perf(agent): open a store without copying the whole file

`read_from_disk` memory-mapped the file and then copied the entire
mapping into a `Vec` to hand to `File::from_bytes` — but `File::open`
memory-maps it itself whenever the facade's `mmap` feature is on, which
it is by default. So every open mapped the file, memcpy'd all of it, and
parsed the copy.

Store open at 100k x 384: 455 ms -> 327 ms, about 28% faster (two runs
after the change, 326.8 and 328.1 ms).

Peak memory is unchanged, which is worth saying because the opposite is
the natural assumption. The footprint harness now tracks a high-water
mark next to the retained figure, and it shows the peak falling after
the parse, during the index build — so a buffer allocated and freed
inside the parse never reaches it. Confirmed rather than assumed:
holding a deliberate extra copy of the whole file across the parse
leaves the peak exactly where it was, which is also what proved the
instrument was working before trusting its answer.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-20 17:49:47 -07:00
co-authored by Claude Opus 5
parent a91df3f1c3
commit 1d767e3b93
4 changed files with 80 additions and 18 deletions
+25
View File
@@ -123,6 +123,31 @@ vectors, and recall is measured against brute-force ground truth rather than
against the f32 index, whose own approximation errors a re-scored search is
entitled to get right.
### Opening a store (`read_from_disk`)
`HDF5Memory::open` memory-mapped the file, copied the whole mapping into a
`Vec`, and handed that to `File::from_bytes` — while `File::open` memory-maps
the file itself. Dropping the copy takes **store open from 455 ms to 327 ms**
at 100 000 x 384 (`--e2e-only --full`; two runs after the change, 326.8 and
328.1 ms).
It does **not** lower the process's peak memory, which is worth stating
precisely because it is the obvious thing to assume. The harness now reports a
high-water mark alongside the retained figure:
| N | reopened MiB | peak during open MiB |
|---:|---:|---:|
| 1 000 | 4 | 5 |
| 10 000 | 44 | 61 |
| 100 000 | 399 | 562 |
The peak is set *after* the parse, by the index build, so a buffer allocated
and freed during the parse never reaches the high-water mark. Holding a
deliberate extra copy of the file across the whole parse leaves the peak
unmoved, which is how this was confirmed rather than assumed. What the change
saves is the copy itself: a full-file memcpy on every open, and the transient
that goes with it.
## Read harness
Produced by `cargo run --release -p clawhdf5-bench --bin read_harness`: a 4096 x