perf(agent): open a store without copying the whole file
`read_from_disk` memory-mapped the file and then copied the entire mapping into a `Vec` to hand to `File::from_bytes` — but `File::open` memory-maps it itself whenever the facade's `mmap` feature is on, which it is by default. So every open mapped the file, memcpy'd all of it, and parsed the copy. Store open at 100k x 384: 455 ms -> 327 ms, about 28% faster (two runs after the change, 326.8 and 328.1 ms). Peak memory is unchanged, which is worth saying because the opposite is the natural assumption. The footprint harness now tracks a high-water mark next to the retained figure, and it shows the peak falling after the parse, during the index build — so a buffer allocated and freed inside the parse never reaches it. Confirmed rather than assumed: holding a deliberate extra copy of the whole file across the parse leaves the peak exactly where it was, which is also what proved the instrument was working before trusting its answer. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
@@ -123,6 +123,31 @@ vectors, and recall is measured against brute-force ground truth rather than
|
||||
against the f32 index, whose own approximation errors a re-scored search is
|
||||
entitled to get right.
|
||||
|
||||
### Opening a store (`read_from_disk`)
|
||||
|
||||
`HDF5Memory::open` memory-mapped the file, copied the whole mapping into a
|
||||
`Vec`, and handed that to `File::from_bytes` — while `File::open` memory-maps
|
||||
the file itself. Dropping the copy takes **store open from 455 ms to 327 ms**
|
||||
at 100 000 x 384 (`--e2e-only --full`; two runs after the change, 326.8 and
|
||||
328.1 ms).
|
||||
|
||||
It does **not** lower the process's peak memory, which is worth stating
|
||||
precisely because it is the obvious thing to assume. The harness now reports a
|
||||
high-water mark alongside the retained figure:
|
||||
|
||||
| N | reopened MiB | peak during open MiB |
|
||||
|---:|---:|---:|
|
||||
| 1 000 | 4 | 5 |
|
||||
| 10 000 | 44 | 61 |
|
||||
| 100 000 | 399 | 562 |
|
||||
|
||||
The peak is set *after* the parse, by the index build, so a buffer allocated
|
||||
and freed during the parse never reaches the high-water mark. Holding a
|
||||
deliberate extra copy of the file across the whole parse leaves the peak
|
||||
unmoved, which is how this was confirmed rather than assumed. What the change
|
||||
saves is the copy itself: a full-file memcpy on every open, and the transient
|
||||
that goes with it.
|
||||
|
||||
## Read harness
|
||||
|
||||
Produced by `cargo run --release -p clawhdf5-bench --bin read_harness`: a 4096 x
|
||||
|
||||
Reference in New Issue
Block a user