quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

perf(agent): open a store without copying the whole file

Browse Source 
`read_from_disk` memory-mapped the file and then copied the entire
mapping into a `Vec` to hand to `File::from_bytes` — but `File::open`
memory-maps it itself whenever the facade's `mmap` feature is on, which
it is by default. So every open mapped the file, memcpy'd all of it, and
parsed the copy.

Store open at 100k x 384: 455 ms -> 327 ms, about 28% faster (two runs
after the change, 326.8 and 328.1 ms).

Peak memory is unchanged, which is worth saying because the opposite is
the natural assumption. The footprint harness now tracks a high-water
mark next to the retained figure, and it shows the peak falling after
the parse, during the index build — so a buffer allocated and freed
inside the parse never reaches it. Confirmed rather than assumed:
holding a deliberate extra copy of the whole file across the parse
leaves the peak exactly where it was, which is also what proved the
instrument was working before trusting its answer.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-20 17:49:47 -07:00
co-authored by Claude Opus 5
parent a91df3f1c3
commit 1d767e3b93
4 changed files with 80 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/clawhdf5-agent/src/storage.rs
+6 -10
View File
@@ -113,13 +113,11 @@ pub type StoreState = (MemoryConfig, MemoryCache, SessionCache, KnowledgeCache);
/// [`read_from_disk`], plus the checkpoint's [`WalMark`] (if any) so the
/// caller can skip WAL entries this file already contains.
pub fn read_from_disk_with_mark(path: &Path) -> Result<(StoreState, Option<WalMark>), MemoryError> {
let mmap = clawhdf5_io::MmapReader::open(path).map_err(MemoryError::Io)?;
// Advise the OS we'll need the whole file for parsing
mmap.advise_willneed(0, mmap.len());
// Parse the HDF5 file from the mmap'd bytes
let file = clawhdf5::File::from_bytes(mmap.as_bytes().to_vec())
// `File::open` memory-maps the file itself (the facade's `mmap` feature is
// on by default). Mapping it here and handing over `as_bytes().to_vec()`
// did the same work and then copied the whole store — a second full copy
// of the file, live for the whole parse, on top of the mapping.
let file = clawhdf5::File::open(path)
.map_err(|e| MemoryError::Hdf5(format!("cannot open {}: {e}", path.display())))?;
let (mut config, cache, sessions, knowledge) = schema::validate_and_load(&file)?;
@@ -133,9 +131,7 @@ pub fn read_from_disk_with_mark(path: &Path) -> Result<(StoreState, Option<WalMa
pub fn read_from_disk_with_meta(
path: &Path,
) -> Result<(StoreState, schema::CheckpointMeta), MemoryError> {
let mmap = clawhdf5_io::MmapReader::open(path).map_err(MemoryError::Io)?;
mmap.advise_willneed(0, mmap.len());
let file = clawhdf5::File::from_bytes(mmap.as_bytes().to_vec())
let file = clawhdf5::File::open(path)
.map_err(|e| MemoryError::Hdf5(format!("cannot open {}: {e}", path.display())))?;
let (mut config, cache, sessions, knowledge) = schema::validate_and_load(&file)?;
config.path = path.to_path_buf();