fix: apply a metadata cache image without copying the file

apply_cache_image returned a copy of the whole file with the image's
entries written in, and File (mmap by default), MmapFile and LazyFile
used that copy for every read: opening a 1 GiB sparse file with an image
needed 2 GB of memory, and an 8 GiB one aborted the process, where
de2a53f (which ignored the image) opened them in a few MB.

The metadata parsers read one contiguous slice, so the image still has
to be laid over the file's bytes; it is now laid over a private copy
that costs only the pages it touches:

- clawhdf5_format::superblock_ext::CacheImage decodes the image into an
  entry list (address, offset in the block, length) and applies it to
  any destination; cache_image_state tells an opener whether the file
  has no image, a loadable one, or one libhdf5 cannot load;
  apply_cache_image_in_place is for readers that own their buffer.
  apply_cache_image and metadata_view (which copied) are gone.
- clawhdf5_io::HDF5Read::private_copy returns a writable private copy
  of a reader's bytes: MmapReader gives a MAP_PRIVATE copy-on-write
  mapping (memmap2 map_copy), so only the pages the entries land on are
  copied; the default copies the bytes (in-memory readers).
- File, MmapFile and LazyFile write the image into that mapping
  (crate::cache_image). File::from_bytes / open_buffered patch their own
  buffer in place, copying only the image block, as libhdf5 does. A
  file without an image is read straight from the mapping, unchanged.

An image entry that runs past the end of file is now refused: libhdf5
checks only that it starts inside the file, and the images libhdf5
writes never do this, but those bytes have nowhere to go in a view of
the file.

Tests: tests/cache_image_memory.rs has libhdf5 (through ctypes) add an
image to a 1 GiB sparse file and bounds resident-memory growth for all
three openers at 256 MiB; it fails on the previous commit (File::open
grew 2,148,720,640 bytes). reader.rs zero_copy_tests check that a file
without an image is read from the mapping itself and that an image goes
into a copy-on-write mapping, not a heap copy; clawhdf5-io checks that
private_copy writes never reach the reader or the file.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 11:42:43 -05:00
co-authored by Claude Opus 5.5
parent a6ed3a5c7d
commit 60502593b7
12 changed files with 695 additions and 176 deletions
+17 -13
View File
@@ -718,8 +718,8 @@ fn main() {
// root group — so a file whose image it cannot load still opens and
// every object fails; File::open refuses such a file outright. The
// probe records the image's error where libhdf5 reports it.
use clawhdf5_format::superblock_ext;
let ext = match guarded(|| superblock_ext::read_superblock_extension(hdf5, &sb).map_err(e)) {
use clawhdf5_format::superblock_ext::{self, CacheImageState};
let state = match guarded(|| superblock_ext::cache_image_state(hdf5, &sb).map_err(e)) {
Ok(x) => x,
Err(msg) => {
top.insert("open_error".into(), Value::String(msg));
@@ -728,18 +728,22 @@ fn main() {
}
};
let mut image_error = None;
let view = match ext.and_then(|x| x.cache_image) {
None => None,
Some(loc) => match guarded(|| {
superblock_ext::apply_cache_image(hdf5, loc, &sb)
.map_err(e)
}) {
Ok(v) => Some(v),
Err(msg) => {
image_error = Some(msg);
None
let view = match state {
CacheImageState::Absent => None,
CacheImageState::Unloadable(err) => {
image_error = Some(e(err));
None
}
CacheImageState::Loaded(image) => {
let mut v = hdf5.to_vec();
match image.block(hdf5).and_then(|b| image.apply(b, &mut v)) {
Ok(()) => Some(v),
Err(err) => {
image_error = Some(e(err));
None
}
}
},
}
};
let hdf5: &[u8] = view.as_deref().unwrap_or(hdf5);
top.insert("superblock_version".into(), json!(sb.version));