fix: apply a metadata cache image without copying the file
apply_cache_image returned a copy of the whole file with the image's
entries written in, and File (mmap by default), MmapFile and LazyFile
used that copy for every read: opening a 1 GiB sparse file with an image
needed 2 GB of memory, and an 8 GiB one aborted the process, where
de2a53f (which ignored the image) opened them in a few MB.
The metadata parsers read one contiguous slice, so the image still has
to be laid over the file's bytes; it is now laid over a private copy
that costs only the pages it touches:
- clawhdf5_format::superblock_ext::CacheImage decodes the image into an
entry list (address, offset in the block, length) and applies it to
any destination; cache_image_state tells an opener whether the file
has no image, a loadable one, or one libhdf5 cannot load;
apply_cache_image_in_place is for readers that own their buffer.
apply_cache_image and metadata_view (which copied) are gone.
- clawhdf5_io::HDF5Read::private_copy returns a writable private copy
of a reader's bytes: MmapReader gives a MAP_PRIVATE copy-on-write
mapping (memmap2 map_copy), so only the pages the entries land on are
copied; the default copies the bytes (in-memory readers).
- File, MmapFile and LazyFile write the image into that mapping
(crate::cache_image). File::from_bytes / open_buffered patch their own
buffer in place, copying only the image block, as libhdf5 does. A
file without an image is read straight from the mapping, unchanged.
An image entry that runs past the end of file is now refused: libhdf5
checks only that it starts inside the file, and the images libhdf5
writes never do this, but those bytes have nowhere to go in a view of
the file.
Tests: tests/cache_image_memory.rs has libhdf5 (through ctypes) add an
image to a 1 GiB sparse file and bounds resident-memory growth for all
three openers at 256 MiB; it fails on the previous commit (File::open
grew 2,148,720,640 bytes). reader.rs zero_copy_tests check that a file
without an image is read from the mapping itself and that an image goes
into a copy-on-write mapping, not a heap copy; clawhdf5-io checks that
private_copy writes never reach the reader or the file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+13
-3
@@ -13,9 +13,19 @@ ZFP filters; the 2 mismatches are the known h5py big-endian VL bug.
|
||||
bytes; in `h5clear_mdc_image.h5` the root group exists only there, and
|
||||
every reader failed with `InvalidObjectHeaderVersion(0)`. `File`,
|
||||
`MmapFile` and `LazyFile` (and `h5rs`) now apply the image at open
|
||||
(`clawhdf5_format::superblock_ext`), with libhdf5's checks. A file whose
|
||||
image libhdf5 cannot load opens in libhdf5 but nothing in it can be read;
|
||||
`File::open` refuses it.
|
||||
(`clawhdf5_format::superblock_ext::CacheImage`), with libhdf5's checks.
|
||||
The file is not copied to do it: a mapped file gets the image's entries
|
||||
written into a private copy-on-write mapping
|
||||
(`clawhdf5_io::HDF5Read::private_copy`, `MAP_PRIVATE`), so only the pages
|
||||
they land on are copied, and a buffer the opener owns (`File::from_bytes`,
|
||||
`open_buffered`) is patched in place; files without an image are read
|
||||
from the mapping exactly as before. (An interim version copied the whole
|
||||
file onto the heap: 2 GB of memory to open a 1 GiB sparse file with an
|
||||
image, and an abort for an 8 GiB one; `tests/cache_image_memory.rs`
|
||||
guards it.) An image entry that runs past the end of file is refused
|
||||
(libhdf5 checks only its start; the images it writes never do this). A
|
||||
file whose image libhdf5 cannot load opens in libhdf5 but nothing in it
|
||||
can be read; `File::open` refuses it.
|
||||
- **The superblock extension is decoded at open, as libhdf5 does:** a File
|
||||
Space Info or Metadata Cache Image message libhdf5 cannot decode makes the
|
||||
open fail (`cve-2020-10810`, `cve-2020-10812` were opened).
|
||||
|
||||
Reference in New Issue
Block a user