fix: apply a metadata cache image without copying the file
apply_cache_image returned a copy of the whole file with the image's
entries written in, and File (mmap by default), MmapFile and LazyFile
used that copy for every read: opening a 1 GiB sparse file with an image
needed 2 GB of memory, and an 8 GiB one aborted the process, where
de2a53f (which ignored the image) opened them in a few MB.
The metadata parsers read one contiguous slice, so the image still has
to be laid over the file's bytes; it is now laid over a private copy
that costs only the pages it touches:
- clawhdf5_format::superblock_ext::CacheImage decodes the image into an
entry list (address, offset in the block, length) and applies it to
any destination; cache_image_state tells an opener whether the file
has no image, a loadable one, or one libhdf5 cannot load;
apply_cache_image_in_place is for readers that own their buffer.
apply_cache_image and metadata_view (which copied) are gone.
- clawhdf5_io::HDF5Read::private_copy returns a writable private copy
of a reader's bytes: MmapReader gives a MAP_PRIVATE copy-on-write
mapping (memmap2 map_copy), so only the pages the entries land on are
copied; the default copies the bytes (in-memory readers).
- File, MmapFile and LazyFile write the image into that mapping
(crate::cache_image). File::from_bytes / open_buffered patch their own
buffer in place, copying only the image block, as libhdf5 does. A
file without an image is read straight from the mapping, unchanged.
An image entry that runs past the end of file is now refused: libhdf5
checks only that it starts inside the file, and the images libhdf5
writes never do this, but those bytes have nowhere to go in a view of
the file.
Tests: tests/cache_image_memory.rs has libhdf5 (through ctypes) add an
image to a 1 GiB sparse file and bounds resident-memory growth for all
three openers at 256 MiB; it fails on the previous commit (File::open
grew 2,148,720,640 bytes). reader.rs zero_copy_tests check that a file
without an image is read from the mapping itself and that an image goes
into a copy-on-write mapping, not a heap copy; clawhdf5-io checks that
private_copy writes never reach the reader or the file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -41,6 +41,65 @@ pub trait HDF5Read {
|
||||
fn is_empty(&self) -> bool {
|
||||
self.as_bytes().is_empty()
|
||||
}
|
||||
|
||||
/// A private, writable copy of [`Self::as_bytes`]: writes to it stay in
|
||||
/// this process and never reach the underlying storage.
|
||||
///
|
||||
/// Readers use it to lay a file's metadata cache image over the file's
|
||||
/// own metadata. The default copies the bytes; a memory-mapped reader
|
||||
/// returns a copy-on-write mapping instead, so only the pages written to
|
||||
/// are copied and the rest stay shared with the page cache.
|
||||
fn private_copy(&self) -> io::Result<PrivateCopy> {
|
||||
Ok(PrivateCopy::Owned(self.as_bytes().to_vec()))
|
||||
}
|
||||
}
|
||||
|
||||
/// A private, writable copy of a file's bytes (see
|
||||
/// [`HDF5Read::private_copy`]).
|
||||
pub enum PrivateCopy {
|
||||
/// The bytes copied onto the heap.
|
||||
Owned(Vec<u8>),
|
||||
/// A copy-on-write mapping of the file: pages are copied only when
|
||||
/// written to.
|
||||
#[cfg(feature = "mmap")]
|
||||
Mapped(memmap2::MmapMut),
|
||||
}
|
||||
|
||||
impl PrivateCopy {
|
||||
/// Whether this is a copy-on-write mapping rather than a heap copy.
|
||||
pub fn is_mapped(&self) -> bool {
|
||||
!matches!(self, PrivateCopy::Owned(_))
|
||||
}
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for PrivateCopy {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("PrivateCopy")
|
||||
.field("len", &self.len())
|
||||
.field("mapped", &self.is_mapped())
|
||||
.finish()
|
||||
}
|
||||
}
|
||||
|
||||
impl std::ops::Deref for PrivateCopy {
|
||||
type Target = [u8];
|
||||
fn deref(&self) -> &[u8] {
|
||||
match self {
|
||||
PrivateCopy::Owned(v) => v,
|
||||
#[cfg(feature = "mmap")]
|
||||
PrivateCopy::Mapped(m) => m,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl std::ops::DerefMut for PrivateCopy {
|
||||
fn deref_mut(&mut self) -> &mut [u8] {
|
||||
match self {
|
||||
PrivateCopy::Owned(v) => v,
|
||||
#[cfg(feature = "mmap")]
|
||||
PrivateCopy::Mapped(m) => m,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Read-write access to HDF5 data.
|
||||
|
||||
Reference in New Issue
Block a user