quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix: apply a metadata cache image without copying the file

Browse Source 
apply_cache_image returned a copy of the whole file with the image's
entries written in, and File (mmap by default), MmapFile and LazyFile
used that copy for every read: opening a 1 GiB sparse file with an image
needed 2 GB of memory, and an 8 GiB one aborted the process, where
de2a53f (which ignored the image) opened them in a few MB.

The metadata parsers read one contiguous slice, so the image still has
to be laid over the file's bytes; it is now laid over a private copy
that costs only the pages it touches:

- clawhdf5_format::superblock_ext::CacheImage decodes the image into an
  entry list (address, offset in the block, length) and applies it to
  any destination; cache_image_state tells an opener whether the file
  has no image, a loadable one, or one libhdf5 cannot load;
  apply_cache_image_in_place is for readers that own their buffer.
  apply_cache_image and metadata_view (which copied) are gone.
- clawhdf5_io::HDF5Read::private_copy returns a writable private copy
  of a reader's bytes: MmapReader gives a MAP_PRIVATE copy-on-write
  mapping (memmap2 map_copy), so only the pages the entries land on are
  copied; the default copies the bytes (in-memory readers).
- File, MmapFile and LazyFile write the image into that mapping
  (crate::cache_image). File::from_bytes / open_buffered patch their own
  buffer in place, copying only the image block, as libhdf5 does. A
  file without an image is read straight from the mapping, unchanged.

An image entry that runs past the end of file is now refused: libhdf5
checks only that it starts inside the file, and the images libhdf5
writes never do this, but those bytes have nowhere to go in a view of
the file.

Tests: tests/cache_image_memory.rs has libhdf5 (through ctypes) add an
image to a 1 GiB sparse file and bounds resident-memory growth for all
three openers at 256 MiB; it fails on the previous commit (File::open
grew 2,148,720,640 bytes). reader.rs zero_copy_tests check that a file
without an image is read from the mapping itself and that an image goes
into a copy-on-write mapping, not a heap copy; clawhdf5-io checks that
private_copy writes never reach the reader or the file.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 11:42:43 -05:00
co-authored by Claude Opus 5.5
parent a6ed3a5c7d
commit 60502593b7
12 changed files with 695 additions and 176 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/clawhdf5-format/src/superblock_ext.rs
+241 -128
View File
@@ -14,9 +14,11 @@
//! `H5C__reconstruct_cache_contents`), and the entries take the place of
//! the file's bytes at their addresses: the file itself may hold stale or
//! no metadata there (in `h5clear_mdc_image.h5` the root group's header is
//! only in the image). [`apply_cache_image`] does the same with bytes: it
//! returns a copy of the file with every entry written at its address, so
//! every parser reads what libhdf5 reads.
//! only in the image). [`CacheImage::apply`] does the same with bytes: it
//! writes every entry at its address, so every parser reads what libhdf5
//! reads. It writes into whatever the opener gives it — a private
//! copy-on-write mapping of the file, or a buffer the opener owns — so the
//! file is never copied whole.
#[cfg(not(feature = "std"))]
use alloc::{collections::BTreeSet, vec::Vec};
@@ -284,150 +286,243 @@ fn decode_fsinfo(data: &[u8], os: u8, ls: u8) -> Result<FileSpaceInfo, FormatErr
/// One entry of a metadata cache image: `len` bytes at `image_offset` in
/// the image block, belonging at file address `address`.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
struct ImageEntry {
address: u64,
image_offset: usize,
len: usize,
}
/// Decode the metadata cache image at `location` in `data` (the file from
/// the superblock on, up to its recorded end of file) and return a copy of
/// `data` with every cached entry written at its address — the metadata
/// libhdf5 reads for this file. The image is checked as libhdf5 checks it
/// (`H5C__decode_cache_image_header`, `H5C__reconstruct_cache_entry`):
/// signature and version, the image length it records, entry types, rings
/// and ages in range, entry addresses inside the file and not repeated,
/// flush-dependency parents already in the cache.
/// A decoded metadata cache image: where its block is, and the entries it
/// holds. [`CacheImage::apply`] writes the entries over a file's bytes.
///
/// libhdf5 does not verify the block's trailing checksum when it loads an
/// image, so neither does this.
pub fn apply_cache_image(
data: &[u8],
/// Only the entry list is kept, never a copy of the file: an opener that
/// maps the file applies the image to a private copy-on-write mapping, so
/// only the pages the entries land on are copied.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CacheImage {
location: CacheImageLocation,
sb: &Superblock,
) -> Result<Vec<u8>, FormatError> {
let (offset_size, length_size) = (sb.offset_size, sb.length_size);
entries: Vec<ImageEntry>,
}
/// What an opener must do about a file's metadata cache image.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum CacheImageState {
/// The file has no image: its bytes are its metadata.
Absent,
/// The file has an image that loads: apply it with [`CacheImage::apply`].
Loaded(CacheImage),
/// The file has an image libhdf5 fails to load. libhdf5 still opens the
/// file (the image loads at the first metadata read), and that read
/// fails with this error.
Unloadable(FormatError),
}
impl CacheImage {
/// Decode the metadata cache image at `location` in `data` (the file
/// from the superblock on, up to its recorded end of file). The image is
/// checked as libhdf5 checks it (`H5C__decode_cache_image_header`,
/// `H5C__reconstruct_cache_entry`): signature and version, the image
/// length it records, entry types, rings and ages in range, entry
/// addresses inside the file and not repeated, flush-dependency parents
/// already in the cache.
///
/// One check is stricter than libhdf5's: an entry must end inside the
/// file. libhdf5 checks only that it starts there, and serves the rest
/// from the image; the images libhdf5 writes never do this (every entry
/// lies below the image block, which is written last), and the bytes an
/// entry would put past the end of file have nowhere to go in a view of
/// the file.
///
/// libhdf5 does not verify the block's trailing checksum when it loads
/// an image, so neither does this.
pub fn decode(
data: &[u8],
location: CacheImageLocation,
sb: &Superblock,
) -> Result<Self, FormatError> {
let (offset_size, length_size) = (sb.offset_size, sb.length_size);
let bad = FormatError::InvalidCacheImage;
let block = image_block(data, location)?;
let eoa = data.len() as u64;
let mut c = Cursor::new(block, bad(RAN_OFF));
// Header: signature, version, flags, image data length, entry count.
if c.take(4)? != MDCI_SIGNATURE {
return Err(bad("bad metadata cache image header signature"));
}
if c.u8()? != 0 {
return Err(bad("bad metadata cache image version"));
}
if c.u8()? & MDCI_HAVE_RESIZE_STATUS != 0 {
return Err(bad("MDC resize status not yet supported"));
}
if c.uint(length_size)? != location.length {
return Err(bad("bad metadata cache image data length"));
}
let n_entries = c.uint(4)?;
if n_entries == 0 {
return Err(bad("bad metadata cache entry count"));
}
let mut entries = Vec::new();
// What is in libhdf5's cache when it loads the image: the superblock
// and the superblock extension's object header (read to find the
// image). Each entry's flush-dependency parents are looked up in the
// cache as the entry is inserted (`H5C__reconstruct_cache_contents`
// searches the index inside the loop that inserts the entries, in
// HDF5 1.14.6 and 2.0.0 alike), so a parent must be one of those or
// an earlier entry.
let mut cached = BTreeSet::new();
cached.insert(0);
if let Some(ext) = sb.superblock_extension_address {
cached.insert(ext);
}
let mut seen = BTreeSet::new();
for _ in 0..n_entries {
let type_id = c.u8()?;
if type_id >= MDCI_NTYPES {
return Err(bad("type id is out of valid range"));
}
let flags = c.u8()?;
if c.u8()? >= MDCI_RING_NTYPES {
return Err(bad("ring is out of valid range"));
}
if c.u8()? > MDCI_AGE_MAX {
return Err(bad("entry age is out of policy range"));
}
let children = c.uint(2)?;
// libhdf5 checks the parent flag against the child count only in
// debug builds (release builds refuse any entry with children);
// the image format's own rule is checked here.
if (flags & MDCI_ENTRY_IS_FD_PARENT != 0) != (children > 0) {
return Err(bad("flush dependency parent flag and child count disagree"));
}
c.uint(2)?; // dirty dependency children: reset for a read-only open
let parents = c.uint(2)?;
if (flags & MDCI_ENTRY_IS_FD_CHILD != 0) != (parents > 0) {
return Err(bad("flush dependency child flag and parent count disagree"));
}
c.uint(4)?; // LRU rank
let address = c
.addr(offset_size)?
.filter(|&a| a < eoa)
.ok_or(bad("invalid entry address range"))?;
let size = c.uint(length_size)?;
if size == 0 {
return Err(bad("invalid entry size"));
}
for _ in 0..parents {
let parent = c
.addr(offset_size)?
.ok_or(bad("invalid flush dependency parent offset"))?;
if !seen.contains(&parent) && !cached.contains(&parent) {
return Err(bad("fd parent not in cache"));
}
}
let len = usize::try_from(size).map_err(|_| bad(RAN_OFF))?;
let image_offset = c.pos;
c.take(len)?;
if address.checked_add(size).is_none_or(|end| end > eoa) {
return Err(bad("entry extends past the end of file"));
}
if !seen.insert(address) {
return Err(bad("duplicate addresses in cache"));
}
entries.push(ImageEntry {
address,
image_offset,
len,
});
}
Ok(CacheImage { location, entries })
}
/// Where the image block is.
pub fn location(&self) -> CacheImageLocation {
self.location
}
/// The number of entries in the image.
pub fn len(&self) -> usize {
self.entries.len()
}
/// Whether the image has no entries (a decoded image always has some).
pub fn is_empty(&self) -> bool {
self.entries.is_empty()
}
/// The file ranges (address, length) the image's entries replace.
pub fn entry_ranges(&self) -> impl Iterator<Item = (u64, usize)> + '_ {
self.entries.iter().map(|e| (e.address, e.len))
}
/// The image block in `data`, the bytes [`Self::decode`] read it from.
pub fn block<'a>(&self, data: &'a [u8]) -> Result<&'a [u8], FormatError> {
image_block(data, self.location)
}
/// Write every entry over `dst`, the file's bytes from the superblock
/// on (as long as the `data` the image was decoded from), taking the
/// entries from `block` (the image block, see [`Self::block`]). `block`
/// must not alias `dst`: an entry may land on the block itself.
pub fn apply(&self, block: &[u8], dst: &mut [u8]) -> Result<(), FormatError> {
let short = || FormatError::InvalidCacheImage("image applied to the wrong file");
for e in &self.entries {
let src = block
.get(e.image_offset..e.image_offset + e.len)
.ok_or_else(short)?;
let at = usize::try_from(e.address).map_err(|_| short())?;
dst.get_mut(at..at + e.len)
.ok_or_else(short)?
.copy_from_slice(src);
}
Ok(())
}
}
fn image_block(data: &[u8], location: CacheImageLocation) -> Result<&[u8], FormatError> {
let bad = FormatError::InvalidCacheImage;
let start = usize::try_from(location.address).map_err(|_| bad("address out of range"))?;
let len = usize::try_from(location.length).map_err(|_| bad("length out of range"))?;
let block = start
start
.checked_add(len)
.and_then(|end| data.get(start..end))
.ok_or(bad("image block extends past the end of the file"))?;
let eoa = data.len() as u64;
let mut c = Cursor::new(block, bad(RAN_OFF));
// Header: signature, version, flags, image data length, entry count.
if c.take(4)? != MDCI_SIGNATURE {
return Err(bad("bad metadata cache image header signature"));
}
if c.u8()? != 0 {
return Err(bad("bad metadata cache image version"));
}
if c.u8()? & MDCI_HAVE_RESIZE_STATUS != 0 {
return Err(bad("MDC resize status not yet supported"));
}
if c.uint(length_size)? != location.length {
return Err(bad("bad metadata cache image data length"));
}
let n_entries = c.uint(4)?;
if n_entries == 0 {
return Err(bad("bad metadata cache entry count"));
}
let mut entries = Vec::new();
// What is in libhdf5's cache when it loads the image: the superblock
// and the superblock extension's object header (read to find the image).
// Each entry's flush-dependency parents are looked up in the cache as
// the entry is inserted (`H5C__reconstruct_cache_contents` searches the
// index inside the loop that inserts the entries, in HDF5 1.14.6 and
// 2.0.0 alike), so a parent must be one of those or an earlier entry.
let mut cached = BTreeSet::new();
cached.insert(0);
if let Some(ext) = sb.superblock_extension_address {
cached.insert(ext);
}
let mut seen = BTreeSet::new();
for _ in 0..n_entries {
let type_id = c.u8()?;
if type_id >= MDCI_NTYPES {
return Err(bad("type id is out of valid range"));
}
let flags = c.u8()?;
if c.u8()? >= MDCI_RING_NTYPES {
return Err(bad("ring is out of valid range"));
}
if c.u8()? > MDCI_AGE_MAX {
return Err(bad("entry age is out of policy range"));
}
let children = c.uint(2)?;
// libhdf5 checks the parent flag against the child count only in
// debug builds (release builds refuse any entry with children); the
// image format's own rule is checked here.
if (flags & MDCI_ENTRY_IS_FD_PARENT != 0) != (children > 0) {
return Err(bad("flush dependency parent flag and child count disagree"));
}
c.uint(2)?; // dirty dependency children: reset for a read-only open
let parents = c.uint(2)?;
if (flags & MDCI_ENTRY_IS_FD_CHILD != 0) != (parents > 0) {
return Err(bad("flush dependency child flag and parent count disagree"));
}
c.uint(4)?; // LRU rank
let address = c
.addr(offset_size)?
.filter(|&a| a < eoa)
.ok_or(bad("invalid entry address range"))?;
let size = c.uint(length_size)?;
if size == 0 {
return Err(bad("invalid entry size"));
}
for _ in 0..parents {
let parent = c
.addr(offset_size)?
.ok_or(bad("invalid flush dependency parent offset"))?;
if !seen.contains(&parent) && !cached.contains(&parent) {
return Err(bad("fd parent not in cache"));
}
}
let len = usize::try_from(size).map_err(|_| bad(RAN_OFF))?;
let image_offset = c.pos;
c.take(len)?;
if !seen.insert(address) {
return Err(bad("duplicate addresses in cache"));
}
entries.push(ImageEntry {
address,
image_offset,
len,
});
}
let mut out = data.to_vec();
for e in &entries {
// address < eoa <= usize::MAX, and the entry's bytes came from the
// block, so neither conversion nor the sum can fail.
let at = e.address as usize;
let end = at + e.len;
if end > out.len() {
out.resize(end, 0);
}
out[at..end].copy_from_slice(&block[e.image_offset..e.image_offset + e.len]);
}
Ok(out)
.ok_or(bad("image block extends past the end of the file"))
}
/// What a reader must do before reading a file's metadata, in one call:
/// check the superblock extension ([`read_superblock_extension`]) and,
/// when the file has a metadata cache image, return the file's bytes with
/// the image applied ([`apply_cache_image`]). `Ok(None)` means read `data`
/// as it is.
pub fn metadata_view(data: &[u8], sb: &Superblock) -> Result<Option<Vec<u8>>, FormatError> {
/// What an opener must do before reading a file's metadata: check the
/// superblock extension ([`read_superblock_extension`]; an error means
/// libhdf5 refuses to open the file) and decode any metadata cache image
/// ([`CacheImage::decode`]). `data` is the file from the superblock on, up
/// to its recorded end of file.
pub fn cache_image_state(data: &[u8], sb: &Superblock) -> Result<CacheImageState, FormatError> {
match read_superblock_extension(data, sb)? {
Some(SuperblockExtension {
cache_image: Some(location),
..
}) => apply_cache_image(data, location, sb).map(Some),
_ => Ok(None),
}) => Ok(match CacheImage::decode(data, location, sb) {
Ok(image) => CacheImageState::Loaded(image),
Err(e) => CacheImageState::Unloadable(e),
}),
_ => Ok(CacheImageState::Absent),
}
}
/// [`cache_image_state`] for a reader that holds the file's bytes in a
/// buffer of its own: check the superblock extension and write any cache
/// image over `data` in place (only the image block is copied). An image
/// libhdf5 cannot load is an error here: such a reader has no way to open
/// the file and fail each object instead.
pub fn apply_cache_image_in_place(data: &mut [u8], sb: &Superblock) -> Result<(), FormatError> {
match cache_image_state(data, sb)? {
CacheImageState::Absent => Ok(()),
CacheImageState::Unloadable(e) => Err(e),
CacheImageState::Loaded(image) => {
let block = image.block(data)?.to_vec();
image.apply(&block, data)
}
}
}
@@ -435,6 +530,18 @@ pub fn metadata_view(data: &[u8], sb: &Superblock) -> Result<Option<Vec<u8>>, Fo
mod tests {
use super::*;
/// The file's bytes with the image at `loc` applied.
fn apply_cache_image(
data: &[u8],
loc: CacheImageLocation,
sb: &Superblock,
) -> Result<Vec<u8>, FormatError> {
let image = CacheImage::decode(data, loc, sb)?;
let mut out = data.to_vec();
image.apply(image.block(data)?, &mut out)?;
Ok(out)
}
fn sb_v2(ext: u64) -> Superblock {
Superblock {
version: 2,
@@ -651,6 +758,12 @@ mod tests {
let mut len = image(&[(16, b"x")]);
len[6] ^= 1;
assert!(matches!(bad(len), FormatError::InvalidCacheImage(_)));
// An entry that starts inside the file (64 bytes, then a 60-byte
// image) but runs past its end.
assert!(matches!(
bad(image(&[(123, b"8 bytes!")])),
FormatError::InvalidCacheImage("entry extends past the end of file")
));
let mut cut = image(&[(16, b"abcdef")]);
let n = cut.len() as u64 - 8;
cut.truncate(cut.len() - 8);