clawhdf5: File::open_storage reads any Storage through the full API

File::open_storage(Arc<dyn Storage + Send + Sync>) opens a file served by
any backend: groups, datasets, attributes, read_*, selections, VL data
and virtual datasets (external sources through the new
File::set_vds_resolver) all read through Storage::read_at/read_ranges.
The file's view (user block skipped, bounded by the recorded end of
file) is itself a Storage; File::open and File::from_bytes keep their
mmap and in-memory paths, now as that view's as_contiguous() fast path.
A storage-backed file's metadata cache image is laid over each read it
covers (new CacheImage::entries), as libhdf5 loads it.

The typed readers keep their fast paths over any storage: a contiguous
dataset is read in one piece and converted (read_f64 and friends), and a
contiguous native selection reads only its runs
(data_read::read_selection_native_in). Zero-copy methods answer
ContiguousStorageRequired when the bytes are not in memory, and
File::as_bytes panics there (File::contiguous_bytes is the fallible
form).

Test: tests/storage_equivalence.rs reads every fixture, and with
CLAWHDF5_STORAGE_CORPUS every conformance-corpus file, through File::open
and through open_storage over a read_at-only CountingStorage — tree,
attributes, and every dataset's values several ways — and requires
identical transcripts; it prints the read_at calls and bytes a one-pass
read costs per file.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 16:41:47 -05:00
co-authored by Claude Opus 5.5
parent 3fa5ed1dda
commit 1c3ef98828
7 changed files with 869 additions and 93 deletions
@@ -490,6 +490,23 @@ impl CacheImage {
image_block_in(file, self.location)
}
/// Every entry as `(file address, its bytes)`, taken from `block` (the
/// image block, see [`Self::block_in`]), in the order [`Self::apply`]
/// writes them: for a reader that cannot write the image over the
/// file's bytes and lays the entries over each read instead.
pub fn entries<'b>(&self, block: &'b [u8]) -> Result<Vec<(u64, &'b [u8])>, FormatError> {
let short = || FormatError::InvalidCacheImage("image applied to the wrong file");
self.entries
.iter()
.map(|e| {
let src = block
.get(e.image_offset..e.image_offset + e.len)
.ok_or_else(short)?;
Ok((e.address, src))
})
.collect()
}
/// Write every entry over `dst`, the file's bytes from the superblock
/// on (as long as the `data` the image was decoded from), taking the
/// entries from `block` (the image block, see [`Self::block`]). `block`