clawhdf5: File::open_storage reads any Storage through the full API
File::open_storage(Arc<dyn Storage + Send + Sync>) opens a file served by any backend: groups, datasets, attributes, read_*, selections, VL data and virtual datasets (external sources through the new File::set_vds_resolver) all read through Storage::read_at/read_ranges. The file's view (user block skipped, bounded by the recorded end of file) is itself a Storage; File::open and File::from_bytes keep their mmap and in-memory paths, now as that view's as_contiguous() fast path. A storage-backed file's metadata cache image is laid over each read it covers (new CacheImage::entries), as libhdf5 loads it. The typed readers keep their fast paths over any storage: a contiguous dataset is read in one piece and converted (read_f64 and friends), and a contiguous native selection reads only its runs (data_read::read_selection_native_in). Zero-copy methods answer ContiguousStorageRequired when the bytes are not in memory, and File::as_bytes panics there (File::contiguous_bytes is the fallible form). Test: tests/storage_equivalence.rs reads every fixture, and with CLAWHDF5_STORAGE_CORPUS every conformance-corpus file, through File::open and through open_storage over a read_at-only CountingStorage — tree, attributes, and every dataset's values several ways — and requires identical transcripts; it prints the read_at calls and bytes a one-pass read costs per file. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -490,6 +490,23 @@ impl CacheImage {
|
||||
image_block_in(file, self.location)
|
||||
}
|
||||
|
||||
/// Every entry as `(file address, its bytes)`, taken from `block` (the
|
||||
/// image block, see [`Self::block_in`]), in the order [`Self::apply`]
|
||||
/// writes them: for a reader that cannot write the image over the
|
||||
/// file's bytes and lays the entries over each read instead.
|
||||
pub fn entries<'b>(&self, block: &'b [u8]) -> Result<Vec<(u64, &'b [u8])>, FormatError> {
|
||||
let short = || FormatError::InvalidCacheImage("image applied to the wrong file");
|
||||
self.entries
|
||||
.iter()
|
||||
.map(|e| {
|
||||
let src = block
|
||||
.get(e.image_offset..e.image_offset + e.len)
|
||||
.ok_or_else(short)?;
|
||||
Ok((e.address, src))
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Write every entry over `dst`, the file's bytes from the superblock
|
||||
/// on (as long as the `data` the image was decoded from), taking the
|
||||
/// entries from `block` (the image block, see [`Self::block`]). `block`
|
||||
|
||||
Reference in New Issue
Block a user