osobhandClaude Fable 5.1
c6a7bbfc67
read_raw_data_selection computed which chunks a selection intersects, threw the answer away, decoded the entire dataset and picked elements out of it — for contiguous layouts too. A 64x64 window of a 64 MB deflate dataset cost 105 ms, about half a full read; every selection cost the same whatever its size. New partial_read module: materialise only the selection's bounding box — the overlapping rows of a contiguous dataset (straight from the file bytes) or the overlapping chunks (only those are decompressed) — then run the existing extractor over that buffer with the selection translated to the box origin, so extraction semantics are exactly the full-read ones. It declines (falling back to the old path) for All/None, compact/virtual/storage-less layouts, and boxes covering more than half the dataset. That window now takes 0.39 ms, one row 2.7 ms, one column 5.2 ms. Selections are validated against the dataset shape first. They were not: a hyperslab past an edge came back padded with zeros and a point with an out-of-range column wrapped into the next row, returning the wrong element with no error. Now FormatError::SelectionOutOfBounds (also rank mismatch and overlapping blocks); the facade's fill-aware path validates too. Tests: equivalence against a reference extraction from a full read over 60 random hyperslabs/point lists per layout (contiguous, chunked, deflate) for ranks 1-3. New read_harness bench binary with before/after in BENCHMARKS.md. Co-Authored-By: Claude Fable 5.1 <[email protected]>
clawhdf5-format
Pure-Rust HDF5 binary format parsing and writing — no C dependencies.
Features
- Zero-copy superblock, object header, and B-tree parsing
- Chunked dataset read/write with filter pipelines
no_stdsupport (disablestdfeature)- Optional parallel reads via Rayon
- SHA-256 provenance tracking
Usage
use clawhdf5_format::Superblock;
let data = std::fs::read("data.h5").unwrap();
let sb = Superblock::from_bytes(&data).unwrap();
println!("HDF5 version {}.{}", sb.version_major(), sb.version_minor());
License
MIT