osobhandClaude Opus 5.5
f0db817678
Full reads of a chunked dataset (the cached reader behind the facade's read_* and the uncached one behind mmap/lazy files and verify_provenance) now decode each chunk into this thread's reusable scratch buffers and copy it straight to its place in the output. Before, the cached reader decoded batches of 128 chunks into fresh Vecs and the uncached one decoded every chunk of the dataset into its own buffer before assembling any: a new 256 KiB allocation (and its page faults) per chunk and per filter stage. Chunks are still inserted into the file's chunk cache when the whole dataset fits in it. With the parallel feature the calling thread now decodes too, sharing the chunks with whichever rayon workers are free (run_with_helpers): a helper the busy pool only starts after the read is done returns at once. Before, the caller handed every chunk to the pool and slept, so readers outside a small pool (2-4 threads) queued behind its workers; with a one-thread pool the reads went sequential. Chunks are placed concurrently only when the index puts them on the chunk grid at distinct places (a corrupt index is read one chunk at a time), and the error returned is still the first failing chunk's. Fix: a chunk stored unfiltered in a filtered dataset (every filter-mask bit set) that is shorter than a chunk read as zeros where its data was missing through the cached reader (the facade's read_*); it is now an error naming the chunk, as the uncached reader already made it. Regression tests, both failing before this change: tests/busy_decode_pool.rs (both workers of a two-thread pool busy, four readers) and short_unfiltered_chunk_of_a_filtered_dataset_is_an_error. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5-format
Pure-Rust HDF5 binary format parsing and writing — no C dependencies.
Features
- Zero-copy superblock, object header, and B-tree parsing
- Chunked dataset read/write with filter pipelines
no_stdsupport (disablestdfeature)- Optional parallel reads via Rayon
- SHA-256 provenance tracking
Usage
use clawhdf5_format::Superblock;
let data = std::fs::read("data.h5").unwrap();
let sb = Superblock::from_bytes(&data).unwrap();
println!("HDF5 version {}.{}", sb.version_major(), sb.version_minor());
License
MIT