osobhandClaude Opus 5.5
9066d34eaa
A File is Send + Sync and keeps one ChunkCache for all its datasets. The cached readers bound that cache to "the current dataset" with ensure_dataset(addr), then checked, built and read its index and its decompressed chunks in separate lock acquisitions. Two threads reading two chunked datasets interleaved those steps, so one could store its chunk index under the other's binding, or get the other's decompressed chunk for the same coordinate: wrong data, or an index-out-of-bounds panic when the ranks differed (16 threads x 40 reads over 24 datasets panicked on every run). The cache now keeps per-dataset state keyed by chunk-index address: the chunk index, ChunkIndex and ChunkLayout per dataset (held as Arcs, built outside the lock, first writer wins), and decompressed chunks keyed by (address, coordinate). The chunked readers use the new addr-taking methods (chunks_for, chunk_layout_for, get/put_decompressed_in, prefetch_hint_in) exclusively. Memory stays bounded: decompressed data by the existing byte/slot budget across datasets, indexes by at most 64 datasets and 2^20 index entries in total, dropping the least recently used dataset's index first. Switching datasets no longer throws away the other datasets' cached chunks. The address-less methods remain and act on the dataset last bound with ensure_dataset; they are documented as not for concurrent readers. Regression: threads_reading_different_datasets_get_their_own_chunks (crates/clawhdf5/tests/concurrent_chunk_cache.rs), plus cache unit tests datasets_sharing_coordinates_stay_separate, dataset_indexes_are_bounded and concurrent_readers_of_different_datasets_see_their_own_chunks. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5-format
Pure-Rust HDF5 binary format parsing and writing — no C dependencies.
Features
- Zero-copy superblock, object header, and B-tree parsing
- Chunked dataset read/write with filter pipelines
no_stdsupport (disablestdfeature)- Optional parallel reads via Rayon
- SHA-256 provenance tracking
Usage
use clawhdf5_format::Superblock;
let data = std::fs::read("data.h5").unwrap();
let sb = Superblock::from_bytes(&data).unwrap();
println!("HDF5 version {}.{}", sb.version_major(), sb.version_minor());
License
MIT