osobhandClaude Fable 5.1
05c665a898
Requesting a filter without chunk dimensions made the whole dataset a single chunk. Any read, even one row, then decompresses everything, and a large dataset cannot be decoded in parallel — which also made the new partial reads pointless for such files. auto_chunk_dims keeps datasets up to 1 MiB as one chunk (unchanged behaviour) and splits larger ones by halving the dimensions in turn, so chunks keep roughly the dataset's proportions, until a chunk is at most 1 MiB — h5py's approach. An empty (unlimited, unwritten) dimension is treated as 1024. The writer passes the element size through resolve_chunk_dims_for; the old resolve_chunk_dims assumes 8-byte elements. Explicit with_chunks always wins. Interop test: h5py reads an auto-chunked 13 MB deflate dataset, sees chunks between 128 KiB and 1 MiB, and a small dataset still has one chunk. Co-Authored-By: Claude Fable 5.1 <[email protected]>
clawhdf5-format
Pure-Rust HDF5 binary format parsing and writing — no C dependencies.
Features
- Zero-copy superblock, object header, and B-tree parsing
- Chunked dataset read/write with filter pipelines
no_stdsupport (disablestdfeature)- Optional parallel reads via Rayon
- SHA-256 provenance tracking
Usage
use clawhdf5_format::Superblock;
let data = std::fs::read("data.h5").unwrap();
let sb = Superblock::from_bytes(&data).unwrap();
println!("HDF5 version {}.{}", sb.version_major(), sb.version_minor());
License
MIT