osobhandClaude Opus 5.5
e162c013fd
Every decode stage was capped at the chunk's decoded size. That holds only when every filter ahead of the codec preserves size; Fletcher32 does not (it appends a 4-byte checksum), so a pipeline with Fletcher32 before deflate (NetCDF-4's fletcher32 -> shuffle -> deflate ordering, h5repack's "all filters") failed with "deflate: output exceeds size limit" on every chunk. decompress_chunk_masked now computes each stage's bound by running the chunk size forward through the filters that precede it in write order (and that the chunk's mask did not skip): shuffle keeps the size, Fletcher32 adds 4, any codec adds at most n/8 + 64. The cap is still a small constant factor of the chunk, so a decompression bomb is rejected as before (tested). Shuffle also had to learn libhdf5's handling of a length that is not a whole number of elements (chunk + checksum): shuffle the whole elements and leave the trailing bytes in place, in both directions. It used to refuse such data. Regression: h5py_fletcher32_before_deflate_reads (fletcher->shuffle-> gzip, fletcher->gzip, shuffle->fletcher->gzip, and a 2-D i32 grid), fletcher32_ahead_of_deflate_stays_bounded and shuffle_leaves_a_partial_trailing_element_in_place. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5-format
Pure-Rust HDF5 binary format parsing and writing — no C dependencies.
Features
- Zero-copy superblock, object header, and B-tree parsing
- Chunked dataset read/write with filter pipelines
no_stdsupport (disablestdfeature)- Optional parallel reads via Rayon
- SHA-256 provenance tracking
Usage
use clawhdf5_format::Superblock;
let data = std::fs::read("data.h5").unwrap();
let sb = Superblock::from_bytes(&data).unwrap();
println!("HDF5 version {}.{}", sb.version_major(), sb.version_minor());
License
MIT