osobhandClaude Opus 5.5
22dc87b07c
B2ND chunks were decoded whole, padding included, with up to 16x the HDF5 chunk size as their limit, so a crafted frame made each chunk allocate and fill up to 16x the output (4 GiB for a 256 MiB HDF5 chunk). The padding is the real bound (prod(ceil(c/b)*b) per chunk), but that can be 2^ndim times the array, so it is no longer held at all: - A Blosc2 chunk is now decoded block by block (decode_blocks), each block handed to a sink as it is ready, with at most three blocks of scratch. blosc2_decompress_chunk and plain frames still collect every block. - reassemble places each B2ND block straight into the output and skips blocks that are all padding (they are not decoded unless the delta filter needs the first block). A frame's NaN chunks are handed over one B2ND block at a time and its zero chunks cost nothing. - A B2ND chunk must decode to exactly its padded size, its Blosc2 blocks must be whole B2ND blocks no larger than the output, and a chunk may not be larger than the array (hdf5-blosc2's chunk is the array), so a block is never larger than the output. Peak allocation for a 10-D array padded to 13x (NaN, repeated-value and stored-block chunks) and for a 16x chunk was 4.5 MB and 17.8 MB for 315 KB and 1 MiB outputs before, and is now within the tests' bound. Blosc2 files written by hdf5plugin in 9-D and 12-D, an 8 MiB single chunk, 1x1x1 and edge-chunk shapes still read exactly. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5-format
Pure-Rust HDF5 binary format parsing and writing — no C dependencies.
Features
- Zero-copy superblock, object header, and B-tree parsing
- Chunked dataset read/write with filter pipelines
no_stdsupport (disablestdfeature)- Optional parallel reads via Rayon
- SHA-256 provenance tracking
Usage
use clawhdf5_format::Superblock;
let data = std::fs::read("data.h5").unwrap();
let sb = Superblock::from_bytes(&data).unwrap();
println!("HDF5 version {}.{}", sb.version_major(), sb.version_minor());
License
MIT