osobhandClaude Opus 5.5
a5bd70216c
HDF5 stores every chunk at the full chunk size (edge chunks are padded before filtering, and with "don't filter partial edge chunks" they are stored raw at full size), so a filter pipeline that decodes to fewer bytes means a corrupt chunk. Every chunk reader padded it with zeros and returned it as data. libhdf5 returns the rest uninitialised, or fails when the filter checks (Blosc with nbytes = 0). New filters::decompress_chunk_exact decodes and then requires exactly the chunk size, with the chunk's coordinates in the error (ChunkedReadError "chunk at [16] decoded to 16 bytes, expected 32"). It replaces decompress_chunk_masked at every chunk read path: the full read (sequential and lane-partitioned), the cached read, the sweep read, the planned-selection read, parallel_read's three decoders and partial_read's box read. decompress_chunk_masked is unchanged (fractal-heap huge objects already checked their own size). Blosc also rejects a frame declaring no data where the chunk size is known. Tests, each failing with the check disabled: filters and parallel_read unit tests; h5py_short_decoded_chunk_is_an_error (gzip chunks rewritten short with write_direct_chunk: 1-D, a 2-D edge chunk, and 40 chunks with shuffle, read through File full/cached/selection reads, a selection that avoids the chunk still reads, MmapFile and LazyFile, with and without the parallel feature); plugin_filters_interop short_decoding_chunks_are_errors (Blosc nbytes=0 and short, LZF and bzip2 short; the Blosc nbytes=0 case read as 16 zeros before). The existing don't-filter-partial-edge-chunks tests still pass. Conformance (tank, 2026-09-26): 573 of 697 ok, and no file changed class, reader result or first issue against the pre-fix run. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5-format
Pure-Rust HDF5 binary format parsing and writing — no C dependencies.
Features
- Zero-copy superblock, object header, and B-tree parsing
- Chunked dataset read/write with filter pipelines
no_stdsupport (disablestdfeature)- Optional parallel reads via Rayon
- SHA-256 provenance tracking
Usage
use clawhdf5_format::Superblock;
let data = std::fs::read("data.h5").unwrap();
let sb = Superblock::from_bytes(&data).unwrap();
println!("HDF5 version {}.{}", sb.version_major(), sb.version_minor());
License
MIT