osobhandClaude Opus 5.5
5935e13866
SZIP-filtered datasets from libhdf5 came back as garbage or zeros with no error (ref_szip.h5, h5repack_szip.h5, noencoder.h5, le_data/be_data Szip_float_data_*), and 64-bit ones failed with "invalid bits per sample" (h5wasm compressed.h5). The decoder called aec_buffer_decode directly, but libhdf5 goes through szlib's SZ_BufftoBuffDecompress (H5Zszip.c), which libaec implements with reshaping (sz_compat.c). Differences, all fixed: - H5Zszip.c prefixes the stream with the 4-byte LE uncompressed size; it was fed to libaec as data. - 32- and 64-bit samples are coded as byte planes of 8-bit samples and must be de-interleaved. - The reference sample interval is ceil(pixels_per_scanline / pixels_per_block), not a fixed 128. - Scanlines that are not a whole number of blocks are padded and must be unpadded. - Byte order comes from the MSB option bit; LE data was decoded as MSB. Test: szip_decodes_libhdf5_chunks_exactly compares chunks from HDF Group test files (noencoder.h5, le_data.h5) and an h5py-written file (64-bit, 16-bit, padded scanlines, NN and EC) byte for byte with h5py's values; it failed before on the first case. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5-format
Pure-Rust HDF5 binary format parsing and writing — no C dependencies.
Features
- Zero-copy superblock, object header, and B-tree parsing
- Chunked dataset read/write with filter pipelines
no_stdsupport (disablestdfeature)- Optional parallel reads via Rayon
- SHA-256 provenance tracking
Usage
use clawhdf5_format::Superblock;
let data = std::fs::read("data.h5").unwrap();
let sb = Superblock::from_bytes(&data).unwrap();
println!("HDF5 version {}.{}", sb.version_major(), sb.version_minor());
License
MIT