osobhandClaude Opus 5.5
aef8e766ae
Filter 32004 chunks were framed as a 4-byte little-endian size plus one LZ4 block. That is not the registered HDF5 LZ4 format (H5Zlz4.c: 8-byte big-endian total size, 4-byte big-endian block size, then per block a 4-byte big-endian compressed length and the block, stored raw when the length equals the block size), so libhdf5 + hdf5plugin could not read our LZ4 datasets and we could not read theirs (h5ex_d_lz4.h5: "lz4: 0 is not a valid match offset"). Write the registered format (cd_values[0] is honoured as the block size, default 1 GiB like the plugin) and read it, multi-block and raw blocks included. Chunks in the old framing stay readable: an HDF5 chunk is under 4 GiB, so a registered chunk always starts with four zero bytes and is at least 12 bytes long, while an old one starts with four zero bytes only when empty (5 bytes). Tests: lz4_reads_registered_hdf5_format (chunk of the HDF Group's h5ex_d_lz4.h5, block size 3), lz4_writes_registered_hdf5_format, lz4_reads_legacy_clawhdf5_format, and hdf5plugin_reads_our_lz4 (ignored interop test; failed before with "filter returned failure during read"). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5-format
Pure-Rust HDF5 binary format parsing and writing — no C dependencies.
Features
- Zero-copy superblock, object header, and B-tree parsing
- Chunked dataset read/write with filter pipelines
no_stdsupport (disablestdfeature)- Optional parallel reads via Rayon
- SHA-256 provenance tracking
Usage
use clawhdf5_format::Superblock;
let data = std::fs::read("data.h5").unwrap();
let sb = Superblock::from_bytes(&data).unwrap();
println!("HDF5 version {}.{}", sb.version_major(), sb.version_minor());
License
MIT