osobhandClaude Opus 5.5
5e4aa1c6bf
Two write-side bugs, both present in every release (the first at least since v2.1.0), made libhdf5 refuse files written by clawhdf5. Our own reader ignores both fields, and the interop suites only ever wrote f64 from our side, so nothing here caught them. - Every f32 dataset: "sign bit position out of bounds". The float datatype encoder hard-coded the sign bit's position (bits 8-15 of the class bit field) to 63, which is right only for f64. It is now derived from the type: bit_offset + bit_precision - 1. This covered every agent store's embeddings, norms and activation weights. - Every empty dataset: "invalid dataset size, likely file corruption". It was written with a real address and size 0, which trips libhdf5's `addr + size <= addr` overflow check. An empty contiguous dataset now gets the undefined address, as libhdf5 writes it. This covered every agent store without sessions or a knowledge graph. Agent stores are rewritten in full at each checkpoint, so they become readable at their next checkpoint on a fixed build; other files with f32 or empty datasets need rewriting. Both are recorded in docs/known-issues.md. Tests: the sign position byte for f32/f64, and h5py reading our f32 datasets (plain and chunked + deflate) bit for bit. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5-format
Pure-Rust HDF5 binary format parsing and writing — no C dependencies.
Features
- Zero-copy superblock, object header, and B-tree parsing
- Chunked dataset read/write with filter pipelines
no_stdsupport (disablestdfeature)- Optional parallel reads via Rayon
- SHA-256 provenance tracking
Usage
use clawhdf5_format::Superblock;
let data = std::fs::read("data.h5").unwrap();
let sb = Superblock::from_bytes(&data).unwrap();
println!("HDF5 version {}.{}", sb.version_major(), sb.version_minor());
License
MIT