osobhandClaude Fable 5.1
12847c6c66
HDF5 allocates lazily: a chunk nobody wrote doesn't exist in the file, and a dataset nobody wrote has no data address. Such regions must read as the dataset's fill value. There was no Fill Value message parser at all, so: - a sparse chunked dataset read its holes as zeros — silently wrong whenever the fill value isn't zero (h5py `fillvalue=-1` came back as 0); - a dataset that was created but never written failed with NoDataAllocated / "no address for chunked layout" where h5py returns a filled array. New clawhdf5_format::fill_value: parses Fill Value messages v1-v3 and the old 0x0004 message (validated against HDF5 2.0 output under default and latest libver), builds a fully filled dataset when there is no storage, and writes the fill value into exactly the chunk-grid cells absent from the chunk index — never mistaking a stored zero for a hole, clipping edge chunks, any rank. It is skipped entirely for the default (zero) fill value. The chunk index dispatch is extracted from read_chunked_data into a reusable list_chunks. The reader, lazy and mmap facades apply it on full reads; selection reads go through a fill-aware full read when the fill value matters. h5py interop test compares against h5py's own readback, including a sparse 2-D dataset and a hyperslab straddling allocated and unallocated chunks. Co-Authored-By: Claude Fable 5.1 <[email protected]>
clawhdf5-format
Pure-Rust HDF5 binary format parsing and writing — no C dependencies.
Features
- Zero-copy superblock, object header, and B-tree parsing
- Chunked dataset read/write with filter pipelines
no_stdsupport (disablestdfeature)- Optional parallel reads via Rayon
- SHA-256 provenance tracking
Usage
use clawhdf5_format::Superblock;
let data = std::fs::read("data.h5").unwrap();
let sb = Superblock::from_bytes(&data).unwrap();
println!("HDF5 version {}.{}", sb.version_major(), sb.version_minor());
License
MIT