feat(format): apply fill values to unallocated storage on read
HDF5 allocates lazily: a chunk nobody wrote doesn't exist in the file, and a dataset nobody wrote has no data address. Such regions must read as the dataset's fill value. There was no Fill Value message parser at all, so: - a sparse chunked dataset read its holes as zeros — silently wrong whenever the fill value isn't zero (h5py `fillvalue=-1` came back as 0); - a dataset that was created but never written failed with NoDataAllocated / "no address for chunked layout" where h5py returns a filled array. New clawhdf5_format::fill_value: parses Fill Value messages v1-v3 and the old 0x0004 message (validated against HDF5 2.0 output under default and latest libver), builds a fully filled dataset when there is no storage, and writes the fill value into exactly the chunk-grid cells absent from the chunk index — never mistaking a stored zero for a hole, clipping edge chunks, any rank. It is skipped entirely for the default (zero) fill value. The chunk index dispatch is extracted from read_chunked_data into a reusable list_chunks. The reader, lazy and mmap facades apply it on full reads; selection reads go through a fill-aware full read when the fill value matters. h5py interop test compares against h5py's own readback, including a sparse 2-D dataset and a hyperslab straddling allocated and unallocated chunks. Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
81e8294048
commit
12847c6c66
@@ -67,6 +67,7 @@ pub mod ea_writer;
|
||||
pub mod error;
|
||||
pub mod extensible_array;
|
||||
pub mod file_writer;
|
||||
pub mod fill_value;
|
||||
pub mod filter_pipeline;
|
||||
pub mod filters;
|
||||
mod filters_szip;
|
||||
|
||||
Reference in New Issue
Block a user