feat(format): apply fill values to unallocated storage on read

HDF5 allocates lazily: a chunk nobody wrote doesn't exist in the file, and a
dataset nobody wrote has no data address. Such regions must read as the
dataset's fill value. There was no Fill Value message parser at all, so:

- a sparse chunked dataset read its holes as zeros — silently wrong whenever
  the fill value isn't zero (h5py `fillvalue=-1` came back as 0);
- a dataset that was created but never written failed with NoDataAllocated /
  "no address for chunked layout" where h5py returns a filled array.

New clawhdf5_format::fill_value: parses Fill Value messages v1-v3 and the old
0x0004 message (validated against HDF5 2.0 output under default and latest
libver), builds a fully filled dataset when there is no storage, and writes the
fill value into exactly the chunk-grid cells absent from the chunk index —
never mistaking a stored zero for a hole, clipping edge chunks, any rank. It is
skipped entirely for the default (zero) fill value. The chunk index dispatch is
extracted from read_chunked_data into a reusable list_chunks.

The reader, lazy and mmap facades apply it on full reads; selection reads go
through a fill-aware full read when the fill value matters. h5py interop test
compares against h5py's own readback, including a sparse 2-D dataset and a
hyperslab straddling allocated and unallocated chunks.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
osobh
2026-09-19 06:35:47 -07:00
co-authored by Claude Fable 5.1
parent 81e8294048
commit 12847c6c66
8 changed files with 582 additions and 20 deletions
+30 -6
View File
@@ -362,15 +362,17 @@ pub fn generate_implicit_chunks(
}
/// Read a chunked dataset, decompressing chunks as needed.
pub fn read_chunked_data(
/// Every allocated chunk of a chunked dataset, for any supported chunk index,
/// plus the spatial chunk dimensions. Chunks the file never allocated (sparse
/// datasets) are simply absent from the list.
pub fn list_chunks(
file_data: &[u8],
layout: &DataLayout,
dataspace: &Dataspace,
datatype: &Datatype,
pipeline: Option<&FilterPipeline>,
elem_size: usize,
offset_size: u8,
length_size: u8,
) -> Result<Vec<u8>, FormatError> {
) -> Result<(Vec<ChunkInfo>, Vec<usize>), FormatError> {
let (
chunk_dimensions,
version,
@@ -404,8 +406,6 @@ pub fn read_chunked_data(
let addr = addr_opt
.ok_or_else(|| FormatError::ChunkedReadError("no address for chunked layout".into()))?;
let elem_size = datatype.type_size() as usize;
// Both v3 and v4 include element size as last dim (rank+1)
let ndims = chunk_dimensions.len();
let rank = ndims
@@ -494,6 +494,30 @@ pub fn read_chunked_data(
}
};
Ok((chunks, chunk_dims))
}
pub fn read_chunked_data(
file_data: &[u8],
layout: &DataLayout,
dataspace: &Dataspace,
datatype: &Datatype,
pipeline: Option<&FilterPipeline>,
offset_size: u8,
length_size: u8,
) -> Result<Vec<u8>, FormatError> {
let elem_size = datatype.type_size() as usize;
let (chunks, chunk_dims) = list_chunks(
file_data,
layout,
dataspace,
elem_size,
offset_size,
length_size,
)?;
let rank = chunk_dims.len();
let ds_dims: Vec<usize> = dataspace.dimensions.iter().map(|&d| d as usize).collect();
// Assemble output
let total_bytes = checked_byte_len(dataspace.checked_num_elements()?, elem_size)?;
if total_bytes == 0 {