feat(format): apply fill values to unallocated storage on read
HDF5 allocates lazily: a chunk nobody wrote doesn't exist in the file, and a dataset nobody wrote has no data address. Such regions must read as the dataset's fill value. There was no Fill Value message parser at all, so: - a sparse chunked dataset read its holes as zeros — silently wrong whenever the fill value isn't zero (h5py `fillvalue=-1` came back as 0); - a dataset that was created but never written failed with NoDataAllocated / "no address for chunked layout" where h5py returns a filled array. New clawhdf5_format::fill_value: parses Fill Value messages v1-v3 and the old 0x0004 message (validated against HDF5 2.0 output under default and latest libver), builds a fully filled dataset when there is no storage, and writes the fill value into exactly the chunk-grid cells absent from the chunk index — never mistaking a stored zero for a hole, clipping edge chunks, any rank. It is skipped entirely for the default (zero) fill value. The chunk index dispatch is extracted from read_chunked_data into a reusable list_chunks. The reader, lazy and mmap facades apply it on full reads; selection reads go through a fill-aware full read when the fill value matters. h5py interop test compares against h5py's own readback, including a sparse 2-D dataset and a hyperslab straddling allocated and unallocated chunks. Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
81e8294048
commit
12847c6c66
@@ -362,15 +362,17 @@ pub fn generate_implicit_chunks(
|
||||
}
|
||||
|
||||
/// Read a chunked dataset, decompressing chunks as needed.
|
||||
pub fn read_chunked_data(
|
||||
/// Every allocated chunk of a chunked dataset, for any supported chunk index,
|
||||
/// plus the spatial chunk dimensions. Chunks the file never allocated (sparse
|
||||
/// datasets) are simply absent from the list.
|
||||
pub fn list_chunks(
|
||||
file_data: &[u8],
|
||||
layout: &DataLayout,
|
||||
dataspace: &Dataspace,
|
||||
datatype: &Datatype,
|
||||
pipeline: Option<&FilterPipeline>,
|
||||
elem_size: usize,
|
||||
offset_size: u8,
|
||||
length_size: u8,
|
||||
) -> Result<Vec<u8>, FormatError> {
|
||||
) -> Result<(Vec<ChunkInfo>, Vec<usize>), FormatError> {
|
||||
let (
|
||||
chunk_dimensions,
|
||||
version,
|
||||
@@ -404,8 +406,6 @@ pub fn read_chunked_data(
|
||||
let addr = addr_opt
|
||||
.ok_or_else(|| FormatError::ChunkedReadError("no address for chunked layout".into()))?;
|
||||
|
||||
let elem_size = datatype.type_size() as usize;
|
||||
|
||||
// Both v3 and v4 include element size as last dim (rank+1)
|
||||
let ndims = chunk_dimensions.len();
|
||||
let rank = ndims
|
||||
@@ -494,6 +494,30 @@ pub fn read_chunked_data(
|
||||
}
|
||||
};
|
||||
|
||||
Ok((chunks, chunk_dims))
|
||||
}
|
||||
|
||||
pub fn read_chunked_data(
|
||||
file_data: &[u8],
|
||||
layout: &DataLayout,
|
||||
dataspace: &Dataspace,
|
||||
datatype: &Datatype,
|
||||
pipeline: Option<&FilterPipeline>,
|
||||
offset_size: u8,
|
||||
length_size: u8,
|
||||
) -> Result<Vec<u8>, FormatError> {
|
||||
let elem_size = datatype.type_size() as usize;
|
||||
let (chunks, chunk_dims) = list_chunks(
|
||||
file_data,
|
||||
layout,
|
||||
dataspace,
|
||||
elem_size,
|
||||
offset_size,
|
||||
length_size,
|
||||
)?;
|
||||
let rank = chunk_dims.len();
|
||||
let ds_dims: Vec<usize> = dataspace.dimensions.iter().map(|&d| d as usize).collect();
|
||||
|
||||
// Assemble output
|
||||
let total_bytes = checked_byte_len(dataspace.checked_num_elements()?, elem_size)?;
|
||||
if total_bytes == 0 {
|
||||
|
||||
Reference in New Issue
Block a user