fix(format): read unmapped VDS elements as the virtual dataset's fill value
Elements of a virtual dataset that no mapping supplies (unmapped regions, a missing source file, a missing source dataset) read as 0 instead of the fill value libhdf5 returns — silent wrong data for any VDS created with a non-zero fillvalue (read-matrix cases 0471/0472: -1 and 7 read as 0). A missing source dataset was an error; libhdf5 reads it as fill. Move VDS assembly into a new vds module following H5Dvirtual.c: vds::read_virtual_dataset takes the dataset's fill value and a VdsFileResolver that can refuse a name, and reports how many elements were unmapped. Sources are read with their own fill value, and a source whose datatype differs from the virtual dataset's is an error (libhdf5 converts). File passes the dataset's fill value, resolves source names against the virtual file's directory, and refuses names that leave it with an error instead of reading them as fill. read_selection on a VDS goes through the same fill-aware path. The raw-read API (read_raw_data_full*) has no fill value, so it now errors for a VDS with unmapped elements instead of guessing zeros. Tests: vds_interop::vds_unmapped_regions_read_as_fill_value (external, same-file, missing file/dataset, sparse source with its own fill, int fill; earliest and latest format) and vds_source_outside_directory_is_an_error_not_fill, both against h5py; integration_test::v4_virtual_dataset_raw_api_refuses_to_guess_the_fill_value. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -275,6 +275,21 @@
|
||||
### Correctness
|
||||
- `clawhdf5-format` virtual datasets (VDS), checked against HDF5 2.0 through
|
||||
h5py (`crates/clawhdf5/tests/vds_interop.rs`):
|
||||
- **Wrong data:** elements no mapping supplies — unmapped regions, and
|
||||
mappings whose source file or dataset is missing — read as 0 instead of
|
||||
the virtual dataset's fill value (e.g. h5py `fillvalue=-1`). Assembly moved
|
||||
to the new `vds` module: `vds::read_virtual_dataset` takes the fill value
|
||||
and a resolver that can refuse a name (`VdsFileResolver`), and `File`
|
||||
passes the dataset's fill value. A missing source *dataset* read as an
|
||||
error; it is fill now, as in libhdf5. Source datasets are read with their
|
||||
own fill value for unallocated chunks, and a source whose datatype differs
|
||||
from the virtual dataset's is an error (libhdf5 converts; we do not).
|
||||
`File` now refuses a source name that leaves the virtual file's directory
|
||||
(`../x.h5`, absolute paths), or any external source of a `File::from_bytes`
|
||||
file, with an error — these used to read as fill.
|
||||
**Behaviour change:** the raw-read API (`read_raw_data_full*`), which has
|
||||
no fill value, now returns an error for a virtual dataset with unmapped
|
||||
elements instead of zeros.
|
||||
- Hyperslab selection versions 1 and 2 were refused ("only version-3
|
||||
hyperslab selections are supported"). Version 1 is what libhdf5 writes for
|
||||
every VDS created with the default format bounds (h5py's default), so
|
||||
|
||||
Reference in New Issue
Block a user