fix(format): read unmapped VDS elements as the virtual dataset's fill value

Elements of a virtual dataset that no mapping supplies (unmapped regions,
a missing source file, a missing source dataset) read as 0 instead of the
fill value libhdf5 returns — silent wrong data for any VDS created with a
non-zero fillvalue (read-matrix cases 0471/0472: -1 and 7 read as 0). A
missing source dataset was an error; libhdf5 reads it as fill.

Move VDS assembly into a new vds module following H5Dvirtual.c:
vds::read_virtual_dataset takes the dataset's fill value and a
VdsFileResolver that can refuse a name, and reports how many elements were
unmapped. Sources are read with their own fill value, and a source whose
datatype differs from the virtual dataset's is an error (libhdf5 converts).
File passes the dataset's fill value, resolves source names against the
virtual file's directory, and refuses names that leave it with an error
instead of reading them as fill. read_selection on a VDS goes through the
same fill-aware path.

The raw-read API (read_raw_data_full*) has no fill value, so it now errors
for a VDS with unmapped elements instead of guessing zeros.

Tests: vds_interop::vds_unmapped_regions_read_as_fill_value (external,
same-file, missing file/dataset, sparse source with its own fill, int
fill; earliest and latest format) and
vds_source_outside_directory_is_an_error_not_fill, both against h5py;
integration_test::v4_virtual_dataset_raw_api_refuses_to_guess_the_fill_value.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-25 22:03:47 -05:00
co-authored by Claude Opus 5.5
parent 2c6c6c176e
commit e94a52a88b
8 changed files with 785 additions and 167 deletions
+15
View File
@@ -275,6 +275,21 @@
### Correctness
- `clawhdf5-format` virtual datasets (VDS), checked against HDF5 2.0 through
h5py (`crates/clawhdf5/tests/vds_interop.rs`):
- **Wrong data:** elements no mapping supplies — unmapped regions, and
mappings whose source file or dataset is missing — read as 0 instead of
the virtual dataset's fill value (e.g. h5py `fillvalue=-1`). Assembly moved
to the new `vds` module: `vds::read_virtual_dataset` takes the fill value
and a resolver that can refuse a name (`VdsFileResolver`), and `File`
passes the dataset's fill value. A missing source *dataset* read as an
error; it is fill now, as in libhdf5. Source datasets are read with their
own fill value for unallocated chunks, and a source whose datatype differs
from the virtual dataset's is an error (libhdf5 converts; we do not).
`File` now refuses a source name that leaves the virtual file's directory
(`../x.h5`, absolute paths), or any external source of a `File::from_bytes`
file, with an error — these used to read as fill.
**Behaviour change:** the raw-read API (`read_raw_data_full*`), which has
no fill value, now returns an error for a virtual dataset with unmapped
elements instead of zeros.
- Hyperslab selection versions 1 and 2 were refused ("only version-3
hyperslab selections are supported"). Version 1 is what libhdf5 writes for
every VDS created with the default format bounds (h5py's default), so