Merge branch 'fix/p1-vds' into fix/p1-read-gaps

# Conflicts:
#	CHANGELOG.md
#	crates/clawhdf5-format/src/data_read.rs
This commit is contained in:
osobh
2026-09-25 22:42:04 -05:00
23 changed files with 2321 additions and 287 deletions
+41
View File
@@ -331,6 +331,47 @@
The legacy per-member dimension fields are now decoded into an array type,
as libhdf5 does; more than four dimensions, or a zero-sized one, is an
error.
- `clawhdf5-format` virtual datasets (VDS), checked against HDF5 2.0 through
h5py (`crates/clawhdf5/tests/vds_interop.rs`):
- **Wrong data:** elements no mapping supplies — unmapped regions, and
mappings whose source file or dataset is missing — read as 0 instead of
the virtual dataset's fill value (e.g. h5py `fillvalue=-1`). Assembly moved
to the new `vds` module: `vds::read_virtual_dataset` takes the fill value
and a resolver that can refuse a name (`VdsFileResolver`), and `File`
passes the dataset's fill value. A missing source *dataset* read as an
error; it is fill now, as in libhdf5. Source datasets are read with their
own fill value for unallocated chunks, and a source whose datatype differs
from the virtual dataset's is an error (libhdf5 converts; we do not).
`File` now refuses a source name that leaves the virtual file's directory
(`../x.h5`, absolute paths), or any external source of a `File::from_bytes`
file, with an error — these used to read as fill.
**Behaviour change:** the raw-read API (`read_raw_data_full*`), which has
no fill value, now returns an error for a virtual dataset with unmapped
elements instead of zeros.
- Unlimited and printf-style mappings are supported (all 7 VDS files in the
libhdf5 test set are such mappings, e.g. Eiger/Percival detector layouts).
`%b` in a source file or dataset name is the block number and `%%` a
literal `%` (other `%` sequences are an error, as in libhdf5); block `j`
is read from the source named with `j`, probing from 0 up to the first
missing source. Unlimited source/virtual selections cover as much as the
source's current extent fills, including a partial last block. As
libhdf5 does on `H5Dget_space`, the extent is recomputed from the sources
present (default "last available" view, printf gap 0) —
`vds::virtual_dataset_extent`, used by `Dataset::shape()` — so e.g.
`vds-eiger.h5` is `[5, 10, 10]`, not its stored `[20, 10, 10]`. A source
stored in the other byte order is byte-swapped (libhdf5 converts);
other type conversions remain an error.
- Hyperslab selection versions 1 and 2 were refused ("only version-3
hyperslab selections are supported"). Version 1 is what libhdf5 writes for
every VDS created with the default format bounds (h5py's default), so
those could not be read at all; version 2 is its encoding of an unlimited
selection. Both are decoded now, as are irregular hyperslabs (a union of
blocks, read in row-major order as libhdf5 iterates them).
`SerializedSelection` exposes the raw form, including unlimited counts.
- The version-1 mapping list HDF5 2.0 writes (low version bound 2.0) was
misparsed: each entry's flags byte was read as the start of the source
file name, and names shared with an earlier entry (stored as that entry's
index) were not followed. Now decoded as `H5D__virtual_load_layout` does.
- `clawhdf5-format` reader — **values returned wrong with no error:**
- Fixed Array and Extensible Array chunk indexes were laid out by the
dataset's current shape instead of its max shape (23 libhdf5 test files,