Merge branch 'fix/p1-vds' into fix/p1-read-gaps
# Conflicts: # CHANGELOG.md # crates/clawhdf5-format/src/data_read.rs
This commit is contained in:
@@ -331,6 +331,47 @@
|
||||
The legacy per-member dimension fields are now decoded into an array type,
|
||||
as libhdf5 does; more than four dimensions, or a zero-sized one, is an
|
||||
error.
|
||||
- `clawhdf5-format` virtual datasets (VDS), checked against HDF5 2.0 through
|
||||
h5py (`crates/clawhdf5/tests/vds_interop.rs`):
|
||||
- **Wrong data:** elements no mapping supplies — unmapped regions, and
|
||||
mappings whose source file or dataset is missing — read as 0 instead of
|
||||
the virtual dataset's fill value (e.g. h5py `fillvalue=-1`). Assembly moved
|
||||
to the new `vds` module: `vds::read_virtual_dataset` takes the fill value
|
||||
and a resolver that can refuse a name (`VdsFileResolver`), and `File`
|
||||
passes the dataset's fill value. A missing source *dataset* read as an
|
||||
error; it is fill now, as in libhdf5. Source datasets are read with their
|
||||
own fill value for unallocated chunks, and a source whose datatype differs
|
||||
from the virtual dataset's is an error (libhdf5 converts; we do not).
|
||||
`File` now refuses a source name that leaves the virtual file's directory
|
||||
(`../x.h5`, absolute paths), or any external source of a `File::from_bytes`
|
||||
file, with an error — these used to read as fill.
|
||||
**Behaviour change:** the raw-read API (`read_raw_data_full*`), which has
|
||||
no fill value, now returns an error for a virtual dataset with unmapped
|
||||
elements instead of zeros.
|
||||
- Unlimited and printf-style mappings are supported (all 7 VDS files in the
|
||||
libhdf5 test set are such mappings, e.g. Eiger/Percival detector layouts).
|
||||
`%b` in a source file or dataset name is the block number and `%%` a
|
||||
literal `%` (other `%` sequences are an error, as in libhdf5); block `j`
|
||||
is read from the source named with `j`, probing from 0 up to the first
|
||||
missing source. Unlimited source/virtual selections cover as much as the
|
||||
source's current extent fills, including a partial last block. As
|
||||
libhdf5 does on `H5Dget_space`, the extent is recomputed from the sources
|
||||
present (default "last available" view, printf gap 0) —
|
||||
`vds::virtual_dataset_extent`, used by `Dataset::shape()` — so e.g.
|
||||
`vds-eiger.h5` is `[5, 10, 10]`, not its stored `[20, 10, 10]`. A source
|
||||
stored in the other byte order is byte-swapped (libhdf5 converts);
|
||||
other type conversions remain an error.
|
||||
- Hyperslab selection versions 1 and 2 were refused ("only version-3
|
||||
hyperslab selections are supported"). Version 1 is what libhdf5 writes for
|
||||
every VDS created with the default format bounds (h5py's default), so
|
||||
those could not be read at all; version 2 is its encoding of an unlimited
|
||||
selection. Both are decoded now, as are irregular hyperslabs (a union of
|
||||
blocks, read in row-major order as libhdf5 iterates them).
|
||||
`SerializedSelection` exposes the raw form, including unlimited counts.
|
||||
- The version-1 mapping list HDF5 2.0 writes (low version bound 2.0) was
|
||||
misparsed: each entry's flags byte was read as the start of the source
|
||||
file name, and names shared with an earlier entry (stored as that entry's
|
||||
index) were not followed. Now decoded as `H5D__virtual_load_layout` does.
|
||||
- `clawhdf5-format` reader — **values returned wrong with no error:**
|
||||
- Fixed Array and Extensible Array chunk indexes were laid out by the
|
||||
dataset's current shape instead of its max shape (23 libhdf5 test files,
|
||||
|
||||
Reference in New Issue
Block a user