test: every scale-offset dataset h5py writes reads as h5py reads it
The scale-offset fix (d110b1d) was covered only by unit vectors from CVE chunks, and documented as three corner cases. The review found it is much bigger: of 1480 scale-offset datasets h5py writes (every integer type i1..u8, f4 and f8, both byte orders, with and without a fill value, scaleoffset 0..full width), v2.7.0's decoder read 332 differently from h5py: 151 returned wrong values with no error (82 integer datasets with scaleoffset=0 and a wide range, 51 full-width i4/u4/i8/u8, 18 f4 D-scale datasets with a large range) and 181 failed to read. The cause in every case is a chunk libhdf5 stores at full width, whose elements were decoded as offsets from minval. tests/scaleoffset_interop.rs generates that matrix with h5py at test time, stores h5py's decoded values uncompressed next to it, and compares every dataset's bytes. It passes on this branch; with the filters.rs befored110b1dit reports "332 of 1480 scale-offset datasets differ from h5py". CHANGELOG: a Correctness entry stating this was silent wrong data in every release that decoded scale-offset (v2.2.0 to v2.7.0), replacing the corner-case wording. docs/known-issues.md: a fixed entry with the affected cases. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+26
-6
@@ -61,12 +61,10 @@ ZFP filters; the 2 mismatches are the known h5py big-endian VL bug.
|
||||
- a simple dataspace of rank 0 holds one element (it held 0;
|
||||
`cve-2020-18494`), and contiguous storage larger than the dataset reads
|
||||
(`cve-2024-32623`, `cve-2025-2309`; libhdf5 ignores the excess);
|
||||
- scale-offset: the packed codes start at byte 21 whatever size the chunk
|
||||
records for `minval`, a chunk with `minbits` 0 and a fill value is all
|
||||
fill values, full-width `minbits` stores the elements as they are
|
||||
(these decoded differently from libhdf5); E-scale is refused, as in
|
||||
libhdf5; codes past the end of the chunk stay an error
|
||||
(`cve-2025-2308`, where HDF5 2.0 reads past its buffer);
|
||||
- scale-offset returned wrong values for ordinary h5py files — see
|
||||
*Correctness* below; E-scale is refused, as in libhdf5; codes past the
|
||||
end of the chunk stay an error (`cve-2025-2308`, where HDF5 2.0 reads
|
||||
past its buffer);
|
||||
- shuffle uses its own parameter as the element size, as libhdf5 does
|
||||
(`cve-2025-44905`);
|
||||
- an unfiltered chunk the index records at other than the chunk's size is
|
||||
@@ -878,6 +876,28 @@ ZFP filters; the 2 mismatches are the known h5py big-endian VL bug.
|
||||
- CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake.
|
||||
|
||||
### Correctness
|
||||
- **Scale-offset data read wrong values in every release that decoded it
|
||||
(v2.2.0 to v2.7.0), silently, on ordinary h5py files** (fixed
|
||||
2026-09-26). Of 1480 scale-offset datasets h5py writes across every
|
||||
integer type (`i1` .. `u8`), `f4` and `f8`, both byte orders, with and
|
||||
without a fill value, and `scaleoffset` from 0 to the full width, 332 did
|
||||
not read as h5py reads them: **151 returned wrong values with no error**
|
||||
and 181 failed to read. The common cause was a chunk libhdf5 stores at
|
||||
full width (`minbits` equal to the type's width), which it does for any
|
||||
full-width `scaleoffset` and on its own whenever a chunk's values span
|
||||
most of the type's range: `scaleoffset=0` integer data with a wide range
|
||||
(82 datasets, all wrong values), full-width `u4`/`i4`/`u8`/`i8` (51 wrong
|
||||
values; the narrower types and the rest failed with "truncated minval" or
|
||||
"implausible minbits"), and `f4` D-scale data with a large range (18,
|
||||
wrong values). Such a chunk holds the elements as they are; they were
|
||||
decoded as offsets from `minval`. Also fixed, found on crafted files: the
|
||||
packed codes start at byte 21 whatever size the chunk records for
|
||||
`minval` (`cve-2025-44905` `/Scale_offset_short_data_be`), and a chunk
|
||||
with `minbits` 0 and a fill value is all fill values (it read as
|
||||
`minval`). The whole matrix is now an interop test
|
||||
(`crates/clawhdf5/tests/scaleoffset_interop.rs`, generated by h5py at
|
||||
test time, every dataset compared); on v2.7.0's decoder it reports the
|
||||
332. See `docs/known-issues.md`.
|
||||
- **Corrupt files libhdf5 refuses are now refused instead of read.** On the
|
||||
HDF Group's CVE reproducers, 18 objects that libhdf5 (HDF5 2.0, through
|
||||
h5py) refuses to open were read by clawhdf5, some as wrong data (a chunk
|
||||
|
||||
Reference in New Issue
Block a user