test: every scale-offset dataset h5py writes reads as h5py reads it

The scale-offset fix (d110b1d) was covered only by unit vectors from
CVE chunks, and documented as three corner cases. The review found it
is much bigger: of 1480 scale-offset datasets h5py writes (every
integer type i1..u8, f4 and f8, both byte orders, with and without a
fill value, scaleoffset 0..full width), v2.7.0's decoder read 332
differently from h5py: 151 returned wrong values with no error (82
integer datasets with scaleoffset=0 and a wide range, 51 full-width
i4/u4/i8/u8, 18 f4 D-scale datasets with a large range) and 181 failed
to read. The cause in every case is a chunk libhdf5 stores at full
width, whose elements were decoded as offsets from minval.

tests/scaleoffset_interop.rs generates that matrix with h5py at test
time, stores h5py's decoded values uncompressed next to it, and
compares every dataset's bytes. It passes on this branch; with the
filters.rs before d110b1d it reports "332 of 1480 scale-offset datasets
differ from h5py".

CHANGELOG: a Correctness entry stating this was silent wrong data in
every release that decoded scale-offset (v2.2.0 to v2.7.0), replacing
the corner-case wording. docs/known-issues.md: a fixed entry with the
affected cases.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 11:51:30 -05:00
co-authored by Claude Opus 5.5
parent d493d4792e
commit 4c01267b76
3 changed files with 216 additions and 7 deletions
+26 -6
View File
@@ -61,12 +61,10 @@ ZFP filters; the 2 mismatches are the known h5py big-endian VL bug.
- a simple dataspace of rank 0 holds one element (it held 0;
`cve-2020-18494`), and contiguous storage larger than the dataset reads
(`cve-2024-32623`, `cve-2025-2309`; libhdf5 ignores the excess);
- scale-offset: the packed codes start at byte 21 whatever size the chunk
records for `minval`, a chunk with `minbits` 0 and a fill value is all
fill values, full-width `minbits` stores the elements as they are
(these decoded differently from libhdf5); E-scale is refused, as in
libhdf5; codes past the end of the chunk stay an error
(`cve-2025-2308`, where HDF5 2.0 reads past its buffer);
- scale-offset returned wrong values for ordinary h5py files — see
*Correctness* below; E-scale is refused, as in libhdf5; codes past the
end of the chunk stay an error (`cve-2025-2308`, where HDF5 2.0 reads
past its buffer);
- shuffle uses its own parameter as the element size, as libhdf5 does
(`cve-2025-44905`);
- an unfiltered chunk the index records at other than the chunk's size is
@@ -878,6 +876,28 @@ ZFP filters; the 2 mismatches are the known h5py big-endian VL bug.
- CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake.
### Correctness
- **Scale-offset data read wrong values in every release that decoded it
(v2.2.0 to v2.7.0), silently, on ordinary h5py files** (fixed
2026-09-26). Of 1480 scale-offset datasets h5py writes across every
integer type (`i1` .. `u8`), `f4` and `f8`, both byte orders, with and
without a fill value, and `scaleoffset` from 0 to the full width, 332 did
not read as h5py reads them: **151 returned wrong values with no error**
and 181 failed to read. The common cause was a chunk libhdf5 stores at
full width (`minbits` equal to the type's width), which it does for any
full-width `scaleoffset` and on its own whenever a chunk's values span
most of the type's range: `scaleoffset=0` integer data with a wide range
(82 datasets, all wrong values), full-width `u4`/`i4`/`u8`/`i8` (51 wrong
values; the narrower types and the rest failed with "truncated minval" or
"implausible minbits"), and `f4` D-scale data with a large range (18,
wrong values). Such a chunk holds the elements as they are; they were
decoded as offsets from `minval`. Also fixed, found on crafted files: the
packed codes start at byte 21 whatever size the chunk records for
`minval` (`cve-2025-44905` `/Scale_offset_short_data_be`), and a chunk
with `minbits` 0 and a fill value is all fill values (it read as
`minval`). The whole matrix is now an interop test
(`crates/clawhdf5/tests/scaleoffset_interop.rs`, generated by h5py at
test time, every dataset compared); on v2.7.0's decoder it reports the
332. See `docs/known-issues.md`.
- **Corrupt files libhdf5 refuses are now refused instead of read.** On the
HDF Group's CVE reproducers, 18 objects that libhdf5 (HDF5 2.0, through
h5py) refuses to open were read by clawhdf5, some as wrong data (a chunk