test: every scale-offset dataset h5py writes reads as h5py reads it

The scale-offset fix (d110b1d) was covered only by unit vectors from
CVE chunks, and documented as three corner cases. The review found it
is much bigger: of 1480 scale-offset datasets h5py writes (every
integer type i1..u8, f4 and f8, both byte orders, with and without a
fill value, scaleoffset 0..full width), v2.7.0's decoder read 332
differently from h5py: 151 returned wrong values with no error (82
integer datasets with scaleoffset=0 and a wide range, 51 full-width
i4/u4/i8/u8, 18 f4 D-scale datasets with a large range) and 181 failed
to read. The cause in every case is a chunk libhdf5 stores at full
width, whose elements were decoded as offsets from minval.

tests/scaleoffset_interop.rs generates that matrix with h5py at test
time, stores h5py's decoded values uncompressed next to it, and
compares every dataset's bytes. It passes on this branch; with the
filters.rs before d110b1d it reports "332 of 1480 scale-offset datasets
differ from h5py".

CHANGELOG: a Correctness entry stating this was silent wrong data in
every release that decoded scale-offset (v2.2.0 to v2.7.0), replacing
the corner-case wording. docs/known-issues.md: a fixed entry with the
affected cases.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 11:51:30 -05:00
co-authored by Claude Opus 5.5
parent d493d4792e
commit 4c01267b76
3 changed files with 216 additions and 7 deletions
+39 -1
View File
@@ -63,6 +63,43 @@ tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`,
`CHANGELOG.md`). The chunked-read scaling item above is still open.
Values are correct; this is speed only.
## Scale-offset data read back wrong values
**Status:** fixed 2026-09-26, after v2.7.0. **Every release that decoded
the scale-offset filter (v2.2.0 to v2.7.0) is affected**, on ordinary
files h5py writes, with no error.
Found by the review of the 2026-09-26 conformance work: h5py wrote 1480
scale-offset datasets (every integer type `i1` .. `u8`, `f4` and `f8`,
little- and big-endian, with no fill value and with the type's minimum,
maximum or another value as fill, random, constant, all-fill and extreme
data, `scaleoffset` 0, 1, 3, full width less one and full width for
integers, decimal scale factors 0, 1, 2, 4 and 7 for floats). v2.7.0's
decoder read 332 of them differently from h5py: **151 returned wrong
values with no error**, 181 failed to read.
| Case | Datasets | v2.7.0 |
|---|---|---|
| integer, `scaleoffset=0` (libhdf5 picks the bits), data spanning most of the type's range | 82 | wrong values |
| integer, `scaleoffset` = full width: `i4`/`u4` (10), `i8`/`u8` (41) | 51 | wrong values |
| integer, `scaleoffset` = full width, the other datasets (every `i1`..`u2` one, most `i4`/`u4`, some `i8`/`u8`) | 181 | "truncated minval" / "implausible minbits" |
| `f4` D-scale, factor 4 or 7, values up to about 10^6 | 18 | wrong values |
In every case libhdf5 stored a chunk at full width (`minbits` equal to the
type's width): the chunk then holds the elements as they are, and they
were decoded as offsets from `minval`. Two more differences were found on
crafted files and fixed with them: the packed codes start at byte 21
whatever size the chunk records for `minval` (`cve-2025-44905`
`/Scale_offset_short_data_be`), and a chunk with `minbits` 0 and a fill
value is all fill values (it read as `minval`).
**Fix:** `clawhdf5_format::filters` decodes a scale-offset chunk as
`H5Z__filter_scaleoffset` does. **Test:** the whole matrix is
`crates/clawhdf5/tests/scaleoffset_interop.rs`, generated by h5py at test
time and compared dataset by dataset; on v2.7.0's decoder it reports the
332. **Existing data:** the files were always right; only reads were
wrong, so re-reading with a fixed build gives the correct values.
## Silent wrong data found by the 2026-09-25 HDF5 audit
**Status:** fixed after v2.7.0 (2026-09-25). **Every release up
@@ -230,7 +267,8 @@ fill-value item that did is fixed).
under *Known not-our-bug*. Scale-offset did decode three cases
differently from libhdf5 (codes after a `minval` of recorded size other
than 8, `minbits` 0 with a fill value, full-width `minbits`): fixed
2026-09-26.
2026-09-26, and the full-width case was silent wrong data on ordinary
h5py files (see *Scale-offset data read back wrong values* above).
- **Filters:** blosc, blosc2, bitshuffle, bzip2, LZF and zfp are not
implemented. **Fixed 2026-09-26** for LZF (default-on `lzf` feature),
bitshuffle, bzip2 and Blosc 1 (`bitshuffle`, `bzip2`, `blosc`, or