test: every scale-offset dataset h5py writes reads as h5py reads it
The scale-offset fix (d110b1d) was covered only by unit vectors from CVE chunks, and documented as three corner cases. The review found it is much bigger: of 1480 scale-offset datasets h5py writes (every integer type i1..u8, f4 and f8, both byte orders, with and without a fill value, scaleoffset 0..full width), v2.7.0's decoder read 332 differently from h5py: 151 returned wrong values with no error (82 integer datasets with scaleoffset=0 and a wide range, 51 full-width i4/u4/i8/u8, 18 f4 D-scale datasets with a large range) and 181 failed to read. The cause in every case is a chunk libhdf5 stores at full width, whose elements were decoded as offsets from minval. tests/scaleoffset_interop.rs generates that matrix with h5py at test time, stores h5py's decoded values uncompressed next to it, and compares every dataset's bytes. It passes on this branch; with the filters.rs befored110b1dit reports "332 of 1480 scale-offset datasets differ from h5py". CHANGELOG: a Correctness entry stating this was silent wrong data in every release that decoded scale-offset (v2.2.0 to v2.7.0), replacing the corner-case wording. docs/known-issues.md: a fixed entry with the affected cases. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+39
-1
@@ -63,6 +63,43 @@ tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`,
|
||||
`CHANGELOG.md`). The chunked-read scaling item above is still open.
|
||||
Values are correct; this is speed only.
|
||||
|
||||
## Scale-offset data read back wrong values
|
||||
|
||||
**Status:** fixed 2026-09-26, after v2.7.0. **Every release that decoded
|
||||
the scale-offset filter (v2.2.0 to v2.7.0) is affected**, on ordinary
|
||||
files h5py writes, with no error.
|
||||
|
||||
Found by the review of the 2026-09-26 conformance work: h5py wrote 1480
|
||||
scale-offset datasets (every integer type `i1` .. `u8`, `f4` and `f8`,
|
||||
little- and big-endian, with no fill value and with the type's minimum,
|
||||
maximum or another value as fill, random, constant, all-fill and extreme
|
||||
data, `scaleoffset` 0, 1, 3, full width less one and full width for
|
||||
integers, decimal scale factors 0, 1, 2, 4 and 7 for floats). v2.7.0's
|
||||
decoder read 332 of them differently from h5py: **151 returned wrong
|
||||
values with no error**, 181 failed to read.
|
||||
|
||||
| Case | Datasets | v2.7.0 |
|
||||
|---|---|---|
|
||||
| integer, `scaleoffset=0` (libhdf5 picks the bits), data spanning most of the type's range | 82 | wrong values |
|
||||
| integer, `scaleoffset` = full width: `i4`/`u4` (10), `i8`/`u8` (41) | 51 | wrong values |
|
||||
| integer, `scaleoffset` = full width, the other datasets (every `i1`..`u2` one, most `i4`/`u4`, some `i8`/`u8`) | 181 | "truncated minval" / "implausible minbits" |
|
||||
| `f4` D-scale, factor 4 or 7, values up to about 10^6 | 18 | wrong values |
|
||||
|
||||
In every case libhdf5 stored a chunk at full width (`minbits` equal to the
|
||||
type's width): the chunk then holds the elements as they are, and they
|
||||
were decoded as offsets from `minval`. Two more differences were found on
|
||||
crafted files and fixed with them: the packed codes start at byte 21
|
||||
whatever size the chunk records for `minval` (`cve-2025-44905`
|
||||
`/Scale_offset_short_data_be`), and a chunk with `minbits` 0 and a fill
|
||||
value is all fill values (it read as `minval`).
|
||||
|
||||
**Fix:** `clawhdf5_format::filters` decodes a scale-offset chunk as
|
||||
`H5Z__filter_scaleoffset` does. **Test:** the whole matrix is
|
||||
`crates/clawhdf5/tests/scaleoffset_interop.rs`, generated by h5py at test
|
||||
time and compared dataset by dataset; on v2.7.0's decoder it reports the
|
||||
332. **Existing data:** the files were always right; only reads were
|
||||
wrong, so re-reading with a fixed build gives the correct values.
|
||||
|
||||
## Silent wrong data found by the 2026-09-25 HDF5 audit
|
||||
|
||||
**Status:** fixed after v2.7.0 (2026-09-25). **Every release up
|
||||
@@ -230,7 +267,8 @@ fill-value item that did is fixed).
|
||||
under *Known not-our-bug*. Scale-offset did decode three cases
|
||||
differently from libhdf5 (codes after a `minval` of recorded size other
|
||||
than 8, `minbits` 0 with a fill value, full-width `minbits`): fixed
|
||||
2026-09-26.
|
||||
2026-09-26, and the full-width case was silent wrong data on ordinary
|
||||
h5py files (see *Scale-offset data read back wrong values* above).
|
||||
- **Filters:** blosc, blosc2, bitshuffle, bzip2, LZF and zfp are not
|
||||
implemented. **Fixed 2026-09-26** for LZF (default-on `lzf` feature),
|
||||
bitshuffle, bzip2 and Blosc 1 (`bitshuffle`, `bzip2`, `blosc`, or
|
||||
|
||||
Reference in New Issue
Block a user