diff --git a/CHANGELOG.md b/CHANGELOG.md index 8912fed..253e9a0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -61,12 +61,10 @@ ZFP filters; the 2 mismatches are the known h5py big-endian VL bug. - a simple dataspace of rank 0 holds one element (it held 0; `cve-2020-18494`), and contiguous storage larger than the dataset reads (`cve-2024-32623`, `cve-2025-2309`; libhdf5 ignores the excess); - - scale-offset: the packed codes start at byte 21 whatever size the chunk - records for `minval`, a chunk with `minbits` 0 and a fill value is all - fill values, full-width `minbits` stores the elements as they are - (these decoded differently from libhdf5); E-scale is refused, as in - libhdf5; codes past the end of the chunk stay an error - (`cve-2025-2308`, where HDF5 2.0 reads past its buffer); + - scale-offset returned wrong values for ordinary h5py files — see + *Correctness* below; E-scale is refused, as in libhdf5; codes past the + end of the chunk stay an error (`cve-2025-2308`, where HDF5 2.0 reads + past its buffer); - shuffle uses its own parameter as the element size, as libhdf5 does (`cve-2025-44905`); - an unfiltered chunk the index records at other than the chunk's size is @@ -878,6 +876,28 @@ ZFP filters; the 2 mismatches are the known h5py big-endian VL bug. - CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake. ### Correctness +- **Scale-offset data read wrong values in every release that decoded it + (v2.2.0 to v2.7.0), silently, on ordinary h5py files** (fixed + 2026-09-26). Of 1480 scale-offset datasets h5py writes across every + integer type (`i1` .. `u8`), `f4` and `f8`, both byte orders, with and + without a fill value, and `scaleoffset` from 0 to the full width, 332 did + not read as h5py reads them: **151 returned wrong values with no error** + and 181 failed to read. The common cause was a chunk libhdf5 stores at + full width (`minbits` equal to the type's width), which it does for any + full-width `scaleoffset` and on its own whenever a chunk's values span + most of the type's range: `scaleoffset=0` integer data with a wide range + (82 datasets, all wrong values), full-width `u4`/`i4`/`u8`/`i8` (51 wrong + values; the narrower types and the rest failed with "truncated minval" or + "implausible minbits"), and `f4` D-scale data with a large range (18, + wrong values). Such a chunk holds the elements as they are; they were + decoded as offsets from `minval`. Also fixed, found on crafted files: the + packed codes start at byte 21 whatever size the chunk records for + `minval` (`cve-2025-44905` `/Scale_offset_short_data_be`), and a chunk + with `minbits` 0 and a fill value is all fill values (it read as + `minval`). The whole matrix is now an interop test + (`crates/clawhdf5/tests/scaleoffset_interop.rs`, generated by h5py at + test time, every dataset compared); on v2.7.0's decoder it reports the + 332. See `docs/known-issues.md`. - **Corrupt files libhdf5 refuses are now refused instead of read.** On the HDF Group's CVE reproducers, 18 objects that libhdf5 (HDF5 2.0, through h5py) refuses to open were read by clawhdf5, some as wrong data (a chunk diff --git a/crates/clawhdf5/tests/scaleoffset_interop.rs b/crates/clawhdf5/tests/scaleoffset_interop.rs new file mode 100644 index 0000000..d31c0db --- /dev/null +++ b/crates/clawhdf5/tests/scaleoffset_interop.rs @@ -0,0 +1,151 @@ +//! Scale-offset datasets written by h5py (libhdf5) read exactly as h5py +//! reads them. +//! +//! h5py writes the whole matrix the filter has: every integer type (`i1` .. +//! `u8`) and `f4`/`f8`, both byte orders, with and without a fill value +//! (the type's minimum, maximum, or another value), with random, constant, +//! all-fill and extreme data, and every interesting `scaleoffset` setting +//! (0 = let libhdf5 choose the bits, a few bits, full width less one, full +//! width; decimal scale factors 0..7 for floats) — about 1480 datasets. For +//! each one h5py's decoded values are stored uncompressed next to it, and +//! the raw bytes clawhdf5 decodes must equal them. +//! +//! Until 2026-09-26 clawhdf5 silently returned wrong values for 332 of +//! these, in every release that decoded scale-offset: ordinary `u8`, `u4`, +//! `i8` and `f4` data with a wide range (where libhdf5 stores the elements +//! as they are, at full width), chunks whose `minval` field was recorded at +//! another size than 8 bytes, and chunks with `minbits` 0 and a fill value. +//! +//! Skipped when python3 with h5py is unavailable, unless +//! `CLAWHDF5_REQUIRE_INTEROP=1`. + +use std::process::Command; + +use clawhdf5::File; + +fn python() -> String { + std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string()) +} + +fn python_available() -> bool { + Command::new(python()) + .args(["-c", "import h5py, numpy"]) + .output() + .map(|o| o.status.success()) + .unwrap_or(false) +} + +const GENERATE: &str = r#" +import sys, h5py, numpy as np +rng = np.random.default_rng(7) +out, expect = sys.argv[1], sys.argv[2] +made = [] +with h5py.File(out, "w") as f: + def make(name, a, so, fv, desc): + try: + d = f.create_dataset(name, data=a, chunks=(16,), scaleoffset=so, fillvalue=fv) + except Exception: + return # a combination libhdf5 refuses to write + d.attrs["desc"] = desc + made.append(name) + i = 0 + for dt in ["i1", "u1", "i2", "u2", "i4", "u4", "i8", "u8"]: + for bo in "<>": + t = np.dtype(bo + dt) + info = np.iinfo(t) + for so in [0, 1, 3, 8 * t.itemsize - 1, 8 * t.itemsize]: + for fill in [None, "min", "max", "mid"]: + for pat in ["rand", "const", "allfill", "extreme"]: + n = 64 + fv = {None: None, "min": info.min, "max": info.max, "mid": t.type(7)}[fill] + if pat == "rand": + lo = max(info.min, -50) if so else info.min + hi = min(info.max, 50) if so else info.max + a = rng.integers(lo, hi, size=n, endpoint=True, + dtype=np.int64 if t.kind == "i" else np.uint64).astype(t) + elif pat == "const": + a = np.full(n, 3, t) + elif pat == "extreme": + a = np.array([info.min, info.max] * (n // 2), t) + else: + if fv is None: + continue + a = np.full(n, fv, t) + make(f"d{i}", a, so, fv, f"{bo}{dt} so={so} fill={fill} pat={pat}") + i += 1 + for dt in ["f4", "f8"]: + for bo in "<>": + t = np.dtype(bo + dt) + for so in [0, 1, 2, 4, 7]: + for fill in [None, -1.5, 0.0]: + for pat in ["rand", "const", "allfill", "neg", "big"]: + n = 50 + if pat == "rand": + a = rng.normal(size=n).astype(t) * 10 + elif pat == "const": + a = np.full(n, 2.25, t) + elif pat == "neg": + a = -np.abs(rng.normal(size=n)).astype(t) * 1000 + elif pat == "big": + a = rng.normal(size=n).astype(t) * 1e6 + else: + if fill is None: + continue + a = np.full(n, fill, t) + make(f"d{i}", a, so, fill, f"{bo}{dt} so={so} fill={fill} pat={pat}") + i += 1 +# What libhdf5 decodes, stored uncompressed in the same datatype. +with h5py.File(out, "r") as f, h5py.File(expect, "w") as e: + for name in made: + e.create_dataset(name, data=f[name][()]) +print(len(made)) +"#; + +#[test] +fn every_scale_offset_dataset_reads_as_h5py_reads_it() { + if !python_available() { + assert!( + !std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1"), + "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available" + ); + eprintln!("SKIP: python3 with h5py not available"); + return; + } + let dir = tempfile::tempdir().unwrap(); + let written = dir.path().join("scaleoffset.h5"); + let expected = dir.path().join("expected.h5"); + let out = Command::new(python()) + .args(["-c", GENERATE]) + .arg(&written) + .arg(&expected) + .output() + .expect("failed to run python"); + assert!( + out.status.success(), + "python failed:\n{}", + String::from_utf8_lossy(&out.stderr) + ); + let count: usize = String::from_utf8_lossy(&out.stdout).trim().parse().unwrap(); + assert!(count > 1400, "only {count} datasets written"); + + let file = File::open(&written).unwrap(); + let reference = File::open(&expected).unwrap(); + let mut names = reference.root().datasets().unwrap(); + names.sort(); + assert_eq!(names.len(), count); + let mut wrong = Vec::new(); + for name in &names { + let want = reference.read_multi(&[name]).unwrap().remove(0); + let got = file.read_multi(&[name]).map(|mut v| v.remove(0)); + if got.as_ref().ok() != Some(&want) { + let desc = file.dataset(name).unwrap().attrs().unwrap().remove("desc"); + wrong.push(format!("{name} {desc:?}: {:?}", got.map(|g| g.len()))); + } + } + assert!( + wrong.is_empty(), + "{} of {count} scale-offset datasets differ from h5py:\n{}", + wrong.len(), + wrong.join("\n") + ); +} diff --git a/docs/known-issues.md b/docs/known-issues.md index 8328f14..5d1aac7 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -63,6 +63,43 @@ tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`, `CHANGELOG.md`). The chunked-read scaling item above is still open. Values are correct; this is speed only. +## Scale-offset data read back wrong values + +**Status:** fixed 2026-09-26, after v2.7.0. **Every release that decoded +the scale-offset filter (v2.2.0 to v2.7.0) is affected**, on ordinary +files h5py writes, with no error. + +Found by the review of the 2026-09-26 conformance work: h5py wrote 1480 +scale-offset datasets (every integer type `i1` .. `u8`, `f4` and `f8`, +little- and big-endian, with no fill value and with the type's minimum, +maximum or another value as fill, random, constant, all-fill and extreme +data, `scaleoffset` 0, 1, 3, full width less one and full width for +integers, decimal scale factors 0, 1, 2, 4 and 7 for floats). v2.7.0's +decoder read 332 of them differently from h5py: **151 returned wrong +values with no error**, 181 failed to read. + +| Case | Datasets | v2.7.0 | +|---|---|---| +| integer, `scaleoffset=0` (libhdf5 picks the bits), data spanning most of the type's range | 82 | wrong values | +| integer, `scaleoffset` = full width: `i4`/`u4` (10), `i8`/`u8` (41) | 51 | wrong values | +| integer, `scaleoffset` = full width, the other datasets (every `i1`..`u2` one, most `i4`/`u4`, some `i8`/`u8`) | 181 | "truncated minval" / "implausible minbits" | +| `f4` D-scale, factor 4 or 7, values up to about 10^6 | 18 | wrong values | + +In every case libhdf5 stored a chunk at full width (`minbits` equal to the +type's width): the chunk then holds the elements as they are, and they +were decoded as offsets from `minval`. Two more differences were found on +crafted files and fixed with them: the packed codes start at byte 21 +whatever size the chunk records for `minval` (`cve-2025-44905` +`/Scale_offset_short_data_be`), and a chunk with `minbits` 0 and a fill +value is all fill values (it read as `minval`). + +**Fix:** `clawhdf5_format::filters` decodes a scale-offset chunk as +`H5Z__filter_scaleoffset` does. **Test:** the whole matrix is +`crates/clawhdf5/tests/scaleoffset_interop.rs`, generated by h5py at test +time and compared dataset by dataset; on v2.7.0's decoder it reports the +332. **Existing data:** the files were always right; only reads were +wrong, so re-reading with a fixed build gives the correct values. + ## Silent wrong data found by the 2026-09-25 HDF5 audit **Status:** fixed after v2.7.0 (2026-09-25). **Every release up @@ -230,7 +267,8 @@ fill-value item that did is fixed). under *Known not-our-bug*. Scale-offset did decode three cases differently from libhdf5 (codes after a `minval` of recorded size other than 8, `minbits` 0 with a fill value, full-width `minbits`): fixed - 2026-09-26. + 2026-09-26, and the full-width case was silent wrong data on ordinary + h5py files (see *Scale-offset data read back wrong values* above). - **Filters:** blosc, blosc2, bitshuffle, bzip2, LZF and zfp are not implemented. **Fixed 2026-09-26** for LZF (default-on `lzf` feature), bitshuffle, bzip2 and Blosc 1 (`bitshuffle`, `bzip2`, `blosc`, or