Merge branch 'fix/p2b-remaining-conformance' into feat/p2b-scale
# Conflicts: # CHANGELOG.md # crates/clawhdf5-format/src/chunked_read.rs
This commit is contained in:
osobh
+79
-6
@@ -74,6 +74,43 @@ tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`,
|
||||
`CHANGELOG.md`). The chunked-read scaling item above is still open.
|
||||
Values are correct; this is speed only.
|
||||
|
||||
## Scale-offset data read back wrong values
|
||||
|
||||
**Status:** fixed 2026-09-26, after v2.7.0. **Every release that decoded
|
||||
the scale-offset filter (v2.2.0 to v2.7.0) is affected**, on ordinary
|
||||
files h5py writes, with no error.
|
||||
|
||||
Found by the review of the 2026-09-26 conformance work: h5py wrote 1480
|
||||
scale-offset datasets (every integer type `i1` .. `u8`, `f4` and `f8`,
|
||||
little- and big-endian, with no fill value and with the type's minimum,
|
||||
maximum or another value as fill, random, constant, all-fill and extreme
|
||||
data, `scaleoffset` 0, 1, 3, full width less one and full width for
|
||||
integers, decimal scale factors 0, 1, 2, 4 and 7 for floats). v2.7.0's
|
||||
decoder read 332 of them differently from h5py: **151 returned wrong
|
||||
values with no error**, 181 failed to read.
|
||||
|
||||
| Case | Datasets | v2.7.0 |
|
||||
|---|---|---|
|
||||
| integer, `scaleoffset=0` (libhdf5 picks the bits), data spanning most of the type's range | 82 | wrong values |
|
||||
| integer, `scaleoffset` = full width: `i4`/`u4` (10), `i8`/`u8` (41) | 51 | wrong values |
|
||||
| integer, `scaleoffset` = full width, the other datasets (every `i1`..`u2` one, most `i4`/`u4`, some `i8`/`u8`) | 181 | "truncated minval" / "implausible minbits" |
|
||||
| `f4` D-scale, factor 4 or 7, values up to about 10^6 | 18 | wrong values |
|
||||
|
||||
In every case libhdf5 stored a chunk at full width (`minbits` equal to the
|
||||
type's width): the chunk then holds the elements as they are, and they
|
||||
were decoded as offsets from `minval`. Two more differences were found on
|
||||
crafted files and fixed with them: the packed codes start at byte 21
|
||||
whatever size the chunk records for `minval` (`cve-2025-44905`
|
||||
`/Scale_offset_short_data_be`), and a chunk with `minbits` 0 and a fill
|
||||
value is all fill values (it read as `minval`).
|
||||
|
||||
**Fix:** `clawhdf5_format::filters` decodes a scale-offset chunk as
|
||||
`H5Z__filter_scaleoffset` does. **Test:** the whole matrix is
|
||||
`crates/clawhdf5/tests/scaleoffset_interop.rs`, generated by h5py at test
|
||||
time and compared dataset by dataset; on v2.7.0's decoder it reports the
|
||||
332. **Existing data:** the files were always right; only reads were
|
||||
wrong, so re-reading with a fixed build gives the correct values.
|
||||
|
||||
## Silent wrong data found by the 2026-09-25 HDF5 audit
|
||||
|
||||
**Status:** fixed after v2.7.0 (2026-09-25). **Every release up
|
||||
@@ -213,9 +250,36 @@ fill-value item that did is fixed).
|
||||
its datatype message stores (12 with 4-byte offsets), and the global
|
||||
heap is read with libhdf5's header padding
|
||||
(`crates/clawhdf5/tests/vl_offset4_interop.rs`).
|
||||
- Metadata cache images are not supported.
|
||||
- Metadata cache images are not supported. **Fixed 2026-09-26:** the
|
||||
image is applied at open, as libhdf5 loads it over the file's metadata
|
||||
(`clawhdf5_format::superblock_ext`); `h5clear_mdc_image.h5` reads
|
||||
(`crates/clawhdf5/tests/metadata_cache_image.rs`), without copying the
|
||||
file (a private copy-on-write mapping takes the image's entries;
|
||||
`tests/cache_image_memory.rs`). A file whose image libhdf5 cannot load
|
||||
(`cve-2025-6269-*`, `cve-2025-6516`) opens, as in libhdf5, and every
|
||||
object lookup fails with the image's error. Differences from libhdf5
|
||||
that remain: libhdf5 fails only the first metadata read and then reads
|
||||
the file's own (possibly stale) metadata, where we keep failing; an
|
||||
image entry that runs past the end of file is refused (libhdf5 checks
|
||||
only its start); a flush-dependency parent flag is checked against the
|
||||
child count as libhdf5's debug build checks it (HDF5 2.0 release
|
||||
builds refuse every entry that has children, even in images they
|
||||
wrote); the superblock extension's driver-info and shared-message table
|
||||
messages are not decoded at open.
|
||||
- x87 long double and binary128 are refused.
|
||||
- N-Bit on 64-bit scale-offset data and some N-Bit parameter layouts fail.
|
||||
**Not our bug (checked 2026-09-26):** both are corrupt files HDF5 2.0
|
||||
reads only by reading past a buffer. `cve-2025-2308`'s
|
||||
`/Scale_offset_long_long_data_le` has scale-offset codes that run past
|
||||
the end of the chunk (libhdf5's develop branch refuses it, "Buffer too
|
||||
short"); `bad_nbit_parms_walk.h5` has an N-Bit parameter list one value
|
||||
short (libhdf5's own `test_filter_bad_params` in `test/dsets.c` now
|
||||
requires that read to fail). We refuse both; `CONFORMANCE.md` lists them
|
||||
under *Known not-our-bug*. Scale-offset did decode three cases
|
||||
differently from libhdf5 (codes after a `minval` of recorded size other
|
||||
than 8, `minbits` 0 with a fill value, full-width `minbits`): fixed
|
||||
2026-09-26, and the full-width case was silent wrong data on ordinary
|
||||
h5py files (see *Scale-offset data read back wrong values* above).
|
||||
- **Filters:** blosc, blosc2, bitshuffle, bzip2, LZF and zfp are not
|
||||
implemented. **Fixed 2026-09-26** for LZF (default-on `lzf` feature),
|
||||
bitshuffle, bzip2 and Blosc 1 (`bitshuffle`, `bzip2`, `blosc`, or
|
||||
@@ -235,7 +299,11 @@ fill-value item that did is fixed).
|
||||
read with zeros for the missing bytes** (any filter; found reviewing the plugin
|
||||
filters). **Fixed 2026-09-26:** it is an error naming the chunk. A corrupt
|
||||
chunk must never read as zeros. Unfiltered chunks are read at their stored
|
||||
size and are not checked this way.
|
||||
size and are not checked this way. **Fixed 2026-09-26** for unfiltered
|
||||
chunks too: in a dataset without filters a chunk the index records at
|
||||
other than the chunk's size is refused, as libhdf5's develop branch
|
||||
refuses it (`cve-2025-44904`, where HDF5 2.0 fills the rest from its
|
||||
buffer).
|
||||
- **Crash:** a hostile Blosc chunk (frame size below its header) panicked in
|
||||
builds with overflow checks. **Fixed 2026-09-26**; the new decoders are
|
||||
fuzzed in the unit tests.
|
||||
@@ -248,13 +316,18 @@ fill-value item that did is fixed).
|
||||
refused. 17 of the 18 now
|
||||
fail as in libhdf5 (conformance on tank, `conformance/run.sh --no-fetch`,
|
||||
2026-09-26: 571 of 697 ok). Still read where libhdf5 refuses:
|
||||
- `cve-2024-32624.h5` `/Dset_OBJREF`: a dataspace whose storage size
|
||||
- ~~`cve-2024-32624.h5` `/Dset_OBJREF`: a dataspace whose storage size
|
||||
overflows 64 bits. `File::dataset` and `shape()` succeed (libhdf5
|
||||
refuses at open); reading the values fails.
|
||||
- `cve-2020-10810.h5`, `cve-2020-10812.h5` (whole files libhdf5 cannot
|
||||
refuses at open); reading the values fails.~~ **Fixed 2026-09-26:**
|
||||
`File::dataset` (and `MmapFile`, `LazyFile`) refuse it at open
|
||||
(`FormatError::InvalidDatasetStorage`), as they do contiguous storage
|
||||
past the end of the file.
|
||||
- ~~`cve-2020-10810.h5`, `cve-2020-10812.h5` (whole files libhdf5 cannot
|
||||
open, not among the 18): libhdf5 decodes the superblock extension's File
|
||||
Space Info and metadata-cache-image messages at open and refuses these
|
||||
files; we do not decode those messages at open.
|
||||
files; we do not decode those messages at open.~~ **Fixed 2026-09-26:**
|
||||
the superblock extension is decoded at open with libhdf5's checks, and
|
||||
both files are refused.
|
||||
- Deliberately not refused, because clawhdf5 up to v2.7.0 wrote them: a
|
||||
float sign bit position outside the type, and a size-0 string type.
|
||||
- Not refused because current libhdf5 reads it though HDF5 2.0.0
|
||||
|
||||
Reference in New Issue
Block a user