A ref-bug file's refused objects were still listed as our-error root
causes. The summary line now names every non-ok class and its count, and
an empty root-cause table says None.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The last 3 our-errors and 2 mismatches were documented as not ours but
still counted against us, on a heuristic (any big-endian VL mismatch) and
a fixed list.
- ref.py checks that the installed h5py returns big-endian VL elements
with the file's bytes under a little-endian dtype (writing and reading
a vlen('>f4') in memory) and, if so, relabels them with the file's byte
order before hashing, marking the object `ref_fix`. The values are now
compared: attr_datatypes.hdf5 /@vlen_uint64 and tcomplex_be.h5
/VariableLengthDatasetFloatComplex are identical to ours (h5dump 1.14.6
prints the same (1, 2), (3, 4, 5), (42)).
- ref_bugs.py re-reads each object h5py reads only through a libhdf5 bug
in six processes with different heaps (import order, MALLOC_PERTURB_).
Values the file determines are the same every time; these three change
(6, 6 and 3 distinct results), so they are over-read memory, not data
clawhdf5 could match. compare.py classifies a file `ref-bug` only when
every difference is such an object confirmed in the same run.
- report.py: the ref-bug class, the evidence table, the corrected
objects; test_ref.py covers both (run in the nightly job).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 does not walk the chunk B-tree to read a dataset: it looks each
chunk up (H5B_find with H5D__btree_cmp3 and H5D__btree_found), asking for
the element-size coordinate as 0. collect_chunk_info_checked now parses
the tree with its keys and returns each stored chunk only when that
lookup, replayed over the scaled keys, finds it.
A key with a non-zero element-size coordinate is therefore found in a
1-D dataset (cmp3 compares only the first coordinate there, and found
compares with <=) and missed in a dataset of rank 2 or more, which reads
fill values. The previous commit refused every such key, which refused
1-D files libhdf5 reads correctly; before that, the rank-2 case read the
chunk's data where h5py reads fill values (cve-2025-44905
/Shuffle_float_data_le, now identical to h5py, so it leaves the
conformance report's list of libhdf5 bugs).
Test: chunk_keys_with_an_element_offset_read_as_libhdf5_reads_them
compares 1-D and 2-D files against h5py's values. It fails on the
previous commit (the 1-D file is refused) and with the refusal removed
(the 2-D file reads 0..23 where h5py reads fill values).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Four of the remaining our-errors are objects the reference (h5py 3.16 /
HDF5 2.0) reads only through a libhdf5 bug, and clawhdf5 refuses:
cve-2025-2308 (scale-offset codes past the end of the chunk),
cve-2025-44904 (short unfiltered chunks), bad_nbit_parms_walk.h5 (an N-Bit
parameter list one value short; libhdf5's own test_filter_bad_params now
requires the read to fail) and cve-2025-44905 /Shuffle_float_data_le (a
chunk key libhdf5's lookup misses, reading fill values). report.py lists
them under Known not-our-bug and counts them in the summary; they stay
our-errors in the class counts.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
conformance/run.sh fetches eight public HDF5 corpora pinned by commit
(conformance/corpus.txt) into a gitignored cache, reads every file with
clawhdf5 (conformance/probe, a crate outside the workspace) and with
h5py/libhdf5 (ref.py), and the CVE files with h5dump, each under a timeout
and an address-space limit; compare.py classifies the files, report.py
writes CONFORMANCE.md and check.py gates on panics/hangs/crashes/OOM and on
regressions against conformance/baseline.json. ~25 s once cached.
Changes from the ad-hoc audit harness:
- the probe compares non-IEEE-layout floats (N-Bit) and integers with a bit
offset or reduced precision as the values libhdf5 converts them to, not
raw file bytes: 8 files that showed as mismatches now read identically;
- ref.py exits without tearing down h5py objects: libhdf5 2.0 aborts while
freeing them for two files about half the time, which flipped them
between ok and h5py-cannot-read from run to run;
- the file list is defined (list_files.py): netCDF classic files are left
out, 11 HDF5 files the ad-hoc sweep missed are in.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
