clawhdf5/conformance/README.md

Conformance sweep

Reads every HDF5 file of eight public corpora with clawhdf5 and with h5py/libhdf5, compares the two readings object by object, and writes CONFORMANCE.md.

CLAWHDF5_PYTHON=/path/to/venv/bin/python conformance/run.sh      # ~30 s once the corpus is cached
conformance/run.sh --no-fetch                                    # use the cached corpus as is
conformance/run.sh --update-baseline                             # after an intended change in results

Latest result (tank, 2026-09-27, conformance/run.sh --no-fetch): 602 of 697 files ok, 0 our-error, 0 mismatch, 3 ref-bug, 92 h5py-cannot-read, and no panic, hang, crash or out-of-memory. The report with every file is CONFORMANCE.md.

Classes

compare.py puts each file in one class:

class	meaning
ok	clawhdf5 and h5py read the same objects with the same values
our-error	h5py reads something clawhdf5 refuses
mismatch	both read it, with different values or structure
h5py-cannot-read	h5py (libhdf5) cannot read the file; not compared
ref-bug	h5py reads an object clawhdf5 refuses, but only through a libhdf5 over-read: ref_bugs.py re-reads it in six processes with different heaps (import order, MALLOC_PERTURB_) and its values change. The file is ref-bug only while that is confirmed in the same run; if the values become stable it counts as our-error again
panic / hang / crash / oom	a clawhdf5 failure under the timeout and address-space limit; the gate fails on any

Where h5py itself returns wrong values through a known h5py bug (the big-endian variable-length bug: elements returned with the file's bytes under a little-endian dtype), ref.py checks that the installed h5py has the bug, corrects the values before hashing and marks them ref_fix, so those objects are still compared. The evidence for the three current ref-bug files is under "Conformance: the last non-ok files" in docs/known-issues.md.

Needs Rust, git, h5dump (Debian/Ubuntu hdf5-tools), libaec (for the probe's szip feature; libaec-dev), and a Python with the packages in requirements.txt. The first run downloads about 450 MB of sparse checkouts.

file	role
corpus.txt	the corpora: git URL, pinned commit, swept root, sparse-checkout patterns
fetch-corpus.sh	shallow, sparse, blob-filtered checkout of each pinned commit into .cache/src/ (gitignored); no-op when already there
list_files.py	which files are probed (HDF5/netCDF-4 extensions minus netCDF classic, plus the CVE reproducers)
probe/	the clawhdf5 side: a standalone crate (outside the workspace, so cargo test --workspace never builds it) that walks a file with clawhdf5-format and prints canonical JSON
ref.py	the h5py side: the same JSON from h5py (values corrected for a known h5py bug are marked ref_fix)
ref_bugs.py	re-reads the objects h5py reads only through a libhdf5 bug in six differently-set-up processes; an object whose values change is confirmed as a libhdf5 over-read
test_ref.py	tests of ref.py's correction and ref_bugs.py's confirmation (python conformance/test_ref.py)
run_one.sh	runs both sides on one file (and h5dump on the CVE corpus) under a timeout and an address-space limit
compare.py	classifies each file (ok / our-error / mismatch / h5py-cannot-read / ref-bug / panic / hang / crash / oom) and groups root causes
report.py	writes CONFORMANCE.md
check.py	the gate: fails on any panic/hang/crash/oom, on an ok count below baseline.json, or on a baseline-ok file that is no longer ok
baseline.json	the ok files the gate holds the line on
requirements.txt	pinned h5py / numpy / hdf5plugin / netCDF4

Results for every file (both sides' JSON and stderr, results.csv, results.json, summary.md) are left in .cache/results/.

The nightly job is .gitea/workflows/conformance.yml; it prints the report into the job log.

The canonical value encoding both sides hash is documented at the top of probe/src/main.rs. Values are compared as libhdf5 presents them: a float with a non-IEEE bit layout (N-Bit) or an integer with a bit offset is compared as the converted number, not as raw file bytes.