# Conformance sweep

Reads every HDF5 file of eight public corpora with clawhdf5 and with
h5py/libhdf5, compares the two readings object by object, and writes
[`CONFORMANCE.md`](../CONFORMANCE.md).

```sh
CLAWHDF5_PYTHON=/path/to/venv/bin/python conformance/run.sh      # ~30 s once the corpus is cached
conformance/run.sh --no-fetch                                    # use the cached corpus as is
conformance/run.sh --update-baseline                             # after an intended change in results
```

Latest result (tank, 2026-09-28 04:29 UTC, `conformance/run.sh --no-fetch
--update-baseline`): 602 of 697 files ok, 1 our-error, 0 mismatch, 2
ref-bug, 92 h5py-cannot-read, and no panic, hang, crash or out-of-memory.
The our-error file is `bad_nbit_parms_walk.h5`, which flips between ref-bug
and our-error from run to run (see `docs/known-issues.md`). The report with every file is
[`CONFORMANCE.md`](../CONFORMANCE.md).

## Classes

`compare.py` puts each file in one class:

| class | meaning |
|---|---|
| **ok** | clawhdf5 and h5py read the same objects with the same values |
| **our-error** | h5py reads something clawhdf5 refuses |
| **mismatch** | both read it, with different values or structure |
| **h5py-cannot-read** | h5py (libhdf5) cannot read the file; not compared |
| **ref-bug** | h5py reads an object clawhdf5 refuses, but only through a libhdf5 over-read: `ref_bugs.py` re-reads it in six processes with different heaps (import order, `MALLOC_PERTURB_`) and its values change. The file is ref-bug only while that is confirmed in the same run; if the values become stable it counts as our-error again |
| **panic / hang / crash / oom** | a clawhdf5 failure under the timeout and address-space limit; the gate fails on any |

Where h5py itself returns wrong values through a known h5py bug (the
big-endian variable-length bug: elements returned with the file's bytes
under a little-endian dtype), `ref.py` checks that the installed h5py has
the bug, corrects the values before hashing and marks them `ref_fix`, so
those objects are still compared. The evidence for the three remaining
non-ok files (ref-bug or, for one, our-error) is under "Conformance: the last non-ok files" in
[`docs/known-issues.md`](../docs/known-issues.md).

Needs Rust, `git`, `h5dump` (Debian/Ubuntu `hdf5-tools`), `libaec` (for the
probe's `szip` feature; `libaec-dev`), and a Python with the packages in
`requirements.txt`. The first run downloads about 450 MB of sparse checkouts.

| file | role |
|---|---|
| `corpus.txt` | the corpora: git URL, pinned commit, swept root, sparse-checkout patterns |
| `fetch-corpus.sh` | shallow, sparse, blob-filtered checkout of each pinned commit into `.cache/src/` (gitignored); no-op when already there |
| `list_files.py` | which files are probed (HDF5/netCDF-4 extensions minus netCDF classic, plus the CVE reproducers) |
| `probe/` | the clawhdf5 side: a standalone crate (outside the workspace, so `cargo test --workspace` never builds it) that walks a file with `clawhdf5-format` and prints canonical JSON |
| `ref.py` | the h5py side: the same JSON from h5py (values corrected for a known h5py bug are marked `ref_fix`) |
| `ref_bugs.py` | re-reads the objects h5py reads only through a libhdf5 bug in six differently-set-up processes; an object whose values change is confirmed as a libhdf5 over-read |
| `test_ref.py` | tests of `ref.py`'s correction and `ref_bugs.py`'s confirmation (`python conformance/test_ref.py`) |
| `run_one.sh` | runs both sides on one file (and `h5dump` on the CVE corpus) under a timeout and an address-space limit |
| `compare.py` | classifies each file (ok / our-error / mismatch / h5py-cannot-read / ref-bug / panic / hang / crash / oom) and groups root causes |
| `report.py` | writes `CONFORMANCE.md` |
| `check.py` | the gate: fails on any panic/hang/crash/oom, on an ok count below `baseline.json`, or on a baseline-ok file that is no longer ok |
| `baseline.json` | the ok files the gate holds the line on |
| `requirements.txt` | pinned h5py / numpy / hdf5plugin / netCDF4 |

Results for every file (both sides' JSON and stderr, `results.csv`,
`results.json`, `summary.md`) are left in `.cache/results/`.

The nightly job is `.gitea/workflows/conformance.yml`; it prints the report
into the job log.

The canonical value encoding both sides hash is documented at the top of
`probe/src/main.rs`. Values are compared as libhdf5 presents them: a float
with a non-IEEE bit layout (N-Bit) or an integer with a bit offset is compared
as the converted number, not as raw file bytes.