feat(conformance): in-repo, reproducible conformance sweep

conformance/run.sh fetches eight public HDF5 corpora pinned by commit
(conformance/corpus.txt) into a gitignored cache, reads every file with
clawhdf5 (conformance/probe, a crate outside the workspace) and with
h5py/libhdf5 (ref.py), and the CVE files with h5dump, each under a timeout
and an address-space limit; compare.py classifies the files, report.py
writes CONFORMANCE.md and check.py gates on panics/hangs/crashes/OOM and on
regressions against conformance/baseline.json. ~25 s once cached.

Changes from the ad-hoc audit harness:
- the probe compares non-IEEE-layout floats (N-Bit) and integers with a bit
  offset or reduced precision as the values libhdf5 converts them to, not
  raw file bytes: 8 files that showed as mismatches now read identically;
- ref.py exits without tearing down h5py objects: libhdf5 2.0 aborts while
  freeing them for two files about half the time, which flipped them
  between ok and h5py-cannot-read from run to run;
- the file list is defined (list_files.py): netCDF classic files are left
  out, 11 HDF5 files the ad-hoc sweep missed are in.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

conformance/README.md

@@ -0,0 +1,39 @@
+# Conformance sweep
+
+Reads every HDF5 file of eight public corpora with clawhdf5 and with
+h5py/libhdf5, compares the two readings object by object, and writes
+[`CONFORMANCE.md`](../CONFORMANCE.md).
+
+```sh
+CLAWHDF5_PYTHON=/path/to/venv/bin/python conformance/run.sh      # ~30 s once the corpus is cached
+conformance/run.sh --update-baseline                             # after an intended change in results
+```
+
+Needs Rust, `git`, `h5dump` (Debian/Ubuntu `hdf5-tools`), `libaec` (for the
+probe's `szip` feature; `libaec-dev`), and a Python with the packages in
+`requirements.txt`. The first run downloads about 450 MB of sparse checkouts.
+
+| file | role |
+|---|---|
+| `corpus.txt` | the corpora: git URL, pinned commit, swept root, sparse-checkout patterns |
+| `fetch-corpus.sh` | shallow, sparse, blob-filtered checkout of each pinned commit into `.cache/src/` (gitignored); no-op when already there |
+| `list_files.py` | which files are probed (HDF5/netCDF-4 extensions minus netCDF classic, plus the CVE reproducers) |
+| `probe/` | the clawhdf5 side: a standalone crate (outside the workspace, so `cargo test --workspace` never builds it) that walks a file with `clawhdf5-format` and prints canonical JSON |
+| `ref.py` | the h5py side: the same JSON from h5py |
+| `run_one.sh` | runs both sides on one file (and `h5dump` on the CVE corpus) under a timeout and an address-space limit |
+| `compare.py` | classifies each file (ok / our-error / mismatch / h5py-cannot-read / panic / hang / crash / oom) and groups root causes |
+| `report.py` | writes `CONFORMANCE.md` |
+| `check.py` | the gate: fails on any panic/hang/crash/oom, on an ok count below `baseline.json`, or on a baseline-ok file that is no longer ok |
+| `baseline.json` | the ok files the gate holds the line on |
+| `requirements.txt` | pinned h5py / numpy / hdf5plugin / netCDF4 |
+
+Results for every file (both sides' JSON and stderr, `results.csv`,
+`results.json`, `summary.md`) are left in `.cache/results/`.
+
+The nightly job is `.gitea/workflows/conformance.yml`; it prints the report
+into the job log.
+
+The canonical value encoding both sides hash is documented at the top of
+`probe/src/main.rs`. Values are compared as libhdf5 presents them: a float
+with a non-IEEE bit layout (N-Bit) or an integer with a bit offset is compared
+as the converted number, not as raw file bytes.