Reproducible HDF5 conformance sweep and nightly CI job #12

2026-09-26T03:50:28Z

osobh commented

2026-09-26 03:50:28 +00:00

Adds a conformance sweep in the repo that anyone can reproduce: conformance/run.sh. This is the "proof package" from Phase 1.

Every HDF5 file in eight public corpora is read by clawhdf5 and by h5py/libhdf5, and the two readings are compared object by object: which objects exist, and each dataset's and attribute's shape and value hash. The corpora are the libhdf5 test files, the HDF Group's cve_hdf5 reproducers, and the netcdf-c, netcdf4-python, pyfive, h5wasm, xarray and h5py corpora, each pinned by commit. The CVE corpus is also run through h5dump. Each process runs under a timeout and a memory limit, so a panic, hang, crash or out-of-memory kill is recorded instead of stopping the sweep.

What's included

conformance/:
- pinned corpus.txt and an idempotent fetch-corpus.sh, which makes sparse checkouts and verifies each commit;
- the Rust probe, as a standalone crate outside the workspace, so cargo test --workspace never builds it;
- ref.py, the reference reader;
- compare.py and report.py;
- check.py, the gate;
- baseline.json.
CONFORMANCE.md, generated on tank at 42b81d9 (current main). It records the date, machine, command, commit and tool versions.
.gitea/workflows/conformance.yml: a nightly job at 03:17 plus workflow_dispatch, running in rust:latest with no JavaScript actions. It fails on any panic, hang, crash or out-of-memory kill, and on any file in the baseline that stops reading correctly.
Probe fixes that make the comparison trustworthy:
- Non-IEEE floats and partial-precision integers are now compared as the values libhdf5 converts them to. Before, the 20-bit N-Bit float files showed as false mismatches.
- ref.py now exits with os._exit(0). libhdf5 2.0 aborts in H5T_close about half the time while h5py frees its objects (free(): chunks in smallbin corrupted), and that made two files flip between runs. Two consecutive sweeps now produce byte-identical results.

Result at `42b81d9`

	files	ok	our-error	mismatch	h5py-cannot-read	panic / hang / crash / oom
all	697	467	123	15	92	0

On the 147 CVE and fuzzer files, clawhdf5 had 0 crashes, h5dump 1.14.6 had 2 segfaults, and h5py/HDF5 2.0 had 1 segfault. The stacked follow-up PR (read gaps: layout v1/v2, VDS, user blocks, dense attributes and links) raises this to 569 ok / 14 our-error.

Notes

Network: the nightly job needs a runner that can reach github.com, where the corpora live. It downloads about 450 MB of sparse checkouts each night, because caching would need a JavaScript action.
Speed: on tank with the corpus cached, a run takes about 25 s. In CI it takes about 63 s from a cold start (checked in docker rust:latest).
README: not linked yet; I left that for you to decide.
Library code: unchanged.

🤖 Generated with Claude Code

Adds a conformance sweep in the repo that anyone can reproduce: `conformance/run.sh`. This is the "proof package" from Phase 1. Every HDF5 file in eight public corpora is read by clawhdf5 and by h5py/libhdf5, and the two readings are compared object by object: which objects exist, and each dataset's and attribute's shape and value hash. The corpora are the libhdf5 test files, the HDF Group's `cve_hdf5` reproducers, and the netcdf-c, netcdf4-python, pyfive, h5wasm, xarray and h5py corpora, each pinned by commit. The CVE corpus is also run through h5dump. Each process runs under a timeout and a memory limit, so a panic, hang, crash or out-of-memory kill is recorded instead of stopping the sweep. ## What's included - `conformance/`: - pinned `corpus.txt` and an idempotent `fetch-corpus.sh`, which makes sparse checkouts and verifies each commit; - the Rust probe, as a standalone crate outside the workspace, so `cargo test --workspace` never builds it; - `ref.py`, the reference reader; - `compare.py` and `report.py`; - `check.py`, the gate; - `baseline.json`. - `CONFORMANCE.md`, generated on tank at `42b81d9` (current main). It records the date, machine, command, commit and tool versions. - `.gitea/workflows/conformance.yml`: a nightly job at 03:17 plus `workflow_dispatch`, running in `rust:latest` with no JavaScript actions. It fails on any panic, hang, crash or out-of-memory kill, and on any file in the baseline that stops reading correctly. - Probe fixes that make the comparison trustworthy: - Non-IEEE floats and partial-precision integers are now compared as the values libhdf5 converts them to. Before, the 20-bit N-Bit float files showed as false mismatches. - `ref.py` now exits with `os._exit(0)`. libhdf5 2.0 aborts in `H5T_close` about half the time while h5py frees its objects (`free(): chunks in smallbin corrupted`), and that made two files flip between runs. Two consecutive sweeps now produce byte-identical results. ## Result at `42b81d9` | | files | ok | our-error | mismatch | h5py-cannot-read | panic / hang / crash / oom | |---|---|---|---|---|---|---| | all | 697 | 467 | 123 | 15 | 92 | **0** | On the 147 CVE and fuzzer files, clawhdf5 had 0 crashes, h5dump 1.14.6 had 2 segfaults, and h5py/HDF5 2.0 had 1 segfault. The stacked follow-up PR (read gaps: layout v1/v2, VDS, user blocks, dense attributes and links) raises this to **569 ok / 14 our-error**. ## Notes - **Network:** the nightly job needs a runner that can reach github.com, where the corpora live. It downloads about 450 MB of sparse checkouts each night, because caching would need a JavaScript action. - **Speed:** on tank with the corpus cached, a run takes about 25 s. In CI it takes about 63 s from a cold start (checked in `docker rust:latest`). - **README:** not linked yet; I left that for you to decide. - **Library code:** unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

osobh added 3 commits 2026-09-26 03:50:29 +00:00

feat(conformance): in-repo, reproducible conformance sweep 9179aa356e

conformance/run.sh fetches eight public HDF5 corpora pinned by commit
(conformance/corpus.txt) into a gitignored cache, reads every file with
clawhdf5 (conformance/probe, a crate outside the workspace) and with
h5py/libhdf5 (ref.py), and the CVE files with h5dump, each under a timeout
and an address-space limit; compare.py classifies the files, report.py
writes CONFORMANCE.md and check.py gates on panics/hangs/crashes/OOM and on
regressions against conformance/baseline.json. ~25 s once cached.

Changes from the ad-hoc audit harness:
- the probe compares non-IEEE-layout floats (N-Bit) and integers with a bit
  offset or reduced precision as the values libhdf5 converts them to, not
  raw file bytes: 8 files that showed as mismatches now read identically;
- ref.py exits without tearing down h5py objects: libhdf5 2.0 aborts while
  freeing them for two files about half the time, which flipped them
  between ok and h5py-cannot-read from run to run;
- the file list is defined (list_files.py): netCDF classic files are left
  out, 11 HDF5 files the ad-hoc sweep missed are in.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

ci: nightly conformance sweep 945b13a1f1

Runs conformance/run.sh in rust:latest on a schedule and on demand, with its
own venv (pinned h5py/numpy/hdf5plugin/netCDF4) and hdf5-tools. Fails on any
panic, hang, crash or OOM in clawhdf5 and on a drop against
conformance/baseline.json; prints CONFORMANCE.md into the job log and
uploads nothing. Plain git checkout, no JavaScript actions.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: first conformance report and baseline (42b81d9, tank)

CI / test-arm64 (pull_request) Successful in 1m6s

Details

CI / test (pull_request) Successful in 5m0s

Details

5c2f656fe7

467 of 697 files read identically to h5py 3.16 / HDF5 2.0, 123 our-error,
15 mismatch (2 an h5py big-endian VL bug), 92 libhdf5 cannot read; no
panics, hangs, crashes or OOM.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

osobh merged commit c9c5337a62 into main

2026-09-26 04:45:34 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: quantumclaw/clawhdf5#12

Reproducible HDF5 conformance sweep and nightly CI job #12

What's included

Result at 42b81d9

Notes

Result at `42b81d9`