From cab952bb007b1435c9cb034a92d729f33a7ed05a Mon Sep 17 00:00:00 2001 From: osobh Date: Mon, 28 Sep 2026 11:13:30 -0500 Subject: [PATCH] docs(conformance): file classes, ref-bug and ref_fix, latest result Explains each class compare.py assigns, how ref_bugs.py confirms a ref-bug in every run and how ref.py corrects h5py's big-endian VL values (ref_fix), and records the latest result (602 of 697 ok, 3 ref-bug, 2026-09-27) with a link to CONFORMANCE.md. Mentions --no-fetch. Co-Authored-By: Claude Opus 5.5 (1M context) --- conformance/README.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/conformance/README.md b/conformance/README.md index 919d1fe..b4b682c 100644 --- a/conformance/README.md +++ b/conformance/README.md @@ -6,9 +6,36 @@ h5py/libhdf5, compares the two readings object by object, and writes ```sh CLAWHDF5_PYTHON=/path/to/venv/bin/python conformance/run.sh # ~30 s once the corpus is cached +conformance/run.sh --no-fetch # use the cached corpus as is conformance/run.sh --update-baseline # after an intended change in results ``` +Latest result (tank, 2026-09-27, `conformance/run.sh --no-fetch`): 602 of +697 files ok, 0 our-error, 0 mismatch, 3 ref-bug, 92 h5py-cannot-read, and +no panic, hang, crash or out-of-memory. The report with every file is +[`CONFORMANCE.md`](../CONFORMANCE.md). + +## Classes + +`compare.py` puts each file in one class: + +| class | meaning | +|---|---| +| **ok** | clawhdf5 and h5py read the same objects with the same values | +| **our-error** | h5py reads something clawhdf5 refuses | +| **mismatch** | both read it, with different values or structure | +| **h5py-cannot-read** | h5py (libhdf5) cannot read the file; not compared | +| **ref-bug** | h5py reads an object clawhdf5 refuses, but only through a libhdf5 over-read: `ref_bugs.py` re-reads it in six processes with different heaps (import order, `MALLOC_PERTURB_`) and its values change. The file is ref-bug only while that is confirmed in the same run; if the values become stable it counts as our-error again | +| **panic / hang / crash / oom** | a clawhdf5 failure under the timeout and address-space limit; the gate fails on any | + +Where h5py itself returns wrong values through a known h5py bug (the +big-endian variable-length bug: elements returned with the file's bytes +under a little-endian dtype), `ref.py` checks that the installed h5py has +the bug, corrects the values before hashing and marks them `ref_fix`, so +those objects are still compared. The evidence for the three current +ref-bug files is under "Conformance: the last non-ok files" in +[`docs/known-issues.md`](../docs/known-issues.md). + Needs Rust, `git`, `h5dump` (Debian/Ubuntu `hdf5-tools`), `libaec` (for the probe's `szip` feature; `libaec-dev`), and a Python with the packages in `requirements.txt`. The first run downloads about 450 MB of sparse checkouts.