conformance: compare h5py's big-endian VL values corrected, confirm libhdf5 over-reads per run

The last 3 our-errors and 2 mismatches were documented as not ours but
still counted against us, on a heuristic (any big-endian VL mismatch) and
a fixed list.

- ref.py checks that the installed h5py returns big-endian VL elements
  with the file's bytes under a little-endian dtype (writing and reading
  a vlen('>f4') in memory) and, if so, relabels them with the file's byte
  order before hashing, marking the object `ref_fix`. The values are now
  compared: attr_datatypes.hdf5 /@vlen_uint64 and tcomplex_be.h5
  /VariableLengthDatasetFloatComplex are identical to ours (h5dump 1.14.6
  prints the same (1, 2), (3, 4, 5), (42)).
- ref_bugs.py re-reads each object h5py reads only through a libhdf5 bug
  in six processes with different heaps (import order, MALLOC_PERTURB_).
  Values the file determines are the same every time; these three change
  (6, 6 and 3 distinct results), so they are over-read memory, not data
  clawhdf5 could match. compare.py classifies a file `ref-bug` only when
  every difference is such an object confirmed in the same run.
- report.py: the ref-bug class, the evidence table, the corrected
  objects; test_ref.py covers both (run in the nightly job).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-27 22:26:32 -05:00
co-authored by Claude Opus 5.5
parent 425585ee71
commit 5c44630ea2
9 changed files with 379 additions and 65 deletions
+2
View File
@@ -40,6 +40,8 @@ jobs:
run: cargo test --release --manifest-path conformance/probe/Cargo.toml
env:
CARGO_TARGET_DIR: conformance/.cache/target
- name: Reference-side tests
run: /opt/conformance/bin/python conformance/test_ref.py
- name: Sweep
# The corpora come from GitHub (pinned commits, conformance/corpus.txt),
# so this job needs a runner that reaches github.com.