conformance: compare h5py's big-endian VL values corrected, confirm libhdf5 over-reads per run

The last 3 our-errors and 2 mismatches were documented as not ours but
still counted against us, on a heuristic (any big-endian VL mismatch) and
a fixed list.

- ref.py checks that the installed h5py returns big-endian VL elements
  with the file's bytes under a little-endian dtype (writing and reading
  a vlen('>f4') in memory) and, if so, relabels them with the file's byte
  order before hashing, marking the object `ref_fix`. The values are now
  compared: attr_datatypes.hdf5 /@vlen_uint64 and tcomplex_be.h5
  /VariableLengthDatasetFloatComplex are identical to ours (h5dump 1.14.6
  prints the same (1, 2), (3, 4, 5), (42)).
- ref_bugs.py re-reads each object h5py reads only through a libhdf5 bug
  in six processes with different heaps (import order, MALLOC_PERTURB_).
  Values the file determines are the same every time; these three change
  (6, 6 and 3 distinct results), so they are over-read memory, not data
  clawhdf5 could match. compare.py classifies a file `ref-bug` only when
  every difference is such an object confirmed in the same run.
- report.py: the ref-bug class, the evidence table, the corrected
  objects; test_ref.py covers both (run in the nightly job).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-27 22:26:32 -05:00
co-authored by Claude Opus 5.5
parent 425585ee71
commit 5c44630ea2
9 changed files with 379 additions and 65 deletions
+2
View File
@@ -71,6 +71,8 @@ xargs -a "$OUT/files.txt" -d '\n' -P "$JOBS" -I{} bash -c '
f="$1"; d="$OUT/runs/${f//\//__}"
case "$f" in cve_hdf5/*) export WITH_H5DUMP=1 ;; esac
"$HERE/run_one.sh" "$C/$f" "$d"' _ {} 2>"$OUT/probe.log"
echo "== re-checking the objects h5py reads only through a libhdf5 bug"
"$PY" "$HERE/ref_bugs.py" "$C" > "$OUT/ref_bugs.json" 2> "$OUT/ref_bugs.err" || true
echo "== comparing"
"$PY" "$HERE/compare.py" "$OUT" >/dev/null
t2=$(date +%s)