conformance: compare h5py's big-endian VL values corrected, confirm libhdf5 over-reads per run

The last 3 our-errors and 2 mismatches were documented as not ours but
still counted against us, on a heuristic (any big-endian VL mismatch) and
a fixed list.

- ref.py checks that the installed h5py returns big-endian VL elements
  with the file's bytes under a little-endian dtype (writing and reading
  a vlen('>f4') in memory) and, if so, relabels them with the file's byte
  order before hashing, marking the object `ref_fix`. The values are now
  compared: attr_datatypes.hdf5 /@vlen_uint64 and tcomplex_be.h5
  /VariableLengthDatasetFloatComplex are identical to ours (h5dump 1.14.6
  prints the same (1, 2), (3, 4, 5), (42)).
- ref_bugs.py re-reads each object h5py reads only through a libhdf5 bug
  in six processes with different heaps (import order, MALLOC_PERTURB_).
  Values the file determines are the same every time; these three change
  (6, 6 and 3 distinct results), so they are over-read memory, not data
  clawhdf5 could match. compare.py classifies a file `ref-bug` only when
  every difference is such an object confirmed in the same run.
- report.py: the ref-bug class, the evidence table, the corrected
  objects; test_ref.py covers both (run in the nightly job).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs/known-issues.md

@@ -25,6 +25,49 @@ An earlier run the same day under load also listed single-thread contiguous
 hyperslab reads as 5.6% slower; the idle rerun puts them at −1.7% with
 overlapping ranges (noise), so that item is withdrawn.
 
+## Conformance: the last non-ok files (checked 2026-09-27)
+
+**Status:** classified with evidence — none is a clawhdf5 bug. The
+conformance report had 3 our-errors and 2 mismatches left, each documented
+as "not ours" but still counted against us. Re-checked on tank
+(h5py 3.16 / HDF5 2.0.0, h5dump 1.14.6):
+
+- **2 mismatches, h5py's big-endian VL bug:**
+  `NCAS-CMS_pyfive/tests/data/attr_datatypes.hdf5` `/@vlen_uint64` and
+  `hdf5/tools/test/testfiles/tcomplex_be.h5`
+  `/VariableLengthDatasetFloatComplex`. h5py returns the elements with the
+  file's big-endian bytes under a little-endian dtype (a `vlen('>f4')`
+  holding `[1.0, 2.0]` reads back as `[4.6e-41, 9.0e-44]`). h5dump 1.14.6
+  prints `(1, 2), (3, 4, 5), (42)` for `vlen_uint64`, which is what we read;
+  it cannot print `tcomplex_be.h5` (complex types are HDF5 2.0). The
+  reference side (`conformance/ref.py`) now checks that the installed h5py
+  has the bug and relabels such elements with the file's byte order, so the
+  values are compared rather than excused: both objects are identical to
+  ours, and both files are now *ok*.
+- **3 our-errors, objects HDF5 2.0 reads only by over-reading memory:**
+  `cve-2025-2308.h5` `/Scale_offset_long_long_data_le` (first chunk records
+  `minbits` 11; its 12 values need 17 bytes of codes and the 26-byte chunk
+  holds 5 after its 21-byte header), `cve-2025-44904.h5`
+  `/Scale_offset_float_data_le` (unfiltered chunks stored as 38 and 37 bytes
+  for 48-byte chunks; 1.14.6's `H5D__chunk_lock` reads the stored bytes
+  into a buffer of that size and uses it as the whole chunk) and
+  `bad_nbit_parms_walk.h5` `/Nbit_int_data_le` (N-Bit parameters
+  `(7, 0, 40, 1, 4, 0, 20)`: an integer needs 8, and the decoder takes the
+  bit offset from `cd_values[7]`, past the list). **Could we match
+  libhdf5?** No: its output is not determined by the file. h5py's values for
+  the first two change from one run to the next (three plain runs gave three
+  different results), and all three change with `MALLOC_PERTURB_` or with
+  whether numpy is imported before h5py (`bad_nbit_parms_walk` then fails
+  with "filter returned failure during read" or reads all zeros); h5dump
+  1.14.6 prints yet other values for the over-read parts (`1280, 0, 0` where
+  h5py gave e.g. `1283, 749, 1713`). libhdf5's develop branch refuses all
+  three. We keep
+  refusing them. `conformance/ref_bugs.py` repeats the check in every
+  conformance run (six reads per object in differently set-up processes) and
+  such a file is classified *ref-bug* only while its values keep changing.
+
+Result (`conformance/run.sh --no-fetch`, 2026-09-27): see CHANGELOG.
+
 ## Files a SWMR writer had open could not be read past a stale end of file
 
 **Status:** fixed 2026-09-27 (branch `feat/p3-m5-swmr-reader`), before any
@@ -505,7 +548,9 @@ fill-value item that did is fixed).
     short"); `bad_nbit_parms_walk.h5` has an N-Bit parameter list one value
     short (libhdf5's own `test_filter_bad_params` in `test/dsets.c` now
     requires that read to fail). We refuse both; `CONFORMANCE.md` lists them
-    under *Known not-our-bug*. Scale-offset did decode three cases
+    under *Known not-our-bug* (since 2026-09-27: the *ref-bug* class, with
+    the evidence re-checked every run — see *Conformance: the last non-ok
+    files* above). Scale-offset did decode three cases
     differently from libhdf5 (codes after a `minval` of recorded size other
     than 8, `minbits` 0 with a fill value, full-width `minbits`): fixed
     2026-09-26, and the full-width case was silent wrong data on ordinary