CHANGELOG.md

@@ -2,6 +2,27 @@
 
 ## Unreleased
 
+### Conformance: no our-errors or mismatches left (2026-09-27)
+- The last 3 our-errors and 2 mismatches were documented as not ours but
+  still counted against us. Re-checked with evidence
+  (`docs/known-issues.md`, "Conformance: the last non-ok files"):
+  - The 2 mismatches were h5py's big-endian VL bug (elements returned with
+    the file's bytes under a little-endian dtype). `conformance/ref.py` now
+    checks the installed h5py has the bug and relabels such elements before
+    hashing, so their values are compared: `attr_datatypes.hdf5` and
+    `tcomplex_be.h5` are identical to clawhdf5's and now **ok**.
+  - The 3 our-errors are objects HDF5 2.0 reads by over-reading memory
+    (scale-offset codes past a chunk, unfiltered chunks shorter than a
+    chunk, an N-Bit parameter list one value short). h5py's values for them
+    change between runs, with `MALLOC_PERTURB_` and with import order, and
+    h5dump 1.14.6 prints others, so they are not the file's data and
+    clawhdf5 keeps refusing them. `conformance/ref_bugs.py` repeats that
+    check in every run, and such a file is the new class **ref-bug** only
+    while its values keep changing.
+- Result (`conformance/run.sh --no-fetch`, tank, 2026-09-27): 602 of 697
+  ok (baseline 600), 0 our-error, 0 mismatch, 3 ref-bug, 92
+  h5py-cannot-read, no panic/hang/crash/oom.
+
 ### Deterministic errors on damaged chunked datasets (2026-09-27)
 - A read through the file's chunk cache listed a damaged dataset's chunks in
   hash-map order, seeded per `File`, so two opens of the same file could

CONFORMANCE.md

@@ -13,15 +13,15 @@ fatal. This file is generated by `conformance/run.sh`; do not edit it by hand.
 
 | | |
 |---|---|
-| date | 2026-09-27 00:34 UTC |
+| date | 2026-09-28 03:33 UTC |
-| clawhdf5 commit | `f37e7ae3263277319dba4bc39be5397194eb00c3` |
+| clawhdf5 commit | `ff0b2f8a4e4ef105d7ff4694349ca475abf3f611` |
 | machine | `tank`: AMD Ryzen 7 7800X3D 8-Core Processor, 16 CPUs, 61 GiB, Linux 7.0.0-34-generic x86_64 |
-| command | `conformance/run.sh --no-fetch --update-baseline` |
+| command | `conformance/run.sh --no-fetch` |
-| rustc | rustc 1.98.1 (48a229cea 2026-09-01) |
+| rustc |  |
 | reference | h5py 3.16.0, HDF5 2.0.0, numpy 2.5.3, hdf5plugin 7.1.0, Python 3.14.4 |
 | h5dump | Version 1.14.6 (CVE corpus only) |
 | limits | 20 s timeout (SIGKILL), 4096 MiB address space, per process; 16 files in parallel |
-| runtime | 21 s probing + comparing (0 s fetch/build before it) |
+| runtime | 21 s probing + comparing (18 s fetch/build before it) |
 
 ## Results
 
@@ -29,25 +29,24 @@ A file's class is the first that applies:
 
 - **panic / hang / crash / oom** — clawhdf5 panicked (caught per object or not), hit the timeout, died on a signal, or failed an allocation. The CI gate fails on any of these.
 - **h5py-cannot-read** — libhdf5 could not open the file (or itself crashed or hung). Nothing to compare against; most are the deliberately malformed CVE reproducers.
+- **ref-bug** — every difference is an object clawhdf5 refuses that h5py reads only through a libhdf5 bug: the values h5py returns for it change with the reading process's heap, re-checked in every run (see *Reference bugs*).
 - **our-error** — clawhdf5 returned an error for something h5py reads.
 - **mismatch** — both read it, but the shapes, values, object set or attribute set differ.
 - **ok** — every object h5py reads, clawhdf5 reads identically.
 
-| corpus | files | ok | our-error | mismatch | h5py-cannot-read | panic | hang | crash | oom |
+| corpus | files | ok | our-error | mismatch | h5py-cannot-read | ref-bug | panic | hang | crash | oom |
-|---|---|---|---|---|---|---|---|---|---|
+|---|---|---|---|---|---|---|---|---|---|---|
-| NCAS-CMS_pyfive | 33 | 32 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
+| NCAS-CMS_pyfive | 33 | 33 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
-| cve_hdf5 | 147 | 113 | 2 | 0 | 32 | 0 | 0 | 0 | 0 |
+| cve_hdf5 | 147 | 113 | 0 | 0 | 32 | 2 | 0 | 0 | 0 | 0 |
-| h5py_data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+| h5py_data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
-| hdf5 | 466 | 404 | 1 | 1 | 60 | 0 | 0 | 0 | 0 |
+| hdf5 | 466 | 405 | 0 | 0 | 60 | 1 | 0 | 0 | 0 | 0 |
-| netcdf-c | 20 | 20 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+| netcdf-c | 20 | 20 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
-| netcdf4-python | 18 | 18 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+| netcdf4-python | 18 | 18 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
-| usnistgov_h5wasm | 5 | 5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+| usnistgov_h5wasm | 5 | 5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
-| xarray-data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+| xarray-data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
-| **all** | **697** | **600** | **3** | **2** | **92** | **0** | **0** | **0** | **0** |
+| **all** | **697** | **602** | **0** | **0** | **92** | **3** | **0** | **0** | **0** | **0** |
 
-2 of the 2 mismatches are a known h5py bug, not ours (see *Known not-our-bug*).
+**Our errors and mismatches: 0.** Files not ok: 92 h5py-cannot-read, 3 ref-bug. 2 object(s) were compared against h5py's values corrected for a known h5py bug (2 identical to clawhdf5's; see *Reference bugs*).
-
-3 of the 3 our-errors are corrupt data that HDF5 2.0 reads only through a bug and clawhdf5 refuses (see *Known not-our-bug*).
 
 Corpora (fetched by `conformance/fetch-corpus.sh` into the gitignored `conformance/.cache/`):
 
@@ -68,18 +67,11 @@ None.
 
 ## Our-error root causes
 
-Grouped by normalised error message. *files* counts files whose class this cause affects.
+None.
-
-| files | objects | error | examples |
-|---:|---:|---|---|
-| 3 | 3 | `ChunkedReadError("…")` | `cve_hdf5/cvefiles/cve-2025-2308.h5`, `cve_hdf5/cvefiles/cve-2025-44904.h5`, `hdf5/test/testfiles/bad_nbit_parms_walk.h5` |
 
 ## Mismatch root causes
 
-| files | objects | cause | examples |
+None.
-|---:|---:|---|---|
-| 1 | 1 | `attr-values: ours=vlen(>u8) h5py=object layout=- filters=-` | `NCAS-CMS_pyfive/tests/data/attr_datatypes.hdf5` |
-| 1 | 1 | `values: ours=vlen({r:>f4,i:>f4}8) h5py=object layout=contiguous filters=-` | `hdf5/tools/test/testfiles/tcomplex_be.h5` |
 
 ## CVE corpus: clawhdf5 vs h5dump vs h5py
 
@@ -203,7 +195,7 @@ columns are.
 | cvefiles/cve-2024-33876.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2024-33877.h5 | error exit | read 8 obj, 1 errors | read 8 obj, 1 errors | ok |
 | cvefiles/cve-2025-2153.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
-| cvefiles/cve-2025-2308.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 2 errors | our-error |
+| cvefiles/cve-2025-2308.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 2 errors | ref-bug |
 | cvefiles/cve-2025-2309.h5 | ok | read 6 obj, 1 errors | read 6 obj | ok |
 | cvefiles/cve-2025-2310.h5 | error exit | read 24 obj, 8 errors | read 24 obj, 8 errors | ok |
 | cvefiles/cve-2025-2912.h5 | error exit | open error | open error | h5py-cannot-read |
@@ -214,7 +206,7 @@ columns are.
 | cvefiles/cve-2025-2924.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2025-2925.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2025-2926.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-44904.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 2 errors | our-error |
+| cvefiles/cve-2025-44904.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 2 errors | ref-bug |
 | cvefiles/cve-2025-44905.h5 | error exit | read 25 obj, 3 errors | read 25 obj, 3 errors | ok |
 | cvefiles/cve-2025-6269-1.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2025-6269-2.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
@@ -249,13 +241,36 @@ columns are.
 
 </details>
 
-## Known not-our-bug
+## Reference bugs
+
+### Objects h5py reads only through a libhdf5 bug (*ref-bug*)
+
+clawhdf5 refuses these objects; h5py 3.16 / HDF5 2.0 returns values for them. `conformance/ref_bugs.py`
+re-reads each with h5py in six fresh processes whose heaps differ (h5py imported before numpy, three
+times and twice more with `MALLOC_PERTURB_`, and numpy imported first). Values the file determines
+come out the same every time; these do not, so they are memory libhdf5 over-reads, not the file's
+data. A file is *ref-bug* only while every one of its differences is such an object confirmed in
+the same run; an object that reads the same every time goes back to *our-error*. Reproducer:
+`python conformance/ref_bugs.py conformance/.cache/corpus` (prints every read's outcome).
+
+| file | object | distinct results in 6 reads | confirmed | what goes wrong |
+|---|---|---:|---|---|
+| `cve_hdf5/cvefiles/cve-2025-2308.h5` | `/Scale_offset_long_long_data_le` | 6 | yes | the first chunk records minbits 11: its 12 values need 17 bytes of codes, and the 26-byte chunk holds 5 after its 21-byte header; libhdf5's scale-offset decoder reads past its buffer, and develop refuses the chunk ("Buffer too short") |
+| `cve_hdf5/cvefiles/cve-2025-44904.h5` | `/Scale_offset_float_data_le` | 6 | yes | unfiltered chunks stored as 38 and 37 bytes for 48-byte chunks: 1.14/2.0 read the stored bytes into a buffer of that size and use it as the whole chunk (H5D__chunk_lock), so the rest is heap memory; develop refuses them ("incorrect chunk size returned from index for unfiltered chunk") |
+| `hdf5/test/testfiles/bad_nbit_parms_walk.h5` | `/Nbit_int_data_le` | 3 | yes | the N-Bit parameter list holds 7 values (cd_values[0] = 7) where an integer needs 8: the decoder takes the bit offset from cd_values[7], past the list; libhdf5's own test (`test_filter_bad_params`, test/dsets.c on develop) requires the read to fail |
+
+### Values corrected for a known h5py bug
 
 - **h5py big-endian variable-length sequences.** h5py returns the elements of a VL sequence
   whose base type is big-endian with the file's big-endian bytes but a native (little-endian)
-  numpy dtype, so the values it reports are byte-swapped garbage; `h5dump` prints the values
+  numpy dtype: a `h5py.vlen_dtype(np.dtype('>f4'))` dataset holding `[1.0, 2.0]` reads back as
-  clawhdf5 reads. Reproducer: `h5py.vlen_dtype(np.dtype('>f4'))` dataset holding `[1.0, 2.0]`
+  `[4.6e-41, 9.0e-44]`; `h5dump` prints the file's values. `ref.py` checks that the installed
-  reads back in h5py as `[4.6e-41, 9.0e-44]`. Affected here: `NCAS-CMS_pyfive/tests/data/attr_datatypes.hdf5`, `hdf5/tools/test/testfiles/tcomplex_be.h5`.
+  h5py still does this (by writing and reading exactly that dataset in memory) and, if so,
+  relabels such elements with the file's byte order before hashing, so the values are still
+  compared. Corrected objects: `NCAS-CMS_pyfive/tests/data/attr_datatypes.hdf5` `/@vlen_uint64` (same as clawhdf5), `hdf5/tools/test/testfiles/tcomplex_be.h5` `/VariableLengthDatasetFloatComplex` (same as clawhdf5).
+
+## Other comparison rules
+
 - **Non-IEEE floats and partial-precision integers (N-Bit).** libhdf5 converts a float whose
   bit layout is not IEEE (e.g. `H5Tset_precision` for the N-Bit filter) or an integer with a
   bit offset / reduced precision into the plain numpy type of the same size. The probe
@@ -264,11 +279,6 @@ columns are.
 - **Types h5py widens.** Where h5py reads a type into a numpy type of a different size
   (FP8 -> float16, bfloat16 -> float32, x87 long double -> float128) the values are not
   compared (shape and presence still are): dataset file type size 1 -> numpy float16 (2) (15x), attr file type size 1 -> numpy float16 (2) (15x), dataset file type size 2 -> numpy float32 (4) (2x), dataset file type size 8 -> numpy float128 (16) (1x), dataset file type size 12 -> numpy float128 (16) (1x), attr file type size 2 -> numpy float32 (4) (1x), dataset file type size 2 -> numpy >f4 (4) (1x), attr file type size 2 -> numpy >f4 (4) (1x).
-- **Corrupt data HDF5 2.0 reads through a bug.** clawhdf5 refuses these objects; h5py 3.16 /
-  HDF5 2.0 returns values for them that the file does not hold:
-  - `cve_hdf5/cvefiles/cve-2025-2308.h5` `/Scale_offset_long_long_data_le`: scale-offset codes run past the end of the chunk: HDF5 2.0 reads past its buffer; libhdf5's develop branch refuses the chunk ("Buffer too short").
-  - `cve_hdf5/cvefiles/cve-2025-44904.h5` `/Scale_offset_float_data_le`: unfiltered chunks of 38 and 37 bytes for 48-byte chunks: HDF5 2.0 fills the rest with whatever its buffer held; libhdf5's develop branch refuses them ("incorrect chunk size returned from index for unfiltered chunk").
-  - `hdf5/test/testfiles/bad_nbit_parms_walk.h5` `/Nbit_int_data_le`: an N-Bit parameter list one value short: HDF5 2.0 reads past the list; libhdf5's own test (`test_filter_bad_params`, test/dsets.c) now requires the read to fail.
 - **References** are compared by presence only (`R`), not by target.
 
 ## Objects h5py fails on but clawhdf5 reads

docs/known-issues.md

@@ -66,7 +66,8 @@ as "not ours" but still counted against us. Re-checked on tank
   conformance run (six reads per object in differently set-up processes) and
   such a file is classified *ref-bug* only while its values keep changing.
 
-Result (`conformance/run.sh --no-fetch`, 2026-09-27): see CHANGELOG.
+Result (`conformance/run.sh --no-fetch`, tank, 2026-09-27): 602 of 697 ok
+(was 600), 0 our-error, 0 mismatch, 3 ref-bug, 92 h5py-cannot-read.
 
 ## Files a SWMR writer had open could not be read past a stale end of file