quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: conformance report with the last non-ok files classified (602 of 697 ok, 0 our-error, 0 mismatch)

Browse Source 
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-27 22:40:03 -05:00
co-authored by Claude Opus 5.5
parent ff0b2f8a4e
commit 05e1136027
3 changed files with 72 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+21
View File
@@ -2,6 +2,27 @@
## Unreleased
### Conformance: no our-errors or mismatches left (2026-09-27)
- The last 3 our-errors and 2 mismatches were documented as not ours but
still counted against us. Re-checked with evidence
(`docs/known-issues.md`, "Conformance: the last non-ok files"):
- The 2 mismatches were h5py's big-endian VL bug (elements returned with
the file's bytes under a little-endian dtype). `conformance/ref.py` now
checks the installed h5py has the bug and relabels such elements before
hashing, so their values are compared: `attr_datatypes.hdf5` and
`tcomplex_be.h5` are identical to clawhdf5's and now **ok**.
- The 3 our-errors are objects HDF5 2.0 reads by over-reading memory
(scale-offset codes past a chunk, unfiltered chunks shorter than a
chunk, an N-Bit parameter list one value short). h5py's values for them
change between runs, with `MALLOC_PERTURB_` and with import order, and
h5dump 1.14.6 prints others, so they are not the file's data and
clawhdf5 keeps refusing them. `conformance/ref_bugs.py` repeats that
check in every run, and such a file is the new class **ref-bug** only
while its values keep changing.
- Result (`conformance/run.sh --no-fetch`, tank, 2026-09-27): 602 of 697
ok (baseline 600), 0 our-error, 0 mismatch, 3 ref-bug, 92
h5py-cannot-read, no panic/hang/crash/oom.
### Deterministic errors on damaged chunked datasets (2026-09-27)
- A read through the file's chunk cache listed a damaged dataset's chunks in
hash-map order, seeded per `File`, so two opens of the same file could
CONFORMANCE.md
+49 -39
View File
@@ -13,15 +13,15 @@ fatal. This file is generated by `conformance/run.sh`; do not edit it by hand.
| | |
|---|---|
| date | 2026-09-27 00:34 UTC |
| clawhdf5 commit | `f37e7ae3263277319dba4bc39be5397194eb00c3` |
| date | 2026-09-28 03:33 UTC |
| clawhdf5 commit | `ff0b2f8a4e4ef105d7ff4694349ca475abf3f611` |
| machine | `tank`: AMD Ryzen 7 7800X3D 8-Core Processor, 16 CPUs, 61 GiB, Linux 7.0.0-34-generic x86_64 |
| command | `conformance/run.sh --no-fetch --update-baseline` |
| rustc | rustc 1.98.1 (48a229cea 2026-09-01) |
| command | `conformance/run.sh --no-fetch` |
| rustc | |
| reference | h5py 3.16.0, HDF5 2.0.0, numpy 2.5.3, hdf5plugin 7.1.0, Python 3.14.4 |
| h5dump | Version 1.14.6 (CVE corpus only) |
| limits | 20 s timeout (SIGKILL), 4096 MiB address space, per process; 16 files in parallel |
| runtime | 21 s probing + comparing (0 s fetch/build before it) |
| runtime | 21 s probing + comparing (18 s fetch/build before it) |
## Results
@@ -29,25 +29,24 @@ A file's class is the first that applies:
- **panic / hang / crash / oom** — clawhdf5 panicked (caught per object or not), hit the timeout, died on a signal, or failed an allocation. The CI gate fails on any of these.
- **h5py-cannot-read** — libhdf5 could not open the file (or itself crashed or hung). Nothing to compare against; most are the deliberately malformed CVE reproducers.
- **ref-bug** — every difference is an object clawhdf5 refuses that h5py reads only through a libhdf5 bug: the values h5py returns for it change with the reading process's heap, re-checked in every run (see *Reference bugs*).
- **our-error** — clawhdf5 returned an error for something h5py reads.
- **mismatch** — both read it, but the shapes, values, object set or attribute set differ.
- **ok** — every object h5py reads, clawhdf5 reads identically.
| corpus | files | ok | our-error | mismatch | h5py-cannot-read | panic | hang | crash | oom |
|---|---|---|---|---|---|---|---|---|---|
| NCAS-CMS_pyfive | 33 | 32 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
| cve_hdf5 | 147 | 113 | 2 | 0 | 32 | 0 | 0 | 0 | 0 |
| h5py_data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| hdf5 | 466 | 404 | 1 | 1 | 60 | 0 | 0 | 0 | 0 |
| netcdf-c | 20 | 20 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| netcdf4-python | 18 | 18 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| usnistgov_h5wasm | 5 | 5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| xarray-data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| **all** | **697** | **600** | **3** | **2** | **92** | **0** | **0** | **0** | **0** |
| corpus | files | ok | our-error | mismatch | h5py-cannot-read | ref-bug | panic | hang | crash | oom |
|---|---|---|---|---|---|---|---|---|---|---|
| NCAS-CMS_pyfive | 33 | 33 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| cve_hdf5 | 147 | 113 | 0 | 0 | 32 | 2 | 0 | 0 | 0 | 0 |
| h5py_data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| hdf5 | 466 | 405 | 0 | 0 | 60 | 1 | 0 | 0 | 0 | 0 |
| netcdf-c | 20 | 20 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| netcdf4-python | 18 | 18 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| usnistgov_h5wasm | 5 | 5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| xarray-data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| **all** | **697** | **602** | **0** | **0** | **92** | **3** | **0** | **0** | **0** | **0** |
2 of the 2 mismatches are a known h5py bug, not ours (see *Known not-our-bug*).
3 of the 3 our-errors are corrupt data that HDF5 2.0 reads only through a bug and clawhdf5 refuses (see *Known not-our-bug*).
**Our errors and mismatches: 0.** Files not ok: 92 h5py-cannot-read, 3 ref-bug. 2 object(s) were compared against h5py's values corrected for a known h5py bug (2 identical to clawhdf5's; see *Reference bugs*).
Corpora (fetched by `conformance/fetch-corpus.sh` into the gitignored `conformance/.cache/`):
@@ -68,18 +67,11 @@ None.
## Our-error root causes
Grouped by normalised error message. *files* counts files whose class this cause affects.
| files | objects | error | examples |
|---:|---:|---|---|
| 3 | 3 | `ChunkedReadError("…")` | `cve_hdf5/cvefiles/cve-2025-2308.h5`, `cve_hdf5/cvefiles/cve-2025-44904.h5`, `hdf5/test/testfiles/bad_nbit_parms_walk.h5` |
None.
## Mismatch root causes
| files | objects | cause | examples |
|---:|---:|---|---|
| 1 | 1 | `attr-values: ours=vlen(>u8) h5py=object layout=- filters=-` | `NCAS-CMS_pyfive/tests/data/attr_datatypes.hdf5` |
| 1 | 1 | `values: ours=vlen({r:>f4,i:>f4}8) h5py=object layout=contiguous filters=-` | `hdf5/tools/test/testfiles/tcomplex_be.h5` |
None.
## CVE corpus: clawhdf5 vs h5dump vs h5py
@@ -203,7 +195,7 @@ columns are.
| cvefiles/cve-2024-33876.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2024-33877.h5 | error exit | read 8 obj, 1 errors | read 8 obj, 1 errors | ok |
| cvefiles/cve-2025-2153.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
| cvefiles/cve-2025-2308.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 2 errors | our-error |
| cvefiles/cve-2025-2308.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 2 errors | ref-bug |
| cvefiles/cve-2025-2309.h5 | ok | read 6 obj, 1 errors | read 6 obj | ok |
| cvefiles/cve-2025-2310.h5 | error exit | read 24 obj, 8 errors | read 24 obj, 8 errors | ok |
| cvefiles/cve-2025-2912.h5 | error exit | open error | open error | h5py-cannot-read |
@@ -214,7 +206,7 @@ columns are.
| cvefiles/cve-2025-2924.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2025-2925.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2025-2926.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2025-44904.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 2 errors | our-error |
| cvefiles/cve-2025-44904.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 2 errors | ref-bug |
| cvefiles/cve-2025-44905.h5 | error exit | read 25 obj, 3 errors | read 25 obj, 3 errors | ok |
| cvefiles/cve-2025-6269-1.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2025-6269-2.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
@@ -249,13 +241,36 @@ columns are.
</details>
## Known not-our-bug
## Reference bugs
### Objects h5py reads only through a libhdf5 bug (*ref-bug*)
clawhdf5 refuses these objects; h5py 3.16 / HDF5 2.0 returns values for them. `conformance/ref_bugs.py`
re-reads each with h5py in six fresh processes whose heaps differ (h5py imported before numpy, three
times and twice more with `MALLOC_PERTURB_`, and numpy imported first). Values the file determines
come out the same every time; these do not, so they are memory libhdf5 over-reads, not the file's
data. A file is *ref-bug* only while every one of its differences is such an object confirmed in
the same run; an object that reads the same every time goes back to *our-error*. Reproducer:
`python conformance/ref_bugs.py conformance/.cache/corpus` (prints every read's outcome).
| file | object | distinct results in 6 reads | confirmed | what goes wrong |
|---|---|---:|---|---|
| `cve_hdf5/cvefiles/cve-2025-2308.h5` | `/Scale_offset_long_long_data_le` | 6 | yes | the first chunk records minbits 11: its 12 values need 17 bytes of codes, and the 26-byte chunk holds 5 after its 21-byte header; libhdf5's scale-offset decoder reads past its buffer, and develop refuses the chunk ("Buffer too short") |
| `cve_hdf5/cvefiles/cve-2025-44904.h5` | `/Scale_offset_float_data_le` | 6 | yes | unfiltered chunks stored as 38 and 37 bytes for 48-byte chunks: 1.14/2.0 read the stored bytes into a buffer of that size and use it as the whole chunk (H5D__chunk_lock), so the rest is heap memory; develop refuses them ("incorrect chunk size returned from index for unfiltered chunk") |
| `hdf5/test/testfiles/bad_nbit_parms_walk.h5` | `/Nbit_int_data_le` | 3 | yes | the N-Bit parameter list holds 7 values (cd_values[0] = 7) where an integer needs 8: the decoder takes the bit offset from cd_values[7], past the list; libhdf5's own test (`test_filter_bad_params`, test/dsets.c on develop) requires the read to fail |
### Values corrected for a known h5py bug
- **h5py big-endian variable-length sequences.** h5py returns the elements of a VL sequence
whose base type is big-endian with the file's big-endian bytes but a native (little-endian)
numpy dtype, so the values it reports are byte-swapped garbage; `h5dump` prints the values
clawhdf5 reads. Reproducer: `h5py.vlen_dtype(np.dtype('>f4'))` dataset holding `[1.0, 2.0]`
reads back in h5py as `[4.6e-41, 9.0e-44]`. Affected here: `NCAS-CMS_pyfive/tests/data/attr_datatypes.hdf5`, `hdf5/tools/test/testfiles/tcomplex_be.h5`.
numpy dtype: a `h5py.vlen_dtype(np.dtype('>f4'))` dataset holding `[1.0, 2.0]` reads back as
`[4.6e-41, 9.0e-44]`; `h5dump` prints the file's values. `ref.py` checks that the installed
h5py still does this (by writing and reading exactly that dataset in memory) and, if so,
relabels such elements with the file's byte order before hashing, so the values are still
compared. Corrected objects: `NCAS-CMS_pyfive/tests/data/attr_datatypes.hdf5` `/@vlen_uint64` (same as clawhdf5), `hdf5/tools/test/testfiles/tcomplex_be.h5` `/VariableLengthDatasetFloatComplex` (same as clawhdf5).
## Other comparison rules
- **Non-IEEE floats and partial-precision integers (N-Bit).** libhdf5 converts a float whose
bit layout is not IEEE (e.g. `H5Tset_precision` for the N-Bit filter) or an integer with a
bit offset / reduced precision into the plain numpy type of the same size. The probe
@@ -264,11 +279,6 @@ columns are.
- **Types h5py widens.** Where h5py reads a type into a numpy type of a different size
(FP8 -> float16, bfloat16 -> float32, x87 long double -> float128) the values are not
compared (shape and presence still are): dataset file type size 1 -> numpy float16 (2) (15x), attr file type size 1 -> numpy float16 (2) (15x), dataset file type size 2 -> numpy float32 (4) (2x), dataset file type size 8 -> numpy float128 (16) (1x), dataset file type size 12 -> numpy float128 (16) (1x), attr file type size 2 -> numpy float32 (4) (1x), dataset file type size 2 -> numpy >f4 (4) (1x), attr file type size 2 -> numpy >f4 (4) (1x).
- **Corrupt data HDF5 2.0 reads through a bug.** clawhdf5 refuses these objects; h5py 3.16 /
HDF5 2.0 returns values for them that the file does not hold:
- `cve_hdf5/cvefiles/cve-2025-2308.h5` `/Scale_offset_long_long_data_le`: scale-offset codes run past the end of the chunk: HDF5 2.0 reads past its buffer; libhdf5's develop branch refuses the chunk ("Buffer too short").
- `cve_hdf5/cvefiles/cve-2025-44904.h5` `/Scale_offset_float_data_le`: unfiltered chunks of 38 and 37 bytes for 48-byte chunks: HDF5 2.0 fills the rest with whatever its buffer held; libhdf5's develop branch refuses them ("incorrect chunk size returned from index for unfiltered chunk").
- `hdf5/test/testfiles/bad_nbit_parms_walk.h5` `/Nbit_int_data_le`: an N-Bit parameter list one value short: HDF5 2.0 reads past the list; libhdf5's own test (`test_filter_bad_params`, test/dsets.c) now requires the read to fail.
- **References** are compared by presence only (`R`), not by target.
## Objects h5py fails on but clawhdf5 reads
docs/known-issues.md
+2 -1
View File
@@ -66,7 +66,8 @@ as "not ours" but still counted against us. Re-checked on tank
conformance run (six reads per object in differently set-up processes) and
such a file is classified *ref-bug* only while its values keep changing.
Result (`conformance/run.sh --no-fetch`, 2026-09-27): see CHANGELOG.
Result (`conformance/run.sh --no-fetch`, tank, 2026-09-27): 602 of 697 ok
(was 600), 0 our-error, 0 mismatch, 3 ref-bug, 92 h5py-cannot-read.
## Files a SWMR writer had open could not be read past a stale end of file