quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: conformance report after the read-gap fixes (569 of 697 ok)

Browse Source 
Regenerated on tank at 10d1029: ok 467 -> 569, our-error 123 -> 14,
mismatch 15 -> 22 (six user-defined-link files moved from our-error to a
listing difference), no panics, hangs, crashes or OOM. Baseline raised.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-25 22:46:37 -05:00
co-authored by Claude Opus 5.5
parent 10d1029ead
commit a7de15534c
2 changed files with 149 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONFORMANCE.md
+33 -35
View File
@@ -13,15 +13,15 @@ fatal. This file is generated by `conformance/run.sh`; do not edit it by hand.
| | |
|---|---|
| date | 2026-09-26 03:05 UTC |
| clawhdf5 commit | `42b81d9f1c3d9bef6050ad8a1326ac8c97f641d3` |
| date | 2026-09-26 03:46 UTC |
| clawhdf5 commit | `10d1029ead524e2fe64c2cd7f61b28067d9e449c` |
| machine | `tank`: AMD Ryzen 7 7800X3D 8-Core Processor, 16 CPUs, 61 GiB, Linux 7.0.0-34-generic x86_64 |
| command | `conformance/run.sh --update-baseline` |
| command | `conformance/run.sh --no-fetch --update-baseline` |
| rustc | rustc 1.98.1 (48a229cea 2026-09-01) |
| reference | h5py 3.16.0, HDF5 2.0.0, numpy 2.5.3, hdf5plugin 7.1.0, Python 3.14.4 |
| h5dump | Version 1.14.6 (CVE corpus only) |
| limits | 20 s timeout (SIGKILL), 4096 MiB address space, per process; 16 files in parallel |
| runtime | 25 s probing + comparing (0 s fetch/build before it) |
| runtime | 22 s probing + comparing (0 s fetch/build before it) |
## Results
@@ -35,17 +35,17 @@ A file's class is the first that applies:
| corpus | files | ok | our-error | mismatch | h5py-cannot-read | panic | hang | crash | oom |
|---|---|---|---|---|---|---|---|---|---|
| NCAS-CMS_pyfive | 33 | 31 | 1 | 1 | 0 | 0 | 0 | 0 | 0 |
| cve_hdf5 | 147 | 87 | 17 | 11 | 32 | 0 | 0 | 0 | 0 |
| NCAS-CMS_pyfive | 33 | 32 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
| cve_hdf5 | 147 | 100 | 6 | 9 | 32 | 0 | 0 | 0 | 0 |
| h5py_data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| hdf5 | 466 | 300 | 103 | 3 | 60 | 0 | 0 | 0 | 0 |
| hdf5 | 466 | 386 | 8 | 12 | 60 | 0 | 0 | 0 | 0 |
| netcdf-c | 20 | 20 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| netcdf4-python | 18 | 16 | 2 | 0 | 0 | 0 | 0 | 0 | 0 |
| netcdf4-python | 18 | 18 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| usnistgov_h5wasm | 5 | 5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| xarray-data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| **all** | **697** | **467** | **123** | **15** | **92** | **0** | **0** | **0** | **0** |
| **all** | **697** | **569** | **14** | **22** | **92** | **0** | **0** | **0** | **0** |
2 of the 15 mismatches are a known h5py bug, not ours (see *Known not-our-bug*).
2 of the 22 mismatches are a known h5py bug, not ours (see *Known not-our-bug*).
Corpora (fetched by `conformance/fetch-corpus.sh` into the gitignored `conformance/.cache/`):
@@ -70,29 +70,26 @@ Grouped by normalised error message. *files* counts files whose class this cause
| files | objects | error | examples |
|---:|---:|---|---|
| 84 | 205 | `InvalidLayoutVersion(N)` | `cve_hdf5/cvefiles/cve-2016-4330.h5`, `cve_hdf5/cvefiles/cve-2016-4333.h5`, `cve_hdf5/cvefiles/cve-2018-11206-old.h5` (+81 more) |
| 10 | 10 | `ChunkedReadError("…")` | `cve_hdf5/cvefiles/cve-2025-2308.h5`, `hdf5/test/testfiles/bad_nbit_parms_walk.h5`, `hdf5/tools/test/testfiles/vds/1_vds.h5` (+7 more) |
| 9 | 17 | `InvalidObjectHeaderVersion(N)` | `cve_hdf5/cvefiles/cve-2021-36977.h5`, `cve_hdf5/cvefiles/unknown-1.h5`, `hdf5/tools/test/testfiles/h5clear_fsm_persist_user_equal.h5` (+6 more) |
| 6 | 6 | `UnsupportedFilter(N)` | `hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_blosc.h5`, `hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_blosc2.h5`, `hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_bshuf.h5` (+3 more) |
| 6 | 6 | `InvalidLinkType(N)` | `hdf5/tools/test/testfiles/bigendian/tall.h5`, `hdf5/tools/test/testfiles/h5diff_types.h5`, `hdf5/tools/test/testfiles/tall.h5` (+3 more) |
| 5 | 5 | `UnexpectedEof { expected: N, available: N }` | `NCAS-CMS_pyfive/tests/data/cmip_bad_eg.nc`, `cve_hdf5/cvefiles/cve-2019-9151.h5`, `hdf5/tools/test/testfiles/h5stat_newgrat.h5` (+2 more) |
| 3 | 3 | `DataSizeMismatch { expected: N, actual: N }` | `cve_hdf5/cvefiles/cve-2020-18494.h5`, `cve_hdf5/cvefiles/cve-2024-32623.h5`, `cve_hdf5/cvefiles/cve-2025-2309.h5` |
| 2 | 2 | `ChunkedReadError("…")` | `cve_hdf5/cvefiles/cve-2025-2308.h5`, `hdf5/test/testfiles/bad_nbit_parms_walk.h5` |
| 1 | 1 | `UnexpectedEof { expected: N, available: N }` | `cve_hdf5/cvefiles/cve-2019-9151.h5` |
| 1 | 1 | `MissingMessage(Dataspace)` | `cve_hdf5/cvefiles/cve-2024-33874.h5` |
| 1 | 2 | `InvalidSharedMessageVersion(N)` | `hdf5/tools/test/testfiles/h5stat_tsohm.h5` |
| 1 | 1 | `InvalidObjectHeaderVersion(N)` | `hdf5/tools/test/testfiles/h5clear_mdc_image.h5` |
## Mismatch root causes
| files | objects | cause | examples |
|---:|---:|---|---|
| 9 | 27 | `missing-object` | `cve_hdf5/cvefiles/cve-2019-8397.h5`, `cve_hdf5/cvefiles/cve-2019-8398.h5`, `cve_hdf5/cvefiles/cve-2021-46243.h5` (+6 more) |
| 5 | 11 | `extra-object` | `cve_hdf5/cvefiles/cve-2021-46244.h5`, `cve_hdf5/cvefiles/cve-2024-32613.h5`, `cve_hdf5/cvefiles/cve-2024-32616.h5` (+2 more) |
| 13 | 14 | `missing-object` | `cve_hdf5/cvefiles/cve-2019-8397.h5`, `cve_hdf5/cvefiles/cve-2019-8398.h5`, `cve_hdf5/cvefiles/cve-2021-46243.h5` (+10 more) |
| 3 | 7 | `extra-attr` | `cve_hdf5/cvefiles/cve-2018-17438`, `cve_hdf5/cvefiles/cve-2018-17439`, `cve_hdf5/cvefiles/cve-2024-33874.h5` |
| 2 | 4 | `missing-attr` | `hdf5/tools/test/testfiles/twithub.h5`, `hdf5/tools/test/testfiles/twithub513.h5` |
| 3 | 6 | `extra-object` | `cve_hdf5/cvefiles/cve-2021-46244.h5`, `hdf5/tools/test/testfiles/h5clear_fsm_persist_less.h5`, `hdf5/tools/test/testfiles/h5stat_err_refcount.h5` |
| 1 | 1 | `attr-values: ours=vlen(>u8) h5py=object layout=- filters=-` | `NCAS-CMS_pyfive/tests/data/attr_datatypes.hdf5` |
| 1 | 1 | `values: ours=<f4 h5py=float32 layout=chunked filters=-` | `cve_hdf5/cvefiles/cve-2025-44904.h5` |
| 1 | 1 | `values: ours=>i2 h5py=>i2 layout=chunked filters=[6]` | `cve_hdf5/cvefiles/cve-2025-44905.h5` |
| 1 | 1 | `values: ours=>f4 h5py=>f4 layout=chunked filters=[2]` | `cve_hdf5/cvefiles/cve-2025-44905.h5` |
| 1 | 1 | `values: ours=<f4 h5py=float32 layout=chunked filters=[2]` | `cve_hdf5/cvefiles/cve-2025-44905.h5` |
| 1 | 1 | `values: ours=((<i4)[6, 3])[4] h5py=(('<i4', (6, 3)), (4,)) layout=contiguous filters=-` | `hdf5/tools/test/testfiles/tarray3.h5` |
| 1 | 1 | `values: ours=vlen({r:>f4,i:>f4}8) h5py=object layout=contiguous filters=-` | `hdf5/tools/test/testfiles/tcomplex_be.h5` |
## CVE corpus: clawhdf5 vs h5dump vs h5py
@@ -113,12 +110,12 @@ columns are.
| file | h5dump | h5py | clawhdf5 | class |
|---|---|---|---|---|
| cvefiles/cve-2016-4330.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | our-error |
| cvefiles/cve-2016-4330.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2016-4331.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 1 errors | ok |
| cvefiles/cve-2016-4332-mtime-new.h5 | error exit | read 25 obj, 1 errors | read 25 obj | ok |
| cvefiles/cve-2016-4332-mtime.h5 | error exit | read 4 obj, 3 errors | read 4 obj, 1 errors | ok |
| cvefiles/cve-2016-4332-mtime.h5 | error exit | read 4 obj, 3 errors | read 4 obj | ok |
| cvefiles/cve-2016-4332-stab.h5 | error exit | open error | read 65 obj | h5py-cannot-read |
| cvefiles/cve-2016-4333.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | our-error |
| cvefiles/cve-2016-4333.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2017-17505.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2017-17506.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2017-17507.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
@@ -129,23 +126,23 @@ columns are.
| cvefiles/cve-2018-11204.h5 | error exit | read 2 obj, 1 errors | read 2 obj | ok |
| cvefiles/cve-2018-11205.h5 | error exit | read 2 obj, 1 errors | read 2 obj | ok |
| cvefiles/cve-2018-11206-new.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-11206-old.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | our-error |
| cvefiles/cve-2018-11206-old.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-11207.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2018-13866.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2018-13867.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2018-13868.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | our-error |
| cvefiles/cve-2018-13868.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-13869.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2018-13870.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2018-13871.h5 | error exit | read 2 obj | read 2 obj | ok |
| cvefiles/cve-2018-13872.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2018-13873.h5 | error exit | read 1 obj, 1 errors | read 1 obj | ok |
| cvefiles/cve-2018-13874.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
| cvefiles/cve-2018-13875.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | our-error |
| cvefiles/cve-2018-13875.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-13876.h5 | error exit | open error | read 2 obj, 1 errors | h5py-cannot-read |
| cvefiles/cve-2018-14031.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | our-error |
| cvefiles/cve-2018-14031.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-14033.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-14034.h5 | error exit | read 1 obj, 2 errors | read 1 obj | ok |
| cvefiles/cve-2018-14035.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | our-error |
| cvefiles/cve-2018-14035.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-14460.h5 | error exit | read 3 obj, 2 errors | read 3 obj, 2 errors | ok |
| cvefiles/cve-2018-15671.h5 | ok | read 1 obj | read 1 obj | ok |
| cvefiles/cve-2018-15672.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
@@ -172,7 +169,7 @@ columns are.
| cvefiles/cve-2020-10812.h5 | error exit | open error | read 2 obj | h5py-cannot-read |
| cvefiles/cve-2020-18232.h5 | error exit | read 3 obj, 2 errors | read 3 obj, 2 errors | ok |
| cvefiles/cve-2020-18494.h5 | ok | read 2 obj | read 2 obj, 1 errors | our-error |
| cvefiles/cve-2021-36977.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | our-error |
| cvefiles/cve-2021-36977.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2021-37501.h5 | error exit | read 18 obj, 1 errors | read 18 obj, 1 errors | ok |
| cvefiles/cve-2021-45829.h5 | error exit | read 1 obj, 2 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2021-45830.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
@@ -187,7 +184,7 @@ columns are.
| cvefiles/cve-2024-29161.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 2 errors | ok |
| cvefiles/cve-2024-29162.h5 | error exit | read 17 obj, 4 errors | read 17 obj, 3 errors | ok |
| cvefiles/cve-2024-29163.h5 | error exit | read 7 obj, 1 errors | read 7 obj, 1 errors | ok |
| cvefiles/cve-2024-29164.h5 | ok | read 3 obj | read 3 obj, 2 errors | our-error |
| cvefiles/cve-2024-29164.h5 | ok | read 3 obj | read 3 obj | ok |
| cvefiles/cve-2024-29165.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2024-29166.h5 | error exit | read 17 obj, 2 errors | read 17 obj | ok |
| cvefiles/cve-2024-32605.h5 | ok | read 6 obj, 1 errors | read 6 obj, 1 errors | ok |
@@ -195,14 +192,14 @@ columns are.
| cvefiles/cve-2024-32607-1.h5 | ok | read 10 obj | read 10 obj | ok |
| cvefiles/cve-2024-32607-2.h5 | error exit | read 9 obj, 1 errors | read 9 obj, 1 errors | ok |
| cvefiles/cve-2024-32608.h5 | error exit | read 6 obj, 1 errors | read 6 obj | ok |
| cvefiles/cve-2024-32609.h5 | error exit | SIGSEGV | read 4 obj, 1 errors | h5py-cannot-read |
| cvefiles/cve-2024-32609.h5 | error exit | SIGSEGV | read 3 obj, 1 errors | h5py-cannot-read |
| cvefiles/cve-2024-32610.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2024-32611.h5 | ok | read 6 obj | read 6 obj | ok |
| cvefiles/cve-2024-32612.h5 | ok | read 3 obj | read 3 obj, 2 errors | our-error |
| cvefiles/cve-2024-32613.h5 | error exit | read 7 obj, 1 errors | read 11 obj | mismatch |
| cvefiles/cve-2024-32612.h5 | ok | read 3 obj | read 3 obj | ok |
| cvefiles/cve-2024-32613.h5 | error exit | read 7 obj, 1 errors | read 7 obj, 1 errors | ok |
| cvefiles/cve-2024-32614.h5 | error exit | read 25 obj, 2 errors | read 25 obj, 1 errors | ok |
| cvefiles/cve-2024-32615.h5 | error exit | read 4 obj, 1 errors | read 4 obj, 1 errors | ok |
| cvefiles/cve-2024-32616.h5 | error exit | read 10 obj, 7 errors | read 11 obj, 6 errors | mismatch |
| cvefiles/cve-2024-32616.h5 | error exit | read 10 obj, 7 errors | read 10 obj, 5 errors | ok |
| cvefiles/cve-2024-32617.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2024-32618.h5 | error exit | read 4 obj, 2 errors | read 3 obj | mismatch |
| cvefiles/cve-2024-32619.h5 | error exit | read 3 obj, 2 errors | read 3 obj | ok |
@@ -251,7 +248,7 @@ columns are.
| cvefiles/cve-2026-26200.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2026-34734.h5 | error exit | read 2 obj, 1 errors | read 2 obj | ok |
| cvefiles/cve-2026-92627.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/unknown-1.h5 | error exit | read 11 obj, 1 errors | read 11 obj, 5 errors | our-error |
| cvefiles/unknown-1.h5 | error exit | read 11 obj, 1 errors | read 11 obj, 1 errors | ok |
| fuzzerfiles/gh-4431-poc-03.h5 | error exit | read 1 obj | read 1 obj | ok |
| fuzzerfiles/gh-4432-poc-05.h5 | SIGSEGV | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| fuzzerfiles/gh-4433-poc-08.h5 | error exit | read 1 obj, 1 errors | read 1 obj | ok |
@@ -282,10 +279,11 @@ columns are.
## Objects h5py fails on but clawhdf5 reads
- 19 x `KeyError: '…'`
- 19 x `OSError: Can't synchronously read data (no appropriate function for conversion path)`
- 17 x `KeyError: '…'`
- 1 x `TypeError: unhandled dtype kind M (dtype('…'))`
- 1 x `OSError: Can't synchronously read data (bad coordinate offset)`
- 1 x `TypeError: No NumPy equivalent for TypeTimeID exists`
- 1 x `KeyError: "…"`
- 1 x `ValueError: Insufficient precision in available types to represent (N, N, N, N, N)`