quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

h5rs tools, browser reader, libhdf5 header checks, plugin filters, concurrency benchmark #14

Merged
osobh merged 60 commits from feat/p1-proof into main 2026-09-26 13:14:39 +00:00
Conversation 0 Commits 60 Files Changed 92
+44 -40
2 changed files with 44 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit 91644d8aaf - Show all commits
CONFORMANCE.md
+29 -31
View File
@@ -13,8 +13,8 @@ fatal. This file is generated by `conformance/run.sh`; do not edit it by hand.
| | |
|---|---|
| date | 2026-09-26 03:46 UTC |
| clawhdf5 commit | `10d1029ead524e2fe64c2cd7f61b28067d9e449c` |
| date | 2026-09-26 06:50 UTC |
| clawhdf5 commit | `72306c601399748616bc9d061be2ebc4c1bea9e0` |
| machine | `tank`: AMD Ryzen 7 7800X3D 8-Core Processor, 16 CPUs, 61 GiB, Linux 7.0.0-34-generic x86_64 |
| command | `conformance/run.sh --no-fetch --update-baseline` |
| rustc | rustc 1.98.1 (48a229cea 2026-09-01) |
@@ -38,14 +38,14 @@ A file's class is the first that applies:
| NCAS-CMS_pyfive | 33 | 32 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
| cve_hdf5 | 147 | 100 | 6 | 9 | 32 | 0 | 0 | 0 | 0 |
| h5py_data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| hdf5 | 466 | 386 | 8 | 12 | 60 | 0 | 0 | 0 | 0 |
| hdf5 | 466 | 392 | 4 | 10 | 60 | 0 | 0 | 0 | 0 |
| netcdf-c | 20 | 20 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| netcdf4-python | 18 | 18 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| usnistgov_h5wasm | 5 | 5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| xarray-data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| **all** | **697** | **569** | **14** | **22** | **92** | **0** | **0** | **0** | **0** |
| **all** | **697** | **575** | **10** | **20** | **92** | **0** | **0** | **0** | **0** |
2 of the 22 mismatches are a known h5py bug, not ours (see *Known not-our-bug*).
2 of the 20 mismatches are a known h5py bug, not ours (see *Known not-our-bug*).
Corpora (fetched by `conformance/fetch-corpus.sh` into the gitignored `conformance/.cache/`):
@@ -70,9 +70,9 @@ Grouped by normalised error message. *files* counts files whose class this cause
| files | objects | error | examples |
|---:|---:|---|---|
| 6 | 6 | `UnsupportedFilter(N)` | `hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_blosc.h5`, `hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_blosc2.h5`, `hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_bshuf.h5` (+3 more) |
| 3 | 3 | `DataSizeMismatch { expected: N, actual: N }` | `cve_hdf5/cvefiles/cve-2020-18494.h5`, `cve_hdf5/cvefiles/cve-2024-32623.h5`, `cve_hdf5/cvefiles/cve-2025-2309.h5` |
| 2 | 2 | `ChunkedReadError("…")` | `cve_hdf5/cvefiles/cve-2025-2308.h5`, `hdf5/test/testfiles/bad_nbit_parms_walk.h5` |
| 2 | 2 | `UnsupportedFilter(N)` | `hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_blosc2.h5`, `hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_zfp.h5` |
| 1 | 1 | `UnexpectedEof { expected: N, available: N }` | `cve_hdf5/cvefiles/cve-2019-9151.h5` |
| 1 | 1 | `MissingMessage(Dataspace)` | `cve_hdf5/cvefiles/cve-2024-33874.h5` |
| 1 | 1 | `InvalidObjectHeaderVersion(N)` | `hdf5/tools/test/testfiles/h5clear_mdc_image.h5` |
@@ -82,9 +82,9 @@ Grouped by normalised error message. *files* counts files whose class this cause
| files | objects | cause | examples |
|---:|---:|---|---|
| 13 | 14 | `missing-object` | `cve_hdf5/cvefiles/cve-2019-8397.h5`, `cve_hdf5/cvefiles/cve-2019-8398.h5`, `cve_hdf5/cvefiles/cve-2021-46243.h5` (+10 more) |
| 3 | 7 | `extra-attr` | `cve_hdf5/cvefiles/cve-2018-17438`, `cve_hdf5/cvefiles/cve-2018-17439`, `cve_hdf5/cvefiles/cve-2024-33874.h5` |
| 3 | 6 | `extra-object` | `cve_hdf5/cvefiles/cve-2021-46244.h5`, `hdf5/tools/test/testfiles/h5clear_fsm_persist_less.h5`, `hdf5/tools/test/testfiles/h5stat_err_refcount.h5` |
| 2 | 6 | `extra-attr` | `cve_hdf5/cvefiles/cve-2018-17438`, `cve_hdf5/cvefiles/cve-2018-17439` |
| 1 | 1 | `attr-values: ours=vlen(>u8) h5py=object layout=- filters=-` | `NCAS-CMS_pyfive/tests/data/attr_datatypes.hdf5` |
| 1 | 4 | `extra-object` | `cve_hdf5/cvefiles/cve-2021-46244.h5` |
| 1 | 1 | `values: ours=<f4 h5py=float32 layout=chunked filters=-` | `cve_hdf5/cvefiles/cve-2025-44904.h5` |
| 1 | 1 | `values: ours=>i2 h5py=>i2 layout=chunked filters=[6]` | `cve_hdf5/cvefiles/cve-2025-44905.h5` |
| 1 | 1 | `values: ours=>f4 h5py=>f4 layout=chunked filters=[2]` | `cve_hdf5/cvefiles/cve-2025-44905.h5` |
@@ -102,7 +102,7 @@ columns are.
| tool | read | error | panic | crash | hang | oom |
|---|---:|---:|---:|---:|---:|---:|
| clawhdf5 | 142 | 5 | 0 | 0 | 0 | 0 |
| clawhdf5 | 140 | 7 | 0 | 0 | 0 | 0 |
| h5dump 1.14.6 | 16 | 129 | 0 | 2 | 0 | 0 |
| h5py 3.16.0 / HDF5 2.0.0 | 115 | 31 | 0 | 1 | 0 | 0 |
@@ -112,19 +112,19 @@ columns are.
|---|---|---|---|---|
| cvefiles/cve-2016-4330.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2016-4331.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 1 errors | ok |
| cvefiles/cve-2016-4332-mtime-new.h5 | error exit | read 25 obj, 1 errors | read 25 obj | ok |
| cvefiles/cve-2016-4332-mtime.h5 | error exit | read 4 obj, 3 errors | read 4 obj | ok |
| cvefiles/cve-2016-4332-stab.h5 | error exit | open error | read 65 obj | h5py-cannot-read |
| cvefiles/cve-2016-4332-mtime-new.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 1 errors | ok |
| cvefiles/cve-2016-4332-mtime.h5 | error exit | read 4 obj, 3 errors | read 4 obj, 3 errors | ok |
| cvefiles/cve-2016-4332-stab.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
| cvefiles/cve-2016-4333.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2017-17505.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2017-17506.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2017-17507.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2017-17508.h5 | error exit | read 2 obj, 1 errors | read 2 obj | ok |
| cvefiles/cve-2017-17508.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2017-17509.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2018-11202.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2018-11203.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2018-11204.h5 | error exit | read 2 obj, 1 errors | read 2 obj | ok |
| cvefiles/cve-2018-11205.h5 | error exit | read 2 obj, 1 errors | read 2 obj | ok |
| cvefiles/cve-2018-11204.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2018-11205.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2018-11206-new.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-11206-old.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-11207.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
@@ -135,10 +135,10 @@ columns are.
| cvefiles/cve-2018-13870.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2018-13871.h5 | error exit | read 2 obj | read 2 obj | ok |
| cvefiles/cve-2018-13872.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2018-13873.h5 | error exit | read 1 obj, 1 errors | read 1 obj | ok |
| cvefiles/cve-2018-13874.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
| cvefiles/cve-2018-13873.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2018-13874.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2018-13875.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-13876.h5 | error exit | open error | read 2 obj, 1 errors | h5py-cannot-read |
| cvefiles/cve-2018-13876.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2018-14031.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-14033.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-14034.h5 | error exit | read 1 obj, 2 errors | read 1 obj | ok |
@@ -176,13 +176,13 @@ columns are.
| cvefiles/cve-2021-45833.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2021-46242.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
| cvefiles/cve-2021-46243.h5 | error exit | read 3 obj, 2 errors | read 2 obj, 1 errors | mismatch |
| cvefiles/cve-2021-46244.h5 | error exit | read 2 obj, 1 errors | read 6 obj, 3 errors | mismatch |
| cvefiles/cve-2021-46244.h5 | error exit | read 2 obj, 1 errors | read 6 obj, 4 errors | mismatch |
| cvefiles/cve-2024-29157.h5 | error exit | read 4 obj, 7 errors | read 4 obj, 7 errors | ok |
| cvefiles/cve-2024-29158.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2024-29159.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2024-29160.h5 | error exit | read 4 obj, 1 errors | read 4 obj, 1 errors | ok |
| cvefiles/cve-2024-29161.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 2 errors | ok |
| cvefiles/cve-2024-29162.h5 | error exit | read 17 obj, 4 errors | read 17 obj, 3 errors | ok |
| cvefiles/cve-2024-29161.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2024-29162.h5 | error exit | read 17 obj, 4 errors | read 17 obj, 4 errors | ok |
| cvefiles/cve-2024-29163.h5 | error exit | read 7 obj, 1 errors | read 7 obj, 1 errors | ok |
| cvefiles/cve-2024-29164.h5 | ok | read 3 obj | read 3 obj | ok |
| cvefiles/cve-2024-29165.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
@@ -197,19 +197,19 @@ columns are.
| cvefiles/cve-2024-32611.h5 | ok | read 6 obj | read 6 obj | ok |
| cvefiles/cve-2024-32612.h5 | ok | read 3 obj | read 3 obj | ok |
| cvefiles/cve-2024-32613.h5 | error exit | read 7 obj, 1 errors | read 7 obj, 1 errors | ok |
| cvefiles/cve-2024-32614.h5 | error exit | read 25 obj, 2 errors | read 25 obj, 1 errors | ok |
| cvefiles/cve-2024-32614.h5 | error exit | read 25 obj, 2 errors | read 25 obj, 2 errors | ok |
| cvefiles/cve-2024-32615.h5 | error exit | read 4 obj, 1 errors | read 4 obj, 1 errors | ok |
| cvefiles/cve-2024-32616.h5 | error exit | read 10 obj, 7 errors | read 10 obj, 5 errors | ok |
| cvefiles/cve-2024-32616.h5 | error exit | read 10 obj, 7 errors | read 10 obj, 6 errors | ok |
| cvefiles/cve-2024-32617.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2024-32618.h5 | error exit | read 4 obj, 2 errors | read 3 obj | mismatch |
| cvefiles/cve-2024-32619.h5 | error exit | read 3 obj, 2 errors | read 3 obj | ok |
| cvefiles/cve-2024-32618.h5 | error exit | read 4 obj, 2 errors | read 3 obj, 1 errors | mismatch |
| cvefiles/cve-2024-32619.h5 | error exit | read 3 obj, 2 errors | read 3 obj, 2 errors | ok |
| cvefiles/cve-2024-32620.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2024-32621.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2024-32622.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2024-32623.h5 | ok | read 6 obj | read 6 obj, 1 errors | our-error |
| cvefiles/cve-2024-32624.h5 | error exit | read 6 obj, 1 errors | read 6 obj | ok |
| cvefiles/cve-2024-33873.h5 | error exit | read 4 obj, 1 errors | read 4 obj | ok |
| cvefiles/cve-2024-33874.h5 | ok | read 6 obj, 1 errors | read 6 obj, 1 errors | our-error |
| cvefiles/cve-2024-33873.h5 | error exit | read 4 obj, 1 errors | read 4 obj, 1 errors | ok |
| cvefiles/cve-2024-33874.h5 | ok | read 6 obj, 1 errors | read 6 obj, 2 errors | our-error |
| cvefiles/cve-2024-33875.h5 | ok | read 2 obj | read 2 obj | ok |
| cvefiles/cve-2024-33876.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2024-33877.h5 | error exit | read 8 obj, 1 errors | read 8 obj, 1 errors | ok |
@@ -246,12 +246,12 @@ columns are.
| cvefiles/cve-2025-7068.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
| cvefiles/cve-2025-7069.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
| cvefiles/cve-2026-26200.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2026-34734.h5 | error exit | read 2 obj, 1 errors | read 2 obj | ok |
| cvefiles/cve-2026-34734.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2026-92627.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/unknown-1.h5 | error exit | read 11 obj, 1 errors | read 11 obj, 1 errors | ok |
| fuzzerfiles/gh-4431-poc-03.h5 | error exit | read 1 obj | read 1 obj | ok |
| fuzzerfiles/gh-4432-poc-05.h5 | SIGSEGV | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| fuzzerfiles/gh-4433-poc-08.h5 | error exit | read 1 obj, 1 errors | read 1 obj | ok |
| fuzzerfiles/gh-4433-poc-08.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| fuzzerfiles/gh-4434-poc-09.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
| fuzzerfiles/gh-4435-poc-10.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| fuzzerfiles/gh-4585.h5 | error exit | open error | open error | h5py-cannot-read |
@@ -279,10 +279,8 @@ columns are.
## Objects h5py fails on but clawhdf5 reads
- 19 x `KeyError: '…'`
- 19 x `OSError: Can't synchronously read data (no appropriate function for conversion path)`
- 1 x `TypeError: unhandled dtype kind M (dtype('…'))`
- 1 x `OSError: Can't synchronously read data (bad coordinate offset)`
- 1 x `TypeError: No NumPy equivalent for TypeTimeID exists`
- 1 x `KeyError: "…"`
- 1 x `ValueError: Insufficient precision in available types to represent (N, N, N, N, N)`
conformance/baseline.json
+15 -9
View File
@@ -1,15 +1,15 @@
{
"comment": "conformance/run.sh fails if the ok count drops below `ok` or a file in `ok_files` stops being ok. Regenerate with `conformance/run.sh --update-baseline` after an intended change.",
"commit": "10d1029ead524e2fe64c2cd7f61b28067d9e449c",
"date": "2026-09-26 03:46 UTC",
"commit": "72306c601399748616bc9d061be2ebc4c1bea9e0",
"date": "2026-09-26 06:50 UTC",
"reference": "h5py 3.16.0 / HDF5 2.0.0",
"files": 697,
"ok": 569,
"ok": 575,
"counts": {
"h5py-cannot-read": 92,
"mismatch": 22,
"ok": 569,
"our-error": 14
"mismatch": 20,
"ok": 575,
"our-error": 10
},
"per_corpus": {
"NCAS-CMS_pyfive": {
@@ -27,9 +27,9 @@
},
"hdf5": {
"h5py-cannot-read": 60,
"mismatch": 12,
"ok": 386,
"our-error": 8
"mismatch": 10,
"ok": 392,
"our-error": 4
},
"netcdf-c": {
"ok": 20
@@ -182,9 +182,13 @@
"h5py_data/vlen_string_dset_utc.h5",
"h5py_data/vlen_string_s390x.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_bitgroom.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_blosc.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_bshuf.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_bzip2.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_granularbr.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_jpeg.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_lz4.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_lzf.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_zstd.h5",
"hdf5/HDF5Examples/C/H5G/16/h5ex_g_iterate.h5",
"hdf5/HDF5Examples/C/H5G/16/h5ex_g_traverse.h5",
@@ -276,6 +280,7 @@
"hdf5/tools/test/testfiles/file_space.h5",
"hdf5/tools/test/testfiles/filter_fail.h5",
"hdf5/tools/test/testfiles/h5clear_fsm_persist_equal.h5",
"hdf5/tools/test/testfiles/h5clear_fsm_persist_less.h5",
"hdf5/tools/test/testfiles/h5clear_fsm_persist_noclose.h5",
"hdf5/tools/test/testfiles/h5clear_fsm_persist_user_equal.h5",
"hdf5/tools/test/testfiles/h5clear_fsm_persist_user_less.h5",
@@ -386,6 +391,7 @@
"hdf5/tools/test/testfiles/h5repack_uint8be_ex.h5",
"hdf5/tools/test/testfiles/h5stat_err_old_fill.h5",
"hdf5/tools/test/testfiles/h5stat_err_old_layout.h5",
"hdf5/tools/test/testfiles/h5stat_err_refcount.h5",
"hdf5/tools/test/testfiles/h5stat_filters.h5",
"hdf5/tools/test/testfiles/h5stat_idx.h5",
"hdf5/tools/test/testfiles/h5stat_newgrat.h5",