osobhandClaude Opus 5.5
3cf8cd86f2
Datatype::parse now makes the checks of libhdf5's H5O__dtype_decode_helper
and fails with InvalidDatatype (libhdf5's own error text) instead of
decoding a corrupt type:
- size 0 ("invalid datatype size"), for every class;
- integer bit offset/precision outside the type, or precision 0;
- float sign/exponent/mantissa outside the type, empty, or overlapping;
normalization 3; bit 6 without bit 0 from version 3;
- compound with no members, a member outside the compound, a duplicate
name, or a member overlapping an earlier one;
- enum whose size differs from its base type's, or an empty member name;
- array of more than 32 dimensions or with a zero-sized one (v1 compound
array members now say so rather than InvalidDatatypeVersion);
- opaque tag length that is not a multiple of 8.
Bit 6 of a version-1/2 float's class bits used to be read as VAX order,
byte-swapping values; libhdf5 ignores it before version 3, and so does
this now.
Only checks HDF5 2.0 (h5py 3.16) makes are added: newer libhdf5 also
checks bit fields, the variable-length kind and array sizes, but h5py
opens files that fail those, so they are left out. Each check was
confirmed against h5py by corrupting a file it wrote.
The conformance probe now decodes committed datatypes, as h5py's f[name]
does. Conformance (cached corpus, tank): 570 ok, unchanged. Objects
libhdf5 refuses that clawhdf5 used to read: cve-2016-4332-mtime (/cmpnd),
cve-2017-17508, cve-2024-32616 (/type1), cve-2024-32618, cve-2026-34734,
bad_compound.h5 (/cmpnd, /dataset); eight more that already failed now
fail with libhdf5's reason (e.g. cve-2024-29163 "mantissa range out of
bounds").
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Conformance sweep
Reads every HDF5 file of eight public corpora with clawhdf5 and with
h5py/libhdf5, compares the two readings object by object, and writes
CONFORMANCE.md.
CLAWHDF5_PYTHON=/path/to/venv/bin/python conformance/run.sh # ~30 s once the corpus is cached
conformance/run.sh --update-baseline # after an intended change in results
Needs Rust, git, h5dump (Debian/Ubuntu hdf5-tools), libaec (for the
probe's szip feature; libaec-dev), and a Python with the packages in
requirements.txt. The first run downloads about 450 MB of sparse checkouts.
| file | role |
|---|---|
corpus.txt |
the corpora: git URL, pinned commit, swept root, sparse-checkout patterns |
fetch-corpus.sh |
shallow, sparse, blob-filtered checkout of each pinned commit into .cache/src/ (gitignored); no-op when already there |
list_files.py |
which files are probed (HDF5/netCDF-4 extensions minus netCDF classic, plus the CVE reproducers) |
probe/ |
the clawhdf5 side: a standalone crate (outside the workspace, so cargo test --workspace never builds it) that walks a file with clawhdf5-format and prints canonical JSON |
ref.py |
the h5py side: the same JSON from h5py |
run_one.sh |
runs both sides on one file (and h5dump on the CVE corpus) under a timeout and an address-space limit |
compare.py |
classifies each file (ok / our-error / mismatch / h5py-cannot-read / panic / hang / crash / oom) and groups root causes |
report.py |
writes CONFORMANCE.md |
check.py |
the gate: fails on any panic/hang/crash/oom, on an ok count below baseline.json, or on a baseline-ok file that is no longer ok |
baseline.json |
the ok files the gate holds the line on |
requirements.txt |
pinned h5py / numpy / hdf5plugin / netCDF4 |
Results for every file (both sides' JSON and stderr, results.csv,
results.json, summary.md) are left in .cache/results/.
The nightly job is .gitea/workflows/conformance.yml; it prints the report
into the job log.
The canonical value encoding both sides hash is documented at the top of
probe/src/main.rs. Values are compared as libhdf5 presents them: a float
with a non-IEEE bit layout (N-Bit) or an integer with a bit offset is compared
as the converted number, not as raw file bytes.