libhdf5 1.14.4+ treats an integer, float or bit field wider than a byte
whose precision and offset leave more than half its bits unused as
corruption when the type sits in a header without a checksum (version 1),
unless the file is opened with H5Pset_relax_file_integrity_checks
(H5T_is_numeric_with_unusual_unused_bits). clawhdf5 read such types,
e.g. a 3-bit integer in 4 bytes (cve-2024-29162) or a 32-bit float in
65525 bytes (cve-2024-32614, tmisc38a.h5).
New Datatype::check_unused_bits (recursive) and Datatype::parse_in_header,
which applies it for version-1 headers. Dataset datatypes (facade File,
LazyFile, MmapFile; clawhdf5-io VOL, MPI VOL, async reader; the
conformance probe) and compact attributes in version-1 headers use it.
Conformance (cached corpus, tank): 570 ok, unchanged; cve-2024-29162,
cve-2024-32614 and tmisc38a.h5 now refuse the object h5py refuses, and
tmisc38b.h5 / unknown-1.h5 now fail with libhdf5's reason.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Datatype::parse now makes the checks of libhdf5's H5O__dtype_decode_helper
and fails with InvalidDatatype (libhdf5's own error text) instead of
decoding a corrupt type:
- size 0 ("invalid datatype size"), for every class;
- integer bit offset/precision outside the type, or precision 0;
- float sign/exponent/mantissa outside the type, empty, or overlapping;
normalization 3; bit 6 without bit 0 from version 3;
- compound with no members, a member outside the compound, a duplicate
name, or a member overlapping an earlier one;
- enum whose size differs from its base type's, or an empty member name;
- array of more than 32 dimensions or with a zero-sized one (v1 compound
array members now say so rather than InvalidDatatypeVersion);
- opaque tag length that is not a multiple of 8.
Bit 6 of a version-1/2 float's class bits used to be read as VAX order,
byte-swapping values; libhdf5 ignores it before version 3, and so does
this now.
Only checks HDF5 2.0 (h5py 3.16) makes are added: newer libhdf5 also
checks bit fields, the variable-length kind and array sizes, but h5py
opens files that fail those, so they are left out. Each check was
confirmed against h5py by corrupting a file it wrote.
The conformance probe now decodes committed datatypes, as h5py's f[name]
does. Conformance (cached corpus, tank): 570 ok, unchanged. Objects
libhdf5 refuses that clawhdf5 used to read: cve-2016-4332-mtime (/cmpnd),
cve-2017-17508, cve-2024-32616 (/type1), cve-2024-32618, cve-2026-34734,
bad_compound.h5 (/cmpnd, /dataset); eight more that already failed now
fail with libhdf5's reason (e.g. cve-2024-29163 "mantissa range out of
bounds").
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Regenerated on tank at 10d1029: ok 467 -> 569, our-error 123 -> 14,
mismatch 15 -> 22 (six user-defined-link files moved from our-error to a
listing difference), no panics, hangs, crashes or OOM. Baseline raised.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Superblock::parse now refuses a user-block offset, and the raw read path
no longer guesses a VDS fill value. The probe looks at the file from the
superblock on and reads virtual datasets with vds::read_virtual_dataset,
the dataset's fill value and its source-derived extent.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
conformance/run.sh fetches eight public HDF5 corpora pinned by commit
(conformance/corpus.txt) into a gitignored cache, reads every file with
clawhdf5 (conformance/probe, a crate outside the workspace) and with
h5py/libhdf5 (ref.py), and the CVE files with h5dump, each under a timeout
and an address-space limit; compare.py classifies the files, report.py
writes CONFORMANCE.md and check.py gates on panics/hangs/crashes/OOM and on
regressions against conformance/baseline.json. ~25 s once cached.
Changes from the ad-hoc audit harness:
- the probe compares non-IEEE-layout floats (N-Bit) and integers with a bit
offset or reduced precision as the values libhdf5 converts them to, not
raw file bytes: 8 files that showed as mismatches now read identically;
- ref.py exits without tearing down h5py objects: libhdf5 2.0 aborts while
freeing them for two files about half the time, which flipped them
between ok and h5py-cannot-read from run to run;
- the file list is defined (list_files.py): netCDF classic files are left
out, 11 HDF5 files the ad-hoc sweep missed are in.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
