Regenerated on tank: ok 575 -> 599, our-error 10 -> 4, mismatch 20 -> 2,
no panics, hangs, crashes or OOM. Newly ok: metadata-cache-image files,
the CVE open-time checks, v1 chunk-key lookup as libhdf5 does it, the
scale-offset fixes, Blosc2, and harness corrections. Baseline raised.
h5rs check --data flags none of the 435 fully-read ok files.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
For a metadata cache image libhdf5 cannot load, libhdf5 opens the file
and fails the first metadata read (the image loads on the first
H5C_protect after open); h5py reports the error on the root group. The
conformance probe reported it that way, but File::open refused the file,
so the gate counted cve-2025-6269-1..4 and cve-2025-6516 as agreeing
with h5py for behaviour the library did not have.
The library now behaves as the probe reports: File (mmap, buffered and
from_bytes) and MmapFile open the file and every object lookup (dataset,
dataset_at, group, group listings and attributes, VL decoding) fails with
the image's error; LazyFile reads the root group's header at open, so
its open is that first read and fails. Probe and library take the
three-way decision (refuse at open / image loads / image cannot load)
from the same clawhdf5_format::superblock_ext::cache_image_state.
One deliberate difference from libhdf5 remains, documented: after the
failed first read libhdf5 reads the file's own metadata, which the image
was meant to replace and may be stale; here every lookup keeps failing.
File::cache_image_error / MmapFile::cache_image_error expose the error
to code that parses as_bytes() itself; h5rs checks it before reading any
object header (h5rs ls on cve-2025-6269-1 said "invalid object header
version: 0" from the stale bytes).
Test: metadata_cache_image.rs an_image_libhdf5_cannot_load_fails_every_object
(the fixture with its image signature broken; h5py opens that file and
fails the first read with "Bad metadata cache image header signature").
It fails on the previous commit, where File::open refuses the file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
apply_cache_image returned a copy of the whole file with the image's
entries written in, and File (mmap by default), MmapFile and LazyFile
used that copy for every read: opening a 1 GiB sparse file with an image
needed 2 GB of memory, and an 8 GiB one aborted the process, where
de2a53f (which ignored the image) opened them in a few MB.
The metadata parsers read one contiguous slice, so the image still has
to be laid over the file's bytes; it is now laid over a private copy
that costs only the pages it touches:
- clawhdf5_format::superblock_ext::CacheImage decodes the image into an
entry list (address, offset in the block, length) and applies it to
any destination; cache_image_state tells an opener whether the file
has no image, a loadable one, or one libhdf5 cannot load;
apply_cache_image_in_place is for readers that own their buffer.
apply_cache_image and metadata_view (which copied) are gone.
- clawhdf5_io::HDF5Read::private_copy returns a writable private copy
of a reader's bytes: MmapReader gives a MAP_PRIVATE copy-on-write
mapping (memmap2 map_copy), so only the pages the entries land on are
copied; the default copies the bytes (in-memory readers).
- File, MmapFile and LazyFile write the image into that mapping
(crate::cache_image). File::from_bytes / open_buffered patch their own
buffer in place, copying only the image block, as libhdf5 does. A
file without an image is read straight from the mapping, unchanged.
An image entry that runs past the end of file is now refused: libhdf5
checks only that it starts inside the file, and the images libhdf5
writes never do this, but those bytes have nowhere to go in a view of
the file.
Tests: tests/cache_image_memory.rs has libhdf5 (through ctypes) add an
image to a 1 GiB sparse file and bounds resident-memory growth for all
three openers at 256 MiB; it fails on the previous commit (File::open
grew 2,148,720,640 bytes). reader.rs zero_copy_tests check that a file
without an image is read from the mapping itself and that an image goes
into a copy-on-write mapping, not a heap copy; clawhdf5-io checks that
private_copy writes never reach the reader or the file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The review suggested libhdf5 loads every image entry and resolves
flush-dependency parents afterwards. It does not:
H5C__reconstruct_cache_contents (HDF5 1.14.6 and 2.0.0, and develop)
inserts each entry and then searches the cache index for its parents in
the same loop, failing with "fd parent not in cache?!?" when one is
missing. So a parent must be an earlier image entry, as before, or
metadata cached before the image loads: the superblock (address 0) and
the superblock extension's object header, which libhdf5 reads to find
the image. Those two were refused as parents; they are now accepted.
A parent listed after its child is still refused, as libhdf5 refuses
it, and so is an entry that is its own parent ("Child entry flush
dependency parent can't be itself").
apply_cache_image takes the superblock to know the extension address.
Test: superblock_ext::tests::flush_dependency_parents_must_already_be_cached
(parent-first loads, child-first refused, extension header accepted,
self-parent refused); the extension-header case fails without the fix.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 does not walk the chunk B-tree to read a dataset: it looks each
chunk up (H5B_find with H5D__btree_cmp3 and H5D__btree_found), asking for
the element-size coordinate as 0. collect_chunk_info_checked now parses
the tree with its keys and returns each stored chunk only when that
lookup, replayed over the scaled keys, finds it.
A key with a non-zero element-size coordinate is therefore found in a
1-D dataset (cmp3 compares only the first coordinate there, and found
compares with <=) and missed in a dataset of rank 2 or more, which reads
fill values. The previous commit refused every such key, which refused
1-D files libhdf5 reads correctly; before that, the rank-2 case read the
chunk's data where h5py reads fill values (cve-2025-44905
/Shuffle_float_data_le, now identical to h5py, so it leaves the
conformance report's list of libhdf5 bugs).
Test: chunk_keys_with_an_element_offset_read_as_libhdf5_reads_them
compares 1-D and 2-D files against h5py's values. It fails on the
previous commit (the 1-D file is refused) and with the refusal removed
(the 2-D file reads 0..23 where h5py reads fill values).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Four of the remaining our-errors are objects the reference (h5py 3.16 /
HDF5 2.0) reads only through a libhdf5 bug, and clawhdf5 refuses:
cve-2025-2308 (scale-offset codes past the end of the chunk),
cve-2025-44904 (short unfiltered chunks), bad_nbit_parms_walk.h5 (an N-Bit
parameter list one value short; libhdf5's own test_filter_bad_params now
requires the read to fail) and cve-2025-44905 /Shuffle_float_data_le (a
chunk key libhdf5's lookup misses, reading fill values). report.py lists
them under Known not-our-bug and counts them in the summary; they stay
our-errors in the class counts.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 loads a metadata cache image when it first reads metadata (the
root group), not at open, so for cve-2025-6269-1..4 and cve-2025-6516 (all
corrupt images) h5py opens the file and fails on "/". The probe reported
the image's error as an open error, which made those files our-errors;
it now records it on the root object, where h5py reports it. File::open
still refuses such a file outright: nothing in it can be read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 decodes the messages of a v2/v3 superblock's extension when it
opens a file (H5F__super_read) and refuses the file when one does not
decode. We never looked at them, so we opened cve-2020-10810 (a File
Space Info message too short for the free-space manager addresses it
announces) and cve-2020-10812 (a metadata cache image past the end of the
file), both of which libhdf5 refuses.
A file written with a metadata cache image keeps its metadata cache
entries in an image block the extension points at; libhdf5 loads them
over the file's own bytes before it reads any metadata
(H5C__load_cache_image, H5C__reconstruct_cache_contents). In
h5clear_mdc_image.h5 the root group's header exists only in the image, so
every reader failed with InvalidObjectHeaderVersion(0).
The new clawhdf5_format::superblock_ext module:
- read_superblock_extension decodes the v1 B-tree K, File Space Info and
Metadata Cache Image messages with libhdf5's checks (versions, page size
512 B .. 1 GiB, the addresses a persisting message lists, the image
inside the file), with the new FormatError::InvalidSuperblockExtension;
- apply_cache_image checks an image block as libhdf5 does (signature,
version, recorded length, entry types, rings, ages, addresses inside
the file and not repeated, flush-dependency parents) and returns the
file's bytes with every entry written at its address
(FormatError::InvalidCacheImage);
- metadata_view does both.
File, MmapFile and LazyFile (and so h5rs) call metadata_view at open and
read an image file through the patched copy; the conformance probe does
the same. The image's trailing checksum is not verified, as libhdf5 does
not verify it. tests/fixtures/h5clear_mdc_image.h5 is libhdf5's own test
file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 checks a dataset's storage when it opens the dataset
(H5D__contig_check, H5D__compact_init): the element count times the
element size must not overflow, contiguous storage must end inside the
file, compact data must be the dataset's size. File::dataset opened
cve-2024-32624's /Dset_OBJREF (2^62 + 2 references of 8 bytes) and
reported its shape; only reading failed.
data_read::check_dataset_storage makes those checks (new
FormatError::InvalidDatasetStorage), and File, MmapFile and LazyFile run
it whenever they open a dataset (by path, by address, from a group), as
does the conformance probe. As before, a datatype, dataspace or layout
that does not decode is left for the read to report, so such a dataset
still opens and its attributes still read. An empty contiguous dataset at
a defined address, which libhdf5 refuses, is still accepted: clawhdf5 up
to v2.7.0 wrote them.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 decides what an object header is in a fixed order
(H5O__obj_class_real): a group if it has a Symbol Table or Link Info
message, a dataset if it has a Datatype *and* a Dataspace message, a named
datatype if it has a Datatype message. The conformance probe called any
header with a Data Layout message a dataset, so cve-2024-33874's /Dset1 (a
datatype and a layout, no dataspace), which h5py opens as a named
datatype, was reported as a dataset we failed to read
(MissingMessage(Dataspace)). The probe now classifies with
object_class().
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ref.py and compare.py reported 13 files as mismatches or our-errors that
were artefacts of the harness, not differences between the readers:
- User-defined links (tall.h5, tudlink.h5, twithub*.h5, tmany.h5, ...):
h5py's `get(name, getlink=True)` reports a user-defined link as a
HardLink, so ref.py listed it as an object. Read the link type from
H5Lget_info instead.
- Objects h5py cannot open (cve-2019-8397/8398, cve-2021-46243,
cve-2024-32618): the probe deduplicates by header address, ref.py by
ObjectID, which an unopenable object does not have, so each extra hard
link to it was listed again. Deduplicate those by link address.
- Nested array types (tarray3.h5): h5py expands them into trailing dims;
hash_values stripped one level and numpy broadcast every element into a
whole subarray. Strip every level.
compare.py no longer compares the attributes or links of an object h5py
could not open at all (cve-2018-17438/17439, cve-2019-9151): h5py read
none, so ours are neither extra nor errors against it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5-format now depends on libc on Linux (huge-page advice for read
buffers); the probe's committed lockfile picks that up.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Checked with h5py on a patched file:
- a VL string with an embedded NUL reads up to the NUL (libhdf5 converts
VL strings to C strings); read_vl_strings returned "a\0b";
- an element whose global heap object is not length x base size bytes is
an error ("Expected global heap object size does not match"); we
returned the object cut to the length;
- a heap address of 0 is a null element whatever its length.
vl_data::VlResolver does this, caching each parsed heap collection:
read_vl_strings parsed the whole collection again for every element.
read_vl_strings and read_vl_bytes use it; check_element_size refuses a VL
type whose stored element size is not 4 + offset size + 4. The
conformance probe resolves VL values through VlResolver instead of its
own lenient copy (575 of 697, unchanged).
The new unit tests fail against the old read_vl_strings.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The superblock records where the file's data ends. libhdf5 refuses to
open a file shorter than that ("truncated file", H5F__super_read) and
fails any read past it ("addr overflow" / "address plus size exceeds
file eoa"). clawhdf5 read whatever was left of a truncated file, and read
bytes after the recorded end as if they belonged to the file.
New Superblock::data_end: FormatError::TruncatedFile for a file shorter
than its recorded end, otherwise where the HDF5 data ends. As in libhdf5,
the recorded end moves with the superblock when its recorded base address
is not where it is (a user block added afterwards; cve-2021-36977, which
h5py reads, depends on it), and the check is skipped for a v3 superblock
still being written in SWMR mode. File, LazyFile, MmapFile and the
conformance probe refuse a truncated file and parse only up to the end.
Files clawhdf5 writes record their true length, and the v2.5.0 agent-store
fixture and files written by v2.7.0 (plain, paged, user-block free) pass
the check.
Interop test: h5py writes a file; a copy missing its last 8 bytes must be
refused by both, a copy with bytes appended and one moved behind a new
512-byte user block must read in both.
Conformance (cached corpus, tank): 570 -> 571 ok
(h5clear_fsm_persist_less.h5, whose data past the recorded end was being
read); ten files h5py refuses as truncated (cve-2018-13874,
cve-2018-13876, the family/multi/subfiling members, h5clear_fsm_persist_
greater/user_greater) are now refused at open instead of read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
scripts/ci-test.sh: the format feature matrix (clippy and tests) adds
plugin-filters; bitshuffle, bzip2 and blosc are each linted alone (blosc
and bitshuffle share code); the facade is linted with plugin-filters; and
the interop section runs tests/plugin_filters_interop.rs with it, against
h5py + hdf5plugin (CI's venv already installs hdf5plugin).
conformance/probe enables plugin-filters. Sweep (tank, 2026-09-26,
conformance/run.sh --no-fetch against the cached corpus): 573 of 697 ok
(baseline 569), no regressions; newly ok: h5ex_d_blosc.h5,
h5ex_d_bshuf.h5, h5ex_d_bzip2.h5, h5ex_d_lzf.h5. h5ex_d_blosc2.h5 and
h5ex_d_zfp.h5 remain UnsupportedFilter.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 1.14.4+ treats an integer, float or bit field wider than a byte
whose precision and offset leave more than half its bits unused as
corruption when the type sits in a header without a checksum (version 1),
unless the file is opened with H5Pset_relax_file_integrity_checks
(H5T_is_numeric_with_unusual_unused_bits). clawhdf5 read such types,
e.g. a 3-bit integer in 4 bytes (cve-2024-29162) or a 32-bit float in
65525 bytes (cve-2024-32614, tmisc38a.h5).
New Datatype::check_unused_bits (recursive) and Datatype::parse_in_header,
which applies it for version-1 headers. Dataset datatypes (facade File,
LazyFile, MmapFile; clawhdf5-io VOL, MPI VOL, async reader; the
conformance probe) and compact attributes in version-1 headers use it.
Conformance (cached corpus, tank): 570 ok, unchanged; cve-2024-29162,
cve-2024-32614 and tmisc38a.h5 now refuse the object h5py refuses, and
tmisc38b.h5 / unknown-1.h5 now fail with libhdf5's reason.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Datatype::parse now makes the checks of libhdf5's H5O__dtype_decode_helper
and fails with InvalidDatatype (libhdf5's own error text) instead of
decoding a corrupt type:
- size 0 ("invalid datatype size"), for every class;
- integer bit offset/precision outside the type, or precision 0;
- float sign/exponent/mantissa outside the type, empty, or overlapping;
normalization 3; bit 6 without bit 0 from version 3;
- compound with no members, a member outside the compound, a duplicate
name, or a member overlapping an earlier one;
- enum whose size differs from its base type's, or an empty member name;
- array of more than 32 dimensions or with a zero-sized one (v1 compound
array members now say so rather than InvalidDatatypeVersion);
- opaque tag length that is not a multiple of 8.
Bit 6 of a version-1/2 float's class bits used to be read as VAX order,
byte-swapping values; libhdf5 ignores it before version 3, and so does
this now.
Only checks HDF5 2.0 (h5py 3.16) makes are added: newer libhdf5 also
checks bit fields, the variable-length kind and array sizes, but h5py
opens files that fail those, so they are left out. Each check was
confirmed against h5py by corrupting a file it wrote.
The conformance probe now decodes committed datatypes, as h5py's f[name]
does. Conformance (cached corpus, tank): 570 ok, unchanged. Objects
libhdf5 refuses that clawhdf5 used to read: cve-2016-4332-mtime (/cmpnd),
cve-2017-17508, cve-2024-32616 (/type1), cve-2024-32618, cve-2026-34734,
bad_compound.h5 (/cmpnd, /dataset); eight more that already failed now
fail with libhdf5's reason (e.g. cve-2024-29163 "mantissa range out of
bounds").
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Regenerated on tank at 10d1029: ok 467 -> 569, our-error 123 -> 14,
mismatch 15 -> 22 (six user-defined-link files moved from our-error to a
listing difference), no panics, hangs, crashes or OOM. Baseline raised.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Superblock::parse now refuses a user-block offset, and the raw read path
no longer guesses a VDS fill value. The probe looks at the file from the
superblock on and reads virtual datasets with vds::read_virtual_dataset,
the dataset's fill value and its source-derived extent.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
conformance/run.sh fetches eight public HDF5 corpora pinned by commit
(conformance/corpus.txt) into a gitignored cache, reads every file with
clawhdf5 (conformance/probe, a crate outside the workspace) and with
h5py/libhdf5 (ref.py), and the CVE files with h5dump, each under a timeout
and an address-space limit; compare.py classifies the files, report.py
writes CONFORMANCE.md and check.py gates on panics/hangs/crashes/OOM and on
regressions against conformance/baseline.json. ~25 s once cached.
Changes from the ad-hoc audit harness:
- the probe compares non-IEEE-layout floats (N-Bit) and integers with a bit
offset or reduced precision as the values libhdf5 converts them to, not
raw file bytes: 8 files that showed as mismatches now read identically;
- ref.py exits without tearing down h5py objects: libhdf5 2.0 aborts while
freeing them for two files about half the time, which flipped them
between ok and h5py-cannot-read from run to run;
- the file list is defined (list_files.py): netCDF classic files are left
out, 11 HDF5 files the ad-hoc sweep missed are in.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
