quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(tools): h5rs check --data follows VL data into the global heap

Browse Source 
The README said check skips only "global heap collections other than
those a value read touches", but read_dataset returns the raw heap IDs,
so no collection was ever read: a file whose global heap collection
claims a 4 GiB object passed `check --data` with no problems, while
h5dump (and h5rs dump/diff) fail on it.

With --data, every variable-length element (strings and sequences, also
inside compounds, arrays and nested sequences) of every dataset and
attribute is followed into its collection. A collection that does not
parse, a missing heap object, or a sequence longer than its heap object
is a problem at the collection's address, once per object; the summary
counts the collections read.

Measured on tank, 2026-09-26: the 418 fully-read conformance ok files
still pass (scripts/h5rs-check-ok-files.sh --data, 0 flagged), and
`check --data` now flags 152 of the 180 CVE-corpus files (was 147); of
the 28 it passes, h5dump 1.14.6 rejects 21 (was 26 of 33).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 01:22:27 -05:00
co-authored by Claude Opus 5.5
parent 386bd1d41e
commit b8492bd28d
5 changed files with 228 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/clawhdf5-tools/README.md
+9 -4
View File
@@ -216,7 +216,12 @@ extension) and checks:
no two pieces overlap.
`--data` also reads every dataset, decoding every chunk through its filters
(which catches corrupt compressed data and Fletcher-32 mismatches). Data the
(which catches corrupt compressed data and Fletcher-32 mismatches), and
follows every variable-length element (strings and sequences, also inside
compounds and arrays) of every dataset and attribute into its global heap
collection: a collection that does not parse, a missing heap object, or a
sequence longer than its heap object is a problem at the collection's
address. Data the
tool cannot decode (a filter it does not implement, such as szip, or a
dataset over `--max-bytes`) is a `note:`, not a problem. Every problem is
printed with the address of the structure involved; the exit status is 0
@@ -228,11 +233,11 @@ also covers the structures HDF5 1.10+ writes (fixed/extensible array and v2
B-tree chunk indexes, and version 3 superblocks).
What it does not check: free-space manager and shared-message (SOHM) table
checksums, global heap collections other than those a value read touches,
and objects reachable only by external links. It validates with
checksums, global heap collections no variable-length value points into (and
none at all without `--data`), and objects reachable only by external links. It validates with
clawhdf5's parsers, so it accepts what they accept: some header damage that
libhdf5 refuses goes unreported (of the 180 files of the HDF Group's CVE
corpus, `check --data` passes 33, and h5dump 1.14.6 rejects 26 of those;
corpus, `check --data` passes 28, and h5dump 1.14.6 rejects 21 of those;
tank, 2026-09-26).
## Robustness