libhdf5 fails to read a VL element whose global heap address is
undefined (all 0xff), even at length 0 ("addr undefined"); we returned
"" (or an empty sequence) in every reader. Checked with h5py first:
libhdf5 writes a null element with address 0, which still reads as
empty, and h5py writes "" as a zero-size heap object at a real address,
so no file they write relies on the old behaviour. read_vl_bytes now
treats address 0 as null whatever the length, as VlResolver does.
Tests, each failing before: vl_data unit test (8- and 4-byte offsets,
lengths 0 and 1); clawhdf5 vl_data_interop
a_vl_element_at_the_undefined_heap_address_fails_like_h5py (also checks
where h5py writes ""); h5rs dump --json and check --data on the patched
`undef` dataset; clawhdf5-wasm vl_strings.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5rs (dump, ls, diff, check --data) kept its own lenient VL decoder:
a heap object longer than its element was cut to the element's length
(libhdf5 and h5py refuse it), a null string printed "" where h5dump
prints NULL, the stored element size was trusted, and every heap
collection was kept as an owned copy for the whole run. It now resolves
each element with VlResolver::element / string_element (new: one element
in place, borrowing from the file), and refuses a VL type whose stored
element size is not 4 + offset size + 4, as File does. H5::heap_object
and its cache are gone. h5diff compares a null VL string equal to an
empty one; so does h5rs diff.
clawhdf5-wasm already resolved VL strings with read_vl_strings; it now
uses VlResolver and checks the stored element size before reading, as
File::read_string does.
Tests (h5py writes the files, patched for "a\0b", a null element and
mis-sized heap objects, with 8- and 4-byte offsets):
- h5rs_interop dump_prints_vl_data_like_h5dump: byte-identical to h5dump;
- dump_json_vl_values_match_h5py: h5py's values, errors where h5py fails;
- check_data_flags_mis_sized_vl_heap_objects;
- clawhdf5-wasm tests/vl_strings.rs: wasm, File and h5py agree.
All four fail before. check --data over the 150 cve_hdf5 CVE and fuzzer
files now passes 15 (h5dump rejects 8 of them), was 16 and 9: the
stored-size check flags cve-2024-32608. h5rs-check-ok-files.sh --data:
0 of 422 flagged; h5rs-fuzz.sh: clean on 180 files.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Dense link and attribute storage keeps its messages in a fractal heap. Its
root indirect block holds direct blocks up to 64 KiB, 512 KiB in all; rows
past that are child indirect blocks. The writer kept adding rows of direct
blocks instead, and libhdf5 and h5rs read them as indirect blocks: a group
with 20 000 links of 20-byte names was written without error and could not
be listed ("incorrect metadata checksum"), and 150 dense attributes of up
to 56 KB could not be opened. The heap writer now follows the doubling
table: rows past the direct ones hold child indirect blocks, each with its
own rows, nested as deep as the heap needs.
Two more heap bugs are fixed on the way. An object bigger than the next
block's free space was written into it anyway and cut off; the block is
now left unallocated and the object goes in the first block big enough, as
libhdf5 skips blocks. And the header's next-block offset was 0, so libhdf5
adding a link to such a group overwrote the heap's first block ("bad
version number for message"); it is now the offset after the last block.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FileWriter wrote the root group plus one level of groups, and refused
path-like names. The writer now flattens its builders into a group tree
(writer_tree.rs) before layout:
- A name may be a path ("a/b/x", "/a/b/x" at the root); missing
intermediate groups are created as h5py does, and GroupBuilder gains
create_group/add_group so builders nest to any depth. A group added at a
path that already holds a group is merged into it (require_group);
any other repeated name, an empty or "." component, or an absolute path
below the root is an error.
- add_soft_link, add_hard_link and add_external_link on FileWriter,
FileBuilder and GroupBuilder. Hard-link targets are resolved to objects
at finish (through other hard links; a missing target, a soft link on the
way or a cycle of paths is an error). Objects with several hard links get
an Object Reference Count message so libhdf5 can delete one link without
freeing the object.
- track_order(true) per group, or as the file default, tracks and indexes
link creation order: Link Info flags and max order, the order in each
Link message, and a type-6 creation-order B-tree for dense groups.
- A group's link index is one B-tree leaf; more than 65535 links is an
error.
Groups are laid out depth-first from the root, datasets group by group,
and untracked groups keep writing datasets, then groups, then other links:
files with one level of groups are byte-identical to before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With the header checks merged, the library refuses truncated files and
misaligned chunk-index keys itself, so check reported only "file cannot
be opened" for a truncated file. It now reports the truncation (stored
end of file vs file length) or the library's error, and the misaligned
chunk test accepts the library's refusal of the key.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A null-padded fixed string inside a compound or an array member printed
trimmed ("" for three NULs, "a" for "a\0b"), where h5dump prints every
byte ("\000\000\000", "a\000b"); only top-level strings were shown in
full. DATA blocks now render elements through one function that keeps
the padding at any depth.
The README now lists the remaining known differences from h5dump:
nested compounds print inline, and long double values are printed as
errors (exit 1) with the datatype as an H5T_FLOAT block.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The README said check skips only "global heap collections other than
those a value read touches", but read_dataset returns the raw heap IDs,
so no collection was ever read: a file whose global heap collection
claims a 4 GiB object passed `check --data` with no problems, while
h5dump (and h5rs dump/diff) fail on it.
With --data, every variable-length element (strings and sequences, also
inside compounds, arrays and nested sequences) of every dataset and
attribute is followed into its collection. A collection that does not
parse, a missing heap object, or a sequence longer than its heap object
is a problem at the collection's address, once per object; the summary
counts the collections read.
Measured on tank, 2026-09-26: the 418 fully-read conformance ok files
still pass (scripts/h5rs-check-ok-files.sh --data, 0 flagged), and
`check --data` now flags 152 of the 180 CVE-corpus files (was 147); of
the 28 it passes, h5dump 1.14.6 rejects 21 (was 26 of 33).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
-c meant "list at most N differences" in h5rs, but in h5diff -c is
--compare (a flag) and the count is -n/--count=N, so a script moved over
from h5diff behaved differently: `h5diff -r -c 2 A B` exits 2 (the 2 is
taken as a file name) while h5rs exited 1.
The count is now -n/--count, -c/--compare is accepted (h5rs always lists
objects that are not comparable), and the --count=N, --delta=D,
--relative=R forms are accepted; exit codes equal h5diff's on 7 cases.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
An OBJ that was a soft link was resolved and its target object compared,
so two files whose /g/s both point at /z differed when /z did: exit 1,
where h5diff (without --follow-symlinks) compares the links' target paths
and exits 0.
A soft link is now compared as a link wherever it is, OBJ included.
--follow-symlinks compares the objects soft links lead to instead, walks
into soft-linked groups, resolves relative targets against the link's
group, and treats two dangling links as the same; exit codes equal
h5diff's on 14 cases. External links are never followed (documented).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With a tolerance, integers were converted to f64 before comparing, so
int64/uint64 values above 2^53 that differ compared equal: -d 0 on 2^60
and 2^60 + 1 exited 0, where h5diff exits 1. Integer pairs are now
compared in i128 (the delta against floor(D), the relative quotient from
an exact difference), and the report prints the exact difference.
h5diff compares exactly when -p is below the f64 epsilon (2^60 and
2^60 + 1 differ at -p 1e-18, and nextafter(2, 0) and 2 at -p 1.5e-16);
h5rs now does the same.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The path walk skipped the second hard link to an object, so a file that
shares one dataset between /x and /y differed from a file holding two
identical copies: "</y> exists only in <B>", exit 1, where h5diff exits 0.
For a hard-linked group every member was reported the same way.
diff now enumerates every path below the start object (a hard link back
to an ancestor is recorded but not descended into), so each name is
compared. A group whose links cannot be read is now an error instead of
an empty group.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
New workspace crate clawhdf5-tools with one binary, h5rs, built only on the
clawhdf5 facade and clawhdf5-format (no libhdf5, no C):
- ls [-r] [-v] FILE[/path]: h5ls's listing (same text in its first two
columns) plus the datatype; -v adds address, link count, layout and chunk
index, chunk size, storage, filters, datatype and attributes.
- dump [--json] [-A] [-p] [-d PATH] FILE: h5dump DDL (byte-identical to
h5dump 1.14.6 on the test files) or hdf5-json.
- stat FILE: h5stat's object/link/rank/layout/filter/attribute counts, raw
data and total size.
- diff [-r] [-q] [-d D] [-p R] A B [OBJ1 [OBJ2]]: structural and value
differences, exit 0/1/2 like h5diff.
- check [--data] FILE: walks every object, parses every message, verifies
the checksums of every v2+ structure (including the fractal heap blocks
the library never checks), checks chunk indexes against their datasets
and raw data for out-of-file or overlapping extents; every problem with
its address.
Values over --max-bytes are reported, not read; dense-storage heaps are
verified before objects are read from them; panics are caught (exit 3).
Tests compare with h5ls, h5stat, h5dump and h5diff and with h5py's values,
and flip the checksum of every checksummed structure in a v1.14-format file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
osobh