Merge branch 'feat/p1-h5-tools' into feat/p1-proof

# Conflicts:
#	docs/known-issues.md
This commit is contained in:
osobh
2026-09-26 01:38:06 -05:00
25 changed files with 7561 additions and 5 deletions
+59
View File
@@ -163,6 +163,65 @@
`quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
out. The CLI's `--quantized-index` is still accepted but is now a no-op.
### Tools
- New crate **`clawhdf5-tools`** with the binary **`h5rs`**: HDF5
command-line tools without libhdf5, built only on the `clawhdf5` facade
and `clawhdf5-format` (no C, so it also builds as a static musl binary).
- `h5rs ls [-r] [-v] FILE[/path]` lists objects like h5ls (its first two
columns are h5ls's text on the test files) plus the datatype; `-v` adds
address, link count, layout and chunk index, chunk size, storage,
filters, datatype and attributes.
- `h5rs dump [--json] [-A] [-p] [-d PATH] FILE` prints DDL text that is
byte-identical to h5dump 1.14.6's (and to Debian's 1.14.5, which CI
uses) on the test files (all layouts and
chunk indexes, v1/v2 groups, compound, enum, strings, links, named
types, attributes; null-padded strings show their NULs at any depth),
or JSON in the HDF Group's hdf5-json layout (schema in the crate
README). Nested compounds print inline and `long double` values as
errors (exit 1); both are listed in the README.
- `h5rs stat FILE` reports h5stat's object, link, rank, layout, filter,
attribute, raw-data and file-size figures (equal to h5stat's on the test
files); metadata space is one figure, not broken down.
- `h5rs diff [-r] [-q] [-n N] [-d D] [-p R] [--follow-symlinks] A B [OBJ1
[OBJ2]]` (option names as h5diff's: `-c` is `--compare`, the count is
`-n`/`--count=N`) compares objects, kinds, datatypes, shapes, attributes, values and link
targets; exit status 0/1/2 as h5diff's. Soft links are compared by
target path, as h5diff's default, or with `--follow-symlinks` by the
objects they lead to (external links are never followed). Every path is
compared, including every name of a hard-linked object and the members
of a hard-linked group; with a `-d`/`-p` tolerance, integers are
compared exactly in integer arithmetic (no loss above 2^53), and a `-p`
below the f64 epsilon compares exactly, as h5diff's. Objects that cannot
be compared count as a difference (h5diff exits 0 for them), and NaN
equals NaN.
- `h5rs check [--data] FILE` is a structural validator: it walks every
object, parses every header message, verifies the checksums of every
version 2+ structure it meets (superblock, object headers and
continuation chunks, v2 B-tree nodes, fractal heap headers and — which
the library's reads do not — every direct and indirect heap block, and
extensible/fixed array chunk indexes), checks each chunk index against
its dataset (aligned, in-extent, unique, plausibly sized chunks), and
that raw data lies inside the file without overlaps. Every problem is
printed with its address; exit 1 when there are any. libhdf5's h5check
reads only the 1.8 format. On the conformance corpus it passes all 418
files that both clawhdf5 and h5py read in full, and `check --data` flags
152 of the 180 files of the CVE corpus (tank, 2026-09-26). `--data` also
follows variable-length data into its global heap collections and
reports a damaged one at its address. It inherits the library's
tolerance, though: 21 of the 28 it passes are files
h5dump 1.14.6 rejects (see `docs/known-issues.md`, header checks).
- Values over `--max-bytes` (default 1 GiB) are reported instead of read;
a panic is caught and reported as an internal error (exit 3).
`scripts/h5rs-fuzz.sh` runs every subcommand over a corpus (default the
CVE reproducers, optionally with byte-flipped copies) with overflow
checks, a timeout and a memory limit, and fails on any panic, crash or
hang; `scripts/h5rs-check-ok-files.sh` runs `check --data` over the
fully-read conformance files.
- Because the library does not verify fractal heap block checksums when
it reads a dense group's links or dense attributes, `h5rs` verifies a
heap's blocks before reading from it and refuses a damaged one, as
libhdf5 does, instead of printing what the damaged block holds.
### Signing
- `clawhdf5-agent`: **Ed25519-signed checkpoints** — the README's
"cryptographically verifiable memory", now true. With