clawhdf5/crates/clawhdf5-tools/README.md

# clawhdf5-tools: `h5rs`

HDF5 command-line tools in pure Rust, built only on the `clawhdf5` facade and
`clawhdf5-format`. No libhdf5 and no C code, so the binary also builds as a
fully static executable: `cargo build --release -p clawhdf5-tools --target
x86_64-unknown-linux-musl` gives a static-pie `h5rs` of 1.8 MB with no
shared-library dependencies (built and run on tank, 2026-09-26, Rust 1.98).

| Command | Modelled on | What it does |
|---------|-------------|--------------|
| `h5rs ls` | `h5ls` | list objects: name, kind, shape, datatype; `-v` adds layout, chunking, storage, filters, attributes |
| `h5rs dump` | `h5dump` | the file's structure and values as DDL text, or as JSON (hdf5-json layout) |
| `h5rs stat` | `h5stat` | object, link, rank, layout, filter and attribute counts; raw-data and file size |
| `h5rs diff` | `h5diff` | structural and value differences between two files or objects |
| `h5rs check` | `h5check` | structural validator: every object and header message, the checksums of version 2+ structures, chunk index consistency |

```bash
cargo install --path crates/clawhdf5-tools     # or: cargo build --release -p clawhdf5-tools
h5rs --help
h5rs <command> --help
```

Every command takes `--max-bytes N` where it reads values (default 1 GiB): a
dataset whose dataspace claims more than that is reported instead of read, so
a corrupt size cannot exhaust memory.

## `h5rs ls`

```console
$ h5rs ls -r data.h5            # an extract of the output
/                        Group
/external                External Link {other.h5//x}
/grp                     Group
/grp/ext2                Dataset {6/Inf, 10/Inf} float32
/grp/gz                  Dataset {1000} int32
/hard2                   Group, same as /grp/sub
/named_t                 Type
/soft                    Soft Link {/contig}
```

The first two columns are h5ls's (`h5ls -r` prints the same text); h5rs adds
the datatype. `FILE/path` lists a group's members or one dataset, as h5ls
does. `-v` prints, per object:

```console
$ h5rs ls -v data.h5/grp/gz
gz                       Dataset {1000/1000}
    Address:   1120
    Links:     1
    Layout:    chunked (fixed array index)
    Chunks:    {100} 400 bytes
    Storage:   4000 logical bytes, 1237 allocated bytes, 323.36% utilization
    Filter-0:  shuffle-2 OPT {4}
    Filter-1:  deflate-1 OPT {4}
    Filter-2:  fletcher32-3
    Type:      32-bit little-endian integer
```

## `h5rs dump`

```console
$ h5rs dump data.h5            # DDL, like h5dump
$ h5rs dump -A data.h5         # no dataset values (attributes still shown), like h5dump -A
$ h5rs dump -d /grp/gz data.h5 # one dataset
$ h5rs dump -p data.h5         # also STORAGE_LAYOUT and FILTERS blocks
$ h5rs dump --json data.h5     # hdf5-json
```

The DDL output is h5dump's: on the test files of `tests/gen_files.py`
(compact, contiguous and chunked datasets with every chunk index; v1 and v2
groups; integers of both byte orders, floats, compound with an array member,
enum, fixed and variable-length strings; soft, external and hard links; a
named datatype; compact and dense attributes) `h5rs dump` and `h5rs dump -A`
print the same bytes as h5dump 1.14.6 and as Debian's h5dump 1.14.5 (the
`hdf5-tools` package CI installs in `rust:latest`; the whole interop suite
was run in that image on 2026-09-26) — `dump_matches_h5dump` in
`tests/h5rs_interop.rs` checks this, and `dump_shows_nul_padding_in_nested_strings`
that null-padded strings show their NULs (`"a\000b"`) at any depth, as
h5dump's do. `dump_prints_vl_data_like_h5dump` covers variable-length
strings (one with an embedded NUL, which prints up to the NUL; empty; null,
which prints `NULL`), variable-length sequences, a VL compound member and
a VL attribute, with 8- and 4-byte offsets. Not covered by those tests:
references, opaque, bitfield, non-ASCII UTF-8 (h5dump prints each byte
above 0x7f as a sign-extended octal escape, h5rs the character) and
virtual datasets. Known differences from
h5dump:

- Floats print at their own precision (a `float32` 0.1 prints as `0.1`),
  which for some values is more digits than h5dump's `%g`.
- A compound nested in a compound prints inline (`{ 1, 2.5 }`) where
  h5dump prints it as an indented block, one member per line; only the
  outer compound is a block.
- `long double` (x87 80-bit) and other floats wider than 64 bits: the
  datatype is printed as an `H5T_FLOAT { ... }` block instead of h5dump's
  one-line description, and each value as `<error: ...>`; `dump` then
  exits 1. The library cannot convert them (see `docs/known-issues.md`).
- The `-p` block is h5rs's own (it names the chunk index), not h5dump's.

### JSON schema

`--json` follows the HDF Group's [hdf5-json](https://github.com/HDFGroup/hdf5-json)
layout:

```json
{
  "apiVersion": "1.1.1",
  "root": "g-0000000000000060",
  "groups":    { "<id>": { "alias": ["/grp"], "attributes": [...], "links": [...] } },
  "datasets":  { "<id>": { "alias": [...], "attributes": [...], "shape": {...},
                           "type": {...}, "creationProperties": {...}, "value": ... } },
  "datatypes": { "<id>": { "alias": [...], "attributes": [...], "type": {...} } }
}
```

- **ids** are `g-`/`d-`/`t-` plus the object header address in 16 hex digits
  (hdf5-json uses UUIDs; these are stable for a given file). `alias` lists
  every path that reaches the object.
- **links**: `{"class": "H5L_TYPE_HARD", "title", "collection", "id"}`,
  `{"class": "H5L_TYPE_SOFT", "title", "h5path"}`,
  `{"class": "H5L_TYPE_EXTERNAL", "title", "file", "h5path"}`,
  `{"class": "H5L_TYPE_USER_DEFINED", "title", "linkClass"}`.
- **shape**: `{"class": "H5S_NULL"}`, `{"class": "H5S_SCALAR"}` or
  `{"class": "H5S_SIMPLE", "dims": [...], "maxdims": [...]}` with
  `"H5S_UNLIMITED"` for an unlimited dimension.
- **type**: `{"class": "H5T_INTEGER" | "H5T_FLOAT" | "H5T_BITFIELD", "base":
  "H5T_STD_I32LE" ...}`, `{"class": "H5T_STRING", "charSet", "strPad",
  "length": n | "H5T_VARIABLE"}`, `{"class": "H5T_COMPOUND", "fields": [{"name",
  "type"}]}`, `{"class": "H5T_ARRAY", "base", "dims"}`, `{"class": "H5T_ENUM",
  "base", "mapping": {"NAME": value}}`, `{"class": "H5T_VLEN", "base"}`,
  `{"class": "H5T_OPAQUE", "size", "tag"}`, `{"class": "H5T_REFERENCE", "base":
  "H5T_STD_REF_OBJ" | "H5T_STD_REF_DSETREG" | "H5T_STD_REF"}`.
- **value**: nested lists in the dataset's shape (a scalar is the bare value,
  a null dataspace `null`). A compound element is a list of its members, an
  enum element its integer value, a string a JSON string, opaque/bitfield data
  a `0x...` hex string, an object reference the referenced object's path,
  NaN/infinities the strings `"NaN"`, `"Infinity"`, `"-Infinity"`, and an
  integer beyond 64 bits a decimal string.
- **creationProperties**: `layout` (`{"class": "H5D_CHUNKED", "dims": [...]}`
  etc.) and `filters` (`[{"id", "name", "class", "parameters"}]`).

A value that cannot be read is replaced by `"value_error": "<reason>"` and
the command exits 1.

## `h5rs stat`

Prints h5stat's report sections with the same labels for the facts it
computes — object and link counts, max links to an object, max objects in a
group, dataset ranks, layout counts, filter counts, attribute counts, total
raw data size and total file size (all equal to h5stat's on the test files;
`stat_matches_h5stat` checks them). It does not break metadata space down by
structure as h5stat does; it reports metadata and free space as one figure.

## `h5rs diff`

```console
$ h5rs diff a.h5 b.h5                # whole files
$ h5rs diff a.h5 b.h5 /grp           # one object and everything below it
$ h5rs diff a.h5 b.h5 /x /y          # different paths in each
$ h5rs diff -r a.h5 b.h5 /d          # list every differing element
$ h5rs diff -d 0.001 a.h5 b.h5       # |a - b| > 0.001 is a difference
$ h5rs diff -p 0.01 a.h5 b.h5        # |a - b| / |a| > 1% is a difference
$ h5rs diff --follow-symlinks a.h5 b.h5 /lnk   # what the soft link /lnk leads to
$ h5rs diff -r -n 10 a.h5 b.h5       # list at most 10 differing elements per object
```

The options are named as h5diff's: `-n N`/`--count=N` limits the listed
elements, `-c`/`--compare` (list objects that are not comparable) is
accepted and always in effect, and `--delta=D`, `--relative=R` work too.

Integers are compared in integer arithmetic, with or without a tolerance, so
64-bit values beyond 2^53 lose no precision (`-d 0` tells 2^60 from
2^60 + 1). A relative tolerance below the f64 epsilon (2.2e-16) compares
exactly, as h5diff's does.

Exit status: 0 no differences, 1 differences, 2 error — the same as h5diff's
on the cases `diff_exit_codes_match_h5diff` runs. Compared: which objects
exist, their kinds, datatypes and shapes, attribute sets and values, dataset
values, and soft/external link targets. A soft link — including an OBJ
that is itself a soft link — is compared as a link, by its target path, as
h5diff does; `--follow-symlinks` compares the objects soft links lead to
instead (and walks into soft-linked groups), and two dangling links are then
the same, as in h5diff. External links are always compared by target (file
and path): `--follow-symlinks` does not open other files, where h5diff's
does. Two dangling soft links with different targets are a difference
(h5diff reports them with `-r`/`-v` but exits 0 without). Every path is compared: an object
hard-linked under two names is compared under both, with everything below
it, so a file that shares one object between two names equals a file that
stores two identical copies. Differences from h5diff, on purpose: objects
that cannot be compared (different shapes or datatype classes) count as a
difference (h5diff warns and exits 0); two NaNs are equal; and the members
of a group reached by a second hard link are compared (h5diff lists them in
one file only and exits 1).

## `h5rs check`

```console
$ h5rs check data.h5
checked data.h5: superblock v3, 33 objects (4 groups, 28 datasets, 1 named datatypes), 139 header messages, 34 chunks
checksums verified: superblock 1, v2 object headers 33, v2 B-trees 3, fractal heaps 3 (+3 blocks), chunk indexes 5
no problems found

$ h5rs check damaged.h5          # one bit flipped in /grp/ext1's chunk index header
problem: 0x79c /grp/ext1: chunk index (extensible array): checksum mismatch: expected 0xcfe2391f, computed 0x2dd59bed
checked damaged.h5: superblock v3, 33 objects (4 groups, 28 datasets, 1 named datatypes), 139 header messages, 27 chunks
checksums verified: superblock 1, v2 object headers 33, v2 B-trees 3, fractal heaps 3 (+3 blocks), chunk indexes 4
1 problem found
```

Each problem line is `problem: <address> <object path>: <what>`; addresses
are HDF5 addresses (relative to the superblock, as h5dump and h5ls print
them).

It walks every object reachable from the root group (and the superblock
extension) and checks:

- the superblock (and its checksum, version 2+) and that the file is not
  shorter than the superblock's end-of-file address;
- every object header, with the checksum of version 2 headers and of their
  continuation chunks, and every header message parsed by type (dataspace,
  datatype, fill value, layout, filter pipeline, attributes, link info,
  links, group info, symbol table), including shared messages;
- groups: the symbol table (v1 B-tree, local heap, symbol nodes), or the
  links, and for dense storage the fractal heap — header, and every direct
  and indirect block with its checksum, back-pointer and heap offset — and
  the v2 B-tree name and creation-order indexes (every node's checksum, and
  the record count against the header's);
- dense attribute storage the same way;
- datasets: the layout against the dataspace and datatype (compact and
  contiguous sizes, chunk rank), and for chunked datasets the whole chunk
  index (v1 B-tree, single chunk, implicit, fixed array, extensible array,
  v2 B-tree, with the checksums of the last three): every chunk's offset must
  be a multiple of the chunk size and inside the extent, appear once, and
  have a plausible size;
- that all raw data (contiguous blocks and chunks) lies inside the file and
  no two pieces overlap.

`--data` also reads every dataset, decoding every chunk through its filters
(which catches corrupt compressed data and Fletcher-32 mismatches), and
follows every variable-length element (strings and sequences, also inside
compounds and arrays) of every dataset and attribute into its global heap
collection: a collection that does not parse or overlaps another, a missing
heap object, or a heap object whose size is not exactly the element's
length times its base size (libhdf5 refuses such an element) is a problem
at the collection's address. Variable-length elements are resolved by the
library's `VlResolver`, as `clawhdf5::File` resolves them. Data the
tool cannot decode (a filter it does not implement, such as szip, or a
dataset over `--max-bytes`) is a `note:`, not a problem. Every problem is
printed with the address of the structure involved; the exit status is 0
when there are none, 1 when there are, 2 for a usage error or a missing
file.

libhdf5's h5check understands only the HDF5 1.8 file format; `h5rs check`
also covers the structures HDF5 1.10+ writes (fixed/extensible array and v2
B-tree chunk indexes, and version 3 superblocks).

What it does not check: free-space manager and shared-message (SOHM) table
checksums, global heap collections no variable-length value points into (and
none at all without `--data`), and objects reachable only by external links. It validates with
clawhdf5's parsers, so it accepts what they accept: some header damage that
libhdf5 refuses goes unreported. Of the 150 CVE and fuzzer files of the
HDF Group's `cve_hdf5` corpus (`cvefiles/` and `fuzzerfiles/`),
`check --data` passes 15, and h5dump 1.14.6 rejects 8 of those (tank,
2026-09-26, `h5rs check --data F` and `h5dump F` per file; before the
library's header checks it passed 28, of which h5dump rejects 21, and 16
and 9 before a VL type's stored element size was checked).

## Robustness

A panic is a bug: `h5rs` catches it, prints `internal error`, and exits 3
(`check` records it against the object and carries on). `scripts/h5rs-fuzz.sh`
runs every subcommand over every file of a corpus (by default the HDF Group's
CVE reproducers, fetched by `conformance/fetch-corpus.sh`), optionally with
byte-flipped copies (`MUTATE=N`), under a timeout and a memory limit, with
overflow checks on, and fails on any panic, crash or hang.
`scripts/h5rs-check-ok-files.sh` runs `check --data` over the conformance
files that both clawhdf5 and h5py read in full, which must all pass.

## Tests

```bash
CLAWHDF5_PYTHON=.venv/bin/python CLAWHDF5_REQUIRE_INTEROP=1 cargo test -p clawhdf5-tools
```

The interop tests write their files with h5py and compare with h5ls, h5stat,
h5dump and h5diff; each skips when what it needs is missing unless
`CLAWHDF5_REQUIRE_INTEROP=1`.