clawhdf5/crates/clawhdf5-tools/README.md

clawhdf5-tools: h5rs

HDF5 command-line tools in pure Rust, built only on the clawhdf5 facade and clawhdf5-format. No libhdf5 and no C code, so the binary also builds as a fully static executable: cargo build --release -p clawhdf5-tools --target x86_64-unknown-linux-musl gives a static-pie h5rs of 1.8 MB with no shared-library dependencies (built and run on tank, 2026-09-26, Rust 1.98).

Command	Modelled on	What it does
h5rs ls	h5ls	list objects: name, kind, shape, datatype; -v adds layout, chunking, storage, filters, attributes
h5rs dump	h5dump	the file's structure and values as DDL text, or as JSON (hdf5-json layout)
h5rs stat	h5stat	object, link, rank, layout, filter and attribute counts; raw-data and file size
h5rs diff	h5diff	structural and value differences between two files or objects
h5rs check	h5check	structural validator: every object and header message, the checksums of version 2+ structures, chunk index consistency

cargo install --path crates/clawhdf5-tools     # or: cargo build --release -p clawhdf5-tools
h5rs --help
h5rs <command> --help

Every command takes --max-bytes N where it reads values (default 1 GiB): a dataset whose dataspace claims more than that is reported instead of read, so a corrupt size cannot exhaust memory.

h5rs ls

$ h5rs ls -r data.h5            # an extract of the output
/                        Group
/external                External Link {other.h5//x}
/grp                     Group
/grp/ext2                Dataset {6/Inf, 10/Inf} float32
/grp/gz                  Dataset {1000} int32
/hard2                   Group, same as /grp/sub
/named_t                 Type
/soft                    Soft Link {/contig}

The first two columns are h5ls's (h5ls -r prints the same text); h5rs adds the datatype. FILE/path lists a group's members or one dataset, as h5ls does. -v prints, per object:

$ h5rs ls -v data.h5/grp/gz
gz                       Dataset {1000/1000}
    Address:   1120
    Links:     1
    Layout:    chunked (fixed array index)
    Chunks:    {100} 400 bytes
    Storage:   4000 logical bytes, 1237 allocated bytes, 323.36% utilization
    Filter-0:  shuffle-2 OPT {4}
    Filter-1:  deflate-1 OPT {4}
    Filter-2:  fletcher32-3
    Type:      32-bit little-endian integer

h5rs dump

$ h5rs dump data.h5            # DDL, like h5dump
$ h5rs dump -A data.h5         # no dataset values (attributes still shown), like h5dump -A
$ h5rs dump -d /grp/gz data.h5 # one dataset
$ h5rs dump -p data.h5         # also STORAGE_LAYOUT and FILTERS blocks
$ h5rs dump --json data.h5     # hdf5-json

The DDL output is h5dump's: on the test files of tests/gen_files.py (compact, contiguous and chunked datasets with every chunk index; v1 and v2 groups; integers of both byte orders, floats, compound with an array member, enum, fixed and variable-length strings; soft, external and hard links; a named datatype; compact and dense attributes) h5rs dump and h5rs dump -A print the same bytes as h5dump 1.14.6 — dump_matches_h5dump in tests/h5rs_interop.rs checks this. Not covered by that test: references, opaque, bitfield, variable-length sequences and virtual datasets. Floats print at their own precision (a float32 0.1 prints as 0.1), which for some values is more digits than h5dump's %g. The -p block is h5rs's own (it names the chunk index), not h5dump's.

JSON schema

--json follows the HDF Group's hdf5-json layout:

{
  "apiVersion": "1.1.1",
  "root": "g-0000000000000060",
  "groups":    { "<id>": { "alias": ["/grp"], "attributes": [...], "links": [...] } },
  "datasets":  { "<id>": { "alias": [...], "attributes": [...], "shape": {...},
                           "type": {...}, "creationProperties": {...}, "value": ... } },
  "datatypes": { "<id>": { "alias": [...], "attributes": [...], "type": {...} } }
}

• ids are g-/d-/t- plus the object header address in 16 hex digits (hdf5-json uses UUIDs; these are stable for a given file). alias lists every path that reaches the object.
• links: {"class": "H5L_TYPE_HARD", "title", "collection", "id"}, {"class": "H5L_TYPE_SOFT", "title", "h5path"}, {"class": "H5L_TYPE_EXTERNAL", "title", "file", "h5path"}, {"class": "H5L_TYPE_USER_DEFINED", "title", "linkClass"}.
• shape: {"class": "H5S_NULL"}, {"class": "H5S_SCALAR"} or {"class": "H5S_SIMPLE", "dims": [...], "maxdims": [...]} with "H5S_UNLIMITED" for an unlimited dimension.
• type: {"class": "H5T_INTEGER" | "H5T_FLOAT" | "H5T_BITFIELD", "base": "H5T_STD_I32LE" ...}, {"class": "H5T_STRING", "charSet", "strPad", "length": n | "H5T_VARIABLE"}, {"class": "H5T_COMPOUND", "fields": [{"name", "type"}]}, {"class": "H5T_ARRAY", "base", "dims"}, {"class": "H5T_ENUM", "base", "mapping": {"NAME": value}}, {"class": "H5T_VLEN", "base"}, {"class": "H5T_OPAQUE", "size", "tag"}, {"class": "H5T_REFERENCE", "base": "H5T_STD_REF_OBJ" | "H5T_STD_REF_DSETREG" | "H5T_STD_REF"}.
• value: nested lists in the dataset's shape (a scalar is the bare value, a null dataspace null). A compound element is a list of its members, an enum element its integer value, a string a JSON string, opaque/bitfield data a 0x... hex string, an object reference the referenced object's path, NaN/infinities the strings "NaN", "Infinity", "-Infinity", and an integer beyond 64 bits a decimal string.
• creationProperties: layout ({"class": "H5D_CHUNKED", "dims": [...]} etc.) and filters ([{"id", "name", "class", "parameters"}]).

A value that cannot be read is replaced by "value_error": "<reason>" and the command exits 1.

h5rs stat

Prints h5stat's report sections with the same labels for the facts it computes — object and link counts, max links to an object, max objects in a group, dataset ranks, layout counts, filter counts, attribute counts, total raw data size and total file size (all equal to h5stat's on the test files; stat_matches_h5stat checks them). It does not break metadata space down by structure as h5stat does; it reports metadata and free space as one figure.

h5rs diff

$ h5rs diff a.h5 b.h5                # whole files
$ h5rs diff a.h5 b.h5 /grp           # one object and everything below it
$ h5rs diff a.h5 b.h5 /x /y          # different paths in each
$ h5rs diff -r a.h5 b.h5 /d          # list every differing element
$ h5rs diff -d 0.001 a.h5 b.h5       # |a - b| > 0.001 is a difference
$ h5rs diff -p 0.01 a.h5 b.h5        # |a - b| / |a| > 1% is a difference

Exit status: 0 no differences, 1 differences, 2 error — the same as h5diff's on the cases diff_exit_codes_match_h5diff runs. Compared: which objects exist, their kinds, datatypes and shapes, attribute sets and values, dataset values, and soft/external link targets. Every path is compared: an object hard-linked under two names is compared under both, with everything below it, so a file that shares one object between two names equals a file that stores two identical copies. Differences from h5diff, on purpose: objects that cannot be compared (different shapes or datatype classes) count as a difference (h5diff warns and exits 0); two NaNs are equal; and the members of a group reached by a second hard link are compared (h5diff lists them in one file only and exits 1).

h5rs check

$ h5rs check data.h5
checked data.h5: superblock v3, 33 objects (4 groups, 28 datasets, 1 named datatypes), 139 header messages, 34 chunks
checksums verified: superblock 1, v2 object headers 33, v2 B-trees 3, fractal heaps 3 (+3 blocks), chunk indexes 5
no problems found

$ h5rs check damaged.h5          # one bit flipped in /grp/ext1's chunk index header
problem: 0x79c /grp/ext1: chunk index (extensible array): checksum mismatch: expected 0xcfe2391f, computed 0x2dd59bed
checked damaged.h5: superblock v3, 33 objects (4 groups, 28 datasets, 1 named datatypes), 139 header messages, 27 chunks
checksums verified: superblock 1, v2 object headers 33, v2 B-trees 3, fractal heaps 3 (+3 blocks), chunk indexes 4
1 problem found

Each problem line is problem: <address> <object path>: <what>; addresses are HDF5 addresses (relative to the superblock, as h5dump and h5ls print them).

It walks every object reachable from the root group (and the superblock extension) and checks:

• the superblock (and its checksum, version 2+) and that the file is not shorter than the superblock's end-of-file address;
• every object header, with the checksum of version 2 headers and of their continuation chunks, and every header message parsed by type (dataspace, datatype, fill value, layout, filter pipeline, attributes, link info, links, group info, symbol table), including shared messages;
• groups: the symbol table (v1 B-tree, local heap, symbol nodes), or the links, and for dense storage the fractal heap — header, and every direct and indirect block with its checksum, back-pointer and heap offset — and the v2 B-tree name and creation-order indexes (every node's checksum, and the record count against the header's);
• dense attribute storage the same way;
• datasets: the layout against the dataspace and datatype (compact and contiguous sizes, chunk rank), and for chunked datasets the whole chunk index (v1 B-tree, single chunk, implicit, fixed array, extensible array, v2 B-tree, with the checksums of the last three): every chunk's offset must be a multiple of the chunk size and inside the extent, appear once, and have a plausible size;
• that all raw data (contiguous blocks and chunks) lies inside the file and no two pieces overlap.

--data also reads every dataset, decoding every chunk through its filters (which catches corrupt compressed data and Fletcher-32 mismatches). Data the tool cannot decode (a filter it does not implement, such as szip, or a dataset over --max-bytes) is a note:, not a problem. Every problem is printed with the address of the structure involved; the exit status is 0 when there are none, 1 when there are, 2 for a usage error or a missing file.

libhdf5's h5check understands only the HDF5 1.8 file format; h5rs check also covers the structures HDF5 1.10+ writes (fixed/extensible array and v2 B-tree chunk indexes, and version 3 superblocks).

What it does not check: free-space manager and shared-message (SOHM) table checksums, global heap collections other than those a value read touches, and objects reachable only by external links. It validates with clawhdf5's parsers, so it accepts what they accept: some header damage that libhdf5 refuses goes unreported (of the 180 files of the HDF Group's CVE corpus, check --data passes 33, and h5dump 1.14.6 rejects 26 of those; tank, 2026-09-26).

Robustness

A panic is a bug: h5rs catches it, prints internal error, and exits 3 (check records it against the object and carries on). scripts/h5rs-fuzz.sh runs every subcommand over every file of a corpus (by default the HDF Group's CVE reproducers, fetched by conformance/fetch-corpus.sh), optionally with byte-flipped copies (MUTATE=N), under a timeout and a memory limit, with overflow checks on, and fails on any panic, crash or hang. scripts/h5rs-check-ok-files.sh runs check --data over the conformance files that both clawhdf5 and h5py read in full, which must all pass.

Tests

CLAWHDF5_PYTHON=.venv/bin/python CLAWHDF5_REQUIRE_INTEROP=1 cargo test -p clawhdf5-tools

The interop tests write their files with h5py and compare with h5ls, h5stat, h5dump and h5diff; each skips when what it needs is missing unless CLAWHDF5_REQUIRE_INTEROP=1.