clawhdf5/crates/clawhdf5-tools

clawhdf5-tools: h5rs

HDF5 command-line tools in pure Rust, built only on the clawhdf5 facade and clawhdf5-format. No libhdf5 and no C code, so the binary also builds as a fully static executable: cargo build --release -p clawhdf5-tools --target x86_64-unknown-linux-musl gives a static-pie h5rs of 1.8 MB with no shared-library dependencies (built and run on tank, 2026-09-26, Rust 1.98).

Command	Modelled on	What it does
h5rs ls	h5ls	list objects: name, kind, shape, datatype; -v adds layout, chunking, storage, filters, attributes
h5rs dump	h5dump	the file's structure and values as DDL text, or as JSON (hdf5-json layout)
h5rs stat	h5stat	object, link, rank, layout, filter and attribute counts; raw-data and file size
h5rs diff	h5diff	structural and value differences between two files or objects
h5rs check	h5check	structural validator: every object and header message, the checksums of version 2+ structures, chunk index consistency

cargo install --path crates/clawhdf5-tools     # or: cargo build --release -p clawhdf5-tools
h5rs --help
h5rs <command> --help

Every command takes --max-bytes N where it reads values (default 1 GiB): a dataset whose dataspace claims more than that is reported instead of read, so a corrupt size cannot exhaust memory.

h5rs ls

$ h5rs ls -r data.h5            # an extract of the output
/                        Group
/external                External Link {other.h5//x}
/grp                     Group
/grp/ext2                Dataset {6/Inf, 10/Inf} float32
/grp/gz                  Dataset {1000} int32
/hard2                   Group, same as /grp/sub
/named_t                 Type
/soft                    Soft Link {/contig}

The first two columns are h5ls's (h5ls -r prints the same text); h5rs adds the datatype. FILE/path lists a group's members or one dataset, as h5ls does. -v prints, per object:

$ h5rs ls -v data.h5/grp/gz
gz                       Dataset {1000/1000}
    Address:   1120
    Links:     1
    Layout:    chunked (fixed array index)
    Chunks:    {100} 400 bytes
    Storage:   4000 logical bytes, 1237 allocated bytes, 323.36% utilization
    Filter-0:  shuffle-2 OPT {4}
    Filter-1:  deflate-1 OPT {4}
    Filter-2:  fletcher32-3
    Type:      32-bit little-endian integer

h5rs dump

$ h5rs dump data.h5            # DDL, like h5dump
$ h5rs dump -A data.h5         # no dataset values (attributes still shown), like h5dump -A
$ h5rs dump -d /grp/gz data.h5 # one dataset
$ h5rs dump -p data.h5         # also STORAGE_LAYOUT and FILTERS blocks
$ h5rs dump --json data.h5     # hdf5-json

The DDL output is h5dump's: on the test files of tests/gen_files.py (compact, contiguous and chunked datasets with every chunk index; v1 and v2 groups; integers of both byte orders, floats, compound with an array member, enum, fixed and variable-length strings; soft, external and hard links; a named datatype; compact and dense attributes) h5rs dump and h5rs dump -A print the same bytes as h5dump 1.14.6 and as Debian's h5dump 1.14.5 (the hdf5-tools package CI installs in rust:latest; the whole interop suite was run in that image on 2026-09-26) — dump_matches_h5dump in tests/h5rs_interop.rs checks this, and dump_shows_nul_padding_in_nested_strings that null-padded strings show their NULs ("a\000b") at any depth, as h5dump's do. Not covered by those tests: references, opaque, bitfield, variable-length sequences and virtual datasets. Known differences from h5dump:

• Floats print at their own precision (a float32 0.1 prints as 0.1), which for some values is more digits than h5dump's %g.
• A compound nested in a compound prints inline ({ 1, 2.5 }) where h5dump prints it as an indented block, one member per line; only the outer compound is a block.
• long double (x87 80-bit) and other floats wider than 64 bits: the datatype is printed as an H5T_FLOAT { ... } block instead of h5dump's one-line description, and each value as <error: ...>; dump then exits 1. The library cannot convert them (see docs/known-issues.md).
• The -p block is h5rs's own (it names the chunk index), not h5dump's.

JSON schema

--json follows the HDF Group's hdf5-json layout:

{
  "apiVersion": "1.1.1",
  "root": "g-0000000000000060",
  "groups":    { "<id>": { "alias": ["/grp"], "attributes": [...], "links": [...] } },
  "datasets":  { "<id>": { "alias": [...], "attributes": [...], "shape": {...},
                           "type": {...}, "creationProperties": {...}, "value": ... } },
  "datatypes": { "<id>": { "alias": [...], "attributes": [...], "type": {...} } }
}

• ids are g-/d-/t- plus the object header address in 16 hex digits (hdf5-json uses UUIDs; these are stable for a given file). alias lists every path that reaches the object.
• links: {"class": "H5L_TYPE_HARD", "title", "collection", "id"}, {"class": "H5L_TYPE_SOFT", "title", "h5path"}, {"class": "H5L_TYPE_EXTERNAL", "title", "file", "h5path"}, {"class": "H5L_TYPE_USER_DEFINED", "title", "linkClass"}.
• shape: {"class": "H5S_NULL"}, {"class": "H5S_SCALAR"} or {"class": "H5S_SIMPLE", "dims": [...], "maxdims": [...]} with "H5S_UNLIMITED" for an unlimited dimension.
• type: {"class": "H5T_INTEGER" | "H5T_FLOAT" | "H5T_BITFIELD", "base": "H5T_STD_I32LE" ...}, {"class": "H5T_STRING", "charSet", "strPad", "length": n | "H5T_VARIABLE"}, {"class": "H5T_COMPOUND", "fields": [{"name", "type"}]}, {"class": "H5T_ARRAY", "base", "dims"}, {"class": "H5T_ENUM", "base", "mapping": {"NAME": value}}, {"class": "H5T_VLEN", "base"}, {"class": "H5T_OPAQUE", "size", "tag"}, {"class": "H5T_REFERENCE", "base": "H5T_STD_REF_OBJ" | "H5T_STD_REF_DSETREG" | "H5T_STD_REF"}.
• value: nested lists in the dataset's shape (a scalar is the bare value, a null dataspace null). A compound element is a list of its members, an enum element its integer value, a string a JSON string, opaque/bitfield data a 0x... hex string, an object reference the referenced object's path, NaN/infinities the strings "NaN", "Infinity", "-Infinity", and an integer beyond 64 bits a decimal string.
• creationProperties: layout ({"class": "H5D_CHUNKED", "dims": [...]} etc.) and filters ([{"id", "name", "class", "parameters"}]).

A value that cannot be read is replaced by "value_error": "<reason>" and the command exits 1.

h5rs stat

Prints h5stat's report sections with the same labels for the facts it computes — object and link counts, max links to an object, max objects in a group, dataset ranks, layout counts, filter counts, attribute counts, total raw data size and total file size (all equal to h5stat's on the test files; stat_matches_h5stat checks them). It does not break metadata space down by structure as h5stat does; it reports metadata and free space as one figure.

h5rs diff

$ h5rs diff a.h5 b.h5                # whole files
$ h5rs diff a.h5 b.h5 /grp           # one object and everything below it
$ h5rs diff a.h5 b.h5 /x /y          # different paths in each
$ h5rs diff -r a.h5 b.h5 /d          # list every differing element
$ h5rs diff -d 0.001 a.h5 b.h5       # |a - b| > 0.001 is a difference
$ h5rs diff -p 0.01 a.h5 b.h5        # |a - b| / |a| > 1% is a difference
$ h5rs diff --follow-symlinks a.h5 b.h5 /lnk   # what the soft link /lnk leads to
$ h5rs diff -r -n 10 a.h5 b.h5       # list at most 10 differing elements per object

The options are named as h5diff's: -n N/--count=N limits the listed elements, -c/--compare (list objects that are not comparable) is accepted and always in effect, and --delta=D, --relative=R work too.

Integers are compared in integer arithmetic, with or without a tolerance, so 64-bit values beyond 2^53 lose no precision (-d 0 tells 2^60 from 2^60 + 1). A relative tolerance below the f64 epsilon (2.2e-16) compares exactly, as h5diff's does.

Exit status: 0 no differences, 1 differences, 2 error — the same as h5diff's on the cases diff_exit_codes_match_h5diff runs. Compared: which objects exist, their kinds, datatypes and shapes, attribute sets and values, dataset values, and soft/external link targets. A soft link — including an OBJ that is itself a soft link — is compared as a link, by its target path, as h5diff does; --follow-symlinks compares the objects soft links lead to instead (and walks into soft-linked groups), and two dangling links are then the same, as in h5diff. External links are always compared by target (file and path): --follow-symlinks does not open other files, where h5diff's does. Two dangling soft links with different targets are a difference (h5diff reports them with -r/-v but exits 0 without). Every path is compared: an object hard-linked under two names is compared under both, with everything below it, so a file that shares one object between two names equals a file that stores two identical copies. Differences from h5diff, on purpose: objects that cannot be compared (different shapes or datatype classes) count as a difference (h5diff warns and exits 0); two NaNs are equal; and the members of a group reached by a second hard link are compared (h5diff lists them in one file only and exits 1).

h5rs check

$ h5rs check data.h5
checked data.h5: superblock v3, 33 objects (4 groups, 28 datasets, 1 named datatypes), 139 header messages, 34 chunks
checksums verified: superblock 1, v2 object headers 33, v2 B-trees 3, fractal heaps 3 (+3 blocks), chunk indexes 5
no problems found

$ h5rs check damaged.h5          # one bit flipped in /grp/ext1's chunk index header
problem: 0x79c /grp/ext1: chunk index (extensible array): checksum mismatch: expected 0xcfe2391f, computed 0x2dd59bed
checked damaged.h5: superblock v3, 33 objects (4 groups, 28 datasets, 1 named datatypes), 139 header messages, 27 chunks
checksums verified: superblock 1, v2 object headers 33, v2 B-trees 3, fractal heaps 3 (+3 blocks), chunk indexes 4
1 problem found

Each problem line is problem: <address> <object path>: <what>; addresses are HDF5 addresses (relative to the superblock, as h5dump and h5ls print them).

It walks every object reachable from the root group (and the superblock extension) and checks:

• the superblock (and its checksum, version 2+) and that the file is not shorter than the superblock's end-of-file address;
• every object header, with the checksum of version 2 headers and of their continuation chunks, and every header message parsed by type (dataspace, datatype, fill value, layout, filter pipeline, attributes, link info, links, group info, symbol table), including shared messages;
• groups: the symbol table (v1 B-tree, local heap, symbol nodes), or the links, and for dense storage the fractal heap — header, and every direct and indirect block with its checksum, back-pointer and heap offset — and the v2 B-tree name and creation-order indexes (every node's checksum, and the record count against the header's);
• dense attribute storage the same way;
• datasets: the layout against the dataspace and datatype (compact and contiguous sizes, chunk rank), and for chunked datasets the whole chunk index (v1 B-tree, single chunk, implicit, fixed array, extensible array, v2 B-tree, with the checksums of the last three): every chunk's offset must be a multiple of the chunk size and inside the extent, appear once, and have a plausible size;
• that all raw data (contiguous blocks and chunks) lies inside the file and no two pieces overlap.

--data also reads every dataset, decoding every chunk through its filters (which catches corrupt compressed data and Fletcher-32 mismatches), and follows every variable-length element (strings and sequences, also inside compounds and arrays) of every dataset and attribute into its global heap collection: a collection that does not parse, a missing heap object, or a sequence longer than its heap object is a problem at the collection's address. Data the tool cannot decode (a filter it does not implement, such as szip, or a dataset over --max-bytes) is a note:, not a problem. Every problem is printed with the address of the structure involved; the exit status is 0 when there are none, 1 when there are, 2 for a usage error or a missing file.

libhdf5's h5check understands only the HDF5 1.8 file format; h5rs check also covers the structures HDF5 1.10+ writes (fixed/extensible array and v2 B-tree chunk indexes, and version 3 superblocks).

What it does not check: free-space manager and shared-message (SOHM) table checksums, global heap collections no variable-length value points into (and none at all without --data), and objects reachable only by external links. It validates with clawhdf5's parsers, so it accepts what they accept: some header damage that libhdf5 refuses goes unreported. Of the 150 CVE and fuzzer files of the HDF Group's cve_hdf5 corpus (cvefiles/ and fuzzerfiles/), check --data passes 16, and h5dump 1.14.6 rejects 9 of those (tank, 2026-09-26, h5rs check --data F and h5dump F per file; before the library's header checks it passed 28, of which h5dump rejects 21).

Robustness

A panic is a bug: h5rs catches it, prints internal error, and exits 3 (check records it against the object and carries on). scripts/h5rs-fuzz.sh runs every subcommand over every file of a corpus (by default the HDF Group's CVE reproducers, fetched by conformance/fetch-corpus.sh), optionally with byte-flipped copies (MUTATE=N), under a timeout and a memory limit, with overflow checks on, and fails on any panic, crash or hang. scripts/h5rs-check-ok-files.sh runs check --data over the conformance files that both clawhdf5 and h5py read in full, which must all pass.

Tests

CLAWHDF5_PYTHON=.venv/bin/python CLAWHDF5_REQUIRE_INTEROP=1 cargo test -p clawhdf5-tools

The interop tests write their files with h5py and compare with h5ls, h5stat, h5dump and h5diff; each skips when what it needs is missing unless CLAWHDF5_REQUIRE_INTEROP=1.