# clawhdf5-tools: `h5rs` HDF5 command-line tools in pure Rust, built only on the `clawhdf5` facade and `clawhdf5-format`. No libhdf5 and no C code, so the binary also builds as a fully static executable: `cargo build --release -p clawhdf5-tools --target x86_64-unknown-linux-musl` gives a static-pie `h5rs` of 1.8 MB with no shared-library dependencies (built and run on tank, 2026-09-26, Rust 1.98). | Command | Modelled on | What it does | |---------|-------------|--------------| | `h5rs ls` | `h5ls` | list objects: name, kind, shape, datatype; `-v` adds layout, chunking, storage, filters, attributes | | `h5rs dump` | `h5dump` | the file's structure and values as DDL text, or as JSON (hdf5-json layout) | | `h5rs stat` | `h5stat` | object, link, rank, layout, filter and attribute counts; raw-data and file size | | `h5rs diff` | `h5diff` | structural and value differences between two files or objects | | `h5rs check` | `h5check` | structural validator: every object and header message, the checksums of version 2+ structures, chunk index consistency | ```bash cargo install --path crates/clawhdf5-tools # or: cargo build --release -p clawhdf5-tools h5rs --help h5rs --help ``` Every command takes `--max-bytes N` where it reads values (default 1 GiB): a dataset whose dataspace claims more than that is reported instead of read, so a corrupt size cannot exhaust memory. ## `h5rs ls` ```console $ h5rs ls -r data.h5 # an extract of the output / Group /external External Link {other.h5//x} /grp Group /grp/ext2 Dataset {6/Inf, 10/Inf} float32 /grp/gz Dataset {1000} int32 /hard2 Group, same as /grp/sub /named_t Type /soft Soft Link {/contig} ``` The first two columns are h5ls's (`h5ls -r` prints the same text); h5rs adds the datatype. `FILE/path` lists a group's members or one dataset, as h5ls does. `-v` prints, per object: ```console $ h5rs ls -v data.h5/grp/gz gz Dataset {1000/1000} Address: 1120 Links: 1 Layout: chunked (fixed array index) Chunks: {100} 400 bytes Storage: 4000 logical bytes, 1237 allocated bytes, 323.36% utilization Filter-0: shuffle-2 OPT {4} Filter-1: deflate-1 OPT {4} Filter-2: fletcher32-3 Type: 32-bit little-endian integer ``` ## `h5rs dump` ```console $ h5rs dump data.h5 # DDL, like h5dump $ h5rs dump -A data.h5 # no dataset values (attributes still shown), like h5dump -A $ h5rs dump -d /grp/gz data.h5 # one dataset $ h5rs dump -p data.h5 # also STORAGE_LAYOUT and FILTERS blocks $ h5rs dump --json data.h5 # hdf5-json ``` The DDL output is h5dump's: on the test files of `tests/gen_files.py` (compact, contiguous and chunked datasets with every chunk index; v1 and v2 groups; integers of both byte orders, floats, compound with an array member, enum, fixed and variable-length strings; soft, external and hard links; a named datatype; compact and dense attributes) `h5rs dump` and `h5rs dump -A` print the same bytes as h5dump 1.14.6 — `dump_matches_h5dump` in `tests/h5rs_interop.rs` checks this. Not covered by that test: references, opaque, bitfield, variable-length sequences and virtual datasets. Floats print at their own precision (a `float32` 0.1 prints as `0.1`), which for some values is more digits than h5dump's `%g`. The `-p` block is h5rs's own (it names the chunk index), not h5dump's. ### JSON schema `--json` follows the HDF Group's [hdf5-json](https://github.com/HDFGroup/hdf5-json) layout: ```json { "apiVersion": "1.1.1", "root": "g-0000000000000060", "groups": { "": { "alias": ["/grp"], "attributes": [...], "links": [...] } }, "datasets": { "": { "alias": [...], "attributes": [...], "shape": {...}, "type": {...}, "creationProperties": {...}, "value": ... } }, "datatypes": { "": { "alias": [...], "attributes": [...], "type": {...} } } } ``` - **ids** are `g-`/`d-`/`t-` plus the object header address in 16 hex digits (hdf5-json uses UUIDs; these are stable for a given file). `alias` lists every path that reaches the object. - **links**: `{"class": "H5L_TYPE_HARD", "title", "collection", "id"}`, `{"class": "H5L_TYPE_SOFT", "title", "h5path"}`, `{"class": "H5L_TYPE_EXTERNAL", "title", "file", "h5path"}`, `{"class": "H5L_TYPE_USER_DEFINED", "title", "linkClass"}`. - **shape**: `{"class": "H5S_NULL"}`, `{"class": "H5S_SCALAR"}` or `{"class": "H5S_SIMPLE", "dims": [...], "maxdims": [...]}` with `"H5S_UNLIMITED"` for an unlimited dimension. - **type**: `{"class": "H5T_INTEGER" | "H5T_FLOAT" | "H5T_BITFIELD", "base": "H5T_STD_I32LE" ...}`, `{"class": "H5T_STRING", "charSet", "strPad", "length": n | "H5T_VARIABLE"}`, `{"class": "H5T_COMPOUND", "fields": [{"name", "type"}]}`, `{"class": "H5T_ARRAY", "base", "dims"}`, `{"class": "H5T_ENUM", "base", "mapping": {"NAME": value}}`, `{"class": "H5T_VLEN", "base"}`, `{"class": "H5T_OPAQUE", "size", "tag"}`, `{"class": "H5T_REFERENCE", "base": "H5T_STD_REF_OBJ" | "H5T_STD_REF_DSETREG" | "H5T_STD_REF"}`. - **value**: nested lists in the dataset's shape (a scalar is the bare value, a null dataspace `null`). A compound element is a list of its members, an enum element its integer value, a string a JSON string, opaque/bitfield data a `0x...` hex string, an object reference the referenced object's path, NaN/infinities the strings `"NaN"`, `"Infinity"`, `"-Infinity"`, and an integer beyond 64 bits a decimal string. - **creationProperties**: `layout` (`{"class": "H5D_CHUNKED", "dims": [...]}` etc.) and `filters` (`[{"id", "name", "class", "parameters"}]`). A value that cannot be read is replaced by `"value_error": ""` and the command exits 1. ## `h5rs stat` Prints h5stat's report sections with the same labels for the facts it computes — object and link counts, max links to an object, max objects in a group, dataset ranks, layout counts, filter counts, attribute counts, total raw data size and total file size (all equal to h5stat's on the test files; `stat_matches_h5stat` checks them). It does not break metadata space down by structure as h5stat does; it reports metadata and free space as one figure. ## `h5rs diff` ```console $ h5rs diff a.h5 b.h5 # whole files $ h5rs diff a.h5 b.h5 /grp # one object and everything below it $ h5rs diff a.h5 b.h5 /x /y # different paths in each $ h5rs diff -r a.h5 b.h5 /d # list every differing element $ h5rs diff -d 0.001 a.h5 b.h5 # |a - b| > 0.001 is a difference $ h5rs diff -p 0.01 a.h5 b.h5 # |a - b| / |a| > 1% is a difference ``` Exit status: 0 no differences, 1 differences, 2 error — the same as h5diff's on the cases `diff_exit_codes_match_h5diff` runs. Compared: which objects exist, their kinds, datatypes and shapes, attribute sets and values, dataset values, and soft/external link targets. Two differences from h5diff, on purpose: objects that cannot be compared (different shapes or datatype classes) count as a difference (h5diff warns and exits 0), and two NaNs are equal. ## `h5rs check` ```console $ h5rs check data.h5 checked data.h5: superblock v3, 33 objects (4 groups, 28 datasets, 1 named datatypes), 139 header messages, 34 chunks checksums verified: superblock 1, v2 object headers 33, v2 B-trees 3, fractal heaps 3 (+3 blocks), chunk indexes 5 no problems found $ h5rs check damaged.h5 # one bit flipped in /grp/ext1's chunk index header problem: 0x79c /grp/ext1: chunk index (extensible array): checksum mismatch: expected 0xcfe2391f, computed 0x2dd59bed checked damaged.h5: superblock v3, 33 objects (4 groups, 28 datasets, 1 named datatypes), 139 header messages, 27 chunks checksums verified: superblock 1, v2 object headers 33, v2 B-trees 3, fractal heaps 3 (+3 blocks), chunk indexes 4 1 problem found ``` Each problem line is `problem:
: `; addresses are HDF5 addresses (relative to the superblock, as h5dump and h5ls print them). It walks every object reachable from the root group (and the superblock extension) and checks: - the superblock (and its checksum, version 2+) and that the file is not shorter than the superblock's end-of-file address; - every object header, with the checksum of version 2 headers and of their continuation chunks, and every header message parsed by type (dataspace, datatype, fill value, layout, filter pipeline, attributes, link info, links, group info, symbol table), including shared messages; - groups: the symbol table (v1 B-tree, local heap, symbol nodes), or the links, and for dense storage the fractal heap — header, and every direct and indirect block with its checksum, back-pointer and heap offset — and the v2 B-tree name and creation-order indexes (every node's checksum, and the record count against the header's); - dense attribute storage the same way; - datasets: the layout against the dataspace and datatype (compact and contiguous sizes, chunk rank), and for chunked datasets the whole chunk index (v1 B-tree, single chunk, implicit, fixed array, extensible array, v2 B-tree, with the checksums of the last three): every chunk's offset must be a multiple of the chunk size and inside the extent, appear once, and have a plausible size; - that all raw data (contiguous blocks and chunks) lies inside the file and no two pieces overlap. `--data` also reads every dataset, decoding every chunk through its filters (which catches corrupt compressed data and Fletcher-32 mismatches). Data the tool cannot decode (a filter it does not implement, such as szip, or a dataset over `--max-bytes`) is a `note:`, not a problem. Every problem is printed with the address of the structure involved; the exit status is 0 when there are none, 1 when there are, 2 for a usage error or a missing file. libhdf5's h5check understands only the HDF5 1.8 file format; `h5rs check` also covers the structures HDF5 1.10+ writes (fixed/extensible array and v2 B-tree chunk indexes, and version 3 superblocks). What it does not check: free-space manager and shared-message (SOHM) table checksums, global heap collections other than those a value read touches, and objects reachable only by external links. ## Robustness A panic is a bug: `h5rs` catches it, prints `internal error`, and exits 3 (`check` records it against the object and carries on). `scripts/h5rs-fuzz.sh` runs every subcommand over every file of a corpus (by default the HDF Group's CVE reproducers, fetched by `conformance/fetch-corpus.sh`), optionally with byte-flipped copies (`MUTATE=N`), under a timeout and a memory limit, with overflow checks on, and fails on any panic, crash or hang. `scripts/h5rs-check-ok-files.sh` runs `check --data` over the conformance files that both clawhdf5 and h5py read in full, which must all pass. ## Tests ```bash CLAWHDF5_PYTHON=.venv/bin/python CLAWHDF5_REQUIRE_INTEROP=1 cargo test -p clawhdf5-tools ``` The interop tests write their files with h5py and compare with h5ls, h5stat, h5dump and h5diff; each skips when what it needs is missing unless `CLAWHDF5_REQUIRE_INTEROP=1`.