feat(tools): h5rs, pure-Rust HDF5 tools (ls, dump, stat, diff, check)
New workspace crate clawhdf5-tools with one binary, h5rs, built only on the clawhdf5 facade and clawhdf5-format (no libhdf5, no C): - ls [-r] [-v] FILE[/path]: h5ls's listing (same text in its first two columns) plus the datatype; -v adds address, link count, layout and chunk index, chunk size, storage, filters, datatype and attributes. - dump [--json] [-A] [-p] [-d PATH] FILE: h5dump DDL (byte-identical to h5dump 1.14.6 on the test files) or hdf5-json. - stat FILE: h5stat's object/link/rank/layout/filter/attribute counts, raw data and total size. - diff [-r] [-q] [-d D] [-p R] A B [OBJ1 [OBJ2]]: structural and value differences, exit 0/1/2 like h5diff. - check [--data] FILE: walks every object, parses every message, verifies the checksums of every v2+ structure (including the fractal heap blocks the library never checks), checks chunk indexes against their datasets and raw data for out-of-file or overlapping extents; every problem with its address. Values over --max-bytes are reported, not read; dense-storage heaps are verified before objects are read from them; panics are caught (exit 3). Tests compare with h5ls, h5stat, h5dump and h5diff and with h5py's values, and flip the checksum of every checksummed structure in a v1.14-format file. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
@@ -0,0 +1,231 @@
|
||||
# clawhdf5-tools: `h5rs`
|
||||
|
||||
HDF5 command-line tools in pure Rust, built only on the `clawhdf5` facade and
|
||||
`clawhdf5-format`. No libhdf5 and no C code, so the binary also builds as a
|
||||
fully static executable: `cargo build --release -p clawhdf5-tools --target
|
||||
x86_64-unknown-linux-musl` gives a static-pie `h5rs` of 1.8 MB with no
|
||||
shared-library dependencies (built and run on tank, 2026-09-26, Rust 1.98).
|
||||
|
||||
| Command | Modelled on | What it does |
|
||||
|---------|-------------|--------------|
|
||||
| `h5rs ls` | `h5ls` | list objects: name, kind, shape, datatype; `-v` adds layout, chunking, storage, filters, attributes |
|
||||
| `h5rs dump` | `h5dump` | the file's structure and values as DDL text, or as JSON (hdf5-json layout) |
|
||||
| `h5rs stat` | `h5stat` | object, link, rank, layout, filter and attribute counts; raw-data and file size |
|
||||
| `h5rs diff` | `h5diff` | structural and value differences between two files or objects |
|
||||
| `h5rs check` | `h5check` | structural validator: every object and header message, the checksums of version 2+ structures, chunk index consistency |
|
||||
|
||||
```bash
|
||||
cargo install --path crates/clawhdf5-tools # or: cargo build --release -p clawhdf5-tools
|
||||
h5rs --help
|
||||
h5rs <command> --help
|
||||
```
|
||||
|
||||
Every command takes `--max-bytes N` where it reads values (default 1 GiB): a
|
||||
dataset whose dataspace claims more than that is reported instead of read, so
|
||||
a corrupt size cannot exhaust memory.
|
||||
|
||||
## `h5rs ls`
|
||||
|
||||
```console
|
||||
$ h5rs ls -r data.h5 # an extract of the output
|
||||
/ Group
|
||||
/external External Link {other.h5//x}
|
||||
/grp Group
|
||||
/grp/ext2 Dataset {6/Inf, 10/Inf} float32
|
||||
/grp/gz Dataset {1000} int32
|
||||
/hard2 Group, same as /grp/sub
|
||||
/named_t Type
|
||||
/soft Soft Link {/contig}
|
||||
```
|
||||
|
||||
The first two columns are h5ls's (`h5ls -r` prints the same text); h5rs adds
|
||||
the datatype. `FILE/path` lists a group's members or one dataset, as h5ls
|
||||
does. `-v` prints, per object:
|
||||
|
||||
```console
|
||||
$ h5rs ls -v data.h5/grp/gz
|
||||
gz Dataset {1000/1000}
|
||||
Address: 1120
|
||||
Links: 1
|
||||
Layout: chunked (fixed array index)
|
||||
Chunks: {100} 400 bytes
|
||||
Storage: 4000 logical bytes, 1237 allocated bytes, 323.36% utilization
|
||||
Filter-0: shuffle-2 OPT {4}
|
||||
Filter-1: deflate-1 OPT {4}
|
||||
Filter-2: fletcher32-3
|
||||
Type: 32-bit little-endian integer
|
||||
```
|
||||
|
||||
## `h5rs dump`
|
||||
|
||||
```console
|
||||
$ h5rs dump data.h5 # DDL, like h5dump
|
||||
$ h5rs dump -A data.h5 # no dataset values (attributes still shown), like h5dump -A
|
||||
$ h5rs dump -d /grp/gz data.h5 # one dataset
|
||||
$ h5rs dump -p data.h5 # also STORAGE_LAYOUT and FILTERS blocks
|
||||
$ h5rs dump --json data.h5 # hdf5-json
|
||||
```
|
||||
|
||||
The DDL output is h5dump's: on the test files of `tests/gen_files.py`
|
||||
(compact, contiguous and chunked datasets with every chunk index; v1 and v2
|
||||
groups; integers of both byte orders, floats, compound with an array member,
|
||||
enum, fixed and variable-length strings; soft, external and hard links; a
|
||||
named datatype; compact and dense attributes) `h5rs dump` and `h5rs dump -A`
|
||||
print the same bytes as h5dump 1.14.6 — `dump_matches_h5dump` in
|
||||
`tests/h5rs_interop.rs` checks this. Not covered by that test: references,
|
||||
opaque, bitfield, variable-length sequences and virtual datasets. Floats print at their own precision (a `float32` 0.1 prints as
|
||||
`0.1`), which for some values is more digits than h5dump's `%g`. The `-p`
|
||||
block is h5rs's own (it names the chunk index), not h5dump's.
|
||||
|
||||
### JSON schema
|
||||
|
||||
`--json` follows the HDF Group's [hdf5-json](https://github.com/HDFGroup/hdf5-json)
|
||||
layout:
|
||||
|
||||
```json
|
||||
{
|
||||
"apiVersion": "1.1.1",
|
||||
"root": "g-0000000000000060",
|
||||
"groups": { "<id>": { "alias": ["/grp"], "attributes": [...], "links": [...] } },
|
||||
"datasets": { "<id>": { "alias": [...], "attributes": [...], "shape": {...},
|
||||
"type": {...}, "creationProperties": {...}, "value": ... } },
|
||||
"datatypes": { "<id>": { "alias": [...], "attributes": [...], "type": {...} } }
|
||||
}
|
||||
```
|
||||
|
||||
- **ids** are `g-`/`d-`/`t-` plus the object header address in 16 hex digits
|
||||
(hdf5-json uses UUIDs; these are stable for a given file). `alias` lists
|
||||
every path that reaches the object.
|
||||
- **links**: `{"class": "H5L_TYPE_HARD", "title", "collection", "id"}`,
|
||||
`{"class": "H5L_TYPE_SOFT", "title", "h5path"}`,
|
||||
`{"class": "H5L_TYPE_EXTERNAL", "title", "file", "h5path"}`,
|
||||
`{"class": "H5L_TYPE_USER_DEFINED", "title", "linkClass"}`.
|
||||
- **shape**: `{"class": "H5S_NULL"}`, `{"class": "H5S_SCALAR"}` or
|
||||
`{"class": "H5S_SIMPLE", "dims": [...], "maxdims": [...]}` with
|
||||
`"H5S_UNLIMITED"` for an unlimited dimension.
|
||||
- **type**: `{"class": "H5T_INTEGER" | "H5T_FLOAT" | "H5T_BITFIELD", "base":
|
||||
"H5T_STD_I32LE" ...}`, `{"class": "H5T_STRING", "charSet", "strPad",
|
||||
"length": n | "H5T_VARIABLE"}`, `{"class": "H5T_COMPOUND", "fields": [{"name",
|
||||
"type"}]}`, `{"class": "H5T_ARRAY", "base", "dims"}`, `{"class": "H5T_ENUM",
|
||||
"base", "mapping": {"NAME": value}}`, `{"class": "H5T_VLEN", "base"}`,
|
||||
`{"class": "H5T_OPAQUE", "size", "tag"}`, `{"class": "H5T_REFERENCE", "base":
|
||||
"H5T_STD_REF_OBJ" | "H5T_STD_REF_DSETREG" | "H5T_STD_REF"}`.
|
||||
- **value**: nested lists in the dataset's shape (a scalar is the bare value,
|
||||
a null dataspace `null`). A compound element is a list of its members, an
|
||||
enum element its integer value, a string a JSON string, opaque/bitfield data
|
||||
a `0x...` hex string, an object reference the referenced object's path,
|
||||
NaN/infinities the strings `"NaN"`, `"Infinity"`, `"-Infinity"`, and an
|
||||
integer beyond 64 bits a decimal string.
|
||||
- **creationProperties**: `layout` (`{"class": "H5D_CHUNKED", "dims": [...]}`
|
||||
etc.) and `filters` (`[{"id", "name", "class", "parameters"}]`).
|
||||
|
||||
A value that cannot be read is replaced by `"value_error": "<reason>"` and
|
||||
the command exits 1.
|
||||
|
||||
## `h5rs stat`
|
||||
|
||||
Prints h5stat's report sections with the same labels for the facts it
|
||||
computes — object and link counts, max links to an object, max objects in a
|
||||
group, dataset ranks, layout counts, filter counts, attribute counts, total
|
||||
raw data size and total file size (all equal to h5stat's on the test files;
|
||||
`stat_matches_h5stat` checks them). It does not break metadata space down by
|
||||
structure as h5stat does; it reports metadata and free space as one figure.
|
||||
|
||||
## `h5rs diff`
|
||||
|
||||
```console
|
||||
$ h5rs diff a.h5 b.h5 # whole files
|
||||
$ h5rs diff a.h5 b.h5 /grp # one object and everything below it
|
||||
$ h5rs diff a.h5 b.h5 /x /y # different paths in each
|
||||
$ h5rs diff -r a.h5 b.h5 /d # list every differing element
|
||||
$ h5rs diff -d 0.001 a.h5 b.h5 # |a - b| > 0.001 is a difference
|
||||
$ h5rs diff -p 0.01 a.h5 b.h5 # |a - b| / |a| > 1% is a difference
|
||||
```
|
||||
|
||||
Exit status: 0 no differences, 1 differences, 2 error — the same as h5diff's
|
||||
on the cases `diff_exit_codes_match_h5diff` runs. Compared: which objects
|
||||
exist, their kinds, datatypes and shapes, attribute sets and values, dataset
|
||||
values, and soft/external link targets. Two differences from h5diff, on
|
||||
purpose: objects that cannot be compared (different shapes or datatype
|
||||
classes) count as a difference (h5diff warns and exits 0), and two NaNs are
|
||||
equal.
|
||||
|
||||
## `h5rs check`
|
||||
|
||||
```console
|
||||
$ h5rs check data.h5
|
||||
checked data.h5: superblock v3, 33 objects (4 groups, 28 datasets, 1 named datatypes), 139 header messages, 34 chunks
|
||||
checksums verified: superblock 1, v2 object headers 33, v2 B-trees 3, fractal heaps 3 (+3 blocks), chunk indexes 5
|
||||
no problems found
|
||||
|
||||
$ h5rs check damaged.h5 # one bit flipped in /grp/ext1's chunk index header
|
||||
problem: 0x79c /grp/ext1: chunk index (extensible array): checksum mismatch: expected 0xcfe2391f, computed 0x2dd59bed
|
||||
checked damaged.h5: superblock v3, 33 objects (4 groups, 28 datasets, 1 named datatypes), 139 header messages, 27 chunks
|
||||
checksums verified: superblock 1, v2 object headers 33, v2 B-trees 3, fractal heaps 3 (+3 blocks), chunk indexes 4
|
||||
1 problem found
|
||||
```
|
||||
|
||||
Each problem line is `problem: <address> <object path>: <what>`; addresses
|
||||
are HDF5 addresses (relative to the superblock, as h5dump and h5ls print
|
||||
them).
|
||||
|
||||
It walks every object reachable from the root group (and the superblock
|
||||
extension) and checks:
|
||||
|
||||
- the superblock (and its checksum, version 2+) and that the file is not
|
||||
shorter than the superblock's end-of-file address;
|
||||
- every object header, with the checksum of version 2 headers and of their
|
||||
continuation chunks, and every header message parsed by type (dataspace,
|
||||
datatype, fill value, layout, filter pipeline, attributes, link info,
|
||||
links, group info, symbol table), including shared messages;
|
||||
- groups: the symbol table (v1 B-tree, local heap, symbol nodes), or the
|
||||
links, and for dense storage the fractal heap — header, and every direct
|
||||
and indirect block with its checksum, back-pointer and heap offset — and
|
||||
the v2 B-tree name and creation-order indexes (every node's checksum, and
|
||||
the record count against the header's);
|
||||
- dense attribute storage the same way;
|
||||
- datasets: the layout against the dataspace and datatype (compact and
|
||||
contiguous sizes, chunk rank), and for chunked datasets the whole chunk
|
||||
index (v1 B-tree, single chunk, implicit, fixed array, extensible array,
|
||||
v2 B-tree, with the checksums of the last three): every chunk's offset must
|
||||
be a multiple of the chunk size and inside the extent, appear once, and
|
||||
have a plausible size;
|
||||
- that all raw data (contiguous blocks and chunks) lies inside the file and
|
||||
no two pieces overlap.
|
||||
|
||||
`--data` also reads every dataset, decoding every chunk through its filters
|
||||
(which catches corrupt compressed data and Fletcher-32 mismatches). Data the
|
||||
tool cannot decode (a filter it does not implement, such as szip, or a
|
||||
dataset over `--max-bytes`) is a `note:`, not a problem. Every problem is
|
||||
printed with the address of the structure involved; the exit status is 0
|
||||
when there are none, 1 when there are, 2 for a usage error or a missing
|
||||
file.
|
||||
|
||||
libhdf5's h5check understands only the HDF5 1.8 file format; `h5rs check`
|
||||
also covers the structures HDF5 1.10+ writes (fixed/extensible array and v2
|
||||
B-tree chunk indexes, and version 3 superblocks).
|
||||
|
||||
What it does not check: free-space manager and shared-message (SOHM) table
|
||||
checksums, global heap collections other than those a value read touches,
|
||||
and objects reachable only by external links.
|
||||
|
||||
## Robustness
|
||||
|
||||
A panic is a bug: `h5rs` catches it, prints `internal error`, and exits 3
|
||||
(`check` records it against the object and carries on). `scripts/h5rs-fuzz.sh`
|
||||
runs every subcommand over every file of a corpus (by default the HDF Group's
|
||||
CVE reproducers, fetched by `conformance/fetch-corpus.sh`), optionally with
|
||||
byte-flipped copies (`MUTATE=N`), under a timeout and a memory limit, with
|
||||
overflow checks on, and fails on any panic, crash or hang.
|
||||
`scripts/h5rs-check-ok-files.sh` runs `check --data` over the conformance
|
||||
files that both clawhdf5 and h5py read in full, which must all pass.
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
CLAWHDF5_PYTHON=.venv/bin/python CLAWHDF5_REQUIRE_INTEROP=1 cargo test -p clawhdf5-tools
|
||||
```
|
||||
|
||||
The interop tests write their files with h5py and compare with h5ls, h5stat,
|
||||
h5dump and h5diff; each skips when what it needs is missing unless
|
||||
`CLAWHDF5_REQUIRE_INTEROP=1`.
|
||||
Reference in New Issue
Block a user