From 068503759384160d5c6a8b8f6cc94b59a40e2283 Mon Sep 17 00:00:00 2001 From: osobh Date: Sat, 26 Sep 2026 00:29:38 -0500 Subject: [PATCH] docs: h5rs in the changelog and the crate table Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 44 ++++++++++++++++++++++++++++++++++++++++++++ CLAUDE.md | 15 +++++++++++++-- 2 files changed, 57 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1fc429d..9321be2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -112,6 +112,50 @@ `quantized_index = false`, or pass `create --f32-index` to the CLI, to opt out. The CLI's `--quantized-index` is still accepted but is now a no-op. +### Tools +- New crate **`clawhdf5-tools`** with the binary **`h5rs`**: HDF5 + command-line tools without libhdf5, built only on the `clawhdf5` facade + and `clawhdf5-format` (no C, so it also builds as a static musl binary). + - `h5rs ls [-r] [-v] FILE[/path]` lists objects like h5ls (its first two + columns are h5ls's text on the test files) plus the datatype; `-v` adds + address, link count, layout and chunk index, chunk size, storage, + filters, datatype and attributes. + - `h5rs dump [--json] [-A] [-p] [-d PATH] FILE` prints DDL text that is + byte-identical to h5dump 1.14.6's on the test files (all layouts and + chunk indexes, v1/v2 groups, compound, enum, strings, links, named + types, attributes), or JSON in the HDF Group's hdf5-json layout (schema + in the crate README). + - `h5rs stat FILE` reports h5stat's object, link, rank, layout, filter, + attribute, raw-data and file-size figures (equal to h5stat's on the test + files); metadata space is one figure, not broken down. + - `h5rs diff [-r] [-q] [-d D] [-p R] A B [OBJ1 [OBJ2]]` compares objects, + kinds, datatypes, shapes, attributes, values and link targets; exit + status 0/1/2 as h5diff's. Objects that cannot be compared count as a + difference (h5diff exits 0 for them), and NaN equals NaN. + - `h5rs check [--data] FILE` is a structural validator: it walks every + object, parses every header message, verifies the checksums of every + version 2+ structure it meets (superblock, object headers and + continuation chunks, v2 B-tree nodes, fractal heap headers and — which + the library's reads do not — every direct and indirect heap block, and + extensible/fixed array chunk indexes), checks each chunk index against + its dataset (aligned, in-extent, unique, plausibly sized chunks), and + that raw data lies inside the file without overlaps. Every problem is + printed with its address; exit 1 when there are any. libhdf5's h5check + reads only the 1.8 format. On the conformance corpus it passes all 418 + files that both clawhdf5 and h5py read in full, and flags 149 of the + 180 CVE reproducers. + - Values over `--max-bytes` (default 1 GiB) are reported instead of read; + a panic is caught and reported as an internal error (exit 3). + `scripts/h5rs-fuzz.sh` runs every subcommand over a corpus (default the + CVE reproducers, optionally with byte-flipped copies) with overflow + checks, a timeout and a memory limit, and fails on any panic, crash or + hang; `scripts/h5rs-check-ok-files.sh` runs `check --data` over the + fully-read conformance files. + - Because the library does not verify fractal heap block checksums when + it reads a dense group's links or dense attributes, `h5rs` verifies a + heap's blocks before reading from it and refuses a damaged one, as + libhdf5 does, instead of printing what the damaged block holds. + ### Signing - `clawhdf5-agent`: **Ed25519-signed checkpoints** — the README's "cryptographically verifiable memory", now true. With diff --git a/CLAUDE.md b/CLAUDE.md index 3f8d9c1..5e8583c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -5,7 +5,7 @@ Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persist ## Architecture -Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature): +Cargo workspace with 17 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature): | Crate | Role | |-------|------| @@ -21,7 +21,8 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F | `clawhdf5-accel` | CPU SIMD acceleration path | | `clawhdf5-migrate` | SQLite → HDF5 agent-memory migration | | `clawhdf5-android` | Android JNI bindings | -| `clawhdf5-cli` | Command-line interface | +| `clawhdf5-cli` | Command-line interface (agent memory) | +| `clawhdf5-tools` | `h5rs`: pure-Rust HDF5 tools — `ls`, `dump` (DDL / hdf5-json), `stat`, `diff`, `check` (structural + checksum validator) | | `clawhdf5-napi` | Node.js native addon bindings | | `clawhdf5-py` | PyO3 Python bindings | | `clawhdf5-bench` | Benchmark suite | @@ -188,6 +189,16 @@ cargo run -p clawhdf5-cli -- --help # create, save, search, recall, stats, flush-wal, agents-md, export, snapshot subcommands ``` +### HDF5 tools (`h5rs`, crate `clawhdf5-tools`) +```bash +cargo run -p clawhdf5-tools -- ls -r file.h5 # also dump [--json], stat, diff, check +bash scripts/h5rs-fuzz.sh # every subcommand over the CVE corpus: no panic/crash/hang +bash scripts/h5rs-check-ok-files.sh --data # check passes every fully-read conformance file +``` +Its interop tests compare against h5ls/h5stat/h5dump/h5diff (Debian +`hdf5-tools`, installed in CI); `dump` must stay byte-identical to h5dump on +the test files. + ### Python bindings ```bash cd crates/clawhdf5-py