quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: CLAUDE.md regrouped into rules, invariants and workflows

Browse Source 
Standing rules (no C by default, h5py must read what we write, one
float16 implementation, claims need evidence, OpenClaw/ZeroClaw
withdrawn, the known-issues rule) are gathered in one place; library and
agent-memory invariants are split; the CI section lists what
ci-test.sh and the conformance workflow run now instead of a dated
"two jobs, green" line. Adds the conformance, remote, wasm, Python and
benchmark workflows (idle load below 2, dated records), and warns that
scripts/run-benchmarks.sh is stale and overwrites BENCHMARKS.md. Crate
roles corrected (clawhdf5-io is I/O adapters; codecs live in
clawhdf5-format). Benchmark figures now live in BENCHMARKS.md only.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-28 11:08:53 -05:00
co-authored by Claude Opus 5.5
parent 24cc9d14d8
commit 38107b90ed
1 changed files with 232 additions and 220 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+232 -220
View File
@@ -1,253 +1,266 @@
# clawhdf5 # clawhdf5
## Purpose ## Purpose
Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated vector search. A standalone library. Its one verified consumer is ClawBrainHub (`.brain` files); no agent framework integrates it (OpenClaw and ZeroClaw claims were withdrawn on 2026-09-25 — neither was ever true). Pure-Rust HDF5 implementation (read, write, in-place edit, remote and browser
reads) plus agent memory on top of it: HNSW vector search, a WAL-backed store,
and GPU vector distances. A standalone library. Its one verified consumer is
ClawBrainHub (`.brain` files); no agent framework integrates it (see
*Standing rules*).
## Architecture ## Architecture
Cargo workspace with 19 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature): Cargo workspace, 19 crates under `crates/` (plus `libaec-sys`, the FFI crate
behind the optional `szip` feature). MSRV 1.92 (`rust-version`, checked in CI).
| Crate | Role | | Crate | Role |
|-------|------| |-------|------|
| `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants | | `clawhdf5-format` | The HDF5 format: parsers and writer (superblock, headers, B-trees, heaps, chunk indexes), the `Storage` trait, the filter pipeline and registry (`filter_registry`), every codec except deflate (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec; pure-Rust LZF, bitshuffle, bzip2, Blosc 1; Blosc2 and ZFP read-only), `float16`, `checksum` |
| `clawhdf5-io` | Read/write implementation | | `clawhdf5-filters` | Deflate backends (zlib-rs default, zlib-ng, Apple Compression) |
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 and ZFP read-only) live in `clawhdf5-format`. | | `clawhdf5-io` | I/O adapters (buffers, mmap, prefetch) |
| `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs | | `clawhdf5-derive` | `#[derive(H5Type)]` for compound types |
| `clawhdf5` | Main facade crate | | `clawhdf5` | Facade: `File`, `FileBuilder`, `Dataset`, `FileEditor` (`src/edit/`), SWMR reading (`src/swmr.rs`) |
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer | | `clawhdf5-netcdf4` | NetCDF-4 read support |
| `clawhdf5-ann` | HNSW approximate nearest-neighbor vector index | | `clawhdf5-remote` | `open_url`: HTTP(S) range requests and object stores (S3, GCS, Azure) through `BlockCache` |
| `clawhdf5-agent` | Agent memory, session history, knowledge graph storage | | `clawhdf5-tools` | `h5rs`: `ls`, `dump` (DDL / hdf5-json), `stat`, `diff`, `check` |
| `clawhdf5-gpu` | GPU vector distance computation via wgpu (hand-written WGSL compute shaders) — not dataset I/O | | `clawhdf5-py` | PyO3 bindings (h5py-like API, remote files, `'r+'` editing) |
| `clawhdf5-accel` | CPU SIMD acceleration path | | `clawhdf5-wasm` | wasm-bindgen browser reader (`open(bytes)`, `openUrl(url)`); demo in `examples/wasm-viewer/` |
| `clawhdf5-migrate` | SQLite → HDF5 agent-memory migration | | `clawhdf5-ann` | HNSW index |
| `clawhdf5-agent` | Agent memory store (`HDF5Memory`), sessions, knowledge graph, BM25 |
| `clawhdf5-accel` | CPU SIMD kernels (AVX2, NEON) |
| `clawhdf5-gpu` | wgpu vector distances (WGSL) — not dataset I/O; HDF5 I/O is CPU-only |
| `clawhdf5-migrate` | SQLite → agent store migration |
| `clawhdf5-cli` | Agent-memory CLI |
| `clawhdf5-napi` | Node.js addon (the `packages/clawhdf5-node` wrapper is broken; `docs/known-issues.md`) |
| `clawhdf5-android` | Android JNI bindings | | `clawhdf5-android` | Android JNI bindings |
| `clawhdf5-cli` | Command-line interface (agent memory) | | `clawhdf5-bench` | Benchmarks and harnesses (`search_harness`, `read_harness`, `concurrent_read`, `longmemeval_bench`, …) |
| `clawhdf5-tools` | `h5rs`: pure-Rust HDF5 tools — `ls`, `dump` (DDL / hdf5-json), `stat`, `diff`, `check` (structural + checksum validator) |
| `clawhdf5-napi` | Node.js native addon bindings |
| `clawhdf5-py` | PyO3 Python bindings |
| `clawhdf5-wasm` | WebAssembly (wasm-bindgen) reader for the browser; demo in `examples/wasm-viewer/` |
| `clawhdf5-remote` | Remote files: `open_url` over HTTP(S) range requests and object stores (`object_store`: S3, GCS, Azure) through a mandatory block cache (`BlockCache`) |
| `clawhdf5-bench` | Benchmark suite |
## Key Features Reference docs: `docs/known-issues.md` (open issues table first — check it
- Zero-C-dependency HDF5 read/write: no libhdf5, and deflate defaults to before calling something a bug or a feature), `BENCHMARKS.md` (headline
pure-Rust zlib-rs (`fast-deflate` opts into zlib-ng, which needs cmake). numbers first), `CONFORMANCE.md` (generated), `docs/design/range-reads.md`
`ci-test.sh` fails if a C-building crate enters the core crates' default and `docs/design/swmr.md`, `CHANGELOG.md` (full detail of every fix).
tree. flate2 must keep `runtime_detection` with zlib-rs — without it zlib-rs
loses SIMD and inflates 3.5x slower. MSRV is 1.92 (`rust-version`, checked ## Standing rules
in CI).
- HNSW vector index for semantic similarity search over agent memories — the - **No C in the default build.** No libhdf5; deflate defaults to pure-Rust
`clawhdf5-agent` `hnsw` feature is **on by default**, so `hybrid_search` uses zlib-rs (`fast-deflate` opts into zlib-ng, which needs cmake). `ci-test.sh`
the approximate `clawhdf5-ann` index for the vector stage (the index mirrors fails if a C-building crate enters the core crates' default tree. Zstd,
the cache and self-heals on drift). Build the agent with SZIP, `https` (ring) and `s3`/`gcs`/`azure` (aws-lc-rs) are opt-in. flate2
`--no-default-features --features float16` to force the exact linear cosine scan. must keep `runtime_detection` with zlib-rs — without it zlib-rs loses SIMD
The agent's `parallel` feature (also default) builds the index on a thread and inflates 3.5x slower.
pool; the graph is identical with or without it. - **Every file we write must open in h5py/libhdf5.** Interop tests compare
The index uses the HNSW paper's diversity heuristic for neighbour selection against h5py and h5dump; `f32` and empty datasets did not open until
(plain closest-M capped recall on clustered data: 0.31 recall@10 at 100K). Its 2026-09-23.
graph is saved to `<store>.h5.ann` at each checkpoint and reloaded by `open()` - **float16 has one implementation:** `clawhdf5_format::float16`.
(tied to the checkpoint by a generation id; stale/damaged sidecars are - **Claims need evidence.** Performance and integration claims in docs must
ignored and the index rebuilt). `MemoryConfig::quantized_index` (**on by be measured, dated (with machine and command), or withdrawn. Benchmark
default** for new stores, persisted; stores predating the setting load as numbers are dated records: never edit a measured value, add a new dated
`false` and keep their f32 index — guarded by section and mark the old one superseded.
`tests/fixtures/store_v2_5_0.h5`; CLI opt-out is `create --f32-index`) - **OpenClaw is not supported** (decided 2026-09-25): clawhdf5 is not and
stores the index's own copy of the embeddings as `i8`, never was an OpenClaw memory plugin; the old `memory.backend = "clawhdf5"`
which roughly halves a loaded store's memory (2.72x -> 1.74x the raw vectors config was never valid. `docs/openclaw.md` records what a real plugin would
at 100K); because quantised distances are approximate and `ef` cannot need. The `openclaw` module's `ClawhdfBackend` is just `search` with
compensate, the query path then re-scores the candidate pool against the re-rank + confidence on.
exact embeddings, which holds recall at the f32 index's level. It is also - **ZeroClaw does not use clawhdf5** (checked 2026-09-25 against upstream
faster at equal recall: 1.63x the QPS on x86-64 (AVX2) and 1.18x on a v0.8.5 and the `osobh/zeroclaw` fork and their history): its memory
Raspberry Pi 5 (`clawhdf5_accel::dot_i8`, NEON `SDOT` via inline asm since backends are its own; `clawhdf5-migrate`'s default SQLite layout is not
the intrinsic is unstable; plain NEON on pre-dotprod cores). The aarch64 ZeroClaw's schema. Don't reintroduce integration claims without an
code is `cfg`'d out on x86, so x86 CI never compiles or lints it — test it integration and a test against the real consumer.
on real ARM (`rpivision02`, 10.0.2.3, is a Pi 5). `hybrid_search` keeps one incremental BM25 - **known-issues.md:** one entry per bug; when fixed, record it in
index for the life of the store and never writes the store: Hebbian `CHANGELOG.md` and move the entry to *Fixed (history)* with date, PR,
activation boosts are persisted by the next checkpoint (or on drop), not per affected releases and what users must do — never delete it.
query. Measure any search-path change with
`cargo run --release -p clawhdf5-bench --bin search_harness` (baselines in ## HDF5 library: invariants and gotchas
`BENCHMARKS.md`).
- WAL (write-ahead log) for crash-safe persistence, with a chained CRC32 - **Remote/range reads** (`docs/design/range-reads.md`, M0-M5 merged in PRs
trailer per entry (each entry's CRC folds in the previous entry's CRC) so a #17-#21): every format-crate read path goes through `Storage`
corrupted, reordered, duplicated, or spliced entry stops replay cleanly (`read_at`/`read_ranges`/`hint`). `File::open_storage` takes any
instead of loading bad or tampered data. The pre-chaining per-entry-CRC `Storage`; `clawhdf5_remote::open_url` wraps HTTP (`HttpStorage`, ureq) or
format (v2) is still fully readable; the oldest no-CRC format (v1) is only `ObjectStoreStorage` in `BlockCache` (1 MiB blocks, LRU budget, in-flight
reachable through the one-time migration path in `HDF5Memory::open`, not dedup, coalesced runs). Remote files are pinned by ETag/Last-Modified and
through the public `WalFile::read_entries`. length (`RemoteError::FileChanged`). Zero-copy APIs and `File::as_bytes`
**What the WAL guarantees:** integrity, ordering, and recovery from a need an in-memory file. Parse through `File::storage()` and the `*_in`
*process* crash at any point — including between a checkpoint and the WAL functions, not `as_bytes`, in new code (the Python bindings do).
truncate (each checkpoint records a `WalMark` in `/meta`, and `open()` skips `ObjectStoreStorage` runs reads on its own small tokio runtime, so it
the WAL prefix the `.h5` already contains, so entries are never applied works from any thread.
twice). Checkpoints and snapshots are made durable as a unit (temp file - **SWMR** (`docs/design/swmr.md`): `File::open_swmr` reads a file a libhdf5
synced, renamed, directory synced). **What it does not guarantee:** SWMR writer is appending to — positioned reads, no chunk cache, bounded
individual WAL appends are *not* fsynced (a deliberate latency trade-off), so retries (100), `Dataset::refresh()`. clawhdf5 has no SWMR writer; remote
saves made since the last checkpoint can be lost on power failure or kernel SWMR is out of scope.
panic. Current header version is 4 (adds the `Update` record used by - **Browser** (`clawhdf5-wasm`, read-only, no Zstd/SZIP): `openUrl` reads
`save_or_update`); v3 files are read and upgraded in place. through the restartable "NeedBytes" cache (`src/lazy.rs`: a call is re-run
- A store has a **single writer**: `HDF5Memory::create`/`open` hold an exclusive after each wave of misses; no block is evicted while a call runs); the HTTP
advisory lock on `<store>.h5.lock` and a second opener gets is JavaScript (`js/remote.js`).
`MemoryError::Locked`. Use `HDF5Memory::open_read_only` for a lock-free, - **In-place editing** (`clawhdf5::FileEditor`): overwrites values, grows and
never-writing point-in-time view (the CLI's `recall`/`stats`/`agents-md`/ shrinks chunked datasets (every chunk index) and sets attributes (compact
`export` do). An unreadable WAL (torn header, bad magic) is quarantined to and dense) without rewriting the file, changing indexes and heaps as
`<store>.h5.wal.corrupt-<ts>` rather than blocking `open()`; a WAL with an libhdf5 does; freed space is reused within one editor. Anything it cannot
unknown *newer* version still fails and is left untouched. do safely is `Error::Unsupported` before any write (limits in
- `MemoryConfig::float16` (**on by default** for new stores, persisted; `docs/known-issues.md`). The algorithms follow libhdf5 `hdf5_1_14_6`
existing stores keep their recorded `false` — guarded by the v2.5.0 (github.com/HDFGroup/hdf5). Test changes with `cargo test -p
fixture in `tests/float16_store.rs`; CLI opt-out is `create --f32`) writes clawhdf5-tools --test edit_interop --test edit_coverage_interop`.
`/memory/embeddings` as IEEE half precision (48% smaller file at 100K; - **Provenance:** `Dataset::verify_provenance()` (facade `provenance`
LongMemEval with real MiniLM embeddings identical to f32). feature, default) re-hashes a dataset against its `_provenance_sha256`
`MemoryCache::half_precision` rounds each embedding as it enters the cache (push, update, WAL replay, and on load of a store still attribute (`DatasetBuilder::with_provenance`). Opt-in per call; unkeyed
`f32` on disk), so memory and file agree bit for bit; the conversions live hash — tamper-evident, not tamper-proof.
in `clawhdf5_format::float16` and must stay the single implementation.
Values beyond ±65504 are `MemoryError::InvalidEntry`. Interop: every file ## Agent memory: invariants and gotchas
must open in h5py — `f32` datasets and empty datasets did not until
2026-09-23 (see `docs/known-issues.md`); the agent's `h5py_interop` test - **Search.** `HDF5Memory::search(query_emb, text, &SearchOptions)` is the
guards a whole store. full path: optional source-channel filter (before ranking; exact scan of
- `HDF5Memory::search(query_emb, text, &SearchOptions)` is the full search the allowed records when cheaper than `pool × M` index distance
path: optional source-channel filter (applied before ranking; exact scan of
the allowed records whenever cheaper than `pool × M` index distance
evaluations, and as the fallback when the pool comes back short), fusion, evaluations, and as the fallback when the pool comes back short), fusion,
activation scaling, optional re-ranking and confidence rejection. activation scaling, optional re-ranking and confidence rejection.
`hybrid_search`/`hybrid_search_with` are thin wrappers; `ClawhdfBackend` `hybrid_search`/`hybrid_search_with` are thin wrappers. It keeps one
(the `openclaw` module) is `search` with re-rank + confidence on. incremental BM25 index for the life of the store and never writes the
- **OpenClaw is not supported** (decided 2026-09-25): clawhdf5 is not an store: Hebbian activation boosts are persisted by the next checkpoint (or
OpenClaw memory plugin and never was — the old `memory.backend = "clawhdf5"` on drop).
config was never valid. Don't reintroduce OpenClaw claims; `docs/openclaw.md` - **HNSW** (`hnsw` feature, default): the approximate `clawhdf5-ann` index
records what a real plugin would need. mirrors the cache and self-heals on drift; build the agent with
- **ZeroClaw does not use clawhdf5** (checked 2026-09-25 against upstream `--no-default-features --features float16` for the exact linear scan.
v0.8.5 and the `osobh/zeroclaw` fork, and their full history): no `parallel` (default) builds it on a thread pool with an identical graph.
`clawhdf5` feature or backend exists; ZeroClaw's memory backends are Neighbour selection uses the HNSW paper's diversity heuristic (closest-M
sqlite/lucid/postgres/qdrant/markdown/none behind its own `Memory` trait. capped recall at 0.31 recall@10 at 100K on clustered data). The graph is
`clawhdf5-migrate`'s default SQLite layout (`memory_chunks`, `sessions`, saved to `<store>.h5.ann` at each checkpoint, tied to it by a generation
`entities`, `relations`) is not ZeroClaw's schema either (ZeroClaw's is a id; a stale or damaged sidecar is ignored and the index rebuilt.
`memories` table). Don't reintroduce integration claims without an - **`MemoryConfig::quantized_index`** (default on for new stores, persisted;
integration and a test against the real consumer. Measure changes with older stores load as `false` — guarded by `tests/fixtures/store_v2_5_0.h5`;
`search_harness --options-study`. CLI `create --f32-index`): the index's copy of the embeddings is `i8`, and
- `MemoryConfig::compression` is off by default; when on, embeddings are the query path re-scores candidates against the exact embeddings. The
deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd). aarch64 kernels (`clawhdf5_accel::dot_i8`, NEON `SDOT` via inline asm) are
- Signed checkpoints (`clawhdf5-agent` `signing` module): with `cfg`'d out on x86, so x86 CI never compiles them — test on real ARM
`HDF5Memory::set_signing_key` every checkpoint stores an Ed25519-signed (`rpivision02`, 10.0.2.3, a Pi 5) or rely on the `test-arm64` job.
manifest (SHA-256 per record in a Merkle tree + settings/sessions/graph - **`MemoryConfig::float16`** (default on for new stores, persisted; older
hashes; per-record hashes in `/integrity/record_hashes`); stores keep `false` — guarded in `tests/float16_store.rs`; CLI `create
`HDF5Memory::verify(path, &pk)` locates edits. The hashes must cover exactly --f32`): `/memory/embeddings` is IEEE half. `MemoryCache::half_precision`
what the file persists in the form the loader returns it (strings lose rounds each embedding as it enters the cache (push, update, WAL replay, and
trailing NULs; an empty WAL mark is not written) or untouched stores stop load of a store still `f32` on disk) so memory and file agree bit for bit.
verifying — `tests/signed_store.rs` round-trips awkward strings. The key is Values beyond ±65504 are `MemoryError::InvalidEntry`. The agent's
never persisted; a signed store refuses to checkpoint without it `h5py_interop` test guards that a whole store opens in h5py.
(`MemoryError::SigningKeyRequired`, and `MemoryError` is `#[non_exhaustive]`). - **WAL.** Chained CRC32 per entry (a corrupted, reordered, duplicated or
spliced entry stops replay cleanly). Header version 4 (`Update` record for
`save_or_update`); v3 is upgraded in place, v2 read, v1 only through the
one-time migration in `HDF5Memory::open`. Each checkpoint records a
`WalMark` in `/meta` so `open()` never applies an entry twice; checkpoints
and snapshots are durable as a unit (temp file synced, renamed, directory
synced). Individual WAL appends are **not** fsynced (deliberate): saves
since the last checkpoint can be lost on power failure or kernel panic.
- **Single writer.** `create`/`open` hold an exclusive lock on
`<store>.h5.lock` (`MemoryError::Locked` for a second opener);
`open_read_only` is a lock-free point-in-time view (CLI `recall`/`stats`/
`agents-md`/`export`). An unreadable WAL is quarantined to
`<store>.h5.wal.corrupt-<ts>`; a WAL of an unknown newer version fails and
is left untouched.
- **Signed checkpoints** (`signing` module): with `set_signing_key` each
checkpoint stores an Ed25519-signed manifest (per-record SHA-256 in a
Merkle tree plus settings/sessions/graph hashes; `/integrity/record_hashes`);
`HDF5Memory::verify(path, &pk)` locates edits. The hashes must cover
exactly what the file persists in the form the loader returns it (strings
lose trailing NULs; an empty WAL mark is not written) —
`tests/signed_store.rs` round-trips awkward strings. The key is never
persisted; a signed store refuses to checkpoint without it
(`MemoryError::SigningKeyRequired`; `MemoryError` is `#[non_exhaustive]`).
WAL entries after the checkpoint are not covered. WAL entries after the checkpoint are not covered.
- `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by - **Write bookkeeping.** `save`/`save_batch`/`save_or_update` feed an
default) recomputes a dataset's SHA-256 and compares it against the in-memory, session-scoped provenance ledger and anomaly detector
`_provenance_sha256` attribute written automatically on save when (`provenance.rs`, `anomaly.rs`); alerts never block a save
`DatasetBuilder::with_provenance` is used. It's opt-in per call, not run (`take_anomaly_alerts`). `MemorySource` is inferred from the caller's
automatically on open — it decodes and hashes the whole dataset. The hash `source_channel` string — a heuristic, not a trust boundary.
is unkeyed (tamper-*evident*, not tamper-*proof*): it detects accidental - `MemoryConfig::compression` is off by default (deflate, or Zstd with the
corruption, not a deliberate actor able to modify both the data and the agent's `zstd` feature, which links libzstd).
stored hash.
- `clawhdf5-agent`'s `HDF5Memory::save`/`save_batch`/`save_or_update` run every
write through an in-memory (session-scoped, not persisted to disk)
provenance ledger and write-anomaly detector: a content hash per record
(`provenance.rs`) for detecting accidental mid-session corruption, plus
rate-limit/injection-pattern/source-distribution checks (`anomaly.rs`).
Alerts never block a save — drain them with `HDF5Memory::take_anomaly_alerts`.
`MemorySource` for this bookkeeping is inferred from the caller-supplied
`source_channel` string (a heuristic, not an authenticated trust boundary).
- In-place modification: `clawhdf5::FileEditor` (`crates/clawhdf5/src/edit/`)
overwrites values, grows and shrinks chunked datasets (every chunk index,
version-2 B-trees included) and sets attributes (compact and dense
storage) in existing files (h5py- or clawhdf5-written) without rewriting
them, changing indexes and heaps as libhdf5 does (index shapes and heap
bookkeeping are compared with libhdf5's in the tests); space an edit
frees is reused by later edits of the same editor. Anything it cannot do
safely is `Error::Unsupported` before any write (limits in
`docs/known-issues.md`). Test changes with
`cargo test -p clawhdf5-tools --test edit_interop --test
edit_coverage_interop` (h5py, h5dump, `h5rs check`, structure comparisons
with libhdf5; libhdf5 sources for the algorithms are at
github.com/HDFGroup/hdf5, tag `hdf5_1_14_6`).
- Remote files (`clawhdf5-remote`, range-read milestone M3 of
`docs/design/range-reads.md`): `open_url("http://…")` gives a
`clawhdf5::File` over `File::open_storage`, read through `BlockCache`
(1 MiB blocks, LRU byte budget, per-block in-flight dedup across threads,
runs coalesced into parallel requests). `HttpStorage` pins the file by
ETag/Last-Modified and length (a change is `RemoteError::FileChanged`),
refuses servers that ignore `Range` unless a full download is allowed,
and retries transient failures. `ObjectStoreStorage` (feature
`object-store`, pure Rust) runs each read on a small owned tokio
runtime and waits on a channel, so it works from any thread, including
inside `spawn_blocking` or another runtime. Default build is plain HTTP with
no C; `https` (rustls + ring) and `s3`/`gcs`/`azure` (aws-lc-rs) are
opt-in. Tests run a std-only HTTP server
(`tests/common/server.rs`, also the `range_server` example);
`CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus` compares every corpus
file over HTTP with `File::open`.
- GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only
- Browser: `clawhdf5-wasm` (wasm-bindgen, read-only; no Zstd/SZIP since
they link C) and the `examples/wasm-viewer/` page. `open(bytes)` holds
the file in memory; `openUrl(url)` (range-read M4) reads it by HTTP range
requests through the restartable "NeedBytes" cache (`src/lazy.rs`: a
call is re-run after each wave of misses; no block evicted while a call
runs), the HTTP in `js/remote.js`. `examples/wasm-viewer/test/run.sh`
builds the package (needs the `wasm-bindgen` CLI at the crate's exact
version) and tests it under Node and headless Chromium (a Playwright
download in `~/.cache/ms-playwright` on tank) against `test/serve.py`
(range server with request counts, 200 MB budget file); the CI container
has neither, so CI runs the native `h5py_interop` and `lazy` tests
(`CLAWHDF5_WASM_CORPUS=conformance/.cache/corpus` for the corpus). Size
numbers in the example's README predate `openUrl`.
- Python and Node.js bindings for cross-language use
- NetCDF-4 compatibility for scientific data interop
## Workflows ## Workflows
### Build Put `$HOME/.cargo/bin` on `PATH`. The h5py/netCDF4 interop tests find their
Python through `CLAWHDF5_PYTHON` (or `.venv/bin/python`); create it with
`python3 -m venv .venv && .venv/bin/pip install h5py numpy netCDF4 hdf5plugin`.
Set `CLAWHDF5_REQUIRE_INTEROP=1` to make a missing interpreter a failure.
```bash ```bash
cargo build --release cargo build --release
```
### Test
```bash
cargo test --workspace cargo test --workspace
bash scripts/ci-test.sh # everything CI runs (see below)
``` ```
### CI ### CI (`.gitea/workflows/`)
`.gitea/workflows/ci.yml` has two jobs, both green as of 2026-09-22: - **`ci.yml` `test`** (`ubuntu-latest`, `rust:latest` container; runners
- **`test`** (`ubuntu-latest`, in `rust:latest`) runs `scripts/ci-test.sh` with `tank`, `architect`): installs h5py/netCDF4/xarray/hdf5plugin/maturin/pytest,
the h5py/netCDF4 interop suites required (`CLAWHDF5_REQUIRE_INTEROP=1`). `hdf5-tools` and `cmake`, then runs `scripts/ci-test.sh` with
Served by the `tank` and `architect` runners. `CLAWHDF5_REQUIRE_INTEROP=1`. The script runs: fmt; clippy (workspace, the
- **`test-arm64`** (`linux_arm64`) lints and tests the aarch64 code — the NEON format feature matrix, each plugin filter alone, parallel, fast-deflate,
kernels are `cfg`'d out on x86, so this is the only place they are built. remote with all backends, h5rs remote); "no C in the default build";
Served by `vision-01` (host mode) and `vision-02` (Docker), so steps must wasm32 build and clippy; `check-32bit-casts.sh`; the wasm package under
work in both. Node when `node` and `wasm-bindgen` exist (not in CI); the MSRV check;
`cargo test` (workspace plus feature variants: format matrix, parallel,
remote/object_store, h5rs URLs, ann parallel, fast-deflate); the h5py
interop suites (`writer_h5py_tests --include-ignored`, plugin filters,
ZFP); the Python package (clippy, `maturin build`, pytest vs h5py);
`cargo bench --no-run`; `check-nostd.sh`; an optional fuzz smoke run
(`CLAWHDF5_FUZZ_SECONDS`).
- **`ci.yml` `test-arm64`** (`linux_arm64`; `vision-01` host mode,
`vision-02` Docker — steps must work in both): clippy and tests of
`clawhdf5-accel`, `-ann`, `-format`; the only place the NEON kernels build.
- **`conformance.yml`** (nightly 03:17 UTC and manual): probe unit tests,
`conformance/test_ref.py`, then `conformance/run.sh` (gate:
`conformance/check.py` against `baseline.json`).
Keep workflows free of JavaScript actions (`actions/checkout`, `actions/cache`, Keep workflows free of JavaScript actions (`actions/checkout`,
…): `rust:latest` has no `node`, and not every runner reaches GitHub, where `actions/cache`, …): `rust:latest` has no `node` and not every runner reaches
they are fetched from. Check out with plain `git` instead. The `test` job GitHub. Check out with plain `git`. Runners are `gitea-runner` 3.5.0 from
installs `cmake` for the opt-in `fast-deflate` (zlib-ng) steps; the default `docker.gitea.com/act_runner` (`gitea/act_runner:latest` on Docker Hub is
build needs no C toolchain, so `test-arm64` does not. frozen at 0.6.1).
All runners are on `gitea-runner` 3.5.0, from `docker.gitea.com/act_runner`
— `gitea/act_runner:latest` on Docker Hub is frozen at 0.6.1.
### CLI ### Conformance
```bash ```bash
cargo run -p clawhdf5-cli -- --help CLAWHDF5_PYTHON=.venv/bin/python bash conformance/run.sh --no-fetch # writes CONFORMANCE.md
# create, save, search, recall, stats, flush-wal, agents-md, export, snapshot subcommands
``` ```
Reads 697 files of eight pinned corpora with clawhdf5 and h5py and compares
them object by object (602 ok as of 2026-09-27). `CONFORMANCE.md` is
generated — never hand-edit it (its wording lives in `conformance/report.py`).
Use `--update-baseline` only after an intended change in results.
`CONFORMANCE_CACHE` points at an existing corpus cache (`conformance/.cache`,
about 450 MB). See `conformance/README.md`.
### HDF5 tools (`h5rs`, crate `clawhdf5-tools`) ### HDF5 tools (`h5rs`)
```bash ```bash
cargo run -p clawhdf5-tools -- ls -r file.h5 # also dump [--json], stat, diff, check cargo run -p clawhdf5-tools -- ls -r file.h5 # also dump [--json], stat, diff, check
bash scripts/h5rs-fuzz.sh # every subcommand over the CVE corpus: no panic/crash/hang bash scripts/h5rs-fuzz.sh # every subcommand over the CVE corpus: no panic/crash/hang
bash scripts/h5rs-check-ok-files.sh --data # check passes every fully-read conformance file bash scripts/h5rs-check-ok-files.sh --data # check passes every fully-read conformance file
``` ```
Its interop tests compare against h5ls/h5stat/h5dump/h5diff (Debian Interop tests compare against h5ls/h5stat/h5dump/h5diff (Debian `hdf5-tools`);
`hdf5-tools`, installed in CI); `dump` must stay byte-identical to h5dump on `dump` must stay byte-identical to h5dump on the test files.
the test files.
### Remote and browser tests
- `clawhdf5-remote` tests run a std-only HTTP server
(`tests/common/server.rs`, also the `range_server` example);
`CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus` compares every corpus
file over HTTP with `File::open`.
- wasm: `bash examples/wasm-viewer/test/run.sh` builds the package (needs the
`wasm-bindgen` CLI at the crate's exact version) and tests it under Node and
headless Chromium (Playwright's download in `~/.cache/ms-playwright` on
tank) against `test/serve.py` (range server with request counts). CI has
neither, so it runs the native `h5py_interop` and `lazy` tests
(`CLAWHDF5_WASM_CORPUS=conformance/.cache/corpus` for the corpus).
### Python bindings ### Python bindings
```bash ```bash
cd crates/clawhdf5-py cd crates/clawhdf5-py && maturin develop
maturin develop python -m pytest crates/clawhdf5-py/tests # compares with h5py; editing tests want CLAWHDF5_H5RS=<path to h5rs>
python -c "import clawhdf5; print(clawhdf5.__version__)" ```
### Benchmarks
- Search path: `cargo run --release -p clawhdf5-bench --bin search_harness`
(`--full`, `--options-study`, `--footprint`, …); reads: `read_harness`,
`concurrent_read`; criterion benches with `cargo bench -p <crate>`.
- Run on an idle machine (1-minute load average below 2; wait otherwise),
alternate base and candidate binaries for A/B comparisons, and record date,
machine, commit and command with every number in `BENCHMARKS.md`.
- `scripts/run-benchmarks.sh` is stale (it benchmarks `rustyhdf5-format` and
overwrites `BENCHMARKS.md`) — do not run it.
### CLI
```bash
cargo run -p clawhdf5-cli -- --help
# create, save, search, recall, stats, flush-wal, agents-md, export, snapshot
``` ```
## Integration ## Integration
@@ -255,8 +268,7 @@ python -c "import clawhdf5; print(clawhdf5.__version__)"
verified consumer: `cbh-core` reads and writes `.brain` files through the verified consumer: `cbh-core` reads and writes `.brain` files through the
facade (`File`, `FileBuilder`, `AttrValue`, `Selection`), `cbh-scanner` facade (`File`, `FileBuilder`, `AttrValue`, `Selection`), `cbh-scanner`
uses the facade, and `cbh-cli` uses `clawhdf5_agent::bm25::BM25Index`. It uses the facade, and `cbh-cli` uses `clawhdf5_agent::bm25::BM25Index`. It
depends on this repo by path (`../clawhdf5`), so it builds against whatever depends on this repo by path (`../clawhdf5`), so changes to those APIs
is checked out — changes to those APIs reach it directly. Verified reach it directly. Verified 2026-09-25 against main: builds, and its 204
2026-09-25 against main: builds, and its 204 tests pass. tests pass.
- OpenClaw and ZeroClaw were both described as consumers; neither integrates - OpenClaw and ZeroClaw integrate nothing (see *Standing rules*).
clawhdf5 (see Key Features and `docs/openclaw.md`).