# clawhdf5

## Purpose
Pure-Rust HDF5 implementation (read, write, in-place edit, remote and browser
reads) plus agent memory on top of it: HNSW vector search, a WAL-backed store,
and GPU vector distances. A standalone library. Its one verified consumer is
ClawBrainHub (`.brain` files); no agent framework integrates it (see
*Standing rules*).

## Architecture

Cargo workspace, 19 crates under `crates/` (plus `libaec-sys`, the FFI crate
behind the optional `szip` feature). MSRV 1.92 (`rust-version`, checked in CI).

| Crate | Role |
|-------|------|
| `clawhdf5-format` | The HDF5 format: parsers and writer (superblock, headers, B-trees, heaps, chunk indexes), the `Storage` trait, the filter pipeline and registry (`filter_registry`), every codec except deflate (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec; pure-Rust LZF, bitshuffle, bzip2, Blosc 1; Blosc2 and ZFP read-only), `float16`, `checksum` |
| `clawhdf5-filters` | Deflate backends (zlib-rs default, zlib-ng, Apple Compression) |
| `clawhdf5-io` | I/O adapters (buffers, mmap, prefetch) |
| `clawhdf5-derive` | `#[derive(H5Type)]` for compound types |
| `clawhdf5` | Facade: `File`, `FileBuilder`, `Dataset`, `FileEditor` (`src/edit/`), SWMR reading (`src/swmr.rs`) |
| `clawhdf5-netcdf4` | NetCDF-4 read support |
| `clawhdf5-remote` | `open_url`: HTTP(S) range requests and object stores (S3, GCS, Azure) through `BlockCache` |
| `clawhdf5-tools` | `h5rs`: `ls`, `dump` (DDL / hdf5-json), `stat`, `diff`, `check` |
| `clawhdf5-py` | PyO3 bindings (h5py-like API, remote files, `'r+'` editing) |
| `clawhdf5-wasm` | wasm-bindgen browser reader (`open(bytes)`, `openUrl(url)`); demo in `examples/wasm-viewer/` |
| `clawhdf5-ann` | HNSW index |
| `clawhdf5-agent` | Agent memory store (`HDF5Memory`), sessions, knowledge graph, BM25 |
| `clawhdf5-accel` | CPU SIMD kernels (AVX2, NEON) |
| `clawhdf5-gpu` | wgpu vector distances (WGSL) — not dataset I/O; HDF5 I/O is CPU-only |
| `clawhdf5-migrate` | SQLite → agent store migration |
| `clawhdf5-cli` | Agent-memory CLI |
| `clawhdf5-napi` | Node.js addon (the `packages/clawhdf5-node` wrapper is broken; `docs/known-issues.md`) |
| `clawhdf5-android` | Android JNI bindings |
| `clawhdf5-bench` | Benchmarks and harnesses (`search_harness`, `read_harness`, `concurrent_read`, `longmemeval_bench`, …) |

Reference docs: `docs/known-issues.md` (open issues table first — check it
before calling something a bug or a feature), `BENCHMARKS.md` (headline
numbers first), `CONFORMANCE.md` (generated), `docs/design/range-reads.md`
and `docs/design/swmr.md`, `CHANGELOG.md` (full detail of every fix).

## Standing rules

- **No C in the default build.** No libhdf5; deflate defaults to pure-Rust
  zlib-rs (`fast-deflate` opts into zlib-ng, which needs cmake). `ci-test.sh`
  fails if a C-building crate enters the core crates' default tree. Zstd,
  SZIP, `https` (ring) and `s3`/`gcs`/`azure` (aws-lc-rs) are opt-in. flate2
  must keep `runtime_detection` with zlib-rs — without it zlib-rs loses SIMD
  and inflates 3.5x slower.
- **Every file we write must open in h5py/libhdf5.** Interop tests compare
  against h5py and h5dump; `f32` and empty datasets did not open until
  2026-09-23.
- **float16 has one implementation:** `clawhdf5_format::float16`.
- **Claims need evidence.** Performance and integration claims in docs must
  be measured, dated (with machine and command), or withdrawn. Benchmark
  numbers are dated records: never edit a measured value, add a new dated
  section and mark the old one superseded.
- **OpenClaw is not supported** (decided 2026-09-25): clawhdf5 is not and
  never was an OpenClaw memory plugin; the old `memory.backend = "clawhdf5"`
  config was never valid. `docs/openclaw.md` records what a real plugin would
  need. The `openclaw` module's `ClawhdfBackend` is just `search` with
  re-rank + confidence on.
- **ZeroClaw does not use clawhdf5** (checked 2026-09-25 against upstream
  v0.8.5 and the `osobh/zeroclaw` fork and their history): its memory
  backends are its own; `clawhdf5-migrate`'s default SQLite layout is not
  ZeroClaw's schema. Don't reintroduce integration claims without an
  integration and a test against the real consumer.
- **known-issues.md:** one entry per bug; when fixed, record it in
  `CHANGELOG.md` and move the entry to *Fixed (history)* with date, PR,
  affected releases and what users must do — never delete it.

## HDF5 library: invariants and gotchas

- **Remote/range reads** (`docs/design/range-reads.md`, M0-M5 merged in PRs
  #17-#19, M4 listing costs cut in #21): every format-crate read path goes through `Storage`
  (`read_at`/`read_ranges`/`hint`). `File::open_storage` takes any
  `Storage`; `clawhdf5_remote::open_url` wraps HTTP (`HttpStorage`, ureq) or
  `ObjectStoreStorage` in `BlockCache` (1 MiB blocks, LRU budget, in-flight
  dedup, coalesced runs). Remote files are pinned by ETag/Last-Modified and
  length (`RemoteError::FileChanged`). Zero-copy APIs and `File::as_bytes`
  need an in-memory file. Parse through `File::storage()` and the `*_in`
  functions, not `as_bytes`, in new code (the Python bindings do).
  `ObjectStoreStorage` runs reads on its own small tokio runtime, so it
  works from any thread.
- **SWMR** (`docs/design/swmr.md`): `File::open_swmr` reads a file a libhdf5
  SWMR writer is appending to — positioned reads, no chunk cache, bounded
  retries (100), `Dataset::refresh()`. clawhdf5 has no SWMR writer; remote
  SWMR is out of scope.
- **Browser** (`clawhdf5-wasm`, read-only, no Zstd/SZIP): `openUrl` reads
  through the restartable "NeedBytes" cache (`src/lazy.rs`: a call is re-run
  after each wave of misses; no block is evicted while a call runs); the HTTP
  is JavaScript (`js/remote.js`).
- **In-place editing** (`clawhdf5::FileEditor`): overwrites values, grows and
  shrinks chunked datasets (every chunk index) and sets attributes (compact
  and dense) without rewriting the file, changing indexes and heaps as
  libhdf5 does; freed space is reused within one editor. Anything it cannot
  do safely is `Error::Unsupported` before any write (limits in
  `docs/known-issues.md`). The algorithms follow libhdf5 `hdf5_1_14_6`
  (github.com/HDFGroup/hdf5). Test changes with `cargo test -p
  clawhdf5-tools --test edit_interop --test edit_coverage_interop`.
- **Provenance:** `Dataset::verify_provenance()` (facade `provenance`
  feature, default) re-hashes a dataset against its `_provenance_sha256`
  attribute (`DatasetBuilder::with_provenance`). Opt-in per call; unkeyed
  hash — tamper-evident, not tamper-proof.

## Agent memory: invariants and gotchas

- **Search.** `HDF5Memory::search(query_emb, text, &SearchOptions)` is the
  full path: optional source-channel filter (before ranking; exact scan of
  the allowed records when cheaper than `pool × M` index distance
  evaluations, and as the fallback when the pool comes back short), fusion,
  activation scaling, optional re-ranking and confidence rejection.
  `hybrid_search`/`hybrid_search_with` are thin wrappers. It keeps one
  incremental BM25 index for the life of the store and never writes the
  store: Hebbian activation boosts are persisted by the next checkpoint (or
  on drop).
- **HNSW** (`hnsw` feature, default): the approximate `clawhdf5-ann` index
  mirrors the cache and self-heals on drift; build the agent with
  `--no-default-features --features float16` for the exact linear scan.
  `parallel` (default) builds it on a thread pool with an identical graph.
  Neighbour selection uses the HNSW paper's diversity heuristic (closest-M
  capped recall at 0.31 recall@10 at 100K on clustered data). The graph is
  saved to `<store>.h5.ann` at each checkpoint, tied to it by a generation
  id; a stale or damaged sidecar is ignored and the index rebuilt.
- **`MemoryConfig::quantized_index`** (default on for new stores, persisted;
  older stores load as `false` — guarded by `tests/fixtures/store_v2_5_0.h5`;
  CLI `create --f32-index`): the index's copy of the embeddings is `i8`, and
  the query path re-scores candidates against the exact embeddings. The
  aarch64 kernels (`clawhdf5_accel::dot_i8`, NEON `SDOT` via inline asm) are
  `cfg`'d out on x86, so x86 CI never compiles them — test on real ARM
  (`rpivision02`, 10.0.2.3, a Pi 5) or rely on the `test-arm64` job.
- **`MemoryConfig::float16`** (default on for new stores, persisted; older
  stores keep `false` — guarded in `tests/float16_store.rs`; CLI `create
  --f32`): `/memory/embeddings` is IEEE half. `MemoryCache::half_precision`
  rounds each embedding as it enters the cache (push, update, WAL replay, and
  load of a store still `f32` on disk) so memory and file agree bit for bit.
  Values beyond ±65504 are `MemoryError::InvalidEntry`. The agent's
  `h5py_interop` test guards that a whole store opens in h5py.
- **WAL.** Chained CRC32 per entry (a corrupted, reordered, duplicated or
  spliced entry stops replay cleanly). Header version 4 (`Update` record for
  `save_or_update`); v3 is upgraded in place, v2 read, v1 only through the
  one-time migration in `HDF5Memory::open`. Each checkpoint records a
  `WalMark` in `/meta` so `open()` never applies an entry twice; checkpoints
  and snapshots are durable as a unit (temp file synced, renamed, directory
  synced). Individual WAL appends are **not** fsynced (deliberate): saves
  since the last checkpoint can be lost on power failure or kernel panic.
- **Single writer.** `create`/`open` hold an exclusive lock on
  `<store>.h5.lock` (`MemoryError::Locked` for a second opener);
  `open_read_only` is a lock-free point-in-time view (CLI `recall`/`stats`/
  `agents-md`/`export`). An unreadable WAL is quarantined to
  `<store>.h5.wal.corrupt-<ts>`; a WAL of an unknown newer version fails and
  is left untouched.
- **Signed checkpoints** (`signing` module): with `set_signing_key` each
  checkpoint stores an Ed25519-signed manifest (per-record SHA-256 in a
  Merkle tree plus settings/sessions/graph hashes; `/integrity/record_hashes`);
  `HDF5Memory::verify(path, &pk)` locates edits. The hashes must cover
  exactly what the file persists in the form the loader returns it (strings
  lose trailing NULs; an empty WAL mark is not written) —
  `tests/signed_store.rs` round-trips awkward strings. The key is never
  persisted; a signed store refuses to checkpoint without it
  (`MemoryError::SigningKeyRequired`; `MemoryError` is `#[non_exhaustive]`).
  WAL entries after the checkpoint are not covered.
- **Write bookkeeping.** `save`/`save_batch`/`save_or_update` feed an
  in-memory, session-scoped provenance ledger and anomaly detector
  (`provenance.rs`, `anomaly.rs`); alerts never block a save
  (`take_anomaly_alerts`). `MemorySource` is inferred from the caller's
  `source_channel` string — a heuristic, not a trust boundary.
- `MemoryConfig::compression` is off by default (deflate, or Zstd with the
  agent's `zstd` feature, which links libzstd).

## Workflows

Put `$HOME/.cargo/bin` on `PATH`. The h5py/netCDF4 interop tests find their
Python through `CLAWHDF5_PYTHON` (or `.venv/bin/python`); create it with
`python3 -m venv .venv && .venv/bin/pip install h5py numpy netCDF4 hdf5plugin`.
Set `CLAWHDF5_REQUIRE_INTEROP=1` to make a missing interpreter a failure.

```bash
cargo build --release
cargo test --workspace
bash scripts/ci-test.sh          # everything CI runs (see below)
```

### CI (`.gitea/workflows/`)
- **`ci.yml` `test`** (`ubuntu-latest`, `rust:latest` container; runners
  `tank`, `architect`): installs h5py/netCDF4/xarray/hdf5plugin/maturin/pytest,
  `hdf5-tools` and `cmake`, then runs `scripts/ci-test.sh` with
  `CLAWHDF5_REQUIRE_INTEROP=1`. The script runs: fmt; clippy (workspace, the
  format feature matrix, each plugin filter alone, parallel, fast-deflate,
  remote with all backends, h5rs remote); "no C in the default build";
  wasm32 build and clippy; `check-32bit-casts.sh`; the wasm package under
  Node when `node` and `wasm-bindgen` exist (not in CI); the MSRV check;
  `cargo test` (workspace plus feature variants: format matrix, parallel,
  remote/object_store, h5rs URLs, ann parallel, fast-deflate); the h5py
  interop suites (`writer_h5py_tests --include-ignored`, plugin filters,
  ZFP); the Python package (clippy, `maturin build`, pytest vs h5py);
  `cargo bench --no-run`; `check-nostd.sh`; an optional fuzz smoke run
  (`CLAWHDF5_FUZZ_SECONDS`).
- **`ci.yml` `test-arm64`** (`linux_arm64`; `vision-01` host mode,
  `vision-02` Docker — steps must work in both): clippy of
  `clawhdf5-accel`, tests of `-accel`, `-ann`, `-format`; the only place the NEON kernels build.
- **`conformance.yml`** (nightly 03:17 UTC and manual): probe unit tests,
  `conformance/test_ref.py`, then `conformance/run.sh` (gate:
  `conformance/check.py` against `baseline.json`).

Keep workflows free of JavaScript actions (`actions/checkout`,
`actions/cache`, …): `rust:latest` has no `node` and not every runner reaches
GitHub. Check out with plain `git`. Runners are `gitea-runner` 3.5.0 from
`docker.gitea.com/act_runner` (`gitea/act_runner:latest` on Docker Hub is
frozen at 0.6.1).

### Conformance
```bash
CLAWHDF5_PYTHON=.venv/bin/python bash conformance/run.sh --no-fetch   # writes CONFORMANCE.md
```
Reads 697 files of eight pinned corpora with clawhdf5 and h5py and compares
them object by object (602 ok in the run of 2026-09-28). `CONFORMANCE.md` is
generated — never hand-edit it (its wording lives in `conformance/report.py`).
Use `--update-baseline` only after an intended change in results.
`CONFORMANCE_CACHE` points at an existing corpus cache (`conformance/.cache`,
about 450 MB). See `conformance/README.md`.

### HDF5 tools (`h5rs`)
```bash
cargo run -p clawhdf5-tools -- ls -r file.h5    # also dump [--json], stat, diff, check
bash scripts/h5rs-fuzz.sh                       # every subcommand over the CVE corpus: no panic/crash/hang
bash scripts/h5rs-check-ok-files.sh --data      # check passes every fully-read conformance file
```
Interop tests compare against h5ls/h5stat/h5dump/h5diff (Debian `hdf5-tools`);
`dump` must stay byte-identical to h5dump on the test files.

### Remote and browser tests
- `clawhdf5-remote` tests run a std-only HTTP server
  (`tests/common/server.rs`, also the `range_server` example);
  `CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus` compares every corpus
  file over HTTP with `File::open`.
- wasm: `bash examples/wasm-viewer/test/run.sh` builds the package (needs the
  `wasm-bindgen` CLI at the crate's exact version) and tests it under Node and
  headless Chromium (Playwright's download in `~/.cache/ms-playwright` on
  tank) against `test/serve.py` (range server with request counts). CI has
  neither, so it runs the native `h5py_interop` and `lazy` tests
  (`CLAWHDF5_WASM_CORPUS=conformance/.cache/corpus` for the corpus).

### Python bindings
```bash
cd crates/clawhdf5-py && maturin develop
python -m pytest crates/clawhdf5-py/tests      # compares with h5py; editing tests want CLAWHDF5_H5RS=<path to h5rs>
```

### Benchmarks
- Search path: `cargo run --release -p clawhdf5-bench --bin search_harness`
  (`--full`, `--options-study`, `--footprint`, …); reads: `read_harness`,
  `concurrent_read`; criterion benches with `cargo bench -p <crate>`.
- Run on an idle machine (1-minute load average below 2; wait otherwise),
  alternate base and candidate binaries for A/B comparisons, and record date,
  machine, commit and command with every number in `BENCHMARKS.md`.
- `BENCHMARKS.md` is written by hand from dated runs; no script regenerates
  it (the old `scripts/run-benchmarks.sh`, which benchmarked the pre-rename
  `rustyhdf5-format` and overwrote the file, was removed on 2026-09-28).

### CLI
```bash
cargo run -p clawhdf5-cli -- --help
# create, save, search, recall, stats, flush-wal, agents-md, export, snapshot, keygen, verify
```

## Integration
- **ClawBrainHub** (`clawverse/clawbrainhub` on git.redclaw.dev) is the one
  verified consumer: `cbh-core` reads and writes `.brain` files through the
  facade (`File`, `FileBuilder`, `AttrValue`, `Selection`), `cbh-scanner`
  uses the facade, and `cbh-cli` uses `clawhdf5_agent::bm25::BM25Index`. It
  depends on this repo by path (`../clawhdf5`), so changes to those APIs
  reach it directly. Verified 2026-09-25 against main: builds, and its 204
  tests pass.
- OpenClaw and ZeroClaw integrate nothing (see *Standing rules*).