clawhdf5/CLAUDE.md

clawhdf5

Purpose

Pure-Rust HDF5 implementation (read, write, in-place edit, remote and browser reads) plus agent memory on top of it: HNSW vector search, a WAL-backed store, and GPU vector distances. A standalone library. Its one verified consumer is ClawBrainHub (.brain files); no agent framework integrates it (see Standing rules).

Architecture

Cargo workspace, 19 crates under crates/ (plus libaec-sys, the FFI crate behind the optional szip feature). MSRV 1.92 (rust-version, checked in CI).

Crate	Role
clawhdf5-format	The HDF5 format: parsers and writer (superblock, headers, B-trees, heaps, chunk indexes), the Storage trait, the filter pipeline and registry (filter_registry), every codec except deflate (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec; pure-Rust LZF, bitshuffle, bzip2, Blosc 1; Blosc2 and ZFP read-only), float16, checksum
clawhdf5-filters	Deflate backends (zlib-rs default, zlib-ng, Apple Compression)
clawhdf5-io	I/O adapters (buffers, mmap, prefetch)
clawhdf5-derive	#[derive(H5Type)] for compound types
clawhdf5	Facade: File, FileBuilder, Dataset, FileEditor (src/edit/), SWMR reading (src/swmr.rs)
clawhdf5-netcdf4	NetCDF-4 read support
clawhdf5-remote	open_url: HTTP(S) range requests and object stores (S3, GCS, Azure) through BlockCache
clawhdf5-tools	h5rs: ls, dump (DDL / hdf5-json), stat, diff, check
clawhdf5-py	PyO3 bindings (h5py-like API, remote files, 'r+' editing)
clawhdf5-wasm	wasm-bindgen browser reader (open(bytes), openUrl(url)); demo in examples/wasm-viewer/
clawhdf5-ann	HNSW index
clawhdf5-agent	Agent memory store (HDF5Memory), sessions, knowledge graph, BM25
clawhdf5-accel	CPU SIMD kernels (AVX2, NEON)
clawhdf5-gpu	wgpu vector distances (WGSL) — not dataset I/O; HDF5 I/O is CPU-only
clawhdf5-migrate	SQLite → agent store migration
clawhdf5-cli	Agent-memory CLI
clawhdf5-napi	Node.js addon (the packages/clawhdf5-node wrapper is broken; docs/known-issues.md)
clawhdf5-android	Android JNI bindings
clawhdf5-bench	Benchmarks and harnesses (search_harness, read_harness, concurrent_read, longmemeval_bench, …)

Reference docs: docs/known-issues.md (open issues table first — check it before calling something a bug or a feature), BENCHMARKS.md (headline numbers first), CONFORMANCE.md (generated), docs/design/range-reads.md and docs/design/swmr.md, CHANGELOG.md (full detail of every fix).

Standing rules

• No C in the default build. No libhdf5; deflate defaults to pure-Rust zlib-rs (fast-deflate opts into zlib-ng, which needs cmake). ci-test.sh fails if a C-building crate enters the core crates' default tree. Zstd, SZIP, https (ring) and s3/gcs/azure (aws-lc-rs) are opt-in. flate2 must keep runtime_detection with zlib-rs — without it zlib-rs loses SIMD and inflates 3.5x slower.
• Every file we write must open in h5py/libhdf5. Interop tests compare against h5py and h5dump; f32 and empty datasets did not open until 2026-09-23.
• float16 has one implementation: clawhdf5_format::float16.
• Claims need evidence. Performance and integration claims in docs must be measured, dated (with machine and command), or withdrawn. Benchmark numbers are dated records: never edit a measured value, add a new dated section and mark the old one superseded.
• OpenClaw is not supported (decided 2026-09-25): clawhdf5 is not and never was an OpenClaw memory plugin; the old memory.backend = "clawhdf5" config was never valid. docs/openclaw.md records what a real plugin would need. The openclaw module's ClawhdfBackend is just search with re-rank + confidence on.
• ZeroClaw does not use clawhdf5 (checked 2026-09-25 against upstream v0.8.5 and the osobh/zeroclaw fork and their history): its memory backends are its own; clawhdf5-migrate's default SQLite layout is not ZeroClaw's schema. Don't reintroduce integration claims without an integration and a test against the real consumer.
• known-issues.md: one entry per bug; when fixed, record it in CHANGELOG.md and move the entry to Fixed (history) with date, PR, affected releases and what users must do — never delete it.

HDF5 library: invariants and gotchas

• Remote/range reads (docs/design/range-reads.md, M0-M5 merged in PRs #17-#19, M4 listing costs cut in #21): every format-crate read path goes through Storage (read_at/read_ranges/hint). File::open_storage takes any Storage; clawhdf5_remote::open_url wraps HTTP (HttpStorage, ureq) or ObjectStoreStorage in BlockCache (1 MiB blocks, LRU budget, in-flight dedup, coalesced runs). Remote files are pinned by ETag/Last-Modified and length (RemoteError::FileChanged). Zero-copy APIs and File::as_bytes need an in-memory file. Parse through File::storage() and the *_in functions, not as_bytes, in new code (the Python bindings do). ObjectStoreStorage runs reads on its own small tokio runtime, so it works from any thread.
• SWMR (docs/design/swmr.md): File::open_swmr reads a file a libhdf5 SWMR writer is appending to — positioned reads, no chunk cache, bounded retries (100), Dataset::refresh(). clawhdf5 has no SWMR writer; remote SWMR is out of scope.
• Browser (clawhdf5-wasm, read-only, no Zstd/SZIP): openUrl reads through the restartable "NeedBytes" cache (src/lazy.rs: a call is re-run after each wave of misses; no block is evicted while a call runs); the HTTP is JavaScript (js/remote.js).
• In-place editing (clawhdf5::FileEditor): overwrites values, grows and shrinks chunked datasets (every chunk index) and sets attributes (compact and dense) without rewriting the file, changing indexes and heaps as libhdf5 does; freed space is reused within one editor. Anything it cannot do safely is Error::Unsupported before any write (limits in docs/known-issues.md). The algorithms follow libhdf5 hdf5_1_14_6 (github.com/HDFGroup/hdf5). Test changes with cargo test -p clawhdf5-tools --test edit_interop --test edit_coverage_interop.
• Provenance: Dataset::verify_provenance() (facade provenance feature, default) re-hashes a dataset against its _provenance_sha256 attribute (DatasetBuilder::with_provenance). Opt-in per call; unkeyed hash — tamper-evident, not tamper-proof.

Agent memory: invariants and gotchas

• Search. HDF5Memory::search(query_emb, text, &SearchOptions) is the full path: optional source-channel filter (before ranking; exact scan of the allowed records when cheaper than pool × M index distance evaluations, and as the fallback when the pool comes back short), fusion, activation scaling, optional re-ranking and confidence rejection. hybrid_search/hybrid_search_with are thin wrappers. It keeps one incremental BM25 index for the life of the store and never writes the store: Hebbian activation boosts are persisted by the next checkpoint (or on drop).
• HNSW (hnsw feature, default): the approximate clawhdf5-ann index mirrors the cache and self-heals on drift; build the agent with --no-default-features --features float16 for the exact linear scan. parallel (default) builds it on a thread pool with an identical graph. Neighbour selection uses the HNSW paper's diversity heuristic (closest-M capped recall at 0.31 recall@10 at 100K on clustered data). The graph is saved to <store>.h5.ann at each checkpoint, tied to it by a generation id; a stale or damaged sidecar is ignored and the index rebuilt.
• MemoryConfig::quantized_index (default on for new stores, persisted; older stores load as false — guarded by tests/fixtures/store_v2_5_0.h5; CLI create --f32-index): the index's copy of the embeddings is i8, and the query path re-scores candidates against the exact embeddings. The aarch64 kernels (clawhdf5_accel::dot_i8, NEON SDOT via inline asm) are cfg'd out on x86, so x86 CI never compiles them — test on real ARM (rpivision02, 10.0.2.3, a Pi 5) or rely on the test-arm64 job.
• MemoryConfig::float16 (default on for new stores, persisted; older stores keep false — guarded in tests/float16_store.rs; CLI create --f32): /memory/embeddings is IEEE half. MemoryCache::half_precision rounds each embedding as it enters the cache (push, update, WAL replay, and load of a store still f32 on disk) so memory and file agree bit for bit. Values beyond ±65504 are MemoryError::InvalidEntry. The agent's h5py_interop test guards that a whole store opens in h5py.
• WAL. Chained CRC32 per entry (a corrupted, reordered, duplicated or spliced entry stops replay cleanly). Header version 4 (Update record for save_or_update); v3 is upgraded in place, v2 read, v1 only through the one-time migration in HDF5Memory::open. Each checkpoint records a WalMark in /meta so open() never applies an entry twice; checkpoints and snapshots are durable as a unit (temp file synced, renamed, directory synced). Individual WAL appends are not fsynced (deliberate): saves since the last checkpoint can be lost on power failure or kernel panic.
• Single writer. create/open hold an exclusive lock on <store>.h5.lock (MemoryError::Locked for a second opener); open_read_only is a lock-free point-in-time view (CLI recall/stats/ agents-md/export). An unreadable WAL is quarantined to <store>.h5.wal.corrupt-<ts>; a WAL of an unknown newer version fails and is left untouched.
• Signed checkpoints (signing module): with set_signing_key each checkpoint stores an Ed25519-signed manifest (per-record SHA-256 in a Merkle tree plus settings/sessions/graph hashes; /integrity/record_hashes); HDF5Memory::verify(path, &pk) locates edits. The hashes must cover exactly what the file persists in the form the loader returns it (strings lose trailing NULs; an empty WAL mark is not written) — tests/signed_store.rs round-trips awkward strings. The key is never persisted; a signed store refuses to checkpoint without it (MemoryError::SigningKeyRequired; MemoryError is #[non_exhaustive]). WAL entries after the checkpoint are not covered.
• Write bookkeeping. save/save_batch/save_or_update feed an in-memory, session-scoped provenance ledger and anomaly detector (provenance.rs, anomaly.rs); alerts never block a save (take_anomaly_alerts). MemorySource is inferred from the caller's source_channel string — a heuristic, not a trust boundary.
• MemoryConfig::compression is off by default (deflate, or Zstd with the agent's zstd feature, which links libzstd).

Workflows

Put $HOME/.cargo/bin on PATH. The h5py/netCDF4 interop tests find their Python through CLAWHDF5_PYTHON (or .venv/bin/python); create it with python3 -m venv .venv && .venv/bin/pip install h5py numpy netCDF4 hdf5plugin. Set CLAWHDF5_REQUIRE_INTEROP=1 to make a missing interpreter a failure.

cargo build --release
cargo test --workspace
bash scripts/ci-test.sh          # everything CI runs (see below)

CI (.gitea/workflows/)

• ci.yml test (ubuntu-latest, rust:latest container; runners tank, architect): installs h5py/netCDF4/xarray/hdf5plugin/maturin/pytest, hdf5-tools and cmake, then runs scripts/ci-test.sh with CLAWHDF5_REQUIRE_INTEROP=1. The script runs: fmt; clippy (workspace, the format feature matrix, each plugin filter alone, parallel, fast-deflate, remote with all backends, h5rs remote); "no C in the default build"; wasm32 build and clippy; check-32bit-casts.sh; the wasm package under Node when node and wasm-bindgen exist (not in CI); the MSRV check; cargo test (workspace plus feature variants: format matrix, parallel, remote/object_store, h5rs URLs, ann parallel, fast-deflate); the h5py interop suites (writer_h5py_tests --include-ignored, plugin filters, ZFP); the Python package (clippy, maturin build, pytest vs h5py); cargo bench --no-run; check-nostd.sh; an optional fuzz smoke run (CLAWHDF5_FUZZ_SECONDS).
• ci.yml test-arm64 (linux_arm64; vision-01 host mode, vision-02 Docker — steps must work in both): clippy of clawhdf5-accel, tests of -accel, -ann, -format; the only place the NEON kernels build.
• conformance.yml (nightly 03:17 UTC and manual): probe unit tests, conformance/test_ref.py, then conformance/run.sh (gate: conformance/check.py against baseline.json).

Keep workflows free of JavaScript actions (actions/checkout, actions/cache, …): rust:latest has no node and not every runner reaches GitHub. Check out with plain git. Runners are gitea-runner 3.5.0 from docker.gitea.com/act_runner (gitea/act_runner:latest on Docker Hub is frozen at 0.6.1).

Conformance

CLAWHDF5_PYTHON=.venv/bin/python bash conformance/run.sh --no-fetch   # writes CONFORMANCE.md

Reads 697 files of eight pinned corpora with clawhdf5 and h5py and compares them object by object (602 ok in the run of 2026-09-28). CONFORMANCE.md is generated — never hand-edit it (its wording lives in conformance/report.py). Use --update-baseline only after an intended change in results. CONFORMANCE_CACHE points at an existing corpus cache (conformance/.cache, about 450 MB). See conformance/README.md.

HDF5 tools (h5rs)

cargo run -p clawhdf5-tools -- ls -r file.h5    # also dump [--json], stat, diff, check
bash scripts/h5rs-fuzz.sh                       # every subcommand over the CVE corpus: no panic/crash/hang
bash scripts/h5rs-check-ok-files.sh --data      # check passes every fully-read conformance file

Interop tests compare against h5ls/h5stat/h5dump/h5diff (Debian hdf5-tools); dump must stay byte-identical to h5dump on the test files.

Remote and browser tests

• clawhdf5-remote tests run a std-only HTTP server (tests/common/server.rs, also the range_server example); CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus compares every corpus file over HTTP with File::open.
• wasm: bash examples/wasm-viewer/test/run.sh builds the package (needs the wasm-bindgen CLI at the crate's exact version) and tests it under Node and headless Chromium (Playwright's download in ~/.cache/ms-playwright on tank) against test/serve.py (range server with request counts). CI has neither, so it runs the native h5py_interop and lazy tests (CLAWHDF5_WASM_CORPUS=conformance/.cache/corpus for the corpus).

Python bindings

cd crates/clawhdf5-py && maturin develop
python -m pytest crates/clawhdf5-py/tests      # compares with h5py; editing tests want CLAWHDF5_H5RS=<path to h5rs>

Benchmarks

• Search path: cargo run --release -p clawhdf5-bench --bin search_harness (--full, --options-study, --footprint, …); reads: read_harness, concurrent_read; criterion benches with cargo bench -p <crate>.
• Run on an idle machine (1-minute load average below 2; wait otherwise), alternate base and candidate binaries for A/B comparisons, and record date, machine, commit and command with every number in BENCHMARKS.md.
• BENCHMARKS.md is written by hand from dated runs; no script regenerates it (the old scripts/run-benchmarks.sh, which benchmarked the pre-rename rustyhdf5-format and overwrote the file, was removed on 2026-09-28).

CLI

cargo run -p clawhdf5-cli -- --help
# create, save, search, recall, stats, flush-wal, agents-md, export, snapshot, keygen, verify

Integration

• ClawBrainHub (clawverse/clawbrainhub on git.redclaw.dev) is the one verified consumer: cbh-core reads and writes .brain files through the facade (File, FileBuilder, AttrValue, Selection), cbh-scanner uses the facade, and cbh-cli uses clawhdf5_agent::bm25::BM25Index. It depends on this repo by path (../clawhdf5), so changes to those APIs reach it directly. Verified 2026-09-25 against main: builds, and its 204 tests pass.
• OpenClaw and ZeroClaw integrate nothing (see Standing rules).