diff --git a/CLAUDE.md b/CLAUDE.md index 8c95cc5..cc90aed 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,253 +1,266 @@ # clawhdf5 ## Purpose -Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated vector search. A standalone library. Its one verified consumer is ClawBrainHub (`.brain` files); no agent framework integrates it (OpenClaw and ZeroClaw claims were withdrawn on 2026-09-25 — neither was ever true). +Pure-Rust HDF5 implementation (read, write, in-place edit, remote and browser +reads) plus agent memory on top of it: HNSW vector search, a WAL-backed store, +and GPU vector distances. A standalone library. Its one verified consumer is +ClawBrainHub (`.brain` files); no agent framework integrates it (see +*Standing rules*). ## Architecture -Cargo workspace with 19 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature): +Cargo workspace, 19 crates under `crates/` (plus `libaec-sys`, the FFI crate +behind the optional `szip` feature). MSRV 1.92 (`rust-version`, checked in CI). | Crate | Role | |-------|------| -| `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants | -| `clawhdf5-io` | Read/write implementation | -| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 and ZFP read-only) live in `clawhdf5-format`. | -| `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs | -| `clawhdf5` | Main facade crate | -| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer | -| `clawhdf5-ann` | HNSW approximate nearest-neighbor vector index | -| `clawhdf5-agent` | Agent memory, session history, knowledge graph storage | -| `clawhdf5-gpu` | GPU vector distance computation via wgpu (hand-written WGSL compute shaders) — not dataset I/O | -| `clawhdf5-accel` | CPU SIMD acceleration path | -| `clawhdf5-migrate` | SQLite → HDF5 agent-memory migration | +| `clawhdf5-format` | The HDF5 format: parsers and writer (superblock, headers, B-trees, heaps, chunk indexes), the `Storage` trait, the filter pipeline and registry (`filter_registry`), every codec except deflate (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec; pure-Rust LZF, bitshuffle, bzip2, Blosc 1; Blosc2 and ZFP read-only), `float16`, `checksum` | +| `clawhdf5-filters` | Deflate backends (zlib-rs default, zlib-ng, Apple Compression) | +| `clawhdf5-io` | I/O adapters (buffers, mmap, prefetch) | +| `clawhdf5-derive` | `#[derive(H5Type)]` for compound types | +| `clawhdf5` | Facade: `File`, `FileBuilder`, `Dataset`, `FileEditor` (`src/edit/`), SWMR reading (`src/swmr.rs`) | +| `clawhdf5-netcdf4` | NetCDF-4 read support | +| `clawhdf5-remote` | `open_url`: HTTP(S) range requests and object stores (S3, GCS, Azure) through `BlockCache` | +| `clawhdf5-tools` | `h5rs`: `ls`, `dump` (DDL / hdf5-json), `stat`, `diff`, `check` | +| `clawhdf5-py` | PyO3 bindings (h5py-like API, remote files, `'r+'` editing) | +| `clawhdf5-wasm` | wasm-bindgen browser reader (`open(bytes)`, `openUrl(url)`); demo in `examples/wasm-viewer/` | +| `clawhdf5-ann` | HNSW index | +| `clawhdf5-agent` | Agent memory store (`HDF5Memory`), sessions, knowledge graph, BM25 | +| `clawhdf5-accel` | CPU SIMD kernels (AVX2, NEON) | +| `clawhdf5-gpu` | wgpu vector distances (WGSL) — not dataset I/O; HDF5 I/O is CPU-only | +| `clawhdf5-migrate` | SQLite → agent store migration | +| `clawhdf5-cli` | Agent-memory CLI | +| `clawhdf5-napi` | Node.js addon (the `packages/clawhdf5-node` wrapper is broken; `docs/known-issues.md`) | | `clawhdf5-android` | Android JNI bindings | -| `clawhdf5-cli` | Command-line interface (agent memory) | -| `clawhdf5-tools` | `h5rs`: pure-Rust HDF5 tools — `ls`, `dump` (DDL / hdf5-json), `stat`, `diff`, `check` (structural + checksum validator) | -| `clawhdf5-napi` | Node.js native addon bindings | -| `clawhdf5-py` | PyO3 Python bindings | -| `clawhdf5-wasm` | WebAssembly (wasm-bindgen) reader for the browser; demo in `examples/wasm-viewer/` | -| `clawhdf5-remote` | Remote files: `open_url` over HTTP(S) range requests and object stores (`object_store`: S3, GCS, Azure) through a mandatory block cache (`BlockCache`) | -| `clawhdf5-bench` | Benchmark suite | +| `clawhdf5-bench` | Benchmarks and harnesses (`search_harness`, `read_harness`, `concurrent_read`, `longmemeval_bench`, …) | -## Key Features -- Zero-C-dependency HDF5 read/write: no libhdf5, and deflate defaults to - pure-Rust zlib-rs (`fast-deflate` opts into zlib-ng, which needs cmake). - `ci-test.sh` fails if a C-building crate enters the core crates' default - tree. flate2 must keep `runtime_detection` with zlib-rs — without it zlib-rs - loses SIMD and inflates 3.5x slower. MSRV is 1.92 (`rust-version`, checked - in CI). -- HNSW vector index for semantic similarity search over agent memories — the - `clawhdf5-agent` `hnsw` feature is **on by default**, so `hybrid_search` uses - the approximate `clawhdf5-ann` index for the vector stage (the index mirrors - the cache and self-heals on drift). Build the agent with - `--no-default-features --features float16` to force the exact linear cosine scan. - The agent's `parallel` feature (also default) builds the index on a thread - pool; the graph is identical with or without it. - The index uses the HNSW paper's diversity heuristic for neighbour selection - (plain closest-M capped recall on clustered data: 0.31 recall@10 at 100K). Its - graph is saved to `.h5.ann` at each checkpoint and reloaded by `open()` - (tied to the checkpoint by a generation id; stale/damaged sidecars are - ignored and the index rebuilt). `MemoryConfig::quantized_index` (**on by - default** for new stores, persisted; stores predating the setting load as - `false` and keep their f32 index — guarded by - `tests/fixtures/store_v2_5_0.h5`; CLI opt-out is `create --f32-index`) - stores the index's own copy of the embeddings as `i8`, - which roughly halves a loaded store's memory (2.72x -> 1.74x the raw vectors - at 100K); because quantised distances are approximate and `ef` cannot - compensate, the query path then re-scores the candidate pool against the - exact embeddings, which holds recall at the f32 index's level. It is also - faster at equal recall: 1.63x the QPS on x86-64 (AVX2) and 1.18x on a - Raspberry Pi 5 (`clawhdf5_accel::dot_i8`, NEON `SDOT` via inline asm since - the intrinsic is unstable; plain NEON on pre-dotprod cores). The aarch64 - code is `cfg`'d out on x86, so x86 CI never compiles or lints it — test it - on real ARM (`rpivision02`, 10.0.2.3, is a Pi 5). `hybrid_search` keeps one incremental BM25 - index for the life of the store and never writes the store: Hebbian - activation boosts are persisted by the next checkpoint (or on drop), not per - query. Measure any search-path change with - `cargo run --release -p clawhdf5-bench --bin search_harness` (baselines in - `BENCHMARKS.md`). -- WAL (write-ahead log) for crash-safe persistence, with a chained CRC32 - trailer per entry (each entry's CRC folds in the previous entry's CRC) so a - corrupted, reordered, duplicated, or spliced entry stops replay cleanly - instead of loading bad or tampered data. The pre-chaining per-entry-CRC - format (v2) is still fully readable; the oldest no-CRC format (v1) is only - reachable through the one-time migration path in `HDF5Memory::open`, not - through the public `WalFile::read_entries`. - **What the WAL guarantees:** integrity, ordering, and recovery from a - *process* crash at any point — including between a checkpoint and the WAL - truncate (each checkpoint records a `WalMark` in `/meta`, and `open()` skips - the WAL prefix the `.h5` already contains, so entries are never applied - twice). Checkpoints and snapshots are made durable as a unit (temp file - synced, renamed, directory synced). **What it does not guarantee:** - individual WAL appends are *not* fsynced (a deliberate latency trade-off), so - saves made since the last checkpoint can be lost on power failure or kernel - panic. Current header version is 4 (adds the `Update` record used by - `save_or_update`); v3 files are read and upgraded in place. -- A store has a **single writer**: `HDF5Memory::create`/`open` hold an exclusive - advisory lock on `.h5.lock` and a second opener gets - `MemoryError::Locked`. Use `HDF5Memory::open_read_only` for a lock-free, - never-writing point-in-time view (the CLI's `recall`/`stats`/`agents-md`/ - `export` do). An unreadable WAL (torn header, bad magic) is quarantined to - `.h5.wal.corrupt-` rather than blocking `open()`; a WAL with an - unknown *newer* version still fails and is left untouched. -- `MemoryConfig::float16` (**on by default** for new stores, persisted; - existing stores keep their recorded `false` — guarded by the v2.5.0 - fixture in `tests/float16_store.rs`; CLI opt-out is `create --f32`) writes - `/memory/embeddings` as IEEE half precision (48% smaller file at 100K; - LongMemEval with real MiniLM embeddings identical to f32). - `MemoryCache::half_precision` rounds each embedding as it enters the cache (push, update, WAL replay, and on load of a store still - `f32` on disk), so memory and file agree bit for bit; the conversions live - in `clawhdf5_format::float16` and must stay the single implementation. - Values beyond ±65504 are `MemoryError::InvalidEntry`. Interop: every file - must open in h5py — `f32` datasets and empty datasets did not until - 2026-09-23 (see `docs/known-issues.md`); the agent's `h5py_interop` test - guards a whole store. -- `HDF5Memory::search(query_emb, text, &SearchOptions)` is the full search - path: optional source-channel filter (applied before ranking; exact scan of - the allowed records whenever cheaper than `pool × M` index distance +Reference docs: `docs/known-issues.md` (open issues table first — check it +before calling something a bug or a feature), `BENCHMARKS.md` (headline +numbers first), `CONFORMANCE.md` (generated), `docs/design/range-reads.md` +and `docs/design/swmr.md`, `CHANGELOG.md` (full detail of every fix). + +## Standing rules + +- **No C in the default build.** No libhdf5; deflate defaults to pure-Rust + zlib-rs (`fast-deflate` opts into zlib-ng, which needs cmake). `ci-test.sh` + fails if a C-building crate enters the core crates' default tree. Zstd, + SZIP, `https` (ring) and `s3`/`gcs`/`azure` (aws-lc-rs) are opt-in. flate2 + must keep `runtime_detection` with zlib-rs — without it zlib-rs loses SIMD + and inflates 3.5x slower. +- **Every file we write must open in h5py/libhdf5.** Interop tests compare + against h5py and h5dump; `f32` and empty datasets did not open until + 2026-09-23. +- **float16 has one implementation:** `clawhdf5_format::float16`. +- **Claims need evidence.** Performance and integration claims in docs must + be measured, dated (with machine and command), or withdrawn. Benchmark + numbers are dated records: never edit a measured value, add a new dated + section and mark the old one superseded. +- **OpenClaw is not supported** (decided 2026-09-25): clawhdf5 is not and + never was an OpenClaw memory plugin; the old `memory.backend = "clawhdf5"` + config was never valid. `docs/openclaw.md` records what a real plugin would + need. The `openclaw` module's `ClawhdfBackend` is just `search` with + re-rank + confidence on. +- **ZeroClaw does not use clawhdf5** (checked 2026-09-25 against upstream + v0.8.5 and the `osobh/zeroclaw` fork and their history): its memory + backends are its own; `clawhdf5-migrate`'s default SQLite layout is not + ZeroClaw's schema. Don't reintroduce integration claims without an + integration and a test against the real consumer. +- **known-issues.md:** one entry per bug; when fixed, record it in + `CHANGELOG.md` and move the entry to *Fixed (history)* with date, PR, + affected releases and what users must do — never delete it. + +## HDF5 library: invariants and gotchas + +- **Remote/range reads** (`docs/design/range-reads.md`, M0-M5 merged in PRs + #17-#21): every format-crate read path goes through `Storage` + (`read_at`/`read_ranges`/`hint`). `File::open_storage` takes any + `Storage`; `clawhdf5_remote::open_url` wraps HTTP (`HttpStorage`, ureq) or + `ObjectStoreStorage` in `BlockCache` (1 MiB blocks, LRU budget, in-flight + dedup, coalesced runs). Remote files are pinned by ETag/Last-Modified and + length (`RemoteError::FileChanged`). Zero-copy APIs and `File::as_bytes` + need an in-memory file. Parse through `File::storage()` and the `*_in` + functions, not `as_bytes`, in new code (the Python bindings do). + `ObjectStoreStorage` runs reads on its own small tokio runtime, so it + works from any thread. +- **SWMR** (`docs/design/swmr.md`): `File::open_swmr` reads a file a libhdf5 + SWMR writer is appending to — positioned reads, no chunk cache, bounded + retries (100), `Dataset::refresh()`. clawhdf5 has no SWMR writer; remote + SWMR is out of scope. +- **Browser** (`clawhdf5-wasm`, read-only, no Zstd/SZIP): `openUrl` reads + through the restartable "NeedBytes" cache (`src/lazy.rs`: a call is re-run + after each wave of misses; no block is evicted while a call runs); the HTTP + is JavaScript (`js/remote.js`). +- **In-place editing** (`clawhdf5::FileEditor`): overwrites values, grows and + shrinks chunked datasets (every chunk index) and sets attributes (compact + and dense) without rewriting the file, changing indexes and heaps as + libhdf5 does; freed space is reused within one editor. Anything it cannot + do safely is `Error::Unsupported` before any write (limits in + `docs/known-issues.md`). The algorithms follow libhdf5 `hdf5_1_14_6` + (github.com/HDFGroup/hdf5). Test changes with `cargo test -p + clawhdf5-tools --test edit_interop --test edit_coverage_interop`. +- **Provenance:** `Dataset::verify_provenance()` (facade `provenance` + feature, default) re-hashes a dataset against its `_provenance_sha256` + attribute (`DatasetBuilder::with_provenance`). Opt-in per call; unkeyed + hash — tamper-evident, not tamper-proof. + +## Agent memory: invariants and gotchas + +- **Search.** `HDF5Memory::search(query_emb, text, &SearchOptions)` is the + full path: optional source-channel filter (before ranking; exact scan of + the allowed records when cheaper than `pool × M` index distance evaluations, and as the fallback when the pool comes back short), fusion, activation scaling, optional re-ranking and confidence rejection. - `hybrid_search`/`hybrid_search_with` are thin wrappers; `ClawhdfBackend` - (the `openclaw` module) is `search` with re-rank + confidence on. -- **OpenClaw is not supported** (decided 2026-09-25): clawhdf5 is not an - OpenClaw memory plugin and never was — the old `memory.backend = "clawhdf5"` - config was never valid. Don't reintroduce OpenClaw claims; `docs/openclaw.md` - records what a real plugin would need. -- **ZeroClaw does not use clawhdf5** (checked 2026-09-25 against upstream - v0.8.5 and the `osobh/zeroclaw` fork, and their full history): no - `clawhdf5` feature or backend exists; ZeroClaw's memory backends are - sqlite/lucid/postgres/qdrant/markdown/none behind its own `Memory` trait. - `clawhdf5-migrate`'s default SQLite layout (`memory_chunks`, `sessions`, - `entities`, `relations`) is not ZeroClaw's schema either (ZeroClaw's is a - `memories` table). Don't reintroduce integration claims without an - integration and a test against the real consumer. Measure changes with - `search_harness --options-study`. -- `MemoryConfig::compression` is off by default; when on, embeddings are - deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd). -- Signed checkpoints (`clawhdf5-agent` `signing` module): with - `HDF5Memory::set_signing_key` every checkpoint stores an Ed25519-signed - manifest (SHA-256 per record in a Merkle tree + settings/sessions/graph - hashes; per-record hashes in `/integrity/record_hashes`); - `HDF5Memory::verify(path, &pk)` locates edits. The hashes must cover exactly - what the file persists in the form the loader returns it (strings lose - trailing NULs; an empty WAL mark is not written) or untouched stores stop - verifying — `tests/signed_store.rs` round-trips awkward strings. The key is - never persisted; a signed store refuses to checkpoint without it - (`MemoryError::SigningKeyRequired`, and `MemoryError` is `#[non_exhaustive]`). + `hybrid_search`/`hybrid_search_with` are thin wrappers. It keeps one + incremental BM25 index for the life of the store and never writes the + store: Hebbian activation boosts are persisted by the next checkpoint (or + on drop). +- **HNSW** (`hnsw` feature, default): the approximate `clawhdf5-ann` index + mirrors the cache and self-heals on drift; build the agent with + `--no-default-features --features float16` for the exact linear scan. + `parallel` (default) builds it on a thread pool with an identical graph. + Neighbour selection uses the HNSW paper's diversity heuristic (closest-M + capped recall at 0.31 recall@10 at 100K on clustered data). The graph is + saved to `.h5.ann` at each checkpoint, tied to it by a generation + id; a stale or damaged sidecar is ignored and the index rebuilt. +- **`MemoryConfig::quantized_index`** (default on for new stores, persisted; + older stores load as `false` — guarded by `tests/fixtures/store_v2_5_0.h5`; + CLI `create --f32-index`): the index's copy of the embeddings is `i8`, and + the query path re-scores candidates against the exact embeddings. The + aarch64 kernels (`clawhdf5_accel::dot_i8`, NEON `SDOT` via inline asm) are + `cfg`'d out on x86, so x86 CI never compiles them — test on real ARM + (`rpivision02`, 10.0.2.3, a Pi 5) or rely on the `test-arm64` job. +- **`MemoryConfig::float16`** (default on for new stores, persisted; older + stores keep `false` — guarded in `tests/float16_store.rs`; CLI `create + --f32`): `/memory/embeddings` is IEEE half. `MemoryCache::half_precision` + rounds each embedding as it enters the cache (push, update, WAL replay, and + load of a store still `f32` on disk) so memory and file agree bit for bit. + Values beyond ±65504 are `MemoryError::InvalidEntry`. The agent's + `h5py_interop` test guards that a whole store opens in h5py. +- **WAL.** Chained CRC32 per entry (a corrupted, reordered, duplicated or + spliced entry stops replay cleanly). Header version 4 (`Update` record for + `save_or_update`); v3 is upgraded in place, v2 read, v1 only through the + one-time migration in `HDF5Memory::open`. Each checkpoint records a + `WalMark` in `/meta` so `open()` never applies an entry twice; checkpoints + and snapshots are durable as a unit (temp file synced, renamed, directory + synced). Individual WAL appends are **not** fsynced (deliberate): saves + since the last checkpoint can be lost on power failure or kernel panic. +- **Single writer.** `create`/`open` hold an exclusive lock on + `.h5.lock` (`MemoryError::Locked` for a second opener); + `open_read_only` is a lock-free point-in-time view (CLI `recall`/`stats`/ + `agents-md`/`export`). An unreadable WAL is quarantined to + `.h5.wal.corrupt-`; a WAL of an unknown newer version fails and + is left untouched. +- **Signed checkpoints** (`signing` module): with `set_signing_key` each + checkpoint stores an Ed25519-signed manifest (per-record SHA-256 in a + Merkle tree plus settings/sessions/graph hashes; `/integrity/record_hashes`); + `HDF5Memory::verify(path, &pk)` locates edits. The hashes must cover + exactly what the file persists in the form the loader returns it (strings + lose trailing NULs; an empty WAL mark is not written) — + `tests/signed_store.rs` round-trips awkward strings. The key is never + persisted; a signed store refuses to checkpoint without it + (`MemoryError::SigningKeyRequired`; `MemoryError` is `#[non_exhaustive]`). WAL entries after the checkpoint are not covered. -- `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by - default) recomputes a dataset's SHA-256 and compares it against the - `_provenance_sha256` attribute written automatically on save when - `DatasetBuilder::with_provenance` is used. It's opt-in per call, not run - automatically on open — it decodes and hashes the whole dataset. The hash - is unkeyed (tamper-*evident*, not tamper-*proof*): it detects accidental - corruption, not a deliberate actor able to modify both the data and the - stored hash. -- `clawhdf5-agent`'s `HDF5Memory::save`/`save_batch`/`save_or_update` run every - write through an in-memory (session-scoped, not persisted to disk) - provenance ledger and write-anomaly detector: a content hash per record - (`provenance.rs`) for detecting accidental mid-session corruption, plus - rate-limit/injection-pattern/source-distribution checks (`anomaly.rs`). - Alerts never block a save — drain them with `HDF5Memory::take_anomaly_alerts`. - `MemorySource` for this bookkeeping is inferred from the caller-supplied - `source_channel` string (a heuristic, not an authenticated trust boundary). -- In-place modification: `clawhdf5::FileEditor` (`crates/clawhdf5/src/edit/`) - overwrites values, grows and shrinks chunked datasets (every chunk index, - version-2 B-trees included) and sets attributes (compact and dense - storage) in existing files (h5py- or clawhdf5-written) without rewriting - them, changing indexes and heaps as libhdf5 does (index shapes and heap - bookkeeping are compared with libhdf5's in the tests); space an edit - frees is reused by later edits of the same editor. Anything it cannot do - safely is `Error::Unsupported` before any write (limits in - `docs/known-issues.md`). Test changes with - `cargo test -p clawhdf5-tools --test edit_interop --test - edit_coverage_interop` (h5py, h5dump, `h5rs check`, structure comparisons - with libhdf5; libhdf5 sources for the algorithms are at - github.com/HDFGroup/hdf5, tag `hdf5_1_14_6`). -- Remote files (`clawhdf5-remote`, range-read milestone M3 of - `docs/design/range-reads.md`): `open_url("http://…")` gives a - `clawhdf5::File` over `File::open_storage`, read through `BlockCache` - (1 MiB blocks, LRU byte budget, per-block in-flight dedup across threads, - runs coalesced into parallel requests). `HttpStorage` pins the file by - ETag/Last-Modified and length (a change is `RemoteError::FileChanged`), - refuses servers that ignore `Range` unless a full download is allowed, - and retries transient failures. `ObjectStoreStorage` (feature - `object-store`, pure Rust) runs each read on a small owned tokio - runtime and waits on a channel, so it works from any thread, including - inside `spawn_blocking` or another runtime. Default build is plain HTTP with - no C; `https` (rustls + ring) and `s3`/`gcs`/`azure` (aws-lc-rs) are - opt-in. Tests run a std-only HTTP server - (`tests/common/server.rs`, also the `range_server` example); - `CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus` compares every corpus - file over HTTP with `File::open`. -- GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only -- Browser: `clawhdf5-wasm` (wasm-bindgen, read-only; no Zstd/SZIP since - they link C) and the `examples/wasm-viewer/` page. `open(bytes)` holds - the file in memory; `openUrl(url)` (range-read M4) reads it by HTTP range - requests through the restartable "NeedBytes" cache (`src/lazy.rs`: a - call is re-run after each wave of misses; no block evicted while a call - runs), the HTTP in `js/remote.js`. `examples/wasm-viewer/test/run.sh` - builds the package (needs the `wasm-bindgen` CLI at the crate's exact - version) and tests it under Node and headless Chromium (a Playwright - download in `~/.cache/ms-playwright` on tank) against `test/serve.py` - (range server with request counts, 200 MB budget file); the CI container - has neither, so CI runs the native `h5py_interop` and `lazy` tests - (`CLAWHDF5_WASM_CORPUS=conformance/.cache/corpus` for the corpus). Size - numbers in the example's README predate `openUrl`. -- Python and Node.js bindings for cross-language use -- NetCDF-4 compatibility for scientific data interop +- **Write bookkeeping.** `save`/`save_batch`/`save_or_update` feed an + in-memory, session-scoped provenance ledger and anomaly detector + (`provenance.rs`, `anomaly.rs`); alerts never block a save + (`take_anomaly_alerts`). `MemorySource` is inferred from the caller's + `source_channel` string — a heuristic, not a trust boundary. +- `MemoryConfig::compression` is off by default (deflate, or Zstd with the + agent's `zstd` feature, which links libzstd). ## Workflows -### Build +Put `$HOME/.cargo/bin` on `PATH`. The h5py/netCDF4 interop tests find their +Python through `CLAWHDF5_PYTHON` (or `.venv/bin/python`); create it with +`python3 -m venv .venv && .venv/bin/pip install h5py numpy netCDF4 hdf5plugin`. +Set `CLAWHDF5_REQUIRE_INTEROP=1` to make a missing interpreter a failure. + ```bash cargo build --release -``` - -### Test -```bash cargo test --workspace +bash scripts/ci-test.sh # everything CI runs (see below) ``` -### CI -`.gitea/workflows/ci.yml` has two jobs, both green as of 2026-09-22: -- **`test`** (`ubuntu-latest`, in `rust:latest`) runs `scripts/ci-test.sh` with - the h5py/netCDF4 interop suites required (`CLAWHDF5_REQUIRE_INTEROP=1`). - Served by the `tank` and `architect` runners. -- **`test-arm64`** (`linux_arm64`) lints and tests the aarch64 code — the NEON - kernels are `cfg`'d out on x86, so this is the only place they are built. - Served by `vision-01` (host mode) and `vision-02` (Docker), so steps must - work in both. +### CI (`.gitea/workflows/`) +- **`ci.yml` `test`** (`ubuntu-latest`, `rust:latest` container; runners + `tank`, `architect`): installs h5py/netCDF4/xarray/hdf5plugin/maturin/pytest, + `hdf5-tools` and `cmake`, then runs `scripts/ci-test.sh` with + `CLAWHDF5_REQUIRE_INTEROP=1`. The script runs: fmt; clippy (workspace, the + format feature matrix, each plugin filter alone, parallel, fast-deflate, + remote with all backends, h5rs remote); "no C in the default build"; + wasm32 build and clippy; `check-32bit-casts.sh`; the wasm package under + Node when `node` and `wasm-bindgen` exist (not in CI); the MSRV check; + `cargo test` (workspace plus feature variants: format matrix, parallel, + remote/object_store, h5rs URLs, ann parallel, fast-deflate); the h5py + interop suites (`writer_h5py_tests --include-ignored`, plugin filters, + ZFP); the Python package (clippy, `maturin build`, pytest vs h5py); + `cargo bench --no-run`; `check-nostd.sh`; an optional fuzz smoke run + (`CLAWHDF5_FUZZ_SECONDS`). +- **`ci.yml` `test-arm64`** (`linux_arm64`; `vision-01` host mode, + `vision-02` Docker — steps must work in both): clippy and tests of + `clawhdf5-accel`, `-ann`, `-format`; the only place the NEON kernels build. +- **`conformance.yml`** (nightly 03:17 UTC and manual): probe unit tests, + `conformance/test_ref.py`, then `conformance/run.sh` (gate: + `conformance/check.py` against `baseline.json`). -Keep workflows free of JavaScript actions (`actions/checkout`, `actions/cache`, -…): `rust:latest` has no `node`, and not every runner reaches GitHub, where -they are fetched from. Check out with plain `git` instead. The `test` job -installs `cmake` for the opt-in `fast-deflate` (zlib-ng) steps; the default -build needs no C toolchain, so `test-arm64` does not. -All runners are on `gitea-runner` 3.5.0, from `docker.gitea.com/act_runner` -— `gitea/act_runner:latest` on Docker Hub is frozen at 0.6.1. +Keep workflows free of JavaScript actions (`actions/checkout`, +`actions/cache`, …): `rust:latest` has no `node` and not every runner reaches +GitHub. Check out with plain `git`. Runners are `gitea-runner` 3.5.0 from +`docker.gitea.com/act_runner` (`gitea/act_runner:latest` on Docker Hub is +frozen at 0.6.1). -### CLI +### Conformance ```bash -cargo run -p clawhdf5-cli -- --help -# create, save, search, recall, stats, flush-wal, agents-md, export, snapshot subcommands +CLAWHDF5_PYTHON=.venv/bin/python bash conformance/run.sh --no-fetch # writes CONFORMANCE.md ``` +Reads 697 files of eight pinned corpora with clawhdf5 and h5py and compares +them object by object (602 ok as of 2026-09-27). `CONFORMANCE.md` is +generated — never hand-edit it (its wording lives in `conformance/report.py`). +Use `--update-baseline` only after an intended change in results. +`CONFORMANCE_CACHE` points at an existing corpus cache (`conformance/.cache`, +about 450 MB). See `conformance/README.md`. -### HDF5 tools (`h5rs`, crate `clawhdf5-tools`) +### HDF5 tools (`h5rs`) ```bash cargo run -p clawhdf5-tools -- ls -r file.h5 # also dump [--json], stat, diff, check bash scripts/h5rs-fuzz.sh # every subcommand over the CVE corpus: no panic/crash/hang bash scripts/h5rs-check-ok-files.sh --data # check passes every fully-read conformance file ``` -Its interop tests compare against h5ls/h5stat/h5dump/h5diff (Debian -`hdf5-tools`, installed in CI); `dump` must stay byte-identical to h5dump on -the test files. +Interop tests compare against h5ls/h5stat/h5dump/h5diff (Debian `hdf5-tools`); +`dump` must stay byte-identical to h5dump on the test files. + +### Remote and browser tests +- `clawhdf5-remote` tests run a std-only HTTP server + (`tests/common/server.rs`, also the `range_server` example); + `CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus` compares every corpus + file over HTTP with `File::open`. +- wasm: `bash examples/wasm-viewer/test/run.sh` builds the package (needs the + `wasm-bindgen` CLI at the crate's exact version) and tests it under Node and + headless Chromium (Playwright's download in `~/.cache/ms-playwright` on + tank) against `test/serve.py` (range server with request counts). CI has + neither, so it runs the native `h5py_interop` and `lazy` tests + (`CLAWHDF5_WASM_CORPUS=conformance/.cache/corpus` for the corpus). ### Python bindings ```bash -cd crates/clawhdf5-py -maturin develop -python -c "import clawhdf5; print(clawhdf5.__version__)" +cd crates/clawhdf5-py && maturin develop +python -m pytest crates/clawhdf5-py/tests # compares with h5py; editing tests want CLAWHDF5_H5RS= +``` + +### Benchmarks +- Search path: `cargo run --release -p clawhdf5-bench --bin search_harness` + (`--full`, `--options-study`, `--footprint`, …); reads: `read_harness`, + `concurrent_read`; criterion benches with `cargo bench -p `. +- Run on an idle machine (1-minute load average below 2; wait otherwise), + alternate base and candidate binaries for A/B comparisons, and record date, + machine, commit and command with every number in `BENCHMARKS.md`. +- `scripts/run-benchmarks.sh` is stale (it benchmarks `rustyhdf5-format` and + overwrites `BENCHMARKS.md`) — do not run it. + +### CLI +```bash +cargo run -p clawhdf5-cli -- --help +# create, save, search, recall, stats, flush-wal, agents-md, export, snapshot ``` ## Integration @@ -255,8 +268,7 @@ python -c "import clawhdf5; print(clawhdf5.__version__)" verified consumer: `cbh-core` reads and writes `.brain` files through the facade (`File`, `FileBuilder`, `AttrValue`, `Selection`), `cbh-scanner` uses the facade, and `cbh-cli` uses `clawhdf5_agent::bm25::BM25Index`. It - depends on this repo by path (`../clawhdf5`), so it builds against whatever - is checked out — changes to those APIs reach it directly. Verified - 2026-09-25 against main: builds, and its 204 tests pass. -- OpenClaw and ZeroClaw were both described as consumers; neither integrates - clawhdf5 (see Key Features and `docs/openclaw.md`). + depends on this repo by path (`../clawhdf5`), so changes to those APIs + reach it directly. Verified 2026-09-25 against main: builds, and its 204 + tests pass. +- OpenClaw and ZeroClaw integrate nothing (see *Standing rules*).