Author SHA1 Message Date
osobh 79dfa78e8f Merge pull request 'Withdraw the OpenClaw integration claims' (#9) from docs/withdraw-openclaw into main
CI / test-arm64 (push) Successful in 1m18s
CI / test (push) Successful in 5m40s
Reviewed-on: #9
2026-09-25 16:14:36 +00:00
osobh bdadf3447c Merge pull request 'Ed25519-signed checkpoints; remove the no-op agent feature' (#8) from feat/signed-checkpoints into main
CI / test-arm64 (push) Successful in 1m16s
CI / test (push) Successful in 5m42s
Reviewed-on: #8
2026-09-25 15:58:19 +00:00
osobhandClaude Opus 5.5 0c65a27b00 docs: withdraw the OpenClaw integration claims
CI / test-arm64 (pull_request) Successful in 1m3s
CI / test (pull_request) Successful in 5m48s
The docs described a "drop-in" OpenClaw memory backend enabled with
`memory.backend = "clawhdf5"`. Checked against OpenClaw's source and
docs (v2026.2.26 through v2026.9.6): that config was never valid —
v2026.2-v2026.7 accepted only "builtin"/"qmd" and rejected unknown
keys, so a Gateway given it refuses to start, and v2026.8.1 (OpenClaw
2.0) removed the key. No plugin was ever built (no manifest, no
registration, no tools), nothing was tested against OpenClaw, the
linked github.com/redclawsystems/openclaw is a 404, and
@redclaw/clawhdf5 was never published.

Decision (2026-09-25): not pursuing an OpenClaw plugin for now; ZeroClaw
is the integration target.

- Remove openclaw-integration.md, openclaw-config.md and
  migration-guide.md; add docs/openclaw.md: the status, what a memory
  plugin needs against v2026.9.6 (plugins.slots.memory, manifest with
  kind "memory", registerMemoryCapability / MemorySearchManager,
  prebuilt native packages), and what this repo has as building blocks.
- README, QUICKSTART, USE_CASES, ROADMAP (Track 7 withdrawn), CLAUDE.md
  and the `openclaw` module docs describe ClawhdfBackend as what it is:
  a Markdown-oriented library backend, not an OpenClaw plugin. The
  QUICKSTART example is corrected (the old one called a three-argument
  create that does not exist) and states its limits.
- packages/clawhdf5-node: marked unpublished and broken, "private": true
  so it cannot be published by accident; its bugs (snake_case vs
  camelCase fields, wrong addon path, no way to store an embedding,
  wrong WAL name) are recorded in docs/known-issues.md.
- Two broken rustdoc links fixed along the way.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-25 10:27:54 -05:00
osobhandClaude Opus 5.5 db9af7972c feat(agent): Ed25519-signed checkpoints
CI / test-arm64 (pull_request) Successful in 1m5s
CI / test (pull_request) Successful in 4m50s
Makes the README's "cryptographically verifiable memory" true.

With HDF5Memory::set_signing_key(key), every checkpoint stores a signed
manifest of the store: a SHA-256 per memory record (text, embedding as
stored, channel, timestamp, session, tags, deleted flag, activation) in
a Merkle tree, plus hashes of the settings (and WAL mark), sessions and
knowledge graph. The signature, public key and manifest hashes go in
/meta; the per-record hashes in /integrity/record_hashes, so
HDF5Memory::verify(path, &public_key) can say which records changed, not
just that something did. A forged manifest fails the signature.

Decisions, as agreed:
- the key is set on the open store and never persisted;
- a signed store refuses to checkpoint without its key
  (MemoryError::SigningKeyRequired); remove_signature() is the
  deliberate way back to unsigned;
- checkpoints only: saves still in the WAL are not covered, and verify
  reports how many there are.

The hashes cover exactly what the file persists, in the form the loader
returns it (strings lose trailing NULs; an empty WAL mark is not
written), so untouched stores verify across any number of reopen and
checkpoint cycles. MemoryError becomes #[non_exhaustive] (it already
gains variants in this unreleased version).

CLI: keygen (owner-only key file), --signing-key / CLAWHDF5_SIGNING_KEY
on writing commands (create signs immediately), verify --public-key
(JSON; exit 2 if not valid), `signed` in create/stats output.

Tests: reopen/checkpoint cycles with awkward strings (f16 and f32),
refusal without the key, wrong and rotated keys, eight kinds of edit
each detected and located, a forged manifest, unsigned stores, NULs in
text, and an edit made in place with h5py that verify pinpoints.

Cost on tank (search_harness --signing-study --full, 3 runs): ~20% of a
checkpoint (+9 ms at 10K, +89-112 ms at 100K), verify 18.6 ms / 247 ms,
32 bytes per record in the file. New deps ed25519-dalek, sha2,
rand_core: pure Rust, the no-C check passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-25 10:13:34 -05:00
osobh 7706697feb Merge pull request 'Complete the consolidation benchmark: cheaper novelty scoring' (#7) from feat/consolidation-scaling into main
CI / test-arm64 (push) Successful in 1m7s
CI / test (push) Successful in 5m28s
Reviewed-on: #7
2026-09-25 15:05:58 +00:00
osobh c0f704c381 Merge pull request 'clawhdf5-migrate writes real agent stores; knowledge-graph fix; dated benchmark re-run' (#6) from feat/migrate-and-benchmarks into main
CI / test-arm64 (push) Successful in 1m19s
CI / test (push) Successful in 5m34s
Reviewed-on: #6
2026-09-25 14:52:35 +00:00
osobh a7920bd4b3 Merge pull request 'Search options (source filters, re-ranking, confidence); float16 default' (#5) from feat/search-options into main
CI / test-arm64 (push) Successful in 1m8s
CI / test (push) Successful in 5m40s
Reviewed-on: #5
2026-09-25 12:44:15 +00:00
osobh 7e43b5366c Merge branch 'main' into feat/search-options
CI / test-arm64 (pull_request) Successful in 54s
CI / test (pull_request) Successful in 4m35s
2026-09-25 12:32:00 +00:00
osobh 73bb068264 Merge pull request 'Files open in h5py again; float16 embedding storage' (#4) from feat/float16-embeddings into main
CI / test-arm64 (push) Successful in 1m17s
CI / test (push) Successful in 5m10s
Reviewed-on: #4
2026-09-25 03:18:07 +00:00
26 changed files with 1522 additions and 851 deletions
+24
View File
@@ -243,6 +243,30 @@ index asked for ~16 000 candidates, where scanning the few hundred or thousand
allowed records is exact and cheap. Re-ranking a 3k candidate pool and allowed records is exact and cheap. Re-ranking a 3k candidate pool and
confidence rejection add about 3%. confidence rejection add about 3%.
### Signed checkpoints
Measured 2026-09-25 on tank (AMD Ryzen 7 7800X3D). A default store (float16,
int8 index), 384-dim; each checkpoint rewrites the whole file, as every
checkpoint does. Medians of five checkpoints and three verifies; three runs
agreed to within the ranges shown.
```bash
cargo run --release -p clawhdf5-bench --bin search_harness -- --signing-study --full
```
| N | checkpoint, unsigned | checkpoint, signed | signing adds | `verify` | file size added |
|---:|---:|---:|---:|---:|---:|
| 1 000 | 5.4 ms | 6.4 ms | 0.7–1.0 ms | 2.1 ms | 0.03 MiB |
| 10 000 | 46 ms | 55 ms | 8.1–9.4 ms | 18.6 ms | 0.31 MiB |
| 100 000 | 495 ms | 598 ms | 89–112 ms | 247 ms | 3.05 MiB |
Signing costs about 20% of a checkpoint: every record is rehashed (SHA-256)
and the Merkle root recomputed each time; the Ed25519 signature itself is
microseconds. Caching per-record hashes between checkpoints would cut this to
the records that changed. The per-record hashes stored for locating edits are
32 bytes each (4% of a 100K float16 store). `verify` reads and rehashes the
whole checkpoint.
### float16 embedding storage (`MemoryConfig::float16`) ### float16 embedding storage (`MemoryConfig::float16`)
Measured 2026-09-23 on tank (AMD Ryzen 7 7800X3D). The same clustered Measured 2026-09-23 on tank (AMD Ryzen 7 7800X3D). The same clustered
+42
View File
@@ -3,6 +3,18 @@
## Unreleased ## Unreleased
### Upgrade Notes ### Upgrade Notes
- **OpenClaw is not supported, and never was.** The docs described a
"drop-in" OpenClaw memory backend enabled with `memory.backend = "clawhdf5"`.
That config was never valid in any OpenClaw release (v2026.2–v2026.7
accepted only `builtin`/`qmd` and rejected unknown keys, so a Gateway given
it refuses to start; OpenClaw 2.0 removed the key), no plugin was ever built,
and `@redclaw/clawhdf5` was never published. The integration docs
(`openclaw-integration.md`, `openclaw-config.md`, `migration-guide.md`) are
removed; `docs/openclaw.md` explains the status and what a real plugin would
need against OpenClaw v2026.9.6. `ClawhdfBackend` stays as a library API.
- **Breaking:** `MemoryError` is now `#[non_exhaustive]` and gained
`SigningKeyRequired`; a `match` on it needs a wildcard arm. Future variants
will no longer be breaking.
- **Breaking:** `clawhdf5-agent`'s `agent` feature is removed. It enabled - **Breaking:** `clawhdf5-agent`'s `agent` feature is removed. It enabled
nothing — the agent layer is always built — but the README and guides told nothing — the agent layer is always built — but the README and guides told
people to pass it; drop `agent` from `features = [...]`. people to pass it; drop `agent` from `features = [...]`.
@@ -60,6 +72,29 @@
`quantized_index = false`, or pass `create --f32-index` to the CLI, to opt `quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
out. The CLI's `--quantized-index` is still accepted but is now a no-op. out. The CLI's `--quantized-index` is still accepted but is now a no-op.
### Signing
- `clawhdf5-agent`: **Ed25519-signed checkpoints** — the README's
"cryptographically verifiable memory", now true. With
`HDF5Memory::set_signing_key(key)`, every checkpoint stores a signed
manifest: a SHA-256 per record (text, embedding as stored, channel,
timestamp, session, tags, deleted flag, activation) in a Merkle tree, plus
hashes of the settings (and WAL mark), sessions and knowledge graph, with
the per-record hashes in `/integrity/record_hashes`.
`HDF5Memory::verify(path, &public_key)` recomputes everything from the file
and reports which part changed and which records (`changed_records`); a
forged manifest fails the signature. The key is never persisted; a signed
store refuses to checkpoint without it (`MemoryError::SigningKeyRequired`),
and `remove_signature()` is the deliberate way back to unsigned. Saves still
in the WAL are not covered (`wal_entries_unsigned`). Tests include every
kind of edit, and an edit made with h5py in place, which verify pinpoints.
Cost: ~20% of a checkpoint, 32 bytes per record (`BENCHMARKS.md`, "Signed
checkpoints"). New dependencies `ed25519-dalek`, `sha2`, `rand_core` — pure
Rust; the no-C check still passes.
- `clawhdf5-cli`: `keygen --out <file>` (owner-only key file),
`--signing-key <file>` / `CLAWHDF5_SIGNING_KEY` on writing commands
(`create` signs immediately), `verify --public-key <hex|file>` (JSON report;
exit status 2 if not valid), and `signed` in `create`/`stats` output.
### Migration ### Migration
- `clawhdf5-migrate`: writes through the agent's own API (`HDF5Memory::create` - `clawhdf5-migrate`: writes through the agent's own API (`HDF5Memory::create`
/ `open`, `save_batch`, the session cache and knowledge graph), so there is / `open`, `save_batch`, the session cache and knowledge graph), so there is
@@ -100,6 +135,13 @@
activation of the `k` results it returns, not of the whole `3k` candidate activation of the `k` results it returns, not of the whole `3k` candidate
pool it re-ranks. pool it re-ranks.
### Documentation
- OpenClaw claims withdrawn across the README, QUICKSTART, USE_CASES, ROADMAP
(Track 7 marked withdrawn) and the `openclaw` module docs; the dead
`github.com/redclawsystems/openclaw` link is gone. The Node package is
marked unpublished and broken (now `"private": true` so it cannot be
published by accident), with its bugs recorded in `docs/known-issues.md`.
### Benchmarks ### Benchmarks
- Every undated or pre-September section of `BENCHMARKS.md` re-run on one - Every undated or pre-September section of `BENCHMARKS.md` re-run on one
machine on one day (tank, 2026-09-24, commit 5c8323c), with the command for machine on one day (tank, 2026-09-24, commit 5c8323c), with the command for
+17 -2
View File
@@ -104,11 +104,26 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
the allowed records whenever cheaper than `pool × M` index distance the allowed records whenever cheaper than `pool × M` index distance
evaluations, and as the fallback when the pool comes back short), fusion, evaluations, and as the fallback when the pool comes back short), fusion,
activation scaling, optional re-ranking and confidence rejection. activation scaling, optional re-ranking and confidence rejection.
`hybrid_search`/`hybrid_search_with` are thin wrappers; the OpenClaw `hybrid_search`/`hybrid_search_with` are thin wrappers; `ClawhdfBackend`
backend is `search` with re-rank + confidence on. Measure changes with (the `openclaw` module) is `search` with re-rank + confidence on.
- **OpenClaw is not supported** (decided 2026-09-25): clawhdf5 is not an
OpenClaw memory plugin and never was — the old `memory.backend = "clawhdf5"`
config was never valid. Don't reintroduce OpenClaw claims; `docs/openclaw.md`
records what a real plugin would need. ZeroClaw is the integration target. Measure changes with
`search_harness --options-study`. `search_harness --options-study`.
- `MemoryConfig::compression` is off by default; when on, embeddings are - `MemoryConfig::compression` is off by default; when on, embeddings are
deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd). deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd).
- Signed checkpoints (`clawhdf5-agent` `signing` module): with
`HDF5Memory::set_signing_key` every checkpoint stores an Ed25519-signed
manifest (SHA-256 per record in a Merkle tree + settings/sessions/graph
hashes; per-record hashes in `/integrity/record_hashes`);
`HDF5Memory::verify(path, &pk)` locates edits. The hashes must cover exactly
what the file persists in the form the loader returns it (strings lose
trailing NULs; an empty WAL mark is not written) or untouched stores stop
verifying — `tests/signed_store.rs` round-trips awkward strings. The key is
never persisted; a signed store refuses to checkpoint without it
(`MemoryError::SigningKeyRequired`, and `MemoryError` is `#[non_exhaustive]`).
WAL entries after the checkpoint are not covered.
- `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by - `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by
default) recomputes a dataset's SHA-256 and compares it against the default) recomputes a dataset's SHA-256 and compares it against the
`_provenance_sha256` attribute written automatically on save when `_provenance_sha256` attribute written automatically on save when
+49 -13
View File
@@ -8,7 +8,7 @@
[![LongMemEval](https://img.shields.io/badge/LongMemEval__s-Turn--Level%20Hit@5%2081.4%25%20hybrid-blue.svg)](BENCHMARKS.md#longmemeval-results) [![LongMemEval](https://img.shields.io/badge/LongMemEval__s-Turn--Level%20Hit@5%2081.4%25%20hybrid-blue.svg)](BENCHMARKS.md#longmemeval-results)
[![Footprint](https://img.shields.io/badge/on--disk-~820%20B%2Frecord%20float16%2C%20synthetic%20text-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1) [![Footprint](https://img.shields.io/badge/on--disk-~820%20B%2Frecord%20float16%2C%20synthetic%20text-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1)
ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file. ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, cryptographically verifiable memory (Ed25519-signed checkpoints) — all stored in a single portable file.
> **Two things live here:** > **Two things live here:**
> - **A general-purpose, pure-Rust HDF5 library** — zero C dependencies, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers. > - **A general-purpose, pure-Rust HDF5 library** — zero C dependencies, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers.
@@ -71,7 +71,7 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md).
100K). It no longer rebuilds BM25 or rewrites the store per query, and the 100K). It no longer rebuilds BM25 or rewrites the store per query, and the
HNSW graph is persisted (v2.4.0). HNSW graph is persisted (v2.4.0).
- Default fusion weights are now the measured 0.4 / 0.6 (v2.5.0). Re-ranking had - Default fusion weights are now the measured 0.4 / 0.6 (v2.5.0). Re-ranking had
been discarding the retrieval score, costing the OpenClaw backend 40.6pp of been discarding the retrieval score, costing the Markdown backend 40.6pp of
Hit@1; fixed in v2.6.0. Hit@1; fixed in v2.6.0.
- Selection reads decode only the chunks they touch (a 64×64 window: 105 ms to - Selection reads decode only the chunks they touch (a 64×64 window: 105 ms to
0.39 ms), and full reads are 1.2–1.9× faster (v2.5.0). 0.39 ms), and full reads are 1.2–1.9× faster (v2.5.0).
@@ -93,7 +93,7 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md).
identical LongMemEval retrieval on real embeddings. identical LongMemEval retrieval on real embeddings.
- `HDF5Memory::search` with `SearchOptions`: filter by source channel (exact - `HDF5Memory::search` with `SearchOptions`: filter by source channel (exact
filtered top-k, never slower than unfiltered), and opt-in re-ranking and filtered top-k, never slower than unfiltered), and opt-in re-ranking and
confidence rejection, which used to be OpenClaw-only. confidence rejection, which used to be reachable only through `ClawhdfBackend`.
**Tooling** **Tooling**
- CI now runs the h5py/netCDF4 interop suites for real (they had been skipping - CI now runs the h5py/netCDF4 interop suites for real (they had been skipping
@@ -113,7 +113,7 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d
| Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers | | Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers |
| Temporal queries | Custom code | Native temporal index (622 ns range query over 10K) | | Temporal queries | Custom code | Native temporal index (622 ns range query over 10K) |
| Multi-modal | Multiple stores | Unified cross-modal search (exact scan: 842 µs over 1K records) | | Multi-modal | Multiple stores | Unified cross-modal search (exact scan: 842 µs over 1K records) |
| Integrity | Hope for the best | Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance | | Integrity | Hope for the best | Ed25519-signed checkpoints that pinpoint any edited record, chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts |
| Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** | | Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** |
--- ---
@@ -228,7 +228,7 @@ is for. The weights matter more than the stages: a sweep of `vector_weight` from
0.0 to 1.0 found the old `0.7/0.3` default is **strictly dominated** by 0.0 to 1.0 found the old `0.7/0.3` default is **strictly dominated** by
`0.4/0.6` — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Since `0.4/0.6` — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Since
v2.5.0 `0.4/0.6` is the default (`hybrid::DEFAULT_FUSION`, used by v2.5.0 `0.4/0.6` is the default (`hybrid::DEFAULT_FUSION`, used by
`unified_search`, `hybrid_search_with` and the OpenClaw backend); callers that `unified_search`, `hybrid_search_with` and `ClawhdfBackend`); callers that
pass weights to `hybrid_search` explicitly choose their own. Use `0.3/0.7` if pass weights to `hybrid_search` explicitly choose their own. Use `0.3/0.7` if
rank-1 precision matters most. Reciprocal rank fusion is selectable rank-1 precision matters most. Reciprocal rank fusion is selectable
(`hybrid::Fusion::Rrf`) but measured worse than the weighted sum. See (`hybrid::Fusion::Rrf`) but measured worse than the weighted sum. See
@@ -329,7 +329,7 @@ ClawhDF5's agent memory engine draws on 15+ recent papers on agentic memory syst
│ × √(Hebbian activation) │ │ × √(Hebbian activation) │
└─────────────────┬──────────────────┘ └─────────────────┬──────────────────┘
│ opt-in (SearchOptions); │ opt-in (SearchOptions);
│ the OpenClaw backend turns both on │ ClawhdfBackend turns both on
┌─────────────────▼──────────────────┐ ┌─────────────────▼──────────────────┐
│ Multi-factor re-ranking │ │ Multi-factor re-ranking │
│ relevance · recency · authority · │ │ relevance · recency · authority · │
@@ -365,13 +365,14 @@ directly; the store persists the records, sessions and graph they work over.
| **`knowledge`** | Entity/relation graph with BFS traversal, spreading activation, fuzzy (Levenshtein) entity resolution | | **`knowledge`** | Entity/relation graph with BFS traversal, spreading activation, fuzzy (Levenshtein) entity resolution |
| **`consolidation`** | Three-tier memory (Working → Episodic → Semantic) with importance scoring, novelty, and time-decay | | **`consolidation`** | Three-tier memory (Working → Episodic → Semantic) with importance scoring, novelty, and time-decay |
| **`hybrid`** | Vector + BM25 fusion. Default is a min-max-normalised weighted sum, vector 0.4 / keyword 0.6 (`hybrid::DEFAULT_FUSION`, tuned on LongMemEval); RRF is available via `Fusion::Rrf` / `hybrid_search_with`. The vector stage uses the HNSW index by default (`hnsw` feature); disable with `--no-default-features --features float16` for an exact linear scan | | **`hybrid`** | Vector + BM25 fusion. Default is a min-max-normalised weighted sum, vector 0.4 / keyword 0.6 (`hybrid::DEFAULT_FUSION`, tuned on LongMemEval); RRF is available via `Fusion::Rrf` / `hybrid_search_with`. The vector stage uses the HNSW index by default (`hnsw` feature); disable with `--no-default-features --features float16` for an exact linear scan |
| **`reranker`** | Multi-factor re-ranking: retrieval relevance (leads, weight 1.0), temporal recency, source authority, activation weight. Opt-in via `SearchOptions::with_rerank`; on in the OpenClaw backend | | **`reranker`** | Multi-factor re-ranking: retrieval relevance (leads, weight 1.0), temporal recency, source authority, activation weight. Opt-in via `SearchOptions::with_rerank`; on in `ClawhdfBackend` |
| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches. Opt-in via `SearchOptions::with_confidence`; on in the OpenClaw backend | | **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches. Opt-in via `SearchOptions::with_confidence`; on in `ClawhdfBackend` |
| **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints | | **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints |
| **`multimodal`** | Cross-modal search across text/image/audio/video embeddings | | **`multimodal`** | Cross-modal search across text/image/audio/video embeddings |
| **`signing`** | Ed25519-signed checkpoints: SHA-256 per record in a Merkle tree, plus hashes of settings, sessions and the knowledge graph; `HDF5Memory::verify` names any edited record |
| **`provenance`** | Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof) | | **`provenance`** | Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof) |
| **`anomaly`** | Write rate limiting, 15 injection-pattern detectors, source-distribution analysis. Alerts never block a save; drain them with `take_anomaly_alerts` | | **`anomaly`** | Write rate limiting, 15 injection-pattern detectors, source-distribution analysis. Alerts never block a save; drain them with `take_anomaly_alerts` |
| **`openclaw`** | OpenClaw integration: MemoryBackend trait, Markdown ↔ HDF5 conversion | | **`openclaw`** | `ClawhdfBackend`: a Markdown-oriented backend (ingest by section, search, read back by path, export). Named for OpenClaw, but **not an OpenClaw plugin** — see [docs/openclaw.md](docs/openclaw.md) |
| **`vector_search`** | Flat cosine, pre-normed, SIMD, BLAS, GPU, parallel search paths | | **`vector_search`** | Flat cosine, pre-normed, SIMD, BLAS, GPU, parallel search paths |
| **`ivf` / `pq`** | Standalone IVF and IVF-PQ indexes (benchmarked to 100K vectors); not used by `HDF5Memory`, whose ANN index is HNSW | | **`ivf` / `pq`** | Standalone IVF and IVF-PQ indexes (benchmarked to 100K vectors); not used by `HDF5Memory`, whose ANN index is HNSW |
| **`bm25`** | Incremental Okapi BM25 inverted index, kept for the life of the store; optional stemming | | **`bm25`** | Incremental Okapi BM25 inverted index, kept for the life of the store; optional stemming |
@@ -447,7 +448,7 @@ let work = memory.search(
); );
// Re-rank by relevance, recency, source authority and activation, then drop // Re-rank by relevance, recency, source authority and activation, then drop
// low-confidence results — the pipeline the OpenClaw backend runs. // low-confidence results — the pipeline ClawhdfBackend runs.
let careful = memory.search( let careful = memory.search(
&query_embedding, &query_embedding,
"user preferences", "user preferences",
@@ -457,6 +458,35 @@ let careful = memory.search(
); );
``` ```
### Signed Checkpoints
```rust
use clawhdf5_agent::signing;
// Once, somewhere safe: keep the secret key, publish the public key.
let key = signing::generate_key();
let public = key.verifying_key();
// Every checkpoint is signed from now on. The key is never written to disk;
// a signed store refuses to checkpoint without it.
memory.set_signing_key(key);
memory.flush_wal()?;
// Anyone holding the public key can check the file, e.g. after copying it.
let report = HDF5Memory::verify(std::path::Path::new("agent.h5"), &public)?;
assert!(report.is_valid());
// On a tampered file: report.changed_records lists the records that differ.
```
The signature covers every record (text, embedding as stored, channel,
timestamp, session, tags, deleted flag, activation), the store's settings,
its sessions and its knowledge graph — a change made with any tool is caught.
It covers checkpoints, not saves still in the WAL
(`report.wal_entries_unsigned` counts those). CLI: `clawhdf5-cli keygen`,
`--signing-key <file>` on writing commands, and `verify --public-key`.
Signing adds about 20% to a checkpoint and 32 bytes per record to the file
([BENCHMARKS.md § Signed checkpoints](BENCHMARKS.md#signed-checkpoints)).
### Knowledge Graph ### Knowledge Graph
```rust ```rust
@@ -527,7 +557,13 @@ let ids = index.range_query(1700000000.0, 1700010800.0);
let recent = index.latest(10); let recent = index.latest(10);
``` ```
### OpenClaw Integration ### Markdown Backend
`ClawhdfBackend` ingests Markdown by section and searches it with the full
pipeline. It is a library API — clawhdf5 is **not** an OpenClaw memory plugin
([docs/openclaw.md](docs/openclaw.md)). Sections stored this way carry no
embedding, so their search is keyword-only unless you save records with
vectors through `save_entry`.
```rust ```rust
use clawhdf5_agent::openclaw::*; use clawhdf5_agent::openclaw::*;
@@ -821,10 +857,10 @@ See [ROADMAP.md](ROADMAP.md) for the full implementation tracker.
- ✅ Temporal reasoning with sub-µs queries - ✅ Temporal reasoning with sub-µs queries
- ✅ Memory security + anomaly detection - ✅ Memory security + anomaly detection
- ✅ Multi-modal memory (text/image/audio/video) - ✅ Multi-modal memory (text/image/audio/video)
- ✅ OpenClaw integration layer - ✅ Markdown ingest/export backend (`ClawhdfBackend`); an OpenClaw plugin was never built — see [docs/openclaw.md](docs/openclaw.md)
- ✅ Comprehensive Criterion benchmarks - ✅ Comprehensive Criterion benchmarks
**Phase 2** — MemoryArena and LongMemEval academic benchmarks are done (see [BENCHMARKS.md](BENCHMARKS.md), reproduced on a second machine); remaining: publish the OpenClaw TypeScript bridge to npm, crates.io/PyPI publishing. **Phase 2** — MemoryArena and LongMemEval academic benchmarks are done (see [BENCHMARKS.md](BENCHMARKS.md), reproduced on a second machine); remaining: crates.io/PyPI publishing. The Node bindings are unpublished and known to be broken ([known issues](docs/known-issues.md)).
--- ---
+14 -8
View File
@@ -105,24 +105,30 @@
--- ---
## Track 7: OpenClaw Integration ## Track 7: OpenClaw Integration — withdrawn (2026-09-25)
**Status:** 🟢 Complete **Status:** ⚪ Withdrawn (the items below were library work; no OpenClaw integration shipped)
**Priority:** Critical (for adoption) **Priority:** Critical (for adoption)
**Crates:** `clawhdf5-agent`, `clawhdf5-napi` **Crates:** `clawhdf5-agent`, `clawhdf5-napi`
- [x] **7.1** Memory backend trait — MemoryBackend with search/get/write/ingest/export/stats - [x] **7.1** Memory backend trait — MemoryBackend with search/get/write/ingest/export/stats
- [x] **7.2** Hybrid retrieval pipeline — ClawhdfBackend wires RRF → reranker → confidence rejection - [x] **7.2** Hybrid retrieval pipeline — ClawhdfBackend wires RRF → reranker → confidence rejection
- [x] **7.3** Markdown import/export — MarkdownParser + MarkdownExporter with line tracking + metadata - [x] **7.3** Markdown import/export — MarkdownParser + MarkdownExporter with line tracking + metadata
- [x] **7.4** memory_search tool — backed by full hybrid retrieval pipeline - [x] **7.4** `search()` — backed by the full hybrid retrieval pipeline (a Rust method; no OpenClaw tool was ever registered)
- [x] **7.5** memory_get tool — get() with path + line range support - [x] **7.5** `get()` — read back by path, with a line slice (not an OpenClaw tool either)
- [x] **7.6** Compaction integration — run_compaction() (decay + compact + WAL flush), run_consolidation() (hippocampal engine), tick_session(), flush_wal() - [x] **7.6** Compaction integration — run_compaction() (decay + compact + WAL flush), run_consolidation() (hippocampal engine), tick_session(), flush_wal()
- [x] **7.7** Config surface — `memory.backend = "clawhdf5"` schema documented in docs/openclaw-config.md - [ ] **7.7** ~~Config surface — `memory.backend = "clawhdf5"`~~ — never valid OpenClaw config; docs removed
- [x] **7.8** Documentation + migration guide — docs/migration-guide.md, docs/openclaw-integration.md (architecture, full API reference, code patterns) - [ ] **7.8** ~~Documentation + migration guide~~ — removed: they described an integration that never worked
**Node.js bridge:** `clawhdf5-napi` (napi-rs) → `@redclaw/clawhdf5` npm package with full TypeScript types. **Node.js bridge:** `clawhdf5-napi` (napi-rs) and a TypeScript wrapper in `packages/clawhdf5-node` exist but are unpublished, untested in CI and known to be broken (docs/known-issues.md).
--- ---
> **Withdrawn.** None of this track produced a working OpenClaw integration: no
> plugin was built, the documented `memory.backend = "clawhdf5"` config was never
> valid in any OpenClaw release, and the Node package was never published. The
> Rust `ClawhdfBackend` remains as a library API. Not pursued for now; see
> [docs/openclaw.md](docs/openclaw.md) for what a plugin would need today.
## Track 8: Benchmarking & Validation ## Track 8: Benchmarking & Validation
**Status:** 🟢 Complete **Status:** 🟢 Complete
**Priority:** High **Priority:** High
@@ -142,7 +148,7 @@
**Phase 1:** ~~Tracks 1, 2, 3 — core memory intelligence~~ 🟢 Complete **Phase 1:** ~~Tracks 1, 2, 3 — core memory intelligence~~ 🟢 Complete
**Phase 2:** ~~Track 4 (temporal) + Track 5 (security)~~ 🟢 Complete **Phase 2:** ~~Track 4 (temporal) + Track 5 (security)~~ 🟢 Complete
**Phase 3:** ~~Track 6 (multi-modal) + Track 7 (OpenClaw integration)~~ 🟢 Complete **Phase 3:** ~~Track 6 (multi-modal)~~ 🟢 Complete; Track 7 (OpenClaw integration) withdrawn
**Phase 4:** ~~Track 8 (benchmarking + validation)~~ 🟢 Complete **Phase 4:** ~~Track 8 (benchmarking + validation)~~ 🟢 Complete
All 8 tracks delivered. 1,650+ tests passing, zero clippy warnings. All 8 tracks delivered. 1,650+ tests passing, zero clippy warnings.
+4
View File
@@ -19,6 +19,10 @@ clawhdf5-ann = { path = "../clawhdf5-ann", version = "2.7.0", optional = true }
clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.7.0", optional = true, default-features = false } clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.7.0", optional = true, default-features = false }
serde = { workspace = true } serde = { workspace = true }
byteorder = "1" byteorder = "1"
# Signed checkpoints (MemoryConfig-independent; see `signing`). Pure Rust.
ed25519-dalek = { version = "2", features = ["rand_core"] }
sha2 = "0.10"
rand_core = { version = "0.6", features = ["getrandom"] }
half = { workspace = true, optional = true } half = { workspace = true, optional = true }
rayon = { version = "1", optional = true } rayon = { version = "1", optional = true }
matrixmultiply = { version = "0.3", optional = true } matrixmultiply = { version = "0.3", optional = true }
+1 -1
View File
@@ -203,7 +203,7 @@ impl ImportanceScorer {
/// Novelty score: 1.0 − max cosine similarity against all existing records. /// Novelty score: 1.0 − max cosine similarity against all existing records.
/// Returns 1.0 when there are no existing memories. /// Returns 1.0 when there are no existing memories.
/// ///
/// Same result as [`Self::cosine_similarity`] against each record, but the /// Same result as the reference cosine similarity against each record, but the
/// new embedding's norm is computed once rather than per record, each /// new embedding's norm is computed once rather than per record, each
/// record costs one fused pass (dot product and its norm together) rather /// record costs one fused pass (dot product and its norm together) rather
/// than three, and a large working set is scored in parallel. Every insert /// than three, and a large working set is scored in parallel. Every insert
+72 -1
View File
@@ -36,6 +36,7 @@ pub mod reranker;
pub mod schema; pub mod schema;
pub mod search; pub mod search;
pub mod session; pub mod session;
pub mod signing;
pub mod storage; pub mod storage;
mod store_lock; mod store_lock;
pub mod temporal; pub mod temporal;
@@ -78,6 +79,7 @@ pub use session::{SessionCache, SessionEntry};
// --- Error type --- // --- Error type ---
#[derive(Debug)] #[derive(Debug)]
#[non_exhaustive]
pub enum MemoryError { pub enum MemoryError {
Io(std::io::Error), Io(std::io::Error),
Hdf5(String), Hdf5(String),
@@ -88,6 +90,11 @@ pub enum MemoryError {
/// A record the store cannot hold as given, e.g. an embedding value /// A record the store cannot hold as given, e.g. an embedding value
/// outside the half-precision range of a `float16` store. /// outside the half-precision range of a `float16` store.
InvalidEntry(String), InvalidEntry(String),
/// The store's checkpoints are signed and no signing key is set, so a
/// checkpoint would leave it unsigned. Set the key with
/// [`HDF5Memory::set_signing_key`], or drop the signature on purpose with
/// [`HDF5Memory::remove_signature`].
SigningKeyRequired(String),
} }
impl std::fmt::Display for MemoryError { impl std::fmt::Display for MemoryError {
@@ -99,6 +106,7 @@ impl std::fmt::Display for MemoryError {
MemoryError::NotFound(e) => write!(f, "not found: {e}"), MemoryError::NotFound(e) => write!(f, "not found: {e}"),
MemoryError::Locked(e) => write!(f, "store is locked: {e}"), MemoryError::Locked(e) => write!(f, "store is locked: {e}"),
MemoryError::InvalidEntry(e) => write!(f, "invalid entry: {e}"), MemoryError::InvalidEntry(e) => write!(f, "invalid entry: {e}"),
MemoryError::SigningKeyRequired(e) => write!(f, "signing key required: {e}"),
} }
} }
} }
@@ -317,6 +325,12 @@ pub struct HDF5Memory {
activations_dirty: bool, activations_dirty: bool,
/// Opened with [`HDF5Memory::open_read_only`]: nothing may reach the disk. /// Opened with [`HDF5Memory::open_read_only`]: nothing may reach the disk.
read_only: bool, read_only: bool,
/// Key that signs every checkpoint; never persisted. See
/// [`HDF5Memory::set_signing_key`].
signing_key: Option<signing::SigningKey>,
/// Checkpoints of this store are signed: the file on disk is, or a key
/// has been set. A checkpoint without a key is then refused.
signed: bool,
/// A WAL that `open()` could not read and moved aside; see /// A WAL that `open()` could not read and moved aside; see
/// [`HDF5Memory::quarantined_wal`]. /// [`HDF5Memory::quarantined_wal`].
quarantined_wal: Option<PathBuf>, quarantined_wal: Option<PathBuf>,
@@ -372,6 +386,8 @@ impl HDF5Memory {
bm25_filter: bm25::TokenFilter::default(), bm25_filter: bm25::TokenFilter::default(),
activations_dirty: false, activations_dirty: false,
read_only: false, read_only: false,
signing_key: None,
signed: false,
quarantined_wal: None, quarantined_wal: None,
_lock: Some(lock), _lock: Some(lock),
}) })
@@ -550,6 +566,8 @@ impl HDF5Memory {
bm25_filter: bm25::TokenFilter::default(), bm25_filter: bm25::TokenFilter::default(),
activations_dirty: false, activations_dirty: false,
read_only, read_only,
signing_key: None,
signed: checkpoint.signed,
quarantined_wal, quarantined_wal,
_lock: lock, _lock: lock,
}) })
@@ -710,6 +728,39 @@ impl HDF5Memory {
} }
} }
/// Sign every checkpoint from now on with `key` (Ed25519). The key is
/// never written anywhere; set it again after every `open`. Once a store
/// is signed, a checkpoint without the key is refused
/// ([`MemoryError::SigningKeyRequired`]) rather than silently leaving it
/// unsigned. Setting a different key re-signs the store under that key
/// from the next checkpoint; a verifier trusting the old key will then
/// reject it, which is the point. Call [`AgentMemory::flush_wal`] to sign
/// right away.
pub fn set_signing_key(&mut self, key: signing::SigningKey) {
self.signing_key = Some(key);
self.signed = true;
}
/// Stop signing: the next checkpoint writes the store unsigned. The
/// deliberate way out of [`MemoryError::SigningKeyRequired`].
pub fn remove_signature(&mut self) {
self.signing_key = None;
self.signed = false;
}
/// Checkpoints of this store are signed (on disk, or from the next
/// checkpoint because a key has been set).
pub fn is_signed(&self) -> bool {
self.signed
}
/// Check the checkpoint at `path` against the public key the caller
/// trusts; see [`signing::verify_store`]. Reads the file only: it works
/// on a store another process has open.
pub fn verify(path: &Path, trusted: &signing::VerifyingKey) -> Result<signing::VerifyReport> {
signing::verify_store(path, trusted)
}
/// Flush current state to disk and truncate the WAL. /// Flush current state to disk and truncate the WAL.
/// ///
/// Every code path that persists the full cache to the .h5 file must /// Every code path that persists the full cache to the .h5 file must
@@ -725,10 +776,28 @@ impl HDF5Memory {
// Record which WAL prefix this checkpoint contains, so a crash before // Record which WAL prefix this checkpoint contains, so a crash before
// the truncate below can't replay those entries a second time. // the truncate below can't replay those entries a second time.
let wal_applied = self.wal.as_ref().map(|w| w.mark()); let wal_applied = self.wal.as_ref().map(|w| w.mark());
let signature = match &self.signing_key {
Some(key) => Some(signing::sign(
key,
&self.config,
&self.cache,
&self.sessions,
&self.knowledge,
wal_applied,
)),
None if self.signed => {
return Err(MemoryError::SigningKeyRequired(format!(
"{} is signed; set its signing key before a checkpoint \
(saves so far are held in the WAL or in memory)",
self.config.path.display()
)));
}
None => None,
};
// Written before the .h5 so a crash in between leaves a sidecar whose // Written before the .h5 so a crash in between leaves a sidecar whose
// generation matches no checkpoint (ignored), never the reverse. // generation matches no checkpoint (ignored), never the reverse.
let ann_generation = self.persist_vector_index(); let ann_generation = self.persist_vector_index();
storage::write_to_disk_with_meta( storage::write_to_disk_signed(
&self.config.path, &self.config.path,
&self.config, &self.config,
&self.cache, &self.cache,
@@ -737,7 +806,9 @@ impl HDF5Memory {
&schema::CheckpointMeta { &schema::CheckpointMeta {
wal_applied, wal_applied,
ann_generation, ann_generation,
signed: signature.is_some(),
}, },
signature.as_ref(),
)?; )?;
if let Some(ref mut w) = self.wal { if let Some(ref mut w) = self.wal {
w.truncate()?; w.truncate()?;
+11 -8
View File
@@ -1,10 +1,12 @@
//! OpenClaw Integration Layer. //! A Markdown-oriented memory backend over [`crate::HDF5Memory`].
//! //!
//! Bridge between OpenClaw agent gateway (Markdown + sqlite-vec) and the //! Named for OpenClaw, whose workspace memory is Markdown, but **not an
//! clawhdf5 HDF5-backed memory backend. Provides: //! OpenClaw plugin**: nothing here registers with OpenClaw, and the
//! integration it was written for never worked (see `docs/openclaw.md`).
//! Provides:
//! //!
//! - [`MemoryBackend`] — the trait OpenClaw implements against. //! - [`MemoryBackend`] — search / read back / write / ingest / export.
//! - [`ClawhdfBackend`] — concrete HDF5-backed implementation. //! - [`ClawhdfBackend`] — the HDF5-backed implementation.
//! - [`MarkdownParser`] — splits Markdown into [`MarkdownSection`] records. //! - [`MarkdownParser`] — splits Markdown into [`MarkdownSection`] records.
//! - [`MarkdownExporter`] — renders sections back to Markdown text. //! - [`MarkdownExporter`] — renders sections back to Markdown text.
@@ -61,7 +63,8 @@ pub struct BackendStats {
// MemoryBackend trait // MemoryBackend trait
// ───────────────────────────────────────────────────────────────────────────── // ─────────────────────────────────────────────────────────────────────────────
/// Interface that OpenClaw uses to interact with a memory backend. /// A Markdown-oriented memory backend: search, read back by path, write,
/// ingest and export.
/// ///
/// Implementors provide persistent storage, full-text + vector search, /// Implementors provide persistent storage, full-text + vector search,
/// Markdown ingestion / export, and statistics. /// Markdown ingestion / export, and statistics.
@@ -318,7 +321,7 @@ impl MarkdownExporter {
/// ///
/// # Path mapping /// # Path mapping
/// ///
/// OpenClaw addresses memories by file path (e.g. `"memory/user.md"`). /// Memories are addressed by file path (e.g. `"memory/user.md"`).
/// Internally every [`MemoryEntry`] stores the originating path as its /// Internally every [`MemoryEntry`] stores the originating path as its
/// `source_channel`. Section sub-paths are stored as /// `source_channel`. Section sub-paths are stored as
/// `"<path>::<heading>"`. /// `"<path>::<heading>"`.
@@ -421,7 +424,7 @@ impl ClawhdfBackend {
// ── Compaction & Consolidation hooks (7.6) ──────────────────────────── // ── Compaction & Consolidation hooks (7.6) ────────────────────────────
/// Run a compaction cycle — called by OpenClaw during session compaction. /// Run a compaction cycle (decay, compaction, WAL flush).
/// ///
/// Sequence: /// Sequence:
/// 1. `tick_session()` — apply Hebbian decay to all activation weights. /// 1. `tick_session()` — apply Hebbian decay to all activation weights.
+1 -1
View File
@@ -2,7 +2,7 @@
//! //!
//! Records the origin, authorship, and a content hash of every memory chunk //! Records the origin, authorship, and a content hash of every memory chunk
//! so the system can detect *accidental* corruption and trace data lineage. //! so the system can detect *accidental* corruption and trace data lineage.
//! The hash is unkeyed (see [`fnv1a_64`]) — this is not a tamper-evidence or //! The hash is unkeyed (FNV-1a) — this is not a tamper-evidence or
//! authenticity guarantee. //! authenticity guarantee.
use std::collections::HashMap; use std::collections::HashMap;
+113 -1
View File
@@ -23,6 +23,7 @@ pub const ZEROCLAW_VERSION: &str = "0.8.0";
const WAL_APPLIED_LEN_ATTR: &str = "wal_applied_len"; const WAL_APPLIED_LEN_ATTR: &str = "wal_applied_len";
const WAL_APPLIED_CRC_ATTR: &str = "wal_applied_crc"; const WAL_APPLIED_CRC_ATTR: &str = "wal_applied_crc";
const ANN_GENERATION_ATTR: &str = "ann_generation"; const ANN_GENERATION_ATTR: &str = "ann_generation";
const SIG_VERSION_ATTR: &str = "sig_version";
/// Build a complete HDF5 file from the in-memory state. /// Build a complete HDF5 file from the in-memory state.
pub fn build_hdf5_file( pub fn build_hdf5_file(
@@ -46,7 +47,7 @@ pub fn build_hdf5_file_with_mark(
) -> Result<Vec<u8>, MemoryError> { ) -> Result<Vec<u8>, MemoryError> {
let meta = CheckpointMeta { let meta = CheckpointMeta {
wal_applied, wal_applied,
ann_generation: None, ..CheckpointMeta::default()
}; };
build_hdf5_file_with_meta(config, cache, sessions, knowledge, &meta) build_hdf5_file_with_meta(config, cache, sessions, knowledge, &meta)
} }
@@ -61,6 +62,10 @@ pub struct CheckpointMeta {
/// one left over from another checkpoint can never be attached to records /// one left over from another checkpoint can never be attached to records
/// it wasn't built from. /// it wasn't built from.
pub ann_generation: Option<u64>, pub ann_generation: Option<u64>,
/// The checkpoint carries an Ed25519 signature (see [`crate::signing`]).
/// Read-only: whether a checkpoint is *written* signed is decided by the
/// signature passed to [`build_hdf5_file_signed`].
pub signed: bool,
} }
/// [`build_hdf5_file`] with checkpoint bookkeeping. /// [`build_hdf5_file`] with checkpoint bookkeeping.
@@ -70,6 +75,19 @@ pub fn build_hdf5_file_with_meta(
sessions: &SessionCache, sessions: &SessionCache,
knowledge: &KnowledgeCache, knowledge: &KnowledgeCache,
checkpoint: &CheckpointMeta, checkpoint: &CheckpointMeta,
) -> Result<Vec<u8>, MemoryError> {
build_hdf5_file_signed(config, cache, sessions, knowledge, checkpoint, None)
}
/// [`build_hdf5_file_with_meta`], plus a signed manifest of the contents
/// (see [`crate::signing`]).
pub fn build_hdf5_file_signed(
config: &MemoryConfig,
cache: &MemoryCache,
sessions: &SessionCache,
knowledge: &KnowledgeCache,
checkpoint: &CheckpointMeta,
signature: Option<&crate::signing::StoredSignature>,
) -> Result<Vec<u8>, MemoryError> { ) -> Result<Vec<u8>, MemoryError> {
let wal_applied = checkpoint.wal_applied; let wal_applied = checkpoint.wal_applied;
let mut builder = clawhdf5::FileBuilder::new(); let mut builder = clawhdf5::FileBuilder::new();
@@ -130,11 +148,42 @@ pub fn build_hdf5_file_with_meta(
// round trip through every reader. // round trip through every reader.
meta.set_attr(ANN_GENERATION_ATTR, AttrValue::I64(generation as i64)); meta.set_attr(ANN_GENERATION_ATTR, AttrValue::I64(generation as i64));
} }
if let Some(sig) = signature {
use crate::signing::to_hex;
let m = &sig.manifest;
meta.set_attr(
SIG_VERSION_ATTR,
AttrValue::I64(crate::signing::MANIFEST_VERSION),
);
meta.set_attr("sig_algorithm", AttrValue::String("ed25519".into()));
meta.set_attr("sig_public_key", AttrValue::String(to_hex(&sig.public_key)));
meta.set_attr("sig_signature", AttrValue::String(to_hex(&sig.signature)));
meta.set_attr("sig_record_count", AttrValue::I64(m.record_count as i64));
meta.set_attr(
"sig_records_root",
AttrValue::String(to_hex(&m.records_root)),
);
meta.set_attr("sig_settings", AttrValue::String(to_hex(&m.settings)));
meta.set_attr("sig_sessions", AttrValue::String(to_hex(&m.sessions)));
meta.set_attr("sig_graph", AttrValue::String(to_hex(&m.graph)));
}
// Need at least one dataset in the group for it to be a proper group // Need at least one dataset in the group for it to be a proper group
meta.create_dataset("_marker").with_u8_data(&[1]).compact(); meta.create_dataset("_marker").with_u8_data(&[1]).compact();
let finished_meta = meta.finish(); let finished_meta = meta.finish();
builder.add_group(finished_meta); builder.add_group(finished_meta);
// /integrity: the signed per-record hashes, so verification can say
// which records changed.
if let Some(sig) = signature {
let mut group = builder.create_group("integrity");
let flat: Vec<u8> = sig.record_hashes.iter().flatten().copied().collect();
group
.create_dataset("record_hashes")
.with_u8_data(&flat)
.with_shape(&[sig.record_hashes.len() as u64, 32]);
builder.add_group(group.finish());
}
// /memory group // /memory group
build_memory_group(&mut builder, config, cache)?; build_memory_group(&mut builder, config, cache)?;
@@ -440,6 +489,64 @@ pub fn read_wal_mark(file: &clawhdf5::File) -> Option<WalMark> {
Some(WalMark { len, crc }) Some(WalMark { len, crc })
} }
/// Read a checkpoint's signature, if it has one. A signature whose
/// attributes are present but malformed is an error, not "unsigned".
pub fn read_signature(
file: &clawhdf5::File,
) -> Result<Option<crate::signing::StoredSignature>, MemoryError> {
use crate::signing::{Manifest, StoredSignature, from_hex};
let attrs = file
.group("meta")
.and_then(|g| g.attrs())
.map_err(|e| MemoryError::Schema(format!("cannot read /meta attrs: {e}")))?;
let version = match attrs.get(SIG_VERSION_ATTR) {
None => return Ok(None),
Some(AttrValue::I64(v)) => *v,
Some(_) => return Err(MemoryError::Schema("malformed sig_version".into())),
};
if version != crate::signing::MANIFEST_VERSION {
return Err(MemoryError::Schema(format!(
"unsupported signature version {version}"
)));
}
fn hex<const N: usize>(
attrs: &std::collections::HashMap<String, AttrValue>,
name: &str,
) -> Result<[u8; N], MemoryError> {
match attrs.get(name) {
Some(AttrValue::String(s)) => from_hex::<N>(s),
_ => None,
}
.ok_or_else(|| MemoryError::Schema(format!("malformed or missing {name}")))
}
let record_count = match attrs.get("sig_record_count") {
Some(AttrValue::I64(v)) if *v >= 0 => *v as u64,
_ => return Err(MemoryError::Schema("malformed sig_record_count".into())),
};
let group = file
.group("integrity")
.map_err(|e| MemoryError::Schema(format!("signed checkpoint without /integrity: {e}")))?;
let flat = read_u8_dataset(&group, "record_hashes")?;
if flat.len() % 32 != 0 {
return Err(MemoryError::Schema(
"/integrity/record_hashes is not a whole number of hashes".into(),
));
}
let record_hashes = flat.as_chunks::<32>().0.to_vec();
Ok(Some(StoredSignature {
manifest: Manifest {
record_count,
records_root: hex::<32>(&attrs, "sig_records_root")?,
settings: hex::<32>(&attrs, "sig_settings")?,
sessions: hex::<32>(&attrs, "sig_sessions")?,
graph: hex::<32>(&attrs, "sig_graph")?,
},
record_hashes,
public_key: hex::<32>(&attrs, "sig_public_key")?,
signature: hex::<64>(&attrs, "sig_signature")?,
}))
}
/// Read the checkpoint bookkeeping from `/meta`. /// Read the checkpoint bookkeeping from `/meta`.
pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta { pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta {
let ann_generation = file let ann_generation = file
@@ -450,9 +557,14 @@ pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta {
Some(AttrValue::I64(v)) => Some(*v as u64), Some(AttrValue::I64(v)) => Some(*v as u64),
_ => None, _ => None,
}); });
let signed = file
.group("meta")
.and_then(|g| g.attrs())
.is_ok_and(|attrs| attrs.contains_key(SIG_VERSION_ATTR));
CheckpointMeta { CheckpointMeta {
wal_applied: read_wal_mark(file), wal_applied: read_wal_mark(file),
ann_generation, ann_generation,
signed,
} }
} }
+419
View File
@@ -0,0 +1,419 @@
//! Ed25519-signed checkpoints.
//!
//! When a signing key is set ([`crate::HDF5Memory::set_signing_key`]), every
//! checkpoint writes a signed manifest of the store: a SHA-256 per memory
//! record rolled into a Merkle root, plus hashes of the store's settings, its
//! sessions and its knowledge graph. [`verify_store`] recomputes all of it from
//! the file and checks the signature against a public key the caller trusts,
//! so any change to the checkpointed file — a record's text or embedding, a
//! setting, a session, a graph edge, made through this crate or any other HDF5
//! tool — is detected, and the per-record hashes say which records changed.
//!
//! What it does not cover: saves still only in the WAL (made since the last
//! checkpoint). [`VerifyReport::wal_entries_unsigned`] counts them.
//!
//! The hashes cover exactly what the file persists, in the form the loader
//! returns it, so a store verifies after any number of reopen/checkpoint
//! cycles. Derived data (L2 norms, the vector index) is not covered; it is
//! recomputed from covered data.
use ed25519_dalek::{Signature, Signer, Verifier};
pub use ed25519_dalek::{SigningKey, VerifyingKey};
use sha2::{Digest, Sha256};
use crate::MemoryConfig;
use crate::cache::MemoryCache;
use crate::knowledge::KnowledgeCache;
use crate::session::SessionCache;
use crate::wal::WalMark;
/// Version of the manifest encoding; part of what is signed.
pub const MANIFEST_VERSION: i64 = 1;
type Hash = [u8; 32];
/// The hashes a signature covers.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Manifest {
pub record_count: u64,
/// Merkle root over the per-record hashes.
pub records_root: Hash,
/// Settings persisted in `/meta`, plus the checkpoint's WAL mark.
pub settings: Hash,
pub sessions: Hash,
pub graph: Hash,
}
impl Manifest {
/// The exact bytes that are signed.
pub fn signed_bytes(&self) -> Vec<u8> {
let mut m = Vec::with_capacity(160);
m.extend_from_slice(b"clawhdf5-agent signed checkpoint\0");
m.extend_from_slice(&MANIFEST_VERSION.to_le_bytes());
m.extend_from_slice(&self.record_count.to_le_bytes());
m.extend_from_slice(&self.records_root);
m.extend_from_slice(&self.settings);
m.extend_from_slice(&self.sessions);
m.extend_from_slice(&self.graph);
m
}
}
/// A signature as stored in a checkpoint.
#[derive(Debug, Clone)]
pub struct StoredSignature {
pub manifest: Manifest,
pub record_hashes: Vec<Hash>,
pub public_key: [u8; 32],
pub signature: [u8; 64],
}
/// Build the manifest (and per-record hashes) for the state about to be
/// checkpointed, and sign it.
pub fn sign(
key: &SigningKey,
config: &MemoryConfig,
cache: &MemoryCache,
sessions: &SessionCache,
knowledge: &KnowledgeCache,
wal_applied: Option<WalMark>,
) -> StoredSignature {
let (manifest, record_hashes) = manifest(config, cache, sessions, knowledge, wal_applied);
let signature = key.sign(&manifest.signed_bytes()).to_bytes();
StoredSignature {
manifest,
record_hashes,
public_key: key.verifying_key().to_bytes(),
signature,
}
}
/// Compute the manifest of a store's state.
pub fn manifest(
config: &MemoryConfig,
cache: &MemoryCache,
sessions: &SessionCache,
knowledge: &KnowledgeCache,
wal_applied: Option<WalMark>,
) -> (Manifest, Vec<Hash>) {
let record_hashes: Vec<Hash> = (0..cache.len()).map(|i| record_hash(cache, i)).collect();
let manifest = Manifest {
record_count: cache.len() as u64,
records_root: merkle_root(&record_hashes),
settings: settings_hash(config, wal_applied),
sessions: sessions_hash(sessions),
graph: graph_hash(knowledge),
};
(manifest, record_hashes)
}
// ---------------------------------------------------------------------------
// Canonical encoding
// ---------------------------------------------------------------------------
/// A SHA-256 over length-prefixed fields, so no two different field lists
/// hash the same bytes.
struct Fields(Sha256);
impl Fields {
fn new(domain: &str) -> Self {
let mut h = Sha256::new();
h.update((domain.len() as u64).to_le_bytes());
h.update(domain.as_bytes());
Self(h)
}
fn bytes(&mut self, b: &[u8]) -> &mut Self {
self.0.update((b.len() as u64).to_le_bytes());
self.0.update(b);
self
}
/// Strings as the loader returns them: stored null-padded, so a trailing
/// NUL cannot survive a round trip and must not be part of the hash.
fn str(&mut self, s: &str) -> &mut Self {
self.bytes(s.trim_end_matches('\0').as_bytes())
}
fn u64(&mut self, v: u64) -> &mut Self {
self.0.update(v.to_le_bytes());
self
}
fn f64(&mut self, v: f64) -> &mut Self {
self.0.update(v.to_bits().to_le_bytes());
self
}
fn f32(&mut self, v: f32) -> &mut Self {
self.0.update(v.to_bits().to_le_bytes());
self
}
fn finish(self) -> Hash {
self.0.finalize().into()
}
}
/// Everything persisted about record `i`, including its position. The
/// embedding is hashed as the cache holds it — for a `float16` store that is
/// the half-rounded value the file holds.
fn record_hash(cache: &MemoryCache, i: usize) -> Hash {
let mut f = Fields::new("clawhdf5-agent/record");
f.u64(i as u64).str(&cache.chunks[i]);
let emb: Vec<u8> = cache.embeddings[i]
.iter()
.flat_map(|v| v.to_bits().to_le_bytes())
.collect();
f.bytes(&emb)
.str(&cache.source_channels[i])
.f64(cache.timestamps[i])
.str(&cache.session_ids[i])
.str(&cache.tags[i])
.u64(u64::from(cache.tombstones[i]))
.f32(cache.activation_weights[i]);
f.finish()
}
/// Binary Merkle tree: leaves are the record hashes; a parent hashes its two
/// children with a node prefix; an odd node is carried up unchanged.
fn merkle_root(leaves: &[Hash]) -> Hash {
if leaves.is_empty() {
return Fields::new("clawhdf5-agent/merkle-empty").finish();
}
let mut level: Vec<Hash> = leaves.to_vec();
while level.len() > 1 {
level = level
.chunks(2)
.map(|pair| match pair {
[l, r] => {
let mut h = Sha256::new();
h.update([1u8]);
h.update(l);
h.update(r);
h.finalize().into()
}
[only] => *only,
_ => unreachable!(),
})
.collect();
}
level[0]
}
fn settings_hash(c: &MemoryConfig, wal_applied: Option<WalMark>) -> Hash {
let mut f = Fields::new("clawhdf5-agent/settings");
f.str(crate::schema::SCHEMA_VERSION)
.str(&c.created_at)
.str(&c.agent_id)
.str(&c.embedder)
.u64(c.embedding_dim as u64)
.u64(c.chunk_size as u64)
.u64(c.overlap as u64)
.u64(u64::from(c.float16))
.u64(u64::from(c.compression))
.u64(u64::from(c.compression_level))
.f32(c.compact_threshold)
.f32(c.hebbian_boost)
.f32(c.decay_factor)
.u64(u64::from(c.wal_enabled))
.u64(c.wal_max_entries as u64)
.u64(u64::from(c.quantized_index))
.u64(c.hnsw_m as u64)
.u64(c.hnsw_ef_construction as u64)
.u64(c.hnsw_ef_search as u64);
// An empty mark is not written to the file, so it must hash as none.
match wal_applied.filter(|m| m.len > 0) {
Some(m) => f.u64(1).u64(m.len).u64(u64::from(m.crc)),
None => f.u64(0),
};
f.finish()
}
fn sessions_hash(s: &SessionCache) -> Hash {
let mut f = Fields::new("clawhdf5-agent/sessions");
f.u64(s.entries.len() as u64);
for (i, e) in s.entries.iter().enumerate() {
f.str(&e.id)
.u64(e.start_idx)
.u64(e.end_idx)
.str(&e.channel)
.f64(e.ts)
.str(s.summaries.get(i).map(String::as_str).unwrap_or(""));
}
f.finish()
}
fn graph_hash(k: &KnowledgeCache) -> Hash {
let mut f = Fields::new("clawhdf5-agent/graph");
f.u64(k.entities.len() as u64);
for e in &k.entities {
f.u64(e.id)
.str(&e.name)
.str(&e.entity_type)
.u64(e.embedding_idx as u64);
}
f.u64(k.relations.len() as u64);
for r in &k.relations {
f.u64(r.src)
.u64(r.tgt)
.str(&r.relation)
.f32(r.weight)
.f64(r.ts);
}
f.u64(k.alias_strings.len() as u64);
for (s, id) in k.alias_strings.iter().zip(&k.alias_entity_ids) {
f.str(s).u64(*id as u64);
}
f.finish()
}
// ---------------------------------------------------------------------------
// Verification
// ---------------------------------------------------------------------------
/// The outcome of [`verify_store`].
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct VerifyReport {
/// The checkpoint carries a signature.
pub signed: bool,
/// The signature was made by the key the caller trusts.
pub key_matches: bool,
/// The signature over the stored manifest is valid.
pub signature_valid: bool,
/// The file's current contents match the signed manifest.
pub records_match: bool,
pub settings_match: bool,
pub sessions_match: bool,
pub graph_match: bool,
/// Records whose contents differ from what was signed (by position),
/// when the stored per-record hashes are themselves authentic.
pub changed_records: Vec<usize>,
/// Records in the file versus in the signed manifest.
pub record_count: u64,
pub signed_record_count: u64,
/// The public key the checkpoint claims to be signed by.
pub public_key: Option<[u8; 32]>,
/// Saves in the WAL after the checkpoint: not covered by the signature.
pub wal_entries_unsigned: usize,
}
impl VerifyReport {
/// Signed by the trusted key, signature valid, and every part of the
/// file unchanged since it was signed.
pub fn is_valid(&self) -> bool {
self.signed
&& self.key_matches
&& self.signature_valid
&& self.records_match
&& self.settings_match
&& self.sessions_match
&& self.graph_match
}
}
/// Check a store file against the public key the caller trusts.
///
/// Reads the checkpoint (not the WAL), recomputes every hash from its
/// contents and checks the signature. Never writes.
pub fn verify_store(
path: &std::path::Path,
trusted: &VerifyingKey,
) -> Result<VerifyReport, crate::MemoryError> {
let file = clawhdf5::File::open(path)
.map_err(|e| crate::MemoryError::Hdf5(format!("cannot open {}: {e}", path.display())))?;
let (config, cache, sessions, knowledge) = crate::schema::validate_and_load(&file)?;
let checkpoint = crate::schema::read_checkpoint_meta(&file);
let stored = crate::schema::read_signature(&file)?;
let wal_entries_unsigned = count_wal_entries_after(path, checkpoint.wal_applied);
let (current, current_hashes) = manifest(
&config,
&cache,
&sessions,
&knowledge,
checkpoint.wal_applied,
);
let Some(stored) = stored else {
return Ok(VerifyReport {
signed: false,
key_matches: false,
signature_valid: false,
records_match: false,
settings_match: false,
sessions_match: false,
graph_match: false,
changed_records: Vec::new(),
record_count: current.record_count,
signed_record_count: 0,
public_key: None,
wal_entries_unsigned,
});
};
let key_matches = stored.public_key == trusted.to_bytes();
let signature_valid = trusted
.verify(
&stored.manifest.signed_bytes(),
&Signature::from_bytes(&stored.signature),
)
.is_ok();
// The stored per-record hashes can localise a change only if they are
// the ones that were signed.
let hashes_authentic = signature_valid
&& stored.record_hashes.len() as u64 == stored.manifest.record_count
&& merkle_root(&stored.record_hashes) == stored.manifest.records_root;
let changed_records = if hashes_authentic {
let n = current_hashes.len().max(stored.record_hashes.len());
(0..n)
.filter(|&i| current_hashes.get(i) != stored.record_hashes.get(i))
.collect()
} else {
Vec::new()
};
Ok(VerifyReport {
signed: true,
key_matches,
signature_valid,
records_match: signature_valid
&& current.record_count == stored.manifest.record_count
&& current.records_root == stored.manifest.records_root,
settings_match: signature_valid && current.settings == stored.manifest.settings,
sessions_match: signature_valid && current.sessions == stored.manifest.sessions,
graph_match: signature_valid && current.graph == stored.manifest.graph,
changed_records,
record_count: current.record_count,
signed_record_count: stored.manifest.record_count,
public_key: Some(stored.public_key),
wal_entries_unsigned,
})
}
fn count_wal_entries_after(store: &std::path::Path, mark: Option<WalMark>) -> usize {
let wal = store.with_extension("h5.wal");
if !wal.exists() {
return 0;
}
crate::wal::WalFile::read_entries_for_migration(&wal, mark)
.map(|e| e.len())
.unwrap_or(0)
}
/// A new random signing key from the operating system's RNG.
pub fn generate_key() -> SigningKey {
SigningKey::generate(&mut rand_core::OsRng)
}
/// Hex encoding for keys and signatures in attributes and the CLI.
pub fn to_hex(bytes: &[u8]) -> String {
bytes.iter().map(|b| format!("{b:02x}")).collect()
}
/// Parse hex into exactly `N` bytes.
pub fn from_hex<const N: usize>(s: &str) -> Option<[u8; N]> {
let s = s.trim();
if s.len() != 2 * N {
return None;
}
let mut out = [0u8; N];
for (i, byte) in out.iter_mut().enumerate() {
*byte = u8::from_str_radix(&s[2 * i..2 * i + 2], 16).ok()?;
}
Some(out)
}
+16 -2
View File
@@ -36,7 +36,7 @@ pub fn write_to_disk_with_mark(
) -> Result<(), MemoryError> { ) -> Result<(), MemoryError> {
let meta = schema::CheckpointMeta { let meta = schema::CheckpointMeta {
wal_applied, wal_applied,
ann_generation: None, ..schema::CheckpointMeta::default()
}; };
write_to_disk_with_meta(path, config, cache, sessions, knowledge, &meta) write_to_disk_with_meta(path, config, cache, sessions, knowledge, &meta)
} }
@@ -50,7 +50,21 @@ pub fn write_to_disk_with_meta(
knowledge: &KnowledgeCache, knowledge: &KnowledgeCache,
checkpoint: &schema::CheckpointMeta, checkpoint: &schema::CheckpointMeta,
) -> Result<(), MemoryError> { ) -> Result<(), MemoryError> {
let bytes = schema::build_hdf5_file_with_meta(config, cache, sessions, knowledge, checkpoint)?; write_to_disk_signed(path, config, cache, sessions, knowledge, checkpoint, None)
}
/// [`write_to_disk_with_meta`] with a signed manifest of the contents.
pub fn write_to_disk_signed(
path: &Path,
config: &MemoryConfig,
cache: &MemoryCache,
sessions: &SessionCache,
knowledge: &KnowledgeCache,
checkpoint: &schema::CheckpointMeta,
signature: Option<&crate::signing::StoredSignature>,
) -> Result<(), MemoryError> {
let bytes =
schema::build_hdf5_file_signed(config, cache, sessions, knowledge, checkpoint, signature)?;
if bytes.is_empty() { if bytes.is_empty() {
return Err(MemoryError::Hdf5("build_hdf5_file produced 0 bytes".into())); return Err(MemoryError::Hdf5("build_hdf5_file produced 0 bytes".into()));
@@ -92,3 +92,64 @@ print(len(names))
assert!(n >= 10, "only {n} datasets"); assert!(n >= 10, "only {n} datasets");
} }
} }
#[test]
fn an_edit_made_with_h5py_breaks_the_signature_and_names_the_record() {
if !h5py_available() {
assert!(
std::env::var("CLAWHDF5_REQUIRE_INTEROP").as_deref() != Ok("1"),
"CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
);
eprintln!("SKIP: python3 with h5py not available");
return;
}
use clawhdf5_agent::signing::SigningKey;
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("signed.h5");
let key = SigningKey::from_bytes(&[42; 32]);
let mut m = HDF5Memory::create(MemoryConfig::new(path.clone(), "agent", 8)).unwrap();
m.set_signing_key(key.clone());
m.save_batch(
(0..10)
.map(|i| MemoryEntry {
chunk: format!("memory {i}"),
embedding: (0..8).map(|j| ((i * 8 + j) as f32).cos()).collect(),
source_channel: "test".into(),
timestamp: i as f64,
session_id: "s".into(),
tags: String::new(),
})
.collect(),
)
.unwrap();
drop(m);
assert!(
HDF5Memory::verify(&path, &key.verifying_key())
.unwrap()
.is_valid()
);
// Someone edits one timestamp in place with h5py.
let script = format!(
r#"
import h5py
with h5py.File("{}", "r+") as f:
ts = f["memory/timestamps"]
ts[3] = 12345.0
"#,
path.display()
);
let out = Command::new(python())
.args(["-c", &script])
.output()
.unwrap();
assert!(
out.status.success(),
"{}",
String::from_utf8_lossy(&out.stderr)
);
let r = HDF5Memory::verify(&path, &key.verifying_key()).unwrap();
assert!(r.signature_valid && !r.is_valid(), "{r:?}");
assert_eq!(r.changed_records, vec![3]);
}
+330
View File
@@ -0,0 +1,330 @@
//! Ed25519-signed checkpoints: `HDF5Memory::set_signing_key` and
//! `HDF5Memory::verify`.
use std::path::Path;
use clawhdf5_agent::signing::{SigningKey, VerifyReport, VerifyingKey};
use clawhdf5_agent::storage;
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, MemoryError, schema};
use tempfile::TempDir;
const DIM: usize = 16;
fn key(seed: u8) -> SigningKey {
SigningKey::from_bytes(&[seed; 32])
}
fn entry(i: usize, chunk: &str) -> MemoryEntry {
MemoryEntry {
chunk: chunk.to_string(),
embedding: (0..DIM)
.map(|j| ((i * DIM + j) as f32 * 0.37).sin())
.collect(),
source_channel: "chat".into(),
timestamp: 1_700_000_000.0 + i as f64,
session_id: format!("s{}", i % 3),
tags: format!("t{i}"),
}
}
/// Awkward strings on purpose: they must hash the same after a round trip.
const TEXTS: [&str; 6] = [
"plain text",
"ünïcödé — 日本語 🙂",
"",
"trailing spaces ",
"tab\tand\nnewline",
"x",
];
fn signed_store(dir: &TempDir, float16: bool, k: &SigningKey) -> std::path::PathBuf {
let mut cfg = MemoryConfig::new(dir.path().join("s.h5"), "agent", DIM);
cfg.float16 = float16;
let path = cfg.path.clone();
let mut m = HDF5Memory::create(cfg).unwrap();
m.set_signing_key(k.clone());
let entries = (0..30).map(|i| entry(i, TEXTS[i % TEXTS.len()])).collect();
m.save_batch(entries).unwrap();
// Some graph and a deleted record, so every part of the manifest is used.
let a = m.knowledge_mut().add_entity("Alice", "person", 0);
let b = m.knowledge_mut().add_entity("Acme", "org", -1);
m.knowledge_mut().add_relation(a, b, "works_at", 0.75);
m.sessions_mut()
.add_at("s0", 0, 9, "chat", "first session", 1_700_000_000.0);
m.delete(4).unwrap();
m.flush_wal().unwrap();
path
}
fn verify(path: &Path, k: &SigningKey) -> VerifyReport {
HDF5Memory::verify(path, &k.verifying_key()).unwrap()
}
#[test]
fn a_signed_store_verifies_through_reopen_and_checkpoint_cycles() {
for float16 in [true, false] {
let dir = TempDir::new().unwrap();
let k = key(7);
let path = signed_store(&dir, float16, &k);
let r = verify(&path, &k);
assert!(r.is_valid(), "float16={float16}: {r:?}");
assert_eq!(r.public_key, Some(k.verifying_key().to_bytes()));
assert_eq!(r.record_count, 30);
assert!(r.changed_records.is_empty());
// Reopen, change nothing, checkpoint again (with the key): still valid.
for _ in 0..3 {
let mut m = HDF5Memory::open(&path).unwrap();
assert!(m.is_signed());
m.set_signing_key(k.clone());
m.flush_wal().unwrap();
drop(m);
assert!(verify(&path, &k).is_valid());
}
// And after real changes, re-signed.
let mut m = HDF5Memory::open(&path).unwrap();
m.set_signing_key(k.clone());
m.save(entry(99, "added later")).unwrap();
m.hybrid_search(&entry(1, "").embedding, "text", 0.4, 0.6, 5);
m.flush_wal().unwrap();
drop(m);
let r = verify(&path, &k);
assert!(r.is_valid(), "{r:?}");
assert_eq!(r.record_count, 31);
}
}
#[test]
fn a_signed_store_refuses_to_checkpoint_without_its_key() {
let dir = TempDir::new().unwrap();
let k = key(1);
let path = signed_store(&dir, true, &k);
let mut m = HDF5Memory::open(&path).unwrap();
m.save(entry(50, "pending")).unwrap();
match m.flush_wal() {
Err(MemoryError::SigningKeyRequired(msg)) => assert!(msg.contains("signed"), "{msg}"),
other => panic!("expected SigningKeyRequired, got {other:?}"),
}
// The file is untouched and still valid; the save is still in the WAL.
let r = verify(&path, &k);
assert!(r.is_valid());
assert_eq!(r.wal_entries_unsigned, 1);
// Supplying the key lets the checkpoint through, signed.
m.set_signing_key(k.clone());
m.flush_wal().unwrap();
drop(m);
let r = verify(&path, &k);
assert!(r.is_valid());
assert_eq!((r.record_count, r.wal_entries_unsigned), (31, 0));
// Removing the signature on purpose writes it unsigned.
let mut m = HDF5Memory::open(&path).unwrap();
m.remove_signature();
m.flush_wal().unwrap();
drop(m);
let r = verify(&path, &k);
assert!(!r.signed && !r.is_valid());
assert!(!HDF5Memory::open(&path).unwrap().is_signed());
}
#[test]
fn the_wrong_key_does_not_verify_and_a_new_key_re_signs() {
let dir = TempDir::new().unwrap();
let (a, b) = (key(1), key(2));
let path = signed_store(&dir, true, &a);
let r = verify(&path, &b);
assert!(r.signed && !r.key_matches && !r.signature_valid && !r.is_valid());
let mut m = HDF5Memory::open(&path).unwrap();
m.set_signing_key(b.clone());
m.flush_wal().unwrap();
drop(m);
assert!(verify(&path, &b).is_valid());
assert!(!verify(&path, &a).is_valid());
}
/// Rewrite the store with changed contents but the *old* signature — what
/// someone with write access to the file, but not the key, can do.
fn tamper(path: &Path, change: impl FnOnce(&mut Tampered)) {
let file = clawhdf5::File::open(path).unwrap();
let (config, cache, sessions, knowledge) = schema::validate_and_load(&file).unwrap();
let checkpoint = schema::read_checkpoint_meta(&file);
let signature = schema::read_signature(&file).unwrap().unwrap();
drop(file);
let mut t = Tampered {
config,
cache,
sessions,
knowledge,
};
change(&mut t);
storage::write_to_disk_signed(
path,
&t.config,
&t.cache,
&t.sessions,
&t.knowledge,
&checkpoint,
Some(&signature),
)
.unwrap();
}
struct Tampered {
config: MemoryConfig,
cache: clawhdf5_agent::cache::MemoryCache,
sessions: clawhdf5_agent::SessionCache,
knowledge: clawhdf5_agent::knowledge::KnowledgeCache,
}
#[test]
fn every_kind_of_edit_is_detected_and_located() {
let k = key(3);
type Edit = Box<dyn FnOnce(&mut Tampered)>;
type Case = (&'static str, Edit, fn(&VerifyReport) -> bool);
let cases: Vec<Case> = vec![
(
"record text",
Box::new(|t: &mut Tampered| t.cache.chunks[7] = "rewritten".into()),
|r| !r.records_match && r.changed_records == vec![7],
),
(
"one embedding value",
Box::new(|t: &mut Tampered| {
let mut e = t.cache.embeddings[12].to_vec();
e[3] = 0.5;
t.cache.embeddings.set(12, &e);
}),
|r| r.changed_records == vec![12],
),
(
"undelete",
Box::new(|t: &mut Tampered| t.cache.tombstones[4] = 0),
|r| r.changed_records == vec![4],
),
(
"timestamp",
Box::new(|t: &mut Tampered| t.cache.timestamps[20] += 1.0),
|r| r.changed_records == vec![20],
),
(
"record appended",
Box::new(|t: &mut Tampered| {
t.cache.push(
"new".into(),
vec![0.1; DIM],
"x".into(),
1.0,
"s".into(),
"".into(),
);
}),
|r| !r.records_match && r.changed_records == vec![30] && r.record_count == 31,
),
(
"setting",
Box::new(|t: &mut Tampered| t.config.agent_id = "someone-else".into()),
|r| !r.settings_match && r.records_match,
),
(
"session summary",
Box::new(|t: &mut Tampered| t.sessions.summaries[0] = "edited".into()),
|r| !r.sessions_match && r.records_match,
),
(
"graph edge",
Box::new(|t: &mut Tampered| t.knowledge.relations[0].weight = 1.0),
|r| !r.graph_match && r.records_match,
),
];
for (name, edit, check) in cases {
let dir = TempDir::new().unwrap();
let path = signed_store(&dir, true, &k);
tamper(&path, edit);
let r = verify(&path, &k);
assert!(
r.signed && r.key_matches && r.signature_valid,
"{name}: {r:?}"
);
assert!(!r.is_valid(), "{name}: edit not detected: {r:?}");
assert!(check(&r), "{name}: {r:?}");
}
}
#[test]
fn a_forged_manifest_fails_the_signature() {
// Recomputing the hashes for tampered contents does not help without the
// key: the signature no longer matches the manifest.
let dir = TempDir::new().unwrap();
let k = key(5);
let path = signed_store(&dir, true, &k);
let file = clawhdf5::File::open(&path).unwrap();
let (config, mut cache, sessions, knowledge) = schema::validate_and_load(&file).unwrap();
let checkpoint = schema::read_checkpoint_meta(&file);
let mut sig = schema::read_signature(&file).unwrap().unwrap();
drop(file);
cache.chunks[0] = "forged".into();
// Re-sign with an attacker key, then splice the victim's public key back.
let forged = clawhdf5_agent::signing::sign(
&key(66),
&config,
&cache,
&sessions,
&knowledge,
checkpoint.wal_applied,
);
sig.manifest = forged.manifest;
sig.record_hashes = forged.record_hashes;
storage::write_to_disk_signed(
&path,
&config,
&cache,
&sessions,
&knowledge,
&checkpoint,
Some(&sig),
)
.unwrap();
let r = verify(&path, &k);
assert!(
r.key_matches && !r.signature_valid && !r.is_valid(),
"{r:?}"
);
}
#[test]
fn an_unsigned_store_reports_unsigned() {
let dir = TempDir::new().unwrap();
let mut m = HDF5Memory::create(MemoryConfig::new(dir.path().join("u.h5"), "a", DIM)).unwrap();
m.save_batch(vec![entry(0, "hello")]).unwrap();
drop(m);
let r = HDF5Memory::verify(&dir.path().join("u.h5"), &VerifyingKey::from(&key(1))).unwrap();
assert!(!r.signed && !r.is_valid());
assert_eq!(r.record_count, 1);
}
#[test]
fn nul_bytes_in_text_still_verify() {
// Strings are stored null-padded; the hash must follow what a reopened
// store actually holds, or an untouched store would fail to verify.
let dir = TempDir::new().unwrap();
let k = key(9);
let mut m = HDF5Memory::create(MemoryConfig::new(dir.path().join("n.h5"), "a", DIM)).unwrap();
m.set_signing_key(k.clone());
m.save_batch(vec![
entry(0, "inner\0nul"),
entry(1, "trailing nul\0"),
entry(2, "\0leading"),
])
.unwrap();
drop(m);
let r = verify(&dir.path().join("n.h5"), &k);
assert!(r.is_valid(), "{r:?}");
let m = HDF5Memory::open(&dir.path().join("n.h5")).unwrap();
eprintln!(
"reloaded: {:?}",
(0..3).map(|i| m.get_chunk(i)).collect::<Vec<_>>()
);
}
@@ -21,6 +21,7 @@
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --ann-only --uniform //! cargo run --release -p clawhdf5-bench --bin search_harness -- --ann-only --uniform
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full //! cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --options-study --full //! cargo run --release -p clawhdf5-bench --bin search_harness -- --options-study --full
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --signing-study --full
//! ``` //! ```
use std::time::{Duration, Instant}; use std::time::{Duration, Instant};
@@ -488,6 +489,81 @@ fn bench_end_to_end(n: usize, json: &mut Vec<serde_json::Value>) {
})); }));
} }
// ---------------------------------------------------------------------------
// Signing study: what does an Ed25519-signed checkpoint cost?
// ---------------------------------------------------------------------------
/// `--signing-study`: checkpoint time unsigned vs signed, `verify` time, and
/// the file-size cost of the stored per-record hashes. Default store
/// settings (float16, int8 index). Medians of five checkpoints / three
/// verifies.
fn signing_study(n: usize) {
use clawhdf5_agent::signing::SigningKey;
let data = make_dataset(n, 0x516 ^ n as u64);
let mut rng = Rng(9);
let entries: Vec<MemoryEntry> = data
.vectors
.iter()
.enumerate()
.map(|(i, v)| MemoryEntry {
chunk: text_for(data.cluster_of[i], i, &mut rng),
embedding: v.clone(),
source_channel: "bench".into(),
timestamp: i as f64,
session_id: format!("s{}", i % 50),
tags: format!("t{i}"),
})
.collect();
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("sign.h5");
let mut mem = HDF5Memory::create(MemoryConfig::new(path.clone(), "bench", DIM)).unwrap();
mem.save_batch(entries).unwrap();
std::hint::black_box(mem.hybrid_search(&data.queries[0], "", 1.0, 0.0, K));
let median = |mut v: Vec<Duration>| {
v.sort();
v[v.len() / 2]
};
let checkpoint = |mem: &mut HDF5Memory| {
median(
(0..5)
.map(|_| {
let t = Instant::now();
mem.flush_wal().unwrap();
t.elapsed()
})
.collect(),
)
};
let unsigned = checkpoint(&mut mem);
let unsigned_bytes = std::fs::metadata(&path).unwrap().len();
let key = SigningKey::from_bytes(&[7; 32]);
mem.set_signing_key(key.clone());
let signed = checkpoint(&mut mem);
let signed_bytes = std::fs::metadata(&path).unwrap().len();
drop(mem);
let vk = key.verifying_key();
let verify = median(
(0..3)
.map(|_| {
let t = Instant::now();
let r = HDF5Memory::verify(&path, &vk).unwrap();
let d = t.elapsed();
assert!(r.is_valid());
d
})
.collect(),
);
println!(
"| {n} | {:.1} | {:.1} | {:+.1} | {:.1} | {:+.2} |",
millis(unsigned),
millis(signed),
millis(signed) - millis(unsigned),
millis(verify),
(signed_bytes as f64 - unsigned_bytes as f64) / (1024.0 * 1024.0),
);
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Search options study: source filters, re-ranking, confidence rejection // Search options study: source filters, re-ranking, confidence rejection
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -960,6 +1036,21 @@ fn main() {
} }
return; return;
} }
if args.iter().any(|a| a == "--signing-study") {
println!("## Signed checkpoints ({DIM}-dim, float16, int8 index)\n");
println!(
"| N | checkpoint ms, unsigned | checkpoint ms, signed | signing adds ms | verify ms | file MiB added |"
);
println!("|---:|---:|---:|---:|---:|---:|");
for &n in if full {
&[1_000, 10_000, 100_000][..]
} else {
&[1_000, 10_000][..]
} {
signing_study(n);
}
return;
}
if args.iter().any(|a| a == "--options-study") { if args.iter().any(|a| a == "--options-study") {
println!("## Search options ({DIM}-dim, k = {K}, Hebbian boost off)\n"); println!("## Search options ({DIM}-dim, k = {K}, Hebbian boost off)\n");
println!("| N | options | filtered recall@10 | p50 ms | p99 ms |"); println!("| N | options | filtered recall@10 | p50 ms | p99 ms |");
+126 -16
View File
@@ -1,15 +1,22 @@
use std::path::PathBuf; use std::path::{Path, PathBuf};
use clap::{Parser, Subcommand}; use clap::{Parser, Subcommand};
use clawhdf5_agent::signing::{self, SigningKey, VerifyingKey};
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry}; use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
/// ClawhDF5 — HDF5-backed cognitive memory for AI agents /// ClawhDF5 — HDF5-backed cognitive memory for AI agents
#[derive(Parser)] #[derive(Parser)]
#[command(name = "clawhdf5", version, about)] #[command(name = "clawhdf5", version, about)]
struct Cli { struct Cli {
/// Path to the .h5 memory file /// Path to the .h5 memory file (not needed for `keygen`)
#[arg(short, long, env = "CLAWHDF5_PATH")] #[arg(short, long, env = "CLAWHDF5_PATH")]
path: PathBuf, path: Option<PathBuf>,
/// File holding an Ed25519 signing key (64 hex characters, from
/// `keygen`). Every checkpoint this command makes is then signed; a
/// signed store refuses to checkpoint without it.
#[arg(long, env = "CLAWHDF5_SIGNING_KEY", global = true)]
signing_key: Option<PathBuf>,
#[command(subcommand)] #[command(subcommand)]
command: Commands, command: Commands,
@@ -91,6 +98,38 @@ enum Commands {
/// Destination path /// Destination path
dest: PathBuf, dest: PathBuf,
}, },
/// Generate an Ed25519 signing key for signed checkpoints
Keygen {
/// Where to write the secret key (created new, owner-only on Unix)
#[arg(long)]
out: PathBuf,
},
/// Verify a signed store against a public key; exit status 2 if not valid
Verify {
/// The trusted public key: 64 hex characters, or a file holding them
#[arg(long)]
public_key: String,
},
}
fn read_signing_key(path: &Path) -> Result<SigningKey, Box<dyn std::error::Error>> {
let text = std::fs::read_to_string(path)
.map_err(|e| format!("cannot read signing key {}: {e}", path.display()))?;
let bytes = signing::from_hex::<32>(&text)
.ok_or_else(|| format!("{} is not a 64-hex-character key", path.display()))?;
Ok(SigningKey::from_bytes(&bytes))
}
/// Open for writing, with the signing key applied if one was given.
fn open_writable(
path: &Path,
key: &Option<SigningKey>,
) -> Result<HDF5Memory, Box<dyn std::error::Error>> {
let mut mem = HDF5Memory::open(path)?;
if let Some(k) = key {
mem.set_signing_key(k.clone());
}
Ok(mem)
} }
fn main() { fn main() {
@@ -103,6 +142,37 @@ fn main() {
} }
fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> { fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
if let Commands::Keygen { out } = &cli.command {
let key = signing::generate_key();
let mut opts = std::fs::OpenOptions::new();
opts.write(true).create_new(true);
#[cfg(unix)]
{
use std::os::unix::fs::OpenOptionsExt;
opts.mode(0o600);
}
use std::io::Write;
let mut f = opts
.open(out)
.map_err(|e| format!("cannot create {}: {e}", out.display()))?;
writeln!(f, "{}", signing::to_hex(&key.to_bytes()))?;
let j = serde_json::json!({
"status": "generated",
"secret_key_file": out.display().to_string(),
"public_key": signing::to_hex(&key.verifying_key().to_bytes()),
});
println!("{}", serde_json::to_string_pretty(&j)?);
return Ok(());
}
let path = cli
.path
.clone()
.ok_or("--path (or CLAWHDF5_PATH) is required")?;
let key = cli
.signing_key
.as_deref()
.map(read_signing_key)
.transpose()?;
match cli.command { match cli.command {
Commands::Create { Commands::Create {
agent_id, agent_id,
@@ -113,7 +183,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
f32, f32,
float16: _, float16: _,
} => { } => {
let mut config = MemoryConfig::new(cli.path.clone(), &agent_id, dim); let mut config = MemoryConfig::new(path.clone(), &agent_id, dim);
config.wal_enabled = wal; config.wal_enabled = wal;
// As with --f32-index: only ever switch the library default off. // As with --f32-index: only ever switch the library default off.
if f32 { if f32 {
@@ -127,15 +197,21 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
config.quantized_index = false; config.quantized_index = false;
} }
let config_quantized = config.quantized_index; let config_quantized = config.quantized_index;
let mem = HDF5Memory::create(config)?; let mut mem = HDF5Memory::create(config)?;
// Sign straight away, so the store is never on disk unsigned.
if let Some(k) = &key {
mem.set_signing_key(k.clone());
mem.flush_wal()?;
}
let j = serde_json::json!({ let j = serde_json::json!({
"status": "created", "status": "created",
"path": cli.path.display().to_string(), "path": path.display().to_string(),
"agent_id": agent_id, "agent_id": agent_id,
"embedding_dim": dim, "embedding_dim": dim,
"wal_enabled": wal, "wal_enabled": wal,
"quantized_index": config_quantized, "quantized_index": config_quantized,
"float16": config_float16, "float16": config_float16,
"signed": mem.is_signed(),
"count": mem.count(), "count": mem.count(),
}); });
println!("{}", serde_json::to_string_pretty(&j)?); println!("{}", serde_json::to_string_pretty(&j)?);
@@ -152,7 +228,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
} }
}; };
let entry: MemoryEntry = serde_json::from_str(&input)?; let entry: MemoryEntry = serde_json::from_str(&input)?;
let mut mem = HDF5Memory::open(&cli.path)?; let mut mem = open_writable(&path, &key)?;
let idx = mem.save(entry)?; let idx = mem.save(entry)?;
let j = serde_json::json!({ "status": "saved", "index": idx, "count": mem.count() }); let j = serde_json::json!({ "status": "saved", "index": idx, "count": mem.count() });
println!("{}", serde_json::to_string(&j)?); println!("{}", serde_json::to_string(&j)?);
@@ -166,7 +242,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
keyword_weight, keyword_weight,
} => { } => {
let emb: Vec<f32> = serde_json::from_str(&embedding)?; let emb: Vec<f32> = serde_json::from_str(&embedding)?;
let mut mem = HDF5Memory::open(&cli.path)?; let mut mem = open_writable(&path, &key)?;
let results = mem.hybrid_search(&emb, &query, vector_weight, keyword_weight, top_k); let results = mem.hybrid_search(&emb, &query, vector_weight, keyword_weight, top_k);
let j: Vec<serde_json::Value> = results let j: Vec<serde_json::Value> = results
.iter() .iter()
@@ -184,7 +260,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
} }
Commands::Recall { index } => { Commands::Recall { index } => {
let mem = HDF5Memory::open_read_only(&cli.path)?; let mem = HDF5Memory::open_read_only(&path)?;
match mem.get_chunk(index) { match mem.get_chunk(index) {
Some(content) => { Some(content) => {
let j = serde_json::json!({ "index": index, "chunk": content }); let j = serde_json::json!({ "index": index, "chunk": content });
@@ -198,22 +274,23 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
} }
Commands::Stats => { Commands::Stats => {
let mem = HDF5Memory::open_read_only(&cli.path)?; let mem = HDF5Memory::open_read_only(&path)?;
let cfg = mem.config(); let cfg = mem.config();
let j = serde_json::json!({ let j = serde_json::json!({
"path": cli.path.display().to_string(), "path": path.display().to_string(),
"agent_id": cfg.agent_id, "agent_id": cfg.agent_id,
"embedding_dim": cfg.embedding_dim, "embedding_dim": cfg.embedding_dim,
"count": mem.count(), "count": mem.count(),
"active": mem.count_active(), "active": mem.count_active(),
"wal_enabled": cfg.wal_enabled, "wal_enabled": cfg.wal_enabled,
"wal_pending": mem.wal_pending_count(), "wal_pending": mem.wal_pending_count(),
"signed": mem.is_signed(),
}); });
println!("{}", serde_json::to_string_pretty(&j)?); println!("{}", serde_json::to_string_pretty(&j)?);
} }
Commands::FlushWal => { Commands::FlushWal => {
let mut mem = HDF5Memory::open(&cli.path)?; let mut mem = open_writable(&path, &key)?;
let before = mem.wal_pending_count(); let before = mem.wal_pending_count();
mem.flush_wal()?; mem.flush_wal()?;
let j = serde_json::json!({ let j = serde_json::json!({
@@ -225,7 +302,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
} }
Commands::AgentsMd { output } => { Commands::AgentsMd { output } => {
let mem = HDF5Memory::open_read_only(&cli.path)?; let mem = HDF5Memory::open_read_only(&path)?;
let md = mem.generate_agents_md(); let md = mem.generate_agents_md();
match output { match output {
Some(p) => { Some(p) => {
@@ -237,7 +314,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
} }
Commands::Export => { Commands::Export => {
let mem = HDF5Memory::open_read_only(&cli.path)?; let mem = HDF5Memory::open_read_only(&path)?;
for i in 0..mem.count() { for i in 0..mem.count() {
if let Some(chunk) = mem.get_chunk(i) { if let Some(chunk) = mem.get_chunk(i) {
let j = serde_json::json!({ "index": i, "chunk": chunk }); let j = serde_json::json!({ "index": i, "chunk": chunk });
@@ -246,11 +323,44 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
} }
} }
Commands::Keygen { .. } => unreachable!("handled before opening a store"),
Commands::Verify { public_key } => {
let text = if Path::new(&public_key).is_file() {
std::fs::read_to_string(&public_key)?
} else {
public_key
};
let bytes = signing::from_hex::<32>(&text)
.ok_or("--public-key must be 64 hex characters or a file holding them")?;
let trusted = VerifyingKey::from_bytes(&bytes)?;
let r = HDF5Memory::verify(&path, &trusted)?;
let j = serde_json::json!({
"valid": r.is_valid(),
"signed": r.signed,
"key_matches": r.key_matches,
"signature_valid": r.signature_valid,
"records_match": r.records_match,
"settings_match": r.settings_match,
"sessions_match": r.sessions_match,
"graph_match": r.graph_match,
"changed_records": r.changed_records,
"record_count": r.record_count,
"signed_record_count": r.signed_record_count,
"signed_by": r.public_key.map(|k| signing::to_hex(&k)),
"wal_entries_unsigned": r.wal_entries_unsigned,
});
println!("{}", serde_json::to_string_pretty(&j)?);
if !r.is_valid() {
std::process::exit(2);
}
}
Commands::Snapshot { dest } => { Commands::Snapshot { dest } => {
let _result = clawhdf5_agent::storage::snapshot_file(&cli.path, &dest)?; let _result = clawhdf5_agent::storage::snapshot_file(&path, &dest)?;
let j = serde_json::json!({ let j = serde_json::json!({
"status": "snapshot_created", "status": "snapshot_created",
"source": cli.path.display().to_string(), "source": path.display().to_string(),
"dest": dest.display().to_string(), "dest": dest.display().to_string(),
}); });
println!("{}", serde_json::to_string(&j)?); println!("{}", serde_json::to_string(&j)?);
+19 -61
View File
@@ -11,7 +11,7 @@ ClawhDF5 serves three audiences with different entry points:
| You Are | You Want | Start Here | | You Are | You Want | Start Here |
|---------|----------|------------| |---------|----------|------------|
| **AI agent developer** | Persistent memory for your agent | [Agent Memory (Rust)](#1-agent-memory-rust-library) | | **AI agent developer** | Persistent memory for your agent | [Agent Memory (Rust)](#1-agent-memory-rust-library) |
| **OpenClaw user** | Better memory for your OpenClaw agent | [OpenClaw Integration](#2-openclaw-integration) | | **OpenClaw user** | clawhdf5 is not an OpenClaw memory plugin | [Status](openclaw.md) |
| **Data scientist** | Read/write HDF5 files in Rust | [HDF5 File I/O](#3-hdf5-file-io) | | **Data scientist** | Read/write HDF5 files in Rust | [HDF5 File I/O](#3-hdf5-file-io) |
| **CLI user** | Inspect and manage agent memories | [CLI Tool](#4-cli-tool) | | **CLI user** | Inspect and manage agent memories | [CLI Tool](#4-cli-tool) |
| **Python user** | Use clawhdf5 from Python | [Python Bindings](#5-python-bindings) | | **Python user** | Use clawhdf5 from Python | [Python Bindings](#5-python-bindings) |
@@ -197,80 +197,38 @@ if let Some(alert) = detector.check_rate_anomaly() {
--- ---
## 2. OpenClaw Integration ## 2. Markdown Memory (and OpenClaw)
ClawhDF5 can serve as the memory backend for [OpenClaw](https://docs.openclaw.ai) agents, replacing the default Markdown + sqlite-vec approach. **clawhdf5 is not an OpenClaw memory backend.** Earlier versions of this guide
described one; it never worked — see [openclaw.md](openclaw.md) for what
happened and what a real plugin would need.
### How It Works What does exist is `ClawhdfBackend`, a library API that ingests Markdown files
by section and searches them with the full pipeline (hybrid retrieval,
``` re-ranking, confidence rejection):
OpenClaw Agent
│
├── memory_search("user preferences")
│ │
│ └── ClawhdfBackend
│ ├── Vector search (cosine)
│ ├── BM25 keyword search
│ ├── Reciprocal Rank Fusion
│ ├── Multi-factor re-ranking
│ └── Low-confidence rejection
│
└── agent_memory.h5 (single file, portable)
```
### Migration from Markdown
```rust ```rust
use clawhdf5_agent::openclaw::*; use clawhdf5_agent::openclaw::*;
use std::path::Path;
// Create a new HDF5 backend let mut backend = ClawhdfBackend::create(Path::new("memory.h5"), 384)?;
let mut backend = ClawhdfBackend::create("memory.h5", "my-agent", 384)?;
// Import your existing MEMORY.md // Each heading becomes a record, stored under "MEMORY.md::<heading>".
let md = std::fs::read_to_string("~/.openclaw/workspace/MEMORY.md")?; let md = std::fs::read_to_string("MEMORY.md")?;
let count = backend.ingest_markdown("MEMORY.md", &md)?; let count = backend.ingest_markdown("MEMORY.md", &md)?;
println!("Imported {} sections", count); println!("Imported {count} sections");
// Import daily logs
for entry in std::fs::read_dir("~/.openclaw/workspace/memory/")? {
let path = entry?.path();
if path.extension().map(|e| e == "md").unwrap_or(false) {
let content = std::fs::read_to_string(&path)?;
let name = path.file_name().unwrap().to_string_lossy();
backend.ingest_markdown(&name, &content)?;
}
}
// Search using the full pipeline
let results = backend.search("what are user preferences", &query_embedding, 5); let results = backend.search("what are user preferences", &query_embedding, 5);
for r in &results { for r in &results {
println!("[{:.3}] {} (from {})", r.score, r.text, r.path); println!("[{:.3}] {} (from {})", r.score, r.text, r.path);
} }
// Export back to Markdown (lossless roundtrip)
let exported = backend.export_markdown("MEMORY.md")?;
``` ```
### What You Get Over sqlite-vec Limits to know: sections ingested this way carry no embedding (search over them
is keyword-only unless you save records with vectors via `save_entry`);
| Feature | sqlite-vec | ClawhDF5 | ingesting the same file again adds the sections again rather than replacing
|---------|-----------|----------| them; and `export_markdown` rewrites every heading as `##`, so it is not a
| Vector search | ✅ | ✅ (8× faster at 100K) | lossless round trip.
| Keyword search | ❌ | ✅ BM25 |
| Hybrid fusion | ❌ | ✅ RRF |
| Re-ranking | ❌ | ✅ Multi-factor |
| Confidence rejection | ❌ | ✅ |
| Knowledge graph | ❌ | ✅ |
| Memory consolidation | ❌ | ✅ |
| Temporal queries | ❌ | ✅ (716ns) |
| Anomaly detection | ❌ | ✅ |
| Provenance tracking | ❌ | ✅ |
| Multi-modal | ❌ | ✅ |
| Single portable file | ❌ (SQLite + MD files) | ✅ |
### Future: Native OpenClaw Plugin
The Phase 2 roadmap includes a native OpenClaw plugin (`memory.backend = "clawhdf5"`) that transparently replaces sqlite-vec. Until then, the Rust library can be wrapped via NAPI or used from the CLI.
--- ---
@@ -547,7 +505,7 @@ let final_results = confidence::reject_low_confidence(
**Why not a vector database?** Pinecone, Qdrant, Weaviate — they're cloud services or heavy servers. Agent memory should be local, portable, and zero-dependency. An agent's memories should travel with it. **Why not a vector database?** Pinecone, Qdrant, Weaviate — they're cloud services or heavy servers. Agent memory should be local, portable, and zero-dependency. An agent's memories should travel with it.
**Why not Markdown?** OpenClaw uses Markdown today and it works for simple cases. But it doesn't scale: no vector search, no knowledge graph, no structured retrieval. ClawhDF5 can import/export Markdown while providing everything Markdown can't. **Why not Markdown?** Plain Markdown files work for simple cases. But it doesn't scale: no vector search, no knowledge graph, no structured retrieval. ClawhDF5 can import/export Markdown while providing everything Markdown can't.
**Why HDF5 specifically?** **Why HDF5 specifically?**
- Native N-dimensional array storage (perfect for embeddings) - Native N-dimensional array storage (perfect for embeddings)
+3 -30
View File
@@ -42,37 +42,10 @@ conversation → embedding → save to agent.h5
--- ---
## 2. OpenClaw Memory Upgrade ## 2. OpenClaw
**Scenario:** You run OpenClaw and the default Markdown + sqlite-vec memory works OK for simple recall but falls short on complex queries like "what did we decide about the deployment architecture last Tuesday?" or "who's responsible for the billing system?" Not supported: clawhdf5 is not an OpenClaw memory plugin, and the config this
section used to show was never valid. See [openclaw.md](openclaw.md).
**Problem:** Markdown files have no semantic structure. sqlite-vec does flat vector search — no keyword fusion, no re-ranking, no temporal reasoning, no knowledge graph.
**ClawhDF5 solution:**
```bash
# Migrate existing memories
clawhdf5 --path memory.h5 create --agent-id openclaw --dim 384
# Import your MEMORY.md and daily logs
# (programmatically via ClawhdfBackend::ingest_markdown)
```
Then in your OpenClaw config (future):
```json
{
"memory": {
"backend": "clawhdf5",
"path": "~/.openclaw/agents/main/memory.h5"
}
}
```
**What changes:**
- "What did we discuss last Tuesday?" → temporal index finds the session, returns memories from that time range
- "Who owns the billing system?" → knowledge graph traversal: billing_system → owned_by → Alice
- "Preferences about deployment" → hybrid search (vector + BM25) finds relevant memories even with different wording
- Bad search results get filtered out by confidence rejection instead of confusing the agent
--- ---
+23
View File
@@ -279,3 +279,26 @@ the same agent-store interop test.
**Fix:** an empty contiguous dataset gets the undefined address (all `0xff`), **Fix:** an empty contiguous dataset gets the undefined address (all `0xff`),
which is what libhdf5 itself writes. which is what libhdf5 itself writes.
## The Node.js package (`packages/clawhdf5-node`) does not work
**Status:** open (found 2026-09-25). Unpublished; not built or tested in CI.
The TypeScript wrapper over `crates/clawhdf5-napi` has never run successfully:
- napi-rs converts `#[napi(object)]` fields to camelCase, but the wrapper reads
snake_case (`r.line_range`, `s.total_records`, `s.working_count`, …), so
every stats and consolidation field comes back `undefined`
(`src/index.ts:76-120`).
- It loads `../clawhdf5.node`, but `napi build --platform` produces
`clawhdf5.<triple>.node`; `main` points at `index.js` while `tsc` writes to
`dist/`; `napi prepublish` expects per-platform packages that are not
defined.
- `save`/`saveBatch` exist in the napi layer but not in the wrapper, so a
TypeScript caller cannot store an embedding at all.
- The WAL for `agent.brain` is `agent.h5.wal` (the store uses
`with_extension("h5.wal")`), not `agent.brain.wal` as the old docs and the
test cleanup assume.
It was written for an OpenClaw integration that is not being pursued (see
`docs/openclaw.md`). Fix and add CI, or remove it, before anyone depends on it.
-213
View File
@@ -1,213 +0,0 @@
# Migration Guide: OpenClaw sqlite-vec → clawhdf5
This guide walks through migrating an OpenClaw agent from its default
sqlite-vec + Markdown file memory to the `clawhdf5` HDF5 backend.
---
## Why migrate?
| Feature | sqlite-vec + Markdown | clawhdf5 |
|---------|----------------------|----------|
| Storage format | SQLite WAL + flat .md files | Single HDF5 binary file |
| Vector search | sqlite-vec (SQLite extension) | Pure-Rust SIMD (clawhdf5-accel) |
| Full-text search | External (FTS5 or plain string match) | Built-in BM25 |
| Hybrid search | Manual combination | Automatic RRF blend |
| Memory tiers | Flat | Working → Episodic → Semantic |
| Hebbian decay | Not built-in | Automatic activation weighting |
| Portability | SQLite binary required | Zero native deps (all Rust) |
| Crash recovery | SQLite WAL | clawhdf5 WAL |
| Compaction | Manual | Auto-threshold + session-end |
| Embedding dim change | New DB required | New file required (same) |
---
## Step 1: Install `@redclaw/clawhdf5`
```bash
npm install @redclaw/clawhdf5
```
Or, if building from the monorepo source:
```bash
npm install -g @napi-rs/cli
cd packages/clawhdf5-node
npm install
npm run build
```
---
## Step 2: Update your OpenClaw config
Change `backend` from `"sqlite-vec"` (or `"markdown"`) to `"clawhdf5"`:
```json
{
"memory": {
"backend": "clawhdf5",
"clawhdf5": {
"path": "./agent.brain",
"embeddingDim": 768
}
}
}
```
See [openclaw-config.md](openclaw-config.md) for the full schema.
---
## Step 3: Run the one-time migration
clawhdf5 ships a migration helper that reads your existing Markdown memory
files and ingests them via `ingestMarkdown()`.
### Automated migration script
```typescript
import { ClawhdfMemory } from '@redclaw/clawhdf5';
import { readFileSync, readdirSync, statSync } from 'fs';
import { join, relative } from 'path';
async function migrate(
memoryDir: string,
brainPath: string,
embeddingDim: number = 768,
): Promise<void> {
const mem = ClawhdfMemory.create(brainPath, embeddingDim);
// Walk all .md files under memoryDir
function walk(dir: string): string[] {
return readdirSync(dir).flatMap((entry) => {
const full = join(dir, entry);
return statSync(full).isDirectory() ? walk(full) : [full];
});
}
const files = walk(memoryDir).filter((f) => f.endsWith('.md'));
let totalSections = 0;
for (const file of files) {
const content = readFileSync(file, 'utf8');
const relPath = relative(process.cwd(), file);
const count = mem.ingestMarkdown(relPath, content);
console.log(` ${relPath}: ${count} sections`);
totalSections += count;
}
// Force WAL merge after bulk import
mem.flushWal();
console.log(`\nMigration complete: ${files.length} files, ${totalSections} sections`);
const s = mem.stats();
console.log(` Total records: ${s.totalRecords}`);
console.log(` File size: ${(s.fileSizeBytes / 1024).toFixed(1)} KB`);
}
// Usage
migrate('./memory', './agent.brain', 768).catch(console.error);
```
### What the migration does
1. Walks all `.md` files under your memory directory.
2. Parses each file into sections using the same `MarkdownParser` used by
OpenClaw (splits on ATX headings `#`, `##`, `###`, …).
3. Stores each section as a separate record in the HDF5 file with the file
path as the `source_channel` (e.g. `memory/user.md::Goals`).
4. Flushes the WAL to merge everything into the `.brain` file.
After migration, **the original `.md` files are not modified or deleted**.
You can keep them as a backup or remove them once you have verified the
migrated data.
---
## Step 4: Verify
```typescript
import { ClawhdfMemory } from '@redclaw/clawhdf5';
const mem = ClawhdfMemory.open('./agent.brain');
const s = mem.stats();
console.log('Records after migration:', s.totalRecords);
// Spot-check: retrieve a known path
const userMd = mem.get('memory/MEMORY.md');
console.log(userMd?.slice(0, 200));
// Round-trip a file back to Markdown
const exported = mem.exportMarkdown('memory/MEMORY.md');
console.log(exported.slice(0, 500));
```
---
## Step 5: Update agent code
If your agent code reads memory files directly from disk, update it to use
the clawhdf5 API instead:
**Before (sqlite-vec + file reads):**
```typescript
const content = readFileSync('memory/user.md', 'utf8');
const sections = parseMarkdown(content);
const results = await vectorSearch(query, sections, k);
```
**After (clawhdf5):**
```typescript
import { ClawhdfMemory } from '@redclaw/clawhdf5';
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
const embedding = await embed(query); // your embedding function
const results = mem.search(query, new Float32Array(embedding), k);
```
---
## Step 6: Session lifecycle hooks
Add compaction at session end for best long-term memory health:
```typescript
// At the start of your agent process
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
// ... agent runs ...
// At the end of each session
mem.tickSession(); // decay activation weights
const stats = mem.runConsolidation(Date.now() / 1000); // promote memories
console.log('[memory] consolidation:', stats);
```
---
## Rollback
If you need to roll back to sqlite-vec:
1. Change `memory.backend` back to `"sqlite-vec"` in your config.
2. The original `.md` files are unchanged (if you kept them).
3. Delete `agent.brain` (and `agent.brain.wal` if present).
---
## Troubleshooting
### `Error: no records found for path: memory/user.md`
The path passed to `get()` or `exportMarkdown()` must exactly match the
relative path used during `ingestMarkdown()`. Check for leading `./`
differences.
### Memory is empty after reopening
Make sure `flushWal()` was called after bulk writes. Without it, entries
remain in the WAL and may be lost if the process exits abnormally.
### Embedding dimension mismatch
The `embeddingDim` passed to `create()` cannot be changed after the file is
created. If you switch embedding models, create a new `.brain` file and
re-run the migration script.
-144
View File
@@ -1,144 +0,0 @@
# OpenClaw × clawhdf5 Configuration Reference
This document describes the full configuration schema for integrating
`clawhdf5` as the memory backend in an OpenClaw agent gateway.
---
## Minimal example
```json
{
"memory": {
"backend": "clawhdf5",
"clawhdf5": {
"path": "./agent.brain",
"embeddingDim": 768
}
}
}
```
---
## Full schema
```json
{
"memory": {
"backend": "clawhdf5",
"clawhdf5": {
"path": "./agent.brain",
"embeddingDim": 768,
"walEnabled": true,
"walMaxEntries": 500,
"consolidation": {
"workingCapacity": 100,
"episodicCapacity": 10000,
"episodicHalfLifeDays": 7,
"semanticHalfLifeDays": 30,
"promotionThreshold": 0.6,
"semanticAccessThreshold": 10
},
"compaction": {
"autoCompactThreshold": 0.3,
"tickOnSessionEnd": true,
"consolidateOnCompaction": true
}
}
}
}
```
---
## Field reference
### Top level
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `memory.backend` | `string` | `"clawhdf5"` | Must be `"clawhdf5"` to activate this backend |
### `clawhdf5`
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `path` | `string` | `"./agent.brain"` | Filesystem path for the `.brain` (HDF5) file. Relative to the OpenClaw working directory. |
| `embeddingDim` | `number` | `768` | Dimension of the embedding vectors. Must match the embedder model. Common values: `384` (MiniLM), `768` (nomic-embed-text, BGE-base), `1536` (OpenAI text-embedding-3-small). |
| `walEnabled` | `boolean` | `true` | Enable the Write-Ahead Log for crash recovery. Disable only on read-only stores or when crash safety is not required. |
| `walMaxEntries` | `number` | `500` | Number of WAL entries to accumulate before an automatic merge to the .h5 file. Lower values = more frequent flushes (safer, slightly slower). |
### `clawhdf5.consolidation`
Controls the hippocampal three-tier memory engine (Working → Episodic →
Semantic).
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `workingCapacity` | `number` | `100` | Maximum records in the Working tier before lowest-decay entries are evicted. |
| `episodicCapacity` | `number` | `10000` | Maximum records in the Episodic tier. |
| `episodicHalfLifeDays` | `number` | `7` | Half-life (in days) for exponential decay of Episodic records. Records not accessed within roughly one half-life drop in importance. |
| `semanticHalfLifeDays` | `number` | `30` | Half-life for Semantic records. Longer than Episodic — semantic knowledge decays slowly. |
| `promotionThreshold` | `number` | `0.6` | Importance score (0–1) above which a Working record is promoted to the Episodic tier. Higher = more selective. |
| `semanticAccessThreshold` | `number` | `10` | Minimum access count for an Episodic record to be promoted to Semantic. |
### `clawhdf5.compaction`
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `autoCompactThreshold` | `number` | `0.3` | Fraction of tombstoned records (0–1) that triggers automatic compaction. `0.3` = compact when 30% of records are deleted. Set to `0` to disable auto-compact. |
| `tickOnSessionEnd` | `boolean` | `true` | Run `tickSession()` (Hebbian decay) automatically when the agent session closes. |
| `consolidateOnCompaction` | `boolean` | `true` | Run the hippocampal consolidation engine after each compaction cycle. |
---
## Embedder compatibility
The `embeddingDim` must remain constant for the lifetime of a `.brain` file.
Mixing embedding models in the same file is not supported.
| Embedder | `embeddingDim` |
|----------|---------------|
| `all-MiniLM-L6-v2` | `384` |
| `nomic-embed-text` | `768` |
| `BGE-base-en-v1.5` | `768` |
| `OpenAI text-embedding-3-small` | `1536` |
| `OpenAI text-embedding-3-large` | `3072` |
---
## OpenClaw integration code
```typescript
import { ClawhdfMemory } from '@redclaw/clawhdf5';
// Load config from your OpenClaw config file
const cfg = loadConfig(); // your config loading logic
const mem = ClawhdfMemory.openOrCreate(
cfg.memory.clawhdf5.path,
cfg.memory.clawhdf5.embeddingDim ?? 768,
);
// On session end
if (cfg.memory.clawhdf5.compaction?.tickOnSessionEnd) {
mem.tickSession();
}
if (cfg.memory.clawhdf5.compaction?.consolidateOnCompaction) {
const stats = mem.runConsolidation(Date.now() / 1000);
console.log('[clawhdf5] consolidation:', stats);
}
```
---
## Environment variables
The following environment variables override config file values when set:
| Variable | Overrides |
|----------|-----------|
| `CLAWHDF5_PATH` | `clawhdf5.path` |
| `CLAWHDF5_EMBEDDING_DIM` | `clawhdf5.embeddingDim` |
| `CLAWHDF5_WAL_ENABLED` | `clawhdf5.walEnabled` (`"true"` / `"false"`) |
-337
View File
@@ -1,337 +0,0 @@
# OpenClaw × clawhdf5 Integration
clawhdf5 provides a drop-in HDF5-backed memory backend for the
[OpenClaw](https://github.com/redclawsystems/openclaw) agent gateway.
This document covers architecture, the full Node.js API reference, and code
examples for common operations.
---
## Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ OpenClaw (Node.js/TypeScript) │
│ │
│ ┌─────────────────┐ ┌──────────────────────────────┐ │
│ │ Agent runtime │───▶│ @redclaw/clawhdf5 (Node.js) │ │
│ └─────────────────┘ │ TypeScript wrapper │ │
│ └──────────────┬─────────────┘ │
│ │ napi-rs FFI │
└────────────────────────────────────────┼────────────────────┘
│
┌────────────────────────────────────────▼────────────────────┐
│ clawhdf5-napi (Rust, cdylib) │
│ │
│ ClawhdfMemory ──▶ ClawhdfBackend ──▶ HDF5Memory │
│ MemoryBackend ├─ MemoryCache │
│ trait impl ├─ WalFile │
│ ├─ SessionCache │
│ └─ KnowledgeCache │
│ │
│ ConsolidationEngine (hippocampal tiers) │
│ Working (100) ──▶ Episodic (10k) ──▶ Semantic (∞) │
└─────────────────────────────────────────────────────────────┘
│
┌────────────────▼────────────┐
│ agent.brain (HDF5 file) │
│ agent.brain.wal (WAL log) │
└─────────────────────────────┘
```
### Key design decisions
- **Single file**: everything lives in one `.brain` HDF5 file (+ WAL sidecar).
- **In-memory cache**: the full embedding matrix and chunk list are loaded into RAM for fast search.
- **Hybrid search**: vector similarity (70%) and BM25 full-text (30%) are blended with Reciprocal Rank Fusion (RRF), then re-ranked by Hebbian activation weight and temporal recency.
- **Hippocampal tiers**: records are classified as Working, Episodic, or Semantic based on importance and access frequency. Tier promotion and eviction happen during `runConsolidation()`.
- **WAL**: writes are journaled before hitting the .h5 file. On crash, the WAL is replayed at next open.
---
## Installation
```bash
npm install @redclaw/clawhdf5
```
See [packages/clawhdf5-node/README.md](../packages/clawhdf5-node/README.md)
for build-from-source instructions.
---
## Node.js API reference
### `ClawhdfMemory` (class)
All instance methods are synchronous. The native Rust code is single-threaded
on the Node.js side; do **not** share a `ClawhdfMemory` instance across Worker
threads without external locking.
---
#### Static factory methods
##### `ClawhdfMemory.create(path: string, embeddingDim: number): ClawhdfMemory`
Create a new `.brain` file. Throws if the file already exists.
```typescript
const mem = ClawhdfMemory.create('./agent.brain', 768);
```
##### `ClawhdfMemory.open(path: string): ClawhdfMemory`
Open an existing file. Replays the WAL automatically.
```typescript
const mem = ClawhdfMemory.open('./agent.brain');
```
##### `ClawhdfMemory.openOrCreate(path: string, embeddingDim: number): ClawhdfMemory`
**Recommended entry point.** Opens if the file exists, otherwise creates it.
```typescript
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
```
---
#### `search(queryText, queryEmbedding, k): MemorySearchResult[]`
Hybrid BM25 + vector search.
```typescript
const embedding = new Float32Array(await embed(query));
const results = mem.search(query, embedding, 10);
for (const r of results) {
console.log(r.score.toFixed(3), r.path, r.text.slice(0, 80));
}
```
Pass an empty `Float32Array` to use BM25 only (no vector similarity).
**Parameters:**
- `queryText: string` — used for BM25 term matching
- `queryEmbedding: Float32Array` — dense vector of length `embeddingDim`
- `k: number` — maximum results to return
**Returns:** `MemorySearchResult[]`
---
#### `get(path, fromLine?, numLines?): string | null`
Retrieve stored content by path.
```typescript
const md = mem.get('memory/user.md'); // all content
const lines = mem.get('memory/user.md', 5, 10); // lines 5–14
const section = mem.get('memory/user.md::Goals'); // specific section
```
Section sub-paths use the `::heading` suffix produced by `ingestMarkdown`.
---
#### `write(path, content): void`
Store raw content at `path`.
```typescript
mem.write('memory/session.md', '# Session\n\nWorking on task X.');
```
---
#### `ingestMarkdown(path, content): number`
Parse `content` as Markdown, split on ATX headings, and store each section
separately. Returns the number of sections ingested.
```typescript
import { readFileSync } from 'fs';
const md = readFileSync('./memory/MEMORY.md', 'utf8');
const count = mem.ingestMarkdown('memory/MEMORY.md', md);
console.log(`Ingested ${count} sections`);
```
Sections are addressable as `memory/MEMORY.md::HeadingName`.
---
#### `exportMarkdown(path): string`
Reconstruct stored sections for `path` back into a Markdown string.
```typescript
const md = mem.exportMarkdown('memory/MEMORY.md');
writeFileSync('./memory/MEMORY.md', md);
```
---
#### `stats(): BackendStats`
Return aggregate statistics.
```typescript
const s = mem.stats();
console.log(`Records: ${s.totalRecords}, Size: ${s.fileSizeBytes} bytes`);
```
---
#### `compact(): number`
Remove tombstoned records from the store. Returns count removed.
---
#### `tickSession(): void`
Apply Hebbian decay to all activation weights. Call at session end.
---
#### `flushWal(): void`
Force a WAL merge: flush `.h5` and truncate the WAL log.
---
#### `runConsolidation(nowSecs: number): ConsolidationStats`
Run one full hippocampal consolidation cycle.
```typescript
const stats = mem.runConsolidation(Date.now() / 1000);
console.log(stats);
// { workingCount: 42, episodicCount: 310, semanticCount: 5,
// totalEvictions: 0, totalPromotions: 7 }
```
---
#### `walPendingCount(): number`
Number of pending WAL entries (0 if WAL is disabled).
---
### Type reference
```typescript
interface MemorySearchResult {
text: string;
score: number; // 0–1, higher = more relevant
path: string; // source file path
lineRange?: [number, number];
timestamp?: number; // Unix epoch seconds
source: string;
}
interface BackendStats {
totalRecords: number;
totalEmbeddings: number;
fileSizeBytes: number;
modalities: string[]; // e.g. ["text"]
lastUpdated?: number; // Unix epoch seconds
}
interface ConsolidationStats {
workingCount: number;
episodicCount: number;
semanticCount: number;
totalEvictions: number;
totalPromotions: number;
}
```
---
## Common patterns
### Session lifecycle
```typescript
import { ClawhdfMemory } from '@redclaw/clawhdf5';
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
// --- agent session runs ---
// On session end: decay + consolidate
mem.tickSession();
const consolidationStats = mem.runConsolidation(Date.now() / 1000);
console.log('[memory] consolidation:', consolidationStats);
```
### Ingest all memory files at startup
```typescript
import { readdirSync, readFileSync, statSync } from 'fs';
import { join, relative } from 'path';
function ingestDirectory(mem: ClawhdfMemory, dir: string): void {
for (const entry of readdirSync(dir)) {
const full = join(dir, entry);
if (statSync(full).isDirectory()) {
ingestDirectory(mem, full);
} else if (entry.endsWith('.md')) {
const content = readFileSync(full, 'utf8');
const path = relative(process.cwd(), full);
mem.ingestMarkdown(path, content);
}
}
mem.flushWal();
}
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
ingestDirectory(mem, './memory');
```
### Search with real embeddings
```typescript
import OpenAI from 'openai';
import { ClawhdfMemory } from '@redclaw/clawhdf5';
const ai = new OpenAI();
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 1536);
async function searchMemory(query: string, k = 5) {
const resp = await ai.embeddings.create({
model: 'text-embedding-3-small',
input: query,
});
const embedding = new Float32Array(resp.data[0].embedding);
return mem.search(query, embedding, k);
}
```
---
## Error handling
All methods that can fail throw a `NapiError` (a standard JS `Error` subclass)
with the Rust error message as `message`.
```typescript
try {
const md = mem.exportMarkdown('nonexistent.md');
} catch (e) {
console.error('Export failed:', (e as Error).message);
// "no records found for path: nonexistent.md"
}
```
---
## See also
- [openclaw-config.md](openclaw-config.md) — Full configuration schema
- [migration-guide.md](migration-guide.md) — Migrating from sqlite-vec
- [packages/clawhdf5-node/README.md](../packages/clawhdf5-node/README.md) — Build instructions
- [BENCHMARKS.md](../BENCHMARKS.md) — Performance results
+73
View File
@@ -0,0 +1,73 @@
# OpenClaw: not supported
**clawhdf5 does not currently work as an [OpenClaw](https://docs.openclaw.ai)
memory backend, and never has.** Earlier versions of these docs described a
"drop-in" backend enabled with `memory.backend = "clawhdf5"`. That
configuration was never valid: from v2026.2 through v2026.7 OpenClaw's
`memory.backend` accepted only `"builtin"` or `"qmd"` and rejected unknown
keys, and since v2026.8.1 ("OpenClaw 2.0") the key no longer exists. A Gateway
given that config refuses to start. No plugin was ever built or tested against
OpenClaw, and the `@redclaw/clawhdf5` npm package was never published.
As of 2026-09-25 we are not pursuing an OpenClaw plugin; the maintained
integration target is ZeroClaw. This page records what a plugin would need,
for when that changes.
## What OpenClaw expects today (v2026.9.6)
Checked against the OpenClaw source at tag `v2026.9.6` and its docs on
2026-09-25. OpenClaw marks every plugin API as experimental, so re-check before
building anything.
- **Memory lives in Markdown files**, which are the source of truth: `MEMORY.md`,
`USER.md`, daily notes in `memory/YYYY-MM-DD.md` in the agent workspace. The
memory engine is an index over them
([concepts/memory](https://docs.openclaw.ai/concepts/memory)).
- **A memory plugin is selected with `plugins.slots.memory: "<plugin-id>"`**
(default `memory-core`), its settings under
`plugins.entries.<plugin-id>.config`, validated against the plugin's own
schema ([gateway/config-extensions](https://docs.openclaw.ai/gateway/config-extensions)).
Memory search settings are under `memory.search`
([reference/memory-config](https://docs.openclaw.ai/reference/memory-config)).
- **A plugin needs** an `openclaw.plugin.json` manifest with `id`,
`configSchema`, `"kind": "memory"` and every tool listed in `contracts.tools`
([plugins/manifest](https://docs.openclaw.ai/plugins/manifest)); a
`package.json` with `openclaw.extensions`, `openclaw.compat.pluginApi` and an
`openclaw` peer dependency; and an entry built with `definePluginEntry`.
- **Two ways to integrate** (both exist upstream): tools only, as
`memory-lancedb` does (`api.registerTool`), or a full memory engine, as
`memory-core` does, through `api.registerMemoryCapability({ runtime, ... })`,
whose runtime returns a `MemorySearchManager` implementing `search`,
`readFile` (returning `status: "ok" | "not_found"`), `status`,
`probeEmbeddingAvailability` and `probeVectorAvailability`. Active Memory
expects `memory_search` and `memory_get` tools
([plugins/sdk-overview/memory-and-context](https://docs.openclaw.ai/plugins/sdk-overview/memory-and-context)).
- **Embeddings come from OpenClaw's providers** (`memory.search.provider`), or a
plugin registers one with `api.registerEmbeddingProvider`.
- **Native code**: plugin installs run with `--ignore-scripts`, so a napi addon
has to ship as prebuilt per-platform packages (the pattern `memory-lancedb`
uses for LanceDB), loaded lazily
([plugins/dependency-resolution](https://docs.openclaw.ai/plugins/dependency-resolution)).
- **Distribution**: `openclaw plugins install` from npm or ClawHub; a first
install from an arbitrary source needs explicit review, and community ClawHub
packages go through a security audit.
- **Churn to plan for**: the memory SDK was reshaped in 2026 (separate
registration functions merged into `registerMemoryCapability`;
`registerMemoryEmbeddingProvider` removed on 2026-08-21), and further SDK
surfaces become eligible for removal on 2026-10-01
([plugins/sdk-migration/removal-timeline](https://docs.openclaw.ai/plugins/sdk-migration/removal-timeline)).
## What this repository has
Building blocks, usable as a library today, but not an OpenClaw plugin:
- `clawhdf5_agent::openclaw::ClawhdfBackend` — a Markdown-oriented backend over
`HDF5Memory`: ingest Markdown by section, hybrid search with re-ranking and
confidence rejection, read back by path, export. Gaps a plugin would have to
close: `write`/`ingest_markdown` store no embeddings (search is keyword-only
for that content unless records are saved with `save_entry`), re-ingesting
appends rather than replaces, there is no delete, `line_range` is never set,
and export rewrites every heading as `##`.
- `crates/clawhdf5-napi` and `packages/clawhdf5-node` — Node bindings and a
TypeScript wrapper. **Not published, not built or tested in CI, and known to
be broken**; see `docs/known-issues.md`.
+9 -9
View File
@@ -5,20 +5,20 @@ HDF5-backed agent memory system with hippocampal consolidation.
Built with [napi-rs](https://napi.rs). Built with [napi-rs](https://napi.rs).
> **Status: unpublished and known to be broken.** This package is not on npm,
> no binaries are built, nothing in CI builds or tests it, and the wrapper
> reads field names the native layer does not produce. It is not an OpenClaw
> plugin. See [known issues](../../docs/known-issues.md) and
> [docs/openclaw.md](../../docs/openclaw.md) before using it.
## Installation ## Installation
Not published. Building from source needs `@napi-rs/cli`:
```bash ```bash
npm install @redclaw/clawhdf5 npm install && npm run build
``` ```
Pre-built binaries are published for:
| Platform | Architecture |
|----------|-------------|
| Linux (glibc) | x64, aarch64 |
| macOS | x64, aarch64 (Apple Silicon) |
| Windows | x64 |
## Quick start ## Quick start
```ts ```ts
+4 -4
View File
@@ -1,7 +1,7 @@
{ {
"name": "@redclaw/clawhdf5", "name": "@redclaw/clawhdf5",
"version": "2.7.0", "version": "2.7.0",
"description": "Node.js bindings for clawhdf5 — HDF5-backed agent memory with hippocampal consolidation", "description": "Node.js bindings for clawhdf5 \u2014 HDF5-backed agent memory with hippocampal consolidation",
"main": "index.js", "main": "index.js",
"types": "index.d.ts", "types": "index.d.ts",
"license": "MIT", "license": "MIT",
@@ -14,8 +14,7 @@
"memory", "memory",
"hdf5", "hdf5",
"vector-search", "vector-search",
"embedding", "embedding"
"openclaw"
], ],
"napi": { "napi": {
"name": "clawhdf5", "name": "clawhdf5",
@@ -51,5 +50,6 @@
}, },
"engines": { "engines": {
"node": ">= 16" "node": ">= 16"
} },
"private": true
} }