Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
79dfa78e8f | ||
|
|
bdadf3447c | ||
|
|
0c65a27b00 | ||
|
|
db9af7972c | ||
|
|
7706697feb | ||
|
|
c0f704c381 | ||
|
|
a7920bd4b3 | ||
|
|
7e43b5366c | ||
|
|
73bb068264 |
@@ -243,6 +243,30 @@ index asked for ~16 000 candidates, where scanning the few hundred or thousand
|
||||
allowed records is exact and cheap. Re-ranking a 3k candidate pool and
|
||||
confidence rejection add about 3%.
|
||||
|
||||
### Signed checkpoints
|
||||
|
||||
Measured 2026-09-25 on tank (AMD Ryzen 7 7800X3D). A default store (float16,
|
||||
int8 index), 384-dim; each checkpoint rewrites the whole file, as every
|
||||
checkpoint does. Medians of five checkpoints and three verifies; three runs
|
||||
agreed to within the ranges shown.
|
||||
|
||||
```bash
|
||||
cargo run --release -p clawhdf5-bench --bin search_harness -- --signing-study --full
|
||||
```
|
||||
|
||||
| N | checkpoint, unsigned | checkpoint, signed | signing adds | `verify` | file size added |
|
||||
|---:|---:|---:|---:|---:|---:|
|
||||
| 1 000 | 5.4 ms | 6.4 ms | 0.7–1.0 ms | 2.1 ms | 0.03 MiB |
|
||||
| 10 000 | 46 ms | 55 ms | 8.1–9.4 ms | 18.6 ms | 0.31 MiB |
|
||||
| 100 000 | 495 ms | 598 ms | 89–112 ms | 247 ms | 3.05 MiB |
|
||||
|
||||
Signing costs about 20% of a checkpoint: every record is rehashed (SHA-256)
|
||||
and the Merkle root recomputed each time; the Ed25519 signature itself is
|
||||
microseconds. Caching per-record hashes between checkpoints would cut this to
|
||||
the records that changed. The per-record hashes stored for locating edits are
|
||||
32 bytes each (4% of a 100K float16 store). `verify` reads and rehashes the
|
||||
whole checkpoint.
|
||||
|
||||
### float16 embedding storage (`MemoryConfig::float16`)
|
||||
|
||||
Measured 2026-09-23 on tank (AMD Ryzen 7 7800X3D). The same clustered
|
||||
|
||||
@@ -3,6 +3,18 @@
|
||||
## Unreleased
|
||||
|
||||
### Upgrade Notes
|
||||
- **OpenClaw is not supported, and never was.** The docs described a
|
||||
"drop-in" OpenClaw memory backend enabled with `memory.backend = "clawhdf5"`.
|
||||
That config was never valid in any OpenClaw release (v2026.2–v2026.7
|
||||
accepted only `builtin`/`qmd` and rejected unknown keys, so a Gateway given
|
||||
it refuses to start; OpenClaw 2.0 removed the key), no plugin was ever built,
|
||||
and `@redclaw/clawhdf5` was never published. The integration docs
|
||||
(`openclaw-integration.md`, `openclaw-config.md`, `migration-guide.md`) are
|
||||
removed; `docs/openclaw.md` explains the status and what a real plugin would
|
||||
need against OpenClaw v2026.9.6. `ClawhdfBackend` stays as a library API.
|
||||
- **Breaking:** `MemoryError` is now `#[non_exhaustive]` and gained
|
||||
`SigningKeyRequired`; a `match` on it needs a wildcard arm. Future variants
|
||||
will no longer be breaking.
|
||||
- **Breaking:** `clawhdf5-agent`'s `agent` feature is removed. It enabled
|
||||
nothing — the agent layer is always built — but the README and guides told
|
||||
people to pass it; drop `agent` from `features = [...]`.
|
||||
@@ -60,6 +72,29 @@
|
||||
`quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
|
||||
out. The CLI's `--quantized-index` is still accepted but is now a no-op.
|
||||
|
||||
### Signing
|
||||
- `clawhdf5-agent`: **Ed25519-signed checkpoints** — the README's
|
||||
"cryptographically verifiable memory", now true. With
|
||||
`HDF5Memory::set_signing_key(key)`, every checkpoint stores a signed
|
||||
manifest: a SHA-256 per record (text, embedding as stored, channel,
|
||||
timestamp, session, tags, deleted flag, activation) in a Merkle tree, plus
|
||||
hashes of the settings (and WAL mark), sessions and knowledge graph, with
|
||||
the per-record hashes in `/integrity/record_hashes`.
|
||||
`HDF5Memory::verify(path, &public_key)` recomputes everything from the file
|
||||
and reports which part changed and which records (`changed_records`); a
|
||||
forged manifest fails the signature. The key is never persisted; a signed
|
||||
store refuses to checkpoint without it (`MemoryError::SigningKeyRequired`),
|
||||
and `remove_signature()` is the deliberate way back to unsigned. Saves still
|
||||
in the WAL are not covered (`wal_entries_unsigned`). Tests include every
|
||||
kind of edit, and an edit made with h5py in place, which verify pinpoints.
|
||||
Cost: ~20% of a checkpoint, 32 bytes per record (`BENCHMARKS.md`, "Signed
|
||||
checkpoints"). New dependencies `ed25519-dalek`, `sha2`, `rand_core` — pure
|
||||
Rust; the no-C check still passes.
|
||||
- `clawhdf5-cli`: `keygen --out <file>` (owner-only key file),
|
||||
`--signing-key <file>` / `CLAWHDF5_SIGNING_KEY` on writing commands
|
||||
(`create` signs immediately), `verify --public-key <hex|file>` (JSON report;
|
||||
exit status 2 if not valid), and `signed` in `create`/`stats` output.
|
||||
|
||||
### Migration
|
||||
- `clawhdf5-migrate`: writes through the agent's own API (`HDF5Memory::create`
|
||||
/ `open`, `save_batch`, the session cache and knowledge graph), so there is
|
||||
@@ -100,6 +135,13 @@
|
||||
activation of the `k` results it returns, not of the whole `3k` candidate
|
||||
pool it re-ranks.
|
||||
|
||||
### Documentation
|
||||
- OpenClaw claims withdrawn across the README, QUICKSTART, USE_CASES, ROADMAP
|
||||
(Track 7 marked withdrawn) and the `openclaw` module docs; the dead
|
||||
`github.com/redclawsystems/openclaw` link is gone. The Node package is
|
||||
marked unpublished and broken (now `"private": true` so it cannot be
|
||||
published by accident), with its bugs recorded in `docs/known-issues.md`.
|
||||
|
||||
### Benchmarks
|
||||
- Every undated or pre-September section of `BENCHMARKS.md` re-run on one
|
||||
machine on one day (tank, 2026-09-24, commit 5c8323c), with the command for
|
||||
|
||||
@@ -104,11 +104,26 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
|
||||
the allowed records whenever cheaper than `pool × M` index distance
|
||||
evaluations, and as the fallback when the pool comes back short), fusion,
|
||||
activation scaling, optional re-ranking and confidence rejection.
|
||||
`hybrid_search`/`hybrid_search_with` are thin wrappers; the OpenClaw
|
||||
backend is `search` with re-rank + confidence on. Measure changes with
|
||||
`hybrid_search`/`hybrid_search_with` are thin wrappers; `ClawhdfBackend`
|
||||
(the `openclaw` module) is `search` with re-rank + confidence on.
|
||||
- **OpenClaw is not supported** (decided 2026-09-25): clawhdf5 is not an
|
||||
OpenClaw memory plugin and never was — the old `memory.backend = "clawhdf5"`
|
||||
config was never valid. Don't reintroduce OpenClaw claims; `docs/openclaw.md`
|
||||
records what a real plugin would need. ZeroClaw is the integration target. Measure changes with
|
||||
`search_harness --options-study`.
|
||||
- `MemoryConfig::compression` is off by default; when on, embeddings are
|
||||
deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd).
|
||||
- Signed checkpoints (`clawhdf5-agent` `signing` module): with
|
||||
`HDF5Memory::set_signing_key` every checkpoint stores an Ed25519-signed
|
||||
manifest (SHA-256 per record in a Merkle tree + settings/sessions/graph
|
||||
hashes; per-record hashes in `/integrity/record_hashes`);
|
||||
`HDF5Memory::verify(path, &pk)` locates edits. The hashes must cover exactly
|
||||
what the file persists in the form the loader returns it (strings lose
|
||||
trailing NULs; an empty WAL mark is not written) or untouched stores stop
|
||||
verifying — `tests/signed_store.rs` round-trips awkward strings. The key is
|
||||
never persisted; a signed store refuses to checkpoint without it
|
||||
(`MemoryError::SigningKeyRequired`, and `MemoryError` is `#[non_exhaustive]`).
|
||||
WAL entries after the checkpoint are not covered.
|
||||
- `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by
|
||||
default) recomputes a dataset's SHA-256 and compares it against the
|
||||
`_provenance_sha256` attribute written automatically on save when
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
[](BENCHMARKS.md#longmemeval-results)
|
||||
[](BENCHMARKS.md#memory-footprint-1)
|
||||
|
||||
ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file.
|
||||
ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, cryptographically verifiable memory (Ed25519-signed checkpoints) — all stored in a single portable file.
|
||||
|
||||
> **Two things live here:**
|
||||
> - **A general-purpose, pure-Rust HDF5 library** — zero C dependencies, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers.
|
||||
@@ -71,7 +71,7 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md).
|
||||
100K). It no longer rebuilds BM25 or rewrites the store per query, and the
|
||||
HNSW graph is persisted (v2.4.0).
|
||||
- Default fusion weights are now the measured 0.4 / 0.6 (v2.5.0). Re-ranking had
|
||||
been discarding the retrieval score, costing the OpenClaw backend 40.6pp of
|
||||
been discarding the retrieval score, costing the Markdown backend 40.6pp of
|
||||
Hit@1; fixed in v2.6.0.
|
||||
- Selection reads decode only the chunks they touch (a 64×64 window: 105 ms to
|
||||
0.39 ms), and full reads are 1.2–1.9× faster (v2.5.0).
|
||||
@@ -93,7 +93,7 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md).
|
||||
identical LongMemEval retrieval on real embeddings.
|
||||
- `HDF5Memory::search` with `SearchOptions`: filter by source channel (exact
|
||||
filtered top-k, never slower than unfiltered), and opt-in re-ranking and
|
||||
confidence rejection, which used to be OpenClaw-only.
|
||||
confidence rejection, which used to be reachable only through `ClawhdfBackend`.
|
||||
|
||||
**Tooling**
|
||||
- CI now runs the h5py/netCDF4 interop suites for real (they had been skipping
|
||||
@@ -113,7 +113,7 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d
|
||||
| Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers |
|
||||
| Temporal queries | Custom code | Native temporal index (622 ns range query over 10K) |
|
||||
| Multi-modal | Multiple stores | Unified cross-modal search (exact scan: 842 µs over 1K records) |
|
||||
| Integrity | Hope for the best | Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance |
|
||||
| Integrity | Hope for the best | Ed25519-signed checkpoints that pinpoint any edited record, chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts |
|
||||
| Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** |
|
||||
|
||||
---
|
||||
@@ -228,7 +228,7 @@ is for. The weights matter more than the stages: a sweep of `vector_weight` from
|
||||
0.0 to 1.0 found the old `0.7/0.3` default is **strictly dominated** by
|
||||
`0.4/0.6` — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Since
|
||||
v2.5.0 `0.4/0.6` is the default (`hybrid::DEFAULT_FUSION`, used by
|
||||
`unified_search`, `hybrid_search_with` and the OpenClaw backend); callers that
|
||||
`unified_search`, `hybrid_search_with` and `ClawhdfBackend`); callers that
|
||||
pass weights to `hybrid_search` explicitly choose their own. Use `0.3/0.7` if
|
||||
rank-1 precision matters most. Reciprocal rank fusion is selectable
|
||||
(`hybrid::Fusion::Rrf`) but measured worse than the weighted sum. See
|
||||
@@ -329,7 +329,7 @@ ClawhDF5's agent memory engine draws on 15+ recent papers on agentic memory syst
|
||||
│ × √(Hebbian activation) │
|
||||
└─────────────────┬──────────────────┘
|
||||
│ opt-in (SearchOptions);
|
||||
│ the OpenClaw backend turns both on
|
||||
│ ClawhdfBackend turns both on
|
||||
┌─────────────────▼──────────────────┐
|
||||
│ Multi-factor re-ranking │
|
||||
│ relevance · recency · authority · │
|
||||
@@ -365,13 +365,14 @@ directly; the store persists the records, sessions and graph they work over.
|
||||
| **`knowledge`** | Entity/relation graph with BFS traversal, spreading activation, fuzzy (Levenshtein) entity resolution |
|
||||
| **`consolidation`** | Three-tier memory (Working → Episodic → Semantic) with importance scoring, novelty, and time-decay |
|
||||
| **`hybrid`** | Vector + BM25 fusion. Default is a min-max-normalised weighted sum, vector 0.4 / keyword 0.6 (`hybrid::DEFAULT_FUSION`, tuned on LongMemEval); RRF is available via `Fusion::Rrf` / `hybrid_search_with`. The vector stage uses the HNSW index by default (`hnsw` feature); disable with `--no-default-features --features float16` for an exact linear scan |
|
||||
| **`reranker`** | Multi-factor re-ranking: retrieval relevance (leads, weight 1.0), temporal recency, source authority, activation weight. Opt-in via `SearchOptions::with_rerank`; on in the OpenClaw backend |
|
||||
| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches. Opt-in via `SearchOptions::with_confidence`; on in the OpenClaw backend |
|
||||
| **`reranker`** | Multi-factor re-ranking: retrieval relevance (leads, weight 1.0), temporal recency, source authority, activation weight. Opt-in via `SearchOptions::with_rerank`; on in `ClawhdfBackend` |
|
||||
| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches. Opt-in via `SearchOptions::with_confidence`; on in `ClawhdfBackend` |
|
||||
| **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints |
|
||||
| **`multimodal`** | Cross-modal search across text/image/audio/video embeddings |
|
||||
| **`signing`** | Ed25519-signed checkpoints: SHA-256 per record in a Merkle tree, plus hashes of settings, sessions and the knowledge graph; `HDF5Memory::verify` names any edited record |
|
||||
| **`provenance`** | Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof) |
|
||||
| **`anomaly`** | Write rate limiting, 15 injection-pattern detectors, source-distribution analysis. Alerts never block a save; drain them with `take_anomaly_alerts` |
|
||||
| **`openclaw`** | OpenClaw integration: MemoryBackend trait, Markdown ↔ HDF5 conversion |
|
||||
| **`openclaw`** | `ClawhdfBackend`: a Markdown-oriented backend (ingest by section, search, read back by path, export). Named for OpenClaw, but **not an OpenClaw plugin** — see [docs/openclaw.md](docs/openclaw.md) |
|
||||
| **`vector_search`** | Flat cosine, pre-normed, SIMD, BLAS, GPU, parallel search paths |
|
||||
| **`ivf` / `pq`** | Standalone IVF and IVF-PQ indexes (benchmarked to 100K vectors); not used by `HDF5Memory`, whose ANN index is HNSW |
|
||||
| **`bm25`** | Incremental Okapi BM25 inverted index, kept for the life of the store; optional stemming |
|
||||
@@ -447,7 +448,7 @@ let work = memory.search(
|
||||
);
|
||||
|
||||
// Re-rank by relevance, recency, source authority and activation, then drop
|
||||
// low-confidence results — the pipeline the OpenClaw backend runs.
|
||||
// low-confidence results — the pipeline ClawhdfBackend runs.
|
||||
let careful = memory.search(
|
||||
&query_embedding,
|
||||
"user preferences",
|
||||
@@ -457,6 +458,35 @@ let careful = memory.search(
|
||||
);
|
||||
```
|
||||
|
||||
### Signed Checkpoints
|
||||
|
||||
```rust
|
||||
use clawhdf5_agent::signing;
|
||||
|
||||
// Once, somewhere safe: keep the secret key, publish the public key.
|
||||
let key = signing::generate_key();
|
||||
let public = key.verifying_key();
|
||||
|
||||
// Every checkpoint is signed from now on. The key is never written to disk;
|
||||
// a signed store refuses to checkpoint without it.
|
||||
memory.set_signing_key(key);
|
||||
memory.flush_wal()?;
|
||||
|
||||
// Anyone holding the public key can check the file, e.g. after copying it.
|
||||
let report = HDF5Memory::verify(std::path::Path::new("agent.h5"), &public)?;
|
||||
assert!(report.is_valid());
|
||||
// On a tampered file: report.changed_records lists the records that differ.
|
||||
```
|
||||
|
||||
The signature covers every record (text, embedding as stored, channel,
|
||||
timestamp, session, tags, deleted flag, activation), the store's settings,
|
||||
its sessions and its knowledge graph — a change made with any tool is caught.
|
||||
It covers checkpoints, not saves still in the WAL
|
||||
(`report.wal_entries_unsigned` counts those). CLI: `clawhdf5-cli keygen`,
|
||||
`--signing-key <file>` on writing commands, and `verify --public-key`.
|
||||
Signing adds about 20% to a checkpoint and 32 bytes per record to the file
|
||||
([BENCHMARKS.md § Signed checkpoints](BENCHMARKS.md#signed-checkpoints)).
|
||||
|
||||
### Knowledge Graph
|
||||
|
||||
```rust
|
||||
@@ -527,7 +557,13 @@ let ids = index.range_query(1700000000.0, 1700010800.0);
|
||||
let recent = index.latest(10);
|
||||
```
|
||||
|
||||
### OpenClaw Integration
|
||||
### Markdown Backend
|
||||
|
||||
`ClawhdfBackend` ingests Markdown by section and searches it with the full
|
||||
pipeline. It is a library API — clawhdf5 is **not** an OpenClaw memory plugin
|
||||
([docs/openclaw.md](docs/openclaw.md)). Sections stored this way carry no
|
||||
embedding, so their search is keyword-only unless you save records with
|
||||
vectors through `save_entry`.
|
||||
|
||||
```rust
|
||||
use clawhdf5_agent::openclaw::*;
|
||||
@@ -821,10 +857,10 @@ See [ROADMAP.md](ROADMAP.md) for the full implementation tracker.
|
||||
- ✅ Temporal reasoning with sub-µs queries
|
||||
- ✅ Memory security + anomaly detection
|
||||
- ✅ Multi-modal memory (text/image/audio/video)
|
||||
- ✅ OpenClaw integration layer
|
||||
- ✅ Markdown ingest/export backend (`ClawhdfBackend`); an OpenClaw plugin was never built — see [docs/openclaw.md](docs/openclaw.md)
|
||||
- ✅ Comprehensive Criterion benchmarks
|
||||
|
||||
**Phase 2** — MemoryArena and LongMemEval academic benchmarks are done (see [BENCHMARKS.md](BENCHMARKS.md), reproduced on a second machine); remaining: publish the OpenClaw TypeScript bridge to npm, crates.io/PyPI publishing.
|
||||
**Phase 2** — MemoryArena and LongMemEval academic benchmarks are done (see [BENCHMARKS.md](BENCHMARKS.md), reproduced on a second machine); remaining: crates.io/PyPI publishing. The Node bindings are unpublished and known to be broken ([known issues](docs/known-issues.md)).
|
||||
|
||||
---
|
||||
|
||||
|
||||
+14
-8
@@ -105,24 +105,30 @@
|
||||
|
||||
---
|
||||
|
||||
## Track 7: OpenClaw Integration
|
||||
**Status:** 🟢 Complete
|
||||
## Track 7: OpenClaw Integration — withdrawn (2026-09-25)
|
||||
**Status:** ⚪ Withdrawn (the items below were library work; no OpenClaw integration shipped)
|
||||
**Priority:** Critical (for adoption)
|
||||
**Crates:** `clawhdf5-agent`, `clawhdf5-napi`
|
||||
|
||||
- [x] **7.1** Memory backend trait — MemoryBackend with search/get/write/ingest/export/stats
|
||||
- [x] **7.2** Hybrid retrieval pipeline — ClawhdfBackend wires RRF → reranker → confidence rejection
|
||||
- [x] **7.3** Markdown import/export — MarkdownParser + MarkdownExporter with line tracking + metadata
|
||||
- [x] **7.4** memory_search tool — backed by full hybrid retrieval pipeline
|
||||
- [x] **7.5** memory_get tool — get() with path + line range support
|
||||
- [x] **7.4** `search()` — backed by the full hybrid retrieval pipeline (a Rust method; no OpenClaw tool was ever registered)
|
||||
- [x] **7.5** `get()` — read back by path, with a line slice (not an OpenClaw tool either)
|
||||
- [x] **7.6** Compaction integration — run_compaction() (decay + compact + WAL flush), run_consolidation() (hippocampal engine), tick_session(), flush_wal()
|
||||
- [x] **7.7** Config surface — `memory.backend = "clawhdf5"` schema documented in docs/openclaw-config.md
|
||||
- [x] **7.8** Documentation + migration guide — docs/migration-guide.md, docs/openclaw-integration.md (architecture, full API reference, code patterns)
|
||||
- [ ] **7.7** ~~Config surface — `memory.backend = "clawhdf5"`~~ — never valid OpenClaw config; docs removed
|
||||
- [ ] **7.8** ~~Documentation + migration guide~~ — removed: they described an integration that never worked
|
||||
|
||||
**Node.js bridge:** `clawhdf5-napi` (napi-rs) → `@redclaw/clawhdf5` npm package with full TypeScript types.
|
||||
**Node.js bridge:** `clawhdf5-napi` (napi-rs) and a TypeScript wrapper in `packages/clawhdf5-node` exist but are unpublished, untested in CI and known to be broken (docs/known-issues.md).
|
||||
|
||||
---
|
||||
|
||||
> **Withdrawn.** None of this track produced a working OpenClaw integration: no
|
||||
> plugin was built, the documented `memory.backend = "clawhdf5"` config was never
|
||||
> valid in any OpenClaw release, and the Node package was never published. The
|
||||
> Rust `ClawhdfBackend` remains as a library API. Not pursued for now; see
|
||||
> [docs/openclaw.md](docs/openclaw.md) for what a plugin would need today.
|
||||
|
||||
## Track 8: Benchmarking & Validation
|
||||
**Status:** 🟢 Complete
|
||||
**Priority:** High
|
||||
@@ -142,7 +148,7 @@
|
||||
|
||||
**Phase 1:** ~~Tracks 1, 2, 3 — core memory intelligence~~ 🟢 Complete
|
||||
**Phase 2:** ~~Track 4 (temporal) + Track 5 (security)~~ 🟢 Complete
|
||||
**Phase 3:** ~~Track 6 (multi-modal) + Track 7 (OpenClaw integration)~~ 🟢 Complete
|
||||
**Phase 3:** ~~Track 6 (multi-modal)~~ 🟢 Complete; Track 7 (OpenClaw integration) withdrawn
|
||||
**Phase 4:** ~~Track 8 (benchmarking + validation)~~ 🟢 Complete
|
||||
|
||||
All 8 tracks delivered. 1,650+ tests passing, zero clippy warnings.
|
||||
|
||||
@@ -19,6 +19,10 @@ clawhdf5-ann = { path = "../clawhdf5-ann", version = "2.7.0", optional = true }
|
||||
clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.7.0", optional = true, default-features = false }
|
||||
serde = { workspace = true }
|
||||
byteorder = "1"
|
||||
# Signed checkpoints (MemoryConfig-independent; see `signing`). Pure Rust.
|
||||
ed25519-dalek = { version = "2", features = ["rand_core"] }
|
||||
sha2 = "0.10"
|
||||
rand_core = { version = "0.6", features = ["getrandom"] }
|
||||
half = { workspace = true, optional = true }
|
||||
rayon = { version = "1", optional = true }
|
||||
matrixmultiply = { version = "0.3", optional = true }
|
||||
|
||||
@@ -203,7 +203,7 @@ impl ImportanceScorer {
|
||||
/// Novelty score: 1.0 − max cosine similarity against all existing records.
|
||||
/// Returns 1.0 when there are no existing memories.
|
||||
///
|
||||
/// Same result as [`Self::cosine_similarity`] against each record, but the
|
||||
/// Same result as the reference cosine similarity against each record, but the
|
||||
/// new embedding's norm is computed once rather than per record, each
|
||||
/// record costs one fused pass (dot product and its norm together) rather
|
||||
/// than three, and a large working set is scored in parallel. Every insert
|
||||
|
||||
@@ -36,6 +36,7 @@ pub mod reranker;
|
||||
pub mod schema;
|
||||
pub mod search;
|
||||
pub mod session;
|
||||
pub mod signing;
|
||||
pub mod storage;
|
||||
mod store_lock;
|
||||
pub mod temporal;
|
||||
@@ -78,6 +79,7 @@ pub use session::{SessionCache, SessionEntry};
|
||||
// --- Error type ---
|
||||
|
||||
#[derive(Debug)]
|
||||
#[non_exhaustive]
|
||||
pub enum MemoryError {
|
||||
Io(std::io::Error),
|
||||
Hdf5(String),
|
||||
@@ -88,6 +90,11 @@ pub enum MemoryError {
|
||||
/// A record the store cannot hold as given, e.g. an embedding value
|
||||
/// outside the half-precision range of a `float16` store.
|
||||
InvalidEntry(String),
|
||||
/// The store's checkpoints are signed and no signing key is set, so a
|
||||
/// checkpoint would leave it unsigned. Set the key with
|
||||
/// [`HDF5Memory::set_signing_key`], or drop the signature on purpose with
|
||||
/// [`HDF5Memory::remove_signature`].
|
||||
SigningKeyRequired(String),
|
||||
}
|
||||
|
||||
impl std::fmt::Display for MemoryError {
|
||||
@@ -99,6 +106,7 @@ impl std::fmt::Display for MemoryError {
|
||||
MemoryError::NotFound(e) => write!(f, "not found: {e}"),
|
||||
MemoryError::Locked(e) => write!(f, "store is locked: {e}"),
|
||||
MemoryError::InvalidEntry(e) => write!(f, "invalid entry: {e}"),
|
||||
MemoryError::SigningKeyRequired(e) => write!(f, "signing key required: {e}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -317,6 +325,12 @@ pub struct HDF5Memory {
|
||||
activations_dirty: bool,
|
||||
/// Opened with [`HDF5Memory::open_read_only`]: nothing may reach the disk.
|
||||
read_only: bool,
|
||||
/// Key that signs every checkpoint; never persisted. See
|
||||
/// [`HDF5Memory::set_signing_key`].
|
||||
signing_key: Option<signing::SigningKey>,
|
||||
/// Checkpoints of this store are signed: the file on disk is, or a key
|
||||
/// has been set. A checkpoint without a key is then refused.
|
||||
signed: bool,
|
||||
/// A WAL that `open()` could not read and moved aside; see
|
||||
/// [`HDF5Memory::quarantined_wal`].
|
||||
quarantined_wal: Option<PathBuf>,
|
||||
@@ -372,6 +386,8 @@ impl HDF5Memory {
|
||||
bm25_filter: bm25::TokenFilter::default(),
|
||||
activations_dirty: false,
|
||||
read_only: false,
|
||||
signing_key: None,
|
||||
signed: false,
|
||||
quarantined_wal: None,
|
||||
_lock: Some(lock),
|
||||
})
|
||||
@@ -550,6 +566,8 @@ impl HDF5Memory {
|
||||
bm25_filter: bm25::TokenFilter::default(),
|
||||
activations_dirty: false,
|
||||
read_only,
|
||||
signing_key: None,
|
||||
signed: checkpoint.signed,
|
||||
quarantined_wal,
|
||||
_lock: lock,
|
||||
})
|
||||
@@ -710,6 +728,39 @@ impl HDF5Memory {
|
||||
}
|
||||
}
|
||||
|
||||
/// Sign every checkpoint from now on with `key` (Ed25519). The key is
|
||||
/// never written anywhere; set it again after every `open`. Once a store
|
||||
/// is signed, a checkpoint without the key is refused
|
||||
/// ([`MemoryError::SigningKeyRequired`]) rather than silently leaving it
|
||||
/// unsigned. Setting a different key re-signs the store under that key
|
||||
/// from the next checkpoint; a verifier trusting the old key will then
|
||||
/// reject it, which is the point. Call [`AgentMemory::flush_wal`] to sign
|
||||
/// right away.
|
||||
pub fn set_signing_key(&mut self, key: signing::SigningKey) {
|
||||
self.signing_key = Some(key);
|
||||
self.signed = true;
|
||||
}
|
||||
|
||||
/// Stop signing: the next checkpoint writes the store unsigned. The
|
||||
/// deliberate way out of [`MemoryError::SigningKeyRequired`].
|
||||
pub fn remove_signature(&mut self) {
|
||||
self.signing_key = None;
|
||||
self.signed = false;
|
||||
}
|
||||
|
||||
/// Checkpoints of this store are signed (on disk, or from the next
|
||||
/// checkpoint because a key has been set).
|
||||
pub fn is_signed(&self) -> bool {
|
||||
self.signed
|
||||
}
|
||||
|
||||
/// Check the checkpoint at `path` against the public key the caller
|
||||
/// trusts; see [`signing::verify_store`]. Reads the file only: it works
|
||||
/// on a store another process has open.
|
||||
pub fn verify(path: &Path, trusted: &signing::VerifyingKey) -> Result<signing::VerifyReport> {
|
||||
signing::verify_store(path, trusted)
|
||||
}
|
||||
|
||||
/// Flush current state to disk and truncate the WAL.
|
||||
///
|
||||
/// Every code path that persists the full cache to the .h5 file must
|
||||
@@ -725,10 +776,28 @@ impl HDF5Memory {
|
||||
// Record which WAL prefix this checkpoint contains, so a crash before
|
||||
// the truncate below can't replay those entries a second time.
|
||||
let wal_applied = self.wal.as_ref().map(|w| w.mark());
|
||||
let signature = match &self.signing_key {
|
||||
Some(key) => Some(signing::sign(
|
||||
key,
|
||||
&self.config,
|
||||
&self.cache,
|
||||
&self.sessions,
|
||||
&self.knowledge,
|
||||
wal_applied,
|
||||
)),
|
||||
None if self.signed => {
|
||||
return Err(MemoryError::SigningKeyRequired(format!(
|
||||
"{} is signed; set its signing key before a checkpoint \
|
||||
(saves so far are held in the WAL or in memory)",
|
||||
self.config.path.display()
|
||||
)));
|
||||
}
|
||||
None => None,
|
||||
};
|
||||
// Written before the .h5 so a crash in between leaves a sidecar whose
|
||||
// generation matches no checkpoint (ignored), never the reverse.
|
||||
let ann_generation = self.persist_vector_index();
|
||||
storage::write_to_disk_with_meta(
|
||||
storage::write_to_disk_signed(
|
||||
&self.config.path,
|
||||
&self.config,
|
||||
&self.cache,
|
||||
@@ -737,7 +806,9 @@ impl HDF5Memory {
|
||||
&schema::CheckpointMeta {
|
||||
wal_applied,
|
||||
ann_generation,
|
||||
signed: signature.is_some(),
|
||||
},
|
||||
signature.as_ref(),
|
||||
)?;
|
||||
if let Some(ref mut w) = self.wal {
|
||||
w.truncate()?;
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
//! OpenClaw Integration Layer.
|
||||
//! A Markdown-oriented memory backend over [`crate::HDF5Memory`].
|
||||
//!
|
||||
//! Bridge between OpenClaw agent gateway (Markdown + sqlite-vec) and the
|
||||
//! clawhdf5 HDF5-backed memory backend. Provides:
|
||||
//! Named for OpenClaw, whose workspace memory is Markdown, but **not an
|
||||
//! OpenClaw plugin**: nothing here registers with OpenClaw, and the
|
||||
//! integration it was written for never worked (see `docs/openclaw.md`).
|
||||
//! Provides:
|
||||
//!
|
||||
//! - [`MemoryBackend`] — the trait OpenClaw implements against.
|
||||
//! - [`ClawhdfBackend`] — concrete HDF5-backed implementation.
|
||||
//! - [`MemoryBackend`] — search / read back / write / ingest / export.
|
||||
//! - [`ClawhdfBackend`] — the HDF5-backed implementation.
|
||||
//! - [`MarkdownParser`] — splits Markdown into [`MarkdownSection`] records.
|
||||
//! - [`MarkdownExporter`] — renders sections back to Markdown text.
|
||||
|
||||
@@ -61,7 +63,8 @@ pub struct BackendStats {
|
||||
// MemoryBackend trait
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
/// Interface that OpenClaw uses to interact with a memory backend.
|
||||
/// A Markdown-oriented memory backend: search, read back by path, write,
|
||||
/// ingest and export.
|
||||
///
|
||||
/// Implementors provide persistent storage, full-text + vector search,
|
||||
/// Markdown ingestion / export, and statistics.
|
||||
@@ -318,7 +321,7 @@ impl MarkdownExporter {
|
||||
///
|
||||
/// # Path mapping
|
||||
///
|
||||
/// OpenClaw addresses memories by file path (e.g. `"memory/user.md"`).
|
||||
/// Memories are addressed by file path (e.g. `"memory/user.md"`).
|
||||
/// Internally every [`MemoryEntry`] stores the originating path as its
|
||||
/// `source_channel`. Section sub-paths are stored as
|
||||
/// `"<path>::<heading>"`.
|
||||
@@ -421,7 +424,7 @@ impl ClawhdfBackend {
|
||||
|
||||
// ── Compaction & Consolidation hooks (7.6) ────────────────────────────
|
||||
|
||||
/// Run a compaction cycle — called by OpenClaw during session compaction.
|
||||
/// Run a compaction cycle (decay, compaction, WAL flush).
|
||||
///
|
||||
/// Sequence:
|
||||
/// 1. `tick_session()` — apply Hebbian decay to all activation weights.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
//!
|
||||
//! Records the origin, authorship, and a content hash of every memory chunk
|
||||
//! so the system can detect *accidental* corruption and trace data lineage.
|
||||
//! The hash is unkeyed (see [`fnv1a_64`]) — this is not a tamper-evidence or
|
||||
//! The hash is unkeyed (FNV-1a) — this is not a tamper-evidence or
|
||||
//! authenticity guarantee.
|
||||
|
||||
use std::collections::HashMap;
|
||||
|
||||
@@ -23,6 +23,7 @@ pub const ZEROCLAW_VERSION: &str = "0.8.0";
|
||||
const WAL_APPLIED_LEN_ATTR: &str = "wal_applied_len";
|
||||
const WAL_APPLIED_CRC_ATTR: &str = "wal_applied_crc";
|
||||
const ANN_GENERATION_ATTR: &str = "ann_generation";
|
||||
const SIG_VERSION_ATTR: &str = "sig_version";
|
||||
|
||||
/// Build a complete HDF5 file from the in-memory state.
|
||||
pub fn build_hdf5_file(
|
||||
@@ -46,7 +47,7 @@ pub fn build_hdf5_file_with_mark(
|
||||
) -> Result<Vec<u8>, MemoryError> {
|
||||
let meta = CheckpointMeta {
|
||||
wal_applied,
|
||||
ann_generation: None,
|
||||
..CheckpointMeta::default()
|
||||
};
|
||||
build_hdf5_file_with_meta(config, cache, sessions, knowledge, &meta)
|
||||
}
|
||||
@@ -61,6 +62,10 @@ pub struct CheckpointMeta {
|
||||
/// one left over from another checkpoint can never be attached to records
|
||||
/// it wasn't built from.
|
||||
pub ann_generation: Option<u64>,
|
||||
/// The checkpoint carries an Ed25519 signature (see [`crate::signing`]).
|
||||
/// Read-only: whether a checkpoint is *written* signed is decided by the
|
||||
/// signature passed to [`build_hdf5_file_signed`].
|
||||
pub signed: bool,
|
||||
}
|
||||
|
||||
/// [`build_hdf5_file`] with checkpoint bookkeeping.
|
||||
@@ -70,6 +75,19 @@ pub fn build_hdf5_file_with_meta(
|
||||
sessions: &SessionCache,
|
||||
knowledge: &KnowledgeCache,
|
||||
checkpoint: &CheckpointMeta,
|
||||
) -> Result<Vec<u8>, MemoryError> {
|
||||
build_hdf5_file_signed(config, cache, sessions, knowledge, checkpoint, None)
|
||||
}
|
||||
|
||||
/// [`build_hdf5_file_with_meta`], plus a signed manifest of the contents
|
||||
/// (see [`crate::signing`]).
|
||||
pub fn build_hdf5_file_signed(
|
||||
config: &MemoryConfig,
|
||||
cache: &MemoryCache,
|
||||
sessions: &SessionCache,
|
||||
knowledge: &KnowledgeCache,
|
||||
checkpoint: &CheckpointMeta,
|
||||
signature: Option<&crate::signing::StoredSignature>,
|
||||
) -> Result<Vec<u8>, MemoryError> {
|
||||
let wal_applied = checkpoint.wal_applied;
|
||||
let mut builder = clawhdf5::FileBuilder::new();
|
||||
@@ -130,11 +148,42 @@ pub fn build_hdf5_file_with_meta(
|
||||
// round trip through every reader.
|
||||
meta.set_attr(ANN_GENERATION_ATTR, AttrValue::I64(generation as i64));
|
||||
}
|
||||
if let Some(sig) = signature {
|
||||
use crate::signing::to_hex;
|
||||
let m = &sig.manifest;
|
||||
meta.set_attr(
|
||||
SIG_VERSION_ATTR,
|
||||
AttrValue::I64(crate::signing::MANIFEST_VERSION),
|
||||
);
|
||||
meta.set_attr("sig_algorithm", AttrValue::String("ed25519".into()));
|
||||
meta.set_attr("sig_public_key", AttrValue::String(to_hex(&sig.public_key)));
|
||||
meta.set_attr("sig_signature", AttrValue::String(to_hex(&sig.signature)));
|
||||
meta.set_attr("sig_record_count", AttrValue::I64(m.record_count as i64));
|
||||
meta.set_attr(
|
||||
"sig_records_root",
|
||||
AttrValue::String(to_hex(&m.records_root)),
|
||||
);
|
||||
meta.set_attr("sig_settings", AttrValue::String(to_hex(&m.settings)));
|
||||
meta.set_attr("sig_sessions", AttrValue::String(to_hex(&m.sessions)));
|
||||
meta.set_attr("sig_graph", AttrValue::String(to_hex(&m.graph)));
|
||||
}
|
||||
// Need at least one dataset in the group for it to be a proper group
|
||||
meta.create_dataset("_marker").with_u8_data(&[1]).compact();
|
||||
let finished_meta = meta.finish();
|
||||
builder.add_group(finished_meta);
|
||||
|
||||
// /integrity: the signed per-record hashes, so verification can say
|
||||
// which records changed.
|
||||
if let Some(sig) = signature {
|
||||
let mut group = builder.create_group("integrity");
|
||||
let flat: Vec<u8> = sig.record_hashes.iter().flatten().copied().collect();
|
||||
group
|
||||
.create_dataset("record_hashes")
|
||||
.with_u8_data(&flat)
|
||||
.with_shape(&[sig.record_hashes.len() as u64, 32]);
|
||||
builder.add_group(group.finish());
|
||||
}
|
||||
|
||||
// /memory group
|
||||
build_memory_group(&mut builder, config, cache)?;
|
||||
|
||||
@@ -440,6 +489,64 @@ pub fn read_wal_mark(file: &clawhdf5::File) -> Option<WalMark> {
|
||||
Some(WalMark { len, crc })
|
||||
}
|
||||
|
||||
/// Read a checkpoint's signature, if it has one. A signature whose
|
||||
/// attributes are present but malformed is an error, not "unsigned".
|
||||
pub fn read_signature(
|
||||
file: &clawhdf5::File,
|
||||
) -> Result<Option<crate::signing::StoredSignature>, MemoryError> {
|
||||
use crate::signing::{Manifest, StoredSignature, from_hex};
|
||||
let attrs = file
|
||||
.group("meta")
|
||||
.and_then(|g| g.attrs())
|
||||
.map_err(|e| MemoryError::Schema(format!("cannot read /meta attrs: {e}")))?;
|
||||
let version = match attrs.get(SIG_VERSION_ATTR) {
|
||||
None => return Ok(None),
|
||||
Some(AttrValue::I64(v)) => *v,
|
||||
Some(_) => return Err(MemoryError::Schema("malformed sig_version".into())),
|
||||
};
|
||||
if version != crate::signing::MANIFEST_VERSION {
|
||||
return Err(MemoryError::Schema(format!(
|
||||
"unsupported signature version {version}"
|
||||
)));
|
||||
}
|
||||
fn hex<const N: usize>(
|
||||
attrs: &std::collections::HashMap<String, AttrValue>,
|
||||
name: &str,
|
||||
) -> Result<[u8; N], MemoryError> {
|
||||
match attrs.get(name) {
|
||||
Some(AttrValue::String(s)) => from_hex::<N>(s),
|
||||
_ => None,
|
||||
}
|
||||
.ok_or_else(|| MemoryError::Schema(format!("malformed or missing {name}")))
|
||||
}
|
||||
let record_count = match attrs.get("sig_record_count") {
|
||||
Some(AttrValue::I64(v)) if *v >= 0 => *v as u64,
|
||||
_ => return Err(MemoryError::Schema("malformed sig_record_count".into())),
|
||||
};
|
||||
let group = file
|
||||
.group("integrity")
|
||||
.map_err(|e| MemoryError::Schema(format!("signed checkpoint without /integrity: {e}")))?;
|
||||
let flat = read_u8_dataset(&group, "record_hashes")?;
|
||||
if flat.len() % 32 != 0 {
|
||||
return Err(MemoryError::Schema(
|
||||
"/integrity/record_hashes is not a whole number of hashes".into(),
|
||||
));
|
||||
}
|
||||
let record_hashes = flat.as_chunks::<32>().0.to_vec();
|
||||
Ok(Some(StoredSignature {
|
||||
manifest: Manifest {
|
||||
record_count,
|
||||
records_root: hex::<32>(&attrs, "sig_records_root")?,
|
||||
settings: hex::<32>(&attrs, "sig_settings")?,
|
||||
sessions: hex::<32>(&attrs, "sig_sessions")?,
|
||||
graph: hex::<32>(&attrs, "sig_graph")?,
|
||||
},
|
||||
record_hashes,
|
||||
public_key: hex::<32>(&attrs, "sig_public_key")?,
|
||||
signature: hex::<64>(&attrs, "sig_signature")?,
|
||||
}))
|
||||
}
|
||||
|
||||
/// Read the checkpoint bookkeeping from `/meta`.
|
||||
pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta {
|
||||
let ann_generation = file
|
||||
@@ -450,9 +557,14 @@ pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta {
|
||||
Some(AttrValue::I64(v)) => Some(*v as u64),
|
||||
_ => None,
|
||||
});
|
||||
let signed = file
|
||||
.group("meta")
|
||||
.and_then(|g| g.attrs())
|
||||
.is_ok_and(|attrs| attrs.contains_key(SIG_VERSION_ATTR));
|
||||
CheckpointMeta {
|
||||
wal_applied: read_wal_mark(file),
|
||||
ann_generation,
|
||||
signed,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,419 @@
|
||||
//! Ed25519-signed checkpoints.
|
||||
//!
|
||||
//! When a signing key is set ([`crate::HDF5Memory::set_signing_key`]), every
|
||||
//! checkpoint writes a signed manifest of the store: a SHA-256 per memory
|
||||
//! record rolled into a Merkle root, plus hashes of the store's settings, its
|
||||
//! sessions and its knowledge graph. [`verify_store`] recomputes all of it from
|
||||
//! the file and checks the signature against a public key the caller trusts,
|
||||
//! so any change to the checkpointed file — a record's text or embedding, a
|
||||
//! setting, a session, a graph edge, made through this crate or any other HDF5
|
||||
//! tool — is detected, and the per-record hashes say which records changed.
|
||||
//!
|
||||
//! What it does not cover: saves still only in the WAL (made since the last
|
||||
//! checkpoint). [`VerifyReport::wal_entries_unsigned`] counts them.
|
||||
//!
|
||||
//! The hashes cover exactly what the file persists, in the form the loader
|
||||
//! returns it, so a store verifies after any number of reopen/checkpoint
|
||||
//! cycles. Derived data (L2 norms, the vector index) is not covered; it is
|
||||
//! recomputed from covered data.
|
||||
|
||||
use ed25519_dalek::{Signature, Signer, Verifier};
|
||||
pub use ed25519_dalek::{SigningKey, VerifyingKey};
|
||||
use sha2::{Digest, Sha256};
|
||||
|
||||
use crate::MemoryConfig;
|
||||
use crate::cache::MemoryCache;
|
||||
use crate::knowledge::KnowledgeCache;
|
||||
use crate::session::SessionCache;
|
||||
use crate::wal::WalMark;
|
||||
|
||||
/// Version of the manifest encoding; part of what is signed.
|
||||
pub const MANIFEST_VERSION: i64 = 1;
|
||||
|
||||
type Hash = [u8; 32];
|
||||
|
||||
/// The hashes a signature covers.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct Manifest {
|
||||
pub record_count: u64,
|
||||
/// Merkle root over the per-record hashes.
|
||||
pub records_root: Hash,
|
||||
/// Settings persisted in `/meta`, plus the checkpoint's WAL mark.
|
||||
pub settings: Hash,
|
||||
pub sessions: Hash,
|
||||
pub graph: Hash,
|
||||
}
|
||||
|
||||
impl Manifest {
|
||||
/// The exact bytes that are signed.
|
||||
pub fn signed_bytes(&self) -> Vec<u8> {
|
||||
let mut m = Vec::with_capacity(160);
|
||||
m.extend_from_slice(b"clawhdf5-agent signed checkpoint\0");
|
||||
m.extend_from_slice(&MANIFEST_VERSION.to_le_bytes());
|
||||
m.extend_from_slice(&self.record_count.to_le_bytes());
|
||||
m.extend_from_slice(&self.records_root);
|
||||
m.extend_from_slice(&self.settings);
|
||||
m.extend_from_slice(&self.sessions);
|
||||
m.extend_from_slice(&self.graph);
|
||||
m
|
||||
}
|
||||
}
|
||||
|
||||
/// A signature as stored in a checkpoint.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct StoredSignature {
|
||||
pub manifest: Manifest,
|
||||
pub record_hashes: Vec<Hash>,
|
||||
pub public_key: [u8; 32],
|
||||
pub signature: [u8; 64],
|
||||
}
|
||||
|
||||
/// Build the manifest (and per-record hashes) for the state about to be
|
||||
/// checkpointed, and sign it.
|
||||
pub fn sign(
|
||||
key: &SigningKey,
|
||||
config: &MemoryConfig,
|
||||
cache: &MemoryCache,
|
||||
sessions: &SessionCache,
|
||||
knowledge: &KnowledgeCache,
|
||||
wal_applied: Option<WalMark>,
|
||||
) -> StoredSignature {
|
||||
let (manifest, record_hashes) = manifest(config, cache, sessions, knowledge, wal_applied);
|
||||
let signature = key.sign(&manifest.signed_bytes()).to_bytes();
|
||||
StoredSignature {
|
||||
manifest,
|
||||
record_hashes,
|
||||
public_key: key.verifying_key().to_bytes(),
|
||||
signature,
|
||||
}
|
||||
}
|
||||
|
||||
/// Compute the manifest of a store's state.
|
||||
pub fn manifest(
|
||||
config: &MemoryConfig,
|
||||
cache: &MemoryCache,
|
||||
sessions: &SessionCache,
|
||||
knowledge: &KnowledgeCache,
|
||||
wal_applied: Option<WalMark>,
|
||||
) -> (Manifest, Vec<Hash>) {
|
||||
let record_hashes: Vec<Hash> = (0..cache.len()).map(|i| record_hash(cache, i)).collect();
|
||||
let manifest = Manifest {
|
||||
record_count: cache.len() as u64,
|
||||
records_root: merkle_root(&record_hashes),
|
||||
settings: settings_hash(config, wal_applied),
|
||||
sessions: sessions_hash(sessions),
|
||||
graph: graph_hash(knowledge),
|
||||
};
|
||||
(manifest, record_hashes)
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Canonical encoding
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// A SHA-256 over length-prefixed fields, so no two different field lists
|
||||
/// hash the same bytes.
|
||||
struct Fields(Sha256);
|
||||
|
||||
impl Fields {
|
||||
fn new(domain: &str) -> Self {
|
||||
let mut h = Sha256::new();
|
||||
h.update((domain.len() as u64).to_le_bytes());
|
||||
h.update(domain.as_bytes());
|
||||
Self(h)
|
||||
}
|
||||
fn bytes(&mut self, b: &[u8]) -> &mut Self {
|
||||
self.0.update((b.len() as u64).to_le_bytes());
|
||||
self.0.update(b);
|
||||
self
|
||||
}
|
||||
/// Strings as the loader returns them: stored null-padded, so a trailing
|
||||
/// NUL cannot survive a round trip and must not be part of the hash.
|
||||
fn str(&mut self, s: &str) -> &mut Self {
|
||||
self.bytes(s.trim_end_matches('\0').as_bytes())
|
||||
}
|
||||
fn u64(&mut self, v: u64) -> &mut Self {
|
||||
self.0.update(v.to_le_bytes());
|
||||
self
|
||||
}
|
||||
fn f64(&mut self, v: f64) -> &mut Self {
|
||||
self.0.update(v.to_bits().to_le_bytes());
|
||||
self
|
||||
}
|
||||
fn f32(&mut self, v: f32) -> &mut Self {
|
||||
self.0.update(v.to_bits().to_le_bytes());
|
||||
self
|
||||
}
|
||||
fn finish(self) -> Hash {
|
||||
self.0.finalize().into()
|
||||
}
|
||||
}
|
||||
|
||||
/// Everything persisted about record `i`, including its position. The
|
||||
/// embedding is hashed as the cache holds it — for a `float16` store that is
|
||||
/// the half-rounded value the file holds.
|
||||
fn record_hash(cache: &MemoryCache, i: usize) -> Hash {
|
||||
let mut f = Fields::new("clawhdf5-agent/record");
|
||||
f.u64(i as u64).str(&cache.chunks[i]);
|
||||
let emb: Vec<u8> = cache.embeddings[i]
|
||||
.iter()
|
||||
.flat_map(|v| v.to_bits().to_le_bytes())
|
||||
.collect();
|
||||
f.bytes(&emb)
|
||||
.str(&cache.source_channels[i])
|
||||
.f64(cache.timestamps[i])
|
||||
.str(&cache.session_ids[i])
|
||||
.str(&cache.tags[i])
|
||||
.u64(u64::from(cache.tombstones[i]))
|
||||
.f32(cache.activation_weights[i]);
|
||||
f.finish()
|
||||
}
|
||||
|
||||
/// Binary Merkle tree: leaves are the record hashes; a parent hashes its two
|
||||
/// children with a node prefix; an odd node is carried up unchanged.
|
||||
fn merkle_root(leaves: &[Hash]) -> Hash {
|
||||
if leaves.is_empty() {
|
||||
return Fields::new("clawhdf5-agent/merkle-empty").finish();
|
||||
}
|
||||
let mut level: Vec<Hash> = leaves.to_vec();
|
||||
while level.len() > 1 {
|
||||
level = level
|
||||
.chunks(2)
|
||||
.map(|pair| match pair {
|
||||
[l, r] => {
|
||||
let mut h = Sha256::new();
|
||||
h.update([1u8]);
|
||||
h.update(l);
|
||||
h.update(r);
|
||||
h.finalize().into()
|
||||
}
|
||||
[only] => *only,
|
||||
_ => unreachable!(),
|
||||
})
|
||||
.collect();
|
||||
}
|
||||
level[0]
|
||||
}
|
||||
|
||||
fn settings_hash(c: &MemoryConfig, wal_applied: Option<WalMark>) -> Hash {
|
||||
let mut f = Fields::new("clawhdf5-agent/settings");
|
||||
f.str(crate::schema::SCHEMA_VERSION)
|
||||
.str(&c.created_at)
|
||||
.str(&c.agent_id)
|
||||
.str(&c.embedder)
|
||||
.u64(c.embedding_dim as u64)
|
||||
.u64(c.chunk_size as u64)
|
||||
.u64(c.overlap as u64)
|
||||
.u64(u64::from(c.float16))
|
||||
.u64(u64::from(c.compression))
|
||||
.u64(u64::from(c.compression_level))
|
||||
.f32(c.compact_threshold)
|
||||
.f32(c.hebbian_boost)
|
||||
.f32(c.decay_factor)
|
||||
.u64(u64::from(c.wal_enabled))
|
||||
.u64(c.wal_max_entries as u64)
|
||||
.u64(u64::from(c.quantized_index))
|
||||
.u64(c.hnsw_m as u64)
|
||||
.u64(c.hnsw_ef_construction as u64)
|
||||
.u64(c.hnsw_ef_search as u64);
|
||||
// An empty mark is not written to the file, so it must hash as none.
|
||||
match wal_applied.filter(|m| m.len > 0) {
|
||||
Some(m) => f.u64(1).u64(m.len).u64(u64::from(m.crc)),
|
||||
None => f.u64(0),
|
||||
};
|
||||
f.finish()
|
||||
}
|
||||
|
||||
fn sessions_hash(s: &SessionCache) -> Hash {
|
||||
let mut f = Fields::new("clawhdf5-agent/sessions");
|
||||
f.u64(s.entries.len() as u64);
|
||||
for (i, e) in s.entries.iter().enumerate() {
|
||||
f.str(&e.id)
|
||||
.u64(e.start_idx)
|
||||
.u64(e.end_idx)
|
||||
.str(&e.channel)
|
||||
.f64(e.ts)
|
||||
.str(s.summaries.get(i).map(String::as_str).unwrap_or(""));
|
||||
}
|
||||
f.finish()
|
||||
}
|
||||
|
||||
fn graph_hash(k: &KnowledgeCache) -> Hash {
|
||||
let mut f = Fields::new("clawhdf5-agent/graph");
|
||||
f.u64(k.entities.len() as u64);
|
||||
for e in &k.entities {
|
||||
f.u64(e.id)
|
||||
.str(&e.name)
|
||||
.str(&e.entity_type)
|
||||
.u64(e.embedding_idx as u64);
|
||||
}
|
||||
f.u64(k.relations.len() as u64);
|
||||
for r in &k.relations {
|
||||
f.u64(r.src)
|
||||
.u64(r.tgt)
|
||||
.str(&r.relation)
|
||||
.f32(r.weight)
|
||||
.f64(r.ts);
|
||||
}
|
||||
f.u64(k.alias_strings.len() as u64);
|
||||
for (s, id) in k.alias_strings.iter().zip(&k.alias_entity_ids) {
|
||||
f.str(s).u64(*id as u64);
|
||||
}
|
||||
f.finish()
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Verification
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// The outcome of [`verify_store`].
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct VerifyReport {
|
||||
/// The checkpoint carries a signature.
|
||||
pub signed: bool,
|
||||
/// The signature was made by the key the caller trusts.
|
||||
pub key_matches: bool,
|
||||
/// The signature over the stored manifest is valid.
|
||||
pub signature_valid: bool,
|
||||
/// The file's current contents match the signed manifest.
|
||||
pub records_match: bool,
|
||||
pub settings_match: bool,
|
||||
pub sessions_match: bool,
|
||||
pub graph_match: bool,
|
||||
/// Records whose contents differ from what was signed (by position),
|
||||
/// when the stored per-record hashes are themselves authentic.
|
||||
pub changed_records: Vec<usize>,
|
||||
/// Records in the file versus in the signed manifest.
|
||||
pub record_count: u64,
|
||||
pub signed_record_count: u64,
|
||||
/// The public key the checkpoint claims to be signed by.
|
||||
pub public_key: Option<[u8; 32]>,
|
||||
/// Saves in the WAL after the checkpoint: not covered by the signature.
|
||||
pub wal_entries_unsigned: usize,
|
||||
}
|
||||
|
||||
impl VerifyReport {
|
||||
/// Signed by the trusted key, signature valid, and every part of the
|
||||
/// file unchanged since it was signed.
|
||||
pub fn is_valid(&self) -> bool {
|
||||
self.signed
|
||||
&& self.key_matches
|
||||
&& self.signature_valid
|
||||
&& self.records_match
|
||||
&& self.settings_match
|
||||
&& self.sessions_match
|
||||
&& self.graph_match
|
||||
}
|
||||
}
|
||||
|
||||
/// Check a store file against the public key the caller trusts.
|
||||
///
|
||||
/// Reads the checkpoint (not the WAL), recomputes every hash from its
|
||||
/// contents and checks the signature. Never writes.
|
||||
pub fn verify_store(
|
||||
path: &std::path::Path,
|
||||
trusted: &VerifyingKey,
|
||||
) -> Result<VerifyReport, crate::MemoryError> {
|
||||
let file = clawhdf5::File::open(path)
|
||||
.map_err(|e| crate::MemoryError::Hdf5(format!("cannot open {}: {e}", path.display())))?;
|
||||
let (config, cache, sessions, knowledge) = crate::schema::validate_and_load(&file)?;
|
||||
let checkpoint = crate::schema::read_checkpoint_meta(&file);
|
||||
let stored = crate::schema::read_signature(&file)?;
|
||||
let wal_entries_unsigned = count_wal_entries_after(path, checkpoint.wal_applied);
|
||||
|
||||
let (current, current_hashes) = manifest(
|
||||
&config,
|
||||
&cache,
|
||||
&sessions,
|
||||
&knowledge,
|
||||
checkpoint.wal_applied,
|
||||
);
|
||||
|
||||
let Some(stored) = stored else {
|
||||
return Ok(VerifyReport {
|
||||
signed: false,
|
||||
key_matches: false,
|
||||
signature_valid: false,
|
||||
records_match: false,
|
||||
settings_match: false,
|
||||
sessions_match: false,
|
||||
graph_match: false,
|
||||
changed_records: Vec::new(),
|
||||
record_count: current.record_count,
|
||||
signed_record_count: 0,
|
||||
public_key: None,
|
||||
wal_entries_unsigned,
|
||||
});
|
||||
};
|
||||
|
||||
let key_matches = stored.public_key == trusted.to_bytes();
|
||||
let signature_valid = trusted
|
||||
.verify(
|
||||
&stored.manifest.signed_bytes(),
|
||||
&Signature::from_bytes(&stored.signature),
|
||||
)
|
||||
.is_ok();
|
||||
// The stored per-record hashes can localise a change only if they are
|
||||
// the ones that were signed.
|
||||
let hashes_authentic = signature_valid
|
||||
&& stored.record_hashes.len() as u64 == stored.manifest.record_count
|
||||
&& merkle_root(&stored.record_hashes) == stored.manifest.records_root;
|
||||
let changed_records = if hashes_authentic {
|
||||
let n = current_hashes.len().max(stored.record_hashes.len());
|
||||
(0..n)
|
||||
.filter(|&i| current_hashes.get(i) != stored.record_hashes.get(i))
|
||||
.collect()
|
||||
} else {
|
||||
Vec::new()
|
||||
};
|
||||
|
||||
Ok(VerifyReport {
|
||||
signed: true,
|
||||
key_matches,
|
||||
signature_valid,
|
||||
records_match: signature_valid
|
||||
&& current.record_count == stored.manifest.record_count
|
||||
&& current.records_root == stored.manifest.records_root,
|
||||
settings_match: signature_valid && current.settings == stored.manifest.settings,
|
||||
sessions_match: signature_valid && current.sessions == stored.manifest.sessions,
|
||||
graph_match: signature_valid && current.graph == stored.manifest.graph,
|
||||
changed_records,
|
||||
record_count: current.record_count,
|
||||
signed_record_count: stored.manifest.record_count,
|
||||
public_key: Some(stored.public_key),
|
||||
wal_entries_unsigned,
|
||||
})
|
||||
}
|
||||
|
||||
fn count_wal_entries_after(store: &std::path::Path, mark: Option<WalMark>) -> usize {
|
||||
let wal = store.with_extension("h5.wal");
|
||||
if !wal.exists() {
|
||||
return 0;
|
||||
}
|
||||
crate::wal::WalFile::read_entries_for_migration(&wal, mark)
|
||||
.map(|e| e.len())
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
/// A new random signing key from the operating system's RNG.
|
||||
pub fn generate_key() -> SigningKey {
|
||||
SigningKey::generate(&mut rand_core::OsRng)
|
||||
}
|
||||
|
||||
/// Hex encoding for keys and signatures in attributes and the CLI.
|
||||
pub fn to_hex(bytes: &[u8]) -> String {
|
||||
bytes.iter().map(|b| format!("{b:02x}")).collect()
|
||||
}
|
||||
|
||||
/// Parse hex into exactly `N` bytes.
|
||||
pub fn from_hex<const N: usize>(s: &str) -> Option<[u8; N]> {
|
||||
let s = s.trim();
|
||||
if s.len() != 2 * N {
|
||||
return None;
|
||||
}
|
||||
let mut out = [0u8; N];
|
||||
for (i, byte) in out.iter_mut().enumerate() {
|
||||
*byte = u8::from_str_radix(&s[2 * i..2 * i + 2], 16).ok()?;
|
||||
}
|
||||
Some(out)
|
||||
}
|
||||
@@ -36,7 +36,7 @@ pub fn write_to_disk_with_mark(
|
||||
) -> Result<(), MemoryError> {
|
||||
let meta = schema::CheckpointMeta {
|
||||
wal_applied,
|
||||
ann_generation: None,
|
||||
..schema::CheckpointMeta::default()
|
||||
};
|
||||
write_to_disk_with_meta(path, config, cache, sessions, knowledge, &meta)
|
||||
}
|
||||
@@ -50,7 +50,21 @@ pub fn write_to_disk_with_meta(
|
||||
knowledge: &KnowledgeCache,
|
||||
checkpoint: &schema::CheckpointMeta,
|
||||
) -> Result<(), MemoryError> {
|
||||
let bytes = schema::build_hdf5_file_with_meta(config, cache, sessions, knowledge, checkpoint)?;
|
||||
write_to_disk_signed(path, config, cache, sessions, knowledge, checkpoint, None)
|
||||
}
|
||||
|
||||
/// [`write_to_disk_with_meta`] with a signed manifest of the contents.
|
||||
pub fn write_to_disk_signed(
|
||||
path: &Path,
|
||||
config: &MemoryConfig,
|
||||
cache: &MemoryCache,
|
||||
sessions: &SessionCache,
|
||||
knowledge: &KnowledgeCache,
|
||||
checkpoint: &schema::CheckpointMeta,
|
||||
signature: Option<&crate::signing::StoredSignature>,
|
||||
) -> Result<(), MemoryError> {
|
||||
let bytes =
|
||||
schema::build_hdf5_file_signed(config, cache, sessions, knowledge, checkpoint, signature)?;
|
||||
|
||||
if bytes.is_empty() {
|
||||
return Err(MemoryError::Hdf5("build_hdf5_file produced 0 bytes".into()));
|
||||
|
||||
@@ -92,3 +92,64 @@ print(len(names))
|
||||
assert!(n >= 10, "only {n} datasets");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_edit_made_with_h5py_breaks_the_signature_and_names_the_record() {
|
||||
if !h5py_available() {
|
||||
assert!(
|
||||
std::env::var("CLAWHDF5_REQUIRE_INTEROP").as_deref() != Ok("1"),
|
||||
"CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
|
||||
);
|
||||
eprintln!("SKIP: python3 with h5py not available");
|
||||
return;
|
||||
}
|
||||
use clawhdf5_agent::signing::SigningKey;
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("signed.h5");
|
||||
let key = SigningKey::from_bytes(&[42; 32]);
|
||||
let mut m = HDF5Memory::create(MemoryConfig::new(path.clone(), "agent", 8)).unwrap();
|
||||
m.set_signing_key(key.clone());
|
||||
m.save_batch(
|
||||
(0..10)
|
||||
.map(|i| MemoryEntry {
|
||||
chunk: format!("memory {i}"),
|
||||
embedding: (0..8).map(|j| ((i * 8 + j) as f32).cos()).collect(),
|
||||
source_channel: "test".into(),
|
||||
timestamp: i as f64,
|
||||
session_id: "s".into(),
|
||||
tags: String::new(),
|
||||
})
|
||||
.collect(),
|
||||
)
|
||||
.unwrap();
|
||||
drop(m);
|
||||
assert!(
|
||||
HDF5Memory::verify(&path, &key.verifying_key())
|
||||
.unwrap()
|
||||
.is_valid()
|
||||
);
|
||||
|
||||
// Someone edits one timestamp in place with h5py.
|
||||
let script = format!(
|
||||
r#"
|
||||
import h5py
|
||||
with h5py.File("{}", "r+") as f:
|
||||
ts = f["memory/timestamps"]
|
||||
ts[3] = 12345.0
|
||||
"#,
|
||||
path.display()
|
||||
);
|
||||
let out = Command::new(python())
|
||||
.args(["-c", &script])
|
||||
.output()
|
||||
.unwrap();
|
||||
assert!(
|
||||
out.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&out.stderr)
|
||||
);
|
||||
|
||||
let r = HDF5Memory::verify(&path, &key.verifying_key()).unwrap();
|
||||
assert!(r.signature_valid && !r.is_valid(), "{r:?}");
|
||||
assert_eq!(r.changed_records, vec![3]);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,330 @@
|
||||
//! Ed25519-signed checkpoints: `HDF5Memory::set_signing_key` and
|
||||
//! `HDF5Memory::verify`.
|
||||
|
||||
use std::path::Path;
|
||||
|
||||
use clawhdf5_agent::signing::{SigningKey, VerifyReport, VerifyingKey};
|
||||
use clawhdf5_agent::storage;
|
||||
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, MemoryError, schema};
|
||||
use tempfile::TempDir;
|
||||
|
||||
const DIM: usize = 16;
|
||||
|
||||
fn key(seed: u8) -> SigningKey {
|
||||
SigningKey::from_bytes(&[seed; 32])
|
||||
}
|
||||
|
||||
fn entry(i: usize, chunk: &str) -> MemoryEntry {
|
||||
MemoryEntry {
|
||||
chunk: chunk.to_string(),
|
||||
embedding: (0..DIM)
|
||||
.map(|j| ((i * DIM + j) as f32 * 0.37).sin())
|
||||
.collect(),
|
||||
source_channel: "chat".into(),
|
||||
timestamp: 1_700_000_000.0 + i as f64,
|
||||
session_id: format!("s{}", i % 3),
|
||||
tags: format!("t{i}"),
|
||||
}
|
||||
}
|
||||
|
||||
/// Awkward strings on purpose: they must hash the same after a round trip.
|
||||
const TEXTS: [&str; 6] = [
|
||||
"plain text",
|
||||
"ünïcödé — 日本語 🙂",
|
||||
"",
|
||||
"trailing spaces ",
|
||||
"tab\tand\nnewline",
|
||||
"x",
|
||||
];
|
||||
|
||||
fn signed_store(dir: &TempDir, float16: bool, k: &SigningKey) -> std::path::PathBuf {
|
||||
let mut cfg = MemoryConfig::new(dir.path().join("s.h5"), "agent", DIM);
|
||||
cfg.float16 = float16;
|
||||
let path = cfg.path.clone();
|
||||
let mut m = HDF5Memory::create(cfg).unwrap();
|
||||
m.set_signing_key(k.clone());
|
||||
let entries = (0..30).map(|i| entry(i, TEXTS[i % TEXTS.len()])).collect();
|
||||
m.save_batch(entries).unwrap();
|
||||
// Some graph and a deleted record, so every part of the manifest is used.
|
||||
let a = m.knowledge_mut().add_entity("Alice", "person", 0);
|
||||
let b = m.knowledge_mut().add_entity("Acme", "org", -1);
|
||||
m.knowledge_mut().add_relation(a, b, "works_at", 0.75);
|
||||
m.sessions_mut()
|
||||
.add_at("s0", 0, 9, "chat", "first session", 1_700_000_000.0);
|
||||
m.delete(4).unwrap();
|
||||
m.flush_wal().unwrap();
|
||||
path
|
||||
}
|
||||
|
||||
fn verify(path: &Path, k: &SigningKey) -> VerifyReport {
|
||||
HDF5Memory::verify(path, &k.verifying_key()).unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_signed_store_verifies_through_reopen_and_checkpoint_cycles() {
|
||||
for float16 in [true, false] {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let k = key(7);
|
||||
let path = signed_store(&dir, float16, &k);
|
||||
let r = verify(&path, &k);
|
||||
assert!(r.is_valid(), "float16={float16}: {r:?}");
|
||||
assert_eq!(r.public_key, Some(k.verifying_key().to_bytes()));
|
||||
assert_eq!(r.record_count, 30);
|
||||
assert!(r.changed_records.is_empty());
|
||||
|
||||
// Reopen, change nothing, checkpoint again (with the key): still valid.
|
||||
for _ in 0..3 {
|
||||
let mut m = HDF5Memory::open(&path).unwrap();
|
||||
assert!(m.is_signed());
|
||||
m.set_signing_key(k.clone());
|
||||
m.flush_wal().unwrap();
|
||||
drop(m);
|
||||
assert!(verify(&path, &k).is_valid());
|
||||
}
|
||||
// And after real changes, re-signed.
|
||||
let mut m = HDF5Memory::open(&path).unwrap();
|
||||
m.set_signing_key(k.clone());
|
||||
m.save(entry(99, "added later")).unwrap();
|
||||
m.hybrid_search(&entry(1, "").embedding, "text", 0.4, 0.6, 5);
|
||||
m.flush_wal().unwrap();
|
||||
drop(m);
|
||||
let r = verify(&path, &k);
|
||||
assert!(r.is_valid(), "{r:?}");
|
||||
assert_eq!(r.record_count, 31);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_signed_store_refuses_to_checkpoint_without_its_key() {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let k = key(1);
|
||||
let path = signed_store(&dir, true, &k);
|
||||
|
||||
let mut m = HDF5Memory::open(&path).unwrap();
|
||||
m.save(entry(50, "pending")).unwrap();
|
||||
match m.flush_wal() {
|
||||
Err(MemoryError::SigningKeyRequired(msg)) => assert!(msg.contains("signed"), "{msg}"),
|
||||
other => panic!("expected SigningKeyRequired, got {other:?}"),
|
||||
}
|
||||
// The file is untouched and still valid; the save is still in the WAL.
|
||||
let r = verify(&path, &k);
|
||||
assert!(r.is_valid());
|
||||
assert_eq!(r.wal_entries_unsigned, 1);
|
||||
|
||||
// Supplying the key lets the checkpoint through, signed.
|
||||
m.set_signing_key(k.clone());
|
||||
m.flush_wal().unwrap();
|
||||
drop(m);
|
||||
let r = verify(&path, &k);
|
||||
assert!(r.is_valid());
|
||||
assert_eq!((r.record_count, r.wal_entries_unsigned), (31, 0));
|
||||
|
||||
// Removing the signature on purpose writes it unsigned.
|
||||
let mut m = HDF5Memory::open(&path).unwrap();
|
||||
m.remove_signature();
|
||||
m.flush_wal().unwrap();
|
||||
drop(m);
|
||||
let r = verify(&path, &k);
|
||||
assert!(!r.signed && !r.is_valid());
|
||||
assert!(!HDF5Memory::open(&path).unwrap().is_signed());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_wrong_key_does_not_verify_and_a_new_key_re_signs() {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let (a, b) = (key(1), key(2));
|
||||
let path = signed_store(&dir, true, &a);
|
||||
let r = verify(&path, &b);
|
||||
assert!(r.signed && !r.key_matches && !r.signature_valid && !r.is_valid());
|
||||
|
||||
let mut m = HDF5Memory::open(&path).unwrap();
|
||||
m.set_signing_key(b.clone());
|
||||
m.flush_wal().unwrap();
|
||||
drop(m);
|
||||
assert!(verify(&path, &b).is_valid());
|
||||
assert!(!verify(&path, &a).is_valid());
|
||||
}
|
||||
|
||||
/// Rewrite the store with changed contents but the *old* signature — what
|
||||
/// someone with write access to the file, but not the key, can do.
|
||||
fn tamper(path: &Path, change: impl FnOnce(&mut Tampered)) {
|
||||
let file = clawhdf5::File::open(path).unwrap();
|
||||
let (config, cache, sessions, knowledge) = schema::validate_and_load(&file).unwrap();
|
||||
let checkpoint = schema::read_checkpoint_meta(&file);
|
||||
let signature = schema::read_signature(&file).unwrap().unwrap();
|
||||
drop(file);
|
||||
let mut t = Tampered {
|
||||
config,
|
||||
cache,
|
||||
sessions,
|
||||
knowledge,
|
||||
};
|
||||
change(&mut t);
|
||||
storage::write_to_disk_signed(
|
||||
path,
|
||||
&t.config,
|
||||
&t.cache,
|
||||
&t.sessions,
|
||||
&t.knowledge,
|
||||
&checkpoint,
|
||||
Some(&signature),
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
struct Tampered {
|
||||
config: MemoryConfig,
|
||||
cache: clawhdf5_agent::cache::MemoryCache,
|
||||
sessions: clawhdf5_agent::SessionCache,
|
||||
knowledge: clawhdf5_agent::knowledge::KnowledgeCache,
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_kind_of_edit_is_detected_and_located() {
|
||||
let k = key(3);
|
||||
type Edit = Box<dyn FnOnce(&mut Tampered)>;
|
||||
type Case = (&'static str, Edit, fn(&VerifyReport) -> bool);
|
||||
let cases: Vec<Case> = vec![
|
||||
(
|
||||
"record text",
|
||||
Box::new(|t: &mut Tampered| t.cache.chunks[7] = "rewritten".into()),
|
||||
|r| !r.records_match && r.changed_records == vec![7],
|
||||
),
|
||||
(
|
||||
"one embedding value",
|
||||
Box::new(|t: &mut Tampered| {
|
||||
let mut e = t.cache.embeddings[12].to_vec();
|
||||
e[3] = 0.5;
|
||||
t.cache.embeddings.set(12, &e);
|
||||
}),
|
||||
|r| r.changed_records == vec![12],
|
||||
),
|
||||
(
|
||||
"undelete",
|
||||
Box::new(|t: &mut Tampered| t.cache.tombstones[4] = 0),
|
||||
|r| r.changed_records == vec![4],
|
||||
),
|
||||
(
|
||||
"timestamp",
|
||||
Box::new(|t: &mut Tampered| t.cache.timestamps[20] += 1.0),
|
||||
|r| r.changed_records == vec![20],
|
||||
),
|
||||
(
|
||||
"record appended",
|
||||
Box::new(|t: &mut Tampered| {
|
||||
t.cache.push(
|
||||
"new".into(),
|
||||
vec![0.1; DIM],
|
||||
"x".into(),
|
||||
1.0,
|
||||
"s".into(),
|
||||
"".into(),
|
||||
);
|
||||
}),
|
||||
|r| !r.records_match && r.changed_records == vec![30] && r.record_count == 31,
|
||||
),
|
||||
(
|
||||
"setting",
|
||||
Box::new(|t: &mut Tampered| t.config.agent_id = "someone-else".into()),
|
||||
|r| !r.settings_match && r.records_match,
|
||||
),
|
||||
(
|
||||
"session summary",
|
||||
Box::new(|t: &mut Tampered| t.sessions.summaries[0] = "edited".into()),
|
||||
|r| !r.sessions_match && r.records_match,
|
||||
),
|
||||
(
|
||||
"graph edge",
|
||||
Box::new(|t: &mut Tampered| t.knowledge.relations[0].weight = 1.0),
|
||||
|r| !r.graph_match && r.records_match,
|
||||
),
|
||||
];
|
||||
for (name, edit, check) in cases {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let path = signed_store(&dir, true, &k);
|
||||
tamper(&path, edit);
|
||||
let r = verify(&path, &k);
|
||||
assert!(
|
||||
r.signed && r.key_matches && r.signature_valid,
|
||||
"{name}: {r:?}"
|
||||
);
|
||||
assert!(!r.is_valid(), "{name}: edit not detected: {r:?}");
|
||||
assert!(check(&r), "{name}: {r:?}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_forged_manifest_fails_the_signature() {
|
||||
// Recomputing the hashes for tampered contents does not help without the
|
||||
// key: the signature no longer matches the manifest.
|
||||
let dir = TempDir::new().unwrap();
|
||||
let k = key(5);
|
||||
let path = signed_store(&dir, true, &k);
|
||||
let file = clawhdf5::File::open(&path).unwrap();
|
||||
let (config, mut cache, sessions, knowledge) = schema::validate_and_load(&file).unwrap();
|
||||
let checkpoint = schema::read_checkpoint_meta(&file);
|
||||
let mut sig = schema::read_signature(&file).unwrap().unwrap();
|
||||
drop(file);
|
||||
cache.chunks[0] = "forged".into();
|
||||
// Re-sign with an attacker key, then splice the victim's public key back.
|
||||
let forged = clawhdf5_agent::signing::sign(
|
||||
&key(66),
|
||||
&config,
|
||||
&cache,
|
||||
&sessions,
|
||||
&knowledge,
|
||||
checkpoint.wal_applied,
|
||||
);
|
||||
sig.manifest = forged.manifest;
|
||||
sig.record_hashes = forged.record_hashes;
|
||||
storage::write_to_disk_signed(
|
||||
&path,
|
||||
&config,
|
||||
&cache,
|
||||
&sessions,
|
||||
&knowledge,
|
||||
&checkpoint,
|
||||
Some(&sig),
|
||||
)
|
||||
.unwrap();
|
||||
let r = verify(&path, &k);
|
||||
assert!(
|
||||
r.key_matches && !r.signature_valid && !r.is_valid(),
|
||||
"{r:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_unsigned_store_reports_unsigned() {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let mut m = HDF5Memory::create(MemoryConfig::new(dir.path().join("u.h5"), "a", DIM)).unwrap();
|
||||
m.save_batch(vec![entry(0, "hello")]).unwrap();
|
||||
drop(m);
|
||||
let r = HDF5Memory::verify(&dir.path().join("u.h5"), &VerifyingKey::from(&key(1))).unwrap();
|
||||
assert!(!r.signed && !r.is_valid());
|
||||
assert_eq!(r.record_count, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn nul_bytes_in_text_still_verify() {
|
||||
// Strings are stored null-padded; the hash must follow what a reopened
|
||||
// store actually holds, or an untouched store would fail to verify.
|
||||
let dir = TempDir::new().unwrap();
|
||||
let k = key(9);
|
||||
let mut m = HDF5Memory::create(MemoryConfig::new(dir.path().join("n.h5"), "a", DIM)).unwrap();
|
||||
m.set_signing_key(k.clone());
|
||||
m.save_batch(vec![
|
||||
entry(0, "inner\0nul"),
|
||||
entry(1, "trailing nul\0"),
|
||||
entry(2, "\0leading"),
|
||||
])
|
||||
.unwrap();
|
||||
drop(m);
|
||||
let r = verify(&dir.path().join("n.h5"), &k);
|
||||
assert!(r.is_valid(), "{r:?}");
|
||||
let m = HDF5Memory::open(&dir.path().join("n.h5")).unwrap();
|
||||
eprintln!(
|
||||
"reloaded: {:?}",
|
||||
(0..3).map(|i| m.get_chunk(i)).collect::<Vec<_>>()
|
||||
);
|
||||
}
|
||||
@@ -21,6 +21,7 @@
|
||||
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --ann-only --uniform
|
||||
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full
|
||||
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --options-study --full
|
||||
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --signing-study --full
|
||||
//! ```
|
||||
|
||||
use std::time::{Duration, Instant};
|
||||
@@ -488,6 +489,81 @@ fn bench_end_to_end(n: usize, json: &mut Vec<serde_json::Value>) {
|
||||
}));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Signing study: what does an Ed25519-signed checkpoint cost?
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// `--signing-study`: checkpoint time unsigned vs signed, `verify` time, and
|
||||
/// the file-size cost of the stored per-record hashes. Default store
|
||||
/// settings (float16, int8 index). Medians of five checkpoints / three
|
||||
/// verifies.
|
||||
fn signing_study(n: usize) {
|
||||
use clawhdf5_agent::signing::SigningKey;
|
||||
let data = make_dataset(n, 0x516 ^ n as u64);
|
||||
let mut rng = Rng(9);
|
||||
let entries: Vec<MemoryEntry> = data
|
||||
.vectors
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, v)| MemoryEntry {
|
||||
chunk: text_for(data.cluster_of[i], i, &mut rng),
|
||||
embedding: v.clone(),
|
||||
source_channel: "bench".into(),
|
||||
timestamp: i as f64,
|
||||
session_id: format!("s{}", i % 50),
|
||||
tags: format!("t{i}"),
|
||||
})
|
||||
.collect();
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("sign.h5");
|
||||
let mut mem = HDF5Memory::create(MemoryConfig::new(path.clone(), "bench", DIM)).unwrap();
|
||||
mem.save_batch(entries).unwrap();
|
||||
std::hint::black_box(mem.hybrid_search(&data.queries[0], "", 1.0, 0.0, K));
|
||||
|
||||
let median = |mut v: Vec<Duration>| {
|
||||
v.sort();
|
||||
v[v.len() / 2]
|
||||
};
|
||||
let checkpoint = |mem: &mut HDF5Memory| {
|
||||
median(
|
||||
(0..5)
|
||||
.map(|_| {
|
||||
let t = Instant::now();
|
||||
mem.flush_wal().unwrap();
|
||||
t.elapsed()
|
||||
})
|
||||
.collect(),
|
||||
)
|
||||
};
|
||||
let unsigned = checkpoint(&mut mem);
|
||||
let unsigned_bytes = std::fs::metadata(&path).unwrap().len();
|
||||
let key = SigningKey::from_bytes(&[7; 32]);
|
||||
mem.set_signing_key(key.clone());
|
||||
let signed = checkpoint(&mut mem);
|
||||
let signed_bytes = std::fs::metadata(&path).unwrap().len();
|
||||
drop(mem);
|
||||
let vk = key.verifying_key();
|
||||
let verify = median(
|
||||
(0..3)
|
||||
.map(|_| {
|
||||
let t = Instant::now();
|
||||
let r = HDF5Memory::verify(&path, &vk).unwrap();
|
||||
let d = t.elapsed();
|
||||
assert!(r.is_valid());
|
||||
d
|
||||
})
|
||||
.collect(),
|
||||
);
|
||||
println!(
|
||||
"| {n} | {:.1} | {:.1} | {:+.1} | {:.1} | {:+.2} |",
|
||||
millis(unsigned),
|
||||
millis(signed),
|
||||
millis(signed) - millis(unsigned),
|
||||
millis(verify),
|
||||
(signed_bytes as f64 - unsigned_bytes as f64) / (1024.0 * 1024.0),
|
||||
);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Search options study: source filters, re-ranking, confidence rejection
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -960,6 +1036,21 @@ fn main() {
|
||||
}
|
||||
return;
|
||||
}
|
||||
if args.iter().any(|a| a == "--signing-study") {
|
||||
println!("## Signed checkpoints ({DIM}-dim, float16, int8 index)\n");
|
||||
println!(
|
||||
"| N | checkpoint ms, unsigned | checkpoint ms, signed | signing adds ms | verify ms | file MiB added |"
|
||||
);
|
||||
println!("|---:|---:|---:|---:|---:|---:|");
|
||||
for &n in if full {
|
||||
&[1_000, 10_000, 100_000][..]
|
||||
} else {
|
||||
&[1_000, 10_000][..]
|
||||
} {
|
||||
signing_study(n);
|
||||
}
|
||||
return;
|
||||
}
|
||||
if args.iter().any(|a| a == "--options-study") {
|
||||
println!("## Search options ({DIM}-dim, k = {K}, Hebbian boost off)\n");
|
||||
println!("| N | options | filtered recall@10 | p50 ms | p99 ms |");
|
||||
|
||||
+126
-16
@@ -1,15 +1,22 @@
|
||||
use std::path::PathBuf;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use clap::{Parser, Subcommand};
|
||||
use clawhdf5_agent::signing::{self, SigningKey, VerifyingKey};
|
||||
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
|
||||
|
||||
/// ClawhDF5 — HDF5-backed cognitive memory for AI agents
|
||||
#[derive(Parser)]
|
||||
#[command(name = "clawhdf5", version, about)]
|
||||
struct Cli {
|
||||
/// Path to the .h5 memory file
|
||||
/// Path to the .h5 memory file (not needed for `keygen`)
|
||||
#[arg(short, long, env = "CLAWHDF5_PATH")]
|
||||
path: PathBuf,
|
||||
path: Option<PathBuf>,
|
||||
|
||||
/// File holding an Ed25519 signing key (64 hex characters, from
|
||||
/// `keygen`). Every checkpoint this command makes is then signed; a
|
||||
/// signed store refuses to checkpoint without it.
|
||||
#[arg(long, env = "CLAWHDF5_SIGNING_KEY", global = true)]
|
||||
signing_key: Option<PathBuf>,
|
||||
|
||||
#[command(subcommand)]
|
||||
command: Commands,
|
||||
@@ -91,6 +98,38 @@ enum Commands {
|
||||
/// Destination path
|
||||
dest: PathBuf,
|
||||
},
|
||||
/// Generate an Ed25519 signing key for signed checkpoints
|
||||
Keygen {
|
||||
/// Where to write the secret key (created new, owner-only on Unix)
|
||||
#[arg(long)]
|
||||
out: PathBuf,
|
||||
},
|
||||
/// Verify a signed store against a public key; exit status 2 if not valid
|
||||
Verify {
|
||||
/// The trusted public key: 64 hex characters, or a file holding them
|
||||
#[arg(long)]
|
||||
public_key: String,
|
||||
},
|
||||
}
|
||||
|
||||
fn read_signing_key(path: &Path) -> Result<SigningKey, Box<dyn std::error::Error>> {
|
||||
let text = std::fs::read_to_string(path)
|
||||
.map_err(|e| format!("cannot read signing key {}: {e}", path.display()))?;
|
||||
let bytes = signing::from_hex::<32>(&text)
|
||||
.ok_or_else(|| format!("{} is not a 64-hex-character key", path.display()))?;
|
||||
Ok(SigningKey::from_bytes(&bytes))
|
||||
}
|
||||
|
||||
/// Open for writing, with the signing key applied if one was given.
|
||||
fn open_writable(
|
||||
path: &Path,
|
||||
key: &Option<SigningKey>,
|
||||
) -> Result<HDF5Memory, Box<dyn std::error::Error>> {
|
||||
let mut mem = HDF5Memory::open(path)?;
|
||||
if let Some(k) = key {
|
||||
mem.set_signing_key(k.clone());
|
||||
}
|
||||
Ok(mem)
|
||||
}
|
||||
|
||||
fn main() {
|
||||
@@ -103,6 +142,37 @@ fn main() {
|
||||
}
|
||||
|
||||
fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
if let Commands::Keygen { out } = &cli.command {
|
||||
let key = signing::generate_key();
|
||||
let mut opts = std::fs::OpenOptions::new();
|
||||
opts.write(true).create_new(true);
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::OpenOptionsExt;
|
||||
opts.mode(0o600);
|
||||
}
|
||||
use std::io::Write;
|
||||
let mut f = opts
|
||||
.open(out)
|
||||
.map_err(|e| format!("cannot create {}: {e}", out.display()))?;
|
||||
writeln!(f, "{}", signing::to_hex(&key.to_bytes()))?;
|
||||
let j = serde_json::json!({
|
||||
"status": "generated",
|
||||
"secret_key_file": out.display().to_string(),
|
||||
"public_key": signing::to_hex(&key.verifying_key().to_bytes()),
|
||||
});
|
||||
println!("{}", serde_json::to_string_pretty(&j)?);
|
||||
return Ok(());
|
||||
}
|
||||
let path = cli
|
||||
.path
|
||||
.clone()
|
||||
.ok_or("--path (or CLAWHDF5_PATH) is required")?;
|
||||
let key = cli
|
||||
.signing_key
|
||||
.as_deref()
|
||||
.map(read_signing_key)
|
||||
.transpose()?;
|
||||
match cli.command {
|
||||
Commands::Create {
|
||||
agent_id,
|
||||
@@ -113,7 +183,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
f32,
|
||||
float16: _,
|
||||
} => {
|
||||
let mut config = MemoryConfig::new(cli.path.clone(), &agent_id, dim);
|
||||
let mut config = MemoryConfig::new(path.clone(), &agent_id, dim);
|
||||
config.wal_enabled = wal;
|
||||
// As with --f32-index: only ever switch the library default off.
|
||||
if f32 {
|
||||
@@ -127,15 +197,21 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
config.quantized_index = false;
|
||||
}
|
||||
let config_quantized = config.quantized_index;
|
||||
let mem = HDF5Memory::create(config)?;
|
||||
let mut mem = HDF5Memory::create(config)?;
|
||||
// Sign straight away, so the store is never on disk unsigned.
|
||||
if let Some(k) = &key {
|
||||
mem.set_signing_key(k.clone());
|
||||
mem.flush_wal()?;
|
||||
}
|
||||
let j = serde_json::json!({
|
||||
"status": "created",
|
||||
"path": cli.path.display().to_string(),
|
||||
"path": path.display().to_string(),
|
||||
"agent_id": agent_id,
|
||||
"embedding_dim": dim,
|
||||
"wal_enabled": wal,
|
||||
"quantized_index": config_quantized,
|
||||
"float16": config_float16,
|
||||
"signed": mem.is_signed(),
|
||||
"count": mem.count(),
|
||||
});
|
||||
println!("{}", serde_json::to_string_pretty(&j)?);
|
||||
@@ -152,7 +228,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
}
|
||||
};
|
||||
let entry: MemoryEntry = serde_json::from_str(&input)?;
|
||||
let mut mem = HDF5Memory::open(&cli.path)?;
|
||||
let mut mem = open_writable(&path, &key)?;
|
||||
let idx = mem.save(entry)?;
|
||||
let j = serde_json::json!({ "status": "saved", "index": idx, "count": mem.count() });
|
||||
println!("{}", serde_json::to_string(&j)?);
|
||||
@@ -166,7 +242,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
keyword_weight,
|
||||
} => {
|
||||
let emb: Vec<f32> = serde_json::from_str(&embedding)?;
|
||||
let mut mem = HDF5Memory::open(&cli.path)?;
|
||||
let mut mem = open_writable(&path, &key)?;
|
||||
let results = mem.hybrid_search(&emb, &query, vector_weight, keyword_weight, top_k);
|
||||
let j: Vec<serde_json::Value> = results
|
||||
.iter()
|
||||
@@ -184,7 +260,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
}
|
||||
|
||||
Commands::Recall { index } => {
|
||||
let mem = HDF5Memory::open_read_only(&cli.path)?;
|
||||
let mem = HDF5Memory::open_read_only(&path)?;
|
||||
match mem.get_chunk(index) {
|
||||
Some(content) => {
|
||||
let j = serde_json::json!({ "index": index, "chunk": content });
|
||||
@@ -198,22 +274,23 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
}
|
||||
|
||||
Commands::Stats => {
|
||||
let mem = HDF5Memory::open_read_only(&cli.path)?;
|
||||
let mem = HDF5Memory::open_read_only(&path)?;
|
||||
let cfg = mem.config();
|
||||
let j = serde_json::json!({
|
||||
"path": cli.path.display().to_string(),
|
||||
"path": path.display().to_string(),
|
||||
"agent_id": cfg.agent_id,
|
||||
"embedding_dim": cfg.embedding_dim,
|
||||
"count": mem.count(),
|
||||
"active": mem.count_active(),
|
||||
"wal_enabled": cfg.wal_enabled,
|
||||
"wal_pending": mem.wal_pending_count(),
|
||||
"signed": mem.is_signed(),
|
||||
});
|
||||
println!("{}", serde_json::to_string_pretty(&j)?);
|
||||
}
|
||||
|
||||
Commands::FlushWal => {
|
||||
let mut mem = HDF5Memory::open(&cli.path)?;
|
||||
let mut mem = open_writable(&path, &key)?;
|
||||
let before = mem.wal_pending_count();
|
||||
mem.flush_wal()?;
|
||||
let j = serde_json::json!({
|
||||
@@ -225,7 +302,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
}
|
||||
|
||||
Commands::AgentsMd { output } => {
|
||||
let mem = HDF5Memory::open_read_only(&cli.path)?;
|
||||
let mem = HDF5Memory::open_read_only(&path)?;
|
||||
let md = mem.generate_agents_md();
|
||||
match output {
|
||||
Some(p) => {
|
||||
@@ -237,7 +314,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
}
|
||||
|
||||
Commands::Export => {
|
||||
let mem = HDF5Memory::open_read_only(&cli.path)?;
|
||||
let mem = HDF5Memory::open_read_only(&path)?;
|
||||
for i in 0..mem.count() {
|
||||
if let Some(chunk) = mem.get_chunk(i) {
|
||||
let j = serde_json::json!({ "index": i, "chunk": chunk });
|
||||
@@ -246,11 +323,44 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
}
|
||||
}
|
||||
|
||||
Commands::Keygen { .. } => unreachable!("handled before opening a store"),
|
||||
|
||||
Commands::Verify { public_key } => {
|
||||
let text = if Path::new(&public_key).is_file() {
|
||||
std::fs::read_to_string(&public_key)?
|
||||
} else {
|
||||
public_key
|
||||
};
|
||||
let bytes = signing::from_hex::<32>(&text)
|
||||
.ok_or("--public-key must be 64 hex characters or a file holding them")?;
|
||||
let trusted = VerifyingKey::from_bytes(&bytes)?;
|
||||
let r = HDF5Memory::verify(&path, &trusted)?;
|
||||
let j = serde_json::json!({
|
||||
"valid": r.is_valid(),
|
||||
"signed": r.signed,
|
||||
"key_matches": r.key_matches,
|
||||
"signature_valid": r.signature_valid,
|
||||
"records_match": r.records_match,
|
||||
"settings_match": r.settings_match,
|
||||
"sessions_match": r.sessions_match,
|
||||
"graph_match": r.graph_match,
|
||||
"changed_records": r.changed_records,
|
||||
"record_count": r.record_count,
|
||||
"signed_record_count": r.signed_record_count,
|
||||
"signed_by": r.public_key.map(|k| signing::to_hex(&k)),
|
||||
"wal_entries_unsigned": r.wal_entries_unsigned,
|
||||
});
|
||||
println!("{}", serde_json::to_string_pretty(&j)?);
|
||||
if !r.is_valid() {
|
||||
std::process::exit(2);
|
||||
}
|
||||
}
|
||||
|
||||
Commands::Snapshot { dest } => {
|
||||
let _result = clawhdf5_agent::storage::snapshot_file(&cli.path, &dest)?;
|
||||
let _result = clawhdf5_agent::storage::snapshot_file(&path, &dest)?;
|
||||
let j = serde_json::json!({
|
||||
"status": "snapshot_created",
|
||||
"source": cli.path.display().to_string(),
|
||||
"source": path.display().to_string(),
|
||||
"dest": dest.display().to_string(),
|
||||
});
|
||||
println!("{}", serde_json::to_string(&j)?);
|
||||
|
||||
+19
-61
@@ -11,7 +11,7 @@ ClawhDF5 serves three audiences with different entry points:
|
||||
| You Are | You Want | Start Here |
|
||||
|---------|----------|------------|
|
||||
| **AI agent developer** | Persistent memory for your agent | [Agent Memory (Rust)](#1-agent-memory-rust-library) |
|
||||
| **OpenClaw user** | Better memory for your OpenClaw agent | [OpenClaw Integration](#2-openclaw-integration) |
|
||||
| **OpenClaw user** | clawhdf5 is not an OpenClaw memory plugin | [Status](openclaw.md) |
|
||||
| **Data scientist** | Read/write HDF5 files in Rust | [HDF5 File I/O](#3-hdf5-file-io) |
|
||||
| **CLI user** | Inspect and manage agent memories | [CLI Tool](#4-cli-tool) |
|
||||
| **Python user** | Use clawhdf5 from Python | [Python Bindings](#5-python-bindings) |
|
||||
@@ -197,80 +197,38 @@ if let Some(alert) = detector.check_rate_anomaly() {
|
||||
|
||||
---
|
||||
|
||||
## 2. OpenClaw Integration
|
||||
## 2. Markdown Memory (and OpenClaw)
|
||||
|
||||
ClawhDF5 can serve as the memory backend for [OpenClaw](https://docs.openclaw.ai) agents, replacing the default Markdown + sqlite-vec approach.
|
||||
**clawhdf5 is not an OpenClaw memory backend.** Earlier versions of this guide
|
||||
described one; it never worked — see [openclaw.md](openclaw.md) for what
|
||||
happened and what a real plugin would need.
|
||||
|
||||
### How It Works
|
||||
|
||||
```
|
||||
OpenClaw Agent
|
||||
│
|
||||
├── memory_search("user preferences")
|
||||
│ │
|
||||
│ └── ClawhdfBackend
|
||||
│ ├── Vector search (cosine)
|
||||
│ ├── BM25 keyword search
|
||||
│ ├── Reciprocal Rank Fusion
|
||||
│ ├── Multi-factor re-ranking
|
||||
│ └── Low-confidence rejection
|
||||
│
|
||||
└── agent_memory.h5 (single file, portable)
|
||||
```
|
||||
|
||||
### Migration from Markdown
|
||||
What does exist is `ClawhdfBackend`, a library API that ingests Markdown files
|
||||
by section and searches them with the full pipeline (hybrid retrieval,
|
||||
re-ranking, confidence rejection):
|
||||
|
||||
```rust
|
||||
use clawhdf5_agent::openclaw::*;
|
||||
use std::path::Path;
|
||||
|
||||
// Create a new HDF5 backend
|
||||
let mut backend = ClawhdfBackend::create("memory.h5", "my-agent", 384)?;
|
||||
let mut backend = ClawhdfBackend::create(Path::new("memory.h5"), 384)?;
|
||||
|
||||
// Import your existing MEMORY.md
|
||||
let md = std::fs::read_to_string("~/.openclaw/workspace/MEMORY.md")?;
|
||||
// Each heading becomes a record, stored under "MEMORY.md::<heading>".
|
||||
let md = std::fs::read_to_string("MEMORY.md")?;
|
||||
let count = backend.ingest_markdown("MEMORY.md", &md)?;
|
||||
println!("Imported {} sections", count);
|
||||
println!("Imported {count} sections");
|
||||
|
||||
// Import daily logs
|
||||
for entry in std::fs::read_dir("~/.openclaw/workspace/memory/")? {
|
||||
let path = entry?.path();
|
||||
if path.extension().map(|e| e == "md").unwrap_or(false) {
|
||||
let content = std::fs::read_to_string(&path)?;
|
||||
let name = path.file_name().unwrap().to_string_lossy();
|
||||
backend.ingest_markdown(&name, &content)?;
|
||||
}
|
||||
}
|
||||
|
||||
// Search using the full pipeline
|
||||
let results = backend.search("what are user preferences", &query_embedding, 5);
|
||||
for r in &results {
|
||||
println!("[{:.3}] {} (from {})", r.score, r.text, r.path);
|
||||
}
|
||||
|
||||
// Export back to Markdown (lossless roundtrip)
|
||||
let exported = backend.export_markdown("MEMORY.md")?;
|
||||
```
|
||||
|
||||
### What You Get Over sqlite-vec
|
||||
|
||||
| Feature | sqlite-vec | ClawhDF5 |
|
||||
|---------|-----------|----------|
|
||||
| Vector search | ✅ | ✅ (8× faster at 100K) |
|
||||
| Keyword search | ❌ | ✅ BM25 |
|
||||
| Hybrid fusion | ❌ | ✅ RRF |
|
||||
| Re-ranking | ❌ | ✅ Multi-factor |
|
||||
| Confidence rejection | ❌ | ✅ |
|
||||
| Knowledge graph | ❌ | ✅ |
|
||||
| Memory consolidation | ❌ | ✅ |
|
||||
| Temporal queries | ❌ | ✅ (716ns) |
|
||||
| Anomaly detection | ❌ | ✅ |
|
||||
| Provenance tracking | ❌ | ✅ |
|
||||
| Multi-modal | ❌ | ✅ |
|
||||
| Single portable file | ❌ (SQLite + MD files) | ✅ |
|
||||
|
||||
### Future: Native OpenClaw Plugin
|
||||
|
||||
The Phase 2 roadmap includes a native OpenClaw plugin (`memory.backend = "clawhdf5"`) that transparently replaces sqlite-vec. Until then, the Rust library can be wrapped via NAPI or used from the CLI.
|
||||
Limits to know: sections ingested this way carry no embedding (search over them
|
||||
is keyword-only unless you save records with vectors via `save_entry`);
|
||||
ingesting the same file again adds the sections again rather than replacing
|
||||
them; and `export_markdown` rewrites every heading as `##`, so it is not a
|
||||
lossless round trip.
|
||||
|
||||
---
|
||||
|
||||
@@ -547,7 +505,7 @@ let final_results = confidence::reject_low_confidence(
|
||||
|
||||
**Why not a vector database?** Pinecone, Qdrant, Weaviate — they're cloud services or heavy servers. Agent memory should be local, portable, and zero-dependency. An agent's memories should travel with it.
|
||||
|
||||
**Why not Markdown?** OpenClaw uses Markdown today and it works for simple cases. But it doesn't scale: no vector search, no knowledge graph, no structured retrieval. ClawhDF5 can import/export Markdown while providing everything Markdown can't.
|
||||
**Why not Markdown?** Plain Markdown files work for simple cases. But it doesn't scale: no vector search, no knowledge graph, no structured retrieval. ClawhDF5 can import/export Markdown while providing everything Markdown can't.
|
||||
|
||||
**Why HDF5 specifically?**
|
||||
- Native N-dimensional array storage (perfect for embeddings)
|
||||
|
||||
+3
-30
@@ -42,37 +42,10 @@ conversation → embedding → save to agent.h5
|
||||
|
||||
---
|
||||
|
||||
## 2. OpenClaw Memory Upgrade
|
||||
## 2. OpenClaw
|
||||
|
||||
**Scenario:** You run OpenClaw and the default Markdown + sqlite-vec memory works OK for simple recall but falls short on complex queries like "what did we decide about the deployment architecture last Tuesday?" or "who's responsible for the billing system?"
|
||||
|
||||
**Problem:** Markdown files have no semantic structure. sqlite-vec does flat vector search — no keyword fusion, no re-ranking, no temporal reasoning, no knowledge graph.
|
||||
|
||||
**ClawhDF5 solution:**
|
||||
|
||||
```bash
|
||||
# Migrate existing memories
|
||||
clawhdf5 --path memory.h5 create --agent-id openclaw --dim 384
|
||||
|
||||
# Import your MEMORY.md and daily logs
|
||||
# (programmatically via ClawhdfBackend::ingest_markdown)
|
||||
```
|
||||
|
||||
Then in your OpenClaw config (future):
|
||||
```json
|
||||
{
|
||||
"memory": {
|
||||
"backend": "clawhdf5",
|
||||
"path": "~/.openclaw/agents/main/memory.h5"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**What changes:**
|
||||
- "What did we discuss last Tuesday?" → temporal index finds the session, returns memories from that time range
|
||||
- "Who owns the billing system?" → knowledge graph traversal: billing_system → owned_by → Alice
|
||||
- "Preferences about deployment" → hybrid search (vector + BM25) finds relevant memories even with different wording
|
||||
- Bad search results get filtered out by confidence rejection instead of confusing the agent
|
||||
Not supported: clawhdf5 is not an OpenClaw memory plugin, and the config this
|
||||
section used to show was never valid. See [openclaw.md](openclaw.md).
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -279,3 +279,26 @@ the same agent-store interop test.
|
||||
|
||||
**Fix:** an empty contiguous dataset gets the undefined address (all `0xff`),
|
||||
which is what libhdf5 itself writes.
|
||||
|
||||
## The Node.js package (`packages/clawhdf5-node`) does not work
|
||||
|
||||
**Status:** open (found 2026-09-25). Unpublished; not built or tested in CI.
|
||||
|
||||
The TypeScript wrapper over `crates/clawhdf5-napi` has never run successfully:
|
||||
|
||||
- napi-rs converts `#[napi(object)]` fields to camelCase, but the wrapper reads
|
||||
snake_case (`r.line_range`, `s.total_records`, `s.working_count`, …), so
|
||||
every stats and consolidation field comes back `undefined`
|
||||
(`src/index.ts:76-120`).
|
||||
- It loads `../clawhdf5.node`, but `napi build --platform` produces
|
||||
`clawhdf5.<triple>.node`; `main` points at `index.js` while `tsc` writes to
|
||||
`dist/`; `napi prepublish` expects per-platform packages that are not
|
||||
defined.
|
||||
- `save`/`saveBatch` exist in the napi layer but not in the wrapper, so a
|
||||
TypeScript caller cannot store an embedding at all.
|
||||
- The WAL for `agent.brain` is `agent.h5.wal` (the store uses
|
||||
`with_extension("h5.wal")`), not `agent.brain.wal` as the old docs and the
|
||||
test cleanup assume.
|
||||
|
||||
It was written for an OpenClaw integration that is not being pursued (see
|
||||
`docs/openclaw.md`). Fix and add CI, or remove it, before anyone depends on it.
|
||||
|
||||
@@ -1,213 +0,0 @@
|
||||
# Migration Guide: OpenClaw sqlite-vec → clawhdf5
|
||||
|
||||
This guide walks through migrating an OpenClaw agent from its default
|
||||
sqlite-vec + Markdown file memory to the `clawhdf5` HDF5 backend.
|
||||
|
||||
---
|
||||
|
||||
## Why migrate?
|
||||
|
||||
| Feature | sqlite-vec + Markdown | clawhdf5 |
|
||||
|---------|----------------------|----------|
|
||||
| Storage format | SQLite WAL + flat .md files | Single HDF5 binary file |
|
||||
| Vector search | sqlite-vec (SQLite extension) | Pure-Rust SIMD (clawhdf5-accel) |
|
||||
| Full-text search | External (FTS5 or plain string match) | Built-in BM25 |
|
||||
| Hybrid search | Manual combination | Automatic RRF blend |
|
||||
| Memory tiers | Flat | Working → Episodic → Semantic |
|
||||
| Hebbian decay | Not built-in | Automatic activation weighting |
|
||||
| Portability | SQLite binary required | Zero native deps (all Rust) |
|
||||
| Crash recovery | SQLite WAL | clawhdf5 WAL |
|
||||
| Compaction | Manual | Auto-threshold + session-end |
|
||||
| Embedding dim change | New DB required | New file required (same) |
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Install `@redclaw/clawhdf5`
|
||||
|
||||
```bash
|
||||
npm install @redclaw/clawhdf5
|
||||
```
|
||||
|
||||
Or, if building from the monorepo source:
|
||||
|
||||
```bash
|
||||
npm install -g @napi-rs/cli
|
||||
cd packages/clawhdf5-node
|
||||
npm install
|
||||
npm run build
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Update your OpenClaw config
|
||||
|
||||
Change `backend` from `"sqlite-vec"` (or `"markdown"`) to `"clawhdf5"`:
|
||||
|
||||
```json
|
||||
{
|
||||
"memory": {
|
||||
"backend": "clawhdf5",
|
||||
"clawhdf5": {
|
||||
"path": "./agent.brain",
|
||||
"embeddingDim": 768
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
See [openclaw-config.md](openclaw-config.md) for the full schema.
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Run the one-time migration
|
||||
|
||||
clawhdf5 ships a migration helper that reads your existing Markdown memory
|
||||
files and ingests them via `ingestMarkdown()`.
|
||||
|
||||
### Automated migration script
|
||||
|
||||
```typescript
|
||||
import { ClawhdfMemory } from '@redclaw/clawhdf5';
|
||||
import { readFileSync, readdirSync, statSync } from 'fs';
|
||||
import { join, relative } from 'path';
|
||||
|
||||
async function migrate(
|
||||
memoryDir: string,
|
||||
brainPath: string,
|
||||
embeddingDim: number = 768,
|
||||
): Promise<void> {
|
||||
const mem = ClawhdfMemory.create(brainPath, embeddingDim);
|
||||
|
||||
// Walk all .md files under memoryDir
|
||||
function walk(dir: string): string[] {
|
||||
return readdirSync(dir).flatMap((entry) => {
|
||||
const full = join(dir, entry);
|
||||
return statSync(full).isDirectory() ? walk(full) : [full];
|
||||
});
|
||||
}
|
||||
|
||||
const files = walk(memoryDir).filter((f) => f.endsWith('.md'));
|
||||
let totalSections = 0;
|
||||
|
||||
for (const file of files) {
|
||||
const content = readFileSync(file, 'utf8');
|
||||
const relPath = relative(process.cwd(), file);
|
||||
const count = mem.ingestMarkdown(relPath, content);
|
||||
console.log(` ${relPath}: ${count} sections`);
|
||||
totalSections += count;
|
||||
}
|
||||
|
||||
// Force WAL merge after bulk import
|
||||
mem.flushWal();
|
||||
|
||||
console.log(`\nMigration complete: ${files.length} files, ${totalSections} sections`);
|
||||
const s = mem.stats();
|
||||
console.log(` Total records: ${s.totalRecords}`);
|
||||
console.log(` File size: ${(s.fileSizeBytes / 1024).toFixed(1)} KB`);
|
||||
}
|
||||
|
||||
// Usage
|
||||
migrate('./memory', './agent.brain', 768).catch(console.error);
|
||||
```
|
||||
|
||||
### What the migration does
|
||||
|
||||
1. Walks all `.md` files under your memory directory.
|
||||
2. Parses each file into sections using the same `MarkdownParser` used by
|
||||
OpenClaw (splits on ATX headings `#`, `##`, `###`, …).
|
||||
3. Stores each section as a separate record in the HDF5 file with the file
|
||||
path as the `source_channel` (e.g. `memory/user.md::Goals`).
|
||||
4. Flushes the WAL to merge everything into the `.brain` file.
|
||||
|
||||
After migration, **the original `.md` files are not modified or deleted**.
|
||||
You can keep them as a backup or remove them once you have verified the
|
||||
migrated data.
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Verify
|
||||
|
||||
```typescript
|
||||
import { ClawhdfMemory } from '@redclaw/clawhdf5';
|
||||
|
||||
const mem = ClawhdfMemory.open('./agent.brain');
|
||||
const s = mem.stats();
|
||||
console.log('Records after migration:', s.totalRecords);
|
||||
|
||||
// Spot-check: retrieve a known path
|
||||
const userMd = mem.get('memory/MEMORY.md');
|
||||
console.log(userMd?.slice(0, 200));
|
||||
|
||||
// Round-trip a file back to Markdown
|
||||
const exported = mem.exportMarkdown('memory/MEMORY.md');
|
||||
console.log(exported.slice(0, 500));
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Update agent code
|
||||
|
||||
If your agent code reads memory files directly from disk, update it to use
|
||||
the clawhdf5 API instead:
|
||||
|
||||
**Before (sqlite-vec + file reads):**
|
||||
```typescript
|
||||
const content = readFileSync('memory/user.md', 'utf8');
|
||||
const sections = parseMarkdown(content);
|
||||
const results = await vectorSearch(query, sections, k);
|
||||
```
|
||||
|
||||
**After (clawhdf5):**
|
||||
```typescript
|
||||
import { ClawhdfMemory } from '@redclaw/clawhdf5';
|
||||
|
||||
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
|
||||
const embedding = await embed(query); // your embedding function
|
||||
const results = mem.search(query, new Float32Array(embedding), k);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Session lifecycle hooks
|
||||
|
||||
Add compaction at session end for best long-term memory health:
|
||||
|
||||
```typescript
|
||||
// At the start of your agent process
|
||||
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
|
||||
|
||||
// ... agent runs ...
|
||||
|
||||
// At the end of each session
|
||||
mem.tickSession(); // decay activation weights
|
||||
const stats = mem.runConsolidation(Date.now() / 1000); // promote memories
|
||||
console.log('[memory] consolidation:', stats);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
If you need to roll back to sqlite-vec:
|
||||
|
||||
1. Change `memory.backend` back to `"sqlite-vec"` in your config.
|
||||
2. The original `.md` files are unchanged (if you kept them).
|
||||
3. Delete `agent.brain` (and `agent.brain.wal` if present).
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### `Error: no records found for path: memory/user.md`
|
||||
The path passed to `get()` or `exportMarkdown()` must exactly match the
|
||||
relative path used during `ingestMarkdown()`. Check for leading `./`
|
||||
differences.
|
||||
|
||||
### Memory is empty after reopening
|
||||
Make sure `flushWal()` was called after bulk writes. Without it, entries
|
||||
remain in the WAL and may be lost if the process exits abnormally.
|
||||
|
||||
### Embedding dimension mismatch
|
||||
The `embeddingDim` passed to `create()` cannot be changed after the file is
|
||||
created. If you switch embedding models, create a new `.brain` file and
|
||||
re-run the migration script.
|
||||
@@ -1,144 +0,0 @@
|
||||
# OpenClaw × clawhdf5 Configuration Reference
|
||||
|
||||
This document describes the full configuration schema for integrating
|
||||
`clawhdf5` as the memory backend in an OpenClaw agent gateway.
|
||||
|
||||
---
|
||||
|
||||
## Minimal example
|
||||
|
||||
```json
|
||||
{
|
||||
"memory": {
|
||||
"backend": "clawhdf5",
|
||||
"clawhdf5": {
|
||||
"path": "./agent.brain",
|
||||
"embeddingDim": 768
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Full schema
|
||||
|
||||
```json
|
||||
{
|
||||
"memory": {
|
||||
"backend": "clawhdf5",
|
||||
"clawhdf5": {
|
||||
"path": "./agent.brain",
|
||||
"embeddingDim": 768,
|
||||
"walEnabled": true,
|
||||
"walMaxEntries": 500,
|
||||
"consolidation": {
|
||||
"workingCapacity": 100,
|
||||
"episodicCapacity": 10000,
|
||||
"episodicHalfLifeDays": 7,
|
||||
"semanticHalfLifeDays": 30,
|
||||
"promotionThreshold": 0.6,
|
||||
"semanticAccessThreshold": 10
|
||||
},
|
||||
"compaction": {
|
||||
"autoCompactThreshold": 0.3,
|
||||
"tickOnSessionEnd": true,
|
||||
"consolidateOnCompaction": true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Field reference
|
||||
|
||||
### Top level
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `memory.backend` | `string` | `"clawhdf5"` | Must be `"clawhdf5"` to activate this backend |
|
||||
|
||||
### `clawhdf5`
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `path` | `string` | `"./agent.brain"` | Filesystem path for the `.brain` (HDF5) file. Relative to the OpenClaw working directory. |
|
||||
| `embeddingDim` | `number` | `768` | Dimension of the embedding vectors. Must match the embedder model. Common values: `384` (MiniLM), `768` (nomic-embed-text, BGE-base), `1536` (OpenAI text-embedding-3-small). |
|
||||
| `walEnabled` | `boolean` | `true` | Enable the Write-Ahead Log for crash recovery. Disable only on read-only stores or when crash safety is not required. |
|
||||
| `walMaxEntries` | `number` | `500` | Number of WAL entries to accumulate before an automatic merge to the .h5 file. Lower values = more frequent flushes (safer, slightly slower). |
|
||||
|
||||
### `clawhdf5.consolidation`
|
||||
|
||||
Controls the hippocampal three-tier memory engine (Working → Episodic →
|
||||
Semantic).
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `workingCapacity` | `number` | `100` | Maximum records in the Working tier before lowest-decay entries are evicted. |
|
||||
| `episodicCapacity` | `number` | `10000` | Maximum records in the Episodic tier. |
|
||||
| `episodicHalfLifeDays` | `number` | `7` | Half-life (in days) for exponential decay of Episodic records. Records not accessed within roughly one half-life drop in importance. |
|
||||
| `semanticHalfLifeDays` | `number` | `30` | Half-life for Semantic records. Longer than Episodic — semantic knowledge decays slowly. |
|
||||
| `promotionThreshold` | `number` | `0.6` | Importance score (0–1) above which a Working record is promoted to the Episodic tier. Higher = more selective. |
|
||||
| `semanticAccessThreshold` | `number` | `10` | Minimum access count for an Episodic record to be promoted to Semantic. |
|
||||
|
||||
### `clawhdf5.compaction`
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `autoCompactThreshold` | `number` | `0.3` | Fraction of tombstoned records (0–1) that triggers automatic compaction. `0.3` = compact when 30% of records are deleted. Set to `0` to disable auto-compact. |
|
||||
| `tickOnSessionEnd` | `boolean` | `true` | Run `tickSession()` (Hebbian decay) automatically when the agent session closes. |
|
||||
| `consolidateOnCompaction` | `boolean` | `true` | Run the hippocampal consolidation engine after each compaction cycle. |
|
||||
|
||||
---
|
||||
|
||||
## Embedder compatibility
|
||||
|
||||
The `embeddingDim` must remain constant for the lifetime of a `.brain` file.
|
||||
Mixing embedding models in the same file is not supported.
|
||||
|
||||
| Embedder | `embeddingDim` |
|
||||
|----------|---------------|
|
||||
| `all-MiniLM-L6-v2` | `384` |
|
||||
| `nomic-embed-text` | `768` |
|
||||
| `BGE-base-en-v1.5` | `768` |
|
||||
| `OpenAI text-embedding-3-small` | `1536` |
|
||||
| `OpenAI text-embedding-3-large` | `3072` |
|
||||
|
||||
---
|
||||
|
||||
## OpenClaw integration code
|
||||
|
||||
```typescript
|
||||
import { ClawhdfMemory } from '@redclaw/clawhdf5';
|
||||
|
||||
// Load config from your OpenClaw config file
|
||||
const cfg = loadConfig(); // your config loading logic
|
||||
|
||||
const mem = ClawhdfMemory.openOrCreate(
|
||||
cfg.memory.clawhdf5.path,
|
||||
cfg.memory.clawhdf5.embeddingDim ?? 768,
|
||||
);
|
||||
|
||||
// On session end
|
||||
if (cfg.memory.clawhdf5.compaction?.tickOnSessionEnd) {
|
||||
mem.tickSession();
|
||||
}
|
||||
if (cfg.memory.clawhdf5.compaction?.consolidateOnCompaction) {
|
||||
const stats = mem.runConsolidation(Date.now() / 1000);
|
||||
console.log('[clawhdf5] consolidation:', stats);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment variables
|
||||
|
||||
The following environment variables override config file values when set:
|
||||
|
||||
| Variable | Overrides |
|
||||
|----------|-----------|
|
||||
| `CLAWHDF5_PATH` | `clawhdf5.path` |
|
||||
| `CLAWHDF5_EMBEDDING_DIM` | `clawhdf5.embeddingDim` |
|
||||
| `CLAWHDF5_WAL_ENABLED` | `clawhdf5.walEnabled` (`"true"` / `"false"`) |
|
||||
@@ -1,337 +0,0 @@
|
||||
# OpenClaw × clawhdf5 Integration
|
||||
|
||||
clawhdf5 provides a drop-in HDF5-backed memory backend for the
|
||||
[OpenClaw](https://github.com/redclawsystems/openclaw) agent gateway.
|
||||
This document covers architecture, the full Node.js API reference, and code
|
||||
examples for common operations.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ OpenClaw (Node.js/TypeScript) │
|
||||
│ │
|
||||
│ ┌─────────────────┐ ┌──────────────────────────────┐ │
|
||||
│ │ Agent runtime │───▶│ @redclaw/clawhdf5 (Node.js) │ │
|
||||
│ └─────────────────┘ │ TypeScript wrapper │ │
|
||||
│ └──────────────┬─────────────┘ │
|
||||
│ │ napi-rs FFI │
|
||||
└────────────────────────────────────────┼────────────────────┘
|
||||
│
|
||||
┌────────────────────────────────────────▼────────────────────┐
|
||||
│ clawhdf5-napi (Rust, cdylib) │
|
||||
│ │
|
||||
│ ClawhdfMemory ──▶ ClawhdfBackend ──▶ HDF5Memory │
|
||||
│ MemoryBackend ├─ MemoryCache │
|
||||
│ trait impl ├─ WalFile │
|
||||
│ ├─ SessionCache │
|
||||
│ └─ KnowledgeCache │
|
||||
│ │
|
||||
│ ConsolidationEngine (hippocampal tiers) │
|
||||
│ Working (100) ──▶ Episodic (10k) ──▶ Semantic (∞) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
┌────────────────▼────────────┐
|
||||
│ agent.brain (HDF5 file) │
|
||||
│ agent.brain.wal (WAL log) │
|
||||
└─────────────────────────────┘
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **Single file**: everything lives in one `.brain` HDF5 file (+ WAL sidecar).
|
||||
- **In-memory cache**: the full embedding matrix and chunk list are loaded into RAM for fast search.
|
||||
- **Hybrid search**: vector similarity (70%) and BM25 full-text (30%) are blended with Reciprocal Rank Fusion (RRF), then re-ranked by Hebbian activation weight and temporal recency.
|
||||
- **Hippocampal tiers**: records are classified as Working, Episodic, or Semantic based on importance and access frequency. Tier promotion and eviction happen during `runConsolidation()`.
|
||||
- **WAL**: writes are journaled before hitting the .h5 file. On crash, the WAL is replayed at next open.
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
npm install @redclaw/clawhdf5
|
||||
```
|
||||
|
||||
See [packages/clawhdf5-node/README.md](../packages/clawhdf5-node/README.md)
|
||||
for build-from-source instructions.
|
||||
|
||||
---
|
||||
|
||||
## Node.js API reference
|
||||
|
||||
### `ClawhdfMemory` (class)
|
||||
|
||||
All instance methods are synchronous. The native Rust code is single-threaded
|
||||
on the Node.js side; do **not** share a `ClawhdfMemory` instance across Worker
|
||||
threads without external locking.
|
||||
|
||||
---
|
||||
|
||||
#### Static factory methods
|
||||
|
||||
##### `ClawhdfMemory.create(path: string, embeddingDim: number): ClawhdfMemory`
|
||||
|
||||
Create a new `.brain` file. Throws if the file already exists.
|
||||
|
||||
```typescript
|
||||
const mem = ClawhdfMemory.create('./agent.brain', 768);
|
||||
```
|
||||
|
||||
##### `ClawhdfMemory.open(path: string): ClawhdfMemory`
|
||||
|
||||
Open an existing file. Replays the WAL automatically.
|
||||
|
||||
```typescript
|
||||
const mem = ClawhdfMemory.open('./agent.brain');
|
||||
```
|
||||
|
||||
##### `ClawhdfMemory.openOrCreate(path: string, embeddingDim: number): ClawhdfMemory`
|
||||
|
||||
**Recommended entry point.** Opens if the file exists, otherwise creates it.
|
||||
|
||||
```typescript
|
||||
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `search(queryText, queryEmbedding, k): MemorySearchResult[]`
|
||||
|
||||
Hybrid BM25 + vector search.
|
||||
|
||||
```typescript
|
||||
const embedding = new Float32Array(await embed(query));
|
||||
const results = mem.search(query, embedding, 10);
|
||||
for (const r of results) {
|
||||
console.log(r.score.toFixed(3), r.path, r.text.slice(0, 80));
|
||||
}
|
||||
```
|
||||
|
||||
Pass an empty `Float32Array` to use BM25 only (no vector similarity).
|
||||
|
||||
**Parameters:**
|
||||
- `queryText: string` — used for BM25 term matching
|
||||
- `queryEmbedding: Float32Array` — dense vector of length `embeddingDim`
|
||||
- `k: number` — maximum results to return
|
||||
|
||||
**Returns:** `MemorySearchResult[]`
|
||||
|
||||
---
|
||||
|
||||
#### `get(path, fromLine?, numLines?): string | null`
|
||||
|
||||
Retrieve stored content by path.
|
||||
|
||||
```typescript
|
||||
const md = mem.get('memory/user.md'); // all content
|
||||
const lines = mem.get('memory/user.md', 5, 10); // lines 5–14
|
||||
const section = mem.get('memory/user.md::Goals'); // specific section
|
||||
```
|
||||
|
||||
Section sub-paths use the `::heading` suffix produced by `ingestMarkdown`.
|
||||
|
||||
---
|
||||
|
||||
#### `write(path, content): void`
|
||||
|
||||
Store raw content at `path`.
|
||||
|
||||
```typescript
|
||||
mem.write('memory/session.md', '# Session\n\nWorking on task X.');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `ingestMarkdown(path, content): number`
|
||||
|
||||
Parse `content` as Markdown, split on ATX headings, and store each section
|
||||
separately. Returns the number of sections ingested.
|
||||
|
||||
```typescript
|
||||
import { readFileSync } from 'fs';
|
||||
const md = readFileSync('./memory/MEMORY.md', 'utf8');
|
||||
const count = mem.ingestMarkdown('memory/MEMORY.md', md);
|
||||
console.log(`Ingested ${count} sections`);
|
||||
```
|
||||
|
||||
Sections are addressable as `memory/MEMORY.md::HeadingName`.
|
||||
|
||||
---
|
||||
|
||||
#### `exportMarkdown(path): string`
|
||||
|
||||
Reconstruct stored sections for `path` back into a Markdown string.
|
||||
|
||||
```typescript
|
||||
const md = mem.exportMarkdown('memory/MEMORY.md');
|
||||
writeFileSync('./memory/MEMORY.md', md);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `stats(): BackendStats`
|
||||
|
||||
Return aggregate statistics.
|
||||
|
||||
```typescript
|
||||
const s = mem.stats();
|
||||
console.log(`Records: ${s.totalRecords}, Size: ${s.fileSizeBytes} bytes`);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `compact(): number`
|
||||
|
||||
Remove tombstoned records from the store. Returns count removed.
|
||||
|
||||
---
|
||||
|
||||
#### `tickSession(): void`
|
||||
|
||||
Apply Hebbian decay to all activation weights. Call at session end.
|
||||
|
||||
---
|
||||
|
||||
#### `flushWal(): void`
|
||||
|
||||
Force a WAL merge: flush `.h5` and truncate the WAL log.
|
||||
|
||||
---
|
||||
|
||||
#### `runConsolidation(nowSecs: number): ConsolidationStats`
|
||||
|
||||
Run one full hippocampal consolidation cycle.
|
||||
|
||||
```typescript
|
||||
const stats = mem.runConsolidation(Date.now() / 1000);
|
||||
console.log(stats);
|
||||
// { workingCount: 42, episodicCount: 310, semanticCount: 5,
|
||||
// totalEvictions: 0, totalPromotions: 7 }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `walPendingCount(): number`
|
||||
|
||||
Number of pending WAL entries (0 if WAL is disabled).
|
||||
|
||||
---
|
||||
|
||||
### Type reference
|
||||
|
||||
```typescript
|
||||
interface MemorySearchResult {
|
||||
text: string;
|
||||
score: number; // 0–1, higher = more relevant
|
||||
path: string; // source file path
|
||||
lineRange?: [number, number];
|
||||
timestamp?: number; // Unix epoch seconds
|
||||
source: string;
|
||||
}
|
||||
|
||||
interface BackendStats {
|
||||
totalRecords: number;
|
||||
totalEmbeddings: number;
|
||||
fileSizeBytes: number;
|
||||
modalities: string[]; // e.g. ["text"]
|
||||
lastUpdated?: number; // Unix epoch seconds
|
||||
}
|
||||
|
||||
interface ConsolidationStats {
|
||||
workingCount: number;
|
||||
episodicCount: number;
|
||||
semanticCount: number;
|
||||
totalEvictions: number;
|
||||
totalPromotions: number;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common patterns
|
||||
|
||||
### Session lifecycle
|
||||
|
||||
```typescript
|
||||
import { ClawhdfMemory } from '@redclaw/clawhdf5';
|
||||
|
||||
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
|
||||
|
||||
// --- agent session runs ---
|
||||
|
||||
// On session end: decay + consolidate
|
||||
mem.tickSession();
|
||||
const consolidationStats = mem.runConsolidation(Date.now() / 1000);
|
||||
console.log('[memory] consolidation:', consolidationStats);
|
||||
```
|
||||
|
||||
### Ingest all memory files at startup
|
||||
|
||||
```typescript
|
||||
import { readdirSync, readFileSync, statSync } from 'fs';
|
||||
import { join, relative } from 'path';
|
||||
|
||||
function ingestDirectory(mem: ClawhdfMemory, dir: string): void {
|
||||
for (const entry of readdirSync(dir)) {
|
||||
const full = join(dir, entry);
|
||||
if (statSync(full).isDirectory()) {
|
||||
ingestDirectory(mem, full);
|
||||
} else if (entry.endsWith('.md')) {
|
||||
const content = readFileSync(full, 'utf8');
|
||||
const path = relative(process.cwd(), full);
|
||||
mem.ingestMarkdown(path, content);
|
||||
}
|
||||
}
|
||||
mem.flushWal();
|
||||
}
|
||||
|
||||
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
|
||||
ingestDirectory(mem, './memory');
|
||||
```
|
||||
|
||||
### Search with real embeddings
|
||||
|
||||
```typescript
|
||||
import OpenAI from 'openai';
|
||||
import { ClawhdfMemory } from '@redclaw/clawhdf5';
|
||||
|
||||
const ai = new OpenAI();
|
||||
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 1536);
|
||||
|
||||
async function searchMemory(query: string, k = 5) {
|
||||
const resp = await ai.embeddings.create({
|
||||
model: 'text-embedding-3-small',
|
||||
input: query,
|
||||
});
|
||||
const embedding = new Float32Array(resp.data[0].embedding);
|
||||
return mem.search(query, embedding, k);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error handling
|
||||
|
||||
All methods that can fail throw a `NapiError` (a standard JS `Error` subclass)
|
||||
with the Rust error message as `message`.
|
||||
|
||||
```typescript
|
||||
try {
|
||||
const md = mem.exportMarkdown('nonexistent.md');
|
||||
} catch (e) {
|
||||
console.error('Export failed:', (e as Error).message);
|
||||
// "no records found for path: nonexistent.md"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## See also
|
||||
|
||||
- [openclaw-config.md](openclaw-config.md) — Full configuration schema
|
||||
- [migration-guide.md](migration-guide.md) — Migrating from sqlite-vec
|
||||
- [packages/clawhdf5-node/README.md](../packages/clawhdf5-node/README.md) — Build instructions
|
||||
- [BENCHMARKS.md](../BENCHMARKS.md) — Performance results
|
||||
@@ -0,0 +1,73 @@
|
||||
# OpenClaw: not supported
|
||||
|
||||
**clawhdf5 does not currently work as an [OpenClaw](https://docs.openclaw.ai)
|
||||
memory backend, and never has.** Earlier versions of these docs described a
|
||||
"drop-in" backend enabled with `memory.backend = "clawhdf5"`. That
|
||||
configuration was never valid: from v2026.2 through v2026.7 OpenClaw's
|
||||
`memory.backend` accepted only `"builtin"` or `"qmd"` and rejected unknown
|
||||
keys, and since v2026.8.1 ("OpenClaw 2.0") the key no longer exists. A Gateway
|
||||
given that config refuses to start. No plugin was ever built or tested against
|
||||
OpenClaw, and the `@redclaw/clawhdf5` npm package was never published.
|
||||
|
||||
As of 2026-09-25 we are not pursuing an OpenClaw plugin; the maintained
|
||||
integration target is ZeroClaw. This page records what a plugin would need,
|
||||
for when that changes.
|
||||
|
||||
## What OpenClaw expects today (v2026.9.6)
|
||||
|
||||
Checked against the OpenClaw source at tag `v2026.9.6` and its docs on
|
||||
2026-09-25. OpenClaw marks every plugin API as experimental, so re-check before
|
||||
building anything.
|
||||
|
||||
- **Memory lives in Markdown files**, which are the source of truth: `MEMORY.md`,
|
||||
`USER.md`, daily notes in `memory/YYYY-MM-DD.md` in the agent workspace. The
|
||||
memory engine is an index over them
|
||||
([concepts/memory](https://docs.openclaw.ai/concepts/memory)).
|
||||
- **A memory plugin is selected with `plugins.slots.memory: "<plugin-id>"`**
|
||||
(default `memory-core`), its settings under
|
||||
`plugins.entries.<plugin-id>.config`, validated against the plugin's own
|
||||
schema ([gateway/config-extensions](https://docs.openclaw.ai/gateway/config-extensions)).
|
||||
Memory search settings are under `memory.search`
|
||||
([reference/memory-config](https://docs.openclaw.ai/reference/memory-config)).
|
||||
- **A plugin needs** an `openclaw.plugin.json` manifest with `id`,
|
||||
`configSchema`, `"kind": "memory"` and every tool listed in `contracts.tools`
|
||||
([plugins/manifest](https://docs.openclaw.ai/plugins/manifest)); a
|
||||
`package.json` with `openclaw.extensions`, `openclaw.compat.pluginApi` and an
|
||||
`openclaw` peer dependency; and an entry built with `definePluginEntry`.
|
||||
- **Two ways to integrate** (both exist upstream): tools only, as
|
||||
`memory-lancedb` does (`api.registerTool`), or a full memory engine, as
|
||||
`memory-core` does, through `api.registerMemoryCapability({ runtime, ... })`,
|
||||
whose runtime returns a `MemorySearchManager` implementing `search`,
|
||||
`readFile` (returning `status: "ok" | "not_found"`), `status`,
|
||||
`probeEmbeddingAvailability` and `probeVectorAvailability`. Active Memory
|
||||
expects `memory_search` and `memory_get` tools
|
||||
([plugins/sdk-overview/memory-and-context](https://docs.openclaw.ai/plugins/sdk-overview/memory-and-context)).
|
||||
- **Embeddings come from OpenClaw's providers** (`memory.search.provider`), or a
|
||||
plugin registers one with `api.registerEmbeddingProvider`.
|
||||
- **Native code**: plugin installs run with `--ignore-scripts`, so a napi addon
|
||||
has to ship as prebuilt per-platform packages (the pattern `memory-lancedb`
|
||||
uses for LanceDB), loaded lazily
|
||||
([plugins/dependency-resolution](https://docs.openclaw.ai/plugins/dependency-resolution)).
|
||||
- **Distribution**: `openclaw plugins install` from npm or ClawHub; a first
|
||||
install from an arbitrary source needs explicit review, and community ClawHub
|
||||
packages go through a security audit.
|
||||
- **Churn to plan for**: the memory SDK was reshaped in 2026 (separate
|
||||
registration functions merged into `registerMemoryCapability`;
|
||||
`registerMemoryEmbeddingProvider` removed on 2026-08-21), and further SDK
|
||||
surfaces become eligible for removal on 2026-10-01
|
||||
([plugins/sdk-migration/removal-timeline](https://docs.openclaw.ai/plugins/sdk-migration/removal-timeline)).
|
||||
|
||||
## What this repository has
|
||||
|
||||
Building blocks, usable as a library today, but not an OpenClaw plugin:
|
||||
|
||||
- `clawhdf5_agent::openclaw::ClawhdfBackend` — a Markdown-oriented backend over
|
||||
`HDF5Memory`: ingest Markdown by section, hybrid search with re-ranking and
|
||||
confidence rejection, read back by path, export. Gaps a plugin would have to
|
||||
close: `write`/`ingest_markdown` store no embeddings (search is keyword-only
|
||||
for that content unless records are saved with `save_entry`), re-ingesting
|
||||
appends rather than replaces, there is no delete, `line_range` is never set,
|
||||
and export rewrites every heading as `##`.
|
||||
- `crates/clawhdf5-napi` and `packages/clawhdf5-node` — Node bindings and a
|
||||
TypeScript wrapper. **Not published, not built or tested in CI, and known to
|
||||
be broken**; see `docs/known-issues.md`.
|
||||
@@ -5,20 +5,20 @@ HDF5-backed agent memory system with hippocampal consolidation.
|
||||
|
||||
Built with [napi-rs](https://napi.rs).
|
||||
|
||||
> **Status: unpublished and known to be broken.** This package is not on npm,
|
||||
> no binaries are built, nothing in CI builds or tests it, and the wrapper
|
||||
> reads field names the native layer does not produce. It is not an OpenClaw
|
||||
> plugin. See [known issues](../../docs/known-issues.md) and
|
||||
> [docs/openclaw.md](../../docs/openclaw.md) before using it.
|
||||
|
||||
## Installation
|
||||
|
||||
Not published. Building from source needs `@napi-rs/cli`:
|
||||
|
||||
```bash
|
||||
npm install @redclaw/clawhdf5
|
||||
npm install && npm run build
|
||||
```
|
||||
|
||||
Pre-built binaries are published for:
|
||||
|
||||
| Platform | Architecture |
|
||||
|----------|-------------|
|
||||
| Linux (glibc) | x64, aarch64 |
|
||||
| macOS | x64, aarch64 (Apple Silicon) |
|
||||
| Windows | x64 |
|
||||
|
||||
## Quick start
|
||||
|
||||
```ts
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "@redclaw/clawhdf5",
|
||||
"version": "2.7.0",
|
||||
"description": "Node.js bindings for clawhdf5 — HDF5-backed agent memory with hippocampal consolidation",
|
||||
"description": "Node.js bindings for clawhdf5 \u2014 HDF5-backed agent memory with hippocampal consolidation",
|
||||
"main": "index.js",
|
||||
"types": "index.d.ts",
|
||||
"license": "MIT",
|
||||
@@ -14,8 +14,7 @@
|
||||
"memory",
|
||||
"hdf5",
|
||||
"vector-search",
|
||||
"embedding",
|
||||
"openclaw"
|
||||
"embedding"
|
||||
],
|
||||
"napi": {
|
||||
"name": "clawhdf5",
|
||||
@@ -51,5 +50,6 @@
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 16"
|
||||
}
|
||||
},
|
||||
"private": true
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user