feat(agent): Ed25519-signed checkpoints
CI / test-arm64 (pull_request) Successful in 1m5s
CI / test (pull_request) Successful in 4m50s

Makes the README's "cryptographically verifiable memory" true.

With HDF5Memory::set_signing_key(key), every checkpoint stores a signed
manifest of the store: a SHA-256 per memory record (text, embedding as
stored, channel, timestamp, session, tags, deleted flag, activation) in
a Merkle tree, plus hashes of the settings (and WAL mark), sessions and
knowledge graph. The signature, public key and manifest hashes go in
/meta; the per-record hashes in /integrity/record_hashes, so
HDF5Memory::verify(path, &public_key) can say which records changed, not
just that something did. A forged manifest fails the signature.

Decisions, as agreed:
- the key is set on the open store and never persisted;
- a signed store refuses to checkpoint without its key
  (MemoryError::SigningKeyRequired); remove_signature() is the
  deliberate way back to unsigned;
- checkpoints only: saves still in the WAL are not covered, and verify
  reports how many there are.

The hashes cover exactly what the file persists, in the form the loader
returns it (strings lose trailing NULs; an empty WAL mark is not
written), so untouched stores verify across any number of reopen and
checkpoint cycles. MemoryError becomes #[non_exhaustive] (it already
gains variants in this unreleased version).

CLI: keygen (owner-only key file), --signing-key / CLAWHDF5_SIGNING_KEY
on writing commands (create signs immediately), verify --public-key
(JSON; exit 2 if not valid), `signed` in create/stats output.

Tests: reopen/checkpoint cycles with awkward strings (f16 and f32),
refusal without the key, wrong and rotated keys, eight kinds of edit
each detected and located, a forged manifest, unsigned stores, NULs in
text, and an edit made in place with h5py that verify pinpoints.

Cost on tank (search_harness --signing-study --full, 3 runs): ~20% of a
checkpoint (+9 ms at 10K, +89-112 ms at 100K), verify 18.6 ms / 247 ms,
32 bytes per record in the file. New deps ed25519-dalek, sha2,
rand_core: pure Rust, the no-C check passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-25 10:13:34 -05:00
co-authored by Claude Opus 5.5
parent 4ecac65f22
commit db9af7972c
13 changed files with 1325 additions and 22 deletions
+32 -2
View File
@@ -8,7 +8,7 @@
[![LongMemEval](https://img.shields.io/badge/LongMemEval__s-Turn--Level%20Hit@5%2081.4%25%20hybrid-blue.svg)](BENCHMARKS.md#longmemeval-results)
[![Footprint](https://img.shields.io/badge/on--disk-~820%20B%2Frecord%20float16%2C%20synthetic%20text-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1)
ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file.
ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, cryptographically verifiable memory (Ed25519-signed checkpoints) — all stored in a single portable file.
> **Two things live here:**
> - **A general-purpose, pure-Rust HDF5 library** — zero C dependencies, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers.
@@ -113,7 +113,7 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d
| Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers |
| Temporal queries | Custom code | Native temporal index (622 ns range query over 10K) |
| Multi-modal | Multiple stores | Unified cross-modal search (exact scan: 842 µs over 1K records) |
| Integrity | Hope for the best | Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance |
| Integrity | Hope for the best | Ed25519-signed checkpoints that pinpoint any edited record, chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts |
| Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** |
---
@@ -369,6 +369,7 @@ directly; the store persists the records, sessions and graph they work over.
| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches. Opt-in via `SearchOptions::with_confidence`; on in the OpenClaw backend |
| **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints |
| **`multimodal`** | Cross-modal search across text/image/audio/video embeddings |
| **`signing`** | Ed25519-signed checkpoints: SHA-256 per record in a Merkle tree, plus hashes of settings, sessions and the knowledge graph; `HDF5Memory::verify` names any edited record |
| **`provenance`** | Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof) |
| **`anomaly`** | Write rate limiting, 15 injection-pattern detectors, source-distribution analysis. Alerts never block a save; drain them with `take_anomaly_alerts` |
| **`openclaw`** | OpenClaw integration: MemoryBackend trait, Markdown ↔ HDF5 conversion |
@@ -457,6 +458,35 @@ let careful = memory.search(
);
```
### Signed Checkpoints
```rust
use clawhdf5_agent::signing;
// Once, somewhere safe: keep the secret key, publish the public key.
let key = signing::generate_key();
let public = key.verifying_key();
// Every checkpoint is signed from now on. The key is never written to disk;
// a signed store refuses to checkpoint without it.
memory.set_signing_key(key);
memory.flush_wal()?;
// Anyone holding the public key can check the file, e.g. after copying it.
let report = HDF5Memory::verify(std::path::Path::new("agent.h5"), &public)?;
assert!(report.is_valid());
// On a tampered file: report.changed_records lists the records that differ.
```
The signature covers every record (text, embedding as stored, channel,
timestamp, session, tags, deleted flag, activation), the store's settings,
its sessions and its knowledge graph — a change made with any tool is caught.
It covers checkpoints, not saves still in the WAL
(`report.wal_entries_unsigned` counts those). CLI: `clawhdf5-cli keygen`,
`--signing-key <file>` on writing commands, and `verify --public-key`.
Signing adds about 20% to a checkpoint and 32 bytes per record to the file
([BENCHMARKS.md § Signed checkpoints](BENCHMARKS.md#signed-checkpoints)).
### Knowledge Graph
```rust