feat(agent): Ed25519-signed checkpoints
CI / test-arm64 (pull_request) Successful in 1m5s
CI / test (pull_request) Successful in 4m50s

Makes the README's "cryptographically verifiable memory" true.

With HDF5Memory::set_signing_key(key), every checkpoint stores a signed
manifest of the store: a SHA-256 per memory record (text, embedding as
stored, channel, timestamp, session, tags, deleted flag, activation) in
a Merkle tree, plus hashes of the settings (and WAL mark), sessions and
knowledge graph. The signature, public key and manifest hashes go in
/meta; the per-record hashes in /integrity/record_hashes, so
HDF5Memory::verify(path, &public_key) can say which records changed, not
just that something did. A forged manifest fails the signature.

Decisions, as agreed:
- the key is set on the open store and never persisted;
- a signed store refuses to checkpoint without its key
  (MemoryError::SigningKeyRequired); remove_signature() is the
  deliberate way back to unsigned;
- checkpoints only: saves still in the WAL are not covered, and verify
  reports how many there are.

The hashes cover exactly what the file persists, in the form the loader
returns it (strings lose trailing NULs; an empty WAL mark is not
written), so untouched stores verify across any number of reopen and
checkpoint cycles. MemoryError becomes #[non_exhaustive] (it already
gains variants in this unreleased version).

CLI: keygen (owner-only key file), --signing-key / CLAWHDF5_SIGNING_KEY
on writing commands (create signs immediately), verify --public-key
(JSON; exit 2 if not valid), `signed` in create/stats output.

Tests: reopen/checkpoint cycles with awkward strings (f16 and f32),
refusal without the key, wrong and rotated keys, eight kinds of edit
each detected and located, a forged manifest, unsigned stores, NULs in
text, and an edit made in place with h5py that verify pinpoints.

Cost on tank (search_harness --signing-study --full, 3 runs): ~20% of a
checkpoint (+9 ms at 10K, +89-112 ms at 100K), verify 18.6 ms / 247 ms,
32 bytes per record in the file. New deps ed25519-dalek, sha2,
rand_core: pure Rust, the no-C check passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-25 10:13:34 -05:00
co-authored by Claude Opus 5.5
parent 4ecac65f22
commit db9af7972c
13 changed files with 1325 additions and 22 deletions
@@ -21,6 +21,7 @@
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --ann-only --uniform
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --options-study --full
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --signing-study --full
//! ```
use std::time::{Duration, Instant};
@@ -488,6 +489,81 @@ fn bench_end_to_end(n: usize, json: &mut Vec<serde_json::Value>) {
}));
}
// ---------------------------------------------------------------------------
// Signing study: what does an Ed25519-signed checkpoint cost?
// ---------------------------------------------------------------------------
/// `--signing-study`: checkpoint time unsigned vs signed, `verify` time, and
/// the file-size cost of the stored per-record hashes. Default store
/// settings (float16, int8 index). Medians of five checkpoints / three
/// verifies.
fn signing_study(n: usize) {
use clawhdf5_agent::signing::SigningKey;
let data = make_dataset(n, 0x516 ^ n as u64);
let mut rng = Rng(9);
let entries: Vec<MemoryEntry> = data
.vectors
.iter()
.enumerate()
.map(|(i, v)| MemoryEntry {
chunk: text_for(data.cluster_of[i], i, &mut rng),
embedding: v.clone(),
source_channel: "bench".into(),
timestamp: i as f64,
session_id: format!("s{}", i % 50),
tags: format!("t{i}"),
})
.collect();
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("sign.h5");
let mut mem = HDF5Memory::create(MemoryConfig::new(path.clone(), "bench", DIM)).unwrap();
mem.save_batch(entries).unwrap();
std::hint::black_box(mem.hybrid_search(&data.queries[0], "", 1.0, 0.0, K));
let median = |mut v: Vec<Duration>| {
v.sort();
v[v.len() / 2]
};
let checkpoint = |mem: &mut HDF5Memory| {
median(
(0..5)
.map(|_| {
let t = Instant::now();
mem.flush_wal().unwrap();
t.elapsed()
})
.collect(),
)
};
let unsigned = checkpoint(&mut mem);
let unsigned_bytes = std::fs::metadata(&path).unwrap().len();
let key = SigningKey::from_bytes(&[7; 32]);
mem.set_signing_key(key.clone());
let signed = checkpoint(&mut mem);
let signed_bytes = std::fs::metadata(&path).unwrap().len();
drop(mem);
let vk = key.verifying_key();
let verify = median(
(0..3)
.map(|_| {
let t = Instant::now();
let r = HDF5Memory::verify(&path, &vk).unwrap();
let d = t.elapsed();
assert!(r.is_valid());
d
})
.collect(),
);
println!(
"| {n} | {:.1} | {:.1} | {:+.1} | {:.1} | {:+.2} |",
millis(unsigned),
millis(signed),
millis(signed) - millis(unsigned),
millis(verify),
(signed_bytes as f64 - unsigned_bytes as f64) / (1024.0 * 1024.0),
);
}
// ---------------------------------------------------------------------------
// Search options study: source filters, re-ranking, confidence rejection
// ---------------------------------------------------------------------------
@@ -960,6 +1036,21 @@ fn main() {
}
return;
}
if args.iter().any(|a| a == "--signing-study") {
println!("## Signed checkpoints ({DIM}-dim, float16, int8 index)\n");
println!(
"| N | checkpoint ms, unsigned | checkpoint ms, signed | signing adds ms | verify ms | file MiB added |"
);
println!("|---:|---:|---:|---:|---:|---:|");
for &n in if full {
&[1_000, 10_000, 100_000][..]
} else {
&[1_000, 10_000][..]
} {
signing_study(n);
}
return;
}
if args.iter().any(|a| a == "--options-study") {
println!("## Search options ({DIM}-dim, k = {K}, Hebbian boost off)\n");
println!("| N | options | filtered recall@10 | p50 ms | p99 ms |");