diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 9866d71..f6e6375 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -243,6 +243,30 @@ index asked for ~16 000 candidates, where scanning the few hundred or thousand allowed records is exact and cheap. Re-ranking a 3k candidate pool and confidence rejection add about 3%. +### Signed checkpoints + +Measured 2026-09-25 on tank (AMD Ryzen 7 7800X3D). A default store (float16, +int8 index), 384-dim; each checkpoint rewrites the whole file, as every +checkpoint does. Medians of five checkpoints and three verifies; three runs +agreed to within the ranges shown. + +```bash +cargo run --release -p clawhdf5-bench --bin search_harness -- --signing-study --full +``` + +| N | checkpoint, unsigned | checkpoint, signed | signing adds | `verify` | file size added | +|---:|---:|---:|---:|---:|---:| +| 1 000 | 5.4 ms | 6.4 ms | 0.7–1.0 ms | 2.1 ms | 0.03 MiB | +| 10 000 | 46 ms | 55 ms | 8.1–9.4 ms | 18.6 ms | 0.31 MiB | +| 100 000 | 495 ms | 598 ms | 89–112 ms | 247 ms | 3.05 MiB | + +Signing costs about 20% of a checkpoint: every record is rehashed (SHA-256) +and the Merkle root recomputed each time; the Ed25519 signature itself is +microseconds. Caching per-record hashes between checkpoints would cut this to +the records that changed. The per-record hashes stored for locating edits are +32 bytes each (4% of a 100K float16 store). `verify` reads and rehashes the +whole checkpoint. + ### float16 embedding storage (`MemoryConfig::float16`) Measured 2026-09-23 on tank (AMD Ryzen 7 7800X3D). The same clustered diff --git a/CHANGELOG.md b/CHANGELOG.md index 97b2342..166600d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,12 @@ ## Unreleased ### Upgrade Notes +- **Breaking:** `MemoryError` is now `#[non_exhaustive]` and gained + `SigningKeyRequired`; a `match` on it needs a wildcard arm. Future variants + will no longer be breaking. +- **Breaking:** `clawhdf5-agent`'s `agent` feature is removed. It enabled + nothing — the agent layer is always built — but the README and guides told + people to pass it; drop `agent` from `features = [...]`. - **`clawhdf5-migrate` now writes a real agent store.** Its output used to be a layout of its own (`/chunks`, `/sessions`, `/entities`, `/relations`, no `/meta`) that `HDF5Memory::open` rejected, so a migrated file could not be @@ -57,6 +63,29 @@ `quantized_index = false`, or pass `create --f32-index` to the CLI, to opt out. The CLI's `--quantized-index` is still accepted but is now a no-op. +### Signing +- `clawhdf5-agent`: **Ed25519-signed checkpoints** — the README's + "cryptographically verifiable memory", now true. With + `HDF5Memory::set_signing_key(key)`, every checkpoint stores a signed + manifest: a SHA-256 per record (text, embedding as stored, channel, + timestamp, session, tags, deleted flag, activation) in a Merkle tree, plus + hashes of the settings (and WAL mark), sessions and knowledge graph, with + the per-record hashes in `/integrity/record_hashes`. + `HDF5Memory::verify(path, &public_key)` recomputes everything from the file + and reports which part changed and which records (`changed_records`); a + forged manifest fails the signature. The key is never persisted; a signed + store refuses to checkpoint without it (`MemoryError::SigningKeyRequired`), + and `remove_signature()` is the deliberate way back to unsigned. Saves still + in the WAL are not covered (`wal_entries_unsigned`). Tests include every + kind of edit, and an edit made with h5py in place, which verify pinpoints. + Cost: ~20% of a checkpoint, 32 bytes per record (`BENCHMARKS.md`, "Signed + checkpoints"). New dependencies `ed25519-dalek`, `sha2`, `rand_core` — pure + Rust; the no-C check still passes. +- `clawhdf5-cli`: `keygen --out ` (owner-only key file), + `--signing-key ` / `CLAWHDF5_SIGNING_KEY` on writing commands + (`create` signs immediately), `verify --public-key ` (JSON report; + exit status 2 if not valid), and `signed` in `create`/`stats` output. + ### Migration - `clawhdf5-migrate`: writes through the agent's own API (`HDF5Memory::create` / `open`, `save_batch`, the session cache and knowledge graph), so there is diff --git a/CLAUDE.md b/CLAUDE.md index 21b47d8..aa94af2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -109,6 +109,17 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F `search_harness --options-study`. - `MemoryConfig::compression` is off by default; when on, embeddings are deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd). +- Signed checkpoints (`clawhdf5-agent` `signing` module): with + `HDF5Memory::set_signing_key` every checkpoint stores an Ed25519-signed + manifest (SHA-256 per record in a Merkle tree + settings/sessions/graph + hashes; per-record hashes in `/integrity/record_hashes`); + `HDF5Memory::verify(path, &pk)` locates edits. The hashes must cover exactly + what the file persists in the form the loader returns it (strings lose + trailing NULs; an empty WAL mark is not written) or untouched stores stop + verifying — `tests/signed_store.rs` round-trips awkward strings. The key is + never persisted; a signed store refuses to checkpoint without it + (`MemoryError::SigningKeyRequired`, and `MemoryError` is `#[non_exhaustive]`). + WAL entries after the checkpoint are not covered. - `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by default) recomputes a dataset's SHA-256 and compares it against the `_provenance_sha256` attribute written automatically on save when diff --git a/README.md b/README.md index cbf37e9..cc5f436 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ [![LongMemEval](https://img.shields.io/badge/LongMemEval__s-Turn--Level%20Hit@5%2081.4%25%20hybrid-blue.svg)](BENCHMARKS.md#longmemeval-results) [![Footprint](https://img.shields.io/badge/on--disk-~820%20B%2Frecord%20float16%2C%20synthetic%20text-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1) -ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file. +ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, cryptographically verifiable memory (Ed25519-signed checkpoints) — all stored in a single portable file. > **Two things live here:** > - **A general-purpose, pure-Rust HDF5 library** — zero C dependencies, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers. @@ -113,7 +113,7 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d | Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers | | Temporal queries | Custom code | Native temporal index (622 ns range query over 10K) | | Multi-modal | Multiple stores | Unified cross-modal search (exact scan: 842 µs over 1K records) | -| Integrity | Hope for the best | Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance | +| Integrity | Hope for the best | Ed25519-signed checkpoints that pinpoint any edited record, chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts | | Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** | --- @@ -369,6 +369,7 @@ directly; the store persists the records, sessions and graph they work over. | **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches. Opt-in via `SearchOptions::with_confidence`; on in the OpenClaw backend | | **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints | | **`multimodal`** | Cross-modal search across text/image/audio/video embeddings | +| **`signing`** | Ed25519-signed checkpoints: SHA-256 per record in a Merkle tree, plus hashes of settings, sessions and the knowledge graph; `HDF5Memory::verify` names any edited record | | **`provenance`** | Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof) | | **`anomaly`** | Write rate limiting, 15 injection-pattern detectors, source-distribution analysis. Alerts never block a save; drain them with `take_anomaly_alerts` | | **`openclaw`** | OpenClaw integration: MemoryBackend trait, Markdown ↔ HDF5 conversion | @@ -457,6 +458,35 @@ let careful = memory.search( ); ``` +### Signed Checkpoints + +```rust +use clawhdf5_agent::signing; + +// Once, somewhere safe: keep the secret key, publish the public key. +let key = signing::generate_key(); +let public = key.verifying_key(); + +// Every checkpoint is signed from now on. The key is never written to disk; +// a signed store refuses to checkpoint without it. +memory.set_signing_key(key); +memory.flush_wal()?; + +// Anyone holding the public key can check the file, e.g. after copying it. +let report = HDF5Memory::verify(std::path::Path::new("agent.h5"), &public)?; +assert!(report.is_valid()); +// On a tampered file: report.changed_records lists the records that differ. +``` + +The signature covers every record (text, embedding as stored, channel, +timestamp, session, tags, deleted flag, activation), the store's settings, +its sessions and its knowledge graph — a change made with any tool is caught. +It covers checkpoints, not saves still in the WAL +(`report.wal_entries_unsigned` counts those). CLI: `clawhdf5-cli keygen`, +`--signing-key ` on writing commands, and `verify --public-key`. +Signing adds about 20% to a checkpoint and 32 bytes per record to the file +([BENCHMARKS.md § Signed checkpoints](BENCHMARKS.md#signed-checkpoints)). + ### Knowledge Graph ```rust @@ -615,7 +645,6 @@ ClawhDF5's agent memory design draws from 15+ recent papers: | `openblas` | no | OpenBLAS (Linux) | | `gpu` | no | GPU search via wgpu | | `async` | no | Tokio async with background flush | -| `agent` | no | Reserved; currently enables nothing (the agent layer is always built) | To opt out of the parallel build: `--no-default-features --features float16,hnsw`. For an exact linear cosine scan instead of HNSW: `--no-default-features --features float16`. diff --git a/crates/clawhdf5-agent/Cargo.toml b/crates/clawhdf5-agent/Cargo.toml index 8af2c6f..ec8adcd 100644 --- a/crates/clawhdf5-agent/Cargo.toml +++ b/crates/clawhdf5-agent/Cargo.toml @@ -19,6 +19,10 @@ clawhdf5-ann = { path = "../clawhdf5-ann", version = "2.7.0", optional = true } clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.7.0", optional = true, default-features = false } serde = { workspace = true } byteorder = "1" +# Signed checkpoints (MemoryConfig-independent; see `signing`). Pure Rust. +ed25519-dalek = { version = "2", features = ["rand_core"] } +sha2 = "0.10" +rand_core = { version = "0.6", features = ["getrandom"] } half = { workspace = true, optional = true } rayon = { version = "1", optional = true } matrixmultiply = { version = "0.3", optional = true } @@ -64,7 +68,6 @@ zstd = ["clawhdf5/zstd"] # `--no-default-features` (plus re-enabling other defaults) to force the exact # linear cosine scan. hnsw = ["clawhdf5-ann"] -agent = [] gpu = ["clawhdf5-gpu/gpu-wgpu"] fast-math = ["matrixmultiply"] accelerate = ["accelerate-src", "cblas-sys"] diff --git a/crates/clawhdf5-agent/src/lib.rs b/crates/clawhdf5-agent/src/lib.rs index 62575fa..8bb7811 100644 --- a/crates/clawhdf5-agent/src/lib.rs +++ b/crates/clawhdf5-agent/src/lib.rs @@ -36,6 +36,7 @@ pub mod reranker; pub mod schema; pub mod search; pub mod session; +pub mod signing; pub mod storage; mod store_lock; pub mod temporal; @@ -78,6 +79,7 @@ pub use session::{SessionCache, SessionEntry}; // --- Error type --- #[derive(Debug)] +#[non_exhaustive] pub enum MemoryError { Io(std::io::Error), Hdf5(String), @@ -88,6 +90,11 @@ pub enum MemoryError { /// A record the store cannot hold as given, e.g. an embedding value /// outside the half-precision range of a `float16` store. InvalidEntry(String), + /// The store's checkpoints are signed and no signing key is set, so a + /// checkpoint would leave it unsigned. Set the key with + /// [`HDF5Memory::set_signing_key`], or drop the signature on purpose with + /// [`HDF5Memory::remove_signature`]. + SigningKeyRequired(String), } impl std::fmt::Display for MemoryError { @@ -99,6 +106,7 @@ impl std::fmt::Display for MemoryError { MemoryError::NotFound(e) => write!(f, "not found: {e}"), MemoryError::Locked(e) => write!(f, "store is locked: {e}"), MemoryError::InvalidEntry(e) => write!(f, "invalid entry: {e}"), + MemoryError::SigningKeyRequired(e) => write!(f, "signing key required: {e}"), } } } @@ -317,6 +325,12 @@ pub struct HDF5Memory { activations_dirty: bool, /// Opened with [`HDF5Memory::open_read_only`]: nothing may reach the disk. read_only: bool, + /// Key that signs every checkpoint; never persisted. See + /// [`HDF5Memory::set_signing_key`]. + signing_key: Option, + /// Checkpoints of this store are signed: the file on disk is, or a key + /// has been set. A checkpoint without a key is then refused. + signed: bool, /// A WAL that `open()` could not read and moved aside; see /// [`HDF5Memory::quarantined_wal`]. quarantined_wal: Option, @@ -372,6 +386,8 @@ impl HDF5Memory { bm25_filter: bm25::TokenFilter::default(), activations_dirty: false, read_only: false, + signing_key: None, + signed: false, quarantined_wal: None, _lock: Some(lock), }) @@ -550,6 +566,8 @@ impl HDF5Memory { bm25_filter: bm25::TokenFilter::default(), activations_dirty: false, read_only, + signing_key: None, + signed: checkpoint.signed, quarantined_wal, _lock: lock, }) @@ -710,6 +728,39 @@ impl HDF5Memory { } } + /// Sign every checkpoint from now on with `key` (Ed25519). The key is + /// never written anywhere; set it again after every `open`. Once a store + /// is signed, a checkpoint without the key is refused + /// ([`MemoryError::SigningKeyRequired`]) rather than silently leaving it + /// unsigned. Setting a different key re-signs the store under that key + /// from the next checkpoint; a verifier trusting the old key will then + /// reject it, which is the point. Call [`AgentMemory::flush_wal`] to sign + /// right away. + pub fn set_signing_key(&mut self, key: signing::SigningKey) { + self.signing_key = Some(key); + self.signed = true; + } + + /// Stop signing: the next checkpoint writes the store unsigned. The + /// deliberate way out of [`MemoryError::SigningKeyRequired`]. + pub fn remove_signature(&mut self) { + self.signing_key = None; + self.signed = false; + } + + /// Checkpoints of this store are signed (on disk, or from the next + /// checkpoint because a key has been set). + pub fn is_signed(&self) -> bool { + self.signed + } + + /// Check the checkpoint at `path` against the public key the caller + /// trusts; see [`signing::verify_store`]. Reads the file only: it works + /// on a store another process has open. + pub fn verify(path: &Path, trusted: &signing::VerifyingKey) -> Result { + signing::verify_store(path, trusted) + } + /// Flush current state to disk and truncate the WAL. /// /// Every code path that persists the full cache to the .h5 file must @@ -725,10 +776,28 @@ impl HDF5Memory { // Record which WAL prefix this checkpoint contains, so a crash before // the truncate below can't replay those entries a second time. let wal_applied = self.wal.as_ref().map(|w| w.mark()); + let signature = match &self.signing_key { + Some(key) => Some(signing::sign( + key, + &self.config, + &self.cache, + &self.sessions, + &self.knowledge, + wal_applied, + )), + None if self.signed => { + return Err(MemoryError::SigningKeyRequired(format!( + "{} is signed; set its signing key before a checkpoint \ + (saves so far are held in the WAL or in memory)", + self.config.path.display() + ))); + } + None => None, + }; // Written before the .h5 so a crash in between leaves a sidecar whose // generation matches no checkpoint (ignored), never the reverse. let ann_generation = self.persist_vector_index(); - storage::write_to_disk_with_meta( + storage::write_to_disk_signed( &self.config.path, &self.config, &self.cache, @@ -737,7 +806,9 @@ impl HDF5Memory { &schema::CheckpointMeta { wal_applied, ann_generation, + signed: signature.is_some(), }, + signature.as_ref(), )?; if let Some(ref mut w) = self.wal { w.truncate()?; diff --git a/crates/clawhdf5-agent/src/schema.rs b/crates/clawhdf5-agent/src/schema.rs index 3555986..679a0b5 100644 --- a/crates/clawhdf5-agent/src/schema.rs +++ b/crates/clawhdf5-agent/src/schema.rs @@ -23,6 +23,7 @@ pub const ZEROCLAW_VERSION: &str = "0.8.0"; const WAL_APPLIED_LEN_ATTR: &str = "wal_applied_len"; const WAL_APPLIED_CRC_ATTR: &str = "wal_applied_crc"; const ANN_GENERATION_ATTR: &str = "ann_generation"; +const SIG_VERSION_ATTR: &str = "sig_version"; /// Build a complete HDF5 file from the in-memory state. pub fn build_hdf5_file( @@ -46,7 +47,7 @@ pub fn build_hdf5_file_with_mark( ) -> Result, MemoryError> { let meta = CheckpointMeta { wal_applied, - ann_generation: None, + ..CheckpointMeta::default() }; build_hdf5_file_with_meta(config, cache, sessions, knowledge, &meta) } @@ -61,6 +62,10 @@ pub struct CheckpointMeta { /// one left over from another checkpoint can never be attached to records /// it wasn't built from. pub ann_generation: Option, + /// The checkpoint carries an Ed25519 signature (see [`crate::signing`]). + /// Read-only: whether a checkpoint is *written* signed is decided by the + /// signature passed to [`build_hdf5_file_signed`]. + pub signed: bool, } /// [`build_hdf5_file`] with checkpoint bookkeeping. @@ -70,6 +75,19 @@ pub fn build_hdf5_file_with_meta( sessions: &SessionCache, knowledge: &KnowledgeCache, checkpoint: &CheckpointMeta, +) -> Result, MemoryError> { + build_hdf5_file_signed(config, cache, sessions, knowledge, checkpoint, None) +} + +/// [`build_hdf5_file_with_meta`], plus a signed manifest of the contents +/// (see [`crate::signing`]). +pub fn build_hdf5_file_signed( + config: &MemoryConfig, + cache: &MemoryCache, + sessions: &SessionCache, + knowledge: &KnowledgeCache, + checkpoint: &CheckpointMeta, + signature: Option<&crate::signing::StoredSignature>, ) -> Result, MemoryError> { let wal_applied = checkpoint.wal_applied; let mut builder = clawhdf5::FileBuilder::new(); @@ -130,11 +148,42 @@ pub fn build_hdf5_file_with_meta( // round trip through every reader. meta.set_attr(ANN_GENERATION_ATTR, AttrValue::I64(generation as i64)); } + if let Some(sig) = signature { + use crate::signing::to_hex; + let m = &sig.manifest; + meta.set_attr( + SIG_VERSION_ATTR, + AttrValue::I64(crate::signing::MANIFEST_VERSION), + ); + meta.set_attr("sig_algorithm", AttrValue::String("ed25519".into())); + meta.set_attr("sig_public_key", AttrValue::String(to_hex(&sig.public_key))); + meta.set_attr("sig_signature", AttrValue::String(to_hex(&sig.signature))); + meta.set_attr("sig_record_count", AttrValue::I64(m.record_count as i64)); + meta.set_attr( + "sig_records_root", + AttrValue::String(to_hex(&m.records_root)), + ); + meta.set_attr("sig_settings", AttrValue::String(to_hex(&m.settings))); + meta.set_attr("sig_sessions", AttrValue::String(to_hex(&m.sessions))); + meta.set_attr("sig_graph", AttrValue::String(to_hex(&m.graph))); + } // Need at least one dataset in the group for it to be a proper group meta.create_dataset("_marker").with_u8_data(&[1]).compact(); let finished_meta = meta.finish(); builder.add_group(finished_meta); + // /integrity: the signed per-record hashes, so verification can say + // which records changed. + if let Some(sig) = signature { + let mut group = builder.create_group("integrity"); + let flat: Vec = sig.record_hashes.iter().flatten().copied().collect(); + group + .create_dataset("record_hashes") + .with_u8_data(&flat) + .with_shape(&[sig.record_hashes.len() as u64, 32]); + builder.add_group(group.finish()); + } + // /memory group build_memory_group(&mut builder, config, cache)?; @@ -440,6 +489,64 @@ pub fn read_wal_mark(file: &clawhdf5::File) -> Option { Some(WalMark { len, crc }) } +/// Read a checkpoint's signature, if it has one. A signature whose +/// attributes are present but malformed is an error, not "unsigned". +pub fn read_signature( + file: &clawhdf5::File, +) -> Result, MemoryError> { + use crate::signing::{Manifest, StoredSignature, from_hex}; + let attrs = file + .group("meta") + .and_then(|g| g.attrs()) + .map_err(|e| MemoryError::Schema(format!("cannot read /meta attrs: {e}")))?; + let version = match attrs.get(SIG_VERSION_ATTR) { + None => return Ok(None), + Some(AttrValue::I64(v)) => *v, + Some(_) => return Err(MemoryError::Schema("malformed sig_version".into())), + }; + if version != crate::signing::MANIFEST_VERSION { + return Err(MemoryError::Schema(format!( + "unsupported signature version {version}" + ))); + } + fn hex( + attrs: &std::collections::HashMap, + name: &str, + ) -> Result<[u8; N], MemoryError> { + match attrs.get(name) { + Some(AttrValue::String(s)) => from_hex::(s), + _ => None, + } + .ok_or_else(|| MemoryError::Schema(format!("malformed or missing {name}"))) + } + let record_count = match attrs.get("sig_record_count") { + Some(AttrValue::I64(v)) if *v >= 0 => *v as u64, + _ => return Err(MemoryError::Schema("malformed sig_record_count".into())), + }; + let group = file + .group("integrity") + .map_err(|e| MemoryError::Schema(format!("signed checkpoint without /integrity: {e}")))?; + let flat = read_u8_dataset(&group, "record_hashes")?; + if flat.len() % 32 != 0 { + return Err(MemoryError::Schema( + "/integrity/record_hashes is not a whole number of hashes".into(), + )); + } + let record_hashes = flat.as_chunks::<32>().0.to_vec(); + Ok(Some(StoredSignature { + manifest: Manifest { + record_count, + records_root: hex::<32>(&attrs, "sig_records_root")?, + settings: hex::<32>(&attrs, "sig_settings")?, + sessions: hex::<32>(&attrs, "sig_sessions")?, + graph: hex::<32>(&attrs, "sig_graph")?, + }, + record_hashes, + public_key: hex::<32>(&attrs, "sig_public_key")?, + signature: hex::<64>(&attrs, "sig_signature")?, + })) +} + /// Read the checkpoint bookkeeping from `/meta`. pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta { let ann_generation = file @@ -450,9 +557,14 @@ pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta { Some(AttrValue::I64(v)) => Some(*v as u64), _ => None, }); + let signed = file + .group("meta") + .and_then(|g| g.attrs()) + .is_ok_and(|attrs| attrs.contains_key(SIG_VERSION_ATTR)); CheckpointMeta { wal_applied: read_wal_mark(file), ann_generation, + signed, } } diff --git a/crates/clawhdf5-agent/src/signing.rs b/crates/clawhdf5-agent/src/signing.rs new file mode 100644 index 0000000..7eb2f15 --- /dev/null +++ b/crates/clawhdf5-agent/src/signing.rs @@ -0,0 +1,419 @@ +//! Ed25519-signed checkpoints. +//! +//! When a signing key is set ([`crate::HDF5Memory::set_signing_key`]), every +//! checkpoint writes a signed manifest of the store: a SHA-256 per memory +//! record rolled into a Merkle root, plus hashes of the store's settings, its +//! sessions and its knowledge graph. [`verify_store`] recomputes all of it from +//! the file and checks the signature against a public key the caller trusts, +//! so any change to the checkpointed file — a record's text or embedding, a +//! setting, a session, a graph edge, made through this crate or any other HDF5 +//! tool — is detected, and the per-record hashes say which records changed. +//! +//! What it does not cover: saves still only in the WAL (made since the last +//! checkpoint). [`VerifyReport::wal_entries_unsigned`] counts them. +//! +//! The hashes cover exactly what the file persists, in the form the loader +//! returns it, so a store verifies after any number of reopen/checkpoint +//! cycles. Derived data (L2 norms, the vector index) is not covered; it is +//! recomputed from covered data. + +use ed25519_dalek::{Signature, Signer, Verifier}; +pub use ed25519_dalek::{SigningKey, VerifyingKey}; +use sha2::{Digest, Sha256}; + +use crate::MemoryConfig; +use crate::cache::MemoryCache; +use crate::knowledge::KnowledgeCache; +use crate::session::SessionCache; +use crate::wal::WalMark; + +/// Version of the manifest encoding; part of what is signed. +pub const MANIFEST_VERSION: i64 = 1; + +type Hash = [u8; 32]; + +/// The hashes a signature covers. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Manifest { + pub record_count: u64, + /// Merkle root over the per-record hashes. + pub records_root: Hash, + /// Settings persisted in `/meta`, plus the checkpoint's WAL mark. + pub settings: Hash, + pub sessions: Hash, + pub graph: Hash, +} + +impl Manifest { + /// The exact bytes that are signed. + pub fn signed_bytes(&self) -> Vec { + let mut m = Vec::with_capacity(160); + m.extend_from_slice(b"clawhdf5-agent signed checkpoint\0"); + m.extend_from_slice(&MANIFEST_VERSION.to_le_bytes()); + m.extend_from_slice(&self.record_count.to_le_bytes()); + m.extend_from_slice(&self.records_root); + m.extend_from_slice(&self.settings); + m.extend_from_slice(&self.sessions); + m.extend_from_slice(&self.graph); + m + } +} + +/// A signature as stored in a checkpoint. +#[derive(Debug, Clone)] +pub struct StoredSignature { + pub manifest: Manifest, + pub record_hashes: Vec, + pub public_key: [u8; 32], + pub signature: [u8; 64], +} + +/// Build the manifest (and per-record hashes) for the state about to be +/// checkpointed, and sign it. +pub fn sign( + key: &SigningKey, + config: &MemoryConfig, + cache: &MemoryCache, + sessions: &SessionCache, + knowledge: &KnowledgeCache, + wal_applied: Option, +) -> StoredSignature { + let (manifest, record_hashes) = manifest(config, cache, sessions, knowledge, wal_applied); + let signature = key.sign(&manifest.signed_bytes()).to_bytes(); + StoredSignature { + manifest, + record_hashes, + public_key: key.verifying_key().to_bytes(), + signature, + } +} + +/// Compute the manifest of a store's state. +pub fn manifest( + config: &MemoryConfig, + cache: &MemoryCache, + sessions: &SessionCache, + knowledge: &KnowledgeCache, + wal_applied: Option, +) -> (Manifest, Vec) { + let record_hashes: Vec = (0..cache.len()).map(|i| record_hash(cache, i)).collect(); + let manifest = Manifest { + record_count: cache.len() as u64, + records_root: merkle_root(&record_hashes), + settings: settings_hash(config, wal_applied), + sessions: sessions_hash(sessions), + graph: graph_hash(knowledge), + }; + (manifest, record_hashes) +} + +// --------------------------------------------------------------------------- +// Canonical encoding +// --------------------------------------------------------------------------- + +/// A SHA-256 over length-prefixed fields, so no two different field lists +/// hash the same bytes. +struct Fields(Sha256); + +impl Fields { + fn new(domain: &str) -> Self { + let mut h = Sha256::new(); + h.update((domain.len() as u64).to_le_bytes()); + h.update(domain.as_bytes()); + Self(h) + } + fn bytes(&mut self, b: &[u8]) -> &mut Self { + self.0.update((b.len() as u64).to_le_bytes()); + self.0.update(b); + self + } + /// Strings as the loader returns them: stored null-padded, so a trailing + /// NUL cannot survive a round trip and must not be part of the hash. + fn str(&mut self, s: &str) -> &mut Self { + self.bytes(s.trim_end_matches('\0').as_bytes()) + } + fn u64(&mut self, v: u64) -> &mut Self { + self.0.update(v.to_le_bytes()); + self + } + fn f64(&mut self, v: f64) -> &mut Self { + self.0.update(v.to_bits().to_le_bytes()); + self + } + fn f32(&mut self, v: f32) -> &mut Self { + self.0.update(v.to_bits().to_le_bytes()); + self + } + fn finish(self) -> Hash { + self.0.finalize().into() + } +} + +/// Everything persisted about record `i`, including its position. The +/// embedding is hashed as the cache holds it — for a `float16` store that is +/// the half-rounded value the file holds. +fn record_hash(cache: &MemoryCache, i: usize) -> Hash { + let mut f = Fields::new("clawhdf5-agent/record"); + f.u64(i as u64).str(&cache.chunks[i]); + let emb: Vec = cache.embeddings[i] + .iter() + .flat_map(|v| v.to_bits().to_le_bytes()) + .collect(); + f.bytes(&emb) + .str(&cache.source_channels[i]) + .f64(cache.timestamps[i]) + .str(&cache.session_ids[i]) + .str(&cache.tags[i]) + .u64(u64::from(cache.tombstones[i])) + .f32(cache.activation_weights[i]); + f.finish() +} + +/// Binary Merkle tree: leaves are the record hashes; a parent hashes its two +/// children with a node prefix; an odd node is carried up unchanged. +fn merkle_root(leaves: &[Hash]) -> Hash { + if leaves.is_empty() { + return Fields::new("clawhdf5-agent/merkle-empty").finish(); + } + let mut level: Vec = leaves.to_vec(); + while level.len() > 1 { + level = level + .chunks(2) + .map(|pair| match pair { + [l, r] => { + let mut h = Sha256::new(); + h.update([1u8]); + h.update(l); + h.update(r); + h.finalize().into() + } + [only] => *only, + _ => unreachable!(), + }) + .collect(); + } + level[0] +} + +fn settings_hash(c: &MemoryConfig, wal_applied: Option) -> Hash { + let mut f = Fields::new("clawhdf5-agent/settings"); + f.str(crate::schema::SCHEMA_VERSION) + .str(&c.created_at) + .str(&c.agent_id) + .str(&c.embedder) + .u64(c.embedding_dim as u64) + .u64(c.chunk_size as u64) + .u64(c.overlap as u64) + .u64(u64::from(c.float16)) + .u64(u64::from(c.compression)) + .u64(u64::from(c.compression_level)) + .f32(c.compact_threshold) + .f32(c.hebbian_boost) + .f32(c.decay_factor) + .u64(u64::from(c.wal_enabled)) + .u64(c.wal_max_entries as u64) + .u64(u64::from(c.quantized_index)) + .u64(c.hnsw_m as u64) + .u64(c.hnsw_ef_construction as u64) + .u64(c.hnsw_ef_search as u64); + // An empty mark is not written to the file, so it must hash as none. + match wal_applied.filter(|m| m.len > 0) { + Some(m) => f.u64(1).u64(m.len).u64(u64::from(m.crc)), + None => f.u64(0), + }; + f.finish() +} + +fn sessions_hash(s: &SessionCache) -> Hash { + let mut f = Fields::new("clawhdf5-agent/sessions"); + f.u64(s.entries.len() as u64); + for (i, e) in s.entries.iter().enumerate() { + f.str(&e.id) + .u64(e.start_idx) + .u64(e.end_idx) + .str(&e.channel) + .f64(e.ts) + .str(s.summaries.get(i).map(String::as_str).unwrap_or("")); + } + f.finish() +} + +fn graph_hash(k: &KnowledgeCache) -> Hash { + let mut f = Fields::new("clawhdf5-agent/graph"); + f.u64(k.entities.len() as u64); + for e in &k.entities { + f.u64(e.id) + .str(&e.name) + .str(&e.entity_type) + .u64(e.embedding_idx as u64); + } + f.u64(k.relations.len() as u64); + for r in &k.relations { + f.u64(r.src) + .u64(r.tgt) + .str(&r.relation) + .f32(r.weight) + .f64(r.ts); + } + f.u64(k.alias_strings.len() as u64); + for (s, id) in k.alias_strings.iter().zip(&k.alias_entity_ids) { + f.str(s).u64(*id as u64); + } + f.finish() +} + +// --------------------------------------------------------------------------- +// Verification +// --------------------------------------------------------------------------- + +/// The outcome of [`verify_store`]. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct VerifyReport { + /// The checkpoint carries a signature. + pub signed: bool, + /// The signature was made by the key the caller trusts. + pub key_matches: bool, + /// The signature over the stored manifest is valid. + pub signature_valid: bool, + /// The file's current contents match the signed manifest. + pub records_match: bool, + pub settings_match: bool, + pub sessions_match: bool, + pub graph_match: bool, + /// Records whose contents differ from what was signed (by position), + /// when the stored per-record hashes are themselves authentic. + pub changed_records: Vec, + /// Records in the file versus in the signed manifest. + pub record_count: u64, + pub signed_record_count: u64, + /// The public key the checkpoint claims to be signed by. + pub public_key: Option<[u8; 32]>, + /// Saves in the WAL after the checkpoint: not covered by the signature. + pub wal_entries_unsigned: usize, +} + +impl VerifyReport { + /// Signed by the trusted key, signature valid, and every part of the + /// file unchanged since it was signed. + pub fn is_valid(&self) -> bool { + self.signed + && self.key_matches + && self.signature_valid + && self.records_match + && self.settings_match + && self.sessions_match + && self.graph_match + } +} + +/// Check a store file against the public key the caller trusts. +/// +/// Reads the checkpoint (not the WAL), recomputes every hash from its +/// contents and checks the signature. Never writes. +pub fn verify_store( + path: &std::path::Path, + trusted: &VerifyingKey, +) -> Result { + let file = clawhdf5::File::open(path) + .map_err(|e| crate::MemoryError::Hdf5(format!("cannot open {}: {e}", path.display())))?; + let (config, cache, sessions, knowledge) = crate::schema::validate_and_load(&file)?; + let checkpoint = crate::schema::read_checkpoint_meta(&file); + let stored = crate::schema::read_signature(&file)?; + let wal_entries_unsigned = count_wal_entries_after(path, checkpoint.wal_applied); + + let (current, current_hashes) = manifest( + &config, + &cache, + &sessions, + &knowledge, + checkpoint.wal_applied, + ); + + let Some(stored) = stored else { + return Ok(VerifyReport { + signed: false, + key_matches: false, + signature_valid: false, + records_match: false, + settings_match: false, + sessions_match: false, + graph_match: false, + changed_records: Vec::new(), + record_count: current.record_count, + signed_record_count: 0, + public_key: None, + wal_entries_unsigned, + }); + }; + + let key_matches = stored.public_key == trusted.to_bytes(); + let signature_valid = trusted + .verify( + &stored.manifest.signed_bytes(), + &Signature::from_bytes(&stored.signature), + ) + .is_ok(); + // The stored per-record hashes can localise a change only if they are + // the ones that were signed. + let hashes_authentic = signature_valid + && stored.record_hashes.len() as u64 == stored.manifest.record_count + && merkle_root(&stored.record_hashes) == stored.manifest.records_root; + let changed_records = if hashes_authentic { + let n = current_hashes.len().max(stored.record_hashes.len()); + (0..n) + .filter(|&i| current_hashes.get(i) != stored.record_hashes.get(i)) + .collect() + } else { + Vec::new() + }; + + Ok(VerifyReport { + signed: true, + key_matches, + signature_valid, + records_match: signature_valid + && current.record_count == stored.manifest.record_count + && current.records_root == stored.manifest.records_root, + settings_match: signature_valid && current.settings == stored.manifest.settings, + sessions_match: signature_valid && current.sessions == stored.manifest.sessions, + graph_match: signature_valid && current.graph == stored.manifest.graph, + changed_records, + record_count: current.record_count, + signed_record_count: stored.manifest.record_count, + public_key: Some(stored.public_key), + wal_entries_unsigned, + }) +} + +fn count_wal_entries_after(store: &std::path::Path, mark: Option) -> usize { + let wal = store.with_extension("h5.wal"); + if !wal.exists() { + return 0; + } + crate::wal::WalFile::read_entries_for_migration(&wal, mark) + .map(|e| e.len()) + .unwrap_or(0) +} + +/// A new random signing key from the operating system's RNG. +pub fn generate_key() -> SigningKey { + SigningKey::generate(&mut rand_core::OsRng) +} + +/// Hex encoding for keys and signatures in attributes and the CLI. +pub fn to_hex(bytes: &[u8]) -> String { + bytes.iter().map(|b| format!("{b:02x}")).collect() +} + +/// Parse hex into exactly `N` bytes. +pub fn from_hex(s: &str) -> Option<[u8; N]> { + let s = s.trim(); + if s.len() != 2 * N { + return None; + } + let mut out = [0u8; N]; + for (i, byte) in out.iter_mut().enumerate() { + *byte = u8::from_str_radix(&s[2 * i..2 * i + 2], 16).ok()?; + } + Some(out) +} diff --git a/crates/clawhdf5-agent/src/storage.rs b/crates/clawhdf5-agent/src/storage.rs index 7ced239..8bb15e6 100644 --- a/crates/clawhdf5-agent/src/storage.rs +++ b/crates/clawhdf5-agent/src/storage.rs @@ -36,7 +36,7 @@ pub fn write_to_disk_with_mark( ) -> Result<(), MemoryError> { let meta = schema::CheckpointMeta { wal_applied, - ann_generation: None, + ..schema::CheckpointMeta::default() }; write_to_disk_with_meta(path, config, cache, sessions, knowledge, &meta) } @@ -50,7 +50,21 @@ pub fn write_to_disk_with_meta( knowledge: &KnowledgeCache, checkpoint: &schema::CheckpointMeta, ) -> Result<(), MemoryError> { - let bytes = schema::build_hdf5_file_with_meta(config, cache, sessions, knowledge, checkpoint)?; + write_to_disk_signed(path, config, cache, sessions, knowledge, checkpoint, None) +} + +/// [`write_to_disk_with_meta`] with a signed manifest of the contents. +pub fn write_to_disk_signed( + path: &Path, + config: &MemoryConfig, + cache: &MemoryCache, + sessions: &SessionCache, + knowledge: &KnowledgeCache, + checkpoint: &schema::CheckpointMeta, + signature: Option<&crate::signing::StoredSignature>, +) -> Result<(), MemoryError> { + let bytes = + schema::build_hdf5_file_signed(config, cache, sessions, knowledge, checkpoint, signature)?; if bytes.is_empty() { return Err(MemoryError::Hdf5("build_hdf5_file produced 0 bytes".into())); diff --git a/crates/clawhdf5-agent/tests/h5py_interop.rs b/crates/clawhdf5-agent/tests/h5py_interop.rs index d82ed46..54cfec7 100644 --- a/crates/clawhdf5-agent/tests/h5py_interop.rs +++ b/crates/clawhdf5-agent/tests/h5py_interop.rs @@ -92,3 +92,64 @@ print(len(names)) assert!(n >= 10, "only {n} datasets"); } } + +#[test] +fn an_edit_made_with_h5py_breaks_the_signature_and_names_the_record() { + if !h5py_available() { + assert!( + std::env::var("CLAWHDF5_REQUIRE_INTEROP").as_deref() != Ok("1"), + "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available" + ); + eprintln!("SKIP: python3 with h5py not available"); + return; + } + use clawhdf5_agent::signing::SigningKey; + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("signed.h5"); + let key = SigningKey::from_bytes(&[42; 32]); + let mut m = HDF5Memory::create(MemoryConfig::new(path.clone(), "agent", 8)).unwrap(); + m.set_signing_key(key.clone()); + m.save_batch( + (0..10) + .map(|i| MemoryEntry { + chunk: format!("memory {i}"), + embedding: (0..8).map(|j| ((i * 8 + j) as f32).cos()).collect(), + source_channel: "test".into(), + timestamp: i as f64, + session_id: "s".into(), + tags: String::new(), + }) + .collect(), + ) + .unwrap(); + drop(m); + assert!( + HDF5Memory::verify(&path, &key.verifying_key()) + .unwrap() + .is_valid() + ); + + // Someone edits one timestamp in place with h5py. + let script = format!( + r#" +import h5py +with h5py.File("{}", "r+") as f: + ts = f["memory/timestamps"] + ts[3] = 12345.0 +"#, + path.display() + ); + let out = Command::new(python()) + .args(["-c", &script]) + .output() + .unwrap(); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + + let r = HDF5Memory::verify(&path, &key.verifying_key()).unwrap(); + assert!(r.signature_valid && !r.is_valid(), "{r:?}"); + assert_eq!(r.changed_records, vec![3]); +} diff --git a/crates/clawhdf5-agent/tests/signed_store.rs b/crates/clawhdf5-agent/tests/signed_store.rs new file mode 100644 index 0000000..7f7c078 --- /dev/null +++ b/crates/clawhdf5-agent/tests/signed_store.rs @@ -0,0 +1,330 @@ +//! Ed25519-signed checkpoints: `HDF5Memory::set_signing_key` and +//! `HDF5Memory::verify`. + +use std::path::Path; + +use clawhdf5_agent::signing::{SigningKey, VerifyReport, VerifyingKey}; +use clawhdf5_agent::storage; +use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, MemoryError, schema}; +use tempfile::TempDir; + +const DIM: usize = 16; + +fn key(seed: u8) -> SigningKey { + SigningKey::from_bytes(&[seed; 32]) +} + +fn entry(i: usize, chunk: &str) -> MemoryEntry { + MemoryEntry { + chunk: chunk.to_string(), + embedding: (0..DIM) + .map(|j| ((i * DIM + j) as f32 * 0.37).sin()) + .collect(), + source_channel: "chat".into(), + timestamp: 1_700_000_000.0 + i as f64, + session_id: format!("s{}", i % 3), + tags: format!("t{i}"), + } +} + +/// Awkward strings on purpose: they must hash the same after a round trip. +const TEXTS: [&str; 6] = [ + "plain text", + "ünïcödé — 日本語 🙂", + "", + "trailing spaces ", + "tab\tand\nnewline", + "x", +]; + +fn signed_store(dir: &TempDir, float16: bool, k: &SigningKey) -> std::path::PathBuf { + let mut cfg = MemoryConfig::new(dir.path().join("s.h5"), "agent", DIM); + cfg.float16 = float16; + let path = cfg.path.clone(); + let mut m = HDF5Memory::create(cfg).unwrap(); + m.set_signing_key(k.clone()); + let entries = (0..30).map(|i| entry(i, TEXTS[i % TEXTS.len()])).collect(); + m.save_batch(entries).unwrap(); + // Some graph and a deleted record, so every part of the manifest is used. + let a = m.knowledge_mut().add_entity("Alice", "person", 0); + let b = m.knowledge_mut().add_entity("Acme", "org", -1); + m.knowledge_mut().add_relation(a, b, "works_at", 0.75); + m.sessions_mut() + .add_at("s0", 0, 9, "chat", "first session", 1_700_000_000.0); + m.delete(4).unwrap(); + m.flush_wal().unwrap(); + path +} + +fn verify(path: &Path, k: &SigningKey) -> VerifyReport { + HDF5Memory::verify(path, &k.verifying_key()).unwrap() +} + +#[test] +fn a_signed_store_verifies_through_reopen_and_checkpoint_cycles() { + for float16 in [true, false] { + let dir = TempDir::new().unwrap(); + let k = key(7); + let path = signed_store(&dir, float16, &k); + let r = verify(&path, &k); + assert!(r.is_valid(), "float16={float16}: {r:?}"); + assert_eq!(r.public_key, Some(k.verifying_key().to_bytes())); + assert_eq!(r.record_count, 30); + assert!(r.changed_records.is_empty()); + + // Reopen, change nothing, checkpoint again (with the key): still valid. + for _ in 0..3 { + let mut m = HDF5Memory::open(&path).unwrap(); + assert!(m.is_signed()); + m.set_signing_key(k.clone()); + m.flush_wal().unwrap(); + drop(m); + assert!(verify(&path, &k).is_valid()); + } + // And after real changes, re-signed. + let mut m = HDF5Memory::open(&path).unwrap(); + m.set_signing_key(k.clone()); + m.save(entry(99, "added later")).unwrap(); + m.hybrid_search(&entry(1, "").embedding, "text", 0.4, 0.6, 5); + m.flush_wal().unwrap(); + drop(m); + let r = verify(&path, &k); + assert!(r.is_valid(), "{r:?}"); + assert_eq!(r.record_count, 31); + } +} + +#[test] +fn a_signed_store_refuses_to_checkpoint_without_its_key() { + let dir = TempDir::new().unwrap(); + let k = key(1); + let path = signed_store(&dir, true, &k); + + let mut m = HDF5Memory::open(&path).unwrap(); + m.save(entry(50, "pending")).unwrap(); + match m.flush_wal() { + Err(MemoryError::SigningKeyRequired(msg)) => assert!(msg.contains("signed"), "{msg}"), + other => panic!("expected SigningKeyRequired, got {other:?}"), + } + // The file is untouched and still valid; the save is still in the WAL. + let r = verify(&path, &k); + assert!(r.is_valid()); + assert_eq!(r.wal_entries_unsigned, 1); + + // Supplying the key lets the checkpoint through, signed. + m.set_signing_key(k.clone()); + m.flush_wal().unwrap(); + drop(m); + let r = verify(&path, &k); + assert!(r.is_valid()); + assert_eq!((r.record_count, r.wal_entries_unsigned), (31, 0)); + + // Removing the signature on purpose writes it unsigned. + let mut m = HDF5Memory::open(&path).unwrap(); + m.remove_signature(); + m.flush_wal().unwrap(); + drop(m); + let r = verify(&path, &k); + assert!(!r.signed && !r.is_valid()); + assert!(!HDF5Memory::open(&path).unwrap().is_signed()); +} + +#[test] +fn the_wrong_key_does_not_verify_and_a_new_key_re_signs() { + let dir = TempDir::new().unwrap(); + let (a, b) = (key(1), key(2)); + let path = signed_store(&dir, true, &a); + let r = verify(&path, &b); + assert!(r.signed && !r.key_matches && !r.signature_valid && !r.is_valid()); + + let mut m = HDF5Memory::open(&path).unwrap(); + m.set_signing_key(b.clone()); + m.flush_wal().unwrap(); + drop(m); + assert!(verify(&path, &b).is_valid()); + assert!(!verify(&path, &a).is_valid()); +} + +/// Rewrite the store with changed contents but the *old* signature — what +/// someone with write access to the file, but not the key, can do. +fn tamper(path: &Path, change: impl FnOnce(&mut Tampered)) { + let file = clawhdf5::File::open(path).unwrap(); + let (config, cache, sessions, knowledge) = schema::validate_and_load(&file).unwrap(); + let checkpoint = schema::read_checkpoint_meta(&file); + let signature = schema::read_signature(&file).unwrap().unwrap(); + drop(file); + let mut t = Tampered { + config, + cache, + sessions, + knowledge, + }; + change(&mut t); + storage::write_to_disk_signed( + path, + &t.config, + &t.cache, + &t.sessions, + &t.knowledge, + &checkpoint, + Some(&signature), + ) + .unwrap(); +} + +struct Tampered { + config: MemoryConfig, + cache: clawhdf5_agent::cache::MemoryCache, + sessions: clawhdf5_agent::SessionCache, + knowledge: clawhdf5_agent::knowledge::KnowledgeCache, +} + +#[test] +fn every_kind_of_edit_is_detected_and_located() { + let k = key(3); + type Edit = Box; + type Case = (&'static str, Edit, fn(&VerifyReport) -> bool); + let cases: Vec = vec![ + ( + "record text", + Box::new(|t: &mut Tampered| t.cache.chunks[7] = "rewritten".into()), + |r| !r.records_match && r.changed_records == vec![7], + ), + ( + "one embedding value", + Box::new(|t: &mut Tampered| { + let mut e = t.cache.embeddings[12].to_vec(); + e[3] = 0.5; + t.cache.embeddings.set(12, &e); + }), + |r| r.changed_records == vec![12], + ), + ( + "undelete", + Box::new(|t: &mut Tampered| t.cache.tombstones[4] = 0), + |r| r.changed_records == vec![4], + ), + ( + "timestamp", + Box::new(|t: &mut Tampered| t.cache.timestamps[20] += 1.0), + |r| r.changed_records == vec![20], + ), + ( + "record appended", + Box::new(|t: &mut Tampered| { + t.cache.push( + "new".into(), + vec![0.1; DIM], + "x".into(), + 1.0, + "s".into(), + "".into(), + ); + }), + |r| !r.records_match && r.changed_records == vec![30] && r.record_count == 31, + ), + ( + "setting", + Box::new(|t: &mut Tampered| t.config.agent_id = "someone-else".into()), + |r| !r.settings_match && r.records_match, + ), + ( + "session summary", + Box::new(|t: &mut Tampered| t.sessions.summaries[0] = "edited".into()), + |r| !r.sessions_match && r.records_match, + ), + ( + "graph edge", + Box::new(|t: &mut Tampered| t.knowledge.relations[0].weight = 1.0), + |r| !r.graph_match && r.records_match, + ), + ]; + for (name, edit, check) in cases { + let dir = TempDir::new().unwrap(); + let path = signed_store(&dir, true, &k); + tamper(&path, edit); + let r = verify(&path, &k); + assert!( + r.signed && r.key_matches && r.signature_valid, + "{name}: {r:?}" + ); + assert!(!r.is_valid(), "{name}: edit not detected: {r:?}"); + assert!(check(&r), "{name}: {r:?}"); + } +} + +#[test] +fn a_forged_manifest_fails_the_signature() { + // Recomputing the hashes for tampered contents does not help without the + // key: the signature no longer matches the manifest. + let dir = TempDir::new().unwrap(); + let k = key(5); + let path = signed_store(&dir, true, &k); + let file = clawhdf5::File::open(&path).unwrap(); + let (config, mut cache, sessions, knowledge) = schema::validate_and_load(&file).unwrap(); + let checkpoint = schema::read_checkpoint_meta(&file); + let mut sig = schema::read_signature(&file).unwrap().unwrap(); + drop(file); + cache.chunks[0] = "forged".into(); + // Re-sign with an attacker key, then splice the victim's public key back. + let forged = clawhdf5_agent::signing::sign( + &key(66), + &config, + &cache, + &sessions, + &knowledge, + checkpoint.wal_applied, + ); + sig.manifest = forged.manifest; + sig.record_hashes = forged.record_hashes; + storage::write_to_disk_signed( + &path, + &config, + &cache, + &sessions, + &knowledge, + &checkpoint, + Some(&sig), + ) + .unwrap(); + let r = verify(&path, &k); + assert!( + r.key_matches && !r.signature_valid && !r.is_valid(), + "{r:?}" + ); +} + +#[test] +fn an_unsigned_store_reports_unsigned() { + let dir = TempDir::new().unwrap(); + let mut m = HDF5Memory::create(MemoryConfig::new(dir.path().join("u.h5"), "a", DIM)).unwrap(); + m.save_batch(vec![entry(0, "hello")]).unwrap(); + drop(m); + let r = HDF5Memory::verify(&dir.path().join("u.h5"), &VerifyingKey::from(&key(1))).unwrap(); + assert!(!r.signed && !r.is_valid()); + assert_eq!(r.record_count, 1); +} + +#[test] +fn nul_bytes_in_text_still_verify() { + // Strings are stored null-padded; the hash must follow what a reopened + // store actually holds, or an untouched store would fail to verify. + let dir = TempDir::new().unwrap(); + let k = key(9); + let mut m = HDF5Memory::create(MemoryConfig::new(dir.path().join("n.h5"), "a", DIM)).unwrap(); + m.set_signing_key(k.clone()); + m.save_batch(vec![ + entry(0, "inner\0nul"), + entry(1, "trailing nul\0"), + entry(2, "\0leading"), + ]) + .unwrap(); + drop(m); + let r = verify(&dir.path().join("n.h5"), &k); + assert!(r.is_valid(), "{r:?}"); + let m = HDF5Memory::open(&dir.path().join("n.h5")).unwrap(); + eprintln!( + "reloaded: {:?}", + (0..3).map(|i| m.get_chunk(i)).collect::>() + ); +} diff --git a/crates/clawhdf5-bench/src/bin/search_harness.rs b/crates/clawhdf5-bench/src/bin/search_harness.rs index 920a44a..25d4023 100644 --- a/crates/clawhdf5-bench/src/bin/search_harness.rs +++ b/crates/clawhdf5-bench/src/bin/search_harness.rs @@ -21,6 +21,7 @@ //! cargo run --release -p clawhdf5-bench --bin search_harness -- --ann-only --uniform //! cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full //! cargo run --release -p clawhdf5-bench --bin search_harness -- --options-study --full +//! cargo run --release -p clawhdf5-bench --bin search_harness -- --signing-study --full //! ``` use std::time::{Duration, Instant}; @@ -488,6 +489,81 @@ fn bench_end_to_end(n: usize, json: &mut Vec) { })); } +// --------------------------------------------------------------------------- +// Signing study: what does an Ed25519-signed checkpoint cost? +// --------------------------------------------------------------------------- + +/// `--signing-study`: checkpoint time unsigned vs signed, `verify` time, and +/// the file-size cost of the stored per-record hashes. Default store +/// settings (float16, int8 index). Medians of five checkpoints / three +/// verifies. +fn signing_study(n: usize) { + use clawhdf5_agent::signing::SigningKey; + let data = make_dataset(n, 0x516 ^ n as u64); + let mut rng = Rng(9); + let entries: Vec = data + .vectors + .iter() + .enumerate() + .map(|(i, v)| MemoryEntry { + chunk: text_for(data.cluster_of[i], i, &mut rng), + embedding: v.clone(), + source_channel: "bench".into(), + timestamp: i as f64, + session_id: format!("s{}", i % 50), + tags: format!("t{i}"), + }) + .collect(); + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("sign.h5"); + let mut mem = HDF5Memory::create(MemoryConfig::new(path.clone(), "bench", DIM)).unwrap(); + mem.save_batch(entries).unwrap(); + std::hint::black_box(mem.hybrid_search(&data.queries[0], "", 1.0, 0.0, K)); + + let median = |mut v: Vec| { + v.sort(); + v[v.len() / 2] + }; + let checkpoint = |mem: &mut HDF5Memory| { + median( + (0..5) + .map(|_| { + let t = Instant::now(); + mem.flush_wal().unwrap(); + t.elapsed() + }) + .collect(), + ) + }; + let unsigned = checkpoint(&mut mem); + let unsigned_bytes = std::fs::metadata(&path).unwrap().len(); + let key = SigningKey::from_bytes(&[7; 32]); + mem.set_signing_key(key.clone()); + let signed = checkpoint(&mut mem); + let signed_bytes = std::fs::metadata(&path).unwrap().len(); + drop(mem); + let vk = key.verifying_key(); + let verify = median( + (0..3) + .map(|_| { + let t = Instant::now(); + let r = HDF5Memory::verify(&path, &vk).unwrap(); + let d = t.elapsed(); + assert!(r.is_valid()); + d + }) + .collect(), + ); + println!( + "| {n} | {:.1} | {:.1} | {:+.1} | {:.1} | {:+.2} |", + millis(unsigned), + millis(signed), + millis(signed) - millis(unsigned), + millis(verify), + (signed_bytes as f64 - unsigned_bytes as f64) / (1024.0 * 1024.0), + ); +} + // --------------------------------------------------------------------------- // Search options study: source filters, re-ranking, confidence rejection // --------------------------------------------------------------------------- @@ -960,6 +1036,21 @@ fn main() { } return; } + if args.iter().any(|a| a == "--signing-study") { + println!("## Signed checkpoints ({DIM}-dim, float16, int8 index)\n"); + println!( + "| N | checkpoint ms, unsigned | checkpoint ms, signed | signing adds ms | verify ms | file MiB added |" + ); + println!("|---:|---:|---:|---:|---:|---:|"); + for &n in if full { + &[1_000, 10_000, 100_000][..] + } else { + &[1_000, 10_000][..] + } { + signing_study(n); + } + return; + } if args.iter().any(|a| a == "--options-study") { println!("## Search options ({DIM}-dim, k = {K}, Hebbian boost off)\n"); println!("| N | options | filtered recall@10 | p50 ms | p99 ms |"); diff --git a/crates/clawhdf5-cli/src/main.rs b/crates/clawhdf5-cli/src/main.rs index 2fc07c0..e728a23 100644 --- a/crates/clawhdf5-cli/src/main.rs +++ b/crates/clawhdf5-cli/src/main.rs @@ -1,15 +1,22 @@ -use std::path::PathBuf; +use std::path::{Path, PathBuf}; use clap::{Parser, Subcommand}; +use clawhdf5_agent::signing::{self, SigningKey, VerifyingKey}; use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry}; /// ClawhDF5 — HDF5-backed cognitive memory for AI agents #[derive(Parser)] #[command(name = "clawhdf5", version, about)] struct Cli { - /// Path to the .h5 memory file + /// Path to the .h5 memory file (not needed for `keygen`) #[arg(short, long, env = "CLAWHDF5_PATH")] - path: PathBuf, + path: Option, + + /// File holding an Ed25519 signing key (64 hex characters, from + /// `keygen`). Every checkpoint this command makes is then signed; a + /// signed store refuses to checkpoint without it. + #[arg(long, env = "CLAWHDF5_SIGNING_KEY", global = true)] + signing_key: Option, #[command(subcommand)] command: Commands, @@ -91,6 +98,38 @@ enum Commands { /// Destination path dest: PathBuf, }, + /// Generate an Ed25519 signing key for signed checkpoints + Keygen { + /// Where to write the secret key (created new, owner-only on Unix) + #[arg(long)] + out: PathBuf, + }, + /// Verify a signed store against a public key; exit status 2 if not valid + Verify { + /// The trusted public key: 64 hex characters, or a file holding them + #[arg(long)] + public_key: String, + }, +} + +fn read_signing_key(path: &Path) -> Result> { + let text = std::fs::read_to_string(path) + .map_err(|e| format!("cannot read signing key {}: {e}", path.display()))?; + let bytes = signing::from_hex::<32>(&text) + .ok_or_else(|| format!("{} is not a 64-hex-character key", path.display()))?; + Ok(SigningKey::from_bytes(&bytes)) +} + +/// Open for writing, with the signing key applied if one was given. +fn open_writable( + path: &Path, + key: &Option, +) -> Result> { + let mut mem = HDF5Memory::open(path)?; + if let Some(k) = key { + mem.set_signing_key(k.clone()); + } + Ok(mem) } fn main() { @@ -103,6 +142,37 @@ fn main() { } fn run(cli: Cli) -> Result<(), Box> { + if let Commands::Keygen { out } = &cli.command { + let key = signing::generate_key(); + let mut opts = std::fs::OpenOptions::new(); + opts.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt; + opts.mode(0o600); + } + use std::io::Write; + let mut f = opts + .open(out) + .map_err(|e| format!("cannot create {}: {e}", out.display()))?; + writeln!(f, "{}", signing::to_hex(&key.to_bytes()))?; + let j = serde_json::json!({ + "status": "generated", + "secret_key_file": out.display().to_string(), + "public_key": signing::to_hex(&key.verifying_key().to_bytes()), + }); + println!("{}", serde_json::to_string_pretty(&j)?); + return Ok(()); + } + let path = cli + .path + .clone() + .ok_or("--path (or CLAWHDF5_PATH) is required")?; + let key = cli + .signing_key + .as_deref() + .map(read_signing_key) + .transpose()?; match cli.command { Commands::Create { agent_id, @@ -113,7 +183,7 @@ fn run(cli: Cli) -> Result<(), Box> { f32, float16: _, } => { - let mut config = MemoryConfig::new(cli.path.clone(), &agent_id, dim); + let mut config = MemoryConfig::new(path.clone(), &agent_id, dim); config.wal_enabled = wal; // As with --f32-index: only ever switch the library default off. if f32 { @@ -127,15 +197,21 @@ fn run(cli: Cli) -> Result<(), Box> { config.quantized_index = false; } let config_quantized = config.quantized_index; - let mem = HDF5Memory::create(config)?; + let mut mem = HDF5Memory::create(config)?; + // Sign straight away, so the store is never on disk unsigned. + if let Some(k) = &key { + mem.set_signing_key(k.clone()); + mem.flush_wal()?; + } let j = serde_json::json!({ "status": "created", - "path": cli.path.display().to_string(), + "path": path.display().to_string(), "agent_id": agent_id, "embedding_dim": dim, "wal_enabled": wal, "quantized_index": config_quantized, "float16": config_float16, + "signed": mem.is_signed(), "count": mem.count(), }); println!("{}", serde_json::to_string_pretty(&j)?); @@ -152,7 +228,7 @@ fn run(cli: Cli) -> Result<(), Box> { } }; let entry: MemoryEntry = serde_json::from_str(&input)?; - let mut mem = HDF5Memory::open(&cli.path)?; + let mut mem = open_writable(&path, &key)?; let idx = mem.save(entry)?; let j = serde_json::json!({ "status": "saved", "index": idx, "count": mem.count() }); println!("{}", serde_json::to_string(&j)?); @@ -166,7 +242,7 @@ fn run(cli: Cli) -> Result<(), Box> { keyword_weight, } => { let emb: Vec = serde_json::from_str(&embedding)?; - let mut mem = HDF5Memory::open(&cli.path)?; + let mut mem = open_writable(&path, &key)?; let results = mem.hybrid_search(&emb, &query, vector_weight, keyword_weight, top_k); let j: Vec = results .iter() @@ -184,7 +260,7 @@ fn run(cli: Cli) -> Result<(), Box> { } Commands::Recall { index } => { - let mem = HDF5Memory::open_read_only(&cli.path)?; + let mem = HDF5Memory::open_read_only(&path)?; match mem.get_chunk(index) { Some(content) => { let j = serde_json::json!({ "index": index, "chunk": content }); @@ -198,22 +274,23 @@ fn run(cli: Cli) -> Result<(), Box> { } Commands::Stats => { - let mem = HDF5Memory::open_read_only(&cli.path)?; + let mem = HDF5Memory::open_read_only(&path)?; let cfg = mem.config(); let j = serde_json::json!({ - "path": cli.path.display().to_string(), + "path": path.display().to_string(), "agent_id": cfg.agent_id, "embedding_dim": cfg.embedding_dim, "count": mem.count(), "active": mem.count_active(), "wal_enabled": cfg.wal_enabled, "wal_pending": mem.wal_pending_count(), + "signed": mem.is_signed(), }); println!("{}", serde_json::to_string_pretty(&j)?); } Commands::FlushWal => { - let mut mem = HDF5Memory::open(&cli.path)?; + let mut mem = open_writable(&path, &key)?; let before = mem.wal_pending_count(); mem.flush_wal()?; let j = serde_json::json!({ @@ -225,7 +302,7 @@ fn run(cli: Cli) -> Result<(), Box> { } Commands::AgentsMd { output } => { - let mem = HDF5Memory::open_read_only(&cli.path)?; + let mem = HDF5Memory::open_read_only(&path)?; let md = mem.generate_agents_md(); match output { Some(p) => { @@ -237,7 +314,7 @@ fn run(cli: Cli) -> Result<(), Box> { } Commands::Export => { - let mem = HDF5Memory::open_read_only(&cli.path)?; + let mem = HDF5Memory::open_read_only(&path)?; for i in 0..mem.count() { if let Some(chunk) = mem.get_chunk(i) { let j = serde_json::json!({ "index": i, "chunk": chunk }); @@ -246,11 +323,44 @@ fn run(cli: Cli) -> Result<(), Box> { } } + Commands::Keygen { .. } => unreachable!("handled before opening a store"), + + Commands::Verify { public_key } => { + let text = if Path::new(&public_key).is_file() { + std::fs::read_to_string(&public_key)? + } else { + public_key + }; + let bytes = signing::from_hex::<32>(&text) + .ok_or("--public-key must be 64 hex characters or a file holding them")?; + let trusted = VerifyingKey::from_bytes(&bytes)?; + let r = HDF5Memory::verify(&path, &trusted)?; + let j = serde_json::json!({ + "valid": r.is_valid(), + "signed": r.signed, + "key_matches": r.key_matches, + "signature_valid": r.signature_valid, + "records_match": r.records_match, + "settings_match": r.settings_match, + "sessions_match": r.sessions_match, + "graph_match": r.graph_match, + "changed_records": r.changed_records, + "record_count": r.record_count, + "signed_record_count": r.signed_record_count, + "signed_by": r.public_key.map(|k| signing::to_hex(&k)), + "wal_entries_unsigned": r.wal_entries_unsigned, + }); + println!("{}", serde_json::to_string_pretty(&j)?); + if !r.is_valid() { + std::process::exit(2); + } + } + Commands::Snapshot { dest } => { - let _result = clawhdf5_agent::storage::snapshot_file(&cli.path, &dest)?; + let _result = clawhdf5_agent::storage::snapshot_file(&path, &dest)?; let j = serde_json::json!({ "status": "snapshot_created", - "source": cli.path.display().to_string(), + "source": path.display().to_string(), "dest": dest.display().to_string(), }); println!("{}", serde_json::to_string(&j)?); diff --git a/docs/QUICKSTART.md b/docs/QUICKSTART.md index 0635ce0..e8acf84 100644 --- a/docs/QUICKSTART.md +++ b/docs/QUICKSTART.md @@ -27,7 +27,7 @@ The core use case. Give your AI agent persistent, searchable memory in a single ```toml # Cargo.toml [dependencies] -clawhdf5-agent = { version = "2.0", features = ["agent"] } +clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } # not on crates.io yet ``` ### Create a Memory Store diff --git a/docs/USE_CASES.md b/docs/USE_CASES.md index 4d0cb4a..132c6f1 100644 --- a/docs/USE_CASES.md +++ b/docs/USE_CASES.md @@ -212,7 +212,7 @@ This is the container image for intelligence. | Your Situation | Features to Enable | Why | |----------------|-------------------|-----| | **Quick prototype** | Default | Vector search works out of the box | -| **Production agent** | `agent`, `float16`, `parallel` | Half-precision saves 50% storage, parallel search for scale | +| **Production agent** | defaults (`float16`, `hnsw`, `parallel`) | HNSW search and a parallel index build; half-precision *storage* is `MemoryConfig::float16`, on by default for new stores | | **macOS** | + `accelerate` | Apple AMX coprocessor for matrix ops | | **Linux server** | + `openblas` or `fast-math` | BLAS acceleration | | **GPU available** | + `gpu` | wgpu-based search, wins at 100K+ scale | @@ -220,14 +220,15 @@ This is the container image for intelligence. | **Edge device** | Default only | Minimal dependencies, smallest binary | ```toml +# Not on crates.io yet: depend on the repository. # Production agent on Linux -clawhdf5-agent = { version = "2.0", features = ["agent", "float16", "parallel", "fast-math"] } +clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5", features = ["fast-math"] } # Edge device -clawhdf5-agent = { version = "2.0", features = ["agent"] } +clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } # macOS with GPU -clawhdf5-agent = { version = "2.0", features = ["agent", "float16", "accelerate", "gpu", "async"] } +clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5", features = ["accelerate", "gpu", "async"] } ``` ---