Ed25519-signed checkpoints; remove the no-op agent feature #8

Merged
osobh merged 2 commits from feat/signed-checkpoints into main 2026-09-25 15:58:19 +00:00
15 changed files with 1334 additions and 29 deletions
+24
View File
@@ -243,6 +243,30 @@ index asked for ~16 000 candidates, where scanning the few hundred or thousand
allowed records is exact and cheap. Re-ranking a 3k candidate pool and allowed records is exact and cheap. Re-ranking a 3k candidate pool and
confidence rejection add about 3%. confidence rejection add about 3%.
### Signed checkpoints
Measured 2026-09-25 on tank (AMD Ryzen 7 7800X3D). A default store (float16,
int8 index), 384-dim; each checkpoint rewrites the whole file, as every
checkpoint does. Medians of five checkpoints and three verifies; three runs
agreed to within the ranges shown.
```bash
cargo run --release -p clawhdf5-bench --bin search_harness -- --signing-study --full
```
| N | checkpoint, unsigned | checkpoint, signed | signing adds | `verify` | file size added |
|---:|---:|---:|---:|---:|---:|
| 1 000 | 5.4 ms | 6.4 ms | 0.7–1.0 ms | 2.1 ms | 0.03 MiB |
| 10 000 | 46 ms | 55 ms | 8.1–9.4 ms | 18.6 ms | 0.31 MiB |
| 100 000 | 495 ms | 598 ms | 89–112 ms | 247 ms | 3.05 MiB |
Signing costs about 20% of a checkpoint: every record is rehashed (SHA-256)
and the Merkle root recomputed each time; the Ed25519 signature itself is
microseconds. Caching per-record hashes between checkpoints would cut this to
the records that changed. The per-record hashes stored for locating edits are
32 bytes each (4% of a 100K float16 store). `verify` reads and rehashes the
whole checkpoint.
### float16 embedding storage (`MemoryConfig::float16`) ### float16 embedding storage (`MemoryConfig::float16`)
Measured 2026-09-23 on tank (AMD Ryzen 7 7800X3D). The same clustered Measured 2026-09-23 on tank (AMD Ryzen 7 7800X3D). The same clustered
+29
View File
@@ -3,6 +3,12 @@
## Unreleased ## Unreleased
### Upgrade Notes ### Upgrade Notes
- **Breaking:** `MemoryError` is now `#[non_exhaustive]` and gained
`SigningKeyRequired`; a `match` on it needs a wildcard arm. Future variants
will no longer be breaking.
- **Breaking:** `clawhdf5-agent`'s `agent` feature is removed. It enabled
nothing — the agent layer is always built — but the README and guides told
people to pass it; drop `agent` from `features = [...]`.
- **`clawhdf5-migrate` now writes a real agent store.** Its output used to be - **`clawhdf5-migrate` now writes a real agent store.** Its output used to be
a layout of its own (`/chunks`, `/sessions`, `/entities`, `/relations`, no a layout of its own (`/chunks`, `/sessions`, `/entities`, `/relations`, no
`/meta`) that `HDF5Memory::open` rejected, so a migrated file could not be `/meta`) that `HDF5Memory::open` rejected, so a migrated file could not be
@@ -57,6 +63,29 @@
`quantized_index = false`, or pass `create --f32-index` to the CLI, to opt `quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
out. The CLI's `--quantized-index` is still accepted but is now a no-op. out. The CLI's `--quantized-index` is still accepted but is now a no-op.
### Signing
- `clawhdf5-agent`: **Ed25519-signed checkpoints** — the README's
"cryptographically verifiable memory", now true. With
`HDF5Memory::set_signing_key(key)`, every checkpoint stores a signed
manifest: a SHA-256 per record (text, embedding as stored, channel,
timestamp, session, tags, deleted flag, activation) in a Merkle tree, plus
hashes of the settings (and WAL mark), sessions and knowledge graph, with
the per-record hashes in `/integrity/record_hashes`.
`HDF5Memory::verify(path, &public_key)` recomputes everything from the file
and reports which part changed and which records (`changed_records`); a
forged manifest fails the signature. The key is never persisted; a signed
store refuses to checkpoint without it (`MemoryError::SigningKeyRequired`),
and `remove_signature()` is the deliberate way back to unsigned. Saves still
in the WAL are not covered (`wal_entries_unsigned`). Tests include every
kind of edit, and an edit made with h5py in place, which verify pinpoints.
Cost: ~20% of a checkpoint, 32 bytes per record (`BENCHMARKS.md`, "Signed
checkpoints"). New dependencies `ed25519-dalek`, `sha2`, `rand_core` — pure
Rust; the no-C check still passes.
- `clawhdf5-cli`: `keygen --out <file>` (owner-only key file),
`--signing-key <file>` / `CLAWHDF5_SIGNING_KEY` on writing commands
(`create` signs immediately), `verify --public-key <hex|file>` (JSON report;
exit status 2 if not valid), and `signed` in `create`/`stats` output.
### Migration ### Migration
- `clawhdf5-migrate`: writes through the agent's own API (`HDF5Memory::create` - `clawhdf5-migrate`: writes through the agent's own API (`HDF5Memory::create`
/ `open`, `save_batch`, the session cache and knowledge graph), so there is / `open`, `save_batch`, the session cache and knowledge graph), so there is
+11
View File
@@ -109,6 +109,17 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
`search_harness --options-study`. `search_harness --options-study`.
- `MemoryConfig::compression` is off by default; when on, embeddings are - `MemoryConfig::compression` is off by default; when on, embeddings are
deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd). deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd).
- Signed checkpoints (`clawhdf5-agent` `signing` module): with
`HDF5Memory::set_signing_key` every checkpoint stores an Ed25519-signed
manifest (SHA-256 per record in a Merkle tree + settings/sessions/graph
hashes; per-record hashes in `/integrity/record_hashes`);
`HDF5Memory::verify(path, &pk)` locates edits. The hashes must cover exactly
what the file persists in the form the loader returns it (strings lose
trailing NULs; an empty WAL mark is not written) or untouched stores stop
verifying — `tests/signed_store.rs` round-trips awkward strings. The key is
never persisted; a signed store refuses to checkpoint without it
(`MemoryError::SigningKeyRequired`, and `MemoryError` is `#[non_exhaustive]`).
WAL entries after the checkpoint are not covered.
- `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by - `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by
default) recomputes a dataset's SHA-256 and compares it against the default) recomputes a dataset's SHA-256 and compares it against the
`_provenance_sha256` attribute written automatically on save when `_provenance_sha256` attribute written automatically on save when
+32 -3
View File
@@ -8,7 +8,7 @@
[![LongMemEval](https://img.shields.io/badge/LongMemEval__s-Turn--Level%20Hit@5%2081.4%25%20hybrid-blue.svg)](BENCHMARKS.md#longmemeval-results) [![LongMemEval](https://img.shields.io/badge/LongMemEval__s-Turn--Level%20Hit@5%2081.4%25%20hybrid-blue.svg)](BENCHMARKS.md#longmemeval-results)
[![Footprint](https://img.shields.io/badge/on--disk-~820%20B%2Frecord%20float16%2C%20synthetic%20text-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1) [![Footprint](https://img.shields.io/badge/on--disk-~820%20B%2Frecord%20float16%2C%20synthetic%20text-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1)
ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file. ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, cryptographically verifiable memory (Ed25519-signed checkpoints) — all stored in a single portable file.
> **Two things live here:** > **Two things live here:**
> - **A general-purpose, pure-Rust HDF5 library** — zero C dependencies, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers. > - **A general-purpose, pure-Rust HDF5 library** — zero C dependencies, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers.
@@ -113,7 +113,7 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d
| Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers | | Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers |
| Temporal queries | Custom code | Native temporal index (622 ns range query over 10K) | | Temporal queries | Custom code | Native temporal index (622 ns range query over 10K) |
| Multi-modal | Multiple stores | Unified cross-modal search (exact scan: 842 µs over 1K records) | | Multi-modal | Multiple stores | Unified cross-modal search (exact scan: 842 µs over 1K records) |
| Integrity | Hope for the best | Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance | | Integrity | Hope for the best | Ed25519-signed checkpoints that pinpoint any edited record, chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts |
| Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** | | Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** |
--- ---
@@ -369,6 +369,7 @@ directly; the store persists the records, sessions and graph they work over.
| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches. Opt-in via `SearchOptions::with_confidence`; on in the OpenClaw backend | | **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches. Opt-in via `SearchOptions::with_confidence`; on in the OpenClaw backend |
| **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints | | **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints |
| **`multimodal`** | Cross-modal search across text/image/audio/video embeddings | | **`multimodal`** | Cross-modal search across text/image/audio/video embeddings |
| **`signing`** | Ed25519-signed checkpoints: SHA-256 per record in a Merkle tree, plus hashes of settings, sessions and the knowledge graph; `HDF5Memory::verify` names any edited record |
| **`provenance`** | Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof) | | **`provenance`** | Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof) |
| **`anomaly`** | Write rate limiting, 15 injection-pattern detectors, source-distribution analysis. Alerts never block a save; drain them with `take_anomaly_alerts` | | **`anomaly`** | Write rate limiting, 15 injection-pattern detectors, source-distribution analysis. Alerts never block a save; drain them with `take_anomaly_alerts` |
| **`openclaw`** | OpenClaw integration: MemoryBackend trait, Markdown ↔ HDF5 conversion | | **`openclaw`** | OpenClaw integration: MemoryBackend trait, Markdown ↔ HDF5 conversion |
@@ -457,6 +458,35 @@ let careful = memory.search(
); );
``` ```
### Signed Checkpoints
```rust
use clawhdf5_agent::signing;
// Once, somewhere safe: keep the secret key, publish the public key.
let key = signing::generate_key();
let public = key.verifying_key();
// Every checkpoint is signed from now on. The key is never written to disk;
// a signed store refuses to checkpoint without it.
memory.set_signing_key(key);
memory.flush_wal()?;
// Anyone holding the public key can check the file, e.g. after copying it.
let report = HDF5Memory::verify(std::path::Path::new("agent.h5"), &public)?;
assert!(report.is_valid());
// On a tampered file: report.changed_records lists the records that differ.
```
The signature covers every record (text, embedding as stored, channel,
timestamp, session, tags, deleted flag, activation), the store's settings,
its sessions and its knowledge graph — a change made with any tool is caught.
It covers checkpoints, not saves still in the WAL
(`report.wal_entries_unsigned` counts those). CLI: `clawhdf5-cli keygen`,
`--signing-key <file>` on writing commands, and `verify --public-key`.
Signing adds about 20% to a checkpoint and 32 bytes per record to the file
([BENCHMARKS.md § Signed checkpoints](BENCHMARKS.md#signed-checkpoints)).
### Knowledge Graph ### Knowledge Graph
```rust ```rust
@@ -615,7 +645,6 @@ ClawhDF5's agent memory design draws from 15+ recent papers:
| `openblas` | no | OpenBLAS (Linux) | | `openblas` | no | OpenBLAS (Linux) |
| `gpu` | no | GPU search via wgpu | | `gpu` | no | GPU search via wgpu |
| `async` | no | Tokio async with background flush | | `async` | no | Tokio async with background flush |
| `agent` | no | Reserved; currently enables nothing (the agent layer is always built) |
To opt out of the parallel build: `--no-default-features --features float16,hnsw`. To opt out of the parallel build: `--no-default-features --features float16,hnsw`.
For an exact linear cosine scan instead of HNSW: `--no-default-features --features float16`. For an exact linear cosine scan instead of HNSW: `--no-default-features --features float16`.
+4 -1
View File
@@ -19,6 +19,10 @@ clawhdf5-ann = { path = "../clawhdf5-ann", version = "2.7.0", optional = true }
clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.7.0", optional = true, default-features = false } clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.7.0", optional = true, default-features = false }
serde = { workspace = true } serde = { workspace = true }
byteorder = "1" byteorder = "1"
# Signed checkpoints (MemoryConfig-independent; see `signing`). Pure Rust.
ed25519-dalek = { version = "2", features = ["rand_core"] }
sha2 = "0.10"
rand_core = { version = "0.6", features = ["getrandom"] }
half = { workspace = true, optional = true } half = { workspace = true, optional = true }
rayon = { version = "1", optional = true } rayon = { version = "1", optional = true }
matrixmultiply = { version = "0.3", optional = true } matrixmultiply = { version = "0.3", optional = true }
@@ -64,7 +68,6 @@ zstd = ["clawhdf5/zstd"]
# `--no-default-features` (plus re-enabling other defaults) to force the exact # `--no-default-features` (plus re-enabling other defaults) to force the exact
# linear cosine scan. # linear cosine scan.
hnsw = ["clawhdf5-ann"] hnsw = ["clawhdf5-ann"]
agent = []
gpu = ["clawhdf5-gpu/gpu-wgpu"] gpu = ["clawhdf5-gpu/gpu-wgpu"]
fast-math = ["matrixmultiply"] fast-math = ["matrixmultiply"]
accelerate = ["accelerate-src", "cblas-sys"] accelerate = ["accelerate-src", "cblas-sys"]
+72 -1
View File
@@ -36,6 +36,7 @@ pub mod reranker;
pub mod schema; pub mod schema;
pub mod search; pub mod search;
pub mod session; pub mod session;
pub mod signing;
pub mod storage; pub mod storage;
mod store_lock; mod store_lock;
pub mod temporal; pub mod temporal;
@@ -78,6 +79,7 @@ pub use session::{SessionCache, SessionEntry};
// --- Error type --- // --- Error type ---
#[derive(Debug)] #[derive(Debug)]
#[non_exhaustive]
pub enum MemoryError { pub enum MemoryError {
Io(std::io::Error), Io(std::io::Error),
Hdf5(String), Hdf5(String),
@@ -88,6 +90,11 @@ pub enum MemoryError {
/// A record the store cannot hold as given, e.g. an embedding value /// A record the store cannot hold as given, e.g. an embedding value
/// outside the half-precision range of a `float16` store. /// outside the half-precision range of a `float16` store.
InvalidEntry(String), InvalidEntry(String),
/// The store's checkpoints are signed and no signing key is set, so a
/// checkpoint would leave it unsigned. Set the key with
/// [`HDF5Memory::set_signing_key`], or drop the signature on purpose with
/// [`HDF5Memory::remove_signature`].
SigningKeyRequired(String),
} }
impl std::fmt::Display for MemoryError { impl std::fmt::Display for MemoryError {
@@ -99,6 +106,7 @@ impl std::fmt::Display for MemoryError {
MemoryError::NotFound(e) => write!(f, "not found: {e}"), MemoryError::NotFound(e) => write!(f, "not found: {e}"),
MemoryError::Locked(e) => write!(f, "store is locked: {e}"), MemoryError::Locked(e) => write!(f, "store is locked: {e}"),
MemoryError::InvalidEntry(e) => write!(f, "invalid entry: {e}"), MemoryError::InvalidEntry(e) => write!(f, "invalid entry: {e}"),
MemoryError::SigningKeyRequired(e) => write!(f, "signing key required: {e}"),
} }
} }
} }
@@ -317,6 +325,12 @@ pub struct HDF5Memory {
activations_dirty: bool, activations_dirty: bool,
/// Opened with [`HDF5Memory::open_read_only`]: nothing may reach the disk. /// Opened with [`HDF5Memory::open_read_only`]: nothing may reach the disk.
read_only: bool, read_only: bool,
/// Key that signs every checkpoint; never persisted. See
/// [`HDF5Memory::set_signing_key`].
signing_key: Option<signing::SigningKey>,
/// Checkpoints of this store are signed: the file on disk is, or a key
/// has been set. A checkpoint without a key is then refused.
signed: bool,
/// A WAL that `open()` could not read and moved aside; see /// A WAL that `open()` could not read and moved aside; see
/// [`HDF5Memory::quarantined_wal`]. /// [`HDF5Memory::quarantined_wal`].
quarantined_wal: Option<PathBuf>, quarantined_wal: Option<PathBuf>,
@@ -372,6 +386,8 @@ impl HDF5Memory {
bm25_filter: bm25::TokenFilter::default(), bm25_filter: bm25::TokenFilter::default(),
activations_dirty: false, activations_dirty: false,
read_only: false, read_only: false,
signing_key: None,
signed: false,
quarantined_wal: None, quarantined_wal: None,
_lock: Some(lock), _lock: Some(lock),
}) })
@@ -550,6 +566,8 @@ impl HDF5Memory {
bm25_filter: bm25::TokenFilter::default(), bm25_filter: bm25::TokenFilter::default(),
activations_dirty: false, activations_dirty: false,
read_only, read_only,
signing_key: None,
signed: checkpoint.signed,
quarantined_wal, quarantined_wal,
_lock: lock, _lock: lock,
}) })
@@ -710,6 +728,39 @@ impl HDF5Memory {
} }
} }
/// Sign every checkpoint from now on with `key` (Ed25519). The key is
/// never written anywhere; set it again after every `open`. Once a store
/// is signed, a checkpoint without the key is refused
/// ([`MemoryError::SigningKeyRequired`]) rather than silently leaving it
/// unsigned. Setting a different key re-signs the store under that key
/// from the next checkpoint; a verifier trusting the old key will then
/// reject it, which is the point. Call [`AgentMemory::flush_wal`] to sign
/// right away.
pub fn set_signing_key(&mut self, key: signing::SigningKey) {
self.signing_key = Some(key);
self.signed = true;
}
/// Stop signing: the next checkpoint writes the store unsigned. The
/// deliberate way out of [`MemoryError::SigningKeyRequired`].
pub fn remove_signature(&mut self) {
self.signing_key = None;
self.signed = false;
}
/// Checkpoints of this store are signed (on disk, or from the next
/// checkpoint because a key has been set).
pub fn is_signed(&self) -> bool {
self.signed
}
/// Check the checkpoint at `path` against the public key the caller
/// trusts; see [`signing::verify_store`]. Reads the file only: it works
/// on a store another process has open.
pub fn verify(path: &Path, trusted: &signing::VerifyingKey) -> Result<signing::VerifyReport> {
signing::verify_store(path, trusted)
}
/// Flush current state to disk and truncate the WAL. /// Flush current state to disk and truncate the WAL.
/// ///
/// Every code path that persists the full cache to the .h5 file must /// Every code path that persists the full cache to the .h5 file must
@@ -725,10 +776,28 @@ impl HDF5Memory {
// Record which WAL prefix this checkpoint contains, so a crash before // Record which WAL prefix this checkpoint contains, so a crash before
// the truncate below can't replay those entries a second time. // the truncate below can't replay those entries a second time.
let wal_applied = self.wal.as_ref().map(|w| w.mark()); let wal_applied = self.wal.as_ref().map(|w| w.mark());
let signature = match &self.signing_key {
Some(key) => Some(signing::sign(
key,
&self.config,
&self.cache,
&self.sessions,
&self.knowledge,
wal_applied,
)),
None if self.signed => {
return Err(MemoryError::SigningKeyRequired(format!(
"{} is signed; set its signing key before a checkpoint \
(saves so far are held in the WAL or in memory)",
self.config.path.display()
)));
}
None => None,
};
// Written before the .h5 so a crash in between leaves a sidecar whose // Written before the .h5 so a crash in between leaves a sidecar whose
// generation matches no checkpoint (ignored), never the reverse. // generation matches no checkpoint (ignored), never the reverse.
let ann_generation = self.persist_vector_index(); let ann_generation = self.persist_vector_index();
storage::write_to_disk_with_meta( storage::write_to_disk_signed(
&self.config.path, &self.config.path,
&self.config, &self.config,
&self.cache, &self.cache,
@@ -737,7 +806,9 @@ impl HDF5Memory {
&schema::CheckpointMeta { &schema::CheckpointMeta {
wal_applied, wal_applied,
ann_generation, ann_generation,
signed: signature.is_some(),
}, },
signature.as_ref(),
)?; )?;
if let Some(ref mut w) = self.wal { if let Some(ref mut w) = self.wal {
w.truncate()?; w.truncate()?;
+113 -1
View File
@@ -23,6 +23,7 @@ pub const ZEROCLAW_VERSION: &str = "0.8.0";
const WAL_APPLIED_LEN_ATTR: &str = "wal_applied_len"; const WAL_APPLIED_LEN_ATTR: &str = "wal_applied_len";
const WAL_APPLIED_CRC_ATTR: &str = "wal_applied_crc"; const WAL_APPLIED_CRC_ATTR: &str = "wal_applied_crc";
const ANN_GENERATION_ATTR: &str = "ann_generation"; const ANN_GENERATION_ATTR: &str = "ann_generation";
const SIG_VERSION_ATTR: &str = "sig_version";
/// Build a complete HDF5 file from the in-memory state. /// Build a complete HDF5 file from the in-memory state.
pub fn build_hdf5_file( pub fn build_hdf5_file(
@@ -46,7 +47,7 @@ pub fn build_hdf5_file_with_mark(
) -> Result<Vec<u8>, MemoryError> { ) -> Result<Vec<u8>, MemoryError> {
let meta = CheckpointMeta { let meta = CheckpointMeta {
wal_applied, wal_applied,
ann_generation: None, ..CheckpointMeta::default()
}; };
build_hdf5_file_with_meta(config, cache, sessions, knowledge, &meta) build_hdf5_file_with_meta(config, cache, sessions, knowledge, &meta)
} }
@@ -61,6 +62,10 @@ pub struct CheckpointMeta {
/// one left over from another checkpoint can never be attached to records /// one left over from another checkpoint can never be attached to records
/// it wasn't built from. /// it wasn't built from.
pub ann_generation: Option<u64>, pub ann_generation: Option<u64>,
/// The checkpoint carries an Ed25519 signature (see [`crate::signing`]).
/// Read-only: whether a checkpoint is *written* signed is decided by the
/// signature passed to [`build_hdf5_file_signed`].
pub signed: bool,
} }
/// [`build_hdf5_file`] with checkpoint bookkeeping. /// [`build_hdf5_file`] with checkpoint bookkeeping.
@@ -70,6 +75,19 @@ pub fn build_hdf5_file_with_meta(
sessions: &SessionCache, sessions: &SessionCache,
knowledge: &KnowledgeCache, knowledge: &KnowledgeCache,
checkpoint: &CheckpointMeta, checkpoint: &CheckpointMeta,
) -> Result<Vec<u8>, MemoryError> {
build_hdf5_file_signed(config, cache, sessions, knowledge, checkpoint, None)
}
/// [`build_hdf5_file_with_meta`], plus a signed manifest of the contents
/// (see [`crate::signing`]).
pub fn build_hdf5_file_signed(
config: &MemoryConfig,
cache: &MemoryCache,
sessions: &SessionCache,
knowledge: &KnowledgeCache,
checkpoint: &CheckpointMeta,
signature: Option<&crate::signing::StoredSignature>,
) -> Result<Vec<u8>, MemoryError> { ) -> Result<Vec<u8>, MemoryError> {
let wal_applied = checkpoint.wal_applied; let wal_applied = checkpoint.wal_applied;
let mut builder = clawhdf5::FileBuilder::new(); let mut builder = clawhdf5::FileBuilder::new();
@@ -130,11 +148,42 @@ pub fn build_hdf5_file_with_meta(
// round trip through every reader. // round trip through every reader.
meta.set_attr(ANN_GENERATION_ATTR, AttrValue::I64(generation as i64)); meta.set_attr(ANN_GENERATION_ATTR, AttrValue::I64(generation as i64));
} }
if let Some(sig) = signature {
use crate::signing::to_hex;
let m = &sig.manifest;
meta.set_attr(
SIG_VERSION_ATTR,
AttrValue::I64(crate::signing::MANIFEST_VERSION),
);
meta.set_attr("sig_algorithm", AttrValue::String("ed25519".into()));
meta.set_attr("sig_public_key", AttrValue::String(to_hex(&sig.public_key)));
meta.set_attr("sig_signature", AttrValue::String(to_hex(&sig.signature)));
meta.set_attr("sig_record_count", AttrValue::I64(m.record_count as i64));
meta.set_attr(
"sig_records_root",
AttrValue::String(to_hex(&m.records_root)),
);
meta.set_attr("sig_settings", AttrValue::String(to_hex(&m.settings)));
meta.set_attr("sig_sessions", AttrValue::String(to_hex(&m.sessions)));
meta.set_attr("sig_graph", AttrValue::String(to_hex(&m.graph)));
}
// Need at least one dataset in the group for it to be a proper group // Need at least one dataset in the group for it to be a proper group
meta.create_dataset("_marker").with_u8_data(&[1]).compact(); meta.create_dataset("_marker").with_u8_data(&[1]).compact();
let finished_meta = meta.finish(); let finished_meta = meta.finish();
builder.add_group(finished_meta); builder.add_group(finished_meta);
// /integrity: the signed per-record hashes, so verification can say
// which records changed.
if let Some(sig) = signature {
let mut group = builder.create_group("integrity");
let flat: Vec<u8> = sig.record_hashes.iter().flatten().copied().collect();
group
.create_dataset("record_hashes")
.with_u8_data(&flat)
.with_shape(&[sig.record_hashes.len() as u64, 32]);
builder.add_group(group.finish());
}
// /memory group // /memory group
build_memory_group(&mut builder, config, cache)?; build_memory_group(&mut builder, config, cache)?;
@@ -440,6 +489,64 @@ pub fn read_wal_mark(file: &clawhdf5::File) -> Option<WalMark> {
Some(WalMark { len, crc }) Some(WalMark { len, crc })
} }
/// Read a checkpoint's signature, if it has one. A signature whose
/// attributes are present but malformed is an error, not "unsigned".
pub fn read_signature(
file: &clawhdf5::File,
) -> Result<Option<crate::signing::StoredSignature>, MemoryError> {
use crate::signing::{Manifest, StoredSignature, from_hex};
let attrs = file
.group("meta")
.and_then(|g| g.attrs())
.map_err(|e| MemoryError::Schema(format!("cannot read /meta attrs: {e}")))?;
let version = match attrs.get(SIG_VERSION_ATTR) {
None => return Ok(None),
Some(AttrValue::I64(v)) => *v,
Some(_) => return Err(MemoryError::Schema("malformed sig_version".into())),
};
if version != crate::signing::MANIFEST_VERSION {
return Err(MemoryError::Schema(format!(
"unsupported signature version {version}"
)));
}
fn hex<const N: usize>(
attrs: &std::collections::HashMap<String, AttrValue>,
name: &str,
) -> Result<[u8; N], MemoryError> {
match attrs.get(name) {
Some(AttrValue::String(s)) => from_hex::<N>(s),
_ => None,
}
.ok_or_else(|| MemoryError::Schema(format!("malformed or missing {name}")))
}
let record_count = match attrs.get("sig_record_count") {
Some(AttrValue::I64(v)) if *v >= 0 => *v as u64,
_ => return Err(MemoryError::Schema("malformed sig_record_count".into())),
};
let group = file
.group("integrity")
.map_err(|e| MemoryError::Schema(format!("signed checkpoint without /integrity: {e}")))?;
let flat = read_u8_dataset(&group, "record_hashes")?;
if flat.len() % 32 != 0 {
return Err(MemoryError::Schema(
"/integrity/record_hashes is not a whole number of hashes".into(),
));
}
let record_hashes = flat.as_chunks::<32>().0.to_vec();
Ok(Some(StoredSignature {
manifest: Manifest {
record_count,
records_root: hex::<32>(&attrs, "sig_records_root")?,
settings: hex::<32>(&attrs, "sig_settings")?,
sessions: hex::<32>(&attrs, "sig_sessions")?,
graph: hex::<32>(&attrs, "sig_graph")?,
},
record_hashes,
public_key: hex::<32>(&attrs, "sig_public_key")?,
signature: hex::<64>(&attrs, "sig_signature")?,
}))
}
/// Read the checkpoint bookkeeping from `/meta`. /// Read the checkpoint bookkeeping from `/meta`.
pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta { pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta {
let ann_generation = file let ann_generation = file
@@ -450,9 +557,14 @@ pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta {
Some(AttrValue::I64(v)) => Some(*v as u64), Some(AttrValue::I64(v)) => Some(*v as u64),
_ => None, _ => None,
}); });
let signed = file
.group("meta")
.and_then(|g| g.attrs())
.is_ok_and(|attrs| attrs.contains_key(SIG_VERSION_ATTR));
CheckpointMeta { CheckpointMeta {
wal_applied: read_wal_mark(file), wal_applied: read_wal_mark(file),
ann_generation, ann_generation,
signed,
} }
} }
+419
View File
@@ -0,0 +1,419 @@
//! Ed25519-signed checkpoints.
//!
//! When a signing key is set ([`crate::HDF5Memory::set_signing_key`]), every
//! checkpoint writes a signed manifest of the store: a SHA-256 per memory
//! record rolled into a Merkle root, plus hashes of the store's settings, its
//! sessions and its knowledge graph. [`verify_store`] recomputes all of it from
//! the file and checks the signature against a public key the caller trusts,
//! so any change to the checkpointed file — a record's text or embedding, a
//! setting, a session, a graph edge, made through this crate or any other HDF5
//! tool — is detected, and the per-record hashes say which records changed.
//!
//! What it does not cover: saves still only in the WAL (made since the last
//! checkpoint). [`VerifyReport::wal_entries_unsigned`] counts them.
//!
//! The hashes cover exactly what the file persists, in the form the loader
//! returns it, so a store verifies after any number of reopen/checkpoint
//! cycles. Derived data (L2 norms, the vector index) is not covered; it is
//! recomputed from covered data.
use ed25519_dalek::{Signature, Signer, Verifier};
pub use ed25519_dalek::{SigningKey, VerifyingKey};
use sha2::{Digest, Sha256};
use crate::MemoryConfig;
use crate::cache::MemoryCache;
use crate::knowledge::KnowledgeCache;
use crate::session::SessionCache;
use crate::wal::WalMark;
/// Version of the manifest encoding; part of what is signed.
pub const MANIFEST_VERSION: i64 = 1;
type Hash = [u8; 32];
/// The hashes a signature covers.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Manifest {
pub record_count: u64,
/// Merkle root over the per-record hashes.
pub records_root: Hash,
/// Settings persisted in `/meta`, plus the checkpoint's WAL mark.
pub settings: Hash,
pub sessions: Hash,
pub graph: Hash,
}
impl Manifest {
/// The exact bytes that are signed.
pub fn signed_bytes(&self) -> Vec<u8> {
let mut m = Vec::with_capacity(160);
m.extend_from_slice(b"clawhdf5-agent signed checkpoint\0");
m.extend_from_slice(&MANIFEST_VERSION.to_le_bytes());
m.extend_from_slice(&self.record_count.to_le_bytes());
m.extend_from_slice(&self.records_root);
m.extend_from_slice(&self.settings);
m.extend_from_slice(&self.sessions);
m.extend_from_slice(&self.graph);
m
}
}
/// A signature as stored in a checkpoint.
#[derive(Debug, Clone)]
pub struct StoredSignature {
pub manifest: Manifest,
pub record_hashes: Vec<Hash>,
pub public_key: [u8; 32],
pub signature: [u8; 64],
}
/// Build the manifest (and per-record hashes) for the state about to be
/// checkpointed, and sign it.
pub fn sign(
key: &SigningKey,
config: &MemoryConfig,
cache: &MemoryCache,
sessions: &SessionCache,
knowledge: &KnowledgeCache,
wal_applied: Option<WalMark>,
) -> StoredSignature {
let (manifest, record_hashes) = manifest(config, cache, sessions, knowledge, wal_applied);
let signature = key.sign(&manifest.signed_bytes()).to_bytes();
StoredSignature {
manifest,
record_hashes,
public_key: key.verifying_key().to_bytes(),
signature,
}
}
/// Compute the manifest of a store's state.
pub fn manifest(
config: &MemoryConfig,
cache: &MemoryCache,
sessions: &SessionCache,
knowledge: &KnowledgeCache,
wal_applied: Option<WalMark>,
) -> (Manifest, Vec<Hash>) {
let record_hashes: Vec<Hash> = (0..cache.len()).map(|i| record_hash(cache, i)).collect();
let manifest = Manifest {
record_count: cache.len() as u64,
records_root: merkle_root(&record_hashes),
settings: settings_hash(config, wal_applied),
sessions: sessions_hash(sessions),
graph: graph_hash(knowledge),
};
(manifest, record_hashes)
}
// ---------------------------------------------------------------------------
// Canonical encoding
// ---------------------------------------------------------------------------
/// A SHA-256 over length-prefixed fields, so no two different field lists
/// hash the same bytes.
struct Fields(Sha256);
impl Fields {
fn new(domain: &str) -> Self {
let mut h = Sha256::new();
h.update((domain.len() as u64).to_le_bytes());
h.update(domain.as_bytes());
Self(h)
}
fn bytes(&mut self, b: &[u8]) -> &mut Self {
self.0.update((b.len() as u64).to_le_bytes());
self.0.update(b);
self
}
/// Strings as the loader returns them: stored null-padded, so a trailing
/// NUL cannot survive a round trip and must not be part of the hash.
fn str(&mut self, s: &str) -> &mut Self {
self.bytes(s.trim_end_matches('\0').as_bytes())
}
fn u64(&mut self, v: u64) -> &mut Self {
self.0.update(v.to_le_bytes());
self
}
fn f64(&mut self, v: f64) -> &mut Self {
self.0.update(v.to_bits().to_le_bytes());
self
}
fn f32(&mut self, v: f32) -> &mut Self {
self.0.update(v.to_bits().to_le_bytes());
self
}
fn finish(self) -> Hash {
self.0.finalize().into()
}
}
/// Everything persisted about record `i`, including its position. The
/// embedding is hashed as the cache holds it — for a `float16` store that is
/// the half-rounded value the file holds.
fn record_hash(cache: &MemoryCache, i: usize) -> Hash {
let mut f = Fields::new("clawhdf5-agent/record");
f.u64(i as u64).str(&cache.chunks[i]);
let emb: Vec<u8> = cache.embeddings[i]
.iter()
.flat_map(|v| v.to_bits().to_le_bytes())
.collect();
f.bytes(&emb)
.str(&cache.source_channels[i])
.f64(cache.timestamps[i])
.str(&cache.session_ids[i])
.str(&cache.tags[i])
.u64(u64::from(cache.tombstones[i]))
.f32(cache.activation_weights[i]);
f.finish()
}
/// Binary Merkle tree: leaves are the record hashes; a parent hashes its two
/// children with a node prefix; an odd node is carried up unchanged.
fn merkle_root(leaves: &[Hash]) -> Hash {
if leaves.is_empty() {
return Fields::new("clawhdf5-agent/merkle-empty").finish();
}
let mut level: Vec<Hash> = leaves.to_vec();
while level.len() > 1 {
level = level
.chunks(2)
.map(|pair| match pair {
[l, r] => {
let mut h = Sha256::new();
h.update([1u8]);
h.update(l);
h.update(r);
h.finalize().into()
}
[only] => *only,
_ => unreachable!(),
})
.collect();
}
level[0]
}
fn settings_hash(c: &MemoryConfig, wal_applied: Option<WalMark>) -> Hash {
let mut f = Fields::new("clawhdf5-agent/settings");
f.str(crate::schema::SCHEMA_VERSION)
.str(&c.created_at)
.str(&c.agent_id)
.str(&c.embedder)
.u64(c.embedding_dim as u64)
.u64(c.chunk_size as u64)
.u64(c.overlap as u64)
.u64(u64::from(c.float16))
.u64(u64::from(c.compression))
.u64(u64::from(c.compression_level))
.f32(c.compact_threshold)
.f32(c.hebbian_boost)
.f32(c.decay_factor)
.u64(u64::from(c.wal_enabled))
.u64(c.wal_max_entries as u64)
.u64(u64::from(c.quantized_index))
.u64(c.hnsw_m as u64)
.u64(c.hnsw_ef_construction as u64)
.u64(c.hnsw_ef_search as u64);
// An empty mark is not written to the file, so it must hash as none.
match wal_applied.filter(|m| m.len > 0) {
Some(m) => f.u64(1).u64(m.len).u64(u64::from(m.crc)),
None => f.u64(0),
};
f.finish()
}
fn sessions_hash(s: &SessionCache) -> Hash {
let mut f = Fields::new("clawhdf5-agent/sessions");
f.u64(s.entries.len() as u64);
for (i, e) in s.entries.iter().enumerate() {
f.str(&e.id)
.u64(e.start_idx)
.u64(e.end_idx)
.str(&e.channel)
.f64(e.ts)
.str(s.summaries.get(i).map(String::as_str).unwrap_or(""));
}
f.finish()
}
fn graph_hash(k: &KnowledgeCache) -> Hash {
let mut f = Fields::new("clawhdf5-agent/graph");
f.u64(k.entities.len() as u64);
for e in &k.entities {
f.u64(e.id)
.str(&e.name)
.str(&e.entity_type)
.u64(e.embedding_idx as u64);
}
f.u64(k.relations.len() as u64);
for r in &k.relations {
f.u64(r.src)
.u64(r.tgt)
.str(&r.relation)
.f32(r.weight)
.f64(r.ts);
}
f.u64(k.alias_strings.len() as u64);
for (s, id) in k.alias_strings.iter().zip(&k.alias_entity_ids) {
f.str(s).u64(*id as u64);
}
f.finish()
}
// ---------------------------------------------------------------------------
// Verification
// ---------------------------------------------------------------------------
/// The outcome of [`verify_store`].
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct VerifyReport {
/// The checkpoint carries a signature.
pub signed: bool,
/// The signature was made by the key the caller trusts.
pub key_matches: bool,
/// The signature over the stored manifest is valid.
pub signature_valid: bool,
/// The file's current contents match the signed manifest.
pub records_match: bool,
pub settings_match: bool,
pub sessions_match: bool,
pub graph_match: bool,
/// Records whose contents differ from what was signed (by position),
/// when the stored per-record hashes are themselves authentic.
pub changed_records: Vec<usize>,
/// Records in the file versus in the signed manifest.
pub record_count: u64,
pub signed_record_count: u64,
/// The public key the checkpoint claims to be signed by.
pub public_key: Option<[u8; 32]>,
/// Saves in the WAL after the checkpoint: not covered by the signature.
pub wal_entries_unsigned: usize,
}
impl VerifyReport {
/// Signed by the trusted key, signature valid, and every part of the
/// file unchanged since it was signed.
pub fn is_valid(&self) -> bool {
self.signed
&& self.key_matches
&& self.signature_valid
&& self.records_match
&& self.settings_match
&& self.sessions_match
&& self.graph_match
}
}
/// Check a store file against the public key the caller trusts.
///
/// Reads the checkpoint (not the WAL), recomputes every hash from its
/// contents and checks the signature. Never writes.
pub fn verify_store(
path: &std::path::Path,
trusted: &VerifyingKey,
) -> Result<VerifyReport, crate::MemoryError> {
let file = clawhdf5::File::open(path)
.map_err(|e| crate::MemoryError::Hdf5(format!("cannot open {}: {e}", path.display())))?;
let (config, cache, sessions, knowledge) = crate::schema::validate_and_load(&file)?;
let checkpoint = crate::schema::read_checkpoint_meta(&file);
let stored = crate::schema::read_signature(&file)?;
let wal_entries_unsigned = count_wal_entries_after(path, checkpoint.wal_applied);
let (current, current_hashes) = manifest(
&config,
&cache,
&sessions,
&knowledge,
checkpoint.wal_applied,
);
let Some(stored) = stored else {
return Ok(VerifyReport {
signed: false,
key_matches: false,
signature_valid: false,
records_match: false,
settings_match: false,
sessions_match: false,
graph_match: false,
changed_records: Vec::new(),
record_count: current.record_count,
signed_record_count: 0,
public_key: None,
wal_entries_unsigned,
});
};
let key_matches = stored.public_key == trusted.to_bytes();
let signature_valid = trusted
.verify(
&stored.manifest.signed_bytes(),
&Signature::from_bytes(&stored.signature),
)
.is_ok();
// The stored per-record hashes can localise a change only if they are
// the ones that were signed.
let hashes_authentic = signature_valid
&& stored.record_hashes.len() as u64 == stored.manifest.record_count
&& merkle_root(&stored.record_hashes) == stored.manifest.records_root;
let changed_records = if hashes_authentic {
let n = current_hashes.len().max(stored.record_hashes.len());
(0..n)
.filter(|&i| current_hashes.get(i) != stored.record_hashes.get(i))
.collect()
} else {
Vec::new()
};
Ok(VerifyReport {
signed: true,
key_matches,
signature_valid,
records_match: signature_valid
&& current.record_count == stored.manifest.record_count
&& current.records_root == stored.manifest.records_root,
settings_match: signature_valid && current.settings == stored.manifest.settings,
sessions_match: signature_valid && current.sessions == stored.manifest.sessions,
graph_match: signature_valid && current.graph == stored.manifest.graph,
changed_records,
record_count: current.record_count,
signed_record_count: stored.manifest.record_count,
public_key: Some(stored.public_key),
wal_entries_unsigned,
})
}
fn count_wal_entries_after(store: &std::path::Path, mark: Option<WalMark>) -> usize {
let wal = store.with_extension("h5.wal");
if !wal.exists() {
return 0;
}
crate::wal::WalFile::read_entries_for_migration(&wal, mark)
.map(|e| e.len())
.unwrap_or(0)
}
/// A new random signing key from the operating system's RNG.
pub fn generate_key() -> SigningKey {
SigningKey::generate(&mut rand_core::OsRng)
}
/// Hex encoding for keys and signatures in attributes and the CLI.
pub fn to_hex(bytes: &[u8]) -> String {
bytes.iter().map(|b| format!("{b:02x}")).collect()
}
/// Parse hex into exactly `N` bytes.
pub fn from_hex<const N: usize>(s: &str) -> Option<[u8; N]> {
let s = s.trim();
if s.len() != 2 * N {
return None;
}
let mut out = [0u8; N];
for (i, byte) in out.iter_mut().enumerate() {
*byte = u8::from_str_radix(&s[2 * i..2 * i + 2], 16).ok()?;
}
Some(out)
}
+16 -2
View File
@@ -36,7 +36,7 @@ pub fn write_to_disk_with_mark(
) -> Result<(), MemoryError> { ) -> Result<(), MemoryError> {
let meta = schema::CheckpointMeta { let meta = schema::CheckpointMeta {
wal_applied, wal_applied,
ann_generation: None, ..schema::CheckpointMeta::default()
}; };
write_to_disk_with_meta(path, config, cache, sessions, knowledge, &meta) write_to_disk_with_meta(path, config, cache, sessions, knowledge, &meta)
} }
@@ -50,7 +50,21 @@ pub fn write_to_disk_with_meta(
knowledge: &KnowledgeCache, knowledge: &KnowledgeCache,
checkpoint: &schema::CheckpointMeta, checkpoint: &schema::CheckpointMeta,
) -> Result<(), MemoryError> { ) -> Result<(), MemoryError> {
let bytes = schema::build_hdf5_file_with_meta(config, cache, sessions, knowledge, checkpoint)?; write_to_disk_signed(path, config, cache, sessions, knowledge, checkpoint, None)
}
/// [`write_to_disk_with_meta`] with a signed manifest of the contents.
pub fn write_to_disk_signed(
path: &Path,
config: &MemoryConfig,
cache: &MemoryCache,
sessions: &SessionCache,
knowledge: &KnowledgeCache,
checkpoint: &schema::CheckpointMeta,
signature: Option<&crate::signing::StoredSignature>,
) -> Result<(), MemoryError> {
let bytes =
schema::build_hdf5_file_signed(config, cache, sessions, knowledge, checkpoint, signature)?;
if bytes.is_empty() { if bytes.is_empty() {
return Err(MemoryError::Hdf5("build_hdf5_file produced 0 bytes".into())); return Err(MemoryError::Hdf5("build_hdf5_file produced 0 bytes".into()));
@@ -92,3 +92,64 @@ print(len(names))
assert!(n >= 10, "only {n} datasets"); assert!(n >= 10, "only {n} datasets");
} }
} }
#[test]
fn an_edit_made_with_h5py_breaks_the_signature_and_names_the_record() {
if !h5py_available() {
assert!(
std::env::var("CLAWHDF5_REQUIRE_INTEROP").as_deref() != Ok("1"),
"CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
);
eprintln!("SKIP: python3 with h5py not available");
return;
}
use clawhdf5_agent::signing::SigningKey;
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("signed.h5");
let key = SigningKey::from_bytes(&[42; 32]);
let mut m = HDF5Memory::create(MemoryConfig::new(path.clone(), "agent", 8)).unwrap();
m.set_signing_key(key.clone());
m.save_batch(
(0..10)
.map(|i| MemoryEntry {
chunk: format!("memory {i}"),
embedding: (0..8).map(|j| ((i * 8 + j) as f32).cos()).collect(),
source_channel: "test".into(),
timestamp: i as f64,
session_id: "s".into(),
tags: String::new(),
})
.collect(),
)
.unwrap();
drop(m);
assert!(
HDF5Memory::verify(&path, &key.verifying_key())
.unwrap()
.is_valid()
);
// Someone edits one timestamp in place with h5py.
let script = format!(
r#"
import h5py
with h5py.File("{}", "r+") as f:
ts = f["memory/timestamps"]
ts[3] = 12345.0
"#,
path.display()
);
let out = Command::new(python())
.args(["-c", &script])
.output()
.unwrap();
assert!(
out.status.success(),
"{}",
String::from_utf8_lossy(&out.stderr)
);
let r = HDF5Memory::verify(&path, &key.verifying_key()).unwrap();
assert!(r.signature_valid && !r.is_valid(), "{r:?}");
assert_eq!(r.changed_records, vec![3]);
}
+330
View File
@@ -0,0 +1,330 @@
//! Ed25519-signed checkpoints: `HDF5Memory::set_signing_key` and
//! `HDF5Memory::verify`.
use std::path::Path;
use clawhdf5_agent::signing::{SigningKey, VerifyReport, VerifyingKey};
use clawhdf5_agent::storage;
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, MemoryError, schema};
use tempfile::TempDir;
const DIM: usize = 16;
fn key(seed: u8) -> SigningKey {
SigningKey::from_bytes(&[seed; 32])
}
fn entry(i: usize, chunk: &str) -> MemoryEntry {
MemoryEntry {
chunk: chunk.to_string(),
embedding: (0..DIM)
.map(|j| ((i * DIM + j) as f32 * 0.37).sin())
.collect(),
source_channel: "chat".into(),
timestamp: 1_700_000_000.0 + i as f64,
session_id: format!("s{}", i % 3),
tags: format!("t{i}"),
}
}
/// Awkward strings on purpose: they must hash the same after a round trip.
const TEXTS: [&str; 6] = [
"plain text",
"ünïcödé — 日本語 🙂",
"",
"trailing spaces ",
"tab\tand\nnewline",
"x",
];
fn signed_store(dir: &TempDir, float16: bool, k: &SigningKey) -> std::path::PathBuf {
let mut cfg = MemoryConfig::new(dir.path().join("s.h5"), "agent", DIM);
cfg.float16 = float16;
let path = cfg.path.clone();
let mut m = HDF5Memory::create(cfg).unwrap();
m.set_signing_key(k.clone());
let entries = (0..30).map(|i| entry(i, TEXTS[i % TEXTS.len()])).collect();
m.save_batch(entries).unwrap();
// Some graph and a deleted record, so every part of the manifest is used.
let a = m.knowledge_mut().add_entity("Alice", "person", 0);
let b = m.knowledge_mut().add_entity("Acme", "org", -1);
m.knowledge_mut().add_relation(a, b, "works_at", 0.75);
m.sessions_mut()
.add_at("s0", 0, 9, "chat", "first session", 1_700_000_000.0);
m.delete(4).unwrap();
m.flush_wal().unwrap();
path
}
fn verify(path: &Path, k: &SigningKey) -> VerifyReport {
HDF5Memory::verify(path, &k.verifying_key()).unwrap()
}
#[test]
fn a_signed_store_verifies_through_reopen_and_checkpoint_cycles() {
for float16 in [true, false] {
let dir = TempDir::new().unwrap();
let k = key(7);
let path = signed_store(&dir, float16, &k);
let r = verify(&path, &k);
assert!(r.is_valid(), "float16={float16}: {r:?}");
assert_eq!(r.public_key, Some(k.verifying_key().to_bytes()));
assert_eq!(r.record_count, 30);
assert!(r.changed_records.is_empty());
// Reopen, change nothing, checkpoint again (with the key): still valid.
for _ in 0..3 {
let mut m = HDF5Memory::open(&path).unwrap();
assert!(m.is_signed());
m.set_signing_key(k.clone());
m.flush_wal().unwrap();
drop(m);
assert!(verify(&path, &k).is_valid());
}
// And after real changes, re-signed.
let mut m = HDF5Memory::open(&path).unwrap();
m.set_signing_key(k.clone());
m.save(entry(99, "added later")).unwrap();
m.hybrid_search(&entry(1, "").embedding, "text", 0.4, 0.6, 5);
m.flush_wal().unwrap();
drop(m);
let r = verify(&path, &k);
assert!(r.is_valid(), "{r:?}");
assert_eq!(r.record_count, 31);
}
}
#[test]
fn a_signed_store_refuses_to_checkpoint_without_its_key() {
let dir = TempDir::new().unwrap();
let k = key(1);
let path = signed_store(&dir, true, &k);
let mut m = HDF5Memory::open(&path).unwrap();
m.save(entry(50, "pending")).unwrap();
match m.flush_wal() {
Err(MemoryError::SigningKeyRequired(msg)) => assert!(msg.contains("signed"), "{msg}"),
other => panic!("expected SigningKeyRequired, got {other:?}"),
}
// The file is untouched and still valid; the save is still in the WAL.
let r = verify(&path, &k);
assert!(r.is_valid());
assert_eq!(r.wal_entries_unsigned, 1);
// Supplying the key lets the checkpoint through, signed.
m.set_signing_key(k.clone());
m.flush_wal().unwrap();
drop(m);
let r = verify(&path, &k);
assert!(r.is_valid());
assert_eq!((r.record_count, r.wal_entries_unsigned), (31, 0));
// Removing the signature on purpose writes it unsigned.
let mut m = HDF5Memory::open(&path).unwrap();
m.remove_signature();
m.flush_wal().unwrap();
drop(m);
let r = verify(&path, &k);
assert!(!r.signed && !r.is_valid());
assert!(!HDF5Memory::open(&path).unwrap().is_signed());
}
#[test]
fn the_wrong_key_does_not_verify_and_a_new_key_re_signs() {
let dir = TempDir::new().unwrap();
let (a, b) = (key(1), key(2));
let path = signed_store(&dir, true, &a);
let r = verify(&path, &b);
assert!(r.signed && !r.key_matches && !r.signature_valid && !r.is_valid());
let mut m = HDF5Memory::open(&path).unwrap();
m.set_signing_key(b.clone());
m.flush_wal().unwrap();
drop(m);
assert!(verify(&path, &b).is_valid());
assert!(!verify(&path, &a).is_valid());
}
/// Rewrite the store with changed contents but the *old* signature — what
/// someone with write access to the file, but not the key, can do.
fn tamper(path: &Path, change: impl FnOnce(&mut Tampered)) {
let file = clawhdf5::File::open(path).unwrap();
let (config, cache, sessions, knowledge) = schema::validate_and_load(&file).unwrap();
let checkpoint = schema::read_checkpoint_meta(&file);
let signature = schema::read_signature(&file).unwrap().unwrap();
drop(file);
let mut t = Tampered {
config,
cache,
sessions,
knowledge,
};
change(&mut t);
storage::write_to_disk_signed(
path,
&t.config,
&t.cache,
&t.sessions,
&t.knowledge,
&checkpoint,
Some(&signature),
)
.unwrap();
}
struct Tampered {
config: MemoryConfig,
cache: clawhdf5_agent::cache::MemoryCache,
sessions: clawhdf5_agent::SessionCache,
knowledge: clawhdf5_agent::knowledge::KnowledgeCache,
}
#[test]
fn every_kind_of_edit_is_detected_and_located() {
let k = key(3);
type Edit = Box<dyn FnOnce(&mut Tampered)>;
type Case = (&'static str, Edit, fn(&VerifyReport) -> bool);
let cases: Vec<Case> = vec![
(
"record text",
Box::new(|t: &mut Tampered| t.cache.chunks[7] = "rewritten".into()),
|r| !r.records_match && r.changed_records == vec![7],
),
(
"one embedding value",
Box::new(|t: &mut Tampered| {
let mut e = t.cache.embeddings[12].to_vec();
e[3] = 0.5;
t.cache.embeddings.set(12, &e);
}),
|r| r.changed_records == vec![12],
),
(
"undelete",
Box::new(|t: &mut Tampered| t.cache.tombstones[4] = 0),
|r| r.changed_records == vec![4],
),
(
"timestamp",
Box::new(|t: &mut Tampered| t.cache.timestamps[20] += 1.0),
|r| r.changed_records == vec![20],
),
(
"record appended",
Box::new(|t: &mut Tampered| {
t.cache.push(
"new".into(),
vec![0.1; DIM],
"x".into(),
1.0,
"s".into(),
"".into(),
);
}),
|r| !r.records_match && r.changed_records == vec![30] && r.record_count == 31,
),
(
"setting",
Box::new(|t: &mut Tampered| t.config.agent_id = "someone-else".into()),
|r| !r.settings_match && r.records_match,
),
(
"session summary",
Box::new(|t: &mut Tampered| t.sessions.summaries[0] = "edited".into()),
|r| !r.sessions_match && r.records_match,
),
(
"graph edge",
Box::new(|t: &mut Tampered| t.knowledge.relations[0].weight = 1.0),
|r| !r.graph_match && r.records_match,
),
];
for (name, edit, check) in cases {
let dir = TempDir::new().unwrap();
let path = signed_store(&dir, true, &k);
tamper(&path, edit);
let r = verify(&path, &k);
assert!(
r.signed && r.key_matches && r.signature_valid,
"{name}: {r:?}"
);
assert!(!r.is_valid(), "{name}: edit not detected: {r:?}");
assert!(check(&r), "{name}: {r:?}");
}
}
#[test]
fn a_forged_manifest_fails_the_signature() {
// Recomputing the hashes for tampered contents does not help without the
// key: the signature no longer matches the manifest.
let dir = TempDir::new().unwrap();
let k = key(5);
let path = signed_store(&dir, true, &k);
let file = clawhdf5::File::open(&path).unwrap();
let (config, mut cache, sessions, knowledge) = schema::validate_and_load(&file).unwrap();
let checkpoint = schema::read_checkpoint_meta(&file);
let mut sig = schema::read_signature(&file).unwrap().unwrap();
drop(file);
cache.chunks[0] = "forged".into();
// Re-sign with an attacker key, then splice the victim's public key back.
let forged = clawhdf5_agent::signing::sign(
&key(66),
&config,
&cache,
&sessions,
&knowledge,
checkpoint.wal_applied,
);
sig.manifest = forged.manifest;
sig.record_hashes = forged.record_hashes;
storage::write_to_disk_signed(
&path,
&config,
&cache,
&sessions,
&knowledge,
&checkpoint,
Some(&sig),
)
.unwrap();
let r = verify(&path, &k);
assert!(
r.key_matches && !r.signature_valid && !r.is_valid(),
"{r:?}"
);
}
#[test]
fn an_unsigned_store_reports_unsigned() {
let dir = TempDir::new().unwrap();
let mut m = HDF5Memory::create(MemoryConfig::new(dir.path().join("u.h5"), "a", DIM)).unwrap();
m.save_batch(vec![entry(0, "hello")]).unwrap();
drop(m);
let r = HDF5Memory::verify(&dir.path().join("u.h5"), &VerifyingKey::from(&key(1))).unwrap();
assert!(!r.signed && !r.is_valid());
assert_eq!(r.record_count, 1);
}
#[test]
fn nul_bytes_in_text_still_verify() {
// Strings are stored null-padded; the hash must follow what a reopened
// store actually holds, or an untouched store would fail to verify.
let dir = TempDir::new().unwrap();
let k = key(9);
let mut m = HDF5Memory::create(MemoryConfig::new(dir.path().join("n.h5"), "a", DIM)).unwrap();
m.set_signing_key(k.clone());
m.save_batch(vec![
entry(0, "inner\0nul"),
entry(1, "trailing nul\0"),
entry(2, "\0leading"),
])
.unwrap();
drop(m);
let r = verify(&dir.path().join("n.h5"), &k);
assert!(r.is_valid(), "{r:?}");
let m = HDF5Memory::open(&dir.path().join("n.h5")).unwrap();
eprintln!(
"reloaded: {:?}",
(0..3).map(|i| m.get_chunk(i)).collect::<Vec<_>>()
);
}
@@ -21,6 +21,7 @@
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --ann-only --uniform //! cargo run --release -p clawhdf5-bench --bin search_harness -- --ann-only --uniform
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full //! cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --options-study --full //! cargo run --release -p clawhdf5-bench --bin search_harness -- --options-study --full
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --signing-study --full
//! ``` //! ```
use std::time::{Duration, Instant}; use std::time::{Duration, Instant};
@@ -488,6 +489,81 @@ fn bench_end_to_end(n: usize, json: &mut Vec<serde_json::Value>) {
})); }));
} }
// ---------------------------------------------------------------------------
// Signing study: what does an Ed25519-signed checkpoint cost?
// ---------------------------------------------------------------------------
/// `--signing-study`: checkpoint time unsigned vs signed, `verify` time, and
/// the file-size cost of the stored per-record hashes. Default store
/// settings (float16, int8 index). Medians of five checkpoints / three
/// verifies.
fn signing_study(n: usize) {
use clawhdf5_agent::signing::SigningKey;
let data = make_dataset(n, 0x516 ^ n as u64);
let mut rng = Rng(9);
let entries: Vec<MemoryEntry> = data
.vectors
.iter()
.enumerate()
.map(|(i, v)| MemoryEntry {
chunk: text_for(data.cluster_of[i], i, &mut rng),
embedding: v.clone(),
source_channel: "bench".into(),
timestamp: i as f64,
session_id: format!("s{}", i % 50),
tags: format!("t{i}"),
})
.collect();
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("sign.h5");
let mut mem = HDF5Memory::create(MemoryConfig::new(path.clone(), "bench", DIM)).unwrap();
mem.save_batch(entries).unwrap();
std::hint::black_box(mem.hybrid_search(&data.queries[0], "", 1.0, 0.0, K));
let median = |mut v: Vec<Duration>| {
v.sort();
v[v.len() / 2]
};
let checkpoint = |mem: &mut HDF5Memory| {
median(
(0..5)
.map(|_| {
let t = Instant::now();
mem.flush_wal().unwrap();
t.elapsed()
})
.collect(),
)
};
let unsigned = checkpoint(&mut mem);
let unsigned_bytes = std::fs::metadata(&path).unwrap().len();
let key = SigningKey::from_bytes(&[7; 32]);
mem.set_signing_key(key.clone());
let signed = checkpoint(&mut mem);
let signed_bytes = std::fs::metadata(&path).unwrap().len();
drop(mem);
let vk = key.verifying_key();
let verify = median(
(0..3)
.map(|_| {
let t = Instant::now();
let r = HDF5Memory::verify(&path, &vk).unwrap();
let d = t.elapsed();
assert!(r.is_valid());
d
})
.collect(),
);
println!(
"| {n} | {:.1} | {:.1} | {:+.1} | {:.1} | {:+.2} |",
millis(unsigned),
millis(signed),
millis(signed) - millis(unsigned),
millis(verify),
(signed_bytes as f64 - unsigned_bytes as f64) / (1024.0 * 1024.0),
);
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Search options study: source filters, re-ranking, confidence rejection // Search options study: source filters, re-ranking, confidence rejection
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -960,6 +1036,21 @@ fn main() {
} }
return; return;
} }
if args.iter().any(|a| a == "--signing-study") {
println!("## Signed checkpoints ({DIM}-dim, float16, int8 index)\n");
println!(
"| N | checkpoint ms, unsigned | checkpoint ms, signed | signing adds ms | verify ms | file MiB added |"
);
println!("|---:|---:|---:|---:|---:|---:|");
for &n in if full {
&[1_000, 10_000, 100_000][..]
} else {
&[1_000, 10_000][..]
} {
signing_study(n);
}
return;
}
if args.iter().any(|a| a == "--options-study") { if args.iter().any(|a| a == "--options-study") {
println!("## Search options ({DIM}-dim, k = {K}, Hebbian boost off)\n"); println!("## Search options ({DIM}-dim, k = {K}, Hebbian boost off)\n");
println!("| N | options | filtered recall@10 | p50 ms | p99 ms |"); println!("| N | options | filtered recall@10 | p50 ms | p99 ms |");
+126 -16
View File
@@ -1,15 +1,22 @@
use std::path::PathBuf; use std::path::{Path, PathBuf};
use clap::{Parser, Subcommand}; use clap::{Parser, Subcommand};
use clawhdf5_agent::signing::{self, SigningKey, VerifyingKey};
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry}; use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
/// ClawhDF5 — HDF5-backed cognitive memory for AI agents /// ClawhDF5 — HDF5-backed cognitive memory for AI agents
#[derive(Parser)] #[derive(Parser)]
#[command(name = "clawhdf5", version, about)] #[command(name = "clawhdf5", version, about)]
struct Cli { struct Cli {
/// Path to the .h5 memory file /// Path to the .h5 memory file (not needed for `keygen`)
#[arg(short, long, env = "CLAWHDF5_PATH")] #[arg(short, long, env = "CLAWHDF5_PATH")]
path: PathBuf, path: Option<PathBuf>,
/// File holding an Ed25519 signing key (64 hex characters, from
/// `keygen`). Every checkpoint this command makes is then signed; a
/// signed store refuses to checkpoint without it.
#[arg(long, env = "CLAWHDF5_SIGNING_KEY", global = true)]
signing_key: Option<PathBuf>,
#[command(subcommand)] #[command(subcommand)]
command: Commands, command: Commands,
@@ -91,6 +98,38 @@ enum Commands {
/// Destination path /// Destination path
dest: PathBuf, dest: PathBuf,
}, },
/// Generate an Ed25519 signing key for signed checkpoints
Keygen {
/// Where to write the secret key (created new, owner-only on Unix)
#[arg(long)]
out: PathBuf,
},
/// Verify a signed store against a public key; exit status 2 if not valid
Verify {
/// The trusted public key: 64 hex characters, or a file holding them
#[arg(long)]
public_key: String,
},
}
fn read_signing_key(path: &Path) -> Result<SigningKey, Box<dyn std::error::Error>> {
let text = std::fs::read_to_string(path)
.map_err(|e| format!("cannot read signing key {}: {e}", path.display()))?;
let bytes = signing::from_hex::<32>(&text)
.ok_or_else(|| format!("{} is not a 64-hex-character key", path.display()))?;
Ok(SigningKey::from_bytes(&bytes))
}
/// Open for writing, with the signing key applied if one was given.
fn open_writable(
path: &Path,
key: &Option<SigningKey>,
) -> Result<HDF5Memory, Box<dyn std::error::Error>> {
let mut mem = HDF5Memory::open(path)?;
if let Some(k) = key {
mem.set_signing_key(k.clone());
}
Ok(mem)
} }
fn main() { fn main() {
@@ -103,6 +142,37 @@ fn main() {
} }
fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> { fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
if let Commands::Keygen { out } = &cli.command {
let key = signing::generate_key();
let mut opts = std::fs::OpenOptions::new();
opts.write(true).create_new(true);
#[cfg(unix)]
{
use std::os::unix::fs::OpenOptionsExt;
opts.mode(0o600);
}
use std::io::Write;
let mut f = opts
.open(out)
.map_err(|e| format!("cannot create {}: {e}", out.display()))?;
writeln!(f, "{}", signing::to_hex(&key.to_bytes()))?;
let j = serde_json::json!({
"status": "generated",
"secret_key_file": out.display().to_string(),
"public_key": signing::to_hex(&key.verifying_key().to_bytes()),
});
println!("{}", serde_json::to_string_pretty(&j)?);
return Ok(());
}
let path = cli
.path
.clone()
.ok_or("--path (or CLAWHDF5_PATH) is required")?;
let key = cli
.signing_key
.as_deref()
.map(read_signing_key)
.transpose()?;
match cli.command { match cli.command {
Commands::Create { Commands::Create {
agent_id, agent_id,
@@ -113,7 +183,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
f32, f32,
float16: _, float16: _,
} => { } => {
let mut config = MemoryConfig::new(cli.path.clone(), &agent_id, dim); let mut config = MemoryConfig::new(path.clone(), &agent_id, dim);
config.wal_enabled = wal; config.wal_enabled = wal;
// As with --f32-index: only ever switch the library default off. // As with --f32-index: only ever switch the library default off.
if f32 { if f32 {
@@ -127,15 +197,21 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
config.quantized_index = false; config.quantized_index = false;
} }
let config_quantized = config.quantized_index; let config_quantized = config.quantized_index;
let mem = HDF5Memory::create(config)?; let mut mem = HDF5Memory::create(config)?;
// Sign straight away, so the store is never on disk unsigned.
if let Some(k) = &key {
mem.set_signing_key(k.clone());
mem.flush_wal()?;
}
let j = serde_json::json!({ let j = serde_json::json!({
"status": "created", "status": "created",
"path": cli.path.display().to_string(), "path": path.display().to_string(),
"agent_id": agent_id, "agent_id": agent_id,
"embedding_dim": dim, "embedding_dim": dim,
"wal_enabled": wal, "wal_enabled": wal,
"quantized_index": config_quantized, "quantized_index": config_quantized,
"float16": config_float16, "float16": config_float16,
"signed": mem.is_signed(),
"count": mem.count(), "count": mem.count(),
}); });
println!("{}", serde_json::to_string_pretty(&j)?); println!("{}", serde_json::to_string_pretty(&j)?);
@@ -152,7 +228,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
} }
}; };
let entry: MemoryEntry = serde_json::from_str(&input)?; let entry: MemoryEntry = serde_json::from_str(&input)?;
let mut mem = HDF5Memory::open(&cli.path)?; let mut mem = open_writable(&path, &key)?;
let idx = mem.save(entry)?; let idx = mem.save(entry)?;
let j = serde_json::json!({ "status": "saved", "index": idx, "count": mem.count() }); let j = serde_json::json!({ "status": "saved", "index": idx, "count": mem.count() });
println!("{}", serde_json::to_string(&j)?); println!("{}", serde_json::to_string(&j)?);
@@ -166,7 +242,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
keyword_weight, keyword_weight,
} => { } => {
let emb: Vec<f32> = serde_json::from_str(&embedding)?; let emb: Vec<f32> = serde_json::from_str(&embedding)?;
let mut mem = HDF5Memory::open(&cli.path)?; let mut mem = open_writable(&path, &key)?;
let results = mem.hybrid_search(&emb, &query, vector_weight, keyword_weight, top_k); let results = mem.hybrid_search(&emb, &query, vector_weight, keyword_weight, top_k);
let j: Vec<serde_json::Value> = results let j: Vec<serde_json::Value> = results
.iter() .iter()
@@ -184,7 +260,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
} }
Commands::Recall { index } => { Commands::Recall { index } => {
let mem = HDF5Memory::open_read_only(&cli.path)?; let mem = HDF5Memory::open_read_only(&path)?;
match mem.get_chunk(index) { match mem.get_chunk(index) {
Some(content) => { Some(content) => {
let j = serde_json::json!({ "index": index, "chunk": content }); let j = serde_json::json!({ "index": index, "chunk": content });
@@ -198,22 +274,23 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
} }
Commands::Stats => { Commands::Stats => {
let mem = HDF5Memory::open_read_only(&cli.path)?; let mem = HDF5Memory::open_read_only(&path)?;
let cfg = mem.config(); let cfg = mem.config();
let j = serde_json::json!({ let j = serde_json::json!({
"path": cli.path.display().to_string(), "path": path.display().to_string(),
"agent_id": cfg.agent_id, "agent_id": cfg.agent_id,
"embedding_dim": cfg.embedding_dim, "embedding_dim": cfg.embedding_dim,
"count": mem.count(), "count": mem.count(),
"active": mem.count_active(), "active": mem.count_active(),
"wal_enabled": cfg.wal_enabled, "wal_enabled": cfg.wal_enabled,
"wal_pending": mem.wal_pending_count(), "wal_pending": mem.wal_pending_count(),
"signed": mem.is_signed(),
}); });
println!("{}", serde_json::to_string_pretty(&j)?); println!("{}", serde_json::to_string_pretty(&j)?);
} }
Commands::FlushWal => { Commands::FlushWal => {
let mut mem = HDF5Memory::open(&cli.path)?; let mut mem = open_writable(&path, &key)?;
let before = mem.wal_pending_count(); let before = mem.wal_pending_count();
mem.flush_wal()?; mem.flush_wal()?;
let j = serde_json::json!({ let j = serde_json::json!({
@@ -225,7 +302,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
} }
Commands::AgentsMd { output } => { Commands::AgentsMd { output } => {
let mem = HDF5Memory::open_read_only(&cli.path)?; let mem = HDF5Memory::open_read_only(&path)?;
let md = mem.generate_agents_md(); let md = mem.generate_agents_md();
match output { match output {
Some(p) => { Some(p) => {
@@ -237,7 +314,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
} }
Commands::Export => { Commands::Export => {
let mem = HDF5Memory::open_read_only(&cli.path)?; let mem = HDF5Memory::open_read_only(&path)?;
for i in 0..mem.count() { for i in 0..mem.count() {
if let Some(chunk) = mem.get_chunk(i) { if let Some(chunk) = mem.get_chunk(i) {
let j = serde_json::json!({ "index": i, "chunk": chunk }); let j = serde_json::json!({ "index": i, "chunk": chunk });
@@ -246,11 +323,44 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
} }
} }
Commands::Keygen { .. } => unreachable!("handled before opening a store"),
Commands::Verify { public_key } => {
let text = if Path::new(&public_key).is_file() {
std::fs::read_to_string(&public_key)?
} else {
public_key
};
let bytes = signing::from_hex::<32>(&text)
.ok_or("--public-key must be 64 hex characters or a file holding them")?;
let trusted = VerifyingKey::from_bytes(&bytes)?;
let r = HDF5Memory::verify(&path, &trusted)?;
let j = serde_json::json!({
"valid": r.is_valid(),
"signed": r.signed,
"key_matches": r.key_matches,
"signature_valid": r.signature_valid,
"records_match": r.records_match,
"settings_match": r.settings_match,
"sessions_match": r.sessions_match,
"graph_match": r.graph_match,
"changed_records": r.changed_records,
"record_count": r.record_count,
"signed_record_count": r.signed_record_count,
"signed_by": r.public_key.map(|k| signing::to_hex(&k)),
"wal_entries_unsigned": r.wal_entries_unsigned,
});
println!("{}", serde_json::to_string_pretty(&j)?);
if !r.is_valid() {
std::process::exit(2);
}
}
Commands::Snapshot { dest } => { Commands::Snapshot { dest } => {
let _result = clawhdf5_agent::storage::snapshot_file(&cli.path, &dest)?; let _result = clawhdf5_agent::storage::snapshot_file(&path, &dest)?;
let j = serde_json::json!({ let j = serde_json::json!({
"status": "snapshot_created", "status": "snapshot_created",
"source": cli.path.display().to_string(), "source": path.display().to_string(),
"dest": dest.display().to_string(), "dest": dest.display().to_string(),
}); });
println!("{}", serde_json::to_string(&j)?); println!("{}", serde_json::to_string(&j)?);
+1 -1
View File
@@ -27,7 +27,7 @@ The core use case. Give your AI agent persistent, searchable memory in a single
```toml ```toml
# Cargo.toml # Cargo.toml
[dependencies] [dependencies]
clawhdf5-agent = { version = "2.0", features = ["agent"] } clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } # not on crates.io yet
``` ```
### Create a Memory Store ### Create a Memory Store
+5 -4
View File
@@ -212,7 +212,7 @@ This is the container image for intelligence.
| Your Situation | Features to Enable | Why | | Your Situation | Features to Enable | Why |
|----------------|-------------------|-----| |----------------|-------------------|-----|
| **Quick prototype** | Default | Vector search works out of the box | | **Quick prototype** | Default | Vector search works out of the box |
| **Production agent** | `agent`, `float16`, `parallel` | Half-precision saves 50% storage, parallel search for scale | | **Production agent** | defaults (`float16`, `hnsw`, `parallel`) | HNSW search and a parallel index build; half-precision *storage* is `MemoryConfig::float16`, on by default for new stores |
| **macOS** | + `accelerate` | Apple AMX coprocessor for matrix ops | | **macOS** | + `accelerate` | Apple AMX coprocessor for matrix ops |
| **Linux server** | + `openblas` or `fast-math` | BLAS acceleration | | **Linux server** | + `openblas` or `fast-math` | BLAS acceleration |
| **GPU available** | + `gpu` | wgpu-based search, wins at 100K+ scale | | **GPU available** | + `gpu` | wgpu-based search, wins at 100K+ scale |
@@ -220,14 +220,15 @@ This is the container image for intelligence.
| **Edge device** | Default only | Minimal dependencies, smallest binary | | **Edge device** | Default only | Minimal dependencies, smallest binary |
```toml ```toml
# Not on crates.io yet: depend on the repository.
# Production agent on Linux # Production agent on Linux
clawhdf5-agent = { version = "2.0", features = ["agent", "float16", "parallel", "fast-math"] } clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5", features = ["fast-math"] }
# Edge device # Edge device
clawhdf5-agent = { version = "2.0", features = ["agent"] } clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
# macOS with GPU # macOS with GPU
clawhdf5-agent = { version = "2.0", features = ["agent", "float16", "accelerate", "gpu", "async"] } clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5", features = ["accelerate", "gpu", "async"] }
``` ```
--- ---