Ed25519-signed checkpoints; remove the no-op agent feature #8
@@ -243,6 +243,30 @@ index asked for ~16 000 candidates, where scanning the few hundred or thousand
|
||||
allowed records is exact and cheap. Re-ranking a 3k candidate pool and
|
||||
confidence rejection add about 3%.
|
||||
|
||||
### Signed checkpoints
|
||||
|
||||
Measured 2026-09-25 on tank (AMD Ryzen 7 7800X3D). A default store (float16,
|
||||
int8 index), 384-dim; each checkpoint rewrites the whole file, as every
|
||||
checkpoint does. Medians of five checkpoints and three verifies; three runs
|
||||
agreed to within the ranges shown.
|
||||
|
||||
```bash
|
||||
cargo run --release -p clawhdf5-bench --bin search_harness -- --signing-study --full
|
||||
```
|
||||
|
||||
| N | checkpoint, unsigned | checkpoint, signed | signing adds | `verify` | file size added |
|
||||
|---:|---:|---:|---:|---:|---:|
|
||||
| 1 000 | 5.4 ms | 6.4 ms | 0.7–1.0 ms | 2.1 ms | 0.03 MiB |
|
||||
| 10 000 | 46 ms | 55 ms | 8.1–9.4 ms | 18.6 ms | 0.31 MiB |
|
||||
| 100 000 | 495 ms | 598 ms | 89–112 ms | 247 ms | 3.05 MiB |
|
||||
|
||||
Signing costs about 20% of a checkpoint: every record is rehashed (SHA-256)
|
||||
and the Merkle root recomputed each time; the Ed25519 signature itself is
|
||||
microseconds. Caching per-record hashes between checkpoints would cut this to
|
||||
the records that changed. The per-record hashes stored for locating edits are
|
||||
32 bytes each (4% of a 100K float16 store). `verify` reads and rehashes the
|
||||
whole checkpoint.
|
||||
|
||||
### float16 embedding storage (`MemoryConfig::float16`)
|
||||
|
||||
Measured 2026-09-23 on tank (AMD Ryzen 7 7800X3D). The same clustered
|
||||
|
||||
@@ -3,6 +3,9 @@
|
||||
## Unreleased
|
||||
|
||||
### Upgrade Notes
|
||||
- **Breaking:** `MemoryError` is now `#[non_exhaustive]` and gained
|
||||
`SigningKeyRequired`; a `match` on it needs a wildcard arm. Future variants
|
||||
will no longer be breaking.
|
||||
- **Breaking:** `clawhdf5-agent`'s `agent` feature is removed. It enabled
|
||||
nothing — the agent layer is always built — but the README and guides told
|
||||
people to pass it; drop `agent` from `features = [...]`.
|
||||
@@ -60,6 +63,29 @@
|
||||
`quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
|
||||
out. The CLI's `--quantized-index` is still accepted but is now a no-op.
|
||||
|
||||
### Signing
|
||||
- `clawhdf5-agent`: **Ed25519-signed checkpoints** — the README's
|
||||
"cryptographically verifiable memory", now true. With
|
||||
`HDF5Memory::set_signing_key(key)`, every checkpoint stores a signed
|
||||
manifest: a SHA-256 per record (text, embedding as stored, channel,
|
||||
timestamp, session, tags, deleted flag, activation) in a Merkle tree, plus
|
||||
hashes of the settings (and WAL mark), sessions and knowledge graph, with
|
||||
the per-record hashes in `/integrity/record_hashes`.
|
||||
`HDF5Memory::verify(path, &public_key)` recomputes everything from the file
|
||||
and reports which part changed and which records (`changed_records`); a
|
||||
forged manifest fails the signature. The key is never persisted; a signed
|
||||
store refuses to checkpoint without it (`MemoryError::SigningKeyRequired`),
|
||||
and `remove_signature()` is the deliberate way back to unsigned. Saves still
|
||||
in the WAL are not covered (`wal_entries_unsigned`). Tests include every
|
||||
kind of edit, and an edit made with h5py in place, which verify pinpoints.
|
||||
Cost: ~20% of a checkpoint, 32 bytes per record (`BENCHMARKS.md`, "Signed
|
||||
checkpoints"). New dependencies `ed25519-dalek`, `sha2`, `rand_core` — pure
|
||||
Rust; the no-C check still passes.
|
||||
- `clawhdf5-cli`: `keygen --out <file>` (owner-only key file),
|
||||
`--signing-key <file>` / `CLAWHDF5_SIGNING_KEY` on writing commands
|
||||
(`create` signs immediately), `verify --public-key <hex|file>` (JSON report;
|
||||
exit status 2 if not valid), and `signed` in `create`/`stats` output.
|
||||
|
||||
### Migration
|
||||
- `clawhdf5-migrate`: writes through the agent's own API (`HDF5Memory::create`
|
||||
/ `open`, `save_batch`, the session cache and knowledge graph), so there is
|
||||
|
||||
@@ -109,6 +109,17 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
|
||||
`search_harness --options-study`.
|
||||
- `MemoryConfig::compression` is off by default; when on, embeddings are
|
||||
deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd).
|
||||
- Signed checkpoints (`clawhdf5-agent` `signing` module): with
|
||||
`HDF5Memory::set_signing_key` every checkpoint stores an Ed25519-signed
|
||||
manifest (SHA-256 per record in a Merkle tree + settings/sessions/graph
|
||||
hashes; per-record hashes in `/integrity/record_hashes`);
|
||||
`HDF5Memory::verify(path, &pk)` locates edits. The hashes must cover exactly
|
||||
what the file persists in the form the loader returns it (strings lose
|
||||
trailing NULs; an empty WAL mark is not written) or untouched stores stop
|
||||
verifying — `tests/signed_store.rs` round-trips awkward strings. The key is
|
||||
never persisted; a signed store refuses to checkpoint without it
|
||||
(`MemoryError::SigningKeyRequired`, and `MemoryError` is `#[non_exhaustive]`).
|
||||
WAL entries after the checkpoint are not covered.
|
||||
- `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by
|
||||
default) recomputes a dataset's SHA-256 and compares it against the
|
||||
`_provenance_sha256` attribute written automatically on save when
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
[](BENCHMARKS.md#longmemeval-results)
|
||||
[](BENCHMARKS.md#memory-footprint-1)
|
||||
|
||||
ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file.
|
||||
ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, cryptographically verifiable memory (Ed25519-signed checkpoints) — all stored in a single portable file.
|
||||
|
||||
> **Two things live here:**
|
||||
> - **A general-purpose, pure-Rust HDF5 library** — zero C dependencies, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers.
|
||||
@@ -113,7 +113,7 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d
|
||||
| Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers |
|
||||
| Temporal queries | Custom code | Native temporal index (622 ns range query over 10K) |
|
||||
| Multi-modal | Multiple stores | Unified cross-modal search (exact scan: 842 µs over 1K records) |
|
||||
| Integrity | Hope for the best | Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance |
|
||||
| Integrity | Hope for the best | Ed25519-signed checkpoints that pinpoint any edited record, chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts |
|
||||
| Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** |
|
||||
|
||||
---
|
||||
@@ -369,6 +369,7 @@ directly; the store persists the records, sessions and graph they work over.
|
||||
| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches. Opt-in via `SearchOptions::with_confidence`; on in the OpenClaw backend |
|
||||
| **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints |
|
||||
| **`multimodal`** | Cross-modal search across text/image/audio/video embeddings |
|
||||
| **`signing`** | Ed25519-signed checkpoints: SHA-256 per record in a Merkle tree, plus hashes of settings, sessions and the knowledge graph; `HDF5Memory::verify` names any edited record |
|
||||
| **`provenance`** | Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof) |
|
||||
| **`anomaly`** | Write rate limiting, 15 injection-pattern detectors, source-distribution analysis. Alerts never block a save; drain them with `take_anomaly_alerts` |
|
||||
| **`openclaw`** | OpenClaw integration: MemoryBackend trait, Markdown ↔ HDF5 conversion |
|
||||
@@ -457,6 +458,35 @@ let careful = memory.search(
|
||||
);
|
||||
```
|
||||
|
||||
### Signed Checkpoints
|
||||
|
||||
```rust
|
||||
use clawhdf5_agent::signing;
|
||||
|
||||
// Once, somewhere safe: keep the secret key, publish the public key.
|
||||
let key = signing::generate_key();
|
||||
let public = key.verifying_key();
|
||||
|
||||
// Every checkpoint is signed from now on. The key is never written to disk;
|
||||
// a signed store refuses to checkpoint without it.
|
||||
memory.set_signing_key(key);
|
||||
memory.flush_wal()?;
|
||||
|
||||
// Anyone holding the public key can check the file, e.g. after copying it.
|
||||
let report = HDF5Memory::verify(std::path::Path::new("agent.h5"), &public)?;
|
||||
assert!(report.is_valid());
|
||||
// On a tampered file: report.changed_records lists the records that differ.
|
||||
```
|
||||
|
||||
The signature covers every record (text, embedding as stored, channel,
|
||||
timestamp, session, tags, deleted flag, activation), the store's settings,
|
||||
its sessions and its knowledge graph — a change made with any tool is caught.
|
||||
It covers checkpoints, not saves still in the WAL
|
||||
(`report.wal_entries_unsigned` counts those). CLI: `clawhdf5-cli keygen`,
|
||||
`--signing-key <file>` on writing commands, and `verify --public-key`.
|
||||
Signing adds about 20% to a checkpoint and 32 bytes per record to the file
|
||||
([BENCHMARKS.md § Signed checkpoints](BENCHMARKS.md#signed-checkpoints)).
|
||||
|
||||
### Knowledge Graph
|
||||
|
||||
```rust
|
||||
|
||||
@@ -19,6 +19,10 @@ clawhdf5-ann = { path = "../clawhdf5-ann", version = "2.7.0", optional = true }
|
||||
clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.7.0", optional = true, default-features = false }
|
||||
serde = { workspace = true }
|
||||
byteorder = "1"
|
||||
# Signed checkpoints (MemoryConfig-independent; see `signing`). Pure Rust.
|
||||
ed25519-dalek = { version = "2", features = ["rand_core"] }
|
||||
sha2 = "0.10"
|
||||
rand_core = { version = "0.6", features = ["getrandom"] }
|
||||
half = { workspace = true, optional = true }
|
||||
rayon = { version = "1", optional = true }
|
||||
matrixmultiply = { version = "0.3", optional = true }
|
||||
|
||||
@@ -36,6 +36,7 @@ pub mod reranker;
|
||||
pub mod schema;
|
||||
pub mod search;
|
||||
pub mod session;
|
||||
pub mod signing;
|
||||
pub mod storage;
|
||||
mod store_lock;
|
||||
pub mod temporal;
|
||||
@@ -78,6 +79,7 @@ pub use session::{SessionCache, SessionEntry};
|
||||
// --- Error type ---
|
||||
|
||||
#[derive(Debug)]
|
||||
#[non_exhaustive]
|
||||
pub enum MemoryError {
|
||||
Io(std::io::Error),
|
||||
Hdf5(String),
|
||||
@@ -88,6 +90,11 @@ pub enum MemoryError {
|
||||
/// A record the store cannot hold as given, e.g. an embedding value
|
||||
/// outside the half-precision range of a `float16` store.
|
||||
InvalidEntry(String),
|
||||
/// The store's checkpoints are signed and no signing key is set, so a
|
||||
/// checkpoint would leave it unsigned. Set the key with
|
||||
/// [`HDF5Memory::set_signing_key`], or drop the signature on purpose with
|
||||
/// [`HDF5Memory::remove_signature`].
|
||||
SigningKeyRequired(String),
|
||||
}
|
||||
|
||||
impl std::fmt::Display for MemoryError {
|
||||
@@ -99,6 +106,7 @@ impl std::fmt::Display for MemoryError {
|
||||
MemoryError::NotFound(e) => write!(f, "not found: {e}"),
|
||||
MemoryError::Locked(e) => write!(f, "store is locked: {e}"),
|
||||
MemoryError::InvalidEntry(e) => write!(f, "invalid entry: {e}"),
|
||||
MemoryError::SigningKeyRequired(e) => write!(f, "signing key required: {e}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -317,6 +325,12 @@ pub struct HDF5Memory {
|
||||
activations_dirty: bool,
|
||||
/// Opened with [`HDF5Memory::open_read_only`]: nothing may reach the disk.
|
||||
read_only: bool,
|
||||
/// Key that signs every checkpoint; never persisted. See
|
||||
/// [`HDF5Memory::set_signing_key`].
|
||||
signing_key: Option<signing::SigningKey>,
|
||||
/// Checkpoints of this store are signed: the file on disk is, or a key
|
||||
/// has been set. A checkpoint without a key is then refused.
|
||||
signed: bool,
|
||||
/// A WAL that `open()` could not read and moved aside; see
|
||||
/// [`HDF5Memory::quarantined_wal`].
|
||||
quarantined_wal: Option<PathBuf>,
|
||||
@@ -372,6 +386,8 @@ impl HDF5Memory {
|
||||
bm25_filter: bm25::TokenFilter::default(),
|
||||
activations_dirty: false,
|
||||
read_only: false,
|
||||
signing_key: None,
|
||||
signed: false,
|
||||
quarantined_wal: None,
|
||||
_lock: Some(lock),
|
||||
})
|
||||
@@ -550,6 +566,8 @@ impl HDF5Memory {
|
||||
bm25_filter: bm25::TokenFilter::default(),
|
||||
activations_dirty: false,
|
||||
read_only,
|
||||
signing_key: None,
|
||||
signed: checkpoint.signed,
|
||||
quarantined_wal,
|
||||
_lock: lock,
|
||||
})
|
||||
@@ -710,6 +728,39 @@ impl HDF5Memory {
|
||||
}
|
||||
}
|
||||
|
||||
/// Sign every checkpoint from now on with `key` (Ed25519). The key is
|
||||
/// never written anywhere; set it again after every `open`. Once a store
|
||||
/// is signed, a checkpoint without the key is refused
|
||||
/// ([`MemoryError::SigningKeyRequired`]) rather than silently leaving it
|
||||
/// unsigned. Setting a different key re-signs the store under that key
|
||||
/// from the next checkpoint; a verifier trusting the old key will then
|
||||
/// reject it, which is the point. Call [`AgentMemory::flush_wal`] to sign
|
||||
/// right away.
|
||||
pub fn set_signing_key(&mut self, key: signing::SigningKey) {
|
||||
self.signing_key = Some(key);
|
||||
self.signed = true;
|
||||
}
|
||||
|
||||
/// Stop signing: the next checkpoint writes the store unsigned. The
|
||||
/// deliberate way out of [`MemoryError::SigningKeyRequired`].
|
||||
pub fn remove_signature(&mut self) {
|
||||
self.signing_key = None;
|
||||
self.signed = false;
|
||||
}
|
||||
|
||||
/// Checkpoints of this store are signed (on disk, or from the next
|
||||
/// checkpoint because a key has been set).
|
||||
pub fn is_signed(&self) -> bool {
|
||||
self.signed
|
||||
}
|
||||
|
||||
/// Check the checkpoint at `path` against the public key the caller
|
||||
/// trusts; see [`signing::verify_store`]. Reads the file only: it works
|
||||
/// on a store another process has open.
|
||||
pub fn verify(path: &Path, trusted: &signing::VerifyingKey) -> Result<signing::VerifyReport> {
|
||||
signing::verify_store(path, trusted)
|
||||
}
|
||||
|
||||
/// Flush current state to disk and truncate the WAL.
|
||||
///
|
||||
/// Every code path that persists the full cache to the .h5 file must
|
||||
@@ -725,10 +776,28 @@ impl HDF5Memory {
|
||||
// Record which WAL prefix this checkpoint contains, so a crash before
|
||||
// the truncate below can't replay those entries a second time.
|
||||
let wal_applied = self.wal.as_ref().map(|w| w.mark());
|
||||
let signature = match &self.signing_key {
|
||||
Some(key) => Some(signing::sign(
|
||||
key,
|
||||
&self.config,
|
||||
&self.cache,
|
||||
&self.sessions,
|
||||
&self.knowledge,
|
||||
wal_applied,
|
||||
)),
|
||||
None if self.signed => {
|
||||
return Err(MemoryError::SigningKeyRequired(format!(
|
||||
"{} is signed; set its signing key before a checkpoint \
|
||||
(saves so far are held in the WAL or in memory)",
|
||||
self.config.path.display()
|
||||
)));
|
||||
}
|
||||
None => None,
|
||||
};
|
||||
// Written before the .h5 so a crash in between leaves a sidecar whose
|
||||
// generation matches no checkpoint (ignored), never the reverse.
|
||||
let ann_generation = self.persist_vector_index();
|
||||
storage::write_to_disk_with_meta(
|
||||
storage::write_to_disk_signed(
|
||||
&self.config.path,
|
||||
&self.config,
|
||||
&self.cache,
|
||||
@@ -737,7 +806,9 @@ impl HDF5Memory {
|
||||
&schema::CheckpointMeta {
|
||||
wal_applied,
|
||||
ann_generation,
|
||||
signed: signature.is_some(),
|
||||
},
|
||||
signature.as_ref(),
|
||||
)?;
|
||||
if let Some(ref mut w) = self.wal {
|
||||
w.truncate()?;
|
||||
|
||||
@@ -23,6 +23,7 @@ pub const ZEROCLAW_VERSION: &str = "0.8.0";
|
||||
const WAL_APPLIED_LEN_ATTR: &str = "wal_applied_len";
|
||||
const WAL_APPLIED_CRC_ATTR: &str = "wal_applied_crc";
|
||||
const ANN_GENERATION_ATTR: &str = "ann_generation";
|
||||
const SIG_VERSION_ATTR: &str = "sig_version";
|
||||
|
||||
/// Build a complete HDF5 file from the in-memory state.
|
||||
pub fn build_hdf5_file(
|
||||
@@ -46,7 +47,7 @@ pub fn build_hdf5_file_with_mark(
|
||||
) -> Result<Vec<u8>, MemoryError> {
|
||||
let meta = CheckpointMeta {
|
||||
wal_applied,
|
||||
ann_generation: None,
|
||||
..CheckpointMeta::default()
|
||||
};
|
||||
build_hdf5_file_with_meta(config, cache, sessions, knowledge, &meta)
|
||||
}
|
||||
@@ -61,6 +62,10 @@ pub struct CheckpointMeta {
|
||||
/// one left over from another checkpoint can never be attached to records
|
||||
/// it wasn't built from.
|
||||
pub ann_generation: Option<u64>,
|
||||
/// The checkpoint carries an Ed25519 signature (see [`crate::signing`]).
|
||||
/// Read-only: whether a checkpoint is *written* signed is decided by the
|
||||
/// signature passed to [`build_hdf5_file_signed`].
|
||||
pub signed: bool,
|
||||
}
|
||||
|
||||
/// [`build_hdf5_file`] with checkpoint bookkeeping.
|
||||
@@ -70,6 +75,19 @@ pub fn build_hdf5_file_with_meta(
|
||||
sessions: &SessionCache,
|
||||
knowledge: &KnowledgeCache,
|
||||
checkpoint: &CheckpointMeta,
|
||||
) -> Result<Vec<u8>, MemoryError> {
|
||||
build_hdf5_file_signed(config, cache, sessions, knowledge, checkpoint, None)
|
||||
}
|
||||
|
||||
/// [`build_hdf5_file_with_meta`], plus a signed manifest of the contents
|
||||
/// (see [`crate::signing`]).
|
||||
pub fn build_hdf5_file_signed(
|
||||
config: &MemoryConfig,
|
||||
cache: &MemoryCache,
|
||||
sessions: &SessionCache,
|
||||
knowledge: &KnowledgeCache,
|
||||
checkpoint: &CheckpointMeta,
|
||||
signature: Option<&crate::signing::StoredSignature>,
|
||||
) -> Result<Vec<u8>, MemoryError> {
|
||||
let wal_applied = checkpoint.wal_applied;
|
||||
let mut builder = clawhdf5::FileBuilder::new();
|
||||
@@ -130,11 +148,42 @@ pub fn build_hdf5_file_with_meta(
|
||||
// round trip through every reader.
|
||||
meta.set_attr(ANN_GENERATION_ATTR, AttrValue::I64(generation as i64));
|
||||
}
|
||||
if let Some(sig) = signature {
|
||||
use crate::signing::to_hex;
|
||||
let m = &sig.manifest;
|
||||
meta.set_attr(
|
||||
SIG_VERSION_ATTR,
|
||||
AttrValue::I64(crate::signing::MANIFEST_VERSION),
|
||||
);
|
||||
meta.set_attr("sig_algorithm", AttrValue::String("ed25519".into()));
|
||||
meta.set_attr("sig_public_key", AttrValue::String(to_hex(&sig.public_key)));
|
||||
meta.set_attr("sig_signature", AttrValue::String(to_hex(&sig.signature)));
|
||||
meta.set_attr("sig_record_count", AttrValue::I64(m.record_count as i64));
|
||||
meta.set_attr(
|
||||
"sig_records_root",
|
||||
AttrValue::String(to_hex(&m.records_root)),
|
||||
);
|
||||
meta.set_attr("sig_settings", AttrValue::String(to_hex(&m.settings)));
|
||||
meta.set_attr("sig_sessions", AttrValue::String(to_hex(&m.sessions)));
|
||||
meta.set_attr("sig_graph", AttrValue::String(to_hex(&m.graph)));
|
||||
}
|
||||
// Need at least one dataset in the group for it to be a proper group
|
||||
meta.create_dataset("_marker").with_u8_data(&[1]).compact();
|
||||
let finished_meta = meta.finish();
|
||||
builder.add_group(finished_meta);
|
||||
|
||||
// /integrity: the signed per-record hashes, so verification can say
|
||||
// which records changed.
|
||||
if let Some(sig) = signature {
|
||||
let mut group = builder.create_group("integrity");
|
||||
let flat: Vec<u8> = sig.record_hashes.iter().flatten().copied().collect();
|
||||
group
|
||||
.create_dataset("record_hashes")
|
||||
.with_u8_data(&flat)
|
||||
.with_shape(&[sig.record_hashes.len() as u64, 32]);
|
||||
builder.add_group(group.finish());
|
||||
}
|
||||
|
||||
// /memory group
|
||||
build_memory_group(&mut builder, config, cache)?;
|
||||
|
||||
@@ -440,6 +489,64 @@ pub fn read_wal_mark(file: &clawhdf5::File) -> Option<WalMark> {
|
||||
Some(WalMark { len, crc })
|
||||
}
|
||||
|
||||
/// Read a checkpoint's signature, if it has one. A signature whose
|
||||
/// attributes are present but malformed is an error, not "unsigned".
|
||||
pub fn read_signature(
|
||||
file: &clawhdf5::File,
|
||||
) -> Result<Option<crate::signing::StoredSignature>, MemoryError> {
|
||||
use crate::signing::{Manifest, StoredSignature, from_hex};
|
||||
let attrs = file
|
||||
.group("meta")
|
||||
.and_then(|g| g.attrs())
|
||||
.map_err(|e| MemoryError::Schema(format!("cannot read /meta attrs: {e}")))?;
|
||||
let version = match attrs.get(SIG_VERSION_ATTR) {
|
||||
None => return Ok(None),
|
||||
Some(AttrValue::I64(v)) => *v,
|
||||
Some(_) => return Err(MemoryError::Schema("malformed sig_version".into())),
|
||||
};
|
||||
if version != crate::signing::MANIFEST_VERSION {
|
||||
return Err(MemoryError::Schema(format!(
|
||||
"unsupported signature version {version}"
|
||||
)));
|
||||
}
|
||||
fn hex<const N: usize>(
|
||||
attrs: &std::collections::HashMap<String, AttrValue>,
|
||||
name: &str,
|
||||
) -> Result<[u8; N], MemoryError> {
|
||||
match attrs.get(name) {
|
||||
Some(AttrValue::String(s)) => from_hex::<N>(s),
|
||||
_ => None,
|
||||
}
|
||||
.ok_or_else(|| MemoryError::Schema(format!("malformed or missing {name}")))
|
||||
}
|
||||
let record_count = match attrs.get("sig_record_count") {
|
||||
Some(AttrValue::I64(v)) if *v >= 0 => *v as u64,
|
||||
_ => return Err(MemoryError::Schema("malformed sig_record_count".into())),
|
||||
};
|
||||
let group = file
|
||||
.group("integrity")
|
||||
.map_err(|e| MemoryError::Schema(format!("signed checkpoint without /integrity: {e}")))?;
|
||||
let flat = read_u8_dataset(&group, "record_hashes")?;
|
||||
if flat.len() % 32 != 0 {
|
||||
return Err(MemoryError::Schema(
|
||||
"/integrity/record_hashes is not a whole number of hashes".into(),
|
||||
));
|
||||
}
|
||||
let record_hashes = flat.as_chunks::<32>().0.to_vec();
|
||||
Ok(Some(StoredSignature {
|
||||
manifest: Manifest {
|
||||
record_count,
|
||||
records_root: hex::<32>(&attrs, "sig_records_root")?,
|
||||
settings: hex::<32>(&attrs, "sig_settings")?,
|
||||
sessions: hex::<32>(&attrs, "sig_sessions")?,
|
||||
graph: hex::<32>(&attrs, "sig_graph")?,
|
||||
},
|
||||
record_hashes,
|
||||
public_key: hex::<32>(&attrs, "sig_public_key")?,
|
||||
signature: hex::<64>(&attrs, "sig_signature")?,
|
||||
}))
|
||||
}
|
||||
|
||||
/// Read the checkpoint bookkeeping from `/meta`.
|
||||
pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta {
|
||||
let ann_generation = file
|
||||
@@ -450,9 +557,14 @@ pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta {
|
||||
Some(AttrValue::I64(v)) => Some(*v as u64),
|
||||
_ => None,
|
||||
});
|
||||
let signed = file
|
||||
.group("meta")
|
||||
.and_then(|g| g.attrs())
|
||||
.is_ok_and(|attrs| attrs.contains_key(SIG_VERSION_ATTR));
|
||||
CheckpointMeta {
|
||||
wal_applied: read_wal_mark(file),
|
||||
ann_generation,
|
||||
signed,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,419 @@
|
||||
//! Ed25519-signed checkpoints.
|
||||
//!
|
||||
//! When a signing key is set ([`crate::HDF5Memory::set_signing_key`]), every
|
||||
//! checkpoint writes a signed manifest of the store: a SHA-256 per memory
|
||||
//! record rolled into a Merkle root, plus hashes of the store's settings, its
|
||||
//! sessions and its knowledge graph. [`verify_store`] recomputes all of it from
|
||||
//! the file and checks the signature against a public key the caller trusts,
|
||||
//! so any change to the checkpointed file — a record's text or embedding, a
|
||||
//! setting, a session, a graph edge, made through this crate or any other HDF5
|
||||
//! tool — is detected, and the per-record hashes say which records changed.
|
||||
//!
|
||||
//! What it does not cover: saves still only in the WAL (made since the last
|
||||
//! checkpoint). [`VerifyReport::wal_entries_unsigned`] counts them.
|
||||
//!
|
||||
//! The hashes cover exactly what the file persists, in the form the loader
|
||||
//! returns it, so a store verifies after any number of reopen/checkpoint
|
||||
//! cycles. Derived data (L2 norms, the vector index) is not covered; it is
|
||||
//! recomputed from covered data.
|
||||
|
||||
use ed25519_dalek::{Signature, Signer, Verifier};
|
||||
pub use ed25519_dalek::{SigningKey, VerifyingKey};
|
||||
use sha2::{Digest, Sha256};
|
||||
|
||||
use crate::MemoryConfig;
|
||||
use crate::cache::MemoryCache;
|
||||
use crate::knowledge::KnowledgeCache;
|
||||
use crate::session::SessionCache;
|
||||
use crate::wal::WalMark;
|
||||
|
||||
/// Version of the manifest encoding; part of what is signed.
|
||||
pub const MANIFEST_VERSION: i64 = 1;
|
||||
|
||||
type Hash = [u8; 32];
|
||||
|
||||
/// The hashes a signature covers.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct Manifest {
|
||||
pub record_count: u64,
|
||||
/// Merkle root over the per-record hashes.
|
||||
pub records_root: Hash,
|
||||
/// Settings persisted in `/meta`, plus the checkpoint's WAL mark.
|
||||
pub settings: Hash,
|
||||
pub sessions: Hash,
|
||||
pub graph: Hash,
|
||||
}
|
||||
|
||||
impl Manifest {
|
||||
/// The exact bytes that are signed.
|
||||
pub fn signed_bytes(&self) -> Vec<u8> {
|
||||
let mut m = Vec::with_capacity(160);
|
||||
m.extend_from_slice(b"clawhdf5-agent signed checkpoint\0");
|
||||
m.extend_from_slice(&MANIFEST_VERSION.to_le_bytes());
|
||||
m.extend_from_slice(&self.record_count.to_le_bytes());
|
||||
m.extend_from_slice(&self.records_root);
|
||||
m.extend_from_slice(&self.settings);
|
||||
m.extend_from_slice(&self.sessions);
|
||||
m.extend_from_slice(&self.graph);
|
||||
m
|
||||
}
|
||||
}
|
||||
|
||||
/// A signature as stored in a checkpoint.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct StoredSignature {
|
||||
pub manifest: Manifest,
|
||||
pub record_hashes: Vec<Hash>,
|
||||
pub public_key: [u8; 32],
|
||||
pub signature: [u8; 64],
|
||||
}
|
||||
|
||||
/// Build the manifest (and per-record hashes) for the state about to be
|
||||
/// checkpointed, and sign it.
|
||||
pub fn sign(
|
||||
key: &SigningKey,
|
||||
config: &MemoryConfig,
|
||||
cache: &MemoryCache,
|
||||
sessions: &SessionCache,
|
||||
knowledge: &KnowledgeCache,
|
||||
wal_applied: Option<WalMark>,
|
||||
) -> StoredSignature {
|
||||
let (manifest, record_hashes) = manifest(config, cache, sessions, knowledge, wal_applied);
|
||||
let signature = key.sign(&manifest.signed_bytes()).to_bytes();
|
||||
StoredSignature {
|
||||
manifest,
|
||||
record_hashes,
|
||||
public_key: key.verifying_key().to_bytes(),
|
||||
signature,
|
||||
}
|
||||
}
|
||||
|
||||
/// Compute the manifest of a store's state.
|
||||
pub fn manifest(
|
||||
config: &MemoryConfig,
|
||||
cache: &MemoryCache,
|
||||
sessions: &SessionCache,
|
||||
knowledge: &KnowledgeCache,
|
||||
wal_applied: Option<WalMark>,
|
||||
) -> (Manifest, Vec<Hash>) {
|
||||
let record_hashes: Vec<Hash> = (0..cache.len()).map(|i| record_hash(cache, i)).collect();
|
||||
let manifest = Manifest {
|
||||
record_count: cache.len() as u64,
|
||||
records_root: merkle_root(&record_hashes),
|
||||
settings: settings_hash(config, wal_applied),
|
||||
sessions: sessions_hash(sessions),
|
||||
graph: graph_hash(knowledge),
|
||||
};
|
||||
(manifest, record_hashes)
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Canonical encoding
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// A SHA-256 over length-prefixed fields, so no two different field lists
|
||||
/// hash the same bytes.
|
||||
struct Fields(Sha256);
|
||||
|
||||
impl Fields {
|
||||
fn new(domain: &str) -> Self {
|
||||
let mut h = Sha256::new();
|
||||
h.update((domain.len() as u64).to_le_bytes());
|
||||
h.update(domain.as_bytes());
|
||||
Self(h)
|
||||
}
|
||||
fn bytes(&mut self, b: &[u8]) -> &mut Self {
|
||||
self.0.update((b.len() as u64).to_le_bytes());
|
||||
self.0.update(b);
|
||||
self
|
||||
}
|
||||
/// Strings as the loader returns them: stored null-padded, so a trailing
|
||||
/// NUL cannot survive a round trip and must not be part of the hash.
|
||||
fn str(&mut self, s: &str) -> &mut Self {
|
||||
self.bytes(s.trim_end_matches('\0').as_bytes())
|
||||
}
|
||||
fn u64(&mut self, v: u64) -> &mut Self {
|
||||
self.0.update(v.to_le_bytes());
|
||||
self
|
||||
}
|
||||
fn f64(&mut self, v: f64) -> &mut Self {
|
||||
self.0.update(v.to_bits().to_le_bytes());
|
||||
self
|
||||
}
|
||||
fn f32(&mut self, v: f32) -> &mut Self {
|
||||
self.0.update(v.to_bits().to_le_bytes());
|
||||
self
|
||||
}
|
||||
fn finish(self) -> Hash {
|
||||
self.0.finalize().into()
|
||||
}
|
||||
}
|
||||
|
||||
/// Everything persisted about record `i`, including its position. The
|
||||
/// embedding is hashed as the cache holds it — for a `float16` store that is
|
||||
/// the half-rounded value the file holds.
|
||||
fn record_hash(cache: &MemoryCache, i: usize) -> Hash {
|
||||
let mut f = Fields::new("clawhdf5-agent/record");
|
||||
f.u64(i as u64).str(&cache.chunks[i]);
|
||||
let emb: Vec<u8> = cache.embeddings[i]
|
||||
.iter()
|
||||
.flat_map(|v| v.to_bits().to_le_bytes())
|
||||
.collect();
|
||||
f.bytes(&emb)
|
||||
.str(&cache.source_channels[i])
|
||||
.f64(cache.timestamps[i])
|
||||
.str(&cache.session_ids[i])
|
||||
.str(&cache.tags[i])
|
||||
.u64(u64::from(cache.tombstones[i]))
|
||||
.f32(cache.activation_weights[i]);
|
||||
f.finish()
|
||||
}
|
||||
|
||||
/// Binary Merkle tree: leaves are the record hashes; a parent hashes its two
|
||||
/// children with a node prefix; an odd node is carried up unchanged.
|
||||
fn merkle_root(leaves: &[Hash]) -> Hash {
|
||||
if leaves.is_empty() {
|
||||
return Fields::new("clawhdf5-agent/merkle-empty").finish();
|
||||
}
|
||||
let mut level: Vec<Hash> = leaves.to_vec();
|
||||
while level.len() > 1 {
|
||||
level = level
|
||||
.chunks(2)
|
||||
.map(|pair| match pair {
|
||||
[l, r] => {
|
||||
let mut h = Sha256::new();
|
||||
h.update([1u8]);
|
||||
h.update(l);
|
||||
h.update(r);
|
||||
h.finalize().into()
|
||||
}
|
||||
[only] => *only,
|
||||
_ => unreachable!(),
|
||||
})
|
||||
.collect();
|
||||
}
|
||||
level[0]
|
||||
}
|
||||
|
||||
fn settings_hash(c: &MemoryConfig, wal_applied: Option<WalMark>) -> Hash {
|
||||
let mut f = Fields::new("clawhdf5-agent/settings");
|
||||
f.str(crate::schema::SCHEMA_VERSION)
|
||||
.str(&c.created_at)
|
||||
.str(&c.agent_id)
|
||||
.str(&c.embedder)
|
||||
.u64(c.embedding_dim as u64)
|
||||
.u64(c.chunk_size as u64)
|
||||
.u64(c.overlap as u64)
|
||||
.u64(u64::from(c.float16))
|
||||
.u64(u64::from(c.compression))
|
||||
.u64(u64::from(c.compression_level))
|
||||
.f32(c.compact_threshold)
|
||||
.f32(c.hebbian_boost)
|
||||
.f32(c.decay_factor)
|
||||
.u64(u64::from(c.wal_enabled))
|
||||
.u64(c.wal_max_entries as u64)
|
||||
.u64(u64::from(c.quantized_index))
|
||||
.u64(c.hnsw_m as u64)
|
||||
.u64(c.hnsw_ef_construction as u64)
|
||||
.u64(c.hnsw_ef_search as u64);
|
||||
// An empty mark is not written to the file, so it must hash as none.
|
||||
match wal_applied.filter(|m| m.len > 0) {
|
||||
Some(m) => f.u64(1).u64(m.len).u64(u64::from(m.crc)),
|
||||
None => f.u64(0),
|
||||
};
|
||||
f.finish()
|
||||
}
|
||||
|
||||
fn sessions_hash(s: &SessionCache) -> Hash {
|
||||
let mut f = Fields::new("clawhdf5-agent/sessions");
|
||||
f.u64(s.entries.len() as u64);
|
||||
for (i, e) in s.entries.iter().enumerate() {
|
||||
f.str(&e.id)
|
||||
.u64(e.start_idx)
|
||||
.u64(e.end_idx)
|
||||
.str(&e.channel)
|
||||
.f64(e.ts)
|
||||
.str(s.summaries.get(i).map(String::as_str).unwrap_or(""));
|
||||
}
|
||||
f.finish()
|
||||
}
|
||||
|
||||
fn graph_hash(k: &KnowledgeCache) -> Hash {
|
||||
let mut f = Fields::new("clawhdf5-agent/graph");
|
||||
f.u64(k.entities.len() as u64);
|
||||
for e in &k.entities {
|
||||
f.u64(e.id)
|
||||
.str(&e.name)
|
||||
.str(&e.entity_type)
|
||||
.u64(e.embedding_idx as u64);
|
||||
}
|
||||
f.u64(k.relations.len() as u64);
|
||||
for r in &k.relations {
|
||||
f.u64(r.src)
|
||||
.u64(r.tgt)
|
||||
.str(&r.relation)
|
||||
.f32(r.weight)
|
||||
.f64(r.ts);
|
||||
}
|
||||
f.u64(k.alias_strings.len() as u64);
|
||||
for (s, id) in k.alias_strings.iter().zip(&k.alias_entity_ids) {
|
||||
f.str(s).u64(*id as u64);
|
||||
}
|
||||
f.finish()
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Verification
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// The outcome of [`verify_store`].
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct VerifyReport {
|
||||
/// The checkpoint carries a signature.
|
||||
pub signed: bool,
|
||||
/// The signature was made by the key the caller trusts.
|
||||
pub key_matches: bool,
|
||||
/// The signature over the stored manifest is valid.
|
||||
pub signature_valid: bool,
|
||||
/// The file's current contents match the signed manifest.
|
||||
pub records_match: bool,
|
||||
pub settings_match: bool,
|
||||
pub sessions_match: bool,
|
||||
pub graph_match: bool,
|
||||
/// Records whose contents differ from what was signed (by position),
|
||||
/// when the stored per-record hashes are themselves authentic.
|
||||
pub changed_records: Vec<usize>,
|
||||
/// Records in the file versus in the signed manifest.
|
||||
pub record_count: u64,
|
||||
pub signed_record_count: u64,
|
||||
/// The public key the checkpoint claims to be signed by.
|
||||
pub public_key: Option<[u8; 32]>,
|
||||
/// Saves in the WAL after the checkpoint: not covered by the signature.
|
||||
pub wal_entries_unsigned: usize,
|
||||
}
|
||||
|
||||
impl VerifyReport {
|
||||
/// Signed by the trusted key, signature valid, and every part of the
|
||||
/// file unchanged since it was signed.
|
||||
pub fn is_valid(&self) -> bool {
|
||||
self.signed
|
||||
&& self.key_matches
|
||||
&& self.signature_valid
|
||||
&& self.records_match
|
||||
&& self.settings_match
|
||||
&& self.sessions_match
|
||||
&& self.graph_match
|
||||
}
|
||||
}
|
||||
|
||||
/// Check a store file against the public key the caller trusts.
|
||||
///
|
||||
/// Reads the checkpoint (not the WAL), recomputes every hash from its
|
||||
/// contents and checks the signature. Never writes.
|
||||
pub fn verify_store(
|
||||
path: &std::path::Path,
|
||||
trusted: &VerifyingKey,
|
||||
) -> Result<VerifyReport, crate::MemoryError> {
|
||||
let file = clawhdf5::File::open(path)
|
||||
.map_err(|e| crate::MemoryError::Hdf5(format!("cannot open {}: {e}", path.display())))?;
|
||||
let (config, cache, sessions, knowledge) = crate::schema::validate_and_load(&file)?;
|
||||
let checkpoint = crate::schema::read_checkpoint_meta(&file);
|
||||
let stored = crate::schema::read_signature(&file)?;
|
||||
let wal_entries_unsigned = count_wal_entries_after(path, checkpoint.wal_applied);
|
||||
|
||||
let (current, current_hashes) = manifest(
|
||||
&config,
|
||||
&cache,
|
||||
&sessions,
|
||||
&knowledge,
|
||||
checkpoint.wal_applied,
|
||||
);
|
||||
|
||||
let Some(stored) = stored else {
|
||||
return Ok(VerifyReport {
|
||||
signed: false,
|
||||
key_matches: false,
|
||||
signature_valid: false,
|
||||
records_match: false,
|
||||
settings_match: false,
|
||||
sessions_match: false,
|
||||
graph_match: false,
|
||||
changed_records: Vec::new(),
|
||||
record_count: current.record_count,
|
||||
signed_record_count: 0,
|
||||
public_key: None,
|
||||
wal_entries_unsigned,
|
||||
});
|
||||
};
|
||||
|
||||
let key_matches = stored.public_key == trusted.to_bytes();
|
||||
let signature_valid = trusted
|
||||
.verify(
|
||||
&stored.manifest.signed_bytes(),
|
||||
&Signature::from_bytes(&stored.signature),
|
||||
)
|
||||
.is_ok();
|
||||
// The stored per-record hashes can localise a change only if they are
|
||||
// the ones that were signed.
|
||||
let hashes_authentic = signature_valid
|
||||
&& stored.record_hashes.len() as u64 == stored.manifest.record_count
|
||||
&& merkle_root(&stored.record_hashes) == stored.manifest.records_root;
|
||||
let changed_records = if hashes_authentic {
|
||||
let n = current_hashes.len().max(stored.record_hashes.len());
|
||||
(0..n)
|
||||
.filter(|&i| current_hashes.get(i) != stored.record_hashes.get(i))
|
||||
.collect()
|
||||
} else {
|
||||
Vec::new()
|
||||
};
|
||||
|
||||
Ok(VerifyReport {
|
||||
signed: true,
|
||||
key_matches,
|
||||
signature_valid,
|
||||
records_match: signature_valid
|
||||
&& current.record_count == stored.manifest.record_count
|
||||
&& current.records_root == stored.manifest.records_root,
|
||||
settings_match: signature_valid && current.settings == stored.manifest.settings,
|
||||
sessions_match: signature_valid && current.sessions == stored.manifest.sessions,
|
||||
graph_match: signature_valid && current.graph == stored.manifest.graph,
|
||||
changed_records,
|
||||
record_count: current.record_count,
|
||||
signed_record_count: stored.manifest.record_count,
|
||||
public_key: Some(stored.public_key),
|
||||
wal_entries_unsigned,
|
||||
})
|
||||
}
|
||||
|
||||
fn count_wal_entries_after(store: &std::path::Path, mark: Option<WalMark>) -> usize {
|
||||
let wal = store.with_extension("h5.wal");
|
||||
if !wal.exists() {
|
||||
return 0;
|
||||
}
|
||||
crate::wal::WalFile::read_entries_for_migration(&wal, mark)
|
||||
.map(|e| e.len())
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
/// A new random signing key from the operating system's RNG.
|
||||
pub fn generate_key() -> SigningKey {
|
||||
SigningKey::generate(&mut rand_core::OsRng)
|
||||
}
|
||||
|
||||
/// Hex encoding for keys and signatures in attributes and the CLI.
|
||||
pub fn to_hex(bytes: &[u8]) -> String {
|
||||
bytes.iter().map(|b| format!("{b:02x}")).collect()
|
||||
}
|
||||
|
||||
/// Parse hex into exactly `N` bytes.
|
||||
pub fn from_hex<const N: usize>(s: &str) -> Option<[u8; N]> {
|
||||
let s = s.trim();
|
||||
if s.len() != 2 * N {
|
||||
return None;
|
||||
}
|
||||
let mut out = [0u8; N];
|
||||
for (i, byte) in out.iter_mut().enumerate() {
|
||||
*byte = u8::from_str_radix(&s[2 * i..2 * i + 2], 16).ok()?;
|
||||
}
|
||||
Some(out)
|
||||
}
|
||||
@@ -36,7 +36,7 @@ pub fn write_to_disk_with_mark(
|
||||
) -> Result<(), MemoryError> {
|
||||
let meta = schema::CheckpointMeta {
|
||||
wal_applied,
|
||||
ann_generation: None,
|
||||
..schema::CheckpointMeta::default()
|
||||
};
|
||||
write_to_disk_with_meta(path, config, cache, sessions, knowledge, &meta)
|
||||
}
|
||||
@@ -50,7 +50,21 @@ pub fn write_to_disk_with_meta(
|
||||
knowledge: &KnowledgeCache,
|
||||
checkpoint: &schema::CheckpointMeta,
|
||||
) -> Result<(), MemoryError> {
|
||||
let bytes = schema::build_hdf5_file_with_meta(config, cache, sessions, knowledge, checkpoint)?;
|
||||
write_to_disk_signed(path, config, cache, sessions, knowledge, checkpoint, None)
|
||||
}
|
||||
|
||||
/// [`write_to_disk_with_meta`] with a signed manifest of the contents.
|
||||
pub fn write_to_disk_signed(
|
||||
path: &Path,
|
||||
config: &MemoryConfig,
|
||||
cache: &MemoryCache,
|
||||
sessions: &SessionCache,
|
||||
knowledge: &KnowledgeCache,
|
||||
checkpoint: &schema::CheckpointMeta,
|
||||
signature: Option<&crate::signing::StoredSignature>,
|
||||
) -> Result<(), MemoryError> {
|
||||
let bytes =
|
||||
schema::build_hdf5_file_signed(config, cache, sessions, knowledge, checkpoint, signature)?;
|
||||
|
||||
if bytes.is_empty() {
|
||||
return Err(MemoryError::Hdf5("build_hdf5_file produced 0 bytes".into()));
|
||||
|
||||
@@ -92,3 +92,64 @@ print(len(names))
|
||||
assert!(n >= 10, "only {n} datasets");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_edit_made_with_h5py_breaks_the_signature_and_names_the_record() {
|
||||
if !h5py_available() {
|
||||
assert!(
|
||||
std::env::var("CLAWHDF5_REQUIRE_INTEROP").as_deref() != Ok("1"),
|
||||
"CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
|
||||
);
|
||||
eprintln!("SKIP: python3 with h5py not available");
|
||||
return;
|
||||
}
|
||||
use clawhdf5_agent::signing::SigningKey;
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("signed.h5");
|
||||
let key = SigningKey::from_bytes(&[42; 32]);
|
||||
let mut m = HDF5Memory::create(MemoryConfig::new(path.clone(), "agent", 8)).unwrap();
|
||||
m.set_signing_key(key.clone());
|
||||
m.save_batch(
|
||||
(0..10)
|
||||
.map(|i| MemoryEntry {
|
||||
chunk: format!("memory {i}"),
|
||||
embedding: (0..8).map(|j| ((i * 8 + j) as f32).cos()).collect(),
|
||||
source_channel: "test".into(),
|
||||
timestamp: i as f64,
|
||||
session_id: "s".into(),
|
||||
tags: String::new(),
|
||||
})
|
||||
.collect(),
|
||||
)
|
||||
.unwrap();
|
||||
drop(m);
|
||||
assert!(
|
||||
HDF5Memory::verify(&path, &key.verifying_key())
|
||||
.unwrap()
|
||||
.is_valid()
|
||||
);
|
||||
|
||||
// Someone edits one timestamp in place with h5py.
|
||||
let script = format!(
|
||||
r#"
|
||||
import h5py
|
||||
with h5py.File("{}", "r+") as f:
|
||||
ts = f["memory/timestamps"]
|
||||
ts[3] = 12345.0
|
||||
"#,
|
||||
path.display()
|
||||
);
|
||||
let out = Command::new(python())
|
||||
.args(["-c", &script])
|
||||
.output()
|
||||
.unwrap();
|
||||
assert!(
|
||||
out.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&out.stderr)
|
||||
);
|
||||
|
||||
let r = HDF5Memory::verify(&path, &key.verifying_key()).unwrap();
|
||||
assert!(r.signature_valid && !r.is_valid(), "{r:?}");
|
||||
assert_eq!(r.changed_records, vec![3]);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,330 @@
|
||||
//! Ed25519-signed checkpoints: `HDF5Memory::set_signing_key` and
|
||||
//! `HDF5Memory::verify`.
|
||||
|
||||
use std::path::Path;
|
||||
|
||||
use clawhdf5_agent::signing::{SigningKey, VerifyReport, VerifyingKey};
|
||||
use clawhdf5_agent::storage;
|
||||
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, MemoryError, schema};
|
||||
use tempfile::TempDir;
|
||||
|
||||
const DIM: usize = 16;
|
||||
|
||||
fn key(seed: u8) -> SigningKey {
|
||||
SigningKey::from_bytes(&[seed; 32])
|
||||
}
|
||||
|
||||
fn entry(i: usize, chunk: &str) -> MemoryEntry {
|
||||
MemoryEntry {
|
||||
chunk: chunk.to_string(),
|
||||
embedding: (0..DIM)
|
||||
.map(|j| ((i * DIM + j) as f32 * 0.37).sin())
|
||||
.collect(),
|
||||
source_channel: "chat".into(),
|
||||
timestamp: 1_700_000_000.0 + i as f64,
|
||||
session_id: format!("s{}", i % 3),
|
||||
tags: format!("t{i}"),
|
||||
}
|
||||
}
|
||||
|
||||
/// Awkward strings on purpose: they must hash the same after a round trip.
|
||||
const TEXTS: [&str; 6] = [
|
||||
"plain text",
|
||||
"ünïcödé — 日本語 🙂",
|
||||
"",
|
||||
"trailing spaces ",
|
||||
"tab\tand\nnewline",
|
||||
"x",
|
||||
];
|
||||
|
||||
fn signed_store(dir: &TempDir, float16: bool, k: &SigningKey) -> std::path::PathBuf {
|
||||
let mut cfg = MemoryConfig::new(dir.path().join("s.h5"), "agent", DIM);
|
||||
cfg.float16 = float16;
|
||||
let path = cfg.path.clone();
|
||||
let mut m = HDF5Memory::create(cfg).unwrap();
|
||||
m.set_signing_key(k.clone());
|
||||
let entries = (0..30).map(|i| entry(i, TEXTS[i % TEXTS.len()])).collect();
|
||||
m.save_batch(entries).unwrap();
|
||||
// Some graph and a deleted record, so every part of the manifest is used.
|
||||
let a = m.knowledge_mut().add_entity("Alice", "person", 0);
|
||||
let b = m.knowledge_mut().add_entity("Acme", "org", -1);
|
||||
m.knowledge_mut().add_relation(a, b, "works_at", 0.75);
|
||||
m.sessions_mut()
|
||||
.add_at("s0", 0, 9, "chat", "first session", 1_700_000_000.0);
|
||||
m.delete(4).unwrap();
|
||||
m.flush_wal().unwrap();
|
||||
path
|
||||
}
|
||||
|
||||
fn verify(path: &Path, k: &SigningKey) -> VerifyReport {
|
||||
HDF5Memory::verify(path, &k.verifying_key()).unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_signed_store_verifies_through_reopen_and_checkpoint_cycles() {
|
||||
for float16 in [true, false] {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let k = key(7);
|
||||
let path = signed_store(&dir, float16, &k);
|
||||
let r = verify(&path, &k);
|
||||
assert!(r.is_valid(), "float16={float16}: {r:?}");
|
||||
assert_eq!(r.public_key, Some(k.verifying_key().to_bytes()));
|
||||
assert_eq!(r.record_count, 30);
|
||||
assert!(r.changed_records.is_empty());
|
||||
|
||||
// Reopen, change nothing, checkpoint again (with the key): still valid.
|
||||
for _ in 0..3 {
|
||||
let mut m = HDF5Memory::open(&path).unwrap();
|
||||
assert!(m.is_signed());
|
||||
m.set_signing_key(k.clone());
|
||||
m.flush_wal().unwrap();
|
||||
drop(m);
|
||||
assert!(verify(&path, &k).is_valid());
|
||||
}
|
||||
// And after real changes, re-signed.
|
||||
let mut m = HDF5Memory::open(&path).unwrap();
|
||||
m.set_signing_key(k.clone());
|
||||
m.save(entry(99, "added later")).unwrap();
|
||||
m.hybrid_search(&entry(1, "").embedding, "text", 0.4, 0.6, 5);
|
||||
m.flush_wal().unwrap();
|
||||
drop(m);
|
||||
let r = verify(&path, &k);
|
||||
assert!(r.is_valid(), "{r:?}");
|
||||
assert_eq!(r.record_count, 31);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_signed_store_refuses_to_checkpoint_without_its_key() {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let k = key(1);
|
||||
let path = signed_store(&dir, true, &k);
|
||||
|
||||
let mut m = HDF5Memory::open(&path).unwrap();
|
||||
m.save(entry(50, "pending")).unwrap();
|
||||
match m.flush_wal() {
|
||||
Err(MemoryError::SigningKeyRequired(msg)) => assert!(msg.contains("signed"), "{msg}"),
|
||||
other => panic!("expected SigningKeyRequired, got {other:?}"),
|
||||
}
|
||||
// The file is untouched and still valid; the save is still in the WAL.
|
||||
let r = verify(&path, &k);
|
||||
assert!(r.is_valid());
|
||||
assert_eq!(r.wal_entries_unsigned, 1);
|
||||
|
||||
// Supplying the key lets the checkpoint through, signed.
|
||||
m.set_signing_key(k.clone());
|
||||
m.flush_wal().unwrap();
|
||||
drop(m);
|
||||
let r = verify(&path, &k);
|
||||
assert!(r.is_valid());
|
||||
assert_eq!((r.record_count, r.wal_entries_unsigned), (31, 0));
|
||||
|
||||
// Removing the signature on purpose writes it unsigned.
|
||||
let mut m = HDF5Memory::open(&path).unwrap();
|
||||
m.remove_signature();
|
||||
m.flush_wal().unwrap();
|
||||
drop(m);
|
||||
let r = verify(&path, &k);
|
||||
assert!(!r.signed && !r.is_valid());
|
||||
assert!(!HDF5Memory::open(&path).unwrap().is_signed());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_wrong_key_does_not_verify_and_a_new_key_re_signs() {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let (a, b) = (key(1), key(2));
|
||||
let path = signed_store(&dir, true, &a);
|
||||
let r = verify(&path, &b);
|
||||
assert!(r.signed && !r.key_matches && !r.signature_valid && !r.is_valid());
|
||||
|
||||
let mut m = HDF5Memory::open(&path).unwrap();
|
||||
m.set_signing_key(b.clone());
|
||||
m.flush_wal().unwrap();
|
||||
drop(m);
|
||||
assert!(verify(&path, &b).is_valid());
|
||||
assert!(!verify(&path, &a).is_valid());
|
||||
}
|
||||
|
||||
/// Rewrite the store with changed contents but the *old* signature — what
|
||||
/// someone with write access to the file, but not the key, can do.
|
||||
fn tamper(path: &Path, change: impl FnOnce(&mut Tampered)) {
|
||||
let file = clawhdf5::File::open(path).unwrap();
|
||||
let (config, cache, sessions, knowledge) = schema::validate_and_load(&file).unwrap();
|
||||
let checkpoint = schema::read_checkpoint_meta(&file);
|
||||
let signature = schema::read_signature(&file).unwrap().unwrap();
|
||||
drop(file);
|
||||
let mut t = Tampered {
|
||||
config,
|
||||
cache,
|
||||
sessions,
|
||||
knowledge,
|
||||
};
|
||||
change(&mut t);
|
||||
storage::write_to_disk_signed(
|
||||
path,
|
||||
&t.config,
|
||||
&t.cache,
|
||||
&t.sessions,
|
||||
&t.knowledge,
|
||||
&checkpoint,
|
||||
Some(&signature),
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
struct Tampered {
|
||||
config: MemoryConfig,
|
||||
cache: clawhdf5_agent::cache::MemoryCache,
|
||||
sessions: clawhdf5_agent::SessionCache,
|
||||
knowledge: clawhdf5_agent::knowledge::KnowledgeCache,
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_kind_of_edit_is_detected_and_located() {
|
||||
let k = key(3);
|
||||
type Edit = Box<dyn FnOnce(&mut Tampered)>;
|
||||
type Case = (&'static str, Edit, fn(&VerifyReport) -> bool);
|
||||
let cases: Vec<Case> = vec![
|
||||
(
|
||||
"record text",
|
||||
Box::new(|t: &mut Tampered| t.cache.chunks[7] = "rewritten".into()),
|
||||
|r| !r.records_match && r.changed_records == vec![7],
|
||||
),
|
||||
(
|
||||
"one embedding value",
|
||||
Box::new(|t: &mut Tampered| {
|
||||
let mut e = t.cache.embeddings[12].to_vec();
|
||||
e[3] = 0.5;
|
||||
t.cache.embeddings.set(12, &e);
|
||||
}),
|
||||
|r| r.changed_records == vec![12],
|
||||
),
|
||||
(
|
||||
"undelete",
|
||||
Box::new(|t: &mut Tampered| t.cache.tombstones[4] = 0),
|
||||
|r| r.changed_records == vec![4],
|
||||
),
|
||||
(
|
||||
"timestamp",
|
||||
Box::new(|t: &mut Tampered| t.cache.timestamps[20] += 1.0),
|
||||
|r| r.changed_records == vec![20],
|
||||
),
|
||||
(
|
||||
"record appended",
|
||||
Box::new(|t: &mut Tampered| {
|
||||
t.cache.push(
|
||||
"new".into(),
|
||||
vec![0.1; DIM],
|
||||
"x".into(),
|
||||
1.0,
|
||||
"s".into(),
|
||||
"".into(),
|
||||
);
|
||||
}),
|
||||
|r| !r.records_match && r.changed_records == vec![30] && r.record_count == 31,
|
||||
),
|
||||
(
|
||||
"setting",
|
||||
Box::new(|t: &mut Tampered| t.config.agent_id = "someone-else".into()),
|
||||
|r| !r.settings_match && r.records_match,
|
||||
),
|
||||
(
|
||||
"session summary",
|
||||
Box::new(|t: &mut Tampered| t.sessions.summaries[0] = "edited".into()),
|
||||
|r| !r.sessions_match && r.records_match,
|
||||
),
|
||||
(
|
||||
"graph edge",
|
||||
Box::new(|t: &mut Tampered| t.knowledge.relations[0].weight = 1.0),
|
||||
|r| !r.graph_match && r.records_match,
|
||||
),
|
||||
];
|
||||
for (name, edit, check) in cases {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let path = signed_store(&dir, true, &k);
|
||||
tamper(&path, edit);
|
||||
let r = verify(&path, &k);
|
||||
assert!(
|
||||
r.signed && r.key_matches && r.signature_valid,
|
||||
"{name}: {r:?}"
|
||||
);
|
||||
assert!(!r.is_valid(), "{name}: edit not detected: {r:?}");
|
||||
assert!(check(&r), "{name}: {r:?}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_forged_manifest_fails_the_signature() {
|
||||
// Recomputing the hashes for tampered contents does not help without the
|
||||
// key: the signature no longer matches the manifest.
|
||||
let dir = TempDir::new().unwrap();
|
||||
let k = key(5);
|
||||
let path = signed_store(&dir, true, &k);
|
||||
let file = clawhdf5::File::open(&path).unwrap();
|
||||
let (config, mut cache, sessions, knowledge) = schema::validate_and_load(&file).unwrap();
|
||||
let checkpoint = schema::read_checkpoint_meta(&file);
|
||||
let mut sig = schema::read_signature(&file).unwrap().unwrap();
|
||||
drop(file);
|
||||
cache.chunks[0] = "forged".into();
|
||||
// Re-sign with an attacker key, then splice the victim's public key back.
|
||||
let forged = clawhdf5_agent::signing::sign(
|
||||
&key(66),
|
||||
&config,
|
||||
&cache,
|
||||
&sessions,
|
||||
&knowledge,
|
||||
checkpoint.wal_applied,
|
||||
);
|
||||
sig.manifest = forged.manifest;
|
||||
sig.record_hashes = forged.record_hashes;
|
||||
storage::write_to_disk_signed(
|
||||
&path,
|
||||
&config,
|
||||
&cache,
|
||||
&sessions,
|
||||
&knowledge,
|
||||
&checkpoint,
|
||||
Some(&sig),
|
||||
)
|
||||
.unwrap();
|
||||
let r = verify(&path, &k);
|
||||
assert!(
|
||||
r.key_matches && !r.signature_valid && !r.is_valid(),
|
||||
"{r:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_unsigned_store_reports_unsigned() {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let mut m = HDF5Memory::create(MemoryConfig::new(dir.path().join("u.h5"), "a", DIM)).unwrap();
|
||||
m.save_batch(vec![entry(0, "hello")]).unwrap();
|
||||
drop(m);
|
||||
let r = HDF5Memory::verify(&dir.path().join("u.h5"), &VerifyingKey::from(&key(1))).unwrap();
|
||||
assert!(!r.signed && !r.is_valid());
|
||||
assert_eq!(r.record_count, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn nul_bytes_in_text_still_verify() {
|
||||
// Strings are stored null-padded; the hash must follow what a reopened
|
||||
// store actually holds, or an untouched store would fail to verify.
|
||||
let dir = TempDir::new().unwrap();
|
||||
let k = key(9);
|
||||
let mut m = HDF5Memory::create(MemoryConfig::new(dir.path().join("n.h5"), "a", DIM)).unwrap();
|
||||
m.set_signing_key(k.clone());
|
||||
m.save_batch(vec![
|
||||
entry(0, "inner\0nul"),
|
||||
entry(1, "trailing nul\0"),
|
||||
entry(2, "\0leading"),
|
||||
])
|
||||
.unwrap();
|
||||
drop(m);
|
||||
let r = verify(&dir.path().join("n.h5"), &k);
|
||||
assert!(r.is_valid(), "{r:?}");
|
||||
let m = HDF5Memory::open(&dir.path().join("n.h5")).unwrap();
|
||||
eprintln!(
|
||||
"reloaded: {:?}",
|
||||
(0..3).map(|i| m.get_chunk(i)).collect::<Vec<_>>()
|
||||
);
|
||||
}
|
||||
@@ -21,6 +21,7 @@
|
||||
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --ann-only --uniform
|
||||
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full
|
||||
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --options-study --full
|
||||
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --signing-study --full
|
||||
//! ```
|
||||
|
||||
use std::time::{Duration, Instant};
|
||||
@@ -488,6 +489,81 @@ fn bench_end_to_end(n: usize, json: &mut Vec<serde_json::Value>) {
|
||||
}));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Signing study: what does an Ed25519-signed checkpoint cost?
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// `--signing-study`: checkpoint time unsigned vs signed, `verify` time, and
|
||||
/// the file-size cost of the stored per-record hashes. Default store
|
||||
/// settings (float16, int8 index). Medians of five checkpoints / three
|
||||
/// verifies.
|
||||
fn signing_study(n: usize) {
|
||||
use clawhdf5_agent::signing::SigningKey;
|
||||
let data = make_dataset(n, 0x516 ^ n as u64);
|
||||
let mut rng = Rng(9);
|
||||
let entries: Vec<MemoryEntry> = data
|
||||
.vectors
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, v)| MemoryEntry {
|
||||
chunk: text_for(data.cluster_of[i], i, &mut rng),
|
||||
embedding: v.clone(),
|
||||
source_channel: "bench".into(),
|
||||
timestamp: i as f64,
|
||||
session_id: format!("s{}", i % 50),
|
||||
tags: format!("t{i}"),
|
||||
})
|
||||
.collect();
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("sign.h5");
|
||||
let mut mem = HDF5Memory::create(MemoryConfig::new(path.clone(), "bench", DIM)).unwrap();
|
||||
mem.save_batch(entries).unwrap();
|
||||
std::hint::black_box(mem.hybrid_search(&data.queries[0], "", 1.0, 0.0, K));
|
||||
|
||||
let median = |mut v: Vec<Duration>| {
|
||||
v.sort();
|
||||
v[v.len() / 2]
|
||||
};
|
||||
let checkpoint = |mem: &mut HDF5Memory| {
|
||||
median(
|
||||
(0..5)
|
||||
.map(|_| {
|
||||
let t = Instant::now();
|
||||
mem.flush_wal().unwrap();
|
||||
t.elapsed()
|
||||
})
|
||||
.collect(),
|
||||
)
|
||||
};
|
||||
let unsigned = checkpoint(&mut mem);
|
||||
let unsigned_bytes = std::fs::metadata(&path).unwrap().len();
|
||||
let key = SigningKey::from_bytes(&[7; 32]);
|
||||
mem.set_signing_key(key.clone());
|
||||
let signed = checkpoint(&mut mem);
|
||||
let signed_bytes = std::fs::metadata(&path).unwrap().len();
|
||||
drop(mem);
|
||||
let vk = key.verifying_key();
|
||||
let verify = median(
|
||||
(0..3)
|
||||
.map(|_| {
|
||||
let t = Instant::now();
|
||||
let r = HDF5Memory::verify(&path, &vk).unwrap();
|
||||
let d = t.elapsed();
|
||||
assert!(r.is_valid());
|
||||
d
|
||||
})
|
||||
.collect(),
|
||||
);
|
||||
println!(
|
||||
"| {n} | {:.1} | {:.1} | {:+.1} | {:.1} | {:+.2} |",
|
||||
millis(unsigned),
|
||||
millis(signed),
|
||||
millis(signed) - millis(unsigned),
|
||||
millis(verify),
|
||||
(signed_bytes as f64 - unsigned_bytes as f64) / (1024.0 * 1024.0),
|
||||
);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Search options study: source filters, re-ranking, confidence rejection
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -960,6 +1036,21 @@ fn main() {
|
||||
}
|
||||
return;
|
||||
}
|
||||
if args.iter().any(|a| a == "--signing-study") {
|
||||
println!("## Signed checkpoints ({DIM}-dim, float16, int8 index)\n");
|
||||
println!(
|
||||
"| N | checkpoint ms, unsigned | checkpoint ms, signed | signing adds ms | verify ms | file MiB added |"
|
||||
);
|
||||
println!("|---:|---:|---:|---:|---:|---:|");
|
||||
for &n in if full {
|
||||
&[1_000, 10_000, 100_000][..]
|
||||
} else {
|
||||
&[1_000, 10_000][..]
|
||||
} {
|
||||
signing_study(n);
|
||||
}
|
||||
return;
|
||||
}
|
||||
if args.iter().any(|a| a == "--options-study") {
|
||||
println!("## Search options ({DIM}-dim, k = {K}, Hebbian boost off)\n");
|
||||
println!("| N | options | filtered recall@10 | p50 ms | p99 ms |");
|
||||
|
||||
+126
-16
@@ -1,15 +1,22 @@
|
||||
use std::path::PathBuf;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use clap::{Parser, Subcommand};
|
||||
use clawhdf5_agent::signing::{self, SigningKey, VerifyingKey};
|
||||
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
|
||||
|
||||
/// ClawhDF5 — HDF5-backed cognitive memory for AI agents
|
||||
#[derive(Parser)]
|
||||
#[command(name = "clawhdf5", version, about)]
|
||||
struct Cli {
|
||||
/// Path to the .h5 memory file
|
||||
/// Path to the .h5 memory file (not needed for `keygen`)
|
||||
#[arg(short, long, env = "CLAWHDF5_PATH")]
|
||||
path: PathBuf,
|
||||
path: Option<PathBuf>,
|
||||
|
||||
/// File holding an Ed25519 signing key (64 hex characters, from
|
||||
/// `keygen`). Every checkpoint this command makes is then signed; a
|
||||
/// signed store refuses to checkpoint without it.
|
||||
#[arg(long, env = "CLAWHDF5_SIGNING_KEY", global = true)]
|
||||
signing_key: Option<PathBuf>,
|
||||
|
||||
#[command(subcommand)]
|
||||
command: Commands,
|
||||
@@ -91,6 +98,38 @@ enum Commands {
|
||||
/// Destination path
|
||||
dest: PathBuf,
|
||||
},
|
||||
/// Generate an Ed25519 signing key for signed checkpoints
|
||||
Keygen {
|
||||
/// Where to write the secret key (created new, owner-only on Unix)
|
||||
#[arg(long)]
|
||||
out: PathBuf,
|
||||
},
|
||||
/// Verify a signed store against a public key; exit status 2 if not valid
|
||||
Verify {
|
||||
/// The trusted public key: 64 hex characters, or a file holding them
|
||||
#[arg(long)]
|
||||
public_key: String,
|
||||
},
|
||||
}
|
||||
|
||||
fn read_signing_key(path: &Path) -> Result<SigningKey, Box<dyn std::error::Error>> {
|
||||
let text = std::fs::read_to_string(path)
|
||||
.map_err(|e| format!("cannot read signing key {}: {e}", path.display()))?;
|
||||
let bytes = signing::from_hex::<32>(&text)
|
||||
.ok_or_else(|| format!("{} is not a 64-hex-character key", path.display()))?;
|
||||
Ok(SigningKey::from_bytes(&bytes))
|
||||
}
|
||||
|
||||
/// Open for writing, with the signing key applied if one was given.
|
||||
fn open_writable(
|
||||
path: &Path,
|
||||
key: &Option<SigningKey>,
|
||||
) -> Result<HDF5Memory, Box<dyn std::error::Error>> {
|
||||
let mut mem = HDF5Memory::open(path)?;
|
||||
if let Some(k) = key {
|
||||
mem.set_signing_key(k.clone());
|
||||
}
|
||||
Ok(mem)
|
||||
}
|
||||
|
||||
fn main() {
|
||||
@@ -103,6 +142,37 @@ fn main() {
|
||||
}
|
||||
|
||||
fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
if let Commands::Keygen { out } = &cli.command {
|
||||
let key = signing::generate_key();
|
||||
let mut opts = std::fs::OpenOptions::new();
|
||||
opts.write(true).create_new(true);
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::OpenOptionsExt;
|
||||
opts.mode(0o600);
|
||||
}
|
||||
use std::io::Write;
|
||||
let mut f = opts
|
||||
.open(out)
|
||||
.map_err(|e| format!("cannot create {}: {e}", out.display()))?;
|
||||
writeln!(f, "{}", signing::to_hex(&key.to_bytes()))?;
|
||||
let j = serde_json::json!({
|
||||
"status": "generated",
|
||||
"secret_key_file": out.display().to_string(),
|
||||
"public_key": signing::to_hex(&key.verifying_key().to_bytes()),
|
||||
});
|
||||
println!("{}", serde_json::to_string_pretty(&j)?);
|
||||
return Ok(());
|
||||
}
|
||||
let path = cli
|
||||
.path
|
||||
.clone()
|
||||
.ok_or("--path (or CLAWHDF5_PATH) is required")?;
|
||||
let key = cli
|
||||
.signing_key
|
||||
.as_deref()
|
||||
.map(read_signing_key)
|
||||
.transpose()?;
|
||||
match cli.command {
|
||||
Commands::Create {
|
||||
agent_id,
|
||||
@@ -113,7 +183,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
f32,
|
||||
float16: _,
|
||||
} => {
|
||||
let mut config = MemoryConfig::new(cli.path.clone(), &agent_id, dim);
|
||||
let mut config = MemoryConfig::new(path.clone(), &agent_id, dim);
|
||||
config.wal_enabled = wal;
|
||||
// As with --f32-index: only ever switch the library default off.
|
||||
if f32 {
|
||||
@@ -127,15 +197,21 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
config.quantized_index = false;
|
||||
}
|
||||
let config_quantized = config.quantized_index;
|
||||
let mem = HDF5Memory::create(config)?;
|
||||
let mut mem = HDF5Memory::create(config)?;
|
||||
// Sign straight away, so the store is never on disk unsigned.
|
||||
if let Some(k) = &key {
|
||||
mem.set_signing_key(k.clone());
|
||||
mem.flush_wal()?;
|
||||
}
|
||||
let j = serde_json::json!({
|
||||
"status": "created",
|
||||
"path": cli.path.display().to_string(),
|
||||
"path": path.display().to_string(),
|
||||
"agent_id": agent_id,
|
||||
"embedding_dim": dim,
|
||||
"wal_enabled": wal,
|
||||
"quantized_index": config_quantized,
|
||||
"float16": config_float16,
|
||||
"signed": mem.is_signed(),
|
||||
"count": mem.count(),
|
||||
});
|
||||
println!("{}", serde_json::to_string_pretty(&j)?);
|
||||
@@ -152,7 +228,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
}
|
||||
};
|
||||
let entry: MemoryEntry = serde_json::from_str(&input)?;
|
||||
let mut mem = HDF5Memory::open(&cli.path)?;
|
||||
let mut mem = open_writable(&path, &key)?;
|
||||
let idx = mem.save(entry)?;
|
||||
let j = serde_json::json!({ "status": "saved", "index": idx, "count": mem.count() });
|
||||
println!("{}", serde_json::to_string(&j)?);
|
||||
@@ -166,7 +242,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
keyword_weight,
|
||||
} => {
|
||||
let emb: Vec<f32> = serde_json::from_str(&embedding)?;
|
||||
let mut mem = HDF5Memory::open(&cli.path)?;
|
||||
let mut mem = open_writable(&path, &key)?;
|
||||
let results = mem.hybrid_search(&emb, &query, vector_weight, keyword_weight, top_k);
|
||||
let j: Vec<serde_json::Value> = results
|
||||
.iter()
|
||||
@@ -184,7 +260,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
}
|
||||
|
||||
Commands::Recall { index } => {
|
||||
let mem = HDF5Memory::open_read_only(&cli.path)?;
|
||||
let mem = HDF5Memory::open_read_only(&path)?;
|
||||
match mem.get_chunk(index) {
|
||||
Some(content) => {
|
||||
let j = serde_json::json!({ "index": index, "chunk": content });
|
||||
@@ -198,22 +274,23 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
}
|
||||
|
||||
Commands::Stats => {
|
||||
let mem = HDF5Memory::open_read_only(&cli.path)?;
|
||||
let mem = HDF5Memory::open_read_only(&path)?;
|
||||
let cfg = mem.config();
|
||||
let j = serde_json::json!({
|
||||
"path": cli.path.display().to_string(),
|
||||
"path": path.display().to_string(),
|
||||
"agent_id": cfg.agent_id,
|
||||
"embedding_dim": cfg.embedding_dim,
|
||||
"count": mem.count(),
|
||||
"active": mem.count_active(),
|
||||
"wal_enabled": cfg.wal_enabled,
|
||||
"wal_pending": mem.wal_pending_count(),
|
||||
"signed": mem.is_signed(),
|
||||
});
|
||||
println!("{}", serde_json::to_string_pretty(&j)?);
|
||||
}
|
||||
|
||||
Commands::FlushWal => {
|
||||
let mut mem = HDF5Memory::open(&cli.path)?;
|
||||
let mut mem = open_writable(&path, &key)?;
|
||||
let before = mem.wal_pending_count();
|
||||
mem.flush_wal()?;
|
||||
let j = serde_json::json!({
|
||||
@@ -225,7 +302,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
}
|
||||
|
||||
Commands::AgentsMd { output } => {
|
||||
let mem = HDF5Memory::open_read_only(&cli.path)?;
|
||||
let mem = HDF5Memory::open_read_only(&path)?;
|
||||
let md = mem.generate_agents_md();
|
||||
match output {
|
||||
Some(p) => {
|
||||
@@ -237,7 +314,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
}
|
||||
|
||||
Commands::Export => {
|
||||
let mem = HDF5Memory::open_read_only(&cli.path)?;
|
||||
let mem = HDF5Memory::open_read_only(&path)?;
|
||||
for i in 0..mem.count() {
|
||||
if let Some(chunk) = mem.get_chunk(i) {
|
||||
let j = serde_json::json!({ "index": i, "chunk": chunk });
|
||||
@@ -246,11 +323,44 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
}
|
||||
}
|
||||
|
||||
Commands::Keygen { .. } => unreachable!("handled before opening a store"),
|
||||
|
||||
Commands::Verify { public_key } => {
|
||||
let text = if Path::new(&public_key).is_file() {
|
||||
std::fs::read_to_string(&public_key)?
|
||||
} else {
|
||||
public_key
|
||||
};
|
||||
let bytes = signing::from_hex::<32>(&text)
|
||||
.ok_or("--public-key must be 64 hex characters or a file holding them")?;
|
||||
let trusted = VerifyingKey::from_bytes(&bytes)?;
|
||||
let r = HDF5Memory::verify(&path, &trusted)?;
|
||||
let j = serde_json::json!({
|
||||
"valid": r.is_valid(),
|
||||
"signed": r.signed,
|
||||
"key_matches": r.key_matches,
|
||||
"signature_valid": r.signature_valid,
|
||||
"records_match": r.records_match,
|
||||
"settings_match": r.settings_match,
|
||||
"sessions_match": r.sessions_match,
|
||||
"graph_match": r.graph_match,
|
||||
"changed_records": r.changed_records,
|
||||
"record_count": r.record_count,
|
||||
"signed_record_count": r.signed_record_count,
|
||||
"signed_by": r.public_key.map(|k| signing::to_hex(&k)),
|
||||
"wal_entries_unsigned": r.wal_entries_unsigned,
|
||||
});
|
||||
println!("{}", serde_json::to_string_pretty(&j)?);
|
||||
if !r.is_valid() {
|
||||
std::process::exit(2);
|
||||
}
|
||||
}
|
||||
|
||||
Commands::Snapshot { dest } => {
|
||||
let _result = clawhdf5_agent::storage::snapshot_file(&cli.path, &dest)?;
|
||||
let _result = clawhdf5_agent::storage::snapshot_file(&path, &dest)?;
|
||||
let j = serde_json::json!({
|
||||
"status": "snapshot_created",
|
||||
"source": cli.path.display().to_string(),
|
||||
"source": path.display().to_string(),
|
||||
"dest": dest.display().to_string(),
|
||||
});
|
||||
println!("{}", serde_json::to_string(&j)?);
|
||||
|
||||
Reference in New Issue
Block a user