feat(agent): Ed25519-signed checkpoints
Makes the README's "cryptographically verifiable memory" true. With HDF5Memory::set_signing_key(key), every checkpoint stores a signed manifest of the store: a SHA-256 per memory record (text, embedding as stored, channel, timestamp, session, tags, deleted flag, activation) in a Merkle tree, plus hashes of the settings (and WAL mark), sessions and knowledge graph. The signature, public key and manifest hashes go in /meta; the per-record hashes in /integrity/record_hashes, so HDF5Memory::verify(path, &public_key) can say which records changed, not just that something did. A forged manifest fails the signature. Decisions, as agreed: - the key is set on the open store and never persisted; - a signed store refuses to checkpoint without its key (MemoryError::SigningKeyRequired); remove_signature() is the deliberate way back to unsigned; - checkpoints only: saves still in the WAL are not covered, and verify reports how many there are. The hashes cover exactly what the file persists, in the form the loader returns it (strings lose trailing NULs; an empty WAL mark is not written), so untouched stores verify across any number of reopen and checkpoint cycles. MemoryError becomes #[non_exhaustive] (it already gains variants in this unreleased version). CLI: keygen (owner-only key file), --signing-key / CLAWHDF5_SIGNING_KEY on writing commands (create signs immediately), verify --public-key (JSON; exit 2 if not valid), `signed` in create/stats output. Tests: reopen/checkpoint cycles with awkward strings (f16 and f32), refusal without the key, wrong and rotated keys, eight kinds of edit each detected and located, a forged manifest, unsigned stores, NULs in text, and an edit made in place with h5py that verify pinpoints. Cost on tank (search_harness --signing-study --full, 3 runs): ~20% of a checkpoint (+9 ms at 10K, +89-112 ms at 100K), verify 18.6 ms / 247 ms, 32 bytes per record in the file. New deps ed25519-dalek, sha2, rand_core: pure Rust, the no-C check passes. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -243,6 +243,30 @@ index asked for ~16 000 candidates, where scanning the few hundred or thousand
|
|||||||
allowed records is exact and cheap. Re-ranking a 3k candidate pool and
|
allowed records is exact and cheap. Re-ranking a 3k candidate pool and
|
||||||
confidence rejection add about 3%.
|
confidence rejection add about 3%.
|
||||||
|
|
||||||
|
### Signed checkpoints
|
||||||
|
|
||||||
|
Measured 2026-09-25 on tank (AMD Ryzen 7 7800X3D). A default store (float16,
|
||||||
|
int8 index), 384-dim; each checkpoint rewrites the whole file, as every
|
||||||
|
checkpoint does. Medians of five checkpoints and three verifies; three runs
|
||||||
|
agreed to within the ranges shown.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo run --release -p clawhdf5-bench --bin search_harness -- --signing-study --full
|
||||||
|
```
|
||||||
|
|
||||||
|
| N | checkpoint, unsigned | checkpoint, signed | signing adds | `verify` | file size added |
|
||||||
|
|---:|---:|---:|---:|---:|---:|
|
||||||
|
| 1 000 | 5.4 ms | 6.4 ms | 0.7–1.0 ms | 2.1 ms | 0.03 MiB |
|
||||||
|
| 10 000 | 46 ms | 55 ms | 8.1–9.4 ms | 18.6 ms | 0.31 MiB |
|
||||||
|
| 100 000 | 495 ms | 598 ms | 89–112 ms | 247 ms | 3.05 MiB |
|
||||||
|
|
||||||
|
Signing costs about 20% of a checkpoint: every record is rehashed (SHA-256)
|
||||||
|
and the Merkle root recomputed each time; the Ed25519 signature itself is
|
||||||
|
microseconds. Caching per-record hashes between checkpoints would cut this to
|
||||||
|
the records that changed. The per-record hashes stored for locating edits are
|
||||||
|
32 bytes each (4% of a 100K float16 store). `verify` reads and rehashes the
|
||||||
|
whole checkpoint.
|
||||||
|
|
||||||
### float16 embedding storage (`MemoryConfig::float16`)
|
### float16 embedding storage (`MemoryConfig::float16`)
|
||||||
|
|
||||||
Measured 2026-09-23 on tank (AMD Ryzen 7 7800X3D). The same clustered
|
Measured 2026-09-23 on tank (AMD Ryzen 7 7800X3D). The same clustered
|
||||||
|
|||||||
@@ -3,6 +3,9 @@
|
|||||||
## Unreleased
|
## Unreleased
|
||||||
|
|
||||||
### Upgrade Notes
|
### Upgrade Notes
|
||||||
|
- **Breaking:** `MemoryError` is now `#[non_exhaustive]` and gained
|
||||||
|
`SigningKeyRequired`; a `match` on it needs a wildcard arm. Future variants
|
||||||
|
will no longer be breaking.
|
||||||
- **Breaking:** `clawhdf5-agent`'s `agent` feature is removed. It enabled
|
- **Breaking:** `clawhdf5-agent`'s `agent` feature is removed. It enabled
|
||||||
nothing — the agent layer is always built — but the README and guides told
|
nothing — the agent layer is always built — but the README and guides told
|
||||||
people to pass it; drop `agent` from `features = [...]`.
|
people to pass it; drop `agent` from `features = [...]`.
|
||||||
@@ -60,6 +63,29 @@
|
|||||||
`quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
|
`quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
|
||||||
out. The CLI's `--quantized-index` is still accepted but is now a no-op.
|
out. The CLI's `--quantized-index` is still accepted but is now a no-op.
|
||||||
|
|
||||||
|
### Signing
|
||||||
|
- `clawhdf5-agent`: **Ed25519-signed checkpoints** — the README's
|
||||||
|
"cryptographically verifiable memory", now true. With
|
||||||
|
`HDF5Memory::set_signing_key(key)`, every checkpoint stores a signed
|
||||||
|
manifest: a SHA-256 per record (text, embedding as stored, channel,
|
||||||
|
timestamp, session, tags, deleted flag, activation) in a Merkle tree, plus
|
||||||
|
hashes of the settings (and WAL mark), sessions and knowledge graph, with
|
||||||
|
the per-record hashes in `/integrity/record_hashes`.
|
||||||
|
`HDF5Memory::verify(path, &public_key)` recomputes everything from the file
|
||||||
|
and reports which part changed and which records (`changed_records`); a
|
||||||
|
forged manifest fails the signature. The key is never persisted; a signed
|
||||||
|
store refuses to checkpoint without it (`MemoryError::SigningKeyRequired`),
|
||||||
|
and `remove_signature()` is the deliberate way back to unsigned. Saves still
|
||||||
|
in the WAL are not covered (`wal_entries_unsigned`). Tests include every
|
||||||
|
kind of edit, and an edit made with h5py in place, which verify pinpoints.
|
||||||
|
Cost: ~20% of a checkpoint, 32 bytes per record (`BENCHMARKS.md`, "Signed
|
||||||
|
checkpoints"). New dependencies `ed25519-dalek`, `sha2`, `rand_core` — pure
|
||||||
|
Rust; the no-C check still passes.
|
||||||
|
- `clawhdf5-cli`: `keygen --out <file>` (owner-only key file),
|
||||||
|
`--signing-key <file>` / `CLAWHDF5_SIGNING_KEY` on writing commands
|
||||||
|
(`create` signs immediately), `verify --public-key <hex|file>` (JSON report;
|
||||||
|
exit status 2 if not valid), and `signed` in `create`/`stats` output.
|
||||||
|
|
||||||
### Migration
|
### Migration
|
||||||
- `clawhdf5-migrate`: writes through the agent's own API (`HDF5Memory::create`
|
- `clawhdf5-migrate`: writes through the agent's own API (`HDF5Memory::create`
|
||||||
/ `open`, `save_batch`, the session cache and knowledge graph), so there is
|
/ `open`, `save_batch`, the session cache and knowledge graph), so there is
|
||||||
|
|||||||
@@ -109,6 +109,17 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
|
|||||||
`search_harness --options-study`.
|
`search_harness --options-study`.
|
||||||
- `MemoryConfig::compression` is off by default; when on, embeddings are
|
- `MemoryConfig::compression` is off by default; when on, embeddings are
|
||||||
deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd).
|
deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd).
|
||||||
|
- Signed checkpoints (`clawhdf5-agent` `signing` module): with
|
||||||
|
`HDF5Memory::set_signing_key` every checkpoint stores an Ed25519-signed
|
||||||
|
manifest (SHA-256 per record in a Merkle tree + settings/sessions/graph
|
||||||
|
hashes; per-record hashes in `/integrity/record_hashes`);
|
||||||
|
`HDF5Memory::verify(path, &pk)` locates edits. The hashes must cover exactly
|
||||||
|
what the file persists in the form the loader returns it (strings lose
|
||||||
|
trailing NULs; an empty WAL mark is not written) or untouched stores stop
|
||||||
|
verifying — `tests/signed_store.rs` round-trips awkward strings. The key is
|
||||||
|
never persisted; a signed store refuses to checkpoint without it
|
||||||
|
(`MemoryError::SigningKeyRequired`, and `MemoryError` is `#[non_exhaustive]`).
|
||||||
|
WAL entries after the checkpoint are not covered.
|
||||||
- `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by
|
- `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by
|
||||||
default) recomputes a dataset's SHA-256 and compares it against the
|
default) recomputes a dataset's SHA-256 and compares it against the
|
||||||
`_provenance_sha256` attribute written automatically on save when
|
`_provenance_sha256` attribute written automatically on save when
|
||||||
|
|||||||
@@ -8,7 +8,7 @@
|
|||||||
[](BENCHMARKS.md#longmemeval-results)
|
[](BENCHMARKS.md#longmemeval-results)
|
||||||
[](BENCHMARKS.md#memory-footprint-1)
|
[](BENCHMARKS.md#memory-footprint-1)
|
||||||
|
|
||||||
ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file.
|
ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, cryptographically verifiable memory (Ed25519-signed checkpoints) — all stored in a single portable file.
|
||||||
|
|
||||||
> **Two things live here:**
|
> **Two things live here:**
|
||||||
> - **A general-purpose, pure-Rust HDF5 library** — zero C dependencies, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers.
|
> - **A general-purpose, pure-Rust HDF5 library** — zero C dependencies, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers.
|
||||||
@@ -113,7 +113,7 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d
|
|||||||
| Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers |
|
| Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers |
|
||||||
| Temporal queries | Custom code | Native temporal index (622 ns range query over 10K) |
|
| Temporal queries | Custom code | Native temporal index (622 ns range query over 10K) |
|
||||||
| Multi-modal | Multiple stores | Unified cross-modal search (exact scan: 842 µs over 1K records) |
|
| Multi-modal | Multiple stores | Unified cross-modal search (exact scan: 842 µs over 1K records) |
|
||||||
| Integrity | Hope for the best | Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance |
|
| Integrity | Hope for the best | Ed25519-signed checkpoints that pinpoint any edited record, chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts |
|
||||||
| Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** |
|
| Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** |
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -369,6 +369,7 @@ directly; the store persists the records, sessions and graph they work over.
|
|||||||
| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches. Opt-in via `SearchOptions::with_confidence`; on in the OpenClaw backend |
|
| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches. Opt-in via `SearchOptions::with_confidence`; on in the OpenClaw backend |
|
||||||
| **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints |
|
| **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints |
|
||||||
| **`multimodal`** | Cross-modal search across text/image/audio/video embeddings |
|
| **`multimodal`** | Cross-modal search across text/image/audio/video embeddings |
|
||||||
|
| **`signing`** | Ed25519-signed checkpoints: SHA-256 per record in a Merkle tree, plus hashes of settings, sessions and the knowledge graph; `HDF5Memory::verify` names any edited record |
|
||||||
| **`provenance`** | Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof) |
|
| **`provenance`** | Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof) |
|
||||||
| **`anomaly`** | Write rate limiting, 15 injection-pattern detectors, source-distribution analysis. Alerts never block a save; drain them with `take_anomaly_alerts` |
|
| **`anomaly`** | Write rate limiting, 15 injection-pattern detectors, source-distribution analysis. Alerts never block a save; drain them with `take_anomaly_alerts` |
|
||||||
| **`openclaw`** | OpenClaw integration: MemoryBackend trait, Markdown ↔ HDF5 conversion |
|
| **`openclaw`** | OpenClaw integration: MemoryBackend trait, Markdown ↔ HDF5 conversion |
|
||||||
@@ -457,6 +458,35 @@ let careful = memory.search(
|
|||||||
);
|
);
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Signed Checkpoints
|
||||||
|
|
||||||
|
```rust
|
||||||
|
use clawhdf5_agent::signing;
|
||||||
|
|
||||||
|
// Once, somewhere safe: keep the secret key, publish the public key.
|
||||||
|
let key = signing::generate_key();
|
||||||
|
let public = key.verifying_key();
|
||||||
|
|
||||||
|
// Every checkpoint is signed from now on. The key is never written to disk;
|
||||||
|
// a signed store refuses to checkpoint without it.
|
||||||
|
memory.set_signing_key(key);
|
||||||
|
memory.flush_wal()?;
|
||||||
|
|
||||||
|
// Anyone holding the public key can check the file, e.g. after copying it.
|
||||||
|
let report = HDF5Memory::verify(std::path::Path::new("agent.h5"), &public)?;
|
||||||
|
assert!(report.is_valid());
|
||||||
|
// On a tampered file: report.changed_records lists the records that differ.
|
||||||
|
```
|
||||||
|
|
||||||
|
The signature covers every record (text, embedding as stored, channel,
|
||||||
|
timestamp, session, tags, deleted flag, activation), the store's settings,
|
||||||
|
its sessions and its knowledge graph — a change made with any tool is caught.
|
||||||
|
It covers checkpoints, not saves still in the WAL
|
||||||
|
(`report.wal_entries_unsigned` counts those). CLI: `clawhdf5-cli keygen`,
|
||||||
|
`--signing-key <file>` on writing commands, and `verify --public-key`.
|
||||||
|
Signing adds about 20% to a checkpoint and 32 bytes per record to the file
|
||||||
|
([BENCHMARKS.md § Signed checkpoints](BENCHMARKS.md#signed-checkpoints)).
|
||||||
|
|
||||||
### Knowledge Graph
|
### Knowledge Graph
|
||||||
|
|
||||||
```rust
|
```rust
|
||||||
|
|||||||
@@ -19,6 +19,10 @@ clawhdf5-ann = { path = "../clawhdf5-ann", version = "2.7.0", optional = true }
|
|||||||
clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.7.0", optional = true, default-features = false }
|
clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.7.0", optional = true, default-features = false }
|
||||||
serde = { workspace = true }
|
serde = { workspace = true }
|
||||||
byteorder = "1"
|
byteorder = "1"
|
||||||
|
# Signed checkpoints (MemoryConfig-independent; see `signing`). Pure Rust.
|
||||||
|
ed25519-dalek = { version = "2", features = ["rand_core"] }
|
||||||
|
sha2 = "0.10"
|
||||||
|
rand_core = { version = "0.6", features = ["getrandom"] }
|
||||||
half = { workspace = true, optional = true }
|
half = { workspace = true, optional = true }
|
||||||
rayon = { version = "1", optional = true }
|
rayon = { version = "1", optional = true }
|
||||||
matrixmultiply = { version = "0.3", optional = true }
|
matrixmultiply = { version = "0.3", optional = true }
|
||||||
|
|||||||
@@ -36,6 +36,7 @@ pub mod reranker;
|
|||||||
pub mod schema;
|
pub mod schema;
|
||||||
pub mod search;
|
pub mod search;
|
||||||
pub mod session;
|
pub mod session;
|
||||||
|
pub mod signing;
|
||||||
pub mod storage;
|
pub mod storage;
|
||||||
mod store_lock;
|
mod store_lock;
|
||||||
pub mod temporal;
|
pub mod temporal;
|
||||||
@@ -78,6 +79,7 @@ pub use session::{SessionCache, SessionEntry};
|
|||||||
// --- Error type ---
|
// --- Error type ---
|
||||||
|
|
||||||
#[derive(Debug)]
|
#[derive(Debug)]
|
||||||
|
#[non_exhaustive]
|
||||||
pub enum MemoryError {
|
pub enum MemoryError {
|
||||||
Io(std::io::Error),
|
Io(std::io::Error),
|
||||||
Hdf5(String),
|
Hdf5(String),
|
||||||
@@ -88,6 +90,11 @@ pub enum MemoryError {
|
|||||||
/// A record the store cannot hold as given, e.g. an embedding value
|
/// A record the store cannot hold as given, e.g. an embedding value
|
||||||
/// outside the half-precision range of a `float16` store.
|
/// outside the half-precision range of a `float16` store.
|
||||||
InvalidEntry(String),
|
InvalidEntry(String),
|
||||||
|
/// The store's checkpoints are signed and no signing key is set, so a
|
||||||
|
/// checkpoint would leave it unsigned. Set the key with
|
||||||
|
/// [`HDF5Memory::set_signing_key`], or drop the signature on purpose with
|
||||||
|
/// [`HDF5Memory::remove_signature`].
|
||||||
|
SigningKeyRequired(String),
|
||||||
}
|
}
|
||||||
|
|
||||||
impl std::fmt::Display for MemoryError {
|
impl std::fmt::Display for MemoryError {
|
||||||
@@ -99,6 +106,7 @@ impl std::fmt::Display for MemoryError {
|
|||||||
MemoryError::NotFound(e) => write!(f, "not found: {e}"),
|
MemoryError::NotFound(e) => write!(f, "not found: {e}"),
|
||||||
MemoryError::Locked(e) => write!(f, "store is locked: {e}"),
|
MemoryError::Locked(e) => write!(f, "store is locked: {e}"),
|
||||||
MemoryError::InvalidEntry(e) => write!(f, "invalid entry: {e}"),
|
MemoryError::InvalidEntry(e) => write!(f, "invalid entry: {e}"),
|
||||||
|
MemoryError::SigningKeyRequired(e) => write!(f, "signing key required: {e}"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -317,6 +325,12 @@ pub struct HDF5Memory {
|
|||||||
activations_dirty: bool,
|
activations_dirty: bool,
|
||||||
/// Opened with [`HDF5Memory::open_read_only`]: nothing may reach the disk.
|
/// Opened with [`HDF5Memory::open_read_only`]: nothing may reach the disk.
|
||||||
read_only: bool,
|
read_only: bool,
|
||||||
|
/// Key that signs every checkpoint; never persisted. See
|
||||||
|
/// [`HDF5Memory::set_signing_key`].
|
||||||
|
signing_key: Option<signing::SigningKey>,
|
||||||
|
/// Checkpoints of this store are signed: the file on disk is, or a key
|
||||||
|
/// has been set. A checkpoint without a key is then refused.
|
||||||
|
signed: bool,
|
||||||
/// A WAL that `open()` could not read and moved aside; see
|
/// A WAL that `open()` could not read and moved aside; see
|
||||||
/// [`HDF5Memory::quarantined_wal`].
|
/// [`HDF5Memory::quarantined_wal`].
|
||||||
quarantined_wal: Option<PathBuf>,
|
quarantined_wal: Option<PathBuf>,
|
||||||
@@ -372,6 +386,8 @@ impl HDF5Memory {
|
|||||||
bm25_filter: bm25::TokenFilter::default(),
|
bm25_filter: bm25::TokenFilter::default(),
|
||||||
activations_dirty: false,
|
activations_dirty: false,
|
||||||
read_only: false,
|
read_only: false,
|
||||||
|
signing_key: None,
|
||||||
|
signed: false,
|
||||||
quarantined_wal: None,
|
quarantined_wal: None,
|
||||||
_lock: Some(lock),
|
_lock: Some(lock),
|
||||||
})
|
})
|
||||||
@@ -550,6 +566,8 @@ impl HDF5Memory {
|
|||||||
bm25_filter: bm25::TokenFilter::default(),
|
bm25_filter: bm25::TokenFilter::default(),
|
||||||
activations_dirty: false,
|
activations_dirty: false,
|
||||||
read_only,
|
read_only,
|
||||||
|
signing_key: None,
|
||||||
|
signed: checkpoint.signed,
|
||||||
quarantined_wal,
|
quarantined_wal,
|
||||||
_lock: lock,
|
_lock: lock,
|
||||||
})
|
})
|
||||||
@@ -710,6 +728,39 @@ impl HDF5Memory {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Sign every checkpoint from now on with `key` (Ed25519). The key is
|
||||||
|
/// never written anywhere; set it again after every `open`. Once a store
|
||||||
|
/// is signed, a checkpoint without the key is refused
|
||||||
|
/// ([`MemoryError::SigningKeyRequired`]) rather than silently leaving it
|
||||||
|
/// unsigned. Setting a different key re-signs the store under that key
|
||||||
|
/// from the next checkpoint; a verifier trusting the old key will then
|
||||||
|
/// reject it, which is the point. Call [`AgentMemory::flush_wal`] to sign
|
||||||
|
/// right away.
|
||||||
|
pub fn set_signing_key(&mut self, key: signing::SigningKey) {
|
||||||
|
self.signing_key = Some(key);
|
||||||
|
self.signed = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Stop signing: the next checkpoint writes the store unsigned. The
|
||||||
|
/// deliberate way out of [`MemoryError::SigningKeyRequired`].
|
||||||
|
pub fn remove_signature(&mut self) {
|
||||||
|
self.signing_key = None;
|
||||||
|
self.signed = false;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Checkpoints of this store are signed (on disk, or from the next
|
||||||
|
/// checkpoint because a key has been set).
|
||||||
|
pub fn is_signed(&self) -> bool {
|
||||||
|
self.signed
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Check the checkpoint at `path` against the public key the caller
|
||||||
|
/// trusts; see [`signing::verify_store`]. Reads the file only: it works
|
||||||
|
/// on a store another process has open.
|
||||||
|
pub fn verify(path: &Path, trusted: &signing::VerifyingKey) -> Result<signing::VerifyReport> {
|
||||||
|
signing::verify_store(path, trusted)
|
||||||
|
}
|
||||||
|
|
||||||
/// Flush current state to disk and truncate the WAL.
|
/// Flush current state to disk and truncate the WAL.
|
||||||
///
|
///
|
||||||
/// Every code path that persists the full cache to the .h5 file must
|
/// Every code path that persists the full cache to the .h5 file must
|
||||||
@@ -725,10 +776,28 @@ impl HDF5Memory {
|
|||||||
// Record which WAL prefix this checkpoint contains, so a crash before
|
// Record which WAL prefix this checkpoint contains, so a crash before
|
||||||
// the truncate below can't replay those entries a second time.
|
// the truncate below can't replay those entries a second time.
|
||||||
let wal_applied = self.wal.as_ref().map(|w| w.mark());
|
let wal_applied = self.wal.as_ref().map(|w| w.mark());
|
||||||
|
let signature = match &self.signing_key {
|
||||||
|
Some(key) => Some(signing::sign(
|
||||||
|
key,
|
||||||
|
&self.config,
|
||||||
|
&self.cache,
|
||||||
|
&self.sessions,
|
||||||
|
&self.knowledge,
|
||||||
|
wal_applied,
|
||||||
|
)),
|
||||||
|
None if self.signed => {
|
||||||
|
return Err(MemoryError::SigningKeyRequired(format!(
|
||||||
|
"{} is signed; set its signing key before a checkpoint \
|
||||||
|
(saves so far are held in the WAL or in memory)",
|
||||||
|
self.config.path.display()
|
||||||
|
)));
|
||||||
|
}
|
||||||
|
None => None,
|
||||||
|
};
|
||||||
// Written before the .h5 so a crash in between leaves a sidecar whose
|
// Written before the .h5 so a crash in between leaves a sidecar whose
|
||||||
// generation matches no checkpoint (ignored), never the reverse.
|
// generation matches no checkpoint (ignored), never the reverse.
|
||||||
let ann_generation = self.persist_vector_index();
|
let ann_generation = self.persist_vector_index();
|
||||||
storage::write_to_disk_with_meta(
|
storage::write_to_disk_signed(
|
||||||
&self.config.path,
|
&self.config.path,
|
||||||
&self.config,
|
&self.config,
|
||||||
&self.cache,
|
&self.cache,
|
||||||
@@ -737,7 +806,9 @@ impl HDF5Memory {
|
|||||||
&schema::CheckpointMeta {
|
&schema::CheckpointMeta {
|
||||||
wal_applied,
|
wal_applied,
|
||||||
ann_generation,
|
ann_generation,
|
||||||
|
signed: signature.is_some(),
|
||||||
},
|
},
|
||||||
|
signature.as_ref(),
|
||||||
)?;
|
)?;
|
||||||
if let Some(ref mut w) = self.wal {
|
if let Some(ref mut w) = self.wal {
|
||||||
w.truncate()?;
|
w.truncate()?;
|
||||||
|
|||||||
@@ -23,6 +23,7 @@ pub const ZEROCLAW_VERSION: &str = "0.8.0";
|
|||||||
const WAL_APPLIED_LEN_ATTR: &str = "wal_applied_len";
|
const WAL_APPLIED_LEN_ATTR: &str = "wal_applied_len";
|
||||||
const WAL_APPLIED_CRC_ATTR: &str = "wal_applied_crc";
|
const WAL_APPLIED_CRC_ATTR: &str = "wal_applied_crc";
|
||||||
const ANN_GENERATION_ATTR: &str = "ann_generation";
|
const ANN_GENERATION_ATTR: &str = "ann_generation";
|
||||||
|
const SIG_VERSION_ATTR: &str = "sig_version";
|
||||||
|
|
||||||
/// Build a complete HDF5 file from the in-memory state.
|
/// Build a complete HDF5 file from the in-memory state.
|
||||||
pub fn build_hdf5_file(
|
pub fn build_hdf5_file(
|
||||||
@@ -46,7 +47,7 @@ pub fn build_hdf5_file_with_mark(
|
|||||||
) -> Result<Vec<u8>, MemoryError> {
|
) -> Result<Vec<u8>, MemoryError> {
|
||||||
let meta = CheckpointMeta {
|
let meta = CheckpointMeta {
|
||||||
wal_applied,
|
wal_applied,
|
||||||
ann_generation: None,
|
..CheckpointMeta::default()
|
||||||
};
|
};
|
||||||
build_hdf5_file_with_meta(config, cache, sessions, knowledge, &meta)
|
build_hdf5_file_with_meta(config, cache, sessions, knowledge, &meta)
|
||||||
}
|
}
|
||||||
@@ -61,6 +62,10 @@ pub struct CheckpointMeta {
|
|||||||
/// one left over from another checkpoint can never be attached to records
|
/// one left over from another checkpoint can never be attached to records
|
||||||
/// it wasn't built from.
|
/// it wasn't built from.
|
||||||
pub ann_generation: Option<u64>,
|
pub ann_generation: Option<u64>,
|
||||||
|
/// The checkpoint carries an Ed25519 signature (see [`crate::signing`]).
|
||||||
|
/// Read-only: whether a checkpoint is *written* signed is decided by the
|
||||||
|
/// signature passed to [`build_hdf5_file_signed`].
|
||||||
|
pub signed: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// [`build_hdf5_file`] with checkpoint bookkeeping.
|
/// [`build_hdf5_file`] with checkpoint bookkeeping.
|
||||||
@@ -70,6 +75,19 @@ pub fn build_hdf5_file_with_meta(
|
|||||||
sessions: &SessionCache,
|
sessions: &SessionCache,
|
||||||
knowledge: &KnowledgeCache,
|
knowledge: &KnowledgeCache,
|
||||||
checkpoint: &CheckpointMeta,
|
checkpoint: &CheckpointMeta,
|
||||||
|
) -> Result<Vec<u8>, MemoryError> {
|
||||||
|
build_hdf5_file_signed(config, cache, sessions, knowledge, checkpoint, None)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// [`build_hdf5_file_with_meta`], plus a signed manifest of the contents
|
||||||
|
/// (see [`crate::signing`]).
|
||||||
|
pub fn build_hdf5_file_signed(
|
||||||
|
config: &MemoryConfig,
|
||||||
|
cache: &MemoryCache,
|
||||||
|
sessions: &SessionCache,
|
||||||
|
knowledge: &KnowledgeCache,
|
||||||
|
checkpoint: &CheckpointMeta,
|
||||||
|
signature: Option<&crate::signing::StoredSignature>,
|
||||||
) -> Result<Vec<u8>, MemoryError> {
|
) -> Result<Vec<u8>, MemoryError> {
|
||||||
let wal_applied = checkpoint.wal_applied;
|
let wal_applied = checkpoint.wal_applied;
|
||||||
let mut builder = clawhdf5::FileBuilder::new();
|
let mut builder = clawhdf5::FileBuilder::new();
|
||||||
@@ -130,11 +148,42 @@ pub fn build_hdf5_file_with_meta(
|
|||||||
// round trip through every reader.
|
// round trip through every reader.
|
||||||
meta.set_attr(ANN_GENERATION_ATTR, AttrValue::I64(generation as i64));
|
meta.set_attr(ANN_GENERATION_ATTR, AttrValue::I64(generation as i64));
|
||||||
}
|
}
|
||||||
|
if let Some(sig) = signature {
|
||||||
|
use crate::signing::to_hex;
|
||||||
|
let m = &sig.manifest;
|
||||||
|
meta.set_attr(
|
||||||
|
SIG_VERSION_ATTR,
|
||||||
|
AttrValue::I64(crate::signing::MANIFEST_VERSION),
|
||||||
|
);
|
||||||
|
meta.set_attr("sig_algorithm", AttrValue::String("ed25519".into()));
|
||||||
|
meta.set_attr("sig_public_key", AttrValue::String(to_hex(&sig.public_key)));
|
||||||
|
meta.set_attr("sig_signature", AttrValue::String(to_hex(&sig.signature)));
|
||||||
|
meta.set_attr("sig_record_count", AttrValue::I64(m.record_count as i64));
|
||||||
|
meta.set_attr(
|
||||||
|
"sig_records_root",
|
||||||
|
AttrValue::String(to_hex(&m.records_root)),
|
||||||
|
);
|
||||||
|
meta.set_attr("sig_settings", AttrValue::String(to_hex(&m.settings)));
|
||||||
|
meta.set_attr("sig_sessions", AttrValue::String(to_hex(&m.sessions)));
|
||||||
|
meta.set_attr("sig_graph", AttrValue::String(to_hex(&m.graph)));
|
||||||
|
}
|
||||||
// Need at least one dataset in the group for it to be a proper group
|
// Need at least one dataset in the group for it to be a proper group
|
||||||
meta.create_dataset("_marker").with_u8_data(&[1]).compact();
|
meta.create_dataset("_marker").with_u8_data(&[1]).compact();
|
||||||
let finished_meta = meta.finish();
|
let finished_meta = meta.finish();
|
||||||
builder.add_group(finished_meta);
|
builder.add_group(finished_meta);
|
||||||
|
|
||||||
|
// /integrity: the signed per-record hashes, so verification can say
|
||||||
|
// which records changed.
|
||||||
|
if let Some(sig) = signature {
|
||||||
|
let mut group = builder.create_group("integrity");
|
||||||
|
let flat: Vec<u8> = sig.record_hashes.iter().flatten().copied().collect();
|
||||||
|
group
|
||||||
|
.create_dataset("record_hashes")
|
||||||
|
.with_u8_data(&flat)
|
||||||
|
.with_shape(&[sig.record_hashes.len() as u64, 32]);
|
||||||
|
builder.add_group(group.finish());
|
||||||
|
}
|
||||||
|
|
||||||
// /memory group
|
// /memory group
|
||||||
build_memory_group(&mut builder, config, cache)?;
|
build_memory_group(&mut builder, config, cache)?;
|
||||||
|
|
||||||
@@ -440,6 +489,64 @@ pub fn read_wal_mark(file: &clawhdf5::File) -> Option<WalMark> {
|
|||||||
Some(WalMark { len, crc })
|
Some(WalMark { len, crc })
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Read a checkpoint's signature, if it has one. A signature whose
|
||||||
|
/// attributes are present but malformed is an error, not "unsigned".
|
||||||
|
pub fn read_signature(
|
||||||
|
file: &clawhdf5::File,
|
||||||
|
) -> Result<Option<crate::signing::StoredSignature>, MemoryError> {
|
||||||
|
use crate::signing::{Manifest, StoredSignature, from_hex};
|
||||||
|
let attrs = file
|
||||||
|
.group("meta")
|
||||||
|
.and_then(|g| g.attrs())
|
||||||
|
.map_err(|e| MemoryError::Schema(format!("cannot read /meta attrs: {e}")))?;
|
||||||
|
let version = match attrs.get(SIG_VERSION_ATTR) {
|
||||||
|
None => return Ok(None),
|
||||||
|
Some(AttrValue::I64(v)) => *v,
|
||||||
|
Some(_) => return Err(MemoryError::Schema("malformed sig_version".into())),
|
||||||
|
};
|
||||||
|
if version != crate::signing::MANIFEST_VERSION {
|
||||||
|
return Err(MemoryError::Schema(format!(
|
||||||
|
"unsupported signature version {version}"
|
||||||
|
)));
|
||||||
|
}
|
||||||
|
fn hex<const N: usize>(
|
||||||
|
attrs: &std::collections::HashMap<String, AttrValue>,
|
||||||
|
name: &str,
|
||||||
|
) -> Result<[u8; N], MemoryError> {
|
||||||
|
match attrs.get(name) {
|
||||||
|
Some(AttrValue::String(s)) => from_hex::<N>(s),
|
||||||
|
_ => None,
|
||||||
|
}
|
||||||
|
.ok_or_else(|| MemoryError::Schema(format!("malformed or missing {name}")))
|
||||||
|
}
|
||||||
|
let record_count = match attrs.get("sig_record_count") {
|
||||||
|
Some(AttrValue::I64(v)) if *v >= 0 => *v as u64,
|
||||||
|
_ => return Err(MemoryError::Schema("malformed sig_record_count".into())),
|
||||||
|
};
|
||||||
|
let group = file
|
||||||
|
.group("integrity")
|
||||||
|
.map_err(|e| MemoryError::Schema(format!("signed checkpoint without /integrity: {e}")))?;
|
||||||
|
let flat = read_u8_dataset(&group, "record_hashes")?;
|
||||||
|
if flat.len() % 32 != 0 {
|
||||||
|
return Err(MemoryError::Schema(
|
||||||
|
"/integrity/record_hashes is not a whole number of hashes".into(),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let record_hashes = flat.as_chunks::<32>().0.to_vec();
|
||||||
|
Ok(Some(StoredSignature {
|
||||||
|
manifest: Manifest {
|
||||||
|
record_count,
|
||||||
|
records_root: hex::<32>(&attrs, "sig_records_root")?,
|
||||||
|
settings: hex::<32>(&attrs, "sig_settings")?,
|
||||||
|
sessions: hex::<32>(&attrs, "sig_sessions")?,
|
||||||
|
graph: hex::<32>(&attrs, "sig_graph")?,
|
||||||
|
},
|
||||||
|
record_hashes,
|
||||||
|
public_key: hex::<32>(&attrs, "sig_public_key")?,
|
||||||
|
signature: hex::<64>(&attrs, "sig_signature")?,
|
||||||
|
}))
|
||||||
|
}
|
||||||
|
|
||||||
/// Read the checkpoint bookkeeping from `/meta`.
|
/// Read the checkpoint bookkeeping from `/meta`.
|
||||||
pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta {
|
pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta {
|
||||||
let ann_generation = file
|
let ann_generation = file
|
||||||
@@ -450,9 +557,14 @@ pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta {
|
|||||||
Some(AttrValue::I64(v)) => Some(*v as u64),
|
Some(AttrValue::I64(v)) => Some(*v as u64),
|
||||||
_ => None,
|
_ => None,
|
||||||
});
|
});
|
||||||
|
let signed = file
|
||||||
|
.group("meta")
|
||||||
|
.and_then(|g| g.attrs())
|
||||||
|
.is_ok_and(|attrs| attrs.contains_key(SIG_VERSION_ATTR));
|
||||||
CheckpointMeta {
|
CheckpointMeta {
|
||||||
wal_applied: read_wal_mark(file),
|
wal_applied: read_wal_mark(file),
|
||||||
ann_generation,
|
ann_generation,
|
||||||
|
signed,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,419 @@
|
|||||||
|
//! Ed25519-signed checkpoints.
|
||||||
|
//!
|
||||||
|
//! When a signing key is set ([`crate::HDF5Memory::set_signing_key`]), every
|
||||||
|
//! checkpoint writes a signed manifest of the store: a SHA-256 per memory
|
||||||
|
//! record rolled into a Merkle root, plus hashes of the store's settings, its
|
||||||
|
//! sessions and its knowledge graph. [`verify_store`] recomputes all of it from
|
||||||
|
//! the file and checks the signature against a public key the caller trusts,
|
||||||
|
//! so any change to the checkpointed file — a record's text or embedding, a
|
||||||
|
//! setting, a session, a graph edge, made through this crate or any other HDF5
|
||||||
|
//! tool — is detected, and the per-record hashes say which records changed.
|
||||||
|
//!
|
||||||
|
//! What it does not cover: saves still only in the WAL (made since the last
|
||||||
|
//! checkpoint). [`VerifyReport::wal_entries_unsigned`] counts them.
|
||||||
|
//!
|
||||||
|
//! The hashes cover exactly what the file persists, in the form the loader
|
||||||
|
//! returns it, so a store verifies after any number of reopen/checkpoint
|
||||||
|
//! cycles. Derived data (L2 norms, the vector index) is not covered; it is
|
||||||
|
//! recomputed from covered data.
|
||||||
|
|
||||||
|
use ed25519_dalek::{Signature, Signer, Verifier};
|
||||||
|
pub use ed25519_dalek::{SigningKey, VerifyingKey};
|
||||||
|
use sha2::{Digest, Sha256};
|
||||||
|
|
||||||
|
use crate::MemoryConfig;
|
||||||
|
use crate::cache::MemoryCache;
|
||||||
|
use crate::knowledge::KnowledgeCache;
|
||||||
|
use crate::session::SessionCache;
|
||||||
|
use crate::wal::WalMark;
|
||||||
|
|
||||||
|
/// Version of the manifest encoding; part of what is signed.
|
||||||
|
pub const MANIFEST_VERSION: i64 = 1;
|
||||||
|
|
||||||
|
type Hash = [u8; 32];
|
||||||
|
|
||||||
|
/// The hashes a signature covers.
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
|
pub struct Manifest {
|
||||||
|
pub record_count: u64,
|
||||||
|
/// Merkle root over the per-record hashes.
|
||||||
|
pub records_root: Hash,
|
||||||
|
/// Settings persisted in `/meta`, plus the checkpoint's WAL mark.
|
||||||
|
pub settings: Hash,
|
||||||
|
pub sessions: Hash,
|
||||||
|
pub graph: Hash,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Manifest {
|
||||||
|
/// The exact bytes that are signed.
|
||||||
|
pub fn signed_bytes(&self) -> Vec<u8> {
|
||||||
|
let mut m = Vec::with_capacity(160);
|
||||||
|
m.extend_from_slice(b"clawhdf5-agent signed checkpoint\0");
|
||||||
|
m.extend_from_slice(&MANIFEST_VERSION.to_le_bytes());
|
||||||
|
m.extend_from_slice(&self.record_count.to_le_bytes());
|
||||||
|
m.extend_from_slice(&self.records_root);
|
||||||
|
m.extend_from_slice(&self.settings);
|
||||||
|
m.extend_from_slice(&self.sessions);
|
||||||
|
m.extend_from_slice(&self.graph);
|
||||||
|
m
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A signature as stored in a checkpoint.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct StoredSignature {
|
||||||
|
pub manifest: Manifest,
|
||||||
|
pub record_hashes: Vec<Hash>,
|
||||||
|
pub public_key: [u8; 32],
|
||||||
|
pub signature: [u8; 64],
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Build the manifest (and per-record hashes) for the state about to be
|
||||||
|
/// checkpointed, and sign it.
|
||||||
|
pub fn sign(
|
||||||
|
key: &SigningKey,
|
||||||
|
config: &MemoryConfig,
|
||||||
|
cache: &MemoryCache,
|
||||||
|
sessions: &SessionCache,
|
||||||
|
knowledge: &KnowledgeCache,
|
||||||
|
wal_applied: Option<WalMark>,
|
||||||
|
) -> StoredSignature {
|
||||||
|
let (manifest, record_hashes) = manifest(config, cache, sessions, knowledge, wal_applied);
|
||||||
|
let signature = key.sign(&manifest.signed_bytes()).to_bytes();
|
||||||
|
StoredSignature {
|
||||||
|
manifest,
|
||||||
|
record_hashes,
|
||||||
|
public_key: key.verifying_key().to_bytes(),
|
||||||
|
signature,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Compute the manifest of a store's state.
|
||||||
|
pub fn manifest(
|
||||||
|
config: &MemoryConfig,
|
||||||
|
cache: &MemoryCache,
|
||||||
|
sessions: &SessionCache,
|
||||||
|
knowledge: &KnowledgeCache,
|
||||||
|
wal_applied: Option<WalMark>,
|
||||||
|
) -> (Manifest, Vec<Hash>) {
|
||||||
|
let record_hashes: Vec<Hash> = (0..cache.len()).map(|i| record_hash(cache, i)).collect();
|
||||||
|
let manifest = Manifest {
|
||||||
|
record_count: cache.len() as u64,
|
||||||
|
records_root: merkle_root(&record_hashes),
|
||||||
|
settings: settings_hash(config, wal_applied),
|
||||||
|
sessions: sessions_hash(sessions),
|
||||||
|
graph: graph_hash(knowledge),
|
||||||
|
};
|
||||||
|
(manifest, record_hashes)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Canonical encoding
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/// A SHA-256 over length-prefixed fields, so no two different field lists
|
||||||
|
/// hash the same bytes.
|
||||||
|
struct Fields(Sha256);
|
||||||
|
|
||||||
|
impl Fields {
|
||||||
|
fn new(domain: &str) -> Self {
|
||||||
|
let mut h = Sha256::new();
|
||||||
|
h.update((domain.len() as u64).to_le_bytes());
|
||||||
|
h.update(domain.as_bytes());
|
||||||
|
Self(h)
|
||||||
|
}
|
||||||
|
fn bytes(&mut self, b: &[u8]) -> &mut Self {
|
||||||
|
self.0.update((b.len() as u64).to_le_bytes());
|
||||||
|
self.0.update(b);
|
||||||
|
self
|
||||||
|
}
|
||||||
|
/// Strings as the loader returns them: stored null-padded, so a trailing
|
||||||
|
/// NUL cannot survive a round trip and must not be part of the hash.
|
||||||
|
fn str(&mut self, s: &str) -> &mut Self {
|
||||||
|
self.bytes(s.trim_end_matches('\0').as_bytes())
|
||||||
|
}
|
||||||
|
fn u64(&mut self, v: u64) -> &mut Self {
|
||||||
|
self.0.update(v.to_le_bytes());
|
||||||
|
self
|
||||||
|
}
|
||||||
|
fn f64(&mut self, v: f64) -> &mut Self {
|
||||||
|
self.0.update(v.to_bits().to_le_bytes());
|
||||||
|
self
|
||||||
|
}
|
||||||
|
fn f32(&mut self, v: f32) -> &mut Self {
|
||||||
|
self.0.update(v.to_bits().to_le_bytes());
|
||||||
|
self
|
||||||
|
}
|
||||||
|
fn finish(self) -> Hash {
|
||||||
|
self.0.finalize().into()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Everything persisted about record `i`, including its position. The
|
||||||
|
/// embedding is hashed as the cache holds it — for a `float16` store that is
|
||||||
|
/// the half-rounded value the file holds.
|
||||||
|
fn record_hash(cache: &MemoryCache, i: usize) -> Hash {
|
||||||
|
let mut f = Fields::new("clawhdf5-agent/record");
|
||||||
|
f.u64(i as u64).str(&cache.chunks[i]);
|
||||||
|
let emb: Vec<u8> = cache.embeddings[i]
|
||||||
|
.iter()
|
||||||
|
.flat_map(|v| v.to_bits().to_le_bytes())
|
||||||
|
.collect();
|
||||||
|
f.bytes(&emb)
|
||||||
|
.str(&cache.source_channels[i])
|
||||||
|
.f64(cache.timestamps[i])
|
||||||
|
.str(&cache.session_ids[i])
|
||||||
|
.str(&cache.tags[i])
|
||||||
|
.u64(u64::from(cache.tombstones[i]))
|
||||||
|
.f32(cache.activation_weights[i]);
|
||||||
|
f.finish()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Binary Merkle tree: leaves are the record hashes; a parent hashes its two
|
||||||
|
/// children with a node prefix; an odd node is carried up unchanged.
|
||||||
|
fn merkle_root(leaves: &[Hash]) -> Hash {
|
||||||
|
if leaves.is_empty() {
|
||||||
|
return Fields::new("clawhdf5-agent/merkle-empty").finish();
|
||||||
|
}
|
||||||
|
let mut level: Vec<Hash> = leaves.to_vec();
|
||||||
|
while level.len() > 1 {
|
||||||
|
level = level
|
||||||
|
.chunks(2)
|
||||||
|
.map(|pair| match pair {
|
||||||
|
[l, r] => {
|
||||||
|
let mut h = Sha256::new();
|
||||||
|
h.update([1u8]);
|
||||||
|
h.update(l);
|
||||||
|
h.update(r);
|
||||||
|
h.finalize().into()
|
||||||
|
}
|
||||||
|
[only] => *only,
|
||||||
|
_ => unreachable!(),
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
}
|
||||||
|
level[0]
|
||||||
|
}
|
||||||
|
|
||||||
|
fn settings_hash(c: &MemoryConfig, wal_applied: Option<WalMark>) -> Hash {
|
||||||
|
let mut f = Fields::new("clawhdf5-agent/settings");
|
||||||
|
f.str(crate::schema::SCHEMA_VERSION)
|
||||||
|
.str(&c.created_at)
|
||||||
|
.str(&c.agent_id)
|
||||||
|
.str(&c.embedder)
|
||||||
|
.u64(c.embedding_dim as u64)
|
||||||
|
.u64(c.chunk_size as u64)
|
||||||
|
.u64(c.overlap as u64)
|
||||||
|
.u64(u64::from(c.float16))
|
||||||
|
.u64(u64::from(c.compression))
|
||||||
|
.u64(u64::from(c.compression_level))
|
||||||
|
.f32(c.compact_threshold)
|
||||||
|
.f32(c.hebbian_boost)
|
||||||
|
.f32(c.decay_factor)
|
||||||
|
.u64(u64::from(c.wal_enabled))
|
||||||
|
.u64(c.wal_max_entries as u64)
|
||||||
|
.u64(u64::from(c.quantized_index))
|
||||||
|
.u64(c.hnsw_m as u64)
|
||||||
|
.u64(c.hnsw_ef_construction as u64)
|
||||||
|
.u64(c.hnsw_ef_search as u64);
|
||||||
|
// An empty mark is not written to the file, so it must hash as none.
|
||||||
|
match wal_applied.filter(|m| m.len > 0) {
|
||||||
|
Some(m) => f.u64(1).u64(m.len).u64(u64::from(m.crc)),
|
||||||
|
None => f.u64(0),
|
||||||
|
};
|
||||||
|
f.finish()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn sessions_hash(s: &SessionCache) -> Hash {
|
||||||
|
let mut f = Fields::new("clawhdf5-agent/sessions");
|
||||||
|
f.u64(s.entries.len() as u64);
|
||||||
|
for (i, e) in s.entries.iter().enumerate() {
|
||||||
|
f.str(&e.id)
|
||||||
|
.u64(e.start_idx)
|
||||||
|
.u64(e.end_idx)
|
||||||
|
.str(&e.channel)
|
||||||
|
.f64(e.ts)
|
||||||
|
.str(s.summaries.get(i).map(String::as_str).unwrap_or(""));
|
||||||
|
}
|
||||||
|
f.finish()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn graph_hash(k: &KnowledgeCache) -> Hash {
|
||||||
|
let mut f = Fields::new("clawhdf5-agent/graph");
|
||||||
|
f.u64(k.entities.len() as u64);
|
||||||
|
for e in &k.entities {
|
||||||
|
f.u64(e.id)
|
||||||
|
.str(&e.name)
|
||||||
|
.str(&e.entity_type)
|
||||||
|
.u64(e.embedding_idx as u64);
|
||||||
|
}
|
||||||
|
f.u64(k.relations.len() as u64);
|
||||||
|
for r in &k.relations {
|
||||||
|
f.u64(r.src)
|
||||||
|
.u64(r.tgt)
|
||||||
|
.str(&r.relation)
|
||||||
|
.f32(r.weight)
|
||||||
|
.f64(r.ts);
|
||||||
|
}
|
||||||
|
f.u64(k.alias_strings.len() as u64);
|
||||||
|
for (s, id) in k.alias_strings.iter().zip(&k.alias_entity_ids) {
|
||||||
|
f.str(s).u64(*id as u64);
|
||||||
|
}
|
||||||
|
f.finish()
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Verification
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/// The outcome of [`verify_store`].
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
|
pub struct VerifyReport {
|
||||||
|
/// The checkpoint carries a signature.
|
||||||
|
pub signed: bool,
|
||||||
|
/// The signature was made by the key the caller trusts.
|
||||||
|
pub key_matches: bool,
|
||||||
|
/// The signature over the stored manifest is valid.
|
||||||
|
pub signature_valid: bool,
|
||||||
|
/// The file's current contents match the signed manifest.
|
||||||
|
pub records_match: bool,
|
||||||
|
pub settings_match: bool,
|
||||||
|
pub sessions_match: bool,
|
||||||
|
pub graph_match: bool,
|
||||||
|
/// Records whose contents differ from what was signed (by position),
|
||||||
|
/// when the stored per-record hashes are themselves authentic.
|
||||||
|
pub changed_records: Vec<usize>,
|
||||||
|
/// Records in the file versus in the signed manifest.
|
||||||
|
pub record_count: u64,
|
||||||
|
pub signed_record_count: u64,
|
||||||
|
/// The public key the checkpoint claims to be signed by.
|
||||||
|
pub public_key: Option<[u8; 32]>,
|
||||||
|
/// Saves in the WAL after the checkpoint: not covered by the signature.
|
||||||
|
pub wal_entries_unsigned: usize,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl VerifyReport {
|
||||||
|
/// Signed by the trusted key, signature valid, and every part of the
|
||||||
|
/// file unchanged since it was signed.
|
||||||
|
pub fn is_valid(&self) -> bool {
|
||||||
|
self.signed
|
||||||
|
&& self.key_matches
|
||||||
|
&& self.signature_valid
|
||||||
|
&& self.records_match
|
||||||
|
&& self.settings_match
|
||||||
|
&& self.sessions_match
|
||||||
|
&& self.graph_match
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Check a store file against the public key the caller trusts.
|
||||||
|
///
|
||||||
|
/// Reads the checkpoint (not the WAL), recomputes every hash from its
|
||||||
|
/// contents and checks the signature. Never writes.
|
||||||
|
pub fn verify_store(
|
||||||
|
path: &std::path::Path,
|
||||||
|
trusted: &VerifyingKey,
|
||||||
|
) -> Result<VerifyReport, crate::MemoryError> {
|
||||||
|
let file = clawhdf5::File::open(path)
|
||||||
|
.map_err(|e| crate::MemoryError::Hdf5(format!("cannot open {}: {e}", path.display())))?;
|
||||||
|
let (config, cache, sessions, knowledge) = crate::schema::validate_and_load(&file)?;
|
||||||
|
let checkpoint = crate::schema::read_checkpoint_meta(&file);
|
||||||
|
let stored = crate::schema::read_signature(&file)?;
|
||||||
|
let wal_entries_unsigned = count_wal_entries_after(path, checkpoint.wal_applied);
|
||||||
|
|
||||||
|
let (current, current_hashes) = manifest(
|
||||||
|
&config,
|
||||||
|
&cache,
|
||||||
|
&sessions,
|
||||||
|
&knowledge,
|
||||||
|
checkpoint.wal_applied,
|
||||||
|
);
|
||||||
|
|
||||||
|
let Some(stored) = stored else {
|
||||||
|
return Ok(VerifyReport {
|
||||||
|
signed: false,
|
||||||
|
key_matches: false,
|
||||||
|
signature_valid: false,
|
||||||
|
records_match: false,
|
||||||
|
settings_match: false,
|
||||||
|
sessions_match: false,
|
||||||
|
graph_match: false,
|
||||||
|
changed_records: Vec::new(),
|
||||||
|
record_count: current.record_count,
|
||||||
|
signed_record_count: 0,
|
||||||
|
public_key: None,
|
||||||
|
wal_entries_unsigned,
|
||||||
|
});
|
||||||
|
};
|
||||||
|
|
||||||
|
let key_matches = stored.public_key == trusted.to_bytes();
|
||||||
|
let signature_valid = trusted
|
||||||
|
.verify(
|
||||||
|
&stored.manifest.signed_bytes(),
|
||||||
|
&Signature::from_bytes(&stored.signature),
|
||||||
|
)
|
||||||
|
.is_ok();
|
||||||
|
// The stored per-record hashes can localise a change only if they are
|
||||||
|
// the ones that were signed.
|
||||||
|
let hashes_authentic = signature_valid
|
||||||
|
&& stored.record_hashes.len() as u64 == stored.manifest.record_count
|
||||||
|
&& merkle_root(&stored.record_hashes) == stored.manifest.records_root;
|
||||||
|
let changed_records = if hashes_authentic {
|
||||||
|
let n = current_hashes.len().max(stored.record_hashes.len());
|
||||||
|
(0..n)
|
||||||
|
.filter(|&i| current_hashes.get(i) != stored.record_hashes.get(i))
|
||||||
|
.collect()
|
||||||
|
} else {
|
||||||
|
Vec::new()
|
||||||
|
};
|
||||||
|
|
||||||
|
Ok(VerifyReport {
|
||||||
|
signed: true,
|
||||||
|
key_matches,
|
||||||
|
signature_valid,
|
||||||
|
records_match: signature_valid
|
||||||
|
&& current.record_count == stored.manifest.record_count
|
||||||
|
&& current.records_root == stored.manifest.records_root,
|
||||||
|
settings_match: signature_valid && current.settings == stored.manifest.settings,
|
||||||
|
sessions_match: signature_valid && current.sessions == stored.manifest.sessions,
|
||||||
|
graph_match: signature_valid && current.graph == stored.manifest.graph,
|
||||||
|
changed_records,
|
||||||
|
record_count: current.record_count,
|
||||||
|
signed_record_count: stored.manifest.record_count,
|
||||||
|
public_key: Some(stored.public_key),
|
||||||
|
wal_entries_unsigned,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn count_wal_entries_after(store: &std::path::Path, mark: Option<WalMark>) -> usize {
|
||||||
|
let wal = store.with_extension("h5.wal");
|
||||||
|
if !wal.exists() {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
crate::wal::WalFile::read_entries_for_migration(&wal, mark)
|
||||||
|
.map(|e| e.len())
|
||||||
|
.unwrap_or(0)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A new random signing key from the operating system's RNG.
|
||||||
|
pub fn generate_key() -> SigningKey {
|
||||||
|
SigningKey::generate(&mut rand_core::OsRng)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Hex encoding for keys and signatures in attributes and the CLI.
|
||||||
|
pub fn to_hex(bytes: &[u8]) -> String {
|
||||||
|
bytes.iter().map(|b| format!("{b:02x}")).collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Parse hex into exactly `N` bytes.
|
||||||
|
pub fn from_hex<const N: usize>(s: &str) -> Option<[u8; N]> {
|
||||||
|
let s = s.trim();
|
||||||
|
if s.len() != 2 * N {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let mut out = [0u8; N];
|
||||||
|
for (i, byte) in out.iter_mut().enumerate() {
|
||||||
|
*byte = u8::from_str_radix(&s[2 * i..2 * i + 2], 16).ok()?;
|
||||||
|
}
|
||||||
|
Some(out)
|
||||||
|
}
|
||||||
@@ -36,7 +36,7 @@ pub fn write_to_disk_with_mark(
|
|||||||
) -> Result<(), MemoryError> {
|
) -> Result<(), MemoryError> {
|
||||||
let meta = schema::CheckpointMeta {
|
let meta = schema::CheckpointMeta {
|
||||||
wal_applied,
|
wal_applied,
|
||||||
ann_generation: None,
|
..schema::CheckpointMeta::default()
|
||||||
};
|
};
|
||||||
write_to_disk_with_meta(path, config, cache, sessions, knowledge, &meta)
|
write_to_disk_with_meta(path, config, cache, sessions, knowledge, &meta)
|
||||||
}
|
}
|
||||||
@@ -50,7 +50,21 @@ pub fn write_to_disk_with_meta(
|
|||||||
knowledge: &KnowledgeCache,
|
knowledge: &KnowledgeCache,
|
||||||
checkpoint: &schema::CheckpointMeta,
|
checkpoint: &schema::CheckpointMeta,
|
||||||
) -> Result<(), MemoryError> {
|
) -> Result<(), MemoryError> {
|
||||||
let bytes = schema::build_hdf5_file_with_meta(config, cache, sessions, knowledge, checkpoint)?;
|
write_to_disk_signed(path, config, cache, sessions, knowledge, checkpoint, None)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// [`write_to_disk_with_meta`] with a signed manifest of the contents.
|
||||||
|
pub fn write_to_disk_signed(
|
||||||
|
path: &Path,
|
||||||
|
config: &MemoryConfig,
|
||||||
|
cache: &MemoryCache,
|
||||||
|
sessions: &SessionCache,
|
||||||
|
knowledge: &KnowledgeCache,
|
||||||
|
checkpoint: &schema::CheckpointMeta,
|
||||||
|
signature: Option<&crate::signing::StoredSignature>,
|
||||||
|
) -> Result<(), MemoryError> {
|
||||||
|
let bytes =
|
||||||
|
schema::build_hdf5_file_signed(config, cache, sessions, knowledge, checkpoint, signature)?;
|
||||||
|
|
||||||
if bytes.is_empty() {
|
if bytes.is_empty() {
|
||||||
return Err(MemoryError::Hdf5("build_hdf5_file produced 0 bytes".into()));
|
return Err(MemoryError::Hdf5("build_hdf5_file produced 0 bytes".into()));
|
||||||
|
|||||||
@@ -92,3 +92,64 @@ print(len(names))
|
|||||||
assert!(n >= 10, "only {n} datasets");
|
assert!(n >= 10, "only {n} datasets");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_edit_made_with_h5py_breaks_the_signature_and_names_the_record() {
|
||||||
|
if !h5py_available() {
|
||||||
|
assert!(
|
||||||
|
std::env::var("CLAWHDF5_REQUIRE_INTEROP").as_deref() != Ok("1"),
|
||||||
|
"CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
|
||||||
|
);
|
||||||
|
eprintln!("SKIP: python3 with h5py not available");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
use clawhdf5_agent::signing::SigningKey;
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let path = dir.path().join("signed.h5");
|
||||||
|
let key = SigningKey::from_bytes(&[42; 32]);
|
||||||
|
let mut m = HDF5Memory::create(MemoryConfig::new(path.clone(), "agent", 8)).unwrap();
|
||||||
|
m.set_signing_key(key.clone());
|
||||||
|
m.save_batch(
|
||||||
|
(0..10)
|
||||||
|
.map(|i| MemoryEntry {
|
||||||
|
chunk: format!("memory {i}"),
|
||||||
|
embedding: (0..8).map(|j| ((i * 8 + j) as f32).cos()).collect(),
|
||||||
|
source_channel: "test".into(),
|
||||||
|
timestamp: i as f64,
|
||||||
|
session_id: "s".into(),
|
||||||
|
tags: String::new(),
|
||||||
|
})
|
||||||
|
.collect(),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
drop(m);
|
||||||
|
assert!(
|
||||||
|
HDF5Memory::verify(&path, &key.verifying_key())
|
||||||
|
.unwrap()
|
||||||
|
.is_valid()
|
||||||
|
);
|
||||||
|
|
||||||
|
// Someone edits one timestamp in place with h5py.
|
||||||
|
let script = format!(
|
||||||
|
r#"
|
||||||
|
import h5py
|
||||||
|
with h5py.File("{}", "r+") as f:
|
||||||
|
ts = f["memory/timestamps"]
|
||||||
|
ts[3] = 12345.0
|
||||||
|
"#,
|
||||||
|
path.display()
|
||||||
|
);
|
||||||
|
let out = Command::new(python())
|
||||||
|
.args(["-c", &script])
|
||||||
|
.output()
|
||||||
|
.unwrap();
|
||||||
|
assert!(
|
||||||
|
out.status.success(),
|
||||||
|
"{}",
|
||||||
|
String::from_utf8_lossy(&out.stderr)
|
||||||
|
);
|
||||||
|
|
||||||
|
let r = HDF5Memory::verify(&path, &key.verifying_key()).unwrap();
|
||||||
|
assert!(r.signature_valid && !r.is_valid(), "{r:?}");
|
||||||
|
assert_eq!(r.changed_records, vec![3]);
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,330 @@
|
|||||||
|
//! Ed25519-signed checkpoints: `HDF5Memory::set_signing_key` and
|
||||||
|
//! `HDF5Memory::verify`.
|
||||||
|
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
use clawhdf5_agent::signing::{SigningKey, VerifyReport, VerifyingKey};
|
||||||
|
use clawhdf5_agent::storage;
|
||||||
|
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, MemoryError, schema};
|
||||||
|
use tempfile::TempDir;
|
||||||
|
|
||||||
|
const DIM: usize = 16;
|
||||||
|
|
||||||
|
fn key(seed: u8) -> SigningKey {
|
||||||
|
SigningKey::from_bytes(&[seed; 32])
|
||||||
|
}
|
||||||
|
|
||||||
|
fn entry(i: usize, chunk: &str) -> MemoryEntry {
|
||||||
|
MemoryEntry {
|
||||||
|
chunk: chunk.to_string(),
|
||||||
|
embedding: (0..DIM)
|
||||||
|
.map(|j| ((i * DIM + j) as f32 * 0.37).sin())
|
||||||
|
.collect(),
|
||||||
|
source_channel: "chat".into(),
|
||||||
|
timestamp: 1_700_000_000.0 + i as f64,
|
||||||
|
session_id: format!("s{}", i % 3),
|
||||||
|
tags: format!("t{i}"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Awkward strings on purpose: they must hash the same after a round trip.
|
||||||
|
const TEXTS: [&str; 6] = [
|
||||||
|
"plain text",
|
||||||
|
"ünïcödé — 日本語 🙂",
|
||||||
|
"",
|
||||||
|
"trailing spaces ",
|
||||||
|
"tab\tand\nnewline",
|
||||||
|
"x",
|
||||||
|
];
|
||||||
|
|
||||||
|
fn signed_store(dir: &TempDir, float16: bool, k: &SigningKey) -> std::path::PathBuf {
|
||||||
|
let mut cfg = MemoryConfig::new(dir.path().join("s.h5"), "agent", DIM);
|
||||||
|
cfg.float16 = float16;
|
||||||
|
let path = cfg.path.clone();
|
||||||
|
let mut m = HDF5Memory::create(cfg).unwrap();
|
||||||
|
m.set_signing_key(k.clone());
|
||||||
|
let entries = (0..30).map(|i| entry(i, TEXTS[i % TEXTS.len()])).collect();
|
||||||
|
m.save_batch(entries).unwrap();
|
||||||
|
// Some graph and a deleted record, so every part of the manifest is used.
|
||||||
|
let a = m.knowledge_mut().add_entity("Alice", "person", 0);
|
||||||
|
let b = m.knowledge_mut().add_entity("Acme", "org", -1);
|
||||||
|
m.knowledge_mut().add_relation(a, b, "works_at", 0.75);
|
||||||
|
m.sessions_mut()
|
||||||
|
.add_at("s0", 0, 9, "chat", "first session", 1_700_000_000.0);
|
||||||
|
m.delete(4).unwrap();
|
||||||
|
m.flush_wal().unwrap();
|
||||||
|
path
|
||||||
|
}
|
||||||
|
|
||||||
|
fn verify(path: &Path, k: &SigningKey) -> VerifyReport {
|
||||||
|
HDF5Memory::verify(path, &k.verifying_key()).unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_signed_store_verifies_through_reopen_and_checkpoint_cycles() {
|
||||||
|
for float16 in [true, false] {
|
||||||
|
let dir = TempDir::new().unwrap();
|
||||||
|
let k = key(7);
|
||||||
|
let path = signed_store(&dir, float16, &k);
|
||||||
|
let r = verify(&path, &k);
|
||||||
|
assert!(r.is_valid(), "float16={float16}: {r:?}");
|
||||||
|
assert_eq!(r.public_key, Some(k.verifying_key().to_bytes()));
|
||||||
|
assert_eq!(r.record_count, 30);
|
||||||
|
assert!(r.changed_records.is_empty());
|
||||||
|
|
||||||
|
// Reopen, change nothing, checkpoint again (with the key): still valid.
|
||||||
|
for _ in 0..3 {
|
||||||
|
let mut m = HDF5Memory::open(&path).unwrap();
|
||||||
|
assert!(m.is_signed());
|
||||||
|
m.set_signing_key(k.clone());
|
||||||
|
m.flush_wal().unwrap();
|
||||||
|
drop(m);
|
||||||
|
assert!(verify(&path, &k).is_valid());
|
||||||
|
}
|
||||||
|
// And after real changes, re-signed.
|
||||||
|
let mut m = HDF5Memory::open(&path).unwrap();
|
||||||
|
m.set_signing_key(k.clone());
|
||||||
|
m.save(entry(99, "added later")).unwrap();
|
||||||
|
m.hybrid_search(&entry(1, "").embedding, "text", 0.4, 0.6, 5);
|
||||||
|
m.flush_wal().unwrap();
|
||||||
|
drop(m);
|
||||||
|
let r = verify(&path, &k);
|
||||||
|
assert!(r.is_valid(), "{r:?}");
|
||||||
|
assert_eq!(r.record_count, 31);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_signed_store_refuses_to_checkpoint_without_its_key() {
|
||||||
|
let dir = TempDir::new().unwrap();
|
||||||
|
let k = key(1);
|
||||||
|
let path = signed_store(&dir, true, &k);
|
||||||
|
|
||||||
|
let mut m = HDF5Memory::open(&path).unwrap();
|
||||||
|
m.save(entry(50, "pending")).unwrap();
|
||||||
|
match m.flush_wal() {
|
||||||
|
Err(MemoryError::SigningKeyRequired(msg)) => assert!(msg.contains("signed"), "{msg}"),
|
||||||
|
other => panic!("expected SigningKeyRequired, got {other:?}"),
|
||||||
|
}
|
||||||
|
// The file is untouched and still valid; the save is still in the WAL.
|
||||||
|
let r = verify(&path, &k);
|
||||||
|
assert!(r.is_valid());
|
||||||
|
assert_eq!(r.wal_entries_unsigned, 1);
|
||||||
|
|
||||||
|
// Supplying the key lets the checkpoint through, signed.
|
||||||
|
m.set_signing_key(k.clone());
|
||||||
|
m.flush_wal().unwrap();
|
||||||
|
drop(m);
|
||||||
|
let r = verify(&path, &k);
|
||||||
|
assert!(r.is_valid());
|
||||||
|
assert_eq!((r.record_count, r.wal_entries_unsigned), (31, 0));
|
||||||
|
|
||||||
|
// Removing the signature on purpose writes it unsigned.
|
||||||
|
let mut m = HDF5Memory::open(&path).unwrap();
|
||||||
|
m.remove_signature();
|
||||||
|
m.flush_wal().unwrap();
|
||||||
|
drop(m);
|
||||||
|
let r = verify(&path, &k);
|
||||||
|
assert!(!r.signed && !r.is_valid());
|
||||||
|
assert!(!HDF5Memory::open(&path).unwrap().is_signed());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_wrong_key_does_not_verify_and_a_new_key_re_signs() {
|
||||||
|
let dir = TempDir::new().unwrap();
|
||||||
|
let (a, b) = (key(1), key(2));
|
||||||
|
let path = signed_store(&dir, true, &a);
|
||||||
|
let r = verify(&path, &b);
|
||||||
|
assert!(r.signed && !r.key_matches && !r.signature_valid && !r.is_valid());
|
||||||
|
|
||||||
|
let mut m = HDF5Memory::open(&path).unwrap();
|
||||||
|
m.set_signing_key(b.clone());
|
||||||
|
m.flush_wal().unwrap();
|
||||||
|
drop(m);
|
||||||
|
assert!(verify(&path, &b).is_valid());
|
||||||
|
assert!(!verify(&path, &a).is_valid());
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Rewrite the store with changed contents but the *old* signature — what
|
||||||
|
/// someone with write access to the file, but not the key, can do.
|
||||||
|
fn tamper(path: &Path, change: impl FnOnce(&mut Tampered)) {
|
||||||
|
let file = clawhdf5::File::open(path).unwrap();
|
||||||
|
let (config, cache, sessions, knowledge) = schema::validate_and_load(&file).unwrap();
|
||||||
|
let checkpoint = schema::read_checkpoint_meta(&file);
|
||||||
|
let signature = schema::read_signature(&file).unwrap().unwrap();
|
||||||
|
drop(file);
|
||||||
|
let mut t = Tampered {
|
||||||
|
config,
|
||||||
|
cache,
|
||||||
|
sessions,
|
||||||
|
knowledge,
|
||||||
|
};
|
||||||
|
change(&mut t);
|
||||||
|
storage::write_to_disk_signed(
|
||||||
|
path,
|
||||||
|
&t.config,
|
||||||
|
&t.cache,
|
||||||
|
&t.sessions,
|
||||||
|
&t.knowledge,
|
||||||
|
&checkpoint,
|
||||||
|
Some(&signature),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
struct Tampered {
|
||||||
|
config: MemoryConfig,
|
||||||
|
cache: clawhdf5_agent::cache::MemoryCache,
|
||||||
|
sessions: clawhdf5_agent::SessionCache,
|
||||||
|
knowledge: clawhdf5_agent::knowledge::KnowledgeCache,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn every_kind_of_edit_is_detected_and_located() {
|
||||||
|
let k = key(3);
|
||||||
|
type Edit = Box<dyn FnOnce(&mut Tampered)>;
|
||||||
|
type Case = (&'static str, Edit, fn(&VerifyReport) -> bool);
|
||||||
|
let cases: Vec<Case> = vec![
|
||||||
|
(
|
||||||
|
"record text",
|
||||||
|
Box::new(|t: &mut Tampered| t.cache.chunks[7] = "rewritten".into()),
|
||||||
|
|r| !r.records_match && r.changed_records == vec![7],
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"one embedding value",
|
||||||
|
Box::new(|t: &mut Tampered| {
|
||||||
|
let mut e = t.cache.embeddings[12].to_vec();
|
||||||
|
e[3] = 0.5;
|
||||||
|
t.cache.embeddings.set(12, &e);
|
||||||
|
}),
|
||||||
|
|r| r.changed_records == vec![12],
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"undelete",
|
||||||
|
Box::new(|t: &mut Tampered| t.cache.tombstones[4] = 0),
|
||||||
|
|r| r.changed_records == vec![4],
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"timestamp",
|
||||||
|
Box::new(|t: &mut Tampered| t.cache.timestamps[20] += 1.0),
|
||||||
|
|r| r.changed_records == vec![20],
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"record appended",
|
||||||
|
Box::new(|t: &mut Tampered| {
|
||||||
|
t.cache.push(
|
||||||
|
"new".into(),
|
||||||
|
vec![0.1; DIM],
|
||||||
|
"x".into(),
|
||||||
|
1.0,
|
||||||
|
"s".into(),
|
||||||
|
"".into(),
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
|r| !r.records_match && r.changed_records == vec![30] && r.record_count == 31,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"setting",
|
||||||
|
Box::new(|t: &mut Tampered| t.config.agent_id = "someone-else".into()),
|
||||||
|
|r| !r.settings_match && r.records_match,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"session summary",
|
||||||
|
Box::new(|t: &mut Tampered| t.sessions.summaries[0] = "edited".into()),
|
||||||
|
|r| !r.sessions_match && r.records_match,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"graph edge",
|
||||||
|
Box::new(|t: &mut Tampered| t.knowledge.relations[0].weight = 1.0),
|
||||||
|
|r| !r.graph_match && r.records_match,
|
||||||
|
),
|
||||||
|
];
|
||||||
|
for (name, edit, check) in cases {
|
||||||
|
let dir = TempDir::new().unwrap();
|
||||||
|
let path = signed_store(&dir, true, &k);
|
||||||
|
tamper(&path, edit);
|
||||||
|
let r = verify(&path, &k);
|
||||||
|
assert!(
|
||||||
|
r.signed && r.key_matches && r.signature_valid,
|
||||||
|
"{name}: {r:?}"
|
||||||
|
);
|
||||||
|
assert!(!r.is_valid(), "{name}: edit not detected: {r:?}");
|
||||||
|
assert!(check(&r), "{name}: {r:?}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_forged_manifest_fails_the_signature() {
|
||||||
|
// Recomputing the hashes for tampered contents does not help without the
|
||||||
|
// key: the signature no longer matches the manifest.
|
||||||
|
let dir = TempDir::new().unwrap();
|
||||||
|
let k = key(5);
|
||||||
|
let path = signed_store(&dir, true, &k);
|
||||||
|
let file = clawhdf5::File::open(&path).unwrap();
|
||||||
|
let (config, mut cache, sessions, knowledge) = schema::validate_and_load(&file).unwrap();
|
||||||
|
let checkpoint = schema::read_checkpoint_meta(&file);
|
||||||
|
let mut sig = schema::read_signature(&file).unwrap().unwrap();
|
||||||
|
drop(file);
|
||||||
|
cache.chunks[0] = "forged".into();
|
||||||
|
// Re-sign with an attacker key, then splice the victim's public key back.
|
||||||
|
let forged = clawhdf5_agent::signing::sign(
|
||||||
|
&key(66),
|
||||||
|
&config,
|
||||||
|
&cache,
|
||||||
|
&sessions,
|
||||||
|
&knowledge,
|
||||||
|
checkpoint.wal_applied,
|
||||||
|
);
|
||||||
|
sig.manifest = forged.manifest;
|
||||||
|
sig.record_hashes = forged.record_hashes;
|
||||||
|
storage::write_to_disk_signed(
|
||||||
|
&path,
|
||||||
|
&config,
|
||||||
|
&cache,
|
||||||
|
&sessions,
|
||||||
|
&knowledge,
|
||||||
|
&checkpoint,
|
||||||
|
Some(&sig),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
let r = verify(&path, &k);
|
||||||
|
assert!(
|
||||||
|
r.key_matches && !r.signature_valid && !r.is_valid(),
|
||||||
|
"{r:?}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_unsigned_store_reports_unsigned() {
|
||||||
|
let dir = TempDir::new().unwrap();
|
||||||
|
let mut m = HDF5Memory::create(MemoryConfig::new(dir.path().join("u.h5"), "a", DIM)).unwrap();
|
||||||
|
m.save_batch(vec![entry(0, "hello")]).unwrap();
|
||||||
|
drop(m);
|
||||||
|
let r = HDF5Memory::verify(&dir.path().join("u.h5"), &VerifyingKey::from(&key(1))).unwrap();
|
||||||
|
assert!(!r.signed && !r.is_valid());
|
||||||
|
assert_eq!(r.record_count, 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn nul_bytes_in_text_still_verify() {
|
||||||
|
// Strings are stored null-padded; the hash must follow what a reopened
|
||||||
|
// store actually holds, or an untouched store would fail to verify.
|
||||||
|
let dir = TempDir::new().unwrap();
|
||||||
|
let k = key(9);
|
||||||
|
let mut m = HDF5Memory::create(MemoryConfig::new(dir.path().join("n.h5"), "a", DIM)).unwrap();
|
||||||
|
m.set_signing_key(k.clone());
|
||||||
|
m.save_batch(vec![
|
||||||
|
entry(0, "inner\0nul"),
|
||||||
|
entry(1, "trailing nul\0"),
|
||||||
|
entry(2, "\0leading"),
|
||||||
|
])
|
||||||
|
.unwrap();
|
||||||
|
drop(m);
|
||||||
|
let r = verify(&dir.path().join("n.h5"), &k);
|
||||||
|
assert!(r.is_valid(), "{r:?}");
|
||||||
|
let m = HDF5Memory::open(&dir.path().join("n.h5")).unwrap();
|
||||||
|
eprintln!(
|
||||||
|
"reloaded: {:?}",
|
||||||
|
(0..3).map(|i| m.get_chunk(i)).collect::<Vec<_>>()
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -21,6 +21,7 @@
|
|||||||
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --ann-only --uniform
|
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --ann-only --uniform
|
||||||
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full
|
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full
|
||||||
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --options-study --full
|
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --options-study --full
|
||||||
|
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --signing-study --full
|
||||||
//! ```
|
//! ```
|
||||||
|
|
||||||
use std::time::{Duration, Instant};
|
use std::time::{Duration, Instant};
|
||||||
@@ -488,6 +489,81 @@ fn bench_end_to_end(n: usize, json: &mut Vec<serde_json::Value>) {
|
|||||||
}));
|
}));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Signing study: what does an Ed25519-signed checkpoint cost?
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/// `--signing-study`: checkpoint time unsigned vs signed, `verify` time, and
|
||||||
|
/// the file-size cost of the stored per-record hashes. Default store
|
||||||
|
/// settings (float16, int8 index). Medians of five checkpoints / three
|
||||||
|
/// verifies.
|
||||||
|
fn signing_study(n: usize) {
|
||||||
|
use clawhdf5_agent::signing::SigningKey;
|
||||||
|
let data = make_dataset(n, 0x516 ^ n as u64);
|
||||||
|
let mut rng = Rng(9);
|
||||||
|
let entries: Vec<MemoryEntry> = data
|
||||||
|
.vectors
|
||||||
|
.iter()
|
||||||
|
.enumerate()
|
||||||
|
.map(|(i, v)| MemoryEntry {
|
||||||
|
chunk: text_for(data.cluster_of[i], i, &mut rng),
|
||||||
|
embedding: v.clone(),
|
||||||
|
source_channel: "bench".into(),
|
||||||
|
timestamp: i as f64,
|
||||||
|
session_id: format!("s{}", i % 50),
|
||||||
|
tags: format!("t{i}"),
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let path = dir.path().join("sign.h5");
|
||||||
|
let mut mem = HDF5Memory::create(MemoryConfig::new(path.clone(), "bench", DIM)).unwrap();
|
||||||
|
mem.save_batch(entries).unwrap();
|
||||||
|
std::hint::black_box(mem.hybrid_search(&data.queries[0], "", 1.0, 0.0, K));
|
||||||
|
|
||||||
|
let median = |mut v: Vec<Duration>| {
|
||||||
|
v.sort();
|
||||||
|
v[v.len() / 2]
|
||||||
|
};
|
||||||
|
let checkpoint = |mem: &mut HDF5Memory| {
|
||||||
|
median(
|
||||||
|
(0..5)
|
||||||
|
.map(|_| {
|
||||||
|
let t = Instant::now();
|
||||||
|
mem.flush_wal().unwrap();
|
||||||
|
t.elapsed()
|
||||||
|
})
|
||||||
|
.collect(),
|
||||||
|
)
|
||||||
|
};
|
||||||
|
let unsigned = checkpoint(&mut mem);
|
||||||
|
let unsigned_bytes = std::fs::metadata(&path).unwrap().len();
|
||||||
|
let key = SigningKey::from_bytes(&[7; 32]);
|
||||||
|
mem.set_signing_key(key.clone());
|
||||||
|
let signed = checkpoint(&mut mem);
|
||||||
|
let signed_bytes = std::fs::metadata(&path).unwrap().len();
|
||||||
|
drop(mem);
|
||||||
|
let vk = key.verifying_key();
|
||||||
|
let verify = median(
|
||||||
|
(0..3)
|
||||||
|
.map(|_| {
|
||||||
|
let t = Instant::now();
|
||||||
|
let r = HDF5Memory::verify(&path, &vk).unwrap();
|
||||||
|
let d = t.elapsed();
|
||||||
|
assert!(r.is_valid());
|
||||||
|
d
|
||||||
|
})
|
||||||
|
.collect(),
|
||||||
|
);
|
||||||
|
println!(
|
||||||
|
"| {n} | {:.1} | {:.1} | {:+.1} | {:.1} | {:+.2} |",
|
||||||
|
millis(unsigned),
|
||||||
|
millis(signed),
|
||||||
|
millis(signed) - millis(unsigned),
|
||||||
|
millis(verify),
|
||||||
|
(signed_bytes as f64 - unsigned_bytes as f64) / (1024.0 * 1024.0),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// Search options study: source filters, re-ranking, confidence rejection
|
// Search options study: source filters, re-ranking, confidence rejection
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -960,6 +1036,21 @@ fn main() {
|
|||||||
}
|
}
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
if args.iter().any(|a| a == "--signing-study") {
|
||||||
|
println!("## Signed checkpoints ({DIM}-dim, float16, int8 index)\n");
|
||||||
|
println!(
|
||||||
|
"| N | checkpoint ms, unsigned | checkpoint ms, signed | signing adds ms | verify ms | file MiB added |"
|
||||||
|
);
|
||||||
|
println!("|---:|---:|---:|---:|---:|---:|");
|
||||||
|
for &n in if full {
|
||||||
|
&[1_000, 10_000, 100_000][..]
|
||||||
|
} else {
|
||||||
|
&[1_000, 10_000][..]
|
||||||
|
} {
|
||||||
|
signing_study(n);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
if args.iter().any(|a| a == "--options-study") {
|
if args.iter().any(|a| a == "--options-study") {
|
||||||
println!("## Search options ({DIM}-dim, k = {K}, Hebbian boost off)\n");
|
println!("## Search options ({DIM}-dim, k = {K}, Hebbian boost off)\n");
|
||||||
println!("| N | options | filtered recall@10 | p50 ms | p99 ms |");
|
println!("| N | options | filtered recall@10 | p50 ms | p99 ms |");
|
||||||
|
|||||||
+126
-16
@@ -1,15 +1,22 @@
|
|||||||
use std::path::PathBuf;
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
use clap::{Parser, Subcommand};
|
use clap::{Parser, Subcommand};
|
||||||
|
use clawhdf5_agent::signing::{self, SigningKey, VerifyingKey};
|
||||||
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
|
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
|
||||||
|
|
||||||
/// ClawhDF5 — HDF5-backed cognitive memory for AI agents
|
/// ClawhDF5 — HDF5-backed cognitive memory for AI agents
|
||||||
#[derive(Parser)]
|
#[derive(Parser)]
|
||||||
#[command(name = "clawhdf5", version, about)]
|
#[command(name = "clawhdf5", version, about)]
|
||||||
struct Cli {
|
struct Cli {
|
||||||
/// Path to the .h5 memory file
|
/// Path to the .h5 memory file (not needed for `keygen`)
|
||||||
#[arg(short, long, env = "CLAWHDF5_PATH")]
|
#[arg(short, long, env = "CLAWHDF5_PATH")]
|
||||||
path: PathBuf,
|
path: Option<PathBuf>,
|
||||||
|
|
||||||
|
/// File holding an Ed25519 signing key (64 hex characters, from
|
||||||
|
/// `keygen`). Every checkpoint this command makes is then signed; a
|
||||||
|
/// signed store refuses to checkpoint without it.
|
||||||
|
#[arg(long, env = "CLAWHDF5_SIGNING_KEY", global = true)]
|
||||||
|
signing_key: Option<PathBuf>,
|
||||||
|
|
||||||
#[command(subcommand)]
|
#[command(subcommand)]
|
||||||
command: Commands,
|
command: Commands,
|
||||||
@@ -91,6 +98,38 @@ enum Commands {
|
|||||||
/// Destination path
|
/// Destination path
|
||||||
dest: PathBuf,
|
dest: PathBuf,
|
||||||
},
|
},
|
||||||
|
/// Generate an Ed25519 signing key for signed checkpoints
|
||||||
|
Keygen {
|
||||||
|
/// Where to write the secret key (created new, owner-only on Unix)
|
||||||
|
#[arg(long)]
|
||||||
|
out: PathBuf,
|
||||||
|
},
|
||||||
|
/// Verify a signed store against a public key; exit status 2 if not valid
|
||||||
|
Verify {
|
||||||
|
/// The trusted public key: 64 hex characters, or a file holding them
|
||||||
|
#[arg(long)]
|
||||||
|
public_key: String,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_signing_key(path: &Path) -> Result<SigningKey, Box<dyn std::error::Error>> {
|
||||||
|
let text = std::fs::read_to_string(path)
|
||||||
|
.map_err(|e| format!("cannot read signing key {}: {e}", path.display()))?;
|
||||||
|
let bytes = signing::from_hex::<32>(&text)
|
||||||
|
.ok_or_else(|| format!("{} is not a 64-hex-character key", path.display()))?;
|
||||||
|
Ok(SigningKey::from_bytes(&bytes))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Open for writing, with the signing key applied if one was given.
|
||||||
|
fn open_writable(
|
||||||
|
path: &Path,
|
||||||
|
key: &Option<SigningKey>,
|
||||||
|
) -> Result<HDF5Memory, Box<dyn std::error::Error>> {
|
||||||
|
let mut mem = HDF5Memory::open(path)?;
|
||||||
|
if let Some(k) = key {
|
||||||
|
mem.set_signing_key(k.clone());
|
||||||
|
}
|
||||||
|
Ok(mem)
|
||||||
}
|
}
|
||||||
|
|
||||||
fn main() {
|
fn main() {
|
||||||
@@ -103,6 +142,37 @@ fn main() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||||
|
if let Commands::Keygen { out } = &cli.command {
|
||||||
|
let key = signing::generate_key();
|
||||||
|
let mut opts = std::fs::OpenOptions::new();
|
||||||
|
opts.write(true).create_new(true);
|
||||||
|
#[cfg(unix)]
|
||||||
|
{
|
||||||
|
use std::os::unix::fs::OpenOptionsExt;
|
||||||
|
opts.mode(0o600);
|
||||||
|
}
|
||||||
|
use std::io::Write;
|
||||||
|
let mut f = opts
|
||||||
|
.open(out)
|
||||||
|
.map_err(|e| format!("cannot create {}: {e}", out.display()))?;
|
||||||
|
writeln!(f, "{}", signing::to_hex(&key.to_bytes()))?;
|
||||||
|
let j = serde_json::json!({
|
||||||
|
"status": "generated",
|
||||||
|
"secret_key_file": out.display().to_string(),
|
||||||
|
"public_key": signing::to_hex(&key.verifying_key().to_bytes()),
|
||||||
|
});
|
||||||
|
println!("{}", serde_json::to_string_pretty(&j)?);
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
let path = cli
|
||||||
|
.path
|
||||||
|
.clone()
|
||||||
|
.ok_or("--path (or CLAWHDF5_PATH) is required")?;
|
||||||
|
let key = cli
|
||||||
|
.signing_key
|
||||||
|
.as_deref()
|
||||||
|
.map(read_signing_key)
|
||||||
|
.transpose()?;
|
||||||
match cli.command {
|
match cli.command {
|
||||||
Commands::Create {
|
Commands::Create {
|
||||||
agent_id,
|
agent_id,
|
||||||
@@ -113,7 +183,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
|||||||
f32,
|
f32,
|
||||||
float16: _,
|
float16: _,
|
||||||
} => {
|
} => {
|
||||||
let mut config = MemoryConfig::new(cli.path.clone(), &agent_id, dim);
|
let mut config = MemoryConfig::new(path.clone(), &agent_id, dim);
|
||||||
config.wal_enabled = wal;
|
config.wal_enabled = wal;
|
||||||
// As with --f32-index: only ever switch the library default off.
|
// As with --f32-index: only ever switch the library default off.
|
||||||
if f32 {
|
if f32 {
|
||||||
@@ -127,15 +197,21 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
|||||||
config.quantized_index = false;
|
config.quantized_index = false;
|
||||||
}
|
}
|
||||||
let config_quantized = config.quantized_index;
|
let config_quantized = config.quantized_index;
|
||||||
let mem = HDF5Memory::create(config)?;
|
let mut mem = HDF5Memory::create(config)?;
|
||||||
|
// Sign straight away, so the store is never on disk unsigned.
|
||||||
|
if let Some(k) = &key {
|
||||||
|
mem.set_signing_key(k.clone());
|
||||||
|
mem.flush_wal()?;
|
||||||
|
}
|
||||||
let j = serde_json::json!({
|
let j = serde_json::json!({
|
||||||
"status": "created",
|
"status": "created",
|
||||||
"path": cli.path.display().to_string(),
|
"path": path.display().to_string(),
|
||||||
"agent_id": agent_id,
|
"agent_id": agent_id,
|
||||||
"embedding_dim": dim,
|
"embedding_dim": dim,
|
||||||
"wal_enabled": wal,
|
"wal_enabled": wal,
|
||||||
"quantized_index": config_quantized,
|
"quantized_index": config_quantized,
|
||||||
"float16": config_float16,
|
"float16": config_float16,
|
||||||
|
"signed": mem.is_signed(),
|
||||||
"count": mem.count(),
|
"count": mem.count(),
|
||||||
});
|
});
|
||||||
println!("{}", serde_json::to_string_pretty(&j)?);
|
println!("{}", serde_json::to_string_pretty(&j)?);
|
||||||
@@ -152,7 +228,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
let entry: MemoryEntry = serde_json::from_str(&input)?;
|
let entry: MemoryEntry = serde_json::from_str(&input)?;
|
||||||
let mut mem = HDF5Memory::open(&cli.path)?;
|
let mut mem = open_writable(&path, &key)?;
|
||||||
let idx = mem.save(entry)?;
|
let idx = mem.save(entry)?;
|
||||||
let j = serde_json::json!({ "status": "saved", "index": idx, "count": mem.count() });
|
let j = serde_json::json!({ "status": "saved", "index": idx, "count": mem.count() });
|
||||||
println!("{}", serde_json::to_string(&j)?);
|
println!("{}", serde_json::to_string(&j)?);
|
||||||
@@ -166,7 +242,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
|||||||
keyword_weight,
|
keyword_weight,
|
||||||
} => {
|
} => {
|
||||||
let emb: Vec<f32> = serde_json::from_str(&embedding)?;
|
let emb: Vec<f32> = serde_json::from_str(&embedding)?;
|
||||||
let mut mem = HDF5Memory::open(&cli.path)?;
|
let mut mem = open_writable(&path, &key)?;
|
||||||
let results = mem.hybrid_search(&emb, &query, vector_weight, keyword_weight, top_k);
|
let results = mem.hybrid_search(&emb, &query, vector_weight, keyword_weight, top_k);
|
||||||
let j: Vec<serde_json::Value> = results
|
let j: Vec<serde_json::Value> = results
|
||||||
.iter()
|
.iter()
|
||||||
@@ -184,7 +260,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
Commands::Recall { index } => {
|
Commands::Recall { index } => {
|
||||||
let mem = HDF5Memory::open_read_only(&cli.path)?;
|
let mem = HDF5Memory::open_read_only(&path)?;
|
||||||
match mem.get_chunk(index) {
|
match mem.get_chunk(index) {
|
||||||
Some(content) => {
|
Some(content) => {
|
||||||
let j = serde_json::json!({ "index": index, "chunk": content });
|
let j = serde_json::json!({ "index": index, "chunk": content });
|
||||||
@@ -198,22 +274,23 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
Commands::Stats => {
|
Commands::Stats => {
|
||||||
let mem = HDF5Memory::open_read_only(&cli.path)?;
|
let mem = HDF5Memory::open_read_only(&path)?;
|
||||||
let cfg = mem.config();
|
let cfg = mem.config();
|
||||||
let j = serde_json::json!({
|
let j = serde_json::json!({
|
||||||
"path": cli.path.display().to_string(),
|
"path": path.display().to_string(),
|
||||||
"agent_id": cfg.agent_id,
|
"agent_id": cfg.agent_id,
|
||||||
"embedding_dim": cfg.embedding_dim,
|
"embedding_dim": cfg.embedding_dim,
|
||||||
"count": mem.count(),
|
"count": mem.count(),
|
||||||
"active": mem.count_active(),
|
"active": mem.count_active(),
|
||||||
"wal_enabled": cfg.wal_enabled,
|
"wal_enabled": cfg.wal_enabled,
|
||||||
"wal_pending": mem.wal_pending_count(),
|
"wal_pending": mem.wal_pending_count(),
|
||||||
|
"signed": mem.is_signed(),
|
||||||
});
|
});
|
||||||
println!("{}", serde_json::to_string_pretty(&j)?);
|
println!("{}", serde_json::to_string_pretty(&j)?);
|
||||||
}
|
}
|
||||||
|
|
||||||
Commands::FlushWal => {
|
Commands::FlushWal => {
|
||||||
let mut mem = HDF5Memory::open(&cli.path)?;
|
let mut mem = open_writable(&path, &key)?;
|
||||||
let before = mem.wal_pending_count();
|
let before = mem.wal_pending_count();
|
||||||
mem.flush_wal()?;
|
mem.flush_wal()?;
|
||||||
let j = serde_json::json!({
|
let j = serde_json::json!({
|
||||||
@@ -225,7 +302,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
Commands::AgentsMd { output } => {
|
Commands::AgentsMd { output } => {
|
||||||
let mem = HDF5Memory::open_read_only(&cli.path)?;
|
let mem = HDF5Memory::open_read_only(&path)?;
|
||||||
let md = mem.generate_agents_md();
|
let md = mem.generate_agents_md();
|
||||||
match output {
|
match output {
|
||||||
Some(p) => {
|
Some(p) => {
|
||||||
@@ -237,7 +314,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
Commands::Export => {
|
Commands::Export => {
|
||||||
let mem = HDF5Memory::open_read_only(&cli.path)?;
|
let mem = HDF5Memory::open_read_only(&path)?;
|
||||||
for i in 0..mem.count() {
|
for i in 0..mem.count() {
|
||||||
if let Some(chunk) = mem.get_chunk(i) {
|
if let Some(chunk) = mem.get_chunk(i) {
|
||||||
let j = serde_json::json!({ "index": i, "chunk": chunk });
|
let j = serde_json::json!({ "index": i, "chunk": chunk });
|
||||||
@@ -246,11 +323,44 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
Commands::Keygen { .. } => unreachable!("handled before opening a store"),
|
||||||
|
|
||||||
|
Commands::Verify { public_key } => {
|
||||||
|
let text = if Path::new(&public_key).is_file() {
|
||||||
|
std::fs::read_to_string(&public_key)?
|
||||||
|
} else {
|
||||||
|
public_key
|
||||||
|
};
|
||||||
|
let bytes = signing::from_hex::<32>(&text)
|
||||||
|
.ok_or("--public-key must be 64 hex characters or a file holding them")?;
|
||||||
|
let trusted = VerifyingKey::from_bytes(&bytes)?;
|
||||||
|
let r = HDF5Memory::verify(&path, &trusted)?;
|
||||||
|
let j = serde_json::json!({
|
||||||
|
"valid": r.is_valid(),
|
||||||
|
"signed": r.signed,
|
||||||
|
"key_matches": r.key_matches,
|
||||||
|
"signature_valid": r.signature_valid,
|
||||||
|
"records_match": r.records_match,
|
||||||
|
"settings_match": r.settings_match,
|
||||||
|
"sessions_match": r.sessions_match,
|
||||||
|
"graph_match": r.graph_match,
|
||||||
|
"changed_records": r.changed_records,
|
||||||
|
"record_count": r.record_count,
|
||||||
|
"signed_record_count": r.signed_record_count,
|
||||||
|
"signed_by": r.public_key.map(|k| signing::to_hex(&k)),
|
||||||
|
"wal_entries_unsigned": r.wal_entries_unsigned,
|
||||||
|
});
|
||||||
|
println!("{}", serde_json::to_string_pretty(&j)?);
|
||||||
|
if !r.is_valid() {
|
||||||
|
std::process::exit(2);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
Commands::Snapshot { dest } => {
|
Commands::Snapshot { dest } => {
|
||||||
let _result = clawhdf5_agent::storage::snapshot_file(&cli.path, &dest)?;
|
let _result = clawhdf5_agent::storage::snapshot_file(&path, &dest)?;
|
||||||
let j = serde_json::json!({
|
let j = serde_json::json!({
|
||||||
"status": "snapshot_created",
|
"status": "snapshot_created",
|
||||||
"source": cli.path.display().to_string(),
|
"source": path.display().to_string(),
|
||||||
"dest": dest.display().to_string(),
|
"dest": dest.display().to_string(),
|
||||||
});
|
});
|
||||||
println!("{}", serde_json::to_string(&j)?);
|
println!("{}", serde_json::to_string(&j)?);
|
||||||
|
|||||||
Reference in New Issue
Block a user