quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

Ed25519-signed checkpoints; remove the no-op agent feature #8

Merged
osobh merged 2 commits from feat/signed-checkpoints into main 2026-09-25 15:58:19 +00:00
Conversation 0 Commits 2 Files Changed 15
+1334 -29
osobh commented 2026-09-25 15:13:53 +00:00
Owner
Copy Link
Copy Source

Summary

Two items from the README claims audit.

Ed25519-signed checkpoints (db9af79)

This makes the README's "cryptographically verifiable memory" true.

  • Signing: with HDF5Memory::set_signing_key(key), every checkpoint stores a signed manifest. It holds a SHA-256 per record in a Merkle tree, plus hashes of the settings (including the WAL position), sessions and knowledge graph.
  • Verifying: HDF5Memory::verify(path, &public_key) recomputes everything from the file. It reports which part changed and which records (changed_records), and a forged manifest fails the signature.

Decisions, as agreed:

  • The key is set on the open store and never persisted.
  • A signed store refuses to checkpoint without its key (MemoryError::SigningKeyRequired). remove_signature() is the deliberate way back to unsigned.
  • Checkpoints only: saves still in the WAL aren't covered, and verify reports how many there are.

CLI: keygen, --signing-key <file> (or CLAWHDF5_SIGNING_KEY), and verify --public-key, which prints a JSON report and exits 2 if the store isn't valid.

tank, 3 runs 1K records 10K 100K
Checkpoint, unsigned → signed 5.4 → 6.4 ms 46 → 55 ms 495 → 598 ms
verify 2.1 ms 18.6 ms 247 ms
File size added (32 bytes per record) 0.03 MiB 0.31 MiB 3.05 MiB

New dependencies ed25519-dalek, sha2 and rand_core are pure Rust; the no-C check still passes.

Breaking: MemoryError is now #[non_exhaustive] and has a new SigningKeyRequired variant, so a match on it needs a wildcard arm.

Remove the no-op agent feature (4ecac65)

It enabled nothing, yet the README, QUICKSTART and USE_CASES told people to pass it. Those snippets are fixed too: they now use git dependencies (the old version = "2.0" never resolved, since nothing is on crates.io). USE_CASES also no longer presents the float16 feature as half-precision storage.

Breaking: drop agent from features = [...].

Test plan

  • scripts/ci-test.sh locally: 14/14, including the no-C check and MSRV 1.92; Python bindings compile
  • tests/signed_store.rs, 7 tests:
    • reopen and checkpoint cycles with awkward strings, for f16 and f32
    • refusal without the key; wrong and rotated keys
    • eight kinds of edit, each detected and located
    • a forged manifest; unsigned stores; NULs in text
  • h5py edits one timestamp in place, and verify names exactly that record
  • CLI end to end: keygen, signed create/save, verify exits 0; refusal without the key exits 1; wrong key exits 2
  • README snippet compiles and runs
  • Gitea CI on this PR

🤖 Generated with Claude Code

## Summary Two items from the README claims audit. ### Ed25519-signed checkpoints (`db9af79`) This makes the README's "cryptographically verifiable memory" true. - **Signing:** with `HDF5Memory::set_signing_key(key)`, every checkpoint stores a signed manifest. It holds a SHA-256 per record in a Merkle tree, plus hashes of the settings (including the WAL position), sessions and knowledge graph. - **Verifying:** `HDF5Memory::verify(path, &public_key)` recomputes everything from the file. It reports which part changed and which records (`changed_records`), and a forged manifest fails the signature. Decisions, as agreed: - The key is set on the open store and **never persisted**. - A signed store **refuses to checkpoint without its key** (`MemoryError::SigningKeyRequired`). `remove_signature()` is the deliberate way back to unsigned. - **Checkpoints only:** saves still in the WAL aren't covered, and `verify` reports how many there are. CLI: `keygen`, `--signing-key <file>` (or `CLAWHDF5_SIGNING_KEY`), and `verify --public-key`, which prints a JSON report and exits 2 if the store isn't valid. | tank, 3 runs | 1K records | 10K | 100K | |---|---:|---:|---:| | Checkpoint, unsigned → signed | 5.4 → 6.4 ms | 46 → 55 ms | 495 → 598 ms | | `verify` | 2.1 ms | 18.6 ms | 247 ms | | File size added (32 bytes per record) | 0.03 MiB | 0.31 MiB | 3.05 MiB | New dependencies `ed25519-dalek`, `sha2` and `rand_core` are pure Rust; the no-C check still passes. **Breaking:** `MemoryError` is now `#[non_exhaustive]` and has a new `SigningKeyRequired` variant, so a `match` on it needs a wildcard arm. ### Remove the no-op `agent` feature (`4ecac65`) It enabled nothing, yet the README, QUICKSTART and USE_CASES told people to pass it. Those snippets are fixed too: they now use git dependencies (the old `version = "2.0"` never resolved, since nothing is on crates.io). USE_CASES also no longer presents the `float16` feature as half-precision storage. **Breaking:** drop `agent` from `features = [...]`. ## Test plan - [x] `scripts/ci-test.sh` locally: 14/14, including the no-C check and MSRV 1.92; Python bindings compile - [x] `tests/signed_store.rs`, 7 tests: - reopen and checkpoint cycles with awkward strings, for `f16` and `f32` - refusal without the key; wrong and rotated keys - eight kinds of edit, each detected and located - a forged manifest; unsigned stores; NULs in text - [x] h5py edits one timestamp in place, and `verify` names exactly that record - [x] CLI end to end: `keygen`, signed `create`/`save`, `verify` exits 0; refusal without the key exits 1; wrong key exits 2 - [x] README snippet compiles and runs - [ ] Gitea CI on this PR 🤖 Generated with [Claude Code](https://claude.com/claude-code)
osobh added 2 commits 2026-09-25 15:13:54 +00:00
chore(agent)!: remove the no-op agent feature 4ecac65f22 
It enabled nothing — the agent layer is always built — yet the README,
QUICKSTART and USE_CASES told people to pass it. Removed, with those
snippets fixed: they now depend on the git repository (nothing is on
crates.io, so `version = "2.0"` never resolved) and USE_CASES no longer
presents the `float16` feature as half-precision storage (that is
MemoryConfig::float16, on by default for new stores).

Breaking for anyone passing `features = ["agent"]`: drop it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
feat(agent): Ed25519-signed checkpoints
CI / test-arm64 (pull_request) Successful in 1m5s
Details
CI / test (pull_request) Successful in 4m50s
Details
db9af7972c 
Makes the README's "cryptographically verifiable memory" true.

With HDF5Memory::set_signing_key(key), every checkpoint stores a signed
manifest of the store: a SHA-256 per memory record (text, embedding as
stored, channel, timestamp, session, tags, deleted flag, activation) in
a Merkle tree, plus hashes of the settings (and WAL mark), sessions and
knowledge graph. The signature, public key and manifest hashes go in
/meta; the per-record hashes in /integrity/record_hashes, so
HDF5Memory::verify(path, &public_key) can say which records changed, not
just that something did. A forged manifest fails the signature.

Decisions, as agreed:
- the key is set on the open store and never persisted;
- a signed store refuses to checkpoint without its key
  (MemoryError::SigningKeyRequired); remove_signature() is the
  deliberate way back to unsigned;
- checkpoints only: saves still in the WAL are not covered, and verify
  reports how many there are.

The hashes cover exactly what the file persists, in the form the loader
returns it (strings lose trailing NULs; an empty WAL mark is not
written), so untouched stores verify across any number of reopen and
checkpoint cycles. MemoryError becomes #[non_exhaustive] (it already
gains variants in this unreleased version).

CLI: keygen (owner-only key file), --signing-key / CLAWHDF5_SIGNING_KEY
on writing commands (create signs immediately), verify --public-key
(JSON; exit 2 if not valid), `signed` in create/stats output.

Tests: reopen/checkpoint cycles with awkward strings (f16 and f32),
refusal without the key, wrong and rotated keys, eight kinds of edit
each detected and located, a forged manifest, unsigned stores, NULs in
text, and an edit made in place with h5py that verify pinpoints.

Cost on tank (search_harness --signing-study --full, 3 runs): ~20% of a
checkpoint (+9 ms at 10K, +89-112 ms at 100K), verify 18.6 ms / 247 ms,
32 bytes per record in the file. New deps ed25519-dalek, sha2,
rand_core: pure Rust, the no-C check passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
osobh merged commit bdadf3447c into main 2026-09-25 15:58:19 +00:00
osobh deleted branch feat/signed-checkpoints 2026-09-25 15:58:53 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
nsobh osobh scottb (Scott B.)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: quantumclaw/clawhdf5#8