clawhdf5/crates/clawhdf5-agent/README.md

clawhdf5-agent

Persistent memory for AI agents in a single HDF5 file: text chunks with embeddings and metadata, hybrid search (HNSW vector search + BM25 keyword search, fused), sessions, a knowledge graph, a write-ahead log for crash safety, and optionally Ed25519-signed checkpoints. Stores open in h5py like any other HDF5 file. Built on clawhdf5, clawhdf5-ann and clawhdf5-accel.

It is a library: no agent framework integrates it (OpenClaw and ZeroClaw integration claims were withdrawn on 2026-09-25; see docs/openclaw.md). The command-line front end is clawhdf5-cli.

Not on crates.io yet; depend on it from git:

[dependencies]
clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }

Usage

use std::path::PathBuf;
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, SearchOptions};

let config = MemoryConfig::new(PathBuf::from("agent.h5"), "my-agent", 384);
let mut mem = HDF5Memory::create(config)?;

mem.save(MemoryEntry {
    chunk: "The deploy key rotates every Monday.".into(),
    embedding: vec![0.01; 384], // from your embedding model
    source_channel: "chat".into(),
    timestamp: 1_790_000_000.0,
    session_id: "s1".into(),
    tags: "ops".into(),
})?;

let query = vec![0.01f32; 384];
let hits = mem.search(&query, "deploy key", &SearchOptions::new(5).with_sources(["chat"]));
for h in &hits {
    println!("{:.3} {}", h.score, h.chunk);
}
mem.flush_wal()?; // checkpoint now; otherwise one is made once the WAL holds more than 500 entries (wal_max_entries)
# Ok::<(), clawhdf5_agent::MemoryError>(())

What is in it

• HDF5Memory — create, open (single writer: an exclusive lock on <store>.h5.lock, a second opener gets MemoryError::Locked), open_read_only (no lock, never writes). Through the AgentMemory trait: save, save_batch, delete, compact, count, snapshot, sessions; also save_or_update, delete_batch, flush_wal.
• Search — search(query_embedding, text, &SearchOptions): optional source-channel filter applied before ranking, vector + BM25 fusion (weighted or RRF), Hebbian activation scaling, optional re-ranking (reranker::ReRankConfig) and confidence rejection (confidence::ConfidenceConfig). hybrid_search and hybrid_search_with are thin wrappers. The vector stage uses the HNSW index (hnsw feature); its graph is saved to <store>.h5.ann at each checkpoint and reloaded on open (rebuilt if stale or damaged).
• Storage settings (MemoryConfig, persisted with the store): float16 embeddings (on by default for new stores; 48% smaller file at 100K records, same retrieval on LongMemEval), quantized_index (int8 copy of the vectors in the index, on by default; re-scored against the exact embeddings), compression (off by default), HNSW m/ef parameters, WAL settings (wal_enabled, on by default; wal_max_entries, 500: the WAL is checkpointed into the .h5 once it holds more).
• WAL (wal) — every write is appended to <store>.h5.wal with a chained CRC32 per entry, so a corrupted, reordered or spliced entry stops replay. Recovers from a process crash at any point, including between a checkpoint and the WAL truncate. WAL appends are not fsynced: saves since the last checkpoint can be lost on power failure. An unreadable WAL is quarantined to <store>.h5.wal.corrupt-<ts>.
• Signed checkpoints (signing) — set_signing_key signs a manifest (SHA-256 Merkle tree over records, plus settings, sessions and graph) at every checkpoint; HDF5Memory::verify(path, &public_key) checks it and locates edits. WAL entries after the checkpoint are not covered.
• Knowledge graph (knowledge, entity_extract) — add_entity, add_entity_alias, add_relation, extract_and_store_entities, traversal and spreading activation.
• Also: sessions (session), temporal index (temporal), consolidation tiers (consolidation), an in-memory TTL tier (ephemeral), multi-modal embeddings (multimodal), AGENTS.md generation (agents_md), query expansion, and a session-scoped provenance ledger and write-anomaly detector on every save (take_anomaly_alerts; alerts never block a save, and the source is inferred from source_channel, not authenticated).
• openclaw::ClawhdfBackend is search with re-ranking and confidence on, plus Markdown import/export. The module name is historical: it is not an OpenClaw plugin.

Features

Feature	Default	What	Builds C
hnsw	yes	HNSW vector index (clawhdf5-ann); without it the vector stage is an exact linear cosine scan	no
parallel	yes	build the HNSW index on a rayon pool (same graph either way)	no
float16	yes	f16 helpers in vector_search (half). Stores' MemoryConfig::float16 works without it.	no
fast-math	no	matrixmultiply batch distances in strategy	no
accelerate	no	Apple Accelerate BLAS in strategy (macOS)	links a system framework
openblas	no	OpenBLAS in strategy	yes (openblas-src)
gpu	no	gpu_search through clawhdf5-gpu (wgpu), used by strategy, not by HDF5Memory::search	no, but needs GPU drivers
zstd	no	Zstd instead of deflate when MemoryConfig::compression is on	yes (libzstd)
async	no	async_memory wrapper on tokio	no

--no-default-features --features float16 forces the exact linear scan.

Measurements and limits

• Search recall and latency, file size, LongMemEval and MemoryArena retrieval numbers: BENCHMARKS.md, measured with the clawhdf5-bench binaries (search_harness, longmemeval_bench, footprint_bench, ...).
• Known issues and their history: docs/known-issues.md.
• Migrating a SQLite memory database: clawhdf5-migrate.

License

MIT