# clawhdf5-agent

Persistent memory for AI agents in a single HDF5 file: text chunks with
embeddings and metadata, hybrid search (HNSW vector search + BM25 keyword
search, fused), sessions, a knowledge graph, a write-ahead log for crash
safety, and optionally Ed25519-signed checkpoints. Stores open in h5py like
any other HDF5 file. Built on [`clawhdf5`](../clawhdf5/README.md),
[`clawhdf5-ann`](../clawhdf5-ann/README.md) and
[`clawhdf5-accel`](../clawhdf5-accel/README.md).

It is a library: no agent framework integrates it (OpenClaw and ZeroClaw
integration claims were withdrawn on 2026-09-25; see
[`docs/openclaw.md`](../../docs/openclaw.md)). The command-line front end
is [`clawhdf5-cli`](../clawhdf5-cli/README.md).

Not on crates.io yet; depend on it from git:

```toml
[dependencies]
clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
```

## Usage

```rust,no_run
use std::path::PathBuf;
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, SearchOptions};

let config = MemoryConfig::new(PathBuf::from("agent.h5"), "my-agent", 384);
let mut mem = HDF5Memory::create(config)?;

mem.save(MemoryEntry {
    chunk: "The deploy key rotates every Monday.".into(),
    embedding: vec![0.01; 384], // from your embedding model
    source_channel: "chat".into(),
    timestamp: 1_790_000_000.0,
    session_id: "s1".into(),
    tags: "ops".into(),
})?;

let query = vec![0.01f32; 384];
let hits = mem.search(&query, "deploy key", &SearchOptions::new(5).with_sources(["chat"]));
for h in &hits {
    println!("{:.3} {}", h.score, h.chunk);
}
mem.flush_wal()?; // checkpoint now; otherwise one is made once the WAL holds more than 500 entries (wal_max_entries)
# Ok::<(), clawhdf5_agent::MemoryError>(())
```

## What is in it

- **`HDF5Memory`** — `create`, `open` (single writer: an exclusive lock on
  `<store>.h5.lock`, a second opener gets `MemoryError::Locked`),
  `open_read_only` (no lock, never writes). Through the `AgentMemory`
  trait: `save`, `save_batch`, `delete`, `compact`, `count`, `snapshot`,
  sessions; also `save_or_update`, `delete_batch`, `flush_wal`.
- **Search** — `search(query_embedding, text, &SearchOptions)`: optional
  source-channel filter applied before ranking, vector + BM25 fusion
  (weighted or RRF), Hebbian activation scaling, optional re-ranking
  (`reranker::ReRankConfig`) and confidence rejection
  (`confidence::ConfidenceConfig`). `hybrid_search` and
  `hybrid_search_with` are thin wrappers. The vector stage uses the HNSW
  index (`hnsw` feature); its graph is saved to `<store>.h5.ann` at each
  checkpoint and reloaded on open (rebuilt if stale or damaged).
- **Storage settings** (`MemoryConfig`, persisted with the store):
  `float16` embeddings (on by default for new stores; 48% smaller file at
  100K records, same retrieval on LongMemEval), `quantized_index` (int8
  copy of the vectors in the index, on by default; re-scored against the
  exact embeddings), `compression` (off by default), HNSW `m`/`ef`
  parameters, WAL settings (`wal_enabled`, on by default; `wal_max_entries`,
  500: the WAL is checkpointed into the `.h5` once it holds more).
- **WAL** (`wal`) — every write is appended to `<store>.h5.wal` with a
  chained CRC32 per entry, so a corrupted, reordered or spliced entry stops
  replay. Recovers from a process crash at any point, including between a
  checkpoint and the WAL truncate. WAL appends are not fsynced: saves since
  the last checkpoint can be lost on power failure. An unreadable WAL is
  quarantined to `<store>.h5.wal.corrupt-<ts>`.
- **Signed checkpoints** (`signing`) — `set_signing_key` signs a manifest
  (SHA-256 Merkle tree over records, plus settings, sessions and graph) at
  every checkpoint; `HDF5Memory::verify(path, &public_key)` checks it and
  locates edits. WAL entries after the checkpoint are not covered.
- **Knowledge graph** (`knowledge`, `entity_extract`) — `add_entity`,
  `add_entity_alias`, `add_relation`, `extract_and_store_entities`,
  traversal and spreading activation.
- **Also:** sessions (`session`), temporal index (`temporal`),
  consolidation tiers (`consolidation`), an in-memory TTL tier
  (`ephemeral`), multi-modal embeddings (`multimodal`), `AGENTS.md`
  generation (`agents_md`), query expansion, and a session-scoped
  provenance ledger and write-anomaly detector on every save
  (`take_anomaly_alerts`; alerts never block a save, and the source is
  inferred from `source_channel`, not authenticated).
- `openclaw::ClawhdfBackend` is `search` with re-ranking and confidence
  on, plus Markdown import/export. The module name is historical: it is not
  an OpenClaw plugin.

## Features

| Feature | Default | What | Builds C |
|---|---|---|---|
| `hnsw` | yes | HNSW vector index (`clawhdf5-ann`); without it the vector stage is an exact linear cosine scan | no |
| `parallel` | yes | build the HNSW index on a rayon pool (same graph either way) | no |
| `float16` | yes | f16 helpers in `vector_search` (`half`). Stores' `MemoryConfig::float16` works without it. | no |
| `fast-math` | no | `matrixmultiply` batch distances in `strategy` | no |
| `accelerate` | no | Apple Accelerate BLAS in `strategy` (macOS) | links a system framework |
| `openblas` | no | OpenBLAS in `strategy` | yes (`openblas-src`) |
| `gpu` | no | `gpu_search` through [`clawhdf5-gpu`](../clawhdf5-gpu/README.md) (wgpu), used by `strategy`, not by `HDF5Memory::search` | no, but needs GPU drivers |
| `zstd` | no | Zstd instead of deflate when `MemoryConfig::compression` is on | yes (libzstd) |
| `async` | no | `async_memory` wrapper on tokio | no |

`--no-default-features --features float16` forces the exact linear scan.

## Measurements and limits

- Search recall and latency, file size, LongMemEval and MemoryArena
  retrieval numbers: [`BENCHMARKS.md`](../../BENCHMARKS.md), measured with
  the `clawhdf5-bench` binaries (`search_harness`, `longmemeval_bench`,
  `footprint_bench`, ...).
- Known issues and their history: [`docs/known-issues.md`](../../docs/known-issues.md).
- Migrating a SQLite memory database:
  [`clawhdf5-migrate`](../clawhdf5-migrate/README.md).

## License

MIT