osobhandClaude Opus 5.5
c27a478e44
Numbers, API names, feature defaults and PR references checked against CONFORMANCE.md, BENCHMARKS.md, CHANGELOG.md, the code and git history. - int8 index figures (1.74x memory, 1.63x QPS) carry the dates git gives them (2026-09-19/20, machine not recorded, not re-run) instead of none; the Pi 5 1.18x carries 2026-09-21. - BENCHMARKS headline: the libhdf5 chunked-write figure is the newest measurement (35x, 2026-09-23), not 45.3x (2026-08-03). - Conformance counts follow the 2026-09-28 run (1 our-error, 2 ref-bug) in conformance/README.md, ROADMAP.md and CLAUDE.md, with a pointer to the bad_nbit_parms_walk.h5 flip. - README: LZ4 is opt-in; the browser refuses reference/opaque/bitfield/ time datasets too; zlib-rs byte-identity scoped to what was measured; macOS default links the system libz for inflate. - Crate READMEs: system-zlib-decompress does something (macOS), SweepDetector lives in prefetch, checkpoint after more than 500 WAL entries, NetCDF-4 unlimited-dimension size warning. - agent-memory.md: string-dataset compression threshold, agents-md prints Markdown, float16 file sizes linked to their study. - known-issues.md: contiguous selection reads, 1.21x vs h5py threads. - docs/README.md, USE_CASES.md, ROADMAP.md, CLAUDE.md: range-read milestones M0-M5 and PRs #17-#19, missing README rows, CLI keygen/verify, dated figures, fast-math is not BLAS. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
125 lines
6.0 KiB
Markdown
125 lines
6.0 KiB
Markdown
# clawhdf5-agent
|
|
|
|
Persistent memory for AI agents in a single HDF5 file: text chunks with
|
|
embeddings and metadata, hybrid search (HNSW vector search + BM25 keyword
|
|
search, fused), sessions, a knowledge graph, a write-ahead log for crash
|
|
safety, and optionally Ed25519-signed checkpoints. Stores open in h5py like
|
|
any other HDF5 file. Built on [`clawhdf5`](../clawhdf5/README.md),
|
|
[`clawhdf5-ann`](../clawhdf5-ann/README.md) and
|
|
[`clawhdf5-accel`](../clawhdf5-accel/README.md).
|
|
|
|
It is a library: no agent framework integrates it (OpenClaw and ZeroClaw
|
|
integration claims were withdrawn on 2026-09-25; see
|
|
[`docs/openclaw.md`](../../docs/openclaw.md)). The command-line front end
|
|
is [`clawhdf5-cli`](../clawhdf5-cli/README.md).
|
|
|
|
Not on crates.io yet; depend on it from git:
|
|
|
|
```toml
|
|
[dependencies]
|
|
clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
|
|
```
|
|
|
|
## Usage
|
|
|
|
```rust,no_run
|
|
use std::path::PathBuf;
|
|
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, SearchOptions};
|
|
|
|
let config = MemoryConfig::new(PathBuf::from("agent.h5"), "my-agent", 384);
|
|
let mut mem = HDF5Memory::create(config)?;
|
|
|
|
mem.save(MemoryEntry {
|
|
chunk: "The deploy key rotates every Monday.".into(),
|
|
embedding: vec![0.01; 384], // from your embedding model
|
|
source_channel: "chat".into(),
|
|
timestamp: 1_790_000_000.0,
|
|
session_id: "s1".into(),
|
|
tags: "ops".into(),
|
|
})?;
|
|
|
|
let query = vec![0.01f32; 384];
|
|
let hits = mem.search(&query, "deploy key", &SearchOptions::new(5).with_sources(["chat"]));
|
|
for h in &hits {
|
|
println!("{:.3} {}", h.score, h.chunk);
|
|
}
|
|
mem.flush_wal()?; // checkpoint now; otherwise one is made once the WAL holds more than 500 entries (wal_max_entries)
|
|
# Ok::<(), clawhdf5_agent::MemoryError>(())
|
|
```
|
|
|
|
## What is in it
|
|
|
|
- **`HDF5Memory`** — `create`, `open` (single writer: an exclusive lock on
|
|
`<store>.h5.lock`, a second opener gets `MemoryError::Locked`),
|
|
`open_read_only` (no lock, never writes). Through the `AgentMemory`
|
|
trait: `save`, `save_batch`, `delete`, `compact`, `count`, `snapshot`,
|
|
sessions; also `save_or_update`, `delete_batch`, `flush_wal`.
|
|
- **Search** — `search(query_embedding, text, &SearchOptions)`: optional
|
|
source-channel filter applied before ranking, vector + BM25 fusion
|
|
(weighted or RRF), Hebbian activation scaling, optional re-ranking
|
|
(`reranker::ReRankConfig`) and confidence rejection
|
|
(`confidence::ConfidenceConfig`). `hybrid_search` and
|
|
`hybrid_search_with` are thin wrappers. The vector stage uses the HNSW
|
|
index (`hnsw` feature); its graph is saved to `<store>.h5.ann` at each
|
|
checkpoint and reloaded on open (rebuilt if stale or damaged).
|
|
- **Storage settings** (`MemoryConfig`, persisted with the store):
|
|
`float16` embeddings (on by default for new stores; 48% smaller file at
|
|
100K records, same retrieval on LongMemEval), `quantized_index` (int8
|
|
copy of the vectors in the index, on by default; re-scored against the
|
|
exact embeddings), `compression` (off by default), HNSW `m`/`ef`
|
|
parameters, WAL settings (`wal_enabled`, on by default; `wal_max_entries`,
|
|
500: the WAL is checkpointed into the `.h5` once it holds more).
|
|
- **WAL** (`wal`) — every write is appended to `<store>.h5.wal` with a
|
|
chained CRC32 per entry, so a corrupted, reordered or spliced entry stops
|
|
replay. Recovers from a process crash at any point, including between a
|
|
checkpoint and the WAL truncate. WAL appends are not fsynced: saves since
|
|
the last checkpoint can be lost on power failure. An unreadable WAL is
|
|
quarantined to `<store>.h5.wal.corrupt-<ts>`.
|
|
- **Signed checkpoints** (`signing`) — `set_signing_key` signs a manifest
|
|
(SHA-256 Merkle tree over records, plus settings, sessions and graph) at
|
|
every checkpoint; `HDF5Memory::verify(path, &public_key)` checks it and
|
|
locates edits. WAL entries after the checkpoint are not covered.
|
|
- **Knowledge graph** (`knowledge`, `entity_extract`) — `add_entity`,
|
|
`add_entity_alias`, `add_relation`, `extract_and_store_entities`,
|
|
traversal and spreading activation.
|
|
- **Also:** sessions (`session`), temporal index (`temporal`),
|
|
consolidation tiers (`consolidation`), an in-memory TTL tier
|
|
(`ephemeral`), multi-modal embeddings (`multimodal`), `AGENTS.md`
|
|
generation (`agents_md`), query expansion, and a session-scoped
|
|
provenance ledger and write-anomaly detector on every save
|
|
(`take_anomaly_alerts`; alerts never block a save, and the source is
|
|
inferred from `source_channel`, not authenticated).
|
|
- `openclaw::ClawhdfBackend` is `search` with re-ranking and confidence
|
|
on, plus Markdown import/export. The module name is historical: it is not
|
|
an OpenClaw plugin.
|
|
|
|
## Features
|
|
|
|
| Feature | Default | What | Builds C |
|
|
|---|---|---|---|
|
|
| `hnsw` | yes | HNSW vector index (`clawhdf5-ann`); without it the vector stage is an exact linear cosine scan | no |
|
|
| `parallel` | yes | build the HNSW index on a rayon pool (same graph either way) | no |
|
|
| `float16` | yes | f16 helpers in `vector_search` (`half`). Stores' `MemoryConfig::float16` works without it. | no |
|
|
| `fast-math` | no | `matrixmultiply` batch distances in `strategy` | no |
|
|
| `accelerate` | no | Apple Accelerate BLAS in `strategy` (macOS) | links a system framework |
|
|
| `openblas` | no | OpenBLAS in `strategy` | yes (`openblas-src`) |
|
|
| `gpu` | no | `gpu_search` through [`clawhdf5-gpu`](../clawhdf5-gpu/README.md) (wgpu), used by `strategy`, not by `HDF5Memory::search` | no, but needs GPU drivers |
|
|
| `zstd` | no | Zstd instead of deflate when `MemoryConfig::compression` is on | yes (libzstd) |
|
|
| `async` | no | `async_memory` wrapper on tokio | no |
|
|
|
|
`--no-default-features --features float16` forces the exact linear scan.
|
|
|
|
## Measurements and limits
|
|
|
|
- Search recall and latency, file size, LongMemEval and MemoryArena
|
|
retrieval numbers: [`BENCHMARKS.md`](../../BENCHMARKS.md), measured with
|
|
the `clawhdf5-bench` binaries (`search_harness`, `longmemeval_bench`,
|
|
`footprint_bench`, ...).
|
|
- Known issues and their history: [`docs/known-issues.md`](../../docs/known-issues.md).
|
|
- Migrating a SQLite memory database:
|
|
[`clawhdf5-migrate`](../clawhdf5-migrate/README.md).
|
|
|
|
## License
|
|
|
|
MIT
|