quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
3c557c9f0a73c50e6afa91f2236e026ff53a42e0
clawhdf5/crates/clawhdf5-agent/README.md
T
osobhandClaude Opus 5.5 b55b24b7ba docs: crate READMEs describe each crate as it is today 
Every crate under crates/ now has a README (android, bench, cli, napi and
wasm had none), each saying what the crate is, its main types and
functions (names checked against the code), its cargo features with
defaults and which ones build C (checked with `cargo tree`), and links to
the top-level docs.

Corrections to the old stubs:
- clawhdf5-derive: the derive is `H5Type`, not `HDF5Type`, and it needs
  clawhdf5-format as a dependency.
- clawhdf5-filters: deflate backends only, and no library crate depends
  on it; the filter pipeline and every other codec are in -format.
- clawhdf5-gpu: vector distance compute, not I/O; not used by
  HDF5Memory::search.
- clawhdf5-io: MpiVol is root-read + broadcast, not collective MPI-IO.
- clawhdf5-ann: from_hdf5/search(q, k) did not exist; load_from_hdf5 and
  search(q, k, ef).
- clawhdf5-accel: checksum::crc32_simd did not exist; the SSE4 and wasm
  backends are reported but run the scalar kernels.
- clawhdf5-gpu: the old example called l2_distances, which does not
  exist (l2_search).
- clawhdf5-agent: it described a "vector store" with "GPU acceleration";
  it now covers HDF5Memory, search options, WAL, signing, the graph.
- crates.io/docs.rs badges removed and `cargo install <crate>` replaced:
  nothing is published; depend on git.
- fuzz: the opt-in CLAWHDF5_FUZZ_SECONDS smoke run in ci-test.sh.
- tools: the FileEditor interop tests that live in this crate.
- remote, py: license, other front ends, limits, File.mode/flush/chunks.

The Rust examples of the facade, format, filters, accel, ann, derive and
agent READMEs were compiled and run as tests (netcdf4, gpu and remote
compiled only) in a scratch crate; the CLI example was run.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-28 11:13:30 -05:00

125 lines
6.0 KiB
Markdown
Raw Blame History

# clawhdf5-agent
Persistent memory for AI agents in a single HDF5 file: text chunks with
embeddings and metadata, hybrid search (HNSW vector search + BM25 keyword
search, fused), sessions, a knowledge graph, a write-ahead log for crash
safety, and optionally Ed25519-signed checkpoints. Stores open in h5py like
any other HDF5 file. Built on [`clawhdf5`](../clawhdf5/README.md),
[`clawhdf5-ann`](../clawhdf5-ann/README.md) and
[`clawhdf5-accel`](../clawhdf5-accel/README.md).
It is a library: no agent framework integrates it (OpenClaw and ZeroClaw
integration claims were withdrawn on 2026-09-25; see
[`docs/openclaw.md`](../../docs/openclaw.md)). The command-line front end
is [`clawhdf5-cli`](../clawhdf5-cli/README.md).
Not on crates.io yet; depend on it from git:
```toml
[dependencies]
clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
```
## Usage
```rust,no_run
use std::path::PathBuf;
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, SearchOptions};
let config = MemoryConfig::new(PathBuf::from("agent.h5"), "my-agent", 384);
let mut mem = HDF5Memory::create(config)?;
mem.save(MemoryEntry {
chunk: "The deploy key rotates every Monday.".into(),
embedding: vec![0.01; 384], // from your embedding model
source_channel: "chat".into(),
timestamp: 1_790_000_000.0,
session_id: "s1".into(),
tags: "ops".into(),
})?;
let query = vec![0.01f32; 384];
let hits = mem.search(&query, "deploy key", &SearchOptions::new(5).with_sources(["chat"]));
for h in &hits {
println!("{:.3} {}", h.score, h.chunk);
}
mem.flush_wal()?; // checkpoint now; otherwise one is made after 500 WAL entries (wal_max_entries)
# Ok::<(), clawhdf5_agent::MemoryError>(())
```
## What is in it
- **`HDF5Memory`** — `create`, `open` (single writer: an exclusive lock on
`<store>.h5.lock`, a second opener gets `MemoryError::Locked`),
`open_read_only` (no lock, never writes). Through the `AgentMemory`
trait: `save`, `save_batch`, `delete`, `compact`, `count`, `snapshot`,
sessions; also `save_or_update`, `delete_batch`, `flush_wal`.
- **Search** — `search(query_embedding, text, &SearchOptions)`: optional
source-channel filter applied before ranking, vector + BM25 fusion
(weighted or RRF), Hebbian activation scaling, optional re-ranking
(`reranker::ReRankConfig`) and confidence rejection
(`confidence::ConfidenceConfig`). `hybrid_search` and
`hybrid_search_with` are thin wrappers. The vector stage uses the HNSW
index (`hnsw` feature); its graph is saved to `<store>.h5.ann` at each
checkpoint and reloaded on open (rebuilt if stale or damaged).
- **Storage settings** (`MemoryConfig`, persisted with the store):
`float16` embeddings (on by default for new stores; 48% smaller file at
100K records, same retrieval on LongMemEval), `quantized_index` (int8
copy of the vectors in the index, on by default; re-scored against the
exact embeddings), `compression` (off by default), HNSW `m`/`ef`
parameters, WAL settings (`wal_enabled`, on by default; `wal_max_entries`,
500: the WAL is checkpointed into the `.h5` once it holds more).
- **WAL** (`wal`) — every write is appended to `<store>.h5.wal` with a
chained CRC32 per entry, so a corrupted, reordered or spliced entry stops
replay. Recovers from a process crash at any point, including between a
checkpoint and the WAL truncate. WAL appends are not fsynced: saves since
the last checkpoint can be lost on power failure. An unreadable WAL is
quarantined to `<store>.h5.wal.corrupt-<ts>`.
- **Signed checkpoints** (`signing`) — `set_signing_key` signs a manifest
(SHA-256 Merkle tree over records, plus settings, sessions and graph) at
every checkpoint; `HDF5Memory::verify(path, &public_key)` checks it and
locates edits. WAL entries after the checkpoint are not covered.
- **Knowledge graph** (`knowledge`, `entity_extract`) — `add_entity`,
`add_entity_alias`, `add_relation`, `extract_and_store_entities`,
traversal and spreading activation.
- **Also:** sessions (`session`), temporal index (`temporal`),
consolidation tiers (`consolidation`), an in-memory TTL tier
(`ephemeral`), multi-modal embeddings (`multimodal`), `AGENTS.md`
generation (`agents_md`), query expansion, and a session-scoped
provenance ledger and write-anomaly detector on every save
(`take_anomaly_alerts`; alerts never block a save, and the source is
inferred from `source_channel`, not authenticated).
- `openclaw::ClawhdfBackend` is `search` with re-ranking and confidence
on, plus Markdown import/export. The module name is historical: it is not
an OpenClaw plugin.
## Features
| Feature | Default | What | Builds C |
|---|---|---|---|
| `hnsw` | yes | HNSW vector index (`clawhdf5-ann`); without it the vector stage is an exact linear cosine scan | no |
| `parallel` | yes | build the HNSW index on a rayon pool (same graph either way) | no |
| `float16` | yes | f16 helpers in `vector_search` (`half`). Stores' `MemoryConfig::float16` works without it. | no |
| `fast-math` | no | `matrixmultiply` batch distances in `strategy` | no |
| `accelerate` | no | Apple Accelerate BLAS in `strategy` (macOS) | links a system framework |
| `openblas` | no | OpenBLAS in `strategy` | yes (`openblas-src`) |
| `gpu` | no | `gpu_search` through [`clawhdf5-gpu`](../clawhdf5-gpu/README.md) (wgpu), used by `strategy`, not by `HDF5Memory::search` | no, but needs GPU drivers |
| `zstd` | no | Zstd instead of deflate when `MemoryConfig::compression` is on | yes (libzstd) |
| `async` | no | `async_memory` wrapper on tokio | no |
`--no-default-features --features float16` forces the exact linear scan.
## Measurements and limits
- Search recall and latency, file size, LongMemEval and MemoryArena
retrieval numbers: [`BENCHMARKS.md`](../../BENCHMARKS.md), measured with
the `clawhdf5-bench` binaries (`search_harness`, `longmemeval_bench`,
`footprint_bench`, ...).
- Known issues and their history: [`docs/known-issues.md`](../../docs/known-issues.md).
- Migrating a SQLite memory database:
[`clawhdf5-migrate`](../clawhdf5-migrate/README.md).
## License
MIT
Reference in New Issue View Git Blame Copy Permalink