docs: crate READMEs describe each crate as it is today
Every crate under crates/ now has a README (android, bench, cli, napi and wasm had none), each saying what the crate is, its main types and functions (names checked against the code), its cargo features with defaults and which ones build C (checked with `cargo tree`), and links to the top-level docs. Corrections to the old stubs: - clawhdf5-derive: the derive is `H5Type`, not `HDF5Type`, and it needs clawhdf5-format as a dependency. - clawhdf5-filters: deflate backends only, and no library crate depends on it; the filter pipeline and every other codec are in -format. - clawhdf5-gpu: vector distance compute, not I/O; not used by HDF5Memory::search. - clawhdf5-io: MpiVol is root-read + broadcast, not collective MPI-IO. - clawhdf5-ann: from_hdf5/search(q, k) did not exist; load_from_hdf5 and search(q, k, ef). - clawhdf5-accel: checksum::crc32_simd did not exist; the SSE4 and wasm backends are reported but run the scalar kernels. - clawhdf5-gpu: the old example called l2_distances, which does not exist (l2_search). - clawhdf5-agent: it described a "vector store" with "GPU acceleration"; it now covers HDF5Memory, search options, WAL, signing, the graph. - crates.io/docs.rs badges removed and `cargo install <crate>` replaced: nothing is published; depend on git. - fuzz: the opt-in CLAWHDF5_FUZZ_SECONDS smoke run in ci-test.sh. - tools: the FileEditor interop tests that live in this crate. - remote, py: license, other front ends, limits, File.mode/flush/chunks. The Rust examples of the facade, format, filters, accel, ann, derive and agent READMEs were compiled and run as tests (netcdf4, gpu and remote compiled only) in a scratch crate; the CLI example was run. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
+112
-16
@@ -1,28 +1,124 @@
|
||||
# clawhdf5-agent
|
||||
|
||||
[](https://crates.io/crates/clawhdf5-agent)
|
||||
[](https://docs.rs/clawhdf5-agent)
|
||||
Persistent memory for AI agents in a single HDF5 file: text chunks with
|
||||
embeddings and metadata, hybrid search (HNSW vector search + BM25 keyword
|
||||
search, fused), sessions, a knowledge graph, a write-ahead log for crash
|
||||
safety, and optionally Ed25519-signed checkpoints. Stores open in h5py like
|
||||
any other HDF5 file. Built on [`clawhdf5`](../clawhdf5/README.md),
|
||||
[`clawhdf5-ann`](../clawhdf5-ann/README.md) and
|
||||
[`clawhdf5-accel`](../clawhdf5-accel/README.md).
|
||||
|
||||
HDF5-backed persistent memory store for on-device AI agents.
|
||||
It is a library: no agent framework integrates it (OpenClaw and ZeroClaw
|
||||
integration claims were withdrawn on 2026-09-25; see
|
||||
[`docs/openclaw.md`](../../docs/openclaw.md)). The command-line front end
|
||||
is [`clawhdf5-cli`](../clawhdf5-cli/README.md).
|
||||
|
||||
Built on [clawhdf5](https://crates.io/crates/clawhdf5), clawhdf5-agent provides a vector-searchable memory backend optimized for edge AI workloads. Store embeddings, text chunks, and metadata in a single HDF5 file with SIMD-accelerated similarity search.
|
||||
|
||||
## Features
|
||||
|
||||
- Persistent vector store in HDF5 format
|
||||
- Cosine similarity and L2 distance search
|
||||
- SIMD-accelerated via clawhdf5-accel (AVX2, NEON)
|
||||
- Optional GPU acceleration via clawhdf5-gpu
|
||||
- Memory-mapped access for large stores
|
||||
- f16 storage support for compact embeddings
|
||||
|
||||
## Usage
|
||||
Not on crates.io yet; depend on it from git:
|
||||
|
||||
```toml
|
||||
[dependencies]
|
||||
clawhdf5-agent = "2.1.0"
|
||||
clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```rust,no_run
|
||||
use std::path::PathBuf;
|
||||
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, SearchOptions};
|
||||
|
||||
let config = MemoryConfig::new(PathBuf::from("agent.h5"), "my-agent", 384);
|
||||
let mut mem = HDF5Memory::create(config)?;
|
||||
|
||||
mem.save(MemoryEntry {
|
||||
chunk: "The deploy key rotates every Monday.".into(),
|
||||
embedding: vec![0.01; 384], // from your embedding model
|
||||
source_channel: "chat".into(),
|
||||
timestamp: 1_790_000_000.0,
|
||||
session_id: "s1".into(),
|
||||
tags: "ops".into(),
|
||||
})?;
|
||||
|
||||
let query = vec![0.01f32; 384];
|
||||
let hits = mem.search(&query, "deploy key", &SearchOptions::new(5).with_sources(["chat"]));
|
||||
for h in &hits {
|
||||
println!("{:.3} {}", h.score, h.chunk);
|
||||
}
|
||||
mem.flush_wal()?; // checkpoint now; otherwise one is made after 500 WAL entries (wal_max_entries)
|
||||
# Ok::<(), clawhdf5_agent::MemoryError>(())
|
||||
```
|
||||
|
||||
## What is in it
|
||||
|
||||
- **`HDF5Memory`** — `create`, `open` (single writer: an exclusive lock on
|
||||
`<store>.h5.lock`, a second opener gets `MemoryError::Locked`),
|
||||
`open_read_only` (no lock, never writes). Through the `AgentMemory`
|
||||
trait: `save`, `save_batch`, `delete`, `compact`, `count`, `snapshot`,
|
||||
sessions; also `save_or_update`, `delete_batch`, `flush_wal`.
|
||||
- **Search** — `search(query_embedding, text, &SearchOptions)`: optional
|
||||
source-channel filter applied before ranking, vector + BM25 fusion
|
||||
(weighted or RRF), Hebbian activation scaling, optional re-ranking
|
||||
(`reranker::ReRankConfig`) and confidence rejection
|
||||
(`confidence::ConfidenceConfig`). `hybrid_search` and
|
||||
`hybrid_search_with` are thin wrappers. The vector stage uses the HNSW
|
||||
index (`hnsw` feature); its graph is saved to `<store>.h5.ann` at each
|
||||
checkpoint and reloaded on open (rebuilt if stale or damaged).
|
||||
- **Storage settings** (`MemoryConfig`, persisted with the store):
|
||||
`float16` embeddings (on by default for new stores; 48% smaller file at
|
||||
100K records, same retrieval on LongMemEval), `quantized_index` (int8
|
||||
copy of the vectors in the index, on by default; re-scored against the
|
||||
exact embeddings), `compression` (off by default), HNSW `m`/`ef`
|
||||
parameters, WAL settings (`wal_enabled`, on by default; `wal_max_entries`,
|
||||
500: the WAL is checkpointed into the `.h5` once it holds more).
|
||||
- **WAL** (`wal`) — every write is appended to `<store>.h5.wal` with a
|
||||
chained CRC32 per entry, so a corrupted, reordered or spliced entry stops
|
||||
replay. Recovers from a process crash at any point, including between a
|
||||
checkpoint and the WAL truncate. WAL appends are not fsynced: saves since
|
||||
the last checkpoint can be lost on power failure. An unreadable WAL is
|
||||
quarantined to `<store>.h5.wal.corrupt-<ts>`.
|
||||
- **Signed checkpoints** (`signing`) — `set_signing_key` signs a manifest
|
||||
(SHA-256 Merkle tree over records, plus settings, sessions and graph) at
|
||||
every checkpoint; `HDF5Memory::verify(path, &public_key)` checks it and
|
||||
locates edits. WAL entries after the checkpoint are not covered.
|
||||
- **Knowledge graph** (`knowledge`, `entity_extract`) — `add_entity`,
|
||||
`add_entity_alias`, `add_relation`, `extract_and_store_entities`,
|
||||
traversal and spreading activation.
|
||||
- **Also:** sessions (`session`), temporal index (`temporal`),
|
||||
consolidation tiers (`consolidation`), an in-memory TTL tier
|
||||
(`ephemeral`), multi-modal embeddings (`multimodal`), `AGENTS.md`
|
||||
generation (`agents_md`), query expansion, and a session-scoped
|
||||
provenance ledger and write-anomaly detector on every save
|
||||
(`take_anomaly_alerts`; alerts never block a save, and the source is
|
||||
inferred from `source_channel`, not authenticated).
|
||||
- `openclaw::ClawhdfBackend` is `search` with re-ranking and confidence
|
||||
on, plus Markdown import/export. The module name is historical: it is not
|
||||
an OpenClaw plugin.
|
||||
|
||||
## Features
|
||||
|
||||
| Feature | Default | What | Builds C |
|
||||
|---|---|---|---|
|
||||
| `hnsw` | yes | HNSW vector index (`clawhdf5-ann`); without it the vector stage is an exact linear cosine scan | no |
|
||||
| `parallel` | yes | build the HNSW index on a rayon pool (same graph either way) | no |
|
||||
| `float16` | yes | f16 helpers in `vector_search` (`half`). Stores' `MemoryConfig::float16` works without it. | no |
|
||||
| `fast-math` | no | `matrixmultiply` batch distances in `strategy` | no |
|
||||
| `accelerate` | no | Apple Accelerate BLAS in `strategy` (macOS) | links a system framework |
|
||||
| `openblas` | no | OpenBLAS in `strategy` | yes (`openblas-src`) |
|
||||
| `gpu` | no | `gpu_search` through [`clawhdf5-gpu`](../clawhdf5-gpu/README.md) (wgpu), used by `strategy`, not by `HDF5Memory::search` | no, but needs GPU drivers |
|
||||
| `zstd` | no | Zstd instead of deflate when `MemoryConfig::compression` is on | yes (libzstd) |
|
||||
| `async` | no | `async_memory` wrapper on tokio | no |
|
||||
|
||||
`--no-default-features --features float16` forces the exact linear scan.
|
||||
|
||||
## Measurements and limits
|
||||
|
||||
- Search recall and latency, file size, LongMemEval and MemoryArena
|
||||
retrieval numbers: [`BENCHMARKS.md`](../../BENCHMARKS.md), measured with
|
||||
the `clawhdf5-bench` binaries (`search_harness`, `longmemeval_bench`,
|
||||
`footprint_bench`, ...).
|
||||
- Known issues and their history: [`docs/known-issues.md`](../../docs/known-issues.md).
|
||||
- Migrating a SQLite memory database:
|
||||
[`clawhdf5-migrate`](../clawhdf5-migrate/README.md).
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
||||
Reference in New Issue
Block a user