docs: bring the README up to date with the last five releases
The README had fallen behind v2.3.0-v2.7.0, and parts of it were not true. Checked every claim against the code and BENCHMARKS.md: - Three of the six Quick Start snippets no longer compiled (Agent Memory, Consolidation, OpenClaw); all six now do. - Hybrid search was described as RRF throughout. The default has been weighted 0.4/0.6 fusion since v2.5.0; re-ranking and confidence rejection run only in the OpenClaw backend. - The `float16` feature does not halve embedding storage (the store always writes f32), `--features agent` enables nothing, "Source Isolation" is not wired in, and nothing backs "billion-scale" IVF-PQ. - "Cryptographically verifiable" overstated an unkeyed, session-scoped FNV-1a ledger; "Zero C dependencies" was false while zlib-ng was the default deflate backend. - Stale numbers: tests (1,650 -> 1,868), Rust badge (1.75 is below edition 2024's floor), 6.5 KB/record on disk (BENCHMARKS.md: 1.7 KB), consolidation and hybrid-search latency, and a feature-flag table broken by a paragraph pasted into it. - The file schema, module table and crate map now match the code. Adds a "What's new (v2.2 -> v2.7)" section for collaborators, leading with the silent Extensible Array read bug fixed in v2.7.0. Footer links point at git.redclaw.dev. CLAUDE.md: clawhdf5-migrate is the SQLite migration tool, and MemoryConfig::compression is off by default. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
@@ -19,7 +19,7 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
|
||||
| `clawhdf5-agent` | Agent memory, session history, knowledge graph storage |
|
||||
| `clawhdf5-gpu` | GPU-accelerated I/O via wgpu (hand-written WGSL compute shaders) |
|
||||
| `clawhdf5-accel` | CPU SIMD acceleration path |
|
||||
| `clawhdf5-migrate` | Schema migration engine |
|
||||
| `clawhdf5-migrate` | SQLite → HDF5 agent-memory migration |
|
||||
| `clawhdf5-android` | Android JNI bindings |
|
||||
| `clawhdf5-cli` | Command-line interface |
|
||||
| `clawhdf5-napi` | Node.js native addon bindings |
|
||||
@@ -82,8 +82,8 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
|
||||
`export` do). An unreadable WAL (torn header, bad magic) is quarantined to
|
||||
`<store>.h5.wal.corrupt-<ts>` rather than blocking `open()`; a WAL with an
|
||||
unknown *newer* version still fails and is left untouched.
|
||||
- `MemoryConfig::compression` uses deflate by default; enable the agent's
|
||||
`zstd` feature to compress embeddings with Zstd instead (links libzstd).
|
||||
- `MemoryConfig::compression` is off by default; when on, embeddings are
|
||||
deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd).
|
||||
- `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by
|
||||
default) recomputes a dataset's SHA-256 and compares it against the
|
||||
`_provenance_sha256` attribute written automatically on save when
|
||||
|
||||
@@ -1,26 +1,85 @@
|
||||
# ClawhDF5
|
||||
|
||||
**The memory layer AI agents deserve. One file. Pure Rust. Zero C dependencies.**
|
||||
**The memory layer AI agents deserve. One file. Pure Rust. No libhdf5.**
|
||||
|
||||
[](LICENSE)
|
||||
[](https://www.rust-lang.org)
|
||||
[](#performance)
|
||||
[](BENCHMARKS.md#longmemeval-results)
|
||||
[](BENCHMARKS.md#memory-footprint)
|
||||
[](https://www.rust-lang.org)
|
||||
[](#building)
|
||||
[](BENCHMARKS.md#longmemeval-results)
|
||||
[](BENCHMARKS.md#memory-footprint-1)
|
||||
|
||||
ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, cryptographically verifiable memory — all stored in a single portable file.
|
||||
ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file.
|
||||
|
||||
> **Two things live here:**
|
||||
> - **A general-purpose, pure-Rust HDF5 library** — zero C dependencies, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers.
|
||||
> - **A general-purpose, pure-Rust HDF5 library** — no libhdf5, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers.
|
||||
> - **An agent memory layer built on top of it** — vector search, knowledge graph, hippocampal-style consolidation, in `clawhdf5-agent`.
|
||||
|
||||
```
|
||||
cargo add clawhdf5 # core HDF5 read/write, no agent layer
|
||||
cargo add clawhdf5-agent --features agent # + agent memory layer
|
||||
The crates are not on crates.io yet, so depend on them from git:
|
||||
|
||||
```toml
|
||||
[dependencies]
|
||||
clawhdf5 = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } # core HDF5 read/write
|
||||
clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } # + agent memory layer
|
||||
```
|
||||
|
||||
> **C dependencies, precisely:** the HDF5 format code is pure Rust and never
|
||||
> links libhdf5. The default deflate backend is zlib-ng (`fast-deflate`), a C
|
||||
> library built from source, so a default build needs `cmake` and a C
|
||||
> compiler. Opt-in codecs (`zstd`, `szip`) and BLAS backends link C too.
|
||||
|
||||
> **New here?** Start with the **[Quickstart Guide](docs/QUICKSTART.md)** · See **[Use Cases](docs/USE_CASES.md)** · Read **[Benchmarks](BENCHMARKS.md)**
|
||||
|
||||
## What's new (v2.2 → v2.7, and unreleased)
|
||||
|
||||
Five releases in September 2026. Details, including upgrade notes and every
|
||||
breaking change, are in [CHANGELOG.md](CHANGELOG.md).
|
||||
|
||||
**HDF5 correctness (read these if you read files with an earlier release)**
|
||||
- **Extensible Array chunk indexes returned wrong data** past the 36th chunk —
|
||||
any dataset with one unlimited dimension. Silent: plausible numbers from the
|
||||
wrong chunks. Fixed in v2.7.0; re-read affected data.
|
||||
- Fixed and Extensible Array checksums are now verified, so a corrupt chunk
|
||||
index is `ChecksumMismatch` instead of wrong data (v2.7.0).
|
||||
- Compound datatypes written with default libver bounds (plain
|
||||
`h5py.File(path, 'w')`) were mis-parsed; HDF5 2.0 compound v5 and native
|
||||
complex (class 11) types now parse (v2.2.0–v2.3.0).
|
||||
- Committed datatypes, fill values, soft links and `H5T_STD_REF` references now
|
||||
read correctly; external links and external raw data are explicit errors;
|
||||
`attrs()` no longer silently drops attributes (v2.3.0–v2.5.0).
|
||||
- Datasets indexed by a version-2 B-tree now read (v2.5.0).
|
||||
|
||||
**Security and robustness**
|
||||
- A crafted file could abort any reader via B-tree v2 recursion or explode it
|
||||
via shared children; both are now fast errors (v2.7.0).
|
||||
- Virtual-dataset source paths are confined to the file's directory; chunked
|
||||
reads use overflow-checked sizes and fallible allocation, and the facade
|
||||
writes files atomically (v2.3.0).
|
||||
- Agent store: single-writer lock plus `open_read_only`; a crash between
|
||||
checkpoint and WAL truncate no longer duplicates entries; unreadable WALs are
|
||||
quarantined instead of blocking `open()` (v2.3.0).
|
||||
|
||||
**Search quality and speed**
|
||||
- HNSW neighbour selection now uses the paper's diversity heuristic: recall@10
|
||||
at 100K went from 0.31 to 0.98 (v2.4.0).
|
||||
- `hybrid_search` is 79–190× faster than v2.3.0 (p50 0.07 ms at 1K, 4.65 ms at
|
||||
100K). It no longer rebuilds BM25 or rewrites the store per query, and the
|
||||
HNSW graph is persisted (v2.4.0).
|
||||
- Default fusion weights are now the measured 0.4 / 0.6 (v2.5.0). Re-ranking had
|
||||
been discarding the retrieval score, costing the OpenClaw backend 40.6pp of
|
||||
Hit@1; fixed in v2.6.0.
|
||||
- Selection reads decode only the chunks they touch (a 64×64 window: 105 ms to
|
||||
0.39 ms), and full reads are 1.2–1.9× faster (v2.5.0).
|
||||
|
||||
**Memory**
|
||||
- A loaded store holds ~30% less (embeddings stored once, v2.6.0), and the
|
||||
int8 HNSW index, **on by default for new stores** (unreleased), brings a
|
||||
100K × 384 store to 1.74× the raw vectors. At equal recall it is also faster
|
||||
than `f32`: 1.63× QPS on AVX2, 1.18× on a Raspberry Pi 5 (NEON `SDOT`).
|
||||
|
||||
**Tooling**
|
||||
- CI now runs the h5py/netCDF4 interop suites for real (they had been skipping
|
||||
silently) and runs an aarch64 job for the NEON kernels.
|
||||
|
||||
---
|
||||
|
||||
## Why ClawhDF5?
|
||||
@@ -35,7 +94,7 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d
|
||||
| Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers |
|
||||
| Temporal queries | Custom code | Native temporal index (716ns) |
|
||||
| Multi-modal | Multiple stores | Unified cross-modal search |
|
||||
| Security | Hope for the best | Provenance tracking + anomaly detection |
|
||||
| Integrity | Hope for the best | Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance |
|
||||
| Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** |
|
||||
|
||||
---
|
||||
@@ -60,6 +119,20 @@ Figures below are from an independent reproduction run on a second machine (AMD
|
||||
|
||||
### Vector Search
|
||||
|
||||
**HNSW (the default backend for `hybrid_search`)** — `search_harness`, clustered
|
||||
384-dim data, M = 16, ef_construction = 64, recall measured against an exact scan.
|
||||
See [BENCHMARKS.md § Search harness](BENCHMARKS.md#search-harness-baseline-v230)
|
||||
and [§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index):
|
||||
|
||||
| N = 100K, ef = 64 | recall@10 | QPS | build |
|
||||
|---|---:|---:|---:|
|
||||
| `f32` index | 0.9945 | 13 399 | 3.2 s |
|
||||
| `i8` index + exact re-score (**default for new stores**) | 0.9940 | **21 848** | **1.8 s** |
|
||||
|
||||
Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31.
|
||||
|
||||
**Brute-force and IVF paths** (Criterion, i7-12650H):
|
||||
|
||||
| Scale | Flat | IVF (nprobe=10) | IVF-PQ | vs MemX¹ |
|
||||
|-------|------|-----------------|--------|----------|
|
||||
| 1K | **54 µs** | — | — | — |
|
||||
@@ -75,7 +148,7 @@ Figures below are from an independent reproduction run on a second machine (AMD
|
||||
|
||||
| Operation | Latency | Scale |
|
||||
|-----------|---------|-------|
|
||||
| Hybrid search (RRF) | **222 µs** | 1K records |
|
||||
| Hybrid search (`HDF5Memory::hybrid_search`, p50) | **70 µs** / 0.49 ms / 4.65 ms | 1K / 10K / 100K records |
|
||||
| BM25 keyword search | **67 µs** | 1K records |
|
||||
| Knowledge graph BFS | **24 µs** | 1K entities |
|
||||
| Spreading activation | **17 µs** | 100 entities |
|
||||
@@ -115,13 +188,17 @@ declaration:
|
||||
|
||||
Hybrid is the strongest configuration, which is what running two retrieval stages
|
||||
is for. The weights matter more than the stages: a sweep of `vector_weight` from
|
||||
0.0 to 1.0 found the long-standing `0.7/0.3` default is **strictly dominated** by
|
||||
`0.4/0.6` — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Use
|
||||
`0.4/0.6`, or `0.3/0.7` if rank-1 precision matters most. See
|
||||
[BENCHMARKS.md § Weight sweep](BENCHMARKS.md#longmemeval-results).
|
||||
0.0 to 1.0 found the old `0.7/0.3` default is **strictly dominated** by
|
||||
`0.4/0.6` — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Since
|
||||
v2.5.0 `0.4/0.6` is the default (`hybrid::DEFAULT_FUSION`, used by
|
||||
`unified_search`, `hybrid_search_with` and the OpenClaw backend); callers that
|
||||
pass weights to `hybrid_search` explicitly choose their own. Use `0.3/0.7` if
|
||||
rank-1 precision matters most. Reciprocal rank fusion is selectable
|
||||
(`hybrid::Fusion::Rrf`) but measured worse than the weighted sum. See
|
||||
[BENCHMARKS.md § Weight sweep](BENCHMARKS.md#weight-sweep--full-haystack-n500).
|
||||
|
||||
Vector embeddings require `--features embeddings`; without it the vector stage is
|
||||
inert and only the BM25 row is produced, which is what every previously published
|
||||
The benchmark's vector stage requires `clawhdf5-bench`'s `embeddings` feature
|
||||
(real MiniLM embeddings); without it the vector stage is inert and only the BM25 row is produced, which is what every previously published
|
||||
number here measured.
|
||||
|
||||
On the easier `longmemeval_oracle` variant (evidence sessions only) the same
|
||||
@@ -146,19 +223,37 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points.
|
||||
|
||||
### Memory Footprint
|
||||
|
||||
| Records | File Size | Bytes/Record | With Compression |
|
||||
|---------|-----------|--------------|------------------|
|
||||
| 1K | ~6.5 MB | ~6.5 KB | ~2.1 MB (3.1x) |
|
||||
| 10K | ~65 MB | ~6.5 KB | ~21 MB (3.1x) |
|
||||
| 100K | ~645 MB | ~6.5 KB | ~208 MB (3.1x) |
|
||||
**On disk** — 384-dim embeddings, 200-char text
|
||||
([BENCHMARKS.md § Memory Footprint](BENCHMARKS.md#memory-footprint-1)):
|
||||
|
||||
| Records | File Size | Bytes/Record | Gzip-6 compressed |
|
||||
|---------|-----------|--------------|-------------------|
|
||||
| 1K | 1.7 MB | 1.8 KB | 277 KB (6.1x) |
|
||||
| 10K | 17.0 MB | 1.7 KB | 2.7 MB (6.2x) |
|
||||
| 100K | 169.8 MB | 1.7 KB | 26.9 MB (6.2x) |
|
||||
|
||||
**In memory** — a store reopened from disk, 384-dim `f32`, measured with a
|
||||
counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-footprint)):
|
||||
|
||||
| Records | Raw vectors | Reopened, `f32` index | Reopened, `i8` index (default) |
|
||||
|---------|-------------|-----------------------|--------------------------------|
|
||||
| 1K | 1 MiB | 4 MiB (2.40x) | 2 MiB (1.64x) |
|
||||
| 10K | 15 MiB | 44 MiB (3.03x) | 27 MiB (1.81x) |
|
||||
| 100K | 146 MiB | 399 MiB (2.72x) | **256 MiB (1.74x)** |
|
||||
|
||||
Down from 505 MiB (3.44x) at 100K before v2.6.0, when the cache held every
|
||||
embedding twice.
|
||||
|
||||
### Consolidation Efficiency
|
||||
|
||||
1,000 records (10 signal + 990 noise), `working_capacity = 100`
|
||||
([BENCHMARKS.md § Consolidation Efficiency](BENCHMARKS.md#consolidation-efficiency)):
|
||||
|
||||
| Metric | Before | After | Delta |
|
||||
|--------|--------|-------|-------|
|
||||
| Records in store | 1,000 | ~110 | −89% |
|
||||
| Hit@1 recall | ~60% | ~90% | +30% |
|
||||
| Search latency | ~2.8 ms | ~0.3 ms | **9x faster** |
|
||||
| Records in store | 1,000 | 100 | −90% |
|
||||
| Hit@1 recall (signal records) | 100% | 100% | no loss |
|
||||
| Search latency | 2.75 ms | 0.31 ms | **8.8x faster** |
|
||||
|
||||
**Full benchmark details: [BENCHMARKS.md](BENCHMARKS.md)**
|
||||
|
||||
@@ -166,74 +261,71 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points.
|
||||
|
||||
## Agent Memory Architecture
|
||||
|
||||
ClawhDF5's agent memory engine implements research from 15+ recent papers on agentic memory systems. It's not a toy — it's the real thing.
|
||||
ClawhDF5's agent memory engine draws on 15+ recent papers on agentic memory systems (see [Research Foundation](#research-foundation)).
|
||||
|
||||
```
|
||||
┌─────────────────┐
|
||||
│ Agent Query │
|
||||
└────────┬────────┘
|
||||
│
|
||||
┌────────────▼────────────┐
|
||||
│ Hybrid Retrieval │
|
||||
│ Vector + BM25 + RRF │
|
||||
└────────────┬────────────┘
|
||||
│
|
||||
┌──────────────────▼──────────────────┐
|
||||
│ Multi-Factor Re-Ranking │
|
||||
│ temporal · authority · activation │
|
||||
└──────────────────┬──────────────────┘
|
||||
│
|
||||
┌────────────▼────────────┐
|
||||
│ Confidence Rejection │
|
||||
┌─────────────────▼──────────────────┐
|
||||
│ HDF5Memory::hybrid_search │
|
||||
│ HNSW vector + BM25 keyword │
|
||||
│ weighted fusion (0.4 / 0.6) │
|
||||
│ × √(Hebbian activation) │
|
||||
└─────────────────┬──────────────────┘
|
||||
│ OpenClaw backend adds:
|
||||
┌─────────────────▼──────────────────┐
|
||||
│ Multi-factor re-ranking │
|
||||
│ relevance · recency · authority · │
|
||||
│ activation │
|
||||
├────────────────────────────────────┤
|
||||
│ Confidence rejection │
|
||||
│ (suppress bad matches) │
|
||||
└────────────┬────────────┘
|
||||
└─────────────────┬──────────────────┘
|
||||
│
|
||||
┌────────────────────────▼────────────────────────┐
|
||||
│ Memory Store (HDF5) │
|
||||
│ │
|
||||
│ ┌───────────┐ ┌───────────┐ ┌───────────────┐ │
|
||||
│ │ Working │→│ Episodic │→│ Semantic │ │
|
||||
│ │ (bounded) │ │ (bounded) │ │ (long-term) │ │
|
||||
│ └───────────┘ └───────────┘ └───────────────┘ │
|
||||
│ │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌────────────────┐ │
|
||||
│ │Knowledge │ │Temporal │ │ Multi-Modal │ │
|
||||
│ │ Graph │ │ Index │ │ Embeddings │ │
|
||||
│ └──────────┘ └──────────┘ └────────────────┘ │
|
||||
│ │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌────────────────┐ │
|
||||
│ │Provenance│ │ Anomaly │ │ Source │ │
|
||||
│ │ Tracking │ │Detection │ │ Isolation │ │
|
||||
│ └──────────┘ └──────────┘ └────────────────┘ │
|
||||
└─────────────────────────────────────────────────┘
|
||||
│
|
||||
┌────────┴────────┐
|
||||
│ agent_memory.h5 │
|
||||
│ single file │
|
||||
└─────────────────┘
|
||||
┌────────────────────────────▼────────────────────────────┐
|
||||
│ In memory │
|
||||
│ cache (flat f32 embeddings) · BM25 index · HNSW index │
|
||||
│ provenance ledger + anomaly alerts (session-scoped) │
|
||||
└────────────────────────────┬────────────────────────────┘
|
||||
│ WAL append; checkpoint
|
||||
┌────────────────────────────▼────────────────────────────┐
|
||||
│ agent_memory.h5 /meta · /memory · /sessions · │
|
||||
│ /knowledge_graph │
|
||||
│ agent_memory.h5.wal chained-CRC write-ahead log │
|
||||
│ agent_memory.h5.ann HNSW graph (derived, rebuildable) │
|
||||
│ agent_memory.h5.lock single-writer lock │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Consolidation tiers (Working → Episodic → Semantic), the knowledge-graph
|
||||
algorithms, temporal and multi-modal indexes are library components you drive
|
||||
directly; the store persists the records, sessions and graph they work over.
|
||||
|
||||
### Module Overview
|
||||
|
||||
| Module | What It Does |
|
||||
|--------|-------------|
|
||||
| **`knowledge`** | Entity/relation graph with BFS traversal, spreading activation, fuzzy entity resolution |
|
||||
| **`consolidation`** | Three-tier memory (Working → Episodic → Semantic) with importance scoring and time-decay |
|
||||
| **`hybrid`** | Vector + BM25 fusion with Reciprocal Rank Fusion (RRF, k=60). The vector stage uses the HNSW index by default (`hnsw` feature, on by default); disable with `--no-default-features --features float16` for an exact linear scan |
|
||||
| **`reranker`** | Multi-factor re-ranking: temporal recency, source authority, activation weight |
|
||||
| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches |
|
||||
| **`knowledge`** | Entity/relation graph with BFS traversal, spreading activation, fuzzy (Levenshtein) entity resolution |
|
||||
| **`consolidation`** | Three-tier memory (Working → Episodic → Semantic) with importance scoring, novelty, and time-decay |
|
||||
| **`hybrid`** | Vector + BM25 fusion. Default is a min-max-normalised weighted sum, vector 0.4 / keyword 0.6 (`hybrid::DEFAULT_FUSION`, tuned on LongMemEval); RRF is available via `Fusion::Rrf` / `hybrid_search_with`. The vector stage uses the HNSW index by default (`hnsw` feature); disable with `--no-default-features --features float16` for an exact linear scan |
|
||||
| **`reranker`** | Multi-factor re-ranking: retrieval relevance (leads, weight 1.0), temporal recency, source authority, activation weight. Used by the OpenClaw backend |
|
||||
| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches (OpenClaw backend) |
|
||||
| **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints |
|
||||
| **`multimodal`** | Cross-modal search across text/image/audio/video embeddings |
|
||||
| **`provenance`** | Source attribution, FNV-1a content hashing, integrity verification |
|
||||
| **`anomaly`** | Write rate limiting, 15 injection pattern detectors, source distribution analysis |
|
||||
| **`provenance`** | Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof) |
|
||||
| **`anomaly`** | Write rate limiting, 15 injection-pattern detectors, source-distribution analysis. Alerts never block a save; drain them with `take_anomaly_alerts` |
|
||||
| **`openclaw`** | OpenClaw integration: MemoryBackend trait, Markdown ↔ HDF5 conversion |
|
||||
| **`vector_search`** | Flat cosine, pre-normed, SIMD, BLAS, GPU, parallel search paths |
|
||||
| **`ivf` / `pq`** | IVF-PQ approximate nearest neighbor for billion-scale search |
|
||||
| **`bm25`** | BM25 keyword index with TF-IDF scoring |
|
||||
| **`ivf` / `pq`** | Standalone IVF and IVF-PQ indexes (benchmarked to 100K vectors); not used by `HDF5Memory`, whose ANN index is HNSW |
|
||||
| **`bm25`** | Incremental Okapi BM25 inverted index, kept for the life of the store; optional stemming |
|
||||
| **`query_expand`** | Synonym / acronym / temporal query expansion |
|
||||
| **`entity_extract`** | Rule-based entity extraction from text chunks into the knowledge graph |
|
||||
| **`wal`** | Write-ahead log for crash-safe persistence; each entry is CRC32-checked on replay, so a corrupted entry stops replay there instead of loading bad data |
|
||||
| **`wal`** | Write-ahead log (v4) with a chained CRC32 per entry, so a corrupted, reordered, duplicated or spliced entry stops replay; checkpoints record a WAL mark so nothing is applied twice. Appends are not fsynced |
|
||||
| **`memory_strategy`** | Pluggable strategies: save-every, semantic-shift, user-correction detection |
|
||||
| **`decision_gate`** | Sub-microsecond trivial/substantive classification |
|
||||
| **`ephemeral`** | In-memory TTL/LFU working tier |
|
||||
| **`async_memory`** | Tokio-based async wrapper over the memory store (`async` feature) |
|
||||
|
||||
---
|
||||
@@ -265,7 +357,7 @@ assert_eq!(values, vec![22.5, 23.1, 21.8]);
|
||||
use clawhdf5_agent::{HDF5Memory, MemoryConfig, MemoryEntry, AgentMemory};
|
||||
|
||||
// Create memory store
|
||||
let config = MemoryConfig::new("agent.h5", "my-agent", 384);
|
||||
let config = MemoryConfig::new("agent.h5".into(), "my-agent", 384);
|
||||
let mut memory = HDF5Memory::create(config)?;
|
||||
|
||||
// Save a memory
|
||||
@@ -278,8 +370,8 @@ memory.save(MemoryEntry {
|
||||
tags: "preference".into(),
|
||||
})?;
|
||||
|
||||
// Search
|
||||
let results = memory.search(&query_embedding, 5)?;
|
||||
// Hybrid search: vector + BM25, weighted 0.4 / 0.6 (the measured default)
|
||||
let results = memory.hybrid_search(&query_embedding, "user preferences", 0.4, 0.6, 5);
|
||||
for result in results {
|
||||
println!("[{:.3}] {}", result.score, result.chunk);
|
||||
}
|
||||
@@ -309,8 +401,8 @@ let neighbors = kg.bfs_neighbors(alice, 2); // 2-hop neighborhood
|
||||
let activated = kg.spreading_activation(&[alice], 0.5, 0.01, 5);
|
||||
|
||||
// Entity resolution — fuzzy matching
|
||||
let resolved = kg.resolve_or_create("alice", "person", -1, 2);
|
||||
// Returns existing Alice entity (Levenshtein distance ≤ 2)
|
||||
let (id, created) = kg.resolve_or_create("alice", "person", -1, 2);
|
||||
// id == alice, created == false: matched the existing entity (Levenshtein distance ≤ 2)
|
||||
```
|
||||
|
||||
### Memory Consolidation
|
||||
@@ -321,15 +413,19 @@ use clawhdf5_agent::consolidation::*;
|
||||
let config = ConsolidationConfig::default();
|
||||
let mut engine = ConsolidationEngine::new(config);
|
||||
|
||||
// Add memories — automatically scored for importance
|
||||
engine.add_memory("User prefers dark mode", vec![0.1, 0.2, ...], MemorySource::User);
|
||||
engine.add_memory("ok", vec![0.0, 0.0, ...], MemorySource::System);
|
||||
let now = 1_700_000_000.0; // seconds since the epoch
|
||||
|
||||
// Add memories — automatically scored for importance.
|
||||
// Elevated sources (System, …) go through a separate, explicit API.
|
||||
let id = engine.add_memory("User prefers dark mode".into(), vec![0.1, 0.2, ...], UntrustedSource::User, now);
|
||||
engine.add_trusted_memory("ok".into(), vec![0.0, 0.0, ...], TrustedSource::System, now);
|
||||
|
||||
// Access a memory (reactivates it)
|
||||
engine.access_memory(0);
|
||||
engine.access_memory(id, now);
|
||||
|
||||
// Run consolidation cycle
|
||||
let stats = engine.consolidate();
|
||||
engine.consolidate(now);
|
||||
let stats = engine.get_stats();
|
||||
// Working memories promote to Episodic (if important enough)
|
||||
// Episodic memories promote to Semantic (if accessed enough)
|
||||
// Low-decay memories get evicted when tiers are full
|
||||
@@ -357,13 +453,13 @@ let recent = index.latest(10);
|
||||
use clawhdf5_agent::openclaw::*;
|
||||
|
||||
// Create backend
|
||||
let mut backend = ClawhdfBackend::create("memory.h5", "agent-1", 384)?;
|
||||
let mut backend = ClawhdfBackend::create(std::path::Path::new("memory.h5"), 384)?;
|
||||
|
||||
// Ingest existing Markdown memory files
|
||||
let md = std::fs::read_to_string("MEMORY.md")?;
|
||||
let count = backend.ingest_markdown("MEMORY.md", &md)?;
|
||||
|
||||
// Search (uses full pipeline: RRF → re-rank → confidence filter)
|
||||
// Search (full pipeline: weighted vector + BM25 fusion → re-rank → confidence filter)
|
||||
let results = backend.search("user preferences", &query_embedding, 5);
|
||||
|
||||
// Export back to Markdown
|
||||
@@ -375,22 +471,23 @@ let exported = backend.export_markdown("MEMORY.md")?;
|
||||
## Crate Map
|
||||
|
||||
```
|
||||
clawhdf5 workspace (16 crates, ~92K lines of Rust; plus libaec-sys, an
|
||||
internal FFI bindings crate for the optional szip feature)
|
||||
clawhdf5 workspace (16 crates, ~86K lines of Rust in src/, ~104K with tests
|
||||
and benches; plus libaec-sys, an internal FFI bindings
|
||||
crate for the optional szip feature)
|
||||
│
|
||||
├── Core HDF5
|
||||
│ ├── clawhdf5-format — Binary parser/writer (no_std), shared type definitions
|
||||
│ ├── clawhdf5-io — I/O abstraction (buffered, mmap, async)
|
||||
│ ├── clawhdf5-format — Binary parser/writer (no_std-capable), shared type definitions
|
||||
│ ├── clawhdf5-io — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI)
|
||||
│ ├── clawhdf5-filters — Fast deflate path (zlib-ng); lz4/zstd/pcodec/szip filters live in clawhdf5-format
|
||||
│ ├── clawhdf5-derive — Proc macros
|
||||
│ ├── clawhdf5 — High-level API
|
||||
│ ├── clawhdf5-netcdf4 — NetCDF-4 support
|
||||
│ ├── clawhdf5-accel — SIMD (NEON, AVX2, AVX-512)
|
||||
│ ├── clawhdf5-accel — SIMD (AVX2, NEON incl. SDOT int8; AVX-512 behind `avx512`)
|
||||
│ └── clawhdf5-gpu — GPU compute (wgpu, hand-written WGSL compute shaders)
|
||||
│
|
||||
├── Agent Memory
|
||||
│ ├── clawhdf5-agent — Memory engine (20.9K lines, 32 modules; WAL is CRC32-checked per entry)
|
||||
│ ├── clawhdf5-ann — HNSW approximate nearest neighbor (default backend; optional `parallel` feature)
|
||||
│ ├── clawhdf5-agent — Memory engine (24.7K lines, 32 modules; chained-CRC WAL)
|
||||
│ ├── clawhdf5-ann — HNSW approximate nearest neighbor (default backend; f32 or int8 storage; `parallel` build)
|
||||
│ ├── clawhdf5-migrate — SQLite → HDF5 migration
|
||||
│ ├── clawhdf5-android — Android JNI bridge
|
||||
│ └── clawhdf5-cli — CLI tool
|
||||
@@ -411,10 +508,10 @@ ClawhDF5's agent memory design draws from 15+ recent papers:
|
||||
|
||||
| Paper | Key Insight | ClawhDF5 Module |
|
||||
|-------|-------------|-----------------|
|
||||
| **MemX** (2026) | RRF + multi-factor re-ranking | `hybrid`, `reranker` |
|
||||
| **Graph-Native Cognitive Memory** (2026) | Graph-structured belief revision | `knowledge` |
|
||||
| **MemX** (2026) | Hybrid fusion + multi-factor re-ranking | `hybrid`, `reranker` |
|
||||
| **Graph-Native Cognitive Memory** (2026) | Graph-structured memory (weighted, timestamped relations; entity timelines) | `knowledge`, `temporal` |
|
||||
| **CraniMem** (2026) | Bounded hippocampal memory | `consolidation` |
|
||||
| **D-MEM** (2026) | Reward prediction error gating | `consolidation` |
|
||||
| **D-MEM** (2026) | Surprise-gated storage (implemented as a novelty score) | `consolidation` |
|
||||
| **SYNAPSE** (2025) | Spreading activation for recall | `knowledge` |
|
||||
| **RAGdb** (2025) | Zero-dependency edge RAG | Architecture |
|
||||
| **MemoryGraft** (2025) | Memory poisoning attacks | `anomaly`, `provenance` |
|
||||
@@ -429,29 +526,35 @@ ClawhDF5's agent memory design draws from 15+ recent papers:
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `agent` | no | Full agent memory layer |
|
||||
| `float16` | **yes** | Half-precision embedding storage (2× compression) |
|
||||
| `float16` | **yes** | Half-precision cosine kernel (`cosine_similarity_f16`). The store itself always writes `f32` embeddings; `MemoryConfig::float16` is recorded in `/meta` but not yet applied |
|
||||
| `hnsw` | **yes** | HNSW approximate vector index for `hybrid_search` (via `clawhdf5-ann`); disable for an exact linear scan |
|
||||
| `parallel` | **yes** | Parallel HNSW bulk build (same graph, ~3× faster on 16 cores) and Rayon brute-force search strategies |
|
||||
| `zstd` | no | Compress embeddings with Zstd instead of deflate when `MemoryConfig::compression` is on (links libzstd) |
|
||||
| `fast-math` | no | BLAS matrix-vector multiply |
|
||||
| `accelerate` | no | Apple Accelerate / AMX (macOS) |
|
||||
| `openblas` | no | OpenBLAS (Linux) |
|
||||
| `gpu` | no | GPU search via wgpu |
|
||||
| `async` | no | Tokio async with background flush |
|
||||
| `agent` | no | Reserved; currently enables nothing (the agent layer is always built) |
|
||||
|
||||
To opt out of the parallel build: `--no-default-features --features float16,hnsw`.
|
||||
For an exact linear cosine scan instead of HNSW: `--no-default-features --features float16`.
|
||||
|
||||
`MemoryConfig::hnsw_m`, `hnsw_ef_construction` and `hnsw_ef_search` tune the
|
||||
vector index (16 / 64 / scale-with-`k` by default) and are stored with the
|
||||
file.
|
||||
|
||||
`MemoryConfig::quantized_index` (**on by default** for new stores) holds the
|
||||
HNSW index's own
|
||||
copy of the embeddings as `i8`, roughly halving a loaded store's memory
|
||||
(2.72x -> 1.74x the raw vectors at 100k x 384). Quantised distances are
|
||||
approximate, so the query path re-scores the candidate pool against the exact
|
||||
embeddings the store already holds, which keeps recall at the `f32` index's
|
||||
level. It is also **faster**: 1.63x the queries per second at equal recall on
|
||||
x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with index builds
|
||||
1.8x and 2.3x faster respectively. See `BENCHMARKS.md`, "Quantising the index copy".
|
||||
| `parallel` | no | Rayon parallel search |
|
||||
| `fast-math` | no | BLAS matrix-vector multiply |
|
||||
| `accelerate` | no | Apple Accelerate / AMX (macOS) |
|
||||
| `openblas` | no | OpenBLAS (Linux) |
|
||||
| `gpu` | no | GPU search via wgpu |
|
||||
| `async` | no | Tokio async with background flush |
|
||||
HNSW index's own copy of the embeddings as `i8`, roughly halving a loaded
|
||||
store's memory (2.72x -> 1.74x the raw vectors at 100k x 384). Quantised
|
||||
distances are approximate, so the query path re-scores the candidate pool
|
||||
against the exact embeddings the store already holds, which keeps recall at the
|
||||
`f32` index's level. It is also **faster**: 1.63x the queries per second at
|
||||
equal recall on x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with
|
||||
index builds 1.8x and 2.3x faster respectively. Stores created before the
|
||||
setting existed keep their `f32` index; opt out for new stores with
|
||||
`quantized_index = false` or `clawhdf5-cli create --f32-index`. See
|
||||
[BENCHMARKS.md § Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index).
|
||||
|
||||
### `clawhdf5-format`
|
||||
|
||||
@@ -461,8 +564,8 @@ x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with index builds
|
||||
| `deflate` | yes | Deflate compression |
|
||||
| `checksum` | yes | Jenkins lookup3 verification |
|
||||
| `provenance` | yes | SHA-256 provenance attributes |
|
||||
| `fast-deflate` | **yes** | zlib-ng backend for faster deflate |
|
||||
| `system-zlib-decompress` | **yes** | Use the system zlib for decompression where available |
|
||||
| `fast-deflate` | **yes** | zlib-ng backend for faster deflate (C; needs `cmake`) |
|
||||
| `system-zlib-decompress` | **yes** | Use Apple's system libz for decompression (macOS only; no effect elsewhere) |
|
||||
| `parallel` | no | Parallel chunk encoding + compression (rayon) |
|
||||
| `fast-checksum` | no | crc32fast-accelerated checksums |
|
||||
| `lz4` | no | LZ4 block compression filter (id 32004) |
|
||||
@@ -470,17 +573,21 @@ x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with index builds
|
||||
| `pcodec` | no | Pcodec lossless numerical codec (id 32023, via `pco` crate) |
|
||||
| `system-zlib` / `zlib-rs` | no | Alternative zlib backends for deflate |
|
||||
| `blake3_hash` | no | BLAKE3 content hashing for provenance |
|
||||
| `szip` | no | SZIP filter (id 4) via libaec (C, through the internal `libaec-sys` crate) |
|
||||
|
||||
### `clawhdf5-ann`
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `parallel` | no | Rayon-parallel neighbor-distance computation during HNSW graph pruning |
|
||||
| `parallel` | no | Batched bulk build runs neighbour planning and back-link pruning on a Rayon pool; the graph is identical with or without it (enabled by `clawhdf5-agent`'s default `parallel`) |
|
||||
|
||||
### `clawhdf5-io`
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `mmap` | no | Memory-mapped reads (`memmap2`) |
|
||||
| `async` | no | Tokio-based async I/O |
|
||||
| `hsds` | no | HSDS (HDF REST service) client |
|
||||
| `mpi-io` | no | MPI-backed I/O via the `mpi` crate |
|
||||
|
||||
> **Parallel I/O (MPI) limitation:** `mpi-io`'s read path is a root-rank read
|
||||
@@ -494,17 +601,17 @@ x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with index builds
|
||||
## Building
|
||||
|
||||
```bash
|
||||
# Default
|
||||
# Default (needs cmake + a C compiler for zlib-ng)
|
||||
cargo build --workspace
|
||||
|
||||
# Agent memory with all accelerations (Linux)
|
||||
cargo build -p clawhdf5-agent --features "agent,float16,parallel,fast-math"
|
||||
cargo build -p clawhdf5-agent --features fast-math
|
||||
|
||||
# Agent memory with Apple Accelerate (macOS)
|
||||
cargo build -p clawhdf5-agent --features "agent,float16,accelerate,parallel,gpu"
|
||||
cargo build -p clawhdf5-agent --features "accelerate,gpu"
|
||||
|
||||
# Tests
|
||||
cargo test --workspace # all 1,650+ tests
|
||||
cargo test --workspace # all 1,850+ tests
|
||||
cargo test -p clawhdf5-agent # agent memory tests
|
||||
scripts/ci-test.sh # what CI runs: fmt, clippy matrix, tests,
|
||||
# h5py/netCDF4 interop, no_std
|
||||
@@ -526,25 +633,41 @@ cargo bench -p clawhdf5-bench # h5bench-equivalent I/O suite
|
||||
|
||||
```
|
||||
agent_memory.h5
|
||||
├── /meta
|
||||
│ ├── schema_version: "1.0"
|
||||
│ ├── agent_id, embedder, embedding_dim
|
||||
│ └── created_at
|
||||
├── /meta (attributes)
|
||||
│ ├── schema_version: "1.0", edgehdf5_version
|
||||
│ ├── agent_id, embedder, embedding_dim, chunk_size, overlap, created_at
|
||||
│ ├── float16, compression, compression_level, compact_threshold,
|
||||
│ │ hebbian_boost, decay_factor, wal_enabled, wal_max_entries
|
||||
│ ├── quantized_index, hnsw_m, hnsw_ef_construction, hnsw_ef_search
|
||||
│ ├── wal_applied_len, wal_applied_crc (WAL mark of the last checkpoint)
|
||||
│ └── ann_generation (ties the .ann sidecar to this checkpoint)
|
||||
├── /memory
|
||||
│ ├── chunks: string[N]
|
||||
│ ├── embeddings: f32[N × D] (or f16 with float16 flag)
|
||||
│ ├── embeddings: f32[N × D] (chunked; deflate, or Zstd with the
|
||||
│ │ `zstd` feature, when compression is on)
|
||||
│ ├── source_channel: string[N]
|
||||
│ ├── timestamps: f64[N]
|
||||
│ ├── session_ids: string[N]
|
||||
│ ├── tags: string[N]
|
||||
│ ├── tombstones: u8[N]
|
||||
│ └── norms: f32[N] (pre-computed L2)
|
||||
│ ├── norms: f32[N] (pre-computed L2)
|
||||
│ └── activation_weights: f32[N] (Hebbian)
|
||||
├── /sessions
|
||||
│ ├── ids: string[S]
|
||||
│ └── summaries: string[S]
|
||||
│ ├── ids, channels, summaries: string[S]
|
||||
│ ├── start_idxs, end_idxs: i64[S]
|
||||
│ └── timestamps: f64[S]
|
||||
└── /knowledge_graph
|
||||
├── entity_names: string[E]
|
||||
├── relation_srcs: i64[R]
|
||||
├── relation_tgts: i64[R]
|
||||
└── relation_types: string[R]
|
||||
├── entity_ids, entity_emb_idxs: i64[E]; entity_names, entity_types: string[E]
|
||||
├── relation_srcs, relation_tgts: i64[R]; relation_types: string[R]
|
||||
├── relation_weights: f32[R]; relation_ts: f64[R]
|
||||
└── alias_strings: string[A]; alias_entity_ids: i64[A] (when aliases exist)
|
||||
```
|
||||
|
||||
Alongside the store: `<store>.h5.wal` (write-ahead log), `<store>.h5.ann`
|
||||
(HNSW graph; derived, safe to delete) and `<store>.h5.lock` (single-writer
|
||||
lock). A second writer gets `MemoryError::Locked`; use
|
||||
`HDF5Memory::open_read_only` for a lock-free point-in-time view.
|
||||
|
||||
---
|
||||
|
||||
## Migration
|
||||
@@ -599,6 +722,6 @@ MIT
|
||||
---
|
||||
|
||||
<p align="center">
|
||||
<em>Built by <a href="https://github.com/redclawsystems">RedClaw Systems</a></em><br>
|
||||
<em>~92,000 lines of Rust. Zero C dependencies. One file to remember everything.</em>
|
||||
<em>Built by <a href="https://git.redclaw.dev/quantumclaw">RedClaw Systems</a></em><br>
|
||||
<em>~86,000 lines of Rust. No libhdf5. One file to remember everything.</em>
|
||||
</p>
|
||||
|
||||
+1
-1
@@ -567,4 +567,4 @@ let final_results = confidence::reject_low_confidence(
|
||||
|
||||
---
|
||||
|
||||
<p align="center"><em>Built by <a href="https://github.com/redclawsystems">RedClaw Systems</a></em></p>
|
||||
<p align="center"><em>Built by <a href="https://git.redclaw.dev/quantumclaw">RedClaw Systems</a></em></p>
|
||||
|
||||
+1
-1
@@ -232,4 +232,4 @@ clawhdf5-agent = { version = "2.0", features = ["agent", "float16", "accelerate"
|
||||
|
||||
---
|
||||
|
||||
<p align="center"><em>Built by <a href="https://github.com/redclawsystems">RedClaw Systems</a></em></p>
|
||||
<p align="center"><em>Built by <a href="https://git.redclaw.dev/quantumclaw">RedClaw Systems</a></em></p>
|
||||
|
||||
Reference in New Issue
Block a user