quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: bring the README up to date with the last five releases

Browse Source 
The README had fallen behind v2.3.0-v2.7.0, and parts of it were not
true. Checked every claim against the code and BENCHMARKS.md:

- Three of the six Quick Start snippets no longer compiled (Agent
  Memory, Consolidation, OpenClaw); all six now do.
- Hybrid search was described as RRF throughout. The default has been
  weighted 0.4/0.6 fusion since v2.5.0; re-ranking and confidence
  rejection run only in the OpenClaw backend.
- The `float16` feature does not halve embedding storage (the store
  always writes f32), `--features agent` enables nothing, "Source
  Isolation" is not wired in, and nothing backs "billion-scale" IVF-PQ.
- "Cryptographically verifiable" overstated an unkeyed, session-scoped
  FNV-1a ledger; "Zero C dependencies" was false while zlib-ng was the
  default deflate backend.
- Stale numbers: tests (1,650 -> 1,868), Rust badge (1.75 is below
  edition 2024's floor), 6.5 KB/record on disk (BENCHMARKS.md: 1.7 KB),
  consolidation and hybrid-search latency, and a feature-flag table
  broken by a paragraph pasted into it.
- The file schema, module table and crate map now match the code.

Adds a "What's new (v2.2 -> v2.7)" section for collaborators, leading
with the silent Extensible Array read bug fixed in v2.7.0. Footer links
point at git.redclaw.dev. CLAUDE.md: clawhdf5-migrate is the SQLite
migration tool, and MemoryConfig::compression is off by default.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-23 11:05:10 -05:00
co-authored by Claude Opus 5.5
parent 4bed8b3765
commit e9b37a9602
4 changed files with 268 additions and 145 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+3 -3
View File
@@ -19,7 +19,7 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
| `clawhdf5-agent` | Agent memory, session history, knowledge graph storage | | `clawhdf5-agent` | Agent memory, session history, knowledge graph storage |
| `clawhdf5-gpu` | GPU-accelerated I/O via wgpu (hand-written WGSL compute shaders) | | `clawhdf5-gpu` | GPU-accelerated I/O via wgpu (hand-written WGSL compute shaders) |
| `clawhdf5-accel` | CPU SIMD acceleration path | | `clawhdf5-accel` | CPU SIMD acceleration path |
| `clawhdf5-migrate` | Schema migration engine | | `clawhdf5-migrate` | SQLite → HDF5 agent-memory migration |
| `clawhdf5-android` | Android JNI bindings | | `clawhdf5-android` | Android JNI bindings |
| `clawhdf5-cli` | Command-line interface | | `clawhdf5-cli` | Command-line interface |
| `clawhdf5-napi` | Node.js native addon bindings | | `clawhdf5-napi` | Node.js native addon bindings |
@@ -82,8 +82,8 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
`export` do). An unreadable WAL (torn header, bad magic) is quarantined to `export` do). An unreadable WAL (torn header, bad magic) is quarantined to
`<store>.h5.wal.corrupt-<ts>` rather than blocking `open()`; a WAL with an `<store>.h5.wal.corrupt-<ts>` rather than blocking `open()`; a WAL with an
unknown *newer* version still fails and is left untouched. unknown *newer* version still fails and is left untouched.
- `MemoryConfig::compression` uses deflate by default; enable the agent's - `MemoryConfig::compression` is off by default; when on, embeddings are
`zstd` feature to compress embeddings with Zstd instead (links libzstd). deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd).
- `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by - `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by
default) recomputes a dataset's SHA-256 and compares it against the default) recomputes a dataset's SHA-256 and compares it against the
`_provenance_sha256` attribute written automatically on save when `_provenance_sha256` attribute written automatically on save when
README.md
+255 -132
View File
@@ -1,26 +1,85 @@
# ClawhDF5 # ClawhDF5
**The memory layer AI agents deserve. One file. Pure Rust. Zero C dependencies.** **The memory layer AI agents deserve. One file. Pure Rust. No libhdf5.**
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Rust](https://img.shields.io/badge/rust-1.75%2B-orange.svg)](https://www.rust-lang.org) [![Rust](https://img.shields.io/badge/rust-1.94%2B-orange.svg)](https://www.rust-lang.org)
[![Tests](https://img.shields.io/badge/tests-1650%2B%20passing-brightgreen.svg)](#performance) [![Tests](https://img.shields.io/badge/tests-1850%2B-brightgreen.svg)](#building)
[![LongMemEval](https://img.shields.io/badge/LongMemEval%20oracle-Turn--Level%20Hit@5%2084%25%20BM25--only-blue.svg)](BENCHMARKS.md#longmemeval-results) [![LongMemEval](https://img.shields.io/badge/LongMemEval__s-Turn--Level%20Hit@5%2081.4%25%20hybrid-blue.svg)](BENCHMARKS.md#longmemeval-results)
[![Footprint](https://img.shields.io/badge/footprint-6.5%20KB%2Frecord-lightgrey.svg)](BENCHMARKS.md#memory-footprint) [![Footprint](https://img.shields.io/badge/on--disk-1.7%20KB%2Frecord-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1)
ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, cryptographically verifiable memory — all stored in a single portable file. ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file.
> **Two things live here:** > **Two things live here:**
> - **A general-purpose, pure-Rust HDF5 library** — zero C dependencies, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers. > - **A general-purpose, pure-Rust HDF5 library** — no libhdf5, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers.
> - **An agent memory layer built on top of it** — vector search, knowledge graph, hippocampal-style consolidation, in `clawhdf5-agent`. > - **An agent memory layer built on top of it** — vector search, knowledge graph, hippocampal-style consolidation, in `clawhdf5-agent`.
``` The crates are not on crates.io yet, so depend on them from git:
cargo add clawhdf5 # core HDF5 read/write, no agent layer
cargo add clawhdf5-agent --features agent # + agent memory layer ```toml
[dependencies]
clawhdf5 = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } # core HDF5 read/write
clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } # + agent memory layer
``` ```
> **C dependencies, precisely:** the HDF5 format code is pure Rust and never
> links libhdf5. The default deflate backend is zlib-ng (`fast-deflate`), a C
> library built from source, so a default build needs `cmake` and a C
> compiler. Opt-in codecs (`zstd`, `szip`) and BLAS backends link C too.
> **New here?** Start with the **[Quickstart Guide](docs/QUICKSTART.md)** · See **[Use Cases](docs/USE_CASES.md)** · Read **[Benchmarks](BENCHMARKS.md)** > **New here?** Start with the **[Quickstart Guide](docs/QUICKSTART.md)** · See **[Use Cases](docs/USE_CASES.md)** · Read **[Benchmarks](BENCHMARKS.md)**
## What's new (v2.2 → v2.7, and unreleased)
Five releases in September 2026. Details, including upgrade notes and every
breaking change, are in [CHANGELOG.md](CHANGELOG.md).
**HDF5 correctness (read these if you read files with an earlier release)**
- **Extensible Array chunk indexes returned wrong data** past the 36th chunk —
any dataset with one unlimited dimension. Silent: plausible numbers from the
wrong chunks. Fixed in v2.7.0; re-read affected data.
- Fixed and Extensible Array checksums are now verified, so a corrupt chunk
index is `ChecksumMismatch` instead of wrong data (v2.7.0).
- Compound datatypes written with default libver bounds (plain
`h5py.File(path, 'w')`) were mis-parsed; HDF5 2.0 compound v5 and native
complex (class 11) types now parse (v2.2.0–v2.3.0).
- Committed datatypes, fill values, soft links and `H5T_STD_REF` references now
read correctly; external links and external raw data are explicit errors;
`attrs()` no longer silently drops attributes (v2.3.0–v2.5.0).
- Datasets indexed by a version-2 B-tree now read (v2.5.0).
**Security and robustness**
- A crafted file could abort any reader via B-tree v2 recursion or explode it
via shared children; both are now fast errors (v2.7.0).
- Virtual-dataset source paths are confined to the file's directory; chunked
reads use overflow-checked sizes and fallible allocation, and the facade
writes files atomically (v2.3.0).
- Agent store: single-writer lock plus `open_read_only`; a crash between
checkpoint and WAL truncate no longer duplicates entries; unreadable WALs are
quarantined instead of blocking `open()` (v2.3.0).
**Search quality and speed**
- HNSW neighbour selection now uses the paper's diversity heuristic: recall@10
at 100K went from 0.31 to 0.98 (v2.4.0).
- `hybrid_search` is 79–190× faster than v2.3.0 (p50 0.07 ms at 1K, 4.65 ms at
100K). It no longer rebuilds BM25 or rewrites the store per query, and the
HNSW graph is persisted (v2.4.0).
- Default fusion weights are now the measured 0.4 / 0.6 (v2.5.0). Re-ranking had
been discarding the retrieval score, costing the OpenClaw backend 40.6pp of
Hit@1; fixed in v2.6.0.
- Selection reads decode only the chunks they touch (a 64×64 window: 105 ms to
0.39 ms), and full reads are 1.2–1.9× faster (v2.5.0).
**Memory**
- A loaded store holds ~30% less (embeddings stored once, v2.6.0), and the
int8 HNSW index, **on by default for new stores** (unreleased), brings a
100K × 384 store to 1.74× the raw vectors. At equal recall it is also faster
than `f32`: 1.63× QPS on AVX2, 1.18× on a Raspberry Pi 5 (NEON `SDOT`).
**Tooling**
- CI now runs the h5py/netCDF4 interop suites for real (they had been skipping
silently) and runs an aarch64 job for the NEON kernels.
--- ---
## Why ClawhDF5? ## Why ClawhDF5?
@@ -35,7 +94,7 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d
| Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers | | Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers |
| Temporal queries | Custom code | Native temporal index (716ns) | | Temporal queries | Custom code | Native temporal index (716ns) |
| Multi-modal | Multiple stores | Unified cross-modal search | | Multi-modal | Multiple stores | Unified cross-modal search |
| Security | Hope for the best | Provenance tracking + anomaly detection | | Integrity | Hope for the best | Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance |
| Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** | | Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** |
--- ---
@@ -60,6 +119,20 @@ Figures below are from an independent reproduction run on a second machine (AMD
### Vector Search ### Vector Search
**HNSW (the default backend for `hybrid_search`)** — `search_harness`, clustered
384-dim data, M = 16, ef_construction = 64, recall measured against an exact scan.
See [BENCHMARKS.md § Search harness](BENCHMARKS.md#search-harness-baseline-v230)
and [§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index):
| N = 100K, ef = 64 | recall@10 | QPS | build |
|---|---:|---:|---:|
| `f32` index | 0.9945 | 13 399 | 3.2 s |
| `i8` index + exact re-score (**default for new stores**) | 0.9940 | **21 848** | **1.8 s** |
Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31.
**Brute-force and IVF paths** (Criterion, i7-12650H):
| Scale | Flat | IVF (nprobe=10) | IVF-PQ | vs MemX¹ | | Scale | Flat | IVF (nprobe=10) | IVF-PQ | vs MemX¹ |
|-------|------|-----------------|--------|----------| |-------|------|-----------------|--------|----------|
| 1K | **54 µs** | — | — | — | | 1K | **54 µs** | — | — | — |
@@ -75,7 +148,7 @@ Figures below are from an independent reproduction run on a second machine (AMD
| Operation | Latency | Scale | | Operation | Latency | Scale |
|-----------|---------|-------| |-----------|---------|-------|
| Hybrid search (RRF) | **222 µs** | 1K records | | Hybrid search (`HDF5Memory::hybrid_search`, p50) | **70 µs** / 0.49 ms / 4.65 ms | 1K / 10K / 100K records |
| BM25 keyword search | **67 µs** | 1K records | | BM25 keyword search | **67 µs** | 1K records |
| Knowledge graph BFS | **24 µs** | 1K entities | | Knowledge graph BFS | **24 µs** | 1K entities |
| Spreading activation | **17 µs** | 100 entities | | Spreading activation | **17 µs** | 100 entities |
@@ -115,13 +188,17 @@ declaration:
Hybrid is the strongest configuration, which is what running two retrieval stages Hybrid is the strongest configuration, which is what running two retrieval stages
is for. The weights matter more than the stages: a sweep of `vector_weight` from is for. The weights matter more than the stages: a sweep of `vector_weight` from
0.0 to 1.0 found the long-standing `0.7/0.3` default is **strictly dominated** by 0.0 to 1.0 found the old `0.7/0.3` default is **strictly dominated** by
`0.4/0.6` — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Use `0.4/0.6` — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Since
`0.4/0.6`, or `0.3/0.7` if rank-1 precision matters most. See v2.5.0 `0.4/0.6` is the default (`hybrid::DEFAULT_FUSION`, used by
[BENCHMARKS.md § Weight sweep](BENCHMARKS.md#longmemeval-results). `unified_search`, `hybrid_search_with` and the OpenClaw backend); callers that
pass weights to `hybrid_search` explicitly choose their own. Use `0.3/0.7` if
rank-1 precision matters most. Reciprocal rank fusion is selectable
(`hybrid::Fusion::Rrf`) but measured worse than the weighted sum. See
[BENCHMARKS.md § Weight sweep](BENCHMARKS.md#weight-sweep--full-haystack-n500).
Vector embeddings require `--features embeddings`; without it the vector stage is The benchmark's vector stage requires `clawhdf5-bench`'s `embeddings` feature
inert and only the BM25 row is produced, which is what every previously published (real MiniLM embeddings); without it the vector stage is inert and only the BM25 row is produced, which is what every previously published
number here measured. number here measured.
On the easier `longmemeval_oracle` variant (evidence sessions only) the same On the easier `longmemeval_oracle` variant (evidence sessions only) the same
@@ -146,19 +223,37 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points.
### Memory Footprint ### Memory Footprint
| Records | File Size | Bytes/Record | With Compression | **On disk** — 384-dim embeddings, 200-char text
|---------|-----------|--------------|------------------| ([BENCHMARKS.md § Memory Footprint](BENCHMARKS.md#memory-footprint-1)):
| 1K | ~6.5 MB | ~6.5 KB | ~2.1 MB (3.1x) |
| 10K | ~65 MB | ~6.5 KB | ~21 MB (3.1x) | | Records | File Size | Bytes/Record | Gzip-6 compressed |
| 100K | ~645 MB | ~6.5 KB | ~208 MB (3.1x) | |---------|-----------|--------------|-------------------|
| 1K | 1.7 MB | 1.8 KB | 277 KB (6.1x) |
| 10K | 17.0 MB | 1.7 KB | 2.7 MB (6.2x) |
| 100K | 169.8 MB | 1.7 KB | 26.9 MB (6.2x) |
**In memory** — a store reopened from disk, 384-dim `f32`, measured with a
counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-footprint)):
| Records | Raw vectors | Reopened, `f32` index | Reopened, `i8` index (default) |
|---------|-------------|-----------------------|--------------------------------|
| 1K | 1 MiB | 4 MiB (2.40x) | 2 MiB (1.64x) |
| 10K | 15 MiB | 44 MiB (3.03x) | 27 MiB (1.81x) |
| 100K | 146 MiB | 399 MiB (2.72x) | **256 MiB (1.74x)** |
Down from 505 MiB (3.44x) at 100K before v2.6.0, when the cache held every
embedding twice.
### Consolidation Efficiency ### Consolidation Efficiency
1,000 records (10 signal + 990 noise), `working_capacity = 100`
([BENCHMARKS.md § Consolidation Efficiency](BENCHMARKS.md#consolidation-efficiency)):
| Metric | Before | After | Delta | | Metric | Before | After | Delta |
|--------|--------|-------|-------| |--------|--------|-------|-------|
| Records in store | 1,000 | ~110 | −89% | | Records in store | 1,000 | 100 | −90% |
| Hit@1 recall | ~60% | ~90% | +30% | | Hit@1 recall (signal records) | 100% | 100% | no loss |
| Search latency | ~2.8 ms | ~0.3 ms | **9x faster** | | Search latency | 2.75 ms | 0.31 ms | **8.8x faster** |
**Full benchmark details: [BENCHMARKS.md](BENCHMARKS.md)** **Full benchmark details: [BENCHMARKS.md](BENCHMARKS.md)**
@@ -166,74 +261,71 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points.
## Agent Memory Architecture ## Agent Memory Architecture
ClawhDF5's agent memory engine implements research from 15+ recent papers on agentic memory systems. It's not a toy — it's the real thing. ClawhDF5's agent memory engine draws on 15+ recent papers on agentic memory systems (see [Research Foundation](#research-foundation)).
``` ```
┌─────────────────┐ ┌─────────────────┐
│ Agent Query │ │ Agent Query │
└────────┬────────┘ └────────┬────────┘
│ │
┌────────────▼────────────┐ ┌─────────────────▼──────────────────┐
│ Hybrid Retrieval │ │ HDF5Memory::hybrid_search │
│ Vector + BM25 + RRF │ │ HNSW vector + BM25 keyword │
└────────────┬────────────┘ │ weighted fusion (0.4 / 0.6) │
│ │ × √(Hebbian activation) │
┌──────────────────▼──────────────────┐ └─────────────────┬──────────────────┘
│ Multi-Factor Re-Ranking │ │ OpenClaw backend adds:
│ temporal · authority · activation │ ┌─────────────────▼──────────────────┐
└──────────────────┬──────────────────┘ │ Multi-factor re-ranking │
│ │ relevance · recency · authority · │
┌────────────▼────────────┐ │ activation │
│ Confidence Rejection │ ├────────────────────────────────────┤
│ Confidence rejection │
│ (suppress bad matches) │ │ (suppress bad matches) │
└────────────┬────────────┘ └─────────────────┬──────────────────┘
│ │
┌────────────────────────▼────────────────────────┐ ┌────────────────────────────▼────────────────────────────┐
│ Memory Store (HDF5) │ │ In memory │
│ │ │ cache (flat f32 embeddings) · BM25 index · HNSW index │
│ ┌───────────┐ ┌───────────┐ ┌───────────────┐ │ │ provenance ledger + anomaly alerts (session-scoped) │
│ │ Working │→│ Episodic │→│ Semantic │ │ └────────────────────────────┬────────────────────────────┘
│ │ (bounded) │ │ (bounded) │ │ (long-term) │ │ │ WAL append; checkpoint
│ └───────────┘ └───────────┘ └───────────────┘ │ ┌────────────────────────────▼────────────────────────────┐
│ │ │ agent_memory.h5 /meta · /memory · /sessions · │
│ ┌──────────┐ ┌──────────┐ ┌────────────────┐ │ │ /knowledge_graph │
│ │Knowledge │ │Temporal │ │ Multi-Modal │ │ │ agent_memory.h5.wal chained-CRC write-ahead log │
│ │ Graph │ │ Index │ │ Embeddings │ │ │ agent_memory.h5.ann HNSW graph (derived, rebuildable) │
│ └──────────┘ └──────────┘ └────────────────┘ │ │ agent_memory.h5.lock single-writer lock │
│ │ └─────────────────────────────────────────────────────────┘
│ ┌──────────┐ ┌──────────┐ ┌────────────────┐ │
│ │Provenance│ │ Anomaly │ │ Source │ │
│ │ Tracking │ │Detection │ │ Isolation │ │
│ └──────────┘ └──────────┘ └────────────────┘ │
└─────────────────────────────────────────────────┘
│
┌────────┴────────┐
│ agent_memory.h5 │
│ single file │
└─────────────────┘
``` ```
Consolidation tiers (Working → Episodic → Semantic), the knowledge-graph
algorithms, temporal and multi-modal indexes are library components you drive
directly; the store persists the records, sessions and graph they work over.
### Module Overview ### Module Overview
| Module | What It Does | | Module | What It Does |
|--------|-------------| |--------|-------------|
| **`knowledge`** | Entity/relation graph with BFS traversal, spreading activation, fuzzy entity resolution | | **`knowledge`** | Entity/relation graph with BFS traversal, spreading activation, fuzzy (Levenshtein) entity resolution |
| **`consolidation`** | Three-tier memory (Working → Episodic → Semantic) with importance scoring and time-decay | | **`consolidation`** | Three-tier memory (Working → Episodic → Semantic) with importance scoring, novelty, and time-decay |
| **`hybrid`** | Vector + BM25 fusion with Reciprocal Rank Fusion (RRF, k=60). The vector stage uses the HNSW index by default (`hnsw` feature, on by default); disable with `--no-default-features --features float16` for an exact linear scan | | **`hybrid`** | Vector + BM25 fusion. Default is a min-max-normalised weighted sum, vector 0.4 / keyword 0.6 (`hybrid::DEFAULT_FUSION`, tuned on LongMemEval); RRF is available via `Fusion::Rrf` / `hybrid_search_with`. The vector stage uses the HNSW index by default (`hnsw` feature); disable with `--no-default-features --features float16` for an exact linear scan |
| **`reranker`** | Multi-factor re-ranking: temporal recency, source authority, activation weight | | **`reranker`** | Multi-factor re-ranking: retrieval relevance (leads, weight 1.0), temporal recency, source authority, activation weight. Used by the OpenClaw backend |
| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches | | **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches (OpenClaw backend) |
| **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints | | **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints |
| **`multimodal`** | Cross-modal search across text/image/audio/video embeddings | | **`multimodal`** | Cross-modal search across text/image/audio/video embeddings |
| **`provenance`** | Source attribution, FNV-1a content hashing, integrity verification | | **`provenance`** | Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof) |
| **`anomaly`** | Write rate limiting, 15 injection pattern detectors, source distribution analysis | | **`anomaly`** | Write rate limiting, 15 injection-pattern detectors, source-distribution analysis. Alerts never block a save; drain them with `take_anomaly_alerts` |
| **`openclaw`** | OpenClaw integration: MemoryBackend trait, Markdown ↔ HDF5 conversion | | **`openclaw`** | OpenClaw integration: MemoryBackend trait, Markdown ↔ HDF5 conversion |
| **`vector_search`** | Flat cosine, pre-normed, SIMD, BLAS, GPU, parallel search paths | | **`vector_search`** | Flat cosine, pre-normed, SIMD, BLAS, GPU, parallel search paths |
| **`ivf` / `pq`** | IVF-PQ approximate nearest neighbor for billion-scale search | | **`ivf` / `pq`** | Standalone IVF and IVF-PQ indexes (benchmarked to 100K vectors); not used by `HDF5Memory`, whose ANN index is HNSW |
| **`bm25`** | BM25 keyword index with TF-IDF scoring | | **`bm25`** | Incremental Okapi BM25 inverted index, kept for the life of the store; optional stemming |
| **`query_expand`** | Synonym / acronym / temporal query expansion |
| **`entity_extract`** | Rule-based entity extraction from text chunks into the knowledge graph | | **`entity_extract`** | Rule-based entity extraction from text chunks into the knowledge graph |
| **`wal`** | Write-ahead log for crash-safe persistence; each entry is CRC32-checked on replay, so a corrupted entry stops replay there instead of loading bad data | | **`wal`** | Write-ahead log (v4) with a chained CRC32 per entry, so a corrupted, reordered, duplicated or spliced entry stops replay; checkpoints record a WAL mark so nothing is applied twice. Appends are not fsynced |
| **`memory_strategy`** | Pluggable strategies: save-every, semantic-shift, user-correction detection | | **`memory_strategy`** | Pluggable strategies: save-every, semantic-shift, user-correction detection |
| **`decision_gate`** | Sub-microsecond trivial/substantive classification | | **`decision_gate`** | Sub-microsecond trivial/substantive classification |
| **`ephemeral`** | In-memory TTL/LFU working tier |
| **`async_memory`** | Tokio-based async wrapper over the memory store (`async` feature) | | **`async_memory`** | Tokio-based async wrapper over the memory store (`async` feature) |
--- ---
@@ -265,7 +357,7 @@ assert_eq!(values, vec![22.5, 23.1, 21.8]);
use clawhdf5_agent::{HDF5Memory, MemoryConfig, MemoryEntry, AgentMemory}; use clawhdf5_agent::{HDF5Memory, MemoryConfig, MemoryEntry, AgentMemory};
// Create memory store // Create memory store
let config = MemoryConfig::new("agent.h5", "my-agent", 384); let config = MemoryConfig::new("agent.h5".into(), "my-agent", 384);
let mut memory = HDF5Memory::create(config)?; let mut memory = HDF5Memory::create(config)?;
// Save a memory // Save a memory
@@ -278,8 +370,8 @@ memory.save(MemoryEntry {
tags: "preference".into(), tags: "preference".into(),
})?; })?;
// Search // Hybrid search: vector + BM25, weighted 0.4 / 0.6 (the measured default)
let results = memory.search(&query_embedding, 5)?; let results = memory.hybrid_search(&query_embedding, "user preferences", 0.4, 0.6, 5);
for result in results { for result in results {
println!("[{:.3}] {}", result.score, result.chunk); println!("[{:.3}] {}", result.score, result.chunk);
} }
@@ -309,8 +401,8 @@ let neighbors = kg.bfs_neighbors(alice, 2); // 2-hop neighborhood
let activated = kg.spreading_activation(&[alice], 0.5, 0.01, 5); let activated = kg.spreading_activation(&[alice], 0.5, 0.01, 5);
// Entity resolution — fuzzy matching // Entity resolution — fuzzy matching
let resolved = kg.resolve_or_create("alice", "person", -1, 2); let (id, created) = kg.resolve_or_create("alice", "person", -1, 2);
// Returns existing Alice entity (Levenshtein distance ≤ 2) // id == alice, created == false: matched the existing entity (Levenshtein distance ≤ 2)
``` ```
### Memory Consolidation ### Memory Consolidation
@@ -321,15 +413,19 @@ use clawhdf5_agent::consolidation::*;
let config = ConsolidationConfig::default(); let config = ConsolidationConfig::default();
let mut engine = ConsolidationEngine::new(config); let mut engine = ConsolidationEngine::new(config);
// Add memories — automatically scored for importance let now = 1_700_000_000.0; // seconds since the epoch
engine.add_memory("User prefers dark mode", vec![0.1, 0.2, ...], MemorySource::User);
engine.add_memory("ok", vec![0.0, 0.0, ...], MemorySource::System); // Add memories — automatically scored for importance.
// Elevated sources (System, …) go through a separate, explicit API.
let id = engine.add_memory("User prefers dark mode".into(), vec![0.1, 0.2, ...], UntrustedSource::User, now);
engine.add_trusted_memory("ok".into(), vec![0.0, 0.0, ...], TrustedSource::System, now);
// Access a memory (reactivates it) // Access a memory (reactivates it)
engine.access_memory(0); engine.access_memory(id, now);
// Run consolidation cycle // Run consolidation cycle
let stats = engine.consolidate(); engine.consolidate(now);
let stats = engine.get_stats();
// Working memories promote to Episodic (if important enough) // Working memories promote to Episodic (if important enough)
// Episodic memories promote to Semantic (if accessed enough) // Episodic memories promote to Semantic (if accessed enough)
// Low-decay memories get evicted when tiers are full // Low-decay memories get evicted when tiers are full
@@ -357,13 +453,13 @@ let recent = index.latest(10);
use clawhdf5_agent::openclaw::*; use clawhdf5_agent::openclaw::*;
// Create backend // Create backend
let mut backend = ClawhdfBackend::create("memory.h5", "agent-1", 384)?; let mut backend = ClawhdfBackend::create(std::path::Path::new("memory.h5"), 384)?;
// Ingest existing Markdown memory files // Ingest existing Markdown memory files
let md = std::fs::read_to_string("MEMORY.md")?; let md = std::fs::read_to_string("MEMORY.md")?;
let count = backend.ingest_markdown("MEMORY.md", &md)?; let count = backend.ingest_markdown("MEMORY.md", &md)?;
// Search (uses full pipeline: RRF → re-rank → confidence filter) // Search (full pipeline: weighted vector + BM25 fusion → re-rank → confidence filter)
let results = backend.search("user preferences", &query_embedding, 5); let results = backend.search("user preferences", &query_embedding, 5);
// Export back to Markdown // Export back to Markdown
@@ -375,22 +471,23 @@ let exported = backend.export_markdown("MEMORY.md")?;
## Crate Map ## Crate Map
``` ```
clawhdf5 workspace (16 crates, ~92K lines of Rust; plus libaec-sys, an clawhdf5 workspace (16 crates, ~86K lines of Rust in src/, ~104K with tests
internal FFI bindings crate for the optional szip feature) and benches; plus libaec-sys, an internal FFI bindings
crate for the optional szip feature)
│ │
├── Core HDF5 ├── Core HDF5
│ ├── clawhdf5-format — Binary parser/writer (no_std), shared type definitions │ ├── clawhdf5-format — Binary parser/writer (no_std-capable), shared type definitions
│ ├── clawhdf5-io — I/O abstraction (buffered, mmap, async) │ ├── clawhdf5-io — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI)
│ ├── clawhdf5-filters — Fast deflate path (zlib-ng); lz4/zstd/pcodec/szip filters live in clawhdf5-format │ ├── clawhdf5-filters — Fast deflate path (zlib-ng); lz4/zstd/pcodec/szip filters live in clawhdf5-format
│ ├── clawhdf5-derive — Proc macros │ ├── clawhdf5-derive — Proc macros
│ ├── clawhdf5 — High-level API │ ├── clawhdf5 — High-level API
│ ├── clawhdf5-netcdf4 — NetCDF-4 support │ ├── clawhdf5-netcdf4 — NetCDF-4 support
│ ├── clawhdf5-accel — SIMD (NEON, AVX2, AVX-512) │ ├── clawhdf5-accel — SIMD (AVX2, NEON incl. SDOT int8; AVX-512 behind `avx512`)
│ └── clawhdf5-gpu — GPU compute (wgpu, hand-written WGSL compute shaders) │ └── clawhdf5-gpu — GPU compute (wgpu, hand-written WGSL compute shaders)
│ │
├── Agent Memory ├── Agent Memory
│ ├── clawhdf5-agent — Memory engine (20.9K lines, 32 modules; WAL is CRC32-checked per entry) │ ├── clawhdf5-agent — Memory engine (24.7K lines, 32 modules; chained-CRC WAL)
│ ├── clawhdf5-ann — HNSW approximate nearest neighbor (default backend; optional `parallel` feature) │ ├── clawhdf5-ann — HNSW approximate nearest neighbor (default backend; f32 or int8 storage; `parallel` build)
│ ├── clawhdf5-migrate — SQLite → HDF5 migration │ ├── clawhdf5-migrate — SQLite → HDF5 migration
│ ├── clawhdf5-android — Android JNI bridge │ ├── clawhdf5-android — Android JNI bridge
│ └── clawhdf5-cli — CLI tool │ └── clawhdf5-cli — CLI tool
@@ -411,10 +508,10 @@ ClawhDF5's agent memory design draws from 15+ recent papers:
| Paper | Key Insight | ClawhDF5 Module | | Paper | Key Insight | ClawhDF5 Module |
|-------|-------------|-----------------| |-------|-------------|-----------------|
| **MemX** (2026) | RRF + multi-factor re-ranking | `hybrid`, `reranker` | | **MemX** (2026) | Hybrid fusion + multi-factor re-ranking | `hybrid`, `reranker` |
| **Graph-Native Cognitive Memory** (2026) | Graph-structured belief revision | `knowledge` | | **Graph-Native Cognitive Memory** (2026) | Graph-structured memory (weighted, timestamped relations; entity timelines) | `knowledge`, `temporal` |
| **CraniMem** (2026) | Bounded hippocampal memory | `consolidation` | | **CraniMem** (2026) | Bounded hippocampal memory | `consolidation` |
| **D-MEM** (2026) | Reward prediction error gating | `consolidation` | | **D-MEM** (2026) | Surprise-gated storage (implemented as a novelty score) | `consolidation` |
| **SYNAPSE** (2025) | Spreading activation for recall | `knowledge` | | **SYNAPSE** (2025) | Spreading activation for recall | `knowledge` |
| **RAGdb** (2025) | Zero-dependency edge RAG | Architecture | | **RAGdb** (2025) | Zero-dependency edge RAG | Architecture |
| **MemoryGraft** (2025) | Memory poisoning attacks | `anomaly`, `provenance` | | **MemoryGraft** (2025) | Memory poisoning attacks | `anomaly`, `provenance` |
@@ -429,29 +526,35 @@ ClawhDF5's agent memory design draws from 15+ recent papers:
| Flag | Default | Description | | Flag | Default | Description |
|------|---------|-------------| |------|---------|-------------|
| `agent` | no | Full agent memory layer | | `float16` | **yes** | Half-precision cosine kernel (`cosine_similarity_f16`). The store itself always writes `f32` embeddings; `MemoryConfig::float16` is recorded in `/meta` but not yet applied |
| `float16` | **yes** | Half-precision embedding storage (2× compression) |
| `hnsw` | **yes** | HNSW approximate vector index for `hybrid_search` (via `clawhdf5-ann`); disable for an exact linear scan | | `hnsw` | **yes** | HNSW approximate vector index for `hybrid_search` (via `clawhdf5-ann`); disable for an exact linear scan |
| `parallel` | **yes** | Parallel HNSW bulk build (same graph, ~3× faster on 16 cores) and Rayon brute-force search strategies |
| `zstd` | no | Compress embeddings with Zstd instead of deflate when `MemoryConfig::compression` is on (links libzstd) |
| `fast-math` | no | BLAS matrix-vector multiply |
| `accelerate` | no | Apple Accelerate / AMX (macOS) |
| `openblas` | no | OpenBLAS (Linux) |
| `gpu` | no | GPU search via wgpu |
| `async` | no | Tokio async with background flush |
| `agent` | no | Reserved; currently enables nothing (the agent layer is always built) |
To opt out of the parallel build: `--no-default-features --features float16,hnsw`.
For an exact linear cosine scan instead of HNSW: `--no-default-features --features float16`.
`MemoryConfig::hnsw_m`, `hnsw_ef_construction` and `hnsw_ef_search` tune the `MemoryConfig::hnsw_m`, `hnsw_ef_construction` and `hnsw_ef_search` tune the
vector index (16 / 64 / scale-with-`k` by default) and are stored with the vector index (16 / 64 / scale-with-`k` by default) and are stored with the
file. file.
`MemoryConfig::quantized_index` (**on by default** for new stores) holds the `MemoryConfig::quantized_index` (**on by default** for new stores) holds the
HNSW index's own HNSW index's own copy of the embeddings as `i8`, roughly halving a loaded
copy of the embeddings as `i8`, roughly halving a loaded store's memory store's memory (2.72x -> 1.74x the raw vectors at 100k x 384). Quantised
(2.72x -> 1.74x the raw vectors at 100k x 384). Quantised distances are distances are approximate, so the query path re-scores the candidate pool
approximate, so the query path re-scores the candidate pool against the exact against the exact embeddings the store already holds, which keeps recall at the
embeddings the store already holds, which keeps recall at the `f32` index's `f32` index's level. It is also **faster**: 1.63x the queries per second at
level. It is also **faster**: 1.63x the queries per second at equal recall on equal recall on x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with
x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with index builds index builds 1.8x and 2.3x faster respectively. Stores created before the
1.8x and 2.3x faster respectively. See `BENCHMARKS.md`, "Quantising the index copy". setting existed keep their `f32` index; opt out for new stores with
| `parallel` | no | Rayon parallel search | `quantized_index = false` or `clawhdf5-cli create --f32-index`. See
| `fast-math` | no | BLAS matrix-vector multiply | [BENCHMARKS.md § Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index).
| `accelerate` | no | Apple Accelerate / AMX (macOS) |
| `openblas` | no | OpenBLAS (Linux) |
| `gpu` | no | GPU search via wgpu |
| `async` | no | Tokio async with background flush |
### `clawhdf5-format` ### `clawhdf5-format`
@@ -461,8 +564,8 @@ x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with index builds
| `deflate` | yes | Deflate compression | | `deflate` | yes | Deflate compression |
| `checksum` | yes | Jenkins lookup3 verification | | `checksum` | yes | Jenkins lookup3 verification |
| `provenance` | yes | SHA-256 provenance attributes | | `provenance` | yes | SHA-256 provenance attributes |
| `fast-deflate` | **yes** | zlib-ng backend for faster deflate | | `fast-deflate` | **yes** | zlib-ng backend for faster deflate (C; needs `cmake`) |
| `system-zlib-decompress` | **yes** | Use the system zlib for decompression where available | | `system-zlib-decompress` | **yes** | Use Apple's system libz for decompression (macOS only; no effect elsewhere) |
| `parallel` | no | Parallel chunk encoding + compression (rayon) | | `parallel` | no | Parallel chunk encoding + compression (rayon) |
| `fast-checksum` | no | crc32fast-accelerated checksums | | `fast-checksum` | no | crc32fast-accelerated checksums |
| `lz4` | no | LZ4 block compression filter (id 32004) | | `lz4` | no | LZ4 block compression filter (id 32004) |
@@ -470,17 +573,21 @@ x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with index builds
| `pcodec` | no | Pcodec lossless numerical codec (id 32023, via `pco` crate) | | `pcodec` | no | Pcodec lossless numerical codec (id 32023, via `pco` crate) |
| `system-zlib` / `zlib-rs` | no | Alternative zlib backends for deflate | | `system-zlib` / `zlib-rs` | no | Alternative zlib backends for deflate |
| `blake3_hash` | no | BLAKE3 content hashing for provenance | | `blake3_hash` | no | BLAKE3 content hashing for provenance |
| `szip` | no | SZIP filter (id 4) via libaec (C, through the internal `libaec-sys` crate) |
### `clawhdf5-ann` ### `clawhdf5-ann`
| Flag | Default | Description | | Flag | Default | Description |
|------|---------|-------------| |------|---------|-------------|
| `parallel` | no | Rayon-parallel neighbor-distance computation during HNSW graph pruning | | `parallel` | no | Batched bulk build runs neighbour planning and back-link pruning on a Rayon pool; the graph is identical with or without it (enabled by `clawhdf5-agent`'s default `parallel`) |
### `clawhdf5-io` ### `clawhdf5-io`
| Flag | Default | Description | | Flag | Default | Description |
|------|---------|-------------| |------|---------|-------------|
| `mmap` | no | Memory-mapped reads (`memmap2`) |
| `async` | no | Tokio-based async I/O |
| `hsds` | no | HSDS (HDF REST service) client |
| `mpi-io` | no | MPI-backed I/O via the `mpi` crate | | `mpi-io` | no | MPI-backed I/O via the `mpi` crate |
> **Parallel I/O (MPI) limitation:** `mpi-io`'s read path is a root-rank read > **Parallel I/O (MPI) limitation:** `mpi-io`'s read path is a root-rank read
@@ -494,17 +601,17 @@ x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with index builds
## Building ## Building
```bash ```bash
# Default # Default (needs cmake + a C compiler for zlib-ng)
cargo build --workspace cargo build --workspace
# Agent memory with all accelerations (Linux) # Agent memory with all accelerations (Linux)
cargo build -p clawhdf5-agent --features "agent,float16,parallel,fast-math" cargo build -p clawhdf5-agent --features fast-math
# Agent memory with Apple Accelerate (macOS) # Agent memory with Apple Accelerate (macOS)
cargo build -p clawhdf5-agent --features "agent,float16,accelerate,parallel,gpu" cargo build -p clawhdf5-agent --features "accelerate,gpu"
# Tests # Tests
cargo test --workspace # all 1,650+ tests cargo test --workspace # all 1,850+ tests
cargo test -p clawhdf5-agent # agent memory tests cargo test -p clawhdf5-agent # agent memory tests
scripts/ci-test.sh # what CI runs: fmt, clippy matrix, tests, scripts/ci-test.sh # what CI runs: fmt, clippy matrix, tests,
# h5py/netCDF4 interop, no_std # h5py/netCDF4 interop, no_std
@@ -526,25 +633,41 @@ cargo bench -p clawhdf5-bench # h5bench-equivalent I/O suite
``` ```
agent_memory.h5 agent_memory.h5
├── /meta ├── /meta (attributes)
│ ├── schema_version: "1.0" │ ├── schema_version: "1.0", edgehdf5_version
│ ├── agent_id, embedder, embedding_dim │ ├── agent_id, embedder, embedding_dim, chunk_size, overlap, created_at
│ └── created_at │ ├── float16, compression, compression_level, compact_threshold,
│ │ hebbian_boost, decay_factor, wal_enabled, wal_max_entries
│ ├── quantized_index, hnsw_m, hnsw_ef_construction, hnsw_ef_search
│ ├── wal_applied_len, wal_applied_crc (WAL mark of the last checkpoint)
│ └── ann_generation (ties the .ann sidecar to this checkpoint)
├── /memory ├── /memory
│ ├── chunks: string[N] │ ├── chunks: string[N]
│ ├── embeddings: f32[N × D] (or f16 with float16 flag) │ ├── embeddings: f32[N × D] (chunked; deflate, or Zstd with the
│ │ `zstd` feature, when compression is on)
│ ├── source_channel: string[N]
│ ├── timestamps: f64[N]
│ ├── session_ids: string[N]
│ ├── tags: string[N]
│ ├── tombstones: u8[N] │ ├── tombstones: u8[N]
│ └── norms: f32[N] (pre-computed L2) │ ├── norms: f32[N] (pre-computed L2)
│ └── activation_weights: f32[N] (Hebbian)
├── /sessions ├── /sessions
│ ├── ids: string[S] │ ├── ids, channels, summaries: string[S]
│ └── summaries: string[S] │ ├── start_idxs, end_idxs: i64[S]
│ └── timestamps: f64[S]
└── /knowledge_graph └── /knowledge_graph
├── entity_names: string[E] ├── entity_ids, entity_emb_idxs: i64[E]; entity_names, entity_types: string[E]
├── relation_srcs: i64[R] ├── relation_srcs, relation_tgts: i64[R]; relation_types: string[R]
├── relation_tgts: i64[R] ├── relation_weights: f32[R]; relation_ts: f64[R]
└── relation_types: string[R] └── alias_strings: string[A]; alias_entity_ids: i64[A] (when aliases exist)
``` ```
Alongside the store: `<store>.h5.wal` (write-ahead log), `<store>.h5.ann`
(HNSW graph; derived, safe to delete) and `<store>.h5.lock` (single-writer
lock). A second writer gets `MemoryError::Locked`; use
`HDF5Memory::open_read_only` for a lock-free point-in-time view.
--- ---
## Migration ## Migration
@@ -599,6 +722,6 @@ MIT
--- ---
<p align="center"> <p align="center">
<em>Built by <a href="https://github.com/redclawsystems">RedClaw Systems</a></em><br> <em>Built by <a href="https://git.redclaw.dev/quantumclaw">RedClaw Systems</a></em><br>
<em>~92,000 lines of Rust. Zero C dependencies. One file to remember everything.</em> <em>~86,000 lines of Rust. No libhdf5. One file to remember everything.</em>
</p> </p>
docs/QUICKSTART.md
+1 -1
View File
@@ -567,4 +567,4 @@ let final_results = confidence::reject_low_confidence(
--- ---
<p align="center"><em>Built by <a href="https://github.com/redclawsystems">RedClaw Systems</a></em></p> <p align="center"><em>Built by <a href="https://git.redclaw.dev/quantumclaw">RedClaw Systems</a></em></p>
docs/USE_CASES.md
+1 -1
View File
@@ -232,4 +232,4 @@ clawhdf5-agent = { version = "2.0", features = ["agent", "float16", "accelerate"
--- ---
<p align="center"><em>Built by <a href="https://github.com/redclawsystems">RedClaw Systems</a></em></p> <p align="center"><em>Built by <a href="https://git.redclaw.dev/quantumclaw">RedClaw Systems</a></em></p>