From e9b37a9602c8911fcd78c332f65efd9fd1077e0d Mon Sep 17 00:00:00 2001 From: osobh Date: Wed, 23 Sep 2026 11:05:10 -0500 Subject: [PATCH] docs: bring the README up to date with the last five releases The README had fallen behind v2.3.0-v2.7.0, and parts of it were not true. Checked every claim against the code and BENCHMARKS.md: - Three of the six Quick Start snippets no longer compiled (Agent Memory, Consolidation, OpenClaw); all six now do. - Hybrid search was described as RRF throughout. The default has been weighted 0.4/0.6 fusion since v2.5.0; re-ranking and confidence rejection run only in the OpenClaw backend. - The `float16` feature does not halve embedding storage (the store always writes f32), `--features agent` enables nothing, "Source Isolation" is not wired in, and nothing backs "billion-scale" IVF-PQ. - "Cryptographically verifiable" overstated an unkeyed, session-scoped FNV-1a ledger; "Zero C dependencies" was false while zlib-ng was the default deflate backend. - Stale numbers: tests (1,650 -> 1,868), Rust badge (1.75 is below edition 2024's floor), 6.5 KB/record on disk (BENCHMARKS.md: 1.7 KB), consolidation and hybrid-search latency, and a feature-flag table broken by a paragraph pasted into it. - The file schema, module table and crate map now match the code. Adds a "What's new (v2.2 -> v2.7)" section for collaborators, leading with the silent Extensible Array read bug fixed in v2.7.0. Footer links point at git.redclaw.dev. CLAUDE.md: clawhdf5-migrate is the SQLite migration tool, and MemoryConfig::compression is off by default. Co-Authored-By: Claude Opus 5.5 (1M context) --- CLAUDE.md | 6 +- README.md | 403 +++++++++++++++++++++++++++++---------------- docs/QUICKSTART.md | 2 +- docs/USE_CASES.md | 2 +- 4 files changed, 268 insertions(+), 145 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 8a93702..6165035 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,7 +19,7 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F | `clawhdf5-agent` | Agent memory, session history, knowledge graph storage | | `clawhdf5-gpu` | GPU-accelerated I/O via wgpu (hand-written WGSL compute shaders) | | `clawhdf5-accel` | CPU SIMD acceleration path | -| `clawhdf5-migrate` | Schema migration engine | +| `clawhdf5-migrate` | SQLite → HDF5 agent-memory migration | | `clawhdf5-android` | Android JNI bindings | | `clawhdf5-cli` | Command-line interface | | `clawhdf5-napi` | Node.js native addon bindings | @@ -82,8 +82,8 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F `export` do). An unreadable WAL (torn header, bad magic) is quarantined to `.h5.wal.corrupt-` rather than blocking `open()`; a WAL with an unknown *newer* version still fails and is left untouched. -- `MemoryConfig::compression` uses deflate by default; enable the agent's - `zstd` feature to compress embeddings with Zstd instead (links libzstd). +- `MemoryConfig::compression` is off by default; when on, embeddings are + deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd). - `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by default) recomputes a dataset's SHA-256 and compares it against the `_provenance_sha256` attribute written automatically on save when diff --git a/README.md b/README.md index e235648..990b1c3 100644 --- a/README.md +++ b/README.md @@ -1,26 +1,85 @@ # ClawhDF5 -**The memory layer AI agents deserve. One file. Pure Rust. Zero C dependencies.** +**The memory layer AI agents deserve. One file. Pure Rust. No libhdf5.** [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) -[![Rust](https://img.shields.io/badge/rust-1.75%2B-orange.svg)](https://www.rust-lang.org) -[![Tests](https://img.shields.io/badge/tests-1650%2B%20passing-brightgreen.svg)](#performance) -[![LongMemEval](https://img.shields.io/badge/LongMemEval%20oracle-Turn--Level%20Hit@5%2084%25%20BM25--only-blue.svg)](BENCHMARKS.md#longmemeval-results) -[![Footprint](https://img.shields.io/badge/footprint-6.5%20KB%2Frecord-lightgrey.svg)](BENCHMARKS.md#memory-footprint) +[![Rust](https://img.shields.io/badge/rust-1.94%2B-orange.svg)](https://www.rust-lang.org) +[![Tests](https://img.shields.io/badge/tests-1850%2B-brightgreen.svg)](#building) +[![LongMemEval](https://img.shields.io/badge/LongMemEval__s-Turn--Level%20Hit@5%2081.4%25%20hybrid-blue.svg)](BENCHMARKS.md#longmemeval-results) +[![Footprint](https://img.shields.io/badge/on--disk-1.7%20KB%2Frecord-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1) -ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, cryptographically verifiable memory — all stored in a single portable file. +ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file. > **Two things live here:** -> - **A general-purpose, pure-Rust HDF5 library** — zero C dependencies, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers. +> - **A general-purpose, pure-Rust HDF5 library** — no libhdf5, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers. > - **An agent memory layer built on top of it** — vector search, knowledge graph, hippocampal-style consolidation, in `clawhdf5-agent`. -``` -cargo add clawhdf5 # core HDF5 read/write, no agent layer -cargo add clawhdf5-agent --features agent # + agent memory layer +The crates are not on crates.io yet, so depend on them from git: + +```toml +[dependencies] +clawhdf5 = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } # core HDF5 read/write +clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } # + agent memory layer ``` +> **C dependencies, precisely:** the HDF5 format code is pure Rust and never +> links libhdf5. The default deflate backend is zlib-ng (`fast-deflate`), a C +> library built from source, so a default build needs `cmake` and a C +> compiler. Opt-in codecs (`zstd`, `szip`) and BLAS backends link C too. + > **New here?** Start with the **[Quickstart Guide](docs/QUICKSTART.md)** · See **[Use Cases](docs/USE_CASES.md)** · Read **[Benchmarks](BENCHMARKS.md)** +## What's new (v2.2 → v2.7, and unreleased) + +Five releases in September 2026. Details, including upgrade notes and every +breaking change, are in [CHANGELOG.md](CHANGELOG.md). + +**HDF5 correctness (read these if you read files with an earlier release)** +- **Extensible Array chunk indexes returned wrong data** past the 36th chunk — + any dataset with one unlimited dimension. Silent: plausible numbers from the + wrong chunks. Fixed in v2.7.0; re-read affected data. +- Fixed and Extensible Array checksums are now verified, so a corrupt chunk + index is `ChecksumMismatch` instead of wrong data (v2.7.0). +- Compound datatypes written with default libver bounds (plain + `h5py.File(path, 'w')`) were mis-parsed; HDF5 2.0 compound v5 and native + complex (class 11) types now parse (v2.2.0–v2.3.0). +- Committed datatypes, fill values, soft links and `H5T_STD_REF` references now + read correctly; external links and external raw data are explicit errors; + `attrs()` no longer silently drops attributes (v2.3.0–v2.5.0). +- Datasets indexed by a version-2 B-tree now read (v2.5.0). + +**Security and robustness** +- A crafted file could abort any reader via B-tree v2 recursion or explode it + via shared children; both are now fast errors (v2.7.0). +- Virtual-dataset source paths are confined to the file's directory; chunked + reads use overflow-checked sizes and fallible allocation, and the facade + writes files atomically (v2.3.0). +- Agent store: single-writer lock plus `open_read_only`; a crash between + checkpoint and WAL truncate no longer duplicates entries; unreadable WALs are + quarantined instead of blocking `open()` (v2.3.0). + +**Search quality and speed** +- HNSW neighbour selection now uses the paper's diversity heuristic: recall@10 + at 100K went from 0.31 to 0.98 (v2.4.0). +- `hybrid_search` is 79–190× faster than v2.3.0 (p50 0.07 ms at 1K, 4.65 ms at + 100K). It no longer rebuilds BM25 or rewrites the store per query, and the + HNSW graph is persisted (v2.4.0). +- Default fusion weights are now the measured 0.4 / 0.6 (v2.5.0). Re-ranking had + been discarding the retrieval score, costing the OpenClaw backend 40.6pp of + Hit@1; fixed in v2.6.0. +- Selection reads decode only the chunks they touch (a 64×64 window: 105 ms to + 0.39 ms), and full reads are 1.2–1.9× faster (v2.5.0). + +**Memory** +- A loaded store holds ~30% less (embeddings stored once, v2.6.0), and the + int8 HNSW index, **on by default for new stores** (unreleased), brings a + 100K × 384 store to 1.74× the raw vectors. At equal recall it is also faster + than `f32`: 1.63× QPS on AVX2, 1.18× on a Raspberry Pi 5 (NEON `SDOT`). + +**Tooling** +- CI now runs the h5py/netCDF4 interop suites for real (they had been skipping + silently) and runs an aarch64 job for the NEON kernels. + --- ## Why ClawhDF5? @@ -35,7 +94,7 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d | Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers | | Temporal queries | Custom code | Native temporal index (716ns) | | Multi-modal | Multiple stores | Unified cross-modal search | -| Security | Hope for the best | Provenance tracking + anomaly detection | +| Integrity | Hope for the best | Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance | | Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** | --- @@ -60,6 +119,20 @@ Figures below are from an independent reproduction run on a second machine (AMD ### Vector Search +**HNSW (the default backend for `hybrid_search`)** — `search_harness`, clustered +384-dim data, M = 16, ef_construction = 64, recall measured against an exact scan. +See [BENCHMARKS.md § Search harness](BENCHMARKS.md#search-harness-baseline-v230) +and [§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index): + +| N = 100K, ef = 64 | recall@10 | QPS | build | +|---|---:|---:|---:| +| `f32` index | 0.9945 | 13 399 | 3.2 s | +| `i8` index + exact re-score (**default for new stores**) | 0.9940 | **21 848** | **1.8 s** | + +Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31. + +**Brute-force and IVF paths** (Criterion, i7-12650H): + | Scale | Flat | IVF (nprobe=10) | IVF-PQ | vs MemX¹ | |-------|------|-----------------|--------|----------| | 1K | **54 µs** | — | — | — | @@ -75,7 +148,7 @@ Figures below are from an independent reproduction run on a second machine (AMD | Operation | Latency | Scale | |-----------|---------|-------| -| Hybrid search (RRF) | **222 µs** | 1K records | +| Hybrid search (`HDF5Memory::hybrid_search`, p50) | **70 µs** / 0.49 ms / 4.65 ms | 1K / 10K / 100K records | | BM25 keyword search | **67 µs** | 1K records | | Knowledge graph BFS | **24 µs** | 1K entities | | Spreading activation | **17 µs** | 100 entities | @@ -115,13 +188,17 @@ declaration: Hybrid is the strongest configuration, which is what running two retrieval stages is for. The weights matter more than the stages: a sweep of `vector_weight` from -0.0 to 1.0 found the long-standing `0.7/0.3` default is **strictly dominated** by -`0.4/0.6` — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Use -`0.4/0.6`, or `0.3/0.7` if rank-1 precision matters most. See -[BENCHMARKS.md § Weight sweep](BENCHMARKS.md#longmemeval-results). +0.0 to 1.0 found the old `0.7/0.3` default is **strictly dominated** by +`0.4/0.6` — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Since +v2.5.0 `0.4/0.6` is the default (`hybrid::DEFAULT_FUSION`, used by +`unified_search`, `hybrid_search_with` and the OpenClaw backend); callers that +pass weights to `hybrid_search` explicitly choose their own. Use `0.3/0.7` if +rank-1 precision matters most. Reciprocal rank fusion is selectable +(`hybrid::Fusion::Rrf`) but measured worse than the weighted sum. See +[BENCHMARKS.md § Weight sweep](BENCHMARKS.md#weight-sweep--full-haystack-n500). -Vector embeddings require `--features embeddings`; without it the vector stage is -inert and only the BM25 row is produced, which is what every previously published +The benchmark's vector stage requires `clawhdf5-bench`'s `embeddings` feature +(real MiniLM embeddings); without it the vector stage is inert and only the BM25 row is produced, which is what every previously published number here measured. On the easier `longmemeval_oracle` variant (evidence sessions only) the same @@ -146,19 +223,37 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points. ### Memory Footprint -| Records | File Size | Bytes/Record | With Compression | -|---------|-----------|--------------|------------------| -| 1K | ~6.5 MB | ~6.5 KB | ~2.1 MB (3.1x) | -| 10K | ~65 MB | ~6.5 KB | ~21 MB (3.1x) | -| 100K | ~645 MB | ~6.5 KB | ~208 MB (3.1x) | +**On disk** — 384-dim embeddings, 200-char text +([BENCHMARKS.md § Memory Footprint](BENCHMARKS.md#memory-footprint-1)): + +| Records | File Size | Bytes/Record | Gzip-6 compressed | +|---------|-----------|--------------|-------------------| +| 1K | 1.7 MB | 1.8 KB | 277 KB (6.1x) | +| 10K | 17.0 MB | 1.7 KB | 2.7 MB (6.2x) | +| 100K | 169.8 MB | 1.7 KB | 26.9 MB (6.2x) | + +**In memory** — a store reopened from disk, 384-dim `f32`, measured with a +counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-footprint)): + +| Records | Raw vectors | Reopened, `f32` index | Reopened, `i8` index (default) | +|---------|-------------|-----------------------|--------------------------------| +| 1K | 1 MiB | 4 MiB (2.40x) | 2 MiB (1.64x) | +| 10K | 15 MiB | 44 MiB (3.03x) | 27 MiB (1.81x) | +| 100K | 146 MiB | 399 MiB (2.72x) | **256 MiB (1.74x)** | + +Down from 505 MiB (3.44x) at 100K before v2.6.0, when the cache held every +embedding twice. ### Consolidation Efficiency +1,000 records (10 signal + 990 noise), `working_capacity = 100` +([BENCHMARKS.md § Consolidation Efficiency](BENCHMARKS.md#consolidation-efficiency)): + | Metric | Before | After | Delta | |--------|--------|-------|-------| -| Records in store | 1,000 | ~110 | −89% | -| Hit@1 recall | ~60% | ~90% | +30% | -| Search latency | ~2.8 ms | ~0.3 ms | **9x faster** | +| Records in store | 1,000 | 100 | −90% | +| Hit@1 recall (signal records) | 100% | 100% | no loss | +| Search latency | 2.75 ms | 0.31 ms | **8.8x faster** | **Full benchmark details: [BENCHMARKS.md](BENCHMARKS.md)** @@ -166,74 +261,71 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points. ## Agent Memory Architecture -ClawhDF5's agent memory engine implements research from 15+ recent papers on agentic memory systems. It's not a toy — it's the real thing. +ClawhDF5's agent memory engine draws on 15+ recent papers on agentic memory systems (see [Research Foundation](#research-foundation)). ``` - ┌─────────────────┐ - │ Agent Query │ - └────────┬────────┘ - │ - ┌────────────▼────────────┐ - │ Hybrid Retrieval │ - │ Vector + BM25 + RRF │ - └────────────┬────────────┘ - │ - ┌──────────────────▼──────────────────┐ - │ Multi-Factor Re-Ranking │ - │ temporal · authority · activation │ - └──────────────────┬──────────────────┘ - │ - ┌────────────▼────────────┐ - │ Confidence Rejection │ - │ (suppress bad matches) │ - └────────────┬────────────┘ - │ - ┌────────────────────────▼────────────────────────┐ - │ Memory Store (HDF5) │ - │ │ - │ ┌───────────┐ ┌───────────┐ ┌───────────────┐ │ - │ │ Working │→│ Episodic │→│ Semantic │ │ - │ │ (bounded) │ │ (bounded) │ │ (long-term) │ │ - │ └───────────┘ └───────────┘ └───────────────┘ │ - │ │ - │ ┌──────────┐ ┌──────────┐ ┌────────────────┐ │ - │ │Knowledge │ │Temporal │ │ Multi-Modal │ │ - │ │ Graph │ │ Index │ │ Embeddings │ │ - │ └──────────┘ └──────────┘ └────────────────┘ │ - │ │ - │ ┌──────────┐ ┌──────────┐ ┌────────────────┐ │ - │ │Provenance│ │ Anomaly │ │ Source │ │ - │ │ Tracking │ │Detection │ │ Isolation │ │ - │ └──────────┘ └──────────┘ └────────────────┘ │ - └─────────────────────────────────────────────────┘ - │ - ┌────────┴────────┐ - │ agent_memory.h5 │ - │ single file │ - └─────────────────┘ + ┌─────────────────┐ + │ Agent Query │ + └────────┬────────┘ + │ + ┌─────────────────▼──────────────────┐ + │ HDF5Memory::hybrid_search │ + │ HNSW vector + BM25 keyword │ + │ weighted fusion (0.4 / 0.6) │ + │ × √(Hebbian activation) │ + └─────────────────┬──────────────────┘ + │ OpenClaw backend adds: + ┌─────────────────▼──────────────────┐ + │ Multi-factor re-ranking │ + │ relevance · recency · authority · │ + │ activation │ + ├────────────────────────────────────┤ + │ Confidence rejection │ + │ (suppress bad matches) │ + └─────────────────┬──────────────────┘ + │ + ┌────────────────────────────▼────────────────────────────┐ + │ In memory │ + │ cache (flat f32 embeddings) · BM25 index · HNSW index │ + │ provenance ledger + anomaly alerts (session-scoped) │ + └────────────────────────────┬────────────────────────────┘ + │ WAL append; checkpoint + ┌────────────────────────────▼────────────────────────────┐ + │ agent_memory.h5 /meta · /memory · /sessions · │ + │ /knowledge_graph │ + │ agent_memory.h5.wal chained-CRC write-ahead log │ + │ agent_memory.h5.ann HNSW graph (derived, rebuildable) │ + │ agent_memory.h5.lock single-writer lock │ + └─────────────────────────────────────────────────────────┘ ``` +Consolidation tiers (Working → Episodic → Semantic), the knowledge-graph +algorithms, temporal and multi-modal indexes are library components you drive +directly; the store persists the records, sessions and graph they work over. + ### Module Overview | Module | What It Does | |--------|-------------| -| **`knowledge`** | Entity/relation graph with BFS traversal, spreading activation, fuzzy entity resolution | -| **`consolidation`** | Three-tier memory (Working → Episodic → Semantic) with importance scoring and time-decay | -| **`hybrid`** | Vector + BM25 fusion with Reciprocal Rank Fusion (RRF, k=60). The vector stage uses the HNSW index by default (`hnsw` feature, on by default); disable with `--no-default-features --features float16` for an exact linear scan | -| **`reranker`** | Multi-factor re-ranking: temporal recency, source authority, activation weight | -| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches | +| **`knowledge`** | Entity/relation graph with BFS traversal, spreading activation, fuzzy (Levenshtein) entity resolution | +| **`consolidation`** | Three-tier memory (Working → Episodic → Semantic) with importance scoring, novelty, and time-decay | +| **`hybrid`** | Vector + BM25 fusion. Default is a min-max-normalised weighted sum, vector 0.4 / keyword 0.6 (`hybrid::DEFAULT_FUSION`, tuned on LongMemEval); RRF is available via `Fusion::Rrf` / `hybrid_search_with`. The vector stage uses the HNSW index by default (`hnsw` feature); disable with `--no-default-features --features float16` for an exact linear scan | +| **`reranker`** | Multi-factor re-ranking: retrieval relevance (leads, weight 1.0), temporal recency, source authority, activation weight. Used by the OpenClaw backend | +| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches (OpenClaw backend) | | **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints | | **`multimodal`** | Cross-modal search across text/image/audio/video embeddings | -| **`provenance`** | Source attribution, FNV-1a content hashing, integrity verification | -| **`anomaly`** | Write rate limiting, 15 injection pattern detectors, source distribution analysis | +| **`provenance`** | Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof) | +| **`anomaly`** | Write rate limiting, 15 injection-pattern detectors, source-distribution analysis. Alerts never block a save; drain them with `take_anomaly_alerts` | | **`openclaw`** | OpenClaw integration: MemoryBackend trait, Markdown ↔ HDF5 conversion | | **`vector_search`** | Flat cosine, pre-normed, SIMD, BLAS, GPU, parallel search paths | -| **`ivf` / `pq`** | IVF-PQ approximate nearest neighbor for billion-scale search | -| **`bm25`** | BM25 keyword index with TF-IDF scoring | +| **`ivf` / `pq`** | Standalone IVF and IVF-PQ indexes (benchmarked to 100K vectors); not used by `HDF5Memory`, whose ANN index is HNSW | +| **`bm25`** | Incremental Okapi BM25 inverted index, kept for the life of the store; optional stemming | +| **`query_expand`** | Synonym / acronym / temporal query expansion | | **`entity_extract`** | Rule-based entity extraction from text chunks into the knowledge graph | -| **`wal`** | Write-ahead log for crash-safe persistence; each entry is CRC32-checked on replay, so a corrupted entry stops replay there instead of loading bad data | +| **`wal`** | Write-ahead log (v4) with a chained CRC32 per entry, so a corrupted, reordered, duplicated or spliced entry stops replay; checkpoints record a WAL mark so nothing is applied twice. Appends are not fsynced | | **`memory_strategy`** | Pluggable strategies: save-every, semantic-shift, user-correction detection | | **`decision_gate`** | Sub-microsecond trivial/substantive classification | +| **`ephemeral`** | In-memory TTL/LFU working tier | | **`async_memory`** | Tokio-based async wrapper over the memory store (`async` feature) | --- @@ -265,7 +357,7 @@ assert_eq!(values, vec![22.5, 23.1, 21.8]); use clawhdf5_agent::{HDF5Memory, MemoryConfig, MemoryEntry, AgentMemory}; // Create memory store -let config = MemoryConfig::new("agent.h5", "my-agent", 384); +let config = MemoryConfig::new("agent.h5".into(), "my-agent", 384); let mut memory = HDF5Memory::create(config)?; // Save a memory @@ -278,8 +370,8 @@ memory.save(MemoryEntry { tags: "preference".into(), })?; -// Search -let results = memory.search(&query_embedding, 5)?; +// Hybrid search: vector + BM25, weighted 0.4 / 0.6 (the measured default) +let results = memory.hybrid_search(&query_embedding, "user preferences", 0.4, 0.6, 5); for result in results { println!("[{:.3}] {}", result.score, result.chunk); } @@ -309,8 +401,8 @@ let neighbors = kg.bfs_neighbors(alice, 2); // 2-hop neighborhood let activated = kg.spreading_activation(&[alice], 0.5, 0.01, 5); // Entity resolution — fuzzy matching -let resolved = kg.resolve_or_create("alice", "person", -1, 2); -// Returns existing Alice entity (Levenshtein distance ≤ 2) +let (id, created) = kg.resolve_or_create("alice", "person", -1, 2); +// id == alice, created == false: matched the existing entity (Levenshtein distance ≤ 2) ``` ### Memory Consolidation @@ -321,15 +413,19 @@ use clawhdf5_agent::consolidation::*; let config = ConsolidationConfig::default(); let mut engine = ConsolidationEngine::new(config); -// Add memories — automatically scored for importance -engine.add_memory("User prefers dark mode", vec![0.1, 0.2, ...], MemorySource::User); -engine.add_memory("ok", vec![0.0, 0.0, ...], MemorySource::System); +let now = 1_700_000_000.0; // seconds since the epoch + +// Add memories — automatically scored for importance. +// Elevated sources (System, …) go through a separate, explicit API. +let id = engine.add_memory("User prefers dark mode".into(), vec![0.1, 0.2, ...], UntrustedSource::User, now); +engine.add_trusted_memory("ok".into(), vec![0.0, 0.0, ...], TrustedSource::System, now); // Access a memory (reactivates it) -engine.access_memory(0); +engine.access_memory(id, now); // Run consolidation cycle -let stats = engine.consolidate(); +engine.consolidate(now); +let stats = engine.get_stats(); // Working memories promote to Episodic (if important enough) // Episodic memories promote to Semantic (if accessed enough) // Low-decay memories get evicted when tiers are full @@ -357,13 +453,13 @@ let recent = index.latest(10); use clawhdf5_agent::openclaw::*; // Create backend -let mut backend = ClawhdfBackend::create("memory.h5", "agent-1", 384)?; +let mut backend = ClawhdfBackend::create(std::path::Path::new("memory.h5"), 384)?; // Ingest existing Markdown memory files let md = std::fs::read_to_string("MEMORY.md")?; let count = backend.ingest_markdown("MEMORY.md", &md)?; -// Search (uses full pipeline: RRF → re-rank → confidence filter) +// Search (full pipeline: weighted vector + BM25 fusion → re-rank → confidence filter) let results = backend.search("user preferences", &query_embedding, 5); // Export back to Markdown @@ -375,22 +471,23 @@ let exported = backend.export_markdown("MEMORY.md")?; ## Crate Map ``` -clawhdf5 workspace (16 crates, ~92K lines of Rust; plus libaec-sys, an - internal FFI bindings crate for the optional szip feature) +clawhdf5 workspace (16 crates, ~86K lines of Rust in src/, ~104K with tests + and benches; plus libaec-sys, an internal FFI bindings + crate for the optional szip feature) │ ├── Core HDF5 -│ ├── clawhdf5-format — Binary parser/writer (no_std), shared type definitions -│ ├── clawhdf5-io — I/O abstraction (buffered, mmap, async) +│ ├── clawhdf5-format — Binary parser/writer (no_std-capable), shared type definitions +│ ├── clawhdf5-io — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI) │ ├── clawhdf5-filters — Fast deflate path (zlib-ng); lz4/zstd/pcodec/szip filters live in clawhdf5-format │ ├── clawhdf5-derive — Proc macros │ ├── clawhdf5 — High-level API │ ├── clawhdf5-netcdf4 — NetCDF-4 support -│ ├── clawhdf5-accel — SIMD (NEON, AVX2, AVX-512) +│ ├── clawhdf5-accel — SIMD (AVX2, NEON incl. SDOT int8; AVX-512 behind `avx512`) │ └── clawhdf5-gpu — GPU compute (wgpu, hand-written WGSL compute shaders) │ ├── Agent Memory -│ ├── clawhdf5-agent — Memory engine (20.9K lines, 32 modules; WAL is CRC32-checked per entry) -│ ├── clawhdf5-ann — HNSW approximate nearest neighbor (default backend; optional `parallel` feature) +│ ├── clawhdf5-agent — Memory engine (24.7K lines, 32 modules; chained-CRC WAL) +│ ├── clawhdf5-ann — HNSW approximate nearest neighbor (default backend; f32 or int8 storage; `parallel` build) │ ├── clawhdf5-migrate — SQLite → HDF5 migration │ ├── clawhdf5-android — Android JNI bridge │ └── clawhdf5-cli — CLI tool @@ -411,10 +508,10 @@ ClawhDF5's agent memory design draws from 15+ recent papers: | Paper | Key Insight | ClawhDF5 Module | |-------|-------------|-----------------| -| **MemX** (2026) | RRF + multi-factor re-ranking | `hybrid`, `reranker` | -| **Graph-Native Cognitive Memory** (2026) | Graph-structured belief revision | `knowledge` | +| **MemX** (2026) | Hybrid fusion + multi-factor re-ranking | `hybrid`, `reranker` | +| **Graph-Native Cognitive Memory** (2026) | Graph-structured memory (weighted, timestamped relations; entity timelines) | `knowledge`, `temporal` | | **CraniMem** (2026) | Bounded hippocampal memory | `consolidation` | -| **D-MEM** (2026) | Reward prediction error gating | `consolidation` | +| **D-MEM** (2026) | Surprise-gated storage (implemented as a novelty score) | `consolidation` | | **SYNAPSE** (2025) | Spreading activation for recall | `knowledge` | | **RAGdb** (2025) | Zero-dependency edge RAG | Architecture | | **MemoryGraft** (2025) | Memory poisoning attacks | `anomaly`, `provenance` | @@ -429,29 +526,35 @@ ClawhDF5's agent memory design draws from 15+ recent papers: | Flag | Default | Description | |------|---------|-------------| -| `agent` | no | Full agent memory layer | -| `float16` | **yes** | Half-precision embedding storage (2× compression) | +| `float16` | **yes** | Half-precision cosine kernel (`cosine_similarity_f16`). The store itself always writes `f32` embeddings; `MemoryConfig::float16` is recorded in `/meta` but not yet applied | | `hnsw` | **yes** | HNSW approximate vector index for `hybrid_search` (via `clawhdf5-ann`); disable for an exact linear scan | +| `parallel` | **yes** | Parallel HNSW bulk build (same graph, ~3× faster on 16 cores) and Rayon brute-force search strategies | +| `zstd` | no | Compress embeddings with Zstd instead of deflate when `MemoryConfig::compression` is on (links libzstd) | +| `fast-math` | no | BLAS matrix-vector multiply | +| `accelerate` | no | Apple Accelerate / AMX (macOS) | +| `openblas` | no | OpenBLAS (Linux) | +| `gpu` | no | GPU search via wgpu | +| `async` | no | Tokio async with background flush | +| `agent` | no | Reserved; currently enables nothing (the agent layer is always built) | + +To opt out of the parallel build: `--no-default-features --features float16,hnsw`. +For an exact linear cosine scan instead of HNSW: `--no-default-features --features float16`. `MemoryConfig::hnsw_m`, `hnsw_ef_construction` and `hnsw_ef_search` tune the vector index (16 / 64 / scale-with-`k` by default) and are stored with the file. `MemoryConfig::quantized_index` (**on by default** for new stores) holds the -HNSW index's own -copy of the embeddings as `i8`, roughly halving a loaded store's memory -(2.72x -> 1.74x the raw vectors at 100k x 384). Quantised distances are -approximate, so the query path re-scores the candidate pool against the exact -embeddings the store already holds, which keeps recall at the `f32` index's -level. It is also **faster**: 1.63x the queries per second at equal recall on -x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with index builds -1.8x and 2.3x faster respectively. See `BENCHMARKS.md`, "Quantising the index copy". -| `parallel` | no | Rayon parallel search | -| `fast-math` | no | BLAS matrix-vector multiply | -| `accelerate` | no | Apple Accelerate / AMX (macOS) | -| `openblas` | no | OpenBLAS (Linux) | -| `gpu` | no | GPU search via wgpu | -| `async` | no | Tokio async with background flush | +HNSW index's own copy of the embeddings as `i8`, roughly halving a loaded +store's memory (2.72x -> 1.74x the raw vectors at 100k x 384). Quantised +distances are approximate, so the query path re-scores the candidate pool +against the exact embeddings the store already holds, which keeps recall at the +`f32` index's level. It is also **faster**: 1.63x the queries per second at +equal recall on x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with +index builds 1.8x and 2.3x faster respectively. Stores created before the +setting existed keep their `f32` index; opt out for new stores with +`quantized_index = false` or `clawhdf5-cli create --f32-index`. See +[BENCHMARKS.md § Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index). ### `clawhdf5-format` @@ -461,8 +564,8 @@ x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with index builds | `deflate` | yes | Deflate compression | | `checksum` | yes | Jenkins lookup3 verification | | `provenance` | yes | SHA-256 provenance attributes | -| `fast-deflate` | **yes** | zlib-ng backend for faster deflate | -| `system-zlib-decompress` | **yes** | Use the system zlib for decompression where available | +| `fast-deflate` | **yes** | zlib-ng backend for faster deflate (C; needs `cmake`) | +| `system-zlib-decompress` | **yes** | Use Apple's system libz for decompression (macOS only; no effect elsewhere) | | `parallel` | no | Parallel chunk encoding + compression (rayon) | | `fast-checksum` | no | crc32fast-accelerated checksums | | `lz4` | no | LZ4 block compression filter (id 32004) | @@ -470,17 +573,21 @@ x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with index builds | `pcodec` | no | Pcodec lossless numerical codec (id 32023, via `pco` crate) | | `system-zlib` / `zlib-rs` | no | Alternative zlib backends for deflate | | `blake3_hash` | no | BLAKE3 content hashing for provenance | +| `szip` | no | SZIP filter (id 4) via libaec (C, through the internal `libaec-sys` crate) | ### `clawhdf5-ann` | Flag | Default | Description | |------|---------|-------------| -| `parallel` | no | Rayon-parallel neighbor-distance computation during HNSW graph pruning | +| `parallel` | no | Batched bulk build runs neighbour planning and back-link pruning on a Rayon pool; the graph is identical with or without it (enabled by `clawhdf5-agent`'s default `parallel`) | ### `clawhdf5-io` | Flag | Default | Description | |------|---------|-------------| +| `mmap` | no | Memory-mapped reads (`memmap2`) | +| `async` | no | Tokio-based async I/O | +| `hsds` | no | HSDS (HDF REST service) client | | `mpi-io` | no | MPI-backed I/O via the `mpi` crate | > **Parallel I/O (MPI) limitation:** `mpi-io`'s read path is a root-rank read @@ -494,17 +601,17 @@ x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with index builds ## Building ```bash -# Default +# Default (needs cmake + a C compiler for zlib-ng) cargo build --workspace # Agent memory with all accelerations (Linux) -cargo build -p clawhdf5-agent --features "agent,float16,parallel,fast-math" +cargo build -p clawhdf5-agent --features fast-math # Agent memory with Apple Accelerate (macOS) -cargo build -p clawhdf5-agent --features "agent,float16,accelerate,parallel,gpu" +cargo build -p clawhdf5-agent --features "accelerate,gpu" # Tests -cargo test --workspace # all 1,650+ tests +cargo test --workspace # all 1,850+ tests cargo test -p clawhdf5-agent # agent memory tests scripts/ci-test.sh # what CI runs: fmt, clippy matrix, tests, # h5py/netCDF4 interop, no_std @@ -526,25 +633,41 @@ cargo bench -p clawhdf5-bench # h5bench-equivalent I/O suite ``` agent_memory.h5 -├── /meta -│ ├── schema_version: "1.0" -│ ├── agent_id, embedder, embedding_dim -│ └── created_at +├── /meta (attributes) +│ ├── schema_version: "1.0", edgehdf5_version +│ ├── agent_id, embedder, embedding_dim, chunk_size, overlap, created_at +│ ├── float16, compression, compression_level, compact_threshold, +│ │ hebbian_boost, decay_factor, wal_enabled, wal_max_entries +│ ├── quantized_index, hnsw_m, hnsw_ef_construction, hnsw_ef_search +│ ├── wal_applied_len, wal_applied_crc (WAL mark of the last checkpoint) +│ └── ann_generation (ties the .ann sidecar to this checkpoint) ├── /memory -│ ├── chunks: string[N] -│ ├── embeddings: f32[N × D] (or f16 with float16 flag) -│ ├── tombstones: u8[N] -│ └── norms: f32[N] (pre-computed L2) +│ ├── chunks: string[N] +│ ├── embeddings: f32[N × D] (chunked; deflate, or Zstd with the +│ │ `zstd` feature, when compression is on) +│ ├── source_channel: string[N] +│ ├── timestamps: f64[N] +│ ├── session_ids: string[N] +│ ├── tags: string[N] +│ ├── tombstones: u8[N] +│ ├── norms: f32[N] (pre-computed L2) +│ └── activation_weights: f32[N] (Hebbian) ├── /sessions -│ ├── ids: string[S] -│ └── summaries: string[S] +│ ├── ids, channels, summaries: string[S] +│ ├── start_idxs, end_idxs: i64[S] +│ └── timestamps: f64[S] └── /knowledge_graph - ├── entity_names: string[E] - ├── relation_srcs: i64[R] - ├── relation_tgts: i64[R] - └── relation_types: string[R] + ├── entity_ids, entity_emb_idxs: i64[E]; entity_names, entity_types: string[E] + ├── relation_srcs, relation_tgts: i64[R]; relation_types: string[R] + ├── relation_weights: f32[R]; relation_ts: f64[R] + └── alias_strings: string[A]; alias_entity_ids: i64[A] (when aliases exist) ``` +Alongside the store: `.h5.wal` (write-ahead log), `.h5.ann` +(HNSW graph; derived, safe to delete) and `.h5.lock` (single-writer +lock). A second writer gets `MemoryError::Locked`; use +`HDF5Memory::open_read_only` for a lock-free point-in-time view. + --- ## Migration @@ -599,6 +722,6 @@ MIT ---

- Built by RedClaw Systems
- ~92,000 lines of Rust. Zero C dependencies. One file to remember everything. + Built by RedClaw Systems
+ ~86,000 lines of Rust. No libhdf5. One file to remember everything.

diff --git a/docs/QUICKSTART.md b/docs/QUICKSTART.md index c751768..0635ce0 100644 --- a/docs/QUICKSTART.md +++ b/docs/QUICKSTART.md @@ -567,4 +567,4 @@ let final_results = confidence::reject_low_confidence( --- -

Built by RedClaw Systems

+

Built by RedClaw Systems

diff --git a/docs/USE_CASES.md b/docs/USE_CASES.md index 1ac85a9..4d0cb4a 100644 --- a/docs/USE_CASES.md +++ b/docs/USE_CASES.md @@ -232,4 +232,4 @@ clawhdf5-agent = { version = "2.0", features = ["agent", "float16", "accelerate" --- -

Built by RedClaw Systems

+

Built by RedClaw Systems