# ClawhDF5 **The memory layer AI agents deserve. One file. Pure Rust. Zero C dependencies.** [](LICENSE) [](https://www.rust-lang.org) [](#performance) [](BENCHMARKS.md#longmemeval-results) [](BENCHMARKS.md#memory-footprint) ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, cryptographically verifiable memory — all stored in a single portable file. > **Two things live here:** > - **A general-purpose, pure-Rust HDF5 library** — zero C dependencies, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers. > - **An agent memory layer built on top of it** — vector search, knowledge graph, hippocampal-style consolidation, in `clawhdf5-agent`. ``` cargo add clawhdf5 # core HDF5 read/write, no agent layer cargo add clawhdf5-agent --features agent # + agent memory layer ``` > **New here?** Start with the **[Quickstart Guide](docs/QUICKSTART.md)** · See **[Use Cases](docs/USE_CASES.md)** · Read **[Benchmarks](BENCHMARKS.md)** --- ## Why ClawhDF5? Every AI agent needs memory. Today that means scattered Markdown files, SQLite databases, cloud-hosted vector stores, and glue code. ClawhDF5 replaces all of it: | Problem | Status Quo | ClawhDF5 | |---------|-----------|----------| | Vector search | External DB (Pinecone, Qdrant) | Built-in, sub-millisecond | | Keyword search | Separate FTS engine | Integrated BM25 | | Knowledge graph | Neo4j or none | In-file graph with spreading activation | | Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers | | Temporal queries | Custom code | Native temporal index (716ns) | | Multi-modal | Multiple stores | Unified cross-modal search | | Security | Hope for the best | Provenance tracking + anomaly detection | | Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** | --- ## Performance Vector search and agent-memory operations below are benchmarked on Intel i7-12650H (10C/16T), 384-dim embeddings, Criterion.rs. The HDF5 Core I/O table immediately below is from a separate, independently reproduced run (see its own hardware note). ### HDF5 Core I/O (vs libhdf5 1.14.6) *Benchmark numbers are being validated in collaboration with engineers from the HDF5 Group to confirm methodology and reproducibility.* Figures below are from an independent reproduction run on a second machine (AMD Ryzen 7 7800X3D, 2026-08-03). Full methodology, the original i7-12650H run, and two additional benchmarks added to close prior coverage gaps (an I/O-inclusive metadata-open comparison and an honest zero-copy-mmap measurement) are in [BENCHMARKS.md § Independent Validation](BENCHMARKS.md#independent-validation-tank-ryzen-7-7800x3d-2026-08-03). | Operation | ClawhDF5 | libhdf5 | Speedup | |-----------|----------|---------|---------| | Attribute write (128 attrs) | 85.2 µs | 877 µs | **10.3×** | | Group create (64 groups) | 130 µs | 1.37 ms | **10.6×** | | Chunked write, deflate-6 (512×512 f32) | 1.44 ms | 65.0 ms | **45.3×** | | Sequential read (100K f32) | 23.3 µs | 63.6 µs | **2.7×** | | Sequential write (100K f32) | 210 µs | 189 µs | **≈ tie** | ### Vector Search | Scale | Flat | IVF (nprobe=10) | IVF-PQ | vs MemX¹ | |-------|------|-----------------|--------|----------| | 1K | **54 µs** | — | — | — | | 10K | 753 µs | **27 µs** | — | — | | 100K | 11.4 ms | 1.32 ms | **1.19 ms** | ~8–76× (see caveat) | > Reproduced on the same second machine (Ryzen 7 7800X3D) with a corrected, > apples-to-apples SIMD/scalar/parallel comparison methodology — see > [BENCHMARKS.md § Independent Validation: tank — LongMemEval & Vector > Search](BENCHMARKS.md#independent-validation-tank--longmemeval--vector-search-ryzen-7-7800x3d-2026-08-05). ### Agent Memory Operations | Operation | Latency | Scale | |-----------|---------|-------| | Hybrid search (RRF) | **222 µs** | 1K records | | BM25 keyword search | **67 µs** | 1K records | | Knowledge graph BFS | **24 µs** | 1K entities | | Spreading activation | **17 µs** | 100 entities | | Temporal range query | **716 ns** | 10K timestamps | | Consolidation cycle | **164 µs** | 1K records | | Memory write (WAL) | **18 µs** | per record (group-commit append; HDF5 batched at flush) | | Importance gate | **61 ns** | per record | ### Chunked Write Throughput (codec comparison) Measured with Criterion on f32 matrices. Auto-shuffle is applied before all compression codecs by default (AoS→SoA byte transpose, +157–204% throughput for float data): | Codec | 128×128 f32 | 512×512 f32 | Notes | |-------|-------------|-------------|-------| | Zstd level 3 | **148 µs / 422 MiB/s** | **1.34 ms / 748 MiB/s** | With auto-shuffle | | Deflate level 6 | 153 µs / 407 MiB/s | 1.39 ms / 719 MiB/s | With auto-shuffle | | Pcodec | 528 µs / 118 MiB/s | 1.69 ms / 591 MiB/s | Best compression ratio | Use `.with_zstd(3)` or `.with_deflate(6)` for write-heavy workloads — both now perform at ~720–750 MiB/s on large matrices. Use `.with_pcodec()` for write-once/read-many workloads where compression ratio matters more than encode speed. Disable auto-shuffle with `.without_shuffle()` for byte arrays that don't benefit from AoS→SoA transposition. > ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search). The ratio overstates the real advantage by an unquantified margin — order-of-magnitude indication only. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171). ### LongMemEval Retrieval Recall Evaluated against the full **`longmemeval_s`** haystack — all 500 questions, 47.7 sessions and 493.5 turns each, with only 4.0% of haystack sessions being evidence sessions. See [BENCHMARKS.md § LongMemEval Results](BENCHMARKS.md#longmemeval-results) for the full scoring-target declaration: | Mode | Turn-Level Hit@5 | Session-Level Hit@5 | |------|------------------|---------------------| | BM25 only | 75.0% | 93.6% | | Vector only (MiniLM) | 71.8% | 94.2% | | Hybrid (0.4/0.6, tuned) | **81.4%** | **96.8%** | Hybrid is the strongest configuration, which is what running two retrieval stages is for. The weights matter more than the stages: a sweep of `vector_weight` from 0.0 to 1.0 found the long-standing `0.7/0.3` default is **strictly dominated** by `0.4/0.6` — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Use `0.4/0.6`, or `0.3/0.7` if rank-1 precision matters most. See [BENCHMARKS.md § Weight sweep](BENCHMARKS.md#longmemeval-results). Vector embeddings require `--features embeddings`; without it the vector stage is inert and only the BM25 row is produced, which is what every previously published number here measured. On the easier `longmemeval_oracle` variant (evidence sessions only) the same harness scores 84.4% turn-level Hit@5 / MRR 0.6597, reproduced identically on a second machine. The 9.4-point gap is the cost of the real haystack, and is why the full-haystack number is the one quoted here. This is **retrieval recall** (did the gold memory appear in the top-k), not the official LongMemEval QA-accuracy metric — the two are not comparable, and retrieval recall reported as QA accuracy typically overstates by 20–30 points. > **Previously reported here and now retracted:** session-level Hit@5 of 100.0% / > MRR 1.0000, and a claim of beating MemX's 51.6%. Those session-level figures were > degenerate on the oracle variant (any returned document is a hit by > construction); the 93.6% above is a different, real measurement on a corpus where > evidence sessions are 4.0% of the haystack. The MemX comparison stays withdrawn — > MemX measures fact-level granularity over 220,349 records, which running the full > haystack does not fix. Details in > [BENCHMARKS.md](BENCHMARKS.md#retracted-session-level-recall-and-the-memx-comparison). > Enable embeddings via `hybrid_search(query_emb, text, 0.4, 0.6, k)` for substantially higher recall. The vector stage is served by the HNSW index by default (the `hnsw` feature is on by default); build with `--no-default-features --features float16` to fall back to an exact linear cosine scan. ### Memory Footprint | Records | File Size | Bytes/Record | With Compression | |---------|-----------|--------------|------------------| | 1K | ~6.5 MB | ~6.5 KB | ~2.1 MB (3.1x) | | 10K | ~65 MB | ~6.5 KB | ~21 MB (3.1x) | | 100K | ~645 MB | ~6.5 KB | ~208 MB (3.1x) | ### Consolidation Efficiency | Metric | Before | After | Delta | |--------|--------|-------|-------| | Records in store | 1,000 | ~110 | −89% | | Hit@1 recall | ~60% | ~90% | +30% | | Search latency | ~2.8 ms | ~0.3 ms | **9x faster** | **Full benchmark details: [BENCHMARKS.md](BENCHMARKS.md)** --- ## Agent Memory Architecture ClawhDF5's agent memory engine implements research from 15+ recent papers on agentic memory systems. It's not a toy — it's the real thing. ``` ┌─────────────────┐ │ Agent Query │ └────────┬────────┘ │ ┌────────────▼────────────┐ │ Hybrid Retrieval │ │ Vector + BM25 + RRF │ └────────────┬────────────┘ │ ┌──────────────────▼──────────────────┐ │ Multi-Factor Re-Ranking │ │ temporal · authority · activation │ └──────────────────┬──────────────────┘ │ ┌────────────▼────────────┐ │ Confidence Rejection │ │ (suppress bad matches) │ └────────────┬────────────┘ │ ┌────────────────────────▼────────────────────────┐ │ Memory Store (HDF5) │ │ │ │ ┌───────────┐ ┌───────────┐ ┌───────────────┐ │ │ │ Working │→│ Episodic │→│ Semantic │ │ │ │ (bounded) │ │ (bounded) │ │ (long-term) │ │ │ └───────────┘ └───────────┘ └───────────────┘ │ │ │ │ ┌──────────┐ ┌──────────┐ ┌────────────────┐ │ │ │Knowledge │ │Temporal │ │ Multi-Modal │ │ │ │ Graph │ │ Index │ │ Embeddings │ │ │ └──────────┘ └──────────┘ └────────────────┘ │ │ │ │ ┌──────────┐ ┌──────────┐ ┌────────────────┐ │ │ │Provenance│ │ Anomaly │ │ Source │ │ │ │ Tracking │ │Detection │ │ Isolation │ │ │ └──────────┘ └──────────┘ └────────────────┘ │ └─────────────────────────────────────────────────┘ │ ┌────────┴────────┐ │ agent_memory.h5 │ │ single file │ └─────────────────┘ ``` ### Module Overview | Module | What It Does | |--------|-------------| | **`knowledge`** | Entity/relation graph with BFS traversal, spreading activation, fuzzy entity resolution | | **`consolidation`** | Three-tier memory (Working → Episodic → Semantic) with importance scoring and time-decay | | **`hybrid`** | Vector + BM25 fusion with Reciprocal Rank Fusion (RRF, k=60). The vector stage uses the HNSW index by default (`hnsw` feature, on by default); disable with `--no-default-features --features float16` for an exact linear scan | | **`reranker`** | Multi-factor re-ranking: temporal recency, source authority, activation weight | | **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches | | **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints | | **`multimodal`** | Cross-modal search across text/image/audio/video embeddings | | **`provenance`** | Source attribution, FNV-1a content hashing, integrity verification | | **`anomaly`** | Write rate limiting, 15 injection pattern detectors, source distribution analysis | | **`openclaw`** | OpenClaw integration: MemoryBackend trait, Markdown ↔ HDF5 conversion | | **`vector_search`** | Flat cosine, pre-normed, SIMD, BLAS, GPU, parallel search paths | | **`ivf` / `pq`** | IVF-PQ approximate nearest neighbor for billion-scale search | | **`bm25`** | BM25 keyword index with TF-IDF scoring | | **`entity_extract`** | Rule-based entity extraction from text chunks into the knowledge graph | | **`wal`** | Write-ahead log for crash-safe persistence; each entry is CRC32-checked on replay, so a corrupted entry stops replay there instead of loading bad data | | **`memory_strategy`** | Pluggable strategies: save-every, semantic-shift, user-correction detection | | **`decision_gate`** | Sub-microsecond trivial/substantive classification | | **`async_memory`** | Tokio-based async wrapper over the memory store (`async` feature) | --- ## Quick Start ### HDF5 File I/O ```rust use clawhdf5::{File, FileBuilder, AttrValue}; // Write let mut builder = FileBuilder::new(); builder.create_dataset("temperatures") .with_f64_data(&[22.5, 23.1, 21.8]) .with_shape(&[3]); builder.write("output.h5")?; // Read let file = File::open("output.h5")?; let ds = file.dataset("temperatures")?; let values = ds.read_f64()?; assert_eq!(values, vec![22.5, 23.1, 21.8]); ``` ### Agent Memory ```rust use clawhdf5_agent::{HDF5Memory, MemoryConfig, MemoryEntry, AgentMemory}; // Create memory store let config = MemoryConfig::new("agent.h5", "my-agent", 384); let mut memory = HDF5Memory::create(config)?; // Save a memory memory.save(MemoryEntry { chunk: "User prefers dark mode and vim keybindings.".into(), embedding: embed("User prefers dark mode..."), // your embedder source_channel: "chat".into(), timestamp: now(), session_id: "session-001".into(), tags: "preference".into(), })?; // Search let results = memory.search(&query_embedding, 5)?; for result in results { println!("[{:.3}] {}", result.score, result.chunk); } ``` ### Knowledge Graph ```rust use clawhdf5_agent::knowledge::KnowledgeCache; let mut kg = KnowledgeCache::new(); // Add entities let alice = kg.add_entity("Alice", "person", -1); let bob = kg.add_entity("Bob", "person", -1); let acme = kg.add_entity("Acme Corp", "company", -1); // Add relations kg.add_relation(alice, acme, "works_at", 1.0); kg.add_relation(bob, acme, "works_at", 1.0); kg.add_relation(alice, bob, "manages", 0.8); // Traverse let neighbors = kg.bfs_neighbors(alice, 2); // 2-hop neighborhood // Spreading activation — find related entities let activated = kg.spreading_activation(&[alice], 0.5, 0.01, 5); // Entity resolution — fuzzy matching let resolved = kg.resolve_or_create("alice", "person", -1, 2); // Returns existing Alice entity (Levenshtein distance ≤ 2) ``` ### Memory Consolidation ```rust use clawhdf5_agent::consolidation::*; let config = ConsolidationConfig::default(); let mut engine = ConsolidationEngine::new(config); // Add memories — automatically scored for importance engine.add_memory("User prefers dark mode", vec![0.1, 0.2, ...], MemorySource::User); engine.add_memory("ok", vec![0.0, 0.0, ...], MemorySource::System); // Access a memory (reactivates it) engine.access_memory(0); // Run consolidation cycle let stats = engine.consolidate(); // Working memories promote to Episodic (if important enough) // Episodic memories promote to Semantic (if accessed enough) // Low-decay memories get evicted when tiers are full ``` ### Temporal Queries ```rust use clawhdf5_agent::temporal::*; let mut index = TemporalIndex::new(); index.insert(1, 1700000000.0); // record 1 at timestamp index.insert(2, 1700003600.0); // record 2, 1 hour later // Range query — "what happened between 2pm and 5pm?" let ids = index.range_query(1700000000.0, 1700010800.0); // Latest 10 memories let recent = index.latest(10); ``` ### OpenClaw Integration ```rust use clawhdf5_agent::openclaw::*; // Create backend let mut backend = ClawhdfBackend::create("memory.h5", "agent-1", 384)?; // Ingest existing Markdown memory files let md = std::fs::read_to_string("MEMORY.md")?; let count = backend.ingest_markdown("MEMORY.md", &md)?; // Search (uses full pipeline: RRF → re-rank → confidence filter) let results = backend.search("user preferences", &query_embedding, 5); // Export back to Markdown let exported = backend.export_markdown("MEMORY.md")?; ``` --- ## Crate Map ``` clawhdf5 workspace (16 crates, ~92K lines of Rust; plus libaec-sys, an internal FFI bindings crate for the optional szip feature) │ ├── Core HDF5 │ ├── clawhdf5-format — Binary parser/writer (no_std), shared type definitions │ ├── clawhdf5-io — I/O abstraction (buffered, mmap, async) │ ├── clawhdf5-filters — Fast deflate path (zlib-ng); lz4/zstd/pcodec/szip filters live in clawhdf5-format │ ├── clawhdf5-derive — Proc macros │ ├── clawhdf5 — High-level API │ ├── clawhdf5-netcdf4 — NetCDF-4 support │ ├── clawhdf5-accel — SIMD (NEON, AVX2, AVX-512) │ └── clawhdf5-gpu — GPU compute (wgpu, hand-written WGSL compute shaders) │ ├── Agent Memory │ ├── clawhdf5-agent — Memory engine (20.9K lines, 32 modules; WAL is CRC32-checked per entry) │ ├── clawhdf5-ann — HNSW approximate nearest neighbor (default backend; optional `parallel` feature) │ ├── clawhdf5-migrate — SQLite → HDF5 migration │ ├── clawhdf5-android — Android JNI bridge │ └── clawhdf5-cli — CLI tool │ ├── Bindings │ ├── clawhdf5-py — Python (PyO3) │ └── clawhdf5-napi — Node.js (napi-rs) │ └── Tooling └── clawhdf5-bench — Benchmark suite ``` --- ## Research Foundation ClawhDF5's agent memory design draws from 15+ recent papers: | Paper | Key Insight | ClawhDF5 Module | |-------|-------------|-----------------| | **MemX** (2026) | RRF + multi-factor re-ranking | `hybrid`, `reranker` | | **Graph-Native Cognitive Memory** (2026) | Graph-structured belief revision | `knowledge` | | **CraniMem** (2026) | Bounded hippocampal memory | `consolidation` | | **D-MEM** (2026) | Reward prediction error gating | `consolidation` | | **SYNAPSE** (2025) | Spreading activation for recall | `knowledge` | | **RAGdb** (2025) | Zero-dependency edge RAG | Architecture | | **MemoryGraft** (2025) | Memory poisoning attacks | `anomaly`, `provenance` | | **MemoryArena** (2026) | Multi-session benchmark | `temporal` | | **AI Hippocampus** (2026) | Memory taxonomy survey | Overall design | --- ## Feature Flags ### `clawhdf5-agent` | Flag | Default | Description | |------|---------|-------------| | `agent` | no | Full agent memory layer | | `float16` | **yes** | Half-precision embedding storage (2× compression) | | `hnsw` | **yes** | HNSW approximate vector index for `hybrid_search` (via `clawhdf5-ann`); disable for an exact linear scan | | `parallel` | no | Rayon parallel search | | `fast-math` | no | BLAS matrix-vector multiply | | `accelerate` | no | Apple Accelerate / AMX (macOS) | | `openblas` | no | OpenBLAS (Linux) | | `gpu` | no | GPU search via wgpu | | `async` | no | Tokio async with background flush | ### `clawhdf5-format` | Flag | Default | Description | |------|---------|-------------| | `std` | yes | Standard library (disable for `no_std`) | | `deflate` | yes | Deflate compression | | `checksum` | yes | Jenkins lookup3 verification | | `provenance` | yes | SHA-256 provenance attributes | | `fast-deflate` | **yes** | zlib-ng backend for faster deflate | | `system-zlib-decompress` | **yes** | Use the system zlib for decompression where available | | `parallel` | no | Parallel chunk encoding + compression (rayon) | | `fast-checksum` | no | crc32fast-accelerated checksums | | `lz4` | no | LZ4 block compression filter (id 32004) | | `zstd` | no | Zstandard compression filter (id 32015) | | `pcodec` | no | Pcodec lossless numerical codec (id 32023, via `pco` crate) | | `system-zlib` / `zlib-rs` | no | Alternative zlib backends for deflate | | `blake3_hash` | no | BLAKE3 content hashing for provenance | ### `clawhdf5-ann` | Flag | Default | Description | |------|---------|-------------| | `parallel` | no | Rayon-parallel neighbor-distance computation during HNSW graph pruning | ### `clawhdf5-io` | Flag | Default | Description | |------|---------|-------------| | `mpi-io` | no | MPI-backed I/O via the `mpi` crate | > **Parallel I/O (MPI) limitation:** `mpi-io`'s read path is a root-rank read > followed by a broadcast, and its write path gathers all ranks' shards to > rank 0 before writing — not true collective I/O > (`MPI_File_read_at_all`/`write_at_all`). It does not provide I/O bandwidth > that scales with rank count; true collective I/O is tracked as future work. --- ## Building ```bash # Default cargo build --workspace # Agent memory with all accelerations (Linux) cargo build -p clawhdf5-agent --features "agent,float16,parallel,fast-math" # Agent memory with Apple Accelerate (macOS) cargo build -p clawhdf5-agent --features "agent,float16,accelerate,parallel,gpu" # Tests cargo test --workspace # all 1,650+ tests cargo test -p clawhdf5-agent # agent memory tests # Benchmarks cargo bench -p clawhdf5-agent # agent memory suite cargo bench -p clawhdf5-bench # h5bench-equivalent I/O suite ``` --- ## HDF5 File Schema ``` agent_memory.h5 ├── /meta │ ├── schema_version: "1.0" │ ├── agent_id, embedder, embedding_dim │ └── created_at ├── /memory │ ├── chunks: string[N] │ ├── embeddings: f32[N × D] (or f16 with float16 flag) │ ├── tombstones: u8[N] │ └── norms: f32[N] (pre-computed L2) ├── /sessions │ ├── ids: string[S] │ └── summaries: string[S] └── /knowledge_graph ├── entity_names: string[E] ├── relation_srcs: i64[R] ├── relation_tgts: i64[R] └── relation_types: string[R] ``` --- ## Migration ### From rustyhdf5 / edgehdf5 Replace in `Cargo.toml` and source: | Old | New | |-----|-----| | `rustyhdf5*` | `clawhdf5*` | | `edgehdf5-memory` | `clawhdf5-agent` | | `edgehdf5` (CLI) | `clawhdf5-cli` | ### From SQLite ```bash cargo install --path crates/clawhdf5-migrate clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedding-dim 384 ``` --- ## Roadmap See [ROADMAP.md](ROADMAP.md) for the full implementation tracker. **Phase 1 complete** — all 8 tracks delivered: - ✅ Knowledge Graph with spreading activation - ✅ Hippocampal memory consolidation - ✅ RRF hybrid retrieval + re-ranking + confidence rejection - ✅ Temporal reasoning with sub-µs queries - ✅ Memory security + anomaly detection - ✅ Multi-modal memory (text/image/audio/video) - ✅ OpenClaw integration layer - ✅ Comprehensive Criterion benchmarks **Phase 2** — MemoryArena and LongMemEval academic benchmarks are done (see [BENCHMARKS.md](BENCHMARKS.md), reproduced on a second machine); remaining: publish the OpenClaw TypeScript bridge to npm, crates.io/PyPI publishing. --- ## Part of the RedClaw Ecosystem ClawhDF5 powers the `.brain` format for [ClawBrainHub](https://clawbrainhub.com) — the brain registry for AI agents. One file that packages identity, skills, memory, knowledge, and cryptographic provenance. --- ## License MIT ---
Built by RedClaw Systems
~92,000 lines of Rust. Zero C dependencies. One file to remember everything.