clawhdf5/README.md

ClawhDF5

The memory layer AI agents deserve. One file. Pure Rust. Zero C dependencies.

ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file.

Two things live here:

• A general-purpose, pure-Rust HDF5 library — zero C dependencies, NetCDF-4 support, SIMD/GPU acceleration. See the Crate Map and BENCHMARKS.md for the libhdf5 head-to-head numbers.
• An agent memory layer built on top of it — vector search, knowledge graph, hippocampal-style consolidation, in clawhdf5-agent.

The crates are not on crates.io yet, so depend on them from git:

[dependencies]
clawhdf5       = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }  # core HDF5 read/write
clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }  # + agent memory layer

C dependencies, precisely: the core crates (clawhdf5, clawhdf5-agent, -format, -io, -filters, -ann, -accel, -netcdf4, -cli) build no C code by default — no libhdf5, and deflate is the pure-Rust zlib-rs, which matches zlib-ng on HDF5 reads and writes and produces byte-identical output (BENCHMARKS.md § Deflate backend). CI fails if a C-building crate enters their default dependency tree. C comes in only when you ask for it: fast-deflate (zlib-ng, needs cmake), zstd, szip, the BLAS backends, clawhdf5-migrate (bundled SQLite) and the Node.js bindings.

New here? Start with the Quickstart Guide · See Use Cases · Read Benchmarks

What's new (v2.2 → v2.7, and unreleased)

Five releases in September 2026. Details, including upgrade notes and every breaking change, are in CHANGELOG.md.

HDF5 correctness (read these if you read files with an earlier release)

• Extensible Array chunk indexes returned wrong data past the 36th chunk — any dataset with one unlimited dimension. Silent: plausible numbers from the wrong chunks. Fixed in v2.7.0; re-read affected data.
• Fixed and Extensible Array checksums are now verified, so a corrupt chunk index is ChecksumMismatch instead of wrong data (v2.7.0).
• Compound datatypes written with default libver bounds (plain h5py.File(path, 'w')) were mis-parsed; HDF5 2.0 compound v5 and native complex (class 11) types now parse (v2.2.0–v2.3.0).
• Committed datatypes, fill values, soft links and H5T_STD_REF references now read correctly; external links and external raw data are explicit errors; attrs() no longer silently drops attributes (v2.3.0–v2.5.0).
• Datasets indexed by a version-2 B-tree now read (v2.5.0).

Security and robustness

• A crafted file could abort any reader via B-tree v2 recursion or explode it via shared children; both are now fast errors (v2.7.0).
• Virtual-dataset source paths are confined to the file's directory; chunked reads use overflow-checked sizes and fallible allocation, and the facade writes files atomically (v2.3.0).
• Agent store: single-writer lock plus open_read_only; a crash between checkpoint and WAL truncate no longer duplicates entries; unreadable WALs are quarantined instead of blocking open() (v2.3.0).

Search quality and speed

• HNSW neighbour selection now uses the paper's diversity heuristic: recall@10 at 100K went from 0.31 to 0.98 (v2.4.0).
• hybrid_search is 79–190× faster than v2.3.0 (p50 0.07 ms at 1K, 4.65 ms at 100K). It no longer rebuilds BM25 or rewrites the store per query, and the HNSW graph is persisted (v2.4.0).
• Default fusion weights are now the measured 0.4 / 0.6 (v2.5.0). Re-ranking had been discarding the retrieval score, costing the OpenClaw backend 40.6pp of Hit@1; fixed in v2.6.0.
• Selection reads decode only the chunks they touch (a 64×64 window: 105 ms to 0.39 ms), and full reads are 1.2–1.9× faster (v2.5.0).

Memory

• A loaded store holds ~30% less (embeddings stored once, v2.6.0), and the int8 HNSW index, on by default for new stores (unreleased), brings a 100K × 384 store to 1.74× the raw vectors. At equal recall it is also faster than f32: 1.63× QPS on AVX2, 1.18× on a Raspberry Pi 5 (NEON SDOT).

Tooling

• CI now runs the h5py/netCDF4 interop suites for real (they had been skipping silently) and runs an aarch64 job for the NEON kernels.

---

Why ClawhDF5?

Every AI agent needs memory. Today that means scattered Markdown files, SQLite databases, cloud-hosted vector stores, and glue code. ClawhDF5 replaces all of it:

Problem	Status Quo	ClawhDF5
Vector search	External DB (Pinecone, Qdrant)	Built-in, sub-millisecond
Keyword search	Separate FTS engine	Integrated BM25
Knowledge graph	Neo4j or none	In-file graph with spreading activation
Memory consolidation	Manual pruning	Hippocampal-inspired automatic tiers
Temporal queries	Custom code	Native temporal index (716ns)
Multi-modal	Multiple stores	Unified cross-modal search
Integrity	Hope for the best	Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance
Portability	Config + DB + files	One .h5 file. Copy it anywhere.

---

Performance

Vector search and agent-memory operations below are benchmarked on Intel i7-12650H (10C/16T), 384-dim embeddings, Criterion.rs. The HDF5 Core I/O table immediately below is from a separate, independently reproduced run (see its own hardware note).

HDF5 Core I/O (vs libhdf5 1.14.6)

Benchmark numbers are being validated in collaboration with engineers from the HDF5 Group to confirm methodology and reproducibility.

Figures below are from an independent reproduction run on a second machine (AMD Ryzen 7 7800X3D, 2026-08-03). Full methodology, the original i7-12650H run, and two additional benchmarks added to close prior coverage gaps (an I/O-inclusive metadata-open comparison and an honest zero-copy-mmap measurement) are in BENCHMARKS.md § Independent Validation.

Operation	ClawhDF5	libhdf5	Speedup
Attribute write (128 attrs)	85.2 µs	877 µs	10.3×
Group create (64 groups)	130 µs	1.37 ms	10.6×
Chunked write, deflate-6 (512×512 f32)	1.44 ms	65.0 ms	45.3×
Sequential read (100K f32)	23.3 µs	63.6 µs	2.7×
Sequential write (100K f32)	210 µs	189 µs	≈ tie

The chunked-write row was re-measured on the same machine on 2026-09-23, after the default deflate backend became pure-Rust zlib-rs: 1.46 ms against libhdf5's 51.4 ms (35×), and 1.48 ms with zlib-ng. libhdf5's own time on that machine moved from 65.0 to 51.4 ms between the two dates, which is most of the difference from 45×; compare same-day numbers only.

Vector Search

HNSW (the default backend for hybrid_search) — search_harness, clustered 384-dim data, M = 16, ef_construction = 64, recall measured against an exact scan. See BENCHMARKS.md § Search harness and § Quantising the index copy:

N = 100K, ef = 64	recall@10	QPS	build
f32 index	0.9945	13 399	3.2 s
i8 index + exact re-score (default for new stores)	0.9940	21 848	1.8 s

Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31.

Brute-force and IVF paths (Criterion, i7-12650H):

Scale	Flat	IVF (nprobe=10)	IVF-PQ	vs MemX¹
1K	54 µs	—	—	—
10K	753 µs	27 µs	—	—
100K	11.4 ms	1.32 ms	1.19 ms	~8–76× (see caveat)

Reproduced on the same second machine (Ryzen 7 7800X3D) with a corrected, apples-to-apples SIMD/scalar/parallel comparison methodology — see BENCHMARKS.md § Independent Validation: tank — LongMemEval & Vector Search.

Agent Memory Operations

Operation	Latency	Scale
Hybrid search (HDF5Memory::hybrid_search, p50)	70 µs / 0.49 ms / 4.65 ms	1K / 10K / 100K records
BM25 keyword search	67 µs	1K records
Knowledge graph BFS	24 µs	1K entities
Spreading activation	17 µs	100 entities
Temporal range query	716 ns	10K timestamps
Consolidation cycle	164 µs	1K records
Memory write (WAL)	18 µs	per record (group-commit append; HDF5 batched at flush)
Importance gate	61 ns	per record

Chunked Write Throughput (codec comparison)

Measured with Criterion on f32 matrices. Auto-shuffle is applied before all compression codecs by default (AoS→SoA byte transpose, +157–204% throughput for float data):

Codec	128×128 f32	512×512 f32	Notes
Zstd level 3	148 µs / 422 MiB/s	1.34 ms / 748 MiB/s	With auto-shuffle
Deflate level 6	153 µs / 407 MiB/s	1.39 ms / 719 MiB/s	With auto-shuffle
Pcodec	528 µs / 118 MiB/s	1.69 ms / 591 MiB/s	Best compression ratio

Use .with_zstd(3) or .with_deflate(6) for write-heavy workloads — both now perform at ~720–750 MiB/s on large matrices. Use .with_pcodec() for write-once/read-many workloads where compression ratio matters more than encode speed. Disable auto-shuffle with .without_shuffle() for byte arrays that don't benefit from AoS→SoA transposition.

¹ MemX (arxiv:2603.16171, March 2026): Rust + libSQL, claims <90ms at 100K records. Not like-for-like: MemX's figure is end-to-end (embeddings + FTS5 + four-factor re-ranking); ours is a single component (raw vector search). The ratio overstates the real advantage by an unquantified margin — order-of-magnitude indication only. See BENCHMARKS.md.

LongMemEval Retrieval Recall

Evaluated against the full longmemeval_s haystack — all 500 questions, 47.7 sessions and 493.5 turns each, with only 4.0% of haystack sessions being evidence sessions. See BENCHMARKS.md § LongMemEval Results for the full scoring-target declaration:

Mode	Turn-Level Hit@5	Session-Level Hit@5
BM25 only	75.0%	93.6%
Vector only (MiniLM)	71.8%	94.2%
Hybrid (0.4/0.6, tuned)	81.4%	96.8%

Hybrid is the strongest configuration, which is what running two retrieval stages is for. The weights matter more than the stages: a sweep of vector_weight from 0.0 to 1.0 found the old 0.7/0.3 default is strictly dominated by 0.4/0.6 — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Since v2.5.0 0.4/0.6 is the default (hybrid::DEFAULT_FUSION, used by unified_search, hybrid_search_with and the OpenClaw backend); callers that pass weights to hybrid_search explicitly choose their own. Use 0.3/0.7 if rank-1 precision matters most. Reciprocal rank fusion is selectable (hybrid::Fusion::Rrf) but measured worse than the weighted sum. See BENCHMARKS.md § Weight sweep.

The benchmark's vector stage requires clawhdf5-bench's embeddings feature (real MiniLM embeddings); without it the vector stage is inert and only the BM25 row is produced, which is what every previously published number here measured.

On the easier longmemeval_oracle variant (evidence sessions only) the same harness scores 84.4% turn-level Hit@5 / MRR 0.6597, reproduced identically on a second machine. The 9.4-point gap is the cost of the real haystack, and is why the full-haystack number is the one quoted here.

This is retrieval recall (did the gold memory appear in the top-k), not the official LongMemEval QA-accuracy metric — the two are not comparable, and retrieval recall reported as QA accuracy typically overstates by 20–30 points.

Previously reported here and now retracted: session-level Hit@5 of 100.0% / MRR 1.0000, and a claim of beating MemX's 51.6%. Those session-level figures were degenerate on the oracle variant (any returned document is a hit by construction); the 93.6% above is a different, real measurement on a corpus where evidence sessions are 4.0% of the haystack. The MemX comparison stays withdrawn — MemX measures fact-level granularity over 220,349 records, which running the full haystack does not fix. Details in BENCHMARKS.md.

Enable embeddings via hybrid_search(query_emb, text, 0.4, 0.6, k) for substantially higher recall. The vector stage is served by the HNSW index by default (the hnsw feature is on by default); build with --no-default-features --features float16 to fall back to an exact linear cosine scan.

Memory Footprint

On disk — 384-dim embeddings, 200-char text (BENCHMARKS.md § Memory Footprint):

Records	File Size	Bytes/Record	Gzip-6 compressed
1K	1.7 MB	1.8 KB	277 KB (6.1x)
10K	17.0 MB	1.7 KB	2.7 MB (6.2x)
100K	169.8 MB	1.7 KB	26.9 MB (6.2x)

In memory — a store reopened from disk, 384-dim f32, measured with a counting allocator (BENCHMARKS.md § Memory footprint):

Records	Raw vectors	Reopened, f32 index	Reopened, i8 index (default)
1K	1 MiB	4 MiB (2.40x)	2 MiB (1.64x)
10K	15 MiB	44 MiB (3.03x)	27 MiB (1.81x)
100K	146 MiB	399 MiB (2.72x)	256 MiB (1.74x)

Down from 505 MiB (3.44x) at 100K before v2.6.0, when the cache held every embedding twice.

Consolidation Efficiency

1,000 records (10 signal + 990 noise), working_capacity = 100 (BENCHMARKS.md § Consolidation Efficiency):

Metric	Before	After	Delta
Records in store	1,000	100	−90%
Hit@1 recall (signal records)	100%	100%	no loss
Search latency	2.75 ms	0.31 ms	8.8x faster

Full benchmark details: BENCHMARKS.md

---

Agent Memory Architecture

ClawhDF5's agent memory engine draws on 15+ recent papers on agentic memory systems (see Research Foundation).

                         ┌─────────────────┐
                         │   Agent Query   │
                         └────────┬────────┘
                                  │
                ┌─────────────────▼──────────────────┐
                │ HDF5Memory::hybrid_search          │
                │ HNSW vector + BM25 keyword         │
                │ weighted fusion (0.4 / 0.6)        │
                │ × √(Hebbian activation)            │
                └─────────────────┬──────────────────┘
                                  │  OpenClaw backend adds:
                ┌─────────────────▼──────────────────┐
                │ Multi-factor re-ranking            │
                │ relevance · recency · authority ·  │
                │ activation                         │
                ├────────────────────────────────────┤
                │ Confidence rejection               │
                │ (suppress bad matches)             │
                └─────────────────┬──────────────────┘
                                  │
     ┌────────────────────────────▼────────────────────────────┐
     │ In memory                                               │
     │  cache (flat f32 embeddings) · BM25 index · HNSW index  │
     │  provenance ledger + anomaly alerts (session-scoped)    │
     └────────────────────────────┬────────────────────────────┘
                                  │ WAL append; checkpoint
     ┌────────────────────────────▼────────────────────────────┐
     │ agent_memory.h5   /meta · /memory · /sessions ·         │
     │                   /knowledge_graph                      │
     │ agent_memory.h5.wal   chained-CRC write-ahead log       │
     │ agent_memory.h5.ann   HNSW graph (derived, rebuildable) │
     │ agent_memory.h5.lock  single-writer lock                │
     └─────────────────────────────────────────────────────────┘

Consolidation tiers (Working → Episodic → Semantic), the knowledge-graph algorithms, temporal and multi-modal indexes are library components you drive directly; the store persists the records, sessions and graph they work over.

Module Overview

Module	What It Does
knowledge	Entity/relation graph with BFS traversal, spreading activation, fuzzy (Levenshtein) entity resolution
consolidation	Three-tier memory (Working → Episodic → Semantic) with importance scoring, novelty, and time-decay
hybrid	Vector + BM25 fusion. Default is a min-max-normalised weighted sum, vector 0.4 / keyword 0.6 (hybrid::DEFAULT_FUSION, tuned on LongMemEval); RRF is available via Fusion::Rrf / hybrid_search_with. The vector stage uses the HNSW index by default (hnsw feature); disable with --no-default-features --features float16 for an exact linear scan
reranker	Multi-factor re-ranking: retrieval relevance (leads, weight 1.0), temporal recency, source authority, activation weight. Used by the OpenClaw backend
confidence	Low-confidence rejection — suppresses spurious recalls when nothing matches (OpenClaw backend)
temporal	Sorted timestamp index, session DAG, entity timeline, temporal query hints
multimodal	Cross-modal search across text/image/audio/video embeddings
provenance	Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof)
anomaly	Write rate limiting, 15 injection-pattern detectors, source-distribution analysis. Alerts never block a save; drain them with take_anomaly_alerts
openclaw	OpenClaw integration: MemoryBackend trait, Markdown ↔ HDF5 conversion
vector_search	Flat cosine, pre-normed, SIMD, BLAS, GPU, parallel search paths
ivf / pq	Standalone IVF and IVF-PQ indexes (benchmarked to 100K vectors); not used by HDF5Memory, whose ANN index is HNSW
bm25	Incremental Okapi BM25 inverted index, kept for the life of the store; optional stemming
query_expand	Synonym / acronym / temporal query expansion
entity_extract	Rule-based entity extraction from text chunks into the knowledge graph
wal	Write-ahead log (v4) with a chained CRC32 per entry, so a corrupted, reordered, duplicated or spliced entry stops replay; checkpoints record a WAL mark so nothing is applied twice. Appends are not fsynced
memory_strategy	Pluggable strategies: save-every, semantic-shift, user-correction detection
decision_gate	Sub-microsecond trivial/substantive classification
ephemeral	In-memory TTL/LFU working tier
async_memory	Tokio-based async wrapper over the memory store (async feature)

---

Quick Start

HDF5 File I/O

use clawhdf5::{File, FileBuilder, AttrValue};

// Write
let mut builder = FileBuilder::new();
builder.create_dataset("temperatures")
    .with_f64_data(&[22.5, 23.1, 21.8])
    .with_shape(&[3]);
builder.write("output.h5")?;

// Read
let file = File::open("output.h5")?;
let ds = file.dataset("temperatures")?;
let values = ds.read_f64()?;
assert_eq!(values, vec![22.5, 23.1, 21.8]);

Agent Memory

use clawhdf5_agent::{HDF5Memory, MemoryConfig, MemoryEntry, AgentMemory};

// Create memory store
let config = MemoryConfig::new("agent.h5".into(), "my-agent", 384);
let mut memory = HDF5Memory::create(config)?;

// Save a memory
memory.save(MemoryEntry {
    chunk: "User prefers dark mode and vim keybindings.".into(),
    embedding: embed("User prefers dark mode..."),  // your embedder
    source_channel: "chat".into(),
    timestamp: now(),
    session_id: "session-001".into(),
    tags: "preference".into(),
})?;

// Hybrid search: vector + BM25, weighted 0.4 / 0.6 (the measured default)
let results = memory.hybrid_search(&query_embedding, "user preferences", 0.4, 0.6, 5);
for result in results {
    println!("[{:.3}] {}", result.score, result.chunk);
}

Knowledge Graph

use clawhdf5_agent::knowledge::KnowledgeCache;

let mut kg = KnowledgeCache::new();

// Add entities
let alice = kg.add_entity("Alice", "person", -1);
let bob = kg.add_entity("Bob", "person", -1);
let acme = kg.add_entity("Acme Corp", "company", -1);

// Add relations
kg.add_relation(alice, acme, "works_at", 1.0);
kg.add_relation(bob, acme, "works_at", 1.0);
kg.add_relation(alice, bob, "manages", 0.8);

// Traverse
let neighbors = kg.bfs_neighbors(alice, 2);  // 2-hop neighborhood

// Spreading activation — find related entities
let activated = kg.spreading_activation(&[alice], 0.5, 0.01, 5);

// Entity resolution — fuzzy matching
let (id, created) = kg.resolve_or_create("alice", "person", -1, 2);
// id == alice, created == false: matched the existing entity (Levenshtein distance ≤ 2)

Memory Consolidation

use clawhdf5_agent::consolidation::*;

let config = ConsolidationConfig::default();
let mut engine = ConsolidationEngine::new(config);

let now = 1_700_000_000.0; // seconds since the epoch

// Add memories — automatically scored for importance.
// Elevated sources (System, …) go through a separate, explicit API.
let id = engine.add_memory("User prefers dark mode".into(), vec![0.1, 0.2, ...], UntrustedSource::User, now);
engine.add_trusted_memory("ok".into(), vec![0.0, 0.0, ...], TrustedSource::System, now);

// Access a memory (reactivates it)
engine.access_memory(id, now);

// Run consolidation cycle
engine.consolidate(now);
let stats = engine.get_stats();
// Working memories promote to Episodic (if important enough)
// Episodic memories promote to Semantic (if accessed enough)
// Low-decay memories get evicted when tiers are full

Temporal Queries

use clawhdf5_agent::temporal::*;

let mut index = TemporalIndex::new();
index.insert(1, 1700000000.0);  // record 1 at timestamp
index.insert(2, 1700003600.0);  // record 2, 1 hour later

// Range query — "what happened between 2pm and 5pm?"
let ids = index.range_query(1700000000.0, 1700010800.0);

// Latest 10 memories
let recent = index.latest(10);

OpenClaw Integration

use clawhdf5_agent::openclaw::*;

// Create backend
let mut backend = ClawhdfBackend::create(std::path::Path::new("memory.h5"), 384)?;

// Ingest existing Markdown memory files
let md = std::fs::read_to_string("MEMORY.md")?;
let count = backend.ingest_markdown("MEMORY.md", &md)?;

// Search (full pipeline: weighted vector + BM25 fusion → re-rank → confidence filter)
let results = backend.search("user preferences", &query_embedding, 5);

// Export back to Markdown
let exported = backend.export_markdown("MEMORY.md")?;

---

Crate Map

clawhdf5 workspace (16 crates, ~86K lines of Rust in src/, ~104K with tests
                     and benches; plus libaec-sys, an internal FFI bindings
                     crate for the optional szip feature)
│
├── Core HDF5
│   ├── clawhdf5-format      — Binary parser/writer (no_std-capable), shared type definitions
│   ├── clawhdf5-io          — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI)
│   ├── clawhdf5-filters     — Fast deflate path (zlib-ng); lz4/zstd/pcodec/szip filters live in clawhdf5-format
│   ├── clawhdf5-derive      — Proc macros
│   ├── clawhdf5             — High-level API
│   ├── clawhdf5-netcdf4     — NetCDF-4 support
│   ├── clawhdf5-accel       — SIMD (AVX2, NEON incl. SDOT int8; AVX-512 behind `avx512`)
│   └── clawhdf5-gpu         — GPU compute (wgpu, hand-written WGSL compute shaders)
│
├── Agent Memory
│   ├── clawhdf5-agent       — Memory engine (24.7K lines, 32 modules; chained-CRC WAL)
│   ├── clawhdf5-ann         — HNSW approximate nearest neighbor (default backend; f32 or int8 storage; `parallel` build)
│   ├── clawhdf5-migrate     — SQLite → HDF5 migration
│   ├── clawhdf5-android     — Android JNI bridge
│   └── clawhdf5-cli         — CLI tool
│
├── Bindings
│   ├── clawhdf5-py          — Python (PyO3)
│   └── clawhdf5-napi        — Node.js (napi-rs)
│
└── Tooling
    └── clawhdf5-bench       — Benchmark suite

---

Research Foundation

ClawhDF5's agent memory design draws from 15+ recent papers:

Paper	Key Insight	ClawhDF5 Module
MemX (2026)	Hybrid fusion + multi-factor re-ranking	hybrid, reranker
Graph-Native Cognitive Memory (2026)	Graph-structured memory (weighted, timestamped relations; entity timelines)	knowledge, temporal
CraniMem (2026)	Bounded hippocampal memory	consolidation
D-MEM (2026)	Surprise-gated storage (implemented as a novelty score)	consolidation
SYNAPSE (2025)	Spreading activation for recall	knowledge
RAGdb (2025)	Zero-dependency edge RAG	Architecture
MemoryGraft (2025)	Memory poisoning attacks	anomaly, provenance
MemoryArena (2026)	Multi-session benchmark	temporal
AI Hippocampus (2026)	Memory taxonomy survey	Overall design

---

Feature Flags

clawhdf5-agent

Flag	Default	Description
float16	yes	Half-precision cosine kernel (cosine_similarity_f16). The store itself always writes f32 embeddings; MemoryConfig::float16 is recorded in /meta but not yet applied
hnsw	yes	HNSW approximate vector index for hybrid_search (via clawhdf5-ann); disable for an exact linear scan
parallel	yes	Parallel HNSW bulk build (same graph, ~3× faster on 16 cores) and Rayon brute-force search strategies
zstd	no	Compress embeddings with Zstd instead of deflate when MemoryConfig::compression is on (links libzstd)
fast-math	no	BLAS matrix-vector multiply
accelerate	no	Apple Accelerate / AMX (macOS)
openblas	no	OpenBLAS (Linux)
gpu	no	GPU search via wgpu
async	no	Tokio async with background flush
agent	no	Reserved; currently enables nothing (the agent layer is always built)

To opt out of the parallel build: --no-default-features --features float16,hnsw. For an exact linear cosine scan instead of HNSW: --no-default-features --features float16.

MemoryConfig::hnsw_m, hnsw_ef_construction and hnsw_ef_search tune the vector index (16 / 64 / scale-with-k by default) and are stored with the file.

MemoryConfig::quantized_index (on by default for new stores) holds the HNSW index's own copy of the embeddings as i8, roughly halving a loaded store's memory (2.72x -> 1.74x the raw vectors at 100k x 384). Quantised distances are approximate, so the query path re-scores the candidate pool against the exact embeddings the store already holds, which keeps recall at the f32 index's level. It is also faster: 1.63x the queries per second at equal recall on x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON SDOT), with index builds 1.8x and 2.3x faster respectively. Stores created before the setting existed keep their f32 index; opt out for new stores with quantized_index = false or clawhdf5-cli create --f32-index. See BENCHMARKS.md § Quantising the index copy.

clawhdf5-format

Flag	Default	Description
std	yes	Standard library (disable for no_std)
deflate	yes	Deflate compression
checksum	yes	Jenkins lookup3 verification
provenance	yes	SHA-256 provenance attributes
zlib-rs	yes	Pure-Rust deflate backend (zlib-rs)
fast-deflate	no	zlib-ng deflate backend instead (C; needs cmake). Overrides zlib-rs when both are on
system-zlib-decompress	yes	Use Apple's system libz for decompression (macOS only; no effect elsewhere)
parallel	no	Parallel chunk encoding + compression (rayon)
fast-checksum	no	crc32fast-accelerated checksums
lz4	no	LZ4 block compression filter (id 32004)
zstd	no	Zstandard compression filter (id 32015)
pcodec	no	Pcodec lossless numerical codec (id 32023, via pco crate)
system-zlib	no	System zlib backend for deflate (C)
blake3_hash	no	BLAKE3 content hashing for provenance
szip	no	SZIP filter (id 4) via libaec (C, through the internal libaec-sys crate)

clawhdf5-ann

Flag	Default	Description
parallel	no	Batched bulk build runs neighbour planning and back-link pruning on a Rayon pool; the graph is identical with or without it (enabled by clawhdf5-agent's default parallel)

clawhdf5-io

Flag	Default	Description
mmap	no	Memory-mapped reads (memmap2)
async	no	Tokio-based async I/O
hsds	no	HSDS (HDF REST service) client
mpi-io	no	MPI-backed I/O via the mpi crate

Parallel I/O (MPI) limitation: mpi-io's read path is a root-rank read followed by a broadcast, and its write path gathers all ranks' shards to rank 0 before writing — not true collective I/O (MPI_File_read_at_all/write_at_all). It does not provide I/O bandwidth that scales with rank count; true collective I/O is tracked as future work.

---

Building

# Default (pure Rust: no cmake or C compiler needed)
cargo build --workspace

# Agent memory with all accelerations (Linux)
cargo build -p clawhdf5-agent --features fast-math

# Agent memory with Apple Accelerate (macOS)
cargo build -p clawhdf5-agent --features "accelerate,gpu"

# Tests
cargo test --workspace            # all 1,850+ tests
cargo test -p clawhdf5-agent      # agent memory tests
scripts/ci-test.sh                # what CI runs: fmt, clippy matrix, tests,
                                  # h5py/netCDF4 interop, no_std

# The interop suites need a Python with h5py; on a PEP 668 system that has to
# be a virtualenv. `ci-test.sh` finds `.venv` on its own, or set
# CLAWHDF5_PYTHON. Without one they skip — set CLAWHDF5_REQUIRE_INTEROP=1 to
# make that a failure instead.
python3 -m venv .venv && .venv/bin/pip install h5py numpy netCDF4 xarray

# Benchmarks
cargo bench -p clawhdf5-agent               # agent memory suite
cargo bench -p clawhdf5-bench               # h5bench-equivalent I/O suite

---

HDF5 File Schema

agent_memory.h5
├── /meta                          (attributes)
│   ├── schema_version: "1.0", edgehdf5_version
│   ├── agent_id, embedder, embedding_dim, chunk_size, overlap, created_at
│   ├── float16, compression, compression_level, compact_threshold,
│   │   hebbian_boost, decay_factor, wal_enabled, wal_max_entries
│   ├── quantized_index, hnsw_m, hnsw_ef_construction, hnsw_ef_search
│   ├── wal_applied_len, wal_applied_crc   (WAL mark of the last checkpoint)
│   └── ann_generation                     (ties the .ann sidecar to this checkpoint)
├── /memory
│   ├── chunks:              string[N]
│   ├── embeddings:          f32[N × D]  (chunked; deflate, or Zstd with the
│   │                                     `zstd` feature, when compression is on)
│   ├── source_channel:      string[N]
│   ├── timestamps:          f64[N]
│   ├── session_ids:         string[N]
│   ├── tags:                string[N]
│   ├── tombstones:          u8[N]
│   ├── norms:               f32[N]      (pre-computed L2)
│   └── activation_weights:  f32[N]      (Hebbian)
├── /sessions
│   ├── ids, channels, summaries:  string[S]
│   ├── start_idxs, end_idxs:      i64[S]
│   └── timestamps:                f64[S]
└── /knowledge_graph
    ├── entity_ids, entity_emb_idxs: i64[E]; entity_names, entity_types: string[E]
    ├── relation_srcs, relation_tgts: i64[R]; relation_types: string[R]
    ├── relation_weights: f32[R]; relation_ts: f64[R]
    └── alias_strings: string[A]; alias_entity_ids: i64[A]   (when aliases exist)

Alongside the store: <store>.h5.wal (write-ahead log), <store>.h5.ann (HNSW graph; derived, safe to delete) and <store>.h5.lock (single-writer lock). A second writer gets MemoryError::Locked; use HDF5Memory::open_read_only for a lock-free point-in-time view.

---

Migration

From rustyhdf5 / edgehdf5

Replace in Cargo.toml and source:

Old	New
rustyhdf5*	clawhdf5*
edgehdf5-memory	clawhdf5-agent
edgehdf5 (CLI)	clawhdf5-cli

From SQLite

cargo install --path crates/clawhdf5-migrate
clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedding-dim 384

---

Roadmap

See ROADMAP.md for the full implementation tracker.

Phase 1 complete — all 8 tracks delivered:

• ✅ Knowledge Graph with spreading activation
• ✅ Hippocampal memory consolidation
• ✅ RRF hybrid retrieval + re-ranking + confidence rejection
• ✅ Temporal reasoning with sub-µs queries
• ✅ Memory security + anomaly detection
• ✅ Multi-modal memory (text/image/audio/video)
• ✅ OpenClaw integration layer
• ✅ Comprehensive Criterion benchmarks

Phase 2 — MemoryArena and LongMemEval academic benchmarks are done (see BENCHMARKS.md, reproduced on a second machine); remaining: publish the OpenClaw TypeScript bridge to npm, crates.io/PyPI publishing.

---

Part of the RedClaw Ecosystem

ClawhDF5 powers the .brain format for ClawBrainHub — the brain registry for AI agents. One file that packages identity, skills, memory, knowledge, and cryptographic provenance.

---

License

MIT

---

Built by RedClaw Systems
~86,000 lines of Rust. Zero C dependencies. One file to remember everything.