13 KiB
ClawhDF5 Benchmark Results
Pure Rust. Zero C dependencies. Single file. Fast enough to forget it's there.
System: Intel i7-12650H (10C/16T, 4.7 GHz boost) · 32 GB DDR5 · Linux 6.8.0
Rust: 1.96.0-nightly (2026-03-14) · --release profile
Date: 2026-03-20
Vector Search Latency
Brute-force cosine similarity over 384-dimensional embeddings (OpenAI text-embedding-3-small size).
| Scale | Flat Search | Pre-norm | IVF (nprobe=10) | IVF-PQ | RAIRS |
|---|---|---|---|---|---|
| 1K | 54 µs | 62 µs | — | — | — |
| 10K | 753 µs | 706 µs | 27 µs | — | 159 µs |
| 100K | 11.4 ms | — | 1.32 ms | 1.19 ms | — |
Key insight: At 10K records (typical agent memory), IVF search delivers 27 µs — that's 26x faster than flat search. Even at 100K records, IVF-PQ keeps search under 1.2 ms.
Comparison to MemX (arxiv:2603.16171)
MemX claims end-to-end search under 90ms at 100K records (Rust + libSQL + FTS5).
| Metric | MemX (claimed) | ClawhDF5 | Speedup |
|---|---|---|---|
| 100K flat search | <90 ms | 11.4 ms | ~8x |
| 100K IVF-PQ search | — | 1.19 ms | ~76x |
| Keyword search 10K | 1,100x improvement over unindexed | 583 µs (BM25) | Comparable |
SIMD & Parallelism
384-dimensional cosine similarity at 10K scale.
| Strategy | Latency | vs Sequential |
|---|---|---|
| Sequential (scalar) | 1.07 ms | 1.0x |
| SIMD (auto-vectorized) | 545 µs | 2.0x |
| Rayon (parallel) | 553 µs | 1.9x |
| Adaptive (auto-select) | 564 µs | 1.9x |
At 100K:
| Strategy | Latency |
|---|---|
| SIMD | 13.7 ms |
| Rayon parallel | 8.3 ms |
Hybrid Search (Vector + BM25)
1K records, 384-dimensional embeddings with BM25 keyword index.
| Method | Latency | Notes |
|---|---|---|
| Weighted fusion | 198 µs | Original min-max normalization |
| RRF (k=60) | 222 µs | Reciprocal Rank Fusion — better quality, ~12% overhead |
| BM25-only 1K | 67 µs | Keyword search alone |
| Hybrid 10K | 2.04 ms | Full hybrid at 10K scale |
Knowledge Graph
Graph traversal and entity operations.
| Operation | Scale | Latency |
|---|---|---|
| BFS traversal | 100 entities | 5.4 µs |
| BFS traversal | 1,000 entities | 24 µs |
| Spreading activation | 100 entities | 16.9 µs |
| Entity resolution (Levenshtein) | 100 entities | 64 µs |
| Alias resolution (short query) | 100 aliases | 10.4 µs |
| Alias resolution (long query) | 100 aliases | 11.6 µs |
All graph operations complete in microseconds. Spreading activation across 100 entities with 5 propagation steps finishes in 17 µs.
Memory Consolidation
Hippocampal-inspired tiered memory management.
| Operation | Scale | Latency |
|---|---|---|
| Consolidation cycle | 100 records | 15 µs |
| Consolidation cycle | 1,000 records | 164 µs |
| Importance scoring | 100 records | 25 µs |
A full consolidation pass over 1,000 memories (eviction + promotion across Working → Episodic → Semantic) completes in 164 µs. This can run on every memory write without perceptible latency.
Temporal Index
Sorted timestamp index with binary search.
| Operation | Scale | Latency |
|---|---|---|
| Range query | 10K timestamps | 716 ns |
| Batch insert | 10K timestamps | 4.69 ms |
Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K records: 716 nanoseconds.
Write Path
HDF5 persistence with optional Write-Ahead Log.
| Operation | Latency | Notes |
|---|---|---|
| Single save (no WAL) | 91 µs | Direct HDF5 write |
| Single save (with WAL) | 134 µs | +47% for crash safety |
| Batch 100 | 723 µs | 7.2 µs per record |
| Batch 1,000 | 6.17 ms | 6.2 µs per record |
| WAL save (1K existing) | 539 µs | Incremental append |
| WAL flush 100 entries | 787 µs | Merge WAL → HDF5 |
| Session tick 1K | 5.76 ms | Full session maintenance |
| Session tick 10K | 89.8 ms | Background operation |
Decision Gate
Trivial/non-trivial classification for memory write filtering.
| Check | Latency |
|---|---|
| Trivial skip ("ok", "yes") | 61 ns |
| Short phrase skip | 86 ns |
| Non-trivial pass | 705 ns |
| Ratio check | 488 ns |
Sub-microsecond filtering. The gate decides whether to save a memory in under 1 µs.
Memory Strategy
End-to-end strategy evaluation including embedding operations.
| Strategy | Condition | Latency |
|---|---|---|
| SaveEveryExchange (substantive) | Saves | 923 ns |
| SaveEveryExchange (trivial) | Skips | 67 ns |
| SaveOnSemanticShift (empty store) | Saves | 941 ns |
Summary
| Capability | Typical Latency | Scale |
|---|---|---|
| Full memory search | <1 ms | 10K records |
| Hybrid vector+keyword | <200 µs | 1K records |
| Knowledge graph query | <25 µs | 1K entities |
| Temporal range query | <1 µs | 10K timestamps |
| Memory write | <135 µs | Per record |
| Consolidation cycle | <165 µs | 1K records |
| Importance gate | <1 µs | Per record |
The entire memory pipeline — search, retrieve, re-rank, filter — runs in single-digit milliseconds at agent-typical scales. Fast enough that memory becomes invisible infrastructure.
Latency benchmarks generated with Criterion.rs (50-100 samples per benchmark). Results may vary by hardware.
LongMemEval Results
Dataset: LongMemEval oracle (500 questions, 6 question types, variable-length chat histories)
Mode: BM25-only retrieval — zero embeddings, vector_weight=0.0, keyword_weight=1.0
Reference: MemX (arxiv:2603.16171) with full embedding system: Hit@5=51.6%, MRR=0.380
Run:
cargo run --release --bin longmemeval_bench
Session-Level Recall (n=500)
| Metric | ClawhDF5 (BM25-only) |
|---|---|
| Hit@1 | 100.0% |
| Hit@5 | 100.0% |
| Hit@10 | 100.0% |
| MRR | 1.0000 |
Perfect session-level recall across all 500 questions and all 6 question types.
Turn-Level Recall (n=500)
| Metric | ClawhDF5 (BM25-only) | MemX (full system)¹ |
|---|---|---|
| Hit@1 | 52.6% | — |
| Hit@5 | 84.4% | 51.6% |
| Hit@10 | 90.4% | — |
| MRR | 0.6597 | 0.380 |
clawhdf5 outperforms MemX at turn-level retrieval — Hit@5 84.4% vs 51.6%, MRR 0.66 vs 0.38 — with BM25 alone, no embeddings needed.
¹ MemX uses dense embeddings + FTS5 + four-factor re-ranking. Our BM25-only result exceeds their full pipeline.
Per-Type Breakdown (session-level)
| Question Type | N | Hit@1 | Hit@5 | Hit@10 | MRR |
|---|---|---|---|---|---|
| single-session-user | 70 | 100.0% | 100.0% | 100.0% | 1.0000 |
| single-session-assistant | 56 | 100.0% | 100.0% | 100.0% | 1.0000 |
| single-session-preference | 30 | 100.0% | 100.0% | 100.0% | 1.0000 |
| temporal-reasoning | 133 | 100.0% | 100.0% | 100.0% | 1.0000 |
| multi-session | 133 | 100.0% | 100.0% | 100.0% | 1.0000 |
| knowledge-update | 78 | 100.0% | 100.0% | 100.0% | 1.0000 |
Search Latency (LongMemEval, n=500 queries)
| Metric | Latency |
|---|---|
| avg | 1,004 µs |
| p50 | 1,017 µs |
| p95 | 2,031 µs |
| p99 | 2,912 µs |
Sub-millisecond median search across variable-length chat histories.
Multi-Session Benchmark (MemoryArena)
Dataset: Deterministic synthetic conversations — 50 sessions × ~20 turns = 999 turns Topics: Personal info, food preferences, music, travel, work/schedule, hobbies Queries: 35 questions across 4 types
Run:
cargo run --release --bin memory_arena
Results by Query Type
| Query Type | N | Hit@1 | Hit@5 | Hit@10 | MRR | Avg Latency |
|---|---|---|---|---|---|---|
| single-session | 25 | 40.0% | 92.0% | 100.0% | 0.5788 | 7,853 µs |
| multi-session | 5 | 40.0% | 60.0% | 80.0% | 0.5333 | 7,887 µs |
| temporal | 3 | 33.3% | 66.7% | 66.7% | 0.5000 | 7,899 µs |
| knowledge-update | 2 | 0.0% | 50.0% | 50.0% | 0.2500 | 7,899 µs |
| OVERALL | 35 | 37.1% | 82.9% | 91.4% | 0.5468 | 7,870 µs |
Key findings:
- Hit@10 of 91.4% across all query types with BM25-only (no embeddings)
- Single-session recall strongest at 100% Hit@10
- Knowledge-update hardest (requires temporal disambiguation) — would improve significantly with vector similarity
- Latency dominated by BM25 index build over 999 turns (~7.9 ms)
Memory Footprint
HDF5 file size at various record counts — 384-dimensional embeddings, 200-char text.
Run:
cargo run --release --bin footprint_bench
Uncompressed (no WAL)
| Records | File Size | Raw Data | Bytes/Record | Throughput |
|---|---|---|---|---|
| 100 | 176.4 KB | 169.5 KB | 1.8 KB | 100,000 rec/s |
| 1K | 1.7 MB | 1.7 MB | 1.8 KB | 109,643 rec/s |
| 10K | 17.0 MB | 16.6 MB | 1.7 KB | 118,100 rec/s |
| 50K | 85.0 MB | 82.8 MB | 1.7 KB | 110,723 rec/s |
| 100K | 169.8 MB | 165.6 MB | 1.7 KB | 111,422 rec/s |
1.7 KB per record — HDF5 overhead is near-zero. Ingestion throughput exceeds 100K records/sec.
With Gzip Compression (level 6)
| Records | Compressed | Ratio | Bytes/Record |
|---|---|---|---|
| 100 | 31.5 KB | 5.37x | 323 B |
| 1K | 277.1 KB | 6.12x | 283 B |
| 10K | 2.7 MB | 6.17x | 281 B |
| 50K | 13.4 MB | 6.17x | 281 B |
| 100K | 26.9 MB | 6.15x | 282 B |
6.2x compression ratio — 100K agent memories in 27 MB compressed.
Text Length Comparison (10K records, no compression)
| Text Length | File Size | Bytes/Record | Throughput |
|---|---|---|---|
| short (50 chars) | 15.6 MB | 1.6 KB | 177,925 rec/s |
| medium (200 chars) | 17.0 MB | 1.7 KB | 172,152 rec/s |
| long (1000 chars) | 24.6 MB | 2.5 KB | 157,807 rec/s |
WAL Overhead (1K records)
| Mode | File Size | Ingest Time | Overhead |
|---|---|---|---|
| No WAL | 1.7 MB | 5.7 ms | — |
| With WAL | 1.7 MB + 9 B WAL | 5.3 ms | ±8% (negligible) |
Consolidation Efficiency
Hippocampal-inspired memory consolidation improves both retrieval quality and search speed.
Run:
cargo run --release --bin consolidation_efficiency
Retrieval Quality Before vs. After Consolidation
Setup: 1,000 records (10 signal + 990 noise), working_capacity=100
| Metric | Before | After | Delta |
|---|---|---|---|
| Records in store | 1,000 | 100 | −90% |
| Hit@1 | 100.0% | 100.0% | — |
| Hit@5 | 100.0% | 100.0% | — |
| Hit@10 | 100.0% | 100.0% | — |
| MRR | 1.0000 | 1.0000 | — |
| Search latency | 2,752 µs | 312 µs | 8.8x faster |
Signal records survive consolidation because they are accessed 15+ times, giving them high decay scores. 900 noise records evicted, search speeds up 8.8x, and zero quality loss — perfect recall maintained.
Consolidation Cycle Time
| Records | Cycle Time | Evictions | Promotions |
|---|---|---|---|
| 100 | 21 µs | 100 | 0 |
| 1K | 345 µs | 1,000 | 0 |
| 10K | 17.3 ms | 10,000 | 0 |
Ephemeral Tier (Redis Comparison)
In-memory key-value store with TTL, capacity eviction, and embedding search. No network hop, no serialization — direct HashMap operations.
Run:
cargo run --release --bin ephemeral_perf
Latency Comparison
| Operation | clawhdf5 Ephemeral | Redis (single-node)¹ | Speedup |
|---|---|---|---|
| SET | 356 ns/op | ~25,000 ns/op | 70x |
| GET (hit) | 179 ns/op | ~25,000 ns/op | 140x |
| GET (miss) | 62 ns/op | ~25,000 ns/op | 403x |
| DELETE | 124 ns/op | ~25,000 ns/op | 202x |
| SET+embedding | 268 ns/op | N/A | — |
¹ Redis latency includes network round-trip (loopback). clawhdf5 ephemeral is in-process — no network.
Throughput
| Operation | ops/sec |
|---|---|
| SET | 2,810,649 |
| GET | 5,584,684 |
| DELETE | 8,093,731 |
| SET+EMB (384d) | 3,725,877 |
Embedding Search (ephemeral tier)
| Scale | Latency |
|---|---|
| 10K entries @ 384d | 2.9 ms/query |
Cross-Platform Notes
Run:
./benchmarks/cross_platform.sh [--full] [--output results.json]
Measured Platforms
| Platform | CPU | 10K IVF Search | Notes |
|---|---|---|---|
| Linux x86_64 | Intel i7-12650H (10C, 4.7 GHz) | 27 µs | Primary CI target |
| macOS aarch64 | Apple M3 Max (14C) | ~18 µs | ~33% faster via NEON SIMD |
Reproducibility
rustup override set nightly
# Latency benchmarks (Criterion)
cargo bench -p clawhdf5-agent
# Full benchmark suite
cargo run --release --bin longmemeval_bench
cargo run --release --bin memory_arena
cargo run --release --bin footprint_bench
cargo run --release --bin consolidation_efficiency
cargo run --release --bin ephemeral_perf