clawhdf5/BENCHMARKS.md

ClawhDF5 Benchmark Results

Pure Rust. Zero C dependencies. Single file. Fast enough to forget it's there.

System: Intel i7-12650H (10C/16T, 4.7 GHz boost) · 32 GB DDR5 · Linux 6.8.0
Rust: 1.96.0-nightly (2026-03-14) · --release profile
Date: 2026-07-01

Traceability note: the "h5bench-Equivalent I/O Benchmarks" and both "Independent Validation: tank" sections below meet a dated, hardware-cited, reproducible standard (explicit date, machine spec, and a runnable command per result) — this now covers "LongMemEval Results", "SIMD & Parallelism", "Vector Search Latency", and "Comparison to MemX" via their tank re-runs. The remaining undated sections above (Hybrid Search, Knowledge Graph, Memory Consolidation, Temporal Index, Write Path, Decision Gate, Memory Strategy, Multi-Session Benchmark, Memory Footprint, Consolidation Efficiency, Ephemeral Tier) do not yet meet that bar — this is a known, tracked documentation gap, not a claim that those numbers are wrong.

Correctness note (2026-08-06). Being dated and reproducible is necessary but not sufficient — a number can be perfectly reproducible and still measure the wrong thing. A methodology audit found two such cases and both have been retracted in place: the session-level LongMemEval figures (degenerate on the oracle variant) and the MemX retrieval comparison (mismatched granularity and corpus). Every cross-system comparison in this file now carries an explicit scoping caveat. Where a section states a scoring target, that declaration is the contract — read it before citing the number.

---

Vector Search Latency

Brute-force cosine similarity over 384-dimensional embeddings (OpenAI text-embedding-3-small size).

Scale	Flat Search	Pre-norm	IVF (nprobe=10)	IVF-PQ	RAIRS
1K	54 µs	62 µs	—	—	—
10K	753 µs	706 µs	27 µs	—	159 µs
100K	11.4 ms	—	1.32 ms	1.19 ms	—

Key insight: At 10K records (typical agent memory), IVF search delivers 27 µs — that's 26x faster than flat search. Even at 100K records, IVF-PQ keeps search under 1.2 ms.

Comparison to MemX (arxiv:2603.16171)

MemX claims end-to-end search under 90ms at 100K records (Rust + libSQL + FTS5).

Caveat — not like-for-like. MemX's <90 ms is end-to-end search across their full pipeline (dense embeddings + FTS5 + four-factor re-ranking). The clawhdf5 figures below are a single component — raw vector search latency, excluding embedding, keyword, fusion, and re-ranking stages. A component measured against a full pipeline will always look favourable; the "speedup" column overstates the real advantage by an unquantified margin and should be read as an order-of-magnitude indication only, not a benchmark result. Matching MemX's measurement boundary is tracked as follow-up work.

Metric	MemX (claimed, end-to-end)	ClawhDF5 (component only)	Ratio
100K flat search	<90 ms	11.4 ms	~8x
100K IVF-PQ search	—	1.19 ms	~76x
Keyword search 10K	1,100x improvement over unindexed	583 µs (BM25)	Comparable

---

SIMD & Parallelism

384-dimensional cosine similarity at 10K scale.

Strategy	Latency	vs Sequential
Sequential (scalar)	1.07 ms	1.0x
SIMD (auto-vectorized)	545 µs	2.0x
Rayon (parallel)	553 µs	1.9x
Adaptive (auto-select)	564 µs	1.9x

At 100K:

Strategy	Latency
SIMD	13.7 ms
Rayon parallel	8.3 ms

---

Hybrid Search (Vector + BM25)

1K records, 384-dimensional embeddings with BM25 keyword index.

Method	Latency	Notes
Weighted fusion	198 µs	Original min-max normalization
RRF (k=60)	222 µs	Reciprocal Rank Fusion — better quality, ~12% overhead
BM25-only 1K	67 µs	Keyword search alone
Hybrid 10K	2.04 ms	Full hybrid at 10K scale

---

Knowledge Graph

Graph traversal and entity operations.

Operation	Scale	Latency
BFS traversal	100 entities	5.4 µs
BFS traversal	1,000 entities	24 µs
Spreading activation	100 entities	16.9 µs
Entity resolution (Levenshtein)	100 entities	64 µs
Alias resolution (short query)	100 aliases	10.4 µs
Alias resolution (long query)	100 aliases	11.6 µs

All graph operations complete in microseconds. Spreading activation across 100 entities with 5 propagation steps finishes in 17 µs.

---

Memory Consolidation

Hippocampal-inspired tiered memory management.

Operation	Scale	Latency
Consolidation cycle	100 records	15 µs
Consolidation cycle	1,000 records	164 µs
Importance scoring	100 records	25 µs

A full consolidation pass over 1,000 memories (eviction + promotion across Working → Episodic → Semantic) completes in 164 µs. This can run on every memory write without perceptible latency.

---

Temporal Index

Sorted timestamp index with binary search.

Operation	Scale	Latency
Range query	10K timestamps	716 ns
Batch insert	10K timestamps	4.69 ms

Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K records: 716 nanoseconds.

---

Write Path

HDF5 persistence with optional Write-Ahead Log.

Operation	Latency	Notes
Single save (no WAL)	61 µs	Direct HDF5 write (owned-Vec IO path)
Single save (with WAL)	18 µs	WAL group-commit append; HDF5 write batched at flush
Batch 100	723 µs	7.2 µs per record
Batch 1,000	6.17 ms	6.2 µs per record
WAL save (1K existing)	539 µs	Incremental append
WAL flush 100 entries	787 µs	Merge WAL → HDF5
Session tick 1K	5.76 ms	Full session maintenance
Session tick 10K	89.8 ms	Background operation

---

Decision Gate

Trivial/non-trivial classification for memory write filtering.

Check	Latency
Trivial skip ("ok", "yes")	61 ns
Short phrase skip	86 ns
Non-trivial pass	705 ns
Ratio check	488 ns

Sub-microsecond filtering. The gate decides whether to save a memory in under 1 µs.

---

Memory Strategy

End-to-end strategy evaluation including embedding operations.

Strategy	Condition	Latency
SaveEveryExchange (substantive)	Saves	923 ns
SaveEveryExchange (trivial)	Skips	67 ns
SaveOnSemanticShift (empty store)	Saves	941 ns

---

Summary

Capability	Typical Latency	Scale
Full memory search	<1 ms	10K records
Hybrid vector+keyword	<200 µs	1K records
Knowledge graph query	<25 µs	1K entities
Temporal range query	<1 µs	10K timestamps
Memory write	<20 µs	Per record (WAL group-commit append)
Consolidation cycle	<165 µs	1K records
Importance gate	<1 µs	Per record

The entire memory pipeline — search, retrieve, re-rank, filter — runs in single-digit milliseconds at agent-typical scales. Fast enough that memory becomes invisible infrastructure.

---

Latency benchmarks generated with Criterion.rs (50-100 samples per benchmark). Results may vary by hardware.

---

LongMemEval Results

Scoring target declaration. Per arXiv 2605.24060, which found that changing scoring target alone alters nDCG on 83–94% of queries and can reverse system rankings, this section states its measurement contract explicitly:

• Dataset variant: both are now reported below — the full longmemeval_s haystack (the headline number) and longmemeval_oracle (evidence sessions only, a substantially easier corpus, kept for continuity). The harness does not trust the filename: it measures evidence-session density from the data and labels the run from that, so a mislabelled input cannot yield a mislabelled result. Measured density is 4.0% on longmemeval_s and 100.0% on the oracle.
• Metric: retrieval recall. A "hit" means the gold-labelled memory appeared in the top-k. No answer is generated and none is scored — the dataset's answer field is deserialized and never read. This is not the official LongMemEval leaderboard metric, which is end-to-end QA accuracy (retrieve → generate → LLM judge). Retrieval recall reported as QA accuracy typically overstates by 20–30 points.
• Granularity: turn-level = the returned memory's source turn had has_answer == true.
• k = 10, n = 500.
• Retrieval mode: all three are reported below. Historically the bench passed zero-vector embeddings with vector_weight=0.0, so the HNSW/vector stage was inert and every published number was BM25 alone. Real all-MiniLM-L6-v2 embeddings are now available via --features embeddings --embeddings <dir>, and BM25-only / vector-only / hybrid are each measured separately.

Mode: BM25-only retrieval — zero embeddings, vector_weight=0.0, keyword_weight=1.0

Run: cargo run --release --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s_cleaned.json (~70 s for all 500 questions on the tank reference machine). Omit the path for the oracle variant; add --limit N for an evenly-strided subsample.

Full haystack — longmemeval_s, n=500 (the number to cite)

47.7 sessions and 493.5 turns per question; 4.0% of haystack sessions are evidence sessions, so retrieval has to actually discriminate.

Metric	Turn-level	Session-level
Hit@1	53.8%	86.2%
Hit@5	75.0%	93.6%
Hit@10	81.6%	96.6%
MRR	0.6320	0.8948

Session-level is reported here because on this corpus it is meaningful — unlike on the oracle variant, where it was degenerate and was retracted (below). At 4.0% evidence density a session-level hit reflects discrimination rather than corpus shape.

Per-type, session-level: single-session-assistant 100.0% Hit@1 (n=56), knowledge-update 96.2% (n=78), single-session-user 94.3% (n=70), multi-session 84.2% (n=133), temporal-reasoning 84.2% (n=133), and single-session-preference 33.3% (n=30) — the one category where BM25 clearly struggles, since a preference question's evidence rarely shares vocabulary with the question.

Retrieval mode ablation — full haystack, n=500

Real 384-d all-MiniLM-L6-v2 embeddings, 190,015 unique texts encoded once on an RTX 5060 Ti (~13 min; the same work on the 8-core CPU was still unfinished after 30 minutes, so the GPU path is not a convenience here). Turn-level:

Mode	Hit@1	Hit@5	Hit@10	MRR
BM25 only (0.0/1.0)	53.8%	75.0%	81.6%	0.6320
Vector only (1.0/0.0)	36.0%	71.8%	81.6%	0.5027
Hybrid (0.7/0.3)	44.4%	79.2%	86.0%	0.5868

Session-level:

Mode	Hit@1	Hit@5	Hit@10	MRR
BM25 only	86.2%	93.6%	96.6%	0.8948
Vector only	85.4%	94.2%	96.6%	0.8901
Hybrid	88.2%	95.8%	97.8%	0.9158

Weight sweep — full haystack, n=500

0.7/0.3 was a documented default, never a searched one. Sweeping vector_weight from 0.0 to 1.0 (--sweep, reusing the one-time embedding table) shows it is not merely suboptimal but strictly dominated:

vector / keyword	Hit@1	Hit@5	Hit@10	MRR	session Hit@5
0.0 / 1.0 (BM25)	53.8%	75.0%	81.6%	0.6320	93.6%
0.1 / 0.9	53.2%	77.4%	83.8%	0.6374	95.0%
0.2 / 0.8	53.6%	78.2%	85.6%	0.6440	95.4%
0.3 / 0.7	53.2%	78.8%	87.2%	0.6463	96.0%
0.4 / 0.6	51.6%	81.4%	87.8%	0.6429	96.8%
0.5 / 0.5	48.2%	81.4%	88.2%	0.6234	97.4%
0.6 / 0.4	46.6%	79.8%	87.4%	0.6069	96.6%
0.7 / 0.3 (old default)	44.4%	79.2%	86.0%	0.5868	95.8%
0.8 / 0.2	40.6%	76.2%	85.4%	0.5571	95.2%
0.9 / 0.1	37.8%	73.4%	84.6%	0.5289	94.2%
1.0 / 0.0 (vector)	36.0%	71.8%	81.6%	0.5027	94.2%

0.4/0.6 beats 0.7/0.3 on every metric at both granularities — Hit@1 +7.2pp, Hit@5 +2.2, Hit@10 +1.8, MRR +0.056. There is no trade being made; the old default was simply on the wrong side of the peak. 0.4/0.6 is the recommended setting, with 0.3/0.7 preferable if rank-1 precision matters most (it takes the best MRR in the sweep and gives up only 0.6pp of Hit@1 against pure BM25).

Correction. An earlier revision of this section, measuring only 0.7/0.3, concluded that fusion "buys deeper recall and pays for it at rank 1" and advised callers taking a single top hit to prefer BM25. That was an artifact of the badly-chosen weight, not a property of fusion. At 0.3/0.7 hybrid beats BM25 on MRR (0.6463 vs 0.6320) and on Hit@5 (78.8% vs 75.0%) while costing 0.6pp of Hit@1. The advice below is corrected accordingly.

Hybrid wins, once the weights are right. At the old 0.7/0.3 the picture looked like a trade: best at Hit@5 and Hit@10, worse than BM25 at Hit@1 and MRR. The sweep above shows that was the weight, not fusion. At 0.4/0.6 hybrid leads Hit@5 and Hit@10 outright; at 0.3/0.7 it also leads MRR and is within 0.6pp of BM25 at Hit@1. Both dominate 0.7/0.3.

The rows below are kept at the three original settings because they are what the mode ablation measured — read them as "the shape of each stage in isolation", and take the operating point from the sweep.

The same pattern shows up independently in omni-cortex's four-signal RRF ablation, where adding BM25 to a dense retriever raised nDCG@5 while lowering Hit@1 and MRR. Two different codebases, two different fusion schemes, same direction.

Vector-only being worse than BM25 at every turn-level cutoff except Hit@10 is worth stating plainly rather than hiding: LongMemEval questions share substantial vocabulary with their evidence turns, which is close to the best case for lexical matching, and MiniLM at 384 dimensions is a small embedding model.

Run: cargo run --release --bin longmemeval_bench --features embeddings -- \ benchmarks/longmemeval/longmemeval_s_cleaned.json --embeddings weights/all-minilm-l6-v2 For the GPU path use --features embeddings-cuda. That requires nvcc on PATH at build time — cudarc's build script shells out to it. The toolkit installs to /usr/local/cuda/bin, which many distributions do not export; check with nvcc --version and, if it is missing, add it somewhere every shell reads (for zsh that is ~/.zshenv, not ~/.zshrc, since build tooling runs non-interactively). The device is selected at runtime with a CPU fallback, so a machine without CUDA still produces correct numbers — just far more slowly, and the bench says so on startup.

Weights: huggingface.co/sentence-transformers/all-MiniLM-L6-v2 — place model.safetensors and tokenizer.json in the --embeddings directory.

Oracle variant — longmemeval_oracle, n=500 (easier corpus, kept for continuity)

Metric	ClawhDF5 (BM25-only, oracle variant)
Hit@1	52.6%
Hit@5	84.4%
Hit@10	90.4%
MRR	0.6597

Turn-level. The 9.4-point gap between this and the full haystack's 75.0% is the price of the harder corpus, and is the reason oracle-only numbers should not be presented as LongMemEval results. Session-level figures on this variant are degenerate — see below.

With real embeddings the same oracle corpus gives BM25-only 84.2% / vector-only 80.4% / hybrid 85.2% Hit@5 turn-level — hybrid ahead at Hit@5 and Hit@10 and behind at Hit@1, matching the full-haystack pattern above. (BM25-only reads 84.2% here against 84.4% with zero embedding vectors: one question of 500 changes rank, with MRR identical at 0.6597. On the full haystack the two agree exactly.)

Retracted: session-level recall and the MemX comparison

Earlier revisions of this file reported session-level Hit@1/5/10 of 100.0% with MRR 1.0000, uniform across all six question types, and claimed clawhdf5 "outperforms MemX at turn-level retrieval (84.4% vs 51.6%)". Both are withdrawn.

The session-level numbers are a degenerate artifact. On the longmemeval_oracle variant, the ingested haystack for a question consists essentially only of that question's evidence sessions. Every returned document therefore belongs to an answer session, so session-level hit rate is ≈1.0 at rank 0 by construction — which is exactly why the result was a uniform 100.0% across every question type. It measured the shape of the corpus, not the retriever.

The MemX comparison was not like-for-like on two independent axes. MemX (arxiv:2603.16171) reports Hit@5 = 51.6% / MRR = 0.380 at fact-level granularity over 220,349 fact-level records drawn from 19,195 sessions, and explicitly notes that fact-level "doubl[es] session-level performance." Our 84.4% is turn-level, on the oracle subset. Different retrieval granularity, and a corpus smaller by orders of magnitude. A higher number on an easier corpus at a different granularity is not an outperformance claim, and it should not have been presented as one.

The full-haystack half of that gap is now closed: the section above reports longmemeval_s over all 500 questions. The granularity mismatch remains — MemX measures fact-level, we measure turn-level and session-level — so no cross-system claim is made here even now. Matching granularity would require fact-level extraction over the haystack, which this harness does not do.

Search Latency (LongMemEval, n=500 queries)

Metric	Latency
avg	1,004 µs
p50	1,017 µs
p95	2,031 µs
p99	2,912 µs

Sub-millisecond median search across variable-length chat histories.

---

Multi-Session Benchmark (MemoryArena)

Dataset: Deterministic synthetic conversations — 50 sessions × ~20 turns = 999 turns Topics: Personal info, food preferences, music, travel, work/schedule, hobbies Queries: 35 questions across 4 types

Run: cargo run --release --bin memory_arena

Results by Query Type

Query Type	N	Hit@1	Hit@5	Hit@10	MRR	Avg Latency
single-session	25	40.0%	92.0%	100.0%	0.5788	7,853 µs
multi-session	5	40.0%	60.0%	80.0%	0.5333	7,887 µs
temporal	3	33.3%	66.7%	66.7%	0.5000	7,899 µs
knowledge-update	2	0.0%	50.0%	50.0%	0.2500	7,899 µs
OVERALL	35	37.1%	82.9%	91.4%	0.5468	7,870 µs

Key findings:

• Hit@10 of 91.4% across all query types with BM25-only (no embeddings)
• Single-session recall strongest at 100% Hit@10
• Knowledge-update hardest (requires temporal disambiguation) — would improve significantly with vector similarity
• Latency dominated by BM25 index build over 999 turns (~7.9 ms)

---

Memory Footprint

HDF5 file size at various record counts — 384-dimensional embeddings, 200-char text.

Run: cargo run --release --bin footprint_bench

Uncompressed (no WAL)

Records	File Size	Raw Data	Bytes/Record	Throughput
100	176.4 KB	169.5 KB	1.8 KB	100,000 rec/s
1K	1.7 MB	1.7 MB	1.8 KB	109,643 rec/s
10K	17.0 MB	16.6 MB	1.7 KB	118,100 rec/s
50K	85.0 MB	82.8 MB	1.7 KB	110,723 rec/s
100K	169.8 MB	165.6 MB	1.7 KB	111,422 rec/s

1.7 KB per record — HDF5 overhead is near-zero. Ingestion throughput exceeds 100K records/sec.

With Gzip Compression (level 6)

Records	Compressed	Ratio	Bytes/Record
100	31.5 KB	5.37x	323 B
1K	277.1 KB	6.12x	283 B
10K	2.7 MB	6.17x	281 B
50K	13.4 MB	6.17x	281 B
100K	26.9 MB	6.15x	282 B

6.2x compression ratio — 100K agent memories in 27 MB compressed.

Text Length Comparison (10K records, no compression)

Text Length	File Size	Bytes/Record	Throughput
short (50 chars)	15.6 MB	1.6 KB	177,925 rec/s
medium (200 chars)	17.0 MB	1.7 KB	172,152 rec/s
long (1000 chars)	24.6 MB	2.5 KB	157,807 rec/s

WAL Overhead (1K records)

Mode	File Size	Ingest Time	Overhead
No WAL	1.7 MB	5.7 ms	—
With WAL	1.7 MB + 9 B WAL	5.3 ms	±8% (negligible)

---

Consolidation Efficiency

Hippocampal-inspired memory consolidation improves both retrieval quality and search speed.

Run: cargo run --release --bin consolidation_efficiency

Retrieval Quality Before vs. After Consolidation

Setup: 1,000 records (10 signal + 990 noise), working_capacity=100

Metric	Before	After	Delta
Records in store	1,000	100	−90%
Hit@1	100.0%	100.0%	—
Hit@5	100.0%	100.0%	—
Hit@10	100.0%	100.0%	—
MRR	1.0000	1.0000	—
Search latency	2,752 µs	312 µs	8.8x faster

Signal records survive consolidation because they are accessed 15+ times, giving them high decay scores. 900 noise records evicted, search speeds up 8.8x, and zero quality loss — perfect recall maintained.

Consolidation Cycle Time

Records	Cycle Time	Evictions	Promotions
100	21 µs	100	0
1K	345 µs	1,000	0
10K	17.3 ms	10,000	0

---

Ephemeral Tier (Redis Comparison)

In-memory key-value store with TTL, capacity eviction, and embedding search. No network hop, no serialization — direct HashMap operations.

Run: cargo run --release --bin ephemeral_perf

Latency Comparison

Operation	clawhdf5 Ephemeral	Redis (single-node)¹	Speedup
SET	356 ns/op	~25,000 ns/op	70x
GET (hit)	179 ns/op	~25,000 ns/op	140x
GET (miss)	62 ns/op	~25,000 ns/op	403x
DELETE	124 ns/op	~25,000 ns/op	202x
SET+embedding	268 ns/op	N/A	—

¹ Redis latency includes network round-trip (loopback). clawhdf5 ephemeral is in-process — no network.

Throughput

Operation	ops/sec
SET	2,810,649
GET	5,584,684
DELETE	8,093,731
SET+EMB (384d)	3,725,877

Embedding Search (ephemeral tier)

Scale	Latency
10K entries @ 384d	2.9 ms/query

---

World-Model Sample Loading (vs h5py / stable-worldmodel shape)

Reproduces the access pattern of stable-worldmodel's HDF5 dataloader (arXiv 2605.21800, LeCun/Balestriero group), which supports HDF5 as one of three native formats and measures generic HDF5 at 1,416-1,474 samples/s (vs Lance 4,815) for per-frame sample loading. This benchmark measures clawhdf5 vs h5py on the same machine and the same file, so the comparison is hardware-controlled.

Absolute numbers are not comparable to the paper's - different hardware (AMD Ryzen 7 7800X3D, local NVMe, warm page cache), smaller frames, and no torch-tensor / transform step. Only the clawhdf5-vs-h5py ratio here is a controlled result. The workload is the dataloader shape: a (N, H, W, C) uint8 observation dataset (20,000 x 64x64x3 = 246 MB), each frame read once per pass in a fixed shuffled (random-access) order, 10 passes.

Both read a file written by h5py - clawhdf5 parsing an externally-produced HDF5 file is itself the interop result. h5py opens SWMR with a 256 MB chunk cache, exactly stable-worldmodel's HDF5Dataset; it materialises each frame as a numpy array (d[i]) and sums it. clawhdf5 mmaps once, takes a zero-copy &[u8] over the contiguous dataset, and indexes frame i as a subslice.

Reader	samples/sec (median of 3)	vs h5py
clawhdf5 (zero-copy view)	593,000	8.1x
clawhdf5 (materialised copy per frame)	518,000	7.1x
h5py (swmr, 256 MB cache)	73,000	1.0x

The materialised-copy row is the fair, equal-work comparison - it to_vec()s every frame so clawhdf5 pays the same per-frame allocation h5py does, and it is still 7.1x faster. That the copy costs almost nothing (518k vs 593k) shows the h5py gap is per-frame call overhead (Python + library dispatch), not data movement. This is an in-page-cache measurement: it isolates the read-path overhead both libraries add on top of the OS, which is the thing that differs - not disk bandwidth, which is shared.

Reproduce (benchmarks/):

python benchmarks/gen_worldmodel_frames.py /tmp/wm_frames.h5 20000
cargo run --release -p clawhdf5-bench --example worldmodel_sampling -- /tmp/wm_frames.h5 10
cargo run --release -p clawhdf5-bench --example worldmodel_sampling -- /tmp/wm_frames.h5 10 --copy
python benchmarks/bench_worldmodel_h5py.py /tmp/wm_frames.h5 10

Measured 2026-08-07 on tank (Ryzen 7 7800X3D, 246 MB dataset in page cache).

Cross-Platform Notes

Run: ./benchmarks/cross_platform.sh [--full] [--output results.json]

Measured Platforms

Platform	CPU	10K IVF Search	Notes
Linux x86_64	Intel i7-12650H (10C, 4.7 GHz)	27 µs	Primary CI target
macOS aarch64	Apple M3 Max (14C)	~18 µs	~33% faster via NEON SIMD

Reproducibility

rustup override set nightly

# Latency benchmarks (Criterion)
cargo bench -p clawhdf5-agent

# Full benchmark suite
cargo run --release --bin longmemeval_bench
cargo run --release --bin memory_arena
cargo run --release --bin footprint_bench
cargo run --release --bin consolidation_efficiency
cargo run --release --bin ephemeral_perf

---

h5bench-Equivalent I/O Benchmarks

Criterion harness mirroring h5bench serial workloads. clawhdf5 benchmarks dated 2026-07-01; libhdf5 1.14.6 head-to-head comparison dated 2026-06-30 (same hardware, same Criterion harness).

cargo bench -p clawhdf5-bench                         # clawhdf5-only
cargo bench -p clawhdf5-bench --features libhdf5-compare  # head-to-head

Sequential Read Throughput

Both read a 1-D contiguous f32 dataset. clawhdf5 parses from Vec<u8> (zero-copy); libhdf5 reads from a temp file including open + read + close overhead.

Workload	n=1K	n=10K	n=100K
clawhdf5 f32	634 ns / 5.9 GiB/s	2.44 µs / 15.3 GiB/s	24.5 µs / 15.2 GiB/s
libhdf5 f32	45.2 µs / 85 MiB/s	47.8 µs / 799 MiB/s	73.9 µs / 5.0 GiB/s
Speedup	71×	20×	3.0×
clawhdf5 f64	743 ns / 10.0 GiB/s	4.17 µs / 17.8 GiB/s	43.3 µs / 17.2 GiB/s
clawhdf5 from_disk (f64, OS I/O)	—	10.1 µs / 7.4 GiB/s	77.6 µs / 9.6 GiB/s
clawhdf5 hyperslab (f64, 10% slice)	—	4.09 µs / 1.8 GiB/s	50.1 µs / 1.5 GiB/s

libhdf5 f64 comparison excluded — clawhdf5's datatype encoding differs from libhdf5's (known gap), making cross-format reads unreliable for comparison.

Chunked Read Throughput

Matrix size	Latency	Throughput
64×64 f32	6.39 µs	2.4 GiB/s
256×256 f32	41.7 µs	5.9 GiB/s
512×512 f32	176 µs	5.5 GiB/s

Sequential Write Throughput

Both write to disk. At 100K elements both converge on the OS write() syscall ceiling.

Workload	n=1K	n=10K	n=100K
clawhdf5 f32	9.44 µs / 404 MiB/s	25 µs / 1.49 GiB/s	228 µs / 1.63 GiB/s
libhdf5 f32	77.9 µs / 49 MiB/s	87.8 µs / 435 MiB/s	214 µs / 1.74 GiB/s
Speedup	8.2×	3.5×	≈ tie
clawhdf5 f64 embeddings	6.50 µs (n=128)	8.67 µs (n=512) / 450 MiB/s	10.27 µs (n=1K) / 761 MiB/s

Chunked Write: Codec Comparison (with auto-shuffle)

Auto-shuffle is applied before all compression codecs by default — AoS→SoA byte transpose, implements byte-grouping pre-filter per arXiv:2506.18062. Shuffle dramatically improves throughput for float/int data by creating long runs of similar bytes.

Matrix size	Zstd-3 + shuffle	Deflate-6 + shuffle	Speedup
32×32 f32	48 µs / 81 MiB/s	39 µs / 100 MiB/s	Deflate 1.23× faster (small chunk)
128×128 f32	148 µs / 422 MiB/s	153 µs / 407 MiB/s	Parity
512×512 f32	1.34 ms / 748 MiB/s	1.39 ms / 719 MiB/s	Zstd 1.04× faster

Impact of auto-shuffle vs no-shuffle baseline:

Matrix size	Zstd-3 speedup	Deflate-6 speedup
32×32	+19%	+38%
128×128	+25%	+204%
512×512	+25%	+157%

Both codecs perform at parity at large sizes (~720–750 MiB/s). Use .with_zstd(3) or .with_deflate(6) for write-heavy workloads. Use .without_shuffle() only for byte arrays or data that doesn't benefit from AoS→SoA transposition.

Chunked Write vs libhdf5 (deflate-6)

clawhdf5 compresses all chunks in memory and issues a single write(). libhdf5 flushes each chunk individually via its Virtual File Layer (one pwrite() per chunk).

Matrix	clawhdf5 deflate-6 + shuffle	libhdf5 deflate-6	Speedup
32×32 f32	39 µs / 100 MiB/s	172 µs / 23 MiB/s	4.4×
128×128 f32	153 µs / 407 MiB/s	3,150 µs / 20 MiB/s	20.6×
512×512 f32	1,390 µs / 719 MiB/s	53,300 µs / 19 MiB/s	38.4×

The 32×32 speedup (4.4×) is lower than the 512×512 speedup (38.4×) because shuffle adds overhead that dominates at 4 KB chunks. libhdf5 was benchmarked without shuffle. The speedup compounds with matrix size because libhdf5's per-chunk VFL overhead is proportional to chunk count while clawhdf5's single-pass cost is constant.

Codec Comparison: Pcodec vs Zstd-3

Pcodec (arXiv:2502.06112) is a pure-Rust lossless numerical codec with 30–94% better compression ratio than Zstd for f32/f64 columns. Both sides benchmarked without auto-shuffle here (shuffle degrades Pcodec which handles byte organization internally; Zstd-3 without shuffle numbers shown for an apples-to-apples comparison).

Matrix size	Pcodec	Zstd-3 (no shuffle)	Winner
32×32 f32	95 µs / 41 MiB/s	57 µs / 68 MiB/s	Zstd-3 (1.66×)
128×128 f32	528 µs / 118 MiB/s	179 µs / 349 MiB/s	Zstd-3 (2.95×)
512×512 f32	1.69 ms / 591 MiB/s	1.64 ms / 610 MiB/s	Parity (3% diff)

Pcodec's fixed per-chunk distributional analysis overhead (~400 µs) dominates at 32×32 (4 KB). At 512×512 (1 MB) the speeds converge. Pcodec's advantage is compression ratio, not encode speed — less data on disk means faster reads and lower storage cost. Enable with .with_pcodec() for write-once/read-many workloads (embedding archives, scientific datasets).

Metadata Throughput

clawhdf5 accumulates all metadata in memory and serializes in one pass. libhdf5 acquires a global file mutex and flushes to disk on every attribute write or group creation.

Attributes and datasets (k = attribute or dataset count):

Workload	k=4	k=16	k=64	k=128
clawhdf5 attrs_write (i64)	8.05 µs / 494 Kop/s	17.2 µs / 932 Kop/s	49.2 µs / 1.30 Mop/s	87.3 µs / 1.47 Mop/s
libhdf5 attrs_write	100 µs / 40 Kop/s	170 µs / 94 Kop/s	472 µs / 136 Kop/s	929 µs / 138 Kop/s
Speedup	12.4×	9.9×	9.6×	10.6×
clawhdf5 attrs_read	1.06 µs / 3.78 Mop/s	3.64 µs / 4.39 Mop/s	15.7 µs / 4.08 Mop/s	31.3 µs / 4.09 Mop/s
clawhdf5 string_attrs (write+read)	5.17 µs / 774 Kop/s	16.5 µs / 967 Kop/s	33.6 µs / 951 Kop/s	—
clawhdf5 multi_dataset_write	10.1 µs / 397 Kop/s	31.5 µs / 508 Kop/s	104 µs / 614 Kop/s	—

Groups (k = group count):

Workload	k=4	k=16	k=32	k=64
clawhdf5 groups_create	12.1 µs / 330 Kop/s	33.7 µs / 475 Kop/s	66.7 µs / 480 Kop/s	121 µs / 529 Kop/s
libhdf5 groups_create	140 µs / 28 Kop/s	433 µs / 37 Kop/s	690 µs / 46 Kop/s	1,340 µs / 48 Kop/s
Speedup	11.6×	12.8×	9.5×	11.1×
clawhdf5 groups_traverse	664 ns / 6.0 Mop/s	3.55 µs / 4.5 Mop/s	4.87 µs / 6.6 Mop/s	10.6 µs / 6.0 Mop/s

---

vs libhdf5 Summary

Workload	clawhdf5	libhdf5	Speedup
Sequential read, 1K f32	634 ns	45.2 µs	71×
Sequential read, 100K f32	24.5 µs · 15.2 GiB/s	73.9 µs · 5.0 GiB/s	3.0×
Sequential write, 100K f32	228 µs · 1.63 GiB/s	214 µs · 1.74 GiB/s	≈ tie
Chunked write deflate-6, 512×512	1,390 µs · 719 MiB/s	53,300 µs · 19 MiB/s	38.4×
Attribute write, 128 attrs	87.3 µs · 1.47 Mop/s	929 µs · 138 Kop/s	10.6×
Group create, 64 groups	121 µs · 529 Kop/s	1,340 µs · 48 Kop/s	11.1×

Why the Gaps

Metadata (10–13×): libhdf5 was designed for MPI parallel filesystems where every metadata write must be immediately visible to other processes. It acquires a global file mutex and flushes to disk per operation. clawhdf5 builds the entire file in memory and writes it in one shot — no locking, no flushing, no C heap allocation per message.

Chunked compressed write (4–38×): libhdf5 writes each chunk individually through its VFL (Virtual File Layer), one pwrite() per chunk. clawhdf5 compresses all chunks in memory (Rayon parallel when > 2 chunks), lays them out contiguously, and issues a single write(). The speedup compounds with matrix size: libhdf5's per-chunk overhead is proportional to chunk count while clawhdf5's architectural cost is constant.

Small reads (20–71×): libhdf5's per-open overhead (chunk cache init, SWMR lock, metadata read) dominates at sub-millisecond payloads. clawhdf5 has no global state — File::from_bytes() starts parsing immediately.

Large contiguous writes (≈ tie at 100K): Both are bottlenecked by the OS write() syscall to the page cache. There is no algorithmic headroom above ~1.7 GiB/s on this hardware.

Caveats

• libhdf5 f64 read comparison excluded — clawhdf5's f32 datatype encoding differs from libhdf5's (known compatibility gap). f64 results are clawhdf5-only.
• Serial benchmarks. clawhdf5 uses Rayon for chunk compression when > 2 chunks; that parallelism is already reflected in the chunked write numbers.
• clawhdf5 reads from Vec<u8> (zero-copy from mmap in production); libhdf5 reads from a temp file. This gives clawhdf5 a structural read advantage that reflects realistic API usage.

---

Independent Validation: tank (Ryzen 7 7800X3D), 2026-08-03

The vs libhdf5 Summary numbers above were re-run on a second, independently administered machine (tank: AMD Ryzen 7 7800X3D, 8C/16T, Ubuntu 26.04, libhdf5 1.14.6 via apt) to confirm they reproduce off the original i7-12650H box, and to add benchmark coverage for two claims that a documentation review found were not traceable to any dated benchmark run (see git history around 2026-08-03 for context). This section documents both.

Reproduction of the vs-libhdf5 Summary table

Workload	clawhdf5 (tank)	libhdf5 (tank)	Speedup (tank)	Speedup (i7-12650H, above)
Sequential read, 1K f32	553 ns	44.2 µs	79.9×	71×
Sequential read, 100K f32	23.3 µs	63.6 µs	2.7×	3.0×
Sequential write, 100K f32	210 µs	189 µs	≈ tie (clawhdf5 ~11% behind)	≈ tie (clawhdf5 ~7% behind)
Chunked write deflate-6, 512×512	1.44 ms	65.0 ms	45.3×	38.4×
Attribute write, 128 attrs	85.2 µs	877 µs	10.3×	10.6×
Group create, 64 groups	130 µs	1.37 ms	10.6×	11.1×

Five of six rows land within ~15% of the original i7-12650H figures — consistent with normal cross-machine variance, not a methodology artifact. The chunked-write row moved further (38.4× → 45.3×, +18%): tank's libhdf5 per-chunk write cost scales worse relative to its own sequential-write throughput than on the i7, likely IPC/ memory-subsystem dependent. Both figures are real and dated; we report both rather than picking one.

New coverage: replacing the retracted "metadata parse / 308×" and "zero-copy mmap / 313 ns" claims

An earlier README revision cited 19 ns vs 2,080 µs (labeled, incorrectly, 308×) for "metadata parse," and 313 ns for "zero-copy mmap" — neither figure traced to any benchmark in this file. Both have been retracted from the README. In their place, two new Criterion benchmarks were added (crates/clawhdf5-bench/benches/h5bench_meta.rs, crates/clawhdf5-bench/benches/h5bench_read.rs) and run on tank:

metadata_open_from_disk — opens a small file from disk (std::fs::read / hdf5::File::open) and resolves one attribute. Both sides pay real OS I/O, unlike the retracted claim.

Operation	clawhdf5	libhdf5	Speedup
Open file + read 1 attribute	4.01 µs	39.3 µs	9.8×

metadata_parse_in_memory (clawhdf5-only) — times File::from_bytes() alone, given bytes already resident in memory, i.e. header-parse cost with disk I/O excluded. There is no fair libhdf5-side equivalent (its API has no "parse from an in-memory buffer, skip the OS open" path), so this is reported standalone rather than as a speedup multiple — this is the honest version of what the old 19 ns number was trying to claim.

Operation	clawhdf5 (in-memory, no I/O)
Parse superblock + resolve 1 attribute	549 ns

read_zerocopy_mmap — opens via MmapFile and reads an f64 dataset through read_f64_zerocopy(), summing every element to force the mapped pages to actually fault in (returning only a slice length, as an earlier draft of this benchmark did, would repeat the exact "measures nothing" mistake being fixed here).

n (f64 elements)	clawhdf5 mmap (zerocopy, page-fault-forced)	clawhdf5 (Vec<u8> copy)	libhdf5 (disk open + copy)
1,000	7.86 µs	4.50 µs	44.2 µs
10,000	19.0 µs	9.53 µs	47.1 µs
100,000	112 µs	72.0 µs	81.2 µs

Honest result: at these sizes, forcing full materialization through the mmap path is not faster than the plain Vec<u8> copy path — mmap()/page-fault overhead per call outweighs the copy it avoids. This contradicts the retracted 313 ns claim outright and is a genuinely useful finding: MmapFile's real advantage is avoiding the allocation/copy for large files or sparse access patterns (lower peak RSS, share pages across processes), not raw single-shot read latency at these sizes. No README claim is made from this row; it's recorded here for the record and to keep future readers from reintroducing the old number.

Reproduce:

cargo bench -p clawhdf5-bench --features libhdf5-compare --bench h5bench_meta -- metadata_open_from_disk
cargo bench -p clawhdf5-bench --features libhdf5-compare --bench h5bench_meta -- metadata_parse_in_memory
cargo bench -p clawhdf5-bench --features libhdf5-compare --bench h5bench_read -- read_zerocopy_mmap

Independent Validation: tank — LongMemEval & Vector Search (Ryzen 7 7800X3D), 2026-08-05

Re-running the "LongMemEval Results" and "SIMD & Parallelism" sections above on tank (AMD Ryzen 7 7800X3D, 8C/16T, Ubuntu 26.04, same machine as the vs-libhdf5 validation above) to give both sections the dated, hardware-cited, reproducible citation the top-of-file traceability note flags them as missing.

LongMemEval Results (reproduction)

cd benchmarks/longmemeval
wget https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_oracle.json
cargo run --release --bin longmemeval_bench

Recall numbers are deterministic (pure BM25 retrieval over a fixed dataset) and reproduce exactly. Scoring target as declared in the LongMemEval section above: retrieval recall, turn-level, k=10, longmemeval_oracle variant, BM25-only.

Metric	Turn-Level
Hit@1	52.6%
Hit@5	84.4%
Hit@10	90.4%
MRR	0.6597

Session-level figures are omitted here — they are degenerate on the oracle variant and have been retracted; see "Retracted: session-level recall and the MemX comparison" above.

Search latency (hardware-dependent, tank numbers):

Metric	avg	p50	p95	p99
Latency	2,431 µs	2,105 µs	7,250 µs	12,018 µs

Higher than the i7-12650H figures at the top of this file (avg 1,004 µs) despite tank's faster single-core performance elsewhere in this document — BM25 search latency here scales with per-question haystack size and this run's variance is wider (p99 is ~5x the mean), suggesting this metric is more sensitive to momentary scheduling/cache effects than the flat-array vector-search benchmarks. Recorded as-is rather than smoothed.

SIMD & Parallelism (reproduction, with a correction)

cargo bench -p clawhdf5-agent --bench bench -- "^(strategy_scalar_10k|strategy_simd_10k|strategy_rayon_10k|adaptive_search_10k|simd_cosine_100k|rayon_cosine_100k)$"

The original 10K table above compares named benchmarks (vector_search, rayon, strategy) that, on inspection, don't all exercise the same scalar-vs-SIMD-vs-parallel axis the table implies — several of the simd_cosine_10k/sequential_cosine_10k-style benchmarks actually call the same underlying function under different names. The adaptive_benches group's strategy_scalar_10k / strategy_simd_10k / strategy_rayon_10k benchmarks are the ones that genuinely hold the dataset fixed and vary only the SearchStrategy enum, so they're the correct apples-to-apples comparison — used here instead.

Strategy	Latency (tank)	vs Sequential
Sequential (scalar)	502 µs	1.0x
SIMD (auto-vectorized)	327 µs	1.53x
Rayon (parallel)	323 µs	1.55x
Adaptive (auto-select)	339 µs	1.48x

Honest finding: the speedup from SIMD/parallelism over scalar is real but smaller here (~1.5x) than the i7-12650H figures above (~2.0x). The Ryzen 7 7800X3D's large L3 cache (96MB 3D V-Cache) measurably narrows the gap versus a naive scalar loop compared to the i7 — this is a genuine hardware-dependent result, not a regression or measurement error, and is recorded rather than reconciled away.

At 100K, no strategy_* benchmark exists in the current suite (adaptive_benches only covers n=10,000), so this row uses the same simd_cosine_100k/ rayon_cosine_100k benchmarks as the original table — not a true scalar baseline, so no "vs Sequential" multiple is reported for it:

Strategy	Latency (tank)
SIMD	6.60 ms
Rayon parallel	4.73 ms

Vector Search Latency & Comparison to MemX (reproduction)

cargo bench -p clawhdf5-agent --bench bench -- "^(vector_search_1k|simd_cosine_10k|simd_cosine_100k|prenorm_search_10k|ivf_search_10k_nprobe10|ivf_search_100k_nprobe10|ivf_pq_search_100k|rairs_search_10k_nprobe10|bm25_search_10k)$"

Scale	Flat Search	Pre-norm	IVF (nprobe=10)	IVF-PQ	RAIRS
1K	47.8 µs	—	—	—	—
10K	501 µs	322 µs	24.8 µs	—	109 µs
100K	6.60 ms	—	608 µs	865 µs	—

(The 1K Pre-norm cell from the original table has no corresponding benchmark in the current suite — not re-verified, left blank rather than guessed.)

Same not-like-for-like caveat as the "Comparison to MemX" section at the top of this file applies — MemX's figure is end-to-end, these are a single component. Ratios are an order-of-magnitude indication, not a benchmark result.

Metric	MemX (claimed, end-to-end)	ClawhDF5 (tank, component only)	Ratio
100K flat search	<90 ms	6.60 ms	~14x
100K IVF-PQ search	—	865 µs	~104x
Keyword search 10K	1,100x improvement over unindexed	520 µs (BM25)	Comparable

Every figure in this subsection is faster than the corresponding i7-12650H number at the top of this file, consistent with the Ryzen 7 7800X3D's higher single-core throughput and larger cache observed in the vs-libhdf5 validation above.