- README's "HDF5 Core I/O" table claimed 19ns/2,080µs labeled 308× (real ratio ~109,000×) and a 313ns zero-copy mmap figure — neither traced to any dated benchmark in BENCHMARKS.md. Replaced the table wholesale with the existing "vs libhdf5 Summary" figures, relabeled from "h5py/C HDF5" to "libhdf5" (BENCHMARKS.md never benchmarks against h5py, only libhdf5 directly). - Added two new Criterion benchmarks to close the coverage gaps that produced the untraceable numbers: metadata_open_from_disk (I/O-inclusive, fair clawhdf5-vs-libhdf5 file-open comparison) and metadata_parse_in_memory (clawhdf5-only, explicitly labeled as excluding I/O) in h5bench_meta.rs; read_zerocopy_mmap in h5bench_read.rs (forces real page-ins by summing elements rather than just returning a slice length — the mmap path turns out to be slower than a plain copy at these sizes, an honest, unflattering but real result now documented instead of a fabricated 313ns). - Re-ran the full existing benchmark suite plus the two new ones on a second, independently administered machine (tank: Ryzen 7 7800X3D) to validate the numbers before publishing them. 5 of 6 rows landed within ~15% of the original i7-12650H figures; recorded both in BENCHMARKS.md's new "Independent Validation" section. README now cites the tank numbers. - Added a short top-of-file README callout naming both halves of the project (general-purpose HDF5 library vs. agent memory layer) with links to BENCHMARKS.md and the Crate Map, so a data-infra reader isn't 60% through a memory-store pitch before finding the part relevant to them. - Added one factual, no-names line noting benchmark numbers are being validated in collaboration with HDF5 Group engineers. - Fixed the same untraceable "2-300x faster than h5py/C HDF5" / "313 ns" claims in docs/QUICKSTART.md, one click from the README's own "New here?" link. Co-Authored-By: Claude Sonnet 5 <[email protected]>
27 KiB
ClawhDF5 Benchmark Results
Pure Rust. Zero C dependencies. Single file. Fast enough to forget it's there.
System: Intel i7-12650H (10C/16T, 4.7 GHz boost) · 32 GB DDR5 · Linux 6.8.0
Rust: 1.96.0-nightly (2026-03-14) · --release profile
Date: 2026-07-01
Vector Search Latency
Brute-force cosine similarity over 384-dimensional embeddings (OpenAI text-embedding-3-small size).
| Scale | Flat Search | Pre-norm | IVF (nprobe=10) | IVF-PQ | RAIRS |
|---|---|---|---|---|---|
| 1K | 54 µs | 62 µs | — | — | — |
| 10K | 753 µs | 706 µs | 27 µs | — | 159 µs |
| 100K | 11.4 ms | — | 1.32 ms | 1.19 ms | — |
Key insight: At 10K records (typical agent memory), IVF search delivers 27 µs — that's 26x faster than flat search. Even at 100K records, IVF-PQ keeps search under 1.2 ms.
Comparison to MemX (arxiv:2603.16171)
MemX claims end-to-end search under 90ms at 100K records (Rust + libSQL + FTS5).
| Metric | MemX (claimed) | ClawhDF5 | Speedup |
|---|---|---|---|
| 100K flat search | <90 ms | 11.4 ms | ~8x |
| 100K IVF-PQ search | — | 1.19 ms | ~76x |
| Keyword search 10K | 1,100x improvement over unindexed | 583 µs (BM25) | Comparable |
SIMD & Parallelism
384-dimensional cosine similarity at 10K scale.
| Strategy | Latency | vs Sequential |
|---|---|---|
| Sequential (scalar) | 1.07 ms | 1.0x |
| SIMD (auto-vectorized) | 545 µs | 2.0x |
| Rayon (parallel) | 553 µs | 1.9x |
| Adaptive (auto-select) | 564 µs | 1.9x |
At 100K:
| Strategy | Latency |
|---|---|
| SIMD | 13.7 ms |
| Rayon parallel | 8.3 ms |
Hybrid Search (Vector + BM25)
1K records, 384-dimensional embeddings with BM25 keyword index.
| Method | Latency | Notes |
|---|---|---|
| Weighted fusion | 198 µs | Original min-max normalization |
| RRF (k=60) | 222 µs | Reciprocal Rank Fusion — better quality, ~12% overhead |
| BM25-only 1K | 67 µs | Keyword search alone |
| Hybrid 10K | 2.04 ms | Full hybrid at 10K scale |
Knowledge Graph
Graph traversal and entity operations.
| Operation | Scale | Latency |
|---|---|---|
| BFS traversal | 100 entities | 5.4 µs |
| BFS traversal | 1,000 entities | 24 µs |
| Spreading activation | 100 entities | 16.9 µs |
| Entity resolution (Levenshtein) | 100 entities | 64 µs |
| Alias resolution (short query) | 100 aliases | 10.4 µs |
| Alias resolution (long query) | 100 aliases | 11.6 µs |
All graph operations complete in microseconds. Spreading activation across 100 entities with 5 propagation steps finishes in 17 µs.
Memory Consolidation
Hippocampal-inspired tiered memory management.
| Operation | Scale | Latency |
|---|---|---|
| Consolidation cycle | 100 records | 15 µs |
| Consolidation cycle | 1,000 records | 164 µs |
| Importance scoring | 100 records | 25 µs |
A full consolidation pass over 1,000 memories (eviction + promotion across Working → Episodic → Semantic) completes in 164 µs. This can run on every memory write without perceptible latency.
Temporal Index
Sorted timestamp index with binary search.
| Operation | Scale | Latency |
|---|---|---|
| Range query | 10K timestamps | 716 ns |
| Batch insert | 10K timestamps | 4.69 ms |
Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K records: 716 nanoseconds.
Write Path
HDF5 persistence with optional Write-Ahead Log.
| Operation | Latency | Notes |
|---|---|---|
| Single save (no WAL) | 61 µs | Direct HDF5 write (owned-Vec IO path) |
| Single save (with WAL) | 18 µs | WAL group-commit append; HDF5 write batched at flush |
| Batch 100 | 723 µs | 7.2 µs per record |
| Batch 1,000 | 6.17 ms | 6.2 µs per record |
| WAL save (1K existing) | 539 µs | Incremental append |
| WAL flush 100 entries | 787 µs | Merge WAL → HDF5 |
| Session tick 1K | 5.76 ms | Full session maintenance |
| Session tick 10K | 89.8 ms | Background operation |
Decision Gate
Trivial/non-trivial classification for memory write filtering.
| Check | Latency |
|---|---|
| Trivial skip ("ok", "yes") | 61 ns |
| Short phrase skip | 86 ns |
| Non-trivial pass | 705 ns |
| Ratio check | 488 ns |
Sub-microsecond filtering. The gate decides whether to save a memory in under 1 µs.
Memory Strategy
End-to-end strategy evaluation including embedding operations.
| Strategy | Condition | Latency |
|---|---|---|
| SaveEveryExchange (substantive) | Saves | 923 ns |
| SaveEveryExchange (trivial) | Skips | 67 ns |
| SaveOnSemanticShift (empty store) | Saves | 941 ns |
Summary
| Capability | Typical Latency | Scale |
|---|---|---|
| Full memory search | <1 ms | 10K records |
| Hybrid vector+keyword | <200 µs | 1K records |
| Knowledge graph query | <25 µs | 1K entities |
| Temporal range query | <1 µs | 10K timestamps |
| Memory write | <20 µs | Per record (WAL group-commit append) |
| Consolidation cycle | <165 µs | 1K records |
| Importance gate | <1 µs | Per record |
The entire memory pipeline — search, retrieve, re-rank, filter — runs in single-digit milliseconds at agent-typical scales. Fast enough that memory becomes invisible infrastructure.
Latency benchmarks generated with Criterion.rs (50-100 samples per benchmark). Results may vary by hardware.
LongMemEval Results
Dataset: LongMemEval oracle (500 questions, 6 question types, variable-length chat histories)
Mode: BM25-only retrieval — zero embeddings, vector_weight=0.0, keyword_weight=1.0
Reference: MemX (arxiv:2603.16171) with full embedding system: Hit@5=51.6%, MRR=0.380
Run:
cargo run --release --bin longmemeval_bench
Session-Level Recall (n=500)
| Metric | ClawhDF5 (BM25-only) |
|---|---|
| Hit@1 | 100.0% |
| Hit@5 | 100.0% |
| Hit@10 | 100.0% |
| MRR | 1.0000 |
Perfect session-level recall across all 500 questions and all 6 question types.
Turn-Level Recall (n=500)
| Metric | ClawhDF5 (BM25-only) | MemX (full system)¹ |
|---|---|---|
| Hit@1 | 52.6% | — |
| Hit@5 | 84.4% | 51.6% |
| Hit@10 | 90.4% | — |
| MRR | 0.6597 | 0.380 |
clawhdf5 outperforms MemX at turn-level retrieval — Hit@5 84.4% vs 51.6%, MRR 0.66 vs 0.38 — with BM25 alone, no embeddings needed.
¹ MemX uses dense embeddings + FTS5 + four-factor re-ranking. Our BM25-only result exceeds their full pipeline.
Per-Type Breakdown (session-level)
| Question Type | N | Hit@1 | Hit@5 | Hit@10 | MRR |
|---|---|---|---|---|---|
| single-session-user | 70 | 100.0% | 100.0% | 100.0% | 1.0000 |
| single-session-assistant | 56 | 100.0% | 100.0% | 100.0% | 1.0000 |
| single-session-preference | 30 | 100.0% | 100.0% | 100.0% | 1.0000 |
| temporal-reasoning | 133 | 100.0% | 100.0% | 100.0% | 1.0000 |
| multi-session | 133 | 100.0% | 100.0% | 100.0% | 1.0000 |
| knowledge-update | 78 | 100.0% | 100.0% | 100.0% | 1.0000 |
Search Latency (LongMemEval, n=500 queries)
| Metric | Latency |
|---|---|
| avg | 1,004 µs |
| p50 | 1,017 µs |
| p95 | 2,031 µs |
| p99 | 2,912 µs |
Sub-millisecond median search across variable-length chat histories.
Multi-Session Benchmark (MemoryArena)
Dataset: Deterministic synthetic conversations — 50 sessions × ~20 turns = 999 turns Topics: Personal info, food preferences, music, travel, work/schedule, hobbies Queries: 35 questions across 4 types
Run:
cargo run --release --bin memory_arena
Results by Query Type
| Query Type | N | Hit@1 | Hit@5 | Hit@10 | MRR | Avg Latency |
|---|---|---|---|---|---|---|
| single-session | 25 | 40.0% | 92.0% | 100.0% | 0.5788 | 7,853 µs |
| multi-session | 5 | 40.0% | 60.0% | 80.0% | 0.5333 | 7,887 µs |
| temporal | 3 | 33.3% | 66.7% | 66.7% | 0.5000 | 7,899 µs |
| knowledge-update | 2 | 0.0% | 50.0% | 50.0% | 0.2500 | 7,899 µs |
| OVERALL | 35 | 37.1% | 82.9% | 91.4% | 0.5468 | 7,870 µs |
Key findings:
- Hit@10 of 91.4% across all query types with BM25-only (no embeddings)
- Single-session recall strongest at 100% Hit@10
- Knowledge-update hardest (requires temporal disambiguation) — would improve significantly with vector similarity
- Latency dominated by BM25 index build over 999 turns (~7.9 ms)
Memory Footprint
HDF5 file size at various record counts — 384-dimensional embeddings, 200-char text.
Run:
cargo run --release --bin footprint_bench
Uncompressed (no WAL)
| Records | File Size | Raw Data | Bytes/Record | Throughput |
|---|---|---|---|---|
| 100 | 176.4 KB | 169.5 KB | 1.8 KB | 100,000 rec/s |
| 1K | 1.7 MB | 1.7 MB | 1.8 KB | 109,643 rec/s |
| 10K | 17.0 MB | 16.6 MB | 1.7 KB | 118,100 rec/s |
| 50K | 85.0 MB | 82.8 MB | 1.7 KB | 110,723 rec/s |
| 100K | 169.8 MB | 165.6 MB | 1.7 KB | 111,422 rec/s |
1.7 KB per record — HDF5 overhead is near-zero. Ingestion throughput exceeds 100K records/sec.
With Gzip Compression (level 6)
| Records | Compressed | Ratio | Bytes/Record |
|---|---|---|---|
| 100 | 31.5 KB | 5.37x | 323 B |
| 1K | 277.1 KB | 6.12x | 283 B |
| 10K | 2.7 MB | 6.17x | 281 B |
| 50K | 13.4 MB | 6.17x | 281 B |
| 100K | 26.9 MB | 6.15x | 282 B |
6.2x compression ratio — 100K agent memories in 27 MB compressed.
Text Length Comparison (10K records, no compression)
| Text Length | File Size | Bytes/Record | Throughput |
|---|---|---|---|
| short (50 chars) | 15.6 MB | 1.6 KB | 177,925 rec/s |
| medium (200 chars) | 17.0 MB | 1.7 KB | 172,152 rec/s |
| long (1000 chars) | 24.6 MB | 2.5 KB | 157,807 rec/s |
WAL Overhead (1K records)
| Mode | File Size | Ingest Time | Overhead |
|---|---|---|---|
| No WAL | 1.7 MB | 5.7 ms | — |
| With WAL | 1.7 MB + 9 B WAL | 5.3 ms | ±8% (negligible) |
Consolidation Efficiency
Hippocampal-inspired memory consolidation improves both retrieval quality and search speed.
Run:
cargo run --release --bin consolidation_efficiency
Retrieval Quality Before vs. After Consolidation
Setup: 1,000 records (10 signal + 990 noise), working_capacity=100
| Metric | Before | After | Delta |
|---|---|---|---|
| Records in store | 1,000 | 100 | −90% |
| Hit@1 | 100.0% | 100.0% | — |
| Hit@5 | 100.0% | 100.0% | — |
| Hit@10 | 100.0% | 100.0% | — |
| MRR | 1.0000 | 1.0000 | — |
| Search latency | 2,752 µs | 312 µs | 8.8x faster |
Signal records survive consolidation because they are accessed 15+ times, giving them high decay scores. 900 noise records evicted, search speeds up 8.8x, and zero quality loss — perfect recall maintained.
Consolidation Cycle Time
| Records | Cycle Time | Evictions | Promotions |
|---|---|---|---|
| 100 | 21 µs | 100 | 0 |
| 1K | 345 µs | 1,000 | 0 |
| 10K | 17.3 ms | 10,000 | 0 |
Ephemeral Tier (Redis Comparison)
In-memory key-value store with TTL, capacity eviction, and embedding search. No network hop, no serialization — direct HashMap operations.
Run:
cargo run --release --bin ephemeral_perf
Latency Comparison
| Operation | clawhdf5 Ephemeral | Redis (single-node)¹ | Speedup |
|---|---|---|---|
| SET | 356 ns/op | ~25,000 ns/op | 70x |
| GET (hit) | 179 ns/op | ~25,000 ns/op | 140x |
| GET (miss) | 62 ns/op | ~25,000 ns/op | 403x |
| DELETE | 124 ns/op | ~25,000 ns/op | 202x |
| SET+embedding | 268 ns/op | N/A | — |
¹ Redis latency includes network round-trip (loopback). clawhdf5 ephemeral is in-process — no network.
Throughput
| Operation | ops/sec |
|---|---|
| SET | 2,810,649 |
| GET | 5,584,684 |
| DELETE | 8,093,731 |
| SET+EMB (384d) | 3,725,877 |
Embedding Search (ephemeral tier)
| Scale | Latency |
|---|---|
| 10K entries @ 384d | 2.9 ms/query |
Cross-Platform Notes
Run:
./benchmarks/cross_platform.sh [--full] [--output results.json]
Measured Platforms
| Platform | CPU | 10K IVF Search | Notes |
|---|---|---|---|
| Linux x86_64 | Intel i7-12650H (10C, 4.7 GHz) | 27 µs | Primary CI target |
| macOS aarch64 | Apple M3 Max (14C) | ~18 µs | ~33% faster via NEON SIMD |
Reproducibility
rustup override set nightly
# Latency benchmarks (Criterion)
cargo bench -p clawhdf5-agent
# Full benchmark suite
cargo run --release --bin longmemeval_bench
cargo run --release --bin memory_arena
cargo run --release --bin footprint_bench
cargo run --release --bin consolidation_efficiency
cargo run --release --bin ephemeral_perf
h5bench-Equivalent I/O Benchmarks
Criterion harness mirroring h5bench serial workloads. clawhdf5 benchmarks dated 2026-07-01; libhdf5 1.14.6 head-to-head comparison dated 2026-06-30 (same hardware, same Criterion harness).
cargo bench -p clawhdf5-bench # clawhdf5-only
cargo bench -p clawhdf5-bench --features libhdf5-compare # head-to-head
Sequential Read Throughput
Both read a 1-D contiguous f32 dataset. clawhdf5 parses from Vec<u8> (zero-copy);
libhdf5 reads from a temp file including open + read + close overhead.
| Workload | n=1K | n=10K | n=100K |
|---|---|---|---|
| clawhdf5 f32 | 634 ns / 5.9 GiB/s | 2.44 µs / 15.3 GiB/s | 24.5 µs / 15.2 GiB/s |
| libhdf5 f32 | 45.2 µs / 85 MiB/s | 47.8 µs / 799 MiB/s | 73.9 µs / 5.0 GiB/s |
| Speedup | 71× | 20× | 3.0× |
| clawhdf5 f64 | 743 ns / 10.0 GiB/s | 4.17 µs / 17.8 GiB/s | 43.3 µs / 17.2 GiB/s |
| clawhdf5 from_disk (f64, OS I/O) | — | 10.1 µs / 7.4 GiB/s | 77.6 µs / 9.6 GiB/s |
| clawhdf5 hyperslab (f64, 10% slice) | — | 4.09 µs / 1.8 GiB/s | 50.1 µs / 1.5 GiB/s |
libhdf5 f64 comparison excluded — clawhdf5's datatype encoding differs from libhdf5's (known gap), making cross-format reads unreliable for comparison.
Chunked Read Throughput
| Matrix size | Latency | Throughput |
|---|---|---|
| 64×64 f32 | 6.39 µs | 2.4 GiB/s |
| 256×256 f32 | 41.7 µs | 5.9 GiB/s |
| 512×512 f32 | 176 µs | 5.5 GiB/s |
Sequential Write Throughput
Both write to disk. At 100K elements both converge on the OS write() syscall ceiling.
| Workload | n=1K | n=10K | n=100K |
|---|---|---|---|
| clawhdf5 f32 | 9.44 µs / 404 MiB/s | 25 µs / 1.49 GiB/s | 228 µs / 1.63 GiB/s |
| libhdf5 f32 | 77.9 µs / 49 MiB/s | 87.8 µs / 435 MiB/s | 214 µs / 1.74 GiB/s |
| Speedup | 8.2× | 3.5× | ≈ tie |
| clawhdf5 f64 embeddings | 6.50 µs (n=128) | 8.67 µs (n=512) / 450 MiB/s | 10.27 µs (n=1K) / 761 MiB/s |
Chunked Write: Codec Comparison (with auto-shuffle)
Auto-shuffle is applied before all compression codecs by default — AoS→SoA byte transpose, implements byte-grouping pre-filter per arXiv:2506.18062. Shuffle dramatically improves throughput for float/int data by creating long runs of similar bytes.
| Matrix size | Zstd-3 + shuffle | Deflate-6 + shuffle | Speedup |
|---|---|---|---|
| 32×32 f32 | 48 µs / 81 MiB/s | 39 µs / 100 MiB/s | Deflate 1.23× faster (small chunk) |
| 128×128 f32 | 148 µs / 422 MiB/s | 153 µs / 407 MiB/s | Parity |
| 512×512 f32 | 1.34 ms / 748 MiB/s | 1.39 ms / 719 MiB/s | Zstd 1.04× faster |
Impact of auto-shuffle vs no-shuffle baseline:
| Matrix size | Zstd-3 speedup | Deflate-6 speedup |
|---|---|---|
| 32×32 | +19% | +38% |
| 128×128 | +25% | +204% |
| 512×512 | +25% | +157% |
Both codecs perform at parity at large sizes (~720–750 MiB/s). Use .with_zstd(3) or
.with_deflate(6) for write-heavy workloads. Use .without_shuffle() only for byte arrays
or data that doesn't benefit from AoS→SoA transposition.
Chunked Write vs libhdf5 (deflate-6)
clawhdf5 compresses all chunks in memory and issues a single write(). libhdf5 flushes each
chunk individually via its Virtual File Layer (one pwrite() per chunk).
| Matrix | clawhdf5 deflate-6 + shuffle | libhdf5 deflate-6 | Speedup |
|---|---|---|---|
| 32×32 f32 | 39 µs / 100 MiB/s | 172 µs / 23 MiB/s | 4.4× |
| 128×128 f32 | 153 µs / 407 MiB/s | 3,150 µs / 20 MiB/s | 20.6× |
| 512×512 f32 | 1,390 µs / 719 MiB/s | 53,300 µs / 19 MiB/s | 38.4× |
The 32×32 speedup (4.4×) is lower than the 512×512 speedup (38.4×) because shuffle adds overhead that dominates at 4 KB chunks. libhdf5 was benchmarked without shuffle. The speedup compounds with matrix size because libhdf5's per-chunk VFL overhead is proportional to chunk count while clawhdf5's single-pass cost is constant.
Codec Comparison: Pcodec vs Zstd-3
Pcodec (arXiv:2502.06112) is a pure-Rust lossless numerical codec with 30–94% better compression ratio than Zstd for f32/f64 columns. Both sides benchmarked without auto-shuffle here (shuffle degrades Pcodec which handles byte organization internally; Zstd-3 without shuffle numbers shown for an apples-to-apples comparison).
| Matrix size | Pcodec | Zstd-3 (no shuffle) | Winner |
|---|---|---|---|
| 32×32 f32 | 95 µs / 41 MiB/s | 57 µs / 68 MiB/s | Zstd-3 (1.66×) |
| 128×128 f32 | 528 µs / 118 MiB/s | 179 µs / 349 MiB/s | Zstd-3 (2.95×) |
| 512×512 f32 | 1.69 ms / 591 MiB/s | 1.64 ms / 610 MiB/s | Parity (3% diff) |
Pcodec's fixed per-chunk distributional analysis overhead (~400 µs) dominates at 32×32 (4 KB).
At 512×512 (1 MB) the speeds converge. Pcodec's advantage is compression ratio, not encode
speed — less data on disk means faster reads and lower storage cost. Enable with
.with_pcodec() for write-once/read-many workloads (embedding archives, scientific datasets).
Metadata Throughput
clawhdf5 accumulates all metadata in memory and serializes in one pass. libhdf5 acquires a global file mutex and flushes to disk on every attribute write or group creation.
Attributes and datasets (k = attribute or dataset count):
| Workload | k=4 | k=16 | k=64 | k=128 |
|---|---|---|---|---|
| clawhdf5 attrs_write (i64) | 8.05 µs / 494 Kop/s | 17.2 µs / 932 Kop/s | 49.2 µs / 1.30 Mop/s | 87.3 µs / 1.47 Mop/s |
| libhdf5 attrs_write | 100 µs / 40 Kop/s | 170 µs / 94 Kop/s | 472 µs / 136 Kop/s | 929 µs / 138 Kop/s |
| Speedup | 12.4× | 9.9× | 9.6× | 10.6× |
| clawhdf5 attrs_read | 1.06 µs / 3.78 Mop/s | 3.64 µs / 4.39 Mop/s | 15.7 µs / 4.08 Mop/s | 31.3 µs / 4.09 Mop/s |
| clawhdf5 string_attrs (write+read) | 5.17 µs / 774 Kop/s | 16.5 µs / 967 Kop/s | 33.6 µs / 951 Kop/s | — |
| clawhdf5 multi_dataset_write | 10.1 µs / 397 Kop/s | 31.5 µs / 508 Kop/s | 104 µs / 614 Kop/s | — |
Groups (k = group count):
| Workload | k=4 | k=16 | k=32 | k=64 |
|---|---|---|---|---|
| clawhdf5 groups_create | 12.1 µs / 330 Kop/s | 33.7 µs / 475 Kop/s | 66.7 µs / 480 Kop/s | 121 µs / 529 Kop/s |
| libhdf5 groups_create | 140 µs / 28 Kop/s | 433 µs / 37 Kop/s | 690 µs / 46 Kop/s | 1,340 µs / 48 Kop/s |
| Speedup | 11.6× | 12.8× | 9.5× | 11.1× |
| clawhdf5 groups_traverse | 664 ns / 6.0 Mop/s | 3.55 µs / 4.5 Mop/s | 4.87 µs / 6.6 Mop/s | 10.6 µs / 6.0 Mop/s |
vs libhdf5 Summary
| Workload | clawhdf5 | libhdf5 | Speedup |
|---|---|---|---|
| Sequential read, 1K f32 | 634 ns | 45.2 µs | 71× |
| Sequential read, 100K f32 | 24.5 µs · 15.2 GiB/s | 73.9 µs · 5.0 GiB/s | 3.0× |
| Sequential write, 100K f32 | 228 µs · 1.63 GiB/s | 214 µs · 1.74 GiB/s | ≈ tie |
| Chunked write deflate-6, 512×512 | 1,390 µs · 719 MiB/s | 53,300 µs · 19 MiB/s | 38.4× |
| Attribute write, 128 attrs | 87.3 µs · 1.47 Mop/s | 929 µs · 138 Kop/s | 10.6× |
| Group create, 64 groups | 121 µs · 529 Kop/s | 1,340 µs · 48 Kop/s | 11.1× |
Why the Gaps
Metadata (10–13×): libhdf5 was designed for MPI parallel filesystems where every metadata write must be immediately visible to other processes. It acquires a global file mutex and flushes to disk per operation. clawhdf5 builds the entire file in memory and writes it in one shot — no locking, no flushing, no C heap allocation per message.
Chunked compressed write (4–38×): libhdf5 writes each chunk individually through its VFL
(Virtual File Layer), one pwrite() per chunk. clawhdf5 compresses all chunks in memory (Rayon
parallel when > 2 chunks), lays them out contiguously, and issues a single write(). The
speedup compounds with matrix size: libhdf5's per-chunk overhead is proportional to chunk count
while clawhdf5's architectural cost is constant.
Small reads (20–71×): libhdf5's per-open overhead (chunk cache init, SWMR lock, metadata
read) dominates at sub-millisecond payloads. clawhdf5 has no global state — File::from_bytes()
starts parsing immediately.
Large contiguous writes (≈ tie at 100K): Both are bottlenecked by the OS write() syscall
to the page cache. There is no algorithmic headroom above ~1.7 GiB/s on this hardware.
Caveats
- libhdf5 f64 read comparison excluded — clawhdf5's f32 datatype encoding differs from libhdf5's (known compatibility gap). f64 results are clawhdf5-only.
- Serial benchmarks. clawhdf5 uses Rayon for chunk compression when > 2 chunks; that parallelism is already reflected in the chunked write numbers.
- clawhdf5 reads from
Vec<u8>(zero-copy from mmap in production); libhdf5 reads from a temp file. This gives clawhdf5 a structural read advantage that reflects realistic API usage.
Independent Validation: tank (Ryzen 7 7800X3D), 2026-08-03
The vs libhdf5 Summary numbers above were re-run on a second, independently
administered machine (tank: AMD Ryzen 7 7800X3D, 8C/16T, Ubuntu 26.04, libhdf5
1.14.6 via apt) to confirm they reproduce off the original i7-12650H box, and to
add benchmark coverage for two claims that a documentation review found were not
traceable to any dated benchmark run (see git history around 2026-08-03 for context).
This section documents both.
Reproduction of the vs-libhdf5 Summary table
| Workload | clawhdf5 (tank) | libhdf5 (tank) | Speedup (tank) | Speedup (i7-12650H, above) |
|---|---|---|---|---|
| Sequential read, 1K f32 | 553 ns | 44.2 µs | 79.9× | 71× |
| Sequential read, 100K f32 | 23.3 µs | 63.6 µs | 2.7× | 3.0× |
| Sequential write, 100K f32 | 210 µs | 189 µs | ≈ tie (clawhdf5 ~11% behind) | ≈ tie (clawhdf5 ~7% behind) |
| Chunked write deflate-6, 512×512 | 1.44 ms | 65.0 ms | 45.3× | 38.4× |
| Attribute write, 128 attrs | 85.2 µs | 877 µs | 10.3× | 10.6× |
| Group create, 64 groups | 130 µs | 1.37 ms | 10.6× | 11.1× |
Five of six rows land within ~15% of the original i7-12650H figures — consistent with normal cross-machine variance, not a methodology artifact. The chunked-write row moved further (38.4× → 45.3×, +18%): tank's libhdf5 per-chunk write cost scales worse relative to its own sequential-write throughput than on the i7, likely IPC/ memory-subsystem dependent. Both figures are real and dated; we report both rather than picking one.
New coverage: replacing the retracted "metadata parse / 308×" and "zero-copy mmap / 313 ns" claims
An earlier README revision cited 19 ns vs 2,080 µs (labeled, incorrectly, 308×)
for "metadata parse," and 313 ns for "zero-copy mmap" — neither figure traced to
any benchmark in this file. Both have been retracted from the README. In their
place, two new Criterion benchmarks were added
(crates/clawhdf5-bench/benches/h5bench_meta.rs,
crates/clawhdf5-bench/benches/h5bench_read.rs) and run on tank:
metadata_open_from_disk — opens a small file from disk (std::fs::read /
hdf5::File::open) and resolves one attribute. Both sides pay real OS I/O, unlike
the retracted claim.
| Operation | clawhdf5 | libhdf5 | Speedup |
|---|---|---|---|
| Open file + read 1 attribute | 4.01 µs | 39.3 µs | 9.8× |
metadata_parse_in_memory (clawhdf5-only) — times File::from_bytes() alone,
given bytes already resident in memory, i.e. header-parse cost with disk I/O
excluded. There is no fair libhdf5-side equivalent (its API has no "parse from an
in-memory buffer, skip the OS open" path), so this is reported standalone rather
than as a speedup multiple — this is the honest version of what the old 19 ns
number was trying to claim.
| Operation | clawhdf5 (in-memory, no I/O) |
|---|---|
| Parse superblock + resolve 1 attribute | 549 ns |
read_zerocopy_mmap — opens via MmapFile and reads an f64 dataset through
read_f64_zerocopy(), summing every element to force the mapped pages to actually
fault in (returning only a slice length, as an earlier draft of this benchmark did,
would repeat the exact "measures nothing" mistake being fixed here).
| n (f64 elements) | clawhdf5 mmap (zerocopy, page-fault-forced) | clawhdf5 (Vec<u8> copy) |
libhdf5 (disk open + copy) |
|---|---|---|---|
| 1,000 | 7.86 µs | 4.50 µs | 44.2 µs |
| 10,000 | 19.0 µs | 9.53 µs | 47.1 µs |
| 100,000 | 112 µs | 72.0 µs | 81.2 µs |
Honest result: at these sizes, forcing full materialization through the mmap path
is not faster than the plain Vec<u8> copy path — mmap()/page-fault overhead
per call outweighs the copy it avoids. This contradicts the retracted 313 ns
claim outright and is a genuinely useful finding: MmapFile's real advantage is
avoiding the allocation/copy for large files or sparse access patterns (lower peak
RSS, share pages across processes), not raw single-shot read latency at these
sizes. No README claim is made from this row; it's recorded here for the record
and to keep future readers from reintroducing the old number.
Reproduce:
cargo bench -p clawhdf5-bench --features libhdf5-compare --bench h5bench_meta -- metadata_open_from_disk
cargo bench -p clawhdf5-bench --features libhdf5-compare --bench h5bench_meta -- metadata_parse_in_memory
cargo bench -p clawhdf5-bench --features libhdf5-compare --bench h5bench_read -- read_zerocopy_mmap