osobhandClaude Opus 5.5
dce5559ff2
Every undated or pre-September section re-run on one machine on one day
(tank, AMD Ryzen 7 7800X3D, 2026-09-24, commit 5c8323c), 24 commands run
serially with the load average checked before each, with the command
recorded for each section. A separate check traced every changed number
back to the raw output; its corrections are applied (e.g. the on-disk
~820 B/record is float16 plus always-deflated text on a synthetic corpus
of 40 distinct texts, not float16 alone).
Two apparent regressions were isolated rather than published:
- knowledge-graph traversal: a real bug, fixed in the previous commit;
- the write path: v2.3.0 built and run on the same machine measures the
same as today, so the old 18 us / 6.17 ms figures (undated, other
hardware) are not reproducible; float16 adds ~2 us per save and the
int8 index nothing (both isolated by switching the bench's config).
Also:
- new multimodal_bench: cross-modal search at 1K/10K records, which the
README claimed but nothing measured;
- footprint_bench reports whether it built float16 or f32 stores and
takes --f32 (it kept printing "f32" after the default changed);
- README: performance tables, the "Why" table figures and the SQLite
migration section (from the previous migrate commit);
- CHANGELOG for this branch.
Not re-run: consolidation_efficiency's 100K row and its memory-reduction
part (stopped for time), and cross_platform.sh.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2193 lines
107 KiB
Markdown
2193 lines
107 KiB
Markdown
# ClawhDF5 Benchmark Results
|
||
|
||
> Pure Rust. Zero C dependencies. Single file. Fast enough to forget it's there.
|
||
|
||
**System (original run):** Intel i7-12650H (10C/16T, 4.7 GHz boost) · 32 GB DDR5 · Linux 6.8.0
|
||
**Rust:** 1.96.0-nightly (2026-03-14) · `--release` profile
|
||
**Date:** 2026-07-01
|
||
|
||
Sections that carry their own "Measured …" line were measured there instead,
|
||
not on the system above. The 2026-09-24 re-run was on tank (AMD Ryzen 7
|
||
7800X3D, 8C/16T, Linux 7.0.0-34-generic, rustc 1.98.1) at commit 5c8323c,
|
||
serially, with no `--warm-up-time`/`--measurement-time` overrides, waiting
|
||
before each command while the 1-minute load average was above 2.0. The
|
||
`bench` targets ran at Criterion's defaults (3 s warm-up, 5 s target
|
||
measurement time, 100 samples). Every `memory_bench` group that was run
|
||
(`vector_search_latency`, `hybrid_search`, `knowledge_graph`,
|
||
`consolidation`, `temporal`) and the multi-modal group set
|
||
`sample_size(50)`, so those sections took 50 samples, not 100. The 5 s is a
|
||
target: Criterion stretched it where 5 s could not hold the samples it needed
|
||
(for example 8.5 s for `vector_search_latency/bench_cosine_search/100k`,
|
||
15.5 s for the 1K consolidation cycle, 252 s for `tick_session_10k`).
|
||
|
||
> **Traceability note:** a section meets the dated, hardware-cited,
|
||
> reproducible standard when it gives an explicit date, the machine, and a
|
||
> runnable command for its results. As of the 2026-09-24 re-run on tank that
|
||
> covers Vector Search Latency, Comparison to MemX, SIMD & Parallelism, Hybrid
|
||
> Search, Knowledge Graph, Memory Consolidation, Temporal Index, Write Path,
|
||
> Decision Gate, Memory Strategy, Summary (derived from those), LongMemEval
|
||
> Results (the BM25 rows), Multi-Session Benchmark, Memory Footprint (on disk
|
||
> and in memory), Consolidation Efficiency, Ephemeral Tier, Multi-modal Search,
|
||
> the Search and Read harnesses, and the "h5bench-Equivalent I/O Benchmarks"
|
||
> and "Independent Validation: tank" sections. What does not yet meet that bar:
|
||
> the LongMemEval rows that need real embeddings (not re-run here, except the
|
||
> dated float16 comparison), the Consolidation Efficiency 100K cycle row and
|
||
> memory-reduction part (the 2026-09-24 run was stopped before it produced
|
||
> them), the int8 side of "Quantising the index copy" (not re-run), and the
|
||
> i7-12650H and macOS M3 Max rows under Cross-Platform Notes. That is a
|
||
> known, tracked documentation gap, not a claim that those numbers are wrong.
|
||
>
|
||
> **Correctness note (2026-08-06).** Being dated and reproducible is necessary but
|
||
> not sufficient — a number can be perfectly reproducible and still measure the
|
||
> wrong thing. A methodology audit found two such cases and both have been
|
||
> retracted in place: the session-level LongMemEval figures (degenerate on the
|
||
> oracle variant) and the MemX retrieval comparison (mismatched granularity and
|
||
> corpus). Every cross-system comparison in this file now carries an explicit
|
||
> scoping caveat. Where a section states a scoring target, that declaration is the
|
||
> contract — read it before citing the number.
|
||
|
||
---
|
||
|
||
## Memory footprint
|
||
|
||
`cargo run --release -p clawhdf5-bench --bin search_harness -- --footprint --full`,
|
||
384-dim `f32`. The figure that matters is **reopened**: a store loaded from
|
||
disk, which is what a long-lived process holds.
|
||
|
||
Measured with a counting global allocator, not RSS. RSS cannot see this from
|
||
inside one process — freeing a large structure returns its pages to the
|
||
allocator's pool rather than to the OS, so allocating the next one shows no
|
||
change at all. Measured that way a store holding the corpus twice and one
|
||
holding it once came out *identical* (1.00x both), which is how the first
|
||
attempt at this measurement went.
|
||
|
||
| N | vectors (raw) | reopened, before | reopened, after |
|
||
|---:|---:|---:|---:|
|
||
| 1 000 | 1 MiB | 5 MiB (3.41x) | 4 MiB (2.39x) |
|
||
| 10 000 | 15 MiB | 50 MiB (3.43x) | 35 MiB (2.42x) |
|
||
| 100 000 | 146 MiB | 505 MiB (3.44x) | **357 MiB (2.43x)** |
|
||
|
||
The cache stored every embedding twice — once as a `Vec<Vec<f32>>` and once
|
||
flattened for the batched kernels, kept in lock-step on every push, update and
|
||
compaction. Storing only the flat buffer and indexing into it gives back
|
||
almost exactly one copy of the corpus (148 MiB at 100k) and one heap
|
||
allocation per entry. Recall and query latency are unchanged.
|
||
|
||
What remains at 2.43x: the flat vectors (1.0x), the HNSW index's own copy of
|
||
them (1.0x), and text, ids and graph (~0.4x). The index copy is the next
|
||
target — it is what a quantised or borrowed representation would address.
|
||
|
||
**Current figures (`f32` index).** Measured 2026-09-24 on tank (AMD Ryzen 7
|
||
7800X3D), commit 5c8323c:
|
||
|
||
> **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --footprint --full`
|
||
|
||
| N | vectors (raw) | entries MiB | store MiB | indexes MiB | reopened MiB | peak during open MiB | reopened / raw |
|
||
|---:|---:|---:|---:|---:|---:|---:|---:|
|
||
| 1 000 | 1 | 2 | 0 | 2 | 4 | 5 | 2.40x |
|
||
| 10 000 | 15 | 17 | 12 | 32 | 44 | 61 | 3.03x |
|
||
| 100 000 | 146 | 172 | 60 | 266 | 399 | 562 | 2.72x |
|
||
|
||
These are the "reopened, f32" figures in the next table, reproduced exactly.
|
||
They are above the "after" column of the table above (44 MiB vs 35 MiB at 10K,
|
||
399 vs 357 MiB at 100K): the 2.43x was measured for 2e7e045, and the store
|
||
already measured 399 MiB (2.72x) when the int8 index landed later the same
|
||
day (c0a9206). The git log does not say what changed in between.
|
||
|
||
### Quantising the index copy (`quantized_index`)
|
||
|
||
`MemoryConfig::quantized_index` stores the index's copy as `i8` instead of
|
||
`f32`. Same harness, same binary, `--footprint --full` with and without
|
||
`--int8`:
|
||
|
||
| N | vectors (raw) | indexes, f32 | indexes, int8 | reopened, f32 | reopened, int8 |
|
||
|---:|---:|---:|---:|---:|---:|
|
||
| 1 000 | 1 MiB | 2 MiB | 1 MiB | 4 MiB (2.40x) | 2 MiB (1.64x) |
|
||
| 10 000 | 15 MiB | 32 MiB | 14 MiB | 44 MiB (3.03x) | 27 MiB (1.81x) |
|
||
| 100 000 | 146 MiB | 266 MiB | **123 MiB** | 399 MiB (2.72x) | **256 MiB (1.74x)** |
|
||
|
||
The scale is **per row**, not global. A unit-length row in `d` dimensions has
|
||
components around `1/sqrt(d)`, so a fixed `[-1, 1]` scale spends fewer than 12
|
||
of the 255 levels on a 128-dimensional vector: measured against an exact
|
||
ranking that gives 0.35 top-10 overlap — unusable. Scaling each row by its own
|
||
largest component brings the same measurement to 0.99.
|
||
|
||
Quantised distances still cost recall on their own, and **`ef` does not buy it
|
||
back**, because the loss is in the distances rather than in the graph
|
||
(`--ann-only --full`, N = 100 000):
|
||
|
||
| ef | recall@10, f32 | recall@10, int8 | recall@10, int8 + re-score |
|
||
|---:|---:|---:|---:|
|
||
| 32 | 0.9775 | 0.9415 | 0.9785 |
|
||
| 64 | 0.9945 | 0.9625 | 0.9940 |
|
||
| 128 | 0.9995 | 0.9670 | 0.9990 |
|
||
| 256 | 0.9995 | 0.9670 (ceiling) | 0.9990 |
|
||
|
||
Re-scoring closes the gap: the store already holds the exact embeddings, so
|
||
the query path re-scores the candidate pool against them before fusion. That
|
||
is done automatically whenever the index is quantised.
|
||
|
||
**On AVX2 this costs nothing — it pays.** The first measurement of this put
|
||
the cost at ~13% of QPS and ~16% of build time, but that compared a scalar
|
||
int8 loop against `clawhdf5-accel`'s hand-written AVX2 kernels for `f32`:
|
||
the gap was a missing kernel, not a property of int8. With
|
||
`clawhdf5_accel::dot_i8` (AVX2: sign-extend to `i16`, then `madd_epi16`),
|
||
medians of three alternating runs at N = 100 000, same binary:
|
||
|
||
| | f32 | int8 | int8 + re-score |
|
||
|---|---:|---:|---:|
|
||
| build | 3197 ms | **1778 ms** | 1826 ms |
|
||
| QPS at ef = 64 | 13 399 | 29 195 | **21 848** |
|
||
| recall@10 at ef = 64 | 0.9945 | 0.9625 | **0.9940** |
|
||
|
||
So at equal recall the quantised index answers **1.63x as many queries per
|
||
second**, builds **1.8x faster**, and holds a quarter of the vectors. (Compare
|
||
only at equal `ef`: with re-scoring the harness raises `ef` to at least the
|
||
candidate pool, so the `ef = 16` and `ef = 32` rows are not like-for-like.)
|
||
|
||
**Re-check, 2026-09-24.** Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D),
|
||
commit 5c8323c. The `f32` memory columns of the first table reproduce exactly
|
||
(see [Memory footprint](#memory-footprint)). Only the `f32` side of the speed
|
||
table was re-run, once, not as alternating medians:
|
||
|
||
> **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --full`
|
||
|
||
At N = 100 000, `ef = 64`, the `f32` index measured recall@10 0.9945 (same),
|
||
19 001 QPS (was 13 399) and a 2700.8 ms build (was 3197 ms). The QPS figure
|
||
moved by more than 20%; the log does not explain it, and a single run is not
|
||
the same-moment paired comparison the table above is. The int8 columns were
|
||
not re-run, so the 1.63x ratio has not been re-checked against the new `f32`
|
||
figure.
|
||
|
||
#### On ARM (Raspberry Pi 5, Cortex-A76)
|
||
|
||
`dot_i8` has two aarch64 kernels: `SDOT` for CPUs with the ARMv8.2
|
||
dot-product extension (Cortex-A76 and later, Neoverse-N1, all Apple Silicon)
|
||
and plain NEON (`vmull_s8` + `vpadalq_s16`) otherwise. Medians of three runs
|
||
at N = 100 000, ef = 64, recall@10 0.9940 in every int8 row against f32's
|
||
0.9945:
|
||
|
||
| int8 kernel | build | QPS | vs f32 |
|
||
|---|---:|---:|---:|
|
||
| *(f32 baseline)* | 33 413 ms | 6 164 | 1.00x |
|
||
| scalar (what v2.7.0 shipped) | 18 950 ms | ~6 190 | 1.00x |
|
||
| plain NEON | ~17 000 ms | 6 640 | 1.08x |
|
||
| **SDOT** | **14 464 ms** | **7 267** | **1.18x** |
|
||
|
||
These are Pi 5 numbers, not "ARM" numbers: a Pi has far less memory bandwidth
|
||
and cache than an Apple M-series or a flagship phone, so the ratios will move
|
||
on other hardware. The plain-NEON row is that code on an A76 with `SDOT`
|
||
disabled, not a measurement of a pre-A76 core.
|
||
|
||
**A correction.** Until this was measured, this section said aarch64 "falls
|
||
back to the scalar loop, where the original trade still applies" — that is,
|
||
that quantised search was ~13% slower than f32 on ARM. That was extrapolated
|
||
from x86 and it was wrong. On x86-64 the portable baseline is SSE2 while the
|
||
f32 kernels are hand-written AVX2, so scalar int8 lost; on aarch64 NEON *is*
|
||
the baseline, the compiler vectorises the scalar loop well, and scalar int8
|
||
already matched f32 for search while building 1.76x faster.
|
||
|
||
So on every configuration measured — x86-64 AVX2, and Pi 5 with each of the
|
||
three int8 kernels — the quantised index is at least as fast as f32 at equal
|
||
recall, builds faster, and holds a quarter of the vectors.
|
||
|
||
A measurement trap worth recording: the synthetic `clustered` generator in the
|
||
`clawhdf5-ann` tests draws clusters far tighter than any real embedding, so
|
||
neighbours there sit closer together than the quantisation error and top-10
|
||
*identity* is noise. Scored on that fixture int8 looks catastrophic (0.57
|
||
overlap) — a fact about the fixture, not the storage. The tests use random
|
||
vectors, and recall is measured against brute-force ground truth rather than
|
||
against the f32 index, whose own approximation errors a re-scored search is
|
||
entitled to get right.
|
||
|
||
### Search options: source filters, re-ranking, confidence
|
||
|
||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D). `HDF5Memory::search`
|
||
with `SearchOptions`, clustered 384-dim data, k = 10, Hebbian boosting off.
|
||
Filters keep 50%, 10% or 1% of the store at random, or two whole clusters
|
||
chosen *away* from each query's own — the case an ANN index handles worst,
|
||
because nothing it finds near the query is allowed. Recall is vector-only
|
||
against an exact scan of the allowed records; latency is full hybrid search
|
||
(vector + BM25). 200 queries, medians of three runs (recall was identical in
|
||
every run).
|
||
|
||
```bash
|
||
cargo run --release -p clawhdf5-bench --bin search_harness -- --options-study --full
|
||
```
|
||
|
||
| N | options | filtered recall@10 | p50 ms | p99 ms |
|
||
|---:|---|---:|---:|---:|
|
||
| 10 000 | no filter | 1.0000 | 0.488 | 0.518 |
|
||
| 10 000 | random 50% | 1.0000 | 0.520 | 0.563 |
|
||
| 10 000 | random 10% | 1.0000 | 0.342 | 0.367 |
|
||
| 10 000 | random 1% | 1.0000 | 0.220 | 0.239 |
|
||
| 10 000 | 2 clusters away from the query | 1.0000 | 0.252 | 0.278 |
|
||
| 10 000 | re-rank | — | 0.562 | 0.674 |
|
||
| 10 000 | re-rank + confidence | — | 0.554 | 0.575 |
|
||
| 100 000 | no filter | 0.9995 | 4.600 | 5.245 |
|
||
| 100 000 | random 50% | 1.0000 | 4.770 | 5.614 |
|
||
| 100 000 | random 10% | 1.0000 | 3.244 | 4.008 |
|
||
| 100 000 | random 1% | 1.0000 | 2.288 | 2.943 |
|
||
| 100 000 | 2 clusters away from the query | 1.0000 | 2.298 | 3.048 |
|
||
| 100 000 | re-rank | — | 4.747 | 5.411 |
|
||
| 100 000 | re-rank + confidence | — | 4.748 | 5.526 |
|
||
|
||
A filtered search finds the exact filtered top 10 and is never slower than an
|
||
unfiltered one. The filter applies before ranking — over-fetching the index in
|
||
proportion to what the filter removes, and scanning the allowed records
|
||
exactly whenever that costs fewer distance evaluations than the index would
|
||
(roughly `pool × M`). The first version compared the over-fetch to the store
|
||
size instead, and that measured badly at 100K: the 1% filter took 5.9 ms at
|
||
recall 0.9965 and the away-from-query filter 12.3 ms at 0.976, both through an
|
||
index asked for ~16 000 candidates, where scanning the few hundred or thousand
|
||
allowed records is exact and cheap. Re-ranking a 3k candidate pool and
|
||
confidence rejection add about 3%.
|
||
|
||
### float16 embedding storage (`MemoryConfig::float16`)
|
||
|
||
Measured 2026-09-23 on tank (AMD Ryzen 7 7800X3D). The same clustered
|
||
384-dim data in an `f32` store and a `float16` store, both with the default
|
||
int8 index and Hebbian boosting off (so every query sees the same store).
|
||
Recall is vector-only `hybrid_search` against an exact scan of the original
|
||
`f32` vectors, 200 queries. Six runs, three with each store going first;
|
||
medians. Nothing depended on the order.
|
||
|
||
```bash
|
||
cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full
|
||
cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full --f16-first
|
||
```
|
||
|
||
| N | embeddings | file MiB | checkpoint ms | open ms | recall@10 | top-10 overlap | hybrid p50 ms |
|
||
|---:|---|---:|---:|---:|---:|---:|---:|
|
||
| 1 000 | f32 | 1.6 | 8 | 1.6 | 1.0000 | | 0.072 |
|
||
| 1 000 | float16 | 0.8 | 6 | 1.9 | 0.9980 | 0.9980 | 0.072 |
|
||
| 10 000 | f32 | 15.4 | 66 | 15.2 | 1.0000 | | 0.495 |
|
||
| 10 000 | float16 | 8.2 | 45 | 18.1 | 1.0000 | 1.0000 | 0.494 |
|
||
| 100 000 | f32 | 154.0 | 752 | 299.8 | 0.9940 | | 4.676 |
|
||
| 100 000 | float16 | **80.8** | **512** | **252.1** | 0.9990 | 0.9940 | 4.654 |
|
||
|
||
The file is 48% smaller, checkpoints write less and open reads less. At
|
||
small N opening is slightly slower (widening halves costs more than the I/O it
|
||
saves: +3 ms at 10K). Recall does not move: half precision keeps about three
|
||
significant digits, far finer than the gaps between neighbours on unit-length
|
||
embeddings. Recall was identical in every run; the 0.999 against 0.994 at
|
||
100K is two slightly different HNSW graphs, not an improvement to claim.
|
||
|
||
The in-memory cache holds the half-rounded values, so the store searches the
|
||
same before and after a reopen; RAM use is unchanged (the cache is still
|
||
`f32`). What `float16` saves is disk, and the I/O that goes with it.
|
||
|
||
**On real embeddings.** The table above is synthetic clustered data. The full
|
||
LongMemEval haystack (`longmemeval_s`, 500 questions, ~494 turns each) with
|
||
real all-MiniLM-L6-v2 embeddings, run once with `f32` stores and once with
|
||
`--float16`, measured 2026-09-24 on tank (embeddings on an RTX 5060 Ti):
|
||
|
||
```bash
|
||
cargo run --release -p clawhdf5-bench --bin longmemeval_bench --features embeddings-cuda -- \
|
||
benchmarks/longmemeval/longmemeval_s.json --embeddings weights/all-minilm-l6-v2 [--float16]
|
||
```
|
||
|
||
| mode | turn Hit@1 | turn Hit@5 | turn Hit@10 | turn MRR | session Hit@5 | session MRR |
|
||
|---|---:|---:|---:|---:|---:|---:|
|
||
| Hybrid 0.4 / 0.6, f32 | 51.6% | 81.4% | 87.8% | 0.6430 | 96.8% | 0.9347 |
|
||
| Hybrid 0.4 / 0.6, float16 | 51.6% | 81.4% | 87.8% | 0.6430 | 96.8% | 0.9347 |
|
||
| Vector only, f32 | 36.0% | 71.8% | 81.6% | 0.5031 | 94.2% | 0.8901 |
|
||
| Vector only, float16 | 36.0% | 71.8% | 81.6% | 0.5031 | 94.2% | 0.8901 |
|
||
|
||
All eight modes the harness runs (BM25, vector, hybrid, RRF, stemmed, and
|
||
both re-rank variants) were identical at every Hit@k and MRR, turn and session
|
||
level, except RRF's session MRR (0.9253 vs 0.9254) and one or two flips in
|
||
which of two gold sessions ranks first, out of ~320. Those flips show the
|
||
half-precision path was in effect; they do not change a single hit. The f32
|
||
run reproduces the published hybrid numbers exactly.
|
||
|
||
### Opening a store (`read_from_disk`)
|
||
|
||
`HDF5Memory::open` memory-mapped the file, copied the whole mapping into a
|
||
`Vec`, and handed that to `File::from_bytes` — while `File::open` memory-maps
|
||
the file itself. Dropping the copy takes **store open from 455 ms to 327 ms**
|
||
at 100 000 x 384 (`--e2e-only --full`; two runs after the change, 326.8 and
|
||
328.1 ms).
|
||
|
||
It does **not** lower the process's peak memory, which is worth stating
|
||
precisely because it is the obvious thing to assume. The harness now reports a
|
||
high-water mark alongside the retained figure:
|
||
|
||
| N | reopened MiB | peak during open MiB |
|
||
|---:|---:|---:|
|
||
| 1 000 | 4 | 5 |
|
||
| 10 000 | 44 | 61 |
|
||
| 100 000 | 399 | 562 |
|
||
|
||
Reproduced exactly on 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit
|
||
5c8323c, by `search_harness -- --footprint --full` (table under
|
||
[Memory footprint](#memory-footprint)).
|
||
|
||
The peak is set *after* the parse, by the index build, so a buffer allocated
|
||
and freed during the parse never reaches the high-water mark. Holding a
|
||
deliberate extra copy of the file across the whole parse leaves the peak
|
||
unmoved, which is how this was confirmed rather than assumed. What the change
|
||
saves is the copy itself: a full-file memcpy on every open, and the transient
|
||
that goes with it.
|
||
|
||
## Read harness
|
||
|
||
Produced by `cargo run --release -p clawhdf5-bench --bin read_harness`: a 4096 x
|
||
2048 `f64` dataset (64 MB) written three ways, read in full and through four
|
||
hyperslab selections, each from a fresh file handle. The last column is the
|
||
point: does a selection cost what the *selection* costs?
|
||
|
||
### Baseline (v2.4.0): every selection decodes the whole dataset
|
||
|
||
4096 x 2048 f64 (64 MB per dataset), chunks 256 x 256, file 129 MB
|
||
|
||
| layout | read | selected | time ms | MB/s of selection | vs full read |
|
||
|---|---|---:|---:|---:|---:|
|
||
| chunked + deflate | full (first) | 64 MB | 181.8 | 352 | |
|
||
| chunked + deflate | full (repeat) | 64 MB | 162.1 | 395 | 1.00x |
|
||
| chunked + deflate | 64 x 64 window (1 chunk) | 0.03 MB | 104.89 | 0 | 0.577x |
|
||
| chunked + deflate | 512 x 512 window (4-9 chunks) | 2.00 MB | 110.26 | 18 | 0.606x |
|
||
| chunked + deflate | one row | 0.02 MB | 105.81 | 0 | 0.582x |
|
||
| chunked + deflate | one column | 0.03 MB | 108.37 | 0 | 0.596x |
|
||
| chunked | full (first) | 64 MB | 97.4 | 657 | |
|
||
| chunked | full (repeat) | 64 MB | 86.7 | 738 | 1.00x |
|
||
| chunked | 64 x 64 window (1 chunk) | 0.03 MB | 40.97 | 1 | 0.420x |
|
||
| chunked | 512 x 512 window (4-9 chunks) | 2.00 MB | 44.66 | 45 | 0.458x |
|
||
| chunked | one row | 0.02 MB | 30.88 | 1 | 0.317x |
|
||
| chunked | one column | 0.03 MB | 30.27 | 1 | 0.311x |
|
||
| contiguous | full (first) | 64 MB | 57.6 | 1112 | |
|
||
| contiguous | full (repeat) | 64 MB | 53.5 | 1195 | 1.00x |
|
||
| contiguous | 64 x 64 window (1 chunk) | 0.03 MB | 30.97 | 1 | 0.538x |
|
||
| contiguous | 512 x 512 window (4-9 chunks) | 2.00 MB | 31.64 | 63 | 0.550x |
|
||
| contiguous | one row | 0.02 MB | 31.90 | 0 | 0.554x |
|
||
| contiguous | one column | 0.03 MB | 29.36 | 1 | 0.510x |
|
||
|
||
### After: partial reads
|
||
|
||
Only the rows of a contiguous dataset, or the chunks, that overlap the
|
||
selection's bounding box are read/decoded. A 64 x 64 window of the compressed
|
||
dataset: **105 -> 0.39 ms**; one row: **106 -> 2.7 ms**; one column:
|
||
**108 -> 5.2 ms**. (Absolute full-read times differ between the two runs
|
||
because the machine's speed drifted; compare the *vs full read* column.)
|
||
|
||
4096 x 2048 f64 (64 MB per dataset), chunks 256 x 256, file 129 MB
|
||
|
||
| layout | read | selected | time ms | MB/s of selection | vs full read |
|
||
|---|---|---:|---:|---:|---:|
|
||
| chunked + deflate | full (first) | 64 MB | 112.5 | 569 | |
|
||
| chunked + deflate | full (repeat) | 64 MB | 104.5 | 612 | 1.00x |
|
||
| chunked + deflate | 64 x 64 window (1 chunk) | 0.03 MB | 0.39 | 81 | 0.003x |
|
||
| chunked + deflate | 512 x 512 window (4-9 chunks) | 2.00 MB | 4.85 | 412 | 0.043x |
|
||
| chunked + deflate | one row | 0.02 MB | 2.69 | 6 | 0.024x |
|
||
| chunked + deflate | one column | 0.03 MB | 5.23 | 6 | 0.046x |
|
||
| chunked | full (first) | 64 MB | 70.0 | 915 | |
|
||
| chunked | full (repeat) | 64 MB | 61.7 | 1037 | 1.00x |
|
||
| chunked | 64 x 64 window (1 chunk) | 0.03 MB | 0.06 | 541 | 0.001x |
|
||
| chunked | 512 x 512 window (4-9 chunks) | 2.00 MB | 1.99 | 1005 | 0.028x |
|
||
| chunked | one row | 0.02 MB | 0.05 | 285 | 0.001x |
|
||
| chunked | one column | 0.03 MB | 0.45 | 69 | 0.006x |
|
||
| contiguous | full (first) | 64 MB | 60.3 | 1062 | |
|
||
| contiguous | full (repeat) | 64 MB | 56.4 | 1134 | 1.00x |
|
||
| contiguous | 64 x 64 window (1 chunk) | 0.03 MB | 0.08 | 396 | 0.001x |
|
||
| contiguous | 512 x 512 window (4-9 chunks) | 2.00 MB | 2.12 | 944 | 0.035x |
|
||
| contiguous | one row | 0.02 MB | 0.03 | 576 | 0.000x |
|
||
| contiguous | one column | 0.03 MB | 2.55 | 12 | 0.042x |
|
||
|
||
### After: parallel cached decode, fewer copies (full reads)
|
||
|
||
Full-read times, old and new binaries run alternately at the same moment (this
|
||
machine's absolute speed drifts over a long session, so only same-moment
|
||
comparisons mean anything):
|
||
|
||
| layout (64 MB `f64`) | before | after |
|
||
|---|---:|---:|
|
||
| chunked + deflate | 110 ms | 69 ms |
|
||
| chunked | 72 ms | 60 ms |
|
||
| contiguous | 56 ms | 30 ms |
|
||
|
||
What changed: the facade's cached read path decompressed chunks one at a time
|
||
(only the uncached reader was parallel) and pushed every chunk through a 16 MiB
|
||
cache that a 64 MB read simply churns; it now decodes cache misses in parallel
|
||
batches and caches only datasets that fit. Unfiltered chunks are copied
|
||
straight from the file bytes instead of via two intermediate buffers. A
|
||
contiguous dataset is converted straight from the file bytes (one copy instead
|
||
of two), and the native-endian conversions no longer zero a buffer they are
|
||
about to overwrite.
|
||
|
||
### Current: read harness (2026-09-24)
|
||
|
||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||
|
||
> **Run:** `cargo run --release -p clawhdf5-bench --bin read_harness`
|
||
|
||
4096 x 2048 f64 (64 MB per dataset), chunks 256 x 256, file 129 MB
|
||
|
||
| layout | read | selected | time ms | MB/s of selection | vs full read |
|
||
|---|---|---:|---:|---:|---:|
|
||
| chunked + deflate | full (first) | 64 MB | 65.6 | 975 | |
|
||
| chunked + deflate | full (repeat) | 64 MB | 63.2 | 1013 | 1.00x |
|
||
| chunked + deflate | 64 x 64 window (1 chunk) | 0.03 MB | 0.18 | 177 | 0.003x |
|
||
| chunked + deflate | 512 x 512 window (4-9 chunks) | 2.00 MB | 4.12 | 485 | 0.063x |
|
||
| chunked + deflate | one row | 0.02 MB | 0.99 | 16 | 0.015x |
|
||
| chunked + deflate | one column | 0.03 MB | 2.02 | 15 | 0.031x |
|
||
| chunked | full (first) | 64 MB | 60.5 | 1057 | |
|
||
| chunked | full (repeat) | 64 MB | 56.7 | 1129 | 1.00x |
|
||
| chunked | 64 x 64 window (1 chunk) | 0.03 MB | 0.10 | 325 | 0.002x |
|
||
| chunked | 512 x 512 window (4-9 chunks) | 2.00 MB | 3.19 | 628 | 0.053x |
|
||
| chunked | one row | 0.02 MB | 0.05 | 284 | 0.001x |
|
||
| chunked | one column | 0.03 MB | 0.49 | 64 | 0.008x |
|
||
| contiguous | full (first) | 64 MB | 30.1 | 2124 | |
|
||
| contiguous | full (repeat) | 64 MB | 26.1 | 2452 | 1.00x |
|
||
| contiguous | 64 x 64 window (1 chunk) | 0.03 MB | 0.08 | 394 | 0.003x |
|
||
| contiguous | 512 x 512 window (4-9 chunks) | 2.00 MB | 3.36 | 595 | 0.112x |
|
||
| contiguous | one row | 0.02 MB | 0.03 | 486 | 0.001x |
|
||
| contiguous | one column | 0.03 MB | 2.61 | 12 | 0.087x |
|
||
|
||
Full reads match the "after" column above (chunked + deflate 69 ms then, 63.2
|
||
to 65.6 ms now). The compressed selections are faster than in "After: partial
|
||
reads" (64 x 64 window 0.39 -> 0.18 ms, one row 2.69 -> 0.99 ms, one column
|
||
5.23 -> 2.02 ms) and match the 2026-09-23 figures in
|
||
[Deflate backend](#deflate-backend-zlib-rs-vs-zlib-ng) (0.18, 1.00 and 2.01
|
||
ms), which were taken after the zlib-rs switch and the one-shot codec calls
|
||
described there; this run does not isolate which change accounts for it.
|
||
|
||
The uncompressed windows went the other way. Against "After: partial reads",
|
||
the 512 x 512 window is about 60% slower (chunked 1.99 -> 3.19 ms, contiguous
|
||
2.12 -> 3.36 ms) and the chunked 64 x 64 window went from 0.06 to 0.10 ms.
|
||
The rows and columns of the uncompressed layouts are within 20% (chunked
|
||
column 0.45 -> 0.49 ms, contiguous column 2.55 -> 2.61 ms). This run does not
|
||
explain the slower windows.
|
||
|
||
## Search harness baseline (v2.3.0)
|
||
|
||
Produced by `cargo run --release -p clawhdf5-bench --bin search_harness -- --full`
|
||
on deterministic **clustered** synthetic data (384-dim, unit-normalised; points =
|
||
cluster centre + noise — uniform random vectors are nearly equidistant in high
|
||
dimension and say nothing about embeddings). Recall is measured against an exact
|
||
brute-force scan, 200 queries. This is the *before* picture for the search
|
||
hot-path work; every change to that path should be justified by a re-run.
|
||
|
||
Two things stand out:
|
||
|
||
* **HNSW recall does not respond to `ef`** and degrades sharply with size
|
||
(0.87 → 0.67 → 0.31 recall@10 at 1K / 10K / 100K). Latency plateaus at the same
|
||
point, i.e. the search exhausts the nodes it can reach: on clustered data the
|
||
graph is poorly connected. The index selects neighbours by plain top-M
|
||
distance rather than the HNSW paper's diversity heuristic.
|
||
* **End-to-end `hybrid_search` is ~1000x slower than its vector stage** (49 ms
|
||
vs ~0.03 ms at 10K; 884 ms at 100K). Each query rebuilds the BM25 index from
|
||
scratch and rewrites the whole `.h5` file. The first query after `open()`
|
||
additionally rebuilds the HNSW index (10.5 s at 100K).
|
||
|
||
### HNSW, N = 1000, dim = 384, M = 16, ef_construction = 64
|
||
|
||
build: 72.4 ms (13818 vectors/s) · exact scan: 3854 QPS, p50 258 µs
|
||
|
||
| ef | recall@10 | QPS | p50 µs | p99 µs |
|
||
|---:|---:|---:|---:|---:|
|
||
| 16 | 0.8710 | 59484 | 16 | 31 |
|
||
| 32 | 0.8730 | 46302 | 21 | 25 |
|
||
| 64 | 0.8730 | 31683 | 31 | 44 |
|
||
| 128 | 0.8730 | 24715 | 40 | 49 |
|
||
| 256 | 0.8730 | 24788 | 40 | 50 |
|
||
|
||
### HNSW, N = 10000, dim = 384, M = 16, ef_construction = 64
|
||
|
||
build: 802.5 ms (12461 vectors/s) · exact scan: 418 QPS, p50 2363 µs
|
||
|
||
| ef | recall@10 | QPS | p50 µs | p99 µs |
|
||
|---:|---:|---:|---:|---:|
|
||
| 16 | 0.6695 | 44031 | 19 | 51 |
|
||
| 32 | 0.6705 | 45066 | 22 | 30 |
|
||
| 64 | 0.6705 | 32746 | 30 | 41 |
|
||
| 128 | 0.6705 | 27542 | 36 | 51 |
|
||
| 256 | 0.6705 | 27754 | 36 | 49 |
|
||
|
||
### HNSW, N = 100000, dim = 384, M = 16, ef_construction = 64
|
||
|
||
build: 9752.6 ms (10254 vectors/s) · exact scan: 40 QPS, p50 24648 µs
|
||
|
||
| ef | recall@10 | QPS | p50 µs | p99 µs |
|
||
|---:|---:|---:|---:|---:|
|
||
| 16 | 0.3085 | 18046 | 57 | 84 |
|
||
| 32 | 0.3110 | 21621 | 43 | 75 |
|
||
| 64 | 0.3130 | 20015 | 49 | 70 |
|
||
| 128 | 0.3135 | 15822 | 63 | 99 |
|
||
| 256 | 0.3135 | 15308 | 66 | 124 |
|
||
|
||
### End to end: `HDF5Memory::hybrid_search` (k = 10, weights 0.7 / 0.3)
|
||
|
||
| N | ingest ms | checkpoint ms | open ms | first query ms | p50 ms | p99 ms | QPS |
|
||
|---:|---:|---:|---:|---:|---:|---:|---:|
|
||
| 1000 | 11 | 3.9 | 0.9 | 68.1 | 5.48 | 5.57 | 182.5 |
|
||
| 10000 | 114 | 32.2 | 10.9 | 845.0 | 48.56 | 78.65 | 19.8 |
|
||
| 100000 | 1486 | 713.0 | 354.5 | 10486.5 | 883.51 | 975.23 | 1.1 |
|
||
wrote /tmp/claude-1000/-home-osobh-projects-clawhdf5/422f755e-dd25-4c35-8613-5439087e3aaa/scratchpad/baseline_full.json
|
||
|
||
### After: HNSW neighbour-selection heuristic
|
||
|
||
Same harness, same data, after replacing closest-M neighbour selection with the
|
||
HNSW paper's diversity heuristic (Algorithm 4, keeping pruned connections) for
|
||
both new links and back-link pruning. Recall@10 at `ef = 64`: **0.87 → 1.00**
|
||
(1K), **0.67 → 1.00** (10K), **0.31 → 0.98** (100K), and it now rises with
|
||
`ef` as it should. The cost is a slower build (extra distance evaluations per
|
||
insert: ~3.5x at 10K); the distance-kernel work that follows targets that.
|
||
|
||
### HNSW, N = 1000, dim = 384, M = 16, ef_construction = 64
|
||
|
||
build: 221.3 ms (4519 vectors/s) · exact scan: 3851 QPS, p50 258 µs
|
||
|
||
| ef | recall@10 | QPS | p50 µs | p99 µs |
|
||
|---:|---:|---:|---:|---:|
|
||
| 16 | 0.9990 | 54760 | 18 | 29 |
|
||
| 32 | 1.0000 | 40422 | 24 | 44 |
|
||
| 64 | 1.0000 | 27744 | 36 | 51 |
|
||
| 128 | 1.0000 | 13164 | 74 | 106 |
|
||
| 256 | 1.0000 | 6879 | 144 | 175 |
|
||
|
||
### HNSW, N = 10000, dim = 384, M = 16, ef_construction = 64
|
||
|
||
build: 2733.5 ms (3658 vectors/s) · exact scan: 423 QPS, p50 2362 µs
|
||
|
||
| ef | recall@10 | QPS | p50 µs | p99 µs |
|
||
|---:|---:|---:|---:|---:|
|
||
| 16 | 0.9975 | 31321 | 27 | 61 |
|
||
| 32 | 1.0000 | 32427 | 29 | 48 |
|
||
| 64 | 1.0000 | 22738 | 42 | 62 |
|
||
| 128 | 1.0000 | 10055 | 99 | 129 |
|
||
| 256 | 1.0000 | 4649 | 214 | 266 |
|
||
|
||
### HNSW, N = 100000, dim = 384, M = 16, ef_construction = 64
|
||
|
||
build: 36472.8 ms (2742 vectors/s) · exact scan: 40 QPS, p50 24644 µs
|
||
|
||
| ef | recall@10 | QPS | p50 µs | p99 µs |
|
||
|---:|---:|---:|---:|---:|
|
||
| 16 | 0.9235 | 11394 | 82 | 194 |
|
||
| 32 | 0.9675 | 12788 | 73 | 161 |
|
||
| 64 | 0.9840 | 10406 | 91 | 186 |
|
||
| 128 | 0.9990 | 7633 | 126 | 248 |
|
||
| 256 | 0.9990 | 2823 | 352 | 510 |
|
||
|
||
### After: persistent keyword index, no store rewrite per query
|
||
|
||
`hybrid_search` used to rebuild the BM25 index from scratch (re-tokenising every
|
||
record) and rewrite the whole `.h5` file on **every query**. The index is now
|
||
kept for the life of the store and updated incrementally, and activation boosts
|
||
are persisted by the next checkpoint instead of inside the query. Steady-state
|
||
p50: **5.5 → 0.24 ms** (1K), **49 → 2.1 ms** (10K), **884 → 23 ms** (100K).
|
||
|
||
The first query after `open()` is slower than before (it pays for the better —
|
||
slower — HNSW build plus the one-off keyword index build); persisting the HNSW
|
||
index removes that.
|
||
|
||
### End to end: `HDF5Memory::hybrid_search` (k = 10, weights 0.7 / 0.3)
|
||
|
||
| N | ingest ms | checkpoint ms | open ms | first query ms | p50 ms | p99 ms | QPS |
|
||
|---:|---:|---:|---:|---:|---:|---:|---:|
|
||
| 1000 | 11 | 3.8 | 0.9 | 195.9 | 0.24 | 0.27 | 4130.4 |
|
||
| 10000 | 104 | 31.1 | 10.9 | 2627.1 | 2.09 | 2.11 | 479.5 |
|
||
| 100000 | 1436 | 684.7 | 278.0 | 36308.1 | 22.90 | 25.46 | 43.5 |
|
||
|
||
### After: vector index persisted with the checkpoint
|
||
|
||
The HNSW graph (not the vectors, which the store already holds) is saved to
|
||
`<store>.h5.ann` at each checkpoint and reloaded by `open()`, tied to that
|
||
checkpoint by a generation id. The index is now built once per store (the *cold
|
||
index build* column — the first query ever), not once per session. First query
|
||
after `open()`: **196 → 1.7 ms** (1K), **2627 → 15 ms** (10K),
|
||
**36308 → 159 ms** (100K); what remains is the one-off keyword index build.
|
||
Batch saves no longer force a full rebuild either: appended records join the
|
||
index incrementally.
|
||
|
||
| N | ingest ms | cold index build ms | checkpoint ms | open ms | first query after open ms | p50 ms | p99 ms | QPS |
|
||
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||
| 1000 | 14 | 220 | 6.1 | 1.2 | 1.7 | 0.24 | 0.27 | 4049.9 |
|
||
| 10000 | 120 | 2916 | 33.1 | 14.0 | 15.4 | 2.15 | 3.30 | 421.2 |
|
||
| 100000 | 1591 | 40515 | 747.3 | 324.7 | 158.9 | 23.07 | 30.42 | 41.3 |
|
||
|
||
### After: unit-vector dot product, reusable visited set
|
||
|
||
Cosine distance recomputed both vector norms on every evaluation; the index now
|
||
stores unit vectors and uses a plain dot product. The per-call `HashSet` of
|
||
visited nodes became a reusable epoch-stamped array. Recall is unchanged.
|
||
Build: **2.75 -> 1.89 s** (10K), **~38 -> 21 s** (100K). QPS at `ef = 64`:
|
||
**22.7K -> 39K** (10K), **10.4K -> 14K** (100K).
|
||
|
||
### HNSW, N = 1000, dim = 384, M = 16, ef_construction = 64
|
||
|
||
build: 113.4 ms (8821 vectors/s) · exact scan: 4375 QPS, p50 225 µs
|
||
|
||
| ef | recall@10 | QPS | p50 µs | p99 µs |
|
||
|---:|---:|---:|---:|---:|
|
||
| 16 | 0.9990 | 144379 | 7 | 15 |
|
||
| 32 | 1.0000 | 110654 | 9 | 17 |
|
||
| 64 | 1.0000 | 80446 | 12 | 25 |
|
||
| 128 | 1.0000 | 38220 | 26 | 36 |
|
||
| 256 | 1.0000 | 20041 | 50 | 62 |
|
||
|
||
### HNSW, N = 10000, dim = 384, M = 16, ef_construction = 64
|
||
|
||
build: 1519.4 ms (6581 vectors/s) · exact scan: 422 QPS, p50 2368 µs
|
||
|
||
| ef | recall@10 | QPS | p50 µs | p99 µs |
|
||
|---:|---:|---:|---:|---:|
|
||
| 16 | 0.9975 | 54608 | 15 | 45 |
|
||
| 32 | 1.0000 | 66009 | 14 | 24 |
|
||
| 64 | 1.0000 | 49854 | 19 | 31 |
|
||
| 128 | 1.0000 | 22403 | 45 | 57 |
|
||
| 256 | 1.0000 | 10096 | 100 | 120 |
|
||
|
||
### HNSW, N = 100000, dim = 384, M = 16, ef_construction = 64
|
||
|
||
build: 21084.6 ms (4743 vectors/s) · exact scan: 39 QPS, p50 24739 µs
|
||
|
||
| ef | recall@10 | QPS | p50 µs | p99 µs |
|
||
|---:|---:|---:|---:|---:|
|
||
| 16 | 0.9235 | 15139 | 61 | 154 |
|
||
| 32 | 0.9675 | 18181 | 53 | 121 |
|
||
| 64 | 0.9840 | 13980 | 70 | 139 |
|
||
| 128 | 0.9990 | 10959 | 86 | 174 |
|
||
| 256 | 0.9990 | 3731 | 254 | 697 |
|
||
|
||
|
||
### After: unranked keyword scores, top-k merge (rankings unchanged)
|
||
|
||
A fusion study (`search_harness --fusion-study`) showed that capping the
|
||
keyword candidate pool is **not** a safe optimisation: against the current
|
||
full-corpus normalisation the final top-10 overlap is only 0.83-0.92 and the
|
||
first result changes for 10-35% of queries, for only a 2x saving. So the fusion
|
||
semantics were left alone and the same answer made cheaper: fusion needs every
|
||
keyword score but not their ranking, so BM25 now returns them unsorted from a
|
||
dense accumulator (it hashed every posting and then sorted every match), and
|
||
the merge selects its top k instead of sorting every candidate. Steady-state
|
||
p50: **0.24 -> 0.07 ms** (1K), **2.1 -> 0.49 ms** (10K), **23 -> 4.65 ms**
|
||
(100K) — **79x / 100x / 190x** faster than the v2.3.0 baseline, with identical
|
||
results.
|
||
|
||
### End to end: `HDF5Memory::hybrid_search` (k = 10, weights 0.7 / 0.3)
|
||
|
||
| N | ingest ms | cold index build ms | checkpoint ms | open ms | first query after open ms | p50 ms | p99 ms | QPS |
|
||
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||
| 1000 | 11 | 112 | 4.0 | 1.1 | 1.4 | 0.07 | 0.08 | 14077.9 |
|
||
| 10000 | 104 | 1487 | 33.8 | 13.7 | 13.9 | 0.49 | 0.51 | 2020.9 |
|
||
| 100000 | 1376 | 20285 | 728.9 | 353.1 | 142.2 | 4.65 | 4.78 | 214.7 |
|
||
|
||
### After: batched bulk build (optionally parallel); deletions handled in search
|
||
|
||
Profiling showed **90% of a build's distance evaluations are in back-link
|
||
pruning**. The bulk build now inserts in batches: plan each node's neighbours
|
||
against the graph as it stood at the start of the batch, link, then prune every
|
||
overflowing list once. That is less work even single-threaded (a node gaining
|
||
several back-links in a batch is pruned once), and with the `parallel` feature
|
||
planning and pruning run on a thread pool. The graph is deterministic and the
|
||
same with or without the feature. Parallelising *within* one insert was tried
|
||
first and gave only 1.45x on 16 cores (tasks too small).
|
||
|
||
| build | 1K | 10K | 100K |
|
||
|---|---:|---:|---:|
|
||
| v2.4.0 | 116 ms | 1676 ms | ~21 s |
|
||
| batched | 83 ms | 1074 ms | 19.2 s |
|
||
| batched + `parallel` (16 cores) | 34 ms | 388 ms | 5.9 s |
|
||
|
||
Recall on clustered data is unchanged or slightly better (100K, `ef = 64`:
|
||
0.984 -> 0.9945). On uniform random data it dips slightly (10K, `ef = 64`:
|
||
0.474 -> 0.444), the cost of batch members not seeing each other while
|
||
planning; batches are capped at 1/16 of the graph and 512 nodes.
|
||
|
||
### Current: search harness (2026-09-24)
|
||
|
||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, default
|
||
features (so the build is batched and `parallel`, 16 threads). The HNSW tables
|
||
use an `f32` index. The end-to-end table uses `MemoryConfig::new`'s defaults,
|
||
which at this commit are `float16` embeddings and the int8 index; every
|
||
earlier end-to-end table in this section was an `f32` store with an `f32`
|
||
index.
|
||
|
||
> **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --full`
|
||
|
||
#### HNSW, N = 1000, dim = 384, M = 16, ef_construction = 64, storage = Float32
|
||
|
||
build: 14.7 ms (67954 vectors/s) · exact scan: 4403 QPS, p50 225 µs
|
||
|
||
| ef | recall@10 | QPS | p50 µs | p99 µs |
|
||
|---:|---:|---:|---:|---:|
|
||
| 16 | 0.9940 | 134065 | 7 | 18 |
|
||
| 32 | 1.0000 | 105134 | 9 | 21 |
|
||
| 64 | 1.0000 | 77541 | 13 | 26 |
|
||
| 128 | 1.0000 | 36938 | 26 | 35 |
|
||
| 256 | 1.0000 | 19249 | 52 | 63 |
|
||
|
||
#### HNSW, N = 10000, dim = 384, M = 16, ef_construction = 64, storage = Float32
|
||
|
||
build: 135.6 ms (73736 vectors/s) · exact scan: 435 QPS, p50 2295 µs
|
||
|
||
| ef | recall@10 | QPS | p50 µs | p99 µs |
|
||
|---:|---:|---:|---:|---:|
|
||
| 16 | 0.9975 | 79614 | 12 | 24 |
|
||
| 32 | 1.0000 | 70556 | 14 | 21 |
|
||
| 64 | 1.0000 | 51313 | 19 | 29 |
|
||
| 128 | 1.0000 | 22846 | 44 | 57 |
|
||
| 256 | 1.0000 | 10267 | 97 | 119 |
|
||
|
||
#### HNSW, N = 100000, dim = 384, M = 16, ef_construction = 64, storage = Float32
|
||
|
||
build: 2700.8 ms (37027 vectors/s) · exact scan: 43 QPS, p50 23433 µs
|
||
|
||
| ef | recall@10 | QPS | p50 µs | p99 µs |
|
||
|---:|---:|---:|---:|---:|
|
||
| 16 | 0.9180 | 20043 | 47 | 104 |
|
||
| 32 | 0.9775 | 21348 | 44 | 104 |
|
||
| 64 | 0.9945 | 19001 | 51 | 101 |
|
||
| 128 | 0.9995 | 13458 | 73 | 137 |
|
||
| 256 | 0.9995 | 5128 | 196 | 282 |
|
||
|
||
#### End to end: `HDF5Memory::hybrid_search` (k = 10, weights 0.7 / 0.3)
|
||
|
||
| N | ingest ms | cold index build ms | checkpoint ms | open ms | first query after open ms | p50 ms | p99 ms | QPS |
|
||
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||
| 1000 | 15 | 14 | 4.4 | 1.9 | 1.5 | 0.07 | 0.08 | 13869.5 |
|
||
| 10000 | 146 | 126 | 33.5 | 18.3 | 14.8 | 0.49 | 0.50 | 2044.1 |
|
||
| 100000 | 1498 | 1833 | 495.7 | 259.9 | 155.1 | 4.69 | 5.22 | 212.7 |
|
||
|
||
Steady-state query latency is where the v2.4.0 work left it (p50 0.07 / 0.49 /
|
||
4.69 ms against 0.07 / 0.49 / 4.65 ms). Figures that moved by more than 20%:
|
||
|
||
- **Build.** 14.7 / 135.6 / 2700.8 ms against 34 / 388 / 5.9 s for
|
||
"batched + `parallel`" above. The git log does not explain the difference.
|
||
- **Cold index build** in the end-to-end table: 14 / 126 / 1833 ms, down from
|
||
112 / 1487 / 20285 ms in the "unranked keyword scores" table, which predates
|
||
the batched and parallel build described above.
|
||
- **QPS at 100K, `ef = 64`**: 19 001 against 13 980 in the "unit-vector"
|
||
table, with recall 0.9945 against 0.984; the recall gain is the one the
|
||
batched build reported. The QPS gain is not explained by the log.
|
||
- **Other HNSW QPS cells**, against the same "unit-vector" tables: 10K
|
||
`ef = 16` 79 614 against 54 608 (+46%); 100K `ef = 16` 20 043 against
|
||
15 139 (+32%), `ef = 128` 13 458 against 10 959 (+23%), `ef = 256` 5 128
|
||
against 3 731 (+37%). The other cells, and every 1K cell, are within 20%.
|
||
The log does not explain these either.
|
||
- **Open** in the end-to-end table: 1.9 / 18.3 / 259.9 ms against 1.1 / 13.7
|
||
/ 353.1 ms in the "unranked keyword scores" table, slower at 1K and 10K and
|
||
faster at 100K. This run does not isolate the cause.
|
||
- **Checkpoint at 100K**: 495.7 ms against 728.9 ms, and ingest up 11 -> 15,
|
||
104 -> 146 ms at 1K / 10K. This store is `float16` with an int8 index, the
|
||
earlier ones `f32`; the [float16 study](#float16-embedding-storage-memoryconfigfloat16)
|
||
measured float16 checkpoints at 512 ms against 752 ms for `f32` at 100K.
|
||
This run does not isolate the ingest change.
|
||
|
||
## Vector Search Latency
|
||
|
||
Brute-force cosine similarity over 384-dimensional embeddings (OpenAI text-embedding-3-small size).
|
||
|
||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||
|
||
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(vector_search_1k|simd_cosine_10k|simd_cosine_100k|prenorm_search_10k|ivf_search_10k_nprobe10|ivf_search_100k_nprobe10|ivf_pq_search_100k|rairs_search_10k_nprobe10)$'`
|
||
|
||
| Scale | Flat Search | Pre-norm | IVF (nprobe=10) | IVF-PQ | RAIRS |
|
||
|-------|-------------|----------|-----------------|--------|-------|
|
||
| **1K** | 47.4 µs | — | — | — | — |
|
||
| **10K** | 500.5 µs | 316.5 µs | 24.8 µs | — | 124.0 µs |
|
||
| **100K** | 6.58 ms | — | 592 µs | 869 µs | — |
|
||
|
||
The 1K pre-norm cell is blank because no 1K pre-norm benchmark exists (the
|
||
old 62 µs has no source). `memory_bench`'s own cosine search
|
||
(`cargo bench -p clawhdf5-agent --bench memory_bench -- '^vector_search_latency/'`)
|
||
agrees: 47.6 µs / 505.1 µs / 6.67 ms at 1K / 10K / 100K.
|
||
|
||
Most cells moved by more than 20% from the previous table (10K flat 753 µs,
|
||
pre-norm 706 µs, RAIRS 159 µs; 100K flat 11.4 ms, IVF 1.32 ms, IVF-PQ 1.19 ms).
|
||
Those were undated figures from the original i7-12650H run. The 2026-08-05
|
||
tank run below is within 4% of the new numbers in every cell but one (1K flat
|
||
47.8 µs, 10K flat 501 µs, pre-norm 322 µs, IVF 24.8 µs; 100K flat 6.60 ms,
|
||
IVF 608 µs, IVF-PQ 865 µs), so for those cells the difference is the machine.
|
||
The exception is RAIRS: 109 µs on 2026-08-05, 124.0 µs now (+14%); this run
|
||
does not explain that change.
|
||
|
||
**Key insight:** at 10K records (typical agent memory), IVF search takes
|
||
**24.8 µs** against 500.5 µs for the flat scan. At 100K, IVF answers in
|
||
**592 µs** and IVF-PQ in 869 µs — both under 1 ms. (RAIRS, 124.0 µs at 10K, is
|
||
slower than plain IVF here.)
|
||
|
||
### Comparison to MemX (arxiv:2603.16171)
|
||
|
||
MemX claims end-to-end search under 90ms at 100K records (Rust + libSQL + FTS5).
|
||
|
||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||
|
||
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(simd_cosine_100k|ivf_pq_search_100k|bm25_search_10k)$'`
|
||
|
||
> **Caveat — not like-for-like.** MemX's `<90 ms` is *end-to-end* search across their
|
||
> full pipeline (dense embeddings + FTS5 + four-factor re-ranking). The clawhdf5
|
||
> figures below are a *single component* — raw vector search latency, excluding
|
||
> embedding, keyword, fusion, and re-ranking stages. A component measured against a
|
||
> full pipeline will always look favourable, so the two columns below cannot be
|
||
> divided into a speedup: any such ratio would overstate the real advantage by an
|
||
> unquantified margin. Read the table as an order-of-magnitude indication only, not a
|
||
> benchmark result. Matching MemX's measurement boundary is tracked as follow-up work.
|
||
|
||
| Metric | MemX (claimed, end-to-end) | ClawhDF5 (component only) |
|
||
|--------|----------------------------|---------------------------|
|
||
| 100K flat search | <90 ms | 6.64 ms |
|
||
| 100K IVF-PQ search | — | 876 µs |
|
||
| Keyword search 10K | 1,100x improvement over unindexed | 205 µs (BM25) |
|
||
|
||
The ratio column is gone: dividing an end-to-end claim by a component
|
||
timing produced a number that looked like a result and was not one. The
|
||
previous figures (100K flat 11.4 ms, IVF-PQ 1.19 ms, BM25 10K 583 µs) were
|
||
undated, from the original i7-12650H run. BM25 at 10K also moved on this
|
||
machine: 520 µs on 2026-08-05, 205 µs now; v2.4.0 changed
|
||
`BM25Index::search` in between (bounded-heap top-k, the unused WAND bound
|
||
removed, IDF per query), but this run does not isolate its effect.
|
||
|
||
---
|
||
|
||
## SIMD & Parallelism
|
||
|
||
384-dimensional cosine similarity at 10K scale, using the `strategy_*`
|
||
benchmarks, which hold the dataset fixed and vary only `SearchStrategy`
|
||
(the 2026-08-05 re-run below explains why the older table's benchmarks were
|
||
not an apples-to-apples comparison).
|
||
|
||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. Rayon
|
||
uses all 16 threads; the 1-minute load average was 4.19 when this command
|
||
finished, most likely from the Rayon benchmarks themselves (not verified).
|
||
|
||
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(strategy_scalar_10k|strategy_simd_10k|strategy_rayon_10k|adaptive_search_10k|simd_cosine_100k|rayon_cosine_100k)$'`
|
||
|
||
| Strategy | Latency |
|
||
|----------|---------|
|
||
| Sequential (scalar), `strategy_scalar_10k` | 501.1 µs |
|
||
| SIMD, `strategy_simd_10k` | 319.9 µs |
|
||
| Rayon (parallel), `strategy_rayon_10k` | **79.0 µs** |
|
||
| Adaptive (auto-select), `adaptive_search_10k` | 317.9 µs |
|
||
|
||
At 100K (no `strategy_*` benchmark exists at this size, so these are the
|
||
`simd_cosine_100k` / `rayon_cosine_100k` benchmarks):
|
||
|
||
| Strategy | Latency |
|
||
|----------|---------|
|
||
| SIMD | 6.56 ms |
|
||
| Rayon parallel | 4.63 ms |
|
||
|
||
The Rayon 10K figure moved furthest: 323 µs on this machine on 2026-08-05,
|
||
79.0 µs now. The git log does not explain it. The other rows are close to
|
||
that run (scalar 502 µs, SIMD 327 µs, adaptive 339 µs; 100K 6.60 / 4.73 ms). Against the original, undated i7-12650H table (scalar 1.07 ms, SIMD
|
||
545 µs, Rayon 553 µs, adaptive 564 µs; 100K 13.7 / 8.3 ms) every figure is
|
||
lower; that table also used the mislabelled benchmarks. The older names
|
||
measured in the same session, for reference (`cargo bench -p clawhdf5-agent
|
||
--bench bench -- '^(sequential_cosine_10k|simd_cosine_10k|rayon_cosine_10k|adaptive_search_10k|simd_cosine_100k|rayon_cosine_100k)$'`):
|
||
`sequential_cosine_10k` 317.8 µs, `simd_cosine_10k` 505.2 µs,
|
||
`rayon_cosine_10k` 274.3 µs — "sequential" faster than "SIMD", which is the
|
||
mislabelling the 2026-08-05 re-run found.
|
||
|
||
---
|
||
|
||
## Hybrid Search (Vector + BM25)
|
||
|
||
1K records, 384-dimensional embeddings with BM25 keyword index. These are the
|
||
flat free functions `hybrid::hybrid_search` / `rrf_hybrid_search` (linear
|
||
scan), not the HNSW-backed `HDF5Memory::hybrid_search`, which is measured in
|
||
the [search harness](#current-search-harness-2026-09-24).
|
||
|
||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||
|
||
> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^hybrid_search/'`
|
||
> and `cargo bench -p clawhdf5-agent --bench bench -- '^(bm25_search_1k|hybrid_search_10k)$'`
|
||
|
||
| Method | Latency | Notes |
|
||
|--------|---------|-------|
|
||
| Weighted fusion | 106.8 µs | Min-max normalization |
|
||
| **RRF (k=60)** | **130.5 µs** | Reciprocal Rank Fusion |
|
||
| BM25-only 1K | 20.4 µs | Keyword search alone |
|
||
| Hybrid 10K | 1.10 ms | Full hybrid at 10K scale |
|
||
|
||
Every row moved by more than 20% from the previous table (198 µs, 222 µs,
|
||
67 µs, 2.04 ms), which was undated and from the original i7-12650H run. The
|
||
cause is not isolated. (The old "better quality" note on RRF is dropped: on
|
||
LongMemEval RRF measured worse than the tuned weighted sum; see
|
||
[Fusion method](#fusion-method--weighted-vs-rrf-full-haystack-n500).)
|
||
|
||
---
|
||
|
||
## Knowledge Graph
|
||
|
||
Graph traversal and entity operations.
|
||
|
||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||
|
||
> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^knowledge_graph/'`
|
||
> and `cargo bench -p clawhdf5-agent --bench bench -- '^alias_resolve_(short|long)_query$'`
|
||
> (the `knowledge_graph/` rows re-run after the adjacency-index fix, same day,
|
||
> load average 0.49 at start)
|
||
|
||
| Operation | Scale | Latency |
|
||
|-----------|-------|---------|
|
||
| BFS traversal | 100 entities | 5.23 µs |
|
||
| BFS traversal | 1,000 entities | 23.1 µs |
|
||
| Spreading activation | 100 entities | 10.1 µs |
|
||
| Entity resolution (Levenshtein) | 100 entities | 48.6 µs |
|
||
| Alias resolution (short query) | 100 aliases | 7.67 µs |
|
||
| Alias resolution (long query) | 100 aliases | 8.18 µs |
|
||
|
||
The traversal rows were measured after a fix made during this re-run. The
|
||
first measurement of the day found **BFS 6.5x slower** than the old table
|
||
said: 17.5 µs and 155.1 µs, against 5.4 µs and 24 µs, with spreading
|
||
activation at 22.8 µs against 16.9 µs. The cause was 1efd82c (2026-08-17),
|
||
which built an adjacency index over the whole graph on every traversal — so a
|
||
2-hop BFS paid to index every entity and relation first. The index is now
|
||
cached on `KnowledgeCache` and checked against a fingerprint of the graph on
|
||
each use (one pass over entity ids and relation endpoints, no allocation), so
|
||
any change to the graph, including direct edits of its public `Vec`s, still
|
||
triggers a rebuild. These benches traverse an unchanged graph, which is the
|
||
cached case; the first traversal after a change pays for one build. Entity
|
||
and alias resolution do not use the index (the 49.8 µs first-run figure for
|
||
entity resolution is within noise of the 48.6 µs here). The old figures were
|
||
undated, from the original i7-12650H run.
|
||
|
||
---
|
||
|
||
## Memory Consolidation
|
||
|
||
Hippocampal-inspired tiered memory management.
|
||
|
||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||
|
||
> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^consolidation/'`
|
||
|
||
| Operation | Scale | Latency |
|
||
|-----------|-------|---------|
|
||
| Consolidation cycle | 100 records | 8.08 µs |
|
||
| Consolidation cycle | 1,000 records | 115.2 µs |
|
||
| Importance scoring | 100 records | 31.0 µs |
|
||
|
||
A full consolidation pass over 1,000 memories (eviction + promotion across Working → Episodic → Semantic) completes in **115 µs**. This can run on every memory write without perceptible latency.
|
||
|
||
The cycle figures were 15 µs and 164 µs, and importance scoring 25 µs, all
|
||
undated from the original i7-12650H run. Eviction's membership check became a
|
||
`HashSet` lookup (603fcf8) and `add_memory` stopped cloning Working-tier
|
||
records (d787ac0), both on 2026-08-17; this run does not isolate their effect.
|
||
|
||
---
|
||
|
||
## Temporal Index
|
||
|
||
Sorted timestamp index with binary search.
|
||
|
||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||
|
||
> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^temporal/'`
|
||
|
||
| Operation | Scale | Latency |
|
||
|-----------|-------|---------|
|
||
| Range query | 10K timestamps | **622 ns** |
|
||
| Batch insert | 10K timestamps | 3.22 ms |
|
||
|
||
Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K records: **622 nanoseconds.**
|
||
Batch insert was 4.69 ms in the previous, undated i7-12650H figure.
|
||
|
||
---
|
||
|
||
## Write Path
|
||
|
||
HDF5 persistence with optional Write-Ahead Log.
|
||
|
||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. The stores
|
||
use `MemoryConfig::new`'s defaults at this commit: `float16` embeddings and
|
||
the int8 index.
|
||
|
||
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(save_without_wal_single|save_with_wal_single|save_batch_100|save_batch_1000|save_wal_1k_existing|wal_flush_100_entries|tick_session_1k|tick_session_10k)$'`
|
||
|
||
| Operation | Latency | Notes |
|
||
|-----------|---------|-------|
|
||
| Single save (no WAL) | 88.3 µs | Direct HDF5 write (owned-Vec IO path) |
|
||
| Single save (with WAL) | 26.1 µs | WAL group-commit append; HDF5 write batched at flush |
|
||
| Batch 100 | 1.08 ms | |
|
||
| Batch 1,000 | 31.7 ms | |
|
||
| WAL save (1K existing) | 282.8 µs | Incremental append |
|
||
| WAL flush 100 entries | 530.1 µs | Merge WAL → HDF5 |
|
||
| Session tick 1K | 3.83 ms | Full session maintenance |
|
||
| Session tick 10K | 35.5 ms | Background operation |
|
||
|
||
Almost every row moved by more than 20% from the previous table, in both
|
||
directions. Slower: single save 61 -> 88.3 µs (no WAL) and 18 -> 26.1 µs
|
||
(WAL), batch 100 723 µs -> 1.08 ms, and **batch 1,000 6.17 -> 31.7 ms**.
|
||
Faster: WAL save into 1K existing 539 -> 282.8 µs, WAL flush 787 -> 530.1 µs,
|
||
session tick 5.76 -> 3.83 ms (1K) and 89.8 -> 35.5 ms (10K). The two single-save
|
||
figures were last updated on 2026-07-01 and the rest are undated, all on the
|
||
i7-12650H with `f32` stores.
|
||
|
||
**Isolated afterwards, same machine, same day** (`--warm-up-time 1
|
||
--measurement-time 3`, the store settings switched through the bench's
|
||
config):
|
||
|
||
| | f16 + int8 (default) | f32 + int8 | f16 + f32 index | f32 + f32 index | v2.3.0 (f32) |
|
||
|---|---:|---:|---:|---:|---:|
|
||
| Single save (WAL) | 26.1 µs | 24.3 µs | 26.1 µs | 24.4 µs | 24.3 µs |
|
||
| Single save (no WAL) | 89.0 µs | 87.3 µs | 88.8 µs | 87.4 µs | 82.6 µs |
|
||
| Batch 100 | 1.09 ms | 0.90 ms | 1.09 ms | 0.90 ms | 0.87 ms |
|
||
| Batch 1,000 | 31.5 ms | 31.1 ms | 31.5 ms | 31.6 ms | 30.3 ms |
|
||
|
||
The int8 index costs nothing on the write path. `float16` costs ~2 µs per
|
||
saved record — the rounding — which is the whole difference at 100 records
|
||
and within noise at 1,000. Built at the v2.3.0 tag (v2.2.0's benches do not
|
||
compile) and run on this machine, the write path measures the same as today,
|
||
so nothing regressed since then; the old 18 µs and 6.17 ms figures cannot be
|
||
reproduced on this hardware and came from an undated run on another machine.
|
||
`save_batch` scales linearly (about 12 µs per record from 500 to 8 000
|
||
records); the bench's figure also includes dropping the store.
|
||
|
||
---
|
||
|
||
## Decision Gate
|
||
|
||
Trivial/non-trivial classification for memory write filtering.
|
||
|
||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||
|
||
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^gate_'`
|
||
|
||
| Check | Latency |
|
||
|-------|---------|
|
||
| Trivial skip ("ok", "yes") | 57.6 ns |
|
||
| Short phrase skip | 83.0 ns |
|
||
| Non-trivial pass | 570.5 ns |
|
||
| Ratio check | 395.2 ns |
|
||
|
||
**Sub-microsecond filtering.** The gate decides whether to save a memory in under 1 µs.
|
||
|
||
---
|
||
|
||
## Memory Strategy
|
||
|
||
End-to-end strategy evaluation including embedding operations.
|
||
|
||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||
|
||
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^strategy_(save_every|semantic_shift)'`
|
||
|
||
| Strategy | Condition | Latency |
|
||
|----------|-----------|---------|
|
||
| SaveEveryExchange (substantive) | Saves | 732.5 ns |
|
||
| SaveEveryExchange (trivial) | Skips | 58.3 ns |
|
||
| SaveOnSemanticShift (empty store) | Saves | 734.9 ns |
|
||
|
||
The two "Saves" rows were 923 ns and 941 ns in the previous table, undated
|
||
and from the original i7-12650H run.
|
||
|
||
---
|
||
|
||
## Multi-modal Search
|
||
|
||
`clawhdf5_agent::multimodal::MultiModalStore`. Each of N records carries two
|
||
384-dim embeddings: a text embedding of its caption and one of its primary
|
||
modality, cycling image / audio / video. `search_cross_modal` scores every
|
||
embedding of every record (2N vectors) and keeps each record's best;
|
||
`search_by_modality(Image, …)` scores only image embeddings (a third of the
|
||
records). Both are exact linear scans followed by a full sort; k = 10. Data is
|
||
from a fixed-seed generator, so every run sees the same corpus.
|
||
|
||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c (with the
|
||
new, uncommitted `crates/clawhdf5-agent/benches/multimodal_bench.rs` in the
|
||
working tree). Criterion, 50 samples.
|
||
|
||
> **Run:** `cargo bench -p clawhdf5-agent --bench multimodal_bench`
|
||
|
||
| Search | 1K records | 10K records |
|
||
|--------|-----------:|------------:|
|
||
| Cross-modal (all modalities, 2N vectors) | 842.0 µs | 8.44 ms |
|
||
| One modality (image, ~N/3 vectors) | 151.2 µs | 1.54 ms |
|
||
|
||
Both scale linearly with N. Cross-modal search is the slow path: at 1K
|
||
records it takes 842.0 µs, where the flat single-vector cosine scan in
|
||
[Vector Search Latency](#vector-search-latency) takes 47.4 µs over 1K vectors,
|
||
and it has no index — at 10K it is 8.44 ms, slower than a 100K flat scan
|
||
(6.58 ms). No earlier figure exists for this section.
|
||
|
||
---
|
||
|
||
## Summary
|
||
|
||
Derived from the 2026-09-24 tables above (tank, AMD Ryzen 7 7800X3D, commit
|
||
5c8323c); no command of its own.
|
||
|
||
| Capability | Typical Latency | Scale |
|
||
|------------|----------------|-------|
|
||
| **Full memory search** (`HDF5Memory::hybrid_search`, p50) | 0.49 ms | 10K records |
|
||
| **Hybrid vector+keyword** (flat, weighted) | 106.8 µs | 1K records |
|
||
| **Knowledge graph query** (BFS) | 23.1 µs | 1K entities |
|
||
| **Temporal range query** | 622 ns | 10K timestamps |
|
||
| **Memory write** | 26.1 µs | Per record (WAL group-commit append) |
|
||
| **Consolidation cycle** | 115.2 µs | 1K records |
|
||
| **Importance gate** | 57.6–570.5 ns | Per record |
|
||
|
||
The previous summary said "<25 µs" for a knowledge-graph query and "<20 µs"
|
||
for a memory write. The first holds again after the adjacency-index fix
|
||
(23.1 µs); the second does not (26.1 µs).
|
||
|
||
**At agent-typical scales a full `HDF5Memory::hybrid_search` (vector + BM25 + fusion) over 10K records takes 0.49 ms at p50, and graph traversal, temporal queries, consolidation and the write gate each take well under a millisecond. At 100K records the hybrid search takes 4.69 ms.**
|
||
|
||
---
|
||
|
||
_Latency benchmarks generated with Criterion.rs (50-100 samples per benchmark). Results may vary by hardware._
|
||
|
||
---
|
||
|
||
## LongMemEval Results
|
||
|
||
> **Scoring target declaration.** Per [arXiv 2605.24060](https://arxiv.org/abs/2605.24060),
|
||
> which found that changing scoring target alone alters nDCG on 83–94% of queries and
|
||
> can reverse system rankings, this section states its measurement contract explicitly:
|
||
>
|
||
> - **Dataset variant:** both are now reported below — the full `longmemeval_s`
|
||
> haystack (**the headline number**) and `longmemeval_oracle` (evidence sessions
|
||
> only, a substantially easier corpus, kept for continuity). The harness does not
|
||
> trust the filename: it measures evidence-session density from the data and
|
||
> labels the run from that, so a mislabelled input cannot yield a mislabelled
|
||
> result. Measured density is 4.0% on `longmemeval_s` and 100.0% on the oracle.
|
||
> - **Metric:** *retrieval recall.* A "hit" means the gold-labelled memory appeared in
|
||
> the top-k. **No answer is generated and none is scored** — the dataset's `answer`
|
||
> field is deserialized and never read. This is **not** the official LongMemEval
|
||
> leaderboard metric, which is end-to-end QA accuracy (retrieve → generate → LLM
|
||
> judge). Retrieval recall reported as QA accuracy typically overstates by 20–30 points.
|
||
> - **Granularity:** turn-level = the returned memory's source turn had `has_answer == true`.
|
||
> - **k = 10**, n = 500.
|
||
> - **Retrieval mode:** all three are reported below. Historically the bench passed
|
||
> zero-vector embeddings with `vector_weight=0.0`, so the HNSW/vector stage was
|
||
> inert and every published number was BM25 alone. Real `all-MiniLM-L6-v2`
|
||
> embeddings are now available via `--features embeddings --embeddings <dir>`,
|
||
> and BM25-only / vector-only / hybrid are each measured separately.
|
||
|
||
**Mode:** BM25-only retrieval — zero embeddings, `vector_weight=0.0`, `keyword_weight=1.0`
|
||
|
||
> **Run:** `cargo run --release -p clawhdf5-bench --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s.json`
|
||
> Omit the path for the oracle variant; add `--limit N` for an evenly-strided
|
||
> subsample. Without the `embeddings` feature this runs two modes, BM25 only
|
||
> and BM25 stemmed.
|
||
|
||
The BM25-only and BM25-stemmed figures in this section were measured again on
|
||
2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, with the command
|
||
above. Every figure of those two modes that this document publishes was
|
||
reproduced exactly: BM25 Hit@1/5/10 and MRR at turn and session level, the
|
||
BM25 per-type session Hit@1 values quoted in prose below, and the stemmed
|
||
turn-level row plus its session Hit@1 in the tokenizer table. The run also
|
||
produced figures this document does not publish (stemmed session Hit@5,
|
||
Hit@10 and MRR, and per-type Hit@5/Hit@10/MRR for both modes), so there was
|
||
nothing to compare them with. Rows that need real embeddings (vector-only,
|
||
hybrid, RRF, re-ranking, the weight sweep) were not re-run.
|
||
|
||
### Full haystack — `longmemeval_s`, n=500 (the number to cite)
|
||
|
||
47.7 sessions and 493.5 turns per question; 4.0% of haystack sessions are evidence
|
||
sessions, so retrieval has to actually discriminate.
|
||
|
||
| Metric | Turn-level | Session-level |
|
||
|--------|-----------|---------------|
|
||
| Hit@1 | 53.8% | 86.2% |
|
||
| Hit@5 | **75.0%** | **93.6%** |
|
||
| Hit@10 | 81.6% | 96.6% |
|
||
| MRR | 0.6320 | 0.8948 |
|
||
|
||
Session-level is reported here because on this corpus it is meaningful — unlike on
|
||
the oracle variant, where it was degenerate and was retracted (below). At 4.0%
|
||
evidence density a session-level hit reflects discrimination rather than corpus
|
||
shape.
|
||
|
||
Per-type, session-level: `single-session-assistant` 100.0% Hit@1 (n=56),
|
||
`knowledge-update` 96.2% (n=78), `single-session-user` 94.3% (n=70),
|
||
`multi-session` 84.2% (n=133), `temporal-reasoning` 84.2% (n=133), and
|
||
`single-session-preference` 33.3% (n=30) — the one category where BM25 clearly
|
||
struggles, since a preference question's evidence rarely shares vocabulary with
|
||
the question.
|
||
|
||
### Retrieval mode ablation — full haystack, n=500
|
||
|
||
Real 384-d `all-MiniLM-L6-v2` embeddings, 190,015 unique texts encoded once on an
|
||
RTX 5060 Ti (~13 min; the same work on the 8-core CPU was still unfinished after
|
||
30 minutes, so the GPU path is not a convenience here). Turn-level:
|
||
|
||
| Mode | Hit@1 | Hit@5 | Hit@10 | MRR |
|
||
|------|-------|-------|--------|-----|
|
||
| BM25 only (`0.0`/`1.0`) | **53.8%** | 75.0% | 81.6% | **0.6320** |
|
||
| Vector only (`1.0`/`0.0`) | 36.0% | 71.8% | 81.6% | 0.5027 |
|
||
| Hybrid (`0.7`/`0.3`) | 44.4% | **79.2%** | **86.0%** | 0.5868 |
|
||
|
||
Session-level:
|
||
|
||
| Mode | Hit@1 | Hit@5 | Hit@10 | MRR |
|
||
|------|-------|-------|--------|-----|
|
||
| BM25 only | 86.2% | 93.6% | 96.6% | 0.8948 |
|
||
| Vector only | 85.4% | 94.2% | 96.6% | 0.8901 |
|
||
| Hybrid | **88.2%** | **95.8%** | **97.8%** | **0.9158** |
|
||
|
||
### Fusion method — weighted vs. RRF, full haystack, n=500
|
||
|
||
Reciprocal rank fusion has been in the codebase since early on but was only
|
||
reachable as a free function over a linear scan, so it had never been compared
|
||
with the weighted sum on equal terms. `HDF5Memory::hybrid_search_with` now
|
||
takes a `Fusion`, and both run over the same HNSW + BM25 candidates:
|
||
|
||
| Mode | turn Hit@1 | Hit@5 | Hit@10 | MRR | session Hit@1 | session MRR |
|
||
|---|---|---|---|---|---|---|
|
||
| BM25 only | **53.8%** | 75.0% | 81.6% | 0.6320 | 86.2% | 0.8948 |
|
||
| Vector only | 36.0% | 71.8% | 81.6% | 0.5031 | 85.4% | 0.8901 |
|
||
| **Weighted 0.4 / 0.6** | 51.6% | **81.4%** | **87.8%** | **0.6430** | **91.0%** | **0.9347** |
|
||
| RRF (k=60) | 45.0% | 78.8% | 87.6% | 0.5967 | 89.6% | 0.9253 |
|
||
|
||
**RRF loses to the tuned weighted sum** — 6.6pp of turn Hit@1 and 0.046 of MRR
|
||
— and lands almost exactly where the old `0.7/0.3` weighting did (44.2% /
|
||
0.5856). That is not a coincidence: RRF combines the two stages by rank with
|
||
*equal* influence, and on this corpus the stages are not equally good. BM25
|
||
alone beats the vector stage by 17.8pp at Hit@1, so any scheme that treats them
|
||
as peers gives up rank-1 accuracy, and RRF discards the score magnitudes that
|
||
would say which stage to believe.
|
||
|
||
This is a property of the corpus, not a defect in RRF: its selling point is
|
||
robustness when the two stages' scores are not comparable and there is no
|
||
labelled data to tune against. Here there is, so the weighted sum is kept as
|
||
the default. `Fusion::Rrf` remains available for callers whose stages are more
|
||
evenly matched.
|
||
|
||
### Keyword tokenizer — stemming, full haystack, n=500
|
||
|
||
The keyword stage lowercases and splits on non-alphanumerics, with no stemming,
|
||
so "training" and "trains" are unrelated terms. `TokenFilter::Stemmed` strips
|
||
common English inflections (plurals, `-ing`/`-ed`, with consonant un-doubling)
|
||
from documents and queries alike. Turn-level:
|
||
|
||
| Mode | Hit@1 | Hit@5 | Hit@10 | MRR | session Hit@1 |
|
||
|---|---|---|---|---|---|
|
||
| BM25 only | **53.8%** | 75.0% | 81.6% | 0.6320 | 86.2% |
|
||
| BM25 only, stemmed | 52.0% | 77.8% | 84.0% | 0.6320 | 88.0% |
|
||
| Hybrid 0.4/0.6 | 51.6% | **81.4%** | 87.8% | **0.6430** | 91.0% |
|
||
| Hybrid 0.4/0.6, stemmed | 50.2% | **81.4%** | **88.2%** | 0.6394 | **91.4%** |
|
||
|
||
**Stemming is a trade, not a win, and the default stays off.** It reliably buys
|
||
depth and costs the top rank: on BM25 alone, +2.8pp Hit@5 and +2.4pp Hit@10 for
|
||
−1.8pp Hit@1, with MRR unchanged to four decimal places — the gains deeper down
|
||
exactly offset the loss at rank 1. That is what conflation does: merging
|
||
"train"/"training"/"trains" surfaces documents an exact-match query would never
|
||
reach, and also lets a near-miss outrank the exact hit.
|
||
|
||
On the configuration that actually ships (hybrid 0.4/0.6) the trade is
|
||
narrower still — Hit@5 identical, Hit@10 +0.4pp, Hit@1 −1.4pp, MRR −0.004 —
|
||
because the vector stage already supplies much of the recall stemming would
|
||
add. There is no case here for changing the default; `TokenFilter::Stemmed`
|
||
is available via `HDF5Memory::set_token_filter` for callers who want Hit@5/@10
|
||
over rank-1 precision.
|
||
|
||
### Re-ranking and recency — full haystack, n=500
|
||
|
||
`reranker::rerank` combines temporal decay, source authority and Hebbian
|
||
activation. Until now its combined score contained **no relevance term at
|
||
all** — `RerankInput` did not carry the retrieval score — so a caller that
|
||
re-ranked its candidates threw the retriever's ordering away and returned them
|
||
ordered by age. The OpenClaw backend did exactly that on every search.
|
||
|
||
Measuring that is unambiguous. "Recency" below is the share of
|
||
`knowledge-update` questions where the newest gold session outranked the stale
|
||
one (see `newest_gold_first`); ~45% is chance.
|
||
|
||
| Mode | Hit@1 | Hit@5 | Hit@10 | MRR | recency |
|
||
|---|---|---|---|---|---|
|
||
| Hybrid 0.4/0.6, no re-rank | 51.6% | **81.4%** | 87.8% | 0.6430 | 45.0% |
|
||
| + re-rank, **metadata only** (pre-fix) | 11.0% | 24.8% | 43.8% | 0.1829 | **87.5%** |
|
||
| + re-rank, relevance-led, half-life 1 day | **52.0%** | 79.8% | 87.8% | 0.6403 | 51.7% |
|
||
| + re-rank, relevance-led, half-life 7 days | 51.8% | 80.8% | 87.6% | **0.6437** | **52.2%** |
|
||
| + re-rank, relevance-led, half-life 30 days | 51.8% | 81.0% | 87.8% | 0.6427 | 51.4% |
|
||
| + re-rank, relevance-led, half-life 90 days | **52.0%** | 80.4% | 87.8% | 0.6425 | 50.8% |
|
||
|
||
**The pre-fix row is the finding.** Ordering candidates by recency alone costs
|
||
40.6pp of Hit@1 and two thirds of MRR: the results are the newest memories in
|
||
the pool rather than the ones that answer the question. It does ace the recency
|
||
metric, which is exactly what makes that metric worth having — a number that
|
||
only goes up when a change is good would not have caught this.
|
||
|
||
With relevance leading, retrieval is preserved (Hit@1 +0.4pp, MRR −0.003
|
||
against no re-ranking) and recency discrimination gains 6–7pp. That is a real
|
||
improvement but not a solved problem: recency only breaks near-ties, so it
|
||
cannot reach the 87.5% the degenerate ordering gets. Those two rows are the
|
||
ends of a trade-off, and the default sits deliberately near the relevance end.
|
||
|
||
**Half-life is not a sensitive knob.** Across 1, 7, 30 and 90 days recency
|
||
moves 1.4pp and MRR 0.003 — inside the noise of a 500-question run — because
|
||
the temporal term is capped by its weight (0.3) while relevance differences
|
||
between candidates are larger. The 24-hour default is kept; there is no
|
||
measured reason to change it, and a corpus-matched value is not the lever it
|
||
looks like.
|
||
|
||
### Weight sweep — full haystack, n=500
|
||
|
||
`0.7/0.3` was a documented default, never a searched one. Sweeping
|
||
`vector_weight` from 0.0 to 1.0 (`--sweep`, reusing the one-time embedding
|
||
table) shows it is not merely suboptimal but **strictly dominated**:
|
||
|
||
| vector / keyword | Hit@1 | Hit@5 | Hit@10 | MRR | session Hit@5 |
|
||
|---|---|---|---|---|---|
|
||
| 0.0 / 1.0 (BM25) | **53.8%** | 75.0% | 81.6% | 0.6320 | 93.6% |
|
||
| 0.1 / 0.9 | 53.2% | 77.4% | 83.8% | 0.6374 | 95.0% |
|
||
| 0.2 / 0.8 | 53.6% | 78.2% | 85.6% | 0.6440 | 95.4% |
|
||
| 0.3 / 0.7 | 53.2% | 78.8% | 87.2% | **0.6463** | 96.0% |
|
||
| **0.4 / 0.6** | 51.6% | **81.4%** | 87.8% | 0.6429 | 96.8% |
|
||
| 0.5 / 0.5 | 48.2% | **81.4%** | **88.2%** | 0.6234 | **97.4%** |
|
||
| 0.6 / 0.4 | 46.6% | 79.8% | 87.4% | 0.6069 | 96.6% |
|
||
| 0.7 / 0.3 *(old default)* | 44.4% | 79.2% | 86.0% | 0.5868 | 95.8% |
|
||
| 0.8 / 0.2 | 40.6% | 76.2% | 85.4% | 0.5571 | 95.2% |
|
||
| 0.9 / 0.1 | 37.8% | 73.4% | 84.6% | 0.5289 | 94.2% |
|
||
| 1.0 / 0.0 (vector) | 36.0% | 71.8% | 81.6% | 0.5027 | 94.2% |
|
||
|
||
**`0.4/0.6` beats `0.7/0.3` on every metric at both granularities** — Hit@1
|
||
+7.2pp, Hit@5 +2.2, Hit@10 +1.8, MRR +0.056. There is no trade being made; the
|
||
old default was simply on the wrong side of the peak. **`0.4/0.6` is the
|
||
recommended setting**, with `0.3/0.7` preferable if rank-1 precision matters
|
||
most (it takes the best MRR in the sweep and gives up only 0.6pp of Hit@1
|
||
against pure BM25).
|
||
|
||
**Correction.** An earlier revision of this section, measuring only `0.7/0.3`,
|
||
concluded that fusion "buys deeper recall and pays for it at rank 1" and advised
|
||
callers taking a single top hit to prefer BM25. That was an artifact of the
|
||
badly-chosen weight, not a property of fusion. At `0.3/0.7` hybrid *beats* BM25
|
||
on MRR (0.6463 vs 0.6320) and on Hit@5 (78.8% vs 75.0%) while costing 0.6pp of
|
||
Hit@1. The advice below is corrected accordingly.
|
||
|
||
**Hybrid wins, once the weights are right.** At the old `0.7/0.3` the picture
|
||
looked like a trade: best at Hit@5 and Hit@10, worse than BM25 at Hit@1 and MRR.
|
||
The sweep above shows that was the weight, not fusion. At `0.4/0.6` hybrid leads
|
||
Hit@5 and Hit@10 outright; at `0.3/0.7` it also leads MRR and is within 0.6pp of
|
||
BM25 at Hit@1. Both dominate `0.7/0.3`.
|
||
|
||
The rows below are kept at the three original settings because they are what the
|
||
mode ablation measured — read them as "the shape of each stage in isolation",
|
||
and take the operating point from the sweep. `0.4/0.6` is now the shipped
|
||
default (`hybrid::DEFAULT_FUSION`).
|
||
|
||
The same pattern shows up independently in omni-cortex's four-signal RRF ablation,
|
||
where adding BM25 to a dense retriever raised nDCG@5 while lowering Hit@1 and MRR.
|
||
Two different codebases, two different fusion schemes, same direction.
|
||
|
||
Vector-only being *worse* than BM25 at every turn-level cutoff except Hit@10 is
|
||
worth stating plainly rather than hiding: LongMemEval questions share substantial
|
||
vocabulary with their evidence turns, which is close to the best case for lexical
|
||
matching, and MiniLM at 384 dimensions is a small embedding model.
|
||
|
||
> **Run:** `cargo run --release --bin longmemeval_bench --features embeddings -- \
|
||
> benchmarks/longmemeval/longmemeval_s_cleaned.json --embeddings weights/all-minilm-l6-v2`
|
||
> For the GPU path use `--features embeddings-cuda`. That requires `nvcc` on
|
||
> `PATH` at *build* time — cudarc's build script shells out to it. The toolkit
|
||
> installs to `/usr/local/cuda/bin`, which many distributions do not export;
|
||
> check with `nvcc --version` and, if it is missing, add it somewhere every
|
||
> shell reads (for zsh that is `~/.zshenv`, not `~/.zshrc`, since build tooling
|
||
> runs non-interactively). The device is selected at runtime with a CPU
|
||
> fallback, so a machine without CUDA still produces correct numbers — just far
|
||
> more slowly, and the bench says so on startup.
|
||
>
|
||
> Weights: `huggingface.co/sentence-transformers/all-MiniLM-L6-v2` — place
|
||
> `model.safetensors` and `tokenizer.json` in the `--embeddings` directory.
|
||
|
||
### Oracle variant — `longmemeval_oracle`, n=500 (easier corpus, kept for continuity)
|
||
|
||
| Metric | ClawhDF5 (BM25-only, oracle variant) |
|
||
|--------|--------------------------------------|
|
||
| Hit@1 | 52.6% |
|
||
| Hit@5 | **84.4%** |
|
||
| Hit@10 | 90.4% |
|
||
| MRR | 0.6597 |
|
||
|
||
Turn-level. The 9.4-point gap between this and the full haystack's 75.0% is the
|
||
price of the harder corpus, and is the reason oracle-only numbers should not be
|
||
presented as LongMemEval results. Session-level figures on this variant are
|
||
degenerate — see below.
|
||
|
||
With real embeddings the same oracle corpus gives BM25-only 84.2% / vector-only
|
||
80.4% / hybrid **85.2%** Hit@5 turn-level — hybrid ahead at Hit@5 and Hit@10 and
|
||
behind at Hit@1, matching the full-haystack pattern above. (BM25-only reads 84.2%
|
||
here against 84.4% with zero embedding vectors: one question of 500 changes rank,
|
||
with MRR identical at 0.6597. On the full haystack the two agree exactly.)
|
||
|
||
### Retracted: session-level recall and the MemX comparison
|
||
|
||
Earlier revisions of this file reported session-level Hit@1/5/10 of **100.0%** with
|
||
MRR **1.0000**, uniform across all six question types, and claimed clawhdf5
|
||
"outperforms MemX at turn-level retrieval (84.4% vs 51.6%)". **Both are withdrawn.**
|
||
|
||
**The session-level numbers are a degenerate artifact.** On the `longmemeval_oracle`
|
||
variant, the ingested haystack for a question consists essentially only of that
|
||
question's evidence sessions. Every returned document therefore belongs to an answer
|
||
session, so session-level hit rate is ≈1.0 at rank 0 *by construction* — which is
|
||
exactly why the result was a uniform 100.0% across every question type. It measured
|
||
the shape of the corpus, not the retriever.
|
||
|
||
**The MemX comparison was not like-for-like on two independent axes.** MemX
|
||
([arxiv:2603.16171](https://arxiv.org/abs/2603.16171)) reports Hit@5 = 51.6% /
|
||
MRR = 0.380 at **fact-level granularity over 220,349 fact-level records drawn from
|
||
19,195 sessions**, and explicitly notes that fact-level "doubl[es] session-level
|
||
performance." Our 84.4% is **turn-level, on the oracle subset**. Different retrieval
|
||
granularity, and a corpus smaller by orders of magnitude. A higher number on an
|
||
easier corpus at a different granularity is not an outperformance claim, and it
|
||
should not have been presented as one.
|
||
|
||
The full-haystack half of that gap is now closed: the section above reports
|
||
`longmemeval_s` over all 500 questions. The **granularity** mismatch remains — MemX
|
||
measures fact-level, we measure turn-level and session-level — so no cross-system
|
||
claim is made here even now. Matching granularity would require fact-level
|
||
extraction over the haystack, which this harness does not do.
|
||
|
||
### Search Latency (LongMemEval, n=500 queries)
|
||
|
||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, full
|
||
`longmemeval_s` haystack (47.7 sessions / 494 turns per question).
|
||
|
||
> **Run:** `cargo run --release -p clawhdf5-bench --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s.json`
|
||
|
||
| Metric | BM25 only | BM25 stemmed |
|
||
|--------|-----------|--------------|
|
||
| avg | 11,034.7 µs | 11,096.2 µs |
|
||
| p50 | 10,943.5 µs | 10,996.5 µs |
|
||
| p95 | 11,749.6 µs | 12,018.8 µs |
|
||
| p99 | 12,644.1 µs | 12,784.7 µs |
|
||
|
||
About 11 ms per query. Each question gets its own store, and the timed call is
|
||
the first `hybrid_search_with` on that store. The first query on a store
|
||
builds its indexes (the "cold index build" column of the search harness), so
|
||
this is a first-query figure, not steady-state search; for that
|
||
see the [search harness](#current-search-harness-2026-09-24).
|
||
|
||
The previous table (avg 1,004 µs, p50 1,017 µs) said "sub-millisecond median
|
||
search", which no longer holds. That figure was undated and predates the
|
||
full-haystack harness (added 2026-08-07, 7d6e269), so it was measured on the
|
||
much smaller `longmemeval_oracle` corpus; on that corpus tank measured avg
|
||
2,431 µs on 2026-08-05 (below).
|
||
|
||
---
|
||
|
||
## Multi-Session Benchmark (MemoryArena)
|
||
|
||
**Dataset:** Deterministic synthetic conversations — 50 sessions × ~20 turns = 999 turns
|
||
**Topics:** Personal info, food preferences, music, travel, work/schedule, hobbies
|
||
**Queries:** 35 questions across 4 types
|
||
|
||
> **Run:** `cargo run --release -p clawhdf5-bench --bin memory_arena`
|
||
|
||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||
|
||
### Results by Query Type
|
||
|
||
| Query Type | N | Hit@1 | Hit@5 | Hit@10 | MRR | Avg Latency |
|
||
|------------|---|-------|-------|--------|-----|-------------|
|
||
| single-session | 25 | 40.0% | 92.0% | 100.0% | 0.5755 | 248.4 µs |
|
||
| multi-session | 5 | 40.0% | 60.0% | 80.0% | 0.5333 | 24.4 µs |
|
||
| temporal | 3 | 33.3% | 66.7% | 66.7% | 0.5000 | 28.0 µs |
|
||
| knowledge-update | 2 | 0.0% | 50.0% | 50.0% | 0.2500 | 38.7 µs |
|
||
| **OVERALL** | **35** | **37.1%** | **82.9%** | **91.4%** | **0.5444** | p50 **30.1 µs**, p95 49.8 µs |
|
||
|
||
**Key findings:**
|
||
- Hit@10 of 91.4% across all query types with BM25-only (no embeddings)
|
||
- Single-session recall strongest at 100% Hit@10
|
||
- Knowledge-update hardest (requires temporal disambiguation) — would improve significantly with vector similarity
|
||
- Search takes about 30 µs at p50 over the 999 turns. All 35 queries run
|
||
against one store, and single-session's higher average (248.4 µs) includes
|
||
that store's first query, which builds its indexes.
|
||
|
||
The previous table put every query at ~7.9 ms ("latency dominated by BM25
|
||
index build over 999 turns"). That was before v2.4.0, when
|
||
`HDF5Memory::hybrid_search` rebuilt the BM25 index from scratch and rewrote
|
||
the store on every query; it now keeps the index for the life of the store.
|
||
The MRR moved slightly (single-session 0.5788 -> 0.5755, overall 0.5468 ->
|
||
0.5444); the Hit@k figures are unchanged.
|
||
|
||
---
|
||
|
||
## Memory Footprint
|
||
|
||
HDF5 file size at various record counts — 384-dimensional embeddings, 200-char text.
|
||
|
||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||
|
||
> **Run:** `cargo run --release -p clawhdf5-bench --bin footprint_bench`
|
||
|
||
**These are `float16` stores.** `footprint_bench` builds its stores with
|
||
`MemoryConfig::new`, which defaults to `float16` embeddings since 5c8323c,
|
||
although the binary still prints "Embedding: 384-dim f32". Its "Raw Data"
|
||
column, and so the ratios, count embeddings as `f32` (1,536 bytes per record)
|
||
plus text. The synthetic embeddings take only 1,000 distinct values, and each
|
||
record's text is fixed by `record_idx % 40` (a 40-word bank walked from that
|
||
start), so a store holds only 40 distinct texts. Both compress far better than
|
||
real data would; read the gzip figures as an upper bound, and see the next
|
||
paragraph for why the other tables are optimistic too. "MB" and "KB" are the
|
||
binary's units (powers of 1024).
|
||
|
||
**No table here is uncompressed.** The store always deflates its string
|
||
datasets: `write_string_dataset` applies deflate level 6 to any string dataset
|
||
of 4 KiB or more, whatever `MemoryConfig::compression` says
|
||
(`crates/clawhdf5-agent/src/schema.rs`, `STRING_COMPRESS_THRESHOLD`, added in
|
||
4fa7e89 on 2026-06-03). With only 40 distinct texts, the chunk text costs
|
||
almost nothing on disk: at 10K records, 50-char text takes 804 B per record
|
||
and 1000-char text 944 B, so 950 more characters cost only 140 B. The
|
||
`float16` embeddings alone are 768 B per record, and 200 bytes of raw text
|
||
would bring that to about 968 B before any other field or overhead; the
|
||
measured 803–829 B is below that only because the text is compressed away.
|
||
Real text will not deflate like this, so expect more than 820 B per record
|
||
with real data. `MemoryConfig::compression` decides only whether the
|
||
embeddings are compressed; the gzip table below is that setting turned on.
|
||
|
||
### `compression` off (no WAL)
|
||
|
||
| Records | File Size | Raw Data (f32) | Bytes/Record | Throughput |
|
||
|---------|-----------|----------------|--------------|------------|
|
||
| 100 | 90.0 KB | 169.5 KB | 921 B | 75,914 rec/s |
|
||
| 1K | 810.4 KB | 1.7 MB | 829 B | 72,266 rec/s |
|
||
| 10K | 7.8 MB | 16.6 MB | 820 B | 74,509 rec/s |
|
||
| 50K | 38.3 MB | 82.8 MB | 803 B | 72,829 rec/s |
|
||
| 100K | 76.7 MB | 165.6 MB | 803 B | 71,102 rec/s |
|
||
|
||
**About 820 B per record** at 10K, for this synthetic corpus (see above: the
|
||
text is deflated regardless and has only 40 distinct values, so real data
|
||
takes more). Ingestion runs at 71,102–75,914 records/sec.
|
||
|
||
### With Gzip Compression (level 6)
|
||
|
||
| Records | Compressed | Ratio vs f32 raw | Bytes/Record |
|
||
|---------|------------|------------------|--------------|
|
||
| 100 | 17.2 KB | 9.85x | 176 B |
|
||
| 1K | 56.4 KB | 30.06x | 57 B |
|
||
| 10K | 471.3 KB | 35.97x | 48 B |
|
||
| 50K | 2.3 MB | 36.67x | 47 B |
|
||
| 100K | 4.5 MB | 36.73x | 47 B |
|
||
|
||
### Text Length Comparison (10K records, `compression` off)
|
||
|
||
| Text Length | File Size | Bytes/Record | Throughput |
|
||
|-------------|-----------|--------------|------------|
|
||
| short (50 chars) | 7.7 MB | 804 B | 97,615 rec/s |
|
||
| medium (200 chars) | 7.8 MB | 820 B | 89,597 rec/s |
|
||
| long (1000 chars) | 9.0 MB | 944 B | 51,462 rec/s |
|
||
|
||
### WAL Overhead (1K records)
|
||
|
||
| Mode | File Size | Ingest Time | Overhead |
|
||
|------|-----------|-------------|----------|
|
||
| No WAL | 810.4 KB | 10.8 ms | — |
|
||
| With WAL | 810.4 KB + 9 B WAL | 10.6 ms | −2% (negligible) |
|
||
|
||
**Compared with the previous tables.** Those were undated `f32` stores from the
|
||
original i7-12650H run: 1.7 KB per record (169.8 MB at 100K), gzip 6.2x
|
||
(26.9 MB at 100K), and more than 100,000 records/sec. Two changes separate
|
||
those tables from these, not one. `float16` embeddings save 768 B per record.
|
||
The old tables (recorded by 2026-05-14) also predate 4fa7e89 (2026-06-03),
|
||
which began deflating string datasets of 4 KiB or more, so their text was
|
||
stored raw; with only 40 distinct texts, deflate now removes most of it.
|
||
An `f32` embedding plus 200 B of text is 1,536 + 200 = 1,736 B; `float16`
|
||
alone would bring that to about 968 B per record, not the measured 803–829 B. This run does not measure how the saving splits between the two
|
||
beyond that, nor explain why the gzip ratio rose so much further, nor
|
||
separate the lower throughput (and the 1K ingest going from 5.7 to 10.8 ms)
|
||
from the change of machine and store configuration. The
|
||
`f32` on-disk figures were not re-measured; the dated
|
||
[float16 study](#float16-embedding-storage-memoryconfigfloat16) compares the
|
||
two directly (100K × 384: 154.0 MiB `f32`, 80.8 MiB `float16`).
|
||
|
||
---
|
||
|
||
## Consolidation Efficiency
|
||
|
||
Hippocampal-inspired memory consolidation improves both retrieval quality and search speed.
|
||
|
||
> **Run:** `cargo run --release -p clawhdf5-bench --bin consolidation_efficiency`
|
||
|
||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. The run
|
||
was stopped (after about 19 minutes on one core) while still computing a
|
||
100K row of the cycle-time table, so that row and the binary's memory-reduction
|
||
part were not produced; neither has ever been published here.
|
||
|
||
### Retrieval Quality Before vs. After Consolidation
|
||
|
||
**Setup:** 1,000 records (10 signal + 990 noise), working_capacity=100
|
||
|
||
| Metric | Before | After | Delta |
|
||
|--------|--------|-------|-------|
|
||
| Records in store | 1,000 | 100 | −90% |
|
||
| Hit@1 | 100.0% | 100.0% | — |
|
||
| Hit@5 | 100.0% | 100.0% | — |
|
||
| Hit@10 | 100.0% | 100.0% | — |
|
||
| MRR | 1.0000 | 1.0000 | — |
|
||
| Search latency (avg) | 2,223.7 µs | 239.2 µs | **9.3x faster** |
|
||
|
||
Signal records survive consolidation because they are accessed 15+ times, giving them high decay scores. 900 noise records evicted, search speeds up 9.3x, and **zero quality loss** — perfect recall maintained. The consolidation cycle itself took 0.13 ms and promoted 2 records.
|
||
|
||
### Consolidation Cycle Time
|
||
|
||
| Records | Cycle Time | Evictions | Promotions |
|
||
|---------|-----------|-----------|------------|
|
||
| 100 | 17 µs | 100 | 0 |
|
||
| 1K | 189 µs | 1,000 | 0 |
|
||
| 10K | 2.16 ms | 10,000 | 0 |
|
||
|
||
The 1K and 10K cycles were 345 µs and 17.3 ms in the previous, undated table
|
||
(before: 2,752 µs / after: 312 µs for search). Eviction's membership check
|
||
changed from a linear scan to a `HashSet` in 603fcf8 (2026-08-17), after those
|
||
figures were recorded; this run does not isolate its effect.
|
||
|
||
---
|
||
|
||
## Ephemeral Tier (Redis Comparison)
|
||
|
||
In-memory key-value store with TTL, capacity eviction, and embedding search.
|
||
No network hop, no serialization — direct HashMap operations.
|
||
|
||
> **Run:** `cargo run --release -p clawhdf5-bench --bin ephemeral_perf`
|
||
|
||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||
|
||
### Latency Comparison
|
||
|
||
| Operation | clawhdf5 Ephemeral | Redis (single-node)¹ | Ratio¹ |
|
||
|-----------|-------------------|---------------------|---------|
|
||
| SET | **366 ns/op** | ~25,000 ns/op (not measured) | 68.3x |
|
||
| GET (hit) | **115 ns/op** | ~25,000 ns/op (not measured) | 216.7x |
|
||
| GET (miss) | **57 ns/op** | — | — |
|
||
| DELETE | **99 ns/op** | — | — |
|
||
| SET+embedding | **236 ns/op** | N/A | — |
|
||
|
||
> ¹ **The Redis figure is not a measurement.** It is a constant hard-coded in
|
||
> `ephemeral_perf` as a "typical single-node" latency, and the binary computes
|
||
> the two ratios from it; only the clawhdf5 column was measured. A typical
|
||
> Redis figure includes a network round-trip, which the in-process ephemeral
|
||
> tier does not have, so the ratios compare different things.
|
||
|
||
### Throughput
|
||
|
||
| Operation | ops/sec |
|
||
|-----------|---------|
|
||
| SET | 2,732,568 |
|
||
| GET (hit) | 8,666,607 |
|
||
| GET (miss) | 17,554,663 |
|
||
| DELETE | 10,109,895 |
|
||
| SET+EMB (384d) | 4,235,766 |
|
||
|
||
### Embedding Search (ephemeral tier)
|
||
|
||
| Scale | Latency |
|
||
|-------|---------|
|
||
| 10K entries @ 384d | 2.85 ms/query |
|
||
|
||
GET (hit) moved from 179 to 115 ns/op and DELETE from 124 to 99 ns/op (GET
|
||
throughput 5,584,684 -> 8,666,607 ops/sec); the previous figures were undated.
|
||
|
||
---
|
||
|
||
## World-Model Sample Loading (vs h5py / stable-worldmodel shape)
|
||
|
||
Reproduces the access pattern of `stable-worldmodel`'s HDF5 dataloader
|
||
([arXiv 2605.21800](https://arxiv.org/abs/2605.21800), LeCun/Balestriero
|
||
group), which supports HDF5 as one of three native formats and measures
|
||
generic HDF5 at **1,416-1,474 samples/s** (vs Lance 4,815) for per-frame
|
||
sample loading. This benchmark measures **clawhdf5 vs h5py on the same
|
||
machine and the same file**, so the comparison is hardware-controlled.
|
||
|
||
**Absolute numbers are not comparable to the paper's** - different hardware
|
||
(AMD Ryzen 7 7800X3D, local NVMe, warm page cache), smaller frames, and no
|
||
torch-tensor / transform step. Only the clawhdf5-vs-h5py ratio *here* is a
|
||
controlled result. The workload is the dataloader shape: a `(N, H, W, C)`
|
||
uint8 observation dataset (20,000 x 64x64x3 = 246 MB), each frame read once
|
||
per pass in a fixed shuffled (random-access) order, 10 passes.
|
||
|
||
Both read a **file written by h5py** - clawhdf5 parsing an
|
||
externally-produced HDF5 file is itself the interop result. h5py opens SWMR
|
||
with a 256 MB chunk cache, exactly `stable-worldmodel`'s `HDF5Dataset`; it
|
||
materialises each frame as a numpy array (`d[i]`) and sums it. clawhdf5
|
||
mmaps once, takes a zero-copy `&[u8]` over the contiguous dataset, and
|
||
indexes frame `i` as a subslice.
|
||
|
||
| Reader | samples/sec (median of 3) | vs h5py |
|
||
|--------|---------------------------|---------|
|
||
| **clawhdf5** (zero-copy view) | **593,000** | **8.1x** |
|
||
| **clawhdf5** (materialised copy per frame) | **518,000** | **7.1x** |
|
||
| h5py (swmr, 256 MB cache) | 73,000 | 1.0x |
|
||
|
||
The **materialised-copy row is the fair, equal-work comparison** - it
|
||
`to_vec()`s every frame so clawhdf5 pays the same per-frame allocation h5py
|
||
does, and it is still **7.1x faster**. That the copy costs almost nothing
|
||
(518k vs 593k) shows the h5py gap is **per-frame call overhead** (Python +
|
||
library dispatch), not data movement. This is an in-page-cache measurement:
|
||
it isolates the read-path overhead both libraries add on top of the OS,
|
||
which is the thing that differs - not disk bandwidth, which is shared.
|
||
|
||
Reproduce (`benchmarks/`):
|
||
|
||
```bash
|
||
python benchmarks/gen_worldmodel_frames.py /tmp/wm_frames.h5 20000
|
||
cargo run --release -p clawhdf5-bench --example worldmodel_sampling -- /tmp/wm_frames.h5 10
|
||
cargo run --release -p clawhdf5-bench --example worldmodel_sampling -- /tmp/wm_frames.h5 10 --copy
|
||
python benchmarks/bench_worldmodel_h5py.py /tmp/wm_frames.h5 10
|
||
```
|
||
|
||
Measured 2026-08-07 on tank (Ryzen 7 7800X3D, 246 MB dataset in page cache).
|
||
|
||
## Cross-Platform Notes
|
||
|
||
> **Run:** `./benchmarks/cross_platform.sh [--full] [--output results.json]`
|
||
|
||
### Measured Platforms
|
||
|
||
The tank row was measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit
|
||
5c8323c, with the single benchmark rather than the whole script:
|
||
|
||
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^ivf_search_10k_nprobe10$'`
|
||
|
||
| Platform | CPU | 10K IVF Search | Notes |
|
||
|----------|-----|----------------|-------|
|
||
| Linux x86_64 | AMD Ryzen 7 7800X3D (8C/16T), tank | 24.8 µs | Measured 2026-09-24 |
|
||
| Linux x86_64 | Intel i7-12650H (10C, 4.7 GHz) | 27 µs | Undated, original run |
|
||
| macOS aarch64 | Apple M3 Max (14C) | ~18 µs | Undated; ~33% faster than the i7-12650H row. Not reproducible on the hardware available for the 2026-09-24 re-run |
|
||
|
||
### Reproducibility
|
||
|
||
```bash
|
||
rustup override set nightly
|
||
|
||
# Latency benchmarks (Criterion)
|
||
cargo bench -p clawhdf5-agent
|
||
|
||
# Full benchmark suite
|
||
cargo run --release --bin longmemeval_bench
|
||
cargo run --release --bin memory_arena
|
||
cargo run --release --bin footprint_bench
|
||
cargo run --release --bin consolidation_efficiency
|
||
cargo run --release --bin ephemeral_perf
|
||
```
|
||
|
||
---
|
||
|
||
## Deflate backend: zlib-rs vs zlib-ng
|
||
|
||
Measured 2026-09-23 on tank (AMD Ryzen 7 7800X3D, 8C/16T). The default
|
||
deflate backend is now **zlib-rs**, a pure-Rust port of zlib-ng; zlib-ng (C,
|
||
built with cmake) was the default before and is still available as
|
||
`fast-deflate`. Both builds were compiled once into separate target
|
||
directories and run **alternately, three rounds each**; figures are medians.
|
||
|
||
```bash
|
||
# zlib-rs (default)
|
||
cargo bench -p clawhdf5-filters --bench deflate_bench
|
||
cargo bench -p clawhdf5-bench --bench h5bench_write --features libhdf5-compare -- '^write_2d_chunked/'
|
||
cargo run --release -p clawhdf5-bench --bin read_harness
|
||
# zlib-ng: add --features fast-deflate (filters) or clawhdf5-format/fast-deflate (bench)
|
||
```
|
||
|
||
| Workload | zlib-rs | zlib-ng | rs / ng |
|
||
|---|---:|---:|---:|
|
||
| HDF5 chunked write, deflate-6, 512×512 f32 | 1.458 ms | 1.484 ms | 0.98 |
|
||
| HDF5 chunked write, deflate-6, 128×128 f32 | 157.6 µs | 152.8 µs | 1.03 |
|
||
| HDF5 chunked write, deflate-6, 32×32 f32 | 62.9 µs | 60.2 µs | 1.05 |
|
||
| HDF5 read, 64 MB chunked + deflate, full | 64.4 ms | 65.2 ms | 0.99 |
|
||
| HDF5 read, 64×64 window (1 chunk) | 0.18 ms | 0.17 ms | 1.06 |
|
||
| HDF5 read, 512×512 window (4–9 chunks) | 4.10 ms | 4.10 ms | 1.00 |
|
||
| HDF5 read, one row / one column | 1.00 / 2.01 ms | 0.95 / 1.95 ms | 1.05 / 1.03 |
|
||
| Raw inflate, 8 MB f64 | 5.92 ms | 6.06 ms | 0.98 |
|
||
| Raw inflate, 1 MB sine | 82.8 µs | 68.6 µs | 1.21 |
|
||
| Raw deflate-6, 8 MB f64 / 1 MB sine | 92.2 / 2.01 ms | 83.1 / 1.84 ms | 1.11 / 1.09 |
|
||
|
||
Compressed output is **byte-identical** between the two at levels 1, 6 and 9
|
||
on all three inputs, so files do not change size. libhdf5 1.14.6 took 51.4 ms
|
||
for the 512×512 write in the same session (35× the zlib-rs figure).
|
||
|
||
On the HDF5 paths zlib-rs is within 6% of zlib-ng everywhere, and ahead on the
|
||
largest write. The raw codec loops show zlib-ng still slightly faster at
|
||
compression (~10%), which chunked writes do not expose because encoding runs
|
||
in parallel across chunks.
|
||
|
||
**Two findings along the way.** The first measurement had zlib-rs 1.2–1.9×
|
||
slower on single-chunk reads and 3.7× slower on a 1 MB inflate — slower even
|
||
than miniz_oxide. Neither was zlib-rs's fault:
|
||
|
||
1. **Runtime CPU detection was off.** zlib-rs needs its `std` feature to
|
||
detect and use SIMD at runtime; flate2 turns it on through its default
|
||
`runtime_detection` feature, which our `default-features = false` flate2
|
||
dependency was disabling. With it, a 1 MB inflate goes 282 → 83 µs.
|
||
`clawhdf5-format/zlib-rs` and `clawhdf5-filters/zlib-rs` now enable it.
|
||
2. **The codec was fed through a 32 KiB buffer.** Both deflate paths used
|
||
flate2's streaming `read::ZlibDecoder` / `write::ZlibEncoder`. A chunk's
|
||
decompressed size is known, so they now hand the codec the whole input in
|
||
one call, into an output buffer sized up front. Worth ~5% on chunked
|
||
writes and ~10% on zlib-ng's 1 MB inflate. It also closed a hole: the
|
||
streaming reader returned a truncated stream's bytes without an error, so
|
||
a truncated chunk read back short; it is now an error.
|
||
|
||
| 1 MB inflate, same build otherwise | zlib-rs | zlib-ng |
|
||
|---|---:|---:|
|
||
| streaming reader, no runtime detection | 284.1 µs | 76.7 µs |
|
||
| one-shot, no runtime detection | 282.3 µs | 68.5 µs |
|
||
| one-shot + runtime detection (shipped) | **82.8 µs** | **68.6 µs** |
|
||
|
||
---
|
||
|
||
## h5bench-Equivalent I/O Benchmarks
|
||
|
||
Criterion harness mirroring h5bench serial workloads. clawhdf5 benchmarks dated 2026-07-01;
|
||
libhdf5 1.14.6 head-to-head comparison dated 2026-06-30 (same hardware, same Criterion harness).
|
||
|
||
```bash
|
||
cargo bench -p clawhdf5-bench # clawhdf5-only
|
||
cargo bench -p clawhdf5-bench --features libhdf5-compare # head-to-head
|
||
```
|
||
|
||
### Sequential Read Throughput
|
||
|
||
Both read a 1-D contiguous f32 dataset. clawhdf5 parses from `Vec<u8>` (zero-copy);
|
||
libhdf5 reads from a temp file including `open` + `read` + `close` overhead.
|
||
|
||
| Workload | n=1K | n=10K | n=100K |
|
||
|----------|------|-------|--------|
|
||
| **clawhdf5** f32 | 634 ns / **5.9 GiB/s** | 2.44 µs / **15.3 GiB/s** | 24.5 µs / **15.2 GiB/s** |
|
||
| libhdf5 f32 | 45.2 µs / 85 MiB/s | 47.8 µs / 799 MiB/s | 73.9 µs / 5.0 GiB/s |
|
||
| **Speedup** | **71×** | **20×** | **3.0×** |
|
||
| clawhdf5 f64 | 743 ns / **10.0 GiB/s** | 4.17 µs / **17.8 GiB/s** | 43.3 µs / **17.2 GiB/s** |
|
||
| clawhdf5 from_disk (f64, OS I/O) | — | 10.1 µs / **7.4 GiB/s** | 77.6 µs / **9.6 GiB/s** |
|
||
| clawhdf5 hyperslab (f64, 10% slice) | — | 4.09 µs / **1.8 GiB/s** | 50.1 µs / **1.5 GiB/s** |
|
||
|
||
libhdf5 f64 comparison excluded — clawhdf5's datatype encoding differs from libhdf5's (known
|
||
gap), making cross-format reads unreliable for comparison.
|
||
|
||
### Chunked Read Throughput
|
||
|
||
| Matrix size | Latency | Throughput |
|
||
|-------------|---------|-----------|
|
||
| 64×64 f32 | 6.39 µs | **2.4 GiB/s** |
|
||
| 256×256 f32 | 41.7 µs | **5.9 GiB/s** |
|
||
| 512×512 f32 | 176 µs | **5.5 GiB/s** |
|
||
|
||
### Sequential Write Throughput
|
||
|
||
Both write to disk. At 100K elements both converge on the OS `write()` syscall ceiling.
|
||
|
||
| Workload | n=1K | n=10K | n=100K |
|
||
|----------|------|-------|--------|
|
||
| **clawhdf5** f32 | 9.44 µs / **404 MiB/s** | 25 µs / **1.49 GiB/s** | 228 µs / **1.63 GiB/s** |
|
||
| libhdf5 f32 | 77.9 µs / 49 MiB/s | 87.8 µs / 435 MiB/s | 214 µs / 1.74 GiB/s |
|
||
| **Speedup** | **8.2×** | **3.5×** | **≈ tie** |
|
||
| clawhdf5 f64 embeddings | 6.50 µs (n=128) | 8.67 µs (n=512) / **450 MiB/s** | 10.27 µs (n=1K) / **761 MiB/s** |
|
||
|
||
### Chunked Write: Codec Comparison (with auto-shuffle)
|
||
|
||
Auto-shuffle is applied before all compression codecs by default — AoS→SoA byte transpose,
|
||
implements byte-grouping pre-filter per arXiv:2506.18062. Shuffle dramatically improves
|
||
throughput for float/int data by creating long runs of similar bytes.
|
||
|
||
| Matrix size | Zstd-3 + shuffle | Deflate-6 + shuffle | Speedup |
|
||
|-------------|-----------------|---------------------|---------|
|
||
| 32×32 f32 | 48 µs / **81 MiB/s** | 39 µs / **100 MiB/s** | Deflate 1.23× faster (small chunk) |
|
||
| 128×128 f32 | **148 µs / 422 MiB/s** | 153 µs / **407 MiB/s** | Parity |
|
||
| 512×512 f32 | **1.34 ms / 748 MiB/s** | 1.39 ms / **719 MiB/s** | Zstd 1.04× faster |
|
||
|
||
Impact of auto-shuffle vs no-shuffle baseline:
|
||
|
||
| Matrix size | Zstd-3 speedup | Deflate-6 speedup |
|
||
|-------------|----------------|-------------------|
|
||
| 32×32 | +19% | +38% |
|
||
| 128×128 | +25% | **+204%** |
|
||
| 512×512 | +25% | **+157%** |
|
||
|
||
Both codecs perform at parity at large sizes (~720–750 MiB/s). Use `.with_zstd(3)` or
|
||
`.with_deflate(6)` for write-heavy workloads. Use `.without_shuffle()` only for byte arrays
|
||
or data that doesn't benefit from AoS→SoA transposition.
|
||
|
||
### Chunked Write vs libhdf5 (deflate-6)
|
||
|
||
clawhdf5 compresses all chunks in memory and issues a single `write()`. libhdf5 flushes each
|
||
chunk individually via its Virtual File Layer (one `pwrite()` per chunk).
|
||
|
||
| Matrix | clawhdf5 deflate-6 + shuffle | libhdf5 deflate-6 | Speedup |
|
||
|--------|------------------------------|-------------------|---------|
|
||
| 32×32 f32 | 39 µs / 100 MiB/s | 172 µs / 23 MiB/s | **4.4×** |
|
||
| 128×128 f32 | 153 µs / 407 MiB/s | 3,150 µs / 20 MiB/s | **20.6×** |
|
||
| 512×512 f32 | 1,390 µs / 719 MiB/s | 53,300 µs / 19 MiB/s | **38.4×** |
|
||
|
||
The 32×32 speedup (4.4×) is lower than the 512×512 speedup (38.4×) because shuffle adds
|
||
overhead that dominates at 4 KB chunks. libhdf5 was benchmarked without shuffle. The speedup
|
||
compounds with matrix size because libhdf5's per-chunk VFL overhead is proportional to chunk
|
||
count while clawhdf5's single-pass cost is constant.
|
||
|
||
### Codec Comparison: Pcodec vs Zstd-3
|
||
|
||
Pcodec (arXiv:2502.06112) is a pure-Rust lossless numerical codec with 30–94% better compression
|
||
ratio than Zstd for f32/f64 columns. Both sides benchmarked **without** auto-shuffle here (shuffle
|
||
degrades Pcodec which handles byte organization internally; Zstd-3 without shuffle numbers shown
|
||
for an apples-to-apples comparison).
|
||
|
||
| Matrix size | Pcodec | Zstd-3 (no shuffle) | Winner |
|
||
|-------------|--------|---------------------|--------|
|
||
| 32×32 f32 | 95 µs / **41 MiB/s** | 57 µs / **68 MiB/s** | Zstd-3 (1.66×) |
|
||
| 128×128 f32 | 528 µs / **118 MiB/s** | 179 µs / **349 MiB/s** | Zstd-3 (2.95×) |
|
||
| 512×512 f32 | 1.69 ms / **591 MiB/s** | 1.64 ms / **610 MiB/s** | Parity (3% diff) |
|
||
|
||
Pcodec's fixed per-chunk distributional analysis overhead (~400 µs) dominates at 32×32 (4 KB).
|
||
At 512×512 (1 MB) the speeds converge. **Pcodec's advantage is compression ratio, not encode
|
||
speed** — less data on disk means faster reads and lower storage cost. Enable with
|
||
`.with_pcodec()` for write-once/read-many workloads (embedding archives, scientific datasets).
|
||
|
||
### Metadata Throughput
|
||
|
||
clawhdf5 accumulates all metadata in memory and serializes in one pass. libhdf5 acquires a
|
||
global file mutex and flushes to disk on every attribute write or group creation.
|
||
|
||
**Attributes and datasets** (k = attribute or dataset count):
|
||
|
||
| Workload | k=4 | k=16 | k=64 | k=128 |
|
||
|----------|-----|------|------|-------|
|
||
| **clawhdf5** attrs_write (i64) | 8.05 µs / 494 Kop/s | 17.2 µs / 932 Kop/s | 49.2 µs / 1.30 Mop/s | 87.3 µs / 1.47 Mop/s |
|
||
| libhdf5 attrs_write | 100 µs / 40 Kop/s | 170 µs / 94 Kop/s | 472 µs / 136 Kop/s | 929 µs / 138 Kop/s |
|
||
| **Speedup** | **12.4×** | **9.9×** | **9.6×** | **10.6×** |
|
||
| clawhdf5 attrs_read | 1.06 µs / 3.78 Mop/s | 3.64 µs / 4.39 Mop/s | 15.7 µs / 4.08 Mop/s | 31.3 µs / 4.09 Mop/s |
|
||
| clawhdf5 string_attrs (write+read) | 5.17 µs / 774 Kop/s | 16.5 µs / 967 Kop/s | 33.6 µs / 951 Kop/s | — |
|
||
| clawhdf5 multi_dataset_write | 10.1 µs / 397 Kop/s | 31.5 µs / 508 Kop/s | 104 µs / 614 Kop/s | — |
|
||
|
||
**Groups** (k = group count):
|
||
|
||
| Workload | k=4 | k=16 | k=32 | k=64 |
|
||
|----------|-----|------|------|------|
|
||
| **clawhdf5** groups_create | 12.1 µs / 330 Kop/s | 33.7 µs / 475 Kop/s | 66.7 µs / 480 Kop/s | 121 µs / 529 Kop/s |
|
||
| libhdf5 groups_create | 140 µs / 28 Kop/s | 433 µs / 37 Kop/s | 690 µs / 46 Kop/s | 1,340 µs / 48 Kop/s |
|
||
| **Speedup** | **11.6×** | **12.8×** | **9.5×** | **11.1×** |
|
||
| clawhdf5 groups_traverse | 664 ns / 6.0 Mop/s | 3.55 µs / 4.5 Mop/s | 4.87 µs / 6.6 Mop/s | 10.6 µs / 6.0 Mop/s |
|
||
|
||
---
|
||
|
||
## vs libhdf5 Summary
|
||
|
||
| Workload | clawhdf5 | libhdf5 | Speedup |
|
||
|----------|----------|---------|---------|
|
||
| Sequential read, 1K f32 | 634 ns | 45.2 µs | **71×** |
|
||
| Sequential read, 100K f32 | 24.5 µs · 15.2 GiB/s | 73.9 µs · 5.0 GiB/s | **3.0×** |
|
||
| Sequential write, 100K f32 | 228 µs · 1.63 GiB/s | 214 µs · 1.74 GiB/s | **≈ tie** |
|
||
| Chunked write deflate-6, 512×512 | 1,390 µs · 719 MiB/s | 53,300 µs · 19 MiB/s | **38.4×** |
|
||
| Attribute write, 128 attrs | 87.3 µs · 1.47 Mop/s | 929 µs · 138 Kop/s | **10.6×** |
|
||
| Group create, 64 groups | 121 µs · 529 Kop/s | 1,340 µs · 48 Kop/s | **11.1×** |
|
||
|
||
### Why the Gaps
|
||
|
||
**Metadata (10–13×):** libhdf5 was designed for MPI parallel filesystems where every metadata
|
||
write must be immediately visible to other processes. It acquires a global file mutex and
|
||
flushes to disk per operation. clawhdf5 builds the entire file in memory and writes it in one
|
||
shot — no locking, no flushing, no C heap allocation per message.
|
||
|
||
**Chunked compressed write (4–38×):** libhdf5 writes each chunk individually through its VFL
|
||
(Virtual File Layer), one `pwrite()` per chunk. clawhdf5 compresses all chunks in memory (Rayon
|
||
parallel when > 2 chunks), lays them out contiguously, and issues a single `write()`. The
|
||
speedup compounds with matrix size: libhdf5's per-chunk overhead is proportional to chunk count
|
||
while clawhdf5's architectural cost is constant.
|
||
|
||
**Small reads (20–71×):** libhdf5's per-open overhead (chunk cache init, SWMR lock, metadata
|
||
read) dominates at sub-millisecond payloads. clawhdf5 has no global state — `File::from_bytes()`
|
||
starts parsing immediately.
|
||
|
||
**Large contiguous writes (≈ tie at 100K):** Both are bottlenecked by the OS `write()` syscall
|
||
to the page cache. There is no algorithmic headroom above ~1.7 GiB/s on this hardware.
|
||
|
||
### Caveats
|
||
|
||
- libhdf5 f64 read comparison excluded — clawhdf5's f32 datatype encoding differs from libhdf5's (known compatibility gap). f64 results are clawhdf5-only.
|
||
- Serial benchmarks. clawhdf5 uses Rayon for chunk compression when > 2 chunks; that parallelism is already reflected in the chunked write numbers.
|
||
- clawhdf5 reads from `Vec<u8>` (zero-copy from mmap in production); libhdf5 reads from a temp file. This gives clawhdf5 a structural read advantage that reflects realistic API usage.
|
||
|
||
---
|
||
|
||
## Independent Validation: tank (Ryzen 7 7800X3D), 2026-08-03
|
||
|
||
The `vs libhdf5 Summary` numbers above were re-run on a second, independently
|
||
administered machine (`tank`: AMD Ryzen 7 7800X3D, 8C/16T, Ubuntu 26.04, libhdf5
|
||
1.14.6 via `apt`) to confirm they reproduce off the original i7-12650H box, and to
|
||
add benchmark coverage for two claims that a documentation review found were not
|
||
traceable to any dated benchmark run (see git history around 2026-08-03 for context).
|
||
This section documents both.
|
||
|
||
### Reproduction of the vs-libhdf5 Summary table
|
||
|
||
| Workload | clawhdf5 (tank) | libhdf5 (tank) | Speedup (tank) | Speedup (i7-12650H, above) |
|
||
|----------|-----------------|-----------------|----------------|------------------------------|
|
||
| Sequential read, 1K f32 | 553 ns | 44.2 µs | **79.9×** | 71× |
|
||
| Sequential read, 100K f32 | 23.3 µs | 63.6 µs | **2.7×** | 3.0× |
|
||
| Sequential write, 100K f32 | 210 µs | 189 µs | **≈ tie** (clawhdf5 ~11% behind) | ≈ tie (clawhdf5 ~7% behind) |
|
||
| Chunked write deflate-6, 512×512 | 1.44 ms | 65.0 ms | **45.3×** | 38.4× |
|
||
| Attribute write, 128 attrs | 85.2 µs | 877 µs | **10.3×** | 10.6× |
|
||
| Group create, 64 groups | 130 µs | 1.37 ms | **10.6×** | 11.1× |
|
||
|
||
Five of six rows land within ~15% of the original i7-12650H figures — consistent
|
||
with normal cross-machine variance, not a methodology artifact. The chunked-write
|
||
row moved further (38.4× → 45.3×, +18%): tank's libhdf5 per-chunk write cost scales
|
||
worse relative to its own sequential-write throughput than on the i7, likely IPC/
|
||
memory-subsystem dependent. Both figures are real and dated; we report both rather
|
||
than picking one.
|
||
|
||
### New coverage: replacing the retracted "metadata parse / 308×" and "zero-copy mmap / 313 ns" claims
|
||
|
||
An earlier README revision cited `19 ns` vs `2,080 µs` (labeled, incorrectly, `308×`)
|
||
for "metadata parse," and `313 ns` for "zero-copy mmap" — neither figure traced to
|
||
any benchmark in this file. Both have been retracted from the README. In their
|
||
place, two new Criterion benchmarks were added
|
||
(`crates/clawhdf5-bench/benches/h5bench_meta.rs`,
|
||
`crates/clawhdf5-bench/benches/h5bench_read.rs`) and run on tank:
|
||
|
||
**`metadata_open_from_disk`** — opens a small file from disk (`std::fs::read` /
|
||
`hdf5::File::open`) and resolves one attribute. Both sides pay real OS I/O, unlike
|
||
the retracted claim.
|
||
|
||
| Operation | clawhdf5 | libhdf5 | Speedup |
|
||
|-----------|----------|---------|---------|
|
||
| Open file + read 1 attribute | 4.01 µs | 39.3 µs | **9.8×** |
|
||
|
||
**`metadata_parse_in_memory`** (clawhdf5-only) — times `File::from_bytes()` alone,
|
||
given bytes already resident in memory, i.e. header-parse cost with disk I/O
|
||
excluded. There is no fair libhdf5-side equivalent (its API has no "parse from an
|
||
in-memory buffer, skip the OS open" path), so this is reported standalone rather
|
||
than as a speedup multiple — this is the honest version of what the old `19 ns`
|
||
number was trying to claim.
|
||
|
||
| Operation | clawhdf5 (in-memory, no I/O) |
|
||
|-----------|------------------------------|
|
||
| Parse superblock + resolve 1 attribute | 549 ns |
|
||
|
||
**`read_zerocopy_mmap`** — opens via `MmapFile` and reads an f64 dataset through
|
||
`read_f64_zerocopy()`, summing every element to force the mapped pages to actually
|
||
fault in (returning only a slice length, as an earlier draft of this benchmark did,
|
||
would repeat the exact "measures nothing" mistake being fixed here).
|
||
|
||
| n (f64 elements) | clawhdf5 mmap (zerocopy, page-fault-forced) | clawhdf5 (`Vec<u8>` copy) | libhdf5 (disk open + copy) |
|
||
|-------------------|----------------------------------------------|----------------------------|------------------------------|
|
||
| 1,000 | 7.86 µs | 4.50 µs | 44.2 µs |
|
||
| 10,000 | 19.0 µs | 9.53 µs | 47.1 µs |
|
||
| 100,000 | 112 µs | 72.0 µs | 81.2 µs |
|
||
|
||
Honest result: at these sizes, forcing full materialization through the mmap path
|
||
is **not** faster than the plain `Vec<u8>` copy path — `mmap()`/page-fault overhead
|
||
per call outweighs the copy it avoids. This contradicts the retracted `313 ns`
|
||
claim outright and is a genuinely useful finding: `MmapFile`'s real advantage is
|
||
avoiding the allocation/copy for large files or sparse access patterns (lower peak
|
||
RSS, share pages across processes), not raw single-shot read latency at these
|
||
sizes. No README claim is made from this row; it's recorded here for the record
|
||
and to keep future readers from reintroducing the old number.
|
||
|
||
**Reproduce:**
|
||
|
||
```bash
|
||
cargo bench -p clawhdf5-bench --features libhdf5-compare --bench h5bench_meta -- metadata_open_from_disk
|
||
cargo bench -p clawhdf5-bench --features libhdf5-compare --bench h5bench_meta -- metadata_parse_in_memory
|
||
cargo bench -p clawhdf5-bench --features libhdf5-compare --bench h5bench_read -- read_zerocopy_mmap
|
||
```
|
||
|
||
## Independent Validation: tank — LongMemEval & Vector Search (Ryzen 7 7800X3D), 2026-08-05
|
||
|
||
Re-running the "LongMemEval Results" and "SIMD & Parallelism" sections above on
|
||
tank (AMD Ryzen 7 7800X3D, 8C/16T, Ubuntu 26.04, same machine as the
|
||
vs-libhdf5 validation above) to give both sections the dated, hardware-cited,
|
||
reproducible citation the top-of-file traceability note flags them as
|
||
missing.
|
||
|
||
> The sections this re-run checked were themselves re-measured on tank on
|
||
> 2026-09-24 and now show those figures; the i7-12650H figures this section
|
||
> compares against ("above", "at the top of this file") are quoted in those
|
||
> sections' notes. Kept as the 2026-08-05 record.
|
||
|
||
### LongMemEval Results (reproduction)
|
||
|
||
```bash
|
||
cd benchmarks/longmemeval
|
||
wget https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_oracle.json
|
||
cargo run --release --bin longmemeval_bench
|
||
```
|
||
|
||
Recall numbers are deterministic (pure BM25 retrieval over a fixed dataset) and
|
||
reproduce exactly. Scoring target as declared in the LongMemEval section above:
|
||
retrieval recall, turn-level, k=10, `longmemeval_oracle` variant, BM25-only.
|
||
|
||
| Metric | Turn-Level |
|
||
|--------|------------|
|
||
| Hit@1 | 52.6% |
|
||
| Hit@5 | **84.4%** |
|
||
| Hit@10 | 90.4% |
|
||
| MRR | 0.6597 |
|
||
|
||
Session-level figures are omitted here — they are degenerate on the oracle variant
|
||
and have been retracted; see "Retracted: session-level recall and the MemX
|
||
comparison" above.
|
||
|
||
Search latency (hardware-dependent, tank numbers):
|
||
|
||
| Metric | avg | p50 | p95 | p99 |
|
||
|--------|-----|-----|-----|-----|
|
||
| Latency | 2,431 µs | 2,105 µs | 7,250 µs | 12,018 µs |
|
||
|
||
Higher than the i7-12650H figures at the top of this file (avg 1,004 µs) despite
|
||
tank's faster single-core performance elsewhere in this document — BM25 search
|
||
latency here scales with per-question haystack size and this run's variance is
|
||
wider (p99 is ~5x the mean), suggesting this metric is more sensitive to
|
||
momentary scheduling/cache effects than the flat-array vector-search benchmarks.
|
||
Recorded as-is rather than smoothed.
|
||
|
||
### SIMD & Parallelism (reproduction, with a correction)
|
||
|
||
```bash
|
||
cargo bench -p clawhdf5-agent --bench bench -- "^(strategy_scalar_10k|strategy_simd_10k|strategy_rayon_10k|adaptive_search_10k|simd_cosine_100k|rayon_cosine_100k)$"
|
||
```
|
||
|
||
The original 10K table above compares named benchmarks (`vector_search`,
|
||
`rayon`, `strategy`) that, on inspection, don't all exercise the same
|
||
scalar-vs-SIMD-vs-parallel axis the table implies — several of the
|
||
`simd_cosine_10k`/`sequential_cosine_10k`-style benchmarks actually call the
|
||
same underlying function under different names. The `adaptive_benches` group's
|
||
`strategy_scalar_10k` / `strategy_simd_10k` / `strategy_rayon_10k` benchmarks
|
||
are the ones that genuinely hold the dataset fixed and vary only the
|
||
`SearchStrategy` enum, so they're the correct apples-to-apples comparison —
|
||
used here instead.
|
||
|
||
| Strategy | Latency (tank) | vs Sequential |
|
||
|----------|-----------------|----------------|
|
||
| Sequential (scalar) | 502 µs | 1.0x |
|
||
| SIMD (auto-vectorized) | 327 µs | **1.53x** |
|
||
| Rayon (parallel) | 323 µs | **1.55x** |
|
||
| Adaptive (auto-select) | 339 µs | **1.48x** |
|
||
|
||
Honest finding: the speedup from SIMD/parallelism over scalar is real but
|
||
smaller here (~1.5x) than the i7-12650H figures above (~2.0x). The Ryzen 7
|
||
7800X3D's large L3 cache (96MB 3D V-Cache) measurably narrows the gap versus a
|
||
naive scalar loop compared to the i7 — this is a genuine hardware-dependent
|
||
result, not a regression or measurement error, and is recorded rather than
|
||
reconciled away.
|
||
|
||
At 100K, no `strategy_*` benchmark exists in the current suite (`adaptive_benches`
|
||
only covers n=10,000), so this row uses the same `simd_cosine_100k`/
|
||
`rayon_cosine_100k` benchmarks as the original table — not a true scalar
|
||
baseline, so no "vs Sequential" multiple is reported for it:
|
||
|
||
| Strategy | Latency (tank) |
|
||
|----------|-----------------|
|
||
| SIMD | 6.60 ms |
|
||
| Rayon parallel | 4.73 ms |
|
||
|
||
### Vector Search Latency & Comparison to MemX (reproduction)
|
||
|
||
```bash
|
||
cargo bench -p clawhdf5-agent --bench bench -- "^(vector_search_1k|simd_cosine_10k|simd_cosine_100k|prenorm_search_10k|ivf_search_10k_nprobe10|ivf_search_100k_nprobe10|ivf_pq_search_100k|rairs_search_10k_nprobe10|bm25_search_10k)$"
|
||
```
|
||
|
||
| Scale | Flat Search | Pre-norm | IVF (nprobe=10) | IVF-PQ | RAIRS |
|
||
|-------|-------------|----------|-----------------|--------|-------|
|
||
| **1K** | 47.8 µs | — | — | — | — |
|
||
| **10K** | 501 µs | 322 µs | 24.8 µs | — | 109 µs |
|
||
| **100K** | 6.60 ms | — | 608 µs | 865 µs | — |
|
||
|
||
(The 1K Pre-norm cell from the original table has no corresponding benchmark
|
||
in the current suite — not re-verified, left blank rather than guessed.)
|
||
|
||
Same not-like-for-like caveat as the "Comparison to MemX" section at the top of this
|
||
file applies — MemX's figure is end-to-end, these are a single component. Ratios are
|
||
an order-of-magnitude indication, not a benchmark result.
|
||
|
||
| Metric | MemX (claimed, end-to-end) | ClawhDF5 (tank, component only) | Ratio |
|
||
|--------|----------------------------|----------------------------------|-------|
|
||
| 100K flat search | <90 ms | 6.60 ms | ~14x |
|
||
| 100K IVF-PQ search | — | 865 µs | ~104x |
|
||
| Keyword search 10K | 1,100x improvement over unindexed | 520 µs (BM25) | Comparable |
|
||
|
||
Every figure in this subsection is faster than the corresponding i7-12650H
|
||
number at the top of this file, consistent with the Ryzen 7 7800X3D's higher
|
||
single-core throughput and larger cache observed in the vs-libhdf5 validation
|
||
above.
|