bench: re-run every stale BENCHMARKS.md section, dated and traced
Every undated or pre-September section re-run on one machine on one day
(tank, AMD Ryzen 7 7800X3D, 2026-09-24, commit 5c8323c), 24 commands run
serially with the load average checked before each, with the command
recorded for each section. A separate check traced every changed number
back to the raw output; its corrections are applied (e.g. the on-disk
~820 B/record is float16 plus always-deflated text on a synthetic corpus
of 40 distinct texts, not float16 alone).
Two apparent regressions were isolated rather than published:
- knowledge-graph traversal: a real bug, fixed in the previous commit;
- the write path: v2.3.0 built and run on the same machine measures the
same as today, so the old 18 us / 6.17 ms figures (undated, other
hardware) are not reproducible; float16 adds ~2 us per save and the
int8 index nothing (both isolated by switching the bench's config).
Also:
- new multimodal_bench: cross-modal search at 1K/10K records, which the
README claimed but nothing measured;
- footprint_bench reports whether it built float16 or f32 stores and
takes --f32 (it kept printing "f32" after the default changed);
- README: performance tables, the "Why" table figures and the SQLite
migration section (from the previous migrate commit);
- CHANGELOG for this branch.
Not re-run: consolidation_efficiency's 100K row and its memory-reduction
part (stopped for time), and cross_platform.sh.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
+614
-139
@@ -2,20 +2,40 @@
|
||||
|
||||
> Pure Rust. Zero C dependencies. Single file. Fast enough to forget it's there.
|
||||
|
||||
**System:** Intel i7-12650H (10C/16T, 4.7 GHz boost) · 32 GB DDR5 · Linux 6.8.0
|
||||
**System (original run):** Intel i7-12650H (10C/16T, 4.7 GHz boost) · 32 GB DDR5 · Linux 6.8.0
|
||||
**Rust:** 1.96.0-nightly (2026-03-14) · `--release` profile
|
||||
**Date:** 2026-07-01
|
||||
|
||||
> **Traceability note:** the "h5bench-Equivalent I/O Benchmarks" and both
|
||||
> "Independent Validation: tank" sections below meet a dated,
|
||||
> hardware-cited, reproducible standard (explicit date, machine spec, and a
|
||||
> runnable command per result) — this now covers "LongMemEval Results",
|
||||
> "SIMD & Parallelism", "Vector Search Latency", and "Comparison to MemX" via
|
||||
> their tank re-runs. The remaining undated sections above (Hybrid Search,
|
||||
> Knowledge Graph, Memory Consolidation, Temporal Index, Write Path, Decision
|
||||
> Gate, Memory Strategy, Multi-Session Benchmark, Memory Footprint,
|
||||
> Consolidation Efficiency, Ephemeral Tier) do not yet meet that bar — this is
|
||||
> a known, tracked documentation gap, not a claim that those numbers are wrong.
|
||||
Sections that carry their own "Measured …" line were measured there instead,
|
||||
not on the system above. The 2026-09-24 re-run was on tank (AMD Ryzen 7
|
||||
7800X3D, 8C/16T, Linux 7.0.0-34-generic, rustc 1.98.1) at commit 5c8323c,
|
||||
serially, with no `--warm-up-time`/`--measurement-time` overrides, waiting
|
||||
before each command while the 1-minute load average was above 2.0. The
|
||||
`bench` targets ran at Criterion's defaults (3 s warm-up, 5 s target
|
||||
measurement time, 100 samples). Every `memory_bench` group that was run
|
||||
(`vector_search_latency`, `hybrid_search`, `knowledge_graph`,
|
||||
`consolidation`, `temporal`) and the multi-modal group set
|
||||
`sample_size(50)`, so those sections took 50 samples, not 100. The 5 s is a
|
||||
target: Criterion stretched it where 5 s could not hold the samples it needed
|
||||
(for example 8.5 s for `vector_search_latency/bench_cosine_search/100k`,
|
||||
15.5 s for the 1K consolidation cycle, 252 s for `tick_session_10k`).
|
||||
|
||||
> **Traceability note:** a section meets the dated, hardware-cited,
|
||||
> reproducible standard when it gives an explicit date, the machine, and a
|
||||
> runnable command for its results. As of the 2026-09-24 re-run on tank that
|
||||
> covers Vector Search Latency, Comparison to MemX, SIMD & Parallelism, Hybrid
|
||||
> Search, Knowledge Graph, Memory Consolidation, Temporal Index, Write Path,
|
||||
> Decision Gate, Memory Strategy, Summary (derived from those), LongMemEval
|
||||
> Results (the BM25 rows), Multi-Session Benchmark, Memory Footprint (on disk
|
||||
> and in memory), Consolidation Efficiency, Ephemeral Tier, Multi-modal Search,
|
||||
> the Search and Read harnesses, and the "h5bench-Equivalent I/O Benchmarks"
|
||||
> and "Independent Validation: tank" sections. What does not yet meet that bar:
|
||||
> the LongMemEval rows that need real embeddings (not re-run here, except the
|
||||
> dated float16 comparison), the Consolidation Efficiency 100K cycle row and
|
||||
> memory-reduction part (the 2026-09-24 run was stopped before it produced
|
||||
> them), the int8 side of "Quantising the index copy" (not re-run), and the
|
||||
> i7-12650H and macOS M3 Max rows under Cross-Platform Notes. That is a
|
||||
> known, tracked documentation gap, not a claim that those numbers are wrong.
|
||||
>
|
||||
> **Correctness note (2026-08-06).** Being dated and reproducible is necessary but
|
||||
> not sufficient — a number can be perfectly reproducible and still measure the
|
||||
@@ -57,6 +77,23 @@ What remains at 2.43x: the flat vectors (1.0x), the HNSW index's own copy of
|
||||
them (1.0x), and text, ids and graph (~0.4x). The index copy is the next
|
||||
target — it is what a quantised or borrowed representation would address.
|
||||
|
||||
**Current figures (`f32` index).** Measured 2026-09-24 on tank (AMD Ryzen 7
|
||||
7800X3D), commit 5c8323c:
|
||||
|
||||
> **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --footprint --full`
|
||||
|
||||
| N | vectors (raw) | entries MiB | store MiB | indexes MiB | reopened MiB | peak during open MiB | reopened / raw |
|
||||
|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 1 000 | 1 | 2 | 0 | 2 | 4 | 5 | 2.40x |
|
||||
| 10 000 | 15 | 17 | 12 | 32 | 44 | 61 | 3.03x |
|
||||
| 100 000 | 146 | 172 | 60 | 266 | 399 | 562 | 2.72x |
|
||||
|
||||
These are the "reopened, f32" figures in the next table, reproduced exactly.
|
||||
They are above the "after" column of the table above (44 MiB vs 35 MiB at 10K,
|
||||
399 vs 357 MiB at 100K): the 2.43x was measured for 2e7e045, and the store
|
||||
already measured 399 MiB (2.72x) when the int8 index landed later the same
|
||||
day (c0a9206). The git log does not say what changed in between.
|
||||
|
||||
### Quantising the index copy (`quantized_index`)
|
||||
|
||||
`MemoryConfig::quantized_index` stores the index's copy as `i8` instead of
|
||||
@@ -108,6 +145,20 @@ second**, builds **1.8x faster**, and holds a quarter of the vectors. (Compare
|
||||
only at equal `ef`: with re-scoring the harness raises `ef` to at least the
|
||||
candidate pool, so the `ef = 16` and `ef = 32` rows are not like-for-like.)
|
||||
|
||||
**Re-check, 2026-09-24.** Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D),
|
||||
commit 5c8323c. The `f32` memory columns of the first table reproduce exactly
|
||||
(see [Memory footprint](#memory-footprint)). Only the `f32` side of the speed
|
||||
table was re-run, once, not as alternating medians:
|
||||
|
||||
> **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --full`
|
||||
|
||||
At N = 100 000, `ef = 64`, the `f32` index measured recall@10 0.9945 (same),
|
||||
19 001 QPS (was 13 399) and a 2700.8 ms build (was 3197 ms). The QPS figure
|
||||
moved by more than 20%; the log does not explain it, and a single run is not
|
||||
the same-moment paired comparison the table above is. The int8 columns were
|
||||
not re-run, so the 1.63x ratio has not been re-checked against the new `f32`
|
||||
figure.
|
||||
|
||||
#### On ARM (Raspberry Pi 5, Cortex-A76)
|
||||
|
||||
`dot_i8` has two aarch64 kernels: `SDOT` for CPUs with the ARMv8.2
|
||||
@@ -268,6 +319,10 @@ high-water mark alongside the retained figure:
|
||||
| 10 000 | 44 | 61 |
|
||||
| 100 000 | 399 | 562 |
|
||||
|
||||
Reproduced exactly on 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit
|
||||
5c8323c, by `search_harness -- --footprint --full` (table under
|
||||
[Memory footprint](#memory-footprint)).
|
||||
|
||||
The peak is set *after* the parse, by the index build, so a buffer allocated
|
||||
and freed during the parse never reaches the high-water mark. Holding a
|
||||
deliberate extra copy of the file across the whole parse leaves the peak
|
||||
@@ -359,6 +414,50 @@ contiguous dataset is converted straight from the file bytes (one copy instead
|
||||
of two), and the native-endian conversions no longer zero a buffer they are
|
||||
about to overwrite.
|
||||
|
||||
### Current: read harness (2026-09-24)
|
||||
|
||||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||||
|
||||
> **Run:** `cargo run --release -p clawhdf5-bench --bin read_harness`
|
||||
|
||||
4096 x 2048 f64 (64 MB per dataset), chunks 256 x 256, file 129 MB
|
||||
|
||||
| layout | read | selected | time ms | MB/s of selection | vs full read |
|
||||
|---|---|---:|---:|---:|---:|
|
||||
| chunked + deflate | full (first) | 64 MB | 65.6 | 975 | |
|
||||
| chunked + deflate | full (repeat) | 64 MB | 63.2 | 1013 | 1.00x |
|
||||
| chunked + deflate | 64 x 64 window (1 chunk) | 0.03 MB | 0.18 | 177 | 0.003x |
|
||||
| chunked + deflate | 512 x 512 window (4-9 chunks) | 2.00 MB | 4.12 | 485 | 0.063x |
|
||||
| chunked + deflate | one row | 0.02 MB | 0.99 | 16 | 0.015x |
|
||||
| chunked + deflate | one column | 0.03 MB | 2.02 | 15 | 0.031x |
|
||||
| chunked | full (first) | 64 MB | 60.5 | 1057 | |
|
||||
| chunked | full (repeat) | 64 MB | 56.7 | 1129 | 1.00x |
|
||||
| chunked | 64 x 64 window (1 chunk) | 0.03 MB | 0.10 | 325 | 0.002x |
|
||||
| chunked | 512 x 512 window (4-9 chunks) | 2.00 MB | 3.19 | 628 | 0.053x |
|
||||
| chunked | one row | 0.02 MB | 0.05 | 284 | 0.001x |
|
||||
| chunked | one column | 0.03 MB | 0.49 | 64 | 0.008x |
|
||||
| contiguous | full (first) | 64 MB | 30.1 | 2124 | |
|
||||
| contiguous | full (repeat) | 64 MB | 26.1 | 2452 | 1.00x |
|
||||
| contiguous | 64 x 64 window (1 chunk) | 0.03 MB | 0.08 | 394 | 0.003x |
|
||||
| contiguous | 512 x 512 window (4-9 chunks) | 2.00 MB | 3.36 | 595 | 0.112x |
|
||||
| contiguous | one row | 0.02 MB | 0.03 | 486 | 0.001x |
|
||||
| contiguous | one column | 0.03 MB | 2.61 | 12 | 0.087x |
|
||||
|
||||
Full reads match the "after" column above (chunked + deflate 69 ms then, 63.2
|
||||
to 65.6 ms now). The compressed selections are faster than in "After: partial
|
||||
reads" (64 x 64 window 0.39 -> 0.18 ms, one row 2.69 -> 0.99 ms, one column
|
||||
5.23 -> 2.02 ms) and match the 2026-09-23 figures in
|
||||
[Deflate backend](#deflate-backend-zlib-rs-vs-zlib-ng) (0.18, 1.00 and 2.01
|
||||
ms), which were taken after the zlib-rs switch and the one-shot codec calls
|
||||
described there; this run does not isolate which change accounts for it.
|
||||
|
||||
The uncompressed windows went the other way. Against "After: partial reads",
|
||||
the 512 x 512 window is about 60% slower (chunked 1.99 -> 3.19 ms, contiguous
|
||||
2.12 -> 3.36 ms) and the chunked 64 x 64 window went from 0.06 to 0.10 ms.
|
||||
The rows and columns of the uncompressed layouts are within 20% (chunked
|
||||
column 0.45 -> 0.49 ms, contiguous column 2.55 -> 2.61 ms). This run does not
|
||||
explain the slower windows.
|
||||
|
||||
## Search harness baseline (v2.3.0)
|
||||
|
||||
Produced by `cargo run --release -p clawhdf5-bench --bin search_harness -- --full`
|
||||
@@ -596,68 +695,217 @@ Recall on clustered data is unchanged or slightly better (100K, `ef = 64`:
|
||||
0.474 -> 0.444), the cost of batch members not seeing each other while
|
||||
planning; batches are capped at 1/16 of the graph and 512 nodes.
|
||||
|
||||
### Current: search harness (2026-09-24)
|
||||
|
||||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, default
|
||||
features (so the build is batched and `parallel`, 16 threads). The HNSW tables
|
||||
use an `f32` index. The end-to-end table uses `MemoryConfig::new`'s defaults,
|
||||
which at this commit are `float16` embeddings and the int8 index; every
|
||||
earlier end-to-end table in this section was an `f32` store with an `f32`
|
||||
index.
|
||||
|
||||
> **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --full`
|
||||
|
||||
#### HNSW, N = 1000, dim = 384, M = 16, ef_construction = 64, storage = Float32
|
||||
|
||||
build: 14.7 ms (67954 vectors/s) · exact scan: 4403 QPS, p50 225 µs
|
||||
|
||||
| ef | recall@10 | QPS | p50 µs | p99 µs |
|
||||
|---:|---:|---:|---:|---:|
|
||||
| 16 | 0.9940 | 134065 | 7 | 18 |
|
||||
| 32 | 1.0000 | 105134 | 9 | 21 |
|
||||
| 64 | 1.0000 | 77541 | 13 | 26 |
|
||||
| 128 | 1.0000 | 36938 | 26 | 35 |
|
||||
| 256 | 1.0000 | 19249 | 52 | 63 |
|
||||
|
||||
#### HNSW, N = 10000, dim = 384, M = 16, ef_construction = 64, storage = Float32
|
||||
|
||||
build: 135.6 ms (73736 vectors/s) · exact scan: 435 QPS, p50 2295 µs
|
||||
|
||||
| ef | recall@10 | QPS | p50 µs | p99 µs |
|
||||
|---:|---:|---:|---:|---:|
|
||||
| 16 | 0.9975 | 79614 | 12 | 24 |
|
||||
| 32 | 1.0000 | 70556 | 14 | 21 |
|
||||
| 64 | 1.0000 | 51313 | 19 | 29 |
|
||||
| 128 | 1.0000 | 22846 | 44 | 57 |
|
||||
| 256 | 1.0000 | 10267 | 97 | 119 |
|
||||
|
||||
#### HNSW, N = 100000, dim = 384, M = 16, ef_construction = 64, storage = Float32
|
||||
|
||||
build: 2700.8 ms (37027 vectors/s) · exact scan: 43 QPS, p50 23433 µs
|
||||
|
||||
| ef | recall@10 | QPS | p50 µs | p99 µs |
|
||||
|---:|---:|---:|---:|---:|
|
||||
| 16 | 0.9180 | 20043 | 47 | 104 |
|
||||
| 32 | 0.9775 | 21348 | 44 | 104 |
|
||||
| 64 | 0.9945 | 19001 | 51 | 101 |
|
||||
| 128 | 0.9995 | 13458 | 73 | 137 |
|
||||
| 256 | 0.9995 | 5128 | 196 | 282 |
|
||||
|
||||
#### End to end: `HDF5Memory::hybrid_search` (k = 10, weights 0.7 / 0.3)
|
||||
|
||||
| N | ingest ms | cold index build ms | checkpoint ms | open ms | first query after open ms | p50 ms | p99 ms | QPS |
|
||||
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 1000 | 15 | 14 | 4.4 | 1.9 | 1.5 | 0.07 | 0.08 | 13869.5 |
|
||||
| 10000 | 146 | 126 | 33.5 | 18.3 | 14.8 | 0.49 | 0.50 | 2044.1 |
|
||||
| 100000 | 1498 | 1833 | 495.7 | 259.9 | 155.1 | 4.69 | 5.22 | 212.7 |
|
||||
|
||||
Steady-state query latency is where the v2.4.0 work left it (p50 0.07 / 0.49 /
|
||||
4.69 ms against 0.07 / 0.49 / 4.65 ms). Figures that moved by more than 20%:
|
||||
|
||||
- **Build.** 14.7 / 135.6 / 2700.8 ms against 34 / 388 / 5.9 s for
|
||||
"batched + `parallel`" above. The git log does not explain the difference.
|
||||
- **Cold index build** in the end-to-end table: 14 / 126 / 1833 ms, down from
|
||||
112 / 1487 / 20285 ms in the "unranked keyword scores" table, which predates
|
||||
the batched and parallel build described above.
|
||||
- **QPS at 100K, `ef = 64`**: 19 001 against 13 980 in the "unit-vector"
|
||||
table, with recall 0.9945 against 0.984; the recall gain is the one the
|
||||
batched build reported. The QPS gain is not explained by the log.
|
||||
- **Other HNSW QPS cells**, against the same "unit-vector" tables: 10K
|
||||
`ef = 16` 79 614 against 54 608 (+46%); 100K `ef = 16` 20 043 against
|
||||
15 139 (+32%), `ef = 128` 13 458 against 10 959 (+23%), `ef = 256` 5 128
|
||||
against 3 731 (+37%). The other cells, and every 1K cell, are within 20%.
|
||||
The log does not explain these either.
|
||||
- **Open** in the end-to-end table: 1.9 / 18.3 / 259.9 ms against 1.1 / 13.7
|
||||
/ 353.1 ms in the "unranked keyword scores" table, slower at 1K and 10K and
|
||||
faster at 100K. This run does not isolate the cause.
|
||||
- **Checkpoint at 100K**: 495.7 ms against 728.9 ms, and ingest up 11 -> 15,
|
||||
104 -> 146 ms at 1K / 10K. This store is `float16` with an int8 index, the
|
||||
earlier ones `f32`; the [float16 study](#float16-embedding-storage-memoryconfigfloat16)
|
||||
measured float16 checkpoints at 512 ms against 752 ms for `f32` at 100K.
|
||||
This run does not isolate the ingest change.
|
||||
|
||||
## Vector Search Latency
|
||||
|
||||
Brute-force cosine similarity over 384-dimensional embeddings (OpenAI text-embedding-3-small size).
|
||||
|
||||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||||
|
||||
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(vector_search_1k|simd_cosine_10k|simd_cosine_100k|prenorm_search_10k|ivf_search_10k_nprobe10|ivf_search_100k_nprobe10|ivf_pq_search_100k|rairs_search_10k_nprobe10)$'`
|
||||
|
||||
| Scale | Flat Search | Pre-norm | IVF (nprobe=10) | IVF-PQ | RAIRS |
|
||||
|-------|-------------|----------|-----------------|--------|-------|
|
||||
| **1K** | 54 µs | 62 µs | — | — | — |
|
||||
| **10K** | 753 µs | 706 µs | 27 µs | — | 159 µs |
|
||||
| **100K** | 11.4 ms | — | 1.32 ms | 1.19 ms | — |
|
||||
| **1K** | 47.4 µs | — | — | — | — |
|
||||
| **10K** | 500.5 µs | 316.5 µs | 24.8 µs | — | 124.0 µs |
|
||||
| **100K** | 6.58 ms | — | 592 µs | 869 µs | — |
|
||||
|
||||
**Key insight:** At 10K records (typical agent memory), IVF search delivers **27 µs** — that's 26x faster than flat search. Even at 100K records, IVF-PQ keeps search under **1.2 ms**.
|
||||
The 1K pre-norm cell is blank because no 1K pre-norm benchmark exists (the
|
||||
old 62 µs has no source). `memory_bench`'s own cosine search
|
||||
(`cargo bench -p clawhdf5-agent --bench memory_bench -- '^vector_search_latency/'`)
|
||||
agrees: 47.6 µs / 505.1 µs / 6.67 ms at 1K / 10K / 100K.
|
||||
|
||||
Most cells moved by more than 20% from the previous table (10K flat 753 µs,
|
||||
pre-norm 706 µs, RAIRS 159 µs; 100K flat 11.4 ms, IVF 1.32 ms, IVF-PQ 1.19 ms).
|
||||
Those were undated figures from the original i7-12650H run. The 2026-08-05
|
||||
tank run below is within 4% of the new numbers in every cell but one (1K flat
|
||||
47.8 µs, 10K flat 501 µs, pre-norm 322 µs, IVF 24.8 µs; 100K flat 6.60 ms,
|
||||
IVF 608 µs, IVF-PQ 865 µs), so for those cells the difference is the machine.
|
||||
The exception is RAIRS: 109 µs on 2026-08-05, 124.0 µs now (+14%); this run
|
||||
does not explain that change.
|
||||
|
||||
**Key insight:** at 10K records (typical agent memory), IVF search takes
|
||||
**24.8 µs** against 500.5 µs for the flat scan. At 100K, IVF answers in
|
||||
**592 µs** and IVF-PQ in 869 µs — both under 1 ms. (RAIRS, 124.0 µs at 10K, is
|
||||
slower than plain IVF here.)
|
||||
|
||||
### Comparison to MemX (arxiv:2603.16171)
|
||||
|
||||
MemX claims end-to-end search under 90ms at 100K records (Rust + libSQL + FTS5).
|
||||
|
||||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||||
|
||||
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(simd_cosine_100k|ivf_pq_search_100k|bm25_search_10k)$'`
|
||||
|
||||
> **Caveat — not like-for-like.** MemX's `<90 ms` is *end-to-end* search across their
|
||||
> full pipeline (dense embeddings + FTS5 + four-factor re-ranking). The clawhdf5
|
||||
> figures below are a *single component* — raw vector search latency, excluding
|
||||
> embedding, keyword, fusion, and re-ranking stages. A component measured against a
|
||||
> full pipeline will always look favourable; the "speedup" column overstates the real
|
||||
> advantage by an unquantified margin and should be read as an order-of-magnitude
|
||||
> indication only, not a benchmark result. Matching MemX's measurement boundary is
|
||||
> tracked as follow-up work.
|
||||
> full pipeline will always look favourable, so the two columns below cannot be
|
||||
> divided into a speedup: any such ratio would overstate the real advantage by an
|
||||
> unquantified margin. Read the table as an order-of-magnitude indication only, not a
|
||||
> benchmark result. Matching MemX's measurement boundary is tracked as follow-up work.
|
||||
|
||||
| Metric | MemX (claimed, end-to-end) | ClawhDF5 (component only) | Ratio |
|
||||
|--------|----------------------------|---------------------------|-------|
|
||||
| 100K flat search | <90 ms | 11.4 ms | ~8x |
|
||||
| 100K IVF-PQ search | — | 1.19 ms | ~76x |
|
||||
| Keyword search 10K | 1,100x improvement over unindexed | 583 µs (BM25) | Comparable |
|
||||
| Metric | MemX (claimed, end-to-end) | ClawhDF5 (component only) |
|
||||
|--------|----------------------------|---------------------------|
|
||||
| 100K flat search | <90 ms | 6.64 ms |
|
||||
| 100K IVF-PQ search | — | 876 µs |
|
||||
| Keyword search 10K | 1,100x improvement over unindexed | 205 µs (BM25) |
|
||||
|
||||
The ratio column is gone: dividing an end-to-end claim by a component
|
||||
timing produced a number that looked like a result and was not one. The
|
||||
previous figures (100K flat 11.4 ms, IVF-PQ 1.19 ms, BM25 10K 583 µs) were
|
||||
undated, from the original i7-12650H run. BM25 at 10K also moved on this
|
||||
machine: 520 µs on 2026-08-05, 205 µs now; v2.4.0 changed
|
||||
`BM25Index::search` in between (bounded-heap top-k, the unused WAND bound
|
||||
removed, IDF per query), but this run does not isolate its effect.
|
||||
|
||||
---
|
||||
|
||||
## SIMD & Parallelism
|
||||
|
||||
384-dimensional cosine similarity at 10K scale.
|
||||
384-dimensional cosine similarity at 10K scale, using the `strategy_*`
|
||||
benchmarks, which hold the dataset fixed and vary only `SearchStrategy`
|
||||
(the 2026-08-05 re-run below explains why the older table's benchmarks were
|
||||
not an apples-to-apples comparison).
|
||||
|
||||
| Strategy | Latency | vs Sequential |
|
||||
|----------|---------|---------------|
|
||||
| Sequential (scalar) | 1.07 ms | 1.0x |
|
||||
| SIMD (auto-vectorized) | 545 µs | **2.0x** |
|
||||
| Rayon (parallel) | 553 µs | **1.9x** |
|
||||
| Adaptive (auto-select) | 564 µs | **1.9x** |
|
||||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. Rayon
|
||||
uses all 16 threads; the 1-minute load average was 4.19 when this command
|
||||
finished, most likely from the Rayon benchmarks themselves (not verified).
|
||||
|
||||
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(strategy_scalar_10k|strategy_simd_10k|strategy_rayon_10k|adaptive_search_10k|simd_cosine_100k|rayon_cosine_100k)$'`
|
||||
|
||||
At 100K:
|
||||
| Strategy | Latency |
|
||||
|----------|---------|
|
||||
| SIMD | 13.7 ms |
|
||||
| Rayon parallel | 8.3 ms |
|
||||
| Sequential (scalar), `strategy_scalar_10k` | 501.1 µs |
|
||||
| SIMD, `strategy_simd_10k` | 319.9 µs |
|
||||
| Rayon (parallel), `strategy_rayon_10k` | **79.0 µs** |
|
||||
| Adaptive (auto-select), `adaptive_search_10k` | 317.9 µs |
|
||||
|
||||
At 100K (no `strategy_*` benchmark exists at this size, so these are the
|
||||
`simd_cosine_100k` / `rayon_cosine_100k` benchmarks):
|
||||
|
||||
| Strategy | Latency |
|
||||
|----------|---------|
|
||||
| SIMD | 6.56 ms |
|
||||
| Rayon parallel | 4.63 ms |
|
||||
|
||||
The Rayon 10K figure moved furthest: 323 µs on this machine on 2026-08-05,
|
||||
79.0 µs now. The git log does not explain it. The other rows are close to
|
||||
that run (scalar 502 µs, SIMD 327 µs, adaptive 339 µs; 100K 6.60 / 4.73 ms). Against the original, undated i7-12650H table (scalar 1.07 ms, SIMD
|
||||
545 µs, Rayon 553 µs, adaptive 564 µs; 100K 13.7 / 8.3 ms) every figure is
|
||||
lower; that table also used the mislabelled benchmarks. The older names
|
||||
measured in the same session, for reference (`cargo bench -p clawhdf5-agent
|
||||
--bench bench -- '^(sequential_cosine_10k|simd_cosine_10k|rayon_cosine_10k|adaptive_search_10k|simd_cosine_100k|rayon_cosine_100k)$'`):
|
||||
`sequential_cosine_10k` 317.8 µs, `simd_cosine_10k` 505.2 µs,
|
||||
`rayon_cosine_10k` 274.3 µs — "sequential" faster than "SIMD", which is the
|
||||
mislabelling the 2026-08-05 re-run found.
|
||||
|
||||
---
|
||||
|
||||
## Hybrid Search (Vector + BM25)
|
||||
|
||||
1K records, 384-dimensional embeddings with BM25 keyword index.
|
||||
1K records, 384-dimensional embeddings with BM25 keyword index. These are the
|
||||
flat free functions `hybrid::hybrid_search` / `rrf_hybrid_search` (linear
|
||||
scan), not the HNSW-backed `HDF5Memory::hybrid_search`, which is measured in
|
||||
the [search harness](#current-search-harness-2026-09-24).
|
||||
|
||||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||||
|
||||
> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^hybrid_search/'`
|
||||
> and `cargo bench -p clawhdf5-agent --bench bench -- '^(bm25_search_1k|hybrid_search_10k)$'`
|
||||
|
||||
| Method | Latency | Notes |
|
||||
|--------|---------|-------|
|
||||
| Weighted fusion | 198 µs | Original min-max normalization |
|
||||
| **RRF (k=60)** | **222 µs** | Reciprocal Rank Fusion — better quality, ~12% overhead |
|
||||
| BM25-only 1K | 67 µs | Keyword search alone |
|
||||
| Hybrid 10K | 2.04 ms | Full hybrid at 10K scale |
|
||||
| Weighted fusion | 106.8 µs | Min-max normalization |
|
||||
| **RRF (k=60)** | **130.5 µs** | Reciprocal Rank Fusion |
|
||||
| BM25-only 1K | 20.4 µs | Keyword search alone |
|
||||
| Hybrid 10K | 1.10 ms | Full hybrid at 10K scale |
|
||||
|
||||
Every row moved by more than 20% from the previous table (198 µs, 222 µs,
|
||||
67 µs, 2.04 ms), which was undated and from the original i7-12650H run. The
|
||||
cause is not isolated. (The old "better quality" note on RRF is dropped: on
|
||||
LongMemEval RRF measured worse than the tuned weighted sum; see
|
||||
[Fusion method](#fusion-method--weighted-vs-rrf-full-haystack-n500).)
|
||||
|
||||
---
|
||||
|
||||
@@ -665,16 +913,36 @@ At 100K:
|
||||
|
||||
Graph traversal and entity operations.
|
||||
|
||||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||||
|
||||
> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^knowledge_graph/'`
|
||||
> and `cargo bench -p clawhdf5-agent --bench bench -- '^alias_resolve_(short|long)_query$'`
|
||||
> (the `knowledge_graph/` rows re-run after the adjacency-index fix, same day,
|
||||
> load average 0.49 at start)
|
||||
|
||||
| Operation | Scale | Latency |
|
||||
|-----------|-------|---------|
|
||||
| BFS traversal | 100 entities | 5.4 µs |
|
||||
| BFS traversal | 1,000 entities | 24 µs |
|
||||
| Spreading activation | 100 entities | 16.9 µs |
|
||||
| Entity resolution (Levenshtein) | 100 entities | 64 µs |
|
||||
| Alias resolution (short query) | 100 aliases | 10.4 µs |
|
||||
| Alias resolution (long query) | 100 aliases | 11.6 µs |
|
||||
| BFS traversal | 100 entities | 5.23 µs |
|
||||
| BFS traversal | 1,000 entities | 23.1 µs |
|
||||
| Spreading activation | 100 entities | 10.1 µs |
|
||||
| Entity resolution (Levenshtein) | 100 entities | 48.6 µs |
|
||||
| Alias resolution (short query) | 100 aliases | 7.67 µs |
|
||||
| Alias resolution (long query) | 100 aliases | 8.18 µs |
|
||||
|
||||
**All graph operations complete in microseconds.** Spreading activation across 100 entities with 5 propagation steps finishes in 17 µs.
|
||||
The traversal rows were measured after a fix made during this re-run. The
|
||||
first measurement of the day found **BFS 6.5x slower** than the old table
|
||||
said: 17.5 µs and 155.1 µs, against 5.4 µs and 24 µs, with spreading
|
||||
activation at 22.8 µs against 16.9 µs. The cause was 1efd82c (2026-08-17),
|
||||
which built an adjacency index over the whole graph on every traversal — so a
|
||||
2-hop BFS paid to index every entity and relation first. The index is now
|
||||
cached on `KnowledgeCache` and checked against a fingerprint of the graph on
|
||||
each use (one pass over entity ids and relation endpoints, no allocation), so
|
||||
any change to the graph, including direct edits of its public `Vec`s, still
|
||||
triggers a rebuild. These benches traverse an unchanged graph, which is the
|
||||
cached case; the first traversal after a change pays for one build. Entity
|
||||
and alias resolution do not use the index (the 49.8 µs first-run figure for
|
||||
entity resolution is within noise of the 48.6 µs here). The old figures were
|
||||
undated, from the original i7-12650H run.
|
||||
|
||||
---
|
||||
|
||||
@@ -682,13 +950,22 @@ Graph traversal and entity operations.
|
||||
|
||||
Hippocampal-inspired tiered memory management.
|
||||
|
||||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||||
|
||||
> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^consolidation/'`
|
||||
|
||||
| Operation | Scale | Latency |
|
||||
|-----------|-------|---------|
|
||||
| Consolidation cycle | 100 records | 15 µs |
|
||||
| Consolidation cycle | 1,000 records | 164 µs |
|
||||
| Importance scoring | 100 records | 25 µs |
|
||||
| Consolidation cycle | 100 records | 8.08 µs |
|
||||
| Consolidation cycle | 1,000 records | 115.2 µs |
|
||||
| Importance scoring | 100 records | 31.0 µs |
|
||||
|
||||
A full consolidation pass over 1,000 memories (eviction + promotion across Working → Episodic → Semantic) completes in **164 µs**. This can run on every memory write without perceptible latency.
|
||||
A full consolidation pass over 1,000 memories (eviction + promotion across Working → Episodic → Semantic) completes in **115 µs**. This can run on every memory write without perceptible latency.
|
||||
|
||||
The cycle figures were 15 µs and 164 µs, and importance scoring 25 µs, all
|
||||
undated from the original i7-12650H run. Eviction's membership check became a
|
||||
`HashSet` lookup (603fcf8) and `add_memory` stopped cloning Working-tier
|
||||
records (d787ac0), both on 2026-08-17; this run does not isolate their effect.
|
||||
|
||||
---
|
||||
|
||||
@@ -696,12 +973,17 @@ A full consolidation pass over 1,000 memories (eviction + promotion across Worki
|
||||
|
||||
Sorted timestamp index with binary search.
|
||||
|
||||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||||
|
||||
> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^temporal/'`
|
||||
|
||||
| Operation | Scale | Latency |
|
||||
|-----------|-------|---------|
|
||||
| Range query | 10K timestamps | **716 ns** |
|
||||
| Batch insert | 10K timestamps | 4.69 ms |
|
||||
| Range query | 10K timestamps | **622 ns** |
|
||||
| Batch insert | 10K timestamps | 3.22 ms |
|
||||
|
||||
Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K records: **716 nanoseconds.**
|
||||
Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K records: **622 nanoseconds.**
|
||||
Batch insert was 4.69 ms in the previous, undated i7-12650H figure.
|
||||
|
||||
---
|
||||
|
||||
@@ -709,16 +991,50 @@ Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K
|
||||
|
||||
HDF5 persistence with optional Write-Ahead Log.
|
||||
|
||||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. The stores
|
||||
use `MemoryConfig::new`'s defaults at this commit: `float16` embeddings and
|
||||
the int8 index.
|
||||
|
||||
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(save_without_wal_single|save_with_wal_single|save_batch_100|save_batch_1000|save_wal_1k_existing|wal_flush_100_entries|tick_session_1k|tick_session_10k)$'`
|
||||
|
||||
| Operation | Latency | Notes |
|
||||
|-----------|---------|-------|
|
||||
| Single save (no WAL) | 61 µs | Direct HDF5 write (owned-Vec IO path) |
|
||||
| Single save (with WAL) | 18 µs | WAL group-commit append; HDF5 write batched at flush |
|
||||
| Batch 100 | 723 µs | 7.2 µs per record |
|
||||
| Batch 1,000 | 6.17 ms | 6.2 µs per record |
|
||||
| WAL save (1K existing) | 539 µs | Incremental append |
|
||||
| WAL flush 100 entries | 787 µs | Merge WAL → HDF5 |
|
||||
| Session tick 1K | 5.76 ms | Full session maintenance |
|
||||
| Session tick 10K | 89.8 ms | Background operation |
|
||||
| Single save (no WAL) | 88.3 µs | Direct HDF5 write (owned-Vec IO path) |
|
||||
| Single save (with WAL) | 26.1 µs | WAL group-commit append; HDF5 write batched at flush |
|
||||
| Batch 100 | 1.08 ms | |
|
||||
| Batch 1,000 | 31.7 ms | |
|
||||
| WAL save (1K existing) | 282.8 µs | Incremental append |
|
||||
| WAL flush 100 entries | 530.1 µs | Merge WAL → HDF5 |
|
||||
| Session tick 1K | 3.83 ms | Full session maintenance |
|
||||
| Session tick 10K | 35.5 ms | Background operation |
|
||||
|
||||
Almost every row moved by more than 20% from the previous table, in both
|
||||
directions. Slower: single save 61 -> 88.3 µs (no WAL) and 18 -> 26.1 µs
|
||||
(WAL), batch 100 723 µs -> 1.08 ms, and **batch 1,000 6.17 -> 31.7 ms**.
|
||||
Faster: WAL save into 1K existing 539 -> 282.8 µs, WAL flush 787 -> 530.1 µs,
|
||||
session tick 5.76 -> 3.83 ms (1K) and 89.8 -> 35.5 ms (10K). The two single-save
|
||||
figures were last updated on 2026-07-01 and the rest are undated, all on the
|
||||
i7-12650H with `f32` stores.
|
||||
|
||||
**Isolated afterwards, same machine, same day** (`--warm-up-time 1
|
||||
--measurement-time 3`, the store settings switched through the bench's
|
||||
config):
|
||||
|
||||
| | f16 + int8 (default) | f32 + int8 | f16 + f32 index | f32 + f32 index | v2.3.0 (f32) |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| Single save (WAL) | 26.1 µs | 24.3 µs | 26.1 µs | 24.4 µs | 24.3 µs |
|
||||
| Single save (no WAL) | 89.0 µs | 87.3 µs | 88.8 µs | 87.4 µs | 82.6 µs |
|
||||
| Batch 100 | 1.09 ms | 0.90 ms | 1.09 ms | 0.90 ms | 0.87 ms |
|
||||
| Batch 1,000 | 31.5 ms | 31.1 ms | 31.5 ms | 31.6 ms | 30.3 ms |
|
||||
|
||||
The int8 index costs nothing on the write path. `float16` costs ~2 µs per
|
||||
saved record — the rounding — which is the whole difference at 100 records
|
||||
and within noise at 1,000. Built at the v2.3.0 tag (v2.2.0's benches do not
|
||||
compile) and run on this machine, the write path measures the same as today,
|
||||
so nothing regressed since then; the old 18 µs and 6.17 ms figures cannot be
|
||||
reproduced on this hardware and came from an undated run on another machine.
|
||||
`save_batch` scales linearly (about 12 µs per record from 500 to 8 000
|
||||
records); the bench's figure also includes dropping the store.
|
||||
|
||||
---
|
||||
|
||||
@@ -726,12 +1042,16 @@ HDF5 persistence with optional Write-Ahead Log.
|
||||
|
||||
Trivial/non-trivial classification for memory write filtering.
|
||||
|
||||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||||
|
||||
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^gate_'`
|
||||
|
||||
| Check | Latency |
|
||||
|-------|---------|
|
||||
| Trivial skip ("ok", "yes") | 61 ns |
|
||||
| Short phrase skip | 86 ns |
|
||||
| Non-trivial pass | 705 ns |
|
||||
| Ratio check | 488 ns |
|
||||
| Trivial skip ("ok", "yes") | 57.6 ns |
|
||||
| Short phrase skip | 83.0 ns |
|
||||
| Non-trivial pass | 570.5 ns |
|
||||
| Ratio check | 395.2 ns |
|
||||
|
||||
**Sub-microsecond filtering.** The gate decides whether to save a memory in under 1 µs.
|
||||
|
||||
@@ -741,27 +1061,70 @@ Trivial/non-trivial classification for memory write filtering.
|
||||
|
||||
End-to-end strategy evaluation including embedding operations.
|
||||
|
||||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||||
|
||||
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^strategy_(save_every|semantic_shift)'`
|
||||
|
||||
| Strategy | Condition | Latency |
|
||||
|----------|-----------|---------|
|
||||
| SaveEveryExchange (substantive) | Saves | 923 ns |
|
||||
| SaveEveryExchange (trivial) | Skips | 67 ns |
|
||||
| SaveOnSemanticShift (empty store) | Saves | 941 ns |
|
||||
| SaveEveryExchange (substantive) | Saves | 732.5 ns |
|
||||
| SaveEveryExchange (trivial) | Skips | 58.3 ns |
|
||||
| SaveOnSemanticShift (empty store) | Saves | 734.9 ns |
|
||||
|
||||
The two "Saves" rows were 923 ns and 941 ns in the previous table, undated
|
||||
and from the original i7-12650H run.
|
||||
|
||||
---
|
||||
|
||||
## Multi-modal Search
|
||||
|
||||
`clawhdf5_agent::multimodal::MultiModalStore`. Each of N records carries two
|
||||
384-dim embeddings: a text embedding of its caption and one of its primary
|
||||
modality, cycling image / audio / video. `search_cross_modal` scores every
|
||||
embedding of every record (2N vectors) and keeps each record's best;
|
||||
`search_by_modality(Image, …)` scores only image embeddings (a third of the
|
||||
records). Both are exact linear scans followed by a full sort; k = 10. Data is
|
||||
from a fixed-seed generator, so every run sees the same corpus.
|
||||
|
||||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c (with the
|
||||
new, uncommitted `crates/clawhdf5-agent/benches/multimodal_bench.rs` in the
|
||||
working tree). Criterion, 50 samples.
|
||||
|
||||
> **Run:** `cargo bench -p clawhdf5-agent --bench multimodal_bench`
|
||||
|
||||
| Search | 1K records | 10K records |
|
||||
|--------|-----------:|------------:|
|
||||
| Cross-modal (all modalities, 2N vectors) | 842.0 µs | 8.44 ms |
|
||||
| One modality (image, ~N/3 vectors) | 151.2 µs | 1.54 ms |
|
||||
|
||||
Both scale linearly with N. Cross-modal search is the slow path: at 1K
|
||||
records it takes 842.0 µs, where the flat single-vector cosine scan in
|
||||
[Vector Search Latency](#vector-search-latency) takes 47.4 µs over 1K vectors,
|
||||
and it has no index — at 10K it is 8.44 ms, slower than a 100K flat scan
|
||||
(6.58 ms). No earlier figure exists for this section.
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Derived from the 2026-09-24 tables above (tank, AMD Ryzen 7 7800X3D, commit
|
||||
5c8323c); no command of its own.
|
||||
|
||||
| Capability | Typical Latency | Scale |
|
||||
|------------|----------------|-------|
|
||||
| **Full memory search** | <1 ms | 10K records |
|
||||
| **Hybrid vector+keyword** | <200 µs | 1K records |
|
||||
| **Knowledge graph query** | <25 µs | 1K entities |
|
||||
| **Temporal range query** | <1 µs | 10K timestamps |
|
||||
| **Memory write** | <20 µs | Per record (WAL group-commit append) |
|
||||
| **Consolidation cycle** | <165 µs | 1K records |
|
||||
| **Importance gate** | <1 µs | Per record |
|
||||
| **Full memory search** (`HDF5Memory::hybrid_search`, p50) | 0.49 ms | 10K records |
|
||||
| **Hybrid vector+keyword** (flat, weighted) | 106.8 µs | 1K records |
|
||||
| **Knowledge graph query** (BFS) | 23.1 µs | 1K entities |
|
||||
| **Temporal range query** | 622 ns | 10K timestamps |
|
||||
| **Memory write** | 26.1 µs | Per record (WAL group-commit append) |
|
||||
| **Consolidation cycle** | 115.2 µs | 1K records |
|
||||
| **Importance gate** | 57.6–570.5 ns | Per record |
|
||||
|
||||
**The entire memory pipeline — search, retrieve, re-rank, filter — runs in single-digit milliseconds at agent-typical scales. Fast enough that memory becomes invisible infrastructure.**
|
||||
The previous summary said "<25 µs" for a knowledge-graph query and "<20 µs"
|
||||
for a memory write. The first holds again after the adjacency-index fix
|
||||
(23.1 µs); the second does not (26.1 µs).
|
||||
|
||||
**At agent-typical scales a full `HDF5Memory::hybrid_search` (vector + BM25 + fusion) over 10K records takes 0.49 ms at p50, and graph traversal, temporal queries, consolidation and the write gate each take well under a millisecond. At 100K records the hybrid search takes 4.69 ms.**
|
||||
|
||||
---
|
||||
|
||||
@@ -796,9 +1159,21 @@ _Latency benchmarks generated with Criterion.rs (50-100 samples per benchmark).
|
||||
|
||||
**Mode:** BM25-only retrieval — zero embeddings, `vector_weight=0.0`, `keyword_weight=1.0`
|
||||
|
||||
> **Run:** `cargo run --release --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s_cleaned.json`
|
||||
> (~70 s for all 500 questions on the tank reference machine). Omit the path for the
|
||||
> oracle variant; add `--limit N` for an evenly-strided subsample.
|
||||
> **Run:** `cargo run --release -p clawhdf5-bench --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s.json`
|
||||
> Omit the path for the oracle variant; add `--limit N` for an evenly-strided
|
||||
> subsample. Without the `embeddings` feature this runs two modes, BM25 only
|
||||
> and BM25 stemmed.
|
||||
|
||||
The BM25-only and BM25-stemmed figures in this section were measured again on
|
||||
2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, with the command
|
||||
above. Every figure of those two modes that this document publishes was
|
||||
reproduced exactly: BM25 Hit@1/5/10 and MRR at turn and session level, the
|
||||
BM25 per-type session Hit@1 values quoted in prose below, and the stemmed
|
||||
turn-level row plus its session Hit@1 in the tokenizer table. The run also
|
||||
produced figures this document does not publish (stemmed session Hit@5,
|
||||
Hit@10 and MRR, and per-type Hit@5/Hit@10/MRR for both modes), so there was
|
||||
nothing to compare them with. Rows that need real embeddings (vector-only,
|
||||
hybrid, RRF, re-ranking, the weight sweep) were not re-run.
|
||||
|
||||
### Full haystack — `longmemeval_s`, n=500 (the number to cite)
|
||||
|
||||
@@ -1058,14 +1433,29 @@ extraction over the haystack, which this harness does not do.
|
||||
|
||||
### Search Latency (LongMemEval, n=500 queries)
|
||||
|
||||
| Metric | Latency |
|
||||
|--------|---------|
|
||||
| avg | 1,004 µs |
|
||||
| p50 | 1,017 µs |
|
||||
| p95 | 2,031 µs |
|
||||
| p99 | 2,912 µs |
|
||||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, full
|
||||
`longmemeval_s` haystack (47.7 sessions / 494 turns per question).
|
||||
|
||||
Sub-millisecond median search across variable-length chat histories.
|
||||
> **Run:** `cargo run --release -p clawhdf5-bench --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s.json`
|
||||
|
||||
| Metric | BM25 only | BM25 stemmed |
|
||||
|--------|-----------|--------------|
|
||||
| avg | 11,034.7 µs | 11,096.2 µs |
|
||||
| p50 | 10,943.5 µs | 10,996.5 µs |
|
||||
| p95 | 11,749.6 µs | 12,018.8 µs |
|
||||
| p99 | 12,644.1 µs | 12,784.7 µs |
|
||||
|
||||
About 11 ms per query. Each question gets its own store, and the timed call is
|
||||
the first `hybrid_search_with` on that store. The first query on a store
|
||||
builds its indexes (the "cold index build" column of the search harness), so
|
||||
this is a first-query figure, not steady-state search; for that
|
||||
see the [search harness](#current-search-harness-2026-09-24).
|
||||
|
||||
The previous table (avg 1,004 µs, p50 1,017 µs) said "sub-millisecond median
|
||||
search", which no longer holds. That figure was undated and predates the
|
||||
full-haystack harness (added 2026-08-07, 7d6e269), so it was measured on the
|
||||
much smaller `longmemeval_oracle` corpus; on that corpus tank measured avg
|
||||
2,431 µs on 2026-08-05 (below).
|
||||
|
||||
---
|
||||
|
||||
@@ -1075,23 +1465,34 @@ Sub-millisecond median search across variable-length chat histories.
|
||||
**Topics:** Personal info, food preferences, music, travel, work/schedule, hobbies
|
||||
**Queries:** 35 questions across 4 types
|
||||
|
||||
> **Run:** `cargo run --release --bin memory_arena`
|
||||
> **Run:** `cargo run --release -p clawhdf5-bench --bin memory_arena`
|
||||
|
||||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||||
|
||||
### Results by Query Type
|
||||
|
||||
| Query Type | N | Hit@1 | Hit@5 | Hit@10 | MRR | Avg Latency |
|
||||
|------------|---|-------|-------|--------|-----|-------------|
|
||||
| single-session | 25 | 40.0% | 92.0% | 100.0% | 0.5788 | 7,853 µs |
|
||||
| multi-session | 5 | 40.0% | 60.0% | 80.0% | 0.5333 | 7,887 µs |
|
||||
| temporal | 3 | 33.3% | 66.7% | 66.7% | 0.5000 | 7,899 µs |
|
||||
| knowledge-update | 2 | 0.0% | 50.0% | 50.0% | 0.2500 | 7,899 µs |
|
||||
| **OVERALL** | **35** | **37.1%** | **82.9%** | **91.4%** | **0.5468** | **7,870 µs** |
|
||||
| single-session | 25 | 40.0% | 92.0% | 100.0% | 0.5755 | 248.4 µs |
|
||||
| multi-session | 5 | 40.0% | 60.0% | 80.0% | 0.5333 | 24.4 µs |
|
||||
| temporal | 3 | 33.3% | 66.7% | 66.7% | 0.5000 | 28.0 µs |
|
||||
| knowledge-update | 2 | 0.0% | 50.0% | 50.0% | 0.2500 | 38.7 µs |
|
||||
| **OVERALL** | **35** | **37.1%** | **82.9%** | **91.4%** | **0.5444** | p50 **30.1 µs**, p95 49.8 µs |
|
||||
|
||||
**Key findings:**
|
||||
- Hit@10 of 91.4% across all query types with BM25-only (no embeddings)
|
||||
- Single-session recall strongest at 100% Hit@10
|
||||
- Knowledge-update hardest (requires temporal disambiguation) — would improve significantly with vector similarity
|
||||
- Latency dominated by BM25 index build over 999 turns (~7.9 ms)
|
||||
- Search takes about 30 µs at p50 over the 999 turns. All 35 queries run
|
||||
against one store, and single-session's higher average (248.4 µs) includes
|
||||
that store's first query, which builds its indexes.
|
||||
|
||||
The previous table put every query at ~7.9 ms ("latency dominated by BM25
|
||||
index build over 999 turns"). That was before v2.4.0, when
|
||||
`HDF5Memory::hybrid_search` rebuilt the BM25 index from scratch and rewrote
|
||||
the store on every query; it now keeps the index for the life of the store.
|
||||
The MRR moved slightly (single-session 0.5788 -> 0.5755, overall 0.5468 ->
|
||||
0.5444); the Hit@k figures are unchanged.
|
||||
|
||||
---
|
||||
|
||||
@@ -1099,46 +1500,89 @@ Sub-millisecond median search across variable-length chat histories.
|
||||
|
||||
HDF5 file size at various record counts — 384-dimensional embeddings, 200-char text.
|
||||
|
||||
> **Run:** `cargo run --release --bin footprint_bench`
|
||||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||||
|
||||
### Uncompressed (no WAL)
|
||||
> **Run:** `cargo run --release -p clawhdf5-bench --bin footprint_bench`
|
||||
|
||||
| Records | File Size | Raw Data | Bytes/Record | Throughput |
|
||||
|---------|-----------|----------|--------------|------------|
|
||||
| 100 | 176.4 KB | 169.5 KB | 1.8 KB | 100,000 rec/s |
|
||||
| 1K | 1.7 MB | 1.7 MB | 1.8 KB | 109,643 rec/s |
|
||||
| 10K | 17.0 MB | 16.6 MB | 1.7 KB | 118,100 rec/s |
|
||||
| 50K | 85.0 MB | 82.8 MB | 1.7 KB | 110,723 rec/s |
|
||||
| 100K | 169.8 MB | 165.6 MB | 1.7 KB | 111,422 rec/s |
|
||||
**These are `float16` stores.** `footprint_bench` builds its stores with
|
||||
`MemoryConfig::new`, which defaults to `float16` embeddings since 5c8323c,
|
||||
although the binary still prints "Embedding: 384-dim f32". Its "Raw Data"
|
||||
column, and so the ratios, count embeddings as `f32` (1,536 bytes per record)
|
||||
plus text. The synthetic embeddings take only 1,000 distinct values, and each
|
||||
record's text is fixed by `record_idx % 40` (a 40-word bank walked from that
|
||||
start), so a store holds only 40 distinct texts. Both compress far better than
|
||||
real data would; read the gzip figures as an upper bound, and see the next
|
||||
paragraph for why the other tables are optimistic too. "MB" and "KB" are the
|
||||
binary's units (powers of 1024).
|
||||
|
||||
**1.7 KB per record** — HDF5 overhead is near-zero. Ingestion throughput exceeds **100K records/sec**.
|
||||
**No table here is uncompressed.** The store always deflates its string
|
||||
datasets: `write_string_dataset` applies deflate level 6 to any string dataset
|
||||
of 4 KiB or more, whatever `MemoryConfig::compression` says
|
||||
(`crates/clawhdf5-agent/src/schema.rs`, `STRING_COMPRESS_THRESHOLD`, added in
|
||||
4fa7e89 on 2026-06-03). With only 40 distinct texts, the chunk text costs
|
||||
almost nothing on disk: at 10K records, 50-char text takes 804 B per record
|
||||
and 1000-char text 944 B, so 950 more characters cost only 140 B. The
|
||||
`float16` embeddings alone are 768 B per record, and 200 bytes of raw text
|
||||
would bring that to about 968 B before any other field or overhead; the
|
||||
measured 803–829 B is below that only because the text is compressed away.
|
||||
Real text will not deflate like this, so expect more than 820 B per record
|
||||
with real data. `MemoryConfig::compression` decides only whether the
|
||||
embeddings are compressed; the gzip table below is that setting turned on.
|
||||
|
||||
### `compression` off (no WAL)
|
||||
|
||||
| Records | File Size | Raw Data (f32) | Bytes/Record | Throughput |
|
||||
|---------|-----------|----------------|--------------|------------|
|
||||
| 100 | 90.0 KB | 169.5 KB | 921 B | 75,914 rec/s |
|
||||
| 1K | 810.4 KB | 1.7 MB | 829 B | 72,266 rec/s |
|
||||
| 10K | 7.8 MB | 16.6 MB | 820 B | 74,509 rec/s |
|
||||
| 50K | 38.3 MB | 82.8 MB | 803 B | 72,829 rec/s |
|
||||
| 100K | 76.7 MB | 165.6 MB | 803 B | 71,102 rec/s |
|
||||
|
||||
**About 820 B per record** at 10K, for this synthetic corpus (see above: the
|
||||
text is deflated regardless and has only 40 distinct values, so real data
|
||||
takes more). Ingestion runs at 71,102–75,914 records/sec.
|
||||
|
||||
### With Gzip Compression (level 6)
|
||||
|
||||
| Records | Compressed | Ratio | Bytes/Record |
|
||||
|---------|------------|-------|--------------|
|
||||
| 100 | 31.5 KB | 5.37x | 323 B |
|
||||
| 1K | 277.1 KB | 6.12x | 283 B |
|
||||
| 10K | 2.7 MB | 6.17x | 281 B |
|
||||
| 50K | 13.4 MB | 6.17x | 281 B |
|
||||
| 100K | 26.9 MB | 6.15x | 282 B |
|
||||
| Records | Compressed | Ratio vs f32 raw | Bytes/Record |
|
||||
|---------|------------|------------------|--------------|
|
||||
| 100 | 17.2 KB | 9.85x | 176 B |
|
||||
| 1K | 56.4 KB | 30.06x | 57 B |
|
||||
| 10K | 471.3 KB | 35.97x | 48 B |
|
||||
| 50K | 2.3 MB | 36.67x | 47 B |
|
||||
| 100K | 4.5 MB | 36.73x | 47 B |
|
||||
|
||||
**6.2x compression ratio** — 100K agent memories in 27 MB compressed.
|
||||
|
||||
### Text Length Comparison (10K records, no compression)
|
||||
### Text Length Comparison (10K records, `compression` off)
|
||||
|
||||
| Text Length | File Size | Bytes/Record | Throughput |
|
||||
|-------------|-----------|--------------|------------|
|
||||
| short (50 chars) | 15.6 MB | 1.6 KB | 177,925 rec/s |
|
||||
| medium (200 chars) | 17.0 MB | 1.7 KB | 172,152 rec/s |
|
||||
| long (1000 chars) | 24.6 MB | 2.5 KB | 157,807 rec/s |
|
||||
| short (50 chars) | 7.7 MB | 804 B | 97,615 rec/s |
|
||||
| medium (200 chars) | 7.8 MB | 820 B | 89,597 rec/s |
|
||||
| long (1000 chars) | 9.0 MB | 944 B | 51,462 rec/s |
|
||||
|
||||
### WAL Overhead (1K records)
|
||||
|
||||
| Mode | File Size | Ingest Time | Overhead |
|
||||
|------|-----------|-------------|----------|
|
||||
| No WAL | 1.7 MB | 5.7 ms | — |
|
||||
| With WAL | 1.7 MB + 9 B WAL | 5.3 ms | ±8% (negligible) |
|
||||
| No WAL | 810.4 KB | 10.8 ms | — |
|
||||
| With WAL | 810.4 KB + 9 B WAL | 10.6 ms | −2% (negligible) |
|
||||
|
||||
**Compared with the previous tables.** Those were undated `f32` stores from the
|
||||
original i7-12650H run: 1.7 KB per record (169.8 MB at 100K), gzip 6.2x
|
||||
(26.9 MB at 100K), and more than 100,000 records/sec. Two changes separate
|
||||
those tables from these, not one. `float16` embeddings save 768 B per record.
|
||||
The old tables (recorded by 2026-05-14) also predate 4fa7e89 (2026-06-03),
|
||||
which began deflating string datasets of 4 KiB or more, so their text was
|
||||
stored raw; with only 40 distinct texts, deflate now removes most of it.
|
||||
An `f32` embedding plus 200 B of text is 1,536 + 200 = 1,736 B; `float16`
|
||||
alone would bring that to about 968 B per record, not the measured 803–829 B. This run does not measure how the saving splits between the two
|
||||
beyond that, nor explain why the gzip ratio rose so much further, nor
|
||||
separate the lower throughput (and the 1K ingest going from 5.7 to 10.8 ms)
|
||||
from the change of machine and store configuration. The
|
||||
`f32` on-disk figures were not re-measured; the dated
|
||||
[float16 study](#float16-embedding-storage-memoryconfigfloat16) compares the
|
||||
two directly (100K × 384: 154.0 MiB `f32`, 80.8 MiB `float16`).
|
||||
|
||||
---
|
||||
|
||||
@@ -1146,7 +1590,12 @@ HDF5 file size at various record counts — 384-dimensional embeddings, 200-char
|
||||
|
||||
Hippocampal-inspired memory consolidation improves both retrieval quality and search speed.
|
||||
|
||||
> **Run:** `cargo run --release --bin consolidation_efficiency`
|
||||
> **Run:** `cargo run --release -p clawhdf5-bench --bin consolidation_efficiency`
|
||||
|
||||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. The run
|
||||
was stopped (after about 19 minutes on one core) while still computing a
|
||||
100K row of the cycle-time table, so that row and the binary's memory-reduction
|
||||
part were not produced; neither has ever been published here.
|
||||
|
||||
### Retrieval Quality Before vs. After Consolidation
|
||||
|
||||
@@ -1159,17 +1608,22 @@ Hippocampal-inspired memory consolidation improves both retrieval quality and se
|
||||
| Hit@5 | 100.0% | 100.0% | — |
|
||||
| Hit@10 | 100.0% | 100.0% | — |
|
||||
| MRR | 1.0000 | 1.0000 | — |
|
||||
| Search latency | 2,752 µs | 312 µs | **8.8x faster** |
|
||||
| Search latency (avg) | 2,223.7 µs | 239.2 µs | **9.3x faster** |
|
||||
|
||||
Signal records survive consolidation because they are accessed 15+ times, giving them high decay scores. 900 noise records evicted, search speeds up 8.8x, and **zero quality loss** — perfect recall maintained.
|
||||
Signal records survive consolidation because they are accessed 15+ times, giving them high decay scores. 900 noise records evicted, search speeds up 9.3x, and **zero quality loss** — perfect recall maintained. The consolidation cycle itself took 0.13 ms and promoted 2 records.
|
||||
|
||||
### Consolidation Cycle Time
|
||||
|
||||
| Records | Cycle Time | Evictions | Promotions |
|
||||
|---------|-----------|-----------|------------|
|
||||
| 100 | 21 µs | 100 | 0 |
|
||||
| 1K | 345 µs | 1,000 | 0 |
|
||||
| 10K | 17.3 ms | 10,000 | 0 |
|
||||
| 100 | 17 µs | 100 | 0 |
|
||||
| 1K | 189 µs | 1,000 | 0 |
|
||||
| 10K | 2.16 ms | 10,000 | 0 |
|
||||
|
||||
The 1K and 10K cycles were 345 µs and 17.3 ms in the previous, undated table
|
||||
(before: 2,752 µs / after: 312 µs for search). Eviction's membership check
|
||||
changed from a linear scan to a `HashSet` in 603fcf8 (2026-08-17), after those
|
||||
figures were recorded; this run does not isolate its effect.
|
||||
|
||||
---
|
||||
|
||||
@@ -1178,34 +1632,44 @@ Signal records survive consolidation because they are accessed 15+ times, giving
|
||||
In-memory key-value store with TTL, capacity eviction, and embedding search.
|
||||
No network hop, no serialization — direct HashMap operations.
|
||||
|
||||
> **Run:** `cargo run --release --bin ephemeral_perf`
|
||||
> **Run:** `cargo run --release -p clawhdf5-bench --bin ephemeral_perf`
|
||||
|
||||
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
|
||||
|
||||
### Latency Comparison
|
||||
|
||||
| Operation | clawhdf5 Ephemeral | Redis (single-node)¹ | Speedup |
|
||||
| Operation | clawhdf5 Ephemeral | Redis (single-node)¹ | Ratio¹ |
|
||||
|-----------|-------------------|---------------------|---------|
|
||||
| SET | **356 ns/op** | ~25,000 ns/op | **70x** |
|
||||
| GET (hit) | **179 ns/op** | ~25,000 ns/op | **140x** |
|
||||
| GET (miss) | **62 ns/op** | ~25,000 ns/op | **403x** |
|
||||
| DELETE | **124 ns/op** | ~25,000 ns/op | **202x** |
|
||||
| SET+embedding | **268 ns/op** | N/A | — |
|
||||
| SET | **366 ns/op** | ~25,000 ns/op (not measured) | 68.3x |
|
||||
| GET (hit) | **115 ns/op** | ~25,000 ns/op (not measured) | 216.7x |
|
||||
| GET (miss) | **57 ns/op** | — | — |
|
||||
| DELETE | **99 ns/op** | — | — |
|
||||
| SET+embedding | **236 ns/op** | N/A | — |
|
||||
|
||||
> ¹ Redis latency includes network round-trip (loopback). clawhdf5 ephemeral is in-process — no network.
|
||||
> ¹ **The Redis figure is not a measurement.** It is a constant hard-coded in
|
||||
> `ephemeral_perf` as a "typical single-node" latency, and the binary computes
|
||||
> the two ratios from it; only the clawhdf5 column was measured. A typical
|
||||
> Redis figure includes a network round-trip, which the in-process ephemeral
|
||||
> tier does not have, so the ratios compare different things.
|
||||
|
||||
### Throughput
|
||||
|
||||
| Operation | ops/sec |
|
||||
|-----------|---------|
|
||||
| SET | 2,810,649 |
|
||||
| GET | 5,584,684 |
|
||||
| DELETE | 8,093,731 |
|
||||
| SET+EMB (384d) | 3,725,877 |
|
||||
| SET | 2,732,568 |
|
||||
| GET (hit) | 8,666,607 |
|
||||
| GET (miss) | 17,554,663 |
|
||||
| DELETE | 10,109,895 |
|
||||
| SET+EMB (384d) | 4,235,766 |
|
||||
|
||||
### Embedding Search (ephemeral tier)
|
||||
|
||||
| Scale | Latency |
|
||||
|-------|---------|
|
||||
| 10K entries @ 384d | 2.9 ms/query |
|
||||
| 10K entries @ 384d | 2.85 ms/query |
|
||||
|
||||
GET (hit) moved from 179 to 115 ns/op and DELETE from 124 to 99 ns/op (GET
|
||||
throughput 5,584,684 -> 8,666,607 ops/sec); the previous figures were undated.
|
||||
|
||||
---
|
||||
|
||||
@@ -1263,10 +1727,16 @@ Measured 2026-08-07 on tank (Ryzen 7 7800X3D, 246 MB dataset in page cache).
|
||||
|
||||
### Measured Platforms
|
||||
|
||||
The tank row was measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit
|
||||
5c8323c, with the single benchmark rather than the whole script:
|
||||
|
||||
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^ivf_search_10k_nprobe10$'`
|
||||
|
||||
| Platform | CPU | 10K IVF Search | Notes |
|
||||
|----------|-----|----------------|-------|
|
||||
| Linux x86_64 | Intel i7-12650H (10C, 4.7 GHz) | 27 µs | Primary CI target |
|
||||
| macOS aarch64 | Apple M3 Max (14C) | ~18 µs | ~33% faster via NEON SIMD |
|
||||
| Linux x86_64 | AMD Ryzen 7 7800X3D (8C/16T), tank | 24.8 µs | Measured 2026-09-24 |
|
||||
| Linux x86_64 | Intel i7-12650H (10C, 4.7 GHz) | 27 µs | Undated, original run |
|
||||
| macOS aarch64 | Apple M3 Max (14C) | ~18 µs | Undated; ~33% faster than the i7-12650H row. Not reproducible on the hardware available for the 2026-09-24 re-run |
|
||||
|
||||
### Reproducibility
|
||||
|
||||
@@ -1610,6 +2080,11 @@ vs-libhdf5 validation above) to give both sections the dated, hardware-cited,
|
||||
reproducible citation the top-of-file traceability note flags them as
|
||||
missing.
|
||||
|
||||
> The sections this re-run checked were themselves re-measured on tank on
|
||||
> 2026-09-24 and now show those figures; the i7-12650H figures this section
|
||||
> compares against ("above", "at the top of this file") are quoted in those
|
||||
> sections' notes. Kept as the 2026-08-05 record.
|
||||
|
||||
### LongMemEval Results (reproduction)
|
||||
|
||||
```bash
|
||||
|
||||
Reference in New Issue
Block a user