quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

bench: re-run every stale BENCHMARKS.md section, dated and traced

Browse Source 
Every undated or pre-September section re-run on one machine on one day
(tank, AMD Ryzen 7 7800X3D, 2026-09-24, commit 5c8323c), 24 commands run
serially with the load average checked before each, with the command
recorded for each section. A separate check traced every changed number
back to the raw output; its corrections are applied (e.g. the on-disk
~820 B/record is float16 plus always-deflated text on a synthetic corpus
of 40 distinct texts, not float16 alone).

Two apparent regressions were isolated rather than published:
- knowledge-graph traversal: a real bug, fixed in the previous commit;
- the write path: v2.3.0 built and run on the same machine measures the
  same as today, so the old 18 us / 6.17 ms figures (undated, other
  hardware) are not reproducible; float16 adds ~2 us per save and the
  int8 index nothing (both isolated by switching the bench's config).

Also:
- new multimodal_bench: cross-modal search at 1K/10K records, which the
  README claimed but nothing measured;
- footprint_bench reports whether it built float16 or f32 stores and
  takes --f32 (it kept printing "f32" after the default changed);
- README: performance tables, the "Why" table figures and the SQLite
  migration section (from the previous migrate commit);
- CHANGELOG for this branch.

Not re-run: consolidation_efficiency's 100K row and its memory-reduction
part (stopped for time), and cross_platform.sh.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-24 23:45:15 -05:00
co-authored by Claude Opus 5.5
parent 1b3bbb054a
commit dce5559ff2
6 changed files with 880 additions and 175 deletions
Download Patch File Download Diff File Expand all files Collapse all files
BENCHMARKS.md
+614 -139
View File
@@ -2,20 +2,40 @@
> Pure Rust. Zero C dependencies. Single file. Fast enough to forget it's there.
**System:** Intel i7-12650H (10C/16T, 4.7 GHz boost) · 32 GB DDR5 · Linux 6.8.0
**System (original run):** Intel i7-12650H (10C/16T, 4.7 GHz boost) · 32 GB DDR5 · Linux 6.8.0
**Rust:** 1.96.0-nightly (2026-03-14) · `--release` profile
**Date:** 2026-07-01
> **Traceability note:** the "h5bench-Equivalent I/O Benchmarks" and both
> "Independent Validation: tank" sections below meet a dated,
> hardware-cited, reproducible standard (explicit date, machine spec, and a
> runnable command per result) — this now covers "LongMemEval Results",
> "SIMD & Parallelism", "Vector Search Latency", and "Comparison to MemX" via
> their tank re-runs. The remaining undated sections above (Hybrid Search,
> Knowledge Graph, Memory Consolidation, Temporal Index, Write Path, Decision
> Gate, Memory Strategy, Multi-Session Benchmark, Memory Footprint,
> Consolidation Efficiency, Ephemeral Tier) do not yet meet that bar — this is
> a known, tracked documentation gap, not a claim that those numbers are wrong.
Sections that carry their own "Measured …" line were measured there instead,
not on the system above. The 2026-09-24 re-run was on tank (AMD Ryzen 7
7800X3D, 8C/16T, Linux 7.0.0-34-generic, rustc 1.98.1) at commit 5c8323c,
serially, with no `--warm-up-time`/`--measurement-time` overrides, waiting
before each command while the 1-minute load average was above 2.0. The
`bench` targets ran at Criterion's defaults (3 s warm-up, 5 s target
measurement time, 100 samples). Every `memory_bench` group that was run
(`vector_search_latency`, `hybrid_search`, `knowledge_graph`,
`consolidation`, `temporal`) and the multi-modal group set
`sample_size(50)`, so those sections took 50 samples, not 100. The 5 s is a
target: Criterion stretched it where 5 s could not hold the samples it needed
(for example 8.5 s for `vector_search_latency/bench_cosine_search/100k`,
15.5 s for the 1K consolidation cycle, 252 s for `tick_session_10k`).
> **Traceability note:** a section meets the dated, hardware-cited,
> reproducible standard when it gives an explicit date, the machine, and a
> runnable command for its results. As of the 2026-09-24 re-run on tank that
> covers Vector Search Latency, Comparison to MemX, SIMD & Parallelism, Hybrid
> Search, Knowledge Graph, Memory Consolidation, Temporal Index, Write Path,
> Decision Gate, Memory Strategy, Summary (derived from those), LongMemEval
> Results (the BM25 rows), Multi-Session Benchmark, Memory Footprint (on disk
> and in memory), Consolidation Efficiency, Ephemeral Tier, Multi-modal Search,
> the Search and Read harnesses, and the "h5bench-Equivalent I/O Benchmarks"
> and "Independent Validation: tank" sections. What does not yet meet that bar:
> the LongMemEval rows that need real embeddings (not re-run here, except the
> dated float16 comparison), the Consolidation Efficiency 100K cycle row and
> memory-reduction part (the 2026-09-24 run was stopped before it produced
> them), the int8 side of "Quantising the index copy" (not re-run), and the
> i7-12650H and macOS M3 Max rows under Cross-Platform Notes. That is a
> known, tracked documentation gap, not a claim that those numbers are wrong.
>
> **Correctness note (2026-08-06).** Being dated and reproducible is necessary but
> not sufficient — a number can be perfectly reproducible and still measure the
@@ -57,6 +77,23 @@ What remains at 2.43x: the flat vectors (1.0x), the HNSW index's own copy of
them (1.0x), and text, ids and graph (~0.4x). The index copy is the next
target — it is what a quantised or borrowed representation would address.
**Current figures (`f32` index).** Measured 2026-09-24 on tank (AMD Ryzen 7
7800X3D), commit 5c8323c:
> **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --footprint --full`
| N | vectors (raw) | entries MiB | store MiB | indexes MiB | reopened MiB | peak during open MiB | reopened / raw |
|---:|---:|---:|---:|---:|---:|---:|---:|
| 1 000 | 1 | 2 | 0 | 2 | 4 | 5 | 2.40x |
| 10 000 | 15 | 17 | 12 | 32 | 44 | 61 | 3.03x |
| 100 000 | 146 | 172 | 60 | 266 | 399 | 562 | 2.72x |
These are the "reopened, f32" figures in the next table, reproduced exactly.
They are above the "after" column of the table above (44 MiB vs 35 MiB at 10K,
399 vs 357 MiB at 100K): the 2.43x was measured for 2e7e045, and the store
already measured 399 MiB (2.72x) when the int8 index landed later the same
day (c0a9206). The git log does not say what changed in between.
### Quantising the index copy (`quantized_index`)
`MemoryConfig::quantized_index` stores the index's copy as `i8` instead of
@@ -108,6 +145,20 @@ second**, builds **1.8x faster**, and holds a quarter of the vectors. (Compare
only at equal `ef`: with re-scoring the harness raises `ef` to at least the
candidate pool, so the `ef = 16` and `ef = 32` rows are not like-for-like.)
**Re-check, 2026-09-24.** Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D),
commit 5c8323c. The `f32` memory columns of the first table reproduce exactly
(see [Memory footprint](#memory-footprint)). Only the `f32` side of the speed
table was re-run, once, not as alternating medians:
> **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --full`
At N = 100 000, `ef = 64`, the `f32` index measured recall@10 0.9945 (same),
19 001 QPS (was 13 399) and a 2700.8 ms build (was 3197 ms). The QPS figure
moved by more than 20%; the log does not explain it, and a single run is not
the same-moment paired comparison the table above is. The int8 columns were
not re-run, so the 1.63x ratio has not been re-checked against the new `f32`
figure.
#### On ARM (Raspberry Pi 5, Cortex-A76)
`dot_i8` has two aarch64 kernels: `SDOT` for CPUs with the ARMv8.2
@@ -268,6 +319,10 @@ high-water mark alongside the retained figure:
| 10 000 | 44 | 61 |
| 100 000 | 399 | 562 |
Reproduced exactly on 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit
5c8323c, by `search_harness -- --footprint --full` (table under
[Memory footprint](#memory-footprint)).
The peak is set *after* the parse, by the index build, so a buffer allocated
and freed during the parse never reaches the high-water mark. Holding a
deliberate extra copy of the file across the whole parse leaves the peak
@@ -359,6 +414,50 @@ contiguous dataset is converted straight from the file bytes (one copy instead
of two), and the native-endian conversions no longer zero a buffer they are
about to overwrite.
### Current: read harness (2026-09-24)
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
> **Run:** `cargo run --release -p clawhdf5-bench --bin read_harness`
4096 x 2048 f64 (64 MB per dataset), chunks 256 x 256, file 129 MB
| layout | read | selected | time ms | MB/s of selection | vs full read |
|---|---|---:|---:|---:|---:|
| chunked + deflate | full (first) | 64 MB | 65.6 | 975 | |
| chunked + deflate | full (repeat) | 64 MB | 63.2 | 1013 | 1.00x |
| chunked + deflate | 64 x 64 window (1 chunk) | 0.03 MB | 0.18 | 177 | 0.003x |
| chunked + deflate | 512 x 512 window (4-9 chunks) | 2.00 MB | 4.12 | 485 | 0.063x |
| chunked + deflate | one row | 0.02 MB | 0.99 | 16 | 0.015x |
| chunked + deflate | one column | 0.03 MB | 2.02 | 15 | 0.031x |
| chunked | full (first) | 64 MB | 60.5 | 1057 | |
| chunked | full (repeat) | 64 MB | 56.7 | 1129 | 1.00x |
| chunked | 64 x 64 window (1 chunk) | 0.03 MB | 0.10 | 325 | 0.002x |
| chunked | 512 x 512 window (4-9 chunks) | 2.00 MB | 3.19 | 628 | 0.053x |
| chunked | one row | 0.02 MB | 0.05 | 284 | 0.001x |
| chunked | one column | 0.03 MB | 0.49 | 64 | 0.008x |
| contiguous | full (first) | 64 MB | 30.1 | 2124 | |
| contiguous | full (repeat) | 64 MB | 26.1 | 2452 | 1.00x |
| contiguous | 64 x 64 window (1 chunk) | 0.03 MB | 0.08 | 394 | 0.003x |
| contiguous | 512 x 512 window (4-9 chunks) | 2.00 MB | 3.36 | 595 | 0.112x |
| contiguous | one row | 0.02 MB | 0.03 | 486 | 0.001x |
| contiguous | one column | 0.03 MB | 2.61 | 12 | 0.087x |
Full reads match the "after" column above (chunked + deflate 69 ms then, 63.2
to 65.6 ms now). The compressed selections are faster than in "After: partial
reads" (64 x 64 window 0.39 -> 0.18 ms, one row 2.69 -> 0.99 ms, one column
5.23 -> 2.02 ms) and match the 2026-09-23 figures in
[Deflate backend](#deflate-backend-zlib-rs-vs-zlib-ng) (0.18, 1.00 and 2.01
ms), which were taken after the zlib-rs switch and the one-shot codec calls
described there; this run does not isolate which change accounts for it.
The uncompressed windows went the other way. Against "After: partial reads",
the 512 x 512 window is about 60% slower (chunked 1.99 -> 3.19 ms, contiguous
2.12 -> 3.36 ms) and the chunked 64 x 64 window went from 0.06 to 0.10 ms.
The rows and columns of the uncompressed layouts are within 20% (chunked
column 0.45 -> 0.49 ms, contiguous column 2.55 -> 2.61 ms). This run does not
explain the slower windows.
## Search harness baseline (v2.3.0)
Produced by `cargo run --release -p clawhdf5-bench --bin search_harness -- --full`
@@ -596,68 +695,217 @@ Recall on clustered data is unchanged or slightly better (100K, `ef = 64`:
0.474 -> 0.444), the cost of batch members not seeing each other while
planning; batches are capped at 1/16 of the graph and 512 nodes.
### Current: search harness (2026-09-24)
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, default
features (so the build is batched and `parallel`, 16 threads). The HNSW tables
use an `f32` index. The end-to-end table uses `MemoryConfig::new`'s defaults,
which at this commit are `float16` embeddings and the int8 index; every
earlier end-to-end table in this section was an `f32` store with an `f32`
index.
> **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --full`
#### HNSW, N = 1000, dim = 384, M = 16, ef_construction = 64, storage = Float32
build: 14.7 ms (67954 vectors/s) · exact scan: 4403 QPS, p50 225 µs
| ef | recall@10 | QPS | p50 µs | p99 µs |
|---:|---:|---:|---:|---:|
| 16 | 0.9940 | 134065 | 7 | 18 |
| 32 | 1.0000 | 105134 | 9 | 21 |
| 64 | 1.0000 | 77541 | 13 | 26 |
| 128 | 1.0000 | 36938 | 26 | 35 |
| 256 | 1.0000 | 19249 | 52 | 63 |
#### HNSW, N = 10000, dim = 384, M = 16, ef_construction = 64, storage = Float32
build: 135.6 ms (73736 vectors/s) · exact scan: 435 QPS, p50 2295 µs
| ef | recall@10 | QPS | p50 µs | p99 µs |
|---:|---:|---:|---:|---:|
| 16 | 0.9975 | 79614 | 12 | 24 |
| 32 | 1.0000 | 70556 | 14 | 21 |
| 64 | 1.0000 | 51313 | 19 | 29 |
| 128 | 1.0000 | 22846 | 44 | 57 |
| 256 | 1.0000 | 10267 | 97 | 119 |
#### HNSW, N = 100000, dim = 384, M = 16, ef_construction = 64, storage = Float32
build: 2700.8 ms (37027 vectors/s) · exact scan: 43 QPS, p50 23433 µs
| ef | recall@10 | QPS | p50 µs | p99 µs |
|---:|---:|---:|---:|---:|
| 16 | 0.9180 | 20043 | 47 | 104 |
| 32 | 0.9775 | 21348 | 44 | 104 |
| 64 | 0.9945 | 19001 | 51 | 101 |
| 128 | 0.9995 | 13458 | 73 | 137 |
| 256 | 0.9995 | 5128 | 196 | 282 |
#### End to end: `HDF5Memory::hybrid_search` (k = 10, weights 0.7 / 0.3)
| N | ingest ms | cold index build ms | checkpoint ms | open ms | first query after open ms | p50 ms | p99 ms | QPS |
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 1000 | 15 | 14 | 4.4 | 1.9 | 1.5 | 0.07 | 0.08 | 13869.5 |
| 10000 | 146 | 126 | 33.5 | 18.3 | 14.8 | 0.49 | 0.50 | 2044.1 |
| 100000 | 1498 | 1833 | 495.7 | 259.9 | 155.1 | 4.69 | 5.22 | 212.7 |
Steady-state query latency is where the v2.4.0 work left it (p50 0.07 / 0.49 /
4.69 ms against 0.07 / 0.49 / 4.65 ms). Figures that moved by more than 20%:
- **Build.** 14.7 / 135.6 / 2700.8 ms against 34 / 388 / 5.9 s for
"batched + `parallel`" above. The git log does not explain the difference.
- **Cold index build** in the end-to-end table: 14 / 126 / 1833 ms, down from
112 / 1487 / 20285 ms in the "unranked keyword scores" table, which predates
the batched and parallel build described above.
- **QPS at 100K, `ef = 64`**: 19 001 against 13 980 in the "unit-vector"
table, with recall 0.9945 against 0.984; the recall gain is the one the
batched build reported. The QPS gain is not explained by the log.
- **Other HNSW QPS cells**, against the same "unit-vector" tables: 10K
`ef = 16` 79 614 against 54 608 (+46%); 100K `ef = 16` 20 043 against
15 139 (+32%), `ef = 128` 13 458 against 10 959 (+23%), `ef = 256` 5 128
against 3 731 (+37%). The other cells, and every 1K cell, are within 20%.
The log does not explain these either.
- **Open** in the end-to-end table: 1.9 / 18.3 / 259.9 ms against 1.1 / 13.7
/ 353.1 ms in the "unranked keyword scores" table, slower at 1K and 10K and
faster at 100K. This run does not isolate the cause.
- **Checkpoint at 100K**: 495.7 ms against 728.9 ms, and ingest up 11 -> 15,
104 -> 146 ms at 1K / 10K. This store is `float16` with an int8 index, the
earlier ones `f32`; the [float16 study](#float16-embedding-storage-memoryconfigfloat16)
measured float16 checkpoints at 512 ms against 752 ms for `f32` at 100K.
This run does not isolate the ingest change.
## Vector Search Latency
Brute-force cosine similarity over 384-dimensional embeddings (OpenAI text-embedding-3-small size).
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(vector_search_1k|simd_cosine_10k|simd_cosine_100k|prenorm_search_10k|ivf_search_10k_nprobe10|ivf_search_100k_nprobe10|ivf_pq_search_100k|rairs_search_10k_nprobe10)$'`
| Scale | Flat Search | Pre-norm | IVF (nprobe=10) | IVF-PQ | RAIRS |
|-------|-------------|----------|-----------------|--------|-------|
| **1K** | 54 µs | 62 µs | — | — | — |
| **10K** | 753 µs | 706 µs | 27 µs | — | 159 µs |
| **100K** | 11.4 ms | — | 1.32 ms | 1.19 ms | — |
| **1K** | 47.4 µs | — | — | — | — |
| **10K** | 500.5 µs | 316.5 µs | 24.8 µs | — | 124.0 µs |
| **100K** | 6.58 ms | — | 592 µs | 869 µs | — |
**Key insight:** At 10K records (typical agent memory), IVF search delivers **27 µs** — that's 26x faster than flat search. Even at 100K records, IVF-PQ keeps search under **1.2 ms**.
The 1K pre-norm cell is blank because no 1K pre-norm benchmark exists (the
old 62 µs has no source). `memory_bench`'s own cosine search
(`cargo bench -p clawhdf5-agent --bench memory_bench -- '^vector_search_latency/'`)
agrees: 47.6 µs / 505.1 µs / 6.67 ms at 1K / 10K / 100K.
Most cells moved by more than 20% from the previous table (10K flat 753 µs,
pre-norm 706 µs, RAIRS 159 µs; 100K flat 11.4 ms, IVF 1.32 ms, IVF-PQ 1.19 ms).
Those were undated figures from the original i7-12650H run. The 2026-08-05
tank run below is within 4% of the new numbers in every cell but one (1K flat
47.8 µs, 10K flat 501 µs, pre-norm 322 µs, IVF 24.8 µs; 100K flat 6.60 ms,
IVF 608 µs, IVF-PQ 865 µs), so for those cells the difference is the machine.
The exception is RAIRS: 109 µs on 2026-08-05, 124.0 µs now (+14%); this run
does not explain that change.
**Key insight:** at 10K records (typical agent memory), IVF search takes
**24.8 µs** against 500.5 µs for the flat scan. At 100K, IVF answers in
**592 µs** and IVF-PQ in 869 µs — both under 1 ms. (RAIRS, 124.0 µs at 10K, is
slower than plain IVF here.)
### Comparison to MemX (arxiv:2603.16171)
MemX claims end-to-end search under 90ms at 100K records (Rust + libSQL + FTS5).
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(simd_cosine_100k|ivf_pq_search_100k|bm25_search_10k)$'`
> **Caveat — not like-for-like.** MemX's `<90 ms` is *end-to-end* search across their
> full pipeline (dense embeddings + FTS5 + four-factor re-ranking). The clawhdf5
> figures below are a *single component* — raw vector search latency, excluding
> embedding, keyword, fusion, and re-ranking stages. A component measured against a
> full pipeline will always look favourable; the "speedup" column overstates the real
> advantage by an unquantified margin and should be read as an order-of-magnitude
> indication only, not a benchmark result. Matching MemX's measurement boundary is
> tracked as follow-up work.
> full pipeline will always look favourable, so the two columns below cannot be
> divided into a speedup: any such ratio would overstate the real advantage by an
> unquantified margin. Read the table as an order-of-magnitude indication only, not a
> benchmark result. Matching MemX's measurement boundary is tracked as follow-up work.
| Metric | MemX (claimed, end-to-end) | ClawhDF5 (component only) | Ratio |
|--------|----------------------------|---------------------------|-------|
| 100K flat search | <90 ms | 11.4 ms | ~8x |
| 100K IVF-PQ search | — | 1.19 ms | ~76x |
| Keyword search 10K | 1,100x improvement over unindexed | 583 µs (BM25) | Comparable |
| Metric | MemX (claimed, end-to-end) | ClawhDF5 (component only) |
|--------|----------------------------|---------------------------|
| 100K flat search | <90 ms | 6.64 ms |
| 100K IVF-PQ search | — | 876 µs |
| Keyword search 10K | 1,100x improvement over unindexed | 205 µs (BM25) |
The ratio column is gone: dividing an end-to-end claim by a component
timing produced a number that looked like a result and was not one. The
previous figures (100K flat 11.4 ms, IVF-PQ 1.19 ms, BM25 10K 583 µs) were
undated, from the original i7-12650H run. BM25 at 10K also moved on this
machine: 520 µs on 2026-08-05, 205 µs now; v2.4.0 changed
`BM25Index::search` in between (bounded-heap top-k, the unused WAND bound
removed, IDF per query), but this run does not isolate its effect.
---
## SIMD & Parallelism
384-dimensional cosine similarity at 10K scale.
384-dimensional cosine similarity at 10K scale, using the `strategy_*`
benchmarks, which hold the dataset fixed and vary only `SearchStrategy`
(the 2026-08-05 re-run below explains why the older table's benchmarks were
not an apples-to-apples comparison).
| Strategy | Latency | vs Sequential |
|----------|---------|---------------|
| Sequential (scalar) | 1.07 ms | 1.0x |
| SIMD (auto-vectorized) | 545 µs | **2.0x** |
| Rayon (parallel) | 553 µs | **1.9x** |
| Adaptive (auto-select) | 564 µs | **1.9x** |
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. Rayon
uses all 16 threads; the 1-minute load average was 4.19 when this command
finished, most likely from the Rayon benchmarks themselves (not verified).
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(strategy_scalar_10k|strategy_simd_10k|strategy_rayon_10k|adaptive_search_10k|simd_cosine_100k|rayon_cosine_100k)$'`
At 100K:
| Strategy | Latency |
|----------|---------|
| SIMD | 13.7 ms |
| Rayon parallel | 8.3 ms |
| Sequential (scalar), `strategy_scalar_10k` | 501.1 µs |
| SIMD, `strategy_simd_10k` | 319.9 µs |
| Rayon (parallel), `strategy_rayon_10k` | **79.0 µs** |
| Adaptive (auto-select), `adaptive_search_10k` | 317.9 µs |
At 100K (no `strategy_*` benchmark exists at this size, so these are the
`simd_cosine_100k` / `rayon_cosine_100k` benchmarks):
| Strategy | Latency |
|----------|---------|
| SIMD | 6.56 ms |
| Rayon parallel | 4.63 ms |
The Rayon 10K figure moved furthest: 323 µs on this machine on 2026-08-05,
79.0 µs now. The git log does not explain it. The other rows are close to
that run (scalar 502 µs, SIMD 327 µs, adaptive 339 µs; 100K 6.60 / 4.73 ms). Against the original, undated i7-12650H table (scalar 1.07 ms, SIMD
545 µs, Rayon 553 µs, adaptive 564 µs; 100K 13.7 / 8.3 ms) every figure is
lower; that table also used the mislabelled benchmarks. The older names
measured in the same session, for reference (`cargo bench -p clawhdf5-agent
--bench bench -- '^(sequential_cosine_10k|simd_cosine_10k|rayon_cosine_10k|adaptive_search_10k|simd_cosine_100k|rayon_cosine_100k)$'`):
`sequential_cosine_10k` 317.8 µs, `simd_cosine_10k` 505.2 µs,
`rayon_cosine_10k` 274.3 µs — "sequential" faster than "SIMD", which is the
mislabelling the 2026-08-05 re-run found.
---
## Hybrid Search (Vector + BM25)
1K records, 384-dimensional embeddings with BM25 keyword index.
1K records, 384-dimensional embeddings with BM25 keyword index. These are the
flat free functions `hybrid::hybrid_search` / `rrf_hybrid_search` (linear
scan), not the HNSW-backed `HDF5Memory::hybrid_search`, which is measured in
the [search harness](#current-search-harness-2026-09-24).
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^hybrid_search/'`
> and `cargo bench -p clawhdf5-agent --bench bench -- '^(bm25_search_1k|hybrid_search_10k)$'`
| Method | Latency | Notes |
|--------|---------|-------|
| Weighted fusion | 198 µs | Original min-max normalization |
| **RRF (k=60)** | **222 µs** | Reciprocal Rank Fusion — better quality, ~12% overhead |
| BM25-only 1K | 67 µs | Keyword search alone |
| Hybrid 10K | 2.04 ms | Full hybrid at 10K scale |
| Weighted fusion | 106.8 µs | Min-max normalization |
| **RRF (k=60)** | **130.5 µs** | Reciprocal Rank Fusion |
| BM25-only 1K | 20.4 µs | Keyword search alone |
| Hybrid 10K | 1.10 ms | Full hybrid at 10K scale |
Every row moved by more than 20% from the previous table (198 µs, 222 µs,
67 µs, 2.04 ms), which was undated and from the original i7-12650H run. The
cause is not isolated. (The old "better quality" note on RRF is dropped: on
LongMemEval RRF measured worse than the tuned weighted sum; see
[Fusion method](#fusion-method--weighted-vs-rrf-full-haystack-n500).)
---
@@ -665,16 +913,36 @@ At 100K:
Graph traversal and entity operations.
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^knowledge_graph/'`
> and `cargo bench -p clawhdf5-agent --bench bench -- '^alias_resolve_(short|long)_query$'`
> (the `knowledge_graph/` rows re-run after the adjacency-index fix, same day,
> load average 0.49 at start)
| Operation | Scale | Latency |
|-----------|-------|---------|
| BFS traversal | 100 entities | 5.4 µs |
| BFS traversal | 1,000 entities | 24 µs |
| Spreading activation | 100 entities | 16.9 µs |
| Entity resolution (Levenshtein) | 100 entities | 64 µs |
| Alias resolution (short query) | 100 aliases | 10.4 µs |
| Alias resolution (long query) | 100 aliases | 11.6 µs |
| BFS traversal | 100 entities | 5.23 µs |
| BFS traversal | 1,000 entities | 23.1 µs |
| Spreading activation | 100 entities | 10.1 µs |
| Entity resolution (Levenshtein) | 100 entities | 48.6 µs |
| Alias resolution (short query) | 100 aliases | 7.67 µs |
| Alias resolution (long query) | 100 aliases | 8.18 µs |
**All graph operations complete in microseconds.** Spreading activation across 100 entities with 5 propagation steps finishes in 17 µs.
The traversal rows were measured after a fix made during this re-run. The
first measurement of the day found **BFS 6.5x slower** than the old table
said: 17.5 µs and 155.1 µs, against 5.4 µs and 24 µs, with spreading
activation at 22.8 µs against 16.9 µs. The cause was 1efd82c (2026-08-17),
which built an adjacency index over the whole graph on every traversal — so a
2-hop BFS paid to index every entity and relation first. The index is now
cached on `KnowledgeCache` and checked against a fingerprint of the graph on
each use (one pass over entity ids and relation endpoints, no allocation), so
any change to the graph, including direct edits of its public `Vec`s, still
triggers a rebuild. These benches traverse an unchanged graph, which is the
cached case; the first traversal after a change pays for one build. Entity
and alias resolution do not use the index (the 49.8 µs first-run figure for
entity resolution is within noise of the 48.6 µs here). The old figures were
undated, from the original i7-12650H run.
---
@@ -682,13 +950,22 @@ Graph traversal and entity operations.
Hippocampal-inspired tiered memory management.
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^consolidation/'`
| Operation | Scale | Latency |
|-----------|-------|---------|
| Consolidation cycle | 100 records | 15 µs |
| Consolidation cycle | 1,000 records | 164 µs |
| Importance scoring | 100 records | 25 µs |
| Consolidation cycle | 100 records | 8.08 µs |
| Consolidation cycle | 1,000 records | 115.2 µs |
| Importance scoring | 100 records | 31.0 µs |
A full consolidation pass over 1,000 memories (eviction + promotion across Working → Episodic → Semantic) completes in **164 µs**. This can run on every memory write without perceptible latency.
A full consolidation pass over 1,000 memories (eviction + promotion across Working → Episodic → Semantic) completes in **115 µs**. This can run on every memory write without perceptible latency.
The cycle figures were 15 µs and 164 µs, and importance scoring 25 µs, all
undated from the original i7-12650H run. Eviction's membership check became a
`HashSet` lookup (603fcf8) and `add_memory` stopped cloning Working-tier
records (d787ac0), both on 2026-08-17; this run does not isolate their effect.
---
@@ -696,12 +973,17 @@ A full consolidation pass over 1,000 memories (eviction + promotion across Worki
Sorted timestamp index with binary search.
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^temporal/'`
| Operation | Scale | Latency |
|-----------|-------|---------|
| Range query | 10K timestamps | **716 ns** |
| Batch insert | 10K timestamps | 4.69 ms |
| Range query | 10K timestamps | **622 ns** |
| Batch insert | 10K timestamps | 3.22 ms |
Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K records: **716 nanoseconds.**
Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K records: **622 nanoseconds.**
Batch insert was 4.69 ms in the previous, undated i7-12650H figure.
---
@@ -709,16 +991,50 @@ Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K
HDF5 persistence with optional Write-Ahead Log.
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. The stores
use `MemoryConfig::new`'s defaults at this commit: `float16` embeddings and
the int8 index.
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(save_without_wal_single|save_with_wal_single|save_batch_100|save_batch_1000|save_wal_1k_existing|wal_flush_100_entries|tick_session_1k|tick_session_10k)$'`
| Operation | Latency | Notes |
|-----------|---------|-------|
| Single save (no WAL) | 61 µs | Direct HDF5 write (owned-Vec IO path) |
| Single save (with WAL) | 18 µs | WAL group-commit append; HDF5 write batched at flush |
| Batch 100 | 723 µs | 7.2 µs per record |
| Batch 1,000 | 6.17 ms | 6.2 µs per record |
| WAL save (1K existing) | 539 µs | Incremental append |
| WAL flush 100 entries | 787 µs | Merge WAL → HDF5 |
| Session tick 1K | 5.76 ms | Full session maintenance |
| Session tick 10K | 89.8 ms | Background operation |
| Single save (no WAL) | 88.3 µs | Direct HDF5 write (owned-Vec IO path) |
| Single save (with WAL) | 26.1 µs | WAL group-commit append; HDF5 write batched at flush |
| Batch 100 | 1.08 ms | |
| Batch 1,000 | 31.7 ms | |
| WAL save (1K existing) | 282.8 µs | Incremental append |
| WAL flush 100 entries | 530.1 µs | Merge WAL → HDF5 |
| Session tick 1K | 3.83 ms | Full session maintenance |
| Session tick 10K | 35.5 ms | Background operation |
Almost every row moved by more than 20% from the previous table, in both
directions. Slower: single save 61 -> 88.3 µs (no WAL) and 18 -> 26.1 µs
(WAL), batch 100 723 µs -> 1.08 ms, and **batch 1,000 6.17 -> 31.7 ms**.
Faster: WAL save into 1K existing 539 -> 282.8 µs, WAL flush 787 -> 530.1 µs,
session tick 5.76 -> 3.83 ms (1K) and 89.8 -> 35.5 ms (10K). The two single-save
figures were last updated on 2026-07-01 and the rest are undated, all on the
i7-12650H with `f32` stores.
**Isolated afterwards, same machine, same day** (`--warm-up-time 1
--measurement-time 3`, the store settings switched through the bench's
config):
| | f16 + int8 (default) | f32 + int8 | f16 + f32 index | f32 + f32 index | v2.3.0 (f32) |
|---|---:|---:|---:|---:|---:|
| Single save (WAL) | 26.1 µs | 24.3 µs | 26.1 µs | 24.4 µs | 24.3 µs |
| Single save (no WAL) | 89.0 µs | 87.3 µs | 88.8 µs | 87.4 µs | 82.6 µs |
| Batch 100 | 1.09 ms | 0.90 ms | 1.09 ms | 0.90 ms | 0.87 ms |
| Batch 1,000 | 31.5 ms | 31.1 ms | 31.5 ms | 31.6 ms | 30.3 ms |
The int8 index costs nothing on the write path. `float16` costs ~2 µs per
saved record — the rounding — which is the whole difference at 100 records
and within noise at 1,000. Built at the v2.3.0 tag (v2.2.0's benches do not
compile) and run on this machine, the write path measures the same as today,
so nothing regressed since then; the old 18 µs and 6.17 ms figures cannot be
reproduced on this hardware and came from an undated run on another machine.
`save_batch` scales linearly (about 12 µs per record from 500 to 8 000
records); the bench's figure also includes dropping the store.
---
@@ -726,12 +1042,16 @@ HDF5 persistence with optional Write-Ahead Log.
Trivial/non-trivial classification for memory write filtering.
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^gate_'`
| Check | Latency |
|-------|---------|
| Trivial skip ("ok", "yes") | 61 ns |
| Short phrase skip | 86 ns |
| Non-trivial pass | 705 ns |
| Ratio check | 488 ns |
| Trivial skip ("ok", "yes") | 57.6 ns |
| Short phrase skip | 83.0 ns |
| Non-trivial pass | 570.5 ns |
| Ratio check | 395.2 ns |
**Sub-microsecond filtering.** The gate decides whether to save a memory in under 1 µs.
@@ -741,27 +1061,70 @@ Trivial/non-trivial classification for memory write filtering.
End-to-end strategy evaluation including embedding operations.
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^strategy_(save_every|semantic_shift)'`
| Strategy | Condition | Latency |
|----------|-----------|---------|
| SaveEveryExchange (substantive) | Saves | 923 ns |
| SaveEveryExchange (trivial) | Skips | 67 ns |
| SaveOnSemanticShift (empty store) | Saves | 941 ns |
| SaveEveryExchange (substantive) | Saves | 732.5 ns |
| SaveEveryExchange (trivial) | Skips | 58.3 ns |
| SaveOnSemanticShift (empty store) | Saves | 734.9 ns |
The two "Saves" rows were 923 ns and 941 ns in the previous table, undated
and from the original i7-12650H run.
---
## Multi-modal Search
`clawhdf5_agent::multimodal::MultiModalStore`. Each of N records carries two
384-dim embeddings: a text embedding of its caption and one of its primary
modality, cycling image / audio / video. `search_cross_modal` scores every
embedding of every record (2N vectors) and keeps each record's best;
`search_by_modality(Image, …)` scores only image embeddings (a third of the
records). Both are exact linear scans followed by a full sort; k = 10. Data is
from a fixed-seed generator, so every run sees the same corpus.
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c (with the
new, uncommitted `crates/clawhdf5-agent/benches/multimodal_bench.rs` in the
working tree). Criterion, 50 samples.
> **Run:** `cargo bench -p clawhdf5-agent --bench multimodal_bench`
| Search | 1K records | 10K records |
|--------|-----------:|------------:|
| Cross-modal (all modalities, 2N vectors) | 842.0 µs | 8.44 ms |
| One modality (image, ~N/3 vectors) | 151.2 µs | 1.54 ms |
Both scale linearly with N. Cross-modal search is the slow path: at 1K
records it takes 842.0 µs, where the flat single-vector cosine scan in
[Vector Search Latency](#vector-search-latency) takes 47.4 µs over 1K vectors,
and it has no index — at 10K it is 8.44 ms, slower than a 100K flat scan
(6.58 ms). No earlier figure exists for this section.
---
## Summary
Derived from the 2026-09-24 tables above (tank, AMD Ryzen 7 7800X3D, commit
5c8323c); no command of its own.
| Capability | Typical Latency | Scale |
|------------|----------------|-------|
| **Full memory search** | <1 ms | 10K records |
| **Hybrid vector+keyword** | <200 µs | 1K records |
| **Knowledge graph query** | <25 µs | 1K entities |
| **Temporal range query** | <1 µs | 10K timestamps |
| **Memory write** | <20 µs | Per record (WAL group-commit append) |
| **Consolidation cycle** | <165 µs | 1K records |
| **Importance gate** | <1 µs | Per record |
| **Full memory search** (`HDF5Memory::hybrid_search`, p50) | 0.49 ms | 10K records |
| **Hybrid vector+keyword** (flat, weighted) | 106.8 µs | 1K records |
| **Knowledge graph query** (BFS) | 23.1 µs | 1K entities |
| **Temporal range query** | 622 ns | 10K timestamps |
| **Memory write** | 26.1 µs | Per record (WAL group-commit append) |
| **Consolidation cycle** | 115.2 µs | 1K records |
| **Importance gate** | 57.6–570.5 ns | Per record |
**The entire memory pipeline — search, retrieve, re-rank, filter — runs in single-digit milliseconds at agent-typical scales. Fast enough that memory becomes invisible infrastructure.**
The previous summary said "<25 µs" for a knowledge-graph query and "<20 µs"
for a memory write. The first holds again after the adjacency-index fix
(23.1 µs); the second does not (26.1 µs).
**At agent-typical scales a full `HDF5Memory::hybrid_search` (vector + BM25 + fusion) over 10K records takes 0.49 ms at p50, and graph traversal, temporal queries, consolidation and the write gate each take well under a millisecond. At 100K records the hybrid search takes 4.69 ms.**
---
@@ -796,9 +1159,21 @@ _Latency benchmarks generated with Criterion.rs (50-100 samples per benchmark).
**Mode:** BM25-only retrieval — zero embeddings, `vector_weight=0.0`, `keyword_weight=1.0`
> **Run:** `cargo run --release --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s_cleaned.json`
> (~70 s for all 500 questions on the tank reference machine). Omit the path for the
> oracle variant; add `--limit N` for an evenly-strided subsample.
> **Run:** `cargo run --release -p clawhdf5-bench --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s.json`
> Omit the path for the oracle variant; add `--limit N` for an evenly-strided
> subsample. Without the `embeddings` feature this runs two modes, BM25 only
> and BM25 stemmed.
The BM25-only and BM25-stemmed figures in this section were measured again on
2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, with the command
above. Every figure of those two modes that this document publishes was
reproduced exactly: BM25 Hit@1/5/10 and MRR at turn and session level, the
BM25 per-type session Hit@1 values quoted in prose below, and the stemmed
turn-level row plus its session Hit@1 in the tokenizer table. The run also
produced figures this document does not publish (stemmed session Hit@5,
Hit@10 and MRR, and per-type Hit@5/Hit@10/MRR for both modes), so there was
nothing to compare them with. Rows that need real embeddings (vector-only,
hybrid, RRF, re-ranking, the weight sweep) were not re-run.
### Full haystack — `longmemeval_s`, n=500 (the number to cite)
@@ -1058,14 +1433,29 @@ extraction over the haystack, which this harness does not do.
### Search Latency (LongMemEval, n=500 queries)
| Metric | Latency |
|--------|---------|
| avg | 1,004 µs |
| p50 | 1,017 µs |
| p95 | 2,031 µs |
| p99 | 2,912 µs |
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, full
`longmemeval_s` haystack (47.7 sessions / 494 turns per question).
Sub-millisecond median search across variable-length chat histories.
> **Run:** `cargo run --release -p clawhdf5-bench --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s.json`
| Metric | BM25 only | BM25 stemmed |
|--------|-----------|--------------|
| avg | 11,034.7 µs | 11,096.2 µs |
| p50 | 10,943.5 µs | 10,996.5 µs |
| p95 | 11,749.6 µs | 12,018.8 µs |
| p99 | 12,644.1 µs | 12,784.7 µs |
About 11 ms per query. Each question gets its own store, and the timed call is
the first `hybrid_search_with` on that store. The first query on a store
builds its indexes (the "cold index build" column of the search harness), so
this is a first-query figure, not steady-state search; for that
see the [search harness](#current-search-harness-2026-09-24).
The previous table (avg 1,004 µs, p50 1,017 µs) said "sub-millisecond median
search", which no longer holds. That figure was undated and predates the
full-haystack harness (added 2026-08-07, 7d6e269), so it was measured on the
much smaller `longmemeval_oracle` corpus; on that corpus tank measured avg
2,431 µs on 2026-08-05 (below).
---
@@ -1075,23 +1465,34 @@ Sub-millisecond median search across variable-length chat histories.
**Topics:** Personal info, food preferences, music, travel, work/schedule, hobbies
**Queries:** 35 questions across 4 types
> **Run:** `cargo run --release --bin memory_arena`
> **Run:** `cargo run --release -p clawhdf5-bench --bin memory_arena`
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
### Results by Query Type
| Query Type | N | Hit@1 | Hit@5 | Hit@10 | MRR | Avg Latency |
|------------|---|-------|-------|--------|-----|-------------|
| single-session | 25 | 40.0% | 92.0% | 100.0% | 0.5788 | 7,853 µs |
| multi-session | 5 | 40.0% | 60.0% | 80.0% | 0.5333 | 7,887 µs |
| temporal | 3 | 33.3% | 66.7% | 66.7% | 0.5000 | 7,899 µs |
| knowledge-update | 2 | 0.0% | 50.0% | 50.0% | 0.2500 | 7,899 µs |
| **OVERALL** | **35** | **37.1%** | **82.9%** | **91.4%** | **0.5468** | **7,870 µs** |
| single-session | 25 | 40.0% | 92.0% | 100.0% | 0.5755 | 248.4 µs |
| multi-session | 5 | 40.0% | 60.0% | 80.0% | 0.5333 | 24.4 µs |
| temporal | 3 | 33.3% | 66.7% | 66.7% | 0.5000 | 28.0 µs |
| knowledge-update | 2 | 0.0% | 50.0% | 50.0% | 0.2500 | 38.7 µs |
| **OVERALL** | **35** | **37.1%** | **82.9%** | **91.4%** | **0.5444** | p50 **30.1 µs**, p95 49.8 µs |
**Key findings:**
- Hit@10 of 91.4% across all query types with BM25-only (no embeddings)
- Single-session recall strongest at 100% Hit@10
- Knowledge-update hardest (requires temporal disambiguation) — would improve significantly with vector similarity
- Latency dominated by BM25 index build over 999 turns (~7.9 ms)
- Search takes about 30 µs at p50 over the 999 turns. All 35 queries run
against one store, and single-session's higher average (248.4 µs) includes
that store's first query, which builds its indexes.
The previous table put every query at ~7.9 ms ("latency dominated by BM25
index build over 999 turns"). That was before v2.4.0, when
`HDF5Memory::hybrid_search` rebuilt the BM25 index from scratch and rewrote
the store on every query; it now keeps the index for the life of the store.
The MRR moved slightly (single-session 0.5788 -> 0.5755, overall 0.5468 ->
0.5444); the Hit@k figures are unchanged.
---
@@ -1099,46 +1500,89 @@ Sub-millisecond median search across variable-length chat histories.
HDF5 file size at various record counts — 384-dimensional embeddings, 200-char text.
> **Run:** `cargo run --release --bin footprint_bench`
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
### Uncompressed (no WAL)
> **Run:** `cargo run --release -p clawhdf5-bench --bin footprint_bench`
| Records | File Size | Raw Data | Bytes/Record | Throughput |
|---------|-----------|----------|--------------|------------|
| 100 | 176.4 KB | 169.5 KB | 1.8 KB | 100,000 rec/s |
| 1K | 1.7 MB | 1.7 MB | 1.8 KB | 109,643 rec/s |
| 10K | 17.0 MB | 16.6 MB | 1.7 KB | 118,100 rec/s |
| 50K | 85.0 MB | 82.8 MB | 1.7 KB | 110,723 rec/s |
| 100K | 169.8 MB | 165.6 MB | 1.7 KB | 111,422 rec/s |
**These are `float16` stores.** `footprint_bench` builds its stores with
`MemoryConfig::new`, which defaults to `float16` embeddings since 5c8323c,
although the binary still prints "Embedding: 384-dim f32". Its "Raw Data"
column, and so the ratios, count embeddings as `f32` (1,536 bytes per record)
plus text. The synthetic embeddings take only 1,000 distinct values, and each
record's text is fixed by `record_idx % 40` (a 40-word bank walked from that
start), so a store holds only 40 distinct texts. Both compress far better than
real data would; read the gzip figures as an upper bound, and see the next
paragraph for why the other tables are optimistic too. "MB" and "KB" are the
binary's units (powers of 1024).
**1.7 KB per record** — HDF5 overhead is near-zero. Ingestion throughput exceeds **100K records/sec**.
**No table here is uncompressed.** The store always deflates its string
datasets: `write_string_dataset` applies deflate level 6 to any string dataset
of 4 KiB or more, whatever `MemoryConfig::compression` says
(`crates/clawhdf5-agent/src/schema.rs`, `STRING_COMPRESS_THRESHOLD`, added in
4fa7e89 on 2026-06-03). With only 40 distinct texts, the chunk text costs
almost nothing on disk: at 10K records, 50-char text takes 804 B per record
and 1000-char text 944 B, so 950 more characters cost only 140 B. The
`float16` embeddings alone are 768 B per record, and 200 bytes of raw text
would bring that to about 968 B before any other field or overhead; the
measured 803–829 B is below that only because the text is compressed away.
Real text will not deflate like this, so expect more than 820 B per record
with real data. `MemoryConfig::compression` decides only whether the
embeddings are compressed; the gzip table below is that setting turned on.
### `compression` off (no WAL)
| Records | File Size | Raw Data (f32) | Bytes/Record | Throughput |
|---------|-----------|----------------|--------------|------------|
| 100 | 90.0 KB | 169.5 KB | 921 B | 75,914 rec/s |
| 1K | 810.4 KB | 1.7 MB | 829 B | 72,266 rec/s |
| 10K | 7.8 MB | 16.6 MB | 820 B | 74,509 rec/s |
| 50K | 38.3 MB | 82.8 MB | 803 B | 72,829 rec/s |
| 100K | 76.7 MB | 165.6 MB | 803 B | 71,102 rec/s |
**About 820 B per record** at 10K, for this synthetic corpus (see above: the
text is deflated regardless and has only 40 distinct values, so real data
takes more). Ingestion runs at 71,102–75,914 records/sec.
### With Gzip Compression (level 6)
| Records | Compressed | Ratio | Bytes/Record |
|---------|------------|-------|--------------|
| 100 | 31.5 KB | 5.37x | 323 B |
| 1K | 277.1 KB | 6.12x | 283 B |
| 10K | 2.7 MB | 6.17x | 281 B |
| 50K | 13.4 MB | 6.17x | 281 B |
| 100K | 26.9 MB | 6.15x | 282 B |
| Records | Compressed | Ratio vs f32 raw | Bytes/Record |
|---------|------------|------------------|--------------|
| 100 | 17.2 KB | 9.85x | 176 B |
| 1K | 56.4 KB | 30.06x | 57 B |
| 10K | 471.3 KB | 35.97x | 48 B |
| 50K | 2.3 MB | 36.67x | 47 B |
| 100K | 4.5 MB | 36.73x | 47 B |
**6.2x compression ratio** — 100K agent memories in 27 MB compressed.
### Text Length Comparison (10K records, no compression)
### Text Length Comparison (10K records, `compression` off)
| Text Length | File Size | Bytes/Record | Throughput |
|-------------|-----------|--------------|------------|
| short (50 chars) | 15.6 MB | 1.6 KB | 177,925 rec/s |
| medium (200 chars) | 17.0 MB | 1.7 KB | 172,152 rec/s |
| long (1000 chars) | 24.6 MB | 2.5 KB | 157,807 rec/s |
| short (50 chars) | 7.7 MB | 804 B | 97,615 rec/s |
| medium (200 chars) | 7.8 MB | 820 B | 89,597 rec/s |
| long (1000 chars) | 9.0 MB | 944 B | 51,462 rec/s |
### WAL Overhead (1K records)
| Mode | File Size | Ingest Time | Overhead |
|------|-----------|-------------|----------|
| No WAL | 1.7 MB | 5.7 ms | — |
| With WAL | 1.7 MB + 9 B WAL | 5.3 ms | ±8% (negligible) |
| No WAL | 810.4 KB | 10.8 ms | — |
| With WAL | 810.4 KB + 9 B WAL | 10.6 ms | −2% (negligible) |
**Compared with the previous tables.** Those were undated `f32` stores from the
original i7-12650H run: 1.7 KB per record (169.8 MB at 100K), gzip 6.2x
(26.9 MB at 100K), and more than 100,000 records/sec. Two changes separate
those tables from these, not one. `float16` embeddings save 768 B per record.
The old tables (recorded by 2026-05-14) also predate 4fa7e89 (2026-06-03),
which began deflating string datasets of 4 KiB or more, so their text was
stored raw; with only 40 distinct texts, deflate now removes most of it.
An `f32` embedding plus 200 B of text is 1,536 + 200 = 1,736 B; `float16`
alone would bring that to about 968 B per record, not the measured 803–829 B. This run does not measure how the saving splits between the two
beyond that, nor explain why the gzip ratio rose so much further, nor
separate the lower throughput (and the 1K ingest going from 5.7 to 10.8 ms)
from the change of machine and store configuration. The
`f32` on-disk figures were not re-measured; the dated
[float16 study](#float16-embedding-storage-memoryconfigfloat16) compares the
two directly (100K × 384: 154.0 MiB `f32`, 80.8 MiB `float16`).
---
@@ -1146,7 +1590,12 @@ HDF5 file size at various record counts — 384-dimensional embeddings, 200-char
Hippocampal-inspired memory consolidation improves both retrieval quality and search speed.
> **Run:** `cargo run --release --bin consolidation_efficiency`
> **Run:** `cargo run --release -p clawhdf5-bench --bin consolidation_efficiency`
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. The run
was stopped (after about 19 minutes on one core) while still computing a
100K row of the cycle-time table, so that row and the binary's memory-reduction
part were not produced; neither has ever been published here.
### Retrieval Quality Before vs. After Consolidation
@@ -1159,17 +1608,22 @@ Hippocampal-inspired memory consolidation improves both retrieval quality and se
| Hit@5 | 100.0% | 100.0% | — |
| Hit@10 | 100.0% | 100.0% | — |
| MRR | 1.0000 | 1.0000 | — |
| Search latency | 2,752 µs | 312 µs | **8.8x faster** |
| Search latency (avg) | 2,223.7 µs | 239.2 µs | **9.3x faster** |
Signal records survive consolidation because they are accessed 15+ times, giving them high decay scores. 900 noise records evicted, search speeds up 8.8x, and **zero quality loss** — perfect recall maintained.
Signal records survive consolidation because they are accessed 15+ times, giving them high decay scores. 900 noise records evicted, search speeds up 9.3x, and **zero quality loss** — perfect recall maintained. The consolidation cycle itself took 0.13 ms and promoted 2 records.
### Consolidation Cycle Time
| Records | Cycle Time | Evictions | Promotions |
|---------|-----------|-----------|------------|
| 100 | 21 µs | 100 | 0 |
| 1K | 345 µs | 1,000 | 0 |
| 10K | 17.3 ms | 10,000 | 0 |
| 100 | 17 µs | 100 | 0 |
| 1K | 189 µs | 1,000 | 0 |
| 10K | 2.16 ms | 10,000 | 0 |
The 1K and 10K cycles were 345 µs and 17.3 ms in the previous, undated table
(before: 2,752 µs / after: 312 µs for search). Eviction's membership check
changed from a linear scan to a `HashSet` in 603fcf8 (2026-08-17), after those
figures were recorded; this run does not isolate its effect.
---
@@ -1178,34 +1632,44 @@ Signal records survive consolidation because they are accessed 15+ times, giving
In-memory key-value store with TTL, capacity eviction, and embedding search.
No network hop, no serialization — direct HashMap operations.
> **Run:** `cargo run --release --bin ephemeral_perf`
> **Run:** `cargo run --release -p clawhdf5-bench --bin ephemeral_perf`
Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
### Latency Comparison
| Operation | clawhdf5 Ephemeral | Redis (single-node)¹ | Speedup |
| Operation | clawhdf5 Ephemeral | Redis (single-node)¹ | Ratio¹ |
|-----------|-------------------|---------------------|---------|
| SET | **356 ns/op** | ~25,000 ns/op | **70x** |
| GET (hit) | **179 ns/op** | ~25,000 ns/op | **140x** |
| GET (miss) | **62 ns/op** | ~25,000 ns/op | **403x** |
| DELETE | **124 ns/op** | ~25,000 ns/op | **202x** |
| SET+embedding | **268 ns/op** | N/A | — |
| SET | **366 ns/op** | ~25,000 ns/op (not measured) | 68.3x |
| GET (hit) | **115 ns/op** | ~25,000 ns/op (not measured) | 216.7x |
| GET (miss) | **57 ns/op** | — | — |
| DELETE | **99 ns/op** | — | — |
| SET+embedding | **236 ns/op** | N/A | — |
> ¹ Redis latency includes network round-trip (loopback). clawhdf5 ephemeral is in-process — no network.
> ¹ **The Redis figure is not a measurement.** It is a constant hard-coded in
> `ephemeral_perf` as a "typical single-node" latency, and the binary computes
> the two ratios from it; only the clawhdf5 column was measured. A typical
> Redis figure includes a network round-trip, which the in-process ephemeral
> tier does not have, so the ratios compare different things.
### Throughput
| Operation | ops/sec |
|-----------|---------|
| SET | 2,810,649 |
| GET | 5,584,684 |
| DELETE | 8,093,731 |
| SET+EMB (384d) | 3,725,877 |
| SET | 2,732,568 |
| GET (hit) | 8,666,607 |
| GET (miss) | 17,554,663 |
| DELETE | 10,109,895 |
| SET+EMB (384d) | 4,235,766 |
### Embedding Search (ephemeral tier)
| Scale | Latency |
|-------|---------|
| 10K entries @ 384d | 2.9 ms/query |
| 10K entries @ 384d | 2.85 ms/query |
GET (hit) moved from 179 to 115 ns/op and DELETE from 124 to 99 ns/op (GET
throughput 5,584,684 -> 8,666,607 ops/sec); the previous figures were undated.
---
@@ -1263,10 +1727,16 @@ Measured 2026-08-07 on tank (Ryzen 7 7800X3D, 246 MB dataset in page cache).
### Measured Platforms
The tank row was measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit
5c8323c, with the single benchmark rather than the whole script:
> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^ivf_search_10k_nprobe10$'`
| Platform | CPU | 10K IVF Search | Notes |
|----------|-----|----------------|-------|
| Linux x86_64 | Intel i7-12650H (10C, 4.7 GHz) | 27 µs | Primary CI target |
| macOS aarch64 | Apple M3 Max (14C) | ~18 µs | ~33% faster via NEON SIMD |
| Linux x86_64 | AMD Ryzen 7 7800X3D (8C/16T), tank | 24.8 µs | Measured 2026-09-24 |
| Linux x86_64 | Intel i7-12650H (10C, 4.7 GHz) | 27 µs | Undated, original run |
| macOS aarch64 | Apple M3 Max (14C) | ~18 µs | Undated; ~33% faster than the i7-12650H row. Not reproducible on the hardware available for the 2026-09-24 re-run |
### Reproducibility
@@ -1610,6 +2080,11 @@ vs-libhdf5 validation above) to give both sections the dated, hardware-cited,
reproducible citation the top-of-file traceability note flags them as
missing.
> The sections this re-run checked were themselves re-measured on tank on
> 2026-09-24 and now show those figures; the i7-12650H figures this section
> compares against ("above", "at the top of this file") are quoted in those
> sections' notes. Kept as the 2026-08-05 record.
### LongMemEval Results (reproduction)
```bash
CHANGELOG.md
+51
View File
@@ -3,6 +3,17 @@
## Unreleased
### Upgrade Notes
- **`clawhdf5-migrate` now writes a real agent store.** Its output used to be
a layout of its own (`/chunks`, `/sessions`, `/entities`, `/relations`, no
`/meta`) that `HDF5Memory::open` rejected, so a migrated file could not be
used as agent memory. Files it wrote before this release are not agent
stores; re-run the migration. Also: embeddings default to `float16` like
any new store (`--f32` opts out; `--float16` is a hidden no-op); a row with
the wrong embedding length is an error instead of being truncated or
padded; `--incremental` now matches rows by content against an existing
store and follows the source's deleted flags; a source with no memory rows
needs `--embedding-dim`. The per-dataset SHA-256 provenance attributes of
the old layout are gone (the agent schema has no place for them).
- **Files written by clawhdf5 now open in h5py and libhdf5.** Every `f32`
dataset we wrote — including every agent store's embeddings — was refused
with "sign bit position out of bounds", and every empty dataset with
@@ -46,6 +57,24 @@
`quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
out. The CLI's `--quantized-index` is still accepted but is now a no-op.
### Migration
- `clawhdf5-migrate`: writes through the agent's own API (`HDF5Memory::create`
/ `open`, `save_batch`, the session cache and knowledge graph), so there is
no second copy of the schema. Sessions and entities/relations carry over;
deleted rows become deleted records (or are left out with
`--skip-deleted`). Every source row is checked before the output is created,
so a source that cannot be migrated leaves an existing store untouched.
Validation reads the result back with `HDF5Memory::open_read_only`, compares
every field (embeddings bit for bit — `round_to_f16` of the source for a
`float16` store) and checks that a migrated record is found by search. The
`half`-based conversion is gone; `clawhdf5_format::float16` is the only one.
42 tests, including h5py opening a migrated store; an adversarial review's
two blocker and four major findings are fixed with regression tests.
- `clawhdf5-agent`: `HDF5Memory::sessions()` / `sessions_mut()`,
`HDF5Memory::delete_batch(&[usize])` (one save, all-or-nothing, never
auto-compacts), `SessionCache::add_at`, and `SessionCache` / `SessionEntry`
re-exported from the crate root.
### Search
- `clawhdf5-agent`: **`HDF5Memory::search` with `SearchOptions`** — source
filtering, re-ranking and confidence rejection in the store's own search
@@ -68,6 +97,20 @@
activation of the `k` results it returns, not of the whole `3k` candidate
pool it re-ranks.
### Benchmarks
- Every undated or pre-September section of `BENCHMARKS.md` re-run on one
machine on one day (tank, 2026-09-24, commit 5c8323c), with the command for
each and every number traced back to the raw output by a separate check.
Where a figure moved, the section says so. Two apparent regressions were
isolated rather than published: knowledge-graph traversal (a real bug,
fixed above) and the write path, which measures the same at v2.3.0 on this
machine — the old 18 µs / 6.17 ms figures came from an undated run on other
hardware; `float16` adds ~2 µs per save and the int8 index nothing.
- New `multimodal_bench`: cross-modal search at 1K and 10K records, which the
README claimed but nothing measured.
- `footprint_bench` reports whether it built `float16` or `f32` stores and
takes `--f32`; it had kept printing "f32" after the default changed.
### Interop
- `clawhdf5-format`: **every `f32` dataset was unreadable by h5py and
libhdf5.** The float datatype encoder hard-coded the sign bit's position to
@@ -146,6 +189,14 @@
knew to ask; it now only ever switches the default off.
### Performance
- `clawhdf5-agent`: **knowledge-graph traversal was 6.5x slower than it
should be.** `bfs_neighbors` and `spreading_activation` built an adjacency
index over the whole graph on every call (1efd82c), so a 2-hop BFS over 1K
entities took 155 µs. The index is now cached on `KnowledgeCache` and
checked against a fingerprint of the graph on each use — one pass over
entity ids and relation endpoints, no allocation — so any change, including
direct edits of its public `Vec`s, still rebuilds it (tested). BFS over 1K
entities: 155.1 -> 23.1 µs; spreading activation over 100: 22.8 -> 10.1 µs.
- `clawhdf5-format`, `clawhdf5-filters`: both deflate paths hand the codec the
whole chunk in one call, into a buffer allocated once, instead of streaming
it through a 32 KiB buffer: about 5% on chunked writes and 10% on zlib-ng's
README.md
+85 -33
View File
@@ -6,7 +6,7 @@
[![Rust](https://img.shields.io/badge/rust-1.92%2B-orange.svg)](https://www.rust-lang.org)
[![Tests](https://img.shields.io/badge/tests-1850%2B-brightgreen.svg)](#building)
[![LongMemEval](https://img.shields.io/badge/LongMemEval__s-Turn--Level%20Hit@5%2081.4%25%20hybrid-blue.svg)](BENCHMARKS.md#longmemeval-results)
[![Footprint](https://img.shields.io/badge/on--disk-1.7%20KB%2Frecord-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1)
[![Footprint](https://img.shields.io/badge/on--disk-~820%20B%2Frecord%20float16%2C%20synthetic%20text-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1)
ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file.
@@ -111,8 +111,8 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d
| Keyword search | Separate FTS engine | Integrated BM25 |
| Knowledge graph | Neo4j or none | In-file graph with spreading activation |
| Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers |
| Temporal queries | Custom code | Native temporal index (716ns) |
| Multi-modal | Multiple stores | Unified cross-modal search |
| Temporal queries | Custom code | Native temporal index (622 ns range query over 10K) |
| Multi-modal | Multiple stores | Unified cross-modal search (exact scan: 842 µs over 1K records) |
| Integrity | Hope for the best | Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance |
| Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** |
@@ -120,7 +120,7 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d
## Performance
Vector search and agent-memory operations below are benchmarked on Intel i7-12650H (10C/16T), 384-dim embeddings, Criterion.rs. The HDF5 Core I/O table immediately below is from a separate, independently reproduced run (see its own hardware note).
The brute-force/IVF vector search, agent-memory, on-disk footprint and consolidation figures below were measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D, 8C/16T), commit 5c8323c, 384-dim embeddings; the commands are in [BENCHMARKS.md](BENCHMARKS.md). Exceptions are marked where they appear: the HDF5 Core I/O table immediately below is from a separate, independently reproduced run (see its own hardware note), and the HNSW `f32`/`i8` table and the in-memory `i8` column were not re-measured on 2026-09-24.
### HDF5 Core I/O (vs libhdf5 1.14.6)
@@ -154,33 +154,45 @@ and [§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quant
| `f32` index | 0.9945 | 13 399 | 3.2 s |
| `i8` index + exact re-score (**default for new stores**) | 0.9940 | **21 848** | **1.8 s** |
Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31.
Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31. These
two rows are a paired comparison (medians of alternating runs, same binary).
A single `f32` run on 2026-09-24 measured recall 0.9945, 19 001 QPS and a
2.7 s build; the int8 row was not re-run, so the pair has not been re-checked
([§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index)).
**Brute-force and IVF paths** (Criterion, i7-12650H):
**Brute-force and IVF paths** (Criterion, tank, 2026-09-24):
| Scale | Flat | IVF (nprobe=10) | IVF-PQ | vs MemX¹ |
| Scale | Flat | IVF (nprobe=10) | IVF-PQ | MemX¹ (claimed, end-to-end) |
|-------|------|-----------------|--------|----------|
| 1K | **54 µs** | — | — | — |
| 10K | 753 µs | **27 µs** | — | — |
| 100K | 11.4 ms | 1.32 ms | **1.19 ms** | ~8–76× (see caveat) |
| 1K | **47.4 µs** | — | — | — |
| 10K | 500.5 µs | **24.8 µs** | — | — |
| 100K | 6.58 ms | 592 µs | **869 µs** | <90 ms |
> Reproduced on the same second machine (Ryzen 7 7800X3D) with a corrected,
> apples-to-apples SIMD/scalar/parallel comparison methodology — see
> [BENCHMARKS.md § Independent Validation: tank — LongMemEval & Vector
> Search](BENCHMARKS.md#independent-validation-tank--longmemeval--vector-search-ryzen-7-7800x3d-2026-08-05).
> These replace figures from the original i7-12650H run (flat 54 µs / 753 µs /
> 11.4 ms); a 2026-08-05 run on tank had already matched the new ones — see
> [BENCHMARKS.md § Vector Search Latency](BENCHMARKS.md#vector-search-latency).
### Agent Memory Operations
| Operation | Latency | Scale |
|-----------|---------|-------|
| Hybrid search (`HDF5Memory::hybrid_search`, p50) | **70 µs** / 0.49 ms / 4.65 ms | 1K / 10K / 100K records |
| BM25 keyword search | **67 µs** | 1K records |
| Knowledge graph BFS | **24 µs** | 1K entities |
| Spreading activation | **17 µs** | 100 entities |
| Temporal range query | **716 ns** | 10K timestamps |
| Consolidation cycle | **164 µs** | 1K records |
| Memory write (WAL) | **18 µs** | per record (group-commit append; HDF5 batched at flush) |
| Importance gate | **61 ns** | per record |
| Hybrid search (`HDF5Memory::hybrid_search`, p50) | **0.07 ms** / 0.49 ms / 4.69 ms | 1K / 10K / 100K records |
| BM25 keyword search | **20.4 µs** | 1K records |
| Knowledge graph BFS | **23.1 µs** | 1K entities |
| Spreading activation | **10.1 µs** | 100 entities |
| Temporal range query | **622 ns** | 10K timestamps |
| Consolidation cycle | **115.2 µs** | 1K records |
| Cross-modal search (exact scan, 2 embeddings per record) | **842.0 µs** / 8.44 ms | 1K / 10K records |
| Memory write (WAL) | **26.1 µs** | per record (group-commit append; HDF5 batched at flush) |
| Importance gate | **57.6 ns** | per record (trivial skip) |
The old 18 µs WAL write was undated, from another machine: v2.3.0 measures
24.3 µs on the same hardware as this table, the same as an `f32` store today.
`float16` stores (the new default) add ~2 µs for rounding; the int8 index adds
nothing. See [BENCHMARKS.md § Write Path](BENCHMARKS.md#write-path).
Knowledge-graph traversal was briefly 6.5x slower (155 µs) until this re-run
found and fixed an adjacency index rebuilt on every traversal; see
[§ Knowledge Graph](BENCHMARKS.md#knowledge-graph).
### Chunked Write Throughput (codec comparison)
@@ -195,7 +207,7 @@ by default (AoS→SoA byte transpose, +157–204% throughput for float data):
Use `.with_zstd(3)` or `.with_deflate(6)` for write-heavy workloads — both now perform at ~720–750 MiB/s on large matrices. Use `.with_pcodec()` for write-once/read-many workloads where compression ratio matters more than encode speed. Disable auto-shuffle with `.without_shuffle()` for byte arrays that don't benefit from AoS→SoA transposition.
> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search). The ratio overstates the real advantage by an unquantified margin — order-of-magnitude indication only. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171).
> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search), so the two columns are not comparable and no ratio is given. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171).
### LongMemEval Retrieval Recall
@@ -248,18 +260,26 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points.
### Memory Footprint
**On disk** — 384-dim embeddings, 200-char text
**On disk** — 384-dim `float16` embeddings (the default for new stores),
200-char text, `footprint_bench`
([BENCHMARKS.md § Memory Footprint](BENCHMARKS.md#memory-footprint-1)):
| Records | File Size | Bytes/Record | Gzip-6 compressed |
|---------|-----------|--------------|-------------------|
| 1K | 1.7 MB | 1.8 KB | 277 KB (6.1x) |
| 10K | 17.0 MB | 1.7 KB | 2.7 MB (6.2x) |
| 100K | 169.8 MB | 1.7 KB | 26.9 MB (6.2x) |
| 1K | 810.4 KB | 829 B | 56.4 KB |
| 10K | 7.8 MB | 820 B | 471.3 KB |
| 100K | 76.7 MB | 803 B | 4.5 MB |
These figures are `f32` embeddings. New agent stores default to
`MemoryConfig::float16`, which halves them: 100K × 384 records take 80.8 MiB
instead of 154.0.
The benchmark's synthetic embeddings and text are far more repetitive than
real data (only 40 distinct texts), so no column here is an expectation for
real data. The compressed column is an upper bound, and the Bytes/Record
column is optimistic too: it is not an uncompressed figure, because the store
always deflates its text (any string dataset of 4 KiB or more) whatever
`MemoryConfig::compression` says. The `float16` embeddings alone are 768 B per
record, so 200 characters of real text would take a record above 820 B.
This table used to show `f32` stores (1.7 KB per record, 169.8 MB at 100K);
those were not re-measured. The float16 study compares the two on the same
data: 100K × 384 records take 80.8 MiB as `float16` and 154.0 MiB as `f32`.
**In memory** — a store reopened from disk, 384-dim `f32`, measured with a
counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-footprint)):
@@ -271,7 +291,8 @@ counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-foo
| 100K | 146 MiB | 399 MiB (2.72x) | **256 MiB (1.74x)** |
Down from 505 MiB (3.44x) at 100K before v2.6.0, when the cache held every
embedding twice.
embedding twice. The `f32` column was re-measured on 2026-09-24 and reproduced
exactly; the `i8` column was not re-run.
### Consolidation Efficiency
@@ -282,7 +303,10 @@ embedding twice.
|--------|--------|-------|-------|
| Records in store | 1,000 | 100 | −90% |
| Hit@1 recall (signal records) | 100% | 100% | no loss |
| Search latency | 2.75 ms | 0.31 ms | **8.8x faster** |
| Search latency (avg) | 2.22 ms | 0.24 ms | **9.3x faster** |
The consolidation cycle that does this took 0.13 ms; at 10K records a cycle
takes 2.16 ms.
**Full benchmark details: [BENCHMARKS.md](BENCHMARKS.md)**
@@ -754,9 +778,37 @@ Replace in `Cargo.toml` and source:
```bash
cargo install --path crates/clawhdf5-migrate
clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedding-dim 384
clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedder minilm
```
The output is an ordinary `clawhdf5-agent` store, written through the agent's
own API: open it with `HDF5Memory::open` (or `clawhdf5-cli --path memory.h5 …`)
and search it straight away. What carries over from the ZeroClaw tables:
| SQLite | Agent store |
|--------|-------------|
| `memory_chunks` | memory records (text, embedding, source channel, timestamp, session id, tags); rows with `deleted = 1` become deleted records, or are left out with `--skip-deleted` |
| `sessions` | sessions (id, start/end index, channel, summary, timestamp) |
| `entities`, `relations` | knowledge graph entities and relations; entities get new ids and relations are re-pointed at them |
The chunk `id` column has no counterpart in the agent store, so records are
written in `id` order and numbered from 0. Embeddings are stored as float16
like any new store; `--f32` keeps full precision (and is required for values
beyond ±65504). The embedding dimension is detected from the first row unless
`--embedding-dim` is given, and every row must have it: a row of another length
is an error, never truncated or padded. A source with no memory records (only
sessions or the graph) needs `--embedding-dim`, since a store's dimension is
fixed when it is created. Every row is checked before the output is created,
so a source that cannot be migrated leaves an existing store at `--hdf5` as it
was. `--incremental` adds to an existing store only the rows it does not
already hold; the source must have the store's dimension, and records already
in the store take the source's deleted flag (a row deleted in SQLite since the
last run is deleted in the store; one un-deleted there is written again, as
the agent has no un-delete). The tool reads the result back with
`HDF5Memory::open_read_only`, compares it with the source (every row with
`--validate-full`) and checks that a migrated record is found by search;
`--dry-run` only counts the rows.
---
## Roadmap
crates/clawhdf5-agent/Cargo.toml
+4
View File
@@ -45,6 +45,10 @@ harness = false
name = "memory_bench"
harness = false
[[bench]]
name = "multimodal_bench"
harness = false
[features]
default = ["float16", "hnsw", "parallel"]
float16 = ["half"]
crates/clawhdf5-agent/benches/multimodal_bench.rs
+107
View File
@@ -0,0 +1,107 @@
//! Multi-modal memory search benchmarks (`clawhdf5_agent::multimodal`).
//!
//! Covers `MultiModalStore::search_cross_modal` (every embedding of every
//! record, whatever its modality) and, for comparison,
//! `MultiModalStore::search_by_modality` restricted to one modality.
//!
//! Corpus: N records (1K and 10K), each carrying two 384-dim embeddings —
//! a text embedding of its caption plus one embedding of its primary modality,
//! cycling Image / Audio / Video — so a cross-modal query scores 2N vectors.
//! All data comes from a fixed-seed LCG, so every run sees the same corpus.
//!
//! Run: `cargo bench -p clawhdf5-agent --bench multimodal_bench`
use std::collections::HashMap;
use clawhdf5_agent::multimodal::{
MediaRef, ModalEmbedding, Modality, MultiModalRecord, MultiModalStore,
};
use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
// ---------------------------------------------------------------------------
// Simple deterministic PRNG (LCG), same as the other agent benches
// ---------------------------------------------------------------------------
struct Rng(u32);
impl Rng {
fn new(seed: u32) -> Self {
Self(seed)
}
fn next_u32(&mut self) -> u32 {
self.0 = self.0.wrapping_mul(1103515245).wrapping_add(12345);
self.0 >> 16
}
fn next_f32(&mut self) -> f32 {
self.next_u32() as f32 / 65536.0 - 0.5
}
}
fn make_vec(rng: &mut Rng, dim: usize) -> Vec<f32> {
(0..dim).map(|_| rng.next_f32()).collect()
}
// ---------------------------------------------------------------------------
// Corpus
// ---------------------------------------------------------------------------
const DIM: usize = 384;
const K: usize = 10;
const MEDIA: [(Modality, &str, &str); 3] = [
(Modality::Image, "image/png", "clip-vit-base"),
(Modality::Audio, "audio/wav", "clap-base"),
(Modality::Video, "video/mp4", "xclip-base"),
];
fn build_store(n: usize, seed: u32) -> MultiModalStore {
let mut rng = Rng::new(seed);
let mut store = MultiModalStore::new();
for i in 0..n {
let (modality, mime, model) = &MEDIA[i % MEDIA.len()];
let embeddings = vec![
ModalEmbedding::new(Modality::Text, make_vec(&mut rng, DIM), "minilm-l6"),
ModalEmbedding::new(modality.clone(), make_vec(&mut rng, DIM), *model),
];
store.add_record(MultiModalRecord {
id: 0,
primary_modality: modality.clone(),
text_content: Some(format!("{modality} memory {i}")),
media_ref: Some(MediaRef::path(format!("/media/{i}"), *mime)),
embeddings,
observation: None,
timestamp: 1_700_000_000.0 + i as f64,
metadata: HashMap::new(),
});
}
store
}
// ---------------------------------------------------------------------------
// Benchmarks
// ---------------------------------------------------------------------------
fn multimodal_search_benches(c: &mut Criterion) {
let query = make_vec(&mut Rng::new(99), DIM);
let mut group = c.benchmark_group("multimodal_search");
group.sample_size(50);
for (label, n) in [("1k", 1_000usize), ("10k", 10_000)] {
let store = build_store(n, 42);
assert_eq!(store.count(), n);
group.bench_with_input(BenchmarkId::new("cross_modal", label), &n, |b, _| {
b.iter(|| store.search_cross_modal(&query, K));
});
group.bench_with_input(BenchmarkId::new("by_modality_image", label), &n, |b, _| {
b.iter(|| store.search_by_modality(&Modality::Image, &query, K));
});
}
group.finish();
}
criterion_group!(multimodal_benches, multimodal_search_benches);
criterion_main!(multimodal_benches);
crates/clawhdf5-bench/src/bin/footprint_bench.rs
+19 -3
View File
@@ -11,12 +11,14 @@
//!
//! Configuration matrix:
//! - Text lengths: short (50 chars), medium (200 chars), long (1000 chars)
//! - Embedding: 384-dim f32 (1536 bytes raw per record)
//! - Embedding: 384-dim, stored as float16 (the default for new stores) or
//! f32 with `--f32`; "raw" bytes are counted as f32 input either way
//! - WAL: enabled and disabled
//!
//! # Usage
//! ```
//! cargo run --release --bin footprint_bench
//! cargo run --release --bin footprint_bench # float16 stores
//! cargo run --release --bin footprint_bench -- --f32 # f32 stores
//! ```
use std::time::Instant;
@@ -24,6 +26,9 @@ use std::time::Instant;
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
use tempfile::TempDir;
/// `--f32`: build f32 stores instead of the library's float16 default.
static F32: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
const EMBEDDING_DIM: usize = 384;
// Raw bytes per record: 384 f32 embeddings + median text + overhead
@@ -152,6 +157,9 @@ fn measure_footprint(
config.compression = compression;
config.compression_level = if compression { 6 } else { 0 };
config.compact_threshold = 0.0;
if F32.load(std::sync::atomic::Ordering::Relaxed) {
config.float16 = false;
}
let mut memory = HDF5Memory::create(config).expect("HDF5Memory::create failed");
@@ -241,11 +249,19 @@ fn fmt_n(n: usize) -> String {
// ---------------------------------------------------------------------------
fn main() {
if std::env::args().skip(1).any(|a| a == "--f32") {
F32.store(true, std::sync::atomic::Ordering::Relaxed);
}
let stored = if F32.load(std::sync::atomic::Ordering::Relaxed) {
"f32 (1,536 bytes per record)"
} else {
"float16 (768 bytes per record; the default for new stores)"
};
println!("=================================================================");
println!(" ClawhDF5 Memory Footprint Benchmark");
println!("=================================================================");
println!();
println!("Embedding: 384-dim f32 = 1,536 bytes raw per record");
println!("Embedding: 384-dim, stored as {stored}; raw input counted as f32");
println!("Text lengths: short=50 chars, medium=200 chars, long=1000 chars");
println!();