From dce5559ff2a83b23fc76c356e37fd0205f132b38 Mon Sep 17 00:00:00 2001 From: osobh Date: Thu, 24 Sep 2026 23:45:15 -0500 Subject: [PATCH] bench: re-run every stale BENCHMARKS.md section, dated and traced Every undated or pre-September section re-run on one machine on one day (tank, AMD Ryzen 7 7800X3D, 2026-09-24, commit 5c8323c), 24 commands run serially with the load average checked before each, with the command recorded for each section. A separate check traced every changed number back to the raw output; its corrections are applied (e.g. the on-disk ~820 B/record is float16 plus always-deflated text on a synthetic corpus of 40 distinct texts, not float16 alone). Two apparent regressions were isolated rather than published: - knowledge-graph traversal: a real bug, fixed in the previous commit; - the write path: v2.3.0 built and run on the same machine measures the same as today, so the old 18 us / 6.17 ms figures (undated, other hardware) are not reproducible; float16 adds ~2 us per save and the int8 index nothing (both isolated by switching the bench's config). Also: - new multimodal_bench: cross-modal search at 1K/10K records, which the README claimed but nothing measured; - footprint_bench reports whether it built float16 or f32 stores and takes --f32 (it kept printing "f32" after the default changed); - README: performance tables, the "Why" table figures and the SQLite migration section (from the previous migrate commit); - CHANGELOG for this branch. Not re-run: consolidation_efficiency's 100K row and its memory-reduction part (stopped for time), and cross_platform.sh. Co-Authored-By: Claude Opus 5.5 (1M context) --- BENCHMARKS.md | 753 ++++++++++++++---- CHANGELOG.md | 51 ++ README.md | 118 ++- crates/clawhdf5-agent/Cargo.toml | 4 + .../benches/multimodal_bench.rs | 107 +++ .../clawhdf5-bench/src/bin/footprint_bench.rs | 22 +- 6 files changed, 880 insertions(+), 175 deletions(-) create mode 100644 crates/clawhdf5-agent/benches/multimodal_bench.rs diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 1e4b98d..fdccb50 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -2,20 +2,40 @@ > Pure Rust. Zero C dependencies. Single file. Fast enough to forget it's there. -**System:** Intel i7-12650H (10C/16T, 4.7 GHz boost) · 32 GB DDR5 · Linux 6.8.0 +**System (original run):** Intel i7-12650H (10C/16T, 4.7 GHz boost) · 32 GB DDR5 · Linux 6.8.0 **Rust:** 1.96.0-nightly (2026-03-14) · `--release` profile **Date:** 2026-07-01 -> **Traceability note:** the "h5bench-Equivalent I/O Benchmarks" and both -> "Independent Validation: tank" sections below meet a dated, -> hardware-cited, reproducible standard (explicit date, machine spec, and a -> runnable command per result) — this now covers "LongMemEval Results", -> "SIMD & Parallelism", "Vector Search Latency", and "Comparison to MemX" via -> their tank re-runs. The remaining undated sections above (Hybrid Search, -> Knowledge Graph, Memory Consolidation, Temporal Index, Write Path, Decision -> Gate, Memory Strategy, Multi-Session Benchmark, Memory Footprint, -> Consolidation Efficiency, Ephemeral Tier) do not yet meet that bar — this is -> a known, tracked documentation gap, not a claim that those numbers are wrong. +Sections that carry their own "Measured …" line were measured there instead, +not on the system above. The 2026-09-24 re-run was on tank (AMD Ryzen 7 +7800X3D, 8C/16T, Linux 7.0.0-34-generic, rustc 1.98.1) at commit 5c8323c, +serially, with no `--warm-up-time`/`--measurement-time` overrides, waiting +before each command while the 1-minute load average was above 2.0. The +`bench` targets ran at Criterion's defaults (3 s warm-up, 5 s target +measurement time, 100 samples). Every `memory_bench` group that was run +(`vector_search_latency`, `hybrid_search`, `knowledge_graph`, +`consolidation`, `temporal`) and the multi-modal group set +`sample_size(50)`, so those sections took 50 samples, not 100. The 5 s is a +target: Criterion stretched it where 5 s could not hold the samples it needed +(for example 8.5 s for `vector_search_latency/bench_cosine_search/100k`, +15.5 s for the 1K consolidation cycle, 252 s for `tick_session_10k`). + +> **Traceability note:** a section meets the dated, hardware-cited, +> reproducible standard when it gives an explicit date, the machine, and a +> runnable command for its results. As of the 2026-09-24 re-run on tank that +> covers Vector Search Latency, Comparison to MemX, SIMD & Parallelism, Hybrid +> Search, Knowledge Graph, Memory Consolidation, Temporal Index, Write Path, +> Decision Gate, Memory Strategy, Summary (derived from those), LongMemEval +> Results (the BM25 rows), Multi-Session Benchmark, Memory Footprint (on disk +> and in memory), Consolidation Efficiency, Ephemeral Tier, Multi-modal Search, +> the Search and Read harnesses, and the "h5bench-Equivalent I/O Benchmarks" +> and "Independent Validation: tank" sections. What does not yet meet that bar: +> the LongMemEval rows that need real embeddings (not re-run here, except the +> dated float16 comparison), the Consolidation Efficiency 100K cycle row and +> memory-reduction part (the 2026-09-24 run was stopped before it produced +> them), the int8 side of "Quantising the index copy" (not re-run), and the +> i7-12650H and macOS M3 Max rows under Cross-Platform Notes. That is a +> known, tracked documentation gap, not a claim that those numbers are wrong. > > **Correctness note (2026-08-06).** Being dated and reproducible is necessary but > not sufficient — a number can be perfectly reproducible and still measure the @@ -57,6 +77,23 @@ What remains at 2.43x: the flat vectors (1.0x), the HNSW index's own copy of them (1.0x), and text, ids and graph (~0.4x). The index copy is the next target — it is what a quantised or borrowed representation would address. +**Current figures (`f32` index).** Measured 2026-09-24 on tank (AMD Ryzen 7 +7800X3D), commit 5c8323c: + +> **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --footprint --full` + +| N | vectors (raw) | entries MiB | store MiB | indexes MiB | reopened MiB | peak during open MiB | reopened / raw | +|---:|---:|---:|---:|---:|---:|---:|---:| +| 1 000 | 1 | 2 | 0 | 2 | 4 | 5 | 2.40x | +| 10 000 | 15 | 17 | 12 | 32 | 44 | 61 | 3.03x | +| 100 000 | 146 | 172 | 60 | 266 | 399 | 562 | 2.72x | + +These are the "reopened, f32" figures in the next table, reproduced exactly. +They are above the "after" column of the table above (44 MiB vs 35 MiB at 10K, +399 vs 357 MiB at 100K): the 2.43x was measured for 2e7e045, and the store +already measured 399 MiB (2.72x) when the int8 index landed later the same +day (c0a9206). The git log does not say what changed in between. + ### Quantising the index copy (`quantized_index`) `MemoryConfig::quantized_index` stores the index's copy as `i8` instead of @@ -108,6 +145,20 @@ second**, builds **1.8x faster**, and holds a quarter of the vectors. (Compare only at equal `ef`: with re-scoring the harness raises `ef` to at least the candidate pool, so the `ef = 16` and `ef = 32` rows are not like-for-like.) +**Re-check, 2026-09-24.** Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), +commit 5c8323c. The `f32` memory columns of the first table reproduce exactly +(see [Memory footprint](#memory-footprint)). Only the `f32` side of the speed +table was re-run, once, not as alternating medians: + +> **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --full` + +At N = 100 000, `ef = 64`, the `f32` index measured recall@10 0.9945 (same), +19 001 QPS (was 13 399) and a 2700.8 ms build (was 3197 ms). The QPS figure +moved by more than 20%; the log does not explain it, and a single run is not +the same-moment paired comparison the table above is. The int8 columns were +not re-run, so the 1.63x ratio has not been re-checked against the new `f32` +figure. + #### On ARM (Raspberry Pi 5, Cortex-A76) `dot_i8` has two aarch64 kernels: `SDOT` for CPUs with the ARMv8.2 @@ -268,6 +319,10 @@ high-water mark alongside the retained figure: | 10 000 | 44 | 61 | | 100 000 | 399 | 562 | +Reproduced exactly on 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit +5c8323c, by `search_harness -- --footprint --full` (table under +[Memory footprint](#memory-footprint)). + The peak is set *after* the parse, by the index build, so a buffer allocated and freed during the parse never reaches the high-water mark. Holding a deliberate extra copy of the file across the whole parse leaves the peak @@ -359,6 +414,50 @@ contiguous dataset is converted straight from the file bytes (one copy instead of two), and the native-endian conversions no longer zero a buffer they are about to overwrite. +### Current: read harness (2026-09-24) + +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. + +> **Run:** `cargo run --release -p clawhdf5-bench --bin read_harness` + +4096 x 2048 f64 (64 MB per dataset), chunks 256 x 256, file 129 MB + +| layout | read | selected | time ms | MB/s of selection | vs full read | +|---|---|---:|---:|---:|---:| +| chunked + deflate | full (first) | 64 MB | 65.6 | 975 | | +| chunked + deflate | full (repeat) | 64 MB | 63.2 | 1013 | 1.00x | +| chunked + deflate | 64 x 64 window (1 chunk) | 0.03 MB | 0.18 | 177 | 0.003x | +| chunked + deflate | 512 x 512 window (4-9 chunks) | 2.00 MB | 4.12 | 485 | 0.063x | +| chunked + deflate | one row | 0.02 MB | 0.99 | 16 | 0.015x | +| chunked + deflate | one column | 0.03 MB | 2.02 | 15 | 0.031x | +| chunked | full (first) | 64 MB | 60.5 | 1057 | | +| chunked | full (repeat) | 64 MB | 56.7 | 1129 | 1.00x | +| chunked | 64 x 64 window (1 chunk) | 0.03 MB | 0.10 | 325 | 0.002x | +| chunked | 512 x 512 window (4-9 chunks) | 2.00 MB | 3.19 | 628 | 0.053x | +| chunked | one row | 0.02 MB | 0.05 | 284 | 0.001x | +| chunked | one column | 0.03 MB | 0.49 | 64 | 0.008x | +| contiguous | full (first) | 64 MB | 30.1 | 2124 | | +| contiguous | full (repeat) | 64 MB | 26.1 | 2452 | 1.00x | +| contiguous | 64 x 64 window (1 chunk) | 0.03 MB | 0.08 | 394 | 0.003x | +| contiguous | 512 x 512 window (4-9 chunks) | 2.00 MB | 3.36 | 595 | 0.112x | +| contiguous | one row | 0.02 MB | 0.03 | 486 | 0.001x | +| contiguous | one column | 0.03 MB | 2.61 | 12 | 0.087x | + +Full reads match the "after" column above (chunked + deflate 69 ms then, 63.2 +to 65.6 ms now). The compressed selections are faster than in "After: partial +reads" (64 x 64 window 0.39 -> 0.18 ms, one row 2.69 -> 0.99 ms, one column +5.23 -> 2.02 ms) and match the 2026-09-23 figures in +[Deflate backend](#deflate-backend-zlib-rs-vs-zlib-ng) (0.18, 1.00 and 2.01 +ms), which were taken after the zlib-rs switch and the one-shot codec calls +described there; this run does not isolate which change accounts for it. + +The uncompressed windows went the other way. Against "After: partial reads", +the 512 x 512 window is about 60% slower (chunked 1.99 -> 3.19 ms, contiguous +2.12 -> 3.36 ms) and the chunked 64 x 64 window went from 0.06 to 0.10 ms. +The rows and columns of the uncompressed layouts are within 20% (chunked +column 0.45 -> 0.49 ms, contiguous column 2.55 -> 2.61 ms). This run does not +explain the slower windows. + ## Search harness baseline (v2.3.0) Produced by `cargo run --release -p clawhdf5-bench --bin search_harness -- --full` @@ -596,68 +695,217 @@ Recall on clustered data is unchanged or slightly better (100K, `ef = 64`: 0.474 -> 0.444), the cost of batch members not seeing each other while planning; batches are capped at 1/16 of the graph and 512 nodes. +### Current: search harness (2026-09-24) + +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, default +features (so the build is batched and `parallel`, 16 threads). The HNSW tables +use an `f32` index. The end-to-end table uses `MemoryConfig::new`'s defaults, +which at this commit are `float16` embeddings and the int8 index; every +earlier end-to-end table in this section was an `f32` store with an `f32` +index. + +> **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --full` + +#### HNSW, N = 1000, dim = 384, M = 16, ef_construction = 64, storage = Float32 + +build: 14.7 ms (67954 vectors/s) · exact scan: 4403 QPS, p50 225 µs + +| ef | recall@10 | QPS | p50 µs | p99 µs | +|---:|---:|---:|---:|---:| +| 16 | 0.9940 | 134065 | 7 | 18 | +| 32 | 1.0000 | 105134 | 9 | 21 | +| 64 | 1.0000 | 77541 | 13 | 26 | +| 128 | 1.0000 | 36938 | 26 | 35 | +| 256 | 1.0000 | 19249 | 52 | 63 | + +#### HNSW, N = 10000, dim = 384, M = 16, ef_construction = 64, storage = Float32 + +build: 135.6 ms (73736 vectors/s) · exact scan: 435 QPS, p50 2295 µs + +| ef | recall@10 | QPS | p50 µs | p99 µs | +|---:|---:|---:|---:|---:| +| 16 | 0.9975 | 79614 | 12 | 24 | +| 32 | 1.0000 | 70556 | 14 | 21 | +| 64 | 1.0000 | 51313 | 19 | 29 | +| 128 | 1.0000 | 22846 | 44 | 57 | +| 256 | 1.0000 | 10267 | 97 | 119 | + +#### HNSW, N = 100000, dim = 384, M = 16, ef_construction = 64, storage = Float32 + +build: 2700.8 ms (37027 vectors/s) · exact scan: 43 QPS, p50 23433 µs + +| ef | recall@10 | QPS | p50 µs | p99 µs | +|---:|---:|---:|---:|---:| +| 16 | 0.9180 | 20043 | 47 | 104 | +| 32 | 0.9775 | 21348 | 44 | 104 | +| 64 | 0.9945 | 19001 | 51 | 101 | +| 128 | 0.9995 | 13458 | 73 | 137 | +| 256 | 0.9995 | 5128 | 196 | 282 | + +#### End to end: `HDF5Memory::hybrid_search` (k = 10, weights 0.7 / 0.3) + +| N | ingest ms | cold index build ms | checkpoint ms | open ms | first query after open ms | p50 ms | p99 ms | QPS | +|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| 1000 | 15 | 14 | 4.4 | 1.9 | 1.5 | 0.07 | 0.08 | 13869.5 | +| 10000 | 146 | 126 | 33.5 | 18.3 | 14.8 | 0.49 | 0.50 | 2044.1 | +| 100000 | 1498 | 1833 | 495.7 | 259.9 | 155.1 | 4.69 | 5.22 | 212.7 | + +Steady-state query latency is where the v2.4.0 work left it (p50 0.07 / 0.49 / +4.69 ms against 0.07 / 0.49 / 4.65 ms). Figures that moved by more than 20%: + +- **Build.** 14.7 / 135.6 / 2700.8 ms against 34 / 388 / 5.9 s for + "batched + `parallel`" above. The git log does not explain the difference. +- **Cold index build** in the end-to-end table: 14 / 126 / 1833 ms, down from + 112 / 1487 / 20285 ms in the "unranked keyword scores" table, which predates + the batched and parallel build described above. +- **QPS at 100K, `ef = 64`**: 19 001 against 13 980 in the "unit-vector" + table, with recall 0.9945 against 0.984; the recall gain is the one the + batched build reported. The QPS gain is not explained by the log. +- **Other HNSW QPS cells**, against the same "unit-vector" tables: 10K + `ef = 16` 79 614 against 54 608 (+46%); 100K `ef = 16` 20 043 against + 15 139 (+32%), `ef = 128` 13 458 against 10 959 (+23%), `ef = 256` 5 128 + against 3 731 (+37%). The other cells, and every 1K cell, are within 20%. + The log does not explain these either. +- **Open** in the end-to-end table: 1.9 / 18.3 / 259.9 ms against 1.1 / 13.7 + / 353.1 ms in the "unranked keyword scores" table, slower at 1K and 10K and + faster at 100K. This run does not isolate the cause. +- **Checkpoint at 100K**: 495.7 ms against 728.9 ms, and ingest up 11 -> 15, + 104 -> 146 ms at 1K / 10K. This store is `float16` with an int8 index, the + earlier ones `f32`; the [float16 study](#float16-embedding-storage-memoryconfigfloat16) + measured float16 checkpoints at 512 ms against 752 ms for `f32` at 100K. + This run does not isolate the ingest change. + ## Vector Search Latency Brute-force cosine similarity over 384-dimensional embeddings (OpenAI text-embedding-3-small size). +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. + +> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(vector_search_1k|simd_cosine_10k|simd_cosine_100k|prenorm_search_10k|ivf_search_10k_nprobe10|ivf_search_100k_nprobe10|ivf_pq_search_100k|rairs_search_10k_nprobe10)$'` + | Scale | Flat Search | Pre-norm | IVF (nprobe=10) | IVF-PQ | RAIRS | |-------|-------------|----------|-----------------|--------|-------| -| **1K** | 54 µs | 62 µs | — | — | — | -| **10K** | 753 µs | 706 µs | 27 µs | — | 159 µs | -| **100K** | 11.4 ms | — | 1.32 ms | 1.19 ms | — | +| **1K** | 47.4 µs | — | — | — | — | +| **10K** | 500.5 µs | 316.5 µs | 24.8 µs | — | 124.0 µs | +| **100K** | 6.58 ms | — | 592 µs | 869 µs | — | -**Key insight:** At 10K records (typical agent memory), IVF search delivers **27 µs** — that's 26x faster than flat search. Even at 100K records, IVF-PQ keeps search under **1.2 ms**. +The 1K pre-norm cell is blank because no 1K pre-norm benchmark exists (the +old 62 µs has no source). `memory_bench`'s own cosine search +(`cargo bench -p clawhdf5-agent --bench memory_bench -- '^vector_search_latency/'`) +agrees: 47.6 µs / 505.1 µs / 6.67 ms at 1K / 10K / 100K. + +Most cells moved by more than 20% from the previous table (10K flat 753 µs, +pre-norm 706 µs, RAIRS 159 µs; 100K flat 11.4 ms, IVF 1.32 ms, IVF-PQ 1.19 ms). +Those were undated figures from the original i7-12650H run. The 2026-08-05 +tank run below is within 4% of the new numbers in every cell but one (1K flat +47.8 µs, 10K flat 501 µs, pre-norm 322 µs, IVF 24.8 µs; 100K flat 6.60 ms, +IVF 608 µs, IVF-PQ 865 µs), so for those cells the difference is the machine. +The exception is RAIRS: 109 µs on 2026-08-05, 124.0 µs now (+14%); this run +does not explain that change. + +**Key insight:** at 10K records (typical agent memory), IVF search takes +**24.8 µs** against 500.5 µs for the flat scan. At 100K, IVF answers in +**592 µs** and IVF-PQ in 869 µs — both under 1 ms. (RAIRS, 124.0 µs at 10K, is +slower than plain IVF here.) ### Comparison to MemX (arxiv:2603.16171) MemX claims end-to-end search under 90ms at 100K records (Rust + libSQL + FTS5). +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. + +> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(simd_cosine_100k|ivf_pq_search_100k|bm25_search_10k)$'` + > **Caveat — not like-for-like.** MemX's `<90 ms` is *end-to-end* search across their > full pipeline (dense embeddings + FTS5 + four-factor re-ranking). The clawhdf5 > figures below are a *single component* — raw vector search latency, excluding > embedding, keyword, fusion, and re-ranking stages. A component measured against a -> full pipeline will always look favourable; the "speedup" column overstates the real -> advantage by an unquantified margin and should be read as an order-of-magnitude -> indication only, not a benchmark result. Matching MemX's measurement boundary is -> tracked as follow-up work. +> full pipeline will always look favourable, so the two columns below cannot be +> divided into a speedup: any such ratio would overstate the real advantage by an +> unquantified margin. Read the table as an order-of-magnitude indication only, not a +> benchmark result. Matching MemX's measurement boundary is tracked as follow-up work. -| Metric | MemX (claimed, end-to-end) | ClawhDF5 (component only) | Ratio | -|--------|----------------------------|---------------------------|-------| -| 100K flat search | <90 ms | 11.4 ms | ~8x | -| 100K IVF-PQ search | — | 1.19 ms | ~76x | -| Keyword search 10K | 1,100x improvement over unindexed | 583 µs (BM25) | Comparable | +| Metric | MemX (claimed, end-to-end) | ClawhDF5 (component only) | +|--------|----------------------------|---------------------------| +| 100K flat search | <90 ms | 6.64 ms | +| 100K IVF-PQ search | — | 876 µs | +| Keyword search 10K | 1,100x improvement over unindexed | 205 µs (BM25) | + +The ratio column is gone: dividing an end-to-end claim by a component +timing produced a number that looked like a result and was not one. The +previous figures (100K flat 11.4 ms, IVF-PQ 1.19 ms, BM25 10K 583 µs) were +undated, from the original i7-12650H run. BM25 at 10K also moved on this +machine: 520 µs on 2026-08-05, 205 µs now; v2.4.0 changed +`BM25Index::search` in between (bounded-heap top-k, the unused WAND bound +removed, IDF per query), but this run does not isolate its effect. --- ## SIMD & Parallelism -384-dimensional cosine similarity at 10K scale. +384-dimensional cosine similarity at 10K scale, using the `strategy_*` +benchmarks, which hold the dataset fixed and vary only `SearchStrategy` +(the 2026-08-05 re-run below explains why the older table's benchmarks were +not an apples-to-apples comparison). -| Strategy | Latency | vs Sequential | -|----------|---------|---------------| -| Sequential (scalar) | 1.07 ms | 1.0x | -| SIMD (auto-vectorized) | 545 µs | **2.0x** | -| Rayon (parallel) | 553 µs | **1.9x** | -| Adaptive (auto-select) | 564 µs | **1.9x** | +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. Rayon +uses all 16 threads; the 1-minute load average was 4.19 when this command +finished, most likely from the Rayon benchmarks themselves (not verified). + +> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(strategy_scalar_10k|strategy_simd_10k|strategy_rayon_10k|adaptive_search_10k|simd_cosine_100k|rayon_cosine_100k)$'` -At 100K: | Strategy | Latency | |----------|---------| -| SIMD | 13.7 ms | -| Rayon parallel | 8.3 ms | +| Sequential (scalar), `strategy_scalar_10k` | 501.1 µs | +| SIMD, `strategy_simd_10k` | 319.9 µs | +| Rayon (parallel), `strategy_rayon_10k` | **79.0 µs** | +| Adaptive (auto-select), `adaptive_search_10k` | 317.9 µs | + +At 100K (no `strategy_*` benchmark exists at this size, so these are the +`simd_cosine_100k` / `rayon_cosine_100k` benchmarks): + +| Strategy | Latency | +|----------|---------| +| SIMD | 6.56 ms | +| Rayon parallel | 4.63 ms | + +The Rayon 10K figure moved furthest: 323 µs on this machine on 2026-08-05, +79.0 µs now. The git log does not explain it. The other rows are close to +that run (scalar 502 µs, SIMD 327 µs, adaptive 339 µs; 100K 6.60 / 4.73 ms). Against the original, undated i7-12650H table (scalar 1.07 ms, SIMD +545 µs, Rayon 553 µs, adaptive 564 µs; 100K 13.7 / 8.3 ms) every figure is +lower; that table also used the mislabelled benchmarks. The older names +measured in the same session, for reference (`cargo bench -p clawhdf5-agent +--bench bench -- '^(sequential_cosine_10k|simd_cosine_10k|rayon_cosine_10k|adaptive_search_10k|simd_cosine_100k|rayon_cosine_100k)$'`): +`sequential_cosine_10k` 317.8 µs, `simd_cosine_10k` 505.2 µs, +`rayon_cosine_10k` 274.3 µs — "sequential" faster than "SIMD", which is the +mislabelling the 2026-08-05 re-run found. --- ## Hybrid Search (Vector + BM25) -1K records, 384-dimensional embeddings with BM25 keyword index. +1K records, 384-dimensional embeddings with BM25 keyword index. These are the +flat free functions `hybrid::hybrid_search` / `rrf_hybrid_search` (linear +scan), not the HNSW-backed `HDF5Memory::hybrid_search`, which is measured in +the [search harness](#current-search-harness-2026-09-24). + +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. + +> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^hybrid_search/'` +> and `cargo bench -p clawhdf5-agent --bench bench -- '^(bm25_search_1k|hybrid_search_10k)$'` | Method | Latency | Notes | |--------|---------|-------| -| Weighted fusion | 198 µs | Original min-max normalization | -| **RRF (k=60)** | **222 µs** | Reciprocal Rank Fusion — better quality, ~12% overhead | -| BM25-only 1K | 67 µs | Keyword search alone | -| Hybrid 10K | 2.04 ms | Full hybrid at 10K scale | +| Weighted fusion | 106.8 µs | Min-max normalization | +| **RRF (k=60)** | **130.5 µs** | Reciprocal Rank Fusion | +| BM25-only 1K | 20.4 µs | Keyword search alone | +| Hybrid 10K | 1.10 ms | Full hybrid at 10K scale | + +Every row moved by more than 20% from the previous table (198 µs, 222 µs, +67 µs, 2.04 ms), which was undated and from the original i7-12650H run. The +cause is not isolated. (The old "better quality" note on RRF is dropped: on +LongMemEval RRF measured worse than the tuned weighted sum; see +[Fusion method](#fusion-method--weighted-vs-rrf-full-haystack-n500).) --- @@ -665,16 +913,36 @@ At 100K: Graph traversal and entity operations. +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. + +> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^knowledge_graph/'` +> and `cargo bench -p clawhdf5-agent --bench bench -- '^alias_resolve_(short|long)_query$'` +> (the `knowledge_graph/` rows re-run after the adjacency-index fix, same day, +> load average 0.49 at start) + | Operation | Scale | Latency | |-----------|-------|---------| -| BFS traversal | 100 entities | 5.4 µs | -| BFS traversal | 1,000 entities | 24 µs | -| Spreading activation | 100 entities | 16.9 µs | -| Entity resolution (Levenshtein) | 100 entities | 64 µs | -| Alias resolution (short query) | 100 aliases | 10.4 µs | -| Alias resolution (long query) | 100 aliases | 11.6 µs | +| BFS traversal | 100 entities | 5.23 µs | +| BFS traversal | 1,000 entities | 23.1 µs | +| Spreading activation | 100 entities | 10.1 µs | +| Entity resolution (Levenshtein) | 100 entities | 48.6 µs | +| Alias resolution (short query) | 100 aliases | 7.67 µs | +| Alias resolution (long query) | 100 aliases | 8.18 µs | -**All graph operations complete in microseconds.** Spreading activation across 100 entities with 5 propagation steps finishes in 17 µs. +The traversal rows were measured after a fix made during this re-run. The +first measurement of the day found **BFS 6.5x slower** than the old table +said: 17.5 µs and 155.1 µs, against 5.4 µs and 24 µs, with spreading +activation at 22.8 µs against 16.9 µs. The cause was 1efd82c (2026-08-17), +which built an adjacency index over the whole graph on every traversal — so a +2-hop BFS paid to index every entity and relation first. The index is now +cached on `KnowledgeCache` and checked against a fingerprint of the graph on +each use (one pass over entity ids and relation endpoints, no allocation), so +any change to the graph, including direct edits of its public `Vec`s, still +triggers a rebuild. These benches traverse an unchanged graph, which is the +cached case; the first traversal after a change pays for one build. Entity +and alias resolution do not use the index (the 49.8 µs first-run figure for +entity resolution is within noise of the 48.6 µs here). The old figures were +undated, from the original i7-12650H run. --- @@ -682,13 +950,22 @@ Graph traversal and entity operations. Hippocampal-inspired tiered memory management. +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. + +> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^consolidation/'` + | Operation | Scale | Latency | |-----------|-------|---------| -| Consolidation cycle | 100 records | 15 µs | -| Consolidation cycle | 1,000 records | 164 µs | -| Importance scoring | 100 records | 25 µs | +| Consolidation cycle | 100 records | 8.08 µs | +| Consolidation cycle | 1,000 records | 115.2 µs | +| Importance scoring | 100 records | 31.0 µs | -A full consolidation pass over 1,000 memories (eviction + promotion across Working → Episodic → Semantic) completes in **164 µs**. This can run on every memory write without perceptible latency. +A full consolidation pass over 1,000 memories (eviction + promotion across Working → Episodic → Semantic) completes in **115 µs**. This can run on every memory write without perceptible latency. + +The cycle figures were 15 µs and 164 µs, and importance scoring 25 µs, all +undated from the original i7-12650H run. Eviction's membership check became a +`HashSet` lookup (603fcf8) and `add_memory` stopped cloning Working-tier +records (d787ac0), both on 2026-08-17; this run does not isolate their effect. --- @@ -696,12 +973,17 @@ A full consolidation pass over 1,000 memories (eviction + promotion across Worki Sorted timestamp index with binary search. +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. + +> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^temporal/'` + | Operation | Scale | Latency | |-----------|-------|---------| -| Range query | 10K timestamps | **716 ns** | -| Batch insert | 10K timestamps | 4.69 ms | +| Range query | 10K timestamps | **622 ns** | +| Batch insert | 10K timestamps | 3.22 ms | -Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K records: **716 nanoseconds.** +Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K records: **622 nanoseconds.** +Batch insert was 4.69 ms in the previous, undated i7-12650H figure. --- @@ -709,16 +991,50 @@ Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K HDF5 persistence with optional Write-Ahead Log. +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. The stores +use `MemoryConfig::new`'s defaults at this commit: `float16` embeddings and +the int8 index. + +> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(save_without_wal_single|save_with_wal_single|save_batch_100|save_batch_1000|save_wal_1k_existing|wal_flush_100_entries|tick_session_1k|tick_session_10k)$'` + | Operation | Latency | Notes | |-----------|---------|-------| -| Single save (no WAL) | 61 µs | Direct HDF5 write (owned-Vec IO path) | -| Single save (with WAL) | 18 µs | WAL group-commit append; HDF5 write batched at flush | -| Batch 100 | 723 µs | 7.2 µs per record | -| Batch 1,000 | 6.17 ms | 6.2 µs per record | -| WAL save (1K existing) | 539 µs | Incremental append | -| WAL flush 100 entries | 787 µs | Merge WAL → HDF5 | -| Session tick 1K | 5.76 ms | Full session maintenance | -| Session tick 10K | 89.8 ms | Background operation | +| Single save (no WAL) | 88.3 µs | Direct HDF5 write (owned-Vec IO path) | +| Single save (with WAL) | 26.1 µs | WAL group-commit append; HDF5 write batched at flush | +| Batch 100 | 1.08 ms | | +| Batch 1,000 | 31.7 ms | | +| WAL save (1K existing) | 282.8 µs | Incremental append | +| WAL flush 100 entries | 530.1 µs | Merge WAL → HDF5 | +| Session tick 1K | 3.83 ms | Full session maintenance | +| Session tick 10K | 35.5 ms | Background operation | + +Almost every row moved by more than 20% from the previous table, in both +directions. Slower: single save 61 -> 88.3 µs (no WAL) and 18 -> 26.1 µs +(WAL), batch 100 723 µs -> 1.08 ms, and **batch 1,000 6.17 -> 31.7 ms**. +Faster: WAL save into 1K existing 539 -> 282.8 µs, WAL flush 787 -> 530.1 µs, +session tick 5.76 -> 3.83 ms (1K) and 89.8 -> 35.5 ms (10K). The two single-save +figures were last updated on 2026-07-01 and the rest are undated, all on the +i7-12650H with `f32` stores. + +**Isolated afterwards, same machine, same day** (`--warm-up-time 1 +--measurement-time 3`, the store settings switched through the bench's +config): + +| | f16 + int8 (default) | f32 + int8 | f16 + f32 index | f32 + f32 index | v2.3.0 (f32) | +|---|---:|---:|---:|---:|---:| +| Single save (WAL) | 26.1 µs | 24.3 µs | 26.1 µs | 24.4 µs | 24.3 µs | +| Single save (no WAL) | 89.0 µs | 87.3 µs | 88.8 µs | 87.4 µs | 82.6 µs | +| Batch 100 | 1.09 ms | 0.90 ms | 1.09 ms | 0.90 ms | 0.87 ms | +| Batch 1,000 | 31.5 ms | 31.1 ms | 31.5 ms | 31.6 ms | 30.3 ms | + +The int8 index costs nothing on the write path. `float16` costs ~2 µs per +saved record — the rounding — which is the whole difference at 100 records +and within noise at 1,000. Built at the v2.3.0 tag (v2.2.0's benches do not +compile) and run on this machine, the write path measures the same as today, +so nothing regressed since then; the old 18 µs and 6.17 ms figures cannot be +reproduced on this hardware and came from an undated run on another machine. +`save_batch` scales linearly (about 12 µs per record from 500 to 8 000 +records); the bench's figure also includes dropping the store. --- @@ -726,12 +1042,16 @@ HDF5 persistence with optional Write-Ahead Log. Trivial/non-trivial classification for memory write filtering. +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. + +> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^gate_'` + | Check | Latency | |-------|---------| -| Trivial skip ("ok", "yes") | 61 ns | -| Short phrase skip | 86 ns | -| Non-trivial pass | 705 ns | -| Ratio check | 488 ns | +| Trivial skip ("ok", "yes") | 57.6 ns | +| Short phrase skip | 83.0 ns | +| Non-trivial pass | 570.5 ns | +| Ratio check | 395.2 ns | **Sub-microsecond filtering.** The gate decides whether to save a memory in under 1 µs. @@ -741,27 +1061,70 @@ Trivial/non-trivial classification for memory write filtering. End-to-end strategy evaluation including embedding operations. +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. + +> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^strategy_(save_every|semantic_shift)'` + | Strategy | Condition | Latency | |----------|-----------|---------| -| SaveEveryExchange (substantive) | Saves | 923 ns | -| SaveEveryExchange (trivial) | Skips | 67 ns | -| SaveOnSemanticShift (empty store) | Saves | 941 ns | +| SaveEveryExchange (substantive) | Saves | 732.5 ns | +| SaveEveryExchange (trivial) | Skips | 58.3 ns | +| SaveOnSemanticShift (empty store) | Saves | 734.9 ns | + +The two "Saves" rows were 923 ns and 941 ns in the previous table, undated +and from the original i7-12650H run. + +--- + +## Multi-modal Search + +`clawhdf5_agent::multimodal::MultiModalStore`. Each of N records carries two +384-dim embeddings: a text embedding of its caption and one of its primary +modality, cycling image / audio / video. `search_cross_modal` scores every +embedding of every record (2N vectors) and keeps each record's best; +`search_by_modality(Image, …)` scores only image embeddings (a third of the +records). Both are exact linear scans followed by a full sort; k = 10. Data is +from a fixed-seed generator, so every run sees the same corpus. + +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c (with the +new, uncommitted `crates/clawhdf5-agent/benches/multimodal_bench.rs` in the +working tree). Criterion, 50 samples. + +> **Run:** `cargo bench -p clawhdf5-agent --bench multimodal_bench` + +| Search | 1K records | 10K records | +|--------|-----------:|------------:| +| Cross-modal (all modalities, 2N vectors) | 842.0 µs | 8.44 ms | +| One modality (image, ~N/3 vectors) | 151.2 µs | 1.54 ms | + +Both scale linearly with N. Cross-modal search is the slow path: at 1K +records it takes 842.0 µs, where the flat single-vector cosine scan in +[Vector Search Latency](#vector-search-latency) takes 47.4 µs over 1K vectors, +and it has no index — at 10K it is 8.44 ms, slower than a 100K flat scan +(6.58 ms). No earlier figure exists for this section. --- ## Summary +Derived from the 2026-09-24 tables above (tank, AMD Ryzen 7 7800X3D, commit +5c8323c); no command of its own. + | Capability | Typical Latency | Scale | |------------|----------------|-------| -| **Full memory search** | <1 ms | 10K records | -| **Hybrid vector+keyword** | <200 µs | 1K records | -| **Knowledge graph query** | <25 µs | 1K entities | -| **Temporal range query** | <1 µs | 10K timestamps | -| **Memory write** | <20 µs | Per record (WAL group-commit append) | -| **Consolidation cycle** | <165 µs | 1K records | -| **Importance gate** | <1 µs | Per record | +| **Full memory search** (`HDF5Memory::hybrid_search`, p50) | 0.49 ms | 10K records | +| **Hybrid vector+keyword** (flat, weighted) | 106.8 µs | 1K records | +| **Knowledge graph query** (BFS) | 23.1 µs | 1K entities | +| **Temporal range query** | 622 ns | 10K timestamps | +| **Memory write** | 26.1 µs | Per record (WAL group-commit append) | +| **Consolidation cycle** | 115.2 µs | 1K records | +| **Importance gate** | 57.6–570.5 ns | Per record | -**The entire memory pipeline — search, retrieve, re-rank, filter — runs in single-digit milliseconds at agent-typical scales. Fast enough that memory becomes invisible infrastructure.** +The previous summary said "<25 µs" for a knowledge-graph query and "<20 µs" +for a memory write. The first holds again after the adjacency-index fix +(23.1 µs); the second does not (26.1 µs). + +**At agent-typical scales a full `HDF5Memory::hybrid_search` (vector + BM25 + fusion) over 10K records takes 0.49 ms at p50, and graph traversal, temporal queries, consolidation and the write gate each take well under a millisecond. At 100K records the hybrid search takes 4.69 ms.** --- @@ -796,9 +1159,21 @@ _Latency benchmarks generated with Criterion.rs (50-100 samples per benchmark). **Mode:** BM25-only retrieval — zero embeddings, `vector_weight=0.0`, `keyword_weight=1.0` -> **Run:** `cargo run --release --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s_cleaned.json` -> (~70 s for all 500 questions on the tank reference machine). Omit the path for the -> oracle variant; add `--limit N` for an evenly-strided subsample. +> **Run:** `cargo run --release -p clawhdf5-bench --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s.json` +> Omit the path for the oracle variant; add `--limit N` for an evenly-strided +> subsample. Without the `embeddings` feature this runs two modes, BM25 only +> and BM25 stemmed. + +The BM25-only and BM25-stemmed figures in this section were measured again on +2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, with the command +above. Every figure of those two modes that this document publishes was +reproduced exactly: BM25 Hit@1/5/10 and MRR at turn and session level, the +BM25 per-type session Hit@1 values quoted in prose below, and the stemmed +turn-level row plus its session Hit@1 in the tokenizer table. The run also +produced figures this document does not publish (stemmed session Hit@5, +Hit@10 and MRR, and per-type Hit@5/Hit@10/MRR for both modes), so there was +nothing to compare them with. Rows that need real embeddings (vector-only, +hybrid, RRF, re-ranking, the weight sweep) were not re-run. ### Full haystack — `longmemeval_s`, n=500 (the number to cite) @@ -1058,14 +1433,29 @@ extraction over the haystack, which this harness does not do. ### Search Latency (LongMemEval, n=500 queries) -| Metric | Latency | -|--------|---------| -| avg | 1,004 µs | -| p50 | 1,017 µs | -| p95 | 2,031 µs | -| p99 | 2,912 µs | +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, full +`longmemeval_s` haystack (47.7 sessions / 494 turns per question). -Sub-millisecond median search across variable-length chat histories. +> **Run:** `cargo run --release -p clawhdf5-bench --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s.json` + +| Metric | BM25 only | BM25 stemmed | +|--------|-----------|--------------| +| avg | 11,034.7 µs | 11,096.2 µs | +| p50 | 10,943.5 µs | 10,996.5 µs | +| p95 | 11,749.6 µs | 12,018.8 µs | +| p99 | 12,644.1 µs | 12,784.7 µs | + +About 11 ms per query. Each question gets its own store, and the timed call is +the first `hybrid_search_with` on that store. The first query on a store +builds its indexes (the "cold index build" column of the search harness), so +this is a first-query figure, not steady-state search; for that +see the [search harness](#current-search-harness-2026-09-24). + +The previous table (avg 1,004 µs, p50 1,017 µs) said "sub-millisecond median +search", which no longer holds. That figure was undated and predates the +full-haystack harness (added 2026-08-07, 7d6e269), so it was measured on the +much smaller `longmemeval_oracle` corpus; on that corpus tank measured avg +2,431 µs on 2026-08-05 (below). --- @@ -1075,23 +1465,34 @@ Sub-millisecond median search across variable-length chat histories. **Topics:** Personal info, food preferences, music, travel, work/schedule, hobbies **Queries:** 35 questions across 4 types -> **Run:** `cargo run --release --bin memory_arena` +> **Run:** `cargo run --release -p clawhdf5-bench --bin memory_arena` + +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. ### Results by Query Type | Query Type | N | Hit@1 | Hit@5 | Hit@10 | MRR | Avg Latency | |------------|---|-------|-------|--------|-----|-------------| -| single-session | 25 | 40.0% | 92.0% | 100.0% | 0.5788 | 7,853 µs | -| multi-session | 5 | 40.0% | 60.0% | 80.0% | 0.5333 | 7,887 µs | -| temporal | 3 | 33.3% | 66.7% | 66.7% | 0.5000 | 7,899 µs | -| knowledge-update | 2 | 0.0% | 50.0% | 50.0% | 0.2500 | 7,899 µs | -| **OVERALL** | **35** | **37.1%** | **82.9%** | **91.4%** | **0.5468** | **7,870 µs** | +| single-session | 25 | 40.0% | 92.0% | 100.0% | 0.5755 | 248.4 µs | +| multi-session | 5 | 40.0% | 60.0% | 80.0% | 0.5333 | 24.4 µs | +| temporal | 3 | 33.3% | 66.7% | 66.7% | 0.5000 | 28.0 µs | +| knowledge-update | 2 | 0.0% | 50.0% | 50.0% | 0.2500 | 38.7 µs | +| **OVERALL** | **35** | **37.1%** | **82.9%** | **91.4%** | **0.5444** | p50 **30.1 µs**, p95 49.8 µs | **Key findings:** - Hit@10 of 91.4% across all query types with BM25-only (no embeddings) - Single-session recall strongest at 100% Hit@10 - Knowledge-update hardest (requires temporal disambiguation) — would improve significantly with vector similarity -- Latency dominated by BM25 index build over 999 turns (~7.9 ms) +- Search takes about 30 µs at p50 over the 999 turns. All 35 queries run + against one store, and single-session's higher average (248.4 µs) includes + that store's first query, which builds its indexes. + +The previous table put every query at ~7.9 ms ("latency dominated by BM25 +index build over 999 turns"). That was before v2.4.0, when +`HDF5Memory::hybrid_search` rebuilt the BM25 index from scratch and rewrote +the store on every query; it now keeps the index for the life of the store. +The MRR moved slightly (single-session 0.5788 -> 0.5755, overall 0.5468 -> +0.5444); the Hit@k figures are unchanged. --- @@ -1099,46 +1500,89 @@ Sub-millisecond median search across variable-length chat histories. HDF5 file size at various record counts — 384-dimensional embeddings, 200-char text. -> **Run:** `cargo run --release --bin footprint_bench` +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. -### Uncompressed (no WAL) +> **Run:** `cargo run --release -p clawhdf5-bench --bin footprint_bench` -| Records | File Size | Raw Data | Bytes/Record | Throughput | -|---------|-----------|----------|--------------|------------| -| 100 | 176.4 KB | 169.5 KB | 1.8 KB | 100,000 rec/s | -| 1K | 1.7 MB | 1.7 MB | 1.8 KB | 109,643 rec/s | -| 10K | 17.0 MB | 16.6 MB | 1.7 KB | 118,100 rec/s | -| 50K | 85.0 MB | 82.8 MB | 1.7 KB | 110,723 rec/s | -| 100K | 169.8 MB | 165.6 MB | 1.7 KB | 111,422 rec/s | +**These are `float16` stores.** `footprint_bench` builds its stores with +`MemoryConfig::new`, which defaults to `float16` embeddings since 5c8323c, +although the binary still prints "Embedding: 384-dim f32". Its "Raw Data" +column, and so the ratios, count embeddings as `f32` (1,536 bytes per record) +plus text. The synthetic embeddings take only 1,000 distinct values, and each +record's text is fixed by `record_idx % 40` (a 40-word bank walked from that +start), so a store holds only 40 distinct texts. Both compress far better than +real data would; read the gzip figures as an upper bound, and see the next +paragraph for why the other tables are optimistic too. "MB" and "KB" are the +binary's units (powers of 1024). -**1.7 KB per record** — HDF5 overhead is near-zero. Ingestion throughput exceeds **100K records/sec**. +**No table here is uncompressed.** The store always deflates its string +datasets: `write_string_dataset` applies deflate level 6 to any string dataset +of 4 KiB or more, whatever `MemoryConfig::compression` says +(`crates/clawhdf5-agent/src/schema.rs`, `STRING_COMPRESS_THRESHOLD`, added in +4fa7e89 on 2026-06-03). With only 40 distinct texts, the chunk text costs +almost nothing on disk: at 10K records, 50-char text takes 804 B per record +and 1000-char text 944 B, so 950 more characters cost only 140 B. The +`float16` embeddings alone are 768 B per record, and 200 bytes of raw text +would bring that to about 968 B before any other field or overhead; the +measured 803–829 B is below that only because the text is compressed away. +Real text will not deflate like this, so expect more than 820 B per record +with real data. `MemoryConfig::compression` decides only whether the +embeddings are compressed; the gzip table below is that setting turned on. + +### `compression` off (no WAL) + +| Records | File Size | Raw Data (f32) | Bytes/Record | Throughput | +|---------|-----------|----------------|--------------|------------| +| 100 | 90.0 KB | 169.5 KB | 921 B | 75,914 rec/s | +| 1K | 810.4 KB | 1.7 MB | 829 B | 72,266 rec/s | +| 10K | 7.8 MB | 16.6 MB | 820 B | 74,509 rec/s | +| 50K | 38.3 MB | 82.8 MB | 803 B | 72,829 rec/s | +| 100K | 76.7 MB | 165.6 MB | 803 B | 71,102 rec/s | + +**About 820 B per record** at 10K, for this synthetic corpus (see above: the +text is deflated regardless and has only 40 distinct values, so real data +takes more). Ingestion runs at 71,102–75,914 records/sec. ### With Gzip Compression (level 6) -| Records | Compressed | Ratio | Bytes/Record | -|---------|------------|-------|--------------| -| 100 | 31.5 KB | 5.37x | 323 B | -| 1K | 277.1 KB | 6.12x | 283 B | -| 10K | 2.7 MB | 6.17x | 281 B | -| 50K | 13.4 MB | 6.17x | 281 B | -| 100K | 26.9 MB | 6.15x | 282 B | +| Records | Compressed | Ratio vs f32 raw | Bytes/Record | +|---------|------------|------------------|--------------| +| 100 | 17.2 KB | 9.85x | 176 B | +| 1K | 56.4 KB | 30.06x | 57 B | +| 10K | 471.3 KB | 35.97x | 48 B | +| 50K | 2.3 MB | 36.67x | 47 B | +| 100K | 4.5 MB | 36.73x | 47 B | -**6.2x compression ratio** — 100K agent memories in 27 MB compressed. - -### Text Length Comparison (10K records, no compression) +### Text Length Comparison (10K records, `compression` off) | Text Length | File Size | Bytes/Record | Throughput | |-------------|-----------|--------------|------------| -| short (50 chars) | 15.6 MB | 1.6 KB | 177,925 rec/s | -| medium (200 chars) | 17.0 MB | 1.7 KB | 172,152 rec/s | -| long (1000 chars) | 24.6 MB | 2.5 KB | 157,807 rec/s | +| short (50 chars) | 7.7 MB | 804 B | 97,615 rec/s | +| medium (200 chars) | 7.8 MB | 820 B | 89,597 rec/s | +| long (1000 chars) | 9.0 MB | 944 B | 51,462 rec/s | ### WAL Overhead (1K records) | Mode | File Size | Ingest Time | Overhead | |------|-----------|-------------|----------| -| No WAL | 1.7 MB | 5.7 ms | — | -| With WAL | 1.7 MB + 9 B WAL | 5.3 ms | ±8% (negligible) | +| No WAL | 810.4 KB | 10.8 ms | — | +| With WAL | 810.4 KB + 9 B WAL | 10.6 ms | −2% (negligible) | + +**Compared with the previous tables.** Those were undated `f32` stores from the +original i7-12650H run: 1.7 KB per record (169.8 MB at 100K), gzip 6.2x +(26.9 MB at 100K), and more than 100,000 records/sec. Two changes separate +those tables from these, not one. `float16` embeddings save 768 B per record. +The old tables (recorded by 2026-05-14) also predate 4fa7e89 (2026-06-03), +which began deflating string datasets of 4 KiB or more, so their text was +stored raw; with only 40 distinct texts, deflate now removes most of it. +An `f32` embedding plus 200 B of text is 1,536 + 200 = 1,736 B; `float16` +alone would bring that to about 968 B per record, not the measured 803–829 B. This run does not measure how the saving splits between the two +beyond that, nor explain why the gzip ratio rose so much further, nor +separate the lower throughput (and the 1K ingest going from 5.7 to 10.8 ms) +from the change of machine and store configuration. The +`f32` on-disk figures were not re-measured; the dated +[float16 study](#float16-embedding-storage-memoryconfigfloat16) compares the +two directly (100K × 384: 154.0 MiB `f32`, 80.8 MiB `float16`). --- @@ -1146,7 +1590,12 @@ HDF5 file size at various record counts — 384-dimensional embeddings, 200-char Hippocampal-inspired memory consolidation improves both retrieval quality and search speed. -> **Run:** `cargo run --release --bin consolidation_efficiency` +> **Run:** `cargo run --release -p clawhdf5-bench --bin consolidation_efficiency` + +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. The run +was stopped (after about 19 minutes on one core) while still computing a +100K row of the cycle-time table, so that row and the binary's memory-reduction +part were not produced; neither has ever been published here. ### Retrieval Quality Before vs. After Consolidation @@ -1159,17 +1608,22 @@ Hippocampal-inspired memory consolidation improves both retrieval quality and se | Hit@5 | 100.0% | 100.0% | — | | Hit@10 | 100.0% | 100.0% | — | | MRR | 1.0000 | 1.0000 | — | -| Search latency | 2,752 µs | 312 µs | **8.8x faster** | +| Search latency (avg) | 2,223.7 µs | 239.2 µs | **9.3x faster** | -Signal records survive consolidation because they are accessed 15+ times, giving them high decay scores. 900 noise records evicted, search speeds up 8.8x, and **zero quality loss** — perfect recall maintained. +Signal records survive consolidation because they are accessed 15+ times, giving them high decay scores. 900 noise records evicted, search speeds up 9.3x, and **zero quality loss** — perfect recall maintained. The consolidation cycle itself took 0.13 ms and promoted 2 records. ### Consolidation Cycle Time | Records | Cycle Time | Evictions | Promotions | |---------|-----------|-----------|------------| -| 100 | 21 µs | 100 | 0 | -| 1K | 345 µs | 1,000 | 0 | -| 10K | 17.3 ms | 10,000 | 0 | +| 100 | 17 µs | 100 | 0 | +| 1K | 189 µs | 1,000 | 0 | +| 10K | 2.16 ms | 10,000 | 0 | + +The 1K and 10K cycles were 345 µs and 17.3 ms in the previous, undated table +(before: 2,752 µs / after: 312 µs for search). Eviction's membership check +changed from a linear scan to a `HashSet` in 603fcf8 (2026-08-17), after those +figures were recorded; this run does not isolate its effect. --- @@ -1178,34 +1632,44 @@ Signal records survive consolidation because they are accessed 15+ times, giving In-memory key-value store with TTL, capacity eviction, and embedding search. No network hop, no serialization — direct HashMap operations. -> **Run:** `cargo run --release --bin ephemeral_perf` +> **Run:** `cargo run --release -p clawhdf5-bench --bin ephemeral_perf` + +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. ### Latency Comparison -| Operation | clawhdf5 Ephemeral | Redis (single-node)¹ | Speedup | +| Operation | clawhdf5 Ephemeral | Redis (single-node)¹ | Ratio¹ | |-----------|-------------------|---------------------|---------| -| SET | **356 ns/op** | ~25,000 ns/op | **70x** | -| GET (hit) | **179 ns/op** | ~25,000 ns/op | **140x** | -| GET (miss) | **62 ns/op** | ~25,000 ns/op | **403x** | -| DELETE | **124 ns/op** | ~25,000 ns/op | **202x** | -| SET+embedding | **268 ns/op** | N/A | — | +| SET | **366 ns/op** | ~25,000 ns/op (not measured) | 68.3x | +| GET (hit) | **115 ns/op** | ~25,000 ns/op (not measured) | 216.7x | +| GET (miss) | **57 ns/op** | — | — | +| DELETE | **99 ns/op** | — | — | +| SET+embedding | **236 ns/op** | N/A | — | -> ¹ Redis latency includes network round-trip (loopback). clawhdf5 ephemeral is in-process — no network. +> ¹ **The Redis figure is not a measurement.** It is a constant hard-coded in +> `ephemeral_perf` as a "typical single-node" latency, and the binary computes +> the two ratios from it; only the clawhdf5 column was measured. A typical +> Redis figure includes a network round-trip, which the in-process ephemeral +> tier does not have, so the ratios compare different things. ### Throughput | Operation | ops/sec | |-----------|---------| -| SET | 2,810,649 | -| GET | 5,584,684 | -| DELETE | 8,093,731 | -| SET+EMB (384d) | 3,725,877 | +| SET | 2,732,568 | +| GET (hit) | 8,666,607 | +| GET (miss) | 17,554,663 | +| DELETE | 10,109,895 | +| SET+EMB (384d) | 4,235,766 | ### Embedding Search (ephemeral tier) | Scale | Latency | |-------|---------| -| 10K entries @ 384d | 2.9 ms/query | +| 10K entries @ 384d | 2.85 ms/query | + +GET (hit) moved from 179 to 115 ns/op and DELETE from 124 to 99 ns/op (GET +throughput 5,584,684 -> 8,666,607 ops/sec); the previous figures were undated. --- @@ -1263,10 +1727,16 @@ Measured 2026-08-07 on tank (Ryzen 7 7800X3D, 246 MB dataset in page cache). ### Measured Platforms +The tank row was measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit +5c8323c, with the single benchmark rather than the whole script: + +> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^ivf_search_10k_nprobe10$'` + | Platform | CPU | 10K IVF Search | Notes | |----------|-----|----------------|-------| -| Linux x86_64 | Intel i7-12650H (10C, 4.7 GHz) | 27 µs | Primary CI target | -| macOS aarch64 | Apple M3 Max (14C) | ~18 µs | ~33% faster via NEON SIMD | +| Linux x86_64 | AMD Ryzen 7 7800X3D (8C/16T), tank | 24.8 µs | Measured 2026-09-24 | +| Linux x86_64 | Intel i7-12650H (10C, 4.7 GHz) | 27 µs | Undated, original run | +| macOS aarch64 | Apple M3 Max (14C) | ~18 µs | Undated; ~33% faster than the i7-12650H row. Not reproducible on the hardware available for the 2026-09-24 re-run | ### Reproducibility @@ -1610,6 +2080,11 @@ vs-libhdf5 validation above) to give both sections the dated, hardware-cited, reproducible citation the top-of-file traceability note flags them as missing. +> The sections this re-run checked were themselves re-measured on tank on +> 2026-09-24 and now show those figures; the i7-12650H figures this section +> compares against ("above", "at the top of this file") are quoted in those +> sections' notes. Kept as the 2026-08-05 record. + ### LongMemEval Results (reproduction) ```bash diff --git a/CHANGELOG.md b/CHANGELOG.md index 2996da6..c32606a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,17 @@ ## Unreleased ### Upgrade Notes +- **`clawhdf5-migrate` now writes a real agent store.** Its output used to be + a layout of its own (`/chunks`, `/sessions`, `/entities`, `/relations`, no + `/meta`) that `HDF5Memory::open` rejected, so a migrated file could not be + used as agent memory. Files it wrote before this release are not agent + stores; re-run the migration. Also: embeddings default to `float16` like + any new store (`--f32` opts out; `--float16` is a hidden no-op); a row with + the wrong embedding length is an error instead of being truncated or + padded; `--incremental` now matches rows by content against an existing + store and follows the source's deleted flags; a source with no memory rows + needs `--embedding-dim`. The per-dataset SHA-256 provenance attributes of + the old layout are gone (the agent schema has no place for them). - **Files written by clawhdf5 now open in h5py and libhdf5.** Every `f32` dataset we wrote — including every agent store's embeddings — was refused with "sign bit position out of bounds", and every empty dataset with @@ -46,6 +57,24 @@ `quantized_index = false`, or pass `create --f32-index` to the CLI, to opt out. The CLI's `--quantized-index` is still accepted but is now a no-op. +### Migration +- `clawhdf5-migrate`: writes through the agent's own API (`HDF5Memory::create` + / `open`, `save_batch`, the session cache and knowledge graph), so there is + no second copy of the schema. Sessions and entities/relations carry over; + deleted rows become deleted records (or are left out with + `--skip-deleted`). Every source row is checked before the output is created, + so a source that cannot be migrated leaves an existing store untouched. + Validation reads the result back with `HDF5Memory::open_read_only`, compares + every field (embeddings bit for bit — `round_to_f16` of the source for a + `float16` store) and checks that a migrated record is found by search. The + `half`-based conversion is gone; `clawhdf5_format::float16` is the only one. + 42 tests, including h5py opening a migrated store; an adversarial review's + two blocker and four major findings are fixed with regression tests. +- `clawhdf5-agent`: `HDF5Memory::sessions()` / `sessions_mut()`, + `HDF5Memory::delete_batch(&[usize])` (one save, all-or-nothing, never + auto-compacts), `SessionCache::add_at`, and `SessionCache` / `SessionEntry` + re-exported from the crate root. + ### Search - `clawhdf5-agent`: **`HDF5Memory::search` with `SearchOptions`** — source filtering, re-ranking and confidence rejection in the store's own search @@ -68,6 +97,20 @@ activation of the `k` results it returns, not of the whole `3k` candidate pool it re-ranks. +### Benchmarks +- Every undated or pre-September section of `BENCHMARKS.md` re-run on one + machine on one day (tank, 2026-09-24, commit 5c8323c), with the command for + each and every number traced back to the raw output by a separate check. + Where a figure moved, the section says so. Two apparent regressions were + isolated rather than published: knowledge-graph traversal (a real bug, + fixed above) and the write path, which measures the same at v2.3.0 on this + machine — the old 18 µs / 6.17 ms figures came from an undated run on other + hardware; `float16` adds ~2 µs per save and the int8 index nothing. +- New `multimodal_bench`: cross-modal search at 1K and 10K records, which the + README claimed but nothing measured. +- `footprint_bench` reports whether it built `float16` or `f32` stores and + takes `--f32`; it had kept printing "f32" after the default changed. + ### Interop - `clawhdf5-format`: **every `f32` dataset was unreadable by h5py and libhdf5.** The float datatype encoder hard-coded the sign bit's position to @@ -146,6 +189,14 @@ knew to ask; it now only ever switches the default off. ### Performance +- `clawhdf5-agent`: **knowledge-graph traversal was 6.5x slower than it + should be.** `bfs_neighbors` and `spreading_activation` built an adjacency + index over the whole graph on every call (1efd82c), so a 2-hop BFS over 1K + entities took 155 µs. The index is now cached on `KnowledgeCache` and + checked against a fingerprint of the graph on each use — one pass over + entity ids and relation endpoints, no allocation — so any change, including + direct edits of its public `Vec`s, still rebuilds it (tested). BFS over 1K + entities: 155.1 -> 23.1 µs; spreading activation over 100: 22.8 -> 10.1 µs. - `clawhdf5-format`, `clawhdf5-filters`: both deflate paths hand the codec the whole chunk in one call, into a buffer allocated once, instead of streaming it through a 32 KiB buffer: about 5% on chunked writes and 10% on zlib-ng's diff --git a/README.md b/README.md index a96553b..588a825 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ [![Rust](https://img.shields.io/badge/rust-1.92%2B-orange.svg)](https://www.rust-lang.org) [![Tests](https://img.shields.io/badge/tests-1850%2B-brightgreen.svg)](#building) [![LongMemEval](https://img.shields.io/badge/LongMemEval__s-Turn--Level%20Hit@5%2081.4%25%20hybrid-blue.svg)](BENCHMARKS.md#longmemeval-results) -[![Footprint](https://img.shields.io/badge/on--disk-1.7%20KB%2Frecord-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1) +[![Footprint](https://img.shields.io/badge/on--disk-~820%20B%2Frecord%20float16%2C%20synthetic%20text-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1) ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file. @@ -111,8 +111,8 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d | Keyword search | Separate FTS engine | Integrated BM25 | | Knowledge graph | Neo4j or none | In-file graph with spreading activation | | Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers | -| Temporal queries | Custom code | Native temporal index (716ns) | -| Multi-modal | Multiple stores | Unified cross-modal search | +| Temporal queries | Custom code | Native temporal index (622 ns range query over 10K) | +| Multi-modal | Multiple stores | Unified cross-modal search (exact scan: 842 µs over 1K records) | | Integrity | Hope for the best | Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance | | Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** | @@ -120,7 +120,7 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d ## Performance -Vector search and agent-memory operations below are benchmarked on Intel i7-12650H (10C/16T), 384-dim embeddings, Criterion.rs. The HDF5 Core I/O table immediately below is from a separate, independently reproduced run (see its own hardware note). +The brute-force/IVF vector search, agent-memory, on-disk footprint and consolidation figures below were measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D, 8C/16T), commit 5c8323c, 384-dim embeddings; the commands are in [BENCHMARKS.md](BENCHMARKS.md). Exceptions are marked where they appear: the HDF5 Core I/O table immediately below is from a separate, independently reproduced run (see its own hardware note), and the HNSW `f32`/`i8` table and the in-memory `i8` column were not re-measured on 2026-09-24. ### HDF5 Core I/O (vs libhdf5 1.14.6) @@ -154,33 +154,45 @@ and [§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quant | `f32` index | 0.9945 | 13 399 | 3.2 s | | `i8` index + exact re-score (**default for new stores**) | 0.9940 | **21 848** | **1.8 s** | -Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31. +Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31. These +two rows are a paired comparison (medians of alternating runs, same binary). +A single `f32` run on 2026-09-24 measured recall 0.9945, 19 001 QPS and a +2.7 s build; the int8 row was not re-run, so the pair has not been re-checked +([§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index)). -**Brute-force and IVF paths** (Criterion, i7-12650H): +**Brute-force and IVF paths** (Criterion, tank, 2026-09-24): -| Scale | Flat | IVF (nprobe=10) | IVF-PQ | vs MemX¹ | +| Scale | Flat | IVF (nprobe=10) | IVF-PQ | MemX¹ (claimed, end-to-end) | |-------|------|-----------------|--------|----------| -| 1K | **54 µs** | — | — | — | -| 10K | 753 µs | **27 µs** | — | — | -| 100K | 11.4 ms | 1.32 ms | **1.19 ms** | ~8–76× (see caveat) | +| 1K | **47.4 µs** | — | — | — | +| 10K | 500.5 µs | **24.8 µs** | — | — | +| 100K | 6.58 ms | 592 µs | **869 µs** | <90 ms | -> Reproduced on the same second machine (Ryzen 7 7800X3D) with a corrected, -> apples-to-apples SIMD/scalar/parallel comparison methodology — see -> [BENCHMARKS.md § Independent Validation: tank — LongMemEval & Vector -> Search](BENCHMARKS.md#independent-validation-tank--longmemeval--vector-search-ryzen-7-7800x3d-2026-08-05). +> These replace figures from the original i7-12650H run (flat 54 µs / 753 µs / +> 11.4 ms); a 2026-08-05 run on tank had already matched the new ones — see +> [BENCHMARKS.md § Vector Search Latency](BENCHMARKS.md#vector-search-latency). ### Agent Memory Operations | Operation | Latency | Scale | |-----------|---------|-------| -| Hybrid search (`HDF5Memory::hybrid_search`, p50) | **70 µs** / 0.49 ms / 4.65 ms | 1K / 10K / 100K records | -| BM25 keyword search | **67 µs** | 1K records | -| Knowledge graph BFS | **24 µs** | 1K entities | -| Spreading activation | **17 µs** | 100 entities | -| Temporal range query | **716 ns** | 10K timestamps | -| Consolidation cycle | **164 µs** | 1K records | -| Memory write (WAL) | **18 µs** | per record (group-commit append; HDF5 batched at flush) | -| Importance gate | **61 ns** | per record | +| Hybrid search (`HDF5Memory::hybrid_search`, p50) | **0.07 ms** / 0.49 ms / 4.69 ms | 1K / 10K / 100K records | +| BM25 keyword search | **20.4 µs** | 1K records | +| Knowledge graph BFS | **23.1 µs** | 1K entities | +| Spreading activation | **10.1 µs** | 100 entities | +| Temporal range query | **622 ns** | 10K timestamps | +| Consolidation cycle | **115.2 µs** | 1K records | +| Cross-modal search (exact scan, 2 embeddings per record) | **842.0 µs** / 8.44 ms | 1K / 10K records | +| Memory write (WAL) | **26.1 µs** | per record (group-commit append; HDF5 batched at flush) | +| Importance gate | **57.6 ns** | per record (trivial skip) | + +The old 18 µs WAL write was undated, from another machine: v2.3.0 measures +24.3 µs on the same hardware as this table, the same as an `f32` store today. +`float16` stores (the new default) add ~2 µs for rounding; the int8 index adds +nothing. See [BENCHMARKS.md § Write Path](BENCHMARKS.md#write-path). +Knowledge-graph traversal was briefly 6.5x slower (155 µs) until this re-run +found and fixed an adjacency index rebuilt on every traversal; see +[§ Knowledge Graph](BENCHMARKS.md#knowledge-graph). ### Chunked Write Throughput (codec comparison) @@ -195,7 +207,7 @@ by default (AoS→SoA byte transpose, +157–204% throughput for float data): Use `.with_zstd(3)` or `.with_deflate(6)` for write-heavy workloads — both now perform at ~720–750 MiB/s on large matrices. Use `.with_pcodec()` for write-once/read-many workloads where compression ratio matters more than encode speed. Disable auto-shuffle with `.without_shuffle()` for byte arrays that don't benefit from AoS→SoA transposition. -> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search). The ratio overstates the real advantage by an unquantified margin — order-of-magnitude indication only. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171). +> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search), so the two columns are not comparable and no ratio is given. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171). ### LongMemEval Retrieval Recall @@ -248,18 +260,26 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points. ### Memory Footprint -**On disk** — 384-dim embeddings, 200-char text +**On disk** — 384-dim `float16` embeddings (the default for new stores), +200-char text, `footprint_bench` ([BENCHMARKS.md § Memory Footprint](BENCHMARKS.md#memory-footprint-1)): | Records | File Size | Bytes/Record | Gzip-6 compressed | |---------|-----------|--------------|-------------------| -| 1K | 1.7 MB | 1.8 KB | 277 KB (6.1x) | -| 10K | 17.0 MB | 1.7 KB | 2.7 MB (6.2x) | -| 100K | 169.8 MB | 1.7 KB | 26.9 MB (6.2x) | +| 1K | 810.4 KB | 829 B | 56.4 KB | +| 10K | 7.8 MB | 820 B | 471.3 KB | +| 100K | 76.7 MB | 803 B | 4.5 MB | -These figures are `f32` embeddings. New agent stores default to -`MemoryConfig::float16`, which halves them: 100K × 384 records take 80.8 MiB -instead of 154.0. +The benchmark's synthetic embeddings and text are far more repetitive than +real data (only 40 distinct texts), so no column here is an expectation for +real data. The compressed column is an upper bound, and the Bytes/Record +column is optimistic too: it is not an uncompressed figure, because the store +always deflates its text (any string dataset of 4 KiB or more) whatever +`MemoryConfig::compression` says. The `float16` embeddings alone are 768 B per +record, so 200 characters of real text would take a record above 820 B. +This table used to show `f32` stores (1.7 KB per record, 169.8 MB at 100K); +those were not re-measured. The float16 study compares the two on the same +data: 100K × 384 records take 80.8 MiB as `float16` and 154.0 MiB as `f32`. **In memory** — a store reopened from disk, 384-dim `f32`, measured with a counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-footprint)): @@ -271,7 +291,8 @@ counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-foo | 100K | 146 MiB | 399 MiB (2.72x) | **256 MiB (1.74x)** | Down from 505 MiB (3.44x) at 100K before v2.6.0, when the cache held every -embedding twice. +embedding twice. The `f32` column was re-measured on 2026-09-24 and reproduced +exactly; the `i8` column was not re-run. ### Consolidation Efficiency @@ -282,7 +303,10 @@ embedding twice. |--------|--------|-------|-------| | Records in store | 1,000 | 100 | −90% | | Hit@1 recall (signal records) | 100% | 100% | no loss | -| Search latency | 2.75 ms | 0.31 ms | **8.8x faster** | +| Search latency (avg) | 2.22 ms | 0.24 ms | **9.3x faster** | + +The consolidation cycle that does this took 0.13 ms; at 10K records a cycle +takes 2.16 ms. **Full benchmark details: [BENCHMARKS.md](BENCHMARKS.md)** @@ -754,9 +778,37 @@ Replace in `Cargo.toml` and source: ```bash cargo install --path crates/clawhdf5-migrate -clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedding-dim 384 +clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedder minilm ``` +The output is an ordinary `clawhdf5-agent` store, written through the agent's +own API: open it with `HDF5Memory::open` (or `clawhdf5-cli --path memory.h5 …`) +and search it straight away. What carries over from the ZeroClaw tables: + +| SQLite | Agent store | +|--------|-------------| +| `memory_chunks` | memory records (text, embedding, source channel, timestamp, session id, tags); rows with `deleted = 1` become deleted records, or are left out with `--skip-deleted` | +| `sessions` | sessions (id, start/end index, channel, summary, timestamp) | +| `entities`, `relations` | knowledge graph entities and relations; entities get new ids and relations are re-pointed at them | + +The chunk `id` column has no counterpart in the agent store, so records are +written in `id` order and numbered from 0. Embeddings are stored as float16 +like any new store; `--f32` keeps full precision (and is required for values +beyond ±65504). The embedding dimension is detected from the first row unless +`--embedding-dim` is given, and every row must have it: a row of another length +is an error, never truncated or padded. A source with no memory records (only +sessions or the graph) needs `--embedding-dim`, since a store's dimension is +fixed when it is created. Every row is checked before the output is created, +so a source that cannot be migrated leaves an existing store at `--hdf5` as it +was. `--incremental` adds to an existing store only the rows it does not +already hold; the source must have the store's dimension, and records already +in the store take the source's deleted flag (a row deleted in SQLite since the +last run is deleted in the store; one un-deleted there is written again, as +the agent has no un-delete). The tool reads the result back with +`HDF5Memory::open_read_only`, compares it with the source (every row with +`--validate-full`) and checks that a migrated record is found by search; +`--dry-run` only counts the rows. + --- ## Roadmap diff --git a/crates/clawhdf5-agent/Cargo.toml b/crates/clawhdf5-agent/Cargo.toml index b8d79ee..8af2c6f 100644 --- a/crates/clawhdf5-agent/Cargo.toml +++ b/crates/clawhdf5-agent/Cargo.toml @@ -45,6 +45,10 @@ harness = false name = "memory_bench" harness = false +[[bench]] +name = "multimodal_bench" +harness = false + [features] default = ["float16", "hnsw", "parallel"] float16 = ["half"] diff --git a/crates/clawhdf5-agent/benches/multimodal_bench.rs b/crates/clawhdf5-agent/benches/multimodal_bench.rs new file mode 100644 index 0000000..798a035 --- /dev/null +++ b/crates/clawhdf5-agent/benches/multimodal_bench.rs @@ -0,0 +1,107 @@ +//! Multi-modal memory search benchmarks (`clawhdf5_agent::multimodal`). +//! +//! Covers `MultiModalStore::search_cross_modal` (every embedding of every +//! record, whatever its modality) and, for comparison, +//! `MultiModalStore::search_by_modality` restricted to one modality. +//! +//! Corpus: N records (1K and 10K), each carrying two 384-dim embeddings — +//! a text embedding of its caption plus one embedding of its primary modality, +//! cycling Image / Audio / Video — so a cross-modal query scores 2N vectors. +//! All data comes from a fixed-seed LCG, so every run sees the same corpus. +//! +//! Run: `cargo bench -p clawhdf5-agent --bench multimodal_bench` + +use std::collections::HashMap; + +use clawhdf5_agent::multimodal::{ + MediaRef, ModalEmbedding, Modality, MultiModalRecord, MultiModalStore, +}; +use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main}; + +// --------------------------------------------------------------------------- +// Simple deterministic PRNG (LCG), same as the other agent benches +// --------------------------------------------------------------------------- + +struct Rng(u32); + +impl Rng { + fn new(seed: u32) -> Self { + Self(seed) + } + fn next_u32(&mut self) -> u32 { + self.0 = self.0.wrapping_mul(1103515245).wrapping_add(12345); + self.0 >> 16 + } + fn next_f32(&mut self) -> f32 { + self.next_u32() as f32 / 65536.0 - 0.5 + } +} + +fn make_vec(rng: &mut Rng, dim: usize) -> Vec { + (0..dim).map(|_| rng.next_f32()).collect() +} + +// --------------------------------------------------------------------------- +// Corpus +// --------------------------------------------------------------------------- + +const DIM: usize = 384; +const K: usize = 10; + +const MEDIA: [(Modality, &str, &str); 3] = [ + (Modality::Image, "image/png", "clip-vit-base"), + (Modality::Audio, "audio/wav", "clap-base"), + (Modality::Video, "video/mp4", "xclip-base"), +]; + +fn build_store(n: usize, seed: u32) -> MultiModalStore { + let mut rng = Rng::new(seed); + let mut store = MultiModalStore::new(); + for i in 0..n { + let (modality, mime, model) = &MEDIA[i % MEDIA.len()]; + let embeddings = vec![ + ModalEmbedding::new(Modality::Text, make_vec(&mut rng, DIM), "minilm-l6"), + ModalEmbedding::new(modality.clone(), make_vec(&mut rng, DIM), *model), + ]; + store.add_record(MultiModalRecord { + id: 0, + primary_modality: modality.clone(), + text_content: Some(format!("{modality} memory {i}")), + media_ref: Some(MediaRef::path(format!("/media/{i}"), *mime)), + embeddings, + observation: None, + timestamp: 1_700_000_000.0 + i as f64, + metadata: HashMap::new(), + }); + } + store +} + +// --------------------------------------------------------------------------- +// Benchmarks +// --------------------------------------------------------------------------- + +fn multimodal_search_benches(c: &mut Criterion) { + let query = make_vec(&mut Rng::new(99), DIM); + + let mut group = c.benchmark_group("multimodal_search"); + group.sample_size(50); + + for (label, n) in [("1k", 1_000usize), ("10k", 10_000)] { + let store = build_store(n, 42); + assert_eq!(store.count(), n); + + group.bench_with_input(BenchmarkId::new("cross_modal", label), &n, |b, _| { + b.iter(|| store.search_cross_modal(&query, K)); + }); + + group.bench_with_input(BenchmarkId::new("by_modality_image", label), &n, |b, _| { + b.iter(|| store.search_by_modality(&Modality::Image, &query, K)); + }); + } + + group.finish(); +} + +criterion_group!(multimodal_benches, multimodal_search_benches); +criterion_main!(multimodal_benches); diff --git a/crates/clawhdf5-bench/src/bin/footprint_bench.rs b/crates/clawhdf5-bench/src/bin/footprint_bench.rs index bce5748..477fd50 100644 --- a/crates/clawhdf5-bench/src/bin/footprint_bench.rs +++ b/crates/clawhdf5-bench/src/bin/footprint_bench.rs @@ -11,12 +11,14 @@ //! //! Configuration matrix: //! - Text lengths: short (50 chars), medium (200 chars), long (1000 chars) -//! - Embedding: 384-dim f32 (1536 bytes raw per record) +//! - Embedding: 384-dim, stored as float16 (the default for new stores) or +//! f32 with `--f32`; "raw" bytes are counted as f32 input either way //! - WAL: enabled and disabled //! //! # Usage //! ``` -//! cargo run --release --bin footprint_bench +//! cargo run --release --bin footprint_bench # float16 stores +//! cargo run --release --bin footprint_bench -- --f32 # f32 stores //! ``` use std::time::Instant; @@ -24,6 +26,9 @@ use std::time::Instant; use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry}; use tempfile::TempDir; +/// `--f32`: build f32 stores instead of the library's float16 default. +static F32: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false); + const EMBEDDING_DIM: usize = 384; // Raw bytes per record: 384 f32 embeddings + median text + overhead @@ -152,6 +157,9 @@ fn measure_footprint( config.compression = compression; config.compression_level = if compression { 6 } else { 0 }; config.compact_threshold = 0.0; + if F32.load(std::sync::atomic::Ordering::Relaxed) { + config.float16 = false; + } let mut memory = HDF5Memory::create(config).expect("HDF5Memory::create failed"); @@ -241,11 +249,19 @@ fn fmt_n(n: usize) -> String { // --------------------------------------------------------------------------- fn main() { + if std::env::args().skip(1).any(|a| a == "--f32") { + F32.store(true, std::sync::atomic::Ordering::Relaxed); + } + let stored = if F32.load(std::sync::atomic::Ordering::Relaxed) { + "f32 (1,536 bytes per record)" + } else { + "float16 (768 bytes per record; the default for new stores)" + }; println!("================================================================="); println!(" ClawhDF5 Memory Footprint Benchmark"); println!("================================================================="); println!(); - println!("Embedding: 384-dim f32 = 1,536 bytes raw per record"); + println!("Embedding: 384-dim, stored as {stored}; raw input counted as f32"); println!("Text lengths: short=50 chars, medium=200 chars, long=1000 chars"); println!();