Merge pull request 'clawhdf5-migrate writes real agent stores; knowledge-graph fix; dated benchmark re-run' (#6) from feat/migrate-and-benchmarks into main

Reviewed-on: #6
2026-09-25 14:52:35 +00:00
parent a7920bd4b3 dce5559ff2
commit c0f704c381
17 changed files with 2829 additions and 1314 deletions
@@ -2,20 +2,40 @@

 > Pure Rust. Zero C dependencies. Single file. Fast enough to forget it's there.

-**System:** Intel i7-12650H (10C/16T, 4.7 GHz boost) · 32 GB DDR5 · Linux 6.8.0  
+**System (original run):** Intel i7-12650H (10C/16T, 4.7 GHz boost) · 32 GB DDR5 · Linux 6.8.0  
 **Rust:** 1.96.0-nightly (2026-03-14) · `--release` profile  
 **Date:** 2026-07-01

-> **Traceability note:** the "h5bench-Equivalent I/O Benchmarks" and both
-> "Independent Validation: tank" sections below meet a dated,
-> hardware-cited, reproducible standard (explicit date, machine spec, and a
-> runnable command per result) — this now covers "LongMemEval Results",
-> "SIMD & Parallelism", "Vector Search Latency", and "Comparison to MemX" via
-> their tank re-runs. The remaining undated sections above (Hybrid Search,
-> Knowledge Graph, Memory Consolidation, Temporal Index, Write Path, Decision
-> Gate, Memory Strategy, Multi-Session Benchmark, Memory Footprint,
-> Consolidation Efficiency, Ephemeral Tier) do not yet meet that bar — this is
-> a known, tracked documentation gap, not a claim that those numbers are wrong.
+Sections that carry their own "Measured …" line were measured there instead,
+not on the system above. The 2026-09-24 re-run was on tank (AMD Ryzen 7
+7800X3D, 8C/16T, Linux 7.0.0-34-generic, rustc 1.98.1) at commit 5c8323c,
+serially, with no `--warm-up-time`/`--measurement-time` overrides, waiting
+before each command while the 1-minute load average was above 2.0. The
+`bench` targets ran at Criterion's defaults (3 s warm-up, 5 s target
+measurement time, 100 samples). Every `memory_bench` group that was run
+(`vector_search_latency`, `hybrid_search`, `knowledge_graph`,
+`consolidation`, `temporal`) and the multi-modal group set
+`sample_size(50)`, so those sections took 50 samples, not 100. The 5 s is a
+target: Criterion stretched it where 5 s could not hold the samples it needed
+(for example 8.5 s for `vector_search_latency/bench_cosine_search/100k`,
+15.5 s for the 1K consolidation cycle, 252 s for `tick_session_10k`).
+
+> **Traceability note:** a section meets the dated, hardware-cited,
+> reproducible standard when it gives an explicit date, the machine, and a
+> runnable command for its results. As of the 2026-09-24 re-run on tank that
+> covers Vector Search Latency, Comparison to MemX, SIMD & Parallelism, Hybrid
+> Search, Knowledge Graph, Memory Consolidation, Temporal Index, Write Path,
+> Decision Gate, Memory Strategy, Summary (derived from those), LongMemEval
+> Results (the BM25 rows), Multi-Session Benchmark, Memory Footprint (on disk
+> and in memory), Consolidation Efficiency, Ephemeral Tier, Multi-modal Search,
+> the Search and Read harnesses, and the "h5bench-Equivalent I/O Benchmarks"
+> and "Independent Validation: tank" sections. What does not yet meet that bar:
+> the LongMemEval rows that need real embeddings (not re-run here, except the
+> dated float16 comparison), the Consolidation Efficiency 100K cycle row and
+> memory-reduction part (the 2026-09-24 run was stopped before it produced
+> them), the int8 side of "Quantising the index copy" (not re-run), and the
+> i7-12650H and macOS M3 Max rows under Cross-Platform Notes. That is a
+> known, tracked documentation gap, not a claim that those numbers are wrong.
 >
 > **Correctness note (2026-08-06).** Being dated and reproducible is necessary but
 > not sufficient — a number can be perfectly reproducible and still measure the
@@ -57,6 +77,23 @@ What remains at 2.43x: the flat vectors (1.0x), the HNSW index's own copy of
 them (1.0x), and text, ids and graph (~0.4x). The index copy is the next
 target — it is what a quantised or borrowed representation would address.

+**Current figures (`f32` index).** Measured 2026-09-24 on tank (AMD Ryzen 7
+7800X3D), commit 5c8323c:
+
+> **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --footprint --full`
+
+| N | vectors (raw) | entries MiB | store MiB | indexes MiB | reopened MiB | peak during open MiB | reopened / raw |
+|---:|---:|---:|---:|---:|---:|---:|---:|
+| 1 000 | 1 | 2 | 0 | 2 | 4 | 5 | 2.40x |
+| 10 000 | 15 | 17 | 12 | 32 | 44 | 61 | 3.03x |
+| 100 000 | 146 | 172 | 60 | 266 | 399 | 562 | 2.72x |
+
+These are the "reopened, f32" figures in the next table, reproduced exactly.
+They are above the "after" column of the table above (44 MiB vs 35 MiB at 10K,
+399 vs 357 MiB at 100K): the 2.43x was measured for 2e7e045, and the store
+already measured 399 MiB (2.72x) when the int8 index landed later the same
+day (c0a9206). The git log does not say what changed in between.
+
 ### Quantising the index copy (`quantized_index`)

 `MemoryConfig::quantized_index` stores the index's copy as `i8` instead of
@@ -108,6 +145,20 @@ second**, builds **1.8x faster**, and holds a quarter of the vectors. (Compare
 only at equal `ef`: with re-scoring the harness raises `ef` to at least the
 candidate pool, so the `ef = 16` and `ef = 32` rows are not like-for-like.)

+**Re-check, 2026-09-24.** Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D),
+commit 5c8323c. The `f32` memory columns of the first table reproduce exactly
+(see [Memory footprint](#memory-footprint)). Only the `f32` side of the speed
+table was re-run, once, not as alternating medians:
+
+> **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --full`
+
+At N = 100 000, `ef = 64`, the `f32` index measured recall@10 0.9945 (same),
+19 001 QPS (was 13 399) and a 2700.8 ms build (was 3197 ms). The QPS figure
+moved by more than 20%; the log does not explain it, and a single run is not
+the same-moment paired comparison the table above is. The int8 columns were
+not re-run, so the 1.63x ratio has not been re-checked against the new `f32`
+figure.
+
 #### On ARM (Raspberry Pi 5, Cortex-A76)

 `dot_i8` has two aarch64 kernels: `SDOT` for CPUs with the ARMv8.2
@@ -268,6 +319,10 @@ high-water mark alongside the retained figure:
 | 10 000 | 44 | 61 |
 | 100 000 | 399 | 562 |

+Reproduced exactly on 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit
+5c8323c, by `search_harness -- --footprint --full` (table under
+[Memory footprint](#memory-footprint)).
+
 The peak is set *after* the parse, by the index build, so a buffer allocated
 and freed during the parse never reaches the high-water mark. Holding a
 deliberate extra copy of the file across the whole parse leaves the peak
@@ -359,6 +414,50 @@ contiguous dataset is converted straight from the file bytes (one copy instead
 of two), and the native-endian conversions no longer zero a buffer they are
 about to overwrite.

+### Current: read harness (2026-09-24)
+
+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
+
+> **Run:** `cargo run --release -p clawhdf5-bench --bin read_harness`
+
+4096 x 2048 f64 (64 MB per dataset), chunks 256 x 256, file 129 MB
+
+| layout | read | selected | time ms | MB/s of selection | vs full read |
+|---|---|---:|---:|---:|---:|
+| chunked + deflate | full (first) | 64 MB | 65.6 | 975 | |
+| chunked + deflate | full (repeat) | 64 MB | 63.2 | 1013 | 1.00x |
+| chunked + deflate | 64 x 64 window (1 chunk) | 0.03 MB | 0.18 | 177 | 0.003x |
+| chunked + deflate | 512 x 512 window (4-9 chunks) | 2.00 MB | 4.12 | 485 | 0.063x |
+| chunked + deflate | one row | 0.02 MB | 0.99 | 16 | 0.015x |
+| chunked + deflate | one column | 0.03 MB | 2.02 | 15 | 0.031x |
+| chunked | full (first) | 64 MB | 60.5 | 1057 | |
+| chunked | full (repeat) | 64 MB | 56.7 | 1129 | 1.00x |
+| chunked | 64 x 64 window (1 chunk) | 0.03 MB | 0.10 | 325 | 0.002x |
+| chunked | 512 x 512 window (4-9 chunks) | 2.00 MB | 3.19 | 628 | 0.053x |
+| chunked | one row | 0.02 MB | 0.05 | 284 | 0.001x |
+| chunked | one column | 0.03 MB | 0.49 | 64 | 0.008x |
+| contiguous | full (first) | 64 MB | 30.1 | 2124 | |
+| contiguous | full (repeat) | 64 MB | 26.1 | 2452 | 1.00x |
+| contiguous | 64 x 64 window (1 chunk) | 0.03 MB | 0.08 | 394 | 0.003x |
+| contiguous | 512 x 512 window (4-9 chunks) | 2.00 MB | 3.36 | 595 | 0.112x |
+| contiguous | one row | 0.02 MB | 0.03 | 486 | 0.001x |
+| contiguous | one column | 0.03 MB | 2.61 | 12 | 0.087x |
+
+Full reads match the "after" column above (chunked + deflate 69 ms then, 63.2
+to 65.6 ms now). The compressed selections are faster than in "After: partial
+reads" (64 x 64 window 0.39 -> 0.18 ms, one row 2.69 -> 0.99 ms, one column
+5.23 -> 2.02 ms) and match the 2026-09-23 figures in
+[Deflate backend](#deflate-backend-zlib-rs-vs-zlib-ng) (0.18, 1.00 and 2.01
+ms), which were taken after the zlib-rs switch and the one-shot codec calls
+described there; this run does not isolate which change accounts for it.
+
+The uncompressed windows went the other way. Against "After: partial reads",
+the 512 x 512 window is about 60% slower (chunked 1.99 -> 3.19 ms, contiguous
+2.12 -> 3.36 ms) and the chunked 64 x 64 window went from 0.06 to 0.10 ms.
+The rows and columns of the uncompressed layouts are within 20% (chunked
+column 0.45 -> 0.49 ms, contiguous column 2.55 -> 2.61 ms). This run does not
+explain the slower windows.
+
 ## Search harness baseline (v2.3.0)

 Produced by `cargo run --release -p clawhdf5-bench --bin search_harness -- --full`
@@ -596,68 +695,217 @@ Recall on clustered data is unchanged or slightly better (100K, `ef = 64`:
 0.474 -> 0.444), the cost of batch members not seeing each other while
 planning; batches are capped at 1/16 of the graph and 512 nodes.

+### Current: search harness (2026-09-24)
+
+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, default
+features (so the build is batched and `parallel`, 16 threads). The HNSW tables
+use an `f32` index. The end-to-end table uses `MemoryConfig::new`'s defaults,
+which at this commit are `float16` embeddings and the int8 index; every
+earlier end-to-end table in this section was an `f32` store with an `f32`
+index.
+
+> **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --full`
+
+#### HNSW, N = 1000, dim = 384, M = 16, ef_construction = 64, storage = Float32
+
+build: 14.7 ms (67954 vectors/s) · exact scan: 4403 QPS, p50 225 µs
+
+| ef | recall@10 | QPS | p50 µs | p99 µs |
+|---:|---:|---:|---:|---:|
+| 16 | 0.9940 | 134065 | 7 | 18 |
+| 32 | 1.0000 | 105134 | 9 | 21 |
+| 64 | 1.0000 | 77541 | 13 | 26 |
+| 128 | 1.0000 | 36938 | 26 | 35 |
+| 256 | 1.0000 | 19249 | 52 | 63 |
+
+#### HNSW, N = 10000, dim = 384, M = 16, ef_construction = 64, storage = Float32
+
+build: 135.6 ms (73736 vectors/s) · exact scan: 435 QPS, p50 2295 µs
+
+| ef | recall@10 | QPS | p50 µs | p99 µs |
+|---:|---:|---:|---:|---:|
+| 16 | 0.9975 | 79614 | 12 | 24 |
+| 32 | 1.0000 | 70556 | 14 | 21 |
+| 64 | 1.0000 | 51313 | 19 | 29 |
+| 128 | 1.0000 | 22846 | 44 | 57 |
+| 256 | 1.0000 | 10267 | 97 | 119 |
+
+#### HNSW, N = 100000, dim = 384, M = 16, ef_construction = 64, storage = Float32
+
+build: 2700.8 ms (37027 vectors/s) · exact scan: 43 QPS, p50 23433 µs
+
+| ef | recall@10 | QPS | p50 µs | p99 µs |
+|---:|---:|---:|---:|---:|
+| 16 | 0.9180 | 20043 | 47 | 104 |
+| 32 | 0.9775 | 21348 | 44 | 104 |
+| 64 | 0.9945 | 19001 | 51 | 101 |
+| 128 | 0.9995 | 13458 | 73 | 137 |
+| 256 | 0.9995 | 5128 | 196 | 282 |
+
+#### End to end: `HDF5Memory::hybrid_search` (k = 10, weights 0.7 / 0.3)
+
+| N | ingest ms | cold index build ms | checkpoint ms | open ms | first query after open ms | p50 ms | p99 ms | QPS |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 1000 | 15 | 14 | 4.4 | 1.9 | 1.5 | 0.07 | 0.08 | 13869.5 |
+| 10000 | 146 | 126 | 33.5 | 18.3 | 14.8 | 0.49 | 0.50 | 2044.1 |
+| 100000 | 1498 | 1833 | 495.7 | 259.9 | 155.1 | 4.69 | 5.22 | 212.7 |
+
+Steady-state query latency is where the v2.4.0 work left it (p50 0.07 / 0.49 /
+4.69 ms against 0.07 / 0.49 / 4.65 ms). Figures that moved by more than 20%:
+
+- **Build.** 14.7 / 135.6 / 2700.8 ms against 34 / 388 / 5.9 s for
+  "batched + `parallel`" above. The git log does not explain the difference.
+- **Cold index build** in the end-to-end table: 14 / 126 / 1833 ms, down from
+  112 / 1487 / 20285 ms in the "unranked keyword scores" table, which predates
+  the batched and parallel build described above.
+- **QPS at 100K, `ef = 64`**: 19 001 against 13 980 in the "unit-vector"
+  table, with recall 0.9945 against 0.984; the recall gain is the one the
+  batched build reported. The QPS gain is not explained by the log.
+- **Other HNSW QPS cells**, against the same "unit-vector" tables: 10K
+  `ef = 16` 79 614 against 54 608 (+46%); 100K `ef = 16` 20 043 against
+  15 139 (+32%), `ef = 128` 13 458 against 10 959 (+23%), `ef = 256` 5 128
+  against 3 731 (+37%). The other cells, and every 1K cell, are within 20%.
+  The log does not explain these either.
+- **Open** in the end-to-end table: 1.9 / 18.3 / 259.9 ms against 1.1 / 13.7
+  / 353.1 ms in the "unranked keyword scores" table, slower at 1K and 10K and
+  faster at 100K. This run does not isolate the cause.
+- **Checkpoint at 100K**: 495.7 ms against 728.9 ms, and ingest up 11 -> 15,
+  104 -> 146 ms at 1K / 10K. This store is `float16` with an int8 index, the
+  earlier ones `f32`; the [float16 study](#float16-embedding-storage-memoryconfigfloat16)
+  measured float16 checkpoints at 512 ms against 752 ms for `f32` at 100K.
+  This run does not isolate the ingest change.
+
 ## Vector Search Latency

 Brute-force cosine similarity over 384-dimensional embeddings (OpenAI text-embedding-3-small size).

+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
+
+> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(vector_search_1k|simd_cosine_10k|simd_cosine_100k|prenorm_search_10k|ivf_search_10k_nprobe10|ivf_search_100k_nprobe10|ivf_pq_search_100k|rairs_search_10k_nprobe10)$'`
+
 | Scale | Flat Search | Pre-norm | IVF (nprobe=10) | IVF-PQ | RAIRS |
 |-------|-------------|----------|-----------------|--------|-------|
-| **1K** | 54 µs | 62 µs | — | — | — |
-| **10K** | 753 µs | 706 µs | 27 µs | — | 159 µs |
-| **100K** | 11.4 ms | — | 1.32 ms | 1.19 ms | — |
+| **1K** | 47.4 µs | — | — | — | — |
+| **10K** | 500.5 µs | 316.5 µs | 24.8 µs | — | 124.0 µs |
+| **100K** | 6.58 ms | — | 592 µs | 869 µs | — |

-**Key insight:** At 10K records (typical agent memory), IVF search delivers **27 µs** — that's 26x faster than flat search. Even at 100K records, IVF-PQ keeps search under **1.2 ms**.
+The 1K pre-norm cell is blank because no 1K pre-norm benchmark exists (the
+old 62 µs has no source). `memory_bench`'s own cosine search
+(`cargo bench -p clawhdf5-agent --bench memory_bench -- '^vector_search_latency/'`)
+agrees: 47.6 µs / 505.1 µs / 6.67 ms at 1K / 10K / 100K.
+
+Most cells moved by more than 20% from the previous table (10K flat 753 µs,
+pre-norm 706 µs, RAIRS 159 µs; 100K flat 11.4 ms, IVF 1.32 ms, IVF-PQ 1.19 ms).
+Those were undated figures from the original i7-12650H run. The 2026-08-05
+tank run below is within 4% of the new numbers in every cell but one (1K flat
+47.8 µs, 10K flat 501 µs, pre-norm 322 µs, IVF 24.8 µs; 100K flat 6.60 ms,
+IVF 608 µs, IVF-PQ 865 µs), so for those cells the difference is the machine.
+The exception is RAIRS: 109 µs on 2026-08-05, 124.0 µs now (+14%); this run
+does not explain that change.
+
+**Key insight:** at 10K records (typical agent memory), IVF search takes
+**24.8 µs** against 500.5 µs for the flat scan. At 100K, IVF answers in
+**592 µs** and IVF-PQ in 869 µs — both under 1 ms. (RAIRS, 124.0 µs at 10K, is
+slower than plain IVF here.)

 ### Comparison to MemX (arxiv:2603.16171)

 MemX claims end-to-end search under 90ms at 100K records (Rust + libSQL + FTS5).

+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
+
+> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(simd_cosine_100k|ivf_pq_search_100k|bm25_search_10k)$'`
+
 > **Caveat — not like-for-like.** MemX's `<90 ms` is *end-to-end* search across their
 > full pipeline (dense embeddings + FTS5 + four-factor re-ranking). The clawhdf5
 > figures below are a *single component* — raw vector search latency, excluding
 > embedding, keyword, fusion, and re-ranking stages. A component measured against a
-> full pipeline will always look favourable; the "speedup" column overstates the real
-> advantage by an unquantified margin and should be read as an order-of-magnitude
-> indication only, not a benchmark result. Matching MemX's measurement boundary is
-> tracked as follow-up work.
+> full pipeline will always look favourable, so the two columns below cannot be
+> divided into a speedup: any such ratio would overstate the real advantage by an
+> unquantified margin. Read the table as an order-of-magnitude indication only, not a
+> benchmark result. Matching MemX's measurement boundary is tracked as follow-up work.

-| Metric | MemX (claimed, end-to-end) | ClawhDF5 (component only) | Ratio |
-|--------|----------------------------|---------------------------|-------|
-| 100K flat search | <90 ms | 11.4 ms | ~8x |
-| 100K IVF-PQ search | — | 1.19 ms | ~76x |
-| Keyword search 10K | 1,100x improvement over unindexed | 583 µs (BM25) | Comparable |
+| Metric | MemX (claimed, end-to-end) | ClawhDF5 (component only) |
+|--------|----------------------------|---------------------------|
+| 100K flat search | <90 ms | 6.64 ms |
+| 100K IVF-PQ search | — | 876 µs |
+| Keyword search 10K | 1,100x improvement over unindexed | 205 µs (BM25) |
+
+The ratio column is gone: dividing an end-to-end claim by a component
+timing produced a number that looked like a result and was not one. The
+previous figures (100K flat 11.4 ms, IVF-PQ 1.19 ms, BM25 10K 583 µs) were
+undated, from the original i7-12650H run. BM25 at 10K also moved on this
+machine: 520 µs on 2026-08-05, 205 µs now; v2.4.0 changed
+`BM25Index::search` in between (bounded-heap top-k, the unused WAND bound
+removed, IDF per query), but this run does not isolate its effect.

 ---

 ## SIMD & Parallelism

-384-dimensional cosine similarity at 10K scale.
+384-dimensional cosine similarity at 10K scale, using the `strategy_*`
+benchmarks, which hold the dataset fixed and vary only `SearchStrategy`
+(the 2026-08-05 re-run below explains why the older table's benchmarks were
+not an apples-to-apples comparison).

-| Strategy | Latency | vs Sequential |
-|----------|---------|---------------|
-| Sequential (scalar) | 1.07 ms | 1.0x |
-| SIMD (auto-vectorized) | 545 µs | **2.0x** |
-| Rayon (parallel) | 553 µs | **1.9x** |
-| Adaptive (auto-select) | 564 µs | **1.9x** |
+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. Rayon
+uses all 16 threads; the 1-minute load average was 4.19 when this command
+finished, most likely from the Rayon benchmarks themselves (not verified).
+
+> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(strategy_scalar_10k|strategy_simd_10k|strategy_rayon_10k|adaptive_search_10k|simd_cosine_100k|rayon_cosine_100k)$'`

-At 100K:
 | Strategy | Latency |
 |----------|---------|
-| SIMD | 13.7 ms |
-| Rayon parallel | 8.3 ms |
+| Sequential (scalar), `strategy_scalar_10k` | 501.1 µs |
+| SIMD, `strategy_simd_10k` | 319.9 µs |
+| Rayon (parallel), `strategy_rayon_10k` | **79.0 µs** |
+| Adaptive (auto-select), `adaptive_search_10k` | 317.9 µs |
+
+At 100K (no `strategy_*` benchmark exists at this size, so these are the
+`simd_cosine_100k` / `rayon_cosine_100k` benchmarks):
+
+| Strategy | Latency |
+|----------|---------|
+| SIMD | 6.56 ms |
+| Rayon parallel | 4.63 ms |
+
+The Rayon 10K figure moved furthest: 323 µs on this machine on 2026-08-05,
+79.0 µs now. The git log does not explain it. The other rows are close to
+that run (scalar 502 µs, SIMD 327 µs, adaptive 339 µs; 100K 6.60 / 4.73 ms). Against the original, undated i7-12650H table (scalar 1.07 ms, SIMD
+545 µs, Rayon 553 µs, adaptive 564 µs; 100K 13.7 / 8.3 ms) every figure is
+lower; that table also used the mislabelled benchmarks. The older names
+measured in the same session, for reference (`cargo bench -p clawhdf5-agent
+--bench bench -- '^(sequential_cosine_10k|simd_cosine_10k|rayon_cosine_10k|adaptive_search_10k|simd_cosine_100k|rayon_cosine_100k)$'`):
+`sequential_cosine_10k` 317.8 µs, `simd_cosine_10k` 505.2 µs,
+`rayon_cosine_10k` 274.3 µs — "sequential" faster than "SIMD", which is the
+mislabelling the 2026-08-05 re-run found.

 ---

 ## Hybrid Search (Vector + BM25)

-1K records, 384-dimensional embeddings with BM25 keyword index.
+1K records, 384-dimensional embeddings with BM25 keyword index. These are the
+flat free functions `hybrid::hybrid_search` / `rrf_hybrid_search` (linear
+scan), not the HNSW-backed `HDF5Memory::hybrid_search`, which is measured in
+the [search harness](#current-search-harness-2026-09-24).
+
+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
+
+> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^hybrid_search/'`
+> and `cargo bench -p clawhdf5-agent --bench bench -- '^(bm25_search_1k|hybrid_search_10k)$'`

 | Method | Latency | Notes |
 |--------|---------|-------|
-| Weighted fusion | 198 µs | Original min-max normalization |
-| **RRF (k=60)** | **222 µs** | Reciprocal Rank Fusion — better quality, ~12% overhead |
-| BM25-only 1K | 67 µs | Keyword search alone |
-| Hybrid 10K | 2.04 ms | Full hybrid at 10K scale |
+| Weighted fusion | 106.8 µs | Min-max normalization |
+| **RRF (k=60)** | **130.5 µs** | Reciprocal Rank Fusion |
+| BM25-only 1K | 20.4 µs | Keyword search alone |
+| Hybrid 10K | 1.10 ms | Full hybrid at 10K scale |
+
+Every row moved by more than 20% from the previous table (198 µs, 222 µs,
+67 µs, 2.04 ms), which was undated and from the original i7-12650H run. The
+cause is not isolated. (The old "better quality" note on RRF is dropped: on
+LongMemEval RRF measured worse than the tuned weighted sum; see
+[Fusion method](#fusion-method--weighted-vs-rrf-full-haystack-n500).)

 ---

@@ -665,16 +913,36 @@ At 100K:

 Graph traversal and entity operations.

+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
+
+> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^knowledge_graph/'`
+> and `cargo bench -p clawhdf5-agent --bench bench -- '^alias_resolve_(short|long)_query$'`
+> (the `knowledge_graph/` rows re-run after the adjacency-index fix, same day,
+> load average 0.49 at start)
+
 | Operation | Scale | Latency |
 |-----------|-------|---------|
-| BFS traversal | 100 entities | 5.4 µs |
-| BFS traversal | 1,000 entities | 24 µs |
-| Spreading activation | 100 entities | 16.9 µs |
-| Entity resolution (Levenshtein) | 100 entities | 64 µs |
-| Alias resolution (short query) | 100 aliases | 10.4 µs |
-| Alias resolution (long query) | 100 aliases | 11.6 µs |
+| BFS traversal | 100 entities | 5.23 µs |
+| BFS traversal | 1,000 entities | 23.1 µs |
+| Spreading activation | 100 entities | 10.1 µs |
+| Entity resolution (Levenshtein) | 100 entities | 48.6 µs |
+| Alias resolution (short query) | 100 aliases | 7.67 µs |
+| Alias resolution (long query) | 100 aliases | 8.18 µs |

-**All graph operations complete in microseconds.** Spreading activation across 100 entities with 5 propagation steps finishes in 17 µs.
+The traversal rows were measured after a fix made during this re-run. The
+first measurement of the day found **BFS 6.5x slower** than the old table
+said: 17.5 µs and 155.1 µs, against 5.4 µs and 24 µs, with spreading
+activation at 22.8 µs against 16.9 µs. The cause was 1efd82c (2026-08-17),
+which built an adjacency index over the whole graph on every traversal — so a
+2-hop BFS paid to index every entity and relation first. The index is now
+cached on `KnowledgeCache` and checked against a fingerprint of the graph on
+each use (one pass over entity ids and relation endpoints, no allocation), so
+any change to the graph, including direct edits of its public `Vec`s, still
+triggers a rebuild. These benches traverse an unchanged graph, which is the
+cached case; the first traversal after a change pays for one build. Entity
+and alias resolution do not use the index (the 49.8 µs first-run figure for
+entity resolution is within noise of the 48.6 µs here). The old figures were
+undated, from the original i7-12650H run.

 ---

@@ -682,13 +950,22 @@ Graph traversal and entity operations.

 Hippocampal-inspired tiered memory management.

+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
+
+> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^consolidation/'`
+
 | Operation | Scale | Latency |
 |-----------|-------|---------|
-| Consolidation cycle | 100 records | 15 µs |
-| Consolidation cycle | 1,000 records | 164 µs |
-| Importance scoring | 100 records | 25 µs |
+| Consolidation cycle | 100 records | 8.08 µs |
+| Consolidation cycle | 1,000 records | 115.2 µs |
+| Importance scoring | 100 records | 31.0 µs |

-A full consolidation pass over 1,000 memories (eviction + promotion across Working → Episodic → Semantic) completes in **164 µs**. This can run on every memory write without perceptible latency.
+A full consolidation pass over 1,000 memories (eviction + promotion across Working → Episodic → Semantic) completes in **115 µs**. This can run on every memory write without perceptible latency.
+
+The cycle figures were 15 µs and 164 µs, and importance scoring 25 µs, all
+undated from the original i7-12650H run. Eviction's membership check became a
+`HashSet` lookup (603fcf8) and `add_memory` stopped cloning Working-tier
+records (d787ac0), both on 2026-08-17; this run does not isolate their effect.

 ---

@@ -696,12 +973,17 @@ A full consolidation pass over 1,000 memories (eviction + promotion across Worki

 Sorted timestamp index with binary search.

+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
+
+> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^temporal/'`
+
 | Operation | Scale | Latency |
 |-----------|-------|---------|
-| Range query | 10K timestamps | **716 ns** |
-| Batch insert | 10K timestamps | 4.69 ms |
+| Range query | 10K timestamps | **622 ns** |
+| Batch insert | 10K timestamps | 3.22 ms |

-Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K records: **716 nanoseconds.**
+Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K records: **622 nanoseconds.**
+Batch insert was 4.69 ms in the previous, undated i7-12650H figure.

 ---

@@ -709,16 +991,50 @@ Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K

 HDF5 persistence with optional Write-Ahead Log.

+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. The stores
+use `MemoryConfig::new`'s defaults at this commit: `float16` embeddings and
+the int8 index.
+
+> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(save_without_wal_single|save_with_wal_single|save_batch_100|save_batch_1000|save_wal_1k_existing|wal_flush_100_entries|tick_session_1k|tick_session_10k)$'`
+
 | Operation | Latency | Notes |
 |-----------|---------|-------|
-| Single save (no WAL) | 61 µs | Direct HDF5 write (owned-Vec IO path) |
-| Single save (with WAL) | 18 µs | WAL group-commit append; HDF5 write batched at flush |
-| Batch 100 | 723 µs | 7.2 µs per record |
-| Batch 1,000 | 6.17 ms | 6.2 µs per record |
-| WAL save (1K existing) | 539 µs | Incremental append |
-| WAL flush 100 entries | 787 µs | Merge WAL → HDF5 |
-| Session tick 1K | 5.76 ms | Full session maintenance |
-| Session tick 10K | 89.8 ms | Background operation |
+| Single save (no WAL) | 88.3 µs | Direct HDF5 write (owned-Vec IO path) |
+| Single save (with WAL) | 26.1 µs | WAL group-commit append; HDF5 write batched at flush |
+| Batch 100 | 1.08 ms | |
+| Batch 1,000 | 31.7 ms | |
+| WAL save (1K existing) | 282.8 µs | Incremental append |
+| WAL flush 100 entries | 530.1 µs | Merge WAL → HDF5 |
+| Session tick 1K | 3.83 ms | Full session maintenance |
+| Session tick 10K | 35.5 ms | Background operation |
+
+Almost every row moved by more than 20% from the previous table, in both
+directions. Slower: single save 61 -> 88.3 µs (no WAL) and 18 -> 26.1 µs
+(WAL), batch 100 723 µs -> 1.08 ms, and **batch 1,000 6.17 -> 31.7 ms**.
+Faster: WAL save into 1K existing 539 -> 282.8 µs, WAL flush 787 -> 530.1 µs,
+session tick 5.76 -> 3.83 ms (1K) and 89.8 -> 35.5 ms (10K). The two single-save
+figures were last updated on 2026-07-01 and the rest are undated, all on the
+i7-12650H with `f32` stores.
+
+**Isolated afterwards, same machine, same day** (`--warm-up-time 1
+--measurement-time 3`, the store settings switched through the bench's
+config):
+
+| | f16 + int8 (default) | f32 + int8 | f16 + f32 index | f32 + f32 index | v2.3.0 (f32) |
+|---|---:|---:|---:|---:|---:|
+| Single save (WAL) | 26.1 µs | 24.3 µs | 26.1 µs | 24.4 µs | 24.3 µs |
+| Single save (no WAL) | 89.0 µs | 87.3 µs | 88.8 µs | 87.4 µs | 82.6 µs |
+| Batch 100 | 1.09 ms | 0.90 ms | 1.09 ms | 0.90 ms | 0.87 ms |
+| Batch 1,000 | 31.5 ms | 31.1 ms | 31.5 ms | 31.6 ms | 30.3 ms |
+
+The int8 index costs nothing on the write path. `float16` costs ~2 µs per
+saved record — the rounding — which is the whole difference at 100 records
+and within noise at 1,000. Built at the v2.3.0 tag (v2.2.0's benches do not
+compile) and run on this machine, the write path measures the same as today,
+so nothing regressed since then; the old 18 µs and 6.17 ms figures cannot be
+reproduced on this hardware and came from an undated run on another machine.
+`save_batch` scales linearly (about 12 µs per record from 500 to 8 000
+records); the bench's figure also includes dropping the store.

 ---

@@ -726,12 +1042,16 @@ HDF5 persistence with optional Write-Ahead Log.

 Trivial/non-trivial classification for memory write filtering.

+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
+
+> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^gate_'`
+
 | Check | Latency |
 |-------|---------|
-| Trivial skip ("ok", "yes") | 61 ns |
-| Short phrase skip | 86 ns |
-| Non-trivial pass | 705 ns |
-| Ratio check | 488 ns |
+| Trivial skip ("ok", "yes") | 57.6 ns |
+| Short phrase skip | 83.0 ns |
+| Non-trivial pass | 570.5 ns |
+| Ratio check | 395.2 ns |

 **Sub-microsecond filtering.** The gate decides whether to save a memory in under 1 µs.

@@ -741,27 +1061,70 @@ Trivial/non-trivial classification for memory write filtering.

 End-to-end strategy evaluation including embedding operations.

+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
+
+> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^strategy_(save_every|semantic_shift)'`
+
 | Strategy | Condition | Latency |
 |----------|-----------|---------|
-| SaveEveryExchange (substantive) | Saves | 923 ns |
-| SaveEveryExchange (trivial) | Skips | 67 ns |
-| SaveOnSemanticShift (empty store) | Saves | 941 ns |
+| SaveEveryExchange (substantive) | Saves | 732.5 ns |
+| SaveEveryExchange (trivial) | Skips | 58.3 ns |
+| SaveOnSemanticShift (empty store) | Saves | 734.9 ns |
+
+The two "Saves" rows were 923 ns and 941 ns in the previous table, undated
+and from the original i7-12650H run.
+
+---
+
+## Multi-modal Search
+
+`clawhdf5_agent::multimodal::MultiModalStore`. Each of N records carries two
+384-dim embeddings: a text embedding of its caption and one of its primary
+modality, cycling image / audio / video. `search_cross_modal` scores every
+embedding of every record (2N vectors) and keeps each record's best;
+`search_by_modality(Image, …)` scores only image embeddings (a third of the
+records). Both are exact linear scans followed by a full sort; k = 10. Data is
+from a fixed-seed generator, so every run sees the same corpus.
+
+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c (with the
+new, uncommitted `crates/clawhdf5-agent/benches/multimodal_bench.rs` in the
+working tree). Criterion, 50 samples.
+
+> **Run:** `cargo bench -p clawhdf5-agent --bench multimodal_bench`
+
+| Search | 1K records | 10K records |
+|--------|-----------:|------------:|
+| Cross-modal (all modalities, 2N vectors) | 842.0 µs | 8.44 ms |
+| One modality (image, ~N/3 vectors) | 151.2 µs | 1.54 ms |
+
+Both scale linearly with N. Cross-modal search is the slow path: at 1K
+records it takes 842.0 µs, where the flat single-vector cosine scan in
+[Vector Search Latency](#vector-search-latency) takes 47.4 µs over 1K vectors,
+and it has no index — at 10K it is 8.44 ms, slower than a 100K flat scan
+(6.58 ms). No earlier figure exists for this section.

 ---

 ## Summary

+Derived from the 2026-09-24 tables above (tank, AMD Ryzen 7 7800X3D, commit
+5c8323c); no command of its own.
+
 | Capability | Typical Latency | Scale |
 |------------|----------------|-------|
-| **Full memory search** | <1 ms | 10K records |
-| **Hybrid vector+keyword** | <200 µs | 1K records |
-| **Knowledge graph query** | <25 µs | 1K entities |
-| **Temporal range query** | <1 µs | 10K timestamps |
-| **Memory write** | <20 µs | Per record (WAL group-commit append) |
-| **Consolidation cycle** | <165 µs | 1K records |
-| **Importance gate** | <1 µs | Per record |
+| **Full memory search** (`HDF5Memory::hybrid_search`, p50) | 0.49 ms | 10K records |
+| **Hybrid vector+keyword** (flat, weighted) | 106.8 µs | 1K records |
+| **Knowledge graph query** (BFS) | 23.1 µs | 1K entities |
+| **Temporal range query** | 622 ns | 10K timestamps |
+| **Memory write** | 26.1 µs | Per record (WAL group-commit append) |
+| **Consolidation cycle** | 115.2 µs | 1K records |
+| **Importance gate** | 57.6–570.5 ns | Per record |

-**The entire memory pipeline — search, retrieve, re-rank, filter — runs in single-digit milliseconds at agent-typical scales. Fast enough that memory becomes invisible infrastructure.**
+The previous summary said "<25 µs" for a knowledge-graph query and "<20 µs"
+for a memory write. The first holds again after the adjacency-index fix
+(23.1 µs); the second does not (26.1 µs).
+
+**At agent-typical scales a full `HDF5Memory::hybrid_search` (vector + BM25 + fusion) over 10K records takes 0.49 ms at p50, and graph traversal, temporal queries, consolidation and the write gate each take well under a millisecond. At 100K records the hybrid search takes 4.69 ms.**

 ---

@@ -796,9 +1159,21 @@ _Latency benchmarks generated with Criterion.rs (50-100 samples per benchmark).

 **Mode:** BM25-only retrieval — zero embeddings, `vector_weight=0.0`, `keyword_weight=1.0`

-> **Run:** `cargo run --release --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s_cleaned.json`
-> (~70 s for all 500 questions on the tank reference machine). Omit the path for the
-> oracle variant; add `--limit N` for an evenly-strided subsample.
+> **Run:** `cargo run --release -p clawhdf5-bench --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s.json`
+> Omit the path for the oracle variant; add `--limit N` for an evenly-strided
+> subsample. Without the `embeddings` feature this runs two modes, BM25 only
+> and BM25 stemmed.
+
+The BM25-only and BM25-stemmed figures in this section were measured again on
+2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, with the command
+above. Every figure of those two modes that this document publishes was
+reproduced exactly: BM25 Hit@1/5/10 and MRR at turn and session level, the
+BM25 per-type session Hit@1 values quoted in prose below, and the stemmed
+turn-level row plus its session Hit@1 in the tokenizer table. The run also
+produced figures this document does not publish (stemmed session Hit@5,
+Hit@10 and MRR, and per-type Hit@5/Hit@10/MRR for both modes), so there was
+nothing to compare them with. Rows that need real embeddings (vector-only,
+hybrid, RRF, re-ranking, the weight sweep) were not re-run.

 ### Full haystack — `longmemeval_s`, n=500 (the number to cite)

@@ -1058,14 +1433,29 @@ extraction over the haystack, which this harness does not do.

 ### Search Latency (LongMemEval, n=500 queries)

-| Metric | Latency |
-|--------|---------|
-| avg | 1,004 µs |
-| p50 | 1,017 µs |
-| p95 | 2,031 µs |
-| p99 | 2,912 µs |
+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, full
+`longmemeval_s` haystack (47.7 sessions / 494 turns per question).

-Sub-millisecond median search across variable-length chat histories.
+> **Run:** `cargo run --release -p clawhdf5-bench --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s.json`
+
+| Metric | BM25 only | BM25 stemmed |
+|--------|-----------|--------------|
+| avg | 11,034.7 µs | 11,096.2 µs |
+| p50 | 10,943.5 µs | 10,996.5 µs |
+| p95 | 11,749.6 µs | 12,018.8 µs |
+| p99 | 12,644.1 µs | 12,784.7 µs |
+
+About 11 ms per query. Each question gets its own store, and the timed call is
+the first `hybrid_search_with` on that store. The first query on a store
+builds its indexes (the "cold index build" column of the search harness), so
+this is a first-query figure, not steady-state search; for that
+see the [search harness](#current-search-harness-2026-09-24).
+
+The previous table (avg 1,004 µs, p50 1,017 µs) said "sub-millisecond median
+search", which no longer holds. That figure was undated and predates the
+full-haystack harness (added 2026-08-07, 7d6e269), so it was measured on the
+much smaller `longmemeval_oracle` corpus; on that corpus tank measured avg
+2,431 µs on 2026-08-05 (below).

 ---

@@ -1075,23 +1465,34 @@ Sub-millisecond median search across variable-length chat histories.
 **Topics:** Personal info, food preferences, music, travel, work/schedule, hobbies
 **Queries:** 35 questions across 4 types

-> **Run:** `cargo run --release --bin memory_arena`
+> **Run:** `cargo run --release -p clawhdf5-bench --bin memory_arena`
+
+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.

 ### Results by Query Type

 | Query Type | N | Hit@1 | Hit@5 | Hit@10 | MRR | Avg Latency |
 |------------|---|-------|-------|--------|-----|-------------|
-| single-session | 25 | 40.0% | 92.0% | 100.0% | 0.5788 | 7,853 µs |
-| multi-session | 5 | 40.0% | 60.0% | 80.0% | 0.5333 | 7,887 µs |
-| temporal | 3 | 33.3% | 66.7% | 66.7% | 0.5000 | 7,899 µs |
-| knowledge-update | 2 | 0.0% | 50.0% | 50.0% | 0.2500 | 7,899 µs |
-| **OVERALL** | **35** | **37.1%** | **82.9%** | **91.4%** | **0.5468** | **7,870 µs** |
+| single-session | 25 | 40.0% | 92.0% | 100.0% | 0.5755 | 248.4 µs |
+| multi-session | 5 | 40.0% | 60.0% | 80.0% | 0.5333 | 24.4 µs |
+| temporal | 3 | 33.3% | 66.7% | 66.7% | 0.5000 | 28.0 µs |
+| knowledge-update | 2 | 0.0% | 50.0% | 50.0% | 0.2500 | 38.7 µs |
+| **OVERALL** | **35** | **37.1%** | **82.9%** | **91.4%** | **0.5444** | p50 **30.1 µs**, p95 49.8 µs |

 **Key findings:**
 - Hit@10 of 91.4% across all query types with BM25-only (no embeddings)
 - Single-session recall strongest at 100% Hit@10
 - Knowledge-update hardest (requires temporal disambiguation) — would improve significantly with vector similarity
- Latency dominated by BM25 index build over 999 turns (~7.9 ms)
+- Search takes about 30 µs at p50 over the 999 turns. All 35 queries run
+  against one store, and single-session's higher average (248.4 µs) includes
+  that store's first query, which builds its indexes.
+
+The previous table put every query at ~7.9 ms ("latency dominated by BM25
+index build over 999 turns"). That was before v2.4.0, when
+`HDF5Memory::hybrid_search` rebuilt the BM25 index from scratch and rewrote
+the store on every query; it now keeps the index for the life of the store.
+The MRR moved slightly (single-session 0.5788 -> 0.5755, overall 0.5468 ->
+0.5444); the Hit@k figures are unchanged.

 ---

@@ -1099,46 +1500,89 @@ Sub-millisecond median search across variable-length chat histories.

 HDF5 file size at various record counts — 384-dimensional embeddings, 200-char text.

-> **Run:** `cargo run --release --bin footprint_bench`
+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.

-### Uncompressed (no WAL)
+> **Run:** `cargo run --release -p clawhdf5-bench --bin footprint_bench`

-| Records | File Size | Raw Data | Bytes/Record | Throughput |
-|---------|-----------|----------|--------------|------------|
-| 100 | 176.4 KB | 169.5 KB | 1.8 KB | 100,000 rec/s |
-| 1K | 1.7 MB | 1.7 MB | 1.8 KB | 109,643 rec/s |
-| 10K | 17.0 MB | 16.6 MB | 1.7 KB | 118,100 rec/s |
-| 50K | 85.0 MB | 82.8 MB | 1.7 KB | 110,723 rec/s |
-| 100K | 169.8 MB | 165.6 MB | 1.7 KB | 111,422 rec/s |
+**These are `float16` stores.** `footprint_bench` builds its stores with
+`MemoryConfig::new`, which defaults to `float16` embeddings since 5c8323c,
+although the binary still prints "Embedding: 384-dim f32". Its "Raw Data"
+column, and so the ratios, count embeddings as `f32` (1,536 bytes per record)
+plus text. The synthetic embeddings take only 1,000 distinct values, and each
+record's text is fixed by `record_idx % 40` (a 40-word bank walked from that
+start), so a store holds only 40 distinct texts. Both compress far better than
+real data would; read the gzip figures as an upper bound, and see the next
+paragraph for why the other tables are optimistic too. "MB" and "KB" are the
+binary's units (powers of 1024).

-**1.7 KB per record** — HDF5 overhead is near-zero. Ingestion throughput exceeds **100K records/sec**.
+**No table here is uncompressed.** The store always deflates its string
+datasets: `write_string_dataset` applies deflate level 6 to any string dataset
+of 4 KiB or more, whatever `MemoryConfig::compression` says
+(`crates/clawhdf5-agent/src/schema.rs`, `STRING_COMPRESS_THRESHOLD`, added in
+4fa7e89 on 2026-06-03). With only 40 distinct texts, the chunk text costs
+almost nothing on disk: at 10K records, 50-char text takes 804 B per record
+and 1000-char text 944 B, so 950 more characters cost only 140 B. The
+`float16` embeddings alone are 768 B per record, and 200 bytes of raw text
+would bring that to about 968 B before any other field or overhead; the
+measured 803–829 B is below that only because the text is compressed away.
+Real text will not deflate like this, so expect more than 820 B per record
+with real data. `MemoryConfig::compression` decides only whether the
+embeddings are compressed; the gzip table below is that setting turned on.
+
+### `compression` off (no WAL)
+
+| Records | File Size | Raw Data (f32) | Bytes/Record | Throughput |
+|---------|-----------|----------------|--------------|------------|
+| 100 | 90.0 KB | 169.5 KB | 921 B | 75,914 rec/s |
+| 1K | 810.4 KB | 1.7 MB | 829 B | 72,266 rec/s |
+| 10K | 7.8 MB | 16.6 MB | 820 B | 74,509 rec/s |
+| 50K | 38.3 MB | 82.8 MB | 803 B | 72,829 rec/s |
+| 100K | 76.7 MB | 165.6 MB | 803 B | 71,102 rec/s |
+
+**About 820 B per record** at 10K, for this synthetic corpus (see above: the
+text is deflated regardless and has only 40 distinct values, so real data
+takes more). Ingestion runs at 71,102–75,914 records/sec.

 ### With Gzip Compression (level 6)

-| Records | Compressed | Ratio | Bytes/Record |
-|---------|------------|-------|--------------|
-| 100 | 31.5 KB | 5.37x | 323 B |
-| 1K | 277.1 KB | 6.12x | 283 B |
-| 10K | 2.7 MB | 6.17x | 281 B |
-| 50K | 13.4 MB | 6.17x | 281 B |
-| 100K | 26.9 MB | 6.15x | 282 B |
+| Records | Compressed | Ratio vs f32 raw | Bytes/Record |
+|---------|------------|------------------|--------------|
+| 100 | 17.2 KB | 9.85x | 176 B |
+| 1K | 56.4 KB | 30.06x | 57 B |
+| 10K | 471.3 KB | 35.97x | 48 B |
+| 50K | 2.3 MB | 36.67x | 47 B |
+| 100K | 4.5 MB | 36.73x | 47 B |

-**6.2x compression ratio** — 100K agent memories in 27 MB compressed.
-
-### Text Length Comparison (10K records, no compression)
+### Text Length Comparison (10K records, `compression` off)

 | Text Length | File Size | Bytes/Record | Throughput |
 |-------------|-----------|--------------|------------|
-| short (50 chars) | 15.6 MB | 1.6 KB | 177,925 rec/s |
-| medium (200 chars) | 17.0 MB | 1.7 KB | 172,152 rec/s |
-| long (1000 chars) | 24.6 MB | 2.5 KB | 157,807 rec/s |
+| short (50 chars) | 7.7 MB | 804 B | 97,615 rec/s |
+| medium (200 chars) | 7.8 MB | 820 B | 89,597 rec/s |
+| long (1000 chars) | 9.0 MB | 944 B | 51,462 rec/s |

 ### WAL Overhead (1K records)

 | Mode | File Size | Ingest Time | Overhead |
 |------|-----------|-------------|----------|
-| No WAL | 1.7 MB | 5.7 ms | — |
-| With WAL | 1.7 MB + 9 B WAL | 5.3 ms | ±8% (negligible) |
+| No WAL | 810.4 KB | 10.8 ms | — |
+| With WAL | 810.4 KB + 9 B WAL | 10.6 ms | −2% (negligible) |
+
+**Compared with the previous tables.** Those were undated `f32` stores from the
+original i7-12650H run: 1.7 KB per record (169.8 MB at 100K), gzip 6.2x
+(26.9 MB at 100K), and more than 100,000 records/sec. Two changes separate
+those tables from these, not one. `float16` embeddings save 768 B per record.
+The old tables (recorded by 2026-05-14) also predate 4fa7e89 (2026-06-03),
+which began deflating string datasets of 4 KiB or more, so their text was
+stored raw; with only 40 distinct texts, deflate now removes most of it.
+An `f32` embedding plus 200 B of text is 1,536 + 200 = 1,736 B; `float16`
+alone would bring that to about 968 B per record, not the measured 803–829 B. This run does not measure how the saving splits between the two
+beyond that, nor explain why the gzip ratio rose so much further, nor
+separate the lower throughput (and the 1K ingest going from 5.7 to 10.8 ms)
+from the change of machine and store configuration. The
+`f32` on-disk figures were not re-measured; the dated
+[float16 study](#float16-embedding-storage-memoryconfigfloat16) compares the
+two directly (100K × 384: 154.0 MiB `f32`, 80.8 MiB `float16`).

 ---

@@ -1146,7 +1590,12 @@ HDF5 file size at various record counts — 384-dimensional embeddings, 200-char

 Hippocampal-inspired memory consolidation improves both retrieval quality and search speed.

-> **Run:** `cargo run --release --bin consolidation_efficiency`
+> **Run:** `cargo run --release -p clawhdf5-bench --bin consolidation_efficiency`
+
+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. The run
+was stopped (after about 19 minutes on one core) while still computing a
+100K row of the cycle-time table, so that row and the binary's memory-reduction
+part were not produced; neither has ever been published here.

 ### Retrieval Quality Before vs. After Consolidation

@@ -1159,17 +1608,22 @@ Hippocampal-inspired memory consolidation improves both retrieval quality and se
 | Hit@5 | 100.0% | 100.0% | — |
 | Hit@10 | 100.0% | 100.0% | — |
 | MRR | 1.0000 | 1.0000 | — |
-| Search latency | 2,752 µs | 312 µs | **8.8x faster** |
+| Search latency (avg) | 2,223.7 µs | 239.2 µs | **9.3x faster** |

-Signal records survive consolidation because they are accessed 15+ times, giving them high decay scores. 900 noise records evicted, search speeds up 8.8x, and **zero quality loss** — perfect recall maintained.
+Signal records survive consolidation because they are accessed 15+ times, giving them high decay scores. 900 noise records evicted, search speeds up 9.3x, and **zero quality loss** — perfect recall maintained. The consolidation cycle itself took 0.13 ms and promoted 2 records.

 ### Consolidation Cycle Time

 | Records | Cycle Time | Evictions | Promotions |
 |---------|-----------|-----------|------------|
-| 100 | 21 µs | 100 | 0 |
-| 1K | 345 µs | 1,000 | 0 |
-| 10K | 17.3 ms | 10,000 | 0 |
+| 100 | 17 µs | 100 | 0 |
+| 1K | 189 µs | 1,000 | 0 |
+| 10K | 2.16 ms | 10,000 | 0 |
+
+The 1K and 10K cycles were 345 µs and 17.3 ms in the previous, undated table
+(before: 2,752 µs / after: 312 µs for search). Eviction's membership check
+changed from a linear scan to a `HashSet` in 603fcf8 (2026-08-17), after those
+figures were recorded; this run does not isolate its effect.

 ---

@@ -1178,34 +1632,44 @@ Signal records survive consolidation because they are accessed 15+ times, giving
 In-memory key-value store with TTL, capacity eviction, and embedding search.
 No network hop, no serialization — direct HashMap operations.

-> **Run:** `cargo run --release --bin ephemeral_perf`
+> **Run:** `cargo run --release -p clawhdf5-bench --bin ephemeral_perf`
+
+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.

 ### Latency Comparison

-| Operation | clawhdf5 Ephemeral | Redis (single-node)¹ | Speedup |
+| Operation | clawhdf5 Ephemeral | Redis (single-node)¹ | Ratio¹ |
 |-----------|-------------------|---------------------|---------|
-| SET | **356 ns/op** | ~25,000 ns/op | **70x** |
-| GET (hit) | **179 ns/op** | ~25,000 ns/op | **140x** |
-| GET (miss) | **62 ns/op** | ~25,000 ns/op | **403x** |
-| DELETE | **124 ns/op** | ~25,000 ns/op | **202x** |
-| SET+embedding | **268 ns/op** | N/A | — |
+| SET | **366 ns/op** | ~25,000 ns/op (not measured) | 68.3x |
+| GET (hit) | **115 ns/op** | ~25,000 ns/op (not measured) | 216.7x |
+| GET (miss) | **57 ns/op** | — | — |
+| DELETE | **99 ns/op** | — | — |
+| SET+embedding | **236 ns/op** | N/A | — |

-> ¹ Redis latency includes network round-trip (loopback). clawhdf5 ephemeral is in-process — no network.
+> ¹ **The Redis figure is not a measurement.** It is a constant hard-coded in
+> `ephemeral_perf` as a "typical single-node" latency, and the binary computes
+> the two ratios from it; only the clawhdf5 column was measured. A typical
+> Redis figure includes a network round-trip, which the in-process ephemeral
+> tier does not have, so the ratios compare different things.

 ### Throughput

 | Operation | ops/sec |
 |-----------|---------|
-| SET | 2,810,649 |
-| GET | 5,584,684 |
-| DELETE | 8,093,731 |
-| SET+EMB (384d) | 3,725,877 |
+| SET | 2,732,568 |
+| GET (hit) | 8,666,607 |
+| GET (miss) | 17,554,663 |
+| DELETE | 10,109,895 |
+| SET+EMB (384d) | 4,235,766 |

 ### Embedding Search (ephemeral tier)

 | Scale | Latency |
 |-------|---------|
-| 10K entries @ 384d | 2.9 ms/query |
+| 10K entries @ 384d | 2.85 ms/query |
+
+GET (hit) moved from 179 to 115 ns/op and DELETE from 124 to 99 ns/op (GET
+throughput 5,584,684 -> 8,666,607 ops/sec); the previous figures were undated.

 ---

@@ -1263,10 +1727,16 @@ Measured 2026-08-07 on tank (Ryzen 7 7800X3D, 246 MB dataset in page cache).

 ### Measured Platforms

+The tank row was measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit
+5c8323c, with the single benchmark rather than the whole script:
+
+> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^ivf_search_10k_nprobe10$'`
+
 | Platform | CPU | 10K IVF Search | Notes |
 |----------|-----|----------------|-------|
-| Linux x86_64 | Intel i7-12650H (10C, 4.7 GHz) | 27 µs | Primary CI target |
-| macOS aarch64 | Apple M3 Max (14C) | ~18 µs | ~33% faster via NEON SIMD |
+| Linux x86_64 | AMD Ryzen 7 7800X3D (8C/16T), tank | 24.8 µs | Measured 2026-09-24 |
+| Linux x86_64 | Intel i7-12650H (10C, 4.7 GHz) | 27 µs | Undated, original run |
+| macOS aarch64 | Apple M3 Max (14C) | ~18 µs | Undated; ~33% faster than the i7-12650H row. Not reproducible on the hardware available for the 2026-09-24 re-run |

 ### Reproducibility

@@ -1610,6 +2080,11 @@ vs-libhdf5 validation above) to give both sections the dated, hardware-cited,
 reproducible citation the top-of-file traceability note flags them as
 missing.

+> The sections this re-run checked were themselves re-measured on tank on
+> 2026-09-24 and now show those figures; the i7-12650H figures this section
+> compares against ("above", "at the top of this file") are quoted in those
+> sections' notes. Kept as the 2026-08-05 record.
+
 ### LongMemEval Results (reproduction)

 ```bash
@@ -3,6 +3,17 @@
 ## Unreleased

 ### Upgrade Notes
+- **`clawhdf5-migrate` now writes a real agent store.** Its output used to be
+  a layout of its own (`/chunks`, `/sessions`, `/entities`, `/relations`, no
+  `/meta`) that `HDF5Memory::open` rejected, so a migrated file could not be
+  used as agent memory. Files it wrote before this release are not agent
+  stores; re-run the migration. Also: embeddings default to `float16` like
+  any new store (`--f32` opts out; `--float16` is a hidden no-op); a row with
+  the wrong embedding length is an error instead of being truncated or
+  padded; `--incremental` now matches rows by content against an existing
+  store and follows the source's deleted flags; a source with no memory rows
+  needs `--embedding-dim`. The per-dataset SHA-256 provenance attributes of
+  the old layout are gone (the agent schema has no place for them).
 - **Files written by clawhdf5 now open in h5py and libhdf5.** Every `f32`
  dataset we wrote — including every agent store's embeddings — was refused
  with "sign bit position out of bounds", and every empty dataset with
@@ -46,6 +57,24 @@
  `quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
  out. The CLI's `--quantized-index` is still accepted but is now a no-op.

+### Migration
+- `clawhdf5-migrate`: writes through the agent's own API (`HDF5Memory::create`
+  / `open`, `save_batch`, the session cache and knowledge graph), so there is
+  no second copy of the schema. Sessions and entities/relations carry over;
+  deleted rows become deleted records (or are left out with
+  `--skip-deleted`). Every source row is checked before the output is created,
+  so a source that cannot be migrated leaves an existing store untouched.
+  Validation reads the result back with `HDF5Memory::open_read_only`, compares
+  every field (embeddings bit for bit — `round_to_f16` of the source for a
+  `float16` store) and checks that a migrated record is found by search. The
+  `half`-based conversion is gone; `clawhdf5_format::float16` is the only one.
+  42 tests, including h5py opening a migrated store; an adversarial review's
+  two blocker and four major findings are fixed with regression tests.
+- `clawhdf5-agent`: `HDF5Memory::sessions()` / `sessions_mut()`,
+  `HDF5Memory::delete_batch(&[usize])` (one save, all-or-nothing, never
+  auto-compacts), `SessionCache::add_at`, and `SessionCache` / `SessionEntry`
+  re-exported from the crate root.
+
 ### Search
 - `clawhdf5-agent`: **`HDF5Memory::search` with `SearchOptions`** — source
  filtering, re-ranking and confidence rejection in the store's own search
@@ -68,6 +97,20 @@
  activation of the `k` results it returns, not of the whole `3k` candidate
  pool it re-ranks.

+### Benchmarks
+- Every undated or pre-September section of `BENCHMARKS.md` re-run on one
+  machine on one day (tank, 2026-09-24, commit 5c8323c), with the command for
+  each and every number traced back to the raw output by a separate check.
+  Where a figure moved, the section says so. Two apparent regressions were
+  isolated rather than published: knowledge-graph traversal (a real bug,
+  fixed above) and the write path, which measures the same at v2.3.0 on this
+  machine — the old 18 µs / 6.17 ms figures came from an undated run on other
+  hardware; `float16` adds ~2 µs per save and the int8 index nothing.
+- New `multimodal_bench`: cross-modal search at 1K and 10K records, which the
+  README claimed but nothing measured.
+- `footprint_bench` reports whether it built `float16` or `f32` stores and
+  takes `--f32`; it had kept printing "f32" after the default changed.
+
 ### Interop
 - `clawhdf5-format`: **every `f32` dataset was unreadable by h5py and
  libhdf5.** The float datatype encoder hard-coded the sign bit's position to
@@ -146,6 +189,14 @@
  knew to ask; it now only ever switches the default off.

 ### Performance
+- `clawhdf5-agent`: **knowledge-graph traversal was 6.5x slower than it
+  should be.** `bfs_neighbors` and `spreading_activation` built an adjacency
+  index over the whole graph on every call (1efd82c), so a 2-hop BFS over 1K
+  entities took 155 µs. The index is now cached on `KnowledgeCache` and
+  checked against a fingerprint of the graph on each use — one pass over
+  entity ids and relation endpoints, no allocation — so any change, including
+  direct edits of its public `Vec`s, still rebuilds it (tested). BFS over 1K
+  entities: 155.1 -> 23.1 µs; spreading activation over 100: 22.8 -> 10.1 µs.
 - `clawhdf5-format`, `clawhdf5-filters`: both deflate paths hand the codec the
  whole chunk in one call, into a buffer allocated once, instead of streaming
  it through a 32 KiB buffer: about 5% on chunked writes and 10% on zlib-ng's
@@ -6,7 +6,7 @@
 [![Rust](https://img.shields.io/badge/rust-1.92%2B-orange.svg)](https://www.rust-lang.org)
 [![Tests](https://img.shields.io/badge/tests-1850%2B-brightgreen.svg)](#building)
 [![LongMemEval](https://img.shields.io/badge/LongMemEval__s-Turn--Level%20Hit@5%2081.4%25%20hybrid-blue.svg)](BENCHMARKS.md#longmemeval-results)
-[![Footprint](https://img.shields.io/badge/on--disk-1.7%20KB%2Frecord-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1)
+[![Footprint](https://img.shields.io/badge/on--disk-~820%20B%2Frecord%20float16%2C%20synthetic%20text-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1)

 ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file.

@@ -111,8 +111,8 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d
 | Keyword search | Separate FTS engine | Integrated BM25 |
 | Knowledge graph | Neo4j or none | In-file graph with spreading activation |
 | Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers |
-| Temporal queries | Custom code | Native temporal index (716ns) |
-| Multi-modal | Multiple stores | Unified cross-modal search |
+| Temporal queries | Custom code | Native temporal index (622 ns range query over 10K) |
+| Multi-modal | Multiple stores | Unified cross-modal search (exact scan: 842 µs over 1K records) |
 | Integrity | Hope for the best | Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance |
 | Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** |

@@ -120,7 +120,7 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d

 ## Performance

-Vector search and agent-memory operations below are benchmarked on Intel i7-12650H (10C/16T), 384-dim embeddings, Criterion.rs. The HDF5 Core I/O table immediately below is from a separate, independently reproduced run (see its own hardware note).
+The brute-force/IVF vector search, agent-memory, on-disk footprint and consolidation figures below were measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D, 8C/16T), commit 5c8323c, 384-dim embeddings; the commands are in [BENCHMARKS.md](BENCHMARKS.md). Exceptions are marked where they appear: the HDF5 Core I/O table immediately below is from a separate, independently reproduced run (see its own hardware note), and the HNSW `f32`/`i8` table and the in-memory `i8` column were not re-measured on 2026-09-24.

 ### HDF5 Core I/O (vs libhdf5 1.14.6)

@@ -154,33 +154,45 @@ and [§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quant
 | `f32` index | 0.9945 | 13 399 | 3.2 s |
 | `i8` index + exact re-score (**default for new stores**) | 0.9940 | **21 848** | **1.8 s** |

-Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31.
+Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31. These
+two rows are a paired comparison (medians of alternating runs, same binary).
+A single `f32` run on 2026-09-24 measured recall 0.9945, 19 001 QPS and a
+2.7 s build; the int8 row was not re-run, so the pair has not been re-checked
+([§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index)).

-**Brute-force and IVF paths** (Criterion, i7-12650H):
+**Brute-force and IVF paths** (Criterion, tank, 2026-09-24):

-| Scale | Flat | IVF (nprobe=10) | IVF-PQ | vs MemX¹ |
+| Scale | Flat | IVF (nprobe=10) | IVF-PQ | MemX¹ (claimed, end-to-end) |
 |-------|------|-----------------|--------|----------|
-| 1K | **54 µs** | — | — | — |
-| 10K | 753 µs | **27 µs** | — | — |
-| 100K | 11.4 ms | 1.32 ms | **1.19 ms** | ~8–76× (see caveat) |
+| 1K | **47.4 µs** | — | — | — |
+| 10K | 500.5 µs | **24.8 µs** | — | — |
+| 100K | 6.58 ms | 592 µs | **869 µs** | <90 ms |

-> Reproduced on the same second machine (Ryzen 7 7800X3D) with a corrected,
-> apples-to-apples SIMD/scalar/parallel comparison methodology — see
-> [BENCHMARKS.md § Independent Validation: tank — LongMemEval & Vector
-> Search](BENCHMARKS.md#independent-validation-tank--longmemeval--vector-search-ryzen-7-7800x3d-2026-08-05).
+> These replace figures from the original i7-12650H run (flat 54 µs / 753 µs /
+> 11.4 ms); a 2026-08-05 run on tank had already matched the new ones — see
+> [BENCHMARKS.md § Vector Search Latency](BENCHMARKS.md#vector-search-latency).

 ### Agent Memory Operations

 | Operation | Latency | Scale |
 |-----------|---------|-------|
-| Hybrid search (`HDF5Memory::hybrid_search`, p50) | **70 µs** / 0.49 ms / 4.65 ms | 1K / 10K / 100K records |
-| BM25 keyword search | **67 µs** | 1K records |
-| Knowledge graph BFS | **24 µs** | 1K entities |
-| Spreading activation | **17 µs** | 100 entities |
-| Temporal range query | **716 ns** | 10K timestamps |
-| Consolidation cycle | **164 µs** | 1K records |
-| Memory write (WAL) | **18 µs** | per record (group-commit append; HDF5 batched at flush) |
-| Importance gate | **61 ns** | per record |
+| Hybrid search (`HDF5Memory::hybrid_search`, p50) | **0.07 ms** / 0.49 ms / 4.69 ms | 1K / 10K / 100K records |
+| BM25 keyword search | **20.4 µs** | 1K records |
+| Knowledge graph BFS | **23.1 µs** | 1K entities |
+| Spreading activation | **10.1 µs** | 100 entities |
+| Temporal range query | **622 ns** | 10K timestamps |
+| Consolidation cycle | **115.2 µs** | 1K records |
+| Cross-modal search (exact scan, 2 embeddings per record) | **842.0 µs** / 8.44 ms | 1K / 10K records |
+| Memory write (WAL) | **26.1 µs** | per record (group-commit append; HDF5 batched at flush) |
+| Importance gate | **57.6 ns** | per record (trivial skip) |
+
+The old 18 µs WAL write was undated, from another machine: v2.3.0 measures
+24.3 µs on the same hardware as this table, the same as an `f32` store today.
+`float16` stores (the new default) add ~2 µs for rounding; the int8 index adds
+nothing. See [BENCHMARKS.md § Write Path](BENCHMARKS.md#write-path).
+Knowledge-graph traversal was briefly 6.5x slower (155 µs) until this re-run
+found and fixed an adjacency index rebuilt on every traversal; see
+[§ Knowledge Graph](BENCHMARKS.md#knowledge-graph).

 ### Chunked Write Throughput (codec comparison)

@@ -195,7 +207,7 @@ by default (AoS→SoA byte transpose, +157–204% throughput for float data):

 Use `.with_zstd(3)` or `.with_deflate(6)` for write-heavy workloads — both now perform at ~720–750 MiB/s on large matrices. Use `.with_pcodec()` for write-once/read-many workloads where compression ratio matters more than encode speed. Disable auto-shuffle with `.without_shuffle()` for byte arrays that don't benefit from AoS→SoA transposition.

-> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search). The ratio overstates the real advantage by an unquantified margin — order-of-magnitude indication only. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171).
+> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search), so the two columns are not comparable and no ratio is given. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171).

 ### LongMemEval Retrieval Recall

@@ -248,18 +260,26 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points.

 ### Memory Footprint

-**On disk** — 384-dim embeddings, 200-char text
+**On disk** — 384-dim `float16` embeddings (the default for new stores),
+200-char text, `footprint_bench`
 ([BENCHMARKS.md § Memory Footprint](BENCHMARKS.md#memory-footprint-1)):

 | Records | File Size | Bytes/Record | Gzip-6 compressed |
 |---------|-----------|--------------|-------------------|
-| 1K | 1.7 MB | 1.8 KB | 277 KB (6.1x) |
-| 10K | 17.0 MB | 1.7 KB | 2.7 MB (6.2x) |
-| 100K | 169.8 MB | 1.7 KB | 26.9 MB (6.2x) |
+| 1K | 810.4 KB | 829 B | 56.4 KB |
+| 10K | 7.8 MB | 820 B | 471.3 KB |
+| 100K | 76.7 MB | 803 B | 4.5 MB |

-These figures are `f32` embeddings. New agent stores default to
-`MemoryConfig::float16`, which halves them: 100K × 384 records take 80.8 MiB
-instead of 154.0.
+The benchmark's synthetic embeddings and text are far more repetitive than
+real data (only 40 distinct texts), so no column here is an expectation for
+real data. The compressed column is an upper bound, and the Bytes/Record
+column is optimistic too: it is not an uncompressed figure, because the store
+always deflates its text (any string dataset of 4 KiB or more) whatever
+`MemoryConfig::compression` says. The `float16` embeddings alone are 768 B per
+record, so 200 characters of real text would take a record above 820 B.
+This table used to show `f32` stores (1.7 KB per record, 169.8 MB at 100K);
+those were not re-measured. The float16 study compares the two on the same
+data: 100K × 384 records take 80.8 MiB as `float16` and 154.0 MiB as `f32`.

 **In memory** — a store reopened from disk, 384-dim `f32`, measured with a
 counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-footprint)):
@@ -271,7 +291,8 @@ counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-foo
 | 100K | 146 MiB | 399 MiB (2.72x) | **256 MiB (1.74x)** |

 Down from 505 MiB (3.44x) at 100K before v2.6.0, when the cache held every
-embedding twice.
+embedding twice. The `f32` column was re-measured on 2026-09-24 and reproduced
+exactly; the `i8` column was not re-run.

 ### Consolidation Efficiency

@@ -282,7 +303,10 @@ embedding twice.
 |--------|--------|-------|-------|
 | Records in store | 1,000 | 100 | −90% |
 | Hit@1 recall (signal records) | 100% | 100% | no loss |
-| Search latency | 2.75 ms | 0.31 ms | **8.8x faster** |
+| Search latency (avg) | 2.22 ms | 0.24 ms | **9.3x faster** |
+
+The consolidation cycle that does this took 0.13 ms; at 10K records a cycle
+takes 2.16 ms.

 **Full benchmark details: [BENCHMARKS.md](BENCHMARKS.md)**

@@ -754,9 +778,37 @@ Replace in `Cargo.toml` and source:

 ```bash
 cargo install --path crates/clawhdf5-migrate
-clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedding-dim 384
+clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedder minilm
 ```

+The output is an ordinary `clawhdf5-agent` store, written through the agent's
+own API: open it with `HDF5Memory::open` (or `clawhdf5-cli --path memory.h5 …`)
+and search it straight away. What carries over from the ZeroClaw tables:
+
+| SQLite | Agent store |
+|--------|-------------|
+| `memory_chunks` | memory records (text, embedding, source channel, timestamp, session id, tags); rows with `deleted = 1` become deleted records, or are left out with `--skip-deleted` |
+| `sessions` | sessions (id, start/end index, channel, summary, timestamp) |
+| `entities`, `relations` | knowledge graph entities and relations; entities get new ids and relations are re-pointed at them |
+
+The chunk `id` column has no counterpart in the agent store, so records are
+written in `id` order and numbered from 0. Embeddings are stored as float16
+like any new store; `--f32` keeps full precision (and is required for values
+beyond ±65504). The embedding dimension is detected from the first row unless
+`--embedding-dim` is given, and every row must have it: a row of another length
+is an error, never truncated or padded. A source with no memory records (only
+sessions or the graph) needs `--embedding-dim`, since a store's dimension is
+fixed when it is created. Every row is checked before the output is created,
+so a source that cannot be migrated leaves an existing store at `--hdf5` as it
+was. `--incremental` adds to an existing store only the rows it does not
+already hold; the source must have the store's dimension, and records already
+in the store take the source's deleted flag (a row deleted in SQLite since the
+last run is deleted in the store; one un-deleted there is written again, as
+the agent has no un-delete). The tool reads the result back with
+`HDF5Memory::open_read_only`, compares it with the source (every row with
+`--validate-full`) and checks that a migrated record is found by search;
+`--dry-run` only counts the rows.
+
 ---

 ## Roadmap
@@ -45,6 +45,10 @@ harness = false
 name = "memory_bench"
 harness = false

+[[bench]]
+name = "multimodal_bench"
+harness = false
+
 [features]
 default = ["float16", "hnsw", "parallel"]
 float16 = ["half"]
@@ -0,0 +1,107 @@
+//! Multi-modal memory search benchmarks (`clawhdf5_agent::multimodal`).
+//!
+//! Covers `MultiModalStore::search_cross_modal` (every embedding of every
+//! record, whatever its modality) and, for comparison,
+//! `MultiModalStore::search_by_modality` restricted to one modality.
+//!
+//! Corpus: N records (1K and 10K), each carrying two 384-dim embeddings —
+//! a text embedding of its caption plus one embedding of its primary modality,
+//! cycling Image / Audio / Video — so a cross-modal query scores 2N vectors.
+//! All data comes from a fixed-seed LCG, so every run sees the same corpus.
+//!
+//! Run: `cargo bench -p clawhdf5-agent --bench multimodal_bench`
+
+use std::collections::HashMap;
+
+use clawhdf5_agent::multimodal::{
+    MediaRef, ModalEmbedding, Modality, MultiModalRecord, MultiModalStore,
+};
+use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
+
+// ---------------------------------------------------------------------------
+// Simple deterministic PRNG (LCG), same as the other agent benches
+// ---------------------------------------------------------------------------
+
+struct Rng(u32);
+
+impl Rng {
+    fn new(seed: u32) -> Self {
+        Self(seed)
+    }
+    fn next_u32(&mut self) -> u32 {
+        self.0 = self.0.wrapping_mul(1103515245).wrapping_add(12345);
+        self.0 >> 16
+    }
+    fn next_f32(&mut self) -> f32 {
+        self.next_u32() as f32 / 65536.0 - 0.5
+    }
+}
+
+fn make_vec(rng: &mut Rng, dim: usize) -> Vec<f32> {
+    (0..dim).map(|_| rng.next_f32()).collect()
+}
+
+// ---------------------------------------------------------------------------
+// Corpus
+// ---------------------------------------------------------------------------
+
+const DIM: usize = 384;
+const K: usize = 10;
+
+const MEDIA: [(Modality, &str, &str); 3] = [
+    (Modality::Image, "image/png", "clip-vit-base"),
+    (Modality::Audio, "audio/wav", "clap-base"),
+    (Modality::Video, "video/mp4", "xclip-base"),
+];
+
+fn build_store(n: usize, seed: u32) -> MultiModalStore {
+    let mut rng = Rng::new(seed);
+    let mut store = MultiModalStore::new();
+    for i in 0..n {
+        let (modality, mime, model) = &MEDIA[i % MEDIA.len()];
+        let embeddings = vec![
+            ModalEmbedding::new(Modality::Text, make_vec(&mut rng, DIM), "minilm-l6"),
+            ModalEmbedding::new(modality.clone(), make_vec(&mut rng, DIM), *model),
+        ];
+        store.add_record(MultiModalRecord {
+            id: 0,
+            primary_modality: modality.clone(),
+            text_content: Some(format!("{modality} memory {i}")),
+            media_ref: Some(MediaRef::path(format!("/media/{i}"), *mime)),
+            embeddings,
+            observation: None,
+            timestamp: 1_700_000_000.0 + i as f64,
+            metadata: HashMap::new(),
+        });
+    }
+    store
+}
+
+// ---------------------------------------------------------------------------
+// Benchmarks
+// ---------------------------------------------------------------------------
+
+fn multimodal_search_benches(c: &mut Criterion) {
+    let query = make_vec(&mut Rng::new(99), DIM);
+
+    let mut group = c.benchmark_group("multimodal_search");
+    group.sample_size(50);
+
+    for (label, n) in [("1k", 1_000usize), ("10k", 10_000)] {
+        let store = build_store(n, 42);
+        assert_eq!(store.count(), n);
+
+        group.bench_with_input(BenchmarkId::new("cross_modal", label), &n, |b, _| {
+            b.iter(|| store.search_cross_modal(&query, K));
+        });
+
+        group.bench_with_input(BenchmarkId::new("by_modality_image", label), &n, |b, _| {
+            b.iter(|| store.search_by_modality(&Modality::Image, &query, K));
+        });
+    }
+
+    group.finish();
+}
+
+criterion_group!(multimodal_benches, multimodal_search_benches);
+criterion_main!(multimodal_benches);
@@ -163,12 +163,13 @@ fn levenshtein(a: &str, b: &str) -> usize {
 /// entities-slice-index map, and an entity-id -> relation-indices map (edges
 /// touching that entity as either source or target).
 ///
-/// Built fresh per traversal call rather than cached on `KnowledgeCache`:
-/// entities/relations are plain `pub` `Vec`s that get pushed to directly
-/// (e.g. `schema.rs`'s load path bypasses `add_entity`/`add_relation`), so a
-/// persistent index would need extra bookkeeping to avoid drifting stale. A
-/// one-off O(V+E) build per call is still a large win over the O(V·E) (BFS)
-/// / O(steps·active·E) (spreading activation) scans it replaces.
+/// Cached on `KnowledgeCache` and checked against a fingerprint of the graph
+/// on every use ([`graph_fingerprint`]). entities/relations are plain `pub`
+/// `Vec`s that get changed directly (e.g. `schema.rs`'s load path bypasses
+/// `add_entity`/`add_relation`), so the cache cannot rely on being told about
+/// changes; the fingerprint notices any of them. Rebuilding it on every
+/// traversal instead made a 2-hop BFS over 1K entities 6.5x slower than the
+/// scan it replaced (24 -> 155 µs; `BENCHMARKS.md`, "Knowledge Graph").
 struct AdjacencyIndex {
    entity_index: HashMap<u64, usize>,
    by_entity: HashMap<u64, Vec<usize>>,
@@ -204,6 +205,45 @@ impl AdjacencyIndex {
    }
 }

+/// A hash of everything [`AdjacencyIndex`] depends on — each entity's id and
+/// position, each relation's endpoints and position. One linear pass, no
+/// allocation: far cheaper than building the index, which hashes the same
+/// values into two maps.
+fn graph_fingerprint(entities: &[Entity], relations: &[Relation]) -> u64 {
+    // splitmix64-style mixing; order matters, so positions are covered.
+    fn mix(h: u64, v: u64) -> u64 {
+        let mut z = (h ^ v).wrapping_add(0x9E37_79B9_7F4A_7C15);
+        z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
+        z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
+        z ^ (z >> 31)
+    }
+    let mut h = mix(entities.len() as u64, relations.len() as u64);
+    for e in entities {
+        h = mix(h, e.id);
+    }
+    for r in relations {
+        h = mix(mix(h, r.src), r.tgt);
+    }
+    h
+}
+
+/// The cached [`AdjacencyIndex`] and the fingerprint it was built for.
+/// Cloning a `KnowledgeCache` starts the clone with an empty cache.
+#[derive(Default)]
+struct AdjacencyCache(std::sync::Mutex<Option<(u64, std::sync::Arc<AdjacencyIndex>)>>);
+
+impl Clone for AdjacencyCache {
+    fn clone(&self) -> Self {
+        Self::default()
+    }
+}
+
+impl std::fmt::Debug for AdjacencyCache {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.write_str("AdjacencyCache")
+    }
+}
+
 // ---------------------------------------------------------------------------
 // KnowledgeCache
 // ---------------------------------------------------------------------------
@@ -216,6 +256,7 @@ pub struct KnowledgeCache {
    pub alias_strings: Vec<String>,
    pub alias_entity_ids: Vec<i64>,
    next_entity_id: u64,
+    adjacency: AdjacencyCache,
 }

 impl KnowledgeCache {
@@ -226,6 +267,7 @@ impl KnowledgeCache {
            alias_strings: Vec::new(),
            alias_entity_ids: Vec::new(),
            next_entity_id: 0,
+            adjacency: AdjacencyCache::default(),
        }
    }

@@ -236,9 +278,29 @@ impl KnowledgeCache {
            alias_strings: Vec::new(),
            alias_entity_ids: Vec::new(),
            next_entity_id: next_id,
+            adjacency: AdjacencyCache::default(),
        }
    }

+    /// The adjacency index for the graph as it is now: the cached one if the
+    /// graph's fingerprint still matches, otherwise rebuilt and cached.
+    fn adjacency_index(&self) -> std::sync::Arc<AdjacencyIndex> {
+        let fp = graph_fingerprint(&self.entities, &self.relations);
+        let mut slot = self
+            .adjacency
+            .0
+            .lock()
+            .unwrap_or_else(std::sync::PoisonError::into_inner);
+        if let Some((cached_fp, idx)) = slot.as_ref()
+            && *cached_fp == fp
+        {
+            return idx.clone();
+        }
+        let idx = std::sync::Arc::new(AdjacencyIndex::build(&self.entities, &self.relations));
+        *slot = Some((fp, idx.clone()));
+        idx
+    }
+
    // -----------------------------------------------------------------------
    // Entity management
    // -----------------------------------------------------------------------
@@ -397,7 +459,7 @@ impl KnowledgeCache {
    /// together with their discovered depth.  The seed entity itself is NOT
    /// included.  Traversal follows both outgoing and incoming relation edges.
    pub fn bfs_neighbors(&self, entity_id: u64, max_depth: usize) -> Vec<(Entity, usize)> {
-        let idx = AdjacencyIndex::build(&self.entities, &self.relations);
+        let idx = self.adjacency_index();
        let mut visited: HashSet<u64> = HashSet::new();
        let mut queue: VecDeque<(u64, usize)> = VecDeque::new();
        let mut results: Vec<(Entity, usize)> = Vec::new();
@@ -502,7 +564,7 @@ impl KnowledgeCache {
        min_activation: f32,
        max_steps: usize,
    ) -> Vec<(u64, f32)> {
-        let idx = AdjacencyIndex::build(&self.entities, &self.relations);
+        let idx = self.adjacency_index();
        let mut activation: HashMap<u64, f32> = HashMap::new();

        // Initialise seeds with activation 1.0.
@@ -631,6 +693,51 @@ impl Default for KnowledgeCache {
 mod tests {
    use super::*;

+    #[test]
+    fn cached_adjacency_sees_direct_changes_to_the_graph() {
+        // The index is cached across traversals, but entities/relations are
+        // pub Vecs anyone can edit; every kind of edit must be seen.
+        let mut kg = KnowledgeCache::new();
+        let a = kg.add_entity("a", "t", -1);
+        let b = kg.add_entity("b", "t", -1);
+        let c = kg.add_entity("c", "t", -1);
+        kg.add_relation(a, b, "r", 1.0);
+        let ids = |kg: &KnowledgeCache| -> Vec<u64> {
+            let mut v: Vec<u64> = kg.bfs_neighbors(a, 3).iter().map(|(e, _)| e.id).collect();
+            v.sort();
+            v
+        };
+        assert_eq!(ids(&kg), vec![b]);
+        assert_eq!(ids(&kg), vec![b], "cached index reused");
+
+        // Pushed directly, bypassing add_relation.
+        kg.relations.push(Relation {
+            src: b,
+            tgt: c,
+            ..Relation::default()
+        });
+        assert_eq!(ids(&kg), vec![b, c]);
+
+        // Rewired in place: same lengths, different edge.
+        kg.relations[1].tgt = a;
+        assert_eq!(ids(&kg), vec![b]);
+
+        // Removed and replaced: same lengths again.
+        kg.relations.pop();
+        kg.relations.push(Relation {
+            src: a,
+            tgt: c,
+            ..Relation::default()
+        });
+        assert_eq!(ids(&kg), vec![b, c]);
+        let act: Vec<u64> = kg
+            .spreading_activation(&[a], 0.5, 0.0, 2)
+            .iter()
+            .map(|(id, _)| *id)
+            .collect();
+        assert!(act.contains(&c));
+    }
+
    // -----------------------------------------------------------------------
    // Original tests — must remain passing
    // -----------------------------------------------------------------------
@@ -73,7 +73,7 @@ pub use ephemeral::{EphemeralEntry, EphemeralStats};
 use knowledge::KnowledgeCache;
 use memory_strategy::{Exchange, MemoryStrategy, StrategyOutput};
 pub use search::SearchOptions;
-use session::SessionCache;
+pub use session::{SessionCache, SessionEntry};

 // --- Error type ---

@@ -1028,6 +1028,18 @@ impl HDF5Memory {
        &self.config
    }

+    /// The sessions recorded in this store.
+    pub fn sessions(&self) -> &SessionCache {
+        &self.sessions
+    }
+
+    /// Mutable access to the sessions, e.g. to add many at once. Changes
+    /// reach the disk at the next checkpoint (any flushing call, such as
+    /// [`HDF5Memory::flush_wal`] or `save_batch`), not immediately.
+    pub fn sessions_mut(&mut self) -> &mut SessionCache {
+        &mut self.sessions
+    }
+
    /// Get a reference to the knowledge cache.
    pub fn knowledge(&self) -> &KnowledgeCache {
        &self.knowledge
@@ -1401,6 +1413,35 @@ impl HDF5Memory {
 }

 impl HDF5Memory {
+    /// Delete many records with a single checkpoint, where
+    /// [`AgentMemory::delete`] checkpoints once per record.
+    ///
+    /// All or nothing: if any id is out of range or already deleted (or
+    /// repeated), nothing is deleted and `MemoryError::NotFound` is returned.
+    /// Unlike `delete`, this never auto-compacts, so the records stay in the
+    /// store as tombstones (their indices unchanged) until [`AgentMemory::compact`]
+    /// is called — importers use it to carry over records that were already
+    /// deleted in the source.
+    pub fn delete_batch(&mut self, ids: &[usize]) -> Result<()> {
+        let mut seen = std::collections::HashSet::with_capacity(ids.len());
+        for &id in ids {
+            if self.cache.tombstones.get(id).copied() != Some(0) || !seen.insert(id) {
+                return Err(MemoryError::NotFound(format!(
+                    "entry {id} not found or already deleted"
+                )));
+            }
+        }
+        if ids.is_empty() {
+            return Ok(());
+        }
+        for &id in ids {
+            self.cache.mark_deleted(id);
+            self.hnsw_on_delete(id);
+            self.bm25_on_delete(id);
+        }
+        self.flush()
+    }
+
    pub fn tick_session(&mut self) -> Result<()> {
        let d = self.config.decay_factor;
        for w in self.cache.activation_weights.iter_mut() {
@@ -1599,6 +1640,79 @@ mod tests {
        }
    }

+    #[test]
+    fn delete_batch_tombstones_without_compacting() {
+        let dir = TempDir::new().unwrap();
+        let path = dir.path().join("test.h5");
+        let mut mem = HDF5Memory::create(make_config(&dir)).unwrap();
+        mem.save_batch(
+            (0..4)
+                .map(|i| make_entry(&format!("record {i}"), &[i as f32, 1.0, 0.0, 0.0]))
+                .collect(),
+        )
+        .unwrap();
+        // 3 of 4 is far past compact_threshold (0.3): delete() would compact.
+        mem.delete_batch(&[0, 1, 3]).unwrap();
+        assert_eq!(mem.count(), 4);
+        assert_eq!(mem.count_active(), 1);
+        drop(mem);
+
+        let mut mem = HDF5Memory::open(&path).unwrap();
+        assert_eq!(mem.cache.tombstones, vec![1, 1, 0, 1]);
+        let hits = mem.hybrid_search(&[0.0, 1.0, 0.0, 0.0], "record", 0.5, 0.5, 10);
+        assert!(
+            hits.iter().all(|r| r.index == 2),
+            "tombstoned record returned"
+        );
+    }
+
+    #[test]
+    fn delete_batch_is_all_or_nothing() {
+        let dir = TempDir::new().unwrap();
+        let mut mem = HDF5Memory::create(make_config(&dir)).unwrap();
+        mem.save_batch(vec![
+            make_entry("a", &[1.0, 0.0, 0.0, 0.0]),
+            make_entry("b", &[0.0, 1.0, 0.0, 0.0]),
+        ])
+        .unwrap();
+        for bad in [&[0, 5][..], &[1, 1][..]] {
+            assert!(matches!(
+                mem.delete_batch(bad),
+                Err(MemoryError::NotFound(_))
+            ));
+            assert_eq!(mem.count_active(), 2, "{bad:?} deleted something");
+        }
+        mem.delete_batch(&[]).unwrap();
+        assert_eq!(mem.count_active(), 2);
+    }
+
+    #[test]
+    fn sessions_mut_add_at_keeps_timestamp_across_reopen() {
+        let dir = TempDir::new().unwrap();
+        let path = dir.path().join("test.h5");
+        let mut mem = HDF5Memory::create(make_config(&dir)).unwrap();
+        mem.sessions_mut()
+            .add_at("s-old", 2, 7, "discord", "old summary", 1.7e15);
+        mem.flush_wal().unwrap();
+        drop(mem);
+
+        let mem = HDF5Memory::open_read_only(&path).unwrap();
+        let s = mem.sessions();
+        assert_eq!(s.len(), 1);
+        let e = &s.entries[0];
+        assert_eq!(
+            (
+                e.id.as_str(),
+                e.start_idx,
+                e.end_idx,
+                e.channel.as_str(),
+                e.ts
+            ),
+            ("s-old", 2, 7, "discord", 1.7e15)
+        );
+        assert_eq!(s.summaries[0], "old summary");
+    }
+
    #[test]
    fn create_new_file() {
        let dir = TempDir::new().unwrap();
@@ -33,7 +33,7 @@ impl SessionCache {
        self.entries.is_empty()
    }

-    /// Add a new session with its summary.
+    /// Add a new session with its summary, timestamped now.
    pub fn add(
        &mut self,
        id: &str,
@@ -47,6 +47,21 @@ impl SessionCache {
            .unwrap_or_default()
            .as_secs_f64()
            * 1_000_000.0; // microseconds
+        self.add_at(id, start_idx, end_idx, channel, summary, ts);
+    }
+
+    /// Add a session with an explicit timestamp (Unix **microseconds**, the
+    /// unit [`SessionEntry::ts`] uses) — for importers carrying sessions over
+    /// from another store, whose original time should be kept.
+    pub fn add_at(
+        &mut self,
+        id: &str,
+        start_idx: usize,
+        end_idx: usize,
+        channel: &str,
+        summary: &str,
+        ts: f64,
+    ) {
        self.entries.push(SessionEntry {
            id: id.to_string(),
            start_idx: start_idx as u64,
@@ -11,12 +11,14 @@
 //!
 //! Configuration matrix:
 //!   - Text lengths: short (50 chars), medium (200 chars), long (1000 chars)
-//!   - Embedding: 384-dim f32 (1536 bytes raw per record)
+//!   - Embedding: 384-dim, stored as float16 (the default for new stores) or
+//!     f32 with `--f32`; "raw" bytes are counted as f32 input either way
 //!   - WAL: enabled and disabled
 //!
 //! # Usage
 //! ```
-//! cargo run --release --bin footprint_bench
+//! cargo run --release --bin footprint_bench           # float16 stores
+//! cargo run --release --bin footprint_bench -- --f32  # f32 stores
 //! ```

 use std::time::Instant;
@@ -24,6 +26,9 @@ use std::time::Instant;
 use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
 use tempfile::TempDir;

+/// `--f32`: build f32 stores instead of the library's float16 default.
+static F32: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
+
 const EMBEDDING_DIM: usize = 384;

 // Raw bytes per record: 384 f32 embeddings + median text + overhead
@@ -152,6 +157,9 @@ fn measure_footprint(
    config.compression = compression;
    config.compression_level = if compression { 6 } else { 0 };
    config.compact_threshold = 0.0;
+    if F32.load(std::sync::atomic::Ordering::Relaxed) {
+        config.float16 = false;
+    }

    let mut memory = HDF5Memory::create(config).expect("HDF5Memory::create failed");

@@ -241,11 +249,19 @@ fn fmt_n(n: usize) -> String {
 // ---------------------------------------------------------------------------

 fn main() {
+    if std::env::args().skip(1).any(|a| a == "--f32") {
+        F32.store(true, std::sync::atomic::Ordering::Relaxed);
+    }
+    let stored = if F32.load(std::sync::atomic::Ordering::Relaxed) {
+        "f32 (1,536 bytes per record)"
+    } else {
+        "float16 (768 bytes per record; the default for new stores)"
+    };
    println!("=================================================================");
    println!("  ClawhDF5 Memory Footprint Benchmark");
    println!("=================================================================");
    println!();
-    println!("Embedding: 384-dim f32 = 1,536 bytes raw per record");
+    println!("Embedding: 384-dim, stored as {stored}; raw input counted as f32");
    println!("Text lengths: short=50 chars, medium=200 chars, long=1000 chars");
    println!();

@@ -3,7 +3,7 @@ name = "clawhdf5-migrate"
 version = "2.7.0"
 edition = "2024"
 rust-version.workspace = true
-description = "CLI to migrate SQLite agent memory databases to HDF5 format"
+description = "CLI to migrate SQLite agent memory databases to clawhdf5-agent stores"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
 readme = "README.md"
@@ -17,10 +17,8 @@ path = "src/main.rs"
 [dependencies]
 clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.7.0" }
 clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
-clawhdf5 = { path = "../clawhdf5", version = "2.7.0" }
 rusqlite = { version = "0.31", features = ["bundled"] }
 clap = { version = "4", features = ["derive"] }
-half = { workspace = true }

 [dev-dependencies]
 tempfile = { workspace = true }
@@ -3,9 +3,12 @@
 [![crates.io](https://img.shields.io/crates/v/clawhdf5-migrate.svg)](https://crates.io/crates/clawhdf5-migrate)
 [![docs.rs](https://img.shields.io/docsrs/clawhdf5-migrate)](https://docs.rs/clawhdf5-migrate)

-CLI tool to migrate SQLite agent memory databases to HDF5 format.
+CLI tool to migrate SQLite agent memory databases (the ZeroClaw layout) to a
+[clawhdf5-agent](https://crates.io/crates/clawhdf5-agent) store.

-Converts existing SQLite-based agent memory stores (embeddings, text chunks, metadata) into the HDF5 format used by [clawhdf5-agent](https://crates.io/crates/clawhdf5-agent).
+The output is written through `clawhdf5-agent`'s own API, so it opens with
+`HDF5Memory::open` and is searchable immediately: memory records, sessions and
+the knowledge graph (entities and relations) are carried over.

 ## Installation

@@ -16,9 +19,19 @@ cargo install clawhdf5-migrate
 ## Usage

 ```bash
-clawhdf5-migrate --input agent.db --output agent.h5
+clawhdf5-migrate --sqlite agent.db --hdf5 agent.h5 --agent-id my-agent
 ```

+Embeddings are stored as float16 (the library default for new stores); pass
+`--f32` for full precision. Every embedding must have the same dimension
+(the first row's, or `--embedding-dim`, which a source with no memory records
+requires); rows are never truncated, and the whole source is checked before an
+existing output store is replaced. `--incremental` adds only new rows to an
+existing store of the same dimension and carries over changes to rows'
+deleted flags, `--skip-deleted` leaves out tombstoned rows, and `--dry-run`
+only counts.
+See `clawhdf5-migrate --help` for every option.
+
 ## License

 MIT
@@ -1,163 +0,0 @@
-//! Read a migration HDF5 file back into the in-memory data model.
-//!
-//! Used to verify migrated content (real validation) and to merge new rows into
-//! an existing output (incremental migration). Mirrors the layout produced by
-//! [`crate::hdf5_writer`].
-
-use clawhdf5::reader::{File, Group};
-use clawhdf5_format::type_builders::AttrValue;
-
-use crate::sqlite_reader::{Entity, MemoryChunk, Relation, Session, SqliteData};
-
-type BoxErr = Box<dyn std::error::Error>;
-
-fn read_strings(group: &Group<'_>, name: &str) -> Result<Vec<String>, BoxErr> {
-    Ok(group.dataset(name)?.read_string()?)
-}
-
-fn read_i64s(group: &Group<'_>, name: &str) -> Result<Vec<i64>, BoxErr> {
-    Ok(group.dataset(name)?.read_i64()?)
-}
-
-fn read_f64s(group: &Group<'_>, name: &str) -> Result<Vec<f64>, BoxErr> {
-    Ok(group.dataset(name)?.read_f64()?)
-}
-
-/// Read the embeddings dataset as a flat `Vec<f32>` of `n * dim` values,
-/// handling both f32 and (lossy) f16 storage.
-fn read_embeddings_flat(group: &Group<'_>) -> Result<Vec<f32>, BoxErr> {
-    Ok(group.dataset("embeddings")?.read_f32()?)
-}
-
-/// Read a migration HDF5 file into a [`SqliteData`].
-pub fn read_hdf5(path: &str) -> Result<SqliteData, BoxErr> {
-    let file = File::open(path)?;
-
-    let embedding_dim = match file.root().attrs()?.get("embedding_dim") {
-        Some(AttrValue::I64(d)) => *d as usize,
-        _ => 0,
-    };
-
-    let chunks = read_chunks(&file, embedding_dim)?;
-    let sessions = read_sessions(&file)?;
-    let entities = read_entities(&file)?;
-    let relations = read_relations(&file)?;
-
-    Ok(SqliteData {
-        chunks,
-        sessions,
-        entities,
-        relations,
-        embedding_dim,
-        // Not a SQLite read — the caller (incremental migration) carries
-        // forward the current run's actual `source_path` from the fresh
-        // SQLite read instead of using this placeholder.
-        source_path: String::new(),
-    })
-}
-
-fn read_chunks(file: &File, dim: usize) -> Result<Vec<MemoryChunk>, BoxErr> {
-    let g = file.group("chunks")?;
-    let count = group_count(&g)?;
-    if count == 0 {
-        return Ok(Vec::new());
-    }
-    let ids = read_i64s(&g, "id")?;
-    let texts = read_strings(&g, "text")?;
-    let channels = read_strings(&g, "source_channel")?;
-    let timestamps = read_f64s(&g, "timestamp")?;
-    let session_ids = read_strings(&g, "session_id")?;
-    let tags = read_strings(&g, "tags")?;
-    let deleted = g.dataset("deleted")?.read_i32()?;
-    let emb_flat = read_embeddings_flat(&g)?;
-    let dim = dim.max(1);
-
-    let mut chunks = Vec::with_capacity(ids.len());
-    for (i, &id) in ids.iter().enumerate() {
-        let embedding = emb_flat
-            .get(i * dim..(i + 1) * dim)
-            .map(|s| s.to_vec())
-            .unwrap_or_default();
-        chunks.push(MemoryChunk {
-            id,
-            chunk: texts.get(i).cloned().unwrap_or_default(),
-            embedding,
-            source_channel: channels.get(i).cloned().unwrap_or_default(),
-            timestamp: timestamps.get(i).copied().unwrap_or(0.0),
-            session_id: session_ids.get(i).cloned().unwrap_or_default(),
-            tags: tags.get(i).cloned().unwrap_or_default(),
-            deleted: deleted.get(i).copied().unwrap_or(0),
-        });
-    }
-    Ok(chunks)
-}
-
-fn read_sessions(file: &File) -> Result<Vec<Session>, BoxErr> {
-    let g = file.group("sessions")?;
-    if group_count(&g)? == 0 {
-        return Ok(Vec::new());
-    }
-    let ids = read_strings(&g, "id")?;
-    let starts = read_i64s(&g, "start_idx")?;
-    let ends = read_i64s(&g, "end_idx")?;
-    let channels = read_strings(&g, "channel")?;
-    let timestamps = read_f64s(&g, "timestamp")?;
-    let summaries = read_strings(&g, "summary")?;
-    Ok((0..ids.len())
-        .map(|i| Session {
-            id: ids[i].clone(),
-            start_idx: starts.get(i).copied().unwrap_or(0),
-            end_idx: ends.get(i).copied().unwrap_or(0),
-            channel: channels.get(i).cloned().unwrap_or_default(),
-            timestamp: timestamps.get(i).copied().unwrap_or(0.0),
-            summary: summaries.get(i).cloned().unwrap_or_default(),
-        })
-        .collect())
-}
-
-fn read_entities(file: &File) -> Result<Vec<Entity>, BoxErr> {
-    let g = file.group("entities")?;
-    if group_count(&g)? == 0 {
-        return Ok(Vec::new());
-    }
-    let ids = read_i64s(&g, "id")?;
-    let names = read_strings(&g, "name")?;
-    let types = read_strings(&g, "type")?;
-    let emb_idxs = read_i64s(&g, "embedding_idx")?;
-    Ok((0..ids.len())
-        .map(|i| Entity {
-            id: ids[i],
-            name: names.get(i).cloned().unwrap_or_default(),
-            entity_type: types.get(i).cloned().unwrap_or_default(),
-            embedding_idx: emb_idxs.get(i).copied().unwrap_or(-1),
-        })
-        .collect())
-}
-
-fn read_relations(file: &File) -> Result<Vec<Relation>, BoxErr> {
-    let g = file.group("relations")?;
-    if group_count(&g)? == 0 {
-        return Ok(Vec::new());
-    }
-    let srcs = read_i64s(&g, "src")?;
-    let tgts = read_i64s(&g, "tgt")?;
-    let rels = read_strings(&g, "relation")?;
-    let weights = read_f64s(&g, "weight")?;
-    let timestamps = read_f64s(&g, "timestamp")?;
-    Ok((0..srcs.len())
-        .map(|i| Relation {
-            src: srcs[i],
-            tgt: tgts.get(i).copied().unwrap_or(0),
-            relation: rels.get(i).cloned().unwrap_or_default(),
-            weight: weights.get(i).copied().unwrap_or(1.0),
-            timestamp: timestamps.get(i).copied().unwrap_or(0.0),
-        })
-        .collect())
-}
-
-fn group_count(group: &Group<'_>) -> Result<u64, BoxErr> {
-    match group.attrs()?.get("count") {
-        Some(AttrValue::I64(n)) => Ok(*n as u64),
-        _ => Ok(0),
-    }
-}
@@ -1,366 +0,0 @@
-use clawhdf5::writer::FileBuilder;
-use clawhdf5_format::datatype::{CharacterSet, Datatype, StringPadding};
-use clawhdf5_format::type_builders::AttrValue;
-
-use crate::sqlite_reader::SqliteData;
-
-/// Options controlling HDF5 output.
-pub struct WriteOptions {
-    pub agent_id: String,
-    pub embedder: String,
-    pub compression: bool,
-    pub compression_level: u32,
-    pub float16: bool,
-}
-
-/// Write SQLite data to an HDF5 file.
-pub fn write_hdf5(
-    path: &str,
-    data: &SqliteData,
-    opts: &WriteOptions,
-) -> Result<(), Box<dyn std::error::Error>> {
-    let mut builder = FileBuilder::new();
-    let timestamp = iso8601_now();
-
-    // Root-level metadata attributes
-    builder.set_attr("agent_id", AttrValue::String(opts.agent_id.clone()));
-    builder.set_attr("embedder", AttrValue::String(opts.embedder.clone()));
-    builder.set_attr("embedding_dim", AttrValue::I64(data.embedding_dim as i64));
-    builder.set_attr("source", AttrValue::String("sqlite-migration".into()));
-    builder.set_attr("version", AttrValue::I64(1));
-    // Lineage: which SQLite database this output was migrated from and when,
-    // plus the migrator tool version — so a chain of `--incremental` runs
-    // still has an audit trail instead of every run overwriting the same
-    // static attributes (see research/03_provenance.md, INT-03).
-    builder.set_attr("source_path", AttrValue::String(data.source_path.clone()));
-    builder.set_attr("migrated_at", AttrValue::String(timestamp.clone()));
-    builder.set_attr(
-        "migrator_version",
-        AttrValue::String(env!("CARGO_PKG_VERSION").to_owned()),
-    );
-
-    write_chunks_group(&mut builder, data, opts, &timestamp);
-    write_sessions_group(&mut builder, data);
-    write_entities_group(&mut builder, data);
-    write_relations_group(&mut builder, data);
-
-    builder.write(path)?;
-    Ok(())
-}
-
-/// Current UTC time formatted as an ISO-8601 / RFC-3339 timestamp
-/// (`YYYY-MM-DDTHH:MM:SSZ`), with no external date/time dependency.
-fn iso8601_now() -> String {
-    let secs = std::time::SystemTime::now()
-        .duration_since(std::time::UNIX_EPOCH)
-        .unwrap_or_default()
-        .as_secs();
-    let days = (secs / 86_400) as i64;
-    let time_of_day = secs % 86_400;
-    let (h, m, s) = (
-        time_of_day / 3600,
-        (time_of_day % 3600) / 60,
-        time_of_day % 60,
-    );
-    let (y, mo, d) = civil_from_days(days);
-    format!("{y:04}-{mo:02}-{d:02}T{h:02}:{m:02}:{s:02}Z")
-}
-
-/// Days-since-epoch to (year, month, day), Howard Hinnant's `civil_from_days`
-/// algorithm (proleptic Gregorian calendar, valid for the full `i64` range).
-fn civil_from_days(z: i64) -> (i64, u32, u32) {
-    let z = z + 719_468;
-    let era = if z >= 0 { z } else { z - 146_096 } / 146_097;
-    let doe = (z - era * 146_097) as u64; // [0, 146096]
-    let yoe = (doe - doe / 1460 + doe / 36_524 - doe / 146_096) / 365; // [0, 399]
-    let y = yoe as i64 + era * 400;
-    let doy = doe - (365 * yoe + yoe / 4 - yoe / 100); // [0, 365]
-    let mp = (5 * doy + 2) / 153; // [0, 11]
-    let d = (doy - (153 * mp + 2) / 5 + 1) as u32; // [1, 31]
-    let m = (if mp < 10 { mp + 3 } else { mp - 9 }) as u32; // [1, 12]
-    let y = if m <= 2 { y + 1 } else { y };
-    (y, m, d)
-}
-
-/// Build a fixed-length string Datatype from the max byte length of the items.
-fn string_dtype(max_len: usize) -> Datatype {
-    Datatype::String {
-        size: max_len.max(1) as u32,
-        padding: StringPadding::NullPad,
-        charset: CharacterSet::Utf8,
-    }
-}
-
-/// Pack a slice of strings into null-padded raw bytes of uniform width.
-fn pack_strings(strings: &[String]) -> (Vec<u8>, usize) {
-    let max_len = strings.iter().map(|s| s.len()).max().unwrap_or(0).max(1);
-    let mut buf = vec![0u8; strings.len() * max_len];
-    for (i, s) in strings.iter().enumerate() {
-        let start = i * max_len;
-        let bytes = s.as_bytes();
-        let copy_len = bytes.len().min(max_len);
-        buf[start..start + copy_len].copy_from_slice(&bytes[..copy_len]);
-    }
-    (buf, max_len)
-}
-
-fn apply_compression(ds: &mut clawhdf5_format::type_builders::DatasetBuilder, opts: &WriteOptions) {
-    if opts.compression {
-        ds.with_deflate(opts.compression_level);
-        ds.with_shuffle();
-    }
-}
-
-fn write_chunks_group(
-    builder: &mut FileBuilder,
-    data: &SqliteData,
-    opts: &WriteOptions,
-    timestamp: &str,
-) {
-    let mut group = builder.create_group("chunks");
-    let n = data.chunks.len() as u64;
-
-    if n == 0 {
-        group.set_attr("count", AttrValue::I64(0));
-        builder.add_group(group.finish());
-        return;
-    }
-
-    group.set_attr("count", AttrValue::I64(n as i64));
-
-    // Source attribution attached directly to the content-bearing datasets
-    // (SHA-256 of the raw bytes + creator/timestamp/source), so the chunk
-    // text and embeddings each carry their own verifiable provenance
-    // (see clawhdf5_format::provenance / `Dataset::verify_provenance`).
-    let source_opt = if data.source_path.is_empty() {
-        None
-    } else {
-        Some(data.source_path.as_str())
-    };
-
-    // ids
-    let ids: Vec<i64> = data.chunks.iter().map(|c| c.id).collect();
-    group.create_dataset("id").with_i64_data(&ids);
-
-    // text
-    let texts: Vec<String> = data.chunks.iter().map(|c| c.chunk.clone()).collect();
-    let (text_raw, text_len) = pack_strings(&texts);
-    group
-        .create_dataset("text")
-        .with_compound_data(string_dtype(text_len), text_raw, n)
-        .with_provenance("clawhdf5-migrate", timestamp, source_opt);
-
-    // embeddings - flatten to [N, dim]
-    let dim = data.embedding_dim;
-    if opts.float16 {
-        let f16_data: Vec<u16> = data
-            .chunks
-            .iter()
-            .flat_map(|c| {
-                c.embedding
-                    .iter()
-                    .map(|&v| half::f16::from_f32(v).to_bits())
-            })
-            .collect();
-        let raw: Vec<u8> = f16_data.iter().flat_map(|v| v.to_le_bytes()).collect();
-        let f16_dtype = Datatype::FloatingPoint {
-            size: 2,
-            byte_order: clawhdf5_format::datatype::DatatypeByteOrder::LittleEndian,
-            bit_offset: 0,
-            bit_precision: 16,
-            exponent_location: 10,
-            exponent_size: 5,
-            mantissa_location: 0,
-            mantissa_size: 10,
-            exponent_bias: 15,
-        };
-        let ds = group
-            .create_dataset("embeddings")
-            .with_compound_data(f16_dtype, raw, n)
-            .with_shape(&[n, dim as u64])
-            .with_provenance("clawhdf5-migrate", timestamp, source_opt);
-        apply_compression(ds, opts);
-    } else {
-        let flat: Vec<f32> = data
-            .chunks
-            .iter()
-            .flat_map(|c| c.embedding.iter().copied())
-            .collect();
-        let ds = group
-            .create_dataset("embeddings")
-            .with_f32_data(&flat)
-            .with_shape(&[n, dim as u64])
-            .with_provenance("clawhdf5-migrate", timestamp, source_opt);
-        apply_compression(ds, opts);
-    }
-
-    // source_channel
-    let channels: Vec<String> = data
-        .chunks
-        .iter()
-        .map(|c| c.source_channel.clone())
-        .collect();
-    let (ch_raw, ch_len) = pack_strings(&channels);
-    group
-        .create_dataset("source_channel")
-        .with_compound_data(string_dtype(ch_len), ch_raw, n);
-
-    // timestamp
-    let timestamps: Vec<f64> = data.chunks.iter().map(|c| c.timestamp).collect();
-    group.create_dataset("timestamp").with_f64_data(&timestamps);
-
-    // session_id
-    let sess_ids: Vec<String> = data.chunks.iter().map(|c| c.session_id.clone()).collect();
-    let (sid_raw, sid_len) = pack_strings(&sess_ids);
-    group
-        .create_dataset("session_id")
-        .with_compound_data(string_dtype(sid_len), sid_raw, n);
-
-    // tags
-    let tags: Vec<String> = data.chunks.iter().map(|c| c.tags.clone()).collect();
-    let (tag_raw, tag_len) = pack_strings(&tags);
-    group
-        .create_dataset("tags")
-        .with_compound_data(string_dtype(tag_len), tag_raw, n);
-
-    // deleted
-    let deleted: Vec<i32> = data.chunks.iter().map(|c| c.deleted).collect();
-    group.create_dataset("deleted").with_i32_data(&deleted);
-
-    builder.add_group(group.finish());
-}
-
-fn write_sessions_group(builder: &mut FileBuilder, data: &SqliteData) {
-    let mut group = builder.create_group("sessions");
-    let n = data.sessions.len() as u64;
-    group.set_attr("count", AttrValue::I64(n as i64));
-
-    if n == 0 {
-        builder.add_group(group.finish());
-        return;
-    }
-
-    let ids: Vec<String> = data.sessions.iter().map(|s| s.id.clone()).collect();
-    let (id_raw, id_len) = pack_strings(&ids);
-    group
-        .create_dataset("id")
-        .with_compound_data(string_dtype(id_len), id_raw, n);
-
-    let start_idxs: Vec<i64> = data.sessions.iter().map(|s| s.start_idx).collect();
-    group.create_dataset("start_idx").with_i64_data(&start_idxs);
-
-    let end_idxs: Vec<i64> = data.sessions.iter().map(|s| s.end_idx).collect();
-    group.create_dataset("end_idx").with_i64_data(&end_idxs);
-
-    let channels: Vec<String> = data.sessions.iter().map(|s| s.channel.clone()).collect();
-    let (ch_raw, ch_len) = pack_strings(&channels);
-    group
-        .create_dataset("channel")
-        .with_compound_data(string_dtype(ch_len), ch_raw, n);
-
-    let timestamps: Vec<f64> = data.sessions.iter().map(|s| s.timestamp).collect();
-    group.create_dataset("timestamp").with_f64_data(&timestamps);
-
-    let summaries: Vec<String> = data.sessions.iter().map(|s| s.summary.clone()).collect();
-    let (sum_raw, sum_len) = pack_strings(&summaries);
-    group
-        .create_dataset("summary")
-        .with_compound_data(string_dtype(sum_len), sum_raw, n);
-
-    builder.add_group(group.finish());
-}
-
-fn write_entities_group(builder: &mut FileBuilder, data: &SqliteData) {
-    let mut group = builder.create_group("entities");
-    let n = data.entities.len() as u64;
-    group.set_attr("count", AttrValue::I64(n as i64));
-
-    if n == 0 {
-        builder.add_group(group.finish());
-        return;
-    }
-
-    let ids: Vec<i64> = data.entities.iter().map(|e| e.id).collect();
-    group.create_dataset("id").with_i64_data(&ids);
-
-    let names: Vec<String> = data.entities.iter().map(|e| e.name.clone()).collect();
-    let (name_raw, name_len) = pack_strings(&names);
-    group
-        .create_dataset("name")
-        .with_compound_data(string_dtype(name_len), name_raw, n);
-
-    let types: Vec<String> = data
-        .entities
-        .iter()
-        .map(|e| e.entity_type.clone())
-        .collect();
-    let (type_raw, type_len) = pack_strings(&types);
-    group
-        .create_dataset("type")
-        .with_compound_data(string_dtype(type_len), type_raw, n);
-
-    let emb_idxs: Vec<i64> = data.entities.iter().map(|e| e.embedding_idx).collect();
-    group
-        .create_dataset("embedding_idx")
-        .with_i64_data(&emb_idxs);
-
-    builder.add_group(group.finish());
-}
-
-fn write_relations_group(builder: &mut FileBuilder, data: &SqliteData) {
-    let mut group = builder.create_group("relations");
-    let n = data.relations.len() as u64;
-    group.set_attr("count", AttrValue::I64(n as i64));
-
-    if n == 0 {
-        builder.add_group(group.finish());
-        return;
-    }
-
-    let srcs: Vec<i64> = data.relations.iter().map(|r| r.src).collect();
-    group.create_dataset("src").with_i64_data(&srcs);
-
-    let tgts: Vec<i64> = data.relations.iter().map(|r| r.tgt).collect();
-    group.create_dataset("tgt").with_i64_data(&tgts);
-
-    let rels: Vec<String> = data.relations.iter().map(|r| r.relation.clone()).collect();
-    let (rel_raw, rel_len) = pack_strings(&rels);
-    group
-        .create_dataset("relation")
-        .with_compound_data(string_dtype(rel_len), rel_raw, n);
-
-    let weights: Vec<f64> = data.relations.iter().map(|r| r.weight).collect();
-    group.create_dataset("weight").with_f64_data(&weights);
-
-    let timestamps: Vec<f64> = data.relations.iter().map(|r| r.timestamp).collect();
-    group.create_dataset("timestamp").with_f64_data(&timestamps);
-
-    builder.add_group(group.finish());
-}
-
-#[cfg(test)]
-mod time_tests {
-    use super::civil_from_days;
-
-    #[test]
-    fn epoch_day_zero_is_1970_01_01() {
-        assert_eq!(civil_from_days(0), (1970, 1, 1));
-    }
-
-    #[test]
-    fn known_dates_roundtrip() {
-        // 2026-08-16 is 20,681 days after 1970-01-01.
-        assert_eq!(civil_from_days(20_681), (2026, 8, 16));
-        // 2000-02-29 (leap day itself) and 2000-03-01 (the day after).
-        assert_eq!(civil_from_days(11_016), (2000, 2, 29));
-        assert_eq!(civil_from_days(11_017), (2000, 3, 1));
-    }
-
-    #[test]
-    fn iso8601_now_has_expected_shape() {
-        let ts = super::iso8601_now();
-        assert_eq!(ts.len(), "2026-08-16T00:00:00Z".len());
-        assert!(ts.starts_with("20")); // sanity: 21st-century year
-        assert!(ts.ends_with('Z'));
-    }
-}
@@ -50,12 +50,8 @@ pub struct SqliteData {
    pub sessions: Vec<Session>,
    pub entities: Vec<Entity>,
    pub relations: Vec<Relation>,
+    /// `--embedding-dim`, or the first row's; 0 when neither exists.
    pub embedding_dim: usize,
-    /// Filesystem path of the SQLite database this data was read from, for
-    /// provenance attribution on the HDF5 output. Empty when the data did
-    /// not come directly from a SQLite read (e.g. re-read of a prior HDF5
-    /// migration output for an incremental merge).
-    pub source_path: String,
 }

 /// A table name plus the ordered column names the reader maps by position.
@@ -167,11 +163,13 @@ pub fn read_counts(
    })
 }

-/// Auto-detect embedding dimension from the first chunk's BLOB size.
+/// Auto-detect embedding dimension from the BLOB size of the first chunk (in
+/// id order, deleted or not).
 fn detect_embedding_dim(conn: &Connection, config: &SchemaConfig) -> SqlResult<Option<usize>> {
    let emb_col = config.chunks.columns.get(2).copied().unwrap_or("embedding");
+    let id_col = config.chunks.columns.first().copied().unwrap_or("id");
    let mut stmt = conn.prepare(&format!(
-        "SELECT {emb_col} FROM {} LIMIT 1",
+        "SELECT {emb_col} FROM {} ORDER BY {id_col} LIMIT 1",
        config.chunks.table
    ))?;
    let mut rows = stmt.query([])?;
@@ -195,24 +193,16 @@ fn blob_to_f32(blob: &[u8]) -> Vec<f32> {
 /// Read all data from a ZeroClaw SQLite database.
 ///
 /// If `skip_deleted` is true, rows with `deleted=1` are excluded from chunks.
-/// If `embedding_dim` is `None`, auto-detect from the first row.
+/// If `embedding_dim` is `None`, auto-detect from the first row (0 when there
+/// are no rows). Embeddings are returned at their full stored length whatever
+/// the dimension: checking that every row matches it is the writer's job
+/// (`store_writer::write_store`), so a mismatch is an error, not silent
+/// truncation.
 pub fn read_sqlite(
    path: &str,
    skip_deleted: bool,
    embedding_dim: Option<usize>,
    config: &SchemaConfig,
-) -> Result<SqliteData, Box<dyn std::error::Error>> {
-    read_sqlite_filtered(path, skip_deleted, embedding_dim, config, 0)
-}
-
-/// Like [`read_sqlite`] but only reads chunks whose id is greater than
-/// `min_chunk_id` (0 = all). Used for incremental migration.
-pub fn read_sqlite_filtered(
-    path: &str,
-    skip_deleted: bool,
-    embedding_dim: Option<usize>,
-    config: &SchemaConfig,
-    min_chunk_id: i64,
 ) -> Result<SqliteData, Box<dyn std::error::Error>> {
    let conn = Connection::open(path)?;

@@ -221,7 +211,7 @@ pub fn read_sqlite_filtered(
        None => detect_embedding_dim(&conn, config)?.unwrap_or(0),
    };

-    let chunks = read_chunks(&conn, skip_deleted, dim, config, min_chunk_id)?;
+    let chunks = read_chunks(&conn, skip_deleted, config)?;
    let sessions = read_sessions(&conn, config)?;
    let entities = read_entities(&conn, config)?;
    let relations = read_relations(&conn, config)?;
@@ -232,42 +222,43 @@ pub fn read_sqlite_filtered(
        entities,
        relations,
        embedding_dim: dim,
-        source_path: path.to_owned(),
    })
 }

 fn read_chunks(
    conn: &Connection,
    skip_deleted: bool,
-    expected_dim: usize,
    config: &SchemaConfig,
-    min_chunk_id: i64,
 ) -> SqlResult<Vec<MemoryChunk>> {
    let id_col = config.chunks.columns.first().copied().unwrap_or("id");
    let deleted_col = config.chunks.columns.get(7).copied().unwrap_or("deleted");
-    let mut conds = Vec::new();
+    let mut where_clause = String::new();
    if skip_deleted {
-        conds.push(format!("{deleted_col} = 0"));
+        where_clause = format!(" WHERE {deleted_col} = 0");
    }
-    if min_chunk_id > 0 {
-        conds.push(format!("{id_col} > {min_chunk_id}"));
-    }
-    let where_clause = if conds.is_empty() {
-        String::new()
-    } else {
-        format!(" WHERE {}", conds.join(" AND "))
-    };
+    // In id order, so the store's records follow the source's order.
+    where_clause.push_str(&format!(" ORDER BY {id_col}"));
    let sql = config.chunks.select(&where_clause);

    let mut stmt = conn.prepare(&sql)?;
    let rows = stmt.query_map([], |row| {
        let blob: Vec<u8> = row.get(2)?;
-        let mut embedding = blob_to_f32(&blob);
-
-        // Validate/truncate to expected dimension
-        if expected_dim > 0 {
-            embedding.truncate(expected_dim);
+        if !blob.len().is_multiple_of(4) {
+            let id: i64 = row.get(0)?;
+            return Err(rusqlite::Error::FromSqlConversionFailure(
+                2,
+                rusqlite::types::Type::Blob,
+                format!(
+                    "chunk id {id}: embedding BLOB is {} bytes, not a whole number of \
+                     little-endian f32 values",
+                    blob.len()
+                )
+                .into(),
+            ));
        }
+        // Read at full length: rows of the wrong dimension are rejected by
+        // the writer, never truncated to fit.
+        let embedding = blob_to_f32(&blob);

        Ok(MemoryChunk {
            id: row.get(0)?,
@@ -0,0 +1,407 @@
+//! Write migrated SQLite data into a clawhdf5-agent store.
+//!
+//! Everything goes through `clawhdf5-agent`'s own API — `HDF5Memory::create`
+//! (or `open` for `--incremental`), `save_batch`, `delete_batch`, the session
+//! cache and the knowledge graph — so the result is an ordinary agent store
+//! that `HDF5Memory::open` accepts, not a second hand-built copy of its schema.
+
+use std::collections::{HashMap, HashSet};
+use std::path::Path;
+
+use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
+use clawhdf5_format::float16::round_to_f16;
+
+use crate::sqlite_reader::{MemoryChunk, SqliteData};
+
+type BoxErr = Box<dyn std::error::Error>;
+
+/// SQLite timestamps are Unix seconds; the agent's session and relation
+/// timestamps are Unix microseconds (memory records stay in seconds).
+pub const US_PER_SEC: f64 = 1_000_000.0;
+
+/// Options controlling the output store.
+#[derive(Debug, Clone)]
+pub struct WriteOptions {
+    pub agent_id: String,
+    pub embedder: String,
+    pub compression: bool,
+    pub compression_level: u32,
+    /// Store full-precision `f32` embeddings instead of the library default
+    /// (half precision). Only applies to a newly created store: an existing
+    /// store keeps the precision it was created with.
+    pub f32: bool,
+    /// Add to the store at the output path if there is one, instead of
+    /// replacing it.
+    pub incremental: bool,
+    /// Leave out deleted source rows that are not in the store. (A deleted
+    /// row that matches an active store record still tombstones it, so pass
+    /// deleted rows in `data` for an incremental run.)
+    pub skip_deleted: bool,
+}
+
+/// What the migration wrote, and where each source row went, so validation
+/// can compare the store with the source row by row.
+#[derive(Debug, Default)]
+pub struct Migration {
+    /// Whether the output store existed and was added to (`--incremental`).
+    pub appended_to_existing: bool,
+    /// The store's embedding precision.
+    pub float16: bool,
+    pub embedding_dim: usize,
+    /// Records in the store after the migration (including tombstones).
+    pub store_count: usize,
+    /// `(store index, source chunk index)` of every record written.
+    pub records: Vec<(usize, usize)>,
+    /// Source chunks already in the store (incremental), not written again.
+    pub chunks_present: usize,
+    /// `(store index, source chunk index)` of records that were active in
+    /// the store but whose source row is now deleted (incremental): they were
+    /// tombstoned by this run.
+    pub deleted_in_store: Vec<(usize, usize)>,
+    /// Source rows that were deleted in the store but are active in the
+    /// source (incremental): the agent has no un-delete, so each was written
+    /// again as a new record (counted in `records` too).
+    pub restored: usize,
+    /// Deleted source rows left out because of `skip_deleted`.
+    pub deleted_skipped: usize,
+    /// `(store session index, source session index)` of each session written.
+    pub sessions: Vec<(usize, usize)>,
+    pub sessions_present: usize,
+    /// `(store entity id, source entity index)` of each entity written.
+    pub entities: Vec<(u64, usize)>,
+    pub entities_present: usize,
+    /// SQLite entity id -> store entity id, for every source entity.
+    pub entity_ids: HashMap<i64, u64>,
+    /// `(store relation index, source relation index)` of each relation written.
+    pub relations: Vec<(usize, usize)>,
+    pub relations_present: usize,
+    /// Source relations naming an entity id that is not in the entities
+    /// table; the knowledge graph cannot hold them, so they are skipped.
+    pub dangling_relations: Vec<usize>,
+    /// Messages of the write-anomaly alerts the agent raised while importing
+    /// (informational; they never block a save — a bulk import typically
+    /// trips the write-rate check).
+    pub anomaly_alerts: Vec<String>,
+}
+
+/// Identity of a memory record for incremental de-duplication: every field
+/// the agent stores except the embedding (whose stored form depends on the
+/// store's precision).
+type RecordKey = (String, String, String, String, u64);
+
+fn record_key(
+    chunk: &str,
+    source_channel: &str,
+    session_id: &str,
+    tags: &str,
+    ts: f64,
+) -> RecordKey {
+    (
+        chunk.to_owned(),
+        source_channel.to_owned(),
+        session_id.to_owned(),
+        tags.to_owned(),
+        ts.to_bits(),
+    )
+}
+
+/// Reject rows the agent would otherwise store differently from the source,
+/// or not at all: an embedding of a different length from the store's
+/// dimension (the agent pads/truncates silently), an empty embedding, or, in
+/// a float16 store, a value beyond the half-precision range.
+///
+/// Every source row is checked, including ones that end up not being written
+/// (already in the store, or deleted and skipped): the source must be
+/// consistent as a whole, and the check runs before the store is touched.
+fn check_chunks(chunks: &[MemoryChunk], dim: usize, float16: bool) -> Result<(), BoxErr> {
+    for c in chunks {
+        if c.embedding.is_empty() {
+            return Err(format!(
+                "chunk id {}: the embedding is empty; an agent store needs an embedding \
+                 for every record",
+                c.id
+            )
+            .into());
+        }
+        if c.embedding.len() != dim {
+            return Err(format!(
+                "chunk id {}: embedding has {} values, expected {dim}; every row must have \
+                 the store's dimension (detected from the first row unless --embedding-dim \
+                 is given), and rows are never truncated or padded to fit",
+                c.id,
+                c.embedding.len()
+            )
+            .into());
+        }
+        if float16
+            && let Some((k, v)) = c
+                .embedding
+                .iter()
+                .enumerate()
+                .find(|&(_, &v)| v.is_finite() && round_to_f16(v).is_infinite())
+        {
+            return Err(format!(
+                "chunk id {}: embedding[{k}] = {v} is outside the half-precision range \
+                 (±65504) of a float16 store; migrate with --f32",
+                c.id
+            )
+            .into());
+        }
+    }
+    Ok(())
+}
+
+/// Migrate `data` into the agent store at `path`.
+///
+/// Without `opts.incremental` (or when nothing exists at `path`) a new store
+/// is created, replacing any file there — but only once every source row has
+/// passed [`check_chunks`], so a source that cannot be migrated leaves an
+/// existing store untouched. With it, the existing store is opened and only
+/// source rows it does not already hold are added: memory records are
+/// matched on their content, sessions on their id, entities on name and
+/// type, relations on (source, target, relation). A matched record then
+/// takes the source row's deleted flag: see [`Migration::deleted_in_store`]
+/// and [`Migration::restored`].
+pub fn write_store(
+    path: &Path,
+    data: &SqliteData,
+    opts: &WriteOptions,
+) -> Result<Migration, BoxErr> {
+    let existing = opts.incremental && path.exists();
+    let mut mem = if existing {
+        // `open` does not modify the store beyond what the agent itself does
+        // on open; the checks below run before anything is written.
+        let mem = HDF5Memory::open(path)?;
+        let dim = mem.config().embedding_dim;
+        // `data.embedding_dim` is 0 only for a source with no records and no
+        // --embedding-dim, which has no dimension to disagree with.
+        if data.embedding_dim != 0 && dim != data.embedding_dim {
+            let hint = if dim == 0 {
+                " (a store created from a source with no memory records; re-create it \
+                 with --embedding-dim)"
+            } else {
+                ""
+            };
+            return Err(format!(
+                "the store at {} has embedding_dim {dim}{hint}, the source {}; \
+                 embeddings of a different dimension cannot be added to it",
+                path.display(),
+                data.embedding_dim
+            )
+            .into());
+        }
+        check_chunks(&data.chunks, dim, mem.config().float16)?;
+        mem
+    } else {
+        // (With records, a dimension of 0 means an empty first embedding,
+        // which `check_chunks` reports more precisely.)
+        if data.embedding_dim == 0 && data.chunks.is_empty() {
+            return Err(
+                "the source has no memory records to detect the embedding dimension \
+                        from; pass --embedding-dim (the dimension of the agent's embedder), \
+                        or the store could never hold a record"
+                    .into(),
+            );
+        }
+        let mut config = MemoryConfig::new(path.to_path_buf(), &opts.agent_id, data.embedding_dim);
+        config.embedder = opts.embedder.clone();
+        config.compression = opts.compression;
+        config.compression_level = opts.compression_level;
+        // Only ever switch the library default off (as `clawhdf5-cli create`).
+        if opts.f32 {
+            config.float16 = false;
+        }
+        // Before `create`, which replaces whatever is at `path`.
+        check_chunks(&data.chunks, config.embedding_dim, config.float16)?;
+        HDF5Memory::create(config)?
+    };
+    let float16 = mem.config().float16;
+    let dim = mem.config().embedding_dim;
+
+    let mut m = Migration {
+        appended_to_existing: existing,
+        float16,
+        embedding_dim: dim,
+        ..Migration::default()
+    };
+
+    // ---- Memory records --------------------------------------------------
+    // Store indices of every record the store already holds, by content, so
+    // a source row that appears twice is only treated as present as often
+    // as the store has it.
+    let mut present: HashMap<RecordKey, Vec<usize>> = HashMap::new();
+    if existing {
+        let c = &mem.cache;
+        for i in 0..c.len() {
+            let key = record_key(
+                &c.chunks[i],
+                &c.source_channels[i],
+                &c.session_ids[i],
+                &c.tags[i],
+                c.timestamps[i],
+            );
+            present.entry(key).or_default().push(i);
+        }
+    }
+    let key_of = |c: &MemoryChunk| {
+        record_key(
+            &c.chunk,
+            &c.source_channel,
+            &c.session_id,
+            &c.tags,
+            c.timestamp,
+        )
+    };
+    let tombstoned = |idx: usize| mem.cache.tombstones[idx] != 0;
+    // Pass 1: a store record in the same deleted state as the source row.
+    let mut unmatched: Vec<usize> = Vec::new();
+    for (i, c) in data.chunks.iter().enumerate() {
+        let src_deleted = c.deleted != 0;
+        let hit = present.get_mut(&key_of(c)).and_then(|idxs| {
+            let at = idxs.iter().position(|&x| tombstoned(x) == src_deleted)?;
+            Some(idxs.remove(at))
+        });
+        match hit {
+            Some(_) => m.chunks_present += 1,
+            None => unmatched.push(i),
+        }
+    }
+    // Pass 2: a store record whose deleted state differs — the source row
+    // was deleted or restored since the last migration. The source wins.
+    let mut new_chunks: Vec<usize> = Vec::with_capacity(unmatched.len());
+    let mut delete_in_store: Vec<usize> = Vec::new();
+    for i in unmatched {
+        let c = &data.chunks[i];
+        let hit = present
+            .get_mut(&key_of(c))
+            .and_then(|idxs| (!idxs.is_empty()).then(|| idxs.remove(0)));
+        match hit {
+            // Active in the store, deleted in the source: tombstone it.
+            Some(idx) if c.deleted != 0 => {
+                m.deleted_in_store.push((idx, i));
+                delete_in_store.push(idx);
+            }
+            // Deleted in the store, active in the source. The agent has no
+            // un-delete, so the row is written again as a new active record
+            // (the tombstone stays until the store is compacted).
+            Some(_) => {
+                m.restored += 1;
+                new_chunks.push(i);
+            }
+            None if c.deleted != 0 && opts.skip_deleted => m.deleted_skipped += 1,
+            None => new_chunks.push(i),
+        }
+    }
+    new_chunks.sort_unstable();
+    let to_write: Vec<&MemoryChunk> = new_chunks.iter().map(|&i| &data.chunks[i]).collect();
+
+    // ---- Sessions (in the cache; persisted by the save_batch checkpoint) ---
+    let known_sessions: HashSet<String> = mem
+        .sessions()
+        .entries
+        .iter()
+        .map(|e| e.id.clone())
+        .collect();
+    for (i, s) in data.sessions.iter().enumerate() {
+        if known_sessions.contains(&s.id) {
+            m.sessions_present += 1;
+            continue;
+        }
+        let sessions = mem.sessions_mut();
+        let at = sessions.len();
+        sessions.add_at(
+            &s.id,
+            s.start_idx.max(0) as usize,
+            s.end_idx.max(0) as usize,
+            &s.channel,
+            &s.summary,
+            s.timestamp * US_PER_SEC,
+        );
+        m.sessions.push((at, i));
+    }
+
+    // ---- Knowledge graph -------------------------------------------------
+    let kg = mem.knowledge_mut();
+    // Matched only against what the store held before this run: the source
+    // itself is copied as it is, duplicates included.
+    let by_name_type: HashMap<(String, String), u64> = kg
+        .entities
+        .iter()
+        .map(|e| ((e.name.clone(), e.entity_type.clone()), e.id))
+        .collect();
+    for (i, e) in data.entities.iter().enumerate() {
+        let key = (e.name.clone(), e.entity_type.clone());
+        let id = match by_name_type.get(&key) {
+            Some(&id) => {
+                m.entities_present += 1;
+                id
+            }
+            None => {
+                let id = kg.add_entity(&e.name, &e.entity_type, e.embedding_idx);
+                m.entities.push((id, i));
+                id
+            }
+        };
+        m.entity_ids.insert(e.id, id);
+    }
+    let known_relations: HashSet<(u64, u64, String)> = kg
+        .relations
+        .iter()
+        .map(|r| (r.src, r.tgt, r.relation.clone()))
+        .collect();
+    for (i, r) in data.relations.iter().enumerate() {
+        let (Some(&src), Some(&tgt)) = (m.entity_ids.get(&r.src), m.entity_ids.get(&r.tgt)) else {
+            m.dangling_relations.push(i);
+            continue;
+        };
+        if known_relations.contains(&(src, tgt, r.relation.clone())) {
+            m.relations_present += 1;
+            continue;
+        }
+        let at = kg.relations.len();
+        kg.add_relation(src, tgt, &r.relation, r.weight as f32);
+        kg.relations[at].ts = r.timestamp * US_PER_SEC;
+        m.relations.push((at, i));
+    }
+
+    // ---- Write: one checkpoint for records, sessions and graph -----------
+    let entries: Vec<MemoryEntry> = to_write
+        .iter()
+        .map(|c| MemoryEntry {
+            chunk: c.chunk.clone(),
+            embedding: c.embedding.clone(),
+            source_channel: c.source_channel.clone(),
+            timestamp: c.timestamp,
+            session_id: c.session_id.clone(),
+            tags: c.tags.clone(),
+        })
+        .collect();
+    let indices = mem.save_batch(entries)?;
+    m.records = indices
+        .iter()
+        .copied()
+        .zip(new_chunks.iter().copied())
+        .collect();
+
+    // Rows deleted in the source stay deleted: tombstones, as the agent's own
+    // `delete` leaves them (not compacted away).
+    // Records matched in the store whose source row has since been deleted
+    // are tombstoned too.
+    let tombstones: Vec<usize> = m
+        .records
+        .iter()
+        .filter(|&&(_, src)| data.chunks[src].deleted != 0)
+        .map(|&(idx, _)| idx)
+        .chain(delete_in_store)
+        .collect();
+    mem.delete_batch(&tombstones)?;
+
+    m.anomaly_alerts = mem
+        .take_anomaly_alerts()
+        .into_iter()
+        .map(|a| a.message)
+        .collect();
+    m.store_count = mem.count();
+    drop(mem); // release the single-writer lock before anyone re-opens it
+    Ok(m)
+}
@@ -1,192 +1,266 @@
-use clawhdf5::reader::File as Hdf5File;
-use clawhdf5_format::provenance::VerifyResult;
+//! Validate a migration by reading the store back the way an agent would:
+//! through `HDF5Memory::open_read_only`, comparing what it loads with the
+//! SQLite source, and running a search for a migrated record.
+
+use std::path::Path;
+
+use clawhdf5_agent::{AgentMemory, HDF5Memory, SearchOptions};
+use clawhdf5_format::float16::round_to_f16;

-use crate::hdf5_reader::read_hdf5;
 use crate::sqlite_reader::SqliteData;
+use crate::store_writer::{Migration, US_PER_SEC};

 type BoxErr = Box<dyn std::error::Error>;

 /// Summary of a migration validation.
 #[derive(Debug)]
 pub struct ValidationSummary {
-    pub chunks: u64,
-    pub sessions: u64,
-    pub entities: u64,
-    pub relations: u64,
-    pub embedding_dim: u64,
-    /// Number of rows whose full content was compared against the source.
+    /// Records in the store (including tombstones).
+    pub count: usize,
+    /// Records in the store that are not deleted.
+    pub active: usize,
+    pub sessions: usize,
+    pub entities: usize,
+    pub relations: usize,
+    pub embedding_dim: usize,
+    pub float16: bool,
+    /// Rows whose full content was compared against the source.
    pub rows_checked: u64,
-    /// Whether the `chunks/text` and `chunks/embeddings` SHINES provenance
-    /// hashes (written via [`crate::hdf5_writer`]) were both present and
-    /// matched their recomputed SHA-256 on read-back. `false` when either
-    /// dataset has no provenance metadata (e.g. an older output file) or
-    /// there are zero chunks to check.
-    pub provenance_verified: bool,
+    /// Whether a search for a migrated record found it (`false` when there
+    /// was no active migrated record with an embedding to search for).
+    pub search_checked: bool,
 }

-/// Validate a migrated HDF5 file against the source data.
+/// Validate the store at `path` against the source rows `migration` wrote.
 ///
-/// Reads the written file back and compares actual content — chunk text,
-/// embeddings, and every session/entity/relation field — to the source, not
-/// just the row counts. When `full` is false a representative sample of chunk
-/// rows is content-checked (counts and all other groups are always checked in
-/// full); when `full` is true every chunk row is compared too. `float16` widens
-/// the embedding tolerance to allow for half-precision quantization.
-pub fn validate_hdf5(
-    path: &str,
+/// Counts and the session / entity / relation rows are always checked in
+/// full. Memory records are content-checked on a representative sample, or
+/// all of them with `full`. Embeddings must match exactly: the source values
+/// themselves in an `f32` store, their [`round_to_f16`] in a `float16` one.
+pub fn validate_store(
+    path: &Path,
    source: &SqliteData,
+    migration: &Migration,
    full: bool,
-    float16: bool,
 ) -> Result<ValidationSummary, BoxErr> {
-    let got = read_hdf5(path)?;
-    let provenance_verified = verify_chunk_provenance(path)?;
+    let mut mem = HDF5Memory::open_read_only(path)?;
+    let float16 = mem.config().float16;
+    let dim = mem.config().embedding_dim;

    // ---- Counts ----
-    check_count("chunk", got.chunks.len(), source.chunks.len())?;
-    check_count("session", got.sessions.len(), source.sessions.len())?;
-    check_count("entity", got.entities.len(), source.entities.len())?;
-    check_count("relation", got.relations.len(), source.relations.len())?;
-    if got.embedding_dim != source.embedding_dim {
+    check_count("record", mem.count(), migration.store_count)?;
+    if float16 != migration.float16 {
        return Err(format!(
-            "embedding_dim mismatch: HDF5 has {}, source has {}",
-            got.embedding_dim, source.embedding_dim
+            "float16 mismatch: store {float16}, expected {}",
+            migration.float16
        )
        .into());
    }
+    if dim != migration.embedding_dim {
+        return Err(format!(
+            "embedding_dim mismatch: store has {dim}, expected {}",
+            migration.embedding_dim
+        )
+        .into());
+    }
+    if !migration.appended_to_existing {
+        check_count("record", mem.count(), migration.records.len())?;
+        check_count("session", mem.sessions().len(), migration.sessions.len())?;
+        check_count(
+            "entity",
+            mem.knowledge().entities.len(),
+            migration.entities.len(),
+        )?;
+        check_count(
+            "relation",
+            mem.knowledge().relations.len(),
+            migration.relations.len(),
+        )?;
+    }

-    // ---- Chunk content (sampled or full) ----
-    let (emb_abs, emb_rel) = if float16 { (1e-2, 1e-2) } else { (1e-4, 0.0) };
+    // ---- Memory records (sampled or full) ----
    let mut rows_checked = 0u64;
-    for i in sample_indices(source.chunks.len(), full) {
-        let (s, g) = (&source.chunks[i], &got.chunks[i]);
-        if s.id != g.id {
-            return Err(field_err("chunk", i, "id", s.id, g.id));
+    let expected_value = |v: f32| if float16 { round_to_f16(v) } else { v };
+    for k in sample_indices(migration.records.len(), full) {
+        let (idx, src) = migration.records[k];
+        let s = &source.chunks[src];
+        let c = &mem.cache;
+        if idx >= c.len() {
+            return Err(
+                format!("record {idx} (chunk id {}) is missing from the store", s.id).into(),
+            );
        }
-        if s.chunk != g.chunk {
+        let id = s.id;
+        if c.chunks[idx] != s.chunk {
            return Err(format!(
-                "chunk[{i}].text mismatch: source {:?}, HDF5 {:?}",
+                "record {idx} (chunk id {id}) text mismatch: source {:?}, store {:?}",
                truncate(&s.chunk),
-                truncate(&g.chunk)
+                truncate(&c.chunks[idx])
            )
            .into());
        }
-        if s.session_id != g.session_id || s.source_channel != g.source_channel || s.tags != g.tags
+        if c.source_channels[idx] != s.source_channel
+            || c.session_ids[idx] != s.session_id
+            || c.tags[idx] != s.tags
        {
-            return Err(format!("chunk[{i}] string field mismatch").into());
+            return Err(format!("record {idx} (chunk id {id}) string field mismatch").into());
        }
-        if s.deleted != g.deleted {
-            return Err(field_err("chunk", i, "deleted", s.deleted, g.deleted));
-        }
-        if s.embedding.len() != g.embedding.len() {
+        if c.timestamps[idx].to_bits() != s.timestamp.to_bits() {
            return Err(format!(
-                "chunk[{i}] embedding length mismatch: {} vs {}",
-                s.embedding.len(),
-                g.embedding.len()
+                "record {idx} (chunk id {id}) timestamp mismatch: source {}, store {}",
+                s.timestamp, c.timestamps[idx]
            )
            .into());
        }
-        for (k, (&a, &b)) in s.embedding.iter().zip(g.embedding.iter()).enumerate() {
-            if (a - b).abs() > emb_abs + emb_rel * a.abs() {
-                return Err(
-                    format!("chunk[{i}].embedding[{k}] mismatch: source {a}, HDF5 {b}").into(),
-                );
+        let deleted = c.tombstones[idx] != 0;
+        if deleted != (s.deleted != 0) {
+            return Err(format!(
+                "record {idx} (chunk id {id}) deleted mismatch: source {}, store {deleted}",
+                s.deleted != 0
+            )
+            .into());
+        }
+        let got = c.embeddings.get(idx).unwrap_or(&[]);
+        if got.len() != s.embedding.len() {
+            return Err(format!(
+                "record {idx} (chunk id {id}) embedding length mismatch: source {}, store {}",
+                s.embedding.len(),
+                got.len()
+            )
+            .into());
+        }
+        for (j, (&a, &b)) in s.embedding.iter().zip(got).enumerate() {
+            let want = expected_value(a);
+            if want.to_bits() != b.to_bits() && !(want.is_nan() && b.is_nan()) {
+                return Err(format!(
+                    "record {idx} (chunk id {id}) embedding[{j}] mismatch: source {a}, \
+                     expected {want}, store {b}"
+                )
+                .into());
            }
        }
        rows_checked += 1;
    }

-    // ---- Other groups (always full — they are small) ----
-    for (i, (s, g)) in source.sessions.iter().zip(got.sessions.iter()).enumerate() {
-        if s.id != g.id
-            || s.start_idx != g.start_idx
-            || s.end_idx != g.end_idx
-            || s.channel != g.channel
-            || s.summary != g.summary
-        {
-            return Err(format!("session[{i}] mismatch").into());
+    // ---- Records tombstoned because their source row was deleted ----
+    for &(idx, src) in &migration.deleted_in_store {
+        let s = &source.chunks[src];
+        let c = &mem.cache;
+        if idx >= c.len() || c.chunks[idx] != s.chunk || c.timestamps[idx] != s.timestamp {
+            return Err(format!("record {idx} (chunk id {}) mismatch or missing", s.id).into());
        }
-        rows_checked += 1;
-    }
-    for (i, (s, g)) in source.entities.iter().zip(got.entities.iter()).enumerate() {
-        if s.id != g.id
-            || s.name != g.name
-            || s.entity_type != g.entity_type
-            || s.embedding_idx != g.embedding_idx
-        {
-            return Err(format!("entity[{i}] mismatch").into());
-        }
-        rows_checked += 1;
-    }
-    for (i, (s, g)) in source
-        .relations
-        .iter()
-        .zip(got.relations.iter())
-        .enumerate()
-    {
-        if s.src != g.src || s.tgt != g.tgt || s.relation != g.relation {
-            return Err(format!("relation[{i}] mismatch").into());
+        if c.tombstones[idx] == 0 {
+            return Err(format!(
+                "record {idx} (chunk id {}) is deleted in the source but active in the store",
+                s.id
+            )
+            .into());
        }
        rows_checked += 1;
    }

+    // ---- Sessions ----
+    let sessions = mem.sessions();
+    for &(at, src) in &migration.sessions {
+        let s = &source.sessions[src];
+        let (Some(e), Some(summary)) = (sessions.entries.get(at), sessions.summaries.get(at))
+        else {
+            return Err(format!("session {:?} is missing from the store", s.id).into());
+        };
+        if e.id != s.id
+            || e.start_idx != s.start_idx.max(0) as u64
+            || e.end_idx != s.end_idx.max(0) as u64
+            || e.channel != s.channel
+            || *summary != s.summary
+            || e.ts != s.timestamp * US_PER_SEC
+        {
+            return Err(format!("session {:?} mismatch", s.id).into());
+        }
+        rows_checked += 1;
+    }
+
+    // ---- Knowledge graph ----
+    let kg = mem.knowledge();
+    for &(id, src) in &migration.entities {
+        let s = &source.entities[src];
+        let Some(e) = kg.get_entity(id) else {
+            return Err(format!(
+                "entity {:?} (id {}) is missing from the store",
+                s.name, s.id
+            )
+            .into());
+        };
+        if e.name != s.name || e.entity_type != s.entity_type || e.embedding_idx != s.embedding_idx
+        {
+            return Err(format!("entity {:?} (id {}) mismatch", s.name, s.id).into());
+        }
+        rows_checked += 1;
+    }
+    for &(at, src) in &migration.relations {
+        let s = &source.relations[src];
+        let r = kg.relations.get(at);
+        let ok = r.is_some_and(|r| {
+            Some(&r.src) == migration.entity_ids.get(&s.src)
+                && Some(&r.tgt) == migration.entity_ids.get(&s.tgt)
+                && r.relation == s.relation
+                && r.weight == s.weight as f32
+                && r.ts == s.timestamp * US_PER_SEC
+        });
+        if !ok {
+            return Err(format!(
+                "relation {} -[{}]-> {} mismatch or missing",
+                s.src, s.relation, s.tgt
+            )
+            .into());
+        }
+        rows_checked += 1;
+    }
+
+    // ---- A migrated record must be findable by search ----
+    let probe = migration
+        .records
+        .iter()
+        .copied()
+        .find(|&(idx, _)| dim > 0 && mem.cache.tombstones[idx] == 0);
+    let search_checked = match probe {
+        None => false,
+        Some((idx, _)) => {
+            let query = mem.cache.embeddings[idx].to_vec();
+            let text = mem.cache.chunks[idx].clone();
+            let hits = mem.search(&query, &text, &SearchOptions::new(10));
+            // A record with the same text is as good a hit: the source may
+            // hold duplicates, and they tie.
+            if !hits.iter().any(|h| h.index == idx || h.chunk == text) {
+                return Err(format!(
+                    "search for migrated record {idx} ({:?}) did not return it",
+                    truncate(&text)
+                )
+                .into());
+            }
+            true
+        }
+    };
+
    Ok(ValidationSummary {
-        chunks: got.chunks.len() as u64,
-        sessions: got.sessions.len() as u64,
-        entities: got.entities.len() as u64,
-        relations: got.relations.len() as u64,
-        embedding_dim: got.embedding_dim as u64,
+        count: mem.count(),
+        active: mem.count_active(),
+        sessions: mem.sessions().len(),
+        entities: mem.knowledge().entities.len(),
+        relations: mem.knowledge().relations.len(),
+        embedding_dim: dim,
+        float16,
        rows_checked,
-        provenance_verified,
+        search_checked,
    })
 }

 fn check_count(kind: &str, got: usize, expected: usize) -> Result<(), BoxErr> {
    if got != expected {
-        return Err(format!("{kind} count mismatch: HDF5 has {got}, source has {expected}").into());
+        return Err(format!("{kind} count mismatch: store has {got}, expected {expected}").into());
    }
    Ok(())
 }

-/// Re-verify the SHA-256 provenance hash of `chunks/text` and
-/// `chunks/embeddings` against their actual stored bytes, catching
-/// post-write corruption that a plain content comparison against the
-/// in-memory source wouldn't (the source is compared against what
-/// `read_hdf5` decoded, not against the raw bytes on disk).
-///
-/// Returns `Ok(true)` only if both datasets exist and both hashes match.
-/// Returns `Ok(false)` (not an error) if a dataset has no provenance
-/// attributes at all (e.g. a file written before this check existed) or
-/// there are zero chunks. Returns an error only on an actual hash mismatch —
-/// that indicates real corruption.
-fn verify_chunk_provenance(path: &str) -> Result<bool, BoxErr> {
-    let file = Hdf5File::open(path)?;
-    let Ok(chunks) = file.group("chunks") else {
-        return Ok(false);
-    };
-    let mut all_present = true;
-    for name in ["text", "embeddings"] {
-        let Ok(ds) = chunks.dataset(name) else {
-            all_present = false;
-            continue;
-        };
-        match ds.verify_provenance()? {
-            VerifyResult::Ok => {}
-            VerifyResult::NoHash => all_present = false,
-            VerifyResult::Mismatch { stored, computed } => {
-                return Err(format!(
-                    "provenance hash mismatch on chunks/{name}: stored {stored}, recomputed {computed} — data may be corrupted"
-                )
-                .into());
-            }
-        }
-    }
-    Ok(all_present)
-}
-
-fn field_err<T: std::fmt::Display>(kind: &str, i: usize, field: &str, s: T, g: T) -> BoxErr {
-    format!("{kind}[{i}].{field} mismatch: source {s}, HDF5 {g}").into()
-}
-
 fn truncate(s: &str) -> String {
    if s.len() <= 40 {
        s.to_string()
@@ -196,7 +270,7 @@ fn truncate(s: &str) -> String {
    }
 }

-/// Indices of chunk rows to content-check. Full = all; otherwise a spread of
+/// Indices of records to content-check. Full = all; otherwise a spread of
 /// representative rows (first/last and evenly-spaced interior samples).
 fn sample_indices(n: usize, full: bool) -> Vec<usize> {
    if n == 0 {