Merge pull request 'clawhdf5-migrate writes real agent stores; knowledge-graph fix; dated benchmark re-run' (#6) from feat/migrate-and-benchmarks into main

Reviewed-on: #6
2026-09-25 14:52:35 +00:00
parent a7920bd4b3 dce5559ff2
commit c0f704c381
17 changed files with 2829 additions and 1314 deletions
@@ -2,20 +2,40 @@
 > Pure Rust. Zero C dependencies. Single file. Fast enough to forget it's there.
-**System:** Intel i7-12650H (10C/16T, 4.7 GHz boost) · 32 GB DDR5 · Linux 6.8.0  
+**System (original run):** Intel i7-12650H (10C/16T, 4.7 GHz boost) · 32 GB DDR5 · Linux 6.8.0  
 **Rust:** 1.96.0-nightly (2026-03-14) · `--release` profile  
 **Date:** 2026-07-01
-> **Traceability note:** the "h5bench-Equivalent I/O Benchmarks" and both
+Sections that carry their own "Measured …" line were measured there instead,
-> "Independent Validation: tank" sections below meet a dated,
+not on the system above. The 2026-09-24 re-run was on tank (AMD Ryzen 7
-> hardware-cited, reproducible standard (explicit date, machine spec, and a
+7800X3D, 8C/16T, Linux 7.0.0-34-generic, rustc 1.98.1) at commit 5c8323c,
-> runnable command per result) — this now covers "LongMemEval Results",
+serially, with no `--warm-up-time`/`--measurement-time` overrides, waiting
-> "SIMD & Parallelism", "Vector Search Latency", and "Comparison to MemX" via
+before each command while the 1-minute load average was above 2.0. The
-> their tank re-runs. The remaining undated sections above (Hybrid Search,
+`bench` targets ran at Criterion's defaults (3 s warm-up, 5 s target
-> Knowledge Graph, Memory Consolidation, Temporal Index, Write Path, Decision
+measurement time, 100 samples). Every `memory_bench` group that was run
-> Gate, Memory Strategy, Multi-Session Benchmark, Memory Footprint,
+(`vector_search_latency`, `hybrid_search`, `knowledge_graph`,
-> Consolidation Efficiency, Ephemeral Tier) do not yet meet that bar — this is
+`consolidation`, `temporal`) and the multi-modal group set
-> a known, tracked documentation gap, not a claim that those numbers are wrong.
+`sample_size(50)`, so those sections took 50 samples, not 100. The 5 s is a
 target: Criterion stretched it where 5 s could not hold the samples it needed
 (for example 8.5 s for `vector_search_latency/bench_cosine_search/100k`,
 15.5 s for the 1K consolidation cycle, 252 s for `tick_session_10k`).
 > **Traceability note:** a section meets the dated, hardware-cited,
 > reproducible standard when it gives an explicit date, the machine, and a
 > runnable command for its results. As of the 2026-09-24 re-run on tank that
 > covers Vector Search Latency, Comparison to MemX, SIMD & Parallelism, Hybrid
 > Search, Knowledge Graph, Memory Consolidation, Temporal Index, Write Path,
 > Decision Gate, Memory Strategy, Summary (derived from those), LongMemEval
 > Results (the BM25 rows), Multi-Session Benchmark, Memory Footprint (on disk
 > and in memory), Consolidation Efficiency, Ephemeral Tier, Multi-modal Search,
 > the Search and Read harnesses, and the "h5bench-Equivalent I/O Benchmarks"
 > and "Independent Validation: tank" sections. What does not yet meet that bar:
 > the LongMemEval rows that need real embeddings (not re-run here, except the
 > dated float16 comparison), the Consolidation Efficiency 100K cycle row and
 > memory-reduction part (the 2026-09-24 run was stopped before it produced
 > them), the int8 side of "Quantising the index copy" (not re-run), and the
 > i7-12650H and macOS M3 Max rows under Cross-Platform Notes. That is a
 > known, tracked documentation gap, not a claim that those numbers are wrong.
 >
 > **Correctness note (2026-08-06).** Being dated and reproducible is necessary but
 > not sufficient — a number can be perfectly reproducible and still measure the
@@ -57,6 +77,23 @@ What remains at 2.43x: the flat vectors (1.0x), the HNSW index's own copy of
 them (1.0x), and text, ids and graph (~0.4x). The index copy is the next
 target — it is what a quantised or borrowed representation would address.
 **Current figures (`f32` index).** Measured 2026-09-24 on tank (AMD Ryzen 7
 7800X3D), commit 5c8323c:
 > **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --footprint --full`
 | N | vectors (raw) | entries MiB | store MiB | indexes MiB | reopened MiB | peak during open MiB | reopened / raw |
 |---:|---:|---:|---:|---:|---:|---:|---:|
 | 1 000 | 1 | 2 | 0 | 2 | 4 | 5 | 2.40x |
 | 10 000 | 15 | 17 | 12 | 32 | 44 | 61 | 3.03x |
 | 100 000 | 146 | 172 | 60 | 266 | 399 | 562 | 2.72x |
 These are the "reopened, f32" figures in the next table, reproduced exactly.
 They are above the "after" column of the table above (44 MiB vs 35 MiB at 10K,
 399 vs 357 MiB at 100K): the 2.43x was measured for 2e7e045, and the store
 already measured 399 MiB (2.72x) when the int8 index landed later the same
 day (c0a9206). The git log does not say what changed in between.
 ### Quantising the index copy (`quantized_index`)
 `MemoryConfig::quantized_index` stores the index's copy as `i8` instead of
@@ -108,6 +145,20 @@ second**, builds **1.8x faster**, and holds a quarter of the vectors. (Compare
 only at equal `ef`: with re-scoring the harness raises `ef` to at least the
 candidate pool, so the `ef = 16` and `ef = 32` rows are not like-for-like.)
 **Re-check, 2026-09-24.** Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D),
 commit 5c8323c. The `f32` memory columns of the first table reproduce exactly
 (see [Memory footprint](#memory-footprint)). Only the `f32` side of the speed
 table was re-run, once, not as alternating medians:
 > **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --full`
 At N = 100 000, `ef = 64`, the `f32` index measured recall@10 0.9945 (same),
 19 001 QPS (was 13 399) and a 2700.8 ms build (was 3197 ms). The QPS figure
 moved by more than 20%; the log does not explain it, and a single run is not
 the same-moment paired comparison the table above is. The int8 columns were
 not re-run, so the 1.63x ratio has not been re-checked against the new `f32`
 figure.
 #### On ARM (Raspberry Pi 5, Cortex-A76)
 `dot_i8` has two aarch64 kernels: `SDOT` for CPUs with the ARMv8.2
@@ -268,6 +319,10 @@ high-water mark alongside the retained figure:
 | 10 000 | 44 | 61 |
 | 100 000 | 399 | 562 |
 Reproduced exactly on 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit
 5c8323c, by `search_harness -- --footprint --full` (table under
 [Memory footprint](#memory-footprint)).
 The peak is set *after* the parse, by the index build, so a buffer allocated
 and freed during the parse never reaches the high-water mark. Holding a
 deliberate extra copy of the file across the whole parse leaves the peak
@@ -359,6 +414,50 @@ contiguous dataset is converted straight from the file bytes (one copy instead
 of two), and the native-endian conversions no longer zero a buffer they are
 about to overwrite.
 ### Current: read harness (2026-09-24)
 Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
 > **Run:** `cargo run --release -p clawhdf5-bench --bin read_harness`
 4096 x 2048 f64 (64 MB per dataset), chunks 256 x 256, file 129 MB
 | layout | read | selected | time ms | MB/s of selection | vs full read |
 |---|---|---:|---:|---:|---:|
 | chunked + deflate | full (first) | 64 MB | 65.6 | 975 | |
 | chunked + deflate | full (repeat) | 64 MB | 63.2 | 1013 | 1.00x |
 | chunked + deflate | 64 x 64 window (1 chunk) | 0.03 MB | 0.18 | 177 | 0.003x |
 | chunked + deflate | 512 x 512 window (4-9 chunks) | 2.00 MB | 4.12 | 485 | 0.063x |
 | chunked + deflate | one row | 0.02 MB | 0.99 | 16 | 0.015x |
 | chunked + deflate | one column | 0.03 MB | 2.02 | 15 | 0.031x |
 | chunked | full (first) | 64 MB | 60.5 | 1057 | |
 | chunked | full (repeat) | 64 MB | 56.7 | 1129 | 1.00x |
 | chunked | 64 x 64 window (1 chunk) | 0.03 MB | 0.10 | 325 | 0.002x |
 | chunked | 512 x 512 window (4-9 chunks) | 2.00 MB | 3.19 | 628 | 0.053x |
 | chunked | one row | 0.02 MB | 0.05 | 284 | 0.001x |
 | chunked | one column | 0.03 MB | 0.49 | 64 | 0.008x |
 | contiguous | full (first) | 64 MB | 30.1 | 2124 | |
 | contiguous | full (repeat) | 64 MB | 26.1 | 2452 | 1.00x |
 | contiguous | 64 x 64 window (1 chunk) | 0.03 MB | 0.08 | 394 | 0.003x |
 | contiguous | 512 x 512 window (4-9 chunks) | 2.00 MB | 3.36 | 595 | 0.112x |
 | contiguous | one row | 0.02 MB | 0.03 | 486 | 0.001x |
 | contiguous | one column | 0.03 MB | 2.61 | 12 | 0.087x |
 Full reads match the "after" column above (chunked + deflate 69 ms then, 63.2
 to 65.6 ms now). The compressed selections are faster than in "After: partial
 reads" (64 x 64 window 0.39 -> 0.18 ms, one row 2.69 -> 0.99 ms, one column
 5.23 -> 2.02 ms) and match the 2026-09-23 figures in
 [Deflate backend](#deflate-backend-zlib-rs-vs-zlib-ng) (0.18, 1.00 and 2.01
 ms), which were taken after the zlib-rs switch and the one-shot codec calls
 described there; this run does not isolate which change accounts for it.
 The uncompressed windows went the other way. Against "After: partial reads",
 the 512 x 512 window is about 60% slower (chunked 1.99 -> 3.19 ms, contiguous
 2.12 -> 3.36 ms) and the chunked 64 x 64 window went from 0.06 to 0.10 ms.
 The rows and columns of the uncompressed layouts are within 20% (chunked
 column 0.45 -> 0.49 ms, contiguous column 2.55 -> 2.61 ms). This run does not
 explain the slower windows.
 ## Search harness baseline (v2.3.0)
 Produced by `cargo run --release -p clawhdf5-bench --bin search_harness -- --full`
@@ -596,68 +695,217 @@ Recall on clustered data is unchanged or slightly better (100K, `ef = 64`:
 0.474 -> 0.444), the cost of batch members not seeing each other while
 planning; batches are capped at 1/16 of the graph and 512 nodes.
 ### Current: search harness (2026-09-24)
 Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, default
 features (so the build is batched and `parallel`, 16 threads). The HNSW tables
 use an `f32` index. The end-to-end table uses `MemoryConfig::new`'s defaults,
 which at this commit are `float16` embeddings and the int8 index; every
 earlier end-to-end table in this section was an `f32` store with an `f32`
 index.
 > **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --full`
 #### HNSW, N = 1000, dim = 384, M = 16, ef_construction = 64, storage = Float32
 build: 14.7 ms (67954 vectors/s) · exact scan: 4403 QPS, p50 225 µs
 | ef | recall@10 | QPS | p50 µs | p99 µs |
 |---:|---:|---:|---:|---:|
 | 16 | 0.9940 | 134065 | 7 | 18 |
 | 32 | 1.0000 | 105134 | 9 | 21 |
 | 64 | 1.0000 | 77541 | 13 | 26 |
 | 128 | 1.0000 | 36938 | 26 | 35 |
 | 256 | 1.0000 | 19249 | 52 | 63 |
 #### HNSW, N = 10000, dim = 384, M = 16, ef_construction = 64, storage = Float32
 build: 135.6 ms (73736 vectors/s) · exact scan: 435 QPS, p50 2295 µs
 | ef | recall@10 | QPS | p50 µs | p99 µs |
 |---:|---:|---:|---:|---:|
 | 16 | 0.9975 | 79614 | 12 | 24 |
 | 32 | 1.0000 | 70556 | 14 | 21 |
 | 64 | 1.0000 | 51313 | 19 | 29 |
 | 128 | 1.0000 | 22846 | 44 | 57 |
 | 256 | 1.0000 | 10267 | 97 | 119 |
 #### HNSW, N = 100000, dim = 384, M = 16, ef_construction = 64, storage = Float32
 build: 2700.8 ms (37027 vectors/s) · exact scan: 43 QPS, p50 23433 µs
 | ef | recall@10 | QPS | p50 µs | p99 µs |
 |---:|---:|---:|---:|---:|
 | 16 | 0.9180 | 20043 | 47 | 104 |
 | 32 | 0.9775 | 21348 | 44 | 104 |
 | 64 | 0.9945 | 19001 | 51 | 101 |
 | 128 | 0.9995 | 13458 | 73 | 137 |
 | 256 | 0.9995 | 5128 | 196 | 282 |
 #### End to end: `HDF5Memory::hybrid_search` (k = 10, weights 0.7 / 0.3)
 | N | ingest ms | cold index build ms | checkpoint ms | open ms | first query after open ms | p50 ms | p99 ms | QPS |
 |---:|---:|---:|---:|---:|---:|---:|---:|---:|
 | 1000 | 15 | 14 | 4.4 | 1.9 | 1.5 | 0.07 | 0.08 | 13869.5 |
 | 10000 | 146 | 126 | 33.5 | 18.3 | 14.8 | 0.49 | 0.50 | 2044.1 |
 | 100000 | 1498 | 1833 | 495.7 | 259.9 | 155.1 | 4.69 | 5.22 | 212.7 |
 Steady-state query latency is where the v2.4.0 work left it (p50 0.07 / 0.49 /
 4.69 ms against 0.07 / 0.49 / 4.65 ms). Figures that moved by more than 20%:
 - **Build.** 14.7 / 135.6 / 2700.8 ms against 34 / 388 / 5.9 s for
  "batched + `parallel`" above. The git log does not explain the difference.
 - **Cold index build** in the end-to-end table: 14 / 126 / 1833 ms, down from
  112 / 1487 / 20285 ms in the "unranked keyword scores" table, which predates
  the batched and parallel build described above.
 - **QPS at 100K, `ef = 64`**: 19 001 against 13 980 in the "unit-vector"
  table, with recall 0.9945 against 0.984; the recall gain is the one the
  batched build reported. The QPS gain is not explained by the log.
 - **Other HNSW QPS cells**, against the same "unit-vector" tables: 10K
  `ef = 16` 79 614 against 54 608 (+46%); 100K `ef = 16` 20 043 against
  15 139 (+32%), `ef = 128` 13 458 against 10 959 (+23%), `ef = 256` 5 128
  against 3 731 (+37%). The other cells, and every 1K cell, are within 20%.
  The log does not explain these either.
 - **Open** in the end-to-end table: 1.9 / 18.3 / 259.9 ms against 1.1 / 13.7
  / 353.1 ms in the "unranked keyword scores" table, slower at 1K and 10K and
  faster at 100K. This run does not isolate the cause.
 - **Checkpoint at 100K**: 495.7 ms against 728.9 ms, and ingest up 11 -> 15,
  104 -> 146 ms at 1K / 10K. This store is `float16` with an int8 index, the
  earlier ones `f32`; the [float16 study](#float16-embedding-storage-memoryconfigfloat16)
  measured float16 checkpoints at 512 ms against 752 ms for `f32` at 100K.
  This run does not isolate the ingest change.
 ## Vector Search Latency
 Brute-force cosine similarity over 384-dimensional embeddings (OpenAI text-embedding-3-small size).
 Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
 > **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(vector_search_1k|simd_cosine_10k|simd_cosine_100k|prenorm_search_10k|ivf_search_10k_nprobe10|ivf_search_100k_nprobe10|ivf_pq_search_100k|rairs_search_10k_nprobe10)$'`
 | Scale | Flat Search | Pre-norm | IVF (nprobe=10) | IVF-PQ | RAIRS |
 |-------|-------------|----------|-----------------|--------|-------|
-| **1K** | 54 µs | 62 µs | — | — | — |
+| **1K** | 47.4 µs | — | — | — | — |
-| **10K** | 753 µs | 706 µs | 27 µs | — | 159 µs |
+| **10K** | 500.5 µs | 316.5 µs | 24.8 µs | — | 124.0 µs |
-| **100K** | 11.4 ms | — | 1.32 ms | 1.19 ms | — |
+| **100K** | 6.58 ms | — | 592 µs | 869 µs | — |
-**Key insight:** At 10K records (typical agent memory), IVF search delivers **27 µs** — that's 26x faster than flat search. Even at 100K records, IVF-PQ keeps search under **1.2 ms**.
+The 1K pre-norm cell is blank because no 1K pre-norm benchmark exists (the
 old 62 µs has no source). `memory_bench`'s own cosine search
 (`cargo bench -p clawhdf5-agent --bench memory_bench -- '^vector_search_latency/'`)
 agrees: 47.6 µs / 505.1 µs / 6.67 ms at 1K / 10K / 100K.
 Most cells moved by more than 20% from the previous table (10K flat 753 µs,
 pre-norm 706 µs, RAIRS 159 µs; 100K flat 11.4 ms, IVF 1.32 ms, IVF-PQ 1.19 ms).
 Those were undated figures from the original i7-12650H run. The 2026-08-05
 tank run below is within 4% of the new numbers in every cell but one (1K flat
 47.8 µs, 10K flat 501 µs, pre-norm 322 µs, IVF 24.8 µs; 100K flat 6.60 ms,
 IVF 608 µs, IVF-PQ 865 µs), so for those cells the difference is the machine.
 The exception is RAIRS: 109 µs on 2026-08-05, 124.0 µs now (+14%); this run
 does not explain that change.
 **Key insight:** at 10K records (typical agent memory), IVF search takes
 **24.8 µs** against 500.5 µs for the flat scan. At 100K, IVF answers in
 **592 µs** and IVF-PQ in 869 µs — both under 1 ms. (RAIRS, 124.0 µs at 10K, is
 slower than plain IVF here.)
 ### Comparison to MemX (arxiv:2603.16171)
 MemX claims end-to-end search under 90ms at 100K records (Rust + libSQL + FTS5).
 Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
 > **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(simd_cosine_100k|ivf_pq_search_100k|bm25_search_10k)$'`
 > **Caveat — not like-for-like.** MemX's `<90 ms` is *end-to-end* search across their
 > full pipeline (dense embeddings + FTS5 + four-factor re-ranking). The clawhdf5
 > figures below are a *single component* — raw vector search latency, excluding
 > embedding, keyword, fusion, and re-ranking stages. A component measured against a
-> full pipeline will always look favourable; the "speedup" column overstates the real
+> full pipeline will always look favourable, so the two columns below cannot be
-> advantage by an unquantified margin and should be read as an order-of-magnitude
+> divided into a speedup: any such ratio would overstate the real advantage by an
-> indication only, not a benchmark result. Matching MemX's measurement boundary is
+> unquantified margin. Read the table as an order-of-magnitude indication only, not a
-> tracked as follow-up work.
+> benchmark result. Matching MemX's measurement boundary is tracked as follow-up work.
-| Metric | MemX (claimed, end-to-end) | ClawhDF5 (component only) | Ratio |
+| Metric | MemX (claimed, end-to-end) | ClawhDF5 (component only) |
-|--------|----------------------------|---------------------------|-------|
+|--------|----------------------------|---------------------------|
-| 100K flat search | <90 ms | 11.4 ms | ~8x |
+| 100K flat search | <90 ms | 6.64 ms |
-| 100K IVF-PQ search | — | 1.19 ms | ~76x |
+| 100K IVF-PQ search | — | 876 µs |
-| Keyword search 10K | 1,100x improvement over unindexed | 583 µs (BM25) | Comparable |
+| Keyword search 10K | 1,100x improvement over unindexed | 205 µs (BM25) |
 The ratio column is gone: dividing an end-to-end claim by a component
 timing produced a number that looked like a result and was not one. The
 previous figures (100K flat 11.4 ms, IVF-PQ 1.19 ms, BM25 10K 583 µs) were
 undated, from the original i7-12650H run. BM25 at 10K also moved on this
 machine: 520 µs on 2026-08-05, 205 µs now; v2.4.0 changed
 `BM25Index::search` in between (bounded-heap top-k, the unused WAND bound
 removed, IDF per query), but this run does not isolate its effect.
 ---
 ## SIMD & Parallelism
-384-dimensional cosine similarity at 10K scale.
+384-dimensional cosine similarity at 10K scale, using the `strategy_*`
 benchmarks, which hold the dataset fixed and vary only `SearchStrategy`
 (the 2026-08-05 re-run below explains why the older table's benchmarks were
 not an apples-to-apples comparison).
-| Strategy | Latency | vs Sequential |
+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. Rayon
-|----------|---------|---------------|
+uses all 16 threads; the 1-minute load average was 4.19 when this command
-| Sequential (scalar) | 1.07 ms | 1.0x |
+finished, most likely from the Rayon benchmarks themselves (not verified).
-| SIMD (auto-vectorized) | 545 µs | **2.0x** |
+
-| Rayon (parallel) | 553 µs | **1.9x** |
+> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(strategy_scalar_10k|strategy_simd_10k|strategy_rayon_10k|adaptive_search_10k|simd_cosine_100k|rayon_cosine_100k)$'`
 | Adaptive (auto-select) | 564 µs | **1.9x** |
 At 100K:
 | Strategy | Latency |
 |----------|---------|
-| SIMD | 13.7 ms |
+| Sequential (scalar), `strategy_scalar_10k` | 501.1 µs |
-| Rayon parallel | 8.3 ms |
+| SIMD, `strategy_simd_10k` | 319.9 µs |
 | Rayon (parallel), `strategy_rayon_10k` | **79.0 µs** |
 | Adaptive (auto-select), `adaptive_search_10k` | 317.9 µs |
 At 100K (no `strategy_*` benchmark exists at this size, so these are the
 `simd_cosine_100k` / `rayon_cosine_100k` benchmarks):
 | Strategy | Latency |
 |----------|---------|
 | SIMD | 6.56 ms |
 | Rayon parallel | 4.63 ms |
 The Rayon 10K figure moved furthest: 323 µs on this machine on 2026-08-05,
 79.0 µs now. The git log does not explain it. The other rows are close to
 that run (scalar 502 µs, SIMD 327 µs, adaptive 339 µs; 100K 6.60 / 4.73 ms). Against the original, undated i7-12650H table (scalar 1.07 ms, SIMD
 545 µs, Rayon 553 µs, adaptive 564 µs; 100K 13.7 / 8.3 ms) every figure is
 lower; that table also used the mislabelled benchmarks. The older names
 measured in the same session, for reference (`cargo bench -p clawhdf5-agent
 --bench bench -- '^(sequential_cosine_10k|simd_cosine_10k|rayon_cosine_10k|adaptive_search_10k|simd_cosine_100k|rayon_cosine_100k)$'`):
 `sequential_cosine_10k` 317.8 µs, `simd_cosine_10k` 505.2 µs,
 `rayon_cosine_10k` 274.3 µs — "sequential" faster than "SIMD", which is the
 mislabelling the 2026-08-05 re-run found.
 ---
 ## Hybrid Search (Vector + BM25)
-1K records, 384-dimensional embeddings with BM25 keyword index.
+1K records, 384-dimensional embeddings with BM25 keyword index. These are the
 flat free functions `hybrid::hybrid_search` / `rrf_hybrid_search` (linear
 scan), not the HNSW-backed `HDF5Memory::hybrid_search`, which is measured in
 the [search harness](#current-search-harness-2026-09-24).
 Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
 > **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^hybrid_search/'`
 > and `cargo bench -p clawhdf5-agent --bench bench -- '^(bm25_search_1k|hybrid_search_10k)$'`
 | Method | Latency | Notes |
 |--------|---------|-------|
-| Weighted fusion | 198 µs | Original min-max normalization |
+| Weighted fusion | 106.8 µs | Min-max normalization |
-| **RRF (k=60)** | **222 µs** | Reciprocal Rank Fusion — better quality, ~12% overhead |
+| **RRF (k=60)** | **130.5 µs** | Reciprocal Rank Fusion |
-| BM25-only 1K | 67 µs | Keyword search alone |
+| BM25-only 1K | 20.4 µs | Keyword search alone |
-| Hybrid 10K | 2.04 ms | Full hybrid at 10K scale |
+| Hybrid 10K | 1.10 ms | Full hybrid at 10K scale |
 Every row moved by more than 20% from the previous table (198 µs, 222 µs,
 67 µs, 2.04 ms), which was undated and from the original i7-12650H run. The
 cause is not isolated. (The old "better quality" note on RRF is dropped: on
 LongMemEval RRF measured worse than the tuned weighted sum; see
 [Fusion method](#fusion-method--weighted-vs-rrf-full-haystack-n500).)
 ---
@@ -665,16 +913,36 @@ At 100K:
 Graph traversal and entity operations.
 Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
 > **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^knowledge_graph/'`
 > and `cargo bench -p clawhdf5-agent --bench bench -- '^alias_resolve_(short|long)_query$'`
 > (the `knowledge_graph/` rows re-run after the adjacency-index fix, same day,
 > load average 0.49 at start)
 | Operation | Scale | Latency |
 |-----------|-------|---------|
-| BFS traversal | 100 entities | 5.4 µs |
+| BFS traversal | 100 entities | 5.23 µs |
-| BFS traversal | 1,000 entities | 24 µs |
+| BFS traversal | 1,000 entities | 23.1 µs |
-| Spreading activation | 100 entities | 16.9 µs |
+| Spreading activation | 100 entities | 10.1 µs |
-| Entity resolution (Levenshtein) | 100 entities | 64 µs |
+| Entity resolution (Levenshtein) | 100 entities | 48.6 µs |
-| Alias resolution (short query) | 100 aliases | 10.4 µs |
+| Alias resolution (short query) | 100 aliases | 7.67 µs |
-| Alias resolution (long query) | 100 aliases | 11.6 µs |
+| Alias resolution (long query) | 100 aliases | 8.18 µs |
-**All graph operations complete in microseconds.** Spreading activation across 100 entities with 5 propagation steps finishes in 17 µs.
+The traversal rows were measured after a fix made during this re-run. The
 first measurement of the day found **BFS 6.5x slower** than the old table
 said: 17.5 µs and 155.1 µs, against 5.4 µs and 24 µs, with spreading
 activation at 22.8 µs against 16.9 µs. The cause was 1efd82c (2026-08-17),
 which built an adjacency index over the whole graph on every traversal — so a
 2-hop BFS paid to index every entity and relation first. The index is now
 cached on `KnowledgeCache` and checked against a fingerprint of the graph on
 each use (one pass over entity ids and relation endpoints, no allocation), so
 any change to the graph, including direct edits of its public `Vec`s, still
 triggers a rebuild. These benches traverse an unchanged graph, which is the
 cached case; the first traversal after a change pays for one build. Entity
 and alias resolution do not use the index (the 49.8 µs first-run figure for
 entity resolution is within noise of the 48.6 µs here). The old figures were
 undated, from the original i7-12650H run.
 ---
@@ -682,13 +950,22 @@ Graph traversal and entity operations.
 Hippocampal-inspired tiered memory management.
 Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
 > **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^consolidation/'`
 | Operation | Scale | Latency |
 |-----------|-------|---------|
-| Consolidation cycle | 100 records | 15 µs |
+| Consolidation cycle | 100 records | 8.08 µs |
-| Consolidation cycle | 1,000 records | 164 µs |
+| Consolidation cycle | 1,000 records | 115.2 µs |
-| Importance scoring | 100 records | 25 µs |
+| Importance scoring | 100 records | 31.0 µs |
-A full consolidation pass over 1,000 memories (eviction + promotion across Working → Episodic → Semantic) completes in **164 µs**. This can run on every memory write without perceptible latency.
+A full consolidation pass over 1,000 memories (eviction + promotion across Working → Episodic → Semantic) completes in **115 µs**. This can run on every memory write without perceptible latency.
 The cycle figures were 15 µs and 164 µs, and importance scoring 25 µs, all
 undated from the original i7-12650H run. Eviction's membership check became a
 `HashSet` lookup (603fcf8) and `add_memory` stopped cloning Working-tier
 records (d787ac0), both on 2026-08-17; this run does not isolate their effect.
 ---
@@ -696,12 +973,17 @@ A full consolidation pass over 1,000 memories (eviction + promotion across Worki
 Sorted timestamp index with binary search.
 Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
 > **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^temporal/'`
 | Operation | Scale | Latency |
 |-----------|-------|---------|
-| Range query | 10K timestamps | **716 ns** |
+| Range query | 10K timestamps | **622 ns** |
-| Batch insert | 10K timestamps | 4.69 ms |
+| Batch insert | 10K timestamps | 3.22 ms |
-Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K records: **716 nanoseconds.**
+Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K records: **622 nanoseconds.**
 Batch insert was 4.69 ms in the previous, undated i7-12650H figure.
 ---
@@ -709,16 +991,50 @@ Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K
 HDF5 persistence with optional Write-Ahead Log.
 Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. The stores
 use `MemoryConfig::new`'s defaults at this commit: `float16` embeddings and
 the int8 index.
 > **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(save_without_wal_single|save_with_wal_single|save_batch_100|save_batch_1000|save_wal_1k_existing|wal_flush_100_entries|tick_session_1k|tick_session_10k)$'`
 | Operation | Latency | Notes |
 |-----------|---------|-------|
-| Single save (no WAL) | 61 µs | Direct HDF5 write (owned-Vec IO path) |
+| Single save (no WAL) | 88.3 µs | Direct HDF5 write (owned-Vec IO path) |
-| Single save (with WAL) | 18 µs | WAL group-commit append; HDF5 write batched at flush |
+| Single save (with WAL) | 26.1 µs | WAL group-commit append; HDF5 write batched at flush |
-| Batch 100 | 723 µs | 7.2 µs per record |
+| Batch 100 | 1.08 ms | |
-| Batch 1,000 | 6.17 ms | 6.2 µs per record |
+| Batch 1,000 | 31.7 ms | |
-| WAL save (1K existing) | 539 µs | Incremental append |
+| WAL save (1K existing) | 282.8 µs | Incremental append |
-| WAL flush 100 entries | 787 µs | Merge WAL → HDF5 |
+| WAL flush 100 entries | 530.1 µs | Merge WAL → HDF5 |
-| Session tick 1K | 5.76 ms | Full session maintenance |
+| Session tick 1K | 3.83 ms | Full session maintenance |
-| Session tick 10K | 89.8 ms | Background operation |
+| Session tick 10K | 35.5 ms | Background operation |
 Almost every row moved by more than 20% from the previous table, in both
 directions. Slower: single save 61 -> 88.3 µs (no WAL) and 18 -> 26.1 µs
 (WAL), batch 100 723 µs -> 1.08 ms, and **batch 1,000 6.17 -> 31.7 ms**.
 Faster: WAL save into 1K existing 539 -> 282.8 µs, WAL flush 787 -> 530.1 µs,
 session tick 5.76 -> 3.83 ms (1K) and 89.8 -> 35.5 ms (10K). The two single-save
 figures were last updated on 2026-07-01 and the rest are undated, all on the
 i7-12650H with `f32` stores.
 **Isolated afterwards, same machine, same day** (`--warm-up-time 1
 --measurement-time 3`, the store settings switched through the bench's
 config):
 | | f16 + int8 (default) | f32 + int8 | f16 + f32 index | f32 + f32 index | v2.3.0 (f32) |
 |---|---:|---:|---:|---:|---:|
 | Single save (WAL) | 26.1 µs | 24.3 µs | 26.1 µs | 24.4 µs | 24.3 µs |
 | Single save (no WAL) | 89.0 µs | 87.3 µs | 88.8 µs | 87.4 µs | 82.6 µs |
 | Batch 100 | 1.09 ms | 0.90 ms | 1.09 ms | 0.90 ms | 0.87 ms |
 | Batch 1,000 | 31.5 ms | 31.1 ms | 31.5 ms | 31.6 ms | 30.3 ms |
 The int8 index costs nothing on the write path. `float16` costs ~2 µs per
 saved record — the rounding — which is the whole difference at 100 records
 and within noise at 1,000. Built at the v2.3.0 tag (v2.2.0's benches do not
 compile) and run on this machine, the write path measures the same as today,
 so nothing regressed since then; the old 18 µs and 6.17 ms figures cannot be
 reproduced on this hardware and came from an undated run on another machine.
 `save_batch` scales linearly (about 12 µs per record from 500 to 8 000
 records); the bench's figure also includes dropping the store.
 ---
@@ -726,12 +1042,16 @@ HDF5 persistence with optional Write-Ahead Log.
 Trivial/non-trivial classification for memory write filtering.
 Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
 > **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^gate_'`
 | Check | Latency |
 |-------|---------|
-| Trivial skip ("ok", "yes") | 61 ns |
+| Trivial skip ("ok", "yes") | 57.6 ns |
-| Short phrase skip | 86 ns |
+| Short phrase skip | 83.0 ns |
-| Non-trivial pass | 705 ns |
+| Non-trivial pass | 570.5 ns |
-| Ratio check | 488 ns |
+| Ratio check | 395.2 ns |
 **Sub-microsecond filtering.** The gate decides whether to save a memory in under 1 µs.
@@ -741,27 +1061,70 @@ Trivial/non-trivial classification for memory write filtering.
 End-to-end strategy evaluation including embedding operations.
 Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
 > **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^strategy_(save_every|semantic_shift)'`
 | Strategy | Condition | Latency |
 |----------|-----------|---------|
-| SaveEveryExchange (substantive) | Saves | 923 ns |
+| SaveEveryExchange (substantive) | Saves | 732.5 ns |
-| SaveEveryExchange (trivial) | Skips | 67 ns |
+| SaveEveryExchange (trivial) | Skips | 58.3 ns |
-| SaveOnSemanticShift (empty store) | Saves | 941 ns |
+| SaveOnSemanticShift (empty store) | Saves | 734.9 ns |
 The two "Saves" rows were 923 ns and 941 ns in the previous table, undated
 and from the original i7-12650H run.
 ---
 ## Multi-modal Search
 `clawhdf5_agent::multimodal::MultiModalStore`. Each of N records carries two
 384-dim embeddings: a text embedding of its caption and one of its primary
 modality, cycling image / audio / video. `search_cross_modal` scores every
 embedding of every record (2N vectors) and keeps each record's best;
 `search_by_modality(Image, …)` scores only image embeddings (a third of the
 records). Both are exact linear scans followed by a full sort; k = 10. Data is
 from a fixed-seed generator, so every run sees the same corpus.
 Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c (with the
 new, uncommitted `crates/clawhdf5-agent/benches/multimodal_bench.rs` in the
 working tree). Criterion, 50 samples.
 > **Run:** `cargo bench -p clawhdf5-agent --bench multimodal_bench`
 | Search | 1K records | 10K records |
 |--------|-----------:|------------:|
 | Cross-modal (all modalities, 2N vectors) | 842.0 µs | 8.44 ms |
 | One modality (image, ~N/3 vectors) | 151.2 µs | 1.54 ms |
 Both scale linearly with N. Cross-modal search is the slow path: at 1K
 records it takes 842.0 µs, where the flat single-vector cosine scan in
 [Vector Search Latency](#vector-search-latency) takes 47.4 µs over 1K vectors,
 and it has no index — at 10K it is 8.44 ms, slower than a 100K flat scan
 (6.58 ms). No earlier figure exists for this section.
 ---
 ## Summary
 Derived from the 2026-09-24 tables above (tank, AMD Ryzen 7 7800X3D, commit
 5c8323c); no command of its own.
 | Capability | Typical Latency | Scale |
 |------------|----------------|-------|
-| **Full memory search** | <1 ms | 10K records |
+| **Full memory search** (`HDF5Memory::hybrid_search`, p50) | 0.49 ms | 10K records |
-| **Hybrid vector+keyword** | <200 µs | 1K records |
+| **Hybrid vector+keyword** (flat, weighted) | 106.8 µs | 1K records |
-| **Knowledge graph query** | <25 µs | 1K entities |
+| **Knowledge graph query** (BFS) | 23.1 µs | 1K entities |
-| **Temporal range query** | <1 µs | 10K timestamps |
+| **Temporal range query** | 622 ns | 10K timestamps |
-| **Memory write** | <20 µs | Per record (WAL group-commit append) |
+| **Memory write** | 26.1 µs | Per record (WAL group-commit append) |
-| **Consolidation cycle** | <165 µs | 1K records |
+| **Consolidation cycle** | 115.2 µs | 1K records |
-| **Importance gate** | <1 µs | Per record |
+| **Importance gate** | 57.6–570.5 ns | Per record |
-**The entire memory pipeline — search, retrieve, re-rank, filter — runs in single-digit milliseconds at agent-typical scales. Fast enough that memory becomes invisible infrastructure.**
+The previous summary said "<25 µs" for a knowledge-graph query and "<20 µs"
 for a memory write. The first holds again after the adjacency-index fix
 (23.1 µs); the second does not (26.1 µs).
 **At agent-typical scales a full `HDF5Memory::hybrid_search` (vector + BM25 + fusion) over 10K records takes 0.49 ms at p50, and graph traversal, temporal queries, consolidation and the write gate each take well under a millisecond. At 100K records the hybrid search takes 4.69 ms.**
 ---
@@ -796,9 +1159,21 @@ _Latency benchmarks generated with Criterion.rs (50-100 samples per benchmark).
 **Mode:** BM25-only retrieval — zero embeddings, `vector_weight=0.0`, `keyword_weight=1.0`
-> **Run:** `cargo run --release --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s_cleaned.json`
+> **Run:** `cargo run --release -p clawhdf5-bench --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s.json`
-> (~70 s for all 500 questions on the tank reference machine). Omit the path for the
+> Omit the path for the oracle variant; add `--limit N` for an evenly-strided
-> oracle variant; add `--limit N` for an evenly-strided subsample.
+> subsample. Without the `embeddings` feature this runs two modes, BM25 only
 > and BM25 stemmed.
 The BM25-only and BM25-stemmed figures in this section were measured again on
 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, with the command
 above. Every figure of those two modes that this document publishes was
 reproduced exactly: BM25 Hit@1/5/10 and MRR at turn and session level, the
 BM25 per-type session Hit@1 values quoted in prose below, and the stemmed
 turn-level row plus its session Hit@1 in the tokenizer table. The run also
 produced figures this document does not publish (stemmed session Hit@5,
 Hit@10 and MRR, and per-type Hit@5/Hit@10/MRR for both modes), so there was
 nothing to compare them with. Rows that need real embeddings (vector-only,
 hybrid, RRF, re-ranking, the weight sweep) were not re-run.
 ### Full haystack — `longmemeval_s`, n=500 (the number to cite)
@@ -1058,14 +1433,29 @@ extraction over the haystack, which this harness does not do.
 ### Search Latency (LongMemEval, n=500 queries)
-| Metric | Latency |
+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, full
-|--------|---------|
+`longmemeval_s` haystack (47.7 sessions / 494 turns per question).
 | avg | 1,004 µs |
 | p50 | 1,017 µs |
 | p95 | 2,031 µs |
 | p99 | 2,912 µs |
-Sub-millisecond median search across variable-length chat histories.
+> **Run:** `cargo run --release -p clawhdf5-bench --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s.json`
 | Metric | BM25 only | BM25 stemmed |
 |--------|-----------|--------------|
 | avg | 11,034.7 µs | 11,096.2 µs |
 | p50 | 10,943.5 µs | 10,996.5 µs |
 | p95 | 11,749.6 µs | 12,018.8 µs |
 | p99 | 12,644.1 µs | 12,784.7 µs |
 About 11 ms per query. Each question gets its own store, and the timed call is
 the first `hybrid_search_with` on that store. The first query on a store
 builds its indexes (the "cold index build" column of the search harness), so
 this is a first-query figure, not steady-state search; for that
 see the [search harness](#current-search-harness-2026-09-24).
 The previous table (avg 1,004 µs, p50 1,017 µs) said "sub-millisecond median
 search", which no longer holds. That figure was undated and predates the
 full-haystack harness (added 2026-08-07, 7d6e269), so it was measured on the
 much smaller `longmemeval_oracle` corpus; on that corpus tank measured avg
 2,431 µs on 2026-08-05 (below).
 ---
@@ -1075,23 +1465,34 @@ Sub-millisecond median search across variable-length chat histories.
 **Topics:** Personal info, food preferences, music, travel, work/schedule, hobbies
 **Queries:** 35 questions across 4 types
-> **Run:** `cargo run --release --bin memory_arena`
+> **Run:** `cargo run --release -p clawhdf5-bench --bin memory_arena`
 Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
 ### Results by Query Type
 | Query Type | N | Hit@1 | Hit@5 | Hit@10 | MRR | Avg Latency |
 |------------|---|-------|-------|--------|-----|-------------|
-| single-session | 25 | 40.0% | 92.0% | 100.0% | 0.5788 | 7,853 µs |
+| single-session | 25 | 40.0% | 92.0% | 100.0% | 0.5755 | 248.4 µs |
-| multi-session | 5 | 40.0% | 60.0% | 80.0% | 0.5333 | 7,887 µs |
+| multi-session | 5 | 40.0% | 60.0% | 80.0% | 0.5333 | 24.4 µs |
-| temporal | 3 | 33.3% | 66.7% | 66.7% | 0.5000 | 7,899 µs |
+| temporal | 3 | 33.3% | 66.7% | 66.7% | 0.5000 | 28.0 µs |
-| knowledge-update | 2 | 0.0% | 50.0% | 50.0% | 0.2500 | 7,899 µs |
+| knowledge-update | 2 | 0.0% | 50.0% | 50.0% | 0.2500 | 38.7 µs |
-| **OVERALL** | **35** | **37.1%** | **82.9%** | **91.4%** | **0.5468** | **7,870 µs** |
+| **OVERALL** | **35** | **37.1%** | **82.9%** | **91.4%** | **0.5444** | p50 **30.1 µs**, p95 49.8 µs |
 **Key findings:**
 - Hit@10 of 91.4% across all query types with BM25-only (no embeddings)
 - Single-session recall strongest at 100% Hit@10
 - Knowledge-update hardest (requires temporal disambiguation) — would improve significantly with vector similarity
- Latency dominated by BM25 index build over 999 turns (~7.9 ms)
+- Search takes about 30 µs at p50 over the 999 turns. All 35 queries run
  against one store, and single-session's higher average (248.4 µs) includes
  that store's first query, which builds its indexes.
 The previous table put every query at ~7.9 ms ("latency dominated by BM25
 index build over 999 turns"). That was before v2.4.0, when
 `HDF5Memory::hybrid_search` rebuilt the BM25 index from scratch and rewrote
 the store on every query; it now keeps the index for the life of the store.
 The MRR moved slightly (single-session 0.5788 -> 0.5755, overall 0.5468 ->
 0.5444); the Hit@k figures are unchanged.
 ---
@@ -1099,46 +1500,89 @@ Sub-millisecond median search across variable-length chat histories.
 HDF5 file size at various record counts — 384-dimensional embeddings, 200-char text.
-> **Run:** `cargo run --release --bin footprint_bench`
+Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
-### Uncompressed (no WAL)
+> **Run:** `cargo run --release -p clawhdf5-bench --bin footprint_bench`
-| Records | File Size | Raw Data | Bytes/Record | Throughput |
+**These are `float16` stores.** `footprint_bench` builds its stores with
-|---------|-----------|----------|--------------|------------|
+`MemoryConfig::new`, which defaults to `float16` embeddings since 5c8323c,
-| 100 | 176.4 KB | 169.5 KB | 1.8 KB | 100,000 rec/s |
+although the binary still prints "Embedding: 384-dim f32". Its "Raw Data"
-| 1K | 1.7 MB | 1.7 MB | 1.8 KB | 109,643 rec/s |
+column, and so the ratios, count embeddings as `f32` (1,536 bytes per record)
-| 10K | 17.0 MB | 16.6 MB | 1.7 KB | 118,100 rec/s |
+plus text. The synthetic embeddings take only 1,000 distinct values, and each
-| 50K | 85.0 MB | 82.8 MB | 1.7 KB | 110,723 rec/s |
+record's text is fixed by `record_idx % 40` (a 40-word bank walked from that
-| 100K | 169.8 MB | 165.6 MB | 1.7 KB | 111,422 rec/s |
+start), so a store holds only 40 distinct texts. Both compress far better than
 real data would; read the gzip figures as an upper bound, and see the next
 paragraph for why the other tables are optimistic too. "MB" and "KB" are the
 binary's units (powers of 1024).
-**1.7 KB per record** — HDF5 overhead is near-zero. Ingestion throughput exceeds **100K records/sec**.
+**No table here is uncompressed.** The store always deflates its string
 datasets: `write_string_dataset` applies deflate level 6 to any string dataset
 of 4 KiB or more, whatever `MemoryConfig::compression` says
 (`crates/clawhdf5-agent/src/schema.rs`, `STRING_COMPRESS_THRESHOLD`, added in
 4fa7e89 on 2026-06-03). With only 40 distinct texts, the chunk text costs
 almost nothing on disk: at 10K records, 50-char text takes 804 B per record
 and 1000-char text 944 B, so 950 more characters cost only 140 B. The
 `float16` embeddings alone are 768 B per record, and 200 bytes of raw text
 would bring that to about 968 B before any other field or overhead; the
 measured 803–829 B is below that only because the text is compressed away.
 Real text will not deflate like this, so expect more than 820 B per record
 with real data. `MemoryConfig::compression` decides only whether the
 embeddings are compressed; the gzip table below is that setting turned on.
 ### `compression` off (no WAL)
 | Records | File Size | Raw Data (f32) | Bytes/Record | Throughput |
 |---------|-----------|----------------|--------------|------------|
 | 100 | 90.0 KB | 169.5 KB | 921 B | 75,914 rec/s |
 | 1K | 810.4 KB | 1.7 MB | 829 B | 72,266 rec/s |
 | 10K | 7.8 MB | 16.6 MB | 820 B | 74,509 rec/s |
 | 50K | 38.3 MB | 82.8 MB | 803 B | 72,829 rec/s |
 | 100K | 76.7 MB | 165.6 MB | 803 B | 71,102 rec/s |
 **About 820 B per record** at 10K, for this synthetic corpus (see above: the
 text is deflated regardless and has only 40 distinct values, so real data
 takes more). Ingestion runs at 71,102–75,914 records/sec.
 ### With Gzip Compression (level 6)
-| Records | Compressed | Ratio | Bytes/Record |
+| Records | Compressed | Ratio vs f32 raw | Bytes/Record |
-|---------|------------|-------|--------------|
+|---------|------------|------------------|--------------|
-| 100 | 31.5 KB | 5.37x | 323 B |
+| 100 | 17.2 KB | 9.85x | 176 B |
-| 1K | 277.1 KB | 6.12x | 283 B |
+| 1K | 56.4 KB | 30.06x | 57 B |
-| 10K | 2.7 MB | 6.17x | 281 B |
+| 10K | 471.3 KB | 35.97x | 48 B |
-| 50K | 13.4 MB | 6.17x | 281 B |
+| 50K | 2.3 MB | 36.67x | 47 B |
-| 100K | 26.9 MB | 6.15x | 282 B |
+| 100K | 4.5 MB | 36.73x | 47 B |
-**6.2x compression ratio** — 100K agent memories in 27 MB compressed.
+### Text Length Comparison (10K records, `compression` off)
 ### Text Length Comparison (10K records, no compression)
 | Text Length | File Size | Bytes/Record | Throughput |
 |-------------|-----------|--------------|------------|
-| short (50 chars) | 15.6 MB | 1.6 KB | 177,925 rec/s |
+| short (50 chars) | 7.7 MB | 804 B | 97,615 rec/s |
-| medium (200 chars) | 17.0 MB | 1.7 KB | 172,152 rec/s |
+| medium (200 chars) | 7.8 MB | 820 B | 89,597 rec/s |
-| long (1000 chars) | 24.6 MB | 2.5 KB | 157,807 rec/s |
+| long (1000 chars) | 9.0 MB | 944 B | 51,462 rec/s |
 ### WAL Overhead (1K records)
 | Mode | File Size | Ingest Time | Overhead |
 |------|-----------|-------------|----------|
-| No WAL | 1.7 MB | 5.7 ms | — |
+| No WAL | 810.4 KB | 10.8 ms | — |
-| With WAL | 1.7 MB + 9 B WAL | 5.3 ms | ±8% (negligible) |
+| With WAL | 810.4 KB + 9 B WAL | 10.6 ms | −2% (negligible) |
 **Compared with the previous tables.** Those were undated `f32` stores from the
 original i7-12650H run: 1.7 KB per record (169.8 MB at 100K), gzip 6.2x
 (26.9 MB at 100K), and more than 100,000 records/sec. Two changes separate
 those tables from these, not one. `float16` embeddings save 768 B per record.
 The old tables (recorded by 2026-05-14) also predate 4fa7e89 (2026-06-03),
 which began deflating string datasets of 4 KiB or more, so their text was
 stored raw; with only 40 distinct texts, deflate now removes most of it.
 An `f32` embedding plus 200 B of text is 1,536 + 200 = 1,736 B; `float16`
 alone would bring that to about 968 B per record, not the measured 803–829 B. This run does not measure how the saving splits between the two
 beyond that, nor explain why the gzip ratio rose so much further, nor
 separate the lower throughput (and the 1K ingest going from 5.7 to 10.8 ms)
 from the change of machine and store configuration. The
 `f32` on-disk figures were not re-measured; the dated
 [float16 study](#float16-embedding-storage-memoryconfigfloat16) compares the
 two directly (100K × 384: 154.0 MiB `f32`, 80.8 MiB `float16`).
 ---
@@ -1146,7 +1590,12 @@ HDF5 file size at various record counts — 384-dimensional embeddings, 200-char
 Hippocampal-inspired memory consolidation improves both retrieval quality and search speed.
-> **Run:** `cargo run --release --bin consolidation_efficiency`
+> **Run:** `cargo run --release -p clawhdf5-bench --bin consolidation_efficiency`
 Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. The run
 was stopped (after about 19 minutes on one core) while still computing a
 100K row of the cycle-time table, so that row and the binary's memory-reduction
 part were not produced; neither has ever been published here.
 ### Retrieval Quality Before vs. After Consolidation
@@ -1159,17 +1608,22 @@ Hippocampal-inspired memory consolidation improves both retrieval quality and se
 | Hit@5 | 100.0% | 100.0% | — |
 | Hit@10 | 100.0% | 100.0% | — |
 | MRR | 1.0000 | 1.0000 | — |
-| Search latency | 2,752 µs | 312 µs | **8.8x faster** |
+| Search latency (avg) | 2,223.7 µs | 239.2 µs | **9.3x faster** |
-Signal records survive consolidation because they are accessed 15+ times, giving them high decay scores. 900 noise records evicted, search speeds up 8.8x, and **zero quality loss** — perfect recall maintained.
+Signal records survive consolidation because they are accessed 15+ times, giving them high decay scores. 900 noise records evicted, search speeds up 9.3x, and **zero quality loss** — perfect recall maintained. The consolidation cycle itself took 0.13 ms and promoted 2 records.
 ### Consolidation Cycle Time
 | Records | Cycle Time | Evictions | Promotions |
 |---------|-----------|-----------|------------|
-| 100 | 21 µs | 100 | 0 |
+| 100 | 17 µs | 100 | 0 |
-| 1K | 345 µs | 1,000 | 0 |
+| 1K | 189 µs | 1,000 | 0 |
-| 10K | 17.3 ms | 10,000 | 0 |
+| 10K | 2.16 ms | 10,000 | 0 |
 The 1K and 10K cycles were 345 µs and 17.3 ms in the previous, undated table
 (before: 2,752 µs / after: 312 µs for search). Eviction's membership check
 changed from a linear scan to a `HashSet` in 603fcf8 (2026-08-17), after those
 figures were recorded; this run does not isolate its effect.
 ---
@@ -1178,34 +1632,44 @@ Signal records survive consolidation because they are accessed 15+ times, giving
 In-memory key-value store with TTL, capacity eviction, and embedding search.
 No network hop, no serialization — direct HashMap operations.
-> **Run:** `cargo run --release --bin ephemeral_perf`
+> **Run:** `cargo run --release -p clawhdf5-bench --bin ephemeral_perf`
 Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
 ### Latency Comparison
-| Operation | clawhdf5 Ephemeral | Redis (single-node)¹ | Speedup |
+| Operation | clawhdf5 Ephemeral | Redis (single-node)¹ | Ratio¹ |
 |-----------|-------------------|---------------------|---------|
-| SET | **356 ns/op** | ~25,000 ns/op | **70x** |
+| SET | **366 ns/op** | ~25,000 ns/op (not measured) | 68.3x |
-| GET (hit) | **179 ns/op** | ~25,000 ns/op | **140x** |
+| GET (hit) | **115 ns/op** | ~25,000 ns/op (not measured) | 216.7x |
-| GET (miss) | **62 ns/op** | ~25,000 ns/op | **403x** |
+| GET (miss) | **57 ns/op** | — | — |
-| DELETE | **124 ns/op** | ~25,000 ns/op | **202x** |
+| DELETE | **99 ns/op** | — | — |
-| SET+embedding | **268 ns/op** | N/A | — |
+| SET+embedding | **236 ns/op** | N/A | — |
-> ¹ Redis latency includes network round-trip (loopback). clawhdf5 ephemeral is in-process — no network.
+> ¹ **The Redis figure is not a measurement.** It is a constant hard-coded in
 > `ephemeral_perf` as a "typical single-node" latency, and the binary computes
 > the two ratios from it; only the clawhdf5 column was measured. A typical
 > Redis figure includes a network round-trip, which the in-process ephemeral
 > tier does not have, so the ratios compare different things.
 ### Throughput
 | Operation | ops/sec |
 |-----------|---------|
-| SET | 2,810,649 |
+| SET | 2,732,568 |
-| GET | 5,584,684 |
+| GET (hit) | 8,666,607 |
-| DELETE | 8,093,731 |
+| GET (miss) | 17,554,663 |
-| SET+EMB (384d) | 3,725,877 |
+| DELETE | 10,109,895 |
 | SET+EMB (384d) | 4,235,766 |
 ### Embedding Search (ephemeral tier)
 | Scale | Latency |
 |-------|---------|
-| 10K entries @ 384d | 2.9 ms/query |
+| 10K entries @ 384d | 2.85 ms/query |
 GET (hit) moved from 179 to 115 ns/op and DELETE from 124 to 99 ns/op (GET
 throughput 5,584,684 -> 8,666,607 ops/sec); the previous figures were undated.
 ---
@@ -1263,10 +1727,16 @@ Measured 2026-08-07 on tank (Ryzen 7 7800X3D, 246 MB dataset in page cache).
 ### Measured Platforms
 The tank row was measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit
 5c8323c, with the single benchmark rather than the whole script:
 > **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^ivf_search_10k_nprobe10$'`
 | Platform | CPU | 10K IVF Search | Notes |
 |----------|-----|----------------|-------|
-| Linux x86_64 | Intel i7-12650H (10C, 4.7 GHz) | 27 µs | Primary CI target |
+| Linux x86_64 | AMD Ryzen 7 7800X3D (8C/16T), tank | 24.8 µs | Measured 2026-09-24 |
-| macOS aarch64 | Apple M3 Max (14C) | ~18 µs | ~33% faster via NEON SIMD |
+| Linux x86_64 | Intel i7-12650H (10C, 4.7 GHz) | 27 µs | Undated, original run |
 | macOS aarch64 | Apple M3 Max (14C) | ~18 µs | Undated; ~33% faster than the i7-12650H row. Not reproducible on the hardware available for the 2026-09-24 re-run |
 ### Reproducibility
@@ -1610,6 +2080,11 @@ vs-libhdf5 validation above) to give both sections the dated, hardware-cited,
 reproducible citation the top-of-file traceability note flags them as
 missing.
 > The sections this re-run checked were themselves re-measured on tank on
 > 2026-09-24 and now show those figures; the i7-12650H figures this section
 > compares against ("above", "at the top of this file") are quoted in those
 > sections' notes. Kept as the 2026-08-05 record.
 ### LongMemEval Results (reproduction)
 ```bash
@@ -3,6 +3,17 @@
 ## Unreleased
 ### Upgrade Notes
 - **`clawhdf5-migrate` now writes a real agent store.** Its output used to be
  a layout of its own (`/chunks`, `/sessions`, `/entities`, `/relations`, no
  `/meta`) that `HDF5Memory::open` rejected, so a migrated file could not be
  used as agent memory. Files it wrote before this release are not agent
  stores; re-run the migration. Also: embeddings default to `float16` like
  any new store (`--f32` opts out; `--float16` is a hidden no-op); a row with
  the wrong embedding length is an error instead of being truncated or
  padded; `--incremental` now matches rows by content against an existing
  store and follows the source's deleted flags; a source with no memory rows
  needs `--embedding-dim`. The per-dataset SHA-256 provenance attributes of
  the old layout are gone (the agent schema has no place for them).
 - **Files written by clawhdf5 now open in h5py and libhdf5.** Every `f32`
  dataset we wrote — including every agent store's embeddings — was refused
  with "sign bit position out of bounds", and every empty dataset with
@@ -46,6 +57,24 @@
  `quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
  out. The CLI's `--quantized-index` is still accepted but is now a no-op.
 ### Migration
 - `clawhdf5-migrate`: writes through the agent's own API (`HDF5Memory::create`
  / `open`, `save_batch`, the session cache and knowledge graph), so there is
  no second copy of the schema. Sessions and entities/relations carry over;
  deleted rows become deleted records (or are left out with
  `--skip-deleted`). Every source row is checked before the output is created,
  so a source that cannot be migrated leaves an existing store untouched.
  Validation reads the result back with `HDF5Memory::open_read_only`, compares
  every field (embeddings bit for bit — `round_to_f16` of the source for a
  `float16` store) and checks that a migrated record is found by search. The
  `half`-based conversion is gone; `clawhdf5_format::float16` is the only one.
  42 tests, including h5py opening a migrated store; an adversarial review's
  two blocker and four major findings are fixed with regression tests.
 - `clawhdf5-agent`: `HDF5Memory::sessions()` / `sessions_mut()`,
  `HDF5Memory::delete_batch(&[usize])` (one save, all-or-nothing, never
  auto-compacts), `SessionCache::add_at`, and `SessionCache` / `SessionEntry`
  re-exported from the crate root.
 ### Search
 - `clawhdf5-agent`: **`HDF5Memory::search` with `SearchOptions`** — source
  filtering, re-ranking and confidence rejection in the store's own search
@@ -68,6 +97,20 @@
  activation of the `k` results it returns, not of the whole `3k` candidate
  pool it re-ranks.
 ### Benchmarks
 - Every undated or pre-September section of `BENCHMARKS.md` re-run on one
  machine on one day (tank, 2026-09-24, commit 5c8323c), with the command for
  each and every number traced back to the raw output by a separate check.
  Where a figure moved, the section says so. Two apparent regressions were
  isolated rather than published: knowledge-graph traversal (a real bug,
  fixed above) and the write path, which measures the same at v2.3.0 on this
  machine — the old 18 µs / 6.17 ms figures came from an undated run on other
  hardware; `float16` adds ~2 µs per save and the int8 index nothing.
 - New `multimodal_bench`: cross-modal search at 1K and 10K records, which the
  README claimed but nothing measured.
 - `footprint_bench` reports whether it built `float16` or `f32` stores and
  takes `--f32`; it had kept printing "f32" after the default changed.
 ### Interop
 - `clawhdf5-format`: **every `f32` dataset was unreadable by h5py and
  libhdf5.** The float datatype encoder hard-coded the sign bit's position to
@@ -146,6 +189,14 @@
  knew to ask; it now only ever switches the default off.
 ### Performance
 - `clawhdf5-agent`: **knowledge-graph traversal was 6.5x slower than it
  should be.** `bfs_neighbors` and `spreading_activation` built an adjacency
  index over the whole graph on every call (1efd82c), so a 2-hop BFS over 1K
  entities took 155 µs. The index is now cached on `KnowledgeCache` and
  checked against a fingerprint of the graph on each use — one pass over
  entity ids and relation endpoints, no allocation — so any change, including
  direct edits of its public `Vec`s, still rebuilds it (tested). BFS over 1K
  entities: 155.1 -> 23.1 µs; spreading activation over 100: 22.8 -> 10.1 µs.
 - `clawhdf5-format`, `clawhdf5-filters`: both deflate paths hand the codec the
  whole chunk in one call, into a buffer allocated once, instead of streaming
  it through a 32 KiB buffer: about 5% on chunked writes and 10% on zlib-ng's
@@ -6,7 +6,7 @@
 [![Rust](https://img.shields.io/badge/rust-1.92%2B-orange.svg)](https://www.rust-lang.org)
 [![Tests](https://img.shields.io/badge/tests-1850%2B-brightgreen.svg)](#building)
 [![LongMemEval](https://img.shields.io/badge/LongMemEval__s-Turn--Level%20Hit@5%2081.4%25%20hybrid-blue.svg)](BENCHMARKS.md#longmemeval-results)
-[![Footprint](https://img.shields.io/badge/on--disk-1.7%20KB%2Frecord-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1)
+[![Footprint](https://img.shields.io/badge/on--disk-~820%20B%2Frecord%20float16%2C%20synthetic%20text-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1)
 ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file.
@@ -111,8 +111,8 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d
 | Keyword search | Separate FTS engine | Integrated BM25 |
 | Knowledge graph | Neo4j or none | In-file graph with spreading activation |
 | Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers |
-| Temporal queries | Custom code | Native temporal index (716ns) |
+| Temporal queries | Custom code | Native temporal index (622 ns range query over 10K) |
-| Multi-modal | Multiple stores | Unified cross-modal search |
+| Multi-modal | Multiple stores | Unified cross-modal search (exact scan: 842 µs over 1K records) |
 | Integrity | Hope for the best | Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance |
 | Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** |
@@ -120,7 +120,7 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d
 ## Performance
-Vector search and agent-memory operations below are benchmarked on Intel i7-12650H (10C/16T), 384-dim embeddings, Criterion.rs. The HDF5 Core I/O table immediately below is from a separate, independently reproduced run (see its own hardware note).
+The brute-force/IVF vector search, agent-memory, on-disk footprint and consolidation figures below were measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D, 8C/16T), commit 5c8323c, 384-dim embeddings; the commands are in [BENCHMARKS.md](BENCHMARKS.md). Exceptions are marked where they appear: the HDF5 Core I/O table immediately below is from a separate, independently reproduced run (see its own hardware note), and the HNSW `f32`/`i8` table and the in-memory `i8` column were not re-measured on 2026-09-24.
 ### HDF5 Core I/O (vs libhdf5 1.14.6)
@@ -154,33 +154,45 @@ and [§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quant
 | `f32` index | 0.9945 | 13 399 | 3.2 s |
 | `i8` index + exact re-score (**default for new stores**) | 0.9940 | **21 848** | **1.8 s** |
-Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31.
+Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31. These
 two rows are a paired comparison (medians of alternating runs, same binary).
 A single `f32` run on 2026-09-24 measured recall 0.9945, 19 001 QPS and a
 2.7 s build; the int8 row was not re-run, so the pair has not been re-checked
 ([§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index)).
-**Brute-force and IVF paths** (Criterion, i7-12650H):
+**Brute-force and IVF paths** (Criterion, tank, 2026-09-24):
-| Scale | Flat | IVF (nprobe=10) | IVF-PQ | vs MemX¹ |
+| Scale | Flat | IVF (nprobe=10) | IVF-PQ | MemX¹ (claimed, end-to-end) |
 |-------|------|-----------------|--------|----------|
-| 1K | **54 µs** | — | — | — |
+| 1K | **47.4 µs** | — | — | — |
-| 10K | 753 µs | **27 µs** | — | — |
+| 10K | 500.5 µs | **24.8 µs** | — | — |
-| 100K | 11.4 ms | 1.32 ms | **1.19 ms** | ~8–76× (see caveat) |
+| 100K | 6.58 ms | 592 µs | **869 µs** | <90 ms |
-> Reproduced on the same second machine (Ryzen 7 7800X3D) with a corrected,
+> These replace figures from the original i7-12650H run (flat 54 µs / 753 µs /
-> apples-to-apples SIMD/scalar/parallel comparison methodology — see
+> 11.4 ms); a 2026-08-05 run on tank had already matched the new ones — see
-> [BENCHMARKS.md § Independent Validation: tank — LongMemEval & Vector
+> [BENCHMARKS.md § Vector Search Latency](BENCHMARKS.md#vector-search-latency).
 > Search](BENCHMARKS.md#independent-validation-tank--longmemeval--vector-search-ryzen-7-7800x3d-2026-08-05).
 ### Agent Memory Operations
 | Operation | Latency | Scale |
 |-----------|---------|-------|
-| Hybrid search (`HDF5Memory::hybrid_search`, p50) | **70 µs** / 0.49 ms / 4.65 ms | 1K / 10K / 100K records |
+| Hybrid search (`HDF5Memory::hybrid_search`, p50) | **0.07 ms** / 0.49 ms / 4.69 ms | 1K / 10K / 100K records |
-| BM25 keyword search | **67 µs** | 1K records |
+| BM25 keyword search | **20.4 µs** | 1K records |
-| Knowledge graph BFS | **24 µs** | 1K entities |
+| Knowledge graph BFS | **23.1 µs** | 1K entities |
-| Spreading activation | **17 µs** | 100 entities |
+| Spreading activation | **10.1 µs** | 100 entities |
-| Temporal range query | **716 ns** | 10K timestamps |
+| Temporal range query | **622 ns** | 10K timestamps |
-| Consolidation cycle | **164 µs** | 1K records |
+| Consolidation cycle | **115.2 µs** | 1K records |
-| Memory write (WAL) | **18 µs** | per record (group-commit append; HDF5 batched at flush) |
+| Cross-modal search (exact scan, 2 embeddings per record) | **842.0 µs** / 8.44 ms | 1K / 10K records |
-| Importance gate | **61 ns** | per record |
+| Memory write (WAL) | **26.1 µs** | per record (group-commit append; HDF5 batched at flush) |
 | Importance gate | **57.6 ns** | per record (trivial skip) |
 The old 18 µs WAL write was undated, from another machine: v2.3.0 measures
 24.3 µs on the same hardware as this table, the same as an `f32` store today.
 `float16` stores (the new default) add ~2 µs for rounding; the int8 index adds
 nothing. See [BENCHMARKS.md § Write Path](BENCHMARKS.md#write-path).
 Knowledge-graph traversal was briefly 6.5x slower (155 µs) until this re-run
 found and fixed an adjacency index rebuilt on every traversal; see
 [§ Knowledge Graph](BENCHMARKS.md#knowledge-graph).
 ### Chunked Write Throughput (codec comparison)
@@ -195,7 +207,7 @@ by default (AoS→SoA byte transpose, +157–204% throughput for float data):
 Use `.with_zstd(3)` or `.with_deflate(6)` for write-heavy workloads — both now perform at ~720–750 MiB/s on large matrices. Use `.with_pcodec()` for write-once/read-many workloads where compression ratio matters more than encode speed. Disable auto-shuffle with `.without_shuffle()` for byte arrays that don't benefit from AoS→SoA transposition.
-> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search). The ratio overstates the real advantage by an unquantified margin — order-of-magnitude indication only. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171).
+> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search), so the two columns are not comparable and no ratio is given. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171).
 ### LongMemEval Retrieval Recall
@@ -248,18 +260,26 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points.
 ### Memory Footprint
-**On disk** — 384-dim embeddings, 200-char text
+**On disk** — 384-dim `float16` embeddings (the default for new stores),
 200-char text, `footprint_bench`
 ([BENCHMARKS.md § Memory Footprint](BENCHMARKS.md#memory-footprint-1)):
 | Records | File Size | Bytes/Record | Gzip-6 compressed |
 |---------|-----------|--------------|-------------------|
-| 1K | 1.7 MB | 1.8 KB | 277 KB (6.1x) |
+| 1K | 810.4 KB | 829 B | 56.4 KB |
-| 10K | 17.0 MB | 1.7 KB | 2.7 MB (6.2x) |
+| 10K | 7.8 MB | 820 B | 471.3 KB |
-| 100K | 169.8 MB | 1.7 KB | 26.9 MB (6.2x) |
+| 100K | 76.7 MB | 803 B | 4.5 MB |
-These figures are `f32` embeddings. New agent stores default to
+The benchmark's synthetic embeddings and text are far more repetitive than
-`MemoryConfig::float16`, which halves them: 100K × 384 records take 80.8 MiB
+real data (only 40 distinct texts), so no column here is an expectation for
-instead of 154.0.
+real data. The compressed column is an upper bound, and the Bytes/Record
 column is optimistic too: it is not an uncompressed figure, because the store
 always deflates its text (any string dataset of 4 KiB or more) whatever
 `MemoryConfig::compression` says. The `float16` embeddings alone are 768 B per
 record, so 200 characters of real text would take a record above 820 B.
 This table used to show `f32` stores (1.7 KB per record, 169.8 MB at 100K);
 those were not re-measured. The float16 study compares the two on the same
 data: 100K × 384 records take 80.8 MiB as `float16` and 154.0 MiB as `f32`.
 **In memory** — a store reopened from disk, 384-dim `f32`, measured with a
 counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-footprint)):
@@ -271,7 +291,8 @@ counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-foo
 | 100K | 146 MiB | 399 MiB (2.72x) | **256 MiB (1.74x)** |
 Down from 505 MiB (3.44x) at 100K before v2.6.0, when the cache held every
-embedding twice.
+embedding twice. The `f32` column was re-measured on 2026-09-24 and reproduced
 exactly; the `i8` column was not re-run.
 ### Consolidation Efficiency
@@ -282,7 +303,10 @@ embedding twice.
 |--------|--------|-------|-------|
 | Records in store | 1,000 | 100 | −90% |
 | Hit@1 recall (signal records) | 100% | 100% | no loss |
-| Search latency | 2.75 ms | 0.31 ms | **8.8x faster** |
+| Search latency (avg) | 2.22 ms | 0.24 ms | **9.3x faster** |
 The consolidation cycle that does this took 0.13 ms; at 10K records a cycle
 takes 2.16 ms.
 **Full benchmark details: [BENCHMARKS.md](BENCHMARKS.md)**
@@ -754,9 +778,37 @@ Replace in `Cargo.toml` and source:
 ```bash
 cargo install --path crates/clawhdf5-migrate
-clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedding-dim 384
+clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedder minilm
 ```
 The output is an ordinary `clawhdf5-agent` store, written through the agent's
 own API: open it with `HDF5Memory::open` (or `clawhdf5-cli --path memory.h5 …`)
 and search it straight away. What carries over from the ZeroClaw tables:
 | SQLite | Agent store |
 |--------|-------------|
 | `memory_chunks` | memory records (text, embedding, source channel, timestamp, session id, tags); rows with `deleted = 1` become deleted records, or are left out with `--skip-deleted` |
 | `sessions` | sessions (id, start/end index, channel, summary, timestamp) |
 | `entities`, `relations` | knowledge graph entities and relations; entities get new ids and relations are re-pointed at them |
 The chunk `id` column has no counterpart in the agent store, so records are
 written in `id` order and numbered from 0. Embeddings are stored as float16
 like any new store; `--f32` keeps full precision (and is required for values
 beyond ±65504). The embedding dimension is detected from the first row unless
 `--embedding-dim` is given, and every row must have it: a row of another length
 is an error, never truncated or padded. A source with no memory records (only
 sessions or the graph) needs `--embedding-dim`, since a store's dimension is
 fixed when it is created. Every row is checked before the output is created,
 so a source that cannot be migrated leaves an existing store at `--hdf5` as it
 was. `--incremental` adds to an existing store only the rows it does not
 already hold; the source must have the store's dimension, and records already
 in the store take the source's deleted flag (a row deleted in SQLite since the
 last run is deleted in the store; one un-deleted there is written again, as
 the agent has no un-delete). The tool reads the result back with
 `HDF5Memory::open_read_only`, compares it with the source (every row with
 `--validate-full`) and checks that a migrated record is found by search;
 `--dry-run` only counts the rows.
 ---
 ## Roadmap
@@ -45,6 +45,10 @@ harness = false
 name = "memory_bench"
 harness = false
 [[bench]]
 name = "multimodal_bench"
 harness = false
 [features]
 default = ["float16", "hnsw", "parallel"]
 float16 = ["half"]
@@ -0,0 +1,107 @@
 //! Multi-modal memory search benchmarks (`clawhdf5_agent::multimodal`).
 //!
 //! Covers `MultiModalStore::search_cross_modal` (every embedding of every
 //! record, whatever its modality) and, for comparison,
 //! `MultiModalStore::search_by_modality` restricted to one modality.
 //!
 //! Corpus: N records (1K and 10K), each carrying two 384-dim embeddings —
 //! a text embedding of its caption plus one embedding of its primary modality,
 //! cycling Image / Audio / Video — so a cross-modal query scores 2N vectors.
 //! All data comes from a fixed-seed LCG, so every run sees the same corpus.
 //!
 //! Run: `cargo bench -p clawhdf5-agent --bench multimodal_bench`
 use std::collections::HashMap;
 use clawhdf5_agent::multimodal::{
    MediaRef, ModalEmbedding, Modality, MultiModalRecord, MultiModalStore,
 };
 use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
 // ---------------------------------------------------------------------------
 // Simple deterministic PRNG (LCG), same as the other agent benches
 // ---------------------------------------------------------------------------
 struct Rng(u32);
 impl Rng {
    fn new(seed: u32) -> Self {
        Self(seed)
    }
    fn next_u32(&mut self) -> u32 {
        self.0 = self.0.wrapping_mul(1103515245).wrapping_add(12345);
        self.0 >> 16
    }
    fn next_f32(&mut self) -> f32 {
        self.next_u32() as f32 / 65536.0 - 0.5
    }
 }
 fn make_vec(rng: &mut Rng, dim: usize) -> Vec<f32> {
    (0..dim).map(|_| rng.next_f32()).collect()
 }
 // ---------------------------------------------------------------------------
 // Corpus
 // ---------------------------------------------------------------------------
 const DIM: usize = 384;
 const K: usize = 10;
 const MEDIA: [(Modality, &str, &str); 3] = [
    (Modality::Image, "image/png", "clip-vit-base"),
    (Modality::Audio, "audio/wav", "clap-base"),
    (Modality::Video, "video/mp4", "xclip-base"),
 ];
 fn build_store(n: usize, seed: u32) -> MultiModalStore {
    let mut rng = Rng::new(seed);
    let mut store = MultiModalStore::new();
    for i in 0..n {
        let (modality, mime, model) = &MEDIA[i % MEDIA.len()];
        let embeddings = vec![
            ModalEmbedding::new(Modality::Text, make_vec(&mut rng, DIM), "minilm-l6"),
            ModalEmbedding::new(modality.clone(), make_vec(&mut rng, DIM), *model),
        ];
        store.add_record(MultiModalRecord {
            id: 0,
            primary_modality: modality.clone(),
            text_content: Some(format!("{modality} memory {i}")),
            media_ref: Some(MediaRef::path(format!("/media/{i}"), *mime)),
            embeddings,
            observation: None,
            timestamp: 1_700_000_000.0 + i as f64,
            metadata: HashMap::new(),
        });
    }
    store
 }
 // ---------------------------------------------------------------------------
 // Benchmarks
 // ---------------------------------------------------------------------------
 fn multimodal_search_benches(c: &mut Criterion) {
    let query = make_vec(&mut Rng::new(99), DIM);
    let mut group = c.benchmark_group("multimodal_search");
    group.sample_size(50);
    for (label, n) in [("1k", 1_000usize), ("10k", 10_000)] {
        let store = build_store(n, 42);
        assert_eq!(store.count(), n);
        group.bench_with_input(BenchmarkId::new("cross_modal", label), &n, |b, _| {
            b.iter(|| store.search_cross_modal(&query, K));
        });
        group.bench_with_input(BenchmarkId::new("by_modality_image", label), &n, |b, _| {
            b.iter(|| store.search_by_modality(&Modality::Image, &query, K));
        });
    }
    group.finish();
 }
 criterion_group!(multimodal_benches, multimodal_search_benches);
 criterion_main!(multimodal_benches);
@@ -163,12 +163,13 @@ fn levenshtein(a: &str, b: &str) -> usize {
 /// entities-slice-index map, and an entity-id -> relation-indices map (edges
 /// touching that entity as either source or target).
 ///
-/// Built fresh per traversal call rather than cached on `KnowledgeCache`:
+/// Cached on `KnowledgeCache` and checked against a fingerprint of the graph
-/// entities/relations are plain `pub` `Vec`s that get pushed to directly
+/// on every use ([`graph_fingerprint`]). entities/relations are plain `pub`
-/// (e.g. `schema.rs`'s load path bypasses `add_entity`/`add_relation`), so a
+/// `Vec`s that get changed directly (e.g. `schema.rs`'s load path bypasses
-/// persistent index would need extra bookkeeping to avoid drifting stale. A
+/// `add_entity`/`add_relation`), so the cache cannot rely on being told about
-/// one-off O(V+E) build per call is still a large win over the O(V·E) (BFS)
+/// changes; the fingerprint notices any of them. Rebuilding it on every
-/// / O(steps·active·E) (spreading activation) scans it replaces.
+/// traversal instead made a 2-hop BFS over 1K entities 6.5x slower than the
 /// scan it replaced (24 -> 155 µs; `BENCHMARKS.md`, "Knowledge Graph").
 struct AdjacencyIndex {
    entity_index: HashMap<u64, usize>,
    by_entity: HashMap<u64, Vec<usize>>,
@@ -204,6 +205,45 @@ impl AdjacencyIndex {
    }
 }
 /// A hash of everything [`AdjacencyIndex`] depends on — each entity's id and
 /// position, each relation's endpoints and position. One linear pass, no
 /// allocation: far cheaper than building the index, which hashes the same
 /// values into two maps.
 fn graph_fingerprint(entities: &[Entity], relations: &[Relation]) -> u64 {
    // splitmix64-style mixing; order matters, so positions are covered.
    fn mix(h: u64, v: u64) -> u64 {
        let mut z = (h ^ v).wrapping_add(0x9E37_79B9_7F4A_7C15);
        z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
        z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
        z ^ (z >> 31)
    }
    let mut h = mix(entities.len() as u64, relations.len() as u64);
    for e in entities {
        h = mix(h, e.id);
    }
    for r in relations {
        h = mix(mix(h, r.src), r.tgt);
    }
    h
 }
 /// The cached [`AdjacencyIndex`] and the fingerprint it was built for.
 /// Cloning a `KnowledgeCache` starts the clone with an empty cache.
 #[derive(Default)]
 struct AdjacencyCache(std::sync::Mutex<Option<(u64, std::sync::Arc<AdjacencyIndex>)>>);
 impl Clone for AdjacencyCache {
    fn clone(&self) -> Self {
        Self::default()
    }
 }
 impl std::fmt::Debug for AdjacencyCache {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        f.write_str("AdjacencyCache")
    }
 }
 // ---------------------------------------------------------------------------
 // KnowledgeCache
 // ---------------------------------------------------------------------------
@@ -216,6 +256,7 @@ pub struct KnowledgeCache {
    pub alias_strings: Vec<String>,
    pub alias_entity_ids: Vec<i64>,
    next_entity_id: u64,
    adjacency: AdjacencyCache,
 }
 impl KnowledgeCache {
@@ -226,6 +267,7 @@ impl KnowledgeCache {
            alias_strings: Vec::new(),
            alias_entity_ids: Vec::new(),
            next_entity_id: 0,
            adjacency: AdjacencyCache::default(),
        }
    }
@@ -236,9 +278,29 @@ impl KnowledgeCache {
            alias_strings: Vec::new(),
            alias_entity_ids: Vec::new(),
            next_entity_id: next_id,
            adjacency: AdjacencyCache::default(),
        }
    }
    /// The adjacency index for the graph as it is now: the cached one if the
    /// graph's fingerprint still matches, otherwise rebuilt and cached.
    fn adjacency_index(&self) -> std::sync::Arc<AdjacencyIndex> {
        let fp = graph_fingerprint(&self.entities, &self.relations);
        let mut slot = self
            .adjacency
            .0
            .lock()
            .unwrap_or_else(std::sync::PoisonError::into_inner);
        if let Some((cached_fp, idx)) = slot.as_ref()
            && *cached_fp == fp
        {
            return idx.clone();
        }
        let idx = std::sync::Arc::new(AdjacencyIndex::build(&self.entities, &self.relations));
        *slot = Some((fp, idx.clone()));
        idx
    }
    // -----------------------------------------------------------------------
    // Entity management
    // -----------------------------------------------------------------------
@@ -397,7 +459,7 @@ impl KnowledgeCache {
    /// together with their discovered depth.  The seed entity itself is NOT
    /// included.  Traversal follows both outgoing and incoming relation edges.
    pub fn bfs_neighbors(&self, entity_id: u64, max_depth: usize) -> Vec<(Entity, usize)> {
-        let idx = AdjacencyIndex::build(&self.entities, &self.relations);
+        let idx = self.adjacency_index();
        let mut visited: HashSet<u64> = HashSet::new();
        let mut queue: VecDeque<(u64, usize)> = VecDeque::new();
        let mut results: Vec<(Entity, usize)> = Vec::new();
@@ -502,7 +564,7 @@ impl KnowledgeCache {
        min_activation: f32,
        max_steps: usize,
    ) -> Vec<(u64, f32)> {
-        let idx = AdjacencyIndex::build(&self.entities, &self.relations);
+        let idx = self.adjacency_index();
        let mut activation: HashMap<u64, f32> = HashMap::new();
        // Initialise seeds with activation 1.0.
@@ -631,6 +693,51 @@ impl Default for KnowledgeCache {
 mod tests {
    use super::*;
    #[test]
    fn cached_adjacency_sees_direct_changes_to_the_graph() {
        // The index is cached across traversals, but entities/relations are
        // pub Vecs anyone can edit; every kind of edit must be seen.
        let mut kg = KnowledgeCache::new();
        let a = kg.add_entity("a", "t", -1);
        let b = kg.add_entity("b", "t", -1);
        let c = kg.add_entity("c", "t", -1);
        kg.add_relation(a, b, "r", 1.0);
        let ids = |kg: &KnowledgeCache| -> Vec<u64> {
            let mut v: Vec<u64> = kg.bfs_neighbors(a, 3).iter().map(|(e, _)| e.id).collect();
            v.sort();
            v
        };
        assert_eq!(ids(&kg), vec![b]);
        assert_eq!(ids(&kg), vec![b], "cached index reused");
        // Pushed directly, bypassing add_relation.
        kg.relations.push(Relation {
            src: b,
            tgt: c,
            ..Relation::default()
        });
        assert_eq!(ids(&kg), vec![b, c]);
        // Rewired in place: same lengths, different edge.
        kg.relations[1].tgt = a;
        assert_eq!(ids(&kg), vec![b]);
        // Removed and replaced: same lengths again.
        kg.relations.pop();
        kg.relations.push(Relation {
            src: a,
            tgt: c,
            ..Relation::default()
        });
        assert_eq!(ids(&kg), vec![b, c]);
        let act: Vec<u64> = kg
            .spreading_activation(&[a], 0.5, 0.0, 2)
            .iter()
            .map(|(id, _)| *id)
            .collect();
        assert!(act.contains(&c));
    }
    // -----------------------------------------------------------------------
    // Original tests — must remain passing
    // -----------------------------------------------------------------------
@@ -73,7 +73,7 @@ pub use ephemeral::{EphemeralEntry, EphemeralStats};
 use knowledge::KnowledgeCache;
 use memory_strategy::{Exchange, MemoryStrategy, StrategyOutput};
 pub use search::SearchOptions;
-use session::SessionCache;
+pub use session::{SessionCache, SessionEntry};
 // --- Error type ---
@@ -1028,6 +1028,18 @@ impl HDF5Memory {
        &self.config
    }
    /// The sessions recorded in this store.
    pub fn sessions(&self) -> &SessionCache {
        &self.sessions
    }
    /// Mutable access to the sessions, e.g. to add many at once. Changes
    /// reach the disk at the next checkpoint (any flushing call, such as
    /// [`HDF5Memory::flush_wal`] or `save_batch`), not immediately.
    pub fn sessions_mut(&mut self) -> &mut SessionCache {
        &mut self.sessions
    }
    /// Get a reference to the knowledge cache.
    pub fn knowledge(&self) -> &KnowledgeCache {
        &self.knowledge
@@ -1401,6 +1413,35 @@ impl HDF5Memory {
 }
 impl HDF5Memory {
    /// Delete many records with a single checkpoint, where
    /// [`AgentMemory::delete`] checkpoints once per record.
    ///
    /// All or nothing: if any id is out of range or already deleted (or
    /// repeated), nothing is deleted and `MemoryError::NotFound` is returned.
    /// Unlike `delete`, this never auto-compacts, so the records stay in the
    /// store as tombstones (their indices unchanged) until [`AgentMemory::compact`]
    /// is called — importers use it to carry over records that were already
    /// deleted in the source.
    pub fn delete_batch(&mut self, ids: &[usize]) -> Result<()> {
        let mut seen = std::collections::HashSet::with_capacity(ids.len());
        for &id in ids {
            if self.cache.tombstones.get(id).copied() != Some(0) || !seen.insert(id) {
                return Err(MemoryError::NotFound(format!(
                    "entry {id} not found or already deleted"
                )));
            }
        }
        if ids.is_empty() {
            return Ok(());
        }
        for &id in ids {
            self.cache.mark_deleted(id);
            self.hnsw_on_delete(id);
            self.bm25_on_delete(id);
        }
        self.flush()
    }
    pub fn tick_session(&mut self) -> Result<()> {
        let d = self.config.decay_factor;
        for w in self.cache.activation_weights.iter_mut() {
@@ -1599,6 +1640,79 @@ mod tests {
        }
    }
    #[test]
    fn delete_batch_tombstones_without_compacting() {
        let dir = TempDir::new().unwrap();
        let path = dir.path().join("test.h5");
        let mut mem = HDF5Memory::create(make_config(&dir)).unwrap();
        mem.save_batch(
            (0..4)
                .map(|i| make_entry(&format!("record {i}"), &[i as f32, 1.0, 0.0, 0.0]))
                .collect(),
        )
        .unwrap();
        // 3 of 4 is far past compact_threshold (0.3): delete() would compact.
        mem.delete_batch(&[0, 1, 3]).unwrap();
        assert_eq!(mem.count(), 4);
        assert_eq!(mem.count_active(), 1);
        drop(mem);
        let mut mem = HDF5Memory::open(&path).unwrap();
        assert_eq!(mem.cache.tombstones, vec![1, 1, 0, 1]);
        let hits = mem.hybrid_search(&[0.0, 1.0, 0.0, 0.0], "record", 0.5, 0.5, 10);
        assert!(
            hits.iter().all(|r| r.index == 2),
            "tombstoned record returned"
        );
    }
    #[test]
    fn delete_batch_is_all_or_nothing() {
        let dir = TempDir::new().unwrap();
        let mut mem = HDF5Memory::create(make_config(&dir)).unwrap();
        mem.save_batch(vec![
            make_entry("a", &[1.0, 0.0, 0.0, 0.0]),
            make_entry("b", &[0.0, 1.0, 0.0, 0.0]),
        ])
        .unwrap();
        for bad in [&[0, 5][..], &[1, 1][..]] {
            assert!(matches!(
                mem.delete_batch(bad),
                Err(MemoryError::NotFound(_))
            ));
            assert_eq!(mem.count_active(), 2, "{bad:?} deleted something");
        }
        mem.delete_batch(&[]).unwrap();
        assert_eq!(mem.count_active(), 2);
    }
    #[test]
    fn sessions_mut_add_at_keeps_timestamp_across_reopen() {
        let dir = TempDir::new().unwrap();
        let path = dir.path().join("test.h5");
        let mut mem = HDF5Memory::create(make_config(&dir)).unwrap();
        mem.sessions_mut()
            .add_at("s-old", 2, 7, "discord", "old summary", 1.7e15);
        mem.flush_wal().unwrap();
        drop(mem);
        let mem = HDF5Memory::open_read_only(&path).unwrap();
        let s = mem.sessions();
        assert_eq!(s.len(), 1);
        let e = &s.entries[0];
        assert_eq!(
            (
                e.id.as_str(),
                e.start_idx,
                e.end_idx,
                e.channel.as_str(),
                e.ts
            ),
            ("s-old", 2, 7, "discord", 1.7e15)
        );
        assert_eq!(s.summaries[0], "old summary");
    }
    #[test]
    fn create_new_file() {
        let dir = TempDir::new().unwrap();
@@ -33,7 +33,7 @@ impl SessionCache {
        self.entries.is_empty()
    }
-    /// Add a new session with its summary.
+    /// Add a new session with its summary, timestamped now.
    pub fn add(
        &mut self,
        id: &str,
@@ -47,6 +47,21 @@ impl SessionCache {
            .unwrap_or_default()
            .as_secs_f64()
            * 1_000_000.0; // microseconds
        self.add_at(id, start_idx, end_idx, channel, summary, ts);
    }
    /// Add a session with an explicit timestamp (Unix **microseconds**, the
    /// unit [`SessionEntry::ts`] uses) — for importers carrying sessions over
    /// from another store, whose original time should be kept.
    pub fn add_at(
        &mut self,
        id: &str,
        start_idx: usize,
        end_idx: usize,
        channel: &str,
        summary: &str,
        ts: f64,
    ) {
        self.entries.push(SessionEntry {
            id: id.to_string(),
            start_idx: start_idx as u64,
@@ -11,12 +11,14 @@
 //!
 //! Configuration matrix:
 //!   - Text lengths: short (50 chars), medium (200 chars), long (1000 chars)
-//!   - Embedding: 384-dim f32 (1536 bytes raw per record)
+//!   - Embedding: 384-dim, stored as float16 (the default for new stores) or
 //!     f32 with `--f32`; "raw" bytes are counted as f32 input either way
 //!   - WAL: enabled and disabled
 //!
 //! # Usage
 //! ```
-//! cargo run --release --bin footprint_bench
+//! cargo run --release --bin footprint_bench           # float16 stores
 //! cargo run --release --bin footprint_bench -- --f32  # f32 stores
 //! ```
 use std::time::Instant;
@@ -24,6 +26,9 @@ use std::time::Instant;
 use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
 use tempfile::TempDir;
 /// `--f32`: build f32 stores instead of the library's float16 default.
 static F32: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
 const EMBEDDING_DIM: usize = 384;
 // Raw bytes per record: 384 f32 embeddings + median text + overhead
@@ -152,6 +157,9 @@ fn measure_footprint(
    config.compression = compression;
    config.compression_level = if compression { 6 } else { 0 };
    config.compact_threshold = 0.0;
    if F32.load(std::sync::atomic::Ordering::Relaxed) {
        config.float16 = false;
    }
    let mut memory = HDF5Memory::create(config).expect("HDF5Memory::create failed");
@@ -241,11 +249,19 @@ fn fmt_n(n: usize) -> String {
 // ---------------------------------------------------------------------------
 fn main() {
    if std::env::args().skip(1).any(|a| a == "--f32") {
        F32.store(true, std::sync::atomic::Ordering::Relaxed);
    }
    let stored = if F32.load(std::sync::atomic::Ordering::Relaxed) {
        "f32 (1,536 bytes per record)"
    } else {
        "float16 (768 bytes per record; the default for new stores)"
    };
    println!("=================================================================");
    println!("  ClawhDF5 Memory Footprint Benchmark");
    println!("=================================================================");
    println!();
-    println!("Embedding: 384-dim f32 = 1,536 bytes raw per record");
+    println!("Embedding: 384-dim, stored as {stored}; raw input counted as f32");
    println!("Text lengths: short=50 chars, medium=200 chars, long=1000 chars");
    println!();
@@ -3,7 +3,7 @@ name = "clawhdf5-migrate"
 version = "2.7.0"
 edition = "2024"
 rust-version.workspace = true
-description = "CLI to migrate SQLite agent memory databases to HDF5 format"
+description = "CLI to migrate SQLite agent memory databases to clawhdf5-agent stores"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
 readme = "README.md"
@@ -17,10 +17,8 @@ path = "src/main.rs"
 [dependencies]
 clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.7.0" }
 clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
 clawhdf5 = { path = "../clawhdf5", version = "2.7.0" }
 rusqlite = { version = "0.31", features = ["bundled"] }
 clap = { version = "4", features = ["derive"] }
 half = { workspace = true }
 [dev-dependencies]
 tempfile = { workspace = true }
@@ -3,9 +3,12 @@
 [![crates.io](https://img.shields.io/crates/v/clawhdf5-migrate.svg)](https://crates.io/crates/clawhdf5-migrate)
 [![docs.rs](https://img.shields.io/docsrs/clawhdf5-migrate)](https://docs.rs/clawhdf5-migrate)
-CLI tool to migrate SQLite agent memory databases to HDF5 format.
+CLI tool to migrate SQLite agent memory databases (the ZeroClaw layout) to a
 [clawhdf5-agent](https://crates.io/crates/clawhdf5-agent) store.
-Converts existing SQLite-based agent memory stores (embeddings, text chunks, metadata) into the HDF5 format used by [clawhdf5-agent](https://crates.io/crates/clawhdf5-agent).
+The output is written through `clawhdf5-agent`'s own API, so it opens with
 `HDF5Memory::open` and is searchable immediately: memory records, sessions and
 the knowledge graph (entities and relations) are carried over.
 ## Installation
@@ -16,9 +19,19 @@ cargo install clawhdf5-migrate
 ## Usage
 ```bash
-clawhdf5-migrate --input agent.db --output agent.h5
+clawhdf5-migrate --sqlite agent.db --hdf5 agent.h5 --agent-id my-agent
 ```
 Embeddings are stored as float16 (the library default for new stores); pass
 `--f32` for full precision. Every embedding must have the same dimension
 (the first row's, or `--embedding-dim`, which a source with no memory records
 requires); rows are never truncated, and the whole source is checked before an
 existing output store is replaced. `--incremental` adds only new rows to an
 existing store of the same dimension and carries over changes to rows'
 deleted flags, `--skip-deleted` leaves out tombstoned rows, and `--dry-run`
 only counts.
 See `clawhdf5-migrate --help` for every option.
 ## License
 MIT
@@ -1,163 +0,0 @@
 //! Read a migration HDF5 file back into the in-memory data model.
 //!
 //! Used to verify migrated content (real validation) and to merge new rows into
 //! an existing output (incremental migration). Mirrors the layout produced by
 //! [`crate::hdf5_writer`].
 use clawhdf5::reader::{File, Group};
 use clawhdf5_format::type_builders::AttrValue;
 use crate::sqlite_reader::{Entity, MemoryChunk, Relation, Session, SqliteData};
 type BoxErr = Box<dyn std::error::Error>;
 fn read_strings(group: &Group<'_>, name: &str) -> Result<Vec<String>, BoxErr> {
    Ok(group.dataset(name)?.read_string()?)
 }
 fn read_i64s(group: &Group<'_>, name: &str) -> Result<Vec<i64>, BoxErr> {
    Ok(group.dataset(name)?.read_i64()?)
 }
 fn read_f64s(group: &Group<'_>, name: &str) -> Result<Vec<f64>, BoxErr> {
    Ok(group.dataset(name)?.read_f64()?)
 }
 /// Read the embeddings dataset as a flat `Vec<f32>` of `n * dim` values,
 /// handling both f32 and (lossy) f16 storage.
 fn read_embeddings_flat(group: &Group<'_>) -> Result<Vec<f32>, BoxErr> {
    Ok(group.dataset("embeddings")?.read_f32()?)
 }
 /// Read a migration HDF5 file into a [`SqliteData`].
 pub fn read_hdf5(path: &str) -> Result<SqliteData, BoxErr> {
    let file = File::open(path)?;
    let embedding_dim = match file.root().attrs()?.get("embedding_dim") {
        Some(AttrValue::I64(d)) => *d as usize,
        _ => 0,
    };
    let chunks = read_chunks(&file, embedding_dim)?;
    let sessions = read_sessions(&file)?;
    let entities = read_entities(&file)?;
    let relations = read_relations(&file)?;
    Ok(SqliteData {
        chunks,
        sessions,
        entities,
        relations,
        embedding_dim,
        // Not a SQLite read — the caller (incremental migration) carries
        // forward the current run's actual `source_path` from the fresh
        // SQLite read instead of using this placeholder.
        source_path: String::new(),
    })
 }
 fn read_chunks(file: &File, dim: usize) -> Result<Vec<MemoryChunk>, BoxErr> {
    let g = file.group("chunks")?;
    let count = group_count(&g)?;
    if count == 0 {
        return Ok(Vec::new());
    }
    let ids = read_i64s(&g, "id")?;
    let texts = read_strings(&g, "text")?;
    let channels = read_strings(&g, "source_channel")?;
    let timestamps = read_f64s(&g, "timestamp")?;
    let session_ids = read_strings(&g, "session_id")?;
    let tags = read_strings(&g, "tags")?;
    let deleted = g.dataset("deleted")?.read_i32()?;
    let emb_flat = read_embeddings_flat(&g)?;
    let dim = dim.max(1);
    let mut chunks = Vec::with_capacity(ids.len());
    for (i, &id) in ids.iter().enumerate() {
        let embedding = emb_flat
            .get(i * dim..(i + 1) * dim)
            .map(|s| s.to_vec())
            .unwrap_or_default();
        chunks.push(MemoryChunk {
            id,
            chunk: texts.get(i).cloned().unwrap_or_default(),
            embedding,
            source_channel: channels.get(i).cloned().unwrap_or_default(),
            timestamp: timestamps.get(i).copied().unwrap_or(0.0),
            session_id: session_ids.get(i).cloned().unwrap_or_default(),
            tags: tags.get(i).cloned().unwrap_or_default(),
            deleted: deleted.get(i).copied().unwrap_or(0),
        });
    }
    Ok(chunks)
 }
 fn read_sessions(file: &File) -> Result<Vec<Session>, BoxErr> {
    let g = file.group("sessions")?;
    if group_count(&g)? == 0 {
        return Ok(Vec::new());
    }
    let ids = read_strings(&g, "id")?;
    let starts = read_i64s(&g, "start_idx")?;
    let ends = read_i64s(&g, "end_idx")?;
    let channels = read_strings(&g, "channel")?;
    let timestamps = read_f64s(&g, "timestamp")?;
    let summaries = read_strings(&g, "summary")?;
    Ok((0..ids.len())
        .map(|i| Session {
            id: ids[i].clone(),
            start_idx: starts.get(i).copied().unwrap_or(0),
            end_idx: ends.get(i).copied().unwrap_or(0),
            channel: channels.get(i).cloned().unwrap_or_default(),
            timestamp: timestamps.get(i).copied().unwrap_or(0.0),
            summary: summaries.get(i).cloned().unwrap_or_default(),
        })
        .collect())
 }
 fn read_entities(file: &File) -> Result<Vec<Entity>, BoxErr> {
    let g = file.group("entities")?;
    if group_count(&g)? == 0 {
        return Ok(Vec::new());
    }
    let ids = read_i64s(&g, "id")?;
    let names = read_strings(&g, "name")?;
    let types = read_strings(&g, "type")?;
    let emb_idxs = read_i64s(&g, "embedding_idx")?;
    Ok((0..ids.len())
        .map(|i| Entity {
            id: ids[i],
            name: names.get(i).cloned().unwrap_or_default(),
            entity_type: types.get(i).cloned().unwrap_or_default(),
            embedding_idx: emb_idxs.get(i).copied().unwrap_or(-1),
        })
        .collect())
 }
 fn read_relations(file: &File) -> Result<Vec<Relation>, BoxErr> {
    let g = file.group("relations")?;
    if group_count(&g)? == 0 {
        return Ok(Vec::new());
    }
    let srcs = read_i64s(&g, "src")?;
    let tgts = read_i64s(&g, "tgt")?;
    let rels = read_strings(&g, "relation")?;
    let weights = read_f64s(&g, "weight")?;
    let timestamps = read_f64s(&g, "timestamp")?;
    Ok((0..srcs.len())
        .map(|i| Relation {
            src: srcs[i],
            tgt: tgts.get(i).copied().unwrap_or(0),
            relation: rels.get(i).cloned().unwrap_or_default(),
            weight: weights.get(i).copied().unwrap_or(1.0),
            timestamp: timestamps.get(i).copied().unwrap_or(0.0),
        })
        .collect())
 }
 fn group_count(group: &Group<'_>) -> Result<u64, BoxErr> {
    match group.attrs()?.get("count") {
        Some(AttrValue::I64(n)) => Ok(*n as u64),
        _ => Ok(0),
    }
 }
@@ -1,366 +0,0 @@
 use clawhdf5::writer::FileBuilder;
 use clawhdf5_format::datatype::{CharacterSet, Datatype, StringPadding};
 use clawhdf5_format::type_builders::AttrValue;
 use crate::sqlite_reader::SqliteData;
 /// Options controlling HDF5 output.
 pub struct WriteOptions {
    pub agent_id: String,
    pub embedder: String,
    pub compression: bool,
    pub compression_level: u32,
    pub float16: bool,
 }
 /// Write SQLite data to an HDF5 file.
 pub fn write_hdf5(
    path: &str,
    data: &SqliteData,
    opts: &WriteOptions,
 ) -> Result<(), Box<dyn std::error::Error>> {
    let mut builder = FileBuilder::new();
    let timestamp = iso8601_now();
    // Root-level metadata attributes
    builder.set_attr("agent_id", AttrValue::String(opts.agent_id.clone()));
    builder.set_attr("embedder", AttrValue::String(opts.embedder.clone()));
    builder.set_attr("embedding_dim", AttrValue::I64(data.embedding_dim as i64));
    builder.set_attr("source", AttrValue::String("sqlite-migration".into()));
    builder.set_attr("version", AttrValue::I64(1));
    // Lineage: which SQLite database this output was migrated from and when,
    // plus the migrator tool version — so a chain of `--incremental` runs
    // still has an audit trail instead of every run overwriting the same
    // static attributes (see research/03_provenance.md, INT-03).
    builder.set_attr("source_path", AttrValue::String(data.source_path.clone()));
    builder.set_attr("migrated_at", AttrValue::String(timestamp.clone()));
    builder.set_attr(
        "migrator_version",
        AttrValue::String(env!("CARGO_PKG_VERSION").to_owned()),
    );
    write_chunks_group(&mut builder, data, opts, &timestamp);
    write_sessions_group(&mut builder, data);
    write_entities_group(&mut builder, data);
    write_relations_group(&mut builder, data);
    builder.write(path)?;
    Ok(())
 }
 /// Current UTC time formatted as an ISO-8601 / RFC-3339 timestamp
 /// (`YYYY-MM-DDTHH:MM:SSZ`), with no external date/time dependency.
 fn iso8601_now() -> String {
    let secs = std::time::SystemTime::now()
        .duration_since(std::time::UNIX_EPOCH)
        .unwrap_or_default()
        .as_secs();
    let days = (secs / 86_400) as i64;
    let time_of_day = secs % 86_400;
    let (h, m, s) = (
        time_of_day / 3600,
        (time_of_day % 3600) / 60,
        time_of_day % 60,
    );
    let (y, mo, d) = civil_from_days(days);
    format!("{y:04}-{mo:02}-{d:02}T{h:02}:{m:02}:{s:02}Z")
 }
 /// Days-since-epoch to (year, month, day), Howard Hinnant's `civil_from_days`
 /// algorithm (proleptic Gregorian calendar, valid for the full `i64` range).
 fn civil_from_days(z: i64) -> (i64, u32, u32) {
    let z = z + 719_468;
    let era = if z >= 0 { z } else { z - 146_096 } / 146_097;
    let doe = (z - era * 146_097) as u64; // [0, 146096]
    let yoe = (doe - doe / 1460 + doe / 36_524 - doe / 146_096) / 365; // [0, 399]
    let y = yoe as i64 + era * 400;
    let doy = doe - (365 * yoe + yoe / 4 - yoe / 100); // [0, 365]
    let mp = (5 * doy + 2) / 153; // [0, 11]
    let d = (doy - (153 * mp + 2) / 5 + 1) as u32; // [1, 31]
    let m = (if mp < 10 { mp + 3 } else { mp - 9 }) as u32; // [1, 12]
    let y = if m <= 2 { y + 1 } else { y };
    (y, m, d)
 }
 /// Build a fixed-length string Datatype from the max byte length of the items.
 fn string_dtype(max_len: usize) -> Datatype {
    Datatype::String {
        size: max_len.max(1) as u32,
        padding: StringPadding::NullPad,
        charset: CharacterSet::Utf8,
    }
 }
 /// Pack a slice of strings into null-padded raw bytes of uniform width.
 fn pack_strings(strings: &[String]) -> (Vec<u8>, usize) {
    let max_len = strings.iter().map(|s| s.len()).max().unwrap_or(0).max(1);
    let mut buf = vec![0u8; strings.len() * max_len];
    for (i, s) in strings.iter().enumerate() {
        let start = i * max_len;
        let bytes = s.as_bytes();
        let copy_len = bytes.len().min(max_len);
        buf[start..start + copy_len].copy_from_slice(&bytes[..copy_len]);
    }
    (buf, max_len)
 }
 fn apply_compression(ds: &mut clawhdf5_format::type_builders::DatasetBuilder, opts: &WriteOptions) {
    if opts.compression {
        ds.with_deflate(opts.compression_level);
        ds.with_shuffle();
    }
 }
 fn write_chunks_group(
    builder: &mut FileBuilder,
    data: &SqliteData,
    opts: &WriteOptions,
    timestamp: &str,
 ) {
    let mut group = builder.create_group("chunks");
    let n = data.chunks.len() as u64;
    if n == 0 {
        group.set_attr("count", AttrValue::I64(0));
        builder.add_group(group.finish());
        return;
    }
    group.set_attr("count", AttrValue::I64(n as i64));
    // Source attribution attached directly to the content-bearing datasets
    // (SHA-256 of the raw bytes + creator/timestamp/source), so the chunk
    // text and embeddings each carry their own verifiable provenance
    // (see clawhdf5_format::provenance / `Dataset::verify_provenance`).
    let source_opt = if data.source_path.is_empty() {
        None
    } else {
        Some(data.source_path.as_str())
    };
    // ids
    let ids: Vec<i64> = data.chunks.iter().map(|c| c.id).collect();
    group.create_dataset("id").with_i64_data(&ids);
    // text
    let texts: Vec<String> = data.chunks.iter().map(|c| c.chunk.clone()).collect();
    let (text_raw, text_len) = pack_strings(&texts);
    group
        .create_dataset("text")
        .with_compound_data(string_dtype(text_len), text_raw, n)
        .with_provenance("clawhdf5-migrate", timestamp, source_opt);
    // embeddings - flatten to [N, dim]
    let dim = data.embedding_dim;
    if opts.float16 {
        let f16_data: Vec<u16> = data
            .chunks
            .iter()
            .flat_map(|c| {
                c.embedding
                    .iter()
                    .map(|&v| half::f16::from_f32(v).to_bits())
            })
            .collect();
        let raw: Vec<u8> = f16_data.iter().flat_map(|v| v.to_le_bytes()).collect();
        let f16_dtype = Datatype::FloatingPoint {
            size: 2,
            byte_order: clawhdf5_format::datatype::DatatypeByteOrder::LittleEndian,
            bit_offset: 0,
            bit_precision: 16,
            exponent_location: 10,
            exponent_size: 5,
            mantissa_location: 0,
            mantissa_size: 10,
            exponent_bias: 15,
        };
        let ds = group
            .create_dataset("embeddings")
            .with_compound_data(f16_dtype, raw, n)
            .with_shape(&[n, dim as u64])
            .with_provenance("clawhdf5-migrate", timestamp, source_opt);
        apply_compression(ds, opts);
    } else {
        let flat: Vec<f32> = data
            .chunks
            .iter()
            .flat_map(|c| c.embedding.iter().copied())
            .collect();
        let ds = group
            .create_dataset("embeddings")
            .with_f32_data(&flat)
            .with_shape(&[n, dim as u64])
            .with_provenance("clawhdf5-migrate", timestamp, source_opt);
        apply_compression(ds, opts);
    }
    // source_channel
    let channels: Vec<String> = data
        .chunks
        .iter()
        .map(|c| c.source_channel.clone())
        .collect();
    let (ch_raw, ch_len) = pack_strings(&channels);
    group
        .create_dataset("source_channel")
        .with_compound_data(string_dtype(ch_len), ch_raw, n);
    // timestamp
    let timestamps: Vec<f64> = data.chunks.iter().map(|c| c.timestamp).collect();
    group.create_dataset("timestamp").with_f64_data(&timestamps);
    // session_id
    let sess_ids: Vec<String> = data.chunks.iter().map(|c| c.session_id.clone()).collect();
    let (sid_raw, sid_len) = pack_strings(&sess_ids);
    group
        .create_dataset("session_id")
        .with_compound_data(string_dtype(sid_len), sid_raw, n);
    // tags
    let tags: Vec<String> = data.chunks.iter().map(|c| c.tags.clone()).collect();
    let (tag_raw, tag_len) = pack_strings(&tags);
    group
        .create_dataset("tags")
        .with_compound_data(string_dtype(tag_len), tag_raw, n);
    // deleted
    let deleted: Vec<i32> = data.chunks.iter().map(|c| c.deleted).collect();
    group.create_dataset("deleted").with_i32_data(&deleted);
    builder.add_group(group.finish());
 }
 fn write_sessions_group(builder: &mut FileBuilder, data: &SqliteData) {
    let mut group = builder.create_group("sessions");
    let n = data.sessions.len() as u64;
    group.set_attr("count", AttrValue::I64(n as i64));
    if n == 0 {
        builder.add_group(group.finish());
        return;
    }
    let ids: Vec<String> = data.sessions.iter().map(|s| s.id.clone()).collect();
    let (id_raw, id_len) = pack_strings(&ids);
    group
        .create_dataset("id")
        .with_compound_data(string_dtype(id_len), id_raw, n);
    let start_idxs: Vec<i64> = data.sessions.iter().map(|s| s.start_idx).collect();
    group.create_dataset("start_idx").with_i64_data(&start_idxs);
    let end_idxs: Vec<i64> = data.sessions.iter().map(|s| s.end_idx).collect();
    group.create_dataset("end_idx").with_i64_data(&end_idxs);
    let channels: Vec<String> = data.sessions.iter().map(|s| s.channel.clone()).collect();
    let (ch_raw, ch_len) = pack_strings(&channels);
    group
        .create_dataset("channel")
        .with_compound_data(string_dtype(ch_len), ch_raw, n);
    let timestamps: Vec<f64> = data.sessions.iter().map(|s| s.timestamp).collect();
    group.create_dataset("timestamp").with_f64_data(&timestamps);
    let summaries: Vec<String> = data.sessions.iter().map(|s| s.summary.clone()).collect();
    let (sum_raw, sum_len) = pack_strings(&summaries);
    group
        .create_dataset("summary")
        .with_compound_data(string_dtype(sum_len), sum_raw, n);
    builder.add_group(group.finish());
 }
 fn write_entities_group(builder: &mut FileBuilder, data: &SqliteData) {
    let mut group = builder.create_group("entities");
    let n = data.entities.len() as u64;
    group.set_attr("count", AttrValue::I64(n as i64));
    if n == 0 {
        builder.add_group(group.finish());
        return;
    }
    let ids: Vec<i64> = data.entities.iter().map(|e| e.id).collect();
    group.create_dataset("id").with_i64_data(&ids);
    let names: Vec<String> = data.entities.iter().map(|e| e.name.clone()).collect();
    let (name_raw, name_len) = pack_strings(&names);
    group
        .create_dataset("name")
        .with_compound_data(string_dtype(name_len), name_raw, n);
    let types: Vec<String> = data
        .entities
        .iter()
        .map(|e| e.entity_type.clone())
        .collect();
    let (type_raw, type_len) = pack_strings(&types);
    group
        .create_dataset("type")
        .with_compound_data(string_dtype(type_len), type_raw, n);
    let emb_idxs: Vec<i64> = data.entities.iter().map(|e| e.embedding_idx).collect();
    group
        .create_dataset("embedding_idx")
        .with_i64_data(&emb_idxs);
    builder.add_group(group.finish());
 }
 fn write_relations_group(builder: &mut FileBuilder, data: &SqliteData) {
    let mut group = builder.create_group("relations");
    let n = data.relations.len() as u64;
    group.set_attr("count", AttrValue::I64(n as i64));
    if n == 0 {
        builder.add_group(group.finish());
        return;
    }
    let srcs: Vec<i64> = data.relations.iter().map(|r| r.src).collect();
    group.create_dataset("src").with_i64_data(&srcs);
    let tgts: Vec<i64> = data.relations.iter().map(|r| r.tgt).collect();
    group.create_dataset("tgt").with_i64_data(&tgts);
    let rels: Vec<String> = data.relations.iter().map(|r| r.relation.clone()).collect();
    let (rel_raw, rel_len) = pack_strings(&rels);
    group
        .create_dataset("relation")
        .with_compound_data(string_dtype(rel_len), rel_raw, n);
    let weights: Vec<f64> = data.relations.iter().map(|r| r.weight).collect();
    group.create_dataset("weight").with_f64_data(&weights);
    let timestamps: Vec<f64> = data.relations.iter().map(|r| r.timestamp).collect();
    group.create_dataset("timestamp").with_f64_data(&timestamps);
    builder.add_group(group.finish());
 }
 #[cfg(test)]
 mod time_tests {
    use super::civil_from_days;
    #[test]
    fn epoch_day_zero_is_1970_01_01() {
        assert_eq!(civil_from_days(0), (1970, 1, 1));
    }
    #[test]
    fn known_dates_roundtrip() {
        // 2026-08-16 is 20,681 days after 1970-01-01.
        assert_eq!(civil_from_days(20_681), (2026, 8, 16));
        // 2000-02-29 (leap day itself) and 2000-03-01 (the day after).
        assert_eq!(civil_from_days(11_016), (2000, 2, 29));
        assert_eq!(civil_from_days(11_017), (2000, 3, 1));
    }
    #[test]
    fn iso8601_now_has_expected_shape() {
        let ts = super::iso8601_now();
        assert_eq!(ts.len(), "2026-08-16T00:00:00Z".len());
        assert!(ts.starts_with("20")); // sanity: 21st-century year
        assert!(ts.ends_with('Z'));
    }
 }
@@ -50,12 +50,8 @@ pub struct SqliteData {
    pub sessions: Vec<Session>,
    pub entities: Vec<Entity>,
    pub relations: Vec<Relation>,
    /// `--embedding-dim`, or the first row's; 0 when neither exists.
    pub embedding_dim: usize,
    /// Filesystem path of the SQLite database this data was read from, for
    /// provenance attribution on the HDF5 output. Empty when the data did
    /// not come directly from a SQLite read (e.g. re-read of a prior HDF5
    /// migration output for an incremental merge).
    pub source_path: String,
 }
 /// A table name plus the ordered column names the reader maps by position.
@@ -167,11 +163,13 @@ pub fn read_counts(
    })
 }
-/// Auto-detect embedding dimension from the first chunk's BLOB size.
+/// Auto-detect embedding dimension from the BLOB size of the first chunk (in
 /// id order, deleted or not).
 fn detect_embedding_dim(conn: &Connection, config: &SchemaConfig) -> SqlResult<Option<usize>> {
    let emb_col = config.chunks.columns.get(2).copied().unwrap_or("embedding");
    let id_col = config.chunks.columns.first().copied().unwrap_or("id");
    let mut stmt = conn.prepare(&format!(
-        "SELECT {emb_col} FROM {} LIMIT 1",
+        "SELECT {emb_col} FROM {} ORDER BY {id_col} LIMIT 1",
        config.chunks.table
    ))?;
    let mut rows = stmt.query([])?;
@@ -195,24 +193,16 @@ fn blob_to_f32(blob: &[u8]) -> Vec<f32> {
 /// Read all data from a ZeroClaw SQLite database.
 ///
 /// If `skip_deleted` is true, rows with `deleted=1` are excluded from chunks.
-/// If `embedding_dim` is `None`, auto-detect from the first row.
+/// If `embedding_dim` is `None`, auto-detect from the first row (0 when there
 /// are no rows). Embeddings are returned at their full stored length whatever
 /// the dimension: checking that every row matches it is the writer's job
 /// (`store_writer::write_store`), so a mismatch is an error, not silent
 /// truncation.
 pub fn read_sqlite(
    path: &str,
    skip_deleted: bool,
    embedding_dim: Option<usize>,
    config: &SchemaConfig,
 ) -> Result<SqliteData, Box<dyn std::error::Error>> {
    read_sqlite_filtered(path, skip_deleted, embedding_dim, config, 0)
 }
 /// Like [`read_sqlite`] but only reads chunks whose id is greater than
 /// `min_chunk_id` (0 = all). Used for incremental migration.
 pub fn read_sqlite_filtered(
    path: &str,
    skip_deleted: bool,
    embedding_dim: Option<usize>,
    config: &SchemaConfig,
    min_chunk_id: i64,
 ) -> Result<SqliteData, Box<dyn std::error::Error>> {
    let conn = Connection::open(path)?;
@@ -221,7 +211,7 @@ pub fn read_sqlite_filtered(
        None => detect_embedding_dim(&conn, config)?.unwrap_or(0),
    };
-    let chunks = read_chunks(&conn, skip_deleted, dim, config, min_chunk_id)?;
+    let chunks = read_chunks(&conn, skip_deleted, config)?;
    let sessions = read_sessions(&conn, config)?;
    let entities = read_entities(&conn, config)?;
    let relations = read_relations(&conn, config)?;
@@ -232,42 +222,43 @@ pub fn read_sqlite_filtered(
        entities,
        relations,
        embedding_dim: dim,
        source_path: path.to_owned(),
    })
 }
 fn read_chunks(
    conn: &Connection,
    skip_deleted: bool,
    expected_dim: usize,
    config: &SchemaConfig,
    min_chunk_id: i64,
 ) -> SqlResult<Vec<MemoryChunk>> {
    let id_col = config.chunks.columns.first().copied().unwrap_or("id");
    let deleted_col = config.chunks.columns.get(7).copied().unwrap_or("deleted");
-    let mut conds = Vec::new();
+    let mut where_clause = String::new();
    if skip_deleted {
-        conds.push(format!("{deleted_col} = 0"));
+        where_clause = format!(" WHERE {deleted_col} = 0");
    }
-    if min_chunk_id > 0 {
+    // In id order, so the store's records follow the source's order.
-        conds.push(format!("{id_col} > {min_chunk_id}"));
+    where_clause.push_str(&format!(" ORDER BY {id_col}"));
    }
    let where_clause = if conds.is_empty() {
        String::new()
    } else {
        format!(" WHERE {}", conds.join(" AND "))
    };
    let sql = config.chunks.select(&where_clause);
    let mut stmt = conn.prepare(&sql)?;
    let rows = stmt.query_map([], |row| {
        let blob: Vec<u8> = row.get(2)?;
-        let mut embedding = blob_to_f32(&blob);
+        if !blob.len().is_multiple_of(4) {
-
+            let id: i64 = row.get(0)?;
-        // Validate/truncate to expected dimension
+            return Err(rusqlite::Error::FromSqlConversionFailure(
-        if expected_dim > 0 {
+                2,
-            embedding.truncate(expected_dim);
+                rusqlite::types::Type::Blob,
                format!(
                    "chunk id {id}: embedding BLOB is {} bytes, not a whole number of \
                     little-endian f32 values",
                    blob.len()
                )
                .into(),
            ));
        }
        // Read at full length: rows of the wrong dimension are rejected by
        // the writer, never truncated to fit.
        let embedding = blob_to_f32(&blob);
        Ok(MemoryChunk {
            id: row.get(0)?,
@@ -0,0 +1,407 @@
 //! Write migrated SQLite data into a clawhdf5-agent store.
 //!
 //! Everything goes through `clawhdf5-agent`'s own API — `HDF5Memory::create`
 //! (or `open` for `--incremental`), `save_batch`, `delete_batch`, the session
 //! cache and the knowledge graph — so the result is an ordinary agent store
 //! that `HDF5Memory::open` accepts, not a second hand-built copy of its schema.
 use std::collections::{HashMap, HashSet};
 use std::path::Path;
 use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
 use clawhdf5_format::float16::round_to_f16;
 use crate::sqlite_reader::{MemoryChunk, SqliteData};
 type BoxErr = Box<dyn std::error::Error>;
 /// SQLite timestamps are Unix seconds; the agent's session and relation
 /// timestamps are Unix microseconds (memory records stay in seconds).
 pub const US_PER_SEC: f64 = 1_000_000.0;
 /// Options controlling the output store.
 #[derive(Debug, Clone)]
 pub struct WriteOptions {
    pub agent_id: String,
    pub embedder: String,
    pub compression: bool,
    pub compression_level: u32,
    /// Store full-precision `f32` embeddings instead of the library default
    /// (half precision). Only applies to a newly created store: an existing
    /// store keeps the precision it was created with.
    pub f32: bool,
    /// Add to the store at the output path if there is one, instead of
    /// replacing it.
    pub incremental: bool,
    /// Leave out deleted source rows that are not in the store. (A deleted
    /// row that matches an active store record still tombstones it, so pass
    /// deleted rows in `data` for an incremental run.)
    pub skip_deleted: bool,
 }
 /// What the migration wrote, and where each source row went, so validation
 /// can compare the store with the source row by row.
 #[derive(Debug, Default)]
 pub struct Migration {
    /// Whether the output store existed and was added to (`--incremental`).
    pub appended_to_existing: bool,
    /// The store's embedding precision.
    pub float16: bool,
    pub embedding_dim: usize,
    /// Records in the store after the migration (including tombstones).
    pub store_count: usize,
    /// `(store index, source chunk index)` of every record written.
    pub records: Vec<(usize, usize)>,
    /// Source chunks already in the store (incremental), not written again.
    pub chunks_present: usize,
    /// `(store index, source chunk index)` of records that were active in
    /// the store but whose source row is now deleted (incremental): they were
    /// tombstoned by this run.
    pub deleted_in_store: Vec<(usize, usize)>,
    /// Source rows that were deleted in the store but are active in the
    /// source (incremental): the agent has no un-delete, so each was written
    /// again as a new record (counted in `records` too).
    pub restored: usize,
    /// Deleted source rows left out because of `skip_deleted`.
    pub deleted_skipped: usize,
    /// `(store session index, source session index)` of each session written.
    pub sessions: Vec<(usize, usize)>,
    pub sessions_present: usize,
    /// `(store entity id, source entity index)` of each entity written.
    pub entities: Vec<(u64, usize)>,
    pub entities_present: usize,
    /// SQLite entity id -> store entity id, for every source entity.
    pub entity_ids: HashMap<i64, u64>,
    /// `(store relation index, source relation index)` of each relation written.
    pub relations: Vec<(usize, usize)>,
    pub relations_present: usize,
    /// Source relations naming an entity id that is not in the entities
    /// table; the knowledge graph cannot hold them, so they are skipped.
    pub dangling_relations: Vec<usize>,
    /// Messages of the write-anomaly alerts the agent raised while importing
    /// (informational; they never block a save — a bulk import typically
    /// trips the write-rate check).
    pub anomaly_alerts: Vec<String>,
 }
 /// Identity of a memory record for incremental de-duplication: every field
 /// the agent stores except the embedding (whose stored form depends on the
 /// store's precision).
 type RecordKey = (String, String, String, String, u64);
 fn record_key(
    chunk: &str,
    source_channel: &str,
    session_id: &str,
    tags: &str,
    ts: f64,
 ) -> RecordKey {
    (
        chunk.to_owned(),
        source_channel.to_owned(),
        session_id.to_owned(),
        tags.to_owned(),
        ts.to_bits(),
    )
 }
 /// Reject rows the agent would otherwise store differently from the source,
 /// or not at all: an embedding of a different length from the store's
 /// dimension (the agent pads/truncates silently), an empty embedding, or, in
 /// a float16 store, a value beyond the half-precision range.
 ///
 /// Every source row is checked, including ones that end up not being written
 /// (already in the store, or deleted and skipped): the source must be
 /// consistent as a whole, and the check runs before the store is touched.
 fn check_chunks(chunks: &[MemoryChunk], dim: usize, float16: bool) -> Result<(), BoxErr> {
    for c in chunks {
        if c.embedding.is_empty() {
            return Err(format!(
                "chunk id {}: the embedding is empty; an agent store needs an embedding \
                 for every record",
                c.id
            )
            .into());
        }
        if c.embedding.len() != dim {
            return Err(format!(
                "chunk id {}: embedding has {} values, expected {dim}; every row must have \
                 the store's dimension (detected from the first row unless --embedding-dim \
                 is given), and rows are never truncated or padded to fit",
                c.id,
                c.embedding.len()
            )
            .into());
        }
        if float16
            && let Some((k, v)) = c
                .embedding
                .iter()
                .enumerate()
                .find(|&(_, &v)| v.is_finite() && round_to_f16(v).is_infinite())
        {
            return Err(format!(
                "chunk id {}: embedding[{k}] = {v} is outside the half-precision range \
                 (±65504) of a float16 store; migrate with --f32",
                c.id
            )
            .into());
        }
    }
    Ok(())
 }
 /// Migrate `data` into the agent store at `path`.
 ///
 /// Without `opts.incremental` (or when nothing exists at `path`) a new store
 /// is created, replacing any file there — but only once every source row has
 /// passed [`check_chunks`], so a source that cannot be migrated leaves an
 /// existing store untouched. With it, the existing store is opened and only
 /// source rows it does not already hold are added: memory records are
 /// matched on their content, sessions on their id, entities on name and
 /// type, relations on (source, target, relation). A matched record then
 /// takes the source row's deleted flag: see [`Migration::deleted_in_store`]
 /// and [`Migration::restored`].
 pub fn write_store(
    path: &Path,
    data: &SqliteData,
    opts: &WriteOptions,
 ) -> Result<Migration, BoxErr> {
    let existing = opts.incremental && path.exists();
    let mut mem = if existing {
        // `open` does not modify the store beyond what the agent itself does
        // on open; the checks below run before anything is written.
        let mem = HDF5Memory::open(path)?;
        let dim = mem.config().embedding_dim;
        // `data.embedding_dim` is 0 only for a source with no records and no
        // --embedding-dim, which has no dimension to disagree with.
        if data.embedding_dim != 0 && dim != data.embedding_dim {
            let hint = if dim == 0 {
                " (a store created from a source with no memory records; re-create it \
                 with --embedding-dim)"
            } else {
                ""
            };
            return Err(format!(
                "the store at {} has embedding_dim {dim}{hint}, the source {}; \
                 embeddings of a different dimension cannot be added to it",
                path.display(),
                data.embedding_dim
            )
            .into());
        }
        check_chunks(&data.chunks, dim, mem.config().float16)?;
        mem
    } else {
        // (With records, a dimension of 0 means an empty first embedding,
        // which `check_chunks` reports more precisely.)
        if data.embedding_dim == 0 && data.chunks.is_empty() {
            return Err(
                "the source has no memory records to detect the embedding dimension \
                        from; pass --embedding-dim (the dimension of the agent's embedder), \
                        or the store could never hold a record"
                    .into(),
            );
        }
        let mut config = MemoryConfig::new(path.to_path_buf(), &opts.agent_id, data.embedding_dim);
        config.embedder = opts.embedder.clone();
        config.compression = opts.compression;
        config.compression_level = opts.compression_level;
        // Only ever switch the library default off (as `clawhdf5-cli create`).
        if opts.f32 {
            config.float16 = false;
        }
        // Before `create`, which replaces whatever is at `path`.
        check_chunks(&data.chunks, config.embedding_dim, config.float16)?;
        HDF5Memory::create(config)?
    };
    let float16 = mem.config().float16;
    let dim = mem.config().embedding_dim;
    let mut m = Migration {
        appended_to_existing: existing,
        float16,
        embedding_dim: dim,
        ..Migration::default()
    };
    // ---- Memory records --------------------------------------------------
    // Store indices of every record the store already holds, by content, so
    // a source row that appears twice is only treated as present as often
    // as the store has it.
    let mut present: HashMap<RecordKey, Vec<usize>> = HashMap::new();
    if existing {
        let c = &mem.cache;
        for i in 0..c.len() {
            let key = record_key(
                &c.chunks[i],
                &c.source_channels[i],
                &c.session_ids[i],
                &c.tags[i],
                c.timestamps[i],
            );
            present.entry(key).or_default().push(i);
        }
    }
    let key_of = |c: &MemoryChunk| {
        record_key(
            &c.chunk,
            &c.source_channel,
            &c.session_id,
            &c.tags,
            c.timestamp,
        )
    };
    let tombstoned = |idx: usize| mem.cache.tombstones[idx] != 0;
    // Pass 1: a store record in the same deleted state as the source row.
    let mut unmatched: Vec<usize> = Vec::new();
    for (i, c) in data.chunks.iter().enumerate() {
        let src_deleted = c.deleted != 0;
        let hit = present.get_mut(&key_of(c)).and_then(|idxs| {
            let at = idxs.iter().position(|&x| tombstoned(x) == src_deleted)?;
            Some(idxs.remove(at))
        });
        match hit {
            Some(_) => m.chunks_present += 1,
            None => unmatched.push(i),
        }
    }
    // Pass 2: a store record whose deleted state differs — the source row
    // was deleted or restored since the last migration. The source wins.
    let mut new_chunks: Vec<usize> = Vec::with_capacity(unmatched.len());
    let mut delete_in_store: Vec<usize> = Vec::new();
    for i in unmatched {
        let c = &data.chunks[i];
        let hit = present
            .get_mut(&key_of(c))
            .and_then(|idxs| (!idxs.is_empty()).then(|| idxs.remove(0)));
        match hit {
            // Active in the store, deleted in the source: tombstone it.
            Some(idx) if c.deleted != 0 => {
                m.deleted_in_store.push((idx, i));
                delete_in_store.push(idx);
            }
            // Deleted in the store, active in the source. The agent has no
            // un-delete, so the row is written again as a new active record
            // (the tombstone stays until the store is compacted).
            Some(_) => {
                m.restored += 1;
                new_chunks.push(i);
            }
            None if c.deleted != 0 && opts.skip_deleted => m.deleted_skipped += 1,
            None => new_chunks.push(i),
        }
    }
    new_chunks.sort_unstable();
    let to_write: Vec<&MemoryChunk> = new_chunks.iter().map(|&i| &data.chunks[i]).collect();
    // ---- Sessions (in the cache; persisted by the save_batch checkpoint) ---
    let known_sessions: HashSet<String> = mem
        .sessions()
        .entries
        .iter()
        .map(|e| e.id.clone())
        .collect();
    for (i, s) in data.sessions.iter().enumerate() {
        if known_sessions.contains(&s.id) {
            m.sessions_present += 1;
            continue;
        }
        let sessions = mem.sessions_mut();
        let at = sessions.len();
        sessions.add_at(
            &s.id,
            s.start_idx.max(0) as usize,
            s.end_idx.max(0) as usize,
            &s.channel,
            &s.summary,
            s.timestamp * US_PER_SEC,
        );
        m.sessions.push((at, i));
    }
    // ---- Knowledge graph -------------------------------------------------
    let kg = mem.knowledge_mut();
    // Matched only against what the store held before this run: the source
    // itself is copied as it is, duplicates included.
    let by_name_type: HashMap<(String, String), u64> = kg
        .entities
        .iter()
        .map(|e| ((e.name.clone(), e.entity_type.clone()), e.id))
        .collect();
    for (i, e) in data.entities.iter().enumerate() {
        let key = (e.name.clone(), e.entity_type.clone());
        let id = match by_name_type.get(&key) {
            Some(&id) => {
                m.entities_present += 1;
                id
            }
            None => {
                let id = kg.add_entity(&e.name, &e.entity_type, e.embedding_idx);
                m.entities.push((id, i));
                id
            }
        };
        m.entity_ids.insert(e.id, id);
    }
    let known_relations: HashSet<(u64, u64, String)> = kg
        .relations
        .iter()
        .map(|r| (r.src, r.tgt, r.relation.clone()))
        .collect();
    for (i, r) in data.relations.iter().enumerate() {
        let (Some(&src), Some(&tgt)) = (m.entity_ids.get(&r.src), m.entity_ids.get(&r.tgt)) else {
            m.dangling_relations.push(i);
            continue;
        };
        if known_relations.contains(&(src, tgt, r.relation.clone())) {
            m.relations_present += 1;
            continue;
        }
        let at = kg.relations.len();
        kg.add_relation(src, tgt, &r.relation, r.weight as f32);
        kg.relations[at].ts = r.timestamp * US_PER_SEC;
        m.relations.push((at, i));
    }
    // ---- Write: one checkpoint for records, sessions and graph -----------
    let entries: Vec<MemoryEntry> = to_write
        .iter()
        .map(|c| MemoryEntry {
            chunk: c.chunk.clone(),
            embedding: c.embedding.clone(),
            source_channel: c.source_channel.clone(),
            timestamp: c.timestamp,
            session_id: c.session_id.clone(),
            tags: c.tags.clone(),
        })
        .collect();
    let indices = mem.save_batch(entries)?;
    m.records = indices
        .iter()
        .copied()
        .zip(new_chunks.iter().copied())
        .collect();
    // Rows deleted in the source stay deleted: tombstones, as the agent's own
    // `delete` leaves them (not compacted away).
    // Records matched in the store whose source row has since been deleted
    // are tombstoned too.
    let tombstones: Vec<usize> = m
        .records
        .iter()
        .filter(|&&(_, src)| data.chunks[src].deleted != 0)
        .map(|&(idx, _)| idx)
        .chain(delete_in_store)
        .collect();
    mem.delete_batch(&tombstones)?;
    m.anomaly_alerts = mem
        .take_anomaly_alerts()
        .into_iter()
        .map(|a| a.message)
        .collect();
    m.store_count = mem.count();
    drop(mem); // release the single-writer lock before anyone re-opens it
    Ok(m)
 }
@@ -1,192 +1,266 @@
-use clawhdf5::reader::File as Hdf5File;
+//! Validate a migration by reading the store back the way an agent would:
-use clawhdf5_format::provenance::VerifyResult;
+//! through `HDF5Memory::open_read_only`, comparing what it loads with the
 //! SQLite source, and running a search for a migrated record.
 use std::path::Path;
 use clawhdf5_agent::{AgentMemory, HDF5Memory, SearchOptions};
 use clawhdf5_format::float16::round_to_f16;
 use crate::hdf5_reader::read_hdf5;
 use crate::sqlite_reader::SqliteData;
 use crate::store_writer::{Migration, US_PER_SEC};
 type BoxErr = Box<dyn std::error::Error>;
 /// Summary of a migration validation.
 #[derive(Debug)]
 pub struct ValidationSummary {
-    pub chunks: u64,
+    /// Records in the store (including tombstones).
-    pub sessions: u64,
+    pub count: usize,
-    pub entities: u64,
+    /// Records in the store that are not deleted.
-    pub relations: u64,
+    pub active: usize,
-    pub embedding_dim: u64,
+    pub sessions: usize,
-    /// Number of rows whose full content was compared against the source.
+    pub entities: usize,
    pub relations: usize,
    pub embedding_dim: usize,
    pub float16: bool,
    /// Rows whose full content was compared against the source.
    pub rows_checked: u64,
-    /// Whether the `chunks/text` and `chunks/embeddings` SHINES provenance
+    /// Whether a search for a migrated record found it (`false` when there
-    /// hashes (written via [`crate::hdf5_writer`]) were both present and
+    /// was no active migrated record with an embedding to search for).
-    /// matched their recomputed SHA-256 on read-back. `false` when either
+    pub search_checked: bool,
    /// dataset has no provenance metadata (e.g. an older output file) or
    /// there are zero chunks to check.
    pub provenance_verified: bool,
 }
-/// Validate a migrated HDF5 file against the source data.
+/// Validate the store at `path` against the source rows `migration` wrote.
 ///
-/// Reads the written file back and compares actual content — chunk text,
+/// Counts and the session / entity / relation rows are always checked in
-/// embeddings, and every session/entity/relation field — to the source, not
+/// full. Memory records are content-checked on a representative sample, or
-/// just the row counts. When `full` is false a representative sample of chunk
+/// all of them with `full`. Embeddings must match exactly: the source values
-/// rows is content-checked (counts and all other groups are always checked in
+/// themselves in an `f32` store, their [`round_to_f16`] in a `float16` one.
-/// full); when `full` is true every chunk row is compared too. `float16` widens
+pub fn validate_store(
-/// the embedding tolerance to allow for half-precision quantization.
+    path: &Path,
 pub fn validate_hdf5(
    path: &str,
    source: &SqliteData,
    migration: &Migration,
    full: bool,
    float16: bool,
 ) -> Result<ValidationSummary, BoxErr> {
-    let got = read_hdf5(path)?;
+    let mut mem = HDF5Memory::open_read_only(path)?;
-    let provenance_verified = verify_chunk_provenance(path)?;
+    let float16 = mem.config().float16;
    let dim = mem.config().embedding_dim;
    // ---- Counts ----
-    check_count("chunk", got.chunks.len(), source.chunks.len())?;
+    check_count("record", mem.count(), migration.store_count)?;
-    check_count("session", got.sessions.len(), source.sessions.len())?;
+    if float16 != migration.float16 {
    check_count("entity", got.entities.len(), source.entities.len())?;
    check_count("relation", got.relations.len(), source.relations.len())?;
    if got.embedding_dim != source.embedding_dim {
        return Err(format!(
-            "embedding_dim mismatch: HDF5 has {}, source has {}",
+            "float16 mismatch: store {float16}, expected {}",
-            got.embedding_dim, source.embedding_dim
+            migration.float16
        )
        .into());
    }
    if dim != migration.embedding_dim {
        return Err(format!(
            "embedding_dim mismatch: store has {dim}, expected {}",
            migration.embedding_dim
        )
        .into());
    }
    if !migration.appended_to_existing {
        check_count("record", mem.count(), migration.records.len())?;
        check_count("session", mem.sessions().len(), migration.sessions.len())?;
        check_count(
            "entity",
            mem.knowledge().entities.len(),
            migration.entities.len(),
        )?;
        check_count(
            "relation",
            mem.knowledge().relations.len(),
            migration.relations.len(),
        )?;
    }
-    // ---- Chunk content (sampled or full) ----
+    // ---- Memory records (sampled or full) ----
    let (emb_abs, emb_rel) = if float16 { (1e-2, 1e-2) } else { (1e-4, 0.0) };
    let mut rows_checked = 0u64;
-    for i in sample_indices(source.chunks.len(), full) {
+    let expected_value = |v: f32| if float16 { round_to_f16(v) } else { v };
-        let (s, g) = (&source.chunks[i], &got.chunks[i]);
+    for k in sample_indices(migration.records.len(), full) {
-        if s.id != g.id {
+        let (idx, src) = migration.records[k];
-            return Err(field_err("chunk", i, "id", s.id, g.id));
+        let s = &source.chunks[src];
        let c = &mem.cache;
        if idx >= c.len() {
            return Err(
                format!("record {idx} (chunk id {}) is missing from the store", s.id).into(),
            );
        }
-        if s.chunk != g.chunk {
+        let id = s.id;
        if c.chunks[idx] != s.chunk {
            return Err(format!(
-                "chunk[{i}].text mismatch: source {:?}, HDF5 {:?}",
+                "record {idx} (chunk id {id}) text mismatch: source {:?}, store {:?}",
                truncate(&s.chunk),
-                truncate(&g.chunk)
+                truncate(&c.chunks[idx])
            )
            .into());
        }
-        if s.session_id != g.session_id || s.source_channel != g.source_channel || s.tags != g.tags
+        if c.source_channels[idx] != s.source_channel
            || c.session_ids[idx] != s.session_id
            || c.tags[idx] != s.tags
        {
-            return Err(format!("chunk[{i}] string field mismatch").into());
+            return Err(format!("record {idx} (chunk id {id}) string field mismatch").into());
        }
-        if s.deleted != g.deleted {
+        if c.timestamps[idx].to_bits() != s.timestamp.to_bits() {
            return Err(field_err("chunk", i, "deleted", s.deleted, g.deleted));
        }
        if s.embedding.len() != g.embedding.len() {
            return Err(format!(
-                "chunk[{i}] embedding length mismatch: {} vs {}",
+                "record {idx} (chunk id {id}) timestamp mismatch: source {}, store {}",
-                s.embedding.len(),
+                s.timestamp, c.timestamps[idx]
                g.embedding.len()
            )
            .into());
        }
-        for (k, (&a, &b)) in s.embedding.iter().zip(g.embedding.iter()).enumerate() {
+        let deleted = c.tombstones[idx] != 0;
-            if (a - b).abs() > emb_abs + emb_rel * a.abs() {
+        if deleted != (s.deleted != 0) {
-                return Err(
+            return Err(format!(
-                    format!("chunk[{i}].embedding[{k}] mismatch: source {a}, HDF5 {b}").into(),
+                "record {idx} (chunk id {id}) deleted mismatch: source {}, store {deleted}",
-                );
+                s.deleted != 0
            )
            .into());
        }
        let got = c.embeddings.get(idx).unwrap_or(&[]);
        if got.len() != s.embedding.len() {
            return Err(format!(
                "record {idx} (chunk id {id}) embedding length mismatch: source {}, store {}",
                s.embedding.len(),
                got.len()
            )
            .into());
        }
        for (j, (&a, &b)) in s.embedding.iter().zip(got).enumerate() {
            let want = expected_value(a);
            if want.to_bits() != b.to_bits() && !(want.is_nan() && b.is_nan()) {
                return Err(format!(
                    "record {idx} (chunk id {id}) embedding[{j}] mismatch: source {a}, \
                     expected {want}, store {b}"
                )
                .into());
            }
        }
        rows_checked += 1;
    }
-    // ---- Other groups (always full — they are small) ----
+    // ---- Records tombstoned because their source row was deleted ----
-    for (i, (s, g)) in source.sessions.iter().zip(got.sessions.iter()).enumerate() {
+    for &(idx, src) in &migration.deleted_in_store {
-        if s.id != g.id
+        let s = &source.chunks[src];
-            || s.start_idx != g.start_idx
+        let c = &mem.cache;
-            || s.end_idx != g.end_idx
+        if idx >= c.len() || c.chunks[idx] != s.chunk || c.timestamps[idx] != s.timestamp {
-            || s.channel != g.channel
+            return Err(format!("record {idx} (chunk id {}) mismatch or missing", s.id).into());
            || s.summary != g.summary
        {
            return Err(format!("session[{i}] mismatch").into());
        }
-        rows_checked += 1;
+        if c.tombstones[idx] == 0 {
-    }
+            return Err(format!(
-    for (i, (s, g)) in source.entities.iter().zip(got.entities.iter()).enumerate() {
+                "record {idx} (chunk id {}) is deleted in the source but active in the store",
-        if s.id != g.id
+                s.id
-            || s.name != g.name
+            )
-            || s.entity_type != g.entity_type
+            .into());
            || s.embedding_idx != g.embedding_idx
        {
            return Err(format!("entity[{i}] mismatch").into());
        }
        rows_checked += 1;
    }
    for (i, (s, g)) in source
        .relations
        .iter()
        .zip(got.relations.iter())
        .enumerate()
    {
        if s.src != g.src || s.tgt != g.tgt || s.relation != g.relation {
            return Err(format!("relation[{i}] mismatch").into());
        }
        rows_checked += 1;
    }
    // ---- Sessions ----
    let sessions = mem.sessions();
    for &(at, src) in &migration.sessions {
        let s = &source.sessions[src];
        let (Some(e), Some(summary)) = (sessions.entries.get(at), sessions.summaries.get(at))
        else {
            return Err(format!("session {:?} is missing from the store", s.id).into());
        };
        if e.id != s.id
            || e.start_idx != s.start_idx.max(0) as u64
            || e.end_idx != s.end_idx.max(0) as u64
            || e.channel != s.channel
            || *summary != s.summary
            || e.ts != s.timestamp * US_PER_SEC
        {
            return Err(format!("session {:?} mismatch", s.id).into());
        }
        rows_checked += 1;
    }
    // ---- Knowledge graph ----
    let kg = mem.knowledge();
    for &(id, src) in &migration.entities {
        let s = &source.entities[src];
        let Some(e) = kg.get_entity(id) else {
            return Err(format!(
                "entity {:?} (id {}) is missing from the store",
                s.name, s.id
            )
            .into());
        };
        if e.name != s.name || e.entity_type != s.entity_type || e.embedding_idx != s.embedding_idx
        {
            return Err(format!("entity {:?} (id {}) mismatch", s.name, s.id).into());
        }
        rows_checked += 1;
    }
    for &(at, src) in &migration.relations {
        let s = &source.relations[src];
        let r = kg.relations.get(at);
        let ok = r.is_some_and(|r| {
            Some(&r.src) == migration.entity_ids.get(&s.src)
                && Some(&r.tgt) == migration.entity_ids.get(&s.tgt)
                && r.relation == s.relation
                && r.weight == s.weight as f32
                && r.ts == s.timestamp * US_PER_SEC
        });
        if !ok {
            return Err(format!(
                "relation {} -[{}]-> {} mismatch or missing",
                s.src, s.relation, s.tgt
            )
            .into());
        }
        rows_checked += 1;
    }
    // ---- A migrated record must be findable by search ----
    let probe = migration
        .records
        .iter()
        .copied()
        .find(|&(idx, _)| dim > 0 && mem.cache.tombstones[idx] == 0);
    let search_checked = match probe {
        None => false,
        Some((idx, _)) => {
            let query = mem.cache.embeddings[idx].to_vec();
            let text = mem.cache.chunks[idx].clone();
            let hits = mem.search(&query, &text, &SearchOptions::new(10));
            // A record with the same text is as good a hit: the source may
            // hold duplicates, and they tie.
            if !hits.iter().any(|h| h.index == idx || h.chunk == text) {
                return Err(format!(
                    "search for migrated record {idx} ({:?}) did not return it",
                    truncate(&text)
                )
                .into());
            }
            true
        }
    };
    Ok(ValidationSummary {
-        chunks: got.chunks.len() as u64,
+        count: mem.count(),
-        sessions: got.sessions.len() as u64,
+        active: mem.count_active(),
-        entities: got.entities.len() as u64,
+        sessions: mem.sessions().len(),
-        relations: got.relations.len() as u64,
+        entities: mem.knowledge().entities.len(),
-        embedding_dim: got.embedding_dim as u64,
+        relations: mem.knowledge().relations.len(),
        embedding_dim: dim,
        float16,
        rows_checked,
-        provenance_verified,
+        search_checked,
    })
 }
 fn check_count(kind: &str, got: usize, expected: usize) -> Result<(), BoxErr> {
    if got != expected {
-        return Err(format!("{kind} count mismatch: HDF5 has {got}, source has {expected}").into());
+        return Err(format!("{kind} count mismatch: store has {got}, expected {expected}").into());
    }
    Ok(())
 }
 /// Re-verify the SHA-256 provenance hash of `chunks/text` and
 /// `chunks/embeddings` against their actual stored bytes, catching
 /// post-write corruption that a plain content comparison against the
 /// in-memory source wouldn't (the source is compared against what
 /// `read_hdf5` decoded, not against the raw bytes on disk).
 ///
 /// Returns `Ok(true)` only if both datasets exist and both hashes match.
 /// Returns `Ok(false)` (not an error) if a dataset has no provenance
 /// attributes at all (e.g. a file written before this check existed) or
 /// there are zero chunks. Returns an error only on an actual hash mismatch —
 /// that indicates real corruption.
 fn verify_chunk_provenance(path: &str) -> Result<bool, BoxErr> {
    let file = Hdf5File::open(path)?;
    let Ok(chunks) = file.group("chunks") else {
        return Ok(false);
    };
    let mut all_present = true;
    for name in ["text", "embeddings"] {
        let Ok(ds) = chunks.dataset(name) else {
            all_present = false;
            continue;
        };
        match ds.verify_provenance()? {
            VerifyResult::Ok => {}
            VerifyResult::NoHash => all_present = false,
            VerifyResult::Mismatch { stored, computed } => {
                return Err(format!(
                    "provenance hash mismatch on chunks/{name}: stored {stored}, recomputed {computed} — data may be corrupted"
                )
                .into());
            }
        }
    }
    Ok(all_present)
 }
 fn field_err<T: std::fmt::Display>(kind: &str, i: usize, field: &str, s: T, g: T) -> BoxErr {
    format!("{kind}[{i}].{field} mismatch: source {s}, HDF5 {g}").into()
 }
 fn truncate(s: &str) -> String {
    if s.len() <= 40 {
        s.to_string()
@@ -196,7 +270,7 @@ fn truncate(s: &str) -> String {
    }
 }
-/// Indices of chunk rows to content-check. Full = all; otherwise a spread of
+/// Indices of records to content-check. Full = all; otherwise a spread of
 /// representative rows (first/last and evenly-spaced interior samples).
 fn sample_indices(n: usize, full: bool) -> Vec<usize> {
    if n == 0 {