diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 1e4b98d..fdccb50 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -2,20 +2,40 @@ > Pure Rust. Zero C dependencies. Single file. Fast enough to forget it's there. -**System:** Intel i7-12650H (10C/16T, 4.7 GHz boost) · 32 GB DDR5 · Linux 6.8.0 +**System (original run):** Intel i7-12650H (10C/16T, 4.7 GHz boost) · 32 GB DDR5 · Linux 6.8.0 **Rust:** 1.96.0-nightly (2026-03-14) · `--release` profile **Date:** 2026-07-01 -> **Traceability note:** the "h5bench-Equivalent I/O Benchmarks" and both -> "Independent Validation: tank" sections below meet a dated, -> hardware-cited, reproducible standard (explicit date, machine spec, and a -> runnable command per result) — this now covers "LongMemEval Results", -> "SIMD & Parallelism", "Vector Search Latency", and "Comparison to MemX" via -> their tank re-runs. The remaining undated sections above (Hybrid Search, -> Knowledge Graph, Memory Consolidation, Temporal Index, Write Path, Decision -> Gate, Memory Strategy, Multi-Session Benchmark, Memory Footprint, -> Consolidation Efficiency, Ephemeral Tier) do not yet meet that bar — this is -> a known, tracked documentation gap, not a claim that those numbers are wrong. +Sections that carry their own "Measured …" line were measured there instead, +not on the system above. The 2026-09-24 re-run was on tank (AMD Ryzen 7 +7800X3D, 8C/16T, Linux 7.0.0-34-generic, rustc 1.98.1) at commit 5c8323c, +serially, with no `--warm-up-time`/`--measurement-time` overrides, waiting +before each command while the 1-minute load average was above 2.0. The +`bench` targets ran at Criterion's defaults (3 s warm-up, 5 s target +measurement time, 100 samples). Every `memory_bench` group that was run +(`vector_search_latency`, `hybrid_search`, `knowledge_graph`, +`consolidation`, `temporal`) and the multi-modal group set +`sample_size(50)`, so those sections took 50 samples, not 100. The 5 s is a +target: Criterion stretched it where 5 s could not hold the samples it needed +(for example 8.5 s for `vector_search_latency/bench_cosine_search/100k`, +15.5 s for the 1K consolidation cycle, 252 s for `tick_session_10k`). + +> **Traceability note:** a section meets the dated, hardware-cited, +> reproducible standard when it gives an explicit date, the machine, and a +> runnable command for its results. As of the 2026-09-24 re-run on tank that +> covers Vector Search Latency, Comparison to MemX, SIMD & Parallelism, Hybrid +> Search, Knowledge Graph, Memory Consolidation, Temporal Index, Write Path, +> Decision Gate, Memory Strategy, Summary (derived from those), LongMemEval +> Results (the BM25 rows), Multi-Session Benchmark, Memory Footprint (on disk +> and in memory), Consolidation Efficiency, Ephemeral Tier, Multi-modal Search, +> the Search and Read harnesses, and the "h5bench-Equivalent I/O Benchmarks" +> and "Independent Validation: tank" sections. What does not yet meet that bar: +> the LongMemEval rows that need real embeddings (not re-run here, except the +> dated float16 comparison), the Consolidation Efficiency 100K cycle row and +> memory-reduction part (the 2026-09-24 run was stopped before it produced +> them), the int8 side of "Quantising the index copy" (not re-run), and the +> i7-12650H and macOS M3 Max rows under Cross-Platform Notes. That is a +> known, tracked documentation gap, not a claim that those numbers are wrong. > > **Correctness note (2026-08-06).** Being dated and reproducible is necessary but > not sufficient — a number can be perfectly reproducible and still measure the @@ -57,6 +77,23 @@ What remains at 2.43x: the flat vectors (1.0x), the HNSW index's own copy of them (1.0x), and text, ids and graph (~0.4x). The index copy is the next target — it is what a quantised or borrowed representation would address. +**Current figures (`f32` index).** Measured 2026-09-24 on tank (AMD Ryzen 7 +7800X3D), commit 5c8323c: + +> **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --footprint --full` + +| N | vectors (raw) | entries MiB | store MiB | indexes MiB | reopened MiB | peak during open MiB | reopened / raw | +|---:|---:|---:|---:|---:|---:|---:|---:| +| 1 000 | 1 | 2 | 0 | 2 | 4 | 5 | 2.40x | +| 10 000 | 15 | 17 | 12 | 32 | 44 | 61 | 3.03x | +| 100 000 | 146 | 172 | 60 | 266 | 399 | 562 | 2.72x | + +These are the "reopened, f32" figures in the next table, reproduced exactly. +They are above the "after" column of the table above (44 MiB vs 35 MiB at 10K, +399 vs 357 MiB at 100K): the 2.43x was measured for 2e7e045, and the store +already measured 399 MiB (2.72x) when the int8 index landed later the same +day (c0a9206). The git log does not say what changed in between. + ### Quantising the index copy (`quantized_index`) `MemoryConfig::quantized_index` stores the index's copy as `i8` instead of @@ -108,6 +145,20 @@ second**, builds **1.8x faster**, and holds a quarter of the vectors. (Compare only at equal `ef`: with re-scoring the harness raises `ef` to at least the candidate pool, so the `ef = 16` and `ef = 32` rows are not like-for-like.) +**Re-check, 2026-09-24.** Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), +commit 5c8323c. The `f32` memory columns of the first table reproduce exactly +(see [Memory footprint](#memory-footprint)). Only the `f32` side of the speed +table was re-run, once, not as alternating medians: + +> **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --full` + +At N = 100 000, `ef = 64`, the `f32` index measured recall@10 0.9945 (same), +19 001 QPS (was 13 399) and a 2700.8 ms build (was 3197 ms). The QPS figure +moved by more than 20%; the log does not explain it, and a single run is not +the same-moment paired comparison the table above is. The int8 columns were +not re-run, so the 1.63x ratio has not been re-checked against the new `f32` +figure. + #### On ARM (Raspberry Pi 5, Cortex-A76) `dot_i8` has two aarch64 kernels: `SDOT` for CPUs with the ARMv8.2 @@ -268,6 +319,10 @@ high-water mark alongside the retained figure: | 10 000 | 44 | 61 | | 100 000 | 399 | 562 | +Reproduced exactly on 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit +5c8323c, by `search_harness -- --footprint --full` (table under +[Memory footprint](#memory-footprint)). + The peak is set *after* the parse, by the index build, so a buffer allocated and freed during the parse never reaches the high-water mark. Holding a deliberate extra copy of the file across the whole parse leaves the peak @@ -359,6 +414,50 @@ contiguous dataset is converted straight from the file bytes (one copy instead of two), and the native-endian conversions no longer zero a buffer they are about to overwrite. +### Current: read harness (2026-09-24) + +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. + +> **Run:** `cargo run --release -p clawhdf5-bench --bin read_harness` + +4096 x 2048 f64 (64 MB per dataset), chunks 256 x 256, file 129 MB + +| layout | read | selected | time ms | MB/s of selection | vs full read | +|---|---|---:|---:|---:|---:| +| chunked + deflate | full (first) | 64 MB | 65.6 | 975 | | +| chunked + deflate | full (repeat) | 64 MB | 63.2 | 1013 | 1.00x | +| chunked + deflate | 64 x 64 window (1 chunk) | 0.03 MB | 0.18 | 177 | 0.003x | +| chunked + deflate | 512 x 512 window (4-9 chunks) | 2.00 MB | 4.12 | 485 | 0.063x | +| chunked + deflate | one row | 0.02 MB | 0.99 | 16 | 0.015x | +| chunked + deflate | one column | 0.03 MB | 2.02 | 15 | 0.031x | +| chunked | full (first) | 64 MB | 60.5 | 1057 | | +| chunked | full (repeat) | 64 MB | 56.7 | 1129 | 1.00x | +| chunked | 64 x 64 window (1 chunk) | 0.03 MB | 0.10 | 325 | 0.002x | +| chunked | 512 x 512 window (4-9 chunks) | 2.00 MB | 3.19 | 628 | 0.053x | +| chunked | one row | 0.02 MB | 0.05 | 284 | 0.001x | +| chunked | one column | 0.03 MB | 0.49 | 64 | 0.008x | +| contiguous | full (first) | 64 MB | 30.1 | 2124 | | +| contiguous | full (repeat) | 64 MB | 26.1 | 2452 | 1.00x | +| contiguous | 64 x 64 window (1 chunk) | 0.03 MB | 0.08 | 394 | 0.003x | +| contiguous | 512 x 512 window (4-9 chunks) | 2.00 MB | 3.36 | 595 | 0.112x | +| contiguous | one row | 0.02 MB | 0.03 | 486 | 0.001x | +| contiguous | one column | 0.03 MB | 2.61 | 12 | 0.087x | + +Full reads match the "after" column above (chunked + deflate 69 ms then, 63.2 +to 65.6 ms now). The compressed selections are faster than in "After: partial +reads" (64 x 64 window 0.39 -> 0.18 ms, one row 2.69 -> 0.99 ms, one column +5.23 -> 2.02 ms) and match the 2026-09-23 figures in +[Deflate backend](#deflate-backend-zlib-rs-vs-zlib-ng) (0.18, 1.00 and 2.01 +ms), which were taken after the zlib-rs switch and the one-shot codec calls +described there; this run does not isolate which change accounts for it. + +The uncompressed windows went the other way. Against "After: partial reads", +the 512 x 512 window is about 60% slower (chunked 1.99 -> 3.19 ms, contiguous +2.12 -> 3.36 ms) and the chunked 64 x 64 window went from 0.06 to 0.10 ms. +The rows and columns of the uncompressed layouts are within 20% (chunked +column 0.45 -> 0.49 ms, contiguous column 2.55 -> 2.61 ms). This run does not +explain the slower windows. + ## Search harness baseline (v2.3.0) Produced by `cargo run --release -p clawhdf5-bench --bin search_harness -- --full` @@ -596,68 +695,217 @@ Recall on clustered data is unchanged or slightly better (100K, `ef = 64`: 0.474 -> 0.444), the cost of batch members not seeing each other while planning; batches are capped at 1/16 of the graph and 512 nodes. +### Current: search harness (2026-09-24) + +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, default +features (so the build is batched and `parallel`, 16 threads). The HNSW tables +use an `f32` index. The end-to-end table uses `MemoryConfig::new`'s defaults, +which at this commit are `float16` embeddings and the int8 index; every +earlier end-to-end table in this section was an `f32` store with an `f32` +index. + +> **Run:** `cargo run --release -p clawhdf5-bench --bin search_harness -- --full` + +#### HNSW, N = 1000, dim = 384, M = 16, ef_construction = 64, storage = Float32 + +build: 14.7 ms (67954 vectors/s) · exact scan: 4403 QPS, p50 225 µs + +| ef | recall@10 | QPS | p50 µs | p99 µs | +|---:|---:|---:|---:|---:| +| 16 | 0.9940 | 134065 | 7 | 18 | +| 32 | 1.0000 | 105134 | 9 | 21 | +| 64 | 1.0000 | 77541 | 13 | 26 | +| 128 | 1.0000 | 36938 | 26 | 35 | +| 256 | 1.0000 | 19249 | 52 | 63 | + +#### HNSW, N = 10000, dim = 384, M = 16, ef_construction = 64, storage = Float32 + +build: 135.6 ms (73736 vectors/s) · exact scan: 435 QPS, p50 2295 µs + +| ef | recall@10 | QPS | p50 µs | p99 µs | +|---:|---:|---:|---:|---:| +| 16 | 0.9975 | 79614 | 12 | 24 | +| 32 | 1.0000 | 70556 | 14 | 21 | +| 64 | 1.0000 | 51313 | 19 | 29 | +| 128 | 1.0000 | 22846 | 44 | 57 | +| 256 | 1.0000 | 10267 | 97 | 119 | + +#### HNSW, N = 100000, dim = 384, M = 16, ef_construction = 64, storage = Float32 + +build: 2700.8 ms (37027 vectors/s) · exact scan: 43 QPS, p50 23433 µs + +| ef | recall@10 | QPS | p50 µs | p99 µs | +|---:|---:|---:|---:|---:| +| 16 | 0.9180 | 20043 | 47 | 104 | +| 32 | 0.9775 | 21348 | 44 | 104 | +| 64 | 0.9945 | 19001 | 51 | 101 | +| 128 | 0.9995 | 13458 | 73 | 137 | +| 256 | 0.9995 | 5128 | 196 | 282 | + +#### End to end: `HDF5Memory::hybrid_search` (k = 10, weights 0.7 / 0.3) + +| N | ingest ms | cold index build ms | checkpoint ms | open ms | first query after open ms | p50 ms | p99 ms | QPS | +|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| 1000 | 15 | 14 | 4.4 | 1.9 | 1.5 | 0.07 | 0.08 | 13869.5 | +| 10000 | 146 | 126 | 33.5 | 18.3 | 14.8 | 0.49 | 0.50 | 2044.1 | +| 100000 | 1498 | 1833 | 495.7 | 259.9 | 155.1 | 4.69 | 5.22 | 212.7 | + +Steady-state query latency is where the v2.4.0 work left it (p50 0.07 / 0.49 / +4.69 ms against 0.07 / 0.49 / 4.65 ms). Figures that moved by more than 20%: + +- **Build.** 14.7 / 135.6 / 2700.8 ms against 34 / 388 / 5.9 s for + "batched + `parallel`" above. The git log does not explain the difference. +- **Cold index build** in the end-to-end table: 14 / 126 / 1833 ms, down from + 112 / 1487 / 20285 ms in the "unranked keyword scores" table, which predates + the batched and parallel build described above. +- **QPS at 100K, `ef = 64`**: 19 001 against 13 980 in the "unit-vector" + table, with recall 0.9945 against 0.984; the recall gain is the one the + batched build reported. The QPS gain is not explained by the log. +- **Other HNSW QPS cells**, against the same "unit-vector" tables: 10K + `ef = 16` 79 614 against 54 608 (+46%); 100K `ef = 16` 20 043 against + 15 139 (+32%), `ef = 128` 13 458 against 10 959 (+23%), `ef = 256` 5 128 + against 3 731 (+37%). The other cells, and every 1K cell, are within 20%. + The log does not explain these either. +- **Open** in the end-to-end table: 1.9 / 18.3 / 259.9 ms against 1.1 / 13.7 + / 353.1 ms in the "unranked keyword scores" table, slower at 1K and 10K and + faster at 100K. This run does not isolate the cause. +- **Checkpoint at 100K**: 495.7 ms against 728.9 ms, and ingest up 11 -> 15, + 104 -> 146 ms at 1K / 10K. This store is `float16` with an int8 index, the + earlier ones `f32`; the [float16 study](#float16-embedding-storage-memoryconfigfloat16) + measured float16 checkpoints at 512 ms against 752 ms for `f32` at 100K. + This run does not isolate the ingest change. + ## Vector Search Latency Brute-force cosine similarity over 384-dimensional embeddings (OpenAI text-embedding-3-small size). +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. + +> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(vector_search_1k|simd_cosine_10k|simd_cosine_100k|prenorm_search_10k|ivf_search_10k_nprobe10|ivf_search_100k_nprobe10|ivf_pq_search_100k|rairs_search_10k_nprobe10)$'` + | Scale | Flat Search | Pre-norm | IVF (nprobe=10) | IVF-PQ | RAIRS | |-------|-------------|----------|-----------------|--------|-------| -| **1K** | 54 µs | 62 µs | — | — | — | -| **10K** | 753 µs | 706 µs | 27 µs | — | 159 µs | -| **100K** | 11.4 ms | — | 1.32 ms | 1.19 ms | — | +| **1K** | 47.4 µs | — | — | — | — | +| **10K** | 500.5 µs | 316.5 µs | 24.8 µs | — | 124.0 µs | +| **100K** | 6.58 ms | — | 592 µs | 869 µs | — | -**Key insight:** At 10K records (typical agent memory), IVF search delivers **27 µs** — that's 26x faster than flat search. Even at 100K records, IVF-PQ keeps search under **1.2 ms**. +The 1K pre-norm cell is blank because no 1K pre-norm benchmark exists (the +old 62 µs has no source). `memory_bench`'s own cosine search +(`cargo bench -p clawhdf5-agent --bench memory_bench -- '^vector_search_latency/'`) +agrees: 47.6 µs / 505.1 µs / 6.67 ms at 1K / 10K / 100K. + +Most cells moved by more than 20% from the previous table (10K flat 753 µs, +pre-norm 706 µs, RAIRS 159 µs; 100K flat 11.4 ms, IVF 1.32 ms, IVF-PQ 1.19 ms). +Those were undated figures from the original i7-12650H run. The 2026-08-05 +tank run below is within 4% of the new numbers in every cell but one (1K flat +47.8 µs, 10K flat 501 µs, pre-norm 322 µs, IVF 24.8 µs; 100K flat 6.60 ms, +IVF 608 µs, IVF-PQ 865 µs), so for those cells the difference is the machine. +The exception is RAIRS: 109 µs on 2026-08-05, 124.0 µs now (+14%); this run +does not explain that change. + +**Key insight:** at 10K records (typical agent memory), IVF search takes +**24.8 µs** against 500.5 µs for the flat scan. At 100K, IVF answers in +**592 µs** and IVF-PQ in 869 µs — both under 1 ms. (RAIRS, 124.0 µs at 10K, is +slower than plain IVF here.) ### Comparison to MemX (arxiv:2603.16171) MemX claims end-to-end search under 90ms at 100K records (Rust + libSQL + FTS5). +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. + +> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(simd_cosine_100k|ivf_pq_search_100k|bm25_search_10k)$'` + > **Caveat — not like-for-like.** MemX's `<90 ms` is *end-to-end* search across their > full pipeline (dense embeddings + FTS5 + four-factor re-ranking). The clawhdf5 > figures below are a *single component* — raw vector search latency, excluding > embedding, keyword, fusion, and re-ranking stages. A component measured against a -> full pipeline will always look favourable; the "speedup" column overstates the real -> advantage by an unquantified margin and should be read as an order-of-magnitude -> indication only, not a benchmark result. Matching MemX's measurement boundary is -> tracked as follow-up work. +> full pipeline will always look favourable, so the two columns below cannot be +> divided into a speedup: any such ratio would overstate the real advantage by an +> unquantified margin. Read the table as an order-of-magnitude indication only, not a +> benchmark result. Matching MemX's measurement boundary is tracked as follow-up work. -| Metric | MemX (claimed, end-to-end) | ClawhDF5 (component only) | Ratio | -|--------|----------------------------|---------------------------|-------| -| 100K flat search | <90 ms | 11.4 ms | ~8x | -| 100K IVF-PQ search | — | 1.19 ms | ~76x | -| Keyword search 10K | 1,100x improvement over unindexed | 583 µs (BM25) | Comparable | +| Metric | MemX (claimed, end-to-end) | ClawhDF5 (component only) | +|--------|----------------------------|---------------------------| +| 100K flat search | <90 ms | 6.64 ms | +| 100K IVF-PQ search | — | 876 µs | +| Keyword search 10K | 1,100x improvement over unindexed | 205 µs (BM25) | + +The ratio column is gone: dividing an end-to-end claim by a component +timing produced a number that looked like a result and was not one. The +previous figures (100K flat 11.4 ms, IVF-PQ 1.19 ms, BM25 10K 583 µs) were +undated, from the original i7-12650H run. BM25 at 10K also moved on this +machine: 520 µs on 2026-08-05, 205 µs now; v2.4.0 changed +`BM25Index::search` in between (bounded-heap top-k, the unused WAND bound +removed, IDF per query), but this run does not isolate its effect. --- ## SIMD & Parallelism -384-dimensional cosine similarity at 10K scale. +384-dimensional cosine similarity at 10K scale, using the `strategy_*` +benchmarks, which hold the dataset fixed and vary only `SearchStrategy` +(the 2026-08-05 re-run below explains why the older table's benchmarks were +not an apples-to-apples comparison). -| Strategy | Latency | vs Sequential | -|----------|---------|---------------| -| Sequential (scalar) | 1.07 ms | 1.0x | -| SIMD (auto-vectorized) | 545 µs | **2.0x** | -| Rayon (parallel) | 553 µs | **1.9x** | -| Adaptive (auto-select) | 564 µs | **1.9x** | +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. Rayon +uses all 16 threads; the 1-minute load average was 4.19 when this command +finished, most likely from the Rayon benchmarks themselves (not verified). + +> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(strategy_scalar_10k|strategy_simd_10k|strategy_rayon_10k|adaptive_search_10k|simd_cosine_100k|rayon_cosine_100k)$'` -At 100K: | Strategy | Latency | |----------|---------| -| SIMD | 13.7 ms | -| Rayon parallel | 8.3 ms | +| Sequential (scalar), `strategy_scalar_10k` | 501.1 µs | +| SIMD, `strategy_simd_10k` | 319.9 µs | +| Rayon (parallel), `strategy_rayon_10k` | **79.0 µs** | +| Adaptive (auto-select), `adaptive_search_10k` | 317.9 µs | + +At 100K (no `strategy_*` benchmark exists at this size, so these are the +`simd_cosine_100k` / `rayon_cosine_100k` benchmarks): + +| Strategy | Latency | +|----------|---------| +| SIMD | 6.56 ms | +| Rayon parallel | 4.63 ms | + +The Rayon 10K figure moved furthest: 323 µs on this machine on 2026-08-05, +79.0 µs now. The git log does not explain it. The other rows are close to +that run (scalar 502 µs, SIMD 327 µs, adaptive 339 µs; 100K 6.60 / 4.73 ms). Against the original, undated i7-12650H table (scalar 1.07 ms, SIMD +545 µs, Rayon 553 µs, adaptive 564 µs; 100K 13.7 / 8.3 ms) every figure is +lower; that table also used the mislabelled benchmarks. The older names +measured in the same session, for reference (`cargo bench -p clawhdf5-agent +--bench bench -- '^(sequential_cosine_10k|simd_cosine_10k|rayon_cosine_10k|adaptive_search_10k|simd_cosine_100k|rayon_cosine_100k)$'`): +`sequential_cosine_10k` 317.8 µs, `simd_cosine_10k` 505.2 µs, +`rayon_cosine_10k` 274.3 µs — "sequential" faster than "SIMD", which is the +mislabelling the 2026-08-05 re-run found. --- ## Hybrid Search (Vector + BM25) -1K records, 384-dimensional embeddings with BM25 keyword index. +1K records, 384-dimensional embeddings with BM25 keyword index. These are the +flat free functions `hybrid::hybrid_search` / `rrf_hybrid_search` (linear +scan), not the HNSW-backed `HDF5Memory::hybrid_search`, which is measured in +the [search harness](#current-search-harness-2026-09-24). + +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. + +> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^hybrid_search/'` +> and `cargo bench -p clawhdf5-agent --bench bench -- '^(bm25_search_1k|hybrid_search_10k)$'` | Method | Latency | Notes | |--------|---------|-------| -| Weighted fusion | 198 µs | Original min-max normalization | -| **RRF (k=60)** | **222 µs** | Reciprocal Rank Fusion — better quality, ~12% overhead | -| BM25-only 1K | 67 µs | Keyword search alone | -| Hybrid 10K | 2.04 ms | Full hybrid at 10K scale | +| Weighted fusion | 106.8 µs | Min-max normalization | +| **RRF (k=60)** | **130.5 µs** | Reciprocal Rank Fusion | +| BM25-only 1K | 20.4 µs | Keyword search alone | +| Hybrid 10K | 1.10 ms | Full hybrid at 10K scale | + +Every row moved by more than 20% from the previous table (198 µs, 222 µs, +67 µs, 2.04 ms), which was undated and from the original i7-12650H run. The +cause is not isolated. (The old "better quality" note on RRF is dropped: on +LongMemEval RRF measured worse than the tuned weighted sum; see +[Fusion method](#fusion-method--weighted-vs-rrf-full-haystack-n500).) --- @@ -665,16 +913,36 @@ At 100K: Graph traversal and entity operations. +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. + +> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^knowledge_graph/'` +> and `cargo bench -p clawhdf5-agent --bench bench -- '^alias_resolve_(short|long)_query$'` +> (the `knowledge_graph/` rows re-run after the adjacency-index fix, same day, +> load average 0.49 at start) + | Operation | Scale | Latency | |-----------|-------|---------| -| BFS traversal | 100 entities | 5.4 µs | -| BFS traversal | 1,000 entities | 24 µs | -| Spreading activation | 100 entities | 16.9 µs | -| Entity resolution (Levenshtein) | 100 entities | 64 µs | -| Alias resolution (short query) | 100 aliases | 10.4 µs | -| Alias resolution (long query) | 100 aliases | 11.6 µs | +| BFS traversal | 100 entities | 5.23 µs | +| BFS traversal | 1,000 entities | 23.1 µs | +| Spreading activation | 100 entities | 10.1 µs | +| Entity resolution (Levenshtein) | 100 entities | 48.6 µs | +| Alias resolution (short query) | 100 aliases | 7.67 µs | +| Alias resolution (long query) | 100 aliases | 8.18 µs | -**All graph operations complete in microseconds.** Spreading activation across 100 entities with 5 propagation steps finishes in 17 µs. +The traversal rows were measured after a fix made during this re-run. The +first measurement of the day found **BFS 6.5x slower** than the old table +said: 17.5 µs and 155.1 µs, against 5.4 µs and 24 µs, with spreading +activation at 22.8 µs against 16.9 µs. The cause was 1efd82c (2026-08-17), +which built an adjacency index over the whole graph on every traversal — so a +2-hop BFS paid to index every entity and relation first. The index is now +cached on `KnowledgeCache` and checked against a fingerprint of the graph on +each use (one pass over entity ids and relation endpoints, no allocation), so +any change to the graph, including direct edits of its public `Vec`s, still +triggers a rebuild. These benches traverse an unchanged graph, which is the +cached case; the first traversal after a change pays for one build. Entity +and alias resolution do not use the index (the 49.8 µs first-run figure for +entity resolution is within noise of the 48.6 µs here). The old figures were +undated, from the original i7-12650H run. --- @@ -682,13 +950,22 @@ Graph traversal and entity operations. Hippocampal-inspired tiered memory management. +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. + +> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^consolidation/'` + | Operation | Scale | Latency | |-----------|-------|---------| -| Consolidation cycle | 100 records | 15 µs | -| Consolidation cycle | 1,000 records | 164 µs | -| Importance scoring | 100 records | 25 µs | +| Consolidation cycle | 100 records | 8.08 µs | +| Consolidation cycle | 1,000 records | 115.2 µs | +| Importance scoring | 100 records | 31.0 µs | -A full consolidation pass over 1,000 memories (eviction + promotion across Working → Episodic → Semantic) completes in **164 µs**. This can run on every memory write without perceptible latency. +A full consolidation pass over 1,000 memories (eviction + promotion across Working → Episodic → Semantic) completes in **115 µs**. This can run on every memory write without perceptible latency. + +The cycle figures were 15 µs and 164 µs, and importance scoring 25 µs, all +undated from the original i7-12650H run. Eviction's membership check became a +`HashSet` lookup (603fcf8) and `add_memory` stopped cloning Working-tier +records (d787ac0), both on 2026-08-17; this run does not isolate their effect. --- @@ -696,12 +973,17 @@ A full consolidation pass over 1,000 memories (eviction + promotion across Worki Sorted timestamp index with binary search. +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. + +> **Run:** `cargo bench -p clawhdf5-agent --bench memory_bench -- '^temporal/'` + | Operation | Scale | Latency | |-----------|-------|---------| -| Range query | 10K timestamps | **716 ns** | -| Batch insert | 10K timestamps | 4.69 ms | +| Range query | 10K timestamps | **622 ns** | +| Batch insert | 10K timestamps | 3.22 ms | -Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K records: **716 nanoseconds.** +Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K records: **622 nanoseconds.** +Batch insert was 4.69 ms in the previous, undated i7-12650H figure. --- @@ -709,16 +991,50 @@ Sub-microsecond temporal queries. "What happened between 3pm and 5pm?" over 10K HDF5 persistence with optional Write-Ahead Log. +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. The stores +use `MemoryConfig::new`'s defaults at this commit: `float16` embeddings and +the int8 index. + +> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^(save_without_wal_single|save_with_wal_single|save_batch_100|save_batch_1000|save_wal_1k_existing|wal_flush_100_entries|tick_session_1k|tick_session_10k)$'` + | Operation | Latency | Notes | |-----------|---------|-------| -| Single save (no WAL) | 61 µs | Direct HDF5 write (owned-Vec IO path) | -| Single save (with WAL) | 18 µs | WAL group-commit append; HDF5 write batched at flush | -| Batch 100 | 723 µs | 7.2 µs per record | -| Batch 1,000 | 6.17 ms | 6.2 µs per record | -| WAL save (1K existing) | 539 µs | Incremental append | -| WAL flush 100 entries | 787 µs | Merge WAL → HDF5 | -| Session tick 1K | 5.76 ms | Full session maintenance | -| Session tick 10K | 89.8 ms | Background operation | +| Single save (no WAL) | 88.3 µs | Direct HDF5 write (owned-Vec IO path) | +| Single save (with WAL) | 26.1 µs | WAL group-commit append; HDF5 write batched at flush | +| Batch 100 | 1.08 ms | | +| Batch 1,000 | 31.7 ms | | +| WAL save (1K existing) | 282.8 µs | Incremental append | +| WAL flush 100 entries | 530.1 µs | Merge WAL → HDF5 | +| Session tick 1K | 3.83 ms | Full session maintenance | +| Session tick 10K | 35.5 ms | Background operation | + +Almost every row moved by more than 20% from the previous table, in both +directions. Slower: single save 61 -> 88.3 µs (no WAL) and 18 -> 26.1 µs +(WAL), batch 100 723 µs -> 1.08 ms, and **batch 1,000 6.17 -> 31.7 ms**. +Faster: WAL save into 1K existing 539 -> 282.8 µs, WAL flush 787 -> 530.1 µs, +session tick 5.76 -> 3.83 ms (1K) and 89.8 -> 35.5 ms (10K). The two single-save +figures were last updated on 2026-07-01 and the rest are undated, all on the +i7-12650H with `f32` stores. + +**Isolated afterwards, same machine, same day** (`--warm-up-time 1 +--measurement-time 3`, the store settings switched through the bench's +config): + +| | f16 + int8 (default) | f32 + int8 | f16 + f32 index | f32 + f32 index | v2.3.0 (f32) | +|---|---:|---:|---:|---:|---:| +| Single save (WAL) | 26.1 µs | 24.3 µs | 26.1 µs | 24.4 µs | 24.3 µs | +| Single save (no WAL) | 89.0 µs | 87.3 µs | 88.8 µs | 87.4 µs | 82.6 µs | +| Batch 100 | 1.09 ms | 0.90 ms | 1.09 ms | 0.90 ms | 0.87 ms | +| Batch 1,000 | 31.5 ms | 31.1 ms | 31.5 ms | 31.6 ms | 30.3 ms | + +The int8 index costs nothing on the write path. `float16` costs ~2 µs per +saved record — the rounding — which is the whole difference at 100 records +and within noise at 1,000. Built at the v2.3.0 tag (v2.2.0's benches do not +compile) and run on this machine, the write path measures the same as today, +so nothing regressed since then; the old 18 µs and 6.17 ms figures cannot be +reproduced on this hardware and came from an undated run on another machine. +`save_batch` scales linearly (about 12 µs per record from 500 to 8 000 +records); the bench's figure also includes dropping the store. --- @@ -726,12 +1042,16 @@ HDF5 persistence with optional Write-Ahead Log. Trivial/non-trivial classification for memory write filtering. +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. + +> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^gate_'` + | Check | Latency | |-------|---------| -| Trivial skip ("ok", "yes") | 61 ns | -| Short phrase skip | 86 ns | -| Non-trivial pass | 705 ns | -| Ratio check | 488 ns | +| Trivial skip ("ok", "yes") | 57.6 ns | +| Short phrase skip | 83.0 ns | +| Non-trivial pass | 570.5 ns | +| Ratio check | 395.2 ns | **Sub-microsecond filtering.** The gate decides whether to save a memory in under 1 µs. @@ -741,27 +1061,70 @@ Trivial/non-trivial classification for memory write filtering. End-to-end strategy evaluation including embedding operations. +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. + +> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^strategy_(save_every|semantic_shift)'` + | Strategy | Condition | Latency | |----------|-----------|---------| -| SaveEveryExchange (substantive) | Saves | 923 ns | -| SaveEveryExchange (trivial) | Skips | 67 ns | -| SaveOnSemanticShift (empty store) | Saves | 941 ns | +| SaveEveryExchange (substantive) | Saves | 732.5 ns | +| SaveEveryExchange (trivial) | Skips | 58.3 ns | +| SaveOnSemanticShift (empty store) | Saves | 734.9 ns | + +The two "Saves" rows were 923 ns and 941 ns in the previous table, undated +and from the original i7-12650H run. + +--- + +## Multi-modal Search + +`clawhdf5_agent::multimodal::MultiModalStore`. Each of N records carries two +384-dim embeddings: a text embedding of its caption and one of its primary +modality, cycling image / audio / video. `search_cross_modal` scores every +embedding of every record (2N vectors) and keeps each record's best; +`search_by_modality(Image, …)` scores only image embeddings (a third of the +records). Both are exact linear scans followed by a full sort; k = 10. Data is +from a fixed-seed generator, so every run sees the same corpus. + +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c (with the +new, uncommitted `crates/clawhdf5-agent/benches/multimodal_bench.rs` in the +working tree). Criterion, 50 samples. + +> **Run:** `cargo bench -p clawhdf5-agent --bench multimodal_bench` + +| Search | 1K records | 10K records | +|--------|-----------:|------------:| +| Cross-modal (all modalities, 2N vectors) | 842.0 µs | 8.44 ms | +| One modality (image, ~N/3 vectors) | 151.2 µs | 1.54 ms | + +Both scale linearly with N. Cross-modal search is the slow path: at 1K +records it takes 842.0 µs, where the flat single-vector cosine scan in +[Vector Search Latency](#vector-search-latency) takes 47.4 µs over 1K vectors, +and it has no index — at 10K it is 8.44 ms, slower than a 100K flat scan +(6.58 ms). No earlier figure exists for this section. --- ## Summary +Derived from the 2026-09-24 tables above (tank, AMD Ryzen 7 7800X3D, commit +5c8323c); no command of its own. + | Capability | Typical Latency | Scale | |------------|----------------|-------| -| **Full memory search** | <1 ms | 10K records | -| **Hybrid vector+keyword** | <200 µs | 1K records | -| **Knowledge graph query** | <25 µs | 1K entities | -| **Temporal range query** | <1 µs | 10K timestamps | -| **Memory write** | <20 µs | Per record (WAL group-commit append) | -| **Consolidation cycle** | <165 µs | 1K records | -| **Importance gate** | <1 µs | Per record | +| **Full memory search** (`HDF5Memory::hybrid_search`, p50) | 0.49 ms | 10K records | +| **Hybrid vector+keyword** (flat, weighted) | 106.8 µs | 1K records | +| **Knowledge graph query** (BFS) | 23.1 µs | 1K entities | +| **Temporal range query** | 622 ns | 10K timestamps | +| **Memory write** | 26.1 µs | Per record (WAL group-commit append) | +| **Consolidation cycle** | 115.2 µs | 1K records | +| **Importance gate** | 57.6–570.5 ns | Per record | -**The entire memory pipeline — search, retrieve, re-rank, filter — runs in single-digit milliseconds at agent-typical scales. Fast enough that memory becomes invisible infrastructure.** +The previous summary said "<25 µs" for a knowledge-graph query and "<20 µs" +for a memory write. The first holds again after the adjacency-index fix +(23.1 µs); the second does not (26.1 µs). + +**At agent-typical scales a full `HDF5Memory::hybrid_search` (vector + BM25 + fusion) over 10K records takes 0.49 ms at p50, and graph traversal, temporal queries, consolidation and the write gate each take well under a millisecond. At 100K records the hybrid search takes 4.69 ms.** --- @@ -796,9 +1159,21 @@ _Latency benchmarks generated with Criterion.rs (50-100 samples per benchmark). **Mode:** BM25-only retrieval — zero embeddings, `vector_weight=0.0`, `keyword_weight=1.0` -> **Run:** `cargo run --release --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s_cleaned.json` -> (~70 s for all 500 questions on the tank reference machine). Omit the path for the -> oracle variant; add `--limit N` for an evenly-strided subsample. +> **Run:** `cargo run --release -p clawhdf5-bench --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s.json` +> Omit the path for the oracle variant; add `--limit N` for an evenly-strided +> subsample. Without the `embeddings` feature this runs two modes, BM25 only +> and BM25 stemmed. + +The BM25-only and BM25-stemmed figures in this section were measured again on +2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, with the command +above. Every figure of those two modes that this document publishes was +reproduced exactly: BM25 Hit@1/5/10 and MRR at turn and session level, the +BM25 per-type session Hit@1 values quoted in prose below, and the stemmed +turn-level row plus its session Hit@1 in the tokenizer table. The run also +produced figures this document does not publish (stemmed session Hit@5, +Hit@10 and MRR, and per-type Hit@5/Hit@10/MRR for both modes), so there was +nothing to compare them with. Rows that need real embeddings (vector-only, +hybrid, RRF, re-ranking, the weight sweep) were not re-run. ### Full haystack — `longmemeval_s`, n=500 (the number to cite) @@ -1058,14 +1433,29 @@ extraction over the haystack, which this harness does not do. ### Search Latency (LongMemEval, n=500 queries) -| Metric | Latency | -|--------|---------| -| avg | 1,004 µs | -| p50 | 1,017 µs | -| p95 | 2,031 µs | -| p99 | 2,912 µs | +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c, full +`longmemeval_s` haystack (47.7 sessions / 494 turns per question). -Sub-millisecond median search across variable-length chat histories. +> **Run:** `cargo run --release -p clawhdf5-bench --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s.json` + +| Metric | BM25 only | BM25 stemmed | +|--------|-----------|--------------| +| avg | 11,034.7 µs | 11,096.2 µs | +| p50 | 10,943.5 µs | 10,996.5 µs | +| p95 | 11,749.6 µs | 12,018.8 µs | +| p99 | 12,644.1 µs | 12,784.7 µs | + +About 11 ms per query. Each question gets its own store, and the timed call is +the first `hybrid_search_with` on that store. The first query on a store +builds its indexes (the "cold index build" column of the search harness), so +this is a first-query figure, not steady-state search; for that +see the [search harness](#current-search-harness-2026-09-24). + +The previous table (avg 1,004 µs, p50 1,017 µs) said "sub-millisecond median +search", which no longer holds. That figure was undated and predates the +full-haystack harness (added 2026-08-07, 7d6e269), so it was measured on the +much smaller `longmemeval_oracle` corpus; on that corpus tank measured avg +2,431 µs on 2026-08-05 (below). --- @@ -1075,23 +1465,34 @@ Sub-millisecond median search across variable-length chat histories. **Topics:** Personal info, food preferences, music, travel, work/schedule, hobbies **Queries:** 35 questions across 4 types -> **Run:** `cargo run --release --bin memory_arena` +> **Run:** `cargo run --release -p clawhdf5-bench --bin memory_arena` + +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. ### Results by Query Type | Query Type | N | Hit@1 | Hit@5 | Hit@10 | MRR | Avg Latency | |------------|---|-------|-------|--------|-----|-------------| -| single-session | 25 | 40.0% | 92.0% | 100.0% | 0.5788 | 7,853 µs | -| multi-session | 5 | 40.0% | 60.0% | 80.0% | 0.5333 | 7,887 µs | -| temporal | 3 | 33.3% | 66.7% | 66.7% | 0.5000 | 7,899 µs | -| knowledge-update | 2 | 0.0% | 50.0% | 50.0% | 0.2500 | 7,899 µs | -| **OVERALL** | **35** | **37.1%** | **82.9%** | **91.4%** | **0.5468** | **7,870 µs** | +| single-session | 25 | 40.0% | 92.0% | 100.0% | 0.5755 | 248.4 µs | +| multi-session | 5 | 40.0% | 60.0% | 80.0% | 0.5333 | 24.4 µs | +| temporal | 3 | 33.3% | 66.7% | 66.7% | 0.5000 | 28.0 µs | +| knowledge-update | 2 | 0.0% | 50.0% | 50.0% | 0.2500 | 38.7 µs | +| **OVERALL** | **35** | **37.1%** | **82.9%** | **91.4%** | **0.5444** | p50 **30.1 µs**, p95 49.8 µs | **Key findings:** - Hit@10 of 91.4% across all query types with BM25-only (no embeddings) - Single-session recall strongest at 100% Hit@10 - Knowledge-update hardest (requires temporal disambiguation) — would improve significantly with vector similarity -- Latency dominated by BM25 index build over 999 turns (~7.9 ms) +- Search takes about 30 µs at p50 over the 999 turns. All 35 queries run + against one store, and single-session's higher average (248.4 µs) includes + that store's first query, which builds its indexes. + +The previous table put every query at ~7.9 ms ("latency dominated by BM25 +index build over 999 turns"). That was before v2.4.0, when +`HDF5Memory::hybrid_search` rebuilt the BM25 index from scratch and rewrote +the store on every query; it now keeps the index for the life of the store. +The MRR moved slightly (single-session 0.5788 -> 0.5755, overall 0.5468 -> +0.5444); the Hit@k figures are unchanged. --- @@ -1099,46 +1500,89 @@ Sub-millisecond median search across variable-length chat histories. HDF5 file size at various record counts — 384-dimensional embeddings, 200-char text. -> **Run:** `cargo run --release --bin footprint_bench` +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. -### Uncompressed (no WAL) +> **Run:** `cargo run --release -p clawhdf5-bench --bin footprint_bench` -| Records | File Size | Raw Data | Bytes/Record | Throughput | -|---------|-----------|----------|--------------|------------| -| 100 | 176.4 KB | 169.5 KB | 1.8 KB | 100,000 rec/s | -| 1K | 1.7 MB | 1.7 MB | 1.8 KB | 109,643 rec/s | -| 10K | 17.0 MB | 16.6 MB | 1.7 KB | 118,100 rec/s | -| 50K | 85.0 MB | 82.8 MB | 1.7 KB | 110,723 rec/s | -| 100K | 169.8 MB | 165.6 MB | 1.7 KB | 111,422 rec/s | +**These are `float16` stores.** `footprint_bench` builds its stores with +`MemoryConfig::new`, which defaults to `float16` embeddings since 5c8323c, +although the binary still prints "Embedding: 384-dim f32". Its "Raw Data" +column, and so the ratios, count embeddings as `f32` (1,536 bytes per record) +plus text. The synthetic embeddings take only 1,000 distinct values, and each +record's text is fixed by `record_idx % 40` (a 40-word bank walked from that +start), so a store holds only 40 distinct texts. Both compress far better than +real data would; read the gzip figures as an upper bound, and see the next +paragraph for why the other tables are optimistic too. "MB" and "KB" are the +binary's units (powers of 1024). -**1.7 KB per record** — HDF5 overhead is near-zero. Ingestion throughput exceeds **100K records/sec**. +**No table here is uncompressed.** The store always deflates its string +datasets: `write_string_dataset` applies deflate level 6 to any string dataset +of 4 KiB or more, whatever `MemoryConfig::compression` says +(`crates/clawhdf5-agent/src/schema.rs`, `STRING_COMPRESS_THRESHOLD`, added in +4fa7e89 on 2026-06-03). With only 40 distinct texts, the chunk text costs +almost nothing on disk: at 10K records, 50-char text takes 804 B per record +and 1000-char text 944 B, so 950 more characters cost only 140 B. The +`float16` embeddings alone are 768 B per record, and 200 bytes of raw text +would bring that to about 968 B before any other field or overhead; the +measured 803–829 B is below that only because the text is compressed away. +Real text will not deflate like this, so expect more than 820 B per record +with real data. `MemoryConfig::compression` decides only whether the +embeddings are compressed; the gzip table below is that setting turned on. + +### `compression` off (no WAL) + +| Records | File Size | Raw Data (f32) | Bytes/Record | Throughput | +|---------|-----------|----------------|--------------|------------| +| 100 | 90.0 KB | 169.5 KB | 921 B | 75,914 rec/s | +| 1K | 810.4 KB | 1.7 MB | 829 B | 72,266 rec/s | +| 10K | 7.8 MB | 16.6 MB | 820 B | 74,509 rec/s | +| 50K | 38.3 MB | 82.8 MB | 803 B | 72,829 rec/s | +| 100K | 76.7 MB | 165.6 MB | 803 B | 71,102 rec/s | + +**About 820 B per record** at 10K, for this synthetic corpus (see above: the +text is deflated regardless and has only 40 distinct values, so real data +takes more). Ingestion runs at 71,102–75,914 records/sec. ### With Gzip Compression (level 6) -| Records | Compressed | Ratio | Bytes/Record | -|---------|------------|-------|--------------| -| 100 | 31.5 KB | 5.37x | 323 B | -| 1K | 277.1 KB | 6.12x | 283 B | -| 10K | 2.7 MB | 6.17x | 281 B | -| 50K | 13.4 MB | 6.17x | 281 B | -| 100K | 26.9 MB | 6.15x | 282 B | +| Records | Compressed | Ratio vs f32 raw | Bytes/Record | +|---------|------------|------------------|--------------| +| 100 | 17.2 KB | 9.85x | 176 B | +| 1K | 56.4 KB | 30.06x | 57 B | +| 10K | 471.3 KB | 35.97x | 48 B | +| 50K | 2.3 MB | 36.67x | 47 B | +| 100K | 4.5 MB | 36.73x | 47 B | -**6.2x compression ratio** — 100K agent memories in 27 MB compressed. - -### Text Length Comparison (10K records, no compression) +### Text Length Comparison (10K records, `compression` off) | Text Length | File Size | Bytes/Record | Throughput | |-------------|-----------|--------------|------------| -| short (50 chars) | 15.6 MB | 1.6 KB | 177,925 rec/s | -| medium (200 chars) | 17.0 MB | 1.7 KB | 172,152 rec/s | -| long (1000 chars) | 24.6 MB | 2.5 KB | 157,807 rec/s | +| short (50 chars) | 7.7 MB | 804 B | 97,615 rec/s | +| medium (200 chars) | 7.8 MB | 820 B | 89,597 rec/s | +| long (1000 chars) | 9.0 MB | 944 B | 51,462 rec/s | ### WAL Overhead (1K records) | Mode | File Size | Ingest Time | Overhead | |------|-----------|-------------|----------| -| No WAL | 1.7 MB | 5.7 ms | — | -| With WAL | 1.7 MB + 9 B WAL | 5.3 ms | ±8% (negligible) | +| No WAL | 810.4 KB | 10.8 ms | — | +| With WAL | 810.4 KB + 9 B WAL | 10.6 ms | −2% (negligible) | + +**Compared with the previous tables.** Those were undated `f32` stores from the +original i7-12650H run: 1.7 KB per record (169.8 MB at 100K), gzip 6.2x +(26.9 MB at 100K), and more than 100,000 records/sec. Two changes separate +those tables from these, not one. `float16` embeddings save 768 B per record. +The old tables (recorded by 2026-05-14) also predate 4fa7e89 (2026-06-03), +which began deflating string datasets of 4 KiB or more, so their text was +stored raw; with only 40 distinct texts, deflate now removes most of it. +An `f32` embedding plus 200 B of text is 1,536 + 200 = 1,736 B; `float16` +alone would bring that to about 968 B per record, not the measured 803–829 B. This run does not measure how the saving splits between the two +beyond that, nor explain why the gzip ratio rose so much further, nor +separate the lower throughput (and the 1K ingest going from 5.7 to 10.8 ms) +from the change of machine and store configuration. The +`f32` on-disk figures were not re-measured; the dated +[float16 study](#float16-embedding-storage-memoryconfigfloat16) compares the +two directly (100K × 384: 154.0 MiB `f32`, 80.8 MiB `float16`). --- @@ -1146,7 +1590,12 @@ HDF5 file size at various record counts — 384-dimensional embeddings, 200-char Hippocampal-inspired memory consolidation improves both retrieval quality and search speed. -> **Run:** `cargo run --release --bin consolidation_efficiency` +> **Run:** `cargo run --release -p clawhdf5-bench --bin consolidation_efficiency` + +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. The run +was stopped (after about 19 minutes on one core) while still computing a +100K row of the cycle-time table, so that row and the binary's memory-reduction +part were not produced; neither has ever been published here. ### Retrieval Quality Before vs. After Consolidation @@ -1159,17 +1608,22 @@ Hippocampal-inspired memory consolidation improves both retrieval quality and se | Hit@5 | 100.0% | 100.0% | — | | Hit@10 | 100.0% | 100.0% | — | | MRR | 1.0000 | 1.0000 | — | -| Search latency | 2,752 µs | 312 µs | **8.8x faster** | +| Search latency (avg) | 2,223.7 µs | 239.2 µs | **9.3x faster** | -Signal records survive consolidation because they are accessed 15+ times, giving them high decay scores. 900 noise records evicted, search speeds up 8.8x, and **zero quality loss** — perfect recall maintained. +Signal records survive consolidation because they are accessed 15+ times, giving them high decay scores. 900 noise records evicted, search speeds up 9.3x, and **zero quality loss** — perfect recall maintained. The consolidation cycle itself took 0.13 ms and promoted 2 records. ### Consolidation Cycle Time | Records | Cycle Time | Evictions | Promotions | |---------|-----------|-----------|------------| -| 100 | 21 µs | 100 | 0 | -| 1K | 345 µs | 1,000 | 0 | -| 10K | 17.3 ms | 10,000 | 0 | +| 100 | 17 µs | 100 | 0 | +| 1K | 189 µs | 1,000 | 0 | +| 10K | 2.16 ms | 10,000 | 0 | + +The 1K and 10K cycles were 345 µs and 17.3 ms in the previous, undated table +(before: 2,752 µs / after: 312 µs for search). Eviction's membership check +changed from a linear scan to a `HashSet` in 603fcf8 (2026-08-17), after those +figures were recorded; this run does not isolate its effect. --- @@ -1178,34 +1632,44 @@ Signal records survive consolidation because they are accessed 15+ times, giving In-memory key-value store with TTL, capacity eviction, and embedding search. No network hop, no serialization — direct HashMap operations. -> **Run:** `cargo run --release --bin ephemeral_perf` +> **Run:** `cargo run --release -p clawhdf5-bench --bin ephemeral_perf` + +Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. ### Latency Comparison -| Operation | clawhdf5 Ephemeral | Redis (single-node)¹ | Speedup | +| Operation | clawhdf5 Ephemeral | Redis (single-node)¹ | Ratio¹ | |-----------|-------------------|---------------------|---------| -| SET | **356 ns/op** | ~25,000 ns/op | **70x** | -| GET (hit) | **179 ns/op** | ~25,000 ns/op | **140x** | -| GET (miss) | **62 ns/op** | ~25,000 ns/op | **403x** | -| DELETE | **124 ns/op** | ~25,000 ns/op | **202x** | -| SET+embedding | **268 ns/op** | N/A | — | +| SET | **366 ns/op** | ~25,000 ns/op (not measured) | 68.3x | +| GET (hit) | **115 ns/op** | ~25,000 ns/op (not measured) | 216.7x | +| GET (miss) | **57 ns/op** | — | — | +| DELETE | **99 ns/op** | — | — | +| SET+embedding | **236 ns/op** | N/A | — | -> ¹ Redis latency includes network round-trip (loopback). clawhdf5 ephemeral is in-process — no network. +> ¹ **The Redis figure is not a measurement.** It is a constant hard-coded in +> `ephemeral_perf` as a "typical single-node" latency, and the binary computes +> the two ratios from it; only the clawhdf5 column was measured. A typical +> Redis figure includes a network round-trip, which the in-process ephemeral +> tier does not have, so the ratios compare different things. ### Throughput | Operation | ops/sec | |-----------|---------| -| SET | 2,810,649 | -| GET | 5,584,684 | -| DELETE | 8,093,731 | -| SET+EMB (384d) | 3,725,877 | +| SET | 2,732,568 | +| GET (hit) | 8,666,607 | +| GET (miss) | 17,554,663 | +| DELETE | 10,109,895 | +| SET+EMB (384d) | 4,235,766 | ### Embedding Search (ephemeral tier) | Scale | Latency | |-------|---------| -| 10K entries @ 384d | 2.9 ms/query | +| 10K entries @ 384d | 2.85 ms/query | + +GET (hit) moved from 179 to 115 ns/op and DELETE from 124 to 99 ns/op (GET +throughput 5,584,684 -> 8,666,607 ops/sec); the previous figures were undated. --- @@ -1263,10 +1727,16 @@ Measured 2026-08-07 on tank (Ryzen 7 7800X3D, 246 MB dataset in page cache). ### Measured Platforms +The tank row was measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit +5c8323c, with the single benchmark rather than the whole script: + +> **Run:** `cargo bench -p clawhdf5-agent --bench bench -- '^ivf_search_10k_nprobe10$'` + | Platform | CPU | 10K IVF Search | Notes | |----------|-----|----------------|-------| -| Linux x86_64 | Intel i7-12650H (10C, 4.7 GHz) | 27 µs | Primary CI target | -| macOS aarch64 | Apple M3 Max (14C) | ~18 µs | ~33% faster via NEON SIMD | +| Linux x86_64 | AMD Ryzen 7 7800X3D (8C/16T), tank | 24.8 µs | Measured 2026-09-24 | +| Linux x86_64 | Intel i7-12650H (10C, 4.7 GHz) | 27 µs | Undated, original run | +| macOS aarch64 | Apple M3 Max (14C) | ~18 µs | Undated; ~33% faster than the i7-12650H row. Not reproducible on the hardware available for the 2026-09-24 re-run | ### Reproducibility @@ -1610,6 +2080,11 @@ vs-libhdf5 validation above) to give both sections the dated, hardware-cited, reproducible citation the top-of-file traceability note flags them as missing. +> The sections this re-run checked were themselves re-measured on tank on +> 2026-09-24 and now show those figures; the i7-12650H figures this section +> compares against ("above", "at the top of this file") are quoted in those +> sections' notes. Kept as the 2026-08-05 record. + ### LongMemEval Results (reproduction) ```bash diff --git a/CHANGELOG.md b/CHANGELOG.md index 2996da6..c32606a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,17 @@ ## Unreleased ### Upgrade Notes +- **`clawhdf5-migrate` now writes a real agent store.** Its output used to be + a layout of its own (`/chunks`, `/sessions`, `/entities`, `/relations`, no + `/meta`) that `HDF5Memory::open` rejected, so a migrated file could not be + used as agent memory. Files it wrote before this release are not agent + stores; re-run the migration. Also: embeddings default to `float16` like + any new store (`--f32` opts out; `--float16` is a hidden no-op); a row with + the wrong embedding length is an error instead of being truncated or + padded; `--incremental` now matches rows by content against an existing + store and follows the source's deleted flags; a source with no memory rows + needs `--embedding-dim`. The per-dataset SHA-256 provenance attributes of + the old layout are gone (the agent schema has no place for them). - **Files written by clawhdf5 now open in h5py and libhdf5.** Every `f32` dataset we wrote — including every agent store's embeddings — was refused with "sign bit position out of bounds", and every empty dataset with @@ -46,6 +57,24 @@ `quantized_index = false`, or pass `create --f32-index` to the CLI, to opt out. The CLI's `--quantized-index` is still accepted but is now a no-op. +### Migration +- `clawhdf5-migrate`: writes through the agent's own API (`HDF5Memory::create` + / `open`, `save_batch`, the session cache and knowledge graph), so there is + no second copy of the schema. Sessions and entities/relations carry over; + deleted rows become deleted records (or are left out with + `--skip-deleted`). Every source row is checked before the output is created, + so a source that cannot be migrated leaves an existing store untouched. + Validation reads the result back with `HDF5Memory::open_read_only`, compares + every field (embeddings bit for bit — `round_to_f16` of the source for a + `float16` store) and checks that a migrated record is found by search. The + `half`-based conversion is gone; `clawhdf5_format::float16` is the only one. + 42 tests, including h5py opening a migrated store; an adversarial review's + two blocker and four major findings are fixed with regression tests. +- `clawhdf5-agent`: `HDF5Memory::sessions()` / `sessions_mut()`, + `HDF5Memory::delete_batch(&[usize])` (one save, all-or-nothing, never + auto-compacts), `SessionCache::add_at`, and `SessionCache` / `SessionEntry` + re-exported from the crate root. + ### Search - `clawhdf5-agent`: **`HDF5Memory::search` with `SearchOptions`** — source filtering, re-ranking and confidence rejection in the store's own search @@ -68,6 +97,20 @@ activation of the `k` results it returns, not of the whole `3k` candidate pool it re-ranks. +### Benchmarks +- Every undated or pre-September section of `BENCHMARKS.md` re-run on one + machine on one day (tank, 2026-09-24, commit 5c8323c), with the command for + each and every number traced back to the raw output by a separate check. + Where a figure moved, the section says so. Two apparent regressions were + isolated rather than published: knowledge-graph traversal (a real bug, + fixed above) and the write path, which measures the same at v2.3.0 on this + machine — the old 18 µs / 6.17 ms figures came from an undated run on other + hardware; `float16` adds ~2 µs per save and the int8 index nothing. +- New `multimodal_bench`: cross-modal search at 1K and 10K records, which the + README claimed but nothing measured. +- `footprint_bench` reports whether it built `float16` or `f32` stores and + takes `--f32`; it had kept printing "f32" after the default changed. + ### Interop - `clawhdf5-format`: **every `f32` dataset was unreadable by h5py and libhdf5.** The float datatype encoder hard-coded the sign bit's position to @@ -146,6 +189,14 @@ knew to ask; it now only ever switches the default off. ### Performance +- `clawhdf5-agent`: **knowledge-graph traversal was 6.5x slower than it + should be.** `bfs_neighbors` and `spreading_activation` built an adjacency + index over the whole graph on every call (1efd82c), so a 2-hop BFS over 1K + entities took 155 µs. The index is now cached on `KnowledgeCache` and + checked against a fingerprint of the graph on each use — one pass over + entity ids and relation endpoints, no allocation — so any change, including + direct edits of its public `Vec`s, still rebuilds it (tested). BFS over 1K + entities: 155.1 -> 23.1 µs; spreading activation over 100: 22.8 -> 10.1 µs. - `clawhdf5-format`, `clawhdf5-filters`: both deflate paths hand the codec the whole chunk in one call, into a buffer allocated once, instead of streaming it through a 32 KiB buffer: about 5% on chunked writes and 10% on zlib-ng's diff --git a/README.md b/README.md index a96553b..588a825 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ [![Rust](https://img.shields.io/badge/rust-1.92%2B-orange.svg)](https://www.rust-lang.org) [![Tests](https://img.shields.io/badge/tests-1850%2B-brightgreen.svg)](#building) [![LongMemEval](https://img.shields.io/badge/LongMemEval__s-Turn--Level%20Hit@5%2081.4%25%20hybrid-blue.svg)](BENCHMARKS.md#longmemeval-results) -[![Footprint](https://img.shields.io/badge/on--disk-1.7%20KB%2Frecord-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1) +[![Footprint](https://img.shields.io/badge/on--disk-~820%20B%2Frecord%20float16%2C%20synthetic%20text-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1) ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file. @@ -111,8 +111,8 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d | Keyword search | Separate FTS engine | Integrated BM25 | | Knowledge graph | Neo4j or none | In-file graph with spreading activation | | Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers | -| Temporal queries | Custom code | Native temporal index (716ns) | -| Multi-modal | Multiple stores | Unified cross-modal search | +| Temporal queries | Custom code | Native temporal index (622 ns range query over 10K) | +| Multi-modal | Multiple stores | Unified cross-modal search (exact scan: 842 µs over 1K records) | | Integrity | Hope for the best | Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance | | Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** | @@ -120,7 +120,7 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d ## Performance -Vector search and agent-memory operations below are benchmarked on Intel i7-12650H (10C/16T), 384-dim embeddings, Criterion.rs. The HDF5 Core I/O table immediately below is from a separate, independently reproduced run (see its own hardware note). +The brute-force/IVF vector search, agent-memory, on-disk footprint and consolidation figures below were measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D, 8C/16T), commit 5c8323c, 384-dim embeddings; the commands are in [BENCHMARKS.md](BENCHMARKS.md). Exceptions are marked where they appear: the HDF5 Core I/O table immediately below is from a separate, independently reproduced run (see its own hardware note), and the HNSW `f32`/`i8` table and the in-memory `i8` column were not re-measured on 2026-09-24. ### HDF5 Core I/O (vs libhdf5 1.14.6) @@ -154,33 +154,45 @@ and [§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quant | `f32` index | 0.9945 | 13 399 | 3.2 s | | `i8` index + exact re-score (**default for new stores**) | 0.9940 | **21 848** | **1.8 s** | -Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31. +Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31. These +two rows are a paired comparison (medians of alternating runs, same binary). +A single `f32` run on 2026-09-24 measured recall 0.9945, 19 001 QPS and a +2.7 s build; the int8 row was not re-run, so the pair has not been re-checked +([§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index)). -**Brute-force and IVF paths** (Criterion, i7-12650H): +**Brute-force and IVF paths** (Criterion, tank, 2026-09-24): -| Scale | Flat | IVF (nprobe=10) | IVF-PQ | vs MemX¹ | +| Scale | Flat | IVF (nprobe=10) | IVF-PQ | MemX¹ (claimed, end-to-end) | |-------|------|-----------------|--------|----------| -| 1K | **54 µs** | — | — | — | -| 10K | 753 µs | **27 µs** | — | — | -| 100K | 11.4 ms | 1.32 ms | **1.19 ms** | ~8–76× (see caveat) | +| 1K | **47.4 µs** | — | — | — | +| 10K | 500.5 µs | **24.8 µs** | — | — | +| 100K | 6.58 ms | 592 µs | **869 µs** | <90 ms | -> Reproduced on the same second machine (Ryzen 7 7800X3D) with a corrected, -> apples-to-apples SIMD/scalar/parallel comparison methodology — see -> [BENCHMARKS.md § Independent Validation: tank — LongMemEval & Vector -> Search](BENCHMARKS.md#independent-validation-tank--longmemeval--vector-search-ryzen-7-7800x3d-2026-08-05). +> These replace figures from the original i7-12650H run (flat 54 µs / 753 µs / +> 11.4 ms); a 2026-08-05 run on tank had already matched the new ones — see +> [BENCHMARKS.md § Vector Search Latency](BENCHMARKS.md#vector-search-latency). ### Agent Memory Operations | Operation | Latency | Scale | |-----------|---------|-------| -| Hybrid search (`HDF5Memory::hybrid_search`, p50) | **70 µs** / 0.49 ms / 4.65 ms | 1K / 10K / 100K records | -| BM25 keyword search | **67 µs** | 1K records | -| Knowledge graph BFS | **24 µs** | 1K entities | -| Spreading activation | **17 µs** | 100 entities | -| Temporal range query | **716 ns** | 10K timestamps | -| Consolidation cycle | **164 µs** | 1K records | -| Memory write (WAL) | **18 µs** | per record (group-commit append; HDF5 batched at flush) | -| Importance gate | **61 ns** | per record | +| Hybrid search (`HDF5Memory::hybrid_search`, p50) | **0.07 ms** / 0.49 ms / 4.69 ms | 1K / 10K / 100K records | +| BM25 keyword search | **20.4 µs** | 1K records | +| Knowledge graph BFS | **23.1 µs** | 1K entities | +| Spreading activation | **10.1 µs** | 100 entities | +| Temporal range query | **622 ns** | 10K timestamps | +| Consolidation cycle | **115.2 µs** | 1K records | +| Cross-modal search (exact scan, 2 embeddings per record) | **842.0 µs** / 8.44 ms | 1K / 10K records | +| Memory write (WAL) | **26.1 µs** | per record (group-commit append; HDF5 batched at flush) | +| Importance gate | **57.6 ns** | per record (trivial skip) | + +The old 18 µs WAL write was undated, from another machine: v2.3.0 measures +24.3 µs on the same hardware as this table, the same as an `f32` store today. +`float16` stores (the new default) add ~2 µs for rounding; the int8 index adds +nothing. See [BENCHMARKS.md § Write Path](BENCHMARKS.md#write-path). +Knowledge-graph traversal was briefly 6.5x slower (155 µs) until this re-run +found and fixed an adjacency index rebuilt on every traversal; see +[§ Knowledge Graph](BENCHMARKS.md#knowledge-graph). ### Chunked Write Throughput (codec comparison) @@ -195,7 +207,7 @@ by default (AoS→SoA byte transpose, +157–204% throughput for float data): Use `.with_zstd(3)` or `.with_deflate(6)` for write-heavy workloads — both now perform at ~720–750 MiB/s on large matrices. Use `.with_pcodec()` for write-once/read-many workloads where compression ratio matters more than encode speed. Disable auto-shuffle with `.without_shuffle()` for byte arrays that don't benefit from AoS→SoA transposition. -> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search). The ratio overstates the real advantage by an unquantified margin — order-of-magnitude indication only. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171). +> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search), so the two columns are not comparable and no ratio is given. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171). ### LongMemEval Retrieval Recall @@ -248,18 +260,26 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points. ### Memory Footprint -**On disk** — 384-dim embeddings, 200-char text +**On disk** — 384-dim `float16` embeddings (the default for new stores), +200-char text, `footprint_bench` ([BENCHMARKS.md § Memory Footprint](BENCHMARKS.md#memory-footprint-1)): | Records | File Size | Bytes/Record | Gzip-6 compressed | |---------|-----------|--------------|-------------------| -| 1K | 1.7 MB | 1.8 KB | 277 KB (6.1x) | -| 10K | 17.0 MB | 1.7 KB | 2.7 MB (6.2x) | -| 100K | 169.8 MB | 1.7 KB | 26.9 MB (6.2x) | +| 1K | 810.4 KB | 829 B | 56.4 KB | +| 10K | 7.8 MB | 820 B | 471.3 KB | +| 100K | 76.7 MB | 803 B | 4.5 MB | -These figures are `f32` embeddings. New agent stores default to -`MemoryConfig::float16`, which halves them: 100K × 384 records take 80.8 MiB -instead of 154.0. +The benchmark's synthetic embeddings and text are far more repetitive than +real data (only 40 distinct texts), so no column here is an expectation for +real data. The compressed column is an upper bound, and the Bytes/Record +column is optimistic too: it is not an uncompressed figure, because the store +always deflates its text (any string dataset of 4 KiB or more) whatever +`MemoryConfig::compression` says. The `float16` embeddings alone are 768 B per +record, so 200 characters of real text would take a record above 820 B. +This table used to show `f32` stores (1.7 KB per record, 169.8 MB at 100K); +those were not re-measured. The float16 study compares the two on the same +data: 100K × 384 records take 80.8 MiB as `float16` and 154.0 MiB as `f32`. **In memory** — a store reopened from disk, 384-dim `f32`, measured with a counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-footprint)): @@ -271,7 +291,8 @@ counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-foo | 100K | 146 MiB | 399 MiB (2.72x) | **256 MiB (1.74x)** | Down from 505 MiB (3.44x) at 100K before v2.6.0, when the cache held every -embedding twice. +embedding twice. The `f32` column was re-measured on 2026-09-24 and reproduced +exactly; the `i8` column was not re-run. ### Consolidation Efficiency @@ -282,7 +303,10 @@ embedding twice. |--------|--------|-------|-------| | Records in store | 1,000 | 100 | −90% | | Hit@1 recall (signal records) | 100% | 100% | no loss | -| Search latency | 2.75 ms | 0.31 ms | **8.8x faster** | +| Search latency (avg) | 2.22 ms | 0.24 ms | **9.3x faster** | + +The consolidation cycle that does this took 0.13 ms; at 10K records a cycle +takes 2.16 ms. **Full benchmark details: [BENCHMARKS.md](BENCHMARKS.md)** @@ -754,9 +778,37 @@ Replace in `Cargo.toml` and source: ```bash cargo install --path crates/clawhdf5-migrate -clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedding-dim 384 +clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedder minilm ``` +The output is an ordinary `clawhdf5-agent` store, written through the agent's +own API: open it with `HDF5Memory::open` (or `clawhdf5-cli --path memory.h5 …`) +and search it straight away. What carries over from the ZeroClaw tables: + +| SQLite | Agent store | +|--------|-------------| +| `memory_chunks` | memory records (text, embedding, source channel, timestamp, session id, tags); rows with `deleted = 1` become deleted records, or are left out with `--skip-deleted` | +| `sessions` | sessions (id, start/end index, channel, summary, timestamp) | +| `entities`, `relations` | knowledge graph entities and relations; entities get new ids and relations are re-pointed at them | + +The chunk `id` column has no counterpart in the agent store, so records are +written in `id` order and numbered from 0. Embeddings are stored as float16 +like any new store; `--f32` keeps full precision (and is required for values +beyond ±65504). The embedding dimension is detected from the first row unless +`--embedding-dim` is given, and every row must have it: a row of another length +is an error, never truncated or padded. A source with no memory records (only +sessions or the graph) needs `--embedding-dim`, since a store's dimension is +fixed when it is created. Every row is checked before the output is created, +so a source that cannot be migrated leaves an existing store at `--hdf5` as it +was. `--incremental` adds to an existing store only the rows it does not +already hold; the source must have the store's dimension, and records already +in the store take the source's deleted flag (a row deleted in SQLite since the +last run is deleted in the store; one un-deleted there is written again, as +the agent has no un-delete). The tool reads the result back with +`HDF5Memory::open_read_only`, compares it with the source (every row with +`--validate-full`) and checks that a migrated record is found by search; +`--dry-run` only counts the rows. + --- ## Roadmap diff --git a/crates/clawhdf5-agent/Cargo.toml b/crates/clawhdf5-agent/Cargo.toml index b8d79ee..8af2c6f 100644 --- a/crates/clawhdf5-agent/Cargo.toml +++ b/crates/clawhdf5-agent/Cargo.toml @@ -45,6 +45,10 @@ harness = false name = "memory_bench" harness = false +[[bench]] +name = "multimodal_bench" +harness = false + [features] default = ["float16", "hnsw", "parallel"] float16 = ["half"] diff --git a/crates/clawhdf5-agent/benches/multimodal_bench.rs b/crates/clawhdf5-agent/benches/multimodal_bench.rs new file mode 100644 index 0000000..798a035 --- /dev/null +++ b/crates/clawhdf5-agent/benches/multimodal_bench.rs @@ -0,0 +1,107 @@ +//! Multi-modal memory search benchmarks (`clawhdf5_agent::multimodal`). +//! +//! Covers `MultiModalStore::search_cross_modal` (every embedding of every +//! record, whatever its modality) and, for comparison, +//! `MultiModalStore::search_by_modality` restricted to one modality. +//! +//! Corpus: N records (1K and 10K), each carrying two 384-dim embeddings — +//! a text embedding of its caption plus one embedding of its primary modality, +//! cycling Image / Audio / Video — so a cross-modal query scores 2N vectors. +//! All data comes from a fixed-seed LCG, so every run sees the same corpus. +//! +//! Run: `cargo bench -p clawhdf5-agent --bench multimodal_bench` + +use std::collections::HashMap; + +use clawhdf5_agent::multimodal::{ + MediaRef, ModalEmbedding, Modality, MultiModalRecord, MultiModalStore, +}; +use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main}; + +// --------------------------------------------------------------------------- +// Simple deterministic PRNG (LCG), same as the other agent benches +// --------------------------------------------------------------------------- + +struct Rng(u32); + +impl Rng { + fn new(seed: u32) -> Self { + Self(seed) + } + fn next_u32(&mut self) -> u32 { + self.0 = self.0.wrapping_mul(1103515245).wrapping_add(12345); + self.0 >> 16 + } + fn next_f32(&mut self) -> f32 { + self.next_u32() as f32 / 65536.0 - 0.5 + } +} + +fn make_vec(rng: &mut Rng, dim: usize) -> Vec { + (0..dim).map(|_| rng.next_f32()).collect() +} + +// --------------------------------------------------------------------------- +// Corpus +// --------------------------------------------------------------------------- + +const DIM: usize = 384; +const K: usize = 10; + +const MEDIA: [(Modality, &str, &str); 3] = [ + (Modality::Image, "image/png", "clip-vit-base"), + (Modality::Audio, "audio/wav", "clap-base"), + (Modality::Video, "video/mp4", "xclip-base"), +]; + +fn build_store(n: usize, seed: u32) -> MultiModalStore { + let mut rng = Rng::new(seed); + let mut store = MultiModalStore::new(); + for i in 0..n { + let (modality, mime, model) = &MEDIA[i % MEDIA.len()]; + let embeddings = vec![ + ModalEmbedding::new(Modality::Text, make_vec(&mut rng, DIM), "minilm-l6"), + ModalEmbedding::new(modality.clone(), make_vec(&mut rng, DIM), *model), + ]; + store.add_record(MultiModalRecord { + id: 0, + primary_modality: modality.clone(), + text_content: Some(format!("{modality} memory {i}")), + media_ref: Some(MediaRef::path(format!("/media/{i}"), *mime)), + embeddings, + observation: None, + timestamp: 1_700_000_000.0 + i as f64, + metadata: HashMap::new(), + }); + } + store +} + +// --------------------------------------------------------------------------- +// Benchmarks +// --------------------------------------------------------------------------- + +fn multimodal_search_benches(c: &mut Criterion) { + let query = make_vec(&mut Rng::new(99), DIM); + + let mut group = c.benchmark_group("multimodal_search"); + group.sample_size(50); + + for (label, n) in [("1k", 1_000usize), ("10k", 10_000)] { + let store = build_store(n, 42); + assert_eq!(store.count(), n); + + group.bench_with_input(BenchmarkId::new("cross_modal", label), &n, |b, _| { + b.iter(|| store.search_cross_modal(&query, K)); + }); + + group.bench_with_input(BenchmarkId::new("by_modality_image", label), &n, |b, _| { + b.iter(|| store.search_by_modality(&Modality::Image, &query, K)); + }); + } + + group.finish(); +} + +criterion_group!(multimodal_benches, multimodal_search_benches); +criterion_main!(multimodal_benches); diff --git a/crates/clawhdf5-agent/src/knowledge.rs b/crates/clawhdf5-agent/src/knowledge.rs index 59c6a01..4470e57 100644 --- a/crates/clawhdf5-agent/src/knowledge.rs +++ b/crates/clawhdf5-agent/src/knowledge.rs @@ -163,12 +163,13 @@ fn levenshtein(a: &str, b: &str) -> usize { /// entities-slice-index map, and an entity-id -> relation-indices map (edges /// touching that entity as either source or target). /// -/// Built fresh per traversal call rather than cached on `KnowledgeCache`: -/// entities/relations are plain `pub` `Vec`s that get pushed to directly -/// (e.g. `schema.rs`'s load path bypasses `add_entity`/`add_relation`), so a -/// persistent index would need extra bookkeeping to avoid drifting stale. A -/// one-off O(V+E) build per call is still a large win over the O(V·E) (BFS) -/// / O(steps·active·E) (spreading activation) scans it replaces. +/// Cached on `KnowledgeCache` and checked against a fingerprint of the graph +/// on every use ([`graph_fingerprint`]). entities/relations are plain `pub` +/// `Vec`s that get changed directly (e.g. `schema.rs`'s load path bypasses +/// `add_entity`/`add_relation`), so the cache cannot rely on being told about +/// changes; the fingerprint notices any of them. Rebuilding it on every +/// traversal instead made a 2-hop BFS over 1K entities 6.5x slower than the +/// scan it replaced (24 -> 155 µs; `BENCHMARKS.md`, "Knowledge Graph"). struct AdjacencyIndex { entity_index: HashMap, by_entity: HashMap>, @@ -204,6 +205,45 @@ impl AdjacencyIndex { } } +/// A hash of everything [`AdjacencyIndex`] depends on — each entity's id and +/// position, each relation's endpoints and position. One linear pass, no +/// allocation: far cheaper than building the index, which hashes the same +/// values into two maps. +fn graph_fingerprint(entities: &[Entity], relations: &[Relation]) -> u64 { + // splitmix64-style mixing; order matters, so positions are covered. + fn mix(h: u64, v: u64) -> u64 { + let mut z = (h ^ v).wrapping_add(0x9E37_79B9_7F4A_7C15); + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + let mut h = mix(entities.len() as u64, relations.len() as u64); + for e in entities { + h = mix(h, e.id); + } + for r in relations { + h = mix(mix(h, r.src), r.tgt); + } + h +} + +/// The cached [`AdjacencyIndex`] and the fingerprint it was built for. +/// Cloning a `KnowledgeCache` starts the clone with an empty cache. +#[derive(Default)] +struct AdjacencyCache(std::sync::Mutex)>>); + +impl Clone for AdjacencyCache { + fn clone(&self) -> Self { + Self::default() + } +} + +impl std::fmt::Debug for AdjacencyCache { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str("AdjacencyCache") + } +} + // --------------------------------------------------------------------------- // KnowledgeCache // --------------------------------------------------------------------------- @@ -216,6 +256,7 @@ pub struct KnowledgeCache { pub alias_strings: Vec, pub alias_entity_ids: Vec, next_entity_id: u64, + adjacency: AdjacencyCache, } impl KnowledgeCache { @@ -226,6 +267,7 @@ impl KnowledgeCache { alias_strings: Vec::new(), alias_entity_ids: Vec::new(), next_entity_id: 0, + adjacency: AdjacencyCache::default(), } } @@ -236,9 +278,29 @@ impl KnowledgeCache { alias_strings: Vec::new(), alias_entity_ids: Vec::new(), next_entity_id: next_id, + adjacency: AdjacencyCache::default(), } } + /// The adjacency index for the graph as it is now: the cached one if the + /// graph's fingerprint still matches, otherwise rebuilt and cached. + fn adjacency_index(&self) -> std::sync::Arc { + let fp = graph_fingerprint(&self.entities, &self.relations); + let mut slot = self + .adjacency + .0 + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if let Some((cached_fp, idx)) = slot.as_ref() + && *cached_fp == fp + { + return idx.clone(); + } + let idx = std::sync::Arc::new(AdjacencyIndex::build(&self.entities, &self.relations)); + *slot = Some((fp, idx.clone())); + idx + } + // ----------------------------------------------------------------------- // Entity management // ----------------------------------------------------------------------- @@ -397,7 +459,7 @@ impl KnowledgeCache { /// together with their discovered depth. The seed entity itself is NOT /// included. Traversal follows both outgoing and incoming relation edges. pub fn bfs_neighbors(&self, entity_id: u64, max_depth: usize) -> Vec<(Entity, usize)> { - let idx = AdjacencyIndex::build(&self.entities, &self.relations); + let idx = self.adjacency_index(); let mut visited: HashSet = HashSet::new(); let mut queue: VecDeque<(u64, usize)> = VecDeque::new(); let mut results: Vec<(Entity, usize)> = Vec::new(); @@ -502,7 +564,7 @@ impl KnowledgeCache { min_activation: f32, max_steps: usize, ) -> Vec<(u64, f32)> { - let idx = AdjacencyIndex::build(&self.entities, &self.relations); + let idx = self.adjacency_index(); let mut activation: HashMap = HashMap::new(); // Initialise seeds with activation 1.0. @@ -631,6 +693,51 @@ impl Default for KnowledgeCache { mod tests { use super::*; + #[test] + fn cached_adjacency_sees_direct_changes_to_the_graph() { + // The index is cached across traversals, but entities/relations are + // pub Vecs anyone can edit; every kind of edit must be seen. + let mut kg = KnowledgeCache::new(); + let a = kg.add_entity("a", "t", -1); + let b = kg.add_entity("b", "t", -1); + let c = kg.add_entity("c", "t", -1); + kg.add_relation(a, b, "r", 1.0); + let ids = |kg: &KnowledgeCache| -> Vec { + let mut v: Vec = kg.bfs_neighbors(a, 3).iter().map(|(e, _)| e.id).collect(); + v.sort(); + v + }; + assert_eq!(ids(&kg), vec![b]); + assert_eq!(ids(&kg), vec![b], "cached index reused"); + + // Pushed directly, bypassing add_relation. + kg.relations.push(Relation { + src: b, + tgt: c, + ..Relation::default() + }); + assert_eq!(ids(&kg), vec![b, c]); + + // Rewired in place: same lengths, different edge. + kg.relations[1].tgt = a; + assert_eq!(ids(&kg), vec![b]); + + // Removed and replaced: same lengths again. + kg.relations.pop(); + kg.relations.push(Relation { + src: a, + tgt: c, + ..Relation::default() + }); + assert_eq!(ids(&kg), vec![b, c]); + let act: Vec = kg + .spreading_activation(&[a], 0.5, 0.0, 2) + .iter() + .map(|(id, _)| *id) + .collect(); + assert!(act.contains(&c)); + } + // ----------------------------------------------------------------------- // Original tests — must remain passing // ----------------------------------------------------------------------- diff --git a/crates/clawhdf5-agent/src/lib.rs b/crates/clawhdf5-agent/src/lib.rs index b74507f..62575fa 100644 --- a/crates/clawhdf5-agent/src/lib.rs +++ b/crates/clawhdf5-agent/src/lib.rs @@ -73,7 +73,7 @@ pub use ephemeral::{EphemeralEntry, EphemeralStats}; use knowledge::KnowledgeCache; use memory_strategy::{Exchange, MemoryStrategy, StrategyOutput}; pub use search::SearchOptions; -use session::SessionCache; +pub use session::{SessionCache, SessionEntry}; // --- Error type --- @@ -1028,6 +1028,18 @@ impl HDF5Memory { &self.config } + /// The sessions recorded in this store. + pub fn sessions(&self) -> &SessionCache { + &self.sessions + } + + /// Mutable access to the sessions, e.g. to add many at once. Changes + /// reach the disk at the next checkpoint (any flushing call, such as + /// [`HDF5Memory::flush_wal`] or `save_batch`), not immediately. + pub fn sessions_mut(&mut self) -> &mut SessionCache { + &mut self.sessions + } + /// Get a reference to the knowledge cache. pub fn knowledge(&self) -> &KnowledgeCache { &self.knowledge @@ -1401,6 +1413,35 @@ impl HDF5Memory { } impl HDF5Memory { + /// Delete many records with a single checkpoint, where + /// [`AgentMemory::delete`] checkpoints once per record. + /// + /// All or nothing: if any id is out of range or already deleted (or + /// repeated), nothing is deleted and `MemoryError::NotFound` is returned. + /// Unlike `delete`, this never auto-compacts, so the records stay in the + /// store as tombstones (their indices unchanged) until [`AgentMemory::compact`] + /// is called — importers use it to carry over records that were already + /// deleted in the source. + pub fn delete_batch(&mut self, ids: &[usize]) -> Result<()> { + let mut seen = std::collections::HashSet::with_capacity(ids.len()); + for &id in ids { + if self.cache.tombstones.get(id).copied() != Some(0) || !seen.insert(id) { + return Err(MemoryError::NotFound(format!( + "entry {id} not found or already deleted" + ))); + } + } + if ids.is_empty() { + return Ok(()); + } + for &id in ids { + self.cache.mark_deleted(id); + self.hnsw_on_delete(id); + self.bm25_on_delete(id); + } + self.flush() + } + pub fn tick_session(&mut self) -> Result<()> { let d = self.config.decay_factor; for w in self.cache.activation_weights.iter_mut() { @@ -1599,6 +1640,79 @@ mod tests { } } + #[test] + fn delete_batch_tombstones_without_compacting() { + let dir = TempDir::new().unwrap(); + let path = dir.path().join("test.h5"); + let mut mem = HDF5Memory::create(make_config(&dir)).unwrap(); + mem.save_batch( + (0..4) + .map(|i| make_entry(&format!("record {i}"), &[i as f32, 1.0, 0.0, 0.0])) + .collect(), + ) + .unwrap(); + // 3 of 4 is far past compact_threshold (0.3): delete() would compact. + mem.delete_batch(&[0, 1, 3]).unwrap(); + assert_eq!(mem.count(), 4); + assert_eq!(mem.count_active(), 1); + drop(mem); + + let mut mem = HDF5Memory::open(&path).unwrap(); + assert_eq!(mem.cache.tombstones, vec![1, 1, 0, 1]); + let hits = mem.hybrid_search(&[0.0, 1.0, 0.0, 0.0], "record", 0.5, 0.5, 10); + assert!( + hits.iter().all(|r| r.index == 2), + "tombstoned record returned" + ); + } + + #[test] + fn delete_batch_is_all_or_nothing() { + let dir = TempDir::new().unwrap(); + let mut mem = HDF5Memory::create(make_config(&dir)).unwrap(); + mem.save_batch(vec![ + make_entry("a", &[1.0, 0.0, 0.0, 0.0]), + make_entry("b", &[0.0, 1.0, 0.0, 0.0]), + ]) + .unwrap(); + for bad in [&[0, 5][..], &[1, 1][..]] { + assert!(matches!( + mem.delete_batch(bad), + Err(MemoryError::NotFound(_)) + )); + assert_eq!(mem.count_active(), 2, "{bad:?} deleted something"); + } + mem.delete_batch(&[]).unwrap(); + assert_eq!(mem.count_active(), 2); + } + + #[test] + fn sessions_mut_add_at_keeps_timestamp_across_reopen() { + let dir = TempDir::new().unwrap(); + let path = dir.path().join("test.h5"); + let mut mem = HDF5Memory::create(make_config(&dir)).unwrap(); + mem.sessions_mut() + .add_at("s-old", 2, 7, "discord", "old summary", 1.7e15); + mem.flush_wal().unwrap(); + drop(mem); + + let mem = HDF5Memory::open_read_only(&path).unwrap(); + let s = mem.sessions(); + assert_eq!(s.len(), 1); + let e = &s.entries[0]; + assert_eq!( + ( + e.id.as_str(), + e.start_idx, + e.end_idx, + e.channel.as_str(), + e.ts + ), + ("s-old", 2, 7, "discord", 1.7e15) + ); + assert_eq!(s.summaries[0], "old summary"); + } + #[test] fn create_new_file() { let dir = TempDir::new().unwrap(); diff --git a/crates/clawhdf5-agent/src/session.rs b/crates/clawhdf5-agent/src/session.rs index 9be1f91..ebe817f 100644 --- a/crates/clawhdf5-agent/src/session.rs +++ b/crates/clawhdf5-agent/src/session.rs @@ -33,7 +33,7 @@ impl SessionCache { self.entries.is_empty() } - /// Add a new session with its summary. + /// Add a new session with its summary, timestamped now. pub fn add( &mut self, id: &str, @@ -47,6 +47,21 @@ impl SessionCache { .unwrap_or_default() .as_secs_f64() * 1_000_000.0; // microseconds + self.add_at(id, start_idx, end_idx, channel, summary, ts); + } + + /// Add a session with an explicit timestamp (Unix **microseconds**, the + /// unit [`SessionEntry::ts`] uses) — for importers carrying sessions over + /// from another store, whose original time should be kept. + pub fn add_at( + &mut self, + id: &str, + start_idx: usize, + end_idx: usize, + channel: &str, + summary: &str, + ts: f64, + ) { self.entries.push(SessionEntry { id: id.to_string(), start_idx: start_idx as u64, diff --git a/crates/clawhdf5-bench/src/bin/footprint_bench.rs b/crates/clawhdf5-bench/src/bin/footprint_bench.rs index bce5748..477fd50 100644 --- a/crates/clawhdf5-bench/src/bin/footprint_bench.rs +++ b/crates/clawhdf5-bench/src/bin/footprint_bench.rs @@ -11,12 +11,14 @@ //! //! Configuration matrix: //! - Text lengths: short (50 chars), medium (200 chars), long (1000 chars) -//! - Embedding: 384-dim f32 (1536 bytes raw per record) +//! - Embedding: 384-dim, stored as float16 (the default for new stores) or +//! f32 with `--f32`; "raw" bytes are counted as f32 input either way //! - WAL: enabled and disabled //! //! # Usage //! ``` -//! cargo run --release --bin footprint_bench +//! cargo run --release --bin footprint_bench # float16 stores +//! cargo run --release --bin footprint_bench -- --f32 # f32 stores //! ``` use std::time::Instant; @@ -24,6 +26,9 @@ use std::time::Instant; use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry}; use tempfile::TempDir; +/// `--f32`: build f32 stores instead of the library's float16 default. +static F32: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false); + const EMBEDDING_DIM: usize = 384; // Raw bytes per record: 384 f32 embeddings + median text + overhead @@ -152,6 +157,9 @@ fn measure_footprint( config.compression = compression; config.compression_level = if compression { 6 } else { 0 }; config.compact_threshold = 0.0; + if F32.load(std::sync::atomic::Ordering::Relaxed) { + config.float16 = false; + } let mut memory = HDF5Memory::create(config).expect("HDF5Memory::create failed"); @@ -241,11 +249,19 @@ fn fmt_n(n: usize) -> String { // --------------------------------------------------------------------------- fn main() { + if std::env::args().skip(1).any(|a| a == "--f32") { + F32.store(true, std::sync::atomic::Ordering::Relaxed); + } + let stored = if F32.load(std::sync::atomic::Ordering::Relaxed) { + "f32 (1,536 bytes per record)" + } else { + "float16 (768 bytes per record; the default for new stores)" + }; println!("================================================================="); println!(" ClawhDF5 Memory Footprint Benchmark"); println!("================================================================="); println!(); - println!("Embedding: 384-dim f32 = 1,536 bytes raw per record"); + println!("Embedding: 384-dim, stored as {stored}; raw input counted as f32"); println!("Text lengths: short=50 chars, medium=200 chars, long=1000 chars"); println!(); diff --git a/crates/clawhdf5-migrate/Cargo.toml b/crates/clawhdf5-migrate/Cargo.toml index a518a22..5bd7459 100644 --- a/crates/clawhdf5-migrate/Cargo.toml +++ b/crates/clawhdf5-migrate/Cargo.toml @@ -3,7 +3,7 @@ name = "clawhdf5-migrate" version = "2.7.0" edition = "2024" rust-version.workspace = true -description = "CLI to migrate SQLite agent memory databases to HDF5 format" +description = "CLI to migrate SQLite agent memory databases to clawhdf5-agent stores" license = "MIT" repository = "https://git.redclaw.dev/quantumclaw/clawhdf5" readme = "README.md" @@ -17,10 +17,8 @@ path = "src/main.rs" [dependencies] clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.7.0" } clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" } -clawhdf5 = { path = "../clawhdf5", version = "2.7.0" } rusqlite = { version = "0.31", features = ["bundled"] } clap = { version = "4", features = ["derive"] } -half = { workspace = true } [dev-dependencies] tempfile = { workspace = true } diff --git a/crates/clawhdf5-migrate/README.md b/crates/clawhdf5-migrate/README.md index acbb0b2..54b74a6 100644 --- a/crates/clawhdf5-migrate/README.md +++ b/crates/clawhdf5-migrate/README.md @@ -3,9 +3,12 @@ [![crates.io](https://img.shields.io/crates/v/clawhdf5-migrate.svg)](https://crates.io/crates/clawhdf5-migrate) [![docs.rs](https://img.shields.io/docsrs/clawhdf5-migrate)](https://docs.rs/clawhdf5-migrate) -CLI tool to migrate SQLite agent memory databases to HDF5 format. +CLI tool to migrate SQLite agent memory databases (the ZeroClaw layout) to a +[clawhdf5-agent](https://crates.io/crates/clawhdf5-agent) store. -Converts existing SQLite-based agent memory stores (embeddings, text chunks, metadata) into the HDF5 format used by [clawhdf5-agent](https://crates.io/crates/clawhdf5-agent). +The output is written through `clawhdf5-agent`'s own API, so it opens with +`HDF5Memory::open` and is searchable immediately: memory records, sessions and +the knowledge graph (entities and relations) are carried over. ## Installation @@ -16,9 +19,19 @@ cargo install clawhdf5-migrate ## Usage ```bash -clawhdf5-migrate --input agent.db --output agent.h5 +clawhdf5-migrate --sqlite agent.db --hdf5 agent.h5 --agent-id my-agent ``` +Embeddings are stored as float16 (the library default for new stores); pass +`--f32` for full precision. Every embedding must have the same dimension +(the first row's, or `--embedding-dim`, which a source with no memory records +requires); rows are never truncated, and the whole source is checked before an +existing output store is replaced. `--incremental` adds only new rows to an +existing store of the same dimension and carries over changes to rows' +deleted flags, `--skip-deleted` leaves out tombstoned rows, and `--dry-run` +only counts. +See `clawhdf5-migrate --help` for every option. + ## License MIT diff --git a/crates/clawhdf5-migrate/src/hdf5_reader.rs b/crates/clawhdf5-migrate/src/hdf5_reader.rs deleted file mode 100644 index 00c8956..0000000 --- a/crates/clawhdf5-migrate/src/hdf5_reader.rs +++ /dev/null @@ -1,163 +0,0 @@ -//! Read a migration HDF5 file back into the in-memory data model. -//! -//! Used to verify migrated content (real validation) and to merge new rows into -//! an existing output (incremental migration). Mirrors the layout produced by -//! [`crate::hdf5_writer`]. - -use clawhdf5::reader::{File, Group}; -use clawhdf5_format::type_builders::AttrValue; - -use crate::sqlite_reader::{Entity, MemoryChunk, Relation, Session, SqliteData}; - -type BoxErr = Box; - -fn read_strings(group: &Group<'_>, name: &str) -> Result, BoxErr> { - Ok(group.dataset(name)?.read_string()?) -} - -fn read_i64s(group: &Group<'_>, name: &str) -> Result, BoxErr> { - Ok(group.dataset(name)?.read_i64()?) -} - -fn read_f64s(group: &Group<'_>, name: &str) -> Result, BoxErr> { - Ok(group.dataset(name)?.read_f64()?) -} - -/// Read the embeddings dataset as a flat `Vec` of `n * dim` values, -/// handling both f32 and (lossy) f16 storage. -fn read_embeddings_flat(group: &Group<'_>) -> Result, BoxErr> { - Ok(group.dataset("embeddings")?.read_f32()?) -} - -/// Read a migration HDF5 file into a [`SqliteData`]. -pub fn read_hdf5(path: &str) -> Result { - let file = File::open(path)?; - - let embedding_dim = match file.root().attrs()?.get("embedding_dim") { - Some(AttrValue::I64(d)) => *d as usize, - _ => 0, - }; - - let chunks = read_chunks(&file, embedding_dim)?; - let sessions = read_sessions(&file)?; - let entities = read_entities(&file)?; - let relations = read_relations(&file)?; - - Ok(SqliteData { - chunks, - sessions, - entities, - relations, - embedding_dim, - // Not a SQLite read — the caller (incremental migration) carries - // forward the current run's actual `source_path` from the fresh - // SQLite read instead of using this placeholder. - source_path: String::new(), - }) -} - -fn read_chunks(file: &File, dim: usize) -> Result, BoxErr> { - let g = file.group("chunks")?; - let count = group_count(&g)?; - if count == 0 { - return Ok(Vec::new()); - } - let ids = read_i64s(&g, "id")?; - let texts = read_strings(&g, "text")?; - let channels = read_strings(&g, "source_channel")?; - let timestamps = read_f64s(&g, "timestamp")?; - let session_ids = read_strings(&g, "session_id")?; - let tags = read_strings(&g, "tags")?; - let deleted = g.dataset("deleted")?.read_i32()?; - let emb_flat = read_embeddings_flat(&g)?; - let dim = dim.max(1); - - let mut chunks = Vec::with_capacity(ids.len()); - for (i, &id) in ids.iter().enumerate() { - let embedding = emb_flat - .get(i * dim..(i + 1) * dim) - .map(|s| s.to_vec()) - .unwrap_or_default(); - chunks.push(MemoryChunk { - id, - chunk: texts.get(i).cloned().unwrap_or_default(), - embedding, - source_channel: channels.get(i).cloned().unwrap_or_default(), - timestamp: timestamps.get(i).copied().unwrap_or(0.0), - session_id: session_ids.get(i).cloned().unwrap_or_default(), - tags: tags.get(i).cloned().unwrap_or_default(), - deleted: deleted.get(i).copied().unwrap_or(0), - }); - } - Ok(chunks) -} - -fn read_sessions(file: &File) -> Result, BoxErr> { - let g = file.group("sessions")?; - if group_count(&g)? == 0 { - return Ok(Vec::new()); - } - let ids = read_strings(&g, "id")?; - let starts = read_i64s(&g, "start_idx")?; - let ends = read_i64s(&g, "end_idx")?; - let channels = read_strings(&g, "channel")?; - let timestamps = read_f64s(&g, "timestamp")?; - let summaries = read_strings(&g, "summary")?; - Ok((0..ids.len()) - .map(|i| Session { - id: ids[i].clone(), - start_idx: starts.get(i).copied().unwrap_or(0), - end_idx: ends.get(i).copied().unwrap_or(0), - channel: channels.get(i).cloned().unwrap_or_default(), - timestamp: timestamps.get(i).copied().unwrap_or(0.0), - summary: summaries.get(i).cloned().unwrap_or_default(), - }) - .collect()) -} - -fn read_entities(file: &File) -> Result, BoxErr> { - let g = file.group("entities")?; - if group_count(&g)? == 0 { - return Ok(Vec::new()); - } - let ids = read_i64s(&g, "id")?; - let names = read_strings(&g, "name")?; - let types = read_strings(&g, "type")?; - let emb_idxs = read_i64s(&g, "embedding_idx")?; - Ok((0..ids.len()) - .map(|i| Entity { - id: ids[i], - name: names.get(i).cloned().unwrap_or_default(), - entity_type: types.get(i).cloned().unwrap_or_default(), - embedding_idx: emb_idxs.get(i).copied().unwrap_or(-1), - }) - .collect()) -} - -fn read_relations(file: &File) -> Result, BoxErr> { - let g = file.group("relations")?; - if group_count(&g)? == 0 { - return Ok(Vec::new()); - } - let srcs = read_i64s(&g, "src")?; - let tgts = read_i64s(&g, "tgt")?; - let rels = read_strings(&g, "relation")?; - let weights = read_f64s(&g, "weight")?; - let timestamps = read_f64s(&g, "timestamp")?; - Ok((0..srcs.len()) - .map(|i| Relation { - src: srcs[i], - tgt: tgts.get(i).copied().unwrap_or(0), - relation: rels.get(i).cloned().unwrap_or_default(), - weight: weights.get(i).copied().unwrap_or(1.0), - timestamp: timestamps.get(i).copied().unwrap_or(0.0), - }) - .collect()) -} - -fn group_count(group: &Group<'_>) -> Result { - match group.attrs()?.get("count") { - Some(AttrValue::I64(n)) => Ok(*n as u64), - _ => Ok(0), - } -} diff --git a/crates/clawhdf5-migrate/src/hdf5_writer.rs b/crates/clawhdf5-migrate/src/hdf5_writer.rs deleted file mode 100644 index 8cf1f18..0000000 --- a/crates/clawhdf5-migrate/src/hdf5_writer.rs +++ /dev/null @@ -1,366 +0,0 @@ -use clawhdf5::writer::FileBuilder; -use clawhdf5_format::datatype::{CharacterSet, Datatype, StringPadding}; -use clawhdf5_format::type_builders::AttrValue; - -use crate::sqlite_reader::SqliteData; - -/// Options controlling HDF5 output. -pub struct WriteOptions { - pub agent_id: String, - pub embedder: String, - pub compression: bool, - pub compression_level: u32, - pub float16: bool, -} - -/// Write SQLite data to an HDF5 file. -pub fn write_hdf5( - path: &str, - data: &SqliteData, - opts: &WriteOptions, -) -> Result<(), Box> { - let mut builder = FileBuilder::new(); - let timestamp = iso8601_now(); - - // Root-level metadata attributes - builder.set_attr("agent_id", AttrValue::String(opts.agent_id.clone())); - builder.set_attr("embedder", AttrValue::String(opts.embedder.clone())); - builder.set_attr("embedding_dim", AttrValue::I64(data.embedding_dim as i64)); - builder.set_attr("source", AttrValue::String("sqlite-migration".into())); - builder.set_attr("version", AttrValue::I64(1)); - // Lineage: which SQLite database this output was migrated from and when, - // plus the migrator tool version — so a chain of `--incremental` runs - // still has an audit trail instead of every run overwriting the same - // static attributes (see research/03_provenance.md, INT-03). - builder.set_attr("source_path", AttrValue::String(data.source_path.clone())); - builder.set_attr("migrated_at", AttrValue::String(timestamp.clone())); - builder.set_attr( - "migrator_version", - AttrValue::String(env!("CARGO_PKG_VERSION").to_owned()), - ); - - write_chunks_group(&mut builder, data, opts, ×tamp); - write_sessions_group(&mut builder, data); - write_entities_group(&mut builder, data); - write_relations_group(&mut builder, data); - - builder.write(path)?; - Ok(()) -} - -/// Current UTC time formatted as an ISO-8601 / RFC-3339 timestamp -/// (`YYYY-MM-DDTHH:MM:SSZ`), with no external date/time dependency. -fn iso8601_now() -> String { - let secs = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .unwrap_or_default() - .as_secs(); - let days = (secs / 86_400) as i64; - let time_of_day = secs % 86_400; - let (h, m, s) = ( - time_of_day / 3600, - (time_of_day % 3600) / 60, - time_of_day % 60, - ); - let (y, mo, d) = civil_from_days(days); - format!("{y:04}-{mo:02}-{d:02}T{h:02}:{m:02}:{s:02}Z") -} - -/// Days-since-epoch to (year, month, day), Howard Hinnant's `civil_from_days` -/// algorithm (proleptic Gregorian calendar, valid for the full `i64` range). -fn civil_from_days(z: i64) -> (i64, u32, u32) { - let z = z + 719_468; - let era = if z >= 0 { z } else { z - 146_096 } / 146_097; - let doe = (z - era * 146_097) as u64; // [0, 146096] - let yoe = (doe - doe / 1460 + doe / 36_524 - doe / 146_096) / 365; // [0, 399] - let y = yoe as i64 + era * 400; - let doy = doe - (365 * yoe + yoe / 4 - yoe / 100); // [0, 365] - let mp = (5 * doy + 2) / 153; // [0, 11] - let d = (doy - (153 * mp + 2) / 5 + 1) as u32; // [1, 31] - let m = (if mp < 10 { mp + 3 } else { mp - 9 }) as u32; // [1, 12] - let y = if m <= 2 { y + 1 } else { y }; - (y, m, d) -} - -/// Build a fixed-length string Datatype from the max byte length of the items. -fn string_dtype(max_len: usize) -> Datatype { - Datatype::String { - size: max_len.max(1) as u32, - padding: StringPadding::NullPad, - charset: CharacterSet::Utf8, - } -} - -/// Pack a slice of strings into null-padded raw bytes of uniform width. -fn pack_strings(strings: &[String]) -> (Vec, usize) { - let max_len = strings.iter().map(|s| s.len()).max().unwrap_or(0).max(1); - let mut buf = vec![0u8; strings.len() * max_len]; - for (i, s) in strings.iter().enumerate() { - let start = i * max_len; - let bytes = s.as_bytes(); - let copy_len = bytes.len().min(max_len); - buf[start..start + copy_len].copy_from_slice(&bytes[..copy_len]); - } - (buf, max_len) -} - -fn apply_compression(ds: &mut clawhdf5_format::type_builders::DatasetBuilder, opts: &WriteOptions) { - if opts.compression { - ds.with_deflate(opts.compression_level); - ds.with_shuffle(); - } -} - -fn write_chunks_group( - builder: &mut FileBuilder, - data: &SqliteData, - opts: &WriteOptions, - timestamp: &str, -) { - let mut group = builder.create_group("chunks"); - let n = data.chunks.len() as u64; - - if n == 0 { - group.set_attr("count", AttrValue::I64(0)); - builder.add_group(group.finish()); - return; - } - - group.set_attr("count", AttrValue::I64(n as i64)); - - // Source attribution attached directly to the content-bearing datasets - // (SHA-256 of the raw bytes + creator/timestamp/source), so the chunk - // text and embeddings each carry their own verifiable provenance - // (see clawhdf5_format::provenance / `Dataset::verify_provenance`). - let source_opt = if data.source_path.is_empty() { - None - } else { - Some(data.source_path.as_str()) - }; - - // ids - let ids: Vec = data.chunks.iter().map(|c| c.id).collect(); - group.create_dataset("id").with_i64_data(&ids); - - // text - let texts: Vec = data.chunks.iter().map(|c| c.chunk.clone()).collect(); - let (text_raw, text_len) = pack_strings(&texts); - group - .create_dataset("text") - .with_compound_data(string_dtype(text_len), text_raw, n) - .with_provenance("clawhdf5-migrate", timestamp, source_opt); - - // embeddings - flatten to [N, dim] - let dim = data.embedding_dim; - if opts.float16 { - let f16_data: Vec = data - .chunks - .iter() - .flat_map(|c| { - c.embedding - .iter() - .map(|&v| half::f16::from_f32(v).to_bits()) - }) - .collect(); - let raw: Vec = f16_data.iter().flat_map(|v| v.to_le_bytes()).collect(); - let f16_dtype = Datatype::FloatingPoint { - size: 2, - byte_order: clawhdf5_format::datatype::DatatypeByteOrder::LittleEndian, - bit_offset: 0, - bit_precision: 16, - exponent_location: 10, - exponent_size: 5, - mantissa_location: 0, - mantissa_size: 10, - exponent_bias: 15, - }; - let ds = group - .create_dataset("embeddings") - .with_compound_data(f16_dtype, raw, n) - .with_shape(&[n, dim as u64]) - .with_provenance("clawhdf5-migrate", timestamp, source_opt); - apply_compression(ds, opts); - } else { - let flat: Vec = data - .chunks - .iter() - .flat_map(|c| c.embedding.iter().copied()) - .collect(); - let ds = group - .create_dataset("embeddings") - .with_f32_data(&flat) - .with_shape(&[n, dim as u64]) - .with_provenance("clawhdf5-migrate", timestamp, source_opt); - apply_compression(ds, opts); - } - - // source_channel - let channels: Vec = data - .chunks - .iter() - .map(|c| c.source_channel.clone()) - .collect(); - let (ch_raw, ch_len) = pack_strings(&channels); - group - .create_dataset("source_channel") - .with_compound_data(string_dtype(ch_len), ch_raw, n); - - // timestamp - let timestamps: Vec = data.chunks.iter().map(|c| c.timestamp).collect(); - group.create_dataset("timestamp").with_f64_data(×tamps); - - // session_id - let sess_ids: Vec = data.chunks.iter().map(|c| c.session_id.clone()).collect(); - let (sid_raw, sid_len) = pack_strings(&sess_ids); - group - .create_dataset("session_id") - .with_compound_data(string_dtype(sid_len), sid_raw, n); - - // tags - let tags: Vec = data.chunks.iter().map(|c| c.tags.clone()).collect(); - let (tag_raw, tag_len) = pack_strings(&tags); - group - .create_dataset("tags") - .with_compound_data(string_dtype(tag_len), tag_raw, n); - - // deleted - let deleted: Vec = data.chunks.iter().map(|c| c.deleted).collect(); - group.create_dataset("deleted").with_i32_data(&deleted); - - builder.add_group(group.finish()); -} - -fn write_sessions_group(builder: &mut FileBuilder, data: &SqliteData) { - let mut group = builder.create_group("sessions"); - let n = data.sessions.len() as u64; - group.set_attr("count", AttrValue::I64(n as i64)); - - if n == 0 { - builder.add_group(group.finish()); - return; - } - - let ids: Vec = data.sessions.iter().map(|s| s.id.clone()).collect(); - let (id_raw, id_len) = pack_strings(&ids); - group - .create_dataset("id") - .with_compound_data(string_dtype(id_len), id_raw, n); - - let start_idxs: Vec = data.sessions.iter().map(|s| s.start_idx).collect(); - group.create_dataset("start_idx").with_i64_data(&start_idxs); - - let end_idxs: Vec = data.sessions.iter().map(|s| s.end_idx).collect(); - group.create_dataset("end_idx").with_i64_data(&end_idxs); - - let channels: Vec = data.sessions.iter().map(|s| s.channel.clone()).collect(); - let (ch_raw, ch_len) = pack_strings(&channels); - group - .create_dataset("channel") - .with_compound_data(string_dtype(ch_len), ch_raw, n); - - let timestamps: Vec = data.sessions.iter().map(|s| s.timestamp).collect(); - group.create_dataset("timestamp").with_f64_data(×tamps); - - let summaries: Vec = data.sessions.iter().map(|s| s.summary.clone()).collect(); - let (sum_raw, sum_len) = pack_strings(&summaries); - group - .create_dataset("summary") - .with_compound_data(string_dtype(sum_len), sum_raw, n); - - builder.add_group(group.finish()); -} - -fn write_entities_group(builder: &mut FileBuilder, data: &SqliteData) { - let mut group = builder.create_group("entities"); - let n = data.entities.len() as u64; - group.set_attr("count", AttrValue::I64(n as i64)); - - if n == 0 { - builder.add_group(group.finish()); - return; - } - - let ids: Vec = data.entities.iter().map(|e| e.id).collect(); - group.create_dataset("id").with_i64_data(&ids); - - let names: Vec = data.entities.iter().map(|e| e.name.clone()).collect(); - let (name_raw, name_len) = pack_strings(&names); - group - .create_dataset("name") - .with_compound_data(string_dtype(name_len), name_raw, n); - - let types: Vec = data - .entities - .iter() - .map(|e| e.entity_type.clone()) - .collect(); - let (type_raw, type_len) = pack_strings(&types); - group - .create_dataset("type") - .with_compound_data(string_dtype(type_len), type_raw, n); - - let emb_idxs: Vec = data.entities.iter().map(|e| e.embedding_idx).collect(); - group - .create_dataset("embedding_idx") - .with_i64_data(&emb_idxs); - - builder.add_group(group.finish()); -} - -fn write_relations_group(builder: &mut FileBuilder, data: &SqliteData) { - let mut group = builder.create_group("relations"); - let n = data.relations.len() as u64; - group.set_attr("count", AttrValue::I64(n as i64)); - - if n == 0 { - builder.add_group(group.finish()); - return; - } - - let srcs: Vec = data.relations.iter().map(|r| r.src).collect(); - group.create_dataset("src").with_i64_data(&srcs); - - let tgts: Vec = data.relations.iter().map(|r| r.tgt).collect(); - group.create_dataset("tgt").with_i64_data(&tgts); - - let rels: Vec = data.relations.iter().map(|r| r.relation.clone()).collect(); - let (rel_raw, rel_len) = pack_strings(&rels); - group - .create_dataset("relation") - .with_compound_data(string_dtype(rel_len), rel_raw, n); - - let weights: Vec = data.relations.iter().map(|r| r.weight).collect(); - group.create_dataset("weight").with_f64_data(&weights); - - let timestamps: Vec = data.relations.iter().map(|r| r.timestamp).collect(); - group.create_dataset("timestamp").with_f64_data(×tamps); - - builder.add_group(group.finish()); -} - -#[cfg(test)] -mod time_tests { - use super::civil_from_days; - - #[test] - fn epoch_day_zero_is_1970_01_01() { - assert_eq!(civil_from_days(0), (1970, 1, 1)); - } - - #[test] - fn known_dates_roundtrip() { - // 2026-08-16 is 20,681 days after 1970-01-01. - assert_eq!(civil_from_days(20_681), (2026, 8, 16)); - // 2000-02-29 (leap day itself) and 2000-03-01 (the day after). - assert_eq!(civil_from_days(11_016), (2000, 2, 29)); - assert_eq!(civil_from_days(11_017), (2000, 3, 1)); - } - - #[test] - fn iso8601_now_has_expected_shape() { - let ts = super::iso8601_now(); - assert_eq!(ts.len(), "2026-08-16T00:00:00Z".len()); - assert!(ts.starts_with("20")); // sanity: 21st-century year - assert!(ts.ends_with('Z')); - } -} diff --git a/crates/clawhdf5-migrate/src/main.rs b/crates/clawhdf5-migrate/src/main.rs index 932f826..4a0f58b 100644 --- a/crates/clawhdf5-migrate/src/main.rs +++ b/crates/clawhdf5-migrate/src/main.rs @@ -1,13 +1,21 @@ -mod hdf5_reader; -mod hdf5_writer; mod sqlite_reader; +mod store_writer; mod validate; +use std::path::Path; + use clap::Parser; use sqlite_reader::SchemaConfig; +use store_writer::Migration; +use validate::ValidationSummary; -/// Migrate ZeroClaw agent memory from SQLite to HDF5 format. +type BoxErr = Box; + +/// Migrate ZeroClaw agent memory from SQLite to a clawhdf5-agent store. +/// +/// The output is an ordinary agent store: open it with +/// `HDF5Memory::open` (or `clawhdf5-cli --path ...`). #[derive(Parser, Debug)] #[command(name = "clawhdf5-migrate", version, about)] struct Cli { @@ -15,27 +23,32 @@ struct Cli { #[arg(long)] sqlite: String, - /// Destination HDF5 file path + /// Destination agent store (.h5). Replaced if it exists, unless + /// --incremental #[arg(long)] hdf5: String, - /// Agent ID for metadata + /// Agent ID recorded in the new store #[arg(long, default_value = "migrated")] agent_id: String, - /// Embedder name for metadata + /// Embedder name recorded in the new store #[arg(long, default_value = "unknown")] embedder: String, - /// Embedding dimension (auto-detect from first row if not specified) + /// Embedding dimension (default: detected from the first row). Every row + /// must have it: a row of another length is an error, never truncated. + /// Required when the source has no memory records, so the new store can + /// take records later #[arg(long)] embedding_dim: Option, - /// Skip deleted/tombstoned entries + /// Skip deleted/tombstoned entries (otherwise they are migrated as + /// deleted records) #[arg(long)] skip_deleted: bool, - /// Enable deflate compression on embeddings + /// Compress the embeddings dataset (deflate) #[arg(long)] compression: bool, @@ -43,8 +56,14 @@ struct Cli { #[arg(long, default_value_t = 4)] compression_level: u32, - /// Store embeddings as float16 (halves storage) + /// Store embeddings as full-precision f32 instead of the default half + /// precision (float16: half the bytes, about three significant digits, + /// values within ±65504) #[arg(long)] + f32: bool, + + /// Accepted for compatibility; float16 is now the default + #[arg(long, hide = true, conflicts_with = "f32")] float16: bool, /// Validate without writing @@ -55,8 +74,12 @@ struct Cli { #[arg(long)] validate_full: bool, - /// Append only rows newer than the existing output (by chunk id), merging - /// into the file at --hdf5 if it exists + /// Add to the store at --hdf5 if it exists, writing only source rows it + /// does not already hold (records matched by content, sessions by id, + /// entities by name and type). Matched records take the source's deleted + /// flag: deleted in the source tombstones the record, active in the source + /// writes a deleted record again. The source must have the store's + /// embedding dimension #[arg(long)] incremental: bool, @@ -99,9 +122,15 @@ fn schema_from_cli(cli: &Cli) -> SchemaConfig { c } -fn main() -> Result<(), Box> { - let cli = Cli::parse(); - let schema = schema_from_cli(&cli); +/// The result of a (non-dry) run. +#[derive(Debug)] +struct Outcome { + migration: Migration, + summary: ValidationSummary, +} + +fn run(cli: &Cli) -> Result, BoxErr> { + let schema = schema_from_cli(cli); // Dry run: a fast count-only pass that does not buffer the database. if cli.dry_run { @@ -111,111 +140,134 @@ fn main() -> Result<(), Box> { "Would migrate: {} chunks, {} sessions, {} entities, {} relations", counts.chunks, counts.sessions, counts.entities, counts.relations ); - return Ok(()); + return Ok(None); } + let out = Path::new(&cli.hdf5); + if cli.embedding_dim == Some(0) { + return Err("--embedding-dim must be at least 1".into()); + } + // The source's dimension is its own (--embedding-dim or the first row), + // never the existing store's: a mismatch must be an error, not a reason + // to reshape the source. + // + // An incremental run reads deleted rows even with --skip-deleted, so a + // row deleted in the source since the last run tombstones its store + // record; the writer still leaves out deleted rows the store lacks. + let appending = cli.incremental && out.exists(); + let reader_skip_deleted = cli.skip_deleted && !appending; if cli.verbose { eprintln!("Reading SQLite database: {}", cli.sqlite); } - - // Incremental: merge new rows into the existing output (if present). - let incremental_base = if cli.incremental && std::path::Path::new(&cli.hdf5).exists() { - Some(hdf5_reader::read_hdf5(&cli.hdf5)?) - } else { - None - }; - let min_chunk_id = incremental_base - .as_ref() - .map(|d| d.chunks.iter().map(|c| c.id).max().unwrap_or(0)) - .unwrap_or(0); - let dim_hint = cli - .embedding_dim - .or_else(|| incremental_base.as_ref().map(|d| d.embedding_dim)); - - let source = if min_chunk_id > 0 { - sqlite_reader::read_sqlite_filtered( - &cli.sqlite, - cli.skip_deleted, - dim_hint, - &schema, - min_chunk_id, - )? - } else { - sqlite_reader::read_sqlite(&cli.sqlite, cli.skip_deleted, dim_hint, &schema)? - }; - - // Build the dataset to write: either the source alone, or the existing - // output plus the newly-read rows (metadata groups refreshed from source). - let data = match incremental_base { - Some(mut base) => { - let added = source.chunks.len(); - base.chunks.extend(source.chunks); - base.sessions = source.sessions; - base.entities = source.entities; - base.relations = source.relations; - base.embedding_dim = source.embedding_dim.max(base.embedding_dim); - // Carry the current run's real SQLite source forward for - // provenance — `base` (re-read from the prior HDF5 output) has - // no meaningful source_path of its own. - base.source_path = source.source_path; - if cli.verbose { - eprintln!("Incremental: appended {added} new chunks (id > {min_chunk_id})"); - } - base - } - None => source, - }; - + let data = + sqlite_reader::read_sqlite(&cli.sqlite, reader_skip_deleted, cli.embedding_dim, &schema)?; if cli.verbose { eprintln!( - "Migrating {} chunks, {} sessions, {} entities, {} relations (dim={})", + "Read {} chunks, {} sessions, {} entities, {} relations (dim={})", data.chunks.len(), data.sessions.len(), data.entities.len(), data.relations.len(), data.embedding_dim ); - eprintln!("Writing HDF5 file: {}", cli.hdf5); + eprintln!("Writing agent store: {}", cli.hdf5); } - let opts = hdf5_writer::WriteOptions { - agent_id: cli.agent_id, - embedder: cli.embedder, + let opts = store_writer::WriteOptions { + agent_id: cli.agent_id.clone(), + embedder: cli.embedder.clone(), compression: cli.compression, compression_level: cli.compression_level.clamp(1, 9), - float16: cli.float16, + f32: cli.f32, + incremental: cli.incremental, + skip_deleted: cli.skip_deleted, }; + let migration = store_writer::write_store(out, &data, &opts)?; - hdf5_writer::write_hdf5(&cli.hdf5, &data, &opts)?; - + if migration.appended_to_existing && cli.f32 && migration.float16 { + eprintln!( + "warning: --f32 ignored: the existing store is float16, and a store keeps the \ + precision it was created with" + ); + } + if !migration.dangling_relations.is_empty() { + eprintln!( + "warning: skipped {} relation(s) naming an entity id not in the entities table", + migration.dangling_relations.len() + ); + } + if !migration.deleted_in_store.is_empty() || migration.restored > 0 { + eprintln!( + "Incremental: {} store record(s) deleted and {} restored (written again) to match \ + the source's deleted flags", + migration.deleted_in_store.len(), + migration.restored + ); + } if cli.verbose { - eprintln!("Validating output (content check)..."); + if migration.appended_to_existing { + eprintln!( + "Incremental: already in the store: {} records, {} sessions, {} entities, {} relations", + migration.chunks_present, + migration.sessions_present, + migration.entities_present, + migration.relations_present + ); + } + if !migration.anomaly_alerts.is_empty() { + eprintln!( + "Note: the agent's write-anomaly detector raised {} alert(s) during the import \ + (informational; nothing was blocked), e.g.:", + migration.anomaly_alerts.len() + ); + for msg in migration.anomaly_alerts.iter().take(3) { + eprintln!(" {msg}"); + } + } + eprintln!("Validating output (reading it back with HDF5Memory::open_read_only)..."); } - let summary = validate::validate_hdf5(&cli.hdf5, &data, cli.validate_full, cli.float16)?; + let summary = validate::validate_store(out, &data, &migration, cli.validate_full)?; + Ok(Some(Outcome { migration, summary })) +} +fn main() -> Result<(), BoxErr> { + let cli = Cli::parse(); + let Some(Outcome { migration, summary }) = run(&cli)? else { + return Ok(()); + }; eprintln!( - "Migration complete: {} chunks, {} sessions, {} entities, {} relations (dim={}); {} rows content-verified", - summary.chunks, + "Migration complete: wrote {} records, {} sessions, {} entities, {} relations. \ + Store: {} records ({} active), {} sessions, {} entities, {} relations, dim={}, {}; \ + {} rows content-verified{}", + migration.records.len(), + migration.sessions.len(), + migration.entities.len(), + migration.relations.len(), + summary.count, + summary.active, summary.sessions, summary.entities, summary.relations, summary.embedding_dim, + if summary.float16 { "float16" } else { "f32" }, summary.rows_checked, + if summary.search_checked { + ", search verified" + } else { + "" + }, ); - if summary.provenance_verified { - eprintln!("Provenance: chunks/text and chunks/embeddings SHA-256 hashes verified."); - } else if cli.verbose { - eprintln!("Provenance: no provenance hash found to verify (older output format?)."); - } - Ok(()) } #[cfg(test)] mod tests { use super::*; + use clawhdf5_agent::{AgentMemory, HDF5Memory, SearchOptions}; + use clawhdf5_format::float16::round_to_f16; use rusqlite::Connection; + use std::path::PathBuf; use tempfile::TempDir; /// Create a test SQLite database with the ZeroClaw schema. @@ -303,12 +355,57 @@ mod tests { (0..dim).map(|i| seed + i as f32 * 0.1).collect() } - // ---------- Test 1: Basic end-to-end migration ---------- + /// A unit-length embedding pointing mostly along axis `axis`, so records + /// are distinguishable by vector search. + fn axis_embedding(dim: usize, axis: usize) -> Vec { + let mut v: Vec = (0..dim).map(|i| 0.01 * (i as f32 + 1.0)).collect(); + v[axis % dim] = 1.0; + let n = v.iter().map(|x| x * x).sum::().sqrt(); + v.iter().map(|x| x / n).collect() + } + + fn out_path(dir: &TempDir, name: &str) -> PathBuf { + dir.path().join(name) + } + + /// Run the CLI exactly as `clawhdf5-migrate --sqlite --hdf5 `. + fn migrate(db: &str, out: &Path, extra: &[&str]) -> Result, BoxErr> { + let mut args = vec![ + "clawhdf5-migrate", + "--sqlite", + db, + "--hdf5", + out.to_str().unwrap(), + ]; + args.extend_from_slice(extra); + run(&Cli::try_parse_from(args)?) + } + + fn migrate_ok(db: &str, out: &Path, extra: &[&str]) -> Outcome { + migrate(db, out, extra).unwrap().expect("not a dry run") + } + + fn write(db: &str, out: &Path, opts_f32: bool) -> (sqlite_reader::SqliteData, Migration) { + let data = sqlite_reader::read_sqlite(db, false, None, &SchemaConfig::default()).unwrap(); + let opts = store_writer::WriteOptions { + agent_id: "t".into(), + embedder: "t".into(), + compression: false, + compression_level: 4, + f32: opts_f32, + incremental: false, + skip_deleted: false, + }; + let m = store_writer::write_store(out, &data, &opts).unwrap(); + (data, m) + } + + // ---------- Basic end-to-end migration ---------- #[test] fn test_basic_migration() { let dir = TempDir::new().unwrap(); let db_path = create_test_db(&dir); - let h5_path = dir.path().join("out.h5"); + let h5_path = out_path(&dir, "out.h5"); let conn = Connection::open(&db_path).unwrap(); insert_chunk(&conn, 1, "hello world", &make_embedding(8, 1.0), 0); @@ -318,32 +415,193 @@ mod tests { insert_relation(&conn, 1, 1, "self"); drop(conn); - let data = - sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap(); - let opts = hdf5_writer::WriteOptions { - agent_id: "test-agent".into(), - embedder: "test-embed".into(), - compression: false, - compression_level: 4, - float16: false, - }; - hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap(); - - let summary = - validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap(); - assert_eq!(summary.chunks, 2); - assert_eq!(summary.sessions, 1); - assert_eq!(summary.entities, 1); - assert_eq!(summary.relations, 1); - assert_eq!(summary.embedding_dim, 8); + let o = migrate_ok(&db_path, &h5_path, &["--agent-id", "test-agent"]); + assert_eq!(o.summary.count, 2); + assert_eq!(o.summary.sessions, 1); + assert_eq!(o.summary.entities, 1); + assert_eq!(o.summary.relations, 1); + assert_eq!(o.summary.embedding_dim, 8); + assert!(o.summary.float16, "float16 is the default"); + assert!(o.summary.search_checked); } - // ---------- Test 2: Skip deleted rows ---------- + // ---------- The migrated file is a real agent store ---------- + + /// Build a source with distinct records, sessions and a small graph, + /// migrate it with `extra` flags, and use the result as an agent would. + fn end_to_end(extra: &[&str], expect_float16: bool) { + let dir = TempDir::new().unwrap(); + let db_path = create_test_db(&dir); + let h5_path = out_path(&dir, "agent.h5"); + let dim = 16; + let texts = [ + "the deploy key rotates every ninety days", + "alice prefers tea over coffee in the morning", + "the build cache lives on the tank runner", + "bob is allergic to peanuts", + "quarterly review is scheduled for october", + ]; + + let conn = Connection::open(&db_path).unwrap(); + for (i, t) in texts.iter().enumerate() { + insert_chunk(&conn, i as i64 + 1, t, &axis_embedding(dim, i * 3), 0); + } + insert_chunk(&conn, 99, "a forgotten memory", &axis_embedding(dim, 15), 1); + conn.execute( + "INSERT INTO sessions VALUES ('sess-a', 0, 2, 'discord', 1700000123.5, 'morning chat')", + [], + ) + .unwrap(); + insert_session(&conn, "sess-b", 3, 5); + insert_entity(&conn, 10, "Alice", "person"); + insert_entity(&conn, 20, "Bob", "person"); + insert_entity(&conn, 30, "tank", "machine"); + insert_relation(&conn, 10, 20, "knows"); + conn.execute( + "INSERT INTO relations VALUES (20, 30, 'uses', 0.25, 1700000456.0)", + [], + ) + .unwrap(); + drop(conn); + + let mut args = vec![ + "--agent-id", + "e2e-agent", + "--embedder", + "minilm", + "--validate-full", + ]; + args.extend_from_slice(extra); + let o = migrate_ok(&db_path, &h5_path, &args); + assert_eq!(o.summary.float16, expect_float16); + assert_eq!(o.summary.count, 6); + assert_eq!(o.summary.active, 5); + + // Read-only view: the whole store, as on disk. + let source = + sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap(); + let ro = HDF5Memory::open_read_only(&h5_path).unwrap(); + assert_eq!(ro.count(), 6); + assert_eq!(ro.count_active(), 5); + assert_eq!(ro.config().agent_id, "e2e-agent"); + assert_eq!(ro.config().embedder, "minilm"); + assert_eq!(ro.config().embedding_dim, dim); + assert_eq!(ro.config().float16, expect_float16); + for (i, c) in source.chunks.iter().enumerate() { + let want: Vec = c + .embedding + .iter() + .map(|&v| if expect_float16 { round_to_f16(v) } else { v }.to_bits()) + .collect(); + let got: Vec = ro.cache.embeddings[i].iter().map(|v| v.to_bits()).collect(); + assert_eq!(got, want, "record {i}"); + } + drop(ro); + + // Writable open: a real agent store, searchable, with sessions and KG. + let mut mem = HDF5Memory::open(&h5_path).unwrap(); + assert_eq!(mem.count(), 6); + let hits = mem.hybrid_search(&axis_embedding(dim, 9), "allergic peanuts", 0.7, 0.3, 3); + assert_eq!(hits[0].chunk, texts[3], "{hits:?}"); + let hits = mem.search( + &axis_embedding(dim, 6), + "build cache runner", + &SearchOptions::new(3).with_sources(["api"]), + ); + assert_eq!(hits[0].chunk, texts[2], "{hits:?}"); + // The deleted source row is a tombstone: never returned. + let hits = mem.hybrid_search(&axis_embedding(dim, 15), "forgotten memory", 0.5, 0.5, 10); + assert!( + hits.iter().all(|h| h.chunk != "a forgotten memory"), + "{hits:?}" + ); + + assert_eq!( + mem.get_session_summary("sess-a").unwrap().as_deref(), + Some("morning chat") + ); + let s = &mem.sessions().entries[0]; + assert_eq!( + (s.start_idx, s.end_idx, s.channel.as_str()), + (0, 2, "discord") + ); + assert_eq!(s.ts, 1700000123.5 * 1e6, "seconds -> microseconds"); + assert_eq!(mem.sessions().len(), 2); + + let kg = mem.knowledge(); + assert_eq!(kg.entities.len(), 3); + assert_eq!(kg.relations.len(), 2); + let name = |id: u64| kg.get_entity(id).unwrap().name.clone(); + let rel: Vec<(String, String, String, f32)> = kg + .relations + .iter() + .map(|r| (name(r.src), r.relation.clone(), name(r.tgt), r.weight)) + .collect(); + assert_eq!( + rel, + vec![ + ("Alice".into(), "knows".into(), "Bob".into(), 1.0), + ("Bob".into(), "uses".into(), "tank".into(), 0.25), + ] + ); + assert_eq!(kg.relations[1].ts, 1700000456.0 * 1e6); + assert_eq!(kg.get_entity(2).unwrap().entity_type, "machine"); + + // And it keeps working as a store: a new save survives a reopen. + mem.save(clawhdf5_agent::MemoryEntry { + chunk: "saved after migration".into(), + embedding: axis_embedding(dim, 1), + source_channel: "api".into(), + timestamp: 1.0, + session_id: String::new(), + tags: String::new(), + }) + .unwrap(); + drop(mem); + assert_eq!(HDF5Memory::open(&h5_path).unwrap().count(), 7); + } + + #[test] + fn end_to_end_float16_default() { + end_to_end(&[], true); + } + + #[test] + fn end_to_end_f32() { + end_to_end(&["--f32"], false); + } + + #[test] + fn hidden_float16_flag_is_a_no_op() { + end_to_end(&["--float16"], true); + } + + #[test] + fn cli_precision_flags() { + let base = ["m", "--sqlite", "a.db", "--hdf5", "b.h5"]; + let parse = |extra: &[&str]| { + let mut args = base.to_vec(); + args.extend_from_slice(extra); + Cli::try_parse_from(args) + }; + let c = parse(&[]).unwrap(); + assert!(!c.f32 && !c.float16); + assert!(parse(&["--f32"]).unwrap().f32); + assert!(parse(&["--float16"]).is_ok()); + assert!(parse(&["--f32", "--float16"]).is_err()); + // --float16 stays accepted but out of the help text. + use clap::CommandFactory; + let help = Cli::command().render_long_help().to_string(); + assert!(help.contains("--f32")); + assert!(!help.contains("--float16"), "{help}"); + } + + // ---------- Skip deleted rows ---------- #[test] fn test_skip_deleted() { let dir = TempDir::new().unwrap(); let db_path = create_test_db(&dir); - let h5_path = dir.path().join("out.h5"); + let h5_path = out_path(&dir, "out.h5"); let conn = Connection::open(&db_path).unwrap(); insert_chunk(&conn, 1, "active", &make_embedding(4, 1.0), 0); @@ -351,41 +609,31 @@ mod tests { insert_chunk(&conn, 3, "also active", &make_embedding(4, 3.0), 0); drop(conn); - let data = - sqlite_reader::read_sqlite(&db_path, true, None, &SchemaConfig::default()).unwrap(); - assert_eq!(data.chunks.len(), 2); - - let opts = hdf5_writer::WriteOptions { - agent_id: "t".into(), - embedder: "t".into(), - compression: false, - compression_level: 4, - float16: false, - }; - hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap(); - - let summary = - validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap(); - assert_eq!(summary.chunks, 2); + let o = migrate_ok(&db_path, &h5_path, &["--skip-deleted"]); + assert_eq!(o.summary.count, 2); + assert_eq!(o.summary.active, 2); } - // ---------- Test 3: Include deleted rows ---------- + // ---------- Include deleted rows (as tombstones) ---------- #[test] fn test_include_deleted() { let dir = TempDir::new().unwrap(); let db_path = create_test_db(&dir); + let h5_path = out_path(&dir, "out.h5"); let conn = Connection::open(&db_path).unwrap(); insert_chunk(&conn, 1, "active", &make_embedding(4, 1.0), 0); insert_chunk(&conn, 2, "deleted", &make_embedding(4, 2.0), 1); drop(conn); - let data = - sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap(); - assert_eq!(data.chunks.len(), 2); + let o = migrate_ok(&db_path, &h5_path, &[]); + assert_eq!(o.summary.count, 2); + assert_eq!(o.summary.active, 1); + let mem = HDF5Memory::open_read_only(&h5_path).unwrap(); + assert_eq!(mem.cache.tombstones, vec![0, 1]); } - // ---------- Test 4: Auto-detect embedding dimension ---------- + // ---------- Auto-detect embedding dimension ---------- #[test] fn test_auto_detect_dim() { let dir = TempDir::new().unwrap(); @@ -400,11 +648,12 @@ mod tests { assert_eq!(data.embedding_dim, 16); } - // ---------- Test 5: Manual embedding dimension ---------- + // ---------- Manual embedding dimension ---------- #[test] fn test_manual_dim() { let dir = TempDir::new().unwrap(); let db_path = create_test_db(&dir); + let h5_path = out_path(&dir, "out.h5"); let conn = Connection::open(&db_path).unwrap(); insert_chunk(&conn, 1, "test", &make_embedding(16, 0.5), 0); @@ -413,56 +662,225 @@ mod tests { let data = sqlite_reader::read_sqlite(&db_path, false, Some(8), &SchemaConfig::default()).unwrap(); assert_eq!(data.embedding_dim, 8); - // Embedding truncated to dim 8 - assert_eq!(data.chunks[0].embedding.len(), 8); + // Read at full length, never truncated to the requested dimension... + assert_eq!(data.chunks[0].embedding.len(), 16); + + // ...so a --embedding-dim that disagrees with the data is an error. + let err = migrate(&db_path, &h5_path, &["--embedding-dim", "8"]) + .unwrap_err() + .to_string(); + assert!( + err.contains("chunk id 1") && err.contains("16 values, expected 8"), + "{err}" + ); + assert!(!h5_path.exists(), "nothing written"); + let o = migrate_ok(&db_path, &h5_path, &["--embedding-dim", "16"]); + assert_eq!(o.summary.embedding_dim, 16); + let err = migrate(&db_path, &h5_path, &["--embedding-dim", "0"]) + .unwrap_err() + .to_string(); + assert!(err.contains("at least 1"), "{err}"); } - // ---------- Test 6: Float16 conversion ---------- + // ---------- A row longer than the first is an error, not truncated ---------- + #[test] + fn long_embedding_is_rejected() { + let dir = TempDir::new().unwrap(); + let db_path = create_test_db(&dir); + let h5_path = out_path(&dir, "out.h5"); + + let conn = Connection::open(&db_path).unwrap(); + insert_chunk(&conn, 1, "eight", &make_embedding(8, 0.5), 0); + insert_chunk(&conn, 2, "twelve", &make_embedding(12, 0.5), 0); + drop(conn); + + let err = migrate(&db_path, &h5_path, &[]).unwrap_err().to_string(); + assert!( + err.contains("chunk id 2") && err.contains("12 values, expected 8"), + "{err}" + ); + assert!(!h5_path.exists(), "nothing written"); + } + + // ---------- A short first row does not set a small dimension for all ---------- + #[test] + fn short_first_row_is_rejected_not_imposed() { + let dir = TempDir::new().unwrap(); + let db_path = create_test_db(&dir); + let h5_path = out_path(&dir, "out.h5"); + + let conn = Connection::open(&db_path).unwrap(); + insert_chunk(&conn, 1, "four", &make_embedding(4, 0.5), 0); + insert_chunk(&conn, 2, "eight", &make_embedding(8, 0.5), 0); + insert_chunk(&conn, 3, "eight too", &make_embedding(8, 0.7), 0); + drop(conn); + + let err = migrate(&db_path, &h5_path, &[]).unwrap_err().to_string(); + assert!(err.contains("chunk id 2"), "{err}"); + let err = migrate(&db_path, &h5_path, &["--embedding-dim", "8"]) + .unwrap_err() + .to_string(); + assert!( + err.contains("chunk id 1") && err.contains("4 values, expected 8"), + "{err}" + ); + assert!(!h5_path.exists(), "nothing written"); + } + + // ---------- A BLOB that is not whole f32 values is an error ---------- + #[test] + fn ragged_embedding_blob_is_rejected() { + let dir = TempDir::new().unwrap(); + let db_path = create_test_db(&dir); + let h5_path = out_path(&dir, "out.h5"); + + let conn = Connection::open(&db_path).unwrap(); + insert_chunk(&conn, 1, "ok", &make_embedding(4, 0.5), 0); + conn.execute( + "INSERT INTO memory_chunks (id, chunk, embedding, timestamp, deleted) + VALUES (2, 'ragged', ?1, 1700000000.0, 0)", + [vec![0u8; 18]], + ) + .unwrap(); + drop(conn); + + let err = migrate(&db_path, &h5_path, &[]).unwrap_err().to_string(); + assert!( + err.contains("chunk id 2") && err.contains("18 bytes"), + "{err}" + ); + } + + // ---------- Rows with no embedding are rejected before anything is written ---------- + #[test] + fn empty_embeddings_are_rejected_and_keep_the_old_store() { + let dir = TempDir::new().unwrap(); + let db_path = create_test_db(&dir); + let h5_path = out_path(&dir, "out.h5"); + + let good = dir.path().join("good.db"); + std::fs::copy(&db_path, &good).unwrap(); + let conn = Connection::open(&good).unwrap(); + insert_chunk(&conn, 1, "one", &make_embedding(4, 1.0), 0); + drop(conn); + migrate_ok(good.to_str().unwrap(), &h5_path, &[]); + + let conn = Connection::open(&db_path).unwrap(); + insert_chunk(&conn, 1, "a", &[], 0); + insert_chunk(&conn, 2, "b", &[], 0); + drop(conn); + for extra in [&[][..], &["--embedding-dim", "4"][..]] { + let err = migrate(&db_path, &h5_path, extra).unwrap_err().to_string(); + assert!( + err.contains("chunk id 1") && err.contains("embedding is empty"), + "{extra:?}: {err}" + ); + } + assert_eq!(HDF5Memory::open(&h5_path).unwrap().count(), 1); + + // The same into a path with nothing there: no store is left behind. + let fresh = out_path(&dir, "fresh.h5"); + assert!(migrate(&db_path, &fresh, &[]).is_err()); + assert!(!fresh.exists()); + } + + // ---------- A failed run leaves the existing store as it was ---------- + #[test] + fn failed_run_keeps_the_existing_store() { + let dir = TempDir::new().unwrap(); + let db_path = create_test_db(&dir); + let h5_path = out_path(&dir, "out.h5"); + + let conn = Connection::open(&db_path).unwrap(); + for i in 1..=3 { + insert_chunk(&conn, i, &format!("r{i}"), &make_embedding(4, i as f32), 0); + } + drop(conn); + migrate_ok(&db_path, &h5_path, &[]); + + let big = dir.path().join("big.db"); + let big = big.to_str().unwrap(); + std::fs::copy(&db_path, big).unwrap(); + let conn = Connection::open(big).unwrap(); + conn.execute("DELETE FROM memory_chunks", []).unwrap(); + insert_chunk(&conn, 1, "huge", &[70000.0, 0.0, 0.0, 0.0], 0); + drop(conn); + + let err = migrate(big, &h5_path, &[]).unwrap_err().to_string(); + assert!(err.contains("--f32"), "{err}"); + let mem = HDF5Memory::open(&h5_path).unwrap(); + assert_eq!(mem.count(), 3); + assert_eq!(mem.cache.chunks, ["r1", "r2", "r3"]); + } + + // ---------- A row with a short embedding is an error, not zero-padded ---------- + #[test] + fn short_embedding_is_rejected() { + let dir = TempDir::new().unwrap(); + let db_path = create_test_db(&dir); + let h5_path = out_path(&dir, "out.h5"); + + let conn = Connection::open(&db_path).unwrap(); + insert_chunk(&conn, 1, "full", &make_embedding(8, 0.5), 0); + insert_chunk(&conn, 2, "short", &make_embedding(5, 0.5), 0); + drop(conn); + + let err = migrate(&db_path, &h5_path, &[]).unwrap_err().to_string(); + assert!( + err.contains("chunk id 2") && err.contains("expected 8"), + "{err}" + ); + } + + // ---------- Float16: exact half-precision values ---------- #[test] fn test_float16_conversion() { let dir = TempDir::new().unwrap(); let db_path = create_test_db(&dir); - let h5_path = dir.path().join("out.h5"); + let h5_path = out_path(&dir, "out.h5"); let conn = Connection::open(&db_path).unwrap(); - let emb = vec![1.0f32, 2.5, -0.5, 3.125]; + // 0.1 and 1/3 are not representable in half precision. + let emb = vec![1.0f32, 0.1, -1.0 / 3.0, 3.125]; insert_chunk(&conn, 1, "test", &emb, 0); drop(conn); - let data = - sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap(); - let opts = hdf5_writer::WriteOptions { - agent_id: "t".into(), - embedder: "t".into(), - compression: false, - compression_level: 4, - float16: true, - }; - hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap(); - - // Content-validate with the float16 tolerance enabled. - let summary = - validate::validate_hdf5(h5_path.to_str().unwrap(), &data, true, true).unwrap(); - assert_eq!(summary.chunks, 1); - - // Verify float16 values are within tolerance - for &v in &emb { - let f16 = half::f16::from_f32(v); - let roundtrip = f16.to_f32(); - assert!( - (v - roundtrip).abs() < 0.01, - "f16 roundtrip too lossy for {v}" - ); + let o = migrate_ok(&db_path, &h5_path, &["--validate-full"]); + assert!(o.summary.float16); + let mem = HDF5Memory::open_read_only(&h5_path).unwrap(); + let got = &mem.cache.embeddings[0]; + for (&v, &g) in emb.iter().zip(got) { + assert_eq!(g.to_bits(), round_to_f16(v).to_bits()); } + assert_ne!(got[1], 0.1, "stored at half precision"); } - // ---------- Test 7: Compression produces valid file ---------- + // ---------- Float16 range: an error that points at --f32 ---------- + #[test] + fn value_beyond_half_range_needs_f32() { + let dir = TempDir::new().unwrap(); + let db_path = create_test_db(&dir); + let h5_path = out_path(&dir, "out.h5"); + + let conn = Connection::open(&db_path).unwrap(); + insert_chunk(&conn, 7, "huge", &[1.0, 70000.0, 0.0, 0.0], 0); + drop(conn); + + let err = migrate(&db_path, &h5_path, &[]).unwrap_err().to_string(); + assert!(err.contains("chunk id 7") && err.contains("--f32"), "{err}"); + let o = migrate_ok(&db_path, &h5_path, &["--f32"]); + assert!(!o.summary.float16); + let mem = HDF5Memory::open_read_only(&h5_path).unwrap(); + assert_eq!(mem.cache.embeddings[0][1], 70000.0); + } + + // ---------- Compression produces a smaller valid store ---------- #[test] fn test_compression() { let dir = TempDir::new().unwrap(); let db_path = create_test_db(&dir); - let h5_compressed = dir.path().join("compressed.h5"); - let h5_uncompressed = dir.path().join("uncompressed.h5"); + let h5_compressed = out_path(&dir, "compressed.h5"); + let h5_uncompressed = out_path(&dir, "uncompressed.h5"); let conn = Connection::open(&db_path).unwrap(); // Insert enough data so compression can be effective @@ -471,27 +889,17 @@ mod tests { } drop(conn); - let data = - sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap(); - - let opts_compressed = hdf5_writer::WriteOptions { - agent_id: "t".into(), - embedder: "t".into(), - compression: true, - compression_level: 6, - float16: false, - }; - hdf5_writer::write_hdf5(h5_compressed.to_str().unwrap(), &data, &opts_compressed).unwrap(); - - let opts_plain = hdf5_writer::WriteOptions { - agent_id: "t".into(), - embedder: "t".into(), - compression: false, - compression_level: 4, - float16: false, - }; - hdf5_writer::write_hdf5(h5_uncompressed.to_str().unwrap(), &data, &opts_plain).unwrap(); + let o = migrate_ok( + &db_path, + &h5_compressed, + &["--compression", "--compression-level", "6"], + ); + assert_eq!(o.summary.count, 100); + migrate_ok(&db_path, &h5_uncompressed, &[]); + let mem = HDF5Memory::open_read_only(&h5_compressed).unwrap(); + assert!(mem.config().compression); + assert_eq!(mem.config().compression_level, 6); let sz_c = std::fs::metadata(&h5_compressed).unwrap().len(); let sz_u = std::fs::metadata(&h5_uncompressed).unwrap().len(); assert!( @@ -500,30 +908,31 @@ mod tests { ); } - // ---------- Test 8: Dry run doesn't create file ---------- + // ---------- Dry run doesn't create file ---------- #[test] fn test_dry_run() { let dir = TempDir::new().unwrap(); let db_path = create_test_db(&dir); - let h5_path = dir.path().join("should_not_exist.h5"); + let h5_path = out_path(&dir, "should_not_exist.h5"); let conn = Connection::open(&db_path).unwrap(); insert_chunk(&conn, 1, "test", &make_embedding(4, 1.0), 0); drop(conn); - // Simulate dry-run: read data but don't write - let data = - sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap(); - assert_eq!(data.chunks.len(), 1); + assert!( + migrate(&db_path, &h5_path, &["--dry-run"]) + .unwrap() + .is_none() + ); assert!(!h5_path.exists()); } - // ---------- Test 9: Empty database migration ---------- + // ---------- Empty database migration ---------- #[test] fn test_empty_db() { let dir = TempDir::new().unwrap(); let db_path = create_test_db(&dir); - let h5_path = dir.path().join("out.h5"); + let h5_path = out_path(&dir, "out.h5"); let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap(); @@ -532,26 +941,51 @@ mod tests { assert_eq!(data.entities.len(), 0); assert_eq!(data.relations.len(), 0); - let opts = hdf5_writer::WriteOptions { - agent_id: "t".into(), - embedder: "t".into(), - compression: false, - compression_level: 4, - float16: false, - }; - hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap(); + // No record to detect the dimension from: a store with dimension 0 + // could never take a record, so --embedding-dim is required. + let err = migrate(&db_path, &h5_path, &[]).unwrap_err().to_string(); + assert!(err.contains("--embedding-dim"), "{err}"); + assert!(!h5_path.exists()); - let summary = - validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap(); - assert_eq!(summary.chunks, 0); + let o = migrate_ok(&db_path, &h5_path, &["--embedding-dim", "8"]); + assert_eq!(o.summary.count, 0); + assert_eq!(o.summary.embedding_dim, 8); + assert!(!o.summary.search_checked); + assert_eq!(HDF5Memory::open(&h5_path).unwrap().count(), 0); + + // ...and a later incremental run can add records to it. + let conn = Connection::open(&db_path).unwrap(); + insert_chunk(&conn, 1, "later", &make_embedding(8, 0.5), 0); + drop(conn); + let o = migrate_ok(&db_path, &h5_path, &["--incremental"]); + assert_eq!((o.summary.count, o.summary.embedding_dim), (1, 8)); + assert!(o.summary.search_checked); } - // ---------- Test 10: Large migration (1000 entries) ---------- + // ---------- A source with only sessions/graph still needs a dimension ---------- + #[test] + fn graph_only_source_needs_embedding_dim() { + let dir = TempDir::new().unwrap(); + let db_path = create_test_db(&dir); + let h5_path = out_path(&dir, "out.h5"); + + let conn = Connection::open(&db_path).unwrap(); + insert_entity(&conn, 1, "Alice", "person"); + insert_session(&conn, "s1", 0, 0); + drop(conn); + let err = migrate(&db_path, &h5_path, &[]).unwrap_err().to_string(); + assert!(err.contains("--embedding-dim"), "{err}"); + let o = migrate_ok(&db_path, &h5_path, &["--embedding-dim", "4"]); + assert_eq!((o.summary.entities, o.summary.sessions), (1, 1)); + assert_eq!(o.summary.embedding_dim, 4); + } + + // ---------- Large migration (1000 entries) ---------- #[test] fn test_large_migration() { let dir = TempDir::new().unwrap(); let db_path = create_test_db(&dir); - let h5_path = dir.path().join("out.h5"); + let h5_path = out_path(&dir, "out.h5"); let conn = Connection::open(&db_path).unwrap(); for i in 0..1000 { @@ -565,30 +999,21 @@ mod tests { } drop(conn); - let data = - sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap(); - assert_eq!(data.chunks.len(), 1000); - - let opts = hdf5_writer::WriteOptions { - agent_id: "t".into(), - embedder: "t".into(), - compression: false, - compression_level: 4, - float16: false, - }; - hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap(); - - let summary = - validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap(); - assert_eq!(summary.chunks, 1000); + let o = migrate_ok(&db_path, &h5_path, &[]); + assert_eq!(o.summary.count, 1000); + assert!(o.summary.search_checked); + // Sampled, not every row, without --validate-full. + assert!(o.summary.rows_checked < 1000); + let o = migrate_ok(&db_path, &h5_path, &["--validate-full"]); + assert_eq!(o.summary.rows_checked, 1000); } - // ---------- Test 11: Session migration ---------- + // ---------- Session migration ---------- #[test] fn test_session_migration() { let dir = TempDir::new().unwrap(); let db_path = create_test_db(&dir); - let h5_path = dir.path().join("out.h5"); + let h5_path = out_path(&dir, "out.h5"); let conn = Connection::open(&db_path).unwrap(); insert_session(&conn, "session-alpha", 0, 10); @@ -596,30 +1021,28 @@ mod tests { insert_session(&conn, "session-gamma", 21, 30); drop(conn); - let data = - sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap(); - assert_eq!(data.sessions.len(), 3); - - let opts = hdf5_writer::WriteOptions { - agent_id: "t".into(), - embedder: "t".into(), - compression: false, - compression_level: 4, - float16: false, - }; - hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap(); - - let summary = - validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap(); - assert_eq!(summary.sessions, 3); + let o = migrate_ok(&db_path, &h5_path, &["--embedding-dim", "8"]); + assert_eq!(o.summary.sessions, 3); + let mem = HDF5Memory::open(&h5_path).unwrap(); + let ids: Vec<&str> = mem + .sessions() + .entries + .iter() + .map(|e| e.id.as_str()) + .collect(); + assert_eq!(ids, ["session-alpha", "session-beta", "session-gamma"]); + assert_eq!( + mem.get_session_summary("session-beta").unwrap().as_deref(), + Some("test summary") + ); } - // ---------- Test 12: Knowledge graph (entities + relations) ---------- + // ---------- Knowledge graph (entities + relations) ---------- #[test] fn test_knowledge_graph_migration() { let dir = TempDir::new().unwrap(); let db_path = create_test_db(&dir); - let h5_path = dir.path().join("out.h5"); + let h5_path = out_path(&dir, "out.h5"); let conn = Connection::open(&db_path).unwrap(); insert_entity(&conn, 1, "Alice", "person"); @@ -630,145 +1053,102 @@ mod tests { insert_relation(&conn, 2, 3, "uses"); drop(conn); - let data = - sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap(); - assert_eq!(data.entities.len(), 3); - assert_eq!(data.relations.len(), 3); - - let opts = hdf5_writer::WriteOptions { - agent_id: "t".into(), - embedder: "t".into(), - compression: false, - compression_level: 4, - float16: false, - }; - hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap(); - - let summary = - validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap(); - assert_eq!(summary.entities, 3); - assert_eq!(summary.relations, 3); + let o = migrate_ok(&db_path, &h5_path, &["--embedding-dim", "8"]); + assert_eq!(o.summary.entities, 3); + assert_eq!(o.summary.relations, 3); + let mem = HDF5Memory::open(&h5_path).unwrap(); + let kg = mem.knowledge(); + let alice = o.migration.entity_ids[&1]; + let targets: Vec<&str> = kg + .get_relations_from(alice) + .iter() + .map(|r| kg.get_entity(r.tgt).unwrap().name.as_str()) + .collect(); + assert_eq!(targets, ["Bob", "Rust"]); } - // ---------- Test 13: Validation catches chunk count mismatch ---------- + // ---------- A relation to an unknown entity is skipped and reported ---------- + #[test] + fn dangling_relation_is_reported() { + let dir = TempDir::new().unwrap(); + let db_path = create_test_db(&dir); + let h5_path = out_path(&dir, "out.h5"); + + let conn = Connection::open(&db_path).unwrap(); + insert_entity(&conn, 1, "Alice", "person"); + insert_relation(&conn, 1, 1, "self"); + // A database that did not enforce its foreign keys. + conn.execute_batch("PRAGMA foreign_keys = OFF").unwrap(); + insert_relation(&conn, 1, 42, "knows"); + drop(conn); + + let o = migrate_ok(&db_path, &h5_path, &["--embedding-dim", "8"]); + assert_eq!(o.summary.relations, 1); + assert_eq!(o.migration.dangling_relations, vec![1]); + } + + // ---------- Validation catches a record count mismatch ---------- #[test] fn test_validation_catches_count_mismatch() { let dir = TempDir::new().unwrap(); let db_path = create_test_db(&dir); - let h5_path = dir.path().join("out.h5"); + let h5_path = out_path(&dir, "out.h5"); let conn = Connection::open(&db_path).unwrap(); insert_chunk(&conn, 1, "test", &make_embedding(4, 1.0), 0); drop(conn); - let data = - sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap(); - let opts = hdf5_writer::WriteOptions { - agent_id: "t".into(), - embedder: "t".into(), - compression: false, - compression_level: 4, - float16: false, - }; - hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap(); - - // Validating against a source with an extra (unwritten) chunk must fail. - let mut bigger = - sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap(); - let mut extra = bigger.chunks[0].clone(); - extra.id = 999; - bigger.chunks.push(extra); - let result = validate::validate_hdf5(h5_path.to_str().unwrap(), &bigger, false, false); - assert!(result.is_err()); + let (data, mut m) = write(&db_path, &h5_path, false); + m.store_count += 1; + let result = validate::validate_store(&h5_path, &data, &m, false); assert!(result.unwrap_err().to_string().contains("count mismatch")); } - // ---------- Test 14: Metadata attributes are stored ---------- + // ---------- Metadata is stored in the agent config ---------- #[test] fn test_metadata_attributes() { let dir = TempDir::new().unwrap(); let db_path = create_test_db(&dir); - let h5_path = dir.path().join("out.h5"); + let h5_path = out_path(&dir, "out.h5"); let conn = Connection::open(&db_path).unwrap(); insert_chunk(&conn, 1, "test", &make_embedding(8, 1.0), 0); drop(conn); - let data = - sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap(); - let opts = hdf5_writer::WriteOptions { - agent_id: "my-agent-42".into(), - embedder: "openai-ada".into(), - compression: false, - compression_level: 4, - float16: false, - }; - hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap(); - - let file = clawhdf5::File::open(h5_path.to_str().unwrap()).unwrap(); - let root = file.root(); - let attrs = root.attrs().unwrap(); - - match attrs.get("agent_id") { - Some(clawhdf5_format::type_builders::AttrValue::String(s)) => { - assert_eq!(s, "my-agent-42"); - } - other => panic!("Expected String agent_id, got {other:?}"), - } - match attrs.get("embedder") { - Some(clawhdf5_format::type_builders::AttrValue::String(s)) => { - assert_eq!(s, "openai-ada"); - } - other => panic!("Expected String embedder, got {other:?}"), - } - match attrs.get("embedding_dim") { - Some(clawhdf5_format::type_builders::AttrValue::I64(d)) => { - assert_eq!(*d, 8); - } - other => panic!("Expected I64 embedding_dim, got {other:?}"), - } + migrate_ok( + &db_path, + &h5_path, + &["--agent-id", "my-agent-42", "--embedder", "openai-ada"], + ); + let mem = HDF5Memory::open_read_only(&h5_path).unwrap(); + assert_eq!(mem.config().agent_id, "my-agent-42"); + assert_eq!(mem.config().embedder, "openai-ada"); + assert_eq!(mem.config().embedding_dim, 8); } - // ---------- Test 15: Embedding values roundtrip correctly ---------- + // ---------- f32 embedding values roundtrip exactly ---------- #[test] fn test_embedding_roundtrip() { let dir = TempDir::new().unwrap(); let db_path = create_test_db(&dir); - let h5_path = dir.path().join("out.h5"); + let h5_path = out_path(&dir, "out.h5"); let conn = Connection::open(&db_path).unwrap(); let emb = vec![0.1, 0.2, 0.3, 0.4]; insert_chunk(&conn, 1, "test", &emb, 0); drop(conn); - let data = - sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap(); - let opts = hdf5_writer::WriteOptions { - agent_id: "t".into(), - embedder: "t".into(), - compression: false, - compression_level: 4, - float16: false, - }; - hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap(); - - let file = clawhdf5::File::open(h5_path.to_str().unwrap()).unwrap(); - let chunks_group = file.group("chunks").unwrap(); - let emb_ds = chunks_group.dataset("embeddings").unwrap(); - let read_back = emb_ds.read_f32().unwrap(); - - assert_eq!(read_back.len(), 4); - for (a, b) in emb.iter().zip(read_back.iter()) { - assert!((a - b).abs() < 1e-6, "Embedding mismatch: {a} vs {b}"); - } + migrate_ok(&db_path, &h5_path, &["--f32"]); + let mem = HDF5Memory::open_read_only(&h5_path).unwrap(); + assert_eq!(&mem.cache.embeddings[0], &emb[..]); } - // ---------- Test 16: Full combined migration ---------- + // ---------- Full combined migration ---------- #[test] fn test_full_combined_migration() { let dir = TempDir::new().unwrap(); let db_path = create_test_db(&dir); - let h5_path = dir.path().join("out.h5"); + let h5_path = out_path(&dir, "out.h5"); let conn = Connection::open(&db_path).unwrap(); for i in 0..5 { @@ -787,89 +1167,74 @@ mod tests { insert_relation(&conn, 1, 2, "knows"); drop(conn); - // Skip deleted - let data = - sqlite_reader::read_sqlite(&db_path, true, None, &SchemaConfig::default()).unwrap(); - assert_eq!(data.chunks.len(), 4); // chunk 3 is deleted - - let opts = hdf5_writer::WriteOptions { - agent_id: "combined-test".into(), - embedder: "test-embedder".into(), - compression: true, - compression_level: 4, - float16: false, - }; - hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap(); - - let summary = - validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap(); - assert_eq!(summary.chunks, 4); - assert_eq!(summary.sessions, 2); - assert_eq!(summary.entities, 2); - assert_eq!(summary.relations, 1); - assert_eq!(summary.embedding_dim, 16); + let o = migrate_ok( + &db_path, + &h5_path, + &[ + "--agent-id", + "combined-test", + "--embedder", + "test-embedder", + "--skip-deleted", + "--compression", + ], + ); + assert_eq!(o.summary.count, 4); // chunk 3 is deleted + assert_eq!(o.summary.sessions, 2); + assert_eq!(o.summary.entities, 2); + assert_eq!(o.summary.relations, 1); + assert_eq!(o.summary.embedding_dim, 16); } - // ---------- Test 17: Validation catches session mismatch ---------- + // ---------- Validation catches a session mismatch ---------- #[test] fn test_validation_session_mismatch() { let dir = TempDir::new().unwrap(); let db_path = create_test_db(&dir); - let h5_path = dir.path().join("out.h5"); + let h5_path = out_path(&dir, "out.h5"); let conn = Connection::open(&db_path).unwrap(); insert_session(&conn, "s1", 0, 10); + insert_chunk(&conn, 1, "one", &make_embedding(8, 1.0), 0); drop(conn); - let data = - sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap(); - let opts = hdf5_writer::WriteOptions { - agent_id: "t".into(), - embedder: "t".into(), - compression: false, - compression_level: 4, - float16: false, - }; - hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap(); - - // Validating against a source whose session content differs must fail. - let mut tampered = - sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap(); - tampered.sessions[0].summary = "DIFFERENT".into(); - let result = validate::validate_hdf5(h5_path.to_str().unwrap(), &tampered, false, false); - assert!(result.is_err()); + let (mut data, m) = write(&db_path, &h5_path, false); + validate::validate_store(&h5_path, &data, &m, false).unwrap(); + data.sessions[0].summary = "DIFFERENT".into(); + let result = validate::validate_store(&h5_path, &data, &m, false); assert!(result.unwrap_err().to_string().contains("session")); } - // ---------- Real content validation catches corrupt embeddings ---------- + // ---------- Content validation catches corrupt embeddings ---------- #[test] fn test_content_validation_catches_embedding_corruption() { let dir = TempDir::new().unwrap(); let db_path = create_test_db(&dir); - let h5_path = dir.path().join("out.h5"); + let h5_path = out_path(&dir, "out.h5"); let conn = Connection::open(&db_path).unwrap(); insert_chunk(&conn, 1, "hello", &make_embedding(8, 1.0), 0); drop(conn); - let data = - sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap(); - let opts = hdf5_writer::WriteOptions { - agent_id: "t".into(), - embedder: "t".into(), - compression: false, - compression_level: 4, - float16: false, - }; - hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap(); - - // A source whose embedding differs (but counts match) must fail validation. - let mut tampered = - sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap(); - tampered.chunks[0].embedding[3] += 9.0; - let result = validate::validate_hdf5(h5_path.to_str().unwrap(), &tampered, true, false); - assert!(result.is_err()); - assert!(result.unwrap_err().to_string().contains("embedding")); + for f32 in [false, true] { + let (mut data, m) = write(&db_path, &h5_path, f32); + validate::validate_store(&h5_path, &data, &m, true).unwrap(); + // Exact, not within a tolerance: one ulp is a mismatch. + let v = &mut data.chunks[0].embedding[3]; + *v = if f32 { + f32::from_bits(v.to_bits() + 1) + } else { + // The next half-precision value up. + clawhdf5_format::float16::f16_bits_to_f32( + clawhdf5_format::float16::f32_to_f16_bits(*v) + 1, + ) + }; + let result = validate::validate_store(&h5_path, &data, &m, true); + assert!( + result.unwrap_err().to_string().contains("embedding"), + "f32={f32}" + ); + } } // ---------- Configurable schema: custom table names ---------- @@ -911,6 +1276,15 @@ mod tests { // Counts pass should also honor the custom table name. let counts = sqlite_reader::read_counts(&path_str, false, &schema).unwrap(); assert_eq!(counts.chunks, 1); + + // And the CLI flag reaches the reader. + let h5_path = out_path(&dir, "out.h5"); + let o = migrate_ok( + &path_str, + &h5_path, + &["--chunks-table", "my_chunks", "--verbose"], + ); + assert_eq!(o.summary.count, 1); } // ---------- Incremental migration appends only new rows ---------- @@ -918,44 +1292,290 @@ mod tests { fn test_incremental_migration() { let dir = TempDir::new().unwrap(); let db_path = create_test_db(&dir); - let h5_path = dir.path().join("out.h5"); - let cfg = SchemaConfig::default(); - let opts = hdf5_writer::WriteOptions { - agent_id: "t".into(), - embedder: "t".into(), - compression: false, - compression_level: 4, - float16: false, - }; + let h5_path = out_path(&dir, "out.h5"); - // First migration: 2 chunks. + // First migration: 2 chunks, a session and an edge. let conn = Connection::open(&db_path).unwrap(); insert_chunk(&conn, 1, "one", &make_embedding(4, 1.0), 0); insert_chunk(&conn, 2, "two", &make_embedding(4, 2.0), 0); + insert_session(&conn, "s1", 0, 1); + insert_entity(&conn, 1, "Alice", "person"); + insert_entity(&conn, 2, "Bob", "person"); + insert_relation(&conn, 1, 2, "knows"); drop(conn); - let data = sqlite_reader::read_sqlite(&db_path, false, None, &cfg).unwrap(); - hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap(); + let o = migrate_ok(&db_path, &h5_path, &["--incremental", "--f32"]); + assert!(!o.migration.appended_to_existing); + assert_eq!(o.summary.count, 2); - // Add two more rows, then migrate incrementally. + // The agent uses the store in between. + let mut mem = HDF5Memory::open(&h5_path).unwrap(); + mem.add_entity("Carol", "person", -1).unwrap(); + drop(mem); + + // Add rows, then migrate incrementally. let conn = Connection::open(&db_path).unwrap(); insert_chunk(&conn, 3, "three", &make_embedding(4, 3.0), 0); - insert_chunk(&conn, 4, "four", &make_embedding(4, 4.0), 0); + insert_chunk(&conn, 4, "four", &make_embedding(4, 4.0), 1); + insert_session(&conn, "s2", 2, 3); + insert_entity(&conn, 3, "Rust", "language"); + insert_relation(&conn, 1, 3, "uses"); drop(conn); - let base = hdf5_reader::read_hdf5(h5_path.to_str().unwrap()).unwrap(); - let max_id = base.chunks.iter().map(|c| c.id).max().unwrap_or(0); - assert_eq!(max_id, 2); - let new = - sqlite_reader::read_sqlite_filtered(&db_path, false, Some(4), &cfg, max_id).unwrap(); - assert_eq!(new.chunks.len(), 2); // only id 3 and 4 + // --f32 is irrelevant here: the store keeps its precision anyway. + let o = migrate_ok(&db_path, &h5_path, &["--incremental", "--validate-full"]); + let m = &o.migration; + assert!(m.appended_to_existing); + assert!(!o.summary.float16, "an existing store keeps its precision"); + assert_eq!((m.records.len(), m.chunks_present), (2, 2)); + assert_eq!((m.sessions.len(), m.sessions_present), (1, 1)); + assert_eq!((m.entities.len(), m.entities_present), (1, 2)); + assert_eq!((m.relations.len(), m.relations_present), (1, 1)); + assert_eq!(o.summary.count, 4); + assert_eq!(o.summary.active, 3); + assert_eq!(o.summary.sessions, 2); + assert_eq!(o.summary.entities, 4); // Alice, Bob, Carol, Rust + assert_eq!(o.summary.relations, 2); - let mut merged = base; - merged.chunks.extend(new.chunks); - hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &merged, &opts).unwrap(); + let mem = HDF5Memory::open_read_only(&h5_path).unwrap(); + assert_eq!(mem.cache.chunks, ["one", "two", "three", "four"]); + assert_eq!(mem.cache.tombstones, [0, 0, 0, 1]); + drop(mem); - let final_data = hdf5_reader::read_hdf5(h5_path.to_str().unwrap()).unwrap(); - assert_eq!(final_data.chunks.len(), 4); - let texts: Vec<&str> = final_data.chunks.iter().map(|c| c.chunk.as_str()).collect(); - assert_eq!(texts, vec!["one", "two", "three", "four"]); + // Nothing new: nothing written. + let o = migrate_ok(&db_path, &h5_path, &["--incremental"]); + assert!(o.migration.records.is_empty() && o.migration.relations.is_empty()); + assert_eq!(o.summary.count, 4); + } + + // ---------- --incremental: a source of another dimension is an error ---------- + #[test] + fn incremental_rejects_a_different_dimension() { + let dir = TempDir::new().unwrap(); + let db_path = create_test_db(&dir); + let h5_path = out_path(&dir, "out.h5"); + + let conn = Connection::open(&db_path).unwrap(); + insert_chunk(&conn, 1, "one", &make_embedding(8, 1.0), 0); + drop(conn); + migrate_ok(&db_path, &h5_path, &[]); + + let other = dir.path().join("other.db"); + let other = other.to_str().unwrap(); + std::fs::copy(&db_path, other).unwrap(); + let conn = Connection::open(other).unwrap(); + conn.execute("DELETE FROM memory_chunks", []).unwrap(); + insert_chunk(&conn, 1, "sixteen", &make_embedding(16, 1.0), 0); + drop(conn); + + for extra in [ + &["--incremental"][..], + &["--incremental", "--embedding-dim", "8"], + ] { + let err = migrate(other, &h5_path, extra).unwrap_err().to_string(); + assert!(err.contains("embedding"), "{extra:?}: {err}"); + } + let err = migrate(other, &h5_path, &["--incremental"]) + .unwrap_err() + .to_string(); + assert!(err.contains("has embedding_dim 8, the source 16"), "{err}"); + let mem = HDF5Memory::open(&h5_path).unwrap(); + assert_eq!(mem.count(), 1); + assert_eq!(mem.cache.embeddings[0].len(), 8); + } + + // ---------- --incremental carries over changes to the deleted flag ---------- + fn deleted_flag_is_reconciled(skip_deleted: bool) { + let dir = TempDir::new().unwrap(); + let db_path = create_test_db(&dir); + let h5_path = out_path(&dir, "out.h5"); + let skip: &[&str] = if skip_deleted { + &["--skip-deleted"] + } else { + &[] + }; + let with = |more: &[&'static str]| -> Vec<&str> { + let mut v = skip.to_vec(); + v.extend_from_slice(more); + v + }; + + let conn = Connection::open(&db_path).unwrap(); + for i in 1..=5 { + insert_chunk( + &conn, + i, + &format!("r{i}"), + &axis_embedding(8, i as usize), + 0, + ); + } + insert_chunk(&conn, 6, "r6", &axis_embedding(8, 6), 1); + drop(conn); + migrate_ok(&db_path, &h5_path, &with(&["--incremental"])); + let before = if skip_deleted { 5 } else { 6 }; + assert_eq!(HDF5Memory::open(&h5_path).unwrap().count(), before); + + // Row 5 deleted and row 6 restored in the source since then. + let conn = Connection::open(&db_path).unwrap(); + conn.execute("UPDATE memory_chunks SET deleted = 1 WHERE id = 5", []) + .unwrap(); + conn.execute("UPDATE memory_chunks SET deleted = 0 WHERE id = 6", []) + .unwrap(); + drop(conn); + let o = migrate_ok( + &db_path, + &h5_path, + &with(&["--incremental", "--validate-full"]), + ); + let m = &o.migration; + assert_eq!(m.deleted_in_store.len(), 1); + assert_eq!(m.restored, usize::from(!skip_deleted)); + assert_eq!(m.records.len(), 1, "r6 written (again)"); + assert_eq!(m.chunks_present, 4); + + let mut mem = HDF5Memory::open_read_only(&h5_path).unwrap(); + let active: Vec<&str> = (0..mem.count()) + .filter(|&i| mem.cache.tombstones[i] == 0) + .map(|i| mem.cache.chunks[i].as_str()) + .collect(); + assert_eq!(active, ["r1", "r2", "r3", "r4", "r6"]); + assert_eq!(o.summary.active, 5); + // r5 is no longer found by search; r6 is. + let hits = mem.search(&axis_embedding(8, 5), "r5", &SearchOptions::new(10)); + assert!(hits.iter().all(|h| h.chunk != "r5"), "{hits:?}"); + let hits = mem.search(&axis_embedding(8, 6), "r6", &SearchOptions::new(10)); + assert!(hits.iter().any(|h| h.chunk == "r6"), "{hits:?}"); + drop(mem); + + // Idempotent: a second run changes nothing. + let o = migrate_ok(&db_path, &h5_path, &with(&["--incremental"])); + let m = &o.migration; + assert!(m.records.is_empty() && m.deleted_in_store.is_empty() && m.restored == 0); + assert_eq!(o.summary.active, 5); + } + + #[test] + fn incremental_follows_source_deletes_and_restores() { + deleted_flag_is_reconciled(false); + } + + #[test] + fn incremental_skip_deleted_still_follows_source_deletes() { + deleted_flag_is_reconciled(true); + } + + // ---------- Without --incremental an existing store is replaced ---------- + #[test] + fn rerun_without_incremental_replaces_store() { + let dir = TempDir::new().unwrap(); + let db_path = create_test_db(&dir); + let h5_path = out_path(&dir, "out.h5"); + + let conn = Connection::open(&db_path).unwrap(); + insert_chunk(&conn, 1, "one", &make_embedding(4, 1.0), 0); + drop(conn); + migrate_ok(&db_path, &h5_path, &[]); + let o = migrate_ok(&db_path, &h5_path, &[]); + assert_eq!(o.summary.count, 1); + assert_eq!(HDF5Memory::open(&h5_path).unwrap().count(), 1); + } + + // ---------- The store is locked while open for writing ---------- + #[test] + fn migrating_into_a_store_in_use_fails() { + let dir = TempDir::new().unwrap(); + let db_path = create_test_db(&dir); + let h5_path = out_path(&dir, "out.h5"); + + let conn = Connection::open(&db_path).unwrap(); + insert_chunk(&conn, 1, "one", &make_embedding(4, 1.0), 0); + drop(conn); + migrate_ok(&db_path, &h5_path, &[]); + let _writer = HDF5Memory::open(&h5_path).unwrap(); + for extra in [&[][..], &["--incremental"][..]] { + let err = migrate(&db_path, &h5_path, extra).unwrap_err().to_string(); + assert!(err.contains("locked"), "{extra:?}: {err}"); + } + } + + // ---------- h5py can open a migrated store ---------- + fn python() -> String { + std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string()) + } + + #[test] + fn h5py_opens_migrated_store() { + let has_h5py = std::process::Command::new(python()) + .args(["-c", "import h5py"]) + .output() + .map(|o| o.status.success()) + .unwrap_or(false); + if !has_h5py { + assert!( + std::env::var("CLAWHDF5_REQUIRE_INTEROP").as_deref() != Ok("1"), + "CLAWHDF5_REQUIRE_INTEROP=1 but {} with h5py is not available", + python() + ); + eprintln!("SKIP: {} with h5py not available", python()); + return; + } + + let dir = TempDir::new().unwrap(); + let db_path = create_test_db(&dir); + let conn = Connection::open(&db_path).unwrap(); + for i in 0..6 { + insert_chunk( + &conn, + i, + &format!("memory {i}"), + &axis_embedding(8, i as usize), + 0, + ); + } + insert_session(&conn, "s1", 0, 5); + insert_entity(&conn, 1, "Alice", "person"); + insert_entity(&conn, 2, "Bob", "person"); + insert_relation(&conn, 1, 2, "knows"); + drop(conn); + + for (extra, dtype) in [(&[][..], "float16"), (&["--f32"][..], "float32")] { + let h5_path = out_path(&dir, &format!("{dtype}.h5")); + migrate_ok(&db_path, &h5_path, extra); + let bits = (0..6) + .flat_map(|i| axis_embedding(8, i)) + .map(|v| v.to_bits().to_string()) + .collect::>() + .join(","); + let script = format!( + r#" +import h5py, numpy as np +with h5py.File(r"{path}", "r") as f: + names = [] + f.visititems(lambda n, o: names.append(n) if isinstance(o, h5py.Dataset) else None) + for n in names: + f[n][()] # every dataset must decode + assert f["meta"].attrs["schema_version"] in (b"1.0", "1.0") + e = f["memory/embeddings"] + assert e.dtype == np.{dtype}, e.dtype + ref = np.array([{bits}], dtype=np.uint32).view(np.float32).astype(np.{dtype}).reshape(6, 8) + assert (e[()] == ref).all() + assert len(f["memory/chunks"]) == 6 + assert len(f["sessions/ids"]) == 1 + assert len(f["knowledge_graph/entity_ids"]) == 2 + assert len(f["knowledge_graph/relation_srcs"]) == 1 +print(len(names)) +"#, + path = h5_path.display() + ); + let out = std::process::Command::new(python()) + .args(["-c", &script]) + .output() + .unwrap(); + assert!( + out.status.success(), + "{dtype}: {}", + String::from_utf8_lossy(&out.stderr) + ); + } } } diff --git a/crates/clawhdf5-migrate/src/sqlite_reader.rs b/crates/clawhdf5-migrate/src/sqlite_reader.rs index 26c04e6..1a20dbe 100644 --- a/crates/clawhdf5-migrate/src/sqlite_reader.rs +++ b/crates/clawhdf5-migrate/src/sqlite_reader.rs @@ -50,12 +50,8 @@ pub struct SqliteData { pub sessions: Vec, pub entities: Vec, pub relations: Vec, + /// `--embedding-dim`, or the first row's; 0 when neither exists. pub embedding_dim: usize, - /// Filesystem path of the SQLite database this data was read from, for - /// provenance attribution on the HDF5 output. Empty when the data did - /// not come directly from a SQLite read (e.g. re-read of a prior HDF5 - /// migration output for an incremental merge). - pub source_path: String, } /// A table name plus the ordered column names the reader maps by position. @@ -167,11 +163,13 @@ pub fn read_counts( }) } -/// Auto-detect embedding dimension from the first chunk's BLOB size. +/// Auto-detect embedding dimension from the BLOB size of the first chunk (in +/// id order, deleted or not). fn detect_embedding_dim(conn: &Connection, config: &SchemaConfig) -> SqlResult> { let emb_col = config.chunks.columns.get(2).copied().unwrap_or("embedding"); + let id_col = config.chunks.columns.first().copied().unwrap_or("id"); let mut stmt = conn.prepare(&format!( - "SELECT {emb_col} FROM {} LIMIT 1", + "SELECT {emb_col} FROM {} ORDER BY {id_col} LIMIT 1", config.chunks.table ))?; let mut rows = stmt.query([])?; @@ -195,24 +193,16 @@ fn blob_to_f32(blob: &[u8]) -> Vec { /// Read all data from a ZeroClaw SQLite database. /// /// If `skip_deleted` is true, rows with `deleted=1` are excluded from chunks. -/// If `embedding_dim` is `None`, auto-detect from the first row. +/// If `embedding_dim` is `None`, auto-detect from the first row (0 when there +/// are no rows). Embeddings are returned at their full stored length whatever +/// the dimension: checking that every row matches it is the writer's job +/// (`store_writer::write_store`), so a mismatch is an error, not silent +/// truncation. pub fn read_sqlite( path: &str, skip_deleted: bool, embedding_dim: Option, config: &SchemaConfig, -) -> Result> { - read_sqlite_filtered(path, skip_deleted, embedding_dim, config, 0) -} - -/// Like [`read_sqlite`] but only reads chunks whose id is greater than -/// `min_chunk_id` (0 = all). Used for incremental migration. -pub fn read_sqlite_filtered( - path: &str, - skip_deleted: bool, - embedding_dim: Option, - config: &SchemaConfig, - min_chunk_id: i64, ) -> Result> { let conn = Connection::open(path)?; @@ -221,7 +211,7 @@ pub fn read_sqlite_filtered( None => detect_embedding_dim(&conn, config)?.unwrap_or(0), }; - let chunks = read_chunks(&conn, skip_deleted, dim, config, min_chunk_id)?; + let chunks = read_chunks(&conn, skip_deleted, config)?; let sessions = read_sessions(&conn, config)?; let entities = read_entities(&conn, config)?; let relations = read_relations(&conn, config)?; @@ -232,42 +222,43 @@ pub fn read_sqlite_filtered( entities, relations, embedding_dim: dim, - source_path: path.to_owned(), }) } fn read_chunks( conn: &Connection, skip_deleted: bool, - expected_dim: usize, config: &SchemaConfig, - min_chunk_id: i64, ) -> SqlResult> { let id_col = config.chunks.columns.first().copied().unwrap_or("id"); let deleted_col = config.chunks.columns.get(7).copied().unwrap_or("deleted"); - let mut conds = Vec::new(); + let mut where_clause = String::new(); if skip_deleted { - conds.push(format!("{deleted_col} = 0")); + where_clause = format!(" WHERE {deleted_col} = 0"); } - if min_chunk_id > 0 { - conds.push(format!("{id_col} > {min_chunk_id}")); - } - let where_clause = if conds.is_empty() { - String::new() - } else { - format!(" WHERE {}", conds.join(" AND ")) - }; + // In id order, so the store's records follow the source's order. + where_clause.push_str(&format!(" ORDER BY {id_col}")); let sql = config.chunks.select(&where_clause); let mut stmt = conn.prepare(&sql)?; let rows = stmt.query_map([], |row| { let blob: Vec = row.get(2)?; - let mut embedding = blob_to_f32(&blob); - - // Validate/truncate to expected dimension - if expected_dim > 0 { - embedding.truncate(expected_dim); + if !blob.len().is_multiple_of(4) { + let id: i64 = row.get(0)?; + return Err(rusqlite::Error::FromSqlConversionFailure( + 2, + rusqlite::types::Type::Blob, + format!( + "chunk id {id}: embedding BLOB is {} bytes, not a whole number of \ + little-endian f32 values", + blob.len() + ) + .into(), + )); } + // Read at full length: rows of the wrong dimension are rejected by + // the writer, never truncated to fit. + let embedding = blob_to_f32(&blob); Ok(MemoryChunk { id: row.get(0)?, diff --git a/crates/clawhdf5-migrate/src/store_writer.rs b/crates/clawhdf5-migrate/src/store_writer.rs new file mode 100644 index 0000000..a707596 --- /dev/null +++ b/crates/clawhdf5-migrate/src/store_writer.rs @@ -0,0 +1,407 @@ +//! Write migrated SQLite data into a clawhdf5-agent store. +//! +//! Everything goes through `clawhdf5-agent`'s own API — `HDF5Memory::create` +//! (or `open` for `--incremental`), `save_batch`, `delete_batch`, the session +//! cache and the knowledge graph — so the result is an ordinary agent store +//! that `HDF5Memory::open` accepts, not a second hand-built copy of its schema. + +use std::collections::{HashMap, HashSet}; +use std::path::Path; + +use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry}; +use clawhdf5_format::float16::round_to_f16; + +use crate::sqlite_reader::{MemoryChunk, SqliteData}; + +type BoxErr = Box; + +/// SQLite timestamps are Unix seconds; the agent's session and relation +/// timestamps are Unix microseconds (memory records stay in seconds). +pub const US_PER_SEC: f64 = 1_000_000.0; + +/// Options controlling the output store. +#[derive(Debug, Clone)] +pub struct WriteOptions { + pub agent_id: String, + pub embedder: String, + pub compression: bool, + pub compression_level: u32, + /// Store full-precision `f32` embeddings instead of the library default + /// (half precision). Only applies to a newly created store: an existing + /// store keeps the precision it was created with. + pub f32: bool, + /// Add to the store at the output path if there is one, instead of + /// replacing it. + pub incremental: bool, + /// Leave out deleted source rows that are not in the store. (A deleted + /// row that matches an active store record still tombstones it, so pass + /// deleted rows in `data` for an incremental run.) + pub skip_deleted: bool, +} + +/// What the migration wrote, and where each source row went, so validation +/// can compare the store with the source row by row. +#[derive(Debug, Default)] +pub struct Migration { + /// Whether the output store existed and was added to (`--incremental`). + pub appended_to_existing: bool, + /// The store's embedding precision. + pub float16: bool, + pub embedding_dim: usize, + /// Records in the store after the migration (including tombstones). + pub store_count: usize, + /// `(store index, source chunk index)` of every record written. + pub records: Vec<(usize, usize)>, + /// Source chunks already in the store (incremental), not written again. + pub chunks_present: usize, + /// `(store index, source chunk index)` of records that were active in + /// the store but whose source row is now deleted (incremental): they were + /// tombstoned by this run. + pub deleted_in_store: Vec<(usize, usize)>, + /// Source rows that were deleted in the store but are active in the + /// source (incremental): the agent has no un-delete, so each was written + /// again as a new record (counted in `records` too). + pub restored: usize, + /// Deleted source rows left out because of `skip_deleted`. + pub deleted_skipped: usize, + /// `(store session index, source session index)` of each session written. + pub sessions: Vec<(usize, usize)>, + pub sessions_present: usize, + /// `(store entity id, source entity index)` of each entity written. + pub entities: Vec<(u64, usize)>, + pub entities_present: usize, + /// SQLite entity id -> store entity id, for every source entity. + pub entity_ids: HashMap, + /// `(store relation index, source relation index)` of each relation written. + pub relations: Vec<(usize, usize)>, + pub relations_present: usize, + /// Source relations naming an entity id that is not in the entities + /// table; the knowledge graph cannot hold them, so they are skipped. + pub dangling_relations: Vec, + /// Messages of the write-anomaly alerts the agent raised while importing + /// (informational; they never block a save — a bulk import typically + /// trips the write-rate check). + pub anomaly_alerts: Vec, +} + +/// Identity of a memory record for incremental de-duplication: every field +/// the agent stores except the embedding (whose stored form depends on the +/// store's precision). +type RecordKey = (String, String, String, String, u64); + +fn record_key( + chunk: &str, + source_channel: &str, + session_id: &str, + tags: &str, + ts: f64, +) -> RecordKey { + ( + chunk.to_owned(), + source_channel.to_owned(), + session_id.to_owned(), + tags.to_owned(), + ts.to_bits(), + ) +} + +/// Reject rows the agent would otherwise store differently from the source, +/// or not at all: an embedding of a different length from the store's +/// dimension (the agent pads/truncates silently), an empty embedding, or, in +/// a float16 store, a value beyond the half-precision range. +/// +/// Every source row is checked, including ones that end up not being written +/// (already in the store, or deleted and skipped): the source must be +/// consistent as a whole, and the check runs before the store is touched. +fn check_chunks(chunks: &[MemoryChunk], dim: usize, float16: bool) -> Result<(), BoxErr> { + for c in chunks { + if c.embedding.is_empty() { + return Err(format!( + "chunk id {}: the embedding is empty; an agent store needs an embedding \ + for every record", + c.id + ) + .into()); + } + if c.embedding.len() != dim { + return Err(format!( + "chunk id {}: embedding has {} values, expected {dim}; every row must have \ + the store's dimension (detected from the first row unless --embedding-dim \ + is given), and rows are never truncated or padded to fit", + c.id, + c.embedding.len() + ) + .into()); + } + if float16 + && let Some((k, v)) = c + .embedding + .iter() + .enumerate() + .find(|&(_, &v)| v.is_finite() && round_to_f16(v).is_infinite()) + { + return Err(format!( + "chunk id {}: embedding[{k}] = {v} is outside the half-precision range \ + (±65504) of a float16 store; migrate with --f32", + c.id + ) + .into()); + } + } + Ok(()) +} + +/// Migrate `data` into the agent store at `path`. +/// +/// Without `opts.incremental` (or when nothing exists at `path`) a new store +/// is created, replacing any file there — but only once every source row has +/// passed [`check_chunks`], so a source that cannot be migrated leaves an +/// existing store untouched. With it, the existing store is opened and only +/// source rows it does not already hold are added: memory records are +/// matched on their content, sessions on their id, entities on name and +/// type, relations on (source, target, relation). A matched record then +/// takes the source row's deleted flag: see [`Migration::deleted_in_store`] +/// and [`Migration::restored`]. +pub fn write_store( + path: &Path, + data: &SqliteData, + opts: &WriteOptions, +) -> Result { + let existing = opts.incremental && path.exists(); + let mut mem = if existing { + // `open` does not modify the store beyond what the agent itself does + // on open; the checks below run before anything is written. + let mem = HDF5Memory::open(path)?; + let dim = mem.config().embedding_dim; + // `data.embedding_dim` is 0 only for a source with no records and no + // --embedding-dim, which has no dimension to disagree with. + if data.embedding_dim != 0 && dim != data.embedding_dim { + let hint = if dim == 0 { + " (a store created from a source with no memory records; re-create it \ + with --embedding-dim)" + } else { + "" + }; + return Err(format!( + "the store at {} has embedding_dim {dim}{hint}, the source {}; \ + embeddings of a different dimension cannot be added to it", + path.display(), + data.embedding_dim + ) + .into()); + } + check_chunks(&data.chunks, dim, mem.config().float16)?; + mem + } else { + // (With records, a dimension of 0 means an empty first embedding, + // which `check_chunks` reports more precisely.) + if data.embedding_dim == 0 && data.chunks.is_empty() { + return Err( + "the source has no memory records to detect the embedding dimension \ + from; pass --embedding-dim (the dimension of the agent's embedder), \ + or the store could never hold a record" + .into(), + ); + } + let mut config = MemoryConfig::new(path.to_path_buf(), &opts.agent_id, data.embedding_dim); + config.embedder = opts.embedder.clone(); + config.compression = opts.compression; + config.compression_level = opts.compression_level; + // Only ever switch the library default off (as `clawhdf5-cli create`). + if opts.f32 { + config.float16 = false; + } + // Before `create`, which replaces whatever is at `path`. + check_chunks(&data.chunks, config.embedding_dim, config.float16)?; + HDF5Memory::create(config)? + }; + let float16 = mem.config().float16; + let dim = mem.config().embedding_dim; + + let mut m = Migration { + appended_to_existing: existing, + float16, + embedding_dim: dim, + ..Migration::default() + }; + + // ---- Memory records -------------------------------------------------- + // Store indices of every record the store already holds, by content, so + // a source row that appears twice is only treated as present as often + // as the store has it. + let mut present: HashMap> = HashMap::new(); + if existing { + let c = &mem.cache; + for i in 0..c.len() { + let key = record_key( + &c.chunks[i], + &c.source_channels[i], + &c.session_ids[i], + &c.tags[i], + c.timestamps[i], + ); + present.entry(key).or_default().push(i); + } + } + let key_of = |c: &MemoryChunk| { + record_key( + &c.chunk, + &c.source_channel, + &c.session_id, + &c.tags, + c.timestamp, + ) + }; + let tombstoned = |idx: usize| mem.cache.tombstones[idx] != 0; + // Pass 1: a store record in the same deleted state as the source row. + let mut unmatched: Vec = Vec::new(); + for (i, c) in data.chunks.iter().enumerate() { + let src_deleted = c.deleted != 0; + let hit = present.get_mut(&key_of(c)).and_then(|idxs| { + let at = idxs.iter().position(|&x| tombstoned(x) == src_deleted)?; + Some(idxs.remove(at)) + }); + match hit { + Some(_) => m.chunks_present += 1, + None => unmatched.push(i), + } + } + // Pass 2: a store record whose deleted state differs — the source row + // was deleted or restored since the last migration. The source wins. + let mut new_chunks: Vec = Vec::with_capacity(unmatched.len()); + let mut delete_in_store: Vec = Vec::new(); + for i in unmatched { + let c = &data.chunks[i]; + let hit = present + .get_mut(&key_of(c)) + .and_then(|idxs| (!idxs.is_empty()).then(|| idxs.remove(0))); + match hit { + // Active in the store, deleted in the source: tombstone it. + Some(idx) if c.deleted != 0 => { + m.deleted_in_store.push((idx, i)); + delete_in_store.push(idx); + } + // Deleted in the store, active in the source. The agent has no + // un-delete, so the row is written again as a new active record + // (the tombstone stays until the store is compacted). + Some(_) => { + m.restored += 1; + new_chunks.push(i); + } + None if c.deleted != 0 && opts.skip_deleted => m.deleted_skipped += 1, + None => new_chunks.push(i), + } + } + new_chunks.sort_unstable(); + let to_write: Vec<&MemoryChunk> = new_chunks.iter().map(|&i| &data.chunks[i]).collect(); + + // ---- Sessions (in the cache; persisted by the save_batch checkpoint) --- + let known_sessions: HashSet = mem + .sessions() + .entries + .iter() + .map(|e| e.id.clone()) + .collect(); + for (i, s) in data.sessions.iter().enumerate() { + if known_sessions.contains(&s.id) { + m.sessions_present += 1; + continue; + } + let sessions = mem.sessions_mut(); + let at = sessions.len(); + sessions.add_at( + &s.id, + s.start_idx.max(0) as usize, + s.end_idx.max(0) as usize, + &s.channel, + &s.summary, + s.timestamp * US_PER_SEC, + ); + m.sessions.push((at, i)); + } + + // ---- Knowledge graph ------------------------------------------------- + let kg = mem.knowledge_mut(); + // Matched only against what the store held before this run: the source + // itself is copied as it is, duplicates included. + let by_name_type: HashMap<(String, String), u64> = kg + .entities + .iter() + .map(|e| ((e.name.clone(), e.entity_type.clone()), e.id)) + .collect(); + for (i, e) in data.entities.iter().enumerate() { + let key = (e.name.clone(), e.entity_type.clone()); + let id = match by_name_type.get(&key) { + Some(&id) => { + m.entities_present += 1; + id + } + None => { + let id = kg.add_entity(&e.name, &e.entity_type, e.embedding_idx); + m.entities.push((id, i)); + id + } + }; + m.entity_ids.insert(e.id, id); + } + let known_relations: HashSet<(u64, u64, String)> = kg + .relations + .iter() + .map(|r| (r.src, r.tgt, r.relation.clone())) + .collect(); + for (i, r) in data.relations.iter().enumerate() { + let (Some(&src), Some(&tgt)) = (m.entity_ids.get(&r.src), m.entity_ids.get(&r.tgt)) else { + m.dangling_relations.push(i); + continue; + }; + if known_relations.contains(&(src, tgt, r.relation.clone())) { + m.relations_present += 1; + continue; + } + let at = kg.relations.len(); + kg.add_relation(src, tgt, &r.relation, r.weight as f32); + kg.relations[at].ts = r.timestamp * US_PER_SEC; + m.relations.push((at, i)); + } + + // ---- Write: one checkpoint for records, sessions and graph ----------- + let entries: Vec = to_write + .iter() + .map(|c| MemoryEntry { + chunk: c.chunk.clone(), + embedding: c.embedding.clone(), + source_channel: c.source_channel.clone(), + timestamp: c.timestamp, + session_id: c.session_id.clone(), + tags: c.tags.clone(), + }) + .collect(); + let indices = mem.save_batch(entries)?; + m.records = indices + .iter() + .copied() + .zip(new_chunks.iter().copied()) + .collect(); + + // Rows deleted in the source stay deleted: tombstones, as the agent's own + // `delete` leaves them (not compacted away). + // Records matched in the store whose source row has since been deleted + // are tombstoned too. + let tombstones: Vec = m + .records + .iter() + .filter(|&&(_, src)| data.chunks[src].deleted != 0) + .map(|&(idx, _)| idx) + .chain(delete_in_store) + .collect(); + mem.delete_batch(&tombstones)?; + + m.anomaly_alerts = mem + .take_anomaly_alerts() + .into_iter() + .map(|a| a.message) + .collect(); + m.store_count = mem.count(); + drop(mem); // release the single-writer lock before anyone re-opens it + Ok(m) +} diff --git a/crates/clawhdf5-migrate/src/validate.rs b/crates/clawhdf5-migrate/src/validate.rs index 4a98d95..f028fb7 100644 --- a/crates/clawhdf5-migrate/src/validate.rs +++ b/crates/clawhdf5-migrate/src/validate.rs @@ -1,192 +1,266 @@ -use clawhdf5::reader::File as Hdf5File; -use clawhdf5_format::provenance::VerifyResult; +//! Validate a migration by reading the store back the way an agent would: +//! through `HDF5Memory::open_read_only`, comparing what it loads with the +//! SQLite source, and running a search for a migrated record. + +use std::path::Path; + +use clawhdf5_agent::{AgentMemory, HDF5Memory, SearchOptions}; +use clawhdf5_format::float16::round_to_f16; -use crate::hdf5_reader::read_hdf5; use crate::sqlite_reader::SqliteData; +use crate::store_writer::{Migration, US_PER_SEC}; type BoxErr = Box; /// Summary of a migration validation. #[derive(Debug)] pub struct ValidationSummary { - pub chunks: u64, - pub sessions: u64, - pub entities: u64, - pub relations: u64, - pub embedding_dim: u64, - /// Number of rows whose full content was compared against the source. + /// Records in the store (including tombstones). + pub count: usize, + /// Records in the store that are not deleted. + pub active: usize, + pub sessions: usize, + pub entities: usize, + pub relations: usize, + pub embedding_dim: usize, + pub float16: bool, + /// Rows whose full content was compared against the source. pub rows_checked: u64, - /// Whether the `chunks/text` and `chunks/embeddings` SHINES provenance - /// hashes (written via [`crate::hdf5_writer`]) were both present and - /// matched their recomputed SHA-256 on read-back. `false` when either - /// dataset has no provenance metadata (e.g. an older output file) or - /// there are zero chunks to check. - pub provenance_verified: bool, + /// Whether a search for a migrated record found it (`false` when there + /// was no active migrated record with an embedding to search for). + pub search_checked: bool, } -/// Validate a migrated HDF5 file against the source data. +/// Validate the store at `path` against the source rows `migration` wrote. /// -/// Reads the written file back and compares actual content — chunk text, -/// embeddings, and every session/entity/relation field — to the source, not -/// just the row counts. When `full` is false a representative sample of chunk -/// rows is content-checked (counts and all other groups are always checked in -/// full); when `full` is true every chunk row is compared too. `float16` widens -/// the embedding tolerance to allow for half-precision quantization. -pub fn validate_hdf5( - path: &str, +/// Counts and the session / entity / relation rows are always checked in +/// full. Memory records are content-checked on a representative sample, or +/// all of them with `full`. Embeddings must match exactly: the source values +/// themselves in an `f32` store, their [`round_to_f16`] in a `float16` one. +pub fn validate_store( + path: &Path, source: &SqliteData, + migration: &Migration, full: bool, - float16: bool, ) -> Result { - let got = read_hdf5(path)?; - let provenance_verified = verify_chunk_provenance(path)?; + let mut mem = HDF5Memory::open_read_only(path)?; + let float16 = mem.config().float16; + let dim = mem.config().embedding_dim; // ---- Counts ---- - check_count("chunk", got.chunks.len(), source.chunks.len())?; - check_count("session", got.sessions.len(), source.sessions.len())?; - check_count("entity", got.entities.len(), source.entities.len())?; - check_count("relation", got.relations.len(), source.relations.len())?; - if got.embedding_dim != source.embedding_dim { + check_count("record", mem.count(), migration.store_count)?; + if float16 != migration.float16 { return Err(format!( - "embedding_dim mismatch: HDF5 has {}, source has {}", - got.embedding_dim, source.embedding_dim + "float16 mismatch: store {float16}, expected {}", + migration.float16 ) .into()); } + if dim != migration.embedding_dim { + return Err(format!( + "embedding_dim mismatch: store has {dim}, expected {}", + migration.embedding_dim + ) + .into()); + } + if !migration.appended_to_existing { + check_count("record", mem.count(), migration.records.len())?; + check_count("session", mem.sessions().len(), migration.sessions.len())?; + check_count( + "entity", + mem.knowledge().entities.len(), + migration.entities.len(), + )?; + check_count( + "relation", + mem.knowledge().relations.len(), + migration.relations.len(), + )?; + } - // ---- Chunk content (sampled or full) ---- - let (emb_abs, emb_rel) = if float16 { (1e-2, 1e-2) } else { (1e-4, 0.0) }; + // ---- Memory records (sampled or full) ---- let mut rows_checked = 0u64; - for i in sample_indices(source.chunks.len(), full) { - let (s, g) = (&source.chunks[i], &got.chunks[i]); - if s.id != g.id { - return Err(field_err("chunk", i, "id", s.id, g.id)); + let expected_value = |v: f32| if float16 { round_to_f16(v) } else { v }; + for k in sample_indices(migration.records.len(), full) { + let (idx, src) = migration.records[k]; + let s = &source.chunks[src]; + let c = &mem.cache; + if idx >= c.len() { + return Err( + format!("record {idx} (chunk id {}) is missing from the store", s.id).into(), + ); } - if s.chunk != g.chunk { + let id = s.id; + if c.chunks[idx] != s.chunk { return Err(format!( - "chunk[{i}].text mismatch: source {:?}, HDF5 {:?}", + "record {idx} (chunk id {id}) text mismatch: source {:?}, store {:?}", truncate(&s.chunk), - truncate(&g.chunk) + truncate(&c.chunks[idx]) ) .into()); } - if s.session_id != g.session_id || s.source_channel != g.source_channel || s.tags != g.tags + if c.source_channels[idx] != s.source_channel + || c.session_ids[idx] != s.session_id + || c.tags[idx] != s.tags { - return Err(format!("chunk[{i}] string field mismatch").into()); + return Err(format!("record {idx} (chunk id {id}) string field mismatch").into()); } - if s.deleted != g.deleted { - return Err(field_err("chunk", i, "deleted", s.deleted, g.deleted)); - } - if s.embedding.len() != g.embedding.len() { + if c.timestamps[idx].to_bits() != s.timestamp.to_bits() { return Err(format!( - "chunk[{i}] embedding length mismatch: {} vs {}", - s.embedding.len(), - g.embedding.len() + "record {idx} (chunk id {id}) timestamp mismatch: source {}, store {}", + s.timestamp, c.timestamps[idx] ) .into()); } - for (k, (&a, &b)) in s.embedding.iter().zip(g.embedding.iter()).enumerate() { - if (a - b).abs() > emb_abs + emb_rel * a.abs() { - return Err( - format!("chunk[{i}].embedding[{k}] mismatch: source {a}, HDF5 {b}").into(), - ); + let deleted = c.tombstones[idx] != 0; + if deleted != (s.deleted != 0) { + return Err(format!( + "record {idx} (chunk id {id}) deleted mismatch: source {}, store {deleted}", + s.deleted != 0 + ) + .into()); + } + let got = c.embeddings.get(idx).unwrap_or(&[]); + if got.len() != s.embedding.len() { + return Err(format!( + "record {idx} (chunk id {id}) embedding length mismatch: source {}, store {}", + s.embedding.len(), + got.len() + ) + .into()); + } + for (j, (&a, &b)) in s.embedding.iter().zip(got).enumerate() { + let want = expected_value(a); + if want.to_bits() != b.to_bits() && !(want.is_nan() && b.is_nan()) { + return Err(format!( + "record {idx} (chunk id {id}) embedding[{j}] mismatch: source {a}, \ + expected {want}, store {b}" + ) + .into()); } } rows_checked += 1; } - // ---- Other groups (always full — they are small) ---- - for (i, (s, g)) in source.sessions.iter().zip(got.sessions.iter()).enumerate() { - if s.id != g.id - || s.start_idx != g.start_idx - || s.end_idx != g.end_idx - || s.channel != g.channel - || s.summary != g.summary - { - return Err(format!("session[{i}] mismatch").into()); + // ---- Records tombstoned because their source row was deleted ---- + for &(idx, src) in &migration.deleted_in_store { + let s = &source.chunks[src]; + let c = &mem.cache; + if idx >= c.len() || c.chunks[idx] != s.chunk || c.timestamps[idx] != s.timestamp { + return Err(format!("record {idx} (chunk id {}) mismatch or missing", s.id).into()); } - rows_checked += 1; - } - for (i, (s, g)) in source.entities.iter().zip(got.entities.iter()).enumerate() { - if s.id != g.id - || s.name != g.name - || s.entity_type != g.entity_type - || s.embedding_idx != g.embedding_idx - { - return Err(format!("entity[{i}] mismatch").into()); - } - rows_checked += 1; - } - for (i, (s, g)) in source - .relations - .iter() - .zip(got.relations.iter()) - .enumerate() - { - if s.src != g.src || s.tgt != g.tgt || s.relation != g.relation { - return Err(format!("relation[{i}] mismatch").into()); + if c.tombstones[idx] == 0 { + return Err(format!( + "record {idx} (chunk id {}) is deleted in the source but active in the store", + s.id + ) + .into()); } rows_checked += 1; } + // ---- Sessions ---- + let sessions = mem.sessions(); + for &(at, src) in &migration.sessions { + let s = &source.sessions[src]; + let (Some(e), Some(summary)) = (sessions.entries.get(at), sessions.summaries.get(at)) + else { + return Err(format!("session {:?} is missing from the store", s.id).into()); + }; + if e.id != s.id + || e.start_idx != s.start_idx.max(0) as u64 + || e.end_idx != s.end_idx.max(0) as u64 + || e.channel != s.channel + || *summary != s.summary + || e.ts != s.timestamp * US_PER_SEC + { + return Err(format!("session {:?} mismatch", s.id).into()); + } + rows_checked += 1; + } + + // ---- Knowledge graph ---- + let kg = mem.knowledge(); + for &(id, src) in &migration.entities { + let s = &source.entities[src]; + let Some(e) = kg.get_entity(id) else { + return Err(format!( + "entity {:?} (id {}) is missing from the store", + s.name, s.id + ) + .into()); + }; + if e.name != s.name || e.entity_type != s.entity_type || e.embedding_idx != s.embedding_idx + { + return Err(format!("entity {:?} (id {}) mismatch", s.name, s.id).into()); + } + rows_checked += 1; + } + for &(at, src) in &migration.relations { + let s = &source.relations[src]; + let r = kg.relations.get(at); + let ok = r.is_some_and(|r| { + Some(&r.src) == migration.entity_ids.get(&s.src) + && Some(&r.tgt) == migration.entity_ids.get(&s.tgt) + && r.relation == s.relation + && r.weight == s.weight as f32 + && r.ts == s.timestamp * US_PER_SEC + }); + if !ok { + return Err(format!( + "relation {} -[{}]-> {} mismatch or missing", + s.src, s.relation, s.tgt + ) + .into()); + } + rows_checked += 1; + } + + // ---- A migrated record must be findable by search ---- + let probe = migration + .records + .iter() + .copied() + .find(|&(idx, _)| dim > 0 && mem.cache.tombstones[idx] == 0); + let search_checked = match probe { + None => false, + Some((idx, _)) => { + let query = mem.cache.embeddings[idx].to_vec(); + let text = mem.cache.chunks[idx].clone(); + let hits = mem.search(&query, &text, &SearchOptions::new(10)); + // A record with the same text is as good a hit: the source may + // hold duplicates, and they tie. + if !hits.iter().any(|h| h.index == idx || h.chunk == text) { + return Err(format!( + "search for migrated record {idx} ({:?}) did not return it", + truncate(&text) + ) + .into()); + } + true + } + }; + Ok(ValidationSummary { - chunks: got.chunks.len() as u64, - sessions: got.sessions.len() as u64, - entities: got.entities.len() as u64, - relations: got.relations.len() as u64, - embedding_dim: got.embedding_dim as u64, + count: mem.count(), + active: mem.count_active(), + sessions: mem.sessions().len(), + entities: mem.knowledge().entities.len(), + relations: mem.knowledge().relations.len(), + embedding_dim: dim, + float16, rows_checked, - provenance_verified, + search_checked, }) } fn check_count(kind: &str, got: usize, expected: usize) -> Result<(), BoxErr> { if got != expected { - return Err(format!("{kind} count mismatch: HDF5 has {got}, source has {expected}").into()); + return Err(format!("{kind} count mismatch: store has {got}, expected {expected}").into()); } Ok(()) } -/// Re-verify the SHA-256 provenance hash of `chunks/text` and -/// `chunks/embeddings` against their actual stored bytes, catching -/// post-write corruption that a plain content comparison against the -/// in-memory source wouldn't (the source is compared against what -/// `read_hdf5` decoded, not against the raw bytes on disk). -/// -/// Returns `Ok(true)` only if both datasets exist and both hashes match. -/// Returns `Ok(false)` (not an error) if a dataset has no provenance -/// attributes at all (e.g. a file written before this check existed) or -/// there are zero chunks. Returns an error only on an actual hash mismatch — -/// that indicates real corruption. -fn verify_chunk_provenance(path: &str) -> Result { - let file = Hdf5File::open(path)?; - let Ok(chunks) = file.group("chunks") else { - return Ok(false); - }; - let mut all_present = true; - for name in ["text", "embeddings"] { - let Ok(ds) = chunks.dataset(name) else { - all_present = false; - continue; - }; - match ds.verify_provenance()? { - VerifyResult::Ok => {} - VerifyResult::NoHash => all_present = false, - VerifyResult::Mismatch { stored, computed } => { - return Err(format!( - "provenance hash mismatch on chunks/{name}: stored {stored}, recomputed {computed} — data may be corrupted" - ) - .into()); - } - } - } - Ok(all_present) -} - -fn field_err(kind: &str, i: usize, field: &str, s: T, g: T) -> BoxErr { - format!("{kind}[{i}].{field} mismatch: source {s}, HDF5 {g}").into() -} - fn truncate(s: &str) -> String { if s.len() <= 40 { s.to_string() @@ -196,7 +270,7 @@ fn truncate(s: &str) -> String { } } -/// Indices of chunk rows to content-check. Full = all; otherwise a spread of +/// Indices of records to content-check. Full = all; otherwise a spread of /// representative rows (first/last and evenly-spaced interior samples). fn sample_indices(n: usize, full: bool) -> Vec { if n == 0 {