Merge pull request 'Search options (source filters, re-ranking, confidence); float16 default' (#5 ) from feat/search-options into main

Reviewed-on: #5
Merge branch 'main' into feat/search-options
2026-09-25 12:44:15 +00:00 · 2026-09-25 12:32:00 +00:00 · 2026-09-25 03:18:07 +00:00
19 changed files with 1309 additions and 2990 deletions
@@ -3,17 +3,6 @@
 ## Unreleased

 ### Upgrade Notes
- **`clawhdf5-migrate` now writes a real agent store.** Its output used to be
-  a layout of its own (`/chunks`, `/sessions`, `/entities`, `/relations`, no
-  `/meta`) that `HDF5Memory::open` rejected, so a migrated file could not be
-  used as agent memory. Files it wrote before this release are not agent
-  stores; re-run the migration. Also: embeddings default to `float16` like
-  any new store (`--f32` opts out; `--float16` is a hidden no-op); a row with
-  the wrong embedding length is an error instead of being truncated or
-  padded; `--incremental` now matches rows by content against an existing
-  store and follows the source's deleted flags; a source with no memory rows
-  needs `--embedding-dim`. The per-dataset SHA-256 provenance attributes of
-  the old layout are gone (the agent schema has no place for them).
 - **Files written by clawhdf5 now open in h5py and libhdf5.** Every `f32`
  dataset we wrote — including every agent store's embeddings — was refused
  with "sign bit position out of bounds", and every empty dataset with
@@ -57,24 +46,6 @@
  `quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
  out. The CLI's `--quantized-index` is still accepted but is now a no-op.

-### Migration
- `clawhdf5-migrate`: writes through the agent's own API (`HDF5Memory::create`
-  / `open`, `save_batch`, the session cache and knowledge graph), so there is
-  no second copy of the schema. Sessions and entities/relations carry over;
-  deleted rows become deleted records (or are left out with
-  `--skip-deleted`). Every source row is checked before the output is created,
-  so a source that cannot be migrated leaves an existing store untouched.
-  Validation reads the result back with `HDF5Memory::open_read_only`, compares
-  every field (embeddings bit for bit — `round_to_f16` of the source for a
-  `float16` store) and checks that a migrated record is found by search. The
-  `half`-based conversion is gone; `clawhdf5_format::float16` is the only one.
-  42 tests, including h5py opening a migrated store; an adversarial review's
-  two blocker and four major findings are fixed with regression tests.
- `clawhdf5-agent`: `HDF5Memory::sessions()` / `sessions_mut()`,
-  `HDF5Memory::delete_batch(&[usize])` (one save, all-or-nothing, never
-  auto-compacts), `SessionCache::add_at`, and `SessionCache` / `SessionEntry`
-  re-exported from the crate root.
-
 ### Search
 - `clawhdf5-agent`: **`HDF5Memory::search` with `SearchOptions`** — source
  filtering, re-ranking and confidence rejection in the store's own search
@@ -97,20 +68,6 @@
  activation of the `k` results it returns, not of the whole `3k` candidate
  pool it re-ranks.

-### Benchmarks
- Every undated or pre-September section of `BENCHMARKS.md` re-run on one
-  machine on one day (tank, 2026-09-24, commit 5c8323c), with the command for
-  each and every number traced back to the raw output by a separate check.
-  Where a figure moved, the section says so. Two apparent regressions were
-  isolated rather than published: knowledge-graph traversal (a real bug,
-  fixed above) and the write path, which measures the same at v2.3.0 on this
-  machine — the old 18 µs / 6.17 ms figures came from an undated run on other
-  hardware; `float16` adds ~2 µs per save and the int8 index nothing.
- New `multimodal_bench`: cross-modal search at 1K and 10K records, which the
-  README claimed but nothing measured.
- `footprint_bench` reports whether it built `float16` or `f32` stores and
-  takes `--f32`; it had kept printing "f32" after the default changed.
-
 ### Interop
 - `clawhdf5-format`: **every `f32` dataset was unreadable by h5py and
  libhdf5.** The float datatype encoder hard-coded the sign bit's position to
@@ -189,24 +146,6 @@
  knew to ask; it now only ever switches the default off.

 ### Performance
- `clawhdf5-agent`: consolidation's novelty scoring (each `add_memory` against
-  the whole working tier) computes the new record's norm once, takes each
-  comparison in one vectorised pass instead of three, and splits a working
-  tier of 4 096+ records across threads — same results, tested against the
-  old formula. It had made `consolidation_efficiency` stall at 100K; the
-  complete run now takes 8 min and fills in the 100K cycle row (46.66 ms) and
-  the memory-reduction table.
- `clawhdf5-bench`: `consolidation_efficiency` no longer prints a record-count
-  ratio as a "BM25 Speedup" (it was never measured), nor claims cycle time
-  grows sub-linearly (its own numbers grow slightly faster than linearly).
- `clawhdf5-agent`: **knowledge-graph traversal was 6.5x slower than it
-  should be.** `bfs_neighbors` and `spreading_activation` built an adjacency
-  index over the whole graph on every call (1efd82c), so a 2-hop BFS over 1K
-  entities took 155 µs. The index is now cached on `KnowledgeCache` and
-  checked against a fingerprint of the graph on each use — one pass over
-  entity ids and relation endpoints, no allocation — so any change, including
-  direct edits of its public `Vec`s, still rebuilds it (tested). BFS over 1K
-  entities: 155.1 -> 23.1 µs; spreading activation over 100: 22.8 -> 10.1 µs.
 - `clawhdf5-format`, `clawhdf5-filters`: both deflate paths hand the codec the
  whole chunk in one call, into a buffer allocated once, instead of streaming
  it through a 32 KiB buffer: about 5% on chunked writes and 10% on zlib-ng's
@@ -6,7 +6,7 @@
 [![Rust](https://img.shields.io/badge/rust-1.92%2B-orange.svg)](https://www.rust-lang.org)
 [![Tests](https://img.shields.io/badge/tests-1850%2B-brightgreen.svg)](#building)
 [![LongMemEval](https://img.shields.io/badge/LongMemEval__s-Turn--Level%20Hit@5%2081.4%25%20hybrid-blue.svg)](BENCHMARKS.md#longmemeval-results)
-[![Footprint](https://img.shields.io/badge/on--disk-~820%20B%2Frecord%20float16%2C%20synthetic%20text-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1)
+[![Footprint](https://img.shields.io/badge/on--disk-1.7%20KB%2Frecord-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1)

 ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file.

@@ -111,8 +111,8 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d
 | Keyword search | Separate FTS engine | Integrated BM25 |
 | Knowledge graph | Neo4j or none | In-file graph with spreading activation |
 | Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers |
-| Temporal queries | Custom code | Native temporal index (622 ns range query over 10K) |
-| Multi-modal | Multiple stores | Unified cross-modal search (exact scan: 842 µs over 1K records) |
+| Temporal queries | Custom code | Native temporal index (716ns) |
+| Multi-modal | Multiple stores | Unified cross-modal search |
 | Integrity | Hope for the best | Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance |
 | Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** |

@@ -120,7 +120,7 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d

 ## Performance

-The brute-force/IVF vector search, agent-memory, on-disk footprint and consolidation figures below were measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D, 8C/16T), commit 5c8323c, 384-dim embeddings; the commands are in [BENCHMARKS.md](BENCHMARKS.md). Exceptions are marked where they appear: the HDF5 Core I/O table immediately below is from a separate, independently reproduced run (see its own hardware note), and the HNSW `f32`/`i8` table and the in-memory `i8` column were not re-measured on 2026-09-24.
+Vector search and agent-memory operations below are benchmarked on Intel i7-12650H (10C/16T), 384-dim embeddings, Criterion.rs. The HDF5 Core I/O table immediately below is from a separate, independently reproduced run (see its own hardware note).

 ### HDF5 Core I/O (vs libhdf5 1.14.6)

@@ -154,45 +154,33 @@ and [§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quant
 | `f32` index | 0.9945 | 13 399 | 3.2 s |
 | `i8` index + exact re-score (**default for new stores**) | 0.9940 | **21 848** | **1.8 s** |

-Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31. These
-two rows are a paired comparison (medians of alternating runs, same binary).
-A single `f32` run on 2026-09-24 measured recall 0.9945, 19 001 QPS and a
-2.7 s build; the int8 row was not re-run, so the pair has not been re-checked
-([§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index)).
+Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31.

-**Brute-force and IVF paths** (Criterion, tank, 2026-09-24):
+**Brute-force and IVF paths** (Criterion, i7-12650H):

-| Scale | Flat | IVF (nprobe=10) | IVF-PQ | MemX¹ (claimed, end-to-end) |
+| Scale | Flat | IVF (nprobe=10) | IVF-PQ | vs MemX¹ |
 |-------|------|-----------------|--------|----------|
-| 1K | **47.4 µs** | — | — | — |
-| 10K | 500.5 µs | **24.8 µs** | — | — |
-| 100K | 6.58 ms | 592 µs | **869 µs** | <90 ms |
+| 1K | **54 µs** | — | — | — |
+| 10K | 753 µs | **27 µs** | — | — |
+| 100K | 11.4 ms | 1.32 ms | **1.19 ms** | ~8–76× (see caveat) |

-> These replace figures from the original i7-12650H run (flat 54 µs / 753 µs /
-> 11.4 ms); a 2026-08-05 run on tank had already matched the new ones — see
-> [BENCHMARKS.md § Vector Search Latency](BENCHMARKS.md#vector-search-latency).
+> Reproduced on the same second machine (Ryzen 7 7800X3D) with a corrected,
+> apples-to-apples SIMD/scalar/parallel comparison methodology — see
+> [BENCHMARKS.md § Independent Validation: tank — LongMemEval & Vector
+> Search](BENCHMARKS.md#independent-validation-tank--longmemeval--vector-search-ryzen-7-7800x3d-2026-08-05).

 ### Agent Memory Operations

 | Operation | Latency | Scale |
 |-----------|---------|-------|
-| Hybrid search (`HDF5Memory::hybrid_search`, p50) | **0.07 ms** / 0.49 ms / 4.69 ms | 1K / 10K / 100K records |
-| BM25 keyword search | **20.4 µs** | 1K records |
-| Knowledge graph BFS | **23.1 µs** | 1K entities |
-| Spreading activation | **10.1 µs** | 100 entities |
-| Temporal range query | **622 ns** | 10K timestamps |
-| Consolidation cycle | **115.2 µs** | 1K records |
-| Cross-modal search (exact scan, 2 embeddings per record) | **842.0 µs** / 8.44 ms | 1K / 10K records |
-| Memory write (WAL) | **26.1 µs** | per record (group-commit append; HDF5 batched at flush) |
-| Importance gate | **57.6 ns** | per record (trivial skip) |
-
-The old 18 µs WAL write was undated, from another machine: v2.3.0 measures
-24.3 µs on the same hardware as this table, the same as an `f32` store today.
-`float16` stores (the new default) add ~2 µs for rounding; the int8 index adds
-nothing. See [BENCHMARKS.md § Write Path](BENCHMARKS.md#write-path).
-Knowledge-graph traversal was briefly 6.5x slower (155 µs) until this re-run
-found and fixed an adjacency index rebuilt on every traversal; see
-[§ Knowledge Graph](BENCHMARKS.md#knowledge-graph).
+| Hybrid search (`HDF5Memory::hybrid_search`, p50) | **70 µs** / 0.49 ms / 4.65 ms | 1K / 10K / 100K records |
+| BM25 keyword search | **67 µs** | 1K records |
+| Knowledge graph BFS | **24 µs** | 1K entities |
+| Spreading activation | **17 µs** | 100 entities |
+| Temporal range query | **716 ns** | 10K timestamps |
+| Consolidation cycle | **164 µs** | 1K records |
+| Memory write (WAL) | **18 µs** | per record (group-commit append; HDF5 batched at flush) |
+| Importance gate | **61 ns** | per record |

 ### Chunked Write Throughput (codec comparison)

@@ -207,7 +195,7 @@ by default (AoS→SoA byte transpose, +157–204% throughput for float data):

 Use `.with_zstd(3)` or `.with_deflate(6)` for write-heavy workloads — both now perform at ~720–750 MiB/s on large matrices. Use `.with_pcodec()` for write-once/read-many workloads where compression ratio matters more than encode speed. Disable auto-shuffle with `.without_shuffle()` for byte arrays that don't benefit from AoS→SoA transposition.

-> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search), so the two columns are not comparable and no ratio is given. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171).
+> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search). The ratio overstates the real advantage by an unquantified margin — order-of-magnitude indication only. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171).

 ### LongMemEval Retrieval Recall

@@ -260,26 +248,18 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points.

 ### Memory Footprint

-**On disk** — 384-dim `float16` embeddings (the default for new stores),
-200-char text, `footprint_bench`
+**On disk** — 384-dim embeddings, 200-char text
 ([BENCHMARKS.md § Memory Footprint](BENCHMARKS.md#memory-footprint-1)):

 | Records | File Size | Bytes/Record | Gzip-6 compressed |
 |---------|-----------|--------------|-------------------|
-| 1K | 810.4 KB | 829 B | 56.4 KB |
-| 10K | 7.8 MB | 820 B | 471.3 KB |
-| 100K | 76.7 MB | 803 B | 4.5 MB |
+| 1K | 1.7 MB | 1.8 KB | 277 KB (6.1x) |
+| 10K | 17.0 MB | 1.7 KB | 2.7 MB (6.2x) |
+| 100K | 169.8 MB | 1.7 KB | 26.9 MB (6.2x) |

-The benchmark's synthetic embeddings and text are far more repetitive than
-real data (only 40 distinct texts), so no column here is an expectation for
-real data. The compressed column is an upper bound, and the Bytes/Record
-column is optimistic too: it is not an uncompressed figure, because the store
-always deflates its text (any string dataset of 4 KiB or more) whatever
-`MemoryConfig::compression` says. The `float16` embeddings alone are 768 B per
-record, so 200 characters of real text would take a record above 820 B.
-This table used to show `f32` stores (1.7 KB per record, 169.8 MB at 100K);
-those were not re-measured. The float16 study compares the two on the same
-data: 100K × 384 records take 80.8 MiB as `float16` and 154.0 MiB as `f32`.
+These figures are `f32` embeddings. New agent stores default to
+`MemoryConfig::float16`, which halves them: 100K × 384 records take 80.8 MiB
+instead of 154.0.

 **In memory** — a store reopened from disk, 384-dim `f32`, measured with a
 counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-footprint)):
@@ -291,8 +271,7 @@ counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-foo
 | 100K | 146 MiB | 399 MiB (2.72x) | **256 MiB (1.74x)** |

 Down from 505 MiB (3.44x) at 100K before v2.6.0, when the cache held every
-embedding twice. The `f32` column was re-measured on 2026-09-24 and reproduced
-exactly; the `i8` column was not re-run.
+embedding twice.

 ### Consolidation Efficiency

@@ -303,10 +282,7 @@ exactly; the `i8` column was not re-run.
 |--------|--------|-------|-------|
 | Records in store | 1,000 | 100 | −90% |
 | Hit@1 recall (signal records) | 100% | 100% | no loss |
-| Search latency (avg) | 2.22 ms | 0.24 ms | **9.3x faster** |
-
-The consolidation cycle that does this took 0.13 ms; a cycle over 10K records
-takes 2.81 ms and over 100K 46.7 ms.
+| Search latency | 2.75 ms | 0.31 ms | **8.8x faster** |

 **Full benchmark details: [BENCHMARKS.md](BENCHMARKS.md)**

@@ -778,37 +754,9 @@ Replace in `Cargo.toml` and source:

 ```bash
 cargo install --path crates/clawhdf5-migrate
-clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedder minilm
+clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedding-dim 384
 ```

-The output is an ordinary `clawhdf5-agent` store, written through the agent's
-own API: open it with `HDF5Memory::open` (or `clawhdf5-cli --path memory.h5 …`)
-and search it straight away. What carries over from the ZeroClaw tables:
-
-| SQLite | Agent store |
-|--------|-------------|
-| `memory_chunks` | memory records (text, embedding, source channel, timestamp, session id, tags); rows with `deleted = 1` become deleted records, or are left out with `--skip-deleted` |
-| `sessions` | sessions (id, start/end index, channel, summary, timestamp) |
-| `entities`, `relations` | knowledge graph entities and relations; entities get new ids and relations are re-pointed at them |
-
-The chunk `id` column has no counterpart in the agent store, so records are
-written in `id` order and numbered from 0. Embeddings are stored as float16
-like any new store; `--f32` keeps full precision (and is required for values
-beyond ±65504). The embedding dimension is detected from the first row unless
-`--embedding-dim` is given, and every row must have it: a row of another length
-is an error, never truncated or padded. A source with no memory records (only
-sessions or the graph) needs `--embedding-dim`, since a store's dimension is
-fixed when it is created. Every row is checked before the output is created,
-so a source that cannot be migrated leaves an existing store at `--hdf5` as it
-was. `--incremental` adds to an existing store only the rows it does not
-already hold; the source must have the store's dimension, and records already
-in the store take the source's deleted flag (a row deleted in SQLite since the
-last run is deleted in the store; one un-deleted there is written again, as
-the agent has no un-delete). The tool reads the result back with
-`HDF5Memory::open_read_only`, compares it with the source (every row with
-`--validate-full`) and checks that a migrated record is found by search;
-`--dry-run` only counts the rows.
-
 ---

 ## Roadmap
@@ -45,10 +45,6 @@ harness = false
 name = "memory_bench"
 harness = false

-[[bench]]
-name = "multimodal_bench"
-harness = false
-
 [features]
 default = ["float16", "hnsw", "parallel"]
 float16 = ["half"]
@@ -1,107 +0,0 @@
-//! Multi-modal memory search benchmarks (`clawhdf5_agent::multimodal`).
-//!
-//! Covers `MultiModalStore::search_cross_modal` (every embedding of every
-//! record, whatever its modality) and, for comparison,
-//! `MultiModalStore::search_by_modality` restricted to one modality.
-//!
-//! Corpus: N records (1K and 10K), each carrying two 384-dim embeddings —
-//! a text embedding of its caption plus one embedding of its primary modality,
-//! cycling Image / Audio / Video — so a cross-modal query scores 2N vectors.
-//! All data comes from a fixed-seed LCG, so every run sees the same corpus.
-//!
-//! Run: `cargo bench -p clawhdf5-agent --bench multimodal_bench`
-
-use std::collections::HashMap;
-
-use clawhdf5_agent::multimodal::{
-    MediaRef, ModalEmbedding, Modality, MultiModalRecord, MultiModalStore,
-};
-use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
-
-// ---------------------------------------------------------------------------
-// Simple deterministic PRNG (LCG), same as the other agent benches
-// ---------------------------------------------------------------------------
-
-struct Rng(u32);
-
-impl Rng {
-    fn new(seed: u32) -> Self {
-        Self(seed)
-    }
-    fn next_u32(&mut self) -> u32 {
-        self.0 = self.0.wrapping_mul(1103515245).wrapping_add(12345);
-        self.0 >> 16
-    }
-    fn next_f32(&mut self) -> f32 {
-        self.next_u32() as f32 / 65536.0 - 0.5
-    }
-}
-
-fn make_vec(rng: &mut Rng, dim: usize) -> Vec<f32> {
-    (0..dim).map(|_| rng.next_f32()).collect()
-}
-
-// ---------------------------------------------------------------------------
-// Corpus
-// ---------------------------------------------------------------------------
-
-const DIM: usize = 384;
-const K: usize = 10;
-
-const MEDIA: [(Modality, &str, &str); 3] = [
-    (Modality::Image, "image/png", "clip-vit-base"),
-    (Modality::Audio, "audio/wav", "clap-base"),
-    (Modality::Video, "video/mp4", "xclip-base"),
-];
-
-fn build_store(n: usize, seed: u32) -> MultiModalStore {
-    let mut rng = Rng::new(seed);
-    let mut store = MultiModalStore::new();
-    for i in 0..n {
-        let (modality, mime, model) = &MEDIA[i % MEDIA.len()];
-        let embeddings = vec![
-            ModalEmbedding::new(Modality::Text, make_vec(&mut rng, DIM), "minilm-l6"),
-            ModalEmbedding::new(modality.clone(), make_vec(&mut rng, DIM), *model),
-        ];
-        store.add_record(MultiModalRecord {
-            id: 0,
-            primary_modality: modality.clone(),
-            text_content: Some(format!("{modality} memory {i}")),
-            media_ref: Some(MediaRef::path(format!("/media/{i}"), *mime)),
-            embeddings,
-            observation: None,
-            timestamp: 1_700_000_000.0 + i as f64,
-            metadata: HashMap::new(),
-        });
-    }
-    store
-}
-
-// ---------------------------------------------------------------------------
-// Benchmarks
-// ---------------------------------------------------------------------------
-
-fn multimodal_search_benches(c: &mut Criterion) {
-    let query = make_vec(&mut Rng::new(99), DIM);
-
-    let mut group = c.benchmark_group("multimodal_search");
-    group.sample_size(50);
-
-    for (label, n) in [("1k", 1_000usize), ("10k", 10_000)] {
-        let store = build_store(n, 42);
-        assert_eq!(store.count(), n);
-
-        group.bench_with_input(BenchmarkId::new("cross_modal", label), &n, |b, _| {
-            b.iter(|| store.search_cross_modal(&query, K));
-        });
-
-        group.bench_with_input(BenchmarkId::new("by_modality_image", label), &n, |b, _| {
-            b.iter(|| store.search_by_modality(&Modality::Image, &query, K));
-        });
-    }
-
-    group.finish();
-}
-
-criterion_group!(multimodal_benches, multimodal_search_benches);
-criterion_main!(multimodal_benches);
@@ -144,44 +144,9 @@ pub struct ConsolidationStats {

 pub struct ImportanceScorer;

-/// Sum of squares, in 8-wide lanes so it vectorises.
-fn sum_of_squares(a: &[f32]) -> f32 {
-    let (blocks, tail) = a.as_chunks::<8>();
-    let mut acc = [0.0f32; 8];
-    for b in blocks {
-        for i in 0..8 {
-            acc[i] += b[i] * b[i];
-        }
-    }
-    acc.iter().sum::<f32>() + tail.iter().map(|x| x * x).sum::<f32>()
-}
-
-/// `(a · b, |b|²)` in one pass over equal-length slices, in 8-wide lanes.
-fn dot_and_norm2(a: &[f32], b: &[f32]) -> (f32, f32) {
-    let (a_blocks, a_tail) = a.as_chunks::<8>();
-    let (b_blocks, b_tail) = b.as_chunks::<8>();
-    let mut dot = [0.0f32; 8];
-    let mut nb = [0.0f32; 8];
-    for (x, y) in a_blocks.iter().zip(b_blocks) {
-        for i in 0..8 {
-            dot[i] += x[i] * y[i];
-            nb[i] += y[i] * y[i];
-        }
-    }
-    let mut d = dot.iter().sum::<f32>();
-    let mut n = nb.iter().sum::<f32>();
-    for (x, y) in a_tail.iter().zip(b_tail) {
-        d += x * y;
-        n += y * y;
-    }
-    (d, n)
-}
-
 impl ImportanceScorer {
    /// Cosine similarity between two embedding slices.
-    /// Returns 0.0 if either norm is zero. The reference that
-    /// [`Self::score_surprise`] is tested against.
-    #[cfg(test)]
+    /// Returns 0.0 if either norm is zero.
    fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 {
        let len = a.len().min(b.len());
        if len == 0 {
@@ -202,54 +167,13 @@ impl ImportanceScorer {

    /// Novelty score: 1.0 − max cosine similarity against all existing records.
    /// Returns 1.0 when there are no existing memories.
-    ///
-    /// Same result as [`Self::cosine_similarity`] against each record, but the
-    /// new embedding's norm is computed once rather than per record, each
-    /// record costs one fused pass (dot product and its norm together) rather
-    /// than three, and a large working set is scored in parallel. Every insert
-    /// scores against the whole working tier, so this is what an unbounded
-    /// working tier pays for: at 100K records it was the difference between a
-    /// benchmark finishing and not (`BENCHMARKS.md`, "Consolidation Efficiency").
    pub fn score_surprise(embedding: &[f32], existing_memories: &[&MemoryRecord]) -> f32 {
        if existing_memories.is_empty() {
            return 1.0;
        }
-        let query_norm2 = sum_of_squares(embedding);
-        let similarity = |r: &&MemoryRecord| -> f32 {
-            let other = &r.embedding;
-            let len = embedding.len().min(other.len());
-            if len == 0 {
-                return 0.0;
-            }
-            let (dot, other_norm2) = dot_and_norm2(&embedding[..len], &other[..len]);
-            // A shorter record compares against the query's matching prefix.
-            let q2 = if len == embedding.len() {
-                query_norm2
-            } else {
-                sum_of_squares(&embedding[..len])
-            };
-            if q2 == 0.0 || other_norm2 == 0.0 {
-                return 0.0;
-            }
-            dot / (q2.sqrt() * other_norm2.sqrt())
-        };
-        #[cfg(feature = "parallel")]
-        let max_sim = if existing_memories.len() >= 4096 {
-            use rayon::prelude::*;
-            existing_memories
-                .par_iter()
-                .map(similarity)
-                .reduce(|| f32::NEG_INFINITY, f32::max)
-        } else {
-            existing_memories
-                .iter()
-                .map(similarity)
-                .fold(f32::NEG_INFINITY, f32::max)
-        };
-        #[cfg(not(feature = "parallel"))]
        let max_sim = existing_memories
            .iter()
-            .map(similarity)
+            .map(|r| Self::cosine_similarity(embedding, &r.embedding))
            .fold(f32::NEG_INFINITY, f32::max);
        (1.0 - max_sim).clamp(0.0, 1.0)
    }
@@ -547,54 +471,6 @@ impl ConsolidationEngine {
 mod tests {
    use super::*;

-    #[test]
-    fn score_surprise_matches_the_reference_cosine() {
-        let mut x = 0x2545_F491_4F6C_DD1Du64;
-        let mut next = || {
-            x ^= x << 13;
-            x ^= x >> 7;
-            x ^= x << 17;
-            (x >> 40) as f32 / (1u64 << 24) as f32 - 0.5
-        };
-        let make = |id: u64, v: Vec<f32>| MemoryRecord {
-            id,
-            chunk: String::new(),
-            embedding: v,
-            tier: MemoryTier::Working,
-            importance: 0.0,
-            access_count: 0,
-            last_accessed: 0.0,
-            created_at: 0.0,
-            source: MemorySource::User,
-        };
-        // Ordinary rows, a shorter one, an empty one and a zero vector; and
-        // enough rows to take the parallel path too.
-        for n in [5usize, 5000] {
-            let mut recs: Vec<MemoryRecord> = (0..n as u64)
-                .map(|i| make(i, (0..37).map(|_| next()).collect()))
-                .collect();
-            recs.push(make(9_000, (0..20).map(|_| next()).collect()));
-            recs.push(make(9_001, Vec::new()));
-            recs.push(make(9_002, vec![0.0; 37]));
-            let refs: Vec<&MemoryRecord> = recs.iter().collect();
-            for _ in 0..5 {
-                let q: Vec<f32> = (0..37).map(|_| next()).collect();
-                let expected = (1.0
-                    - refs
-                        .iter()
-                        .map(|r| ImportanceScorer::cosine_similarity(&q, &r.embedding))
-                        .fold(f32::NEG_INFINITY, f32::max))
-                .clamp(0.0, 1.0);
-                let got = ImportanceScorer::score_surprise(&q, &refs);
-                assert!((got - expected).abs() < 1e-5, "n={n}: {got} vs {expected}");
-            }
-        }
-        assert_eq!(
-            ImportanceScorer::score_surprise(&[0.0; 4], &[&make(1, vec![1.0; 4])]),
-            1.0
-        );
-    }
-
    // Helper: build a simple normalised embedding of given dimension.
    fn unit_vec(dim: usize, hot: usize) -> Vec<f32> {
        let mut v = vec![0.0f32; dim];
@@ -163,13 +163,12 @@ fn levenshtein(a: &str, b: &str) -> usize {
 /// entities-slice-index map, and an entity-id -> relation-indices map (edges
 /// touching that entity as either source or target).
 ///
-/// Cached on `KnowledgeCache` and checked against a fingerprint of the graph
-/// on every use ([`graph_fingerprint`]). entities/relations are plain `pub`
-/// `Vec`s that get changed directly (e.g. `schema.rs`'s load path bypasses
-/// `add_entity`/`add_relation`), so the cache cannot rely on being told about
-/// changes; the fingerprint notices any of them. Rebuilding it on every
-/// traversal instead made a 2-hop BFS over 1K entities 6.5x slower than the
-/// scan it replaced (24 -> 155 µs; `BENCHMARKS.md`, "Knowledge Graph").
+/// Built fresh per traversal call rather than cached on `KnowledgeCache`:
+/// entities/relations are plain `pub` `Vec`s that get pushed to directly
+/// (e.g. `schema.rs`'s load path bypasses `add_entity`/`add_relation`), so a
+/// persistent index would need extra bookkeeping to avoid drifting stale. A
+/// one-off O(V+E) build per call is still a large win over the O(V·E) (BFS)
+/// / O(steps·active·E) (spreading activation) scans it replaces.
 struct AdjacencyIndex {
    entity_index: HashMap<u64, usize>,
    by_entity: HashMap<u64, Vec<usize>>,
@@ -205,45 +204,6 @@ impl AdjacencyIndex {
    }
 }

-/// A hash of everything [`AdjacencyIndex`] depends on — each entity's id and
-/// position, each relation's endpoints and position. One linear pass, no
-/// allocation: far cheaper than building the index, which hashes the same
-/// values into two maps.
-fn graph_fingerprint(entities: &[Entity], relations: &[Relation]) -> u64 {
-    // splitmix64-style mixing; order matters, so positions are covered.
-    fn mix(h: u64, v: u64) -> u64 {
-        let mut z = (h ^ v).wrapping_add(0x9E37_79B9_7F4A_7C15);
-        z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
-        z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
-        z ^ (z >> 31)
-    }
-    let mut h = mix(entities.len() as u64, relations.len() as u64);
-    for e in entities {
-        h = mix(h, e.id);
-    }
-    for r in relations {
-        h = mix(mix(h, r.src), r.tgt);
-    }
-    h
-}
-
-/// The cached [`AdjacencyIndex`] and the fingerprint it was built for.
-/// Cloning a `KnowledgeCache` starts the clone with an empty cache.
-#[derive(Default)]
-struct AdjacencyCache(std::sync::Mutex<Option<(u64, std::sync::Arc<AdjacencyIndex>)>>);
-
-impl Clone for AdjacencyCache {
-    fn clone(&self) -> Self {
-        Self::default()
-    }
-}
-
-impl std::fmt::Debug for AdjacencyCache {
-    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
-        f.write_str("AdjacencyCache")
-    }
-}
-
 // ---------------------------------------------------------------------------
 // KnowledgeCache
 // ---------------------------------------------------------------------------
@@ -256,7 +216,6 @@ pub struct KnowledgeCache {
    pub alias_strings: Vec<String>,
    pub alias_entity_ids: Vec<i64>,
    next_entity_id: u64,
-    adjacency: AdjacencyCache,
 }

 impl KnowledgeCache {
@@ -267,7 +226,6 @@ impl KnowledgeCache {
            alias_strings: Vec::new(),
            alias_entity_ids: Vec::new(),
            next_entity_id: 0,
-            adjacency: AdjacencyCache::default(),
        }
    }

@@ -278,29 +236,9 @@ impl KnowledgeCache {
            alias_strings: Vec::new(),
            alias_entity_ids: Vec::new(),
            next_entity_id: next_id,
-            adjacency: AdjacencyCache::default(),
        }
    }

-    /// The adjacency index for the graph as it is now: the cached one if the
-    /// graph's fingerprint still matches, otherwise rebuilt and cached.
-    fn adjacency_index(&self) -> std::sync::Arc<AdjacencyIndex> {
-        let fp = graph_fingerprint(&self.entities, &self.relations);
-        let mut slot = self
-            .adjacency
-            .0
-            .lock()
-            .unwrap_or_else(std::sync::PoisonError::into_inner);
-        if let Some((cached_fp, idx)) = slot.as_ref()
-            && *cached_fp == fp
-        {
-            return idx.clone();
-        }
-        let idx = std::sync::Arc::new(AdjacencyIndex::build(&self.entities, &self.relations));
-        *slot = Some((fp, idx.clone()));
-        idx
-    }
-
    // -----------------------------------------------------------------------
    // Entity management
    // -----------------------------------------------------------------------
@@ -459,7 +397,7 @@ impl KnowledgeCache {
    /// together with their discovered depth.  The seed entity itself is NOT
    /// included.  Traversal follows both outgoing and incoming relation edges.
    pub fn bfs_neighbors(&self, entity_id: u64, max_depth: usize) -> Vec<(Entity, usize)> {
-        let idx = self.adjacency_index();
+        let idx = AdjacencyIndex::build(&self.entities, &self.relations);
        let mut visited: HashSet<u64> = HashSet::new();
        let mut queue: VecDeque<(u64, usize)> = VecDeque::new();
        let mut results: Vec<(Entity, usize)> = Vec::new();
@@ -564,7 +502,7 @@ impl KnowledgeCache {
        min_activation: f32,
        max_steps: usize,
    ) -> Vec<(u64, f32)> {
-        let idx = self.adjacency_index();
+        let idx = AdjacencyIndex::build(&self.entities, &self.relations);
        let mut activation: HashMap<u64, f32> = HashMap::new();

        // Initialise seeds with activation 1.0.
@@ -693,51 +631,6 @@ impl Default for KnowledgeCache {
 mod tests {
    use super::*;

-    #[test]
-    fn cached_adjacency_sees_direct_changes_to_the_graph() {
-        // The index is cached across traversals, but entities/relations are
-        // pub Vecs anyone can edit; every kind of edit must be seen.
-        let mut kg = KnowledgeCache::new();
-        let a = kg.add_entity("a", "t", -1);
-        let b = kg.add_entity("b", "t", -1);
-        let c = kg.add_entity("c", "t", -1);
-        kg.add_relation(a, b, "r", 1.0);
-        let ids = |kg: &KnowledgeCache| -> Vec<u64> {
-            let mut v: Vec<u64> = kg.bfs_neighbors(a, 3).iter().map(|(e, _)| e.id).collect();
-            v.sort();
-            v
-        };
-        assert_eq!(ids(&kg), vec![b]);
-        assert_eq!(ids(&kg), vec![b], "cached index reused");
-
-        // Pushed directly, bypassing add_relation.
-        kg.relations.push(Relation {
-            src: b,
-            tgt: c,
-            ..Relation::default()
-        });
-        assert_eq!(ids(&kg), vec![b, c]);
-
-        // Rewired in place: same lengths, different edge.
-        kg.relations[1].tgt = a;
-        assert_eq!(ids(&kg), vec![b]);
-
-        // Removed and replaced: same lengths again.
-        kg.relations.pop();
-        kg.relations.push(Relation {
-            src: a,
-            tgt: c,
-            ..Relation::default()
-        });
-        assert_eq!(ids(&kg), vec![b, c]);
-        let act: Vec<u64> = kg
-            .spreading_activation(&[a], 0.5, 0.0, 2)
-            .iter()
-            .map(|(id, _)| *id)
-            .collect();
-        assert!(act.contains(&c));
-    }
-
    // -----------------------------------------------------------------------
    // Original tests — must remain passing
    // -----------------------------------------------------------------------
@@ -73,7 +73,7 @@ pub use ephemeral::{EphemeralEntry, EphemeralStats};
 use knowledge::KnowledgeCache;
 use memory_strategy::{Exchange, MemoryStrategy, StrategyOutput};
 pub use search::SearchOptions;
-pub use session::{SessionCache, SessionEntry};
+use session::SessionCache;

 // --- Error type ---

@@ -1028,18 +1028,6 @@ impl HDF5Memory {
        &self.config
    }

-    /// The sessions recorded in this store.
-    pub fn sessions(&self) -> &SessionCache {
-        &self.sessions
-    }
-
-    /// Mutable access to the sessions, e.g. to add many at once. Changes
-    /// reach the disk at the next checkpoint (any flushing call, such as
-    /// [`HDF5Memory::flush_wal`] or `save_batch`), not immediately.
-    pub fn sessions_mut(&mut self) -> &mut SessionCache {
-        &mut self.sessions
-    }
-
    /// Get a reference to the knowledge cache.
    pub fn knowledge(&self) -> &KnowledgeCache {
        &self.knowledge
@@ -1413,35 +1401,6 @@ impl HDF5Memory {
 }

 impl HDF5Memory {
-    /// Delete many records with a single checkpoint, where
-    /// [`AgentMemory::delete`] checkpoints once per record.
-    ///
-    /// All or nothing: if any id is out of range or already deleted (or
-    /// repeated), nothing is deleted and `MemoryError::NotFound` is returned.
-    /// Unlike `delete`, this never auto-compacts, so the records stay in the
-    /// store as tombstones (their indices unchanged) until [`AgentMemory::compact`]
-    /// is called — importers use it to carry over records that were already
-    /// deleted in the source.
-    pub fn delete_batch(&mut self, ids: &[usize]) -> Result<()> {
-        let mut seen = std::collections::HashSet::with_capacity(ids.len());
-        for &id in ids {
-            if self.cache.tombstones.get(id).copied() != Some(0) || !seen.insert(id) {
-                return Err(MemoryError::NotFound(format!(
-                    "entry {id} not found or already deleted"
-                )));
-            }
-        }
-        if ids.is_empty() {
-            return Ok(());
-        }
-        for &id in ids {
-            self.cache.mark_deleted(id);
-            self.hnsw_on_delete(id);
-            self.bm25_on_delete(id);
-        }
-        self.flush()
-    }
-
    pub fn tick_session(&mut self) -> Result<()> {
        let d = self.config.decay_factor;
        for w in self.cache.activation_weights.iter_mut() {
@@ -1640,79 +1599,6 @@ mod tests {
        }
    }

-    #[test]
-    fn delete_batch_tombstones_without_compacting() {
-        let dir = TempDir::new().unwrap();
-        let path = dir.path().join("test.h5");
-        let mut mem = HDF5Memory::create(make_config(&dir)).unwrap();
-        mem.save_batch(
-            (0..4)
-                .map(|i| make_entry(&format!("record {i}"), &[i as f32, 1.0, 0.0, 0.0]))
-                .collect(),
-        )
-        .unwrap();
-        // 3 of 4 is far past compact_threshold (0.3): delete() would compact.
-        mem.delete_batch(&[0, 1, 3]).unwrap();
-        assert_eq!(mem.count(), 4);
-        assert_eq!(mem.count_active(), 1);
-        drop(mem);
-
-        let mut mem = HDF5Memory::open(&path).unwrap();
-        assert_eq!(mem.cache.tombstones, vec![1, 1, 0, 1]);
-        let hits = mem.hybrid_search(&[0.0, 1.0, 0.0, 0.0], "record", 0.5, 0.5, 10);
-        assert!(
-            hits.iter().all(|r| r.index == 2),
-            "tombstoned record returned"
-        );
-    }
-
-    #[test]
-    fn delete_batch_is_all_or_nothing() {
-        let dir = TempDir::new().unwrap();
-        let mut mem = HDF5Memory::create(make_config(&dir)).unwrap();
-        mem.save_batch(vec![
-            make_entry("a", &[1.0, 0.0, 0.0, 0.0]),
-            make_entry("b", &[0.0, 1.0, 0.0, 0.0]),
-        ])
-        .unwrap();
-        for bad in [&[0, 5][..], &[1, 1][..]] {
-            assert!(matches!(
-                mem.delete_batch(bad),
-                Err(MemoryError::NotFound(_))
-            ));
-            assert_eq!(mem.count_active(), 2, "{bad:?} deleted something");
-        }
-        mem.delete_batch(&[]).unwrap();
-        assert_eq!(mem.count_active(), 2);
-    }
-
-    #[test]
-    fn sessions_mut_add_at_keeps_timestamp_across_reopen() {
-        let dir = TempDir::new().unwrap();
-        let path = dir.path().join("test.h5");
-        let mut mem = HDF5Memory::create(make_config(&dir)).unwrap();
-        mem.sessions_mut()
-            .add_at("s-old", 2, 7, "discord", "old summary", 1.7e15);
-        mem.flush_wal().unwrap();
-        drop(mem);
-
-        let mem = HDF5Memory::open_read_only(&path).unwrap();
-        let s = mem.sessions();
-        assert_eq!(s.len(), 1);
-        let e = &s.entries[0];
-        assert_eq!(
-            (
-                e.id.as_str(),
-                e.start_idx,
-                e.end_idx,
-                e.channel.as_str(),
-                e.ts
-            ),
-            ("s-old", 2, 7, "discord", 1.7e15)
-        );
-        assert_eq!(s.summaries[0], "old summary");
-    }
-
    #[test]
    fn create_new_file() {
        let dir = TempDir::new().unwrap();
@@ -33,7 +33,7 @@ impl SessionCache {
        self.entries.is_empty()
    }

-    /// Add a new session with its summary, timestamped now.
+    /// Add a new session with its summary.
    pub fn add(
        &mut self,
        id: &str,
@@ -47,21 +47,6 @@ impl SessionCache {
            .unwrap_or_default()
            .as_secs_f64()
            * 1_000_000.0; // microseconds
-        self.add_at(id, start_idx, end_idx, channel, summary, ts);
-    }
-
-    /// Add a session with an explicit timestamp (Unix **microseconds**, the
-    /// unit [`SessionEntry::ts`] uses) — for importers carrying sessions over
-    /// from another store, whose original time should be kept.
-    pub fn add_at(
-        &mut self,
-        id: &str,
-        start_idx: usize,
-        end_idx: usize,
-        channel: &str,
-        summary: &str,
-        ts: f64,
-    ) {
        self.entries.push(SessionEntry {
            id: id.to_string(),
            start_idx: start_idx as u64,
@@ -396,7 +396,7 @@ fn run_memory_reduction_benchmark() {
    println!();
    println!(
        "{:>8}  {:>10}  {:>10}  {:>10}  {:>12}",
-        "Initial", "Remaining", "Eviction%", "Signal OK?", "Records ÷"
+        "Initial", "Remaining", "Eviction%", "Signal OK?", "BM25 Speedup"
    );
    println!("{}", "-".repeat(58));

@@ -440,8 +440,7 @@ fn run_memory_reduction_benchmark() {
        // Check all signal records survived
        let signal_survived = signal_ids.iter().all(|&id| engine.get_by_id(id).is_some());

-        // How many times fewer records there are. Not a measured speedup —
-        // Part 1 measures search latency before and after.
+        // Rough speedup: BM25 scales roughly linearly with record count
        let speedup = before_count as f64 / after_count.max(1) as f64;

        println!(
@@ -481,7 +480,7 @@ fn main() {
    println!("  3. Reducing search latency proportional to record reduction");
    println!();
    println!(
-        "Cycle time grows a little faster than linearly: 100 records ~microseconds, 100K records ~tens of ms."
+        "Cycle time scales sub-linearly: 100 records ~microseconds, 100K records ~tens of ms."
    );
    println!("Signal records with Correction source + high access_count survive eviction.");
 }
@@ -11,14 +11,12 @@
 //!
 //! Configuration matrix:
 //!   - Text lengths: short (50 chars), medium (200 chars), long (1000 chars)
-//!   - Embedding: 384-dim, stored as float16 (the default for new stores) or
-//!     f32 with `--f32`; "raw" bytes are counted as f32 input either way
+//!   - Embedding: 384-dim f32 (1536 bytes raw per record)
 //!   - WAL: enabled and disabled
 //!
 //! # Usage
 //! ```
-//! cargo run --release --bin footprint_bench           # float16 stores
-//! cargo run --release --bin footprint_bench -- --f32  # f32 stores
+//! cargo run --release --bin footprint_bench
 //! ```

 use std::time::Instant;
@@ -26,9 +24,6 @@ use std::time::Instant;
 use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
 use tempfile::TempDir;

-/// `--f32`: build f32 stores instead of the library's float16 default.
-static F32: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
-
 const EMBEDDING_DIM: usize = 384;

 // Raw bytes per record: 384 f32 embeddings + median text + overhead
@@ -157,9 +152,6 @@ fn measure_footprint(
    config.compression = compression;
    config.compression_level = if compression { 6 } else { 0 };
    config.compact_threshold = 0.0;
-    if F32.load(std::sync::atomic::Ordering::Relaxed) {
-        config.float16 = false;
-    }

    let mut memory = HDF5Memory::create(config).expect("HDF5Memory::create failed");

@@ -249,19 +241,11 @@ fn fmt_n(n: usize) -> String {
 // ---------------------------------------------------------------------------

 fn main() {
-    if std::env::args().skip(1).any(|a| a == "--f32") {
-        F32.store(true, std::sync::atomic::Ordering::Relaxed);
-    }
-    let stored = if F32.load(std::sync::atomic::Ordering::Relaxed) {
-        "f32 (1,536 bytes per record)"
-    } else {
-        "float16 (768 bytes per record; the default for new stores)"
-    };
    println!("=================================================================");
    println!("  ClawhDF5 Memory Footprint Benchmark");
    println!("=================================================================");
    println!();
-    println!("Embedding: 384-dim, stored as {stored}; raw input counted as f32");
+    println!("Embedding: 384-dim f32 = 1,536 bytes raw per record");
    println!("Text lengths: short=50 chars, medium=200 chars, long=1000 chars");
    println!();

@@ -3,7 +3,7 @@ name = "clawhdf5-migrate"
 version = "2.7.0"
 edition = "2024"
 rust-version.workspace = true
-description = "CLI to migrate SQLite agent memory databases to clawhdf5-agent stores"
+description = "CLI to migrate SQLite agent memory databases to HDF5 format"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
 readme = "README.md"
@@ -17,8 +17,10 @@ path = "src/main.rs"
 [dependencies]
 clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.7.0" }
 clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
+clawhdf5 = { path = "../clawhdf5", version = "2.7.0" }
 rusqlite = { version = "0.31", features = ["bundled"] }
 clap = { version = "4", features = ["derive"] }
+half = { workspace = true }

 [dev-dependencies]
 tempfile = { workspace = true }
@@ -3,12 +3,9 @@
 [![crates.io](https://img.shields.io/crates/v/clawhdf5-migrate.svg)](https://crates.io/crates/clawhdf5-migrate)
 [![docs.rs](https://img.shields.io/docsrs/clawhdf5-migrate)](https://docs.rs/clawhdf5-migrate)

-CLI tool to migrate SQLite agent memory databases (the ZeroClaw layout) to a
-[clawhdf5-agent](https://crates.io/crates/clawhdf5-agent) store.
+CLI tool to migrate SQLite agent memory databases to HDF5 format.

-The output is written through `clawhdf5-agent`'s own API, so it opens with
-`HDF5Memory::open` and is searchable immediately: memory records, sessions and
-the knowledge graph (entities and relations) are carried over.
+Converts existing SQLite-based agent memory stores (embeddings, text chunks, metadata) into the HDF5 format used by [clawhdf5-agent](https://crates.io/crates/clawhdf5-agent).

 ## Installation

@@ -19,19 +16,9 @@ cargo install clawhdf5-migrate
 ## Usage

 ```bash
-clawhdf5-migrate --sqlite agent.db --hdf5 agent.h5 --agent-id my-agent
+clawhdf5-migrate --input agent.db --output agent.h5
 ```

-Embeddings are stored as float16 (the library default for new stores); pass
-`--f32` for full precision. Every embedding must have the same dimension
-(the first row's, or `--embedding-dim`, which a source with no memory records
-requires); rows are never truncated, and the whole source is checked before an
-existing output store is replaced. `--incremental` adds only new rows to an
-existing store of the same dimension and carries over changes to rows'
-deleted flags, `--skip-deleted` leaves out tombstoned rows, and `--dry-run`
-only counts.
-See `clawhdf5-migrate --help` for every option.
-
 ## License

 MIT
@@ -0,0 +1,163 @@
+//! Read a migration HDF5 file back into the in-memory data model.
+//!
+//! Used to verify migrated content (real validation) and to merge new rows into
+//! an existing output (incremental migration). Mirrors the layout produced by
+//! [`crate::hdf5_writer`].
+
+use clawhdf5::reader::{File, Group};
+use clawhdf5_format::type_builders::AttrValue;
+
+use crate::sqlite_reader::{Entity, MemoryChunk, Relation, Session, SqliteData};
+
+type BoxErr = Box<dyn std::error::Error>;
+
+fn read_strings(group: &Group<'_>, name: &str) -> Result<Vec<String>, BoxErr> {
+    Ok(group.dataset(name)?.read_string()?)
+}
+
+fn read_i64s(group: &Group<'_>, name: &str) -> Result<Vec<i64>, BoxErr> {
+    Ok(group.dataset(name)?.read_i64()?)
+}
+
+fn read_f64s(group: &Group<'_>, name: &str) -> Result<Vec<f64>, BoxErr> {
+    Ok(group.dataset(name)?.read_f64()?)
+}
+
+/// Read the embeddings dataset as a flat `Vec<f32>` of `n * dim` values,
+/// handling both f32 and (lossy) f16 storage.
+fn read_embeddings_flat(group: &Group<'_>) -> Result<Vec<f32>, BoxErr> {
+    Ok(group.dataset("embeddings")?.read_f32()?)
+}
+
+/// Read a migration HDF5 file into a [`SqliteData`].
+pub fn read_hdf5(path: &str) -> Result<SqliteData, BoxErr> {
+    let file = File::open(path)?;
+
+    let embedding_dim = match file.root().attrs()?.get("embedding_dim") {
+        Some(AttrValue::I64(d)) => *d as usize,
+        _ => 0,
+    };
+
+    let chunks = read_chunks(&file, embedding_dim)?;
+    let sessions = read_sessions(&file)?;
+    let entities = read_entities(&file)?;
+    let relations = read_relations(&file)?;
+
+    Ok(SqliteData {
+        chunks,
+        sessions,
+        entities,
+        relations,
+        embedding_dim,
+        // Not a SQLite read — the caller (incremental migration) carries
+        // forward the current run's actual `source_path` from the fresh
+        // SQLite read instead of using this placeholder.
+        source_path: String::new(),
+    })
+}
+
+fn read_chunks(file: &File, dim: usize) -> Result<Vec<MemoryChunk>, BoxErr> {
+    let g = file.group("chunks")?;
+    let count = group_count(&g)?;
+    if count == 0 {
+        return Ok(Vec::new());
+    }
+    let ids = read_i64s(&g, "id")?;
+    let texts = read_strings(&g, "text")?;
+    let channels = read_strings(&g, "source_channel")?;
+    let timestamps = read_f64s(&g, "timestamp")?;
+    let session_ids = read_strings(&g, "session_id")?;
+    let tags = read_strings(&g, "tags")?;
+    let deleted = g.dataset("deleted")?.read_i32()?;
+    let emb_flat = read_embeddings_flat(&g)?;
+    let dim = dim.max(1);
+
+    let mut chunks = Vec::with_capacity(ids.len());
+    for (i, &id) in ids.iter().enumerate() {
+        let embedding = emb_flat
+            .get(i * dim..(i + 1) * dim)
+            .map(|s| s.to_vec())
+            .unwrap_or_default();
+        chunks.push(MemoryChunk {
+            id,
+            chunk: texts.get(i).cloned().unwrap_or_default(),
+            embedding,
+            source_channel: channels.get(i).cloned().unwrap_or_default(),
+            timestamp: timestamps.get(i).copied().unwrap_or(0.0),
+            session_id: session_ids.get(i).cloned().unwrap_or_default(),
+            tags: tags.get(i).cloned().unwrap_or_default(),
+            deleted: deleted.get(i).copied().unwrap_or(0),
+        });
+    }
+    Ok(chunks)
+}
+
+fn read_sessions(file: &File) -> Result<Vec<Session>, BoxErr> {
+    let g = file.group("sessions")?;
+    if group_count(&g)? == 0 {
+        return Ok(Vec::new());
+    }
+    let ids = read_strings(&g, "id")?;
+    let starts = read_i64s(&g, "start_idx")?;
+    let ends = read_i64s(&g, "end_idx")?;
+    let channels = read_strings(&g, "channel")?;
+    let timestamps = read_f64s(&g, "timestamp")?;
+    let summaries = read_strings(&g, "summary")?;
+    Ok((0..ids.len())
+        .map(|i| Session {
+            id: ids[i].clone(),
+            start_idx: starts.get(i).copied().unwrap_or(0),
+            end_idx: ends.get(i).copied().unwrap_or(0),
+            channel: channels.get(i).cloned().unwrap_or_default(),
+            timestamp: timestamps.get(i).copied().unwrap_or(0.0),
+            summary: summaries.get(i).cloned().unwrap_or_default(),
+        })
+        .collect())
+}
+
+fn read_entities(file: &File) -> Result<Vec<Entity>, BoxErr> {
+    let g = file.group("entities")?;
+    if group_count(&g)? == 0 {
+        return Ok(Vec::new());
+    }
+    let ids = read_i64s(&g, "id")?;
+    let names = read_strings(&g, "name")?;
+    let types = read_strings(&g, "type")?;
+    let emb_idxs = read_i64s(&g, "embedding_idx")?;
+    Ok((0..ids.len())
+        .map(|i| Entity {
+            id: ids[i],
+            name: names.get(i).cloned().unwrap_or_default(),
+            entity_type: types.get(i).cloned().unwrap_or_default(),
+            embedding_idx: emb_idxs.get(i).copied().unwrap_or(-1),
+        })
+        .collect())
+}
+
+fn read_relations(file: &File) -> Result<Vec<Relation>, BoxErr> {
+    let g = file.group("relations")?;
+    if group_count(&g)? == 0 {
+        return Ok(Vec::new());
+    }
+    let srcs = read_i64s(&g, "src")?;
+    let tgts = read_i64s(&g, "tgt")?;
+    let rels = read_strings(&g, "relation")?;
+    let weights = read_f64s(&g, "weight")?;
+    let timestamps = read_f64s(&g, "timestamp")?;
+    Ok((0..srcs.len())
+        .map(|i| Relation {
+            src: srcs[i],
+            tgt: tgts.get(i).copied().unwrap_or(0),
+            relation: rels.get(i).cloned().unwrap_or_default(),
+            weight: weights.get(i).copied().unwrap_or(1.0),
+            timestamp: timestamps.get(i).copied().unwrap_or(0.0),
+        })
+        .collect())
+}
+
+fn group_count(group: &Group<'_>) -> Result<u64, BoxErr> {
+    match group.attrs()?.get("count") {
+        Some(AttrValue::I64(n)) => Ok(*n as u64),
+        _ => Ok(0),
+    }
+}
@@ -0,0 +1,366 @@
+use clawhdf5::writer::FileBuilder;
+use clawhdf5_format::datatype::{CharacterSet, Datatype, StringPadding};
+use clawhdf5_format::type_builders::AttrValue;
+
+use crate::sqlite_reader::SqliteData;
+
+/// Options controlling HDF5 output.
+pub struct WriteOptions {
+    pub agent_id: String,
+    pub embedder: String,
+    pub compression: bool,
+    pub compression_level: u32,
+    pub float16: bool,
+}
+
+/// Write SQLite data to an HDF5 file.
+pub fn write_hdf5(
+    path: &str,
+    data: &SqliteData,
+    opts: &WriteOptions,
+) -> Result<(), Box<dyn std::error::Error>> {
+    let mut builder = FileBuilder::new();
+    let timestamp = iso8601_now();
+
+    // Root-level metadata attributes
+    builder.set_attr("agent_id", AttrValue::String(opts.agent_id.clone()));
+    builder.set_attr("embedder", AttrValue::String(opts.embedder.clone()));
+    builder.set_attr("embedding_dim", AttrValue::I64(data.embedding_dim as i64));
+    builder.set_attr("source", AttrValue::String("sqlite-migration".into()));
+    builder.set_attr("version", AttrValue::I64(1));
+    // Lineage: which SQLite database this output was migrated from and when,
+    // plus the migrator tool version — so a chain of `--incremental` runs
+    // still has an audit trail instead of every run overwriting the same
+    // static attributes (see research/03_provenance.md, INT-03).
+    builder.set_attr("source_path", AttrValue::String(data.source_path.clone()));
+    builder.set_attr("migrated_at", AttrValue::String(timestamp.clone()));
+    builder.set_attr(
+        "migrator_version",
+        AttrValue::String(env!("CARGO_PKG_VERSION").to_owned()),
+    );
+
+    write_chunks_group(&mut builder, data, opts, &timestamp);
+    write_sessions_group(&mut builder, data);
+    write_entities_group(&mut builder, data);
+    write_relations_group(&mut builder, data);
+
+    builder.write(path)?;
+    Ok(())
+}
+
+/// Current UTC time formatted as an ISO-8601 / RFC-3339 timestamp
+/// (`YYYY-MM-DDTHH:MM:SSZ`), with no external date/time dependency.
+fn iso8601_now() -> String {
+    let secs = std::time::SystemTime::now()
+        .duration_since(std::time::UNIX_EPOCH)
+        .unwrap_or_default()
+        .as_secs();
+    let days = (secs / 86_400) as i64;
+    let time_of_day = secs % 86_400;
+    let (h, m, s) = (
+        time_of_day / 3600,
+        (time_of_day % 3600) / 60,
+        time_of_day % 60,
+    );
+    let (y, mo, d) = civil_from_days(days);
+    format!("{y:04}-{mo:02}-{d:02}T{h:02}:{m:02}:{s:02}Z")
+}
+
+/// Days-since-epoch to (year, month, day), Howard Hinnant's `civil_from_days`
+/// algorithm (proleptic Gregorian calendar, valid for the full `i64` range).
+fn civil_from_days(z: i64) -> (i64, u32, u32) {
+    let z = z + 719_468;
+    let era = if z >= 0 { z } else { z - 146_096 } / 146_097;
+    let doe = (z - era * 146_097) as u64; // [0, 146096]
+    let yoe = (doe - doe / 1460 + doe / 36_524 - doe / 146_096) / 365; // [0, 399]
+    let y = yoe as i64 + era * 400;
+    let doy = doe - (365 * yoe + yoe / 4 - yoe / 100); // [0, 365]
+    let mp = (5 * doy + 2) / 153; // [0, 11]
+    let d = (doy - (153 * mp + 2) / 5 + 1) as u32; // [1, 31]
+    let m = (if mp < 10 { mp + 3 } else { mp - 9 }) as u32; // [1, 12]
+    let y = if m <= 2 { y + 1 } else { y };
+    (y, m, d)
+}
+
+/// Build a fixed-length string Datatype from the max byte length of the items.
+fn string_dtype(max_len: usize) -> Datatype {
+    Datatype::String {
+        size: max_len.max(1) as u32,
+        padding: StringPadding::NullPad,
+        charset: CharacterSet::Utf8,
+    }
+}
+
+/// Pack a slice of strings into null-padded raw bytes of uniform width.
+fn pack_strings(strings: &[String]) -> (Vec<u8>, usize) {
+    let max_len = strings.iter().map(|s| s.len()).max().unwrap_or(0).max(1);
+    let mut buf = vec![0u8; strings.len() * max_len];
+    for (i, s) in strings.iter().enumerate() {
+        let start = i * max_len;
+        let bytes = s.as_bytes();
+        let copy_len = bytes.len().min(max_len);
+        buf[start..start + copy_len].copy_from_slice(&bytes[..copy_len]);
+    }
+    (buf, max_len)
+}
+
+fn apply_compression(ds: &mut clawhdf5_format::type_builders::DatasetBuilder, opts: &WriteOptions) {
+    if opts.compression {
+        ds.with_deflate(opts.compression_level);
+        ds.with_shuffle();
+    }
+}
+
+fn write_chunks_group(
+    builder: &mut FileBuilder,
+    data: &SqliteData,
+    opts: &WriteOptions,
+    timestamp: &str,
+) {
+    let mut group = builder.create_group("chunks");
+    let n = data.chunks.len() as u64;
+
+    if n == 0 {
+        group.set_attr("count", AttrValue::I64(0));
+        builder.add_group(group.finish());
+        return;
+    }
+
+    group.set_attr("count", AttrValue::I64(n as i64));
+
+    // Source attribution attached directly to the content-bearing datasets
+    // (SHA-256 of the raw bytes + creator/timestamp/source), so the chunk
+    // text and embeddings each carry their own verifiable provenance
+    // (see clawhdf5_format::provenance / `Dataset::verify_provenance`).
+    let source_opt = if data.source_path.is_empty() {
+        None
+    } else {
+        Some(data.source_path.as_str())
+    };
+
+    // ids
+    let ids: Vec<i64> = data.chunks.iter().map(|c| c.id).collect();
+    group.create_dataset("id").with_i64_data(&ids);
+
+    // text
+    let texts: Vec<String> = data.chunks.iter().map(|c| c.chunk.clone()).collect();
+    let (text_raw, text_len) = pack_strings(&texts);
+    group
+        .create_dataset("text")
+        .with_compound_data(string_dtype(text_len), text_raw, n)
+        .with_provenance("clawhdf5-migrate", timestamp, source_opt);
+
+    // embeddings - flatten to [N, dim]
+    let dim = data.embedding_dim;
+    if opts.float16 {
+        let f16_data: Vec<u16> = data
+            .chunks
+            .iter()
+            .flat_map(|c| {
+                c.embedding
+                    .iter()
+                    .map(|&v| half::f16::from_f32(v).to_bits())
+            })
+            .collect();
+        let raw: Vec<u8> = f16_data.iter().flat_map(|v| v.to_le_bytes()).collect();
+        let f16_dtype = Datatype::FloatingPoint {
+            size: 2,
+            byte_order: clawhdf5_format::datatype::DatatypeByteOrder::LittleEndian,
+            bit_offset: 0,
+            bit_precision: 16,
+            exponent_location: 10,
+            exponent_size: 5,
+            mantissa_location: 0,
+            mantissa_size: 10,
+            exponent_bias: 15,
+        };
+        let ds = group
+            .create_dataset("embeddings")
+            .with_compound_data(f16_dtype, raw, n)
+            .with_shape(&[n, dim as u64])
+            .with_provenance("clawhdf5-migrate", timestamp, source_opt);
+        apply_compression(ds, opts);
+    } else {
+        let flat: Vec<f32> = data
+            .chunks
+            .iter()
+            .flat_map(|c| c.embedding.iter().copied())
+            .collect();
+        let ds = group
+            .create_dataset("embeddings")
+            .with_f32_data(&flat)
+            .with_shape(&[n, dim as u64])
+            .with_provenance("clawhdf5-migrate", timestamp, source_opt);
+        apply_compression(ds, opts);
+    }
+
+    // source_channel
+    let channels: Vec<String> = data
+        .chunks
+        .iter()
+        .map(|c| c.source_channel.clone())
+        .collect();
+    let (ch_raw, ch_len) = pack_strings(&channels);
+    group
+        .create_dataset("source_channel")
+        .with_compound_data(string_dtype(ch_len), ch_raw, n);
+
+    // timestamp
+    let timestamps: Vec<f64> = data.chunks.iter().map(|c| c.timestamp).collect();
+    group.create_dataset("timestamp").with_f64_data(&timestamps);
+
+    // session_id
+    let sess_ids: Vec<String> = data.chunks.iter().map(|c| c.session_id.clone()).collect();
+    let (sid_raw, sid_len) = pack_strings(&sess_ids);
+    group
+        .create_dataset("session_id")
+        .with_compound_data(string_dtype(sid_len), sid_raw, n);
+
+    // tags
+    let tags: Vec<String> = data.chunks.iter().map(|c| c.tags.clone()).collect();
+    let (tag_raw, tag_len) = pack_strings(&tags);
+    group
+        .create_dataset("tags")
+        .with_compound_data(string_dtype(tag_len), tag_raw, n);
+
+    // deleted
+    let deleted: Vec<i32> = data.chunks.iter().map(|c| c.deleted).collect();
+    group.create_dataset("deleted").with_i32_data(&deleted);
+
+    builder.add_group(group.finish());
+}
+
+fn write_sessions_group(builder: &mut FileBuilder, data: &SqliteData) {
+    let mut group = builder.create_group("sessions");
+    let n = data.sessions.len() as u64;
+    group.set_attr("count", AttrValue::I64(n as i64));
+
+    if n == 0 {
+        builder.add_group(group.finish());
+        return;
+    }
+
+    let ids: Vec<String> = data.sessions.iter().map(|s| s.id.clone()).collect();
+    let (id_raw, id_len) = pack_strings(&ids);
+    group
+        .create_dataset("id")
+        .with_compound_data(string_dtype(id_len), id_raw, n);
+
+    let start_idxs: Vec<i64> = data.sessions.iter().map(|s| s.start_idx).collect();
+    group.create_dataset("start_idx").with_i64_data(&start_idxs);
+
+    let end_idxs: Vec<i64> = data.sessions.iter().map(|s| s.end_idx).collect();
+    group.create_dataset("end_idx").with_i64_data(&end_idxs);
+
+    let channels: Vec<String> = data.sessions.iter().map(|s| s.channel.clone()).collect();
+    let (ch_raw, ch_len) = pack_strings(&channels);
+    group
+        .create_dataset("channel")
+        .with_compound_data(string_dtype(ch_len), ch_raw, n);
+
+    let timestamps: Vec<f64> = data.sessions.iter().map(|s| s.timestamp).collect();
+    group.create_dataset("timestamp").with_f64_data(&timestamps);
+
+    let summaries: Vec<String> = data.sessions.iter().map(|s| s.summary.clone()).collect();
+    let (sum_raw, sum_len) = pack_strings(&summaries);
+    group
+        .create_dataset("summary")
+        .with_compound_data(string_dtype(sum_len), sum_raw, n);
+
+    builder.add_group(group.finish());
+}
+
+fn write_entities_group(builder: &mut FileBuilder, data: &SqliteData) {
+    let mut group = builder.create_group("entities");
+    let n = data.entities.len() as u64;
+    group.set_attr("count", AttrValue::I64(n as i64));
+
+    if n == 0 {
+        builder.add_group(group.finish());
+        return;
+    }
+
+    let ids: Vec<i64> = data.entities.iter().map(|e| e.id).collect();
+    group.create_dataset("id").with_i64_data(&ids);
+
+    let names: Vec<String> = data.entities.iter().map(|e| e.name.clone()).collect();
+    let (name_raw, name_len) = pack_strings(&names);
+    group
+        .create_dataset("name")
+        .with_compound_data(string_dtype(name_len), name_raw, n);
+
+    let types: Vec<String> = data
+        .entities
+        .iter()
+        .map(|e| e.entity_type.clone())
+        .collect();
+    let (type_raw, type_len) = pack_strings(&types);
+    group
+        .create_dataset("type")
+        .with_compound_data(string_dtype(type_len), type_raw, n);
+
+    let emb_idxs: Vec<i64> = data.entities.iter().map(|e| e.embedding_idx).collect();
+    group
+        .create_dataset("embedding_idx")
+        .with_i64_data(&emb_idxs);
+
+    builder.add_group(group.finish());
+}
+
+fn write_relations_group(builder: &mut FileBuilder, data: &SqliteData) {
+    let mut group = builder.create_group("relations");
+    let n = data.relations.len() as u64;
+    group.set_attr("count", AttrValue::I64(n as i64));
+
+    if n == 0 {
+        builder.add_group(group.finish());
+        return;
+    }
+
+    let srcs: Vec<i64> = data.relations.iter().map(|r| r.src).collect();
+    group.create_dataset("src").with_i64_data(&srcs);
+
+    let tgts: Vec<i64> = data.relations.iter().map(|r| r.tgt).collect();
+    group.create_dataset("tgt").with_i64_data(&tgts);
+
+    let rels: Vec<String> = data.relations.iter().map(|r| r.relation.clone()).collect();
+    let (rel_raw, rel_len) = pack_strings(&rels);
+    group
+        .create_dataset("relation")
+        .with_compound_data(string_dtype(rel_len), rel_raw, n);
+
+    let weights: Vec<f64> = data.relations.iter().map(|r| r.weight).collect();
+    group.create_dataset("weight").with_f64_data(&weights);
+
+    let timestamps: Vec<f64> = data.relations.iter().map(|r| r.timestamp).collect();
+    group.create_dataset("timestamp").with_f64_data(&timestamps);
+
+    builder.add_group(group.finish());
+}
+
+#[cfg(test)]
+mod time_tests {
+    use super::civil_from_days;
+
+    #[test]
+    fn epoch_day_zero_is_1970_01_01() {
+        assert_eq!(civil_from_days(0), (1970, 1, 1));
+    }
+
+    #[test]
+    fn known_dates_roundtrip() {
+        // 2026-08-16 is 20,681 days after 1970-01-01.
+        assert_eq!(civil_from_days(20_681), (2026, 8, 16));
+        // 2000-02-29 (leap day itself) and 2000-03-01 (the day after).
+        assert_eq!(civil_from_days(11_016), (2000, 2, 29));
+        assert_eq!(civil_from_days(11_017), (2000, 3, 1));
+    }
+
+    #[test]
+    fn iso8601_now_has_expected_shape() {
+        let ts = super::iso8601_now();
+        assert_eq!(ts.len(), "2026-08-16T00:00:00Z".len());
+        assert!(ts.starts_with("20")); // sanity: 21st-century year
+        assert!(ts.ends_with('Z'));
+    }
+}
@@ -50,8 +50,12 @@ pub struct SqliteData {
    pub sessions: Vec<Session>,
    pub entities: Vec<Entity>,
    pub relations: Vec<Relation>,
-    /// `--embedding-dim`, or the first row's; 0 when neither exists.
    pub embedding_dim: usize,
+    /// Filesystem path of the SQLite database this data was read from, for
+    /// provenance attribution on the HDF5 output. Empty when the data did
+    /// not come directly from a SQLite read (e.g. re-read of a prior HDF5
+    /// migration output for an incremental merge).
+    pub source_path: String,
 }

 /// A table name plus the ordered column names the reader maps by position.
@@ -163,13 +167,11 @@ pub fn read_counts(
    })
 }

-/// Auto-detect embedding dimension from the BLOB size of the first chunk (in
-/// id order, deleted or not).
+/// Auto-detect embedding dimension from the first chunk's BLOB size.
 fn detect_embedding_dim(conn: &Connection, config: &SchemaConfig) -> SqlResult<Option<usize>> {
    let emb_col = config.chunks.columns.get(2).copied().unwrap_or("embedding");
-    let id_col = config.chunks.columns.first().copied().unwrap_or("id");
    let mut stmt = conn.prepare(&format!(
-        "SELECT {emb_col} FROM {} ORDER BY {id_col} LIMIT 1",
+        "SELECT {emb_col} FROM {} LIMIT 1",
        config.chunks.table
    ))?;
    let mut rows = stmt.query([])?;
@@ -193,16 +195,24 @@ fn blob_to_f32(blob: &[u8]) -> Vec<f32> {
 /// Read all data from a ZeroClaw SQLite database.
 ///
 /// If `skip_deleted` is true, rows with `deleted=1` are excluded from chunks.
-/// If `embedding_dim` is `None`, auto-detect from the first row (0 when there
-/// are no rows). Embeddings are returned at their full stored length whatever
-/// the dimension: checking that every row matches it is the writer's job
-/// (`store_writer::write_store`), so a mismatch is an error, not silent
-/// truncation.
+/// If `embedding_dim` is `None`, auto-detect from the first row.
 pub fn read_sqlite(
    path: &str,
    skip_deleted: bool,
    embedding_dim: Option<usize>,
    config: &SchemaConfig,
+) -> Result<SqliteData, Box<dyn std::error::Error>> {
+    read_sqlite_filtered(path, skip_deleted, embedding_dim, config, 0)
+}
+
+/// Like [`read_sqlite`] but only reads chunks whose id is greater than
+/// `min_chunk_id` (0 = all). Used for incremental migration.
+pub fn read_sqlite_filtered(
+    path: &str,
+    skip_deleted: bool,
+    embedding_dim: Option<usize>,
+    config: &SchemaConfig,
+    min_chunk_id: i64,
 ) -> Result<SqliteData, Box<dyn std::error::Error>> {
    let conn = Connection::open(path)?;

@@ -211,7 +221,7 @@ pub fn read_sqlite(
        None => detect_embedding_dim(&conn, config)?.unwrap_or(0),
    };

-    let chunks = read_chunks(&conn, skip_deleted, config)?;
+    let chunks = read_chunks(&conn, skip_deleted, dim, config, min_chunk_id)?;
    let sessions = read_sessions(&conn, config)?;
    let entities = read_entities(&conn, config)?;
    let relations = read_relations(&conn, config)?;
@@ -222,43 +232,42 @@ pub fn read_sqlite(
        entities,
        relations,
        embedding_dim: dim,
+        source_path: path.to_owned(),
    })
 }

 fn read_chunks(
    conn: &Connection,
    skip_deleted: bool,
+    expected_dim: usize,
    config: &SchemaConfig,
+    min_chunk_id: i64,
 ) -> SqlResult<Vec<MemoryChunk>> {
    let id_col = config.chunks.columns.first().copied().unwrap_or("id");
    let deleted_col = config.chunks.columns.get(7).copied().unwrap_or("deleted");
-    let mut where_clause = String::new();
+    let mut conds = Vec::new();
    if skip_deleted {
-        where_clause = format!(" WHERE {deleted_col} = 0");
+        conds.push(format!("{deleted_col} = 0"));
    }
-    // In id order, so the store's records follow the source's order.
-    where_clause.push_str(&format!(" ORDER BY {id_col}"));
+    if min_chunk_id > 0 {
+        conds.push(format!("{id_col} > {min_chunk_id}"));
+    }
+    let where_clause = if conds.is_empty() {
+        String::new()
+    } else {
+        format!(" WHERE {}", conds.join(" AND "))
+    };
    let sql = config.chunks.select(&where_clause);

    let mut stmt = conn.prepare(&sql)?;
    let rows = stmt.query_map([], |row| {
        let blob: Vec<u8> = row.get(2)?;
-        if !blob.len().is_multiple_of(4) {
-            let id: i64 = row.get(0)?;
-            return Err(rusqlite::Error::FromSqlConversionFailure(
-                2,
-                rusqlite::types::Type::Blob,
-                format!(
-                    "chunk id {id}: embedding BLOB is {} bytes, not a whole number of \
-                     little-endian f32 values",
-                    blob.len()
-                )
-                .into(),
-            ));
+        let mut embedding = blob_to_f32(&blob);
+
+        // Validate/truncate to expected dimension
+        if expected_dim > 0 {
+            embedding.truncate(expected_dim);
        }
-        // Read at full length: rows of the wrong dimension are rejected by
-        // the writer, never truncated to fit.
-        let embedding = blob_to_f32(&blob);

        Ok(MemoryChunk {
            id: row.get(0)?,
@@ -1,407 +0,0 @@
-//! Write migrated SQLite data into a clawhdf5-agent store.
-//!
-//! Everything goes through `clawhdf5-agent`'s own API — `HDF5Memory::create`
-//! (or `open` for `--incremental`), `save_batch`, `delete_batch`, the session
-//! cache and the knowledge graph — so the result is an ordinary agent store
-//! that `HDF5Memory::open` accepts, not a second hand-built copy of its schema.
-
-use std::collections::{HashMap, HashSet};
-use std::path::Path;
-
-use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
-use clawhdf5_format::float16::round_to_f16;
-
-use crate::sqlite_reader::{MemoryChunk, SqliteData};
-
-type BoxErr = Box<dyn std::error::Error>;
-
-/// SQLite timestamps are Unix seconds; the agent's session and relation
-/// timestamps are Unix microseconds (memory records stay in seconds).
-pub const US_PER_SEC: f64 = 1_000_000.0;
-
-/// Options controlling the output store.
-#[derive(Debug, Clone)]
-pub struct WriteOptions {
-    pub agent_id: String,
-    pub embedder: String,
-    pub compression: bool,
-    pub compression_level: u32,
-    /// Store full-precision `f32` embeddings instead of the library default
-    /// (half precision). Only applies to a newly created store: an existing
-    /// store keeps the precision it was created with.
-    pub f32: bool,
-    /// Add to the store at the output path if there is one, instead of
-    /// replacing it.
-    pub incremental: bool,
-    /// Leave out deleted source rows that are not in the store. (A deleted
-    /// row that matches an active store record still tombstones it, so pass
-    /// deleted rows in `data` for an incremental run.)
-    pub skip_deleted: bool,
-}
-
-/// What the migration wrote, and where each source row went, so validation
-/// can compare the store with the source row by row.
-#[derive(Debug, Default)]
-pub struct Migration {
-    /// Whether the output store existed and was added to (`--incremental`).
-    pub appended_to_existing: bool,
-    /// The store's embedding precision.
-    pub float16: bool,
-    pub embedding_dim: usize,
-    /// Records in the store after the migration (including tombstones).
-    pub store_count: usize,
-    /// `(store index, source chunk index)` of every record written.
-    pub records: Vec<(usize, usize)>,
-    /// Source chunks already in the store (incremental), not written again.
-    pub chunks_present: usize,
-    /// `(store index, source chunk index)` of records that were active in
-    /// the store but whose source row is now deleted (incremental): they were
-    /// tombstoned by this run.
-    pub deleted_in_store: Vec<(usize, usize)>,
-    /// Source rows that were deleted in the store but are active in the
-    /// source (incremental): the agent has no un-delete, so each was written
-    /// again as a new record (counted in `records` too).
-    pub restored: usize,
-    /// Deleted source rows left out because of `skip_deleted`.
-    pub deleted_skipped: usize,
-    /// `(store session index, source session index)` of each session written.
-    pub sessions: Vec<(usize, usize)>,
-    pub sessions_present: usize,
-    /// `(store entity id, source entity index)` of each entity written.
-    pub entities: Vec<(u64, usize)>,
-    pub entities_present: usize,
-    /// SQLite entity id -> store entity id, for every source entity.
-    pub entity_ids: HashMap<i64, u64>,
-    /// `(store relation index, source relation index)` of each relation written.
-    pub relations: Vec<(usize, usize)>,
-    pub relations_present: usize,
-    /// Source relations naming an entity id that is not in the entities
-    /// table; the knowledge graph cannot hold them, so they are skipped.
-    pub dangling_relations: Vec<usize>,
-    /// Messages of the write-anomaly alerts the agent raised while importing
-    /// (informational; they never block a save — a bulk import typically
-    /// trips the write-rate check).
-    pub anomaly_alerts: Vec<String>,
-}
-
-/// Identity of a memory record for incremental de-duplication: every field
-/// the agent stores except the embedding (whose stored form depends on the
-/// store's precision).
-type RecordKey = (String, String, String, String, u64);
-
-fn record_key(
-    chunk: &str,
-    source_channel: &str,
-    session_id: &str,
-    tags: &str,
-    ts: f64,
-) -> RecordKey {
-    (
-        chunk.to_owned(),
-        source_channel.to_owned(),
-        session_id.to_owned(),
-        tags.to_owned(),
-        ts.to_bits(),
-    )
-}
-
-/// Reject rows the agent would otherwise store differently from the source,
-/// or not at all: an embedding of a different length from the store's
-/// dimension (the agent pads/truncates silently), an empty embedding, or, in
-/// a float16 store, a value beyond the half-precision range.
-///
-/// Every source row is checked, including ones that end up not being written
-/// (already in the store, or deleted and skipped): the source must be
-/// consistent as a whole, and the check runs before the store is touched.
-fn check_chunks(chunks: &[MemoryChunk], dim: usize, float16: bool) -> Result<(), BoxErr> {
-    for c in chunks {
-        if c.embedding.is_empty() {
-            return Err(format!(
-                "chunk id {}: the embedding is empty; an agent store needs an embedding \
-                 for every record",
-                c.id
-            )
-            .into());
-        }
-        if c.embedding.len() != dim {
-            return Err(format!(
-                "chunk id {}: embedding has {} values, expected {dim}; every row must have \
-                 the store's dimension (detected from the first row unless --embedding-dim \
-                 is given), and rows are never truncated or padded to fit",
-                c.id,
-                c.embedding.len()
-            )
-            .into());
-        }
-        if float16
-            && let Some((k, v)) = c
-                .embedding
-                .iter()
-                .enumerate()
-                .find(|&(_, &v)| v.is_finite() && round_to_f16(v).is_infinite())
-        {
-            return Err(format!(
-                "chunk id {}: embedding[{k}] = {v} is outside the half-precision range \
-                 (±65504) of a float16 store; migrate with --f32",
-                c.id
-            )
-            .into());
-        }
-    }
-    Ok(())
-}
-
-/// Migrate `data` into the agent store at `path`.
-///
-/// Without `opts.incremental` (or when nothing exists at `path`) a new store
-/// is created, replacing any file there — but only once every source row has
-/// passed [`check_chunks`], so a source that cannot be migrated leaves an
-/// existing store untouched. With it, the existing store is opened and only
-/// source rows it does not already hold are added: memory records are
-/// matched on their content, sessions on their id, entities on name and
-/// type, relations on (source, target, relation). A matched record then
-/// takes the source row's deleted flag: see [`Migration::deleted_in_store`]
-/// and [`Migration::restored`].
-pub fn write_store(
-    path: &Path,
-    data: &SqliteData,
-    opts: &WriteOptions,
-) -> Result<Migration, BoxErr> {
-    let existing = opts.incremental && path.exists();
-    let mut mem = if existing {
-        // `open` does not modify the store beyond what the agent itself does
-        // on open; the checks below run before anything is written.
-        let mem = HDF5Memory::open(path)?;
-        let dim = mem.config().embedding_dim;
-        // `data.embedding_dim` is 0 only for a source with no records and no
-        // --embedding-dim, which has no dimension to disagree with.
-        if data.embedding_dim != 0 && dim != data.embedding_dim {
-            let hint = if dim == 0 {
-                " (a store created from a source with no memory records; re-create it \
-                 with --embedding-dim)"
-            } else {
-                ""
-            };
-            return Err(format!(
-                "the store at {} has embedding_dim {dim}{hint}, the source {}; \
-                 embeddings of a different dimension cannot be added to it",
-                path.display(),
-                data.embedding_dim
-            )
-            .into());
-        }
-        check_chunks(&data.chunks, dim, mem.config().float16)?;
-        mem
-    } else {
-        // (With records, a dimension of 0 means an empty first embedding,
-        // which `check_chunks` reports more precisely.)
-        if data.embedding_dim == 0 && data.chunks.is_empty() {
-            return Err(
-                "the source has no memory records to detect the embedding dimension \
-                        from; pass --embedding-dim (the dimension of the agent's embedder), \
-                        or the store could never hold a record"
-                    .into(),
-            );
-        }
-        let mut config = MemoryConfig::new(path.to_path_buf(), &opts.agent_id, data.embedding_dim);
-        config.embedder = opts.embedder.clone();
-        config.compression = opts.compression;
-        config.compression_level = opts.compression_level;
-        // Only ever switch the library default off (as `clawhdf5-cli create`).
-        if opts.f32 {
-            config.float16 = false;
-        }
-        // Before `create`, which replaces whatever is at `path`.
-        check_chunks(&data.chunks, config.embedding_dim, config.float16)?;
-        HDF5Memory::create(config)?
-    };
-    let float16 = mem.config().float16;
-    let dim = mem.config().embedding_dim;
-
-    let mut m = Migration {
-        appended_to_existing: existing,
-        float16,
-        embedding_dim: dim,
-        ..Migration::default()
-    };
-
-    // ---- Memory records --------------------------------------------------
-    // Store indices of every record the store already holds, by content, so
-    // a source row that appears twice is only treated as present as often
-    // as the store has it.
-    let mut present: HashMap<RecordKey, Vec<usize>> = HashMap::new();
-    if existing {
-        let c = &mem.cache;
-        for i in 0..c.len() {
-            let key = record_key(
-                &c.chunks[i],
-                &c.source_channels[i],
-                &c.session_ids[i],
-                &c.tags[i],
-                c.timestamps[i],
-            );
-            present.entry(key).or_default().push(i);
-        }
-    }
-    let key_of = |c: &MemoryChunk| {
-        record_key(
-            &c.chunk,
-            &c.source_channel,
-            &c.session_id,
-            &c.tags,
-            c.timestamp,
-        )
-    };
-    let tombstoned = |idx: usize| mem.cache.tombstones[idx] != 0;
-    // Pass 1: a store record in the same deleted state as the source row.
-    let mut unmatched: Vec<usize> = Vec::new();
-    for (i, c) in data.chunks.iter().enumerate() {
-        let src_deleted = c.deleted != 0;
-        let hit = present.get_mut(&key_of(c)).and_then(|idxs| {
-            let at = idxs.iter().position(|&x| tombstoned(x) == src_deleted)?;
-            Some(idxs.remove(at))
-        });
-        match hit {
-            Some(_) => m.chunks_present += 1,
-            None => unmatched.push(i),
-        }
-    }
-    // Pass 2: a store record whose deleted state differs — the source row
-    // was deleted or restored since the last migration. The source wins.
-    let mut new_chunks: Vec<usize> = Vec::with_capacity(unmatched.len());
-    let mut delete_in_store: Vec<usize> = Vec::new();
-    for i in unmatched {
-        let c = &data.chunks[i];
-        let hit = present
-            .get_mut(&key_of(c))
-            .and_then(|idxs| (!idxs.is_empty()).then(|| idxs.remove(0)));
-        match hit {
-            // Active in the store, deleted in the source: tombstone it.
-            Some(idx) if c.deleted != 0 => {
-                m.deleted_in_store.push((idx, i));
-                delete_in_store.push(idx);
-            }
-            // Deleted in the store, active in the source. The agent has no
-            // un-delete, so the row is written again as a new active record
-            // (the tombstone stays until the store is compacted).
-            Some(_) => {
-                m.restored += 1;
-                new_chunks.push(i);
-            }
-            None if c.deleted != 0 && opts.skip_deleted => m.deleted_skipped += 1,
-            None => new_chunks.push(i),
-        }
-    }
-    new_chunks.sort_unstable();
-    let to_write: Vec<&MemoryChunk> = new_chunks.iter().map(|&i| &data.chunks[i]).collect();
-
-    // ---- Sessions (in the cache; persisted by the save_batch checkpoint) ---
-    let known_sessions: HashSet<String> = mem
-        .sessions()
-        .entries
-        .iter()
-        .map(|e| e.id.clone())
-        .collect();
-    for (i, s) in data.sessions.iter().enumerate() {
-        if known_sessions.contains(&s.id) {
-            m.sessions_present += 1;
-            continue;
-        }
-        let sessions = mem.sessions_mut();
-        let at = sessions.len();
-        sessions.add_at(
-            &s.id,
-            s.start_idx.max(0) as usize,
-            s.end_idx.max(0) as usize,
-            &s.channel,
-            &s.summary,
-            s.timestamp * US_PER_SEC,
-        );
-        m.sessions.push((at, i));
-    }
-
-    // ---- Knowledge graph -------------------------------------------------
-    let kg = mem.knowledge_mut();
-    // Matched only against what the store held before this run: the source
-    // itself is copied as it is, duplicates included.
-    let by_name_type: HashMap<(String, String), u64> = kg
-        .entities
-        .iter()
-        .map(|e| ((e.name.clone(), e.entity_type.clone()), e.id))
-        .collect();
-    for (i, e) in data.entities.iter().enumerate() {
-        let key = (e.name.clone(), e.entity_type.clone());
-        let id = match by_name_type.get(&key) {
-            Some(&id) => {
-                m.entities_present += 1;
-                id
-            }
-            None => {
-                let id = kg.add_entity(&e.name, &e.entity_type, e.embedding_idx);
-                m.entities.push((id, i));
-                id
-            }
-        };
-        m.entity_ids.insert(e.id, id);
-    }
-    let known_relations: HashSet<(u64, u64, String)> = kg
-        .relations
-        .iter()
-        .map(|r| (r.src, r.tgt, r.relation.clone()))
-        .collect();
-    for (i, r) in data.relations.iter().enumerate() {
-        let (Some(&src), Some(&tgt)) = (m.entity_ids.get(&r.src), m.entity_ids.get(&r.tgt)) else {
-            m.dangling_relations.push(i);
-            continue;
-        };
-        if known_relations.contains(&(src, tgt, r.relation.clone())) {
-            m.relations_present += 1;
-            continue;
-        }
-        let at = kg.relations.len();
-        kg.add_relation(src, tgt, &r.relation, r.weight as f32);
-        kg.relations[at].ts = r.timestamp * US_PER_SEC;
-        m.relations.push((at, i));
-    }
-
-    // ---- Write: one checkpoint for records, sessions and graph -----------
-    let entries: Vec<MemoryEntry> = to_write
-        .iter()
-        .map(|c| MemoryEntry {
-            chunk: c.chunk.clone(),
-            embedding: c.embedding.clone(),
-            source_channel: c.source_channel.clone(),
-            timestamp: c.timestamp,
-            session_id: c.session_id.clone(),
-            tags: c.tags.clone(),
-        })
-        .collect();
-    let indices = mem.save_batch(entries)?;
-    m.records = indices
-        .iter()
-        .copied()
-        .zip(new_chunks.iter().copied())
-        .collect();
-
-    // Rows deleted in the source stay deleted: tombstones, as the agent's own
-    // `delete` leaves them (not compacted away).
-    // Records matched in the store whose source row has since been deleted
-    // are tombstoned too.
-    let tombstones: Vec<usize> = m
-        .records
-        .iter()
-        .filter(|&&(_, src)| data.chunks[src].deleted != 0)
-        .map(|&(idx, _)| idx)
-        .chain(delete_in_store)
-        .collect();
-    mem.delete_batch(&tombstones)?;
-
-    m.anomaly_alerts = mem
-        .take_anomaly_alerts()
-        .into_iter()
-        .map(|a| a.message)
-        .collect();
-    m.store_count = mem.count();
-    drop(mem); // release the single-writer lock before anyone re-opens it
-    Ok(m)
-}
@@ -1,266 +1,192 @@
-//! Validate a migration by reading the store back the way an agent would:
-//! through `HDF5Memory::open_read_only`, comparing what it loads with the
-//! SQLite source, and running a search for a migrated record.
-
-use std::path::Path;
-
-use clawhdf5_agent::{AgentMemory, HDF5Memory, SearchOptions};
-use clawhdf5_format::float16::round_to_f16;
+use clawhdf5::reader::File as Hdf5File;
+use clawhdf5_format::provenance::VerifyResult;

+use crate::hdf5_reader::read_hdf5;
 use crate::sqlite_reader::SqliteData;
-use crate::store_writer::{Migration, US_PER_SEC};

 type BoxErr = Box<dyn std::error::Error>;

 /// Summary of a migration validation.
 #[derive(Debug)]
 pub struct ValidationSummary {
-    /// Records in the store (including tombstones).
-    pub count: usize,
-    /// Records in the store that are not deleted.
-    pub active: usize,
-    pub sessions: usize,
-    pub entities: usize,
-    pub relations: usize,
-    pub embedding_dim: usize,
-    pub float16: bool,
-    /// Rows whose full content was compared against the source.
+    pub chunks: u64,
+    pub sessions: u64,
+    pub entities: u64,
+    pub relations: u64,
+    pub embedding_dim: u64,
+    /// Number of rows whose full content was compared against the source.
    pub rows_checked: u64,
-    /// Whether a search for a migrated record found it (`false` when there
-    /// was no active migrated record with an embedding to search for).
-    pub search_checked: bool,
+    /// Whether the `chunks/text` and `chunks/embeddings` SHINES provenance
+    /// hashes (written via [`crate::hdf5_writer`]) were both present and
+    /// matched their recomputed SHA-256 on read-back. `false` when either
+    /// dataset has no provenance metadata (e.g. an older output file) or
+    /// there are zero chunks to check.
+    pub provenance_verified: bool,
 }

-/// Validate the store at `path` against the source rows `migration` wrote.
+/// Validate a migrated HDF5 file against the source data.
 ///
-/// Counts and the session / entity / relation rows are always checked in
-/// full. Memory records are content-checked on a representative sample, or
-/// all of them with `full`. Embeddings must match exactly: the source values
-/// themselves in an `f32` store, their [`round_to_f16`] in a `float16` one.
-pub fn validate_store(
-    path: &Path,
+/// Reads the written file back and compares actual content — chunk text,
+/// embeddings, and every session/entity/relation field — to the source, not
+/// just the row counts. When `full` is false a representative sample of chunk
+/// rows is content-checked (counts and all other groups are always checked in
+/// full); when `full` is true every chunk row is compared too. `float16` widens
+/// the embedding tolerance to allow for half-precision quantization.
+pub fn validate_hdf5(
+    path: &str,
    source: &SqliteData,
-    migration: &Migration,
    full: bool,
+    float16: bool,
 ) -> Result<ValidationSummary, BoxErr> {
-    let mut mem = HDF5Memory::open_read_only(path)?;
-    let float16 = mem.config().float16;
-    let dim = mem.config().embedding_dim;
+    let got = read_hdf5(path)?;
+    let provenance_verified = verify_chunk_provenance(path)?;

    // ---- Counts ----
-    check_count("record", mem.count(), migration.store_count)?;
-    if float16 != migration.float16 {
+    check_count("chunk", got.chunks.len(), source.chunks.len())?;
+    check_count("session", got.sessions.len(), source.sessions.len())?;
+    check_count("entity", got.entities.len(), source.entities.len())?;
+    check_count("relation", got.relations.len(), source.relations.len())?;
+    if got.embedding_dim != source.embedding_dim {
        return Err(format!(
-            "float16 mismatch: store {float16}, expected {}",
-            migration.float16
+            "embedding_dim mismatch: HDF5 has {}, source has {}",
+            got.embedding_dim, source.embedding_dim
        )
        .into());
    }
-    if dim != migration.embedding_dim {
-        return Err(format!(
-            "embedding_dim mismatch: store has {dim}, expected {}",
-            migration.embedding_dim
-        )
-        .into());
-    }
-    if !migration.appended_to_existing {
-        check_count("record", mem.count(), migration.records.len())?;
-        check_count("session", mem.sessions().len(), migration.sessions.len())?;
-        check_count(
-            "entity",
-            mem.knowledge().entities.len(),
-            migration.entities.len(),
-        )?;
-        check_count(
-            "relation",
-            mem.knowledge().relations.len(),
-            migration.relations.len(),
-        )?;
-    }

-    // ---- Memory records (sampled or full) ----
+    // ---- Chunk content (sampled or full) ----
+    let (emb_abs, emb_rel) = if float16 { (1e-2, 1e-2) } else { (1e-4, 0.0) };
    let mut rows_checked = 0u64;
-    let expected_value = |v: f32| if float16 { round_to_f16(v) } else { v };
-    for k in sample_indices(migration.records.len(), full) {
-        let (idx, src) = migration.records[k];
-        let s = &source.chunks[src];
-        let c = &mem.cache;
-        if idx >= c.len() {
-            return Err(
-                format!("record {idx} (chunk id {}) is missing from the store", s.id).into(),
-            );
+    for i in sample_indices(source.chunks.len(), full) {
+        let (s, g) = (&source.chunks[i], &got.chunks[i]);
+        if s.id != g.id {
+            return Err(field_err("chunk", i, "id", s.id, g.id));
        }
-        let id = s.id;
-        if c.chunks[idx] != s.chunk {
+        if s.chunk != g.chunk {
            return Err(format!(
-                "record {idx} (chunk id {id}) text mismatch: source {:?}, store {:?}",
+                "chunk[{i}].text mismatch: source {:?}, HDF5 {:?}",
                truncate(&s.chunk),
-                truncate(&c.chunks[idx])
+                truncate(&g.chunk)
            )
            .into());
        }
-        if c.source_channels[idx] != s.source_channel
-            || c.session_ids[idx] != s.session_id
-            || c.tags[idx] != s.tags
+        if s.session_id != g.session_id || s.source_channel != g.source_channel || s.tags != g.tags
        {
-            return Err(format!("record {idx} (chunk id {id}) string field mismatch").into());
+            return Err(format!("chunk[{i}] string field mismatch").into());
        }
-        if c.timestamps[idx].to_bits() != s.timestamp.to_bits() {
-            return Err(format!(
-                "record {idx} (chunk id {id}) timestamp mismatch: source {}, store {}",
-                s.timestamp, c.timestamps[idx]
-            )
-            .into());
+        if s.deleted != g.deleted {
+            return Err(field_err("chunk", i, "deleted", s.deleted, g.deleted));
        }
-        let deleted = c.tombstones[idx] != 0;
-        if deleted != (s.deleted != 0) {
+        if s.embedding.len() != g.embedding.len() {
            return Err(format!(
-                "record {idx} (chunk id {id}) deleted mismatch: source {}, store {deleted}",
-                s.deleted != 0
-            )
-            .into());
-        }
-        let got = c.embeddings.get(idx).unwrap_or(&[]);
-        if got.len() != s.embedding.len() {
-            return Err(format!(
-                "record {idx} (chunk id {id}) embedding length mismatch: source {}, store {}",
+                "chunk[{i}] embedding length mismatch: {} vs {}",
                s.embedding.len(),
-                got.len()
+                g.embedding.len()
            )
            .into());
        }
-        for (j, (&a, &b)) in s.embedding.iter().zip(got).enumerate() {
-            let want = expected_value(a);
-            if want.to_bits() != b.to_bits() && !(want.is_nan() && b.is_nan()) {
-                return Err(format!(
-                    "record {idx} (chunk id {id}) embedding[{j}] mismatch: source {a}, \
-                     expected {want}, store {b}"
-                )
-                .into());
+        for (k, (&a, &b)) in s.embedding.iter().zip(g.embedding.iter()).enumerate() {
+            if (a - b).abs() > emb_abs + emb_rel * a.abs() {
+                return Err(
+                    format!("chunk[{i}].embedding[{k}] mismatch: source {a}, HDF5 {b}").into(),
+                );
            }
        }
        rows_checked += 1;
    }

-    // ---- Records tombstoned because their source row was deleted ----
-    for &(idx, src) in &migration.deleted_in_store {
-        let s = &source.chunks[src];
-        let c = &mem.cache;
-        if idx >= c.len() || c.chunks[idx] != s.chunk || c.timestamps[idx] != s.timestamp {
-            return Err(format!("record {idx} (chunk id {}) mismatch or missing", s.id).into());
-        }
-        if c.tombstones[idx] == 0 {
-            return Err(format!(
-                "record {idx} (chunk id {}) is deleted in the source but active in the store",
-                s.id
-            )
-            .into());
-        }
-        rows_checked += 1;
-    }
-
-    // ---- Sessions ----
-    let sessions = mem.sessions();
-    for &(at, src) in &migration.sessions {
-        let s = &source.sessions[src];
-        let (Some(e), Some(summary)) = (sessions.entries.get(at), sessions.summaries.get(at))
-        else {
-            return Err(format!("session {:?} is missing from the store", s.id).into());
-        };
-        if e.id != s.id
-            || e.start_idx != s.start_idx.max(0) as u64
-            || e.end_idx != s.end_idx.max(0) as u64
-            || e.channel != s.channel
-            || *summary != s.summary
-            || e.ts != s.timestamp * US_PER_SEC
+    // ---- Other groups (always full — they are small) ----
+    for (i, (s, g)) in source.sessions.iter().zip(got.sessions.iter()).enumerate() {
+        if s.id != g.id
+            || s.start_idx != g.start_idx
+            || s.end_idx != g.end_idx
+            || s.channel != g.channel
+            || s.summary != g.summary
        {
-            return Err(format!("session {:?} mismatch", s.id).into());
+            return Err(format!("session[{i}] mismatch").into());
        }
        rows_checked += 1;
    }
-
-    // ---- Knowledge graph ----
-    let kg = mem.knowledge();
-    for &(id, src) in &migration.entities {
-        let s = &source.entities[src];
-        let Some(e) = kg.get_entity(id) else {
-            return Err(format!(
-                "entity {:?} (id {}) is missing from the store",
-                s.name, s.id
-            )
-            .into());
-        };
-        if e.name != s.name || e.entity_type != s.entity_type || e.embedding_idx != s.embedding_idx
+    for (i, (s, g)) in source.entities.iter().zip(got.entities.iter()).enumerate() {
+        if s.id != g.id
+            || s.name != g.name
+            || s.entity_type != g.entity_type
+            || s.embedding_idx != g.embedding_idx
        {
-            return Err(format!("entity {:?} (id {}) mismatch", s.name, s.id).into());
+            return Err(format!("entity[{i}] mismatch").into());
        }
        rows_checked += 1;
    }
-    for &(at, src) in &migration.relations {
-        let s = &source.relations[src];
-        let r = kg.relations.get(at);
-        let ok = r.is_some_and(|r| {
-            Some(&r.src) == migration.entity_ids.get(&s.src)
-                && Some(&r.tgt) == migration.entity_ids.get(&s.tgt)
-                && r.relation == s.relation
-                && r.weight == s.weight as f32
-                && r.ts == s.timestamp * US_PER_SEC
-        });
-        if !ok {
-            return Err(format!(
-                "relation {} -[{}]-> {} mismatch or missing",
-                s.src, s.relation, s.tgt
-            )
-            .into());
-        }
-        rows_checked += 1;
-    }
-
-    // ---- A migrated record must be findable by search ----
-    let probe = migration
-        .records
+    for (i, (s, g)) in source
+        .relations
        .iter()
-        .copied()
-        .find(|&(idx, _)| dim > 0 && mem.cache.tombstones[idx] == 0);
-    let search_checked = match probe {
-        None => false,
-        Some((idx, _)) => {
-            let query = mem.cache.embeddings[idx].to_vec();
-            let text = mem.cache.chunks[idx].clone();
-            let hits = mem.search(&query, &text, &SearchOptions::new(10));
-            // A record with the same text is as good a hit: the source may
-            // hold duplicates, and they tie.
-            if !hits.iter().any(|h| h.index == idx || h.chunk == text) {
-                return Err(format!(
-                    "search for migrated record {idx} ({:?}) did not return it",
-                    truncate(&text)
-                )
-                .into());
-            }
-            true
+        .zip(got.relations.iter())
+        .enumerate()
+    {
+        if s.src != g.src || s.tgt != g.tgt || s.relation != g.relation {
+            return Err(format!("relation[{i}] mismatch").into());
        }
-    };
+        rows_checked += 1;
+    }

    Ok(ValidationSummary {
-        count: mem.count(),
-        active: mem.count_active(),
-        sessions: mem.sessions().len(),
-        entities: mem.knowledge().entities.len(),
-        relations: mem.knowledge().relations.len(),
-        embedding_dim: dim,
-        float16,
+        chunks: got.chunks.len() as u64,
+        sessions: got.sessions.len() as u64,
+        entities: got.entities.len() as u64,
+        relations: got.relations.len() as u64,
+        embedding_dim: got.embedding_dim as u64,
        rows_checked,
-        search_checked,
+        provenance_verified,
    })
 }

 fn check_count(kind: &str, got: usize, expected: usize) -> Result<(), BoxErr> {
    if got != expected {
-        return Err(format!("{kind} count mismatch: store has {got}, expected {expected}").into());
+        return Err(format!("{kind} count mismatch: HDF5 has {got}, source has {expected}").into());
    }
    Ok(())
 }

+/// Re-verify the SHA-256 provenance hash of `chunks/text` and
+/// `chunks/embeddings` against their actual stored bytes, catching
+/// post-write corruption that a plain content comparison against the
+/// in-memory source wouldn't (the source is compared against what
+/// `read_hdf5` decoded, not against the raw bytes on disk).
+///
+/// Returns `Ok(true)` only if both datasets exist and both hashes match.
+/// Returns `Ok(false)` (not an error) if a dataset has no provenance
+/// attributes at all (e.g. a file written before this check existed) or
+/// there are zero chunks. Returns an error only on an actual hash mismatch —
+/// that indicates real corruption.
+fn verify_chunk_provenance(path: &str) -> Result<bool, BoxErr> {
+    let file = Hdf5File::open(path)?;
+    let Ok(chunks) = file.group("chunks") else {
+        return Ok(false);
+    };
+    let mut all_present = true;
+    for name in ["text", "embeddings"] {
+        let Ok(ds) = chunks.dataset(name) else {
+            all_present = false;
+            continue;
+        };
+        match ds.verify_provenance()? {
+            VerifyResult::Ok => {}
+            VerifyResult::NoHash => all_present = false,
+            VerifyResult::Mismatch { stored, computed } => {
+                return Err(format!(
+                    "provenance hash mismatch on chunks/{name}: stored {stored}, recomputed {computed} — data may be corrupted"
+                )
+                .into());
+            }
+        }
+    }
+    Ok(all_present)
+}
+
+fn field_err<T: std::fmt::Display>(kind: &str, i: usize, field: &str, s: T, g: T) -> BoxErr {
+    format!("{kind}[{i}].{field} mismatch: source {s}, HDF5 {g}").into()
+}
+
 fn truncate(s: &str) -> String {
    if s.len() <= 40 {
        s.to_string()
@@ -270,7 +196,7 @@ fn truncate(s: &str) -> String {
    }
 }

-/// Indices of records to content-check. Full = all; otherwise a spread of
+/// Indices of chunk rows to content-check. Full = all; otherwise a spread of
 /// representative rows (first/last and evenly-spaced interior samples).
 fn sample_indices(n: usize, full: bool) -> Vec<usize> {
    if n == 0 {
Author	SHA1	Message	Date
osobh	a7920bd4b3	Merge pull request 'Search options (source filters, re-ranking, confidence); float16 default' (#5 ) from feat/search-options into main CI / test-arm64 (push) Successful in 1m8s Details CI / test (push) Successful in 5m40s Details Reviewed-on: #5	2026-09-25 12:44:15 +00:00
osobh	7e43b5366c	Merge branch 'main' into feat/search-options CI / test-arm64 (pull_request) Successful in 54s Details CI / test (pull_request) Successful in 4m35s Details	2026-09-25 12:32:00 +00:00
osobh	73bb068264	Merge pull request 'Files open in h5py again; float16 embedding storage' (#4 ) from feat/float16-embeddings into main CI / test-arm64 (push) Successful in 1m17s Details CI / test (push) Successful in 5m10s Details Reviewed-on: #4	2026-09-25 03:18:07 +00:00