perf(agent): persist the vector index graph; incremental catch-up

open() marked the HNSW index dirty, so the first search of every session
rebuilt it from scratch — 36 s at 100K records with the (better, slower)
heuristic build. First query after open is now 1.7 / 15 / 159 ms at
1K / 10K / 100K; what remains is the one-off keyword index build.

- clawhdf5-ann: HnswIndex::graph_to_bytes / from_graph_bytes serialize the
  graph only (levels, tombstones, adjacency as u32, CRC32). The existing HDF5
  serializer embeds a full copy of every vector, which would double a store
  that already holds them. Loading validates everything — counts, levels vs
  layer count, connection limits, every neighbour id and the layer it must
  exist on — so a damaged graph, or a hostile one with a valid checksum, is an
  error rather than an out-of-bounds walk during search.
- clawhdf5-agent: each checkpoint writes the graph to <store>.h5.ann (synced,
  atomic, before the .h5) and records a fresh generation id in /meta. open()
  loads the sidecar only if its generation matches that checkpoint; missing,
  stale, damaged or mismatched sidecars are ignored and the index rebuilt.
  Records appended through WAL replay join the loaded index incrementally; a
  replayed Update or Tombstone invalidates it. snapshot() copies it. Only an
  index that exactly mirrors the cache is saved; otherwise a stale sidecar is
  removed.
- ensure_hnsw_fresh inserts records appended since the last sync instead of
  rebuilding, so save_batch no longer marks the whole index dirty.
- CheckpointMeta { wal_applied, ann_generation } with *_with_meta build/write/
  read functions; the *_with_mark ones delegate.
- Harness reports the one-off cold index build separately from the first query
  after a reopen.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

crates/clawhdf5-bench/src/bin/search_harness.rs

@@ -7,8 +7,8 @@
 //! * **ANN** — index build time, and for each `ef`: recall@10 against an exact
 //!   brute-force scan, queries/second, and p50/p99 latency.
 //! * **End to end** — `HDF5Memory`: ingest time, checkpoint time, `open()`
-//!   time, the first query after open (which pays for any index rebuild), and
-//!   steady-state `hybrid_search` p50/p99 at each store size.
+//!   time, the one-off cold index build (first query ever), the first query
+//!   after a reopen, and steady-state `hybrid_search` p50/p99 at each size.
 //!
 //! Data is *clustered* (points = cluster centre + noise, unit-normalised), not
 //! uniform: uniform random high-dimensional vectors are nearly equidistant,
@@ -310,6 +310,13 @@ fn bench_end_to_end(n: usize, json: &mut Vec<serde_json::Value>) {
     let t = Instant::now();
     mem.save_batch(entries).unwrap();
     let ingest = t.elapsed();
+    // The very first query builds the vector and keyword indexes from
+    // scratch. It happens once per store, not once per session: the checkpoint
+    // below saves the vector index, so a later `open()` reloads it.
+    let t = Instant::now();
+    std::hint::black_box(mem.hybrid_search(&data.queries[1], &query_texts[1], 0.7, 0.3, K));
+    let cold_build = t.elapsed();
+
     let t = Instant::now();
     mem.flush_wal().unwrap();
     let checkpoint = t.elapsed();
@@ -343,8 +350,9 @@ fn bench_end_to_end(n: usize, json: &mut Vec<serde_json::Value>) {
     );
 
     println!(
-        "| {n} | {:.0} | {:.1} | {:.1} | {:.1} | {:.2} | {:.2} | {:.1} |",
+        "| {n} | {:.0} | {:.0} | {:.1} | {:.1} | {:.1} | {:.2} | {:.2} | {:.1} |",
         millis(ingest),
+        millis(cold_build),
         millis(checkpoint),
         millis(open),
         millis(first_query),
@@ -354,7 +362,8 @@ fn bench_end_to_end(n: usize, json: &mut Vec<serde_json::Value>) {
     );
     json.push(serde_json::json!({
         "bench": "hybrid_search", "n": n,
-        "ingest_ms": millis(ingest), "checkpoint_ms": millis(checkpoint),
+        "ingest_ms": millis(ingest), "cold_index_build_ms": millis(cold_build),
+        "checkpoint_ms": millis(checkpoint),
         "open_ms": millis(open), "first_query_ms": millis(first_query),
         "p50_ms": millis(steady.p50), "p99_ms": millis(steady.p99), "qps": steady.qps,
     }));
@@ -394,9 +403,9 @@ fn main() {
     }
     println!("\n### End to end: `HDF5Memory::hybrid_search` (k = {K}, weights 0.7 / 0.3)\n");
     println!(
-        "| N | ingest ms | checkpoint ms | open ms | first query ms | p50 ms | p99 ms | QPS |"
+        "| N | ingest ms | cold index build ms | checkpoint ms | open ms | first query after open ms | p50 ms | p99 ms | QPS |"
     );
-    println!("|---:|---:|---:|---:|---:|---:|---:|---:|");
+    println!("|---:|---:|---:|---:|---:|---:|---:|---:|---:|");
     for &n in sizes {
         bench_end_to_end(n, &mut json);
     }