perf(agent): persist the vector index graph; incremental catch-up

open() marked the HNSW index dirty, so the first search of every session
rebuilt it from scratch — 36 s at 100K records with the (better, slower)
heuristic build. First query after open is now 1.7 / 15 / 159 ms at
1K / 10K / 100K; what remains is the one-off keyword index build.

- clawhdf5-ann: HnswIndex::graph_to_bytes / from_graph_bytes serialize the
  graph only (levels, tombstones, adjacency as u32, CRC32). The existing HDF5
  serializer embeds a full copy of every vector, which would double a store
  that already holds them. Loading validates everything — counts, levels vs
  layer count, connection limits, every neighbour id and the layer it must
  exist on — so a damaged graph, or a hostile one with a valid checksum, is an
  error rather than an out-of-bounds walk during search.
- clawhdf5-agent: each checkpoint writes the graph to <store>.h5.ann (synced,
  atomic, before the .h5) and records a fresh generation id in /meta. open()
  loads the sidecar only if its generation matches that checkpoint; missing,
  stale, damaged or mismatched sidecars are ignored and the index rebuilt.
  Records appended through WAL replay join the loaded index incrementally; a
  replayed Update or Tombstone invalidates it. snapshot() copies it. Only an
  index that exactly mirrors the cache is saved; otherwise a stale sidecar is
  removed.
- ensure_hnsw_fresh inserts records appended since the last sync instead of
  rebuilding, so save_batch no longer marks the whole index dirty.
- CheckpointMeta { wal_applied, ann_generation } with *_with_meta build/write/
  read functions; the *_with_mark ones delegate.
- Harness reports the one-off cold index build separately from the first query
  after a reopen.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

crates/clawhdf5-agent/src/schema.rs

@@ -22,6 +22,7 @@ pub const ZEROCLAW_VERSION: &str = "0.8.0";
 /// the checkpoint was taken with an empty WAL.
 const WAL_APPLIED_LEN_ATTR: &str = "wal_applied_len";
 const WAL_APPLIED_CRC_ATTR: &str = "wal_applied_crc";
+const ANN_GENERATION_ATTR: &str = "ann_generation";
 
 /// Build a complete HDF5 file from the in-memory state.
 pub fn build_hdf5_file(
@@ -43,6 +44,34 @@ pub fn build_hdf5_file_with_mark(
     knowledge: &KnowledgeCache,
     wal_applied: Option<WalMark>,
 ) -> Result<Vec<u8>, MemoryError> {
+    let meta = CheckpointMeta {
+        wal_applied,
+        ann_generation: None,
+    };
+    build_hdf5_file_with_meta(config, cache, sessions, knowledge, &meta)
+}
+
+/// Bookkeeping a checkpoint records in `/meta` beside the store's contents.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct CheckpointMeta {
+    /// The WAL prefix this checkpoint already contains; see [`WalMark`].
+    pub wal_applied: Option<WalMark>,
+    /// Identifies the vector-index sidecar (`<store>.h5.ann`) written with this
+    /// checkpoint. A sidecar is loaded only if it carries the same value, so
+    /// one left over from another checkpoint can never be attached to records
+    /// it wasn't built from.
+    pub ann_generation: Option<u64>,
+}
+
+/// [`build_hdf5_file`] with checkpoint bookkeeping.
+pub fn build_hdf5_file_with_meta(
+    config: &MemoryConfig,
+    cache: &MemoryCache,
+    sessions: &SessionCache,
+    knowledge: &KnowledgeCache,
+    checkpoint: &CheckpointMeta,
+) -> Result<Vec<u8>, MemoryError> {
+    let wal_applied = checkpoint.wal_applied;
     let mut builder = clawhdf5::FileBuilder::new();
 
     // /meta group with schema attributes
@@ -83,6 +112,11 @@ pub fn build_hdf5_file_with_mark(
         meta.set_attr(WAL_APPLIED_LEN_ATTR, AttrValue::I64(mark.len as i64));
         meta.set_attr(WAL_APPLIED_CRC_ATTR, AttrValue::I64(i64::from(mark.crc)));
     }
+    if let Some(generation) = checkpoint.ann_generation {
+        // Stored as the i64 with the same bits; attributes have no u64 scalar
+        // round trip through every reader.
+        meta.set_attr(ANN_GENERATION_ATTR, AttrValue::I64(generation as i64));
+    }
     // Need at least one dataset in the group for it to be a proper group
     meta.create_dataset("_marker").with_u8_data(&[1]).compact();
     let finished_meta = meta.finish();
@@ -386,6 +420,22 @@ pub fn read_wal_mark(file: &clawhdf5::File) -> Option<WalMark> {
     Some(WalMark { len, crc })
 }
 
+/// Read the checkpoint bookkeeping from `/meta`.
+pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta {
+    let ann_generation = file
+        .group("meta")
+        .ok()
+        .and_then(|g| g.attrs().ok())
+        .and_then(|attrs| match attrs.get(ANN_GENERATION_ATTR) {
+            Some(AttrValue::I64(v)) => Some(*v as u64),
+            _ => None,
+        });
+    CheckpointMeta {
+        wal_applied: read_wal_mark(file),
+        ann_generation,
+    }
+}
+
 pub fn validate_and_load(
     file: &clawhdf5::File,
 ) -> Result<(MemoryConfig, MemoryCache, SessionCache, KnowledgeCache), MemoryError> {