perf(agent): persistent incremental BM25 index; no store rewrite per query

hybrid_search rebuilt the BM25 index from scratch (re-tokenising every record)
and rewrote the whole .h5 file on every single query, so a query cost O(store
size) in both CPU and disk I/O. Steady-state p50 per the search harness:
5.5 -> 0.24 ms (1K), 49 -> 2.1 ms (10K), 884 -> 23 ms (100K).

- BM25Index is incremental: add_document / remove_document keep it exactly
  equivalent to a fresh build over the same live documents (property test: 60
  random op sequences compared against BM25Index::build after every step). IDF
  moves to query time since it depends on the live document count. Top-k uses
  a bounded heap, ties break by doc id (results were HashMap-ordered), and the
  "WAND" code that computed a bound and then discarded it is removed.
- HDF5Memory keeps one index for its lifetime, built lazily. Appends are
  picked up by ensure_bm25_fresh whatever path added them; delete and in-place
  update report themselves; compaction drops the index. A test drives every
  mutation and compares against a fresh build.
- A query no longer calls flush(). Activation boosts are marked dirty and
  persisted by the next checkpoint, including a best-effort one on drop so a
  search-only session keeps them (approved behaviour change). Activation
  weights are capped at 16.0; they previously grew without bound.

The archived mission branch's BM25 cache was reviewed and not used: it was
invalidated by every write, so interleaved save/search still rebuilt per
query, and it changed the default fusion weights.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

crates/clawhdf5-agent/src/search.rs

@@ -4,7 +4,7 @@ use std::path::Path;
 
 use crate::bm25;
 use crate::hybrid;
-use crate::{HDF5Memory, MemoryError, Result, SearchResult};
+use crate::{HDF5Memory, MAX_ACTIVATION_WEIGHT, MemoryError, Result, SearchResult};
 
 impl HDF5Memory {
     /// Vector + keyword scoring stage of [`HDF5Memory::hybrid_search`].
@@ -90,7 +90,11 @@ impl HDF5Memory {
         keyword_weight: f32,
         k: usize,
     ) -> Vec<SearchResult> {
-        let bm25 = bm25::BM25Index::build(&self.cache.chunks, &self.cache.tombstones);
+        // The keyword index lives for the life of the store and is updated
+        // incrementally. Take it out for the duration of the call so the
+        // vector stage can borrow `self` mutably, then put it back.
+        self.ensure_bm25_fresh();
+        let bm25 = self.bm25.take().expect("ensure_bm25_fresh leaves an index");
         let scored = self.vector_keyword_search(
             query_embedding,
             query_text,
@@ -132,15 +136,26 @@ impl HDF5Memory {
             .map(|r| r.index)
             .collect();
         self.apply_hebbian_boost(&hit_indices);
-        self.flush().ok();
+        self.bm25 = Some(bm25);
 
         results
     }
 
+    /// Reinforce the records a query returned. The new weights are persisted by
+    /// the next checkpoint (any write that flushes, `flush_wal`, or drop) — not
+    /// by rewriting the whole store inside the query, which is what made
+    /// `hybrid_search` cost O(store size) in disk I/O. They are a ranking hint,
+    /// not user data: a crash before the next checkpoint only forgets the
+    /// boosts since the last one.
     fn apply_hebbian_boost(&mut self, hit_indices: &[usize]) {
-        for &idx in hit_indices {
-            self.cache.activation_weights[idx] += self.config.hebbian_boost;
+        if hit_indices.is_empty() || self.config.hebbian_boost == 0.0 {
+            return;
         }
+        for &idx in hit_indices {
+            let w = &mut self.cache.activation_weights[idx];
+            *w = (*w + self.config.hebbian_boost).min(MAX_ACTIVATION_WEIGHT);
+        }
+        self.activations_dirty = true;
     }
 
     /// Get the chunk text for a memory entry by index.