perf(agent): persistent incremental BM25 index; no store rewrite per query

hybrid_search rebuilt the BM25 index from scratch (re-tokenising every record)
and rewrote the whole .h5 file on every single query, so a query cost O(store
size) in both CPU and disk I/O. Steady-state p50 per the search harness:
5.5 -> 0.24 ms (1K), 49 -> 2.1 ms (10K), 884 -> 23 ms (100K).

- BM25Index is incremental: add_document / remove_document keep it exactly
  equivalent to a fresh build over the same live documents (property test: 60
  random op sequences compared against BM25Index::build after every step). IDF
  moves to query time since it depends on the live document count. Top-k uses
  a bounded heap, ties break by doc id (results were HashMap-ordered), and the
  "WAND" code that computed a bound and then discarded it is removed.
- HDF5Memory keeps one index for its lifetime, built lazily. Appends are
  picked up by ensure_bm25_fresh whatever path added them; delete and in-place
  update report themselves; compaction drops the index. A test drives every
  mutation and compares against a fresh build.
- A query no longer calls flush(). Activation boosts are marked dirty and
  persisted by the next checkpoint, including a best-effort one on drop so a
  search-only session keeps them (approved behaviour change). Activation
  weights are capped at 16.0; they previously grew without bound.

The archived mission branch's BM25 cache was reviewed and not used: it was
invalidated by every write, so interleaved save/search still rebuilt per
query, and it changed the default fusion weights.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

CHANGELOG.md

@@ -12,6 +12,20 @@
   0.98 and responds to `ef`. Builds are slower (~3.5x at 10K). Existing
   persisted indexes keep their old graph until rebuilt; the agent rebuilds its
   index from the cache, so stores pick this up automatically.
+- `clawhdf5-agent`: **`hybrid_search` is 23-39x faster in steady state** (p50
+  5.5 -> 0.24 ms at 1K records, 49 -> 2.1 ms at 10K, 884 -> 23 ms at 100K).
+  Every query used to rebuild the BM25 index from scratch and rewrite the whole
+  `.h5` file. The keyword index now lives for the life of the store and is
+  updated incrementally (add / remove / in-place update, exactly equivalent to
+  a fresh build - property-tested), and a query no longer writes the store.
+  **Behaviour change:** Hebbian activation boosts are persisted by the next
+  checkpoint (any flushing write, `flush_wal`, or drop) rather than
+  immediately; a crash in between forgets only the boosts since the last
+  checkpoint. Activation weights are now capped (16.0) - they grew without
+  bound.
+- `clawhdf5-agent`: BM25 results are deterministic (ties break by record id),
+  top-k uses a bounded heap, and the "WAND early termination" that computed a
+  bound and then ignored it is gone. IDF is computed per query.
 - `clawhdf5-bench`: new `search_harness` binary — HNSW recall@10 / QPS / latency
   per `ef` against an exact scan, and end-to-end `hybrid_search` timings, on
   deterministic clustered (or `--uniform`) data. Baseline in `BENCHMARKS.md`.