perf(agent): cheaper novelty scoring; complete the consolidation benchmark

consolidation_efficiency never finished: stopped after 19 minutes on one
core while building its 100K case. Not the consolidation cycle (linear:
17 us at 100 records, 2.16 ms at 10K) but the setup — every add_memory
scores the new record's novelty against the whole working tier, the
benchmark lets that tier reach 50K, and each comparison recomputed both
norms: ~5e9 comparisons of three passes each.

ImportanceScorer::score_surprise now computes the new record's norm
once, takes each comparison in one fused, 8-lane pass (dot product and
the other norm together), and splits a working tier of 4096+ records
across threads with the `parallel` feature. Same results: tested against
the old cosine formula, including shorter, empty and zero vectors and
the parallel path. The work stays quadratic in the working-tier size by
design; with regular consolidation the tier stays near
working_capacity (100) and inserts are cheap.

The complete run takes 8 min 10 s on tank and fills in the 100K cycle
row (46.66 ms) and the memory-reduction table, which had never been
published. The binary no longer prints a record-count ratio as a
"BM25 Speedup" (never measured; Part 1 measures search latency) or
claims sub-linear cycle scaling (its own numbers grow slightly faster
than linearly).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

BENCHMARKS.md

@@ -1592,10 +1592,10 @@ Hippocampal-inspired memory consolidation improves both retrieval quality and se
 
 > **Run:** `cargo run --release -p clawhdf5-bench --bin consolidation_efficiency`
 
-Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. The run
-was stopped (after about 19 minutes on one core) while still computing a
-100K row of the cycle-time table, so that row and the binary's memory-reduction
-part were not produced; neither has ever been published here.
+Part 1 measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
+The cycle-time and memory-reduction tables are from a complete run on
+2026-09-25, same machine: the first run was stopped after about 19 minutes on
+one core, still building the 100K case (see "Why the first run stalled").
 
 ### Retrieval Quality Before vs. After Consolidation
 
@@ -1616,14 +1616,45 @@ Signal records survive consolidation because they are accessed 15+ times, giving
 
 | Records | Cycle Time | Evictions | Promotions |
 |---------|-----------|-----------|------------|
-| 100 | 17 µs | 100 | 0 |
-| 1K | 189 µs | 1,000 | 0 |
-| 10K | 2.16 ms | 10,000 | 0 |
+| 100 | 19 µs | 100 | 0 |
+| 1K | 207 µs | 1,000 | 0 |
+| 10K | 2.81 ms | 10,000 | 0 |
+| 100K | 46.66 ms | 100,000 | 0 |
 
-The 1K and 10K cycles were 345 µs and 17.3 ms in the previous, undated table
-(before: 2,752 µs / after: 312 µs for search). Eviction's membership check
-changed from a linear scan to a `HashSet` in 603fcf8 (2026-08-17), after those
-figures were recorded; this run does not isolate its effect.
+Each 10x in records costs 11–17x in cycle time — a little worse than linear.
+The 2026-09-24 partial run measured 17 µs, 189 µs and 2.16 ms for the first
+three rows: single-shot timings, so read the digits after the first as noise.
+The 1K and 10K cycles were 345 µs and 17.3 ms in the previous, undated table.
+Eviction's membership check changed from a linear scan to a `HashSet` in
+603fcf8 (2026-08-17), after those figures were recorded.
+
+### Memory Reduction After Consolidation
+
+Working capacity = 20% of the initial records; signal records accessed 15x.
+
+| Initial | Remaining | Evicted | Signal records kept |
+|--------:|----------:|--------:|:-------------------:|
+| 100 | 20 | 80.0% | all |
+| 1,000 | 200 | 80.0% | all |
+| 10,000 | 2,000 | 80.0% | all |
+
+The binary used to print a "BM25 Speedup" column here that was only the ratio
+of record counts, never measured; it is now labelled as the ratio. The
+measured effect on search latency is the before/after table above.
+
+### Why the first run stalled
+
+Not the cycle: building the 100K case. Each `add_memory` scores the new
+record's novelty (1 − its highest cosine similarity) against every record in
+the working tier, and this benchmark lets the working tier grow to 50K before
+consolidating, so the setup does about 5×10⁹ comparisons. Every comparison
+also recomputed both vectors' norms. Scoring now computes the new record's
+norm once, takes each comparison in one vectorised pass, and splits a working
+tier of 4 096 or more across threads (the `parallel` feature, on by default),
+with the same results (tested against the old formula). The complete run took
+8 min 10 s on 16 threads (97 CPU-minutes). The work is still quadratic in the
+working-tier size by design; with consolidation running regularly the tier
+stays near `working_capacity` (100 by default) and each insert is cheap.
 
 ---