Merge pull request 'Complete the consolidation benchmark: cheaper novelty scoring' (#7) from feat/consolidation-scaling into main

Reviewed-on: #7

BENCHMARKS.md

@@ -1592,10 +1592,10 @@ Hippocampal-inspired memory consolidation improves both retrieval quality and se
 
 > **Run:** `cargo run --release -p clawhdf5-bench --bin consolidation_efficiency`
 
-Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. The run
+Part 1 measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c.
-was stopped (after about 19 minutes on one core) while still computing a
+The cycle-time and memory-reduction tables are from a complete run on
-100K row of the cycle-time table, so that row and the binary's memory-reduction
+2026-09-25, same machine: the first run was stopped after about 19 minutes on
-part were not produced; neither has ever been published here.
+one core, still building the 100K case (see "Why the first run stalled").
 
 ### Retrieval Quality Before vs. After Consolidation
 
@@ -1616,14 +1616,45 @@ Signal records survive consolidation because they are accessed 15+ times, giving
 
 | Records | Cycle Time | Evictions | Promotions |
 |---------|-----------|-----------|------------|
-| 100 | 17 µs | 100 | 0 |
+| 100 | 19 µs | 100 | 0 |
-| 1K | 189 µs | 1,000 | 0 |
+| 1K | 207 µs | 1,000 | 0 |
-| 10K | 2.16 ms | 10,000 | 0 |
+| 10K | 2.81 ms | 10,000 | 0 |
+| 100K | 46.66 ms | 100,000 | 0 |
 
-The 1K and 10K cycles were 345 µs and 17.3 ms in the previous, undated table
+Each 10x in records costs 11–17x in cycle time — a little worse than linear.
-(before: 2,752 µs / after: 312 µs for search). Eviction's membership check
+The 2026-09-24 partial run measured 17 µs, 189 µs and 2.16 ms for the first
-changed from a linear scan to a `HashSet` in 603fcf8 (2026-08-17), after those
+three rows: single-shot timings, so read the digits after the first as noise.
-figures were recorded; this run does not isolate its effect.
+The 1K and 10K cycles were 345 µs and 17.3 ms in the previous, undated table.
+Eviction's membership check changed from a linear scan to a `HashSet` in
+603fcf8 (2026-08-17), after those figures were recorded.
+
+### Memory Reduction After Consolidation
+
+Working capacity = 20% of the initial records; signal records accessed 15x.
+
+| Initial | Remaining | Evicted | Signal records kept |
+|--------:|----------:|--------:|:-------------------:|
+| 100 | 20 | 80.0% | all |
+| 1,000 | 200 | 80.0% | all |
+| 10,000 | 2,000 | 80.0% | all |
+
+The binary used to print a "BM25 Speedup" column here that was only the ratio
+of record counts, never measured; it is now labelled as the ratio. The
+measured effect on search latency is the before/after table above.
+
+### Why the first run stalled
+
+Not the cycle: building the 100K case. Each `add_memory` scores the new
+record's novelty (1 − its highest cosine similarity) against every record in
+the working tier, and this benchmark lets the working tier grow to 50K before
+consolidating, so the setup does about 5×10⁹ comparisons. Every comparison
+also recomputed both vectors' norms. Scoring now computes the new record's
+norm once, takes each comparison in one vectorised pass, and splits a working
+tier of 4 096 or more across threads (the `parallel` feature, on by default),
+with the same results (tested against the old formula). The complete run took
+8 min 10 s on 16 threads (97 CPU-minutes). The work is still quadratic in the
+working-tier size by design; with consolidation running regularly the tier
+stays near `working_capacity` (100 by default) and each insert is cheap.
 
 ---
 

CHANGELOG.md

@@ -189,6 +189,16 @@
   knew to ask; it now only ever switches the default off.
 
 ### Performance
+- `clawhdf5-agent`: consolidation's novelty scoring (each `add_memory` against
+  the whole working tier) computes the new record's norm once, takes each
+  comparison in one vectorised pass instead of three, and splits a working
+  tier of 4 096+ records across threads — same results, tested against the
+  old formula. It had made `consolidation_efficiency` stall at 100K; the
+  complete run now takes 8 min and fills in the 100K cycle row (46.66 ms) and
+  the memory-reduction table.
+- `clawhdf5-bench`: `consolidation_efficiency` no longer prints a record-count
+  ratio as a "BM25 Speedup" (it was never measured), nor claims cycle time
+  grows sub-linearly (its own numbers grow slightly faster than linearly).
 - `clawhdf5-agent`: **knowledge-graph traversal was 6.5x slower than it
   should be.** `bfs_neighbors` and `spreading_activation` built an adjacency
   index over the whole graph on every call (1efd82c), so a 2-hop BFS over 1K

README.md

@@ -305,8 +305,8 @@ exactly; the `i8` column was not re-run.
 | Hit@1 recall (signal records) | 100% | 100% | no loss |
 | Search latency (avg) | 2.22 ms | 0.24 ms | **9.3x faster** |
 
-The consolidation cycle that does this took 0.13 ms; at 10K records a cycle
+The consolidation cycle that does this took 0.13 ms; a cycle over 10K records
-takes 2.16 ms.
+takes 2.81 ms and over 100K 46.7 ms.
 
 **Full benchmark details: [BENCHMARKS.md](BENCHMARKS.md)**
 

crates/clawhdf5-agent/src/consolidation.rs

@@ -144,9 +144,44 @@ pub struct ConsolidationStats {
 
 pub struct ImportanceScorer;
 
+/// Sum of squares, in 8-wide lanes so it vectorises.
+fn sum_of_squares(a: &[f32]) -> f32 {
+    let (blocks, tail) = a.as_chunks::<8>();
+    let mut acc = [0.0f32; 8];
+    for b in blocks {
+        for i in 0..8 {
+            acc[i] += b[i] * b[i];
+        }
+    }
+    acc.iter().sum::<f32>() + tail.iter().map(|x| x * x).sum::<f32>()
+}
+
+/// `(a · b, |b|²)` in one pass over equal-length slices, in 8-wide lanes.
+fn dot_and_norm2(a: &[f32], b: &[f32]) -> (f32, f32) {
+    let (a_blocks, a_tail) = a.as_chunks::<8>();
+    let (b_blocks, b_tail) = b.as_chunks::<8>();
+    let mut dot = [0.0f32; 8];
+    let mut nb = [0.0f32; 8];
+    for (x, y) in a_blocks.iter().zip(b_blocks) {
+        for i in 0..8 {
+            dot[i] += x[i] * y[i];
+            nb[i] += y[i] * y[i];
+        }
+    }
+    let mut d = dot.iter().sum::<f32>();
+    let mut n = nb.iter().sum::<f32>();
+    for (x, y) in a_tail.iter().zip(b_tail) {
+        d += x * y;
+        n += y * y;
+    }
+    (d, n)
+}
+
 impl ImportanceScorer {
     /// Cosine similarity between two embedding slices.
-    /// Returns 0.0 if either norm is zero.
+    /// Returns 0.0 if either norm is zero. The reference that
+    /// [`Self::score_surprise`] is tested against.
+    #[cfg(test)]
     fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 {
         let len = a.len().min(b.len());
         if len == 0 {
@@ -167,13 +202,54 @@ impl ImportanceScorer {
 
     /// Novelty score: 1.0 − max cosine similarity against all existing records.
     /// Returns 1.0 when there are no existing memories.
+    ///
+    /// Same result as [`Self::cosine_similarity`] against each record, but the
+    /// new embedding's norm is computed once rather than per record, each
+    /// record costs one fused pass (dot product and its norm together) rather
+    /// than three, and a large working set is scored in parallel. Every insert
+    /// scores against the whole working tier, so this is what an unbounded
+    /// working tier pays for: at 100K records it was the difference between a
+    /// benchmark finishing and not (`BENCHMARKS.md`, "Consolidation Efficiency").
     pub fn score_surprise(embedding: &[f32], existing_memories: &[&MemoryRecord]) -> f32 {
         if existing_memories.is_empty() {
             return 1.0;
         }
+        let query_norm2 = sum_of_squares(embedding);
+        let similarity = |r: &&MemoryRecord| -> f32 {
+            let other = &r.embedding;
+            let len = embedding.len().min(other.len());
+            if len == 0 {
+                return 0.0;
+            }
+            let (dot, other_norm2) = dot_and_norm2(&embedding[..len], &other[..len]);
+            // A shorter record compares against the query's matching prefix.
+            let q2 = if len == embedding.len() {
+                query_norm2
+            } else {
+                sum_of_squares(&embedding[..len])
+            };
+            if q2 == 0.0 || other_norm2 == 0.0 {
+                return 0.0;
+            }
+            dot / (q2.sqrt() * other_norm2.sqrt())
+        };
+        #[cfg(feature = "parallel")]
+        let max_sim = if existing_memories.len() >= 4096 {
+            use rayon::prelude::*;
+            existing_memories
+                .par_iter()
+                .map(similarity)
+                .reduce(|| f32::NEG_INFINITY, f32::max)
+        } else {
+            existing_memories
+                .iter()
+                .map(similarity)
+                .fold(f32::NEG_INFINITY, f32::max)
+        };
+        #[cfg(not(feature = "parallel"))]
         let max_sim = existing_memories
             .iter()
-            .map(|r| Self::cosine_similarity(embedding, &r.embedding))
+            .map(similarity)
             .fold(f32::NEG_INFINITY, f32::max);
         (1.0 - max_sim).clamp(0.0, 1.0)
     }
@@ -471,6 +547,54 @@ impl ConsolidationEngine {
 mod tests {
     use super::*;
 
+    #[test]
+    fn score_surprise_matches_the_reference_cosine() {
+        let mut x = 0x2545_F491_4F6C_DD1Du64;
+        let mut next = || {
+            x ^= x << 13;
+            x ^= x >> 7;
+            x ^= x << 17;
+            (x >> 40) as f32 / (1u64 << 24) as f32 - 0.5
+        };
+        let make = |id: u64, v: Vec<f32>| MemoryRecord {
+            id,
+            chunk: String::new(),
+            embedding: v,
+            tier: MemoryTier::Working,
+            importance: 0.0,
+            access_count: 0,
+            last_accessed: 0.0,
+            created_at: 0.0,
+            source: MemorySource::User,
+        };
+        // Ordinary rows, a shorter one, an empty one and a zero vector; and
+        // enough rows to take the parallel path too.
+        for n in [5usize, 5000] {
+            let mut recs: Vec<MemoryRecord> = (0..n as u64)
+                .map(|i| make(i, (0..37).map(|_| next()).collect()))
+                .collect();
+            recs.push(make(9_000, (0..20).map(|_| next()).collect()));
+            recs.push(make(9_001, Vec::new()));
+            recs.push(make(9_002, vec![0.0; 37]));
+            let refs: Vec<&MemoryRecord> = recs.iter().collect();
+            for _ in 0..5 {
+                let q: Vec<f32> = (0..37).map(|_| next()).collect();
+                let expected = (1.0
+                    - refs
+                        .iter()
+                        .map(|r| ImportanceScorer::cosine_similarity(&q, &r.embedding))
+                        .fold(f32::NEG_INFINITY, f32::max))
+                .clamp(0.0, 1.0);
+                let got = ImportanceScorer::score_surprise(&q, &refs);
+                assert!((got - expected).abs() < 1e-5, "n={n}: {got} vs {expected}");
+            }
+        }
+        assert_eq!(
+            ImportanceScorer::score_surprise(&[0.0; 4], &[&make(1, vec![1.0; 4])]),
+            1.0
+        );
+    }
+
     // Helper: build a simple normalised embedding of given dimension.
     fn unit_vec(dim: usize, hot: usize) -> Vec<f32> {
         let mut v = vec![0.0f32; dim];

crates/clawhdf5-bench/src/bin/consolidation_efficiency.rs

@@ -396,7 +396,7 @@ fn run_memory_reduction_benchmark() {
     println!();
     println!(
         "{:>8}  {:>10}  {:>10}  {:>10}  {:>12}",
-        "Initial", "Remaining", "Eviction%", "Signal OK?", "BM25 Speedup"
+        "Initial", "Remaining", "Eviction%", "Signal OK?", "Records ÷"
     );
     println!("{}", "-".repeat(58));
 
@@ -440,7 +440,8 @@ fn run_memory_reduction_benchmark() {
         // Check all signal records survived
         let signal_survived = signal_ids.iter().all(|&id| engine.get_by_id(id).is_some());
 
-        // Rough speedup: BM25 scales roughly linearly with record count
+        // How many times fewer records there are. Not a measured speedup —
+        // Part 1 measures search latency before and after.
         let speedup = before_count as f64 / after_count.max(1) as f64;
 
         println!(
@@ -480,7 +481,7 @@ fn main() {
     println!("  3. Reducing search latency proportional to record reduction");
     println!();
     println!(
-        "Cycle time scales sub-linearly: 100 records ~microseconds, 100K records ~tens of ms."
+        "Cycle time grows a little faster than linearly: 100 records ~microseconds, 100K records ~tens of ms."
     );
     println!("Signal records with Correction source + high access_count survive eviction.");
 }