diff --git a/BENCHMARKS.md b/BENCHMARKS.md index fdccb50..9866d71 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -1592,10 +1592,10 @@ Hippocampal-inspired memory consolidation improves both retrieval quality and se > **Run:** `cargo run --release -p clawhdf5-bench --bin consolidation_efficiency` -Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. The run -was stopped (after about 19 minutes on one core) while still computing a -100K row of the cycle-time table, so that row and the binary's memory-reduction -part were not produced; neither has ever been published here. +Part 1 measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D), commit 5c8323c. +The cycle-time and memory-reduction tables are from a complete run on +2026-09-25, same machine: the first run was stopped after about 19 minutes on +one core, still building the 100K case (see "Why the first run stalled"). ### Retrieval Quality Before vs. After Consolidation @@ -1616,14 +1616,45 @@ Signal records survive consolidation because they are accessed 15+ times, giving | Records | Cycle Time | Evictions | Promotions | |---------|-----------|-----------|------------| -| 100 | 17 µs | 100 | 0 | -| 1K | 189 µs | 1,000 | 0 | -| 10K | 2.16 ms | 10,000 | 0 | +| 100 | 19 µs | 100 | 0 | +| 1K | 207 µs | 1,000 | 0 | +| 10K | 2.81 ms | 10,000 | 0 | +| 100K | 46.66 ms | 100,000 | 0 | -The 1K and 10K cycles were 345 µs and 17.3 ms in the previous, undated table -(before: 2,752 µs / after: 312 µs for search). Eviction's membership check -changed from a linear scan to a `HashSet` in 603fcf8 (2026-08-17), after those -figures were recorded; this run does not isolate its effect. +Each 10x in records costs 11–17x in cycle time — a little worse than linear. +The 2026-09-24 partial run measured 17 µs, 189 µs and 2.16 ms for the first +three rows: single-shot timings, so read the digits after the first as noise. +The 1K and 10K cycles were 345 µs and 17.3 ms in the previous, undated table. +Eviction's membership check changed from a linear scan to a `HashSet` in +603fcf8 (2026-08-17), after those figures were recorded. + +### Memory Reduction After Consolidation + +Working capacity = 20% of the initial records; signal records accessed 15x. + +| Initial | Remaining | Evicted | Signal records kept | +|--------:|----------:|--------:|:-------------------:| +| 100 | 20 | 80.0% | all | +| 1,000 | 200 | 80.0% | all | +| 10,000 | 2,000 | 80.0% | all | + +The binary used to print a "BM25 Speedup" column here that was only the ratio +of record counts, never measured; it is now labelled as the ratio. The +measured effect on search latency is the before/after table above. + +### Why the first run stalled + +Not the cycle: building the 100K case. Each `add_memory` scores the new +record's novelty (1 − its highest cosine similarity) against every record in +the working tier, and this benchmark lets the working tier grow to 50K before +consolidating, so the setup does about 5×10⁹ comparisons. Every comparison +also recomputed both vectors' norms. Scoring now computes the new record's +norm once, takes each comparison in one vectorised pass, and splits a working +tier of 4 096 or more across threads (the `parallel` feature, on by default), +with the same results (tested against the old formula). The complete run took +8 min 10 s on 16 threads (97 CPU-minutes). The work is still quadratic in the +working-tier size by design; with consolidation running regularly the tier +stays near `working_capacity` (100 by default) and each insert is cheap. --- diff --git a/CHANGELOG.md b/CHANGELOG.md index c32606a..97b2342 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -189,6 +189,16 @@ knew to ask; it now only ever switches the default off. ### Performance +- `clawhdf5-agent`: consolidation's novelty scoring (each `add_memory` against + the whole working tier) computes the new record's norm once, takes each + comparison in one vectorised pass instead of three, and splits a working + tier of 4 096+ records across threads — same results, tested against the + old formula. It had made `consolidation_efficiency` stall at 100K; the + complete run now takes 8 min and fills in the 100K cycle row (46.66 ms) and + the memory-reduction table. +- `clawhdf5-bench`: `consolidation_efficiency` no longer prints a record-count + ratio as a "BM25 Speedup" (it was never measured), nor claims cycle time + grows sub-linearly (its own numbers grow slightly faster than linearly). - `clawhdf5-agent`: **knowledge-graph traversal was 6.5x slower than it should be.** `bfs_neighbors` and `spreading_activation` built an adjacency index over the whole graph on every call (1efd82c), so a 2-hop BFS over 1K diff --git a/README.md b/README.md index 588a825..cbf37e9 100644 --- a/README.md +++ b/README.md @@ -305,8 +305,8 @@ exactly; the `i8` column was not re-run. | Hit@1 recall (signal records) | 100% | 100% | no loss | | Search latency (avg) | 2.22 ms | 0.24 ms | **9.3x faster** | -The consolidation cycle that does this took 0.13 ms; at 10K records a cycle -takes 2.16 ms. +The consolidation cycle that does this took 0.13 ms; a cycle over 10K records +takes 2.81 ms and over 100K 46.7 ms. **Full benchmark details: [BENCHMARKS.md](BENCHMARKS.md)** diff --git a/crates/clawhdf5-agent/src/consolidation.rs b/crates/clawhdf5-agent/src/consolidation.rs index 420b7b6..49ed656 100644 --- a/crates/clawhdf5-agent/src/consolidation.rs +++ b/crates/clawhdf5-agent/src/consolidation.rs @@ -144,9 +144,44 @@ pub struct ConsolidationStats { pub struct ImportanceScorer; +/// Sum of squares, in 8-wide lanes so it vectorises. +fn sum_of_squares(a: &[f32]) -> f32 { + let (blocks, tail) = a.as_chunks::<8>(); + let mut acc = [0.0f32; 8]; + for b in blocks { + for i in 0..8 { + acc[i] += b[i] * b[i]; + } + } + acc.iter().sum::() + tail.iter().map(|x| x * x).sum::() +} + +/// `(a · b, |b|²)` in one pass over equal-length slices, in 8-wide lanes. +fn dot_and_norm2(a: &[f32], b: &[f32]) -> (f32, f32) { + let (a_blocks, a_tail) = a.as_chunks::<8>(); + let (b_blocks, b_tail) = b.as_chunks::<8>(); + let mut dot = [0.0f32; 8]; + let mut nb = [0.0f32; 8]; + for (x, y) in a_blocks.iter().zip(b_blocks) { + for i in 0..8 { + dot[i] += x[i] * y[i]; + nb[i] += y[i] * y[i]; + } + } + let mut d = dot.iter().sum::(); + let mut n = nb.iter().sum::(); + for (x, y) in a_tail.iter().zip(b_tail) { + d += x * y; + n += y * y; + } + (d, n) +} + impl ImportanceScorer { /// Cosine similarity between two embedding slices. - /// Returns 0.0 if either norm is zero. + /// Returns 0.0 if either norm is zero. The reference that + /// [`Self::score_surprise`] is tested against. + #[cfg(test)] fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 { let len = a.len().min(b.len()); if len == 0 { @@ -167,13 +202,54 @@ impl ImportanceScorer { /// Novelty score: 1.0 − max cosine similarity against all existing records. /// Returns 1.0 when there are no existing memories. + /// + /// Same result as [`Self::cosine_similarity`] against each record, but the + /// new embedding's norm is computed once rather than per record, each + /// record costs one fused pass (dot product and its norm together) rather + /// than three, and a large working set is scored in parallel. Every insert + /// scores against the whole working tier, so this is what an unbounded + /// working tier pays for: at 100K records it was the difference between a + /// benchmark finishing and not (`BENCHMARKS.md`, "Consolidation Efficiency"). pub fn score_surprise(embedding: &[f32], existing_memories: &[&MemoryRecord]) -> f32 { if existing_memories.is_empty() { return 1.0; } + let query_norm2 = sum_of_squares(embedding); + let similarity = |r: &&MemoryRecord| -> f32 { + let other = &r.embedding; + let len = embedding.len().min(other.len()); + if len == 0 { + return 0.0; + } + let (dot, other_norm2) = dot_and_norm2(&embedding[..len], &other[..len]); + // A shorter record compares against the query's matching prefix. + let q2 = if len == embedding.len() { + query_norm2 + } else { + sum_of_squares(&embedding[..len]) + }; + if q2 == 0.0 || other_norm2 == 0.0 { + return 0.0; + } + dot / (q2.sqrt() * other_norm2.sqrt()) + }; + #[cfg(feature = "parallel")] + let max_sim = if existing_memories.len() >= 4096 { + use rayon::prelude::*; + existing_memories + .par_iter() + .map(similarity) + .reduce(|| f32::NEG_INFINITY, f32::max) + } else { + existing_memories + .iter() + .map(similarity) + .fold(f32::NEG_INFINITY, f32::max) + }; + #[cfg(not(feature = "parallel"))] let max_sim = existing_memories .iter() - .map(|r| Self::cosine_similarity(embedding, &r.embedding)) + .map(similarity) .fold(f32::NEG_INFINITY, f32::max); (1.0 - max_sim).clamp(0.0, 1.0) } @@ -471,6 +547,54 @@ impl ConsolidationEngine { mod tests { use super::*; + #[test] + fn score_surprise_matches_the_reference_cosine() { + let mut x = 0x2545_F491_4F6C_DD1Du64; + let mut next = || { + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + (x >> 40) as f32 / (1u64 << 24) as f32 - 0.5 + }; + let make = |id: u64, v: Vec| MemoryRecord { + id, + chunk: String::new(), + embedding: v, + tier: MemoryTier::Working, + importance: 0.0, + access_count: 0, + last_accessed: 0.0, + created_at: 0.0, + source: MemorySource::User, + }; + // Ordinary rows, a shorter one, an empty one and a zero vector; and + // enough rows to take the parallel path too. + for n in [5usize, 5000] { + let mut recs: Vec = (0..n as u64) + .map(|i| make(i, (0..37).map(|_| next()).collect())) + .collect(); + recs.push(make(9_000, (0..20).map(|_| next()).collect())); + recs.push(make(9_001, Vec::new())); + recs.push(make(9_002, vec![0.0; 37])); + let refs: Vec<&MemoryRecord> = recs.iter().collect(); + for _ in 0..5 { + let q: Vec = (0..37).map(|_| next()).collect(); + let expected = (1.0 + - refs + .iter() + .map(|r| ImportanceScorer::cosine_similarity(&q, &r.embedding)) + .fold(f32::NEG_INFINITY, f32::max)) + .clamp(0.0, 1.0); + let got = ImportanceScorer::score_surprise(&q, &refs); + assert!((got - expected).abs() < 1e-5, "n={n}: {got} vs {expected}"); + } + } + assert_eq!( + ImportanceScorer::score_surprise(&[0.0; 4], &[&make(1, vec![1.0; 4])]), + 1.0 + ); + } + // Helper: build a simple normalised embedding of given dimension. fn unit_vec(dim: usize, hot: usize) -> Vec { let mut v = vec![0.0f32; dim]; diff --git a/crates/clawhdf5-bench/src/bin/consolidation_efficiency.rs b/crates/clawhdf5-bench/src/bin/consolidation_efficiency.rs index 629a5d6..5a1fd98 100644 --- a/crates/clawhdf5-bench/src/bin/consolidation_efficiency.rs +++ b/crates/clawhdf5-bench/src/bin/consolidation_efficiency.rs @@ -396,7 +396,7 @@ fn run_memory_reduction_benchmark() { println!(); println!( "{:>8} {:>10} {:>10} {:>10} {:>12}", - "Initial", "Remaining", "Eviction%", "Signal OK?", "BM25 Speedup" + "Initial", "Remaining", "Eviction%", "Signal OK?", "Records ÷" ); println!("{}", "-".repeat(58)); @@ -440,7 +440,8 @@ fn run_memory_reduction_benchmark() { // Check all signal records survived let signal_survived = signal_ids.iter().all(|&id| engine.get_by_id(id).is_some()); - // Rough speedup: BM25 scales roughly linearly with record count + // How many times fewer records there are. Not a measured speedup — + // Part 1 measures search latency before and after. let speedup = before_count as f64 / after_count.max(1) as f64; println!( @@ -480,7 +481,7 @@ fn main() { println!(" 3. Reducing search latency proportional to record reduction"); println!(); println!( - "Cycle time scales sub-linearly: 100 records ~microseconds, 100K records ~tens of ms." + "Cycle time grows a little faster than linearly: 100 records ~microseconds, 100K records ~tens of ms." ); println!("Signal records with Correction source + high access_count survive eviction."); }