Complete the consolidation benchmark: cheaper novelty scoring #7

2026-09-25T14:53:11Z

osobh commented

2026-09-25 14:53:11 +00:00

Summary

This completes the one BENCHMARKS.md section the dated re-run (#6) couldn't finish, and fixes what made it stall.

Why it stalled. consolidation_efficiency was stopped after 19 minutes on one core, still building its 100K case. The consolidation cycle wasn't the problem; it's linear. The setup was: every add_memory scores the new record's novelty against the whole working tier. The benchmark lets that tier grow to 50K records, and every comparison recomputed both norms, which comes to roughly 5×10⁹ comparisons at three passes each.

The fix. ImportanceScorer::score_surprise now:

computes the new record's norm once,
takes each comparison in a single fused, vectorised pass (dot product and the other norm together),
splits a working tier of 4,096 or more records across threads, with the parallel feature.

Results are unchanged; a new test checks them against the old cosine formula, including shorter, empty and zero vectors and the parallel path. The work is still quadratic in the working-tier size by design. With regular consolidation the tier stays near working_capacity (100), so inserts stay cheap.

Results (tank, complete run, 8 min 10 s):

Consolidation cycle	Time
100 / 1K / 10K records	19 µs / 207 µs / 2.81 ms
100K records	46.66 ms (never published before)

The memory-reduction table is published for the first time: 80% of records evicted at 100, 1K and 10K, with every signal record kept.

Unmeasured claims removed from the binary: a "BM25 Speedup" column that was only a ratio of record counts, and "cycle time scales sub-linearly", which its own numbers contradict.

Test plan

score_surprise_matches_the_reference_cosine (includes the parallel path), 17 consolidation tests; agent suite passes, with and without default features
clippy -D warnings on the agent and bench crates
consolidation_efficiency runs to completion
Gitea CI on this PR

Note: one agent test failed once while the machine was heavily loaded, then passed 23 runs in a row (three under the same load). I couldn't identify or reproduce it. The new test is deterministic, so it isn't the cause.

🤖 Generated with Claude Code

## Summary This completes the one BENCHMARKS.md section the dated re-run (#6) couldn't finish, and fixes what made it stall. **Why it stalled.** `consolidation_efficiency` was stopped after 19 minutes on one core, still building its 100K case. The consolidation cycle wasn't the problem; it's linear. The setup was: every `add_memory` scores the new record's novelty against the whole working tier. The benchmark lets that tier grow to 50K records, and every comparison recomputed both norms, which comes to roughly 5×10⁹ comparisons at three passes each. **The fix.** `ImportanceScorer::score_surprise` now: - computes the new record's norm once, - takes each comparison in a single fused, vectorised pass (dot product and the other norm together), - splits a working tier of 4,096 or more records across threads, with the `parallel` feature. Results are unchanged; a new test checks them against the old cosine formula, including shorter, empty and zero vectors and the parallel path. The work is still quadratic in the working-tier size by design. With regular consolidation the tier stays near `working_capacity` (100), so inserts stay cheap. **Results** (tank, complete run, 8 min 10 s): | Consolidation cycle | Time | |---|---:| | 100 / 1K / 10K records | 19 µs / 207 µs / 2.81 ms | | **100K records** | **46.66 ms** (never published before) | The memory-reduction table is published for the first time: 80% of records evicted at 100, 1K and 10K, with every signal record kept. **Unmeasured claims removed from the binary:** a "BM25 Speedup" column that was only a ratio of record counts, and "cycle time scales sub-linearly", which its own numbers contradict. ## Test plan - [x] `score_surprise_matches_the_reference_cosine` (includes the parallel path), 17 consolidation tests; agent suite passes, with and without default features - [x] clippy `-D warnings` on the agent and bench crates - [x] `consolidation_efficiency` runs to completion - [ ] Gitea CI on this PR Note: one agent test failed once while the machine was heavily loaded, then passed 23 runs in a row (three under the same load). I couldn't identify or reproduce it. The new test is deterministic, so it isn't the cause. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

osobh added 1 commit 2026-09-25 14:53:12 +00:00

perf(agent): cheaper novelty scoring; complete the consolidation benchmark

CI / test-arm64 (pull_request) Successful in 1m5s

Details

CI / test (pull_request) Successful in 5m39s

Details

00b0cb0035

consolidation_efficiency never finished: stopped after 19 minutes on one
core while building its 100K case. Not the consolidation cycle (linear:
17 us at 100 records, 2.16 ms at 10K) but the setup — every add_memory
scores the new record's novelty against the whole working tier, the
benchmark lets that tier reach 50K, and each comparison recomputed both
norms: ~5e9 comparisons of three passes each.

ImportanceScorer::score_surprise now computes the new record's norm
once, takes each comparison in one fused, 8-lane pass (dot product and
the other norm together), and splits a working tier of 4096+ records
across threads with the `parallel` feature. Same results: tested against
the old cosine formula, including shorter, empty and zero vectors and
the parallel path. The work stays quadratic in the working-tier size by
design; with regular consolidation the tier stays near
working_capacity (100) and inserts are cheap.

The complete run takes 8 min 10 s on tank and fills in the 100K cycle
row (46.66 ms) and the memory-reduction table, which had never been
published. The binary no longer prints a record-count ratio as a
"BM25 Speedup" (never measured; Part 1 measures search latency) or
claims sub-linear cycle scaling (its own numbers grow slightly faster
than linearly).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

osobh merged commit 7706697feb into main

2026-09-25 15:05:59 +00:00

osobh deleted branch feat/consolidation-scaling

2026-09-25 15:06:02 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: quantumclaw/clawhdf5#7