perf(agent): cheaper novelty scoring; complete the consolidation benchmark
CI / test-arm64 (pull_request) Successful in 1m5s
CI / test (pull_request) Successful in 5m39s

consolidation_efficiency never finished: stopped after 19 minutes on one
core while building its 100K case. Not the consolidation cycle (linear:
17 us at 100 records, 2.16 ms at 10K) but the setup — every add_memory
scores the new record's novelty against the whole working tier, the
benchmark lets that tier reach 50K, and each comparison recomputed both
norms: ~5e9 comparisons of three passes each.

ImportanceScorer::score_surprise now computes the new record's norm
once, takes each comparison in one fused, 8-lane pass (dot product and
the other norm together), and splits a working tier of 4096+ records
across threads with the `parallel` feature. Same results: tested against
the old cosine formula, including shorter, empty and zero vectors and
the parallel path. The work stays quadratic in the working-tier size by
design; with regular consolidation the tier stays near
working_capacity (100) and inserts are cheap.

The complete run takes 8 min 10 s on tank and fills in the 100K cycle
row (46.66 ms) and the memory-reduction table, which had never been
published. The binary no longer prints a record-count ratio as a
"BM25 Speedup" (never measured; Part 1 measures search latency) or
claims sub-linear cycle scaling (its own numbers grow slightly faster
than linearly).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-25 08:51:08 -05:00
co-authored by Claude Opus 5.5
parent dce5559ff2
commit 00b0cb0035
5 changed files with 184 additions and 18 deletions
+10
View File
@@ -189,6 +189,16 @@
knew to ask; it now only ever switches the default off.
### Performance
- `clawhdf5-agent`: consolidation's novelty scoring (each `add_memory` against
the whole working tier) computes the new record's norm once, takes each
comparison in one vectorised pass instead of three, and splits a working
tier of 4 096+ records across threads — same results, tested against the
old formula. It had made `consolidation_efficiency` stall at 100K; the
complete run now takes 8 min and fills in the 100K cycle row (46.66 ms) and
the memory-reduction table.
- `clawhdf5-bench`: `consolidation_efficiency` no longer prints a record-count
ratio as a "BM25 Speedup" (it was never measured), nor claims cycle time
grows sub-linearly (its own numbers grow slightly faster than linearly).
- `clawhdf5-agent`: **knowledge-graph traversal was 6.5x slower than it
should be.** `bfs_neighbors` and `spreading_activation` built an adjacency
index over the whole graph on every call (1efd82c), so a 2-hop BFS over 1K