perf(agent): cheaper novelty scoring; complete the consolidation benchmark
consolidation_efficiency never finished: stopped after 19 minutes on one core while building its 100K case. Not the consolidation cycle (linear: 17 us at 100 records, 2.16 ms at 10K) but the setup — every add_memory scores the new record's novelty against the whole working tier, the benchmark lets that tier reach 50K, and each comparison recomputed both norms: ~5e9 comparisons of three passes each. ImportanceScorer::score_surprise now computes the new record's norm once, takes each comparison in one fused, 8-lane pass (dot product and the other norm together), and splits a working tier of 4096+ records across threads with the `parallel` feature. Same results: tested against the old cosine formula, including shorter, empty and zero vectors and the parallel path. The work stays quadratic in the working-tier size by design; with regular consolidation the tier stays near working_capacity (100) and inserts are cheap. The complete run takes 8 min 10 s on tank and fills in the 100K cycle row (46.66 ms) and the memory-reduction table, which had never been published. The binary no longer prints a record-count ratio as a "BM25 Speedup" (never measured; Part 1 measures search latency) or claims sub-linear cycle scaling (its own numbers grow slightly faster than linearly). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -189,6 +189,16 @@
|
||||
knew to ask; it now only ever switches the default off.
|
||||
|
||||
### Performance
|
||||
- `clawhdf5-agent`: consolidation's novelty scoring (each `add_memory` against
|
||||
the whole working tier) computes the new record's norm once, takes each
|
||||
comparison in one vectorised pass instead of three, and splits a working
|
||||
tier of 4 096+ records across threads — same results, tested against the
|
||||
old formula. It had made `consolidation_efficiency` stall at 100K; the
|
||||
complete run now takes 8 min and fills in the 100K cycle row (46.66 ms) and
|
||||
the memory-reduction table.
|
||||
- `clawhdf5-bench`: `consolidation_efficiency` no longer prints a record-count
|
||||
ratio as a "BM25 Speedup" (it was never measured), nor claims cycle time
|
||||
grows sub-linearly (its own numbers grow slightly faster than linearly).
|
||||
- `clawhdf5-agent`: **knowledge-graph traversal was 6.5x slower than it
|
||||
should be.** `bfs_neighbors` and `spreading_activation` built an adjacency
|
||||
index over the whole graph on every call (1efd82c), so a 2-hop BFS over 1K
|
||||
|
||||
Reference in New Issue
Block a user