Commit Graph
4 Commits
Author SHA1 Message Date
osobhandClaude Opus 5.5 00b0cb0035 perf(agent): cheaper novelty scoring; complete the consolidation benchmark
CI / test-arm64 (pull_request) Successful in 1m5s
CI / test (pull_request) Successful in 5m39s
consolidation_efficiency never finished: stopped after 19 minutes on one
core while building its 100K case. Not the consolidation cycle (linear:
17 us at 100 records, 2.16 ms at 10K) but the setup — every add_memory
scores the new record's novelty against the whole working tier, the
benchmark lets that tier reach 50K, and each comparison recomputed both
norms: ~5e9 comparisons of three passes each.

ImportanceScorer::score_surprise now computes the new record's norm
once, takes each comparison in one fused, 8-lane pass (dot product and
the other norm together), and splits a working tier of 4096+ records
across threads with the `parallel` feature. Same results: tested against
the old cosine formula, including shorter, empty and zero vectors and
the parallel path. The work stays quadratic in the working-tier size by
design; with regular consolidation the tier stays near
working_capacity (100) and inserts are cheap.

The complete run takes 8 min 10 s on tank and fills in the 100K cycle
row (46.66 ms) and the memory-reduction table, which had never been
published. The binary no longer prints a record-count ratio as a
"BM25 Speedup" (never measured; Part 1 measures search latency) or
claims sub-linear cycle scaling (its own numbers grow slightly faster
than linearly).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-25 08:51:08 -05:00
osobhandClaude Fable 5.1 a3f7c6fe89 style: cargo fmt --all
Formatting only. cargo fmt --check was already failing on main (accel SIMD
kernels, agent, format, migrate, bench); CI now enforces it.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-09-19 05:36:23 -07:00
ClawHDF5 Coding Agent 3c7c229e20 security(agent): gate elevated MemorySource construction behind a distinct API
Two related trust-boundary gaps, both closed:

1. ConsolidationEngine::add_memory took a plain `source: MemorySource`
   parameter, so any caller could claim MemorySource::System/Correction —
   which get elevated importance weighting in score_correction — for
   content whose actual origin the caller doesn't control or hasn't
   verified. Split into add_memory(UntrustedSource) for ordinary
   caller-supplied content (User/Tool/Retrieval only, no elevated variant
   exists to claim) and add_trusted_memory(TrustedSource) for content whose
   elevated trust the caller has independently verified (System/
   Correction). Updated the one production consumer outside this crate
   (clawhdf5-bench's consolidation_efficiency benchmark) and all tests.

2. The provenance/anomaly wiring added in the previous commit introduced
   the same pattern: infer_memory_source mapped source_channel == "system"
   or "correction" straight to the elevated MemorySource variants. Since
   MemoryEntry.source_channel is unvalidated caller-supplied text, this let
   a write dodge check_source_anomaly's User-flood detection by simply
   self-labeling source_channel = "system". infer_memory_source now never
   returns System/Correction — only Tool/Retrieval (recognized channel
   names) or User (everything else, the conservative default).

INT-05
2026-08-17 00:49:34 +00:00
redclawsystems 3f222f6956 Merge pull request 'docs(clawhdf5): document DType variants, fix unresolved doc links' (#17) from sdlc-docs/clawhdf5-types-20260514-165210 into main 2026-05-14 23:54:48 +00:00