docs: the saturated-score finding, and its fix verified on a fresh harvest

01a0c940 (old question): relevance 2.93-3.00, spread 0.07. 01a0c950 (kind+evidence): spread 2.98, a survey at 0.0 and a measured method paper at 2.98, 8 of 9 tagged. The research agent reading the old manifest had already called the score overscored, unprompted. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
2026-09-22 08:34:06 -05:00
parent 9628b26795
commit 4aafbca2c5
1 changed files with 24 additions and 0 deletions
@@ -434,6 +434,30 @@ existed, is filled per paper by a Choice over the mission's topics, plus
 a four-level `relevance` Score the ranking phase can start from; PORTICO's
 abstract scored 3.0 at 1.0.
 **The paper triage ran live, and the first thing it proved was its own
 defect** (`48606f2`). Mission `01a0c940`, 10 papers: 10 tagged, 10 scored,
 every answer confident — and the relevance scores were **2.93–3.00, a
 spread of 0.07** on a 4-level scale. The harvest runs the operator's own
 arXiv topic queries, so "is this relevant to an agent platform" is true by
 construction and the scale had no room. The research agent reading that
 manifest said so itself, unprompted, about BabelArena: *"`score: 2.98` —
 overscored. A score around 1.0–1.5 would be honest."*
 Measured on the same ten abstracts: actionability saturated too (0.20);
 **evidence strength** — position piece → measured with ablations — spread
 **1.64**. The manifest now carries `evidence` and `kind` (method /
 benchmark / measurement / survey / position, with the confidence of that
 call) and no relevance number. Verified on a fresh harvest (`01a0c950`,
 9 papers, new topics): **spread 2.98** — a survey at 0.0 labelled `survey`
 at 0.93, a measured method paper at 2.98 — and 8 of 9 tagged, the untagged
 one genuinely fitting neither topic.
 Generalised so it cannot recur quietly: `patterns::spread()` +
 `SATURATED_BELOW`, and `triage_papers` warns when a live harvest's own
 scores collapse. **A score that returns the same number for everything is
 a defect in the question, not a fact about the population** — and it looks
 exactly like a working feature: N confident numbers, no information.
 **Next for this tier:** accumulate skill-triage agreement rows from real
 missions before any promotion from shadow to selection; a local backend
 only if the vendor dependency bites (logit read-out over the 9B fleet