fix(triage): a relevance score on a pre-filtered corpus ranks nothing — measure evidence instead
deploy / test (push) Successful in 5m30s
deploy / build (push) Successful in 5m57s

Mission 01a0c940 ran the paper triage live for the first time: 10 papers,
10 tagged, 10 scored — and the relevance scores were 2.93, 2.94, 2.97,
2.97, 2.98, 2.99, 2.99, 2.99, 3.00, 3.00. A spread of 0.07 across a
4-level scale, every answer confident, no ranking information at all. Of
course: the harvest runs the operator's own arXiv topic queries, so every
paper in the file is about agents by construction. Asked on the same ten
abstracts, 'how actionable is it' saturated the same way (spread 0.20).

What separated them was the strength of the evidence behind the claims:
1.36 (a benchmark paper) to 3.00 (measured on real systems with
ablations), spread 1.64 — and what KIND of paper it is (method /
benchmark / measurement / survey / position), with the confidence of that
call beside it so an unplaceable paper reads as unplaceable. The manifest
now carries those two and no relevance number, and arxiv-daily.md tells
the reading agents what each means and why there is no relevance.

The general rule, since this class of mistake is invisible — a saturated
score looks exactly like a working feature: patterns::spread() with
SATURATED_BELOW, and triage_papers warns when a live harvest's scores
span less than that. A question that returns the same number for
everything is a defect in the question, not a fact about the population.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
This commit is contained in:
Omar Sobh
2026-09-22 08:19:41 -05:00
co-authored by Claude Opus 5
parent 3b6dd3970d
commit 48606f2fe3
3 changed files with 136 additions and 37 deletions
+13 -1
View File
@@ -16,9 +16,21 @@ Your input is the result:
```
ContinuousResearch/<date>/harvest.jsonl
{ source, url, title, snippet, first_seen, topic_tags }
{ source, url, title, snippet, first_seen, topic_tags, kind, evidence }
```
`topic_tags`, `kind` and `evidence` are filled by a decision model before you
see the file. `kind.is` is one of benchmark / method / measurement / survey /
position, with the confidence of that call beside it — a low confidence means
the paper does not sit cleanly in one, so read before you trust the label.
`evidence.score` runs 0 (a position piece with no experiments) to 3 (measured
on real systems with ablations); it is the one dimension that separates papers
in a harvest, because every paper in the file already matched your topics.
There is deliberately no "relevance" number: measured on a real harvest it
came back 2.933.00 for every paper, which ranks nothing. All three keys are
absent or empty when the triage could not run; that is not a signal about the
paper.
One line per paper that is **new since the last run**. Papers already covered
are not in it — by design, not omission.