fix(triage): a relevance score on a pre-filtered corpus ranks nothing — measure evidence instead
Mission 01a0c940 ran the paper triage live for the first time: 10 papers, 10 tagged, 10 scored — and the relevance scores were 2.93, 2.94, 2.97, 2.97, 2.98, 2.99, 2.99, 2.99, 3.00, 3.00. A spread of 0.07 across a 4-level scale, every answer confident, no ranking information at all. Of course: the harvest runs the operator's own arXiv topic queries, so every paper in the file is about agents by construction. Asked on the same ten abstracts, 'how actionable is it' saturated the same way (spread 0.20). What separated them was the strength of the evidence behind the claims: 1.36 (a benchmark paper) to 3.00 (measured on real systems with ablations), spread 1.64 — and what KIND of paper it is (method / benchmark / measurement / survey / position), with the confidence of that call beside it so an unplaceable paper reads as unplaceable. The manifest now carries those two and no relevance number, and arxiv-daily.md tells the reading agents what each means and why there is no relevance. The general rule, since this class of mistake is invisible — a saturated score looks exactly like a working feature: patterns::spread() with SATURATED_BELOW, and triage_papers warns when a live harvest's scores span less than that. A question that returns the same number for everything is a defect in the question, not a fact about the population. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
This commit is contained in:
co-authored by
Claude Opus 5
parent
3b6dd3970d
commit
48606f2fe3
@@ -16,9 +16,21 @@ Your input is the result:
|
||||
|
||||
```
|
||||
ContinuousResearch/<date>/harvest.jsonl
|
||||
{ source, url, title, snippet, first_seen, topic_tags }
|
||||
{ source, url, title, snippet, first_seen, topic_tags, kind, evidence }
|
||||
```
|
||||
|
||||
`topic_tags`, `kind` and `evidence` are filled by a decision model before you
|
||||
see the file. `kind.is` is one of benchmark / method / measurement / survey /
|
||||
position, with the confidence of that call beside it — a low confidence means
|
||||
the paper does not sit cleanly in one, so read before you trust the label.
|
||||
`evidence.score` runs 0 (a position piece with no experiments) to 3 (measured
|
||||
on real systems with ablations); it is the one dimension that separates papers
|
||||
in a harvest, because every paper in the file already matched your topics.
|
||||
There is deliberately no "relevance" number: measured on a real harvest it
|
||||
came back 2.93–3.00 for every paper, which ranks nothing. All three keys are
|
||||
absent or empty when the triage could not run; that is not a signal about the
|
||||
paper.
|
||||
|
||||
One line per paper that is **new since the last run**. Papers already covered
|
||||
are not in it — by design, not omission.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user