fix(triage): a relevance score on a pre-filtered corpus ranks nothing — measure evidence instead

Mission 01a0c940 ran the paper triage live for the first time: 10 papers,
10 tagged, 10 scored — and the relevance scores were 2.93, 2.94, 2.97,
2.97, 2.98, 2.99, 2.99, 2.99, 3.00, 3.00. A spread of 0.07 across a
4-level scale, every answer confident, no ranking information at all. Of
course: the harvest runs the operator's own arXiv topic queries, so every
paper in the file is about agents by construction. Asked on the same ten
abstracts, 'how actionable is it' saturated the same way (spread 0.20).

What separated them was the strength of the evidence behind the claims:
1.36 (a benchmark paper) to 3.00 (measured on real systems with
ablations), spread 1.64 — and what KIND of paper it is (method /
benchmark / measurement / survey / position), with the confidence of that
call beside it so an unplaceable paper reads as unplaceable. The manifest
now carries those two and no relevance number, and arxiv-daily.md tells
the reading agents what each means and why there is no relevance.

The general rule, since this class of mistake is invisible — a saturated
score looks exactly like a working feature: patterns::spread() with
SATURATED_BELOW, and triage_papers warns when a live harvest's scores
span less than that. A question that returns the same number for
everything is a defect in the question, not a fact about the population.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz

crates/cm-decide/src/patterns.rs

@@ -55,6 +55,28 @@ pub fn composite(parts: &[(f64, usize, f64)]) -> f64 {
         / total
 }
 
+/// How much a Score actually separated a population: `max - min`, in
+/// levels. A dimension that returns the same score for everything ranks
+/// nothing, however confident each answer is — measured on a real harvest,
+/// "how relevant is this paper" spanned 0.07 of a 3-level scale because the
+/// corpus was selected to be relevant, while "how strong is the evidence"
+/// spanned 1.64 on the same ten papers. Callers that rank should report
+/// this and say so when it collapses; see `SATURATED_BELOW`.
+pub fn spread(scores: &[f64]) -> f64 {
+    match (
+        scores.iter().cloned().fold(f64::INFINITY, f64::min),
+        scores.iter().cloned().fold(f64::NEG_INFINITY, f64::max),
+    ) {
+        (lo, hi) if lo.is_finite() && hi.is_finite() => hi - lo,
+        _ => 0.0,
+    }
+}
+
+/// A spread under this, on a population of more than a couple of items, is
+/// a question that is not discriminating: act on it as a defect in the
+/// question, not as a fact about the population.
+pub const SATURATED_BELOW: f64 = 0.5;
+
 /// Rerank: order candidates by a per-candidate Noul, highest first.
 pub fn rerank<T>(mut items: Vec<(T, f64)>) -> Vec<(T, f64)> {
     items.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));
@@ -80,6 +102,18 @@ mod tests {
         assert!((c - 0.85).abs() < 1e-9);
     }
 
+    #[test]
+    fn spread_reports_what_a_score_separated() {
+        // The measured numbers: relevance on ten harvested papers, then
+        // evidence on the same ten.
+        let relevance = [3.0, 2.99, 2.99, 2.98, 2.97, 2.97, 2.94, 3.0, 2.99, 3.0];
+        let evidence = [3.0, 2.93, 2.86, 2.85, 2.82, 2.59, 2.88, 2.14, 2.16, 1.36];
+        assert!(spread(&relevance) < SATURATED_BELOW, "{}", spread(&relevance));
+        assert!(spread(&evidence) > SATURATED_BELOW, "{}", spread(&evidence));
+        assert_eq!(spread(&[]), 0.0);
+        assert_eq!(spread(&[1.5]), 0.0);
+    }
+
     #[test]
     fn rerank_is_descending() {
         let r = rerank(vec![("a", 0.2), ("b", 0.9), ("c", 0.5)]);