Mission 01a0c940 ran the paper triage live for the first time: 10 papers, 10 tagged, 10 scored — and the relevance scores were 2.93, 2.94, 2.97, 2.97, 2.98, 2.99, 2.99, 2.99, 3.00, 3.00. A spread of 0.07 across a 4-level scale, every answer confident, no ranking information at all. Of course: the harvest runs the operator's own arXiv topic queries, so every paper in the file is about agents by construction. Asked on the same ten abstracts, 'how actionable is it' saturated the same way (spread 0.20). What separated them was the strength of the evidence behind the claims: 1.36 (a benchmark paper) to 3.00 (measured on real systems with ablations), spread 1.64 — and what KIND of paper it is (method / benchmark / measurement / survey / position), with the confidence of that call beside it so an unplaceable paper reads as unplaceable. The manifest now carries those two and no relevance number, and arxiv-daily.md tells the reading agents what each means and why there is no relevance. The general rule, since this class of mistake is invisible — a saturated score looks exactly like a working feature: patterns::spread() with SATURATED_BELOW, and triage_papers warns when a live harvest's scores span less than that. A question that returns the same number for everything is a defect in the question, not a fact about the population. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
123 lines
4.4 KiB
Rust
123 lines
4.4 KiB
Rust
//! The composition rules, as code. Each is one of the vendor's documented
|
|
//! patterns, kept here because they are the right way to use ANY calibrated
|
|
//! classifier and should not live in one call site's `if` chain.
|
|
|
|
use crate::Answer;
|
|
|
|
/// Confidence-gated routing: three outcomes, not two. The answer says what;
|
|
/// the confidence (or a Noul's distance from even) says whether to act.
|
|
///
|
|
/// `act_above` is the probability/confidence at which code may act on its
|
|
/// own; `dismiss_below` the one under which the answer is a clear no. In
|
|
/// between is the band a person, or a slower model, gets to decide.
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
|
pub enum Gate {
|
|
Act,
|
|
Review,
|
|
Dismiss,
|
|
}
|
|
|
|
pub fn gate_noul(p: f64, dismiss_below: f64, act_above: f64) -> Gate {
|
|
debug_assert!(dismiss_below <= act_above);
|
|
if p >= act_above {
|
|
Gate::Act
|
|
} else if p < dismiss_below {
|
|
Gate::Dismiss
|
|
} else {
|
|
Gate::Review
|
|
}
|
|
}
|
|
|
|
/// Gate a Choice or Score on its confidence alone.
|
|
pub fn gate_confidence(answer: &Answer, review_below: f64) -> Gate {
|
|
if answer.confidence() >= review_below {
|
|
Gate::Act
|
|
} else {
|
|
Gate::Review
|
|
}
|
|
}
|
|
|
|
/// Composite scoring: normalise each Score to 0..1 by its top level and
|
|
/// combine with weights the caller owns. `parts` is `(score, levels, weight)`.
|
|
/// Weights need not sum to one; the result is divided by their sum.
|
|
pub fn composite(parts: &[(f64, usize, f64)]) -> f64 {
|
|
let total: f64 = parts.iter().map(|(_, _, w)| w).sum();
|
|
if total <= 0.0 {
|
|
return 0.0;
|
|
}
|
|
parts
|
|
.iter()
|
|
.map(|(score, levels, w)| {
|
|
let top = (*levels as f64 - 1.0).max(1.0);
|
|
(score / top).clamp(0.0, 1.0) * w
|
|
})
|
|
.sum::<f64>()
|
|
/ total
|
|
}
|
|
|
|
/// How much a Score actually separated a population: `max - min`, in
|
|
/// levels. A dimension that returns the same score for everything ranks
|
|
/// nothing, however confident each answer is — measured on a real harvest,
|
|
/// "how relevant is this paper" spanned 0.07 of a 3-level scale because the
|
|
/// corpus was selected to be relevant, while "how strong is the evidence"
|
|
/// spanned 1.64 on the same ten papers. Callers that rank should report
|
|
/// this and say so when it collapses; see `SATURATED_BELOW`.
|
|
pub fn spread(scores: &[f64]) -> f64 {
|
|
match (
|
|
scores.iter().cloned().fold(f64::INFINITY, f64::min),
|
|
scores.iter().cloned().fold(f64::NEG_INFINITY, f64::max),
|
|
) {
|
|
(lo, hi) if lo.is_finite() && hi.is_finite() => hi - lo,
|
|
_ => 0.0,
|
|
}
|
|
}
|
|
|
|
/// A spread under this, on a population of more than a couple of items, is
|
|
/// a question that is not discriminating: act on it as a defect in the
|
|
/// question, not as a fact about the population.
|
|
pub const SATURATED_BELOW: f64 = 0.5;
|
|
|
|
/// Rerank: order candidates by a per-candidate Noul, highest first.
|
|
pub fn rerank<T>(mut items: Vec<(T, f64)>) -> Vec<(T, f64)> {
|
|
items.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));
|
|
items
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
#[test]
|
|
fn a_noul_gates_three_ways() {
|
|
assert_eq!(gate_noul(0.95, 0.2, 0.8), Gate::Act);
|
|
assert_eq!(gate_noul(0.05, 0.2, 0.8), Gate::Dismiss);
|
|
assert_eq!(gate_noul(0.5, 0.2, 0.8), Gate::Review);
|
|
}
|
|
|
|
#[test]
|
|
fn composite_normalises_by_top_level_and_weights() {
|
|
// severity 2 of 0..2 at weight 0.6, frustration 1 of 0..2 at 0.3,
|
|
// report quality 3 of 0..3 at 0.1 → 0.6*1 + 0.3*0.5 + 0.1*1 = 0.85
|
|
let c = composite(&[(2.0, 3, 0.6), (1.0, 3, 0.3), (3.0, 4, 0.1)]);
|
|
assert!((c - 0.85).abs() < 1e-9);
|
|
}
|
|
|
|
#[test]
|
|
fn spread_reports_what_a_score_separated() {
|
|
// The measured numbers: relevance on ten harvested papers, then
|
|
// evidence on the same ten.
|
|
let relevance = [3.0, 2.99, 2.99, 2.98, 2.97, 2.97, 2.94, 3.0, 2.99, 3.0];
|
|
let evidence = [3.0, 2.93, 2.86, 2.85, 2.82, 2.59, 2.88, 2.14, 2.16, 1.36];
|
|
assert!(spread(&relevance) < SATURATED_BELOW, "{}", spread(&relevance));
|
|
assert!(spread(&evidence) > SATURATED_BELOW, "{}", spread(&evidence));
|
|
assert_eq!(spread(&[]), 0.0);
|
|
assert_eq!(spread(&[1.5]), 0.0);
|
|
}
|
|
|
|
#[test]
|
|
fn rerank_is_descending() {
|
|
let r = rerank(vec![("a", 0.2), ("b", 0.9), ("c", 0.5)]);
|
|
assert_eq!(r.iter().map(|x| x.0).collect::<Vec<_>>(), ["b", "c", "a"]);
|
|
}
|
|
}
|