Mission 01a0c940 ran the paper triage live for the first time: 10 papers,
10 tagged, 10 scored — and the relevance scores were 2.93, 2.94, 2.97,
2.97, 2.98, 2.99, 2.99, 2.99, 3.00, 3.00. A spread of 0.07 across a
4-level scale, every answer confident, no ranking information at all. Of
course: the harvest runs the operator's own arXiv topic queries, so every
paper in the file is about agents by construction. Asked on the same ten
abstracts, 'how actionable is it' saturated the same way (spread 0.20).
What separated them was the strength of the evidence behind the claims:
1.36 (a benchmark paper) to 3.00 (measured on real systems with
ablations), spread 1.64 — and what KIND of paper it is (method /
benchmark / measurement / survey / position), with the confidence of that
call beside it so an unplaceable paper reads as unplaceable. The manifest
now carries those two and no relevance number, and arxiv-daily.md tells
the reading agents what each means and why there is no relevance.
The general rule, since this class of mistake is invisible — a saturated
score looks exactly like a working feature: patterns::spread() with
SATURATED_BELOW, and triage_papers warns when a live harvest's scores
span less than that. A question that returns the same number for
everything is a defect in the question, not a fact about the population.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
A third kind of decision-maker between deterministic code and a full LLM
call: Choice / Score / Noul questions answered as probability
distributions with a confidence, behind one Decider trait, with the
composition patterns (confidence gating, composite scoring, rerank) as
code. Two backends: TypeSafe's Jev over HTTP, and a DeBERTa-v3 MNLI
cross-encoder run in-process with candle (feature nli; metal/cuda).
decide-eval measures a backend on labelled cases the way judge-eval
measures the judge. eval/skill-triage.json: 20 mission tasks × 53 skills,
75 positives, hand-labelled. Measured 2026-09-21:
lexical overlap AUROC 0.851 [email protected] 0.47 top-k 48/75 ECE 0.095
jev (named wording) AUROC 0.989 [email protected] 0.84 top-k 63/75 ECE 0.064 213 ms
jev (plain wording) AUROC 0.970 [email protected] 0.66 top-k 52/75
nli mnli-base AUROC 0.790 [email protected] 0.28 top-k 38/75 ECE 0.263 1.5 s
nli zeroshot-v2 AUROC 0.782 [email protected] 0.43 top-k 39/75 ECE 0.054 1.2 s
The vendor's calibration claim survives our data; the local cross-encoder
ranks below keyword overlap on either checkpoint or wording and is kept
as the measured negative, not shipped. A local backend would need the
logit-readout route over the fleet's 9B model — a separate spike.
Shadow: one Jev call per phase launch (spawned, 10 s cap, silent without
TYPESAFE_API_KEY) records a skill.triage event; the Skill-Use report
carries triage_p beside each skill's Trigger verdict. It selects nothing.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
