Omar SobhandClaude Opus 5
0a2bd6f868
A third kind of decision-maker between deterministic code and a full LLM call: Choice / Score / Noul questions answered as probability distributions with a confidence, behind one Decider trait, with the composition patterns (confidence gating, composite scoring, rerank) as code. Two backends: TypeSafe's Jev over HTTP, and a DeBERTa-v3 MNLI cross-encoder run in-process with candle (feature nli; metal/cuda). decide-eval measures a backend on labelled cases the way judge-eval measures the judge. eval/skill-triage.json: 20 mission tasks × 53 skills, 75 positives, hand-labelled. Measured 2026-09-21: lexical overlap AUROC 0.851 [email protected] 0.47 top-k 48/75 ECE 0.095 jev (named wording) AUROC 0.989 [email protected] 0.84 top-k 63/75 ECE 0.064 213 ms jev (plain wording) AUROC 0.970 [email protected] 0.66 top-k 52/75 nli mnli-base AUROC 0.790 [email protected] 0.28 top-k 38/75 ECE 0.263 1.5 s nli zeroshot-v2 AUROC 0.782 [email protected] 0.43 top-k 39/75 ECE 0.054 1.2 s The vendor's calibration claim survives our data; the local cross-encoder ranks below keyword overlap on either checkpoint or wording and is kept as the measured negative, not shipped. A local backend would need the logit-readout route over the fleet's 9B model — a separate spike. Shadow: one Jev call per phase launch (spawned, 10 s cap, silent without TYPESAFE_API_KEY) records a skill.triage event; the Skill-Use report carries triage_p beside each skill's Trigger verdict. It selects nothing. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
23 lines
907 B
Rust
23 lines
907 B
Rust
//! The skill-triage question, in one place.
|
||
//!
|
||
//! Measured 2026-09-21 on `eval/skill-triage.json` (20 tasks × 53 skills):
|
||
//! this wording — the skill's NAME plus its `when_to_use` line — scored
|
||
//! AUROC 0.989 / F1 0.84 on Jev against 0.970 / 0.66 for the when_to_use
|
||
//! line alone. The name carries signal. The server's shadow path and the
|
||
//! eval both call this, so what is measured is what runs.
|
||
|
||
use crate::Question;
|
||
|
||
pub const WORDING: &str = "named";
|
||
|
||
pub fn question(name: &str, when_to_use: &str) -> Question {
|
||
Question::noul(format!(
|
||
"This task calls for the skill \"{name}\", which applies when: {when_to_use}"
|
||
))
|
||
}
|
||
|
||
/// The probability at which a skill counts as "applies" when the triage is
|
||
/// read back. 0.5 is where Jev's best F1 sat (0.84 at 0.51); it is not a
|
||
/// gate on anything yet — shadow mode records, it does not select.
|
||
pub const APPLIES_AT: f64 = 0.5;
|