Omar SobhandClaude Opus 5
0a2bd6f868
A third kind of decision-maker between deterministic code and a full LLM call: Choice / Score / Noul questions answered as probability distributions with a confidence, behind one Decider trait, with the composition patterns (confidence gating, composite scoring, rerank) as code. Two backends: TypeSafe's Jev over HTTP, and a DeBERTa-v3 MNLI cross-encoder run in-process with candle (feature nli; metal/cuda). decide-eval measures a backend on labelled cases the way judge-eval measures the judge. eval/skill-triage.json: 20 mission tasks × 53 skills, 75 positives, hand-labelled. Measured 2026-09-21: lexical overlap AUROC 0.851 [email protected] 0.47 top-k 48/75 ECE 0.095 jev (named wording) AUROC 0.989 [email protected] 0.84 top-k 63/75 ECE 0.064 213 ms jev (plain wording) AUROC 0.970 [email protected] 0.66 top-k 52/75 nli mnli-base AUROC 0.790 [email protected] 0.28 top-k 38/75 ECE 0.263 1.5 s nli zeroshot-v2 AUROC 0.782 [email protected] 0.43 top-k 39/75 ECE 0.054 1.2 s The vendor's calibration claim survives our data; the local cross-encoder ranks below keyword overlap on either checkpoint or wording and is kept as the measured negative, not shipped. A local backend would need the logit-readout route over the fleet's 9B model — a separate spike. Shadow: one Jev call per phase launch (spawned, 10 s cap, silent without TYPESAFE_API_KEY) records a skill.triage event; the Skill-Use report carries triage_p beside each skill's Trigger verdict. It selects nothing. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
89 lines
2.8 KiB
Rust
89 lines
2.8 KiB
Rust
//! The composition rules, as code. Each is one of the vendor's documented
|
|
//! patterns, kept here because they are the right way to use ANY calibrated
|
|
//! classifier and should not live in one call site's `if` chain.
|
|
|
|
use crate::Answer;
|
|
|
|
/// Confidence-gated routing: three outcomes, not two. The answer says what;
|
|
/// the confidence (or a Noul's distance from even) says whether to act.
|
|
///
|
|
/// `act_above` is the probability/confidence at which code may act on its
|
|
/// own; `dismiss_below` the one under which the answer is a clear no. In
|
|
/// between is the band a person, or a slower model, gets to decide.
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
|
pub enum Gate {
|
|
Act,
|
|
Review,
|
|
Dismiss,
|
|
}
|
|
|
|
pub fn gate_noul(p: f64, dismiss_below: f64, act_above: f64) -> Gate {
|
|
debug_assert!(dismiss_below <= act_above);
|
|
if p >= act_above {
|
|
Gate::Act
|
|
} else if p < dismiss_below {
|
|
Gate::Dismiss
|
|
} else {
|
|
Gate::Review
|
|
}
|
|
}
|
|
|
|
/// Gate a Choice or Score on its confidence alone.
|
|
pub fn gate_confidence(answer: &Answer, review_below: f64) -> Gate {
|
|
if answer.confidence() >= review_below {
|
|
Gate::Act
|
|
} else {
|
|
Gate::Review
|
|
}
|
|
}
|
|
|
|
/// Composite scoring: normalise each Score to 0..1 by its top level and
|
|
/// combine with weights the caller owns. `parts` is `(score, levels, weight)`.
|
|
/// Weights need not sum to one; the result is divided by their sum.
|
|
pub fn composite(parts: &[(f64, usize, f64)]) -> f64 {
|
|
let total: f64 = parts.iter().map(|(_, _, w)| w).sum();
|
|
if total <= 0.0 {
|
|
return 0.0;
|
|
}
|
|
parts
|
|
.iter()
|
|
.map(|(score, levels, w)| {
|
|
let top = (*levels as f64 - 1.0).max(1.0);
|
|
(score / top).clamp(0.0, 1.0) * w
|
|
})
|
|
.sum::<f64>()
|
|
/ total
|
|
}
|
|
|
|
/// Rerank: order candidates by a per-candidate Noul, highest first.
|
|
pub fn rerank<T>(mut items: Vec<(T, f64)>) -> Vec<(T, f64)> {
|
|
items.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));
|
|
items
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
#[test]
|
|
fn a_noul_gates_three_ways() {
|
|
assert_eq!(gate_noul(0.95, 0.2, 0.8), Gate::Act);
|
|
assert_eq!(gate_noul(0.05, 0.2, 0.8), Gate::Dismiss);
|
|
assert_eq!(gate_noul(0.5, 0.2, 0.8), Gate::Review);
|
|
}
|
|
|
|
#[test]
|
|
fn composite_normalises_by_top_level_and_weights() {
|
|
// severity 2 of 0..2 at weight 0.6, frustration 1 of 0..2 at 0.3,
|
|
// report quality 3 of 0..3 at 0.1 → 0.6*1 + 0.3*0.5 + 0.1*1 = 0.85
|
|
let c = composite(&[(2.0, 3, 0.6), (1.0, 3, 0.3), (3.0, 4, 0.1)]);
|
|
assert!((c - 0.85).abs() < 1e-9);
|
|
}
|
|
|
|
#[test]
|
|
fn rerank_is_descending() {
|
|
let r = rerank(vec![("a", 0.2), ("b", 0.9), ("c", 0.5)]);
|
|
assert_eq!(r.iter().map(|x| x.0).collect::<Vec<_>>(), ["b", "c", "a"]);
|
|
}
|
|
}
|