feat(decide): cm-decide — typed calibrated decisions; Jev + local NLI backends; skill triage in shadow
A third kind of decision-maker between deterministic code and a full LLM call: Choice / Score / Noul questions answered as probability distributions with a confidence, behind one Decider trait, with the composition patterns (confidence gating, composite scoring, rerank) as code. Two backends: TypeSafe's Jev over HTTP, and a DeBERTa-v3 MNLI cross-encoder run in-process with candle (feature nli; metal/cuda). decide-eval measures a backend on labelled cases the way judge-eval measures the judge. eval/skill-triage.json: 20 mission tasks × 53 skills, 75 positives, hand-labelled. Measured 2026-09-21: lexical overlap AUROC 0.851 [email protected] 0.47 top-k 48/75 ECE 0.095 jev (named wording) AUROC 0.989 [email protected] 0.84 top-k 63/75 ECE 0.064 213 ms jev (plain wording) AUROC 0.970 [email protected] 0.66 top-k 52/75 nli mnli-base AUROC 0.790 [email protected] 0.28 top-k 38/75 ECE 0.263 1.5 s nli zeroshot-v2 AUROC 0.782 [email protected] 0.43 top-k 39/75 ECE 0.054 1.2 s The vendor's calibration claim survives our data; the local cross-encoder ranks below keyword overlap on either checkpoint or wording and is kept as the measured negative, not shipped. A local backend would need the logit-readout route over the fleet's 9B model — a separate spike. Shadow: one Jev call per phase launch (spawned, 10 s cap, silent without TYPESAFE_API_KEY) records a skill.triage event; the Skill-Use report carries triage_p beside each skill's Trigger verdict. It selects nothing. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
This commit is contained in:
Omar Sobh
co-authored by
Claude Opus 5
Claude Opus 5
parent
650a556029
commit
0a2bd6f868
@@ -0,0 +1,348 @@
|
||||
//! Measure a decision backend against labelled cases.
|
||||
//!
|
||||
//! The judge got an eval before it got trusted (`scripts/judge-eval.sh`);
|
||||
//! a triage model gets the same. Every backend answers the SAME Noul per
|
||||
//! (task, skill) pair, and is scored on ranking (AUROC), on the operating
|
||||
//! point (best-F1 threshold and F1 at 0.5), and on calibration (Brier, ECE)
|
||||
//! — because a calibrated middle band is the whole reason to have this tier,
|
||||
//! and a backend that ranks well but says 0.9 to everything has none.
|
||||
//!
|
||||
//! decide-eval [--backend jev|nli|lexical|all] [--wording named|plain]
|
||||
//! [--set path] [--skills dir] [--dump dir]
|
||||
//!
|
||||
//! `--dump` writes every pair's probability so a temperature can be fitted
|
||||
//! offline without re-running the model.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::time::Instant;
|
||||
|
||||
use cm_decide::{Answer, DecideError, Decider, Decision, Question};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct EvalSet {
|
||||
cases: Vec<Case>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize, Clone)]
|
||||
struct Case {
|
||||
id: String,
|
||||
task: String,
|
||||
positives: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
struct Skill {
|
||||
name: String,
|
||||
when_to_use: String,
|
||||
}
|
||||
|
||||
/// The question a backend gets. `--wording` picks the template; a backend
|
||||
/// is reported under the wording it was run with, and each backend ships
|
||||
/// with the wording that measured best FOR IT — a prompt is part of the
|
||||
/// backend, and an NLI cross-encoder and a decision model do not want the
|
||||
/// same sentence. Both are run on both so the table says so.
|
||||
fn question_for(skill: &Skill, wording: &str) -> Question {
|
||||
let text = match wording {
|
||||
// The shipped question, shared with the server's shadow path.
|
||||
"named" => return cm_decide::triage::question(&skill.name, &skill.when_to_use),
|
||||
// The when_to_use line alone, second person rewritten to the agent,
|
||||
// as a plain declarative the NLI head was trained on.
|
||||
"plain" => third_person(&skill.when_to_use),
|
||||
other => panic!("unknown wording {other}"),
|
||||
};
|
||||
Question::noul(text)
|
||||
}
|
||||
|
||||
/// "You're the tester on a mobile team" → "The agent is the tester on a
|
||||
/// mobile team". Crude on purpose: it is a rewrite of a dozen fixed
|
||||
/// openings, not a grammar, and it exists so the NLI hypothesis reads like
|
||||
/// an MNLI hypothesis.
|
||||
fn third_person(when: &str) -> String {
|
||||
let w = when.trim();
|
||||
let rules: &[(&str, &str)] = &[
|
||||
("You're the ", "The agent is the "),
|
||||
("You are the ", "The agent is the "),
|
||||
("You're a ", "The agent is a "),
|
||||
("You are a ", "The agent is a "),
|
||||
("You're on a ", "The agent is on a "),
|
||||
("You are on a ", "The agent is on a "),
|
||||
("You're ", "The agent is "),
|
||||
("You are ", "The agent is "),
|
||||
("You need ", "The agent needs "),
|
||||
("You have ", "The agent has "),
|
||||
];
|
||||
for (from, to) in rules {
|
||||
if let Some(rest) = w.strip_prefix(from) {
|
||||
return format!("{to}{rest}");
|
||||
}
|
||||
}
|
||||
format!("In this task, {}", w.trim_end_matches('.').to_string() + ".")
|
||||
}
|
||||
|
||||
fn load_skills(dir: &Path) -> Vec<Skill> {
|
||||
let mut out = Vec::new();
|
||||
fn walk(dir: &Path, out: &mut Vec<Skill>) {
|
||||
let Ok(rd) = std::fs::read_dir(dir) else { return };
|
||||
let mut entries: Vec<_> = rd.flatten().collect();
|
||||
entries.sort_by_key(|e| e.path());
|
||||
for e in entries {
|
||||
let p = e.path();
|
||||
if p.is_dir() {
|
||||
walk(&p, out);
|
||||
} else if p.extension().is_some_and(|x| x == "md") {
|
||||
let Ok(text) = std::fs::read_to_string(&p) else { continue };
|
||||
let field = |k: &str| {
|
||||
text.lines()
|
||||
.find_map(|l| l.strip_prefix(k).map(|v| v.trim().trim_matches('"').to_string()))
|
||||
};
|
||||
if let (Some(name), Some(when)) = (field("name:"), field("when_to_use:")) {
|
||||
out.push(Skill { name, when_to_use: when });
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
walk(dir, &mut out);
|
||||
out
|
||||
}
|
||||
|
||||
/// Keyword overlap between the task and the skill's name + when_to_use:
|
||||
/// the bar a model has to clear to be worth a network call.
|
||||
struct Lexical;
|
||||
|
||||
fn tokens(s: &str) -> std::collections::BTreeSet<String> {
|
||||
s.to_ascii_lowercase()
|
||||
.split(|c: char| !c.is_alphanumeric())
|
||||
.filter(|w| w.len() > 3)
|
||||
.map(str::to_string)
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl Decider for Lexical {
|
||||
fn name(&self) -> &str {
|
||||
"lexical"
|
||||
}
|
||||
async fn decide(
|
||||
&self,
|
||||
state: &str,
|
||||
questions: &BTreeMap<String, Question>,
|
||||
) -> Result<Decision, DecideError> {
|
||||
let started = Instant::now();
|
||||
let st = tokens(state);
|
||||
let answers = questions
|
||||
.iter()
|
||||
.map(|(id, q)| {
|
||||
let Question::Noul { instructions, .. } = q else { unreachable!() };
|
||||
let ht = tokens(instructions);
|
||||
let inter = st.intersection(&ht).count() as f64;
|
||||
let p = if ht.is_empty() { 0.0 } else { (inter / ht.len() as f64 * 4.0).min(1.0) };
|
||||
(id.clone(), Answer::Noul { noul: p })
|
||||
})
|
||||
.collect();
|
||||
Ok(Decision { model: "lexical".into(), answers, usage: None, latency: started.elapsed() })
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct Pair {
|
||||
case: String,
|
||||
skill: String,
|
||||
label: bool,
|
||||
p: f64,
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct Report {
|
||||
pairs: Vec<Pair>,
|
||||
latency_ms: Vec<u128>,
|
||||
input_tokens: u64,
|
||||
per_case_topk_hits: Vec<(usize, usize)>,
|
||||
errors: usize,
|
||||
}
|
||||
|
||||
fn auroc(pairs: &[Pair]) -> f64 {
|
||||
// Mann–Whitney: fraction of (positive, negative) pairs ranked correctly.
|
||||
let pos: Vec<f64> = pairs.iter().filter(|p| p.label).map(|p| p.p).collect();
|
||||
let neg: Vec<f64> = pairs.iter().filter(|p| !p.label).map(|p| p.p).collect();
|
||||
if pos.is_empty() || neg.is_empty() {
|
||||
return f64::NAN;
|
||||
}
|
||||
let mut s = 0.0;
|
||||
for a in &pos {
|
||||
for b in &neg {
|
||||
s += if a > b { 1.0 } else if a == b { 0.5 } else { 0.0 };
|
||||
}
|
||||
}
|
||||
s / (pos.len() * neg.len()) as f64
|
||||
}
|
||||
|
||||
fn f1_at(pairs: &[Pair], t: f64) -> (f64, f64, f64) {
|
||||
let (mut tp, mut fp, mut fn_) = (0.0, 0.0, 0.0);
|
||||
for p in pairs {
|
||||
match (p.p >= t, p.label) {
|
||||
(true, true) => tp += 1.0,
|
||||
(true, false) => fp += 1.0,
|
||||
(false, true) => fn_ += 1.0,
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
let prec = if tp + fp > 0.0 { tp / (tp + fp) } else { 0.0 };
|
||||
let rec = if tp + fn_ > 0.0 { tp / (tp + fn_) } else { 0.0 };
|
||||
let f1 = if prec + rec > 0.0 { 2.0 * prec * rec / (prec + rec) } else { 0.0 };
|
||||
(f1, prec, rec)
|
||||
}
|
||||
|
||||
fn brier(pairs: &[Pair]) -> f64 {
|
||||
pairs.iter().map(|p| (p.p - if p.label { 1.0 } else { 0.0 }).powi(2)).sum::<f64>() / pairs.len() as f64
|
||||
}
|
||||
|
||||
/// Expected calibration error, ten equal-width bins.
|
||||
fn ece(pairs: &[Pair]) -> f64 {
|
||||
let mut bins = vec![(0usize, 0.0f64, 0.0f64); 10];
|
||||
for p in pairs {
|
||||
let b = ((p.p * 10.0).floor() as usize).min(9);
|
||||
bins[b].0 += 1;
|
||||
bins[b].1 += p.p;
|
||||
bins[b].2 += if p.label { 1.0 } else { 0.0 };
|
||||
}
|
||||
let n = pairs.len() as f64;
|
||||
bins.iter()
|
||||
.filter(|(c, _, _)| *c > 0)
|
||||
.map(|(c, sp, sy)| (*c as f64 / n) * ((sp / *c as f64) - (sy / *c as f64)).abs())
|
||||
.sum()
|
||||
}
|
||||
|
||||
async fn run(backend: &dyn Decider, cases: &[Case], skills: &[Skill], wording: &str) -> Report {
|
||||
let mut r = Report::default();
|
||||
let questions: BTreeMap<String, Question> =
|
||||
skills.iter().map(|s| (s.name.clone(), question_for(s, wording))).collect();
|
||||
for case in cases {
|
||||
let d = match backend.decide(&case.task, &questions).await {
|
||||
Ok(d) => d,
|
||||
Err(e) => {
|
||||
eprintln!(" {}: {} failed: {e}", backend.name(), case.id);
|
||||
r.errors += 1;
|
||||
continue;
|
||||
}
|
||||
};
|
||||
r.latency_ms.push(d.latency.as_millis());
|
||||
if let Some(u) = d.usage {
|
||||
r.input_tokens += u.input_tokens;
|
||||
}
|
||||
let mut ranked: Vec<(String, f64)> = Vec::new();
|
||||
for s in skills {
|
||||
let p = match d.answers.get(&s.name) {
|
||||
Some(Answer::Noul { noul }) => *noul,
|
||||
_ => 0.0,
|
||||
};
|
||||
let label = case.positives.iter().any(|x| x == &s.name);
|
||||
r.pairs.push(Pair { case: case.id.clone(), skill: s.name.clone(), label, p });
|
||||
ranked.push((s.name.clone(), p));
|
||||
}
|
||||
ranked.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap());
|
||||
let k = case.positives.len();
|
||||
let hits = ranked.iter().take(k).filter(|(n, _)| case.positives.contains(n)).count();
|
||||
r.per_case_topk_hits.push((hits, k));
|
||||
}
|
||||
r
|
||||
}
|
||||
|
||||
fn print_report(name: &str, r: &Report) {
|
||||
if r.pairs.is_empty() {
|
||||
println!("{name:<24} no results ({} errors)", r.errors);
|
||||
return;
|
||||
}
|
||||
let best = (5..=95)
|
||||
.map(|i| i as f64 / 100.0)
|
||||
.map(|t| (t, f1_at(&r.pairs, t)))
|
||||
.max_by(|a, b| a.1 .0.partial_cmp(&b.1 .0).unwrap())
|
||||
.unwrap();
|
||||
let (f1_half, p_half, r_half) = f1_at(&r.pairs, 0.5);
|
||||
let topk: (usize, usize) = r.per_case_topk_hits.iter().fold((0, 0), |a, b| (a.0 + b.0, a.1 + b.1));
|
||||
let lat = if r.latency_ms.is_empty() { 0 } else { r.latency_ms.iter().sum::<u128>() / r.latency_ms.len() as u128 };
|
||||
println!(
|
||||
"{name:<24} AUROC {:.3} [email protected] {:.2} (P {:.2} R {:.2}) bestF1 {:.2}@{:.2} top-k {}/{} Brier {:.3} ECE {:.3} {} ms/call {} tok errors {}",
|
||||
auroc(&r.pairs), f1_half, p_half, r_half, best.1 .0, best.0, topk.0, topk.1,
|
||||
brier(&r.pairs), ece(&r.pairs), lat, r.input_tokens, r.errors
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() {
|
||||
let mut args = std::env::args().skip(1);
|
||||
let (mut backend, mut set, mut skills_dir, mut dump, mut wording) = (
|
||||
"all".to_string(),
|
||||
PathBuf::from("crates/cm-decide/eval/skill-triage.json"),
|
||||
PathBuf::from("skills"),
|
||||
None::<PathBuf>,
|
||||
"named".to_string(),
|
||||
);
|
||||
while let Some(a) = args.next() {
|
||||
match a.as_str() {
|
||||
"--backend" => backend = args.next().unwrap_or_default(),
|
||||
"--wording" => wording = args.next().unwrap_or_default(),
|
||||
"--set" => set = args.next().map(PathBuf::from).unwrap_or(set),
|
||||
"--skills" => skills_dir = args.next().map(PathBuf::from).unwrap_or(skills_dir),
|
||||
"--dump" => dump = args.next().map(PathBuf::from),
|
||||
other => {
|
||||
eprintln!("unknown arg {other}");
|
||||
std::process::exit(2);
|
||||
}
|
||||
}
|
||||
}
|
||||
let text = std::fs::read_to_string(&set).expect("read eval set");
|
||||
let set: EvalSet = serde_json::from_str(&text).expect("parse eval set");
|
||||
let skills = load_skills(&skills_dir);
|
||||
// Every positive must name a real skill, or the label is a typo scored
|
||||
// as a miss against every backend.
|
||||
for c in &set.cases {
|
||||
for p in &c.positives {
|
||||
assert!(skills.iter().any(|s| &s.name == p), "case {}: unknown skill {p:?}", c.id);
|
||||
}
|
||||
}
|
||||
println!(
|
||||
"{} cases × {} skills = {} pairs, {} positive\n",
|
||||
set.cases.len(),
|
||||
skills.len(),
|
||||
set.cases.len() * skills.len(),
|
||||
set.cases.iter().map(|c| c.positives.len()).sum::<usize>()
|
||||
);
|
||||
|
||||
let mut backends: Vec<Box<dyn Decider>> = Vec::new();
|
||||
if backend == "all" || backend == "lexical" {
|
||||
backends.push(Box::new(Lexical));
|
||||
}
|
||||
#[cfg(feature = "jev")]
|
||||
if backend == "all" || backend == "jev" {
|
||||
match cm_decide::jev::Jev::from_env() {
|
||||
Some(j) => backends.push(Box::new(j)),
|
||||
None => eprintln!("jev: TYPESAFE_API_KEY unset — skipped"),
|
||||
}
|
||||
}
|
||||
#[cfg(feature = "nli")]
|
||||
if backend == "all" || backend == "nli" {
|
||||
let t0 = Instant::now();
|
||||
match cm_decide::nli::Nli::load(cm_decide::nli::NliConfig::from_env()) {
|
||||
Ok(n) => {
|
||||
eprintln!("nli: loaded {} in {:?}", n.name(), t0.elapsed());
|
||||
backends.push(Box::new(n));
|
||||
}
|
||||
Err(e) => eprintln!("nli: could not load — {e}"),
|
||||
}
|
||||
}
|
||||
if backends.is_empty() {
|
||||
eprintln!("no backend to run");
|
||||
std::process::exit(2);
|
||||
}
|
||||
for b in &backends {
|
||||
let r = run(b.as_ref(), &set.cases, &skills, &wording).await;
|
||||
print_report(&format!("{} [{wording}]", b.name()), &r);
|
||||
if let Some(dir) = &dump {
|
||||
std::fs::create_dir_all(dir).ok();
|
||||
let path = dir.join(format!("{}-{wording}.json", b.name().replace([':', '/'], "-")));
|
||||
std::fs::write(&path, serde_json::to_string_pretty(&r.pairs).unwrap()).ok();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
//! TypeSafe's Jev over `POST /v1/systemone`.
|
||||
//!
|
||||
//! Text-only, 64 K context (32 K for the state), no tools, no reasoning.
|
||||
//! Priced on input tokens only. Not trained on customer requests. The key is
|
||||
//! `TYPESAFE_API_KEY` and lives in the server's environment — it is never
|
||||
//! handed to a mission container, and this client is only ever called from
|
||||
//! the server.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use serde::Deserialize;
|
||||
|
||||
use crate::{Answer, DecideError, Decider, Decision, Question, Usage};
|
||||
|
||||
pub const ENDPOINT: &str = "https://api.typesafe.ai/v1/systemone";
|
||||
pub const DEFAULT_MODEL: &str = "jev-latest";
|
||||
|
||||
pub struct Jev {
|
||||
client: reqwest::Client,
|
||||
key: String,
|
||||
model: String,
|
||||
}
|
||||
|
||||
impl Jev {
|
||||
/// From `TYPESAFE_API_KEY` (and `TYPESAFE_MODEL`, default `jev-latest`).
|
||||
/// `None` when the key is unset: the caller decides whether that means
|
||||
/// "skip the decision" or "use another backend" — it never means guess.
|
||||
pub fn from_env() -> Option<Self> {
|
||||
let key = std::env::var("TYPESAFE_API_KEY").ok()?;
|
||||
let key = key.trim().to_string();
|
||||
if key.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let model = std::env::var("TYPESAFE_MODEL")
|
||||
.ok()
|
||||
.filter(|m| !m.trim().is_empty())
|
||||
.unwrap_or_else(|| DEFAULT_MODEL.to_string());
|
||||
Some(Self::new(key, model))
|
||||
}
|
||||
|
||||
pub fn new(key: String, model: String) -> Self {
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(Duration::from_secs(30))
|
||||
.build()
|
||||
.expect("reqwest client");
|
||||
Self { client, key, model }
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct Reply {
|
||||
model: String,
|
||||
answers: BTreeMap<String, RawAnswer>,
|
||||
#[serde(default)]
|
||||
usage: Option<RawUsage>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct RawUsage {
|
||||
#[serde(default)]
|
||||
input_tokens: u64,
|
||||
#[serde(default)]
|
||||
output_tokens: u64,
|
||||
}
|
||||
|
||||
/// Their answer shapes. Score keys `probabilities` by level number as a
|
||||
/// STRING ("0", "1", …); a `legend` mirrors the criteria and is dropped.
|
||||
#[derive(Deserialize)]
|
||||
#[serde(tag = "type", rename_all = "lowercase")]
|
||||
enum RawAnswer {
|
||||
Choice {
|
||||
choice: String,
|
||||
probabilities: BTreeMap<String, f64>,
|
||||
confidence: f64,
|
||||
},
|
||||
Score {
|
||||
score: f64,
|
||||
probabilities: BTreeMap<String, f64>,
|
||||
confidence: f64,
|
||||
},
|
||||
Noul {
|
||||
noul: f64,
|
||||
},
|
||||
}
|
||||
|
||||
impl RawAnswer {
|
||||
fn into_answer(self) -> Result<Answer, DecideError> {
|
||||
Ok(match self {
|
||||
RawAnswer::Choice { choice, probabilities, confidence } => Answer::Choice {
|
||||
choice,
|
||||
probabilities,
|
||||
confidence,
|
||||
},
|
||||
RawAnswer::Score { score, probabilities, confidence } => {
|
||||
let mut levels: Vec<(usize, f64)> = probabilities
|
||||
.into_iter()
|
||||
.map(|(k, v)| {
|
||||
k.parse::<usize>()
|
||||
.map(|i| (i, v))
|
||||
.map_err(|_| DecideError::Shape(format!("score level key {k:?}")))
|
||||
})
|
||||
.collect::<Result<_, _>>()?;
|
||||
levels.sort_by_key(|(i, _)| *i);
|
||||
Answer::Score {
|
||||
score,
|
||||
probabilities: levels.into_iter().map(|(_, p)| p).collect(),
|
||||
confidence,
|
||||
}
|
||||
}
|
||||
RawAnswer::Noul { noul } => Answer::Noul { noul },
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl Decider for Jev {
|
||||
fn name(&self) -> &str {
|
||||
"jev"
|
||||
}
|
||||
|
||||
async fn decide(
|
||||
&self,
|
||||
state: &str,
|
||||
questions: &BTreeMap<String, Question>,
|
||||
) -> Result<Decision, DecideError> {
|
||||
let body = serde_json::json!({
|
||||
"state": state,
|
||||
"model": self.model,
|
||||
"questions": questions,
|
||||
});
|
||||
let started = Instant::now();
|
||||
let resp = self
|
||||
.client
|
||||
.post(ENDPOINT)
|
||||
.bearer_auth(&self.key)
|
||||
.json(&body)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| DecideError::Transport(e.to_string()))?;
|
||||
let status = resp.status();
|
||||
let text = resp
|
||||
.text()
|
||||
.await
|
||||
.map_err(|e| DecideError::Transport(e.to_string()))?;
|
||||
if !status.is_success() {
|
||||
return Err(DecideError::Model(format!(
|
||||
"HTTP {status}: {}",
|
||||
text.chars().take(300).collect::<String>()
|
||||
)));
|
||||
}
|
||||
let reply: Reply =
|
||||
serde_json::from_str(&text).map_err(|e| DecideError::Shape(e.to_string()))?;
|
||||
let mut answers = BTreeMap::new();
|
||||
for (id, raw) in reply.answers {
|
||||
answers.insert(id, raw.into_answer()?);
|
||||
}
|
||||
Ok(Decision {
|
||||
model: reply.model,
|
||||
answers,
|
||||
usage: reply.usage.map(|u| Usage {
|
||||
input_tokens: u.input_tokens,
|
||||
output_tokens: u.output_tokens,
|
||||
}),
|
||||
latency: started.elapsed(),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// The documented response, read back into our shapes — including the
|
||||
/// string-keyed Score levels, which arrive unordered in a JSON object.
|
||||
#[test]
|
||||
fn the_documented_reply_parses() {
|
||||
let text = r#"{"model":"jev-1.13.0","answers":{
|
||||
"department":{"type":"choice","choice":"technical","confidence":0.78,
|
||||
"probabilities":{"technical":0.85,"sales":0.0,"billing":0.15}},
|
||||
"frustration":{"type":"score","score":1.0,"confidence":1.0,
|
||||
"legend":{"0":"calm","1":"frustrated","2":"angry"},
|
||||
"probabilities":{"2":0.0,"0":0.0,"1":1.0}},
|
||||
"is_urgent":{"type":"noul","noul":1.0}},
|
||||
"usage":{"input_tokens":392,"output_tokens":65}}"#;
|
||||
let reply: Reply = serde_json::from_str(text).unwrap();
|
||||
let a = reply.answers.get("frustration").unwrap();
|
||||
let RawAnswer::Score { .. } = a else { panic!() };
|
||||
let converted: BTreeMap<String, Answer> = reply
|
||||
.answers
|
||||
.into_iter()
|
||||
.map(|(k, v)| (k, v.into_answer().unwrap()))
|
||||
.collect();
|
||||
match &converted["frustration"] {
|
||||
Answer::Score { probabilities, score, .. } => {
|
||||
assert_eq!(probabilities, &vec![0.0, 1.0, 0.0]);
|
||||
assert_eq!(*score, 1.0);
|
||||
}
|
||||
other => panic!("{other:?}"),
|
||||
}
|
||||
match &converted["department"] {
|
||||
Answer::Choice { choice, .. } => assert_eq!(choice, "technical"),
|
||||
other => panic!("{other:?}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,270 @@
|
||||
//! Typed gut-check decisions for the platform's code to branch on.
|
||||
//!
|
||||
//! ClawMates has had exactly two kinds of decision-maker: deterministic code,
|
||||
//! and a full LLM chat call parsed back into a boolean. There was no cheap,
|
||||
//! calibrated classifier in between — every model decision was forced binary,
|
||||
//! took seconds, and drew on a provider quota that has emptied twice. This
|
||||
//! crate is that middle tier, in the shape TypeSafe's Jev popularised
|
||||
//! (`docs.typesafe.ai`): a [`Question`] is a **Choice** over labelled options,
|
||||
//! a **Score** over ordered levels, or a **Noul** (a yes/no); an [`Answer`] is
|
||||
//! a probability distribution plus a confidence, never text.
|
||||
//!
|
||||
//! Two backends implement [`Decider`]: [`jev::Jev`] (hosted) and
|
||||
//! [`nli::Nli`] (a DeBERTa-v3 MNLI cross-encoder run in-process). Neither
|
||||
//! reasons or runs tools, and that is the point — the judge and the tool
|
||||
//! gate stay where they are. What goes here is triage: which skills apply to
|
||||
//! a task, whether an outbound action looks like exfiltration, which past
|
||||
//! verdict is relevant. Every use is measured on labelled cases first
|
||||
//! (`decide-eval`) and shipped in shadow mode before it changes behaviour.
|
||||
//!
|
||||
//! The `patterns` module carries the composition rules as code, because the
|
||||
//! vendor's advice is right and worth keeping regardless of vendor: ask every
|
||||
//! question in one call, gate on confidence, combine dimensions with weights
|
||||
//! your code owns.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::time::Duration;
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[cfg(feature = "jev")]
|
||||
pub mod jev;
|
||||
#[cfg(feature = "nli")]
|
||||
pub mod nli;
|
||||
pub mod patterns;
|
||||
pub mod triage;
|
||||
|
||||
/// One question, evaluated on its own against the state.
|
||||
///
|
||||
/// Each variant answers a different kind of question, and the shape of the
|
||||
/// answer follows: an option, a position, or a probability.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||
#[serde(tag = "type", rename_all = "lowercase")]
|
||||
pub enum Question {
|
||||
/// One option from a fixed, unordered set. `criteria` maps an option name
|
||||
/// to its description (`None` when the name is clear on its own).
|
||||
Choice {
|
||||
instructions: String,
|
||||
criteria: BTreeMap<String, Option<String>>,
|
||||
},
|
||||
/// A position on a spectrum described in steps, low to high. At least two
|
||||
/// levels; each is a situation, not a degree ("workaround exists", not
|
||||
/// "moderately severe").
|
||||
Score {
|
||||
instructions: String,
|
||||
criteria: Vec<String>,
|
||||
},
|
||||
/// Yes or no. `criteria` optionally says what a yes and a no look like.
|
||||
Noul {
|
||||
instructions: String,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
criteria: Option<NoulCriteria>,
|
||||
},
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||
pub struct NoulCriteria {
|
||||
#[serde(rename = "true")]
|
||||
pub yes: String,
|
||||
#[serde(rename = "false")]
|
||||
pub no: String,
|
||||
}
|
||||
|
||||
impl Question {
|
||||
pub fn noul(instructions: impl Into<String>) -> Self {
|
||||
Question::Noul {
|
||||
instructions: instructions.into(),
|
||||
criteria: None,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn choice<K: Into<String>, V: Into<String>>(
|
||||
instructions: impl Into<String>,
|
||||
options: impl IntoIterator<Item = (K, Option<V>)>,
|
||||
) -> Self {
|
||||
Question::Choice {
|
||||
instructions: instructions.into(),
|
||||
criteria: options
|
||||
.into_iter()
|
||||
.map(|(k, v)| (k.into(), v.map(Into::into)))
|
||||
.collect(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn score<L: Into<String>>(
|
||||
instructions: impl Into<String>,
|
||||
levels: impl IntoIterator<Item = L>,
|
||||
) -> Self {
|
||||
Question::Score {
|
||||
instructions: instructions.into(),
|
||||
criteria: levels.into_iter().map(Into::into).collect(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The typed answer to one question.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||
#[serde(tag = "type", rename_all = "lowercase")]
|
||||
pub enum Answer {
|
||||
Choice {
|
||||
/// The option with the highest probability.
|
||||
choice: String,
|
||||
/// Every option; sums to 1.
|
||||
probabilities: BTreeMap<String, f64>,
|
||||
/// How peaked the distribution is: 1.0 on a single option, toward 0
|
||||
/// as it flattens. See [`confidence_of`].
|
||||
confidence: f64,
|
||||
},
|
||||
Score {
|
||||
/// Probability-weighted position on the level line, 0 to `levels-1`.
|
||||
score: f64,
|
||||
/// Per level, in order; sums to 1.
|
||||
probabilities: Vec<f64>,
|
||||
confidence: f64,
|
||||
},
|
||||
Noul {
|
||||
/// Probability that the answer is yes.
|
||||
noul: f64,
|
||||
},
|
||||
}
|
||||
|
||||
impl Answer {
|
||||
/// Confidence for the shapes that have one; a Noul's value already
|
||||
/// describes its whole two-outcome distribution, so it reports how far
|
||||
/// the value sits from 0.5.
|
||||
pub fn confidence(&self) -> f64 {
|
||||
match self {
|
||||
Answer::Choice { confidence, .. } | Answer::Score { confidence, .. } => *confidence,
|
||||
Answer::Noul { noul } => (noul - 0.5).abs() * 2.0,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// What a backend returned for one call: every answer, who answered, what
|
||||
/// it cost. `usage` is `None` for a local backend (nothing is billed) and
|
||||
/// `latency` is wall-clock as this process saw it.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct Decision {
|
||||
pub model: String,
|
||||
pub answers: BTreeMap<String, Answer>,
|
||||
#[serde(default)]
|
||||
pub usage: Option<Usage>,
|
||||
#[serde(with = "duration_ms")]
|
||||
pub latency: Duration,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)]
|
||||
pub struct Usage {
|
||||
pub input_tokens: u64,
|
||||
pub output_tokens: u64,
|
||||
}
|
||||
|
||||
mod duration_ms {
|
||||
use serde::{Deserialize, Deserializer, Serializer};
|
||||
use std::time::Duration;
|
||||
pub fn serialize<S: Serializer>(d: &Duration, s: S) -> Result<S::Ok, S::Error> {
|
||||
s.serialize_u64(d.as_millis() as u64)
|
||||
}
|
||||
pub fn deserialize<'de, D: Deserializer<'de>>(d: D) -> Result<Duration, D::Error> {
|
||||
Ok(Duration::from_millis(u64::deserialize(d)?))
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum DecideError {
|
||||
#[error("{0}")]
|
||||
Config(String),
|
||||
#[error("transport: {0}")]
|
||||
Transport(String),
|
||||
#[error("the backend answered in a shape this crate does not read: {0}")]
|
||||
Shape(String),
|
||||
#[error("model: {0}")]
|
||||
Model(String),
|
||||
}
|
||||
|
||||
/// A source of typed answers. Every question in `questions` is answered
|
||||
/// against the same `state` in one call; backends are expected to evaluate
|
||||
/// them independently, so an answer never depends on which other questions
|
||||
/// were asked.
|
||||
#[async_trait::async_trait]
|
||||
pub trait Decider: Send + Sync {
|
||||
/// Stable name for logs and eval tables (`jev`, `nli`).
|
||||
fn name(&self) -> &str;
|
||||
async fn decide(
|
||||
&self,
|
||||
state: &str,
|
||||
questions: &BTreeMap<String, Question>,
|
||||
) -> Result<Decision, DecideError>;
|
||||
}
|
||||
|
||||
/// Confidence from a distribution: `1 − H(p) / ln(n)`, so a single peak is
|
||||
/// 1.0 and a flat spread is 0.0. The vendor documents only that confidence
|
||||
/// is "computed from how the probabilities are spread"; this is our
|
||||
/// definition and it is applied to the local backend's answers. Jev's own
|
||||
/// figure is passed through untouched, so the two are comparable in shape
|
||||
/// but not guaranteed identical in value.
|
||||
pub fn confidence_of(probabilities: &[f64]) -> f64 {
|
||||
let n = probabilities.len();
|
||||
if n < 2 {
|
||||
return 1.0;
|
||||
}
|
||||
let h: f64 = probabilities
|
||||
.iter()
|
||||
.filter(|p| **p > 0.0)
|
||||
.map(|p| -p * p.ln())
|
||||
.sum();
|
||||
(1.0 - h / (n as f64).ln()).clamp(0.0, 1.0)
|
||||
}
|
||||
|
||||
/// Softmax over raw scores, numerically stable.
|
||||
pub fn softmax(logits: &[f64]) -> Vec<f64> {
|
||||
let max = logits.iter().cloned().fold(f64::NEG_INFINITY, f64::max);
|
||||
let exps: Vec<f64> = logits.iter().map(|l| (l - max).exp()).collect();
|
||||
let sum: f64 = exps.iter().sum();
|
||||
exps.iter().map(|e| e / sum).collect()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn confidence_is_one_on_a_peak_and_zero_when_flat() {
|
||||
assert!((confidence_of(&[1.0, 0.0, 0.0]) - 1.0).abs() < 1e-9);
|
||||
assert!(confidence_of(&[1.0 / 3.0; 3]).abs() < 1e-9);
|
||||
let mid = confidence_of(&[0.6, 0.4]);
|
||||
assert!(mid > 0.0 && mid < 1.0);
|
||||
// A two-outcome distribution and a Noul report the same certainty
|
||||
// ordering: further from even is more confident.
|
||||
assert!(
|
||||
Answer::Noul { noul: 0.9 }.confidence() > Answer::Noul { noul: 0.6 }.confidence()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn softmax_sums_to_one_and_keeps_order() {
|
||||
let p = softmax(&[2.0, 1.0, 0.1]);
|
||||
assert!((p.iter().sum::<f64>() - 1.0).abs() < 1e-9);
|
||||
assert!(p[0] > p[1] && p[1] > p[2]);
|
||||
}
|
||||
|
||||
/// The wire shape matches the vendor's, so a question written for one
|
||||
/// backend is the same question for the other.
|
||||
#[test]
|
||||
fn questions_serialise_in_the_vendor_shape() {
|
||||
let q = Question::choice(
|
||||
"Which team?",
|
||||
[("billing", Some("charges")), ("returns", None)],
|
||||
);
|
||||
let v = serde_json::to_value(&q).unwrap();
|
||||
assert_eq!(v["type"], "choice");
|
||||
assert_eq!(v["criteria"]["billing"], "charges");
|
||||
assert!(v["criteria"]["returns"].is_null());
|
||||
let n = Question::Noul {
|
||||
instructions: "Is it urgent?".into(),
|
||||
criteria: Some(NoulCriteria { yes: "y".into(), no: "n".into() }),
|
||||
};
|
||||
let v = serde_json::to_value(&n).unwrap();
|
||||
assert_eq!(v["criteria"]["true"], "y");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,349 @@
|
||||
//! A local decision model: a DeBERTa-v3 MNLI cross-encoder, in-process.
|
||||
//!
|
||||
//! A Noul IS an entailment probability — P(the hypothesis follows from the
|
||||
//! state) — and natural-language inference is the oldest working form of
|
||||
//! zero-shot classification. A Choice is one hypothesis per option with a
|
||||
//! softmax over their entailment logits; a Score is the same over ordered
|
||||
//! levels, then the probability-weighted position. Nothing here reasons or
|
||||
//! generates: one forward pass per (state, hypothesis) pair, the answer read
|
||||
//! from the three-way head.
|
||||
//!
|
||||
//! What this does NOT have is the vendor's calibration training. Its
|
||||
//! probabilities are as calibrated as MNLI made them, which on our domain is
|
||||
//! an open question until `decide-eval` answers it; temperature scaling on
|
||||
//! our own labelled cases is the standard fix and the `temperature` knob is
|
||||
//! where it goes.
|
||||
//!
|
||||
//! Model files come from the Hugging Face hub on first use
|
||||
//! (`CLAWMATES_NLI_MODEL`, default `cross-encoder/nli-deberta-v3-base`;
|
||||
//! `-xsmall` is 4× faster and worse) or from `CLAWMATES_NLI_MODEL_DIR` for a
|
||||
//! host with no egress. CPU by default; `metal`/`cuda` features select a GPU.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::path::PathBuf;
|
||||
use std::sync::Mutex;
|
||||
use std::time::Instant;
|
||||
|
||||
use candle_core::{DType, Device, Tensor};
|
||||
use candle_nn::VarBuilder;
|
||||
use candle_transformers::models::debertav2::{Config, DebertaV2SeqClassificationModel, Id2Label};
|
||||
use tokenizers::{EncodeInput, InputSequence, PaddingParams, Tokenizer, TruncationParams};
|
||||
|
||||
use crate::{confidence_of, softmax, Answer, DecideError, Decider, Decision, Question};
|
||||
|
||||
pub const DEFAULT_MODEL: &str = "cross-encoder/nli-deberta-v3-base";
|
||||
|
||||
/// Token budget per pair. DeBERTa-v3 was trained at 512; the STATE is what
|
||||
/// gets cut when a pair is too long (`OnlyFirst`), so the hypothesis — the
|
||||
/// question — always survives whole.
|
||||
const MAX_TOKENS: usize = 512;
|
||||
|
||||
pub struct Nli {
|
||||
model: Mutex<DebertaV2SeqClassificationModel>,
|
||||
tokenizer: Tokenizer,
|
||||
device: Device,
|
||||
/// Index of each MNLI label in the head, read from `config.json`.
|
||||
entail: usize,
|
||||
contra: usize,
|
||||
/// Divides the logits before the softmax. 1.0 = the model's own
|
||||
/// calibration; fitted on labelled cases by the eval when it disagrees.
|
||||
temperature: f64,
|
||||
name: String,
|
||||
}
|
||||
|
||||
pub struct NliConfig {
|
||||
pub model: String,
|
||||
pub dir: Option<PathBuf>,
|
||||
pub temperature: f64,
|
||||
}
|
||||
|
||||
impl NliConfig {
|
||||
pub fn from_env() -> Self {
|
||||
Self {
|
||||
model: std::env::var("CLAWMATES_NLI_MODEL")
|
||||
.ok()
|
||||
.filter(|m| !m.trim().is_empty())
|
||||
.unwrap_or_else(|| DEFAULT_MODEL.to_string()),
|
||||
dir: std::env::var("CLAWMATES_NLI_MODEL_DIR").ok().map(PathBuf::from),
|
||||
temperature: std::env::var("CLAWMATES_NLI_TEMPERATURE")
|
||||
.ok()
|
||||
.and_then(|t| t.parse().ok())
|
||||
.unwrap_or(1.0),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Nli {
|
||||
pub fn load(cfg: NliConfig) -> Result<Self, DecideError> {
|
||||
let (config_path, tokenizer_path, weights_path) = match &cfg.dir {
|
||||
Some(dir) => (
|
||||
dir.join("config.json"),
|
||||
dir.join("tokenizer.json"),
|
||||
dir.join("model.safetensors"),
|
||||
),
|
||||
None => {
|
||||
let api = hf_hub::api::sync::Api::new()
|
||||
.map_err(|e| DecideError::Config(format!("hf hub: {e}")))?;
|
||||
let repo = api.model(cfg.model.clone());
|
||||
let get = |f: &str| {
|
||||
repo.get(f)
|
||||
.map_err(|e| DecideError::Config(format!("fetch {f} for {}: {e}", cfg.model)))
|
||||
};
|
||||
(get("config.json")?, get("tokenizer.json")?, get("model.safetensors")?)
|
||||
}
|
||||
};
|
||||
let config_text = std::fs::read_to_string(&config_path)
|
||||
.map_err(|e| DecideError::Config(format!("{}: {e}", config_path.display())))?;
|
||||
let config: Config = serde_json::from_str(&config_text)
|
||||
.map_err(|e| DecideError::Config(format!("config.json: {e}")))?;
|
||||
// The head's label order is the model's, not ours: read it.
|
||||
let id2label: Id2Label = serde_json::from_str::<serde_json::Value>(&config_text)
|
||||
.ok()
|
||||
.and_then(|v| v.get("id2label").cloned())
|
||||
.and_then(|v| {
|
||||
v.as_object().map(|m| {
|
||||
m.iter()
|
||||
.filter_map(|(k, v)| {
|
||||
Some((k.parse::<u32>().ok()?, v.as_str()?.to_lowercase()))
|
||||
})
|
||||
.collect()
|
||||
})
|
||||
})
|
||||
.ok_or_else(|| DecideError::Config("config.json has no id2label".into()))?;
|
||||
let find = |name: &str| {
|
||||
id2label
|
||||
.iter()
|
||||
.find(|(_, v)| v.as_str() == name)
|
||||
.map(|(k, _)| *k as usize)
|
||||
.ok_or_else(|| DecideError::Config(format!("head has no {name:?} label")))
|
||||
};
|
||||
let entail = find("entailment")?;
|
||||
// MNLI heads have three labels; the zero-shot-tuned checkpoints
|
||||
// collapse to entailment / not_entailment. Either "no" label works.
|
||||
let contra = find("contradiction").or_else(|_| find("not_entailment"))?;
|
||||
|
||||
let device = pick_device();
|
||||
// Buffered, not mmap'd: the workspace denies `unsafe`, and the weights
|
||||
// (740 MB for -base) are read once into memory and kept for the life
|
||||
// of the process anyway.
|
||||
let bytes = std::fs::read(&weights_path)
|
||||
.map_err(|e| DecideError::Config(format!("{}: {e}", weights_path.display())))?;
|
||||
let vb = VarBuilder::from_buffered_safetensors(bytes, DType::F32, &device)
|
||||
.map_err(|e| DecideError::Config(format!("weights: {e}")))?;
|
||||
// The backbone's tensors are `deberta.*`; the pooler and classifier
|
||||
// read from the root. candle's own example does exactly this.
|
||||
let vb = vb.set_prefix("deberta");
|
||||
let model = DebertaV2SeqClassificationModel::load(vb, &config, Some(id2label))
|
||||
.map_err(|e| DecideError::Config(format!("model: {e}")))?;
|
||||
|
||||
let mut tokenizer = Tokenizer::from_file(&tokenizer_path)
|
||||
.map_err(|e| DecideError::Config(format!("tokenizer: {e}")))?;
|
||||
tokenizer.with_padding(Some(PaddingParams::default()));
|
||||
tokenizer
|
||||
.with_truncation(Some(TruncationParams {
|
||||
max_length: MAX_TOKENS,
|
||||
strategy: tokenizers::TruncationStrategy::OnlyFirst,
|
||||
..Default::default()
|
||||
}))
|
||||
.map_err(|e| DecideError::Config(format!("truncation: {e}")))?;
|
||||
|
||||
Ok(Self {
|
||||
model: Mutex::new(model),
|
||||
tokenizer,
|
||||
device,
|
||||
entail,
|
||||
contra,
|
||||
temperature: cfg.temperature.max(1e-3),
|
||||
name: format!("nli:{}", cfg.model.rsplit('/').next().unwrap_or(&cfg.model)),
|
||||
})
|
||||
}
|
||||
|
||||
/// Entailment and contradiction logits for every (state, hypothesis)
|
||||
/// pair, one batch, one forward pass.
|
||||
fn logits(&self, state: &str, hypotheses: &[String]) -> Result<Vec<(f64, f64)>, DecideError> {
|
||||
if hypotheses.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let inputs: Vec<EncodeInput> = hypotheses
|
||||
.iter()
|
||||
.map(|h| {
|
||||
EncodeInput::Dual(
|
||||
InputSequence::Raw(state.into()),
|
||||
InputSequence::Raw(h.as_str().into()),
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
let encodings = self
|
||||
.tokenizer
|
||||
.encode_batch(inputs, true)
|
||||
.map_err(|e| DecideError::Model(format!("tokenize: {e}")))?;
|
||||
let to_tensor = |rows: Vec<&[u32]>| -> Result<Tensor, DecideError> {
|
||||
let stacked: Vec<Tensor> = rows
|
||||
.iter()
|
||||
.map(|r| Tensor::new(*r, &self.device))
|
||||
.collect::<Result<_, _>>()
|
||||
.map_err(|e| DecideError::Model(e.to_string()))?;
|
||||
Tensor::stack(&stacked, 0).map_err(|e| DecideError::Model(e.to_string()))
|
||||
};
|
||||
let ids = to_tensor(encodings.iter().map(|e| e.get_ids()).collect())?;
|
||||
let mask = to_tensor(encodings.iter().map(|e| e.get_attention_mask()).collect())?;
|
||||
let types = to_tensor(encodings.iter().map(|e| e.get_type_ids()).collect())?;
|
||||
let out = {
|
||||
let model = self.model.lock().map_err(|_| DecideError::Model("model lock".into()))?;
|
||||
model
|
||||
.forward(&ids, Some(types), Some(mask))
|
||||
.map_err(|e| DecideError::Model(format!("forward: {e}")))?
|
||||
};
|
||||
let rows: Vec<Vec<f32>> = out
|
||||
.to_dtype(DType::F32)
|
||||
.and_then(|t| t.to_vec2())
|
||||
.map_err(|e| DecideError::Model(e.to_string()))?;
|
||||
Ok(rows
|
||||
.iter()
|
||||
.map(|r| {
|
||||
(
|
||||
r[self.entail] as f64 / self.temperature,
|
||||
r[self.contra] as f64 / self.temperature,
|
||||
)
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// P(yes) for one hypothesis: entailment against contradiction, the
|
||||
/// neutral class left out — the standard zero-shot NLI reading.
|
||||
fn p_yes(entail: f64, contra: f64) -> f64 {
|
||||
softmax(&[entail, contra])[0]
|
||||
}
|
||||
}
|
||||
|
||||
fn pick_device() -> Device {
|
||||
#[cfg(feature = "cuda")]
|
||||
if let Ok(d) = Device::new_cuda(0) {
|
||||
return d;
|
||||
}
|
||||
#[cfg(feature = "metal")]
|
||||
if let Ok(d) = Device::new_metal(0) {
|
||||
return d;
|
||||
}
|
||||
Device::Cpu
|
||||
}
|
||||
|
||||
/// The hypothesis a question becomes. Kept in one place because the wording
|
||||
/// is the whole "prompt" this backend has, and the eval is what tunes it.
|
||||
pub fn hypotheses(question: &Question) -> Vec<String> {
|
||||
match question {
|
||||
Question::Noul { instructions, criteria } => match criteria {
|
||||
Some(c) => vec![
|
||||
format!("{instructions} {}", c.yes),
|
||||
format!("{instructions} {}", c.no),
|
||||
],
|
||||
None => vec![instructions.clone()],
|
||||
},
|
||||
Question::Choice { instructions, criteria } => criteria
|
||||
.iter()
|
||||
.map(|(name, desc)| match desc {
|
||||
Some(d) => format!("{instructions} {name}: {d}"),
|
||||
None => format!("{instructions} {name}"),
|
||||
})
|
||||
.collect(),
|
||||
Question::Score { instructions, criteria } => criteria
|
||||
.iter()
|
||||
.map(|level| format!("{instructions} {level}"))
|
||||
.collect(),
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl Decider for Nli {
|
||||
fn name(&self) -> &str {
|
||||
&self.name
|
||||
}
|
||||
|
||||
async fn decide(
|
||||
&self,
|
||||
state: &str,
|
||||
questions: &BTreeMap<String, Question>,
|
||||
) -> Result<Decision, DecideError> {
|
||||
let started = Instant::now();
|
||||
// Every hypothesis of every question in one batch; then read each
|
||||
// question's slice back out.
|
||||
let mut all: Vec<String> = Vec::new();
|
||||
let mut spans: Vec<(String, usize, usize)> = Vec::new();
|
||||
for (id, q) in questions {
|
||||
let hs = hypotheses(q);
|
||||
let start = all.len();
|
||||
all.extend(hs);
|
||||
spans.push((id.clone(), start, all.len()));
|
||||
}
|
||||
let logits = self.logits(state, &all)?;
|
||||
let mut answers = BTreeMap::new();
|
||||
for (id, start, end) in spans {
|
||||
let slice = &logits[start..end];
|
||||
let q = &questions[&id];
|
||||
let answer = match q {
|
||||
Question::Noul { criteria, .. } => {
|
||||
let p = if criteria.is_some() {
|
||||
// yes-description vs no-description: which does the
|
||||
// state entail more?
|
||||
softmax(&[slice[0].0, slice[1].0])[0]
|
||||
} else {
|
||||
Self::p_yes(slice[0].0, slice[0].1)
|
||||
};
|
||||
Answer::Noul { noul: p }
|
||||
}
|
||||
Question::Choice { criteria, .. } => {
|
||||
let probs = softmax(&slice.iter().map(|(e, _)| *e).collect::<Vec<_>>());
|
||||
let names: Vec<&String> = criteria.keys().collect();
|
||||
let (best, _) = probs
|
||||
.iter()
|
||||
.enumerate()
|
||||
.fold((0, f64::NEG_INFINITY), |acc, (i, p)| if *p > acc.1 { (i, *p) } else { acc });
|
||||
Answer::Choice {
|
||||
choice: names[best].clone(),
|
||||
confidence: confidence_of(&probs),
|
||||
probabilities: names.iter().map(|n| (*n).clone()).zip(probs).collect(),
|
||||
}
|
||||
}
|
||||
Question::Score { .. } => {
|
||||
let probs = softmax(&slice.iter().map(|(e, _)| *e).collect::<Vec<_>>());
|
||||
let score = probs.iter().enumerate().map(|(i, p)| i as f64 * p).sum();
|
||||
Answer::Score {
|
||||
score,
|
||||
confidence: confidence_of(&probs),
|
||||
probabilities: probs,
|
||||
}
|
||||
}
|
||||
};
|
||||
answers.insert(id, answer);
|
||||
}
|
||||
Ok(Decision {
|
||||
model: self.name.clone(),
|
||||
answers,
|
||||
usage: None,
|
||||
latency: started.elapsed(),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn every_question_becomes_the_expected_hypotheses() {
|
||||
let n = Question::noul("The task needs web search.");
|
||||
assert_eq!(hypotheses(&n), vec!["The task needs web search."]);
|
||||
let c = Question::choice("This work is", [("research", Some("reading")), ("coding", None)]);
|
||||
let h = hypotheses(&c);
|
||||
assert_eq!(h, vec!["This work is coding", "This work is research: reading"]);
|
||||
let s = Question::score("Severity:", ["cosmetic", "blocking"]);
|
||||
assert_eq!(hypotheses(&s).len(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn p_yes_is_entailment_against_contradiction() {
|
||||
assert!(Nli::p_yes(3.0, -3.0) > 0.99);
|
||||
assert!(Nli::p_yes(-3.0, 3.0) < 0.01);
|
||||
assert!((Nli::p_yes(0.0, 0.0) - 0.5).abs() < 1e-9);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
//! The composition rules, as code. Each is one of the vendor's documented
|
||||
//! patterns, kept here because they are the right way to use ANY calibrated
|
||||
//! classifier and should not live in one call site's `if` chain.
|
||||
|
||||
use crate::Answer;
|
||||
|
||||
/// Confidence-gated routing: three outcomes, not two. The answer says what;
|
||||
/// the confidence (or a Noul's distance from even) says whether to act.
|
||||
///
|
||||
/// `act_above` is the probability/confidence at which code may act on its
|
||||
/// own; `dismiss_below` the one under which the answer is a clear no. In
|
||||
/// between is the band a person, or a slower model, gets to decide.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum Gate {
|
||||
Act,
|
||||
Review,
|
||||
Dismiss,
|
||||
}
|
||||
|
||||
pub fn gate_noul(p: f64, dismiss_below: f64, act_above: f64) -> Gate {
|
||||
debug_assert!(dismiss_below <= act_above);
|
||||
if p >= act_above {
|
||||
Gate::Act
|
||||
} else if p < dismiss_below {
|
||||
Gate::Dismiss
|
||||
} else {
|
||||
Gate::Review
|
||||
}
|
||||
}
|
||||
|
||||
/// Gate a Choice or Score on its confidence alone.
|
||||
pub fn gate_confidence(answer: &Answer, review_below: f64) -> Gate {
|
||||
if answer.confidence() >= review_below {
|
||||
Gate::Act
|
||||
} else {
|
||||
Gate::Review
|
||||
}
|
||||
}
|
||||
|
||||
/// Composite scoring: normalise each Score to 0..1 by its top level and
|
||||
/// combine with weights the caller owns. `parts` is `(score, levels, weight)`.
|
||||
/// Weights need not sum to one; the result is divided by their sum.
|
||||
pub fn composite(parts: &[(f64, usize, f64)]) -> f64 {
|
||||
let total: f64 = parts.iter().map(|(_, _, w)| w).sum();
|
||||
if total <= 0.0 {
|
||||
return 0.0;
|
||||
}
|
||||
parts
|
||||
.iter()
|
||||
.map(|(score, levels, w)| {
|
||||
let top = (*levels as f64 - 1.0).max(1.0);
|
||||
(score / top).clamp(0.0, 1.0) * w
|
||||
})
|
||||
.sum::<f64>()
|
||||
/ total
|
||||
}
|
||||
|
||||
/// Rerank: order candidates by a per-candidate Noul, highest first.
|
||||
pub fn rerank<T>(mut items: Vec<(T, f64)>) -> Vec<(T, f64)> {
|
||||
items.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));
|
||||
items
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn a_noul_gates_three_ways() {
|
||||
assert_eq!(gate_noul(0.95, 0.2, 0.8), Gate::Act);
|
||||
assert_eq!(gate_noul(0.05, 0.2, 0.8), Gate::Dismiss);
|
||||
assert_eq!(gate_noul(0.5, 0.2, 0.8), Gate::Review);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn composite_normalises_by_top_level_and_weights() {
|
||||
// severity 2 of 0..2 at weight 0.6, frustration 1 of 0..2 at 0.3,
|
||||
// report quality 3 of 0..3 at 0.1 → 0.6*1 + 0.3*0.5 + 0.1*1 = 0.85
|
||||
let c = composite(&[(2.0, 3, 0.6), (1.0, 3, 0.3), (3.0, 4, 0.1)]);
|
||||
assert!((c - 0.85).abs() < 1e-9);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rerank_is_descending() {
|
||||
let r = rerank(vec![("a", 0.2), ("b", 0.9), ("c", 0.5)]);
|
||||
assert_eq!(r.iter().map(|x| x.0).collect::<Vec<_>>(), ["b", "c", "a"]);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
//! The skill-triage question, in one place.
|
||||
//!
|
||||
//! Measured 2026-09-21 on `eval/skill-triage.json` (20 tasks × 53 skills):
|
||||
//! this wording — the skill's NAME plus its `when_to_use` line — scored
|
||||
//! AUROC 0.989 / F1 0.84 on Jev against 0.970 / 0.66 for the when_to_use
|
||||
//! line alone. The name carries signal. The server's shadow path and the
|
||||
//! eval both call this, so what is measured is what runs.
|
||||
|
||||
use crate::Question;
|
||||
|
||||
pub const WORDING: &str = "named";
|
||||
|
||||
pub fn question(name: &str, when_to_use: &str) -> Question {
|
||||
Question::noul(format!(
|
||||
"This task calls for the skill \"{name}\", which applies when: {when_to_use}"
|
||||
))
|
||||
}
|
||||
|
||||
/// The probability at which a skill counts as "applies" when the triage is
|
||||
/// read back. 0.5 is where Jev's best F1 sat (0.84 at 0.51); it is not a
|
||||
/// gate on anything yet — shadow mode records, it does not select.
|
||||
pub const APPLIES_AT: f64 = 0.5;
|
||||
Reference in New Issue
Block a user