feat(decide): cm-decide — typed calibrated decisions; Jev + local NLI backends; skill triage in shadow

A third kind of decision-maker between deterministic code and a full LLM
call: Choice / Score / Noul questions answered as probability
distributions with a confidence, behind one Decider trait, with the
composition patterns (confidence gating, composite scoring, rerank) as
code. Two backends: TypeSafe's Jev over HTTP, and a DeBERTa-v3 MNLI
cross-encoder run in-process with candle (feature nli; metal/cuda).

decide-eval measures a backend on labelled cases the way judge-eval
measures the judge. eval/skill-triage.json: 20 mission tasks × 53 skills,
75 positives, hand-labelled. Measured 2026-09-21:

  lexical overlap        AUROC 0.851  [email protected] 0.47  top-k 48/75  ECE 0.095
  jev (named wording)    AUROC 0.989  [email protected] 0.84  top-k 63/75  ECE 0.064  213 ms
  jev (plain wording)    AUROC 0.970  [email protected] 0.66  top-k 52/75
  nli mnli-base          AUROC 0.790  [email protected] 0.28  top-k 38/75  ECE 0.263  1.5 s
  nli zeroshot-v2        AUROC 0.782  [email protected] 0.43  top-k 39/75  ECE 0.054  1.2 s

The vendor's calibration claim survives our data; the local cross-encoder
ranks below keyword overlap on either checkpoint or wording and is kept
as the measured negative, not shipped. A local backend would need the
logit-readout route over the fleet's 9B model — a separate spike.

Shadow: one Jev call per phase launch (spawned, 10 s cap, silent without
TYPESAFE_API_KEY) records a skill.triage event; the Skill-Use report
carries triage_p beside each skill's Trigger verdict. It selects nothing.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz

.gitignore

@@ -31,3 +31,4 @@ deploy/compose/.env.*
 # deploy/compose/docker-compose.override.yml is TRACKED as of 2026-09-18: it
 # holds the fixes for the five local bring-up gaps and every credential in it is
 # a ${VAR:?} reference into .env. It lived only on one laptop until then.
+crates/cm-decide/eval/out/

Cargo.toml

@@ -7,6 +7,7 @@ members = [
     "crates/cm-config",
     "crates/cm-db",
     "crates/cm-llm",
+    "crates/cm-decide",
     "crates/cm-runtime",
     "crates/cm-tools",
     "crates/cm-safety",

crates/cm-api/Cargo.toml

@@ -31,6 +31,7 @@ cm-billing = { path = "../cm-billing" }
 cm-brain = { path = "../cm-brain" }
 cm-config = { path = "../cm-config" }
 cm-db = { path = "../cm-db" }
+cm-decide = { path = "../cm-decide" }
 cm-domain = { path = "../cm-domain" }
 cm-files = { path = "../cm-files" }
 tar = { workspace = true }

crates/cm-api/src/lib.rs

@@ -57,6 +57,7 @@ pub mod container_tool_hooks;
 pub mod gateway_preflight;
 pub mod skill_delivery;
 pub mod skill_self_authoring;
+pub mod skill_triage;
 pub mod skill_use;
 pub mod skills_loader;
 pub mod subscription;

crates/cm-api/src/phase_runner.rs

@@ -1231,6 +1231,11 @@ async fn launch_phase(
         .await
         .unwrap_or(None);
     let task = phase_task_text(kind, title, description, phase_task, has_repo);
+    // Shadow skill triage on the task as the operator wrote it — before the
+    // judge's guidance and the project memory are appended, because those
+    // are not what a skill's `when_to_use` describes. Spawned; records an
+    // event; changes nothing.
+    crate::skill_triage::spawn(pool.clone(), mission_id, phase_id, workspace_id, task.clone());
     let task = match prior {
         Some((iter, false, guidance)) => format!(
             "{task}\n\nPASS {} DID NOT SATISFY THE COMPLETION CONDITION. What is \

crates/cm-api/src/skill_triage.rs

@@ -0,0 +1,131 @@
+//! Skill triage, in shadow: which of the visible skills a phase's task calls
+//! for, by a calibrated decision model, recorded beside what the agent then
+//! actually read.
+//!
+//! SRA-Bench (arXiv 2604.24594) found agents load skills at the same rate
+//! whether or not one applies — the bottleneck is knowing WHEN, and the
+//! agent's only signal today is the `when_to_use` line in its own prompt. A
+//! host-side oracle that answers the same question in 200 ms is the thing
+//! to measure against that. `cm_decide::jev` scored AUROC 0.989 on the
+//! labelled set (`crates/cm-decide/eval`); this records its answer per phase
+//! as a `skill.triage` event and the Skill-Use scorer reads it back next to
+//! the agent's Trigger. It selects nothing: the files arm still installs
+//! every visible skill. Promotion to a real selector is a later, measured
+//! step, once the agreement numbers from real missions say what the
+//! oracle's misses cost.
+//!
+//! One call per phase launch, spawned so the launch never waits on it, and
+//! silent when `TYPESAFE_API_KEY` is unset. The key never leaves the server.
+
+use std::collections::BTreeMap;
+
+use cm_decide::{Answer, Decider};
+use sqlx::PgPool;
+use uuid::Uuid;
+
+pub const EVENT: &str = "skill.triage";
+
+/// How long a shadow decision may take before it is dropped. Jev measures
+/// ~200 ms; a backend that takes ten seconds is not the one to shadow.
+const TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10);
+
+/// Fire the triage for one phase and record it. Best-effort throughout: a
+/// missing key, a failed call, or a timeout leaves no event and one log line.
+pub fn spawn(pool: PgPool, mission_id: Uuid, phase_id: Uuid, workspace_id: Uuid, task: String) {
+    let Some(jev) = cm_decide::jev::Jev::from_env() else {
+        return;
+    };
+    tokio::spawn(async move {
+        let skills = match cm_db::repo::skills_catalog::list_visible(&pool, workspace_id).await {
+            Ok(s) => s,
+            Err(e) => {
+                eprintln!("skill_triage: could not list skills for {mission_id}: {e}");
+                return;
+            }
+        };
+        let questions: BTreeMap<String, cm_decide::Question> = skills
+            .iter()
+            .map(|s| {
+                (
+                    s.name.clone(),
+                    cm_decide::triage::question(&s.name, s.when_to_use.as_deref().unwrap_or(&s.description)),
+                )
+            })
+            .collect();
+        if questions.is_empty() {
+            return;
+        }
+        let decision = match tokio::time::timeout(TIMEOUT, jev.decide(&task, &questions)).await {
+            Ok(Ok(d)) => d,
+            Ok(Err(e)) => {
+                eprintln!("skill_triage: {} failed for phase {phase_id}: {e}", jev.name());
+                return;
+            }
+            Err(_) => {
+                eprintln!("skill_triage: {} timed out for phase {phase_id}", jev.name());
+                return;
+            }
+        };
+        let probabilities: BTreeMap<&str, f64> = decision
+            .answers
+            .iter()
+            .filter_map(|(k, a)| match a {
+                Answer::Noul { noul } => Some((k.as_str(), *noul)),
+                _ => None,
+            })
+            .collect();
+        let applies = probabilities
+            .iter()
+            .filter(|(_, p)| **p >= cm_decide::triage::APPLIES_AT)
+            .count();
+        eprintln!(
+            "skill_triage: phase {phase_id} — {} says {applies} of {} skills apply ({} ms, {} tokens)",
+            decision.model,
+            probabilities.len(),
+            decision.latency.as_millis(),
+            decision.usage.map(|u| u.input_tokens).unwrap_or(0),
+        );
+        crate::mission_events::record(
+            &pool,
+            crate::mission_events::MissionEvent::new(mission_id, EVENT)
+                .phase(phase_id)
+                .detail(serde_json::json!({
+                    "backend": jev.name(),
+                    "model": decision.model,
+                    "wording": cm_decide::triage::WORDING,
+                    "latency_ms": decision.latency.as_millis() as u64,
+                    "input_tokens": decision.usage.map(|u| u.input_tokens),
+                    "applies_at": cm_decide::triage::APPLIES_AT,
+                    "skills": probabilities,
+                })),
+        )
+        .await;
+    });
+}
+
+/// The recorded triage for a mission: skill → highest probability any phase
+/// gave it. Empty when no event was recorded (no key, or before this existed).
+pub async fn recorded(pool: &PgPool, mission_id: Uuid) -> BTreeMap<String, f64> {
+    let rows: Vec<(serde_json::Value,)> = sqlx::query_as(
+        "SELECT detail FROM mission_events WHERE mission_id = $1 AND kind = $2 ORDER BY id",
+    )
+    .bind(mission_id)
+    .bind(EVENT)
+    .fetch_all(pool)
+    .await
+    .unwrap_or_default();
+    let mut out: BTreeMap<String, f64> = BTreeMap::new();
+    for (detail,) in rows {
+        if let Some(map) = detail.get("skills").and_then(|s| s.as_object()) {
+            for (name, p) in map {
+                if let Some(p) = p.as_f64() {
+                    let e = out.entry(name.clone()).or_insert(0.0);
+                    if p > *e {
+                        *e = p;
+                    }
+                }
+            }
+        }
+    }
+    out
+}

crates/cm-api/src/skill_use.rs

@@ -99,6 +99,13 @@ pub struct SkillUse {
     pub trigger: Verdict,
     pub compliance: Verdict,
     pub boundary: Verdict,
+    /// What the shadow triage said BEFORE the phase ran: the probability
+    /// that this skill applies to the task (`skill_triage`). `None` when no
+    /// triage was recorded. Read next to `trigger`: a high probability with a
+    /// skipped skill is a miss by the agent or by the oracle, and only real
+    /// missions say which.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub triage_p: Option<f64>,
 }
 
 /// The skills a prompt actually delivered.
@@ -305,6 +312,7 @@ pub fn score(
                 compliance,
                 boundary,
                 skill,
+                triage_p: None,
             }
         })
         .collect()
@@ -1120,14 +1128,19 @@ pub async fn score_mission(
     .into_iter()
     .collect();
 
-    Ok(score(&prompts, &Evidence::new(&outputs, &tools), &|name| {
+    let triage = crate::skill_triage::recorded(pool, mission_id).await;
+    let mut scores = score(&prompts, &Evidence::new(&outputs, &tools), &|name| {
         kinds
             .get(name)
             .cloned()
             // A skill in a prompt with no catalogue row was delivered and then
             // deleted. Naming that explicitly beats defaulting it to builtin.
             .unwrap_or_else(|| "unknown (no catalogue row)".to_string())
-    }))
+    });
+    for s in &mut scores {
+        s.triage_p = triage.get(&s.skill).copied();
+    }
+    Ok(scores)
 }
 
 #[cfg(test)]

crates/cm-decide/Cargo.toml

@@ -0,0 +1,36 @@
+[package]
+name = "cm-decide"
+version = "0.1.0"
+edition.workspace = true
+rust-version.workspace = true
+license.workspace = true
+publish.workspace = true
+description = "Typed, calibrated gut-check decisions (Choice / Score / Noul) behind one trait, with a hosted and a local backend"
+
+[features]
+default = ["jev"]
+# TypeSafe's Jev over HTTP.
+jev = ["dep:reqwest"]
+# A local NLI cross-encoder (DeBERTa-v3 MNLI) run in-process with candle.
+nli = ["dep:candle-core", "dep:candle-nn", "dep:candle-transformers", "dep:tokenizers", "dep:hf-hub"]
+metal = ["candle-core/metal", "candle-nn/metal", "candle-transformers/metal"]
+cuda = ["candle-core/cuda", "candle-nn/cuda", "candle-transformers/cuda"]
+
+[dependencies]
+async-trait = "0.1"
+serde = { workspace = true }
+serde_json = { workspace = true }
+thiserror = { workspace = true }
+tokio = { workspace = true }
+reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"], optional = true }
+candle-core = { version = "0.11", optional = true }
+candle-nn = { version = "0.11", optional = true }
+candle-transformers = { version = "0.11", optional = true }
+tokenizers = { version = "0.21", default-features = false, features = ["onig"], optional = true }
+hf-hub = { version = "0.4", optional = true }
+
+[dev-dependencies]
+tokio = { workspace = true }
+
+[lints]
+workspace = true

crates/cm-decide/eval/skill-triage.json

@@ -0,0 +1,45 @@
+{
+  "_about": "Labelled skill-triage cases: for each mission task, which of the skills under skills/ an agent should read. Labelled by hand on 2026-09-21 against each skill's when_to_use line; role-pinned skills are marked positive when the task is the kind of work the pin is for (a Rust coding task pins the Rust and commit-protocol skills). The set is the bar every triage backend is measured against — disputed rows are the eval's business, not the backend's.",
+  "cases": [
+    {"id": "fc-latency-brief", "task": "Write a research brief on Firecracker microVM cold-start latency. Sweep the open web for measurements, prefer primary sources (papers, vendor docs, benchmark repos) over aggregators, and make every empirical claim traceable to a source. Deliver research/BRIEF.md.",
+     "positives": ["web-search-triage", "scientific-writing-conventions", "structured-paper-summary"]},
+    {"id": "continuous-research-digest", "task": "Process this week's harvest manifest for the Continuous Research mission: drop items we have covered before, rank the rest by signal, decide which matter to the platform and why, and write the digest entries into the valhalla vault under the usual conventions.",
+     "positives": ["arxiv-daily", "duplicate-detection", "signal-to-noise-ranking", "paper-to-project-relevance", "executive-summary-writing", "structured-paper-summary", "obsidian-vault-conventions"]},
+    {"id": "podcast-script", "task": "Script phase: turn research/analysis.md into script.md and episode.json for the Continuous Research episode — two hosts, plain speech, every claim from the analysis and nothing invented.",
+     "positives": ["podcast-dialogue-writing"]},
+    {"id": "axum-paginated-endpoint", "task": "Add GET /api/missions/{id}/evaluations to the Rust axum server, returning a paginated list of judge verdicts for the mission. Design the contract first, write the tests before the handler, keep commits small, and commit on the mission branch.",
+     "positives": ["api-pagination-day-1", "openapi-contract-first", "tdd-red-green-refactor", "cargo-test-driven-development", "rust-error-handling", "rust-async-tokio-idioms", "write-rust-current-edition", "workspace-repo-commit-protocol", "small-focused-commits", "int-xx-marker-protocol"]},
+    {"id": "slow-postgres-query", "task": "The query behind /api/world/live over mission_events has become slow. Find out why with EXPLAIN ANALYZE, decide whether an index fixes it, add it as a forward-only migration, and cover the query with an integration test against a real Postgres.",
+     "positives": ["postgres-explain-analyze", "postgres-index-selection", "postgres-migrations-forward-only", "postgres-integration-testing", "workspace-repo-commit-protocol", "small-focused-commits"]},
+    {"id": "criterion-baseline", "task": "Benchmark the add hot path with criterion and record the measured baseline (ns per iteration, harness, host) in BASELINE.md at the repository root. Commit the bench and the file.",
+     "positives": ["criterion-benchmarking", "write-rust-current-edition", "workspace-repo-commit-protocol", "small-focused-commits"]},
+    {"id": "security-scan", "task": "Security phase: run cargo audit against the advisory database and gitleaks over the full history, and write SECURITY.md naming each tool, its version, and the count of findings, with a line per finding.",
+     "positives": ["cargo-audit-workflow", "secret-scanning-gitleaks", "workspace-repo-commit-protocol"]},
+    {"id": "review-judge-pr", "task": "Review the diff that adds the commit-first round to the evaluator. Report defects, missing tests, and anything that weakens an existing guarantee; do not edit the code.",
+     "positives": ["code-review-checklist"]},
+    {"id": "map-unfamiliar-repo", "task": "Before changing anything, map the yc-software/qm repository: its crates, the entry points, how a request flows from the HTTP layer to the executor, and where a TurnExecutor seam would go. Write ARCHITECTURE.md.",
+     "positives": ["ast-grep-repo-index", "request-lifecycle-tracing"]},
+    {"id": "regression-forensics", "task": "Mission resume started dropping the runtime pairing code sometime in August. Find the commit that introduced the behaviour, explain why the code is the way it is, and propose the smallest fix.",
+     "positives": ["git-log-forensics", "request-lifecycle-tracing"]},
+    {"id": "react-metrics-panel", "task": "Build the per-agent metrics band for the agents page: a React 19 server component in the Next.js 15 app, styled with Tailwind v4, showing loading, empty, error and populated states, keyboard-navigable and screen-reader labelled. Commit on the mission branch.",
+     "positives": ["react-19-server-components", "tailwind-v4-idioms", "component-4-state-model", "a11y-checklist", "workspace-repo-commit-protocol", "small-focused-commits"]},
+    {"id": "playwright-login", "task": "Write Playwright end-to-end tests for the login flow: success, wrong password, and session expiry. Make them stable under CI's slower machines.",
+     "positives": ["playwright-e2e-patterns", "workspace-repo-commit-protocol"]},
+    {"id": "expo-missions-list", "task": "Start the mobile app: an Expo React Native project showing the missions list as a fast scrollable feed, with platform-appropriate navigation on iOS and Android.",
+     "positives": ["expo-managed-vs-bare", "rn-flashlist-perf", "mobile-platform-conventions", "component-4-state-model", "workspace-repo-commit-protocol"]},
+    {"id": "mobile-e2e-setup", "task": "Set up end-to-end runs for the mobile app on the iOS simulator and an Android emulator, and get the login test green on both.",
+     "positives": ["mobile-e2e-and-simulators", "playwright-e2e-patterns"]},
+    {"id": "cuda-reduction", "task": "Write a CUDA reduction kernel for the histogram step, profile it, reason about memory coalescing and occupancy, place it on the roofline, and show with a benchmark that it beats the current implementation.",
+     "positives": ["gpu-kernel-authoring", "gpu-coalescing-and-occupancy", "gpu-profiling-workflow", "roofline-model", "criterion-benchmarking", "workspace-repo-commit-protocol"]},
+    {"id": "threejs-world-scene", "task": "Plan the three.js scene graph for the Gource-style World view, write the particle shader in GLSL with a WGSL port, and fix the frame drops the replay scrubber causes.",
+     "positives": ["scene-graph-planning", "shader-authoring-glsl-wgsl", "threejs-perf-and-teardown", "webgl-frame-profiling"]},
+    {"id": "agent-level-up", "task": "Inspect the researcher agent's .brain, decide whether its definition matches what it actually does, propose a level-up changing its skills and model, and show with baseline metrics whether the last change helped.",
+     "positives": ["brain-file-reading", "level-up-proposal-shape", "metrics-baseline-comparison"]},
+    {"id": "planner-decompose", "task": "Planner: read research/IMPLEMENTATION_BRIEF.md and decompose it into INT-XX items with the marker protocol every coder role must follow; output the task list, do not implement anything.",
+     "positives": ["decompose-int-items", "int-xx-marker-protocol"]},
+    {"id": "paper-draft-novelty", "task": "Draft the evaluation section of the paper from our measured skill-retrieval numbers, and check whether the files-arm delivery idea is novel before we claim it.",
+     "positives": ["scientific-writing-conventions", "prior-art-search"]},
+    {"id": "migration-with-backfill", "task": "Add a nullable mission_id column to auth_sessions as a forward-only migration with a partial index, backfill nothing, and prove with an integration test against Postgres that inserts and the cascade delete behave.",
+     "positives": ["postgres-migrations-forward-only", "postgres-integration-testing", "workspace-repo-commit-protocol", "small-focused-commits"]}
+  ]
+}

crates/cm-decide/src/bin/decide-eval.rs

@@ -0,0 +1,348 @@
+//! Measure a decision backend against labelled cases.
+//!
+//! The judge got an eval before it got trusted (`scripts/judge-eval.sh`);
+//! a triage model gets the same. Every backend answers the SAME Noul per
+//! (task, skill) pair, and is scored on ranking (AUROC), on the operating
+//! point (best-F1 threshold and F1 at 0.5), and on calibration (Brier, ECE)
+//! — because a calibrated middle band is the whole reason to have this tier,
+//! and a backend that ranks well but says 0.9 to everything has none.
+//!
+//!   decide-eval [--backend jev|nli|lexical|all] [--wording named|plain]
+//!               [--set path] [--skills dir] [--dump dir]
+//!
+//! `--dump` writes every pair's probability so a temperature can be fitted
+//! offline without re-running the model.
+
+use std::collections::BTreeMap;
+use std::path::{Path, PathBuf};
+use std::time::Instant;
+
+use cm_decide::{Answer, DecideError, Decider, Decision, Question};
+use serde::{Deserialize, Serialize};
+
+#[derive(Deserialize)]
+struct EvalSet {
+    cases: Vec<Case>,
+}
+
+#[derive(Deserialize, Clone)]
+struct Case {
+    id: String,
+    task: String,
+    positives: Vec<String>,
+}
+
+#[derive(Clone)]
+struct Skill {
+    name: String,
+    when_to_use: String,
+}
+
+/// The question a backend gets. `--wording` picks the template; a backend
+/// is reported under the wording it was run with, and each backend ships
+/// with the wording that measured best FOR IT — a prompt is part of the
+/// backend, and an NLI cross-encoder and a decision model do not want the
+/// same sentence. Both are run on both so the table says so.
+fn question_for(skill: &Skill, wording: &str) -> Question {
+    let text = match wording {
+        // The shipped question, shared with the server's shadow path.
+        "named" => return cm_decide::triage::question(&skill.name, &skill.when_to_use),
+        // The when_to_use line alone, second person rewritten to the agent,
+        // as a plain declarative the NLI head was trained on.
+        "plain" => third_person(&skill.when_to_use),
+        other => panic!("unknown wording {other}"),
+    };
+    Question::noul(text)
+}
+
+/// "You're the tester on a mobile team" → "The agent is the tester on a
+/// mobile team". Crude on purpose: it is a rewrite of a dozen fixed
+/// openings, not a grammar, and it exists so the NLI hypothesis reads like
+/// an MNLI hypothesis.
+fn third_person(when: &str) -> String {
+    let w = when.trim();
+    let rules: &[(&str, &str)] = &[
+        ("You're the ", "The agent is the "),
+        ("You are the ", "The agent is the "),
+        ("You're a ", "The agent is a "),
+        ("You are a ", "The agent is a "),
+        ("You're on a ", "The agent is on a "),
+        ("You are on a ", "The agent is on a "),
+        ("You're ", "The agent is "),
+        ("You are ", "The agent is "),
+        ("You need ", "The agent needs "),
+        ("You have ", "The agent has "),
+    ];
+    for (from, to) in rules {
+        if let Some(rest) = w.strip_prefix(from) {
+            return format!("{to}{rest}");
+        }
+    }
+    format!("In this task, {}", w.trim_end_matches('.').to_string() + ".")
+}
+
+fn load_skills(dir: &Path) -> Vec<Skill> {
+    let mut out = Vec::new();
+    fn walk(dir: &Path, out: &mut Vec<Skill>) {
+        let Ok(rd) = std::fs::read_dir(dir) else { return };
+        let mut entries: Vec<_> = rd.flatten().collect();
+        entries.sort_by_key(|e| e.path());
+        for e in entries {
+            let p = e.path();
+            if p.is_dir() {
+                walk(&p, out);
+            } else if p.extension().is_some_and(|x| x == "md") {
+                let Ok(text) = std::fs::read_to_string(&p) else { continue };
+                let field = |k: &str| {
+                    text.lines()
+                        .find_map(|l| l.strip_prefix(k).map(|v| v.trim().trim_matches('"').to_string()))
+                };
+                if let (Some(name), Some(when)) = (field("name:"), field("when_to_use:")) {
+                    out.push(Skill { name, when_to_use: when });
+                }
+            }
+        }
+    }
+    walk(dir, &mut out);
+    out
+}
+
+/// Keyword overlap between the task and the skill's name + when_to_use:
+/// the bar a model has to clear to be worth a network call.
+struct Lexical;
+
+fn tokens(s: &str) -> std::collections::BTreeSet<String> {
+    s.to_ascii_lowercase()
+        .split(|c: char| !c.is_alphanumeric())
+        .filter(|w| w.len() > 3)
+        .map(str::to_string)
+        .collect()
+}
+
+#[async_trait::async_trait]
+impl Decider for Lexical {
+    fn name(&self) -> &str {
+        "lexical"
+    }
+    async fn decide(
+        &self,
+        state: &str,
+        questions: &BTreeMap<String, Question>,
+    ) -> Result<Decision, DecideError> {
+        let started = Instant::now();
+        let st = tokens(state);
+        let answers = questions
+            .iter()
+            .map(|(id, q)| {
+                let Question::Noul { instructions, .. } = q else { unreachable!() };
+                let ht = tokens(instructions);
+                let inter = st.intersection(&ht).count() as f64;
+                let p = if ht.is_empty() { 0.0 } else { (inter / ht.len() as f64 * 4.0).min(1.0) };
+                (id.clone(), Answer::Noul { noul: p })
+            })
+            .collect();
+        Ok(Decision { model: "lexical".into(), answers, usage: None, latency: started.elapsed() })
+    }
+}
+
+#[derive(Serialize)]
+struct Pair {
+    case: String,
+    skill: String,
+    label: bool,
+    p: f64,
+}
+
+#[derive(Default)]
+struct Report {
+    pairs: Vec<Pair>,
+    latency_ms: Vec<u128>,
+    input_tokens: u64,
+    per_case_topk_hits: Vec<(usize, usize)>,
+    errors: usize,
+}
+
+fn auroc(pairs: &[Pair]) -> f64 {
+    // Mann–Whitney: fraction of (positive, negative) pairs ranked correctly.
+    let pos: Vec<f64> = pairs.iter().filter(|p| p.label).map(|p| p.p).collect();
+    let neg: Vec<f64> = pairs.iter().filter(|p| !p.label).map(|p| p.p).collect();
+    if pos.is_empty() || neg.is_empty() {
+        return f64::NAN;
+    }
+    let mut s = 0.0;
+    for a in &pos {
+        for b in &neg {
+            s += if a > b { 1.0 } else if a == b { 0.5 } else { 0.0 };
+        }
+    }
+    s / (pos.len() * neg.len()) as f64
+}
+
+fn f1_at(pairs: &[Pair], t: f64) -> (f64, f64, f64) {
+    let (mut tp, mut fp, mut fn_) = (0.0, 0.0, 0.0);
+    for p in pairs {
+        match (p.p >= t, p.label) {
+            (true, true) => tp += 1.0,
+            (true, false) => fp += 1.0,
+            (false, true) => fn_ += 1.0,
+            _ => {}
+        }
+    }
+    let prec = if tp + fp > 0.0 { tp / (tp + fp) } else { 0.0 };
+    let rec = if tp + fn_ > 0.0 { tp / (tp + fn_) } else { 0.0 };
+    let f1 = if prec + rec > 0.0 { 2.0 * prec * rec / (prec + rec) } else { 0.0 };
+    (f1, prec, rec)
+}
+
+fn brier(pairs: &[Pair]) -> f64 {
+    pairs.iter().map(|p| (p.p - if p.label { 1.0 } else { 0.0 }).powi(2)).sum::<f64>() / pairs.len() as f64
+}
+
+/// Expected calibration error, ten equal-width bins.
+fn ece(pairs: &[Pair]) -> f64 {
+    let mut bins = vec![(0usize, 0.0f64, 0.0f64); 10];
+    for p in pairs {
+        let b = ((p.p * 10.0).floor() as usize).min(9);
+        bins[b].0 += 1;
+        bins[b].1 += p.p;
+        bins[b].2 += if p.label { 1.0 } else { 0.0 };
+    }
+    let n = pairs.len() as f64;
+    bins.iter()
+        .filter(|(c, _, _)| *c > 0)
+        .map(|(c, sp, sy)| (*c as f64 / n) * ((sp / *c as f64) - (sy / *c as f64)).abs())
+        .sum()
+}
+
+async fn run(backend: &dyn Decider, cases: &[Case], skills: &[Skill], wording: &str) -> Report {
+    let mut r = Report::default();
+    let questions: BTreeMap<String, Question> =
+        skills.iter().map(|s| (s.name.clone(), question_for(s, wording))).collect();
+    for case in cases {
+        let d = match backend.decide(&case.task, &questions).await {
+            Ok(d) => d,
+            Err(e) => {
+                eprintln!("  {}: {} failed: {e}", backend.name(), case.id);
+                r.errors += 1;
+                continue;
+            }
+        };
+        r.latency_ms.push(d.latency.as_millis());
+        if let Some(u) = d.usage {
+            r.input_tokens += u.input_tokens;
+        }
+        let mut ranked: Vec<(String, f64)> = Vec::new();
+        for s in skills {
+            let p = match d.answers.get(&s.name) {
+                Some(Answer::Noul { noul }) => *noul,
+                _ => 0.0,
+            };
+            let label = case.positives.iter().any(|x| x == &s.name);
+            r.pairs.push(Pair { case: case.id.clone(), skill: s.name.clone(), label, p });
+            ranked.push((s.name.clone(), p));
+        }
+        ranked.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap());
+        let k = case.positives.len();
+        let hits = ranked.iter().take(k).filter(|(n, _)| case.positives.contains(n)).count();
+        r.per_case_topk_hits.push((hits, k));
+    }
+    r
+}
+
+fn print_report(name: &str, r: &Report) {
+    if r.pairs.is_empty() {
+        println!("{name:<24} no results ({} errors)", r.errors);
+        return;
+    }
+    let best = (5..=95)
+        .map(|i| i as f64 / 100.0)
+        .map(|t| (t, f1_at(&r.pairs, t)))
+        .max_by(|a, b| a.1 .0.partial_cmp(&b.1 .0).unwrap())
+        .unwrap();
+    let (f1_half, p_half, r_half) = f1_at(&r.pairs, 0.5);
+    let topk: (usize, usize) = r.per_case_topk_hits.iter().fold((0, 0), |a, b| (a.0 + b.0, a.1 + b.1));
+    let lat = if r.latency_ms.is_empty() { 0 } else { r.latency_ms.iter().sum::<u128>() / r.latency_ms.len() as u128 };
+    println!(
+        "{name:<24} AUROC {:.3}  [email protected] {:.2} (P {:.2} R {:.2})  bestF1 {:.2}@{:.2}  top-k {}/{}  Brier {:.3}  ECE {:.3}  {} ms/call  {} tok  errors {}",
+        auroc(&r.pairs), f1_half, p_half, r_half, best.1 .0, best.0, topk.0, topk.1,
+        brier(&r.pairs), ece(&r.pairs), lat, r.input_tokens, r.errors
+    );
+}
+
+#[tokio::main]
+async fn main() {
+    let mut args = std::env::args().skip(1);
+    let (mut backend, mut set, mut skills_dir, mut dump, mut wording) = (
+        "all".to_string(),
+        PathBuf::from("crates/cm-decide/eval/skill-triage.json"),
+        PathBuf::from("skills"),
+        None::<PathBuf>,
+        "named".to_string(),
+    );
+    while let Some(a) = args.next() {
+        match a.as_str() {
+            "--backend" => backend = args.next().unwrap_or_default(),
+            "--wording" => wording = args.next().unwrap_or_default(),
+            "--set" => set = args.next().map(PathBuf::from).unwrap_or(set),
+            "--skills" => skills_dir = args.next().map(PathBuf::from).unwrap_or(skills_dir),
+            "--dump" => dump = args.next().map(PathBuf::from),
+            other => {
+                eprintln!("unknown arg {other}");
+                std::process::exit(2);
+            }
+        }
+    }
+    let text = std::fs::read_to_string(&set).expect("read eval set");
+    let set: EvalSet = serde_json::from_str(&text).expect("parse eval set");
+    let skills = load_skills(&skills_dir);
+    // Every positive must name a real skill, or the label is a typo scored
+    // as a miss against every backend.
+    for c in &set.cases {
+        for p in &c.positives {
+            assert!(skills.iter().any(|s| &s.name == p), "case {}: unknown skill {p:?}", c.id);
+        }
+    }
+    println!(
+        "{} cases × {} skills = {} pairs, {} positive\n",
+        set.cases.len(),
+        skills.len(),
+        set.cases.len() * skills.len(),
+        set.cases.iter().map(|c| c.positives.len()).sum::<usize>()
+    );
+
+    let mut backends: Vec<Box<dyn Decider>> = Vec::new();
+    if backend == "all" || backend == "lexical" {
+        backends.push(Box::new(Lexical));
+    }
+    #[cfg(feature = "jev")]
+    if backend == "all" || backend == "jev" {
+        match cm_decide::jev::Jev::from_env() {
+            Some(j) => backends.push(Box::new(j)),
+            None => eprintln!("jev: TYPESAFE_API_KEY unset — skipped"),
+        }
+    }
+    #[cfg(feature = "nli")]
+    if backend == "all" || backend == "nli" {
+        let t0 = Instant::now();
+        match cm_decide::nli::Nli::load(cm_decide::nli::NliConfig::from_env()) {
+            Ok(n) => {
+                eprintln!("nli: loaded {} in {:?}", n.name(), t0.elapsed());
+                backends.push(Box::new(n));
+            }
+            Err(e) => eprintln!("nli: could not load — {e}"),
+        }
+    }
+    if backends.is_empty() {
+        eprintln!("no backend to run");
+        std::process::exit(2);
+    }
+    for b in &backends {
+        let r = run(b.as_ref(), &set.cases, &skills, &wording).await;
+        print_report(&format!("{} [{wording}]", b.name()), &r);
+        if let Some(dir) = &dump {
+            std::fs::create_dir_all(dir).ok();
+            let path = dir.join(format!("{}-{wording}.json", b.name().replace([':', '/'], "-")));
+            std::fs::write(&path, serde_json::to_string_pretty(&r.pairs).unwrap()).ok();
+        }
+    }
+}

crates/cm-decide/src/jev.rs

@@ -0,0 +1,206 @@
+//! TypeSafe's Jev over `POST /v1/systemone`.
+//!
+//! Text-only, 64 K context (32 K for the state), no tools, no reasoning.
+//! Priced on input tokens only. Not trained on customer requests. The key is
+//! `TYPESAFE_API_KEY` and lives in the server's environment — it is never
+//! handed to a mission container, and this client is only ever called from
+//! the server.
+
+use std::collections::BTreeMap;
+use std::time::{Duration, Instant};
+
+use serde::Deserialize;
+
+use crate::{Answer, DecideError, Decider, Decision, Question, Usage};
+
+pub const ENDPOINT: &str = "https://api.typesafe.ai/v1/systemone";
+pub const DEFAULT_MODEL: &str = "jev-latest";
+
+pub struct Jev {
+    client: reqwest::Client,
+    key: String,
+    model: String,
+}
+
+impl Jev {
+    /// From `TYPESAFE_API_KEY` (and `TYPESAFE_MODEL`, default `jev-latest`).
+    /// `None` when the key is unset: the caller decides whether that means
+    /// "skip the decision" or "use another backend" — it never means guess.
+    pub fn from_env() -> Option<Self> {
+        let key = std::env::var("TYPESAFE_API_KEY").ok()?;
+        let key = key.trim().to_string();
+        if key.is_empty() {
+            return None;
+        }
+        let model = std::env::var("TYPESAFE_MODEL")
+            .ok()
+            .filter(|m| !m.trim().is_empty())
+            .unwrap_or_else(|| DEFAULT_MODEL.to_string());
+        Some(Self::new(key, model))
+    }
+
+    pub fn new(key: String, model: String) -> Self {
+        let client = reqwest::Client::builder()
+            .timeout(Duration::from_secs(30))
+            .build()
+            .expect("reqwest client");
+        Self { client, key, model }
+    }
+}
+
+#[derive(Deserialize)]
+struct Reply {
+    model: String,
+    answers: BTreeMap<String, RawAnswer>,
+    #[serde(default)]
+    usage: Option<RawUsage>,
+}
+
+#[derive(Deserialize)]
+struct RawUsage {
+    #[serde(default)]
+    input_tokens: u64,
+    #[serde(default)]
+    output_tokens: u64,
+}
+
+/// Their answer shapes. Score keys `probabilities` by level number as a
+/// STRING ("0", "1", …); a `legend` mirrors the criteria and is dropped.
+#[derive(Deserialize)]
+#[serde(tag = "type", rename_all = "lowercase")]
+enum RawAnswer {
+    Choice {
+        choice: String,
+        probabilities: BTreeMap<String, f64>,
+        confidence: f64,
+    },
+    Score {
+        score: f64,
+        probabilities: BTreeMap<String, f64>,
+        confidence: f64,
+    },
+    Noul {
+        noul: f64,
+    },
+}
+
+impl RawAnswer {
+    fn into_answer(self) -> Result<Answer, DecideError> {
+        Ok(match self {
+            RawAnswer::Choice { choice, probabilities, confidence } => Answer::Choice {
+                choice,
+                probabilities,
+                confidence,
+            },
+            RawAnswer::Score { score, probabilities, confidence } => {
+                let mut levels: Vec<(usize, f64)> = probabilities
+                    .into_iter()
+                    .map(|(k, v)| {
+                        k.parse::<usize>()
+                            .map(|i| (i, v))
+                            .map_err(|_| DecideError::Shape(format!("score level key {k:?}")))
+                    })
+                    .collect::<Result<_, _>>()?;
+                levels.sort_by_key(|(i, _)| *i);
+                Answer::Score {
+                    score,
+                    probabilities: levels.into_iter().map(|(_, p)| p).collect(),
+                    confidence,
+                }
+            }
+            RawAnswer::Noul { noul } => Answer::Noul { noul },
+        })
+    }
+}
+
+#[async_trait::async_trait]
+impl Decider for Jev {
+    fn name(&self) -> &str {
+        "jev"
+    }
+
+    async fn decide(
+        &self,
+        state: &str,
+        questions: &BTreeMap<String, Question>,
+    ) -> Result<Decision, DecideError> {
+        let body = serde_json::json!({
+            "state": state,
+            "model": self.model,
+            "questions": questions,
+        });
+        let started = Instant::now();
+        let resp = self
+            .client
+            .post(ENDPOINT)
+            .bearer_auth(&self.key)
+            .json(&body)
+            .send()
+            .await
+            .map_err(|e| DecideError::Transport(e.to_string()))?;
+        let status = resp.status();
+        let text = resp
+            .text()
+            .await
+            .map_err(|e| DecideError::Transport(e.to_string()))?;
+        if !status.is_success() {
+            return Err(DecideError::Model(format!(
+                "HTTP {status}: {}",
+                text.chars().take(300).collect::<String>()
+            )));
+        }
+        let reply: Reply =
+            serde_json::from_str(&text).map_err(|e| DecideError::Shape(e.to_string()))?;
+        let mut answers = BTreeMap::new();
+        for (id, raw) in reply.answers {
+            answers.insert(id, raw.into_answer()?);
+        }
+        Ok(Decision {
+            model: reply.model,
+            answers,
+            usage: reply.usage.map(|u| Usage {
+                input_tokens: u.input_tokens,
+                output_tokens: u.output_tokens,
+            }),
+            latency: started.elapsed(),
+        })
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    /// The documented response, read back into our shapes — including the
+    /// string-keyed Score levels, which arrive unordered in a JSON object.
+    #[test]
+    fn the_documented_reply_parses() {
+        let text = r#"{"model":"jev-1.13.0","answers":{
+            "department":{"type":"choice","choice":"technical","confidence":0.78,
+                "probabilities":{"technical":0.85,"sales":0.0,"billing":0.15}},
+            "frustration":{"type":"score","score":1.0,"confidence":1.0,
+                "legend":{"0":"calm","1":"frustrated","2":"angry"},
+                "probabilities":{"2":0.0,"0":0.0,"1":1.0}},
+            "is_urgent":{"type":"noul","noul":1.0}},
+            "usage":{"input_tokens":392,"output_tokens":65}}"#;
+        let reply: Reply = serde_json::from_str(text).unwrap();
+        let a = reply.answers.get("frustration").unwrap();
+        let RawAnswer::Score { .. } = a else { panic!() };
+        let converted: BTreeMap<String, Answer> = reply
+            .answers
+            .into_iter()
+            .map(|(k, v)| (k, v.into_answer().unwrap()))
+            .collect();
+        match &converted["frustration"] {
+            Answer::Score { probabilities, score, .. } => {
+                assert_eq!(probabilities, &vec![0.0, 1.0, 0.0]);
+                assert_eq!(*score, 1.0);
+            }
+            other => panic!("{other:?}"),
+        }
+        match &converted["department"] {
+            Answer::Choice { choice, .. } => assert_eq!(choice, "technical"),
+            other => panic!("{other:?}"),
+        }
+    }
+}

crates/cm-decide/src/lib.rs

@@ -0,0 +1,270 @@
+//! Typed gut-check decisions for the platform's code to branch on.
+//!
+//! ClawMates has had exactly two kinds of decision-maker: deterministic code,
+//! and a full LLM chat call parsed back into a boolean. There was no cheap,
+//! calibrated classifier in between — every model decision was forced binary,
+//! took seconds, and drew on a provider quota that has emptied twice. This
+//! crate is that middle tier, in the shape TypeSafe's Jev popularised
+//! (`docs.typesafe.ai`): a [`Question`] is a **Choice** over labelled options,
+//! a **Score** over ordered levels, or a **Noul** (a yes/no); an [`Answer`] is
+//! a probability distribution plus a confidence, never text.
+//!
+//! Two backends implement [`Decider`]: [`jev::Jev`] (hosted) and
+//! [`nli::Nli`] (a DeBERTa-v3 MNLI cross-encoder run in-process). Neither
+//! reasons or runs tools, and that is the point — the judge and the tool
+//! gate stay where they are. What goes here is triage: which skills apply to
+//! a task, whether an outbound action looks like exfiltration, which past
+//! verdict is relevant. Every use is measured on labelled cases first
+//! (`decide-eval`) and shipped in shadow mode before it changes behaviour.
+//!
+//! The `patterns` module carries the composition rules as code, because the
+//! vendor's advice is right and worth keeping regardless of vendor: ask every
+//! question in one call, gate on confidence, combine dimensions with weights
+//! your code owns.
+
+use std::collections::BTreeMap;
+use std::time::Duration;
+
+use serde::{Deserialize, Serialize};
+
+#[cfg(feature = "jev")]
+pub mod jev;
+#[cfg(feature = "nli")]
+pub mod nli;
+pub mod patterns;
+pub mod triage;
+
+/// One question, evaluated on its own against the state.
+///
+/// Each variant answers a different kind of question, and the shape of the
+/// answer follows: an option, a position, or a probability.
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+#[serde(tag = "type", rename_all = "lowercase")]
+pub enum Question {
+    /// One option from a fixed, unordered set. `criteria` maps an option name
+    /// to its description (`None` when the name is clear on its own).
+    Choice {
+        instructions: String,
+        criteria: BTreeMap<String, Option<String>>,
+    },
+    /// A position on a spectrum described in steps, low to high. At least two
+    /// levels; each is a situation, not a degree ("workaround exists", not
+    /// "moderately severe").
+    Score {
+        instructions: String,
+        criteria: Vec<String>,
+    },
+    /// Yes or no. `criteria` optionally says what a yes and a no look like.
+    Noul {
+        instructions: String,
+        #[serde(default, skip_serializing_if = "Option::is_none")]
+        criteria: Option<NoulCriteria>,
+    },
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+pub struct NoulCriteria {
+    #[serde(rename = "true")]
+    pub yes: String,
+    #[serde(rename = "false")]
+    pub no: String,
+}
+
+impl Question {
+    pub fn noul(instructions: impl Into<String>) -> Self {
+        Question::Noul {
+            instructions: instructions.into(),
+            criteria: None,
+        }
+    }
+
+    pub fn choice<K: Into<String>, V: Into<String>>(
+        instructions: impl Into<String>,
+        options: impl IntoIterator<Item = (K, Option<V>)>,
+    ) -> Self {
+        Question::Choice {
+            instructions: instructions.into(),
+            criteria: options
+                .into_iter()
+                .map(|(k, v)| (k.into(), v.map(Into::into)))
+                .collect(),
+        }
+    }
+
+    pub fn score<L: Into<String>>(
+        instructions: impl Into<String>,
+        levels: impl IntoIterator<Item = L>,
+    ) -> Self {
+        Question::Score {
+            instructions: instructions.into(),
+            criteria: levels.into_iter().map(Into::into).collect(),
+        }
+    }
+}
+
+/// The typed answer to one question.
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+#[serde(tag = "type", rename_all = "lowercase")]
+pub enum Answer {
+    Choice {
+        /// The option with the highest probability.
+        choice: String,
+        /// Every option; sums to 1.
+        probabilities: BTreeMap<String, f64>,
+        /// How peaked the distribution is: 1.0 on a single option, toward 0
+        /// as it flattens. See [`confidence_of`].
+        confidence: f64,
+    },
+    Score {
+        /// Probability-weighted position on the level line, 0 to `levels-1`.
+        score: f64,
+        /// Per level, in order; sums to 1.
+        probabilities: Vec<f64>,
+        confidence: f64,
+    },
+    Noul {
+        /// Probability that the answer is yes.
+        noul: f64,
+    },
+}
+
+impl Answer {
+    /// Confidence for the shapes that have one; a Noul's value already
+    /// describes its whole two-outcome distribution, so it reports how far
+    /// the value sits from 0.5.
+    pub fn confidence(&self) -> f64 {
+        match self {
+            Answer::Choice { confidence, .. } | Answer::Score { confidence, .. } => *confidence,
+            Answer::Noul { noul } => (noul - 0.5).abs() * 2.0,
+        }
+    }
+}
+
+/// What a backend returned for one call: every answer, who answered, what
+/// it cost. `usage` is `None` for a local backend (nothing is billed) and
+/// `latency` is wall-clock as this process saw it.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct Decision {
+    pub model: String,
+    pub answers: BTreeMap<String, Answer>,
+    #[serde(default)]
+    pub usage: Option<Usage>,
+    #[serde(with = "duration_ms")]
+    pub latency: Duration,
+}
+
+#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)]
+pub struct Usage {
+    pub input_tokens: u64,
+    pub output_tokens: u64,
+}
+
+mod duration_ms {
+    use serde::{Deserialize, Deserializer, Serializer};
+    use std::time::Duration;
+    pub fn serialize<S: Serializer>(d: &Duration, s: S) -> Result<S::Ok, S::Error> {
+        s.serialize_u64(d.as_millis() as u64)
+    }
+    pub fn deserialize<'de, D: Deserializer<'de>>(d: D) -> Result<Duration, D::Error> {
+        Ok(Duration::from_millis(u64::deserialize(d)?))
+    }
+}
+
+#[derive(Debug, thiserror::Error)]
+pub enum DecideError {
+    #[error("{0}")]
+    Config(String),
+    #[error("transport: {0}")]
+    Transport(String),
+    #[error("the backend answered in a shape this crate does not read: {0}")]
+    Shape(String),
+    #[error("model: {0}")]
+    Model(String),
+}
+
+/// A source of typed answers. Every question in `questions` is answered
+/// against the same `state` in one call; backends are expected to evaluate
+/// them independently, so an answer never depends on which other questions
+/// were asked.
+#[async_trait::async_trait]
+pub trait Decider: Send + Sync {
+    /// Stable name for logs and eval tables (`jev`, `nli`).
+    fn name(&self) -> &str;
+    async fn decide(
+        &self,
+        state: &str,
+        questions: &BTreeMap<String, Question>,
+    ) -> Result<Decision, DecideError>;
+}
+
+/// Confidence from a distribution: `1 − H(p) / ln(n)`, so a single peak is
+/// 1.0 and a flat spread is 0.0. The vendor documents only that confidence
+/// is "computed from how the probabilities are spread"; this is our
+/// definition and it is applied to the local backend's answers. Jev's own
+/// figure is passed through untouched, so the two are comparable in shape
+/// but not guaranteed identical in value.
+pub fn confidence_of(probabilities: &[f64]) -> f64 {
+    let n = probabilities.len();
+    if n < 2 {
+        return 1.0;
+    }
+    let h: f64 = probabilities
+        .iter()
+        .filter(|p| **p > 0.0)
+        .map(|p| -p * p.ln())
+        .sum();
+    (1.0 - h / (n as f64).ln()).clamp(0.0, 1.0)
+}
+
+/// Softmax over raw scores, numerically stable.
+pub fn softmax(logits: &[f64]) -> Vec<f64> {
+    let max = logits.iter().cloned().fold(f64::NEG_INFINITY, f64::max);
+    let exps: Vec<f64> = logits.iter().map(|l| (l - max).exp()).collect();
+    let sum: f64 = exps.iter().sum();
+    exps.iter().map(|e| e / sum).collect()
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn confidence_is_one_on_a_peak_and_zero_when_flat() {
+        assert!((confidence_of(&[1.0, 0.0, 0.0]) - 1.0).abs() < 1e-9);
+        assert!(confidence_of(&[1.0 / 3.0; 3]).abs() < 1e-9);
+        let mid = confidence_of(&[0.6, 0.4]);
+        assert!(mid > 0.0 && mid < 1.0);
+        // A two-outcome distribution and a Noul report the same certainty
+        // ordering: further from even is more confident.
+        assert!(
+            Answer::Noul { noul: 0.9 }.confidence() > Answer::Noul { noul: 0.6 }.confidence()
+        );
+    }
+
+    #[test]
+    fn softmax_sums_to_one_and_keeps_order() {
+        let p = softmax(&[2.0, 1.0, 0.1]);
+        assert!((p.iter().sum::<f64>() - 1.0).abs() < 1e-9);
+        assert!(p[0] > p[1] && p[1] > p[2]);
+    }
+
+    /// The wire shape matches the vendor's, so a question written for one
+    /// backend is the same question for the other.
+    #[test]
+    fn questions_serialise_in_the_vendor_shape() {
+        let q = Question::choice(
+            "Which team?",
+            [("billing", Some("charges")), ("returns", None)],
+        );
+        let v = serde_json::to_value(&q).unwrap();
+        assert_eq!(v["type"], "choice");
+        assert_eq!(v["criteria"]["billing"], "charges");
+        assert!(v["criteria"]["returns"].is_null());
+        let n = Question::Noul {
+            instructions: "Is it urgent?".into(),
+            criteria: Some(NoulCriteria { yes: "y".into(), no: "n".into() }),
+        };
+        let v = serde_json::to_value(&n).unwrap();
+        assert_eq!(v["criteria"]["true"], "y");
+    }
+}

crates/cm-decide/src/nli.rs

@@ -0,0 +1,349 @@
+//! A local decision model: a DeBERTa-v3 MNLI cross-encoder, in-process.
+//!
+//! A Noul IS an entailment probability — P(the hypothesis follows from the
+//! state) — and natural-language inference is the oldest working form of
+//! zero-shot classification. A Choice is one hypothesis per option with a
+//! softmax over their entailment logits; a Score is the same over ordered
+//! levels, then the probability-weighted position. Nothing here reasons or
+//! generates: one forward pass per (state, hypothesis) pair, the answer read
+//! from the three-way head.
+//!
+//! What this does NOT have is the vendor's calibration training. Its
+//! probabilities are as calibrated as MNLI made them, which on our domain is
+//! an open question until `decide-eval` answers it; temperature scaling on
+//! our own labelled cases is the standard fix and the `temperature` knob is
+//! where it goes.
+//!
+//! Model files come from the Hugging Face hub on first use
+//! (`CLAWMATES_NLI_MODEL`, default `cross-encoder/nli-deberta-v3-base`;
+//! `-xsmall` is 4× faster and worse) or from `CLAWMATES_NLI_MODEL_DIR` for a
+//! host with no egress. CPU by default; `metal`/`cuda` features select a GPU.
+
+use std::collections::BTreeMap;
+use std::path::PathBuf;
+use std::sync::Mutex;
+use std::time::Instant;
+
+use candle_core::{DType, Device, Tensor};
+use candle_nn::VarBuilder;
+use candle_transformers::models::debertav2::{Config, DebertaV2SeqClassificationModel, Id2Label};
+use tokenizers::{EncodeInput, InputSequence, PaddingParams, Tokenizer, TruncationParams};
+
+use crate::{confidence_of, softmax, Answer, DecideError, Decider, Decision, Question};
+
+pub const DEFAULT_MODEL: &str = "cross-encoder/nli-deberta-v3-base";
+
+/// Token budget per pair. DeBERTa-v3 was trained at 512; the STATE is what
+/// gets cut when a pair is too long (`OnlyFirst`), so the hypothesis — the
+/// question — always survives whole.
+const MAX_TOKENS: usize = 512;
+
+pub struct Nli {
+    model: Mutex<DebertaV2SeqClassificationModel>,
+    tokenizer: Tokenizer,
+    device: Device,
+    /// Index of each MNLI label in the head, read from `config.json`.
+    entail: usize,
+    contra: usize,
+    /// Divides the logits before the softmax. 1.0 = the model's own
+    /// calibration; fitted on labelled cases by the eval when it disagrees.
+    temperature: f64,
+    name: String,
+}
+
+pub struct NliConfig {
+    pub model: String,
+    pub dir: Option<PathBuf>,
+    pub temperature: f64,
+}
+
+impl NliConfig {
+    pub fn from_env() -> Self {
+        Self {
+            model: std::env::var("CLAWMATES_NLI_MODEL")
+                .ok()
+                .filter(|m| !m.trim().is_empty())
+                .unwrap_or_else(|| DEFAULT_MODEL.to_string()),
+            dir: std::env::var("CLAWMATES_NLI_MODEL_DIR").ok().map(PathBuf::from),
+            temperature: std::env::var("CLAWMATES_NLI_TEMPERATURE")
+                .ok()
+                .and_then(|t| t.parse().ok())
+                .unwrap_or(1.0),
+        }
+    }
+}
+
+impl Nli {
+    pub fn load(cfg: NliConfig) -> Result<Self, DecideError> {
+        let (config_path, tokenizer_path, weights_path) = match &cfg.dir {
+            Some(dir) => (
+                dir.join("config.json"),
+                dir.join("tokenizer.json"),
+                dir.join("model.safetensors"),
+            ),
+            None => {
+                let api = hf_hub::api::sync::Api::new()
+                    .map_err(|e| DecideError::Config(format!("hf hub: {e}")))?;
+                let repo = api.model(cfg.model.clone());
+                let get = |f: &str| {
+                    repo.get(f)
+                        .map_err(|e| DecideError::Config(format!("fetch {f} for {}: {e}", cfg.model)))
+                };
+                (get("config.json")?, get("tokenizer.json")?, get("model.safetensors")?)
+            }
+        };
+        let config_text = std::fs::read_to_string(&config_path)
+            .map_err(|e| DecideError::Config(format!("{}: {e}", config_path.display())))?;
+        let config: Config = serde_json::from_str(&config_text)
+            .map_err(|e| DecideError::Config(format!("config.json: {e}")))?;
+        // The head's label order is the model's, not ours: read it.
+        let id2label: Id2Label = serde_json::from_str::<serde_json::Value>(&config_text)
+            .ok()
+            .and_then(|v| v.get("id2label").cloned())
+            .and_then(|v| {
+                v.as_object().map(|m| {
+                    m.iter()
+                        .filter_map(|(k, v)| {
+                            Some((k.parse::<u32>().ok()?, v.as_str()?.to_lowercase()))
+                        })
+                        .collect()
+                })
+            })
+            .ok_or_else(|| DecideError::Config("config.json has no id2label".into()))?;
+        let find = |name: &str| {
+            id2label
+                .iter()
+                .find(|(_, v)| v.as_str() == name)
+                .map(|(k, _)| *k as usize)
+                .ok_or_else(|| DecideError::Config(format!("head has no {name:?} label")))
+        };
+        let entail = find("entailment")?;
+        // MNLI heads have three labels; the zero-shot-tuned checkpoints
+        // collapse to entailment / not_entailment. Either "no" label works.
+        let contra = find("contradiction").or_else(|_| find("not_entailment"))?;
+
+        let device = pick_device();
+        // Buffered, not mmap'd: the workspace denies `unsafe`, and the weights
+        // (740 MB for -base) are read once into memory and kept for the life
+        // of the process anyway.
+        let bytes = std::fs::read(&weights_path)
+            .map_err(|e| DecideError::Config(format!("{}: {e}", weights_path.display())))?;
+        let vb = VarBuilder::from_buffered_safetensors(bytes, DType::F32, &device)
+            .map_err(|e| DecideError::Config(format!("weights: {e}")))?;
+        // The backbone's tensors are `deberta.*`; the pooler and classifier
+        // read from the root. candle's own example does exactly this.
+        let vb = vb.set_prefix("deberta");
+        let model = DebertaV2SeqClassificationModel::load(vb, &config, Some(id2label))
+            .map_err(|e| DecideError::Config(format!("model: {e}")))?;
+
+        let mut tokenizer = Tokenizer::from_file(&tokenizer_path)
+            .map_err(|e| DecideError::Config(format!("tokenizer: {e}")))?;
+        tokenizer.with_padding(Some(PaddingParams::default()));
+        tokenizer
+            .with_truncation(Some(TruncationParams {
+                max_length: MAX_TOKENS,
+                strategy: tokenizers::TruncationStrategy::OnlyFirst,
+                ..Default::default()
+            }))
+            .map_err(|e| DecideError::Config(format!("truncation: {e}")))?;
+
+        Ok(Self {
+            model: Mutex::new(model),
+            tokenizer,
+            device,
+            entail,
+            contra,
+            temperature: cfg.temperature.max(1e-3),
+            name: format!("nli:{}", cfg.model.rsplit('/').next().unwrap_or(&cfg.model)),
+        })
+    }
+
+    /// Entailment and contradiction logits for every (state, hypothesis)
+    /// pair, one batch, one forward pass.
+    fn logits(&self, state: &str, hypotheses: &[String]) -> Result<Vec<(f64, f64)>, DecideError> {
+        if hypotheses.is_empty() {
+            return Ok(Vec::new());
+        }
+        let inputs: Vec<EncodeInput> = hypotheses
+            .iter()
+            .map(|h| {
+                EncodeInput::Dual(
+                    InputSequence::Raw(state.into()),
+                    InputSequence::Raw(h.as_str().into()),
+                )
+            })
+            .collect();
+        let encodings = self
+            .tokenizer
+            .encode_batch(inputs, true)
+            .map_err(|e| DecideError::Model(format!("tokenize: {e}")))?;
+        let to_tensor = |rows: Vec<&[u32]>| -> Result<Tensor, DecideError> {
+            let stacked: Vec<Tensor> = rows
+                .iter()
+                .map(|r| Tensor::new(*r, &self.device))
+                .collect::<Result<_, _>>()
+                .map_err(|e| DecideError::Model(e.to_string()))?;
+            Tensor::stack(&stacked, 0).map_err(|e| DecideError::Model(e.to_string()))
+        };
+        let ids = to_tensor(encodings.iter().map(|e| e.get_ids()).collect())?;
+        let mask = to_tensor(encodings.iter().map(|e| e.get_attention_mask()).collect())?;
+        let types = to_tensor(encodings.iter().map(|e| e.get_type_ids()).collect())?;
+        let out = {
+            let model = self.model.lock().map_err(|_| DecideError::Model("model lock".into()))?;
+            model
+                .forward(&ids, Some(types), Some(mask))
+                .map_err(|e| DecideError::Model(format!("forward: {e}")))?
+        };
+        let rows: Vec<Vec<f32>> = out
+            .to_dtype(DType::F32)
+            .and_then(|t| t.to_vec2())
+            .map_err(|e| DecideError::Model(e.to_string()))?;
+        Ok(rows
+            .iter()
+            .map(|r| {
+                (
+                    r[self.entail] as f64 / self.temperature,
+                    r[self.contra] as f64 / self.temperature,
+                )
+            })
+            .collect())
+    }
+
+    /// P(yes) for one hypothesis: entailment against contradiction, the
+    /// neutral class left out — the standard zero-shot NLI reading.
+    fn p_yes(entail: f64, contra: f64) -> f64 {
+        softmax(&[entail, contra])[0]
+    }
+}
+
+fn pick_device() -> Device {
+    #[cfg(feature = "cuda")]
+    if let Ok(d) = Device::new_cuda(0) {
+        return d;
+    }
+    #[cfg(feature = "metal")]
+    if let Ok(d) = Device::new_metal(0) {
+        return d;
+    }
+    Device::Cpu
+}
+
+/// The hypothesis a question becomes. Kept in one place because the wording
+/// is the whole "prompt" this backend has, and the eval is what tunes it.
+pub fn hypotheses(question: &Question) -> Vec<String> {
+    match question {
+        Question::Noul { instructions, criteria } => match criteria {
+            Some(c) => vec![
+                format!("{instructions} {}", c.yes),
+                format!("{instructions} {}", c.no),
+            ],
+            None => vec![instructions.clone()],
+        },
+        Question::Choice { instructions, criteria } => criteria
+            .iter()
+            .map(|(name, desc)| match desc {
+                Some(d) => format!("{instructions} {name}: {d}"),
+                None => format!("{instructions} {name}"),
+            })
+            .collect(),
+        Question::Score { instructions, criteria } => criteria
+            .iter()
+            .map(|level| format!("{instructions} {level}"))
+            .collect(),
+    }
+}
+
+#[async_trait::async_trait]
+impl Decider for Nli {
+    fn name(&self) -> &str {
+        &self.name
+    }
+
+    async fn decide(
+        &self,
+        state: &str,
+        questions: &BTreeMap<String, Question>,
+    ) -> Result<Decision, DecideError> {
+        let started = Instant::now();
+        // Every hypothesis of every question in one batch; then read each
+        // question's slice back out.
+        let mut all: Vec<String> = Vec::new();
+        let mut spans: Vec<(String, usize, usize)> = Vec::new();
+        for (id, q) in questions {
+            let hs = hypotheses(q);
+            let start = all.len();
+            all.extend(hs);
+            spans.push((id.clone(), start, all.len()));
+        }
+        let logits = self.logits(state, &all)?;
+        let mut answers = BTreeMap::new();
+        for (id, start, end) in spans {
+            let slice = &logits[start..end];
+            let q = &questions[&id];
+            let answer = match q {
+                Question::Noul { criteria, .. } => {
+                    let p = if criteria.is_some() {
+                        // yes-description vs no-description: which does the
+                        // state entail more?
+                        softmax(&[slice[0].0, slice[1].0])[0]
+                    } else {
+                        Self::p_yes(slice[0].0, slice[0].1)
+                    };
+                    Answer::Noul { noul: p }
+                }
+                Question::Choice { criteria, .. } => {
+                    let probs = softmax(&slice.iter().map(|(e, _)| *e).collect::<Vec<_>>());
+                    let names: Vec<&String> = criteria.keys().collect();
+                    let (best, _) = probs
+                        .iter()
+                        .enumerate()
+                        .fold((0, f64::NEG_INFINITY), |acc, (i, p)| if *p > acc.1 { (i, *p) } else { acc });
+                    Answer::Choice {
+                        choice: names[best].clone(),
+                        confidence: confidence_of(&probs),
+                        probabilities: names.iter().map(|n| (*n).clone()).zip(probs).collect(),
+                    }
+                }
+                Question::Score { .. } => {
+                    let probs = softmax(&slice.iter().map(|(e, _)| *e).collect::<Vec<_>>());
+                    let score = probs.iter().enumerate().map(|(i, p)| i as f64 * p).sum();
+                    Answer::Score {
+                        score,
+                        confidence: confidence_of(&probs),
+                        probabilities: probs,
+                    }
+                }
+            };
+            answers.insert(id, answer);
+        }
+        Ok(Decision {
+            model: self.name.clone(),
+            answers,
+            usage: None,
+            latency: started.elapsed(),
+        })
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn every_question_becomes_the_expected_hypotheses() {
+        let n = Question::noul("The task needs web search.");
+        assert_eq!(hypotheses(&n), vec!["The task needs web search."]);
+        let c = Question::choice("This work is", [("research", Some("reading")), ("coding", None)]);
+        let h = hypotheses(&c);
+        assert_eq!(h, vec!["This work is coding", "This work is research: reading"]);
+        let s = Question::score("Severity:", ["cosmetic", "blocking"]);
+        assert_eq!(hypotheses(&s).len(), 2);
+    }
+
+    #[test]
+    fn p_yes_is_entailment_against_contradiction() {
+        assert!(Nli::p_yes(3.0, -3.0) > 0.99);
+        assert!(Nli::p_yes(-3.0, 3.0) < 0.01);
+        assert!((Nli::p_yes(0.0, 0.0) - 0.5).abs() < 1e-9);
+    }
+}

crates/cm-decide/src/patterns.rs

@@ -0,0 +1,88 @@
+//! The composition rules, as code. Each is one of the vendor's documented
+//! patterns, kept here because they are the right way to use ANY calibrated
+//! classifier and should not live in one call site's `if` chain.
+
+use crate::Answer;
+
+/// Confidence-gated routing: three outcomes, not two. The answer says what;
+/// the confidence (or a Noul's distance from even) says whether to act.
+///
+/// `act_above` is the probability/confidence at which code may act on its
+/// own; `dismiss_below` the one under which the answer is a clear no. In
+/// between is the band a person, or a slower model, gets to decide.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum Gate {
+    Act,
+    Review,
+    Dismiss,
+}
+
+pub fn gate_noul(p: f64, dismiss_below: f64, act_above: f64) -> Gate {
+    debug_assert!(dismiss_below <= act_above);
+    if p >= act_above {
+        Gate::Act
+    } else if p < dismiss_below {
+        Gate::Dismiss
+    } else {
+        Gate::Review
+    }
+}
+
+/// Gate a Choice or Score on its confidence alone.
+pub fn gate_confidence(answer: &Answer, review_below: f64) -> Gate {
+    if answer.confidence() >= review_below {
+        Gate::Act
+    } else {
+        Gate::Review
+    }
+}
+
+/// Composite scoring: normalise each Score to 0..1 by its top level and
+/// combine with weights the caller owns. `parts` is `(score, levels, weight)`.
+/// Weights need not sum to one; the result is divided by their sum.
+pub fn composite(parts: &[(f64, usize, f64)]) -> f64 {
+    let total: f64 = parts.iter().map(|(_, _, w)| w).sum();
+    if total <= 0.0 {
+        return 0.0;
+    }
+    parts
+        .iter()
+        .map(|(score, levels, w)| {
+            let top = (*levels as f64 - 1.0).max(1.0);
+            (score / top).clamp(0.0, 1.0) * w
+        })
+        .sum::<f64>()
+        / total
+}
+
+/// Rerank: order candidates by a per-candidate Noul, highest first.
+pub fn rerank<T>(mut items: Vec<(T, f64)>) -> Vec<(T, f64)> {
+    items.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));
+    items
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn a_noul_gates_three_ways() {
+        assert_eq!(gate_noul(0.95, 0.2, 0.8), Gate::Act);
+        assert_eq!(gate_noul(0.05, 0.2, 0.8), Gate::Dismiss);
+        assert_eq!(gate_noul(0.5, 0.2, 0.8), Gate::Review);
+    }
+
+    #[test]
+    fn composite_normalises_by_top_level_and_weights() {
+        // severity 2 of 0..2 at weight 0.6, frustration 1 of 0..2 at 0.3,
+        // report quality 3 of 0..3 at 0.1 → 0.6*1 + 0.3*0.5 + 0.1*1 = 0.85
+        let c = composite(&[(2.0, 3, 0.6), (1.0, 3, 0.3), (3.0, 4, 0.1)]);
+        assert!((c - 0.85).abs() < 1e-9);
+    }
+
+    #[test]
+    fn rerank_is_descending() {
+        let r = rerank(vec![("a", 0.2), ("b", 0.9), ("c", 0.5)]);
+        assert_eq!(r.iter().map(|x| x.0).collect::<Vec<_>>(), ["b", "c", "a"]);
+    }
+}

crates/cm-decide/src/triage.rs

@@ -0,0 +1,22 @@
+//! The skill-triage question, in one place.
+//!
+//! Measured 2026-09-21 on `eval/skill-triage.json` (20 tasks × 53 skills):
+//! this wording — the skill's NAME plus its `when_to_use` line — scored
+//! AUROC 0.989 / F1 0.84 on Jev against 0.970 / 0.66 for the when_to_use
+//! line alone. The name carries signal. The server's shadow path and the
+//! eval both call this, so what is measured is what runs.
+
+use crate::Question;
+
+pub const WORDING: &str = "named";
+
+pub fn question(name: &str, when_to_use: &str) -> Question {
+    Question::noul(format!(
+        "This task calls for the skill \"{name}\", which applies when: {when_to_use}"
+    ))
+}
+
+/// The probability at which a skill counts as "applies" when the triage is
+/// read back. 0.5 is where Jev's best F1 sat (0.84 at 0.51); it is not a
+/// gate on anything yet — shadow mode records, it does not select.
+pub const APPLIES_AT: f64 = 0.5;

deploy/compose/docker-compose.override.yml

@@ -130,6 +130,9 @@ services:
       CLAWMATES_DOOR_GOVERNOR: "1"
       # Read by the `glm` provider's api_key_env in clawmates.toml.
       ZAI_API_KEY: ${ZAI_API_KEY:?set in .env}
+      # TypeSafe Jev — shadow skill triage (cm-api skill_triage.rs). Optional:
+      # unset, the triage records nothing. Server-side only; never a mission's.
+      TYPESAFE_API_KEY: ${TYPESAFE_API_KEY:-}
       # ElevenLabs GenFM, for rendering the Continuous Research episode.
       #
       # SERVER-SIDE ONLY, deliberately. It is NOT in