feat(decide): cm-decide — typed calibrated decisions; Jev + local NLI backends; skill triage in shadow

A third kind of decision-maker between deterministic code and a full LLM
call: Choice / Score / Noul questions answered as probability
distributions with a confidence, behind one Decider trait, with the
composition patterns (confidence gating, composite scoring, rerank) as
code. Two backends: TypeSafe's Jev over HTTP, and a DeBERTa-v3 MNLI
cross-encoder run in-process with candle (feature nli; metal/cuda).

decide-eval measures a backend on labelled cases the way judge-eval
measures the judge. eval/skill-triage.json: 20 mission tasks × 53 skills,
75 positives, hand-labelled. Measured 2026-09-21:

  lexical overlap        AUROC 0.851  [email protected] 0.47  top-k 48/75  ECE 0.095
  jev (named wording)    AUROC 0.989  [email protected] 0.84  top-k 63/75  ECE 0.064  213 ms
  jev (plain wording)    AUROC 0.970  [email protected] 0.66  top-k 52/75
  nli mnli-base          AUROC 0.790  [email protected] 0.28  top-k 38/75  ECE 0.263  1.5 s
  nli zeroshot-v2        AUROC 0.782  [email protected] 0.43  top-k 39/75  ECE 0.054  1.2 s

The vendor's calibration claim survives our data; the local cross-encoder
ranks below keyword overlap on either checkpoint or wording and is kept
as the measured negative, not shipped. A local backend would need the
logit-readout route over the fleet's 9B model — a separate spike.

Shadow: one Jev call per phase launch (spawned, 10 s cap, silent without
TYPESAFE_API_KEY) records a skill.triage event; the Skill-Use report
carries triage_p beside each skill's Trigger verdict. It selects nothing.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz

crates/cm-api/Cargo.toml

@@ -31,6 +31,7 @@ cm-billing = { path = "../cm-billing" }
 cm-brain = { path = "../cm-brain" }
 cm-config = { path = "../cm-config" }
 cm-db = { path = "../cm-db" }
+cm-decide = { path = "../cm-decide" }
 cm-domain = { path = "../cm-domain" }
 cm-files = { path = "../cm-files" }
 tar = { workspace = true }

crates/cm-api/src/lib.rs

@@ -57,6 +57,7 @@ pub mod container_tool_hooks;
 pub mod gateway_preflight;
 pub mod skill_delivery;
 pub mod skill_self_authoring;
+pub mod skill_triage;
 pub mod skill_use;
 pub mod skills_loader;
 pub mod subscription;

crates/cm-api/src/phase_runner.rs

@@ -1231,6 +1231,11 @@ async fn launch_phase(
         .await
         .unwrap_or(None);
     let task = phase_task_text(kind, title, description, phase_task, has_repo);
+    // Shadow skill triage on the task as the operator wrote it — before the
+    // judge's guidance and the project memory are appended, because those
+    // are not what a skill's `when_to_use` describes. Spawned; records an
+    // event; changes nothing.
+    crate::skill_triage::spawn(pool.clone(), mission_id, phase_id, workspace_id, task.clone());
     let task = match prior {
         Some((iter, false, guidance)) => format!(
             "{task}\n\nPASS {} DID NOT SATISFY THE COMPLETION CONDITION. What is \

crates/cm-api/src/skill_triage.rs

@@ -0,0 +1,131 @@
+//! Skill triage, in shadow: which of the visible skills a phase's task calls
+//! for, by a calibrated decision model, recorded beside what the agent then
+//! actually read.
+//!
+//! SRA-Bench (arXiv 2604.24594) found agents load skills at the same rate
+//! whether or not one applies — the bottleneck is knowing WHEN, and the
+//! agent's only signal today is the `when_to_use` line in its own prompt. A
+//! host-side oracle that answers the same question in 200 ms is the thing
+//! to measure against that. `cm_decide::jev` scored AUROC 0.989 on the
+//! labelled set (`crates/cm-decide/eval`); this records its answer per phase
+//! as a `skill.triage` event and the Skill-Use scorer reads it back next to
+//! the agent's Trigger. It selects nothing: the files arm still installs
+//! every visible skill. Promotion to a real selector is a later, measured
+//! step, once the agreement numbers from real missions say what the
+//! oracle's misses cost.
+//!
+//! One call per phase launch, spawned so the launch never waits on it, and
+//! silent when `TYPESAFE_API_KEY` is unset. The key never leaves the server.
+
+use std::collections::BTreeMap;
+
+use cm_decide::{Answer, Decider};
+use sqlx::PgPool;
+use uuid::Uuid;
+
+pub const EVENT: &str = "skill.triage";
+
+/// How long a shadow decision may take before it is dropped. Jev measures
+/// ~200 ms; a backend that takes ten seconds is not the one to shadow.
+const TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10);
+
+/// Fire the triage for one phase and record it. Best-effort throughout: a
+/// missing key, a failed call, or a timeout leaves no event and one log line.
+pub fn spawn(pool: PgPool, mission_id: Uuid, phase_id: Uuid, workspace_id: Uuid, task: String) {
+    let Some(jev) = cm_decide::jev::Jev::from_env() else {
+        return;
+    };
+    tokio::spawn(async move {
+        let skills = match cm_db::repo::skills_catalog::list_visible(&pool, workspace_id).await {
+            Ok(s) => s,
+            Err(e) => {
+                eprintln!("skill_triage: could not list skills for {mission_id}: {e}");
+                return;
+            }
+        };
+        let questions: BTreeMap<String, cm_decide::Question> = skills
+            .iter()
+            .map(|s| {
+                (
+                    s.name.clone(),
+                    cm_decide::triage::question(&s.name, s.when_to_use.as_deref().unwrap_or(&s.description)),
+                )
+            })
+            .collect();
+        if questions.is_empty() {
+            return;
+        }
+        let decision = match tokio::time::timeout(TIMEOUT, jev.decide(&task, &questions)).await {
+            Ok(Ok(d)) => d,
+            Ok(Err(e)) => {
+                eprintln!("skill_triage: {} failed for phase {phase_id}: {e}", jev.name());
+                return;
+            }
+            Err(_) => {
+                eprintln!("skill_triage: {} timed out for phase {phase_id}", jev.name());
+                return;
+            }
+        };
+        let probabilities: BTreeMap<&str, f64> = decision
+            .answers
+            .iter()
+            .filter_map(|(k, a)| match a {
+                Answer::Noul { noul } => Some((k.as_str(), *noul)),
+                _ => None,
+            })
+            .collect();
+        let applies = probabilities
+            .iter()
+            .filter(|(_, p)| **p >= cm_decide::triage::APPLIES_AT)
+            .count();
+        eprintln!(
+            "skill_triage: phase {phase_id} — {} says {applies} of {} skills apply ({} ms, {} tokens)",
+            decision.model,
+            probabilities.len(),
+            decision.latency.as_millis(),
+            decision.usage.map(|u| u.input_tokens).unwrap_or(0),
+        );
+        crate::mission_events::record(
+            &pool,
+            crate::mission_events::MissionEvent::new(mission_id, EVENT)
+                .phase(phase_id)
+                .detail(serde_json::json!({
+                    "backend": jev.name(),
+                    "model": decision.model,
+                    "wording": cm_decide::triage::WORDING,
+                    "latency_ms": decision.latency.as_millis() as u64,
+                    "input_tokens": decision.usage.map(|u| u.input_tokens),
+                    "applies_at": cm_decide::triage::APPLIES_AT,
+                    "skills": probabilities,
+                })),
+        )
+        .await;
+    });
+}
+
+/// The recorded triage for a mission: skill → highest probability any phase
+/// gave it. Empty when no event was recorded (no key, or before this existed).
+pub async fn recorded(pool: &PgPool, mission_id: Uuid) -> BTreeMap<String, f64> {
+    let rows: Vec<(serde_json::Value,)> = sqlx::query_as(
+        "SELECT detail FROM mission_events WHERE mission_id = $1 AND kind = $2 ORDER BY id",
+    )
+    .bind(mission_id)
+    .bind(EVENT)
+    .fetch_all(pool)
+    .await
+    .unwrap_or_default();
+    let mut out: BTreeMap<String, f64> = BTreeMap::new();
+    for (detail,) in rows {
+        if let Some(map) = detail.get("skills").and_then(|s| s.as_object()) {
+            for (name, p) in map {
+                if let Some(p) = p.as_f64() {
+                    let e = out.entry(name.clone()).or_insert(0.0);
+                    if p > *e {
+                        *e = p;
+                    }
+                }
+            }
+        }
+    }
+    out
+}

crates/cm-api/src/skill_use.rs

@@ -99,6 +99,13 @@ pub struct SkillUse {
     pub trigger: Verdict,
     pub compliance: Verdict,
     pub boundary: Verdict,
+    /// What the shadow triage said BEFORE the phase ran: the probability
+    /// that this skill applies to the task (`skill_triage`). `None` when no
+    /// triage was recorded. Read next to `trigger`: a high probability with a
+    /// skipped skill is a miss by the agent or by the oracle, and only real
+    /// missions say which.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub triage_p: Option<f64>,
 }
 
 /// The skills a prompt actually delivered.
@@ -305,6 +312,7 @@ pub fn score(
                 compliance,
                 boundary,
                 skill,
+                triage_p: None,
             }
         })
         .collect()
@@ -1120,14 +1128,19 @@ pub async fn score_mission(
     .into_iter()
     .collect();
 
-    Ok(score(&prompts, &Evidence::new(&outputs, &tools), &|name| {
+    let triage = crate::skill_triage::recorded(pool, mission_id).await;
+    let mut scores = score(&prompts, &Evidence::new(&outputs, &tools), &|name| {
         kinds
             .get(name)
             .cloned()
             // A skill in a prompt with no catalogue row was delivered and then
             // deleted. Naming that explicitly beats defaulting it to builtin.
             .unwrap_or_else(|| "unknown (no catalogue row)".to_string())
-    }))
+    });
+    for s in &mut scores {
+        s.triage_p = triage.get(&s.skill).copied();
+    }
+    Ok(scores)
 }
 
 #[cfg(test)]