feat(evaluator): Slice 2 — an independent judge, from a different provider, with the same teeth

Claude writes the code and Claude judges it. That is a correlated failure: the
model that talked itself into a shortcut is the one disposed to accept it, and it
is the structural cause of the "early victory" failure Anthropic documents and of
our own Goodhart incident.

`glm` and `kimi` are both already registered in production, so the fix needed no
new credential path.

THE UNLOCK: `judge_with_tools` took `&AnthropicProvider`, but `LlmProvider` is a
single method — `stream(ChatRequest)` — and the loop only ever used that. The
concrete type was incidental. Widening it to `&dyn LlmProvider` means a
cross-provider judge runs the SAME allow-listed command loop. Before, independence
and real verification were mutually exclusive: the tool loop existed only on the
subscription path and every other route "judged claims only", so choosing an
independent judge meant giving up the checks that make a verdict evidence. GLM is
registered in anthropic format, so tool calling reaches it unchanged.

`CLAWMATES_VALIDATOR_MODEL` (e.g. `glm:glm-4.7`) selects it. Three refusals, each
protecting the claim the field makes:
  - a spec in the implementer's own family is rejected, not used — `opus` judging
    `sonnet` is not independence, they share a lineage and most failure modes
  - a spec naming a provider this deployment never registered is rejected.
    `Runtime::resolve_provider` silently falls back to the DEFAULT provider when
    the registry has no such name, which would hand back Claude while the caller
    believed it had GLM. Detectable because the returned model keeps its `name:`
    prefix, so it is checked rather than trusted.
  - an independent judge that FAILS does not fall through to the house judge. A
    verdict quietly produced by a same-family model would claim a property it does
    not have. The pass stays unmet, says why, and the next sweep retries.

`Verdict.independent` records it, `#[serde(default)]` so verdicts stored before
this field read back as not independent — which is what they were. An unrecognised
model family resolves to "unknown", never to ours: guessing would report
independence nobody established.

474 tests pass, clippy clean. Not yet enabled in production — the env var is unset,
so behaviour is identical until it is set deliberately.

crates/cm-api/src/evaluator.rs

@@ -55,6 +55,17 @@ pub struct Verdict {
     /// verification is how a broken sandbox comes to claim it proved
     /// something.
     pub checks: Vec<crate::evaluator_tools::CheckOutcome>,
+    /// Whether the judge came from a DIFFERENT provider family than the agent
+    /// that did the work.
+    ///
+    /// The default judge is Claude judging Claude's output, which is a correlated
+    /// failure: the same model that talked itself into a shortcut is disposed to
+    /// accept it. Independence is the structural fix, and it is recorded rather
+    /// than assumed — a verdict that cannot say who judged it cannot be audited
+    /// for this. `serde(default)` so verdicts stored before this field existed
+    /// read back as "not independent", which is what they were.
+    #[serde(default)]
+    pub independent: bool,
 }
 
 impl Verdict {
@@ -78,6 +89,7 @@ impl Verdict {
             guidance: reason.clone(),
             reason,
             model: model.to_string(),
+            independent: false,
             error,
             checks: Vec::new(),
         }
@@ -189,6 +201,90 @@ If you find any of these, the condition is NOT met — say which one you found.
 If you cannot verify a claim, it is not met: absence of evidence is not \
 satisfaction.";
 
+/// Which provider family a model spec belongs to.
+///
+/// `"glm:glm-4.7"` → `glm`, `"kimi:k2"` → `kimi`, `"claude-opus-4-8"` → `anthropic`.
+/// Used for one decision only: whether the judge is independent of the agent that
+/// produced the work. A family, not a model — two Claude models share a lineage,
+/// a fine-tune and most of their failure modes, so `opus` judging `sonnet` is not
+/// independence.
+pub fn provider_family(spec: &str) -> String {
+    if let Some((name, _)) = spec.split_once(':') {
+        // `runtime:<alias>` routes through an agent container, which is running
+        // Claude — the prefix names the transport, not the family.
+        return if name == "runtime" {
+            "anthropic".into()
+        } else {
+            name.to_ascii_lowercase()
+        };
+    }
+    let s = spec.to_ascii_lowercase();
+    for (needle, family) in [
+        ("claude", "anthropic"),
+        ("opus", "anthropic"),
+        ("sonnet", "anthropic"),
+        ("haiku", "anthropic"),
+        ("glm", "glm"),
+        ("kimi", "kimi"),
+        ("moonshot", "kimi"),
+        ("gemini", "gemini"),
+        ("llama", "groq"),
+    ] {
+        if s.contains(needle) {
+            return family.into();
+        }
+    }
+    // Not "anthropic". An unknown model must not be assumed to be the house
+    // one — that assumption would report independence we never established.
+    "unknown".into()
+}
+
+/// The provider family the mission's agent ran on.
+///
+/// Today every mission backend is Claude Code (`agent-claude`), including the
+/// microVM path. When `agent-glm` / `agent-kimi` images exist this should read
+/// `missions.backend`; until then, hardcoding the truth is better than plumbing a
+/// parameter that only ever has one value.
+const IMPLEMENTER_FAMILY: &str = "anthropic";
+
+/// A judge from a different provider family, if one is configured and REGISTERED.
+///
+/// `CLAWMATES_VALIDATOR_MODEL` holds a registry spec such as `glm:glm-4.7`.
+/// Returns `None` — never a same-family judge — when it is unset, names the
+/// implementer's own family, or names a provider this deployment did not register.
+///
+/// That last case is the trap worth naming: `Runtime::resolve_provider` falls back
+/// to the DEFAULT provider when the registry has no such name, which would hand
+/// back Claude while the caller believed it had asked for GLM. The fallback is
+/// detectable because the returned model still carries the `name:` prefix, and it
+/// is checked here rather than trusted.
+fn cross_provider_judge(
+    runtime: &cm_runtime::Runtime,
+) -> Option<(std::sync::Arc<dyn cm_llm::LlmProvider>, String)> {
+    let spec = std::env::var("CLAWMATES_VALIDATOR_MODEL").ok()?;
+    let spec = spec.trim();
+    if spec.is_empty() {
+        return None;
+    }
+    let family = provider_family(spec);
+    if family == IMPLEMENTER_FAMILY {
+        eprintln!(
+            "evaluator: CLAWMATES_VALIDATOR_MODEL={spec} is the same provider family as the \
+             agent ({IMPLEMENTER_FAMILY}) — that is not an independent check, ignoring it"
+        );
+        return None;
+    }
+    let (provider, model) = runtime.resolve_provider(spec);
+    if model.contains(':') {
+        eprintln!(
+            "evaluator: no provider registered for {spec} — refusing to judge with the \
+             default provider and call it independent"
+        );
+        return None;
+    }
+    Some((provider, model))
+}
+
 /// The model spec to judge with.
 ///
 /// Defaults to [`cm_runtime::judge_model`] so a single knob configures both
@@ -253,6 +349,47 @@ pub async fn evaluate(
     );
     let sandbox = crate::evaluator_tools::Sandbox::for_mission(mission_id);
 
+    // Most preferred: a judge from a DIFFERENT provider family, with the same
+    // allow-listed tool loop. Claude judging Claude's work is a correlated
+    // failure — the model that talked itself into a shortcut is the one disposed
+    // to accept it — and the tool loop is what makes the check evidence rather
+    // than opinion, so an independent judge must have it too.
+    if let Some((provider, model)) = cross_provider_judge(runtime) {
+        let system = match &sandbox {
+            Some(_) => format!("{EVAL_SYSTEM_VERIFYING}\n\n{VERDICT_CONTRACT}"),
+            None => format!("{EVAL_SYSTEM_EVIDENCE_ONLY}\n\n{VERDICT_CONTRACT}"),
+        };
+        eprintln!(
+            "evaluator: mission {mission_id} judged independently by {} ({})",
+            model,
+            provider_family(&model)
+        );
+        match judge_with_tools(provider.as_ref(), &system, &user, &model, sandbox.as_ref()).await {
+            Ok((text, checks)) => {
+                let mut v = parse_verdict(&model, &text);
+                v.guidance = sanitize_guidance(condition, evidence, &v.guidance);
+                v.checks = checks;
+                v.independent = true;
+                return v;
+            }
+            // Deliberately NOT a silent fall-through to the house judge. An
+            // independent check that failed and was quietly replaced by a
+            // same-family one would leave a verdict claiming a property it does
+            // not have. The phase stays unmet this pass and says why; the next
+            // sweep retries.
+            Err(e) => {
+                eprintln!(
+                    "evaluator: the independent judge ({model}) failed — NOT falling back to                      the agent's own provider: {e}"
+                );
+                return Verdict::not_met(
+                    &model,
+                    "the independent validator could not be reached this pass",
+                    Some(e),
+                );
+            }
+        }
+    }
+
     // Preferred: a bare Messages API call on the subscription token. See
     // `subscription_judge` for why this beats routing through an agent.
     if let Some(provider) = subscription_judge() {
@@ -262,6 +399,7 @@ pub async fn evaluate(
             None => format!("{EVAL_SYSTEM_EVIDENCE_ONLY}\n\n{VERDICT_CONTRACT}"),
         };
         let outcome = judge_with_tools(&provider, &system, &user, &model, sandbox.as_ref()).await;
+        // Same family as the agent; `independent` stays false below.
         return match outcome {
             Err(e) => Verdict::not_met(
                 &model,
@@ -346,14 +484,22 @@ fn verify_tool() -> cm_llm::ToolDescriptor {
 ///
 /// With no sandbox this degenerates to a single call — same shape, no tools
 /// offered — so there is one code path for both kinds of phase.
+/// `&dyn LlmProvider`, not `&AnthropicProvider`.
+///
+/// The trait is a single method — `stream(ChatRequest)` — and this loop only ever
+/// used that, so the concrete type was incidental. Widening it is what lets a
+/// CROSS-PROVIDER judge run the same allow-listed checks: before this, independence
+/// and real verification were mutually exclusive, because the tool loop lived only
+/// on the subscription path and every other route "judged claims only".
+/// GLM is registered in anthropic format, so tool calling reaches it unchanged.
 async fn judge_with_tools(
-    provider: &cm_llm::AnthropicProvider,
+    provider: &dyn cm_llm::LlmProvider,
     system: &str,
     user: &str,
     model: &str,
     sandbox: Option<&crate::evaluator_tools::Sandbox>,
 ) -> Result<(String, Vec<crate::evaluator_tools::CheckOutcome>), String> {
-    use cm_llm::{ChatMessage, ChatRequest, ChatRole, ContentPart, LlmEvent, LlmProvider};
+    use cm_llm::{ChatMessage, ChatRequest, ChatRole, ContentPart, LlmEvent};
     use futures::StreamExt as _;
 
     let tools = match sandbox {
@@ -507,6 +653,8 @@ fn parse_verdict(model: &str, text: &str) -> Verdict {
         reason,
         guidance,
         model: model.to_string(),
+        // Set by the caller: only `evaluate` knows which provider judged.
+        independent: false,
         error: None,
         checks: Vec::new(),
     }
@@ -581,6 +729,78 @@ pub async fn latest(
     }))
 }
 
+#[cfg(test)]
+mod cross_provider_tests {
+    use super::*;
+
+    /// A family, not a model. Two Claude models share a lineage and most of their
+    /// failure modes, so `opus` judging `sonnet` is not an independent check.
+    #[test]
+    fn every_anthropic_spelling_is_one_family() {
+        for spec in [
+            "claude-opus-4-8",
+            "claude-sonnet-5",
+            "claude-haiku-4-5-20251001",
+            "opus",
+            "runtime:claw_1234", // routes through an agent container running Claude
+        ] {
+            assert_eq!(provider_family(spec), "anthropic", "{spec}");
+        }
+    }
+
+    /// The registry prefix is what actually selects a different provider.
+    #[test]
+    fn a_registry_prefix_names_the_family() {
+        assert_eq!(provider_family("glm:glm-4.7"), "glm");
+        assert_eq!(provider_family("kimi:kimi-k2"), "kimi");
+        assert_eq!(provider_family("GLM:GLM-4.7"), "glm");
+    }
+
+    /// An unrecognised model must NOT be assumed to be the house one. Guessing
+    /// "anthropic" would understate independence; guessing anything else would
+    /// claim independence we never established. So: unknown.
+    #[test]
+    fn an_unrecognised_model_is_not_assumed_to_be_ours() {
+        assert_eq!(provider_family("some-new-model-v9"), "unknown");
+        assert_ne!(provider_family("some-new-model-v9"), IMPLEMENTER_FAMILY);
+    }
+
+    /// The whole point: a judge in the implementer's own family is not
+    /// independent, whichever model it is.
+    #[test]
+    fn a_same_family_judge_is_never_independent() {
+        for spec in ["claude-opus-4-8", "runtime:claw_x", "sonnet"] {
+            assert_eq!(
+                provider_family(spec),
+                IMPLEMENTER_FAMILY,
+                "{spec} would have to be rejected as a validator"
+            );
+        }
+        for spec in ["glm:glm-4.7", "kimi:kimi-k2"] {
+            assert_ne!(provider_family(spec), IMPLEMENTER_FAMILY, "{spec}");
+        }
+    }
+
+    /// A verdict that has not been marked independent must not read as one. This
+    /// is the field's default, and old rows stored before it existed deserialize
+    /// to exactly that.
+    #[test]
+    fn a_verdict_defaults_to_not_independent() {
+        let v = Verdict::not_met("claude-opus-4-8", "nope", None);
+        assert!(!v.independent);
+
+        let stored = serde_json::json!({
+            "met": true, "reason": "r", "guidance": "", "model": "claude-opus-4-8",
+            "error": null, "checks": []
+        });
+        let old: Verdict = serde_json::from_value(stored).expect("an old verdict still reads");
+        assert!(
+            !old.independent,
+            "a verdict written before independence was recorded was not independent"
+        );
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;

crates/cm-api/tests/phase_conditions.rs

@@ -278,6 +278,7 @@ async fn evaluations_are_unique_per_iteration_and_upsert() {
         model: "runtime:coordinator".into(),
         error: None,
         checks: Vec::new(),
+            independent: false,
     };
     cm_api::evaluator::record(&pool, mission, phase, 0, &first)
         .await
@@ -296,6 +297,7 @@ async fn evaluations_are_unique_per_iteration_and_upsert() {
             exit_code: Some(0),
             evidence: "exit status: 0".into(),
         }],
+        independent: false,
     };
     cm_api::evaluator::record(&pool, mission, phase, 0, &second)
         .await
@@ -353,6 +355,7 @@ async fn latest_returns_the_most_recent_iteration() {
                 model: "m".into(),
                 error: None,
                 checks: Vec::new(),
+            independent: false,
             },
         )
         .await