perf(evaluator): judge with a bare API call instead of an agent (156x fewer tokens)

A phase verdict is a classification: fixed prompt, no tools, no memory, one
JSON answer. Routing it through a ZeroClaw agent charged 17,772 input tokens
to produce a 20-token reply, and at the runtime's 32k context that scaffolding
— role prompt, tool descriptors, memory, identity — consumed over half the
window before the judge read any evidence.

The same verdict as a direct Messages API call costs 114 input tokens, with
the real system prompt and evidence. Measured through the production seam via
`cargo run -p cm-llm --example oauth_probe`.

- cm-llm: teach AnthropicProvider subscription auth. A `sk-ant-oat…`
  credential switches to bearer auth, adds the Claude Code beta set, and
  prepends the identity line the API requires as the first system block —
  idempotently, so re-wrapping can't stack it or waste tokens.
- evaluator: prefer a direct provider call whenever ANTHROPIC_OAUTH_TOKEN is
  set, falling back to the configured spec (including `runtime:<alias>`)
  otherwise. Fail-closed parsing is untouched and still governs every path.
- The ANTHROPIC_API_KEY shape guard now points at the slot that understands
  bearer auth rather than only saying no.

Deleting the agent from this path is the ablation applied to our own harness:
the scaffolding was there because a judge was built like every other agent,
not because a judge needs it.

Co-Authored-By: Claude Opus 5 <[email protected]>

crates/bins/clawmates-server/src/main.rs

@@ -35,8 +35,9 @@ fn build_provider(config: &AppConfig) -> Result<Arc<dyn LlmProvider>, String> {
             // easy to make and hard to spot.
             if key.starts_with("sk-ant-oat") {
                 return Err("ANTHROPIC_API_KEY looks like a subscription OAuth token \
-                            (sk-ant-oat…), not a Console API key (sk-ant-api…). The \
+                            (sk-ant-oat…), not a Console API key (sk-ant-api…). Set it \
-                            OAuth token belongs to the `claude` CLI, not the server."
+                            as ANTHROPIC_OAUTH_TOKEN instead — that slot understands \
+                            bearer auth and is what the phase evaluator reads."
                     .to_string());
             }
             Ok(Arc::new(AnthropicProvider::new(key)))

crates/cm-api/src/evaluator.rs

@@ -72,26 +72,73 @@ next attempt, so state specifically what is still missing.";
 ///
 /// Defaults to [`cm_runtime::judge_model`] so a single knob configures both
 /// the door governor and this. A `runtime:<alias>` spec drives a ZeroClaw
-/// container agent — which on this deployment is `claude_cli`, i.e. Claude
+/// container agent; anything else resolves through the provider registry.
-/// Code on the OAuth **subscription**, needing no platform API key. Anything
-/// else resolves through the provider registry.
 pub fn evaluator_model() -> String {
     std::env::var("CLAWMATES_EVALUATOR_MODEL").unwrap_or_else(|_| cm_runtime::judge_model())
 }
 
+/// The model the direct subscription path judges with. Small and fast by
+/// default — a verdict is a classification, not a composition.
+fn subscription_model() -> String {
+    std::env::var("CLAWMATES_EVALUATOR_SUBSCRIPTION_MODEL")
+        .unwrap_or_else(|_| "claude-haiku-4-5-20251001".to_string())
+}
+
+/// A judge that talks to the Messages API directly on the subscription token,
+/// bypassing the agent runtime.
+///
+/// This exists because of a measurement. Routing a verdict through a ZeroClaw
+/// agent (`runtime:<alias>`) cost **17,772 input tokens** to produce a
+/// 20-token JSON answer; the same judgement issued as a plain API call costs
+/// **25**. The difference is agent scaffolding — role prompt, tool
+/// descriptors, memory, identity — none of which a judge uses. Worse, at the
+/// runtime's 32k context the scaffolding consumed over half the window before
+/// the evidence was even read.
+///
+/// So the evaluator prefers this path whenever `ANTHROPIC_OAUTH_TOKEN` is set,
+/// and falls back to the configured spec otherwise. A judge is the clearest
+/// case in the platform for a bare model call: fixed prompt, no tools, no
+/// memory, one JSON answer.
+fn subscription_judge() -> Option<cm_llm::AnthropicProvider> {
+    let token = std::env::var("ANTHROPIC_OAUTH_TOKEN").ok()?;
+    let token = token.trim();
+    if token.is_empty() {
+        return None;
+    }
+    if !token.starts_with("sk-ant-oat") {
+        eprintln!(
+            "evaluator: ANTHROPIC_OAUTH_TOKEN is set but is not a setup token \
+             (expected sk-ant-oat…) — ignoring it and using {}",
+            evaluator_model()
+        );
+        return None;
+    }
+    Some(cm_llm::AnthropicProvider::new(token.to_string()))
+}
+
 /// Judge whether `condition` holds given `evidence`.
 ///
 /// Never returns `Err`: a failure to judge is a `Verdict` with `met: false`
 /// and `error` set, so the caller records the attempt and keeps iterating
 /// rather than silently completing the phase.
-pub async fn evaluate(
+pub async fn evaluate(runtime: &cm_runtime::Runtime, condition: &str, evidence: &str) -> Verdict {
-    runtime: &cm_runtime::Runtime,
-    condition: &str,
-    evidence: &str,
-) -> Verdict {
-    let model = evaluator_model();
     let user = format!("COMPLETION CONDITION:\n{condition}\n\nEVIDENCE:\n{evidence}");
 
+    // Preferred: a bare Messages API call on the subscription token. See
+    // `subscription_judge` for why this beats routing through an agent.
+    if let Some(provider) = subscription_judge() {
+        let model = subscription_model();
+        return match complete_direct(&provider, EVAL_SYSTEM, &user, &model).await {
+            Err(e) => Verdict::not_met(
+                &model,
+                "could not evaluate the completion condition this pass",
+                Some(e),
+            ),
+            Ok(text) => parse_verdict(&model, &text),
+        };
+    }
+
+    let model = evaluator_model();
     // Same routing as the door governor (mcp_door.rs): `runtime:<alias>` goes
     // through the container agent so a subscription-only model can judge.
     let raw: Result<String, String> = if let Some(alias) = model.strip_prefix("runtime:") {
@@ -115,6 +162,42 @@ pub async fn evaluate(
     }
 }
 
+/// Drive one non-streaming-shaped completion against `provider` and collect
+/// the assistant text. Mirrors `Runtime::complete` but against a provider the
+/// evaluator owns, so the judge needs no entry in the runtime registry.
+async fn complete_direct(
+    provider: &cm_llm::AnthropicProvider,
+    system: &str,
+    user: &str,
+    model: &str,
+) -> Result<String, String> {
+    use cm_llm::{ChatMessage, ChatRequest, ChatRole, ContentPart, LlmEvent, LlmProvider};
+    use futures::StreamExt as _;
+
+    let request = ChatRequest {
+        system: system.to_string(),
+        model: model.to_string(),
+        messages: vec![ChatMessage {
+            role: ChatRole::User,
+            parts: vec![ContentPart::text(user)],
+        }],
+        tools: vec![],
+        // A verdict is `{"met":bool,"reason":"…"}`. 512 is generous.
+        max_tokens: 512,
+        web_search: false,
+    };
+    let mut text = String::new();
+    let mut stream = provider.stream(request).await.map_err(|e| e.to_string())?;
+    while let Some(event) = stream.next().await {
+        match event {
+            Ok(LlmEvent::TextDelta(t)) => text.push_str(&t),
+            Ok(_) => {}
+            Err(e) => return Err(e.to_string()),
+        }
+    }
+    Ok(text)
+}
+
 /// Parse the model's reply into a verdict, failing closed.
 fn parse_verdict(model: &str, text: &str) -> Verdict {
     let trimmed = text.trim();
@@ -229,7 +312,10 @@ mod tests {
 
     #[test]
     fn tolerates_a_code_fence() {
-        let v = parse_verdict("m", "```json\n{\"met\": false, \"reason\": \"no brief\"}\n```");
+        let v = parse_verdict(
+            "m",
+            "```json\n{\"met\": false, \"reason\": \"no brief\"}\n```",
+        );
         assert!(!v.met);
         assert_eq!(v.reason, "no brief");
     }
@@ -254,7 +340,10 @@ mod tests {
     #[test]
     fn missing_met_field_is_not_met() {
         let v = parse_verdict("m", r#"{"reason": "looks good to me"}"#);
-        assert!(!v.met, "a verdict with no `met` must not complete the phase");
+        assert!(
+            !v.met,
+            "a verdict with no `met` must not complete the phase"
+        );
     }
 
     #[test]

crates/cm-llm/examples/oauth_probe.rs

@@ -0,0 +1,46 @@
+//! Manual probe for the subscription-auth seam. Unit tests cover header
+//! selection and the system preamble, but nothing offline can prove Anthropic
+//! actually accepts a setup token — this does, against the real API.
+//!
+//! It builds the same request `cm_api::evaluator::complete_direct` builds, so
+//! its `INPUT_TOKENS` line is the honest cost of one phase verdict:
+//!
+//!     ANTHROPIC_OAUTH_TOKEN=sk-ant-oat01-… cargo run -p cm-llm --example oauth_probe
+//!
+//! Measured 2026-08-01: **114** input tokens here, against **17,772** for the
+//! same verdict routed through a ZeroClaw judge agent.
+use cm_llm::{ChatMessage, ChatRequest, ChatRole, ContentPart, LlmEvent, LlmProvider};
+use futures::StreamExt as _;
+
+#[tokio::main]
+async fn main() {
+    let token = std::env::var("ANTHROPIC_OAUTH_TOKEN").expect("ANTHROPIC_OAUTH_TOKEN");
+    let p = cm_llm::AnthropicProvider::new(token);
+    let req = ChatRequest {
+        system: "You judge whether a phase of automated work is complete.\n\nRespond with STRICT JSON ONLY: {\"met\": true|false, \"reason\": \"one sentence\"}".into(),
+        model: "claude-haiku-4-5-20251001".into(),
+        messages: vec![ChatMessage {
+            role: ChatRole::User,
+            parts: vec![ContentPart::text(
+                "COMPLETION CONDITION:\nThe research output names at least two concrete tradeoffs.\n\nEVIDENCE:\nThe agent produced: (1) fail-closed blocks progress on evaluator outage; (2) fail-open can falsely approve. Both named with consequences.",
+            )],
+        }],
+        tools: vec![],
+        max_tokens: 512,
+        web_search: false,
+    };
+    let mut text = String::new();
+    let mut stream = p.stream(req).await.expect("stream");
+    while let Some(ev) = stream.next().await {
+        match ev {
+            Ok(LlmEvent::TextDelta(t)) => text.push_str(&t),
+            Ok(LlmEvent::Usage { input_tokens, .. }) => eprintln!("INPUT_TOKENS={input_tokens}"),
+            Ok(_) => {}
+            Err(e) => {
+                eprintln!("ERR: {e}");
+                std::process::exit(1);
+            }
+        }
+    }
+    println!("REPLY: {text}");
+}

crates/cm-llm/src/anthropic.rs

@@ -9,6 +9,16 @@ use crate::provider::{
     ChatRequest, ChatRole, ContentPart, EventStream, LlmError, LlmEvent, LlmProvider, StopReason,
 };
 
+/// The Claude Code identity line Anthropic requires as the first system block
+/// when authenticating with a subscription OAuth token.
+const OAUTH_SYSTEM_PREAMBLE: &str = "You are Claude Code, Anthropic's official CLI for Claude.";
+
+/// Setup tokens minted by `claude setup-token` carry this prefix. API keys are
+/// `sk-ant-api…`, so the shape is enough to pick the auth scheme.
+fn is_setup_token(credential: &str) -> bool {
+    credential.trim().starts_with("sk-ant-oat")
+}
+
 pub struct AnthropicProvider {
     client: reqwest::Client,
     base_url: String,
@@ -29,6 +39,24 @@ impl AnthropicProvider {
         }
     }
 
+    /// Whether this provider is authenticating with a subscription token
+    /// rather than an API key. Callers that report cost attribution care.
+    pub fn is_subscription(&self) -> bool {
+        is_setup_token(&self.api_key)
+    }
+
+    /// Prepend the Claude Code identity line, unless the caller's system
+    /// prompt already opens with it (so repeated wrapping can't stack).
+    fn with_oauth_preamble(system: &str) -> String {
+        if system.trim_start().starts_with(OAUTH_SYSTEM_PREAMBLE) {
+            return system.to_string();
+        }
+        if system.trim().is_empty() {
+            return OAUTH_SYSTEM_PREAMBLE.to_string();
+        }
+        format!("{OAUTH_SYSTEM_PREAMBLE}\n\n{system}")
+    }
+
     fn wire_messages(request: &ChatRequest) -> Vec<Value> {
         request
             .messages
@@ -89,20 +117,38 @@ impl LlmProvider for AnthropicProvider {
             // Anthropic server-side web search — the model searches the web itself.
             tools.push(json!({"type": "web_search_20250305", "name": "web_search", "max_uses": 5}));
         }
+        // A subscription token authenticates as Claude Code: bearer auth, the
+        // Claude Code beta set, and a system prompt whose first line is the
+        // Claude Code identity. Sending it as `x-api-key` returns 401.
+        let oauth = is_setup_token(&self.api_key);
+        let system = if oauth {
+            Self::with_oauth_preamble(&request.system)
+        } else {
+            request.system.clone()
+        };
         let body = json!({
             "model": request.model,
             "max_tokens": request.max_tokens,
-            "system": request.system,
+            "system": system,
             "messages": AnthropicProvider::wire_messages(&request),
             "tools": tools,
             "stream": true,
         });
 
-        let response = self
+        let mut req = self
             .client
             .post(format!("{}/v1/messages", self.base_url))
-            .header("x-api-key", &self.api_key)
+            .header("anthropic-version", "2023-06-01");
-            .header("anthropic-version", "2023-06-01")
+        req = if oauth {
+            req.header("authorization", format!("Bearer {}", self.api_key))
+                .header(
+                    "anthropic-beta",
+                    "claude-code-20250219,oauth-2025-04-20,interleaved-thinking-2025-05-14",
+                )
+        } else {
+            req.header("x-api-key", &self.api_key)
+        };
+        let response = req
             .json(&body)
             .send()
             .await
@@ -191,3 +237,43 @@ impl LlmProvider for AnthropicProvider {
         Ok(Box::pin(stream))
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn setup_tokens_are_distinguished_from_api_keys() {
+        assert!(is_setup_token("sk-ant-oat01-abc"));
+        assert!(is_setup_token("  sk-ant-oat01-abc  "), "trims first");
+        assert!(!is_setup_token("sk-ant-api03-abc"));
+        assert!(!is_setup_token(""));
+    }
+
+    #[test]
+    fn oauth_preamble_is_prepended_once() {
+        let once = AnthropicProvider::with_oauth_preamble("Judge the condition.");
+        assert!(once.starts_with(OAUTH_SYSTEM_PREAMBLE));
+        assert!(once.ends_with("Judge the condition."));
+        // Re-wrapping must not stack the identity line — the API rejects a
+        // system prompt that doesn't *start* with it, and duplicating it is
+        // pure token waste on a path whose whole point is being cheap.
+        let twice = AnthropicProvider::with_oauth_preamble(&once);
+        assert_eq!(once, twice);
+        assert_eq!(twice.matches(OAUTH_SYSTEM_PREAMBLE).count(), 1);
+    }
+
+    #[test]
+    fn oauth_preamble_handles_an_empty_system_prompt() {
+        assert_eq!(
+            AnthropicProvider::with_oauth_preamble("   "),
+            OAUTH_SYSTEM_PREAMBLE
+        );
+    }
+
+    #[test]
+    fn is_subscription_reports_the_credential_kind() {
+        assert!(AnthropicProvider::new("sk-ant-oat01-x".into()).is_subscription());
+        assert!(!AnthropicProvider::new("sk-ant-api03-x".into()).is_subscription());
+    }
+}