perf(evaluator): judge with a bare API call instead of an agent (156x fewer tokens)
ci / gates (push) Successful in 5s
ci / rust (push) Failing after 8s
ci / frontend (push) Failing after 18s
ci / e2e (push) Skipped
ci / publish (push) Skipped

A phase verdict is a classification: fixed prompt, no tools, no memory, one
JSON answer. Routing it through a ZeroClaw agent charged 17,772 input tokens
to produce a 20-token reply, and at the runtime's 32k context that scaffolding
— role prompt, tool descriptors, memory, identity — consumed over half the
window before the judge read any evidence.

The same verdict as a direct Messages API call costs 114 input tokens, with
the real system prompt and evidence. Measured through the production seam via
`cargo run -p cm-llm --example oauth_probe`.

- cm-llm: teach AnthropicProvider subscription auth. A `sk-ant-oat…`
  credential switches to bearer auth, adds the Claude Code beta set, and
  prepends the identity line the API requires as the first system block —
  idempotently, so re-wrapping can't stack it or waste tokens.
- evaluator: prefer a direct provider call whenever ANTHROPIC_OAUTH_TOKEN is
  set, falling back to the configured spec (including `runtime:<alias>`)
  otherwise. Fail-closed parsing is untouched and still governs every path.
- The ANTHROPIC_API_KEY shape guard now points at the slot that understands
  bearer auth rather than only saying no.

Deleting the agent from this path is the ablation applied to our own harness:
the scaffolding was there because a judge was built like every other agent,
not because a judge needs it.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-07-31 20:21:22 -07:00
co-authored by Claude Opus 5
parent 2eb0880fc0
commit 09486ec759
4 changed files with 239 additions and 17 deletions
+3 -2
View File
@@ -35,8 +35,9 @@ fn build_provider(config: &AppConfig) -> Result<Arc<dyn LlmProvider>, String> {
// easy to make and hard to spot.
if key.starts_with("sk-ant-oat") {
return Err("ANTHROPIC_API_KEY looks like a subscription OAuth token \
(sk-ant-oat…), not a Console API key (sk-ant-api…). The \
OAuth token belongs to the `claude` CLI, not the server."
(sk-ant-oat…), not a Console API key (sk-ant-api…). Set it \
as ANTHROPIC_OAUTH_TOKEN instead — that slot understands \
bearer auth and is what the phase evaluator reads."
.to_string());
}
Ok(Arc::new(AnthropicProvider::new(key)))