Commit Graph
1 Commits
Author SHA1 Message Date
Omar SobhandClaude Opus 5 09486ec759 perf(evaluator): judge with a bare API call instead of an agent (156x fewer tokens)
ci / gates (push) Successful in 5s
ci / rust (push) Failing after 8s
ci / frontend (push) Failing after 18s
ci / e2e (push) Skipped
ci / publish (push) Skipped
A phase verdict is a classification: fixed prompt, no tools, no memory, one
JSON answer. Routing it through a ZeroClaw agent charged 17,772 input tokens
to produce a 20-token reply, and at the runtime's 32k context that scaffolding
— role prompt, tool descriptors, memory, identity — consumed over half the
window before the judge read any evidence.

The same verdict as a direct Messages API call costs 114 input tokens, with
the real system prompt and evidence. Measured through the production seam via
`cargo run -p cm-llm --example oauth_probe`.

- cm-llm: teach AnthropicProvider subscription auth. A `sk-ant-oat…`
  credential switches to bearer auth, adds the Claude Code beta set, and
  prepends the identity line the API requires as the first system block —
  idempotently, so re-wrapping can't stack it or waste tokens.
- evaluator: prefer a direct provider call whenever ANTHROPIC_OAUTH_TOKEN is
  set, falling back to the configured spec (including `runtime:<alias>`)
  otherwise. Fail-closed parsing is untouched and still governs every path.
- The ANTHROPIC_API_KEY shape guard now points at the slot that understands
  bearer auth rather than only saying no.

Deleting the agent from this path is the ablation applied to our own harness:
the scaffolding was there because a judge was built like every other agent,
not because a judge needs it.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-07-31 20:21:22 -07:00