perf(evaluator): judge with a bare API call instead of an agent (156x fewer tokens)
A phase verdict is a classification: fixed prompt, no tools, no memory, one JSON answer. Routing it through a ZeroClaw agent charged 17,772 input tokens to produce a 20-token reply, and at the runtime's 32k context that scaffolding — role prompt, tool descriptors, memory, identity — consumed over half the window before the judge read any evidence. The same verdict as a direct Messages API call costs 114 input tokens, with the real system prompt and evidence. Measured through the production seam via `cargo run -p cm-llm --example oauth_probe`. - cm-llm: teach AnthropicProvider subscription auth. A `sk-ant-oat…` credential switches to bearer auth, adds the Claude Code beta set, and prepends the identity line the API requires as the first system block — idempotently, so re-wrapping can't stack it or waste tokens. - evaluator: prefer a direct provider call whenever ANTHROPIC_OAUTH_TOKEN is set, falling back to the configured spec (including `runtime:<alias>`) otherwise. Fail-closed parsing is untouched and still governs every path. - The ANTHROPIC_API_KEY shape guard now points at the slot that understands bearer auth rather than only saying no. Deleting the agent from this path is the ablation applied to our own harness: the scaffolding was there because a judge was built like every other agent, not because a judge needs it. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
2eb0880fc0
commit
09486ec759
@@ -35,8 +35,9 @@ fn build_provider(config: &AppConfig) -> Result<Arc<dyn LlmProvider>, String> {
|
||||
// easy to make and hard to spot.
|
||||
if key.starts_with("sk-ant-oat") {
|
||||
return Err("ANTHROPIC_API_KEY looks like a subscription OAuth token \
|
||||
(sk-ant-oat…), not a Console API key (sk-ant-api…). The \
|
||||
OAuth token belongs to the `claude` CLI, not the server."
|
||||
(sk-ant-oat…), not a Console API key (sk-ant-api…). Set it \
|
||||
as ANTHROPIC_OAUTH_TOKEN instead — that slot understands \
|
||||
bearer auth and is what the phase evaluator reads."
|
||||
.to_string());
|
||||
}
|
||||
Ok(Arc::new(AnthropicProvider::new(key)))
|
||||
|
||||
Reference in New Issue
Block a user