Omar SobhandClaude Opus 5
09486ec759
A phase verdict is a classification: fixed prompt, no tools, no memory, one JSON answer. Routing it through a ZeroClaw agent charged 17,772 input tokens to produce a 20-token reply, and at the runtime's 32k context that scaffolding — role prompt, tool descriptors, memory, identity — consumed over half the window before the judge read any evidence. The same verdict as a direct Messages API call costs 114 input tokens, with the real system prompt and evidence. Measured through the production seam via `cargo run -p cm-llm --example oauth_probe`. - cm-llm: teach AnthropicProvider subscription auth. A `sk-ant-oat…` credential switches to bearer auth, adds the Claude Code beta set, and prepends the identity line the API requires as the first system block — idempotently, so re-wrapping can't stack it or waste tokens. - evaluator: prefer a direct provider call whenever ANTHROPIC_OAUTH_TOKEN is set, falling back to the configured spec (including `runtime:<alias>`) otherwise. Fail-closed parsing is untouched and still governs every path. - The ANTHROPIC_API_KEY shape guard now points at the slot that understands bearer auth rather than only saying no. Deleting the agent from this path is the ablation applied to our own harness: the scaffolding was there because a judge was built like every other agent, not because a judge needs it. Co-Authored-By: Claude Opus 5 <[email protected]>
47 lines
2.1 KiB
Rust
47 lines
2.1 KiB
Rust
//! Manual probe for the subscription-auth seam. Unit tests cover header
|
|
//! selection and the system preamble, but nothing offline can prove Anthropic
|
|
//! actually accepts a setup token — this does, against the real API.
|
|
//!
|
|
//! It builds the same request `cm_api::evaluator::complete_direct` builds, so
|
|
//! its `INPUT_TOKENS` line is the honest cost of one phase verdict:
|
|
//!
|
|
//! ANTHROPIC_OAUTH_TOKEN=sk-ant-oat01-… cargo run -p cm-llm --example oauth_probe
|
|
//!
|
|
//! Measured 2026-08-01: **114** input tokens here, against **17,772** for the
|
|
//! same verdict routed through a ZeroClaw judge agent.
|
|
use cm_llm::{ChatMessage, ChatRequest, ChatRole, ContentPart, LlmEvent, LlmProvider};
|
|
use futures::StreamExt as _;
|
|
|
|
#[tokio::main]
|
|
async fn main() {
|
|
let token = std::env::var("ANTHROPIC_OAUTH_TOKEN").expect("ANTHROPIC_OAUTH_TOKEN");
|
|
let p = cm_llm::AnthropicProvider::new(token);
|
|
let req = ChatRequest {
|
|
system: "You judge whether a phase of automated work is complete.\n\nRespond with STRICT JSON ONLY: {\"met\": true|false, \"reason\": \"one sentence\"}".into(),
|
|
model: "claude-haiku-4-5-20251001".into(),
|
|
messages: vec![ChatMessage {
|
|
role: ChatRole::User,
|
|
parts: vec![ContentPart::text(
|
|
"COMPLETION CONDITION:\nThe research output names at least two concrete tradeoffs.\n\nEVIDENCE:\nThe agent produced: (1) fail-closed blocks progress on evaluator outage; (2) fail-open can falsely approve. Both named with consequences.",
|
|
)],
|
|
}],
|
|
tools: vec![],
|
|
max_tokens: 512,
|
|
web_search: false,
|
|
};
|
|
let mut text = String::new();
|
|
let mut stream = p.stream(req).await.expect("stream");
|
|
while let Some(ev) = stream.next().await {
|
|
match ev {
|
|
Ok(LlmEvent::TextDelta(t)) => text.push_str(&t),
|
|
Ok(LlmEvent::Usage { input_tokens, .. }) => eprintln!("INPUT_TOKENS={input_tokens}"),
|
|
Ok(_) => {}
|
|
Err(e) => {
|
|
eprintln!("ERR: {e}");
|
|
std::process::exit(1);
|
|
}
|
|
}
|
|
}
|
|
println!("REPLY: {text}");
|
|
}
|