Omar SobhandClaude Opus 5.5
527384c5fb
The crate-wide guard exists so nobody hand-rolls a model call around the
provider layer. The proxy originates no calls: it relays a mission container's
own request and swaps the credential, reading the same env and auth mode. CI
caught the new module (72fb0dc); local runs had covered only touched modules.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
693 lines
31 KiB
Rust
693 lines
31 KiB
Rust
//! The Anthropic provider backed by the SUBSCRIPTION token, not the metered key.
|
|
//!
|
|
//! Two Anthropic credentials reach this server and they bill differently:
|
|
//!
|
|
//! - `ANTHROPIC_API_KEY` (`sk-ant-api…`) — metered, pay-as-you-go, and the thing
|
|
//! that runs out. Every mission VM already avoids it: `mission_runtime` sends
|
|
//! only the subscription token into a guest, deliberately.
|
|
//! - `ANTHROPIC_OAUTH_TOKEN` / `CLAUDE_CODE_OAUTH_TOKEN` (`sk-ant-oat…`) — the
|
|
//! Claude Code subscription, which is what the CLI inside every VM runs on.
|
|
//!
|
|
//! Server-side model calls that went through `Runtime::complete` with a bare
|
|
//! model name resolved to the DEFAULT provider — the metered key. So the roster
|
|
//! planner died with
|
|
//! `400 … "Your credit balance is too low to access the Anthropic API"` while
|
|
//! every mission on the same machine kept running fine on the subscription.
|
|
//! The harness reported it honestly as FAIL-NORUN rather than a passing scenario,
|
|
//! which is the only reason it was visible at all.
|
|
//!
|
|
//! This is the one place that turns the subscription token into a provider.
|
|
//! `evaluator::subscription_judge` had its own copy; there is now one.
|
|
|
|
/// The subscription-backed provider, or `None` when no usable token is present.
|
|
///
|
|
/// Checks the `sk-ant-oat` prefix rather than trusting the variable name: an
|
|
/// `sk-ant-api` key pasted into the OAuth slot would authenticate and then bill
|
|
/// the metered account, which is the failure this module exists to prevent —
|
|
/// silently, and with the same error weeks later.
|
|
pub fn provider() -> Option<cm_llm::AnthropicProvider> {
|
|
for var in ["ANTHROPIC_OAUTH_TOKEN", "CLAUDE_CODE_OAUTH_TOKEN"] {
|
|
let Ok(token) = std::env::var(var) else {
|
|
continue;
|
|
};
|
|
let token = token.trim();
|
|
if token.is_empty() {
|
|
continue;
|
|
}
|
|
if !is_subscription_token(token) {
|
|
eprintln!(
|
|
"subscription: {var} is set but is not a Claude Code setup token \
|
|
(expected sk-ant-oat…) — ignoring it rather than billing the \
|
|
metered key by accident"
|
|
);
|
|
continue;
|
|
}
|
|
return Some(cm_llm::AnthropicProvider::new(token.to_string()));
|
|
}
|
|
None
|
|
}
|
|
|
|
/// Whether a token is a Claude Code subscription token rather than an API key.
|
|
pub fn is_subscription_token(token: &str) -> bool {
|
|
token.trim().starts_with("sk-ant-oat")
|
|
}
|
|
|
|
/// One completion on the subscription, mirroring `Runtime::complete`'s contract
|
|
/// so a caller can swap between them without reshaping its call.
|
|
///
|
|
/// Falls back to the caller's runtime when no subscription token exists, so a
|
|
/// deployment without one behaves exactly as it did before.
|
|
pub async fn complete_or(
|
|
runtime: &cm_runtime::Runtime,
|
|
system: &str,
|
|
user: &str,
|
|
model: &str,
|
|
max_tokens: u32,
|
|
// Carried explicitly rather than defaulted. The Master Planner and the claw
|
|
// enhancer both pass `true`, and a helper that quietly dropped it would take
|
|
// web search away from two features while every test still passed.
|
|
web_search: bool,
|
|
) -> Result<String, String> {
|
|
// A `name:model` spec is an operator's explicit provider choice — the swarm
|
|
// worker model is literally configured that way (`kimi:kimi-k2.6`), and
|
|
// `Runtime::resolve_provider` honours it. Forcing that onto Anthropic would
|
|
// silently run someone's chosen model on the wrong provider, which is the
|
|
// same class of bug as this module exists to fix, only pointed the other
|
|
// way. Only a BARE name is ambiguous, and a bare name is what resolves to
|
|
// the default provider — the metered key.
|
|
if !is_bare_model_name(model) || provider().is_none() {
|
|
return runtime
|
|
.complete(system, user, model, max_tokens, web_search)
|
|
.await;
|
|
}
|
|
let provider = provider().expect("checked just above");
|
|
complete_with(&provider, system, user, model, max_tokens, web_search).await
|
|
}
|
|
|
|
/// Whether a model string names a model without naming a provider.
|
|
pub fn is_bare_model_name(model: &str) -> bool {
|
|
!model.contains(':')
|
|
}
|
|
|
|
/// How long to wait before each retry. Four attempts, ~30s of patience total.
|
|
///
|
|
/// The subscription has no credit wall, but it does have a rate limit, and a
|
|
/// roster proposal is a single one-shot call: a 429 that a browser would shrug
|
|
/// off used to fail the whole "propose a team" button. Measured on this
|
|
/// deployment — moving the roster onto the subscription turned
|
|
/// `400 credit balance too low` into `429 rate_limit_error`, i.e. a wall that
|
|
/// clears on its own became the failure mode, so waiting is the right answer.
|
|
const BACKOFF_SECS: &[u64] = &[2, 8, 20];
|
|
|
|
/// Whether an error is worth waiting out rather than reporting.
|
|
///
|
|
/// Deliberately narrow. A 400 (bad request), 401 (wrong token) or 404 (unknown
|
|
/// model) will never succeed on a retry, and retrying them turns a legible
|
|
/// error into a 30-second hang followed by the same error.
|
|
fn is_transient(e: &cm_llm::LlmError) -> bool {
|
|
use cm_llm::LlmError;
|
|
match e {
|
|
// The transport never reached Anthropic — a dropped connection or a
|
|
// DNS blip, not a rejected request.
|
|
LlmError::Transport(_) => true,
|
|
LlmError::Api(detail) => {
|
|
// `anthropic.rs` formats these as `"{status}: {body}"`.
|
|
detail.starts_with("429")
|
|
|| detail.starts_with("500")
|
|
|| detail.starts_with("502")
|
|
|| detail.starts_with("503")
|
|
|| detail.starts_with("529")
|
|
|| detail.contains("rate_limit")
|
|
|| detail.contains("overloaded")
|
|
}
|
|
LlmError::Scenario(_) | LlmError::Wire(_) => false,
|
|
}
|
|
}
|
|
|
|
/// Models to try, in order, when the requested one is rate limited.
|
|
///
|
|
/// The order is capability first, then independence:
|
|
///
|
|
/// opus -> sonnet -> haiku one account, three tiers. A throttle usually
|
|
/// hits a tier, so stepping down often clears it.
|
|
/// -> kimi -> glm two separately funded accounts. Now an
|
|
/// Anthropic outage, not just a throttle, is
|
|
/// survivable.
|
|
/// -> local our own GPU. Nothing left to be down.
|
|
///
|
|
/// Every model id here was probed on this deployment 2026-08-09 and answered
|
|
/// 200: the four Anthropic tiers on the subscription, `kimi-k2.7-code` on
|
|
/// api.kimi.com/coding, `glm-4.7` on z.ai, and `ornith-fleet:9b` on the fleet.
|
|
/// Configured is not the same as working — see `preflight`, which re-checks
|
|
/// them at boot, because a link nobody exercises is discovered broken during
|
|
/// the outage it existed for.
|
|
///
|
|
/// The last link runs on our OWN hardware. Every other entry — and every other
|
|
/// link above it — depends on somebody else's account staying funded and
|
|
/// unthrottled; `local:` depends on a GPU in the next room. It is last because
|
|
/// it is the weakest model, and present because a chain whose every link is
|
|
/// external is not a fallback chain, it is one outage in a trench coat.
|
|
///
|
|
/// Note the model half contains a colon (`ornith-fleet:9b`), which is why
|
|
/// `resolve_provider` splits on the FIRST one only.
|
|
///
|
|
/// Override with `CLAWMATES_MODEL_FALLBACK` (comma-separated). An empty value
|
|
/// disables fallback and restores plain "503 and wait".
|
|
///
|
|
/// Ordered by the operator's model policy: sonnet-5 is the working tier, and
|
|
/// haiku sits BELOW it as a last-resort Anthropic link rather than as a peer —
|
|
/// a degraded answer beats a 503, but it must never be reached while a capable
|
|
/// model has capacity.
|
|
const DEFAULT_FALLBACK: &str = "claude-sonnet-5,claude-haiku-4-5-20251001,\
|
|
kimi:kimi-k2.7-code,glm:glm-4.7,local:ornith-fleet:9b";
|
|
|
|
/// The chain to walk after `requested`, with `requested` itself removed so a
|
|
/// capped model is never retried as its own fallback.
|
|
pub fn fallback_chain(requested: &str) -> Vec<String> {
|
|
let raw =
|
|
std::env::var("CLAWMATES_MODEL_FALLBACK").unwrap_or_else(|_| DEFAULT_FALLBACK.to_string());
|
|
raw.split(',')
|
|
.map(str::trim)
|
|
.filter(|m| !m.is_empty() && *m != requested.trim())
|
|
.map(str::to_string)
|
|
.collect()
|
|
}
|
|
|
|
/// Whether a failure means "this model has no capacity right now" as opposed
|
|
/// to "this request was wrong".
|
|
///
|
|
/// The distinction is the whole safety of the chain: walking it on a malformed
|
|
/// prompt would ask three models the same bad question and report the third
|
|
/// one's confusion, while walking it on a rate limit is exactly the point.
|
|
pub fn is_capacity_failure(err: &str) -> bool {
|
|
err.contains("rate_limit") || err.contains("429") || err.contains("credit balance")
|
|
}
|
|
|
|
/// One completion, stepping down `fallback_chain` when a model has no capacity.
|
|
///
|
|
/// Returns the text **and the model that actually produced it**. Callers must
|
|
/// persist that second value: a plan drafted by the third link in the chain and
|
|
/// filed as an opus plan is a silent quality change, which is the failure shape
|
|
/// this project keeps paying for. Every hop is logged.
|
|
pub async fn complete_with_fallback(
|
|
runtime: &cm_runtime::Runtime,
|
|
system: &str,
|
|
user: &str,
|
|
model: &str,
|
|
max_tokens: u32,
|
|
web_search: bool,
|
|
) -> Result<(String, String), String> {
|
|
let mut last = match complete_or(runtime, system, user, model, max_tokens, web_search).await {
|
|
Ok(text) => return Ok((text, model.to_string())),
|
|
Err(e) if is_capacity_failure(&e) => e,
|
|
// A real error. Do not launder it through two more models.
|
|
Err(e) => return Err(e),
|
|
};
|
|
for next in fallback_chain(model) {
|
|
eprintln!("model fallback: {model} has no capacity ({last}) — trying {next}");
|
|
match complete_or(runtime, system, user, &next, max_tokens, web_search).await {
|
|
Ok(text) => {
|
|
eprintln!("model fallback: {next} answered in place of {model}");
|
|
return Ok((text, next));
|
|
}
|
|
Err(e) if is_capacity_failure(&e) => last = e,
|
|
Err(e) => return Err(format!("fallback {next}: {e}")),
|
|
}
|
|
}
|
|
Err(last)
|
|
}
|
|
|
|
/// What a probe of one link found.
|
|
///
|
|
/// `Throttled` is deliberately NOT a failure. A 429 means the spec resolved, the
|
|
/// credential authenticated, and the provider simply had no capacity this
|
|
/// second — which is the exact condition the chain exists to route around. A
|
|
/// report that painted it red would train an operator to ignore the red.
|
|
#[derive(Debug, Clone, PartialEq)]
|
|
pub enum LinkStatus {
|
|
Answered,
|
|
Throttled(String),
|
|
/// Never came back. Its own state because it is the one that used to make
|
|
/// the whole report vanish: with no timeout, a single hung provider meant
|
|
/// silence from the tool built to prevent silence.
|
|
TimedOut,
|
|
/// The spec named a provider the registry does not have, so
|
|
/// `resolve_provider` silently fell back to the DEFAULT provider. The link
|
|
/// would "work" while running on entirely the wrong model.
|
|
Unregistered,
|
|
Broken(String),
|
|
}
|
|
|
|
impl LinkStatus {
|
|
pub fn usable(&self) -> bool {
|
|
matches!(self, LinkStatus::Answered | LinkStatus::Throttled(_))
|
|
}
|
|
fn label(&self) -> String {
|
|
match self {
|
|
LinkStatus::Answered => "ok".into(),
|
|
LinkStatus::Throttled(_) => "throttled (configured, no capacity now)".into(),
|
|
LinkStatus::TimedOut => {
|
|
format!("TIMED OUT after {}s — treat as down", PROBE_TIMEOUT.as_secs())
|
|
}
|
|
LinkStatus::Unregistered => "UNREGISTERED — resolves to the DEFAULT provider".into(),
|
|
LinkStatus::Broken(e) => format!("BROKEN: {}", e.chars().take(120).collect::<String>()),
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Probe every link of the chain, head model included.
|
|
///
|
|
/// Eight tokens each, through the SAME path a real call takes, so it proves
|
|
/// resolution and reachability rather than that a string is present in a config
|
|
/// file. The distinction matters here more than usual: `resolve_provider` falls
|
|
/// back to the default provider for an unknown provider name, so a typo in
|
|
/// `kimi:` does not error — it quietly runs on Anthropic, and the chain reads
|
|
/// as five providers while being one.
|
|
/// Per-link ceiling. Generous on purpose: `complete_or` spends up to 30s in its
|
|
/// own backoff before giving up, so anything under that would report a merely
|
|
/// throttled link as hung.
|
|
const PROBE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(60);
|
|
|
|
pub async fn preflight(runtime: &cm_runtime::Runtime, head: &str) -> Vec<(String, LinkStatus)> {
|
|
let mut out = Vec::new();
|
|
for spec in std::iter::once(head.to_string()).chain(fallback_chain(head)) {
|
|
// A qualified spec whose provider is missing resolves to the default —
|
|
// detected the same way `cross_provider_judge` does it, by asking what
|
|
// the model half came back as.
|
|
if spec.contains(':') {
|
|
// Unrouted specs come back WHOLE; routed ones come back as the part
|
|
// after the FIRST colon. Testing "does it still contain a colon"
|
|
// reads the same and is wrong: `local:ornith-fleet:9b` resolves
|
|
// correctly to model `ornith-fleet:9b`, which does. This probe
|
|
// reported a provider the server had just registered as
|
|
// UNREGISTERED on its first live run, which is how the same latent
|
|
// bug was found in `evaluator::cross_provider_judge`.
|
|
let (_, resolved) = runtime.resolve_provider(&spec);
|
|
if resolved == spec {
|
|
out.push((spec.clone(), LinkStatus::Unregistered));
|
|
continue;
|
|
}
|
|
}
|
|
// A non-empty system prompt. Kimi rejects an empty one outright —
|
|
// `400 the message at position 0 with role 'system' must not be empty` —
|
|
// so an empty probe reported a healthy provider as BROKEN on the first
|
|
// live run. The probe must look like the traffic it stands in for.
|
|
// NOT awaited here — the timeout has to wrap the FUTURE. Awaiting first
|
|
// and wrapping the result compiles, reads correctly, and bounds nothing.
|
|
let probe = complete_or(
|
|
runtime,
|
|
"You are a reachability probe.",
|
|
"Reply with exactly: OK",
|
|
&spec,
|
|
8,
|
|
false,
|
|
);
|
|
let status = match tokio::time::timeout(PROBE_TIMEOUT, probe).await {
|
|
Err(_) => LinkStatus::TimedOut,
|
|
Ok(Ok(_)) => LinkStatus::Answered,
|
|
Ok(Err(e)) if is_capacity_failure(&e) => LinkStatus::Throttled(e),
|
|
Ok(Err(e)) => LinkStatus::Broken(e),
|
|
};
|
|
// Emitted as it resolves, not collected and printed at the end. A later
|
|
// link that hangs must not be able to hide the ones already checked.
|
|
eprintln!("fallback chain: {spec:<32} {}", status.label());
|
|
out.push((spec, status));
|
|
}
|
|
out
|
|
}
|
|
|
|
/// Probe the chain at boot and write the result to stderr.
|
|
///
|
|
/// Spawned rather than awaited, like `runtime_preflight`: this is diagnostic and
|
|
/// must never delay the server coming up. Loud when a link is unusable, because
|
|
/// the whole point of a chain is that nobody looks at it until the day it has to
|
|
/// work.
|
|
pub fn report_at_boot(runtime: cm_runtime::Runtime) {
|
|
tokio::spawn(async move {
|
|
let head = std::env::var("CLAWMATES_PREFLIGHT_HEAD")
|
|
.unwrap_or_else(|_| "claude-opus-5".to_string());
|
|
let links = preflight(&runtime, &head).await;
|
|
let bad: Vec<_> = links.iter().filter(|(_, s)| !s.usable()).collect();
|
|
eprintln!(
|
|
"fallback chain ({} link(s), {} usable):",
|
|
links.len(),
|
|
links.len() - bad.len()
|
|
);
|
|
for (spec, status) in &links {
|
|
eprintln!(" {spec:<32} {}", status.label());
|
|
}
|
|
if !bad.is_empty() {
|
|
eprintln!(
|
|
"fallback chain: WARNING — {} link(s) are NOT usable. The chain is \
|
|
shorter than it reads, and the shortfall only shows up during the \
|
|
outage it exists for.",
|
|
bad.len()
|
|
);
|
|
}
|
|
});
|
|
}
|
|
|
|
/// Turn a `complete_or` failure into the right API error.
|
|
///
|
|
/// A rate limit that outlived the backoff is not a bug in this server, and
|
|
/// reporting it as one costs an operator a trip through the logs to find out
|
|
/// the answer was "wait". Measured: a bare 16-token probe with the same token
|
|
/// returned 429 with `x-should-retry: true` — Anthropic itself says try again.
|
|
pub fn as_api_error(err: &str) -> crate::error::ApiError {
|
|
if err.contains("rate_limit") || err.contains("429") {
|
|
return crate::error::ApiError::Unavailable(
|
|
"the Claude Code subscription is rate limited right now — this \
|
|
clears on its own; try again shortly"
|
|
.into(),
|
|
);
|
|
}
|
|
crate::error::ApiError::Internal
|
|
}
|
|
|
|
/// Stream one request and collect its text, waiting out transient failures.
|
|
async fn complete_with(
|
|
provider: &cm_llm::AnthropicProvider,
|
|
system: &str,
|
|
user: &str,
|
|
model: &str,
|
|
max_tokens: u32,
|
|
web_search: bool,
|
|
) -> Result<String, String> {
|
|
let mut attempt = 0usize;
|
|
loop {
|
|
match attempt_once(provider, system, user, model, max_tokens, web_search).await {
|
|
Ok(text) => return Ok(text),
|
|
Err((stage, e)) => {
|
|
let Some(delay) = BACKOFF_SECS.get(attempt).copied().filter(|_| is_transient(&e))
|
|
else {
|
|
return Err(format!("subscription {stage}: {e}"));
|
|
};
|
|
eprintln!(
|
|
"subscription {stage}: {e} — retrying in {delay}s \
|
|
(attempt {} of {})",
|
|
attempt + 2,
|
|
BACKOFF_SECS.len() + 1
|
|
);
|
|
tokio::time::sleep(std::time::Duration::from_secs(delay)).await;
|
|
attempt += 1;
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
/// One attempt. The collected text is discarded on failure, so a retry never
|
|
/// concatenates a partial answer onto a whole one.
|
|
async fn attempt_once(
|
|
provider: &cm_llm::AnthropicProvider,
|
|
system: &str,
|
|
user: &str,
|
|
model: &str,
|
|
max_tokens: u32,
|
|
web_search: bool,
|
|
) -> Result<String, (&'static str, cm_llm::LlmError)> {
|
|
use cm_llm::{ChatMessage, ChatRequest, ChatRole, ContentPart, LlmEvent, LlmProvider};
|
|
use futures::StreamExt as _;
|
|
|
|
let request = ChatRequest {
|
|
system: system.to_string(),
|
|
model: model.to_string(),
|
|
messages: vec![ChatMessage {
|
|
role: ChatRole::User,
|
|
parts: vec![ContentPart::text(user)],
|
|
}],
|
|
tools: vec![],
|
|
max_tokens,
|
|
web_search,
|
|
};
|
|
let mut stream = provider.stream(request).await.map_err(|e| ("call", e))?;
|
|
let mut text = String::new();
|
|
while let Some(event) = stream.next().await {
|
|
match event {
|
|
Ok(LlmEvent::TextDelta(t)) => text.push_str(&t),
|
|
Ok(_) => {}
|
|
Err(e) => return Err(("stream", e)),
|
|
}
|
|
}
|
|
Ok(text)
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
/// Every server-side model call that should be on the subscription IS.
|
|
///
|
|
/// `validator_preflight` is the deliberate exception: it probes whatever
|
|
/// spec an operator configured (today `glm:glm-4.7`), and forcing it onto
|
|
/// Anthropic would make it prove the wrong thing — it exists to answer "is
|
|
/// the configured validator reachable".
|
|
/// The first version of this test grepped for the literal
|
|
/// `runtime.complete(` and passed while FOUR more call sites — the phase
|
|
/// planner, both swarm calls, and a second enhance path — still billed the
|
|
/// metered key. They were spelled `state.runtime` or wrapped across lines,
|
|
/// so the receiver name was never the thing to look for. Match the METHOD.
|
|
#[test]
|
|
fn no_server_side_call_silently_uses_the_metered_key() {
|
|
let sources = [
|
|
("routes/mission_roster.rs", include_str!("routes/mission_roster.rs")),
|
|
("routes/mission_plan.rs", include_str!("routes/mission_plan.rs")),
|
|
("routes/planner.rs", include_str!("routes/planner.rs")),
|
|
("routes/claws.rs", include_str!("routes/claws.rs")),
|
|
("swarm.rs", include_str!("swarm.rs")),
|
|
];
|
|
for (name, src) in sources {
|
|
assert!(
|
|
!src.contains(".complete("),
|
|
"{name} calls Runtime::complete directly — a bare model name there \
|
|
resolves to the DEFAULT provider, which is the metered API key. \
|
|
Use `subscription::complete_or`, which passes a `name:model` \
|
|
spec through untouched."
|
|
);
|
|
}
|
|
// And the exception stays an exception, on purpose.
|
|
assert!(
|
|
include_str!("validator_preflight.rs").contains("runtime.complete("),
|
|
"validator_preflight must keep probing the CONFIGURED spec"
|
|
);
|
|
}
|
|
|
|
/// Only errors that can clear on their own are waited out.
|
|
///
|
|
/// The negative half is the point: a 400 or a 401 retried three times is a
|
|
/// 30-second hang ending in the identical message, which reads as a stall
|
|
/// rather than a bad request — the failure mode this project keeps hitting.
|
|
#[test]
|
|
fn a_wall_that_clears_is_waited_out_and_one_that_does_not_is_not() {
|
|
use cm_llm::LlmError;
|
|
let api = |s: &str| LlmError::Api(s.to_string());
|
|
|
|
assert!(is_transient(&api(
|
|
"429 Too Many Requests: {\"type\":\"rate_limit_error\"}"
|
|
)));
|
|
assert!(is_transient(&api("529: overloaded_error")));
|
|
assert!(is_transient(&api("503 Service Unavailable")));
|
|
assert!(is_transient(&LlmError::Transport("connection reset".into())));
|
|
|
|
// The exact error that started this: it never clears by waiting, it
|
|
// clears by moving to the other credential — which is now done.
|
|
assert!(!is_transient(&api(
|
|
"400 Bad Request: Your credit balance is too low"
|
|
)));
|
|
assert!(!is_transient(&api("401 Unauthorized: invalid x-api-key")));
|
|
assert!(!is_transient(&api("404 Not Found: model not found")));
|
|
assert!(!is_transient(&LlmError::Wire("bad json".into())));
|
|
}
|
|
|
|
/// Nobody hand-rolls their own Anthropic HTTP call.
|
|
///
|
|
/// `phase_summarizer` did — its own `reqwest` POST to `api.anthropic.com`
|
|
/// with `x-api-key: $ANTHROPIC_API_KEY`. No audit of `.complete(` call
|
|
/// sites could ever have found it, and it was the last thing on this
|
|
/// deployment still billing an account with no credit: every phase summary
|
|
/// died with "credit balance is too low" while the phases themselves ran.
|
|
/// A call site is only routable if it goes through a provider, so walk the
|
|
/// whole crate rather than a hand-listed set of files.
|
|
#[test]
|
|
fn no_module_talks_to_anthropic_behind_the_providers_back() {
|
|
fn walk(dir: &std::path::Path, out: &mut Vec<std::path::PathBuf>) {
|
|
for entry in std::fs::read_dir(dir).expect("readable source dir") {
|
|
let path = entry.expect("readable entry").path();
|
|
if path.is_dir() {
|
|
walk(&path, out);
|
|
} else if path.extension().is_some_and(|e| e == "rs") {
|
|
out.push(path);
|
|
}
|
|
}
|
|
}
|
|
let root = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("src");
|
|
let mut files = Vec::new();
|
|
walk(&root, &mut files);
|
|
assert!(files.len() > 20, "source walk found suspiciously few files");
|
|
|
|
for path in files {
|
|
// This module names the host in prose; it is the one that may.
|
|
if path.ends_with("subscription.rs") {
|
|
continue;
|
|
}
|
|
// The LLM proxy never ORIGINATES a model call: it relays a mission
|
|
// container's own Claude Code request byte for byte and swaps in
|
|
// the credential. Routing it through `complete_or` would re-build
|
|
// (and could re-route) what the agent asked for. Credential choice
|
|
// for relayed calls is `llm_proxy::upstream`, and it reads the same
|
|
// env and auth mode (`runtime_auth_mode`) as everything else.
|
|
if path.ends_with("llm_proxy.rs") {
|
|
continue;
|
|
}
|
|
let src = std::fs::read_to_string(&path).expect("readable source");
|
|
for needle in ["api.anthropic.com", "\"x-api-key\""] {
|
|
assert!(
|
|
!src.contains(needle),
|
|
"{} contains {needle} — build the request through cm_llm and \
|
|
route it via `subscription::complete_or`, so credential \
|
|
choice and the capacity fallback live in ONE place",
|
|
path.display()
|
|
);
|
|
}
|
|
}
|
|
}
|
|
|
|
/// A model name may contain a colon, and "unregistered" must not mean that.
|
|
///
|
|
/// `resolve_provider` returns the spec unchanged when it does not recognise
|
|
/// the provider and the part after the FIRST colon when it does. The obvious
|
|
/// test — "does the model half still contain a colon" — reads the same and
|
|
/// is wrong the moment a model id has one. `ornith-fleet:9b` has one, and
|
|
/// the live preflight reported a provider the server had just registered as
|
|
/// UNREGISTERED. The identical bug was in `cross_provider_judge`, where it
|
|
/// would have refused a perfectly good independent judge.
|
|
#[test]
|
|
fn a_colon_in_the_model_name_is_not_a_missing_provider() {
|
|
// What `resolve_provider` returns in each case.
|
|
fn routed(spec: &str) -> &str {
|
|
spec.split_once(':').map(|(_, m)| m).unwrap_or(spec)
|
|
}
|
|
|
|
for spec in ["local:ornith-fleet:9b", "glm:glm-4.7", "kimi:kimi-k2.7-code"] {
|
|
assert_ne!(routed(spec), spec, "{spec} routed must not equal the whole spec");
|
|
}
|
|
// An unrecognised provider comes back WHOLE — the only true signal.
|
|
assert_eq!(routed("nosuch"), "nosuch");
|
|
// And the case that made the naive colon test look correct for so long.
|
|
assert!(routed("local:ornith-fleet:9b").contains(':'));
|
|
}
|
|
|
|
/// A throttled link is usable; an unregistered one is not.
|
|
///
|
|
/// The second is the dangerous one and the reason `preflight` checks
|
|
/// resolution separately from reachability. `resolve_provider` falls back to
|
|
/// the DEFAULT provider when it does not recognise a provider name, so a
|
|
/// typo in `kimi:` does not error — it quietly runs on Anthropic, and a
|
|
/// chain that reads as three accounts is really one. A reachability-only
|
|
/// probe would call that link green.
|
|
#[test]
|
|
fn only_a_link_that_could_never_answer_counts_as_unusable() {
|
|
assert!(LinkStatus::Answered.usable());
|
|
assert!(LinkStatus::Throttled("429 rate_limit".into()).usable());
|
|
|
|
assert!(!LinkStatus::Unregistered.usable());
|
|
assert!(!LinkStatus::Broken("401 invalid key".into()).usable());
|
|
|
|
// The labels must not read alike: "throttled" is a wait and
|
|
// "unregistered" is a config bug, and an operator acts differently on
|
|
// each.
|
|
assert!(LinkStatus::Throttled(String::new()).label().contains("configured"));
|
|
assert!(LinkStatus::Unregistered.label().contains("DEFAULT provider"));
|
|
}
|
|
|
|
/// The chain never retries the capped model as its own fallback.
|
|
///
|
|
/// Without the filter, asking for haiku while haiku is capped would try
|
|
/// haiku, fail, and try haiku again — a chain that looks like resilience
|
|
/// and delivers none.
|
|
#[test]
|
|
fn the_chain_excludes_the_model_that_just_failed() {
|
|
// No env override in scope: this asserts the SHIPPED default.
|
|
let chain = fallback_chain("claude-opus-5");
|
|
assert_eq!(
|
|
chain,
|
|
vec![
|
|
"claude-sonnet-5",
|
|
"claude-haiku-4-5-20251001",
|
|
"kimi:kimi-k2.7-code",
|
|
"glm:glm-4.7",
|
|
"local:ornith-fleet:9b",
|
|
]
|
|
);
|
|
// Three providers behind five links. A chain that steps down three
|
|
// Anthropic tiers and stops is a tier ladder, not a fallback chain: one
|
|
// account being unreachable would end it.
|
|
let families: std::collections::BTreeSet<_> = chain
|
|
.iter()
|
|
.map(|m| m.split_once(':').map(|(p, _)| p).unwrap_or("anthropic"))
|
|
.collect();
|
|
assert!(
|
|
families.len() >= 3,
|
|
"the chain must span more than one account, got {families:?}"
|
|
);
|
|
// The last link must survive `resolve_provider`'s split, which takes the
|
|
// FIRST colon only — `local:ornith-fleet:9b` is provider `local`, model
|
|
// `ornith-fleet:9b`, and a split on the last colon would ask for a
|
|
// provider named `local:ornith-fleet`.
|
|
let last = chain.last().unwrap();
|
|
let (provider, model) = last.split_once(':').expect("a provider-qualified spec");
|
|
assert_eq!(provider, "local");
|
|
assert_eq!(model, "ornith-fleet:9b");
|
|
assert!(!fallback_chain("claude-haiku-4-5-20251001")
|
|
.iter()
|
|
.any(|m| m == "claude-haiku-4-5-20251001"));
|
|
}
|
|
|
|
/// The chain is walked for "no capacity" and NOT for "bad request".
|
|
///
|
|
/// Walking it on a malformed prompt would ask three models the same bad
|
|
/// question and report the third one's confusion as the answer, burning
|
|
/// the two credentials that still work in order to hide the real error.
|
|
#[test]
|
|
fn only_a_capacity_failure_steps_down_the_chain() {
|
|
assert!(is_capacity_failure(
|
|
"subscription call: provider returned an error: 429 Too Many Requests"
|
|
));
|
|
assert!(is_capacity_failure("rate_limit_error"));
|
|
// The metered key's wall counts too — same meaning, different wording.
|
|
assert!(is_capacity_failure(
|
|
"400: Your credit balance is too low to access the Anthropic API"
|
|
));
|
|
|
|
assert!(!is_capacity_failure("400: messages.0: text content is empty"));
|
|
assert!(!is_capacity_failure("401: invalid x-api-key"));
|
|
assert!(!is_capacity_failure("404: model not found"));
|
|
}
|
|
|
|
/// An operator's explicit provider choice is never hijacked.
|
|
///
|
|
/// The swarm worker model is a configured `name:model` spec. Routing that
|
|
/// onto the subscription would run someone's chosen Kimi or GLM model on
|
|
/// Anthropic and report success — the same silent-substitution bug as the
|
|
/// metered key, aimed the other way.
|
|
#[test]
|
|
fn a_provider_qualified_spec_is_left_alone() {
|
|
assert!(is_bare_model_name("claude-opus-4-8"));
|
|
assert!(is_bare_model_name("claude-haiku-4-5-20251001"));
|
|
assert!(!is_bare_model_name("kimi:kimi-k2.6"));
|
|
assert!(!is_bare_model_name("glm:glm-4.7"));
|
|
}
|
|
|
|
/// A metered key in the OAuth slot must be REFUSED, not used.
|
|
///
|
|
/// Accepting it would authenticate, work, and bill the pay-as-you-go account
|
|
/// — the exact bill this module exists to stop, discovered weeks later when
|
|
/// it runs out mid-mission.
|
|
#[test]
|
|
fn only_a_setup_token_counts_as_the_subscription() {
|
|
assert!(is_subscription_token("sk-ant-oat01-abc"));
|
|
assert!(!is_subscription_token("sk-ant-api03-abc"));
|
|
assert!(!is_subscription_token(""));
|
|
assert!(!is_subscription_token("oat-but-not-anthropic"));
|
|
}
|
|
}
|