osobh/clawmates
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(llm): six-link fallback chain, and a preflight that proves it

Browse Source 
opus -> sonnet -> haiku -> kimi -> glm -> local. The order is capability first,
then independence: three Anthropic tiers on one account (a throttle usually hits
a tier, so stepping down often clears it), then two separately funded accounts
(now an outage, not just a throttle, is survivable), then our own GPU (nothing
left to be down). Every id was probed on this deployment and answered 200.

The preflight is the more important half. Configured is not working, and this
chain has a specific way of lying: `resolve_provider` falls back to the DEFAULT
provider when it does not recognise a provider name, so a typo in `kimi:` does
not error — it quietly runs on Anthropic, and a chain that reads as three
accounts is really one. A reachability-only probe calls that link green.

So `preflight` checks resolution and reachability separately, eight tokens per
link through the REAL call path, and reports four states. `Throttled` is
deliberately not a failure: a 429 means the spec resolved, the credential
authenticated, and there was no capacity this second — the exact condition the
chain exists to route around, and painting it red would train an operator to
ignore red. `Unregistered` and `Broken` are failures, and they get different
words because they need different fixes.

It runs at boot alongside validator_preflight and runtime_preflight, spawned so
it cannot delay startup. A chain is the one piece of infrastructure nobody looks
at until the day it has to work, so it is now checked on the days it does not.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-09 14:38:55 -07:00
co-authored by Claude Opus 5
parent 5afcf63324
commit c3ad5672fc
2 changed files with 153 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/cm-api/src/subscription.rs
+149 -7
View File
@@ -126,11 +126,21 @@ fn is_transient(e: &cm_llm::LlmError) -> bool {
/// Models to try, in order, when the requested one is rate limited.
///
/// Measured on this deployment 2026-08-08: opus and sonnet returned 429 on the
/// subscription while haiku returned 200, and GLM — funded separately, and
/// already the project's measured-best independent judge — returned 200. So the
/// default chain steps down within the subscription first, then leaves Anthropic
/// entirely rather than making a capped window mean "the planner is gone".
/// The order is capability first, then independence:
///
/// opus -> sonnet -> haiku one account, three tiers. A throttle usually
/// hits a tier, so stepping down often clears it.
/// -> kimi -> glm two separately funded accounts. Now an
/// Anthropic outage, not just a throttle, is
/// survivable.
/// -> local our own GPU. Nothing left to be down.
///
/// Every model id here was probed on this deployment 2026-08-09 and answered
/// 200: the four Anthropic tiers on the subscription, `kimi-k2.7-code` on
/// api.kimi.com/coding, `glm-4.7` on z.ai, and `ornith-fleet:9b` on the fleet.
/// Configured is not the same as working — see `preflight`, which re-checks
/// them at boot, because a link nobody exercises is discovered broken during
/// the outage it existed for.
///
/// The last link runs on our OWN hardware. Every other entry — and every other
/// link above it — depends on somebody else's account staying funded and
@@ -143,7 +153,8 @@ fn is_transient(e: &cm_llm::LlmError) -> bool {
///
/// Override with `CLAWMATES_MODEL_FALLBACK` (comma-separated). An empty value
/// disables fallback and restores plain "503 and wait".
const DEFAULT_FALLBACK: &str = "claude-haiku-4-5-20251001,glm:glm-4.7,local:ornith-fleet:9b";
const DEFAULT_FALLBACK: &str = "claude-sonnet-4-6,claude-haiku-4-5-20251001,\
kimi:kimi-k2.7-code,glm:glm-4.7,local:ornith-fleet:9b";
/// The chain to walk after `requested`, with `requested` itself removed so a
/// capped model is never retried as its own fallback.
@@ -201,6 +212,101 @@ pub async fn complete_with_fallback(
Err(last)
}
/// What a probe of one link found.
///
/// `Throttled` is deliberately NOT a failure. A 429 means the spec resolved, the
/// credential authenticated, and the provider simply had no capacity this
/// second — which is the exact condition the chain exists to route around. A
/// report that painted it red would train an operator to ignore the red.
#[derive(Debug, Clone, PartialEq)]
pub enum LinkStatus {
Answered,
Throttled(String),
/// The spec named a provider the registry does not have, so
/// `resolve_provider` silently fell back to the DEFAULT provider. The link
/// would "work" while running on entirely the wrong model.
Unregistered,
Broken(String),
}
impl LinkStatus {
pub fn usable(&self) -> bool {
matches!(self, LinkStatus::Answered | LinkStatus::Throttled(_))
}
fn label(&self) -> String {
match self {
LinkStatus::Answered => "ok".into(),
LinkStatus::Throttled(_) => "throttled (configured, no capacity now)".into(),
LinkStatus::Unregistered => "UNREGISTERED — resolves to the DEFAULT provider".into(),
LinkStatus::Broken(e) => format!("BROKEN: {}", e.chars().take(120).collect::<String>()),
}
}
}
/// Probe every link of the chain, head model included.
///
/// Eight tokens each, through the SAME path a real call takes, so it proves
/// resolution and reachability rather than that a string is present in a config
/// file. The distinction matters here more than usual: `resolve_provider` falls
/// back to the default provider for an unknown provider name, so a typo in
/// `kimi:` does not error — it quietly runs on Anthropic, and the chain reads
/// as five providers while being one.
pub async fn preflight(runtime: &cm_runtime::Runtime, head: &str) -> Vec<(String, LinkStatus)> {
let mut out = Vec::new();
for spec in std::iter::once(head.to_string()).chain(fallback_chain(head)) {
// A qualified spec whose provider is missing resolves to the default —
// detected the same way `cross_provider_judge` does it, by asking what
// the model half came back as.
if let Some((name, _)) = spec.split_once(':') {
let (_, resolved) = runtime.resolve_provider(&spec);
if resolved.contains(':') || resolved == spec {
let _ = name;
out.push((spec.clone(), LinkStatus::Unregistered));
continue;
}
}
let status = match complete_or(runtime, "", "Reply with exactly: OK", &spec, 8, false).await
{
Ok(_) => LinkStatus::Answered,
Err(e) if is_capacity_failure(&e) => LinkStatus::Throttled(e),
Err(e) => LinkStatus::Broken(e),
};
out.push((spec, status));
}
out
}
/// Probe the chain at boot and write the result to stderr.
///
/// Spawned rather than awaited, like `runtime_preflight`: this is diagnostic and
/// must never delay the server coming up. Loud when a link is unusable, because
/// the whole point of a chain is that nobody looks at it until the day it has to
/// work.
pub fn report_at_boot(runtime: cm_runtime::Runtime) {
tokio::spawn(async move {
let head = std::env::var("CLAWMATES_PREFLIGHT_HEAD")
.unwrap_or_else(|_| "claude-opus-4-8".to_string());
let links = preflight(&runtime, &head).await;
let bad: Vec<_> = links.iter().filter(|(_, s)| !s.usable()).collect();
eprintln!(
"fallback chain ({} link(s), {} usable):",
links.len(),
links.len() - bad.len()
);
for (spec, status) in &links {
eprintln!(" {spec:<32} {}", status.label());
}
if !bad.is_empty() {
eprintln!(
"fallback chain: WARNING — {} link(s) are NOT usable. The chain is \
shorter than it reads, and the shortfall only shows up during the \
outage it exists for.",
bad.len()
);
}
});
}
/// Turn a `complete_or` failure into the right API error.
///
/// A rate limit that outlived the backoff is not a bug in this server, and
@@ -396,6 +502,29 @@ mod tests {
}
}
/// A throttled link is usable; an unregistered one is not.
///
/// The second is the dangerous one and the reason `preflight` checks
/// resolution separately from reachability. `resolve_provider` falls back to
/// the DEFAULT provider when it does not recognise a provider name, so a
/// typo in `kimi:` does not error — it quietly runs on Anthropic, and a
/// chain that reads as three accounts is really one. A reachability-only
/// probe would call that link green.
#[test]
fn only_a_link_that_could_never_answer_counts_as_unusable() {
assert!(LinkStatus::Answered.usable());
assert!(LinkStatus::Throttled("429 rate_limit".into()).usable());
assert!(!LinkStatus::Unregistered.usable());
assert!(!LinkStatus::Broken("401 invalid key".into()).usable());
// The labels must not read alike: "throttled" is a wait and
// "unregistered" is a config bug, and an operator acts differently on
// each.
assert!(LinkStatus::Throttled(String::new()).label().contains("configured"));
assert!(LinkStatus::Unregistered.label().contains("DEFAULT provider"));
}
/// The chain never retries the capped model as its own fallback.
///
/// Without the filter, asking for haiku while haiku is capped would try
@@ -408,11 +537,24 @@ mod tests {
assert_eq!(
chain,
vec![
"claude-sonnet-4-6",
"claude-haiku-4-5-20251001",
"kimi:kimi-k2.7-code",
"glm:glm-4.7",
"local:ornith-fleet:9b"
"local:ornith-fleet:9b",
]
);
// Three providers behind five links. A chain that steps down three
// Anthropic tiers and stops is a tier ladder, not a fallback chain: one
// account being unreachable would end it.
let families: std::collections::BTreeSet<_> = chain
.iter()
.map(|m| m.split_once(':').map(|(p, _)| p).unwrap_or("anthropic"))
.collect();
assert!(
families.len() >= 3,
"the chain must span more than one account, got {families:?}"
);
// The last link must survive `resolve_provider`'s split, which takes the
// FIRST colon only — `local:ornith-fleet:9b` is provider `local`, model
// `ornith-fleet:9b`, and a split on the last colon would ask for a