feat(llm): six-link fallback chain, and a preflight that proves it

opus -> sonnet -> haiku -> kimi -> glm -> local. The order is capability first,
then independence: three Anthropic tiers on one account (a throttle usually hits
a tier, so stepping down often clears it), then two separately funded accounts
(now an outage, not just a throttle, is survivable), then our own GPU (nothing
left to be down). Every id was probed on this deployment and answered 200.

The preflight is the more important half. Configured is not working, and this
chain has a specific way of lying: `resolve_provider` falls back to the DEFAULT
provider when it does not recognise a provider name, so a typo in `kimi:` does
not error — it quietly runs on Anthropic, and a chain that reads as three
accounts is really one. A reachability-only probe calls that link green.

So `preflight` checks resolution and reachability separately, eight tokens per
link through the REAL call path, and reports four states. `Throttled` is
deliberately not a failure: a 429 means the spec resolved, the credential
authenticated, and there was no capacity this second — the exact condition the
chain exists to route around, and painting it red would train an operator to
ignore red. `Unregistered` and `Broken` are failures, and they get different
words because they need different fixes.

It runs at boot alongside validator_preflight and runtime_preflight, spawned so
it cannot delay startup. A chain is the one piece of infrastructure nobody looks
at until the day it has to work, so it is now checked on the days it does not.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-09 14:38:55 -07:00
co-authored by Claude Opus 5
parent 5afcf63324
commit c3ad5672fc
2 changed files with 153 additions and 7 deletions
+4
View File
@@ -376,6 +376,10 @@ async fn run() -> Result<(), String> {
// done_when phase unmeetable, and without this the first symptom is a
// mission failing after its VMs have already run.
cm_api::validator_preflight::report_at_boot(runtime.clone());
// Every link of the model fallback chain, probed through the real call path.
// A chain is the one piece of infrastructure nobody looks at until the day it
// has to work, so it is checked on the days it does not.
cm_api::subscription::report_at_boot(runtime.clone());
cm_api::phase_runner::spawn(pool.clone(), runtime.clone(), node_hub.clone());
// Per-mission runtime container sweeper (C3): tears down mission
// runtime containers 30 min after the mission reaches a terminal