feat(llm): six-link fallback chain, and a preflight that proves it
opus -> sonnet -> haiku -> kimi -> glm -> local. The order is capability first, then independence: three Anthropic tiers on one account (a throttle usually hits a tier, so stepping down often clears it), then two separately funded accounts (now an outage, not just a throttle, is survivable), then our own GPU (nothing left to be down). Every id was probed on this deployment and answered 200. The preflight is the more important half. Configured is not working, and this chain has a specific way of lying: `resolve_provider` falls back to the DEFAULT provider when it does not recognise a provider name, so a typo in `kimi:` does not error — it quietly runs on Anthropic, and a chain that reads as three accounts is really one. A reachability-only probe calls that link green. So `preflight` checks resolution and reachability separately, eight tokens per link through the REAL call path, and reports four states. `Throttled` is deliberately not a failure: a 429 means the spec resolved, the credential authenticated, and there was no capacity this second — the exact condition the chain exists to route around, and painting it red would train an operator to ignore red. `Unregistered` and `Broken` are failures, and they get different words because they need different fixes. It runs at boot alongside validator_preflight and runtime_preflight, spawned so it cannot delay startup. A chain is the one piece of infrastructure nobody looks at until the day it has to work, so it is now checked on the days it does not. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
5afcf63324
commit
c3ad5672fc
@@ -376,6 +376,10 @@ async fn run() -> Result<(), String> {
|
||||
// done_when phase unmeetable, and without this the first symptom is a
|
||||
// mission failing after its VMs have already run.
|
||||
cm_api::validator_preflight::report_at_boot(runtime.clone());
|
||||
// Every link of the model fallback chain, probed through the real call path.
|
||||
// A chain is the one piece of infrastructure nobody looks at until the day it
|
||||
// has to work, so it is checked on the days it does not.
|
||||
cm_api::subscription::report_at_boot(runtime.clone());
|
||||
cm_api::phase_runner::spawn(pool.clone(), runtime.clone(), node_hub.clone());
|
||||
// Per-mission runtime container sweeper (C3): tears down mission
|
||||
// runtime containers 30 min after the mission reaches a terminal
|
||||
|
||||
Reference in New Issue
Block a user