feat(llm): the fallback chain's last link runs on our own hardware
`local:ornith-fleet:9b` joins opus -> haiku -> glm as the final link. Every entry above it depends on somebody else's account staying funded and unthrottled; this one depends on a GPU in the next room. It is last because it is the weakest model, and present because a chain whose every link is external is not a fallback chain, it is one outage in a trench coat. Three small changes make it work: - `build_provider_registry` accepts a provider with an empty `api_key_env`. A model on our own hardware has nothing to authenticate to, and the old behaviour SKIPPED a keyless provider — leaving the chain quietly one link shorter than it reads, which is the failure mode this whole area keeps producing. - `provider_family` learns `ornith`/`ollama` for BARE names. A qualified `local:` spec was already answered by the split, but a bare one fell through to "unknown", and `cross_provider_judge` would then refuse a judge that is genuinely a different family from the Anthropic implementer. - A test pins that the last link survives `resolve_provider`'s split-on-FIRST- colon: `local:ornith-fleet:9b` is provider `local`, model `ornith-fleet:9b`. Splitting on the last colon would ask for a provider named `local:ornith-fleet`, and the symptom would be a silent fall back to the default provider. Infra: Ollama on tank and architect now binds 0.0.0.0 so the gateway (which has no GPU) can reach it. `tailscale serve` cannot — Ollama rejects a non-local Host header as a DNS-rebinding guard and OLLAMA_ORIGINS is CORS-only, so it 403s. 0.0.0.0 still includes loopback, so the microVM vsock pipe is unaffected; verified on both nodes. This is an explicit trade: Ollama has no auth and its API can pull and delete models, so it is now reachable from the LAN as well as the tailnet. The drop-in carries the ufw one-liner to close the LAN side. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
774f17d194
commit
b18e62041b
@@ -132,9 +132,18 @@ fn is_transient(e: &cm_llm::LlmError) -> bool {
|
||||
/// default chain steps down within the subscription first, then leaves Anthropic
|
||||
/// entirely rather than making a capped window mean "the planner is gone".
|
||||
///
|
||||
/// The last link runs on our OWN hardware. Every other entry — and every other
|
||||
/// link above it — depends on somebody else's account staying funded and
|
||||
/// unthrottled; `local:` depends on a GPU in the next room. It is last because
|
||||
/// it is the weakest model, and present because a chain whose every link is
|
||||
/// external is not a fallback chain, it is one outage in a trench coat.
|
||||
///
|
||||
/// Note the model half contains a colon (`ornith-fleet:9b`), which is why
|
||||
/// `resolve_provider` splits on the FIRST one only.
|
||||
///
|
||||
/// Override with `CLAWMATES_MODEL_FALLBACK` (comma-separated). An empty value
|
||||
/// disables fallback and restores plain "503 and wait".
|
||||
const DEFAULT_FALLBACK: &str = "claude-haiku-4-5-20251001,glm:glm-4.7";
|
||||
const DEFAULT_FALLBACK: &str = "claude-haiku-4-5-20251001,glm:glm-4.7,local:ornith-fleet:9b";
|
||||
|
||||
/// The chain to walk after `requested`, with `requested` itself removed so a
|
||||
/// capped model is never retried as its own fallback.
|
||||
@@ -396,7 +405,22 @@ mod tests {
|
||||
fn the_chain_excludes_the_model_that_just_failed() {
|
||||
// No env override in scope: this asserts the SHIPPED default.
|
||||
let chain = fallback_chain("claude-opus-4-8");
|
||||
assert_eq!(chain, vec!["claude-haiku-4-5-20251001", "glm:glm-4.7"]);
|
||||
assert_eq!(
|
||||
chain,
|
||||
vec![
|
||||
"claude-haiku-4-5-20251001",
|
||||
"glm:glm-4.7",
|
||||
"local:ornith-fleet:9b"
|
||||
]
|
||||
);
|
||||
// The last link must survive `resolve_provider`'s split, which takes the
|
||||
// FIRST colon only — `local:ornith-fleet:9b` is provider `local`, model
|
||||
// `ornith-fleet:9b`, and a split on the last colon would ask for a
|
||||
// provider named `local:ornith-fleet`.
|
||||
let last = chain.last().unwrap();
|
||||
let (provider, model) = last.split_once(':').expect("a provider-qualified spec");
|
||||
assert_eq!(provider, "local");
|
||||
assert_eq!(model, "ornith-fleet:9b");
|
||||
assert!(!fallback_chain("claude-haiku-4-5-20251001")
|
||||
.iter()
|
||||
.any(|m| m == "claude-haiku-4-5-20251001"));
|
||||
|
||||
Reference in New Issue
Block a user