feat(llm): the fallback chain's last link runs on our own hardware

`local:ornith-fleet:9b` joins opus -> haiku -> glm as the final link. Every
entry above it depends on somebody else's account staying funded and
unthrottled; this one depends on a GPU in the next room. It is last because it
is the weakest model, and present because a chain whose every link is external
is not a fallback chain, it is one outage in a trench coat.

Three small changes make it work:

- `build_provider_registry` accepts a provider with an empty `api_key_env`.
  A model on our own hardware has nothing to authenticate to, and the old
  behaviour SKIPPED a keyless provider — leaving the chain quietly one link
  shorter than it reads, which is the failure mode this whole area keeps
  producing.
- `provider_family` learns `ornith`/`ollama` for BARE names. A qualified
  `local:` spec was already answered by the split, but a bare one fell through
  to "unknown", and `cross_provider_judge` would then refuse a judge that is
  genuinely a different family from the Anthropic implementer.
- A test pins that the last link survives `resolve_provider`'s split-on-FIRST-
  colon: `local:ornith-fleet:9b` is provider `local`, model `ornith-fleet:9b`.
  Splitting on the last colon would ask for a provider named
  `local:ornith-fleet`, and the symptom would be a silent fall back to the
  default provider.

Infra: Ollama on tank and architect now binds 0.0.0.0 so the gateway (which has
no GPU) can reach it. `tailscale serve` cannot — Ollama rejects a non-local Host
header as a DNS-rebinding guard and OLLAMA_ORIGINS is CORS-only, so it 403s.
0.0.0.0 still includes loopback, so the microVM vsock pipe is unaffected;
verified on both nodes. This is an explicit trade: Ollama has no auth and its
API can pull and delete models, so it is now reachable from the LAN as well as
the tailnet. The drop-in carries the ufw one-liner to close the LAN side.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-09 14:13:05 -07:00
co-authored by Claude Opus 5
parent 774f17d194
commit b18e62041b
3 changed files with 49 additions and 4 deletions
+26 -2
View File
@@ -132,9 +132,18 @@ fn is_transient(e: &cm_llm::LlmError) -> bool {
/// default chain steps down within the subscription first, then leaves Anthropic
/// entirely rather than making a capped window mean "the planner is gone".
///
/// The last link runs on our OWN hardware. Every other entry — and every other
/// link above it — depends on somebody else's account staying funded and
/// unthrottled; `local:` depends on a GPU in the next room. It is last because
/// it is the weakest model, and present because a chain whose every link is
/// external is not a fallback chain, it is one outage in a trench coat.
///
/// Note the model half contains a colon (`ornith-fleet:9b`), which is why
/// `resolve_provider` splits on the FIRST one only.
///
/// Override with `CLAWMATES_MODEL_FALLBACK` (comma-separated). An empty value
/// disables fallback and restores plain "503 and wait".
const DEFAULT_FALLBACK: &str = "claude-haiku-4-5-20251001,glm:glm-4.7";
const DEFAULT_FALLBACK: &str = "claude-haiku-4-5-20251001,glm:glm-4.7,local:ornith-fleet:9b";
/// The chain to walk after `requested`, with `requested` itself removed so a
/// capped model is never retried as its own fallback.
@@ -396,7 +405,22 @@ mod tests {
fn the_chain_excludes_the_model_that_just_failed() {
// No env override in scope: this asserts the SHIPPED default.
let chain = fallback_chain("claude-opus-4-8");
assert_eq!(chain, vec!["claude-haiku-4-5-20251001", "glm:glm-4.7"]);
assert_eq!(
chain,
vec![
"claude-haiku-4-5-20251001",
"glm:glm-4.7",
"local:ornith-fleet:9b"
]
);
// The last link must survive `resolve_provider`'s split, which takes the
// FIRST colon only — `local:ornith-fleet:9b` is provider `local`, model
// `ornith-fleet:9b`, and a split on the last colon would ask for a
// provider named `local:ornith-fleet`.
let last = chain.last().unwrap();
let (provider, model) = last.split_once(':').expect("a provider-qualified spec");
assert_eq!(provider, "local");
assert_eq!(model, "ornith-fleet:9b");
assert!(!fallback_chain("claude-haiku-4-5-20251001")
.iter()
.any(|m| m == "claude-haiku-4-5-20251001"));