# Local fallback — what we run, and why not LiteRT How an APESS Uno Q node answers when the cloud is slow, rate-limited, or gone. Supersedes the *"LiteRT-LM on UNO Q 4GB — Technical Spike Plan v1.0"* (July 2026), which was never executed. See **Why not LiteRT** for the decision, and **Lessons carried forward** for the design constraints it got wrong. --- ## 1. What we actually run **Cloud-first, on-board Qwen as fallback.** The local model is a safety net for a guaranteed-offline condition — not the primary reasoner. ### Agent → provider routing From [`config.template.toml`](./config.template.toml): | alias | provider | behaviour | |---|---|---| | `default` | `custom.claude` | cloud first, `fallback = ["llamacpp.local"]` — the workshop default | | `cloud` | `custom.cloud` | cloud only, no fallback | | `local` | `llamacpp.local` | on-board Qwen only (fully offline) | | `chaos` | `custom.dead` | points at a dead `:9099` so it *always* fails over — the outage demo | `chaos` exists to make resilience demonstrable on command: nothing listens on `127.0.0.1:9099`, so any prompt routed there is guaranteed to fall through to the local model. It is the only honest way to show failover without unplugging WiFi. The `fallback = ["llamacpp.local"]` array on `custom.claude` and `custom.dead` is the **only** fallback configured on the board. There is no multi-cloud chain locally. ### Local serving From [`systemd/zeroclaw-llama.service`](./systemd/zeroclaw-llama.service): ``` llama-server -m /home/arduino/models/qwen.gguf \ --host 127.0.0.1 --port 8083 -np 1 -c 8192 --jinja --mlock ``` - **Persistent server, loopback-only.** The model is loaded once and stays resident. It is never spawned per request. - **`-np 1` is mandatory** — a larger slot count makes the tiny model 500 on context. - **`--mlock`** pins the weights so they aren't swapped out under memory pressure. - Health at `:8083/health`. The ZeroClaw daemon is on `:8080` and is ordered `After=zeroclaw-llama.service`. > ### Config drift — repo vs. dev board (2026-07-19) > > The dev board (serial 65301572) had been hand-edited away from the repo and never > propagated back. Benchmarking settled it: **the repo was right.** > > | | Repo before | Board before | Both now | > |---|---|---|---| > | Model | `qwen.gguf` | `qwen-coder.gguf` (1.1 GB) | ✅ `qwen.gguf` (409 MB) | > | Context | `-c 16384` | `-c 8192` | ✅ `-c 8192` | > > **Model — resolved in the repo's favour.** The board ran Qwen2.5-**Coder**-1.5B, which is > both larger and specialized for the wrong task. Swapping to Qwen2.5-0.5B-Instruct made tool > calls **~6× faster** (24 s → 3.8 s warm) at **6/6 correctness**, and freed ~700 MB. > A pre-swap backup sits at `~/zeroclaw-supervisor.sh.bak-preswap` on the board. > > **Context — resolved in the board's favour: standardised on `-c 8192`** across > `zeroclaw-llama.service`, `zeroclaw-supervisor.sh`, and `provision-uno-q.sh`. At ~17 tok/s > prefill, filling even an 8k window costs ~8 minutes, so a 16k window was capacity that > could never be afforded and doubled KV footprint for nothing. > > **Gotcha for whoever changes this:** the supervisor reads `MODEL` into a shell variable > at launch. Editing the file does **not** affect the running process — you must restart > the supervisor itself, not just llama-server, or it will relaunch the old model. ### Keeping it alive [`zeroclaw-supervisor.sh`](./zeroclaw-supervisor.sh) is the no-root watchdog for boards where systemd isn't usable. Two properties matter: - It polls **`/health` endpoints, not `pgrep`** — a wedged process passes `pgrep` but fails a health check. Process liveness is not service liveness. - It respects a **warmup grace** (`LLAMA_WARMUP=300`); reaping mid-load produces an infinite restart loop that never converges. The 3–5 minute cold-load figure was measured against the 1.1 GB coder model — the 409 MB `qwen.gguf` we standardised on came up **healthy in ~5 s** from a cold boot (2026-07-20). The 300 s grace is now generous rather than necessary, which is harmless. Children start under `setsid` so they survive the launching shell closing — plain `nohup … &` inside an `adb shell` does **not** give you that, which is why services died on a previous USB disconnect. ### Workshop-proxy variant [`../workshop-llm/zeroclaw-board.toml`](../workshop-llm/zeroclaw-board.toml) points the `cloud` agent at a LiteLLM proxy on `:4000`. The board asks for one model name (`workshop`) and never sees the routing: the proxy pools Kimi + GLM and falls back to Groq → Gemini. **Cloud-side fallback lives in `litellm-config.yaml`, not on the board.** ### Secrets Provider config is **file-based**; the only `ZEROCLAW_`-prefixed variable in the repo is `ZEROCLAW_BIN` (a binary path). The Claude Max setup-token is **env-only at runtime and must never be written to disk or committed** — not into `config.toml`, not into a unit file, not into a provisioning script. Same for attendee virtual keys and paired tokens. --- ## 2. Why not LiteRT The spike proposed LiteRT-LM + FunctionGemma 270M, invoked by shelling out to the `litert-lm` CLI per classification. We did not pursue it: - **Subprocess-per-call reloads the model every request.** On a quad-A53 that cost dominates the entire latency budget. We already had a persistent server working. - **Fleet provisioning burden.** It required an export pipeline (`litert-torch`, run off-board) plus a per-board `huggingface-cli login` with a personal token. - **The strategy inverted.** The spike framed the local model as a primary classifier. The shipped decision is cloud-default with local reserved for guaranteed-offline — so the bar the spike was measuring against stopped being the bar. - **Its premise was pre-pivot.** Every prompt and pass gate assumed `nominal`/`anomalous`/`critical` structural-health classification. The workshop now teaches domain-driven design (Domain → Skills → Policies → Harness → Loops) over an open, team-chosen domain, so a fixed three-label classifier is not the target. None of this says LiteRT is a bad runtime. It says it solved a problem we'd already solved, in a shape that cost more to operate. --- ## 3. Measured baseline **Measured 2026-07-19** on board 65301572 at `-c 8192`, warm, 4 threads (NEON + ARM_FMA + OPENMP + REPACK, `n_batch=2048`). Figures come from llama-server's own `timings` block, not wall clock. Prefix caching was **deliberately defeated** with a unique leading nonce, so these are honest uncached numbers — the cached case is far better and is what real workloads see (see below). Two models compared: **`qwen.gguf`** = Qwen2.5-0.5B-Instruct (630 M params, 24 layers, n_embd 896) and **`qwen-coder.gguf`** = Qwen2.5-Coder-1.5B-Instruct (1.78 B, 28 layers, n_embd 1536). | prompt tok | 0.5B prefill | 1.5B-coder prefill | 0.5B decode | coder decode | |---|---|---|---|---| | ~260 | **19.6 tok/s** | 5.30 | 10.9 | 3.5 | | ~484 | **19.3 tok/s** | 5.28 | 10.0 | 3.3 | | ~933 | **18.5 tok/s** | 5.14 | 9.4 | 3.0 | | ~1833 | **17.1 tok/s** | 4.94 | 6.3 | 2.33 | - **Prefill is linear** in both — cost is simply proportional to context loaded. - **The 0.5B is ~3.6× faster at prefill, ~3× at decode.** That slightly exceeds the 2.8× parameter ratio because cost tracks compute per layer (layers × n_embd² predicts ~3.4×), not parameter count. - **Decode degrades as KV grows** in both (0.5B: 10.9 → 6.3 tok/s). Output cost is not constant; it worsens with how much context you loaded. - Full-window extrapolation (0.5B): 4k ctx ≈ 4 min, a full 8k ≈ 8 min. - RAM with the 0.5B resident: ~3.0 GB of 3.67 GB available. The coder model costs ~700 MB more. ### Tool calling — the offline task class that matters Warm, prefix-cached function calling on the **0.5B**, stable system+tools preamble, 3 tools (`record_reading`, `update_document`, `set_led`), 6 requests (reproduce: `bench-tools.py`): | | result | |---|---| | Correct tool + valid JSON args | **6 / 6** | | First call (cold preamble, 397 tok) | 24.3 s | | Warm calls | **avg 3.8 s** (3.0–4.5 s) | | `prompt_n` warm / `cache_n` | 14 / 377 | `prompt_n` collapsing from 397 → ~14 once the cache fills is the whole story: the preamble prefills once, later calls pay only for the user's sentence. Calls came back as **native `tool_calls`**, not scraped from prose. **Caveats:** 6 unambiguous single-tool cases is a floor check, not a ceiling — it says nothing about ambiguous or multi-step requests. And this exercised llama-server's *native* tool path, whereas the board config sets `native_tools = false` and uses ZeroClaw's text parser. **The path ZeroClaw actually takes is still unvalidated.** ### Prefix caching is the dominant lever The table above is the *uncached* worst case. In real tool-calling workloads the system prompt and tool schemas are byte-identical on every call, so llama.cpp prefills them once and later calls pay only for the delta. A ~600-token preamble + ~60-token request + ~50-token tool call is **~2 minutes uncached but ~25–30 s warm**. Three ways to destroy that advantage — all easy to do by accident: - **Varying the preamble.** A timestamp, session id, or rotating skill list injected into the system prompt invalidates the cache every call. - **Unbounded `max_tokens`.** At ~3 tok/s, a 500-token answer is 2.5 minutes of pure decode. - **`max_tool_iterations = 6`.** Each iteration is a full model call — ~3 min per task. Acceptable for a background loop, not for anything interactive. ### The offline capability boundary (measured 2026-07-20) Speed is not the only limit — there is a hard capability cliff. Same board, same lean profile, same on-board 0.5B: | task | result | |---|---| | Call a simple tool (`i2cdetect`, 708-token prompt) | **works — 20 s**, tool call fired, real answer | | Write + compile + flash a sketch (1,997-token prompt) | **never completed** — 450 s, two identical requests, no tool call ever emitted | So offline the node can **sense and decide, but it cannot author new code**. Code generation needs the cloud model. This is the concrete degradation boundary to state in ADD Layer 4: what still works with no network is the sensing and decision loop over *already-flashed* firmware — not writing new firmware. Two related prompt-shape findings from the same session: - **Imperative, not interrogative.** `"List the I2C devices on the bus."` fires a tool call in 20 s; `"What sensors can you find on the I2C bus?"` produced **no tool call at all** in 200 s. Same agent, same 708-token prompt — phrasing was the only variable. - **It does not hallucinate hardware.** Asked to list I2C devices on a board with an empty bus, it ran the tool and reported the bus numbers rather than inventing a sensor. Note also that `BuildFlash` in the SPA routes to the **`cloud`** agent, not `local` — so the student build/flash exercise does not depend on the boundary above. ### What this means for offline work On the **0.5B** (measured or extrapolated from the curve above): | task | cost | verdict | |---|---|---| | Function/tool call (warm, cached preamble) | **3.8 s** *(measured)* | viable, near-interactive | | Process a result → decide → emit | ~4–6 s | viable | | Update / retag / summarize one short doc | ~25–35 s | viable | | Q&A retrieving several notes (1.5k ctx) | ~1.5 min | marginal, usable offline | | Code generation | — | not viable (quality, not speed) | An 8-hour overnight loop processes roughly **500k prefill tokens** on the 0.5B — several hundred to a thousand short notes. **The offline task class Omar specified — function calls, processing a specific result, updating documents, nothing needing complex reasoning — is comfortably served by the 0.5B.** Sub-4-second tool calls are fast enough that this is not merely a background queue worker. ### ⚠️ The harness, not the model, is the bottleneck Raw model speed is *not* what makes or breaks offline. **Prompt size is.** Measured 2026-07-19 driving the real ZeroClaw agent loop (`native_tools = false`, the text parser): | | default profile | lean `offline` profile | |---|---|---| | Prompt tokens | **4,718** | **706** (−85%) | | Prefix-cache match (`sim_best`) | 0.435 | **0.966** | | Tokens re-prefilled when warm | ~2,600 | **24** | | Cold turn | cancelled mid-prefill | **~29 s** | | Warm turn | never completed | **11 s** | | Tool actually fired | no | **yes** | The 11 s warm figure is a *complete agentic turn* — two LLM calls plus tool execution — ending in a real answer (`"The I2C devices on the bus are: i2c-0, i2c-1, and i2c-2."`). **The text parser was never the problem.** With the default profile the agent sent a 4,718-token prompt that took ~4.6 min to prefill at 17 tok/s, so the client cancelled before the model could respond. Disabling skills injection alone made it *worse* (10,901 tokens — past the 8k window, hard failure) because 84 accumulated memories in `brain.db` flooded the prompt. Flipping `native_tools = true` changed nothing (still 4,718): tool schemas were never the bulk. ### The lean offline profile Everything that controls prompt size lives on the **runtime profile**, not the agent (`effective_skills_prompt_mode`, `schema.rs:4192`, lets a runtime profile override the global `[skills] prompt_injection_mode`): ```toml [runtime_profiles.offline] agentic = true max_tool_iterations = 3 compact_context = true # 6000 chars / 2 RAG chunks; also drops Channel Capabilities prompt_injection_mode = "compact" max_system_prompt_chars = 2000 max_context_tokens = 3000 max_history_messages = 2 memory_recall_limit = 1 parallel_tools = false [risk_profiles.sense_only] # narrow tools ⇒ also drops the hardware block + level = "supervised" # Uno-Q flash imperative from the system prompt allowed_tools = ["i2cdetect"] auto_approve = ["i2cdetect"] [agents.sense] model_provider = "llamacpp.local" risk_profile = "sense_only" runtime_profile = "offline" skill_bundles = [] # the 12 bundled skills were most of the 4,718 mcp_bundles = [] ``` **Two zero-value footguns:** - `memory_recall_limit = 0` means **unlimited** (`usize::MAX`), not disabled. Use `1`. - `max_actions_per_hour = 0` is a **hard zero** that blocks everything; unlimited is `u32::MAX`. ### Per-step prompts for multi-step operations Cron `JobType::Agent` carries its own `prompt`, `allowed_tools`, and `uses_memory` per job (`crates/zeroclaw-runtime/src/cron/types.rs:32,143`), bound to an agent via `AliasedAgentConfig.cron_jobs`. So a multi-step offline operation is a sequence of jobs, each with the prompt and tool surface for its own step — and `uses_memory = false` skips memory injection entirely for steps that don't need it. Agent-to-agent `delegates` is the other path. --- ## 4. Lessons carried forward Design constraints the spike document got wrong. Worth keeping even though the spike itself is dead. - **A fail-safe must never be "nominal".** The spike degraded both error paths to a synthetic `nominal` response. In any monitoring context that is a silent all-clear manufactured by an outage. Degrade to `unknown` / `escalate` — never to "fine". - **Fallback must trigger on more than "network unavailable".** Timeouts, 429s, DNS failures, and auth errors are the realistic degradations — a flaky workshop LAN far more often than a cleanly absent one. Matching only a `NetworkUnavailable` variant routes the common cases straight past the local model. - **Health-check endpoints, not processes.** A wedged process passes `pgrep`. - **Persistent server, never per-call subprocess.** Model load time dominates. - **Check exit status and stderr on any subprocess.** Otherwise a crashed child surfaces as an unrelated JSON parse error, which is miserable to debug on a board. - **Don't gate on accuracy against undefined thresholds.** The spike asked a model to choose among three labels without ever defining their boundaries, then scored it against the author's own intuition. That measures prompt underspecification, not model capability. Define the thresholds or don't call it accuracy. --- ## See also - [`README.md`](./README.md) — provisioning, the three modalities, open → locked lifecycle - [`config.template.toml`](./config.template.toml) — providers, agents, risk profile - [`zeroclaw-supervisor.sh`](./zeroclaw-supervisor.sh) · [`recover-uno-q.sh`](./recover-uno-q.sh)