apress/deploy/uno-q/LOCAL-FALLBACK.md

# Local fallback — what we run, and why not LiteRT

How an APESS Uno Q node answers when the cloud is slow, rate-limited, or gone.
Supersedes the *"LiteRT-LM on UNO Q 4GB — Technical Spike Plan v1.0"* (July 2026),
which was never executed. See **Why not LiteRT** for the decision, and
**Lessons carried forward** for the design constraints it got wrong.

---

## 1. What we actually run

**Cloud-first, on-board Qwen as fallback.** The local model is a safety net for a
guaranteed-offline condition — not the primary reasoner.

### Agent → provider routing

From [`config.template.toml`](./config.template.toml):

| alias | provider | behaviour |
|---|---|---|
| `default` | `custom.claude` | cloud first, `fallback = ["llamacpp.local"]` — the workshop default |
| `cloud` | `custom.cloud` | cloud only, no fallback |
| `local` | `llamacpp.local` | on-board Qwen only (fully offline) |
| `chaos` | `custom.dead` | points at a dead `:9099` so it *always* fails over — the outage demo |

`chaos` exists to make resilience demonstrable on command: nothing listens on
`127.0.0.1:9099`, so any prompt routed there is guaranteed to fall through to the
local model. It is the only honest way to show failover without unplugging WiFi.

The `fallback = ["llamacpp.local"]` array on `custom.claude` and `custom.dead` is the
**only** fallback configured on the board. There is no multi-cloud chain locally.

### Local serving

From [`systemd/zeroclaw-llama.service`](./systemd/zeroclaw-llama.service):

```
llama-server -m /home/arduino/models/qwen.gguf \
  --host 127.0.0.1 --port 8083 -np 1 -c 8192 --jinja --mlock
```

- **Persistent server, loopback-only.** The model is loaded once and stays resident.
  It is never spawned per request.
- **`-np 1` is mandatory** — a larger slot count makes the tiny model 500 on context.
- **`--mlock`** pins the weights so they aren't swapped out under memory pressure.
- Health at `:8083/health`. The ZeroClaw daemon is on `:8080` and is ordered
  `After=zeroclaw-llama.service`.

> ### Config drift — repo vs. dev board (2026-07-19)
>
> The dev board (serial 65301572) had been hand-edited away from the repo and never
> propagated back. Benchmarking settled it: **the repo was right.**
>
> | | Repo before | Board before | Both now |
> |---|---|---|---|
> | Model | `qwen.gguf` | `qwen-coder.gguf` (1.1 GB) | ✅ `qwen.gguf` (409 MB) |
> | Context | `-c 16384` | `-c 8192` | ✅ `-c 8192` |
>
> **Model — resolved in the repo's favour.** The board ran Qwen2.5-**Coder**-1.5B, which is
> both larger and specialized for the wrong task. Swapping to Qwen2.5-0.5B-Instruct made tool
> calls **~6× faster** (24 s → 3.8 s warm) at **6/6 correctness**, and freed ~700 MB.
> A pre-swap backup sits at `~/zeroclaw-supervisor.sh.bak-preswap` on the board.
>
> **Context — resolved in the board's favour: standardised on `-c 8192`** across
> `zeroclaw-llama.service`, `zeroclaw-supervisor.sh`, and `provision-uno-q.sh`. At ~17 tok/s
> prefill, filling even an 8k window costs ~8 minutes, so a 16k window was capacity that
> could never be afforded and doubled KV footprint for nothing.
>
> **Gotcha for whoever changes this:** the supervisor reads `MODEL` into a shell variable
> at launch. Editing the file does **not** affect the running process — you must restart
> the supervisor itself, not just llama-server, or it will relaunch the old model.

### Keeping it alive

[`zeroclaw-supervisor.sh`](./zeroclaw-supervisor.sh) is the no-root watchdog for boards
where systemd isn't usable. Two properties matter:

- It polls **`/health` endpoints, not `pgrep`** — a wedged process passes `pgrep` but
  fails a health check. Process liveness is not service liveness.
- It respects a **warmup grace** (`LLAMA_WARMUP=300`); reaping mid-load produces an
  infinite restart loop that never converges. The 3–5 minute cold-load figure was measured
  against the 1.1 GB coder model — the 409 MB `qwen.gguf` we standardised on came up
  **healthy in ~5 s** from a cold boot (2026-07-20). The 300 s grace is now generous rather
  than necessary, which is harmless.

Children start under `setsid` so they survive the launching shell closing — plain
`nohup … &` inside an `adb shell` does **not** give you that, which is why services
died on a previous USB disconnect.

### Workshop-proxy variant

[`../workshop-llm/zeroclaw-board.toml`](../workshop-llm/zeroclaw-board.toml) points the
`cloud` agent at a LiteLLM proxy on `:4000`. The board asks for one model name
(`workshop`) and never sees the routing: the proxy pools Kimi + GLM and falls back to
Groq → Gemini. **Cloud-side fallback lives in `litellm-config.yaml`, not on the board.**

### Secrets

Provider config is **file-based**; the only `ZEROCLAW_`-prefixed variable in the repo is
`ZEROCLAW_BIN` (a binary path). The Claude Max setup-token is **env-only at runtime and
must never be written to disk or committed** — not into `config.toml`, not into a unit
file, not into a provisioning script. Same for attendee virtual keys and paired tokens.

---

## 2. Why not LiteRT

The spike proposed LiteRT-LM + FunctionGemma 270M, invoked by shelling out to the
`litert-lm` CLI per classification. We did not pursue it:

- **Subprocess-per-call reloads the model every request.** On a quad-A53 that cost
  dominates the entire latency budget. We already had a persistent server working.
- **Fleet provisioning burden.** It required an export pipeline (`litert-torch`, run
  off-board) plus a per-board `huggingface-cli login` with a personal token.
- **The strategy inverted.** The spike framed the local model as a primary classifier.
  The shipped decision is cloud-default with local reserved for guaranteed-offline —
  so the bar the spike was measuring against stopped being the bar.
- **Its premise was pre-pivot.** Every prompt and pass gate assumed
  `nominal`/`anomalous`/`critical` structural-health classification. The workshop now
  teaches domain-driven design (Domain → Skills → Policies → Harness → Loops) over an
  open, team-chosen domain, so a fixed three-label classifier is not the target.

None of this says LiteRT is a bad runtime. It says it solved a problem we'd already
solved, in a shape that cost more to operate.

---

## 3. Measured baseline

**Measured 2026-07-19** on board 65301572 at `-c 8192`, warm, 4 threads
(NEON + ARM_FMA + OPENMP + REPACK, `n_batch=2048`). Figures come from llama-server's own
`timings` block, not wall clock. Prefix caching was **deliberately defeated** with a unique
leading nonce, so these are honest uncached numbers — the cached case is far better and is
what real workloads see (see below).

Two models compared: **`qwen.gguf`** = Qwen2.5-0.5B-Instruct (630 M params, 24 layers,
n_embd 896) and **`qwen-coder.gguf`** = Qwen2.5-Coder-1.5B-Instruct (1.78 B, 28 layers,
n_embd 1536).

| prompt tok | 0.5B prefill | 1.5B-coder prefill | 0.5B decode | coder decode |
|---|---|---|---|---|
| ~260 | **19.6 tok/s** | 5.30 | 10.9 | 3.5 |
| ~484 | **19.3 tok/s** | 5.28 | 10.0 | 3.3 |
| ~933 | **18.5 tok/s** | 5.14 | 9.4 | 3.0 |
| ~1833 | **17.1 tok/s** | 4.94 | 6.3 | 2.33 |

- **Prefill is linear** in both — cost is simply proportional to context loaded.
- **The 0.5B is ~3.6× faster at prefill, ~3× at decode.** That slightly exceeds the 2.8×
  parameter ratio because cost tracks compute per layer (layers × n_embd² predicts ~3.4×),
  not parameter count.
- **Decode degrades as KV grows** in both (0.5B: 10.9 → 6.3 tok/s). Output cost is not
  constant; it worsens with how much context you loaded.
- Full-window extrapolation (0.5B): 4k ctx ≈ 4 min, a full 8k ≈ 8 min.
- RAM with the 0.5B resident: ~3.0 GB of 3.67 GB available. The coder model costs ~700 MB more.

### Tool calling — the offline task class that matters

Warm, prefix-cached function calling on the **0.5B**, stable system+tools preamble,
3 tools (`record_reading`, `update_document`, `set_led`), 6 requests
(reproduce: `bench-tools.py`):

| | result |
|---|---|
| Correct tool + valid JSON args | **6 / 6** |
| First call (cold preamble, 397 tok) | 24.3 s |
| Warm calls | **avg 3.8 s** (3.0–4.5 s) |
| `prompt_n` warm / `cache_n` | 14 / 377 |

`prompt_n` collapsing from 397 → ~14 once the cache fills is the whole story: the preamble
prefills once, later calls pay only for the user's sentence. Calls came back as **native
`tool_calls`**, not scraped from prose.

**Caveats:** 6 unambiguous single-tool cases is a floor check, not a ceiling — it says nothing
about ambiguous or multi-step requests. And this exercised llama-server's *native* tool path,
whereas the board config sets `native_tools = false` and uses ZeroClaw's text parser.
**The path ZeroClaw actually takes is still unvalidated.**

### Prefix caching is the dominant lever

The table above is the *uncached* worst case. In real tool-calling workloads the system
prompt and tool schemas are byte-identical on every call, so llama.cpp prefills them once
and later calls pay only for the delta. A ~600-token preamble + ~60-token request + ~50-token
tool call is **~2 minutes uncached but ~25–30 s warm**.

Three ways to destroy that advantage — all easy to do by accident:

- **Varying the preamble.** A timestamp, session id, or rotating skill list injected into the
  system prompt invalidates the cache every call.
- **Unbounded `max_tokens`.** At ~3 tok/s, a 500-token answer is 2.5 minutes of pure decode.
- **`max_tool_iterations = 6`.** Each iteration is a full model call — ~3 min per task.
  Acceptable for a background loop, not for anything interactive.

### The offline capability boundary (measured 2026-07-20)

Speed is not the only limit — there is a hard capability cliff. Same board, same lean
profile, same on-board 0.5B:

| task | result |
|---|---|
| Call a simple tool (`i2cdetect`, 708-token prompt) | **works — 20 s**, tool call fired, real answer |
| Write + compile + flash a sketch (1,997-token prompt) | **never completed** — 450 s, two identical requests, no tool call ever emitted |

So offline the node can **sense and decide, but it cannot author new code**. Code
generation needs the cloud model. This is the concrete degradation boundary to state in
ADD Layer 4: what still works with no network is the sensing and decision loop over
*already-flashed* firmware — not writing new firmware.

Two related prompt-shape findings from the same session:

- **Imperative, not interrogative.** `"List the I2C devices on the bus."` fires a tool call
  in 20 s; `"What sensors can you find on the I2C bus?"` produced **no tool call at all**
  in 200 s. Same agent, same 708-token prompt — phrasing was the only variable.
- **It does not hallucinate hardware.** Asked to list I2C devices on a board with an empty
  bus, it ran the tool and reported the bus numbers rather than inventing a sensor.

Note also that `BuildFlash` in the SPA routes to the **`cloud`** agent, not `local` — so the
student build/flash exercise does not depend on the boundary above.

### What this means for offline work

On the **0.5B** (measured or extrapolated from the curve above):

| task | cost | verdict |
|---|---|---|
| Function/tool call (warm, cached preamble) | **3.8 s** *(measured)* | viable, near-interactive |
| Process a result → decide → emit | ~4–6 s | viable |
| Update / retag / summarize one short doc | ~25–35 s | viable |
| Q&A retrieving several notes (1.5k ctx) | ~1.5 min | marginal, usable offline |
| Code generation | — | not viable (quality, not speed) |

An 8-hour overnight loop processes roughly **500k prefill tokens** on the 0.5B — several
hundred to a thousand short notes.

**The offline task class Omar specified — function calls, processing a specific result,
updating documents, nothing needing complex reasoning — is comfortably served by the 0.5B.**
Sub-4-second tool calls are fast enough that this is not merely a background queue worker.

### ⚠️ The harness, not the model, is the bottleneck

Raw model speed is *not* what makes or breaks offline. **Prompt size is.** Measured
2026-07-19 driving the real ZeroClaw agent loop (`native_tools = false`, the text parser):

| | default profile | lean `offline` profile |
|---|---|---|
| Prompt tokens | **4,718** | **706** (−85%) |
| Prefix-cache match (`sim_best`) | 0.435 | **0.966** |
| Tokens re-prefilled when warm | ~2,600 | **24** |
| Cold turn | cancelled mid-prefill | **~29 s** |
| Warm turn | never completed | **11 s** |
| Tool actually fired | no | **yes** |

The 11 s warm figure is a *complete agentic turn* — two LLM calls plus tool execution —
ending in a real answer (`"The I2C devices on the bus are: i2c-0, i2c-1, and i2c-2."`).

**The text parser was never the problem.** With the default profile the agent sent a
4,718-token prompt that took ~4.6 min to prefill at 17 tok/s, so the client cancelled before
the model could respond. Disabling skills injection alone made it *worse* (10,901 tokens —
past the 8k window, hard failure) because 84 accumulated memories in `brain.db` flooded the
prompt. Flipping `native_tools = true` changed nothing (still 4,718): tool schemas were never
the bulk.

### The lean offline profile

Everything that controls prompt size lives on the **runtime profile**, not the agent
(`effective_skills_prompt_mode`, `schema.rs:4192`, lets a runtime profile override the global
`[skills] prompt_injection_mode`):

```toml
[runtime_profiles.offline]
agentic = true
max_tool_iterations = 3
compact_context = true              # 6000 chars / 2 RAG chunks; also drops Channel Capabilities
prompt_injection_mode = "compact"
max_system_prompt_chars = 2000
max_context_tokens = 3000
max_history_messages = 2
memory_recall_limit = 1
parallel_tools = false

[risk_profiles.sense_only]          # narrow tools ⇒ also drops the hardware block +
level = "supervised"                # Uno-Q flash imperative from the system prompt
allowed_tools = ["i2cdetect"]
auto_approve = ["i2cdetect"]

[agents.sense]
model_provider = "llamacpp.local"
risk_profile = "sense_only"
runtime_profile = "offline"
skill_bundles = []                  # the 12 bundled skills were most of the 4,718
mcp_bundles = []
```

**Two zero-value footguns:**
- `memory_recall_limit = 0` means **unlimited** (`usize::MAX`), not disabled. Use `1`.
- `max_actions_per_hour = 0` is a **hard zero** that blocks everything; unlimited is `u32::MAX`.

### Per-step prompts for multi-step operations

Cron `JobType::Agent` carries its own `prompt`, `allowed_tools`, and `uses_memory` per job
(`crates/zeroclaw-runtime/src/cron/types.rs:32,143`), bound to an agent via
`AliasedAgentConfig.cron_jobs`. So a multi-step offline operation is a sequence of jobs, each
with the prompt and tool surface for its own step — and `uses_memory = false` skips memory
injection entirely for steps that don't need it. Agent-to-agent `delegates` is the other path.

---

## 4. Lessons carried forward

Design constraints the spike document got wrong. Worth keeping even though the spike
itself is dead.

- **A fail-safe must never be "nominal".** The spike degraded both error paths to a
  synthetic `nominal` response. In any monitoring context that is a silent all-clear
  manufactured by an outage. Degrade to `unknown` / `escalate` — never to "fine".
- **Fallback must trigger on more than "network unavailable".** Timeouts, 429s, DNS
  failures, and auth errors are the realistic degradations — a flaky workshop LAN far
  more often than a cleanly absent one. Matching only a `NetworkUnavailable` variant
  routes the common cases straight past the local model.
- **Health-check endpoints, not processes.** A wedged process passes `pgrep`.
- **Persistent server, never per-call subprocess.** Model load time dominates.
- **Check exit status and stderr on any subprocess.** Otherwise a crashed child
  surfaces as an unrelated JSON parse error, which is miserable to debug on a board.
- **Don't gate on accuracy against undefined thresholds.** The spike asked a model to
  choose among three labels without ever defining their boundaries, then scored it
  against the author's own intuition. That measures prompt underspecification, not
  model capability. Define the thresholds or don't call it accuracy.

---

## See also

- [`README.md`](./README.md) — provisioning, the three modalities, open → locked lifecycle
- [`config.template.toml`](./config.template.toml) — providers, agents, risk profile
- [`zeroclaw-supervisor.sh`](./zeroclaw-supervisor.sh) · [`recover-uno-q.sh`](./recover-uno-q.sh)