apress/deploy/uno-q/LOCAL-FALLBACK.md

Local fallback — what we run, and why not LiteRT

How an APESS Uno Q node answers when the cloud is slow, rate-limited, or gone. Supersedes the "LiteRT-LM on UNO Q 4GB — Technical Spike Plan v1.0" (July 2026), which was never executed. See Why not LiteRT for the decision, and Lessons carried forward for the design constraints it got wrong.

---

1. What we actually run

Cloud-first, on-board Qwen as fallback. The local model is a safety net for a guaranteed-offline condition — not the primary reasoner.

Agent → provider routing

From config.template.toml:

alias	provider	behaviour
default	custom.claude	cloud first, fallback = ["llamacpp.local"] — the workshop default
cloud	custom.cloud	cloud only, no fallback
local	llamacpp.local	on-board Qwen only (fully offline)
chaos	custom.dead	points at a dead :9099 so it always fails over — the outage demo

chaos exists to make resilience demonstrable on command: nothing listens on 127.0.0.1:9099, so any prompt routed there is guaranteed to fall through to the local model. It is the only honest way to show failover without unplugging WiFi.

The fallback = ["llamacpp.local"] array on custom.claude and custom.dead is the only fallback configured on the board. There is no multi-cloud chain locally.

Local serving

From systemd/zeroclaw-llama.service:

llama-server -m /home/arduino/models/qwen.gguf \
  --host 127.0.0.1 --port 8083 -np 1 -c 8192 --jinja --mlock

• Persistent server, loopback-only. The model is loaded once and stays resident. It is never spawned per request.
• -np 1 is mandatory — a larger slot count makes the tiny model 500 on context.
• --mlock pins the weights so they aren't swapped out under memory pressure.
• Health at :8083/health. The ZeroClaw daemon is on :8080 and is ordered After=zeroclaw-llama.service.

Config drift — repo vs. dev board (2026-07-19)

The dev board (serial 65301572) had been hand-edited away from the repo and never propagated back. Benchmarking settled it: the repo was right.

	Repo before	Board before	Both now
Model	qwen.gguf	qwen-coder.gguf (1.1 GB)	✅ qwen.gguf (409 MB)
Context	-c 16384	-c 8192	✅ -c 8192

Model — resolved in the repo's favour. The board ran Qwen2.5-Coder-1.5B, which is both larger and specialized for the wrong task. Swapping to Qwen2.5-0.5B-Instruct made tool calls ~6× faster (24 s → 3.8 s warm) at 6/6 correctness, and freed ~700 MB. A pre-swap backup sits at ~/zeroclaw-supervisor.sh.bak-preswap on the board.

Context — resolved in the board's favour: standardised on -c 8192 across zeroclaw-llama.service, zeroclaw-supervisor.sh, and provision-uno-q.sh. At ~17 tok/s prefill, filling even an 8k window costs ~8 minutes, so a 16k window was capacity that could never be afforded and doubled KV footprint for nothing.

Gotcha for whoever changes this: the supervisor reads MODEL into a shell variable at launch. Editing the file does not affect the running process — you must restart the supervisor itself, not just llama-server, or it will relaunch the old model.

Keeping it alive

zeroclaw-supervisor.sh is the no-root watchdog for boards where systemd isn't usable. Two properties matter:

• It polls /health endpoints, not pgrep — a wedged process passes pgrep but fails a health check. Process liveness is not service liveness.
• It respects a warmup grace (LLAMA_WARMUP=300). A cold GGUF load takes 3–5 minutes; reaping mid-load produces an infinite restart loop that never converges.

Children start under setsid so they survive the launching shell closing — plain nohup … & inside an adb shell does not give you that, which is why services died on a previous USB disconnect.

Workshop-proxy variant

../workshop-llm/zeroclaw-board.toml points the cloud agent at a LiteLLM proxy on :4000. The board asks for one model name (workshop) and never sees the routing: the proxy pools Kimi + GLM and falls back to Groq → Gemini. Cloud-side fallback lives in litellm-config.yaml, not on the board.

Secrets

Provider config is file-based; the only ZEROCLAW_-prefixed variable in the repo is ZEROCLAW_BIN (a binary path). The Claude Max setup-token is env-only at runtime and must never be written to disk or committed — not into config.toml, not into a unit file, not into a provisioning script. Same for attendee virtual keys and paired tokens.

---

2. Why not LiteRT

The spike proposed LiteRT-LM + FunctionGemma 270M, invoked by shelling out to the litert-lm CLI per classification. We did not pursue it:

• Subprocess-per-call reloads the model every request. On a quad-A53 that cost dominates the entire latency budget. We already had a persistent server working.
• Fleet provisioning burden. It required an export pipeline (litert-torch, run off-board) plus a per-board huggingface-cli login with a personal token.
• The strategy inverted. The spike framed the local model as a primary classifier. The shipped decision is cloud-default with local reserved for guaranteed-offline — so the bar the spike was measuring against stopped being the bar.
• Its premise was pre-pivot. Every prompt and pass gate assumed nominal/anomalous/critical structural-health classification. The workshop now teaches domain-driven design (Domain → Skills → Policies → Harness → Loops) over an open, team-chosen domain, so a fixed three-label classifier is not the target.

None of this says LiteRT is a bad runtime. It says it solved a problem we'd already solved, in a shape that cost more to operate.

---

3. Measured baseline

Measured 2026-07-19 on board 65301572 at -c 8192, warm, 4 threads (NEON + ARM_FMA + OPENMP + REPACK, n_batch=2048). Figures come from llama-server's own timings block, not wall clock. Prefix caching was deliberately defeated with a unique leading nonce, so these are honest uncached numbers — the cached case is far better and is what real workloads see (see below).

Two models compared: qwen.gguf = Qwen2.5-0.5B-Instruct (630 M params, 24 layers, n_embd 896) and qwen-coder.gguf = Qwen2.5-Coder-1.5B-Instruct (1.78 B, 28 layers, n_embd 1536).

prompt tok	0.5B prefill	1.5B-coder prefill	0.5B decode	coder decode
~260	19.6 tok/s	5.30	10.9	3.5
~484	19.3 tok/s	5.28	10.0	3.3
~933	18.5 tok/s	5.14	9.4	3.0
~1833	17.1 tok/s	4.94	6.3	2.33

• Prefill is linear in both — cost is simply proportional to context loaded.
• The 0.5B is ~3.6× faster at prefill, ~3× at decode. That slightly exceeds the 2.8× parameter ratio because cost tracks compute per layer (layers × n_embd² predicts ~3.4×), not parameter count.
• Decode degrades as KV grows in both (0.5B: 10.9 → 6.3 tok/s). Output cost is not constant; it worsens with how much context you loaded.
• Full-window extrapolation (0.5B): 4k ctx ≈ 4 min, a full 8k ≈ 8 min.
• RAM with the 0.5B resident: ~3.0 GB of 3.67 GB available. The coder model costs ~700 MB more.

Tool calling — the offline task class that matters

Warm, prefix-cached function calling on the 0.5B, stable system+tools preamble, 3 tools (record_reading, update_document, set_led), 6 requests (reproduce: bench-tools.py):

	result
Correct tool + valid JSON args	6 / 6
First call (cold preamble, 397 tok)	24.3 s
Warm calls	avg 3.8 s (3.0–4.5 s)
prompt_n warm / cache_n	14 / 377

prompt_n collapsing from 397 → ~14 once the cache fills is the whole story: the preamble prefills once, later calls pay only for the user's sentence. Calls came back as native tool_calls, not scraped from prose.

Caveats: 6 unambiguous single-tool cases is a floor check, not a ceiling — it says nothing about ambiguous or multi-step requests. And this exercised llama-server's native tool path, whereas the board config sets native_tools = false and uses ZeroClaw's text parser. The path ZeroClaw actually takes is still unvalidated.

Prefix caching is the dominant lever

The table above is the uncached worst case. In real tool-calling workloads the system prompt and tool schemas are byte-identical on every call, so llama.cpp prefills them once and later calls pay only for the delta. A ~600-token preamble + ~60-token request + ~50-token tool call is ~2 minutes uncached but ~25–30 s warm.

Three ways to destroy that advantage — all easy to do by accident:

• Varying the preamble. A timestamp, session id, or rotating skill list injected into the system prompt invalidates the cache every call.
• Unbounded max_tokens. At ~3 tok/s, a 500-token answer is 2.5 minutes of pure decode.
• max_tool_iterations = 6. Each iteration is a full model call — ~3 min per task. Acceptable for a background loop, not for anything interactive.

What this means for offline work

On the 0.5B (measured or extrapolated from the curve above):

task	cost	verdict
Function/tool call (warm, cached preamble)	3.8 s (measured)	viable, near-interactive
Process a result → decide → emit	~4–6 s	viable
Update / retag / summarize one short doc	~25–35 s	viable
Q&A retrieving several notes (1.5k ctx)	~1.5 min	marginal, usable offline
Code generation	—	not viable (quality, not speed)

An 8-hour overnight loop processes roughly 500k prefill tokens on the 0.5B — several hundred to a thousand short notes.

The offline task class Omar specified — function calls, processing a specific result, updating documents, nothing needing complex reasoning — is comfortably served by the 0.5B. Sub-4-second tool calls are fast enough that this is not merely a background queue worker.

⚠️ The harness, not the model, is the bottleneck

Raw model speed is not what makes or breaks offline. Prompt size is. Measured 2026-07-19 driving the real ZeroClaw agent loop (native_tools = false, the text parser):

	default profile	lean offline profile
Prompt tokens	4,718	706 (−85%)
Prefix-cache match (sim_best)	0.435	0.966
Tokens re-prefilled when warm	~2,600	24
Cold turn	cancelled mid-prefill	~29 s
Warm turn	never completed	11 s
Tool actually fired	no	yes

The 11 s warm figure is a complete agentic turn — two LLM calls plus tool execution — ending in a real answer ("The I2C devices on the bus are: i2c-0, i2c-1, and i2c-2.").

The text parser was never the problem. With the default profile the agent sent a 4,718-token prompt that took ~4.6 min to prefill at 17 tok/s, so the client cancelled before the model could respond. Disabling skills injection alone made it worse (10,901 tokens — past the 8k window, hard failure) because 84 accumulated memories in brain.db flooded the prompt. Flipping native_tools = true changed nothing (still 4,718): tool schemas were never the bulk.

The lean offline profile

Everything that controls prompt size lives on the runtime profile, not the agent (effective_skills_prompt_mode, schema.rs:4192, lets a runtime profile override the global [skills] prompt_injection_mode):

[runtime_profiles.offline]
agentic = true
max_tool_iterations = 3
compact_context = true              # 6000 chars / 2 RAG chunks; also drops Channel Capabilities
prompt_injection_mode = "compact"
max_system_prompt_chars = 2000
max_context_tokens = 3000
max_history_messages = 2
memory_recall_limit = 1
parallel_tools = false

[risk_profiles.sense_only]          # narrow tools ⇒ also drops the hardware block +
level = "supervised"                # Uno-Q flash imperative from the system prompt
allowed_tools = ["i2cdetect"]
auto_approve = ["i2cdetect"]

[agents.sense]
model_provider = "llamacpp.local"
risk_profile = "sense_only"
runtime_profile = "offline"
skill_bundles = []                  # the 12 bundled skills were most of the 4,718
mcp_bundles = []

Two zero-value footguns:

• memory_recall_limit = 0 means unlimited (usize::MAX), not disabled. Use 1.
• max_actions_per_hour = 0 is a hard zero that blocks everything; unlimited is u32::MAX.

Per-step prompts for multi-step operations

Cron JobType::Agent carries its own prompt, allowed_tools, and uses_memory per job (crates/zeroclaw-runtime/src/cron/types.rs:32,143), bound to an agent via AliasedAgentConfig.cron_jobs. So a multi-step offline operation is a sequence of jobs, each with the prompt and tool surface for its own step — and uses_memory = false skips memory injection entirely for steps that don't need it. Agent-to-agent delegates is the other path.

---

4. Lessons carried forward

Design constraints the spike document got wrong. Worth keeping even though the spike itself is dead.

• A fail-safe must never be "nominal". The spike degraded both error paths to a synthetic nominal response. In any monitoring context that is a silent all-clear manufactured by an outage. Degrade to unknown / escalate — never to "fine".
• Fallback must trigger on more than "network unavailable". Timeouts, 429s, DNS failures, and auth errors are the realistic degradations — a flaky workshop LAN far more often than a cleanly absent one. Matching only a NetworkUnavailable variant routes the common cases straight past the local model.
• Health-check endpoints, not processes. A wedged process passes pgrep.
• Persistent server, never per-call subprocess. Model load time dominates.
• Check exit status and stderr on any subprocess. Otherwise a crashed child surfaces as an unrelated JSON parse error, which is miserable to debug on a board.
• Don't gate on accuracy against undefined thresholds. The spike asked a model to choose among three labels without ever defining their boundaries, then scored it against the author's own intuition. That measures prompt underspecification, not model capability. Define the thresholds or don't call it accuracy.

---

See also

• README.md — provisioning, the three modalities, open → locked lifecycle
• config.template.toml — providers, agents, risk profile
• zeroclaw-supervisor.sh · recover-uno-q.sh