Replaces the never-executed "LiteRT-LM on UNO Q 4GB" spike plan with a measured record of the local-fallback path we actually run. Measured on board 65301572 (Qwen2.5-0.5B-Instruct, -c 8192, 4 threads), using llama-server's own timings rather than wall clock: prefill ~17-20 tok/s (linear), decode ~6-11 tok/s (degrades with KV) warm prefix-cached tool call: 3.8s, 6/6 correct structured calls Two findings that changed the deployment: 1. The board had drifted onto Qwen2.5-Coder-1.5B - larger and tuned for the wrong task. Reverting to the repo's 0.5B made tool calls ~6x faster (24s -> 3.8s) and freed ~700MB. The repo was right. 2. The harness, not the model, was the bottleneck. The default agent profile sent a 4718-token prompt (~4.6 min prefill) and the client cancelled before the model could answer. A lean runtime profile cuts that to 706 tokens, lifts prefix-cache match 0.435 -> 0.966, and completes a full agentic turn with a real tool call in 11s warm. The ZeroClaw text parser was never at fault. Prompt cost model for budgeting profiles: ~706 base (1 tool), ~244/additional tool, +315 for uno_q_flash (schema + flash imperative), ~53/skill in compact mode. Also standardises context on -c 8192 across all three provisioning paths (a 16k window costs ~16 min to fill at this speed and doubles KV for nothing), and fixes stale references to the deleted src/lib/harness.ts. Adds bench-prefill.sh and bench-tools.py as reproducible baselines. Co-Authored-By: Claude Opus 4.8 <[email protected]>
16 KiB
Uno Q workshop node — provisioning
Turns an Arduino Uno Q into an APESS workshop node: a ZeroClaw daemon that exposes an HTTP/SSE gateway, drives the on-board MCU (generate → compile → flash), and answers with cloud-first / on-board-Qwen-fallback inference. APESS proxies each board and streams its activity to participants and instructors.
What a provisioned board runs
zeroclaw daemonon:8080— the gateway APESS talks to (/pair,/webhook,/ws/chat,/api/events,/health). Usedaemon, notgateway start— only the daemon/agent/channel paths register the hardware tools (uno_q_flash, …).llama-serveron:8083— on-board Qwen for offline/fallback inference.zeroclaw-reload-watcher(loopback) — applies a team's browser config edits (e.g. enabling their Telegram bot) via/admin/reload. See modalities.- Four agents, selected per request via
?agent=(callers pass the alias explicitly — seesendPrompt(teamId, message, agent?)insrc/lib/api.ts):alias provider behaviour defaultcustom.claudecloud, fallback to on-board Qwen cloudcustom.cloudcloud only localllamacpp.localon-board Qwen only (fully offline) chaoscustom.deaddead endpoint → always fails over to Qwen (outage demo)
Talking to the node — the three modalities
Beyond the APESS-proxied prompt path, a team reaches its own node three ways.
All bind to the default (cloud-first) agent, so responses are fast; local
stays the offline fallback.
- Web chat by IP. The board is LAN-open during setup (
config.template.tomlsetshost = "0.0.0.0",allow_public_bind = true,require_pairing = false), so a team openshttp://<board-lan-ip>:8080/— the embedded ZeroClaw dashboard +/ws/chat— straight from the "Open your node →" link APESS shows after they claim. No token while open. - Telegram. Each board is its own agent, so each needs its own bot. A team
makes one via @BotFather, then in the dashboard (Config → channels →
telegram.default) pastes thebot_token, adds their username toallowed_users, and setsenabled = true. Dashboard writes only setpending_reload; thezeroclaw-reload-watcher(loopback) applies them within a few seconds — no shell. The channel is pre-bound todefault. - Voice (browser-mic streaming). Talk in the web chat, the agent speaks
back — requires a daemon built with the voice feature (see Voice below),
then uncomment
[channels.voice_duplex.default]in the config.
Open → locked lifecycle
Boards boot open so setup is frictionless on the isolated workshop LAN. When
a team is done, harden the node with zeroclaw-lockdown.sh (run on the board
or over adb — pair-code minting + reload are loopback-only, so it can't be a LAN
button):
adb -s <serial> shell '/home/arduino/zeroclaw-lockdown.sh' # prints a pair code
It flips require_pairing = true, reloads, and mints a pair code the team enters
once at http://<board-ip>:8080/pairing. After lockdown only their paired
devices reach the node. (Note: locking down also cuts the shared APESS proxy's
access unless you re-register a fresh token — a deliberate "graduate to a private
node" transition; validate against your run's needs.)
Voice — building a voice-capable board binary
gateway-voice-duplex is a Cargo feature and the SPA is embedded at compile
time, so the board binary must be built as:
cargo xtask web build # build the SPA into web/dist first (embedded-web needs it)
cargo build --release --target aarch64-unknown-linux-gnu \
--features "hardware,peripheral-rpi,embedded-web,zeroclaw-gateway/gateway-voice-duplex"
gateway-voice-duplex has no root-crate alias, so it's namespaced. The web chat
does client-side STT (Web Speech API) and plays the streamed tts_chunk audio;
the server handle_voice_event path is finished as part of this work. This is
the one modality that needs an on-board build + hardware test to verify.
Default agent skills
Every node ships with a bundle of UNO Q skills (skills/) installed into every
agent's workspace, so the agent knows this board's specifics instead of guessing
generic Arduino. Two complementary sets:
arduino-uno-q— one comprehensive reference skill (SKILL.md+ 7references/*.md): the dual-brain mental model, App Lab vs IDE, remote access, the Bridge/RPC deep-dive, full pin tables, wireless, and a workshop playbook. Best for the cloud agent viaread_skill→ references.- The granular set (
bridge,flashing,led-matrix,uno-q-hardware,sketch-patterns,modulino,linux-led,audio,vision,wireless,arduino-app-lab) — small, keyword-triggered skills. The fork's eager skill-inliner has rules for these names, so the on-board Qwen auto-inlines them (noread_skillround-trip).flashing+led-matrixcarry the exactuno_q_flash+ frame-API /ArduinoGraphics-not-installed detail.
It must live in each agent's own workspace, not a shared bundle. ZeroClaw's
read_skill returns only SKILL.md; the agent reads a references/*.md on demand
via the workspace-sandboxed file_read tool, so the references are only
reachable when the skill sits under ~/.zeroclaw/agents/<alias>/workspace/skills/.
push-skill.sh installs it into every agent workspace; provision-fleet.sh does
this per board. The risk profile (config.template.toml) auto-approves
read_skill + file_read so agents load it without a human approver.
./push-skill.sh <serial> # install the default skill into every agent
./push-skill.sh <serial> skills/<other-skill> # any SKILL.md + references/ skill
# then restart the daemon so it re-discovers skills at agent construction
Notes: the skill NAME is the directory name (arduino-uno-q); keep
description: single-line (ZeroClaw's frontmatter parser is a flat scanner, not
full YAML); no symlinks/scripts inside a skill dir (the auditor rejects them). The
fork's eager keyword→skill inliner has no rule for arduino-uno-q, so small local
models load it via a read_skill round-trip rather than auto-inlining — fine for
the cloud default; add a skill_preroute rule if you want the on-board Qwen to
auto-inline it.
Prerequisites on the board
/home/arduino/zeroclaw— the ZeroClaw binary (aarch64), with the gateway-peripheral-registration fix (merged toosobh/zeroclawmain)./home/arduino/llama/llama-server+/home/arduino/models/qwen.gguf.- Arduino Zephyr core
arduino:zephyr:unoq(0.51.0) + OpenOCD for flashing, andArduinoGraphicsif using scroll text.
Provision (dev — over USB/adb)
export APESS_ADMIN_CODE=adm-xxxxxxxx # instructor code
./provision-uno-q.sh <adb-serial> team-07 https://apess-api.redclaw.dev
Steps it runs: install config.template.toml (cloud endpoint substituted) →
start llama-server → start zeroclaw daemon → pair for a bearer token →
POST /nodes to APESS. Idempotent; re-run to re-provision.
Override the cloud endpoint (default is the local claude_shim on :8090):
CLOUD_URI=https://api.anthropic.com/v1 CLOUD_MODEL=claude-haiku-4-5 \
APESS_ADMIN_CODE=adm-xxxx ./provision-uno-q.sh <serial> team-07
The shim on
:8090is a bring-up convenience (one Mac). A real fleet pointsCLOUD_URIat a shared cloud endpoint (Anthropic / OpenRouter / LiteLLM) with a key, so boards don't each need a tunnel.
Onboarding — attendee self-serve (self-register + claim)
The workshop path: boards are preloaded and, on boot, announce themselves into APESS's unclaimed pool; an attendee then claims their board to their team from the web app (kit + 6-digit code) — no operator, no admin code, no adb.
board boot ──self-register {kitId,url,token}──▶ APESS unclaimed pool
▲
attendee: scan kit QR → team name → 6-digit code ──POST /claim──┘
(bearer token moves pool → node bridge, never touches the browser)
Fleet prep (host, once). Mint per-kit codes + per-board env files + the QR sticker CSV:
FLEET_SECRET=<shared-secret> APESS_URL=https://apess-api.redclaw.dev \
APESS_WEB=https://apess.redclaw.dev ./gen-kit-codes.sh 15
# → kit-codes/env/KIT-NN.env (one per board) kit-codes/kit-codes.csv (stickers)
The same FLEET_SECRET must be set on the APESS API (FLEET_SECRET env) — it
gates /nodes/self-register so only your boards can seed the pool. Each sticker
QR encodes …/workshop?kit=KIT-NN&code=NNNNNN, so scanning it pre-fills both.
All boards at once (recommended). Map each kit to its board's adb serial in
fleet.csv (see fleet.csv.example), then provision the fleet in one command —
per board it pushes the env + self-register script, enables the boot/refresh
timer (root) or an equivalent cron (MODE=cron, the default, no root), and
flashes the default boot animation (sketches/matrix_rain) onto the MCU:
cp fleet.csv.example fleet.csv # fill in kit,serial for each board
./provision-fleet.sh fleet.csv # MODE=systemd for root boards; FLASH_DEFAULT=0 to skip the animation
One board (manual equivalent):
adb -s <serial> push kit-codes/env/KIT-07.env /home/arduino/.zeroclaw/apess-node.env
adb -s <serial> push apess-selfregister.sh /home/arduino/
adb -s <serial> shell 'chmod +x /home/arduino/apess-selfregister.sh'
# with root: enable the boot + refresh timer
sudo cp systemd/apess-selfregister.service systemd/apess-selfregister.timer /etc/systemd/system/
sudo systemctl enable --now apess-selfregister.timer
# no root: run it once now (and let cron re-run it)
/home/arduino/apess-selfregister.sh
Print the claim stickers. Turn the sticker CSV into a self-contained, printable QR sheet (QRs baked in as inline SVG — no network needed to print):
brew install qrencode # one-time (build-time only)
./gen-qr-sheet.sh # kit-codes/kit-codes.csv → kit-codes/qr-sheet.html
open kit-codes/qr-sheet.html # print A4 at 100%; dashed borders are cut lines
provision-uno-q.sh (admin-driven, binds a board straight to a known team) is
still there for pre-provisioning / demo boards — the paths coexist.
Provision (production — systemd, boots on power-up)
With root on the board:
sudo cp systemd/zeroclaw-llama.service systemd/zeroclaw-daemon.service /etc/systemd/system/
sudo systemctl enable --now zeroclaw-llama zeroclaw-daemon
For a LAN fleet (participants reach the board's WiFi IP directly), the template
now ships this by default (host = "0.0.0.0", allow_public_bind = true,
require_pairing = false) so the dashboard/web-chat is reachable + open during
setup — see modalities above. Enable the reload-watcher alongside the daemon:
sudo cp systemd/zeroclaw-reload-watcher.service /etc/systemd/system/
sudo systemctl enable --now zeroclaw-reload-watcher
Then register once so APESS has the board's { url, token } (step 5; under
open-boot the token is a placeholder — the open board ignores auth).
Resilience — surviving a disconnect (no-root boards)
A sudden USB/adb drop breaks things that don't self-heal: the adb tunnels
vanish (board loses the cloud shim), held-shell services die, llama can wedge
(process alive but :8083 dead), and flashes silently stop landing while the
tool still reports success. The MCU keeps its last sketch; the paired token
survives.
The systemd units above are the clean answer when you have root. Some dev
boards don't — an expired account blocks sudo and there's no user session bus,
so neither system nor user units can run. For those, use the no-root pieces:
-
zeroclaw-supervisor.sh(runs on the board) — a watchdog that polls the/healthendpoints (a wedged process passespgrepbut fails here) and restarts llama / the daemon when they die or wedge. Children are launched withsetsid … execso they survive the shell that started them — the property plainnohup … &insideadb shelldoes not give you. Install + persist:adb -s <serial> push zeroclaw-supervisor.sh /home/arduino/ && \ adb -s <serial> shell 'chmod +x /home/arduino/zeroclaw-supervisor.sh; \ setsid nohup /home/arduino/zeroclaw-supervisor.sh >/dev/null 2>&1 </dev/null &' # boot persistence (no root; cron must be running): adb -s <serial> shell '(crontab -l 2>/dev/null | grep -v zeroclaw-supervisor.sh; \ echo "@reboot /home/arduino/zeroclaw-supervisor.sh") | crontab -' -
recover-uno-q.sh(runs on the host) — after the board is physically back, re-does adb + both tunnels, ensures the supervisor is up, and health-checks every hop by endpoint:./recover-uno-q.sh <serial> [cloud-shim-port] # default shim port 8090
See the unoq-disconnect-recovery note for the full failure-mode list.
Verify
adb -s <serial> forward tcp:8080 tcp:8080
curl -s localhost:8080/health # {"status":"ok"}
# each agent alias resolves (needs a bearer token from /pair):
curl -s -X POST 'localhost:8080/webhook?agent=local' -H "authorization: Bearer $TOK" \
-H 'content-type: application/json' -d '{"message":"one word: local"}'
?agent=local routes to Qwen (offline), ?agent=cloud to the cloud, ?agent=default
cloud-with-fallback. In APESS, the team's provider/fallback toggle picks the alias.
Files
config.template.toml— the node config (secrets stripped;__CLOUD_URI__/__CLOUD_MODEL__substituted at provision time).provision-uno-q.sh— one-shot provisioner (adb-driven, binds to a known team).gen-kit-codes.sh— host: mint per-kit claim codes + per-board env files + sticker CSV.provision-fleet.sh— host: provision a whole fleet fromfleet.csv(env + self-register + timer/cron).fleet.csv.example— kit→adb-serial assignment template forprovision-fleet.sh.gen-qr-sheet.sh— host: render the sticker CSV into a self-contained printable QR sheet (needsqrencode).flash-sketch.sh— host: compile + flash a sketch onto a board's MCU (sets the boot animation; defaultsketches/matrix_rain).sketches/— LED-matrix animations (matrix_rainis the board default;matrix_effectsis a sampler). Seesketches/README.md.apess-selfregister.sh— on-board: announce into APESS's unclaimed pool on boot.apess-node.env.example— per-board onboarding identity (KIT_ID / CLAIM_CODE / FLEET_SECRET / APESS_URL).systemd/*.service+apess-selfregister.timer— production units (need root).zeroclaw-supervisor.sh— on-board no-root watchdog (endpoint health + restart).zeroclaw-reload-watcher.sh— on-board: applies browser dashboard config edits by firing the loopback/admin/reloadwhenconfig.tomlchanges (makes Telegram-enable / lockdown self-serve without a shell). Root unit:systemd/zeroclaw-reload-watcher.service; no-root: launch viasetsid nohup.zeroclaw-lockdown.sh— on-board: the harden step — flipsrequire_pairing=true, reloads, and mints a pair code (open → locked).push-skill.sh— host: install a skill (or a whole skills dir) into every agent's workspace on a board.provision-fleet.shruns it per board overskills/.skills/— the bundled UNO Q skills (comprehensivearduino-uno-q+ the granular set). Installed per-agent so each node has them by default.recover-uno-q.sh— host-side post-disconnect recovery (re-tunnel + health-check).