Files
apress/deploy/uno-q/README.md
T
Omar SobhandClaude Opus 4.8 8130b0e30a feat(uno-q): disconnect resilience — on-board supervisor + host recovery
A sudden USB/adb drop breaks the Uno Q node in ways that don't self-heal:
adb tunnels vanish, held-shell services die, llama wedges (alive but not
serving), and flashes silently stop landing. systemd is the clean fix with
root — but the dev board's account is expired (no sudo) and has no user
session bus, so neither system nor user units run. These are the no-root
equivalents, both validated on hardware:

- zeroclaw-supervisor.sh — on-board watchdog. Polls the /health ENDPOINTS
  (a wedged process passes pgrep but fails here) and restarts llama / the
  daemon on death or wedge. Launches children with `setsid … exec` so they
  survive the launching shell — the property `nohup … &` in adb shell lacks.
  Startup grace avoids reaping llama mid-cold-load; single-instance lock;
  no per-restart shell leak. Proven: kill -9 the daemon → auto-restarted.
  Boot-persisted via `@reboot` crontab (no root).

- recover-uno-q.sh — host-side. After the board is back, re-does adb + both
  tunnels (out :8080, back :8090 shim), ensures the supervisor is running,
  and health-checks every hop by endpoint. Proven end-to-end after a
  simulated tunnel drop.

README: new Resilience section documenting the no-root reality + both tools.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-07-03 13:24:36 -07:00

5.5 KiB
Raw Blame History

Uno Q workshop node — provisioning

Turns an Arduino Uno Q into an APESS workshop node: a ZeroClaw daemon that exposes an HTTP/SSE gateway, drives the on-board MCU (generate → compile → flash), and answers with cloud-first / on-board-Qwen-fallback inference. APESS proxies each board and streams its activity to participants and instructors.

What a provisioned board runs

  • zeroclaw daemon on :8080 — the gateway APESS talks to (/pair, /webhook, /ws/chat, /api/events, /health). Use daemon, not gateway start — only the daemon/agent/channel paths register the hardware tools (uno_q_flash, …).
  • llama-server on :8083 — on-board Qwen for offline/fallback inference.
  • Three agents, selected per request via ?agent= (APESS maps the harness provider choice → alias; see src/lib/harness.ts harnessToAgent):
    alias provider behaviour
    default custom.claude cloud, fallback to on-board Qwen
    cloud custom.cloud cloud only
    local llamacpp.local on-board Qwen only (fully offline)

Prerequisites on the board

  • /home/arduino/zeroclaw — the ZeroClaw binary (aarch64), with the gateway-peripheral-registration fix (merged to osobh/zeroclaw main).
  • /home/arduino/llama/llama-server + /home/arduino/models/qwen.gguf.
  • Arduino Zephyr core arduino:zephyr:unoq (0.51.0) + OpenOCD for flashing, and ArduinoGraphics if using scroll text.

Provision (dev — over USB/adb)

export APESS_ADMIN_CODE=adm-xxxxxxxx          # instructor code
./provision-uno-q.sh <adb-serial> team-07 https://apess-api.redclaw.dev

Steps it runs: install config.template.toml (cloud endpoint substituted) → start llama-server → start zeroclaw daemon → pair for a bearer token → POST /nodes to APESS. Idempotent; re-run to re-provision.

Override the cloud endpoint (default is the local claude_shim on :8090):

CLOUD_URI=https://api.anthropic.com/v1 CLOUD_MODEL=claude-haiku-4-5 \
  APESS_ADMIN_CODE=adm-xxxx ./provision-uno-q.sh <serial> team-07

The shim on :8090 is a bring-up convenience (one Mac). A real fleet points CLOUD_URI at a shared cloud endpoint (Anthropic / OpenRouter / LiteLLM) with a key, so boards don't each need a tunnel.

Provision (production — systemd, boots on power-up)

With root on the board:

sudo cp systemd/zeroclaw-llama.service systemd/zeroclaw-daemon.service /etc/systemd/system/
sudo systemctl enable --now zeroclaw-llama zeroclaw-daemon

For a LAN fleet (participants reach the board's WiFi IP directly), set in config.toml:

[gateway]
host = "0.0.0.0"
allow_public_bind = true

Then pair + register once (steps 45 of the script) so APESS has the board's { url, token }.

Resilience — surviving a disconnect (no-root boards)

A sudden USB/adb drop breaks things that don't self-heal: the adb tunnels vanish (board loses the cloud shim), held-shell services die, llama can wedge (process alive but :8083 dead), and flashes silently stop landing while the tool still reports success. The MCU keeps its last sketch; the paired token survives.

The systemd units above are the clean answer when you have root. Some dev boards don't — an expired account blocks sudo and there's no user session bus, so neither system nor user units can run. For those, use the no-root pieces:

  • zeroclaw-supervisor.sh (runs on the board) — a watchdog that polls the /health endpoints (a wedged process passes pgrep but fails here) and restarts llama / the daemon when they die or wedge. Children are launched with setsid … exec so they survive the shell that started them — the property plain nohup … & inside adb shell does not give you. Install + persist:

    adb -s <serial> push zeroclaw-supervisor.sh /home/arduino/ && \
      adb -s <serial> shell 'chmod +x /home/arduino/zeroclaw-supervisor.sh; \
        setsid nohup /home/arduino/zeroclaw-supervisor.sh >/dev/null 2>&1 </dev/null &'
    # boot persistence (no root; cron must be running):
    adb -s <serial> shell '(crontab -l 2>/dev/null | grep -v zeroclaw-supervisor.sh; \
      echo "@reboot /home/arduino/zeroclaw-supervisor.sh") | crontab -'
    
  • recover-uno-q.sh (runs on the host) — after the board is physically back, re-does adb + both tunnels, ensures the supervisor is up, and health-checks every hop by endpoint:

    ./recover-uno-q.sh <serial> [cloud-shim-port]   # default shim port 8090
    

See the unoq-disconnect-recovery note for the full failure-mode list.

Verify

adb -s <serial> forward tcp:8080 tcp:8080
curl -s localhost:8080/health                       # {"status":"ok"}
# each agent alias resolves (needs a bearer token from /pair):
curl -s -X POST 'localhost:8080/webhook?agent=local'  -H "authorization: Bearer $TOK" \
  -H 'content-type: application/json' -d '{"message":"one word: local"}'

?agent=local routes to Qwen (offline), ?agent=cloud to the cloud, ?agent=default cloud-with-fallback. In APESS, the team's provider/fallback toggle picks the alias.

Files

  • config.template.toml — the node config (secrets stripped; __CLOUD_URI__ / __CLOUD_MODEL__ substituted at provision time).
  • provision-uno-q.sh — one-shot provisioner (adb-driven).
  • systemd/*.service — production units (need root).
  • zeroclaw-supervisor.sh — on-board no-root watchdog (endpoint health + restart).
  • recover-uno-q.sh — host-side post-disconnect recovery (re-tunnel + health-check).