feat(uno-q): disconnect resilience — on-board supervisor + host recovery

A sudden USB/adb drop breaks the Uno Q node in ways that don't self-heal: adb tunnels vanish, held-shell services die, llama wedges (alive but not serving), and flashes silently stop landing. systemd is the clean fix with root — but the dev board's account is expired (no sudo) and has no user session bus, so neither system nor user units run. These are the no-root equivalents, both validated on hardware: - zeroclaw-supervisor.sh — on-board watchdog. Polls the /health ENDPOINTS (a wedged process passes pgrep but fails here) and restarts llama / the daemon on death or wedge. Launches children with `setsid … exec` so they survive the launching shell — the property `nohup … &` in adb shell lacks. Startup grace avoids reaping llama mid-cold-load; single-instance lock; no per-restart shell leak. Proven: kill -9 the daemon → auto-restarted. Boot-persisted via `@reboot` crontab (no root). - recover-uno-q.sh — host-side. After the board is back, re-does adb + both tunnels (out :8080, back :8090 shim), ensures the supervisor is running, and health-checks every hop by endpoint. Proven end-to-end after a simulated tunnel drop. README: new Resilience section documenting the no-root reality + both tools. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-07-03 13:24:36 -07:00
parent 56418f1742
commit 8130b0e30a
3 changed files with 180 additions and 1 deletions
@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+# recover-uno-q.sh — recover an APESS Uno Q node after a sudden USB/adb
+# disconnect. Run this from the host (Mac) once the board is physically back.
+#
+# A disconnect drops the adb tunnels, kills held-shell services, and can leave
+# llama wedged — none of it self-heals (see memory: unoq-disconnect-recovery).
+# This re-establishes every hop and verifies it by ENDPOINT, not by pgrep.
+#
+#   ./recover-uno-q.sh <adb-serial> [cloud-shim-port]
+#
+# Steps: reconnect adb → re-tunnel (8080 gateway out, shim port back) → ensure
+# the on-board supervisor is running (it restarts llama + daemon) → health-check
+# gateway, llama, and board→shim reachability.
+set -uo pipefail
+
+SERIAL="${1:?usage: recover-uno-q.sh <adb-serial> [cloud-shim-port]}"
+SHIM_PORT="${2:-8090}"
+SUPERVISOR="/home/arduino/zeroclaw-supervisor.sh"
+a() { adb -s "$SERIAL" "$@"; }
+
+echo "==> [1/5] reconnect adb"
+adb kill-server  >/dev/null 2>&1 || true
+adb start-server >/dev/null 2>&1 || true
+for _ in $(seq 1 15); do
+  [ "$(a get-state 2>/dev/null || true)" = "device" ] && break
+  sleep 1
+done
+[ "$(a get-state 2>/dev/null || true)" = "device" ] \
+  || { echo "    board $SERIAL not found — a physical re-plug is required"; exit 1; }
+echo "    device up"
+
+echo "==> [2/5] re-establish tunnels (out :8080 gateway, back :$SHIM_PORT shim)"
+a forward tcp:8080 tcp:8080 >/dev/null
+a reverse "tcp:$SHIM_PORT" "tcp:$SHIM_PORT" >/dev/null
+echo "    forward :8080 · reverse :$SHIM_PORT"
+
+echo "==> [3/5] ensure on-board supervisor is running"
+if a shell "test -x $SUPERVISOR" 2>/dev/null; then
+  a shell "pgrep -f zeroclaw-supervisor >/dev/null 2>&1 \
+    || (setsid nohup $SUPERVISOR >/dev/null 2>&1 < /dev/null &)" >/dev/null 2>&1
+  echo "    supervisor ensured (restarts llama + daemon on death/wedge)"
+else
+  echo "    WARN: $SUPERVISOR not on board — falling back to a one-shot daemon start"
+  a shell 'pkill -f "zeroclaw daemon" 2>/dev/null; sleep 1; \
+    cd /home/arduino && TMPDIR=/tmp setsid nohup ./zeroclaw daemon >> zc-daemon.log 2>&1 < /dev/null &' >/dev/null 2>&1
+fi
+
+echo "==> [4/5] wait for board services (endpoint, not pgrep)"
+for _ in $(seq 1 20); do
+  a shell 'curl -sf --max-time 3 http://127.0.0.1:8080/health >/dev/null' 2>/dev/null \
+    && { echo "    gateway :8080 healthy"; break; }
+  sleep 3
+done
+a shell 'curl -sf --max-time 3 http://127.0.0.1:8083/health >/dev/null' 2>/dev/null \
+  && echo "    llama :8083 healthy" \
+  || echo "    llama :8083 still warming (cold GGUF load is 3–5 min)"
+
+echo "==> [5/5] verify cloud shim reachable from the board"
+a shell "curl -sf --max-time 4 http://127.0.0.1:$SHIM_PORT/v1/models >/dev/null" 2>/dev/null \
+  && echo "    board can reach cloud shim :$SHIM_PORT" \
+  || echo "    WARN: shim unreachable from board — is it running on the host?"
+
+echo "==> recovery complete for $SERIAL"