Omar SobhandClaude Opus 4.8
21fc93556e
A board power-cycle exposed two bugs that left the node dead after boot: - recover-uno-q.sh step 3 ran `pgrep -f zeroclaw-supervisor`, which matches the pgrep's OWN shell (its args contain the supervisor path) — so it always concluded "already running" and never launched the supervisor. Bracket the pattern (`[z]eroclaw-supervisor.sh`) so only the real process matches. - The supervisor's single-instance lock only checked that the locked pid was alive. After a reboot the stale pid can be recycled by an unrelated process, falsely blocking startup. Now require the live pid's /proc/<pid>/cmdline to actually be a supervisor before deferring — otherwise treat the lock as stale. - Also export a full PATH in the supervisor for cron's minimal @reboot env. Validated on hardware: discriminator matches a real supervisor and rejects init's pid; fixed recover detects the running supervisor without spawning a duplicate; board comes back healthy (gateway + llama up). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
66 lines
2.9 KiB
Bash
Executable File
66 lines
2.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
||
# recover-uno-q.sh — recover an APESS Uno Q node after a sudden USB/adb
|
||
# disconnect. Run this from the host (Mac) once the board is physically back.
|
||
#
|
||
# A disconnect drops the adb tunnels, kills held-shell services, and can leave
|
||
# llama wedged — none of it self-heals (see memory: unoq-disconnect-recovery).
|
||
# This re-establishes every hop and verifies it by ENDPOINT, not by pgrep.
|
||
#
|
||
# ./recover-uno-q.sh <adb-serial> [cloud-shim-port]
|
||
#
|
||
# Steps: reconnect adb → re-tunnel (8080 gateway out, shim port back) → ensure
|
||
# the on-board supervisor is running (it restarts llama + daemon) → health-check
|
||
# gateway, llama, and board→shim reachability.
|
||
set -uo pipefail
|
||
|
||
SERIAL="${1:?usage: recover-uno-q.sh <adb-serial> [cloud-shim-port]}"
|
||
SHIM_PORT="${2:-8090}"
|
||
SUPERVISOR="/home/arduino/zeroclaw-supervisor.sh"
|
||
a() { adb -s "$SERIAL" "$@"; }
|
||
|
||
echo "==> [1/5] reconnect adb"
|
||
adb kill-server >/dev/null 2>&1 || true
|
||
adb start-server >/dev/null 2>&1 || true
|
||
for _ in $(seq 1 15); do
|
||
[ "$(a get-state 2>/dev/null || true)" = "device" ] && break
|
||
sleep 1
|
||
done
|
||
[ "$(a get-state 2>/dev/null || true)" = "device" ] \
|
||
|| { echo " board $SERIAL not found — a physical re-plug is required"; exit 1; }
|
||
echo " device up"
|
||
|
||
echo "==> [2/5] re-establish tunnels (out :8080 gateway, back :$SHIM_PORT shim)"
|
||
a forward tcp:8080 tcp:8080 >/dev/null
|
||
a reverse "tcp:$SHIM_PORT" "tcp:$SHIM_PORT" >/dev/null
|
||
echo " forward :8080 · reverse :$SHIM_PORT"
|
||
|
||
echo "==> [3/5] ensure on-board supervisor is running"
|
||
if a shell "test -x $SUPERVISOR" 2>/dev/null; then
|
||
# Bracket the pattern so the pgrep's own shell (whose args contain the
|
||
# supervisor path) doesn't self-match and skip the launch.
|
||
a shell "pgrep -f '[z]eroclaw-supervisor.sh' >/dev/null 2>&1 \
|
||
|| (setsid nohup $SUPERVISOR >/dev/null 2>&1 < /dev/null &)" >/dev/null 2>&1
|
||
echo " supervisor ensured (restarts llama + daemon on death/wedge)"
|
||
else
|
||
echo " WARN: $SUPERVISOR not on board — falling back to a one-shot daemon start"
|
||
a shell 'pkill -f "zeroclaw daemon" 2>/dev/null; sleep 1; \
|
||
cd /home/arduino && TMPDIR=/tmp setsid nohup ./zeroclaw daemon >> zc-daemon.log 2>&1 < /dev/null &' >/dev/null 2>&1
|
||
fi
|
||
|
||
echo "==> [4/5] wait for board services (endpoint, not pgrep)"
|
||
for _ in $(seq 1 20); do
|
||
a shell 'curl -sf --max-time 3 http://127.0.0.1:8080/health >/dev/null' 2>/dev/null \
|
||
&& { echo " gateway :8080 healthy"; break; }
|
||
sleep 3
|
||
done
|
||
a shell 'curl -sf --max-time 3 http://127.0.0.1:8083/health >/dev/null' 2>/dev/null \
|
||
&& echo " llama :8083 healthy" \
|
||
|| echo " llama :8083 still warming (cold GGUF load is 3–5 min)"
|
||
|
||
echo "==> [5/5] verify cloud shim reachable from the board"
|
||
a shell "curl -sf --max-time 4 http://127.0.0.1:$SHIM_PORT/v1/models >/dev/null" 2>/dev/null \
|
||
&& echo " board can reach cloud shim :$SHIM_PORT" \
|
||
|| echo " WARN: shim unreachable from board — is it running on the host?"
|
||
|
||
echo "==> recovery complete for $SERIAL"
|