Fleet: robust real-time connectivity + mosh-inspired reconnecting terminal

Nodes flapped online/offline and the terminal died on the first blip. WebSockets
are the right transport (outbound, NAT-friendly); the fixes harden around it.

Server (cm-api):
- Anti-clobber connection epoch: a reconnecting daemon gets a fresh epoch; a stale
  run_channel's teardown only clears the hub + sets offline if it still owns the
  slot — so a lingering old channel can't flip a live reconnection offline (the
  main false-offline cause).
- WS keepalive: run_channel now pings every 15s and tears down if no inbound
  frame (incl. pong) for 35s — dead links detected in seconds, not minutes.
- Staleness sweeper backstop: spawn_node_sweeper (8s tick / 20s window) wired in
  clawmates-server, so a vanished node goes offline within ~28s even if its
  channel hangs (mark_stale_offline was defined but never called).

Daemon (clawmates-node v0.3.0):
- Heartbeats off the select thread (dedicated thread owns System + blocking
  docker/tailscale/disk CLIs) so a slow op never starves heartbeats/pongs.
- Each handle_frame runs on its own task; added a 40s inbound idle deadline so a
  half-open socket triggers a reconnect.

Frontend:
- useNodes streams /api/nodes/live (SSE push) instead of a 3s poll; isLive()
  derives online from lastSeen freshness (<15s) so a transient column flip never
  shows a healthy node down.
- Node terminal: clean auto-reconnect loop (re-mint ticket -> reconnect -> tmux
  re-attaches and redraws the live screen = mosh-style snap-to-state over TCP),
  replacing the [disconnected] dead-end.

Mosh evaluated: harvest principles (session/transport decoupling, snap-to-state,
already given by tmux), don't adopt — UDP is incompatible with our browser+CF+NAT
topology and it's GPLv3. Removed temporary terminal debug traces + /api/debug route.

Verified: node holds steadily online (heartbeat 1-3s, no flap) and goes cleanly
offline when the daemon stops.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

frontend/src/components/dashboard/fleet/FleetPanels.tsx

@@ -48,6 +48,20 @@ const STATUS_COLOR: Record<FleetNode["status"], string> = {
   draining: "#ff8a7a",
 };
 
+/** A node is "live" if the channel reports online OR its last heartbeat is fresh
+ *  (<15s) — so a transient status-column flip never shows a healthy node down. */
+export function isLive(n: FleetNode): boolean {
+  if (n.status === "pending" || n.status === "draining") return false;
+  if (n.lastSeen != null && Date.now() / 1000 - n.lastSeen < 15) return true;
+  return n.status === "online";
+}
+
+/** The status to render: pending/draining pass through; otherwise online iff live. */
+export function displayStatus(n: FleetNode): FleetNode["status"] {
+  if (n.status === "pending" || n.status === "draining") return n.status;
+  return isLive(n) ? "online" : "offline";
+}
+
 function fmtBytes(b: number): string {
   if (b >= 1e12) return `${(b / 1e12).toFixed(1)} TB`;
   if (b >= 1e9) return `${(b / 1e9).toFixed(1)} GB`;
@@ -55,14 +69,32 @@ function fmtBytes(b: number): string {
   return `${b} B`;
 }
 
-/** Poll the workspace's nodes every 3s. */
+/** Stream the workspace's nodes live over SSE (status updates push instantly,
+ *  no 3s poll lag). `refresh` does a one-shot GET for an immediate nudge after
+ *  pairing a new host. */
 export function useNodes(): { nodes: FleetNode[]; refresh: () => void } {
-  const { data, refresh } = useFetchJson<{ nodes: FleetNode[] }>("/api/nodes");
+  const [nodes, setNodes] = useState<FleetNode[]>([]);
+  const refresh = useCallback(() => {
+    fetch("/api/nodes")
+      .then((r) => (r.ok ? r.json() : null))
+      .then((d: { nodes: FleetNode[] } | null) => {
+        if (d?.nodes) setNodes(d.nodes);
+      })
+      .catch(() => {});
+  }, []);
   useEffect(() => {
-    const t = setInterval(refresh, 3000);
-    return () => clearInterval(t);
+    refresh();
+    const es = new EventSource("/api/nodes/live");
+    es.addEventListener("nodes", (e) => {
+      try {
+        setNodes(JSON.parse((e as MessageEvent).data) as FleetNode[]);
+      } catch {
+        /* ignore malformed frame */
+      }
+    });
+    return () => es.close();
   }, [refresh]);
-  return { nodes: data?.nodes ?? [], refresh };
+  return { nodes, refresh };
 }
 
 function Bar({ label, pct, detail, color }: { label: string; pct: number; detail: string; color: string }) {