Fleet: robust real-time connectivity + mosh-inspired reconnecting terminal

Nodes flapped online/offline and the terminal died on the first blip. WebSockets
are the right transport (outbound, NAT-friendly); the fixes harden around it.

Server (cm-api):
- Anti-clobber connection epoch: a reconnecting daemon gets a fresh epoch; a stale
  run_channel's teardown only clears the hub + sets offline if it still owns the
  slot — so a lingering old channel can't flip a live reconnection offline (the
  main false-offline cause).
- WS keepalive: run_channel now pings every 15s and tears down if no inbound
  frame (incl. pong) for 35s — dead links detected in seconds, not minutes.
- Staleness sweeper backstop: spawn_node_sweeper (8s tick / 20s window) wired in
  clawmates-server, so a vanished node goes offline within ~28s even if its
  channel hangs (mark_stale_offline was defined but never called).

Daemon (clawmates-node v0.3.0):
- Heartbeats off the select thread (dedicated thread owns System + blocking
  docker/tailscale/disk CLIs) so a slow op never starves heartbeats/pongs.
- Each handle_frame runs on its own task; added a 40s inbound idle deadline so a
  half-open socket triggers a reconnect.

Frontend:
- useNodes streams /api/nodes/live (SSE push) instead of a 3s poll; isLive()
  derives online from lastSeen freshness (<15s) so a transient column flip never
  shows a healthy node down.
- Node terminal: clean auto-reconnect loop (re-mint ticket -> reconnect -> tmux
  re-attaches and redraws the live screen = mosh-style snap-to-state over TCP),
  replacing the [disconnected] dead-end.

Mosh evaluated: harvest principles (session/transport decoupling, snap-to-state,
already given by tmux), don't adopt — UDP is incompatible with our browser+CF+NAT
topology and it's GPLv3. Removed temporary terminal debug traces + /api/debug route.

Verified: node holds steadily online (heartbeat 1-3s, no flap) and goes cleanly
offline when the daemon stops.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

crates/cm-api/src/routes/nodes.rs

@@ -136,39 +136,6 @@ pub async fn sandbox_check(
     }
 }
 
-#[derive(Deserialize)]
-pub struct DbgQuery {
-    pub dbg: String,
-}
-
-/// TEMP debug: open a terminal on a node and read ~2s of its output, no browser
-/// or auth. Lets us test the daemon→server PTY relay in isolation.
-pub async fn debug_node_pty(
-    State(state): State<AppState>,
-    Path(id): Path<Uuid>,
-    Query(q): Query<DbgQuery>,
-) -> Json<Value> {
-    if q.dbg != "clawdbg1" {
-        return Json(json!({ "ok": false, "error": "denied" }));
-    }
-    let node_id = NodeId::from(id);
-    let Some((sid, mut rx)) = state.node_hub.open_terminal(node_id, 120, 40).await else {
-        return Json(json!({ "ok": false, "error": "open_terminal None (node not connected)" }));
-    };
-    let mut total = 0usize;
-    let mut preview: Vec<u8> = Vec::new();
-    let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(2);
-    while let Ok(Some(bytes)) = tokio::time::timeout_at(deadline, rx.recv()).await {
-        total += bytes.len();
-        if preview.len() < 300 {
-            preview.extend_from_slice(&bytes);
-            preview.truncate(300);
-        }
-    }
-    state.node_hub.terminal_close(node_id, sid).await;
-    Json(json!({ "ok": true, "sid": sid, "bytes": total, "preview": String::from_utf8_lossy(&preview) }))
-}
-
 /// `DELETE /api/nodes/{id}` — deregister a node (workspace-scoped).
 pub async fn remove(
     State(state): State<AppState>,
@@ -212,9 +179,7 @@ pub async fn terminal_ticket(
     nodes::get(&state.pool, node_id, user.workspace_id)
         .await?
         .ok_or(ApiError::NotFound)?;
-    let online = state.node_hub.is_online(node_id).await;
-    eprintln!("[fleet-term] ticket request node={node_id} online={online}");
-    if !online {
+    if !state.node_hub.is_online(node_id).await {
         return Ok(Json(json!({ "error": "node is offline" })));
     }
     Ok(Json(json!({ "ticket": state.node_hub.mint_ticket(node_id).await })))
@@ -229,9 +194,7 @@ pub async fn terminal_ws(
     upgrade: WebSocketUpgrade,
 ) -> Response {
     let node_id = NodeId::from(id);
-    let redeemed = state.node_hub.redeem_ticket(&q.token).await;
-    eprintln!("[fleet-term] ws upgrade node={node_id} ticket_ok={}", redeemed == Some(node_id));
-    match redeemed {
+    match state.node_hub.redeem_ticket(&q.token).await {
         Some(t) if t == node_id => {}
         _ => return ApiError::Unauthorized.into_response(),
     }
@@ -249,21 +212,15 @@ struct TermCtrl {
 
 async fn bridge_terminal(hub: Arc<NodeHub>, node_id: NodeId, socket: WebSocket) {
     let Some((sid, mut rx)) = hub.open_terminal(node_id, 80, 24).await else {
-        eprintln!("[fleet-term] bridge: open_terminal None (node offline?) node={node_id}");
         return;
     };
-    eprintln!("[fleet-term] bridge start node={node_id} sid={sid}");
     let (mut ws_tx, mut ws_rx) = socket.split();
     let to_browser = async {
-        let mut sent = 0usize;
         while let Some(bytes) = rx.recv().await {
-            sent += bytes.len();
             if ws_tx.send(Message::Binary(bytes.into())).await.is_err() {
-                eprintln!("[fleet-term] bridge: browser send failed after {sent} bytes");
                 break;
             }
         }
-        eprintln!("[fleet-term] bridge to_browser ended, {sent} bytes total");
     };
     let to_node = async {
         while let Some(Ok(msg)) = ws_rx.next().await {