Fleet: robust real-time connectivity + mosh-inspired reconnecting terminal
ci / gates (push) Failing after 6s
ci / frontend (push) Has been skipped
ci / rust (push) Has been skipped
ci / sandbox-k8s (push) Has been skipped
ci / e2e (push) Has been skipped

Nodes flapped online/offline and the terminal died on the first blip. WebSockets
are the right transport (outbound, NAT-friendly); the fixes harden around it.

Server (cm-api):
- Anti-clobber connection epoch: a reconnecting daemon gets a fresh epoch; a stale
  run_channel's teardown only clears the hub + sets offline if it still owns the
  slot — so a lingering old channel can't flip a live reconnection offline (the
  main false-offline cause).
- WS keepalive: run_channel now pings every 15s and tears down if no inbound
  frame (incl. pong) for 35s — dead links detected in seconds, not minutes.
- Staleness sweeper backstop: spawn_node_sweeper (8s tick / 20s window) wired in
  clawmates-server, so a vanished node goes offline within ~28s even if its
  channel hangs (mark_stale_offline was defined but never called).

Daemon (clawmates-node v0.3.0):
- Heartbeats off the select thread (dedicated thread owns System + blocking
  docker/tailscale/disk CLIs) so a slow op never starves heartbeats/pongs.
- Each handle_frame runs on its own task; added a 40s inbound idle deadline so a
  half-open socket triggers a reconnect.

Frontend:
- useNodes streams /api/nodes/live (SSE push) instead of a 3s poll; isLive()
  derives online from lastSeen freshness (<15s) so a transient column flip never
  shows a healthy node down.
- Node terminal: clean auto-reconnect loop (re-mint ticket -> reconnect -> tmux
  re-attaches and redraws the live screen = mosh-style snap-to-state over TCP),
  replacing the [disconnected] dead-end.

Mosh evaluated: harvest principles (session/transport decoupling, snap-to-state,
already given by tmux), don't adopt — UDP is incompatible with our browser+CF+NAT
topology and it's GPLv3. Removed temporary terminal debug traces + /api/debug route.

Verified: node holds steadily online (heartbeat 1-3s, no flap) and goes cleanly
offline when the daemon stops.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
Omar Sobh
2026-06-25 07:18:05 -07:00
co-authored by Claude Opus 4.8
parent 95bd022d07
commit 94828ed887
10 changed files with 258 additions and 184 deletions
+3
View File
@@ -266,6 +266,9 @@ async fn run() -> Result<(), String> {
// Expiry/retention sweep: expires stale auth/oauth rows and prunes old
// journal/audit rows hourly so unbounded tables don't accumulate.
cm_api::cleanup_sweeper::spawn(pool.clone(), std::time::Duration::from_secs(3600));
// Fleet backstop: a node whose heartbeats stop (without a clean channel
// close) goes offline within ~28s even if its control channel hangs.
cm_api::fleet::spawn_node_sweeper(pool.clone(), std::time::Duration::from_secs(8), 20);
// Hosted identity (Clerk / OIDC): pin the issuer and load its JWKS.
let auth_verifier = match config.auth.mode {