Fleet: robust real-time connectivity + mosh-inspired reconnecting terminal
Nodes flapped online/offline and the terminal died on the first blip. WebSockets are the right transport (outbound, NAT-friendly); the fixes harden around it. Server (cm-api): - Anti-clobber connection epoch: a reconnecting daemon gets a fresh epoch; a stale run_channel's teardown only clears the hub + sets offline if it still owns the slot — so a lingering old channel can't flip a live reconnection offline (the main false-offline cause). - WS keepalive: run_channel now pings every 15s and tears down if no inbound frame (incl. pong) for 35s — dead links detected in seconds, not minutes. - Staleness sweeper backstop: spawn_node_sweeper (8s tick / 20s window) wired in clawmates-server, so a vanished node goes offline within ~28s even if its channel hangs (mark_stale_offline was defined but never called). Daemon (clawmates-node v0.3.0): - Heartbeats off the select thread (dedicated thread owns System + blocking docker/tailscale/disk CLIs) so a slow op never starves heartbeats/pongs. - Each handle_frame runs on its own task; added a 40s inbound idle deadline so a half-open socket triggers a reconnect. Frontend: - useNodes streams /api/nodes/live (SSE push) instead of a 3s poll; isLive() derives online from lastSeen freshness (<15s) so a transient column flip never shows a healthy node down. - Node terminal: clean auto-reconnect loop (re-mint ticket -> reconnect -> tmux re-attaches and redraws the live screen = mosh-style snap-to-state over TCP), replacing the [disconnected] dead-end. Mosh evaluated: harvest principles (session/transport decoupling, snap-to-state, already given by tmux), don't adopt — UDP is incompatible with our browser+CF+NAT topology and it's GPLv3. Removed temporary terminal debug traces + /api/debug route. Verified: node holds steadily online (heartbeat 1-3s, no flap) and goes cleanly offline when the daemon stops. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
95bd022d07
commit
94828ed887
@@ -266,6 +266,9 @@ async fn run() -> Result<(), String> {
|
||||
// Expiry/retention sweep: expires stale auth/oauth rows and prunes old
|
||||
// journal/audit rows hourly so unbounded tables don't accumulate.
|
||||
cm_api::cleanup_sweeper::spawn(pool.clone(), std::time::Duration::from_secs(3600));
|
||||
// Fleet backstop: a node whose heartbeats stop (without a clean channel
|
||||
// close) goes offline within ~28s even if its control channel hangs.
|
||||
cm_api::fleet::spawn_node_sweeper(pool.clone(), std::time::Duration::from_secs(8), 20);
|
||||
|
||||
// Hosted identity (Clerk / OIDC): pin the issuer and load its JWKS.
|
||||
let auth_verifier = match config.auth.mode {
|
||||
|
||||
Reference in New Issue
Block a user