7c1af2e070c63109400e33923c4c88be0864c432
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar SobhandClaude Opus 4.8
|
4de2f31b50 |
Fleet terminal: WebRTC DataChannel direct path (low-latency) + WS fallback
Terminal keystrokes were ~400ms because every byte relayed browser→Cloudflare→
gw-04 (Europe)→tailscale→node, even when the node is on the user's own LAN. Add a
direct browser↔node WebRTC DataChannel so co-located terminals run at LAN speed;
the gateway is reduced to signaling; the WebSocket relay stays as the automatic
fallback (graceful degradation — never worse than before).
Daemon (clawmates-node v0.4.0, new src/rtc.rs):
- Add the `webrtc` crate (reuses the ring crypto provider we already install — no
conflict). Browser is the offerer; we answer, trickle ICE back over the control
channel, and on DataChannel open spawn a host PTY (tmux) bridged DIRECTLY to the
channel. Refactor open_pty → spawn_terminal_pty shared by both transports.
iceServers: STUN + auto host/tailnet candidates (direct, no relay, for LAN/tailnet).
Server (cm-api):
- NodeConn.signal_sinks; Uplink WebRtcAnswer/WebRtcIce/WebRtcFailed routed to the
browser; NodeHub webrtc_offer/ice/close + open_session/open_pty (open_terminal
split so the PTY opens only once the transport is chosen). bridge_terminal relays
signaling over the existing ticket-authed WS and opens the relay PTY on
{type:"fallback"}.
Browser (NodeTerminalApp):
- RTCPeerConnection + reliable/ordered DataChannel; offer/answer/ICE over the WS;
2.5s race → use the DataChannel if it opens, else fall back to the WS relay.
Reconnect wraps both. A direct⚡/relayed indicator shows the live transport.
Deployed; both nodes (morpheus, tank) updated to v0.4.0 and steady online. Direct-
path proof is a browser action (the ⚡ indicator + latency); confirmable from the
daemon's [rtc] logs.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
94828ed887 |
Fleet: robust real-time connectivity + mosh-inspired reconnecting terminal
Nodes flapped online/offline and the terminal died on the first blip. WebSockets are the right transport (outbound, NAT-friendly); the fixes harden around it. Server (cm-api): - Anti-clobber connection epoch: a reconnecting daemon gets a fresh epoch; a stale run_channel's teardown only clears the hub + sets offline if it still owns the slot — so a lingering old channel can't flip a live reconnection offline (the main false-offline cause). - WS keepalive: run_channel now pings every 15s and tears down if no inbound frame (incl. pong) for 35s — dead links detected in seconds, not minutes. - Staleness sweeper backstop: spawn_node_sweeper (8s tick / 20s window) wired in clawmates-server, so a vanished node goes offline within ~28s even if its channel hangs (mark_stale_offline was defined but never called). Daemon (clawmates-node v0.3.0): - Heartbeats off the select thread (dedicated thread owns System + blocking docker/tailscale/disk CLIs) so a slow op never starves heartbeats/pongs. - Each handle_frame runs on its own task; added a 40s inbound idle deadline so a half-open socket triggers a reconnect. Frontend: - useNodes streams /api/nodes/live (SSE push) instead of a 3s poll; isLive() derives online from lastSeen freshness (<15s) so a transient column flip never shows a healthy node down. - Node terminal: clean auto-reconnect loop (re-mint ticket -> reconnect -> tmux re-attaches and redraws the live screen = mosh-style snap-to-state over TCP), replacing the [disconnected] dead-end. Mosh evaluated: harvest principles (session/transport decoupling, snap-to-state, already given by tmux), don't adopt — UDP is incompatible with our browser+CF+NAT topology and it's GPLv3. Removed temporary terminal debug traces + /api/debug route. Verified: node holds steadily online (heartbeat 1-3s, no flap) and goes cleanly offline when the daemon stops. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
95bd022d07 |
Fleet terminal: fix tmux nesting-refusal + frontend resize storm
Root cause (from the server trace: 69 banner bytes then immediate "to_browser
ended" = pty_exit): tmux spawned and exited instantly, writing its "sessions
should be nested with care" warning to stderr (not the PTY). The operator runs
the daemon inside their own tmux, so $TMUX was inherited and the spawned tmux
refused to nest → browser saw nothing.
- daemon (v0.2.2): spawn tmux on a DEDICATED socket (`tmux -L clawmates
new-session -A -s main`) and `env_remove("TMUX")`, so it can never collide with
or be refused by the operator's tmux.
- NodeTerminalApp: debounce the ResizeObserver (150ms). The pull-out animates
open, firing the observer on every pixel — previously ~80 resize frames per
open, each fit()+SIGWINCH. Now one resize after layout settles.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
27f5d05f96 |
Fleet terminal: daemon self-diagnosis + immediate banner + debug route
The server trace showed pty_open is sent but the daemon (morpheus, v0.2.0) emits
no pty_out — so the PTY spawn was dying silently. Instrument it:
- daemon open_pty: log open/tmux/first-read/EOF/error/total to stdout, and send
an IMMEDIATE banner pty_out ("[clawmates] host shell on <host> — starting…") so
the browser confirms the relay even before the shell draws. If open_pty fails,
send the error as pty_out (was a silent pty_exit). Bump to v0.2.1.
- cm-api: temp GET /api/debug/node-pty/{id}?dbg=… opens a node terminal and reads
~2s of output with no browser/auth, to test the relay in isolation.
- NodeTerminalApp + ticket/ws routes already log each hop ([node-term]/[fleet-term]).
Diagnostic logic: banner shows + shell doesn't → relay ok, shell is the problem;
nothing shows → relay broken; daemon "EOF after N bytes" → shell exited.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
8e71c8403b |
clawmates-node: bump to v0.2.0 (selftest + tmux re-hosted)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
cfa16751c5 |
clawmates-node: fix rustls CryptoProvider panic + non-intrusive Tailscale
Two bugs surfaced running the daemon on a real node:
- Linking cm-sandbox (bollard) brought a second rustls provider into the graph,
so rustls couldn't auto-pick one and panicked at the WSS TLS handshake. Install
the ring provider explicitly at startup (rustls dep + install_default()).
- The daemon auto-ran `tailscale set --ssh`, which tries to reroute the user's
live SSH session and aborts ("will result in your session disconnecting"). Now
Tailscale is only touched when an auth key is explicitly passed (opt-in), with
--accept-risk=lose-ssh to avoid the interactive abort.
Rebuilt + re-hosted both binaries (linux-amd64, darwin-arm64).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
fb59378aa2 |
Fleet P2b: run agent sandboxes on connected nodes (RemoteDriver + placement)
Agents can now provision their sandbox on a connected fleet node instead of the gateway host. Local stays the strict default, so existing agents are byte-for- byte unaffected until explicitly placed elsewhere. Security parity: the daemon links the REAL cm-sandbox DockerDriver and runs the typed container ops (sb_provision/sb_exec/sb_destroy/sb_health/sb_list) through it — identical hardening (cap-drop ALL, seccomp, no-net, read-only, non-root) to local sandboxes. cm-sandbox spec types are now Serialize/Deserialize so the spec crosses the channel. - cm-api: RemoteDriver (impl SandboxDriver over the node channel) + HubDriverProvider (impl cm_runtime::NodeDriverProvider, hands out a driver only for connected nodes via a sync online set) + NodeHub.call/is_connected. AppState.with_node_hub so the hub is shared with the placement provider. - cm-runtime SandboxManager: driver_for(node_id) routes by the recorded agent_containers.node_id (local default = existing driver, identical path); placement_node() reads the workspace setting and falls back to local if the node is offline; exec/release route accordingly. NodeDriverProvider trait. - DB: 0020_workspace_placement + repo (for_agent/get/set/clear). - main.rs: build the NodeHub first; inject HubDriverProvider into the agent manager + share the hub with AppState. - API+UI: GET/PUT /api/fleet/placement + a "Run agents on: Local / <node>" selector in the Fleet overview. Note: a node must be able to pull the agent image (the daemon docker-pulls it); interactive PTY for agent containers on remote nodes is not wired (Terminal app stays local) — the in-dashboard node shell already covers host access. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
f5f96508eb |
Fleet P2a: in-dashboard remote terminal (PTY over the WSS channel)
You can now open a real shell on any connected node from the dashboard — the
daemon spawns a host PTY and streams it over the existing outbound control
channel (no inbound port, no Tailscale brokering needed).
Daemon:
- portable-pty host shell sessions: pty_open/pty_in/pty_resize/pty_close ops; a
reader thread streams base64 pty_out frames. Outbound frames now funnel through
one mpsc channel so PTY output and heartbeats interleave.
cm-api NodeHub:
- per-connection pty_sinks + sid multiplexing; open_terminal/terminal_input/
terminal_resize/terminal_close; in-memory single-use terminal tickets (the
browser WS can't carry a bearer, and the session is instance-local anyway).
- routes/nodes.rs: POST /api/nodes/{id}/terminal/ticket + GET .../terminal/ws
(bridges browser xterm <-> node PTY: binary = keystrokes, text = resize).
Frontend:
- NodeTerminal xterm modal (reuses the agent Terminal's xterm setup); a Terminal
button on each online node card opens a shell.
This proves the bidirectional streaming-over-channel mechanism the RemoteDriver
will reuse. Remaining P2: RemoteDriver + placement (run agent workloads on nodes).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
2bdd0a23e8 |
Fleet P0: node registry + daemon + health + connect-host wizard
Users can connect their own local-hardware nodes into a fleet. Each node runs a
new Rust daemon that dials home over an outbound WebSocket, reports host health,
and runs commands we send.
Backend:
- migrations/0018_fleet_nodes.sql: nodes + node_health tables + agent_containers
(node_id, workspace_id) index. cm-domain NodeId.
- cm-db repo/nodes.rs: create/auth/list+health/get/heartbeat/set_status/delete
(unchecked sqlx, no .sqlx regen).
- cm-api fleet.rs NodeHub: live daemon channels (node_id→sender) + the WS channel
runner (heartbeat→DB upsert, exec request/response framing). routes/nodes.rs:
POST /pair, GET /nodes, SSE /nodes/live, POST /{id}/exec-test, DELETE /{id},
WS /nodes/agent (token-auth). Wired into AppState + router.
Daemon (new crate crates/bins/clawmates-node):
- sysinfo host metrics (cpu/mem/pressure/swap/disk/load/containers), outbound WSS
dial + reconnect, heartbeat loop, exec command handling, tailscale-ip probe.
install.sh convenience installer.
Frontend:
- Fleet sidebar item + FleetOverview + LocalHardware node-health cards (live via
/api/nodes, 3s poll) + ConnectHostWizard (install → verify connection →
exec-test). InfraStage dispatches fleet/local; default selection = fleet.
Deferred: P1 (BYO Tailscale + network metrics), P2 (RemoteDriver + placement so
agents actually run on connected nodes).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|