0785ac9c793fa5a74bf4dac5fe270c2d519a3367
18
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar Sobh
|
d8c8793c4a |
ci fixes: cargo fmt, eslint entities, max-lines split
CI on
|
||
|
Omar Sobh
|
bf4af48c80 |
herdr phase 3: INFRA tier Herdr sessions surface
New INFRA category "Herdr sessions" (purple sparkles icon between
Fleet and Local hardware). Shows a card per online fleet node with:
- Node name + hostname + IP
- Per-workspace agent state pills (working / blocked / done /
idle / unknown), colored dots + pane count
- "Open" button → renders that node's full Herdr TUI inline via
xterm.js (same nodeHerdrConnector + WebRTC-with-fallback the
MissionCanvas Live Pane uses)
Backend:
- node daemon: herdr_workspaces + herdr_snapshot ops
(`herdr workspace list`, `herdr api snapshot`)
- fleet_herdr::snapshot helper on top of hub.call_timeout
- GET /api/nodes/{id}/herdr/session route
Fetch flow: /api/nodes filtered to status='online' → for each,
/api/nodes/{id}/herdr/session in parallel. Snapshot errors surface
per-card without failing the whole grid.
The "Open" xterm is separate from the MissionCanvas Live Pane —
this one is scoped to the whole node's Herdr TUI (any workspace),
not a specific mission's pane. Operator toggles between nodes via
the buttons.
Verified: cargo check --workspace + tsc --noEmit both green.
|
||
|
Omar Sobh
|
a5588b0289 |
herdr phase 2: Live Pane tab (xterm.js → node's herdr TUI)
The killer UX feature: click a mission's Live Pane tab and watch the
actual Herdr TUI on the target node in the browser — cursor, colors,
tool output, all live. WebRTC DataChannel direct where the browser
can reach the node peer-to-peer, WS-relayed fallback otherwise
(same auto-negotiation the INFRA node terminal already uses).
Zero new deployment infra — reuses the existing terminal_ticket +
terminal_ws + PTY-over-control-channel machinery. The one primitive
we grew: PtyTarget::Command variant so the node can spawn an
arbitrary program (\`herdr\`) in the PTY instead of the login shell.
Node daemon (clawmates-node):
- PtyTarget grows a Command { argv } variant
- spawn_command_pty resolves bare names against user + system bin
dirs (matches how tool_update finds claude/kimi)
- PtyTarget::from_frame reads the `command` array from the pty_open
frame; precedence Command > Container > Host
cm-api:
- NodeHub::open_pty grows an optional command argv; when set, the
frame carries it and the daemon spawns the program directly.
- routes::nodes::TermCtrl gains a `command: Vec<String>`; the
fallback branch threads it through.
Frontend:
- core.ts::webrtcConnector takes an optional commandOverride
that ships inside the fallback frame
- nodeHerdrConnector(nodeId) — mints the standard ticket + WS URL
but overrides command to ["herdr"]
- MissionCanvas grows a "pane" tab, visible only when
runtime_kind='local_herdr'. LivePane subcomponent uses xterm.js
(already a workspace dep) via useResilientTerminal, shows a
connecting/relayed/direct pill in the corner.
To watch a mission live: pick "On a fleet node (Herdr)" + target
node in the wizard, launch, click Pane tab → node's Herdr TUI
appears. Navigate to the mission workspace in the Herdr sidebar
(mouse or prefix+w) to zoom into the mission's pane.
Focus-a-specific-pane-directly is a later enhancement — Herdr has
no CLI arg for it yet, so operator navigates the sidebar for now.
Verified: cargo check --workspace + tsc --noEmit both green.
|
||
|
Omar SobhandClaude Opus 4.8
|
1a531e91cd |
Fleet tools: add Rust version card (probe rustc + nightly latest + rustup update)
Adds Rust to the per-node dev-tool cards: daemon probes rustc → reports 'rust'; nightly checker fetches latest stable from GitHub rust-lang/rust; GET endpoint maps rust→Rust (after docker); one-click update runs 'rustup update stable'. Frontend is data-driven (no change). $HOME/.cargo/bin added to probe candidate dirs. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
ed339c2121 |
Fleet tools: one-click per-node update (Phase 2)
The ↑ badge on each tool card is now a button: confirm → POST
/api/nodes/{id}/tools/{tool}/update → daemon runs the tool's own updater + re-probes.
- daemon: tool_update op (spawned task so the 170s update can't stall the WS loop;
re-probes + re-sends node_tools after). Fixed command allow-list (no arbitrary
shell): claude/glm → `claude update`; kimi → `uv tool upgrade kimi-cli`; ollama →
brew upgrade (mac) / install.sh (linux); else unsupported. 4KB output cap.
- cm-api: call_timeout/request_timeout (long ops); POST .../tools/{tool}/update
(workspace-scoped, allow-list) → {ok,output}.
- frontend: ↑latest becomes an Update button → confirm → spinner → refresh/err.
Note: claude/kimi/glm are user-space (no sudo); ollama on Linux uses install.sh
(needs sudo — works on passwordless nodes, returns an error otherwise; surfaced in UI).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
9263418fcb |
Fleet: per-node dev-tool version cards + nightly latest-check (Phase 1, read-only)
Each node card now shows installed versions of Docker / Claude Code / Kimi / GLM /
Ollama (conditional per node) under the ssh card, with an "update available" badge.
- daemon: probe_tools() finds docker/claude/kimi-cli/ollama across candidate bin dirs,
extracts semver from --version, reports {"t":"node_tools",...} on connect + every 15m.
- migration node_tools + tool_latest; cm-db repo node_tools (upsert/list/latest).
- cm-api: fleet.rs NodeTools uplink → upsert; tool_versions.rs spawn_latest_checker
(24h, npm/pypi/github; docker display-only); GET /api/nodes/{id}/tools (glm mirrors
claude). Spawned in clawmates-server.
- frontend: NodeTools cards on each HostCard with the ↑latest badge.
Phase 2 (one-click update execution) intentionally deferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
3554a3aaf2 |
CI: remove k8s stages, fix the Docker-level pipeline green
Survey + fixes so the pipeline passes at the Docker level (no k8s).
- Remove k8s: drop the `sandbox-k8s` job (kind/Calico/--features k8s-tests) and the
"Helm chart lints" gate step. release.yml was already k8s-clean.
- Rust job:
- `cargo fmt --all` — fix pre-existing formatting drift (fmt --check was failing).
- clippy -D warnings: fix 3 lib warnings (cm-brain sort_by_key→Reverse, cm-api
fleet.rs doc list indentation, node_rules map_or→is_none_or).
- Regenerate the .sqlx offline cache (was missing the cm-runtime run_loop test
query → offline compile failed). DB-backed tests use testcontainers at runtime.
- Set SQLX_OFFLINE=true on the rust + e2e jobs so query! macros compile against
the committed cache deterministically (no DB needed at compile time).
- Frontend job:
- Fix the 1 ESLint error (useAgentTelemetry: no setState-synchronously-in-effect;
tag the slice with agentId + derive null on mismatch).
- Fix 2 stale panel-params tests (`terminal` is a valid app id now; assert the
current APP_IDS + use a genuinely-unknown id for the reject case).
Verified locally: fmt clean, clippy --all-targets -D warnings clean (offline),
frontend lint 0 errors, tsc clean, 86/86 frontend tests pass, build OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
10c89f5157 |
Node-placed agent terminal: container PTY on the agent's node + WebRTC, shared node-local drives
Completes "agent on a node" (single-node): when an agent's placement points at a fleet node, its terminal container runs there and the browser reaches it over a direct WebRTC DataChannel (LAN speed), sharing a node-local volume with the sandbox. gw-04-local agents are byte-identical to before. - cm-sandbox/docker.rs: empty drive subpath → mount the whole volume at the target (volume_options None), so a per-agent node-local volume auto-creates at ~/drives. - cm-api/fleet.rs: NodeHub.open_pty/webrtc_offer carry optional container+session (injected only when Some); node-terminal caller passes None (host shell unchanged). - cm-runtime/terminals.rs: TerminalManager gains node_provider + placement (mirrors SandboxManager, draining-aware); node_local_drive_mount(agent) = clawmates_agent_<id> at ~/drives; placement_for() ensures + locates the container; attach uses driver_for(node) (local byte-identical). - cm-runtime/sandboxes.rs: a node-placed agent sandbox mounts the same per-agent volume → shares files with the terminal on that node. - cm-api/routes/terminal.rs: ticket response gains `node`; ws() bridges node-placed agents through the NodeHub relay (WebRTC + fallback) execing into the container; local path unchanged. server main wires with_node_provider. - frontend: agentTerminalConnector mints the ticket then picks WebRTC (node-placed, ⚡ direct / relayed badge) vs WS (local); webrtcConnector generalized to be endpoint-agnostic (node terminal reuses it). Known follow-up: terminal (uid 65532) and sandbox (uid 10001) share the volume but differ in uid — cross-container writes need an aligned uid/gid (group-writable). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
36a227566b |
Fleet: Beszel hub integration — rich per-node metrics + per-node monitor (Phase 1)
Tap each node's Beszel metrics (GPU/temps/disk-IO/network/per-container — beyond
our basic heartbeat) by reading the workspace's Beszel hub. The agents run in
WS-only mode with no locally-readable socket, so (per the de-risk) the server taps
the hub's PocketBase API instead of the daemon reading agents — no daemon changes.
- migrations: workspace_beszel (BYO hub URL + login, server-side only, mirrors the
Tailscale BYO pattern) + node_metrics (latest scalar columns + JSONB blob).
- cm-db: repo/fleet_beszel.rs, repo/node_metrics.rs; nodes SELECT joins node_metrics
(gpu_pct/temp_max surfaced on node_json for the live cards).
- cm-api: beszel.rs client (auth-with-password, poll `systems`, map to nodes by
hostname, upsert metrics) + a 15s spawn_poller; routes/beszel.rs (connect/status/
disconnect + GET /api/nodes/{id}/metrics with history proxied live from the hub).
- frontend: HostCard gains a GPU/temp readout + a Monitor button; NodeMonitor is a
full-width per-node page (current panel + CPU/mem/GPU/temp/net/disk charts from the
hub's 1m history); a "Beszel monitoring" connect form in the Local view.
Reachability confirmed: gw-04 → the hub over the tailnet (100.123.224.84:8090). Needs
the user to connect their hub login to activate the poller.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
4de2f31b50 |
Fleet terminal: WebRTC DataChannel direct path (low-latency) + WS fallback
Terminal keystrokes were ~400ms because every byte relayed browser→Cloudflare→
gw-04 (Europe)→tailscale→node, even when the node is on the user's own LAN. Add a
direct browser↔node WebRTC DataChannel so co-located terminals run at LAN speed;
the gateway is reduced to signaling; the WebSocket relay stays as the automatic
fallback (graceful degradation — never worse than before).
Daemon (clawmates-node v0.4.0, new src/rtc.rs):
- Add the `webrtc` crate (reuses the ring crypto provider we already install — no
conflict). Browser is the offerer; we answer, trickle ICE back over the control
channel, and on DataChannel open spawn a host PTY (tmux) bridged DIRECTLY to the
channel. Refactor open_pty → spawn_terminal_pty shared by both transports.
iceServers: STUN + auto host/tailnet candidates (direct, no relay, for LAN/tailnet).
Server (cm-api):
- NodeConn.signal_sinks; Uplink WebRtcAnswer/WebRtcIce/WebRtcFailed routed to the
browser; NodeHub webrtc_offer/ice/close + open_session/open_pty (open_terminal
split so the PTY opens only once the transport is chosen). bridge_terminal relays
signaling over the existing ticket-authed WS and opens the relay PTY on
{type:"fallback"}.
Browser (NodeTerminalApp):
- RTCPeerConnection + reliable/ordered DataChannel; offer/answer/ICE over the WS;
2.5s race → use the DataChannel if it opens, else fall back to the WS relay.
Reconnect wraps both. A direct⚡/relayed indicator shows the live transport.
Deployed; both nodes (morpheus, tank) updated to v0.4.0 and steady online. Direct-
path proof is a browser action (the ⚡ indicator + latency); confirmable from the
daemon's [rtc] logs.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
94828ed887 |
Fleet: robust real-time connectivity + mosh-inspired reconnecting terminal
Nodes flapped online/offline and the terminal died on the first blip. WebSockets are the right transport (outbound, NAT-friendly); the fixes harden around it. Server (cm-api): - Anti-clobber connection epoch: a reconnecting daemon gets a fresh epoch; a stale run_channel's teardown only clears the hub + sets offline if it still owns the slot — so a lingering old channel can't flip a live reconnection offline (the main false-offline cause). - WS keepalive: run_channel now pings every 15s and tears down if no inbound frame (incl. pong) for 35s — dead links detected in seconds, not minutes. - Staleness sweeper backstop: spawn_node_sweeper (8s tick / 20s window) wired in clawmates-server, so a vanished node goes offline within ~28s even if its channel hangs (mark_stale_offline was defined but never called). Daemon (clawmates-node v0.3.0): - Heartbeats off the select thread (dedicated thread owns System + blocking docker/tailscale/disk CLIs) so a slow op never starves heartbeats/pongs. - Each handle_frame runs on its own task; added a 40s inbound idle deadline so a half-open socket triggers a reconnect. Frontend: - useNodes streams /api/nodes/live (SSE push) instead of a 3s poll; isLive() derives online from lastSeen freshness (<15s) so a transient column flip never shows a healthy node down. - Node terminal: clean auto-reconnect loop (re-mint ticket -> reconnect -> tmux re-attaches and redraws the live screen = mosh-style snap-to-state over TCP), replacing the [disconnected] dead-end. Mosh evaluated: harvest principles (session/transport decoupling, snap-to-state, already given by tmux), don't adopt — UDP is incompatible with our browser+CF+NAT topology and it's GPLv3. Removed temporary terminal debug traces + /api/debug route. Verified: node holds steadily online (heartbeat 1-3s, no flap) and goes cleanly offline when the daemon stops. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
27f5d05f96 |
Fleet terminal: daemon self-diagnosis + immediate banner + debug route
The server trace showed pty_open is sent but the daemon (morpheus, v0.2.0) emits
no pty_out — so the PTY spawn was dying silently. Instrument it:
- daemon open_pty: log open/tmux/first-read/EOF/error/total to stdout, and send
an IMMEDIATE banner pty_out ("[clawmates] host shell on <host> — starting…") so
the browser confirms the relay even before the shell draws. If open_pty fails,
send the error as pty_out (was a silent pty_exit). Bump to v0.2.1.
- cm-api: temp GET /api/debug/node-pty/{id}?dbg=… opens a node terminal and reads
~2s of output with no browser/auth, to test the relay in isolation.
- NodeTerminalApp + ticket/ws routes already log each hop ([node-term]/[fleet-term]).
Diagnostic logic: banner shows + shell doesn't → relay ok, shell is the problem;
nothing shows → relay broken; daemon "EOF after N bytes" → shell exited.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
c2a0309ad7 |
clawmates-node --selftest + server terminal tracing (diagnose blank terminal)
- Daemon: factor the terminal command into terminal_command(); add `--selftest` which opens the host terminal PTY locally and prints ~2.5s of raw output, so you can confirm tmux/zsh actually draws on a given node without the browser. (Verified locally: tmux spawns zsh + draws its status bar.) - cm-api: temporary [fleet-term] eprintln tracing in open_terminal, the pty_out router, and the browser bridge (byte counts + sink presence) to locate where output stops between daemon→server→browser. To be removed once diagnosed. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
cf6c331b02 |
Fleet: node hostname/IP on register + node terminal in the infra computer
Hostname/IP: - Daemon reports the machine's hostname (sysinfo) + primary outbound IPv4 on each heartbeat. migrations/0021 adds nodes.hostname/local_ip; cm-db heartbeat stores them; node JSON exposes them. Cards now title on the real hostname (falling back to name) + show the IP, instead of the "New node" placeholder. `name` stays user-overridable (rename). Terminal moved into the pull-out computer (no more per-card modal): - New infra computer app NodeTerminalApp (computer/apps/infra) — xterm bridged to a node's host shell over the node control channel, filling the app window (mirrors the agent Terminal's layout + ResizeObserver). Added "terminal" to the INFRA_CATALOG grid; a ?node= panel param targets a specific node (picker when unset). Clicking Terminal on a node card now opens the infra computer to that node's shell instead of a separate full-screen window. Deleted NodeTerminal.tsx. Rebuilt + re-hosted both daemon binaries (hostname change). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
33aa9c0693 |
Fleet P2b: node sandbox-readiness check (hardened workload on a node)
Proves a connected node can host hardened agent workloads end-to-end, without
touching the agent run loop (zero blast radius on existing agents).
- Daemon: typed `sb_check` op — pulls a tiny image and runs it fully locked down
(cap-drop ALL, no-new-privileges, no network, read-only rootfs, non-root,
memory/pids caps), then tears it down. Fixed command; nothing caller-supplied
runs (preserves the exec-hardening invariant).
- cm-api: NodeHub.sandbox_check + POST /api/nodes/{id}/sandbox-check.
- UI: a shield "sandbox check" button on each online node card streams the
result (✓ SANDBOX READY + container id/uname).
This validates the full provision→run→destroy mechanism on nodes. The remaining
P2 work — wiring real agent deploys to auto-place onto nodes — is its own
subsystem (a RemoteDriver reusing the local DockerDriver for security parity,
agent-image distribution to nodes, and node-routing in SandboxManager) and is
best done as a focused pass; it is intentionally NOT bundled here to keep the
core agent path untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
f5f96508eb |
Fleet P2a: in-dashboard remote terminal (PTY over the WSS channel)
You can now open a real shell on any connected node from the dashboard — the
daemon spawns a host PTY and streams it over the existing outbound control
channel (no inbound port, no Tailscale brokering needed).
Daemon:
- portable-pty host shell sessions: pty_open/pty_in/pty_resize/pty_close ops; a
reader thread streams base64 pty_out frames. Outbound frames now funnel through
one mpsc channel so PTY output and heartbeats interleave.
cm-api NodeHub:
- per-connection pty_sinks + sid multiplexing; open_terminal/terminal_input/
terminal_resize/terminal_close; in-memory single-use terminal tickets (the
browser WS can't carry a bearer, and the session is instance-local anyway).
- routes/nodes.rs: POST /api/nodes/{id}/terminal/ticket + GET .../terminal/ws
(bridges browser xterm <-> node PTY: binary = keystrokes, text = resize).
Frontend:
- NodeTerminal xterm modal (reuses the agent Terminal's xterm setup); a Terminal
button on each online node card opens a shell.
This proves the bidirectional streaming-over-channel mechanism the RemoteDriver
will reuse. Remaining P2: RemoteDriver + placement (run agent workloads on nodes).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
7332d69f8a |
Fleet P1: BYO Tailscale + network metrics, Tailscale SSH, exec hardening
Security hardening: - The gateway no longer sends arbitrary shell to nodes. The WSS exec op is replaced by a typed `verify` op the daemon runs itself (fixed host+docker check); future container ops are typed too. cm-api NodeHub.verify() + the daemon's handle_command only dispatches vetted ops. BYO Tailscale: - migrations/0019_workspace_tailscale.sql + cm-db fleet_tailscale repo (store the user's Tailscale API key + tailnet, server-side only). - cm-api routes/tailscale.rs: POST/GET/DELETE /api/fleet/tailscale + GET /api/fleet/tailscale/devices (proxies api.tailscale.com device list). - Daemon: --tailscale-authkey → `tailscale up --authkey … --ssh` (enables Tailscale SSH for keyless user access); else `tailscale set --ssh=true`. Reports its tailscale IP (already). UI: - Fleet overview gains a Tailscale section: connect (key+tailnet) + live tailnet device status (online/last-seen/IP/os). Node cards show a copyable Tailscale SSH target (ssh <ip>). Remaining: P2 — RemoteDriver + placement (run agents on nodes) and the in-UI remote terminal (PTY proxied over the WSS channel). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
2bdd0a23e8 |
Fleet P0: node registry + daemon + health + connect-host wizard
Users can connect their own local-hardware nodes into a fleet. Each node runs a
new Rust daemon that dials home over an outbound WebSocket, reports host health,
and runs commands we send.
Backend:
- migrations/0018_fleet_nodes.sql: nodes + node_health tables + agent_containers
(node_id, workspace_id) index. cm-domain NodeId.
- cm-db repo/nodes.rs: create/auth/list+health/get/heartbeat/set_status/delete
(unchecked sqlx, no .sqlx regen).
- cm-api fleet.rs NodeHub: live daemon channels (node_id→sender) + the WS channel
runner (heartbeat→DB upsert, exec request/response framing). routes/nodes.rs:
POST /pair, GET /nodes, SSE /nodes/live, POST /{id}/exec-test, DELETE /{id},
WS /nodes/agent (token-auth). Wired into AppState + router.
Daemon (new crate crates/bins/clawmates-node):
- sysinfo host metrics (cpu/mem/pressure/swap/disk/load/containers), outbound WSS
dial + reconnect, heartbeat loop, exec command handling, tailscale-ip probe.
install.sh convenience installer.
Frontend:
- Fleet sidebar item + FleetOverview + LocalHardware node-health cards (live via
/api/nodes, 3s poll) + ConnectHostWizard (install → verify connection →
exec-test). InfraStage dispatches fleet/local; default selection = fleet.
Deferred: P1 (BYO Tailscale + network metrics), P2 (RemoteDriver + placement so
agents actually run on connected nodes).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|