Commit Graph
22 Commits
Author SHA1 Message Date
Omar SobhandClaude Opus 5 16cfc29074 fix(ui): the backend picker showed two options meaning the same thing
`default` is the generic `rootfs.ext4` and `claude` is the named one, and
`microvm_credential_for` gives them the identical contract — so the list came
back with both under the same label, and whichever a user picked they got the
same thing. Collapsed to the named one where it exists; the generic keeps a
label of its own for a fleet that only has that.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-09 15:52:30 -07:00
Omar SobhandClaude Opus 5 529497febb fix(placement): a composed graph needs every backend its nodes name
The full harness found it — 12 of 13 scenarios green, `roster` red:

  roster: the planner sized this mission at 2 member(s)              PASS
  roster: the approved roster is on the mission (2 nodes, composed)  PASS
  roster: this run added 1 line(s) for a 2-member roster             FAIL

  topology_runs.error: turn executor failed: node n1 in a microVM:
    vm_create failed: no rootfs for backend "canary-claude" on this node

The roster proposed `verifier@canary-claude`. Placement asked
`online_for_backend` about the MISSION's backend — `claude` — and architect
answered, holding `claude` and `local-ornith`. The graph's first node ran and
delivered, the second could not boot, and the mission finished half-done. The
question placement asked was true and insufficient.

A composed graph runs on ONE node, so that node needs every image its nodes ask
for. `required_backends` collects the mission's plus each
`config.roster.nodes[].attrs.backend`, and `online_for_backends` passes the
whole set to the same jsonb `@>` — containment already means "contains ALL of
these", so the query shape did not have to change, only what it was asked.

This is the failure mode the roster feature creates by existing: its entire
purpose is putting a verifier on a different provider, which is exactly what
makes one node insufficient. Nothing before the full suite had a reason to
exercise it — the composed scenario uses one backend for all five nodes.

`NoCapableNode` now names the set and says why one node must hold all of them.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-09 15:36:40 -07:00
Omar SobhandClaude Opus 5 c66c3c6377 feat(ui): the microVM path is reachable from the mission wizard
Everything built today — Firecracker missions, the four backends, the local GPU
one — was unreachable from the dashboard. The wizard offered `zeroclaw` and
`local_herdr` and nothing else, so a mission created in the UI could not be a
microVM mission at all, and `local-ornith`/`glm`/`kimi` were API-only. Testing
"our workflows in the UI" would have exercised none of it.

Adds the runtime option and a backend picker, fed by a new
`GET /api/fleet/backends` that returns `mission_roster::available_backends`
verbatim — the SAME list the roster planner is handed, not a second one. Its two
rules are both load-bearing and neither is visible from a node's capabilities
alone: the image must be built on an online node, and the backend must have a
credential contract. `agent-terminal` passes the first and fails the second —
bootable, with nothing for the agent inside to authenticate with — so offering
it would produce a mission that validates, launches, and dies at the agent turn.

Ids are deployment vocabulary, so the picker labels them: a user choosing
between `local-ornith` and `canary-claude` should not have to know which company
each one bills. An empty list says why (no rootfs built) instead of showing an
empty dropdown, and no node is chosen for a microVM mission because
`vm_placement` picks it per phase.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-09 15:09:07 -07:00
Omar SobhandClaude Opus 5 dc0443de34 feat(fleet): GET /api/fleet/capacity returns the scheduler's own survey
Pulled forward from the observability phase because the capacity harness
scenario needs it. A test that recomputed the slot arithmetic in bash would
drift from `vm_placement` and then agree with itself while the scheduler did
something else — the same shape as every silent-success bug in this codebase.

Returns `survey()` + `rank()` unmodified, and keeps `unfit` as its own list:
"the fleet is full" and "we could not read the fleet" send an operator to
different places, so they must not be summed into one number.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-09 05:11:21 -07:00
Omar Sobh d8c8793c4a ci fixes: cargo fmt, eslint entities, max-lines split
ci / gates (push) Successful in 8s
ci / frontend (push) Successful in 26s
ci / rust (push) Successful in 4m25s
ci / e2e (push) Skipped
ci / publish (push) Successful in 2m46s
CI on 6ffbe97 failed on two auto-fixable gates. Both fixed:

  * cargo fmt --all — rustfmt applied across the surface touched
    by the last ~20 commits (world.rs, security_scan.rs,
    routes/{missions,nodes,terminal}.rs, fleet_herdr.rs,
    mission_workspace.rs, benchmark_runner.rs, mission_refiner.rs,
    lib.rs, tests/mission_orchestrator.rs, cm-db/repo/{missions,teams}.rs,
    bins/clawmates-node/src/main.rs)
  * eslint apostrophe escapes in HerdrSessions + MissionWizard
  * eslint max-lines: extracted EditMissionModal + RefineDiffModal
    (each ~200 LoC) into their own files. MissionCanvas drops from
    1424 to 1026, comfortably under both the 1250 eslint cap and the
    1500 CI budget.

New files:
  frontend/src/components/dashboard/EditMissionModal.tsx  (211 LoC)
  frontend/src/components/dashboard/RefineDiffModal.tsx   (208 LoC)

Verified locally: cargo fmt --check clean, cargo check clean,
mission_orchestrator test 3/3 pass, tsc + eslint --quiet both silent.
2026-07-20 12:03:43 -07:00
Omar Sobh bf4af48c80 herdr phase 3: INFRA tier Herdr sessions surface
New INFRA category "Herdr sessions" (purple sparkles icon between
Fleet and Local hardware). Shows a card per online fleet node with:

  - Node name + hostname + IP
  - Per-workspace agent state pills (working / blocked / done /
    idle / unknown), colored dots + pane count
  - "Open" button → renders that node's full Herdr TUI inline via
    xterm.js (same nodeHerdrConnector + WebRTC-with-fallback the
    MissionCanvas Live Pane uses)

Backend:
  - node daemon: herdr_workspaces + herdr_snapshot ops
    (`herdr workspace list`, `herdr api snapshot`)
  - fleet_herdr::snapshot helper on top of hub.call_timeout
  - GET /api/nodes/{id}/herdr/session route

Fetch flow: /api/nodes filtered to status='online' → for each,
/api/nodes/{id}/herdr/session in parallel. Snapshot errors surface
per-card without failing the whole grid.

The "Open" xterm is separate from the MissionCanvas Live Pane —
this one is scoped to the whole node's Herdr TUI (any workspace),
not a specific mission's pane. Operator toggles between nodes via
the buttons.

Verified: cargo check --workspace + tsc --noEmit both green.
2026-07-20 11:30:52 -07:00
Omar Sobh a5588b0289 herdr phase 2: Live Pane tab (xterm.js → node's herdr TUI)
The killer UX feature: click a mission's Live Pane tab and watch the
actual Herdr TUI on the target node in the browser — cursor, colors,
tool output, all live. WebRTC DataChannel direct where the browser
can reach the node peer-to-peer, WS-relayed fallback otherwise
(same auto-negotiation the INFRA node terminal already uses).

Zero new deployment infra — reuses the existing terminal_ticket +
terminal_ws + PTY-over-control-channel machinery. The one primitive
we grew: PtyTarget::Command variant so the node can spawn an
arbitrary program (\`herdr\`) in the PTY instead of the login shell.

Node daemon (clawmates-node):
  - PtyTarget grows a Command { argv } variant
  - spawn_command_pty resolves bare names against user + system bin
    dirs (matches how tool_update finds claude/kimi)
  - PtyTarget::from_frame reads the `command` array from the pty_open
    frame; precedence Command > Container > Host

cm-api:
  - NodeHub::open_pty grows an optional command argv; when set, the
    frame carries it and the daemon spawns the program directly.
  - routes::nodes::TermCtrl gains a `command: Vec<String>`; the
    fallback branch threads it through.

Frontend:
  - core.ts::webrtcConnector takes an optional commandOverride
    that ships inside the fallback frame
  - nodeHerdrConnector(nodeId) — mints the standard ticket + WS URL
    but overrides command to ["herdr"]
  - MissionCanvas grows a "pane" tab, visible only when
    runtime_kind='local_herdr'. LivePane subcomponent uses xterm.js
    (already a workspace dep) via useResilientTerminal, shows a
    connecting/relayed/direct pill in the corner.

To watch a mission live: pick "On a fleet node (Herdr)" + target
node in the wizard, launch, click Pane tab → node's Herdr TUI
appears. Navigate to the mission workspace in the Herdr sidebar
(mouse or prefix+w) to zoom into the mission's pane.

Focus-a-specific-pane-directly is a later enhancement — Herdr has
no CLI arg for it yet, so operator navigates the sidebar for now.

Verified: cargo check --workspace + tsc --noEmit both green.
2026-07-20 10:58:45 -07:00
Omar SobhandClaude Opus 4.8 1a531e91cd Fleet tools: add Rust version card (probe rustc + nightly latest + rustup update)
ci / gates (push) Successful in 5s
ci / frontend (push) Successful in 24s
ci / rust (push) Failing after 2m39s
ci / e2e (push) Has been skipped
Adds Rust to the per-node dev-tool cards: daemon probes rustc → reports 'rust';
nightly checker fetches latest stable from GitHub rust-lang/rust; GET endpoint maps
rust→Rust (after docker); one-click update runs 'rustup update stable'. Frontend is
data-driven (no change). $HOME/.cargo/bin added to probe candidate dirs.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-27 13:04:16 -07:00
Omar SobhandClaude Opus 4.8 ed339c2121 Fleet tools: one-click per-node update (Phase 2)
ci / gates (push) Successful in 5s
ci / rust (push) Failing after 7s
ci / frontend (push) Successful in 23s
ci / e2e (push) Has been skipped
The ↑ badge on each tool card is now a button: confirm → POST
/api/nodes/{id}/tools/{tool}/update → daemon runs the tool's own updater + re-probes.

- daemon: tool_update op (spawned task so the 170s update can't stall the WS loop;
  re-probes + re-sends node_tools after). Fixed command allow-list (no arbitrary
  shell): claude/glm → `claude update`; kimi → `uv tool upgrade kimi-cli`; ollama →
  brew upgrade (mac) / install.sh (linux); else unsupported. 4KB output cap.
- cm-api: call_timeout/request_timeout (long ops); POST .../tools/{tool}/update
  (workspace-scoped, allow-list) → {ok,output}.
- frontend: ↑latest becomes an Update button → confirm → spinner → refresh/err.

Note: claude/kimi/glm are user-space (no sudo); ollama on Linux uses install.sh
(needs sudo — works on passwordless nodes, returns an error otherwise; surfaced in UI).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-27 06:25:30 -07:00
Omar SobhandClaude Opus 4.8 9263418fcb Fleet: per-node dev-tool version cards + nightly latest-check (Phase 1, read-only)
ci / gates (push) Successful in 5s
ci / rust (push) Failing after 7s
ci / frontend (push) Successful in 23s
ci / e2e (push) Has been skipped
Each node card now shows installed versions of Docker / Claude Code / Kimi / GLM /
Ollama (conditional per node) under the ssh card, with an "update available" badge.

- daemon: probe_tools() finds docker/claude/kimi-cli/ollama across candidate bin dirs,
  extracts semver from --version, reports {"t":"node_tools",...} on connect + every 15m.
- migration node_tools + tool_latest; cm-db repo node_tools (upsert/list/latest).
- cm-api: fleet.rs NodeTools uplink → upsert; tool_versions.rs spawn_latest_checker
  (24h, npm/pypi/github; docker display-only); GET /api/nodes/{id}/tools (glm mirrors
  claude). Spawned in clawmates-server.
- frontend: NodeTools cards on each HostCard with the ↑latest badge.

Phase 2 (one-click update execution) intentionally deferred.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-27 06:01:51 -07:00
Omar SobhandClaude Opus 4.8 3554a3aaf2 CI: remove k8s stages, fix the Docker-level pipeline green
ci / frontend (push) Successful in 23s
ci / rust (push) Failing after 27s
ci / e2e (push) Has been skipped
ci / gates (push) Successful in 5s
Survey + fixes so the pipeline passes at the Docker level (no k8s).

- Remove k8s: drop the `sandbox-k8s` job (kind/Calico/--features k8s-tests) and the
  "Helm chart lints" gate step. release.yml was already k8s-clean.
- Rust job:
  - `cargo fmt --all` — fix pre-existing formatting drift (fmt --check was failing).
  - clippy -D warnings: fix 3 lib warnings (cm-brain sort_by_key→Reverse, cm-api
    fleet.rs doc list indentation, node_rules map_or→is_none_or).
  - Regenerate the .sqlx offline cache (was missing the cm-runtime run_loop test
    query → offline compile failed). DB-backed tests use testcontainers at runtime.
  - Set SQLX_OFFLINE=true on the rust + e2e jobs so query! macros compile against
    the committed cache deterministically (no DB needed at compile time).
- Frontend job:
  - Fix the 1 ESLint error (useAgentTelemetry: no setState-synchronously-in-effect;
    tag the slice with agentId + derive null on mismatch).
  - Fix 2 stale panel-params tests (`terminal` is a valid app id now; assert the
    current APP_IDS + use a genuinely-unknown id for the reject case).

Verified locally: fmt clean, clippy --all-targets -D warnings clean (offline),
frontend lint 0 errors, tsc clean, 86/86 frontend tests pass, build OK.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-26 18:15:31 -07:00
Omar SobhandClaude Opus 4.8 10c89f5157 Node-placed agent terminal: container PTY on the agent's node + WebRTC, shared node-local drives
Completes "agent on a node" (single-node): when an agent's placement points at a
fleet node, its terminal container runs there and the browser reaches it over a
direct WebRTC DataChannel (LAN speed), sharing a node-local volume with the
sandbox. gw-04-local agents are byte-identical to before.

- cm-sandbox/docker.rs: empty drive subpath → mount the whole volume at the target
  (volume_options None), so a per-agent node-local volume auto-creates at ~/drives.
- cm-api/fleet.rs: NodeHub.open_pty/webrtc_offer carry optional container+session
  (injected only when Some); node-terminal caller passes None (host shell unchanged).
- cm-runtime/terminals.rs: TerminalManager gains node_provider + placement
  (mirrors SandboxManager, draining-aware); node_local_drive_mount(agent) =
  clawmates_agent_<id> at ~/drives; placement_for() ensures + locates the container;
  attach uses driver_for(node) (local byte-identical).
- cm-runtime/sandboxes.rs: a node-placed agent sandbox mounts the same per-agent
  volume → shares files with the terminal on that node.
- cm-api/routes/terminal.rs: ticket response gains `node`; ws() bridges node-placed
  agents through the NodeHub relay (WebRTC + fallback) execing into the container;
  local path unchanged. server main wires with_node_provider.
- frontend: agentTerminalConnector mints the ticket then picks WebRTC (node-placed,
   direct / relayed badge) vs WS (local); webrtcConnector generalized to be
  endpoint-agnostic (node terminal reuses it).

Known follow-up: terminal (uid 65532) and sandbox (uid 10001) share the volume but
differ in uid — cross-container writes need an aligned uid/gid (group-writable).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-26 09:11:18 -07:00
Omar SobhandClaude Opus 4.8 36a227566b Fleet: Beszel hub integration — rich per-node metrics + per-node monitor (Phase 1)
ci / gates (push) Failing after 5s
ci / rust (push) Has been skipped
ci / sandbox-k8s (push) Has been skipped
ci / frontend (push) Has been skipped
ci / e2e (push) Has been skipped
Tap each node's Beszel metrics (GPU/temps/disk-IO/network/per-container — beyond
our basic heartbeat) by reading the workspace's Beszel hub. The agents run in
WS-only mode with no locally-readable socket, so (per the de-risk) the server taps
the hub's PocketBase API instead of the daemon reading agents — no daemon changes.

- migrations: workspace_beszel (BYO hub URL + login, server-side only, mirrors the
  Tailscale BYO pattern) + node_metrics (latest scalar columns + JSONB blob).
- cm-db: repo/fleet_beszel.rs, repo/node_metrics.rs; nodes SELECT joins node_metrics
  (gpu_pct/temp_max surfaced on node_json for the live cards).
- cm-api: beszel.rs client (auth-with-password, poll `systems`, map to nodes by
  hostname, upsert metrics) + a 15s spawn_poller; routes/beszel.rs (connect/status/
  disconnect + GET /api/nodes/{id}/metrics with history proxied live from the hub).
- frontend: HostCard gains a GPU/temp readout + a Monitor button; NodeMonitor is a
  full-width per-node page (current panel + CPU/mem/GPU/temp/net/disk charts from the
  hub's 1m history); a "Beszel monitoring" connect form in the Local view.

Reachability confirmed: gw-04 → the hub over the tailnet (100.123.224.84:8090). Needs
the user to connect their hub login to activate the poller.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-25 23:28:29 -07:00
Omar SobhandClaude Opus 4.8 4de2f31b50 Fleet terminal: WebRTC DataChannel direct path (low-latency) + WS fallback
ci / gates (push) Failing after 5s
ci / rust (push) Has been skipped
ci / sandbox-k8s (push) Has been skipped
ci / frontend (push) Has been skipped
ci / e2e (push) Has been skipped
Terminal keystrokes were ~400ms because every byte relayed browser→Cloudflare→
gw-04 (Europe)→tailscale→node, even when the node is on the user's own LAN. Add a
direct browser↔node WebRTC DataChannel so co-located terminals run at LAN speed;
the gateway is reduced to signaling; the WebSocket relay stays as the automatic
fallback (graceful degradation — never worse than before).

Daemon (clawmates-node v0.4.0, new src/rtc.rs):
- Add the `webrtc` crate (reuses the ring crypto provider we already install — no
  conflict). Browser is the offerer; we answer, trickle ICE back over the control
  channel, and on DataChannel open spawn a host PTY (tmux) bridged DIRECTLY to the
  channel. Refactor open_pty → spawn_terminal_pty shared by both transports.
  iceServers: STUN + auto host/tailnet candidates (direct, no relay, for LAN/tailnet).

Server (cm-api):
- NodeConn.signal_sinks; Uplink WebRtcAnswer/WebRtcIce/WebRtcFailed routed to the
  browser; NodeHub webrtc_offer/ice/close + open_session/open_pty (open_terminal
  split so the PTY opens only once the transport is chosen). bridge_terminal relays
  signaling over the existing ticket-authed WS and opens the relay PTY on
  {type:"fallback"}.

Browser (NodeTerminalApp):
- RTCPeerConnection + reliable/ordered DataChannel; offer/answer/ICE over the WS;
  2.5s race → use the DataChannel if it opens, else fall back to the WS relay.
  Reconnect wraps both. A direct/relayed indicator shows the live transport.

Deployed; both nodes (morpheus, tank) updated to v0.4.0 and steady online. Direct-
path proof is a browser action (the  indicator + latency); confirmable from the
daemon's [rtc] logs.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-25 16:55:32 -07:00
Omar SobhandClaude Opus 4.8 94828ed887 Fleet: robust real-time connectivity + mosh-inspired reconnecting terminal
ci / gates (push) Failing after 6s
ci / frontend (push) Has been skipped
ci / rust (push) Has been skipped
ci / sandbox-k8s (push) Has been skipped
ci / e2e (push) Has been skipped
Nodes flapped online/offline and the terminal died on the first blip. WebSockets
are the right transport (outbound, NAT-friendly); the fixes harden around it.

Server (cm-api):
- Anti-clobber connection epoch: a reconnecting daemon gets a fresh epoch; a stale
  run_channel's teardown only clears the hub + sets offline if it still owns the
  slot — so a lingering old channel can't flip a live reconnection offline (the
  main false-offline cause).
- WS keepalive: run_channel now pings every 15s and tears down if no inbound
  frame (incl. pong) for 35s — dead links detected in seconds, not minutes.
- Staleness sweeper backstop: spawn_node_sweeper (8s tick / 20s window) wired in
  clawmates-server, so a vanished node goes offline within ~28s even if its
  channel hangs (mark_stale_offline was defined but never called).

Daemon (clawmates-node v0.3.0):
- Heartbeats off the select thread (dedicated thread owns System + blocking
  docker/tailscale/disk CLIs) so a slow op never starves heartbeats/pongs.
- Each handle_frame runs on its own task; added a 40s inbound idle deadline so a
  half-open socket triggers a reconnect.

Frontend:
- useNodes streams /api/nodes/live (SSE push) instead of a 3s poll; isLive()
  derives online from lastSeen freshness (<15s) so a transient column flip never
  shows a healthy node down.
- Node terminal: clean auto-reconnect loop (re-mint ticket -> reconnect -> tmux
  re-attaches and redraws the live screen = mosh-style snap-to-state over TCP),
  replacing the [disconnected] dead-end.

Mosh evaluated: harvest principles (session/transport decoupling, snap-to-state,
already given by tmux), don't adopt — UDP is incompatible with our browser+CF+NAT
topology and it's GPLv3. Removed temporary terminal debug traces + /api/debug route.

Verified: node holds steadily online (heartbeat 1-3s, no flap) and goes cleanly
offline when the daemon stops.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-25 07:18:05 -07:00
Omar SobhandClaude Opus 4.8 27f5d05f96 Fleet terminal: daemon self-diagnosis + immediate banner + debug route
ci / gates (push) Failing after 5s
ci / rust (push) Has been skipped
ci / sandbox-k8s (push) Has been skipped
ci / frontend (push) Has been skipped
ci / e2e (push) Has been skipped
The server trace showed pty_open is sent but the daemon (morpheus, v0.2.0) emits
no pty_out — so the PTY spawn was dying silently. Instrument it:
- daemon open_pty: log open/tmux/first-read/EOF/error/total to stdout, and send
  an IMMEDIATE banner pty_out ("[clawmates] host shell on <host> — starting…") so
  the browser confirms the relay even before the shell draws. If open_pty fails,
  send the error as pty_out (was a silent pty_exit). Bump to v0.2.1.
- cm-api: temp GET /api/debug/node-pty/{id}?dbg=… opens a node terminal and reads
  ~2s of output with no browser/auth, to test the relay in isolation.
- NodeTerminalApp + ticket/ws routes already log each hop ([node-term]/[fleet-term]).

Diagnostic logic: banner shows + shell doesn't → relay ok, shell is the problem;
nothing shows → relay broken; daemon "EOF after N bytes" → shell exited.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-25 00:06:51 -07:00
Omar SobhandClaude Opus 4.8 c2a0309ad7 clawmates-node --selftest + server terminal tracing (diagnose blank terminal)
ci / gates (push) Failing after 5s
ci / rust (push) Has been skipped
ci / sandbox-k8s (push) Has been skipped
ci / frontend (push) Has been skipped
ci / e2e (push) Has been skipped
- Daemon: factor the terminal command into terminal_command(); add `--selftest`
  which opens the host terminal PTY locally and prints ~2.5s of raw output, so
  you can confirm tmux/zsh actually draws on a given node without the browser.
  (Verified locally: tmux spawns zsh + draws its status bar.)
- cm-api: temporary [fleet-term] eprintln tracing in open_terminal, the pty_out
  router, and the browser bridge (byte counts + sink presence) to locate where
  output stops between daemon→server→browser. To be removed once diagnosed.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-24 21:28:26 -07:00
Omar SobhandClaude Opus 4.8 cf6c331b02 Fleet: node hostname/IP on register + node terminal in the infra computer
ci / gates (push) Failing after 5s
ci / rust (push) Has been skipped
ci / sandbox-k8s (push) Has been skipped
ci / frontend (push) Has been skipped
ci / e2e (push) Has been skipped
Hostname/IP:
- Daemon reports the machine's hostname (sysinfo) + primary outbound IPv4 on each
  heartbeat. migrations/0021 adds nodes.hostname/local_ip; cm-db heartbeat stores
  them; node JSON exposes them. Cards now title on the real hostname (falling back
  to name) + show the IP, instead of the "New node" placeholder. `name` stays
  user-overridable (rename).

Terminal moved into the pull-out computer (no more per-card modal):
- New infra computer app NodeTerminalApp (computer/apps/infra) — xterm bridged to
  a node's host shell over the node control channel, filling the app window
  (mirrors the agent Terminal's layout + ResizeObserver). Added "terminal" to the
  INFRA_CATALOG grid; a ?node= panel param targets a specific node (picker when
  unset). Clicking Terminal on a node card now opens the infra computer to that
  node's shell instead of a separate full-screen window. Deleted NodeTerminal.tsx.

Rebuilt + re-hosted both daemon binaries (hostname change).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-24 20:13:15 -07:00
Omar SobhandClaude Opus 4.8 33aa9c0693 Fleet P2b: node sandbox-readiness check (hardened workload on a node)
ci / gates (push) Failing after 5s
ci / rust (push) Has been skipped
ci / sandbox-k8s (push) Has been skipped
ci / frontend (push) Has been skipped
ci / e2e (push) Has been skipped
Proves a connected node can host hardened agent workloads end-to-end, without
touching the agent run loop (zero blast radius on existing agents).

- Daemon: typed `sb_check` op — pulls a tiny image and runs it fully locked down
  (cap-drop ALL, no-new-privileges, no network, read-only rootfs, non-root,
  memory/pids caps), then tears it down. Fixed command; nothing caller-supplied
  runs (preserves the exec-hardening invariant).
- cm-api: NodeHub.sandbox_check + POST /api/nodes/{id}/sandbox-check.
- UI: a shield "sandbox check" button on each online node card streams the
  result (✓ SANDBOX READY + container id/uname).

This validates the full provision→run→destroy mechanism on nodes. The remaining
P2 work — wiring real agent deploys to auto-place onto nodes — is its own
subsystem (a RemoteDriver reusing the local DockerDriver for security parity,
agent-image distribution to nodes, and node-routing in SandboxManager) and is
best done as a focused pass; it is intentionally NOT bundled here to keep the
core agent path untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-24 12:31:50 -07:00
Omar SobhandClaude Opus 4.8 f5f96508eb Fleet P2a: in-dashboard remote terminal (PTY over the WSS channel)
ci / frontend (push) Has been skipped
ci / e2e (push) Has been skipped
ci / gates (push) Failing after 5s
ci / rust (push) Has been skipped
ci / sandbox-k8s (push) Has been skipped
You can now open a real shell on any connected node from the dashboard — the
daemon spawns a host PTY and streams it over the existing outbound control
channel (no inbound port, no Tailscale brokering needed).

Daemon:
- portable-pty host shell sessions: pty_open/pty_in/pty_resize/pty_close ops; a
  reader thread streams base64 pty_out frames. Outbound frames now funnel through
  one mpsc channel so PTY output and heartbeats interleave.

cm-api NodeHub:
- per-connection pty_sinks + sid multiplexing; open_terminal/terminal_input/
  terminal_resize/terminal_close; in-memory single-use terminal tickets (the
  browser WS can't carry a bearer, and the session is instance-local anyway).
- routes/nodes.rs: POST /api/nodes/{id}/terminal/ticket + GET .../terminal/ws
  (bridges browser xterm <-> node PTY: binary = keystrokes, text = resize).

Frontend:
- NodeTerminal xterm modal (reuses the agent Terminal's xterm setup); a Terminal
  button on each online node card opens a shell.

This proves the bidirectional streaming-over-channel mechanism the RemoteDriver
will reuse. Remaining P2: RemoteDriver + placement (run agent workloads on nodes).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-24 12:13:24 -07:00
Omar SobhandClaude Opus 4.8 7332d69f8a Fleet P1: BYO Tailscale + network metrics, Tailscale SSH, exec hardening
ci / gates (push) Failing after 6s
ci / rust (push) Has been skipped
ci / sandbox-k8s (push) Has been skipped
ci / frontend (push) Has been skipped
ci / e2e (push) Has been skipped
Security hardening:
- The gateway no longer sends arbitrary shell to nodes. The WSS exec op is
  replaced by a typed `verify` op the daemon runs itself (fixed host+docker
  check); future container ops are typed too. cm-api NodeHub.verify() + the
  daemon's handle_command only dispatches vetted ops.

BYO Tailscale:
- migrations/0019_workspace_tailscale.sql + cm-db fleet_tailscale repo (store the
  user's Tailscale API key + tailnet, server-side only).
- cm-api routes/tailscale.rs: POST/GET/DELETE /api/fleet/tailscale + GET
  /api/fleet/tailscale/devices (proxies api.tailscale.com device list).
- Daemon: --tailscale-authkey → `tailscale up --authkey … --ssh` (enables
  Tailscale SSH for keyless user access); else `tailscale set --ssh=true`. Reports
  its tailscale IP (already).

UI:
- Fleet overview gains a Tailscale section: connect (key+tailnet) + live tailnet
  device status (online/last-seen/IP/os). Node cards show a copyable Tailscale SSH
  target (ssh <ip>).

Remaining: P2 — RemoteDriver + placement (run agents on nodes) and the in-UI
remote terminal (PTY proxied over the WSS channel).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-24 10:27:58 -07:00
Omar SobhandClaude Opus 4.8 2bdd0a23e8 Fleet P0: node registry + daemon + health + connect-host wizard
ci / gates (push) Failing after 5s
ci / rust (push) Has been skipped
ci / sandbox-k8s (push) Has been skipped
ci / frontend (push) Has been skipped
ci / e2e (push) Has been skipped
Users can connect their own local-hardware nodes into a fleet. Each node runs a
new Rust daemon that dials home over an outbound WebSocket, reports host health,
and runs commands we send.

Backend:
- migrations/0018_fleet_nodes.sql: nodes + node_health tables + agent_containers
  (node_id, workspace_id) index. cm-domain NodeId.
- cm-db repo/nodes.rs: create/auth/list+health/get/heartbeat/set_status/delete
  (unchecked sqlx, no .sqlx regen).
- cm-api fleet.rs NodeHub: live daemon channels (node_id→sender) + the WS channel
  runner (heartbeat→DB upsert, exec request/response framing). routes/nodes.rs:
  POST /pair, GET /nodes, SSE /nodes/live, POST /{id}/exec-test, DELETE /{id},
  WS /nodes/agent (token-auth). Wired into AppState + router.

Daemon (new crate crates/bins/clawmates-node):
- sysinfo host metrics (cpu/mem/pressure/swap/disk/load/containers), outbound WSS
  dial + reconnect, heartbeat loop, exec command handling, tailscale-ip probe.
  install.sh convenience installer.

Frontend:
- Fleet sidebar item + FleetOverview + LocalHardware node-health cards (live via
  /api/nodes, 3s poll) + ConnectHostWizard (install → verify connection →
  exec-test). InfraStage dispatches fleet/local; default selection = fleet.

Deferred: P1 (BYO Tailscale + network metrics), P2 (RemoteDriver + placement so
agents actually run on connected nodes).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-24 08:09:14 -07:00