Second commit of the Research + Loops arc. Builds on 0030 by lighting up
the container CRUD, the agent-slot attach/detach, the standby→processing
transition, and the wizard's one-shot LLM refine.
GET /api/research list workspace's topics
POST /api/research create (accepts wizard output)
GET /api/research/:id detail (topic + attached agents)
PATCH /api/research/:id update non-status fields
POST /api/research/:id/agents attach agent (idempotent)
DELETE /api/research/:id/agents/:agent detach
POST /api/research/:id/start standby → processing
POST /api/research/wizard/refine one-shot LLM refine
The refine endpoint accumulates the workspace's default LLM provider's
stream into a JSON object (`{title, description}`) with the tight system
prompt at the top of the module. Same provider routing as agent runs
(via runtime.provider()), so a workspace already using GLM/Kimi gets it
for free.
State machine's remaining transitions (processing → reviewing on last
run_completed; reviewing → publishing via approvals gate) land with the
orchestrator hookup + approvals extension. Publish approval and loops
are separate commits still to come.
Adds cm-llm as a direct cm-api dep (previously only pulled transitively
via cm-runtime) so the refine endpoint can build a ChatRequest. Uses
sqlx::query! for compile-time verification; .sqlx cache generated on
morpheus against a fresh migrated DB.
The a2a merge (d0d8e7f) landed with a handful of pre-existing rustfmt
diffs that were failing `cargo fmt --all --check` in CI. Pure whitespace
reformatting from `cargo fmt --all`; no semantic changes. Files touched:
mcp_door.rs, quota.rs, routes/a2a.rs, routes/world.rs, runtime_provision.rs,
chat_repos.rs test, and tools/chat.rs.
pty_sinks was an unbounded mpsc, which means a runaway PTY (say
`cat /var/log/huge`) with a browser that stopped consuming would
accumulate megabytes in the hub's memory indefinitely. Switching to a
bounded mpsc::Sender means the send fails when the buffer is full; the
NodeHub then closes that session cleanly instead of holding output
forever. Signal sinks stay unbounded — they carry small control frames.
Adds a Quota.max_active_runs ceiling (queued + running topology runs at
once, per workspace) to stop a single workspace flooding the shared queue.
Free tier: 10, Pro: 25, Team: 100. Enforced at every /run enqueue site:
run_org, run_company, run_team, and the webhook trigger. Webhooks return
429 rather than 402 so external callers can back off — the guard is what
stops a leaked webhook token from being weaponized into a queue flood.
A single team run also spawns a tier-tree of children, so the practical
cap grows with the topology — this counts the outer runs, not every step.
Brings a2a rooms, delegation, and A2A ingress work back into main. Prod's
DB has migrations 0026 (group_rooms) and 0027 (a2a) applied from an
earlier hand-tagged fleet21 build cut from this branch, but main never got
them — so the CI-built server image from main refused to start against
prod's DB with "migration 26 was previously applied but is missing".
Landing the branch closes that gap: main + prod DB now share the same
migration state, so images built from main can safely roll onto gw-04.
Included:
- 0026_group_rooms.sql / 0027_a2a.sql — align main with prod's schema
- cm-api routes/a2a.rs + mcp_door.rs updates — A2A ingress and MCP door
- cm-runtime tools/delegate.rs + tools/chat.rs — delegation + N-way rooms
- frontend TeamObserver + FleetPanels + AgentObserver updates
- taxonomy.ts — delegation + A2A signals in the live world feed
Not included (still WIP on the local checkout):
- 0028_backfill_on_delete.sql / 0029_hot_query_indexes.sql
- broker pool + team-run quota changes
- Dashboard.tsx UI rename (Large World → Visualizations)
- Node write.send timeout (aaab663) — separate concern
The SSE resume fix (202e853) is preserved by auto-merge — approvals.rs
still awaits resume_run inline so the channel is ready before reply.
The `decide()` helper (used by both approve and reject) wrapped
`runtime.resume_run(ready).await` inside `tokio::spawn`, so the handler
returned 200 to the client before the resume had even started. When a
client (or test) immediately reconnected to /api/gateway with resumeFrom,
the run's broadcast channel didn't exist yet — `runtime.subscribe(run_id)`
returned None, the journal was still empty (no new events yet), and the
SSE stream closed with zero events. Symptom: approvals_api tests flaky in
CI (`unwrap on None` at line 282 of reject_over_http_executes_nothing,
sometimes `missing step_finished` on the accept path).
resume_run's *own* awaited portion is only the setup — claim_resume,
checkpoint load, and open_channel. It internally spawns the long-running
multi-step work. Awaiting it inline means we wait milliseconds for setup,
then return. By the time the client reconnects, the channel exists, so
subscribe() finds it and live-tail works. The sweeper still handles the
crash-between-decision-and-resume case for durability.
Edge-initiated inter-agent events (gated delegation, A2A ingress) bypass the
run loop, so the world SSE now polls the append-only audit log (cursor on the
BIGINT id, seeded to max on first pass) and emits:
- delegation.invoked -> agent.delegate {fromAgentId,toAgentId,toName,task}
- a2a.invoked -> a2a.invoked {agentId}
TeamObserver renders agent.delegate as an A->B handoff in the team timeline;
adds the agent.delegate taxonomy type. Reliable live (the synthesized-run path
never streamed — active_runs + first-sight cursor jump skip it).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Adds GET /api/a2a/settings (enabled + publicBaseUrl) and an A2ASection in the
Fleet overview: enable/disable A2A for the workspace, mint/list/revoke external
bearer tokens (token shown once), and the discovery URL. Claw/skill publishing
is configured per-claw (follow-up).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The agent command center's BRAIN+SURFACE columns are now a grid of compact icon
cards (system prompt / how it operates / personality / skills / capabilities /
tools / memory / safety). Clicking a card opens a modal with the full content,
nicely formatted (markdown/persona/tags), editable for the four brain text files
(system_prompt, agent_md, persona, skills_md) and saved back to the .brain via
PATCH /api/claws/{id}/brain. Capabilities/tools/memory/safety render read-only
(tools keeps its Add affordance). LIVE column unchanged.
- frontend: new AnatomyGrid (icon cards + SectionModal); ClawCommandCenter swaps the
two verbose columns for it; skills_md added to RawBrain types.
- backend: cm-brain skills_md() getter + skills_md in ClawBrainResponse so the skills
editor pre-fills (avoids blank-overwrite).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
A new "Observe" button (Eye) sits next to "+ New" in the agent chat header; clicking
flips the card into a read-only observer of that agent's conversations with other
agents, updating live as messages happen.
- frontend: AgentObserver (history from /api/claw-chat/* + live agent.message overlay
filtered to the agent, read-only banner, no composer); ClawChatSection toggle + flip.
- backend: emit a live agent.message run-event when chat.send succeeds — events.rs
AgentMessage variant, chat.send returns to_id, runtime emits in both tool paths,
world.rs normalizes agent_message → agent.message SSE. No migration, no new table.
Roadmap (not built): group/multi-party rooms; A2A protocol (a2a-rs) adoption.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The ↑ badge on each tool card is now a button: confirm → POST
/api/nodes/{id}/tools/{tool}/update → daemon runs the tool's own updater + re-probes.
- daemon: tool_update op (spawned task so the 170s update can't stall the WS loop;
re-probes + re-sends node_tools after). Fixed command allow-list (no arbitrary
shell): claude/glm → `claude update`; kimi → `uv tool upgrade kimi-cli`; ollama →
brew upgrade (mac) / install.sh (linux); else unsupported. 4KB output cap.
- cm-api: call_timeout/request_timeout (long ops); POST .../tools/{tool}/update
(workspace-scoped, allow-list) → {ok,output}.
- frontend: ↑latest becomes an Update button → confirm → spinner → refresh/err.
Note: claude/kimi/glm are user-space (no sudo); ollama on Linux uses install.sh
(needs sudo — works on passwordless nodes, returns an error otherwise; surfaced in UI).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Each node card now shows installed versions of Docker / Claude Code / Kimi / GLM /
Ollama (conditional per node) under the ssh card, with an "update available" badge.
- daemon: probe_tools() finds docker/claude/kimi-cli/ollama across candidate bin dirs,
extracts semver from --version, reports {"t":"node_tools",...} on connect + every 15m.
- migration node_tools + tool_latest; cm-db repo node_tools (upsert/list/latest).
- cm-api: fleet.rs NodeTools uplink → upsert; tool_versions.rs spawn_latest_checker
(24h, npm/pypi/github; docker display-only); GET /api/nodes/{id}/tools (glm mirrors
claude). Spawned in clawmates-server.
- frontend: NodeTools cards on each HostCard with the ↑latest badge.
Phase 2 (one-click update execution) intentionally deferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Survey + fixes so the pipeline passes at the Docker level (no k8s).
- Remove k8s: drop the `sandbox-k8s` job (kind/Calico/--features k8s-tests) and the
"Helm chart lints" gate step. release.yml was already k8s-clean.
- Rust job:
- `cargo fmt --all` — fix pre-existing formatting drift (fmt --check was failing).
- clippy -D warnings: fix 3 lib warnings (cm-brain sort_by_key→Reverse, cm-api
fleet.rs doc list indentation, node_rules map_or→is_none_or).
- Regenerate the .sqlx offline cache (was missing the cm-runtime run_loop test
query → offline compile failed). DB-backed tests use testcontainers at runtime.
- Set SQLX_OFFLINE=true on the rust + e2e jobs so query! macros compile against
the committed cache deterministically (no DB needed at compile time).
- Frontend job:
- Fix the 1 ESLint error (useAgentTelemetry: no setState-synchronously-in-effect;
tag the slice with agentId + derive null on mismatch).
- Fix 2 stale panel-params tests (`terminal` is a valid app id now; assert the
current APP_IDS + use a genuinely-unknown id for the reject case).
Verified locally: fmt clean, clippy --all-targets -D warnings clean (offline),
frontend lint 0 errors, tsc clean, 86/86 frontend tests pass, build OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add PATCH /api/claws/{id}/brain (edit_brain): writes any of system_prompt / agent_md
/ persona / skills_md into the claw's .brain (best-effort) + persists system_prompt
to Postgres (authoritative) + commits a ClawSync revision.
Frontend: a reusable EditableSection (pencil → textarea → Save/Cancel → PATCH →
re-fetch brain). The SYSTEM PROMPT, HOW I OPERATE (AGENTS.md), and PERSONALITY cards
in the command center are now editable inline; saving writes back to the mapped
brain section. AGENTS.md card now always shows (so it can be authored when empty).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Gap 1 — the brain's identity sections were stored but UI-only. cm-runtime/brain.rs
`compose_system` now folds the brain's AGENTS.md ("## How I operate") + personality
into the live system prompt (after the Postgres-authoritative base, before skills +
memory; falls back to the brain's soul_md when the base is empty). Mirrors the
OpenClaw/ZeroClaw render order.
Mapping — expose `agent_md` on `GET /api/claws/{id}/brain` (ClawBrainResponse) and
render it as a collapsible "HOW I OPERATE · AGENTS.md" card in the command center's
BRAIN column (RawBrain gains agent_md; richBrain passes it through). So the section
that's now in the prompt is also visible in the UI.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Replace the single centered "anatomy profile" + resizable chat split with an
operator command center: compact 62px identity strip → 5-tile per-agent metrics
band → three independently-scrolling LIVE · BRAIN · SURFACE columns. Chat moves to
the computer's Chat app; the right computer pullout (DevicePanel) is untouched.
- backend: cm-api/routes/world.rs emits per-agent `telemetry{agentId,tokensPerMin,
costPerHr,loops,doorsPending}` in the SSE loop, from 4 batched GROUP BY queries
(usage_events tokens/min + credits/hr, active routines, pending approvals) — all
real, no migration. taxonomy `telemetry` gains optional agentId; stateKey now
keys it per-agent so slices don't clobber.
- frontend: new ClawCommandCenter + anatomy-cards (shared cards extracted from
Dashboard); useAgentTelemetry(agentId) feeds the metric band (Doors amber>0/
green=0); LIVE column streams the agent's task.update / reasoning.delta /
tool.call (replaces the mocked VitalsCard heatmap with a live activity chart).
- Dashboard: left region → full-height ClawCommandCenter; chat launcher opens the
computer Chat app; removed the dead anatomy cluster + unused imports.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Completes "agent on a node" (single-node): when an agent's placement points at a
fleet node, its terminal container runs there and the browser reaches it over a
direct WebRTC DataChannel (LAN speed), sharing a node-local volume with the
sandbox. gw-04-local agents are byte-identical to before.
- cm-sandbox/docker.rs: empty drive subpath → mount the whole volume at the target
(volume_options None), so a per-agent node-local volume auto-creates at ~/drives.
- cm-api/fleet.rs: NodeHub.open_pty/webrtc_offer carry optional container+session
(injected only when Some); node-terminal caller passes None (host shell unchanged).
- cm-runtime/terminals.rs: TerminalManager gains node_provider + placement
(mirrors SandboxManager, draining-aware); node_local_drive_mount(agent) =
clawmates_agent_<id> at ~/drives; placement_for() ensures + locates the container;
attach uses driver_for(node) (local byte-identical).
- cm-runtime/sandboxes.rs: a node-placed agent sandbox mounts the same per-agent
volume → shares files with the terminal on that node.
- cm-api/routes/terminal.rs: ticket response gains `node`; ws() bridges node-placed
agents through the NodeHub relay (WebRTC + fallback) execing into the container;
local path unchanged. server main wires with_node_provider.
- frontend: agentTerminalConnector mints the ticket then picks WebRTC (node-placed,
⚡ direct / relayed badge) vs WS (local); webrtcConnector generalized to be
endpoint-agnostic (node terminal reuses it).
Known follow-up: terminal (uid 65532) and sandbox (uid 10001) share the volume but
differ in uid — cross-container writes need an aligned uid/gid (group-writable).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Turn the Beszel-tapped metrics into a self-managing loop.
- migration node_rules (workspace/node-scoped: metric op threshold, for_seconds,
action JSONB, last_fired).
- cm-db: repo/node_rules.rs (CRUD + list_enabled); node_metrics::eval_all merges
Beszel + heartbeat scalars per node + a headroom() heuristic; nodes::status_of;
heartbeat now PRESERVES a `draining` status across heartbeats (so a cordon sticks).
- cm-api: node_rules.rs evaluator (spawn_evaluator, 20s) — when a metric condition
holds for the rule's window it fires drain / undrain / alert (in-memory sustained
+ cooldown tracking, modeled on the node sweeper); routes/beszel.rs rules CRUD
(GET/POST/PATCH/DELETE /api/fleet/rules); spawned in clawmates-server.
- cm-runtime: placement_node() is metrics-aware — a `draining` node stops receiving
new agent sandboxes (falls back to local), so the drain rule is actionable.
- frontend: FleetRules section in the Local view — build rules (node · metric · op ·
threshold · duration → action), toggle/delete, with fired-history.
The loop: hot/overloaded node → rule drains it → placement avoids it → recovers →
undrain rule brings it back. Deployed; node_rules migration applied.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Tap each node's Beszel metrics (GPU/temps/disk-IO/network/per-container — beyond
our basic heartbeat) by reading the workspace's Beszel hub. The agents run in
WS-only mode with no locally-readable socket, so (per the de-risk) the server taps
the hub's PocketBase API instead of the daemon reading agents — no daemon changes.
- migrations: workspace_beszel (BYO hub URL + login, server-side only, mirrors the
Tailscale BYO pattern) + node_metrics (latest scalar columns + JSONB blob).
- cm-db: repo/fleet_beszel.rs, repo/node_metrics.rs; nodes SELECT joins node_metrics
(gpu_pct/temp_max surfaced on node_json for the live cards).
- cm-api: beszel.rs client (auth-with-password, poll `systems`, map to nodes by
hostname, upsert metrics) + a 15s spawn_poller; routes/beszel.rs (connect/status/
disconnect + GET /api/nodes/{id}/metrics with history proxied live from the hub).
- frontend: HostCard gains a GPU/temp readout + a Monitor button; NodeMonitor is a
full-width per-node page (current panel + CPU/mem/GPU/temp/net/disk charts from the
hub's 1m history); a "Beszel monitoring" connect form in the Local view.
Reachability confirmed: gw-04 → the hub over the tailnet (100.123.224.84:8090). Needs
the user to connect their hub login to activate the poller.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Terminal keystrokes were ~400ms because every byte relayed browser→Cloudflare→
gw-04 (Europe)→tailscale→node, even when the node is on the user's own LAN. Add a
direct browser↔node WebRTC DataChannel so co-located terminals run at LAN speed;
the gateway is reduced to signaling; the WebSocket relay stays as the automatic
fallback (graceful degradation — never worse than before).
Daemon (clawmates-node v0.4.0, new src/rtc.rs):
- Add the `webrtc` crate (reuses the ring crypto provider we already install — no
conflict). Browser is the offerer; we answer, trickle ICE back over the control
channel, and on DataChannel open spawn a host PTY (tmux) bridged DIRECTLY to the
channel. Refactor open_pty → spawn_terminal_pty shared by both transports.
iceServers: STUN + auto host/tailnet candidates (direct, no relay, for LAN/tailnet).
Server (cm-api):
- NodeConn.signal_sinks; Uplink WebRtcAnswer/WebRtcIce/WebRtcFailed routed to the
browser; NodeHub webrtc_offer/ice/close + open_session/open_pty (open_terminal
split so the PTY opens only once the transport is chosen). bridge_terminal relays
signaling over the existing ticket-authed WS and opens the relay PTY on
{type:"fallback"}.
Browser (NodeTerminalApp):
- RTCPeerConnection + reliable/ordered DataChannel; offer/answer/ICE over the WS;
2.5s race → use the DataChannel if it opens, else fall back to the WS relay.
Reconnect wraps both. A direct⚡/relayed indicator shows the live transport.
Deployed; both nodes (morpheus, tank) updated to v0.4.0 and steady online. Direct-
path proof is a browser action (the ⚡ indicator + latency); confirmable from the
daemon's [rtc] logs.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Nodes flapped online/offline and the terminal died on the first blip. WebSockets
are the right transport (outbound, NAT-friendly); the fixes harden around it.
Server (cm-api):
- Anti-clobber connection epoch: a reconnecting daemon gets a fresh epoch; a stale
run_channel's teardown only clears the hub + sets offline if it still owns the
slot — so a lingering old channel can't flip a live reconnection offline (the
main false-offline cause).
- WS keepalive: run_channel now pings every 15s and tears down if no inbound
frame (incl. pong) for 35s — dead links detected in seconds, not minutes.
- Staleness sweeper backstop: spawn_node_sweeper (8s tick / 20s window) wired in
clawmates-server, so a vanished node goes offline within ~28s even if its
channel hangs (mark_stale_offline was defined but never called).
Daemon (clawmates-node v0.3.0):
- Heartbeats off the select thread (dedicated thread owns System + blocking
docker/tailscale/disk CLIs) so a slow op never starves heartbeats/pongs.
- Each handle_frame runs on its own task; added a 40s inbound idle deadline so a
half-open socket triggers a reconnect.
Frontend:
- useNodes streams /api/nodes/live (SSE push) instead of a 3s poll; isLive()
derives online from lastSeen freshness (<15s) so a transient column flip never
shows a healthy node down.
- Node terminal: clean auto-reconnect loop (re-mint ticket -> reconnect -> tmux
re-attaches and redraws the live screen = mosh-style snap-to-state over TCP),
replacing the [disconnected] dead-end.
Mosh evaluated: harvest principles (session/transport decoupling, snap-to-state,
already given by tmux), don't adopt — UDP is incompatible with our browser+CF+NAT
topology and it's GPLv3. Removed temporary terminal debug traces + /api/debug route.
Verified: node holds steadily online (heartbeat 1-3s, no flap) and goes cleanly
offline when the daemon stops.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The server trace showed pty_open is sent but the daemon (morpheus, v0.2.0) emits
no pty_out — so the PTY spawn was dying silently. Instrument it:
- daemon open_pty: log open/tmux/first-read/EOF/error/total to stdout, and send
an IMMEDIATE banner pty_out ("[clawmates] host shell on <host> — starting…") so
the browser confirms the relay even before the shell draws. If open_pty fails,
send the error as pty_out (was a silent pty_exit). Bump to v0.2.1.
- cm-api: temp GET /api/debug/node-pty/{id}?dbg=… opens a node terminal and reads
~2s of output with no browser/auth, to test the relay in isolation.
- NodeTerminalApp + ticket/ws routes already log each hop ([node-term]/[fleet-term]).
Diagnostic logic: banner shows + shell doesn't → relay ok, shell is the problem;
nothing shows → relay broken; daemon "EOF after N bytes" → shell exited.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- Daemon: factor the terminal command into terminal_command(); add `--selftest`
which opens the host terminal PTY locally and prints ~2.5s of raw output, so
you can confirm tmux/zsh actually draws on a given node without the browser.
(Verified locally: tmux spawns zsh + draws its status bar.)
- cm-api: temporary [fleet-term] eprintln tracing in open_terminal, the pty_out
router, and the browser bridge (byte counts + sink presence) to locate where
output stops between daemon→server→browser. To be removed once diagnosed.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Hostname/IP:
- Daemon reports the machine's hostname (sysinfo) + primary outbound IPv4 on each
heartbeat. migrations/0021 adds nodes.hostname/local_ip; cm-db heartbeat stores
them; node JSON exposes them. Cards now title on the real hostname (falling back
to name) + show the IP, instead of the "New node" placeholder. `name` stays
user-overridable (rename).
Terminal moved into the pull-out computer (no more per-card modal):
- New infra computer app NodeTerminalApp (computer/apps/infra) — xterm bridged to
a node's host shell over the node control channel, filling the app window
(mirrors the agent Terminal's layout + ResizeObserver). Added "terminal" to the
INFRA_CATALOG grid; a ?node= panel param targets a specific node (picker when
unset). Clicking Terminal on a node card now opens the infra computer to that
node's shell instead of a separate full-screen window. Deleted NodeTerminal.tsx.
Rebuilt + re-hosted both daemon binaries (hostname change).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Agents can now provision their sandbox on a connected fleet node instead of the
gateway host. Local stays the strict default, so existing agents are byte-for-
byte unaffected until explicitly placed elsewhere.
Security parity: the daemon links the REAL cm-sandbox DockerDriver and runs the
typed container ops (sb_provision/sb_exec/sb_destroy/sb_health/sb_list) through
it — identical hardening (cap-drop ALL, seccomp, no-net, read-only, non-root) to
local sandboxes. cm-sandbox spec types are now Serialize/Deserialize so the spec
crosses the channel.
- cm-api: RemoteDriver (impl SandboxDriver over the node channel) + HubDriverProvider
(impl cm_runtime::NodeDriverProvider, hands out a driver only for connected
nodes via a sync online set) + NodeHub.call/is_connected. AppState.with_node_hub
so the hub is shared with the placement provider.
- cm-runtime SandboxManager: driver_for(node_id) routes by the recorded
agent_containers.node_id (local default = existing driver, identical path);
placement_node() reads the workspace setting and falls back to local if the
node is offline; exec/release route accordingly. NodeDriverProvider trait.
- DB: 0020_workspace_placement + repo (for_agent/get/set/clear).
- main.rs: build the NodeHub first; inject HubDriverProvider into the agent
manager + share the hub with AppState.
- API+UI: GET/PUT /api/fleet/placement + a "Run agents on: Local / <node>"
selector in the Fleet overview.
Note: a node must be able to pull the agent image (the daemon docker-pulls it);
interactive PTY for agent containers on remote nodes is not wired (Terminal app
stays local) — the in-dashboard node shell already covers host access.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Proves a connected node can host hardened agent workloads end-to-end, without
touching the agent run loop (zero blast radius on existing agents).
- Daemon: typed `sb_check` op — pulls a tiny image and runs it fully locked down
(cap-drop ALL, no-new-privileges, no network, read-only rootfs, non-root,
memory/pids caps), then tears it down. Fixed command; nothing caller-supplied
runs (preserves the exec-hardening invariant).
- cm-api: NodeHub.sandbox_check + POST /api/nodes/{id}/sandbox-check.
- UI: a shield "sandbox check" button on each online node card streams the
result (✓ SANDBOX READY + container id/uname).
This validates the full provision→run→destroy mechanism on nodes. The remaining
P2 work — wiring real agent deploys to auto-place onto nodes — is its own
subsystem (a RemoteDriver reusing the local DockerDriver for security parity,
agent-image distribution to nodes, and node-routing in SandboxManager) and is
best done as a focused pass; it is intentionally NOT bundled here to keep the
core agent path untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
You can now open a real shell on any connected node from the dashboard — the
daemon spawns a host PTY and streams it over the existing outbound control
channel (no inbound port, no Tailscale brokering needed).
Daemon:
- portable-pty host shell sessions: pty_open/pty_in/pty_resize/pty_close ops; a
reader thread streams base64 pty_out frames. Outbound frames now funnel through
one mpsc channel so PTY output and heartbeats interleave.
cm-api NodeHub:
- per-connection pty_sinks + sid multiplexing; open_terminal/terminal_input/
terminal_resize/terminal_close; in-memory single-use terminal tickets (the
browser WS can't carry a bearer, and the session is instance-local anyway).
- routes/nodes.rs: POST /api/nodes/{id}/terminal/ticket + GET .../terminal/ws
(bridges browser xterm <-> node PTY: binary = keystrokes, text = resize).
Frontend:
- NodeTerminal xterm modal (reuses the agent Terminal's xterm setup); a Terminal
button on each online node card opens a shell.
This proves the bidirectional streaming-over-channel mechanism the RemoteDriver
will reuse. Remaining P2: RemoteDriver + placement (run agent workloads on nodes).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Security hardening:
- The gateway no longer sends arbitrary shell to nodes. The WSS exec op is
replaced by a typed `verify` op the daemon runs itself (fixed host+docker
check); future container ops are typed too. cm-api NodeHub.verify() + the
daemon's handle_command only dispatches vetted ops.
BYO Tailscale:
- migrations/0019_workspace_tailscale.sql + cm-db fleet_tailscale repo (store the
user's Tailscale API key + tailnet, server-side only).
- cm-api routes/tailscale.rs: POST/GET/DELETE /api/fleet/tailscale + GET
/api/fleet/tailscale/devices (proxies api.tailscale.com device list).
- Daemon: --tailscale-authkey → `tailscale up --authkey … --ssh` (enables
Tailscale SSH for keyless user access); else `tailscale set --ssh=true`. Reports
its tailscale IP (already).
UI:
- Fleet overview gains a Tailscale section: connect (key+tailnet) + live tailnet
device status (online/last-seen/IP/os). Node cards show a copyable Tailscale SSH
target (ssh <ip>).
Remaining: P2 — RemoteDriver + placement (run agents on nodes) and the in-UI
remote terminal (PTY proxied over the WSS channel).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- Explosive sparks on file/project I/O: /api/world/live tags file-ish tools
(read/write/drive/vault/obsidian/edit/save) with touch weight 1; the engine
carries it as a node "burst" and the renderer fires a big fast particle blast.
- The formation tabs are now genuinely different views:
- Flat — the previous 2D React-Flow forest (WorldFlow), overlaid.
- Live — the WebGL Gource clone (glow sprites + sparks + trails + beams).
- Hierarchy — a NEW neural-brain view: agents are neurons in a dormant
two-hemisphere mesh; active agents fire signal particles along pathway edges
to their nearest neighbours (chaining), so the brain lights up as agents work.
- Render groups (gource/brain) + 3D particle system so signals travel in depth;
shared particle update; OrbitControls (rotate + pinch-zoom) across all WebGL views.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
/api/world/live now tails each active run's journal (run_events) and normalizes
the runner's existing events into taxonomy events — no runner change needed:
- text_delta -> agent.reasoning.delta (the live reasoning stream)
- step_started -> agent.tool.call + node.activity + world.touch on the tool
node (agents visibly converge on the tool they're using)
- approval_required -> door.request
Per-run seq cursor streams forward only (skips backlog on first sight). This
lights up REAL data for both the World view and the upcoming Observe surface.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- /api/world/replay?hours= — reconstructs a sorted, timestamped taxonomy timeline
from the workspace's run history (agent_runs ⋈ sessions) for client playback.
- engine.clearWorldNodes() — drop transient world nodes/targets for a loop restart.
- WorldCanvas: a WorldClock control (Live ⇄ Replay) feeding the same engine —
play/pause, 1x/2x/4x speed, seek bar, loop. Live subscriptions detach in replay.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- engine: when there are no service/event touch-nodes (a quiet feed), pawns roam
the real org/company/team structure so the view is always alive with zero
synthetic data; real touch-nodes take priority when present.
- /api/world/live: emit world.touch + node.activity per currently-running run
(agent_runs joined to sessions, state='running') so each working agent visibly
converges on its active-run node; fold running agents into working status;
telemetry.loops = active run count.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Replaces the static React-Flow world forest with a real-time WebGL view of the
swarm working, plus the live event contract that feeds it.
Phase A — the live contract + feed:
- frontend/src/lib/live/taxonomy.ts — the 13-event Clawmates Event Taxonomy as
typed TS (the shared World/Observe contract).
- frontend/src/lib/live/useClawmatesLive.ts — native shared-singleton live client
(EventSource to /api/world/live, typed fan-out, replay-of-last-state to late
subscribers, refcounted, synthetic fallback so views are always alive).
- crates/cm-api/src/routes/world.rs — GET /api/world/live, authed + workspace-
scoped SSE emitting the taxonomy (real agent.status from live-container state,
a topology.update of the workspace's agents, telemetry with real doorsPending),
mirroring run_events_sse. normalize() seam documented for run_events->world.touch.
Phase B — the WebGL engine:
- frontend/src/components/world/engine.ts — Gource-inspired force-directed model:
org/company/team tree (sibling repulsion + parent spring + friction), agent
pawns that converge on the touched node and beam it, world nodes that glow with
heat and fade when idle. Framework-agnostic (renderer-independent) state+math.
- frontend/src/components/world/WorldCanvas.tsx — three.js scene (ortho cam,
UnrealBloom), render loop syncing engine state, camera auto-fit, HTML labels,
raycast click->select, Hierarchy/Flat/Live formation switch.
- Dashboard.tsx: swap <WorldFlow/> -> <WorldCanvas/> at the world-tier seam
(shared claw-tier pieces untouched; WorldFlow.tsx kept for now).
- Adds three (+ @types/three).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Decouples "many users" + "many server replicas" from "many machines" so the
platform is tenant-isolated and horizontally safe on the current single node.
- Per-signup workspaces (cm-auth): a new hosted-identity sign-in provisions and
owns its own workspace instead of joining the first. Config-gated by
auth.per_signup_workspace (default off); concurrent first-logins serialized by
a per-subject advisory lock so no duplicate workspaces.
- Terminal tickets in Postgres (migration 0016, hashed, single-use): any replica
can redeem a ticket minted by another. Drops the in-process ticket map.
- Container registry in Postgres (migration 0017, agent_containers): Terminal
and Sandbox managers resolve an agent's container through a shared registry,
so a 2nd replica reuses it instead of spawning a duplicate. node_id recorded
as 'local' (Phase 2 hook). Boot reconcile removes only true orphans, so
terminals now survive a redeploy (tmux sessions resume).
- Per-workspace quotas (cm-api/quota.rs): plan-tier caps on agents + live
containers, enforced at agent create + terminal spin-up (reconnects allowed),
returned as HTTP 402. New GET /api/quota surfaces usage vs limits.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Terminal app (xterm ⇄ WebSocket ⇄ per-agent themed container):
- zsh + oh-my-zsh + powerlevel10k image (agent-terminal), runs as uid 65532 to
share read-write ownership of the file-drive volume with the server.
- Interactive PTY in cm-sandbox (bollard exec tty/attach + resize) + a
TerminalManager; ticket-authed WS bridge routed straight to the backend via a
Traefik PathRegexp(/ws) rule. MOTD greets the user by name.
- tmux resumable sessions; multi-tab (one tmux session per tab, same container),
drag-to-reorder, rename, and a Save that persists named tabs to the server
(terminal_tabs, migration 0014) so they survive logout / a new device.
- Files drives mounted per-agent (subpath) at ~/drives/{documents,received,
shared}; a reconciler keeps the Files app's index in sync with terminal writes.
Storage moved to a shared `filedata` volume (CLAWMATES_STORAGE__DATA_DIR).
Obsidian vault (a markdown "second brain" per agent):
- New `vault` FileDrive (migration 0015) mounted into the terminal at ~/obsidian;
a file-content read route; a purple Obsidian tile + a vault viewer app.
Computer UI:
- Draggable computer-panel width (min = phone preset) keeping the size presets.
- Green Terminal glyph, "Claw Chat" → "Chat", colored gradient-outline app icons.
- Agent page: avatar↔activity-grid spacing + larger, uniform section fonts with
colored section-tinted tag chips.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
A team = a baseline topology staffed with real claws. Migration 0010 (teams +
team_members node→claw bindings) + cm-db repo/teams.rs. runtime_provision.rs
turns a claw into a live runtime agent claw_<id> via the synced gateway config
API (#7468): create agent + bind model_provider (mapped from chosen model) +
risk_profile=toolfree + clawmates_door bundle — atomic, immediately drivable.
routes/teams.rs: POST /api/teams (create claws + provision + build(kind,roles)
+ bind node.attrs["agent"]=claw_<id> + persist), GET /api/teams[/{id}],
POST /api/teams/{id}/run (enqueue a durable run of the team graph — reuses the
topology worker + SSE). v1 persona = topology role via the prompt builder; the
claw's system_prompt stays its chat identity.
Spike confirmed: runtime agent provisioning works; IDENTITY.md persona works
for API models (Gemini/Groq), masked by CLI models (Claude/Kimi Code). 16
cm-api tests + provision unit tests pass, clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The §15 governor can now be judged by a ZeroClaw runtime agent instead of a
server-side registry/API model. ZeroClawDriveExecutor::judge drives an agent
with the governor prompt and parses ALLOW/DENY (fail-open). mcp_door routes to
it when CLAWMATES_JUDGE_MODEL=runtime:<alias>.
This unblocks Kimi-as-judge with NO Kimi Platform key: Kimi runs on the
membership via kimi_cli, so CLAWMATES_JUDGE_MODEL=runtime:judge_kimi makes the
coding agent the governor. (Same path works for any subscription-only model.)
cm-runtime re-exports judge_model. clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
#2 GLM judge (closes#114): registry NamedProvider gains a `format` field;
build_provider_registry builds an AnthropicProvider for format="anthropic".
GLM's coding/OpenAI endpoint is ToS-throttled for raw SDK, but its Anthropic
endpoint (api.z.ai/api/anthropic) accepts raw API calls (verified x-api-key
-> glm-4.7), so CLAWMATES_JUDGE_MODEL=glm:glm-4.7 routes the door governor /
topology judge through GLM with no runtime-routing. (Kimi-as-judge still needs
a Platform key — coding key is agent-only.)
#3 run-control: POST /api/topology-runs/{id}/cancel (workspace-scoped,
queued/running only); the worker honors it at the step boundary (checks
current_status in the checkpoint callback) and won't clobber a cancel with
`failed`. Frontend Run tab gains a Cancel button and clickable recent runs
that deep-link into a live/replayed stream (SSE replays from checkpoint).
cm-* tests (incl. new cancel test) + clippy + frontend lint/typecheck green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Backend: GET /api/topology-runs/{id}/events streams Server-Sent Events by
tailing the durable per-step checkpoint the worker already writes — a `step`
event per newly-completed step (replayed on connect so reload/reconnect
re-attaches), then a terminal `done` event with the final output/error. Each
step carries its index as the SSE id, so the browser's Last-Event-ID resumes
without duplicates on reconnect. No new table, no worker change — reuses the
checkpoint; avoids run_events' agent_runs FK.
Frontend: the Run tab now opens an EventSource (through the same-origin proxy,
which adds the bearer) instead of polling — appending steps as they stream and
finalizing on `done`. One streaming connection, lower latency, auto-resume.
cm-api builds + clippy clean; frontend lint + typecheck + next build clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
POST /api/topologies/run now ENQUEUES a durable job and returns 202
{run_id, status:queued} instead of executing inside the HTTP request — the
prerequisite for long-horizon runs (no client/proxy/LB timeout, survives
restarts).
topology_worker: a spawned loop that requeues stale running jobs, claims the
next queued one (CAS via FOR UPDATE SKIP LOCKED), drives it through
execute_resumable, and checkpoints RunProgress after every step; on crash the
stale sweep requeues it and the next claim resumes from the last checkpoint.
Wired into server startup beside the scheduler + resume sweeper.
GET /api/topology-runs/{id} now reports lifecycle status/kind/error/checkpoint
+ the result blob (kept the `comparison` field name for back-compat with the
compare UI; null until completed). list_runs includes status + kind.
Tests: durable lifecycle (enqueue→claim→checkpoint→complete) + stale-requeue
resume, both green; p0 endpoints (compare path) unchanged. 13 + 2 tests pass,
clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Evolve topology_runs into a durable-job table (migration 0009): status state
machine (queued/running/completed/failed/cancelled), kind, input graph,
per-step checkpoint, error, last_event_id, timestamps. comparison becomes
nullable (the result blob, absent until completion). Back-compat: existing
rows default to completed/compare.
cm-db repo gains the durable-job ops: enqueue_run, claim_next_queued (CAS via
FOR UPDATE SKIP LOCKED), checkpoint, complete, fail, requeue_stale (resume
sweep), and status(). Regenerated .sqlx cache. Also fix two pre-existing test
RuntimeConfig literals missing the providers field (from the registry work).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Omar Sobh