Tap each node's Beszel metrics (GPU/temps/disk-IO/network/per-container — beyond
our basic heartbeat) by reading the workspace's Beszel hub. The agents run in
WS-only mode with no locally-readable socket, so (per the de-risk) the server taps
the hub's PocketBase API instead of the daemon reading agents — no daemon changes.
- migrations: workspace_beszel (BYO hub URL + login, server-side only, mirrors the
Tailscale BYO pattern) + node_metrics (latest scalar columns + JSONB blob).
- cm-db: repo/fleet_beszel.rs, repo/node_metrics.rs; nodes SELECT joins node_metrics
(gpu_pct/temp_max surfaced on node_json for the live cards).
- cm-api: beszel.rs client (auth-with-password, poll `systems`, map to nodes by
hostname, upsert metrics) + a 15s spawn_poller; routes/beszel.rs (connect/status/
disconnect + GET /api/nodes/{id}/metrics with history proxied live from the hub).
- frontend: HostCard gains a GPU/temp readout + a Monitor button; NodeMonitor is a
full-width per-node page (current panel + CPU/mem/GPU/temp/net/disk charts from the
hub's 1m history); a "Beszel monitoring" connect form in the Local view.
Reachability confirmed: gw-04 → the hub over the tailnet (100.123.224.84:8090). Needs
the user to connect their hub login to activate the poller.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Terminal keystrokes were ~400ms because every byte relayed browser→Cloudflare→
gw-04 (Europe)→tailscale→node, even when the node is on the user's own LAN. Add a
direct browser↔node WebRTC DataChannel so co-located terminals run at LAN speed;
the gateway is reduced to signaling; the WebSocket relay stays as the automatic
fallback (graceful degradation — never worse than before).
Daemon (clawmates-node v0.4.0, new src/rtc.rs):
- Add the `webrtc` crate (reuses the ring crypto provider we already install — no
conflict). Browser is the offerer; we answer, trickle ICE back over the control
channel, and on DataChannel open spawn a host PTY (tmux) bridged DIRECTLY to the
channel. Refactor open_pty → spawn_terminal_pty shared by both transports.
iceServers: STUN + auto host/tailnet candidates (direct, no relay, for LAN/tailnet).
Server (cm-api):
- NodeConn.signal_sinks; Uplink WebRtcAnswer/WebRtcIce/WebRtcFailed routed to the
browser; NodeHub webrtc_offer/ice/close + open_session/open_pty (open_terminal
split so the PTY opens only once the transport is chosen). bridge_terminal relays
signaling over the existing ticket-authed WS and opens the relay PTY on
{type:"fallback"}.
Browser (NodeTerminalApp):
- RTCPeerConnection + reliable/ordered DataChannel; offer/answer/ICE over the WS;
2.5s race → use the DataChannel if it opens, else fall back to the WS relay.
Reconnect wraps both. A direct⚡/relayed indicator shows the live transport.
Deployed; both nodes (morpheus, tank) updated to v0.4.0 and steady online. Direct-
path proof is a browser action (the ⚡ indicator + latency); confirmable from the
daemon's [rtc] logs.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Nodes flapped online/offline and the terminal died on the first blip. WebSockets
are the right transport (outbound, NAT-friendly); the fixes harden around it.
Server (cm-api):
- Anti-clobber connection epoch: a reconnecting daemon gets a fresh epoch; a stale
run_channel's teardown only clears the hub + sets offline if it still owns the
slot — so a lingering old channel can't flip a live reconnection offline (the
main false-offline cause).
- WS keepalive: run_channel now pings every 15s and tears down if no inbound
frame (incl. pong) for 35s — dead links detected in seconds, not minutes.
- Staleness sweeper backstop: spawn_node_sweeper (8s tick / 20s window) wired in
clawmates-server, so a vanished node goes offline within ~28s even if its
channel hangs (mark_stale_offline was defined but never called).
Daemon (clawmates-node v0.3.0):
- Heartbeats off the select thread (dedicated thread owns System + blocking
docker/tailscale/disk CLIs) so a slow op never starves heartbeats/pongs.
- Each handle_frame runs on its own task; added a 40s inbound idle deadline so a
half-open socket triggers a reconnect.
Frontend:
- useNodes streams /api/nodes/live (SSE push) instead of a 3s poll; isLive()
derives online from lastSeen freshness (<15s) so a transient column flip never
shows a healthy node down.
- Node terminal: clean auto-reconnect loop (re-mint ticket -> reconnect -> tmux
re-attaches and redraws the live screen = mosh-style snap-to-state over TCP),
replacing the [disconnected] dead-end.
Mosh evaluated: harvest principles (session/transport decoupling, snap-to-state,
already given by tmux), don't adopt — UDP is incompatible with our browser+CF+NAT
topology and it's GPLv3. Removed temporary terminal debug traces + /api/debug route.
Verified: node holds steadily online (heartbeat 1-3s, no flap) and goes cleanly
offline when the daemon stops.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Root cause (from the server trace: 69 banner bytes then immediate "to_browser
ended" = pty_exit): tmux spawned and exited instantly, writing its "sessions
should be nested with care" warning to stderr (not the PTY). The operator runs
the daemon inside their own tmux, so $TMUX was inherited and the spawned tmux
refused to nest → browser saw nothing.
- daemon (v0.2.2): spawn tmux on a DEDICATED socket (`tmux -L clawmates
new-session -A -s main`) and `env_remove("TMUX")`, so it can never collide with
or be refused by the operator's tmux.
- NodeTerminalApp: debounce the ResizeObserver (150ms). The pull-out animates
open, firing the observer on every pixel — previously ~80 resize frames per
open, each fit()+SIGWINCH. Now one resize after layout settles.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The server trace showed pty_open is sent but the daemon (morpheus, v0.2.0) emits
no pty_out — so the PTY spawn was dying silently. Instrument it:
- daemon open_pty: log open/tmux/first-read/EOF/error/total to stdout, and send
an IMMEDIATE banner pty_out ("[clawmates] host shell on <host> — starting…") so
the browser confirms the relay even before the shell draws. If open_pty fails,
send the error as pty_out (was a silent pty_exit). Bump to v0.2.1.
- cm-api: temp GET /api/debug/node-pty/{id}?dbg=… opens a node terminal and reads
~2s of output with no browser/auth, to test the relay in isolation.
- NodeTerminalApp + ticket/ws routes already log each hop ([node-term]/[fleet-term]).
Diagnostic logic: banner shows + shell doesn't → relay ok, shell is the problem;
nothing shows → relay broken; daemon "EOF after N bytes" → shell exited.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- Daemon: factor the terminal command into terminal_command(); add `--selftest`
which opens the host terminal PTY locally and prints ~2.5s of raw output, so
you can confirm tmux/zsh actually draws on a given node without the browser.
(Verified locally: tmux spawns zsh + draws its status bar.)
- cm-api: temporary [fleet-term] eprintln tracing in open_terminal, the pty_out
router, and the browser bridge (byte counts + sink presence) to locate where
output stops between daemon→server→browser. To be removed once diagnosed.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The node terminal opened a bare bash PTY — bash prints its prompt once, so a
freshly-attached xterm showed nothing. Switch to `tmux new-session -A -s
clawmates` on the HOST (mirrors the agent terminal, but on the node itself, not
in a container): tmux redraws the whole screen on attach (no blank), and the
session is resumable across reopens. Starts in $HOME. Falls back to a login
shell if tmux is unavailable. ensure_tmux() best-effort installs tmux via the
host package manager when the daemon runs as root (systemd); otherwise logs a
hint to `apt install tmux`. Rebuilt + re-hosted both binaries.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Hostname/IP:
- Daemon reports the machine's hostname (sysinfo) + primary outbound IPv4 on each
heartbeat. migrations/0021 adds nodes.hostname/local_ip; cm-db heartbeat stores
them; node JSON exposes them. Cards now title on the real hostname (falling back
to name) + show the IP, instead of the "New node" placeholder. `name` stays
user-overridable (rename).
Terminal moved into the pull-out computer (no more per-card modal):
- New infra computer app NodeTerminalApp (computer/apps/infra) — xterm bridged to
a node's host shell over the node control channel, filling the app window
(mirrors the agent Terminal's layout + ResizeObserver). Added "terminal" to the
INFRA_CATALOG grid; a ?node= panel param targets a specific node (picker when
unset). Clicking Terminal on a node card now opens the infra computer to that
node's shell instead of a separate full-screen window. Deleted NodeTerminal.tsx.
Rebuilt + re-hosted both daemon binaries (hostname change).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Two bugs surfaced running the daemon on a real node:
- Linking cm-sandbox (bollard) brought a second rustls provider into the graph,
so rustls couldn't auto-pick one and panicked at the WSS TLS handshake. Install
the ring provider explicitly at startup (rustls dep + install_default()).
- The daemon auto-ran `tailscale set --ssh`, which tries to reroute the user's
live SSH session and aborts ("will result in your session disconnecting"). Now
Tailscale is only touched when an auth key is explicitly passed (opt-in), with
--accept-risk=lose-ssh to avoid the interactive abort.
Rebuilt + re-hosted both binaries (linux-amd64, darwin-arm64).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Agents can now provision their sandbox on a connected fleet node instead of the
gateway host. Local stays the strict default, so existing agents are byte-for-
byte unaffected until explicitly placed elsewhere.
Security parity: the daemon links the REAL cm-sandbox DockerDriver and runs the
typed container ops (sb_provision/sb_exec/sb_destroy/sb_health/sb_list) through
it — identical hardening (cap-drop ALL, seccomp, no-net, read-only, non-root) to
local sandboxes. cm-sandbox spec types are now Serialize/Deserialize so the spec
crosses the channel.
- cm-api: RemoteDriver (impl SandboxDriver over the node channel) + HubDriverProvider
(impl cm_runtime::NodeDriverProvider, hands out a driver only for connected
nodes via a sync online set) + NodeHub.call/is_connected. AppState.with_node_hub
so the hub is shared with the placement provider.
- cm-runtime SandboxManager: driver_for(node_id) routes by the recorded
agent_containers.node_id (local default = existing driver, identical path);
placement_node() reads the workspace setting and falls back to local if the
node is offline; exec/release route accordingly. NodeDriverProvider trait.
- DB: 0020_workspace_placement + repo (for_agent/get/set/clear).
- main.rs: build the NodeHub first; inject HubDriverProvider into the agent
manager + share the hub with AppState.
- API+UI: GET/PUT /api/fleet/placement + a "Run agents on: Local / <node>"
selector in the Fleet overview.
Note: a node must be able to pull the agent image (the daemon docker-pulls it);
interactive PTY for agent containers on remote nodes is not wired (Terminal app
stays local) — the in-dashboard node shell already covers host access.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Proves a connected node can host hardened agent workloads end-to-end, without
touching the agent run loop (zero blast radius on existing agents).
- Daemon: typed `sb_check` op — pulls a tiny image and runs it fully locked down
(cap-drop ALL, no-new-privileges, no network, read-only rootfs, non-root,
memory/pids caps), then tears it down. Fixed command; nothing caller-supplied
runs (preserves the exec-hardening invariant).
- cm-api: NodeHub.sandbox_check + POST /api/nodes/{id}/sandbox-check.
- UI: a shield "sandbox check" button on each online node card streams the
result (✓ SANDBOX READY + container id/uname).
This validates the full provision→run→destroy mechanism on nodes. The remaining
P2 work — wiring real agent deploys to auto-place onto nodes — is its own
subsystem (a RemoteDriver reusing the local DockerDriver for security parity,
agent-image distribution to nodes, and node-routing in SandboxManager) and is
best done as a focused pass; it is intentionally NOT bundled here to keep the
core agent path untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
You can now open a real shell on any connected node from the dashboard — the
daemon spawns a host PTY and streams it over the existing outbound control
channel (no inbound port, no Tailscale brokering needed).
Daemon:
- portable-pty host shell sessions: pty_open/pty_in/pty_resize/pty_close ops; a
reader thread streams base64 pty_out frames. Outbound frames now funnel through
one mpsc channel so PTY output and heartbeats interleave.
cm-api NodeHub:
- per-connection pty_sinks + sid multiplexing; open_terminal/terminal_input/
terminal_resize/terminal_close; in-memory single-use terminal tickets (the
browser WS can't carry a bearer, and the session is instance-local anyway).
- routes/nodes.rs: POST /api/nodes/{id}/terminal/ticket + GET .../terminal/ws
(bridges browser xterm <-> node PTY: binary = keystrokes, text = resize).
Frontend:
- NodeTerminal xterm modal (reuses the agent Terminal's xterm setup); a Terminal
button on each online node card opens a shell.
This proves the bidirectional streaming-over-channel mechanism the RemoteDriver
will reuse. Remaining P2: RemoteDriver + placement (run agent workloads on nodes).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Security hardening:
- The gateway no longer sends arbitrary shell to nodes. The WSS exec op is
replaced by a typed `verify` op the daemon runs itself (fixed host+docker
check); future container ops are typed too. cm-api NodeHub.verify() + the
daemon's handle_command only dispatches vetted ops.
BYO Tailscale:
- migrations/0019_workspace_tailscale.sql + cm-db fleet_tailscale repo (store the
user's Tailscale API key + tailnet, server-side only).
- cm-api routes/tailscale.rs: POST/GET/DELETE /api/fleet/tailscale + GET
/api/fleet/tailscale/devices (proxies api.tailscale.com device list).
- Daemon: --tailscale-authkey → `tailscale up --authkey … --ssh` (enables
Tailscale SSH for keyless user access); else `tailscale set --ssh=true`. Reports
its tailscale IP (already).
UI:
- Fleet overview gains a Tailscale section: connect (key+tailnet) + live tailnet
device status (online/last-seen/IP/os). Node cards show a copyable Tailscale SSH
target (ssh <ip>).
Remaining: P2 — RemoteDriver + placement (run agents on nodes) and the in-UI
remote terminal (PTY proxied over the WSS channel).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- Explosive sparks on file/project I/O: /api/world/live tags file-ish tools
(read/write/drive/vault/obsidian/edit/save) with touch weight 1; the engine
carries it as a node "burst" and the renderer fires a big fast particle blast.
- The formation tabs are now genuinely different views:
- Flat — the previous 2D React-Flow forest (WorldFlow), overlaid.
- Live — the WebGL Gource clone (glow sprites + sparks + trails + beams).
- Hierarchy — a NEW neural-brain view: agents are neurons in a dormant
two-hemisphere mesh; active agents fire signal particles along pathway edges
to their nearest neighbours (chaining), so the brain lights up as agents work.
- Render groups (gource/brain) + 3D particle system so signals travel in depth;
shared particle update; OrbitControls (rotate + pinch-zoom) across all WebGL views.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
/api/world/live now tails each active run's journal (run_events) and normalizes
the runner's existing events into taxonomy events — no runner change needed:
- text_delta -> agent.reasoning.delta (the live reasoning stream)
- step_started -> agent.tool.call + node.activity + world.touch on the tool
node (agents visibly converge on the tool they're using)
- approval_required -> door.request
Per-run seq cursor streams forward only (skips backlog on first sight). This
lights up REAL data for both the World view and the upcoming Observe surface.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- /api/world/replay?hours= — reconstructs a sorted, timestamped taxonomy timeline
from the workspace's run history (agent_runs ⋈ sessions) for client playback.
- engine.clearWorldNodes() — drop transient world nodes/targets for a loop restart.
- WorldCanvas: a WorldClock control (Live ⇄ Replay) feeding the same engine —
play/pause, 1x/2x/4x speed, seek bar, loop. Live subscriptions detach in replay.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- engine: when there are no service/event touch-nodes (a quiet feed), pawns roam
the real org/company/team structure so the view is always alive with zero
synthetic data; real touch-nodes take priority when present.
- /api/world/live: emit world.touch + node.activity per currently-running run
(agent_runs joined to sessions, state='running') so each working agent visibly
converges on its active-run node; fold running agents into working status;
telemetry.loops = active run count.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Replaces the static React-Flow world forest with a real-time WebGL view of the
swarm working, plus the live event contract that feeds it.
Phase A — the live contract + feed:
- frontend/src/lib/live/taxonomy.ts — the 13-event Clawmates Event Taxonomy as
typed TS (the shared World/Observe contract).
- frontend/src/lib/live/useClawmatesLive.ts — native shared-singleton live client
(EventSource to /api/world/live, typed fan-out, replay-of-last-state to late
subscribers, refcounted, synthetic fallback so views are always alive).
- crates/cm-api/src/routes/world.rs — GET /api/world/live, authed + workspace-
scoped SSE emitting the taxonomy (real agent.status from live-container state,
a topology.update of the workspace's agents, telemetry with real doorsPending),
mirroring run_events_sse. normalize() seam documented for run_events->world.touch.
Phase B — the WebGL engine:
- frontend/src/components/world/engine.ts — Gource-inspired force-directed model:
org/company/team tree (sibling repulsion + parent spring + friction), agent
pawns that converge on the touched node and beam it, world nodes that glow with
heat and fade when idle. Framework-agnostic (renderer-independent) state+math.
- frontend/src/components/world/WorldCanvas.tsx — three.js scene (ortho cam,
UnrealBloom), render loop syncing engine state, camera auto-fit, HTML labels,
raycast click->select, Hierarchy/Flat/Live formation switch.
- Dashboard.tsx: swap <WorldFlow/> -> <WorldCanvas/> at the world-tier seam
(shared claw-tier pieces untouched; WorldFlow.tsx kept for now).
- Adds three (+ @types/three).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Decouples "many users" + "many server replicas" from "many machines" so the
platform is tenant-isolated and horizontally safe on the current single node.
- Per-signup workspaces (cm-auth): a new hosted-identity sign-in provisions and
owns its own workspace instead of joining the first. Config-gated by
auth.per_signup_workspace (default off); concurrent first-logins serialized by
a per-subject advisory lock so no duplicate workspaces.
- Terminal tickets in Postgres (migration 0016, hashed, single-use): any replica
can redeem a ticket minted by another. Drops the in-process ticket map.
- Container registry in Postgres (migration 0017, agent_containers): Terminal
and Sandbox managers resolve an agent's container through a shared registry,
so a 2nd replica reuses it instead of spawning a duplicate. node_id recorded
as 'local' (Phase 2 hook). Boot reconcile removes only true orphans, so
terminals now survive a redeploy (tmux sessions resume).
- Per-workspace quotas (cm-api/quota.rs): plan-tier caps on agents + live
containers, enforced at agent create + terminal spin-up (reconnects allowed),
returned as HTTP 402. New GET /api/quota surfaces usage vs limits.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Terminal app (xterm ⇄ WebSocket ⇄ per-agent themed container):
- zsh + oh-my-zsh + powerlevel10k image (agent-terminal), runs as uid 65532 to
share read-write ownership of the file-drive volume with the server.
- Interactive PTY in cm-sandbox (bollard exec tty/attach + resize) + a
TerminalManager; ticket-authed WS bridge routed straight to the backend via a
Traefik PathRegexp(/ws) rule. MOTD greets the user by name.
- tmux resumable sessions; multi-tab (one tmux session per tab, same container),
drag-to-reorder, rename, and a Save that persists named tabs to the server
(terminal_tabs, migration 0014) so they survive logout / a new device.
- Files drives mounted per-agent (subpath) at ~/drives/{documents,received,
shared}; a reconciler keeps the Files app's index in sync with terminal writes.
Storage moved to a shared `filedata` volume (CLAWMATES_STORAGE__DATA_DIR).
Obsidian vault (a markdown "second brain" per agent):
- New `vault` FileDrive (migration 0015) mounted into the terminal at ~/obsidian;
a file-content read route; a purple Obsidian tile + a vault viewer app.
Computer UI:
- Draggable computer-panel width (min = phone preset) keeping the size presets.
- Green Terminal glyph, "Claw Chat" → "Chat", colored gradient-outline app icons.
- Agent page: avatar↔activity-grid spacing + larger, uniform section fonts with
colored section-tinted tag chips.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
A team = a baseline topology staffed with real claws. Migration 0010 (teams +
team_members node→claw bindings) + cm-db repo/teams.rs. runtime_provision.rs
turns a claw into a live runtime agent claw_<id> via the synced gateway config
API (#7468): create agent + bind model_provider (mapped from chosen model) +
risk_profile=toolfree + clawmates_door bundle — atomic, immediately drivable.
routes/teams.rs: POST /api/teams (create claws + provision + build(kind,roles)
+ bind node.attrs["agent"]=claw_<id> + persist), GET /api/teams[/{id}],
POST /api/teams/{id}/run (enqueue a durable run of the team graph — reuses the
topology worker + SSE). v1 persona = topology role via the prompt builder; the
claw's system_prompt stays its chat identity.
Spike confirmed: runtime agent provisioning works; IDENTITY.md persona works
for API models (Gemini/Groq), masked by CLI models (Claude/Kimi Code). 16
cm-api tests + provision unit tests pass, clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The §15 governor can now be judged by a ZeroClaw runtime agent instead of a
server-side registry/API model. ZeroClawDriveExecutor::judge drives an agent
with the governor prompt and parses ALLOW/DENY (fail-open). mcp_door routes to
it when CLAWMATES_JUDGE_MODEL=runtime:<alias>.
This unblocks Kimi-as-judge with NO Kimi Platform key: Kimi runs on the
membership via kimi_cli, so CLAWMATES_JUDGE_MODEL=runtime:judge_kimi makes the
coding agent the governor. (Same path works for any subscription-only model.)
cm-runtime re-exports judge_model. clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The §15 door's email_send queues to `outbox` but nothing delivered it. Add a
real transport: cm-db outbox repo (list_queued/mark_sent/mark_failed) + a
cm-runtime drainer — an EmailSender trait (testable), a lettre STARTTLS
LettreSender, drain_once (queued -> sent/failed), and spawn_drainer wired into
the server beside the scheduler/sweeper/topology-worker.
Config-gated: inert (logs "outbox delivery DISABLED") until CLAWMATES_SMTP_*
is set, so it ships safely before credentials exist. The agent never holds the
SMTP credential — it only writes to outbox through the gated door; the server
owns the transport.
NOTE: live delivery is still credential-blocked — Migadu's API can't send
(SMTP-only) and the admin token is invalid; no SMTP creds exist. The transport
is built + tested (drain_marks_sent_and_failed via a mock sender); set
CLAWMATES_SMTP_* to go live with zero further code. clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
#2 GLM judge (closes#114): registry NamedProvider gains a `format` field;
build_provider_registry builds an AnthropicProvider for format="anthropic".
GLM's coding/OpenAI endpoint is ToS-throttled for raw SDK, but its Anthropic
endpoint (api.z.ai/api/anthropic) accepts raw API calls (verified x-api-key
-> glm-4.7), so CLAWMATES_JUDGE_MODEL=glm:glm-4.7 routes the door governor /
topology judge through GLM with no runtime-routing. (Kimi-as-judge still needs
a Platform key — coding key is agent-only.)
#3 run-control: POST /api/topology-runs/{id}/cancel (workspace-scoped,
queued/running only); the worker honors it at the step boundary (checks
current_status in the checkpoint callback) and won't clobber a cancel with
`failed`. Frontend Run tab gains a Cancel button and clickable recent runs
that deep-link into a live/replayed stream (SSE replays from checkpoint).
cm-* tests (incl. new cancel test) + clippy + frontend lint/typecheck green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Backend: GET /api/topology-runs/{id}/events streams Server-Sent Events by
tailing the durable per-step checkpoint the worker already writes — a `step`
event per newly-completed step (replayed on connect so reload/reconnect
re-attaches), then a terminal `done` event with the final output/error. Each
step carries its index as the SSE id, so the browser's Last-Event-ID resumes
without duplicates on reconnect. No new table, no worker change — reuses the
checkpoint; avoids run_events' agent_runs FK.
Frontend: the Run tab now opens an EventSource (through the same-origin proxy,
which adds the bearer) instead of polling — appending steps as they stream and
finalizing on `done`. One streaming connection, lower latency, auto-resume.
cm-api builds + clippy clean; frontend lint + typecheck + next build clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
POST /api/topologies/run now ENQUEUES a durable job and returns 202
{run_id, status:queued} instead of executing inside the HTTP request — the
prerequisite for long-horizon runs (no client/proxy/LB timeout, survives
restarts).
topology_worker: a spawned loop that requeues stale running jobs, claims the
next queued one (CAS via FOR UPDATE SKIP LOCKED), drives it through
execute_resumable, and checkpoints RunProgress after every step; on crash the
stale sweep requeues it and the next claim resumes from the last checkpoint.
Wired into server startup beside the scheduler + resume sweeper.
GET /api/topology-runs/{id} now reports lifecycle status/kind/error/checkpoint
+ the result blob (kept the `comparison` field name for back-compat with the
compare UI; null until completed). list_runs includes status + kind.
Tests: durable lifecycle (enqueue→claim→checkpoint→complete) + stale-requeue
resume, both green; p0 endpoints (compare path) unchanged. 13 + 2 tests pass,
clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Split execute() into a thin wrapper over new execute_resumable(), which starts
from a prior RunProgress checkpoint (completed step count + outputs + records +
metrics) and invokes an async on_step callback after each newly completed step.
The step plan is re-derived from the graph (planners are deterministic), so only
completed outputs need persisting; the callback owns persistence, keeping the
orchestrator storage-agnostic. RunProgress + the journal types (StepRecord,
RunMetrics, StepPhase, GatedAction) gain Deserialize for JSONB round-trip.
New test: resume-from-checkpoint runs only the remaining steps and reproduces
the full run's output. 14 tests pass, clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Evolve topology_runs into a durable-job table (migration 0009): status state
machine (queued/running/completed/failed/cancelled), kind, input graph,
per-step checkpoint, error, last_event_id, timestamps. comparison becomes
nullable (the result blob, absent until completion). Back-compat: existing
rows default to completed/compare.
cm-db repo gains the durable-job ops: enqueue_run, claim_next_queued (CAS via
FOR UPDATE SKIP LOCKED), checkpoint, complete, fail, requeue_stale (resume
sweep), and status(). Regenerated .sqlx cache. Also fix two pre-existing test
RuntimeConfig literals missing the providers field (from the registry work).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
TurnRequest gains an optional `agent` sourced from the graph node's
attrs["agent"]. The ZeroClaw executor binds a node to that alias directly
when present, falling back to the role→alias map otherwise. This lets a
single POST /api/topologies/run specify a different model per role
(heterogeneous topologies) entirely in the graph JSON — no server
ZEROCLAW_AGENT_MAP change or recreate per configuration, which makes
quota-frugal model×role sweeps practical.
cm-orchestrator 13 + cm-api topology_exec 4 tests pass, clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adds a multi-provider registry so judges and topology execution can run on
providers beyond the default. cm-config gains [[llm.providers]] (name, base_url,
api_key_env) — all OpenAI-compatible (GLM, Kimi/Moonshot). The server builds an
Arc<dyn LlmProvider> per entry (OpenAiCompatProvider) keyed by name; a missing
key is skipped with a warning, not a boot failure. cm-runtime RuntimeConfig
carries a ProviderRegistry; Runtime::resolve_provider("<name>:<model>") selects a
registry provider (else the default). The door governor (Runtime::judge) and the
topology compare endpoint both resolve through it, so CLAWMATES_JUDGE_MODEL and
CLAWMATES_TOPOLOGY_EXEC_MODEL accept "glm:glm-4.6" / "kimi:kimi-k2".
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Both LLM-as-judge sites — the door governor (Runtime::judge) and the topology
comparison scorer (JudgeScorer) — now use the judge model (default
claude-opus-4-8, override CLAWMATES_JUDGE_MODEL). Execution/reasoning turns keep
using the configured default model (claude-sonnet-4-6).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adds Runtime::judge (the configured model returns ALLOW/DENY + reason,
fail-open) and wires it into the door policy behind CLAWMATES_DOOR_GOVERNOR: a
governor agent judges each outbound action and can veto exfiltration / spam /
secret-leakage, atop the deterministic rules. Realizes the self-governing-
topology path — authority decided by an agent, not a human, still audited.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
(c) policy_decide is now async with real governance: kill switch +
per-workspace hourly rate cap (audit_log count) + email recipient-domain
allowlist + a governor extension point. Still allow-all by default (autonomous).
(b) the door now supports broker-executed tools: for a broker tool it mints an
auto-approved approval + single-use grant (agent->session->run->approval->
decide), then executes via the runtime so the broker consumes the grant and
reveals the credential — the agent never holds it. Exposes slack_post.
cm-runtime gains tool_broker_executed + tool_preview accessors.
4 door unit tests + clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Tool-free ZeroClaw agents get one actuator: an MCP server (POST /mcp, JSON-RPC
2.0, protocol 2024-11-05) that fronts the existing §15 machinery. The human
approver is replaced by an automated policy (default allow-all → agents are
autonomous; CLAWMATES_DOOR_POLICY=deny is a kill switch), but the governed parts
stay: every action is journaled to the append-only audit log, actions execute
through the runtime's gated-tool path, and broker credential-custody is wired in
for v2 tools. Synchronous execution returns the real result inline.
- crates/cm-api/src/mcp_door.rs: initialize/tools.list/tools.call handler; auth
(bearer -> workspace), classify effects, policy auto-decide, execute, audit.
v1 exposes email_send (-> outbox); slack/pay (broker+grant chain) is next.
- crates/cm-runtime: Runtime::{tool_descriptor_json, tool_gate_category,
execute_door_tool} — door-facing entry that builds ToolContext and consumes a
grant for runtime-executed gated tools.
- deploy/clawmates-runtime: agents now carry mcp_bundles=["clawmates_door"];
[[mcp.servers]] points at the door (bearer injected at deploy, not committed).
Validated live on gw-04: tools/call email_send -> isError:false + outbox row +
"agent|door.executed" audit, no human. 4 door unit tests + clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
ZeroClaw pairing codes are single-use, so a static ZEROCLAW_PAIRING_CODE only
works for the first run. Add ZEROCLAW_TOKEN: pair once out-of-band, set the
durable bearer, and runs are repeatable. PAIRING_CODE stays as a fallback; one
of the two is required.
Validated live on gw-04: a single-node pipeline driven through the deployed
authed POST /api/topologies/run drove a real ZeroClaw role-agent over /ws/chat
and returned a RunRecord with real output + token metering, persisted to
topology_runs.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
cm-orchestrator owns the topology graph; each turn now drives a real ZeroClaw
role-agent in a container via the proven gateway drive recipe, instead of a
tool-free cm-llm call.
- crates/cm-api/src/topology_exec.rs: ZeroClawDriveExecutor impl TurnExecutor —
pair (POST /pair + X-Pairing-Code, token cached) -> ws /ws/chat?agent=<alias>
-> send {type:message,content} -> drain chunk/done/approval_request/error.
approval_request is recorded as a BLOCKED GatedAction, never auto-approved (§15).
Role->alias via ZEROCLAW_AGENT_MAP, fallback ZEROCLAW_DEFAULT_AGENT (scout).
- POST /api/topologies/run {task,graph} -> execute() -> RunRecord, persisted
best-effort to the existing topology_runs table (no migration). compare stays
tool-free. from_env() is read in-handler so cm-api still boots unset.
- deploy/clawmates-runtime: example config now declares a tool-free multi-agent
role-cast; README documents the ZEROCLAW_* knobs + run endpoint.
tokio-tungstenite 0.26 (already in lock) + dev axum `ws` for the hermetic test.
3 lib tests green, clippy clean, SQLX_OFFLINE build clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Save each comparison and let users reload past ones.
- migration 0008: topology_runs (workspace-scoped; full comparison as JSONB).
- cm-db repo::topology_runs (insert / list_recent / get) + regenerated .sqlx.
- cm-api: compare persists best-effort (never loses the LLM result on a DB
hiccup); GET /api/topology-runs (recent) + GET /api/topology-runs/{id}.
Integration test asserts persist → list → get.
- frontend: "Recent comparisons" list on the Compare tab; click to reload a
saved run. e2e p8 green (39 suite); offline build + clippy clean.
Server self-migrates at boot (cm_db::MIGRATOR), so 0008 applies on deploy.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Run a task across topologies via the API and return a leaderboard + quality/cost
Pareto front. The handler builds a ProviderExecutor + JudgeScorer over the
runtime's configured provider/model, then calls cm_orchestrator::compare.
- cm-runtime: expose provider()/model()/max_tokens() accessors on Runtime.
- cm-api: depend on cm-orchestrator (provider feature); add the compare route.
- Integration test runs a 2-topology comparison through the real server
(scripted provider) → 200 with results + leaderboard. 4 topology tests green;
offline build + clippy clean.
The topology endpoints (catalog/classify/build/compare) ship to gw-04 with the
upcoming ReactFlow UI in one server+frontend redeploy.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Stateless, auth-gated routes backing the topology builder UI:
- GET /api/topologies — catalog of the 12 kinds + descriptions + role mix
- POST /api/topologies/classify — infer a kind + metrics from a posted graph
- POST /api/topologies/build — build a canonical graph from {kind, roles}
Add Serialize to cm-topology Classification/GraphMetrics; add ApiError::BadRequest
(400) for invalid build input; add cm-topology dep. 3 integration tests
(catalog/build/auth) green against Postgres; offline build + clippy clean.
No DB or provider yet — running/comparing topologies is a later provider-backed
endpoint. Server redeploy will batch with the ReactFlow UI that consumes these.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
cm-topology::build(kind, roles) instantiates a canonical graph for any of the 12
kinds from a role list (the template catalog + the search generator).
cm-orchestrator::evolve searches a (kinds × team-size) grid using the comparison
machinery as fitness: build a candidate per cell, run the task, score it, and
keep a MAP-Elites-style archive of per-cell elites + a quality/cost Pareto front
and the global best. evolve_all() covers every kind at full size. This is the
bridge toward Autonomous Organizational Evolution on a safe substrate — every
candidate still executes via safe turns (§15 invariant holds).
Demoed in topology_bench (auto-picks the best topology + Pareto kinds).
cm-topology 20 tests; cm-orchestrator 17 (--features provider); clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Milestone-A second capability: chain whole topology runs in sequence, threading
each stage's final output into the next stage's task (e.g. swarm brainstorm →
hierarchical execute → debate review). Each stage is a full safe topology run,
so §15 holds at every step. WorkflowRecord aggregates per-stage RunRecords +
totals. Demonstrated in the benchmark example. 14 tests; clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Every TopologyKind now runs, mapped to five execution patterns:
- hierarchical ← hub_spoke, star_moe, market
- pipeline ← ring
- swarm ← flat, holacratic
- mesh (new) ← blackboard (two peer-exchange rounds + aggregate)
- debate (new) (propose → critique → revise → judge)
execute()'s match is now exhaustive (adding a kind upstream forces an executor),
so the Unsupported error is gone. Benchmark spans all five distinct patterns.
14 tests with --features provider; clippy clean. Doc updated.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
`cargo run -p cm-orchestrator --example topology_bench --features provider`
runs one task across hierarchical/pipeline/swarm topologies and prints a
leaderboard + quality/cost Pareto front. Offline-deterministic via the scripted
provider; set ANTHROPIC_API_KEY to run against a real model. Demonstrates
milestone B end-to-end and seeds the reproducible paper harness.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Make Scorer async and add JudgeScorer (behind the `provider` feature): asks a
cm-llm model to rate a run's output 0-100 vs the task and normalizes to [0,1],
giving the comparison harness real quality numbers. Robust integer parsing
(handles "Score: 92/100", clamps >100); provider errors score 0.0.
11 tests with --features provider (judge incl. parse + scripted-provider score);
core stays 7. Clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
