Third commit of the Research + Loops arc. Lights up loops as durable
recurring topology executions:
GET /api/loops list workspace's loops
POST /api/loops create — returns webhook_token +
signing_key ONCE when webhook trigger
is enabled; never exposed again
GET /api/loops/:id detail
PATCH /api/loops/:id update definition
DELETE /api/loops/:id delete
POST /api/loops/:id/run trigger one iteration NOW
POST /api/loops/:id/enable set enabled=true
POST /api/loops/:id/disable set enabled=false
POST /webhooks/loops/:token public; HMAC-SHA256-verified
Scheduler (cm_runtime::spawn_loop_scheduler) wakes every 10s, queries the
partial index on (next_fire_at) for due loops, enqueues one topology_runs
row per fire with loop_id + iteration + parent_run_id chained back to the
previous iteration. Uses croner via the existing scheduling::next_occurrence
helper. Missed windows fire ONCE and skip the backlog — next_fire_at is
always computed strictly AFTER now(), so a late scheduler doesn't drain a
buildup.
Webhook signatures follow the same pattern as the Stripe billing webhook
(HMAC-SHA256 with constant-time hex compare). Token + signing key are
24-byte OS-RNG values; the URL uses base64-url for the token, and the
signing key is base64-std. Both surface exactly once at create time.
All three fire paths (scheduler, immediate-run, webhook) funnel through
`cm_db::repo::loops::enqueue_iteration` so the invariants stay in one
place. `iters` repeat policy is enforced by the scheduler tick; `until`
and `on_completion` land with the orchestrator hook in commit 4.
Adds cm-llm as a direct cm-api dep, getrandom for the webhook material
generator, and wires the scheduler spawn into the server binary alongside
the resume sweeper and outbox drainer.
Under concurrent door executions the broker was queueing on its 5-connection
pool, adding latency to tool calls that fanned out from the same run. Bump
the default to 8 (one connection per concurrent door before queueing) and
expose CLAWMATES_BROKER_POOL_SIZE so the fleet can tune it up as the load
grows without a rebuild.
Under a half-open TCP (server side closed, client OS still buffering
writes), the daemon's write.send() inside the tokio::select! branch
blocks forever. tokio::select does not preempt a running future, so
the whole loop freezes — idle_tick never gets to check last_rx, no
'channel ended' log ever fires, and the daemon silently spins on a
dead socket for hours.
Observed on architect Jul 5 2026: daemon connected at 12:59:13,
heartbeats worked for ~2.5 min, then went silent. TCP session showed
ESTABLISHED on the node side, no ESTABLISHED on the gateway side.
Restarting the daemon 'fixed' it — but it re-hung within minutes.
Fix: wrap both write.send() call sites in a tokio::time::timeout of
WRITE_DEADLINE=10s. If a send stalls past that, we log and return Ok()
to trigger the main-loop reconnect. Short enough that it fires long
before the 40s read-idle would (which was our only escape hatch and
never triggered because the loop was frozen).
After 'claude update' replaces the binary, the daemon's re-probe ran the fresh
binary which macOS Gatekeeper re-verifies (>2s) — the 2s probe cap missed the new
version, so the UI didn't refresh (update worked but looked stale). Bump the cap to
8s; frontend polls the tools endpoint a few times post-update to catch the re-probe.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The ↑ badge on each tool card is now a button: confirm → POST
/api/nodes/{id}/tools/{tool}/update → daemon runs the tool's own updater + re-probes.
- daemon: tool_update op (spawned task so the 170s update can't stall the WS loop;
re-probes + re-sends node_tools after). Fixed command allow-list (no arbitrary
shell): claude/glm → `claude update`; kimi → `uv tool upgrade kimi-cli`; ollama →
brew upgrade (mac) / install.sh (linux); else unsupported. 4KB output cap.
- cm-api: call_timeout/request_timeout (long ops); POST .../tools/{tool}/update
(workspace-scoped, allow-list) → {ok,output}.
- frontend: ↑latest becomes an Update button → confirm → spinner → refresh/err.
Note: claude/kimi/glm are user-space (no sudo); ollama on Linux uses install.sh
(needs sudo — works on passwordless nodes, returns an error otherwise; surfaced in UI).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Each node card now shows installed versions of Docker / Claude Code / Kimi / GLM /
Ollama (conditional per node) under the ssh card, with an "update available" badge.
- daemon: probe_tools() finds docker/claude/kimi-cli/ollama across candidate bin dirs,
extracts semver from --version, reports {"t":"node_tools",...} on connect + every 15m.
- migration node_tools + tool_latest; cm-db repo node_tools (upsert/list/latest).
- cm-api: fleet.rs NodeTools uplink → upsert; tool_versions.rs spawn_latest_checker
(24h, npm/pypi/github; docker display-only); GET /api/nodes/{id}/tools (glm mirrors
claude). Spawned in clawmates-server.
- frontend: NodeTools cards on each HostCard with the ↑latest badge.
Phase 2 (one-click update execution) intentionally deferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Survey + fixes so the pipeline passes at the Docker level (no k8s).
- Remove k8s: drop the `sandbox-k8s` job (kind/Calico/--features k8s-tests) and the
"Helm chart lints" gate step. release.yml was already k8s-clean.
- Rust job:
- `cargo fmt --all` — fix pre-existing formatting drift (fmt --check was failing).
- clippy -D warnings: fix 3 lib warnings (cm-brain sort_by_key→Reverse, cm-api
fleet.rs doc list indentation, node_rules map_or→is_none_or).
- Regenerate the .sqlx offline cache (was missing the cm-runtime run_loop test
query → offline compile failed). DB-backed tests use testcontainers at runtime.
- Set SQLX_OFFLINE=true on the rust + e2e jobs so query! macros compile against
the committed cache deterministically (no DB needed at compile time).
- Frontend job:
- Fix the 1 ESLint error (useAgentTelemetry: no setState-synchronously-in-effect;
tag the slice with agentId + derive null on mismatch).
- Fix 2 stale panel-params tests (`terminal` is a valid app id now; assert the
current APP_IDS + use a genuinely-unknown id for the reject case).
Verified locally: fmt clean, clippy --all-targets -D warnings clean (offline),
frontend lint 0 errors, tsc clean, 86/86 frontend tests pass, build OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Completes "agent on a node" (single-node): when an agent's placement points at a
fleet node, its terminal container runs there and the browser reaches it over a
direct WebRTC DataChannel (LAN speed), sharing a node-local volume with the
sandbox. gw-04-local agents are byte-identical to before.
- cm-sandbox/docker.rs: empty drive subpath → mount the whole volume at the target
(volume_options None), so a per-agent node-local volume auto-creates at ~/drives.
- cm-api/fleet.rs: NodeHub.open_pty/webrtc_offer carry optional container+session
(injected only when Some); node-terminal caller passes None (host shell unchanged).
- cm-runtime/terminals.rs: TerminalManager gains node_provider + placement
(mirrors SandboxManager, draining-aware); node_local_drive_mount(agent) =
clawmates_agent_<id> at ~/drives; placement_for() ensures + locates the container;
attach uses driver_for(node) (local byte-identical).
- cm-runtime/sandboxes.rs: a node-placed agent sandbox mounts the same per-agent
volume → shares files with the terminal on that node.
- cm-api/routes/terminal.rs: ticket response gains `node`; ws() bridges node-placed
agents through the NodeHub relay (WebRTC + fallback) execing into the container;
local path unchanged. server main wires with_node_provider.
- frontend: agentTerminalConnector mints the ticket then picks WebRTC (node-placed,
⚡ direct / relayed badge) vs WS (local); webrtcConnector generalized to be
endpoint-agnostic (node terminal reuses it).
Known follow-up: terminal (uid 65532) and sandbox (uid 10001) share the volume but
differ in uid — cross-container writes need an aligned uid/gid (group-writable).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Generalize the daemon's PTY spawn so a session can target either the node's host
shell (today) or `docker exec -it <container> tmux …` (the node-placed agent
terminal, which shares the container's node-local ~/drives). A `container` (+
optional `session`) field on pty_open/webrtc_offer selects the container path;
absent it, the host shell path is byte-identical to before — so the Infra node
terminal is unaffected. Threads PtyTarget through open_pty + rtc handle_offer →
build_peer → bridge_pty. Additive; nothing emits `container` yet.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Turn the Beszel-tapped metrics into a self-managing loop.
- migration node_rules (workspace/node-scoped: metric op threshold, for_seconds,
action JSONB, last_fired).
- cm-db: repo/node_rules.rs (CRUD + list_enabled); node_metrics::eval_all merges
Beszel + heartbeat scalars per node + a headroom() heuristic; nodes::status_of;
heartbeat now PRESERVES a `draining` status across heartbeats (so a cordon sticks).
- cm-api: node_rules.rs evaluator (spawn_evaluator, 20s) — when a metric condition
holds for the rule's window it fires drain / undrain / alert (in-memory sustained
+ cooldown tracking, modeled on the node sweeper); routes/beszel.rs rules CRUD
(GET/POST/PATCH/DELETE /api/fleet/rules); spawned in clawmates-server.
- cm-runtime: placement_node() is metrics-aware — a `draining` node stops receiving
new agent sandboxes (falls back to local), so the drain rule is actionable.
- frontend: FleetRules section in the Local view — build rules (node · metric · op ·
threshold · duration → action), toggle/delete, with fired-history.
The loop: hot/overloaded node → rule drains it → placement avoids it → recovers →
undrain rule brings it back. Deployed; node_rules migration applied.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Tap each node's Beszel metrics (GPU/temps/disk-IO/network/per-container — beyond
our basic heartbeat) by reading the workspace's Beszel hub. The agents run in
WS-only mode with no locally-readable socket, so (per the de-risk) the server taps
the hub's PocketBase API instead of the daemon reading agents — no daemon changes.
- migrations: workspace_beszel (BYO hub URL + login, server-side only, mirrors the
Tailscale BYO pattern) + node_metrics (latest scalar columns + JSONB blob).
- cm-db: repo/fleet_beszel.rs, repo/node_metrics.rs; nodes SELECT joins node_metrics
(gpu_pct/temp_max surfaced on node_json for the live cards).
- cm-api: beszel.rs client (auth-with-password, poll `systems`, map to nodes by
hostname, upsert metrics) + a 15s spawn_poller; routes/beszel.rs (connect/status/
disconnect + GET /api/nodes/{id}/metrics with history proxied live from the hub).
- frontend: HostCard gains a GPU/temp readout + a Monitor button; NodeMonitor is a
full-width per-node page (current panel + CPU/mem/GPU/temp/net/disk charts from the
hub's 1m history); a "Beszel monitoring" connect form in the Local view.
Reachability confirmed: gw-04 → the hub over the tailnet (100.123.224.84:8090). Needs
the user to connect their hub login to activate the poller.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Terminal keystrokes were ~400ms because every byte relayed browser→Cloudflare→
gw-04 (Europe)→tailscale→node, even when the node is on the user's own LAN. Add a
direct browser↔node WebRTC DataChannel so co-located terminals run at LAN speed;
the gateway is reduced to signaling; the WebSocket relay stays as the automatic
fallback (graceful degradation — never worse than before).
Daemon (clawmates-node v0.4.0, new src/rtc.rs):
- Add the `webrtc` crate (reuses the ring crypto provider we already install — no
conflict). Browser is the offerer; we answer, trickle ICE back over the control
channel, and on DataChannel open spawn a host PTY (tmux) bridged DIRECTLY to the
channel. Refactor open_pty → spawn_terminal_pty shared by both transports.
iceServers: STUN + auto host/tailnet candidates (direct, no relay, for LAN/tailnet).
Server (cm-api):
- NodeConn.signal_sinks; Uplink WebRtcAnswer/WebRtcIce/WebRtcFailed routed to the
browser; NodeHub webrtc_offer/ice/close + open_session/open_pty (open_terminal
split so the PTY opens only once the transport is chosen). bridge_terminal relays
signaling over the existing ticket-authed WS and opens the relay PTY on
{type:"fallback"}.
Browser (NodeTerminalApp):
- RTCPeerConnection + reliable/ordered DataChannel; offer/answer/ICE over the WS;
2.5s race → use the DataChannel if it opens, else fall back to the WS relay.
Reconnect wraps both. A direct⚡/relayed indicator shows the live transport.
Deployed; both nodes (morpheus, tank) updated to v0.4.0 and steady online. Direct-
path proof is a browser action (the ⚡ indicator + latency); confirmable from the
daemon's [rtc] logs.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Nodes flapped online/offline and the terminal died on the first blip. WebSockets
are the right transport (outbound, NAT-friendly); the fixes harden around it.
Server (cm-api):
- Anti-clobber connection epoch: a reconnecting daemon gets a fresh epoch; a stale
run_channel's teardown only clears the hub + sets offline if it still owns the
slot — so a lingering old channel can't flip a live reconnection offline (the
main false-offline cause).
- WS keepalive: run_channel now pings every 15s and tears down if no inbound
frame (incl. pong) for 35s — dead links detected in seconds, not minutes.
- Staleness sweeper backstop: spawn_node_sweeper (8s tick / 20s window) wired in
clawmates-server, so a vanished node goes offline within ~28s even if its
channel hangs (mark_stale_offline was defined but never called).
Daemon (clawmates-node v0.3.0):
- Heartbeats off the select thread (dedicated thread owns System + blocking
docker/tailscale/disk CLIs) so a slow op never starves heartbeats/pongs.
- Each handle_frame runs on its own task; added a 40s inbound idle deadline so a
half-open socket triggers a reconnect.
Frontend:
- useNodes streams /api/nodes/live (SSE push) instead of a 3s poll; isLive()
derives online from lastSeen freshness (<15s) so a transient column flip never
shows a healthy node down.
- Node terminal: clean auto-reconnect loop (re-mint ticket -> reconnect -> tmux
re-attaches and redraws the live screen = mosh-style snap-to-state over TCP),
replacing the [disconnected] dead-end.
Mosh evaluated: harvest principles (session/transport decoupling, snap-to-state,
already given by tmux), don't adopt — UDP is incompatible with our browser+CF+NAT
topology and it's GPLv3. Removed temporary terminal debug traces + /api/debug route.
Verified: node holds steadily online (heartbeat 1-3s, no flap) and goes cleanly
offline when the daemon stops.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Root cause (from the server trace: 69 banner bytes then immediate "to_browser
ended" = pty_exit): tmux spawned and exited instantly, writing its "sessions
should be nested with care" warning to stderr (not the PTY). The operator runs
the daemon inside their own tmux, so $TMUX was inherited and the spawned tmux
refused to nest → browser saw nothing.
- daemon (v0.2.2): spawn tmux on a DEDICATED socket (`tmux -L clawmates
new-session -A -s main`) and `env_remove("TMUX")`, so it can never collide with
or be refused by the operator's tmux.
- NodeTerminalApp: debounce the ResizeObserver (150ms). The pull-out animates
open, firing the observer on every pixel — previously ~80 resize frames per
open, each fit()+SIGWINCH. Now one resize after layout settles.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The server trace showed pty_open is sent but the daemon (morpheus, v0.2.0) emits
no pty_out — so the PTY spawn was dying silently. Instrument it:
- daemon open_pty: log open/tmux/first-read/EOF/error/total to stdout, and send
an IMMEDIATE banner pty_out ("[clawmates] host shell on <host> — starting…") so
the browser confirms the relay even before the shell draws. If open_pty fails,
send the error as pty_out (was a silent pty_exit). Bump to v0.2.1.
- cm-api: temp GET /api/debug/node-pty/{id}?dbg=… opens a node terminal and reads
~2s of output with no browser/auth, to test the relay in isolation.
- NodeTerminalApp + ticket/ws routes already log each hop ([node-term]/[fleet-term]).
Diagnostic logic: banner shows + shell doesn't → relay ok, shell is the problem;
nothing shows → relay broken; daemon "EOF after N bytes" → shell exited.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- Daemon: factor the terminal command into terminal_command(); add `--selftest`
which opens the host terminal PTY locally and prints ~2.5s of raw output, so
you can confirm tmux/zsh actually draws on a given node without the browser.
(Verified locally: tmux spawns zsh + draws its status bar.)
- cm-api: temporary [fleet-term] eprintln tracing in open_terminal, the pty_out
router, and the browser bridge (byte counts + sink presence) to locate where
output stops between daemon→server→browser. To be removed once diagnosed.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The node terminal opened a bare bash PTY — bash prints its prompt once, so a
freshly-attached xterm showed nothing. Switch to `tmux new-session -A -s
clawmates` on the HOST (mirrors the agent terminal, but on the node itself, not
in a container): tmux redraws the whole screen on attach (no blank), and the
session is resumable across reopens. Starts in $HOME. Falls back to a login
shell if tmux is unavailable. ensure_tmux() best-effort installs tmux via the
host package manager when the daemon runs as root (systemd); otherwise logs a
hint to `apt install tmux`. Rebuilt + re-hosted both binaries.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Hostname/IP:
- Daemon reports the machine's hostname (sysinfo) + primary outbound IPv4 on each
heartbeat. migrations/0021 adds nodes.hostname/local_ip; cm-db heartbeat stores
them; node JSON exposes them. Cards now title on the real hostname (falling back
to name) + show the IP, instead of the "New node" placeholder. `name` stays
user-overridable (rename).
Terminal moved into the pull-out computer (no more per-card modal):
- New infra computer app NodeTerminalApp (computer/apps/infra) — xterm bridged to
a node's host shell over the node control channel, filling the app window
(mirrors the agent Terminal's layout + ResizeObserver). Added "terminal" to the
INFRA_CATALOG grid; a ?node= panel param targets a specific node (picker when
unset). Clicking Terminal on a node card now opens the infra computer to that
node's shell instead of a separate full-screen window. Deleted NodeTerminal.tsx.
Rebuilt + re-hosted both daemon binaries (hostname change).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Two bugs surfaced running the daemon on a real node:
- Linking cm-sandbox (bollard) brought a second rustls provider into the graph,
so rustls couldn't auto-pick one and panicked at the WSS TLS handshake. Install
the ring provider explicitly at startup (rustls dep + install_default()).
- The daemon auto-ran `tailscale set --ssh`, which tries to reroute the user's
live SSH session and aborts ("will result in your session disconnecting"). Now
Tailscale is only touched when an auth key is explicitly passed (opt-in), with
--accept-risk=lose-ssh to avoid the interactive abort.
Rebuilt + re-hosted both binaries (linux-amd64, darwin-arm64).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Agents can now provision their sandbox on a connected fleet node instead of the
gateway host. Local stays the strict default, so existing agents are byte-for-
byte unaffected until explicitly placed elsewhere.
Security parity: the daemon links the REAL cm-sandbox DockerDriver and runs the
typed container ops (sb_provision/sb_exec/sb_destroy/sb_health/sb_list) through
it — identical hardening (cap-drop ALL, seccomp, no-net, read-only, non-root) to
local sandboxes. cm-sandbox spec types are now Serialize/Deserialize so the spec
crosses the channel.
- cm-api: RemoteDriver (impl SandboxDriver over the node channel) + HubDriverProvider
(impl cm_runtime::NodeDriverProvider, hands out a driver only for connected
nodes via a sync online set) + NodeHub.call/is_connected. AppState.with_node_hub
so the hub is shared with the placement provider.
- cm-runtime SandboxManager: driver_for(node_id) routes by the recorded
agent_containers.node_id (local default = existing driver, identical path);
placement_node() reads the workspace setting and falls back to local if the
node is offline; exec/release route accordingly. NodeDriverProvider trait.
- DB: 0020_workspace_placement + repo (for_agent/get/set/clear).
- main.rs: build the NodeHub first; inject HubDriverProvider into the agent
manager + share the hub with AppState.
- API+UI: GET/PUT /api/fleet/placement + a "Run agents on: Local / <node>"
selector in the Fleet overview.
Note: a node must be able to pull the agent image (the daemon docker-pulls it);
interactive PTY for agent containers on remote nodes is not wired (Terminal app
stays local) — the in-dashboard node shell already covers host access.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Proves a connected node can host hardened agent workloads end-to-end, without
touching the agent run loop (zero blast radius on existing agents).
- Daemon: typed `sb_check` op — pulls a tiny image and runs it fully locked down
(cap-drop ALL, no-new-privileges, no network, read-only rootfs, non-root,
memory/pids caps), then tears it down. Fixed command; nothing caller-supplied
runs (preserves the exec-hardening invariant).
- cm-api: NodeHub.sandbox_check + POST /api/nodes/{id}/sandbox-check.
- UI: a shield "sandbox check" button on each online node card streams the
result (✓ SANDBOX READY + container id/uname).
This validates the full provision→run→destroy mechanism on nodes. The remaining
P2 work — wiring real agent deploys to auto-place onto nodes — is its own
subsystem (a RemoteDriver reusing the local DockerDriver for security parity,
agent-image distribution to nodes, and node-routing in SandboxManager) and is
best done as a focused pass; it is intentionally NOT bundled here to keep the
core agent path untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
You can now open a real shell on any connected node from the dashboard — the
daemon spawns a host PTY and streams it over the existing outbound control
channel (no inbound port, no Tailscale brokering needed).
Daemon:
- portable-pty host shell sessions: pty_open/pty_in/pty_resize/pty_close ops; a
reader thread streams base64 pty_out frames. Outbound frames now funnel through
one mpsc channel so PTY output and heartbeats interleave.
cm-api NodeHub:
- per-connection pty_sinks + sid multiplexing; open_terminal/terminal_input/
terminal_resize/terminal_close; in-memory single-use terminal tickets (the
browser WS can't carry a bearer, and the session is instance-local anyway).
- routes/nodes.rs: POST /api/nodes/{id}/terminal/ticket + GET .../terminal/ws
(bridges browser xterm <-> node PTY: binary = keystrokes, text = resize).
Frontend:
- NodeTerminal xterm modal (reuses the agent Terminal's xterm setup); a Terminal
button on each online node card opens a shell.
This proves the bidirectional streaming-over-channel mechanism the RemoteDriver
will reuse. Remaining P2: RemoteDriver + placement (run agent workloads on nodes).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Security hardening:
- The gateway no longer sends arbitrary shell to nodes. The WSS exec op is
replaced by a typed `verify` op the daemon runs itself (fixed host+docker
check); future container ops are typed too. cm-api NodeHub.verify() + the
daemon's handle_command only dispatches vetted ops.
BYO Tailscale:
- migrations/0019_workspace_tailscale.sql + cm-db fleet_tailscale repo (store the
user's Tailscale API key + tailnet, server-side only).
- cm-api routes/tailscale.rs: POST/GET/DELETE /api/fleet/tailscale + GET
/api/fleet/tailscale/devices (proxies api.tailscale.com device list).
- Daemon: --tailscale-authkey → `tailscale up --authkey … --ssh` (enables
Tailscale SSH for keyless user access); else `tailscale set --ssh=true`. Reports
its tailscale IP (already).
UI:
- Fleet overview gains a Tailscale section: connect (key+tailnet) + live tailnet
device status (online/last-seen/IP/os). Node cards show a copyable Tailscale SSH
target (ssh <ip>).
Remaining: P2 — RemoteDriver + placement (run agents on nodes) and the in-UI
remote terminal (PTY proxied over the WSS channel).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Decouples "many users" + "many server replicas" from "many machines" so the
platform is tenant-isolated and horizontally safe on the current single node.
- Per-signup workspaces (cm-auth): a new hosted-identity sign-in provisions and
owns its own workspace instead of joining the first. Config-gated by
auth.per_signup_workspace (default off); concurrent first-logins serialized by
a per-subject advisory lock so no duplicate workspaces.
- Terminal tickets in Postgres (migration 0016, hashed, single-use): any replica
can redeem a ticket minted by another. Drops the in-process ticket map.
- Container registry in Postgres (migration 0017, agent_containers): Terminal
and Sandbox managers resolve an agent's container through a shared registry,
so a 2nd replica reuses it instead of spawning a duplicate. node_id recorded
as 'local' (Phase 2 hook). Boot reconcile removes only true orphans, so
terminals now survive a redeploy (tmux sessions resume).
- Per-workspace quotas (cm-api/quota.rs): plan-tier caps on agents + live
containers, enforced at agent create + terminal spin-up (reconnects allowed),
returned as HTTP 402. New GET /api/quota surfaces usage vs limits.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Terminal app (xterm ⇄ WebSocket ⇄ per-agent themed container):
- zsh + oh-my-zsh + powerlevel10k image (agent-terminal), runs as uid 65532 to
share read-write ownership of the file-drive volume with the server.
- Interactive PTY in cm-sandbox (bollard exec tty/attach + resize) + a
TerminalManager; ticket-authed WS bridge routed straight to the backend via a
Traefik PathRegexp(/ws) rule. MOTD greets the user by name.
- tmux resumable sessions; multi-tab (one tmux session per tab, same container),
drag-to-reorder, rename, and a Save that persists named tabs to the server
(terminal_tabs, migration 0014) so they survive logout / a new device.
- Files drives mounted per-agent (subpath) at ~/drives/{documents,received,
shared}; a reconciler keeps the Files app's index in sync with terminal writes.
Storage moved to a shared `filedata` volume (CLAWMATES_STORAGE__DATA_DIR).
Obsidian vault (a markdown "second brain" per agent):
- New `vault` FileDrive (migration 0015) mounted into the terminal at ~/obsidian;
a file-content read route; a purple Obsidian tile + a vault viewer app.
Computer UI:
- Draggable computer-panel width (min = phone preset) keeping the size presets.
- Green Terminal glyph, "Claw Chat" → "Chat", colored gradient-outline app icons.
- Agent page: avatar↔activity-grid spacing + larger, uniform section fonts with
colored section-tinted tag chips.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The §15 door's email_send queues to `outbox` but nothing delivered it. Add a
real transport: cm-db outbox repo (list_queued/mark_sent/mark_failed) + a
cm-runtime drainer — an EmailSender trait (testable), a lettre STARTTLS
LettreSender, drain_once (queued -> sent/failed), and spawn_drainer wired into
the server beside the scheduler/sweeper/topology-worker.
Config-gated: inert (logs "outbox delivery DISABLED") until CLAWMATES_SMTP_*
is set, so it ships safely before credentials exist. The agent never holds the
SMTP credential — it only writes to outbox through the gated door; the server
owns the transport.
NOTE: live delivery is still credential-blocked — Migadu's API can't send
(SMTP-only) and the admin token is invalid; no SMTP creds exist. The transport
is built + tested (drain_marks_sent_and_failed via a mock sender); set
CLAWMATES_SMTP_* to go live with zero further code. clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
#2 GLM judge (closes#114): registry NamedProvider gains a `format` field;
build_provider_registry builds an AnthropicProvider for format="anthropic".
GLM's coding/OpenAI endpoint is ToS-throttled for raw SDK, but its Anthropic
endpoint (api.z.ai/api/anthropic) accepts raw API calls (verified x-api-key
-> glm-4.7), so CLAWMATES_JUDGE_MODEL=glm:glm-4.7 routes the door governor /
topology judge through GLM with no runtime-routing. (Kimi-as-judge still needs
a Platform key — coding key is agent-only.)
#3 run-control: POST /api/topology-runs/{id}/cancel (workspace-scoped,
queued/running only); the worker honors it at the step boundary (checks
current_status in the checkpoint callback) and won't clobber a cancel with
`failed`. Frontend Run tab gains a Cancel button and clickable recent runs
that deep-link into a live/replayed stream (SSE replays from checkpoint).
cm-* tests (incl. new cancel test) + clippy + frontend lint/typecheck green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
POST /api/topologies/run now ENQUEUES a durable job and returns 202
{run_id, status:queued} instead of executing inside the HTTP request — the
prerequisite for long-horizon runs (no client/proxy/LB timeout, survives
restarts).
topology_worker: a spawned loop that requeues stale running jobs, claims the
next queued one (CAS via FOR UPDATE SKIP LOCKED), drives it through
execute_resumable, and checkpoints RunProgress after every step; on crash the
stale sweep requeues it and the next claim resumes from the last checkpoint.
Wired into server startup beside the scheduler + resume sweeper.
GET /api/topology-runs/{id} now reports lifecycle status/kind/error/checkpoint
+ the result blob (kept the `comparison` field name for back-compat with the
compare UI; null until completed). list_runs includes status + kind.
Tests: durable lifecycle (enqueue→claim→checkpoint→complete) + stale-requeue
resume, both green; p0 endpoints (compare path) unchanged. 13 + 2 tests pass,
clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adds a multi-provider registry so judges and topology execution can run on
providers beyond the default. cm-config gains [[llm.providers]] (name, base_url,
api_key_env) — all OpenAI-compatible (GLM, Kimi/Moonshot). The server builds an
Arc<dyn LlmProvider> per entry (OpenAiCompatProvider) keyed by name; a missing
key is skipped with a warning, not a boot failure. cm-runtime RuntimeConfig
carries a ProviderRegistry; Runtime::resolve_provider("<name>:<model>") selects a
registry provider (else the default). The door governor (Runtime::judge) and the
topology compare endpoint both resolve through it, so CLAWMATES_JUDGE_MODEL and
CLAWMATES_TOPOLOGY_EXEC_MODEL accept "glm:glm-4.6" / "kimi:kimi-k2".
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- GET /api/team/orgchart — each member grouped with the claws they manage
(agents.managed_by); GET /api/team/leaderboard — every claw ranked by
its real usage_events rollup (credits/tokens/runs, zeros included).
Both tested against real Postgres.
- Stripe Buy-credits (the Slack/Clerk integration pattern): [billing]
config (stripe keys + price + webhook secret + credits_per_pack);
POST /api/credits/checkout opens a real Checkout Session; POST
/api/billing/stripe verifies Stripe's t=,v1= HMAC (constant-time) and
grants one credit lot, idempotent on the session id; GET
/api/billing/config gates the button (honest degradation when unset).
Offline tests: signed grant + replay no-double-grant + forged-sig 400 +
config flag. Live checkout creation deferred to a CM_LIVE_STRIPE test.
- /apps global page support: clawId now optional on connect + directory;
absent => workspace-wide connection (app_connections.agent_id NULL) via
new connections::list_for_workspace.
- ApiError gains a From<sqlx::Error> so inline queries use ? cleanly.
cm-api 10 test files incl. team_tabs (2) + stripe_billing (3); clippy clean.
Co-Authored-By: Claude Fable 5 <[email protected]>
Made the Docker Compose route turn-key for a real self-host, then stood
the whole stack up and drove a live chat through it.
- First-owner bootstrap (cm-auth::bootstrap_owner): a fresh local-auth
install has no users and no signup route, so the initial Owner +
workspace are provisioned ONCE from CLAWMATES_BOOTSTRAP_* env on first
boot — idempotent, never clobbers an existing install (keys on 'any
workspace exists'). Two real-Postgres tests (creates + signs in;
second call is a no-op). Wired into server boot, guarded on a
non-empty password
- deploy/compose/README.md: full production bring-up — services, the
security topology, every config knob, Anthropic vs local-LLM, the
broker-key backup, ops, and TLS/SSE proxy notes
- .env.example fleshed out (bootstrap, LLM, auth mode, OTLP); compose
uses optional env_file so only the knobs you set are injected (unset
options never override clawmates.toml with empty strings)
- volume-init one-shot chowns the broker's named volumes so the non-root
scratch broker can write its socket + generated master key
Deployed locally and verified end to end: all 5 containers healthy,
broker generated its key, server bootstrapped owner@…, login + /api/user/me
work, and a real message streamed a live Anthropic response through the
gateway. Captured screenshots of login, workspace home, chat, and the
Computer panel.
166 Rust tests (+2 bootstrap).
Co-Authored-By: Claude Fable 5 <[email protected]>
Full-depth rename per the approved plan; the 'claw' product vocabulary
(claws, /claws routes, clawId, Claw Chat) stays — it is now the brand.
- Display brand: Clawmates (manifest, titles, hero, login/rail logo
'clawmates'); default host app.clawmates.work; registry
ghcr.io/clawmates
- Crates tc-* -> cm-* (16 crates + all imports); binaries
clawmates-server/broker/bundler; images clawmates/*; env prefix
CLAWMATES_* (+ CM_TEST_DATABASE_URL / CM_LIVE_LLM); config
clawmates.toml; helm chart deploy/helm/clawmates with clawmates-*
resources; db names clawmates*; sockets /run/clawmates; cookie
cm_session; kind cluster clawmates-test; seccomp node profile
clawmates-agent-profile.json
- All 9 Playwright brand assertions updated in lockstep; historical
spec document left untouched as the only remaining 'TeamClaw'
- Local env migrated: dev pg clawmates-dev-pg/clawmates_dev, shared
test server clawmates-test-pg, kind cluster recreated with image +
profile, compose images rebuilt under clawmates/*
Verified end to end: 161 Rust + 68 frontend tests, 29 Playwright
journeys, 4 live kind tests, helm/install/LOC/placeholder gates, and
the clean-room install rehearsal serving the clawmates login page from
a signed bundle of the rebuilt images.
Co-Authored-By: Claude Fable 5 <[email protected]>
- tc-telemetry: fmt subscriber always; with [telemetry] otlp_endpoint
set, spans batch-export over OTLP/HTTP. Tested against a REAL OTLP
receiver decoding the actual protobuf (official proto types): the
emitted span and service.name arrive on the wire. No endpoint = no
export = no network (air-gap stance). tower-http TraceLayer gives
every API request a span
- The broker finally has its own image (images/broker.Dockerfile,
9.5MB from scratch) — the Helm chart referenced one that never
existed — and the compose deployment now RUNS the broker, sharing a
socket volume with the server (the unix-socket equivalent of the K8s
sidecar). Compose secret flows were silently dead before this
- server.Dockerfile fixes surfaced by the rehearsal: the workspace
build needs tools/ (bundler joined the workspace) and
images/seccomp/ (include_str! profile) in the build context
- scripts/rehearse-install.sh (plan: clean-VM rehearsal): assembles a
REAL signed bundle from the built images (server/frontend/broker/
postgres/socket-proxy), runs the customer path — offline verify,
docker load, compose up — and asserts /healthz plus the served login
page before teardown. Passing locally; wired as a release.yml step,
which also builds/ships the broker + socket-proxy images now
161 Rust tests + 29 journeys; clean-room rehearsal green.
Co-Authored-By: Claude Fable 5 <[email protected]>
- K8sDriver.with_localhost_seccomp(profile): sandbox pods run under the
STRICT allowlist instead of the runtime default. Proven live on kind:
the harness installs the profile onto the node, a pod runs ordinary
work as uid 10001, and unshare is kernel-denied inside the pod — the
same probe the Docker suite uses, now passing on both targets
- Helm: sandbox.seccomp=localhost renders a DaemonSet that installs the
chart-shipped profile into /var/lib/kubelet/seccomp on every node
(ConfigMap + hostPath); ci/check-helm.sh enforces the chart copy stays
byte-identical to images/seccomp/agent-profile.json and asserts the
hardened render (DaemonSet + profile + HPA)
- server HPA (autoscaling/v2, CPU target) behind
server.autoscaling.enabled
- SandboxManager.warm(n): a background warmer keeps n pre-provisioned
sandboxes ready so an agent's first exec skips container startup;
unhealthy pool entries are discarded, reuse never drains the pool,
shutdown destroys assigned AND pooled. [sandbox] warm_pool config
(default 0). Real-Docker test: prefill -> assign -> refill -> reuse ->
clean shutdown
160 Rust tests + 4 live kind tests.
Co-Authored-By: Claude Fable 5 <[email protected]>
- tc-auth JwtVerifier: OIDC discovery -> JWKS, RS256 with the issuer
pinned, 5s leeway (the crate's default 60s would double the life of
Clerk's 60s session tokens), key cache with one refresh on unknown kid
(Clerk rotates). Serves auth.mode = clerk AND generic oidc — a Clerk
instance IS an OIDC issuer, so one verifier covers both
- AuthService.authenticate dispatches: JWT-shaped bearers take the
hosted-identity path, everything else stays a local opaque session.
External users JIT-provision keyed by the stable sub claim
(users.auth_subject, unique partial index in migration 0007); an
existing local account with the same email is LINKED, not duplicated;
role tracks the issuer claim every request (org:admin -> Owner)
- Config auth.mode = "clerk" (requires issuer_url; validated), server
pins the issuer at boot, Helm values/configmap accept mode=clerk
- Tests with REAL crypto, no mocks: fresh RSA keypairs, a live local
issuer publishing real discovery + JWKS docs, Clerk-shaped tokens —
JIT + role mapping, repeat-subject no-dup, expired refused (leeway
regression), wrong-key forgery refused, foreign issuer refused, and
the full router round trip with Authorization: Bearer <session JWT>
- docs/clerk.md: dashboard session-token customization (email + org
role claims), config, @clerk/nextjs getToken() wiring, what CI proves
157 Rust + 63 frontend tests + 29 journeys. Air-gapped installs keep
local auth — Clerk is a cloud-only alternative, not a replacement.
Co-Authored-By: Claude Fable 5 <[email protected]>
- SandboxSpec gains an egress flag (default false — the kernel suite
still proves zero-network for agent sandboxes). Egress-enabled
containers exist ONLY for the browser: no credentials, no broker
route, bridge network with host-gateway alias for local test pages
- images/agent-browser: Alpine Chromium, uid 10001, setuid bits
stripped — same non-root hardening as agent-base
- browser.goto tool: headless chromium --dump-dom in the agent's
browser container; HTML stripped to readable text (4k cap) and
returned with output_taint=web; viewport screenshot captured,
base64'd out of the container, stored in the blob store
- Taint semantics tightened: the step that PRODUCED untrusted output
now carries its own taint (recorded before the step row), not just
later steps — chat.inbox test updated to the stricter §15 reading
- GET /api/claws/{id}/browser/viewport.png serves the latest capture;
BrowserApp polls it and renders the live viewport (spec §7.1),
keeping the empty state until the agent has browsed
- Proven end to end with REAL Chromium against a REAL local page:
content 'Revenue up 14 percent' returned tainted web; the gated
email.send that follows carries 'web' in its approval taint_sources
(untrusted content can never quietly reach outward); screenshot
verified by PNG magic bytes
152 Rust tests + 63 frontend + 27 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- SandboxManager (tc-runtime): one container per agent, provisioned
lazily on first use, reused for the manager's lifetime, replaced
transparently if dead, destroyed on shutdown
- shell.exec tool: sh -lc inside the agent's sandbox; stdout/stderr/
exit_code return to the model as the step output. No external effects
declared — the sandbox boundary (uid 10001, no caps, seccomp
allowlist, read-only rootfs, zero egress) is the §15 control here,
not an approval gate
- RuntimeConfig.sandboxes (+ with_sandboxes builder); [sandbox] config
{image, enabled}; the server connects the Docker driver at boot and
tolerates an absent engine (shell.exec reports it per-call)
- Tests with the REAL DockerDriver: a scripted run executes two
commands — output proves uid 10001 from inside, and /home/agent state
written by the first call is read by the second (same sandbox); a
deployment without a sandbox runtime records honest error steps and
the run still completes
151 Rust tests + 27 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- S3BlobStore (object_store, path-style) behind the same BlobStore trait,
tested against a REAL MinIO container (round trip, overwrite, NotFound
on get and delete, nested keys); [storage] backend=local|s3 config with
validation + server-side selection (S3 creds via env overlay)
- Helm chart: server pod with the secret broker as a SIDECAR sharing a
private emptyDir unix socket (no network hop carries credentials),
frontend, optional local PVC vs S3, OIDC/oauth values, unbuffered-SSE
ingress annotations, NetworkPolicies (frontend->server only), hardened
securityContexts; ci/check-helm.sh lints AND asserts the rendered
topology properties
- deploy/airgapped/install.sh: offline signature+checksum verification via
the bundled teamclaw-bundler BEFORE any docker load; --verify-only mode;
ci/test-install.sh rehearses clean/tampered/wrong-key paths with the
real binary
- CI: helm gate + installer rehearsal wired in
149 Rust tests; helm lint + rendered assertions green; installer
verify-path rehearsal green.
Co-Authored-By: Claude Fable 5 <[email protected]>
- LlmEvent::Usage across all three providers (Scripted deterministic
word-count accounting; Anthropic message_start/delta usage; OpenAI-compat
stream_options include_usage)
- tc-billing: ceil(tokens/1000) min 1 credit; lots drain oldest-first under
FOR UPDATE; balance clamps at zero while the usage ledger records the
full obligation; promo codes redeem exactly once via CAS (migration 0006)
- Runtime charges every completed run (billing failure never fails a run);
proven: 1 token in + 3 out -> 1 credit deducted
- API: GET /api/team/usage, POST /api/credits/redeem (409 on reuse, audited)
- Credits page: balance, 7-day usage meter with runway estimate, PromoRedeem
- /claws/new is the full §9 wizard: ?step=identity|access|slack deep-linked
progress, accent swatches + name randomizer, access toggles, optional
Slack step, explicit review-and-confirm (creation = live agent), animated
provisioning state -> straight into chat
- E2E: chat decrements the visible balance and fills the usage meter;
WELCOME500 adds exactly 500 once then refuses; wizard round trip
140 Rust + 63 frontend tests + 23 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- migration 0005 oauth_states: one-time states (10-min TTL), consumed by a
CAS DELETE on callback — replays and forgeries both 404
- POST /api/apps/oauth/start: OIDC discovery on the configured issuer (or
the custom MCP issuer for authType=mcp_oauth), state row, authorize URL
- GET /api/apps/oauth/callback: code exchanged at the REAL token endpoint
(client id+secret form POST); the access token goes straight to the
broker (test proves it never appears unencrypted in Postgres); connection
row + audit; redirects to the claw's Add Apps panel
- [oauth] config (issuer/client/redirect_base) wired through AppState
- Tests against a real local IdP server (discovery + validating token
endpoint): full round trip, broker-held token, replay/forged state
refused, bad code fails exchange, mcp_oauth uses the custom issuer while
plain oauth refuses without a configured IdP
- AddAppsApp: live connection badges + inline API-key connect per app
(E2E: connect Notion by key from the directory)
136 Rust + 63 frontend tests + 21 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- app_connections repo; POST /api/apps/connect (keys/basic): the credential
goes to the secret broker over its socket and only the encrypted ref lands
in the row; disconnect endpoint; /api/apps directory merged with live
connection status; audit rows for connect/disconnect
- Broker protocol: InvokeHttp carries a JSON body
- slack.post tool (SendsExternally -> gated): marked broker_executed — the
runtime skips its own grant consumption and the BROKER independently
verifies + consumes the single-use grant, then calls Slack with the bot
token injected; the runtime never sees the credential
- Config: [broker] socket_path + [slack] base_url; e2e harness spawns the
real teamclaw-broker daemon and the server hosts an e2e-only /__slack sink
- SlackApp: Connection tab stores the token via the broker; connected state
- Integration test: blocked while pending -> approved -> sink received
exactly one post with 'Bearer xoxb-test-token' -> grant replay refused
- E2E journey: connect Slack in the panel -> gated post card with preview ->
sink empty while pending -> approve -> exactly one post, queue clear
133 Rust + 63 frontend tests + 21 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
