Nodes flapped online/offline and the terminal died on the first blip. WebSockets
are the right transport (outbound, NAT-friendly); the fixes harden around it.
Server (cm-api):
- Anti-clobber connection epoch: a reconnecting daemon gets a fresh epoch; a stale
run_channel's teardown only clears the hub + sets offline if it still owns the
slot — so a lingering old channel can't flip a live reconnection offline (the
main false-offline cause).
- WS keepalive: run_channel now pings every 15s and tears down if no inbound
frame (incl. pong) for 35s — dead links detected in seconds, not minutes.
- Staleness sweeper backstop: spawn_node_sweeper (8s tick / 20s window) wired in
clawmates-server, so a vanished node goes offline within ~28s even if its
channel hangs (mark_stale_offline was defined but never called).
Daemon (clawmates-node v0.3.0):
- Heartbeats off the select thread (dedicated thread owns System + blocking
docker/tailscale/disk CLIs) so a slow op never starves heartbeats/pongs.
- Each handle_frame runs on its own task; added a 40s inbound idle deadline so a
half-open socket triggers a reconnect.
Frontend:
- useNodes streams /api/nodes/live (SSE push) instead of a 3s poll; isLive()
derives online from lastSeen freshness (<15s) so a transient column flip never
shows a healthy node down.
- Node terminal: clean auto-reconnect loop (re-mint ticket -> reconnect -> tmux
re-attaches and redraws the live screen = mosh-style snap-to-state over TCP),
replacing the [disconnected] dead-end.
Mosh evaluated: harvest principles (session/transport decoupling, snap-to-state,
already given by tmux), don't adopt — UDP is incompatible with our browser+CF+NAT
topology and it's GPLv3. Removed temporary terminal debug traces + /api/debug route.
Verified: node holds steadily online (heartbeat 1-3s, no flap) and goes cleanly
offline when the daemon stops.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Agents can now provision their sandbox on a connected fleet node instead of the
gateway host. Local stays the strict default, so existing agents are byte-for-
byte unaffected until explicitly placed elsewhere.
Security parity: the daemon links the REAL cm-sandbox DockerDriver and runs the
typed container ops (sb_provision/sb_exec/sb_destroy/sb_health/sb_list) through
it — identical hardening (cap-drop ALL, seccomp, no-net, read-only, non-root) to
local sandboxes. cm-sandbox spec types are now Serialize/Deserialize so the spec
crosses the channel.
- cm-api: RemoteDriver (impl SandboxDriver over the node channel) + HubDriverProvider
(impl cm_runtime::NodeDriverProvider, hands out a driver only for connected
nodes via a sync online set) + NodeHub.call/is_connected. AppState.with_node_hub
so the hub is shared with the placement provider.
- cm-runtime SandboxManager: driver_for(node_id) routes by the recorded
agent_containers.node_id (local default = existing driver, identical path);
placement_node() reads the workspace setting and falls back to local if the
node is offline; exec/release route accordingly. NodeDriverProvider trait.
- DB: 0020_workspace_placement + repo (for_agent/get/set/clear).
- main.rs: build the NodeHub first; inject HubDriverProvider into the agent
manager + share the hub with AppState.
- API+UI: GET/PUT /api/fleet/placement + a "Run agents on: Local / <node>"
selector in the Fleet overview.
Note: a node must be able to pull the agent image (the daemon docker-pulls it);
interactive PTY for agent containers on remote nodes is not wired (Terminal app
stays local) — the in-dashboard node shell already covers host access.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Decouples "many users" + "many server replicas" from "many machines" so the
platform is tenant-isolated and horizontally safe on the current single node.
- Per-signup workspaces (cm-auth): a new hosted-identity sign-in provisions and
owns its own workspace instead of joining the first. Config-gated by
auth.per_signup_workspace (default off); concurrent first-logins serialized by
a per-subject advisory lock so no duplicate workspaces.
- Terminal tickets in Postgres (migration 0016, hashed, single-use): any replica
can redeem a ticket minted by another. Drops the in-process ticket map.
- Container registry in Postgres (migration 0017, agent_containers): Terminal
and Sandbox managers resolve an agent's container through a shared registry,
so a 2nd replica reuses it instead of spawning a duplicate. node_id recorded
as 'local' (Phase 2 hook). Boot reconcile removes only true orphans, so
terminals now survive a redeploy (tmux sessions resume).
- Per-workspace quotas (cm-api/quota.rs): plan-tier caps on agents + live
containers, enforced at agent create + terminal spin-up (reconnects allowed),
returned as HTTP 402. New GET /api/quota surfaces usage vs limits.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Terminal app (xterm ⇄ WebSocket ⇄ per-agent themed container):
- zsh + oh-my-zsh + powerlevel10k image (agent-terminal), runs as uid 65532 to
share read-write ownership of the file-drive volume with the server.
- Interactive PTY in cm-sandbox (bollard exec tty/attach + resize) + a
TerminalManager; ticket-authed WS bridge routed straight to the backend via a
Traefik PathRegexp(/ws) rule. MOTD greets the user by name.
- tmux resumable sessions; multi-tab (one tmux session per tab, same container),
drag-to-reorder, rename, and a Save that persists named tabs to the server
(terminal_tabs, migration 0014) so they survive logout / a new device.
- Files drives mounted per-agent (subpath) at ~/drives/{documents,received,
shared}; a reconciler keeps the Files app's index in sync with terminal writes.
Storage moved to a shared `filedata` volume (CLAWMATES_STORAGE__DATA_DIR).
Obsidian vault (a markdown "second brain" per agent):
- New `vault` FileDrive (migration 0015) mounted into the terminal at ~/obsidian;
a file-content read route; a purple Obsidian tile + a vault viewer app.
Computer UI:
- Draggable computer-panel width (min = phone preset) keeping the size presets.
- Green Terminal glyph, "Claw Chat" → "Chat", colored gradient-outline app icons.
- Agent page: avatar↔activity-grid spacing + larger, uniform section fonts with
colored section-tinted tag chips.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The §15 door's email_send queues to `outbox` but nothing delivered it. Add a
real transport: cm-db outbox repo (list_queued/mark_sent/mark_failed) + a
cm-runtime drainer — an EmailSender trait (testable), a lettre STARTTLS
LettreSender, drain_once (queued -> sent/failed), and spawn_drainer wired into
the server beside the scheduler/sweeper/topology-worker.
Config-gated: inert (logs "outbox delivery DISABLED") until CLAWMATES_SMTP_*
is set, so it ships safely before credentials exist. The agent never holds the
SMTP credential — it only writes to outbox through the gated door; the server
owns the transport.
NOTE: live delivery is still credential-blocked — Migadu's API can't send
(SMTP-only) and the admin token is invalid; no SMTP creds exist. The transport
is built + tested (drain_marks_sent_and_failed via a mock sender); set
CLAWMATES_SMTP_* to go live with zero further code. clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
#2 GLM judge (closes#114): registry NamedProvider gains a `format` field;
build_provider_registry builds an AnthropicProvider for format="anthropic".
GLM's coding/OpenAI endpoint is ToS-throttled for raw SDK, but its Anthropic
endpoint (api.z.ai/api/anthropic) accepts raw API calls (verified x-api-key
-> glm-4.7), so CLAWMATES_JUDGE_MODEL=glm:glm-4.7 routes the door governor /
topology judge through GLM with no runtime-routing. (Kimi-as-judge still needs
a Platform key — coding key is agent-only.)
#3 run-control: POST /api/topology-runs/{id}/cancel (workspace-scoped,
queued/running only); the worker honors it at the step boundary (checks
current_status in the checkpoint callback) and won't clobber a cancel with
`failed`. Frontend Run tab gains a Cancel button and clickable recent runs
that deep-link into a live/replayed stream (SSE replays from checkpoint).
cm-* tests (incl. new cancel test) + clippy + frontend lint/typecheck green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
POST /api/topologies/run now ENQUEUES a durable job and returns 202
{run_id, status:queued} instead of executing inside the HTTP request — the
prerequisite for long-horizon runs (no client/proxy/LB timeout, survives
restarts).
topology_worker: a spawned loop that requeues stale running jobs, claims the
next queued one (CAS via FOR UPDATE SKIP LOCKED), drives it through
execute_resumable, and checkpoints RunProgress after every step; on crash the
stale sweep requeues it and the next claim resumes from the last checkpoint.
Wired into server startup beside the scheduler + resume sweeper.
GET /api/topology-runs/{id} now reports lifecycle status/kind/error/checkpoint
+ the result blob (kept the `comparison` field name for back-compat with the
compare UI; null until completed). list_runs includes status + kind.
Tests: durable lifecycle (enqueue→claim→checkpoint→complete) + stale-requeue
resume, both green; p0 endpoints (compare path) unchanged. 13 + 2 tests pass,
clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adds a multi-provider registry so judges and topology execution can run on
providers beyond the default. cm-config gains [[llm.providers]] (name, base_url,
api_key_env) — all OpenAI-compatible (GLM, Kimi/Moonshot). The server builds an
Arc<dyn LlmProvider> per entry (OpenAiCompatProvider) keyed by name; a missing
key is skipped with a warning, not a boot failure. cm-runtime RuntimeConfig
carries a ProviderRegistry; Runtime::resolve_provider("<name>:<model>") selects a
registry provider (else the default). The door governor (Runtime::judge) and the
topology compare endpoint both resolve through it, so CLAWMATES_JUDGE_MODEL and
CLAWMATES_TOPOLOGY_EXEC_MODEL accept "glm:glm-4.6" / "kimi:kimi-k2".
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- GET /api/team/orgchart — each member grouped with the claws they manage
(agents.managed_by); GET /api/team/leaderboard — every claw ranked by
its real usage_events rollup (credits/tokens/runs, zeros included).
Both tested against real Postgres.
- Stripe Buy-credits (the Slack/Clerk integration pattern): [billing]
config (stripe keys + price + webhook secret + credits_per_pack);
POST /api/credits/checkout opens a real Checkout Session; POST
/api/billing/stripe verifies Stripe's t=,v1= HMAC (constant-time) and
grants one credit lot, idempotent on the session id; GET
/api/billing/config gates the button (honest degradation when unset).
Offline tests: signed grant + replay no-double-grant + forged-sig 400 +
config flag. Live checkout creation deferred to a CM_LIVE_STRIPE test.
- /apps global page support: clawId now optional on connect + directory;
absent => workspace-wide connection (app_connections.agent_id NULL) via
new connections::list_for_workspace.
- ApiError gains a From<sqlx::Error> so inline queries use ? cleanly.
cm-api 10 test files incl. team_tabs (2) + stripe_billing (3); clippy clean.
Co-Authored-By: Claude Fable 5 <[email protected]>
Made the Docker Compose route turn-key for a real self-host, then stood
the whole stack up and drove a live chat through it.
- First-owner bootstrap (cm-auth::bootstrap_owner): a fresh local-auth
install has no users and no signup route, so the initial Owner +
workspace are provisioned ONCE from CLAWMATES_BOOTSTRAP_* env on first
boot — idempotent, never clobbers an existing install (keys on 'any
workspace exists'). Two real-Postgres tests (creates + signs in;
second call is a no-op). Wired into server boot, guarded on a
non-empty password
- deploy/compose/README.md: full production bring-up — services, the
security topology, every config knob, Anthropic vs local-LLM, the
broker-key backup, ops, and TLS/SSE proxy notes
- .env.example fleshed out (bootstrap, LLM, auth mode, OTLP); compose
uses optional env_file so only the knobs you set are injected (unset
options never override clawmates.toml with empty strings)
- volume-init one-shot chowns the broker's named volumes so the non-root
scratch broker can write its socket + generated master key
Deployed locally and verified end to end: all 5 containers healthy,
broker generated its key, server bootstrapped owner@…, login + /api/user/me
work, and a real message streamed a live Anthropic response through the
gateway. Captured screenshots of login, workspace home, chat, and the
Computer panel.
166 Rust tests (+2 bootstrap).
Co-Authored-By: Claude Fable 5 <[email protected]>
Full-depth rename per the approved plan; the 'claw' product vocabulary
(claws, /claws routes, clawId, Claw Chat) stays — it is now the brand.
- Display brand: Clawmates (manifest, titles, hero, login/rail logo
'clawmates'); default host app.clawmates.work; registry
ghcr.io/clawmates
- Crates tc-* -> cm-* (16 crates + all imports); binaries
clawmates-server/broker/bundler; images clawmates/*; env prefix
CLAWMATES_* (+ CM_TEST_DATABASE_URL / CM_LIVE_LLM); config
clawmates.toml; helm chart deploy/helm/clawmates with clawmates-*
resources; db names clawmates*; sockets /run/clawmates; cookie
cm_session; kind cluster clawmates-test; seccomp node profile
clawmates-agent-profile.json
- All 9 Playwright brand assertions updated in lockstep; historical
spec document left untouched as the only remaining 'TeamClaw'
- Local env migrated: dev pg clawmates-dev-pg/clawmates_dev, shared
test server clawmates-test-pg, kind cluster recreated with image +
profile, compose images rebuilt under clawmates/*
Verified end to end: 161 Rust + 68 frontend tests, 29 Playwright
journeys, 4 live kind tests, helm/install/LOC/placeholder gates, and
the clean-room install rehearsal serving the clawmates login page from
a signed bundle of the rebuilt images.
Co-Authored-By: Claude Fable 5 <[email protected]>
