Root cause of the "reuse must not drain the pool" flake: the test called
`manager.exec(AgentId::new(), ...)` with a random UUID that had no agents
row. `agent_containers::upsert` uses `INSERT ... FROM agents WHERE a.id = $1`,
which silently inserts zero rows when no agent matches — so the "assigned"
sandbox never persisted to the DB. The next exec's reuse lookup returned
None, fell through to the provision branch, and popped from the warm pool
instead of reusing the assigned sandbox. When the warmer hadn't refilled by
the time we asserted, pool_size == 1 instead of 2.
Seed a workspace + owner + agent up front (mirrors soak.rs's setup). Now
upsert commits a real row, the second exec hits the reuse branch, and the
pool stays whole — the test asserts what its name claims.
The tolerant-health-check change in cm-runtime (15cffba) stays as a
defensive improvement for prod under docker daemon load, but the real fix
for the test is here.
Survey + fixes so the pipeline passes at the Docker level (no k8s).
- Remove k8s: drop the `sandbox-k8s` job (kind/Calico/--features k8s-tests) and the
"Helm chart lints" gate step. release.yml was already k8s-clean.
- Rust job:
- `cargo fmt --all` — fix pre-existing formatting drift (fmt --check was failing).
- clippy -D warnings: fix 3 lib warnings (cm-brain sort_by_key→Reverse, cm-api
fleet.rs doc list indentation, node_rules map_or→is_none_or).
- Regenerate the .sqlx offline cache (was missing the cm-runtime run_loop test
query → offline compile failed). DB-backed tests use testcontainers at runtime.
- Set SQLX_OFFLINE=true on the rust + e2e jobs so query! macros compile against
the committed cache deterministically (no DB needed at compile time).
- Frontend job:
- Fix the 1 ESLint error (useAgentTelemetry: no setState-synchronously-in-effect;
tag the slice with agentId + derive null on mismatch).
- Fix 2 stale panel-params tests (`terminal` is a valid app id now; assert the
current APP_IDS + use a genuinely-unknown id for the reject case).
Verified locally: fmt clean, clippy --all-targets -D warnings clean (offline),
frontend lint 0 errors, tsc clean, 86/86 frontend tests pass, build OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Decouples "many users" + "many server replicas" from "many machines" so the
platform is tenant-isolated and horizontally safe on the current single node.
- Per-signup workspaces (cm-auth): a new hosted-identity sign-in provisions and
owns its own workspace instead of joining the first. Config-gated by
auth.per_signup_workspace (default off); concurrent first-logins serialized by
a per-subject advisory lock so no duplicate workspaces.
- Terminal tickets in Postgres (migration 0016, hashed, single-use): any replica
can redeem a ticket minted by another. Drops the in-process ticket map.
- Container registry in Postgres (migration 0017, agent_containers): Terminal
and Sandbox managers resolve an agent's container through a shared registry,
so a 2nd replica reuses it instead of spawning a duplicate. node_id recorded
as 'local' (Phase 2 hook). Boot reconcile removes only true orphans, so
terminals now survive a redeploy (tmux sessions resume).
- Per-workspace quotas (cm-api/quota.rs): plan-tier caps on agents + live
containers, enforced at agent create + terminal spin-up (reconnects allowed),
returned as HTTP 402. New GET /api/quota surfaces usage vs limits.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Terminal app (xterm ⇄ WebSocket ⇄ per-agent themed container):
- zsh + oh-my-zsh + powerlevel10k image (agent-terminal), runs as uid 65532 to
share read-write ownership of the file-drive volume with the server.
- Interactive PTY in cm-sandbox (bollard exec tty/attach + resize) + a
TerminalManager; ticket-authed WS bridge routed straight to the backend via a
Traefik PathRegexp(/ws) rule. MOTD greets the user by name.
- tmux resumable sessions; multi-tab (one tmux session per tab, same container),
drag-to-reorder, rename, and a Save that persists named tabs to the server
(terminal_tabs, migration 0014) so they survive logout / a new device.
- Files drives mounted per-agent (subpath) at ~/drives/{documents,received,
shared}; a reconciler keeps the Files app's index in sync with terminal writes.
Storage moved to a shared `filedata` volume (CLAWMATES_STORAGE__DATA_DIR).
Obsidian vault (a markdown "second brain" per agent):
- New `vault` FileDrive (migration 0015) mounted into the terminal at ~/obsidian;
a file-content read route; a purple Obsidian tile + a vault viewer app.
Computer UI:
- Draggable computer-panel width (min = phone preset) keeping the size presets.
- Green Terminal glyph, "Claw Chat" → "Chat", colored gradient-outline app icons.
- Agent page: avatar↔activity-grid spacing + larger, uniform section fonts with
colored section-tinted tag chips.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The §15 door's email_send queues to `outbox` but nothing delivered it. Add a
real transport: cm-db outbox repo (list_queued/mark_sent/mark_failed) + a
cm-runtime drainer — an EmailSender trait (testable), a lettre STARTTLS
LettreSender, drain_once (queued -> sent/failed), and spawn_drainer wired into
the server beside the scheduler/sweeper/topology-worker.
Config-gated: inert (logs "outbox delivery DISABLED") until CLAWMATES_SMTP_*
is set, so it ships safely before credentials exist. The agent never holds the
SMTP credential — it only writes to outbox through the gated door; the server
owns the transport.
NOTE: live delivery is still credential-blocked — Migadu's API can't send
(SMTP-only) and the admin token is invalid; no SMTP creds exist. The transport
is built + tested (drain_marks_sent_and_failed via a mock sender); set
CLAWMATES_SMTP_* to go live with zero further code. clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Evolve topology_runs into a durable-job table (migration 0009): status state
machine (queued/running/completed/failed/cancelled), kind, input graph,
per-step checkpoint, error, last_event_id, timestamps. comparison becomes
nullable (the result blob, absent until completion). Back-compat: existing
rows default to completed/compare.
cm-db repo gains the durable-job ops: enqueue_run, claim_next_queued (CAS via
FOR UPDATE SKIP LOCKED), checkpoint, complete, fail, requeue_stale (resume
sweep), and status(). Regenerated .sqlx cache. Also fix two pre-existing test
RuntimeConfig literals missing the providers field (from the registry work).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Running the opt-in live suite (CM_LIVE_LLM=1 + ANTHROPIC_API_KEY) against
the real API immediately surfaced a launch blocker: Anthropic (and
OpenAI) restrict tool names to ^[a-zA-Z0-9_-]{1,128}$ — our ENTIRE
registry uses dotted names (clock.now, email.send, shell.exec, ...).
The scripted provider never enforced the pattern, so every real-model
deployment would have 400'd on the first tool call.
- Fix at the provider boundary, where it belongs: wire_tool_name /
internal_tool_name codec (dots <-> __) applied in BOTH HTTP providers
at all three sites (tools list, assistant tool_use echo, inbound
tool_use decode). Internal naming (DB step rows, scenarios, UI traces)
unchanged. Offline unit test round-trips every registry name through
the wire pattern
- New live tests, all passing against api.anthropic.com (Haiku 4.5):
- provider tool ROUND TRIP: real ToolUse arrives, ToolResult ships
back exactly as a checkpoint would reassemble it, model completes,
real usage events on the wire
- full runtime loop: real model calls clock.now, run completes, REAL
token usage metered, credits decremented
- the #1-risk validation: a real model's email.send intercepted ->
suspended -> approved -> checkpoint RESUMED against the live API ->
completed -> outbox exactly 1 (checkpoint/resume fidelity end to end)
- Stray TC_OPENAI_COMPAT_* envs renamed to CM_OPENAI_COMPAT_*
No credentials stored anywhere; the key was passed via env only.
163 Rust tests (+5 live, key-gated).
Co-Authored-By: Claude Fable 5 <[email protected]>
Full-depth rename per the approved plan; the 'claw' product vocabulary
(claws, /claws routes, clawId, Claw Chat) stays — it is now the brand.
- Display brand: Clawmates (manifest, titles, hero, login/rail logo
'clawmates'); default host app.clawmates.work; registry
ghcr.io/clawmates
- Crates tc-* -> cm-* (16 crates + all imports); binaries
clawmates-server/broker/bundler; images clawmates/*; env prefix
CLAWMATES_* (+ CM_TEST_DATABASE_URL / CM_LIVE_LLM); config
clawmates.toml; helm chart deploy/helm/clawmates with clawmates-*
resources; db names clawmates*; sockets /run/clawmates; cookie
cm_session; kind cluster clawmates-test; seccomp node profile
clawmates-agent-profile.json
- All 9 Playwright brand assertions updated in lockstep; historical
spec document left untouched as the only remaining 'TeamClaw'
- Local env migrated: dev pg clawmates-dev-pg/clawmates_dev, shared
test server clawmates-test-pg, kind cluster recreated with image +
profile, compose images rebuilt under clawmates/*
Verified end to end: 161 Rust + 68 frontend tests, 29 Playwright
journeys, 4 live kind tests, helm/install/LOC/placeholder gates, and
the clean-room install rehearsal serving the clawmates login page from
a signed bundle of the rebuilt images.
Co-Authored-By: Claude Fable 5 <[email protected]>
Omar Sobh