TWO FINDINGS, one from cleaning up and one from refusing to keep guessing.
THE LEAK. `./scripts/test.sh` left three containers running every time — 289 had
accumulated. The cause was a comment that lied: `warm_pool.rs` said "Shutdown
destroys assigned AND pooled sandboxes", while `SandboxManager::shutdown` drains
the POOL only. Its own doc says why — assigned sandboxes persist deliberately so
a redeploy can reuse them, and production reaps the strays with
`reconcile_orphans` at boot. A test has no next boot, so each one that assigned a
sandbox simply left it running. The three tests now call the `release_agent` that
already existed, and the comment says what the code does. Verified: 0 leaked,
where the same run leaked 3 before.
THE EVAL. The independent judge failed the same correct phase FOUR times, each
time citing a different invented requirement. I blamed the condition's wording
twice and rewrote it twice — the second rewrite made it worse, by naming a
command a tool-using judge then ran in its own container. Then a control showed
the same model answering MET to the same question asked directly, and a third
wording test showed a STRICTER phrasing scoring UNMET. Prose wording was not the
variable. Continuing to iterate would have been fitting the fixture to noise.
`scripts/judge-eval.sh` measures the thing instead: five cases drawn from real
incidents, each with an answer a careful human would agree with. This project has
557 tests and had zero evals, which is backwards — a test pins OUR code, an eval
pins the MODEL, and the model changes without us touching anything.
The result is why it was worth building:
glm-4.7 4/5 — wrong on kernel-ok: says UNMET when MET
kimi-for-coding 4/5 — wrong on goodhart: says MET when UNMET
Identical scores, opposite failure modes. GLM fails good work; KIMI passes work
where 14 assertions were deleted and the failing module removed to make a suite
"pass" — the exact incident the verifying judge was built after. Swapping the
validator to Kimi because it passes our failing case would have installed a
rubber stamp. Keep GLM: a judge that is too strict costs a re-run, a judge that
is too lenient costs the guarantee.
The eval also caught a bug in itself before I trusted it: Kimi answers with a
`thinking` block first, and a 160-token budget was consumed entirely by it, which
the harness scored as NO-ANSWER. An eval that misreads a model is worse than no
eval, so it now reads thinking blocks as a fallback and has room to answer.
557 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
Decouples "many users" + "many server replicas" from "many machines" so the
platform is tenant-isolated and horizontally safe on the current single node.
- Per-signup workspaces (cm-auth): a new hosted-identity sign-in provisions and
owns its own workspace instead of joining the first. Config-gated by
auth.per_signup_workspace (default off); concurrent first-logins serialized by
a per-subject advisory lock so no duplicate workspaces.
- Terminal tickets in Postgres (migration 0016, hashed, single-use): any replica
can redeem a ticket minted by another. Drops the in-process ticket map.
- Container registry in Postgres (migration 0017, agent_containers): Terminal
and Sandbox managers resolve an agent's container through a shared registry,
so a 2nd replica reuses it instead of spawning a duplicate. node_id recorded
as 'local' (Phase 2 hook). Boot reconcile removes only true orphans, so
terminals now survive a redeploy (tmux sessions resume).
- Per-workspace quotas (cm-api/quota.rs): plan-tier caps on agents + live
containers, enforced at agent create + terminal spin-up (reconnects allowed),
returned as HTTP 402. New GET /api/quota surfaces usage vs limits.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Full-depth rename per the approved plan; the 'claw' product vocabulary
(claws, /claws routes, clawId, Claw Chat) stays — it is now the brand.
- Display brand: Clawmates (manifest, titles, hero, login/rail logo
'clawmates'); default host app.clawmates.work; registry
ghcr.io/clawmates
- Crates tc-* -> cm-* (16 crates + all imports); binaries
clawmates-server/broker/bundler; images clawmates/*; env prefix
CLAWMATES_* (+ CM_TEST_DATABASE_URL / CM_LIVE_LLM); config
clawmates.toml; helm chart deploy/helm/clawmates with clawmates-*
resources; db names clawmates*; sockets /run/clawmates; cookie
cm_session; kind cluster clawmates-test; seccomp node profile
clawmates-agent-profile.json
- All 9 Playwright brand assertions updated in lockstep; historical
spec document left untouched as the only remaining 'TeamClaw'
- Local env migrated: dev pg clawmates-dev-pg/clawmates_dev, shared
test server clawmates-test-pg, kind cluster recreated with image +
profile, compose images rebuilt under clawmates/*
Verified end to end: 161 Rust + 68 frontend tests, 29 Playwright
journeys, 4 live kind tests, helm/install/LOC/placeholder gates, and
the clean-room install rehearsal serving the clawmates login page from
a signed bundle of the rebuilt images.
Co-Authored-By: Claude Fable 5 <[email protected]>
