Made the Docker Compose route turn-key for a real self-host, then stood
the whole stack up and drove a live chat through it.
- First-owner bootstrap (cm-auth::bootstrap_owner): a fresh local-auth
install has no users and no signup route, so the initial Owner +
workspace are provisioned ONCE from CLAWMATES_BOOTSTRAP_* env on first
boot — idempotent, never clobbers an existing install (keys on 'any
workspace exists'). Two real-Postgres tests (creates + signs in;
second call is a no-op). Wired into server boot, guarded on a
non-empty password
- deploy/compose/README.md: full production bring-up — services, the
security topology, every config knob, Anthropic vs local-LLM, the
broker-key backup, ops, and TLS/SSE proxy notes
- .env.example fleshed out (bootstrap, LLM, auth mode, OTLP); compose
uses optional env_file so only the knobs you set are injected (unset
options never override clawmates.toml with empty strings)
- volume-init one-shot chowns the broker's named volumes so the non-root
scratch broker can write its socket + generated master key
Deployed locally and verified end to end: all 5 containers healthy,
broker generated its key, server bootstrapped owner@…, login + /api/user/me
work, and a real message streamed a live Anthropic response through the
gateway. Captured screenshots of login, workspace home, chat, and the
Computer panel.
166 Rust tests (+2 bootstrap).
Co-Authored-By: Claude Fable 5 <[email protected]>
Last open item from the roadmap + post-1.0 list. Run against the live
Clerk instance closing-seasnail-39.clerk.accounts.dev.
- Backend (crates/cm-auth/tests/live_clerk.rs, CM_LIVE_CLERK=1): pulls
REAL discovery + JWKS from the live instance, mints a REAL session JWT
via Clerk's Backend API (create user -> open session -> session token),
and runs it through AuthService::authenticate — verify + JIT provision
(keyed on the real sub), duplicate-subject suppression, tamper
rejection against the live JWKS. Decodes the instance domain from the
publishable key; cleans up the test user after. PASSING
- Frontend: built with AUTH_MODE=clerk + real keys, next start serves
Clerk's <SignIn /> at /login wired to the instance (instance domain +
data-clerk attributes present in the HTML). Both halves confirmed
end to end against production Clerk
- docs/clerk.md: documented the smoke procedure for both halves
166 Rust tests (+6 live, key-gated). Keys used via env only, never
stored — rotate them (they passed through chat).
Co-Authored-By: Claude Fable 5 <[email protected]>
- scripts/netpol-cluster.sh: a kind cluster with Calico (default CNI
disabled) — the only way to PROVE the §15 default-deny NetworkPolicy,
since kindnet accepts the object but never enforces it. New live test
on that cluster: outbound connect to 1.1.1.1 dropped, DNS egress
dropped, while API-server exec keeps working (not pod network).
Kernel-level enforcement of the sandbox egress claim, demonstrated
- K8sDriver::connect_with_context: pin a kubeconfig context instead of
ambient. The whole k8s suite now pins its cluster explicitly — the
netpol cluster's creation had silently switched the current context
and stranded the seccomp test on the wrong cluster (fixed and made
impossible to recur)
- CI: netpol-cluster up + calico egress test in the sandbox-k8s job
- Visual-regression lock (plan P6): @visual Playwright spec with
animation-disabled, masked-dynamic-region screenshots of login,
workspace home, chat welcome, computer panel, credits; darwin
baselines committed (5 PNGs); CI excludes @visual until linux
baselines are generated there. Full local suite: 33 journeys
165 Rust tests + 5 live kind tests (2 clusters) + 33 journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
Running the opt-in live suite (CM_LIVE_LLM=1 + ANTHROPIC_API_KEY) against
the real API immediately surfaced a launch blocker: Anthropic (and
OpenAI) restrict tool names to ^[a-zA-Z0-9_-]{1,128}$ — our ENTIRE
registry uses dotted names (clock.now, email.send, shell.exec, ...).
The scripted provider never enforced the pattern, so every real-model
deployment would have 400'd on the first tool call.
- Fix at the provider boundary, where it belongs: wire_tool_name /
internal_tool_name codec (dots <-> __) applied in BOTH HTTP providers
at all three sites (tools list, assistant tool_use echo, inbound
tool_use decode). Internal naming (DB step rows, scenarios, UI traces)
unchanged. Offline unit test round-trips every registry name through
the wire pattern
- New live tests, all passing against api.anthropic.com (Haiku 4.5):
- provider tool ROUND TRIP: real ToolUse arrives, ToolResult ships
back exactly as a checkpoint would reassemble it, model completes,
real usage events on the wire
- full runtime loop: real model calls clock.now, run completes, REAL
token usage metered, credits decremented
- the #1-risk validation: a real model's email.send intercepted ->
suspended -> approved -> checkpoint RESUMED against the live API ->
completed -> outbox exactly 1 (checkpoint/resume fidelity end to end)
- Stray TC_OPENAI_COMPAT_* envs renamed to CM_OPENAI_COMPAT_*
No credentials stored anywhere; the key was passed via env only.
163 Rust tests (+5 live, key-gated).
Co-Authored-By: Claude Fable 5 <[email protected]>
Full-depth rename per the approved plan; the 'claw' product vocabulary
(claws, /claws routes, clawId, Claw Chat) stays — it is now the brand.
- Display brand: Clawmates (manifest, titles, hero, login/rail logo
'clawmates'); default host app.clawmates.work; registry
ghcr.io/clawmates
- Crates tc-* -> cm-* (16 crates + all imports); binaries
clawmates-server/broker/bundler; images clawmates/*; env prefix
CLAWMATES_* (+ CM_TEST_DATABASE_URL / CM_LIVE_LLM); config
clawmates.toml; helm chart deploy/helm/clawmates with clawmates-*
resources; db names clawmates*; sockets /run/clawmates; cookie
cm_session; kind cluster clawmates-test; seccomp node profile
clawmates-agent-profile.json
- All 9 Playwright brand assertions updated in lockstep; historical
spec document left untouched as the only remaining 'TeamClaw'
- Local env migrated: dev pg clawmates-dev-pg/clawmates_dev, shared
test server clawmates-test-pg, kind cluster recreated with image +
profile, compose images rebuilt under clawmates/*
Verified end to end: 161 Rust + 68 frontend tests, 29 Playwright
journeys, 4 live kind tests, helm/install/LOC/placeholder gates, and
the clean-room install rehearsal serving the clawmates login page from
a signed bundle of the rebuilt images.
Co-Authored-By: Claude Fable 5 <[email protected]>
- tc-telemetry: fmt subscriber always; with [telemetry] otlp_endpoint
set, spans batch-export over OTLP/HTTP. Tested against a REAL OTLP
receiver decoding the actual protobuf (official proto types): the
emitted span and service.name arrive on the wire. No endpoint = no
export = no network (air-gap stance). tower-http TraceLayer gives
every API request a span
- The broker finally has its own image (images/broker.Dockerfile,
9.5MB from scratch) — the Helm chart referenced one that never
existed — and the compose deployment now RUNS the broker, sharing a
socket volume with the server (the unix-socket equivalent of the K8s
sidecar). Compose secret flows were silently dead before this
- server.Dockerfile fixes surfaced by the rehearsal: the workspace
build needs tools/ (bundler joined the workspace) and
images/seccomp/ (include_str! profile) in the build context
- scripts/rehearse-install.sh (plan: clean-VM rehearsal): assembles a
REAL signed bundle from the built images (server/frontend/broker/
postgres/socket-proxy), runs the customer path — offline verify,
docker load, compose up — and asserts /healthz plus the served login
page before teardown. Passing locally; wired as a release.yml step,
which also builds/ships the broker + socket-proxy images now
161 Rust tests + 29 journeys; clean-room rehearsal green.
Co-Authored-By: Claude Fable 5 <[email protected]>
- K8sDriver.with_localhost_seccomp(profile): sandbox pods run under the
STRICT allowlist instead of the runtime default. Proven live on kind:
the harness installs the profile onto the node, a pod runs ordinary
work as uid 10001, and unshare is kernel-denied inside the pod — the
same probe the Docker suite uses, now passing on both targets
- Helm: sandbox.seccomp=localhost renders a DaemonSet that installs the
chart-shipped profile into /var/lib/kubelet/seccomp on every node
(ConfigMap + hostPath); ci/check-helm.sh enforces the chart copy stays
byte-identical to images/seccomp/agent-profile.json and asserts the
hardened render (DaemonSet + profile + HPA)
- server HPA (autoscaling/v2, CPU target) behind
server.autoscaling.enabled
- SandboxManager.warm(n): a background warmer keeps n pre-provisioned
sandboxes ready so an agent's first exec skips container startup;
unhealthy pool entries are discarded, reuse never drains the pool,
shutdown destroys assigned AND pooled. [sandbox] warm_pool config
(default 0). Real-Docker test: prefill -> assign -> refill -> reuse ->
clean shutdown
160 Rust tests + 4 live kind tests.
Co-Authored-By: Claude Fable 5 <[email protected]>
- The compose server NEVER sees the raw Docker socket (plan risk #5):
tecnativa/docker-socket-proxy on an isolated engine_net with exactly
CONTAINERS/POST/EXEC/DELETE/VERSION allowed; server reaches it via
DOCKER_HOST. DockerDriver honors DOCKER_HOST (connect_to). Proven by a
REAL proxy test: full sandbox lifecycle works through the allowlist
while /networks, /secrets, and /images all 403 — the blast-radius cap
if the server is ever owned. (This also fixes compose deployments,
where sandbox provisioning previously had no engine access at all.)
- Gateway load test (plan P6): 40 concurrent SSE streams against one
server — every run completes with the full §13 event vocabulary,
every journal strictly monotonic, every resumeFrom=0 replay byte-equal
to its live stream
- release.yml: SBOMs (syft, spdx-json) for all four images shipped
INSIDE the signed bundle; final verification now runs in a
--network none container — proving the customer's verify path needs
no internet, not just claiming it
159 Rust tests.
Co-Authored-By: Claude Fable 5 <[email protected]>
- tc-auth JwtVerifier: OIDC discovery -> JWKS, RS256 with the issuer
pinned, 5s leeway (the crate's default 60s would double the life of
Clerk's 60s session tokens), key cache with one refresh on unknown kid
(Clerk rotates). Serves auth.mode = clerk AND generic oidc — a Clerk
instance IS an OIDC issuer, so one verifier covers both
- AuthService.authenticate dispatches: JWT-shaped bearers take the
hosted-identity path, everything else stays a local opaque session.
External users JIT-provision keyed by the stable sub claim
(users.auth_subject, unique partial index in migration 0007); an
existing local account with the same email is LINKED, not duplicated;
role tracks the issuer claim every request (org:admin -> Owner)
- Config auth.mode = "clerk" (requires issuer_url; validated), server
pins the issuer at boot, Helm values/configmap accept mode=clerk
- Tests with REAL crypto, no mocks: fresh RSA keypairs, a live local
issuer publishing real discovery + JWKS docs, Clerk-shaped tokens —
JIT + role mapping, repeat-subject no-dup, expired refused (leeway
regression), wrong-key forgery refused, foreign issuer refused, and
the full router round trip with Authorization: Bearer <session JWT>
- docs/clerk.md: dashboard session-token customization (email + org
role claims), config, @clerk/nextjs getToken() wiring, what CI proves
157 Rust + 63 frontend tests + 29 journeys. Air-gapped installs keep
local auth — Clerk is a cloud-only alternative, not a replacement.
Co-Authored-By: Claude Fable 5 <[email protected]>
- SandboxSpec gains an egress flag (default false — the kernel suite
still proves zero-network for agent sandboxes). Egress-enabled
containers exist ONLY for the browser: no credentials, no broker
route, bridge network with host-gateway alias for local test pages
- images/agent-browser: Alpine Chromium, uid 10001, setuid bits
stripped — same non-root hardening as agent-base
- browser.goto tool: headless chromium --dump-dom in the agent's
browser container; HTML stripped to readable text (4k cap) and
returned with output_taint=web; viewport screenshot captured,
base64'd out of the container, stored in the blob store
- Taint semantics tightened: the step that PRODUCED untrusted output
now carries its own taint (recorded before the step row), not just
later steps — chat.inbox test updated to the stricter §15 reading
- GET /api/claws/{id}/browser/viewport.png serves the latest capture;
BrowserApp polls it and renders the live viewport (spec §7.1),
keeping the empty state until the agent has browsed
- Proven end to end with REAL Chromium against a REAL local page:
content 'Revenue up 14 percent' returned tainted web; the gated
email.send that follows carries 'web' in its approval taint_sources
(untrusted content can never quietly reach outward); screenshot
verified by PNG magic bytes
152 Rust tests + 63 frontend + 27 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- SandboxManager (tc-runtime): one container per agent, provisioned
lazily on first use, reused for the manager's lifetime, replaced
transparently if dead, destroyed on shutdown
- shell.exec tool: sh -lc inside the agent's sandbox; stdout/stderr/
exit_code return to the model as the step output. No external effects
declared — the sandbox boundary (uid 10001, no caps, seccomp
allowlist, read-only rootfs, zero egress) is the §15 control here,
not an approval gate
- RuntimeConfig.sandboxes (+ with_sandboxes builder); [sandbox] config
{image, enabled}; the server connects the Docker driver at boot and
tolerates an absent engine (shell.exec reports it per-call)
- Tests with the REAL DockerDriver: a scripted run executes two
commands — output proves uid 10001 from inside, and /home/agent state
written by the first call is read by the second (same sandbox); a
deployment without a sandbox runtime records honest error steps and
the run still completes
151 Rust tests + 27 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- images/seccomp/agent-profile.json is now a TRUE ALLOWLIST: Docker's
default profile (vendored from moby v27.5.1, defaultAction ERRNO) with
18 syscalls an agent never needs stripped from the allow groups
(unshare, ptrace, bpf, mount family, setns, module loading,
perf_event_open, process_vm_*, reboot, quotactl, ...); arch map trimmed
to x86_64 + aarch64. All 6 Docker kernel assertions still green.
- K8sDriver (tc-sandbox feature 'k8s', kube-rs): one hardened pod per
sandbox — runAsUser 10001, cap-drop ALL, no-new-privs via
allowPrivilegeEscalation=false, RuntimeDefault seccomp, read-only
rootfs with emptyDir /tmp + /home/agent, resource limits, no service
account token — in a PSS-restricted namespace carrying a default-deny
NetworkPolicy (applied server-side apply, idempotent). Exec via the API
server attach channel with exit codes parsed from v1.Status.
- Live suite (feature 'k8s-tests') against a REAL kind cluster: uid /
CapEff==0 / NoNewPrivs / rootfs probes from inside pods, PSS label +
deny-all policy asserted via the API, lifecycle. Honest limits in the
rustdoc: Localhost seccomp profile and CNI-enforced egress are
per-cluster provisioning (kindnet does not enforce NetworkPolicy).
- rustls 0.23 process provider pinned to ring at driver connect.
- CI: dedicated sandbox-k8s job (helm/kind-action) running the suite.
149 Rust tests + 3 live kind tests; clippy clean including the k8s feature.
Co-Authored-By: Claude Fable 5 <[email protected]>
- S3BlobStore (object_store, path-style) behind the same BlobStore trait,
tested against a REAL MinIO container (round trip, overwrite, NotFound
on get and delete, nested keys); [storage] backend=local|s3 config with
validation + server-side selection (S3 creds via env overlay)
- Helm chart: server pod with the secret broker as a SIDECAR sharing a
private emptyDir unix socket (no network hop carries credentials),
frontend, optional local PVC vs S3, OIDC/oauth values, unbuffered-SSE
ingress annotations, NetworkPolicies (frontend->server only), hardened
securityContexts; ci/check-helm.sh lints AND asserts the rendered
topology properties
- deploy/airgapped/install.sh: offline signature+checksum verification via
the bundled teamclaw-bundler BEFORE any docker load; --verify-only mode;
ci/test-install.sh rehearses clean/tampered/wrong-key paths with the
real binary
- CI: helm gate + installer rehearsal wired in
149 Rust tests; helm lint + rendered assertions green; installer
verify-path rehearsal green.
Co-Authored-By: Claude Fable 5 <[email protected]>
- Concurrency soak (exit criterion): 12 concurrent gated runs, every
decision attempted twice concurrently, explicit resumes racing the
durable sweeper — exactly one execution per approval, grants consumed
at most once, every decision audited, zero stuck runs, zero unaudited
executions. (Testkit pool raised to 20 connections; the 5-connection
pool starved the storm.)
- axe a11y sweep (exit criterion): serious+critical violations fail CI on
login, shell, chat, computer home, settings app, all global pages, and
the wizard. Two real violations found and fixed: aria-label on a plain
div (wizard progress -> role=group) and a button directly inside a <dl>
(settings -> plain bordered list).
- tools/bundler (exit criterion): keygen / assemble / verify CLI — copies
artifacts, writes manifest.json + sha256 checksums.txt + a detached
ed25519 signature; verification is fully offline (keyless signing is
internet-dependent and disqualified). Tests: round trip, tampered
artifact caught by hash, tampered checksum list caught by signature,
wrong key refused, missing artifact reported.
147 Rust + 63 frontend tests + 27 Playwright journeys (incl. 4 a11y).
Co-Authored-By: Claude Fable 5 <[email protected]>
- LlmEvent::Usage across all three providers (Scripted deterministic
word-count accounting; Anthropic message_start/delta usage; OpenAI-compat
stream_options include_usage)
- tc-billing: ceil(tokens/1000) min 1 credit; lots drain oldest-first under
FOR UPDATE; balance clamps at zero while the usage ledger records the
full obligation; promo codes redeem exactly once via CAS (migration 0006)
- Runtime charges every completed run (billing failure never fails a run);
proven: 1 token in + 3 out -> 1 credit deducted
- API: GET /api/team/usage, POST /api/credits/redeem (409 on reuse, audited)
- Credits page: balance, 7-day usage meter with runway estimate, PromoRedeem
- /claws/new is the full §9 wizard: ?step=identity|access|slack deep-linked
progress, accent swatches + name randomizer, access toggles, optional
Slack step, explicit review-and-confirm (creation = live agent), animated
provisioning state -> straight into chat
- E2E: chat decrements the visible balance and fills the usage meter;
WELCOME500 adds exactly 500 once then refuses; wizard round trip
140 Rust + 63 frontend tests + 23 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- migration 0005 oauth_states: one-time states (10-min TTL), consumed by a
CAS DELETE on callback — replays and forgeries both 404
- POST /api/apps/oauth/start: OIDC discovery on the configured issuer (or
the custom MCP issuer for authType=mcp_oauth), state row, authorize URL
- GET /api/apps/oauth/callback: code exchanged at the REAL token endpoint
(client id+secret form POST); the access token goes straight to the
broker (test proves it never appears unencrypted in Postgres); connection
row + audit; redirects to the claw's Add Apps panel
- [oauth] config (issuer/client/redirect_base) wired through AppState
- Tests against a real local IdP server (discovery + validating token
endpoint): full round trip, broker-held token, replay/forged state
refused, bad code fails exchange, mcp_oauth uses the custom issuer while
plain oauth refuses without a configured IdP
- AddAppsApp: live connection badges + inline API-key connect per app
(E2E: connect Notion by key from the directory)
136 Rust + 63 frontend tests + 21 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- Broker op VerifySlackSignature: v0 HMAC-SHA256 computed INSIDE the broker
(constant-time compare); the signing secret never crosses the socket.
Slack secrets are one JSON credential {bot_token, signing_secret}; the
broker extracts the right field per operation
- Public POST /api/slack/events: signature verified against connected slack
connections via the broker; forged signatures 401; url_verification
handshake echoed only when signed; app_mention starts a real run in the
agent's dedicated '💬 Slack' session — and the agent's reply is itself a
gated outbound post
- SlackApp Connection tab captures bot token + signing secret
- Integration test: forged 401, signed challenge, signed mention -> run ->
slack.post pending in the approval queue
- E2E: full loop — connect, gated outbound (sink empty -> exactly one post),
then a node-crypto-signed mention -> approval card -> approve -> 'On it!'
lands in the sink
134 Rust + 63 frontend tests + 21 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- app_connections repo; POST /api/apps/connect (keys/basic): the credential
goes to the secret broker over its socket and only the encrypted ref lands
in the row; disconnect endpoint; /api/apps directory merged with live
connection status; audit rows for connect/disconnect
- Broker protocol: InvokeHttp carries a JSON body
- slack.post tool (SendsExternally -> gated): marked broker_executed — the
runtime skips its own grant consumption and the BROKER independently
verifies + consumes the single-use grant, then calls Slack with the bot
token injected; the runtime never sees the credential
- Config: [broker] socket_path + [slack] base_url; e2e harness spawns the
real teamclaw-broker daemon and the server hosts an e2e-only /__slack sink
- SlackApp: Connection tab stores the token via the broker; connected state
- Integration test: blocked while pending -> approved -> sink received
exactly one post with 'Bearer xoxb-test-token' -> grant replay refused
- E2E journey: connect Slack in the panel -> gated post card with preview ->
sink empty while pending -> approve -> exactly one post, queue clear
133 Rust + 63 frontend tests + 21 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- tc-db: sessions/messages/steps/runs/run_events repos (atomic seq assignment,
history with ordered step traces, journal replay-from-offset); migration 0003
- tc-llm: provider-neutral ChatRequest/LlmEvent; ScriptedProvider (scenario
TOML, word-level deltas, multi-turn tool legs — ships in production for
e2e/air-gap smoke), AnthropicProvider (Messages SSE), OpenAiCompatProvider
(vLLM/Ollama/llama.cpp); opt-in live tests via TC_LIVE_LLM=1
- tc-runtime: run loop with persist-before-emit event journal, real built-in
clock.now tool, step rows on the reply message, tool-error resilience,
broadcast channels for live attach
- tc-api: agent CRUD + settings/full (tenant-isolated, RBAC'd, audited),
sessions create/list/history?tools=true, POST /api/gateway SSE with
monotonic ids and exact resumeFrom journal replay (tested equal to live)
- teamclaw-server: config-driven provider factory
83 Rust tests green, all against real Postgres / real TCP.
Co-Authored-By: Claude Fable 5 <[email protected]>
