The server COPYs templates/ and skills/ then drops to USER 65532. When the
build context arrives with mode-700 dirs (e.g. rsync -a preserving a dev's
local perms), COPY bakes 700 into the image and the nonroot runtime user
can't read them — the skills/team-template builtin seed silently skips
("Permission denied (os error 13)"). chmod -R a+rX after the COPYs makes
the seed dirs readable regardless of source perms.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Two durability fixes closing recurring flakes:
CI flake wrapper (broker + server Dockerfiles):
Wrapped the cargo build step in a 3-attempt retry loop with
linear backoff (10s / 20s). Directly targets the crates.io
transient network errors that keep hitting CI on the runners
('curl failed: SSL_ERROR_SYSCALL, errno 0'). Each build only
loses time on transient failures; a real compile error still
fails all 3 attempts and surfaces the last error normally.
Runtime systemd unit (deploy/clawmates-runtime/):
Replaces the manual 'docker run' that had been starting the
ZeroClaw runtime with no persistence for its network topology.
Ephemeral prod fixes at 09:30 PDT 2026-07-21 (task #38) were:
- anthropic.default provider block added to
/root/clawmates-runtime/data/.zeroclaw/config.toml (already
durable — bind-mounted from host)
- docker network connect clawmates_edge clawmates-runtime
(NOT durable — vanishes on container recreate)
New systemd unit clawmates-runtime.service (installed +
enabled on gw-04):
- ExecStart docker-runs the container attached to
clawmates_core, then connects clawmates_edge in the same
shell command, then docker waits.
- Bind-mounts both /root/clawmates-runtime/data and
/var/lib/clawmates-missions (for security_scan +
benchmark_runner).
- --rm so upgrading is just docker pull + systemctl restart.
- Restart=on-failure with 5s backoff.
Closes task #38 and preemptively closes the CI flake pattern.
Watches mission_artifacts for MD entries with render_pdf_status='pending'
and turns them into styled PDFs via:
1. Read source MD from <mission_root>/<path>
2. Call configured LLM (default gemini-2.5-flash) with a document-
typesetter system prompt that constrains style to a self-contained
HTML doc with inline CSS + our color palette
3. Print to PDF via `chromium --headless=new --print-to-pdf`
4. Save alongside source MD (foo.md → foo.pdf) + update
mission_artifacts.rendered_pdf_path + render_pdf_status='done'
Graceful degradation: GEMINI_API_KEY unset OR chromium missing =
row marked failed with a descriptive error, worker keeps ticking.
The frontend's "Open PDF" affordance (Slice 2) light up automatically
when render succeeds.
Boot ordering: PDF worker spawns after task_card_worker. Poll every
30s over up to MAX_PARALLEL=2 rows at a time — respects LLM rate
limits and keeps chromium's peak RAM under control.
Env knobs:
GEMINI_API_KEY — required for LLM step
CLAWMATES_PDF_RENDERER_MODEL — model id, default gemini-2.5-flash
CHROMIUM_BIN — chromium binary, default `chromium`
CLAWMATES_MISSIONS_ROOT — artifact dir root, default /var/lib/clawmates-missions
Dockerfile now installs chromium + fonts-liberation and sets
CHROMIUM_BIN=/usr/bin/chromium so the container image has everything
the renderer needs.
Also bumps workspace tokio deps to include the `process` feature
(required for tokio::process::Command).
Follow-ups:
- Anthropic + OpenAI provider variants (only Gemini in this slice)
- SSE stream on /api/missions/{id}/artifacts for the "PDF ready"
notification instead of poll-via-mission-GET
- Per-template PDF style overrides (currently one house style
for all missions)
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Team templates are the canonical rosters + tool bundles that mint
concrete teams for a mission. Every builtin ships as a TOML recipe
under templates/teams/*.toml, loaded into the DB at server boot.
Migration 0048 adds:
- team_templates (id, key, name, stack, default_topology,
risk_profile, mcp_bundles, version, source,
workspace_id)
- template_roles (m2m: template_id + slot; system_prompt,
skills[], brain_seed)
- teams gets template_id + template_version for level-up lineage
Ships 6 builtins:
- rust_sdlc — planner/coder/tester/reviewer/committer for Rust
- backend — api_designer/db_engineer/coder/tester/committer
(Postgres, DuckDB, graph DBs, wire protocols)
- frontend — designer/coder/tester/committer (React + Tailwind + ShadCN)
- mobile — designer/coder/tester/committer (Expo, RN, iOS, Android)
- gpu — arch_analyst/kernel_author/bench_engineer/coder/committer
(CUDA, Metal, ROCm from Rust)
- threejs — scene_designer/coder/shader_author/perf_engineer/
committer (three.js, WebGL, WebGPU)
Each role has a versioned system_prompt + skill list + brain_seed
markdown. Skills column is a name array today; Slice 3.5a promotes it
to a typed m2m join with the real skills catalog.
Server boot:
- team_template_loader::load_builtins reads TOML from
/etc/clawmates/templates/teams (container) or templates/teams (dev),
upserts idempotently. Deterministic uuid per template key (sha256
of a fixed namespace + key) so ids are stable across boots.
- Dockerfile copies templates/ to /etc/clawmates/templates.
Read API:
- GET /api/team-templates — list all
- GET /api/team-templates/{id} — detail with roles
Wizard:
- Step 3 rewired from a raw team_id text field to a template picker
with "LLM auto-provision" as the default option + one card per
builtin, showing stack, topology, risk profile, and description.
- Mission create now passes team_template_id (not team_id) so phase
execution knows which template to mint from.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Distroless static-debian12 has no git binary, so
routes/research_setup::ensure_repo_workspace's `tokio::process::Command::new("git")`
returned "No such file or directory" for every wizard-materialized
research topic. Silent failure — the topic's repo_workspace_path
stayed NULL, prepare_topic_runtime bailed out, the run was enqueued
against the shared workspace-wide gateway which then 401'd or timed
out on cold auth.
Switching to debian:12-slim + apt-installing git + ca-certificates
(needed by clone-over-https). Same UID 65532 as before so bind-mount
hosts don't need re-chown.
Image size grows from ~40MB to ~80-100MB but this is the correct
base for a service that shells out to git. Static distroless was a
premature optimization once the server's job description grew a
"clone repos" responsibility.
Follow-up: research migrating to the `gix` pure-Rust crate so we can
go back to distroless. Nontrivial refactor of ensure_repo_workspace
so not attempted in this fix.
Same fix pattern that unblocked broker in 534eb72. The `rustup target add`
was landing on the base image's default toolchain, but rust-toolchain.toml
(channel = 1.96.0) can prompt rustup to resolve to a distinct pinned
toolchain the target hasn't been installed on. Move the target install
after the COPY so it lands on the toolchain cargo actually uses.
Ordering matters. `rust:1.96-slim` ships with toolchain 1.96 already
installed under one identifier, but rust-toolchain.toml (channel = 1.96.0)
can prompt rustup to resolve to a distinct pinned toolchain. `rustup
target add` runs BEFORE rust-toolchain.toml lands, so it adds musl to the
wrong toolchain — cargo's later build picks up the workspace pin and can't
find core for the target. Move the COPY of rust-toolchain.toml before the
rustup target add so the target lands on the toolchain cargo actually
uses. Matches server.Dockerfile's known-working structure.
rust-toolchain.toml pins the workspace to 1.96.0. broker.Dockerfile used
rust:1-bookworm — some newer 1.x version — and `rustup target add` inside
that image installs the musl target under the container's default
toolchain. When cargo then reads rust-toolchain.toml and activates 1.96.0,
the target isn't there for that toolchain, so the build fails with
`E0463: can't find crate for core`. server.Dockerfile already uses
rust:1.96-slim; align broker to match.
server.Dockerfile already has CARGO_NET_GIT_FETCH_WITH_CLI=true plus git,
cmake, make, pkg-config, and the musl-gcc CC vars needed for the clawhdf5
git dep (libgit2 fails against Gitea smart-HTTP with "invalid packet line";
the git CLI works fine). broker.Dockerfile was missing all of it, so the
CI publish job failed at the very first `cargo build --release -p
clawmates-broker` inside the Docker builder. Copy the pattern.
So a fresh per-agent node-local volume (node-placed terminal) seeds with 65532
ownership and the non-root shell can write ~/drives, instead of a root-owned mount.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Terminal app (xterm ⇄ WebSocket ⇄ per-agent themed container):
- zsh + oh-my-zsh + powerlevel10k image (agent-terminal), runs as uid 65532 to
share read-write ownership of the file-drive volume with the server.
- Interactive PTY in cm-sandbox (bollard exec tty/attach + resize) + a
TerminalManager; ticket-authed WS bridge routed straight to the backend via a
Traefik PathRegexp(/ws) rule. MOTD greets the user by name.
- tmux resumable sessions; multi-tab (one tmux session per tab, same container),
drag-to-reorder, rename, and a Save that persists named tabs to the server
(terminal_tabs, migration 0014) so they survive logout / a new device.
- Files drives mounted per-agent (subpath) at ~/drives/{documents,received,
shared}; a reconciler keeps the Files app's index in sync with terminal writes.
Storage moved to a shared `filedata` volume (CLAWMATES_STORAGE__DATA_DIR).
Obsidian vault (a markdown "second brain" per agent):
- New `vault` FileDrive (migration 0015) mounted into the terminal at ~/obsidian;
a file-content read route; a purple Obsidian tile + a vault viewer app.
Computer UI:
- Draggable computer-panel width (min = phone preset) keeping the size presets.
- Green Terminal glyph, "Claw Chat" → "Chat", colored gradient-outline app icons.
- Agent page: avatar↔activity-grid spacing + larger, uniform section fonts with
colored section-tinted tag chips.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Full-depth rename per the approved plan; the 'claw' product vocabulary
(claws, /claws routes, clawId, Claw Chat) stays — it is now the brand.
- Display brand: Clawmates (manifest, titles, hero, login/rail logo
'clawmates'); default host app.clawmates.work; registry
ghcr.io/clawmates
- Crates tc-* -> cm-* (16 crates + all imports); binaries
clawmates-server/broker/bundler; images clawmates/*; env prefix
CLAWMATES_* (+ CM_TEST_DATABASE_URL / CM_LIVE_LLM); config
clawmates.toml; helm chart deploy/helm/clawmates with clawmates-*
resources; db names clawmates*; sockets /run/clawmates; cookie
cm_session; kind cluster clawmates-test; seccomp node profile
clawmates-agent-profile.json
- All 9 Playwright brand assertions updated in lockstep; historical
spec document left untouched as the only remaining 'TeamClaw'
- Local env migrated: dev pg clawmates-dev-pg/clawmates_dev, shared
test server clawmates-test-pg, kind cluster recreated with image +
profile, compose images rebuilt under clawmates/*
Verified end to end: 161 Rust + 68 frontend tests, 29 Playwright
journeys, 4 live kind tests, helm/install/LOC/placeholder gates, and
the clean-room install rehearsal serving the clawmates login page from
a signed bundle of the rebuilt images.
Co-Authored-By: Claude Fable 5 <[email protected]>
- tc-telemetry: fmt subscriber always; with [telemetry] otlp_endpoint
set, spans batch-export over OTLP/HTTP. Tested against a REAL OTLP
receiver decoding the actual protobuf (official proto types): the
emitted span and service.name arrive on the wire. No endpoint = no
export = no network (air-gap stance). tower-http TraceLayer gives
every API request a span
- The broker finally has its own image (images/broker.Dockerfile,
9.5MB from scratch) — the Helm chart referenced one that never
existed — and the compose deployment now RUNS the broker, sharing a
socket volume with the server (the unix-socket equivalent of the K8s
sidecar). Compose secret flows were silently dead before this
- server.Dockerfile fixes surfaced by the rehearsal: the workspace
build needs tools/ (bundler joined the workspace) and
images/seccomp/ (include_str! profile) in the build context
- scripts/rehearse-install.sh (plan: clean-VM rehearsal): assembles a
REAL signed bundle from the built images (server/frontend/broker/
postgres/socket-proxy), runs the customer path — offline verify,
docker load, compose up — and asserts /healthz plus the served login
page before teardown. Passing locally; wired as a release.yml step,
which also builds/ships the broker + socket-proxy images now
161 Rust tests + 29 journeys; clean-room rehearsal green.
Co-Authored-By: Claude Fable 5 <[email protected]>
- SandboxSpec gains an egress flag (default false — the kernel suite
still proves zero-network for agent sandboxes). Egress-enabled
containers exist ONLY for the browser: no credentials, no broker
route, bridge network with host-gateway alias for local test pages
- images/agent-browser: Alpine Chromium, uid 10001, setuid bits
stripped — same non-root hardening as agent-base
- browser.goto tool: headless chromium --dump-dom in the agent's
browser container; HTML stripped to readable text (4k cap) and
returned with output_taint=web; viewport screenshot captured,
base64'd out of the container, stored in the blob store
- Taint semantics tightened: the step that PRODUCED untrusted output
now carries its own taint (recorded before the step row), not just
later steps — chat.inbox test updated to the stricter §15 reading
- GET /api/claws/{id}/browser/viewport.png serves the latest capture;
BrowserApp polls it and renders the live viewport (spec §7.1),
keeping the empty state until the agent has browsed
- Proven end to end with REAL Chromium against a REAL local page:
content 'Revenue up 14 percent' returned tainted web; the gated
email.send that follows carries 'web' in its approval taint_sources
(untrusted content can never quietly reach outward); screenshot
verified by PNG magic bytes
152 Rust tests + 63 frontend + 27 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- images/seccomp/agent-profile.json is now a TRUE ALLOWLIST: Docker's
default profile (vendored from moby v27.5.1, defaultAction ERRNO) with
18 syscalls an agent never needs stripped from the allow groups
(unshare, ptrace, bpf, mount family, setns, module loading,
perf_event_open, process_vm_*, reboot, quotactl, ...); arch map trimmed
to x86_64 + aarch64. All 6 Docker kernel assertions still green.
- K8sDriver (tc-sandbox feature 'k8s', kube-rs): one hardened pod per
sandbox — runAsUser 10001, cap-drop ALL, no-new-privs via
allowPrivilegeEscalation=false, RuntimeDefault seccomp, read-only
rootfs with emptyDir /tmp + /home/agent, resource limits, no service
account token — in a PSS-restricted namespace carrying a default-deny
NetworkPolicy (applied server-side apply, idempotent). Exec via the API
server attach channel with exit codes parsed from v1.Status.
- Live suite (feature 'k8s-tests') against a REAL kind cluster: uid /
CapEff==0 / NoNewPrivs / rootfs probes from inside pods, PSS label +
deny-all policy asserted via the API, lifecycle. Honest limits in the
rustdoc: Localhost seccomp profile and CNI-enforced egress are
per-cluster provisioning (kindnet does not enforce NetworkPolicy).
- rustls 0.23 process provider pinned to ring at driver connect.
- CI: dedicated sandbox-k8s job (helm/kind-action) running the suite.
149 Rust tests + 3 live kind tests; clippy clean including the k8s feature.
Co-Authored-By: Claude Fable 5 <[email protected]>
