`clawmates-runtime` shipped with `gcc` and `make` but no `cmake`, no `g++` and
no `python3-dev`. Measured on clawhdf5, three probes:
no cmake → "is `cmake` not installed?" exit 101 after 13s
no python3-dev → "cannot find -lpython3.11" exit 101 at link
with both → cargo test PASSES exit 0 after 69s
This is not only the delivery gate. The AGENTS run in this image, so a coding
phase was writing Rust it had no way to compile or test — which reframes the
last run's 11 agent commits as unverifiable by construction.
`images/agent-toolchain/Dockerfile` (the microVM path) has had `cmake
build-essential` all along, and its own header warns about precisely this:
"if `cargo` is present in one image and absent in another, the same mission
passes or fails depending on which backend it landed on, and nothing says why."
Both images now install the same set — it was missing `python3-dev` too.
`images/runtime-toolchain.Dockerfile` is a thin local overlay so the laptop can
run today without recompiling zeroclaw from the fork; it is meant to be deleted
once a runtime image built from the corrected deploy/ Dockerfile is published.
Also: a build failure is no longer reported as a red suite. Both are cargo exit
101, and `verify_tests` mapped every non-zero to `Failed(code)` — so a missing
toolchain was recorded as the USER's tests failing. It now returns
`CouldNotRun` with the reason when the output shows a compile or link failure.
Deliberately narrow: a failing `assert!` still reads as red, because letting
broken code past `on_green_tests` is the expensive direction to be wrong in.
Both directions are pinned by tests built from today's two real samples.
And the coding phase finally has a loop: `research_and_code.toml` declared
`loop = "until_no_more_int_items"`, which `phase_config.rs` lists as
DECLARED_BUT_UNREAD. Iteration is driven by `max_iterations` + `done_when`, and
with `max_iterations = 1` and no `done_when` the phase ran ONCE and was never
judged — reporting `completed` whatever it produced. Now 3 passes against a
stated goal, wording per the measured rule (say what the tree must CONTAIN).
Co-Authored-By: Claude Opus 5 <[email protected]>
Two things.
1. The architecture_mapper proposal, applied AND made durable.
The GLM proposal (019fddd9) was accepted in full: the agent's system_prompt now
carries the Mermaid-first constraint and its brain was rewritten. Both verified
against the live row and the .h5 file.
But `apply_identity` writes `UPDATE agents SET system_prompt` and
`apply_brain_consolidation` writes that agent's brain — neither touches the team
TEMPLATE. That agent is mission-scoped, so the improvement would have died with
the mission. The model's actual insight was sharp and worth keeping: "Mermaid
diagrams beat prose" lived in the brain SEED and not in the system PROMPT, so it
only applied when the agent happened to consult its brain. That constraint is
now in templates/teams/codebase_research.toml, where every future Codebase
Research team inherits it.
(The proposal's second item mostly restated anti-patterns the seed already
lists, so the seed is unchanged. Applying an LLM's suggestion is not the same as
agreeing with all of it.)
2. Gemini is gone.
Removed: the `gemini.default` provider alias and its `is_exact_provider_match`
prefix, GEMINI_API_KEY forwarding to agent containers, the evaluator's
gemini->gemini family row, the model selectors in claws/teams/planner and in
TeamWizard + AgentComputer, and the commented provider block in the runtime
config example (whose ZEROCLAW_AGENT_MAP example still mapped a worker_gemini
that no longer existed).
`provider_alias_for("gemini")` now returns claude_cli.default via the
unrecognised-model branch, which LOGS. A stray gemini binding degrades visibly
rather than resolving to a provider row we no longer ship. A test pins that, and
another pins that GEMINI_API_KEY is forwarded in NEITHER auth mode, so adding it
back to the list is a visible change rather than an accident.
Avatar generation is DELETED, not disabled — it called Gemini's image model, and
there is no alternative: Claude and Kimi are text-only, and z.ai answers
"Unknown Model" for cogview-3-flash and cogview-4 on our plan (measured, not
assumed). AvatarModal keeps UPLOAD, which never needed a provider; only the
prompt-generation half is gone.
240 backend lib tests, 89 frontend tests, clean tsc + eslint, build succeeds.
Monday 07:00, Persistent=true so a week missed to downtime fires on next
boot rather than leaving a silently empty library. 30-minute timeout so a
wedged run cannot hold the slot until the following week.
The script is deliberately thin — it calls the API and reports — so it
never needs changing when the harvest does. Auth is a long-lived operator
session in /etc/clawmates/library.token (root-only, 600); rotate by
replacing the file.
Exit status follows `healthy`, not paper count. A mature library shelves
nothing most weeks and that is success; a run that errored is a failure
even if it shelved something.
The first manual fire caught a real bug in this script, in the opposite
direction to this week's usual: the harvest genuinely shelved 15 papers
and pushed them, and the reporter crashed on an escaped quote inside an
f-string, so systemd marked the unit FAILED. A false failure destroys
trust in the signal exactly as a false success does. The reporter now
avoids backslashes entirely (it is embedded in a single-quoted shell
string) and was proved against the real response shape before being
trusted.
Verified end to end on gw-04:
run 1: 25 candidates, 10 already held, 15 shelved, pushed
run 2: 25 candidates, 25 already held, 0 shelved, no branch, healthy
Co-Authored-By: Claude Opus 5 <[email protected]>
Two defects found by running a real coding mission (019fc372) rather than a
test. Both made a coding phase look like it produced nothing.
**Capture measured the wrong baseline.** It diffed the working tree against
HEAD, which is correct only while work stays uncommitted. `rust_sdlc` has a
*committer* role, so committing is the intended path — meaning a mission that
did its job properly leaves a clean tree and captured nothing. That is exactly
what happened: the agent created `DELIVERY_PROBE.md`, committed it as
`aa3be95`, and the artifact recorded `empty: true` beside a commit that
plainly contained the work.
`mission_workspace` now records the clone point in `.git/clawmates-base` (in
`.git/`, so it travels with the checkout, stays invisible to the repository,
and cannot be reached by an agent through its pinned workspace), refreshed
whenever `fetch_and_reset` moves HEAD. Capture diffs from there, covering
committed, staged and unstaged changes in one pass. Checkouts predating the
marker fall back to HEAD and say so via `base_recorded: false`.
**Agents could not create files.** `coding_readwrite` granted `file_edit` but
not `file_write`. `file_edit` replaces an exact existing string and rejects an
empty `old_string`, so creating a new file was impossible. The mission
transcript is unambiguous: "the tool rejected empty old_string... the shell is
restricted", after which the agent worked around it through `shell`. The
comment above that profile has claimed it grants file_write since the day it
was written; the list never contained it.
Also broadens capture from coding/benchmark/security_scan to every phase kind
of a repo-bearing mission: `phase_task_text` tells research phases to "save
findings under /mission/repo/research/", so filtering by kind would have
discarded every research brief such a mission produced.
Regression tests cover committed-only and committed-plus-uncommitted work
against a real git repo.
Co-Authored-By: Claude Opus 5 <[email protected]>
`templates/teams/rust_sdlc.toml` instructs the coder to run `cargo test`; the
`done_when` evaluator runs a project's own suite to verify a claim rather than
believe it; `security_scan.rs` shells out to cargo-audit, gitleaks, trivy and
semgrep. The runtime image contained none of them.
The security consequence was the worse one. With no scanners present, a scan
emitted four `<tool>:tool_error` task rows and completed — a scan that scanned
nothing and reported cleanly. Same class of false signal as a verifier that
never ran a command.
Adds gitleaks 8.30.1, trivy 0.72.0, semgrep (in its own venv so its pinned
dependency tree cannot collide), and a minimal Rust stable toolchain with
cargo-audit. Versions are pinned as build args and were taken from the
releases API — the first attempt used plausible-looking numbers that 404'd.
Layers are ordered cheapest-and-most-stable first so bumping a scanner does
not invalidate the Rust layer, and the cargo registry is dropped after
`cargo install`.
Measured: 864 MB -> 3.13 GB (scanners +350 MB, Rust +1.23 GB, semgrep
+680 MB). Note this image is NOT in `AGENT_IMAGES` — it never ships to fleet
nodes, only gw-04 holds it, against 112 GB free. An earlier note claiming
otherwise was wrong. The real cost is a slower `docker save | load` per
rebuild.
Verified in the built image: rustc 1.97.1, cargo-audit 0.22.2, gitleaks
8.30.1, trivy 0.72.0, semgrep 1.172.0, python 3.11.2, plus the existing git,
claude and node.
Co-Authored-By: Claude Opus 5 <[email protected]>
Mission agents were burning ~275K tokens producing nothing: the coder had
only file_read and its workspace was the empty ephemeral sandbox, so it
dumped a full spec inline instead of writing files. Two root causes:
1. Risk-profile allowlists used pre-0.8 tool names. `coding_readwrite`
allow-listed `file_write` (renamed to `file_edit` in ZeroClaw 0.8, and
`file_write` now refuses on ephemeral workspaces) and omitted file_edit
/ content_search / glob_search / git_operations — the exact tools the
phase prompt tells agents to use. Since allowed_tools is a strict
allowlist, agents were effectively read-only. Documents the correct
profiles in agent.config.example.toml (they only lived in host config;
the live runtime profiles were corrected via its config API).
2. workspace.path never got set. `agents.<alias>.workspace.path` is an
Option<PathBuf> the ZeroClaw Configurable macro skips from prop
enumeration, so provision_claw's set_prop always 404'd and the whole
call errored into a swallowed eprintln. Removes the dead set_prop and
pins the workspace out-of-band: MissionRuntimeProvisioner::
pin_agent_workspaces patches the shared config file on the per-mission
container (format-preserving via toml_edit, atomic temp+mv); the daemon
applies it on the same reload that surfaces the freshly-provisioned
claws. Covered by unit tests for the TOML stamp.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- mission_runtime::spawn_sweeper: force-removes runtime containers
for missions terminal for >=30 min, clears runtime_endpoint. Wired
into clawmates-server main().
- docker-compose socket-proxy: NETWORKS=1 so bollard.connect_network
can attach containers to clawmates_edge for provider egress.
- phase_runner ordering: ensure_checkout BEFORE ensure_container so
the mission dir exists before docker mounts it.
- provisioner: mkdir_p the mission dir defensively for research-only
missions that skip checkout entirely.
Two durability fixes closing recurring flakes:
CI flake wrapper (broker + server Dockerfiles):
Wrapped the cargo build step in a 3-attempt retry loop with
linear backoff (10s / 20s). Directly targets the crates.io
transient network errors that keep hitting CI on the runners
('curl failed: SSL_ERROR_SYSCALL, errno 0'). Each build only
loses time on transient failures; a real compile error still
fails all 3 attempts and surfaces the last error normally.
Runtime systemd unit (deploy/clawmates-runtime/):
Replaces the manual 'docker run' that had been starting the
ZeroClaw runtime with no persistence for its network topology.
Ephemeral prod fixes at 09:30 PDT 2026-07-21 (task #38) were:
- anthropic.default provider block added to
/root/clawmates-runtime/data/.zeroclaw/config.toml (already
durable — bind-mounted from host)
- docker network connect clawmates_edge clawmates-runtime
(NOT durable — vanishes on container recreate)
New systemd unit clawmates-runtime.service (installed +
enabled on gw-04):
- ExecStart docker-runs the container attached to
clawmates_core, then connects clawmates_edge in the same
shell command, then docker waits.
- Bind-mounts both /root/clawmates-runtime/data and
/var/lib/clawmates-missions (for security_scan +
benchmark_runner).
- --rm so upgrading is just docker pull + systemctl restart.
- Restart=on-failure with 5s backoff.
Closes task #38 and preemptively closes the CI flake pattern.
Prod's Gemini prepayment credits are depleted (429 on every refine
attempt). Switching to Anthropic Claude Opus 4.8 for the mission
Refine flow — ANTHROPIC_API_KEY is already set in prod for ZeroClaw's
provider config, so no new secret plumbing.
- crates/cm-api/src/mission_refiner.rs:
* DEFAULT_MODEL: gemini-2.5-flash → claude-opus-4-8
* call_gemini → call_anthropic against
https://api.anthropic.com/v1/messages with the standard
x-api-key + anthropic-version headers
* Response parser reads content[type='text'].text (Messages API
block shape) instead of Gemini's candidates path
* Timeout raised 60s → 90s (Opus can be slower than Flash on
long briefs; still bounded so a stuck call fails fast)
- deploy/compose/.env.example: doc block rewritten. Refine now
reuses ANTHROPIC_API_KEY; Level-Up keeps GEMINI_API_KEY because
it needs JSON-mode structured output.
Level-Up is NOT switched in this commit — it uses Gemini's JSON mode
which has no drop-in Anthropic equivalent (needs tool-use rewrite).
Filed as a separate concern; Refine is what was actively broken.
Verified: SQLX_OFFLINE=true cargo check -p cm-api clean;
cargo fmt --all clean.
Phase 0 started Herdr via nohup — died on reboot / logout. Replaces
that with proper service management:
- deploy/fleet/herdr-persistence/herdr.service — systemd user unit
(Linux). Restart=on-failure with an 8-in-24h burst cap so a
broken binary doesn't hot-loop. Requires linger enabled so the
user's systemd manager runs without a login session; install.sh
does that via loginctl.
- deploy/fleet/herdr-persistence/dev.herdr.plist — launchd
LaunchAgent (macOS). ProgramArguments + PATH templated so the
install script substitutes actual paths at deploy time.
- deploy/fleet/herdr-persistence/install.sh — idempotent installer
that autodetects OS, drops the unit/plist in the right place,
enables + starts, prints status.
Rolled to tank + architect + morpheus (systemd) + smith + macbook
(launchd). All 5 nodes confirmed `status: running` post-install.
Phase 0 gap closed: Herdr now survives node reboots and logouts,
which is the prerequisite for the fleet_herdr dispatch path to be
reliable across mission_orchestrator restarts.
Named docker volume required the separately-managed clawmates-runtime
container to know the volume's on-disk path (varies by docker root).
A predictable host path (/var/lib/clawmates-missions) means the
runtime's spawn command can bind-mount the same path directly.
Closes the follow-up gap flagged when task #23 landed. security_scan
and benchmark_runner now exec against $CLAWMATES_MISSIONS_ROOT/
{mission_id}/repo — this commit is what actually puts a repo there.
- crates/cm-api/src/mission_workspace.rs — new module.
ensure_checkout(pool, workspace_id, mission_id):
* mission with no repo_id → Ok(None), no-op
* repo cloned into $ROOT/{id}/repo (--depth 1)
* dir already a git repo → fetch + reset --hard origin/{branch}
(idempotent — every launch brings the tree in sync with the
remote default_branch)
Auth uses the process's ambient git credential setup (SSH agent /
.netrc / helper). Tokens deliberately not embedded in URLs.
- crates/cm-api/src/mission_orchestrator.rs — on_launch calls
ensure_checkout after team materialization + team_id bind.
Non-fatal: clone failures log and continue so research_only
missions (no repo needed) don't get blocked.
- deploy/compose/docker-compose.yml — new named volume
missions_workspaces mounted at /var/lib/clawmates-missions on
both the server (writer) and where the clawmates-runtime
container will mount it (reader for docker exec). CLAWMATES_
MISSIONS_ROOT + CLAWMATES_RUNTIME_CONTAINER env vars set on
the server so mission_workspace + exec_target read the same
canonical values.
The scan/bench trigger buttons now actually produce findings once
you (a) run a mission whose repo_id is set, (b) have the
clawmates-runtime container bind-mounting missions_workspaces at
/var/lib/clawmates-missions.
Verified: SQLX_OFFLINE=true cargo check -p cm-api +
cargo test -p cm-api --test mission_orchestrator both green.
1. Dashboard.tsx — mount LevelUpInbox as a bottom panel on the
MISSIONS tier sidebar (below MissionsList, above the tier rail).
Capped at 38% height so it never crowds the mission list; scrolls
independently when the proposal count grows.
2. deploy/compose/.env.example — document GEMINI_API_KEY +
CLAWMATES_REFINER_MODEL + CLAWMATES_LEVEL_UP_MODEL. Refine + Level-
Up silently 500 without the key; the model overrides default to
gemini-2.5-flash so setting the key is the only required step.
Closes the two loose ends I called out last turn (inbox not mounted,
env vars undocumented). The full flow — Refine → mission launch →
security scan / benchmark → level-up review — is now wireable
end-to-end on a fresh deploy just by copying .env.example → .env
and setting POSTGRES_PASSWORD + GEMINI_API_KEY.
Prep for the SDLC coding team. Adds two upstream Gitea binaries to
the clawmates-runtime image so team-scoped agents can drive
git.redclaw.dev without custom MCP code:
- tea v0.14.2 (shell CLI: clone/push/pr checkout, git remote helper)
- gitea-mcp v1.3.0 (native MCP server: list_repo_pull_requests,
create_pull_request, create_file, update_file, create_branch,
create_issue, get_file_content, etc.)
Both installed by arch (amd64 / arm64) at image build time; also
adds git to the runtime layer since it was missing (needed by tea's
shell delegates).
research_container::inherited_env now propagates GITEA_ prefixed
vars so a team container inherits GITEA_TOKEN (+ optional GITEA_HOST)
from the server env. Team-scoped daemons can then start gitea-mcp
via stdio with --token-env GITEA_TOKEN.
Followup on gw-04:
1. rebuild clawmates-runtime image against the new Dockerfile
2. add [mcp_bundles.gitea_forge] to the runtime template
3. set GITEA_TOKEN in the server compose env
Two cosmetic bugs surfaced by the successful v0.8.3 pipeline run:
1. **'produced Invalid Date'** — research_outcomes.created_at was
an OffsetDateTime serialized by time's default array format
(`[y, ordinal, hh, mm, ss, ns, tz]`), which browser's
`new Date(...)` can't parse. Add `#[serde(with =
"time::serde::rfc3339")]` matching the pattern already in
threads.rs / routine_runs.rs.
2. **Stale error text on pipeline card** — the runs stage's
`latest_error` walked every run by `created_at DESC` and
returned the first non-empty error, so a topic with an earlier
failed run + a later completed run kept displaying the old
error next to '1 completed'. Now the error only surfaces when
the MOST RECENT run itself failed. Historical failures stay in
the run count but don't leak their message.
Root cause: claude CLI in the per-topic research container runs as
uid=0(root). ZeroClaw's claude_cli provider passes
--dangerously-skip-permissions which Claude CLI rejects under root
for security — so the CLI hangs waiting for interactive permission
approval that never arrives, hitting the 600s provider timeout with
zero step records journaled.
Three-part fix:
1. Claude settings bind-mount (research_container.rs):
Optional CLAWMATES_CLAUDE_SETTINGS_PATH env — when set, mount the
host file at /root/.claude/settings.json (read-only) in every
spawned team container. deploy/claude-settings.json ships the
canonical config (permissions.defaultMode = bypassPermissions +
hasCompletedOnboarding). CLI accepts requests immediately with no
--dangerously-skip-permissions flag needed.
2. Verbose per-step log line (topology_worker.rs):
Every checkpoint now writes to stderr:
topology_worker::step run_id=X step=N node=Y role=Z phase=W
output_bytes=B tokens=T gated=G
Visible in docker logs clawmates_server_1 — gives us live
'topology is flowing' signal without opening the canvas, and
makes it obvious when a topology_kind is skipping stages it
shouldn't.
3. Stuck-container reaper (topology_worker.rs):
New 60s-tick loop reap_stuck_runs: for any research topology_run
older than 15 min with zero checkpoint.records, docker-stop its
container and mark the run failed with a diagnostic error. Only
reaps research-bound runs (non-research runs don't own a
container). The existing 180s stale-checkpoint requeuer stays
in place for other failure modes.
Deploy: gw-04 needs
ln -sf /path/to/repo/deploy/claude-settings.json /opt/clawmates/claude-settings.json
CLAWMATES_CLAUDE_SETTINGS_PATH=/opt/clawmates/claude-settings.json
in the server env, plus the timeout lowered from 600 -> 120 in
compose. Both handled in the deploy step outside this commit.
Broker acts as the §14 door for outbound provider fetches (Gitea +
GitHub + GitLab repo lists, Slack sends, OAuth token exchange). The
prior 'internal: true'-only core network gave the container no route
to the internet, so BrokerClient::fetch_authorized() failed at DNS
before it could reach api.github.com or a self-hosted Gitea.
Adding edge keeps the broker inbound-tight (still only listens on the
unix socket inside broker_run — no exposed TCP port) while granting
outbound. Postgres access via core is preserved.
Applied by hand to /opt/clawmates/docker-compose.yml on gw-04 to
unblock the current repo-connect flow; committing here so the next
re-provision doesn't regress it. The auto-deploy timer only pulls
images — compose file drift lives with us until a future
compose-sync step is added.
Two of the fleet's Gitea Actions runners (morpheus, architect) already
had 8080 permanently bound by unrelated services (nginx on morpheus,
envio-hasura on architect) — every e2e run scheduled there died at
playwright's webServer preflight with "http://127.0.0.1:8080/healthz is
already used". 18080 is unused across morpheus/tank/architect.
Swap 8080 → 18080 in the eight e2e-scoped sites: clawmates.e2e.toml
(listen_addr + slack base_url + oauth redirect_base), dex.yaml (client
redirect URIs must match backend), playwright.config.ts + tests
(p4-slack, p6-oauth), the http.ts dev-fallback origin, and the two
shell scripts (e2e-backend safety check, rehearse-install healthz probe).
Prod compose (/opt/clawmates/docker-compose.yml on gw-04) is untouched;
prod continues to expose the server on 8080 internally on the compose
network (that's per-network, not host-shared).
Move the running stack off root ownership. The systemd service now runs
as User=clawmates:clawmates with WorkingDirectory=/opt/clawmates, and the
script's COMPOSE_DIR default follows. This closes the "rootful compose
stack" ask from the original ship-readiness audit — deploys no longer
require any part of the pipeline to run as root beyond docker access
(the clawmates user gets that via the docker group).
Docker-managed volumes (pgdata, broker_run, broker_key, brains, filedata)
stay put; the compose project name is unchanged so docker resolves them
to the same physical volumes. The old /root/clawmates directory stays in
place as an emergency rollback for a week, then gets removed as follow-up.
The compose file on gw-04 was migrated to registry-prefixed image
references (100.94.185.103:5000/clawmates/<svc>:latest), which lets
`docker compose up` pick up the pulled image directly. The old script
retagged each pulled image to `clawmates/<svc>:latest` as a bridge so
the previous compose file (which used bare names) would find it —
that step is now unnecessary and just added a small window where the
un-prefixed tag could diverge from the registry.
Drift check now compares against the registry-prefixed tag directly.
`docker compose` v2 preferred with `docker-compose` v1 fallback stays.
Two fixes needed to make the timer actually roll correctly on gw-04:
1. Drift check compares the running container's image ID against the
local `clawmates/<svc>:latest` tag, not just pre/post-pull digests.
The pre/post check only catches new pulls — if a previous roll failed
between the retag and `docker compose up` (e.g. compose CLI failed),
the tag was updated but the container wasn't, and the next tick saw
no drift and silently left the stale container running. The
running-vs-tag check catches that case on the next tick.
2. Prefer `docker compose` (v2 plugin) but fall back to legacy
`docker-compose` (v1). GW-04 ships v1 only right now, and calling
`docker compose up -d` failed with "unknown shorthand flag: 'd'"
because docker had no `compose` subcommand at all. The fallback
keeps the script portable when the stack moves to a host with v2.
Brings a2a rooms, delegation, and A2A ingress work back into main. Prod's
DB has migrations 0026 (group_rooms) and 0027 (a2a) applied from an
earlier hand-tagged fleet21 build cut from this branch, but main never got
them — so the CI-built server image from main refused to start against
prod's DB with "migration 26 was previously applied but is missing".
Landing the branch closes that gap: main + prod DB now share the same
migration state, so images built from main can safely roll onto gw-04.
Included:
- 0026_group_rooms.sql / 0027_a2a.sql — align main with prod's schema
- cm-api routes/a2a.rs + mcp_door.rs updates — A2A ingress and MCP door
- cm-runtime tools/delegate.rs + tools/chat.rs — delegation + N-way rooms
- frontend TeamObserver + FleetPanels + AgentObserver updates
- taxonomy.ts — delegation + A2A signals in the live world feed
Not included (still WIP on the local checkout):
- 0028_backfill_on_delete.sql / 0029_hot_query_indexes.sql
- broker pool + team-run quota changes
- Dashboard.tsx UI rename (Large World → Visualizations)
- Node write.send timeout (aaab663) — separate concern
The SSE resume fix (202e853) is preserved by auto-merge — approvals.rs
still awaits resume_run inline so the channel is ready before reply.
Adds a `publish` job to ci.yml that fires only on green pushes to main. It
builds broker, server, and frontend from images/*.Dockerfile, tags each with
:main-<sha> + :latest, and pushes to the fleet registry at
redclaw-web-01:5000 (via its Tailscale IP 100.94.185.103, which the daemons
already trust in insecure-registries).
Adds a small systemd oneshot + 1-minute timer for gw-04 that polls :latest
of each service, pulls on drift, retags to the un-prefixed name the current
compose file uses, and rolls only the changed services. The retag keeps
/root/clawmates/docker-compose.yml unchanged for now — a follow-up can
migrate the compose file to registry-prefixed names once we're confident.
End-to-end: push to main -> tests -> images pushed -> gw-04 timer pulls
within ~1 min -> prod updated. Rollback = docker tag <old-sha> :latest and
`docker compose up -d`.
CI no longer references k8s; delete the dead k8s surface:
- deploy/helm/ (the chart), ci/check-helm.sh, scripts/netpol-cluster.sh
- cm-sandbox: the feature-gated K8sDriver (src/k8s.rs) + k8s_security test +
the `k8s`/`k8s-tests` features + the optional kube/k8s-openapi/rustls deps
(Cargo.lock drops the kube-rs tree). Nothing outside cm-sandbox referenced it.
Docker (bollard) DockerDriver is the sole sandbox driver. cm-sandbox + cm-runtime
compile, fmt + clippy clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The §15 door's email_send queues to `outbox` but nothing delivered it. Add a
real transport: cm-db outbox repo (list_queued/mark_sent/mark_failed) + a
cm-runtime drainer — an EmailSender trait (testable), a lettre STARTTLS
LettreSender, drain_once (queued -> sent/failed), and spawn_drainer wired into
the server beside the scheduler/sweeper/topology-worker.
Config-gated: inert (logs "outbox delivery DISABLED") until CLAWMATES_SMTP_*
is set, so it ships safely before credentials exist. The agent never holds the
SMTP credential — it only writes to outbox through the gated door; the server
owns the transport.
NOTE: live delivery is still credential-blocked — Migadu's API can't send
(SMTP-only) and the admin token is invalid; no SMTP creds exist. The transport
is built + tested (drain_marks_sent_and_failed via a mock sender); set
CLAWMATES_SMTP_* to go live with zero further code. clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Single-role drafter on a fixed concision-weighted brief, one turn per backend,
judge-ranked: GLM-4.7 > Claude > Kimi > GLM-5.2. Notable inversion — the
flagship GLM-5.2 ranked last by over-writing the 25-word brief, underscoring
that the best model is role/task/rubric-dependent. Adds the worker_glm5
(glm-5.2 Opus-class) tier to the example config. Gemini excluded this run
(Google June-19 API-key-restriction 403; infra, not model).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Gemini via the built-in `gemini` API family (gemini-2.5-flash default,
gemini-3-pro-preview flagship) and Groq via `groq` — both wired as topology
worker aliases (worker_gemini, worker_groq), no image rebuild. Documents the
Phase C recipe: map semantic roles to per-model aliases via the server
ZEROCLAW_AGENT_MAP, then POST /api/topologies/run.
Five subscription/API backends now selectable per role: Claude, Kimi, GLM,
Gemini, Groq. Proven live on gw-04 — a 3-stage pipeline ran researcher=GLM-4.7
-> analyst=Kimi-K2 -> writer=Gemini-2.5-flash end-to-end (RunRecord persisted).
Quota-care notes added (5h/weekly caps, low concurrency -> prefer sequential
pipelines, short prompts).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
GLM Coding Plan has no CLI of its own and ToS-throttles raw SDK/API access
("1113 Insufficient Balance"); it is served only inside officially supported
tools, and Claude Code is the primary one. So GLM rides the existing
claude_cli provider with a z.ai env override (ANTHROPIC_BASE_URL=
api.z.ai/api/anthropic + ANTHROPIC_AUTH_TOKEN) — no new provider, no image
rebuild. Fresh HOME + CLAUDE_CODE_OAUTH_TOKEN="" so the container's cached
Claude subscription doesn't override the routing; key via $NAME indirection.
Deployed & verified on gw-04: worker_glm answers "I am GLM-4.7, built by
Zhipu AI" over /ws/chat. Three subscription models now co-reside on one
runtime — Claude (coordinator), Kimi (worker_kimi), GLM (worker_glm).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Runtime image now installs Node 22 (nodesource) + both agent CLIs —
@anthropic-ai/claude-code and @moonshot-ai/kimi-code — so the zeroclaw
daemon can spawn `kimi -p` for the new kimi_cli provider on a Kimi
membership, parallel to `claude -p` (claude_cli). Kimi needs Node >=22.19;
the image was on Node 18.
Documents the kimi_cli worker recipe in agent.config.example.toml (the
coding endpoint is User-Agent gated → drive Kimi through its own CLI, not a
raw OpenAI-compat client) and keeps GLM via the registry family.
Deployed & verified on gw-04: coordinator (claude_cli) answers "Anthropic
built me", worker_kimi (kimi_cli) answers "I am Kimi, built by Moonshot AI"
— heterogeneous Claude + Kimi on one runtime, over /ws/chat.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Phase A wiring (config-only): server registry kimi provider now targets the
Kimi Code coding endpoint (api.kimi.com/coding/v1, model kimi-for-coding,
subscription-billed via a Kimi Code Console key). Runtime example documents a
custom.kimi worker provider (OpenAI-compatible coding endpoint) + GLM, selectable
per topology role via ZEROCLAW_AGENT_MAP. Activates on KIMI_API_KEY; no rebuild.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
ZeroClaw already supports glm + moonshot (Kimi) as model providers; document the
pattern to run a topology worker on them (env model+key, agent on the alias,
role->alias via ZEROCLAW_AGENT_MAP).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adds the claude_cli.door provider pattern (mcp_config + allowed/disallowed
tools) so a subscription claude_cli agent reaches the Clawmates door via Claude
Code's own MCP client, with native tools locked out (door = only actuator).
Validated live on gw-04: subdoor agent (claude_cli, subscription) → door →
outbox + agent|door.executed audit, cost 0.0; shell command denied.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Set the tool-free risk profile to level="full" so ZeroClaw raises no in-channel
approval (the human is removed; the Clawmates door auto-governs). Document that
claude_cli is text-only (claude -p doesn't surface tool calls), so agents that
ACT through the door need a tool-capable provider (groq/anthropic) — with a
commented actor-agent example.
Validated live: a groq-brained actor agent at full autonomy emitted
clawmates__email_send with no approval prompt; the door executed it (outbox +
agent|door.executed audit). No human in the loop.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Tool-free ZeroClaw agents get one actuator: an MCP server (POST /mcp, JSON-RPC
2.0, protocol 2024-11-05) that fronts the existing §15 machinery. The human
approver is replaced by an automated policy (default allow-all → agents are
autonomous; CLAWMATES_DOOR_POLICY=deny is a kill switch), but the governed parts
stay: every action is journaled to the append-only audit log, actions execute
through the runtime's gated-tool path, and broker credential-custody is wired in
for v2 tools. Synchronous execution returns the real result inline.
- crates/cm-api/src/mcp_door.rs: initialize/tools.list/tools.call handler; auth
(bearer -> workspace), classify effects, policy auto-decide, execute, audit.
v1 exposes email_send (-> outbox); slack/pay (broker+grant chain) is next.
- crates/cm-runtime: Runtime::{tool_descriptor_json, tool_gate_category,
execute_door_tool} — door-facing entry that builds ToolContext and consumes a
grant for runtime-executed gated tools.
- deploy/clawmates-runtime: agents now carry mcp_bundles=["clawmates_door"];
[[mcp.servers]] points at the door (bearer injected at deploy, not committed).
Validated live on gw-04: tools/call email_send -> isError:false + outbox row +
"agent|door.executed" audit, no human. 4 door unit tests + clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Switch the per-tenant runtime to ZeroClaw's new native `claude_cli` model
provider (added in the zeroclaw repo): every agent turn spawns `claude -p`
headless, authed by CLAUDE_CODE_OAUTH_TOKEN — i.e. the Claude subscription, no
per-minute API TPM ceiling (the groq free tier 429'd multi-turn pipelines).
- Dockerfile: install the Claude Code CLI (@anthropic-ai/claude-code) into the
runtime image.
- agent.config.example.toml: role-cast on model_provider = "claude_cli.default".
NOTE: set the model via env (ZEROCLAW_providers__models__claude_cli__default__model)
not the TOML sub-table, else resolve_default_model() reads empty -> NEEDS_ONBOARDING.
- .gitignore: token.key (local Claude OAuth token, never committed).
Validated live on gw-04: a 2-node researcher->writer pipeline ran via the
deployed POST /api/topologies/run -> claude_cli -> real claude turns; 2-step
RunRecord, cost_usd 0.0 (subscription), no rate limiting.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
cm-orchestrator owns the topology graph; each turn now drives a real ZeroClaw
role-agent in a container via the proven gateway drive recipe, instead of a
tool-free cm-llm call.
- crates/cm-api/src/topology_exec.rs: ZeroClawDriveExecutor impl TurnExecutor —
pair (POST /pair + X-Pairing-Code, token cached) -> ws /ws/chat?agent=<alias>
-> send {type:message,content} -> drain chunk/done/approval_request/error.
approval_request is recorded as a BLOCKED GatedAction, never auto-approved (§15).
Role->alias via ZEROCLAW_AGENT_MAP, fallback ZEROCLAW_DEFAULT_AGENT (scout).
- POST /api/topologies/run {task,graph} -> execute() -> RunRecord, persisted
best-effort to the existing topology_runs table (no migration). compare stays
tool-free. from_env() is read in-handler so cm-api still boots unset.
- deploy/clawmates-runtime: example config now declares a tool-free multi-agent
role-cast; README documents the ZEROCLAW_* knobs + run endpoint.
tokio-tungstenite 0.26 (already in lock) + dev axum `ws` for the hermetic test.
3 lib tests green, clippy clean, SQLX_OFFLINE build clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Made the Docker Compose route turn-key for a real self-host, then stood
the whole stack up and drove a live chat through it.
- First-owner bootstrap (cm-auth::bootstrap_owner): a fresh local-auth
install has no users and no signup route, so the initial Owner +
workspace are provisioned ONCE from CLAWMATES_BOOTSTRAP_* env on first
boot — idempotent, never clobbers an existing install (keys on 'any
workspace exists'). Two real-Postgres tests (creates + signs in;
second call is a no-op). Wired into server boot, guarded on a
non-empty password
- deploy/compose/README.md: full production bring-up — services, the
security topology, every config knob, Anthropic vs local-LLM, the
broker-key backup, ops, and TLS/SSE proxy notes
- .env.example fleshed out (bootstrap, LLM, auth mode, OTLP); compose
uses optional env_file so only the knobs you set are injected (unset
options never override clawmates.toml with empty strings)
- volume-init one-shot chowns the broker's named volumes so the non-root
scratch broker can write its socket + generated master key
Deployed locally and verified end to end: all 5 containers healthy,
broker generated its key, server bootstrapped owner@…, login + /api/user/me
work, and a real message streamed a live Anthropic response through the
gateway. Captured screenshots of login, workspace home, chat, and the
Computer panel.
166 Rust tests (+2 bootstrap).
Co-Authored-By: Claude Fable 5 <[email protected]>
Full-depth rename per the approved plan; the 'claw' product vocabulary
(claws, /claws routes, clawId, Claw Chat) stays — it is now the brand.
- Display brand: Clawmates (manifest, titles, hero, login/rail logo
'clawmates'); default host app.clawmates.work; registry
ghcr.io/clawmates
- Crates tc-* -> cm-* (16 crates + all imports); binaries
clawmates-server/broker/bundler; images clawmates/*; env prefix
CLAWMATES_* (+ CM_TEST_DATABASE_URL / CM_LIVE_LLM); config
clawmates.toml; helm chart deploy/helm/clawmates with clawmates-*
resources; db names clawmates*; sockets /run/clawmates; cookie
cm_session; kind cluster clawmates-test; seccomp node profile
clawmates-agent-profile.json
- All 9 Playwright brand assertions updated in lockstep; historical
spec document left untouched as the only remaining 'TeamClaw'
- Local env migrated: dev pg clawmates-dev-pg/clawmates_dev, shared
test server clawmates-test-pg, kind cluster recreated with image +
profile, compose images rebuilt under clawmates/*
Verified end to end: 161 Rust + 68 frontend tests, 29 Playwright
journeys, 4 live kind tests, helm/install/LOC/placeholder gates, and
the clean-room install rehearsal serving the clawmates login page from
a signed bundle of the rebuilt images.
Co-Authored-By: Claude Fable 5 <[email protected]>
- tc-telemetry: fmt subscriber always; with [telemetry] otlp_endpoint
set, spans batch-export over OTLP/HTTP. Tested against a REAL OTLP
receiver decoding the actual protobuf (official proto types): the
emitted span and service.name arrive on the wire. No endpoint = no
export = no network (air-gap stance). tower-http TraceLayer gives
every API request a span
- The broker finally has its own image (images/broker.Dockerfile,
9.5MB from scratch) — the Helm chart referenced one that never
existed — and the compose deployment now RUNS the broker, sharing a
socket volume with the server (the unix-socket equivalent of the K8s
sidecar). Compose secret flows were silently dead before this
- server.Dockerfile fixes surfaced by the rehearsal: the workspace
build needs tools/ (bundler joined the workspace) and
images/seccomp/ (include_str! profile) in the build context
- scripts/rehearse-install.sh (plan: clean-VM rehearsal): assembles a
REAL signed bundle from the built images (server/frontend/broker/
postgres/socket-proxy), runs the customer path — offline verify,
docker load, compose up — and asserts /healthz plus the served login
page before teardown. Passing locally; wired as a release.yml step,
which also builds/ships the broker + socket-proxy images now
161 Rust tests + 29 journeys; clean-room rehearsal green.
Co-Authored-By: Claude Fable 5 <[email protected]>
- K8sDriver.with_localhost_seccomp(profile): sandbox pods run under the
STRICT allowlist instead of the runtime default. Proven live on kind:
the harness installs the profile onto the node, a pod runs ordinary
work as uid 10001, and unshare is kernel-denied inside the pod — the
same probe the Docker suite uses, now passing on both targets
- Helm: sandbox.seccomp=localhost renders a DaemonSet that installs the
chart-shipped profile into /var/lib/kubelet/seccomp on every node
(ConfigMap + hostPath); ci/check-helm.sh enforces the chart copy stays
byte-identical to images/seccomp/agent-profile.json and asserts the
hardened render (DaemonSet + profile + HPA)
- server HPA (autoscaling/v2, CPU target) behind
server.autoscaling.enabled
- SandboxManager.warm(n): a background warmer keeps n pre-provisioned
sandboxes ready so an agent's first exec skips container startup;
unhealthy pool entries are discarded, reuse never drains the pool,
shutdown destroys assigned AND pooled. [sandbox] warm_pool config
(default 0). Real-Docker test: prefill -> assign -> refill -> reuse ->
clean shutdown
160 Rust tests + 4 live kind tests.
Co-Authored-By: Claude Fable 5 <[email protected]>
- The compose server NEVER sees the raw Docker socket (plan risk #5):
tecnativa/docker-socket-proxy on an isolated engine_net with exactly
CONTAINERS/POST/EXEC/DELETE/VERSION allowed; server reaches it via
DOCKER_HOST. DockerDriver honors DOCKER_HOST (connect_to). Proven by a
REAL proxy test: full sandbox lifecycle works through the allowlist
while /networks, /secrets, and /images all 403 — the blast-radius cap
if the server is ever owned. (This also fixes compose deployments,
where sandbox provisioning previously had no engine access at all.)
- Gateway load test (plan P6): 40 concurrent SSE streams against one
server — every run completes with the full §13 event vocabulary,
every journal strictly monotonic, every resumeFrom=0 replay byte-equal
to its live stream
- release.yml: SBOMs (syft, spdx-json) for all four images shipped
INSIDE the signed bundle; final verification now runs in a
--network none container — proving the customer's verify path needs
no internet, not just claiming it
159 Rust tests.
Co-Authored-By: Claude Fable 5 <[email protected]>
- src/lib/auth/bearer.ts is the single identity dispatch for both
server-side token consumers (RSC apiFetch and the /api proxy route):
local -> httpOnly tc_session cookie; clerk -> Clerk getToken() session
JWT. The Clerk SDK is imported lazily, so the air-gapped/local path
never loads it
- Runtime env (AUTH_MODE / CLERK_PUBLISHABLE_KEY / CLERK_SECRET_KEY),
deliberately NOT build-time NEXT_PUBLIC_*: the same standalone image
serves both deployment targets
- Conditional <ClerkProvider> in the root layout (publishableKey passed
at render from runtime env); /login renders Clerk's <SignIn /> in
clerk mode and the local form otherwise; proxy.ts middleware delegates
to clerkMiddleware() only when active
- Helm: frontend deployment injects the Clerk keys from a Secret when
auth.mode=clerk
- mode.ts unit-tested (default local, exact-match clerk, loud failure
without the publishable key); the local path stays proven by all 29
journeys; the Clerk branch is thin delegation to the SDK, exercised in
deployment smoke per docs/clerk.md
157 Rust + 68 frontend tests + 29 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- tc-auth JwtVerifier: OIDC discovery -> JWKS, RS256 with the issuer
pinned, 5s leeway (the crate's default 60s would double the life of
Clerk's 60s session tokens), key cache with one refresh on unknown kid
(Clerk rotates). Serves auth.mode = clerk AND generic oidc — a Clerk
instance IS an OIDC issuer, so one verifier covers both
- AuthService.authenticate dispatches: JWT-shaped bearers take the
hosted-identity path, everything else stays a local opaque session.
External users JIT-provision keyed by the stable sub claim
(users.auth_subject, unique partial index in migration 0007); an
existing local account with the same email is LINKED, not duplicated;
role tracks the issuer claim every request (org:admin -> Owner)
- Config auth.mode = "clerk" (requires issuer_url; validated), server
pins the issuer at boot, Helm values/configmap accept mode=clerk
- Tests with REAL crypto, no mocks: fresh RSA keypairs, a live local
issuer publishing real discovery + JWKS docs, Clerk-shaped tokens —
JIT + role mapping, repeat-subject no-dup, expired refused (leeway
regression), wrong-key forgery refused, foreign issuer refused, and
the full router round trip with Authorization: Bearer <session JWT>
- docs/clerk.md: dashboard session-token customization (email + org
role claims), config, @clerk/nextjs getToken() wiring, what CI proves
157 Rust + 63 frontend tests + 29 journeys. Air-gapped installs keep
local auth — Clerk is a cloud-only alternative, not a replacement.
Co-Authored-By: Claude Fable 5 <[email protected]>
- PWA (§16): hand-rolled 60-line service worker (network-first pages with
offline fallback, cache-first hashed statics, /api NEVER touched — SSE
and approvals stay live), app manifest with §2 identity, stdlib-
generated coral claw icons, prod-only registration. E2E asserts
manifest, real PNG icons, an ACTIVATED service worker, and the /api
bypass. (Serwist was tried and dropped: its webpack plugin fights
Next 16's Turbopack builds; sixty lines we own beat a plugin we fight.)
- Route motion (§3): (workspace) template re-mounts per navigation with a
quiet fade-rise, zeroed under prefers-reduced-motion. The a11y sweep
now settles running animations before scanning — axe was reading
mid-fade opacity as contrast failures
- OAuth browser flow vs REAL dex: the e2e harness boots dexidp/dex with
static client + password; the journey drives the actual dex login form
from /api/apps/oauth/start through the callback 303 and asserts the
app reads connected (closing the P4 deferral honestly)
- release.yml: tag-triggered — builds all four images + postgres, saves
tarballs, assembles the SIGNED air-gapped bundle (compose, config,
migrations, seccomp profile, installer, bundler binary), derives the
public key via the new Could not find command "pubkey". subcommand (tested), verifies
the bundle customer-style with the public half only, attaches tarball
+ public key to the GitHub release
153 Rust + 63 frontend tests + 29 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- S3BlobStore (object_store, path-style) behind the same BlobStore trait,
tested against a REAL MinIO container (round trip, overwrite, NotFound
on get and delete, nested keys); [storage] backend=local|s3 config with
validation + server-side selection (S3 creds via env overlay)
- Helm chart: server pod with the secret broker as a SIDECAR sharing a
private emptyDir unix socket (no network hop carries credentials),
frontend, optional local PVC vs S3, OIDC/oauth values, unbuffered-SSE
ingress annotations, NetworkPolicies (frontend->server only), hardened
securityContexts; ci/check-helm.sh lints AND asserts the rendered
topology properties
- deploy/airgapped/install.sh: offline signature+checksum verification via
the bundled teamclaw-bundler BEFORE any docker load; --verify-only mode;
ci/test-install.sh rehearses clean/tampered/wrong-key paths with the
real binary
- CI: helm gate + installer rehearsal wired in
149 Rust tests; helm lint + rendered assertions green; installer
verify-path rehearsal green.
Co-Authored-By: Claude Fable 5 <[email protected]>
- Broker op VerifySlackSignature: v0 HMAC-SHA256 computed INSIDE the broker
(constant-time compare); the signing secret never crosses the socket.
Slack secrets are one JSON credential {bot_token, signing_secret}; the
broker extracts the right field per operation
- Public POST /api/slack/events: signature verified against connected slack
connections via the broker; forged signatures 401; url_verification
handshake echoed only when signed; app_mention starts a real run in the
agent's dedicated '💬 Slack' session — and the agent's reply is itself a
gated outbound post
- SlackApp Connection tab captures bot token + signing secret
- Integration test: forged 401, signed challenge, signed mention -> run ->
slack.post pending in the approval queue
- E2E: full loop — connect, gated outbound (sink empty -> exactly one post),
then a node-crypto-signed mention -> approval card -> approve -> 'On it!'
lands in the sink
134 Rust + 63 frontend tests + 21 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- app_connections repo; POST /api/apps/connect (keys/basic): the credential
goes to the secret broker over its socket and only the encrypted ref lands
in the row; disconnect endpoint; /api/apps directory merged with live
connection status; audit rows for connect/disconnect
- Broker protocol: InvokeHttp carries a JSON body
- slack.post tool (SendsExternally -> gated): marked broker_executed — the
runtime skips its own grant consumption and the BROKER independently
verifies + consumes the single-use grant, then calls Slack with the bot
token injected; the runtime never sees the credential
- Config: [broker] socket_path + [slack] base_url; e2e harness spawns the
real teamclaw-broker daemon and the server hosts an e2e-only /__slack sink
- SlackApp: Connection tab stores the token via the broker; connected state
- Integration test: blocked while pending -> approved -> sink received
exactly one post with 'Bearer xoxb-test-token' -> grant replay refused
- E2E journey: connect Slack in the panel -> gated post card with preview ->
sink empty while pending -> approve -> exactly one post, queue clear
133 Rust + 63 frontend tests + 21 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
