Switching agents to claude_cli left missions hanging: the per-mission
container had claude_cli configured but no credential, so `claude -p`
waited forever. A phase sat at `running` for ten minutes with nothing in
the logs — no error, because there is nothing to error on.
The original subscription design assumed a persisted `claude /login`
under a bind-mounted $HOME. That holds for the shared runtime and NOT for
a mission container, which gets its own data dir and therefore no login.
So subscription mode now forwards CLAUDE_CODE_OAUTH_TOKEN.
The two Anthropic credentials remain mutually exclusive, and there is now
a test asserting it in both directions: Claude Code ranks ANTHROPIC_API_KEY
above the OAuth token, so shipping both bills the API while the deployment
believes it is on the subscription — visible only on the invoice.
Deployment: CLAWMATES_RUNTIME_AUTH=subscription and CLAUDE_CODE_OAUTH_TOKEN
added to compose + .env on gw-04.
Co-Authored-By: Claude Opus 5 <[email protected]>
The captured diff from mission 019fc3ba contained the deliverable and, beside
it, a file the agent had invented:
+++ b/.gitconfig_temp
+[safe]
+ directory = /mission/repo
The server clones as uid 65532 and the mission container runs as root, so
every `git` an agent runs is refused with "detected dubious ownership". Agents
do not surface that as a failure — they improvise around it, and the
improvisation lands in the repository. Left alone it would have been committed
and pushed to the user's repo alongside the real work.
The judge got `GIT_CONFIG_*` for this in dd8dad2; the mission containers never
did. They do now — git's environment form of `-c`, inherited by subprocesses,
so it covers the agent's own git, the `git_operations` tool, and anything that
shells out. Scoped to the checkout, never `--global`.
`.gitconfig_temp` is also added to the capture exclusions. The cause is fixed,
but a stray workaround from some future agent should not reach a user's
repository, and the exclusion costs nothing.
Co-Authored-By: Claude Opus 5 <[email protected]>
Wires diff capture into the two sweeps that matter.
`phase_runner::sweep_once` gains `capture_finished_coding_phases`, guarded by
`NOT EXISTS (code_diff for this phase)`. Deliberately a separate step rather
than a hook on `close_finished_phases` or `evaluate_finished_phases`: a phase
reaches `completed` through one or the other depending on whether it declared
a `done_when`, so hanging capture off either would silently skip half the
missions. The guard also makes it retryable — a capture that errors is simply
re-selected next tick.
`mission_runtime::sweep_once` captures anything still outstanding immediately
before `teardown_container`, which deletes the checkout. This covers what the
phase sweep structurally cannot: a mission that ended `failed` mid-coding
still has real work on disk, and reaping it unexamined destroys the only
evidence of what the agents actually did.
Applies to coding, benchmark and security_scan phases — all three operate on
a repo.
Co-Authored-By: Claude Opus 5 <[email protected]>
`evaluator_tools::Sandbox::run` shelled out to `tokio::process::Command::new
("docker")`. The server image installs `git ca-certificates chromium
fonts-liberation` and nothing else, so in production every verification
command failed to spawn.
The failure was invisible in the worst way. `Sandbox::run` deliberately turns
execution failures into evidence text rather than errors, so a judge reasons
about "that command did not run" instead of the pass collapsing. With no
`docker` binary every command returned COULD NOT RUN, the judge correctly
concluded it could not verify, and fail-closed returned "not met". The
verdicts were right. The verification never happened — and the adversarial
validation that appeared to prove the feature working proved fail-closed
working instead.
The second defect made it worse: `checks` recorded the *attempt*, pushed
before the command ran, so a verdict reached with a dead sandbox reported
"verified by 10 checks" — a stronger claim than "no checks at all", made on
weaker evidence.
- New `container_exec` routes execution through the Docker API via bollard,
which was already a dependency and already reaches the daemon through the
socket proxy. Captures the exit code (absent from the old helper) and keeps
stdout and stderr apart (`LogOutput`'s Display merged them, which is why
nothing downstream could tell JSON from a progress bar). `security_scan`
parses stdout alone; `benchmark_runner` needs both.
- `ExecOutput::success()` requires `Some(0)`. An unreadable status is not
success — `commit_policy = "on_green_tests"` will gate on this, and
"unknown" reading as "green" would push untested work.
- `Sandbox::run` returns a `CheckOutcome` carrying `ran`/`refused`/
`exit_code`. `Verdict::verified_checks()` counts executions, not attempts.
- The UI gains a third state: "could not verify (N attempted, 0 ran)" —
precisely the case that used to render as verified.
- Regression tests reproduce the production shape: two checks recorded,
neither executed, `was_verified() == false`; plus a failing suite (exit 101)
still counting as verification, because that is something the judge learned
rather than was told.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude Code resolves credentials in a fixed priority order and ranks
ANTHROPIC_API_KEY ABOVE the subscription's CLAUDE_CODE_OAUTH_TOKEN.
mission_runtime forwarded that key into every per-mission container
unconditionally, so on a runtime authenticated with `claude /login` the key
would silently win: `claude` still works, agents still run, and every mission
bills the API while appearing to use the subscription. There is no error to
observe -- the only symptom is the invoice.
CLAWMATES_RUNTIME_AUTH = subscription | api_key now gates the forward list.
In subscription mode ANTHROPIC_API_KEY is withheld; Gemini/Groq/OpenAI still
forward in both modes since they have no subscription equivalent. The mode is
logged per container so it is visible in the deploy log rather than inferred.
Default is api_key -- today's behaviour exactly. An unset or misspelled value
falls back to it too, because defaulting to subscription on a typo would strip
the key and leave missions with no credential at all.
forwarded_provider_keys() is the single source for the list, called by both
ensure_container and the tests, so the two cannot drift -- the failure mode
here is invisible, which is precisely when duplicated knowledge is worst.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Mission turns execute against the per-mission runtime container, but claws
were provisioned via RuntimeProvisioner::from_env() — i.e. the GLOBAL gateway.
That daemon loads config once at boot and never re-reads the file, so the
per-mission daemon had no claw_* agents at all: querying it for a mission
claw's risk_profile returned 404 while the global daemon returned 200. With
the alias unresolvable, the daemon silently fell back to the default `scout`
agent, which is jailed to the global workspace — agents reported "the scout
agent workspace" and "/mission/repo isn't accessible", produced no files, and
burned tokens. This is the deeper cause behind the empty-output runs; the
tool-allowlist and workspace-pin fixes were necessary but not sufficient.
- RuntimeProvisioner::for_gateway(url) — aim the provisioner at a specific
gateway (mirrors ZeroClawDriveExecutor::from_env_for_gateway); from_env now
delegates to it.
- mission_orchestrator captures the per-mission endpoint from ensure_container
and provisions every claw there, falling back to the global gateway only
when there is no per-mission runtime (dev/no-docker).
- workspace.path is file-only (the config prop API cannot set a PathBuf), and
the daemon never re-reads the file, so pin_agent_workspaces is now followed
by restart_container(): restart + wait for /health to answer. Agents created
through the daemon's own config API are already persisted to that file, so
they survive; the pairing code is re-minted on every launch.
The readiness probe inspects the /health BODY — exec_capture only fails on
docker errors, so a curl that cannot connect still "succeeds".
Co-Authored-By: Claude Opus 4.8 <[email protected]>
DELETE /api/missions/{id} was a bare `DELETE FROM missions` relying on FK
cascades that only cover mission-owned tables. Everything the mission
provisioned leaked: per-mission runtime container, host workspace dir,
teams (created lifecycle=permanent, so no cascade + skipped by the
ephemeral-teardown path), and every claw's ZeroClaw config, .brain files,
and DB rows. Observed live with 0 missions in the DB: 174 orphaned gateway
claw configs, 7 orphaned teams, 31 agents, 39 .brain files, 6 workspace
dirs, a 4-day-old orphaned container, and 123 detached topology_runs.
delete() now calls reap_mission_resources() before the row delete:
- resolve the mission's teams (mission_teams) → claws (team_members)
- per claw: deprovision_claw (gateway) + rm .brain files + hard_purge (DB),
reusing the manual agent-reap pattern in routes/claws.rs
- delete the permanent-lifecycle teams (team_members cascades)
- delete the mission's topology_runs (else they linger with mission_id
nulled by the cascade and accumulate)
- teardown_container(), now extended to also rm the /mission/repo workspace
dir and tolerate an already-gone container (idempotent for the sweeper +
delete paths)
Runtime-side steps are best-effort (Postgres authoritative; fleet sweeper
reconciles daemon config); DB purges are logged on failure but never block.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Mission agents were burning ~275K tokens producing nothing: the coder had
only file_read and its workspace was the empty ephemeral sandbox, so it
dumped a full spec inline instead of writing files. Two root causes:
1. Risk-profile allowlists used pre-0.8 tool names. `coding_readwrite`
allow-listed `file_write` (renamed to `file_edit` in ZeroClaw 0.8, and
`file_write` now refuses on ephemeral workspaces) and omitted file_edit
/ content_search / glob_search / git_operations — the exact tools the
phase prompt tells agents to use. Since allowed_tools is a strict
allowlist, agents were effectively read-only. Documents the correct
profiles in agent.config.example.toml (they only lived in host config;
the live runtime profiles were corrected via its config API).
2. workspace.path never got set. `agents.<alias>.workspace.path` is an
Option<PathBuf> the ZeroClaw Configurable macro skips from prop
enumeration, so provision_claw's set_prop always 404'd and the whole
call errored into a swallowed eprintln. Removes the dead set_prop and
pins the workspace out-of-band: MissionRuntimeProvisioner::
pin_agent_workspaces patches the shared config file on the per-mission
container (format-preserving via toml_edit, atomic temp+mv); the daemon
applies it on the same reload that surfaces the freshly-provisioned
claws. Covered by unit tests for the TOML stamp.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Deprecated ZEROCLAW_WORKSPACE env var (schema.rs:17467) is used by
the daemon as a legacy config-dir pointer that overrides everything
else. Setting it to /mission/repo made the mission daemon compute
its config dir as /mission/repo/.zeroclaw (empty) and fall back to
defaults — zero agents loaded.
This is the actual root cause of Unknown agent errors on WS. The
seed-mount + admin/paircode/new + per-node-agent-injection fixes
we shipped earlier were correct but couldnt take effect because
the daemon wasnt reading our bind-mounted config at all.
Per-agent workspace pinning belongs in config.toml as
agents.<alias>.workspace, not env.
Fresh runtimes had zero agents in their config so WS handshake with
?agent=scout returned 400. Bind-mount the shared runtimes data dir
so per-mission gateways inherit the seeded claw_* agents.
Per-mission pairing (minted via /admin/paircode/new) still works
against the shared devices.db — each mission gets its own accepted
token. Concurrency caveat on sqlite sessions.db documented in the
const doc comment.
Fresh gateways sometimes boot claim-ing already paired (no
pairing_code in the log banner), which broke the log-scrape approach.
Instead, docker exec into the container and hit the localhost
/admin/paircode/new endpoint that always mints a fresh one-time
code and returns JSON we can parse.
The seed-mount approach didnt work: even with the shared runtimes
data dir bind-mounted, a fresh gateway instance mints a new pairing
key and requires re-pairing. The topology_worker connect returned
401 forever.
New approach — per-mission gateways self-pair:
- Provisioner tails container logs after start, extracts the
X-Pairing-Code from the boot banner
- Persists it on missions.runtime_pairing_code (migration 0059)
- topology_worker constructs ZeroClawDriveExecutor with THAT code
via from_env_for_gateway_with_code, which triggers the lazy
/pair handshake on first turn and caches the returned bearer
Drops the shared-runtime data-dir mount — each per-mission gateway
now owns its own state, restoring the C3 isolation guarantee.
Fresh mission runtime containers had no ZEROCLAW pairing token so
the topology_worker got 401 Unauthorized on WS connect. Mount the
shared runtimes /root/clawmates-runtime/data as /zeroclaw-data so
the gateway boots pre-paired and accepts the servers ZEROCLAW_TOKEN.
Seed dir overridable via CLAWMATES_RUNTIME_SEED_DIR.
Known caveat: sqlite sessions dir is shared across concurrent
mission runtimes. Fine while topology_worker runs sequentially per
mission; next iteration should copy-on-write per-mission.
UUIDv7 encodes time in the leading bytes so 12-hex prefixes are
NOT unique across missions minted in the same second. Docker
accepts up to 253 chars; use the full uuid.
- mission_runtime::spawn_sweeper: force-removes runtime containers
for missions terminal for >=30 min, clears runtime_endpoint. Wired
into clawmates-server main().
- docker-compose socket-proxy: NETWORKS=1 so bollard.connect_network
can attach containers to clawmates_edge for provider egress.
- phase_runner ordering: ensure_checkout BEFORE ensure_container so
the mission dir exists before docker mounts it.
- provisioner: mkdir_p the mission dir defensively for research-only
missions that skip checkout entirely.
- migration 0058: adds missions.runtime_container_name + runtime_endpoint
- new mission_runtime module (bollard): ensure_container /
teardown_container. Container is spawned on clawmates_core +
clawmates_edge networks with just /var/lib/clawmates-missions/{id}
bind-mounted so agents scoped to /mission/repo can only see this
missions repo.
- provider API keys forwarded from the server envs so per-mission
runtimes inherit them.
- Mission struct + repo helpers updated for the two new columns +
set_runtime_binding().
- Unit tests cover container naming determinism + entropy.
Not wired to the orchestrator yet — that lands in slice 2.
