on_launch demanded a team template too — "pick teams in the wizard" — so a
microvm mission still could not launch after the first two gates were exempted.
Three separate places required a claw graph for a path that runs one `claude -p`
inside a VM: routes::missions (draft→running), phase_runner::launch_phase (no
matching teams → stay pending), and here.
Returning before team materialisation rather than filtering its picks: claws that
never run are not a cheaper version of the same thing, they are a runtime binding
and a pairing code describing something nothing speaks to.
Also stops provisioning the per-mission ZeroClaw container for a microvm mission.
The first real run was observed starting one and leaving it holding a pairing code
and ~3 GB of image for the life of a mission that never contacts it.
461 tests pass, clippy clean.
Found by running one: the mission was created with runtime_kind='microvm' and
then refused to launch with a bare 400, because draft→running requires a
materializable team. Past that, `launch_phase` returns early when a phase has no
matching teams — so even with the launch allowed, the phase would have sat
`pending` forever while the log said only "no matching teams", and the executor
would never have been reached.
Neither check applies to this path: microvm_executor runs the agent CLI directly
in the VM, so there is no claw graph to materialise. Satisfying the checks by
attaching a team template would have provisioned claws that never run.
The repo checkout still happens — the VM needs the repository.
461 tests pass, clippy clean.
`runtime_kind='microvm'` passes the DB CHECK, is honoured by placement, and now
has an executor — but `POST /api/missions` rejected the value with 400, so the
only interface that creates missions could not produce one. And `backend`, which
selects the per-CLI rootfs, was not in the create payload at all: it existed as a
column and as a parameter to `vm_create`, with nothing able to set it.
microvm needs no target_node_id at create time, unlike local_herdr: placement
resolves a KVM-capable node at launch and fails the launch when there is none, so
an explicit target is a request rather than a requirement.
461 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
`runtime_kind='microvm'` placed a mission on a KVM-capable node and then nothing
executed it: config accepted without a reader, one of the four seams this project
keeps closing. This is the reader.
`microvm_executor` — inject → run → collect → destroy, the shape copy mode
already proved for containers with a VM boundary instead of a namespace one. The
checkout goes in as a tar, the work comes back as a tar over the SAME host path,
so `mission_delivery::capture_phase_diff_at` needs no change at all.
The agent is told NOT to push, unlike the container path's session prompt. Two
reasons: delivery is already host-side and diffs the collected tree against the
recorded clone point (covering committed, staged and unstaged work in one pass),
so pushing would add a second untested way for work to arrive; and pushing would
mean forge credentials inside the VM, when the point of collecting is that the
guest never holds them.
Exactly ONE topology_runs row (tier='microvm'), mirroring launch_direct_session:
close_finished_phases, evaluation, capture and delivery all key off those rows,
and a second completion path would be a second way for a phase to finish with one
of them untested. The row and the phase flip happen BEFORE any fallible VM work,
so a missing token or a node that lost its capability shows up as a failed run an
operator can see — not a phase that stays pending and retries every ten seconds.
Fail-closed points, each the reader for a guarantee built earlier:
- credentials resolve BEFORE the VM boots, so a missing subscription token
fails the phase instead of booting a VM whose agent sits unauthenticated
- a VM reporting egress:false is REFUSED, which is what makes create's
egress/egress_host/egress_guest fields more than decoration — a turn without
egress does not fail, it hangs
- the injected checkout is PROVEN present in the guest before an agent turn is
spent on it; an inject that reports success while landing nothing would
otherwise become an agent reporting an empty repository
- work is collected even when the agent exits non-zero — a turn that failed
partway still wrote files, and a retry needs to see them
- a turn that ran but could not be collected is a FAILED phase, not a happy one
- destroy runs on every exit path, or an 8 GB sparse rootfs leaks
Two integration gaps found while wiring, both of which would have produced a
mission that completed having delivered nothing:
- `capture_finished_coding_phases` pulls work out of a CONTAINER before
capturing. A microvm mission has none, so the docker connect would fail, the
loop would `continue`, and capture would be skipped forever while the phase
sat marked completed. Its work is already collected by the executor.
- `launch_phase` provisioned a runtime container, copied the checkout into it
and wrote a runtime binding + pairing code describing a runtime nothing uses;
and the orchestrator's workspace pin — deliberately FATAL — would have failed
a microVM launch on a container it was never going to use.
461 tests pass, clippy clean.
NOT YET PROVEN END TO END: no mission has run through this path. The pieces under
it are each verified on tank (image, credentials, egress, a real agent turn), but
this executor has only been compiled and unit-tested. Deploy + one real microvm
mission is the remaining step.
Co-Authored-By: Claude Opus 5 <[email protected]>
The guest still has no network interface, and now that is the design rather than
a gap. Its only route out is an HTTP CONNECT proxy: agent CLI -> 127.0.0.1:3128
in the guest -> vsock 9002 -> a per-VM Unix socket on the host -> TLS to an
allow-listed host.
Why not TAP + iptables, which is what the Firecracker write-ups do — measured,
not argued:
- `ip tuntap add` is DENIED to the daemon user (needs CAP_NET_ADMIN), so TAP
would need root to pre-provision devices, the same privilege detour the
loop-mounted rootfs already forced.
- tank's FORWARD policy is DROP with Docker and Tailscale chains, so rules
would have to be inserted at position 1; appended ones die silently.
- a leaked TAP is a new class of host litter to reap.
CONNECT needs no privilege at all and is better on the merits: the client hands
us the HOSTNAME, so resolution happens host-side and the guest needs no DNS or
resolv.conf; the allow-list is by name, not address; and nothing in the guest can
reach the network except through one function. The guest end parses nothing and
enforces nothing, so a compromised agent cannot argue with the policy.
Rests on one measured fact: `claude` honours HTTPS_PROXY. With the proxy at a
closed port, `claude -p` fails ConnectionRefused instead of answering.
THE RESULT: a real agent turn now completes inside a VM with no network card, on
subscription auth — `claude -p` replies VM-OK. The selftest asks for it whenever
CLAUDE_CODE_OAUTH_TOKEN is present and SKIPS loudly when it is not, since it
spends a little of the plan.
The audit log earns its keep immediately: during that turn the proxy logged
`egress DENIED http-intake.logs.us5.datadoghq.com` — the CLI's telemetry, which
the mission container permits today without anyone deciding to.
Three bugs found by the checks rather than by review:
- `env_pairs` returned early when a caller sent no env, so the proxy address
was never added and `curl` in a VM with a working tunnel reported "Could not
resolve host". Absent env means "the caller sent none", not "this command
needs no environment".
- the deny check PASSED for the wrong reason — DNS was failing, so nothing was
refused by the allow-list at all. It now requires a 403 from the proxy, so it
cannot go green on a broken tunnel.
- `host_allowed` accepted `evil.test/api.anthropic.com`, which ends with an
allowed suffix. Hostnames are now validated against a character class, which
also refuses IP literals so an address cannot sidestep a list of names.
- `BufReader::into_inner()` discards buffered bytes: wrapping the stream twice
would have dropped the start of the TLS handshake and stalled a tunnel that
looked established. One reader now spans the request, and anything buffered
past the headers is forwarded as payload.
`iproute2` is in agent-toolchain because it is load-bearing: the guest's `lo`
starts DOWN, and while it is down a listener on loopback BINDS and then refuses
every connection with ENETUNREACH. fcagent finds `ip` by absolute path — as pid 1
its PATH comes from the kernel, and execvp's fallback excludes /usr/sbin, where
Debian puts it.
Egress needs both ends up, so `create` reports `egress` and the guest's `ping`
reports its own half. A VM without it is legal but never silent.
Verified on tank: 16/16 with backend=claude (create 1428 ms), 12/12 on the
default rootfs, no leaked processes, VM dirs or proxy sockets. 457 tests pass,
clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
B4.4 had the microVM path share `forwarded_provider_env(auth)` with the
container path, on the reasoning that the two must not diverge. That was wrong
in the one direction that costs money: gw-04 has CLAWMATES_RUNTIME_AUTH unset,
so the container path forwards ANTHROPIC_API_KEY today — and a VM would have
received it. Claude Code ranks the API key ABOVE the subscription's OAuth token,
so the VM would have worked perfectly while billing per-token against a plan we
already pay for. No error, no symptom but the invoice.
`microvm_provider_env` is subscription-only BY CONSTRUCTION: it does not take
the auth mode as an argument and does not read CLAWMATES_RUNTIME_AUTH at all.
Taking the mode as a parameter would mean one unset variable on a new host
silently turns the API key back on. The container path is unchanged and still
honours the operator's mode — the divergence is now deliberate, with the reason
at the definition.
Two other fail-closed rules fall out of it:
- A missing or blank subscription token REFUSES the launch rather than
returning an empty environment. A VM with no credential does not error;
`claude -p` hangs, which reads as a phase stuck at `running` with nothing in
the logs. The refusal names the variable.
- An unrecognised backend is refused rather than handed the Anthropic token.
GLM and Kimi reach their own endpoints via ANTHROPIC_BASE_URL and that
contract is not settled yet; guessing it would send a subscription
credential to z.ai.
Measured on tank, and this is the end-to-end proof B4.4 could not give:
`claude -p` in the agent-claude image with the real subscription token replies
"OK". Injecting the token in a VM moves the failure from "Not logged in" to a
network error, so the credential channel is accepted by the CLI — the VM's
remaining problem is egress (#49), not auth.
447 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
`claude -p` in the VM failed with "Not logged in". The credential now travels on
the exec op: `env` on `vm_exec` → fcagent → the command's environment. An env var
rather than a file because the per-VM rootfs dies with the VM but an env var
never touches the guest disk at all.
**Every problem in an env entry fails the exec.** The tempting alternative —
skip the entry we cannot use and run anyway — produces a `claude -p` with no
credential, and that does not error, it HANGS. A phase stuck at `running` for
ten minutes with nothing in the logs is exactly what a missing token looked like
on the container path. Names are validated ('=' or NUL would define a different
variable than the one asked for via putenv semantics), values must be strings,
and errors name the key and never the value — an error string travels back over
the wire and into logs.
One list of which credentials travel: `forwarded_provider_env` reuses
`forwarded_provider_keys`, and the container path now reads it too. If the two
execution paths diverged, a mission would behave differently depending on where
it landed — including the expensive way, where one path forwards
ANTHROPIC_API_KEY and bills it while the other uses the subscription. A blank
value is omitted rather than forwarded empty, so `claude` reports having no
credential instead of failing authentication with one.
Verified on tank (`--vm-selftest` backend=claude, 13/13, create 1532 ms): an
injected var reaches the guest command over the real vsock wire, and an
unusable entry comes back ok:false with no rc.
FINDING — the CLI leg remains UNPROVEN, and deliberately so. The guest has no
network interface: `create` writes boot-source, drives, machine-config and vsock
and no `network-interfaces` key, and a booted guest has no routes, no
resolv.conf, no DNS and no TCP. So `claude -p` cannot reach the API whatever
credential it holds. Injecting the real token would have proven nothing, because
the failure would have been network and not auth. Filed as B4.6 (task #49) with
the TAP-vs-vsock-proxy trade-off; B4.5 is now blocked on it.
444 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
The only rootfs on this track came from clawmates/agent-terminal:dev. Mounted,
it held git and nothing else: no claude, no node, no cargo. A VM booted from it
looks perfect and cannot run a mission, so B4.5 could have been written and
never verified.
images/agent-toolchain — the shared mission toolchain (node 22, git, rust +
cargo-audit, gitleaks/trivy/semgrep, tea/gitea-mcp), lifted from the proven
deploy/clawmates-runtime image minus the zeroclaw daemon: a microVM mission runs
the direct-session model, so there is no daemon to host. A base image rather
than three self-contained Dockerfiles because this layer is ~3 GB and the real
risk is scanner and toolchain versions drifting between per-CLI images — the
evaluator runs the project's own suite to check a claim, so `cargo` present in
one image and absent in another makes the same mission pass or fail by backend
with nothing saying why.
images/agent-claude — plan A6, first of three: the pinned CLI and its env
contract only, so bumping Claude Code does not rebuild the toolchain and cannot
disturb agent-kimi / agent-glm. HOME=/root with an empty .claude for B4.4 to
inject into; no ANTHROPIC_API_KEY, since it silently overrides the subscription
OAuth we already pay for.
Both the builder and the node selftest now ASK the guest for the CLI the image
is named for, instead of trusting the name. `required_cli` maps claude/kimi/glm
to a probe; an unrecognised backend reports unchecked and prints SKIP rather
than passing quietly.
Verified on tank:
- rootfs-claude.ext4 boots; git, node, cargo, a real git commit all work
- `claude --version` → 2.1.220 over vsock, in both the builder and
`--vm-selftest` (11/11, create 1498 ms)
- negative control: the same builder run against agent-terminal with
FC_CLI forced reports `cli rc=127 claude: not found` and exits 1, so the
green result above is a measurement and not a default
- `claude -p hello` fails with "Not logged in · Please run /login" — the CLI
runs headless in the VM, and B4.4 only has to supply the credential
- no leaked firecracker processes or vm dirs afterwards
437 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
`vm_create` takes a backend name and boots `rootfs-<backend>.ext4`; NULL
or "default" boots the golden image. Makes the per-CLI images from B4.1
actually reachable (one image per CLI, per A6).
A missing image is an ERROR naming the file and how to build it, never a
quiet fall back to the default. That fallback is the tempting version and
the wrong one: it would run a claude mission in a kimi VM, or in a rootfs
with no CLI at all, and report success for whatever came out. Verified on
real hardware, not just in a unit test — the selftest asks for an image
that does not exist and FAILS if it boots.
`create` now reports the rootfs that actually booted, not the one that was
requested, so a mission artifact can show the wrong VM ran.
The migration adds no CHECK constraint listing the CLIs. Which images
exist is a property of the NODES, not the schema; a constraint would need
migrating for every new image while still not guaranteeing the image
exists anywhere. The node validates and names what is missing. Backend
names are `[A-Za-z0-9_-]` and rejected rather than sanitised, since they
become filenames.
Verified on tank: default backend 8/8; `CLAWMATES_FC_BACKEND=agent-terminal`
9/9 including the absent-image check, create in 910ms on a rootfs built
from a real Docker image. 435 tests green, no leaked processes or VM dirs.
Co-Authored-By: Claude Opus 5 <[email protected]>
The python guest agent only ever worked because Firecracker's CI Ubuntu
image happens to ship python3. NONE of our images do — agent-base has
neither python nor git, agent-terminal has git but no python — so it
could never have run in a real mission rootfs. An agent that dictates
what must be installed in the image has the dependency backwards.
crates/bins/fcagent is a 905K static x86_64-unknown-linux-musl binary
that needs nothing from the rootfs it is dropped into. The wire is
unchanged on purpose — 4-byte BE length + JSON, ops ping/exec/put/get —
so microvm.rs and microvm_client.rs needed no edit at all.
std has no AF_VSOCK and the workspace denies `unsafe`, so it uses the
`vsock` crate. `process_group(0)` gives each command its own group without
unsafe, so a command that spawns background children can be killed
wholesale rather than outliving the run.
A unit test caught a bug that would have broken EVERY exec: sourcing the
image-env file with `. env.sh 2>/dev/null; cmd` returns rc=1 WITHOUT
running cmd, because `.` on a missing file makes a non-interactive POSIX
shell exit immediately. On any rootfs lacking that file every command
would have failed while looking like an ordinary non-zero exit. Guarded
with `if [ -f ]` now.
Other places a failure must not borrow an outcome's representation: a
killed command reports ok:false with no rc (not rc=124, which would read
as a build failure); `get` on a missing path is an error, not an empty
archive; a signalled process reports 128+signal rather than success.
Verified on tank: --vm-selftest still 8/8 with the agent swapped
(create 949ms, wire identical), fc-node-setup 8/8, and — the point of the
change — a rootfs built from clawmates/agent-terminal:dev, which has NO
python3, boots and reports `git version 2.39.5` from inside the VM.
Also fixes a shell bug in fc-build-rootfs.sh: $HOME in a double-quoted
default expanded on this Mac, so it looked for the node's binary under
/Users/quantum on a Linux host.
Co-Authored-By: Claude Opus 5 <[email protected]>
Until now microVMs booted Firecracker's CI Ubuntu image with a python
guest agent bolted on: no git, no toolchain, no CLI. Fine for proving
vsock, useless for running a mission.
Builds FROM a Docker image rather than debootstrapping, because the
per-CLI images (agent-claude / agent-kimi / agent-glm, per A6) are
already Dockerfiles with a tested env contract. Rebuilding that as a VM
image by hand would mean maintaining the same facts twice and finding the
drift in production.
Two things the obvious version gets wrong and this does not:
- `docker export` gives the filesystem with NONE of the image metadata:
no ENV, no ENTRYPOINT, no WORKDIR. A CLI relying on ENV PATH or HOME
would silently behave differently in the VM. The env is extracted
separately and written to /etc/profile.d.
- the ext4 is filled through a mount, not `mkfs -d`, which cannot
handle the device nodes and hard links a container image may contain
and fails late and cryptically when it hits one.
The guest agent is copied from the golden rootfs rather than re-emitted,
so there is ONE copy of the protocol on the node instead of two that can
drift.
It boots what it builds and asks the image for what a mission needs —
git, the profile env, a writable /mission — rather than assuming. An
image that builds and cannot boot is worse than no image, because it
looks finished.
FINDING, and it blocks B4.2: NONE of our images ship python3, so the
python guest agent cannot run in any of them. agent-terminal has git but
no python; agent-base has neither. The guest agent must not dictate the
image's contents — it needs to be a static binary. This script correctly
refuses to build an image whose agent cannot run, so the failure is
visible rather than a VM that boots into nothing.
Co-Authored-By: Claude Opus 5 <[email protected]>
cm_api::microvm_client::MicroVm wraps the node's vm_* ops as typed calls
over the existing hub request/response channel: create / inject / exec /
collect / destroy, plus list() for reaping. No new transport.
Fixes a wire-contract mismatch B2 would have shipped. `Uplink::Result`
declares `output: String`, but the node's vm_* handler returned a JSON
object. The frame then failed to deserialize and hit the uplink match's
`Err(_) => {}` arm, so the reply VANISHED and every vm_* call would have
timed out after 20s with nothing anywhere explaining why. The node now
sends a string, matching the contract rather than what looked tidier.
That silent arm is fixed too: an unparseable frame now logs the node, the
parse error and the frame head, and says explicitly that the request it
was answering will time out. It is the arm that would have hidden this.
Two more places where a failure must not borrow a legitimate outcome's
representation:
- vm_exec returning no `rc` is an error, not a zero. A missing exit code
means the guest did not report one; reading it as success is how a
failed command becomes a passing phase.
- vm_collect on a missing path is an error, not an empty archive — an
empty tar looks exactly like a run that produced nothing.
Timeouts: the hub's deadline is the guest's plus 30s, saturating. A
caller passing a huge budget would otherwise wrap to a tiny timeout and
turn a long agent turn into a spurious transport failure. clippy caught
the tautological assertion in the first version of that test, which is
what surfaced the overflow.
Verified: `--vm-selftest` on tank still 8/8 after the output-type change
(create 950ms), 427 tests green.
Co-Authored-By: Claude Opus 5 <[email protected]>
create / inject / exec / collect / destroy / list, riding the node's
existing frame dispatch ({t, id, …} -> {t:"result", id, ok, output}), so
no protocol change was needed. Control is length-prefixed JSON over
vsock; the serial console stays a log, because feeding a guest over stdin
races its startup and arrives half-consumed.
DEVIATION FROM THE PLAN, deliberately: this does NOT implement
cm_sandbox::SandboxDriver. That trait is container-shaped —
attach_pty/resize_pty/argv exec — while missions need
create -> inject -> run -> collect -> destroy. Conforming would mean
building PTY-over-vsock and window-resize semantics that no mission path
calls, purely to satisfy a signature. We give up automatic RemoteDriver
marshalling; orphan reaping is a label/id sweep either way.
Three traps from the B0 spike are handled in code rather than remembered:
- Firecracker does NOT unlink its vsock UDS on exit, so destroy unlinks
it explicitly, and the selftest ASSERTS it is gone. Assuming the VM
tidies up after itself is how the mission checkout accumulated four
uid bugs.
- firecracker is spawned via setsid and killed as a process GROUP, so a
background child cannot outlive the VM holding its workdir open.
- create does not return until the guest agent has answered a ping. A
VM that booted but serves nothing is worse than one that failed, so a
half-created VM is destroyed rather than left registered.
A vm id becomes a path component, so ids are restricted to [A-Za-z0-9_-]
and REJECTED rather than sanitised — a caller that sent `../../etc`
wanted something we should not guess at.
Verified on tank through the real Rust path, as the daemon user, with no
sudo: `clawmates-node --vm-selftest` -> 8/8, create in 986ms, and the
host left with zero firecracker processes and zero VM directories. The
selftest asserts every step, including that a destroyed VM can no longer
be exec'd; a test that only reports the steps it completed cannot
distinguish "passed" from "stopped early".
Co-Authored-By: Claude Opus 5 <[email protected]>
The capability probe reported `kvm: false` on tank and morpheus while the
device sat right there: /dev/kvm is `crw-rw---- root:kvm` and the kvm
group was EMPTY, so the daemon — an ordinary user — could not open it.
The B0 spike missed this entirely because it ran everything under sudo.
This is exactly why the probe opens the device rather than stat-ing it;
a stat-based check would have reported both nodes capable and every
microvm mission would have failed at launch instead of at placement.
fc-node-setup.sh now fixes the group itself, or says precisely what to
run when it cannot.
Co-Authored-By: Claude Opus 5 <[email protected]>
Phase B step 1, on top of the B0 spike that proved microVMs boot here.
KVM is a HARD predicate, not a preference. gw-04 — where every mission
runs today — is itself a VM without nested virtualisation and has no
/dev/kvm, so a microvm mission landing there cannot start at all. The
scheduler therefore has to be able to tell nodes apart, which means the
node has to report what it can host.
Nodes gain a `capabilities` jsonb, populated from a probe on the node
rather than from configuration: /dev/kvm either exists there or it does
not, and nothing on the server can make it appear. The probe OPENS the
device rather than stat-ing it, because it can exist while being
unopenable (wrong group, or a container without the device passed
through) — which is precisely how firecracker will fail.
`microvm` requires BOTH kvm and a firecracker binary. A node with KVM
but no binary looks capable by the obvious test and fails at launch; a
node with the binary but no KVM is gw-04.
Placement fails the launch when no capable node exists, rather than
letting a mission sit in 'running' with nowhere to run. An explicit
target_node_id is treated as a request, not a guarantee — it is honoured
only if that node actually reports the capability.
`capabilities` defaults to '{}' NOT NULL so a node that has never
reported fails every predicate: an unqueried node and an incapable node
must be indistinguishable to the scheduler, because scheduling onto a
node whose abilities are unknown is how you get a mission that cannot
start and does not say why. The report replaces rather than merges, so a
capability the node has LOST disappears instead of leaving a stale true.
Co-Authored-By: Claude Opus 5 <[email protected]>
Phase B step 0. Before writing any driver, establish that Firecracker
works on this hardware — the plan called it greenfield, and an
orchestrator built against an unproven runtime is a lot of code betting
on an assumption.
It works, and comfortably: a microVM boots, runs our init, writes a
file and shuts down in ~650-910ms wall clock, with the kernel reaching
our init at 234ms. Host->guest RPC over vsock (AF_VSOCK port 9001, no
network stack) round-trips in 27ms.
The script installs and then PROVES, because installing is not working.
It reports success only after a VM has actually booted and run our code.
Four findings from the spike that the driver must account for:
- Firecracker does NOT unlink its vsock UDS on exit, and leaves it
owned by whoever ran the VM. A driver running as anyone else cannot
clean it up — the same uid trap that cost this codebase four bugs on
the mission checkout. The driver owns the socket path lifecycle.
- tank's FORWARD policy is DROP (Tailscale/Docker), confirming the
article's warning: VM networking rules must be inserted at position
1, not appended, or return traffic dies silently.
- Feeding commands to the guest over the serial console races the
shell's startup and arrives half-consumed (`# ho FC-GUEST-ALIVE`).
The guest runs an init script; stdin is not a control channel.
- `sha256sum -c` compares by filename, so a download saved under any
other name fails for a reason unrelated to integrity. A check that
fails for the wrong reason teaches you to ignore it — compare the
hashes directly.
tank and morpheus are ready. architect requires interactive sudo, so it
is deliberately not provisioned rather than worked around.
Co-Authored-By: Claude Opus 5 <[email protected]>
Verified on the deployed stack: verify-mission-delivery.sh all → 9/9,
with the noop negative control showing 'phase 0 failed 0' where the same
shape read 'completed' in mission 019fcf62.
Co-Authored-By: Claude Opus 5 <[email protected]>
The last open item in the silent-success class: a coding phase that
changed no files reported `completed` — the same status a phase gets for
delivering tested, reviewed, pushed work. Mission `019fcf62` completed
that way with its agents silently unpinned from the repo, and nothing in
the platform disagreed; it was found by a script diffing the forge.
The verdict is applied at capture rather than at completion, because
capture selects on `status = 'completed'` — the platform does not know
whether a phase produced anything until after it has already finished.
Three conditions must hold before failing a phase, because a false
positive here fails honest work: the phase is a coding phase (research
phases legitimately write nothing to the tree), the diff was actually
computed (an uncomputable diff also reports zero files — blaming the
agent for a platform fault is the same defect wearing different
clothes), and `allow_empty` is not set. Only an explicit `true` opts
out, so a typo leaves the check armed. Registered in phase_config with
its reader named, per the seam-2 rule.
Also closes an ordering hazard this exposed: capture is batched and runs
after a phase completes, so a backlogged mission could close as
'completed' and only then have capture discover an empty phase — leaving
a 'completed' mission holding a 'failed' phase, unfixable because the
mission-close CASE only touches 'running' rows. A repo-bearing mission
now waits for its work to be captured before closing.
Adds a `noop` scenario to the harness: a phase told to change nothing,
which PASSES only when the phase comes back `failed`. Same discipline as
the uid self-test — a check that has never been seen to fire has not
been shown to work.
Co-Authored-By: Claude Opus 5 <[email protected]>
Verified with CLAWMATES_MISSION_FS removed from gw-04's .env — compose
passes it through as empty, which under the old opt-in logic would have
selected bind. scripts/verify-mission-delivery.sh all → 7/7.
Co-Authored-By: Claude Opus 5 <[email protected]>
Copy mode shipped opt-in so that changing how every mission receives its
code required someone to type it. Four production missions and a
fail-closed harness later, opt-in is the riskier setting: the bind path
is the one with four documented work-loss incidents, and leaving it as
the default means the untested path runs whenever nobody sets the
variable. `CLAWMATES_MISSION_FS=bind` still selects it; anything else —
unset, empty, misspelt — gets copy mode, so a typo lands on the safer
path rather than the one being retired.
Also fixes a real leak found while scoping the deletion below: the git
helper built its `safe.directory` argument with `Box::leak`, justified as
"the process is short-lived". That is true of a CLI and false of cm-api,
which is a long-running server — so it leaked one allocation per git
call, growing with every phase of every mission.
The A5 deletion is NOT done here, and two of its items should never be
done:
- `scrub_remote_credentials` is a security control, not a uid
workaround. Copy mode uploads the whole `.git` into a container the
agent controls as root, which makes stripping the token from
`.git/config` more necessary, not less.
- `has_local_work` / `checkout_in_use` guard `fetch_and_reset` at every
phase launch and have nothing to do with who writes the checkout.
The host checkout still persists across phases under copy mode —
mission `019fcf62` shows the marker firing there. Deleting them
reintroduces PRIOR-PHASE-WORK-WAS-LOST.
The rest (`share_repository_across_uids`, `clear_stale_commit_editmsg`,
`-c safe.directory`) are genuinely obsolete under copy mode but stay
while `bind` remains selectable: a workaround may only be deleted once
the situation it works around can no longer be chosen.
Co-Authored-By: Claude Opus 5 <[email protected]>
Verified against the deployed stack: scripts/verify-mission-delivery.sh all
→ 7/7, chain phase 0 now files=1 pushed=true (was 0 files, no error).
Co-Authored-By: Claude Opus 5 <[email protected]>
A verification run against the deployed stack found a chain mission whose
phase 0 reported `completed` with zero files, no commit error and no push
error — indistinguishable from a phase that correctly had nothing to do.
Three separate defects had to line up, each of them the same shape: a
failure sharing its representation with a legitimate negative result.
1. `pin_agent_workspaces` embedded the whole config in one `sh -c` argv.
That works until the file grows — config gains a block per provisioned
claw — then fails with `argument list too long`. Now written through
the tar upload API, which has no argv limit, so the failure mode is
gone rather than merely further away.
2. A failed pin was logged "(continuing)". Without the pin, agents write
to their sandboxes and the committer finds nothing in /mission/repo —
the mission cannot deliver, so the launch now fails where someone is
still looking. The restart that applies the pin is fatal for the same
reason.
3. `capture_phase_diff_at` swallowed `git diff` failures with
`unwrap_or_default`, so an unreadable base landed `empty: true,
files_changed: 0` — byte-identical to an honest no-op. The error is now
recorded as `diff_error`, and an empty patch that came from a failed
diff is no longer trusted to mean an unchanged tree.
Adds scripts/verify-mission-delivery.sh, which found #1 and #2 on its
first real run. Its probes are fail-closed: no placeholder values, a
self-test that proves the uid probe can detect the split it looks for,
and FAIL-NORUN for a scenario that never executed. Its own first version
had this bug too — a `die` inside `$(...)` exited the subshell, so a run
that could not authenticate printed "all checks passed" and exited 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
With the flag set, ensure_container omits the /mission bind, the checkout
is pushed into the container at phase launch, and the agent's work is
pulled back before capture.
The simplification that makes this small: sync_out unpacks over the SAME
host path the checkout came from. The host directory stays a server-owned
staging area with exactly one writer, and capture_phase_diff_at needs no
change at all — it still finds a normal checkout exactly where it always
has. Delivery, gating, commit and push are untouched.
Two failures are deliberately loud rather than silent:
- copy-IN failure fails the phase launch. Continuing would start a phase
against an empty directory, and the agent would cheerfully report having
done work on a repo that was not there.
- copy-OUT failure SKIPS capture. Capturing anyway would diff a stale host
tree and record "no changes" for work that exists — success reported for
nothing, which is the exact failure mode this codebase keeps paying for.
Opt-in: the bind path is what production has run since the beginning, and
the test asserts a near-miss value leaves it there rather than silently
switching every mission.
414 tests, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
The first half of removing the shared bind mount. Not wired yet — this
adds the mechanism and its tests.
One cause, four fixes so far: .git/objects permission denied
(core.sharedRepository), the capture base being overwritten each phase,
COMMIT_EDITMSG root-owned, and reset --hard deleting a prior phase's work
(.git/clawmates-in-use). core.sharedRepository was never a general
solution — it covers objects and refs, and every OTHER file git touches
is a fresh opportunity. Copy-in/copy-out removes the cause instead: the
agent owns its filesystem with no second writer.
Measured before building, because the plan named copy cost as the open
risk: a real 65 MB checkout of this repo copies in 0.23s and out 0.18s on
gw-04. Not a risk at this size; re-measure an order of magnitude larger.
No compression — the payload crosses a local socket, so gzip would spend
CPU to save nothing.
Two safety properties, both tested:
- The archive comes back from a container the agent controls as ROOT, so
it is untrusted input. A `../ESCAPED` entry must not write outside the
destination. The test writes the tar header bytes by hand because the
tar crate refuses to BUILD such an entry through its safe API — which
is reassuring, but means the hostile case has to be constructed the way
an attacker would.
- Symlinks are packed as links, never dereferenced. Following them on
copy-IN would smuggle host files into the container; the test plants a
host secret behind a symlink and asserts its contents never appear in
the archive.
Ownership is deliberately not preserved on unpack: the archive's uids are
the container's root, and re-applying them on the host would recreate the
exact uid split this exists to remove.
413 tests, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
Checking before deploying caught a mistake in the previous commit. The
seed dir on gw-04 is 1.7 GB and the first version copied all of it per
mission — tens of seconds each, and ~17 GB across ten concurrent
missions.
1.5 GB of that is .rustup: a Rust toolchain that installed itself into
the data dir back when HOME=/zeroclaw-data and the image had no
toolchain. The image now ships Rust at /usr/local/cargo, which is what
the container's PATH actually resolves — verified live. The data-dir copy
is dead weight and is not even reachable.
SEEDED_PATHS now copies only what carries per-mission identity or
secrets: .zeroclaw (config.toml with the door token, sessions.db,
devices.db), clawmates-mcp.json, .claude + .claude.json, .kimi-code,
glm-home, agents. Roughly 46 MB instead of 1.7 GB — about 37x smaller.
Caches and toolchains are deliberately excluded: .rustup, .npm, .cargo,
.cache, .local. They hold no secrets and a mission reads the image's.
Absent paths are tolerated: a fresh deployment has no .kimi-code until
Kimi is first used, and that must not fail container creation.
The test asserts both directions — the token-bearing paths ARE copied
and the caches are NOT — because either mistake is silent: copying
everything just makes missions slow, and copying nothing quietly
restores the credential sharing.
409 tests, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
Every per-mission container bind-mounted the SAME host seed dir as
/zeroclaw-data — shared with each other AND with the singleton runtime.
That directory holds config.toml, which carries the §15 door bearer
token, plus sessions.db and devices.db.
So one mission could read another mission's credential, and anything it
wrote there was inherited by every later mission. teardown_container
only removes /var/lib/clawmates-missions/{id}, so the shared directory
was never cleaned — the contamination was permanent.
The code already knew. The comment on DEFAULT_SEED_DIR names the sqlite
race and calls copy-on-write per mission the long-term fix. This is that
fix: seed_runtime_data copies the seed into
<missions_root>/<mission>/runtime-data at container create, and the
mount points there. Cleanup is free — teardown already removes that tree.
The copy runs in a throwaway container because cm-api cannot see the seed
dir: it hands that host path to Docker but never mounts it itself. The
runtime image is reused so nothing extra is pulled, and `cp -a /seed/.`
copies dotfiles — `/seed/*` would silently skip .zeroclaw/ and produce a
runtime with no config at all.
A copy failure is fatal to container creation on purpose. Falling back to
the shared mount would silently restore the credential sharing this
removes, and silent fallback to a weaker posture is the failure mode this
codebase keeps paying for.
The test asserts path shape rather than behaviour: an edit that points
the mount back at the seed dir restores credential sharing with no other
visible symptom, so the path IS the invariant.
408 tests, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
All three providers run through the SAME `claude` binary, verified live:
Anthropic CLAUDE_CODE_OAUTH_TOKEN -> ANTHROPIC-OK
GLM ANTHROPIC_BASE_URL=https://api.z.ai/api/anthropic -> GLM-OK
Kimi ANTHROPIC_BASE_URL=https://api.kimi.com/coding/ -> KIMI-OK
That is a stronger multi-provider story than a provider-per-implementation:
skills, subagents, MCP, hooks and tool policy are identical across all
three because it is literally the same harness.
The `kimi` CLI (0.31.1, shipped in the image) 401s on this key and is not
needed -- the claude binary reaches Kimi's Anthropic-compatible endpoint
directly. Worth knowing before someone debugs the CLI.
forwarded_provider_keys now ships ZAI_API_KEY and KIMI_API_KEY into
mission containers in BOTH auth modes: they are unrelated to the Anthropic
credential, so the api_key/subscription split does not apply to them. A
mission that selects a backend without its key present would otherwise
fail at the first turn.
Keys persisted in /opt/clawmates/.env and passed through compose.
Co-Authored-By: Claude Opus 5 <[email protected]>
Mission 019fcd0c produced correct work — a reviewed, tested function plus
a REVIEW.md quoting a real cargo test summary — and delivered none of it:
git commit → exit 128: could not open '.git/COMMIT_EDITMSG': Permission denied
The agent ran `git commit` itself inside the mission container (as root),
leaving that file owned by root at 0644. core.sharedRepository covers
objects and refs — .git/index lands at 0666, which is why commits work at
all — but not COMMIT_EDITMSG, which git writes with the default umask.
Unlinking works where overwriting does not: removing a file needs write
permission on the DIRECTORY, and .git/ is owned by the server. Silent on
failure by design, so the commit reports the real error rather than this
speculative cleanup.
Third distinct instance of the same uid-split class (objects, then the
capture base, now this). The pattern holds: the checkout is one directory
written by two users, and each new file git touches is a new opportunity.
Co-Authored-By: Claude Opus 5 <[email protected]>
CLAWMATES_MISSION_EXECUTOR=session makes launch_phase run the whole phase
as a single `claude -p` against /mission/repo instead of driving turns
through ZeroClaw. Opt-in, because silently changing how every mission
executes is exactly the sort of change that should require someone to
have typed it.
It still writes ONE topology_runs row. The entire downstream lifecycle --
close_finished_phases, evaluation, capture, commit, gate, publish -- keys
off those rows, and inventing a second completion path would mean two ways
for a phase to finish with one of them untested. The session is simply a
run with tier='session' and an empty graph.
Spawned rather than awaited: launch_phase runs inside the sweep loop, and
blocking it for the length of a coding session would stall every other
mission.
The session's own summary is logged as diagnostics only. Whether the phase
actually did anything is still decided downstream by capture and delivery
against the repository -- a 0-exit session that pushed nothing was measured
at ~5%, so the agent's account can never be the verdict.
406 tests, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
Rescues session_executor from the stranded spike branch. Multi-provider
missions are not needed for now, so the direct path is worth nailing down:
run a mission as one `claude -p` session against its checkout instead of
routing turns through ZeroClaw.
Measured today against a real checkout in the runtime container, using the
executor's exact argv:
direct `claude -p` 7s, file written
via ZeroClaw minutes per turn, and THREE config failures before
it worked at all (no credential in the mission
container; Write/Edit denied; no tools granted --
the last of which COMPLETED a mission having
written nothing)
Each of those failures came from the same root: with claude_cli, ZeroClaw
is a WebSocket-to-subprocess adapter whose own controls (risk profiles,
tool gating, memory) do not reach the subprocess. The adapter adds
failure modes without adding governance.
What ZeroClaw still earns for the rest of the platform is unchanged and
not in question here: interactive chat, the brain, A2A and door identity,
terminal, agent routines, and non-Claude providers.
Not yet wired into phase_runner — this commit only makes the executor
reachable and keeps it building. SessionOutcome::delivered() still
requires a clean exit AND an observed branch, because a 0-exit session
that pushed nothing was measured at ~5%.
405 tests, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
Switching agents to claude_cli left missions hanging: the per-mission
container had claude_cli configured but no credential, so `claude -p`
waited forever. A phase sat at `running` for ten minutes with nothing in
the logs — no error, because there is nothing to error on.
The original subscription design assumed a persisted `claude /login`
under a bind-mounted $HOME. That holds for the shared runtime and NOT for
a mission container, which gets its own data dir and therefore no login.
So subscription mode now forwards CLAUDE_CODE_OAUTH_TOKEN.
The two Anthropic credentials remain mutually exclusive, and there is now
a test asserting it in both directions: Claude Code ranks ANTHROPIC_API_KEY
above the OAuth token, so shipping both bills the API while the deployment
believes it is on the subscription — visible only on the invoice.
Deployment: CLAWMATES_RUNTIME_AUTH=subscription and CLAUDE_CODE_OAUTH_TOKEN
added to compose + .env on gw-04.
Co-Authored-By: Claude Opus 5 <[email protected]>
provider_alias_for now resolves Claude models to `claude_cli.default`,
which spawns the actual `claude` binary, instead of `anthropic.default`,
which posts to the raw API with Claude Code identity headers. Agent work
is ~99% of tokens, so this moves essentially all of it onto the Max
subscription and onto the supported client.
The judge deliberately stays on the API key. If both rode one credential,
a single subscription limit would blind the verifier at exactly the
moment there is most to verify; this way a throttle degrades missions but
verification keeps working.
Runtime config (applied on gw-04, reloaded via loopback — remote admin
reload is disabled by design):
- [providers.models.claude_cli.default] with mcp_config pointing at the
§15 door, so a subscription agent can ACT and not merely reason
- disallowed_tools denies Claude Code's own Bash/Write/Edit/WebFetch so
the gated door is the ONLY actuator and nothing bypasses the audit log
- env CLAUDE_CODE_OAUTH_TOKEN = "$CLAUDE_CODE_OAUTH_TOKEN" — the $NAME
form reads the daemon env, keeping the token out of config.toml
- anthropic.judge swapped to the API key (0 oat01 left in config)
Verified before changing anything: the real binary returned SUBSCRIPTION-OK
through the token, then ENV-OK once the daemon carried it in env.
Note for future readers: /api/config/prop reflects what is CONFIGURED, not
what the binary supports — `openai` 404s there too. An earlier note that
the image "has no claude_cli in its schema" was true of the old :sync
image and is not true of the rebuilt one.
400 tests, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
`corpus_items.mission_id` has existed since the table landed and nothing
could populate it. `POST /api/library/runs` now accepts `missionId`, which
is the seam the wizard needs: a mission-driven run is the same run, tagged.
`corpus::contributed()` answers the question a continuous mission has to
be able to answer — did THIS run add anything new. Because `record` never
reassigns mission_id on conflict, the mission that first found a source
keeps the credit, so a rerun cannot inflate its own count by re-recording
what an earlier run already held. The test asserts exactly that: two
missions see the same paper, the finder reports 1 and the rerun reports 0.
This is the check the 0030-0044 generation of continuous research did not
have. It could run weekly forever and every run looked like success.
The test also earned its FK: the first version attributed to a bare UUID
and the database refused it. Attribution to a mission that does not exist
is not attribution, so the test now seeds real mission rows.
400 tests, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
Closes the branch pile-up: a catalogue branch that only adds notes now
merges into main by itself, so the work is actually in the vault rather
than waiting in a branch nobody opened.
Additive-only is measured from the diff, not assumed from the mission
type. Three conditions, all required: the type declares additive_only,
the run verified, and `git diff --name-status base...branch` contains
only A entries. A research harvest that somehow rewrote a hand-written
note is refused by the same check that lets its new notes through —
which is the case the test pins down, asserting README.md on main is
byte-identical afterwards.
Renames and deletes count as non-additive. A rename is a delete plus an
add and the delete half can destroy hand-written work.
Unknown merge_policy values fail closed to Never. A typo must not grant
auto-merge.
The diff is taken against FETCH_HEAD, freshly fetched, using `...` so an
unrelated commit landing on main meanwhile is not misread as ours. A
conflicted merge aborts and leaves the branch for a human rather than
wedging the checkout for the next run.
merge_reason is always populated and surfaced in the API: a branch that
quietly did not merge is indistinguishable from one never delivered.
399 tests, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
Monday 07:00, Persistent=true so a week missed to downtime fires on next
boot rather than leaving a silently empty library. 30-minute timeout so a
wedged run cannot hold the slot until the following week.
The script is deliberately thin — it calls the API and reports — so it
never needs changing when the harvest does. Auth is a long-lived operator
session in /etc/clawmates/library.token (root-only, 600); rotate by
replacing the file.
Exit status follows `healthy`, not paper count. A mature library shelves
nothing most weeks and that is success; a run that errored is a failure
even if it shelved something.
The first manual fire caught a real bug in this script, in the opposite
direction to this week's usual: the harvest genuinely shelved 15 papers
and pushed them, and the reporter crashed on an escaped quote inside an
f-string, so systemd marked the unit FAILED. A false failure destroys
trust in the signal exactly as a false success does. The reporter now
avoids backslashes entirely (it is embedded in a single-quoted shell
string) and was proved against the real response shape before being
trusted.
Verified end to end on gw-04:
run 1: 25 candidates, 10 already held, 15 shelved, pushed
run 2: 25 candidates, 25 already held, 0 shelved, no branch, healthy
Co-Authored-By: Claude Opus 5 <[email protected]>
Omar Sobh