A REGRESSION I INTRODUCED ONE COMMIT AGO. `capture_finished_coding_phases`
selects on `mp.status = 'completed'`, so the moment an unmet phase correctly began
reporting `failed`, its diff stopped being captured, committed or pushed — the work
was silently discarded. Found by the new harness scenario, whose phase legitimately
missed its condition and then had no artifact at all.
What was produced, and whether the goal was met, are different facts. The artifact
records the first; `mp.status` records the second. Capture now covers terminal
phases (`completed`, `failed`), so a phase that did real work and missed its goal
still delivers a reviewable diff — which is exactly what the next pass needs.
`scripts/verify-mission-delivery.sh microvm` — the regression net this session was
missing. Everything the microVM track proved by hand was guarded by nothing:
- THE KERNEL LINE is the assertion that cannot pass by accident. Every other
check would also pass if the phase had quietly run in a container on the
gateway; only the kernel says WHERE it ran. Compared against the real gateway
and node kernels read at start-up rather than pinned to a version, so
upgrading vmlinux does not manufacture a failure.
- subagent count > 0, from the server's own count of Claude Code's per-subagent
transcripts. Before `Agent` was in the allowlist this was structurally
impossible and nothing said so. A probe that could not run reports "?" and
FAILS the check rather than reading as zero.
- the verdict's judge and whether it was independent.
- negative control, observed passing: a mission whose backend no node can run is
refused at launch and stays draft. Without it the positive scenario would pass
just as well against a scheduler that ignored `backend` entirely — which is
what it did until the first real microvm mission landed on a node with no such
rootfs.
Also fixed in the harness: `api` now sends the JSON body on STDIN (`curl -d @-`)
instead of interpolating it into a single-quoted argument inside a double-quoted
ssh command. A task description containing "the crate's test suite" ended the
quoting and killed the remote shell; two attempts to escape it were themselves
wrong, because the backslashes must survive bash AND sed AND sh. Removing the
interpolation removes the class, and the next author does not need to know that
apostrophes were forbidden.
475 tests pass, clippy clean.
Found by the Goodhart test for the independent judge, which is exactly what it was
built to find.
The test: a phase whose `done_when` demanded a passing suite, and a task that
deliberately left a failing test. glm-4.7 judged it, ran `cargo test` itself, saw
`parity_is_wrong_on_purpose ... FAILED` (exit 101), and returned met=false quoting
the assertion — while the agent's own summary said "All three steps are implemented
exactly as specified and independently verified". The verdict and the agent's
account diverged, which is the whole point of an independent judge.
And then the mission closed `completed`.
`if verdict.met || last_pass` marked BOTH outcomes completed, so a phase that ran
out of passes without ever meeting its condition reported success — and through
`close_finished_missions`, so did the mission. The verdict said met=false in a
column nobody reads before believing a green status. Anything consuming mission
status rather than digging into the verdict saw a goal that was never reached as a
goal achieved. Exhausted-and-unmet is now `failed`, and the log names the judge and
whether it was independent.
This changes observable behaviour: missions that would previously have finished
green with an unmet condition now finish failed. That is the correction, not a
regression — but it is worth knowing before the next scheduled run.
Also: `Verdict.independent` had no column. The field existed in the struct and in
the logs, so the audit question the mechanism exists to answer — was this checked
by something other than the model that wrote it? — could not be asked of the
database. Migration 0067 adds it, defaulting to false, which is the truth about
every row written before now.
Verified in production before the fix: glm-4.7, 4 checks all executed, the real
cargo failure quoted, met=false. 475 tests pass, clippy clean.
Note for whoever rebases: `sqlx::migrate!` embeds migrations at COMPILE time, so a
new migration needs cm-db rebuilt (`touch crates/cm-db/src/lib.rs`) or the
integration tests fail on a column that exists in the file and not in the binary.
Claude writes the code and Claude judges it. That is a correlated failure: the
model that talked itself into a shortcut is the one disposed to accept it, and it
is the structural cause of the "early victory" failure Anthropic documents and of
our own Goodhart incident.
`glm` and `kimi` are both already registered in production, so the fix needed no
new credential path.
THE UNLOCK: `judge_with_tools` took `&AnthropicProvider`, but `LlmProvider` is a
single method — `stream(ChatRequest)` — and the loop only ever used that. The
concrete type was incidental. Widening it to `&dyn LlmProvider` means a
cross-provider judge runs the SAME allow-listed command loop. Before, independence
and real verification were mutually exclusive: the tool loop existed only on the
subscription path and every other route "judged claims only", so choosing an
independent judge meant giving up the checks that make a verdict evidence. GLM is
registered in anthropic format, so tool calling reaches it unchanged.
`CLAWMATES_VALIDATOR_MODEL` (e.g. `glm:glm-4.7`) selects it. Three refusals, each
protecting the claim the field makes:
- a spec in the implementer's own family is rejected, not used — `opus` judging
`sonnet` is not independence, they share a lineage and most failure modes
- a spec naming a provider this deployment never registered is rejected.
`Runtime::resolve_provider` silently falls back to the DEFAULT provider when
the registry has no such name, which would hand back Claude while the caller
believed it had GLM. Detectable because the returned model keeps its `name:`
prefix, so it is checked rather than trusted.
- an independent judge that FAILS does not fall through to the house judge. A
verdict quietly produced by a same-family model would claim a property it does
not have. The pass stays unmet, says why, and the next sweep retries.
`Verdict.independent` records it, `#[serde(default)]` so verdicts stored before
this field read back as not independent — which is what they were. An unrecognised
model family resolves to "unknown", never to ours: guessing would report
independence nobody established.
474 tests pass, clippy clean. Not yet enabled in production — the env var is unset,
so behaviour is identical until it is set deliberately.
Reviewed the changelog rather than bumping on principle. 2.1.220 → 2.1.223 for one
reason that bears on how we use subagents:
2.1.222 — "Fixed PreToolUse auto-allow hooks bypassing tool restrictions in
background agent tasks."
Subagents run in the background by default since 2.1.198, and the `verifier`
role's entire guarantee is a TOOL restriction — no Edit, no Write. So on 2.1.220
the one property we rely on was the one that bug could undo. 2.1.221 also fixes
`--mcp-config` servers not connecting before the first turn in print mode, which
is the mode we run and will matter when the MCP door reaches a VM.
Two findings from the changelog that we already had at 2.1.220, both worth knowing:
- 2.1.219: subagents can nest to depth 3 (was 1), so our roles can delegate
further than assumed.
- 2.1.212: a subagent inherits the parent's permission mode, which confirms the
verifier's read-only property must come from `tools` and not from permissions.
That is how it was written; now the reasoning is recorded next to it.
And a correctness fix that follows from the background default: the verifier is now
`background: false`. A background verifier lets the lead carry on and write its
report before the check has finished — the finding would arrive after the
conclusion it was supposed to inform.
Verified on tank: image reports 2.1.223, rootfs rebuilt, `--vm-selftest` all green
including a real agent turn on subscription auth, egress allow and deny both firing.
`microvm_executor` passed `--allowedTools Read Edit Write Bash`, which omits the
`Agent` tool, so Claude Code could not spawn a single subagent in any of our VMs.
The tool existed, the model knew how to use it, and the allowlist quietly removed
the ability. Nothing in any output said so.
Now: `Agent` in the allowlist, two roles supplied as `--agents` JSON, and a probe
that counts what actually ran.
Roles are JSON on the command line, not files, because `/mission/repo` is
collected and diffed — a role definition written into the checkout would arrive in
the delivered patch as if the agent had authored it.
Two roles only, and the choice is the research talking:
- `verifier` — the one multi-agent pattern Anthropic endorses for coding work.
It gets Read/Grep/Glob/Bash and deliberately NOT Edit or Write: an agent that
can fix what it is checking will fix it and report success, and the report is
then about a tree nobody reviewed. Its prompt demands the COMPLETE suite,
which is the counter to the "early victory problem" — the same failure as our
own Goodhart incident.
- `explorer` — context protection, read-only.
Roles like "tester" or "committer" are absent on purpose: splitting sequential
phases of the same work is a named anti-pattern, and it is the shape our pipeline
templates already have.
THREE THINGS THE IMAGE CORRECTED, none of which review would have caught:
1. `CLAUDE_AGENT_SDK_DISABLE_BUILTIN_AGENTS=1` (in the plan) removes EVERY agent
type, including the ones `--agents` defines. Measured: the lead reported "an
empty available-agents list" after trying four role names and — to its credit —
refused to fabricate a subagent result. Worse, the unit test asserting
"builtins off is paired with our own roles" PASSED throughout, because the
pairing holds in our code and not in the CLI. Dropped, and the test rewritten
to assert only what a unit test can speak to.
2. `--forward-subagent-text` refuses to run without `--output-format=stream-json`,
which would change how this module reads output. Dropped.
3. `--append-subagent-system-prompt` does not exist in 2.1.220 despite being
documented. The anti-shortcut rule is inlined per role instead — better anyway,
since a verifier and an explorer need different wording.
Evidence instead of assumption: Claude Code writes a per-subagent transcript at
`<session>/subagents/agent-*.jsonl`, so the guest is asked to count them before
collection (they live in /root, outside the collected tree). `VmOutcome.subagents`
is `Option<u32>` and the phase log prints it: `None`/"?" means the probe could not
run, which is a different fact from "delegated to nobody" and only one of those is
about the agent.
Verified in a container against the real CLI on tank before any of this shipped:
`FANOUT-OK`, a subagent transcript on disk, and zero errored Agent calls.
469 tests pass, clippy clean.
Observed on the negative control: a mission with backend='kimi' was correctly
refused, but the message read "no online node reports microvm capability" — and
both nodes do report it. What one lacked was the image. That first clause would
have sent an operator to reinstall firecracker on a node that already had it.
The refusal now names the backend, and the remedy still names both halves.
The first real microVM mission was placed on morpheus because it reports
{"microvm": true}, while only tank had rootfs-claude.ext4. It failed by name
rather than booting the wrong image — but whether a mission ran came down to
which capable node was listed first, which is a coin flip dressed as scheduling.
`missions.backend` was invisible to the scheduler.
The node now enumerates the images on its disk and reports them as a `rootfs`
ARRAY. `microvm::available_backends` lives beside `rootfs_for`, its inverse,
because the two must agree on what a backend name means; split apart, one drifts
and the scheduler starts promising images the booter cannot find. It only
advertises names `rootfs_for` would accept, and reports an empty array rather than
omitting the key — set_capabilities REPLACES, so a deleted image stops being
advertised instead of leaving a stale claim.
`nodes::online_for_backend` requires microvm AND that the node's list contains the
mission's backend. A node on an older daemon has no `rootfs` key and matches
nothing: unknown is not permission, the same treatment every other capability
gets. `backend_key` maps the three spellings of "the default image" to the one
name the node advertises, and is tested — a mismatch there would reject every node
for an ordinary mission with no backend set.
The launch error now names both halves of the fix, since "no capable node" was
true but unhelpful when the node was capable and merely lacked the image.
Mission gains `backend` on the domain struct; it was a column the executor read
from the phase query while the struct that placement uses could not see it.
464 tests pass, clippy clean.
Self-inflicted, one commit old, and found by running a real mission: the early
return I added for "a microvm mission materialises no team" sat ABOVE the microVM
placement block in the same function, so on_launch returned before ever choosing
a node. The mission then failed with the executor's own guard — "mission has no
target_node_id ... a microvm mission cannot run on the gateway, which has no
/dev/kvm" — which is the guard working exactly as designed, on a cause one layer
further up.
Placement now runs first. Worth noting the shape: adding an early return to a
long function silently skipped everything below it that the same runtime_kind
depends on.
461 tests pass, clippy clean.
on_launch demanded a team template too — "pick teams in the wizard" — so a
microvm mission still could not launch after the first two gates were exempted.
Three separate places required a claw graph for a path that runs one `claude -p`
inside a VM: routes::missions (draft→running), phase_runner::launch_phase (no
matching teams → stay pending), and here.
Returning before team materialisation rather than filtering its picks: claws that
never run are not a cheaper version of the same thing, they are a runtime binding
and a pairing code describing something nothing speaks to.
Also stops provisioning the per-mission ZeroClaw container for a microvm mission.
The first real run was observed starting one and leaving it holding a pairing code
and ~3 GB of image for the life of a mission that never contacts it.
461 tests pass, clippy clean.
Found by running one: the mission was created with runtime_kind='microvm' and
then refused to launch with a bare 400, because draft→running requires a
materializable team. Past that, `launch_phase` returns early when a phase has no
matching teams — so even with the launch allowed, the phase would have sat
`pending` forever while the log said only "no matching teams", and the executor
would never have been reached.
Neither check applies to this path: microvm_executor runs the agent CLI directly
in the VM, so there is no claw graph to materialise. Satisfying the checks by
attaching a team template would have provisioned claws that never run.
The repo checkout still happens — the VM needs the repository.
461 tests pass, clippy clean.
`runtime_kind='microvm'` passes the DB CHECK, is honoured by placement, and now
has an executor — but `POST /api/missions` rejected the value with 400, so the
only interface that creates missions could not produce one. And `backend`, which
selects the per-CLI rootfs, was not in the create payload at all: it existed as a
column and as a parameter to `vm_create`, with nothing able to set it.
microvm needs no target_node_id at create time, unlike local_herdr: placement
resolves a KVM-capable node at launch and fails the launch when there is none, so
an explicit target is a request rather than a requirement.
461 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
`runtime_kind='microvm'` placed a mission on a KVM-capable node and then nothing
executed it: config accepted without a reader, one of the four seams this project
keeps closing. This is the reader.
`microvm_executor` — inject → run → collect → destroy, the shape copy mode
already proved for containers with a VM boundary instead of a namespace one. The
checkout goes in as a tar, the work comes back as a tar over the SAME host path,
so `mission_delivery::capture_phase_diff_at` needs no change at all.
The agent is told NOT to push, unlike the container path's session prompt. Two
reasons: delivery is already host-side and diffs the collected tree against the
recorded clone point (covering committed, staged and unstaged work in one pass),
so pushing would add a second untested way for work to arrive; and pushing would
mean forge credentials inside the VM, when the point of collecting is that the
guest never holds them.
Exactly ONE topology_runs row (tier='microvm'), mirroring launch_direct_session:
close_finished_phases, evaluation, capture and delivery all key off those rows,
and a second completion path would be a second way for a phase to finish with one
of them untested. The row and the phase flip happen BEFORE any fallible VM work,
so a missing token or a node that lost its capability shows up as a failed run an
operator can see — not a phase that stays pending and retries every ten seconds.
Fail-closed points, each the reader for a guarantee built earlier:
- credentials resolve BEFORE the VM boots, so a missing subscription token
fails the phase instead of booting a VM whose agent sits unauthenticated
- a VM reporting egress:false is REFUSED, which is what makes create's
egress/egress_host/egress_guest fields more than decoration — a turn without
egress does not fail, it hangs
- the injected checkout is PROVEN present in the guest before an agent turn is
spent on it; an inject that reports success while landing nothing would
otherwise become an agent reporting an empty repository
- work is collected even when the agent exits non-zero — a turn that failed
partway still wrote files, and a retry needs to see them
- a turn that ran but could not be collected is a FAILED phase, not a happy one
- destroy runs on every exit path, or an 8 GB sparse rootfs leaks
Two integration gaps found while wiring, both of which would have produced a
mission that completed having delivered nothing:
- `capture_finished_coding_phases` pulls work out of a CONTAINER before
capturing. A microvm mission has none, so the docker connect would fail, the
loop would `continue`, and capture would be skipped forever while the phase
sat marked completed. Its work is already collected by the executor.
- `launch_phase` provisioned a runtime container, copied the checkout into it
and wrote a runtime binding + pairing code describing a runtime nothing uses;
and the orchestrator's workspace pin — deliberately FATAL — would have failed
a microVM launch on a container it was never going to use.
461 tests pass, clippy clean.
NOT YET PROVEN END TO END: no mission has run through this path. The pieces under
it are each verified on tank (image, credentials, egress, a real agent turn), but
this executor has only been compiled and unit-tested. Deploy + one real microvm
mission is the remaining step.
Co-Authored-By: Claude Opus 5 <[email protected]>
The guest still has no network interface, and now that is the design rather than
a gap. Its only route out is an HTTP CONNECT proxy: agent CLI -> 127.0.0.1:3128
in the guest -> vsock 9002 -> a per-VM Unix socket on the host -> TLS to an
allow-listed host.
Why not TAP + iptables, which is what the Firecracker write-ups do — measured,
not argued:
- `ip tuntap add` is DENIED to the daemon user (needs CAP_NET_ADMIN), so TAP
would need root to pre-provision devices, the same privilege detour the
loop-mounted rootfs already forced.
- tank's FORWARD policy is DROP with Docker and Tailscale chains, so rules
would have to be inserted at position 1; appended ones die silently.
- a leaked TAP is a new class of host litter to reap.
CONNECT needs no privilege at all and is better on the merits: the client hands
us the HOSTNAME, so resolution happens host-side and the guest needs no DNS or
resolv.conf; the allow-list is by name, not address; and nothing in the guest can
reach the network except through one function. The guest end parses nothing and
enforces nothing, so a compromised agent cannot argue with the policy.
Rests on one measured fact: `claude` honours HTTPS_PROXY. With the proxy at a
closed port, `claude -p` fails ConnectionRefused instead of answering.
THE RESULT: a real agent turn now completes inside a VM with no network card, on
subscription auth — `claude -p` replies VM-OK. The selftest asks for it whenever
CLAUDE_CODE_OAUTH_TOKEN is present and SKIPS loudly when it is not, since it
spends a little of the plan.
The audit log earns its keep immediately: during that turn the proxy logged
`egress DENIED http-intake.logs.us5.datadoghq.com` — the CLI's telemetry, which
the mission container permits today without anyone deciding to.
Three bugs found by the checks rather than by review:
- `env_pairs` returned early when a caller sent no env, so the proxy address
was never added and `curl` in a VM with a working tunnel reported "Could not
resolve host". Absent env means "the caller sent none", not "this command
needs no environment".
- the deny check PASSED for the wrong reason — DNS was failing, so nothing was
refused by the allow-list at all. It now requires a 403 from the proxy, so it
cannot go green on a broken tunnel.
- `host_allowed` accepted `evil.test/api.anthropic.com`, which ends with an
allowed suffix. Hostnames are now validated against a character class, which
also refuses IP literals so an address cannot sidestep a list of names.
- `BufReader::into_inner()` discards buffered bytes: wrapping the stream twice
would have dropped the start of the TLS handshake and stalled a tunnel that
looked established. One reader now spans the request, and anything buffered
past the headers is forwarded as payload.
`iproute2` is in agent-toolchain because it is load-bearing: the guest's `lo`
starts DOWN, and while it is down a listener on loopback BINDS and then refuses
every connection with ENETUNREACH. fcagent finds `ip` by absolute path — as pid 1
its PATH comes from the kernel, and execvp's fallback excludes /usr/sbin, where
Debian puts it.
Egress needs both ends up, so `create` reports `egress` and the guest's `ping`
reports its own half. A VM without it is legal but never silent.
Verified on tank: 16/16 with backend=claude (create 1428 ms), 12/12 on the
default rootfs, no leaked processes, VM dirs or proxy sockets. 457 tests pass,
clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
B4.4 had the microVM path share `forwarded_provider_env(auth)` with the
container path, on the reasoning that the two must not diverge. That was wrong
in the one direction that costs money: gw-04 has CLAWMATES_RUNTIME_AUTH unset,
so the container path forwards ANTHROPIC_API_KEY today — and a VM would have
received it. Claude Code ranks the API key ABOVE the subscription's OAuth token,
so the VM would have worked perfectly while billing per-token against a plan we
already pay for. No error, no symptom but the invoice.
`microvm_provider_env` is subscription-only BY CONSTRUCTION: it does not take
the auth mode as an argument and does not read CLAWMATES_RUNTIME_AUTH at all.
Taking the mode as a parameter would mean one unset variable on a new host
silently turns the API key back on. The container path is unchanged and still
honours the operator's mode — the divergence is now deliberate, with the reason
at the definition.
Two other fail-closed rules fall out of it:
- A missing or blank subscription token REFUSES the launch rather than
returning an empty environment. A VM with no credential does not error;
`claude -p` hangs, which reads as a phase stuck at `running` with nothing in
the logs. The refusal names the variable.
- An unrecognised backend is refused rather than handed the Anthropic token.
GLM and Kimi reach their own endpoints via ANTHROPIC_BASE_URL and that
contract is not settled yet; guessing it would send a subscription
credential to z.ai.
Measured on tank, and this is the end-to-end proof B4.4 could not give:
`claude -p` in the agent-claude image with the real subscription token replies
"OK". Injecting the token in a VM moves the failure from "Not logged in" to a
network error, so the credential channel is accepted by the CLI — the VM's
remaining problem is egress (#49), not auth.
447 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
`claude -p` in the VM failed with "Not logged in". The credential now travels on
the exec op: `env` on `vm_exec` → fcagent → the command's environment. An env var
rather than a file because the per-VM rootfs dies with the VM but an env var
never touches the guest disk at all.
**Every problem in an env entry fails the exec.** The tempting alternative —
skip the entry we cannot use and run anyway — produces a `claude -p` with no
credential, and that does not error, it HANGS. A phase stuck at `running` for
ten minutes with nothing in the logs is exactly what a missing token looked like
on the container path. Names are validated ('=' or NUL would define a different
variable than the one asked for via putenv semantics), values must be strings,
and errors name the key and never the value — an error string travels back over
the wire and into logs.
One list of which credentials travel: `forwarded_provider_env` reuses
`forwarded_provider_keys`, and the container path now reads it too. If the two
execution paths diverged, a mission would behave differently depending on where
it landed — including the expensive way, where one path forwards
ANTHROPIC_API_KEY and bills it while the other uses the subscription. A blank
value is omitted rather than forwarded empty, so `claude` reports having no
credential instead of failing authentication with one.
Verified on tank (`--vm-selftest` backend=claude, 13/13, create 1532 ms): an
injected var reaches the guest command over the real vsock wire, and an
unusable entry comes back ok:false with no rc.
FINDING — the CLI leg remains UNPROVEN, and deliberately so. The guest has no
network interface: `create` writes boot-source, drives, machine-config and vsock
and no `network-interfaces` key, and a booted guest has no routes, no
resolv.conf, no DNS and no TCP. So `claude -p` cannot reach the API whatever
credential it holds. Injecting the real token would have proven nothing, because
the failure would have been network and not auth. Filed as B4.6 (task #49) with
the TAP-vs-vsock-proxy trade-off; B4.5 is now blocked on it.
444 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
The only rootfs on this track came from clawmates/agent-terminal:dev. Mounted,
it held git and nothing else: no claude, no node, no cargo. A VM booted from it
looks perfect and cannot run a mission, so B4.5 could have been written and
never verified.
images/agent-toolchain — the shared mission toolchain (node 22, git, rust +
cargo-audit, gitleaks/trivy/semgrep, tea/gitea-mcp), lifted from the proven
deploy/clawmates-runtime image minus the zeroclaw daemon: a microVM mission runs
the direct-session model, so there is no daemon to host. A base image rather
than three self-contained Dockerfiles because this layer is ~3 GB and the real
risk is scanner and toolchain versions drifting between per-CLI images — the
evaluator runs the project's own suite to check a claim, so `cargo` present in
one image and absent in another makes the same mission pass or fail by backend
with nothing saying why.
images/agent-claude — plan A6, first of three: the pinned CLI and its env
contract only, so bumping Claude Code does not rebuild the toolchain and cannot
disturb agent-kimi / agent-glm. HOME=/root with an empty .claude for B4.4 to
inject into; no ANTHROPIC_API_KEY, since it silently overrides the subscription
OAuth we already pay for.
Both the builder and the node selftest now ASK the guest for the CLI the image
is named for, instead of trusting the name. `required_cli` maps claude/kimi/glm
to a probe; an unrecognised backend reports unchecked and prints SKIP rather
than passing quietly.
Verified on tank:
- rootfs-claude.ext4 boots; git, node, cargo, a real git commit all work
- `claude --version` → 2.1.220 over vsock, in both the builder and
`--vm-selftest` (11/11, create 1498 ms)
- negative control: the same builder run against agent-terminal with
FC_CLI forced reports `cli rc=127 claude: not found` and exits 1, so the
green result above is a measurement and not a default
- `claude -p hello` fails with "Not logged in · Please run /login" — the CLI
runs headless in the VM, and B4.4 only has to supply the credential
- no leaked firecracker processes or vm dirs afterwards
437 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
`vm_create` takes a backend name and boots `rootfs-<backend>.ext4`; NULL
or "default" boots the golden image. Makes the per-CLI images from B4.1
actually reachable (one image per CLI, per A6).
A missing image is an ERROR naming the file and how to build it, never a
quiet fall back to the default. That fallback is the tempting version and
the wrong one: it would run a claude mission in a kimi VM, or in a rootfs
with no CLI at all, and report success for whatever came out. Verified on
real hardware, not just in a unit test — the selftest asks for an image
that does not exist and FAILS if it boots.
`create` now reports the rootfs that actually booted, not the one that was
requested, so a mission artifact can show the wrong VM ran.
The migration adds no CHECK constraint listing the CLIs. Which images
exist is a property of the NODES, not the schema; a constraint would need
migrating for every new image while still not guaranteeing the image
exists anywhere. The node validates and names what is missing. Backend
names are `[A-Za-z0-9_-]` and rejected rather than sanitised, since they
become filenames.
Verified on tank: default backend 8/8; `CLAWMATES_FC_BACKEND=agent-terminal`
9/9 including the absent-image check, create in 910ms on a rootfs built
from a real Docker image. 435 tests green, no leaked processes or VM dirs.
Co-Authored-By: Claude Opus 5 <[email protected]>
The python guest agent only ever worked because Firecracker's CI Ubuntu
image happens to ship python3. NONE of our images do — agent-base has
neither python nor git, agent-terminal has git but no python — so it
could never have run in a real mission rootfs. An agent that dictates
what must be installed in the image has the dependency backwards.
crates/bins/fcagent is a 905K static x86_64-unknown-linux-musl binary
that needs nothing from the rootfs it is dropped into. The wire is
unchanged on purpose — 4-byte BE length + JSON, ops ping/exec/put/get —
so microvm.rs and microvm_client.rs needed no edit at all.
std has no AF_VSOCK and the workspace denies `unsafe`, so it uses the
`vsock` crate. `process_group(0)` gives each command its own group without
unsafe, so a command that spawns background children can be killed
wholesale rather than outliving the run.
A unit test caught a bug that would have broken EVERY exec: sourcing the
image-env file with `. env.sh 2>/dev/null; cmd` returns rc=1 WITHOUT
running cmd, because `.` on a missing file makes a non-interactive POSIX
shell exit immediately. On any rootfs lacking that file every command
would have failed while looking like an ordinary non-zero exit. Guarded
with `if [ -f ]` now.
Other places a failure must not borrow an outcome's representation: a
killed command reports ok:false with no rc (not rc=124, which would read
as a build failure); `get` on a missing path is an error, not an empty
archive; a signalled process reports 128+signal rather than success.
Verified on tank: --vm-selftest still 8/8 with the agent swapped
(create 949ms, wire identical), fc-node-setup 8/8, and — the point of the
change — a rootfs built from clawmates/agent-terminal:dev, which has NO
python3, boots and reports `git version 2.39.5` from inside the VM.
Also fixes a shell bug in fc-build-rootfs.sh: $HOME in a double-quoted
default expanded on this Mac, so it looked for the node's binary under
/Users/quantum on a Linux host.
Co-Authored-By: Claude Opus 5 <[email protected]>
Until now microVMs booted Firecracker's CI Ubuntu image with a python
guest agent bolted on: no git, no toolchain, no CLI. Fine for proving
vsock, useless for running a mission.
Builds FROM a Docker image rather than debootstrapping, because the
per-CLI images (agent-claude / agent-kimi / agent-glm, per A6) are
already Dockerfiles with a tested env contract. Rebuilding that as a VM
image by hand would mean maintaining the same facts twice and finding the
drift in production.
Two things the obvious version gets wrong and this does not:
- `docker export` gives the filesystem with NONE of the image metadata:
no ENV, no ENTRYPOINT, no WORKDIR. A CLI relying on ENV PATH or HOME
would silently behave differently in the VM. The env is extracted
separately and written to /etc/profile.d.
- the ext4 is filled through a mount, not `mkfs -d`, which cannot
handle the device nodes and hard links a container image may contain
and fails late and cryptically when it hits one.
The guest agent is copied from the golden rootfs rather than re-emitted,
so there is ONE copy of the protocol on the node instead of two that can
drift.
It boots what it builds and asks the image for what a mission needs —
git, the profile env, a writable /mission — rather than assuming. An
image that builds and cannot boot is worse than no image, because it
looks finished.
FINDING, and it blocks B4.2: NONE of our images ship python3, so the
python guest agent cannot run in any of them. agent-terminal has git but
no python; agent-base has neither. The guest agent must not dictate the
image's contents — it needs to be a static binary. This script correctly
refuses to build an image whose agent cannot run, so the failure is
visible rather than a VM that boots into nothing.
Co-Authored-By: Claude Opus 5 <[email protected]>
cm_api::microvm_client::MicroVm wraps the node's vm_* ops as typed calls
over the existing hub request/response channel: create / inject / exec /
collect / destroy, plus list() for reaping. No new transport.
Fixes a wire-contract mismatch B2 would have shipped. `Uplink::Result`
declares `output: String`, but the node's vm_* handler returned a JSON
object. The frame then failed to deserialize and hit the uplink match's
`Err(_) => {}` arm, so the reply VANISHED and every vm_* call would have
timed out after 20s with nothing anywhere explaining why. The node now
sends a string, matching the contract rather than what looked tidier.
That silent arm is fixed too: an unparseable frame now logs the node, the
parse error and the frame head, and says explicitly that the request it
was answering will time out. It is the arm that would have hidden this.
Two more places where a failure must not borrow a legitimate outcome's
representation:
- vm_exec returning no `rc` is an error, not a zero. A missing exit code
means the guest did not report one; reading it as success is how a
failed command becomes a passing phase.
- vm_collect on a missing path is an error, not an empty archive — an
empty tar looks exactly like a run that produced nothing.
Timeouts: the hub's deadline is the guest's plus 30s, saturating. A
caller passing a huge budget would otherwise wrap to a tiny timeout and
turn a long agent turn into a spurious transport failure. clippy caught
the tautological assertion in the first version of that test, which is
what surfaced the overflow.
Verified: `--vm-selftest` on tank still 8/8 after the output-type change
(create 950ms), 427 tests green.
Co-Authored-By: Claude Opus 5 <[email protected]>
create / inject / exec / collect / destroy / list, riding the node's
existing frame dispatch ({t, id, …} -> {t:"result", id, ok, output}), so
no protocol change was needed. Control is length-prefixed JSON over
vsock; the serial console stays a log, because feeding a guest over stdin
races its startup and arrives half-consumed.
DEVIATION FROM THE PLAN, deliberately: this does NOT implement
cm_sandbox::SandboxDriver. That trait is container-shaped —
attach_pty/resize_pty/argv exec — while missions need
create -> inject -> run -> collect -> destroy. Conforming would mean
building PTY-over-vsock and window-resize semantics that no mission path
calls, purely to satisfy a signature. We give up automatic RemoteDriver
marshalling; orphan reaping is a label/id sweep either way.
Three traps from the B0 spike are handled in code rather than remembered:
- Firecracker does NOT unlink its vsock UDS on exit, so destroy unlinks
it explicitly, and the selftest ASSERTS it is gone. Assuming the VM
tidies up after itself is how the mission checkout accumulated four
uid bugs.
- firecracker is spawned via setsid and killed as a process GROUP, so a
background child cannot outlive the VM holding its workdir open.
- create does not return until the guest agent has answered a ping. A
VM that booted but serves nothing is worse than one that failed, so a
half-created VM is destroyed rather than left registered.
A vm id becomes a path component, so ids are restricted to [A-Za-z0-9_-]
and REJECTED rather than sanitised — a caller that sent `../../etc`
wanted something we should not guess at.
Verified on tank through the real Rust path, as the daemon user, with no
sudo: `clawmates-node --vm-selftest` -> 8/8, create in 986ms, and the
host left with zero firecracker processes and zero VM directories. The
selftest asserts every step, including that a destroyed VM can no longer
be exec'd; a test that only reports the steps it completed cannot
distinguish "passed" from "stopped early".
Co-Authored-By: Claude Opus 5 <[email protected]>
The capability probe reported `kvm: false` on tank and morpheus while the
device sat right there: /dev/kvm is `crw-rw---- root:kvm` and the kvm
group was EMPTY, so the daemon — an ordinary user — could not open it.
The B0 spike missed this entirely because it ran everything under sudo.
This is exactly why the probe opens the device rather than stat-ing it;
a stat-based check would have reported both nodes capable and every
microvm mission would have failed at launch instead of at placement.
fc-node-setup.sh now fixes the group itself, or says precisely what to
run when it cannot.
Co-Authored-By: Claude Opus 5 <[email protected]>
Phase B step 1, on top of the B0 spike that proved microVMs boot here.
KVM is a HARD predicate, not a preference. gw-04 — where every mission
runs today — is itself a VM without nested virtualisation and has no
/dev/kvm, so a microvm mission landing there cannot start at all. The
scheduler therefore has to be able to tell nodes apart, which means the
node has to report what it can host.
Nodes gain a `capabilities` jsonb, populated from a probe on the node
rather than from configuration: /dev/kvm either exists there or it does
not, and nothing on the server can make it appear. The probe OPENS the
device rather than stat-ing it, because it can exist while being
unopenable (wrong group, or a container without the device passed
through) — which is precisely how firecracker will fail.
`microvm` requires BOTH kvm and a firecracker binary. A node with KVM
but no binary looks capable by the obvious test and fails at launch; a
node with the binary but no KVM is gw-04.
Placement fails the launch when no capable node exists, rather than
letting a mission sit in 'running' with nowhere to run. An explicit
target_node_id is treated as a request, not a guarantee — it is honoured
only if that node actually reports the capability.
`capabilities` defaults to '{}' NOT NULL so a node that has never
reported fails every predicate: an unqueried node and an incapable node
must be indistinguishable to the scheduler, because scheduling onto a
node whose abilities are unknown is how you get a mission that cannot
start and does not say why. The report replaces rather than merges, so a
capability the node has LOST disappears instead of leaving a stale true.
Co-Authored-By: Claude Opus 5 <[email protected]>
Phase B step 0. Before writing any driver, establish that Firecracker
works on this hardware — the plan called it greenfield, and an
orchestrator built against an unproven runtime is a lot of code betting
on an assumption.
It works, and comfortably: a microVM boots, runs our init, writes a
file and shuts down in ~650-910ms wall clock, with the kernel reaching
our init at 234ms. Host->guest RPC over vsock (AF_VSOCK port 9001, no
network stack) round-trips in 27ms.
The script installs and then PROVES, because installing is not working.
It reports success only after a VM has actually booted and run our code.
Four findings from the spike that the driver must account for:
- Firecracker does NOT unlink its vsock UDS on exit, and leaves it
owned by whoever ran the VM. A driver running as anyone else cannot
clean it up — the same uid trap that cost this codebase four bugs on
the mission checkout. The driver owns the socket path lifecycle.
- tank's FORWARD policy is DROP (Tailscale/Docker), confirming the
article's warning: VM networking rules must be inserted at position
1, not appended, or return traffic dies silently.
- Feeding commands to the guest over the serial console races the
shell's startup and arrives half-consumed (`# ho FC-GUEST-ALIVE`).
The guest runs an init script; stdin is not a control channel.
- `sha256sum -c` compares by filename, so a download saved under any
other name fails for a reason unrelated to integrity. A check that
fails for the wrong reason teaches you to ignore it — compare the
hashes directly.
tank and morpheus are ready. architect requires interactive sudo, so it
is deliberately not provisioned rather than worked around.
Co-Authored-By: Claude Opus 5 <[email protected]>
Verified on the deployed stack: verify-mission-delivery.sh all → 9/9,
with the noop negative control showing 'phase 0 failed 0' where the same
shape read 'completed' in mission 019fcf62.
Co-Authored-By: Claude Opus 5 <[email protected]>
The last open item in the silent-success class: a coding phase that
changed no files reported `completed` — the same status a phase gets for
delivering tested, reviewed, pushed work. Mission `019fcf62` completed
that way with its agents silently unpinned from the repo, and nothing in
the platform disagreed; it was found by a script diffing the forge.
The verdict is applied at capture rather than at completion, because
capture selects on `status = 'completed'` — the platform does not know
whether a phase produced anything until after it has already finished.
Three conditions must hold before failing a phase, because a false
positive here fails honest work: the phase is a coding phase (research
phases legitimately write nothing to the tree), the diff was actually
computed (an uncomputable diff also reports zero files — blaming the
agent for a platform fault is the same defect wearing different
clothes), and `allow_empty` is not set. Only an explicit `true` opts
out, so a typo leaves the check armed. Registered in phase_config with
its reader named, per the seam-2 rule.
Also closes an ordering hazard this exposed: capture is batched and runs
after a phase completes, so a backlogged mission could close as
'completed' and only then have capture discover an empty phase — leaving
a 'completed' mission holding a 'failed' phase, unfixable because the
mission-close CASE only touches 'running' rows. A repo-bearing mission
now waits for its work to be captured before closing.
Adds a `noop` scenario to the harness: a phase told to change nothing,
which PASSES only when the phase comes back `failed`. Same discipline as
the uid self-test — a check that has never been seen to fire has not
been shown to work.
Co-Authored-By: Claude Opus 5 <[email protected]>
Verified with CLAWMATES_MISSION_FS removed from gw-04's .env — compose
passes it through as empty, which under the old opt-in logic would have
selected bind. scripts/verify-mission-delivery.sh all → 7/7.
Co-Authored-By: Claude Opus 5 <[email protected]>
Copy mode shipped opt-in so that changing how every mission receives its
code required someone to type it. Four production missions and a
fail-closed harness later, opt-in is the riskier setting: the bind path
is the one with four documented work-loss incidents, and leaving it as
the default means the untested path runs whenever nobody sets the
variable. `CLAWMATES_MISSION_FS=bind` still selects it; anything else —
unset, empty, misspelt — gets copy mode, so a typo lands on the safer
path rather than the one being retired.
Also fixes a real leak found while scoping the deletion below: the git
helper built its `safe.directory` argument with `Box::leak`, justified as
"the process is short-lived". That is true of a CLI and false of cm-api,
which is a long-running server — so it leaked one allocation per git
call, growing with every phase of every mission.
The A5 deletion is NOT done here, and two of its items should never be
done:
- `scrub_remote_credentials` is a security control, not a uid
workaround. Copy mode uploads the whole `.git` into a container the
agent controls as root, which makes stripping the token from
`.git/config` more necessary, not less.
- `has_local_work` / `checkout_in_use` guard `fetch_and_reset` at every
phase launch and have nothing to do with who writes the checkout.
The host checkout still persists across phases under copy mode —
mission `019fcf62` shows the marker firing there. Deleting them
reintroduces PRIOR-PHASE-WORK-WAS-LOST.
The rest (`share_repository_across_uids`, `clear_stale_commit_editmsg`,
`-c safe.directory`) are genuinely obsolete under copy mode but stay
while `bind` remains selectable: a workaround may only be deleted once
the situation it works around can no longer be chosen.
Co-Authored-By: Claude Opus 5 <[email protected]>
Verified against the deployed stack: scripts/verify-mission-delivery.sh all
→ 7/7, chain phase 0 now files=1 pushed=true (was 0 files, no error).
Co-Authored-By: Claude Opus 5 <[email protected]>
A verification run against the deployed stack found a chain mission whose
phase 0 reported `completed` with zero files, no commit error and no push
error — indistinguishable from a phase that correctly had nothing to do.
Three separate defects had to line up, each of them the same shape: a
failure sharing its representation with a legitimate negative result.
1. `pin_agent_workspaces` embedded the whole config in one `sh -c` argv.
That works until the file grows — config gains a block per provisioned
claw — then fails with `argument list too long`. Now written through
the tar upload API, which has no argv limit, so the failure mode is
gone rather than merely further away.
2. A failed pin was logged "(continuing)". Without the pin, agents write
to their sandboxes and the committer finds nothing in /mission/repo —
the mission cannot deliver, so the launch now fails where someone is
still looking. The restart that applies the pin is fatal for the same
reason.
3. `capture_phase_diff_at` swallowed `git diff` failures with
`unwrap_or_default`, so an unreadable base landed `empty: true,
files_changed: 0` — byte-identical to an honest no-op. The error is now
recorded as `diff_error`, and an empty patch that came from a failed
diff is no longer trusted to mean an unchanged tree.
Adds scripts/verify-mission-delivery.sh, which found #1 and #2 on its
first real run. Its probes are fail-closed: no placeholder values, a
self-test that proves the uid probe can detect the split it looks for,
and FAIL-NORUN for a scenario that never executed. Its own first version
had this bug too — a `die` inside `$(...)` exited the subshell, so a run
that could not authenticate printed "all checks passed" and exited 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
With the flag set, ensure_container omits the /mission bind, the checkout
is pushed into the container at phase launch, and the agent's work is
pulled back before capture.
The simplification that makes this small: sync_out unpacks over the SAME
host path the checkout came from. The host directory stays a server-owned
staging area with exactly one writer, and capture_phase_diff_at needs no
change at all — it still finds a normal checkout exactly where it always
has. Delivery, gating, commit and push are untouched.
Two failures are deliberately loud rather than silent:
- copy-IN failure fails the phase launch. Continuing would start a phase
against an empty directory, and the agent would cheerfully report having
done work on a repo that was not there.
- copy-OUT failure SKIPS capture. Capturing anyway would diff a stale host
tree and record "no changes" for work that exists — success reported for
nothing, which is the exact failure mode this codebase keeps paying for.
Opt-in: the bind path is what production has run since the beginning, and
the test asserts a near-miss value leaves it there rather than silently
switching every mission.
414 tests, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
The first half of removing the shared bind mount. Not wired yet — this
adds the mechanism and its tests.
One cause, four fixes so far: .git/objects permission denied
(core.sharedRepository), the capture base being overwritten each phase,
COMMIT_EDITMSG root-owned, and reset --hard deleting a prior phase's work
(.git/clawmates-in-use). core.sharedRepository was never a general
solution — it covers objects and refs, and every OTHER file git touches
is a fresh opportunity. Copy-in/copy-out removes the cause instead: the
agent owns its filesystem with no second writer.
Measured before building, because the plan named copy cost as the open
risk: a real 65 MB checkout of this repo copies in 0.23s and out 0.18s on
gw-04. Not a risk at this size; re-measure an order of magnitude larger.
No compression — the payload crosses a local socket, so gzip would spend
CPU to save nothing.
Two safety properties, both tested:
- The archive comes back from a container the agent controls as ROOT, so
it is untrusted input. A `../ESCAPED` entry must not write outside the
destination. The test writes the tar header bytes by hand because the
tar crate refuses to BUILD such an entry through its safe API — which
is reassuring, but means the hostile case has to be constructed the way
an attacker would.
- Symlinks are packed as links, never dereferenced. Following them on
copy-IN would smuggle host files into the container; the test plants a
host secret behind a symlink and asserts its contents never appear in
the archive.
Ownership is deliberately not preserved on unpack: the archive's uids are
the container's root, and re-applying them on the host would recreate the
exact uid split this exists to remove.
413 tests, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
Checking before deploying caught a mistake in the previous commit. The
seed dir on gw-04 is 1.7 GB and the first version copied all of it per
mission — tens of seconds each, and ~17 GB across ten concurrent
missions.
1.5 GB of that is .rustup: a Rust toolchain that installed itself into
the data dir back when HOME=/zeroclaw-data and the image had no
toolchain. The image now ships Rust at /usr/local/cargo, which is what
the container's PATH actually resolves — verified live. The data-dir copy
is dead weight and is not even reachable.
SEEDED_PATHS now copies only what carries per-mission identity or
secrets: .zeroclaw (config.toml with the door token, sessions.db,
devices.db), clawmates-mcp.json, .claude + .claude.json, .kimi-code,
glm-home, agents. Roughly 46 MB instead of 1.7 GB — about 37x smaller.
Caches and toolchains are deliberately excluded: .rustup, .npm, .cargo,
.cache, .local. They hold no secrets and a mission reads the image's.
Absent paths are tolerated: a fresh deployment has no .kimi-code until
Kimi is first used, and that must not fail container creation.
The test asserts both directions — the token-bearing paths ARE copied
and the caches are NOT — because either mistake is silent: copying
everything just makes missions slow, and copying nothing quietly
restores the credential sharing.
409 tests, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
Every per-mission container bind-mounted the SAME host seed dir as
/zeroclaw-data — shared with each other AND with the singleton runtime.
That directory holds config.toml, which carries the §15 door bearer
token, plus sessions.db and devices.db.
So one mission could read another mission's credential, and anything it
wrote there was inherited by every later mission. teardown_container
only removes /var/lib/clawmates-missions/{id}, so the shared directory
was never cleaned — the contamination was permanent.
The code already knew. The comment on DEFAULT_SEED_DIR names the sqlite
race and calls copy-on-write per mission the long-term fix. This is that
fix: seed_runtime_data copies the seed into
<missions_root>/<mission>/runtime-data at container create, and the
mount points there. Cleanup is free — teardown already removes that tree.
The copy runs in a throwaway container because cm-api cannot see the seed
dir: it hands that host path to Docker but never mounts it itself. The
runtime image is reused so nothing extra is pulled, and `cp -a /seed/.`
copies dotfiles — `/seed/*` would silently skip .zeroclaw/ and produce a
runtime with no config at all.
A copy failure is fatal to container creation on purpose. Falling back to
the shared mount would silently restore the credential sharing this
removes, and silent fallback to a weaker posture is the failure mode this
codebase keeps paying for.
The test asserts path shape rather than behaviour: an edit that points
the mount back at the seed dir restores credential sharing with no other
visible symptom, so the path IS the invariant.
408 tests, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
All three providers run through the SAME `claude` binary, verified live:
Anthropic CLAUDE_CODE_OAUTH_TOKEN -> ANTHROPIC-OK
GLM ANTHROPIC_BASE_URL=https://api.z.ai/api/anthropic -> GLM-OK
Kimi ANTHROPIC_BASE_URL=https://api.kimi.com/coding/ -> KIMI-OK
That is a stronger multi-provider story than a provider-per-implementation:
skills, subagents, MCP, hooks and tool policy are identical across all
three because it is literally the same harness.
The `kimi` CLI (0.31.1, shipped in the image) 401s on this key and is not
needed -- the claude binary reaches Kimi's Anthropic-compatible endpoint
directly. Worth knowing before someone debugs the CLI.
forwarded_provider_keys now ships ZAI_API_KEY and KIMI_API_KEY into
mission containers in BOTH auth modes: they are unrelated to the Anthropic
credential, so the api_key/subscription split does not apply to them. A
mission that selects a backend without its key present would otherwise
fail at the first turn.
Keys persisted in /opt/clawmates/.env and passed through compose.
Co-Authored-By: Claude Opus 5 <[email protected]>
Mission 019fcd0c produced correct work — a reviewed, tested function plus
a REVIEW.md quoting a real cargo test summary — and delivered none of it:
git commit → exit 128: could not open '.git/COMMIT_EDITMSG': Permission denied
The agent ran `git commit` itself inside the mission container (as root),
leaving that file owned by root at 0644. core.sharedRepository covers
objects and refs — .git/index lands at 0666, which is why commits work at
all — but not COMMIT_EDITMSG, which git writes with the default umask.
Unlinking works where overwriting does not: removing a file needs write
permission on the DIRECTORY, and .git/ is owned by the server. Silent on
failure by design, so the commit reports the real error rather than this
speculative cleanup.
Third distinct instance of the same uid-split class (objects, then the
capture base, now this). The pattern holds: the checkout is one directory
written by two users, and each new file git touches is a new opportunity.
Co-Authored-By: Claude Opus 5 <[email protected]>
Omar Sobh