Two halves, and the first was worse than the plan assumed. `run_phase_in_vm`
packed `<missions_root>/<mission>/repo` unconditionally — a directory a
repo-less mission does not have — and then required `/mission/repo/.git` inside
the guest before spending a turn. So a repo-less microVM phase did not merely
go uncaptured: it failed before the agent ran.
A repo-less mission now gets an EMPTY workspace at the same guest path, created
host-side so the collect unpacks back over it with no special case, and the
readiness probe asks for what was actually sent — the directory rather than a
`.git` that was never going to be there.
`mission_outputs` then drops its `runtime_kind <> 'microvm'` exclusion, whose
stated reason ("a microVM mission always has a checkout") is exactly what
stopped being true. Where the files come from now depends on the runtime, and
the difference is not cosmetic: a container mission's output is still inside a
running container, while a VM's has already been unpacked onto the host by the
end-of-turn collect. Asking docker for a VM mission's files would query a
container that never existed.
The recursive copy skips symlinks rather than following them — a link out of
the tree would publish whatever it points at.
`research-vm` is the proof, added to the suite as well as the dispatch: the same
assertions as `research-only` with `runtime_kind: microvm`. A scenario nobody
runs is a scenario that does not exist.
Co-Authored-By: Claude Opus 5 <[email protected]>
Added to the case dispatch when it was written, and not to `all` — so the
newest backend, and the only one that runs on hardware we own, was excluded from
the one run that claims to check everything. A scenario nobody runs is a
scenario that does not exist.
Co-Authored-By: Claude Opus 5 <[email protected]>
Forcing the planner onto the local link produced a green chain and a red
assertion:
roster: the planner sized this mission at 1 member(s) PASS
roster: ROSTER.md has 3 line(s) for a 1-member roster FAIL
The model was right and the check was wrong. ROSTER.md does not start empty —
the auto-merge work put an earlier run's two lines onto main — so a 1-member
roster that correctly appended one line delivered three, and the scenario
reported a model that had ignored its own proposal.
It now measures the DELTA against main. Any assertion against a scratch repo
that accumulates has to, or it decays into a test of how many times it has been
run before.
Proven on the local model end to end: opus 429 -> local:ornith-fleet:9b
answered -> `mission_roster: ... local:ornith-fleet:9b proposed 1 member(s)` ->
the composed graph ran -> the branch added exactly one line. 5/5.
CLAWMATES_MODEL_FALLBACK is removed from gw-04's .env again; it was set only to
force the last link for this test, and the deployed default is the full chain.
Co-Authored-By: Claude Opus 5 <[email protected]>
`local-ornith` scenario, green on its first real run against tank:
local-ornith: a locally-served model delivered a guest kernel (6.1.128)
local-ornith: no Anthropic egress from a locally-served mission
local-ornith: the node bound its local-model socket for this VM
local-ornith: checkout has exactly one writer (uid=65532)
Three things had to be true at once and only a real run shows all three: the
agent reached a model at all (a pipe to a closed port produces a turn that HANGS
rather than errors, which is why this is a scenario and not a unit test), the
work came back and landed on a branch, and the VM still could not reach
api.anthropic.com.
That last one is not theoretical. The node log for this VM is a column of
`egress DENIED api.anthropic.com` — Claude Code's own telemetry, correctly
refused — while the model traffic went through the vsock pipe and Ollama logged
loading ornith-fleet:9b at 100% GPU with CONTEXT 131072. A local backend that
quietly kept Anthropic egress would be a credential path nobody asked for.
The egress check asks the NODE's proxy log rather than the agent, for the same
reason the GLM measurement did: a model's account of where its tokens came from
has no evidential value, and the proxy's record of what it dialled does.
Co-Authored-By: Claude Opus 5 <[email protected]>
`drain-midmission` found this. `choose` treated its `want` argument as a hard
requirement, and the only caller passes `missions.target_node_id` — which is
not an operator's choice, only where the PREVIOUS phase happened to run. Two
consequences, both wrong:
- A node drained or filled between phases produced `TargetUnfit`, which
`is_transient()` says false to, so `phase_runner` FAILED the phase rather
than queueing or moving it. The queue silently did not apply to the second
phase of any mission.
- While the node stayed fit, every later phase went straight back to it
regardless of ranking — accidental mission-to-node affinity, which this
module's own header says must not exist.
Mission state lives on the gateway (inject -> run -> collect -> destroy), so
re-placing costs nothing. The pin is now advisory: preferred while it fits,
and when it does not, the reason is logged and ranking proceeds. `TargetUnfit`
is deleted rather than left unconstructed, so it cannot come back as a
non-transient failure by accident.
The scenario had its own race: it waited for phase 0 to COMPLETE before
draining, but warm phases finish in ~80s against a 10s placement sweep, so
phase 1 was often already placed — and the run then blamed the platform for
running on a node that was not yet drained. It now drains while phase 0 is
still running, which does not disturb a live VM and is the more faithful test.
Co-Authored-By: Claude Opus 5 <[email protected]>
The first bursts took 25 minutes because every VM paid a cold 2.4 GB rootfs
copy. Warm, the same 16 missions finish in 70-140s each and the whole burst is
over in about two minutes — so a sampler that waited ~90s for its launch check
and then ticked every 15s caught three samples of the tail and reported
"architect peaked at 1 of 6" for a run that sat at 6/6/2.
Sampling now starts at the first tick, runs every 5s, and folds the launch
check into the same query so verifying the launches costs no observation
window. The 10-sample floor that produced the last NORUN is gone; it was
measuring how long the burst took, not how well it was watched.
And "nothing queued" no longer has one verdict for two causes. If the fleet
never actually filled — a slot can free before the sweep reaches the 15th
mission — the queue was not reached and this scenario did not test it: NORUN,
naming the high-water mark. Only a burst that DID saturate can call an absent
queue a failure.
Co-Authored-By: Claude Opus 5 <[email protected]>
The burst re-run printed "architect peaked at 1 of 6" and "nothing ever
queued" for a run I could watch sitting at architect=6 tank=6 morpheus=2 with
2 phases queued. The fleet was right; the sampler was blind.
Three separate ssh+psql calls per 15s tick, each with stderr to /dev/null, and
under the load of 16 concurrent missions most came back empty. Empty was then
read as "nothing running" — absence encoded as a legitimate value, which is the
exact seam the header of this file was written about, reproduced in a scenario
added to catch it.
One query per tick now, returning done/blocked/per-node in a single row, and
unreadable samples are COUNTED rather than silently treated as zeroes. Fewer
than ten usable samples is NORUN: a sampler that barely looked must not be able
to describe itself as a fleet that was idle.
Co-Authored-By: Claude Opus 5 <[email protected]>
The re-run reported FAIL-NORUN "the burst did not finish in 1800s". The fleet
was fine — 3 of the 16 missions were still in `draft`. Each PATCH-to-running is
an ssh plus a `docker run curl`, and 16 at once does not reliably land; the
response was going to /dev/null, so a launch that never happened spent the full
timeout looking like a platform stall.
That is precisely the swallowed-error shape this file was written to catch,
committed inside the file itself. Launches are now verified against the mission
rows, retried once for the stragglers, and reported as "the burst never
happened" rather than as a timeout — a scenario that did not run must not be
able to describe itself as a slow one.
Co-Authored-By: Claude Opus 5 <[email protected]>
Phase 1 shipped placement-at-phase-launch and a queue made of
`start_pending_phases` leaving a phase `pending`, both deployed unproven under
load — the exact condition this project keeps getting burned by: the code is
right, the system is wrong, and nothing errors.
`capacity` launches `slots + 2` microVM missions simultaneously and asserts two
things. That no node ever exceeds the slots `vm_placement` gave it — overcommit
does not fail loudly, it swaps, and every mission on that node gets slow rather
than dead. And that the excess QUEUES: a burst that drops the extras and one
that wedges them both look identical to any check that only reads the end
state. `capacity_blocked_since` is cleared the instant a phase is placed, so
the evidence only exists mid-flight; the scenario samples while it runs.
Capacity comes from `/api/fleet/capacity`, never recomputed here — a bash copy
of the slot arithmetic would drift from the scheduler and then agree with
itself. A burst that does not exceed capacity is reported NORUN, per rule 3.
`drain-midmission` drains the node phase 0 ran on, before phase 1 is placed,
and asserts phase 1 lands elsewhere AND still reads phase 0's file. That is the
test of the affinity decision: mission state lives on the gateway, so
re-placement is free — if it were not, this would either strand the mission or
silently lose the earlier work, and "silently lose" is what a status-only check
calls success. The node is restored before any assertion runs, so a failure
cannot leave the fleet permanently one node smaller.
Smoke-checked at CAPACITY_BURST=2: sampling, spread and completion all report,
and the queue check correctly returned NORUN rather than a green tick.
Co-Authored-By: Claude Opus 5 <[email protected]>
The retry landed and still failed: all four attempts returned 429. A bare
16-token probe with the same token, straight from gw-04, also returned 429
with `x-should-retry: true` — the Claude Code subscription itself is limited
right now, and no amount of backoff inside one HTTP request will outlast it.
So stop pretending it is a server bug. New `ApiError::Unavailable` → 503,
carrying the one sentence the operator can act on ("clears on its own; try
again shortly"), instead of an opaque `internal error` that sends them into
the logs. The harness now prints the response body rather than the generic
"the planner produced no usable proposal", which is what hid both walls —
first the credit balance, now this.
Co-Authored-By: Claude Opus 5 <[email protected]>
First canary run failed: phase failed, nothing delivered, and the streamed log
said exactly why — "Failed to authenticate. API Error: 403 api.anthropic.com is
not on the egress allow-list".
Not a 2.1.226 regression. `canary-claude` was added to the server's credential
map and not to the node's `provider_hosts`, so the VM booted with a valid
subscription token and a door that only opened onto the forge. The fail-closed
branch was working correctly: a backend nobody taught that function about
reaches no model API, deliberately, so it cannot silently borrow another
provider's door.
Both maps now name it, each pointing at the other, with a test asserting the
canary reaches the same provider as `claude` AND that unknown backends still
resolve to nothing.
Worth noting what made this a five-second diagnosis instead of an afternoon: the
live log streaming built earlier today. The failure was a 403 inside a microVM
that no longer exists, and its reason was sitting in the run's checkpoint.
Fourth instance of the same defect, and the last of the three commands that run
as root against a mission tree.
`verify_tests` execs the project's test command with `workdir = repo` — the live
checkout — inside a container running as ROOT. `cargo test` writes `target/`, so
the checkout ends up owned by two uids and the next phase's cargo hits
permission-denied. The harness reported `uids=0,65532` the first time this gate
ever ran end to end.
It survived because it had never run. Every one of the ten harness fixtures used
`commit_policy: "always"`; `on_green_tests` and `on_reviewer_approval` were
parsed, implemented, and never exercised — and `Gate`'s own doc already records
that three recipes carried this policy while it "did precisely nothing" for want
of a reader. A policy that is never exercised is indistinguishable from one that
is ignored.
Consolidated rather than fixed a third time. `root_copy` now owns the pattern —
copy through `mission_fs::pack_dir` into a SIBLING of the mission dir, run there,
and purge FROM INSIDE THE CONTAINER, because the copy's `target/` is root-owned
and the server (uid 65532) cannot delete it. `benchmark_runner` moved onto it;
`evaluator_tools::Sandbox` keeps its own copy logic for now (it carries an
allow-list and a judge-facing API, so folding it in is a larger change than this
moment warrants — noted, not done).
The gate fails CLOSED if the copy cannot be made: an unverifiable suite must not
license a push.
Also adds the `refactor` scenario, which is what found this. I had written it off
as "structurally identical to four existing scenarios" — wrong: it is the only
recipe carrying `on_green_tests`, and that made it the only one testing this
code path at all.
245 lib tests, 20 test binaries.
The third recipe whose defining phase is not `coding`, and so the third that
nothing could fail before `PRODUCING_KINDS` widened: a `security_scan` phase
that ran no scanner and wrote nothing reported success.
One phase, not the recipe's full scan->research->code chain — what is under test
is the phase KIND, and the other two kinds are already covered.
Two assertions, because the first alone is weak. "Delivered a file" is satisfied
by an agent that writes "I scanned it, all clear" and runs nothing — the
letter-not-purpose shape this codebase keeps paying for. So the delivered patch
must also carry the scanner's OWN output. Verified against the real run: the
agent produced gitleaks' banner, INF/ERR lines, byte counts and exit code, not a
claim about them.
Only `refactor` is now uncovered, and deliberately: its single phase is `coding`,
structurally identical to chain/multirole/microvm/noop. It would add runtime and
no new signal.
security 4/4 against the live fleet.
The portal offers five workflow recipes. Every one of the harness's seven
fixtures was `research_and_code`, so four recipes had never run end to end —
and that is not a theoretical gap. `research_only` DESTROYED its output for as
long as it existed: `requires_repo = false`, so the capture query's
`AND m.repo_id IS NOT NULL` skipped it, the container was reaped unread, and
eight ClawHDF5 research documents were lost while the mission reported
`completed`. Nothing in 550+ tests could see it, because nothing ran the recipe.
`research-only` asserts the whole chain the loss ran through, not just the
happy end of it:
- the phase completes
- document artifacts exist AT ALL (the missing thing)
- the agent's seven identity files (SOUL.md, MEMORY.md, …) are NOT published
— the first live capture published all seven, because `.git/info/exclude`
cannot protect a mission with no `.git`
- the captured text reads back through the content endpoint, since an
artifact row pointing at nothing is a 404 with no explanation
`benchmark` covers the other half: a benchmark mission is ONE benchmark phase,
and while `empty_delivery_is_a_failure` tested `kind == "coding"` that phase was
exempt — nothing in the platform could fail it. The scenario asserts it both
completes AND delivers files.
Also: `run_scenario` takes an optional `no-checkout`. The single-writer uid probe
is a property OF A CHECKOUT, and a repo-less mission has none by design, so
probing reports a platform fault that is really a category error. It is declared
per scenario rather than inferred from a missing directory — that inference would
silently excuse a repo-BACKED mission whose checkout was reaped early, which is
the exact condition the probe exists to catch.
research-only 4/4, benchmark 3/3 against the live fleet.
The unit tests prove the plumbing GIVEN `released_at_cap: Some(true)`. They
cannot prove the guest writes the marker, that the probe reads it back across
the vsock, or that the phase lands `failed` for the right reason — and every
one of those is where this class of bug has actually lived.
The check is `exit 1`: impossible by construction, so the run exercises the
release path rather than hoping to catch it.
`blocks` reaching the cap is deliberately NOT the assertion. A healthy agent
blocked three times and succeeding on the fourth reports the same 3. The phase
STATUS is the assertion; the block count and the failure reason are corroborating
checks, so a phase that failed for some unrelated reason cannot pass this.
Measured on gw-04 against b36ae00, all 4 checks green:
phase 0 failed
the gate spent all 3 blocks before giving up
the failure names the cap release as the reason
Before b36ae00 that same mission completed green.
The microvm scenario's judge assertion failed four runs straight. I blamed the
wording twice and rewrote it twice; the second rewrite made it worse. That was
guessing.
With scripts/judge-eval.sh in place the question is cheap to settle. Three
candidate conditions, three draws each, same evidence and same system prompt:
"its second line is …" MET UNMET MET flaky
"records the kernel version …" MET MET UNMET flaky
"contains both … and …" MET MET MET stable
So it was never noise in general — it is a reproducible weakness with
POSITIONAL and EXCLUSIVE phrasings. "its second line is X and nothing else"
invites this judge to invent requirements about the other lines, which is
exactly the reason it kept citing ("the first line contains 'test result: ok'").
Both fixtures now state what the file CONTAINS. The composed one was checked in
both directions — 3/3 MET on good evidence, 3/3 UNMET when the versions are
missing — because a wording that always answers MET would look stable and prove
nothing.
The eval keeps `kernel-ok` failing on purpose; it is the case production hit,
and tuning it green would turn a measurement into a decoration.
Harness: 24/24, including the assertion that had failed four times.
My previous attempt at this made it worse, which is the useful part.
The condition said "a Linux kernel release string" and the judge rejected
`6.1.128` as "not a Linux kernel release string such as 'Linux 6.1.128'". I
rewrote it as "the exact output of `uname -r`" — and the next verdict was that
line 2 should be `27.0.0`. The judge has a sandbox and allow-listed commands, so
naming a command told it to RUN that command, in ITS OWN container, and compare
the file against the answer it got there. The file records a microVM's kernel;
the judge was comparing it against the machine the judge runs on. Those are
different machines by design — that is the entire point of the assertion.
So a `done_when` for a tool-using judge must describe the VALUE's shape, never a
command that produces it: "a bare kernel version of the form MAJOR.MINOR.PATCH
(for example 6.1.128) and nothing else", plus an explicit instruction not to run
uname and not to compare against the local machine, because the file records a
different one.
The general rule, worth carrying into how `done_when` is written anywhere: a
condition phrased as "the output of X" is ambiguous about WHERE X runs, and a
judge with tools resolves that ambiguity by running X where it stands. Conditions
about a remote or past environment must be stated as properties of the recorded
value.
The scenario's real proof that the agent ran in a guest is unchanged: a separate
comparison of that line against the actual gateway and node kernels, which has
passed on every run including the two where the judge disagreed.
Co-Authored-By: Claude Opus 5 <[email protected]>
The restored GLM judge failed a phase that had done the work: MICROVM.md existed
with two lines and the second was `6.1.128`, and the verdict was "a kernel
version number, not a Linux kernel release string such as 'Linux 6.1.128'".
The judge is wrong on the fact — `6.1.128` is exactly what `uname -r` prints,
and "release" is the term for it — but the CONDITION was ambiguous, and it is our
fixture. "A Linux kernel release string" can be read as either `uname -r` output
or `Linux x.y.z`, and a stricter reader is entitled to the second. Both scenarios
now say what they mean: the exact output of `uname -r`, a bare version, no prefix.
This is not weakening the assertion. The scenario's own kernel check — the one
that proves the agent ran in a guest rather than on a host — is a separate,
unchanged comparison against the real host kernels, and it PASSED on the same
run. What changed is only that the mission-level `done_when` now describes an
observable fact precisely, which is what this codebase's own plan-authoring
prompt tells models to do.
Worth recording rather than papering over: an over-strict independent judge is a
much safer failure mode than an over-lenient one, and this is evidence the judge
READS the tree instead of rubber-stamping it — the Goodhart incident that
motivated cross-provider validation was the opposite failure. But it does mean a
vague `done_when` can now cost a phase, which raises the value of
`done_when_check` (a shell command, judged by exit status) for anything
mechanical.
Co-Authored-By: Claude Opus 5 <[email protected]>
Three threads, all of which end at the same place: a mission whose verifier does
not share a model with the coder it reviews.
**GLM has a credential contract now.** `microvm_credential_for` returned one env
var name, which quietly assumed every provider reads its secret from the same
place Anthropic does. It returns a `Credential { source, target }` instead —
z.ai's key lives in the server's `ZAI_API_KEY` and Claude Code reads it as
`ANTHROPIC_AUTH_TOKEN`, and collapsing those two names is what forces a guess at
the other end. A wrong guess here sends one provider's credential to another
provider's endpoint.
`images/agent-glm` is the same CLI at the same pinned version as `agent-claude`
with `ANTHROPIC_BASE_URL` baked in. The split is deliberate: the ENDPOINT is a
property of the image, the CREDENTIAL is a property of the turn. That makes the
dangerous mix-up unrepresentable — a GLM VM cannot be handed an Anthropic
subscription token, and a claude VM cannot be pointed at z.ai. Asserted both
ways, because "the GLM VM must not carry CLAUDE_CODE_OAUTH_TOKEN" is the
property that costs a credential if it ever stops holding.
Kimi stays refused. `KIMI_API_KEY` is set and Moonshot serves an
Anthropic-compatible API, but I have not verified its base URL against the
running service, and this function is precisely where guessing a URL is
expensive. It becomes an arm the day someone measures it.
`api.z.ai` joins the node's default egress allow-list. A default that cannot
run the images we ship is a trap rather than a policy — the alternative is an
operator discovering it as a hung agent with no model access.
**Per-role models for claws** (migration 0071). `template_roles` had no model
column, so `mint_team_from_template` bound every role of every mission team to
one literal — a template whose whole point is an independent reviewer minted a
reviewer sharing a model with the coder. A role may now name its own; roles that
say nothing still take the mint's default, so every template written before this
behaves exactly as it did. The literal is now that default rather than a
hardcode.
**A harness scenario for the roster flow.** `verify-mission-delivery.sh roster`
runs the whole Slice 5 loop — planner proposes, human approves, mission runs —
and asserts the roster LANDED on the mission row rather than trusting the API's
answer. That distinction is not theoretical: the first live approval returned an
error while leaving the proposal marked approved.
Built and proven on tank ahead of the deploy: `clawmates/agent-glm:dev` reports
`2.1.223` and `BASE=https://api.z.ai/api/anthropic`, and
`fc-build-rootfs.sh … glm 8G` boots a VM from it that has git, can write
/mission, and answers `claude --version`.
533 tests pass, clippy clean. Migration 0071.
Co-Authored-By: Claude Opus 5 <[email protected]>
`verify-mission-delivery.sh composed` runs a `team_engine=composed` mission and
checks the one property that cannot be checked any other way: a VM is inject →
run → collect → destroy, so unless the tree is carried node to node, node 2 boots
from the original checkout, sees nothing of node 1's work, and still reports
success. The task makes each node append ONE line to STAGES.md, so the delivered
file IS the evidence — a run that lost the handoff delivers one line, and no
amount of agent confidence can fabricate the missing ones.
It also asserts the run's tier is `microvm_graph`. A composed mission that
quietly fell back to the solo path would deliver a one-line file and look exactly
like a graph that ran one node.
`assert_stop_gate` reads the count `phase_runner` reports and distinguishes three
outcomes that matter: a number (installed, fired that often), `0` (installed,
never needed), and `-` (could NOT be installed — usually a CLI in the image with
no `--settings`). Wired into the microvm scenario rather than its own, because it
applies to every coding phase on that path.
Both ran against the deployed stack:
composed — 4/4. STAGES.md carried 5 stage lines through 5 separate VMs
(planner → coder → tester → reviewer → committer), each stamped with the guest
kernel 6.1.128 rather than the gateway's 6.8.0 or the node's 7.0.0. The run
checkpointed 5 steps on the worker, and `updated_at` stayed ~2s old mid-turn,
which is the keepalive doing its job — without it `requeue_stale` flips a live
run at 180 seconds.
microvm — 6/6, including the gate installed in a real VM (`blocks: 0`), one
subagent, the GLM judge, and the unavailable-backend negative control.
Co-Authored-By: Claude Opus 5 <[email protected]>
`select met || ' ' || independent` casts the booleans to `true`/`false`, but the
pattern matched `t`/`f` — psql's *column display* form. So the check reported "no
verdict recorded for the phase" while the row sat in the table saying met=true,
independent=true, glm-4.7.
A check that fails for a reason unrelated to what it checks is worse than no check:
it trains you to ignore the output. The booleans are cast explicitly now so the
shape cannot drift again, and the failure message prints what it actually got.
`verify-mission-delivery.sh microvm` now passes 5/5 against production:
- the agent ran under guest kernel 6.1.128, not the gateway's 6.8.0-124 or the
node's 7.0.0-28 — the one assertion that cannot pass by accident
- the lead delegated to 1 subagent
- the condition was met and judged INDEPENDENTLY by glm-4.7
- the checkout has exactly one writer (uid 65532)
- negative control: a backend no node can run is refused at launch
A REGRESSION I INTRODUCED ONE COMMIT AGO. `capture_finished_coding_phases`
selects on `mp.status = 'completed'`, so the moment an unmet phase correctly began
reporting `failed`, its diff stopped being captured, committed or pushed — the work
was silently discarded. Found by the new harness scenario, whose phase legitimately
missed its condition and then had no artifact at all.
What was produced, and whether the goal was met, are different facts. The artifact
records the first; `mp.status` records the second. Capture now covers terminal
phases (`completed`, `failed`), so a phase that did real work and missed its goal
still delivers a reviewable diff — which is exactly what the next pass needs.
`scripts/verify-mission-delivery.sh microvm` — the regression net this session was
missing. Everything the microVM track proved by hand was guarded by nothing:
- THE KERNEL LINE is the assertion that cannot pass by accident. Every other
check would also pass if the phase had quietly run in a container on the
gateway; only the kernel says WHERE it ran. Compared against the real gateway
and node kernels read at start-up rather than pinned to a version, so
upgrading vmlinux does not manufacture a failure.
- subagent count > 0, from the server's own count of Claude Code's per-subagent
transcripts. Before `Agent` was in the allowlist this was structurally
impossible and nothing said so. A probe that could not run reports "?" and
FAILS the check rather than reading as zero.
- the verdict's judge and whether it was independent.
- negative control, observed passing: a mission whose backend no node can run is
refused at launch and stays draft. Without it the positive scenario would pass
just as well against a scheduler that ignored `backend` entirely — which is
what it did until the first real microvm mission landed on a node with no such
rootfs.
Also fixed in the harness: `api` now sends the JSON body on STDIN (`curl -d @-`)
instead of interpolating it into a single-quoted argument inside a double-quoted
ssh command. A task description containing "the crate's test suite" ended the
quoting and killed the remote shell; two attempts to escape it were themselves
wrong, because the backslashes must survive bash AND sed AND sh. Removing the
interpolation removes the class, and the next author does not need to know that
apostrophes were forbidden.
475 tests pass, clippy clean.
The last open item in the silent-success class: a coding phase that
changed no files reported `completed` — the same status a phase gets for
delivering tested, reviewed, pushed work. Mission `019fcf62` completed
that way with its agents silently unpinned from the repo, and nothing in
the platform disagreed; it was found by a script diffing the forge.
The verdict is applied at capture rather than at completion, because
capture selects on `status = 'completed'` — the platform does not know
whether a phase produced anything until after it has already finished.
Three conditions must hold before failing a phase, because a false
positive here fails honest work: the phase is a coding phase (research
phases legitimately write nothing to the tree), the diff was actually
computed (an uncomputable diff also reports zero files — blaming the
agent for a platform fault is the same defect wearing different
clothes), and `allow_empty` is not set. Only an explicit `true` opts
out, so a typo leaves the check armed. Registered in phase_config with
its reader named, per the seam-2 rule.
Also closes an ordering hazard this exposed: capture is batched and runs
after a phase completes, so a backlogged mission could close as
'completed' and only then have capture discover an empty phase — leaving
a 'completed' mission holding a 'failed' phase, unfixable because the
mission-close CASE only touches 'running' rows. A repo-bearing mission
now waits for its work to be captured before closing.
Adds a `noop` scenario to the harness: a phase told to change nothing,
which PASSES only when the phase comes back `failed`. Same discipline as
the uid self-test — a check that has never been seen to fire has not
been shown to work.
Co-Authored-By: Claude Opus 5 <[email protected]>
A verification run against the deployed stack found a chain mission whose
phase 0 reported `completed` with zero files, no commit error and no push
error — indistinguishable from a phase that correctly had nothing to do.
Three separate defects had to line up, each of them the same shape: a
failure sharing its representation with a legitimate negative result.
1. `pin_agent_workspaces` embedded the whole config in one `sh -c` argv.
That works until the file grows — config gains a block per provisioned
claw — then fails with `argument list too long`. Now written through
the tar upload API, which has no argv limit, so the failure mode is
gone rather than merely further away.
2. A failed pin was logged "(continuing)". Without the pin, agents write
to their sandboxes and the committer finds nothing in /mission/repo —
the mission cannot deliver, so the launch now fails where someone is
still looking. The restart that applies the pin is fatal for the same
reason.
3. `capture_phase_diff_at` swallowed `git diff` failures with
`unwrap_or_default`, so an unreadable base landed `empty: true,
files_changed: 0` — byte-identical to an honest no-op. The error is now
recorded as `diff_error`, and an empty patch that came from a failed
diff is no longer trusted to mean an unchanged tree.
Adds scripts/verify-mission-delivery.sh, which found #1 and #2 on its
first real run. Its probes are fail-closed: no placeholder values, a
self-test that proves the uid probe can detect the split it looks for,
and FAIL-NORUN for a scenario that never executed. Its own first version
had this bug too — a `die` inside `$(...)` exited the subshell, so a run
that could not authenticate printed "all checks passed" and exited 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
