Two halves, and the first was worse than the plan assumed. `run_phase_in_vm`
packed `<missions_root>/<mission>/repo` unconditionally — a directory a
repo-less mission does not have — and then required `/mission/repo/.git` inside
the guest before spending a turn. So a repo-less microVM phase did not merely
go uncaptured: it failed before the agent ran.
A repo-less mission now gets an EMPTY workspace at the same guest path, created
host-side so the collect unpacks back over it with no special case, and the
readiness probe asks for what was actually sent — the directory rather than a
`.git` that was never going to be there.
`mission_outputs` then drops its `runtime_kind <> 'microvm'` exclusion, whose
stated reason ("a microVM mission always has a checkout") is exactly what
stopped being true. Where the files come from now depends on the runtime, and
the difference is not cosmetic: a container mission's output is still inside a
running container, while a VM's has already been unpacked onto the host by the
end-of-turn collect. Asking docker for a VM mission's files would query a
container that never existed.
The recursive copy skips symlinks rather than following them — a link out of
the tree would publish whatever it points at.
`research-vm` is the proof, added to the suite as well as the dispatch: the same
assertions as `research-only` with `runtime_kind: microvm`. A scenario nobody
runs is a scenario that does not exist.
Co-Authored-By: Claude Opus 5 <[email protected]>
Five fixed steps for every mission type, and getting a research document out of
it meant naming a team, choosing a runtime, and writing per-phase completion
conditions under a paragraph explaining what a model checker can and cannot
prove. Two of those steps asked for things the mission does not use, and one of
them blocked outright.
1 What do you want to do?
2 Title, a description with a Polish button, repo ONLY if the type needs one
3 Review -> Launch, plus one collapsed Advanced section
Two hard defects fixed on the way:
- The microVM runtime could not be selected AT ALL. Step 4 gated Next on
`targetNodeId`, which microVM deliberately never sets because placement picks
the node per phase. Everything shipped today, the local-GPU backend included,
was unreachable from the UI.
- Step 3 required a team while every workflow TOML already names one in
`default_team_template` — which this file ignored. The answer was always
available and the question was always asked. It is now resolved by key, with a
category fallback, and shown under Advanced so an operator can see WHICH
default rather than having to supply one.
A failed `/api/team-templates` request and a genuinely empty list rendered the
identical red banner, which sends the reader looking for missing template files
when the request had 401'd. They now say different things.
`phases[]` is no longer sent unless someone set a completion condition.
`recipeToPreset` strips each phase's `config`, so posting the stripped list
overrode the recipe's real settings — tools, commit policy, loop mode — with
nothing. Omitting it lets `phases_for_create` use the recipe, which is both
simpler and more correct.
Launch keeps its own gate, since Advanced can still produce an unlaunchable
combination — but it names what is missing instead of greying out in silence.
Artifacts get a Download link. Deliberately a plain link to the streaming route
rather than a Blob built from what "Read" already fetched: that content is
capped at 2 MiB and UTF-8-decoded, so reusing it would silently produce a
truncated or undownloadable file for exactly the artifacts worth downloading.
Co-Authored-By: Claude Opus 5 <[email protected]>
Two endpoints the wizard redesign needs.
`POST /api/missions/refine-draft` — the polish button fires while the user is
still typing, before anything is created, so it has no id to route on.
`refine` deliberately requires a saved draft because its Accept writes back;
this one has nothing to write back to and returns the text. Same system prompt,
same model chain. The phase list comes from the workflow recipe rather than the
caller, for the same reason `phases_for_create` prefers it: a client that
guessed would have the model write acceptance criteria for phases the mission
will not run.
`GET /api/missions/{id}/artifacts/{artifact_id}/download` — the file itself.
`artifact_content` caps at 2 MiB and reads as UTF-8, so a large or binary
artifact is unreachable by any means today; this streams the bytes with a
filename attached and no ceiling.
Both artifact routes now resolve through ONE containment check. Two copies of
"is this path under _outputs" is two chances for one of them to be the lenient
one, and the lenient one is an arbitrary read of the gateway's filesystem — so a
test asserts there is a single resolver and that both routes call it.
The download filename was chosen by an AGENT and lands in a header every browser
parses, so quotes, backslashes and control characters are stripped rather than
escaped; the test covers a header-injection attempt.
Co-Authored-By: Claude Opus 5 <[email protected]>
Clearing the fleet's four leftover agents returned 404 on every one. They had
been soft-deleted back in June — correctly invisible in the UI ever since — and
`agents::get` filters `deleted_at IS NULL`, so `workspace_agent` could not find
them. Every route uses it, including `batch-delete`, the one that exists to
HARD-purge. So a soft-deleted agent was unreachable from the application
entirely and its row stayed forever.
`get_any` sees them, and only the purge path uses it: hiding soft-deleted rows
is right for every read, and wrong for the one operation whose whole job is
removing them. Written with `query_as` rather than the checked macro so it does
not force an offline-cache regeneration on every machine that builds this.
`fleet-reset.sh` now uses `batch-delete` for agents rather than
`DELETE /api/claws/{id}`. The latter is a SOFT delete, so pointing a reset
script at it would have quietly added to the pile it was meant to clear.
Co-Authored-By: Claude Opus 5 <[email protected]>
`default` is the generic `rootfs.ext4` and `claude` is the named one, and
`microvm_credential_for` gives them the identical contract — so the list came
back with both under the same label, and whichever a user picked they got the
same thing. Collapsed to the named one where it exists; the generic keeps a
label of its own for a fleet that only has that.
Co-Authored-By: Claude Opus 5 <[email protected]>
The full harness found it — 12 of 13 scenarios green, `roster` red:
roster: the planner sized this mission at 2 member(s) PASS
roster: the approved roster is on the mission (2 nodes, composed) PASS
roster: this run added 1 line(s) for a 2-member roster FAIL
topology_runs.error: turn executor failed: node n1 in a microVM:
vm_create failed: no rootfs for backend "canary-claude" on this node
The roster proposed `verifier@canary-claude`. Placement asked
`online_for_backend` about the MISSION's backend — `claude` — and architect
answered, holding `claude` and `local-ornith`. The graph's first node ran and
delivered, the second could not boot, and the mission finished half-done. The
question placement asked was true and insufficient.
A composed graph runs on ONE node, so that node needs every image its nodes ask
for. `required_backends` collects the mission's plus each
`config.roster.nodes[].attrs.backend`, and `online_for_backends` passes the
whole set to the same jsonb `@>` — containment already means "contains ALL of
these", so the query shape did not have to change, only what it was asked.
This is the failure mode the roster feature creates by existing: its entire
purpose is putting a verifier on a different provider, which is exactly what
makes one node insufficient. Nothing before the full suite had a reason to
exercise it — the composed scenario uses one backend for all five nodes.
`NoCapableNode` now names the set and says why one node must hold all of them.
Co-Authored-By: Claude Opus 5 <[email protected]>
For a clean slate before a UI session, and for the thing that keeps being true
here: deleting a row has never deleted a directory. A full harness run leaves
~35 missions, each with a repo checkout and a runtime-data tree, on the smallest
disk in the fleet. There are 125 rows and 117 directories right now.
Deletes through the API, never with SQL. `missions::delete` tears down the
per-mission runtime container, hard-purges the FK graph in order, and removes
the workspace directory — falling back to a root purge for the files the
per-mission daemon leaves as root. A `DELETE FROM missions` skips all three and
orphans every one of them, which is how the orphans got there.
Then it checks, because rows gone is not bytes back and every incarnation of
this cleanup has managed the first while silently failing the second: it names
each directory left without a row, and counts root-owned residue separately
because that is the specific way it fails.
Refuses outright while any mission is RUNNING. Yanking a live mission's checkout
leaves a VM writing into a directory that no longer exists, and the symptom is a
phase that hangs rather than one that fails. Verified: it stopped exactly there
against the in-flight harness.
Dry by default; `--yes` to act; `KEEP=<substring>` to spare some.
Co-Authored-By: Claude Opus 5 <[email protected]>
Added to the case dispatch when it was written, and not to `all` — so the
newest backend, and the only one that runs on hardware we own, was excluded from
the one run that claims to check everything. A scenario nobody runs is a
scenario that does not exist.
Co-Authored-By: Claude Opus 5 <[email protected]>
Everything built today — Firecracker missions, the four backends, the local GPU
one — was unreachable from the dashboard. The wizard offered `zeroclaw` and
`local_herdr` and nothing else, so a mission created in the UI could not be a
microVM mission at all, and `local-ornith`/`glm`/`kimi` were API-only. Testing
"our workflows in the UI" would have exercised none of it.
Adds the runtime option and a backend picker, fed by a new
`GET /api/fleet/backends` that returns `mission_roster::available_backends`
verbatim — the SAME list the roster planner is handed, not a second one. Its two
rules are both load-bearing and neither is visible from a node's capabilities
alone: the image must be built on an online node, and the backend must have a
credential contract. `agent-terminal` passes the first and fails the second —
bootable, with nothing for the agent inside to authenticate with — so offering
it would produce a mission that validates, launches, and dies at the agent turn.
Ids are deployment vocabulary, so the picker labels them: a user choosing
between `local-ornith` and `canary-claude` should not have to know which company
each one bills. An empty list says why (no rootfs built) instead of showing an
empty dropdown, and no node is chosen for a microVM mission because
`vm_placement` picks it per phase.
Co-Authored-By: Claude Opus 5 <[email protected]>
`cleanup_sweeper` prunes ROWS. Deleting a row has never deleted a directory,
and `teardown_container` only runs while a mission still exists to tear down —
so a mission removed by any path that skipped teardown left its tree behind
permanently, on the smallest disk in the fleet (150 GB, shared with postgres and
every checkout). 106 mission directories are sitting there now.
Filesystem-first, deliberately: the DB is the PREDICATE, never the enumerator.
Enumerating from the database is exactly how these became invisible — a
directory whose row is gone is the one a row-driven sweep cannot see.
Three reapers, one deletion path. Orphan mission dirs (no row, past a 2h grace),
scratch trees (_bench/_gate/_verify/_merge past 6h — all four have leaked
before), and _outputs past 90d, whose artifact rows are marked only AFTER the
files are gone, because the other order claims artifacts are reaped while they
are still on disk.
The single removal path escalates: the server is uid 65532 and cannot delete
what the per-mission daemon leaves as root, so PermissionDenied falls back to
`root_copy::purge` and shouts if the tree survives even that. A GC that cannot
collect is the thing being fixed, so failures are counted and reported, never
swallowed.
Guards worth naming: `_cargo` is a SHARED cache every mission writes to and
lives under the same root, so an underscore-prefixed sibling treated as an
orphan mission would delete it out from under running work and look like a slow
cargo build. Only a well-formed mission id is ever a candidate — a directory
whose name is not an id can have no row by construction, so without that gate
every unrecognised directory looks orphaned.
Co-Authored-By: Claude Opus 5 <[email protected]>
Deployed, and the report simply did not appear — from the tool built to stop
things failing silently. Two causes, both worth keeping:
There was no timeout anywhere in the probe, so one slow provider swallowed the
entire report. Each link is now bounded at 60s (generous: `complete_or` spends
up to 30s in its own backoff, so a tighter cap would report a merely throttled
link as hung) with `TimedOut` as its own state, and every line is emitted AS IT
RESOLVES rather than collected and printed at the end — a later link that hangs
must not be able to hide the ones already checked.
The first attempt at the timeout awaited the probe and then wrapped the result:
let probe = complete_or(...).await;
timeout(PROBE_TIMEOUT, async { probe }).await
That compiles, reads correctly, and bounds nothing. The timeout has to wrap the
future.
Co-Authored-By: Claude Opus 5 <[email protected]>
Committed and deployed while its test compile was failing: the verify step was
`cargo test | grep -E "^error|test result" && git commit`, and grep exits 0 when
it MATCHES, so finding the error is what let the commit proceed. The library
built fine, so the deploy was sound, but the check that was supposed to gate it
did the opposite of gating.
The error itself was a borrow in a test closure; a plain fn fixes it.
Co-Authored-By: Claude Opus 5 <[email protected]>
fallback chain (6 link(s), 4 usable):
claude-opus-4-8 ok
claude-sonnet-4-6 ok
claude-haiku-4-5-20251001 ok
kimi:kimi-k2.7-code BROKEN: 400 ... role 'system' must not be empty
glm:glm-4.7 ok
local:ornith-fleet:9b UNREGISTERED — resolves to the DEFAULT provider
Neither link was actually broken.
The probe sent an EMPTY system prompt, which Kimi rejects outright. A probe has
to look like the traffic it stands in for, or it measures itself.
The second is the one worth keeping. `resolve_provider` returns a spec unchanged
when it does not recognise the provider, and the part after the FIRST colon when
it does — so the obvious test, "does the model half still contain a colon",
reads correctly and is wrong the moment a model id has one. `ornith-fleet:9b`
has one. The probe reported a provider the server had just finished registering
as UNREGISTERED.
`evaluator::cross_provider_judge` had the identical check, and would therefore
have refused a local judge as "not independent" — silently falling back to a
same-family one, which is the exact claim that path exists to make honestly.
Both now compare against the whole spec.
That bug was written into the codebase before a model name with a colon existed,
was correct at the time, and became wrong when one arrived. Nothing would have
reported it; a boot-time probe of every link did, on its first run.
Co-Authored-By: Claude Opus 5 <[email protected]>
opus -> sonnet -> haiku -> kimi -> glm -> local. The order is capability first,
then independence: three Anthropic tiers on one account (a throttle usually hits
a tier, so stepping down often clears it), then two separately funded accounts
(now an outage, not just a throttle, is survivable), then our own GPU (nothing
left to be down). Every id was probed on this deployment and answered 200.
The preflight is the more important half. Configured is not working, and this
chain has a specific way of lying: `resolve_provider` falls back to the DEFAULT
provider when it does not recognise a provider name, so a typo in `kimi:` does
not error — it quietly runs on Anthropic, and a chain that reads as three
accounts is really one. A reachability-only probe calls that link green.
So `preflight` checks resolution and reachability separately, eight tokens per
link through the REAL call path, and reports four states. `Throttled` is
deliberately not a failure: a 429 means the spec resolved, the credential
authenticated, and there was no capacity this second — the exact condition the
chain exists to route around, and painting it red would train an operator to
ignore red. `Unregistered` and `Broken` are failures, and they get different
words because they need different fixes.
It runs at boot alongside validator_preflight and runtime_preflight, spawned so
it cannot delay startup. A chain is the one piece of infrastructure nobody looks
at until the day it has to work, so it is now checked on the days it does not.
Co-Authored-By: Claude Opus 5 <[email protected]>
Forcing the planner onto the local link produced a green chain and a red
assertion:
roster: the planner sized this mission at 1 member(s) PASS
roster: ROSTER.md has 3 line(s) for a 1-member roster FAIL
The model was right and the check was wrong. ROSTER.md does not start empty —
the auto-merge work put an earlier run's two lines onto main — so a 1-member
roster that correctly appended one line delivered three, and the scenario
reported a model that had ignored its own proposal.
It now measures the DELTA against main. Any assertion against a scratch repo
that accumulates has to, or it decays into a test of how many times it has been
run before.
Proven on the local model end to end: opus 429 -> local:ornith-fleet:9b
answered -> `mission_roster: ... local:ornith-fleet:9b proposed 1 member(s)` ->
the composed graph ran -> the branch added exactly one line. 5/5.
CLAWMATES_MODEL_FALLBACK is removed from gw-04's .env again; it was set only to
force the last link for this test, and the deployed default is the full chain.
Co-Authored-By: Claude Opus 5 <[email protected]>
`local:ornith-fleet:9b` joins opus -> haiku -> glm as the final link. Every
entry above it depends on somebody else's account staying funded and
unthrottled; this one depends on a GPU in the next room. It is last because it
is the weakest model, and present because a chain whose every link is external
is not a fallback chain, it is one outage in a trench coat.
Three small changes make it work:
- `build_provider_registry` accepts a provider with an empty `api_key_env`.
A model on our own hardware has nothing to authenticate to, and the old
behaviour SKIPPED a keyless provider — leaving the chain quietly one link
shorter than it reads, which is the failure mode this whole area keeps
producing.
- `provider_family` learns `ornith`/`ollama` for BARE names. A qualified
`local:` spec was already answered by the split, but a bare one fell through
to "unknown", and `cross_provider_judge` would then refuse a judge that is
genuinely a different family from the Anthropic implementer.
- A test pins that the last link survives `resolve_provider`'s split-on-FIRST-
colon: `local:ornith-fleet:9b` is provider `local`, model `ornith-fleet:9b`.
Splitting on the last colon would ask for a provider named
`local:ornith-fleet`, and the symptom would be a silent fall back to the
default provider.
Infra: Ollama on tank and architect now binds 0.0.0.0 so the gateway (which has
no GPU) can reach it. `tailscale serve` cannot — Ollama rejects a non-local Host
header as a DNS-rebinding guard and OLLAMA_ORIGINS is CORS-only, so it 403s.
0.0.0.0 still includes loopback, so the microVM vsock pipe is unaffected;
verified on both nodes. This is an explicit trade: Ollama has no auth and its
API can pull and delete models, so it is now reachable from the LAN as well as
the tailnet. The drop-in carries the ufw one-liner to close the LAN side.
Co-Authored-By: Claude Opus 5 <[email protected]>
`local-ornith` scenario, green on its first real run against tank:
local-ornith: a locally-served model delivered a guest kernel (6.1.128)
local-ornith: no Anthropic egress from a locally-served mission
local-ornith: the node bound its local-model socket for this VM
local-ornith: checkout has exactly one writer (uid=65532)
Three things had to be true at once and only a real run shows all three: the
agent reached a model at all (a pipe to a closed port produces a turn that HANGS
rather than errors, which is why this is a scenario and not a unit test), the
work came back and landed on a branch, and the VM still could not reach
api.anthropic.com.
That last one is not theoretical. The node log for this VM is a column of
`egress DENIED api.anthropic.com` — Claude Code's own telemetry, correctly
refused — while the model traffic went through the vsock pipe and Ollama logged
loading ornith-fleet:9b at 100% GPU with CONTEXT 131072. A local backend that
quietly kept Anthropic egress would be a credential path nobody asked for.
The egress check asks the NODE's proxy log rather than the agent, for the same
reason the GLM measurement did: a model's account of where its tokens came from
has no evidential value, and the proxy's record of what it dialled does.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude Code pointed at the Ollama already installed on every GPU node. Ollama
has served a native Anthropic-compatible /v1/messages since v0.14, so this is
an env contract rather than a translation layer — the fourth variation on the
same idea as agent-glm and agent-kimi.
The route is NOT the egress proxy, and that is the design. `egress` speaks
CONNECT, takes a destination from the guest, resolves it and decides; every one
of those powers is a liability, which is why it refuses non-443 ports and IP
literals after a unit test caught them being bypassed. Routing a local model
through it would have meant relaxing both.
`local_model` is the opposite shape: there is no destination in the protocol.
fcagent listens on guest 127.0.0.1:11434 and pumps to vsock 9003; the node
splices that onto its own 127.0.0.1:11434 and copies bytes. A compromised guest
cannot redirect it because there is nothing to redirect — it is a pipe, not a
proxy, and strictly narrower than anything an allow-list could express. The
bytes never touch a network, so there is no wire for TLS to protect, and Ollama
stays bound to loopback rather than being exposed on the tailnet.
The socket is bound only for a backend declared to use a local model, so a
`local-ornith` VM reaches the forge through egress and nothing else, while every
other backend's guest port simply refuses. Both halves have negative controls.
`scripts/fleet-model-setup.sh` exists because of one measurement: stock
ornith:9b reported input_tokens=2050 for a 48000-word prompt and answered as
though nothing had been dropped. Ollama's default window is ~2K whatever the
model card says, and it truncates silently — the exact failure an agent turn
would hit and never report. The script pins num_ctx=131072 into a derived tag
and then PROVES both the window and tool calling before declaring success.
Verified on architect: ~65536 words -> 65604 input tokens, stop_reason=tool_use.
Placement needs no new capability key: building the rootfs only on GPU nodes
means `nodes::online_for_backend`'s existing `rootfs @> ["local-ornith"]`
predicate does the affinity, so morpheus never offers the backend.
Co-Authored-By: Claude Opus 5 <[email protected]>
Phase 1 of the local-model plan: prove the model before writing any plumbing.
`JUDGE=local` runs the existing done_when eval against Ollama on a GPU node.
Requests originate on that node rather than the gateway, because the model is
bound to 127.0.0.1 deliberately — it has no network exposure at all — and the
gateway has no GPU.
MEASURED on tank, 3 draws per case, against the incumbent on the same cases:
local (ornith-fleet:9b) 14/15 — one UNPARSED, never a wrong verdict
glm (glm-4.7) 13/15 — two WRONG verdicts on kernel-ok
kernel-ok is the case production actually hit and the one this script's header
says is expected to fail on glm-4.7. A 5.6 GB model on hardware we already own
did not get it wrong once in three draws.
The tag is `ornith-fleet:9b`, not `ornith:9b`, and that is the finding worth
keeping. Ollama defaults to a ~2K window whatever the model claims: stock
ornith:9b reported input_tokens=2050 for a 48000-word prompt and answered as
though nothing had been dropped — silent truncation, confidently. The fleet tag
pins num_ctx=131072, which measures 9.3 GB resident of a 16 GB card (the full
262144 also fits, at 13.6 GB, 100% GPU). These eval cases are a few hundred
tokens, so this eval would have passed either way; that is exactly why the tag
under test has to be the one production would use.
Also measured: Anthropic /v1/messages returns well-formed tool_use with
stop_reason=tool_use on both nodes; the reported count_tokens?beta=true hang is
absent in 0.31.1 (clean 404, server unaffected); ~60 tok/s generate, ~2800
tok/s prefill, 120072-token prompts accepted end to end.
Co-Authored-By: Claude Opus 5 <[email protected]>
The server runs as uid 65532, so `remove_dir_all` on a mission directory
returns PermissionDenied the moment anything root-owned is left in it — and the
old code logged that at the same level as "file not found" and moved on. The
directory then lived forever.
After the seed-copy fix a mission holds 3281 files owned by 65532 and 26 owned
by root: `.claude.json` and the session jsonl the per-mission ZeroClaw daemon
writes itself, after the copy has been chowned. Twenty-six files is small
enough to keep every mission directory alive without anyone noticing why.
PermissionDenied now falls back to `root_copy::purge`, which deletes from
inside the container as root — the same escape hatch `container_exec` keeps for
exactly this, clearing debris a root process created. And if the directory
survives even that, it says so, because a cleanup that silently failed is the
thing being fixed.
Removing the last 26 properly means running the per-mission daemon as 65532,
which needs `/mission` pre-created in the image with that ownership — the
daemon creates it at boot today and cannot at a lower uid. That is an image
change, deliberately not bundled here.
Co-Authored-By: Claude Opus 5 <[email protected]>
Running the seed copier as 65532 broke mission launch, and broke it quietly.
The seed dir is root-owned with parts at mode 0600 (`.claude.json`,
`clawmates-mcp.json`), so uid 65532 cannot READ them: `cp` failed on the first
unreadable entry, `set -e` abandoned the rest, and the mission came up with a
runtime-data holding `.zeroclaw` and nothing else — no Claude credentials, no
door config. The daemon then never created its agents' workspace, and the phase
failed 200 lines later on "Could not find the file /mission in container",
which points nowhere near the cause.
It was quiet because `seed_runtime_data` polled for the container to STOP and
returned Ok without ever reading its exit code. A copier that died on a
permission error and one that finished cleanly were indistinguishable. It now
reads the status and says what went wrong.
So: root for the read, `chown -R 65532:65532 /dst` for the result. Both halves
matter and they pull opposite ways — root is needed to read the seed, and
65532 is needed because everything else in the missions tree is 65532 and a GC
running as 65532 cannot delete what root left behind.
Co-Authored-By: Claude Opus 5 <[email protected]>
The uid fix landed and the CHECKOUT came back completely clean — 0 non-65532
files under `repo/` after a benchmark run that builds and tests Rust. But the
same mission still held 3247 root-owned files, all under `runtime-data/`.
`seed_runtime_data` spawns a throwaway container to `cp -a` the runtime seed
into the mission's directory and never set `user`, so it ran as root — the
identical absent-`user` omission `container_exec` had, in a container create
instead of an exec. The seed source is 65532-owned and the destination is
created by the server (which itself runs as 65532), so the copy never had a
reason to out-rank either.
This is the tree a gateway GC has to be able to delete, and a GC running as
65532 cannot remove root-owned files — the cleanup-that-cannot-clean-up shape,
found before writing the GC rather than after.
Co-Authored-By: Claude Opus 5 <[email protected]>
`CreateExecOptions` never set `user`. Not a wrong value — an ABSENT one: the
daemon defaults to root, and twelve callers inherited that without any of them
choosing it. That single omission is the origin of four separate patches —
root-owned `target/` directories inside a checkout owned by 65532, `root_copy`
existing at all, and a cleanup that had to re-enter the container as root to
undo its own mess.
The rule is positional and lives in ONE place: an exec whose workdir is inside
`missions_root()` runs as 65532; anything else (preflight probes, image checks)
keeps the daemon default so unrelated call sites cannot break. Twelve callers
each remembering to pass a uid is twelve chances to forget, and the one that
forgets leaves debris the other eleven cannot remove.
Non-root needs an environment the image does not provide. Measured in the
deployed image: uid 65532's HOME (/zeroclaw-data) and /usr/local/cargo are both
root-owned and unwritable, so this would otherwise break every cargo call — the
benchmark runner, the judge's sandbox, the delivery test gate — far more quietly
than the leak it fixes. The missions root IS bind-mounted and writable by 65532,
so HOME/CARGO_HOME move there and the cargo cache is shared across missions
rather than re-fetched per mission. Verified on gw-04: a clean `cargo build` as
65532 with those three variables produces output owned entirely by 65532.
Root remains reachable only through `exec_as_root`, whose name says so, and
which exists solely to clear debris earlier root execs left. `runtime_preflight`
now probes the whole policy at boot, so an image that moves or tightens that
mount fails loudly instead of failing every cargo call for a reason no error
message would connect to a uid. evaluator_tools' inlined fourth copy of the
purge is replaced by `root_copy::purge`.
Co-Authored-By: Claude Opus 5 <[email protected]>
drain-midmission passed 3/3 twice, but the "re-placing this phase" line the
last commit added never appeared in the log. It cannot: `online_for_backend`
filters on `status = 'online'`, so a draining node is not a candidate, never
reaches `unfit`, and the pin simply falls through to ranking — on the old code
as well as the new.
So the scenario passes either way and proves the affinity decision, not the
`TargetUnfit` bug. The path that genuinely used to fail a phase is "the
previous phase's node has since FILLED UP": that puts it in `unfit`, which
returned a non-transient error, which never reached the queue. That is what the
unit test now says, in place of a claim about draining the harness does not
support.
The accidental mission-to-node affinity was real and unconditional either way.
Co-Authored-By: Claude Opus 5 <[email protected]>
`drain-midmission` found this. `choose` treated its `want` argument as a hard
requirement, and the only caller passes `missions.target_node_id` — which is
not an operator's choice, only where the PREVIOUS phase happened to run. Two
consequences, both wrong:
- A node drained or filled between phases produced `TargetUnfit`, which
`is_transient()` says false to, so `phase_runner` FAILED the phase rather
than queueing or moving it. The queue silently did not apply to the second
phase of any mission.
- While the node stayed fit, every later phase went straight back to it
regardless of ranking — accidental mission-to-node affinity, which this
module's own header says must not exist.
Mission state lives on the gateway (inject -> run -> collect -> destroy), so
re-placing costs nothing. The pin is now advisory: preferred while it fits,
and when it does not, the reason is logged and ranking proceeds. `TargetUnfit`
is deleted rather than left unconstructed, so it cannot come back as a
non-transient failure by accident.
The scenario had its own race: it waited for phase 0 to COMPLETE before
draining, but warm phases finish in ~80s against a 10s placement sweep, so
phase 1 was often already placed — and the run then blamed the platform for
running on a node that was not yet drained. It now drains while phase 0 is
still running, which does not disturb a live VM and is the more faithful test.
Co-Authored-By: Claude Opus 5 <[email protected]>
The first bursts took 25 minutes because every VM paid a cold 2.4 GB rootfs
copy. Warm, the same 16 missions finish in 70-140s each and the whole burst is
over in about two minutes — so a sampler that waited ~90s for its launch check
and then ticked every 15s caught three samples of the tail and reported
"architect peaked at 1 of 6" for a run that sat at 6/6/2.
Sampling now starts at the first tick, runs every 5s, and folds the launch
check into the same query so verifying the launches costs no observation
window. The 10-sample floor that produced the last NORUN is gone; it was
measuring how long the burst took, not how well it was watched.
And "nothing queued" no longer has one verdict for two causes. If the fleet
never actually filled — a slot can free before the sweep reaches the 15th
mission — the queue was not reached and this scenario did not test it: NORUN,
naming the high-water mark. Only a burst that DID saturate can call an absent
queue a failure.
Co-Authored-By: Claude Opus 5 <[email protected]>
The burst re-run printed "architect peaked at 1 of 6" and "nothing ever
queued" for a run I could watch sitting at architect=6 tank=6 morpheus=2 with
2 phases queued. The fleet was right; the sampler was blind.
Three separate ssh+psql calls per 15s tick, each with stderr to /dev/null, and
under the load of 16 concurrent missions most came back empty. Empty was then
read as "nothing running" — absence encoded as a legitimate value, which is the
exact seam the header of this file was written about, reproduced in a scenario
added to catch it.
One query per tick now, returning done/blocked/per-node in a single row, and
unreadable samples are COUNTED rather than silently treated as zeroes. Fewer
than ten usable samples is NORUN: a sampler that barely looked must not be able
to describe itself as a fleet that was idle.
Co-Authored-By: Claude Opus 5 <[email protected]>
`mission_workspace::missions_root()` is now the only place that answers "where
does mission state live". It had fragmented into five: this function, private
`env::var("CLAWMATES_MISSIONS_ROOT")` copies in security_scan, benchmark_runner
and mission_outputs, and a hardcoded `MISSIONS_HOST_ROOT` const in
mission_runtime that read no env at all.
They agree on the deployed value, so nothing has broken. The risk is entirely
in what comes next: anything that sweeps or reclaims this tree has to be
sweeping the same tree the writers use, and five definitions cannot promise
that — a reaper written against one would silently leave the others' directories
behind forever, which is how the orphans got there in the first place.
A source-walk test fails any module outside `mission_workspace` that reads the
env var itself.
Co-Authored-By: Claude Opus 5 <[email protected]>
The re-run reported FAIL-NORUN "the burst did not finish in 1800s". The fleet
was fine — 3 of the 16 missions were still in `draft`. Each PATCH-to-running is
an ssh plus a `docker run curl`, and 16 at once does not reliably land; the
response was going to /dev/null, so a launch that never happened spent the full
timeout looking like a platform stall.
That is precisely the swallowed-error shape this file was written to catch,
committed inside the file itself. Launches are now verified against the mission
rows, retried once for the stragglers, and reported as "the burst never
happened" rather than as a timeout — a scenario that did not run must not be
able to describe itself as a slow one.
Co-Authored-By: Claude Opus 5 <[email protected]>
The capacity harness scenario, on its first full run, caught what it was
written to catch:
capacity: architect peaked at 6 of 6 slot(s)
FAIL capacity: 'morpheus' peaked at 3 concurrent VM(s) with only 2 slot(s)
capacity: tank peaked at 6 of 6 slot(s)
PASS capacity: the over-capacity missions QUEUED
PASS capacity: all 16 queued/placed missions completed
`capacity_of` inferred the host's own footprint by subtracting the VMs' FULL
8 GiB claim from observed usage — which assumes they have already consumed it.
A VM booted seconds ago holds about an eighth. On morpheus (31757 MiB total,
4314 MiB idle, 2 slots) with 2 young VMs at ~6314 MiB observed, the inference
6314 - 16384 goes negative, clamps to the 2048 floor, and invents 2266 MiB —
exactly enough for a third VM on a two-slot node.
The footprint is only honestly MEASURABLE when nothing is committed, so
remember it then: `nodes.mem_baseline_mib`, sampled by `survey` whenever it
observes an idle node with fresh health. When VMs are committed, take the
LARGER of the remembered reading and the old inference — a node that was once
idle at 4 GiB and is now running a 20 GiB build must not be scored as idle,
which would be the same over-commit arrived at from the other direction. Both
directions have a test; the second is the one that would otherwise rot.
Raising HOST_BASELINE_FLOOR_MIB would have made this one node's numbers pass
and drifted the moment the fleet changed shape.
Co-Authored-By: Claude Opus 5 <[email protected]>
Phase 1 shipped placement-at-phase-launch and a queue made of
`start_pending_phases` leaving a phase `pending`, both deployed unproven under
load — the exact condition this project keeps getting burned by: the code is
right, the system is wrong, and nothing errors.
`capacity` launches `slots + 2` microVM missions simultaneously and asserts two
things. That no node ever exceeds the slots `vm_placement` gave it — overcommit
does not fail loudly, it swaps, and every mission on that node gets slow rather
than dead. And that the excess QUEUES: a burst that drops the extras and one
that wedges them both look identical to any check that only reads the end
state. `capacity_blocked_since` is cleared the instant a phase is placed, so
the evidence only exists mid-flight; the scenario samples while it runs.
Capacity comes from `/api/fleet/capacity`, never recomputed here — a bash copy
of the slot arithmetic would drift from the scheduler and then agree with
itself. A burst that does not exceed capacity is reported NORUN, per rule 3.
`drain-midmission` drains the node phase 0 ran on, before phase 1 is placed,
and asserts phase 1 lands elsewhere AND still reads phase 0's file. That is the
test of the affinity decision: mission state lives on the gateway, so
re-placement is free — if it were not, this would either strand the mission or
silently lose the earlier work, and "silently lose" is what a status-only check
calls success. The node is restored before any assertion runs, so a failure
cannot leave the fleet permanently one node smaller.
Smoke-checked at CAPACITY_BURST=2: sampling, spread and completion all report,
and the queue check correctly returned NORUN rather than a green tick.
Co-Authored-By: Claude Opus 5 <[email protected]>
Pulled forward from the observability phase because the capacity harness
scenario needs it. A test that recomputed the slot arithmetic in bash would
drift from `vm_placement` and then agree with itself while the scheduler did
something else — the same shape as every silent-success bug in this codebase.
Returns `survey()` + `rank()` unmodified, and keeps `unfit` as its own list:
"the fleet is full" and "we could not read the fleet" send an operator to
different places, so they must not be summed into one number.
Co-Authored-By: Claude Opus 5 <[email protected]>
The research scenario passed 4/4 and the log underneath it said:
phase_summarizer: ... failed: anthropic 400 Bad Request: "Your credit
balance is too low to access the Anthropic API"
`phase_summarizer` and `mission_refiner` each built their own reqwest POST to
the Messages API with `x-api-key: $ANTHROPIC_API_KEY`. No audit of `.complete(`
call sites could have found them — they never touched a provider — so every
phase summary and every mission-brief refinement on this deployment had been
failing against an empty account while the phases themselves ran fine. The
summarizer even persisted an error row per phase, which is why nothing ever
retried loudly enough to notice.
Both now go through `subscription::complete_with_fallback`, so they inherit the
subscription-first credential choice, the 429 backoff, and the opus -> haiku ->
glm chain. The summarizer records the model that ANSWERED in
mission_phase_summaries.model rather than the one it asked for.
The guard is a source WALK, not a file list: any .rs under cm-api/src that
mentions the Messages API host or `x-api-key` fails the test. A hand-listed set
of files is exactly what let these two hide.
Co-Authored-By: Claude Opus 5 <[email protected]>
`Runtime::judge` returns "governor unreachable (fail-open)" whenever the
provider never answers, and the door caller drops `reason` on every allow —
so a judge model that is rate limited or uncredited turns the governor into a
rubber stamp with nothing anywhere saying so. Fail-open stays (a governor
outage must not halt agents), but it is now loud.
Found while removing the metered key as a dependency: the governor reads
CLAWMATES_JUDGE_MODEL, which was `claude-opus-4-8` — a model that is 429 on
this deployment's subscription. gw-04's .env now points it at `glm:glm-4.7`,
matching CLAWMATES_VALIDATOR_MODEL: funded separately, uncapped, and a
different family from the agent it judges.
Co-Authored-By: Claude Opus 5 <[email protected]>
Two changes so an empty metered account stops being a platform outage.
1. `build_provider` prefers the subscription token over ANTHROPIC_API_KEY.
A bare model name resolves to whatever this returns, so making it the
subscription means no server-side call can reach the metered key by
construction — rather than by a source-grep test that already missed four
call sites once. The metered key remains a fallback and now warns loudly
when it is the one in use; boot no longer requires it at all.
2. `complete_with_fallback` walks a declared chain when a model has no
capacity: opus -> haiku -> glm:glm-4.7 by default, overridable via
CLAWMATES_MODEL_FALLBACK, empty to disable. Measured on gw-04 today: opus
and sonnet return 429 on the subscription while haiku, GLM and Kimi all
return 200, so a capped window no longer means "the planner is gone".
The chain returns the model that ANSWERED, and every caller persists it —
mission_plan_proposals.author_model, mission_team_proposals.author_model, and
the swarm's step role. A plan drafted by the third link and filed as an opus
plan is a silent quality change, which is the failure shape this project keeps
paying for. Two negative controls hold the design: the chain never retries the
model that just failed as its own fallback, and it steps down ONLY for a
capacity failure — walking it on a malformed prompt would ask three models the
same bad question and report the third one's confusion.
Co-Authored-By: Claude Opus 5 <[email protected]>
The test that was supposed to prevent this grepped for the literal
`runtime.complete(` and passed while the phase planner (`mission_plan.rs`),
both swarm calls, and a second enhance path in `claws.rs` still billed the
pay-as-you-go account. They spell the receiver `state.runtime` or wrap the
call across lines, so the receiver name was never the thing to match. The
test now matches the METHOD, and covers all five files.
`complete_or` gains the rule that makes it safe to apply everywhere: a
`name:model` spec is an operator's explicit provider choice — the swarm
worker model is configured exactly that way — and is passed straight to
`Runtime::resolve_provider` untouched. Only a bare name is ambiguous, and a
bare name is precisely what resolves to the default provider. Hijacking a
chosen Kimi or GLM model onto Anthropic would be the same silent-substitution
bug pointed the other way.
`validator_preflight` and the evaluator judge keep calling the runtime
directly, on purpose: both exist to exercise the CONFIGURED spec.
Co-Authored-By: Claude Opus 5 <[email protected]>
The retry landed and still failed: all four attempts returned 429. A bare
16-token probe with the same token, straight from gw-04, also returned 429
with `x-should-retry: true` — the Claude Code subscription itself is limited
right now, and no amount of backoff inside one HTTP request will outlast it.
So stop pretending it is a server bug. New `ApiError::Unavailable` → 503,
carrying the one sentence the operator can act on ("clears on its own; try
again shortly"), instead of an opaque `internal error` that sends them into
the logs. The harness now prints the response body rather than the generic
"the planner produced no usable proposal", which is what hid both walls —
first the credit balance, now this.
Co-Authored-By: Claude Opus 5 <[email protected]>
Moving the roster and planner onto the subscription removed the credit wall
and revealed the next one: the harness went from
`400 credit balance too low` to `429 rate_limit_error`. A one-shot proposal
call had no retry — there is no retry convention anywhere in cm-llm — so a
limit that clears in seconds killed the "propose a team" button outright.
Four attempts, 2/8/20s backoff, and only for errors that can actually clear:
429/5xx/transport. A 400, 401 or 404 returns immediately, because retrying
those is a 30s hang ending in the identical message, which reads to an
operator as a stall rather than a bad request.
Co-Authored-By: Claude Opus 5 <[email protected]>
The roster planner died with `400 — "Your credit balance is too low to access
the Anthropic API"` while every mission on the same machine kept running. Two
Anthropic credentials reach this server and they bill differently:
`ANTHROPIC_API_KEY` (sk-ant-api, metered, runs out) and the Claude Code
subscription token (sk-ant-oat) that every VM already uses.
`Runtime::complete` with a BARE model name — "claude-opus-4-8" — resolves to the
default provider, which is the metered key. Three server-side callers did that:
the roster planner, the Master Planner, and the claw enhancer. Missions were
never affected because `mission_runtime` deliberately sends only the
subscription token into a guest; the server had no equivalent rule.
`subscription::complete_or` is now that rule, and it is the ONE place a
subscription token becomes a provider — `evaluator::subscription_judge` had its
own copy, and two of them is how one ends up with a prefix check the other
lacks.
The `sk-ant-oat` prefix is checked rather than the variable name trusted: an
API key pasted into the OAuth slot would authenticate, work, and bill the
metered account — the same failure again, discovered weeks later.
`web_search` is carried explicitly rather than defaulted. The Master Planner and
the claw enhancer both pass `true`, and a helper that quietly dropped it would
have taken web search away from two features while every test still passed.
`validator_preflight` deliberately keeps `Runtime::complete`: it probes whatever
validator spec is configured (today `glm:glm-4.7`), and forcing it onto Anthropic
would make it prove the wrong thing. A test pins both halves — no other
server-side caller may regress to the metered key, and preflight must keep
probing the configured spec.
258 lib tests.
Phase 1b: wires the capacity model from 3a2d76a into the launch path, and turns
the existing pending-phase loop into the queue.
Placement moves from mission launch to PHASE launch. A node chosen at launch is
chosen once, minutes before the first VM boots and hours before the last — and
re-placing between phases is free, because mission state lives on the gateway
checkout and every VM is inject -> run -> collect -> destroy. Pinning early
bought nothing and cost the ability to react to a node filling or draining
mid-mission. One call site serves both the solo and composed paths so they cannot
disagree; the composed worker reads `missions.target_node_id`, which placement
writes before dispatch.
QUEUEING, with no new machinery: a phase with no admissible node keeps its
`pending` status and creates no `topology_runs` row. `start_pending_phases`
retries every 10s — that loop already was a queue; nothing downstream ever sees a
run that did not happen.
The risk that creates is the one this codebase keeps paying for: a phase waiting
for capacity looks exactly like a phase nothing is working on. So the wait is
RECORDED, not merely logged — migration 0073 adds `capacity_blocked_since` and
`capacity_note`, stamped once and preserved across retries so the wait is
measured from the first refusal. It is bounded at two full turns: a fleet that
frees will free within one, and a phase that waited two hours must say so rather
than sit pending forever looking like a bug.
Mission launch still fails when NO node could ever run the backend — that is not
transient, waiting will not fix it, and `microvm-negctl` asserts such a mission
stays `draft`. Capacity refusals are transient and queue; capability refusals are
not and fail. The two are separate variants precisely so they cannot be confused.
257 lib tests, 20 binaries.
Phase 1a of the fleet-intelligence plan: the arithmetic and the inputs. Nothing
is wired to it yet; the launch path still picks `capable.first()`.
Placement has been `ORDER BY last_seen DESC` + `.first()` — the most recently
heartbeated node. Among healthy nodes all heartbeating every 5s that is
arbitrary, and it consults nothing about load, so two missions launched together
land on the same machine. It did not matter while tank held the only rootfs
image. All three nodes serve `claude` as of today.
THE correctness point, and the reason this is not a sort change: a VM that booted
30 seconds ago holds a fraction of its 8 GiB claim, so `mem_pct` reports a
sold-out node as nearly idle. `capacity_of` takes the WORSE of observed usage and
committed usage. The negative control pins it with the measured case — tank at
60 GiB total / 12 GiB observed / 5 VMs booted: utilisation alone says 5 more fit,
the node has room for 1. Booking those five is a node in swap, which slows every
VM on it together.
Commitments are unioned BY IDENTITY, never added: `vm_list` reports booted VMs,
`nodes::pinned_microvm_phases` reports phases chosen but not yet booted (a
window of seconds in which a real 8 GiB claim exists that no node can report).
The deterministic `vm_id_for` is what lets the same phase be recognised in both —
counting it twice would shrink the fleet by the number of phases starting.
`EvalRow::headroom()` finally gets a caller. It was written with the doc comment
"for placement ranking" and has had zero callers since. It is a TIEBREAK, not a
gate: ranking is slots first (spread, don't stack), then live headroom, then node
id so the same fleet state yields the same answer twice — which `last_seen DESC`
could never promise.
Fail-closed per house convention: draining, stale health (>30s, tuned just above
the 20s offline sweeper), and an unanswerable `vm_list` are all INELIGIBLE rather
than low-scoring. Stale Beszel metrics are the one exception — they demote a node
to zero headroom instead of excluding it, because they only ever break ties.
`FleetAtCapacity` and `FleetUnreadable` are separate variants with a test
asserting the second never says "at capacity": an operator sent hunting a load
problem that is really a dead daemon wastes the outage.
Also names the two nodes that were both called "New node" (tank, morpheus) — a
capacity report naming two machines identically is one nobody can act on.
257 lib tests.
Four places copy a mission checkout so a ROOT command can run against it without
touching the live tree: the judge, the benchmark runner, the on_green_tests gate,
and — until now — the judge again, with its own implementation predating the
shared one.
`evaluator_tools::Sandbox::for_checkout` now builds through `root_copy::RootCopy`.
Same packer, same exclusion list, same reasoning in one place.
It needs the copy to OUTLIVE the handle, because the judge has not run when
`for_checkout` returns and a firing `Drop` would delete the tree out from under
it. That is `into_workdir`, a method rather than a `mem::forget` at the call
site: the transfer of cleanup responsibility is then visible in the type instead
of implied by a leak. `Sandbox::purge` remains what actually clears it, since
only a container running as root can remove the root-owned `target/`.
What did NOT move: `pack_dir` in the microVM inject/collect path. That marshals a
tree to and from a guest over vsock — a transport, not a host-side copy — and
folding it in would merge two things that only look alike.
249 lib tests.
Canary first, production second — the point of pinning is that the upgrade is a
decision, and the point of `canary-claude` is that the decision has evidence.
Taken for 2.1.225's fix to a transient 401 that replaced a long-lived
CLAUDE_CODE_OAUTH_TOKEN with a short-lived one and broke HEADLESS sessions until
restart. Our sessions are headless and our VMs are per-mission, so "until
restart" reads as a failed phase.
The 2.1.225 workspace-trust prompt does NOT apply: `--help` states the dialog is
skipped in non-interactive mode (`-p`, or stdout not a TTY) and we satisfy both.
Read from the CLI in a booted 2.1.226 VM rather than inferred from the changelog.
Verified on the production path after the rebuild: guest kernel 6.1.128,
delegation to a subagent, `--settings` stop gate installed, judge independent
(glm-4.7), single writer. 6/6.
`rootfs-canary-claude.ext4` is left on tank as the mechanism for the next
candidate, not as a leftover.
First canary run failed: phase failed, nothing delivered, and the streamed log
said exactly why — "Failed to authenticate. API Error: 403 api.anthropic.com is
not on the egress allow-list".
Not a 2.1.226 regression. `canary-claude` was added to the server's credential
map and not to the node's `provider_hosts`, so the VM booted with a valid
subscription token and a door that only opened onto the forge. The fail-closed
branch was working correctly: a backend nobody taught that function about
reaches no model API, deliberately, so it cannot silently borrow another
provider's door.
Both maps now name it, each pointing at the other, with a test asserting the
canary reaches the same provider as `claude` AND that unknown backends still
resolve to nothing.
Worth noting what made this a five-second diagnosis instead of an afternoon: the
live log streaming built earlier today. The failure was a 403 inside a microVM
that no longer exists, and its reason was sitting in the run's checkpoint.
Claude Code 2.1.223 -> 2.1.226 is worth taking (2.1.225 fixes a transient 401
that replaced a long-lived CLAUDE_CODE_OAUTH_TOKEN with a short-lived one and
broke HEADLESS sessions until restart — which for us means a failed phase). But
the image every mission uses is not the place to find out whether a new CLI
still delegates, still accepts `--settings`, and still finishes.
`canary-claude` is a real rootfs built from the candidate version, credentialed
identically to `claude`, so a mission can exercise it through the production
path: egress, stop gate, delegation, delivery, streaming. Testing a new CLI
against a different provider would not be testing the thing we are about to ship.
Named explicitly rather than matched on a prefix. An unrecognised backend must
still be refused at launch — that is what `backend_can_run_a_mission` and the
harness's `microvm-negctl` scenario assert — and loosening the credential map is
exactly how that guard gets softened by accident. A test pins both halves.
Already cleared by direct measurement in a booted 2.1.226 VM, before this:
- `--settings` and `--agents` still exist
- the workspace trust prompt added in 2.1.225 does NOT apply: `--help` states
the dialog is skipped in non-interactive mode (`-p`, or stdout not a TTY).
We use both.
Composed missions streamed ZERO bytes while solo missions streamed fine. Same
executor, same command, same guest — `HubVms::run` is a straight passthrough —
and the node logged a tail starting for all five graph nodes against the correct
outer run id, with no errors. The bytes simply were not there at the end.
Two writers, one column. `fleet.rs` appends live output under `checkpoint.log`;
`topology_runs::checkpoint` wrote `SET checkpoint = $2`, replacing the whole
object. A composed run checkpoints after EVERY graph node, so each node's
progress silently erased the log written during it. A solo run has no second
writer, which is exactly why it looked like it worked.
Now merged with `||`. The keys are disjoint, so the progress object still wins
for everything it owns.
I was wrong about the cause twice before finding this. First I blamed the guest
agent's serial accept loop — real, fixed, and not this. Then I blamed pipe
buffering racing the abort at turn end — plausible, and the drain fix is right on
its own merits, but composed still streamed zero afterwards, which is what ruled
it out. The thing that actually located it was noticing solo and composed differ
by a WRITER, not by a code path.
Composed runs streamed NOTHING while solo runs streamed fine — same code path,
`HubVms::run` is a straight passthrough, and the node logged a tail starting for
all five graph nodes with the correct outer run id. The difference was timing.
`claude -p ... | tee` makes stdout a PIPE, so the CLI block-buffers and flushes
at EXIT. The most valuable output — the agent's summary of what it did — arrives
in the instant the turn ends. The node aborted the tail the moment `handle_op`
returned, so that flush was a race: a solo turn (minutes long, output already
flushed by size) won it and streamed 337 bytes; each node of a composed run
(~20s) lost it and streamed zero.
The tail now DRAINS. A flag is set when the turn returns, and the loop exits only
after a pass that read nothing new — checked AFTER a read, never before one,
because exiting on the flag alone would drop exactly the bytes this exists to
capture. Bounded by a 20s timeout with the abort kept as a backstop rather than
the mechanism, so a VM that stopped answering cannot hold the task open.
Worth naming: 5 tails started, 5 logged cleanly, 0 bytes arrived. Every
individual step reported success and the feature did nothing — the same shape as
the empty Live tab this whole thread began with, one layer down.
The other half of the merge button. Merging told you the branch went in; nothing
checked that what came out still worked.
Verified BEFORE publishing, not reverted after. `merge_locally` and
`push_merged` are separate functions so the caller can run the project's tests
between them, which means a merge that breaks the base is simply never pushed —
`main` is not broken for however long it takes someone to notice. A test asserts
`merge_locally` contains no push, because the moment it does, verification
becomes after-the-fact and the guarantee is gone.
Outcomes, all reported to the operator rather than swallowed:
Passed -> published
NoSuite -> published, and SAID so; a repo with no tests is a fact about the
repo, not a pass
Failed -> not published, exit code reported, branch untouched so it can be
fixed and merged again
CouldNotRun -> not published. Fail closed: a suite that could not run has not
passed, and publishing on "we could not check" is how a green
main stops meaning anything.
`verify_tests` runs `cargo test` as ROOT in a container, so the merge workdir
ends up holding a root-owned `target/` the server (uid 65532) cannot delete —
the same leak found three times today. Purged through the container before the
ordinary cleanup.
248 lib tests.
First live test of the streaming path: mission passed 6/6, `checkpoint.log` was
0 bytes, and the node logged nothing at all.
`stream_vm_log` treated "no progress" as "the turn finished writing". But the
guest's `tail` reports EOF after every idle window, and the FIRST idle window is
always the one before any output exists — the VM is still booting and the CLI
still starting. So the tail returned `at == 0`, the node concluded the turn was
done, and it stopped seconds into a run that then went on for minutes.
The abort is the terminator, not idleness: the caller already aborts this task
when the exec returns, so waiting cannot outlive the turn. No-progress now sleeps
and retries instead of returning.
Also logs when a tail STARTS. The bug was invisible in exactly the way this
session keeps finding: silence on the success path, silence on the give-up path,
and an empty Live tab that looked identical to a feature nobody had wired.
Method note, since it cost time: I tried to confirm the deployed binary by
grepping it for `vm_out` and found zero — then found zero for `pty_out` and
`vm_exec` too, in a binary whose PTY streaming demonstrably works. Binary-grep is
not a reliable presence test for these literals; `stream_vm_log` and `tail of`
being present is what actually showed the code had shipped.
The Live tab showed nothing while a turn ran, and the agent's own account of it
went to stderr on the node and nowhere a user could reach. This is the path that
carries it.
The blocker was the guest agent. `fcagent` handled one connection at a time,
inline, so during an hour-long turn the VM accepted nothing — which is why every
existing probe (subagents, stop-gate blocks, cap) runs AFTER the turn rather than
during it. It now spawns a thread per connection, wrapped in `catch_unwind`
because this process is pid 1: a panic used to take the accept loop with it, and
an unbootable VM is a far worse outcome than a missing log. A failed spawn logs
and keeps accepting rather than dropping the listener.
PROVED against a live VM before building on it, since "sound reasoning about this
system" and "measurement" have diverged repeatedly today. Patched rootfs, booted
under Firecracker, ran an 8s exec and a concurrent tail:
exec took 8.0s ok=True
+0.0s 'line1\nline2\n' +1.2s 'line4\n' +3.2s 'line6\n' +6.0s 'DONE\n'
VERDICT: CONCURRENT — tail returned data before exec finished
The rest is the pattern the terminal already uses. New `tail` op streams a file
by OFFSET (so a dropped link resumes instead of replaying, and the tail always
terminates — one that never returns pins a thread for the life of the VM). The
node follows the log alongside the turn and pushes `Uplink::VmOut { run_id, at,
data }` over the WebSocket it already holds, mirroring `PtyOut`. The server does
what `PtyOut` deliberately does not: it APPENDS to the run's checkpoint as well
as fanning out, because a terminal has no history worth keeping and a mission log
is the record of what the agent did. `run_events_sse` emits the new bytes as
`step` events, which the live pane already renders — no frontend change.
The turn is `tee`d, not redirected: the file feeds the live stream and stdout
still becomes `VmOutcome::summary`. A redirect would have produced a live view
and an empty summary, which is the same green-and-empty shape as the bug this
fixes. Tested, along with the log living outside the collected tree so it never
lands in a user's delivered diff.
246 lib tests, 20 binaries; node and fcagent build clean.
