774f17d1949b772c7c10187af6f84f7ca4fae240
447
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar SobhandClaude Opus 5
|
a33dbdcdc3 |
feat(missions): W1/#13 — let a model author the mission's phases
The last unstarted item from the missions-as-workflows plan, and the other half
of Slice 5: that one lets a model size the TEAM, this lets it decide what the
work IS.
Every mission's phases come from one of five hand-written recipes in
`templates/workflows/*.toml`, chosen by `template_kind` before anyone saw the
mission. That is the "do it this way: 1, 2, 3" over-specification that makes a
capable model follow a worse plan than it would have chosen. The recipes stay —
they are still the default for a mission nobody proposes a plan for, and the
fallback when a proposal is refused.
Same three verbs and the same review gate as the roster, deliberately: propose
and decide are separate because only the second changes a mission, and a second
shape would be a second thing to get right. Approving REPLACES the phases (a
plan is an answer to "what is this mission", not an addition to one), draft-only.
GROUNDED IN WHAT THE PLATFORM ACTUALLY READS, which is the part that makes this
more than a copy. `phase_config::KNOWN_KEYS` already names every phase-config key
and the code that reads it — the registry built after `task` sat unread through
every mission. A plan is validated against it, so a model cannot propose a phase
whose settings nothing will act on: the failure that registry exists to EXPOSE is
one this path cannot create. Phase kinds are checked the same way, because an
unknown kind does not error — it falls through to the catch-all purpose and runs
as a generic phase that looks like it worked.
TWO THINGS THE WORK ITSELF FOUND, both the same shape:
- `done_when_check` — the stop-gate key added earlier today — was never
registered in `phase_config`, so every mission that set it has been logging
it as an unknown key. Found by a test written for a different purpose, which
is the registry doing exactly its job. Now registered with its reader.
- `done_when` and `max_iterations` are COLUMNS promoted out of config by
`missions::create`; the evaluator sweep filters on the column in SQL every
tick. My first insert wrote the config blob alone, which would have stored a
plan's completion condition where nothing judges it. NEGATIVE CONTROL run:
binding NULL instead of the promoted value fails
`an_approved_plan_replaces_the_missions_phases`.
`order_idx` comes from the array's own order rather than a field the model sets:
two sources for one fact is how a plan ends up with two phase 0s, and order_idx
is what `start_pending_phases` sequences on.
MAX_PHASES is 4 and the prompt argues for one. Each phase is a full agent run in
sequence, and splitting one change into plan → implement → test is the documented
anti-pattern — a single agent doing all three keeps the context that makes the
later steps good.
543 tests pass, clippy clean. Migration 0072.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
a48d78f8eb |
test(missions): a composed node is offered the same help as a solo one
Every composed run so far reports `subagents: 0`, and the honest question is whether that is the tasks being small or the capability being absent. It is the former, and this is what says so: a composed node's task text is built by `microvm_turn_executor` and then wrapped by the SAME `vm_prompt` inside `run_inside`, so one prompt builder serves both paths and both carry the `Agent` tool offer and the `verifier` / `explorer` roles. Asserted rather than left to code reading, because if someone gave composed nodes their own prompt without the offer, the difference would show up only as a count nobody was watching — and "the graph fanned out but no node did" is indistinguishable from "no node needed to". The roles are read from `agent_definitions()` rather than spelled out, so adding a role without mentioning it in the prompt fails here instead of shipping a role the lead is never told about. 535 tests pass, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
742724e53c |
feat(fleet): Kimi as a microVM backend — the URL settled by measurement
The base URL took three measurements to find, and the first two were wrong in instructive ways. `api.moonshot.ai/anthropic/v1/messages` EXISTS and speaks the protocol — it answers with Moonshot's own structured error rather than a 404. It also rejects an `sk-kimi-` key, because it belongs to the platform.moonshot.ai account namespace. Two endpoints that both "work" for different accounts is precisely the shape that makes a guessed URL look like a broken key, and it is why this was refused rather than guessed for as long as it was. The Kimi CODE service is the one an `sk-kimi-` key belongs to: `POST https://api.kimi.com/coding/v1/messages` returns a real Anthropic Messages body — `msg_` id, `content` blocks, a `thinking` block with a signature. So `ANTHROPIC_BASE_URL=https://api.kimi.com/coding`, WITHOUT the `/v1`: Claude Code appends `/v1/messages` itself, and `/v1/v1/messages` would 404 in a way that reads as a broken image rather than a bad URL. Two more measured, each otherwise a silent failure at the first turn: `Authorization: Bearer` is accepted (so ANTHROPIC_AUTH_TOKEN is the right injection channel), and a `claude-*` model id is ACCEPTED AND ANSWERED — Kimi maps it onto `kimi-for-coding` exactly as z.ai does, so no ANTHROPIC_MODEL override is needed. Claude Code rather than Moonshot's own `kimi` CLI, deliberately. The mission harness is Claude-Code-shaped throughout: `--agents` JSON roles, the verifier's tool allowlist, the `Stop` hook behind the completion gate, the per-subagent transcripts counted as delegation evidence. `kimi` has none of those flags — its equivalents are TOML files and markdown agent dirs — so using it would mean a second executor with its own untested failure modes. TWO STALE MAPS, caught by the rootfs harness refusing to bless the image: both `fc-build-rootfs.sh` and the node's `required_cli` expected backend `kimi` to contain Moonshot's `kimi` binary. That assumption predates the measurement, and it failed a rootfs that was correct. Both now say `claude` for glm and kimi alike — the binary is the same in all three images; only the endpoint differs. Egress for `kimi` is `api.kimi.com` alone: not moonshot.ai (wrong namespace), not z.ai, not Anthropic. Asserted both ways, like the other two. The image and rootfs are built on tank and the rootfs passes all four checks (boots, git, writable /mission, `claude --version`). 534 tests, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
d3a53e7bf1 |
fix(fleet): a VM reaches its OWN provider and no other, measured not assumed
The GLM backend works — and proving it produced a better boundary than the one I shipped an hour ago. WHAT THE FIRST GLM MISSION SHOWED. It completed, and the delivered file said the model was "claude-opus-5". The node's egress log said the VM had dialled `api.anthropic.com` five times before `api.z.ai`. Either reading alone is consistent with a "GLM backend" that silently runs Anthropic — the exact silent-success shape this project keeps closing — so I did not accept either. THE ABLATION, run on tank rather than reasoned about: deny `anthropic.com` at the proxy and run the same mission again. It **completed**, dialling only `api.z.ai`. So the completions genuinely come from z.ai; Claude Code's calls to anthropic.com are its own telemetry, not its model traffic. And that same agent — served exclusively by z.ai, with Anthropic unreachable — still described itself as "Claude Opus 5 (1M context)". **A model's account of which model it is has no evidential value.** The proxy's log of which host it dialled does. This is the `uname -r` lesson again in a new place: ask the infrastructure, not the agent. So the allow-list is now PER BACKEND rather than a union: a `claude` VM reaches Anthropic and the forge, a `glm` VM reaches z.ai and the forge, and neither can reach the other's endpoint. A union was defensible when it was one host; once the measurement showed a GLM VM never needs Anthropic, keeping it would mean a credential mix-up upstream could still put one provider's secret on another provider's wire. Now it fails at a closed door instead. An unknown backend gets the forge and NO model API — it cannot run anyway, and borrowing somebody else's door is the failure this split prevents. An explicit `CLAWMATES_FC_EGRESS_ALLOW` still wins outright: an operator who set it drew a boundary on purpose. `DEFAULT_ALLOW` is deleted rather than left beside the new function, so there is one answer to "what may a mission reach" and not two. 534 tests pass, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
f7f3dfe495 |
feat(fleet): GLM as a real microVM backend, and per-role models for claws
Three threads, all of which end at the same place: a mission whose verifier does
not share a model with the coder it reviews.
**GLM has a credential contract now.** `microvm_credential_for` returned one env
var name, which quietly assumed every provider reads its secret from the same
place Anthropic does. It returns a `Credential { source, target }` instead —
z.ai's key lives in the server's `ZAI_API_KEY` and Claude Code reads it as
`ANTHROPIC_AUTH_TOKEN`, and collapsing those two names is what forces a guess at
the other end. A wrong guess here sends one provider's credential to another
provider's endpoint.
`images/agent-glm` is the same CLI at the same pinned version as `agent-claude`
with `ANTHROPIC_BASE_URL` baked in. The split is deliberate: the ENDPOINT is a
property of the image, the CREDENTIAL is a property of the turn. That makes the
dangerous mix-up unrepresentable — a GLM VM cannot be handed an Anthropic
subscription token, and a claude VM cannot be pointed at z.ai. Asserted both
ways, because "the GLM VM must not carry CLAUDE_CODE_OAUTH_TOKEN" is the
property that costs a credential if it ever stops holding.
Kimi stays refused. `KIMI_API_KEY` is set and Moonshot serves an
Anthropic-compatible API, but I have not verified its base URL against the
running service, and this function is precisely where guessing a URL is
expensive. It becomes an arm the day someone measures it.
`api.z.ai` joins the node's default egress allow-list. A default that cannot
run the images we ship is a trap rather than a policy — the alternative is an
operator discovering it as a hung agent with no model access.
**Per-role models for claws** (migration 0071). `template_roles` had no model
column, so `mint_team_from_template` bound every role of every mission team to
one literal — a template whose whole point is an independent reviewer minted a
reviewer sharing a model with the coder. A role may now name its own; roles that
say nothing still take the mint's default, so every template written before this
behaves exactly as it did. The literal is now that default rather than a
hardcode.
**A harness scenario for the roster flow.** `verify-mission-delivery.sh roster`
runs the whole Slice 5 loop — planner proposes, human approves, mission runs —
and asserts the roster LANDED on the mission row rather than trusting the API's
answer. That distinction is not theoretical: the first live approval returned an
error while leaving the proposal marked approved.
Built and proven on tank ahead of the deploy: `clawmates/agent-glm:dev` reports
`2.1.223` and `BASE=https://api.z.ai/api/anthropic`, and
`fc-build-rootfs.sh … glm 8G` boots a VM from it that has git, can write
/mission, and answers `claude --version`.
533 tests pass, clippy clean. Migration 0071.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
75d09241fb |
fix(missions): the first real approval found two bugs the tests could not
Deploying Slice 5 and approving one roster in production broke it twice, in ways
528 green tests had nothing to say about.
**1. `jsonb_set` refuses a scalar.** A mission created through the API without a
`config` stores jsonb `null` — a scalar — and `jsonb_set` fails on it with
"cannot set path in scalar". The guard was `coalesce(config, '{}')`, which
protects against SQL NULL; this is a perfectly good JSON null of the wrong shape,
and coalesce passes it straight through. Every test wrote `'{}'::jsonb` because
that is what a test author types. Production types nothing at all.
**2. The approval was not atomic, and failing halfway is permanent.** The claim
and the mission write were two statements, claim first, so when the write failed
the proposal stood `approved` with nothing applied — and the partial unique index
then makes that state unrecoverable: no other proposal for that mission can ever
be approved. The mission ran solo with `team_engine` still NULL while its
proposal said otherwise.
`approve_and_apply` is now one transaction: claim, write, commit or roll back.
The type guard is `CASE WHEN jsonb_typeof(config) = 'object' THEN config ELSE
'{}'::jsonb END`, which answers the question that was actually being asked.
Both regressions are tested in the shape production had, and both NEGATIVE
CONTROLS were run rather than assumed:
- restore `coalesce` → `a_roster_applies_to_a_mission_whose_config_is_json_null`
FAILS with Postgres's own "cannot set path in scalar", the exact production
error.
- commit instead of roll back on a failed apply →
`a_failed_apply_leaves_the_proposal_undecided` FAILS with the proposal stuck
`approved`.
Worth stating plainly: the API returned 500 for that approval, so this was not
silent to the caller — but the row it left behind claimed the mission had a
roster it never received, and the mission then ran and delivered, which is the
shape that gets believed.
530 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
aa470091aa |
fix(missions): a bootable rootfs is not a runnable one
Found by looking at what the fleet actually reports, not by reasoning about it: tank's `capabilities.rootfs` is `["agent-terminal", "claude", "default"]`. Slice 5 offered that list to the planner as the menu of backends and validated proposals against it — so a roster naming `agent-terminal` would have been proposed, validated, approved and launched, and then failed at the agent turn, because `microvm_credential_for` has no contract for it and refuses rather than forward an Anthropic subscription token to an unknown endpoint. Refusing at boot is correct and is exactly the wrong PLACE: it is three steps and one human approval after the point where the answer was already knowable. The menu is now the intersection of "a node can boot it" and "a mission agent can authenticate in it", which is what the question meant all along. `backend_can_run_a_mission` derives from the credential contract rather than restating it, so a backend gaining one (GLM and Kimi, when B4.6's base-URL contract is settled) becomes proposable in the same commit that makes it runnable — instead of in a second list someone has to remember. 528 tests pass, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
1797669296 |
feat(missions): Slice 5 — let a model size the mission's team
`routes/planner.rs` has had Opus proposing rosters since the Master Planner
shipped, and none of it ever reached a mission: the proposal lived in React state
and died with the tab. A mission's shape came from a team template instead —
fixed roles, and every claw minted `claude-sonnet-5` from a literal in
`mint_team_from_template`. That literal is why no mission has ever run more than
one provider.
A roster is `(topology_kind, [(role, backend)])`, which is exactly what the
composed executor already consumes: `Roster::graph` builds a `TopologyGraph` with
the backend in `attrs`, and `MicroVmTurnExecutor` reads `attrs["backend"]` per
node. So a verifier on another provider's rootfs stops being a bolt-on and
becomes a graph node — the correlated-failure break the independent judge exists
for, one layer down.
Three verbs, and the split is the point. **suggest** asks the model and persists
the answer, changing nothing. **decide** approves (writes `config.roster` and
switches the mission to the composed engine) or rejects. A proposal is never
applied on arrival: a model sizing a team is a suggestion about how many VMs to
boot, and this codebase treats model output that costs money as evidence for a
decision, not the decision.
Fail-closed at every seam, because each of these otherwise surfaces much later
and much more expensively:
- a backend no ONLINE node can boot is refused when PROPOSED, naming the ones
the fleet actually has. Placement would refuse it too — at launch, after the
roster was approved and someone believed the mission would run. The model is
handed that same list in its prompt, so the usual case never arises.
- an invented `topology_kind` is refused, not defaulted. `parse_topology_kind`
defaults to hub-spoke, which is right for a template we wrote and wrong for a
string a model just produced: running a `pipeline` proposal as a hub-and-spoke
changes what every node sees and nothing would say so.
- the roster is validated BEFORE it is stored, so a stored proposal is always
one that could be approved; and again at approval, against the fleet as it is
then — a node can go offline in between.
- `MAX_MEMBERS = 6`. Each member is a whole VM, not a subagent, and a model
asked to size a team proposes twelve happily.
Two properties live in SQL rather than in the handler: at most one approved
roster per mission (partial unique index — two approved rosters are two answers
to "what shape is this mission", and the executor reads one field), and
decide-once (`WHERE status = 'proposed'`, so a double-clicked approve claims
nothing the second time). Both tested against a real database, including that the
second approval is refused by Postgres rather than merely losing a race.
NEGATIVE CONTROL, run rather than assumed: with the roster preference removed
from `composed_graph`, `an_approved_roster_outranks_the_template` FAILS — 3 nodes
from the template instead of the roster's 2. A stored roster that is silently
ignored at launch is precisely the shape this project keeps paying for.
Not closed: per-role models for CLAWS. `template_roles` has no model column, so a
ZeroClaw team still mints one model for every role. The literal is now a named
constant that says so and points at the roster path, rather than sitting inline
where nobody reads it.
527 tests pass, clippy clean. Migration 0070. Not yet exercised against the
deployed stack — the route has never been called with a live model.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
6991e21f94 |
feat(missions): the completion gate, moved into the agent's own loop
Every check this platform makes on a phase runs AFTER the agent has stopped: the
evaluator judges `done_when`, capture notices a coding phase delivered nothing,
and either verdict costs a whole new VM — a fresh boot, a fresh inject, and an
agent starting over with none of the context that got it that far. Meanwhile the
documented failure mode of a long-running agent is that it stops too early.
MEASURED FIRST, because the plan's chosen seam does not exist here. Probing every
hook name under `claude -p` (2.1.222, hermetic `--settings` file): `SessionStart`,
`UserPromptSubmit`, `PreToolUse`, `PostToolUse`, `SubagentStop` and `Stop` fire;
`TaskCreated`, `TaskCompleted`, `TeammateIdle`, `SessionEnd`, `Notification` and
`PreCompact` do not. The agent-teams hooks Slice 3 deferred are inert on our path
BY CONSTRUCTION — no team forms in print mode at all — so `done_when` could never
have been wired through `TaskCompleted` exit 2. `Stop` is the seam.
`vm_stop_gate` generates a POSIX `sh` hook installed via `--settings`, under
`/root/gate` and never under `/mission/repo` (anything there is collected and
arrives in the user's delivered patch). It refuses a stop when:
- the phase must deliver and the repository is untouched — asked as TWO
questions, since an agent that committed leaves a clean tree and an agent
that did not leaves HEAD alone; only both together mean nothing happened;
- `config.done_when_check` — a command the phase author wrote — exits nonzero,
in which case its OUTPUT is the feedback, not just "the check failed".
Deliberately mechanical. NOT the `done_when` verdict: that is an LLM judgement
made host-side by a different provider on purpose, and re-running it inside the
VM would put the agent's own environment in charge of grading the agent — the
correlated failure the independent judge exists to break.
THE CAP IS LOAD-BEARING. Without a ceiling a stuck agent is blocked, retries, is
blocked again, and burns the hour-long turn budget instead of failing visibly.
After 3 blocks the gate lets it stop, records that it gave up, and leaves the
verdict to the existing post-hoc path, which is unchanged.
PROVEN AGAINST A LIVE AGENT with the REAL generated artifacts, not a paraphrase:
- a read-only task → blocked 3 times with our exact message, released at
exactly the cap, and the agent took the escape hatch the message offers
("if the task genuinely requires no code change, say so explicitly") rather
than touching a file to satisfy the gate. It did not Goodhart it.
- a task that needs an edit → `blocks: 0`, log says `pass`. No false positives.
Two things that could fail silently, both closed. `--settings` is PROBED in the
image before use (`claude --help | grep`), because an unknown option is a hard
CLI error that would turn every gated phase into a failed one; a build without
it degrades to ungated and says so, since losing a check is better than losing
the work. And `stop_blocks` is reported out of the guest — `None` for no gate,
`0` for got-it-right-first-time — so a gate that never fires is distinguishable
from one that was never installed.
`require_changes` does NOT apply per node on a composed run: a graph's verifier
node is SUPPOSED to leave the tree alone, and a per-node gate would refuse its
stop three times for doing its job. `StopGate::per_node` drops it and keeps the
declared check. The phase-level rule still runs post-hoc against what the last
node collected.
An ungated phase's command is byte-identical to before, asserted by test — most
phases are gated, so the ungated path is the one nobody would notice breaking.
517 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
1d554396f4 |
fix(delivery): four failures from the #55 trace — auth, prompts, truncation, retry
All four were surfaced while tracing #55 and left open. Each one on its own is
small; together they are why a two-line git rejection took hours to read.
**1. `with_ambient_auth` failed open.** It matched one literal prefix,
`https://git.redclaw.dev/`, and returned the URL unchanged for everything else
with no log line. An `http://` remote, an explicit port, a different case in the
host, an ssh remote, a URL that already carried userinfo — all came back
unauthenticated and looked identical to success. It now returns `Authed`, which
carries the URL AND why no credential reached it, and recognises the forge in
every shape a remote can be written (host parsed with userinfo stripped BEFORE
the port, or `oauth2:token@host` reports its username as the host — the first
version of this function did exactly that and failed its own test).
**2. Nothing set `GIT_TERMINAL_PROMPT=0`.** So a credential-less URL did not
fail — git opened `/dev/tty`, and in a server container that surfaces as
`No such device or address`, several layers from the missing token. Now set on
every git invocation that can reach the network. And `push_url_for` refuses
outright when the URL is on OUR forge and unauthenticated: that push cannot
succeed, and letting it proceed only buys a symptom that looks like something
else.
**3. The truncation fix went to the wrong path.**
|
||
|
Omar SobhandClaude Opus 5
|
12147a1e01 |
feat(missions): Slice 4 — the two engines composed, with the file handoff proven
`team_engine='composed'` (the third name migration 0069 anticipated) runs a
mission as a durable ZeroClaw graph whose every node is a whole
Claude-Code-in-a-microVM session. Engine Z owns checkpoint/resume, cancellation
and per-node heterogeneity; Engine C owns shared context and cheap fan-out;
neither has the other's asset, which is why this is a composition and not a
compromise.
`MicroVmTurnExecutor` implements the existing `TurnExecutor`, so it inherits the
planners, the checkpoint, the stale-run recovery, `close_finished_phases`, the
evaluator, capture and delivery unchanged — the same trick `SubTopologyExecutor`
already plays with a heavy `run_turn`. Producer side emits ONE `queued` row
carrying the real graph and lets the worker claim it: the durability IS being
worker-driven, and the solo path's `tokio::spawn` has none of it. Still exactly
one `topology_runs` row per unit of work and one completion path — `finish()` is
now that one place, shared by every tier.
THE TRAP, solved and proven. A VM is inject → run → collect → destroy, so a
per-node VM with text-only handoff silently loses every file an earlier node
wrote: node 2 boots from the original checkout, sees nothing, and still reports
success. The mission's host checkout is the medium — every node injects from it
and collects back over it — and two properties make that safe rather than lucky:
`execute_resumable` is strictly sequential, so two VMs never write one directory;
and the vm id is deterministic per (phase, iteration, step), so a duplicate is
refused by the node ("vm already exists") instead of becoming a second writer.
NEGATIVE CONTROL, run rather than assumed: with `repo` swapped for a private
per-node workspace, `a_later_node_sees_an_earlier_nodes_files` FAILS with
`saw:[]`; restored, it passes. The `PhaseVm` seam exists for exactly this — it
models inject/collect through the real `mission_fs` tar path in milliseconds.
Two durability traps this tier walks into, both closed:
- `requeue_stale` fires at 180s on `updated_at`, and one node here can run for
an hour. `SubTopologyExecutor` keeps its parent alive from each leaf step;
there is nothing between the start and end of a VM turn, so the turn holds a
ticker that touches `updated_at` every 30s and aborts on drop. Without it a
healthy composed run is requeued mid-node and boots a second VM.
- the 15-minute stuck-run reaper asks "any step records since it was CREATED?",
which describes a healthy composed run as readily as a wedged one. Hence
`REAPABLE_TIERS` — worker-driven minus this tier. Reaping it would be #54 in
a different costume.
`on_launch` mints no team for a microVM mission, deliberately: claws in
containers are what a VM mission does not use. So `mission_orchestrator::
composed_graph` builds the shape from the team template directly — nodes, roles
and pattern, zero claws provisioned. Per-node `attrs["backend"]` and
`attrs["node_id"]` override the mission's, which is what makes a validator node
on another provider's image a first-class graph node; a malformed `node_id`
fails the node rather than quietly running it where the graph did not ask.
Refusals are recorded as a failed run, not returned as an error: `launch_phase`
is swept every ten seconds, so a returned error is a phase that retries forever
while the log repeats itself.
501 tests pass, clippy clean. NOT yet proven end to end: no composed mission has
run on the fleet, so the resume-after-a-killed-worker leg is argued from the DB
test and the step-numbering test, not from a real two-node run.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar Sobh
|
e31688bac5 |
fix(delivery): keep the TAIL of a push error — git prints its reason last
A failed push recorded two identical auth lines, a URL, and a branch name cut off mid-word. The reject reason was on the next line and the 500-char head clamp ate it, so the artifact preserved the noise and dropped the answer. That is what left #55 unresolvable: the evidence needed to distinguish "no credentials" from "non-fast-forward" had been truncated away. `evaluator_tools::clamp_output` already existed for exactly this — head AND tail with a byte count of what it dropped, on char boundaries so multi-byte output cannot panic. Reused rather than reinvented. Investigation notes recorded on #55. Two hypotheses were disproved by measurement: push credentials are rebuilt per push from GITEA_TOKEN + repos.clone_url and never live on disk (the clone-time scrub guarantees it, and every checkout on the host — including ones that pushed — has an identical credential-free origin), and neither of the two ways `with_ambient_auth` can silently return an unauthenticated URL applies here: the clone_url matches its required prefix and the token is non-empty in a container that predates the failure. 489 tests pass, clippy clean. |
||
|
Omar Sobh
|
66f730ad16 |
test(db): a regression net for the three-minute bug, with its negative control
The #54 fix had no test that could see it. Its defining property is that it only appears past 180 seconds, and `verify-mission-delivery.sh microvm` runs a 90-second mission — so the end-to-end harness written to catch silent failure was structurally blind to this one. A unit test asserting the allowlist's membership helps, but would not notice a NEW sweeper added without the filter. `crates/cm-db/tests/self_driven_runs.rs` tests the real SQL against a migrated database, in milliseconds instead of eight minutes: - a `microvm` and a `session` run, 30 minutes idle and still `running`, must be left alone by `requeue_stale` — that is the bug, in one assertion - a `team` run in the SAME state must still be requeued, so the fix is "sweep the right rows" and not "stop sweeping" - the worker must not CLAIM a queued self-driven row, which is what turned a healthy run into "missing or invalid graph" - the allowlist names only worker-driven tiers NEGATIVE CONTROL, run rather than assumed: with the tier filter removed from `requeue_stale`, `requeue_stale_leaves_self_driven_runs_alone` FAILS; restored, it passes. A guard that cannot detect the bug it was written for is decoration, and this project has shipped one of those before. 489 tests pass, clippy clean. |
||
|
Omar Sobh
|
d49acaed5e |
fix(missions): exclude build output from COLLECT too, not just inject
The other half of the same bug. The previous commit filtered `mission_fs::pack_dir` (the inject side) and left the guest's `op_get` tarring everything, so the re-run that proved the #54 fix — it survived 480s where it used to die at 210 — still lost its work to `vm_collect ... node timed out`. Two modules written, four subagents used, nothing delivered. `op_get` now takes an `exclude` list, sent by the host from `mission_fs::transport_excludes()` — the same list `mission_delivery` uses for the diff. Policy in one place, applied at both ends of the wire. Matched on directory NAME at any depth, so a workspace's per-crate `target/` dirs are all covered, with a test that plants a nested one and asserts it does not come along. Also proven by that run: the worker no longer kills a live microVM run. It ran 480 seconds straight through the 180s requeue window and the 210s mark where mission 019fd43e died, untouched. And `subagents: 4` — the team addendum did drive real fan-out this time, which is the first evidence the Slice 3 switch does anything. 483 tests pass, clippy clean. Still to prove: a >3-minute mission that actually DELIVERS. The collect fix is tested in isolation but has not yet carried a real mission's work back, and the guest agent needs rebuilding into the rootfs before it can. |
||
|
Omar Sobh
|
4efcde9d4f |
fix(missions): #54 — the worker was killing live microVM runs at 180 seconds
My hypothesis in #54 was WRONG, and it was wrong because I built it on a bad measurement: `grep -c 'microvm phase'` returned 0, so I concluded the completion log never printed and blamed the 15-minute reaper. The line was there all along, at 14:17:45. The real cause is worse. `requeue_stale` has NO TIER FILTER. A microvm run's `updated_at` is written once at insert and never again — it is driven by a `tokio::spawn` that owns it start to finish, and nothing in `microvm_executor` writes `topology_runs`. So at 180s the sweeper declared a perfectly healthy run stale and flipped it to `queued`; `claim_next_queued` (no tier filter either) handed it to the worker; `run_job` tried to parse the microvm graph placeholder, which `TopologyGraph` cannot deserialize; and it failed the run with "missing or invalid graph". Mission 019fd43e: run created 14:11:16, mission failed ~14:14:46. 210 seconds — the 180s window plus a tick. The agent went on working and finished at 14:17:45 with three modules written, by which time the phase was already dead and the VM was orphaned. A firecracker process was still alive 1h37m later. THE UNCOMFORTABLE PART: every microVM mission that appeared to work this session did so only by finishing inside three minutes. The 90-second ones dodged this. The harness scenario dodges it. Nothing about that was visible. `WORKER_DRIVEN_TIERS` (team, company, org, swarm, compare) is now the allowlist for all three sweep paths — claim, requeue, reap. An allowlist rather than a denylist so the next self-driven tier is safe by default instead of exposed until someone remembers the file. `tier='session'` had exactly the same exposure and is covered too. A unit test asserts microvm and session are NOT in it, next to the code that inserts them. Two more fixes from the same wreckage: - `destroy` reported `killed: pgid.is_some()` — true whenever there was a pgid to signal, whether or not anything died. It now sends the signal, polls /proc for the group leader, retries, and reports what it OBSERVED; `signalled` keeps the old meaning so "nothing to kill" is distinguishable from "it would not die". - the run-status update is now guarded with `AND status <> 'cancelled'`. An operator cancelling is a decision; this task reporting an outcome minutes later is an observation, and it must not overwrite one with the other. And the root cause of the collect timeout itself: `mission_fs::pack_dir` shipped `target/` in both directions. `mission_delivery` has excluded build output from the DIFF since day one; the TRANSPORT never knew. The host checkout was 9.4 MB of which 8.9 MB was `target/`, tarred and base64'd over vsock each way. `EXCLUDED_PATHS` is now one list shared by both layers, matched on directory name at any depth so a workspace's per-crate `target/` dirs are all covered. 483 tests pass, clippy clean. |
||
|
Omar Sobh
|
0d25a94a84 |
fix(missions): agent teams do not form in print mode — say so where it is set
MEASURED, against the CLI in our own image (2.1.223): with
CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1 and an explicit request to "spawn two
teammates", `claude -p` did the work with two SUBAGENTS, wrote both files, and
created no ~/.claude/teams/ directory at all. The docs allow for it — "Claude may
sometimes use subagents instead of creating a team" — and headless appears to be
always: the whole feature is described around an interactive agent panel, which a
print-mode session does not have.
So Slice 3's switch, as written yesterday, set a flag with no mechanism behind it.
The first team mission caught it, because the probe was built to look for teammates
rather than to assume them.
Corrected rather than removed:
- `team_env` documents the measurement at the point the flag is set, so the next
reader does not have to rediscover it. The flag stays: harmless, and free if a
later version supports teams non-interactively.
- the prompt addendum now asks for parallel DELEGATION rather than naming
teammates, which is what print mode can actually deliver — and it keeps the two
anti-patterns worth stating (own different files; do not split one change into
stages).
- the "no teammates" warning was blaming the flag and the config path. It now
judges on the SUBAGENT count, which is the mechanism in play, and a zero
teammate count is documented as expected rather than as a fault.
What the switch buys today is real but smaller than the plan assumed: it changes
the prompt so the lead parallelises across files instead of working through them
alone. Whether that beats solo on our own missions is still unmeasured, and the
plan's prediction — that it will not be faster — stands untested.
482 tests pass, clippy clean.
UNEXPLAINED, filed as #54: that team run's `topology_runs` row is `failed` while
the log line that sits immediately before the UPDATE never printed — zero matches
for 'microvm phase' in the container's whole log. The prime suspect is
`topology_worker`'s stuck-run reaper, which fails runs that are `running` with no
step records and does not filter by tier; a microvm run has no step records by
design. If that is it, any sufficiently long VM phase is failed out from under
itself. The system failed safely here — the empty-delivery guard caught that
nothing was produced, and nothing false was reported — but the cause is not known
and it is not being written up as if it were.
|
||
|
Omar Sobh
|
cb48f7ff3b |
feat(missions): Slice 3 — agent teams behind a per-mission switch, solo by default
`missions.team_engine` (0069): NULL = solo, `'claude_code'` = Claude Code agent teams inside the mission's VM. Solo stays the default deliberately — Anthropic measure multi-agent at 3-10x the tokens with wall-clock often LONGER, since the benefit is thoroughness rather than speed — so a mission that said nothing does not get a team. In-process teammates live in the lead's process, so ONE VM hosts the whole team. That is why this is a prompt-and-env change rather than an orchestration one: no N-VM fan-out, no placement per teammate, no new completion path. The lead decides its own team size and there is no flag that limits it, so the cap (4) is stated in the prompt. The addendum also carries the two anti-patterns from Anthropic's guidance, because they are exactly the shapes our pipeline templates have: teammates must own DIFFERENT FILES (two in one file overwrite each other), and one change must not be split into stages across teammates (a handoff loses context at every step). And: wait for your teammates — a summary written before they report is the lead's own guess. A solo mission's prompt and env are byte-identical to before this change. That is enforced by test, not by intention: the comparison between solo and team is only meaningful if the solo side did not also move. Evidence, because a team mission that forms no team is silently just a solo run that looked fine and spent fewer tokens: a second probe counts members in `~/.claude/teams/*/config.json` (minus the lead), reported separately from the subagent count, and a team mission with zero teammates logs loudly with the two likely causes. The teammate path is DOCUMENTED BUT NOT YET VERIFIED in our image, unlike the subagent transcript path which was measured — so a zero there means "no evidence found", and the first real team mission is what turns it into a fact. `Option<u32>`: None means no team was asked for or the probe could not run. Hooks (`TaskCompleted` / `TeammateIdle` exit 2, which would move `done_when` from post-hoc into the agent's own loop) are the highest-value part of this slice and are deliberately NOT here — they deserve their own pass rather than a rushed tail. 482 tests pass, clippy clean. |
||
|
Omar Sobh
|
c840688adb |
feat(missions): choose the independent validator per mission (#53)
`CLAWMATES_VALIDATOR_MODEL` is deployment-wide, so proving Slice 2 put a second
provider on the critical path of EVERY phase verdict. `cross_provider_judge`
deliberately does not fall back when the independent judge fails — a verdict
quietly produced by a same-family model would claim a property it does not have —
so a z.ai outage makes phases unmeetable rather than merely unverified. That is a
per-mission trade, not a per-deployment one.
`missions.validator_model` (0068), settable at create, with three distinct states
because an empty string and NULL mean opposite things in a nullable text column:
NULL use the deployment default
'' explicitly NO independent validator — judge with the house model.
The default must not quietly reinstate independence a mission was
told to skip.
'glm:glm-4.7' this spec, subject to the same three refusals as before:
same-family rejected, unregistered provider rejected, and a failed
independent judge does not fall back.
Whitespace counts as empty: a column hand-set to " " meant to say nothing.
478 tests pass, clippy clean. Behaviour is unchanged for existing missions — they
have NULL and so keep following the deployment default.
|
||
|
Omar Sobh
|
9aed20b6d0 |
fix(missions): capture a failed phase's work; harness gains a microvm scenario (#51)
A REGRESSION I INTRODUCED ONE COMMIT AGO. `capture_finished_coding_phases`
selects on `mp.status = 'completed'`, so the moment an unmet phase correctly began
reporting `failed`, its diff stopped being captured, committed or pushed — the work
was silently discarded. Found by the new harness scenario, whose phase legitimately
missed its condition and then had no artifact at all.
What was produced, and whether the goal was met, are different facts. The artifact
records the first; `mp.status` records the second. Capture now covers terminal
phases (`completed`, `failed`), so a phase that did real work and missed its goal
still delivers a reviewable diff — which is exactly what the next pass needs.
`scripts/verify-mission-delivery.sh microvm` — the regression net this session was
missing. Everything the microVM track proved by hand was guarded by nothing:
- THE KERNEL LINE is the assertion that cannot pass by accident. Every other
check would also pass if the phase had quietly run in a container on the
gateway; only the kernel says WHERE it ran. Compared against the real gateway
and node kernels read at start-up rather than pinned to a version, so
upgrading vmlinux does not manufacture a failure.
- subagent count > 0, from the server's own count of Claude Code's per-subagent
transcripts. Before `Agent` was in the allowlist this was structurally
impossible and nothing said so. A probe that could not run reports "?" and
FAILS the check rather than reading as zero.
- the verdict's judge and whether it was independent.
- negative control, observed passing: a mission whose backend no node can run is
refused at launch and stays draft. Without it the positive scenario would pass
just as well against a scheduler that ignored `backend` entirely — which is
what it did until the first real microvm mission landed on a node with no such
rootfs.
Also fixed in the harness: `api` now sends the JSON body on STDIN (`curl -d @-`)
instead of interpolating it into a single-quoted argument inside a double-quoted
ssh command. A task description containing "the crate's test suite" ended the
quoting and killed the remote shell; two attempts to escape it were themselves
wrong, because the backslashes must survive bash AND sed AND sh. Removing the
interpolation removes the class, and the next author does not need to know that
apostrophes were forbidden.
475 tests pass, clippy clean.
|
||
|
Omar Sobh
|
bb807c2f3a |
fix(missions): an unmet goal condition is no longer reported as success
Found by the Goodhart test for the independent judge, which is exactly what it was built to find. The test: a phase whose `done_when` demanded a passing suite, and a task that deliberately left a failing test. glm-4.7 judged it, ran `cargo test` itself, saw `parity_is_wrong_on_purpose ... FAILED` (exit 101), and returned met=false quoting the assertion — while the agent's own summary said "All three steps are implemented exactly as specified and independently verified". The verdict and the agent's account diverged, which is the whole point of an independent judge. And then the mission closed `completed`. `if verdict.met || last_pass` marked BOTH outcomes completed, so a phase that ran out of passes without ever meeting its condition reported success — and through `close_finished_missions`, so did the mission. The verdict said met=false in a column nobody reads before believing a green status. Anything consuming mission status rather than digging into the verdict saw a goal that was never reached as a goal achieved. Exhausted-and-unmet is now `failed`, and the log names the judge and whether it was independent. This changes observable behaviour: missions that would previously have finished green with an unmet condition now finish failed. That is the correction, not a regression — but it is worth knowing before the next scheduled run. Also: `Verdict.independent` had no column. The field existed in the struct and in the logs, so the audit question the mechanism exists to answer — was this checked by something other than the model that wrote it? — could not be asked of the database. Migration 0067 adds it, defaulting to false, which is the truth about every row written before now. Verified in production before the fix: glm-4.7, 4 checks all executed, the real cargo failure quoted, met=false. 475 tests pass, clippy clean. Note for whoever rebases: `sqlx::migrate!` embeds migrations at COMPILE time, so a new migration needs cm-db rebuilt (`touch crates/cm-db/src/lib.rs`) or the integration tests fail on a column that exists in the file and not in the binary. |
||
|
Omar Sobh
|
8796fbbcbb |
feat(evaluator): Slice 2 — an independent judge, from a different provider, with the same teeth
Claude writes the code and Claude judges it. That is a correlated failure: the
model that talked itself into a shortcut is the one disposed to accept it, and it
is the structural cause of the "early victory" failure Anthropic documents and of
our own Goodhart incident.
`glm` and `kimi` are both already registered in production, so the fix needed no
new credential path.
THE UNLOCK: `judge_with_tools` took `&AnthropicProvider`, but `LlmProvider` is a
single method — `stream(ChatRequest)` — and the loop only ever used that. The
concrete type was incidental. Widening it to `&dyn LlmProvider` means a
cross-provider judge runs the SAME allow-listed command loop. Before, independence
and real verification were mutually exclusive: the tool loop existed only on the
subscription path and every other route "judged claims only", so choosing an
independent judge meant giving up the checks that make a verdict evidence. GLM is
registered in anthropic format, so tool calling reaches it unchanged.
`CLAWMATES_VALIDATOR_MODEL` (e.g. `glm:glm-4.7`) selects it. Three refusals, each
protecting the claim the field makes:
- a spec in the implementer's own family is rejected, not used — `opus` judging
`sonnet` is not independence, they share a lineage and most failure modes
- a spec naming a provider this deployment never registered is rejected.
`Runtime::resolve_provider` silently falls back to the DEFAULT provider when
the registry has no such name, which would hand back Claude while the caller
believed it had GLM. Detectable because the returned model keeps its `name:`
prefix, so it is checked rather than trusted.
- an independent judge that FAILS does not fall through to the house judge. A
verdict quietly produced by a same-family model would claim a property it does
not have. The pass stays unmet, says why, and the next sweep retries.
`Verdict.independent` records it, `#[serde(default)]` so verdicts stored before
this field read back as not independent — which is what they were. An unrecognised
model family resolves to "unknown", never to ours: guessing would report
independence nobody established.
474 tests pass, clippy clean. Not yet enabled in production — the env var is unset,
so behaviour is identical until it is set deliberately.
|
||
|
Omar Sobh
|
11b274edc6 |
chore(images): Claude Code 2.1.223, and make the verifier foreground
Reviewed the changelog rather than bumping on principle. 2.1.220 → 2.1.223 for one
reason that bears on how we use subagents:
2.1.222 — "Fixed PreToolUse auto-allow hooks bypassing tool restrictions in
background agent tasks."
Subagents run in the background by default since 2.1.198, and the `verifier`
role's entire guarantee is a TOOL restriction — no Edit, no Write. So on 2.1.220
the one property we rely on was the one that bug could undo. 2.1.221 also fixes
`--mcp-config` servers not connecting before the first turn in print mode, which
is the mode we run and will matter when the MCP door reaches a VM.
Two findings from the changelog that we already had at 2.1.220, both worth knowing:
- 2.1.219: subagents can nest to depth 3 (was 1), so our roles can delegate
further than assumed.
- 2.1.212: a subagent inherits the parent's permission mode, which confirms the
verifier's read-only property must come from `tools` and not from permissions.
That is how it was written; now the reasoning is recorded next to it.
And a correctness fix that follows from the background default: the verifier is now
`background: false`. A background verifier lets the lead carry on and write its
report before the check has finished — the finding would arrive after the
conclusion it was supposed to inform.
Verified on tank: image reports 2.1.223, rootfs rebuilt, `--vm-selftest` all green
including a real agent turn on subscription auth, egress allow and deny both firing.
|
||
|
Omar Sobh
|
2dee941080 |
feat(missions): Slice 1 — a microVM agent can delegate, and we can see that it did
`microvm_executor` passed `--allowedTools Read Edit Write Bash`, which omits the
`Agent` tool, so Claude Code could not spawn a single subagent in any of our VMs.
The tool existed, the model knew how to use it, and the allowlist quietly removed
the ability. Nothing in any output said so.
Now: `Agent` in the allowlist, two roles supplied as `--agents` JSON, and a probe
that counts what actually ran.
Roles are JSON on the command line, not files, because `/mission/repo` is
collected and diffed — a role definition written into the checkout would arrive in
the delivered patch as if the agent had authored it.
Two roles only, and the choice is the research talking:
- `verifier` — the one multi-agent pattern Anthropic endorses for coding work.
It gets Read/Grep/Glob/Bash and deliberately NOT Edit or Write: an agent that
can fix what it is checking will fix it and report success, and the report is
then about a tree nobody reviewed. Its prompt demands the COMPLETE suite,
which is the counter to the "early victory problem" — the same failure as our
own Goodhart incident.
- `explorer` — context protection, read-only.
Roles like "tester" or "committer" are absent on purpose: splitting sequential
phases of the same work is a named anti-pattern, and it is the shape our pipeline
templates already have.
THREE THINGS THE IMAGE CORRECTED, none of which review would have caught:
1. `CLAUDE_AGENT_SDK_DISABLE_BUILTIN_AGENTS=1` (in the plan) removes EVERY agent
type, including the ones `--agents` defines. Measured: the lead reported "an
empty available-agents list" after trying four role names and — to its credit —
refused to fabricate a subagent result. Worse, the unit test asserting
"builtins off is paired with our own roles" PASSED throughout, because the
pairing holds in our code and not in the CLI. Dropped, and the test rewritten
to assert only what a unit test can speak to.
2. `--forward-subagent-text` refuses to run without `--output-format=stream-json`,
which would change how this module reads output. Dropped.
3. `--append-subagent-system-prompt` does not exist in 2.1.220 despite being
documented. The anti-shortcut rule is inlined per role instead — better anyway,
since a verifier and an explorer need different wording.
Evidence instead of assumption: Claude Code writes a per-subagent transcript at
`<session>/subagents/agent-*.jsonl`, so the guest is asked to count them before
collection (they live in /root, outside the collected tree). `VmOutcome.subagents`
is `Option<u32>` and the phase log prints it: `None`/"?" means the probe could not
run, which is a different fact from "delegated to nobody" and only one of those is
about the agent.
Verified in a container against the real CLI on tank before any of this shipped:
`FANOUT-OK`, a subagent transcript on disk, and zero errored Agent calls.
469 tests pass, clippy clean.
|
||
|
Omar Sobh
|
7696009b25 |
fix(fleet): name the BACKEND in a placement refusal, not just "microvm capability"
Observed on the negative control: a mission with backend='kimi' was correctly refused, but the message read "no online node reports microvm capability" — and both nodes do report it. What one lacked was the image. That first clause would have sent an operator to reinstall firecracker on a node that already had it. The refusal now names the backend, and the remedy still names both halves. |
||
|
Omar Sobh
|
d9f53a3f96 |
fix(fleet): placement requires the backend's rootfs image, not just KVM
The first real microVM mission was placed on morpheus because it reports
{"microvm": true}, while only tank had rootfs-claude.ext4. It failed by name
rather than booting the wrong image — but whether a mission ran came down to
which capable node was listed first, which is a coin flip dressed as scheduling.
`missions.backend` was invisible to the scheduler.
The node now enumerates the images on its disk and reports them as a `rootfs`
ARRAY. `microvm::available_backends` lives beside `rootfs_for`, its inverse,
because the two must agree on what a backend name means; split apart, one drifts
and the scheduler starts promising images the booter cannot find. It only
advertises names `rootfs_for` would accept, and reports an empty array rather than
omitting the key — set_capabilities REPLACES, so a deleted image stops being
advertised instead of leaving a stale claim.
`nodes::online_for_backend` requires microvm AND that the node's list contains the
mission's backend. A node on an older daemon has no `rootfs` key and matches
nothing: unknown is not permission, the same treatment every other capability
gets. `backend_key` maps the three spellings of "the default image" to the one
name the node advertises, and is tested — a mismatch there would reject every node
for an ordinary mission with no backend set.
The launch error now names both halves of the fix, since "no capable node" was
true but unhelpful when the node was capable and merely lacked the image.
Mission gains `backend` on the domain struct; it was a column the executor read
from the phase query while the struct that placement uses could not see it.
464 tests pass, clippy clean.
|
||
|
Omar Sobh
|
1cd81a8b2a |
fix(missions): place a microvm mission before returning from on_launch
Self-inflicted, one commit old, and found by running a real mission: the early return I added for "a microvm mission materialises no team" sat ABOVE the microVM placement block in the same function, so on_launch returned before ever choosing a node. The mission then failed with the executor's own guard — "mission has no target_node_id ... a microvm mission cannot run on the gateway, which has no /dev/kvm" — which is the guard working exactly as designed, on a cause one layer further up. Placement now runs first. Worth noting the shape: adding an early return to a long function silently skipped everything below it that the same runtime_kind depends on. 461 tests pass, clippy clean. |
||
|
Omar Sobh
|
521b8dea10 |
fix(missions): the third team gate, and a container a microvm mission never uses
on_launch demanded a team template too — "pick teams in the wizard" — so a microvm mission still could not launch after the first two gates were exempted. Three separate places required a claw graph for a path that runs one `claude -p` inside a VM: routes::missions (draft→running), phase_runner::launch_phase (no matching teams → stay pending), and here. Returning before team materialisation rather than filtering its picks: claws that never run are not a cheaper version of the same thing, they are a runtime binding and a pairing code describing something nothing speaks to. Also stops provisioning the per-mission ZeroClaw container for a microvm mission. The first real run was observed starting one and leaving it holding a pairing code and ~3 GB of image for the life of a mission that never contacts it. 461 tests pass, clippy clean. |
||
|
Omar Sobh
|
0a9747091f |
fix(missions): a microvm mission needs no team, and two checks required one
Found by running one: the mission was created with runtime_kind='microvm' and then refused to launch with a bare 400, because draft→running requires a materializable team. Past that, `launch_phase` returns early when a phase has no matching teams — so even with the launch allowed, the phase would have sat `pending` forever while the log said only "no matching teams", and the executor would never have been reached. Neither check applies to this path: microvm_executor runs the agent CLI directly in the VM, so there is no claw graph to materialise. Satisfying the checks by attaching a team template would have provisioned claws that never run. The repo checkout still happens — the VM needs the repository. 461 tests pass, clippy clean. |
||
|
Omar SobhandClaude Opus 5
|
c9b7d8b6ca |
fix(missions): a microvm mission could not be created at all
`runtime_kind='microvm'` passes the DB CHECK, is honoured by placement, and now has an executor — but `POST /api/missions` rejected the value with 400, so the only interface that creates missions could not produce one. And `backend`, which selects the per-CLI rootfs, was not in the create payload at all: it existed as a column and as a parameter to `vm_create`, with nothing able to set it. microvm needs no target_node_id at create time, unlike local_herdr: placement resolves a KVM-capable node at launch and fails the launch when there is none, so an explicit target is a request rather than a requirement. 461 tests pass, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
4f07430e92 |
feat(missions): B4.5 — phase_runner runs a microvm mission in a VM
`runtime_kind='microvm'` placed a mission on a KVM-capable node and then nothing
executed it: config accepted without a reader, one of the four seams this project
keeps closing. This is the reader.
`microvm_executor` — inject → run → collect → destroy, the shape copy mode
already proved for containers with a VM boundary instead of a namespace one. The
checkout goes in as a tar, the work comes back as a tar over the SAME host path,
so `mission_delivery::capture_phase_diff_at` needs no change at all.
The agent is told NOT to push, unlike the container path's session prompt. Two
reasons: delivery is already host-side and diffs the collected tree against the
recorded clone point (covering committed, staged and unstaged work in one pass),
so pushing would add a second untested way for work to arrive; and pushing would
mean forge credentials inside the VM, when the point of collecting is that the
guest never holds them.
Exactly ONE topology_runs row (tier='microvm'), mirroring launch_direct_session:
close_finished_phases, evaluation, capture and delivery all key off those rows,
and a second completion path would be a second way for a phase to finish with one
of them untested. The row and the phase flip happen BEFORE any fallible VM work,
so a missing token or a node that lost its capability shows up as a failed run an
operator can see — not a phase that stays pending and retries every ten seconds.
Fail-closed points, each the reader for a guarantee built earlier:
- credentials resolve BEFORE the VM boots, so a missing subscription token
fails the phase instead of booting a VM whose agent sits unauthenticated
- a VM reporting egress:false is REFUSED, which is what makes create's
egress/egress_host/egress_guest fields more than decoration — a turn without
egress does not fail, it hangs
- the injected checkout is PROVEN present in the guest before an agent turn is
spent on it; an inject that reports success while landing nothing would
otherwise become an agent reporting an empty repository
- work is collected even when the agent exits non-zero — a turn that failed
partway still wrote files, and a retry needs to see them
- a turn that ran but could not be collected is a FAILED phase, not a happy one
- destroy runs on every exit path, or an 8 GB sparse rootfs leaks
Two integration gaps found while wiring, both of which would have produced a
mission that completed having delivered nothing:
- `capture_finished_coding_phases` pulls work out of a CONTAINER before
capturing. A microvm mission has none, so the docker connect would fail, the
loop would `continue`, and capture would be skipped forever while the phase
sat marked completed. Its work is already collected by the executor.
- `launch_phase` provisioned a runtime container, copied the checkout into it
and wrote a runtime binding + pairing code describing a runtime nothing uses;
and the orchestrator's workspace pin — deliberately FATAL — would have failed
a microVM launch on a container it was never going to use.
461 tests pass, clippy clean.
NOT YET PROVEN END TO END: no mission has run through this path. The pieces under
it are each verified on tank (image, credentials, egress, a real agent turn), but
this executor has only been compiled and unit-tested. Deploy + one real microvm
mission is the remaining step.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
ebdba34da6 |
feat(fleet): B4.6 — a microVM reaches the API through a vsock CONNECT proxy, with an allow-list
The guest still has no network interface, and now that is the design rather than
a gap. Its only route out is an HTTP CONNECT proxy: agent CLI -> 127.0.0.1:3128
in the guest -> vsock 9002 -> a per-VM Unix socket on the host -> TLS to an
allow-listed host.
Why not TAP + iptables, which is what the Firecracker write-ups do — measured,
not argued:
- `ip tuntap add` is DENIED to the daemon user (needs CAP_NET_ADMIN), so TAP
would need root to pre-provision devices, the same privilege detour the
loop-mounted rootfs already forced.
- tank's FORWARD policy is DROP with Docker and Tailscale chains, so rules
would have to be inserted at position 1; appended ones die silently.
- a leaked TAP is a new class of host litter to reap.
CONNECT needs no privilege at all and is better on the merits: the client hands
us the HOSTNAME, so resolution happens host-side and the guest needs no DNS or
resolv.conf; the allow-list is by name, not address; and nothing in the guest can
reach the network except through one function. The guest end parses nothing and
enforces nothing, so a compromised agent cannot argue with the policy.
Rests on one measured fact: `claude` honours HTTPS_PROXY. With the proxy at a
closed port, `claude -p` fails ConnectionRefused instead of answering.
THE RESULT: a real agent turn now completes inside a VM with no network card, on
subscription auth — `claude -p` replies VM-OK. The selftest asks for it whenever
CLAUDE_CODE_OAUTH_TOKEN is present and SKIPS loudly when it is not, since it
spends a little of the plan.
The audit log earns its keep immediately: during that turn the proxy logged
`egress DENIED http-intake.logs.us5.datadoghq.com` — the CLI's telemetry, which
the mission container permits today without anyone deciding to.
Three bugs found by the checks rather than by review:
- `env_pairs` returned early when a caller sent no env, so the proxy address
was never added and `curl` in a VM with a working tunnel reported "Could not
resolve host". Absent env means "the caller sent none", not "this command
needs no environment".
- the deny check PASSED for the wrong reason — DNS was failing, so nothing was
refused by the allow-list at all. It now requires a 403 from the proxy, so it
cannot go green on a broken tunnel.
- `host_allowed` accepted `evil.test/api.anthropic.com`, which ends with an
allowed suffix. Hostnames are now validated against a character class, which
also refuses IP literals so an address cannot sidestep a list of names.
- `BufReader::into_inner()` discards buffered bytes: wrapping the stream twice
would have dropped the start of the TLS handshake and stalled a tunnel that
looked established. One reader now spans the request, and anything buffered
past the headers is forwarded as payload.
`iproute2` is in agent-toolchain because it is load-bearing: the guest's `lo`
starts DOWN, and while it is down a listener on loopback BINDS and then refuses
every connection with ENETUNREACH. fcagent finds `ip` by absolute path — as pid 1
its PATH comes from the kernel, and execvp's fallback excludes /usr/sbin, where
Debian puts it.
Egress needs both ends up, so `create` reports `egress` and the guest's `ping`
reports its own half. A VM without it is legal but never silent.
Verified on tank: 16/16 with backend=claude (create 1428 ms), 12/12 on the
default rootfs, no leaked processes, VM dirs or proxy sockets. 457 tests pass,
clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
2edafdaf0d |
fix(fleet): a microVM authenticates by subscription only — never with an API key
B4.4 had the microVM path share `forwarded_provider_env(auth)` with the
container path, on the reasoning that the two must not diverge. That was wrong
in the one direction that costs money: gw-04 has CLAWMATES_RUNTIME_AUTH unset,
so the container path forwards ANTHROPIC_API_KEY today — and a VM would have
received it. Claude Code ranks the API key ABOVE the subscription's OAuth token,
so the VM would have worked perfectly while billing per-token against a plan we
already pay for. No error, no symptom but the invoice.
`microvm_provider_env` is subscription-only BY CONSTRUCTION: it does not take
the auth mode as an argument and does not read CLAWMATES_RUNTIME_AUTH at all.
Taking the mode as a parameter would mean one unset variable on a new host
silently turns the API key back on. The container path is unchanged and still
honours the operator's mode — the divergence is now deliberate, with the reason
at the definition.
Two other fail-closed rules fall out of it:
- A missing or blank subscription token REFUSES the launch rather than
returning an empty environment. A VM with no credential does not error;
`claude -p` hangs, which reads as a phase stuck at `running` with nothing in
the logs. The refusal names the variable.
- An unrecognised backend is refused rather than handed the Anthropic token.
GLM and Kimi reach their own endpoints via ANTHROPIC_BASE_URL and that
contract is not settled yet; guessing it would send a subscription
credential to z.ai.
Measured on tank, and this is the end-to-end proof B4.4 could not give:
`claude -p` in the agent-claude image with the real subscription token replies
"OK". Injecting the token in a VM moves the failure from "Not logged in" to a
network error, so the credential channel is accepted by the CLI — the VM's
remaining problem is egress (#49), not auth.
447 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
c3297b86cf |
feat(fleet): B4.4 — credentials reach the microVM guest as exec env, and a bad entry refuses the exec
`claude -p` in the VM failed with "Not logged in". The credential now travels on
the exec op: `env` on `vm_exec` → fcagent → the command's environment. An env var
rather than a file because the per-VM rootfs dies with the VM but an env var
never touches the guest disk at all.
**Every problem in an env entry fails the exec.** The tempting alternative —
skip the entry we cannot use and run anyway — produces a `claude -p` with no
credential, and that does not error, it HANGS. A phase stuck at `running` for
ten minutes with nothing in the logs is exactly what a missing token looked like
on the container path. Names are validated ('=' or NUL would define a different
variable than the one asked for via putenv semantics), values must be strings,
and errors name the key and never the value — an error string travels back over
the wire and into logs.
One list of which credentials travel: `forwarded_provider_env` reuses
`forwarded_provider_keys`, and the container path now reads it too. If the two
execution paths diverged, a mission would behave differently depending on where
it landed — including the expensive way, where one path forwards
ANTHROPIC_API_KEY and bills it while the other uses the subscription. A blank
value is omitted rather than forwarded empty, so `claude` reports having no
credential instead of failing authentication with one.
Verified on tank (`--vm-selftest` backend=claude, 13/13, create 1532 ms): an
injected var reaches the guest command over the real vsock wire, and an
unusable entry comes back ok:false with no rc.
FINDING — the CLI leg remains UNPROVEN, and deliberately so. The guest has no
network interface: `create` writes boot-source, drives, machine-config and vsock
and no `network-interfaces` key, and a booted guest has no routes, no
resolv.conf, no DNS and no TCP. So `claude -p` cannot reach the API whatever
credential it holds. Injecting the real token would have proven nothing, because
the failure would have been network and not auth. Filed as B4.6 (task #49) with
the TAP-vs-vsock-proxy trade-off; B4.5 is now blocked on it.
444 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
0c291ed1bb |
feat(fleet): B4.4a — a real agent-claude microVM image, and a check that it has an agent in it
The only rootfs on this track came from clawmates/agent-terminal:dev. Mounted,
it held git and nothing else: no claude, no node, no cargo. A VM booted from it
looks perfect and cannot run a mission, so B4.5 could have been written and
never verified.
images/agent-toolchain — the shared mission toolchain (node 22, git, rust +
cargo-audit, gitleaks/trivy/semgrep, tea/gitea-mcp), lifted from the proven
deploy/clawmates-runtime image minus the zeroclaw daemon: a microVM mission runs
the direct-session model, so there is no daemon to host. A base image rather
than three self-contained Dockerfiles because this layer is ~3 GB and the real
risk is scanner and toolchain versions drifting between per-CLI images — the
evaluator runs the project's own suite to check a claim, so `cargo` present in
one image and absent in another makes the same mission pass or fail by backend
with nothing saying why.
images/agent-claude — plan A6, first of three: the pinned CLI and its env
contract only, so bumping Claude Code does not rebuild the toolchain and cannot
disturb agent-kimi / agent-glm. HOME=/root with an empty .claude for B4.4 to
inject into; no ANTHROPIC_API_KEY, since it silently overrides the subscription
OAuth we already pay for.
Both the builder and the node selftest now ASK the guest for the CLI the image
is named for, instead of trusting the name. `required_cli` maps claude/kimi/glm
to a probe; an unrecognised backend reports unchecked and prints SKIP rather
than passing quietly.
Verified on tank:
- rootfs-claude.ext4 boots; git, node, cargo, a real git commit all work
- `claude --version` → 2.1.220 over vsock, in both the builder and
`--vm-selftest` (11/11, create 1498 ms)
- negative control: the same builder run against agent-terminal with
FC_CLI forced reports `cli rc=127 claude: not found` and exits 1, so the
green result above is a measurement and not a default
- `claude -p hello` fails with "Not logged in · Please run /login" — the CLI
runs headless in the VM, and B4.4 only has to supply the credential
- no leaked firecracker processes or vm dirs afterwards
437 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
6687f8b808 |
feat(fleet): B4.3 — per-mission rootfs selection (missions.backend)
`vm_create` takes a backend name and boots `rootfs-<backend>.ext4`; NULL or "default" boots the golden image. Makes the per-CLI images from B4.1 actually reachable (one image per CLI, per A6). A missing image is an ERROR naming the file and how to build it, never a quiet fall back to the default. That fallback is the tempting version and the wrong one: it would run a claude mission in a kimi VM, or in a rootfs with no CLI at all, and report success for whatever came out. Verified on real hardware, not just in a unit test — the selftest asks for an image that does not exist and FAILS if it boots. `create` now reports the rootfs that actually booted, not the one that was requested, so a mission artifact can show the wrong VM ran. The migration adds no CHECK constraint listing the CLIs. Which images exist is a property of the NODES, not the schema; a constraint would need migrating for every new image while still not guaranteeing the image exists anywhere. The node validates and names what is missing. Backend names are `[A-Za-z0-9_-]` and rejected rather than sanitised, since they become filenames. Verified on tank: default backend 8/8; `CLAWMATES_FC_BACKEND=agent-terminal` 9/9 including the absent-image check, create in 910ms on a rootfs built from a real Docker image. 435 tests green, no leaked processes or VM dirs. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
08847e6a63 |
feat(fleet): B4.2 — static Rust guest agent replaces the python one
The python guest agent only ever worked because Firecracker's CI Ubuntu image happens to ship python3. NONE of our images do — agent-base has neither python nor git, agent-terminal has git but no python — so it could never have run in a real mission rootfs. An agent that dictates what must be installed in the image has the dependency backwards. crates/bins/fcagent is a 905K static x86_64-unknown-linux-musl binary that needs nothing from the rootfs it is dropped into. The wire is unchanged on purpose — 4-byte BE length + JSON, ops ping/exec/put/get — so microvm.rs and microvm_client.rs needed no edit at all. std has no AF_VSOCK and the workspace denies `unsafe`, so it uses the `vsock` crate. `process_group(0)` gives each command its own group without unsafe, so a command that spawns background children can be killed wholesale rather than outliving the run. A unit test caught a bug that would have broken EVERY exec: sourcing the image-env file with `. env.sh 2>/dev/null; cmd` returns rc=1 WITHOUT running cmd, because `.` on a missing file makes a non-interactive POSIX shell exit immediately. On any rootfs lacking that file every command would have failed while looking like an ordinary non-zero exit. Guarded with `if [ -f ]` now. Other places a failure must not borrow an outcome's representation: a killed command reports ok:false with no rc (not rc=124, which would read as a build failure); `get` on a missing path is an error, not an empty archive; a signalled process reports 128+signal rather than success. Verified on tank: --vm-selftest still 8/8 with the agent swapped (create 949ms, wire identical), fc-node-setup 8/8, and — the point of the change — a rootfs built from clawmates/agent-terminal:dev, which has NO python3, boots and reports `git version 2.39.5` from inside the VM. Also fixes a shell bug in fc-build-rootfs.sh: $HOME in a double-quoted default expanded on this Mac, so it looked for the node's binary under /Users/quantum on a Linux host. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
02ba557c3e |
feat(fleet): B3 — server-side microVM client over NodeHub
cm_api::microvm_client::MicroVm wraps the node's vm_* ops as typed calls
over the existing hub request/response channel: create / inject / exec /
collect / destroy, plus list() for reaping. No new transport.
Fixes a wire-contract mismatch B2 would have shipped. `Uplink::Result`
declares `output: String`, but the node's vm_* handler returned a JSON
object. The frame then failed to deserialize and hit the uplink match's
`Err(_) => {}` arm, so the reply VANISHED and every vm_* call would have
timed out after 20s with nothing anywhere explaining why. The node now
sends a string, matching the contract rather than what looked tidier.
That silent arm is fixed too: an unparseable frame now logs the node, the
parse error and the frame head, and says explicitly that the request it
was answering will time out. It is the arm that would have hidden this.
Two more places where a failure must not borrow a legitimate outcome's
representation:
- vm_exec returning no `rc` is an error, not a zero. A missing exit code
means the guest did not report one; reading it as success is how a
failed command becomes a passing phase.
- vm_collect on a missing path is an error, not an empty archive — an
empty tar looks exactly like a run that produced nothing.
Timeouts: the hub's deadline is the guest's plus 30s, saturating. A
caller passing a huge budget would otherwise wrap to a tiny timeout and
turn a long agent turn into a spurious transport failure. clippy caught
the tautological assertion in the first version of that test, which is
what surfaced the overflow.
Verified: `--vm-selftest` on tank still 8/8 after the output-type change
(create 950ms), 427 tests green.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
2d04c5e257 |
feat(fleet): B2 — vm_* node ops for Firecracker microVMs
create / inject / exec / collect / destroy / list, riding the node's
existing frame dispatch ({t, id, …} -> {t:"result", id, ok, output}), so
no protocol change was needed. Control is length-prefixed JSON over
vsock; the serial console stays a log, because feeding a guest over stdin
races its startup and arrives half-consumed.
DEVIATION FROM THE PLAN, deliberately: this does NOT implement
cm_sandbox::SandboxDriver. That trait is container-shaped —
attach_pty/resize_pty/argv exec — while missions need
create -> inject -> run -> collect -> destroy. Conforming would mean
building PTY-over-vsock and window-resize semantics that no mission path
calls, purely to satisfy a signature. We give up automatic RemoteDriver
marshalling; orphan reaping is a label/id sweep either way.
Three traps from the B0 spike are handled in code rather than remembered:
- Firecracker does NOT unlink its vsock UDS on exit, so destroy unlinks
it explicitly, and the selftest ASSERTS it is gone. Assuming the VM
tidies up after itself is how the mission checkout accumulated four
uid bugs.
- firecracker is spawned via setsid and killed as a process GROUP, so a
background child cannot outlive the VM holding its workdir open.
- create does not return until the guest agent has answered a ping. A
VM that booted but serves nothing is worse than one that failed, so a
half-created VM is destroyed rather than left registered.
A vm id becomes a path component, so ids are restricted to [A-Za-z0-9_-]
and REJECTED rather than sanitised — a caller that sent `../../etc`
wanted something we should not guess at.
Verified on tank through the real Rust path, as the daemon user, with no
sudo: `clawmates-node --vm-selftest` -> 8/8, create in 986ms, and the
host left with zero firecracker processes and zero VM directories. The
selftest asserts every step, including that a destroyed VM can no longer
be exec'd; a test that only reports the steps it completed cannot
distinguish "passed" from "stopped early".
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
0f7fa31f86 |
feat(fleet): B1 — microvm runtime kind and KVM placement predicate
Phase B step 1, on top of the B0 spike that proved microVMs boot here.
KVM is a HARD predicate, not a preference. gw-04 — where every mission
runs today — is itself a VM without nested virtualisation and has no
/dev/kvm, so a microvm mission landing there cannot start at all. The
scheduler therefore has to be able to tell nodes apart, which means the
node has to report what it can host.
Nodes gain a `capabilities` jsonb, populated from a probe on the node
rather than from configuration: /dev/kvm either exists there or it does
not, and nothing on the server can make it appear. The probe OPENS the
device rather than stat-ing it, because it can exist while being
unopenable (wrong group, or a container without the device passed
through) — which is precisely how firecracker will fail.
`microvm` requires BOTH kvm and a firecracker binary. A node with KVM
but no binary looks capable by the obvious test and fails at launch; a
node with the binary but no KVM is gw-04.
Placement fails the launch when no capable node exists, rather than
letting a mission sit in 'running' with nowhere to run. An explicit
target_node_id is treated as a request, not a guarantee — it is honoured
only if that node actually reports the capability.
`capabilities` defaults to '{}' NOT NULL so a node that has never
reported fails every predicate: an unqueried node and an incapable node
must be indistinguishable to the scheduler, because scheduling onto a
node whose abilities are unknown is how you get a mission that cannot
start and does not say why. The report replaces rather than merges, so a
capability the node has LOST disappears instead of leaving a stale true.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
da3731d753 |
fix(missions): a coding phase that delivers nothing is a failure
The last open item in the silent-success class: a coding phase that changed no files reported `completed` — the same status a phase gets for delivering tested, reviewed, pushed work. Mission `019fcf62` completed that way with its agents silently unpinned from the repo, and nothing in the platform disagreed; it was found by a script diffing the forge. The verdict is applied at capture rather than at completion, because capture selects on `status = 'completed'` — the platform does not know whether a phase produced anything until after it has already finished. Three conditions must hold before failing a phase, because a false positive here fails honest work: the phase is a coding phase (research phases legitimately write nothing to the tree), the diff was actually computed (an uncomputable diff also reports zero files — blaming the agent for a platform fault is the same defect wearing different clothes), and `allow_empty` is not set. Only an explicit `true` opts out, so a typo leaves the check armed. Registered in phase_config with its reader named, per the seam-2 rule. Also closes an ordering hazard this exposed: capture is batched and runs after a phase completes, so a backlogged mission could close as 'completed' and only then have capture discover an empty phase — leaving a 'completed' mission holding a 'failed' phase, unfixable because the mission-close CASE only touches 'running' rows. A repo-bearing mission now waits for its work to be captured before closing. Adds a `noop` scenario to the harness: a phase told to change nothing, which PASSES only when the phase comes back `failed`. Same discipline as the uid self-test — a check that has never been seen to fire has not been shown to work. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
4f6719c80e |
feat(missions): make copy-in/copy-out the default filesystem model
Copy mode shipped opt-in so that changing how every mission receives its
code required someone to type it. Four production missions and a
fail-closed harness later, opt-in is the riskier setting: the bind path
is the one with four documented work-loss incidents, and leaving it as
the default means the untested path runs whenever nobody sets the
variable. `CLAWMATES_MISSION_FS=bind` still selects it; anything else —
unset, empty, misspelt — gets copy mode, so a typo lands on the safer
path rather than the one being retired.
Also fixes a real leak found while scoping the deletion below: the git
helper built its `safe.directory` argument with `Box::leak`, justified as
"the process is short-lived". That is true of a CLI and false of cm-api,
which is a long-running server — so it leaked one allocation per git
call, growing with every phase of every mission.
The A5 deletion is NOT done here, and two of its items should never be
done:
- `scrub_remote_credentials` is a security control, not a uid
workaround. Copy mode uploads the whole `.git` into a container the
agent controls as root, which makes stripping the token from
`.git/config` more necessary, not less.
- `has_local_work` / `checkout_in_use` guard `fetch_and_reset` at every
phase launch and have nothing to do with who writes the checkout.
The host checkout still persists across phases under copy mode —
mission `019fcf62` shows the marker firing there. Deleting them
reintroduces PRIOR-PHASE-WORK-WAS-LOST.
The rest (`share_repository_across_uids`, `clear_stale_commit_editmsg`,
`-c safe.directory`) are genuinely obsolete under copy mode but stay
while `bind` remains selectable: a workaround may only be deleted once
the situation it works around can no longer be chosen.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
1253595ba7 |
fix(missions): stop three launch failures from passing as success
A verification run against the deployed stack found a chain mission whose phase 0 reported `completed` with zero files, no commit error and no push error — indistinguishable from a phase that correctly had nothing to do. Three separate defects had to line up, each of them the same shape: a failure sharing its representation with a legitimate negative result. 1. `pin_agent_workspaces` embedded the whole config in one `sh -c` argv. That works until the file grows — config gains a block per provisioned claw — then fails with `argument list too long`. Now written through the tar upload API, which has no argv limit, so the failure mode is gone rather than merely further away. 2. A failed pin was logged "(continuing)". Without the pin, agents write to their sandboxes and the committer finds nothing in /mission/repo — the mission cannot deliver, so the launch now fails where someone is still looking. The restart that applies the pin is fatal for the same reason. 3. `capture_phase_diff_at` swallowed `git diff` failures with `unwrap_or_default`, so an unreadable base landed `empty: true, files_changed: 0` — byte-identical to an honest no-op. The error is now recorded as `diff_error`, and an empty patch that came from a failed diff is no longer trusted to mean an unchanged tree. Adds scripts/verify-mission-delivery.sh, which found #1 and #2 on its first real run. Its probes are fail-closed: no placeholder values, a self-test that proves the uid probe can detect the split it looks for, and FAIL-NORUN for a scenario that never executed. Its own first version had this bug too — a `die` inside `$(...)` exited the subshell, so a run that could not authenticate printed "all checks passed" and exited 0. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
7e07c389c6 |
feat(missions): wire copy-in/copy-out behind CLAWMATES_MISSION_FS=copy
With the flag set, ensure_container omits the /mission bind, the checkout is pushed into the container at phase launch, and the agent's work is pulled back before capture. The simplification that makes this small: sync_out unpacks over the SAME host path the checkout came from. The host directory stays a server-owned staging area with exactly one writer, and capture_phase_diff_at needs no change at all — it still finds a normal checkout exactly where it always has. Delivery, gating, commit and push are untouched. Two failures are deliberately loud rather than silent: - copy-IN failure fails the phase launch. Continuing would start a phase against an empty directory, and the agent would cheerfully report having done work on a repo that was not there. - copy-OUT failure SKIPS capture. Capturing anyway would diff a stale host tree and record "no changes" for work that exists — success reported for nothing, which is the exact failure mode this codebase keeps paying for. Opt-in: the bind path is what production has run since the beginning, and the test asserts a near-miss value leaves it there rather than silently switching every mission. 414 tests, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
389b41f8e6 |
feat(missions): copy-in/copy-out primitive for the mission checkout
The first half of removing the shared bind mount. Not wired yet — this adds the mechanism and its tests. One cause, four fixes so far: .git/objects permission denied (core.sharedRepository), the capture base being overwritten each phase, COMMIT_EDITMSG root-owned, and reset --hard deleting a prior phase's work (.git/clawmates-in-use). core.sharedRepository was never a general solution — it covers objects and refs, and every OTHER file git touches is a fresh opportunity. Copy-in/copy-out removes the cause instead: the agent owns its filesystem with no second writer. Measured before building, because the plan named copy cost as the open risk: a real 65 MB checkout of this repo copies in 0.23s and out 0.18s on gw-04. Not a risk at this size; re-measure an order of magnitude larger. No compression — the payload crosses a local socket, so gzip would spend CPU to save nothing. Two safety properties, both tested: - The archive comes back from a container the agent controls as ROOT, so it is untrusted input. A `../ESCAPED` entry must not write outside the destination. The test writes the tar header bytes by hand because the tar crate refuses to BUILD such an entry through its safe API — which is reassuring, but means the hostile case has to be constructed the way an attacker would. - Symlinks are packed as links, never dereferenced. Following them on copy-IN would smuggle host files into the container; the test plants a host secret behind a symlink and asserts its contents never appear in the archive. Ownership is deliberately not preserved on unpack: the archive's uids are the container's root, and re-applying them on the host would recreate the exact uid split this exists to remove. 413 tests, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
0d9498ec6e |
fix(missions): copy an allow-list, not the whole 1.7GB seed dir
Checking before deploying caught a mistake in the previous commit. The seed dir on gw-04 is 1.7 GB and the first version copied all of it per mission — tens of seconds each, and ~17 GB across ten concurrent missions. 1.5 GB of that is .rustup: a Rust toolchain that installed itself into the data dir back when HOME=/zeroclaw-data and the image had no toolchain. The image now ships Rust at /usr/local/cargo, which is what the container's PATH actually resolves — verified live. The data-dir copy is dead weight and is not even reachable. SEEDED_PATHS now copies only what carries per-mission identity or secrets: .zeroclaw (config.toml with the door token, sessions.db, devices.db), clawmates-mcp.json, .claude + .claude.json, .kimi-code, glm-home, agents. Roughly 46 MB instead of 1.7 GB — about 37x smaller. Caches and toolchains are deliberately excluded: .rustup, .npm, .cargo, .cache, .local. They hold no secrets and a mission reads the image's. Absent paths are tolerated: a fresh deployment has no .kimi-code until Kimi is first used, and that must not fail container creation. The test asserts both directions — the token-bearing paths ARE copied and the caches are NOT — because either mistake is silent: copying everything just makes missions slow, and copying nothing quietly restores the credential sharing. 409 tests, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
15e7608e4a |
fix(missions): give each mission its own runtime data
Every per-mission container bind-mounted the SAME host seed dir as
/zeroclaw-data — shared with each other AND with the singleton runtime.
That directory holds config.toml, which carries the §15 door bearer
token, plus sessions.db and devices.db.
So one mission could read another mission's credential, and anything it
wrote there was inherited by every later mission. teardown_container
only removes /var/lib/clawmates-missions/{id}, so the shared directory
was never cleaned — the contamination was permanent.
The code already knew. The comment on DEFAULT_SEED_DIR names the sqlite
race and calls copy-on-write per mission the long-term fix. This is that
fix: seed_runtime_data copies the seed into
<missions_root>/<mission>/runtime-data at container create, and the
mount points there. Cleanup is free — teardown already removes that tree.
The copy runs in a throwaway container because cm-api cannot see the seed
dir: it hands that host path to Docker but never mounts it itself. The
runtime image is reused so nothing extra is pulled, and `cp -a /seed/.`
copies dotfiles — `/seed/*` would silently skip .zeroclaw/ and produce a
runtime with no config at all.
A copy failure is fatal to container creation on purpose. Falling back to
the shared mount would silently restore the credential sharing this
removes, and silent fallback to a weaker posture is the failure mode this
codebase keeps paying for.
The test asserts path shape rather than behaviour: an edit that points
the mount back at the seed dir restores credential sharing with no other
visible symptom, so the path IS the invariant.
408 tests, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
5d98fcf44a |
feat(missions): forward ZAI/KIMI keys so one binary serves three backends
All three providers run through the SAME `claude` binary, verified live: Anthropic CLAUDE_CODE_OAUTH_TOKEN -> ANTHROPIC-OK GLM ANTHROPIC_BASE_URL=https://api.z.ai/api/anthropic -> GLM-OK Kimi ANTHROPIC_BASE_URL=https://api.kimi.com/coding/ -> KIMI-OK That is a stronger multi-provider story than a provider-per-implementation: skills, subagents, MCP, hooks and tool policy are identical across all three because it is literally the same harness. The `kimi` CLI (0.31.1, shipped in the image) 401s on this key and is not needed -- the claude binary reaches Kimi's Anthropic-compatible endpoint directly. Worth knowing before someone debugs the CLI. forwarded_provider_keys now ships ZAI_API_KEY and KIMI_API_KEY into mission containers in BOTH auth modes: they are unrelated to the Anthropic credential, so the api_key/subscription split does not apply to them. A mission that selects a backend without its key present would otherwise fail at the first turn. Keys persisted in /opt/clawmates/.env and passed through compose. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
4ff4e6f7ee |
fix(missions): a root-owned COMMIT_EDITMSG must not block delivery
Mission 019fcd0c produced correct work — a reviewed, tested function plus a REVIEW.md quoting a real cargo test summary — and delivered none of it: git commit → exit 128: could not open '.git/COMMIT_EDITMSG': Permission denied The agent ran `git commit` itself inside the mission container (as root), leaving that file owned by root at 0644. core.sharedRepository covers objects and refs — .git/index lands at 0666, which is why commits work at all — but not COMMIT_EDITMSG, which git writes with the default umask. Unlinking works where overwriting does not: removing a file needs write permission on the DIRECTORY, and .git/ is owned by the server. Silent on failure by design, so the commit reports the real error rather than this speculative cleanup. Third distinct instance of the same uid-split class (objects, then the capture base, now this). The pattern holds: the checkout is one directory written by two users, and each new file git touches is a new opportunity. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
758b2dbd96 |
feat(missions): run a phase as one direct session, behind a flag
CLAWMATES_MISSION_EXECUTOR=session makes launch_phase run the whole phase as a single `claude -p` against /mission/repo instead of driving turns through ZeroClaw. Opt-in, because silently changing how every mission executes is exactly the sort of change that should require someone to have typed it. It still writes ONE topology_runs row. The entire downstream lifecycle -- close_finished_phases, evaluation, capture, commit, gate, publish -- keys off those rows, and inventing a second completion path would mean two ways for a phase to finish with one of them untested. The session is simply a run with tier='session' and an empty graph. Spawned rather than awaited: launch_phase runs inside the sweep loop, and blocking it for the length of a coding session would stall every other mission. The session's own summary is logged as diagnostics only. Whether the phase actually did anything is still decided downstream by capture and delivery against the repository -- a 0-exit session that pushed nothing was measured at ~5%, so the agent's account can never be the verdict. 406 tests, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
37fac288d2 |
feat(missions): bring the direct session executor onto a live branch
Rescues session_executor from the stranded spike branch. Multi-provider
missions are not needed for now, so the direct path is worth nailing down:
run a mission as one `claude -p` session against its checkout instead of
routing turns through ZeroClaw.
Measured today against a real checkout in the runtime container, using the
executor's exact argv:
direct `claude -p` 7s, file written
via ZeroClaw minutes per turn, and THREE config failures before
it worked at all (no credential in the mission
container; Write/Edit denied; no tools granted --
the last of which COMPLETED a mission having
written nothing)
Each of those failures came from the same root: with claude_cli, ZeroClaw
is a WebSocket-to-subprocess adapter whose own controls (risk profiles,
tool gating, memory) do not reach the subprocess. The adapter adds
failure modes without adding governance.
What ZeroClaw still earns for the rest of the platform is unchanged and
not in question here: interactive chat, the brain, A2A and door identity,
terminal, agent routines, and non-Claude providers.
Not yet wired into phase_runner — this commit only makes the executor
reachable and keeps it building. SessionOutcome::delivered() still
requires a clean exit AND an observed branch, because a 0-exit session
that pushed nothing was measured at ~5%.
405 tests, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|