//! Run a mission phase inside a Firecracker microVM. //! //! The step that makes the whole microVM track load-bearing. Before this, //! `runtime_kind = 'microvm'` placed a mission on a KVM-capable node and then //! nothing executed it — config accepted without a reader, which is one of the //! four seams this project keeps closing. //! //! # The model //! //! **inject → run → collect → destroy**, the same shape copy mode already proved //! for containers, with a VM boundary instead of a namespace boundary. Nothing is //! shared: the checkout goes in as a tar, the work comes back as a tar, and the //! guest's filesystem dies with it. //! //! # Why the agent does not push //! //! `session_executor` tells its agent to commit and push to a branch, and the //! forge is on the VM's egress allow-list, so it could. It must not: //! //! - `mission_delivery::capture_phase_diff_at` is already host-side and diffs the //! collected tree against the recorded clone point, covering committed, staged //! and unstaged work in one pass. Pushing from the guest would add a second, //! untested way for work to arrive. //! - Pushing needs forge credentials in the VM. The point of collecting is that //! the guest never holds them. //! //! So the prompt says explicitly not to push, and the work is collected over the //! same host path the checkout came from — leaving the host directory a //! server-owned staging area with exactly one writer. //! //! # One run row //! //! Like `launch_direct_session`, this creates exactly ONE `topology_runs` row //! (tier `microvm`). The whole downstream lifecycle — `close_finished_phases`, //! evaluation, capture, delivery — keys off those rows, and a second completion //! path would mean two ways for a phase to finish with one of them untested. use cm_domain::NodeId; use uuid::Uuid; use crate::fleet::NodeHub; use crate::microvm_client::MicroVm; /// Resources per phase VM. Generous enough to build: the mission toolchain in /// `agent-claude` includes rustc, and a 512 MB guest OOMs partway through a /// `cargo build` in a way that looks like an agent giving up. const VCPUS: u32 = 4; pub(crate) const MEM_MIB: u32 = 8192; /// Budget for one agent turn inside the VM, matching the container path's. const TURN_SECS: u64 = 3600; /// Where the checkout lands in the guest. Same path as the container path uses, /// so a prompt or a tool that hardcodes it behaves identically either way. const GUEST_REPO: &str = "/mission/repo"; /// A vm id must be `[A-Za-z0-9_-]` — the node rejects anything else, since it /// becomes a path component. Hyphenated uuids qualify; this also keeps the id /// readable in `vm_list` output and in the node's egress log. /// /// `step` distinguishes the nodes of a composed run, whose graph runs several VMs /// for one phase and one iteration. Deliberately DETERMINISTIC rather than /// random: the node refuses to create a vm id that already exists, so if a /// restarted worker resumes a step whose VM is somehow still alive, the second /// attempt fails loudly instead of running a duplicate agent against the same /// checkout. A random id would make that collision invisible and let two VMs /// collect over each other's work. pub(crate) fn vm_id_for(phase_id: Uuid, iteration: i32, step: Option) -> String { let base = format!("m-{}-{}", &phase_id.simple().to_string()[..12], iteration); match step { Some(s) => format!("{base}-s{s}"), None => base, } } /// Ceiling on teammates, stated in the prompt. /// /// Anthropic's own guidance is 3-5, and their research system uses a lead plus /// 3-5 subagents. The cap is in the prompt rather than enforced by us because the /// lead is the thing that decides team size — there is no flag that limits it — /// so the honest options are to tell it the budget or to not know what it spent. const MAX_TEAMMATES: u32 = 4; /// Whether this mission asked for a Claude Code agent team. fn wants_claude_code_team(engine: Option<&str>) -> bool { matches!(engine, Some("claude_code")) } /// Environment that turns agent teams on. Empty for a solo mission. /// /// **MEASURED: agent teams do not form under `claude -p`.** With this flag set and /// an explicit request to "spawn two teammates", the CLI in our image (2.1.223) /// did the work with two SUBAGENTS, wrote both files, and created no /// `~/.claude/teams/` directory at all. The docs allow for it — "Claude may /// sometimes use subagents instead of creating a team" — and headless appears to be /// always. The whole agent-teams feature is described around an interactive agent /// panel, which a print-mode session does not have. /// /// The flag is still set, because it is harmless and costs nothing if a later /// version does support teams non-interactively. What the switch actually buys /// today is the prompt addendum, which does change behaviour: it gets the lead to /// parallelise across files via subagents instead of working through them alone. /// Read the outcome from the subagent count, not from a teammate count. /// /// Kept separate from /// [`crate::mission_runtime::microvm_provider_env`] so the credential path stays /// exactly as narrow as it is — that function is subscription-only by /// construction, and widening it to carry feature flags is how an API key ends up /// back in a VM. fn team_env(engine: Option<&str>) -> Vec<(String, String)> { if !wants_claude_code_team(engine) { return Vec::new(); } vec![( "CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS".to_string(), "1".to_string(), )] } /// What the agent is told. Deliberately not `session_executor::session_prompt`: /// that one instructs a push, and on this path pushing is the host's job. fn vm_prompt(task: &str) -> String { format!( "You are working in the git repository at {GUEST_REPO}.\n\ \n\ TASK\n\ {task}\n\ \n\ HELP AVAILABLE TO YOU\n\ You can delegate with the Agent tool. Two roles exist:\n\ - `verifier` — checks work against the project's own tests. It cannot \ edit files. Use it when you believe you are finished, and treat what it \ reports as the outcome rather than your own impression.\n\ - `explorer` — answers a question about the codebase by reading it, \ without filling your context with search output.\n\ Delegating is optional. For a small, self-contained change, doing it \ yourself is usually better: a handoff costs context and coordination, and \ one careful pass beats an assembly line.\n\ \n\ WHEN THE WORK IS DONE\n\ Leave it in the working tree. Do NOT push, and do not add a remote — \ this machine has no access to the forge. Committing locally is fine but \ not required: the whole tree is collected when you finish, and the work \ is recorded from the repository itself either way.\n\ \n\ If the task cannot be completed as written — a file it refers to does \ not exist, a premise is wrong, the tests cannot run — say so plainly. An \ honest report that the work could not be done is worth more than a tree \ that looks finished.\n" ) } /// The extra paragraph a team mission gets. /// /// Separate from [`vm_prompt`] so a solo mission's prompt is byte-identical to /// what it was before teams existed — the comparison between solo and team is /// only meaningful if the solo side did not also change. fn team_prompt_addendum() -> String { format!( "\nWORKING AS A TEAM\n\ Delegate the independent parts of this task and work them in parallel, \ rather than doing them one after another yourself. Use at most \ {MAX_TEAMMATES}, and fewer when fewer will do: teammates cost tokens \ and coordination, and three focused ones beat five scattered ones.\n\ Split the work so each teammate owns DIFFERENT FILES. Two teammates \ editing one file overwrite each other. Do not split one file's work into \ stages across teammates — a handoff loses context, and sequential phases \ of the same change are better done by one agent in one pass.\n\ Wait for your teammates to finish before you conclude. Their findings are \ the point; a summary written before they report is your own guess.\n" ) } /// Shell-quote for `sh -c`. The guest agent runs one command string, and a task /// description contains apostrophes, quotes and newlines as a matter of course. fn shell_quote(s: &str) -> String { format!("'{}'", s.replace('\'', r"'\''")) } /// The tool that spawns a subagent. /// /// **Verified against `claude --help` and the tool docs in the image we ship /// (2.1.220), not remembered.** It was named `Task` in older versions, and the /// cost of guessing is invisible: an unrecognised entry in `--allowedTools` does /// not error, it simply means the agent has no way to delegate and works alone. /// Nothing in the output says so. const SUBAGENT_TOOL: &str = "Agent"; /// Tools the lead may use. `Agent` is what makes fan-out possible at all — before /// it was added the allowlist was `Read Edit Write Bash`, so Claude Code could not /// spawn a subagent in any of our VMs. const LEAD_TOOLS: &[&str] = &["Read", "Edit", "Write", "Bash", SUBAGENT_TOOL]; /// The rule every subagent carries, in its own prompt. /// /// `--append-subagent-system-prompt` does **not** exist in 2.1.220 despite being /// documented — checked against `--help` in the image — so this is inlined per /// definition instead. That is the better shape anyway: a verifier and an explorer /// need different wording, and a single appended blob would say the same thing to /// both. const NO_SHORTCUTS: &str = "Report what you actually observed. Never present an \ expected result as an observed one, and never weaken, skip or narrow a check \ to make it pass — the work is judged against the repository itself, so a \ shortcut is found later and costs more than an honest failure."; /// Subagent definitions, handed over as `--agents` JSON. /// /// JSON on the command line rather than files, for a specific reason: the guest's /// `/mission/repo` is collected and diffed, so a role definition written into the /// checkout would arrive in the delivered patch as if the agent had authored it. /// `--agents` is session-scoped and touches no disk. /// /// These two are deliberately the only ones. `verifier` is the single multi-agent /// pattern Anthropic endorses for coding work — implement, then check with a /// separate agent that sees only the artifact and the criteria. `explorer` exists /// for context protection, the other justification that survives scrutiny. Roles /// like "tester" or "committer" are absent on purpose: splitting sequential phases /// of the same work is a documented anti-pattern, and it is the shape our pipeline /// templates already have. fn agent_definitions() -> serde_json::Value { serde_json::json!({ "verifier": { "description": "Independently verifies that work is complete and correct. \ Use after implementing something, to check it rather than \ to trust it.", // No Edit and no Write, by construction: an agent that can fix what it // is checking will fix it and report success, and the report is then // about a tree nobody reviewed. // // This restriction is enforced by the tool allowlist, NOT by permission // mode — since 2.1.212 a subagent inherits the parent's permission mode, // so `acceptEdits` reaches it either way. It is also why the image is // pinned to 2.1.223: 2.1.222 fixed background subagents being able to // bypass tool restrictions, and this is the tool restriction in question. "tools": "Read, Grep, Glob, Bash", // Foreground, against the default since 2.1.198. A background verifier // lets the lead carry on and write its report before the check has // finished — the finding would arrive after the conclusion. The whole // point is that the verdict comes from the verifier, so the lead waits. "background": false, "prompt": format!( "You verify work you did not do. You are given an artifact and the \ criteria it must meet, and nothing else — you do not know how it was \ built and you do not need to.\n\ \n\ Run the project's OWN checks, in full. If it has a test suite, run \ the COMPLETE suite, not the tests that look relevant: stopping after \ the first one or two that pass is the most common way a verifier \ reports success on broken work.\n\ \n\ You cannot edit or write files. If something is wrong, say exactly \ what you ran, what you expected and what you saw. {NO_SHORTCUTS}" ), }, "explorer": { "description": "Reads and searches the codebase to answer a specific \ question. Use when finding something out would otherwise \ fill the main context with files and search output.", "tools": "Read, Grep, Glob", "prompt": format!( "You answer one question about this codebase by reading it. Return the \ answer and the paths that support it — not a transcript of your \ search. You cannot modify anything. {NO_SHORTCUTS}" ), }, }) } /// Claude Code writes a separate transcript per subagent, under the session's own /// directory. Its existence is the evidence that delegation happened — measured in /// the shipped image, not inferred: /// /// ```text /// /root/.claude/projects/-w/.jsonl /// /root/.claude/projects/-w//subagents/agent-.jsonl /// ``` /// /// The alternative, `--forward-subagent-text`, is unusable here: it refuses to run /// without `--output-format=stream-json`, which would change how this module reads /// the agent's output entirely. Counting transcripts costs nothing and cannot be /// confused with the lead merely *claiming* it delegated. const SUBAGENT_PROBE: &str = "ls -1 /root/.claude/projects/*/*/subagents/*.jsonl 2>/dev/null | wc -l"; /// How many TEAMMATES the lead spawned, from the team config it maintains. /// /// Teammates are not subagents: they are separate Claude Code sessions sharing a /// task list, and the documented state lives at /// `~/.claude/teams/{team}/config.json` with a `members` array (the lead itself is /// one member, carrying agent type `team-lead`). Counting members and subtracting /// the lead gives teammates. /// /// Documented but NOT yet verified in our image, unlike the subagent transcript /// path — so a zero here means "no evidence found", and the first real team /// mission is what turns this into a fact. It is reported rather than asserted for /// that reason. const TEAMMATE_PROBE: &str = "cat /root/.claude/teams/*/config.json 2>/dev/null \ | tr ',' '\\n' | grep -c '\"name\"' || true"; /// Does the CLI in THIS image accept `--settings`? /// /// Asked of the binary rather than assumed from a version number. The stop gate /// is installed through that flag, and an unknown option is a hard CLI error — /// so without this probe, a Claude Code build that dropped the flag would turn /// every gated phase into a failed one. With it, the gate degrades to absent and /// says so, which is the difference between losing a check and losing the work. const SETTINGS_PROBE: &str = "claude --help 2>&1 | grep -q -- '--settings' && echo SETTINGS-OK"; /// How many times the stop gate refused to let the agent finish. const BLOCKS_PROBE: &str = "cat /root/gate/blocks 2>/dev/null || echo 0"; /// Did the gate give up and release the agent with its condition still failing? /// /// Asked separately from [`BLOCKS_PROBE`] because the count cannot answer it — /// see [`crate::vm_stop_gate::CAPPED_FILE`]. `cat` of a missing file prints /// nothing and exits non-zero, so the `||` supplies the "no" that a gate which /// never capped never wrote. const CAPPED_PROBE: &str = "cat /root/gate/capped 2>/dev/null || echo 0"; /// The command that runs the agent in the guest. /// /// `settings` is the path to the stop-gate settings file, when a gate is /// installed. It lives under `/root`, never in the repo — see [`vm_stop_gate`]. fn agent_command(prompt: &str, settings: Option<&str>) -> String { // --permission-mode acceptEdits, matching the container path: the VM IS the // boundary, so prompting for permission inside it would only mean a turn that // waits for an answer nobody can give. // // NOT `CLAUDE_AGENT_SDK_DISABLE_BUILTIN_AGENTS=1`. It was in the plan, and it // is wrong: measured in the image, it removes EVERY agent type including the // ones `--agents` defines. The lead reported "an empty available-agents list" // after trying four role names and — to its credit — refused to fabricate a // result. A unit test asserting "builtins off is paired with our own roles" // passed the whole time, because the pairing holds in our code and not in the // CLI. So the built-in roles stay available alongside ours. // `tee` rather than a redirect: the file is what the node tails to stream // the turn live, and the exec's own stdout is what becomes `VmOutcome // ::summary`. Redirecting would give us the live view and an empty summary. // `2>&1` first so stderr — where agent CLIs put their progress — is included. format!( "cd {GUEST_REPO} && {{ claude -p --allowedTools {} \ --permission-mode acceptEdits --agents {}{} {} ; }} 2>&1 | tee {GUEST_LOG}", LEAD_TOOLS.join(" "), shell_quote(&agent_definitions().to_string()), match settings { Some(path) => format!(" --settings {}", shell_quote(path)), None => String::new(), }, shell_quote(prompt) ) } /// Where a turn's combined output is teed inside the guest, for the node to /// follow. Under `/root`, never the repo: anything in `/mission/repo` is /// collected and would arrive in the user's delivered diff. pub const GUEST_LOG: &str = "/root/agent.log"; /// Outcome of one phase VM, as observed from outside it. pub struct VmOutcome { /// The agent's closing text. Diagnostic only — never evidence. Whether the /// phase succeeded is decided downstream against the repository. pub summary: String, pub rc: i64, /// Whether the work came back. A turn that ran and could not be collected is /// a failure even if the agent was happy. pub collected: bool, /// How many subagents the lead actually spawned, counted from Claude Code's /// own per-subagent transcripts in the guest. /// /// Observed, not claimed. A lead that says it "had the verifier check this" /// while never spawning one reads identically in its summary; this does not. /// `None` means the probe could not run, which is distinct from zero. pub subagents: Option, /// Teammates the lead spawned, for a mission that asked for a team. `None` /// when no team was requested, or when the probe could not run. pub teammates: Option, /// How many times the stop gate refused to let the agent finish. `None` when /// no gate was installed. Zero means the agent got it right first time, /// which is a different fact from "there was no gate". pub stop_blocks: Option, /// Whether the gate ran out of blocks and let the agent stop anyway, with /// the phase's `done_when_check` still exiting non-zero (or the tree still /// unchanged). `None` when no gate was installed, or the probe could not run /// — neither of which is "it did not cap". /// /// `Some(true)` is a FAILED turn. The gate is the only thing that ever runs /// a `done_when_check`, so if it gave up, nothing downstream will notice. pub released_at_cap: Option, /// What the agent's tools touched, drained from the guest tap. /// /// Empty when no tap was installed, when the hook never fired, or when the /// agent genuinely called nothing. Those are three different facts and this /// field cannot tell them apart — the unmatched-frame log and the install /// error are what separate them. pub tools: Vec, } /// Boot a VM, run the phase in it, collect the result, and destroy it. /// /// `destroy` runs on every exit path. A leaked VM holds an 8 GB sparse rootfs and /// a firecracker process, and the node's orphan sweep is a backstop, not a plan. pub async fn run_phase_in_vm(hub: &NodeHub, p: VmPhase<'_>) -> Result { // Resolved BEFORE the VM boots: a missing subscription token must fail the // phase, not boot a VM whose agent will sit there unauthenticated. let env = crate::mission_runtime::microvm_provider_env(p.backend)?; let vm = MicroVm::new(hub, p.node_id, vm_id_for(p.phase_id, p.iteration, p.step)); let created = vm.create(VCPUS, MEM_MIB, p.backend).await?; // From here on every early return must still destroy the VM, so the work is // one call whose result is held while teardown runs unconditionally. let outcome = run_inside( &vm, &created, p.task, p.repo, p.has_repo, &env, p.team_engine, p.gate, p.run_id, ) .await; if let Err(e) = vm.destroy().await { // Not fatal to the phase — the work may already be collected — but loud, // because the alternative is an 8 GB rootfs nobody is looking for. eprintln!( "microvm_executor: mission {} phase {}: destroy {} FAILED: {e}", p.mission_id, p.phase_id, vm.vm_id() ); } outcome } /// One phase to run in one VM. pub struct VmPhase<'a> { /// The topology run this turn belongs to. Live output is keyed by it, since /// that is what a browser subscribes to. pub run_id: Option, pub node_id: NodeId, pub mission_id: Uuid, pub phase_id: Uuid, /// Which pass. Part of the vm id, so two passes of the same phase cannot /// collide on a node. pub iteration: i32, pub task: &'a str, /// `missions.backend` — which rootfs image. `None` boots the node's default. pub backend: Option<&'a str>, /// The host checkout, injected as a tar and collected back over the same /// path so `mission_delivery` needs no change. pub repo: &'a std::path::Path, /// Whether the mission has a repository at all. /// /// A repo-less mission still gets a `/mission/repo` — the agents need /// somewhere to write and the collect brings it back — but it is an empty /// workspace rather than a checkout. Without this the inject packed a /// directory that does not exist and the readiness probe demanded a `.git` /// that never would, so a repo-less microVM phase failed before the agent /// ran. It is the same `has_repo` `phase_runner` already threads through to /// choose the prompt. pub has_repo: bool, /// `missions.team_engine` — `Some("claude_code")` asks the lead to form a /// team. `None` is solo, which is the default. pub team_engine: Option<&'a str>, /// What must hold before the agent is allowed to stop, installed as a /// Claude Code `Stop` hook inside the guest. `None` leaves the turn exactly /// as it was. pub gate: Option<&'a crate::vm_stop_gate::StopGate>, /// Which node of a composed graph this VM is running, if any. `None` is the /// solo path, where the phase is one VM and the id needs no further /// qualification. Part of the vm id, so the nodes of one phase-iteration /// cannot collide on a fleet node. pub step: Option, } /// Runs one phase (or one graph node of one) in a VM. /// /// A seam with exactly one production implementation, and it exists for one /// reason: the composed executor's defining property is that node 2 sees node 1's /// files, and that property is untestable against real Firecracker in a unit /// test. A fake that models inject → run → collect faithfully can prove it in /// milliseconds, including the negative control where the handoff is broken. #[allow(async_fn_in_trait)] pub trait PhaseVm { /// Boot a VM, run this phase in it, collect the result, destroy it. async fn run(&self, p: VmPhase<'_>) -> Result; } /// The real thing: VMs on the fleet, over the node hub. pub struct HubVms { hub: std::sync::Arc, } impl HubVms { pub fn new(hub: std::sync::Arc) -> Self { Self { hub } } } impl PhaseVm for HubVms { async fn run(&self, p: VmPhase<'_>) -> Result { run_phase_in_vm(&self.hub, p).await } } async fn run_inside( vm: &MicroVm<'_>, created: &serde_json::Value, task: &str, repo: &std::path::Path, // See `VmPhase::has_repo`: a repo-less mission gets an EMPTY workspace at // the same guest path, and is proven present differently. has_repo: bool, env: &[(String, String)], engine: Option<&str>, gate: Option<&crate::vm_stop_gate::StopGate>, // The run this turn's live output belongs to. `None` on paths with no // subscriber, which simply means the node does not follow the log. run_id: Option, ) -> Result { // An agent CLI cannot reach its API without the tunnel, and a turn without // egress does not fail — it hangs, or reports a network error the operator // then has to trace back through three layers. `create` measures both ends; // this is the reader that makes those fields matter. if created.get("egress").and_then(serde_json::Value::as_bool) != Some(true) { return Err(format!( "vm {} has no egress (host={}, guest={}), so the agent could not reach \ its API — refusing to run the phase in it", vm.vm_id(), created.get("egress_host").unwrap_or(&serde_json::Value::Null), created.get("egress_guest").unwrap_or(&serde_json::Value::Null), )); } // The checkout, as a tar. `pack_dir` names the entry `repo`, and the guest // unpacks it under /mission, so it lands at /mission/repo. // // A repo-less mission has no directory to pack. Create it — empty — rather // than skipping the inject: the guest needs the workspace to exist before // the agent writes into it, and creating it host-side means the collect // unpacks back over the same path with no special case. if !has_repo && !repo.exists() { std::fs::create_dir_all(repo) .map_err(|e| format!("create empty workspace {}: {e}", repo.display()))?; } let archive = crate::mission_fs::pack_dir(repo, "repo")?; let injected = archive.len(); vm.inject("/mission", &archive).await?; // Prove the guest actually has the checkout before spending an agent turn on // it. An inject that reports success while landing nothing would otherwise // become an agent reporting that the repository is empty. // What "landed" means depends on what was sent. A checkout is proven by its // `.git`; an empty workspace can only be proven by the directory itself, // and demanding `.git` of it failed every repo-less microVM phase before // the agent got a turn. let want = if has_repo { format!("{GUEST_REPO}/.git") } else { GUEST_REPO.to_string() }; let probe = vm .exec(&format!("test -d {want} && echo REPO-PRESENT"), None, 60, &[]) .await?; if !probe.stdout.contains("REPO-PRESENT") { return Err(format!( "the workspace did not land in the guest ({injected} bytes injected, \ {want} is absent) — rc={} {}", probe.rc, probe.stderr )); } // The team addendum is appended only when the mission asked, so a solo // mission's prompt is byte-identical to what it was before teams existed. let prompt = match wants_claude_code_team(engine) { true => format!("{}{}", vm_prompt(task), team_prompt_addendum()), false => vm_prompt(task), }; let mut turn_env = env.to_vec(); turn_env.extend(team_env(engine)); // The stop gate, installed before the turn. Every failure here is degraded // to "no gate" and logged: the gate makes a phase converge in ONE VM instead // of a second one, so losing it costs money and time — while failing the // turn over it would cost the work. // Asked ONCE, of the binary rather than of a version number, and shared by // both hooks: `--settings` is how either of them gets installed at all. let settings_supported = vm .exec(SETTINGS_PROBE, None, 60, &[]) .await .map(|p| p.stdout.contains("SETTINGS-OK")) .unwrap_or(false); let gate_dir = match gate { None => None, Some(g) => { if !settings_supported { eprintln!( "microvm_executor: {} has no `claude --settings`, so the stop gate \ cannot be installed — this phase runs ungated and its completion \ is judged only after the fact", vm.vm_id() ); None } else { let install = g.install_command(GUEST_REPO, crate::vm_stop_gate::GATE_DIR); match vm.exec(&install, None, 60, &[]).await { Ok(o) if o.rc == 0 => Some(crate::vm_stop_gate::GATE_DIR), Ok(o) => { eprintln!( "microvm_executor: could not install the stop gate on {} \ (rc={}): {} — running ungated", vm.vm_id(), o.rc, o.stderr ); None } Err(e) => { eprintln!( "microvm_executor: could not install the stop gate on {}: {e} \ — running ungated", vm.vm_id() ); None } } } } }; // The tool tap, on the same `--settings` seam as the gate. Best-effort in // exactly the same way: a phase that runs without telemetry is a phase that // still delivers, and failing the turn to protect a picture would be the // wrong trade. let tap_dir = match settings_supported { false => None, true => match vm .exec(&crate::vm_tool_tap::install_command(crate::vm_tool_tap::TAP_DIR), None, 60, &[]) .await { Ok(o) if o.rc == 0 => Some(crate::vm_tool_tap::TAP_DIR), Ok(o) => { eprintln!( "microvm_executor: could not install the tool tap on {} (rc={}): {} \ — this phase's actions will not appear in the World", vm.vm_id(), o.rc, o.stderr ); None } Err(e) => { eprintln!( "microvm_executor: could not install the tool tap on {}: {e} \ — this phase's actions will not appear in the World", vm.vm_id() ); None } }, }; // ONE settings document, written once, carrying whichever hooks installed. // Two writers here is the silent clobber `guest_settings` exists to stop: // whichever ran second would erase the other's hook with no error at all. let settings = match (gate_dir, tap_dir) { (None, None) => None, (g, t) => { let doc = crate::vm_tool_tap::guest_settings(g, t); let cmd = crate::vm_tool_tap::settings_command( crate::vm_tool_tap::SETTINGS_PATH, &doc, ); match vm.exec(&cmd, None, 60, &[]).await { Ok(o) if o.rc == 0 => Some(crate::vm_tool_tap::SETTINGS_PATH.to_string()), other => { eprintln!( "microvm_executor: could not write the guest settings on {} ({other:?}) \ — running with NO hooks: neither the stop gate nor the tool tap", vm.vm_id() ); None } } } }; let out = vm .exec_attributed( &agent_command(&prompt, settings.as_deref()), None, TURN_SECS, &turn_env, run_id, Some(GUEST_LOG), ) .await?; // The tool tap, drained BEFORE collect: it lives in /root, outside the // collected tree, and the VM is destroyed moments later. This is the only // chance to read it. let tools = match tap_dir { None => Vec::new(), Some(_) => match vm.exec(crate::vm_tool_tap::DRAIN_PROBE, None, 60, &[]).await { Ok(p) => crate::vm_tool_tap::parse(&p.stdout), Err(e) => { eprintln!("microvm_executor: tap drain failed on {}: {e}", vm.vm_id()); Vec::new() } }, }; if tap_dir.is_some() && tools.is_empty() { // A tap that installed and drained nothing is the silent case: the // phase looks the same as it did before the tap existed. Say so, or the // next person debugging an empty World has no thread to pull. eprintln!( "microvm_executor: {} installed the tool tap and drained ZERO tool calls — \ either the agent called no tools or `PostToolUse` did not fire in this image", vm.vm_id() ); } // Ask the guest how many subagents ran, before collecting: the transcripts // live in /root, outside the collected tree, so this is the only chance. // Failure to probe is `None`, never 0 — "we could not look" and "it delegated // to nobody" are different facts and only one of them is about the agent. let subagents = match vm.exec(SUBAGENT_PROBE, None, 60, &[]).await { Ok(p) => p.stdout.trim().parse::().ok(), Err(e) => { eprintln!("microvm_executor: subagent probe failed on {}: {e}", vm.vm_id()); None } }; // Teammates, when a team was asked for. A team mission that formed no team is // silently solo otherwise — it would still deliver, still look fine, and the // only difference from a solo run would be the tokens it did not spend. let teammates = match wants_claude_code_team(engine) { false => None, true => match vm.exec(TEAMMATE_PROBE, None, 60, &[]).await { // `members` includes the lead, so teammates = members - 1. Ok(p) => p.stdout.trim().parse::().ok().map(|n| n.saturating_sub(1)), Err(e) => { eprintln!("microvm_executor: teammate probe failed on {}: {e}", vm.vm_id()); None } }, }; // A team mission that fanned out to NOBODY is just a solo run that spent the // same tokens, and nothing else in the output would say so. Judged on the // subagent count, because that is the mechanism print mode actually uses — // teammates never form here (see `team_env`), so a zero teammate count is // expected and not itself a problem. if wants_claude_code_team(engine) && subagents.unwrap_or(0) == 0 { eprintln!( "microvm_executor: {} asked for a team and delegated to nobody — it ran \ SOLO. Print mode satisfies a team request with subagents rather than \ teammates, so check that the Agent tool is in the allowlist and that \ the task is actually divisible.", vm.vm_id() ); } // Collect regardless of the agent's exit code. A turn that failed partway // still wrote files, and throwing them away because the CLI exited non-zero // would discard exactly the work a retry needs to see. let tar = vm.collect(GUEST_REPO, crate::mission_fs::transport_excludes()).await; let collected = match tar { Ok(bytes) => { let parent = repo .parent() .ok_or_else(|| format!("{} has no parent", repo.display()))?; // Unpacked over the SAME host path the checkout came from, so // `capture_phase_diff_at` finds a normal checkout exactly where it // always has and needs no change at all. crate::mission_fs::unpack_into(&bytes, parent)?; true } Err(e) => { // Reported, not swallowed: an uncollected turn is a failed phase even // when the agent said it finished. eprintln!("microvm_executor: collect from {} failed: {e}", vm.vm_id()); false } }; // How many times the gate sent the agent back. Read only when a gate was // actually installed, so `None` never means "it never fired". let stop_blocks = match settings { None => None, Some(_) => match vm.exec(BLOCKS_PROBE, None, 60, &[]).await { Ok(p) => p.stdout.trim().parse::().ok(), Err(e) => { eprintln!("microvm_executor: stop-gate probe failed on {}: {e}", vm.vm_id()); None } }, }; // Probed only when a gate existed, so `None` never reads as "it did not cap". let released_at_cap = match settings { None => None, Some(_) => match vm.exec(CAPPED_PROBE, None, 60, &[]).await { Ok(p) => Some(p.stdout.trim() == "1"), Err(e) => { eprintln!("microvm_executor: cap probe failed on {}: {e}", vm.vm_id()); None } }, }; Ok(VmOutcome { summary: format!("{}{}", out.stdout, out.stderr), rc: out.rc, collected, subagents, teammates, stop_blocks, released_at_cap, tools, }) } #[cfg(test)] mod tests { use super::*; /// The turn must be observable WHILE it runs and still report its output. /// /// `tee`, not `>`: the log file is what the node follows to stream the turn /// live, and the command's own stdout is what becomes `VmOutcome::summary`. /// A redirect would give a live view and an empty summary — which is the /// same "green and empty" shape this codebase keeps finding. #[test] fn a_turn_is_teed_so_it_streams_and_still_reports() { let cmd = agent_command("do the thing", None); assert!(cmd.contains(&format!("tee {GUEST_LOG}")), "{cmd}"); assert!( !cmd.contains(&format!("> {GUEST_LOG}")), "a redirect would empty the summary: {cmd}" ); // stderr must be included: agent CLIs report progress there. assert!(cmd.contains("2>&1"), "{cmd}"); // And the log must live outside the collected tree, or it arrives in the // user's delivered diff. assert!(GUEST_LOG.starts_with("/root/"), "{GUEST_LOG}"); assert!(!GUEST_LOG.starts_with(GUEST_REPO), "{GUEST_LOG}"); } /// The id becomes a path component on the node, which rejects anything /// outside `[A-Za-z0-9_-]` rather than sanitising it. #[test] fn a_vm_id_is_acceptable_to_the_node() { for step in [None, Some(0), Some(11)] { let id = vm_id_for(Uuid::now_v7(), 3, step); assert!( id.chars() .all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'), "{id}" ); assert!(id.len() <= 64, "{id}"); } } /// Two passes of the same phase must not collide: the second would fail with /// "vm already exists" while the first is still running. #[test] fn each_iteration_gets_its_own_vm() { let p = Uuid::now_v7(); assert_ne!(vm_id_for(p, 1, None), vm_id_for(p, 2, None)); } /// A composed run boots one VM per graph node within a single phase and /// iteration. Without the step in the id they would all be the same vm, and /// the second node would fail with "vm already exists" — or worse, if the /// first were already destroyed, succeed while looking like a re-run. #[test] fn each_graph_node_gets_its_own_vm() { let p = Uuid::now_v7(); assert_ne!(vm_id_for(p, 1, Some(0)), vm_id_for(p, 1, Some(1))); // And a composed node never collides with the solo id for the same pass. assert_ne!(vm_id_for(p, 1, Some(0)), vm_id_for(p, 1, None)); // Deterministic: resuming the same step asks for the same vm, which is // what makes a still-live duplicate fail loudly on the node. assert_eq!(vm_id_for(p, 1, Some(2)), vm_id_for(p, 1, Some(2))); } /// A task description contains quotes and newlines as a matter of course, /// and the whole command is handed to `sh -c` in the guest. #[test] fn a_quoted_task_cannot_break_out_of_the_command() { let nasty = "it's a 'test'; rm -rf /\n$(whoami) `id` \"quoted\""; let cmd = agent_command(&vm_prompt(nasty), None); // Everything after the opening quote of the prompt is inside it: the only // way out of a single-quoted string is a quote, and each one is escaped. let body = cmd.split_once('\'').expect("a quoted argument").1; for danger in ["; rm -rf /", "$(whoami)", "`id`"] { let at = body.find(danger).expect("the text is still present"); let before = &body[..at]; // An odd number of unescaped quotes before it would mean it had // escaped the quoting. let unescaped = before.matches('\'').count() - before.matches(r"'\''").count() * 3; assert_eq!(unescaped % 2, 0, "{danger} is not quoted in: {cmd}"); } } /// The regression that blocked fan-out entirely: the allowlist was /// `Read Edit Write Bash`, so Claude Code had no way to spawn a subagent in any /// of our VMs. An unrecognised or missing tool name does not error — the agent /// just works alone and nothing says so — which is why this is asserted rather /// than assumed. #[test] fn the_lead_can_spawn_subagents_at_all() { assert!( LEAD_TOOLS.contains(&SUBAGENT_TOOL), "without {SUBAGENT_TOOL} in the allowlist there is no delegation, silently" ); let cmd = agent_command(&vm_prompt("t"), None); assert!(cmd.contains(&format!("--allowedTools {}", LEAD_TOOLS.join(" "))), "{cmd}"); // `--forward-subagent-text` is deliberately absent: it refuses to run // without `--output-format=stream-json`, which would change how this // module reads output. Evidence comes from the per-subagent transcript // instead. assert!(!cmd.contains("--forward-subagent-text"), "{cmd}"); assert!( SUBAGENT_PROBE.contains("subagents"), "the probe must look at the per-subagent transcripts: {SUBAGENT_PROBE}" ); } /// Our definitions must exist and be well-formed, because they are the only /// roles we control. /// /// This test used to assert that disabling the built-in roles was "paired with" /// defining our own. It passed while the feature was broken: setting /// `CLAUDE_AGENT_SDK_DISABLE_BUILTIN_AGENTS=1` removes OUR definitions too, so /// the pairing held in this file and nowhere else. The lesson is in the code /// now (see `agent_command`), and what is left here is the part a unit test can /// actually speak to. #[test] fn every_role_we_define_is_well_formed() { let defs = agent_definitions(); let defs = defs.as_object().expect("an object of name → definition"); assert!(!defs.is_empty(), "no roles defined means nothing to delegate to"); for (name, d) in defs { assert!( d.get("description").and_then(|v| v.as_str()).is_some_and(|s| !s.is_empty()), "{name} needs a description — it is what Claude matches on to delegate" ); assert!( d.get("prompt").and_then(|v| v.as_str()).is_some_and(|s| !s.is_empty()), "{name} needs a prompt" ); } } /// A verifier that can edit will fix what it was asked to check and then /// report success, and the report is about a tree nobody reviewed. #[test] fn the_verifier_cannot_modify_what_it_checks() { let defs = agent_definitions(); let tools = defs["verifier"]["tools"].as_str().expect("a tools allowlist"); for forbidden in ["Edit", "Write"] { assert!( !tools.contains(forbidden), "the verifier must not have {forbidden}: {tools}" ); } assert!(tools.contains("Bash"), "it has to be able to run the suite: {tools}"); // Subagents default to background since 2.1.198. A background verifier lets // the lead write its conclusion before the check finishes, so the finding // would arrive after the report that was supposed to contain it. assert_eq!( defs["verifier"]["background"], serde_json::json!(false), "the lead must wait for its verifier" ); } /// The early-victory failure: a verifier that stops after the first passing /// test reports success on broken work. Anthropic names it, and we have paid /// for it once already. #[test] fn the_verifier_is_told_to_run_the_whole_suite() { let defs = agent_definitions(); let p = defs["verifier"]["prompt"].as_str().unwrap().to_ascii_lowercase(); assert!(p.contains("complete suite"), "{p}"); assert!(p.contains(&NO_SHORTCUTS[..40].to_ascii_lowercase()), "anti-shortcut rule missing"); } /// The definitions travel as ONE shell argument through `sh -c`, and they /// contain double quotes, braces and apostrophes ("the project's OWN checks"). /// A quoting bug here would hand `claude` a fragment of JSON, which it would /// reject — leaving the lead with no roles and no fan-out. /// /// Tested by round-tripping rather than by banning apostrophes: the first /// version of this test asserted the JSON contained none, which failed on /// correctly-quoted text and said nothing about whether the quoting worked. #[test] fn the_agent_definitions_survive_shell_quoting() { /// Undo `shell_quote`, the way `sh` would. fn unquote(s: &str) -> String { let inner = s .strip_prefix('\'') .and_then(|s| s.strip_suffix('\'')) .expect("a single-quoted argument"); inner.replace(r"'\''", "'") } let json = agent_definitions().to_string(); let quoted = shell_quote(&json); assert_eq!(unquote("ed), json, "the JSON did not survive quoting"); // And it is still valid JSON on the other side. let back: serde_json::Value = serde_json::from_str(&unquote("ed)).expect("valid JSON after quoting"); assert!(back.get("verifier").is_some(), "{back}"); assert!(agent_command(&vm_prompt("task"), None).contains(&format!("--agents {quoted}"))); } /// The tier this executor inserts must NOT be one the topology worker drives. /// A microvm run is owned by its own spawned task for its whole life; if the /// worker also considers it fair game, it requeues it at 180s, fails it on a /// graph it was never meant to parse, and orphans a live VM. Mission 019fd43e /// died at 210 seconds that way. #[test] fn a_microvm_run_is_not_worker_driven() { assert!( !cm_db::repo::topology_runs::WORKER_DRIVEN_TIERS.contains(&"microvm"), "the worker must not claim, requeue or reap a self-driven run" ); // Same exposure, same reason. assert!(!cm_db::repo::topology_runs::WORKER_DRIVEN_TIERS.contains(&"session")); // And the tiers it does drive are still there, or nothing runs at all. for t in ["team", "swarm"] { assert!(cm_db::repo::topology_runs::WORKER_DRIVEN_TIERS.contains(&t), "{t}"); } // The composed tier is the opposite case and must not be confused with // the solo one: its durability comes FROM being worker-driven. assert!(cm_db::repo::topology_runs::WORKER_DRIVEN_TIERS.contains(&"microvm_graph")); // But the 15-minute stuck-run reaper must not touch it: one of its nodes // is a whole agent session, so journaling nothing for 15 minutes is what // a healthy composed run looks like. assert!( !cm_db::repo::topology_runs::REAPABLE_TIERS.contains(&"microvm_graph"), "the reaper would kill a healthy composed run and orphan its VM" ); for t in cm_db::repo::topology_runs::REAPABLE_TIERS { assert!( cm_db::repo::topology_runs::WORKER_DRIVEN_TIERS.contains(t), "{t} is reapable but not worker-driven" ); } } /// Solo is the default, and asking for a team must be explicit. Multi-agent /// costs 3-10x the tokens and is often slower, so a mission that said nothing /// must not get one. #[test] fn only_an_explicit_request_forms_a_team() { assert!(wants_claude_code_team(Some("claude_code"))); for engine in [None, Some(""), Some("zeroclaw"), Some("solo"), Some("CLAUDE_CODE")] { assert!(!wants_claude_code_team(engine), "{engine:?} must not form a team"); } } /// The flag is what enables agent teams in the CLI, and it must not leak into /// a solo mission — a solo run has to stay byte-identical to what it was /// before teams existed, or the comparison between the two is meaningless. #[test] fn the_team_flag_reaches_only_a_team_mission() { let on = team_env(Some("claude_code")); assert_eq!(on.len(), 1); assert_eq!(on[0].0, "CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS"); assert_eq!(on[0].1, "1"); assert!(team_env(None).is_empty(), "a solo mission gets no team flag"); assert!(team_env(Some("zeroclaw")).is_empty()); } /// A solo prompt must not mention teammates, and a team prompt must state the /// cap — the lead decides its own size and there is no flag that limits it, so /// the prompt is the only place the budget can be said. #[test] fn the_team_addendum_states_the_cap_and_stays_out_of_solo_prompts() { let solo = vm_prompt("t"); assert!(!solo.to_lowercase().contains("teammate"), "{solo}"); let addendum = team_prompt_addendum(); assert!(addendum.contains(&MAX_TEAMMATES.to_string()), "{addendum}"); // The anti-patterns worth naming, per Anthropic's own guidance: two // teammates in one file overwrite each other, and splitting one change // into stages across teammates loses context at every handoff. assert!(addendum.contains("DIFFERENT FILES"), "{addendum}"); assert!(addendum.to_lowercase().contains("wait for your teammates"), "{addendum}"); // Print mode delivers this as subagents, so the addendum must not depend on // teammates existing — it asks for parallel delegation, whatever the // mechanism turns out to be. assert!(addendum.contains("Delegate"), "{addendum}"); } /// The teammate probe reads the team config, not the subagent transcripts — /// teammates and subagents are different things and counting one as the other /// would report a team that never formed. #[test] fn the_two_fan_out_probes_look_in_different_places() { assert!(SUBAGENT_PROBE.contains("subagents")); assert!(TEAMMATE_PROBE.contains("teams")); assert!(!TEAMMATE_PROBE.contains("subagents")); } /// An ungated phase's command must be byte-identical to what it was before /// the stop gate existed. Most phases are gated, so the ungated path is the /// one nobody would notice breaking. #[test] fn a_phase_without_a_gate_gets_the_command_it_always_had() { let cmd = agent_command(&vm_prompt("t"), None); assert!(!cmd.contains("--settings"), "{cmd}"); let gated = agent_command(&vm_prompt("t"), Some("/root/gate/settings.json")); assert!(gated.contains("--settings '/root/gate/settings.json'"), "{gated}"); // The flag is the ONLY difference — a gate must not quietly change the // tools, the permission mode or the roles. assert_eq!( gated.replace(" --settings '/root/gate/settings.json'", ""), cmd ); } /// The gate is installed through `--settings`, so whether the CLI in the /// image HAS that flag is a fact about the image, not about our plan. An /// unknown option is a hard error, which would turn every gated phase into a /// failed one — hence a probe rather than a version check. #[test] fn the_settings_flag_is_probed_before_it_is_used() { assert!(SETTINGS_PROBE.contains("--help"), "{SETTINGS_PROBE}"); assert!(SETTINGS_PROBE.contains("--settings"), "{SETTINGS_PROBE}"); assert!(SETTINGS_PROBE.contains("SETTINGS-OK"), "{SETTINGS_PROBE}"); // And the evidence probe reads the gate's own counter, outside the tree // that gets collected and diffed. assert!(BLOCKS_PROBE.contains(crate::vm_stop_gate::GATE_DIR), "{BLOCKS_PROBE}"); assert!(!BLOCKS_PROBE.contains(GUEST_REPO), "{BLOCKS_PROBE}"); } /// EVERY VM turn is offered delegation, composed nodes included. /// /// A composed node's task text is built by `microvm_turn_executor` and then /// wrapped by this same `vm_prompt` inside `run_inside`, so one prompt /// builder serves both paths. Composed runs report `subagents: 0` so far, /// and this is what says that is the TASKS being small rather than the /// capability being absent — if someone gave composed nodes their own /// prompt without the offer, the difference would otherwise show up only as /// a count nobody was watching. #[test] fn every_vm_turn_is_offered_the_same_help() { let p = vm_prompt("do the thing"); assert!(p.contains("Agent tool"), "{p}"); for role in agent_definitions().as_object().unwrap().keys() { assert!(p.contains(role.as_str()), "the prompt must name the `{role}` role: {p}"); } } /// The agent must not be told to push: delivery is host-side, and the guest /// deliberately holds no forge credentials. #[test] fn the_prompt_does_not_ask_the_agent_to_push() { let p = vm_prompt("do the thing"); assert!(p.contains("Do NOT push"), "{p}"); assert!( !p.contains("push to a new branch"), "the container path's push instruction must not leak in: {p}" ); } }