The World could draw a mission's shape but nothing about the work. The
detail existed only as prose in checkpoint.log and model output, where a
tool name is indistinguishable from an agent *talking about* a tool — so
it was never parsed, deliberately. `mission_events` is the structured
channel that replaces it.
Three taps, one table:
- Container tier: the `_ => {}` at the end of topology_exec's typed frame
stream now matches `tool_call` and reads the tool's JSON ARGUMENTS for a
path. Never the prose summary — a path scraped from a sentence would put
files on the map that no agent opened, and the test proves a Grep whose
summary says "src/main.rs" produces no file touch. The frame name itself
is unverified, so the same commit ships an unmatched-frame-type
histogram: a tap that matches nothing looks exactly like a mission that
used no tools, and this is how one gw-04 run names the real frame.
- microVM tier: a `PostToolUse` hook, the seam vm_stop_gate already proved
fires under `claude -p`. It copies stdin to /root/tap and exits 0
unconditionally — a non-zero PostToolUse hook talks back to the model,
which would turn the observer into a participant. Drained before collect,
since the VM is destroyed moments later.
- Phase transitions: five identical copies of the pending→running UPDATE
became one `mark_phase_running`, and `close_finished_phases` grew
RETURNING. Its CASE decides each phase's status inside SQL from rows the
statement does not change, so it cannot be re-derived afterwards without
writing that CASE twice — without RETURNING it emits zero phase.completed
and reports success.
The settings.json hazard the plan called out: the stop gate wrote the
WHOLE document, so a second hook writer would have silently erased it and
a coding phase would then complete having written nothing — the exact
failure the gate exists to catch. There is now one composer,
`vm_tool_tap::guest_settings`, one writer, and a source-walk test that
fails if anything else writes a settings document.
`mission_events.run_id` carries no FK on purpose: phase_runner DELETEs
topology_runs on retry, and a cascade would erase a phase's whole history
the moment it retried — silently, since a cascade is not an error.
world.rs streams it with a cursor that separates backfill from motion.
Everything already in the table when a subscriber arrives is drawn as
settled history; only what lands afterwards animates. Otherwise opening a
finished mission replays an hour of tool calls as a burst storm.
Bounded twice: 400 events per phase (enforced inside the INSERT, since
two concurrent taps would each read a count below the cap) and a 7-day
retention sweep in mission_gc.
Co-Authored-By: Claude Opus 5 <[email protected]>
1175 lines
55 KiB
Rust
1175 lines
55 KiB
Rust
//! Run a mission phase inside a Firecracker microVM.
|
|
//!
|
|
//! The step that makes the whole microVM track load-bearing. Before this,
|
|
//! `runtime_kind = 'microvm'` placed a mission on a KVM-capable node and then
|
|
//! nothing executed it — config accepted without a reader, which is one of the
|
|
//! four seams this project keeps closing.
|
|
//!
|
|
//! # The model
|
|
//!
|
|
//! **inject → run → collect → destroy**, the same shape copy mode already proved
|
|
//! for containers, with a VM boundary instead of a namespace boundary. Nothing is
|
|
//! shared: the checkout goes in as a tar, the work comes back as a tar, and the
|
|
//! guest's filesystem dies with it.
|
|
//!
|
|
//! # Why the agent does not push
|
|
//!
|
|
//! `session_executor` tells its agent to commit and push to a branch, and the
|
|
//! forge is on the VM's egress allow-list, so it could. It must not:
|
|
//!
|
|
//! - `mission_delivery::capture_phase_diff_at` is already host-side and diffs the
|
|
//! collected tree against the recorded clone point, covering committed, staged
|
|
//! and unstaged work in one pass. Pushing from the guest would add a second,
|
|
//! untested way for work to arrive.
|
|
//! - Pushing needs forge credentials in the VM. The point of collecting is that
|
|
//! the guest never holds them.
|
|
//!
|
|
//! So the prompt says explicitly not to push, and the work is collected over the
|
|
//! same host path the checkout came from — leaving the host directory a
|
|
//! server-owned staging area with exactly one writer.
|
|
//!
|
|
//! # One run row
|
|
//!
|
|
//! Like `launch_direct_session`, this creates exactly ONE `topology_runs` row
|
|
//! (tier `microvm`). The whole downstream lifecycle — `close_finished_phases`,
|
|
//! evaluation, capture, delivery — keys off those rows, and a second completion
|
|
//! path would mean two ways for a phase to finish with one of them untested.
|
|
|
|
use cm_domain::NodeId;
|
|
use uuid::Uuid;
|
|
|
|
use crate::fleet::NodeHub;
|
|
use crate::microvm_client::MicroVm;
|
|
|
|
/// Resources per phase VM. Generous enough to build: the mission toolchain in
|
|
/// `agent-claude` includes rustc, and a 512 MB guest OOMs partway through a
|
|
/// `cargo build` in a way that looks like an agent giving up.
|
|
const VCPUS: u32 = 4;
|
|
pub(crate) const MEM_MIB: u32 = 8192;
|
|
|
|
/// Budget for one agent turn inside the VM, matching the container path's.
|
|
const TURN_SECS: u64 = 3600;
|
|
|
|
/// Where the checkout lands in the guest. Same path as the container path uses,
|
|
/// so a prompt or a tool that hardcodes it behaves identically either way.
|
|
const GUEST_REPO: &str = "/mission/repo";
|
|
|
|
/// A vm id must be `[A-Za-z0-9_-]` — the node rejects anything else, since it
|
|
/// becomes a path component. Hyphenated uuids qualify; this also keeps the id
|
|
/// readable in `vm_list` output and in the node's egress log.
|
|
///
|
|
/// `step` distinguishes the nodes of a composed run, whose graph runs several VMs
|
|
/// for one phase and one iteration. Deliberately DETERMINISTIC rather than
|
|
/// random: the node refuses to create a vm id that already exists, so if a
|
|
/// restarted worker resumes a step whose VM is somehow still alive, the second
|
|
/// attempt fails loudly instead of running a duplicate agent against the same
|
|
/// checkout. A random id would make that collision invisible and let two VMs
|
|
/// collect over each other's work.
|
|
pub(crate) fn vm_id_for(phase_id: Uuid, iteration: i32, step: Option<u32>) -> String {
|
|
let base = format!("m-{}-{}", &phase_id.simple().to_string()[..12], iteration);
|
|
match step {
|
|
Some(s) => format!("{base}-s{s}"),
|
|
None => base,
|
|
}
|
|
}
|
|
|
|
/// Ceiling on teammates, stated in the prompt.
|
|
///
|
|
/// Anthropic's own guidance is 3-5, and their research system uses a lead plus
|
|
/// 3-5 subagents. The cap is in the prompt rather than enforced by us because the
|
|
/// lead is the thing that decides team size — there is no flag that limits it —
|
|
/// so the honest options are to tell it the budget or to not know what it spent.
|
|
const MAX_TEAMMATES: u32 = 4;
|
|
|
|
/// Whether this mission asked for a Claude Code agent team.
|
|
fn wants_claude_code_team(engine: Option<&str>) -> bool {
|
|
matches!(engine, Some("claude_code"))
|
|
}
|
|
|
|
/// Environment that turns agent teams on. Empty for a solo mission.
|
|
///
|
|
/// **MEASURED: agent teams do not form under `claude -p`.** With this flag set and
|
|
/// an explicit request to "spawn two teammates", the CLI in our image (2.1.223)
|
|
/// did the work with two SUBAGENTS, wrote both files, and created no
|
|
/// `~/.claude/teams/` directory at all. The docs allow for it — "Claude may
|
|
/// sometimes use subagents instead of creating a team" — and headless appears to be
|
|
/// always. The whole agent-teams feature is described around an interactive agent
|
|
/// panel, which a print-mode session does not have.
|
|
///
|
|
/// The flag is still set, because it is harmless and costs nothing if a later
|
|
/// version does support teams non-interactively. What the switch actually buys
|
|
/// today is the prompt addendum, which does change behaviour: it gets the lead to
|
|
/// parallelise across files via subagents instead of working through them alone.
|
|
/// Read the outcome from the subagent count, not from a teammate count.
|
|
///
|
|
/// Kept separate from
|
|
/// [`crate::mission_runtime::microvm_provider_env`] so the credential path stays
|
|
/// exactly as narrow as it is — that function is subscription-only by
|
|
/// construction, and widening it to carry feature flags is how an API key ends up
|
|
/// back in a VM.
|
|
fn team_env(engine: Option<&str>) -> Vec<(String, String)> {
|
|
if !wants_claude_code_team(engine) {
|
|
return Vec::new();
|
|
}
|
|
vec![(
|
|
"CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS".to_string(),
|
|
"1".to_string(),
|
|
)]
|
|
}
|
|
|
|
/// What the agent is told. Deliberately not `session_executor::session_prompt`:
|
|
/// that one instructs a push, and on this path pushing is the host's job.
|
|
fn vm_prompt(task: &str) -> String {
|
|
format!(
|
|
"You are working in the git repository at {GUEST_REPO}.\n\
|
|
\n\
|
|
TASK\n\
|
|
{task}\n\
|
|
\n\
|
|
HELP AVAILABLE TO YOU\n\
|
|
You can delegate with the Agent tool. Two roles exist:\n\
|
|
- `verifier` — checks work against the project's own tests. It cannot \
|
|
edit files. Use it when you believe you are finished, and treat what it \
|
|
reports as the outcome rather than your own impression.\n\
|
|
- `explorer` — answers a question about the codebase by reading it, \
|
|
without filling your context with search output.\n\
|
|
Delegating is optional. For a small, self-contained change, doing it \
|
|
yourself is usually better: a handoff costs context and coordination, and \
|
|
one careful pass beats an assembly line.\n\
|
|
\n\
|
|
WHEN THE WORK IS DONE\n\
|
|
Leave it in the working tree. Do NOT push, and do not add a remote — \
|
|
this machine has no access to the forge. Committing locally is fine but \
|
|
not required: the whole tree is collected when you finish, and the work \
|
|
is recorded from the repository itself either way.\n\
|
|
\n\
|
|
If the task cannot be completed as written — a file it refers to does \
|
|
not exist, a premise is wrong, the tests cannot run — say so plainly. An \
|
|
honest report that the work could not be done is worth more than a tree \
|
|
that looks finished.\n"
|
|
)
|
|
}
|
|
|
|
/// The extra paragraph a team mission gets.
|
|
///
|
|
/// Separate from [`vm_prompt`] so a solo mission's prompt is byte-identical to
|
|
/// what it was before teams existed — the comparison between solo and team is
|
|
/// only meaningful if the solo side did not also change.
|
|
fn team_prompt_addendum() -> String {
|
|
format!(
|
|
"\nWORKING AS A TEAM\n\
|
|
Delegate the independent parts of this task and work them in parallel, \
|
|
rather than doing them one after another yourself. Use at most \
|
|
{MAX_TEAMMATES}, and fewer when fewer will do: teammates cost tokens \
|
|
and coordination, and three focused ones beat five scattered ones.\n\
|
|
Split the work so each teammate owns DIFFERENT FILES. Two teammates \
|
|
editing one file overwrite each other. Do not split one file's work into \
|
|
stages across teammates — a handoff loses context, and sequential phases \
|
|
of the same change are better done by one agent in one pass.\n\
|
|
Wait for your teammates to finish before you conclude. Their findings are \
|
|
the point; a summary written before they report is your own guess.\n"
|
|
)
|
|
}
|
|
|
|
/// Shell-quote for `sh -c`. The guest agent runs one command string, and a task
|
|
/// description contains apostrophes, quotes and newlines as a matter of course.
|
|
fn shell_quote(s: &str) -> String {
|
|
format!("'{}'", s.replace('\'', r"'\''"))
|
|
}
|
|
|
|
/// The tool that spawns a subagent.
|
|
///
|
|
/// **Verified against `claude --help` and the tool docs in the image we ship
|
|
/// (2.1.220), not remembered.** It was named `Task` in older versions, and the
|
|
/// cost of guessing is invisible: an unrecognised entry in `--allowedTools` does
|
|
/// not error, it simply means the agent has no way to delegate and works alone.
|
|
/// Nothing in the output says so.
|
|
const SUBAGENT_TOOL: &str = "Agent";
|
|
|
|
/// Tools the lead may use. `Agent` is what makes fan-out possible at all — before
|
|
/// it was added the allowlist was `Read Edit Write Bash`, so Claude Code could not
|
|
/// spawn a subagent in any of our VMs.
|
|
const LEAD_TOOLS: &[&str] = &["Read", "Edit", "Write", "Bash", SUBAGENT_TOOL];
|
|
|
|
/// The rule every subagent carries, in its own prompt.
|
|
///
|
|
/// `--append-subagent-system-prompt` does **not** exist in 2.1.220 despite being
|
|
/// documented — checked against `--help` in the image — so this is inlined per
|
|
/// definition instead. That is the better shape anyway: a verifier and an explorer
|
|
/// need different wording, and a single appended blob would say the same thing to
|
|
/// both.
|
|
const NO_SHORTCUTS: &str = "Report what you actually observed. Never present an \
|
|
expected result as an observed one, and never weaken, skip or narrow a check \
|
|
to make it pass — the work is judged against the repository itself, so a \
|
|
shortcut is found later and costs more than an honest failure.";
|
|
|
|
/// Subagent definitions, handed over as `--agents` JSON.
|
|
///
|
|
/// JSON on the command line rather than files, for a specific reason: the guest's
|
|
/// `/mission/repo` is collected and diffed, so a role definition written into the
|
|
/// checkout would arrive in the delivered patch as if the agent had authored it.
|
|
/// `--agents` is session-scoped and touches no disk.
|
|
///
|
|
/// These two are deliberately the only ones. `verifier` is the single multi-agent
|
|
/// pattern Anthropic endorses for coding work — implement, then check with a
|
|
/// separate agent that sees only the artifact and the criteria. `explorer` exists
|
|
/// for context protection, the other justification that survives scrutiny. Roles
|
|
/// like "tester" or "committer" are absent on purpose: splitting sequential phases
|
|
/// of the same work is a documented anti-pattern, and it is the shape our pipeline
|
|
/// templates already have.
|
|
fn agent_definitions() -> serde_json::Value {
|
|
serde_json::json!({
|
|
"verifier": {
|
|
"description": "Independently verifies that work is complete and correct. \
|
|
Use after implementing something, to check it rather than \
|
|
to trust it.",
|
|
// No Edit and no Write, by construction: an agent that can fix what it
|
|
// is checking will fix it and report success, and the report is then
|
|
// about a tree nobody reviewed.
|
|
//
|
|
// This restriction is enforced by the tool allowlist, NOT by permission
|
|
// mode — since 2.1.212 a subagent inherits the parent's permission mode,
|
|
// so `acceptEdits` reaches it either way. It is also why the image is
|
|
// pinned to 2.1.223: 2.1.222 fixed background subagents being able to
|
|
// bypass tool restrictions, and this is the tool restriction in question.
|
|
"tools": "Read, Grep, Glob, Bash",
|
|
// Foreground, against the default since 2.1.198. A background verifier
|
|
// lets the lead carry on and write its report before the check has
|
|
// finished — the finding would arrive after the conclusion. The whole
|
|
// point is that the verdict comes from the verifier, so the lead waits.
|
|
"background": false,
|
|
"prompt": format!(
|
|
"You verify work you did not do. You are given an artifact and the \
|
|
criteria it must meet, and nothing else — you do not know how it was \
|
|
built and you do not need to.\n\
|
|
\n\
|
|
Run the project's OWN checks, in full. If it has a test suite, run \
|
|
the COMPLETE suite, not the tests that look relevant: stopping after \
|
|
the first one or two that pass is the most common way a verifier \
|
|
reports success on broken work.\n\
|
|
\n\
|
|
You cannot edit or write files. If something is wrong, say exactly \
|
|
what you ran, what you expected and what you saw. {NO_SHORTCUTS}"
|
|
),
|
|
},
|
|
"explorer": {
|
|
"description": "Reads and searches the codebase to answer a specific \
|
|
question. Use when finding something out would otherwise \
|
|
fill the main context with files and search output.",
|
|
"tools": "Read, Grep, Glob",
|
|
"prompt": format!(
|
|
"You answer one question about this codebase by reading it. Return the \
|
|
answer and the paths that support it — not a transcript of your \
|
|
search. You cannot modify anything. {NO_SHORTCUTS}"
|
|
),
|
|
},
|
|
})
|
|
}
|
|
|
|
/// Claude Code writes a separate transcript per subagent, under the session's own
|
|
/// directory. Its existence is the evidence that delegation happened — measured in
|
|
/// the shipped image, not inferred:
|
|
///
|
|
/// ```text
|
|
/// /root/.claude/projects/-w/<session>.jsonl
|
|
/// /root/.claude/projects/-w/<session>/subagents/agent-<id>.jsonl
|
|
/// ```
|
|
///
|
|
/// The alternative, `--forward-subagent-text`, is unusable here: it refuses to run
|
|
/// without `--output-format=stream-json`, which would change how this module reads
|
|
/// the agent's output entirely. Counting transcripts costs nothing and cannot be
|
|
/// confused with the lead merely *claiming* it delegated.
|
|
const SUBAGENT_PROBE: &str =
|
|
"ls -1 /root/.claude/projects/*/*/subagents/*.jsonl 2>/dev/null | wc -l";
|
|
|
|
/// How many TEAMMATES the lead spawned, from the team config it maintains.
|
|
///
|
|
/// Teammates are not subagents: they are separate Claude Code sessions sharing a
|
|
/// task list, and the documented state lives at
|
|
/// `~/.claude/teams/{team}/config.json` with a `members` array (the lead itself is
|
|
/// one member, carrying agent type `team-lead`). Counting members and subtracting
|
|
/// the lead gives teammates.
|
|
///
|
|
/// Documented but NOT yet verified in our image, unlike the subagent transcript
|
|
/// path — so a zero here means "no evidence found", and the first real team
|
|
/// mission is what turns this into a fact. It is reported rather than asserted for
|
|
/// that reason.
|
|
const TEAMMATE_PROBE: &str = "cat /root/.claude/teams/*/config.json 2>/dev/null \
|
|
| tr ',' '\\n' | grep -c '\"name\"' || true";
|
|
|
|
/// Does the CLI in THIS image accept `--settings`?
|
|
///
|
|
/// Asked of the binary rather than assumed from a version number. The stop gate
|
|
/// is installed through that flag, and an unknown option is a hard CLI error —
|
|
/// so without this probe, a Claude Code build that dropped the flag would turn
|
|
/// every gated phase into a failed one. With it, the gate degrades to absent and
|
|
/// says so, which is the difference between losing a check and losing the work.
|
|
const SETTINGS_PROBE: &str = "claude --help 2>&1 | grep -q -- '--settings' && echo SETTINGS-OK";
|
|
|
|
/// How many times the stop gate refused to let the agent finish.
|
|
const BLOCKS_PROBE: &str = "cat /root/gate/blocks 2>/dev/null || echo 0";
|
|
|
|
/// Did the gate give up and release the agent with its condition still failing?
|
|
///
|
|
/// Asked separately from [`BLOCKS_PROBE`] because the count cannot answer it —
|
|
/// see [`crate::vm_stop_gate::CAPPED_FILE`]. `cat` of a missing file prints
|
|
/// nothing and exits non-zero, so the `||` supplies the "no" that a gate which
|
|
/// never capped never wrote.
|
|
const CAPPED_PROBE: &str = "cat /root/gate/capped 2>/dev/null || echo 0";
|
|
|
|
/// The command that runs the agent in the guest.
|
|
///
|
|
/// `settings` is the path to the stop-gate settings file, when a gate is
|
|
/// installed. It lives under `/root`, never in the repo — see [`vm_stop_gate`].
|
|
fn agent_command(prompt: &str, settings: Option<&str>) -> String {
|
|
// --permission-mode acceptEdits, matching the container path: the VM IS the
|
|
// boundary, so prompting for permission inside it would only mean a turn that
|
|
// waits for an answer nobody can give.
|
|
//
|
|
// NOT `CLAUDE_AGENT_SDK_DISABLE_BUILTIN_AGENTS=1`. It was in the plan, and it
|
|
// is wrong: measured in the image, it removes EVERY agent type including the
|
|
// ones `--agents` defines. The lead reported "an empty available-agents list"
|
|
// after trying four role names and — to its credit — refused to fabricate a
|
|
// result. A unit test asserting "builtins off is paired with our own roles"
|
|
// passed the whole time, because the pairing holds in our code and not in the
|
|
// CLI. So the built-in roles stay available alongside ours.
|
|
// `tee` rather than a redirect: the file is what the node tails to stream
|
|
// the turn live, and the exec's own stdout is what becomes `VmOutcome
|
|
// ::summary`. Redirecting would give us the live view and an empty summary.
|
|
// `2>&1` first so stderr — where agent CLIs put their progress — is included.
|
|
format!(
|
|
"cd {GUEST_REPO} && {{ claude -p --allowedTools {} \
|
|
--permission-mode acceptEdits --agents {}{} {} ; }} 2>&1 | tee {GUEST_LOG}",
|
|
LEAD_TOOLS.join(" "),
|
|
shell_quote(&agent_definitions().to_string()),
|
|
match settings {
|
|
Some(path) => format!(" --settings {}", shell_quote(path)),
|
|
None => String::new(),
|
|
},
|
|
shell_quote(prompt)
|
|
)
|
|
}
|
|
|
|
/// Where a turn's combined output is teed inside the guest, for the node to
|
|
/// follow. Under `/root`, never the repo: anything in `/mission/repo` is
|
|
/// collected and would arrive in the user's delivered diff.
|
|
pub const GUEST_LOG: &str = "/root/agent.log";
|
|
|
|
/// Outcome of one phase VM, as observed from outside it.
|
|
pub struct VmOutcome {
|
|
/// The agent's closing text. Diagnostic only — never evidence. Whether the
|
|
/// phase succeeded is decided downstream against the repository.
|
|
pub summary: String,
|
|
pub rc: i64,
|
|
/// Whether the work came back. A turn that ran and could not be collected is
|
|
/// a failure even if the agent was happy.
|
|
pub collected: bool,
|
|
/// How many subagents the lead actually spawned, counted from Claude Code's
|
|
/// own per-subagent transcripts in the guest.
|
|
///
|
|
/// Observed, not claimed. A lead that says it "had the verifier check this"
|
|
/// while never spawning one reads identically in its summary; this does not.
|
|
/// `None` means the probe could not run, which is distinct from zero.
|
|
pub subagents: Option<u32>,
|
|
/// Teammates the lead spawned, for a mission that asked for a team. `None`
|
|
/// when no team was requested, or when the probe could not run.
|
|
pub teammates: Option<u32>,
|
|
/// How many times the stop gate refused to let the agent finish. `None` when
|
|
/// no gate was installed. Zero means the agent got it right first time,
|
|
/// which is a different fact from "there was no gate".
|
|
pub stop_blocks: Option<u32>,
|
|
/// Whether the gate ran out of blocks and let the agent stop anyway, with
|
|
/// the phase's `done_when_check` still exiting non-zero (or the tree still
|
|
/// unchanged). `None` when no gate was installed, or the probe could not run
|
|
/// — neither of which is "it did not cap".
|
|
///
|
|
/// `Some(true)` is a FAILED turn. The gate is the only thing that ever runs
|
|
/// a `done_when_check`, so if it gave up, nothing downstream will notice.
|
|
pub released_at_cap: Option<bool>,
|
|
/// What the agent's tools touched, drained from the guest tap.
|
|
///
|
|
/// Empty when no tap was installed, when the hook never fired, or when the
|
|
/// agent genuinely called nothing. Those are three different facts and this
|
|
/// field cannot tell them apart — the unmatched-frame log and the install
|
|
/// error are what separate them.
|
|
pub tools: Vec<crate::vm_tool_tap::Observed>,
|
|
}
|
|
|
|
/// Boot a VM, run the phase in it, collect the result, and destroy it.
|
|
///
|
|
/// `destroy` runs on every exit path. A leaked VM holds an 8 GB sparse rootfs and
|
|
/// a firecracker process, and the node's orphan sweep is a backstop, not a plan.
|
|
pub async fn run_phase_in_vm(hub: &NodeHub, p: VmPhase<'_>) -> Result<VmOutcome, String> {
|
|
// Resolved BEFORE the VM boots: a missing subscription token must fail the
|
|
// phase, not boot a VM whose agent will sit there unauthenticated.
|
|
let env = crate::mission_runtime::microvm_provider_env(p.backend)?;
|
|
|
|
let vm = MicroVm::new(hub, p.node_id, vm_id_for(p.phase_id, p.iteration, p.step));
|
|
let created = vm.create(VCPUS, MEM_MIB, p.backend).await?;
|
|
|
|
// From here on every early return must still destroy the VM, so the work is
|
|
// one call whose result is held while teardown runs unconditionally.
|
|
let outcome = run_inside(
|
|
&vm,
|
|
&created,
|
|
p.task,
|
|
p.repo,
|
|
p.has_repo,
|
|
&env,
|
|
p.team_engine,
|
|
p.gate,
|
|
p.run_id,
|
|
)
|
|
.await;
|
|
|
|
if let Err(e) = vm.destroy().await {
|
|
// Not fatal to the phase — the work may already be collected — but loud,
|
|
// because the alternative is an 8 GB rootfs nobody is looking for.
|
|
eprintln!(
|
|
"microvm_executor: mission {} phase {}: destroy {} FAILED: {e}",
|
|
p.mission_id,
|
|
p.phase_id,
|
|
vm.vm_id()
|
|
);
|
|
}
|
|
outcome
|
|
}
|
|
|
|
/// One phase to run in one VM.
|
|
pub struct VmPhase<'a> {
|
|
/// The topology run this turn belongs to. Live output is keyed by it, since
|
|
/// that is what a browser subscribes to.
|
|
pub run_id: Option<Uuid>,
|
|
pub node_id: NodeId,
|
|
pub mission_id: Uuid,
|
|
pub phase_id: Uuid,
|
|
/// Which pass. Part of the vm id, so two passes of the same phase cannot
|
|
/// collide on a node.
|
|
pub iteration: i32,
|
|
pub task: &'a str,
|
|
/// `missions.backend` — which rootfs image. `None` boots the node's default.
|
|
pub backend: Option<&'a str>,
|
|
/// The host checkout, injected as a tar and collected back over the same
|
|
/// path so `mission_delivery` needs no change.
|
|
pub repo: &'a std::path::Path,
|
|
/// Whether the mission has a repository at all.
|
|
///
|
|
/// A repo-less mission still gets a `/mission/repo` — the agents need
|
|
/// somewhere to write and the collect brings it back — but it is an empty
|
|
/// workspace rather than a checkout. Without this the inject packed a
|
|
/// directory that does not exist and the readiness probe demanded a `.git`
|
|
/// that never would, so a repo-less microVM phase failed before the agent
|
|
/// ran. It is the same `has_repo` `phase_runner` already threads through to
|
|
/// choose the prompt.
|
|
pub has_repo: bool,
|
|
/// `missions.team_engine` — `Some("claude_code")` asks the lead to form a
|
|
/// team. `None` is solo, which is the default.
|
|
pub team_engine: Option<&'a str>,
|
|
/// What must hold before the agent is allowed to stop, installed as a
|
|
/// Claude Code `Stop` hook inside the guest. `None` leaves the turn exactly
|
|
/// as it was.
|
|
pub gate: Option<&'a crate::vm_stop_gate::StopGate>,
|
|
/// Which node of a composed graph this VM is running, if any. `None` is the
|
|
/// solo path, where the phase is one VM and the id needs no further
|
|
/// qualification. Part of the vm id, so the nodes of one phase-iteration
|
|
/// cannot collide on a fleet node.
|
|
pub step: Option<u32>,
|
|
}
|
|
|
|
/// Runs one phase (or one graph node of one) in a VM.
|
|
///
|
|
/// A seam with exactly one production implementation, and it exists for one
|
|
/// reason: the composed executor's defining property is that node 2 sees node 1's
|
|
/// files, and that property is untestable against real Firecracker in a unit
|
|
/// test. A fake that models inject → run → collect faithfully can prove it in
|
|
/// milliseconds, including the negative control where the handoff is broken.
|
|
#[allow(async_fn_in_trait)]
|
|
pub trait PhaseVm {
|
|
/// Boot a VM, run this phase in it, collect the result, destroy it.
|
|
async fn run(&self, p: VmPhase<'_>) -> Result<VmOutcome, String>;
|
|
}
|
|
|
|
/// The real thing: VMs on the fleet, over the node hub.
|
|
pub struct HubVms {
|
|
hub: std::sync::Arc<NodeHub>,
|
|
}
|
|
|
|
impl HubVms {
|
|
pub fn new(hub: std::sync::Arc<NodeHub>) -> Self {
|
|
Self { hub }
|
|
}
|
|
}
|
|
|
|
impl PhaseVm for HubVms {
|
|
async fn run(&self, p: VmPhase<'_>) -> Result<VmOutcome, String> {
|
|
run_phase_in_vm(&self.hub, p).await
|
|
}
|
|
}
|
|
|
|
async fn run_inside(
|
|
vm: &MicroVm<'_>,
|
|
created: &serde_json::Value,
|
|
task: &str,
|
|
repo: &std::path::Path,
|
|
// See `VmPhase::has_repo`: a repo-less mission gets an EMPTY workspace at
|
|
// the same guest path, and is proven present differently.
|
|
has_repo: bool,
|
|
env: &[(String, String)],
|
|
engine: Option<&str>,
|
|
gate: Option<&crate::vm_stop_gate::StopGate>,
|
|
// The run this turn's live output belongs to. `None` on paths with no
|
|
// subscriber, which simply means the node does not follow the log.
|
|
run_id: Option<Uuid>,
|
|
) -> Result<VmOutcome, String> {
|
|
// An agent CLI cannot reach its API without the tunnel, and a turn without
|
|
// egress does not fail — it hangs, or reports a network error the operator
|
|
// then has to trace back through three layers. `create` measures both ends;
|
|
// this is the reader that makes those fields matter.
|
|
if created.get("egress").and_then(serde_json::Value::as_bool) != Some(true) {
|
|
return Err(format!(
|
|
"vm {} has no egress (host={}, guest={}), so the agent could not reach \
|
|
its API — refusing to run the phase in it",
|
|
vm.vm_id(),
|
|
created.get("egress_host").unwrap_or(&serde_json::Value::Null),
|
|
created.get("egress_guest").unwrap_or(&serde_json::Value::Null),
|
|
));
|
|
}
|
|
|
|
// The checkout, as a tar. `pack_dir` names the entry `repo`, and the guest
|
|
// unpacks it under /mission, so it lands at /mission/repo.
|
|
//
|
|
// A repo-less mission has no directory to pack. Create it — empty — rather
|
|
// than skipping the inject: the guest needs the workspace to exist before
|
|
// the agent writes into it, and creating it host-side means the collect
|
|
// unpacks back over the same path with no special case.
|
|
if !has_repo && !repo.exists() {
|
|
std::fs::create_dir_all(repo)
|
|
.map_err(|e| format!("create empty workspace {}: {e}", repo.display()))?;
|
|
}
|
|
let archive = crate::mission_fs::pack_dir(repo, "repo")?;
|
|
let injected = archive.len();
|
|
vm.inject("/mission", &archive).await?;
|
|
|
|
// Prove the guest actually has the checkout before spending an agent turn on
|
|
// it. An inject that reports success while landing nothing would otherwise
|
|
// become an agent reporting that the repository is empty.
|
|
// What "landed" means depends on what was sent. A checkout is proven by its
|
|
// `.git`; an empty workspace can only be proven by the directory itself,
|
|
// and demanding `.git` of it failed every repo-less microVM phase before
|
|
// the agent got a turn.
|
|
let want = if has_repo {
|
|
format!("{GUEST_REPO}/.git")
|
|
} else {
|
|
GUEST_REPO.to_string()
|
|
};
|
|
let probe = vm
|
|
.exec(&format!("test -d {want} && echo REPO-PRESENT"), None, 60, &[])
|
|
.await?;
|
|
if !probe.stdout.contains("REPO-PRESENT") {
|
|
return Err(format!(
|
|
"the workspace did not land in the guest ({injected} bytes injected, \
|
|
{want} is absent) — rc={} {}",
|
|
probe.rc, probe.stderr
|
|
));
|
|
}
|
|
|
|
// The team addendum is appended only when the mission asked, so a solo
|
|
// mission's prompt is byte-identical to what it was before teams existed.
|
|
let prompt = match wants_claude_code_team(engine) {
|
|
true => format!("{}{}", vm_prompt(task), team_prompt_addendum()),
|
|
false => vm_prompt(task),
|
|
};
|
|
let mut turn_env = env.to_vec();
|
|
turn_env.extend(team_env(engine));
|
|
|
|
// The stop gate, installed before the turn. Every failure here is degraded
|
|
// to "no gate" and logged: the gate makes a phase converge in ONE VM instead
|
|
// of a second one, so losing it costs money and time — while failing the
|
|
// turn over it would cost the work.
|
|
// Asked ONCE, of the binary rather than of a version number, and shared by
|
|
// both hooks: `--settings` is how either of them gets installed at all.
|
|
let settings_supported = vm
|
|
.exec(SETTINGS_PROBE, None, 60, &[])
|
|
.await
|
|
.map(|p| p.stdout.contains("SETTINGS-OK"))
|
|
.unwrap_or(false);
|
|
let gate_dir = match gate {
|
|
None => None,
|
|
Some(g) => {
|
|
if !settings_supported {
|
|
eprintln!(
|
|
"microvm_executor: {} has no `claude --settings`, so the stop gate \
|
|
cannot be installed — this phase runs ungated and its completion \
|
|
is judged only after the fact",
|
|
vm.vm_id()
|
|
);
|
|
None
|
|
} else {
|
|
let install = g.install_command(GUEST_REPO, crate::vm_stop_gate::GATE_DIR);
|
|
match vm.exec(&install, None, 60, &[]).await {
|
|
Ok(o) if o.rc == 0 => Some(crate::vm_stop_gate::GATE_DIR),
|
|
Ok(o) => {
|
|
eprintln!(
|
|
"microvm_executor: could not install the stop gate on {} \
|
|
(rc={}): {} — running ungated",
|
|
vm.vm_id(),
|
|
o.rc,
|
|
o.stderr
|
|
);
|
|
None
|
|
}
|
|
Err(e) => {
|
|
eprintln!(
|
|
"microvm_executor: could not install the stop gate on {}: {e} \
|
|
— running ungated",
|
|
vm.vm_id()
|
|
);
|
|
None
|
|
}
|
|
}
|
|
}
|
|
}
|
|
};
|
|
|
|
// The tool tap, on the same `--settings` seam as the gate. Best-effort in
|
|
// exactly the same way: a phase that runs without telemetry is a phase that
|
|
// still delivers, and failing the turn to protect a picture would be the
|
|
// wrong trade.
|
|
let tap_dir = match settings_supported {
|
|
false => None,
|
|
true => match vm
|
|
.exec(&crate::vm_tool_tap::install_command(crate::vm_tool_tap::TAP_DIR), None, 60, &[])
|
|
.await
|
|
{
|
|
Ok(o) if o.rc == 0 => Some(crate::vm_tool_tap::TAP_DIR),
|
|
Ok(o) => {
|
|
eprintln!(
|
|
"microvm_executor: could not install the tool tap on {} (rc={}): {} \
|
|
— this phase's actions will not appear in the World",
|
|
vm.vm_id(),
|
|
o.rc,
|
|
o.stderr
|
|
);
|
|
None
|
|
}
|
|
Err(e) => {
|
|
eprintln!(
|
|
"microvm_executor: could not install the tool tap on {}: {e} \
|
|
— this phase's actions will not appear in the World",
|
|
vm.vm_id()
|
|
);
|
|
None
|
|
}
|
|
},
|
|
};
|
|
|
|
// ONE settings document, written once, carrying whichever hooks installed.
|
|
// Two writers here is the silent clobber `guest_settings` exists to stop:
|
|
// whichever ran second would erase the other's hook with no error at all.
|
|
let settings = match (gate_dir, tap_dir) {
|
|
(None, None) => None,
|
|
(g, t) => {
|
|
let doc = crate::vm_tool_tap::guest_settings(g, t);
|
|
let cmd = crate::vm_tool_tap::settings_command(
|
|
crate::vm_tool_tap::SETTINGS_PATH,
|
|
&doc,
|
|
);
|
|
match vm.exec(&cmd, None, 60, &[]).await {
|
|
Ok(o) if o.rc == 0 => Some(crate::vm_tool_tap::SETTINGS_PATH.to_string()),
|
|
other => {
|
|
eprintln!(
|
|
"microvm_executor: could not write the guest settings on {} ({other:?}) \
|
|
— running with NO hooks: neither the stop gate nor the tool tap",
|
|
vm.vm_id()
|
|
);
|
|
None
|
|
}
|
|
}
|
|
}
|
|
};
|
|
|
|
let out = vm
|
|
.exec_attributed(
|
|
&agent_command(&prompt, settings.as_deref()),
|
|
None,
|
|
TURN_SECS,
|
|
&turn_env,
|
|
run_id,
|
|
Some(GUEST_LOG),
|
|
)
|
|
.await?;
|
|
|
|
// The tool tap, drained BEFORE collect: it lives in /root, outside the
|
|
// collected tree, and the VM is destroyed moments later. This is the only
|
|
// chance to read it.
|
|
let tools = match tap_dir {
|
|
None => Vec::new(),
|
|
Some(_) => match vm.exec(crate::vm_tool_tap::DRAIN_PROBE, None, 60, &[]).await {
|
|
Ok(p) => crate::vm_tool_tap::parse(&p.stdout),
|
|
Err(e) => {
|
|
eprintln!("microvm_executor: tap drain failed on {}: {e}", vm.vm_id());
|
|
Vec::new()
|
|
}
|
|
},
|
|
};
|
|
if tap_dir.is_some() && tools.is_empty() {
|
|
// A tap that installed and drained nothing is the silent case: the
|
|
// phase looks the same as it did before the tap existed. Say so, or the
|
|
// next person debugging an empty World has no thread to pull.
|
|
eprintln!(
|
|
"microvm_executor: {} installed the tool tap and drained ZERO tool calls — \
|
|
either the agent called no tools or `PostToolUse` did not fire in this image",
|
|
vm.vm_id()
|
|
);
|
|
}
|
|
|
|
// Ask the guest how many subagents ran, before collecting: the transcripts
|
|
// live in /root, outside the collected tree, so this is the only chance.
|
|
// Failure to probe is `None`, never 0 — "we could not look" and "it delegated
|
|
// to nobody" are different facts and only one of them is about the agent.
|
|
let subagents = match vm.exec(SUBAGENT_PROBE, None, 60, &[]).await {
|
|
Ok(p) => p.stdout.trim().parse::<u32>().ok(),
|
|
Err(e) => {
|
|
eprintln!("microvm_executor: subagent probe failed on {}: {e}", vm.vm_id());
|
|
None
|
|
}
|
|
};
|
|
// Teammates, when a team was asked for. A team mission that formed no team is
|
|
// silently solo otherwise — it would still deliver, still look fine, and the
|
|
// only difference from a solo run would be the tokens it did not spend.
|
|
let teammates = match wants_claude_code_team(engine) {
|
|
false => None,
|
|
true => match vm.exec(TEAMMATE_PROBE, None, 60, &[]).await {
|
|
// `members` includes the lead, so teammates = members - 1.
|
|
Ok(p) => p.stdout.trim().parse::<u32>().ok().map(|n| n.saturating_sub(1)),
|
|
Err(e) => {
|
|
eprintln!("microvm_executor: teammate probe failed on {}: {e}", vm.vm_id());
|
|
None
|
|
}
|
|
},
|
|
};
|
|
// A team mission that fanned out to NOBODY is just a solo run that spent the
|
|
// same tokens, and nothing else in the output would say so. Judged on the
|
|
// subagent count, because that is the mechanism print mode actually uses —
|
|
// teammates never form here (see `team_env`), so a zero teammate count is
|
|
// expected and not itself a problem.
|
|
if wants_claude_code_team(engine) && subagents.unwrap_or(0) == 0 {
|
|
eprintln!(
|
|
"microvm_executor: {} asked for a team and delegated to nobody — it ran \
|
|
SOLO. Print mode satisfies a team request with subagents rather than \
|
|
teammates, so check that the Agent tool is in the allowlist and that \
|
|
the task is actually divisible.",
|
|
vm.vm_id()
|
|
);
|
|
}
|
|
|
|
// Collect regardless of the agent's exit code. A turn that failed partway
|
|
// still wrote files, and throwing them away because the CLI exited non-zero
|
|
// would discard exactly the work a retry needs to see.
|
|
let tar = vm.collect(GUEST_REPO, crate::mission_fs::transport_excludes()).await;
|
|
let collected = match tar {
|
|
Ok(bytes) => {
|
|
let parent = repo
|
|
.parent()
|
|
.ok_or_else(|| format!("{} has no parent", repo.display()))?;
|
|
// Unpacked over the SAME host path the checkout came from, so
|
|
// `capture_phase_diff_at` finds a normal checkout exactly where it
|
|
// always has and needs no change at all.
|
|
crate::mission_fs::unpack_into(&bytes, parent)?;
|
|
true
|
|
}
|
|
Err(e) => {
|
|
// Reported, not swallowed: an uncollected turn is a failed phase even
|
|
// when the agent said it finished.
|
|
eprintln!("microvm_executor: collect from {} failed: {e}", vm.vm_id());
|
|
false
|
|
}
|
|
};
|
|
|
|
// How many times the gate sent the agent back. Read only when a gate was
|
|
// actually installed, so `None` never means "it never fired".
|
|
let stop_blocks = match settings {
|
|
None => None,
|
|
Some(_) => match vm.exec(BLOCKS_PROBE, None, 60, &[]).await {
|
|
Ok(p) => p.stdout.trim().parse::<u32>().ok(),
|
|
Err(e) => {
|
|
eprintln!("microvm_executor: stop-gate probe failed on {}: {e}", vm.vm_id());
|
|
None
|
|
}
|
|
},
|
|
};
|
|
|
|
// Probed only when a gate existed, so `None` never reads as "it did not cap".
|
|
let released_at_cap = match settings {
|
|
None => None,
|
|
Some(_) => match vm.exec(CAPPED_PROBE, None, 60, &[]).await {
|
|
Ok(p) => Some(p.stdout.trim() == "1"),
|
|
Err(e) => {
|
|
eprintln!("microvm_executor: cap probe failed on {}: {e}", vm.vm_id());
|
|
None
|
|
}
|
|
},
|
|
};
|
|
|
|
Ok(VmOutcome {
|
|
summary: format!("{}{}", out.stdout, out.stderr),
|
|
rc: out.rc,
|
|
collected,
|
|
subagents,
|
|
teammates,
|
|
stop_blocks,
|
|
released_at_cap,
|
|
tools,
|
|
})
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
/// The turn must be observable WHILE it runs and still report its output.
|
|
///
|
|
/// `tee`, not `>`: the log file is what the node follows to stream the turn
|
|
/// live, and the command's own stdout is what becomes `VmOutcome::summary`.
|
|
/// A redirect would give a live view and an empty summary — which is the
|
|
/// same "green and empty" shape this codebase keeps finding.
|
|
#[test]
|
|
fn a_turn_is_teed_so_it_streams_and_still_reports() {
|
|
let cmd = agent_command("do the thing", None);
|
|
assert!(cmd.contains(&format!("tee {GUEST_LOG}")), "{cmd}");
|
|
assert!(
|
|
!cmd.contains(&format!("> {GUEST_LOG}")),
|
|
"a redirect would empty the summary: {cmd}"
|
|
);
|
|
// stderr must be included: agent CLIs report progress there.
|
|
assert!(cmd.contains("2>&1"), "{cmd}");
|
|
// And the log must live outside the collected tree, or it arrives in the
|
|
// user's delivered diff.
|
|
assert!(GUEST_LOG.starts_with("/root/"), "{GUEST_LOG}");
|
|
assert!(!GUEST_LOG.starts_with(GUEST_REPO), "{GUEST_LOG}");
|
|
}
|
|
|
|
/// The id becomes a path component on the node, which rejects anything
|
|
/// outside `[A-Za-z0-9_-]` rather than sanitising it.
|
|
#[test]
|
|
fn a_vm_id_is_acceptable_to_the_node() {
|
|
for step in [None, Some(0), Some(11)] {
|
|
let id = vm_id_for(Uuid::now_v7(), 3, step);
|
|
assert!(
|
|
id.chars()
|
|
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'),
|
|
"{id}"
|
|
);
|
|
assert!(id.len() <= 64, "{id}");
|
|
}
|
|
}
|
|
|
|
/// Two passes of the same phase must not collide: the second would fail with
|
|
/// "vm already exists" while the first is still running.
|
|
#[test]
|
|
fn each_iteration_gets_its_own_vm() {
|
|
let p = Uuid::now_v7();
|
|
assert_ne!(vm_id_for(p, 1, None), vm_id_for(p, 2, None));
|
|
}
|
|
|
|
/// A composed run boots one VM per graph node within a single phase and
|
|
/// iteration. Without the step in the id they would all be the same vm, and
|
|
/// the second node would fail with "vm already exists" — or worse, if the
|
|
/// first were already destroyed, succeed while looking like a re-run.
|
|
#[test]
|
|
fn each_graph_node_gets_its_own_vm() {
|
|
let p = Uuid::now_v7();
|
|
assert_ne!(vm_id_for(p, 1, Some(0)), vm_id_for(p, 1, Some(1)));
|
|
// And a composed node never collides with the solo id for the same pass.
|
|
assert_ne!(vm_id_for(p, 1, Some(0)), vm_id_for(p, 1, None));
|
|
// Deterministic: resuming the same step asks for the same vm, which is
|
|
// what makes a still-live duplicate fail loudly on the node.
|
|
assert_eq!(vm_id_for(p, 1, Some(2)), vm_id_for(p, 1, Some(2)));
|
|
}
|
|
|
|
/// A task description contains quotes and newlines as a matter of course,
|
|
/// and the whole command is handed to `sh -c` in the guest.
|
|
#[test]
|
|
fn a_quoted_task_cannot_break_out_of_the_command() {
|
|
let nasty = "it's a 'test'; rm -rf /\n$(whoami) `id` \"quoted\"";
|
|
let cmd = agent_command(&vm_prompt(nasty), None);
|
|
// Everything after the opening quote of the prompt is inside it: the only
|
|
// way out of a single-quoted string is a quote, and each one is escaped.
|
|
let body = cmd.split_once('\'').expect("a quoted argument").1;
|
|
for danger in ["; rm -rf /", "$(whoami)", "`id`"] {
|
|
let at = body.find(danger).expect("the text is still present");
|
|
let before = &body[..at];
|
|
// An odd number of unescaped quotes before it would mean it had
|
|
// escaped the quoting.
|
|
let unescaped = before.matches('\'').count() - before.matches(r"'\''").count() * 3;
|
|
assert_eq!(unescaped % 2, 0, "{danger} is not quoted in: {cmd}");
|
|
}
|
|
}
|
|
|
|
/// The regression that blocked fan-out entirely: the allowlist was
|
|
/// `Read Edit Write Bash`, so Claude Code had no way to spawn a subagent in any
|
|
/// of our VMs. An unrecognised or missing tool name does not error — the agent
|
|
/// just works alone and nothing says so — which is why this is asserted rather
|
|
/// than assumed.
|
|
#[test]
|
|
fn the_lead_can_spawn_subagents_at_all() {
|
|
assert!(
|
|
LEAD_TOOLS.contains(&SUBAGENT_TOOL),
|
|
"without {SUBAGENT_TOOL} in the allowlist there is no delegation, silently"
|
|
);
|
|
let cmd = agent_command(&vm_prompt("t"), None);
|
|
assert!(cmd.contains(&format!("--allowedTools {}", LEAD_TOOLS.join(" "))), "{cmd}");
|
|
// `--forward-subagent-text` is deliberately absent: it refuses to run
|
|
// without `--output-format=stream-json`, which would change how this
|
|
// module reads output. Evidence comes from the per-subagent transcript
|
|
// instead.
|
|
assert!(!cmd.contains("--forward-subagent-text"), "{cmd}");
|
|
assert!(
|
|
SUBAGENT_PROBE.contains("subagents"),
|
|
"the probe must look at the per-subagent transcripts: {SUBAGENT_PROBE}"
|
|
);
|
|
}
|
|
|
|
/// Our definitions must exist and be well-formed, because they are the only
|
|
/// roles we control.
|
|
///
|
|
/// This test used to assert that disabling the built-in roles was "paired with"
|
|
/// defining our own. It passed while the feature was broken: setting
|
|
/// `CLAUDE_AGENT_SDK_DISABLE_BUILTIN_AGENTS=1` removes OUR definitions too, so
|
|
/// the pairing held in this file and nowhere else. The lesson is in the code
|
|
/// now (see `agent_command`), and what is left here is the part a unit test can
|
|
/// actually speak to.
|
|
#[test]
|
|
fn every_role_we_define_is_well_formed() {
|
|
let defs = agent_definitions();
|
|
let defs = defs.as_object().expect("an object of name → definition");
|
|
assert!(!defs.is_empty(), "no roles defined means nothing to delegate to");
|
|
for (name, d) in defs {
|
|
assert!(
|
|
d.get("description").and_then(|v| v.as_str()).is_some_and(|s| !s.is_empty()),
|
|
"{name} needs a description — it is what Claude matches on to delegate"
|
|
);
|
|
assert!(
|
|
d.get("prompt").and_then(|v| v.as_str()).is_some_and(|s| !s.is_empty()),
|
|
"{name} needs a prompt"
|
|
);
|
|
}
|
|
}
|
|
|
|
/// A verifier that can edit will fix what it was asked to check and then
|
|
/// report success, and the report is about a tree nobody reviewed.
|
|
#[test]
|
|
fn the_verifier_cannot_modify_what_it_checks() {
|
|
let defs = agent_definitions();
|
|
let tools = defs["verifier"]["tools"].as_str().expect("a tools allowlist");
|
|
for forbidden in ["Edit", "Write"] {
|
|
assert!(
|
|
!tools.contains(forbidden),
|
|
"the verifier must not have {forbidden}: {tools}"
|
|
);
|
|
}
|
|
assert!(tools.contains("Bash"), "it has to be able to run the suite: {tools}");
|
|
// Subagents default to background since 2.1.198. A background verifier lets
|
|
// the lead write its conclusion before the check finishes, so the finding
|
|
// would arrive after the report that was supposed to contain it.
|
|
assert_eq!(
|
|
defs["verifier"]["background"],
|
|
serde_json::json!(false),
|
|
"the lead must wait for its verifier"
|
|
);
|
|
}
|
|
|
|
/// The early-victory failure: a verifier that stops after the first passing
|
|
/// test reports success on broken work. Anthropic names it, and we have paid
|
|
/// for it once already.
|
|
#[test]
|
|
fn the_verifier_is_told_to_run_the_whole_suite() {
|
|
let defs = agent_definitions();
|
|
let p = defs["verifier"]["prompt"].as_str().unwrap().to_ascii_lowercase();
|
|
assert!(p.contains("complete suite"), "{p}");
|
|
assert!(p.contains(&NO_SHORTCUTS[..40].to_ascii_lowercase()), "anti-shortcut rule missing");
|
|
}
|
|
|
|
/// The definitions travel as ONE shell argument through `sh -c`, and they
|
|
/// contain double quotes, braces and apostrophes ("the project's OWN checks").
|
|
/// A quoting bug here would hand `claude` a fragment of JSON, which it would
|
|
/// reject — leaving the lead with no roles and no fan-out.
|
|
///
|
|
/// Tested by round-tripping rather than by banning apostrophes: the first
|
|
/// version of this test asserted the JSON contained none, which failed on
|
|
/// correctly-quoted text and said nothing about whether the quoting worked.
|
|
#[test]
|
|
fn the_agent_definitions_survive_shell_quoting() {
|
|
/// Undo `shell_quote`, the way `sh` would.
|
|
fn unquote(s: &str) -> String {
|
|
let inner = s
|
|
.strip_prefix('\'')
|
|
.and_then(|s| s.strip_suffix('\''))
|
|
.expect("a single-quoted argument");
|
|
inner.replace(r"'\''", "'")
|
|
}
|
|
|
|
let json = agent_definitions().to_string();
|
|
let quoted = shell_quote(&json);
|
|
assert_eq!(unquote("ed), json, "the JSON did not survive quoting");
|
|
// And it is still valid JSON on the other side.
|
|
let back: serde_json::Value =
|
|
serde_json::from_str(&unquote("ed)).expect("valid JSON after quoting");
|
|
assert!(back.get("verifier").is_some(), "{back}");
|
|
|
|
assert!(agent_command(&vm_prompt("task"), None).contains(&format!("--agents {quoted}")));
|
|
}
|
|
|
|
/// The tier this executor inserts must NOT be one the topology worker drives.
|
|
/// A microvm run is owned by its own spawned task for its whole life; if the
|
|
/// worker also considers it fair game, it requeues it at 180s, fails it on a
|
|
/// graph it was never meant to parse, and orphans a live VM. Mission 019fd43e
|
|
/// died at 210 seconds that way.
|
|
#[test]
|
|
fn a_microvm_run_is_not_worker_driven() {
|
|
assert!(
|
|
!cm_db::repo::topology_runs::WORKER_DRIVEN_TIERS.contains(&"microvm"),
|
|
"the worker must not claim, requeue or reap a self-driven run"
|
|
);
|
|
// Same exposure, same reason.
|
|
assert!(!cm_db::repo::topology_runs::WORKER_DRIVEN_TIERS.contains(&"session"));
|
|
// And the tiers it does drive are still there, or nothing runs at all.
|
|
for t in ["team", "swarm"] {
|
|
assert!(cm_db::repo::topology_runs::WORKER_DRIVEN_TIERS.contains(&t), "{t}");
|
|
}
|
|
// The composed tier is the opposite case and must not be confused with
|
|
// the solo one: its durability comes FROM being worker-driven.
|
|
assert!(cm_db::repo::topology_runs::WORKER_DRIVEN_TIERS.contains(&"microvm_graph"));
|
|
// But the 15-minute stuck-run reaper must not touch it: one of its nodes
|
|
// is a whole agent session, so journaling nothing for 15 minutes is what
|
|
// a healthy composed run looks like.
|
|
assert!(
|
|
!cm_db::repo::topology_runs::REAPABLE_TIERS.contains(&"microvm_graph"),
|
|
"the reaper would kill a healthy composed run and orphan its VM"
|
|
);
|
|
for t in cm_db::repo::topology_runs::REAPABLE_TIERS {
|
|
assert!(
|
|
cm_db::repo::topology_runs::WORKER_DRIVEN_TIERS.contains(t),
|
|
"{t} is reapable but not worker-driven"
|
|
);
|
|
}
|
|
}
|
|
|
|
/// Solo is the default, and asking for a team must be explicit. Multi-agent
|
|
/// costs 3-10x the tokens and is often slower, so a mission that said nothing
|
|
/// must not get one.
|
|
#[test]
|
|
fn only_an_explicit_request_forms_a_team() {
|
|
assert!(wants_claude_code_team(Some("claude_code")));
|
|
for engine in [None, Some(""), Some("zeroclaw"), Some("solo"), Some("CLAUDE_CODE")] {
|
|
assert!(!wants_claude_code_team(engine), "{engine:?} must not form a team");
|
|
}
|
|
}
|
|
|
|
/// The flag is what enables agent teams in the CLI, and it must not leak into
|
|
/// a solo mission — a solo run has to stay byte-identical to what it was
|
|
/// before teams existed, or the comparison between the two is meaningless.
|
|
#[test]
|
|
fn the_team_flag_reaches_only_a_team_mission() {
|
|
let on = team_env(Some("claude_code"));
|
|
assert_eq!(on.len(), 1);
|
|
assert_eq!(on[0].0, "CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS");
|
|
assert_eq!(on[0].1, "1");
|
|
assert!(team_env(None).is_empty(), "a solo mission gets no team flag");
|
|
assert!(team_env(Some("zeroclaw")).is_empty());
|
|
}
|
|
|
|
/// A solo prompt must not mention teammates, and a team prompt must state the
|
|
/// cap — the lead decides its own size and there is no flag that limits it, so
|
|
/// the prompt is the only place the budget can be said.
|
|
#[test]
|
|
fn the_team_addendum_states_the_cap_and_stays_out_of_solo_prompts() {
|
|
let solo = vm_prompt("t");
|
|
assert!(!solo.to_lowercase().contains("teammate"), "{solo}");
|
|
let addendum = team_prompt_addendum();
|
|
assert!(addendum.contains(&MAX_TEAMMATES.to_string()), "{addendum}");
|
|
// The anti-patterns worth naming, per Anthropic's own guidance: two
|
|
// teammates in one file overwrite each other, and splitting one change
|
|
// into stages across teammates loses context at every handoff.
|
|
assert!(addendum.contains("DIFFERENT FILES"), "{addendum}");
|
|
assert!(addendum.to_lowercase().contains("wait for your teammates"), "{addendum}");
|
|
// Print mode delivers this as subagents, so the addendum must not depend on
|
|
// teammates existing — it asks for parallel delegation, whatever the
|
|
// mechanism turns out to be.
|
|
assert!(addendum.contains("Delegate"), "{addendum}");
|
|
}
|
|
|
|
/// The teammate probe reads the team config, not the subagent transcripts —
|
|
/// teammates and subagents are different things and counting one as the other
|
|
/// would report a team that never formed.
|
|
#[test]
|
|
fn the_two_fan_out_probes_look_in_different_places() {
|
|
assert!(SUBAGENT_PROBE.contains("subagents"));
|
|
assert!(TEAMMATE_PROBE.contains("teams"));
|
|
assert!(!TEAMMATE_PROBE.contains("subagents"));
|
|
}
|
|
|
|
/// An ungated phase's command must be byte-identical to what it was before
|
|
/// the stop gate existed. Most phases are gated, so the ungated path is the
|
|
/// one nobody would notice breaking.
|
|
#[test]
|
|
fn a_phase_without_a_gate_gets_the_command_it_always_had() {
|
|
let cmd = agent_command(&vm_prompt("t"), None);
|
|
assert!(!cmd.contains("--settings"), "{cmd}");
|
|
|
|
let gated = agent_command(&vm_prompt("t"), Some("/root/gate/settings.json"));
|
|
assert!(gated.contains("--settings '/root/gate/settings.json'"), "{gated}");
|
|
// The flag is the ONLY difference — a gate must not quietly change the
|
|
// tools, the permission mode or the roles.
|
|
assert_eq!(
|
|
gated.replace(" --settings '/root/gate/settings.json'", ""),
|
|
cmd
|
|
);
|
|
}
|
|
|
|
/// The gate is installed through `--settings`, so whether the CLI in the
|
|
/// image HAS that flag is a fact about the image, not about our plan. An
|
|
/// unknown option is a hard error, which would turn every gated phase into a
|
|
/// failed one — hence a probe rather than a version check.
|
|
#[test]
|
|
fn the_settings_flag_is_probed_before_it_is_used() {
|
|
assert!(SETTINGS_PROBE.contains("--help"), "{SETTINGS_PROBE}");
|
|
assert!(SETTINGS_PROBE.contains("--settings"), "{SETTINGS_PROBE}");
|
|
assert!(SETTINGS_PROBE.contains("SETTINGS-OK"), "{SETTINGS_PROBE}");
|
|
// And the evidence probe reads the gate's own counter, outside the tree
|
|
// that gets collected and diffed.
|
|
assert!(BLOCKS_PROBE.contains(crate::vm_stop_gate::GATE_DIR), "{BLOCKS_PROBE}");
|
|
assert!(!BLOCKS_PROBE.contains(GUEST_REPO), "{BLOCKS_PROBE}");
|
|
}
|
|
|
|
/// EVERY VM turn is offered delegation, composed nodes included.
|
|
///
|
|
/// A composed node's task text is built by `microvm_turn_executor` and then
|
|
/// wrapped by this same `vm_prompt` inside `run_inside`, so one prompt
|
|
/// builder serves both paths. Composed runs report `subagents: 0` so far,
|
|
/// and this is what says that is the TASKS being small rather than the
|
|
/// capability being absent — if someone gave composed nodes their own
|
|
/// prompt without the offer, the difference would otherwise show up only as
|
|
/// a count nobody was watching.
|
|
#[test]
|
|
fn every_vm_turn_is_offered_the_same_help() {
|
|
let p = vm_prompt("do the thing");
|
|
assert!(p.contains("Agent tool"), "{p}");
|
|
for role in agent_definitions().as_object().unwrap().keys() {
|
|
assert!(p.contains(role.as_str()), "the prompt must name the `{role}` role: {p}");
|
|
}
|
|
}
|
|
|
|
/// The agent must not be told to push: delivery is host-side, and the guest
|
|
/// deliberately holds no forge credentials.
|
|
#[test]
|
|
fn the_prompt_does_not_ask_the_agent_to_push() {
|
|
let p = vm_prompt("do the thing");
|
|
assert!(p.contains("Do NOT push"), "{p}");
|
|
assert!(
|
|
!p.contains("push to a new branch"),
|
|
"the container path's push instruction must not leak in: {p}"
|
|
);
|
|
}
|
|
}
|