feat(missions): Slice 1 — a microVM agent can delegate, and we can see that it did
`microvm_executor` passed `--allowedTools Read Edit Write Bash`, which omits the
`Agent` tool, so Claude Code could not spawn a single subagent in any of our VMs.
The tool existed, the model knew how to use it, and the allowlist quietly removed
the ability. Nothing in any output said so.
Now: `Agent` in the allowlist, two roles supplied as `--agents` JSON, and a probe
that counts what actually ran.
Roles are JSON on the command line, not files, because `/mission/repo` is
collected and diffed — a role definition written into the checkout would arrive in
the delivered patch as if the agent had authored it.
Two roles only, and the choice is the research talking:
- `verifier` — the one multi-agent pattern Anthropic endorses for coding work.
It gets Read/Grep/Glob/Bash and deliberately NOT Edit or Write: an agent that
can fix what it is checking will fix it and report success, and the report is
then about a tree nobody reviewed. Its prompt demands the COMPLETE suite,
which is the counter to the "early victory problem" — the same failure as our
own Goodhart incident.
- `explorer` — context protection, read-only.
Roles like "tester" or "committer" are absent on purpose: splitting sequential
phases of the same work is a named anti-pattern, and it is the shape our pipeline
templates already have.
THREE THINGS THE IMAGE CORRECTED, none of which review would have caught:
1. `CLAUDE_AGENT_SDK_DISABLE_BUILTIN_AGENTS=1` (in the plan) removes EVERY agent
type, including the ones `--agents` defines. Measured: the lead reported "an
empty available-agents list" after trying four role names and — to its credit —
refused to fabricate a subagent result. Worse, the unit test asserting
"builtins off is paired with our own roles" PASSED throughout, because the
pairing holds in our code and not in the CLI. Dropped, and the test rewritten
to assert only what a unit test can speak to.
2. `--forward-subagent-text` refuses to run without `--output-format=stream-json`,
which would change how this module reads output. Dropped.
3. `--append-subagent-system-prompt` does not exist in 2.1.220 despite being
documented. The anti-shortcut rule is inlined per role instead — better anyway,
since a verifier and an explorer need different wording.
Evidence instead of assumption: Claude Code writes a per-subagent transcript at
`<session>/subagents/agent-*.jsonl`, so the guest is asked to count them before
collection (they live in /root, outside the collected tree). `VmOutcome.subagents`
is `Option<u32>` and the phase log prints it: `None`/"?" means the probe could not
run, which is a different fact from "delegated to nobody" and only one of those is
about the agent.
Verified in a container against the real CLI on tank before any of this shipped:
`FANOUT-OK`, a subagent transcript on disk, and zero errored Agent calls.
469 tests pass, clippy clean.
This commit is contained in:
@@ -70,6 +70,17 @@ fn vm_prompt(task: &str) -> String {
|
|||||||
TASK\n\
|
TASK\n\
|
||||||
{task}\n\
|
{task}\n\
|
||||||
\n\
|
\n\
|
||||||
|
HELP AVAILABLE TO YOU\n\
|
||||||
|
You can delegate with the Agent tool. Two roles exist:\n\
|
||||||
|
- `verifier` — checks work against the project's own tests. It cannot \
|
||||||
|
edit files. Use it when you believe you are finished, and treat what it \
|
||||||
|
reports as the outcome rather than your own impression.\n\
|
||||||
|
- `explorer` — answers a question about the codebase by reading it, \
|
||||||
|
without filling your context with search output.\n\
|
||||||
|
Delegating is optional. For a small, self-contained change, doing it \
|
||||||
|
yourself is usually better: a handoff costs context and coordination, and \
|
||||||
|
one careful pass beats an assembly line.\n\
|
||||||
|
\n\
|
||||||
WHEN THE WORK IS DONE\n\
|
WHEN THE WORK IS DONE\n\
|
||||||
Leave it in the working tree. Do NOT push, and do not add a remote — \
|
Leave it in the working tree. Do NOT push, and do not add a remote — \
|
||||||
this machine has no access to the forge. Committing locally is fine but \
|
this machine has no access to the forge. Committing locally is fine but \
|
||||||
@@ -89,14 +100,118 @@ fn shell_quote(s: &str) -> String {
|
|||||||
format!("'{}'", s.replace('\'', r"'\''"))
|
format!("'{}'", s.replace('\'', r"'\''"))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The tool that spawns a subagent.
|
||||||
|
///
|
||||||
|
/// **Verified against `claude --help` and the tool docs in the image we ship
|
||||||
|
/// (2.1.220), not remembered.** It was named `Task` in older versions, and the
|
||||||
|
/// cost of guessing is invisible: an unrecognised entry in `--allowedTools` does
|
||||||
|
/// not error, it simply means the agent has no way to delegate and works alone.
|
||||||
|
/// Nothing in the output says so.
|
||||||
|
const SUBAGENT_TOOL: &str = "Agent";
|
||||||
|
|
||||||
|
/// Tools the lead may use. `Agent` is what makes fan-out possible at all — before
|
||||||
|
/// it was added the allowlist was `Read Edit Write Bash`, so Claude Code could not
|
||||||
|
/// spawn a subagent in any of our VMs.
|
||||||
|
const LEAD_TOOLS: &[&str] = &["Read", "Edit", "Write", "Bash", SUBAGENT_TOOL];
|
||||||
|
|
||||||
|
/// The rule every subagent carries, in its own prompt.
|
||||||
|
///
|
||||||
|
/// `--append-subagent-system-prompt` does **not** exist in 2.1.220 despite being
|
||||||
|
/// documented — checked against `--help` in the image — so this is inlined per
|
||||||
|
/// definition instead. That is the better shape anyway: a verifier and an explorer
|
||||||
|
/// need different wording, and a single appended blob would say the same thing to
|
||||||
|
/// both.
|
||||||
|
const NO_SHORTCUTS: &str = "Report what you actually observed. Never present an \
|
||||||
|
expected result as an observed one, and never weaken, skip or narrow a check \
|
||||||
|
to make it pass — the work is judged against the repository itself, so a \
|
||||||
|
shortcut is found later and costs more than an honest failure.";
|
||||||
|
|
||||||
|
/// Subagent definitions, handed over as `--agents` JSON.
|
||||||
|
///
|
||||||
|
/// JSON on the command line rather than files, for a specific reason: the guest's
|
||||||
|
/// `/mission/repo` is collected and diffed, so a role definition written into the
|
||||||
|
/// checkout would arrive in the delivered patch as if the agent had authored it.
|
||||||
|
/// `--agents` is session-scoped and touches no disk.
|
||||||
|
///
|
||||||
|
/// These two are deliberately the only ones. `verifier` is the single multi-agent
|
||||||
|
/// pattern Anthropic endorses for coding work — implement, then check with a
|
||||||
|
/// separate agent that sees only the artifact and the criteria. `explorer` exists
|
||||||
|
/// for context protection, the other justification that survives scrutiny. Roles
|
||||||
|
/// like "tester" or "committer" are absent on purpose: splitting sequential phases
|
||||||
|
/// of the same work is a documented anti-pattern, and it is the shape our pipeline
|
||||||
|
/// templates already have.
|
||||||
|
fn agent_definitions() -> serde_json::Value {
|
||||||
|
serde_json::json!({
|
||||||
|
"verifier": {
|
||||||
|
"description": "Independently verifies that work is complete and correct. \
|
||||||
|
Use after implementing something, to check it rather than \
|
||||||
|
to trust it.",
|
||||||
|
// No Edit and no Write, by construction: an agent that can fix what it
|
||||||
|
// is checking will fix it and report success, and the report is then
|
||||||
|
// about a tree nobody reviewed.
|
||||||
|
"tools": "Read, Grep, Glob, Bash",
|
||||||
|
"prompt": format!(
|
||||||
|
"You verify work you did not do. You are given an artifact and the \
|
||||||
|
criteria it must meet, and nothing else — you do not know how it was \
|
||||||
|
built and you do not need to.\n\
|
||||||
|
\n\
|
||||||
|
Run the project's OWN checks, in full. If it has a test suite, run \
|
||||||
|
the COMPLETE suite, not the tests that look relevant: stopping after \
|
||||||
|
the first one or two that pass is the most common way a verifier \
|
||||||
|
reports success on broken work.\n\
|
||||||
|
\n\
|
||||||
|
You cannot edit or write files. If something is wrong, say exactly \
|
||||||
|
what you ran, what you expected and what you saw. {NO_SHORTCUTS}"
|
||||||
|
),
|
||||||
|
},
|
||||||
|
"explorer": {
|
||||||
|
"description": "Reads and searches the codebase to answer a specific \
|
||||||
|
question. Use when finding something out would otherwise \
|
||||||
|
fill the main context with files and search output.",
|
||||||
|
"tools": "Read, Grep, Glob",
|
||||||
|
"prompt": format!(
|
||||||
|
"You answer one question about this codebase by reading it. Return the \
|
||||||
|
answer and the paths that support it — not a transcript of your \
|
||||||
|
search. You cannot modify anything. {NO_SHORTCUTS}"
|
||||||
|
),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Claude Code writes a separate transcript per subagent, under the session's own
|
||||||
|
/// directory. Its existence is the evidence that delegation happened — measured in
|
||||||
|
/// the shipped image, not inferred:
|
||||||
|
///
|
||||||
|
/// ```text
|
||||||
|
/// /root/.claude/projects/-w/<session>.jsonl
|
||||||
|
/// /root/.claude/projects/-w/<session>/subagents/agent-<id>.jsonl
|
||||||
|
/// ```
|
||||||
|
///
|
||||||
|
/// The alternative, `--forward-subagent-text`, is unusable here: it refuses to run
|
||||||
|
/// without `--output-format=stream-json`, which would change how this module reads
|
||||||
|
/// the agent's output entirely. Counting transcripts costs nothing and cannot be
|
||||||
|
/// confused with the lead merely *claiming* it delegated.
|
||||||
|
const SUBAGENT_PROBE: &str =
|
||||||
|
"ls -1 /root/.claude/projects/*/*/subagents/*.jsonl 2>/dev/null | wc -l";
|
||||||
|
|
||||||
/// The command that runs the agent in the guest.
|
/// The command that runs the agent in the guest.
|
||||||
fn agent_command(prompt: &str) -> String {
|
fn agent_command(prompt: &str) -> String {
|
||||||
// --permission-mode acceptEdits, matching the container path: the VM IS the
|
// --permission-mode acceptEdits, matching the container path: the VM IS the
|
||||||
// boundary, so prompting for permission inside it would only mean a turn that
|
// boundary, so prompting for permission inside it would only mean a turn that
|
||||||
// waits for an answer nobody can give.
|
// waits for an answer nobody can give.
|
||||||
|
//
|
||||||
|
// NOT `CLAUDE_AGENT_SDK_DISABLE_BUILTIN_AGENTS=1`. It was in the plan, and it
|
||||||
|
// is wrong: measured in the image, it removes EVERY agent type including the
|
||||||
|
// ones `--agents` defines. The lead reported "an empty available-agents list"
|
||||||
|
// after trying four role names and — to its credit — refused to fabricate a
|
||||||
|
// result. A unit test asserting "builtins off is paired with our own roles"
|
||||||
|
// passed the whole time, because the pairing holds in our code and not in the
|
||||||
|
// CLI. So the built-in roles stay available alongside ours.
|
||||||
format!(
|
format!(
|
||||||
"cd {GUEST_REPO} && claude -p --allowedTools Read Edit Write Bash \
|
"cd {GUEST_REPO} && claude -p --allowedTools {} \
|
||||||
--permission-mode acceptEdits {}",
|
--permission-mode acceptEdits --agents {} {}",
|
||||||
|
LEAD_TOOLS.join(" "),
|
||||||
|
shell_quote(&agent_definitions().to_string()),
|
||||||
shell_quote(prompt)
|
shell_quote(prompt)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
@@ -110,6 +225,13 @@ pub struct VmOutcome {
|
|||||||
/// Whether the work came back. A turn that ran and could not be collected is
|
/// Whether the work came back. A turn that ran and could not be collected is
|
||||||
/// a failure even if the agent was happy.
|
/// a failure even if the agent was happy.
|
||||||
pub collected: bool,
|
pub collected: bool,
|
||||||
|
/// How many subagents the lead actually spawned, counted from Claude Code's
|
||||||
|
/// own per-subagent transcripts in the guest.
|
||||||
|
///
|
||||||
|
/// Observed, not claimed. A lead that says it "had the verifier check this"
|
||||||
|
/// while never spawning one reads identically in its summary; this does not.
|
||||||
|
/// `None` means the probe could not run, which is distinct from zero.
|
||||||
|
pub subagents: Option<u32>,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Boot a VM, run the phase in it, collect the result, and destroy it.
|
/// Boot a VM, run the phase in it, collect the result, and destroy it.
|
||||||
@@ -207,6 +329,18 @@ async fn run_inside(
|
|||||||
.exec(&agent_command(&vm_prompt(task)), None, TURN_SECS, env)
|
.exec(&agent_command(&vm_prompt(task)), None, TURN_SECS, env)
|
||||||
.await?;
|
.await?;
|
||||||
|
|
||||||
|
// Ask the guest how many subagents ran, before collecting: the transcripts
|
||||||
|
// live in /root, outside the collected tree, so this is the only chance.
|
||||||
|
// Failure to probe is `None`, never 0 — "we could not look" and "it delegated
|
||||||
|
// to nobody" are different facts and only one of them is about the agent.
|
||||||
|
let subagents = match vm.exec(SUBAGENT_PROBE, None, 60, &[]).await {
|
||||||
|
Ok(p) => p.stdout.trim().parse::<u32>().ok(),
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!("microvm_executor: subagent probe failed on {}: {e}", vm.vm_id());
|
||||||
|
None
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
// Collect regardless of the agent's exit code. A turn that failed partway
|
// Collect regardless of the agent's exit code. A turn that failed partway
|
||||||
// still wrote files, and throwing them away because the CLI exited non-zero
|
// still wrote files, and throwing them away because the CLI exited non-zero
|
||||||
// would discard exactly the work a retry needs to see.
|
// would discard exactly the work a retry needs to see.
|
||||||
@@ -234,6 +368,7 @@ async fn run_inside(
|
|||||||
summary: format!("{}{}", out.stdout, out.stderr),
|
summary: format!("{}{}", out.stdout, out.stderr),
|
||||||
rc: out.rc,
|
rc: out.rc,
|
||||||
collected,
|
collected,
|
||||||
|
subagents,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -281,6 +416,112 @@ mod tests {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The regression that blocked fan-out entirely: the allowlist was
|
||||||
|
/// `Read Edit Write Bash`, so Claude Code had no way to spawn a subagent in any
|
||||||
|
/// of our VMs. An unrecognised or missing tool name does not error — the agent
|
||||||
|
/// just works alone and nothing says so — which is why this is asserted rather
|
||||||
|
/// than assumed.
|
||||||
|
#[test]
|
||||||
|
fn the_lead_can_spawn_subagents_at_all() {
|
||||||
|
assert!(
|
||||||
|
LEAD_TOOLS.contains(&SUBAGENT_TOOL),
|
||||||
|
"without {SUBAGENT_TOOL} in the allowlist there is no delegation, silently"
|
||||||
|
);
|
||||||
|
let cmd = agent_command(&vm_prompt("t"));
|
||||||
|
assert!(cmd.contains(&format!("--allowedTools {}", LEAD_TOOLS.join(" "))), "{cmd}");
|
||||||
|
// `--forward-subagent-text` is deliberately absent: it refuses to run
|
||||||
|
// without `--output-format=stream-json`, which would change how this
|
||||||
|
// module reads output. Evidence comes from the per-subagent transcript
|
||||||
|
// instead.
|
||||||
|
assert!(!cmd.contains("--forward-subagent-text"), "{cmd}");
|
||||||
|
assert!(
|
||||||
|
SUBAGENT_PROBE.contains("subagents"),
|
||||||
|
"the probe must look at the per-subagent transcripts: {SUBAGENT_PROBE}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Our definitions must exist and be well-formed, because they are the only
|
||||||
|
/// roles we control.
|
||||||
|
///
|
||||||
|
/// This test used to assert that disabling the built-in roles was "paired with"
|
||||||
|
/// defining our own. It passed while the feature was broken: setting
|
||||||
|
/// `CLAUDE_AGENT_SDK_DISABLE_BUILTIN_AGENTS=1` removes OUR definitions too, so
|
||||||
|
/// the pairing held in this file and nowhere else. The lesson is in the code
|
||||||
|
/// now (see `agent_command`), and what is left here is the part a unit test can
|
||||||
|
/// actually speak to.
|
||||||
|
#[test]
|
||||||
|
fn every_role_we_define_is_well_formed() {
|
||||||
|
let defs = agent_definitions();
|
||||||
|
let defs = defs.as_object().expect("an object of name → definition");
|
||||||
|
assert!(!defs.is_empty(), "no roles defined means nothing to delegate to");
|
||||||
|
for (name, d) in defs {
|
||||||
|
assert!(
|
||||||
|
d.get("description").and_then(|v| v.as_str()).is_some_and(|s| !s.is_empty()),
|
||||||
|
"{name} needs a description — it is what Claude matches on to delegate"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
d.get("prompt").and_then(|v| v.as_str()).is_some_and(|s| !s.is_empty()),
|
||||||
|
"{name} needs a prompt"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A verifier that can edit will fix what it was asked to check and then
|
||||||
|
/// report success, and the report is about a tree nobody reviewed.
|
||||||
|
#[test]
|
||||||
|
fn the_verifier_cannot_modify_what_it_checks() {
|
||||||
|
let defs = agent_definitions();
|
||||||
|
let tools = defs["verifier"]["tools"].as_str().expect("a tools allowlist");
|
||||||
|
for forbidden in ["Edit", "Write"] {
|
||||||
|
assert!(
|
||||||
|
!tools.contains(forbidden),
|
||||||
|
"the verifier must not have {forbidden}: {tools}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
assert!(tools.contains("Bash"), "it has to be able to run the suite: {tools}");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The early-victory failure: a verifier that stops after the first passing
|
||||||
|
/// test reports success on broken work. Anthropic names it, and we have paid
|
||||||
|
/// for it once already.
|
||||||
|
#[test]
|
||||||
|
fn the_verifier_is_told_to_run_the_whole_suite() {
|
||||||
|
let defs = agent_definitions();
|
||||||
|
let p = defs["verifier"]["prompt"].as_str().unwrap().to_ascii_lowercase();
|
||||||
|
assert!(p.contains("complete suite"), "{p}");
|
||||||
|
assert!(p.contains(&NO_SHORTCUTS[..40].to_ascii_lowercase()), "anti-shortcut rule missing");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The definitions travel as ONE shell argument through `sh -c`, and they
|
||||||
|
/// contain double quotes, braces and apostrophes ("the project's OWN checks").
|
||||||
|
/// A quoting bug here would hand `claude` a fragment of JSON, which it would
|
||||||
|
/// reject — leaving the lead with no roles and no fan-out.
|
||||||
|
///
|
||||||
|
/// Tested by round-tripping rather than by banning apostrophes: the first
|
||||||
|
/// version of this test asserted the JSON contained none, which failed on
|
||||||
|
/// correctly-quoted text and said nothing about whether the quoting worked.
|
||||||
|
#[test]
|
||||||
|
fn the_agent_definitions_survive_shell_quoting() {
|
||||||
|
/// Undo `shell_quote`, the way `sh` would.
|
||||||
|
fn unquote(s: &str) -> String {
|
||||||
|
let inner = s
|
||||||
|
.strip_prefix('\'')
|
||||||
|
.and_then(|s| s.strip_suffix('\''))
|
||||||
|
.expect("a single-quoted argument");
|
||||||
|
inner.replace(r"'\''", "'")
|
||||||
|
}
|
||||||
|
|
||||||
|
let json = agent_definitions().to_string();
|
||||||
|
let quoted = shell_quote(&json);
|
||||||
|
assert_eq!(unquote("ed), json, "the JSON did not survive quoting");
|
||||||
|
// And it is still valid JSON on the other side.
|
||||||
|
let back: serde_json::Value =
|
||||||
|
serde_json::from_str(&unquote("ed)).expect("valid JSON after quoting");
|
||||||
|
assert!(back.get("verifier").is_some(), "{back}");
|
||||||
|
|
||||||
|
assert!(agent_command(&vm_prompt("task")).contains(&format!("--agents {quoted}")));
|
||||||
|
}
|
||||||
|
|
||||||
/// The agent must not be told to push: delivery is host-side, and the guest
|
/// The agent must not be told to push: delivery is host-side, and the guest
|
||||||
/// deliberately holds no forge credentials.
|
/// deliberately holds no forge credentials.
|
||||||
#[test]
|
#[test]
|
||||||
|
|||||||
@@ -698,6 +698,12 @@ async fn launch_microvm_phase(
|
|||||||
|
|
||||||
// The agent's own account is diagnostic only. Whether the phase succeeded
|
// The agent's own account is diagnostic only. Whether the phase succeeded
|
||||||
// is decided downstream by capture + delivery against the repository.
|
// is decided downstream by capture + delivery against the repository.
|
||||||
|
// Counted from the guest's own per-subagent transcripts. "-" means the
|
||||||
|
// probe could not run, which is not the same as "it delegated to nobody".
|
||||||
|
let subagents = match &outcome {
|
||||||
|
Ok(o) => o.subagents.map(|n| n.to_string()).unwrap_or_else(|| "?".into()),
|
||||||
|
Err(_) => "-".into(),
|
||||||
|
};
|
||||||
let (status, note) = match outcome {
|
let (status, note) = match outcome {
|
||||||
Ok(o) if o.rc == 0 && o.collected => ("completed", o.summary),
|
Ok(o) if o.rc == 0 && o.collected => ("completed", o.summary),
|
||||||
// A turn that ran and could not be collected is a failure even when
|
// A turn that ran and could not be collected is a failure even when
|
||||||
@@ -711,7 +717,8 @@ async fn launch_microvm_phase(
|
|||||||
Err(e) => ("failed", e),
|
Err(e) => ("failed", e),
|
||||||
};
|
};
|
||||||
eprintln!(
|
eprintln!(
|
||||||
"phase_runner: microvm phase {phase_id} of mission {mission_id} → {status} — {}",
|
"phase_runner: microvm phase {phase_id} of mission {mission_id} → {status} \
|
||||||
|
(subagents: {subagents}) — {}",
|
||||||
note.chars().take(300).collect::<String>()
|
note.chars().take(300).collect::<String>()
|
||||||
);
|
);
|
||||||
if let Err(e) = sqlx::query(
|
if let Err(e) = sqlx::query(
|
||||||
|
|||||||
Reference in New Issue
Block a user