feat(missions): the completion gate, moved into the agent's own loop
Every check this platform makes on a phase runs AFTER the agent has stopped: the
evaluator judges `done_when`, capture notices a coding phase delivered nothing,
and either verdict costs a whole new VM — a fresh boot, a fresh inject, and an
agent starting over with none of the context that got it that far. Meanwhile the
documented failure mode of a long-running agent is that it stops too early.
MEASURED FIRST, because the plan's chosen seam does not exist here. Probing every
hook name under `claude -p` (2.1.222, hermetic `--settings` file): `SessionStart`,
`UserPromptSubmit`, `PreToolUse`, `PostToolUse`, `SubagentStop` and `Stop` fire;
`TaskCreated`, `TaskCompleted`, `TeammateIdle`, `SessionEnd`, `Notification` and
`PreCompact` do not. The agent-teams hooks Slice 3 deferred are inert on our path
BY CONSTRUCTION — no team forms in print mode at all — so `done_when` could never
have been wired through `TaskCompleted` exit 2. `Stop` is the seam.
`vm_stop_gate` generates a POSIX `sh` hook installed via `--settings`, under
`/root/gate` and never under `/mission/repo` (anything there is collected and
arrives in the user's delivered patch). It refuses a stop when:
- the phase must deliver and the repository is untouched — asked as TWO
questions, since an agent that committed leaves a clean tree and an agent
that did not leaves HEAD alone; only both together mean nothing happened;
- `config.done_when_check` — a command the phase author wrote — exits nonzero,
in which case its OUTPUT is the feedback, not just "the check failed".
Deliberately mechanical. NOT the `done_when` verdict: that is an LLM judgement
made host-side by a different provider on purpose, and re-running it inside the
VM would put the agent's own environment in charge of grading the agent — the
correlated failure the independent judge exists to break.
THE CAP IS LOAD-BEARING. Without a ceiling a stuck agent is blocked, retries, is
blocked again, and burns the hour-long turn budget instead of failing visibly.
After 3 blocks the gate lets it stop, records that it gave up, and leaves the
verdict to the existing post-hoc path, which is unchanged.
PROVEN AGAINST A LIVE AGENT with the REAL generated artifacts, not a paraphrase:
- a read-only task → blocked 3 times with our exact message, released at
exactly the cap, and the agent took the escape hatch the message offers
("if the task genuinely requires no code change, say so explicitly") rather
than touching a file to satisfy the gate. It did not Goodhart it.
- a task that needs an edit → `blocks: 0`, log says `pass`. No false positives.
Two things that could fail silently, both closed. `--settings` is PROBED in the
image before use (`claude --help | grep`), because an unknown option is a hard
CLI error that would turn every gated phase into a failed one; a build without
it degrades to ungated and says so, since losing a check is better than losing
the work. And `stop_blocks` is reported out of the guest — `None` for no gate,
`0` for got-it-right-first-time — so a gate that never fires is distinguishable
from one that was never installed.
`require_changes` does NOT apply per node on a composed run: a graph's verifier
node is SUPPOSED to leave the tree alone, and a per-node gate would refuse its
stop three times for doing its job. `StopGate::per_node` drops it and keeps the
declared check. The phase-level rule still runs post-hoc against what the last
node collected.
An ungated phase's command is byte-identical to before, asserted by test — most
phases are gated, so the ungated path is the one nobody would notice breaking.
517 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
1d554396f4
commit
6991e21f94
@@ -24,6 +24,7 @@ pub mod mission_delivery;
|
|||||||
pub mod microvm_client;
|
pub mod microvm_client;
|
||||||
pub mod microvm_executor;
|
pub mod microvm_executor;
|
||||||
pub mod microvm_turn_executor;
|
pub mod microvm_turn_executor;
|
||||||
|
pub mod vm_stop_gate;
|
||||||
pub mod mission_fs;
|
pub mod mission_fs;
|
||||||
pub mod papers;
|
pub mod papers;
|
||||||
pub mod phase_config;
|
pub mod phase_config;
|
||||||
|
|||||||
@@ -297,8 +297,23 @@ const SUBAGENT_PROBE: &str =
|
|||||||
const TEAMMATE_PROBE: &str = "cat /root/.claude/teams/*/config.json 2>/dev/null \
|
const TEAMMATE_PROBE: &str = "cat /root/.claude/teams/*/config.json 2>/dev/null \
|
||||||
| tr ',' '\\n' | grep -c '\"name\"' || true";
|
| tr ',' '\\n' | grep -c '\"name\"' || true";
|
||||||
|
|
||||||
|
/// Does the CLI in THIS image accept `--settings`?
|
||||||
|
///
|
||||||
|
/// Asked of the binary rather than assumed from a version number. The stop gate
|
||||||
|
/// is installed through that flag, and an unknown option is a hard CLI error —
|
||||||
|
/// so without this probe, a Claude Code build that dropped the flag would turn
|
||||||
|
/// every gated phase into a failed one. With it, the gate degrades to absent and
|
||||||
|
/// says so, which is the difference between losing a check and losing the work.
|
||||||
|
const SETTINGS_PROBE: &str = "claude --help 2>&1 | grep -q -- '--settings' && echo SETTINGS-OK";
|
||||||
|
|
||||||
|
/// How many times the stop gate refused to let the agent finish.
|
||||||
|
const BLOCKS_PROBE: &str = "cat /root/gate/blocks 2>/dev/null || echo 0";
|
||||||
|
|
||||||
/// The command that runs the agent in the guest.
|
/// The command that runs the agent in the guest.
|
||||||
fn agent_command(prompt: &str) -> String {
|
///
|
||||||
|
/// `settings` is the path to the stop-gate settings file, when a gate is
|
||||||
|
/// installed. It lives under `/root`, never in the repo — see [`vm_stop_gate`].
|
||||||
|
fn agent_command(prompt: &str, settings: Option<&str>) -> String {
|
||||||
// --permission-mode acceptEdits, matching the container path: the VM IS the
|
// --permission-mode acceptEdits, matching the container path: the VM IS the
|
||||||
// boundary, so prompting for permission inside it would only mean a turn that
|
// boundary, so prompting for permission inside it would only mean a turn that
|
||||||
// waits for an answer nobody can give.
|
// waits for an answer nobody can give.
|
||||||
@@ -312,9 +327,13 @@ fn agent_command(prompt: &str) -> String {
|
|||||||
// CLI. So the built-in roles stay available alongside ours.
|
// CLI. So the built-in roles stay available alongside ours.
|
||||||
format!(
|
format!(
|
||||||
"cd {GUEST_REPO} && claude -p --allowedTools {} \
|
"cd {GUEST_REPO} && claude -p --allowedTools {} \
|
||||||
--permission-mode acceptEdits --agents {} {}",
|
--permission-mode acceptEdits --agents {}{} {}",
|
||||||
LEAD_TOOLS.join(" "),
|
LEAD_TOOLS.join(" "),
|
||||||
shell_quote(&agent_definitions().to_string()),
|
shell_quote(&agent_definitions().to_string()),
|
||||||
|
match settings {
|
||||||
|
Some(path) => format!(" --settings {}", shell_quote(path)),
|
||||||
|
None => String::new(),
|
||||||
|
},
|
||||||
shell_quote(prompt)
|
shell_quote(prompt)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
@@ -338,6 +357,10 @@ pub struct VmOutcome {
|
|||||||
/// Teammates the lead spawned, for a mission that asked for a team. `None`
|
/// Teammates the lead spawned, for a mission that asked for a team. `None`
|
||||||
/// when no team was requested, or when the probe could not run.
|
/// when no team was requested, or when the probe could not run.
|
||||||
pub teammates: Option<u32>,
|
pub teammates: Option<u32>,
|
||||||
|
/// How many times the stop gate refused to let the agent finish. `None` when
|
||||||
|
/// no gate was installed. Zero means the agent got it right first time,
|
||||||
|
/// which is a different fact from "there was no gate".
|
||||||
|
pub stop_blocks: Option<u32>,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Boot a VM, run the phase in it, collect the result, and destroy it.
|
/// Boot a VM, run the phase in it, collect the result, and destroy it.
|
||||||
@@ -354,7 +377,7 @@ pub async fn run_phase_in_vm(hub: &NodeHub, p: VmPhase<'_>) -> Result<VmOutcome,
|
|||||||
|
|
||||||
// From here on every early return must still destroy the VM, so the work is
|
// From here on every early return must still destroy the VM, so the work is
|
||||||
// one call whose result is held while teardown runs unconditionally.
|
// one call whose result is held while teardown runs unconditionally.
|
||||||
let outcome = run_inside(&vm, &created, p.task, p.repo, &env, p.team_engine).await;
|
let outcome = run_inside(&vm, &created, p.task, p.repo, &env, p.team_engine, p.gate).await;
|
||||||
|
|
||||||
if let Err(e) = vm.destroy().await {
|
if let Err(e) = vm.destroy().await {
|
||||||
// Not fatal to the phase — the work may already be collected — but loud,
|
// Not fatal to the phase — the work may already be collected — but loud,
|
||||||
@@ -386,6 +409,10 @@ pub struct VmPhase<'a> {
|
|||||||
/// `missions.team_engine` — `Some("claude_code")` asks the lead to form a
|
/// `missions.team_engine` — `Some("claude_code")` asks the lead to form a
|
||||||
/// team. `None` is solo, which is the default.
|
/// team. `None` is solo, which is the default.
|
||||||
pub team_engine: Option<&'a str>,
|
pub team_engine: Option<&'a str>,
|
||||||
|
/// What must hold before the agent is allowed to stop, installed as a
|
||||||
|
/// Claude Code `Stop` hook inside the guest. `None` leaves the turn exactly
|
||||||
|
/// as it was.
|
||||||
|
pub gate: Option<&'a crate::vm_stop_gate::StopGate>,
|
||||||
/// Which node of a composed graph this VM is running, if any. `None` is the
|
/// Which node of a composed graph this VM is running, if any. `None` is the
|
||||||
/// solo path, where the phase is one VM and the id needs no further
|
/// solo path, where the phase is one VM and the id needs no further
|
||||||
/// qualification. Part of the vm id, so the nodes of one phase-iteration
|
/// qualification. Part of the vm id, so the nodes of one phase-iteration
|
||||||
@@ -430,6 +457,7 @@ async fn run_inside(
|
|||||||
repo: &std::path::Path,
|
repo: &std::path::Path,
|
||||||
env: &[(String, String)],
|
env: &[(String, String)],
|
||||||
engine: Option<&str>,
|
engine: Option<&str>,
|
||||||
|
gate: Option<&crate::vm_stop_gate::StopGate>,
|
||||||
) -> Result<VmOutcome, String> {
|
) -> Result<VmOutcome, String> {
|
||||||
// An agent CLI cannot reach its API without the tunnel, and a turn without
|
// An agent CLI cannot reach its API without the tunnel, and a turn without
|
||||||
// egress does not fail — it hangs, or reports a network error the operator
|
// egress does not fail — it hangs, or reports a network error the operator
|
||||||
@@ -479,8 +507,63 @@ async fn run_inside(
|
|||||||
let mut turn_env = env.to_vec();
|
let mut turn_env = env.to_vec();
|
||||||
turn_env.extend(team_env(engine));
|
turn_env.extend(team_env(engine));
|
||||||
|
|
||||||
|
// The stop gate, installed before the turn. Every failure here is degraded
|
||||||
|
// to "no gate" and logged: the gate makes a phase converge in ONE VM instead
|
||||||
|
// of a second one, so losing it costs money and time — while failing the
|
||||||
|
// turn over it would cost the work.
|
||||||
|
let settings = match gate {
|
||||||
|
None => None,
|
||||||
|
Some(g) => {
|
||||||
|
let supported = vm
|
||||||
|
.exec(SETTINGS_PROBE, None, 60, &[])
|
||||||
|
.await
|
||||||
|
.map(|p| p.stdout.contains("SETTINGS-OK"))
|
||||||
|
.unwrap_or(false);
|
||||||
|
if !supported {
|
||||||
|
eprintln!(
|
||||||
|
"microvm_executor: {} has no `claude --settings`, so the stop gate \
|
||||||
|
cannot be installed — this phase runs ungated and its completion \
|
||||||
|
is judged only after the fact",
|
||||||
|
vm.vm_id()
|
||||||
|
);
|
||||||
|
None
|
||||||
|
} else {
|
||||||
|
let install = g.install_command(GUEST_REPO, crate::vm_stop_gate::GATE_DIR);
|
||||||
|
match vm.exec(&install, None, 60, &[]).await {
|
||||||
|
Ok(o) if o.rc == 0 => Some(format!(
|
||||||
|
"{}/settings.json",
|
||||||
|
crate::vm_stop_gate::GATE_DIR
|
||||||
|
)),
|
||||||
|
Ok(o) => {
|
||||||
|
eprintln!(
|
||||||
|
"microvm_executor: could not install the stop gate on {} \
|
||||||
|
(rc={}): {} — running ungated",
|
||||||
|
vm.vm_id(),
|
||||||
|
o.rc,
|
||||||
|
o.stderr
|
||||||
|
);
|
||||||
|
None
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!(
|
||||||
|
"microvm_executor: could not install the stop gate on {}: {e} \
|
||||||
|
— running ungated",
|
||||||
|
vm.vm_id()
|
||||||
|
);
|
||||||
|
None
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
let out = vm
|
let out = vm
|
||||||
.exec(&agent_command(&prompt), None, TURN_SECS, &turn_env)
|
.exec(
|
||||||
|
&agent_command(&prompt, settings.as_deref()),
|
||||||
|
None,
|
||||||
|
TURN_SECS,
|
||||||
|
&turn_env,
|
||||||
|
)
|
||||||
.await?;
|
.await?;
|
||||||
|
|
||||||
// Ask the guest how many subagents ran, before collecting: the transcripts
|
// Ask the guest how many subagents ran, before collecting: the transcripts
|
||||||
@@ -546,12 +629,26 @@ async fn run_inside(
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// How many times the gate sent the agent back. Read only when a gate was
|
||||||
|
// actually installed, so `None` never means "it never fired".
|
||||||
|
let stop_blocks = match settings {
|
||||||
|
None => None,
|
||||||
|
Some(_) => match vm.exec(BLOCKS_PROBE, None, 60, &[]).await {
|
||||||
|
Ok(p) => p.stdout.trim().parse::<u32>().ok(),
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!("microvm_executor: stop-gate probe failed on {}: {e}", vm.vm_id());
|
||||||
|
None
|
||||||
|
}
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
Ok(VmOutcome {
|
Ok(VmOutcome {
|
||||||
summary: format!("{}{}", out.stdout, out.stderr),
|
summary: format!("{}{}", out.stdout, out.stderr),
|
||||||
rc: out.rc,
|
rc: out.rc,
|
||||||
collected,
|
collected,
|
||||||
subagents,
|
subagents,
|
||||||
teammates,
|
teammates,
|
||||||
|
stop_blocks,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -602,7 +699,7 @@ mod tests {
|
|||||||
#[test]
|
#[test]
|
||||||
fn a_quoted_task_cannot_break_out_of_the_command() {
|
fn a_quoted_task_cannot_break_out_of_the_command() {
|
||||||
let nasty = "it's a 'test'; rm -rf /\n$(whoami) `id` \"quoted\"";
|
let nasty = "it's a 'test'; rm -rf /\n$(whoami) `id` \"quoted\"";
|
||||||
let cmd = agent_command(&vm_prompt(nasty));
|
let cmd = agent_command(&vm_prompt(nasty), None);
|
||||||
// Everything after the opening quote of the prompt is inside it: the only
|
// Everything after the opening quote of the prompt is inside it: the only
|
||||||
// way out of a single-quoted string is a quote, and each one is escaped.
|
// way out of a single-quoted string is a quote, and each one is escaped.
|
||||||
let body = cmd.split_once('\'').expect("a quoted argument").1;
|
let body = cmd.split_once('\'').expect("a quoted argument").1;
|
||||||
@@ -627,7 +724,7 @@ mod tests {
|
|||||||
LEAD_TOOLS.contains(&SUBAGENT_TOOL),
|
LEAD_TOOLS.contains(&SUBAGENT_TOOL),
|
||||||
"without {SUBAGENT_TOOL} in the allowlist there is no delegation, silently"
|
"without {SUBAGENT_TOOL} in the allowlist there is no delegation, silently"
|
||||||
);
|
);
|
||||||
let cmd = agent_command(&vm_prompt("t"));
|
let cmd = agent_command(&vm_prompt("t"), None);
|
||||||
assert!(cmd.contains(&format!("--allowedTools {}", LEAD_TOOLS.join(" "))), "{cmd}");
|
assert!(cmd.contains(&format!("--allowedTools {}", LEAD_TOOLS.join(" "))), "{cmd}");
|
||||||
// `--forward-subagent-text` is deliberately absent: it refuses to run
|
// `--forward-subagent-text` is deliberately absent: it refuses to run
|
||||||
// without `--output-format=stream-json`, which would change how this
|
// without `--output-format=stream-json`, which would change how this
|
||||||
@@ -727,7 +824,7 @@ mod tests {
|
|||||||
serde_json::from_str(&unquote("ed)).expect("valid JSON after quoting");
|
serde_json::from_str(&unquote("ed)).expect("valid JSON after quoting");
|
||||||
assert!(back.get("verifier").is_some(), "{back}");
|
assert!(back.get("verifier").is_some(), "{back}");
|
||||||
|
|
||||||
assert!(agent_command(&vm_prompt("task")).contains(&format!("--agents {quoted}")));
|
assert!(agent_command(&vm_prompt("task"), None).contains(&format!("--agents {quoted}")));
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The tier this executor inserts must NOT be one the topology worker drives.
|
/// The tier this executor inserts must NOT be one the topology worker drives.
|
||||||
@@ -819,6 +916,39 @@ mod tests {
|
|||||||
assert!(!TEAMMATE_PROBE.contains("subagents"));
|
assert!(!TEAMMATE_PROBE.contains("subagents"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// An ungated phase's command must be byte-identical to what it was before
|
||||||
|
/// the stop gate existed. Most phases are gated, so the ungated path is the
|
||||||
|
/// one nobody would notice breaking.
|
||||||
|
#[test]
|
||||||
|
fn a_phase_without_a_gate_gets_the_command_it_always_had() {
|
||||||
|
let cmd = agent_command(&vm_prompt("t"), None);
|
||||||
|
assert!(!cmd.contains("--settings"), "{cmd}");
|
||||||
|
|
||||||
|
let gated = agent_command(&vm_prompt("t"), Some("/root/gate/settings.json"));
|
||||||
|
assert!(gated.contains("--settings '/root/gate/settings.json'"), "{gated}");
|
||||||
|
// The flag is the ONLY difference — a gate must not quietly change the
|
||||||
|
// tools, the permission mode or the roles.
|
||||||
|
assert_eq!(
|
||||||
|
gated.replace(" --settings '/root/gate/settings.json'", ""),
|
||||||
|
cmd
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The gate is installed through `--settings`, so whether the CLI in the
|
||||||
|
/// image HAS that flag is a fact about the image, not about our plan. An
|
||||||
|
/// unknown option is a hard error, which would turn every gated phase into a
|
||||||
|
/// failed one — hence a probe rather than a version check.
|
||||||
|
#[test]
|
||||||
|
fn the_settings_flag_is_probed_before_it_is_used() {
|
||||||
|
assert!(SETTINGS_PROBE.contains("--help"), "{SETTINGS_PROBE}");
|
||||||
|
assert!(SETTINGS_PROBE.contains("--settings"), "{SETTINGS_PROBE}");
|
||||||
|
assert!(SETTINGS_PROBE.contains("SETTINGS-OK"), "{SETTINGS_PROBE}");
|
||||||
|
// And the evidence probe reads the gate's own counter, outside the tree
|
||||||
|
// that gets collected and diffed.
|
||||||
|
assert!(BLOCKS_PROBE.contains(crate::vm_stop_gate::GATE_DIR), "{BLOCKS_PROBE}");
|
||||||
|
assert!(!BLOCKS_PROBE.contains(GUEST_REPO), "{BLOCKS_PROBE}");
|
||||||
|
}
|
||||||
|
|
||||||
/// The agent must not be told to push: delivery is host-side, and the guest
|
/// The agent must not be told to push: delivery is host-side, and the guest
|
||||||
/// deliberately holds no forge credentials.
|
/// deliberately holds no forge credentials.
|
||||||
#[test]
|
#[test]
|
||||||
|
|||||||
@@ -87,6 +87,8 @@ pub struct MicroVmTurnExecutor<V: PhaseVm> {
|
|||||||
/// `missions.team_engine`, passed through so a composed node can itself ask
|
/// `missions.team_engine`, passed through so a composed node can itself ask
|
||||||
/// for Claude Code fan-out inside its VM.
|
/// for Claude Code fan-out inside its VM.
|
||||||
team_engine: Option<String>,
|
team_engine: Option<String>,
|
||||||
|
/// The phase's completion gate, enforced inside every node's VM.
|
||||||
|
gate: Option<crate::vm_stop_gate::StopGate>,
|
||||||
/// Which step is next. `execute_resumable` is sequential and gives the
|
/// Which step is next. `execute_resumable` is sequential and gives the
|
||||||
/// executor no index, so the executor counts — and the count starts from the
|
/// executor no index, so the executor counts — and the count starts from the
|
||||||
/// checkpoint on resume, or two VMs would share an id across a restart.
|
/// checkpoint on resume, or two VMs would share an id across a restart.
|
||||||
@@ -103,6 +105,8 @@ pub struct ComposedRun {
|
|||||||
pub target_node_id: Option<Uuid>,
|
pub target_node_id: Option<Uuid>,
|
||||||
pub backend: Option<String>,
|
pub backend: Option<String>,
|
||||||
pub team_engine: Option<String>,
|
pub team_engine: Option<String>,
|
||||||
|
/// What must hold before a node's agent may stop. See [`crate::vm_stop_gate`].
|
||||||
|
pub gate: Option<crate::vm_stop_gate::StopGate>,
|
||||||
/// Steps already completed, from the durable checkpoint. Nonzero on resume.
|
/// Steps already completed, from the durable checkpoint. Nonzero on resume.
|
||||||
pub completed_steps: u32,
|
pub completed_steps: u32,
|
||||||
}
|
}
|
||||||
@@ -120,6 +124,7 @@ impl<V: PhaseVm> MicroVmTurnExecutor<V> {
|
|||||||
default_fleet_node: r.target_node_id,
|
default_fleet_node: r.target_node_id,
|
||||||
default_backend: r.backend,
|
default_backend: r.backend,
|
||||||
team_engine: r.team_engine,
|
team_engine: r.team_engine,
|
||||||
|
gate: r.gate,
|
||||||
step: std::sync::atomic::AtomicU32::new(r.completed_steps),
|
step: std::sync::atomic::AtomicU32::new(r.completed_steps),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -189,6 +194,10 @@ impl<V: PhaseVm> TurnExecutor for MicroVmTurnExecutor<V> {
|
|||||||
backend: backend.as_deref(),
|
backend: backend.as_deref(),
|
||||||
repo: &self.repo,
|
repo: &self.repo,
|
||||||
team_engine: self.team_engine.as_deref(),
|
team_engine: self.team_engine.as_deref(),
|
||||||
|
// Each node is its own agent session, so each carries the
|
||||||
|
// phase's gate. Threaded from the run rather than rebuilt here:
|
||||||
|
// one source for what "done" means, whichever executor asks.
|
||||||
|
gate: self.gate.as_ref(),
|
||||||
step: Some(step),
|
step: Some(step),
|
||||||
})
|
})
|
||||||
.await
|
.await
|
||||||
@@ -366,6 +375,7 @@ mod tests {
|
|||||||
collected: true,
|
collected: true,
|
||||||
subagents: Some(0),
|
subagents: Some(0),
|
||||||
teammates: None,
|
teammates: None,
|
||||||
|
stop_blocks: None,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -401,6 +411,7 @@ mod tests {
|
|||||||
target_node_id: Some(Uuid::now_v7()),
|
target_node_id: Some(Uuid::now_v7()),
|
||||||
backend: Some("claude".into()),
|
backend: Some("claude".into()),
|
||||||
team_engine: None,
|
team_engine: None,
|
||||||
|
gate: None,
|
||||||
completed_steps: 0,
|
completed_steps: 0,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
@@ -495,6 +506,7 @@ mod tests {
|
|||||||
target_node_id: Some(Uuid::now_v7()),
|
target_node_id: Some(Uuid::now_v7()),
|
||||||
backend: None,
|
backend: None,
|
||||||
team_engine: None,
|
team_engine: None,
|
||||||
|
gate: None,
|
||||||
completed_steps: 2,
|
completed_steps: 2,
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
@@ -554,6 +566,7 @@ mod tests {
|
|||||||
collected: false,
|
collected: false,
|
||||||
subagents: None,
|
subagents: None,
|
||||||
teammates: None,
|
teammates: None,
|
||||||
|
stop_blocks: None,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -251,6 +251,11 @@ async fn start_pending_phases(
|
|||||||
let rows = sqlx::query(
|
let rows = sqlx::query(
|
||||||
"SELECT mp.id, mp.mission_id, mp.kind, mp.order_idx, mp.iteration,
|
"SELECT mp.id, mp.mission_id, mp.kind, mp.order_idx, mp.iteration,
|
||||||
mp.config->>'task' AS phase_task,
|
mp.config->>'task' AS phase_task,
|
||||||
|
-- The whole config, not just the task: the stop gate is built
|
||||||
|
-- from `allow_empty` and `done_when_check`, and a phase that
|
||||||
|
-- declares a completion check gets it enforced in the agent's
|
||||||
|
-- own loop rather than only after it has finished.
|
||||||
|
mp.config,
|
||||||
m.workspace_id, m.title, m.description,
|
m.workspace_id, m.title, m.description,
|
||||||
-- Where and how this mission executes. `runtime_kind` decides
|
-- Where and how this mission executes. `runtime_kind` decides
|
||||||
-- which executor takes the phase; without it 'microvm' is a
|
-- which executor takes the phase; without it 'microvm' is a
|
||||||
@@ -280,6 +285,7 @@ async fn start_pending_phases(
|
|||||||
let title: String = row.get("title");
|
let title: String = row.get("title");
|
||||||
let description: Option<String> = row.get("description");
|
let description: Option<String> = row.get("description");
|
||||||
let phase_task: Option<String> = row.get("phase_task");
|
let phase_task: Option<String> = row.get("phase_task");
|
||||||
|
let config: serde_json::Value = row.get("config");
|
||||||
let iteration: i32 = row.get("iteration");
|
let iteration: i32 = row.get("iteration");
|
||||||
let runtime_kind: String = row.get("runtime_kind");
|
let runtime_kind: String = row.get("runtime_kind");
|
||||||
let backend: Option<String> = row.get("backend");
|
let backend: Option<String> = row.get("backend");
|
||||||
@@ -297,6 +303,7 @@ async fn start_pending_phases(
|
|||||||
title: &title,
|
title: &title,
|
||||||
description: description.as_deref(),
|
description: description.as_deref(),
|
||||||
phase_task: phase_task.as_deref(),
|
phase_task: phase_task.as_deref(),
|
||||||
|
config: &config,
|
||||||
iteration,
|
iteration,
|
||||||
runtime_kind: &runtime_kind,
|
runtime_kind: &runtime_kind,
|
||||||
backend: backend.as_deref(),
|
backend: backend.as_deref(),
|
||||||
@@ -329,6 +336,9 @@ struct PhaseLaunch<'a> {
|
|||||||
/// coding phases with distinct `task` values both produced the same two
|
/// coding phases with distinct `task` values both produced the same two
|
||||||
/// files, because neither phase ever saw its own instructions.
|
/// files, because neither phase ever saw its own instructions.
|
||||||
phase_task: Option<&'a str>,
|
phase_task: Option<&'a str>,
|
||||||
|
/// `mission_phases.config`, for the settings that shape execution rather
|
||||||
|
/// than describe the work — currently the stop gate's two.
|
||||||
|
config: &'a serde_json::Value,
|
||||||
/// Which pass this is, 0-based. Stamped onto the runs so the completion
|
/// Which pass this is, 0-based. Stamped onto the runs so the completion
|
||||||
/// check can tell this pass's work from the previous one's.
|
/// check can tell this pass's work from the previous one's.
|
||||||
iteration: i32,
|
iteration: i32,
|
||||||
@@ -359,6 +369,7 @@ async fn launch_phase(
|
|||||||
iteration,
|
iteration,
|
||||||
// Destructured but read through `p` below, so the compiler keeps this
|
// Destructured but read through `p` below, so the compiler keeps this
|
||||||
// pattern honest if a field is added.
|
// pattern honest if a field is added.
|
||||||
|
config: _,
|
||||||
runtime_kind: _,
|
runtime_kind: _,
|
||||||
backend: _,
|
backend: _,
|
||||||
target_node_id: _,
|
target_node_id: _,
|
||||||
@@ -561,6 +572,7 @@ async fn launch_phase(
|
|||||||
p.backend,
|
p.backend,
|
||||||
p.target_node_id,
|
p.target_node_id,
|
||||||
p.team_engine,
|
p.team_engine,
|
||||||
|
crate::vm_stop_gate::StopGate::for_phase(kind, p.config),
|
||||||
)
|
)
|
||||||
.await;
|
.await;
|
||||||
}
|
}
|
||||||
@@ -816,6 +828,7 @@ async fn launch_microvm_phase(
|
|||||||
backend: Option<&str>,
|
backend: Option<&str>,
|
||||||
target_node_id: Option<Uuid>,
|
target_node_id: Option<Uuid>,
|
||||||
team_engine: Option<&str>,
|
team_engine: Option<&str>,
|
||||||
|
gate: Option<crate::vm_stop_gate::StopGate>,
|
||||||
) -> Result<(), String> {
|
) -> Result<(), String> {
|
||||||
sqlx::query(
|
sqlx::query(
|
||||||
"DELETE FROM topology_runs
|
"DELETE FROM topology_runs
|
||||||
@@ -889,6 +902,7 @@ async fn launch_microvm_phase(
|
|||||||
backend: backend.as_deref(),
|
backend: backend.as_deref(),
|
||||||
repo: &repo,
|
repo: &repo,
|
||||||
team_engine: team_engine.as_deref(),
|
team_engine: team_engine.as_deref(),
|
||||||
|
gate: gate.as_ref(),
|
||||||
// The solo path is one VM for the whole phase; only a
|
// The solo path is one VM for the whole phase; only a
|
||||||
// composed run needs the id qualified per graph node.
|
// composed run needs the id qualified per graph node.
|
||||||
step: None,
|
step: None,
|
||||||
@@ -911,6 +925,13 @@ async fn launch_microvm_phase(
|
|||||||
Ok(o) => o.teammates.map(|n| n.to_string()).unwrap_or_else(|| "-".into()),
|
Ok(o) => o.teammates.map(|n| n.to_string()).unwrap_or_else(|| "-".into()),
|
||||||
Err(_) => "-".into(),
|
Err(_) => "-".into(),
|
||||||
};
|
};
|
||||||
|
// How often the completion gate sent the agent back inside its own turn.
|
||||||
|
// "-" is no gate; a number is how many second chances it took, which is
|
||||||
|
// the whole measurement of whether the gate is worth its hook.
|
||||||
|
let blocked = match &outcome {
|
||||||
|
Ok(o) => o.stop_blocks.map(|n| n.to_string()).unwrap_or_else(|| "-".into()),
|
||||||
|
Err(_) => "-".into(),
|
||||||
|
};
|
||||||
let (status, note) = match outcome {
|
let (status, note) = match outcome {
|
||||||
Ok(o) if o.rc == 0 && o.collected => ("completed", o.summary),
|
Ok(o) if o.rc == 0 && o.collected => ("completed", o.summary),
|
||||||
// A turn that ran and could not be collected is a failure even when
|
// A turn that ran and could not be collected is a failure even when
|
||||||
@@ -925,7 +946,8 @@ async fn launch_microvm_phase(
|
|||||||
};
|
};
|
||||||
eprintln!(
|
eprintln!(
|
||||||
"phase_runner: microvm phase {phase_id} of mission {mission_id} → {status} \
|
"phase_runner: microvm phase {phase_id} of mission {mission_id} → {status} \
|
||||||
(subagents: {subagents}, teammates: {teammates}) — {}",
|
(subagents: {subagents}, teammates: {teammates}, stop-gate blocks: \
|
||||||
|
{blocked}) — {}",
|
||||||
note.chars().take(300).collect::<String>()
|
note.chars().take(300).collect::<String>()
|
||||||
);
|
);
|
||||||
// Never overwrite a cancellation. The operator asking to stop is a decision;
|
// Never overwrite a cancellation. The operator asking to stop is a decision;
|
||||||
|
|||||||
@@ -293,6 +293,16 @@ async fn run_composed(
|
|||||||
.await
|
.await
|
||||||
.map_err(|e| OrchestratorError::Executor(format!("load mission {mission_id}: {e}")))?;
|
.map_err(|e| OrchestratorError::Executor(format!("load mission {mission_id}: {e}")))?;
|
||||||
|
|
||||||
|
// The phase's completion gate, read here rather than carried on the run row
|
||||||
|
// so an edited `done_when_check` takes effect on the next node instead of at
|
||||||
|
// the next mission.
|
||||||
|
let phase: (String, serde_json::Value) =
|
||||||
|
sqlx::query_as("SELECT kind, config FROM mission_phases WHERE id = $1")
|
||||||
|
.bind(phase_id)
|
||||||
|
.fetch_one(pool)
|
||||||
|
.await
|
||||||
|
.map_err(|e| OrchestratorError::Executor(format!("load phase {phase_id}: {e}")))?;
|
||||||
|
|
||||||
let exec = crate::microvm_turn_executor::for_fleet(
|
let exec = crate::microvm_turn_executor::for_fleet(
|
||||||
hub.clone(),
|
hub.clone(),
|
||||||
pool.clone(),
|
pool.clone(),
|
||||||
@@ -305,6 +315,8 @@ async fn run_composed(
|
|||||||
target_node_id: mission.0,
|
target_node_id: mission.0,
|
||||||
backend: mission.1,
|
backend: mission.1,
|
||||||
team_engine: mission.2,
|
team_engine: mission.2,
|
||||||
|
gate: crate::vm_stop_gate::StopGate::for_phase(&phase.0, &phase.1)
|
||||||
|
.and_then(crate::vm_stop_gate::StopGate::per_node),
|
||||||
// Resume continues the step numbering; restarting it would re-use a
|
// Resume continues the step numbering; restarting it would re-use a
|
||||||
// finished node's vm id.
|
// finished node's vm id.
|
||||||
completed_steps: progress.completed as u32,
|
completed_steps: progress.completed as u32,
|
||||||
|
|||||||
@@ -0,0 +1,442 @@
|
|||||||
|
//! The completion gate, moved into the agent's own loop.
|
||||||
|
//!
|
||||||
|
//! Every check this platform has on a phase runs **after** the agent has
|
||||||
|
//! finished: the evaluator judges `done_when`, capture notices that a coding
|
||||||
|
//! phase delivered nothing, and either verdict costs a whole new VM — a fresh
|
||||||
|
//! boot, a fresh inject, and an agent starting again with none of the context
|
||||||
|
//! that got it that far. Meanwhile the documented failure of a long-running
|
||||||
|
//! agent is that it *stops too early*.
|
||||||
|
//!
|
||||||
|
//! Claude Code's `Stop` hook is the seam. **Exit code 2 blocks the stop and
|
||||||
|
//! feeds stderr back to the model as the reason.** Measured, not read off docs —
|
||||||
|
//! an agent told "say hello and do nothing else", whose `Stop` hook exited 2
|
||||||
|
//! saying `evidence.txt` was missing, created `evidence.txt` and then stopped.
|
||||||
|
//!
|
||||||
|
//! # What it may and may not check
|
||||||
|
//!
|
||||||
|
//! Deliberately mechanical: whether the repository changed, and whether a
|
||||||
|
//! command the phase author wrote exits 0. NOT the `done_when` verdict — that is
|
||||||
|
//! an LLM judgement made host-side by a *different provider* on purpose
|
||||||
|
//! ([[evaluator-verification]]), and re-implementing it inside the VM would put
|
||||||
|
//! the agent's own environment in charge of grading the agent, which is the
|
||||||
|
//! correlated failure the independent judge exists to break.
|
||||||
|
//!
|
||||||
|
//! # The cap is load-bearing
|
||||||
|
//!
|
||||||
|
//! A gate with no ceiling turns a stuck agent into a wedged one: it would be
|
||||||
|
//! blocked, retry, be blocked again, and burn the hour-long turn budget instead
|
||||||
|
//! of failing in a way the operator can see. After [`MAX_BLOCKS`] the gate lets
|
||||||
|
//! the agent stop, records that it did, and leaves the verdict to the existing
|
||||||
|
//! post-hoc path — which still runs, unchanged.
|
||||||
|
//!
|
||||||
|
//! # Which hooks exist here
|
||||||
|
//!
|
||||||
|
//! `TaskCompleted` / `TeammateIdle` were the plan's chosen seam. Measured under
|
||||||
|
//! `claude -p`: they never fire, because no team forms in print mode at all.
|
||||||
|
//! `Stop`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `UserPromptSubmit` and
|
||||||
|
//! `SessionStart` do.
|
||||||
|
|
||||||
|
use serde_json::json;
|
||||||
|
|
||||||
|
/// How many times the gate may refuse a stop before it gives up and lets the
|
||||||
|
/// agent finish. Three is enough for "you wrote nothing" → "you wrote something"
|
||||||
|
/// → "your check passes" without ever approaching the turn budget.
|
||||||
|
pub const MAX_BLOCKS: u32 = 3;
|
||||||
|
|
||||||
|
/// Where the gate lives in the guest.
|
||||||
|
///
|
||||||
|
/// Under `/root`, never under the repository. Anything written into
|
||||||
|
/// `/mission/repo` is collected and diffed, so a gate script placed there would
|
||||||
|
/// arrive in the user's delivered patch as if an agent had authored it.
|
||||||
|
pub const GATE_DIR: &str = "/root/gate";
|
||||||
|
|
||||||
|
/// What must hold before this phase's agent is allowed to stop.
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
|
pub struct StopGate {
|
||||||
|
/// The phase must leave the repository changed. Set for coding phases that
|
||||||
|
/// have not declared `allow_empty` — the same rule
|
||||||
|
/// `empty_delivery_is_a_failure` applies post-hoc, applied while the agent
|
||||||
|
/// can still do something about it.
|
||||||
|
pub require_changes: bool,
|
||||||
|
/// `config.done_when_check`: a shell command, run in the repo, that must
|
||||||
|
/// exit 0. The deterministic half of a completion condition — a command,
|
||||||
|
/// not a judgement.
|
||||||
|
pub check: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl StopGate {
|
||||||
|
/// The gate for a phase, or `None` when there is nothing to enforce.
|
||||||
|
///
|
||||||
|
/// `None` matters: installing a hook that can never block would still cost a
|
||||||
|
/// process per stop and would put a `--settings` flag on the command line
|
||||||
|
/// for no reason.
|
||||||
|
pub fn for_phase(kind: &str, config: &serde_json::Value) -> Option<StopGate> {
|
||||||
|
let allow_empty = config.get("allow_empty").and_then(|v| v.as_bool()) == Some(true);
|
||||||
|
let check = config
|
||||||
|
.get("done_when_check")
|
||||||
|
.and_then(|v| v.as_str())
|
||||||
|
.map(str::trim)
|
||||||
|
.filter(|s| !s.is_empty())
|
||||||
|
.map(str::to_string);
|
||||||
|
let require_changes = kind == "coding" && !allow_empty;
|
||||||
|
if !require_changes && check.is_none() {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
Some(StopGate {
|
||||||
|
require_changes,
|
||||||
|
check,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The same gate, for ONE NODE of a composed run.
|
||||||
|
///
|
||||||
|
/// `require_changes` is a property of the phase, not of every node in it: a
|
||||||
|
/// graph whose second node reviews or verifies is *supposed* to leave the
|
||||||
|
/// tree alone, and a per-node gate would refuse its stop three times for
|
||||||
|
/// doing exactly its job. The phase-level check still runs post-hoc against
|
||||||
|
/// what the last node collected, so nothing is lost — only misapplied.
|
||||||
|
///
|
||||||
|
/// A declared `check` DOES apply per node: it is a command the phase author
|
||||||
|
/// wrote, and every stage of the work should satisfy it.
|
||||||
|
pub fn per_node(self) -> Option<StopGate> {
|
||||||
|
self.check.map(|check| StopGate {
|
||||||
|
require_changes: false,
|
||||||
|
check: Some(check),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The hook script, as POSIX `sh`.
|
||||||
|
///
|
||||||
|
/// `repo` and `dir` are parameters rather than the constants above so a test
|
||||||
|
/// can run this script — the real one, not a paraphrase — against a real git
|
||||||
|
/// repository in a temp directory.
|
||||||
|
pub fn script(&self, repo: &str, dir: &str) -> String {
|
||||||
|
let mut s = String::from("#!/bin/sh\n# ClawMates stop gate. Exit 2 refuses the stop.\n");
|
||||||
|
s.push_str(&format!("REPO={}\nGATE={}\nMAX={MAX_BLOCKS}\n", q(repo), q(dir)));
|
||||||
|
s.push_str("N=$(cat \"$GATE/blocks\" 2>/dev/null || echo 0)\nreason=''\n");
|
||||||
|
|
||||||
|
if self.require_changes {
|
||||||
|
// Two questions, because either alone is answerable "no" by a
|
||||||
|
// perfectly good phase: an agent that committed its work leaves a
|
||||||
|
// clean tree, and an agent that did not commit leaves HEAD where it
|
||||||
|
// was. Only both together mean nothing happened.
|
||||||
|
s.push_str(
|
||||||
|
"BASE=$(cat \"$REPO/.git/clawmates-base\" 2>/dev/null || echo '')\n\
|
||||||
|
DIRTY=$(git -C \"$REPO\" status --porcelain 2>/dev/null | head -c 400)\n\
|
||||||
|
HEAD=$(git -C \"$REPO\" rev-parse HEAD 2>/dev/null || echo '')\n\
|
||||||
|
if [ -z \"$DIRTY\" ] && [ -n \"$BASE\" ] && [ \"$HEAD\" = \"$BASE\" ]; then\n\
|
||||||
|
\x20 reason='This phase has changed nothing: the working tree is clean and \
|
||||||
|
HEAD is still the commit you started from. Do the work the task describes \
|
||||||
|
and leave it in the tree. If the task genuinely requires no code change, \
|
||||||
|
say so explicitly in your final message.'\n\
|
||||||
|
fi\n",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
if let Some(check) = &self.check {
|
||||||
|
s.push_str(&format!(
|
||||||
|
"if [ -z \"$reason\" ]; then\n\
|
||||||
|
\x20 out=$(cd \"$REPO\" && sh -c {} 2>&1); rc=$?\n\
|
||||||
|
\x20 if [ \"$rc\" -ne 0 ]; then\n\
|
||||||
|
\x20 reason=\"This phase's completion check exited $rc, so the work is not \
|
||||||
|
done yet. The check is: {}\n\nIts output:\n$(printf '%s' \"$out\" | tail -c 1500)\"\n\
|
||||||
|
\x20 fi\n\
|
||||||
|
fi\n",
|
||||||
|
q(check),
|
||||||
|
// Inside a double-quoted assignment, so the command text itself
|
||||||
|
// must not carry a `\"` or a `$` that the shell would expand.
|
||||||
|
check.replace('\\', "\\\\").replace('"', "'").replace('$', "\\$"),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
s.push_str(
|
||||||
|
"if [ -z \"$reason\" ]; then echo pass >> \"$GATE/log\"; exit 0; fi\n\
|
||||||
|
if [ \"$N\" -ge \"$MAX\" ]; then\n\
|
||||||
|
\x20 echo \"cap: $reason\" >> \"$GATE/log\"\n\
|
||||||
|
\x20 exit 0\n\
|
||||||
|
fi\n\
|
||||||
|
N=$((N+1)); echo \"$N\" > \"$GATE/blocks\"\n\
|
||||||
|
echo \"block $N: $reason\" >> \"$GATE/log\"\n\
|
||||||
|
printf '%s\\n' \"$reason\" >&2\n\
|
||||||
|
exit 2\n",
|
||||||
|
);
|
||||||
|
s
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The settings file that installs the script as a `Stop` hook.
|
||||||
|
pub fn settings(&self, dir: &str) -> serde_json::Value {
|
||||||
|
json!({
|
||||||
|
"hooks": {
|
||||||
|
"Stop": [{
|
||||||
|
"hooks": [{
|
||||||
|
"type": "command",
|
||||||
|
"command": format!("{dir}/stop-gate.sh"),
|
||||||
|
}]
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One shell command that writes the gate into the guest.
|
||||||
|
///
|
||||||
|
/// Written by `printf` through an exec rather than injected as part of the
|
||||||
|
/// tar: the tar lands in `/mission/repo`, which is exactly where this must
|
||||||
|
/// not be.
|
||||||
|
pub fn install_command(&self, repo: &str, dir: &str) -> String {
|
||||||
|
format!(
|
||||||
|
"mkdir -p {d} && rm -f {d}/blocks {d}/log && printf '%s' {script} > {d}/stop-gate.sh \
|
||||||
|
&& chmod +x {d}/stop-gate.sh && printf '%s' {settings} > {d}/settings.json",
|
||||||
|
d = dir,
|
||||||
|
script = q(&self.script(repo, dir)),
|
||||||
|
settings = q(&self.settings(dir).to_string()),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Single-quote for `sh`. Same rule as `microvm_executor::shell_quote`, kept
|
||||||
|
/// local so this module has no dependency on the executor it is used by.
|
||||||
|
fn q(s: &str) -> String {
|
||||||
|
format!("'{}'", s.replace('\'', r"'\''"))
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use std::path::Path;
|
||||||
|
use std::process::Command;
|
||||||
|
|
||||||
|
fn sh(script: &str, dir: &Path) -> std::process::Output {
|
||||||
|
let path = dir.join("stop-gate.sh");
|
||||||
|
std::fs::write(&path, script).unwrap();
|
||||||
|
Command::new("sh").arg(&path).output().expect("run the gate")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A git repo with one commit and the clone-point marker the real checkout
|
||||||
|
/// carries (`mission_workspace::record_base_commit` writes it).
|
||||||
|
fn repo_with_base(root: &Path) -> std::path::PathBuf {
|
||||||
|
let repo = root.join("repo");
|
||||||
|
std::fs::create_dir_all(&repo).unwrap();
|
||||||
|
let git = |args: &[&str]| {
|
||||||
|
let o = Command::new("git")
|
||||||
|
.arg("-C")
|
||||||
|
.arg(&repo)
|
||||||
|
.args(args)
|
||||||
|
.output()
|
||||||
|
.unwrap();
|
||||||
|
assert!(o.status.success(), "git {args:?}: {:?}", o);
|
||||||
|
};
|
||||||
|
git(&["init", "--quiet"]);
|
||||||
|
git(&["config", "user.email", "t@t"]);
|
||||||
|
git(&["config", "user.name", "T"]);
|
||||||
|
std::fs::write(repo.join("README.md"), "base\n").unwrap();
|
||||||
|
git(&["add", "."]);
|
||||||
|
git(&["commit", "--quiet", "-m", "base"]);
|
||||||
|
let head = Command::new("git")
|
||||||
|
.arg("-C")
|
||||||
|
.arg(&repo)
|
||||||
|
.args(["rev-parse", "HEAD"])
|
||||||
|
.output()
|
||||||
|
.unwrap();
|
||||||
|
std::fs::write(
|
||||||
|
repo.join(".git/clawmates-base"),
|
||||||
|
String::from_utf8_lossy(&head.stdout).trim(),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
repo
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The failure this exists for: an agent that stops having written nothing.
|
||||||
|
/// Post-hoc that costs a whole new VM; here it costs one sentence.
|
||||||
|
#[test]
|
||||||
|
fn an_agent_that_changed_nothing_is_not_allowed_to_stop() {
|
||||||
|
let tmp = tempfile::tempdir().unwrap();
|
||||||
|
let repo = repo_with_base(tmp.path());
|
||||||
|
let gate = StopGate {
|
||||||
|
require_changes: true,
|
||||||
|
check: None,
|
||||||
|
};
|
||||||
|
let script = gate.script(&repo.display().to_string(), &tmp.path().display().to_string());
|
||||||
|
|
||||||
|
let out = sh(&script, tmp.path());
|
||||||
|
assert_eq!(out.status.code(), Some(2), "the stop must be refused");
|
||||||
|
let why = String::from_utf8_lossy(&out.stderr);
|
||||||
|
assert!(why.contains("changed nothing"), "{why}");
|
||||||
|
|
||||||
|
// Uncommitted work counts — the usual case, since the agent is told to
|
||||||
|
// leave its work in the tree rather than commit it.
|
||||||
|
std::fs::write(repo.join("new.rs"), "fn done() {}\n").unwrap();
|
||||||
|
let out = sh(&script, tmp.path());
|
||||||
|
assert_eq!(out.status.code(), Some(0), "{:?}", out);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// And committed work counts too. An agent that committed leaves a CLEAN
|
||||||
|
/// tree, so a gate that only looked at `git status` would refuse the stop of
|
||||||
|
/// a phase that had done everything asked of it.
|
||||||
|
#[test]
|
||||||
|
fn work_the_agent_committed_satisfies_the_gate() {
|
||||||
|
let tmp = tempfile::tempdir().unwrap();
|
||||||
|
let repo = repo_with_base(tmp.path());
|
||||||
|
std::fs::write(repo.join("new.rs"), "fn done() {}\n").unwrap();
|
||||||
|
for args in [vec!["add", "."], vec!["commit", "--quiet", "-m", "work"]] {
|
||||||
|
Command::new("git")
|
||||||
|
.arg("-C")
|
||||||
|
.arg(&repo)
|
||||||
|
.args(&args)
|
||||||
|
.output()
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
let gate = StopGate {
|
||||||
|
require_changes: true,
|
||||||
|
check: None,
|
||||||
|
};
|
||||||
|
let out = sh(
|
||||||
|
&gate.script(&repo.display().to_string(), &tmp.path().display().to_string()),
|
||||||
|
tmp.path(),
|
||||||
|
);
|
||||||
|
assert_eq!(out.status.code(), Some(0), "{:?}", out);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The cap. Without it a stuck agent is blocked, retries, is blocked again,
|
||||||
|
/// and spends the whole hour-long turn budget instead of failing where an
|
||||||
|
/// operator can see it.
|
||||||
|
#[test]
|
||||||
|
fn the_gate_gives_up_after_the_cap_and_says_so() {
|
||||||
|
let tmp = tempfile::tempdir().unwrap();
|
||||||
|
let repo = repo_with_base(tmp.path());
|
||||||
|
let gate = StopGate {
|
||||||
|
require_changes: true,
|
||||||
|
check: None,
|
||||||
|
};
|
||||||
|
let script = gate.script(&repo.display().to_string(), &tmp.path().display().to_string());
|
||||||
|
|
||||||
|
for i in 1..=MAX_BLOCKS {
|
||||||
|
assert_eq!(
|
||||||
|
sh(&script, tmp.path()).status.code(),
|
||||||
|
Some(2),
|
||||||
|
"block {i} of {MAX_BLOCKS}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
sh(&script, tmp.path()).status.code(),
|
||||||
|
Some(0),
|
||||||
|
"past the cap the agent must be allowed to stop"
|
||||||
|
);
|
||||||
|
let log = std::fs::read_to_string(tmp.path().join("log")).unwrap();
|
||||||
|
assert!(log.contains("cap:"), "giving up is recorded: {log}");
|
||||||
|
assert_eq!(
|
||||||
|
std::fs::read_to_string(tmp.path().join("blocks"))
|
||||||
|
.unwrap()
|
||||||
|
.trim(),
|
||||||
|
MAX_BLOCKS.to_string(),
|
||||||
|
"and the count is exact, so the host can report it"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A phase-declared check runs in the repo, and its OUTPUT comes back — a
|
||||||
|
/// gate that said only "the check failed" would send the agent guessing.
|
||||||
|
#[test]
|
||||||
|
fn a_declared_check_must_pass_and_its_output_is_the_feedback() {
|
||||||
|
let tmp = tempfile::tempdir().unwrap();
|
||||||
|
let repo = repo_with_base(tmp.path());
|
||||||
|
let gate = StopGate {
|
||||||
|
require_changes: false,
|
||||||
|
check: Some("test -f wanted.txt || { echo 'wanted.txt is missing'; exit 3; }".into()),
|
||||||
|
};
|
||||||
|
let script = gate.script(&repo.display().to_string(), &tmp.path().display().to_string());
|
||||||
|
|
||||||
|
let out = sh(&script, tmp.path());
|
||||||
|
assert_eq!(out.status.code(), Some(2));
|
||||||
|
let why = String::from_utf8_lossy(&out.stderr);
|
||||||
|
assert!(why.contains("exited 3"), "{why}");
|
||||||
|
assert!(why.contains("wanted.txt is missing"), "{why}");
|
||||||
|
|
||||||
|
std::fs::write(repo.join("wanted.txt"), "here\n").unwrap();
|
||||||
|
assert_eq!(sh(&script, tmp.path()).status.code(), Some(0));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A check with quotes, `$` and apostrophes is ordinary. It travels through
|
||||||
|
/// `sh -c` inside a script that itself travels through `sh -c` to reach the
|
||||||
|
/// guest, and a quoting bug at either layer would run something else.
|
||||||
|
#[test]
|
||||||
|
fn a_check_with_shell_metacharacters_survives_both_layers() {
|
||||||
|
let tmp = tempfile::tempdir().unwrap();
|
||||||
|
let repo = repo_with_base(tmp.path());
|
||||||
|
std::fs::write(repo.join("it's here.txt"), "x\n").unwrap();
|
||||||
|
let gate = StopGate {
|
||||||
|
require_changes: false,
|
||||||
|
check: Some("test -f \"it's here.txt\" && echo $HOME > /dev/null".into()),
|
||||||
|
};
|
||||||
|
let out = sh(
|
||||||
|
&gate.script(&repo.display().to_string(), &tmp.path().display().to_string()),
|
||||||
|
tmp.path(),
|
||||||
|
);
|
||||||
|
assert_eq!(out.status.code(), Some(0), "{:?}", out);
|
||||||
|
// And the install command it is embedded in is still one shell argument.
|
||||||
|
let install = gate.install_command("/mission/repo", GATE_DIR);
|
||||||
|
assert!(install.contains("stop-gate.sh"), "{install}");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Nothing the gate writes may land under the repository: `/mission/repo` is
|
||||||
|
/// collected and diffed, so a file there arrives in the user's patch as if
|
||||||
|
/// an agent had written it.
|
||||||
|
#[test]
|
||||||
|
fn the_gate_never_writes_into_the_delivered_tree() {
|
||||||
|
let gate = StopGate {
|
||||||
|
require_changes: true,
|
||||||
|
check: Some("cargo test".into()),
|
||||||
|
};
|
||||||
|
assert!(GATE_DIR.starts_with("/root/"), "{GATE_DIR}");
|
||||||
|
let install = gate.install_command("/mission/repo", GATE_DIR);
|
||||||
|
for write in ["> /mission/repo", "/mission/repo/stop", "/mission/repo/.claude"] {
|
||||||
|
assert!(!install.contains(write), "{install}");
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
gate.settings(GATE_DIR)["hooks"]["Stop"][0]["hooks"][0]["command"],
|
||||||
|
json!("/root/gate/stop-gate.sh")
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A composed run's nodes must not each be held to "this phase changed
|
||||||
|
/// something". The graph's verifier node changes nothing BY DESIGN, and a
|
||||||
|
/// per-node gate would refuse its stop until the cap — three wasted agent
|
||||||
|
/// turns for doing its job correctly.
|
||||||
|
#[test]
|
||||||
|
fn a_composed_node_is_not_held_to_the_whole_phases_delivery() {
|
||||||
|
let phase = StopGate::for_phase("coding", &json!({})).unwrap();
|
||||||
|
assert!(phase.require_changes);
|
||||||
|
assert!(
|
||||||
|
phase.per_node().is_none(),
|
||||||
|
"with nothing but the delivery rule, a node has no gate at all"
|
||||||
|
);
|
||||||
|
|
||||||
|
let with_check =
|
||||||
|
StopGate::for_phase("coding", &json!({ "done_when_check": "cargo test" })).unwrap();
|
||||||
|
let node = with_check.per_node().expect("the declared check still applies");
|
||||||
|
assert!(!node.require_changes);
|
||||||
|
assert_eq!(node.check.as_deref(), Some("cargo test"));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A gate with nothing to enforce must not be installed at all — a hook that
|
||||||
|
/// can never block still costs a process per stop and a flag on the command
|
||||||
|
/// line.
|
||||||
|
#[test]
|
||||||
|
fn a_phase_with_nothing_to_enforce_gets_no_gate() {
|
||||||
|
let none = json!({});
|
||||||
|
assert!(StopGate::for_phase("research", &none).is_none());
|
||||||
|
assert!(StopGate::for_phase("coding", &json!({ "allow_empty": true })).is_none());
|
||||||
|
|
||||||
|
let coding = StopGate::for_phase("coding", &none).expect("a coding phase must deliver");
|
||||||
|
assert!(coding.require_changes);
|
||||||
|
assert!(coding.check.is_none());
|
||||||
|
|
||||||
|
// A declared check applies to any kind, including one that is allowed to
|
||||||
|
// change nothing — a verification phase's whole job is that check.
|
||||||
|
let verify = StopGate::for_phase(
|
||||||
|
"research",
|
||||||
|
&json!({ "allow_empty": true, "done_when_check": " ./verify.sh " }),
|
||||||
|
)
|
||||||
|
.expect("a declared check is a gate on its own");
|
||||||
|
assert!(!verify.require_changes);
|
||||||
|
assert_eq!(verify.check.as_deref(), Some("./verify.sh"));
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user