feat(missions): Slice 4 — the two engines composed, with the file handoff proven

`team_engine='composed'` (the third name migration 0069 anticipated) runs a
mission as a durable ZeroClaw graph whose every node is a whole
Claude-Code-in-a-microVM session. Engine Z owns checkpoint/resume, cancellation
and per-node heterogeneity; Engine C owns shared context and cheap fan-out;
neither has the other's asset, which is why this is a composition and not a
compromise.

`MicroVmTurnExecutor` implements the existing `TurnExecutor`, so it inherits the
planners, the checkpoint, the stale-run recovery, `close_finished_phases`, the
evaluator, capture and delivery unchanged — the same trick `SubTopologyExecutor`
already plays with a heavy `run_turn`. Producer side emits ONE `queued` row
carrying the real graph and lets the worker claim it: the durability IS being
worker-driven, and the solo path's `tokio::spawn` has none of it. Still exactly
one `topology_runs` row per unit of work and one completion path — `finish()` is
now that one place, shared by every tier.

THE TRAP, solved and proven. A VM is inject → run → collect → destroy, so a
per-node VM with text-only handoff silently loses every file an earlier node
wrote: node 2 boots from the original checkout, sees nothing, and still reports
success. The mission's host checkout is the medium — every node injects from it
and collects back over it — and two properties make that safe rather than lucky:
`execute_resumable` is strictly sequential, so two VMs never write one directory;
and the vm id is deterministic per (phase, iteration, step), so a duplicate is
refused by the node ("vm already exists") instead of becoming a second writer.

NEGATIVE CONTROL, run rather than assumed: with `repo` swapped for a private
per-node workspace, `a_later_node_sees_an_earlier_nodes_files` FAILS with
`saw:[]`; restored, it passes. The `PhaseVm` seam exists for exactly this — it
models inject/collect through the real `mission_fs` tar path in milliseconds.

Two durability traps this tier walks into, both closed:

  - `requeue_stale` fires at 180s on `updated_at`, and one node here can run for
    an hour. `SubTopologyExecutor` keeps its parent alive from each leaf step;
    there is nothing between the start and end of a VM turn, so the turn holds a
    ticker that touches `updated_at` every 30s and aborts on drop. Without it a
    healthy composed run is requeued mid-node and boots a second VM.
  - the 15-minute stuck-run reaper asks "any step records since it was CREATED?",
    which describes a healthy composed run as readily as a wedged one. Hence
    `REAPABLE_TIERS` — worker-driven minus this tier. Reaping it would be #54 in
    a different costume.

`on_launch` mints no team for a microVM mission, deliberately: claws in
containers are what a VM mission does not use. So `mission_orchestrator::
composed_graph` builds the shape from the team template directly — nodes, roles
and pattern, zero claws provisioned. Per-node `attrs["backend"]` and
`attrs["node_id"]` override the mission's, which is what makes a validator node
on another provider's image a first-class graph node; a malformed `node_id`
fails the node rather than quietly running it where the graph did not ask.

Refusals are recorded as a failed run, not returned as an error: `launch_phase`
is swept every ten seconds, so a returned error is a phase that retries forever
while the log repeats itself.

501 tests pass, clippy clean. NOT yet proven end to end: no composed mission has
run on the fleet, so the resume-after-a-killed-worker leg is argued from the DB
test and the step-numbering test, not from a real two-node run.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-06 13:00:44 -07:00
co-authored by Claude Opus 5
parent e31688bac5
commit 12147a1e01
9 changed files with 1141 additions and 17 deletions
+3
View File
@@ -280,6 +280,9 @@ async fn run() -> Result<(), String> {
cm_api::topology_worker::spawn( cm_api::topology_worker::spawn(
pool.clone(), pool.clone(),
runtime.clone(), runtime.clone(),
// The composed tier (`microvm_graph`) runs each graph node as a VM on a
// fleet node, so the worker needs the same hub the phase runner uses.
node_hub.clone(),
std::time::Duration::from_secs(3), std::time::Duration::from_secs(3),
); );
// Boot-time content loaders — skills first, then team templates // Boot-time content loaders — skills first, then team templates
+1
View File
@@ -23,6 +23,7 @@ pub mod library;
pub mod mission_delivery; pub mod mission_delivery;
pub mod microvm_client; pub mod microvm_client;
pub mod microvm_executor; pub mod microvm_executor;
pub mod microvm_turn_executor;
pub mod mission_fs; pub mod mission_fs;
pub mod papers; pub mod papers;
pub mod phase_config; pub mod phase_config;
+91 -11
View File
@@ -57,8 +57,20 @@ const GUEST_REPO: &str = "/mission/repo";
/// A vm id must be `[A-Za-z0-9_-]` — the node rejects anything else, since it /// A vm id must be `[A-Za-z0-9_-]` — the node rejects anything else, since it
/// becomes a path component. Hyphenated uuids qualify; this also keeps the id /// becomes a path component. Hyphenated uuids qualify; this also keeps the id
/// readable in `vm_list` output and in the node's egress log. /// readable in `vm_list` output and in the node's egress log.
fn vm_id_for(phase_id: Uuid, iteration: i32) -> String { ///
format!("m-{}-{}", &phase_id.simple().to_string()[..12], iteration) /// `step` distinguishes the nodes of a composed run, whose graph runs several VMs
/// for one phase and one iteration. Deliberately DETERMINISTIC rather than
/// random: the node refuses to create a vm id that already exists, so if a
/// restarted worker resumes a step whose VM is somehow still alive, the second
/// attempt fails loudly instead of running a duplicate agent against the same
/// checkout. A random id would make that collision invisible and let two VMs
/// collect over each other's work.
fn vm_id_for(phase_id: Uuid, iteration: i32, step: Option<u32>) -> String {
let base = format!("m-{}-{}", &phase_id.simple().to_string()[..12], iteration);
match step {
Some(s) => format!("{base}-s{s}"),
None => base,
}
} }
/// Ceiling on teammates, stated in the prompt. /// Ceiling on teammates, stated in the prompt.
@@ -337,7 +349,7 @@ pub async fn run_phase_in_vm(hub: &NodeHub, p: VmPhase<'_>) -> Result<VmOutcome,
// phase, not boot a VM whose agent will sit there unauthenticated. // phase, not boot a VM whose agent will sit there unauthenticated.
let env = crate::mission_runtime::microvm_provider_env(p.backend)?; let env = crate::mission_runtime::microvm_provider_env(p.backend)?;
let vm = MicroVm::new(hub, p.node_id, vm_id_for(p.phase_id, p.iteration)); let vm = MicroVm::new(hub, p.node_id, vm_id_for(p.phase_id, p.iteration, p.step));
let created = vm.create(VCPUS, MEM_MIB, p.backend).await?; let created = vm.create(VCPUS, MEM_MIB, p.backend).await?;
// From here on every early return must still destroy the VM, so the work is // From here on every early return must still destroy the VM, so the work is
@@ -374,6 +386,41 @@ pub struct VmPhase<'a> {
/// `missions.team_engine` — `Some("claude_code")` asks the lead to form a /// `missions.team_engine` — `Some("claude_code")` asks the lead to form a
/// team. `None` is solo, which is the default. /// team. `None` is solo, which is the default.
pub team_engine: Option<&'a str>, pub team_engine: Option<&'a str>,
/// Which node of a composed graph this VM is running, if any. `None` is the
/// solo path, where the phase is one VM and the id needs no further
/// qualification. Part of the vm id, so the nodes of one phase-iteration
/// cannot collide on a fleet node.
pub step: Option<u32>,
}
/// Runs one phase (or one graph node of one) in a VM.
///
/// A seam with exactly one production implementation, and it exists for one
/// reason: the composed executor's defining property is that node 2 sees node 1's
/// files, and that property is untestable against real Firecracker in a unit
/// test. A fake that models inject → run → collect faithfully can prove it in
/// milliseconds, including the negative control where the handoff is broken.
#[allow(async_fn_in_trait)]
pub trait PhaseVm {
/// Boot a VM, run this phase in it, collect the result, destroy it.
async fn run(&self, p: VmPhase<'_>) -> Result<VmOutcome, String>;
}
/// The real thing: VMs on the fleet, over the node hub.
pub struct HubVms {
hub: std::sync::Arc<NodeHub>,
}
impl HubVms {
pub fn new(hub: std::sync::Arc<NodeHub>) -> Self {
Self { hub }
}
}
impl PhaseVm for HubVms {
async fn run(&self, p: VmPhase<'_>) -> Result<VmOutcome, String> {
run_phase_in_vm(&self.hub, p).await
}
} }
async fn run_inside( async fn run_inside(
@@ -516,13 +563,15 @@ mod tests {
/// outside `[A-Za-z0-9_-]` rather than sanitising it. /// outside `[A-Za-z0-9_-]` rather than sanitising it.
#[test] #[test]
fn a_vm_id_is_acceptable_to_the_node() { fn a_vm_id_is_acceptable_to_the_node() {
let id = vm_id_for(Uuid::now_v7(), 3); for step in [None, Some(0), Some(11)] {
assert!( let id = vm_id_for(Uuid::now_v7(), 3, step);
id.chars() assert!(
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'), id.chars()
"{id}" .all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'),
); "{id}"
assert!(id.len() <= 64, "{id}"); );
assert!(id.len() <= 64, "{id}");
}
} }
/// Two passes of the same phase must not collide: the second would fail with /// Two passes of the same phase must not collide: the second would fail with
@@ -530,7 +579,22 @@ mod tests {
#[test] #[test]
fn each_iteration_gets_its_own_vm() { fn each_iteration_gets_its_own_vm() {
let p = Uuid::now_v7(); let p = Uuid::now_v7();
assert_ne!(vm_id_for(p, 1), vm_id_for(p, 2)); assert_ne!(vm_id_for(p, 1, None), vm_id_for(p, 2, None));
}
/// A composed run boots one VM per graph node within a single phase and
/// iteration. Without the step in the id they would all be the same vm, and
/// the second node would fail with "vm already exists" — or worse, if the
/// first were already destroyed, succeed while looking like a re-run.
#[test]
fn each_graph_node_gets_its_own_vm() {
let p = Uuid::now_v7();
assert_ne!(vm_id_for(p, 1, Some(0)), vm_id_for(p, 1, Some(1)));
// And a composed node never collides with the solo id for the same pass.
assert_ne!(vm_id_for(p, 1, Some(0)), vm_id_for(p, 1, None));
// Deterministic: resuming the same step asks for the same vm, which is
// what makes a still-live duplicate fail loudly on the node.
assert_eq!(vm_id_for(p, 1, Some(2)), vm_id_for(p, 1, Some(2)));
} }
/// A task description contains quotes and newlines as a matter of course, /// A task description contains quotes and newlines as a matter of course,
@@ -683,6 +747,22 @@ mod tests {
for t in ["team", "swarm"] { for t in ["team", "swarm"] {
assert!(cm_db::repo::topology_runs::WORKER_DRIVEN_TIERS.contains(&t), "{t}"); assert!(cm_db::repo::topology_runs::WORKER_DRIVEN_TIERS.contains(&t), "{t}");
} }
// The composed tier is the opposite case and must not be confused with
// the solo one: its durability comes FROM being worker-driven.
assert!(cm_db::repo::topology_runs::WORKER_DRIVEN_TIERS.contains(&"microvm_graph"));
// But the 15-minute stuck-run reaper must not touch it: one of its nodes
// is a whole agent session, so journaling nothing for 15 minutes is what
// a healthy composed run looks like.
assert!(
!cm_db::repo::topology_runs::REAPABLE_TIERS.contains(&"microvm_graph"),
"the reaper would kill a healthy composed run and orphan its VM"
);
for t in cm_db::repo::topology_runs::REAPABLE_TIERS {
assert!(
cm_db::repo::topology_runs::WORKER_DRIVEN_TIERS.contains(t),
"{t} is reapable but not worker-driven"
);
}
} }
/// Solo is the default, and asking for a team must be explicit. Multi-agent /// Solo is the default, and asking for a team must be explicit. Multi-agent
+579
View File
@@ -0,0 +1,579 @@
//! The two engines composed — Slice 4.
//!
//! Engine Z (the ZeroClaw graph in `cm_orchestrator`) owns durability and
//! heterogeneity: deterministic planners, per-step checkpoint/resume, a stale-run
//! sweep, cancellation, and a different model per node. Engine C (Claude Code in
//! a microVM) owns shared context, self-sizing and cheap fan-out. Neither has the
//! other's asset, which is why keeping both is a composition rather than a
//! compromise.
//!
//! This module is the join: a [`TurnExecutor`] whose "turn" is a whole
//! Claude-Code-in-a-VM session. Because `topology_worker` already dispatches by
//! tier, implementing the existing trait inherits the planners, checkpointing,
//! reaper, cancellation, `close_finished_phases`, evaluation, capture and
//! delivery unchanged. `recursive_exec::SubTopologyExecutor` is the precedent: a
//! `run_turn` may be arbitrarily heavy.
//!
//! # The file-handoff trap
//!
//! A VM is inject-tar → run → collect-tar → destroy. A graph of per-node VMs with
//! **text-only** handoff would silently lose every file an earlier node wrote:
//! node 2 would boot from the original checkout, see none of node 1's work, and
//! still report success — the exact silent-success shape this project keeps
//! paying for.
//!
//! The answer here is that the mission's **host checkout is the medium**. Every
//! node injects from `repo` and collects back over `repo`, so the tree carries
//! forward node to node and the last node's tree is what delivery diffs. Two
//! properties make that safe rather than lucky:
//!
//! - `execute_resumable` runs steps strictly **sequentially**, so two VMs are
//! never writing the same host directory at once;
//! - the vm id is deterministic per (phase, iteration, step), so a resumed step
//! whose VM is somehow still alive is refused by the node ("vm already exists")
//! instead of quietly producing a second writer.
//!
//! `a_later_node_sees_an_earlier_nodes_files` proves the handoff, and
//! `text_only_handoff_loses_the_earlier_nodes_work` is its negative control.
//!
//! # Keeping a long turn alive
//!
//! `requeue_stale` requeues a `running` job that has not touched `updated_at` in
//! 180 seconds, and one node here can run for an hour. `SubTopologyExecutor`
//! keeps its parent alive from each *leaf step*, which it has and this does not:
//! there is nothing between the start and end of a VM turn. So the turn holds a
//! ticker that touches `updated_at` every [`KEEPALIVE_SECS`] and is aborted on
//! drop. Without it a healthy composed run is requeued mid-node, claimed again,
//! and boots a second VM against the same checkout.
use std::path::PathBuf;
use std::sync::Arc;
use std::time::Duration;
use cm_domain::NodeId;
use cm_orchestrator::{OrchestratorError, TurnExecutor, TurnOutcome, TurnRequest};
use sqlx::PgPool;
use uuid::Uuid;
use crate::microvm_executor::{PhaseVm, VmPhase};
/// How often a running VM turn touches its run's `updated_at`.
///
/// Comfortably inside the 180s stale window, and cheap: one UPDATE per node per
/// half minute against a row nothing else is writing.
const KEEPALIVE_SECS: u64 = 30;
/// A [`TurnExecutor`] that runs each graph node as a full Claude-Code session
/// inside its own microVM, against the mission's shared host checkout.
pub struct MicroVmTurnExecutor<V: PhaseVm> {
vms: V,
pool: PgPool,
/// The durable outer run. Touched for keepalive; its status gates the turn.
run_id: Uuid,
mission_id: Uuid,
phase_id: Uuid,
iteration: i32,
/// The mission's host checkout — injected into every node's VM and collected
/// back over, which is how file work survives a node boundary.
repo: PathBuf,
/// `missions.target_node_id`: the fleet node a mission was placed on. A node
/// may override it with `attrs["node_id"]`.
default_fleet_node: Option<Uuid>,
/// `missions.backend`: which rootfs image. A node may override it with
/// `attrs["backend"]`, which is what makes a graph heterogeneous — a
/// `validator` node on a different provider's image is then a first-class
/// graph node rather than a bolt-on.
default_backend: Option<String>,
/// `missions.team_engine`, passed through so a composed node can itself ask
/// for Claude Code fan-out inside its VM.
team_engine: Option<String>,
/// Which step is next. `execute_resumable` is sequential and gives the
/// executor no index, so the executor counts — and the count starts from the
/// checkpoint on resume, or two VMs would share an id across a restart.
step: std::sync::atomic::AtomicU32,
}
/// Everything a composed run needs that is not the graph itself.
pub struct ComposedRun {
pub run_id: Uuid,
pub mission_id: Uuid,
pub phase_id: Uuid,
pub iteration: i32,
pub repo: PathBuf,
pub target_node_id: Option<Uuid>,
pub backend: Option<String>,
pub team_engine: Option<String>,
/// Steps already completed, from the durable checkpoint. Nonzero on resume.
pub completed_steps: u32,
}
impl<V: PhaseVm> MicroVmTurnExecutor<V> {
pub fn new(vms: V, pool: PgPool, r: ComposedRun) -> Self {
Self {
vms,
pool,
run_id: r.run_id,
mission_id: r.mission_id,
phase_id: r.phase_id,
iteration: r.iteration,
repo: r.repo,
default_fleet_node: r.target_node_id,
default_backend: r.backend,
team_engine: r.team_engine,
step: std::sync::atomic::AtomicU32::new(r.completed_steps),
}
}
/// Which fleet node this graph node runs on.
///
/// Fail-closed on a malformed override: placing a node on the mission's node
/// because its own `node_id` did not parse would run the work somewhere the
/// graph did not ask for and say nothing.
fn fleet_node(&self, req: &TurnRequest) -> Result<NodeId, OrchestratorError> {
let id = match req.attrs.get("node_id") {
Some(raw) => Uuid::parse_str(raw.trim()).map_err(|_| {
OrchestratorError::Executor(format!(
"node {} has an invalid node_id attr: {raw}",
req.node_id
))
})?,
None => self.default_fleet_node.ok_or_else(|| {
OrchestratorError::Executor(format!(
"node {} has no node_id attr and the mission has no \
target_node_id — a microVM node cannot run on the gateway, \
which has no /dev/kvm",
req.node_id
))
})?,
};
Ok(NodeId::from(id))
}
}
impl<V: PhaseVm> TurnExecutor for MicroVmTurnExecutor<V> {
async fn run_turn(&self, req: TurnRequest) -> Result<TurnOutcome, OrchestratorError> {
let fleet_node = self.fleet_node(&req)?;
let backend = req
.attrs
.get("backend")
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.or_else(|| self.default_backend.clone());
if !self.repo.is_dir() {
return Err(OrchestratorError::Executor(format!(
"mission has no checkout at {} — a composed node needs the \
repository, and it is also how the previous node's work reaches \
this one",
self.repo.display()
)));
}
let step = self
.step
.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
// Held for the length of the VM turn: an hour of silence would otherwise
// look exactly like a dead worker to `requeue_stale`.
let _alive = Keepalive::spawn(self.pool.clone(), self.run_id);
let task = node_task_text(&req);
let outcome = self
.vms
.run(VmPhase {
node_id: fleet_node,
mission_id: self.mission_id,
phase_id: self.phase_id,
iteration: self.iteration,
task: &task,
backend: backend.as_deref(),
repo: &self.repo,
team_engine: self.team_engine.as_deref(),
step: Some(step),
})
.await
.map_err(|e| {
OrchestratorError::Executor(format!("node {} in a microVM: {e}", req.node_id))
})?;
// A node whose work never came back must fail the run rather than hand
// the next node a tree missing the previous one's edits. On this path an
// uncollected turn is worse than on the solo one: the loss is silent,
// because the next node still boots from a checkout that looks fine.
if !outcome.collected {
return Err(OrchestratorError::Executor(format!(
"node {}'s work could not be collected from its VM, so the next \
node would not see it: {}",
req.node_id,
outcome.summary.chars().take(400).collect::<String>()
)));
}
if outcome.rc != 0 {
return Err(OrchestratorError::Executor(format!(
"node {} exited {}: {}",
req.node_id,
outcome.rc,
outcome.summary.chars().take(400).collect::<String>()
)));
}
eprintln!(
"microvm_turn_executor: run {} node {} (role {}, step {}) ok — subagents: {}",
self.run_id,
req.node_id,
req.role,
step,
outcome
.subagents
.map(|n| n.to_string())
.unwrap_or_else(|| "?".into()),
);
Ok(TurnOutcome {
output: outcome.summary,
// `claude -p` does not report token usage on stdout, and inventing a
// number here would corrupt the run totals the harness reads. Zero is
// the honest value for "not measured on this path".
tokens: 0,
gated: Vec::new(),
})
}
}
/// What one graph node is told.
///
/// The upstream outputs are included as context, but the load-bearing sentence is
/// that the previous node's *files* are already in the tree: a node told only
/// about the text would re-do work it is standing on.
fn node_task_text(req: &TurnRequest) -> String {
let mut s = format!(
"You are the `{}` stage of a multi-stage mission.\n\nMISSION TASK\n{}\n",
req.role, req.task
);
if !req.context.is_empty() {
s.push_str(
"\nWHAT CAME BEFORE\nThe earlier stages' work is ALREADY IN THIS \
WORKING TREE — the repository you have been given is their output, \
not a fresh checkout. Read the files before changing them, and do \
not redo what is already done. Their closing reports:\n",
);
for (i, c) in req.context.iter().enumerate() {
s.push_str(&format!("\n--- stage {} ---\n{}\n", i + 1, c));
}
}
s
}
/// Touches a run's `updated_at` until dropped.
struct Keepalive(tokio::task::JoinHandle<()>);
impl Keepalive {
fn spawn(pool: PgPool, run_id: Uuid) -> Self {
Keepalive(tokio::spawn(async move {
let mut ticker = tokio::time::interval(Duration::from_secs(KEEPALIVE_SECS));
loop {
ticker.tick().await;
let _ = cm_db::repo::topology_runs::touch(&pool, run_id).await;
}
}))
}
}
impl Drop for Keepalive {
fn drop(&mut self) {
self.0.abort();
}
}
/// Build the executor the worker uses, over real VMs on the fleet.
pub fn for_fleet(
hub: Arc<crate::fleet::NodeHub>,
pool: PgPool,
r: ComposedRun,
) -> MicroVmTurnExecutor<crate::microvm_executor::HubVms> {
MicroVmTurnExecutor::new(crate::microvm_executor::HubVms::new(hub), pool, r)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::microvm_executor::VmOutcome;
use std::collections::BTreeMap;
use std::sync::Mutex;
/// A VM modelled honestly: the host tree is packed in, the "agent" works on a
/// COPY that no host path points at, and the result is unpacked back over the
/// host tree. That is the real inject → run → collect shape, which is what
/// makes the negative control below meaningful — remove the collect and the
/// handoff breaks exactly as it would in production.
struct FakeVms {
/// Whether the guest's tree is collected back to the host.
collect: bool,
/// vm ids used, in order — the id is what stops two nodes colliding.
ids: Mutex<Vec<String>>,
/// (backend, fleet node) per call, for the heterogeneity assertions.
placements: Mutex<Vec<(Option<String>, NodeId)>>,
}
impl FakeVms {
fn new(collect: bool) -> Self {
Self {
collect,
ids: Mutex::new(Vec::new()),
placements: Mutex::new(Vec::new()),
}
}
}
impl PhaseVm for FakeVms {
async fn run(&self, p: VmPhase<'_>) -> Result<VmOutcome, String> {
self.ids
.lock()
.unwrap()
.push(format!("{}-{:?}", p.phase_id.simple(), p.step));
self.placements
.lock()
.unwrap()
.push((p.backend.map(str::to_string), p.node_id));
// inject: the host checkout goes in as a tar.
let tar = crate::mission_fs::pack_dir(p.repo, "repo")?;
let guest = tempfile::tempdir().map_err(|e| e.to_string())?;
crate::mission_fs::unpack_into(&tar, guest.path())?;
let guest_repo = guest.path().join("repo");
// run: the agent records that it was here, and reports what it found
// of the previous stages — the observation the handoff test reads.
let seen: Vec<String> = std::fs::read_dir(&guest_repo)
.map_err(|e| e.to_string())?
.filter_map(|e| e.ok())
.map(|e| e.file_name().to_string_lossy().to_string())
.filter(|n| n.starts_with("stage-"))
.collect();
let mine = guest_repo.join(format!("stage-{}.txt", p.step.unwrap_or(0)));
std::fs::write(&mine, "work").map_err(|e| e.to_string())?;
// collect: the guest tree comes back over the same host path.
if self.collect {
let back = crate::mission_fs::pack_dir(&guest_repo, "repo")?;
let parent = p.repo.parent().ok_or("no parent")?;
crate::mission_fs::unpack_into(&back, parent)?;
}
Ok(VmOutcome {
summary: format!("saw:[{}]", seen.join(",")),
rc: 0,
collected: true,
subagents: Some(0),
teammates: None,
})
}
}
fn req(node: &str, role: &str, context: Vec<String>) -> TurnRequest {
TurnRequest {
node_id: node.into(),
role: role.into(),
agent: None,
attrs: BTreeMap::new(),
task: "build the thing".into(),
context,
}
}
fn exec<V: PhaseVm>(vms: V, repo: PathBuf) -> MicroVmTurnExecutor<V> {
// A pool that is never connected: every test here fails the turn before
// any query, or drives one whose only DB touch is the best-effort
// keepalive (which swallows its own errors by design).
let pool = sqlx::postgres::PgPoolOptions::new()
.max_connections(1)
.connect_lazy("postgres://invalid/invalid")
.expect("a lazy pool never dials");
MicroVmTurnExecutor::new(
vms,
pool,
ComposedRun {
run_id: Uuid::now_v7(),
mission_id: Uuid::now_v7(),
phase_id: Uuid::now_v7(),
iteration: 1,
repo,
target_node_id: Some(Uuid::now_v7()),
backend: Some("claude".into()),
team_engine: None,
completed_steps: 0,
},
)
}
fn a_checkout() -> tempfile::TempDir {
let d = tempfile::tempdir().unwrap();
std::fs::create_dir_all(d.path().join("repo")).unwrap();
std::fs::write(d.path().join("repo").join("README.md"), "hello").unwrap();
d
}
/// THE trap this slice exists to solve. A per-node VM is destroyed with its
/// filesystem, so unless the tree is carried forward, node 2 works from the
/// original checkout and silently loses node 1's edits — while still
/// reporting success.
#[tokio::test]
async fn a_later_node_sees_an_earlier_nodes_files() {
let d = a_checkout();
let e = exec(FakeVms::new(true), d.path().join("repo"));
let first = e.run_turn(req("n1", "implementer", vec![])).await.unwrap();
assert_eq!(first.output, "saw:[]", "the first node starts clean");
let second = e
.run_turn(req("n2", "verifier", vec![first.output.clone()]))
.await
.unwrap();
assert!(
second.output.contains("stage-0.txt"),
"node 2 could not see node 1's file: {}",
second.output
);
// And the host tree — what delivery diffs — holds both nodes' work.
for f in ["stage-0.txt", "stage-1.txt"] {
assert!(d.path().join("repo").join(f).exists(), "{f} missing on the host");
}
}
/// The negative control, run rather than assumed: with the collect removed —
/// i.e. a text-only handoff between nodes — the test above fails. A guard
/// that cannot detect the bug it was written for is decoration.
#[tokio::test]
async fn text_only_handoff_loses_the_earlier_nodes_work() {
let d = a_checkout();
let e = exec(FakeVms::new(false), d.path().join("repo"));
e.run_turn(req("n1", "implementer", vec![])).await.unwrap();
let second = e.run_turn(req("n2", "verifier", vec![])).await.unwrap();
assert_eq!(
second.output, "saw:[]",
"without a collect, node 2 must NOT see node 1's work — if it does, \
this test is no longer controlling anything"
);
assert!(!d.path().join("repo").join("stage-0.txt").exists());
}
/// Each node gets its own vm id within one phase and iteration. Two nodes
/// sharing an id means the second is refused by the fleet node while the
/// first is alive, and indistinguishable from a re-run once it is not.
#[tokio::test]
async fn every_node_runs_in_its_own_vm() {
let d = a_checkout();
let vms = FakeVms::new(true);
let e = exec(vms, d.path().join("repo"));
for n in ["n1", "n2", "n3"] {
e.run_turn(req(n, "worker", vec![])).await.unwrap();
}
let ids = e.vms.ids.lock().unwrap().clone();
let unique: std::collections::HashSet<_> = ids.iter().collect();
assert_eq!(unique.len(), ids.len(), "{ids:?}");
}
/// Resume must not re-use a completed step's vm id. The executor counts steps
/// itself, so the count has to start where the checkpoint left off.
#[tokio::test]
async fn a_resumed_run_continues_the_step_numbering() {
let d = a_checkout();
let pool = sqlx::postgres::PgPoolOptions::new()
.max_connections(1)
.connect_lazy("postgres://invalid/invalid")
.unwrap();
let e = MicroVmTurnExecutor::new(
FakeVms::new(true),
pool,
ComposedRun {
run_id: Uuid::now_v7(),
mission_id: Uuid::now_v7(),
phase_id: Uuid::now_v7(),
iteration: 1,
repo: d.path().join("repo"),
target_node_id: Some(Uuid::now_v7()),
backend: None,
team_engine: None,
completed_steps: 2,
},
);
e.run_turn(req("n3", "worker", vec![])).await.unwrap();
let ids = e.vms.ids.lock().unwrap().clone();
assert!(
ids[0].ends_with("Some(2)"),
"the first step after a resume must be step 2, not 0: {ids:?}"
);
}
/// Per-node `backend` is what makes the outer graph heterogeneous — a
/// validator node on another provider's image. It must override the
/// mission's, and the mission's must still apply to nodes that say nothing.
#[tokio::test]
async fn a_node_may_pick_its_own_backend_and_fleet_node() {
let d = a_checkout();
let e = exec(FakeVms::new(true), d.path().join("repo"));
let elsewhere = Uuid::now_v7();
let mut r = req("n1", "worker", vec![]);
r.attrs.insert("backend".into(), "glm".into());
r.attrs.insert("node_id".into(), elsewhere.to_string());
e.run_turn(r).await.unwrap();
e.run_turn(req("n2", "worker", vec![])).await.unwrap();
let p = e.vms.placements.lock().unwrap().clone();
assert_eq!(p[0].0.as_deref(), Some("glm"));
assert_eq!(p[0].1, NodeId::from(elsewhere));
assert_eq!(p[1].0.as_deref(), Some("claude"), "the mission default");
assert_ne!(p[1].1, NodeId::from(elsewhere));
}
/// A malformed `node_id` must fail the node, not fall back to the mission's.
/// Silently running work somewhere the graph did not ask for is the same
/// class of bug as an alias that serde dropped.
#[tokio::test]
async fn a_malformed_node_placement_fails_closed() {
let d = a_checkout();
let e = exec(FakeVms::new(true), d.path().join("repo"));
let mut r = req("n1", "worker", vec![]);
r.attrs.insert("node_id".into(), "not-a-uuid".into());
let err = e.run_turn(r).await.unwrap_err().to_string();
assert!(err.contains("invalid node_id"), "{err}");
}
/// An uncollected node is a failed run here, not a warning: the next node
/// would boot from a tree that looks fine and is missing this node's work.
#[tokio::test]
async fn an_uncollected_node_fails_the_run() {
struct Lost;
impl PhaseVm for Lost {
async fn run(&self, _p: VmPhase<'_>) -> Result<VmOutcome, String> {
Ok(VmOutcome {
summary: "did plenty".into(),
rc: 0,
collected: false,
subagents: None,
teammates: None,
})
}
}
let d = a_checkout();
let e = exec(Lost, d.path().join("repo"));
let err = e.run_turn(req("n1", "worker", vec![])).await.unwrap_err().to_string();
assert!(err.contains("could not be collected"), "{err}");
}
/// A node must be told its predecessors' files are already in the tree.
/// Given only the text, an agent re-does work it is standing on.
#[test]
fn a_downstream_node_is_told_the_work_is_already_in_the_tree() {
let solo = node_task_text(&req("n1", "implementer", vec![]));
assert!(solo.contains("build the thing"));
assert!(!solo.contains("WHAT CAME BEFORE"), "{solo}");
let later = node_task_text(&req("n2", "verifier", vec!["I wrote foo.rs".into()]));
assert!(later.contains("ALREADY IN THIS WORKING TREE"), "{later}");
assert!(later.contains("I wrote foo.rs"), "{later}");
assert!(later.contains("verifier"), "the node's role: {later}");
}
}
+67
View File
@@ -574,6 +574,73 @@ async fn mint_team_from_template(
Ok(team_id) Ok(team_id)
} }
/// The graph a COMPOSED microVM mission runs, built from its team template
/// without minting a single claw.
///
/// A composed mission needs the template's *shape* — how many nodes, in what
/// pattern, playing what roles — and nothing else it carries. Its nodes are VMs,
/// so provisioning claws for them would create agents, containers and `.brain`
/// files that nothing ever dials; that is exactly why `on_launch` returns early
/// for a microVM mission, and this is how the composed path gets its graph
/// anyway rather than by undoing that.
///
/// `purposes` is the phase's purpose list, matched against `config.phase_teams`;
/// missions using the legacy single `team_template_id` fall back to it.
/// Returns `None` when the mission picked no template at all.
pub async fn composed_graph(
pool: &PgPool,
mission_id: Uuid,
purposes: &[&str],
) -> Result<Option<serde_json::Value>, String> {
let row: Option<(serde_json::Value, Option<Uuid>)> =
sqlx::query_as("SELECT config, team_template_id FROM missions WHERE id = $1")
.bind(mission_id)
.fetch_optional(pool)
.await
.map_err(|e| format!("load mission {mission_id}: {e}"))?;
let Some((config, legacy_template)) = row else {
return Err(format!("mission {mission_id} not found"));
};
let template_id = config
.get("phase_teams")
.and_then(|v| v.as_object())
.and_then(|pt| {
// First template named by any purpose this phase answers to, in the
// phase's own preference order — the same order `launch_phase` uses
// to pick teams, so a composed mission and a ZeroClaw one resolve the
// same template for the same phase.
purposes.iter().find_map(|p| {
pt.get(*p)
.and_then(|v| v.as_array())
.and_then(|a| a.first())
.and_then(|v| v.as_str())
.and_then(|s| Uuid::parse_str(s).ok())
})
})
.or(legacy_template);
let Some(template_id) = template_id else {
return Ok(None);
};
let template = cm_db::repo::team_templates::get(pool, template_id)
.await
.map_err(|e| format!("load template {template_id}: {e}"))?
.ok_or_else(|| format!("template {template_id} not found"))?;
let roles: Vec<&str> = template.roles.iter().map(|r| r.slot.as_str()).collect();
if roles.is_empty() {
return Err(format!("template {template_id} defines no roles"));
}
let graph = cm_topology::build(
parse_topology_kind(&template.template.default_topology),
&roles,
)
.map_err(|e| format!("build topology graph for template {template_id}: {e}"))?;
serde_json::to_value(&graph)
.map(Some)
.map_err(|e| format!("serialize topology graph: {e}"))
}
fn parse_topology_kind(s: &str) -> cm_topology::TopologyKind { fn parse_topology_kind(s: &str) -> cm_topology::TopologyKind {
use cm_topology::TopologyKind; use cm_topology::TopologyKind;
match s { match s {
+215
View File
@@ -524,6 +524,32 @@ async fn launch_phase(
// this mission specifically, and an env var that happens to be set must not // this mission specifically, and an env var that happens to be set must not
// silently run it somewhere else. // silently run it somewhere else.
if p.runtime_kind == "microvm" { if p.runtime_kind == "microvm" {
// A composed mission's graph comes from the same `mission_teams` row a
// ZeroClaw mission would use — only its nodes run as VMs instead of
// claws. That is the whole point of composing the engines: the graph,
// the planners and the durability are Engine Z's, unchanged.
if wants_composed(p.team_engine) {
let team = team_rows
.iter()
.map(|r| {
(
r.get::<Uuid, _>("team_id"),
r.get::<serde_json::Value, _>("graph"),
)
})
.collect::<Vec<_>>();
return launch_composed_microvm_phase(
pool,
mission_id,
phase_id,
workspace_id,
iteration,
&task,
team,
purposes,
)
.await;
}
return launch_microvm_phase( return launch_microvm_phase(
pool, pool,
hub, hub,
@@ -611,6 +637,162 @@ async fn launch_phase(
/// Returns as soon as the session is spawned: `launch_phase` runs inside the /// Returns as soon as the session is spawned: `launch_phase` runs inside the
/// sweep loop, and blocking it for the length of a coding session would stall /// sweep loop, and blocking it for the length of a coding session would stall
/// every other mission. /// every other mission.
/// Does this mission want the composed engines — a durable ZeroClaw graph whose
/// every node is a Claude-Code-in-a-microVM session?
///
/// The third `team_engine` name migration 0069 anticipated. Exact match, like
/// `wants_claude_code_team`: a typo must run solo rather than half-compose.
fn wants_composed(team_engine: Option<&str>) -> bool {
matches!(team_engine, Some("composed"))
}
/// Enqueue a composed run — Slice 4's producer side.
///
/// Unlike every other executor here this one does **not** run the work: it emits
/// ONE `queued` row carrying the real graph and lets `topology_worker` claim it.
/// That is where the durability comes from — the checkpoint, the resume after a
/// crash, and the cancellation all belong to the worker, and a run that drove
/// itself from a `tokio::spawn` (as the solo microVM path does) would have none
/// of them.
///
/// One row, not one per team, and the constraint is physical: every node injects
/// from and collects back over the SAME host checkout, so two concurrent runs of
/// one phase would be two VMs writing one directory. A mission with two matching
/// teams is refused, visibly, rather than silently running only the first.
#[allow(clippy::too_many_arguments)]
async fn launch_composed_microvm_phase(
pool: &PgPool,
mission_id: Uuid,
phase_id: Uuid,
workspace_id: Uuid,
iteration: i32,
task: &str,
teams: Vec<(Uuid, serde_json::Value)>,
purposes: &[&str],
) -> Result<(), String> {
sqlx::query(
"DELETE FROM topology_runs
WHERE mission_phase_id = $1 AND status IN ('failed', 'cancelled')",
)
.bind(phase_id)
.execute(pool)
.await
.map_err(|e| format!("purge prior runs for phase {phase_id}: {e}"))?;
// Refusals are recorded as a failed run rather than returned as an error:
// `launch_phase` is called from the sweep every ten seconds, so a returned
// error is a phase that retries forever while the log repeats itself. A
// failed row closes the phase and says why on the card.
let refuse = |why: String| async move {
eprintln!("phase_runner: composed phase {phase_id} of mission {mission_id} refused: {why}");
let _ = sqlx::query(
"INSERT INTO topology_runs
(id, workspace_id, task, kind, status, graph, tier,
mission_id, mission_phase_id, iteration, error)
VALUES ($1, $2, $3, 'run', 'failed', $4, 'microvm_graph', $5, $6, $7, $8)",
)
.bind(Uuid::now_v7())
.bind(workspace_id)
.bind(task)
.bind(serde_json::json!({ "nodes": [], "edges": [] }))
.bind(mission_id)
.bind(phase_id)
.bind(iteration)
.bind(&why)
.execute(pool)
.await;
// The phase must still leave `pending`, or the sweep re-launches it.
let _ = sqlx::query(
"UPDATE mission_phases SET status = 'running', started_at = now()
WHERE id = $1 AND status = 'pending'",
)
.bind(phase_id)
.execute(pool)
.await;
Ok(())
};
let (team_id, graph) = match teams.len() {
1 => {
let (id, g) = teams.into_iter().next().expect("len == 1");
(Some(id), g)
}
// The usual case, and not an error: `mission_orchestrator::on_launch`
// deliberately mints NO team for a microVM mission, because claws in
// containers are exactly what a VM mission does not use. A composed run
// needs the template's shape, not its claws, so it builds the graph from
// the template directly.
0 => match crate::mission_orchestrator::composed_graph(pool, mission_id, purposes).await {
Ok(Some(g)) => (None, g),
Ok(None) => {
return refuse(
"team_engine='composed' needs a team template to give the run its \
shape, and this mission picked none — a composed mission is a \
ZeroClaw graph whose nodes happen to be VMs, so without the graph \
there is nothing to compose"
.to_string(),
)
.await
}
Err(e) => return refuse(format!("could not build the composed graph: {e}")).await,
},
n => {
return refuse(format!(
"{n} teams match this phase, and a composed run must be exactly one: \
every node injects from and collects back over the same host \
checkout, so two runs would be two VMs writing one directory"
))
.await
}
};
// Parse here rather than letting the worker discover it: a graph the
// orchestrator cannot plan would otherwise be claimed, fail with "missing or
// invalid graph", and look like a runtime fault instead of a bad team.
if let Err(e) = serde_json::from_value::<cm_topology::TopologyGraph>(graph.clone()) {
return refuse(format!(
"the graph for this composed phase (team {team_id:?}) is not a runnable \
topology: {e}"
))
.await;
}
let run_id = Uuid::now_v7();
sqlx::query(
"INSERT INTO topology_runs
(id, workspace_id, task, kind, status, graph, tier,
team_id, mission_id, mission_phase_id, iteration)
VALUES ($1, $2, $3, 'run', 'queued', $4, 'microvm_graph', $5, $6, $7, $8)",
)
.bind(run_id)
.bind(workspace_id)
.bind(task)
.bind(&graph)
.bind(team_id)
.bind(mission_id)
.bind(phase_id)
.bind(iteration)
.execute(pool)
.await
.map_err(|e| format!("enqueue composed run for phase {phase_id}: {e}"))?;
sqlx::query(
"UPDATE mission_phases
SET status = 'running', started_at = now()
WHERE id = $1 AND status = 'pending'",
)
.bind(phase_id)
.execute(pool)
.await
.map_err(|e| format!("mark phase {phase_id} running: {e}"))?;
eprintln!(
"phase_runner: mission {mission_id} phase {phase_id} queued as a COMPOSED run \
{run_id} (team {team_id:?}) — a ZeroClaw graph with microVM nodes"
);
Ok(())
}
/// Launch a phase inside a Firecracker microVM on the mission's placed node. /// Launch a phase inside a Firecracker microVM on the mission's placed node.
/// ///
/// Mirrors [`launch_direct_session`] on purpose, down to creating exactly ONE /// Mirrors [`launch_direct_session`] on purpose, down to creating exactly ONE
@@ -707,6 +889,9 @@ async fn launch_microvm_phase(
backend: backend.as_deref(), backend: backend.as_deref(),
repo: &repo, repo: &repo,
team_engine: team_engine.as_deref(), team_engine: team_engine.as_deref(),
// The solo path is one VM for the whole phase; only a
// composed run needs the id qualified per graph node.
step: None,
}, },
) )
.await .await
@@ -1247,6 +1432,36 @@ mod tests {
pairs.iter().map(|(k, v)| (k.to_string(), *v)).collect() pairs.iter().map(|(k, v)| (k.to_string(), *v)).collect()
} }
/// Composing the engines must be asked for by name. A mission that says
/// nothing — or says something adjacent — runs the solo VM path it ran
/// before, one VM for the whole phase.
#[test]
fn only_an_explicit_request_composes_the_engines() {
assert!(wants_composed(Some("composed")));
for engine in [
None,
Some(""),
Some("claude_code"),
Some("zeroclaw"),
Some("Composed"),
Some("compose"),
] {
assert!(!wants_composed(engine), "{engine:?} must not compose");
}
}
/// The composed tier the producer inserts and the tier the worker dispatches
/// on are the same string, and nothing but this test connects them. A typo
/// would enqueue a row no worker ever claims: the phase would sit `running`
/// with a `queued` run behind it and no error anywhere.
#[test]
fn the_composed_tier_is_one_the_worker_actually_drives() {
assert!(
cm_db::repo::topology_runs::WORKER_DRIVEN_TIERS.contains(&"microvm_graph"),
"the producer would enqueue a run nobody claims"
);
}
/// The regression from mission `019fcf62`: a coding phase whose agents were /// The regression from mission `019fcf62`: a coding phase whose agents were
/// silently unpinned from the repo wrote nothing and reported `completed`, /// silently unpinned from the repo wrote nothing and reported `completed`,
/// the same status a fully delivered phase gets. /// the same status a fully delivered phase gets.
+84 -4
View File
@@ -33,7 +33,12 @@ const REAP_STUCK_AFTER_SECS: i64 = 15 * 60;
/// Spawn the durable topology job worker. Polls for queued jobs every `poll` /// Spawn the durable topology job worker. Polls for queued jobs every `poll`
/// interval; runs each to completion (or failure), checkpointing per step. /// interval; runs each to completion (or failure), checkpointing per step.
pub fn spawn(pool: PgPool, runtime: cm_runtime::Runtime, poll: Duration) { pub fn spawn(
pool: PgPool,
runtime: cm_runtime::Runtime,
hub: Arc<crate::fleet::NodeHub>,
poll: Duration,
) {
// Fire the stuck-container reaper on its own cadence — checking // Fire the stuck-container reaper on its own cadence — checking
// once a minute is plenty and keeps this off the hot claim loop. // once a minute is plenty and keeps this off the hot claim loop.
let reaper_pool = pool.clone(); let reaper_pool = pool.clone();
@@ -55,7 +60,7 @@ pub fn spawn(pool: PgPool, runtime: cm_runtime::Runtime, poll: Duration) {
eprintln!("topology_worker: requeue_stale failed: {e}"); eprintln!("topology_worker: requeue_stale failed: {e}");
} }
match cm_db::repo::topology_runs::claim_next_queued(&pool).await { match cm_db::repo::topology_runs::claim_next_queued(&pool).await {
Ok(Some(job)) => run_job(&pool, &runtime, job).await, Ok(Some(job)) => run_job(&pool, &runtime, &hub, job).await,
Ok(None) => tokio::time::sleep(poll).await, Ok(None) => tokio::time::sleep(poll).await,
Err(e) => { Err(e) => {
eprintln!("topology_worker: claim failed: {e}"); eprintln!("topology_worker: claim failed: {e}");
@@ -90,7 +95,11 @@ async fn reap_stuck_runs(pool: &PgPool) -> Result<(), sqlx::Error> {
) )
.bind(REAP_STUCK_AFTER_SECS as f64) .bind(REAP_STUCK_AFTER_SECS as f64)
.bind( .bind(
cm_db::repo::topology_runs::WORKER_DRIVEN_TIERS // REAPABLE, not worker-driven: `microvm_graph` is driven by this worker
// and must NOT be reaped — one of its steps is a whole agent session in a
// VM, so "no step records in 15 minutes" describes a healthy composed run
// as readily as a wedged one.
cm_db::repo::topology_runs::REAPABLE_TIERS
.iter() .iter()
.map(|s| (*s).to_string()) .map(|s| (*s).to_string())
.collect::<Vec<_>>(), .collect::<Vec<_>>(),
@@ -119,6 +128,7 @@ async fn reap_stuck_runs(pool: &PgPool) -> Result<(), sqlx::Error> {
async fn run_job( async fn run_job(
pool: &PgPool, pool: &PgPool,
runtime: &cm_runtime::Runtime, runtime: &cm_runtime::Runtime,
hub: &Arc<crate::fleet::NodeHub>,
job: cm_db::repo::topology_runs::ClaimedTopologyRun, job: cm_db::repo::topology_runs::ClaimedTopologyRun,
) { ) {
let id = job.id; let id = job.id;
@@ -156,9 +166,21 @@ async fn run_job(
// Resume from the last checkpoint, or start fresh. // Resume from the last checkpoint, or start fresh.
let progress: RunProgress = job let progress: RunProgress = job
.checkpoint .checkpoint
.clone()
.and_then(|c| serde_json::from_value(c).ok()) .and_then(|c| serde_json::from_value(c).ok())
.unwrap_or_default(); .unwrap_or_default();
// The composed engines (Slice 4): this graph's nodes are not claws, they are
// Claude-Code-in-a-microVM sessions. Branched BEFORE the leaf executor is
// built, because that build reads the ZeroClaw gateway config — a composed
// run must not fail for want of a runtime it never dials.
if job.tier == "microvm_graph" {
let result = run_composed(pool, hub, &job, &graph, progress).await;
finish(pool, id, result).await;
maybe_teardown_ephemeral_team(pool, runtime, id).await;
return;
}
// C3: prefer the mission's per-run runtime endpoint when set on // C3: prefer the mission's per-run runtime endpoint when set on
// the missions row; else fall back to the shared env-derived // the missions row; else fall back to the shared env-derived
// gateway (pre-C3 missions + non-mission runs). This is what // gateway (pre-C3 missions + non-mission runs). This is what
@@ -212,6 +234,14 @@ async fn run_job(
_ => drive(pool, id, &graph, &job.task, progress, &leaf).await, _ => drive(pool, id, &graph, &job.task, progress, &leaf).await,
}; };
finish(pool, id, result).await;
maybe_teardown_ephemeral_team(pool, runtime, id).await;
}
/// Write a driven run's terminal state. The single place a run finishes, shared
/// by every tier — a second one would be a second completion path, which is where
/// every microVM bug this project has hit came from.
async fn finish(pool: &PgPool, id: Uuid, result: Result<RunRecord, OrchestratorError>) {
match result { match result {
Ok(record) => { Ok(record) => {
let value = serde_json::to_value(&record).unwrap_or(serde_json::Value::Null); let value = serde_json::to_value(&record).unwrap_or(serde_json::Value::Null);
@@ -230,7 +260,57 @@ async fn run_job(
} }
} }
} }
maybe_teardown_ephemeral_team(pool, runtime, id).await; }
/// Drive a composed run: the outer graph is Engine Z, every node is a
/// Claude-Code-in-a-microVM session (Engine C).
///
/// The mission columns are read here rather than carried on the run row so a
/// re-placed or re-backed mission takes effect on resume, and so the composed
/// path has exactly one source of truth for where a VM boots.
async fn run_composed(
pool: &PgPool,
hub: &Arc<crate::fleet::NodeHub>,
job: &cm_db::repo::topology_runs::ClaimedTopologyRun,
graph: &TopologyGraph,
progress: RunProgress,
) -> Result<RunRecord, OrchestratorError> {
let mission_id = job.mission_id.ok_or_else(|| {
OrchestratorError::Executor(
"a composed run has no mission, so there is no checkout for its nodes \
to share"
.into(),
)
})?;
let phase_id = job.mission_phase_id.ok_or_else(|| {
OrchestratorError::Executor("a composed run must belong to a mission phase".into())
})?;
let mission: (Option<Uuid>, Option<String>, Option<String>) =
sqlx::query_as("SELECT target_node_id, backend, team_engine FROM missions WHERE id = $1")
.bind(mission_id)
.fetch_one(pool)
.await
.map_err(|e| OrchestratorError::Executor(format!("load mission {mission_id}: {e}")))?;
let exec = crate::microvm_turn_executor::for_fleet(
hub.clone(),
pool.clone(),
crate::microvm_turn_executor::ComposedRun {
run_id: job.id,
mission_id,
phase_id,
iteration: job.iteration.unwrap_or(1),
repo: crate::mission_workspace::checkout_path(mission_id),
target_node_id: mission.0,
backend: mission.1,
team_engine: mission.2,
// Resume continues the step numbering; restarting it would re-use a
// finished node's vm id.
completed_steps: progress.completed as u32,
},
);
drive(pool, job.id, graph, &job.task, progress, &exec).await
} }
/// Post-terminal hook: if this run's team is `ephemeral` and no siblings are /// Post-terminal hook: if this run's team is `ephemeral` and no siblings are
+42 -2
View File
@@ -54,6 +54,15 @@ pub struct ClaimedTopologyRun {
/// Deploy tier: `team` drives claws directly; `company`/`org` drive the /// Deploy tier: `team` drives claws directly; `company`/`org` drive the
/// recursive sub-topology executor. /// recursive sub-topology executor.
pub tier: String, pub tier: String,
/// The mission this run belongs to, when it belongs to one. The composed
/// (`microvm_graph`) tier needs it: its nodes share the mission's checkout,
/// and that shared tree is how file work survives a node boundary.
pub mission_id: Option<Uuid>,
/// The mission phase, for the same reason — the phase and pass identify the
/// VMs a composed run may boot.
pub mission_phase_id: Option<Uuid>,
/// Which pass of the phase produced this run.
pub iteration: Option<i32>,
} }
/// Lifecycle status + progress for a durable run (status endpoint). /// Lifecycle status + progress for a durable run (status endpoint).
@@ -222,7 +231,34 @@ pub async fn check_ephemeral_teardown(
/// ///
/// An allowlist rather than a denylist on purpose: the next self-driven tier is /// An allowlist rather than a denylist on purpose: the next self-driven tier is
/// then safe by default, instead of exposed until someone remembers this file. /// then safe by default, instead of exposed until someone remembers this file.
pub const WORKER_DRIVEN_TIERS: &[&str] = &["team", "company", "org", "swarm", "compare"]; pub const WORKER_DRIVEN_TIERS: &[&str] = &[
"team",
"company",
"org",
"swarm",
"compare",
// The composed engines: a ZeroClaw graph whose every node is a
// Claude-Code-in-a-microVM session. Worker-driven BY DESIGN — the outer
// graph's durability (checkpoint, resume, cancellation) is the entire reason
// the tier exists, and it comes from being claimed like any other job. It
// survives `requeue_stale` because the executor touches `updated_at` from a
// ticker for the whole length of a VM turn, not only between steps.
"microvm_graph",
];
/// Tiers the stuck-run reaper may fail.
///
/// A subset of [`WORKER_DRIVEN_TIERS`], and the difference matters. The reaper
/// asks "has this run journaled a step within 15 minutes of being CREATED?",
/// which assumes a step is short. A `microvm_graph` node is a whole agent session
/// in a VM with an hour's budget, so a healthy composed run can legitimately
/// journal nothing for far longer than the reaper's patience — it would kill the
/// run and orphan a live VM, which is #54 wearing a different tier.
///
/// Losing the reaper for that tier costs little: a composed run that genuinely
/// wedges stops touching `updated_at` and `requeue_stale` recovers it at 180s,
/// which is the mechanism the reaper was a backstop for in the first place.
pub const REAPABLE_TIERS: &[&str] = &["team", "company", "org", "swarm", "compare"];
/// The allowlist as owned strings, for binding as `text[]`. /// The allowlist as owned strings, for binding as `text[]`.
fn worker_driven() -> Vec<String> { fn worker_driven() -> Vec<String> {
@@ -251,7 +287,8 @@ pub async fn claim_next_queued(pool: &PgPool) -> Result<Option<ClaimedTopologyRu
FOR UPDATE SKIP LOCKED FOR UPDATE SKIP LOCKED
LIMIT 1 LIMIT 1
) )
RETURNING id, workspace_id, task, graph, checkpoint, last_event_id, tier", RETURNING id, workspace_id, task, graph, checkpoint, last_event_id, tier,
mission_id, mission_phase_id, iteration",
) )
.bind(worker_driven()) .bind(worker_driven())
.fetch_optional(pool) .fetch_optional(pool)
@@ -264,6 +301,9 @@ pub async fn claim_next_queued(pool: &PgPool) -> Result<Option<ClaimedTopologyRu
checkpoint: r.get("checkpoint"), checkpoint: r.get("checkpoint"),
last_event_id: r.get("last_event_id"), last_event_id: r.get("last_event_id"),
tier: r.get("tier"), tier: r.get("tier"),
mission_id: r.get("mission_id"),
mission_phase_id: r.get("mission_phase_id"),
iteration: r.get("iteration"),
})) }))
} }
+59
View File
@@ -139,6 +139,53 @@ async fn the_worker_will_not_claim_a_self_driven_run() {
assert_eq!(status_of(&pool, id).await, "queued", "and it must be left as it was"); assert_eq!(status_of(&pool, id).await, "queued", "and it must be left as it was");
} }
/// The composed tier is the mirror image of the two above and must not be
/// mistaken for them: it runs VMs, but the WORKER drives its graph, so being
/// claimed and requeued is exactly what gives it checkpointing and resume.
#[tokio::test]
async fn the_worker_claims_and_rescues_a_composed_run() {
let pool = cm_testkit::test_pool().await;
let ws = workspace(&pool).await;
let id = Uuid::now_v7();
topology_runs::enqueue_run_tier(
&pool,
id,
ws,
"compose the engines",
// A real graph, unlike the self-driven placeholder: the worker plans it.
&serde_json::json!({
"kind": "pipeline",
"nodes": [{ "id": "a", "role": "worker", "attrs": {} }],
"edges": []
}),
"microvm_graph",
)
.await
.expect("enqueue");
let claimed = topology_runs::claim_next_queued(&pool)
.await
.expect("claim")
.expect("a composed run must be claimable, or it never runs at all");
assert_eq!(claimed.tier, "microvm_graph");
assert_eq!(claimed.id, id);
// And a composed run whose worker died must come back: its checkpoint is
// what makes resume possible, and requeue is what triggers it.
sqlx::query("UPDATE topology_runs SET updated_at = now() - interval '30 minutes' WHERE id = $1")
.bind(id)
.execute(&pool)
.await
.expect("age the run");
topology_runs::requeue_stale(&pool, 180.0).await.expect("requeue");
assert_eq!(
status_of(&pool, id).await,
"queued",
"a composed run orphaned by a dead worker must be recovered"
);
}
/// The allowlist is the single place this policy lives, so assert its membership /// The allowlist is the single place this policy lives, so assert its membership
/// directly — a new self-driven tier added without touching it would otherwise be /// directly — a new self-driven tier added without touching it would otherwise be
/// exposed exactly as microvm was. /// exposed exactly as microvm was.
@@ -156,4 +203,16 @@ fn the_allowlist_names_only_worker_driven_tiers() {
"{self_driven} owns its own lifecycle; sweeping it kills live work" "{self_driven} owns its own lifecycle; sweeping it kills live work"
); );
} }
// `microvm_graph` is worker-driven but NOT reapable: one of its steps is a
// whole agent session in a VM, so "no step records in 15 minutes" is what a
// healthy composed run looks like, and reaping it would orphan a live VM —
// #54 in a different tier.
assert!(topology_runs::WORKER_DRIVEN_TIERS.contains(&"microvm_graph"));
assert!(!topology_runs::REAPABLE_TIERS.contains(&"microvm_graph"));
for reapable in topology_runs::REAPABLE_TIERS {
assert!(
topology_runs::WORKER_DRIVEN_TIERS.contains(reapable),
"{reapable} is reaped but never driven"
);
}
} }