feat(missions): Slice 4 — the two engines composed, with the file handoff proven

`team_engine='composed'` (the third name migration 0069 anticipated) runs a
mission as a durable ZeroClaw graph whose every node is a whole
Claude-Code-in-a-microVM session. Engine Z owns checkpoint/resume, cancellation
and per-node heterogeneity; Engine C owns shared context and cheap fan-out;
neither has the other's asset, which is why this is a composition and not a
compromise.

`MicroVmTurnExecutor` implements the existing `TurnExecutor`, so it inherits the
planners, the checkpoint, the stale-run recovery, `close_finished_phases`, the
evaluator, capture and delivery unchanged — the same trick `SubTopologyExecutor`
already plays with a heavy `run_turn`. Producer side emits ONE `queued` row
carrying the real graph and lets the worker claim it: the durability IS being
worker-driven, and the solo path's `tokio::spawn` has none of it. Still exactly
one `topology_runs` row per unit of work and one completion path — `finish()` is
now that one place, shared by every tier.

THE TRAP, solved and proven. A VM is inject → run → collect → destroy, so a
per-node VM with text-only handoff silently loses every file an earlier node
wrote: node 2 boots from the original checkout, sees nothing, and still reports
success. The mission's host checkout is the medium — every node injects from it
and collects back over it — and two properties make that safe rather than lucky:
`execute_resumable` is strictly sequential, so two VMs never write one directory;
and the vm id is deterministic per (phase, iteration, step), so a duplicate is
refused by the node ("vm already exists") instead of becoming a second writer.

NEGATIVE CONTROL, run rather than assumed: with `repo` swapped for a private
per-node workspace, `a_later_node_sees_an_earlier_nodes_files` FAILS with
`saw:[]`; restored, it passes. The `PhaseVm` seam exists for exactly this — it
models inject/collect through the real `mission_fs` tar path in milliseconds.

Two durability traps this tier walks into, both closed:

  - `requeue_stale` fires at 180s on `updated_at`, and one node here can run for
    an hour. `SubTopologyExecutor` keeps its parent alive from each leaf step;
    there is nothing between the start and end of a VM turn, so the turn holds a
    ticker that touches `updated_at` every 30s and aborts on drop. Without it a
    healthy composed run is requeued mid-node and boots a second VM.
  - the 15-minute stuck-run reaper asks "any step records since it was CREATED?",
    which describes a healthy composed run as readily as a wedged one. Hence
    `REAPABLE_TIERS` — worker-driven minus this tier. Reaping it would be #54 in
    a different costume.

`on_launch` mints no team for a microVM mission, deliberately: claws in
containers are what a VM mission does not use. So `mission_orchestrator::
composed_graph` builds the shape from the team template directly — nodes, roles
and pattern, zero claws provisioned. Per-node `attrs["backend"]` and
`attrs["node_id"]` override the mission's, which is what makes a validator node
on another provider's image a first-class graph node; a malformed `node_id`
fails the node rather than quietly running it where the graph did not ask.

Refusals are recorded as a failed run, not returned as an error: `launch_phase`
is swept every ten seconds, so a returned error is a phase that retries forever
while the log repeats itself.

501 tests pass, clippy clean. NOT yet proven end to end: no composed mission has
run on the fleet, so the resume-after-a-killed-worker leg is argued from the DB
test and the step-numbering test, not from a real two-node run.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-06 13:00:44 -07:00
co-authored by Claude Opus 5
parent e31688bac5
commit 12147a1e01
9 changed files with 1141 additions and 17 deletions
+67
View File
@@ -574,6 +574,73 @@ async fn mint_team_from_template(
Ok(team_id)
}
/// The graph a COMPOSED microVM mission runs, built from its team template
/// without minting a single claw.
///
/// A composed mission needs the template's *shape* — how many nodes, in what
/// pattern, playing what roles — and nothing else it carries. Its nodes are VMs,
/// so provisioning claws for them would create agents, containers and `.brain`
/// files that nothing ever dials; that is exactly why `on_launch` returns early
/// for a microVM mission, and this is how the composed path gets its graph
/// anyway rather than by undoing that.
///
/// `purposes` is the phase's purpose list, matched against `config.phase_teams`;
/// missions using the legacy single `team_template_id` fall back to it.
/// Returns `None` when the mission picked no template at all.
pub async fn composed_graph(
pool: &PgPool,
mission_id: Uuid,
purposes: &[&str],
) -> Result<Option<serde_json::Value>, String> {
let row: Option<(serde_json::Value, Option<Uuid>)> =
sqlx::query_as("SELECT config, team_template_id FROM missions WHERE id = $1")
.bind(mission_id)
.fetch_optional(pool)
.await
.map_err(|e| format!("load mission {mission_id}: {e}"))?;
let Some((config, legacy_template)) = row else {
return Err(format!("mission {mission_id} not found"));
};
let template_id = config
.get("phase_teams")
.and_then(|v| v.as_object())
.and_then(|pt| {
// First template named by any purpose this phase answers to, in the
// phase's own preference order — the same order `launch_phase` uses
// to pick teams, so a composed mission and a ZeroClaw one resolve the
// same template for the same phase.
purposes.iter().find_map(|p| {
pt.get(*p)
.and_then(|v| v.as_array())
.and_then(|a| a.first())
.and_then(|v| v.as_str())
.and_then(|s| Uuid::parse_str(s).ok())
})
})
.or(legacy_template);
let Some(template_id) = template_id else {
return Ok(None);
};
let template = cm_db::repo::team_templates::get(pool, template_id)
.await
.map_err(|e| format!("load template {template_id}: {e}"))?
.ok_or_else(|| format!("template {template_id} not found"))?;
let roles: Vec<&str> = template.roles.iter().map(|r| r.slot.as_str()).collect();
if roles.is_empty() {
return Err(format!("template {template_id} defines no roles"));
}
let graph = cm_topology::build(
parse_topology_kind(&template.template.default_topology),
&roles,
)
.map_err(|e| format!("build topology graph for template {template_id}: {e}"))?;
serde_json::to_value(&graph)
.map(Some)
.map_err(|e| format!("serialize topology graph: {e}"))
}
fn parse_topology_kind(s: &str) -> cm_topology::TopologyKind {
use cm_topology::TopologyKind;
match s {