feat(missions): Slice 4 — the two engines composed, with the file handoff proven

`team_engine='composed'` (the third name migration 0069 anticipated) runs a
mission as a durable ZeroClaw graph whose every node is a whole
Claude-Code-in-a-microVM session. Engine Z owns checkpoint/resume, cancellation
and per-node heterogeneity; Engine C owns shared context and cheap fan-out;
neither has the other's asset, which is why this is a composition and not a
compromise.

`MicroVmTurnExecutor` implements the existing `TurnExecutor`, so it inherits the
planners, the checkpoint, the stale-run recovery, `close_finished_phases`, the
evaluator, capture and delivery unchanged — the same trick `SubTopologyExecutor`
already plays with a heavy `run_turn`. Producer side emits ONE `queued` row
carrying the real graph and lets the worker claim it: the durability IS being
worker-driven, and the solo path's `tokio::spawn` has none of it. Still exactly
one `topology_runs` row per unit of work and one completion path — `finish()` is
now that one place, shared by every tier.

THE TRAP, solved and proven. A VM is inject → run → collect → destroy, so a
per-node VM with text-only handoff silently loses every file an earlier node
wrote: node 2 boots from the original checkout, sees nothing, and still reports
success. The mission's host checkout is the medium — every node injects from it
and collects back over it — and two properties make that safe rather than lucky:
`execute_resumable` is strictly sequential, so two VMs never write one directory;
and the vm id is deterministic per (phase, iteration, step), so a duplicate is
refused by the node ("vm already exists") instead of becoming a second writer.

NEGATIVE CONTROL, run rather than assumed: with `repo` swapped for a private
per-node workspace, `a_later_node_sees_an_earlier_nodes_files` FAILS with
`saw:[]`; restored, it passes. The `PhaseVm` seam exists for exactly this — it
models inject/collect through the real `mission_fs` tar path in milliseconds.

Two durability traps this tier walks into, both closed:

  - `requeue_stale` fires at 180s on `updated_at`, and one node here can run for
    an hour. `SubTopologyExecutor` keeps its parent alive from each leaf step;
    there is nothing between the start and end of a VM turn, so the turn holds a
    ticker that touches `updated_at` every 30s and aborts on drop. Without it a
    healthy composed run is requeued mid-node and boots a second VM.
  - the 15-minute stuck-run reaper asks "any step records since it was CREATED?",
    which describes a healthy composed run as readily as a wedged one. Hence
    `REAPABLE_TIERS` — worker-driven minus this tier. Reaping it would be #54 in
    a different costume.

`on_launch` mints no team for a microVM mission, deliberately: claws in
containers are what a VM mission does not use. So `mission_orchestrator::
composed_graph` builds the shape from the team template directly — nodes, roles
and pattern, zero claws provisioned. Per-node `attrs["backend"]` and
`attrs["node_id"]` override the mission's, which is what makes a validator node
on another provider's image a first-class graph node; a malformed `node_id`
fails the node rather than quietly running it where the graph did not ask.

Refusals are recorded as a failed run, not returned as an error: `launch_phase`
is swept every ten seconds, so a returned error is a phase that retries forever
while the log repeats itself.

501 tests pass, clippy clean. NOT yet proven end to end: no composed mission has
run on the fleet, so the resume-after-a-killed-worker leg is argued from the DB
test and the step-numbering test, not from a real two-node run.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-06 13:00:44 -07:00
co-authored by Claude Opus 5
parent e31688bac5
commit 12147a1e01
9 changed files with 1141 additions and 17 deletions
+42 -2
View File
@@ -54,6 +54,15 @@ pub struct ClaimedTopologyRun {
/// Deploy tier: `team` drives claws directly; `company`/`org` drive the
/// recursive sub-topology executor.
pub tier: String,
/// The mission this run belongs to, when it belongs to one. The composed
/// (`microvm_graph`) tier needs it: its nodes share the mission's checkout,
/// and that shared tree is how file work survives a node boundary.
pub mission_id: Option<Uuid>,
/// The mission phase, for the same reason — the phase and pass identify the
/// VMs a composed run may boot.
pub mission_phase_id: Option<Uuid>,
/// Which pass of the phase produced this run.
pub iteration: Option<i32>,
}
/// Lifecycle status + progress for a durable run (status endpoint).
@@ -222,7 +231,34 @@ pub async fn check_ephemeral_teardown(
///
/// An allowlist rather than a denylist on purpose: the next self-driven tier is
/// then safe by default, instead of exposed until someone remembers this file.
pub const WORKER_DRIVEN_TIERS: &[&str] = &["team", "company", "org", "swarm", "compare"];
pub const WORKER_DRIVEN_TIERS: &[&str] = &[
"team",
"company",
"org",
"swarm",
"compare",
// The composed engines: a ZeroClaw graph whose every node is a
// Claude-Code-in-a-microVM session. Worker-driven BY DESIGN — the outer
// graph's durability (checkpoint, resume, cancellation) is the entire reason
// the tier exists, and it comes from being claimed like any other job. It
// survives `requeue_stale` because the executor touches `updated_at` from a
// ticker for the whole length of a VM turn, not only between steps.
"microvm_graph",
];
/// Tiers the stuck-run reaper may fail.
///
/// A subset of [`WORKER_DRIVEN_TIERS`], and the difference matters. The reaper
/// asks "has this run journaled a step within 15 minutes of being CREATED?",
/// which assumes a step is short. A `microvm_graph` node is a whole agent session
/// in a VM with an hour's budget, so a healthy composed run can legitimately
/// journal nothing for far longer than the reaper's patience — it would kill the
/// run and orphan a live VM, which is #54 wearing a different tier.
///
/// Losing the reaper for that tier costs little: a composed run that genuinely
/// wedges stops touching `updated_at` and `requeue_stale` recovers it at 180s,
/// which is the mechanism the reaper was a backstop for in the first place.
pub const REAPABLE_TIERS: &[&str] = &["team", "company", "org", "swarm", "compare"];
/// The allowlist as owned strings, for binding as `text[]`.
fn worker_driven() -> Vec<String> {
@@ -251,7 +287,8 @@ pub async fn claim_next_queued(pool: &PgPool) -> Result<Option<ClaimedTopologyRu
FOR UPDATE SKIP LOCKED
LIMIT 1
)
RETURNING id, workspace_id, task, graph, checkpoint, last_event_id, tier",
RETURNING id, workspace_id, task, graph, checkpoint, last_event_id, tier,
mission_id, mission_phase_id, iteration",
)
.bind(worker_driven())
.fetch_optional(pool)
@@ -264,6 +301,9 @@ pub async fn claim_next_queued(pool: &PgPool) -> Result<Option<ClaimedTopologyRu
checkpoint: r.get("checkpoint"),
last_event_id: r.get("last_event_id"),
tier: r.get("tier"),
mission_id: r.get("mission_id"),
mission_phase_id: r.get("mission_phase_id"),
iteration: r.get("iteration"),
}))
}