feat(missions): Slice 4 — the two engines composed, with the file handoff proven
`team_engine='composed'` (the third name migration 0069 anticipated) runs a
mission as a durable ZeroClaw graph whose every node is a whole
Claude-Code-in-a-microVM session. Engine Z owns checkpoint/resume, cancellation
and per-node heterogeneity; Engine C owns shared context and cheap fan-out;
neither has the other's asset, which is why this is a composition and not a
compromise.
`MicroVmTurnExecutor` implements the existing `TurnExecutor`, so it inherits the
planners, the checkpoint, the stale-run recovery, `close_finished_phases`, the
evaluator, capture and delivery unchanged — the same trick `SubTopologyExecutor`
already plays with a heavy `run_turn`. Producer side emits ONE `queued` row
carrying the real graph and lets the worker claim it: the durability IS being
worker-driven, and the solo path's `tokio::spawn` has none of it. Still exactly
one `topology_runs` row per unit of work and one completion path — `finish()` is
now that one place, shared by every tier.
THE TRAP, solved and proven. A VM is inject → run → collect → destroy, so a
per-node VM with text-only handoff silently loses every file an earlier node
wrote: node 2 boots from the original checkout, sees nothing, and still reports
success. The mission's host checkout is the medium — every node injects from it
and collects back over it — and two properties make that safe rather than lucky:
`execute_resumable` is strictly sequential, so two VMs never write one directory;
and the vm id is deterministic per (phase, iteration, step), so a duplicate is
refused by the node ("vm already exists") instead of becoming a second writer.
NEGATIVE CONTROL, run rather than assumed: with `repo` swapped for a private
per-node workspace, `a_later_node_sees_an_earlier_nodes_files` FAILS with
`saw:[]`; restored, it passes. The `PhaseVm` seam exists for exactly this — it
models inject/collect through the real `mission_fs` tar path in milliseconds.
Two durability traps this tier walks into, both closed:
- `requeue_stale` fires at 180s on `updated_at`, and one node here can run for
an hour. `SubTopologyExecutor` keeps its parent alive from each leaf step;
there is nothing between the start and end of a VM turn, so the turn holds a
ticker that touches `updated_at` every 30s and aborts on drop. Without it a
healthy composed run is requeued mid-node and boots a second VM.
- the 15-minute stuck-run reaper asks "any step records since it was CREATED?",
which describes a healthy composed run as readily as a wedged one. Hence
`REAPABLE_TIERS` — worker-driven minus this tier. Reaping it would be #54 in
a different costume.
`on_launch` mints no team for a microVM mission, deliberately: claws in
containers are what a VM mission does not use. So `mission_orchestrator::
composed_graph` builds the shape from the team template directly — nodes, roles
and pattern, zero claws provisioned. Per-node `attrs["backend"]` and
`attrs["node_id"]` override the mission's, which is what makes a validator node
on another provider's image a first-class graph node; a malformed `node_id`
fails the node rather than quietly running it where the graph did not ask.
Refusals are recorded as a failed run, not returned as an error: `launch_phase`
is swept every ten seconds, so a returned error is a phase that retries forever
while the log repeats itself.
501 tests pass, clippy clean. NOT yet proven end to end: no composed mission has
run on the fleet, so the resume-after-a-killed-worker leg is argued from the DB
test and the step-numbering test, not from a real two-node run.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
co-authored by
Claude Opus 5
Claude Opus 5
parent
e31688bac5
commit
12147a1e01
@@ -33,7 +33,12 @@ const REAP_STUCK_AFTER_SECS: i64 = 15 * 60;
|
||||
|
||||
/// Spawn the durable topology job worker. Polls for queued jobs every `poll`
|
||||
/// interval; runs each to completion (or failure), checkpointing per step.
|
||||
pub fn spawn(pool: PgPool, runtime: cm_runtime::Runtime, poll: Duration) {
|
||||
pub fn spawn(
|
||||
pool: PgPool,
|
||||
runtime: cm_runtime::Runtime,
|
||||
hub: Arc<crate::fleet::NodeHub>,
|
||||
poll: Duration,
|
||||
) {
|
||||
// Fire the stuck-container reaper on its own cadence — checking
|
||||
// once a minute is plenty and keeps this off the hot claim loop.
|
||||
let reaper_pool = pool.clone();
|
||||
@@ -55,7 +60,7 @@ pub fn spawn(pool: PgPool, runtime: cm_runtime::Runtime, poll: Duration) {
|
||||
eprintln!("topology_worker: requeue_stale failed: {e}");
|
||||
}
|
||||
match cm_db::repo::topology_runs::claim_next_queued(&pool).await {
|
||||
Ok(Some(job)) => run_job(&pool, &runtime, job).await,
|
||||
Ok(Some(job)) => run_job(&pool, &runtime, &hub, job).await,
|
||||
Ok(None) => tokio::time::sleep(poll).await,
|
||||
Err(e) => {
|
||||
eprintln!("topology_worker: claim failed: {e}");
|
||||
@@ -90,7 +95,11 @@ async fn reap_stuck_runs(pool: &PgPool) -> Result<(), sqlx::Error> {
|
||||
)
|
||||
.bind(REAP_STUCK_AFTER_SECS as f64)
|
||||
.bind(
|
||||
cm_db::repo::topology_runs::WORKER_DRIVEN_TIERS
|
||||
// REAPABLE, not worker-driven: `microvm_graph` is driven by this worker
|
||||
// and must NOT be reaped — one of its steps is a whole agent session in a
|
||||
// VM, so "no step records in 15 minutes" describes a healthy composed run
|
||||
// as readily as a wedged one.
|
||||
cm_db::repo::topology_runs::REAPABLE_TIERS
|
||||
.iter()
|
||||
.map(|s| (*s).to_string())
|
||||
.collect::<Vec<_>>(),
|
||||
@@ -119,6 +128,7 @@ async fn reap_stuck_runs(pool: &PgPool) -> Result<(), sqlx::Error> {
|
||||
async fn run_job(
|
||||
pool: &PgPool,
|
||||
runtime: &cm_runtime::Runtime,
|
||||
hub: &Arc<crate::fleet::NodeHub>,
|
||||
job: cm_db::repo::topology_runs::ClaimedTopologyRun,
|
||||
) {
|
||||
let id = job.id;
|
||||
@@ -156,9 +166,21 @@ async fn run_job(
|
||||
// Resume from the last checkpoint, or start fresh.
|
||||
let progress: RunProgress = job
|
||||
.checkpoint
|
||||
.clone()
|
||||
.and_then(|c| serde_json::from_value(c).ok())
|
||||
.unwrap_or_default();
|
||||
|
||||
// The composed engines (Slice 4): this graph's nodes are not claws, they are
|
||||
// Claude-Code-in-a-microVM sessions. Branched BEFORE the leaf executor is
|
||||
// built, because that build reads the ZeroClaw gateway config — a composed
|
||||
// run must not fail for want of a runtime it never dials.
|
||||
if job.tier == "microvm_graph" {
|
||||
let result = run_composed(pool, hub, &job, &graph, progress).await;
|
||||
finish(pool, id, result).await;
|
||||
maybe_teardown_ephemeral_team(pool, runtime, id).await;
|
||||
return;
|
||||
}
|
||||
|
||||
// C3: prefer the mission's per-run runtime endpoint when set on
|
||||
// the missions row; else fall back to the shared env-derived
|
||||
// gateway (pre-C3 missions + non-mission runs). This is what
|
||||
@@ -212,6 +234,14 @@ async fn run_job(
|
||||
_ => drive(pool, id, &graph, &job.task, progress, &leaf).await,
|
||||
};
|
||||
|
||||
finish(pool, id, result).await;
|
||||
maybe_teardown_ephemeral_team(pool, runtime, id).await;
|
||||
}
|
||||
|
||||
/// Write a driven run's terminal state. The single place a run finishes, shared
|
||||
/// by every tier — a second one would be a second completion path, which is where
|
||||
/// every microVM bug this project has hit came from.
|
||||
async fn finish(pool: &PgPool, id: Uuid, result: Result<RunRecord, OrchestratorError>) {
|
||||
match result {
|
||||
Ok(record) => {
|
||||
let value = serde_json::to_value(&record).unwrap_or(serde_json::Value::Null);
|
||||
@@ -230,7 +260,57 @@ async fn run_job(
|
||||
}
|
||||
}
|
||||
}
|
||||
maybe_teardown_ephemeral_team(pool, runtime, id).await;
|
||||
}
|
||||
|
||||
/// Drive a composed run: the outer graph is Engine Z, every node is a
|
||||
/// Claude-Code-in-a-microVM session (Engine C).
|
||||
///
|
||||
/// The mission columns are read here rather than carried on the run row so a
|
||||
/// re-placed or re-backed mission takes effect on resume, and so the composed
|
||||
/// path has exactly one source of truth for where a VM boots.
|
||||
async fn run_composed(
|
||||
pool: &PgPool,
|
||||
hub: &Arc<crate::fleet::NodeHub>,
|
||||
job: &cm_db::repo::topology_runs::ClaimedTopologyRun,
|
||||
graph: &TopologyGraph,
|
||||
progress: RunProgress,
|
||||
) -> Result<RunRecord, OrchestratorError> {
|
||||
let mission_id = job.mission_id.ok_or_else(|| {
|
||||
OrchestratorError::Executor(
|
||||
"a composed run has no mission, so there is no checkout for its nodes \
|
||||
to share"
|
||||
.into(),
|
||||
)
|
||||
})?;
|
||||
let phase_id = job.mission_phase_id.ok_or_else(|| {
|
||||
OrchestratorError::Executor("a composed run must belong to a mission phase".into())
|
||||
})?;
|
||||
|
||||
let mission: (Option<Uuid>, Option<String>, Option<String>) =
|
||||
sqlx::query_as("SELECT target_node_id, backend, team_engine FROM missions WHERE id = $1")
|
||||
.bind(mission_id)
|
||||
.fetch_one(pool)
|
||||
.await
|
||||
.map_err(|e| OrchestratorError::Executor(format!("load mission {mission_id}: {e}")))?;
|
||||
|
||||
let exec = crate::microvm_turn_executor::for_fleet(
|
||||
hub.clone(),
|
||||
pool.clone(),
|
||||
crate::microvm_turn_executor::ComposedRun {
|
||||
run_id: job.id,
|
||||
mission_id,
|
||||
phase_id,
|
||||
iteration: job.iteration.unwrap_or(1),
|
||||
repo: crate::mission_workspace::checkout_path(mission_id),
|
||||
target_node_id: mission.0,
|
||||
backend: mission.1,
|
||||
team_engine: mission.2,
|
||||
// Resume continues the step numbering; restarting it would re-use a
|
||||
// finished node's vm id.
|
||||
completed_steps: progress.completed as u32,
|
||||
},
|
||||
);
|
||||
drive(pool, job.id, graph, &job.task, progress, &exec).await
|
||||
}
|
||||
|
||||
/// Post-terminal hook: if this run's team is `ephemeral` and no siblings are
|
||||
|
||||
Reference in New Issue
Block a user