feat(missions): Slice 4 — the two engines composed, with the file handoff proven
`team_engine='composed'` (the third name migration 0069 anticipated) runs a
mission as a durable ZeroClaw graph whose every node is a whole
Claude-Code-in-a-microVM session. Engine Z owns checkpoint/resume, cancellation
and per-node heterogeneity; Engine C owns shared context and cheap fan-out;
neither has the other's asset, which is why this is a composition and not a
compromise.
`MicroVmTurnExecutor` implements the existing `TurnExecutor`, so it inherits the
planners, the checkpoint, the stale-run recovery, `close_finished_phases`, the
evaluator, capture and delivery unchanged — the same trick `SubTopologyExecutor`
already plays with a heavy `run_turn`. Producer side emits ONE `queued` row
carrying the real graph and lets the worker claim it: the durability IS being
worker-driven, and the solo path's `tokio::spawn` has none of it. Still exactly
one `topology_runs` row per unit of work and one completion path — `finish()` is
now that one place, shared by every tier.
THE TRAP, solved and proven. A VM is inject → run → collect → destroy, so a
per-node VM with text-only handoff silently loses every file an earlier node
wrote: node 2 boots from the original checkout, sees nothing, and still reports
success. The mission's host checkout is the medium — every node injects from it
and collects back over it — and two properties make that safe rather than lucky:
`execute_resumable` is strictly sequential, so two VMs never write one directory;
and the vm id is deterministic per (phase, iteration, step), so a duplicate is
refused by the node ("vm already exists") instead of becoming a second writer.
NEGATIVE CONTROL, run rather than assumed: with `repo` swapped for a private
per-node workspace, `a_later_node_sees_an_earlier_nodes_files` FAILS with
`saw:[]`; restored, it passes. The `PhaseVm` seam exists for exactly this — it
models inject/collect through the real `mission_fs` tar path in milliseconds.
Two durability traps this tier walks into, both closed:
- `requeue_stale` fires at 180s on `updated_at`, and one node here can run for
an hour. `SubTopologyExecutor` keeps its parent alive from each leaf step;
there is nothing between the start and end of a VM turn, so the turn holds a
ticker that touches `updated_at` every 30s and aborts on drop. Without it a
healthy composed run is requeued mid-node and boots a second VM.
- the 15-minute stuck-run reaper asks "any step records since it was CREATED?",
which describes a healthy composed run as readily as a wedged one. Hence
`REAPABLE_TIERS` — worker-driven minus this tier. Reaping it would be #54 in
a different costume.
`on_launch` mints no team for a microVM mission, deliberately: claws in
containers are what a VM mission does not use. So `mission_orchestrator::
composed_graph` builds the shape from the team template directly — nodes, roles
and pattern, zero claws provisioned. Per-node `attrs["backend"]` and
`attrs["node_id"]` override the mission's, which is what makes a validator node
on another provider's image a first-class graph node; a malformed `node_id`
fails the node rather than quietly running it where the graph did not ask.
Refusals are recorded as a failed run, not returned as an error: `launch_phase`
is swept every ten seconds, so a returned error is a phase that retries forever
while the log repeats itself.
501 tests pass, clippy clean. NOT yet proven end to end: no composed mission has
run on the fleet, so the resume-after-a-killed-worker leg is argued from the DB
test and the step-numbering test, not from a real two-node run.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
e31688bac5
commit
12147a1e01
@@ -54,6 +54,15 @@ pub struct ClaimedTopologyRun {
|
||||
/// Deploy tier: `team` drives claws directly; `company`/`org` drive the
|
||||
/// recursive sub-topology executor.
|
||||
pub tier: String,
|
||||
/// The mission this run belongs to, when it belongs to one. The composed
|
||||
/// (`microvm_graph`) tier needs it: its nodes share the mission's checkout,
|
||||
/// and that shared tree is how file work survives a node boundary.
|
||||
pub mission_id: Option<Uuid>,
|
||||
/// The mission phase, for the same reason — the phase and pass identify the
|
||||
/// VMs a composed run may boot.
|
||||
pub mission_phase_id: Option<Uuid>,
|
||||
/// Which pass of the phase produced this run.
|
||||
pub iteration: Option<i32>,
|
||||
}
|
||||
|
||||
/// Lifecycle status + progress for a durable run (status endpoint).
|
||||
@@ -222,7 +231,34 @@ pub async fn check_ephemeral_teardown(
|
||||
///
|
||||
/// An allowlist rather than a denylist on purpose: the next self-driven tier is
|
||||
/// then safe by default, instead of exposed until someone remembers this file.
|
||||
pub const WORKER_DRIVEN_TIERS: &[&str] = &["team", "company", "org", "swarm", "compare"];
|
||||
pub const WORKER_DRIVEN_TIERS: &[&str] = &[
|
||||
"team",
|
||||
"company",
|
||||
"org",
|
||||
"swarm",
|
||||
"compare",
|
||||
// The composed engines: a ZeroClaw graph whose every node is a
|
||||
// Claude-Code-in-a-microVM session. Worker-driven BY DESIGN — the outer
|
||||
// graph's durability (checkpoint, resume, cancellation) is the entire reason
|
||||
// the tier exists, and it comes from being claimed like any other job. It
|
||||
// survives `requeue_stale` because the executor touches `updated_at` from a
|
||||
// ticker for the whole length of a VM turn, not only between steps.
|
||||
"microvm_graph",
|
||||
];
|
||||
|
||||
/// Tiers the stuck-run reaper may fail.
|
||||
///
|
||||
/// A subset of [`WORKER_DRIVEN_TIERS`], and the difference matters. The reaper
|
||||
/// asks "has this run journaled a step within 15 minutes of being CREATED?",
|
||||
/// which assumes a step is short. A `microvm_graph` node is a whole agent session
|
||||
/// in a VM with an hour's budget, so a healthy composed run can legitimately
|
||||
/// journal nothing for far longer than the reaper's patience — it would kill the
|
||||
/// run and orphan a live VM, which is #54 wearing a different tier.
|
||||
///
|
||||
/// Losing the reaper for that tier costs little: a composed run that genuinely
|
||||
/// wedges stops touching `updated_at` and `requeue_stale` recovers it at 180s,
|
||||
/// which is the mechanism the reaper was a backstop for in the first place.
|
||||
pub const REAPABLE_TIERS: &[&str] = &["team", "company", "org", "swarm", "compare"];
|
||||
|
||||
/// The allowlist as owned strings, for binding as `text[]`.
|
||||
fn worker_driven() -> Vec<String> {
|
||||
@@ -251,7 +287,8 @@ pub async fn claim_next_queued(pool: &PgPool) -> Result<Option<ClaimedTopologyRu
|
||||
FOR UPDATE SKIP LOCKED
|
||||
LIMIT 1
|
||||
)
|
||||
RETURNING id, workspace_id, task, graph, checkpoint, last_event_id, tier",
|
||||
RETURNING id, workspace_id, task, graph, checkpoint, last_event_id, tier,
|
||||
mission_id, mission_phase_id, iteration",
|
||||
)
|
||||
.bind(worker_driven())
|
||||
.fetch_optional(pool)
|
||||
@@ -264,6 +301,9 @@ pub async fn claim_next_queued(pool: &PgPool) -> Result<Option<ClaimedTopologyRu
|
||||
checkpoint: r.get("checkpoint"),
|
||||
last_event_id: r.get("last_event_id"),
|
||||
tier: r.get("tier"),
|
||||
mission_id: r.get("mission_id"),
|
||||
mission_phase_id: r.get("mission_phase_id"),
|
||||
iteration: r.get("iteration"),
|
||||
}))
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user