fix(missions): #54 — the worker was killing live microVM runs at 180 seconds

My hypothesis in #54 was WRONG, and it was wrong because I built it on a bad
measurement: `grep -c 'microvm phase'` returned 0, so I concluded the completion
log never printed and blamed the 15-minute reaper. The line was there all along, at
14:17:45. The real cause is worse.

`requeue_stale` has NO TIER FILTER. A microvm run's `updated_at` is written once at
insert and never again — it is driven by a `tokio::spawn` that owns it start to
finish, and nothing in `microvm_executor` writes `topology_runs`. So at 180s the
sweeper declared a perfectly healthy run stale and flipped it to `queued`;
`claim_next_queued` (no tier filter either) handed it to the worker; `run_job`
tried to parse the microvm graph placeholder, which `TopologyGraph` cannot
deserialize; and it failed the run with "missing or invalid graph".

Mission 019fd43e: run created 14:11:16, mission failed ~14:14:46. 210 seconds — the
180s window plus a tick. The agent went on working and finished at 14:17:45 with
three modules written, by which time the phase was already dead and the VM was
orphaned. A firecracker process was still alive 1h37m later.

THE UNCOMFORTABLE PART: every microVM mission that appeared to work this session
did so only by finishing inside three minutes. The 90-second ones dodged this. The
harness scenario dodges it. Nothing about that was visible.

`WORKER_DRIVEN_TIERS` (team, company, org, swarm, compare) is now the allowlist for
all three sweep paths — claim, requeue, reap. An allowlist rather than a denylist so
the next self-driven tier is safe by default instead of exposed until someone
remembers the file. `tier='session'` had exactly the same exposure and is covered
too. A unit test asserts microvm and session are NOT in it, next to the code that
inserts them.

Two more fixes from the same wreckage:

  - `destroy` reported `killed: pgid.is_some()` — true whenever there was a pgid to
    signal, whether or not anything died. It now sends the signal, polls /proc for
    the group leader, retries, and reports what it OBSERVED; `signalled` keeps the
    old meaning so "nothing to kill" is distinguishable from "it would not die".
  - the run-status update is now guarded with `AND status <> 'cancelled'`. An
    operator cancelling is a decision; this task reporting an outcome minutes later
    is an observation, and it must not overwrite one with the other.

And the root cause of the collect timeout itself: `mission_fs::pack_dir` shipped
`target/` in both directions. `mission_delivery` has excluded build output from the
DIFF since day one; the TRANSPORT never knew. The host checkout was 9.4 MB of which
8.9 MB was `target/`, tarred and base64'd over vsock each way. `EXCLUDED_PATHS` is
now one list shared by both layers, matched on directory name at any depth so a
workspace's per-crate `target/` dirs are all covered.

483 tests pass, clippy clean.
This commit is contained in:
Omar Sobh
2026-08-06 09:01:04 -07:00
parent 0d25a94a84
commit 4efcde9d4f
7 changed files with 200 additions and 20 deletions
+11
View File
@@ -80,10 +80,21 @@ async fn reap_stuck_runs(pool: &PgPool) -> Result<(), sqlx::Error> {
FROM topology_runs
WHERE status = 'running'
AND mission_id IS NOT NULL
-- Only jobs this worker drives. mission_id IS NOT NULL used to mean
-- the same thing as orchestrator-driven, and the microvm and session
-- tiers broke that: their checkpoint is NULL for life BY DESIGN, so the
-- zero-step-records test below is true of a perfectly healthy run.
AND tier = ANY($2)
AND created_at < now() - make_interval(secs => $1::float)
AND coalesce(jsonb_array_length(coalesce(checkpoint->'records', '[]'::jsonb)), 0) = 0",
)
.bind(REAP_STUCK_AFTER_SECS as f64)
.bind(
cm_db::repo::topology_runs::WORKER_DRIVEN_TIERS
.iter()
.map(|s| (*s).to_string())
.collect::<Vec<_>>(),
)
.fetch_all(pool)
.await?;