fix(missions): #54 — the worker was killing live microVM runs at 180 seconds
My hypothesis in #54 was WRONG, and it was wrong because I built it on a bad measurement: `grep -c 'microvm phase'` returned 0, so I concluded the completion log never printed and blamed the 15-minute reaper. The line was there all along, at 14:17:45. The real cause is worse. `requeue_stale` has NO TIER FILTER. A microvm run's `updated_at` is written once at insert and never again — it is driven by a `tokio::spawn` that owns it start to finish, and nothing in `microvm_executor` writes `topology_runs`. So at 180s the sweeper declared a perfectly healthy run stale and flipped it to `queued`; `claim_next_queued` (no tier filter either) handed it to the worker; `run_job` tried to parse the microvm graph placeholder, which `TopologyGraph` cannot deserialize; and it failed the run with "missing or invalid graph". Mission 019fd43e: run created 14:11:16, mission failed ~14:14:46. 210 seconds — the 180s window plus a tick. The agent went on working and finished at 14:17:45 with three modules written, by which time the phase was already dead and the VM was orphaned. A firecracker process was still alive 1h37m later. THE UNCOMFORTABLE PART: every microVM mission that appeared to work this session did so only by finishing inside three minutes. The 90-second ones dodged this. The harness scenario dodges it. Nothing about that was visible. `WORKER_DRIVEN_TIERS` (team, company, org, swarm, compare) is now the allowlist for all three sweep paths — claim, requeue, reap. An allowlist rather than a denylist so the next self-driven tier is safe by default instead of exposed until someone remembers the file. `tier='session'` had exactly the same exposure and is covered too. A unit test asserts microvm and session are NOT in it, next to the code that inserts them. Two more fixes from the same wreckage: - `destroy` reported `killed: pgid.is_some()` — true whenever there was a pgid to signal, whether or not anything died. It now sends the signal, polls /proc for the group leader, retries, and reports what it OBSERVED; `signalled` keeps the old meaning so "nothing to kill" is distinguishable from "it would not die". - the run-status update is now guarded with `AND status <> 'cancelled'`. An operator cancelling is a decision; this task reporting an outcome minutes later is an observation, and it must not overwrite one with the other. And the root cause of the collect timeout itself: `mission_fs::pack_dir` shipped `target/` in both directions. `mission_delivery` has excluded build output from the DIFF since day one; the TRANSPORT never knew. The host checkout was 9.4 MB of which 8.9 MB was `target/`, tarred and base64'd over vsock each way. `EXCLUDED_PATHS` is now one list shared by both layers, matched on directory name at any depth so a workspace's per-crate `target/` dirs are all covered. 483 tests pass, clippy clean.
This commit is contained in:
@@ -475,17 +475,46 @@ pub async fn destroy(vms: &Vms, vm_id: &str) -> Result<Value, String> {
|
||||
(None, wd.clone(), uds, eg)
|
||||
}
|
||||
};
|
||||
// Killed means OBSERVED GONE, not asked-to-die.
|
||||
//
|
||||
// This used to report `killed: pgid.is_some()` — true whenever there was a
|
||||
// pgid to signal, whether or not anything died. A firecracker process was
|
||||
// found alive 1h37m after its VM was destroyed, holding a config file in a
|
||||
// workdir that no longer existed, while destroy had reported success. So the
|
||||
// signal is sent, the process is polled, and the answer is what was seen.
|
||||
let mut killed = false;
|
||||
if let Some(pgid) = pgid {
|
||||
kill_group(pgid).await;
|
||||
for attempt in 0..20 {
|
||||
kill_group(pgid).await;
|
||||
tokio::time::sleep(std::time::Duration::from_millis(100)).await;
|
||||
// The pgid is the group leader's pid, so /proc/<pgid> is the check.
|
||||
if !std::path::Path::new(&format!("/proc/{pgid}")).exists() {
|
||||
killed = true;
|
||||
break;
|
||||
}
|
||||
if attempt == 19 {
|
||||
eprintln!(
|
||||
"microvm {vm_id}: process group {pgid} SURVIVED {} kill attempts — \
|
||||
a firecracker process is still holding this VM's resources",
|
||||
attempt + 1
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
// Give the group a moment to die before removing the files it has open.
|
||||
tokio::time::sleep(std::time::Duration::from_millis(100)).await;
|
||||
let _ = tokio::fs::remove_file(&uds).await;
|
||||
// Same trap as firecracker's own socket: nothing unlinks these for us, and a
|
||||
// stale file makes the next bind fail with EADDRINUSE.
|
||||
let _ = tokio::fs::remove_file(&egress_uds).await;
|
||||
let removed = tokio::fs::remove_dir_all(&workdir).await.is_ok();
|
||||
Ok(json!({ "vm_id": vm_id, "killed": pgid.is_some(), "workdir_removed": removed }))
|
||||
// `killed` is now observed rather than assumed; `signalled` keeps the old
|
||||
// meaning so a caller can tell "there was nothing to kill" from
|
||||
// "we tried and it would not die".
|
||||
Ok(json!({
|
||||
"vm_id": vm_id,
|
||||
"killed": killed,
|
||||
"signalled": pgid.is_some(),
|
||||
"workdir_removed": removed,
|
||||
}))
|
||||
}
|
||||
|
||||
/// VMs this node currently holds, so the server can reap orphans.
|
||||
@@ -869,6 +898,9 @@ pub async fn selftest() -> bool {
|
||||
}
|
||||
|
||||
let r = destroy(&vms, id).await;
|
||||
// `killed` is now an OBSERVATION — the process was polled and found gone —
|
||||
// rather than "we sent a signal". A firecracker was found alive 1h37m after a
|
||||
// destroy that had reported success.
|
||||
check(
|
||||
r.as_ref()
|
||||
.map(|v| v["killed"] == json!(true) && v["workdir_removed"] == json!(true))
|
||||
|
||||
Reference in New Issue
Block a user