fix(missions): #54 — the worker was killing live microVM runs at 180 seconds
My hypothesis in #54 was WRONG, and it was wrong because I built it on a bad measurement: `grep -c 'microvm phase'` returned 0, so I concluded the completion log never printed and blamed the 15-minute reaper. The line was there all along, at 14:17:45. The real cause is worse. `requeue_stale` has NO TIER FILTER. A microvm run's `updated_at` is written once at insert and never again — it is driven by a `tokio::spawn` that owns it start to finish, and nothing in `microvm_executor` writes `topology_runs`. So at 180s the sweeper declared a perfectly healthy run stale and flipped it to `queued`; `claim_next_queued` (no tier filter either) handed it to the worker; `run_job` tried to parse the microvm graph placeholder, which `TopologyGraph` cannot deserialize; and it failed the run with "missing or invalid graph". Mission 019fd43e: run created 14:11:16, mission failed ~14:14:46. 210 seconds — the 180s window plus a tick. The agent went on working and finished at 14:17:45 with three modules written, by which time the phase was already dead and the VM was orphaned. A firecracker process was still alive 1h37m later. THE UNCOMFORTABLE PART: every microVM mission that appeared to work this session did so only by finishing inside three minutes. The 90-second ones dodged this. The harness scenario dodges it. Nothing about that was visible. `WORKER_DRIVEN_TIERS` (team, company, org, swarm, compare) is now the allowlist for all three sweep paths — claim, requeue, reap. An allowlist rather than a denylist so the next self-driven tier is safe by default instead of exposed until someone remembers the file. `tier='session'` had exactly the same exposure and is covered too. A unit test asserts microvm and session are NOT in it, next to the code that inserts them. Two more fixes from the same wreckage: - `destroy` reported `killed: pgid.is_some()` — true whenever there was a pgid to signal, whether or not anything died. It now sends the signal, polls /proc for the group leader, retries, and reports what it OBSERVED; `signalled` keeps the old meaning so "nothing to kill" is distinguishable from "it would not die". - the run-status update is now guarded with `AND status <> 'cancelled'`. An operator cancelling is a decision; this task reporting an outcome minutes later is an observation, and it must not overwrite one with the other. And the root cause of the collect timeout itself: `mission_fs::pack_dir` shipped `target/` in both directions. `mission_delivery` has excluded build output from the DIFF since day one; the TRANSPORT never knew. The host checkout was 9.4 MB of which 8.9 MB was `target/`, tarred and base64'd over vsock each way. `EXCLUDED_PATHS` is now one list shared by both layers, matched on directory name at any depth so a workspace's per-crate `target/` dirs are all covered. 483 tests pass, clippy clean.
This commit is contained in:
@@ -203,32 +203,67 @@ pub async fn check_ephemeral_teardown(
|
||||
}))
|
||||
}
|
||||
|
||||
/// Tiers the topology worker drives, and therefore the only ones it may claim,
|
||||
/// requeue or reap.
|
||||
///
|
||||
/// **A load-bearing allowlist, not tidiness.** Both sweepers were written when
|
||||
/// every `running` row was a `cm_orchestrator` job that checkpointed after each
|
||||
/// step. `tier='microvm'` and `tier='session'` broke that assumption: they are
|
||||
/// inserted directly as `running` by `phase_runner`, driven by a `tokio::spawn`
|
||||
/// that owns them start to finish, and they never write `updated_at` or
|
||||
/// `checkpoint` while in flight.
|
||||
///
|
||||
/// Measured cost of the omission: `requeue_stale` flipped an in-flight microVM run
|
||||
/// to `queued` at 180s, `claim_next_queued` handed it to the worker, and the worker
|
||||
/// failed it with "missing or invalid graph" — a microvm run's graph is a
|
||||
/// placeholder `TopologyGraph` cannot parse. Mission 019fd43e died at 210 seconds
|
||||
/// with its agent still working and its VM orphaned. Every microVM mission that
|
||||
/// appeared to work did so only by finishing inside three minutes.
|
||||
///
|
||||
/// An allowlist rather than a denylist on purpose: the next self-driven tier is
|
||||
/// then safe by default, instead of exposed until someone remembers this file.
|
||||
pub const WORKER_DRIVEN_TIERS: &[&str] = &["team", "company", "org", "swarm", "compare"];
|
||||
|
||||
/// The allowlist as owned strings, for binding as `text[]`.
|
||||
fn worker_driven() -> Vec<String> {
|
||||
WORKER_DRIVEN_TIERS.iter().map(|s| (*s).to_string()).collect()
|
||||
}
|
||||
|
||||
/// Atomically claim the oldest queued job, flipping it to `running`. Uses
|
||||
/// `FOR UPDATE SKIP LOCKED` so multiple workers never claim the same job.
|
||||
/// Returns `None` when the queue is empty.
|
||||
pub async fn claim_next_queued(pool: &PgPool) -> Result<Option<ClaimedTopologyRun>, DbError> {
|
||||
let row = sqlx::query!(
|
||||
use sqlx::Row as _;
|
||||
// A runtime query rather than `query!` so the tier allowlist can be bound
|
||||
// without regenerating the offline metadata on a machine with no database.
|
||||
let row = sqlx::query(
|
||||
"UPDATE topology_runs
|
||||
SET status = 'running', started_at = COALESCE(started_at, now()), updated_at = now()
|
||||
WHERE id = (
|
||||
SELECT id FROM topology_runs
|
||||
WHERE status = 'queued'
|
||||
-- Defence in depth. Even if a self-driven row somehow reaches
|
||||
-- `queued`, the worker must not adopt a job it cannot execute:
|
||||
-- doing so is what turned a live microVM run into a
|
||||
-- missing-or-invalid-graph failure.
|
||||
AND tier = ANY($1)
|
||||
ORDER BY created_at
|
||||
FOR UPDATE SKIP LOCKED
|
||||
LIMIT 1
|
||||
)
|
||||
RETURNING id, workspace_id, task, graph, checkpoint, last_event_id, tier",
|
||||
)
|
||||
.bind(worker_driven())
|
||||
.fetch_optional(pool)
|
||||
.await?;
|
||||
Ok(row.map(|r| ClaimedTopologyRun {
|
||||
id: r.id,
|
||||
workspace_id: r.workspace_id,
|
||||
task: r.task,
|
||||
graph: r.graph,
|
||||
checkpoint: r.checkpoint,
|
||||
last_event_id: r.last_event_id,
|
||||
tier: r.tier,
|
||||
id: r.get("id"),
|
||||
workspace_id: r.get("workspace_id"),
|
||||
task: r.get("task"),
|
||||
graph: r.get("graph"),
|
||||
checkpoint: r.get("checkpoint"),
|
||||
last_event_id: r.get("last_event_id"),
|
||||
tier: r.get("tier"),
|
||||
}))
|
||||
}
|
||||
|
||||
@@ -325,12 +360,19 @@ pub async fn current_status(pool: &PgPool, id: Uuid) -> Result<Option<String>, D
|
||||
/// touch within `older_than_secs`). The next claim resumes them from checkpoint.
|
||||
/// Returns how many were requeued.
|
||||
pub async fn requeue_stale(pool: &PgPool, older_than_secs: f64) -> Result<u64, DbError> {
|
||||
let result = sqlx::query!(
|
||||
let result = sqlx::query(
|
||||
"UPDATE topology_runs
|
||||
SET status = 'queued', updated_at = now()
|
||||
WHERE status = 'running' AND updated_at < now() - make_interval(secs => $1)",
|
||||
older_than_secs,
|
||||
WHERE status = 'running'
|
||||
AND updated_at < now() - make_interval(secs => $1)
|
||||
-- Only jobs the WORKER drives. A self-driven run (microvm, session) is
|
||||
-- owned by its own task for its whole life and never touches
|
||||
-- `updated_at`, so without this every one of them looked stale after
|
||||
-- three minutes and was requeued out from under a live VM.
|
||||
AND tier = ANY($2)",
|
||||
)
|
||||
.bind(older_than_secs)
|
||||
.bind(worker_driven())
|
||||
.execute(pool)
|
||||
.await?;
|
||||
Ok(result.rows_affected())
|
||||
|
||||
Reference in New Issue
Block a user