fix(missions): a failed phase stranded its mission at running forever

Found by counting containers during a cleanup, not by a test. gw-04 was holding
a per-mission runtime container for a mission whose only topology run had failed
three days earlier — phases `pending,failed`, mission still `running`.

The interaction, which lived entirely between two queries' predicates:
`start_pending_phases` launches a phase only when EVERY lower-order phase is
`completed`, so once one fails the phases after it can never run. They stayed
`pending`. `close_finished_missions` closes a mission only when NO phase is
outside ('completed','failed','skipped') — so a `pending` phase that would never
run kept the mission `running` indefinitely. And `mission_runtime`'s sweeper
fires N minutes after a TERMINAL state, so the container was never reaped.

One leaked container per failed multi-phase mission, accumulating silently, with
nothing in any log saying so. Neither query is wrong alone; the bug is that
nothing marked the phases the failure had made unreachable.

`skip_unreachable_phases` says it: a `pending` phase with a `failed` phase at a
LOWER order_idx becomes `skipped` — strictly earlier, because order is what makes
a phase unreachable, and a failure later in the list says nothing about one still
queued ahead of it. `skipped` is not a new concept: `close_finished_missions`
already treats it as terminal, and it is the honest word for a phase that was
never run, as distinct from one that failed.

RETRY HAD TO MOVE WITH IT, or this trades one bug for another. `retry_phase`
required the mission to be `running`, so closing failed missions would have made
the one outcome you would actually want to retry the one you could not. It now
accepts `failed` too, and in one transaction: resets the phase, REOPENS the
phases its failure had skipped (without that, a retry runs the phase and stops,
because everything after it is terminal-by-skip), and puts the mission back to
`running` — every launcher and closer keys off that status. `completed` and
`cancelled` stay refused; reopening those is a different decision.

557 tests pass, clippy clean. Three DB tests against real SQL, including that a
phase queued BEFORE the failure is untouched and that a draft's phases are never
swept.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-07 06:58:26 -07:00
co-authored by Claude Opus 5
parent bf2055e725
commit d24823b6f3
3 changed files with 246 additions and 3 deletions
+39 -3
View File
@@ -701,9 +701,16 @@ pub async fn retry_phase(
let mission = cm_db::repo::missions::get(&state.pool, id, user.workspace_id.as_uuid())
.await?
.ok_or(ApiError::NotFound)?;
if mission.status != "running" {
// `failed` is retryable, and has to be: a failed phase now closes its
// mission (its later phases are marked unreachable so the mission can
// finish at all), so refusing anything but `running` would mean the one
// outcome you would actually want to retry is the one you cannot.
// `completed` and `cancelled` stay refused — reopening those is a different
// decision than re-running a phase that failed.
if mission.status != "running" && mission.status != "failed" {
return Err(ApiError::BadRequest);
}
let mut tx = state.pool.begin().await?;
let r = sqlx::query(
"UPDATE mission_phases
SET status = 'pending', started_at = NULL, completed_at = NULL
@@ -712,12 +719,41 @@ pub async fn retry_phase(
)
.bind(phase_id)
.bind(id)
.execute(&state.pool)
.execute(&mut *tx)
.await?;
if r.rows_affected() == 0 {
tx.rollback().await?;
return Err(ApiError::NotFound);
}
Ok(Json(serde_json::json!({ "reset": true })))
// Reopen the phases this one's failure had made unreachable. Without this a
// retry runs the failed phase and then stops, because everything after it
// is terminal-by-skip — the mission would close again the moment this phase
// finished, having done only part of the work.
let reopened = sqlx::query(
"UPDATE mission_phases mp
SET status = 'pending', started_at = NULL, completed_at = NULL
WHERE mp.mission_id = $1
AND mp.status = 'skipped'
AND mp.order_idx > (SELECT order_idx FROM mission_phases WHERE id = $2)",
)
.bind(id)
.bind(phase_id)
.execute(&mut *tx)
.await?
.rows_affected();
// And put the mission back to running, or nothing sweeps the phase: every
// launcher and closer keys off `missions.status = 'running'`.
sqlx::query(
"UPDATE missions SET status = 'running', completed_at = NULL, updated_at = now()
WHERE id = $1 AND status = 'failed'",
)
.bind(id)
.execute(&mut *tx)
.await?;
tx.commit().await?;
Ok(Json(
serde_json::json!({ "reset": true, "reopened_phases": reopened }),
))
}
/// GET /api/missions/{id}/phases/{phase_id}/summary — the completion