fix(missions): a failed phase stranded its mission at running forever
Found by counting containers during a cleanup, not by a test. gw-04 was holding
a per-mission runtime container for a mission whose only topology run had failed
three days earlier — phases `pending,failed`, mission still `running`.
The interaction, which lived entirely between two queries' predicates:
`start_pending_phases` launches a phase only when EVERY lower-order phase is
`completed`, so once one fails the phases after it can never run. They stayed
`pending`. `close_finished_missions` closes a mission only when NO phase is
outside ('completed','failed','skipped') — so a `pending` phase that would never
run kept the mission `running` indefinitely. And `mission_runtime`'s sweeper
fires N minutes after a TERMINAL state, so the container was never reaped.
One leaked container per failed multi-phase mission, accumulating silently, with
nothing in any log saying so. Neither query is wrong alone; the bug is that
nothing marked the phases the failure had made unreachable.
`skip_unreachable_phases` says it: a `pending` phase with a `failed` phase at a
LOWER order_idx becomes `skipped` — strictly earlier, because order is what makes
a phase unreachable, and a failure later in the list says nothing about one still
queued ahead of it. `skipped` is not a new concept: `close_finished_missions`
already treats it as terminal, and it is the honest word for a phase that was
never run, as distinct from one that failed.
RETRY HAD TO MOVE WITH IT, or this trades one bug for another. `retry_phase`
required the mission to be `running`, so closing failed missions would have made
the one outcome you would actually want to retry the one you could not. It now
accepts `failed` too, and in one transaction: resets the phase, REOPENS the
phases its failure had skipped (without that, a retry runs the phase and stops,
because everything after it is terminal-by-skip), and puts the mission back to
`running` — every launcher and closer keys off that status. `completed` and
`cancelled` stay refused; reopening those is a different decision.
557 tests pass, clippy clean. Three DB tests against real SQL, including that a
phase queued BEFORE the failure is untouched and that a draft's phases are never
swept.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
bf2055e725
commit
d24823b6f3
@@ -701,9 +701,16 @@ pub async fn retry_phase(
|
||||
let mission = cm_db::repo::missions::get(&state.pool, id, user.workspace_id.as_uuid())
|
||||
.await?
|
||||
.ok_or(ApiError::NotFound)?;
|
||||
if mission.status != "running" {
|
||||
// `failed` is retryable, and has to be: a failed phase now closes its
|
||||
// mission (its later phases are marked unreachable so the mission can
|
||||
// finish at all), so refusing anything but `running` would mean the one
|
||||
// outcome you would actually want to retry is the one you cannot.
|
||||
// `completed` and `cancelled` stay refused — reopening those is a different
|
||||
// decision than re-running a phase that failed.
|
||||
if mission.status != "running" && mission.status != "failed" {
|
||||
return Err(ApiError::BadRequest);
|
||||
}
|
||||
let mut tx = state.pool.begin().await?;
|
||||
let r = sqlx::query(
|
||||
"UPDATE mission_phases
|
||||
SET status = 'pending', started_at = NULL, completed_at = NULL
|
||||
@@ -712,12 +719,41 @@ pub async fn retry_phase(
|
||||
)
|
||||
.bind(phase_id)
|
||||
.bind(id)
|
||||
.execute(&state.pool)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
if r.rows_affected() == 0 {
|
||||
tx.rollback().await?;
|
||||
return Err(ApiError::NotFound);
|
||||
}
|
||||
Ok(Json(serde_json::json!({ "reset": true })))
|
||||
// Reopen the phases this one's failure had made unreachable. Without this a
|
||||
// retry runs the failed phase and then stops, because everything after it
|
||||
// is terminal-by-skip — the mission would close again the moment this phase
|
||||
// finished, having done only part of the work.
|
||||
let reopened = sqlx::query(
|
||||
"UPDATE mission_phases mp
|
||||
SET status = 'pending', started_at = NULL, completed_at = NULL
|
||||
WHERE mp.mission_id = $1
|
||||
AND mp.status = 'skipped'
|
||||
AND mp.order_idx > (SELECT order_idx FROM mission_phases WHERE id = $2)",
|
||||
)
|
||||
.bind(id)
|
||||
.bind(phase_id)
|
||||
.execute(&mut *tx)
|
||||
.await?
|
||||
.rows_affected();
|
||||
// And put the mission back to running, or nothing sweeps the phase: every
|
||||
// launcher and closer keys off `missions.status = 'running'`.
|
||||
sqlx::query(
|
||||
"UPDATE missions SET status = 'running', completed_at = NULL, updated_at = now()
|
||||
WHERE id = $1 AND status = 'failed'",
|
||||
)
|
||||
.bind(id)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
tx.commit().await?;
|
||||
Ok(Json(
|
||||
serde_json::json!({ "reset": true, "reopened_phases": reopened }),
|
||||
))
|
||||
}
|
||||
|
||||
/// GET /api/missions/{id}/phases/{phase_id}/summary — the completion
|
||||
|
||||
Reference in New Issue
Block a user