fix(worker): the stuck-run reaper was killing healthy sonnet-5 turns
deploy / test (push) Successful in 5m14s
deploy / build (push) Successful in 5m42s

REAP_STUCK_AFTER_SECS was 15 minutes; the runtime grants a single turn
`timeout_secs = 3000` (50 minutes). A run journals its first step record when
its first step COMPLETES, so a turn still legitimately in flight is
indistinguishable from a wedged container — and with a window shorter than the
turn timeout the reaper does not detect stuck runs, it kills slow healthy ones.

The old value was calibrated on haiku, where "healthy first-step latency is
typically 5-60s" held. Moving mission agents to sonnet-5 made first turns
longer than the window: mission 01a00c41's research phase was reaped at 900s
having already written +402/-39 across 13 files. We only know it was healthy
because the delivery path captured and pushed that work anyway, to branch
clawmates/mission-01a00c41-421200ee at b08df7b6.

Raised to 60 minutes, above the turn timeout, with the invariant written down
so the next person changing either number sees the relationship.

Generalises: a liveness timeout calibrated against one model becomes a
correctness bug when the model changes.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-16 13:48:03 -07:00
co-authored by Claude Opus 5
parent b31a79f650
commit 99dd29cc8a
+18 -4
View File
@@ -26,10 +26,24 @@ use crate::topology_exec::ZeroClawDriveExecutor;
const STALE_AFTER_SECS: f64 = 180.0;
/// Maximum age a `running` run may spend WITHOUT journaling any step
/// records before the reaper kills its container and fails it. 15 min
/// is generous: healthy first-step latency is typically 560s; anything
/// past this is a stuck container (usually a wedged provider CLI).
const REAP_STUCK_AFTER_SECS: i64 = 15 * 60;
/// records before the reaper kills its container and fails it.
///
/// **This must stay LONGER than the runtime's per-turn timeout.** A run
/// journals its first record when its first step COMPLETES, so any turn still
/// legitimately in flight looks identical to a wedged container. The runtime
/// grants a turn `timeout_secs = 3000` (50 min), so a shorter reaper window
/// does not detect stuck runs — it kills healthy slow ones.
///
/// This was 15 minutes, chosen when "healthy first-step latency is typically
/// 560s" was true of the model in use. It was, on haiku. Moving the mission
/// agents to sonnet-5 made first turns longer than the window, and mission
/// 01a00c41's research phase was reaped at 900s having already written 402
/// lines across 13 files — work the delivery path then captured and pushed,
/// which is the only reason we could tell the run was healthy at all.
///
/// The lesson generalises past this constant: a liveness timeout calibrated
/// against one model silently becomes a correctness bug when the model changes.
const REAP_STUCK_AFTER_SECS: i64 = 60 * 60;
/// Spawn the durable topology job worker. Polls for queued jobs every `poll`
/// interval; runs each to completion (or failure), checkpointing per step.