fix(worker): the stuck-run reaper was killing healthy sonnet-5 turns
REAP_STUCK_AFTER_SECS was 15 minutes; the runtime grants a single turn `timeout_secs = 3000` (50 minutes). A run journals its first step record when its first step COMPLETES, so a turn still legitimately in flight is indistinguishable from a wedged container — and with a window shorter than the turn timeout the reaper does not detect stuck runs, it kills slow healthy ones. The old value was calibrated on haiku, where "healthy first-step latency is typically 5-60s" held. Moving mission agents to sonnet-5 made first turns longer than the window: mission 01a00c41's research phase was reaped at 900s having already written +402/-39 across 13 files. We only know it was healthy because the delivery path captured and pushed that work anyway, to branch clawmates/mission-01a00c41-421200ee at b08df7b6. Raised to 60 minutes, above the turn timeout, with the invariant written down so the next person changing either number sees the relationship. Generalises: a liveness timeout calibrated against one model becomes a correctness bug when the model changes. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
b31a79f650
commit
99dd29cc8a
@@ -26,10 +26,24 @@ use crate::topology_exec::ZeroClawDriveExecutor;
|
||||
const STALE_AFTER_SECS: f64 = 180.0;
|
||||
|
||||
/// Maximum age a `running` run may spend WITHOUT journaling any step
|
||||
/// records before the reaper kills its container and fails it. 15 min
|
||||
/// is generous: healthy first-step latency is typically 5–60s; anything
|
||||
/// past this is a stuck container (usually a wedged provider CLI).
|
||||
const REAP_STUCK_AFTER_SECS: i64 = 15 * 60;
|
||||
/// records before the reaper kills its container and fails it.
|
||||
///
|
||||
/// **This must stay LONGER than the runtime's per-turn timeout.** A run
|
||||
/// journals its first record when its first step COMPLETES, so any turn still
|
||||
/// legitimately in flight looks identical to a wedged container. The runtime
|
||||
/// grants a turn `timeout_secs = 3000` (50 min), so a shorter reaper window
|
||||
/// does not detect stuck runs — it kills healthy slow ones.
|
||||
///
|
||||
/// This was 15 minutes, chosen when "healthy first-step latency is typically
|
||||
/// 5–60s" was true of the model in use. It was, on haiku. Moving the mission
|
||||
/// agents to sonnet-5 made first turns longer than the window, and mission
|
||||
/// 01a00c41's research phase was reaped at 900s having already written 402
|
||||
/// lines across 13 files — work the delivery path then captured and pushed,
|
||||
/// which is the only reason we could tell the run was healthy at all.
|
||||
///
|
||||
/// The lesson generalises past this constant: a liveness timeout calibrated
|
||||
/// against one model silently becomes a correctness bug when the model changes.
|
||||
const REAP_STUCK_AFTER_SECS: i64 = 60 * 60;
|
||||
|
||||
/// Spawn the durable topology job worker. Polls for queued jobs every `poll`
|
||||
/// interval; runs each to completion (or failure), checkpointing per step.
|
||||
|
||||
Reference in New Issue
Block a user