Omar SobhandClaude Opus 5
8d6310f126
The retry ran on the sweep's own 10s tick for a 30-minute window, so a phase whose judge was unreachable re-judged up to 180 times. A verdict is not one request either: `evaluator` is agentic and loops up to `MAX_TOOL_CALLS + 1` rounds, resending the whole growing history each time, against evidence the code's own comment sizes at ~120 KB. One unjudgeable phase could therefore issue on the order of 2,000 model requests. That is most of why the z.ai weekly plan kept emptying with no mission having visibly done anything expensive — twice now, 2026-08-29 and 2026-09-09. Nothing recorded it, because `usage_events` carries no provider or model column. Two changes: Read the error before retrying. z.ai answers an exhausted plan with a 429 carrying code 1310 and its own reset timestamp. Retrying that is arithmetic, not optimism: the reset was two days out and the phase spent its whole window asking anyway. It now fails immediately and says which problem this is — "the judge provider's plan limit is exhausted until 2026-09-11 10:01:33" sends you to the plan, where "the independent validator could not be reached" sent you into the mission. The classifier is deliberately conservative; anything that does not positively identify itself as an exhausted plan stays retryable, because giving up on a transport blip costs a phase that did nothing wrong — which is how mission 01a011bf lost its script phase. Back off. Waiting as long as we have already waited doubles total elapsed per attempt, so the schedule is exponential with no attempt counter to store: 10, 20, 40, 80, 160, 300, 300 … — about ten attempts in the same window instead of a hundred and eighty. `judge_retry_after` holds the clock and the sweep's SELECT honours it; a landed verdict clears it alongside `judge_blocked_since`. Verified rather than asserted: the migration applies and rolls back against a real postgres, and replacing the backoff with the old fixed tick makes `the_backoff_is_exponential_and_capped` fail (181 attempts, not ~10). Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
21 lines
1.1 KiB
SQL
21 lines
1.1 KiB
SQL
-- When the judge may next be attempted for this phase.
|
|
--
|
|
-- The retry was a fixed 10s sweep for 30 minutes: up to 180 full re-evaluations
|
|
-- of one phase. A verdict is not one request either — the evaluator is agentic
|
|
-- and loops up to MAX_TOOL_CALLS + 1 times, resending the whole growing history
|
|
-- each round. So one unjudgeable phase could issue on the order of 2,000 model
|
|
-- requests, and a transport error (which is billed — the request was processed,
|
|
-- only its response failed to decode) issued them for real.
|
|
--
|
|
-- That is most of why the z.ai weekly plan kept emptying with no mission having
|
|
-- obviously done anything expensive. Nothing recorded it, because usage_events
|
|
-- carries no provider or model column.
|
|
--
|
|
-- NULL means "attempt on the next sweep", so existing rows and the first
|
|
-- attempt of every new phase behave exactly as before.
|
|
ALTER TABLE mission_phases
|
|
ADD COLUMN IF NOT EXISTS judge_retry_after TIMESTAMPTZ;
|
|
|
|
COMMENT ON COLUMN mission_phases.judge_retry_after IS
|
|
'Earliest next judge attempt; set on a retryable judge error, cleared on a verdict.';
|