fix(judge): stop asking an exhausted plan the same question 180 times

The retry ran on the sweep's own 10s tick for a 30-minute window, so a phase
whose judge was unreachable re-judged up to 180 times. A verdict is not one
request either: `evaluator` is agentic and loops up to `MAX_TOOL_CALLS + 1`
rounds, resending the whole growing history each time, against evidence the
code's own comment sizes at ~120 KB. One unjudgeable phase could therefore
issue on the order of 2,000 model requests.

That is most of why the z.ai weekly plan kept emptying with no mission having
visibly done anything expensive — twice now, 2026-08-29 and 2026-09-09. Nothing
recorded it, because `usage_events` carries no provider or model column.

Two changes:

Read the error before retrying. z.ai answers an exhausted plan with a 429
carrying code 1310 and its own reset timestamp. Retrying that is arithmetic,
not optimism: the reset was two days out and the phase spent its whole window
asking anyway. It now fails immediately and says which problem this is —
"the judge provider's plan limit is exhausted until 2026-09-11 10:01:33" sends
you to the plan, where "the independent validator could not be reached" sent
you into the mission. The classifier is deliberately conservative; anything
that does not positively identify itself as an exhausted plan stays retryable,
because giving up on a transport blip costs a phase that did nothing wrong —
which is how mission 01a011bf lost its script phase.

Back off. Waiting as long as we have already waited doubles total elapsed per
attempt, so the schedule is exponential with no attempt counter to store:
10, 20, 40, 80, 160, 300, 300 … — about ten attempts in the same window instead
of a hundred and eighty. `judge_retry_after` holds the clock and the sweep's
SELECT honours it; a landed verdict clears it alongside `judge_blocked_since`.

Verified rather than asserted: the migration applies and rolls back against a
real postgres, and replacing the backoff with the old fixed tick makes
`the_backoff_is_exponential_and_capped` fail (181 attempts, not ~10).

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz

crates/cm-api/src/phase_runner.rs

@@ -740,6 +740,59 @@ const CAPACITY_WAIT_MAX_SECS: f64 = 2.0 * 3600.0;
 /// one surfaces as a failure that names the transport error.
 const JUDGE_WAIT_MAX_SECS: f64 = 30.0 * 60.0;
 
+/// Longest gap between two judge attempts on the same phase.
+///
+/// The retry used to run on the sweep's own 10s tick, so a phase whose judge
+/// was unreachable re-judged 180 times in its 30-minute window. A verdict is
+/// not one request — [`crate::evaluator`] loops up to `MAX_TOOL_CALLS + 1`
+/// times and resends the whole growing history each round — so that was on the
+/// order of 2,000 model requests for one phase nobody could judge, and on a
+/// transport error they are billed: the request was processed, only its
+/// response failed to decode.
+const JUDGE_RETRY_MAX_BACKOFF_SECS: f64 = 5.0 * 60.0;
+
+/// How long to wait before the next judge attempt, given how long this phase
+/// has already been blocked.
+///
+/// Waiting as long as we have already waited doubles the total elapsed time per
+/// attempt, so the schedule is exponential without storing an attempt counter:
+/// 10, 20, 40, 80, 160, 300, 300 … — about ten attempts across the same
+/// 30-minute window instead of a hundred and eighty.
+fn judge_backoff_secs(blocked_for: f64) -> f64 {
+    blocked_for.clamp(POLL_INTERVAL.as_secs_f64(), JUDGE_RETRY_MAX_BACKOFF_SECS)
+}
+
+/// A judge error that retrying cannot fix before a time the error itself names.
+///
+/// z.ai answers an exhausted plan with a 429 carrying code `1310` and its own
+/// reset timestamp. Retrying that is not optimism, it is arithmetic: the reset
+/// was two days out when this fired on 2026-09-09, and the phase spent its full
+/// 30-minute window asking a question whose answer could not change. Fail
+/// immediately instead, and say what is actually wrong — "quota exhausted until
+/// X" sends you to the plan, where "the independent validator could not be
+/// reached" sends you into the mission.
+///
+/// Conservative on purpose: an error that does not positively identify itself
+/// as an exhausted plan is treated as retryable, because giving up on a
+/// transient blip costs a phase that had done nothing wrong.
+fn judge_error_is_exhausted_plan(why: &str) -> Option<String> {
+    if !(why.contains("Limit Exhausted") || why.contains(r#""code":"1310""#)) {
+        return None;
+    }
+    let until: Option<String> = why.find("reset at ").map(|i| {
+        why[i + "reset at ".len()..]
+            .chars()
+            .take_while(|c| *c != ']' && *c != '"')
+            .collect::<String>()
+            .trim()
+            .to_string()
+    });
+    Some(match until.filter(|u| !u.is_empty()) {
+        Some(u) => format!("the judge provider's plan limit is exhausted until {u}"),
+        None => "the judge provider's plan limit is exhausted".to_string(),
+    })
+}
+
 /// Stamp why a phase is waiting, returning how long it has waited so far.
 ///
 /// The timestamp is set once and preserved across retries, so the wait is
@@ -2413,6 +2466,7 @@ async fn evaluate_finished_phases(
            FROM mission_phases mp
            JOIN missions m ON m.id = mp.mission_id
           WHERE mp.status = 'evaluating' AND m.status = 'running'
+            AND (mp.judge_retry_after IS NULL OR mp.judge_retry_after <= now())
           LIMIT 5",
     )
     .fetch_all(pool)
@@ -2503,34 +2557,59 @@ async fn evaluate_finished_phases(
             .map_err(|e| format!("mark judge-blocked {phase_id}: {e}"))?
             .flatten();
 
-            if blocked_for.unwrap_or(0.0) < JUDGE_WAIT_MAX_SECS {
+            // An exhausted plan names the time it resets. Waiting cannot
+            // reach it, so stop now rather than spending the window — and say
+            // which of the two very different problems this is.
+            if let Some(plain) = judge_error_is_exhausted_plan(why) {
+                eprintln!(
+                    "phase_runner: phase {phase_id} ({kind}) — NOT retrying: {plain}. \
+                     Pass {} of {} NOT consumed; the agent's work is untouched and the \
+                     phase is failing on the judge, not on itself.",
+                    iteration + 1,
+                    max_iterations,
+                );
+                judge_gave_up = true;
+            } else if blocked_for.unwrap_or(0.0) < JUDGE_WAIT_MAX_SECS {
+                let wait = judge_backoff_secs(blocked_for.unwrap_or(0.0));
+                let _ = sqlx::query(
+                    "UPDATE mission_phases
+                        SET judge_retry_after = now() + make_interval(secs => $2)
+                      WHERE id = $1",
+                )
+                .bind(phase_id)
+                .bind(wait)
+                .execute(pool)
+                .await;
                 eprintln!(
                     "phase_runner: phase {phase_id} ({kind}) — judge unreachable ({why}); \
-                     leaving it evaluating so the next sweep retries. Pass {} of {} NOT \
-                     consumed; blocked {:.0}s of {JUDGE_WAIT_MAX_SECS:.0}s.",
+                     retrying in {wait:.0}s. Pass {} of {} NOT consumed; blocked {:.0}s of \
+                     {JUDGE_WAIT_MAX_SECS:.0}s.",
                     iteration + 1,
                     max_iterations,
                     blocked_for.unwrap_or(0.0)
                 );
                 continue;
+            } else {
+                // Waited long enough. Fail with the transport reason rather
+                // than sitting `evaluating` forever — an invisible hang is
+                // worse than an honest failure that names what could not be
+                // reached.
+                eprintln!(
+                    "phase_runner: phase {phase_id} ({kind}) — judge unreachable for {:.0}s, \
+                     giving up: {why}",
+                    blocked_for.unwrap_or(0.0)
+                );
+                // Fail NOW rather than requeueing. Re-running the phase would
+                // spend a container and a model budget re-doing work that was
+                // never the problem — the judge was.
+                judge_gave_up = true;
             }
-            // Waited long enough. Fail with the transport reason rather than
-            // sitting `evaluating` forever — an invisible hang is worse than an
-            // honest failure that names what could not be reached.
-            eprintln!(
-                "phase_runner: phase {phase_id} ({kind}) — judge unreachable for {:.0}s, \
-                 giving up: {why}",
-                blocked_for.unwrap_or(0.0)
-            );
-            // Fail NOW rather than requeueing. Re-running the phase would spend
-            // a container and a model budget re-doing work that was never the
-            // problem — the judge was.
-            judge_gave_up = true;
         } else {
             // A real verdict landed: stop the clock.
             let _ = sqlx::query(
-                "UPDATE mission_phases SET judge_blocked_since = NULL
-                  WHERE id = $1 AND judge_blocked_since IS NOT NULL",
+                "UPDATE mission_phases SET judge_blocked_since = NULL, judge_retry_after = NULL
+                  WHERE id = $1 AND (judge_blocked_since IS NOT NULL
+                                     OR judge_retry_after IS NOT NULL)",
             )
             .bind(phase_id)
             .execute(pool)
@@ -2817,6 +2896,56 @@ mod tests {
 
     /// The give-up path must FAIL, never requeue: re-running the phase spends a
     /// container and a model budget re-doing work that was never the problem.
+    /// The real 429 z.ai returns for an exhausted plan. Retrying it is not
+    /// optimism, it is arithmetic: on 2026-09-09 the reset was two days out and
+    /// the phase spent its whole 30-minute window asking anyway.
+    #[test]
+    fn an_exhausted_plan_is_recognised_and_names_its_reset() {
+        let why = r#"provider returned an error: 429 Too Many Requests: {"type":"error","error":{"type":"rate_limit_error","code":"1310","message":"[1310][Weekly/Monthly Limit Exhausted. Your limit will reset at 2026-09-11 10:01:33][2026090911555755ef730ec0404849]"}}"#;
+        let plain = super::judge_error_is_exhausted_plan(why).expect("recognised");
+        assert!(plain.contains("2026-09-11 10:01:33"), "{plain}");
+        assert!(plain.contains("exhausted"), "{plain}");
+    }
+
+    /// Conservative by design. A transport blip must stay retryable — giving up
+    /// on one costs a phase that had done nothing wrong, which is the failure
+    /// mission 01a011bf actually suffered.
+    #[test]
+    fn a_transient_error_stays_retryable() {
+        assert!(super::judge_error_is_exhausted_plan(
+            "transport error: error decoding response body"
+        )
+        .is_none());
+        assert!(super::judge_error_is_exhausted_plan("429 Too Many Requests").is_none());
+        assert!(super::judge_error_is_exhausted_plan("").is_none());
+    }
+
+    /// Waiting as long as we have already waited doubles total elapsed per
+    /// attempt, so the window holds ~10 attempts instead of 180.
+    #[test]
+    fn the_backoff_is_exponential_and_capped() {
+        assert_eq!(super::judge_backoff_secs(0.0), 10.0, "first retry is one sweep");
+        assert_eq!(super::judge_backoff_secs(20.0), 20.0);
+        assert_eq!(super::judge_backoff_secs(160.0), 160.0);
+        assert_eq!(
+            super::judge_backoff_secs(1_000.0),
+            super::JUDGE_RETRY_MAX_BACKOFF_SECS,
+            "capped, or a long outage stops retrying at all"
+        );
+
+        // Count the attempts the 30-minute window now allows.
+        let mut elapsed = 0.0f64;
+        let mut attempts = 1;
+        while elapsed < super::JUDGE_WAIT_MAX_SECS {
+            elapsed += super::judge_backoff_secs(elapsed);
+            attempts += 1;
+        }
+        assert!(
+            (5..=15).contains(&attempts),
+            "expected roughly ten attempts, got {attempts}"
+        );
+    }
+
     #[test]
     fn giving_up_on_the_judge_closes_the_phase() {
         let src = include_str!("phase_runner.rs");