fix(missions): harvest before the checkout, and stop an unreachable judge failing done work

Two bugs from the first real Continuous Research run, both found by running it.

**1. The harvest ran AFTER the checkout.** `on_launch` cloned the vault and then
harvested, so the mission's working copy predated the manifest push. The reader
agent found no `harvest.jsonl` and — being resourceful — queried arXiv itself
and wrote its own. That is exactly what `skills/research/arxiv-daily.md`
forbids: the papers it found are not checked off in `corpus_items`, so the next
run re-offers them, while the 13 the real harvest DID shelve went unread. The
harvest now runs first, so the clone contains the manifest.

The analysis it produced was otherwise very good — it named
`crates/clawhdf5-ann/src/hnsw.rs`, cited the ROADMAP's serial insert loop and
proposed a concrete pre-build probe — which is the behaviour the whole design
is for. It was reading the wrong papers.

**2. An unreachable judge consumed a pass.** `Verdict.error` exists to
distinguish "could not judge" from "judged incomplete" and nothing acted on it.
glm-5.3 returned "transport error: error decoding response body", the phase
counted it as a failed pass, and with two budgeted that single outage failed a
phase whose work was done and committed. The evaluator was right to refuse a
same-family fallback — that would trade independence for availability — so the
fix belongs here: an unreachable judge no longer spends an iteration.

Retrying forever would trade a wrong failure for an invisible hang, so the wait
is bounded by `judge_blocked_since` (migration 0078), mirroring how
`capacity_blocked_since` bounds a phase waiting on a VM slot. Thirty minutes is
many sweep ticks, so a blip recovers inside it; past that the phase FAILS with
the transport reason rather than requeueing, because re-running spends a
container re-doing work that was never the problem.

346 tests pass.

Co-Authored-By: Claude Opus 5 <[email protected]>

crates/cm-api/src/mission_orchestrator.rs

@@ -54,25 +54,16 @@ pub async fn on_launch(
         return Err("mission not found".into());
     };
 
-    // ensure_checkout is idempotent (fetch+reset on existing clones,
+    // A Continuous Research mission harvests BEFORE its checkout is taken.
-    // clone on missing dirs) so we run it BEFORE the team_id short-
+    //
-    // circuit: a re-launched or retried mission still needs a fresh
+    // The ORDER here is load-bearing and was wrong: the harvest ran after
-    // repo checkout even though its team was minted on the first
+    // `ensure_checkout`, so the mission cloned the vault before the manifest
-    // launch. Non-fatal — logs and continues on failure.
+    // was pushed to it. The reader agent found no harvest.jsonl, and — being
-    match crate::mission_workspace::ensure_checkout(pool, workspace_id, mission_id).await {
+    // resourceful — queried arXiv itself and wrote its own. That is precisely
-        Ok(Some(path)) => eprintln!(
+    // what `skills/research/arxiv-daily.md` forbids: the papers it found are
-            "mission_orchestrator: repo checked out at {} for mission {mission_id}",
+    // not checked off in `corpus_items`, so the next run re-offers them, and
-            path.display()
+    // the 13 the real harvest DID shelve went unread. Harvest first, then
-        ),
+    // clone, so the checkout contains the manifest.
-        Ok(None) => eprintln!(
-            "mission_orchestrator: mission {mission_id} has no repo bound, skipping checkout"
-        ),
-        Err(e) => eprintln!(
-            "mission_orchestrator: repo checkout for {mission_id} failed (continuing): {e}"
-        ),
-    }
-
-    // A Continuous Research mission harvests BEFORE its agents start.
     //
     // Finding papers is not agent work: `library::run_to_vault` searches arXiv,
     // checks the `corpus_items` seen-set, fetches and verifies each PDF, shelves
@@ -115,6 +106,25 @@ pub async fn on_launch(
         }
     }
 
+
+    // ensure_checkout is idempotent (fetch+reset on existing clones,
+    // clone on missing dirs) so we run it BEFORE the team_id short-
+    // circuit: a re-launched or retried mission still needs a fresh
+    // repo checkout even though its team was minted on the first
+    // launch. Non-fatal — logs and continues on failure.
+    match crate::mission_workspace::ensure_checkout(pool, workspace_id, mission_id).await {
+        Ok(Some(path)) => eprintln!(
+            "mission_orchestrator: repo checked out at {} for mission {mission_id}",
+            path.display()
+        ),
+        Ok(None) => eprintln!(
+            "mission_orchestrator: mission {mission_id} has no repo bound, skipping checkout"
+        ),
+        Err(e) => eprintln!(
+            "mission_orchestrator: repo checkout for {mission_id} failed (continuing): {e}"
+        ),
+    }
+
     // Provision the per-mission ZeroClaw runtime container (C3).
     // Idempotent: returns the endpoint if the container is already
     // running. Falls back silently when docker is unreachable so

crates/cm-api/src/phase_runner.rs

@@ -409,6 +409,15 @@ const PRODUCING_KINDS: &[&str] = &["coding", "research", "benchmark", "security_
 /// so rather than sit `pending` forever looking like a bug.
 const CAPACITY_WAIT_MAX_SECS: f64 = 2.0 * 3600.0;
 
+/// How long a phase may wait for a judge it cannot reach before it is failed.
+///
+/// Not consuming a pass for an unreachable judge is right; retrying forever is
+/// not, because a permanently dead validator would leave the phase `evaluating`
+/// in silence — a wrong failure traded for an invisible hang. Thirty minutes is
+/// many sweep ticks, so a transient outage recovers well inside it, and a real
+/// one surfaces as a failure that names the transport error.
+const JUDGE_WAIT_MAX_SECS: f64 = 30.0 * 60.0;
+
 /// Stamp why a phase is waiting, returning how long it has waited so far.
 ///
 /// The timestamp is set once and preserved across retries, so the wait is
@@ -1880,8 +1889,70 @@ async fn evaluate_finished_phases(
             eprintln!("phase_runner: recording evaluation for {phase_id} failed: {e}");
         }
 
+        // A judge that could not be REACHED has not judged. `Verdict.error` is
+        // set only when the evaluator itself failed — "could not judge" as
+        // distinct from "judged incomplete" — and spending one of the phase's
+        // passes on it charges the agent for an outage it had no part in.
+        //
+        // Mission 01a011bf lost its script phase exactly this way: the work was
+        // done and committed, glm-5.3 returned "transport error: error decoding
+        // response body", and with two passes budgeted that single unreachable
+        // judge was enough to fail the phase. Leave it `evaluating` so the next
+        // sweep re-judges the same pass; the phase is not re-run, only re-read.
+        //
+        // Deliberately NOT a fallback to the agent's own provider: `evaluate`
+        // already refuses that, because a judge from the same family is not an
+        // independent check and quietly becoming one is worse than waiting.
+        let mut judge_gave_up = false;
+        if let Some(why) = verdict.error.as_deref() {
+            let blocked_for: Option<f64> = sqlx::query_scalar(
+                "UPDATE mission_phases
+                    SET judge_blocked_since = COALESCE(judge_blocked_since, now())
+                  WHERE id = $1
+              RETURNING EXTRACT(EPOCH FROM now() - judge_blocked_since)::float8",
+            )
+            .bind(phase_id)
+            .fetch_optional(pool)
+            .await
+            .map_err(|e| format!("mark judge-blocked {phase_id}: {e}"))?
+            .flatten();
+
+            if blocked_for.unwrap_or(0.0) < JUDGE_WAIT_MAX_SECS {
+                eprintln!(
+                    "phase_runner: phase {phase_id} ({kind}) — judge unreachable ({why}); \
+                     leaving it evaluating so the next sweep retries. Pass {} of {} NOT \
+                     consumed; blocked {:.0}s of {JUDGE_WAIT_MAX_SECS:.0}s.",
+                    iteration + 1,
+                    max_iterations,
+                    blocked_for.unwrap_or(0.0)
+                );
+                continue;
+            }
+            // Waited long enough. Fail with the transport reason rather than
+            // sitting `evaluating` forever — an invisible hang is worse than an
+            // honest failure that names what could not be reached.
+            eprintln!(
+                "phase_runner: phase {phase_id} ({kind}) — judge unreachable for {:.0}s, \
+                 giving up: {why}",
+                blocked_for.unwrap_or(0.0)
+            );
+            // Fail NOW rather than requeueing. Re-running the phase would spend
+            // a container and a model budget re-doing work that was never the
+            // problem — the judge was.
+            judge_gave_up = true;
+        } else {
+            // A real verdict landed: stop the clock.
+            let _ = sqlx::query(
+                "UPDATE mission_phases SET judge_blocked_since = NULL
+                  WHERE id = $1 AND judge_blocked_since IS NOT NULL",
+            )
+            .bind(phase_id)
+            .execute(pool)
+            .await;
+        }
+
         let last_pass = iteration + 1 >= max_iterations;
-        if verdict.met || last_pass {
+        if verdict.met || last_pass || judge_gave_up {
             // A phase that ran out of passes WITHOUT meeting its condition did not
             // succeed, and must not say it did. This used to mark both outcomes
             // `completed`: the verdict recorded met=false while the phase — and
@@ -2136,6 +2207,48 @@ fn apply_node_agents(
 
 #[cfg(test)]
 mod tests {
+
+    /// An unreachable judge must not spend the phase's iteration budget, and
+    /// must not retry forever either. Both halves are the property: the first
+    /// stops an outage failing work that was done, the second stops a dead
+    /// validator leaving a phase `evaluating` in silence.
+    #[test]
+    fn the_judge_wait_is_bounded_and_shorter_than_the_capacity_wait() {
+        assert!(
+            JUDGE_WAIT_MAX_SECS > 0.0,
+            "a zero wait would fail on the first transient error"
+        );
+        assert!(
+            JUDGE_WAIT_MAX_SECS < CAPACITY_WAIT_MAX_SECS,
+            "waiting on a judge is cheaper to abandon than waiting on a VM slot"
+        );
+        // Many sweep ticks, so a blip recovers well inside the window.
+        assert!(
+            JUDGE_WAIT_MAX_SECS >= 10.0 * 60.0,
+            "too short and a normal provider blip fails the phase"
+        );
+    }
+
+    /// The give-up path must FAIL, never requeue: re-running the phase spends a
+    /// container and a model budget re-doing work that was never the problem.
+    #[test]
+    fn giving_up_on_the_judge_closes_the_phase() {
+        let src = include_str!("phase_runner.rs");
+        let block = src
+            .split("let mut judge_gave_up = false;")
+            .nth(1)
+            .expect("the guard exists");
+        let head = &block[..block.find("let last_pass").unwrap_or(block.len())];
+        assert!(
+            head.contains("judge_gave_up = true;"),
+            "the timeout branch must set the flag"
+        );
+        assert!(
+            block.contains("verdict.met || last_pass || judge_gave_up"),
+            "the flag must reach the close decision, or it requeues instead"
+        );
+    }
+
     use super::*;
 
     fn by_node(pairs: &[(&str, Uuid)]) -> std::collections::HashMap<String, Uuid> {

migrations/0078_phase_judge_blocked.sql

@@ -0,0 +1,22 @@
+-- How long a phase has been waiting on a judge it cannot reach.
+--
+-- `Verdict.error` distinguishes "could not judge" from "judged incomplete", and
+-- until now nothing acted on it: an unreachable judge counted as a failed pass,
+-- so an outage on the validator's side spent the phase's iteration budget.
+-- Mission 01a011bf lost its script phase that way — the work was done and
+-- committed, glm-5.3 returned "transport error: error decoding response body",
+-- and with two passes budgeted that one unreachable judge failed the phase.
+--
+-- Not consuming the pass is right, but it cannot mean retrying forever: a
+-- permanently unreachable judge would leave the phase `evaluating` in silence,
+-- which trades a wrong failure for an invisible hang. This column is the clock
+-- that bounds the wait, exactly as `capacity_blocked_since` bounds a phase
+-- waiting for a VM slot.
+--
+-- Set on the first unreachable verdict, cleared the moment a real verdict
+-- lands. NULL therefore means "not currently blocked", not "never was".
+ALTER TABLE mission_phases
+    ADD COLUMN IF NOT EXISTS judge_blocked_since TIMESTAMPTZ;
+
+COMMENT ON COLUMN mission_phases.judge_blocked_since IS
+    'When this phase first got an unreachable-judge verdict. Cleared when a real verdict lands; bounds how long an evaluating phase may wait before it is failed with the transport reason.';