fix(missions): a server restart no longer kills a running mission

Mission 01a00538 ("ClawHDF5 REsearch and Refactor") failed 19 minutes and 93,762
tokens into its research phase with `pair failed: 403 Forbidden`, and its coding
phase was then correctly skipped as unreachable. The cause was not the coding
phase and not the model — it was pairing.

A per-mission runtime is authenticated with a SINGLE-USE pairing code, and the
bearer token it returns was cached in memory only. Any restart of the server
discarded that token; the next turn re-paired with a code the gateway had
already spent and got 403 — permanently, for that mission. A deploy, a crash or
an OOM would each do it. The durable-run machinery exists precisely so work
survives a restart; pairing was the one thread that did not, and it failed
closed.

`missions.runtime_token` persists the token at the moment pairing succeeds, and
the worker seeds the executor's cache from it, so a new process reuses the
credential instead of re-pairing. Persisting is best-effort: failing to save
must not fail a turn that just paired successfully.

Verified by reproducing the original failure: launched a mission, confirmed the
token was written, restarted the server MID-PHASE, and watched the mission run
to completion with no pairing failure.

Co-Authored-By: Claude Opus 5 <[email protected]>

crates/cm-api/src/topology_exec.rs

@@ -178,6 +178,21 @@ impl ZeroClawDriveExecutor {
     /// new one-time code at startup. The env-derived ZEROCLAW_TOKEN
     /// is ignored (belongs to the shared runtime) so the lazy pair
     /// path runs and issues a bearer for this specific gateway.
+    /// Reuse a token that was already paired and persisted.
+    ///
+    /// The pairing code is single-use, so a restarted server cannot pair again:
+    /// it gets 403 and the mission is unrecoverable. Seeding the cache from
+    /// `missions.runtime_token` is what makes a mission survive a restart.
+    pub fn with_token(self, token: Option<String>) -> Self {
+        if let Some(t) = token.filter(|t| !t.trim().is_empty()) {
+            // try_lock: this runs at construction, before any turn holds it.
+            if let Ok(mut g) = self.token.try_lock() {
+                *g = Some(t);
+            }
+        }
+        self
+    }
+
     pub fn from_env_for_gateway_with_code(
         gateway_url: String,
         pairing_code: String,
@@ -239,6 +254,25 @@ impl ZeroClawDriveExecutor {
             .ok_or_else(|| OrchestratorError::Executor("pair response had no token".into()))?
             .to_string();
         *guard = Some(token.clone());
+        // Persist it. The code we just spent cannot be used again, so if this
+        // token only ever lives in memory the next server process has no way
+        // back in — that is the 403 that killed a 93k-token research phase.
+        // Best-effort: failing to save must not fail a turn that just paired
+        // successfully; the cost is that a restart before the next write
+        // re-opens the original hole.
+        if let Some(tap) = self.tap.as_ref() {
+            if let Err(e) = sqlx::query("UPDATE missions SET runtime_token = $1 WHERE id = $2")
+                .bind(&token)
+                .bind(tap.mission_id)
+                .execute(&tap.pool)
+                .await
+            {
+                eprintln!(
+                    "topology_exec: could not persist runtime token for mission {}: {e}",
+                    tap.mission_id
+                );
+            }
+        }
         Ok(token)
     }
 

crates/cm-api/src/topology_worker.rs

@@ -185,9 +185,16 @@ async fn run_job(
     // the missions row; else fall back to the shared env-derived
     // gateway (pre-C3 missions + non-mission runs). This is what
     // isolates agents' workspace filesystem to that mission's repo.
-    type MissionBinding = (Option<String>, Option<String>, Uuid, Option<Uuid>);
+    type MissionBinding = (
+        Option<String>,
+        Option<String>,
+        Uuid,
+        Option<Uuid>,
+        Option<String>,
+    );
     let mission_binding: Option<MissionBinding> = sqlx::query_as::<_, MissionBinding>(
-        "SELECT m.runtime_endpoint, m.runtime_pairing_code, m.id, r.mission_phase_id
+        "SELECT m.runtime_endpoint, m.runtime_pairing_code, m.id, r.mission_phase_id,
+                m.runtime_token
            FROM topology_runs r
            JOIN missions m ON m.id = r.mission_id
           WHERE r.id = $1",
@@ -202,7 +209,7 @@ async fn run_job(
     let tap = mission_binding
         .as_ref()
         .map(
-            |(_, _, mission_id, phase_id)| crate::topology_exec::MissionTap {
+            |(_, _, mission_id, phase_id, _)| crate::topology_exec::MissionTap {
                 pool: pool.clone(),
                 workspace_id: job.workspace_id,
                 mission_id: *mission_id,
@@ -211,10 +218,17 @@ async fn run_job(
             },
         );
     let leaf_result = match mission_binding {
-        Some((Some(url), Some(code), _, _)) => {
+        // Seed the cached bearer from `runtime_token` when we have one: the
+        // pairing code is single-use, so after a restart it is the only way in.
+        Some((Some(url), Some(code), _, _, tok)) => {
             ZeroClawDriveExecutor::from_env_for_gateway_with_code(url, code)
+                .map(|e| e.with_token(tok))
+        }
+        // No pairing code (pre-C3 missions): the persisted token is the only
+        // credential, so seed it here too.
+        Some((Some(url), None, _, _, tok)) => {
+            ZeroClawDriveExecutor::from_env_for_gateway(url).map(|e| e.with_token(tok))
         }
-        Some((Some(url), None, _, _)) => ZeroClawDriveExecutor::from_env_for_gateway(url),
         _ => ZeroClawDriveExecutor::from_env(),
     };
     let leaf = match leaf_result {

migrations/0076_mission_runtime_token.sql

@@ -0,0 +1,22 @@
+-- Persist the per-mission runtime's bearer token.
+--
+-- The pairing code in `runtime_pairing_code` is SINGLE USE: once the gateway is
+-- paired it reports "already paired" and mints no new code. The token that pair
+-- returns was cached in memory only (`ZeroClawDriveExecutor::token`), so any
+-- restart of the server discarded it — and the next turn re-paired with a code
+-- that had already been spent and got 403 Forbidden, permanently, for that
+-- mission.
+--
+-- Mission 01a00538 ("ClawHDF5 REsearch and Refactor") died exactly that way: 19
+-- minutes and 93,762 tokens into its research phase, killed by a server restart
+-- it should have survived, and its coding phase was then correctly skipped as
+-- unreachable. The durable-run machinery exists so work survives a restart;
+-- pairing was the one thread that did not.
+--
+-- Nullable: a mission that has not paired yet has no token, and the pre-C3
+-- missions never will.
+ALTER TABLE missions
+    ADD COLUMN IF NOT EXISTS runtime_token TEXT;
+
+COMMENT ON COLUMN missions.runtime_token IS
+    'Bearer token for this mission runtime gateway, persisted so a server restart reuses it instead of re-pairing with a spent single-use code.';