fix(missions): a server restart no longer kills a running mission
deploy / test (push) Successful in 4m25s
deploy / build (push) Successful in 5m34s

Mission 01a00538 ("ClawHDF5 REsearch and Refactor") failed 19 minutes and 93,762
tokens into its research phase with `pair failed: 403 Forbidden`, and its coding
phase was then correctly skipped as unreachable. The cause was not the coding
phase and not the model — it was pairing.

A per-mission runtime is authenticated with a SINGLE-USE pairing code, and the
bearer token it returns was cached in memory only. Any restart of the server
discarded that token; the next turn re-paired with a code the gateway had
already spent and got 403 — permanently, for that mission. A deploy, a crash or
an OOM would each do it. The durable-run machinery exists precisely so work
survives a restart; pairing was the one thread that did not, and it failed
closed.

`missions.runtime_token` persists the token at the moment pairing succeeds, and
the worker seeds the executor's cache from it, so a new process reuses the
credential instead of re-pairing. Persisting is best-effort: failing to save
must not fail a turn that just paired successfully.

Verified by reproducing the original failure: launched a mission, confirmed the
token was written, restarted the server MID-PHASE, and watched the mission run
to completion with no pairing failure.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-15 21:43:05 -07:00
co-authored by Claude Opus 5
parent 43436d7181
commit 6e8785f159
3 changed files with 75 additions and 5 deletions
+34
View File
@@ -178,6 +178,21 @@ impl ZeroClawDriveExecutor {
/// new one-time code at startup. The env-derived ZEROCLAW_TOKEN
/// is ignored (belongs to the shared runtime) so the lazy pair
/// path runs and issues a bearer for this specific gateway.
/// Reuse a token that was already paired and persisted.
///
/// The pairing code is single-use, so a restarted server cannot pair again:
/// it gets 403 and the mission is unrecoverable. Seeding the cache from
/// `missions.runtime_token` is what makes a mission survive a restart.
pub fn with_token(self, token: Option<String>) -> Self {
if let Some(t) = token.filter(|t| !t.trim().is_empty()) {
// try_lock: this runs at construction, before any turn holds it.
if let Ok(mut g) = self.token.try_lock() {
*g = Some(t);
}
}
self
}
pub fn from_env_for_gateway_with_code(
gateway_url: String,
pairing_code: String,
@@ -239,6 +254,25 @@ impl ZeroClawDriveExecutor {
.ok_or_else(|| OrchestratorError::Executor("pair response had no token".into()))?
.to_string();
*guard = Some(token.clone());
// Persist it. The code we just spent cannot be used again, so if this
// token only ever lives in memory the next server process has no way
// back in — that is the 403 that killed a 93k-token research phase.
// Best-effort: failing to save must not fail a turn that just paired
// successfully; the cost is that a restart before the next write
// re-opens the original hole.
if let Some(tap) = self.tap.as_ref() {
if let Err(e) = sqlx::query("UPDATE missions SET runtime_token = $1 WHERE id = $2")
.bind(&token)
.bind(tap.mission_id)
.execute(&tap.pool)
.await
{
eprintln!(
"topology_exec: could not persist runtime token for mission {}: {e}",
tap.mission_id
);
}
}
Ok(token)
}