fix(missions): a server restart no longer kills a running mission
Mission 01a00538 ("ClawHDF5 REsearch and Refactor") failed 19 minutes and 93,762
tokens into its research phase with `pair failed: 403 Forbidden`, and its coding
phase was then correctly skipped as unreachable. The cause was not the coding
phase and not the model — it was pairing.
A per-mission runtime is authenticated with a SINGLE-USE pairing code, and the
bearer token it returns was cached in memory only. Any restart of the server
discarded that token; the next turn re-paired with a code the gateway had
already spent and got 403 — permanently, for that mission. A deploy, a crash or
an OOM would each do it. The durable-run machinery exists precisely so work
survives a restart; pairing was the one thread that did not, and it failed
closed.
`missions.runtime_token` persists the token at the moment pairing succeeds, and
the worker seeds the executor's cache from it, so a new process reuses the
credential instead of re-pairing. Persisting is best-effort: failing to save
must not fail a turn that just paired successfully.
Verified by reproducing the original failure: launched a mission, confirmed the
token was written, restarted the server MID-PHASE, and watched the mission run
to completion with no pairing failure.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
43436d7181
commit
6e8785f159
@@ -178,6 +178,21 @@ impl ZeroClawDriveExecutor {
|
||||
/// new one-time code at startup. The env-derived ZEROCLAW_TOKEN
|
||||
/// is ignored (belongs to the shared runtime) so the lazy pair
|
||||
/// path runs and issues a bearer for this specific gateway.
|
||||
/// Reuse a token that was already paired and persisted.
|
||||
///
|
||||
/// The pairing code is single-use, so a restarted server cannot pair again:
|
||||
/// it gets 403 and the mission is unrecoverable. Seeding the cache from
|
||||
/// `missions.runtime_token` is what makes a mission survive a restart.
|
||||
pub fn with_token(self, token: Option<String>) -> Self {
|
||||
if let Some(t) = token.filter(|t| !t.trim().is_empty()) {
|
||||
// try_lock: this runs at construction, before any turn holds it.
|
||||
if let Ok(mut g) = self.token.try_lock() {
|
||||
*g = Some(t);
|
||||
}
|
||||
}
|
||||
self
|
||||
}
|
||||
|
||||
pub fn from_env_for_gateway_with_code(
|
||||
gateway_url: String,
|
||||
pairing_code: String,
|
||||
@@ -239,6 +254,25 @@ impl ZeroClawDriveExecutor {
|
||||
.ok_or_else(|| OrchestratorError::Executor("pair response had no token".into()))?
|
||||
.to_string();
|
||||
*guard = Some(token.clone());
|
||||
// Persist it. The code we just spent cannot be used again, so if this
|
||||
// token only ever lives in memory the next server process has no way
|
||||
// back in — that is the 403 that killed a 93k-token research phase.
|
||||
// Best-effort: failing to save must not fail a turn that just paired
|
||||
// successfully; the cost is that a restart before the next write
|
||||
// re-opens the original hole.
|
||||
if let Some(tap) = self.tap.as_ref() {
|
||||
if let Err(e) = sqlx::query("UPDATE missions SET runtime_token = $1 WHERE id = $2")
|
||||
.bind(&token)
|
||||
.bind(tap.mission_id)
|
||||
.execute(&tap.pool)
|
||||
.await
|
||||
{
|
||||
eprintln!(
|
||||
"topology_exec: could not persist runtime token for mission {}: {e}",
|
||||
tap.mission_id
|
||||
);
|
||||
}
|
||||
}
|
||||
Ok(token)
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user