fix(diag): "turn timed out" now says what the agent was doing
A research_and_code mission failed with: turn executor failed: turn timed out turn executor failed: turn timed out and that is the entire record. Investigating it found: the run produced zero steps and zero output, it died at exactly 700s (TURN_TIMEOUT), the node→claw aliases were bound correctly, and the same zeroclaw team path passes in the `multirole` scenario. So the platform path is fine and the agent simply never finished a turn — but the one place the reason lived, the per-mission runtime container, is torn down after the phase and takes its log with it. By the time anyone looks, all that survives is the string. The timeout now reads the last 40 lines out of that container while it still exists, and reports which agent alias and which gateway it was driving. Best-effort by construction: it runs on a path that is ALREADY failing, so a docker error there degrades to a note rather than replacing the real failure with a second one. `container_name` derives the container from the gateway URL and returns None rather than guessing, because this feeds a diagnostic — a wrong name would put a different container's log under a failure and send the reader somewhere else entirely. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
e417247e7e
commit
b2e2735583
@@ -219,13 +219,50 @@ impl ZeroClawDriveExecutor {
|
||||
.await
|
||||
.map_err(|e| OrchestratorError::Executor(format!("ws send failed: {e}")))?;
|
||||
|
||||
let outcome = tokio::time::timeout(TURN_TIMEOUT, Self::drain(&mut ws))
|
||||
.await
|
||||
.map_err(|_| OrchestratorError::Executor("turn timed out".into()))??;
|
||||
let outcome = match tokio::time::timeout(TURN_TIMEOUT, Self::drain(&mut ws)).await {
|
||||
Ok(res) => res?,
|
||||
Err(_) => {
|
||||
// "turn timed out" on its own is unactionable, and the one place
|
||||
// the reason lives — the per-mission runtime container — is torn
|
||||
// down after the phase, taking its log with it. Read the tail
|
||||
// while it still exists.
|
||||
//
|
||||
// MEASURED: a research phase timed out at exactly 700s having
|
||||
// produced zero steps and zero output, and the container was
|
||||
// already gone by the time anyone looked. All that survived was
|
||||
// the string.
|
||||
let container = self.container_name();
|
||||
let tail = match &container {
|
||||
Some(c) => crate::container_exec::tail_logs(c, 40).await,
|
||||
None => "(could not derive the container name from the gateway url)".into(),
|
||||
};
|
||||
return Err(OrchestratorError::Executor(format!(
|
||||
"turn timed out after {}s driving agent {alias} on {} — the agent \
|
||||
never finished a turn. Last lines from {}:\n{tail}",
|
||||
TURN_TIMEOUT.as_secs(),
|
||||
self.gateway_url,
|
||||
container.as_deref().unwrap_or("its runtime container"),
|
||||
)));
|
||||
}
|
||||
};
|
||||
let _ = ws.close(None).await;
|
||||
Ok(outcome)
|
||||
}
|
||||
|
||||
/// The runtime container behind this executor, derived from its gateway URL
|
||||
/// (`http://cm-runtime-mission-<hex>:42617`). Used only to fetch a log tail
|
||||
/// for an error message, so an unparseable URL is `None` rather than a
|
||||
/// failure.
|
||||
fn container_name(&self) -> Option<String> {
|
||||
let rest = self
|
||||
.gateway_url
|
||||
.split("://")
|
||||
.nth(1)
|
||||
.unwrap_or(&self.gateway_url);
|
||||
let host = rest.split('/').next()?.split(':').next()?;
|
||||
(!host.is_empty()).then(|| host.to_string())
|
||||
}
|
||||
|
||||
/// Use a runtime agent as a governance judge: drive `alias` with the judge
|
||||
/// prompt and parse the verdict (`DENY` anywhere ⇒ deny, else allow). This
|
||||
/// lets a **subscription-only** model (e.g. Kimi via `kimi_cli`) be the judge
|
||||
@@ -406,6 +443,32 @@ fn parse_agent_map(s: &str) -> HashMap<String, String> {
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
/// The container name comes out of the gateway URL, or nothing does.
|
||||
///
|
||||
/// This is only used to fetch a log tail for a failure message, so a URL
|
||||
/// shape it does not recognise must degrade to "no log" rather than to a
|
||||
/// second error on top of the first.
|
||||
#[test]
|
||||
fn the_container_name_is_derived_or_absent_never_wrong() {
|
||||
let ex = |url: &str| ZeroClawDriveExecutor::new(
|
||||
url.to_string(),
|
||||
String::new(),
|
||||
std::collections::HashMap::new(),
|
||||
"scout".into(),
|
||||
);
|
||||
assert_eq!(
|
||||
ex("http://cm-runtime-mission-019fec2d596f:42617").container_name().as_deref(),
|
||||
Some("cm-runtime-mission-019fec2d596f")
|
||||
);
|
||||
assert_eq!(
|
||||
ex("https://host.example:8443/base").container_name().as_deref(),
|
||||
Some("host.example")
|
||||
);
|
||||
// No scheme is still a host.
|
||||
assert_eq!(ex("clawmates-runtime:42617").container_name().as_deref(), Some("clawmates-runtime"));
|
||||
assert_eq!(ex("").container_name(), None);
|
||||
}
|
||||
|
||||
use super::*;
|
||||
use axum::extract::ws::{Message as AxMsg, WebSocket, WebSocketUpgrade};
|
||||
use axum::response::Response;
|
||||
|
||||
Reference in New Issue
Block a user