feat(viz): what the agents actually did, as structured events
The World could draw a mission's shape but nothing about the work. The
detail existed only as prose in checkpoint.log and model output, where a
tool name is indistinguishable from an agent *talking about* a tool — so
it was never parsed, deliberately. `mission_events` is the structured
channel that replaces it.
Three taps, one table:
- Container tier: the `_ => {}` at the end of topology_exec's typed frame
stream now matches `tool_call` and reads the tool's JSON ARGUMENTS for a
path. Never the prose summary — a path scraped from a sentence would put
files on the map that no agent opened, and the test proves a Grep whose
summary says "src/main.rs" produces no file touch. The frame name itself
is unverified, so the same commit ships an unmatched-frame-type
histogram: a tap that matches nothing looks exactly like a mission that
used no tools, and this is how one gw-04 run names the real frame.
- microVM tier: a `PostToolUse` hook, the seam vm_stop_gate already proved
fires under `claude -p`. It copies stdin to /root/tap and exits 0
unconditionally — a non-zero PostToolUse hook talks back to the model,
which would turn the observer into a participant. Drained before collect,
since the VM is destroyed moments later.
- Phase transitions: five identical copies of the pending→running UPDATE
became one `mark_phase_running`, and `close_finished_phases` grew
RETURNING. Its CASE decides each phase's status inside SQL from rows the
statement does not change, so it cannot be re-derived afterwards without
writing that CASE twice — without RETURNING it emits zero phase.completed
and reports success.
The settings.json hazard the plan called out: the stop gate wrote the
WHOLE document, so a second hook writer would have silently erased it and
a coding phase would then complete having written nothing — the exact
failure the gate exists to catch. There is now one composer,
`vm_tool_tap::guest_settings`, one writer, and a source-walk test that
fails if anything else writes a settings document.
`mission_events.run_id` carries no FK on purpose: phase_runner DELETEs
topology_runs on retry, and a cascade would erase a phase's whole history
the moment it retried — silently, since a cascade is not an error.
world.rs streams it with a cursor that separates backfill from motion.
Everything already in the table when a subscriber arrives is drawn as
settled history; only what lands afterwards animates. Otherwise opening a
finished mission replays an hour of tool calls as a burst storm.
Bounded twice: 400 events per phase (enforced inside the INSERT, since
two concurrent taps would each read a count below the cap) and a 7-day
retention sweep in mission_gc.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
co-authored by
Claude Opus 5
Claude Opus 5
parent
c2fa8067e1
commit
9e61e3ba35
@@ -54,6 +54,46 @@ pub struct ZeroClawDriveExecutor {
|
||||
/// Bearer token, paired lazily and reused across turns.
|
||||
token: Arc<Mutex<Option<String>>>,
|
||||
http: reqwest::Client,
|
||||
/// Where this executor's turns record what they did. `None` on every path
|
||||
/// that is not a mission phase (the governor, the door, the evaluator) —
|
||||
/// those turns belong to no phase and have nothing to attribute to.
|
||||
tap: Option<Arc<MissionTap>>,
|
||||
}
|
||||
|
||||
/// Where a turn's tool activity is written, and what it belongs to.
|
||||
///
|
||||
/// Carried on the executor rather than passed per turn because `TurnRequest`
|
||||
/// is the shared orchestrator contract: threading a mission id through it would
|
||||
/// put mission concepts into every tier that has no missions.
|
||||
pub struct MissionTap {
|
||||
pub pool: sqlx::PgPool,
|
||||
pub mission_id: uuid::Uuid,
|
||||
pub phase_id: Option<uuid::Uuid>,
|
||||
pub run_id: Option<uuid::Uuid>,
|
||||
}
|
||||
|
||||
/// One tool call, as the frame stream reported it.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct ToolCall {
|
||||
pub tool: String,
|
||||
/// The path the tool's **arguments** named, if any. Never extracted from a
|
||||
/// prose summary — see [`crate::mission_events::tool_path`].
|
||||
pub path: Option<String>,
|
||||
}
|
||||
|
||||
/// What one turn's frames said about the work, beside its text.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq)]
|
||||
pub struct ToolTrace {
|
||||
pub calls: Vec<ToolCall>,
|
||||
/// Frame `type` values this drain did not recognise, counted.
|
||||
///
|
||||
/// Shipped in the same change as the tap on purpose: the frame name
|
||||
/// `tool_call` is taken from a comment in this file, not from a captured
|
||||
/// frame. If the runtime calls it something else, the tap records nothing
|
||||
/// and nothing anywhere errors — the World simply stays as sparse as it was
|
||||
/// before. This histogram is how one gw-04 run names the real frame instead
|
||||
/// of a bisect.
|
||||
pub unmatched: std::collections::BTreeMap<String, u32>,
|
||||
}
|
||||
|
||||
impl ZeroClawDriveExecutor {
|
||||
@@ -71,9 +111,17 @@ impl ZeroClawDriveExecutor {
|
||||
default_alias,
|
||||
token: Arc::new(Mutex::new(None)),
|
||||
http: reqwest::Client::new(),
|
||||
tap: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Attach the mission this executor's turns belong to, so their tool calls
|
||||
/// are recorded. Without it the executor behaves exactly as it did.
|
||||
pub fn with_tap(mut self, tap: MissionTap) -> Self {
|
||||
self.tap = Some(Arc::new(tap));
|
||||
self
|
||||
}
|
||||
|
||||
/// Build from the environment:
|
||||
/// - `ZEROCLAW_GATEWAY_URL` (required) e.g. `http://127.0.0.1:42617`
|
||||
/// - `ZEROCLAW_TOKEN` (preferred) a durable bearer token — pair once
|
||||
@@ -203,6 +251,19 @@ impl ZeroClawDriveExecutor {
|
||||
}
|
||||
|
||||
pub async fn drive(&self, alias: &str, prompt: &str) -> Result<TurnOutcome, OrchestratorError> {
|
||||
self.drive_traced(alias, prompt).await.map(|(o, _)| o)
|
||||
}
|
||||
|
||||
/// [`Self::drive`], also returning what the turn's frames said it did.
|
||||
///
|
||||
/// Exists so the tool tap is testable at all: `drive` discards the trace
|
||||
/// after recording it, and a tap whose extraction is never asserted is
|
||||
/// exactly the kind of code that silently records nothing.
|
||||
pub(crate) async fn drive_traced(
|
||||
&self,
|
||||
alias: &str,
|
||||
prompt: &str,
|
||||
) -> Result<(TurnOutcome, ToolTrace), OrchestratorError> {
|
||||
let token = self.ensure_paired().await?;
|
||||
let ws_base = if let Some(rest) = self.gateway_url.strip_prefix("https") {
|
||||
format!("wss{rest}")
|
||||
@@ -226,7 +287,8 @@ impl ZeroClawDriveExecutor {
|
||||
.await
|
||||
.map_err(|e| OrchestratorError::Executor(format!("ws send failed: {e}")))?;
|
||||
|
||||
let outcome = match tokio::time::timeout(TURN_TIMEOUT, Self::drain(&mut ws)).await {
|
||||
let (outcome, trace) = match tokio::time::timeout(TURN_TIMEOUT, Self::drain(&mut ws)).await
|
||||
{
|
||||
Ok(res) => res?,
|
||||
Err(_) => {
|
||||
// "turn timed out" on its own is unactionable, and the one place
|
||||
@@ -253,7 +315,60 @@ impl ZeroClawDriveExecutor {
|
||||
}
|
||||
};
|
||||
let _ = ws.close(None).await;
|
||||
Ok(outcome)
|
||||
self.record_trace(alias, &trace).await;
|
||||
Ok((outcome, trace))
|
||||
}
|
||||
|
||||
/// Persist what this turn's frames said the agent did.
|
||||
///
|
||||
/// Best-effort and after the fact: a telemetry write must not be able to
|
||||
/// fail a turn that already succeeded.
|
||||
async fn record_trace(&self, alias: &str, trace: &ToolTrace) {
|
||||
if !trace.unmatched.is_empty() {
|
||||
// Logged whether or not a tap is attached — the point is to learn
|
||||
// the real frame names, and the paths with no tap see the same
|
||||
// stream.
|
||||
eprintln!(
|
||||
"topology_exec: unmatched frame types this turn ({alias}): {:?}",
|
||||
trace.unmatched
|
||||
);
|
||||
}
|
||||
let Some(tap) = self.tap.as_ref() else { return };
|
||||
if trace.calls.is_empty() {
|
||||
return;
|
||||
}
|
||||
let agent_id = crate::runtime_provision::claw_from_alias(alias);
|
||||
let event = |kind: &str, target: String, detail: serde_json::Value| {
|
||||
crate::mission_events::MissionEvent {
|
||||
mission_id: tap.mission_id,
|
||||
phase_id: tap.phase_id,
|
||||
run_id: tap.run_id,
|
||||
agent_id,
|
||||
kind: kind.to_string(),
|
||||
target: Some(target),
|
||||
detail,
|
||||
}
|
||||
};
|
||||
let mut events = Vec::new();
|
||||
for call in &trace.calls {
|
||||
events.push(event(
|
||||
crate::mission_events::TOOL_CALL,
|
||||
call.tool.clone(),
|
||||
serde_json::Value::Null,
|
||||
));
|
||||
// A file touch is a SECOND event, not a replacement: the tool call
|
||||
// happened whether or not we could name a path in its arguments,
|
||||
// and collapsing the two would make every unparseable tool call
|
||||
// disappear from the record entirely.
|
||||
if let Some(path) = &call.path {
|
||||
events.push(event(
|
||||
crate::mission_events::FILE_TOUCH,
|
||||
crate::mission_events::repo_relative(path, GUEST_ROOTS),
|
||||
serde_json::json!({ "tool": call.tool }),
|
||||
));
|
||||
}
|
||||
}
|
||||
crate::mission_events::record_all(&tap.pool, events).await;
|
||||
}
|
||||
|
||||
/// The runtime container behind this executor, derived from its gateway URL
|
||||
@@ -330,7 +445,12 @@ impl ZeroClawDriveExecutor {
|
||||
}
|
||||
|
||||
/// Read frames until a terminal (`done`/`error`/`approval_request`) event.
|
||||
async fn drain<S>(ws: &mut S) -> Result<TurnOutcome, OrchestratorError>
|
||||
///
|
||||
/// Returns the turn's outcome AND what its frames said the agent did. The
|
||||
/// trace is separate from [`TurnOutcome`] deliberately: that type is the
|
||||
/// shared orchestrator contract used by every tier, and tool telemetry is a
|
||||
/// mission concern.
|
||||
async fn drain<S>(ws: &mut S) -> Result<(TurnOutcome, ToolTrace), OrchestratorError>
|
||||
where
|
||||
S: StreamExt<Item = Result<Message, tokio_tungstenite::tungstenite::Error>>
|
||||
+ SinkExt<Message>
|
||||
@@ -339,6 +459,7 @@ impl ZeroClawDriveExecutor {
|
||||
let mut output = String::new();
|
||||
let mut tokens: u64 = 0;
|
||||
let mut gated: Vec<GatedAction> = Vec::new();
|
||||
let mut trace = ToolTrace::default();
|
||||
|
||||
while let Some(frame) = ws.next().await {
|
||||
let msg = frame.map_err(|e| OrchestratorError::Executor(format!("ws recv: {e}")))?;
|
||||
@@ -384,8 +505,39 @@ impl ZeroClawDriveExecutor {
|
||||
"aborted" => {
|
||||
return Err(OrchestratorError::Executor("turn aborted".into()));
|
||||
}
|
||||
// session_start, thinking, tool_call, tool_result, …
|
||||
_ => {}
|
||||
// The action channel. `arguments` is read as JSON and
|
||||
// nothing else is: the frame also carries a prose
|
||||
// summary, and a path pulled out of THAT would be right
|
||||
// often enough to be believed and wrong often enough to
|
||||
// put files on the map that nobody edited.
|
||||
"tool_call" => {
|
||||
let tool = v
|
||||
.get("tool")
|
||||
.or_else(|| v.get("name"))
|
||||
.and_then(|t| t.as_str())
|
||||
.unwrap_or("")
|
||||
.trim()
|
||||
.to_string();
|
||||
if !tool.is_empty() {
|
||||
let args = v
|
||||
.get("arguments")
|
||||
.or_else(|| v.get("input"))
|
||||
.cloned()
|
||||
.unwrap_or(serde_json::Value::Null);
|
||||
trace.calls.push(ToolCall {
|
||||
path: crate::mission_events::tool_path(&args),
|
||||
tool,
|
||||
});
|
||||
}
|
||||
}
|
||||
// session_start, thinking, tool_result, …
|
||||
other => {
|
||||
// Counted, not ignored. See `ToolTrace::unmatched`:
|
||||
// the frame name above is unverified, and a tap
|
||||
// that matches nothing looks exactly like a mission
|
||||
// that used no tools.
|
||||
*trace.unmatched.entry(other.to_string()).or_insert(0) += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
Message::Ping(p) => {
|
||||
@@ -396,14 +548,21 @@ impl ZeroClawDriveExecutor {
|
||||
}
|
||||
}
|
||||
|
||||
Ok(TurnOutcome {
|
||||
output: output.trim().to_string(),
|
||||
tokens,
|
||||
gated,
|
||||
})
|
||||
Ok((
|
||||
TurnOutcome {
|
||||
output: output.trim().to_string(),
|
||||
tokens,
|
||||
gated,
|
||||
},
|
||||
trace,
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
/// Guest workspace roots, stripped so a tool's absolute path becomes the
|
||||
/// repo-relative one a person recognises.
|
||||
const GUEST_ROOTS: &[&str] = &["/mission/repo", "/workspace", "/repo"];
|
||||
|
||||
impl TurnExecutor for ZeroClawDriveExecutor {
|
||||
async fn run_turn(&self, req: TurnRequest) -> Result<TurnOutcome, OrchestratorError> {
|
||||
// An explicit per-node agent (graph `node.attrs["agent"]`) wins, so one
|
||||
@@ -505,6 +664,30 @@ mod tests {
|
||||
})
|
||||
}
|
||||
|
||||
/// A stream carrying tool calls and one frame type we do not know.
|
||||
async fn tool_ws(ws: WebSocketUpgrade) -> Response {
|
||||
ws.on_upgrade(|mut socket: WebSocket| async move {
|
||||
let _ = socket.recv().await;
|
||||
for f in [
|
||||
json!({"type": "session_start"}),
|
||||
json!({"type": "tool_call", "tool": "Read",
|
||||
"arguments": {"file_path": "/mission/repo/src/a.rs"}}),
|
||||
// A tool whose arguments name no path at all.
|
||||
json!({"type": "tool_call", "tool": "Bash",
|
||||
"arguments": {"command": "cargo test"}}),
|
||||
// Prose that MENTIONS a path. It must not become a file touch.
|
||||
json!({"type": "tool_call", "tool": "Grep",
|
||||
"arguments_summary": "searching src/main.rs",
|
||||
"arguments": {"pattern": "fn main"}}),
|
||||
json!({"type": "a_frame_we_have_never_seen"}),
|
||||
json!({"type": "a_frame_we_have_never_seen"}),
|
||||
json!({"type": "done", "input_tokens": 1, "output_tokens": 1}),
|
||||
] {
|
||||
let _ = socket.send(AxMsg::Text(f.to_string().into())).await;
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
async fn approval_ws(ws: WebSocketUpgrade) -> Response {
|
||||
ws.on_upgrade(|mut socket: WebSocket| async move {
|
||||
let _ = socket.recv().await;
|
||||
@@ -569,6 +752,40 @@ mod tests {
|
||||
assert!(out.gated.is_empty());
|
||||
}
|
||||
|
||||
/// Tool detail comes from arguments, and unknown frames are counted.
|
||||
///
|
||||
/// The two halves are one test because they are one risk. The frame type
|
||||
/// `tool_call` is taken from a comment in this file, not from a captured
|
||||
/// frame — so if it is wrong, the tap records nothing, the World stays as
|
||||
/// sparse as it was, and NOTHING errors. The histogram is what turns that
|
||||
/// into a log line naming the real frame.
|
||||
#[tokio::test]
|
||||
async fn tool_frames_give_up_their_arguments_and_unknown_frames_are_counted() {
|
||||
let router = Router::new()
|
||||
.route("/pair", post(pair))
|
||||
.route("/ws/chat", get(tool_ws));
|
||||
let base = serve(router).await;
|
||||
let exec = ZeroClawDriveExecutor::new(base, "code".into(), HashMap::new(), "scout".into());
|
||||
|
||||
let (_out, trace) = exec.drive_traced("scout", "go").await.unwrap();
|
||||
|
||||
assert_eq!(
|
||||
trace.calls,
|
||||
vec![
|
||||
ToolCall { tool: "Read".into(), path: Some("/mission/repo/src/a.rs".into()) },
|
||||
ToolCall { tool: "Bash".into(), path: None },
|
||||
// `arguments_summary` said "src/main.rs". It is prose, so it is
|
||||
// not a file touch — a path scraped from a sentence would put
|
||||
// files on the map that no agent opened.
|
||||
ToolCall { tool: "Grep".into(), path: None },
|
||||
]
|
||||
);
|
||||
assert_eq!(trace.unmatched.get("a_frame_we_have_never_seen"), Some(&2));
|
||||
assert_eq!(trace.unmatched.get("session_start"), Some(&1));
|
||||
// `done` terminates the drain and is not an unmatched frame.
|
||||
assert!(!trace.unmatched.contains_key("done"), "{:?}", trace.unmatched);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn approval_request_is_recorded_as_blocked() {
|
||||
let router = Router::new()
|
||||
|
||||
Reference in New Issue
Block a user