feat(observability): stream a microVM turn's stdout/stderr to the platform live

The Live tab showed nothing while a turn ran, and the agent's own account of it
went to stderr on the node and nowhere a user could reach. This is the path that
carries it.

The blocker was the guest agent. `fcagent` handled one connection at a time,
inline, so during an hour-long turn the VM accepted nothing — which is why every
existing probe (subagents, stop-gate blocks, cap) runs AFTER the turn rather than
during it. It now spawns a thread per connection, wrapped in `catch_unwind`
because this process is pid 1: a panic used to take the accept loop with it, and
an unbootable VM is a far worse outcome than a missing log. A failed spawn logs
and keeps accepting rather than dropping the listener.

PROVED against a live VM before building on it, since "sound reasoning about this
system" and "measurement" have diverged repeatedly today. Patched rootfs, booted
under Firecracker, ran an 8s exec and a concurrent tail:

    exec took 8.0s ok=True
    +0.0s 'line1\nline2\n'  +1.2s 'line4\n'  +3.2s 'line6\n'  +6.0s 'DONE\n'
    VERDICT: CONCURRENT — tail returned data before exec finished

The rest is the pattern the terminal already uses. New `tail` op streams a file
by OFFSET (so a dropped link resumes instead of replaying, and the tail always
terminates — one that never returns pins a thread for the life of the VM). The
node follows the log alongside the turn and pushes `Uplink::VmOut { run_id, at,
data }` over the WebSocket it already holds, mirroring `PtyOut`. The server does
what `PtyOut` deliberately does not: it APPENDS to the run's checkpoint as well
as fanning out, because a terminal has no history worth keeping and a mission log
is the record of what the agent did. `run_events_sse` emits the new bytes as
`step` events, which the live pane already renders — no frontend change.

The turn is `tee`d, not redirected: the file feeds the live stream and stdout
still becomes `VmOutcome::summary`. A redirect would have produced a live view
and an empty summary, which is the same green-and-empty shape as the bug this
fixes. Tested, along with the log living outside the collected tree so it never
lands in a user's delivered diff.

246 lib tests, 20 binaries; node and fcagent build clean.
This commit is contained in:
Omar Sobh
2026-08-07 21:07:12 -07:00
parent 62509a5090
commit 0b89b8316c
9 changed files with 449 additions and 15 deletions
+80
View File
@@ -621,7 +621,40 @@ async fn handle_frame(
// Running it inline would stall heartbeats and the daemon would
// be declared offline mid-mission.
tokio::spawn(async move {
// While an `exec` runs, follow the turn's log and push each
// chunk to the server as it appears. The guest agent accepts
// concurrent connections (proved against a live VM: a tail
// returned data second-by-second while an 8s exec was still
// running), so this does not wait for, or delay, the turn.
//
// Only for `vm_exec`, and only when the caller named a run to
// attribute the output to — a probe exec has nothing to
// stream and no subscriber.
let tail = (op == "vm_exec")
.then(|| {
let run_id = v.get("run_id").and_then(Value::as_str)?.to_string();
let log_path = v
.get("log_path")
.and_then(Value::as_str)
.unwrap_or("/root/agent.log")
.to_string();
let vm_id = v.get("vm_id").and_then(Value::as_str)?.to_string();
Some(tokio::spawn(stream_vm_log(
vms.clone(),
vm_id,
run_id,
log_path,
out.clone(),
)))
})
.flatten();
let (ok, output) = microvm::handle_op(&op, &v, &vms).await;
// The turn is over; the tail's own idle timeout will end it,
// but aborting is immediate and leaves no thread waiting on a
// file that stopped growing.
if let Some(t) = tail {
t.abort();
}
let _ = out.send(
json!({ "t": "result", "id": id, "ok": ok, "output": output }).to_string(),
);
@@ -859,6 +892,53 @@ fn spawn_command_pty(argv: &[String], cols: u16, rows: u16) -> Result<PtyParts,
spawn_pty(c, cols, rows)
}
/// Follow a running turn's log inside a VM and push each chunk to the server.
///
/// The other half of the observability path: the guest tails the file, this
/// forwards what it reads over the WebSocket the daemon already holds, and the
/// server appends it to the run so the live pane and the Output tab both have it.
///
/// Reconnects on a dropped tail, resuming from the last offset — following by
/// OFFSET rather than holding one socket open forever is what makes that cheap.
/// It gives up after a few consecutive failures rather than spinning: by then
/// the VM is gone and the turn's own result is the record.
async fn stream_vm_log(
vms: microvm::Vms,
vm_id: String,
run_id: String,
log_path: String,
out: tokio::sync::mpsc::UnboundedSender<String>,
) {
let mut at: u64 = 0;
let mut failures = 0;
while failures < 3 {
let sent = out.clone();
let rid = run_id.clone();
match microvm::tail_into(&vms, &vm_id, &log_path, at, move |offset, data| {
let _ = sent.send(
json!({ "t": "vm_out", "run_id": rid, "at": offset, "data": data }).to_string(),
);
})
.await
{
Ok(reached) => {
// No progress and no error means the guest reported EOF: the log
// stopped growing, so the turn is done writing.
if reached == at {
return;
}
at = reached;
failures = 0;
}
Err(e) => {
failures += 1;
eprintln!("clawmates-node: tail of {vm_id} for run {run_id} failed: {e}");
tokio::time::sleep(std::time::Duration::from_secs(2)).await;
}
}
}
}
/// Spawn a host login shell in a PTY; stream its output back as pty_out frames.
async fn open_pty(
sid: u64,