feat(observability): stream a microVM turn's stdout/stderr to the platform live

The Live tab showed nothing while a turn ran, and the agent's own account of it
went to stderr on the node and nowhere a user could reach. This is the path that
carries it.

The blocker was the guest agent. `fcagent` handled one connection at a time,
inline, so during an hour-long turn the VM accepted nothing — which is why every
existing probe (subagents, stop-gate blocks, cap) runs AFTER the turn rather than
during it. It now spawns a thread per connection, wrapped in `catch_unwind`
because this process is pid 1: a panic used to take the accept loop with it, and
an unbootable VM is a far worse outcome than a missing log. A failed spawn logs
and keeps accepting rather than dropping the listener.

PROVED against a live VM before building on it, since "sound reasoning about this
system" and "measurement" have diverged repeatedly today. Patched rootfs, booted
under Firecracker, ran an 8s exec and a concurrent tail:

    exec took 8.0s ok=True
    +0.0s 'line1\nline2\n'  +1.2s 'line4\n'  +3.2s 'line6\n'  +6.0s 'DONE\n'
    VERDICT: CONCURRENT — tail returned data before exec finished

The rest is the pattern the terminal already uses. New `tail` op streams a file
by OFFSET (so a dropped link resumes instead of replaying, and the tail always
terminates — one that never returns pins a thread for the life of the VM). The
node follows the log alongside the turn and pushes `Uplink::VmOut { run_id, at,
data }` over the WebSocket it already holds, mirroring `PtyOut`. The server does
what `PtyOut` deliberately does not: it APPENDS to the run's checkpoint as well
as fanning out, because a terminal has no history worth keeping and a mission log
is the record of what the agent did. `run_events_sse` emits the new bytes as
`step` events, which the live pane already renders — no frontend change.

The turn is `tee`d, not redirected: the file feeds the live stream and stdout
still becomes `VmOutcome::summary`. A redirect would have produced a live view
and an empty summary, which is the same green-and-empty shape as the bug this
fixes. Tested, along with the log living outside the collected tree so it never
lands in a user's delivered diff.

246 lib tests, 20 binaries; node and fcagent build clean.
This commit is contained in:
Omar Sobh
2026-08-07 21:07:12 -07:00
parent 62509a5090
commit 0b89b8316c
9 changed files with 449 additions and 15 deletions
+22
View File
@@ -309,6 +309,10 @@ pub async fn run_events_sse(
.map(|n| n + 1)
.unwrap_or(0);
// Bytes of `checkpoint.log` already sent. The step cursor above counts
// RECORDS; this counts BYTES, because a log grows continuously rather than
// in discrete entries. Two sources, two cursors.
let mut log_sent: usize = 0;
let stream = async_stream::stream! {
loop {
match cm_db::repo::topology_runs::status(&pool, id, ws).await {
@@ -327,6 +331,24 @@ pub async fn run_events_sse(
sent += 1;
}
}
// Live stdout/stderr from a microVM turn, appended by the
// node over the fleet WebSocket (`Uplink::VmOut`). Emitted
// as `step` so the existing reader renders it with no
// frontend change — it already reads `data.text`.
if let Some(log) = st
.checkpoint
.as_ref()
.and_then(|c| c.get("log"))
.and_then(|v| v.as_str())
{
if log.len() > log_sent {
let fresh = &log[log_sent..];
log_sent = log.len();
yield Ok::<Event, Infallible>(Event::default().event("step").data(
serde_json::json!({ "kind": "output", "text": fresh }).to_string(),
));
}
}
if matches!(st.status.as_str(), "completed" | "failed" | "cancelled") {
let done = serde_json::json!({
"status": st.status,