feat(observability): stream a microVM turn's stdout/stderr to the platform live
The Live tab showed nothing while a turn ran, and the agent's own account of it
went to stderr on the node and nowhere a user could reach. This is the path that
carries it.
The blocker was the guest agent. `fcagent` handled one connection at a time,
inline, so during an hour-long turn the VM accepted nothing — which is why every
existing probe (subagents, stop-gate blocks, cap) runs AFTER the turn rather than
during it. It now spawns a thread per connection, wrapped in `catch_unwind`
because this process is pid 1: a panic used to take the accept loop with it, and
an unbootable VM is a far worse outcome than a missing log. A failed spawn logs
and keeps accepting rather than dropping the listener.
PROVED against a live VM before building on it, since "sound reasoning about this
system" and "measurement" have diverged repeatedly today. Patched rootfs, booted
under Firecracker, ran an 8s exec and a concurrent tail:
exec took 8.0s ok=True
+0.0s 'line1\nline2\n' +1.2s 'line4\n' +3.2s 'line6\n' +6.0s 'DONE\n'
VERDICT: CONCURRENT — tail returned data before exec finished
The rest is the pattern the terminal already uses. New `tail` op streams a file
by OFFSET (so a dropped link resumes instead of replaying, and the tail always
terminates — one that never returns pins a thread for the life of the VM). The
node follows the log alongside the turn and pushes `Uplink::VmOut { run_id, at,
data }` over the WebSocket it already holds, mirroring `PtyOut`. The server does
what `PtyOut` deliberately does not: it APPENDS to the run's checkpoint as well
as fanning out, because a terminal has no history worth keeping and a mission log
is the record of what the agent did. `run_events_sse` emits the new bytes as
`step` events, which the live pane already renders — no frontend change.
The turn is `tee`d, not redirected: the file feeds the live stream and stdout
still becomes `VmOutcome::summary`. A redirect would have produced a live view
and an empty summary, which is the same green-and-empty shape as the bug this
fixes. Tested, along with the log living outside the collected tree so it never
lands in a user's delivered diff.
246 lib tests, 20 binaries; node and fcagent build clean.
This commit is contained in:
@@ -309,6 +309,10 @@ pub async fn run_events_sse(
|
||||
.map(|n| n + 1)
|
||||
.unwrap_or(0);
|
||||
|
||||
// Bytes of `checkpoint.log` already sent. The step cursor above counts
|
||||
// RECORDS; this counts BYTES, because a log grows continuously rather than
|
||||
// in discrete entries. Two sources, two cursors.
|
||||
let mut log_sent: usize = 0;
|
||||
let stream = async_stream::stream! {
|
||||
loop {
|
||||
match cm_db::repo::topology_runs::status(&pool, id, ws).await {
|
||||
@@ -327,6 +331,24 @@ pub async fn run_events_sse(
|
||||
sent += 1;
|
||||
}
|
||||
}
|
||||
// Live stdout/stderr from a microVM turn, appended by the
|
||||
// node over the fleet WebSocket (`Uplink::VmOut`). Emitted
|
||||
// as `step` so the existing reader renders it with no
|
||||
// frontend change — it already reads `data.text`.
|
||||
if let Some(log) = st
|
||||
.checkpoint
|
||||
.as_ref()
|
||||
.and_then(|c| c.get("log"))
|
||||
.and_then(|v| v.as_str())
|
||||
{
|
||||
if log.len() > log_sent {
|
||||
let fresh = &log[log_sent..];
|
||||
log_sent = log.len();
|
||||
yield Ok::<Event, Infallible>(Event::default().event("step").data(
|
||||
serde_json::json!({ "kind": "output", "text": fresh }).to_string(),
|
||||
));
|
||||
}
|
||||
}
|
||||
if matches!(st.status.as_str(), "completed" | "failed" | "cancelled") {
|
||||
let done = serde_json::json!({
|
||||
"status": st.status,
|
||||
|
||||
Reference in New Issue
Block a user