feat(observability): stream a microVM turn's stdout/stderr to the platform live

The Live tab showed nothing while a turn ran, and the agent's own account of it
went to stderr on the node and nowhere a user could reach. This is the path that
carries it.

The blocker was the guest agent. `fcagent` handled one connection at a time,
inline, so during an hour-long turn the VM accepted nothing — which is why every
existing probe (subagents, stop-gate blocks, cap) runs AFTER the turn rather than
during it. It now spawns a thread per connection, wrapped in `catch_unwind`
because this process is pid 1: a panic used to take the accept loop with it, and
an unbootable VM is a far worse outcome than a missing log. A failed spawn logs
and keeps accepting rather than dropping the listener.

PROVED against a live VM before building on it, since "sound reasoning about this
system" and "measurement" have diverged repeatedly today. Patched rootfs, booted
under Firecracker, ran an 8s exec and a concurrent tail:

    exec took 8.0s ok=True
    +0.0s 'line1\nline2\n'  +1.2s 'line4\n'  +3.2s 'line6\n'  +6.0s 'DONE\n'
    VERDICT: CONCURRENT — tail returned data before exec finished

The rest is the pattern the terminal already uses. New `tail` op streams a file
by OFFSET (so a dropped link resumes instead of replaying, and the tail always
terminates — one that never returns pins a thread for the life of the VM). The
node follows the log alongside the turn and pushes `Uplink::VmOut { run_id, at,
data }` over the WebSocket it already holds, mirroring `PtyOut`. The server does
what `PtyOut` deliberately does not: it APPENDS to the run's checkpoint as well
as fanning out, because a terminal has no history worth keeping and a mission log
is the record of what the agent did. `run_events_sse` emits the new bytes as
`step` events, which the live pane already renders — no frontend change.

The turn is `tee`d, not redirected: the file feeds the live stream and stdout
still becomes `VmOutcome::summary`. A redirect would have produced a live view
and an empty summary, which is the same green-and-empty shape as the bug this
fixes. Tested, along with the log living outside the collected tree so it never
lands in a user's delivered diff.

246 lib tests, 20 binaries; node and fcagent build clean.
This commit is contained in:
Omar Sobh
2026-08-07 21:07:12 -07:00
parent 62509a5090
commit 0b89b8316c
9 changed files with 449 additions and 15 deletions
+83 -2
View File
@@ -88,9 +88,10 @@ fn check_id(vm_id: &str) -> Result<(), String> {
/// in pieces, and reading "whatever was available" would parse a truncated
/// object as a complete one. Claude's `stream-json` output makes that routine
/// rather than theoretical.
async fn rpc(uds: &Path, req: &Value) -> Result<Value, String> {
const MAX_REPLY: u32 = 256 * 1024 * 1024;
/// Largest reply frame accepted from a guest, for both one-shot rpc and tail.
const MAX_REPLY: u32 = 256 * 1024 * 1024;
async fn rpc(uds: &Path, req: &Value) -> Result<Value, String> {
let mut s = UnixStream::connect(uds)
.await
.map_err(|e| format!("connect {}: {e}", uds.display()))?;
@@ -133,6 +134,86 @@ async fn rpc(uds: &Path, req: &Value) -> Result<Value, String> {
serde_json::from_slice(&buf).map_err(|e| format!("decode reply: {e}"))
}
/// Follow a file inside a guest, handing each chunk to `on_chunk` as it arrives.
///
/// Unlike [`rpc`], which is one request and one reply, this keeps the connection
/// open and reads MANY framed replies — the guest's `tail` op emits one per
/// chunk and a final `eof`. That is what makes a turn's output visible while the
/// turn is still running, and it works only because the guest agent now accepts
/// concurrent connections.
///
/// Returns the byte offset reached, so a caller that reconnects resumes instead
/// of replaying.
pub async fn tail_into<F>(
vms: &Vms,
vm_id: &str,
path: &str,
from: u64,
mut on_chunk: F,
) -> Result<u64, String>
where
F: FnMut(u64, String),
{
let uds = uds_of(vms, vm_id).await?;
let mut s = UnixStream::connect(&uds)
.await
.map_err(|e| format!("connect {}: {e}", uds.display()))?;
s.write_all(b"CONNECT 9001\n")
.await
.map_err(|e| format!("vsock CONNECT: {e}"))?;
let mut ack = [0u8; 64];
let n = s
.read(&mut ack)
.await
.map_err(|e| format!("vsock CONNECT ack: {e}"))?;
if !String::from_utf8_lossy(&ack[..n]).starts_with("OK") {
return Err("vsock refused the tail connection".into());
}
let req = json!({
"op": "tail", "path": path, "from": from,
// Long enough that a quiet agent is not mistaken for a finished one,
// short enough that the thread is released soon after the turn ends.
"idle_ms": 15_000, "max_secs": 3_600,
});
let body = serde_json::to_vec(&req).map_err(|e| format!("encode tail: {e}"))?;
s.write_all(&(body.len() as u32).to_be_bytes())
.await
.map_err(|e| format!("write tail length: {e}"))?;
s.write_all(&body)
.await
.map_err(|e| format!("write tail body: {e}"))?;
let mut at = from;
loop {
let mut len = [0u8; 4];
if s.read_exact(&mut len).await.is_err() {
// The guest hung up: the turn ended or the VM went away. Not an
// error — the caller already has the turn's own result.
return Ok(at);
}
let len = u32::from_be_bytes(len);
if len > MAX_REPLY {
return Err(format!("tail frame of {len} bytes exceeds the cap"));
}
let mut buf = vec![0u8; len as usize];
if s.read_exact(&mut buf).await.is_err() {
return Ok(at);
}
let v: Value = match serde_json::from_slice(&buf) {
Ok(v) => v,
Err(e) => return Err(format!("decode tail frame: {e}")),
};
if let Some(d) = v.get("data").and_then(Value::as_str) {
at = v.get("at").and_then(Value::as_u64).unwrap_or(at);
on_chunk(at, d.to_string());
}
if v.get("eof").and_then(Value::as_bool) == Some(true) {
return Ok(v.get("at").and_then(Value::as_u64).unwrap_or(at));
}
}
}
/// Resolve a backend name to the rootfs image on this node.
///
/// `None` (or `"default"`) means the golden `rootfs.ext4`; anything else selects