feat(observability): stream a microVM turn's stdout/stderr to the platform live
The Live tab showed nothing while a turn ran, and the agent's own account of it
went to stderr on the node and nowhere a user could reach. This is the path that
carries it.
The blocker was the guest agent. `fcagent` handled one connection at a time,
inline, so during an hour-long turn the VM accepted nothing — which is why every
existing probe (subagents, stop-gate blocks, cap) runs AFTER the turn rather than
during it. It now spawns a thread per connection, wrapped in `catch_unwind`
because this process is pid 1: a panic used to take the accept loop with it, and
an unbootable VM is a far worse outcome than a missing log. A failed spawn logs
and keeps accepting rather than dropping the listener.
PROVED against a live VM before building on it, since "sound reasoning about this
system" and "measurement" have diverged repeatedly today. Patched rootfs, booted
under Firecracker, ran an 8s exec and a concurrent tail:
exec took 8.0s ok=True
+0.0s 'line1\nline2\n' +1.2s 'line4\n' +3.2s 'line6\n' +6.0s 'DONE\n'
VERDICT: CONCURRENT — tail returned data before exec finished
The rest is the pattern the terminal already uses. New `tail` op streams a file
by OFFSET (so a dropped link resumes instead of replaying, and the tail always
terminates — one that never returns pins a thread for the life of the VM). The
node follows the log alongside the turn and pushes `Uplink::VmOut { run_id, at,
data }` over the WebSocket it already holds, mirroring `PtyOut`. The server does
what `PtyOut` deliberately does not: it APPENDS to the run's checkpoint as well
as fanning out, because a terminal has no history worth keeping and a mission log
is the record of what the agent did. `run_events_sse` emits the new bytes as
`step` events, which the live pane already renders — no frontend change.
The turn is `tee`d, not redirected: the file feeds the live stream and stdout
still becomes `VmOutcome::summary`. A redirect would have produced a live view
and an empty summary, which is the same green-and-empty shape as the bug this
fixes. Tested, along with the log living outside the collected tree so it never
lands in a user's delivered diff.
246 lib tests, 20 binaries; node and fcagent build clean.
This commit is contained in:
@@ -88,9 +88,10 @@ fn check_id(vm_id: &str) -> Result<(), String> {
|
||||
/// in pieces, and reading "whatever was available" would parse a truncated
|
||||
/// object as a complete one. Claude's `stream-json` output makes that routine
|
||||
/// rather than theoretical.
|
||||
async fn rpc(uds: &Path, req: &Value) -> Result<Value, String> {
|
||||
const MAX_REPLY: u32 = 256 * 1024 * 1024;
|
||||
/// Largest reply frame accepted from a guest, for both one-shot rpc and tail.
|
||||
const MAX_REPLY: u32 = 256 * 1024 * 1024;
|
||||
|
||||
async fn rpc(uds: &Path, req: &Value) -> Result<Value, String> {
|
||||
let mut s = UnixStream::connect(uds)
|
||||
.await
|
||||
.map_err(|e| format!("connect {}: {e}", uds.display()))?;
|
||||
@@ -133,6 +134,86 @@ async fn rpc(uds: &Path, req: &Value) -> Result<Value, String> {
|
||||
serde_json::from_slice(&buf).map_err(|e| format!("decode reply: {e}"))
|
||||
}
|
||||
|
||||
/// Follow a file inside a guest, handing each chunk to `on_chunk` as it arrives.
|
||||
///
|
||||
/// Unlike [`rpc`], which is one request and one reply, this keeps the connection
|
||||
/// open and reads MANY framed replies — the guest's `tail` op emits one per
|
||||
/// chunk and a final `eof`. That is what makes a turn's output visible while the
|
||||
/// turn is still running, and it works only because the guest agent now accepts
|
||||
/// concurrent connections.
|
||||
///
|
||||
/// Returns the byte offset reached, so a caller that reconnects resumes instead
|
||||
/// of replaying.
|
||||
pub async fn tail_into<F>(
|
||||
vms: &Vms,
|
||||
vm_id: &str,
|
||||
path: &str,
|
||||
from: u64,
|
||||
mut on_chunk: F,
|
||||
) -> Result<u64, String>
|
||||
where
|
||||
F: FnMut(u64, String),
|
||||
{
|
||||
let uds = uds_of(vms, vm_id).await?;
|
||||
let mut s = UnixStream::connect(&uds)
|
||||
.await
|
||||
.map_err(|e| format!("connect {}: {e}", uds.display()))?;
|
||||
s.write_all(b"CONNECT 9001\n")
|
||||
.await
|
||||
.map_err(|e| format!("vsock CONNECT: {e}"))?;
|
||||
let mut ack = [0u8; 64];
|
||||
let n = s
|
||||
.read(&mut ack)
|
||||
.await
|
||||
.map_err(|e| format!("vsock CONNECT ack: {e}"))?;
|
||||
if !String::from_utf8_lossy(&ack[..n]).starts_with("OK") {
|
||||
return Err("vsock refused the tail connection".into());
|
||||
}
|
||||
|
||||
let req = json!({
|
||||
"op": "tail", "path": path, "from": from,
|
||||
// Long enough that a quiet agent is not mistaken for a finished one,
|
||||
// short enough that the thread is released soon after the turn ends.
|
||||
"idle_ms": 15_000, "max_secs": 3_600,
|
||||
});
|
||||
let body = serde_json::to_vec(&req).map_err(|e| format!("encode tail: {e}"))?;
|
||||
s.write_all(&(body.len() as u32).to_be_bytes())
|
||||
.await
|
||||
.map_err(|e| format!("write tail length: {e}"))?;
|
||||
s.write_all(&body)
|
||||
.await
|
||||
.map_err(|e| format!("write tail body: {e}"))?;
|
||||
|
||||
let mut at = from;
|
||||
loop {
|
||||
let mut len = [0u8; 4];
|
||||
if s.read_exact(&mut len).await.is_err() {
|
||||
// The guest hung up: the turn ended or the VM went away. Not an
|
||||
// error — the caller already has the turn's own result.
|
||||
return Ok(at);
|
||||
}
|
||||
let len = u32::from_be_bytes(len);
|
||||
if len > MAX_REPLY {
|
||||
return Err(format!("tail frame of {len} bytes exceeds the cap"));
|
||||
}
|
||||
let mut buf = vec![0u8; len as usize];
|
||||
if s.read_exact(&mut buf).await.is_err() {
|
||||
return Ok(at);
|
||||
}
|
||||
let v: Value = match serde_json::from_slice(&buf) {
|
||||
Ok(v) => v,
|
||||
Err(e) => return Err(format!("decode tail frame: {e}")),
|
||||
};
|
||||
if let Some(d) = v.get("data").and_then(Value::as_str) {
|
||||
at = v.get("at").and_then(Value::as_u64).unwrap_or(at);
|
||||
on_chunk(at, d.to_string());
|
||||
}
|
||||
if v.get("eof").and_then(Value::as_bool) == Some(true) {
|
||||
return Ok(v.get("at").and_then(Value::as_u64).unwrap_or(at));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Resolve a backend name to the rootfs image on this node.
|
||||
///
|
||||
/// `None` (or `"default"`) means the golden `rootfs.ext4`; anything else selects
|
||||
|
||||
Reference in New Issue
Block a user