feat(observability): stream a microVM turn's stdout/stderr to the platform live
The Live tab showed nothing while a turn ran, and the agent's own account of it
went to stderr on the node and nowhere a user could reach. This is the path that
carries it.
The blocker was the guest agent. `fcagent` handled one connection at a time,
inline, so during an hour-long turn the VM accepted nothing — which is why every
existing probe (subagents, stop-gate blocks, cap) runs AFTER the turn rather than
during it. It now spawns a thread per connection, wrapped in `catch_unwind`
because this process is pid 1: a panic used to take the accept loop with it, and
an unbootable VM is a far worse outcome than a missing log. A failed spawn logs
and keeps accepting rather than dropping the listener.
PROVED against a live VM before building on it, since "sound reasoning about this
system" and "measurement" have diverged repeatedly today. Patched rootfs, booted
under Firecracker, ran an 8s exec and a concurrent tail:
exec took 8.0s ok=True
+0.0s 'line1\nline2\n' +1.2s 'line4\n' +3.2s 'line6\n' +6.0s 'DONE\n'
VERDICT: CONCURRENT — tail returned data before exec finished
The rest is the pattern the terminal already uses. New `tail` op streams a file
by OFFSET (so a dropped link resumes instead of replaying, and the tail always
terminates — one that never returns pins a thread for the life of the VM). The
node follows the log alongside the turn and pushes `Uplink::VmOut { run_id, at,
data }` over the WebSocket it already holds, mirroring `PtyOut`. The server does
what `PtyOut` deliberately does not: it APPENDS to the run's checkpoint as well
as fanning out, because a terminal has no history worth keeping and a mission log
is the record of what the agent did. `run_events_sse` emits the new bytes as
`step` events, which the live pane already renders — no frontend change.
The turn is `tee`d, not redirected: the file feeds the live stream and stdout
still becomes `VmOutcome::summary`. A redirect would have produced a live view
and an empty summary, which is the same green-and-empty shape as the bug this
fixes. Tested, along with the log living outside the collected tree so it never
lands in a user's delivered diff.
246 lib tests, 20 binaries; node and fcagent build clean.
This commit is contained in:
@@ -621,7 +621,40 @@ async fn handle_frame(
|
||||
// Running it inline would stall heartbeats and the daemon would
|
||||
// be declared offline mid-mission.
|
||||
tokio::spawn(async move {
|
||||
// While an `exec` runs, follow the turn's log and push each
|
||||
// chunk to the server as it appears. The guest agent accepts
|
||||
// concurrent connections (proved against a live VM: a tail
|
||||
// returned data second-by-second while an 8s exec was still
|
||||
// running), so this does not wait for, or delay, the turn.
|
||||
//
|
||||
// Only for `vm_exec`, and only when the caller named a run to
|
||||
// attribute the output to — a probe exec has nothing to
|
||||
// stream and no subscriber.
|
||||
let tail = (op == "vm_exec")
|
||||
.then(|| {
|
||||
let run_id = v.get("run_id").and_then(Value::as_str)?.to_string();
|
||||
let log_path = v
|
||||
.get("log_path")
|
||||
.and_then(Value::as_str)
|
||||
.unwrap_or("/root/agent.log")
|
||||
.to_string();
|
||||
let vm_id = v.get("vm_id").and_then(Value::as_str)?.to_string();
|
||||
Some(tokio::spawn(stream_vm_log(
|
||||
vms.clone(),
|
||||
vm_id,
|
||||
run_id,
|
||||
log_path,
|
||||
out.clone(),
|
||||
)))
|
||||
})
|
||||
.flatten();
|
||||
let (ok, output) = microvm::handle_op(&op, &v, &vms).await;
|
||||
// The turn is over; the tail's own idle timeout will end it,
|
||||
// but aborting is immediate and leaves no thread waiting on a
|
||||
// file that stopped growing.
|
||||
if let Some(t) = tail {
|
||||
t.abort();
|
||||
}
|
||||
let _ = out.send(
|
||||
json!({ "t": "result", "id": id, "ok": ok, "output": output }).to_string(),
|
||||
);
|
||||
@@ -859,6 +892,53 @@ fn spawn_command_pty(argv: &[String], cols: u16, rows: u16) -> Result<PtyParts,
|
||||
spawn_pty(c, cols, rows)
|
||||
}
|
||||
|
||||
/// Follow a running turn's log inside a VM and push each chunk to the server.
|
||||
///
|
||||
/// The other half of the observability path: the guest tails the file, this
|
||||
/// forwards what it reads over the WebSocket the daemon already holds, and the
|
||||
/// server appends it to the run so the live pane and the Output tab both have it.
|
||||
///
|
||||
/// Reconnects on a dropped tail, resuming from the last offset — following by
|
||||
/// OFFSET rather than holding one socket open forever is what makes that cheap.
|
||||
/// It gives up after a few consecutive failures rather than spinning: by then
|
||||
/// the VM is gone and the turn's own result is the record.
|
||||
async fn stream_vm_log(
|
||||
vms: microvm::Vms,
|
||||
vm_id: String,
|
||||
run_id: String,
|
||||
log_path: String,
|
||||
out: tokio::sync::mpsc::UnboundedSender<String>,
|
||||
) {
|
||||
let mut at: u64 = 0;
|
||||
let mut failures = 0;
|
||||
while failures < 3 {
|
||||
let sent = out.clone();
|
||||
let rid = run_id.clone();
|
||||
match microvm::tail_into(&vms, &vm_id, &log_path, at, move |offset, data| {
|
||||
let _ = sent.send(
|
||||
json!({ "t": "vm_out", "run_id": rid, "at": offset, "data": data }).to_string(),
|
||||
);
|
||||
})
|
||||
.await
|
||||
{
|
||||
Ok(reached) => {
|
||||
// No progress and no error means the guest reported EOF: the log
|
||||
// stopped growing, so the turn is done writing.
|
||||
if reached == at {
|
||||
return;
|
||||
}
|
||||
at = reached;
|
||||
failures = 0;
|
||||
}
|
||||
Err(e) => {
|
||||
failures += 1;
|
||||
eprintln!("clawmates-node: tail of {vm_id} for run {run_id} failed: {e}");
|
||||
tokio::time::sleep(std::time::Duration::from_secs(2)).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Spawn a host login shell in a PTY; stream its output back as pty_out frames.
|
||||
async fn open_pty(
|
||||
sid: u64,
|
||||
|
||||
Reference in New Issue
Block a user