osobh/clawmates
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(fleet): B2 — vm_* node ops for Firecracker microVMs

Browse Source 
create / inject / exec / collect / destroy / list, riding the node's
existing frame dispatch ({t, id, …} -> {t:"result", id, ok, output}), so
no protocol change was needed. Control is length-prefixed JSON over
vsock; the serial console stays a log, because feeding a guest over stdin
races its startup and arrives half-consumed.

DEVIATION FROM THE PLAN, deliberately: this does NOT implement
cm_sandbox::SandboxDriver. That trait is container-shaped —
attach_pty/resize_pty/argv exec — while missions need
create -> inject -> run -> collect -> destroy. Conforming would mean
building PTY-over-vsock and window-resize semantics that no mission path
calls, purely to satisfy a signature. We give up automatic RemoteDriver
marshalling; orphan reaping is a label/id sweep either way.

Three traps from the B0 spike are handled in code rather than remembered:

  - Firecracker does NOT unlink its vsock UDS on exit, so destroy unlinks
    it explicitly, and the selftest ASSERTS it is gone. Assuming the VM
    tidies up after itself is how the mission checkout accumulated four
    uid bugs.
  - firecracker is spawned via setsid and killed as a process GROUP, so a
    background child cannot outlive the VM holding its workdir open.
  - create does not return until the guest agent has answered a ping. A
    VM that booted but serves nothing is worse than one that failed, so a
    half-created VM is destroyed rather than left registered.

A vm id becomes a path component, so ids are restricted to [A-Za-z0-9_-]
and REJECTED rather than sanitised — a caller that sent `../../etc`
wanted something we should not guess at.

Verified on tank through the real Rust path, as the daemon user, with no
sudo: `clawmates-node --vm-selftest` -> 8/8, create in 986ms, and the
host left with zero firecracker processes and zero VM directories. The
selftest asserts every step, including that a destroyed VM can no longer
be exec'd; a test that only reports the steps it completed cannot
distinguish "passed" from "stopped early".

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-05 07:35:13 -07:00
co-authored by Claude Opus 5
parent b87d89f9fa
commit 2d04c5e257
5 changed files with 710 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/bins/clawmates-node/src/main.rs
+35 -1
View File
@@ -19,6 +19,7 @@ use sysinfo::{Disks, System};
use tokio::sync::{mpsc, Mutex};
use tokio_tungstenite::tungstenite::Message;
mod microvm;
mod rtc;
const B64: base64::engine::general_purpose::GeneralPurpose =
@@ -43,6 +44,15 @@ async fn main() {
selftest();
return;
}
// Exercise the microVM lifecycle against a real VM on this node. Separate
// from --selftest because it needs KVM, so it can only pass on a node that
// actually reports microvm capability.
if std::env::args().any(|a| a == "--vm-selftest") {
if !microvm::selftest().await {
std::process::exit(1);
}
return;
}
let (server, token, ts_authkey) = parse_args();
if server.is_empty() || token.is_empty() {
eprintln!("usage: clawmates-node --server <https://gateway> --token <token> [--tailscale-authkey <key>]");
@@ -108,6 +118,11 @@ async fn run(ws_url: &str) -> Result<(), Box<dyn std::error::Error>> {
let (out_tx, mut out_rx) = mpsc::unbounded_channel::<String>();
let ptys: Ptys = Arc::new(Mutex::new(HashMap::new()));
let peers: rtc::RtcPeers = Arc::new(Mutex::new(HashMap::new()));
// microVMs this connection started. Scoped to the connection deliberately:
// a reconnect must not inherit VMs it cannot prove are still alive, and
// `vm_destroy` cleans a workdir by path even for an unregistered id, so a
// VM from a previous incarnation is reapable rather than orphaned.
let vms = microvm::new_vms();
// Collect heartbeats on a dedicated thread: the metric helpers shell out to
// docker/tailscale and stat disks (blocking), which must never stall the
// async select loop (or heartbeats/pongs would starve during a slow op).
@@ -188,8 +203,9 @@ async fn run(ws_url: &str) -> Result<(), Box<dyn std::error::Error>> {
let out = out_tx.clone();
let ptys = ptys.clone();
let peers = peers.clone();
let vms = vms.clone();
let text = t.to_string();
tokio::spawn(async move { handle_frame(&text, &out, &ptys, &peers).await; });
tokio::spawn(async move { handle_frame(&text, &out, &ptys, &peers, &vms).await; });
}
Some(Ok(Message::Ping(p))) => {
match tokio::time::timeout(WRITE_DEADLINE, write.send(Message::Pong(p))).await {
@@ -520,6 +536,7 @@ async fn handle_frame(
out: &mpsc::UnboundedSender<String>,
ptys: &Ptys,
peers: &rtc::RtcPeers,
vms: &microvm::Vms,
) {
let Ok(v) = serde_json::from_str::<Value>(text) else {
return;
@@ -582,6 +599,23 @@ async fn handle_frame(
// Agent-sandbox container ops: drive the REAL DockerDriver so the
// hardening (cap-drop ALL, seccomp, no-net, read-only, non-root) is
// byte-identical to the gateway's local sandboxes.
// microVM ops. Same envelope as every other op, so adding them needed
// no protocol change. `vm_create` blocks until the guest agent answers:
// a VM that booted but serves nothing is worse than one that failed.
op @ ("vm_create" | "vm_inject" | "vm_exec" | "vm_collect" | "vm_destroy" | "vm_list") => {
if let Some(id) = v.get("id").and_then(Value::as_u64) {
let (op, v, out, vms) = (op.to_string(), v.clone(), out.clone(), vms.clone());
// Spawned: a VM boot takes ~1s and an exec can take an hour.
// Running it inline would stall heartbeats and the daemon would
// be declared offline mid-mission.
tokio::spawn(async move {
let (ok, output) = microvm::handle_op(&op, &v, &vms).await;
let _ = out.send(
json!({ "t": "result", "id": id, "ok": ok, "output": output }).to_string(),
);
});
}
}
op @ ("sb_provision" | "sb_exec" | "sb_destroy" | "sb_health" | "sb_list") => {
if let Some(id) = v.get("id").and_then(Value::as_u64) {
let (ok, output) = sb_op(op, &v).await;