osobh/clawmates
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(fleet): B4.6 — a microVM reaches the API through a vsock CONNECT proxy, with an allow-list

Browse Source 
The guest still has no network interface, and now that is the design rather than
a gap. Its only route out is an HTTP CONNECT proxy: agent CLI -> 127.0.0.1:3128
in the guest -> vsock 9002 -> a per-VM Unix socket on the host -> TLS to an
allow-listed host.

Why not TAP + iptables, which is what the Firecracker write-ups do — measured,
not argued:
  - `ip tuntap add` is DENIED to the daemon user (needs CAP_NET_ADMIN), so TAP
    would need root to pre-provision devices, the same privilege detour the
    loop-mounted rootfs already forced.
  - tank's FORWARD policy is DROP with Docker and Tailscale chains, so rules
    would have to be inserted at position 1; appended ones die silently.
  - a leaked TAP is a new class of host litter to reap.
CONNECT needs no privilege at all and is better on the merits: the client hands
us the HOSTNAME, so resolution happens host-side and the guest needs no DNS or
resolv.conf; the allow-list is by name, not address; and nothing in the guest can
reach the network except through one function. The guest end parses nothing and
enforces nothing, so a compromised agent cannot argue with the policy.

Rests on one measured fact: `claude` honours HTTPS_PROXY. With the proxy at a
closed port, `claude -p` fails ConnectionRefused instead of answering.

THE RESULT: a real agent turn now completes inside a VM with no network card, on
subscription auth — `claude -p` replies VM-OK. The selftest asks for it whenever
CLAUDE_CODE_OAUTH_TOKEN is present and SKIPS loudly when it is not, since it
spends a little of the plan.

The audit log earns its keep immediately: during that turn the proxy logged
`egress DENIED http-intake.logs.us5.datadoghq.com` — the CLI's telemetry, which
the mission container permits today without anyone deciding to.

Three bugs found by the checks rather than by review:
  - `env_pairs` returned early when a caller sent no env, so the proxy address
    was never added and `curl` in a VM with a working tunnel reported "Could not
    resolve host". Absent env means "the caller sent none", not "this command
    needs no environment".
  - the deny check PASSED for the wrong reason — DNS was failing, so nothing was
    refused by the allow-list at all. It now requires a 403 from the proxy, so it
    cannot go green on a broken tunnel.
  - `host_allowed` accepted `evil.test/api.anthropic.com`, which ends with an
    allowed suffix. Hostnames are now validated against a character class, which
    also refuses IP literals so an address cannot sidestep a list of names.
  - `BufReader::into_inner()` discards buffered bytes: wrapping the stream twice
    would have dropped the start of the TLS handshake and stalled a tunnel that
    looked established. One reader now spans the request, and anything buffered
    past the headers is forwarded as payload.

`iproute2` is in agent-toolchain because it is load-bearing: the guest's `lo`
starts DOWN, and while it is down a listener on loopback BINDS and then refuses
every connection with ENETUNREACH. fcagent finds `ip` by absolute path — as pid 1
its PATH comes from the kernel, and execvp's fallback excludes /usr/sbin, where
Debian puts it.

Egress needs both ends up, so `create` reports `egress` and the guest's `ping`
reports its own half. A VM without it is legal but never silent.

Verified on tank: 16/16 with backend=claude (create 1428 ms), 12/12 on the
default rootfs, no leaked processes, VM dirs or proxy sockets. 457 tests pass,
clippy clean.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-05 12:44:59 -07:00
co-authored by Claude Opus 5
parent abc4160a89
commit ebdba34da6
5 changed files with 794 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/bins/clawmates-node/src/microvm.rs
+173 -12
View File
@@ -52,6 +52,12 @@ pub struct Vm {
pgid: i32,
workdir: PathBuf,
uds: PathBuf,
/// This VM's egress proxy (see [`crate::egress`]) and its socket. Held so
/// destroy can abort the task: a listener outliving its VM would accept a
/// connection from the NEXT VM to reuse the path and proxy it under the dead
/// one's identity.
egress: Option<tokio::task::JoinHandle<()>>,
egress_uds: PathBuf,
}
pub type Vms = Arc<Mutex<HashMap<String, Vm>>>;
@@ -245,6 +251,19 @@ pub async fn create(
.await
.map_err(|e| format!("write {}: {e}", cfg_path.display()))?;
// Bound BEFORE firecracker starts: a guest that dials the host before the
// host is listening gets a connection refused it will not retry.
let (egress_uds, egress_task) = match crate::egress::start(&uds, vm_id) {
Ok((p, t)) => (p, Some(t)),
// Not fatal — a VM is still useful for work that needs no network — but
// it must be visible. `create`'s reply says whether egress exists, and
// the guest's own `proxy` flag says whether the guest end came up.
Err(e) => {
eprintln!("microvm {vm_id}: NO EGRESS ({e}) — the guest cannot reach any network");
(PathBuf::from(format!("{}_{}", uds.display(), crate::egress::EGRESS_PORT)), None)
}
};
let log = std::fs::File::create(workdir.join("console.log"))
.map_err(|e| format!("create console.log: {e}"))?;
let errlog = log
@@ -268,16 +287,23 @@ pub async fn create(
// setsid's child IS the new group leader, and its pid is the pgid.
let pgid = child.id().ok_or("firecracker exited immediately")? as i32;
let host_egress = egress_task.is_some();
let vm = Vm {
pgid,
workdir: workdir.clone(),
uds: uds.clone(),
egress: egress_task,
egress_uds: egress_uds.clone(),
};
// Poll for the agent. 10s is generous: the measured boot-to-agent is under
// a second, so anything near the ceiling means something is wrong.
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
let mut last = String::new();
// Assigned on the only path that reaches the use below; the timeout path
// returns. Declared without a default so a future edit cannot make "we never
// asked" read as "the guest said no".
let guest_egress;
loop {
if std::time::Instant::now() > deadline {
let console = tokio::fs::read_to_string(workdir.join("console.log"))
@@ -291,7 +317,13 @@ pub async fn create(
));
}
match rpc(&uds, &json!({"op": "ping"})).await {
Ok(v) if v.get("ok").and_then(Value::as_bool) == Some(true) => break,
Ok(v) if v.get("ok").and_then(Value::as_bool) == Some(true) => {
// The guest reports whether ITS end of the tunnel is listening.
// Both ends must be up for the VM to have egress, and only the
// guest knows whether its image could bring loopback up.
guest_egress = v.get("proxy").and_then(Value::as_bool) == Some(true);
break;
}
Ok(v) => last = v.to_string(),
Err(e) => last = e,
}
@@ -306,6 +338,12 @@ pub async fn create(
// Which image actually booted, not which was asked for. A mission
// artifact that records the request cannot show that the wrong VM ran.
"rootfs": golden.display().to_string(),
// Egress needs BOTH ends. Reported rather than assumed so a caller that
// requires the network can refuse the VM up front, instead of a mission
// discovering it as an agent that cannot reach its API.
"egress": host_egress && guest_egress,
"egress_host": host_egress,
"egress_guest": guest_egress,
}))
}
@@ -381,13 +419,22 @@ async fn kill_group(pgid: i32) {
pub async fn destroy(vms: &Vms, vm_id: &str) -> Result<Value, String> {
check_id(vm_id)?;
let vm = vms.lock().await.remove(vm_id);
let (pgid, workdir, uds) = match vm {
Some(v) => (Some(v.pgid), v.workdir, v.uds),
// Not registered: still clean the path, so a VM created by a previous
let (pgid, workdir, uds, egress_uds) = match vm {
Some(v) => {
// Abort first: a live listener would keep accepting on a path the
// next VM is about to reuse.
if let Some(t) = v.egress {
t.abort();
}
(Some(v.pgid), v.workdir, v.uds, v.egress_uds)
}
// Not registered: still clean the paths, so a VM created by a previous
// incarnation of the daemon can be reaped rather than orphaned forever.
None => {
let wd = work_root().join("vms").join(vm_id);
(None, wd.clone(), wd.join("v.sock"))
let uds = wd.join("v.sock");
let eg = PathBuf::from(format!("{}_{}", uds.display(), crate::egress::EGRESS_PORT));
(None, wd.clone(), uds, eg)
}
};
if let Some(pgid) = pgid {
@@ -396,6 +443,9 @@ pub async fn destroy(vms: &Vms, vm_id: &str) -> Result<Value, String> {
// Give the group a moment to die before removing the files it has open.
tokio::time::sleep(std::time::Duration::from_millis(100)).await;
let _ = tokio::fs::remove_file(&uds).await;
// Same trap as firecracker's own socket: nothing unlinks these for us, and a
// stale file makes the next bind fail with EADDRINUSE.
let _ = tokio::fs::remove_file(&egress_uds).await;
let removed = tokio::fs::remove_dir_all(&workdir).await.is_ok();
Ok(json!({ "vm_id": vm_id, "killed": pgid.is_some(), "workdir_removed": removed }))
}
@@ -517,18 +567,21 @@ pub async fn selftest() -> bool {
}
let started = std::time::Instant::now();
match create(&vms, id, 2, 1024, backend.as_deref()).await {
Ok(v) => check(
true,
&format!("create ({} ms) {}", started.elapsed().as_millis(), v["vm_id"]),
String::new(),
),
let created = match create(&vms, id, 2, 1024, backend.as_deref()).await {
Ok(v) => {
check(
true,
&format!("create ({} ms) {}", started.elapsed().as_millis(), v["vm_id"]),
String::new(),
);
v
}
Err(e) => {
check(false, "create", e);
println!("\n1 or more checks failed");
return false;
}
}
};
// Inject a tar the way the mission checkout will travel.
let mut tar = tar::Builder::new(Vec::new());
@@ -608,6 +661,77 @@ pub async fn selftest() -> bool {
format!("{r:?}"),
);
// Egress. Both ends of the tunnel must be up, and `create` says so rather
// than leaving the caller to find out from an agent that cannot reach its
// API. Only asserted for a backend whose image is expected to have iproute2.
if backend.is_some() {
check(
created["egress"] == json!(true),
"the VM reports working egress (both ends of the tunnel)",
format!(
"host={} guest={}",
created["egress_host"], created["egress_guest"]
),
);
// The allow-list has to actually let the model API through — this is the
// check that says a mission could run — and it goes over the real tunnel:
// guest loopback → vsock → host proxy → TLS to Anthropic. `-sS -o
// /dev/null -w %{http_code}` because any 2xx/4xx from the API proves the
// connection completed; only a transport failure gives no code at all.
let r = exec(
&vms,
id,
"curl -sS -o /dev/null -w '%{http_code}' --max-time 25 https://api.anthropic.com/v1/messages",
None,
40,
None,
)
.await;
let code = r
.as_ref()
.map(|v| v["stdout"].as_str().unwrap_or_default().trim().to_string())
.unwrap_or_default();
check(
code.len() == 3 && code != "000",
"an allow-listed host is reachable from inside the VM",
format!("http_code={code:?} {r:?}"),
);
// And the denial must fire. Without this the allow-list is decoration:
// a proxy that allows everything passes the check above just as well.
let r = exec(
&vms,
id,
"curl -sS -o /dev/null -w '%{http_code}' --max-time 25 https://example.com",
None,
40,
None,
)
.await;
let out = r
.as_ref()
.map(|v| {
format!(
"{}{}",
v["stdout"].as_str().unwrap_or_default(),
v["stderr"].as_str().unwrap_or_default()
)
})
.unwrap_or_default();
// The refusal must come from THE PROXY, not from a dead network. When
// this check merely asserted "did not reach it", it passed while the
// guest had no proxy env at all and curl was failing with "Could not
// resolve host" — a green result for a broken tunnel, which is the exact
// failure shape this project keeps paying for. `403` is the status the
// proxy returns after CONNECT for a host off the list.
check(
out.contains("403") || out.contains("allow-list"),
"a host that is NOT allow-listed is refused BY THE PROXY",
format!("expected a 403 from the proxy, got: {out:?}"),
);
}
// Work produced in the guest must come back out.
let _ = exec(&vms, id, "echo PRODUCED-OK > /work/out.txt", None, 30, None).await;
let r = collect(&vms, id, "/work").await;
@@ -662,6 +786,43 @@ pub async fn selftest() -> bool {
"the guest provides git",
format!("{r:?}"),
);
// The whole track in one check: a real agent turn, in a VM with no
// network card, reaching the API through the vsock tunnel on the
// subscription credential. Everything above can pass while this
// fails, which is why it is asked separately.
//
// Gated on a token being present, and SKIPPED loudly when it is not —
// it spends a small amount of subscription budget, so it must be a
// deliberate act rather than something every infra check does.
match std::env::var("CLAUDE_CODE_OAUTH_TOKEN") {
Ok(t) if !t.trim().is_empty() => {
let r = exec(
&vms,
id,
"cd /work && claude -p 'Reply with exactly: VM-OK'",
None,
180,
// Subscription only. An ANTHROPIC_API_KEY would outrank
// this token and bill the API instead of the plan.
Some(&json!({ "CLAUDE_CODE_OAUTH_TOKEN": t })),
)
.await;
let said = r
.as_ref()
.map(|v| v["stdout"].as_str().unwrap_or_default().to_string())
.unwrap_or_default();
check(
said.contains("VM-OK"),
"the agent completes a real turn inside the VM (subscription auth)",
format!("said {said:?} — {r:?}"),
);
}
_ => println!(
"SKIP no CLAUDE_CODE_OAUTH_TOKEN in the environment — the agent's \
own turn is UNPROVEN by this run"
),
}
}
None => println!(
"SKIP no agent CLI is required of backend {} — its contents are unchecked",