feat(fleet): B4.6 — a microVM reaches the API through a vsock CONNECT proxy, with an allow-list
The guest still has no network interface, and now that is the design rather than
a gap. Its only route out is an HTTP CONNECT proxy: agent CLI -> 127.0.0.1:3128
in the guest -> vsock 9002 -> a per-VM Unix socket on the host -> TLS to an
allow-listed host.
Why not TAP + iptables, which is what the Firecracker write-ups do — measured,
not argued:
- `ip tuntap add` is DENIED to the daemon user (needs CAP_NET_ADMIN), so TAP
would need root to pre-provision devices, the same privilege detour the
loop-mounted rootfs already forced.
- tank's FORWARD policy is DROP with Docker and Tailscale chains, so rules
would have to be inserted at position 1; appended ones die silently.
- a leaked TAP is a new class of host litter to reap.
CONNECT needs no privilege at all and is better on the merits: the client hands
us the HOSTNAME, so resolution happens host-side and the guest needs no DNS or
resolv.conf; the allow-list is by name, not address; and nothing in the guest can
reach the network except through one function. The guest end parses nothing and
enforces nothing, so a compromised agent cannot argue with the policy.
Rests on one measured fact: `claude` honours HTTPS_PROXY. With the proxy at a
closed port, `claude -p` fails ConnectionRefused instead of answering.
THE RESULT: a real agent turn now completes inside a VM with no network card, on
subscription auth — `claude -p` replies VM-OK. The selftest asks for it whenever
CLAUDE_CODE_OAUTH_TOKEN is present and SKIPS loudly when it is not, since it
spends a little of the plan.
The audit log earns its keep immediately: during that turn the proxy logged
`egress DENIED http-intake.logs.us5.datadoghq.com` — the CLI's telemetry, which
the mission container permits today without anyone deciding to.
Three bugs found by the checks rather than by review:
- `env_pairs` returned early when a caller sent no env, so the proxy address
was never added and `curl` in a VM with a working tunnel reported "Could not
resolve host". Absent env means "the caller sent none", not "this command
needs no environment".
- the deny check PASSED for the wrong reason — DNS was failing, so nothing was
refused by the allow-list at all. It now requires a 403 from the proxy, so it
cannot go green on a broken tunnel.
- `host_allowed` accepted `evil.test/api.anthropic.com`, which ends with an
allowed suffix. Hostnames are now validated against a character class, which
also refuses IP literals so an address cannot sidestep a list of names.
- `BufReader::into_inner()` discards buffered bytes: wrapping the stream twice
would have dropped the start of the TLS handshake and stalled a tunnel that
looked established. One reader now spans the request, and anything buffered
past the headers is forwarded as payload.
`iproute2` is in agent-toolchain because it is load-bearing: the guest's `lo`
starts DOWN, and while it is down a listener on loopback BINDS and then refuses
every connection with ENETUNREACH. fcagent finds `ip` by absolute path — as pid 1
its PATH comes from the kernel, and execvp's fallback excludes /usr/sbin, where
Debian puts it.
Egress needs both ends up, so `create` reports `egress` and the guest's `ping`
reports its own half. A VM without it is legal but never silent.
Verified on tank: 16/16 with backend=claude (create 1428 ms), 12/12 on the
default rootfs, no leaked processes, VM dirs or proxy sockets. 457 tests pass,
clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
co-authored by
Claude Opus 5
Claude Opus 5
parent
abc4160a89
commit
ebdba34da6
+215
-13
@@ -25,15 +25,39 @@
|
||||
//! and `crates/cm-api/src/microvm_client.rs` speak this and needed no change.
|
||||
|
||||
use std::io::{Read, Write};
|
||||
use std::net::TcpListener;
|
||||
use std::os::unix::process::CommandExt;
|
||||
use std::path::Path;
|
||||
use std::process::{Command, Stdio};
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use base64::Engine;
|
||||
use serde_json::{json, Value};
|
||||
|
||||
const PORT: u32 = 9001;
|
||||
/// Guest-side egress proxy. The VM has **no network interface at all** — see
|
||||
/// `microvm.rs`, whose machine config declares no `network-interfaces` — so an
|
||||
/// agent CLI cannot reach the model API on its own. It reaches it by honouring
|
||||
/// `HTTPS_PROXY`, which is measured, not assumed: with the proxy pointed at a
|
||||
/// closed port, `claude -p` fails with `ConnectionRefused` instead of answering.
|
||||
///
|
||||
/// This listener is a dumb byte pump. It parses nothing and enforces nothing:
|
||||
/// the `CONNECT` request travels verbatim to the host, which speaks HTTP CONNECT
|
||||
/// and owns the allow-list. Keeping policy on the host means nothing running in
|
||||
/// the guest — including a compromised agent — can talk it into a different
|
||||
/// answer.
|
||||
const PROXY_PORT: u16 = 3128;
|
||||
/// Host-side vsock port the tunnel lands on. Firecracker's convention for a
|
||||
/// guest-initiated connection is that the HOST listens on `<uds_path>_<port>`.
|
||||
const EGRESS_PORT: u32 = 9002;
|
||||
/// `VMADDR_CID_HOST` — the hypervisor side of the vsock.
|
||||
const HOST_CID: u32 = 2;
|
||||
|
||||
/// Whether the egress proxy is actually listening. Reported by `ping` so the
|
||||
/// host can refuse to hand a mission to a VM with no way out, rather than
|
||||
/// discovering it as an agent that hangs.
|
||||
static PROXY_UP: AtomicBool = AtomicBool::new(false);
|
||||
/// Cap on a single request. A hostile or broken host must not be able to make
|
||||
/// pid 1 allocate without bound and get the VM OOM-killed.
|
||||
const MAX_REQUEST: u32 = 512 * 1024 * 1024;
|
||||
@@ -59,6 +83,8 @@ fn main() {
|
||||
.status();
|
||||
}
|
||||
|
||||
start_egress_proxy();
|
||||
|
||||
let listener = match vsock::VsockListener::bind_with_cid_port(libc_vmaddr_cid_any(), PORT) {
|
||||
Ok(l) => l,
|
||||
Err(e) => {
|
||||
@@ -94,6 +120,99 @@ fn libc_vmaddr_cid_any() -> u32 {
|
||||
u32::MAX
|
||||
}
|
||||
|
||||
/// Bring up loopback and start the egress tunnel.
|
||||
///
|
||||
/// Loopback is not optional and not free: the guest's `lo` exists but starts
|
||||
/// **down**, and while it is down a listener on 127.0.0.1 *binds successfully*
|
||||
/// and then refuses every connection with `ENETUNREACH`. A bind-only check would
|
||||
/// have reported a working proxy. So `lo` goes up first, via `ip` — which is why
|
||||
/// `iproute2` is in the agent images.
|
||||
///
|
||||
/// Failure here is recorded, not fatal: exec still works, so a VM is still
|
||||
/// useful for work that needs no network. It is reported through `ping` so the
|
||||
/// host can decide, instead of a mission discovering it as an agent that hangs.
|
||||
fn start_egress_proxy() {
|
||||
// Absolute paths, not `Command::new("ip")`. This process is pid 1, so its
|
||||
// PATH is whatever the kernel handed it — and when PATH is unset, `execvp`
|
||||
// falls back to a default that does NOT include `/usr/sbin`, which is exactly
|
||||
// where Debian puts `ip`. Searching by name would fail on an image that has
|
||||
// it, and the symptom would be a VM with no egress and no explanation.
|
||||
const IP_CANDIDATES: &[&str] = &["/usr/sbin/ip", "/sbin/ip", "/usr/bin/ip", "/bin/ip"];
|
||||
let Some(ip) = IP_CANDIDATES.iter().find(|p| Path::new(p).exists()) else {
|
||||
eprintln!(
|
||||
"FC-AGENT-NO-PROXY no `ip` binary in {IP_CANDIDATES:?} — no egress; \
|
||||
add iproute2 to this image"
|
||||
);
|
||||
return;
|
||||
};
|
||||
match Command::new(ip).args(["link", "set", "lo", "up"]).status() {
|
||||
Ok(s) if s.success() => {}
|
||||
other => {
|
||||
eprintln!("FC-AGENT-NO-PROXY `{ip} link set lo up` failed ({other:?}) — no egress");
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
let listener = match TcpListener::bind(("127.0.0.1", PROXY_PORT)) {
|
||||
Ok(l) => l,
|
||||
Err(e) => {
|
||||
eprintln!("FC-AGENT-NO-PROXY could not listen on 127.0.0.1:{PROXY_PORT}: {e}");
|
||||
return;
|
||||
}
|
||||
};
|
||||
PROXY_UP.store(true, Ordering::Relaxed);
|
||||
println!("FC-AGENT-PROXY listening on 127.0.0.1:{PROXY_PORT} -> vsock {EGRESS_PORT}");
|
||||
let _ = std::io::stdout().flush();
|
||||
|
||||
std::thread::spawn(move || {
|
||||
for c in listener.incoming() {
|
||||
match c {
|
||||
// One thread per connection. An agent CLI opens several at once,
|
||||
// and serving them in sequence would look like a hang.
|
||||
Ok(tcp) => {
|
||||
std::thread::spawn(move || {
|
||||
if let Err(e) = tunnel(tcp) {
|
||||
eprintln!("FC-AGENT-PROXY-ERROR {e}");
|
||||
}
|
||||
});
|
||||
}
|
||||
Err(e) => eprintln!("FC-AGENT-PROXY-ERROR accept: {e}"),
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/// Splice one TCP connection onto a fresh vsock connection to the host.
|
||||
///
|
||||
/// No parsing: whatever the client sent — `CONNECT host:443`, or an absolute-form
|
||||
/// request — is the host's business. The host answers with real HTTP, so a
|
||||
/// refusal reaches the client as a status code rather than a dropped socket.
|
||||
fn tunnel(tcp: std::net::TcpStream) -> Result<(), String> {
|
||||
let vs = vsock::VsockStream::connect_with_cid_port(HOST_CID, EGRESS_PORT)
|
||||
.map_err(|e| format!("vsock connect to host:{EGRESS_PORT}: {e}"))?;
|
||||
|
||||
let (mut tcp_r, mut tcp_w) = (
|
||||
tcp.try_clone().map_err(|e| format!("clone tcp: {e}"))?,
|
||||
tcp,
|
||||
);
|
||||
let (mut vs_r, mut vs_w) = (
|
||||
vs.try_clone().map_err(|e| format!("clone vsock: {e}"))?,
|
||||
vs,
|
||||
);
|
||||
|
||||
// Each direction gets its own thread, and each shuts its peer's write side
|
||||
// down when it ends. Without the shutdown the other half blocks forever on a
|
||||
// half-closed connection and the CLI waits out its own timeout.
|
||||
let up = std::thread::spawn(move || {
|
||||
let _ = std::io::copy(&mut tcp_r, &mut vs_w);
|
||||
let _ = vs_w.shutdown(std::net::Shutdown::Write);
|
||||
});
|
||||
let _ = std::io::copy(&mut vs_r, &mut tcp_w);
|
||||
let _ = tcp_w.shutdown(std::net::Shutdown::Write);
|
||||
let _ = up.join();
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn serve_one(s: &mut vsock::VsockStream) -> Result<(), String> {
|
||||
let mut len = [0u8; 4];
|
||||
s.read_exact(&mut len)
|
||||
@@ -127,7 +246,14 @@ fn reply(s: &mut vsock::VsockStream, v: &Value) -> Result<(), String> {
|
||||
fn handle(req: &Value) -> Value {
|
||||
let op = req.get("op").and_then(Value::as_str).unwrap_or_default();
|
||||
match op {
|
||||
"ping" => json!({ "ok": true, "pid": std::process::id() }),
|
||||
"ping" => json!({
|
||||
"ok": true,
|
||||
"pid": std::process::id(),
|
||||
// The host refuses to run a mission in a VM with no way out; this is
|
||||
// how it knows. Reported rather than assumed because the image, not
|
||||
// this binary, decides whether loopback can come up.
|
||||
"proxy": PROXY_UP.load(Ordering::Relaxed),
|
||||
}),
|
||||
"exec" => op_exec(req),
|
||||
"put" => op_put(req),
|
||||
"get" => op_get(req),
|
||||
@@ -152,18 +278,21 @@ fn handle(req: &Value) -> Value {
|
||||
/// Errors name the key and never the value: the value is the secret, and an
|
||||
/// error string travels back over the wire and into logs.
|
||||
fn env_pairs(req: &Value) -> Result<Vec<(String, String)>, String> {
|
||||
let Some(env) = req.get("env") else {
|
||||
return Ok(Vec::new());
|
||||
// Absent or `null` means the caller sent no variables of its own — which is
|
||||
// NOT the same as "this command needs no environment". Both cases still get
|
||||
// the proxy address below; returning early here meant every exec that passed
|
||||
// no env ran with no HTTPS_PROXY, and the symptom was `curl` reporting
|
||||
// "Could not resolve host" from a guest that had a working tunnel.
|
||||
let empty = serde_json::Map::new();
|
||||
let map = match req.get("env") {
|
||||
None => &empty,
|
||||
Some(v) if v.is_null() => &empty,
|
||||
// Anything else that is not an object is a caller bug.
|
||||
Some(v) => v
|
||||
.as_object()
|
||||
.ok_or("exec env must be an object of name → string")?,
|
||||
};
|
||||
// `null` means "nothing to add" — that is what a caller with no credentials
|
||||
// serialises. Anything else that is not an object is a caller bug.
|
||||
if env.is_null() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let Some(map) = env.as_object() else {
|
||||
return Err("exec env must be an object of name → string".into());
|
||||
};
|
||||
let mut out = Vec::with_capacity(map.len());
|
||||
let mut out = Vec::with_capacity(map.len() + 3);
|
||||
for (k, v) in map {
|
||||
let Some(val) = v.as_str() else {
|
||||
return Err(format!("exec env {k}: value must be a string"));
|
||||
@@ -182,7 +311,38 @@ fn env_pairs(req: &Value) -> Result<Vec<(String, String)>, String> {
|
||||
}
|
||||
out.push((k.clone(), val.to_string()));
|
||||
}
|
||||
Ok(out)
|
||||
|
||||
Ok(with_proxy_env(out, PROXY_UP.load(Ordering::Relaxed)))
|
||||
}
|
||||
|
||||
/// Add the proxy variables the guest's own listener serves.
|
||||
///
|
||||
/// The agent runs the proxy, so the agent declares where it is. Deriving this on
|
||||
/// the host would mean two places agreeing on a port number, and the one that
|
||||
/// drifts is the one nobody tests.
|
||||
///
|
||||
/// Explicit caller values win: a caller can still point a command elsewhere or
|
||||
/// switch the proxy off for it. Matched case-insensitively because the lowercase
|
||||
/// spellings are equally conventional and a duplicate would leave which one
|
||||
/// applies up to the shell.
|
||||
fn with_proxy_env(mut env: Vec<(String, String)>, proxy_up: bool) -> Vec<(String, String)> {
|
||||
if !proxy_up {
|
||||
return env;
|
||||
}
|
||||
let addr = format!("http://127.0.0.1:{PROXY_PORT}");
|
||||
for (k, v) in [
|
||||
("HTTPS_PROXY", addr.as_str()),
|
||||
("HTTP_PROXY", addr.as_str()),
|
||||
// Without this the client would ask the proxy to reach the proxy.
|
||||
("NO_PROXY", "localhost,127.0.0.1"),
|
||||
] {
|
||||
// `eq_ignore_ascii_case` covers the lowercase spelling, which is equally
|
||||
// conventional; setting both would leave which one applies to the client.
|
||||
if !env.iter().any(|(have, _)| have.eq_ignore_ascii_case(k)) {
|
||||
env.push((k.to_string(), v.to_string()));
|
||||
}
|
||||
}
|
||||
env
|
||||
}
|
||||
|
||||
fn op_exec(req: &Value) -> Value {
|
||||
@@ -347,6 +507,48 @@ fn op_get(req: &Value) -> Value {
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// The CLI reaches the API only by honouring HTTPS_PROXY (measured: with the
|
||||
/// proxy at a closed port, `claude -p` fails ConnectionRefused instead of
|
||||
/// answering), so a VM whose proxy is up must hand it the address.
|
||||
#[test]
|
||||
fn the_proxy_address_is_declared_when_the_proxy_is_up() {
|
||||
let env = with_proxy_env(vec![], true);
|
||||
let get = |k: &str| {
|
||||
env.iter()
|
||||
.find(|(a, _)| a == k)
|
||||
.map(|(_, v)| v.as_str())
|
||||
.unwrap_or("")
|
||||
};
|
||||
assert_eq!(get("HTTPS_PROXY"), "http://127.0.0.1:3128");
|
||||
assert_eq!(get("HTTP_PROXY"), "http://127.0.0.1:3128");
|
||||
// Otherwise the client asks the proxy to reach the proxy.
|
||||
assert!(get("NO_PROXY").contains("127.0.0.1"));
|
||||
}
|
||||
|
||||
/// And a VM with no proxy must not claim one: pointing a CLI at a listener
|
||||
/// that is not there turns "no egress" into a connection error mid-run
|
||||
/// instead of a fact the host can check before it starts.
|
||||
#[test]
|
||||
fn no_proxy_address_is_declared_when_the_proxy_is_down() {
|
||||
assert!(with_proxy_env(vec![], false).is_empty());
|
||||
}
|
||||
|
||||
/// An explicit value from the caller wins, in either spelling — otherwise
|
||||
/// both would be set and which one applies would be up to the client.
|
||||
#[test]
|
||||
fn an_explicit_proxy_setting_is_not_overridden() {
|
||||
let env = with_proxy_env(
|
||||
vec![("https_proxy".into(), "http://elsewhere:8080".into())],
|
||||
true,
|
||||
);
|
||||
let proxies: Vec<&str> = env
|
||||
.iter()
|
||||
.filter(|(k, _)| k.eq_ignore_ascii_case("https_proxy"))
|
||||
.map(|(_, v)| v.as_str())
|
||||
.collect();
|
||||
assert_eq!(proxies, vec!["http://elsewhere:8080"]);
|
||||
}
|
||||
|
||||
/// The credential has to actually reach the command. This is the whole
|
||||
/// point of the op, and the failure it prevents is silent: a `claude -p`
|
||||
/// with no token hangs rather than erroring.
|
||||
|
||||
Reference in New Issue
Block a user