feat(backend): local-ornith — a mission backend served by the node's own GPU
Claude Code pointed at the Ollama already installed on every GPU node. Ollama has served a native Anthropic-compatible /v1/messages since v0.14, so this is an env contract rather than a translation layer — the fourth variation on the same idea as agent-glm and agent-kimi. The route is NOT the egress proxy, and that is the design. `egress` speaks CONNECT, takes a destination from the guest, resolves it and decides; every one of those powers is a liability, which is why it refuses non-443 ports and IP literals after a unit test caught them being bypassed. Routing a local model through it would have meant relaxing both. `local_model` is the opposite shape: there is no destination in the protocol. fcagent listens on guest 127.0.0.1:11434 and pumps to vsock 9003; the node splices that onto its own 127.0.0.1:11434 and copies bytes. A compromised guest cannot redirect it because there is nothing to redirect — it is a pipe, not a proxy, and strictly narrower than anything an allow-list could express. The bytes never touch a network, so there is no wire for TLS to protect, and Ollama stays bound to loopback rather than being exposed on the tailnet. The socket is bound only for a backend declared to use a local model, so a `local-ornith` VM reaches the forge through egress and nothing else, while every other backend's guest port simply refuses. Both halves have negative controls. `scripts/fleet-model-setup.sh` exists because of one measurement: stock ornith:9b reported input_tokens=2050 for a 48000-word prompt and answered as though nothing had been dropped. Ollama's default window is ~2K whatever the model card says, and it truncates silently — the exact failure an agent turn would hit and never report. The script pins num_ctx=131072 into a derived tag and then PROVES both the window and tool calling before declaring success. Verified on architect: ~65536 words -> 65604 input tokens, stop_reason=tool_use. Placement needs no new capability key: building the rootfs only on GPU nodes means `nodes::online_for_backend`'s existing `rootfs @> ["local-ornith"]` predicate does the affinity, so morpheus never offers the backend. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
e96c5143bc
commit
f56d41f5b7
@@ -79,6 +79,15 @@ fn provider_hosts(backend: Option<&str>) -> &'static [&'static str] {
|
|||||||
// a different account namespace and rejects that key. Only the host the
|
// a different account namespace and rejects that key. Only the host the
|
||||||
// `agent-kimi` image bakes in.
|
// `agent-kimi` image bakes in.
|
||||||
Some("kimi") => &["api.kimi.com"],
|
Some("kimi") => &["api.kimi.com"],
|
||||||
|
// A locally-hosted model reaches NOTHING through this proxy. Its route
|
||||||
|
// is `crate::local_model` — a vsock pipe to the node's own loopback,
|
||||||
|
// with no destination in the protocol — so the correct allow-list here
|
||||||
|
// is the empty one, and it falls through to the branch below.
|
||||||
|
//
|
||||||
|
// Spelled out rather than left implicit because the temptation was to
|
||||||
|
// widen this proxy instead: an entry here would have meant relaxing the
|
||||||
|
// 443-only rule AND the IP-literal refusal, both of which exist because
|
||||||
|
// a unit test caught them being bypassed.
|
||||||
// Fail closed: a backend nobody taught this function about reaches the
|
// Fail closed: a backend nobody taught this function about reaches the
|
||||||
// forge and no model API. It cannot silently borrow another provider's
|
// forge and no model API. It cannot silently borrow another provider's
|
||||||
// door, which is the failure this split exists to prevent.
|
// door, which is the failure this split exists to prevent.
|
||||||
@@ -449,6 +458,27 @@ mod tests {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// A raw address must not sidestep a list written in names.
|
/// A raw address must not sidestep a list written in names.
|
||||||
|
/// A local-model backend gets NO egress, and the 443 rule is untouched.
|
||||||
|
///
|
||||||
|
/// The alternative design routed the node's Ollama through this proxy, which
|
||||||
|
/// would have meant permitting port 11434 and an address the guest names.
|
||||||
|
/// Both are refused here, still, and a `local-ornith` VM reaches the forge
|
||||||
|
/// and nothing else — its model lives on the other socket entirely.
|
||||||
|
#[test]
|
||||||
|
fn a_local_model_backend_gets_no_egress_and_no_new_port() {
|
||||||
|
let allow = allow_list_for(Some("local-ornith"));
|
||||||
|
assert!(
|
||||||
|
allow.iter().all(|a| a == "git.redclaw.dev"),
|
||||||
|
"a local backend must reach only the forge, got {allow:?}"
|
||||||
|
);
|
||||||
|
for h in ["api.anthropic.com", "api.z.ai", "api.kimi.com", "127.0.0.1"] {
|
||||||
|
assert!(!host_allowed(h, &allow), "{h} must NOT be reachable");
|
||||||
|
}
|
||||||
|
// The rules this design exists to avoid loosening.
|
||||||
|
assert!(parse_target("anything:11434").is_err());
|
||||||
|
assert!(parse_target("127.0.0.1:443").is_ok_and(|(h, _)| !host_allowed(&h, &allow)));
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn an_ip_literal_is_not_allowed() {
|
fn an_ip_literal_is_not_allowed() {
|
||||||
let a = vec![".anthropic.com".to_string()];
|
let a = vec![".anthropic.com".to_string()];
|
||||||
|
|||||||
@@ -0,0 +1,167 @@
|
|||||||
|
//! Host side of a microVM's route to the node's OWN locally-hosted model.
|
||||||
|
//!
|
||||||
|
//! # Why this is not the egress proxy
|
||||||
|
//!
|
||||||
|
//! [`crate::egress`] exists so an agent can reach the public internet under an
|
||||||
|
//! allow-list: it speaks HTTP `CONNECT`, takes a destination from the guest,
|
||||||
|
//! resolves it, and decides. Every one of those powers is a liability, which is
|
||||||
|
//! why that module is careful about ports, IP literals and suffix matching.
|
||||||
|
//!
|
||||||
|
//! This is the opposite shape. There is **no destination in the protocol**. The
|
||||||
|
//! guest opens a socket; the host connects it to `127.0.0.1:11434` on the node
|
||||||
|
//! and copies bytes. A compromised guest can ask for nothing else, because there
|
||||||
|
//! is nothing to ask — it is a pipe, not a proxy. That is strictly narrower than
|
||||||
|
//! anything the allow-list could express, and it is why routing a local model
|
||||||
|
//! through `egress` would have been the worse design: it would have meant
|
||||||
|
//! relaxing the 443-only rule and the IP-literal refusal, both of which exist
|
||||||
|
//! because a unit test caught them being bypassed.
|
||||||
|
//!
|
||||||
|
//! # Why plaintext is right here
|
||||||
|
//!
|
||||||
|
//! The bytes go guest loopback → vsock → host loopback. They never touch a
|
||||||
|
//! network, so there is no wire for TLS to protect. Ollama stays bound to
|
||||||
|
//! `127.0.0.1` on the node and is never exposed to the tailnet, which is a
|
||||||
|
//! stronger position than terminating TLS in front of it would have been.
|
||||||
|
//!
|
||||||
|
//! # Why it is per-backend
|
||||||
|
//!
|
||||||
|
//! The node binds this socket only for a backend declared to use a local model.
|
||||||
|
//! On every other backend the guest's listener is still there and simply gets a
|
||||||
|
//! refusal — the same fail-closed default `provider_hosts` applies to egress.
|
||||||
|
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
|
use tokio::net::{TcpStream, UnixListener};
|
||||||
|
|
||||||
|
/// Host-side vsock port. Must match `fcagent`'s `MODEL_VSOCK_PORT`.
|
||||||
|
pub const MODEL_PORT: u32 = 9003;
|
||||||
|
|
||||||
|
/// Where the node's model server listens. Loopback, and not configurable from
|
||||||
|
/// the guest by design — see the module docs.
|
||||||
|
const OLLAMA_ADDR: &str = "127.0.0.1:11434";
|
||||||
|
|
||||||
|
/// Whether a backend is served by a model running on the node itself.
|
||||||
|
///
|
||||||
|
/// Named individually rather than by prefix. An unrecognised backend must not
|
||||||
|
/// acquire a route to anything by accident, which is the same rule
|
||||||
|
/// `egress::provider_hosts` and `mission_runtime::microvm_credential_for`
|
||||||
|
/// already apply from their own side.
|
||||||
|
pub fn uses_local_model(backend: Option<&str>) -> bool {
|
||||||
|
matches!(backend, Some("local-ornith"))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Bind the guest's local-model socket, if this backend has one.
|
||||||
|
///
|
||||||
|
/// `Ok(None)` means "this backend does not use a local model" and is the normal
|
||||||
|
/// case. An error means it should have had one and could not — reported by the
|
||||||
|
/// caller, never silently swallowed, because the symptom otherwise is an agent
|
||||||
|
/// that hangs on its first turn.
|
||||||
|
pub fn start(
|
||||||
|
uds: &Path,
|
||||||
|
vm_id: &str,
|
||||||
|
backend: Option<&str>,
|
||||||
|
) -> Result<Option<(PathBuf, tokio::task::JoinHandle<()>)>, String> {
|
||||||
|
if !uses_local_model(backend) {
|
||||||
|
return Ok(None);
|
||||||
|
}
|
||||||
|
let path = PathBuf::from(format!("{}_{}", uds.display(), MODEL_PORT));
|
||||||
|
// Firecracker leaves these behind exactly as it does its own socket, and a
|
||||||
|
// stale file makes bind fail with EADDRINUSE.
|
||||||
|
let _ = std::fs::remove_file(&path);
|
||||||
|
let listener =
|
||||||
|
UnixListener::bind(&path).map_err(|e| format!("bind {}: {e}", path.display()))?;
|
||||||
|
|
||||||
|
eprintln!(
|
||||||
|
"microvm {vm_id}: local model socket on {} -> {OLLAMA_ADDR}",
|
||||||
|
path.display()
|
||||||
|
);
|
||||||
|
let vm = vm_id.to_string();
|
||||||
|
let task = tokio::spawn(async move {
|
||||||
|
loop {
|
||||||
|
match listener.accept().await {
|
||||||
|
Ok((s, _)) => {
|
||||||
|
let vm = vm.clone();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
if let Err(e) = pipe(s).await {
|
||||||
|
// Loud, because the failure a mission sees is a turn
|
||||||
|
// that never answers. A refused connection here means
|
||||||
|
// the node's model server is down, and that is worth
|
||||||
|
// saying out loud rather than leaving to a timeout.
|
||||||
|
eprintln!("microvm {vm}: local model pipe failed: {e}");
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!("microvm {vm}: local model accept failed: {e}");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
Ok(Some((path, task)))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Splice one guest connection onto a fresh connection to the node's model.
|
||||||
|
async fn pipe(mut guest: tokio::net::UnixStream) -> Result<(), String> {
|
||||||
|
let mut model = TcpStream::connect(OLLAMA_ADDR)
|
||||||
|
.await
|
||||||
|
.map_err(|e| format!("connect {OLLAMA_ADDR}: {e}"))?;
|
||||||
|
tokio::io::copy_bidirectional(&mut guest, &mut model)
|
||||||
|
.await
|
||||||
|
.map(|_| ())
|
||||||
|
.map_err(|e| format!("copy: {e}"))
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
/// Only the backends that are meant to have a local model get one.
|
||||||
|
///
|
||||||
|
/// The negative half is the point: an unrecognised backend acquiring a route
|
||||||
|
/// to the node's own model server would be a hole opened by a typo, and it
|
||||||
|
/// would be invisible because the mission would simply work.
|
||||||
|
#[test]
|
||||||
|
fn a_local_route_is_never_granted_by_accident() {
|
||||||
|
assert!(uses_local_model(Some("local-ornith")));
|
||||||
|
|
||||||
|
for other in [
|
||||||
|
None,
|
||||||
|
Some(""),
|
||||||
|
Some("default"),
|
||||||
|
Some("claude"),
|
||||||
|
Some("canary-claude"),
|
||||||
|
Some("glm"),
|
||||||
|
Some("kimi"),
|
||||||
|
Some("local"),
|
||||||
|
Some("local-ornith-typo"),
|
||||||
|
Some("ornith"),
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
!uses_local_model(other),
|
||||||
|
"{other:?} must not reach the node's model server"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The guest cannot name a destination, so there is nothing to validate.
|
||||||
|
///
|
||||||
|
/// This asserts the property that makes this module safe enough to skip the
|
||||||
|
/// allow-list entirely: the upstream address is a constant. If it ever
|
||||||
|
/// becomes a parameter, this file needs everything `egress` has.
|
||||||
|
#[test]
|
||||||
|
fn the_upstream_address_is_a_constant_not_an_input() {
|
||||||
|
let src = include_str!("local_model.rs");
|
||||||
|
// Needles are split so they do not match themselves in this file.
|
||||||
|
assert_eq!(
|
||||||
|
src.matches(concat!("TcpStream", "::connect(")).count(),
|
||||||
|
1,
|
||||||
|
"exactly one dial site, and it must use the constant"
|
||||||
|
);
|
||||||
|
assert!(src.contains(concat!("TcpStream", "::connect(OLLAMA_ADDR)")));
|
||||||
|
assert!(
|
||||||
|
OLLAMA_ADDR.starts_with("127.0.0.1:"),
|
||||||
|
"the model server must be reached on loopback only"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -20,6 +20,7 @@ use tokio::sync::{mpsc, Mutex};
|
|||||||
use tokio_tungstenite::tungstenite::Message;
|
use tokio_tungstenite::tungstenite::Message;
|
||||||
|
|
||||||
mod egress;
|
mod egress;
|
||||||
|
mod local_model;
|
||||||
mod microvm;
|
mod microvm;
|
||||||
mod rtc;
|
mod rtc;
|
||||||
|
|
||||||
|
|||||||
@@ -58,6 +58,14 @@ pub struct Vm {
|
|||||||
/// one's identity.
|
/// one's identity.
|
||||||
egress: Option<tokio::task::JoinHandle<()>>,
|
egress: Option<tokio::task::JoinHandle<()>>,
|
||||||
egress_uds: PathBuf,
|
egress_uds: PathBuf,
|
||||||
|
/// The local-model listener, held for the SAME reason as `egress`: a
|
||||||
|
/// listener that outlives its VM would accept a connection from the next VM
|
||||||
|
/// to reuse the path and serve it under the dead one's identity.
|
||||||
|
model: Option<tokio::task::JoinHandle<()>>,
|
||||||
|
/// Present only for a backend served by a model on this node. Removed on
|
||||||
|
/// destroy alongside the egress socket — a leaked unix socket is the same
|
||||||
|
/// class of host litter the TAP approach was rejected for.
|
||||||
|
model_uds: Option<PathBuf>,
|
||||||
}
|
}
|
||||||
|
|
||||||
pub type Vms = Arc<Mutex<HashMap<String, Vm>>>;
|
pub type Vms = Arc<Mutex<HashMap<String, Vm>>>;
|
||||||
@@ -388,6 +396,23 @@ pub async fn create(
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// The node's own model, for a backend that has one. Bound before firecracker
|
||||||
|
// starts for the same reason egress is: a guest that dials before the host
|
||||||
|
// listens gets a refusal it will not retry.
|
||||||
|
let (model_uds, model_task) = match crate::local_model::start(&uds, vm_id, backend) {
|
||||||
|
Ok(Some((p, t))) => (Some(p), Some(t)),
|
||||||
|
Ok(None) => (None, None),
|
||||||
|
// This backend was supposed to have a local model and does not. Not
|
||||||
|
// fatal, but the mission WILL fail on its first turn, so say why here
|
||||||
|
// rather than leaving it to look like a hung agent.
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!(
|
||||||
|
"microvm {vm_id}: NO LOCAL MODEL ({e}) — a {backend:?} turn cannot reach one"
|
||||||
|
);
|
||||||
|
(None, None)
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
let log = std::fs::File::create(workdir.join("console.log"))
|
let log = std::fs::File::create(workdir.join("console.log"))
|
||||||
.map_err(|e| format!("create console.log: {e}"))?;
|
.map_err(|e| format!("create console.log: {e}"))?;
|
||||||
let errlog = log
|
let errlog = log
|
||||||
@@ -418,6 +443,8 @@ pub async fn create(
|
|||||||
uds: uds.clone(),
|
uds: uds.clone(),
|
||||||
egress: egress_task,
|
egress: egress_task,
|
||||||
egress_uds: egress_uds.clone(),
|
egress_uds: egress_uds.clone(),
|
||||||
|
model: model_task,
|
||||||
|
model_uds: model_uds.clone(),
|
||||||
};
|
};
|
||||||
|
|
||||||
// Poll for the agent. 10s is generous: the measured boot-to-agent is under
|
// Poll for the agent. 10s is generous: the measured boot-to-agent is under
|
||||||
@@ -548,14 +575,17 @@ async fn kill_group(pgid: i32) {
|
|||||||
pub async fn destroy(vms: &Vms, vm_id: &str) -> Result<Value, String> {
|
pub async fn destroy(vms: &Vms, vm_id: &str) -> Result<Value, String> {
|
||||||
check_id(vm_id)?;
|
check_id(vm_id)?;
|
||||||
let vm = vms.lock().await.remove(vm_id);
|
let vm = vms.lock().await.remove(vm_id);
|
||||||
let (pgid, workdir, uds, egress_uds) = match vm {
|
let (pgid, workdir, uds, egress_uds, model_uds) = match vm {
|
||||||
Some(v) => {
|
Some(v) => {
|
||||||
// Abort first: a live listener would keep accepting on a path the
|
// Abort first: a live listener would keep accepting on a path the
|
||||||
// next VM is about to reuse.
|
// next VM is about to reuse.
|
||||||
if let Some(t) = v.egress {
|
if let Some(t) = v.egress {
|
||||||
t.abort();
|
t.abort();
|
||||||
}
|
}
|
||||||
(Some(v.pgid), v.workdir, v.uds, v.egress_uds)
|
if let Some(t) = v.model {
|
||||||
|
t.abort();
|
||||||
|
}
|
||||||
|
(Some(v.pgid), v.workdir, v.uds, v.egress_uds, v.model_uds)
|
||||||
}
|
}
|
||||||
// Not registered: still clean the paths, so a VM created by a previous
|
// Not registered: still clean the paths, so a VM created by a previous
|
||||||
// incarnation of the daemon can be reaped rather than orphaned forever.
|
// incarnation of the daemon can be reaped rather than orphaned forever.
|
||||||
@@ -563,7 +593,12 @@ pub async fn destroy(vms: &Vms, vm_id: &str) -> Result<Value, String> {
|
|||||||
let wd = work_root().join("vms").join(vm_id);
|
let wd = work_root().join("vms").join(vm_id);
|
||||||
let uds = wd.join("v.sock");
|
let uds = wd.join("v.sock");
|
||||||
let eg = PathBuf::from(format!("{}_{}", uds.display(), crate::egress::EGRESS_PORT));
|
let eg = PathBuf::from(format!("{}_{}", uds.display(), crate::egress::EGRESS_PORT));
|
||||||
(None, wd.clone(), uds, eg)
|
let md = PathBuf::from(format!(
|
||||||
|
"{}_{}",
|
||||||
|
uds.display(),
|
||||||
|
crate::local_model::MODEL_PORT
|
||||||
|
));
|
||||||
|
(None, wd.clone(), uds, eg, Some(md))
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
// Killed means OBSERVED GONE, not asked-to-die.
|
// Killed means OBSERVED GONE, not asked-to-die.
|
||||||
@@ -596,6 +631,9 @@ pub async fn destroy(vms: &Vms, vm_id: &str) -> Result<Value, String> {
|
|||||||
// Same trap as firecracker's own socket: nothing unlinks these for us, and a
|
// Same trap as firecracker's own socket: nothing unlinks these for us, and a
|
||||||
// stale file makes the next bind fail with EADDRINUSE.
|
// stale file makes the next bind fail with EADDRINUSE.
|
||||||
let _ = tokio::fs::remove_file(&egress_uds).await;
|
let _ = tokio::fs::remove_file(&egress_uds).await;
|
||||||
|
if let Some(m) = &model_uds {
|
||||||
|
let _ = tokio::fs::remove_file(m).await;
|
||||||
|
}
|
||||||
let removed = tokio::fs::remove_dir_all(&workdir).await.is_ok();
|
let removed = tokio::fs::remove_dir_all(&workdir).await.is_ok();
|
||||||
// `killed` is now observed rather than assumed; `signalled` keeps the old
|
// `killed` is now observed rather than assumed; `signalled` keeps the old
|
||||||
// meaning so a caller can tell "there was nothing to kill" from
|
// meaning so a caller can tell "there was nothing to kill" from
|
||||||
|
|||||||
@@ -51,6 +51,25 @@ const PROXY_PORT: u16 = 3128;
|
|||||||
/// Host-side vsock port the tunnel lands on. Firecracker's convention for a
|
/// Host-side vsock port the tunnel lands on. Firecracker's convention for a
|
||||||
/// guest-initiated connection is that the HOST listens on `<uds_path>_<port>`.
|
/// guest-initiated connection is that the HOST listens on `<uds_path>_<port>`.
|
||||||
const EGRESS_PORT: u32 = 9002;
|
const EGRESS_PORT: u32 = 9002;
|
||||||
|
/// Guest-side port for a LOCALLY HOSTED model, and the vsock port it lands on.
|
||||||
|
///
|
||||||
|
/// Separate from the egress proxy on purpose, and simpler than it. The egress
|
||||||
|
/// path exists to let an agent reach the public internet under an allow-list;
|
||||||
|
/// this one reaches exactly one thing — the Ollama the node itself is running,
|
||||||
|
/// on its own loopback — and can reach nothing else, because the host end is a
|
||||||
|
/// pipe to a fixed address rather than a proxy that takes a destination.
|
||||||
|
///
|
||||||
|
/// It therefore needs no `CONNECT`, no TLS and no allow-list. The bytes travel
|
||||||
|
/// guest loopback → vsock → host loopback and never touch a network, so there is
|
||||||
|
/// nothing on a wire for TLS to protect. `NO_PROXY` already contains
|
||||||
|
/// `127.0.0.1`, so an agent pointed at `http://127.0.0.1:11434` bypasses the
|
||||||
|
/// egress proxy entirely rather than trying to CONNECT through it.
|
||||||
|
///
|
||||||
|
/// The guest always listens. Whether anything answers is the HOST's decision:
|
||||||
|
/// the node only binds the vsock end for a backend that is meant to have a
|
||||||
|
/// local model, so on every other backend this port simply refuses.
|
||||||
|
const MODEL_PORT: u16 = 11434;
|
||||||
|
const MODEL_VSOCK_PORT: u32 = 9003;
|
||||||
/// `VMADDR_CID_HOST` — the hypervisor side of the vsock.
|
/// `VMADDR_CID_HOST` — the hypervisor side of the vsock.
|
||||||
const HOST_CID: u32 = 2;
|
const HOST_CID: u32 = 2;
|
||||||
|
|
||||||
@@ -193,7 +212,24 @@ fn start_egress_proxy() {
|
|||||||
PROXY_UP.store(true, Ordering::Relaxed);
|
PROXY_UP.store(true, Ordering::Relaxed);
|
||||||
println!("FC-AGENT-PROXY listening on 127.0.0.1:{PROXY_PORT} -> vsock {EGRESS_PORT}");
|
println!("FC-AGENT-PROXY listening on 127.0.0.1:{PROXY_PORT} -> vsock {EGRESS_PORT}");
|
||||||
let _ = std::io::stdout().flush();
|
let _ = std::io::stdout().flush();
|
||||||
|
pump(listener, EGRESS_PORT, "PROXY");
|
||||||
|
|
||||||
|
// The local-model port. Failure to bind is reported and non-fatal, exactly
|
||||||
|
// like the egress proxy: a VM whose backend does not use a local model is
|
||||||
|
// still perfectly useful, and a fatal error here would take out every
|
||||||
|
// backend to serve one.
|
||||||
|
match TcpListener::bind(("127.0.0.1", MODEL_PORT)) {
|
||||||
|
Ok(l) => {
|
||||||
|
println!("FC-AGENT-MODEL listening on 127.0.0.1:{MODEL_PORT} -> vsock {MODEL_VSOCK_PORT}");
|
||||||
|
let _ = std::io::stdout().flush();
|
||||||
|
pump(l, MODEL_VSOCK_PORT, "MODEL");
|
||||||
|
}
|
||||||
|
Err(e) => eprintln!("FC-AGENT-NO-MODEL could not listen on 127.0.0.1:{MODEL_PORT}: {e}"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Accept forever, splicing each connection onto its own vsock stream.
|
||||||
|
fn pump(listener: TcpListener, vsock_port: u32, tag: &'static str) {
|
||||||
std::thread::spawn(move || {
|
std::thread::spawn(move || {
|
||||||
for c in listener.incoming() {
|
for c in listener.incoming() {
|
||||||
match c {
|
match c {
|
||||||
@@ -201,12 +237,12 @@ fn start_egress_proxy() {
|
|||||||
// and serving them in sequence would look like a hang.
|
// and serving them in sequence would look like a hang.
|
||||||
Ok(tcp) => {
|
Ok(tcp) => {
|
||||||
std::thread::spawn(move || {
|
std::thread::spawn(move || {
|
||||||
if let Err(e) = tunnel(tcp) {
|
if let Err(e) = tunnel(tcp, vsock_port) {
|
||||||
eprintln!("FC-AGENT-PROXY-ERROR {e}");
|
eprintln!("FC-AGENT-{tag}-ERROR {e}");
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
Err(e) => eprintln!("FC-AGENT-PROXY-ERROR accept: {e}"),
|
Err(e) => eprintln!("FC-AGENT-{tag}-ERROR accept: {e}"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
@@ -217,9 +253,9 @@ fn start_egress_proxy() {
|
|||||||
/// No parsing: whatever the client sent — `CONNECT host:443`, or an absolute-form
|
/// No parsing: whatever the client sent — `CONNECT host:443`, or an absolute-form
|
||||||
/// request — is the host's business. The host answers with real HTTP, so a
|
/// request — is the host's business. The host answers with real HTTP, so a
|
||||||
/// refusal reaches the client as a status code rather than a dropped socket.
|
/// refusal reaches the client as a status code rather than a dropped socket.
|
||||||
fn tunnel(tcp: std::net::TcpStream) -> Result<(), String> {
|
fn tunnel(tcp: std::net::TcpStream, vsock_port: u32) -> Result<(), String> {
|
||||||
let vs = vsock::VsockStream::connect_with_cid_port(HOST_CID, EGRESS_PORT)
|
let vs = vsock::VsockStream::connect_with_cid_port(HOST_CID, vsock_port)
|
||||||
.map_err(|e| format!("vsock connect to host:{EGRESS_PORT}: {e}"))?;
|
.map_err(|e| format!("vsock connect to host:{vsock_port}: {e}"))?;
|
||||||
|
|
||||||
let (mut tcp_r, mut tcp_w) = (
|
let (mut tcp_r, mut tcp_w) = (
|
||||||
tcp.try_clone().map_err(|e| format!("clone tcp: {e}"))?,
|
tcp.try_clone().map_err(|e| format!("clone tcp: {e}"))?,
|
||||||
|
|||||||
@@ -199,6 +199,20 @@ fn microvm_credential_for(backend: Option<&str>) -> Result<Credential, String> {
|
|||||||
source: "KIMI_API_KEY",
|
source: "KIMI_API_KEY",
|
||||||
target: "ANTHROPIC_AUTH_TOKEN",
|
target: "ANTHROPIC_AUTH_TOKEN",
|
||||||
}),
|
}),
|
||||||
|
// A model running on the NODE ITSELF. There is no credential and there
|
||||||
|
// is nothing to protect: the guest reaches it over a vsock pipe to the
|
||||||
|
// node's own loopback (`clawmates-node::local_model`), which has no
|
||||||
|
// destination in its protocol and can therefore reach nothing else.
|
||||||
|
//
|
||||||
|
// It still goes through this map rather than around it. Ollama ignores
|
||||||
|
// the bearer but Claude Code refuses to start without one, so the value
|
||||||
|
// is a literal — and routing it here keeps the rule that a backend with
|
||||||
|
// no entry cannot launch, which is what stops a typo'd backend from
|
||||||
|
// quietly inheriting the subscription token.
|
||||||
|
Some("local-ornith") => Ok(Credential {
|
||||||
|
source: "CLAWMATES_LOCAL_MODEL_TOKEN",
|
||||||
|
target: "ANTHROPIC_AUTH_TOKEN",
|
||||||
|
}),
|
||||||
// Kimi, on the same split as GLM: endpoint in the image
|
// Kimi, on the same split as GLM: endpoint in the image
|
||||||
// (`https://api.kimi.com/coding`), credential in the turn.
|
// (`https://api.kimi.com/coding`), credential in the turn.
|
||||||
//
|
//
|
||||||
@@ -222,6 +236,14 @@ fn microvm_provider_env_from(
|
|||||||
lookup: impl Fn(&str) -> Option<String>,
|
lookup: impl Fn(&str) -> Option<String>,
|
||||||
) -> Result<Vec<(String, String)>, String> {
|
) -> Result<Vec<(String, String)>, String> {
|
||||||
let want = microvm_credential_for(backend)?;
|
let want = microvm_credential_for(backend)?;
|
||||||
|
// A local model has no secret to look up. Defaulted rather than demanded:
|
||||||
|
// requiring an operator to set a variable whose value is ignored is a step
|
||||||
|
// that only ever fails, and its failure mode here is a hung agent.
|
||||||
|
let lookup = |k: &str| {
|
||||||
|
lookup(k).or_else(|| {
|
||||||
|
(k == "CLAWMATES_LOCAL_MODEL_TOKEN").then(|| "local-model-no-auth".to_string())
|
||||||
|
})
|
||||||
|
};
|
||||||
let token = lookup(want.source)
|
let token = lookup(want.source)
|
||||||
.filter(|v| !v.trim().is_empty())
|
.filter(|v| !v.trim().is_empty())
|
||||||
.ok_or_else(|| {
|
.ok_or_else(|| {
|
||||||
@@ -1199,6 +1221,38 @@ mod tests {
|
|||||||
/// `CLAWMATES_RUNTIME_AUTH` at all. If it did, the single unset variable on
|
/// `CLAWMATES_RUNTIME_AUTH` at all. If it did, the single unset variable on
|
||||||
/// gw-04 today would put an API key inside every VM, and Claude Code ranks
|
/// gw-04 today would put an API key inside every VM, and Claude Code ranks
|
||||||
/// the key above the subscription token: it would work, and bill per-token
|
/// the key above the subscription token: it would work, and bill per-token
|
||||||
|
/// A local backend carries no secret, and no secret reaches it.
|
||||||
|
///
|
||||||
|
/// The credential map is the one place a backend can acquire a token, and
|
||||||
|
/// the point of routing `local-ornith` through it — rather than special-
|
||||||
|
/// casing it earlier — is that the "unknown backend cannot launch" rule
|
||||||
|
/// still holds. A typo like `local-ornit` must fail closed, not inherit the
|
||||||
|
/// subscription token on its way to somebody else's endpoint.
|
||||||
|
#[test]
|
||||||
|
fn a_local_backend_gets_a_placeholder_and_never_a_real_credential() {
|
||||||
|
let env = microvm_provider_env_from(Some("local-ornith"), |_| None)
|
||||||
|
.expect("a local backend needs nothing from the operator");
|
||||||
|
let token = env
|
||||||
|
.iter()
|
||||||
|
.find(|(k, _)| k == "ANTHROPIC_AUTH_TOKEN")
|
||||||
|
.map(|(_, v)| v.clone())
|
||||||
|
.expect("Claude Code refuses to start without a bearer");
|
||||||
|
assert!(
|
||||||
|
!token.starts_with("sk-"),
|
||||||
|
"a local backend must never be handed a real credential, got {token:?}"
|
||||||
|
);
|
||||||
|
assert!(!env.iter().any(|(k, _)| k == "ANTHROPIC_API_KEY"));
|
||||||
|
assert!(!env.iter().any(|(k, _)| k == "CLAUDE_CODE_OAUTH_TOKEN"));
|
||||||
|
|
||||||
|
// And a near-miss still cannot launch.
|
||||||
|
for typo in ["local-ornit", "ornith", "local", "local-ornith2"] {
|
||||||
|
assert!(
|
||||||
|
microvm_credential_for(Some(typo)).is_err(),
|
||||||
|
"{typo} must be refused, not resolved to something"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// against a plan already paid for, with no symptom but the invoice.
|
/// against a plan already paid for, with no symptom but the invoice.
|
||||||
#[test]
|
#[test]
|
||||||
fn a_microvm_never_receives_an_anthropic_api_key() {
|
fn a_microvm_never_receives_an_anthropic_api_key() {
|
||||||
|
|||||||
@@ -0,0 +1,55 @@
|
|||||||
|
# Claude Code pointed at a model running on the NODE ITSELF.
|
||||||
|
#
|
||||||
|
# The third variation on one idea: `agent-claude` talks to Anthropic,
|
||||||
|
# `agent-glm` to z.ai, `agent-kimi` to Moonshot, and this one to the Ollama
|
||||||
|
# already installed on every GPU node. Ollama has served a native
|
||||||
|
# Anthropic-compatible `/v1/messages` since v0.14, so this costs an env contract
|
||||||
|
# rather than a translation layer — MEASURED on both nodes: a tools request
|
||||||
|
# comes back with a well-formed `tool_use` block and `stop_reason=tool_use`.
|
||||||
|
#
|
||||||
|
# Why the base URL is loopback INSIDE the guest, not a hostname:
|
||||||
|
#
|
||||||
|
# The VM has no network interface. `fcagent` listens on 127.0.0.1:11434 and
|
||||||
|
# pumps to vsock 9003, where `clawmates-node::local_model` splices it onto the
|
||||||
|
# node's own `127.0.0.1:11434`. There is no destination anywhere in that path —
|
||||||
|
# it is a pipe to a fixed address, so a compromised guest cannot redirect it.
|
||||||
|
# `NO_PROXY` already contains 127.0.0.1, so this bypasses the egress proxy
|
||||||
|
# rather than trying to CONNECT through it.
|
||||||
|
#
|
||||||
|
# Build (on a node with a GPU — architect or tank):
|
||||||
|
#
|
||||||
|
# ssh architect "cd ~/clawmates && \
|
||||||
|
# docker build -f images/agent-ornith/Dockerfile -t clawmates/agent-ornith:dev images/agent-ornith/"
|
||||||
|
# scripts/fc-build-rootfs.sh architect clawmates/agent-ornith:dev local-ornith 8G
|
||||||
|
#
|
||||||
|
# Building it ONLY on GPU nodes is deliberate and is the whole placement story:
|
||||||
|
# a node advertises `rootfs-local-ornith.ext4` in `capabilities.rootfs`, and
|
||||||
|
# `nodes::online_for_backend` already filters on exactly that. No GPU capability
|
||||||
|
# key, no migration — morpheus simply never offers this backend.
|
||||||
|
FROM clawmates/agent-toolchain:dev
|
||||||
|
|
||||||
|
# Pinned to the SAME version as agent-claude and agent-glm. A run that differs
|
||||||
|
# by provider should differ by nothing else, or "the local backend behaved
|
||||||
|
# differently" is ambiguous between the model and the harness.
|
||||||
|
ARG CLAUDE_CODE_VERSION=2.1.223
|
||||||
|
RUN npm install -g "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}" \
|
||||||
|
&& npm cache clean --force \
|
||||||
|
&& rm -rf /root/.npm \
|
||||||
|
&& claude --version
|
||||||
|
|
||||||
|
# `ornith-fleet:9b`, NOT `ornith:9b`. Ollama defaults to a ~2K context window
|
||||||
|
# whatever the model card says, and it truncates silently: MEASURED, stock
|
||||||
|
# ornith:9b reported input_tokens=2050 for a 48000-word prompt and answered as
|
||||||
|
# though nothing had been dropped. The fleet tag pins num_ctx=131072 and is
|
||||||
|
# created by `scripts/fleet-model-setup.sh`. An agent turn is exactly the
|
||||||
|
# workload that would hit the default and never say so.
|
||||||
|
ENV HOME=/root \
|
||||||
|
CLAWMATES_AGENT_CLI=claude \
|
||||||
|
ANTHROPIC_BASE_URL=http://127.0.0.1:11434 \
|
||||||
|
ANTHROPIC_MODEL=ornith-fleet:9b \
|
||||||
|
ANTHROPIC_SMALL_FAST_MODEL=ornith-fleet:9b
|
||||||
|
RUN mkdir -p /root/.claude
|
||||||
|
|
||||||
|
# No provider key of any kind. `microvm_credential_for` hands this backend a
|
||||||
|
# literal placeholder because Ollama ignores the bearer and Claude Code refuses
|
||||||
|
# to start without one. There is no secret in this image and none reaches it.
|
||||||
@@ -54,7 +54,10 @@ case "${FC_CLI-unset}" in
|
|||||||
# image is `claude` for all three. Moonshot ship their own `kimi`
|
# image is `claude` for all three. Moonshot ship their own `kimi`
|
||||||
# CLI, and this map used to expect it — a leftover from before the
|
# CLI, and this map used to expect it — a leftover from before the
|
||||||
# endpoint was measured, which failed a perfectly good rootfs.
|
# endpoint was measured, which failed a perfectly good rootfs.
|
||||||
claude|glm|kimi) FC_CLI="claude --version" ;;
|
# `local-ornith` joins them: Claude Code again, pointed at the
|
||||||
|
# node's own Ollama. The binary in the image is `claude` for all
|
||||||
|
# four; only ANTHROPIC_BASE_URL differs.
|
||||||
|
claude|glm|kimi|local-ornith) FC_CLI="claude --version" ;;
|
||||||
*) FC_CLI="" ;;
|
*) FC_CLI="" ;;
|
||||||
esac ;;
|
esac ;;
|
||||||
esac
|
esac
|
||||||
|
|||||||
Executable
+104
@@ -0,0 +1,104 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Put a locally-hosted model on a GPU node, configured the way the fleet needs it.
|
||||||
|
#
|
||||||
|
# The whole reason this is a script and not two commands in a README:
|
||||||
|
#
|
||||||
|
# MEASURED on tank — stock `ornith:9b` reported input_tokens=2050 for a
|
||||||
|
# 48000-word prompt, and answered as though nothing had been dropped.
|
||||||
|
#
|
||||||
|
# Ollama defaults to a ~2K context window whatever the model card says, and it
|
||||||
|
# truncates SILENTLY. An agent turn is exactly the workload that hits that
|
||||||
|
# default, and the symptom is not an error: it is a model that answers
|
||||||
|
# confidently about a prompt it never saw. So the fleet never uses the stock tag.
|
||||||
|
# It uses a derived one with `num_ctx` pinned, and that tag is what
|
||||||
|
# `images/agent-ornith` bakes into ANTHROPIC_MODEL.
|
||||||
|
#
|
||||||
|
# Sizing, measured on a 16 GB RTX 5060 Ti with a 5.6 GB model:
|
||||||
|
#
|
||||||
|
# num_ctx 32768 -> 6.3 GB resident
|
||||||
|
# num_ctx 65536 -> 7.2 GB
|
||||||
|
# num_ctx 131072 -> 9.3 GB <- the default here
|
||||||
|
# num_ctx 262144 -> 13.6 GB (fits, 100% GPU, but leaves little headroom)
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# scripts/fleet-model-setup.sh <ssh-host> [base-model] [fleet-tag] [num_ctx]
|
||||||
|
# scripts/fleet-model-setup.sh architect
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
HOST="${1:?usage: $0 <ssh-host> [base-model] [fleet-tag] [num_ctx]}"
|
||||||
|
BASE="${2:-ornith:9b}"
|
||||||
|
TAG="${3:-ornith-fleet:9b}"
|
||||||
|
CTX="${4:-131072}"
|
||||||
|
|
||||||
|
die() { printf 'ABORT %s\n' "$*" >&2; exit 2; }
|
||||||
|
ok() { printf 'OK %s\n' "$*"; }
|
||||||
|
|
||||||
|
ssh -o BatchMode=yes -o ConnectTimeout=10 "$HOST" true 2>/dev/null || die "cannot ssh to $HOST"
|
||||||
|
|
||||||
|
# A GPU is not optional. This model on CPU is slow enough that a mission would
|
||||||
|
# time out rather than fail, which is the worse outcome.
|
||||||
|
ssh "$HOST" 'command -v nvidia-smi >/dev/null && nvidia-smi -L' \
|
||||||
|
|| die "$HOST has no NVIDIA GPU — a local backend belongs on a GPU node"
|
||||||
|
|
||||||
|
ssh "$HOST" 'command -v ollama >/dev/null' \
|
||||||
|
|| die "$HOST has no ollama (the fleet page's update button installs it)"
|
||||||
|
|
||||||
|
# Version gate, with the reason. architect ran 0.30.5 and could not pull this
|
||||||
|
# model AT ALL — "requires a newer version of Ollama" — which is at least a loud
|
||||||
|
# failure. The quiet one is older still: /v1/messages only exists from v0.14.
|
||||||
|
have=$(ssh "$HOST" 'ollama --version' | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1)
|
||||||
|
[ -n "$have" ] || die "could not read ollama's version on $HOST"
|
||||||
|
printf 'ollama %s on %s\n' "$have" "$HOST"
|
||||||
|
|
||||||
|
ssh "$HOST" "ollama pull '$BASE'" >/dev/null 2>&1 \
|
||||||
|
|| die "could not pull $BASE on $HOST — is ollama new enough for it?"
|
||||||
|
ok "$BASE pulled"
|
||||||
|
|
||||||
|
# The derived tag. `ollama create` is idempotent, so re-running is safe.
|
||||||
|
ssh "$HOST" "printf 'FROM %s\nPARAMETER num_ctx %s\n' '$BASE' '$CTX' > /tmp/Modelfile.fleet \
|
||||||
|
&& ollama create '$TAG' -f /tmp/Modelfile.fleet" >/dev/null 2>&1 \
|
||||||
|
|| die "could not create $TAG on $HOST"
|
||||||
|
ok "$TAG created with num_ctx=$CTX"
|
||||||
|
|
||||||
|
# PROVE the window, do not assume it. This is the check the whole script exists
|
||||||
|
# for: a Modelfile that silently failed to apply looks exactly like one that
|
||||||
|
# worked, right up until an agent loses its context mid-mission.
|
||||||
|
words=$((CTX / 2))
|
||||||
|
reported=$(ssh "$HOST" "python3 - <<'PY'
|
||||||
|
import json, urllib.request
|
||||||
|
filler = 'the quick brown fox jumps over the lazy dog ' * ($words // 9)
|
||||||
|
body = {'model': '$TAG', 'max_tokens': 8,
|
||||||
|
'messages': [{'role': 'user', 'content': 'Log:\n' + filler + '\nReply OK.'}]}
|
||||||
|
req = urllib.request.Request('http://127.0.0.1:11434/v1/messages',
|
||||||
|
data=json.dumps(body).encode(),
|
||||||
|
headers={'content-type': 'application/json', 'x-api-key': 'ollama'})
|
||||||
|
try:
|
||||||
|
print(json.load(urllib.request.urlopen(req, timeout=1800))['usage']['input_tokens'])
|
||||||
|
except Exception as e:
|
||||||
|
print('ERR', e)
|
||||||
|
PY")
|
||||||
|
case "$reported" in
|
||||||
|
''|*[!0-9]*) die "context probe failed on $HOST: $reported" ;;
|
||||||
|
esac
|
||||||
|
# Allow for tokenisation slack, but nothing like a truncation. 2050 is what the
|
||||||
|
# stock tag reports for ANY prompt above it; anything near that means the
|
||||||
|
# num_ctx did not take.
|
||||||
|
floor=$((words / 2))
|
||||||
|
[ "$reported" -gt "$floor" ] \
|
||||||
|
|| die "$TAG reported only $reported input tokens for a ~$words-word prompt — \
|
||||||
|
the context window did not take, and it would truncate silently in a mission"
|
||||||
|
ok "context proven: ~$words words -> $reported input tokens"
|
||||||
|
|
||||||
|
# Tool calling, the other thing a mission agent cannot work without.
|
||||||
|
tools=$(ssh "$HOST" "curl -s -m 300 http://127.0.0.1:11434/v1/messages \
|
||||||
|
-H 'content-type: application/json' -H 'x-api-key: ollama' \
|
||||||
|
-d '{\"model\":\"$TAG\",\"max_tokens\":300,\"tools\":[{\"name\":\"read_file\",\"description\":\"Read a file\",\"input_schema\":{\"type\":\"object\",\"properties\":{\"path\":{\"type\":\"string\"}},\"required\":[\"path\"]}}],\"messages\":[{\"role\":\"user\",\"content\":\"Read src/lib.rs using the tool.\"}]}' \
|
||||||
|
| python3 -c \"import json,sys; d=json.load(sys.stdin); print(d.get('stop_reason'), [b.get('type') for b in d.get('content',[])])\"")
|
||||||
|
case "$tools" in
|
||||||
|
tool_use*) ok "tool calling proven: $tools" ;;
|
||||||
|
*) die "no tool_use from $TAG ($tools) — Claude Code cannot drive a model that cannot call tools" ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
printf '\n%s is ready on %s. Next: build the rootfs there —\n' "$TAG" "$HOST"
|
||||||
|
printf ' ssh %s "cd ~/clawmates && docker build -f images/agent-ornith/Dockerfile -t clawmates/agent-ornith:dev images/agent-ornith/"\n' "$HOST"
|
||||||
|
printf ' scripts/fc-build-rootfs.sh %s clawmates/agent-ornith:dev local-ornith 8G\n' "$HOST"
|
||||||
Reference in New Issue
Block a user