Fleet P2b: run agent sandboxes on connected nodes (RemoteDriver + placement)
ci / gates (push) Failing after 5s
ci / rust (push) Has been skipped
ci / sandbox-k8s (push) Has been skipped
ci / frontend (push) Has been skipped
ci / e2e (push) Has been skipped

Agents can now provision their sandbox on a connected fleet node instead of the
gateway host. Local stays the strict default, so existing agents are byte-for-
byte unaffected until explicitly placed elsewhere.

Security parity: the daemon links the REAL cm-sandbox DockerDriver and runs the
typed container ops (sb_provision/sb_exec/sb_destroy/sb_health/sb_list) through
it — identical hardening (cap-drop ALL, seccomp, no-net, read-only, non-root) to
local sandboxes. cm-sandbox spec types are now Serialize/Deserialize so the spec
crosses the channel.

- cm-api: RemoteDriver (impl SandboxDriver over the node channel) + HubDriverProvider
  (impl cm_runtime::NodeDriverProvider, hands out a driver only for connected
  nodes via a sync online set) + NodeHub.call/is_connected. AppState.with_node_hub
  so the hub is shared with the placement provider.
- cm-runtime SandboxManager: driver_for(node_id) routes by the recorded
  agent_containers.node_id (local default = existing driver, identical path);
  placement_node() reads the workspace setting and falls back to local if the
  node is offline; exec/release route accordingly. NodeDriverProvider trait.
- DB: 0020_workspace_placement + repo (for_agent/get/set/clear).
- main.rs: build the NodeHub first; inject HubDriverProvider into the agent
  manager + share the hub with AppState.
- API+UI: GET/PUT /api/fleet/placement + a "Run agents on: Local / <node>"
  selector in the Fleet overview.

Note: a node must be able to pull the agent image (the daemon docker-pulls it);
interactive PTY for agent containers on remote nodes is not wired (Terminal app
stays local) — the in-dashboard node shell already covers host access.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
Omar Sobh
2026-06-24 12:48:50 -07:00
co-authored by Claude Opus 4.8
parent 33aa9c0693
commit fb59378aa2
30 changed files with 5740 additions and 54 deletions
+15 -6
View File
@@ -144,6 +144,9 @@ async fn run() -> Result<(), String> {
.map_err(|e| format!("s3 storage: {e}"))?,
),
};
// The fleet node hub (live daemon channels) is shared between the sandbox
// placement provider and the API routes, so it must exist before the managers.
let node_hub = std::sync::Arc::new(cm_api::fleet::NodeHub::new());
// Environment tools need a container engine; absence is tolerated
// (shell.exec reports it per-call) so the API still serves.
let (sandboxes, browser, terminals) = if config.sandbox.enabled {
@@ -151,12 +154,17 @@ async fn run() -> Result<(), String> {
Ok(driver) => {
let driver: std::sync::Arc<dyn cm_sandbox::SandboxDriver> =
std::sync::Arc::new(driver);
let agents = std::sync::Arc::new(cm_runtime::SandboxManager::new(
driver.clone(),
pool.clone(),
"local",
&config.sandbox.image,
));
let agents = std::sync::Arc::new(
cm_runtime::SandboxManager::new(
driver.clone(),
pool.clone(),
"local",
&config.sandbox.image,
)
.with_node_provider(std::sync::Arc::new(
cm_api::fleet::HubDriverProvider::new(node_hub.clone()),
)),
);
let browser = std::sync::Arc::new(
cm_runtime::SandboxManager::new(
driver.clone(),
@@ -278,6 +286,7 @@ async fn run() -> Result<(), String> {
let mut app = cm_api::router(
cm_api::AppState::new(pool, runtime)
.with_node_hub(node_hub.clone())
.with_broker(PathBuf::from(&config.broker.socket_path))
.with_oauth(config.oauth.clone())
.with_billing(config.billing.clone())