Omar SobhandClaude Opus 5
869c3adcb7
docs/MISSION-EGRESS.md measured that a mission container reaches the entire
tailnet and SSH on its own host, and left the remediation unapplied. Applying
it on 2026-09-18 found why five iptables lines were never going to be enough:
missions egressed from clawmates_edge, the SERVER's network, and the server
needs the tailnet — Beszel on architect, Ollama for the local backend, the
node daemons for exec-test and node-placed terminals. A tailnet drop scoped to
172.23/16 cut the server off from architect:8090 inside a minute.
Missions now egress from clawmates_missions, 172.25.0.0/16, pinned so the
firewall can name it and declared in both compose files with the same shape
edge has. Compose v1 does not create a network no service uses, so on gw-04
it was created by hand with compose's own labels; the server's attach failure
message now says to check for it. core is unchanged: the door and API are
still reached over 172.20.
The policy itself (/usr/local/sbin/clawmates-egress.sh on gw-04, systemd unit
+ drop-ins on docker and tailscaled) lives in mangle/PREROUTING with
--ctstate NEW. Two earlier placements failed measurably: filter/FORWARD loses
to tailscaled re-inserting ts-forward above it on every restart, and
raw/PREROUTING runs before conntrack, so it dropped the server's replies to
tailnet clients and took the API off 100.102.112.85:8088. Verified from the
mission subnet (tailnet, host ssh, link-local blocked; public and core open),
from edge (tailnet open, ssh blocked), and inbound from tank; and proved to
survive restarting both daemons.
Also: deploy/compose/docker-compose.override.yml is tracked now. It holds the
fixes for the five local bring-up gaps and every credential in it is a
${VAR:?} reference, and it had lived on one laptop that lost a volume this
week.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
2095 lines
93 KiB
Rust
2095 lines
93 KiB
Rust
//! Per-mission ZeroClaw runtime container lifecycle.
|
|
//!
|
|
//! C3 workspace-isolation model: every mission gets its own ZeroClaw
|
|
//! daemon container, so agents' sandboxed filesystem is scoped to that
|
|
//! mission's repo checkout instead of the shared `/zeroclaw-data/
|
|
//! workspace` on the singleton `clawmates-runtime` daemon.
|
|
//!
|
|
//! Container naming: `cm-runtime-mission-{first 12 chars of mission uuid}`.
|
|
//! Endpoint: `http://<container_name>:42617` (well-known ZeroClaw port,
|
|
//! reachable over the `clawmates_core` docker network).
|
|
//!
|
|
//! Lifecycle:
|
|
//! - `ensure_container(mission_id)` — idempotent; spawns the container
|
|
//! if not present, returns its endpoint. Called from
|
|
//! `mission_orchestrator::on_launch` and (as a fallback for
|
|
//! pre-C3 missions) `phase_runner::launch_phase`.
|
|
//! - `teardown_container(mission_id)` — force-removes the container.
|
|
//! Called by the sweeper (slice 3) N minutes after a mission
|
|
//! reaches a terminal state, so the operator has a window to
|
|
//! re-open the ClawmateS UI and pull the last checkpoint before
|
|
//! the daemon disappears.
|
|
//!
|
|
//! State: `missions.runtime_container_name` and `missions.runtime_endpoint`
|
|
//! carry the current binding (both null when torn down or never
|
|
//! provisioned).
|
|
|
|
use bollard::exec::{CreateExecOptions, StartExecResults};
|
|
use bollard::models::{
|
|
ContainerCreateBody, EndpointSettings, HostConfig, Mount, MountTypeEnum, NetworkConnectRequest,
|
|
};
|
|
use bollard::query_parameters::{
|
|
CreateContainerOptions, InspectContainerOptions, RemoveContainerOptions, StartContainerOptions,
|
|
};
|
|
use bollard::Docker;
|
|
use futures::StreamExt;
|
|
use std::collections::HashMap;
|
|
use uuid::Uuid;
|
|
|
|
/// Docker image the per-mission runtime uses, overridable per-deploy via
|
|
/// `CLAWMATES_RUNTIME_IMAGE` (for a canary; the default is the shipped one).
|
|
///
|
|
/// A MOVING tag, and that is the hazard: it names whatever the host last
|
|
/// tagged. gw-04's `sync` was two zeroclaw releases behind the canary every
|
|
/// mission was actually running, and there was no way to see it — a moving tag
|
|
/// that points somewhere old resolves perfectly, starts perfectly, and runs old
|
|
/// code. The registry copy of the same tag was a DIFFERENT image again, without
|
|
/// the Rust toolchain, so a host that pulled rather than retagged would have
|
|
/// silently lost the on-green test gate.
|
|
///
|
|
/// Two things make that visible now: this module logs the image it resolved and
|
|
/// `runtime_preflight` reports the versions inside the container, not just that
|
|
/// the binaries exist.
|
|
const DEFAULT_IMAGE: &str = "clawmates-runtime:sync";
|
|
|
|
/// Well-known ZeroClaw gateway port.
|
|
const GATEWAY_PORT: u16 = 42617;
|
|
|
|
/// How the ZeroClaw runtime authenticates to Anthropic.
|
|
///
|
|
/// The runtime image ships the official `claude` CLI, which can authenticate
|
|
/// either with a platform API key or with a subscription login stored under
|
|
/// `$HOME` (a persisted bind mount, so one login survives container
|
|
/// recreation). These are mutually exclusive in practice because Claude Code
|
|
/// prefers `ANTHROPIC_API_KEY` over the subscription credential.
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
|
pub enum RuntimeAuth {
|
|
/// Forward the platform's `ANTHROPIC_API_KEY`. Metered per token.
|
|
ApiKey,
|
|
/// Withhold the API key so the runtime's own `claude /login` credential is
|
|
/// used. Only valid for a single-operator deployment — a subscription
|
|
/// credential must never serve another person's work.
|
|
Subscription,
|
|
}
|
|
|
|
impl RuntimeAuth {
|
|
pub fn as_str(self) -> &'static str {
|
|
match self {
|
|
RuntimeAuth::ApiKey => "api_key",
|
|
RuntimeAuth::Subscription => "subscription",
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Provider credential env vars forwarded into a runtime container.
|
|
///
|
|
/// `ANTHROPIC_API_KEY` is conditional, and the reason is subtle enough to be
|
|
/// worth stating at the definition: Claude Code resolves credentials in a fixed
|
|
/// priority order and ranks `ANTHROPIC_API_KEY` **above** the subscription's
|
|
/// `CLAUDE_CODE_OAUTH_TOKEN`. On a runtime authenticated via `claude /login`,
|
|
/// forwarding the key silently wins — `claude` still works, agents still run,
|
|
/// and every mission bills the API while appearing to use the subscription.
|
|
/// There is no error to surface; the only symptom is the invoice.
|
|
///
|
|
/// The other three are unrelated providers with no subscription equivalent, so
|
|
/// they forward in both modes.
|
|
///
|
|
/// In subscription mode `CLAUDE_CODE_OAUTH_TOKEN` forwards instead. The
|
|
/// original design assumed a persisted `claude /login` under a bind-mounted
|
|
/// `$HOME`, but a *mission* container gets its own data dir and therefore no
|
|
/// login — so the token has to travel. Missing it is not a loud failure:
|
|
/// `claude -p` simply hangs with no credential, which is what a phase stuck
|
|
/// at `running` for ten minutes looked like when this was first switched on.
|
|
pub fn forwarded_provider_keys(auth: RuntimeAuth) -> Vec<&'static str> {
|
|
// ZAI/KIMI reach their backends through the SAME `claude` binary via
|
|
// ANTHROPIC_BASE_URL, so a mission that selects one needs its key present
|
|
// in the container. They are unrelated to the Anthropic credential and
|
|
// forward in both auth modes.
|
|
let mut keys = vec![
|
|
"GROQ_API_KEY",
|
|
"OPENAI_API_KEY",
|
|
"ZAI_API_KEY",
|
|
"KIMI_API_KEY",
|
|
];
|
|
match auth {
|
|
RuntimeAuth::ApiKey => keys.push("ANTHROPIC_API_KEY"),
|
|
RuntimeAuth::Subscription => keys.push("CLAUDE_CODE_OAUTH_TOKEN"),
|
|
}
|
|
keys
|
|
}
|
|
|
|
/// The forwarded credentials that are actually SET on this server, as pairs.
|
|
///
|
|
/// `forwarded_provider_keys` above stays the single list of *what* forwards; this
|
|
/// is the single reader of the environment, so the container and microVM paths
|
|
/// cannot disagree about how a value is read (blank handling in particular).
|
|
///
|
|
/// A key that is unset is simply absent: an empty-string value would make
|
|
/// `claude` believe it has a credential and fail authentication instead of
|
|
/// reporting that it has none.
|
|
pub fn forwarded_provider_env(auth: RuntimeAuth) -> Vec<(String, String)> {
|
|
provider_env_from(auth, |k| std::env::var(k).ok())
|
|
}
|
|
|
|
/// Credentials for a microVM mission. **Subscription only, by construction.**
|
|
///
|
|
/// Deliberately NOT parameterised by [`runtime_auth_mode`], and that is the
|
|
/// whole point. The container path honours the operator's mode, and gw-04 has
|
|
/// `CLAWMATES_RUNTIME_AUTH` unset today, so it forwards `ANTHROPIC_API_KEY`. A
|
|
/// microVM must not: Claude Code ranks the API key **above** the subscription's
|
|
/// OAuth token, so a VM that received both would bill per-token against a plan
|
|
/// we already pay for, silently — `claude` still works, agents still run, and
|
|
/// the only symptom is the invoice. Taking the mode as an argument would mean a
|
|
/// single unset env var on a new host turns that back on.
|
|
///
|
|
/// A missing subscription token is an **error**, not an empty list. A VM launched
|
|
/// without a credential does not fail: `claude -p` hangs, which is a phase stuck
|
|
/// at `running` with nothing in the logs.
|
|
pub fn microvm_provider_env(backend: Option<&str>) -> Result<Vec<(String, String)>, String> {
|
|
microvm_provider_env_from(backend, |k| std::env::var(k).ok())
|
|
}
|
|
|
|
/// Can a mission agent authenticate in a VM booted from this image?
|
|
///
|
|
/// The honest answer to "which backends can run a mission", which is NOT the
|
|
/// same as "which rootfs images exist on a node". A fleet node reports every
|
|
/// image it has built — `agent-terminal` among them — and booting a mission into
|
|
/// one with no credential contract fails at the agent turn, after the mission
|
|
/// was created and its roster approved.
|
|
pub fn backend_can_run_a_mission(backend: &str) -> bool {
|
|
microvm_credential_for(Some(backend)).is_ok()
|
|
}
|
|
|
|
/// Where a backend's secret comes from, and what the guest's CLI reads it as.
|
|
///
|
|
/// Two names, not one, because they differ for every provider but Anthropic:
|
|
/// z.ai's key lives in the server's `ZAI_API_KEY` and Claude Code reads it as
|
|
/// `ANTHROPIC_AUTH_TOKEN`. Collapsing them into one string is what forces a
|
|
/// guess at the other end, and a wrong guess here sends one provider's
|
|
/// credential to another provider's endpoint.
|
|
struct Credential {
|
|
/// The env var on THIS SERVER holding the secret.
|
|
source: &'static str,
|
|
/// The env var the GUEST's CLI reads it from.
|
|
target: &'static str,
|
|
}
|
|
|
|
/// The credential a backend's CLI authenticates with inside a VM.
|
|
///
|
|
/// Unknown backends are refused rather than given the Anthropic token: sending a
|
|
/// subscription credential to whatever endpoint an unrecognised backend points
|
|
/// at is worse than not launching.
|
|
fn microvm_credential_for(backend: Option<&str>) -> Result<Credential, String> {
|
|
match backend {
|
|
// `canary-*` backends are a REAL rootfs built from a candidate CLI
|
|
// version, run through the production path — egress, stop gate,
|
|
// delegation, delivery — before that version is promoted to the image
|
|
// every mission uses. They carry the same Anthropic subscription
|
|
// credential as `claude`, because testing a new CLI against a different
|
|
// provider would not be testing the thing we are about to ship.
|
|
//
|
|
// Named rather than pattern-matched on anything looser: an unrecognised
|
|
// backend must still be refused at launch, which is what
|
|
// `backend_can_run_a_mission` and the harness's negative control assert.
|
|
None | Some("") | Some("default") | Some("claude") | Some("canary-claude") => {
|
|
Ok(Credential {
|
|
source: "CLAUDE_CODE_OAUTH_TOKEN",
|
|
target: "CLAUDE_CODE_OAUTH_TOKEN",
|
|
})
|
|
}
|
|
// GLM. `images/agent-glm` bakes `ANTHROPIC_BASE_URL=https://api.z.ai/
|
|
// api/anthropic` into the rootfs, so the endpoint is a property of the
|
|
// image and the credential is a property of the turn. That split is what
|
|
// makes the dangerous mix-up unrepresentable: this VM cannot be handed
|
|
// an Anthropic token, and a `claude` VM cannot be pointed at z.ai.
|
|
Some("glm") => Ok(Credential {
|
|
source: "ZAI_API_KEY",
|
|
target: "ANTHROPIC_AUTH_TOKEN",
|
|
}),
|
|
Some("kimi") => Ok(Credential {
|
|
source: "KIMI_API_KEY",
|
|
target: "ANTHROPIC_AUTH_TOKEN",
|
|
}),
|
|
// A model running on the NODE ITSELF. There is no credential and there
|
|
// is nothing to protect: the guest reaches it over a vsock pipe to the
|
|
// node's own loopback (`clawmates-node::local_model`), which has no
|
|
// destination in its protocol and can therefore reach nothing else.
|
|
//
|
|
// It still goes through this map rather than around it. Ollama ignores
|
|
// the bearer but Claude Code refuses to start without one, so the value
|
|
// is a literal — and routing it here keeps the rule that a backend with
|
|
// no entry cannot launch, which is what stops a typo'd backend from
|
|
// quietly inheriting the subscription token.
|
|
Some("local-ornith") => Ok(Credential {
|
|
source: "CLAWMATES_LOCAL_MODEL_TOKEN",
|
|
target: "ANTHROPIC_AUTH_TOKEN",
|
|
}),
|
|
// Kimi, on the same split as GLM: endpoint in the image
|
|
// (`https://api.kimi.com/coding`), credential in the turn.
|
|
//
|
|
// Which HOST took a measurement to find. `api.moonshot.ai/anthropic`
|
|
// exists and speaks the protocol, and rejects an `sk-kimi-` key — it is
|
|
// the platform.moonshot.ai account namespace. The Kimi CODE service at
|
|
// `api.kimi.com/coding` is where such a key is valid, and it answers
|
|
// `/v1/messages` with a real Anthropic body. Two endpoints that both
|
|
// "work" for different accounts is exactly the shape that makes a
|
|
// guessed URL look like a broken key.
|
|
Some(other) => Err(format!(
|
|
"backend {other:?} has no defined microVM credential contract yet — \
|
|
refusing to launch rather than forward one provider's credential to \
|
|
another provider's endpoint"
|
|
)),
|
|
}
|
|
}
|
|
|
|
fn microvm_provider_env_from(
|
|
backend: Option<&str>,
|
|
lookup: impl Fn(&str) -> Option<String>,
|
|
) -> Result<Vec<(String, String)>, String> {
|
|
let want = microvm_credential_for(backend)?;
|
|
// A local model has no secret to look up. Defaulted rather than demanded:
|
|
// requiring an operator to set a variable whose value is ignored is a step
|
|
// that only ever fails, and its failure mode here is a hung agent.
|
|
let lookup = |k: &str| {
|
|
lookup(k).or_else(|| {
|
|
(k == "CLAWMATES_LOCAL_MODEL_TOKEN").then(|| "local-model-no-auth".to_string())
|
|
})
|
|
};
|
|
let token = lookup(want.source)
|
|
.filter(|v| !v.trim().is_empty())
|
|
.ok_or_else(|| {
|
|
format!(
|
|
"{} is not set on this server, so a microVM mission on backend \
|
|
{backend:?} would run `claude -p` with no credential — which hangs \
|
|
rather than failing. Set it, or run the mission on another backend.",
|
|
want.source
|
|
)
|
|
})?;
|
|
|
|
let mut env = vec![(want.target.to_string(), token)];
|
|
// Non-Anthropic providers a mission's tools may need, forwarded when set.
|
|
// ANTHROPIC_API_KEY is absent from this list and must stay absent — see the
|
|
// doc comment above.
|
|
for k in ["GROQ_API_KEY", "OPENAI_API_KEY"] {
|
|
if let Some(v) = lookup(k).filter(|v| !v.trim().is_empty()) {
|
|
env.push((k.to_string(), v));
|
|
}
|
|
}
|
|
debug_assert!(
|
|
!env.iter().any(|(k, _)| k == "ANTHROPIC_API_KEY"),
|
|
"the microVM path must never forward ANTHROPIC_API_KEY"
|
|
);
|
|
Ok(env)
|
|
}
|
|
|
|
/// The testable half of [`forwarded_provider_env`]. The lookup is a parameter
|
|
/// because a test cannot set process environment variables here — the workspace
|
|
/// denies `unsafe`, and `set_var` is racy across test threads regardless.
|
|
fn provider_env_from(
|
|
auth: RuntimeAuth,
|
|
lookup: impl Fn(&str) -> Option<String>,
|
|
) -> Vec<(String, String)> {
|
|
forwarded_provider_keys(auth)
|
|
.into_iter()
|
|
.filter_map(|k| {
|
|
lookup(k)
|
|
.filter(|v| !v.trim().is_empty())
|
|
.map(|v| (k.to_string(), v))
|
|
})
|
|
.collect()
|
|
}
|
|
|
|
/// Read `CLAWMATES_RUNTIME_AUTH`, defaulting to `api_key`.
|
|
///
|
|
/// Defaulting to the existing behaviour is deliberate: an unset or misspelled
|
|
/// value must not silently strip the API key and leave missions unable to
|
|
/// reach a model at all.
|
|
pub fn runtime_auth_mode() -> RuntimeAuth {
|
|
match std::env::var("CLAWMATES_RUNTIME_AUTH")
|
|
.unwrap_or_default()
|
|
.trim()
|
|
.to_ascii_lowercase()
|
|
.as_str()
|
|
{
|
|
"subscription" => RuntimeAuth::Subscription,
|
|
"" | "api_key" => RuntimeAuth::ApiKey,
|
|
other => {
|
|
eprintln!(
|
|
"mission_runtime: unknown CLAWMATES_RUNTIME_AUTH={other:?} — \
|
|
defaulting to api_key"
|
|
);
|
|
RuntimeAuth::ApiKey
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Docker networks the mission runtime container must be attached to.
|
|
/// - `clawmates_core`: talks to the server (skills door, API) + database
|
|
/// - `clawmates_missions`: has egress for outbound provider calls and fetches
|
|
///
|
|
/// Missions used to egress from `clawmates_edge`, the same network as the
|
|
/// SERVER. That made the host's egress policy impossible to scope: a mission
|
|
/// agent runs model-generated shell over content it fetched from the open web
|
|
/// and must not reach the tailnet, but the server on the same subnet must —
|
|
/// Beszel, Ollama, the node daemons. Measured 2026-09-18: the tailnet rule
|
|
/// written for missions cut the server off from architect:8090 within the
|
|
/// minute. A subnet of their own is what lets `clawmates-egress.sh` on gw-04
|
|
/// drop tailnet/private/link-local/ssh for missions and nothing else.
|
|
///
|
|
/// The network is declared by compose (prod: /opt/clawmates/docker-compose.yml;
|
|
/// local: deploy/compose/docker-compose.yml) with the same shape `edge` has —
|
|
/// not internal, so it routes off the host.
|
|
const CORE_NETWORK: &str = "clawmates_core";
|
|
const EDGE_NETWORK: &str = "clawmates_missions";
|
|
|
|
/// Host path (as seen by the docker engine, NOT the server container)
|
|
/// where the mission's checkouts live. Matches the mount source used
|
|
/// by `clawmates-runtime.service`.
|
|
///
|
|
/// This was a hardcoded const that read no env at all, while every writer of
|
|
/// the same tree went through `CLAWMATES_MISSIONS_ROOT`. They agree on this
|
|
/// deployment; they are not guaranteed to, and a reaper pointed at one of them
|
|
/// would have silently left the other's directories behind forever.
|
|
fn missions_host_root() -> String {
|
|
crate::mission_workspace::missions_root()
|
|
.to_string_lossy()
|
|
.into_owned()
|
|
}
|
|
|
|
/// Host path holding the shared ZeroClaw config + seeded agent library
|
|
/// (built up over time by the shared clawmates-runtime.service). Per-
|
|
/// mission runtime containers bind-mount this so their gateway
|
|
/// inherits all the `claw_*` agents that team_template_loader has
|
|
/// provisioned. Each per-mission container then MINTS ITS OWN pairing
|
|
/// code via /admin/paircode/new so the accepted-tokens list is
|
|
/// independent per mission. Overridable for dev via
|
|
/// `CLAWMATES_RUNTIME_SEED_DIR`.
|
|
///
|
|
/// Concurrency caveat: the sqlite files under .zeroclaw/data/ are
|
|
/// currently shared across all per-mission runtimes AND the shared
|
|
/// runtime. Concurrent daemons opening the same sessions.db can
|
|
/// interleave; in practice topology_worker sequentializes runs per
|
|
/// mission so this rarely bites. Long-term: copy-on-write per mission.
|
|
use crate::container_exec::MISSION_UID;
|
|
|
|
const DEFAULT_SEED_DIR: &str = "/root/clawmates-runtime/data";
|
|
|
|
/// What a mission gets its own copy of.
|
|
///
|
|
/// An allow-list, not the whole directory. The seed dir is **1.7 GB** on
|
|
/// gw-04 and 1.5 GB of that is a vestigial `.rustup` — a Rust toolchain that
|
|
/// installed itself into the data dir back when `HOME=/zeroclaw-data` and the
|
|
/// image had no toolchain. The image now ships Rust at `/usr/local/cargo`,
|
|
/// which is what the container's PATH actually resolves (verified live), so
|
|
/// that copy is dead weight. Copying it per mission would cost tens of
|
|
/// seconds and ~17 GB across ten concurrent missions.
|
|
///
|
|
/// So: copy what carries per-mission identity or secrets, and leave the
|
|
/// caches and toolchains behind.
|
|
const SEEDED_PATHS: &[&str] = &[
|
|
// The whole point: config.toml carries the §15 door bearer token, and
|
|
// data/ holds sessions.db + devices.db. ~26 MB.
|
|
".zeroclaw",
|
|
// Door MCP config — also a bearer token.
|
|
"clawmates-mcp.json",
|
|
// Claude Code's own state and credentials (~16 MB). Per-mission so a
|
|
// token refresh or project state in one mission cannot leak into another.
|
|
".claude",
|
|
".claude.json",
|
|
// Per-CLI state for the alternate backends; small. `glm-home` and
|
|
// `kimi-home` are separate HOMEs for the two `claude_cli` fallback
|
|
// aliases: each needs its own `.claude.json` so the binary skips
|
|
// onboarding, and keeping them apart stops two concurrent fallbacks from
|
|
// sharing one Claude Code session directory.
|
|
".kimi-code",
|
|
"glm-home",
|
|
"kimi-home",
|
|
// The seeded agent library.
|
|
"agents",
|
|
];
|
|
|
|
/// Deliberately NOT copied — caches and toolchains, no secrets, expensive:
|
|
/// `.rustup` (1.5 GB, vestigial), `.npm` (85 MB), `.cargo`, `.cache`,
|
|
/// `.local`. A mission that needs them reads the image's copies.
|
|
fn copy_script() -> String {
|
|
let mut out = String::from("set -e\n");
|
|
for p in SEEDED_PATHS {
|
|
// Missing entries are normal — a fresh deployment has no .kimi-code
|
|
// until Kimi is first used — so absence must not fail the copy.
|
|
out.push_str(&format!(
|
|
"if [ -e '/seed/{p}' ]; then cp -a '/seed/{p}' /dst/; fi\n"
|
|
));
|
|
}
|
|
// The copy itself must run as root: the seed dir is root-owned and parts of
|
|
// it are mode 0600 (`.claude.json`, `clawmates-mcp.json`), so uid 65532
|
|
// cannot even READ them. Running the container as 65532 made `cp` fail on
|
|
// the first unreadable entry, `set -e` aborted the rest, and the mission
|
|
// got a runtime-data holding `.zeroclaw` and nothing else — no Claude
|
|
// credentials, no door config.
|
|
//
|
|
// So keep root for the read, and hand the RESULT to 65532. That is what the
|
|
// destination needs: every other writer into the missions tree is 65532,
|
|
// and a GC running as 65532 cannot delete what root left behind.
|
|
out.push_str(&format!("chown -R {MISSION_UID} /dst\n"));
|
|
out
|
|
}
|
|
|
|
/// Give this mission its own copy of the runtime seed data.
|
|
///
|
|
/// Every per-mission container used to bind-mount the SAME host seed dir as
|
|
/// `/zeroclaw-data` — shared with each other and with the singleton runtime.
|
|
/// That directory holds `config.toml`, which carries the §15 door bearer
|
|
/// token, plus `sessions.db` and `devices.db`. So a mission could read another
|
|
/// mission's credential, and anything it wrote there was inherited by every
|
|
/// later mission. Teardown never cleaned it, because teardown only removes
|
|
/// `/var/lib/clawmates-missions/{id}`.
|
|
///
|
|
/// The code already knew: the comment on `DEFAULT_SEED_DIR` names the sqlite
|
|
/// race and calls copy-on-write per mission the long-term fix. This is that.
|
|
///
|
|
/// The copy runs in a throwaway container because cm-api cannot see the seed
|
|
/// dir — it hands that host path to Docker but never mounts it itself. The
|
|
/// runtime image is reused so nothing extra is pulled.
|
|
///
|
|
/// Failure is fatal to container creation on purpose. Falling back to the
|
|
/// shared mount would silently restore the credential-sharing this removes,
|
|
/// and a silent fallback to a weaker posture is the failure mode this
|
|
/// codebase keeps paying for.
|
|
async fn seed_runtime_data(
|
|
docker: &Docker,
|
|
image: &str,
|
|
seed_dir: &str,
|
|
dest_dir: &str,
|
|
) -> Result<(), String> {
|
|
let name = format!("cm-seed-{}", Uuid::now_v7().simple());
|
|
let config = ContainerCreateBody {
|
|
image: Some(image.to_string()),
|
|
entrypoint: Some(vec!["/bin/sh".to_string()]),
|
|
cmd: Some(vec!["-c".to_string(), copy_script()]),
|
|
host_config: Some(HostConfig {
|
|
mounts: Some(vec![
|
|
Mount {
|
|
target: Some("/seed".to_string()),
|
|
source: Some(seed_dir.to_string()),
|
|
typ: Some(MountTypeEnum::BIND),
|
|
read_only: Some(true),
|
|
..Default::default()
|
|
},
|
|
Mount {
|
|
target: Some("/dst".to_string()),
|
|
source: Some(dest_dir.to_string()),
|
|
typ: Some(MountTypeEnum::BIND),
|
|
read_only: Some(false),
|
|
..Default::default()
|
|
},
|
|
]),
|
|
auto_remove: Some(true),
|
|
..Default::default()
|
|
}),
|
|
..Default::default()
|
|
};
|
|
|
|
docker
|
|
.create_container(
|
|
Some(CreateContainerOptions {
|
|
name: Some(name.clone()),
|
|
..Default::default()
|
|
}),
|
|
config,
|
|
)
|
|
.await
|
|
.map_err(|e| format!("create seed copier: {e}"))?;
|
|
docker
|
|
.start_container(&name, None::<StartContainerOptions>)
|
|
.await
|
|
.map_err(|e| format!("start seed copier: {e}"))?;
|
|
|
|
// `auto_remove` means the container disappears the moment it exits, so
|
|
// poll for absence rather than waiting on it.
|
|
for _ in 0..120 {
|
|
tokio::time::sleep(std::time::Duration::from_millis(250)).await;
|
|
match docker
|
|
.inspect_container(&name, None::<InspectContainerOptions>)
|
|
.await
|
|
{
|
|
// `auto_remove` already took it away, which only happens after a
|
|
// clean-ish exit; the status is gone with it. Treated as success
|
|
// because the alternative — failing every mission on a race with
|
|
// the reaper — is worse, and `stamp_workspace_paths` fails loudly
|
|
// downstream if the config did not arrive.
|
|
Err(_) => return Ok(()),
|
|
Ok(info) => {
|
|
let state = info.state.as_ref();
|
|
let running = state.and_then(|st| st.running).unwrap_or(false);
|
|
if !running {
|
|
// The exit code was never read. A `cp` that died on an
|
|
// unreadable seed file left `set -e` to abort the rest, and
|
|
// this returned Ok — so the mission came up with a
|
|
// half-seeded runtime, the daemon never created its agents'
|
|
// workspace, and the phase then failed on
|
|
// "Could not find the file /mission in container",
|
|
// 200 lines and one indirection away from the real cause.
|
|
let code = state.and_then(|st| st.exit_code).unwrap_or(0);
|
|
if code != 0 {
|
|
return Err(format!(
|
|
"seed copier exited {code} — the mission's runtime-data is \
|
|
incomplete (the seed dir is root-owned and partly mode 0600; \
|
|
check that this container still runs as root)"
|
|
));
|
|
}
|
|
return Ok(());
|
|
}
|
|
}
|
|
}
|
|
}
|
|
Err(format!("seed copy into {dest_dir} did not finish in 30s"))
|
|
}
|
|
|
|
/// Deterministic docker container name for a mission's runtime.
|
|
/// Uses the full UUID hex — UUIDv7 encodes time in the leading bytes,
|
|
/// so a short prefix isn't guaranteed unique across missions minted
|
|
/// in the same second. Docker permits up to 253 characters in a name,
|
|
/// so the extra length is free.
|
|
pub fn container_name(mission_id: Uuid) -> String {
|
|
format!("cm-runtime-mission-{}", mission_id.simple())
|
|
}
|
|
|
|
/// Endpoint URL the topology_worker's ZeroClawDriveExecutor will dial.
|
|
/// Uses the container name as hostname — resolves within the shared
|
|
/// `clawmates_core` docker network.
|
|
pub fn endpoint_url(container_name: &str) -> String {
|
|
format!("http://{}:{}", container_name, GATEWAY_PORT)
|
|
}
|
|
|
|
pub struct MissionRuntimeProvisioner {
|
|
docker: Docker,
|
|
image: String,
|
|
}
|
|
|
|
/// What `ensure_container` returns: everything the caller needs to
|
|
/// point a topology_worker at this mission's fresh gateway.
|
|
#[derive(Debug, Clone)]
|
|
pub struct EnsuredContainer {
|
|
pub endpoint: String,
|
|
/// Where the tool hooks were written, or `None` if installing them failed.
|
|
///
|
|
/// Carried out to the caller rather than only logged, because the caller
|
|
/// has the pool and this struct's producer does not. Before this field
|
|
/// the outcome went to stderr and nowhere else, so a mission whose gate
|
|
/// never installed left a record indistinguishable from one whose gate
|
|
/// stood there all night and matched nothing.
|
|
pub hooks: Option<String>,
|
|
/// One-time pairing code minted by the daemon at boot; may be
|
|
/// None on the reuse-existing path when we couldn't scrape it
|
|
/// back (log rotation). Callers keep the previously-persisted
|
|
/// value in that case.
|
|
pub pairing_code: Option<String>,
|
|
}
|
|
|
|
impl MissionRuntimeProvisioner {
|
|
/// Connect to the docker engine. Honors `DOCKER_HOST` (set in
|
|
/// compose to the socket proxy) and falls back to the local
|
|
/// socket. Returns None when docker is unreachable, so callers
|
|
/// can degrade gracefully (missions still launch, just against
|
|
/// the shared runtime).
|
|
pub fn from_env() -> Option<MissionRuntimeProvisioner> {
|
|
let docker = if let Ok(host) = std::env::var("DOCKER_HOST") {
|
|
Docker::connect_with_http(&host, 30, bollard::API_DEFAULT_VERSION).ok()?
|
|
} else {
|
|
Docker::connect_with_local_defaults().ok()?
|
|
};
|
|
let (image, source) = match std::env::var("CLAWMATES_RUNTIME_IMAGE") {
|
|
Ok(v) if !v.trim().is_empty() => (v, "CLAWMATES_RUNTIME_IMAGE"),
|
|
_ => (DEFAULT_IMAGE.to_string(), "the built-in default"),
|
|
};
|
|
// ONCE per process. `from_env` is called per use — on every launch, and
|
|
// from the reaper sweep — so a bare eprintln here is a line every sweep
|
|
// tick forever, which is how a log stops being read at all.
|
|
static ANNOUNCED: std::sync::Once = std::sync::Once::new();
|
|
ANNOUNCED.call_once(|| {
|
|
eprintln!("mission_runtime: per-mission runtime image = {image} (from {source})");
|
|
});
|
|
Some(MissionRuntimeProvisioner { docker, image })
|
|
}
|
|
|
|
/// Is this container actually attached to the egress network?
|
|
///
|
|
/// Asked only when the attach reported an error, to tell "already connected"
|
|
/// apart from "not connected". Inspect failing is treated as NOT attached:
|
|
/// the whole point is to stop guessing that egress is present.
|
|
async fn is_on_edge_network(&self, name: &str) -> bool {
|
|
self.docker
|
|
.inspect_container(name, None::<bollard::query_parameters::InspectContainerOptions>)
|
|
.await
|
|
.ok()
|
|
.and_then(|c| c.network_settings?.networks)
|
|
.is_some_and(|nets| nets.contains_key(EDGE_NETWORK))
|
|
}
|
|
|
|
/// Idempotent: returns the endpoint URL, creating the container
|
|
/// on first call. If the container exists but is stopped, starts
|
|
/// it. If it exists and is running, returns its endpoint.
|
|
pub async fn ensure_container(&self, mission_id: Uuid) -> Result<EnsuredContainer, String> {
|
|
let name = container_name(mission_id);
|
|
// Fast path: already running.
|
|
if let Ok(inspect) = self
|
|
.docker
|
|
.inspect_container(&name, None::<InspectContainerOptions>)
|
|
.await
|
|
{
|
|
let running = inspect
|
|
.state
|
|
.as_ref()
|
|
.and_then(|s| s.running)
|
|
.unwrap_or(false);
|
|
if running {
|
|
// Reuse; try to re-scrape the pairing code from logs,
|
|
// but it may have rotated out — caller falls back to
|
|
// the previously-persisted value in that case.
|
|
// Re-install on reuse: the container outlives the server
|
|
// process, and a hook that exists only on first creation is a
|
|
// hook that quietly disappears after a redeploy.
|
|
let hooks = crate::container_tool_hooks::install(&self.docker, &name).await;
|
|
let pairing_code = self.mint_pairing_code(&name).await;
|
|
return Ok(EnsuredContainer {
|
|
endpoint: endpoint_url(&name),
|
|
pairing_code,
|
|
hooks,
|
|
});
|
|
}
|
|
// Exists but not running — remove + recreate below rather
|
|
// than trying to restart a dirty-state container.
|
|
let _ = self
|
|
.docker
|
|
.remove_container(
|
|
&name,
|
|
Some(RemoveContainerOptions {
|
|
force: true,
|
|
..Default::default()
|
|
}),
|
|
)
|
|
.await;
|
|
}
|
|
|
|
// Create fresh. Ensure the bind source exists first — research-
|
|
// only missions (no repo checkout) still need the directory
|
|
// present or docker start fails with EACCES/ENOENT.
|
|
let mission_dir = format!("{}/{mission_id}", missions_host_root());
|
|
let _ = tokio::fs::create_dir_all(&mission_dir).await;
|
|
let seed_dir = std::env::var("CLAWMATES_RUNTIME_SEED_DIR")
|
|
.unwrap_or_else(|_| DEFAULT_SEED_DIR.to_string());
|
|
// Per-mission copy of the seed data. See `seed_runtime_data`: sharing
|
|
// one directory meant sharing the door token and letting any mission
|
|
// poison every later one.
|
|
let runtime_data_dir = format!("{mission_dir}/runtime-data");
|
|
let _ = tokio::fs::create_dir_all(&runtime_data_dir).await;
|
|
seed_runtime_data(&self.docker, &self.image, &seed_dir, &runtime_data_dir).await?;
|
|
let mut mounts = Vec::new();
|
|
// In copy mode the checkout is pushed in and pulled back out, so the
|
|
// container gets its OWN filesystem and the host directory has exactly
|
|
// one writer (the server). Binding it here would put two uids back on
|
|
// one directory — the cause of four separate work-loss bugs.
|
|
if !crate::mission_fs::copy_mode() {
|
|
mounts.push(Mount {
|
|
target: Some("/mission".to_string()),
|
|
source: Some(mission_dir.clone()),
|
|
typ: Some(MountTypeEnum::BIND),
|
|
read_only: Some(false),
|
|
..Default::default()
|
|
});
|
|
}
|
|
mounts.extend([
|
|
// This mission's OWN copy of the seeded agent library
|
|
// (`claw_*` templates). Copied rather than shared, so its
|
|
// config.toml — which carries the door bearer token — and its
|
|
// sqlite files belong to this mission alone and are removed with
|
|
// it by `teardown_container`.
|
|
Mount {
|
|
target: Some("/zeroclaw-data".to_string()),
|
|
source: Some(runtime_data_dir),
|
|
typ: Some(MountTypeEnum::BIND),
|
|
read_only: Some(false),
|
|
..Default::default()
|
|
},
|
|
]);
|
|
|
|
let host_config = HostConfig {
|
|
mounts: Some(mounts),
|
|
restart_policy: Some(bollard::models::RestartPolicy {
|
|
name: Some(bollard::models::RestartPolicyNameEnum::UNLESS_STOPPED),
|
|
..Default::default()
|
|
}),
|
|
network_mode: Some(CORE_NETWORK.to_string()),
|
|
..Default::default()
|
|
};
|
|
|
|
// NOTE: do NOT set ZEROCLAW_WORKSPACE — despite the name, the
|
|
// daemon uses it (schema.rs:17467) as a legacy config-dir
|
|
// pointer that overrides ZEROCLAW_CONFIG_DIR/DATA_DIR. Setting
|
|
// it to /mission/repo makes the daemon compute its config dir
|
|
// as /mission/repo/.zeroclaw (empty!) and boot with a fresh
|
|
// defaults-only config — zero agents loaded.
|
|
//
|
|
// Per-agent workspace pinning belongs in the shared config
|
|
// under agents.<alias>.workspace = "/mission/repo", not env.
|
|
let mut env = vec![
|
|
format!("ZEROCLAW_GATEWAY_PORT={GATEWAY_PORT}"),
|
|
format!("CM_MISSION_ID={mission_id}"),
|
|
// Let the agents' git read the checkout.
|
|
//
|
|
// The server clones as uid 65532; this container runs as root, so
|
|
// every `git` an agent runs hits "detected dubious ownership" and
|
|
// refuses the repository. Agents do not report that as a failure —
|
|
// they improvise. On mission 019fc3ba one wrote a `.gitconfig_temp`
|
|
// containing `[safe] directory = /mission/repo` into the repository
|
|
// root, which then showed up in the captured diff and would have
|
|
// been committed and pushed to the user's repo alongside the real
|
|
// work.
|
|
//
|
|
// `GIT_CONFIG_*` is git's environment form of `-c` and is
|
|
// inherited by subprocesses, so it covers the agent's own git, any
|
|
// tool that shells out to git, and the `git_operations` tool alike.
|
|
// Scoped to the checkout; never `--global`.
|
|
"GIT_CONFIG_COUNT=1".to_string(),
|
|
"GIT_CONFIG_KEY_0=safe.directory".to_string(),
|
|
"GIT_CONFIG_VALUE_0=/mission/repo".to_string(),
|
|
];
|
|
// Provider credentials forwarded into the container.
|
|
//
|
|
// ANTHROPIC_API_KEY is conditional, and the reason is subtle enough to
|
|
// be worth stating: Claude Code resolves credentials in a fixed
|
|
// priority order, and ANTHROPIC_API_KEY ranks ABOVE the subscription's
|
|
// CLAUDE_CODE_OAUTH_TOKEN. So on a runtime authenticated via `claude
|
|
// /login`, forwarding the key here silently wins — `claude` still works,
|
|
// the agents still run, and every mission bills the API while appearing
|
|
// to use the subscription. Failing loudly is impossible; the only fix
|
|
// is not to send it.
|
|
//
|
|
// The other three are unrelated providers (Gemini/Groq/OpenAI) with no
|
|
// subscription equivalent, so they forward in both modes.
|
|
let auth_mode = runtime_auth_mode();
|
|
for (key, v) in forwarded_provider_env(auth_mode) {
|
|
env.push(format!("{key}={v}"));
|
|
}
|
|
eprintln!(
|
|
"mission_runtime: mission {mission_id} container auth mode = {} \
|
|
(ANTHROPIC_API_KEY {})",
|
|
auth_mode.as_str(),
|
|
if auth_mode == RuntimeAuth::ApiKey {
|
|
"forwarded"
|
|
} else {
|
|
"withheld so the runtime's subscription login is used"
|
|
}
|
|
);
|
|
|
|
let mut labels = HashMap::new();
|
|
labels.insert("clawmates.role".to_string(), "mission-runtime".to_string());
|
|
labels.insert("clawmates.mission_id".to_string(), mission_id.to_string());
|
|
|
|
let config = ContainerCreateBody {
|
|
image: Some(self.image.clone()),
|
|
cmd: Some(vec![
|
|
"daemon".to_string(),
|
|
"--host".to_string(),
|
|
"0.0.0.0".to_string(),
|
|
]),
|
|
env: Some(env),
|
|
host_config: Some(host_config),
|
|
labels: Some(labels),
|
|
..Default::default()
|
|
};
|
|
|
|
self.docker
|
|
.create_container(
|
|
Some(CreateContainerOptions {
|
|
name: Some(name.clone()),
|
|
..Default::default()
|
|
}),
|
|
config,
|
|
)
|
|
.await
|
|
.map_err(|e| format!("create mission runtime container: {e}"))?;
|
|
|
|
// Attach to the edge network for outbound provider egress.
|
|
//
|
|
// `clawmates_core` is `internal: true` and has NO default route —
|
|
// verified from a container on it, where every external address is
|
|
// unreachable. So this attach is not an optimisation: without it the
|
|
// mission cannot reach a provider, cannot fetch anything, and cannot do
|
|
// its work. The result used to be discarded, which made a failure here
|
|
// indistinguishable from success and produced the green-with-nothing
|
|
// shape this codebase keeps meeting.
|
|
//
|
|
// Not fatal on the error alone: re-attaching an already-connected
|
|
// container is an error too, and a benign one on any relaunch path. The
|
|
// container's own network list is the fact that settles it.
|
|
if let Err(e) = self
|
|
.docker
|
|
.connect_network(
|
|
EDGE_NETWORK,
|
|
NetworkConnectRequest {
|
|
container: Some(name.clone()),
|
|
endpoint_config: Some(EndpointSettings::default()),
|
|
},
|
|
)
|
|
.await
|
|
{
|
|
if self.is_on_edge_network(&name).await {
|
|
eprintln!(
|
|
"mission_runtime: {name} was already on {EDGE_NETWORK} ({e}) — \
|
|
egress is present, continuing"
|
|
);
|
|
} else {
|
|
return Err(format!(
|
|
"attach mission runtime container to {EDGE_NETWORK}: {e} — \
|
|
{CORE_NETWORK} is internal and has no route off the host, so \
|
|
this mission would run with no egress at all: every provider \
|
|
call and every fetch would fail while the phase still \
|
|
reported completion. Is the `missions` network declared in \
|
|
the compose file and created (`docker network ls`)?"
|
|
));
|
|
}
|
|
}
|
|
|
|
self.docker
|
|
.start_container(&name, None::<StartContainerOptions>)
|
|
.await
|
|
.map_err(|e| format!("start mission runtime container: {e}"))?;
|
|
|
|
// Give the daemon a moment to print its boot banner, then
|
|
// scrape the pairing code. Poll with a short deadline so a
|
|
// slow boot doesn't hang the launch — the topology_worker
|
|
// will retry pairing later if we came up empty.
|
|
let pairing_code = self.wait_for_pairing_code(&name).await;
|
|
|
|
// Gate and observe the tools claude runs inside its own subprocess.
|
|
// Best-effort by design: a phase that runs unhooked still delivers, and
|
|
// failing the launch to protect telemetry would be the wrong trade.
|
|
let hooks = crate::container_tool_hooks::install(&self.docker, &name).await;
|
|
|
|
Ok(EnsuredContainer {
|
|
endpoint: endpoint_url(&name),
|
|
pairing_code,
|
|
hooks,
|
|
})
|
|
}
|
|
|
|
/// Poll the daemon's localhost admin endpoint until it responds
|
|
/// with a fresh pairing code. Boot takes ~1-2s; deadline is 15s.
|
|
/// Returns None on timeout so callers see a clear paired=false
|
|
/// signal in the mission binding log.
|
|
async fn wait_for_pairing_code(&self, name: &str) -> Option<String> {
|
|
let deadline = std::time::Duration::from_secs(15);
|
|
let start = std::time::Instant::now();
|
|
while start.elapsed() < deadline {
|
|
if let Some(code) = self.mint_pairing_code(name).await {
|
|
return Some(code);
|
|
}
|
|
tokio::time::sleep(std::time::Duration::from_millis(500)).await;
|
|
}
|
|
None
|
|
}
|
|
|
|
/// `docker exec` into the container and hit the local admin
|
|
/// endpoint that mints a fresh pairing code. This works whether
|
|
/// the daemon booted "already paired" (no code in the log) or
|
|
/// "pairing required" (code in the log) — both surfaces mint on
|
|
/// demand.
|
|
async fn mint_pairing_code(&self, name: &str) -> Option<String> {
|
|
let exec = self
|
|
.docker
|
|
.create_exec(
|
|
name,
|
|
CreateExecOptions {
|
|
cmd: Some(
|
|
[
|
|
"curl",
|
|
"-fs",
|
|
"-X",
|
|
"POST",
|
|
"http://127.0.0.1:42617/admin/paircode/new",
|
|
]
|
|
.iter()
|
|
.map(|s| s.to_string())
|
|
.collect(),
|
|
),
|
|
attach_stdout: Some(true),
|
|
attach_stderr: Some(true),
|
|
..Default::default()
|
|
},
|
|
)
|
|
.await
|
|
.ok()?;
|
|
let started = self.docker.start_exec(&exec.id, None).await.ok()?;
|
|
let StartExecResults::Attached { mut output, .. } = started else {
|
|
return None;
|
|
};
|
|
let mut buf = String::new();
|
|
while let Some(chunk) = output.next().await {
|
|
if let Ok(c) = chunk {
|
|
buf.push_str(&c.to_string());
|
|
if buf.len() > 8_000 {
|
|
break;
|
|
}
|
|
}
|
|
}
|
|
extract_pairing_code_from_json(&buf)
|
|
}
|
|
|
|
/// Stamp `[agents.<alias>.workspace] path = "<workspace_path>"` into the
|
|
/// shared runtime config for each provisioned claw, by editing the
|
|
/// config file directly on the per-mission container. This is the
|
|
/// out-of-band path for workspace pinning: the ZeroClaw config prop API
|
|
/// cannot set `workspace.path` (an `Option<PathBuf>` the `Configurable`
|
|
/// macro skips from prop enumeration), so `provision_claw` leaves it
|
|
/// unset and we stamp it here. The daemon applies the change on the
|
|
/// same config reload that surfaces the freshly-provisioned claws for
|
|
/// the run.
|
|
///
|
|
/// Concurrency caveat (mirrors the seed-dir note above): the config
|
|
/// file is shared across the persistent runtime and every per-mission
|
|
/// daemon, so this read-modify-write can race a provision from another
|
|
/// mission launching at the same instant. Missions launch one at a
|
|
/// time in practice; the durable fix is per-mission config isolation.
|
|
pub async fn pin_agent_workspaces(
|
|
&self,
|
|
mission_id: Uuid,
|
|
claws: &[cm_domain::AgentId],
|
|
workspace_path: &str,
|
|
) -> Result<(), String> {
|
|
if claws.is_empty() {
|
|
return Ok(());
|
|
}
|
|
const CONFIG_PATH: &str = "/zeroclaw-data/.zeroclaw/config.toml";
|
|
let name = container_name(mission_id);
|
|
|
|
let raw = self
|
|
.exec_capture(&name, vec!["cat".into(), CONFIG_PATH.into()])
|
|
.await?;
|
|
let aliases: Vec<String> = claws
|
|
.iter()
|
|
.map(|c| crate::runtime_provision::claw_alias(c.as_uuid()))
|
|
.collect();
|
|
let (edited, pinned) = stamp_workspace_paths(&raw, &aliases, workspace_path)?;
|
|
// Pinning NOTHING is a failure, not a no-op. Returning Ok here meant the
|
|
// caller's deliberately-fatal guard could not fire, so a mission whose
|
|
// aliases were missing from the config launched anyway with every agent
|
|
// writing into its own sandbox and delivering nothing — the outcome that
|
|
// guard's error message already describes. Name the aliases: the only
|
|
// way this happens is a config/alias mismatch, and the aliases are the
|
|
// evidence needed to find it.
|
|
if pinned == 0 {
|
|
return Err(format!(
|
|
"pinned 0 of {} agent workspace(s) to {workspace_path} — none of these \
|
|
aliases exist in {CONFIG_PATH}: {}",
|
|
aliases.len(),
|
|
aliases.join(", ")
|
|
));
|
|
}
|
|
|
|
// Upload rather than exec. This used to base64 the whole config into a
|
|
// single `sh -c` argument, which works until the file grows — config
|
|
// gains a block per provisioned claw — and then fails with
|
|
// `argument list too long`. Mission `019fcf62` hit exactly that: the
|
|
// pin never applied, its agents never saw `/mission/repo`, and phase 0
|
|
// completed having written nothing. The tar API has no argv limit, so
|
|
// the failure mode is gone rather than merely further away.
|
|
crate::mission_fs::put_file(&self.docker, &name, CONFIG_PATH, edited.as_bytes())
|
|
.await
|
|
.map_err(|e| format!("write runtime config.toml: {e}"))?;
|
|
eprintln!(
|
|
"mission_runtime: pinned {pinned} workspace(s) → {workspace_path} for mission {mission_id}"
|
|
);
|
|
Ok(())
|
|
}
|
|
|
|
/// Run a command in the mission container and return its combined
|
|
/// stdout+stderr as a String. Used for small config round-trips.
|
|
async fn exec_capture(&self, name: &str, cmd: Vec<String>) -> Result<String, String> {
|
|
let exec = self
|
|
.docker
|
|
.create_exec(
|
|
name,
|
|
CreateExecOptions {
|
|
cmd: Some(cmd),
|
|
attach_stdout: Some(true),
|
|
attach_stderr: Some(true),
|
|
..Default::default()
|
|
},
|
|
)
|
|
.await
|
|
.map_err(|e| format!("create_exec on {name}: {e}"))?;
|
|
let started = self
|
|
.docker
|
|
.start_exec(&exec.id, None)
|
|
.await
|
|
.map_err(|e| format!("start_exec on {name}: {e}"))?;
|
|
let StartExecResults::Attached { mut output, .. } = started else {
|
|
return Err(format!("exec on {name} returned a detached result"));
|
|
};
|
|
let mut buf = String::new();
|
|
while let Some(chunk) = output.next().await {
|
|
match chunk {
|
|
Ok(c) => buf.push_str(&c.to_string()),
|
|
Err(e) => return Err(format!("exec output stream on {name}: {e}")),
|
|
}
|
|
}
|
|
Ok(buf)
|
|
}
|
|
}
|
|
|
|
/// Format-preserving stamp of `[agents.<alias>.workspace] path = "<path>"`
|
|
/// for each alias present in `raw`. Keeps the operator's comments, ordering,
|
|
/// and every untouched byte intact; only the `path` keys change. Aliases not
|
|
/// already present are skipped (never fabricated — a bare agent table would
|
|
/// drop that agent's model/risk_profile/bundles). Returns the edited document
|
|
/// and how many agents were pinned.
|
|
fn stamp_workspace_paths(
|
|
raw: &str,
|
|
aliases: &[String],
|
|
workspace_path: &str,
|
|
) -> Result<(String, usize), String> {
|
|
let mut doc = raw
|
|
.parse::<toml_edit::DocumentMut>()
|
|
.map_err(|e| format!("parse runtime config.toml: {e}"))?;
|
|
let Some(agents) = doc.get_mut("agents").and_then(|i| i.as_table_like_mut()) else {
|
|
return Err("runtime config has no [agents] table".to_string());
|
|
};
|
|
let mut pinned = 0usize;
|
|
for alias in aliases {
|
|
let Some(agent) = agents.get_mut(alias).and_then(|i| i.as_table_like_mut()) else {
|
|
eprintln!("mission_runtime: pin skip — {alias} absent from config");
|
|
continue;
|
|
};
|
|
if agent.get("workspace").is_none() {
|
|
agent.insert("workspace", toml_edit::Item::Table(toml_edit::Table::new()));
|
|
}
|
|
if let Some(ws) = agent
|
|
.get_mut("workspace")
|
|
.and_then(|i| i.as_table_like_mut())
|
|
{
|
|
ws.insert("path", toml_edit::value(workspace_path));
|
|
pinned += 1;
|
|
}
|
|
}
|
|
Ok((doc.to_string(), pinned))
|
|
}
|
|
|
|
/// Parse the JSON `{ "pairing_code": "NNNNNN", ... }` body from the
|
|
/// admin/paircode/new endpoint.
|
|
fn extract_pairing_code_from_json(body: &str) -> Option<String> {
|
|
let v: serde_json::Value = serde_json::from_str(body.trim()).ok()?;
|
|
v.get("pairing_code")
|
|
.and_then(|x| x.as_str())
|
|
.filter(|s| !s.is_empty())
|
|
.map(String::from)
|
|
}
|
|
|
|
impl MissionRuntimeProvisioner {
|
|
/// Restart the mission's runtime container and wait for its gateway to
|
|
/// answer again. Needed after `pin_agent_workspaces`: the daemon reads
|
|
/// config once at boot and never re-reads the file, so a file-only setting
|
|
/// (`workspace.path`, which the config prop API cannot set) only takes
|
|
/// effect across a restart. Agents provisioned through the daemon's own
|
|
/// config API are already persisted to that file, so they survive.
|
|
pub async fn restart_container(&self, mission_id: Uuid) -> Result<(), String> {
|
|
let name = container_name(mission_id);
|
|
self.docker
|
|
.restart_container(
|
|
&name,
|
|
None::<bollard::query_parameters::RestartContainerOptions>,
|
|
)
|
|
.await
|
|
.map_err(|e| format!("restart mission runtime container: {e}"))?;
|
|
// Wait for the gateway to serve again so the caller can launch a run
|
|
// immediately after. ~20s ceiling; the daemon normally boots in ~2s.
|
|
for _ in 0..40 {
|
|
tokio::time::sleep(std::time::Duration::from_millis(500)).await;
|
|
// NOTE: exec_capture only fails on docker errors — a curl that
|
|
// can't connect still "succeeds" (it prints `curl: (7) …`), so we
|
|
// must inspect the BODY. /health answers `{"paired":…}`.
|
|
let body = self
|
|
.exec_capture(
|
|
&name,
|
|
vec![
|
|
"curl".into(),
|
|
"-fsS".into(),
|
|
"-m".into(),
|
|
"2".into(),
|
|
format!("http://127.0.0.1:{GATEWAY_PORT}/health"),
|
|
],
|
|
)
|
|
.await
|
|
.unwrap_or_default();
|
|
if body.contains("\"paired\"") {
|
|
eprintln!("mission_runtime: restarted {name}, gateway healthy");
|
|
return Ok(());
|
|
}
|
|
}
|
|
Err(format!("{name} gateway did not come back after restart"))
|
|
}
|
|
|
|
/// Does this mission's runtime container still exist, in any state?
|
|
///
|
|
/// Used by the sweeper to decide whether a failed teardown left something
|
|
/// behind. Deliberately treats an inspect error as "gone": the caller uses
|
|
/// this to decide whether to KEEP a binding for a retry, and answering
|
|
/// "still there" when docker cannot be reached would pin the binding open
|
|
/// on an unreachable daemon rather than on a real container.
|
|
/// Every `cm-runtime-mission-*` container on this engine, running or not.
|
|
///
|
|
/// The piece the row-driven sweep never had. Without it "which containers
|
|
/// exist" is a question the platform cannot ask, and a container the
|
|
/// database has forgotten is not merely unreaped — it is unseeable.
|
|
pub async fn list_mission_containers(&self) -> Result<Vec<(String, Option<i64>)>, String> {
|
|
let mut filters = std::collections::HashMap::new();
|
|
filters.insert("name".to_string(), vec!["cm-runtime-mission-".to_string()]);
|
|
let opts = bollard::query_parameters::ListContainersOptionsBuilder::default()
|
|
.all(true)
|
|
.filters(&filters)
|
|
.build();
|
|
let list = self
|
|
.docker
|
|
.list_containers(Some(opts))
|
|
.await
|
|
.map_err(|e| format!("list mission containers: {e}"))?;
|
|
Ok(list
|
|
.into_iter()
|
|
.filter_map(|c| {
|
|
let name = c
|
|
.names
|
|
.unwrap_or_default()
|
|
.into_iter()
|
|
// Docker returns names with a leading slash.
|
|
.map(|n| n.trim_start_matches('/').to_string())
|
|
.find(|n| n.starts_with("cm-runtime-mission-"))?;
|
|
Some((name, c.created))
|
|
})
|
|
.collect())
|
|
}
|
|
|
|
/// How long ago docker says this container was created.
|
|
///
|
|
/// Taken from the listing rather than a second `inspect`: `created` is
|
|
/// already a unix timestamp there, so this needs neither a date parser nor
|
|
/// another round-trip. `None` when docker reported none, and the caller
|
|
/// treats that as "do not reap" — a container we cannot date is exactly the
|
|
/// one worth leaving.
|
|
pub fn container_age(created_epoch: Option<i64>) -> Option<std::time::Duration> {
|
|
let created = created_epoch?;
|
|
let now = std::time::SystemTime::now()
|
|
.duration_since(std::time::UNIX_EPOCH)
|
|
.ok()?
|
|
.as_secs() as i64;
|
|
u64::try_from(now - created).ok().map(std::time::Duration::from_secs)
|
|
}
|
|
|
|
/// Does this container's checkout hold commits no remote has?
|
|
///
|
|
/// Answered by `git` inside the container, because only it knows which
|
|
/// refs the remote had. `--not --remotes` lists every commit reachable
|
|
/// from any local ref and from no remote-tracking ref — which is exactly
|
|
/// "work that exists only here".
|
|
///
|
|
/// Every failure path returns `SomeOrUnknown`. A container we cannot
|
|
/// question is not a container we may delete.
|
|
pub async fn unpushed_commits(&self, name: &str) -> UnpushedWork {
|
|
let script = "cd /mission/repo 2>/dev/null || exit 91; \
|
|
git rev-list --all --not --remotes 2>/dev/null | wc -l";
|
|
let argv = vec!["sh".to_string(), "-lc".to_string(), script.to_string()];
|
|
let out = match crate::container_exec::exec_as_root(
|
|
&self.docker,
|
|
name,
|
|
None,
|
|
&argv,
|
|
std::time::Duration::from_secs(30),
|
|
)
|
|
.await
|
|
{
|
|
Ok(o) => o,
|
|
Err(e) => {
|
|
return UnpushedWork::SomeOrUnknown(format!("could not ask git ({e})"));
|
|
}
|
|
};
|
|
if out.exit_code == Some(91) {
|
|
// No checkout at all — nothing to lose.
|
|
return UnpushedWork::None;
|
|
}
|
|
if out.exit_code != Some(0) {
|
|
return UnpushedWork::SomeOrUnknown(format!(
|
|
"git probe exited {:?}",
|
|
out.exit_code
|
|
));
|
|
}
|
|
match out.stdout.trim().parse::<u64>() {
|
|
Ok(0) => UnpushedWork::None,
|
|
Ok(n) => UnpushedWork::SomeOrUnknown(format!(
|
|
"{n} commit(s) in its checkout are on no remote"
|
|
)),
|
|
Err(_) => UnpushedWork::SomeOrUnknown(format!(
|
|
"unreadable git output {:?}",
|
|
out.stdout.trim()
|
|
)),
|
|
}
|
|
}
|
|
|
|
pub async fn container_exists(&self, mission_id: Uuid) -> bool {
|
|
self.docker
|
|
.inspect_container(&container_name(mission_id), None::<InspectContainerOptions>)
|
|
.await
|
|
.is_ok()
|
|
}
|
|
|
|
/// Force-remove the mission's runtime container AND its host workspace
|
|
/// dir (the `/mission/repo` checkout bind source). Idempotent: a missing
|
|
/// container or dir is not an error — this is called both by the terminal
|
|
/// sweeper and by mission delete, where the container may already be gone.
|
|
pub async fn teardown_container(&self, mission_id: Uuid) -> Result<(), String> {
|
|
let name = container_name(mission_id);
|
|
if let Err(e) = self
|
|
.docker
|
|
.remove_container(
|
|
&name,
|
|
Some(RemoveContainerOptions {
|
|
force: true,
|
|
..Default::default()
|
|
}),
|
|
)
|
|
.await
|
|
{
|
|
// 404 (already gone) is fine; anything else is worth surfacing.
|
|
let msg = e.to_string();
|
|
if !msg.contains("No such container") && !msg.contains("404") {
|
|
return Err(format!("remove mission runtime container: {e}"));
|
|
}
|
|
}
|
|
// Remove the per-mission workspace dir (repo checkout + scratch). This
|
|
// path is bind-mounted into cm-api, so we can reap it directly.
|
|
let mission_dir = format!("{}/{mission_id}", missions_host_root());
|
|
if let Err(e) = tokio::fs::remove_dir_all(&mission_dir).await {
|
|
match e.kind() {
|
|
std::io::ErrorKind::NotFound => {}
|
|
// The server runs as 65532 and cannot delete root-owned files.
|
|
// Almost everything under a mission is 65532 now, but the
|
|
// per-mission ZeroClaw daemon still runs as root and leaves ~26
|
|
// of its own files (`.claude.json`, session jsonl) behind after
|
|
// the seed copy has been chowned. Without this fallback those
|
|
// few files keep the whole directory alive forever — the
|
|
// cleanup-that-cannot-clean-up shape, at a scale small enough
|
|
// to go unnoticed for a long time.
|
|
std::io::ErrorKind::PermissionDenied => {
|
|
eprintln!(
|
|
"mission_runtime: {mission_dir} holds root-owned files — removing it from inside the runtime container"
|
|
);
|
|
let container = std::env::var("CLAWMATES_RUNTIME_CONTAINER")
|
|
.unwrap_or_else(|_| "clawmates-runtime".to_string());
|
|
crate::root_copy::purge(&container, std::path::Path::new(&mission_dir)).await;
|
|
if tokio::fs::metadata(&mission_dir).await.is_ok() {
|
|
eprintln!(
|
|
"mission_runtime: {mission_dir} SURVIVED the root purge — it will keep accumulating"
|
|
);
|
|
}
|
|
}
|
|
_ => eprintln!("mission_runtime: rm workspace dir {mission_dir}: {e}"),
|
|
}
|
|
}
|
|
Ok(())
|
|
}
|
|
}
|
|
|
|
/// Background sweeper: force-remove runtime containers for missions
|
|
/// that reached a terminal state ≥ `grace` ago. Keeps the container
|
|
/// around briefly after `completed`/`failed`/`cancelled` so the
|
|
/// operator can re-open the UI and pull the last checkpoint before
|
|
/// the daemon disappears. Runs on the same cadence as the phase
|
|
/// runner (10s) with a much longer per-mission grace.
|
|
pub fn spawn_sweeper(pool: sqlx::PgPool, grace: std::time::Duration) {
|
|
tokio::spawn(async move {
|
|
// Wait past the phase_runner boot so we don't fight over
|
|
// just-launched missions.
|
|
tokio::time::sleep(std::time::Duration::from_secs(30)).await;
|
|
let mut ticker = tokio::time::interval(std::time::Duration::from_secs(60));
|
|
ticker.tick().await;
|
|
loop {
|
|
ticker.tick().await;
|
|
if let Err(e) = sweep_once(&pool, grace).await {
|
|
eprintln!("mission_runtime::sweeper: sweep failed: {e}");
|
|
}
|
|
if let Err(e) = sweep_orphans(&pool, ORPHAN_GRACE).await {
|
|
eprintln!("mission_runtime::sweeper: orphan sweep failed: {e}");
|
|
}
|
|
}
|
|
});
|
|
}
|
|
|
|
/// How long a container with no mission row may sit before it is reaped.
|
|
///
|
|
/// Long, deliberately. The row-driven sweep above handles every container the
|
|
/// platform still knows about, so anything reaching this path is already
|
|
/// unexpected — and the one real orphan we have seen held ten unpushed commits.
|
|
/// A day of disk is cheaper than being wrong about that.
|
|
const ORPHAN_GRACE: std::time::Duration = std::time::Duration::from_secs(24 * 3600);
|
|
|
|
/// Reap `cm-runtime-mission-*` containers that no `missions` row points at.
|
|
///
|
|
/// [`sweep_once`] selects `FROM missions`, and `teardown_container` is only
|
|
/// ever called with an id that came from that query. So a container whose row
|
|
/// is gone is invisible to every reaper: nothing enumerates docker, nothing
|
|
/// errors, and the only symptom is disk.
|
|
///
|
|
/// Found on gw-04 2026-08-21 — a container `Up` for nine days holding 2.5G,
|
|
/// against a `missions` table with **zero rows**.
|
|
///
|
|
/// # It refuses to reap work that exists nowhere else
|
|
///
|
|
/// That container's checkout held **ten commits on a branch that had never
|
|
/// been pushed** (+3451/-30 across 30 files). A reaper that deleted on sight
|
|
/// would have destroyed all of it, silently, as its designed behaviour. So
|
|
/// before removing anything this asks the checkout whether it holds commits
|
|
/// that no remote has, and leaves the container alone — loudly, every tick —
|
|
/// when it does.
|
|
///
|
|
/// The check is deliberately one-sided in the safe direction: an inspection
|
|
/// that fails for any reason counts as "might hold work", never as "safe to
|
|
/// delete". Losing a day of disk to an unreadable container is recoverable;
|
|
/// the other way round is not.
|
|
pub async fn sweep_orphans(pool: &sqlx::PgPool, grace: std::time::Duration) -> Result<(), String> {
|
|
let Some(prov) = MissionRuntimeProvisioner::from_env() else {
|
|
return Ok(());
|
|
};
|
|
let names = prov.list_mission_containers().await?;
|
|
if names.is_empty() {
|
|
return Ok(());
|
|
}
|
|
for (name, created) in names {
|
|
let Some(id) = mission_id_from_container(&name) else {
|
|
continue;
|
|
};
|
|
// `WHERE id = $1` across every workspace on purpose: the question is
|
|
// whether ANY row still points at this container, not whether one the
|
|
// caller can see does.
|
|
let known: Option<(Uuid,)> = sqlx::query_as("SELECT id FROM missions WHERE id = $1")
|
|
.bind(id)
|
|
.fetch_optional(pool)
|
|
.await
|
|
.map_err(|e| format!("look up mission {id}: {e}"))?;
|
|
if known.is_some() {
|
|
continue;
|
|
}
|
|
match MissionRuntimeProvisioner::container_age(created) {
|
|
Some(age) if age < grace => continue,
|
|
None => continue,
|
|
Some(_) => {}
|
|
}
|
|
match prov.unpushed_commits(&name).await {
|
|
// The safe answer, and the one an error also produces.
|
|
UnpushedWork::SomeOrUnknown(why) => {
|
|
eprintln!(
|
|
"mission_runtime::orphans: {name} has no mission row and is older than \
|
|
the grace period, but it is NOT safe to reap: {why}. Recover the work \
|
|
(`git bundle create … origin/main..HEAD`, or push the branch) and then \
|
|
remove it by hand."
|
|
);
|
|
}
|
|
UnpushedWork::None => {
|
|
eprintln!(
|
|
"mission_runtime::orphans: reaping {name} — no mission row, older than \
|
|
the grace period, and its checkout holds nothing a remote does not"
|
|
);
|
|
if let Err(e) = prov.teardown_container(id).await {
|
|
eprintln!("mission_runtime::orphans: reap {name}: {e}");
|
|
}
|
|
}
|
|
}
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
/// Whether an orphan's checkout holds commits no remote has.
|
|
#[derive(Debug, PartialEq, Eq)]
|
|
pub enum UnpushedWork {
|
|
/// Every commit is reachable from a remote ref — nothing is lost.
|
|
None,
|
|
/// There IS unpushed work, or the question could not be answered. One
|
|
/// variant for both, because the reaper must treat them identically.
|
|
SomeOrUnknown(String),
|
|
}
|
|
|
|
/// The mission id encoded in a runtime container's name, if it is one.
|
|
///
|
|
/// The inverse of [`container_name`], which formats the uuid `simple` (no
|
|
/// dashes). Anything that does not parse is not ours and is left alone.
|
|
pub fn mission_id_from_container(name: &str) -> Option<Uuid> {
|
|
Uuid::parse_str(name.strip_prefix("cm-runtime-mission-")?).ok()
|
|
}
|
|
|
|
async fn sweep_once(pool: &sqlx::PgPool, grace: std::time::Duration) -> Result<(), String> {
|
|
use sqlx::Row;
|
|
let grace_secs = grace.as_secs() as f64;
|
|
let rows = sqlx::query(
|
|
"SELECT id, workspace_id, runtime_container_name
|
|
FROM missions
|
|
WHERE status IN ('completed', 'failed', 'cancelled')
|
|
AND runtime_endpoint IS NOT NULL
|
|
AND completed_at IS NOT NULL
|
|
AND completed_at < now() - make_interval(secs => $1::float)
|
|
LIMIT 20",
|
|
)
|
|
.bind(grace_secs)
|
|
.fetch_all(pool)
|
|
.await
|
|
.map_err(|e| format!("query terminal missions: {e}"))?;
|
|
if rows.is_empty() {
|
|
return Ok(());
|
|
}
|
|
let Some(prov) = MissionRuntimeProvisioner::from_env() else {
|
|
return Ok(());
|
|
};
|
|
for row in rows {
|
|
let id: Uuid = row.get("id");
|
|
let workspace_id: Uuid = row.get("workspace_id");
|
|
// Last chance. `teardown_container` deletes the checkout, so anything
|
|
// not captured by now is gone for good. The phase sweep should have
|
|
// handled this minutes ago; this covers the cases it cannot — a phase
|
|
// that ended `failed` rather than `completed`, or a capture that kept
|
|
// erroring until the grace window ran out.
|
|
if let Err(e) = capture_outstanding_phases(pool, id).await {
|
|
eprintln!("mission_runtime::sweeper: last-chance capture for {id}: {e}");
|
|
}
|
|
// Clear the binding only once the container is actually GONE, which is
|
|
// not the same as "teardown returned Ok".
|
|
//
|
|
// This used to clear unconditionally, reasoning that a not-found is
|
|
// expected and retrying forever is worse. But `teardown_container`
|
|
// already maps 404/"No such container" to `Ok`, so an `Err` here means
|
|
// a real docker failure — and this sweep selects on
|
|
// `runtime_endpoint IS NOT NULL`, so clearing after a failed teardown
|
|
// hides the surviving container from the only thing that would ever
|
|
// retry it. One transient docker error would strand a running
|
|
// container until somebody deleted the mission by hand.
|
|
//
|
|
// Asking docker whether the container still exists settles it without
|
|
// reintroducing the infinite retry: if it is gone, drop the binding
|
|
// however the call reported itself; if it survives, keep the binding so
|
|
// the next sweep tries again and the operator sees a recurring log
|
|
// rather than silence.
|
|
let teardown = prov.teardown_container(id).await;
|
|
if let Err(e) = &teardown {
|
|
if prov.container_exists(id).await {
|
|
eprintln!(
|
|
"mission_runtime::sweeper: teardown mission {id}: {e} — container still \
|
|
present, keeping the runtime binding so the next sweep retries"
|
|
);
|
|
continue;
|
|
}
|
|
eprintln!(
|
|
"mission_runtime::sweeper: teardown mission {id}: {e} — container is gone \
|
|
anyway, clearing the binding"
|
|
);
|
|
}
|
|
if let Err(e) =
|
|
cm_db::repo::missions::set_runtime_binding(pool, id, workspace_id, None, None, None)
|
|
.await
|
|
{
|
|
eprintln!("mission_runtime::sweeper: clear binding for {id}: {e}");
|
|
}
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
/// Capture any phase of `mission_id` that has a repo and no `code_diff` yet,
|
|
/// regardless of how the phase ended.
|
|
///
|
|
/// The phase sweep only captures `completed` phases. A mission that failed
|
|
/// mid-coding still has real work in its checkout, and deleting it
|
|
/// unexamined is how a debugging session loses the only evidence of what the
|
|
/// agents actually did.
|
|
async fn capture_outstanding_phases(pool: &sqlx::PgPool, mission_id: Uuid) -> Result<(), String> {
|
|
use sqlx::Row;
|
|
let rows = sqlx::query(
|
|
"SELECT mp.id
|
|
FROM mission_phases mp
|
|
JOIN missions m ON m.id = mp.mission_id
|
|
WHERE mp.mission_id = $1
|
|
AND m.repo_id IS NOT NULL
|
|
AND NOT EXISTS (
|
|
SELECT 1 FROM mission_artifacts a
|
|
WHERE a.mission_id = mp.mission_id
|
|
AND a.phase_id = mp.id
|
|
AND a.kind = 'code_diff'
|
|
)",
|
|
)
|
|
.bind(mission_id)
|
|
.fetch_all(pool)
|
|
.await
|
|
.map_err(|e| format!("select uncaptured phases: {e}"))?;
|
|
|
|
for row in rows {
|
|
let phase_id: Uuid = row.get("id");
|
|
if let Err(e) =
|
|
crate::mission_delivery::capture_phase_diff(pool, mission_id, phase_id).await
|
|
{
|
|
eprintln!(
|
|
"mission_runtime::sweeper: capture mission {mission_id} phase {phase_id}: {e}"
|
|
);
|
|
}
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
/// A canary backend runs the production path, so it needs the production
|
|
/// credential — and an UNKNOWN backend must still be refused.
|
|
///
|
|
/// The second half is the one that matters: `backend_can_run_a_mission` is
|
|
/// what stops a mission being launched against a rootfs no node has, and
|
|
/// widening the credential map is exactly how that guard gets softened by
|
|
/// accident.
|
|
#[test]
|
|
fn a_canary_backend_is_credentialed_but_an_unknown_one_is_not() {
|
|
assert!(backend_can_run_a_mission("canary-claude"));
|
|
assert!(backend_can_run_a_mission("claude"));
|
|
for unknown in ["test226", "definitely-not-built", "canary", "canary-"] {
|
|
assert!(
|
|
!backend_can_run_a_mission(unknown),
|
|
"{unknown} must be refused at launch"
|
|
);
|
|
}
|
|
}
|
|
|
|
use super::*;
|
|
|
|
/// The regression guard for the whole subscription feature.
|
|
///
|
|
/// Claude Code ranks `ANTHROPIC_API_KEY` above the subscription's OAuth
|
|
/// credential, so forwarding it into a container whose runtime is logged in
|
|
/// means every mission silently bills the API while looking correct. There
|
|
/// is no error to observe — only the invoice. If this test ever goes red,
|
|
/// the subscription path is off even though nothing appears broken.
|
|
/// The microVM path is subscription-only BY CONSTRUCTION — it does not read
|
|
/// `CLAWMATES_RUNTIME_AUTH` at all. If it did, the single unset variable on
|
|
/// gw-04 today would put an API key inside every VM, and Claude Code ranks
|
|
/// the key above the subscription token: it would work, and bill per-token
|
|
/// A local backend carries no secret, and no secret reaches it.
|
|
///
|
|
/// The credential map is the one place a backend can acquire a token, and
|
|
/// the point of routing `local-ornith` through it — rather than special-
|
|
/// casing it earlier — is that the "unknown backend cannot launch" rule
|
|
/// still holds. A typo like `local-ornit` must fail closed, not inherit the
|
|
/// subscription token on its way to somebody else's endpoint.
|
|
#[test]
|
|
fn a_local_backend_gets_a_placeholder_and_never_a_real_credential() {
|
|
let env = microvm_provider_env_from(Some("local-ornith"), |_| None)
|
|
.expect("a local backend needs nothing from the operator");
|
|
let token = env
|
|
.iter()
|
|
.find(|(k, _)| k == "ANTHROPIC_AUTH_TOKEN")
|
|
.map(|(_, v)| v.clone())
|
|
.expect("Claude Code refuses to start without a bearer");
|
|
assert!(
|
|
!token.starts_with("sk-"),
|
|
"a local backend must never be handed a real credential, got {token:?}"
|
|
);
|
|
assert!(!env.iter().any(|(k, _)| k == "ANTHROPIC_API_KEY"));
|
|
assert!(!env.iter().any(|(k, _)| k == "CLAUDE_CODE_OAUTH_TOKEN"));
|
|
|
|
// And a near-miss still cannot launch.
|
|
for typo in ["local-ornit", "ornith", "local", "local-ornith2"] {
|
|
assert!(
|
|
microvm_credential_for(Some(typo)).is_err(),
|
|
"{typo} must be refused, not resolved to something"
|
|
);
|
|
}
|
|
}
|
|
|
|
/// against a plan already paid for, with no symptom but the invoice.
|
|
#[test]
|
|
fn a_microvm_never_receives_an_anthropic_api_key() {
|
|
// Everything set, including the key the container path would forward.
|
|
let env = microvm_provider_env_from(Some("claude"), |k| Some(format!("value-of-{k}")))
|
|
.expect("a set token should launch");
|
|
let keys: Vec<&str> = env.iter().map(|(k, _)| k.as_str()).collect();
|
|
assert!(
|
|
!keys.contains(&"ANTHROPIC_API_KEY"),
|
|
"the API key outranks the subscription token; forwarding it bills the \
|
|
API silently. Forwarded: {keys:?}"
|
|
);
|
|
assert!(
|
|
keys.contains(&"CLAUDE_CODE_OAUTH_TOKEN"),
|
|
"the subscription credential must travel: {keys:?}"
|
|
);
|
|
}
|
|
|
|
/// No token is a refusal to launch, not an empty environment. A VM without a
|
|
/// credential does not error — `claude -p` hangs, which reads as a phase
|
|
/// stuck at `running` with nothing in the logs.
|
|
#[test]
|
|
fn a_microvm_without_a_subscription_token_refuses_to_launch() {
|
|
let r = microvm_provider_env_from(Some("claude"), |k| {
|
|
// The API key is present; the subscription token is not. This must
|
|
// NOT be treated as "we have a credential".
|
|
(k == "ANTHROPIC_API_KEY").then(|| "sk-ant-whatever".to_string())
|
|
});
|
|
let err = r.expect_err("no subscription token must refuse the launch");
|
|
assert!(err.contains("CLAUDE_CODE_OAUTH_TOKEN"), "{err}");
|
|
// A blank token is the same as no token.
|
|
assert!(microvm_provider_env_from(Some("claude"), |k| {
|
|
(k == "CLAUDE_CODE_OAUTH_TOKEN").then(|| " ".to_string())
|
|
})
|
|
.is_err());
|
|
}
|
|
|
|
/// The credential each provider gets, and — the part that matters — the one
|
|
/// it must never get. A GLM VM handed `CLAUDE_CODE_OAUTH_TOKEN` would send an
|
|
/// Anthropic subscription credential to z.ai, verbatim, on the first turn.
|
|
/// The two providers' secrets must not cross.
|
|
#[test]
|
|
fn each_provider_gets_its_own_credential_and_only_its_own() {
|
|
let env = microvm_provider_env_from(Some("glm"), |k| Some(format!("value-of-{k}")))
|
|
.expect("glm has a settled contract");
|
|
let by_key: std::collections::HashMap<_, _> = env.into_iter().collect();
|
|
// Claude Code reads a custom-endpoint credential as ANTHROPIC_AUTH_TOKEN,
|
|
// and the value is the SERVER's ZAI_API_KEY — two different names, which
|
|
// is exactly why the contract carries both.
|
|
assert_eq!(
|
|
by_key.get("ANTHROPIC_AUTH_TOKEN").map(String::as_str),
|
|
Some("value-of-ZAI_API_KEY"),
|
|
"{by_key:?}"
|
|
);
|
|
for forbidden in ["CLAUDE_CODE_OAUTH_TOKEN", "ANTHROPIC_API_KEY"] {
|
|
assert!(
|
|
!by_key.contains_key(forbidden),
|
|
"a GLM VM must never carry {forbidden} — it would be sent to z.ai: {by_key:?}"
|
|
);
|
|
}
|
|
|
|
// Kimi shares the target var with GLM and NOTHING else: same protocol,
|
|
// different endpoint baked into a different image. What must not happen
|
|
// is one provider's key travelling under the other's name.
|
|
let kimi = microvm_provider_env_from(Some("kimi"), |k| Some(format!("value-of-{k}")))
|
|
.expect("kimi has a settled contract");
|
|
let kimi: std::collections::HashMap<_, _> = kimi.into_iter().collect();
|
|
assert_eq!(
|
|
kimi.get("ANTHROPIC_AUTH_TOKEN").map(String::as_str),
|
|
Some("value-of-KIMI_API_KEY"),
|
|
"{kimi:?}"
|
|
);
|
|
assert!(!kimi.contains_key("CLAUDE_CODE_OAUTH_TOKEN"), "{kimi:?}");
|
|
|
|
// And the reverse: a claude VM carries no z.ai key.
|
|
let env = microvm_provider_env_from(Some("claude"), |k| Some(format!("value-of-{k}")))
|
|
.expect("claude");
|
|
let keys: Vec<&str> = env.iter().map(|(k, _)| k.as_str()).collect();
|
|
assert!(!keys.contains(&"ANTHROPIC_AUTH_TOKEN"), "{keys:?}");
|
|
assert!(!keys.contains(&"ZAI_API_KEY"), "{keys:?}");
|
|
}
|
|
|
|
/// A GLM mission with no z.ai key refuses to launch, and says which var is
|
|
/// missing. It must NOT fall back to the Anthropic token that IS set.
|
|
#[test]
|
|
fn a_glm_mission_without_a_zai_key_refuses_rather_than_falls_back() {
|
|
let err = microvm_provider_env_from(Some("glm"), |k| {
|
|
(k == "CLAUDE_CODE_OAUTH_TOKEN").then(|| "sub-token".to_string())
|
|
})
|
|
.expect_err("no z.ai key must refuse the launch");
|
|
assert!(err.contains("ZAI_API_KEY"), "{err}");
|
|
}
|
|
|
|
/// A backend whose credential contract is not settled is refused, rather
|
|
/// than handed another provider's credential to send at its endpoint.
|
|
#[test]
|
|
fn an_undefined_backend_is_refused_rather_than_given_the_anthropic_token() {
|
|
// `kimi` has since moved out of this list — its URL was measured.
|
|
for b in [Some("something-new"), Some("agent-terminal")] {
|
|
let r = microvm_provider_env_from(b, |_| Some("set".into()));
|
|
assert!(r.is_err(), "backend {b:?} should be refused: {r:?}");
|
|
}
|
|
// The default image is the claude one, and does have a contract.
|
|
for b in [None, Some(""), Some("default"), Some("claude")] {
|
|
assert!(
|
|
microvm_provider_env_from(b, |_| Some("set".into())).is_ok(),
|
|
"backend {b:?}"
|
|
);
|
|
}
|
|
}
|
|
|
|
/// The container path still honours the operator's mode, and reads values
|
|
/// the same way. Only the microVM path is pinned.
|
|
#[test]
|
|
fn the_container_path_forwards_its_configured_mode() {
|
|
for auth in [RuntimeAuth::ApiKey, RuntimeAuth::Subscription] {
|
|
let keys = forwarded_provider_keys(auth);
|
|
// Every key set in the environment, and nothing else.
|
|
let pairs = provider_env_from(auth, |k| Some(format!("value-of-{k}")));
|
|
let got: Vec<&str> = pairs.iter().map(|(k, _)| k.as_str()).collect();
|
|
assert_eq!(got, keys, "{}", auth.as_str());
|
|
for (k, v) in &pairs {
|
|
assert_eq!(v, &format!("value-of-{k}"), "value must pass through");
|
|
}
|
|
}
|
|
}
|
|
|
|
/// An unset or blank credential is ABSENT, not empty. On the microVM path an
|
|
/// empty string would make `claude` believe it has a credential and fail
|
|
/// authentication, instead of reporting that it has none.
|
|
#[test]
|
|
fn a_blank_credential_is_omitted_rather_than_forwarded_empty() {
|
|
let pairs = provider_env_from(RuntimeAuth::Subscription, |k| match k {
|
|
"CLAUDE_CODE_OAUTH_TOKEN" => Some(" ".into()),
|
|
"OPENAI_API_KEY" => Some(String::new()),
|
|
"GROQ_API_KEY" => Some("real".into()),
|
|
_ => None,
|
|
});
|
|
assert_eq!(
|
|
pairs,
|
|
vec![("GROQ_API_KEY".to_string(), "real".to_string())]
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn subscription_mode_withholds_the_anthropic_api_key() {
|
|
let keys = forwarded_provider_keys(RuntimeAuth::Subscription);
|
|
assert!(
|
|
!keys.contains(&"ANTHROPIC_API_KEY"),
|
|
"ANTHROPIC_API_KEY outranks the subscription credential; forwarding \
|
|
it silently bills the API. Forwarded: {keys:?}"
|
|
);
|
|
// Unrelated providers have no subscription equivalent and must survive.
|
|
for k in [
|
|
"GROQ_API_KEY",
|
|
"OPENAI_API_KEY",
|
|
"ZAI_API_KEY",
|
|
"KIMI_API_KEY",
|
|
] {
|
|
assert!(keys.contains(&k), "{k} should still be forwarded");
|
|
}
|
|
// And the subscription credential MUST travel. A mission container
|
|
// has its own data dir, so unlike the shared runtime it has no
|
|
// persisted `claude /login` to fall back on. Without this the CLI
|
|
// has no credential and simply hangs — a phase stuck at `running`
|
|
// with nothing in the logs, which is exactly how this was found.
|
|
assert!(
|
|
keys.contains(&"CLAUDE_CODE_OAUTH_TOKEN"),
|
|
"subscription mode must forward the token; without it `claude -p` \
|
|
hangs with no credential. Forwarded: {keys:?}"
|
|
);
|
|
}
|
|
|
|
/// The two credentials must never travel together: Claude Code would pick
|
|
/// the API key and bill it while the deployment believes it is on the
|
|
/// subscription.
|
|
#[test]
|
|
fn the_two_anthropic_credentials_are_mutually_exclusive() {
|
|
for mode in [RuntimeAuth::ApiKey, RuntimeAuth::Subscription] {
|
|
let keys = forwarded_provider_keys(mode);
|
|
let both =
|
|
keys.contains(&"ANTHROPIC_API_KEY") && keys.contains(&"CLAUDE_CODE_OAUTH_TOKEN");
|
|
assert!(!both, "{mode:?} forwards both credentials: {keys:?}");
|
|
}
|
|
}
|
|
|
|
/// Default behaviour is unchanged, so a deployment that never opts in keeps
|
|
/// working exactly as before.
|
|
#[test]
|
|
fn api_key_mode_forwards_everything() {
|
|
let keys = forwarded_provider_keys(RuntimeAuth::ApiKey);
|
|
for k in ["ANTHROPIC_API_KEY", "GROQ_API_KEY", "OPENAI_API_KEY"] {
|
|
assert!(keys.contains(&k), "{k} should be forwarded in api_key mode");
|
|
}
|
|
// Gemini is stripped from the platform entirely: no provider row, no
|
|
// selector entry, and no key forwarded to agent containers. Asserted so
|
|
// a future "just add it back to the list" restores the dependency
|
|
// visibly rather than by accident.
|
|
for mode in [RuntimeAuth::ApiKey, RuntimeAuth::Subscription] {
|
|
assert!(
|
|
!forwarded_provider_keys(mode).contains(&"GEMINI_API_KEY"),
|
|
"GEMINI_API_KEY must not be forwarded in {mode:?}"
|
|
);
|
|
}
|
|
assert!(
|
|
!keys.contains(&"CLAUDE_CODE_OAUTH_TOKEN"),
|
|
"api_key mode must not also ship the subscription token"
|
|
);
|
|
}
|
|
|
|
/// An unset or misspelled value must fall back to the *existing* behaviour.
|
|
/// Defaulting to subscription on a typo would leave missions with no
|
|
/// credential at all.
|
|
#[test]
|
|
fn auth_mode_defaults_to_api_key() {
|
|
// Can't safely mutate process env in a parallel test binary, so assert
|
|
// the mapping the parser implements rather than the env read itself.
|
|
for (input, expected) in [
|
|
("subscription", RuntimeAuth::Subscription),
|
|
("SUBSCRIPTION", RuntimeAuth::Subscription),
|
|
("api_key", RuntimeAuth::ApiKey),
|
|
("", RuntimeAuth::ApiKey),
|
|
("nonsense", RuntimeAuth::ApiKey),
|
|
] {
|
|
let got = match input.trim().to_ascii_lowercase().as_str() {
|
|
"subscription" => RuntimeAuth::Subscription,
|
|
"" | "api_key" => RuntimeAuth::ApiKey,
|
|
_ => RuntimeAuth::ApiKey,
|
|
};
|
|
assert_eq!(got, expected, "input {input:?}");
|
|
}
|
|
}
|
|
|
|
/// The name→id round trip the orphan sweep depends on.
|
|
///
|
|
/// If this is wrong the sweep either skips every orphan (harmless) or
|
|
/// resolves a container to the WRONG mission id and asks the database
|
|
/// about a mission that does exist — reading "still live, leave it" for a
|
|
/// container that is not. Cheap to get right, expensive to get wrong.
|
|
#[test]
|
|
fn a_container_name_round_trips_to_its_mission() {
|
|
let id = Uuid::now_v7();
|
|
assert_eq!(mission_id_from_container(&container_name(id)), Some(id));
|
|
// Not ours, and not a panic.
|
|
assert_eq!(mission_id_from_container("clawmates_server_1"), None);
|
|
assert_eq!(mission_id_from_container("cm-runtime-mission-nonsense"), None);
|
|
assert_eq!(mission_id_from_container("cm-sandbox-abc"), None);
|
|
}
|
|
|
|
/// A container docker will not date must not be reaped.
|
|
#[test]
|
|
fn an_undatable_container_has_no_age() {
|
|
assert_eq!(MissionRuntimeProvisioner::container_age(None), None);
|
|
let now = std::time::SystemTime::now()
|
|
.duration_since(std::time::UNIX_EPOCH)
|
|
.unwrap()
|
|
.as_secs() as i64;
|
|
let age = MissionRuntimeProvisioner::container_age(Some(now - 3600)).expect("age");
|
|
assert!(age.as_secs() >= 3500 && age.as_secs() <= 3700, "{age:?}");
|
|
// A clock skew that puts creation in the future must not underflow into
|
|
// a colossal age that reads as "long past the grace period".
|
|
assert_eq!(MissionRuntimeProvisioner::container_age(Some(now + 600)), None);
|
|
}
|
|
|
|
/// The grace period is long, and that is the point.
|
|
#[test]
|
|
fn the_orphan_grace_is_generous() {
|
|
assert!(
|
|
ORPHAN_GRACE >= std::time::Duration::from_secs(12 * 3600),
|
|
"the row-driven sweep already handles everything the platform knows \
|
|
about, so anything reaching the orphan path is unexpected — and the \
|
|
one real orphan held ten unpushed commits"
|
|
);
|
|
}
|
|
|
|
const SAMPLE_CONFIG: &str = r#"# top comment
|
|
[agents.claw_a]
|
|
model_provider = "anthropic.default"
|
|
risk_profile = "coding_readwrite"
|
|
mcp_bundles = ["clawmates_door"]
|
|
|
|
[agents.claw_a.workspace]
|
|
unrestricted_filesystem = false
|
|
|
|
[agents.claw_b]
|
|
risk_profile = "research_readonly"
|
|
|
|
[risk_profiles.coding_readwrite]
|
|
# keep this comment
|
|
allowed_tools = ["file_read", "file_edit"]
|
|
"#;
|
|
|
|
#[test]
|
|
fn stamp_pins_path_and_preserves_existing_workspace_fields() {
|
|
let (out, n) =
|
|
stamp_workspace_paths(SAMPLE_CONFIG, &["claw_a".to_string()], "/mission/repo").unwrap();
|
|
assert_eq!(n, 1);
|
|
assert!(out.contains(r#"path = "/mission/repo""#));
|
|
// The sibling field in the same table is untouched.
|
|
assert!(out.contains("unrestricted_filesystem = false"));
|
|
// Comments and unrelated sections survive the round-trip.
|
|
assert!(out.contains("# top comment"));
|
|
assert!(out.contains("# keep this comment"));
|
|
assert!(out.contains("[risk_profiles.coding_readwrite]"));
|
|
}
|
|
|
|
#[test]
|
|
fn stamp_creates_workspace_table_when_absent() {
|
|
let (out, n) =
|
|
stamp_workspace_paths(SAMPLE_CONFIG, &["claw_b".to_string()], "/mission/repo").unwrap();
|
|
assert_eq!(n, 1);
|
|
// claw_b had no [workspace] table; it now has one with the path.
|
|
let doc = out.parse::<toml_edit::DocumentMut>().unwrap();
|
|
assert_eq!(
|
|
doc["agents"]["claw_b"]["workspace"]["path"].as_str(),
|
|
Some("/mission/repo")
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn stamp_skips_absent_aliases_without_fabricating_them() {
|
|
let (out, n) = stamp_workspace_paths(
|
|
SAMPLE_CONFIG,
|
|
&["claw_missing".to_string()],
|
|
"/mission/repo",
|
|
)
|
|
.unwrap();
|
|
assert_eq!(n, 0);
|
|
assert!(!out.contains("claw_missing"));
|
|
}
|
|
|
|
#[test]
|
|
fn stamp_is_idempotent_overwriting_a_prior_path() {
|
|
let once = stamp_workspace_paths(SAMPLE_CONFIG, &["claw_a".to_string()], "/old/path")
|
|
.unwrap()
|
|
.0;
|
|
let (twice, n) =
|
|
stamp_workspace_paths(&once, &["claw_a".to_string()], "/mission/repo").unwrap();
|
|
assert_eq!(n, 1);
|
|
assert!(twice.contains(r#"path = "/mission/repo""#));
|
|
assert!(!twice.contains("/old/path"));
|
|
// Exactly one path key for claw_a (no duplication).
|
|
assert_eq!(twice.matches("path = ").count(), 1);
|
|
}
|
|
|
|
#[test]
|
|
fn container_name_is_stable_and_prefixed() {
|
|
let id = Uuid::parse_str("019f84a0-f2a2-7bd0-be9b-86713ec73693").unwrap();
|
|
let name = container_name(id);
|
|
assert_eq!(name, "cm-runtime-mission-019f84a0f2a27bd0be9b86713ec73693");
|
|
// Determinism: same input → same output.
|
|
assert_eq!(name, container_name(id));
|
|
}
|
|
|
|
#[test]
|
|
fn container_names_differ_across_missions() {
|
|
// Two UUIDs differing only in the trailing hex char — full-uuid
|
|
// naming must distinguish them (UUIDv7's timestamp shares
|
|
// leading bytes for missions minted in the same second).
|
|
let a = container_name(Uuid::parse_str("019f84a0-f2a2-7bd0-be9b-86713ec73693").unwrap());
|
|
let b = container_name(Uuid::parse_str("019f84a0-f2a2-7bd0-be9b-86713ec7369f").unwrap());
|
|
assert_ne!(a, b);
|
|
}
|
|
|
|
#[test]
|
|
fn endpoint_url_uses_gateway_port() {
|
|
let url = endpoint_url("cm-runtime-mission-abc");
|
|
assert_eq!(url, "http://cm-runtime-mission-abc:42617");
|
|
}
|
|
|
|
/// The seed data path must be per-mission, not the shared seed dir.
|
|
///
|
|
/// Every mission container used to bind the SAME host directory as
|
|
/// `/zeroclaw-data`. It holds `config.toml`, which carries the §15 door
|
|
/// bearer token, plus `sessions.db`/`devices.db`. Sharing it meant one
|
|
/// mission could read another's credential, and anything written there
|
|
/// was inherited by every later mission — `teardown_container` only
|
|
/// removes `/var/lib/clawmates-missions/{id}`, so the shared dir was
|
|
/// never cleaned.
|
|
///
|
|
/// The seed copy reads as root and leaves the result to 65532.
|
|
///
|
|
/// Both halves are load-bearing and they pull in opposite directions. The
|
|
/// seed dir is root-owned with parts at mode 0600, so a copier running as
|
|
/// 65532 cannot read it — that was tried, `cp` failed on the first
|
|
/// unreadable entry, `set -e` abandoned the rest, and the mission booted
|
|
/// with `.zeroclaw` and nothing else. But leaving the RESULT root-owned is
|
|
/// what put ~3200 undeletable files per mission into the missions tree.
|
|
#[test]
|
|
fn the_seed_copy_hands_its_result_to_the_mission_uid() {
|
|
let script = copy_script();
|
|
assert!(
|
|
script.contains(&format!("chown -R {MISSION_UID} /dst")),
|
|
"the copied seed must end up owned by the mission uid:\n{script}"
|
|
);
|
|
// Every seeded path is attempted, and absence is tolerated — a fresh
|
|
// deployment genuinely has no `.kimi-code` yet.
|
|
for p in SEEDED_PATHS {
|
|
assert!(
|
|
script.contains(&format!("if [ -e '/seed/{p}' ]")),
|
|
"{p} is not guarded by an existence check"
|
|
);
|
|
}
|
|
// The chown must come AFTER the copies, or it chowns an empty dir.
|
|
let last_cp = script.rfind("cp -a").expect("at least one copy");
|
|
let chown = script.find("chown -R").expect("a chown");
|
|
assert!(chown > last_cp, "chown must run after the copies");
|
|
}
|
|
|
|
/// The seed data path must be per-mission, not the shared seed dir.
|
|
///
|
|
/// Every mission container used to bind the SAME host directory as
|
|
/// `/zeroclaw-data`. It holds `config.toml`, which carries the §15 door
|
|
/// bearer token, plus `sessions.db`/`devices.db`. Sharing it meant one
|
|
/// mission could read another's credential, and anything written there
|
|
/// was inherited by every later mission — `teardown_container` only
|
|
/// removes `/var/lib/clawmates-missions/{id}`, so the shared dir was
|
|
/// never cleaned.
|
|
///
|
|
/// Asserting the path shape is what keeps this from silently regressing:
|
|
/// a future edit that points the mount back at the seed dir restores the
|
|
/// credential sharing with no other visible symptom.
|
|
///
|
|
/// This test did not RUN for some time: an edit stranded its `#[test]`
|
|
/// above the neighbouring function, leaving two attributes there and none
|
|
/// here. rustc said so twice — "duplicated attribute" and "function is
|
|
/// never used" — and both read as ordinary warnings in a noisy build.
|
|
#[test]
|
|
fn runtime_data_is_scoped_to_one_mission() {
|
|
let a = Uuid::now_v7();
|
|
let b = Uuid::now_v7();
|
|
let root = missions_host_root();
|
|
let path = |id: Uuid| format!("{root}/{id}/runtime-data");
|
|
|
|
assert_ne!(path(a), path(b), "two missions must not share runtime data");
|
|
assert!(
|
|
path(a).starts_with(&format!("{root}/{a}")),
|
|
"runtime data must live under the mission dir so teardown removes it"
|
|
);
|
|
assert_ne!(
|
|
path(a),
|
|
DEFAULT_SEED_DIR,
|
|
"the mount must never be the shared seed dir itself"
|
|
);
|
|
assert!(
|
|
!path(a).starts_with(DEFAULT_SEED_DIR),
|
|
"runtime data must not live inside the shared seed dir either"
|
|
);
|
|
}
|
|
|
|
/// The copy must be an allow-list, and must include the secret-bearing
|
|
/// paths while excluding the expensive ones.
|
|
///
|
|
/// Measured on gw-04: the seed dir is 1.7 GB, of which 1.5 GB is a
|
|
/// vestigial `.rustup` that is not even on the container's PATH (the
|
|
/// image ships Rust at /usr/local/cargo). Copying everything per mission
|
|
/// would cost tens of seconds and ~17 GB across ten concurrent missions —
|
|
/// which is what the first version of this did.
|
|
#[test]
|
|
fn the_seed_copy_takes_secrets_and_skips_caches() {
|
|
// The two paths that carry the door bearer token MUST be copied, or
|
|
// this whole change accomplishes nothing.
|
|
assert!(SEEDED_PATHS.contains(&".zeroclaw"));
|
|
assert!(SEEDED_PATHS.contains(&"clawmates-mcp.json"));
|
|
// Claude Code's credentials and state.
|
|
assert!(SEEDED_PATHS.contains(&".claude"));
|
|
|
|
// The expensive, secret-free ones must NOT be.
|
|
for cache in [".rustup", ".npm", ".cargo", ".cache"] {
|
|
assert!(
|
|
!SEEDED_PATHS.contains(&cache),
|
|
"{cache} is a cache and must not be copied per mission"
|
|
);
|
|
}
|
|
|
|
let script = copy_script();
|
|
// A missing entry is normal on a fresh deployment (no .kimi-code
|
|
// until Kimi is first used) and must not fail the copy.
|
|
assert!(
|
|
script.contains("if [ -e "),
|
|
"absent paths must be tolerated"
|
|
);
|
|
assert!(script.contains("/seed/.zeroclaw"));
|
|
assert!(!script.contains("/seed/.rustup"));
|
|
for p in SEEDED_PATHS {
|
|
assert!(script.contains(p), "{p} missing from the copy script");
|
|
}
|
|
}
|
|
}
|