Omar SobhandClaude Opus 5
79a6119f4e
ActGov's second layer (arXiv 2609.24446), in the honest form our evidence supports. The paper binds each task to its minimum tools; 171 recorded tool calls cannot justify a per-task minimum, but they do justify the line this draws: files, commands, search, web, delegation and skills are the work surface and pass; ListAgents, ScheduleWakeup, CronCreate, SendMessage and the rest reach the platform itself and do not. That line is not theoretical. ListAgents and ScheduleWakeup were both called by microVM missions whose --allowedTools is Read Edit Write Bash Agent. Neither is on that list; both ran, because the flag governs permission prompting and not availability. Our gate is the only place this can be enforced. TaskPolicy is rendered into the same guest script as the floor and the role policies. A phase names its own set with "agent_tools" — NOT "tools", which security_scan already owns for its scanner list; both are now in phase_config::KNOWN_KEYS, adjacent, each saying what the other is. SHADOW BY DEFAULT. The gate records what it would have refused to would-deny.jsonl and allows the call; the host drains it into gate.would_deny on both tiers. CLAWMATES_TASK_PERMISSION=enforce flips it. A policy tightened on a guess and enforced on day one is how an agent learns to work around the gate, and a shadow mode nobody can read is an off switch with extra steps. The VM probe needed a sentinel: a refusal and a call that merely would have been refused are both JSON objects with the same keys, and telling them apart by content would confuse the one distinction shadow mode exists to make. Asymmetry, stated rather than hidden: a VM is per-phase and honours the phase's own agent_tools; a container serves every phase of its mission and gets the mission-wide default. Narrowing per phase there needs a re-install between phases and is not done. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
2117 lines
94 KiB
Rust
2117 lines
94 KiB
Rust
//! Per-mission ZeroClaw runtime container lifecycle.
|
|
//!
|
|
//! C3 workspace-isolation model: every mission gets its own ZeroClaw
|
|
//! daemon container, so agents' sandboxed filesystem is scoped to that
|
|
//! mission's repo checkout instead of the shared `/zeroclaw-data/
|
|
//! workspace` on the singleton `clawmates-runtime` daemon.
|
|
//!
|
|
//! Container naming: `cm-runtime-mission-{first 12 chars of mission uuid}`.
|
|
//! Endpoint: `http://<container_name>:42617` (well-known ZeroClaw port,
|
|
//! reachable over the `clawmates_core` docker network).
|
|
//!
|
|
//! Lifecycle:
|
|
//! - `ensure_container(mission_id)` — idempotent; spawns the container
|
|
//! if not present, returns its endpoint. Called from
|
|
//! `mission_orchestrator::on_launch` and (as a fallback for
|
|
//! pre-C3 missions) `phase_runner::launch_phase`.
|
|
//! - `teardown_container(mission_id)` — force-removes the container.
|
|
//! Called by the sweeper (slice 3) N minutes after a mission
|
|
//! reaches a terminal state, so the operator has a window to
|
|
//! re-open the ClawmateS UI and pull the last checkpoint before
|
|
//! the daemon disappears.
|
|
//!
|
|
//! State: `missions.runtime_container_name` and `missions.runtime_endpoint`
|
|
//! carry the current binding (both null when torn down or never
|
|
//! provisioned).
|
|
|
|
use bollard::exec::{CreateExecOptions, StartExecResults};
|
|
use bollard::models::{
|
|
ContainerCreateBody, EndpointSettings, HostConfig, Mount, MountTypeEnum, NetworkConnectRequest,
|
|
};
|
|
use bollard::query_parameters::{
|
|
CreateContainerOptions, InspectContainerOptions, RemoveContainerOptions, StartContainerOptions,
|
|
};
|
|
use bollard::Docker;
|
|
use futures::StreamExt;
|
|
use std::collections::HashMap;
|
|
use uuid::Uuid;
|
|
|
|
/// Docker image the per-mission runtime uses, overridable per-deploy via
|
|
/// `CLAWMATES_RUNTIME_IMAGE` (for a canary; the default is the shipped one).
|
|
///
|
|
/// A MOVING tag, and that is the hazard: it names whatever the host last
|
|
/// tagged. gw-04's `sync` was two zeroclaw releases behind the canary every
|
|
/// mission was actually running, and there was no way to see it — a moving tag
|
|
/// that points somewhere old resolves perfectly, starts perfectly, and runs old
|
|
/// code. The registry copy of the same tag was a DIFFERENT image again, without
|
|
/// the Rust toolchain, so a host that pulled rather than retagged would have
|
|
/// silently lost the on-green test gate.
|
|
///
|
|
/// Two things make that visible now: this module logs the image it resolved and
|
|
/// `runtime_preflight` reports the versions inside the container, not just that
|
|
/// the binaries exist.
|
|
const DEFAULT_IMAGE: &str = "clawmates-runtime:sync";
|
|
|
|
/// Well-known ZeroClaw gateway port.
|
|
const GATEWAY_PORT: u16 = 42617;
|
|
|
|
/// How the ZeroClaw runtime authenticates to Anthropic.
|
|
///
|
|
/// The runtime image ships the official `claude` CLI, which can authenticate
|
|
/// either with a platform API key or with a subscription login stored under
|
|
/// `$HOME` (a persisted bind mount, so one login survives container
|
|
/// recreation). These are mutually exclusive in practice because Claude Code
|
|
/// prefers `ANTHROPIC_API_KEY` over the subscription credential.
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
|
pub enum RuntimeAuth {
|
|
/// Forward the platform's `ANTHROPIC_API_KEY`. Metered per token.
|
|
ApiKey,
|
|
/// Withhold the API key so the runtime's own `claude /login` credential is
|
|
/// used. Only valid for a single-operator deployment — a subscription
|
|
/// credential must never serve another person's work.
|
|
Subscription,
|
|
}
|
|
|
|
impl RuntimeAuth {
|
|
pub fn as_str(self) -> &'static str {
|
|
match self {
|
|
RuntimeAuth::ApiKey => "api_key",
|
|
RuntimeAuth::Subscription => "subscription",
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Provider credential env vars forwarded into a runtime container.
|
|
///
|
|
/// `ANTHROPIC_API_KEY` is conditional, and the reason is subtle enough to be
|
|
/// worth stating at the definition: Claude Code resolves credentials in a fixed
|
|
/// priority order and ranks `ANTHROPIC_API_KEY` **above** the subscription's
|
|
/// `CLAUDE_CODE_OAUTH_TOKEN`. On a runtime authenticated via `claude /login`,
|
|
/// forwarding the key silently wins — `claude` still works, agents still run,
|
|
/// and every mission bills the API while appearing to use the subscription.
|
|
/// There is no error to surface; the only symptom is the invoice.
|
|
///
|
|
/// The other three are unrelated providers with no subscription equivalent, so
|
|
/// they forward in both modes.
|
|
///
|
|
/// In subscription mode `CLAUDE_CODE_OAUTH_TOKEN` forwards instead. The
|
|
/// original design assumed a persisted `claude /login` under a bind-mounted
|
|
/// `$HOME`, but a *mission* container gets its own data dir and therefore no
|
|
/// login — so the token has to travel. Missing it is not a loud failure:
|
|
/// `claude -p` simply hangs with no credential, which is what a phase stuck
|
|
/// at `running` for ten minutes looked like when this was first switched on.
|
|
pub fn forwarded_provider_keys(auth: RuntimeAuth) -> Vec<&'static str> {
|
|
// ZAI/KIMI reach their backends through the SAME `claude` binary via
|
|
// ANTHROPIC_BASE_URL, so a mission that selects one needs its key present
|
|
// in the container. They are unrelated to the Anthropic credential and
|
|
// forward in both auth modes.
|
|
let mut keys = vec![
|
|
"GROQ_API_KEY",
|
|
"OPENAI_API_KEY",
|
|
"ZAI_API_KEY",
|
|
"KIMI_API_KEY",
|
|
];
|
|
match auth {
|
|
RuntimeAuth::ApiKey => keys.push("ANTHROPIC_API_KEY"),
|
|
RuntimeAuth::Subscription => keys.push("CLAUDE_CODE_OAUTH_TOKEN"),
|
|
}
|
|
keys
|
|
}
|
|
|
|
/// The forwarded credentials that are actually SET on this server, as pairs.
|
|
///
|
|
/// `forwarded_provider_keys` above stays the single list of *what* forwards; this
|
|
/// is the single reader of the environment, so the container and microVM paths
|
|
/// cannot disagree about how a value is read (blank handling in particular).
|
|
///
|
|
/// A key that is unset is simply absent: an empty-string value would make
|
|
/// `claude` believe it has a credential and fail authentication instead of
|
|
/// reporting that it has none.
|
|
pub fn forwarded_provider_env(auth: RuntimeAuth) -> Vec<(String, String)> {
|
|
provider_env_from(auth, |k| std::env::var(k).ok())
|
|
}
|
|
|
|
/// Credentials for a microVM mission. **Subscription only, by construction.**
|
|
///
|
|
/// Deliberately NOT parameterised by [`runtime_auth_mode`], and that is the
|
|
/// whole point. The container path honours the operator's mode, and gw-04 has
|
|
/// `CLAWMATES_RUNTIME_AUTH` unset today, so it forwards `ANTHROPIC_API_KEY`. A
|
|
/// microVM must not: Claude Code ranks the API key **above** the subscription's
|
|
/// OAuth token, so a VM that received both would bill per-token against a plan
|
|
/// we already pay for, silently — `claude` still works, agents still run, and
|
|
/// the only symptom is the invoice. Taking the mode as an argument would mean a
|
|
/// single unset env var on a new host turns that back on.
|
|
///
|
|
/// A missing subscription token is an **error**, not an empty list. A VM launched
|
|
/// without a credential does not fail: `claude -p` hangs, which is a phase stuck
|
|
/// at `running` with nothing in the logs.
|
|
pub fn microvm_provider_env(backend: Option<&str>) -> Result<Vec<(String, String)>, String> {
|
|
microvm_provider_env_from(backend, |k| std::env::var(k).ok())
|
|
}
|
|
|
|
/// Can a mission agent authenticate in a VM booted from this image?
|
|
///
|
|
/// The honest answer to "which backends can run a mission", which is NOT the
|
|
/// same as "which rootfs images exist on a node". A fleet node reports every
|
|
/// image it has built — `agent-terminal` among them — and booting a mission into
|
|
/// one with no credential contract fails at the agent turn, after the mission
|
|
/// was created and its roster approved.
|
|
pub fn backend_can_run_a_mission(backend: &str) -> bool {
|
|
microvm_credential_for(Some(backend)).is_ok()
|
|
}
|
|
|
|
/// Where a backend's secret comes from, and what the guest's CLI reads it as.
|
|
///
|
|
/// Two names, not one, because they differ for every provider but Anthropic:
|
|
/// z.ai's key lives in the server's `ZAI_API_KEY` and Claude Code reads it as
|
|
/// `ANTHROPIC_AUTH_TOKEN`. Collapsing them into one string is what forces a
|
|
/// guess at the other end, and a wrong guess here sends one provider's
|
|
/// credential to another provider's endpoint.
|
|
struct Credential {
|
|
/// The env var on THIS SERVER holding the secret.
|
|
source: &'static str,
|
|
/// The env var the GUEST's CLI reads it from.
|
|
target: &'static str,
|
|
}
|
|
|
|
/// The credential a backend's CLI authenticates with inside a VM.
|
|
///
|
|
/// Unknown backends are refused rather than given the Anthropic token: sending a
|
|
/// subscription credential to whatever endpoint an unrecognised backend points
|
|
/// at is worse than not launching.
|
|
fn microvm_credential_for(backend: Option<&str>) -> Result<Credential, String> {
|
|
match backend {
|
|
// `canary-*` backends are a REAL rootfs built from a candidate CLI
|
|
// version, run through the production path — egress, stop gate,
|
|
// delegation, delivery — before that version is promoted to the image
|
|
// every mission uses. They carry the same Anthropic subscription
|
|
// credential as `claude`, because testing a new CLI against a different
|
|
// provider would not be testing the thing we are about to ship.
|
|
//
|
|
// Named rather than pattern-matched on anything looser: an unrecognised
|
|
// backend must still be refused at launch, which is what
|
|
// `backend_can_run_a_mission` and the harness's negative control assert.
|
|
None | Some("") | Some("default") | Some("claude") | Some("canary-claude") => {
|
|
Ok(Credential {
|
|
source: "CLAUDE_CODE_OAUTH_TOKEN",
|
|
target: "CLAUDE_CODE_OAUTH_TOKEN",
|
|
})
|
|
}
|
|
// GLM. `images/agent-glm` bakes `ANTHROPIC_BASE_URL=https://api.z.ai/
|
|
// api/anthropic` into the rootfs, so the endpoint is a property of the
|
|
// image and the credential is a property of the turn. That split is what
|
|
// makes the dangerous mix-up unrepresentable: this VM cannot be handed
|
|
// an Anthropic token, and a `claude` VM cannot be pointed at z.ai.
|
|
Some("glm") => Ok(Credential {
|
|
source: "ZAI_API_KEY",
|
|
target: "ANTHROPIC_AUTH_TOKEN",
|
|
}),
|
|
Some("kimi") => Ok(Credential {
|
|
source: "KIMI_API_KEY",
|
|
target: "ANTHROPIC_AUTH_TOKEN",
|
|
}),
|
|
// A model running on the NODE ITSELF. There is no credential and there
|
|
// is nothing to protect: the guest reaches it over a vsock pipe to the
|
|
// node's own loopback (`clawmates-node::local_model`), which has no
|
|
// destination in its protocol and can therefore reach nothing else.
|
|
//
|
|
// It still goes through this map rather than around it. Ollama ignores
|
|
// the bearer but Claude Code refuses to start without one, so the value
|
|
// is a literal — and routing it here keeps the rule that a backend with
|
|
// no entry cannot launch, which is what stops a typo'd backend from
|
|
// quietly inheriting the subscription token.
|
|
Some("local-ornith") => Ok(Credential {
|
|
source: "CLAWMATES_LOCAL_MODEL_TOKEN",
|
|
target: "ANTHROPIC_AUTH_TOKEN",
|
|
}),
|
|
// Kimi, on the same split as GLM: endpoint in the image
|
|
// (`https://api.kimi.com/coding`), credential in the turn.
|
|
//
|
|
// Which HOST took a measurement to find. `api.moonshot.ai/anthropic`
|
|
// exists and speaks the protocol, and rejects an `sk-kimi-` key — it is
|
|
// the platform.moonshot.ai account namespace. The Kimi CODE service at
|
|
// `api.kimi.com/coding` is where such a key is valid, and it answers
|
|
// `/v1/messages` with a real Anthropic body. Two endpoints that both
|
|
// "work" for different accounts is exactly the shape that makes a
|
|
// guessed URL look like a broken key.
|
|
Some(other) => Err(format!(
|
|
"backend {other:?} has no defined microVM credential contract yet — \
|
|
refusing to launch rather than forward one provider's credential to \
|
|
another provider's endpoint"
|
|
)),
|
|
}
|
|
}
|
|
|
|
fn microvm_provider_env_from(
|
|
backend: Option<&str>,
|
|
lookup: impl Fn(&str) -> Option<String>,
|
|
) -> Result<Vec<(String, String)>, String> {
|
|
let want = microvm_credential_for(backend)?;
|
|
// A local model has no secret to look up. Defaulted rather than demanded:
|
|
// requiring an operator to set a variable whose value is ignored is a step
|
|
// that only ever fails, and its failure mode here is a hung agent.
|
|
let lookup = |k: &str| {
|
|
lookup(k).or_else(|| {
|
|
(k == "CLAWMATES_LOCAL_MODEL_TOKEN").then(|| "local-model-no-auth".to_string())
|
|
})
|
|
};
|
|
let token = lookup(want.source)
|
|
.filter(|v| !v.trim().is_empty())
|
|
.ok_or_else(|| {
|
|
format!(
|
|
"{} is not set on this server, so a microVM mission on backend \
|
|
{backend:?} would run `claude -p` with no credential — which hangs \
|
|
rather than failing. Set it, or run the mission on another backend.",
|
|
want.source
|
|
)
|
|
})?;
|
|
|
|
let mut env = vec![(want.target.to_string(), token)];
|
|
// Non-Anthropic providers a mission's tools may need, forwarded when set.
|
|
// ANTHROPIC_API_KEY is absent from this list and must stay absent — see the
|
|
// doc comment above.
|
|
for k in ["GROQ_API_KEY", "OPENAI_API_KEY"] {
|
|
if let Some(v) = lookup(k).filter(|v| !v.trim().is_empty()) {
|
|
env.push((k.to_string(), v));
|
|
}
|
|
}
|
|
debug_assert!(
|
|
!env.iter().any(|(k, _)| k == "ANTHROPIC_API_KEY"),
|
|
"the microVM path must never forward ANTHROPIC_API_KEY"
|
|
);
|
|
Ok(env)
|
|
}
|
|
|
|
/// The testable half of [`forwarded_provider_env`]. The lookup is a parameter
|
|
/// because a test cannot set process environment variables here — the workspace
|
|
/// denies `unsafe`, and `set_var` is racy across test threads regardless.
|
|
fn provider_env_from(
|
|
auth: RuntimeAuth,
|
|
lookup: impl Fn(&str) -> Option<String>,
|
|
) -> Vec<(String, String)> {
|
|
forwarded_provider_keys(auth)
|
|
.into_iter()
|
|
.filter_map(|k| {
|
|
lookup(k)
|
|
.filter(|v| !v.trim().is_empty())
|
|
.map(|v| (k.to_string(), v))
|
|
})
|
|
.collect()
|
|
}
|
|
|
|
/// Read `CLAWMATES_RUNTIME_AUTH`, defaulting to `api_key`.
|
|
///
|
|
/// Defaulting to the existing behaviour is deliberate: an unset or misspelled
|
|
/// value must not silently strip the API key and leave missions unable to
|
|
/// reach a model at all.
|
|
pub fn runtime_auth_mode() -> RuntimeAuth {
|
|
match std::env::var("CLAWMATES_RUNTIME_AUTH")
|
|
.unwrap_or_default()
|
|
.trim()
|
|
.to_ascii_lowercase()
|
|
.as_str()
|
|
{
|
|
"subscription" => RuntimeAuth::Subscription,
|
|
"" | "api_key" => RuntimeAuth::ApiKey,
|
|
other => {
|
|
eprintln!(
|
|
"mission_runtime: unknown CLAWMATES_RUNTIME_AUTH={other:?} — \
|
|
defaulting to api_key"
|
|
);
|
|
RuntimeAuth::ApiKey
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Docker networks the mission runtime container must be attached to.
|
|
/// - `clawmates_core`: talks to the server (skills door, API) + database
|
|
/// - `clawmates_missions`: has egress for outbound provider calls and fetches
|
|
///
|
|
/// Missions used to egress from `clawmates_edge`, the same network as the
|
|
/// SERVER. That made the host's egress policy impossible to scope: a mission
|
|
/// agent runs model-generated shell over content it fetched from the open web
|
|
/// and must not reach the tailnet, but the server on the same subnet must —
|
|
/// Beszel, Ollama, the node daemons. Measured 2026-09-18: the tailnet rule
|
|
/// written for missions cut the server off from architect:8090 within the
|
|
/// minute. A subnet of their own is what lets `clawmates-egress.sh` on gw-04
|
|
/// drop tailnet/private/link-local/ssh for missions and nothing else.
|
|
///
|
|
/// The network is declared by compose (prod: /opt/clawmates/docker-compose.yml;
|
|
/// local: deploy/compose/docker-compose.yml) with the same shape `edge` has —
|
|
/// not internal, so it routes off the host.
|
|
const CORE_NETWORK: &str = "clawmates_core";
|
|
const EDGE_NETWORK: &str = "clawmates_missions";
|
|
|
|
/// Host path (as seen by the docker engine, NOT the server container)
|
|
/// where the mission's checkouts live. Matches the mount source used
|
|
/// by `clawmates-runtime.service`.
|
|
///
|
|
/// This was a hardcoded const that read no env at all, while every writer of
|
|
/// the same tree went through `CLAWMATES_MISSIONS_ROOT`. They agree on this
|
|
/// deployment; they are not guaranteed to, and a reaper pointed at one of them
|
|
/// would have silently left the other's directories behind forever.
|
|
fn missions_host_root() -> String {
|
|
crate::mission_workspace::missions_root()
|
|
.to_string_lossy()
|
|
.into_owned()
|
|
}
|
|
|
|
/// Host path holding the shared ZeroClaw config + seeded agent library
|
|
/// (built up over time by the shared clawmates-runtime.service). Per-
|
|
/// mission runtime containers bind-mount this so their gateway
|
|
/// inherits all the `claw_*` agents that team_template_loader has
|
|
/// provisioned. Each per-mission container then MINTS ITS OWN pairing
|
|
/// code via /admin/paircode/new so the accepted-tokens list is
|
|
/// independent per mission. Overridable for dev via
|
|
/// `CLAWMATES_RUNTIME_SEED_DIR`.
|
|
///
|
|
/// Concurrency caveat: the sqlite files under .zeroclaw/data/ are
|
|
/// currently shared across all per-mission runtimes AND the shared
|
|
/// runtime. Concurrent daemons opening the same sessions.db can
|
|
/// interleave; in practice topology_worker sequentializes runs per
|
|
/// mission so this rarely bites. Long-term: copy-on-write per mission.
|
|
use crate::container_exec::MISSION_UID;
|
|
|
|
const DEFAULT_SEED_DIR: &str = "/root/clawmates-runtime/data";
|
|
|
|
/// What a mission gets its own copy of.
|
|
///
|
|
/// An allow-list, not the whole directory. The seed dir is **1.7 GB** on
|
|
/// gw-04 and 1.5 GB of that is a vestigial `.rustup` — a Rust toolchain that
|
|
/// installed itself into the data dir back when `HOME=/zeroclaw-data` and the
|
|
/// image had no toolchain. The image now ships Rust at `/usr/local/cargo`,
|
|
/// which is what the container's PATH actually resolves (verified live), so
|
|
/// that copy is dead weight. Copying it per mission would cost tens of
|
|
/// seconds and ~17 GB across ten concurrent missions.
|
|
///
|
|
/// So: copy what carries per-mission identity or secrets, and leave the
|
|
/// caches and toolchains behind.
|
|
const SEEDED_PATHS: &[&str] = &[
|
|
// The whole point: config.toml carries the §15 door bearer token, and
|
|
// data/ holds sessions.db + devices.db. ~26 MB.
|
|
".zeroclaw",
|
|
// Door MCP config — also a bearer token.
|
|
"clawmates-mcp.json",
|
|
// Claude Code's own state and credentials (~16 MB). Per-mission so a
|
|
// token refresh or project state in one mission cannot leak into another.
|
|
".claude",
|
|
".claude.json",
|
|
// Per-CLI state for the alternate backends; small. `glm-home` and
|
|
// `kimi-home` are separate HOMEs for the two `claude_cli` fallback
|
|
// aliases: each needs its own `.claude.json` so the binary skips
|
|
// onboarding, and keeping them apart stops two concurrent fallbacks from
|
|
// sharing one Claude Code session directory.
|
|
".kimi-code",
|
|
"glm-home",
|
|
"kimi-home",
|
|
// The seeded agent library.
|
|
"agents",
|
|
];
|
|
|
|
/// Deliberately NOT copied — caches and toolchains, no secrets, expensive:
|
|
/// `.rustup` (1.5 GB, vestigial), `.npm` (85 MB), `.cargo`, `.cache`,
|
|
/// `.local`. A mission that needs them reads the image's copies.
|
|
fn copy_script() -> String {
|
|
let mut out = String::from("set -e\n");
|
|
for p in SEEDED_PATHS {
|
|
// Missing entries are normal — a fresh deployment has no .kimi-code
|
|
// until Kimi is first used — so absence must not fail the copy.
|
|
out.push_str(&format!(
|
|
"if [ -e '/seed/{p}' ]; then cp -a '/seed/{p}' /dst/; fi\n"
|
|
));
|
|
}
|
|
// The copy itself must run as root: the seed dir is root-owned and parts of
|
|
// it are mode 0600 (`.claude.json`, `clawmates-mcp.json`), so uid 65532
|
|
// cannot even READ them. Running the container as 65532 made `cp` fail on
|
|
// the first unreadable entry, `set -e` aborted the rest, and the mission
|
|
// got a runtime-data holding `.zeroclaw` and nothing else — no Claude
|
|
// credentials, no door config.
|
|
//
|
|
// So keep root for the read, and hand the RESULT to 65532. That is what the
|
|
// destination needs: every other writer into the missions tree is 65532,
|
|
// and a GC running as 65532 cannot delete what root left behind.
|
|
out.push_str(&format!("chown -R {MISSION_UID} /dst\n"));
|
|
out
|
|
}
|
|
|
|
/// Give this mission its own copy of the runtime seed data.
|
|
///
|
|
/// Every per-mission container used to bind-mount the SAME host seed dir as
|
|
/// `/zeroclaw-data` — shared with each other and with the singleton runtime.
|
|
/// That directory holds `config.toml`, which carries the §15 door bearer
|
|
/// token, plus `sessions.db` and `devices.db`. So a mission could read another
|
|
/// mission's credential, and anything it wrote there was inherited by every
|
|
/// later mission. Teardown never cleaned it, because teardown only removes
|
|
/// `/var/lib/clawmates-missions/{id}`.
|
|
///
|
|
/// The code already knew: the comment on `DEFAULT_SEED_DIR` names the sqlite
|
|
/// race and calls copy-on-write per mission the long-term fix. This is that.
|
|
///
|
|
/// The copy runs in a throwaway container because cm-api cannot see the seed
|
|
/// dir — it hands that host path to Docker but never mounts it itself. The
|
|
/// runtime image is reused so nothing extra is pulled.
|
|
///
|
|
/// Failure is fatal to container creation on purpose. Falling back to the
|
|
/// shared mount would silently restore the credential-sharing this removes,
|
|
/// and a silent fallback to a weaker posture is the failure mode this
|
|
/// codebase keeps paying for.
|
|
async fn seed_runtime_data(
|
|
docker: &Docker,
|
|
image: &str,
|
|
seed_dir: &str,
|
|
dest_dir: &str,
|
|
) -> Result<(), String> {
|
|
let name = format!("cm-seed-{}", Uuid::now_v7().simple());
|
|
let config = ContainerCreateBody {
|
|
image: Some(image.to_string()),
|
|
entrypoint: Some(vec!["/bin/sh".to_string()]),
|
|
cmd: Some(vec!["-c".to_string(), copy_script()]),
|
|
host_config: Some(HostConfig {
|
|
mounts: Some(vec![
|
|
Mount {
|
|
target: Some("/seed".to_string()),
|
|
source: Some(seed_dir.to_string()),
|
|
typ: Some(MountTypeEnum::BIND),
|
|
read_only: Some(true),
|
|
..Default::default()
|
|
},
|
|
Mount {
|
|
target: Some("/dst".to_string()),
|
|
source: Some(dest_dir.to_string()),
|
|
typ: Some(MountTypeEnum::BIND),
|
|
read_only: Some(false),
|
|
..Default::default()
|
|
},
|
|
]),
|
|
auto_remove: Some(true),
|
|
..Default::default()
|
|
}),
|
|
..Default::default()
|
|
};
|
|
|
|
docker
|
|
.create_container(
|
|
Some(CreateContainerOptions {
|
|
name: Some(name.clone()),
|
|
..Default::default()
|
|
}),
|
|
config,
|
|
)
|
|
.await
|
|
.map_err(|e| format!("create seed copier: {e}"))?;
|
|
docker
|
|
.start_container(&name, None::<StartContainerOptions>)
|
|
.await
|
|
.map_err(|e| format!("start seed copier: {e}"))?;
|
|
|
|
// `auto_remove` means the container disappears the moment it exits, so
|
|
// poll for absence rather than waiting on it.
|
|
for _ in 0..120 {
|
|
tokio::time::sleep(std::time::Duration::from_millis(250)).await;
|
|
match docker
|
|
.inspect_container(&name, None::<InspectContainerOptions>)
|
|
.await
|
|
{
|
|
// `auto_remove` already took it away, which only happens after a
|
|
// clean-ish exit; the status is gone with it. Treated as success
|
|
// because the alternative — failing every mission on a race with
|
|
// the reaper — is worse, and `stamp_workspace_paths` fails loudly
|
|
// downstream if the config did not arrive.
|
|
Err(_) => return Ok(()),
|
|
Ok(info) => {
|
|
let state = info.state.as_ref();
|
|
let running = state.and_then(|st| st.running).unwrap_or(false);
|
|
if !running {
|
|
// The exit code was never read. A `cp` that died on an
|
|
// unreadable seed file left `set -e` to abort the rest, and
|
|
// this returned Ok — so the mission came up with a
|
|
// half-seeded runtime, the daemon never created its agents'
|
|
// workspace, and the phase then failed on
|
|
// "Could not find the file /mission in container",
|
|
// 200 lines and one indirection away from the real cause.
|
|
let code = state.and_then(|st| st.exit_code).unwrap_or(0);
|
|
if code != 0 {
|
|
return Err(format!(
|
|
"seed copier exited {code} — the mission's runtime-data is \
|
|
incomplete (the seed dir is root-owned and partly mode 0600; \
|
|
check that this container still runs as root)"
|
|
));
|
|
}
|
|
return Ok(());
|
|
}
|
|
}
|
|
}
|
|
}
|
|
Err(format!("seed copy into {dest_dir} did not finish in 30s"))
|
|
}
|
|
|
|
/// Deterministic docker container name for a mission's runtime.
|
|
/// Uses the full UUID hex — UUIDv7 encodes time in the leading bytes,
|
|
/// so a short prefix isn't guaranteed unique across missions minted
|
|
/// in the same second. Docker permits up to 253 characters in a name,
|
|
/// so the extra length is free.
|
|
pub fn container_name(mission_id: Uuid) -> String {
|
|
format!("cm-runtime-mission-{}", mission_id.simple())
|
|
}
|
|
|
|
/// Endpoint URL the topology_worker's ZeroClawDriveExecutor will dial.
|
|
/// Uses the container name as hostname — resolves within the shared
|
|
/// `clawmates_core` docker network.
|
|
pub fn endpoint_url(container_name: &str) -> String {
|
|
format!("http://{}:{}", container_name, GATEWAY_PORT)
|
|
}
|
|
|
|
pub struct MissionRuntimeProvisioner {
|
|
docker: Docker,
|
|
image: String,
|
|
}
|
|
|
|
/// What `ensure_container` returns: everything the caller needs to
|
|
/// point a topology_worker at this mission's fresh gateway.
|
|
#[derive(Debug, Clone)]
|
|
pub struct EnsuredContainer {
|
|
pub endpoint: String,
|
|
/// Where the tool hooks were written, or `None` if installing them failed.
|
|
///
|
|
/// Carried out to the caller rather than only logged, because the caller
|
|
/// has the pool and this struct's producer does not. Before this field
|
|
/// the outcome went to stderr and nowhere else, so a mission whose gate
|
|
/// never installed left a record indistinguishable from one whose gate
|
|
/// stood there all night and matched nothing.
|
|
pub hooks: Option<String>,
|
|
/// One-time pairing code minted by the daemon at boot; may be
|
|
/// None on the reuse-existing path when we couldn't scrape it
|
|
/// back (log rotation). Callers keep the previously-persisted
|
|
/// value in that case.
|
|
pub pairing_code: Option<String>,
|
|
}
|
|
|
|
impl MissionRuntimeProvisioner {
|
|
/// Connect to the docker engine. Honors `DOCKER_HOST` (set in
|
|
/// compose to the socket proxy) and falls back to the local
|
|
/// socket. Returns None when docker is unreachable, so callers
|
|
/// can degrade gracefully (missions still launch, just against
|
|
/// the shared runtime).
|
|
pub fn from_env() -> Option<MissionRuntimeProvisioner> {
|
|
let docker = if let Ok(host) = std::env::var("DOCKER_HOST") {
|
|
Docker::connect_with_http(&host, 30, bollard::API_DEFAULT_VERSION).ok()?
|
|
} else {
|
|
Docker::connect_with_local_defaults().ok()?
|
|
};
|
|
let (image, source) = match std::env::var("CLAWMATES_RUNTIME_IMAGE") {
|
|
Ok(v) if !v.trim().is_empty() => (v, "CLAWMATES_RUNTIME_IMAGE"),
|
|
_ => (DEFAULT_IMAGE.to_string(), "the built-in default"),
|
|
};
|
|
// ONCE per process. `from_env` is called per use — on every launch, and
|
|
// from the reaper sweep — so a bare eprintln here is a line every sweep
|
|
// tick forever, which is how a log stops being read at all.
|
|
static ANNOUNCED: std::sync::Once = std::sync::Once::new();
|
|
ANNOUNCED.call_once(|| {
|
|
eprintln!("mission_runtime: per-mission runtime image = {image} (from {source})");
|
|
});
|
|
Some(MissionRuntimeProvisioner { docker, image })
|
|
}
|
|
|
|
/// Is this container actually attached to the egress network?
|
|
///
|
|
/// Asked only when the attach reported an error, to tell "already connected"
|
|
/// apart from "not connected". Inspect failing is treated as NOT attached:
|
|
/// the whole point is to stop guessing that egress is present.
|
|
async fn is_on_edge_network(&self, name: &str) -> bool {
|
|
self.docker
|
|
.inspect_container(name, None::<bollard::query_parameters::InspectContainerOptions>)
|
|
.await
|
|
.ok()
|
|
.and_then(|c| c.network_settings?.networks)
|
|
.is_some_and(|nets| nets.contains_key(EDGE_NETWORK))
|
|
}
|
|
|
|
/// Idempotent: returns the endpoint URL, creating the container
|
|
/// on first call. If the container exists but is stopped, starts
|
|
/// it. If it exists and is running, returns its endpoint.
|
|
pub async fn ensure_container(&self, mission_id: Uuid) -> Result<EnsuredContainer, String> {
|
|
let name = container_name(mission_id);
|
|
// Fast path: already running.
|
|
if let Ok(inspect) = self
|
|
.docker
|
|
.inspect_container(&name, None::<InspectContainerOptions>)
|
|
.await
|
|
{
|
|
let running = inspect
|
|
.state
|
|
.as_ref()
|
|
.and_then(|s| s.running)
|
|
.unwrap_or(false);
|
|
if running {
|
|
// Reuse; try to re-scrape the pairing code from logs,
|
|
// but it may have rotated out — caller falls back to
|
|
// the previously-persisted value in that case.
|
|
// Re-install on reuse: the container outlives the server
|
|
// process, and a hook that exists only on first creation is a
|
|
// hook that quietly disappears after a redeploy.
|
|
let hooks = crate::container_tool_hooks::install_with(
|
|
&self.docker,
|
|
&name,
|
|
// A container serves every phase of the mission, so the
|
|
// policy installed here is the mission-wide default. A
|
|
// phase's own `agent_tools` is honoured on the microVM
|
|
// tier, where the VM is per-phase; narrowing per phase
|
|
// here would need a re-install between phases and is not
|
|
// done.
|
|
Some(&crate::vm_tool_gate::TaskPolicy::default_shadow()),
|
|
)
|
|
.await;
|
|
let pairing_code = self.mint_pairing_code(&name).await;
|
|
return Ok(EnsuredContainer {
|
|
endpoint: endpoint_url(&name),
|
|
pairing_code,
|
|
hooks,
|
|
});
|
|
}
|
|
// Exists but not running — remove + recreate below rather
|
|
// than trying to restart a dirty-state container.
|
|
let _ = self
|
|
.docker
|
|
.remove_container(
|
|
&name,
|
|
Some(RemoveContainerOptions {
|
|
force: true,
|
|
..Default::default()
|
|
}),
|
|
)
|
|
.await;
|
|
}
|
|
|
|
// Create fresh. Ensure the bind source exists first — research-
|
|
// only missions (no repo checkout) still need the directory
|
|
// present or docker start fails with EACCES/ENOENT.
|
|
let mission_dir = format!("{}/{mission_id}", missions_host_root());
|
|
let _ = tokio::fs::create_dir_all(&mission_dir).await;
|
|
let seed_dir = std::env::var("CLAWMATES_RUNTIME_SEED_DIR")
|
|
.unwrap_or_else(|_| DEFAULT_SEED_DIR.to_string());
|
|
// Per-mission copy of the seed data. See `seed_runtime_data`: sharing
|
|
// one directory meant sharing the door token and letting any mission
|
|
// poison every later one.
|
|
let runtime_data_dir = format!("{mission_dir}/runtime-data");
|
|
let _ = tokio::fs::create_dir_all(&runtime_data_dir).await;
|
|
seed_runtime_data(&self.docker, &self.image, &seed_dir, &runtime_data_dir).await?;
|
|
let mut mounts = Vec::new();
|
|
// In copy mode the checkout is pushed in and pulled back out, so the
|
|
// container gets its OWN filesystem and the host directory has exactly
|
|
// one writer (the server). Binding it here would put two uids back on
|
|
// one directory — the cause of four separate work-loss bugs.
|
|
if !crate::mission_fs::copy_mode() {
|
|
mounts.push(Mount {
|
|
target: Some("/mission".to_string()),
|
|
source: Some(mission_dir.clone()),
|
|
typ: Some(MountTypeEnum::BIND),
|
|
read_only: Some(false),
|
|
..Default::default()
|
|
});
|
|
}
|
|
mounts.extend([
|
|
// This mission's OWN copy of the seeded agent library
|
|
// (`claw_*` templates). Copied rather than shared, so its
|
|
// config.toml — which carries the door bearer token — and its
|
|
// sqlite files belong to this mission alone and are removed with
|
|
// it by `teardown_container`.
|
|
Mount {
|
|
target: Some("/zeroclaw-data".to_string()),
|
|
source: Some(runtime_data_dir),
|
|
typ: Some(MountTypeEnum::BIND),
|
|
read_only: Some(false),
|
|
..Default::default()
|
|
},
|
|
]);
|
|
|
|
let host_config = HostConfig {
|
|
mounts: Some(mounts),
|
|
restart_policy: Some(bollard::models::RestartPolicy {
|
|
name: Some(bollard::models::RestartPolicyNameEnum::UNLESS_STOPPED),
|
|
..Default::default()
|
|
}),
|
|
network_mode: Some(CORE_NETWORK.to_string()),
|
|
..Default::default()
|
|
};
|
|
|
|
// NOTE: do NOT set ZEROCLAW_WORKSPACE — despite the name, the
|
|
// daemon uses it (schema.rs:17467) as a legacy config-dir
|
|
// pointer that overrides ZEROCLAW_CONFIG_DIR/DATA_DIR. Setting
|
|
// it to /mission/repo makes the daemon compute its config dir
|
|
// as /mission/repo/.zeroclaw (empty!) and boot with a fresh
|
|
// defaults-only config — zero agents loaded.
|
|
//
|
|
// Per-agent workspace pinning belongs in the shared config
|
|
// under agents.<alias>.workspace = "/mission/repo", not env.
|
|
let mut env = vec![
|
|
format!("ZEROCLAW_GATEWAY_PORT={GATEWAY_PORT}"),
|
|
format!("CM_MISSION_ID={mission_id}"),
|
|
// Let the agents' git read the checkout.
|
|
//
|
|
// The server clones as uid 65532; this container runs as root, so
|
|
// every `git` an agent runs hits "detected dubious ownership" and
|
|
// refuses the repository. Agents do not report that as a failure —
|
|
// they improvise. On mission 019fc3ba one wrote a `.gitconfig_temp`
|
|
// containing `[safe] directory = /mission/repo` into the repository
|
|
// root, which then showed up in the captured diff and would have
|
|
// been committed and pushed to the user's repo alongside the real
|
|
// work.
|
|
//
|
|
// `GIT_CONFIG_*` is git's environment form of `-c` and is
|
|
// inherited by subprocesses, so it covers the agent's own git, any
|
|
// tool that shells out to git, and the `git_operations` tool alike.
|
|
// Scoped to the checkout; never `--global`.
|
|
"GIT_CONFIG_COUNT=1".to_string(),
|
|
"GIT_CONFIG_KEY_0=safe.directory".to_string(),
|
|
"GIT_CONFIG_VALUE_0=/mission/repo".to_string(),
|
|
];
|
|
// Provider credentials forwarded into the container.
|
|
//
|
|
// ANTHROPIC_API_KEY is conditional, and the reason is subtle enough to
|
|
// be worth stating: Claude Code resolves credentials in a fixed
|
|
// priority order, and ANTHROPIC_API_KEY ranks ABOVE the subscription's
|
|
// CLAUDE_CODE_OAUTH_TOKEN. So on a runtime authenticated via `claude
|
|
// /login`, forwarding the key here silently wins — `claude` still works,
|
|
// the agents still run, and every mission bills the API while appearing
|
|
// to use the subscription. Failing loudly is impossible; the only fix
|
|
// is not to send it.
|
|
//
|
|
// The other three are unrelated providers (Gemini/Groq/OpenAI) with no
|
|
// subscription equivalent, so they forward in both modes.
|
|
let auth_mode = runtime_auth_mode();
|
|
for (key, v) in forwarded_provider_env(auth_mode) {
|
|
env.push(format!("{key}={v}"));
|
|
}
|
|
eprintln!(
|
|
"mission_runtime: mission {mission_id} container auth mode = {} \
|
|
(ANTHROPIC_API_KEY {})",
|
|
auth_mode.as_str(),
|
|
if auth_mode == RuntimeAuth::ApiKey {
|
|
"forwarded"
|
|
} else {
|
|
"withheld so the runtime's subscription login is used"
|
|
}
|
|
);
|
|
|
|
let mut labels = HashMap::new();
|
|
labels.insert("clawmates.role".to_string(), "mission-runtime".to_string());
|
|
labels.insert("clawmates.mission_id".to_string(), mission_id.to_string());
|
|
|
|
let config = ContainerCreateBody {
|
|
image: Some(self.image.clone()),
|
|
cmd: Some(vec![
|
|
"daemon".to_string(),
|
|
"--host".to_string(),
|
|
"0.0.0.0".to_string(),
|
|
]),
|
|
env: Some(env),
|
|
host_config: Some(host_config),
|
|
labels: Some(labels),
|
|
..Default::default()
|
|
};
|
|
|
|
self.docker
|
|
.create_container(
|
|
Some(CreateContainerOptions {
|
|
name: Some(name.clone()),
|
|
..Default::default()
|
|
}),
|
|
config,
|
|
)
|
|
.await
|
|
.map_err(|e| format!("create mission runtime container: {e}"))?;
|
|
|
|
// Attach to the edge network for outbound provider egress.
|
|
//
|
|
// `clawmates_core` is `internal: true` and has NO default route —
|
|
// verified from a container on it, where every external address is
|
|
// unreachable. So this attach is not an optimisation: without it the
|
|
// mission cannot reach a provider, cannot fetch anything, and cannot do
|
|
// its work. The result used to be discarded, which made a failure here
|
|
// indistinguishable from success and produced the green-with-nothing
|
|
// shape this codebase keeps meeting.
|
|
//
|
|
// Not fatal on the error alone: re-attaching an already-connected
|
|
// container is an error too, and a benign one on any relaunch path. The
|
|
// container's own network list is the fact that settles it.
|
|
if let Err(e) = self
|
|
.docker
|
|
.connect_network(
|
|
EDGE_NETWORK,
|
|
NetworkConnectRequest {
|
|
container: Some(name.clone()),
|
|
endpoint_config: Some(EndpointSettings::default()),
|
|
},
|
|
)
|
|
.await
|
|
{
|
|
if self.is_on_edge_network(&name).await {
|
|
eprintln!(
|
|
"mission_runtime: {name} was already on {EDGE_NETWORK} ({e}) — \
|
|
egress is present, continuing"
|
|
);
|
|
} else {
|
|
return Err(format!(
|
|
"attach mission runtime container to {EDGE_NETWORK}: {e} — \
|
|
{CORE_NETWORK} is internal and has no route off the host, so \
|
|
this mission would run with no egress at all: every provider \
|
|
call and every fetch would fail while the phase still \
|
|
reported completion. Is the `missions` network declared in \
|
|
the compose file and created (`docker network ls`)?"
|
|
));
|
|
}
|
|
}
|
|
|
|
self.docker
|
|
.start_container(&name, None::<StartContainerOptions>)
|
|
.await
|
|
.map_err(|e| format!("start mission runtime container: {e}"))?;
|
|
|
|
// Give the daemon a moment to print its boot banner, then
|
|
// scrape the pairing code. Poll with a short deadline so a
|
|
// slow boot doesn't hang the launch — the topology_worker
|
|
// will retry pairing later if we came up empty.
|
|
let pairing_code = self.wait_for_pairing_code(&name).await;
|
|
|
|
// Gate and observe the tools claude runs inside its own subprocess.
|
|
// Best-effort by design: a phase that runs unhooked still delivers, and
|
|
// failing the launch to protect telemetry would be the wrong trade.
|
|
let hooks = crate::container_tool_hooks::install_with(
|
|
&self.docker,
|
|
&name,
|
|
// A container serves every phase of the mission, so the
|
|
// policy installed here is the mission-wide default. A
|
|
// phase's own `agent_tools` is honoured on the microVM
|
|
// tier, where the VM is per-phase; narrowing per phase
|
|
// here would need a re-install between phases and is not
|
|
// done.
|
|
Some(&crate::vm_tool_gate::TaskPolicy::default_shadow()),
|
|
)
|
|
.await;
|
|
|
|
Ok(EnsuredContainer {
|
|
endpoint: endpoint_url(&name),
|
|
pairing_code,
|
|
hooks,
|
|
})
|
|
}
|
|
|
|
/// Poll the daemon's localhost admin endpoint until it responds
|
|
/// with a fresh pairing code. Boot takes ~1-2s; deadline is 15s.
|
|
/// Returns None on timeout so callers see a clear paired=false
|
|
/// signal in the mission binding log.
|
|
async fn wait_for_pairing_code(&self, name: &str) -> Option<String> {
|
|
let deadline = std::time::Duration::from_secs(15);
|
|
let start = std::time::Instant::now();
|
|
while start.elapsed() < deadline {
|
|
if let Some(code) = self.mint_pairing_code(name).await {
|
|
return Some(code);
|
|
}
|
|
tokio::time::sleep(std::time::Duration::from_millis(500)).await;
|
|
}
|
|
None
|
|
}
|
|
|
|
/// `docker exec` into the container and hit the local admin
|
|
/// endpoint that mints a fresh pairing code. This works whether
|
|
/// the daemon booted "already paired" (no code in the log) or
|
|
/// "pairing required" (code in the log) — both surfaces mint on
|
|
/// demand.
|
|
async fn mint_pairing_code(&self, name: &str) -> Option<String> {
|
|
let exec = self
|
|
.docker
|
|
.create_exec(
|
|
name,
|
|
CreateExecOptions {
|
|
cmd: Some(
|
|
[
|
|
"curl",
|
|
"-fs",
|
|
"-X",
|
|
"POST",
|
|
"http://127.0.0.1:42617/admin/paircode/new",
|
|
]
|
|
.iter()
|
|
.map(|s| s.to_string())
|
|
.collect(),
|
|
),
|
|
attach_stdout: Some(true),
|
|
attach_stderr: Some(true),
|
|
..Default::default()
|
|
},
|
|
)
|
|
.await
|
|
.ok()?;
|
|
let started = self.docker.start_exec(&exec.id, None).await.ok()?;
|
|
let StartExecResults::Attached { mut output, .. } = started else {
|
|
return None;
|
|
};
|
|
let mut buf = String::new();
|
|
while let Some(chunk) = output.next().await {
|
|
if let Ok(c) = chunk {
|
|
buf.push_str(&c.to_string());
|
|
if buf.len() > 8_000 {
|
|
break;
|
|
}
|
|
}
|
|
}
|
|
extract_pairing_code_from_json(&buf)
|
|
}
|
|
|
|
/// Stamp `[agents.<alias>.workspace] path = "<workspace_path>"` into the
|
|
/// shared runtime config for each provisioned claw, by editing the
|
|
/// config file directly on the per-mission container. This is the
|
|
/// out-of-band path for workspace pinning: the ZeroClaw config prop API
|
|
/// cannot set `workspace.path` (an `Option<PathBuf>` the `Configurable`
|
|
/// macro skips from prop enumeration), so `provision_claw` leaves it
|
|
/// unset and we stamp it here. The daemon applies the change on the
|
|
/// same config reload that surfaces the freshly-provisioned claws for
|
|
/// the run.
|
|
///
|
|
/// Concurrency caveat (mirrors the seed-dir note above): the config
|
|
/// file is shared across the persistent runtime and every per-mission
|
|
/// daemon, so this read-modify-write can race a provision from another
|
|
/// mission launching at the same instant. Missions launch one at a
|
|
/// time in practice; the durable fix is per-mission config isolation.
|
|
pub async fn pin_agent_workspaces(
|
|
&self,
|
|
mission_id: Uuid,
|
|
claws: &[cm_domain::AgentId],
|
|
workspace_path: &str,
|
|
) -> Result<(), String> {
|
|
if claws.is_empty() {
|
|
return Ok(());
|
|
}
|
|
const CONFIG_PATH: &str = "/zeroclaw-data/.zeroclaw/config.toml";
|
|
let name = container_name(mission_id);
|
|
|
|
let raw = self
|
|
.exec_capture(&name, vec!["cat".into(), CONFIG_PATH.into()])
|
|
.await?;
|
|
let aliases: Vec<String> = claws
|
|
.iter()
|
|
.map(|c| crate::runtime_provision::claw_alias(c.as_uuid()))
|
|
.collect();
|
|
let (edited, pinned) = stamp_workspace_paths(&raw, &aliases, workspace_path)?;
|
|
// Pinning NOTHING is a failure, not a no-op. Returning Ok here meant the
|
|
// caller's deliberately-fatal guard could not fire, so a mission whose
|
|
// aliases were missing from the config launched anyway with every agent
|
|
// writing into its own sandbox and delivering nothing — the outcome that
|
|
// guard's error message already describes. Name the aliases: the only
|
|
// way this happens is a config/alias mismatch, and the aliases are the
|
|
// evidence needed to find it.
|
|
if pinned == 0 {
|
|
return Err(format!(
|
|
"pinned 0 of {} agent workspace(s) to {workspace_path} — none of these \
|
|
aliases exist in {CONFIG_PATH}: {}",
|
|
aliases.len(),
|
|
aliases.join(", ")
|
|
));
|
|
}
|
|
|
|
// Upload rather than exec. This used to base64 the whole config into a
|
|
// single `sh -c` argument, which works until the file grows — config
|
|
// gains a block per provisioned claw — and then fails with
|
|
// `argument list too long`. Mission `019fcf62` hit exactly that: the
|
|
// pin never applied, its agents never saw `/mission/repo`, and phase 0
|
|
// completed having written nothing. The tar API has no argv limit, so
|
|
// the failure mode is gone rather than merely further away.
|
|
crate::mission_fs::put_file(&self.docker, &name, CONFIG_PATH, edited.as_bytes())
|
|
.await
|
|
.map_err(|e| format!("write runtime config.toml: {e}"))?;
|
|
eprintln!(
|
|
"mission_runtime: pinned {pinned} workspace(s) → {workspace_path} for mission {mission_id}"
|
|
);
|
|
Ok(())
|
|
}
|
|
|
|
/// Run a command in the mission container and return its combined
|
|
/// stdout+stderr as a String. Used for small config round-trips.
|
|
async fn exec_capture(&self, name: &str, cmd: Vec<String>) -> Result<String, String> {
|
|
let exec = self
|
|
.docker
|
|
.create_exec(
|
|
name,
|
|
CreateExecOptions {
|
|
cmd: Some(cmd),
|
|
attach_stdout: Some(true),
|
|
attach_stderr: Some(true),
|
|
..Default::default()
|
|
},
|
|
)
|
|
.await
|
|
.map_err(|e| format!("create_exec on {name}: {e}"))?;
|
|
let started = self
|
|
.docker
|
|
.start_exec(&exec.id, None)
|
|
.await
|
|
.map_err(|e| format!("start_exec on {name}: {e}"))?;
|
|
let StartExecResults::Attached { mut output, .. } = started else {
|
|
return Err(format!("exec on {name} returned a detached result"));
|
|
};
|
|
let mut buf = String::new();
|
|
while let Some(chunk) = output.next().await {
|
|
match chunk {
|
|
Ok(c) => buf.push_str(&c.to_string()),
|
|
Err(e) => return Err(format!("exec output stream on {name}: {e}")),
|
|
}
|
|
}
|
|
Ok(buf)
|
|
}
|
|
}
|
|
|
|
/// Format-preserving stamp of `[agents.<alias>.workspace] path = "<path>"`
|
|
/// for each alias present in `raw`. Keeps the operator's comments, ordering,
|
|
/// and every untouched byte intact; only the `path` keys change. Aliases not
|
|
/// already present are skipped (never fabricated — a bare agent table would
|
|
/// drop that agent's model/risk_profile/bundles). Returns the edited document
|
|
/// and how many agents were pinned.
|
|
fn stamp_workspace_paths(
|
|
raw: &str,
|
|
aliases: &[String],
|
|
workspace_path: &str,
|
|
) -> Result<(String, usize), String> {
|
|
let mut doc = raw
|
|
.parse::<toml_edit::DocumentMut>()
|
|
.map_err(|e| format!("parse runtime config.toml: {e}"))?;
|
|
let Some(agents) = doc.get_mut("agents").and_then(|i| i.as_table_like_mut()) else {
|
|
return Err("runtime config has no [agents] table".to_string());
|
|
};
|
|
let mut pinned = 0usize;
|
|
for alias in aliases {
|
|
let Some(agent) = agents.get_mut(alias).and_then(|i| i.as_table_like_mut()) else {
|
|
eprintln!("mission_runtime: pin skip — {alias} absent from config");
|
|
continue;
|
|
};
|
|
if agent.get("workspace").is_none() {
|
|
agent.insert("workspace", toml_edit::Item::Table(toml_edit::Table::new()));
|
|
}
|
|
if let Some(ws) = agent
|
|
.get_mut("workspace")
|
|
.and_then(|i| i.as_table_like_mut())
|
|
{
|
|
ws.insert("path", toml_edit::value(workspace_path));
|
|
pinned += 1;
|
|
}
|
|
}
|
|
Ok((doc.to_string(), pinned))
|
|
}
|
|
|
|
/// Parse the JSON `{ "pairing_code": "NNNNNN", ... }` body from the
|
|
/// admin/paircode/new endpoint.
|
|
fn extract_pairing_code_from_json(body: &str) -> Option<String> {
|
|
let v: serde_json::Value = serde_json::from_str(body.trim()).ok()?;
|
|
v.get("pairing_code")
|
|
.and_then(|x| x.as_str())
|
|
.filter(|s| !s.is_empty())
|
|
.map(String::from)
|
|
}
|
|
|
|
impl MissionRuntimeProvisioner {
|
|
/// Restart the mission's runtime container and wait for its gateway to
|
|
/// answer again. Needed after `pin_agent_workspaces`: the daemon reads
|
|
/// config once at boot and never re-reads the file, so a file-only setting
|
|
/// (`workspace.path`, which the config prop API cannot set) only takes
|
|
/// effect across a restart. Agents provisioned through the daemon's own
|
|
/// config API are already persisted to that file, so they survive.
|
|
pub async fn restart_container(&self, mission_id: Uuid) -> Result<(), String> {
|
|
let name = container_name(mission_id);
|
|
self.docker
|
|
.restart_container(
|
|
&name,
|
|
None::<bollard::query_parameters::RestartContainerOptions>,
|
|
)
|
|
.await
|
|
.map_err(|e| format!("restart mission runtime container: {e}"))?;
|
|
// Wait for the gateway to serve again so the caller can launch a run
|
|
// immediately after. ~20s ceiling; the daemon normally boots in ~2s.
|
|
for _ in 0..40 {
|
|
tokio::time::sleep(std::time::Duration::from_millis(500)).await;
|
|
// NOTE: exec_capture only fails on docker errors — a curl that
|
|
// can't connect still "succeeds" (it prints `curl: (7) …`), so we
|
|
// must inspect the BODY. /health answers `{"paired":…}`.
|
|
let body = self
|
|
.exec_capture(
|
|
&name,
|
|
vec![
|
|
"curl".into(),
|
|
"-fsS".into(),
|
|
"-m".into(),
|
|
"2".into(),
|
|
format!("http://127.0.0.1:{GATEWAY_PORT}/health"),
|
|
],
|
|
)
|
|
.await
|
|
.unwrap_or_default();
|
|
if body.contains("\"paired\"") {
|
|
eprintln!("mission_runtime: restarted {name}, gateway healthy");
|
|
return Ok(());
|
|
}
|
|
}
|
|
Err(format!("{name} gateway did not come back after restart"))
|
|
}
|
|
|
|
/// Does this mission's runtime container still exist, in any state?
|
|
///
|
|
/// Used by the sweeper to decide whether a failed teardown left something
|
|
/// behind. Deliberately treats an inspect error as "gone": the caller uses
|
|
/// this to decide whether to KEEP a binding for a retry, and answering
|
|
/// "still there" when docker cannot be reached would pin the binding open
|
|
/// on an unreachable daemon rather than on a real container.
|
|
/// Every `cm-runtime-mission-*` container on this engine, running or not.
|
|
///
|
|
/// The piece the row-driven sweep never had. Without it "which containers
|
|
/// exist" is a question the platform cannot ask, and a container the
|
|
/// database has forgotten is not merely unreaped — it is unseeable.
|
|
pub async fn list_mission_containers(&self) -> Result<Vec<(String, Option<i64>)>, String> {
|
|
let mut filters = std::collections::HashMap::new();
|
|
filters.insert("name".to_string(), vec!["cm-runtime-mission-".to_string()]);
|
|
let opts = bollard::query_parameters::ListContainersOptionsBuilder::default()
|
|
.all(true)
|
|
.filters(&filters)
|
|
.build();
|
|
let list = self
|
|
.docker
|
|
.list_containers(Some(opts))
|
|
.await
|
|
.map_err(|e| format!("list mission containers: {e}"))?;
|
|
Ok(list
|
|
.into_iter()
|
|
.filter_map(|c| {
|
|
let name = c
|
|
.names
|
|
.unwrap_or_default()
|
|
.into_iter()
|
|
// Docker returns names with a leading slash.
|
|
.map(|n| n.trim_start_matches('/').to_string())
|
|
.find(|n| n.starts_with("cm-runtime-mission-"))?;
|
|
Some((name, c.created))
|
|
})
|
|
.collect())
|
|
}
|
|
|
|
/// How long ago docker says this container was created.
|
|
///
|
|
/// Taken from the listing rather than a second `inspect`: `created` is
|
|
/// already a unix timestamp there, so this needs neither a date parser nor
|
|
/// another round-trip. `None` when docker reported none, and the caller
|
|
/// treats that as "do not reap" — a container we cannot date is exactly the
|
|
/// one worth leaving.
|
|
pub fn container_age(created_epoch: Option<i64>) -> Option<std::time::Duration> {
|
|
let created = created_epoch?;
|
|
let now = std::time::SystemTime::now()
|
|
.duration_since(std::time::UNIX_EPOCH)
|
|
.ok()?
|
|
.as_secs() as i64;
|
|
u64::try_from(now - created).ok().map(std::time::Duration::from_secs)
|
|
}
|
|
|
|
/// Does this container's checkout hold commits no remote has?
|
|
///
|
|
/// Answered by `git` inside the container, because only it knows which
|
|
/// refs the remote had. `--not --remotes` lists every commit reachable
|
|
/// from any local ref and from no remote-tracking ref — which is exactly
|
|
/// "work that exists only here".
|
|
///
|
|
/// Every failure path returns `SomeOrUnknown`. A container we cannot
|
|
/// question is not a container we may delete.
|
|
pub async fn unpushed_commits(&self, name: &str) -> UnpushedWork {
|
|
let script = "cd /mission/repo 2>/dev/null || exit 91; \
|
|
git rev-list --all --not --remotes 2>/dev/null | wc -l";
|
|
let argv = vec!["sh".to_string(), "-lc".to_string(), script.to_string()];
|
|
let out = match crate::container_exec::exec_as_root(
|
|
&self.docker,
|
|
name,
|
|
None,
|
|
&argv,
|
|
std::time::Duration::from_secs(30),
|
|
)
|
|
.await
|
|
{
|
|
Ok(o) => o,
|
|
Err(e) => {
|
|
return UnpushedWork::SomeOrUnknown(format!("could not ask git ({e})"));
|
|
}
|
|
};
|
|
if out.exit_code == Some(91) {
|
|
// No checkout at all — nothing to lose.
|
|
return UnpushedWork::None;
|
|
}
|
|
if out.exit_code != Some(0) {
|
|
return UnpushedWork::SomeOrUnknown(format!(
|
|
"git probe exited {:?}",
|
|
out.exit_code
|
|
));
|
|
}
|
|
match out.stdout.trim().parse::<u64>() {
|
|
Ok(0) => UnpushedWork::None,
|
|
Ok(n) => UnpushedWork::SomeOrUnknown(format!(
|
|
"{n} commit(s) in its checkout are on no remote"
|
|
)),
|
|
Err(_) => UnpushedWork::SomeOrUnknown(format!(
|
|
"unreadable git output {:?}",
|
|
out.stdout.trim()
|
|
)),
|
|
}
|
|
}
|
|
|
|
pub async fn container_exists(&self, mission_id: Uuid) -> bool {
|
|
self.docker
|
|
.inspect_container(&container_name(mission_id), None::<InspectContainerOptions>)
|
|
.await
|
|
.is_ok()
|
|
}
|
|
|
|
/// Force-remove the mission's runtime container AND its host workspace
|
|
/// dir (the `/mission/repo` checkout bind source). Idempotent: a missing
|
|
/// container or dir is not an error — this is called both by the terminal
|
|
/// sweeper and by mission delete, where the container may already be gone.
|
|
pub async fn teardown_container(&self, mission_id: Uuid) -> Result<(), String> {
|
|
let name = container_name(mission_id);
|
|
if let Err(e) = self
|
|
.docker
|
|
.remove_container(
|
|
&name,
|
|
Some(RemoveContainerOptions {
|
|
force: true,
|
|
..Default::default()
|
|
}),
|
|
)
|
|
.await
|
|
{
|
|
// 404 (already gone) is fine; anything else is worth surfacing.
|
|
let msg = e.to_string();
|
|
if !msg.contains("No such container") && !msg.contains("404") {
|
|
return Err(format!("remove mission runtime container: {e}"));
|
|
}
|
|
}
|
|
// Remove the per-mission workspace dir (repo checkout + scratch). This
|
|
// path is bind-mounted into cm-api, so we can reap it directly.
|
|
let mission_dir = format!("{}/{mission_id}", missions_host_root());
|
|
if let Err(e) = tokio::fs::remove_dir_all(&mission_dir).await {
|
|
match e.kind() {
|
|
std::io::ErrorKind::NotFound => {}
|
|
// The server runs as 65532 and cannot delete root-owned files.
|
|
// Almost everything under a mission is 65532 now, but the
|
|
// per-mission ZeroClaw daemon still runs as root and leaves ~26
|
|
// of its own files (`.claude.json`, session jsonl) behind after
|
|
// the seed copy has been chowned. Without this fallback those
|
|
// few files keep the whole directory alive forever — the
|
|
// cleanup-that-cannot-clean-up shape, at a scale small enough
|
|
// to go unnoticed for a long time.
|
|
std::io::ErrorKind::PermissionDenied => {
|
|
eprintln!(
|
|
"mission_runtime: {mission_dir} holds root-owned files — removing it from inside the runtime container"
|
|
);
|
|
let container = std::env::var("CLAWMATES_RUNTIME_CONTAINER")
|
|
.unwrap_or_else(|_| "clawmates-runtime".to_string());
|
|
crate::root_copy::purge(&container, std::path::Path::new(&mission_dir)).await;
|
|
if tokio::fs::metadata(&mission_dir).await.is_ok() {
|
|
eprintln!(
|
|
"mission_runtime: {mission_dir} SURVIVED the root purge — it will keep accumulating"
|
|
);
|
|
}
|
|
}
|
|
_ => eprintln!("mission_runtime: rm workspace dir {mission_dir}: {e}"),
|
|
}
|
|
}
|
|
Ok(())
|
|
}
|
|
}
|
|
|
|
/// Background sweeper: force-remove runtime containers for missions
|
|
/// that reached a terminal state ≥ `grace` ago. Keeps the container
|
|
/// around briefly after `completed`/`failed`/`cancelled` so the
|
|
/// operator can re-open the UI and pull the last checkpoint before
|
|
/// the daemon disappears. Runs on the same cadence as the phase
|
|
/// runner (10s) with a much longer per-mission grace.
|
|
pub fn spawn_sweeper(pool: sqlx::PgPool, grace: std::time::Duration) {
|
|
tokio::spawn(async move {
|
|
// Wait past the phase_runner boot so we don't fight over
|
|
// just-launched missions.
|
|
tokio::time::sleep(std::time::Duration::from_secs(30)).await;
|
|
let mut ticker = tokio::time::interval(std::time::Duration::from_secs(60));
|
|
ticker.tick().await;
|
|
loop {
|
|
ticker.tick().await;
|
|
if let Err(e) = sweep_once(&pool, grace).await {
|
|
eprintln!("mission_runtime::sweeper: sweep failed: {e}");
|
|
}
|
|
if let Err(e) = sweep_orphans(&pool, ORPHAN_GRACE).await {
|
|
eprintln!("mission_runtime::sweeper: orphan sweep failed: {e}");
|
|
}
|
|
}
|
|
});
|
|
}
|
|
|
|
/// How long a container with no mission row may sit before it is reaped.
|
|
///
|
|
/// Long, deliberately. The row-driven sweep above handles every container the
|
|
/// platform still knows about, so anything reaching this path is already
|
|
/// unexpected — and the one real orphan we have seen held ten unpushed commits.
|
|
/// A day of disk is cheaper than being wrong about that.
|
|
const ORPHAN_GRACE: std::time::Duration = std::time::Duration::from_secs(24 * 3600);
|
|
|
|
/// Reap `cm-runtime-mission-*` containers that no `missions` row points at.
|
|
///
|
|
/// [`sweep_once`] selects `FROM missions`, and `teardown_container` is only
|
|
/// ever called with an id that came from that query. So a container whose row
|
|
/// is gone is invisible to every reaper: nothing enumerates docker, nothing
|
|
/// errors, and the only symptom is disk.
|
|
///
|
|
/// Found on gw-04 2026-08-21 — a container `Up` for nine days holding 2.5G,
|
|
/// against a `missions` table with **zero rows**.
|
|
///
|
|
/// # It refuses to reap work that exists nowhere else
|
|
///
|
|
/// That container's checkout held **ten commits on a branch that had never
|
|
/// been pushed** (+3451/-30 across 30 files). A reaper that deleted on sight
|
|
/// would have destroyed all of it, silently, as its designed behaviour. So
|
|
/// before removing anything this asks the checkout whether it holds commits
|
|
/// that no remote has, and leaves the container alone — loudly, every tick —
|
|
/// when it does.
|
|
///
|
|
/// The check is deliberately one-sided in the safe direction: an inspection
|
|
/// that fails for any reason counts as "might hold work", never as "safe to
|
|
/// delete". Losing a day of disk to an unreadable container is recoverable;
|
|
/// the other way round is not.
|
|
pub async fn sweep_orphans(pool: &sqlx::PgPool, grace: std::time::Duration) -> Result<(), String> {
|
|
let Some(prov) = MissionRuntimeProvisioner::from_env() else {
|
|
return Ok(());
|
|
};
|
|
let names = prov.list_mission_containers().await?;
|
|
if names.is_empty() {
|
|
return Ok(());
|
|
}
|
|
for (name, created) in names {
|
|
let Some(id) = mission_id_from_container(&name) else {
|
|
continue;
|
|
};
|
|
// `WHERE id = $1` across every workspace on purpose: the question is
|
|
// whether ANY row still points at this container, not whether one the
|
|
// caller can see does.
|
|
let known: Option<(Uuid,)> = sqlx::query_as("SELECT id FROM missions WHERE id = $1")
|
|
.bind(id)
|
|
.fetch_optional(pool)
|
|
.await
|
|
.map_err(|e| format!("look up mission {id}: {e}"))?;
|
|
if known.is_some() {
|
|
continue;
|
|
}
|
|
match MissionRuntimeProvisioner::container_age(created) {
|
|
Some(age) if age < grace => continue,
|
|
None => continue,
|
|
Some(_) => {}
|
|
}
|
|
match prov.unpushed_commits(&name).await {
|
|
// The safe answer, and the one an error also produces.
|
|
UnpushedWork::SomeOrUnknown(why) => {
|
|
eprintln!(
|
|
"mission_runtime::orphans: {name} has no mission row and is older than \
|
|
the grace period, but it is NOT safe to reap: {why}. Recover the work \
|
|
(`git bundle create … origin/main..HEAD`, or push the branch) and then \
|
|
remove it by hand."
|
|
);
|
|
}
|
|
UnpushedWork::None => {
|
|
eprintln!(
|
|
"mission_runtime::orphans: reaping {name} — no mission row, older than \
|
|
the grace period, and its checkout holds nothing a remote does not"
|
|
);
|
|
if let Err(e) = prov.teardown_container(id).await {
|
|
eprintln!("mission_runtime::orphans: reap {name}: {e}");
|
|
}
|
|
}
|
|
}
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
/// Whether an orphan's checkout holds commits no remote has.
|
|
#[derive(Debug, PartialEq, Eq)]
|
|
pub enum UnpushedWork {
|
|
/// Every commit is reachable from a remote ref — nothing is lost.
|
|
None,
|
|
/// There IS unpushed work, or the question could not be answered. One
|
|
/// variant for both, because the reaper must treat them identically.
|
|
SomeOrUnknown(String),
|
|
}
|
|
|
|
/// The mission id encoded in a runtime container's name, if it is one.
|
|
///
|
|
/// The inverse of [`container_name`], which formats the uuid `simple` (no
|
|
/// dashes). Anything that does not parse is not ours and is left alone.
|
|
pub fn mission_id_from_container(name: &str) -> Option<Uuid> {
|
|
Uuid::parse_str(name.strip_prefix("cm-runtime-mission-")?).ok()
|
|
}
|
|
|
|
async fn sweep_once(pool: &sqlx::PgPool, grace: std::time::Duration) -> Result<(), String> {
|
|
use sqlx::Row;
|
|
let grace_secs = grace.as_secs() as f64;
|
|
let rows = sqlx::query(
|
|
"SELECT id, workspace_id, runtime_container_name
|
|
FROM missions
|
|
WHERE status IN ('completed', 'failed', 'cancelled')
|
|
AND runtime_endpoint IS NOT NULL
|
|
AND completed_at IS NOT NULL
|
|
AND completed_at < now() - make_interval(secs => $1::float)
|
|
LIMIT 20",
|
|
)
|
|
.bind(grace_secs)
|
|
.fetch_all(pool)
|
|
.await
|
|
.map_err(|e| format!("query terminal missions: {e}"))?;
|
|
if rows.is_empty() {
|
|
return Ok(());
|
|
}
|
|
let Some(prov) = MissionRuntimeProvisioner::from_env() else {
|
|
return Ok(());
|
|
};
|
|
for row in rows {
|
|
let id: Uuid = row.get("id");
|
|
let workspace_id: Uuid = row.get("workspace_id");
|
|
// Last chance. `teardown_container` deletes the checkout, so anything
|
|
// not captured by now is gone for good. The phase sweep should have
|
|
// handled this minutes ago; this covers the cases it cannot — a phase
|
|
// that ended `failed` rather than `completed`, or a capture that kept
|
|
// erroring until the grace window ran out.
|
|
if let Err(e) = capture_outstanding_phases(pool, id).await {
|
|
eprintln!("mission_runtime::sweeper: last-chance capture for {id}: {e}");
|
|
}
|
|
// Clear the binding only once the container is actually GONE, which is
|
|
// not the same as "teardown returned Ok".
|
|
//
|
|
// This used to clear unconditionally, reasoning that a not-found is
|
|
// expected and retrying forever is worse. But `teardown_container`
|
|
// already maps 404/"No such container" to `Ok`, so an `Err` here means
|
|
// a real docker failure — and this sweep selects on
|
|
// `runtime_endpoint IS NOT NULL`, so clearing after a failed teardown
|
|
// hides the surviving container from the only thing that would ever
|
|
// retry it. One transient docker error would strand a running
|
|
// container until somebody deleted the mission by hand.
|
|
//
|
|
// Asking docker whether the container still exists settles it without
|
|
// reintroducing the infinite retry: if it is gone, drop the binding
|
|
// however the call reported itself; if it survives, keep the binding so
|
|
// the next sweep tries again and the operator sees a recurring log
|
|
// rather than silence.
|
|
let teardown = prov.teardown_container(id).await;
|
|
if let Err(e) = &teardown {
|
|
if prov.container_exists(id).await {
|
|
eprintln!(
|
|
"mission_runtime::sweeper: teardown mission {id}: {e} — container still \
|
|
present, keeping the runtime binding so the next sweep retries"
|
|
);
|
|
continue;
|
|
}
|
|
eprintln!(
|
|
"mission_runtime::sweeper: teardown mission {id}: {e} — container is gone \
|
|
anyway, clearing the binding"
|
|
);
|
|
}
|
|
if let Err(e) =
|
|
cm_db::repo::missions::set_runtime_binding(pool, id, workspace_id, None, None, None)
|
|
.await
|
|
{
|
|
eprintln!("mission_runtime::sweeper: clear binding for {id}: {e}");
|
|
}
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
/// Capture any phase of `mission_id` that has a repo and no `code_diff` yet,
|
|
/// regardless of how the phase ended.
|
|
///
|
|
/// The phase sweep only captures `completed` phases. A mission that failed
|
|
/// mid-coding still has real work in its checkout, and deleting it
|
|
/// unexamined is how a debugging session loses the only evidence of what the
|
|
/// agents actually did.
|
|
async fn capture_outstanding_phases(pool: &sqlx::PgPool, mission_id: Uuid) -> Result<(), String> {
|
|
use sqlx::Row;
|
|
let rows = sqlx::query(
|
|
"SELECT mp.id
|
|
FROM mission_phases mp
|
|
JOIN missions m ON m.id = mp.mission_id
|
|
WHERE mp.mission_id = $1
|
|
AND m.repo_id IS NOT NULL
|
|
AND NOT EXISTS (
|
|
SELECT 1 FROM mission_artifacts a
|
|
WHERE a.mission_id = mp.mission_id
|
|
AND a.phase_id = mp.id
|
|
AND a.kind = 'code_diff'
|
|
)",
|
|
)
|
|
.bind(mission_id)
|
|
.fetch_all(pool)
|
|
.await
|
|
.map_err(|e| format!("select uncaptured phases: {e}"))?;
|
|
|
|
for row in rows {
|
|
let phase_id: Uuid = row.get("id");
|
|
if let Err(e) =
|
|
crate::mission_delivery::capture_phase_diff(pool, mission_id, phase_id).await
|
|
{
|
|
eprintln!(
|
|
"mission_runtime::sweeper: capture mission {mission_id} phase {phase_id}: {e}"
|
|
);
|
|
}
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
/// A canary backend runs the production path, so it needs the production
|
|
/// credential — and an UNKNOWN backend must still be refused.
|
|
///
|
|
/// The second half is the one that matters: `backend_can_run_a_mission` is
|
|
/// what stops a mission being launched against a rootfs no node has, and
|
|
/// widening the credential map is exactly how that guard gets softened by
|
|
/// accident.
|
|
#[test]
|
|
fn a_canary_backend_is_credentialed_but_an_unknown_one_is_not() {
|
|
assert!(backend_can_run_a_mission("canary-claude"));
|
|
assert!(backend_can_run_a_mission("claude"));
|
|
for unknown in ["test226", "definitely-not-built", "canary", "canary-"] {
|
|
assert!(
|
|
!backend_can_run_a_mission(unknown),
|
|
"{unknown} must be refused at launch"
|
|
);
|
|
}
|
|
}
|
|
|
|
use super::*;
|
|
|
|
/// The regression guard for the whole subscription feature.
|
|
///
|
|
/// Claude Code ranks `ANTHROPIC_API_KEY` above the subscription's OAuth
|
|
/// credential, so forwarding it into a container whose runtime is logged in
|
|
/// means every mission silently bills the API while looking correct. There
|
|
/// is no error to observe — only the invoice. If this test ever goes red,
|
|
/// the subscription path is off even though nothing appears broken.
|
|
/// The microVM path is subscription-only BY CONSTRUCTION — it does not read
|
|
/// `CLAWMATES_RUNTIME_AUTH` at all. If it did, the single unset variable on
|
|
/// gw-04 today would put an API key inside every VM, and Claude Code ranks
|
|
/// the key above the subscription token: it would work, and bill per-token
|
|
/// A local backend carries no secret, and no secret reaches it.
|
|
///
|
|
/// The credential map is the one place a backend can acquire a token, and
|
|
/// the point of routing `local-ornith` through it — rather than special-
|
|
/// casing it earlier — is that the "unknown backend cannot launch" rule
|
|
/// still holds. A typo like `local-ornit` must fail closed, not inherit the
|
|
/// subscription token on its way to somebody else's endpoint.
|
|
#[test]
|
|
fn a_local_backend_gets_a_placeholder_and_never_a_real_credential() {
|
|
let env = microvm_provider_env_from(Some("local-ornith"), |_| None)
|
|
.expect("a local backend needs nothing from the operator");
|
|
let token = env
|
|
.iter()
|
|
.find(|(k, _)| k == "ANTHROPIC_AUTH_TOKEN")
|
|
.map(|(_, v)| v.clone())
|
|
.expect("Claude Code refuses to start without a bearer");
|
|
assert!(
|
|
!token.starts_with("sk-"),
|
|
"a local backend must never be handed a real credential, got {token:?}"
|
|
);
|
|
assert!(!env.iter().any(|(k, _)| k == "ANTHROPIC_API_KEY"));
|
|
assert!(!env.iter().any(|(k, _)| k == "CLAUDE_CODE_OAUTH_TOKEN"));
|
|
|
|
// And a near-miss still cannot launch.
|
|
for typo in ["local-ornit", "ornith", "local", "local-ornith2"] {
|
|
assert!(
|
|
microvm_credential_for(Some(typo)).is_err(),
|
|
"{typo} must be refused, not resolved to something"
|
|
);
|
|
}
|
|
}
|
|
|
|
/// against a plan already paid for, with no symptom but the invoice.
|
|
#[test]
|
|
fn a_microvm_never_receives_an_anthropic_api_key() {
|
|
// Everything set, including the key the container path would forward.
|
|
let env = microvm_provider_env_from(Some("claude"), |k| Some(format!("value-of-{k}")))
|
|
.expect("a set token should launch");
|
|
let keys: Vec<&str> = env.iter().map(|(k, _)| k.as_str()).collect();
|
|
assert!(
|
|
!keys.contains(&"ANTHROPIC_API_KEY"),
|
|
"the API key outranks the subscription token; forwarding it bills the \
|
|
API silently. Forwarded: {keys:?}"
|
|
);
|
|
assert!(
|
|
keys.contains(&"CLAUDE_CODE_OAUTH_TOKEN"),
|
|
"the subscription credential must travel: {keys:?}"
|
|
);
|
|
}
|
|
|
|
/// No token is a refusal to launch, not an empty environment. A VM without a
|
|
/// credential does not error — `claude -p` hangs, which reads as a phase
|
|
/// stuck at `running` with nothing in the logs.
|
|
#[test]
|
|
fn a_microvm_without_a_subscription_token_refuses_to_launch() {
|
|
let r = microvm_provider_env_from(Some("claude"), |k| {
|
|
// The API key is present; the subscription token is not. This must
|
|
// NOT be treated as "we have a credential".
|
|
(k == "ANTHROPIC_API_KEY").then(|| "sk-ant-whatever".to_string())
|
|
});
|
|
let err = r.expect_err("no subscription token must refuse the launch");
|
|
assert!(err.contains("CLAUDE_CODE_OAUTH_TOKEN"), "{err}");
|
|
// A blank token is the same as no token.
|
|
assert!(microvm_provider_env_from(Some("claude"), |k| {
|
|
(k == "CLAUDE_CODE_OAUTH_TOKEN").then(|| " ".to_string())
|
|
})
|
|
.is_err());
|
|
}
|
|
|
|
/// The credential each provider gets, and — the part that matters — the one
|
|
/// it must never get. A GLM VM handed `CLAUDE_CODE_OAUTH_TOKEN` would send an
|
|
/// Anthropic subscription credential to z.ai, verbatim, on the first turn.
|
|
/// The two providers' secrets must not cross.
|
|
#[test]
|
|
fn each_provider_gets_its_own_credential_and_only_its_own() {
|
|
let env = microvm_provider_env_from(Some("glm"), |k| Some(format!("value-of-{k}")))
|
|
.expect("glm has a settled contract");
|
|
let by_key: std::collections::HashMap<_, _> = env.into_iter().collect();
|
|
// Claude Code reads a custom-endpoint credential as ANTHROPIC_AUTH_TOKEN,
|
|
// and the value is the SERVER's ZAI_API_KEY — two different names, which
|
|
// is exactly why the contract carries both.
|
|
assert_eq!(
|
|
by_key.get("ANTHROPIC_AUTH_TOKEN").map(String::as_str),
|
|
Some("value-of-ZAI_API_KEY"),
|
|
"{by_key:?}"
|
|
);
|
|
for forbidden in ["CLAUDE_CODE_OAUTH_TOKEN", "ANTHROPIC_API_KEY"] {
|
|
assert!(
|
|
!by_key.contains_key(forbidden),
|
|
"a GLM VM must never carry {forbidden} — it would be sent to z.ai: {by_key:?}"
|
|
);
|
|
}
|
|
|
|
// Kimi shares the target var with GLM and NOTHING else: same protocol,
|
|
// different endpoint baked into a different image. What must not happen
|
|
// is one provider's key travelling under the other's name.
|
|
let kimi = microvm_provider_env_from(Some("kimi"), |k| Some(format!("value-of-{k}")))
|
|
.expect("kimi has a settled contract");
|
|
let kimi: std::collections::HashMap<_, _> = kimi.into_iter().collect();
|
|
assert_eq!(
|
|
kimi.get("ANTHROPIC_AUTH_TOKEN").map(String::as_str),
|
|
Some("value-of-KIMI_API_KEY"),
|
|
"{kimi:?}"
|
|
);
|
|
assert!(!kimi.contains_key("CLAUDE_CODE_OAUTH_TOKEN"), "{kimi:?}");
|
|
|
|
// And the reverse: a claude VM carries no z.ai key.
|
|
let env = microvm_provider_env_from(Some("claude"), |k| Some(format!("value-of-{k}")))
|
|
.expect("claude");
|
|
let keys: Vec<&str> = env.iter().map(|(k, _)| k.as_str()).collect();
|
|
assert!(!keys.contains(&"ANTHROPIC_AUTH_TOKEN"), "{keys:?}");
|
|
assert!(!keys.contains(&"ZAI_API_KEY"), "{keys:?}");
|
|
}
|
|
|
|
/// A GLM mission with no z.ai key refuses to launch, and says which var is
|
|
/// missing. It must NOT fall back to the Anthropic token that IS set.
|
|
#[test]
|
|
fn a_glm_mission_without_a_zai_key_refuses_rather_than_falls_back() {
|
|
let err = microvm_provider_env_from(Some("glm"), |k| {
|
|
(k == "CLAUDE_CODE_OAUTH_TOKEN").then(|| "sub-token".to_string())
|
|
})
|
|
.expect_err("no z.ai key must refuse the launch");
|
|
assert!(err.contains("ZAI_API_KEY"), "{err}");
|
|
}
|
|
|
|
/// A backend whose credential contract is not settled is refused, rather
|
|
/// than handed another provider's credential to send at its endpoint.
|
|
#[test]
|
|
fn an_undefined_backend_is_refused_rather_than_given_the_anthropic_token() {
|
|
// `kimi` has since moved out of this list — its URL was measured.
|
|
for b in [Some("something-new"), Some("agent-terminal")] {
|
|
let r = microvm_provider_env_from(b, |_| Some("set".into()));
|
|
assert!(r.is_err(), "backend {b:?} should be refused: {r:?}");
|
|
}
|
|
// The default image is the claude one, and does have a contract.
|
|
for b in [None, Some(""), Some("default"), Some("claude")] {
|
|
assert!(
|
|
microvm_provider_env_from(b, |_| Some("set".into())).is_ok(),
|
|
"backend {b:?}"
|
|
);
|
|
}
|
|
}
|
|
|
|
/// The container path still honours the operator's mode, and reads values
|
|
/// the same way. Only the microVM path is pinned.
|
|
#[test]
|
|
fn the_container_path_forwards_its_configured_mode() {
|
|
for auth in [RuntimeAuth::ApiKey, RuntimeAuth::Subscription] {
|
|
let keys = forwarded_provider_keys(auth);
|
|
// Every key set in the environment, and nothing else.
|
|
let pairs = provider_env_from(auth, |k| Some(format!("value-of-{k}")));
|
|
let got: Vec<&str> = pairs.iter().map(|(k, _)| k.as_str()).collect();
|
|
assert_eq!(got, keys, "{}", auth.as_str());
|
|
for (k, v) in &pairs {
|
|
assert_eq!(v, &format!("value-of-{k}"), "value must pass through");
|
|
}
|
|
}
|
|
}
|
|
|
|
/// An unset or blank credential is ABSENT, not empty. On the microVM path an
|
|
/// empty string would make `claude` believe it has a credential and fail
|
|
/// authentication, instead of reporting that it has none.
|
|
#[test]
|
|
fn a_blank_credential_is_omitted_rather_than_forwarded_empty() {
|
|
let pairs = provider_env_from(RuntimeAuth::Subscription, |k| match k {
|
|
"CLAUDE_CODE_OAUTH_TOKEN" => Some(" ".into()),
|
|
"OPENAI_API_KEY" => Some(String::new()),
|
|
"GROQ_API_KEY" => Some("real".into()),
|
|
_ => None,
|
|
});
|
|
assert_eq!(
|
|
pairs,
|
|
vec![("GROQ_API_KEY".to_string(), "real".to_string())]
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn subscription_mode_withholds_the_anthropic_api_key() {
|
|
let keys = forwarded_provider_keys(RuntimeAuth::Subscription);
|
|
assert!(
|
|
!keys.contains(&"ANTHROPIC_API_KEY"),
|
|
"ANTHROPIC_API_KEY outranks the subscription credential; forwarding \
|
|
it silently bills the API. Forwarded: {keys:?}"
|
|
);
|
|
// Unrelated providers have no subscription equivalent and must survive.
|
|
for k in [
|
|
"GROQ_API_KEY",
|
|
"OPENAI_API_KEY",
|
|
"ZAI_API_KEY",
|
|
"KIMI_API_KEY",
|
|
] {
|
|
assert!(keys.contains(&k), "{k} should still be forwarded");
|
|
}
|
|
// And the subscription credential MUST travel. A mission container
|
|
// has its own data dir, so unlike the shared runtime it has no
|
|
// persisted `claude /login` to fall back on. Without this the CLI
|
|
// has no credential and simply hangs — a phase stuck at `running`
|
|
// with nothing in the logs, which is exactly how this was found.
|
|
assert!(
|
|
keys.contains(&"CLAUDE_CODE_OAUTH_TOKEN"),
|
|
"subscription mode must forward the token; without it `claude -p` \
|
|
hangs with no credential. Forwarded: {keys:?}"
|
|
);
|
|
}
|
|
|
|
/// The two credentials must never travel together: Claude Code would pick
|
|
/// the API key and bill it while the deployment believes it is on the
|
|
/// subscription.
|
|
#[test]
|
|
fn the_two_anthropic_credentials_are_mutually_exclusive() {
|
|
for mode in [RuntimeAuth::ApiKey, RuntimeAuth::Subscription] {
|
|
let keys = forwarded_provider_keys(mode);
|
|
let both =
|
|
keys.contains(&"ANTHROPIC_API_KEY") && keys.contains(&"CLAUDE_CODE_OAUTH_TOKEN");
|
|
assert!(!both, "{mode:?} forwards both credentials: {keys:?}");
|
|
}
|
|
}
|
|
|
|
/// Default behaviour is unchanged, so a deployment that never opts in keeps
|
|
/// working exactly as before.
|
|
#[test]
|
|
fn api_key_mode_forwards_everything() {
|
|
let keys = forwarded_provider_keys(RuntimeAuth::ApiKey);
|
|
for k in ["ANTHROPIC_API_KEY", "GROQ_API_KEY", "OPENAI_API_KEY"] {
|
|
assert!(keys.contains(&k), "{k} should be forwarded in api_key mode");
|
|
}
|
|
// Gemini is stripped from the platform entirely: no provider row, no
|
|
// selector entry, and no key forwarded to agent containers. Asserted so
|
|
// a future "just add it back to the list" restores the dependency
|
|
// visibly rather than by accident.
|
|
for mode in [RuntimeAuth::ApiKey, RuntimeAuth::Subscription] {
|
|
assert!(
|
|
!forwarded_provider_keys(mode).contains(&"GEMINI_API_KEY"),
|
|
"GEMINI_API_KEY must not be forwarded in {mode:?}"
|
|
);
|
|
}
|
|
assert!(
|
|
!keys.contains(&"CLAUDE_CODE_OAUTH_TOKEN"),
|
|
"api_key mode must not also ship the subscription token"
|
|
);
|
|
}
|
|
|
|
/// An unset or misspelled value must fall back to the *existing* behaviour.
|
|
/// Defaulting to subscription on a typo would leave missions with no
|
|
/// credential at all.
|
|
#[test]
|
|
fn auth_mode_defaults_to_api_key() {
|
|
// Can't safely mutate process env in a parallel test binary, so assert
|
|
// the mapping the parser implements rather than the env read itself.
|
|
for (input, expected) in [
|
|
("subscription", RuntimeAuth::Subscription),
|
|
("SUBSCRIPTION", RuntimeAuth::Subscription),
|
|
("api_key", RuntimeAuth::ApiKey),
|
|
("", RuntimeAuth::ApiKey),
|
|
("nonsense", RuntimeAuth::ApiKey),
|
|
] {
|
|
let got = match input.trim().to_ascii_lowercase().as_str() {
|
|
"subscription" => RuntimeAuth::Subscription,
|
|
"" | "api_key" => RuntimeAuth::ApiKey,
|
|
_ => RuntimeAuth::ApiKey,
|
|
};
|
|
assert_eq!(got, expected, "input {input:?}");
|
|
}
|
|
}
|
|
|
|
/// The name→id round trip the orphan sweep depends on.
|
|
///
|
|
/// If this is wrong the sweep either skips every orphan (harmless) or
|
|
/// resolves a container to the WRONG mission id and asks the database
|
|
/// about a mission that does exist — reading "still live, leave it" for a
|
|
/// container that is not. Cheap to get right, expensive to get wrong.
|
|
#[test]
|
|
fn a_container_name_round_trips_to_its_mission() {
|
|
let id = Uuid::now_v7();
|
|
assert_eq!(mission_id_from_container(&container_name(id)), Some(id));
|
|
// Not ours, and not a panic.
|
|
assert_eq!(mission_id_from_container("clawmates_server_1"), None);
|
|
assert_eq!(mission_id_from_container("cm-runtime-mission-nonsense"), None);
|
|
assert_eq!(mission_id_from_container("cm-sandbox-abc"), None);
|
|
}
|
|
|
|
/// A container docker will not date must not be reaped.
|
|
#[test]
|
|
fn an_undatable_container_has_no_age() {
|
|
assert_eq!(MissionRuntimeProvisioner::container_age(None), None);
|
|
let now = std::time::SystemTime::now()
|
|
.duration_since(std::time::UNIX_EPOCH)
|
|
.unwrap()
|
|
.as_secs() as i64;
|
|
let age = MissionRuntimeProvisioner::container_age(Some(now - 3600)).expect("age");
|
|
assert!(age.as_secs() >= 3500 && age.as_secs() <= 3700, "{age:?}");
|
|
// A clock skew that puts creation in the future must not underflow into
|
|
// a colossal age that reads as "long past the grace period".
|
|
assert_eq!(MissionRuntimeProvisioner::container_age(Some(now + 600)), None);
|
|
}
|
|
|
|
/// The grace period is long, and that is the point.
|
|
#[test]
|
|
fn the_orphan_grace_is_generous() {
|
|
assert!(
|
|
ORPHAN_GRACE >= std::time::Duration::from_secs(12 * 3600),
|
|
"the row-driven sweep already handles everything the platform knows \
|
|
about, so anything reaching the orphan path is unexpected — and the \
|
|
one real orphan held ten unpushed commits"
|
|
);
|
|
}
|
|
|
|
const SAMPLE_CONFIG: &str = r#"# top comment
|
|
[agents.claw_a]
|
|
model_provider = "anthropic.default"
|
|
risk_profile = "coding_readwrite"
|
|
mcp_bundles = ["clawmates_door"]
|
|
|
|
[agents.claw_a.workspace]
|
|
unrestricted_filesystem = false
|
|
|
|
[agents.claw_b]
|
|
risk_profile = "research_readonly"
|
|
|
|
[risk_profiles.coding_readwrite]
|
|
# keep this comment
|
|
allowed_tools = ["file_read", "file_edit"]
|
|
"#;
|
|
|
|
#[test]
|
|
fn stamp_pins_path_and_preserves_existing_workspace_fields() {
|
|
let (out, n) =
|
|
stamp_workspace_paths(SAMPLE_CONFIG, &["claw_a".to_string()], "/mission/repo").unwrap();
|
|
assert_eq!(n, 1);
|
|
assert!(out.contains(r#"path = "/mission/repo""#));
|
|
// The sibling field in the same table is untouched.
|
|
assert!(out.contains("unrestricted_filesystem = false"));
|
|
// Comments and unrelated sections survive the round-trip.
|
|
assert!(out.contains("# top comment"));
|
|
assert!(out.contains("# keep this comment"));
|
|
assert!(out.contains("[risk_profiles.coding_readwrite]"));
|
|
}
|
|
|
|
#[test]
|
|
fn stamp_creates_workspace_table_when_absent() {
|
|
let (out, n) =
|
|
stamp_workspace_paths(SAMPLE_CONFIG, &["claw_b".to_string()], "/mission/repo").unwrap();
|
|
assert_eq!(n, 1);
|
|
// claw_b had no [workspace] table; it now has one with the path.
|
|
let doc = out.parse::<toml_edit::DocumentMut>().unwrap();
|
|
assert_eq!(
|
|
doc["agents"]["claw_b"]["workspace"]["path"].as_str(),
|
|
Some("/mission/repo")
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn stamp_skips_absent_aliases_without_fabricating_them() {
|
|
let (out, n) = stamp_workspace_paths(
|
|
SAMPLE_CONFIG,
|
|
&["claw_missing".to_string()],
|
|
"/mission/repo",
|
|
)
|
|
.unwrap();
|
|
assert_eq!(n, 0);
|
|
assert!(!out.contains("claw_missing"));
|
|
}
|
|
|
|
#[test]
|
|
fn stamp_is_idempotent_overwriting_a_prior_path() {
|
|
let once = stamp_workspace_paths(SAMPLE_CONFIG, &["claw_a".to_string()], "/old/path")
|
|
.unwrap()
|
|
.0;
|
|
let (twice, n) =
|
|
stamp_workspace_paths(&once, &["claw_a".to_string()], "/mission/repo").unwrap();
|
|
assert_eq!(n, 1);
|
|
assert!(twice.contains(r#"path = "/mission/repo""#));
|
|
assert!(!twice.contains("/old/path"));
|
|
// Exactly one path key for claw_a (no duplication).
|
|
assert_eq!(twice.matches("path = ").count(), 1);
|
|
}
|
|
|
|
#[test]
|
|
fn container_name_is_stable_and_prefixed() {
|
|
let id = Uuid::parse_str("019f84a0-f2a2-7bd0-be9b-86713ec73693").unwrap();
|
|
let name = container_name(id);
|
|
assert_eq!(name, "cm-runtime-mission-019f84a0f2a27bd0be9b86713ec73693");
|
|
// Determinism: same input → same output.
|
|
assert_eq!(name, container_name(id));
|
|
}
|
|
|
|
#[test]
|
|
fn container_names_differ_across_missions() {
|
|
// Two UUIDs differing only in the trailing hex char — full-uuid
|
|
// naming must distinguish them (UUIDv7's timestamp shares
|
|
// leading bytes for missions minted in the same second).
|
|
let a = container_name(Uuid::parse_str("019f84a0-f2a2-7bd0-be9b-86713ec73693").unwrap());
|
|
let b = container_name(Uuid::parse_str("019f84a0-f2a2-7bd0-be9b-86713ec7369f").unwrap());
|
|
assert_ne!(a, b);
|
|
}
|
|
|
|
#[test]
|
|
fn endpoint_url_uses_gateway_port() {
|
|
let url = endpoint_url("cm-runtime-mission-abc");
|
|
assert_eq!(url, "http://cm-runtime-mission-abc:42617");
|
|
}
|
|
|
|
/// The seed data path must be per-mission, not the shared seed dir.
|
|
///
|
|
/// Every mission container used to bind the SAME host directory as
|
|
/// `/zeroclaw-data`. It holds `config.toml`, which carries the §15 door
|
|
/// bearer token, plus `sessions.db`/`devices.db`. Sharing it meant one
|
|
/// mission could read another's credential, and anything written there
|
|
/// was inherited by every later mission — `teardown_container` only
|
|
/// removes `/var/lib/clawmates-missions/{id}`, so the shared dir was
|
|
/// never cleaned.
|
|
///
|
|
/// The seed copy reads as root and leaves the result to 65532.
|
|
///
|
|
/// Both halves are load-bearing and they pull in opposite directions. The
|
|
/// seed dir is root-owned with parts at mode 0600, so a copier running as
|
|
/// 65532 cannot read it — that was tried, `cp` failed on the first
|
|
/// unreadable entry, `set -e` abandoned the rest, and the mission booted
|
|
/// with `.zeroclaw` and nothing else. But leaving the RESULT root-owned is
|
|
/// what put ~3200 undeletable files per mission into the missions tree.
|
|
#[test]
|
|
fn the_seed_copy_hands_its_result_to_the_mission_uid() {
|
|
let script = copy_script();
|
|
assert!(
|
|
script.contains(&format!("chown -R {MISSION_UID} /dst")),
|
|
"the copied seed must end up owned by the mission uid:\n{script}"
|
|
);
|
|
// Every seeded path is attempted, and absence is tolerated — a fresh
|
|
// deployment genuinely has no `.kimi-code` yet.
|
|
for p in SEEDED_PATHS {
|
|
assert!(
|
|
script.contains(&format!("if [ -e '/seed/{p}' ]")),
|
|
"{p} is not guarded by an existence check"
|
|
);
|
|
}
|
|
// The chown must come AFTER the copies, or it chowns an empty dir.
|
|
let last_cp = script.rfind("cp -a").expect("at least one copy");
|
|
let chown = script.find("chown -R").expect("a chown");
|
|
assert!(chown > last_cp, "chown must run after the copies");
|
|
}
|
|
|
|
/// The seed data path must be per-mission, not the shared seed dir.
|
|
///
|
|
/// Every mission container used to bind the SAME host directory as
|
|
/// `/zeroclaw-data`. It holds `config.toml`, which carries the §15 door
|
|
/// bearer token, plus `sessions.db`/`devices.db`. Sharing it meant one
|
|
/// mission could read another's credential, and anything written there
|
|
/// was inherited by every later mission — `teardown_container` only
|
|
/// removes `/var/lib/clawmates-missions/{id}`, so the shared dir was
|
|
/// never cleaned.
|
|
///
|
|
/// Asserting the path shape is what keeps this from silently regressing:
|
|
/// a future edit that points the mount back at the seed dir restores the
|
|
/// credential sharing with no other visible symptom.
|
|
///
|
|
/// This test did not RUN for some time: an edit stranded its `#[test]`
|
|
/// above the neighbouring function, leaving two attributes there and none
|
|
/// here. rustc said so twice — "duplicated attribute" and "function is
|
|
/// never used" — and both read as ordinary warnings in a noisy build.
|
|
#[test]
|
|
fn runtime_data_is_scoped_to_one_mission() {
|
|
let a = Uuid::now_v7();
|
|
let b = Uuid::now_v7();
|
|
let root = missions_host_root();
|
|
let path = |id: Uuid| format!("{root}/{id}/runtime-data");
|
|
|
|
assert_ne!(path(a), path(b), "two missions must not share runtime data");
|
|
assert!(
|
|
path(a).starts_with(&format!("{root}/{a}")),
|
|
"runtime data must live under the mission dir so teardown removes it"
|
|
);
|
|
assert_ne!(
|
|
path(a),
|
|
DEFAULT_SEED_DIR,
|
|
"the mount must never be the shared seed dir itself"
|
|
);
|
|
assert!(
|
|
!path(a).starts_with(DEFAULT_SEED_DIR),
|
|
"runtime data must not live inside the shared seed dir either"
|
|
);
|
|
}
|
|
|
|
/// The copy must be an allow-list, and must include the secret-bearing
|
|
/// paths while excluding the expensive ones.
|
|
///
|
|
/// Measured on gw-04: the seed dir is 1.7 GB, of which 1.5 GB is a
|
|
/// vestigial `.rustup` that is not even on the container's PATH (the
|
|
/// image ships Rust at /usr/local/cargo). Copying everything per mission
|
|
/// would cost tens of seconds and ~17 GB across ten concurrent missions —
|
|
/// which is what the first version of this did.
|
|
#[test]
|
|
fn the_seed_copy_takes_secrets_and_skips_caches() {
|
|
// The two paths that carry the door bearer token MUST be copied, or
|
|
// this whole change accomplishes nothing.
|
|
assert!(SEEDED_PATHS.contains(&".zeroclaw"));
|
|
assert!(SEEDED_PATHS.contains(&"clawmates-mcp.json"));
|
|
// Claude Code's credentials and state.
|
|
assert!(SEEDED_PATHS.contains(&".claude"));
|
|
|
|
// The expensive, secret-free ones must NOT be.
|
|
for cache in [".rustup", ".npm", ".cargo", ".cache"] {
|
|
assert!(
|
|
!SEEDED_PATHS.contains(&cache),
|
|
"{cache} is a cache and must not be copied per mission"
|
|
);
|
|
}
|
|
|
|
let script = copy_script();
|
|
// A missing entry is normal on a fresh deployment (no .kimi-code
|
|
// until Kimi is first used) and must not fail the copy.
|
|
assert!(
|
|
script.contains("if [ -e "),
|
|
"absent paths must be tolerated"
|
|
);
|
|
assert!(script.contains("/seed/.zeroclaw"));
|
|
assert!(!script.contains("/seed/.rustup"));
|
|
for p in SEEDED_PATHS {
|
|
assert!(script.contains(p), "{p} missing from the copy script");
|
|
}
|
|
}
|
|
}
|