feat(microvm): Claude Code 2.1.276 rootfs pins, and a VM run that says what it ran
deploy / test (push) Successful in 5m18s
deploy / build (push) Successful in 5m54s

Every rootfs on the fleet had sat on Claude Code 2.1.223–2.1.226 since August
while the container tier moved to 2.1.276, and nothing recorded either. GLM
and Kimi exist only as microVM backends, so "have we upgraded GLM and Kimi"
is this change and the rebuild it drives.

Pins. All four agent-* images pin 2.1.276 — as separate ARGs, since Docker has
no include and each file has to stay reproducible alone — and
scripts/fc-build-rootfs.sh refuses to build if they disagree, naming the odd
one out. They had already drifted (claude 226, the rest 223) under comments
saying "same version on purpose". Between 2.1.226 and 2.1.276, 2.1.265 and
2.1.275 each broke every turn on ANTHROPIC_BASE_URL endpoints, which is how
glm and kimi reach `claude` inside a VM; the container-tier verification never
exercised that path, so the VM runs on those backends are the real test.

Provenance. `VmOutcome` carries the rootfs the node reported booting and the
guest's own `claude --version`; `launch_microvm_phase` persists both as
`checkpoint.vm` beside `records` (the two readers parse only `records`) and
names them in its log line. "Which image and CLI did this mission run on" is
a query now.

Independence. `evaluator` derived the implementer family from a constant
`"anthropic"`, true while every backend was Claude on Anthropic. With glm and
kimi rootfs it made a glm mission judged by glm:glm-5.3 read as
`independent = true` — the one claim that path exists to make honestly.
`implementer_family(missions.backend)` mirrors `microvm_credential_for`; the
subscription judge is now independent exactly when the agent did NOT run on
Anthropic.

Harness. `verify-mission-delivery.sh glm|kimi` run the microvm scenario on
each backend and add the proof the mission itself cannot give: the placed
node's journal must show the VM dialling that provider's host, never being
denied it, and dialling nothing else but the forge — a model's self-report is
measured worthless here. `assert_cli_version` reads checkpoint.vm. The stale
scratch-repo default (dead since the 09-14 wipe) is the re-synced id.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
This commit is contained in:
Omar Sobh
2026-09-18 20:16:06 -05:00
co-authored by Claude Opus 5
parent 3755699b41
commit 794f2124bc
10 changed files with 259 additions and 24 deletions
+64 -14
View File
@@ -291,13 +291,27 @@ fn names_a_provider(spec: &str) -> bool {
spec.contains(':') spec.contains(':')
} }
/// The provider family the mission's agent ran on. /// The provider family the mission's agent ran on, from `missions.backend`.
/// ///
/// Today every mission backend is Claude Code (`agent-claude`), including the /// Mirrors `mission_runtime::microvm_credential_for`: the backend decides which
/// microVM path. When `agent-glm` / `agent-kimi` images exist this should read /// credential the guest gets and which host its egress proxy allows, so it is
/// `missions.backend`; until then, hardcoding the truth is better than plumbing a /// the one honest source for "who answered the agent's turns". This was a
/// parameter that only ever has one value. /// hardcoded `"anthropic"` while every backend was Claude Code on Anthropic;
const IMPLEMENTER_FAMILY: &str = "anthropic"; /// once `glm` and `kimi` rootfs existed that constant made a glm-backend
/// mission judged by `glm:glm-5.3` read as `independent = true`, which is the
/// one claim this path exists to make honestly.
///
/// `unknown` for anything unrecognised, for the same reason `provider_family`
/// says it: a guess in either direction misstates independence.
pub fn implementer_family(backend: Option<&str>) -> &'static str {
match backend.map(str::trim) {
None | Some("") | Some("default") | Some("claude") | Some("canary-claude") => "anthropic",
Some("glm") => "glm",
Some("kimi") => "kimi",
Some("local-ornith") => "local",
Some(_) => "unknown",
}
}
/// Which validator spec applies, given the mission's own setting and the /// Which validator spec applies, given the mission's own setting and the
/// deployment default. /// deployment default.
@@ -333,9 +347,23 @@ fn resolve_validator_spec(mission: Option<&str>, deployment: Option<&str>) -> Op
/// back Claude while the caller believed it had asked for GLM. The fallback is /// back Claude while the caller believed it had asked for GLM. The fallback is
/// detectable because the returned model still carries the `name:` prefix, and it /// detectable because the returned model still carries the `name:` prefix, and it
/// is checked here rather than trusted. /// is checked here rather than trusted.
/// The mission's implementer family, read from its row. `anthropic` when the
/// row cannot be read — the pre-2026-09-18 behaviour, and the family every
/// backend actually had until then.
async fn mission_implementer_family(runtime: &cm_runtime::Runtime, mission_id: Uuid) -> &'static str {
let backend: Option<String> = sqlx::query_scalar("SELECT backend FROM missions WHERE id = $1")
.bind(mission_id)
.fetch_optional(runtime.pool())
.await
.unwrap_or(None)
.flatten();
implementer_family(backend.as_deref())
}
async fn cross_provider_judge( async fn cross_provider_judge(
runtime: &cm_runtime::Runtime, runtime: &cm_runtime::Runtime,
mission_id: Uuid, mission_id: Uuid,
implementer: &str,
) -> Option<(std::sync::Arc<dyn cm_llm::LlmProvider>, String)> { ) -> Option<(std::sync::Arc<dyn cm_llm::LlmProvider>, String)> {
// Read per mission rather than widening `Mission` for one caller. One extra // Read per mission rather than widening `Mission` for one caller. One extra
// query per evaluation, against a path that is about to make a model call. // query per evaluation, against a path that is about to make a model call.
@@ -352,10 +380,10 @@ async fn cross_provider_judge(
)?; )?;
let spec = spec.as_str(); let spec = spec.as_str();
let family = provider_family(spec); let family = provider_family(spec);
if family == IMPLEMENTER_FAMILY { if family == implementer {
eprintln!( eprintln!(
"evaluator: CLAWMATES_VALIDATOR_MODEL={spec} is the same provider family as the \ "evaluator: CLAWMATES_VALIDATOR_MODEL={spec} is the same provider family as the \
agent ({IMPLEMENTER_FAMILY}) — that is not an independent check, ignoring it" agent ({implementer}) — that is not an independent check, ignoring it"
); );
return None; return None;
} }
@@ -474,7 +502,8 @@ pub async fn evaluate(
// failure — the model that talked itself into a shortcut is the one disposed // failure — the model that talked itself into a shortcut is the one disposed
// to accept it — and the tool loop is what makes the check evidence rather // to accept it — and the tool loop is what makes the check evidence rather
// than opinion, so an independent judge must have it too. // than opinion, so an independent judge must have it too.
if let Some((provider, model)) = cross_provider_judge(runtime, mission_id).await { let implementer = mission_implementer_family(runtime, mission_id).await;
if let Some((provider, model)) = cross_provider_judge(runtime, mission_id, implementer).await {
let system = match &sandbox { let system = match &sandbox {
Some(_) => format!("{EVAL_SYSTEM_VERIFYING}\n\n{VERDICT_CONTRACT}"), Some(_) => format!("{EVAL_SYSTEM_VERIFYING}\n\n{VERDICT_CONTRACT}"),
None => format!("{EVAL_SYSTEM_EVIDENCE_ONLY}\n\n{VERDICT_CONTRACT}"), None => format!("{EVAL_SYSTEM_EVIDENCE_ONLY}\n\n{VERDICT_CONTRACT}"),
@@ -535,7 +564,9 @@ pub async fn evaluate(
let outcome = let outcome =
judge_with_tools(&provider, &system, &user, &model, sandbox.as_ref(), &mut usage) judge_with_tools(&provider, &system, &user, &model, sandbox.as_ref(), &mut usage)
.await; .await;
// Same family as the agent; `independent` stays false below. // Independent exactly when the agent did NOT run on Anthropic: a
// glm- or kimi-backend mission judged by the Anthropic subscription
// is a cross-provider check, and a claude-backend one is not.
let mut v = match outcome { let mut v = match outcome {
Err(e) => Verdict::not_met( Err(e) => Verdict::not_met(
&model, &model,
@@ -550,6 +581,7 @@ pub async fn evaluate(
} }
}; };
v.usage = usage; v.usage = usage;
v.independent = implementer != "anthropic";
return v; return v;
} }
@@ -1088,7 +1120,23 @@ mod cross_provider_tests {
#[test] #[test]
fn an_unrecognised_model_is_not_assumed_to_be_ours() { fn an_unrecognised_model_is_not_assumed_to_be_ours() {
assert_eq!(provider_family("some-new-model-v9"), "unknown"); assert_eq!(provider_family("some-new-model-v9"), "unknown");
assert_ne!(provider_family("some-new-model-v9"), IMPLEMENTER_FAMILY); assert_ne!(provider_family("some-new-model-v9"), implementer_family(None));
}
/// The implementer family comes from the mission's backend, and the two
/// readers have to agree on the spelling of a family or a glm mission
/// judged by glm reads as independent — which it did, while this was a
/// constant.
#[test]
fn the_implementer_family_follows_the_backend() {
assert_eq!(implementer_family(None), "anthropic");
for b in ["", "default", "claude", "canary-claude"] {
assert_eq!(implementer_family(Some(b)), "anthropic", "{b}");
}
assert_eq!(implementer_family(Some("glm")), provider_family("glm:glm-5.3"));
assert_eq!(implementer_family(Some("kimi")), provider_family("kimi:kimi-k2"));
assert_ne!(implementer_family(Some("claude")), provider_family("glm:glm-5.3"));
assert_eq!(implementer_family(Some("something-else")), "unknown");
} }
/// The whole point: a judge in the implementer's own family is not /// The whole point: a judge in the implementer's own family is not
@@ -1098,13 +1146,15 @@ mod cross_provider_tests {
for spec in ["claude-opus-4-8", "runtime:claw_x", "sonnet"] { for spec in ["claude-opus-4-8", "runtime:claw_x", "sonnet"] {
assert_eq!( assert_eq!(
provider_family(spec), provider_family(spec),
IMPLEMENTER_FAMILY, implementer_family(Some("claude")),
"{spec} would have to be rejected as a validator" "{spec} would have to be rejected as a validator of a claude mission"
); );
} }
for spec in ["glm:glm-4.7", "kimi:kimi-k2"] { for spec in ["glm:glm-4.7", "kimi:kimi-k2"] {
assert_ne!(provider_family(spec), IMPLEMENTER_FAMILY, "{spec}"); assert_ne!(provider_family(spec), implementer_family(Some("claude")), "{spec}");
} }
// And the other way round: glm judging a glm mission is the same trap.
assert_eq!(provider_family("glm:glm-5.3"), implementer_family(Some("glm")));
} }
/// A mission's own choice wins over the deployment default. /// A mission's own choice wins over the deployment default.
+25
View File
@@ -306,6 +306,13 @@ const TEAMMATE_PROBE: &str = "cat /root/.claude/teams/*/config.json 2>/dev/null
/// says so, which is the difference between losing a check and losing the work. /// says so, which is the difference between losing a check and losing the work.
const SETTINGS_PROBE: &str = "claude --help 2>&1 | grep -q -- '--settings' && echo SETTINGS-OK"; const SETTINGS_PROBE: &str = "claude --help 2>&1 | grep -q -- '--settings' && echo SETTINGS-OK";
/// What the guest's `claude` reports itself as. Recorded beside the rootfs the
/// node said it booted, so "which CLI did this mission run on" is a query
/// against `topology_runs`, not an archaeology of image mtimes. Found necessary
/// on 2026-09-18: every rootfs on the fleet had been on 2.1.2232.1.226 for a
/// month while the container tier moved to 2.1.276, and nothing recorded either.
const CLI_VERSION_PROBE: &str = "claude --version 2>/dev/null | head -c 80";
/// How many times the stop gate refused to let the agent finish. /// How many times the stop gate refused to let the agent finish.
const BLOCKS_PROBE: &str = "cat /root/gate/blocks 2>/dev/null || echo 0"; const BLOCKS_PROBE: &str = "cat /root/gate/blocks 2>/dev/null || echo 0";
@@ -400,6 +407,12 @@ pub struct VmOutcome {
/// field cannot tell them apart — the unmatched-frame log and the install /// field cannot tell them apart — the unmatched-frame log and the install
/// error are what separate them. /// error are what separate them.
pub tools: Vec<crate::vm_tool_tap::Observed>, pub tools: Vec<crate::vm_tool_tap::Observed>,
/// The rootfs the node reported booting (`vm_create` reply), e.g.
/// `/opt/clawmates-fc/rootfs-glm.ext4`. `None` if the reply carried none.
pub rootfs: Option<String>,
/// The guest's own `claude --version`, e.g. `2.1.276 (Claude Code)`.
/// `None` if the probe failed — which is a fact worth seeing, not a zero.
pub cli_version: Option<String>,
} }
/// Boot a VM, run the phase in it, collect the result, and destroy it. /// Boot a VM, run the phase in it, collect the result, and destroy it.
@@ -618,6 +631,16 @@ async fn run_inside(
.await .await
.map(|p| p.stdout.contains("SETTINGS-OK")) .map(|p| p.stdout.contains("SETTINGS-OK"))
.unwrap_or(false); .unwrap_or(false);
let cli_version = vm
.exec(CLI_VERSION_PROBE, None, 60, &[])
.await
.ok()
.map(|p| p.stdout.trim().to_string())
.filter(|v| !v.is_empty());
let rootfs = created
.get("rootfs")
.and_then(serde_json::Value::as_str)
.map(str::to_string);
let gate_dir = match gate { let gate_dir = match gate {
None => None, None => None,
Some(g) => { Some(g) => {
@@ -954,6 +977,8 @@ async fn run_inside(
stop_blocks, stop_blocks,
released_at_cap, released_at_cap,
tools, tools,
rootfs,
cli_version,
}) })
} }
@@ -429,6 +429,8 @@ mod tests {
stop_blocks: None, stop_blocks: None,
released_at_cap: None, released_at_cap: None,
tools: Vec::new(), tools: Vec::new(),
rootfs: None,
cli_version: None,
}) })
} }
} }
@@ -624,6 +626,8 @@ mod tests {
stop_blocks: None, stop_blocks: None,
released_at_cap: None, released_at_cap: None,
tools: Vec::new(), tools: Vec::new(),
rootfs: None,
cli_version: None,
}) })
} }
} }
@@ -656,6 +660,8 @@ mod tests {
stop_blocks: Some(crate::vm_stop_gate::MAX_BLOCKS), stop_blocks: Some(crate::vm_stop_gate::MAX_BLOCKS),
released_at_cap: Some(true), released_at_cap: Some(true),
tools: Vec::new(), tools: Vec::new(),
rootfs: None,
cli_version: None,
}) })
} }
} }
@@ -683,6 +689,8 @@ mod tests {
stop_blocks: Some(crate::vm_stop_gate::MAX_BLOCKS), stop_blocks: Some(crate::vm_stop_gate::MAX_BLOCKS),
released_at_cap: Some(false), released_at_cap: Some(false),
tools: Vec::new(), tools: Vec::new(),
rootfs: None,
cli_version: None,
}) })
} }
} }
+18 -2
View File
@@ -1781,6 +1781,18 @@ async fn launch_microvm_phase(
if let Ok(o) = &outcome { if let Ok(o) = &outcome {
record_vm_tools(&pool2, mission_id, phase_id, run_id, &o.tools, &[]).await; record_vm_tools(&pool2, mission_id, phase_id, run_id, &o.tools, &[]).await;
} }
// What actually ran: the rootfs the node booted and the CLI the guest
// reported. Persisted on the run so "which image and version did this
// mission use" is a query, not an inference from file mtimes — on
// 2026-09-18 every fleet rootfs had sat on 2.1.2232.1.226 for a month
// while the container tier moved on, and nothing had recorded either.
let vm = serde_json::json!({
"vm_id": crate::microvm_executor::vm_id_for(phase_id, iteration, None),
"node_id": target_node_id,
"backend": backend,
"rootfs": outcome.as_ref().ok().and_then(|o| o.rootfs.clone()),
"cli_version": outcome.as_ref().ok().and_then(|o| o.cli_version.clone()),
});
let (status, note) = match outcome { let (status, note) = match outcome {
// The gate gave up. It is the ONLY thing that runs a // The gate gave up. It is the ONLY thing that runs a
// `done_when_check`, so a release at the cap means the phase's own // `done_when_check`, so a release at the cap means the phase's own
@@ -1813,7 +1825,9 @@ async fn launch_microvm_phase(
eprintln!( eprintln!(
"phase_runner: microvm phase {phase_id} of mission {mission_id}{status} \ "phase_runner: microvm phase {phase_id} of mission {mission_id}{status} \
(subagents: {subagents}, teammates: {teammates}, stop-gate blocks: \ (subagents: {subagents}, teammates: {teammates}, stop-gate blocks: \
{blocked}) — {}", {blocked}; rootfs: {}, cli: {}) — {}",
vm["rootfs"].as_str().unwrap_or("?"),
vm["cli_version"].as_str().unwrap_or("?"),
note.chars().take(300).collect::<String>() note.chars().take(300).collect::<String>()
); );
// Never overwrite a cancellation. The operator asking to stop is a decision; // Never overwrite a cancellation. The operator asking to stop is a decision;
@@ -1843,7 +1857,9 @@ async fn launch_microvm_phase(
"output": note, "output": note,
"tokens": 0, "tokens": 0,
"gated": [], "gated": [],
}] }],
// Beside `records`, not inside: the two readers parse only `records`.
"vm": vm,
}); });
if let Err(e) = sqlx::query( if let Err(e) = sqlx::query(
"UPDATE topology_runs "UPDATE topology_runs
+8 -1
View File
@@ -27,7 +27,14 @@ FROM clawmates/agent-toolchain:dev
# could undo. 2.1.221 also fixes `--mcp-config` servers not connecting before the # could undo. 2.1.221 also fixes `--mcp-config` servers not connecting before the
# first turn in print mode, which is exactly the mode we run and will matter when # first turn in print mode, which is exactly the mode we run and will matter when
# the MCP door reaches a VM. # the MCP door reaches a VM.
ARG CLAUDE_CODE_VERSION=2.1.226 # 2.1.276, 2026-09-18. Between 2.1.226 and here, 2.1.265 and 2.1.275 each broke
# every turn on ANTHROPIC_BASE_URL endpoints (HTTP 400) — the path the glm and
# kimi images use — and 2.1.276 is the first version after both that is fixed.
# All four agent-* images pin the SAME version; scripts/fc-build-rootfs.sh
# refuses to build if they drift. Bump them together, on purpose, and run a
# mission on each backend before promoting (see deploy/clawmates-runtime/Dockerfile
# for the same rule on the container tier).
ARG CLAUDE_CODE_VERSION=2.1.276
RUN npm install -g "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}" \ RUN npm install -g "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}" \
&& npm cache clean --force \ && npm cache clean --force \
&& rm -rf /root/.npm \ && rm -rf /root/.npm \
+8 -1
View File
@@ -21,7 +21,14 @@ FROM clawmates/agent-toolchain:dev
# composed run's verifier node should differ by provider and by nothing else; two # composed run's verifier node should differ by provider and by nothing else; two
# CLI versions in one graph would make "the verifier disagreed" ambiguous between # CLI versions in one graph would make "the verifier disagreed" ambiguous between
# the model and the harness. # the model and the harness.
ARG CLAUDE_CODE_VERSION=2.1.223 # 2.1.276, 2026-09-18. Between 2.1.226 and here, 2.1.265 and 2.1.275 each broke
# every turn on ANTHROPIC_BASE_URL endpoints (HTTP 400) — the path the glm and
# kimi images use — and 2.1.276 is the first version after both that is fixed.
# All four agent-* images pin the SAME version; scripts/fc-build-rootfs.sh
# refuses to build if they drift. Bump them together, on purpose, and run a
# mission on each backend before promoting (see deploy/clawmates-runtime/Dockerfile
# for the same rule on the container tier).
ARG CLAUDE_CODE_VERSION=2.1.276
RUN npm install -g "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}" \ RUN npm install -g "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}" \
&& npm cache clean --force \ && npm cache clean --force \
&& rm -rf /root/.npm \ && rm -rf /root/.npm \
+8 -1
View File
@@ -34,7 +34,14 @@
# own model exactly as z.ai does. No ANTHROPIC_MODEL override is needed. # own model exactly as z.ai does. No ANTHROPIC_MODEL override is needed.
FROM clawmates/agent-toolchain:dev FROM clawmates/agent-toolchain:dev
ARG CLAUDE_CODE_VERSION=2.1.223 # 2.1.276, 2026-09-18. Between 2.1.226 and here, 2.1.265 and 2.1.275 each broke
# every turn on ANTHROPIC_BASE_URL endpoints (HTTP 400) — the path the glm and
# kimi images use — and 2.1.276 is the first version after both that is fixed.
# All four agent-* images pin the SAME version; scripts/fc-build-rootfs.sh
# refuses to build if they drift. Bump them together, on purpose, and run a
# mission on each backend before promoting (see deploy/clawmates-runtime/Dockerfile
# for the same rule on the container tier).
ARG CLAUDE_CODE_VERSION=2.1.276
RUN npm install -g "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}" \ RUN npm install -g "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}" \
&& npm cache clean --force \ && npm cache clean --force \
&& rm -rf /root/.npm \ && rm -rf /root/.npm \
+8 -1
View File
@@ -31,7 +31,14 @@ FROM clawmates/agent-toolchain:dev
# Pinned to the SAME version as agent-claude and agent-glm. A run that differs # Pinned to the SAME version as agent-claude and agent-glm. A run that differs
# by provider should differ by nothing else, or "the local backend behaved # by provider should differ by nothing else, or "the local backend behaved
# differently" is ambiguous between the model and the harness. # differently" is ambiguous between the model and the harness.
ARG CLAUDE_CODE_VERSION=2.1.223 # 2.1.276, 2026-09-18. Between 2.1.226 and here, 2.1.265 and 2.1.275 each broke
# every turn on ANTHROPIC_BASE_URL endpoints (HTTP 400) — the path the glm and
# kimi images use — and 2.1.276 is the first version after both that is fixed.
# All four agent-* images pin the SAME version; scripts/fc-build-rootfs.sh
# refuses to build if they drift. Bump them together, on purpose, and run a
# mission on each backend before promoting (see deploy/clawmates-runtime/Dockerfile
# for the same rule on the container tier).
ARG CLAUDE_CODE_VERSION=2.1.276
RUN npm install -g "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}" \ RUN npm install -g "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}" \
&& npm cache clean --force \ && npm cache clean --force \
&& rm -rf /root/.npm \ && rm -rf /root/.npm \
+13
View File
@@ -23,6 +23,19 @@
set -uo pipefail set -uo pipefail
# Drift guard. The four agent-* images must pin ONE Claude Code version: a solo
# run and a composed run's verifier should differ by provider and by nothing
# else, and on 2026-09-18 they had already drifted (claude 2.1.226, the rest
# 2.1.223) under comments saying "same version on purpose". Refuse to build a
# rootfs from a set that disagrees, and say which.
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
pins=$(grep -h '^ARG CLAUDE_CODE_VERSION=' "$repo_root"/images/agent-{claude,glm,kimi,ornith}/Dockerfile 2>/dev/null | sort -u)
if [ "$(printf '%s\n' "$pins" | grep -c .)" -ne 1 ]; then
echo "fc-build-rootfs: the agent-* images pin different Claude Code versions — fix that first:" >&2
grep -H '^ARG CLAUDE_CODE_VERSION=' "$repo_root"/images/agent-{claude,glm,kimi,ornith}/Dockerfile >&2
exit 2
fi
WORK="${FC_WORK:-/opt/clawmates-fc}" WORK="${FC_WORK:-/opt/clawmates-fc}"
FAILURES=0 FAILURES=0
pass() { printf 'PASS %s\n' "$*"; } pass() { printf 'PASS %s\n' "$*"; }
+99 -4
View File
@@ -31,6 +31,8 @@
# scripts/verify-mission-delivery.sh multirole # 3 roles + real tests # scripts/verify-mission-delivery.sh multirole # 3 roles + real tests
# scripts/verify-mission-delivery.sh noop # empty phase must FAIL # scripts/verify-mission-delivery.sh noop # empty phase must FAIL
# scripts/verify-mission-delivery.sh microvm # runs in a guest kernel + fans out # scripts/verify-mission-delivery.sh microvm # runs in a guest kernel + fans out
# scripts/verify-mission-delivery.sh glm # microvm on the z.ai backend; provider proven by the node's egress log
# scripts/verify-mission-delivery.sh kimi # microvm on the Kimi backend; same proof
# scripts/verify-mission-delivery.sh capacity # a burst > the fleet must QUEUE # scripts/verify-mission-delivery.sh capacity # a burst > the fleet must QUEUE
# scripts/verify-mission-delivery.sh drain-midmission # a drained node hands the mission on # scripts/verify-mission-delivery.sh drain-midmission # a drained node hands the mission on
# scripts/verify-mission-delivery.sh all # everything # scripts/verify-mission-delivery.sh all # everything
@@ -57,7 +59,11 @@ MISSIONS_ROOT="${CLAWMATES_MISSIONS_ROOT:-/var/lib/clawmates-missions}"
# vmlinux version that an upgrade would invalidate. # vmlinux version that an upgrade would invalidate.
GW_KERNEL=$(ssh "$HOST" 'uname -r' 2>/dev/null | tr -d '[:space:]') GW_KERNEL=$(ssh "$HOST" 'uname -r' 2>/dev/null | tr -d '[:space:]')
NODE_KERNEL=$(ssh "${FLEET_NODE:-osobh@tank}" 'uname -r' 2>/dev/null | tr -d '[:space:]') NODE_KERNEL=$(ssh "${FLEET_NODE:-osobh@tank}" 'uname -r' 2>/dev/null | tr -d '[:space:]')
REPO_ID="${CLAWMATES_REPO_ID:-f8bbe4d7-2878-40c8-b657-7a7f6031def1}" # The scratch repo is re-registered whenever the Gitea connection re-syncs, and
# gets a new id each time (f8bbe4d7-… died in the 2026-09-14 wipe). Override
# with CLAWMATES_REPO_ID, or find it: select id from repos where name =
# 'clawmates-delivery-scratch'.
REPO_ID="${CLAWMATES_REPO_ID:-01a0052f-dc7f-7b73-8afd-2984a91338bb}"
TEAM_TEMPLATE="${CLAWMATES_TEAM_TEMPLATE:-7e453826-41c4-4425-bab5-8f11fd0a14d7}" TEAM_TEMPLATE="${CLAWMATES_TEAM_TEMPLATE:-7e453826-41c4-4425-bab5-8f11fd0a14d7}"
MISSION_TIMEOUT="${MISSION_TIMEOUT:-1800}" MISSION_TIMEOUT="${MISSION_TIMEOUT:-1800}"
@@ -1111,6 +1117,79 @@ assert_local() { # <token> <mission> <report>
fi fi
} }
# ── Provider proof for a microVM mission ─────────────────────────
#
# Which provider served a VM's turns is answered by the NODE's egress proxy
# log and by nothing the agent wrote: a z.ai-served agent called itself Claude
# Opus 5 (memory: glm-microvm-backend). The proxy allow-lists one provider host
# per backend and logs every dial and every denial, so "dialled the right host,
# never denied it, dialled no other" is the whole proof — and it is what the
# 2.1.276 rollout had to establish for glm and kimi, whose turns ride
# ANTHROPIC_BASE_URL through a CLI that broke that path twice between 2.1.226
# and 2.1.276.
placed_node() { # <mission> → ssh target of the node that ran it, or 1
local node
node=$(ssh "$HOST" "docker exec clawmates_postgres_1 psql -U postgres -d clawmates -tAc \
\"select coalesce(n.name,'') from missions m left join nodes n on n.id = m.target_node_id where m.id='$1';\"" \
| head -1 | tr -d '[:space:]')
case "$node" in '') return 1 ;; tank) echo osobh@tank ;; *) echo "$node" ;; esac
}
vm_journal() { # <ssh-target> <mission> → this mission's VM lines only
# The VM id is m-<first 12 hex of the phase id>-<iteration>
# (microvm_executor::vm_id_for), so a grep on that prefix cannot pick up a
# neighbour's VM the way a time window can.
local p12
p12=$(ssh "$HOST" "docker exec clawmates_postgres_1 psql -U postgres -d clawmates -tAc \
\"select substr(replace(id::text,'-',''),1,12) from mission_phases where mission_id='$2' order by order_idx limit 1;\"" \
| head -1 | tr -d '[:space:]')
[ -n "$p12" ] || return 1
ssh "$1" "journalctl -u clawmates-node --since '-3 hours' --no-pager 2>/dev/null | grep -F 'microvm m-$p12-'"
}
assert_provider_egress() { # <label> <mission> <provider-host>
local label="$1" mission="$2" want="$3" target lines hits denied other
target=$(placed_node "$mission") || { norun "$label: could not tell which node ran it — provider UNPROVEN"; return 1; }
lines=$(vm_journal "$target" "$mission") || { norun "$label: no journal lines for this mission's VM on $target"; return 1; }
hits=$(printf '%s\n' "$lines" | grep -c "egress -> $want")
denied=$(printf '%s\n' "$lines" | grep -c "egress DENIED $want")
if [ "$hits" -gt 0 ] && [ "$denied" -eq 0 ]; then
pass "$label: the VM dialled $want ${hits}x and was never denied it"
else
fail "$label: egress -> $want ${hits}x, DENIED ${denied}x — provider NOT proven"
fi
# Dials only. A DENIED api.anthropic.com on a glm/kimi VM is Claude Code's
# telemetry being refused, which is the proxy working, not a second provider.
other=$(printf '%s\n' "$lines" | grep 'egress -> ' | grep -v -e "egress -> $want" -e 'git.redclaw.dev' \
| sed 's/.*egress -> //' | sort -u | tr '\n' ' ')
if [ -z "$other" ]; then
pass "$label: nothing but the provider and the forge was reached"
else
fail "$label: other hosts reached: $other"
fi
}
assert_cli_version() { # <label> <mission> — needs checkpoint.vm (phase_runner, 2026-09-18)
local label="$1" mission="$2" got want="${CLAWMATES_EXPECT_CLI:-2.1.276}"
got=$(ssh "$HOST" "docker exec clawmates_postgres_1 psql -U postgres -d clawmates -tAc \
\"select coalesce(checkpoint->'vm'->>'cli_version','') from topology_runs where mission_id='$mission' limit 1;\"" \
| head -1 | tr -d '\r')
case "$got" in
"$want"*) pass "$label: the guest ran Claude Code $got" ;;
'') fail "$label: no cli_version recorded on the run (server predates checkpoint.vm, or the probe failed)" ;;
*) fail "$label: guest CLI was '$got', expected $want" ;;
esac
got=$(ssh "$HOST" "docker exec clawmates_postgres_1 psql -U postgres -d clawmates -tAc \
\"select coalesce(checkpoint->'vm'->>'rootfs','') from topology_runs where mission_id='$mission' limit 1;\"" \
| head -1 | tr -d '\r')
[ -n "$got" ] && pass "$label: booted $got" || fail "$label: no rootfs recorded on the run"
}
assert_claude_vm() { assert_microvm "$@"; assert_provider_egress microvm "$2" api.anthropic.com; assert_cli_version microvm "$2"; }
assert_glm() { assert_microvm "$@"; assert_provider_egress glm "$2" api.z.ai; assert_cli_version glm "$2"; }
assert_kimi() { assert_microvm "$@"; assert_provider_egress kimi "$2" api.kimi.com; assert_cli_version kimi "$2"; }
# ── Scenario: a burst larger than the fleet QUEUES, and spreads ─── # ── Scenario: a burst larger than the fleet QUEUES, and spreads ───
# #
# Phase 1 of the fleet-intelligence plan shipped placement-at-phase-launch and # Phase 1 of the fleet-intelligence plan shipped placement-at-phase-launch and
@@ -1506,9 +1585,23 @@ case "${1:-all}" in
assert_microvm assert_microvm
;; ;;
microvm) microvm)
run_scenario microvm "$(echo "$MICROVM_BODY" | tr -d '\n')" assert_microvm run_scenario microvm "$(echo "$MICROVM_BODY" | tr -d '\n')" assert_claude_vm
scenario_microvm_unavailable_backend scenario_microvm_unavailable_backend
;; ;;
glm)
# The microvm scenario on the z.ai backend. The provider assertion is the
# point: this CLI reaches z.ai through ANTHROPIC_BASE_URL, a path 2.1.265 and
# 2.1.275 both broke, and "the mission completed" alone cannot say who
# answered.
run_scenario glm \
"$(echo "$MICROVM_BODY" | sed 's/"backend":"claude"/"backend":"glm"/' | tr -d '\n')" \
assert_glm
;;
kimi)
run_scenario kimi \
"$(echo "$MICROVM_BODY" | sed 's/"backend":"claude"/"backend":"kimi"/' | tr -d '\n')" \
assert_kimi
;;
gatecap) gatecap)
run_scenario gatecap "$(echo "$GATECAP_BODY" | tr -d '\n')" assert_gate_cap run_scenario gatecap "$(echo "$GATECAP_BODY" | tr -d '\n')" assert_gate_cap
;; ;;
@@ -1548,8 +1641,10 @@ case "${1:-all}" in
body=${MULTIROLE_BODY//__TEAM__/$TEAM_TEMPLATE} body=${MULTIROLE_BODY//__TEAM__/$TEAM_TEMPLATE}
run_scenario multirole "${body//__REPO__/$REPO_ID}" assert_multirole run_scenario multirole "${body//__REPO__/$REPO_ID}" assert_multirole
run_scenario noop "$NOOP_BODY" assert_noop run_scenario noop "$NOOP_BODY" assert_noop
run_scenario microvm "$(echo "$MICROVM_BODY" | tr -d '\n')" assert_microvm run_scenario microvm "$(echo "$MICROVM_BODY" | tr -d '\n')" assert_claude_vm
scenario_microvm_unavailable_backend scenario_microvm_unavailable_backend
run_scenario glm "$(echo "$MICROVM_BODY" | sed 's/"backend":"claude"/"backend":"glm"/' | tr -d '\n')" assert_glm
run_scenario kimi "$(echo "$MICROVM_BODY" | sed 's/"backend":"claude"/"backend":"kimi"/' | tr -d '\n')" assert_kimi
run_scenario gatecap "$(echo "$GATECAP_BODY" | tr -d '\n')" assert_gate_cap run_scenario gatecap "$(echo "$GATECAP_BODY" | tr -d '\n')" assert_gate_cap
run_scenario research-only "$(echo "$RESEARCH_ONLY_BODY" | tr -d '\n')" assert_research_only no-checkout run_scenario research-only "$(echo "$RESEARCH_ONLY_BODY" | tr -d '\n')" assert_research_only no-checkout
run_scenario research-vm "$(echo "$RESEARCH_VM_BODY" | tr -d '\n')" assert_research_only no-checkout run_scenario research-vm "$(echo "$RESEARCH_VM_BODY" | tr -d '\n')" assert_research_only no-checkout
@@ -1563,7 +1658,7 @@ case "${1:-all}" in
scenario_drain_midmission scenario_drain_midmission
;; ;;
*) *)
die "unknown scenario: $1 (selftest|uids|chain|multirole|noop|microvm|canary|gatecap|research-only|research-vm|benchmark|security|refactor|composed|roster|local-ornith|capacity|drain-midmission|all)" die "unknown scenario: $1 (selftest|uids|chain|multirole|noop|microvm|canary|glm|kimi|gatecap|research-only|research-vm|benchmark|security|refactor|composed|roster|local-ornith|capacity|drain-midmission|all)"
;; ;;
esac esac