feat(microvm): Claude Code 2.1.276 rootfs pins, and a VM run that says what it ran
Every rootfs on the fleet had sat on Claude Code 2.1.223–2.1.226 since August while the container tier moved to 2.1.276, and nothing recorded either. GLM and Kimi exist only as microVM backends, so "have we upgraded GLM and Kimi" is this change and the rebuild it drives. Pins. All four agent-* images pin 2.1.276 — as separate ARGs, since Docker has no include and each file has to stay reproducible alone — and scripts/fc-build-rootfs.sh refuses to build if they disagree, naming the odd one out. They had already drifted (claude 226, the rest 223) under comments saying "same version on purpose". Between 2.1.226 and 2.1.276, 2.1.265 and 2.1.275 each broke every turn on ANTHROPIC_BASE_URL endpoints, which is how glm and kimi reach `claude` inside a VM; the container-tier verification never exercised that path, so the VM runs on those backends are the real test. Provenance. `VmOutcome` carries the rootfs the node reported booting and the guest's own `claude --version`; `launch_microvm_phase` persists both as `checkpoint.vm` beside `records` (the two readers parse only `records`) and names them in its log line. "Which image and CLI did this mission run on" is a query now. Independence. `evaluator` derived the implementer family from a constant `"anthropic"`, true while every backend was Claude on Anthropic. With glm and kimi rootfs it made a glm mission judged by glm:glm-5.3 read as `independent = true` — the one claim that path exists to make honestly. `implementer_family(missions.backend)` mirrors `microvm_credential_for`; the subscription judge is now independent exactly when the agent did NOT run on Anthropic. Harness. `verify-mission-delivery.sh glm|kimi` run the microvm scenario on each backend and add the proof the mission itself cannot give: the placed node's journal must show the VM dialling that provider's host, never being denied it, and dialling nothing else but the forge — a model's self-report is measured worthless here. `assert_cli_version` reads checkpoint.vm. The stale scratch-repo default (dead since the 09-14 wipe) is the re-synced id. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
This commit is contained in:
Omar Sobh
co-authored by
Claude Opus 5
Claude Opus 5
parent
3755699b41
commit
794f2124bc
@@ -31,6 +31,8 @@
|
||||
# scripts/verify-mission-delivery.sh multirole # 3 roles + real tests
|
||||
# scripts/verify-mission-delivery.sh noop # empty phase must FAIL
|
||||
# scripts/verify-mission-delivery.sh microvm # runs in a guest kernel + fans out
|
||||
# scripts/verify-mission-delivery.sh glm # microvm on the z.ai backend; provider proven by the node's egress log
|
||||
# scripts/verify-mission-delivery.sh kimi # microvm on the Kimi backend; same proof
|
||||
# scripts/verify-mission-delivery.sh capacity # a burst > the fleet must QUEUE
|
||||
# scripts/verify-mission-delivery.sh drain-midmission # a drained node hands the mission on
|
||||
# scripts/verify-mission-delivery.sh all # everything
|
||||
@@ -57,7 +59,11 @@ MISSIONS_ROOT="${CLAWMATES_MISSIONS_ROOT:-/var/lib/clawmates-missions}"
|
||||
# vmlinux version that an upgrade would invalidate.
|
||||
GW_KERNEL=$(ssh "$HOST" 'uname -r' 2>/dev/null | tr -d '[:space:]')
|
||||
NODE_KERNEL=$(ssh "${FLEET_NODE:-osobh@tank}" 'uname -r' 2>/dev/null | tr -d '[:space:]')
|
||||
REPO_ID="${CLAWMATES_REPO_ID:-f8bbe4d7-2878-40c8-b657-7a7f6031def1}"
|
||||
# The scratch repo is re-registered whenever the Gitea connection re-syncs, and
|
||||
# gets a new id each time (f8bbe4d7-… died in the 2026-09-14 wipe). Override
|
||||
# with CLAWMATES_REPO_ID, or find it: select id from repos where name =
|
||||
# 'clawmates-delivery-scratch'.
|
||||
REPO_ID="${CLAWMATES_REPO_ID:-01a0052f-dc7f-7b73-8afd-2984a91338bb}"
|
||||
TEAM_TEMPLATE="${CLAWMATES_TEAM_TEMPLATE:-7e453826-41c4-4425-bab5-8f11fd0a14d7}"
|
||||
MISSION_TIMEOUT="${MISSION_TIMEOUT:-1800}"
|
||||
|
||||
@@ -1111,6 +1117,79 @@ assert_local() { # <token> <mission> <report>
|
||||
fi
|
||||
}
|
||||
|
||||
# ── Provider proof for a microVM mission ─────────────────────────
|
||||
#
|
||||
# Which provider served a VM's turns is answered by the NODE's egress proxy
|
||||
# log and by nothing the agent wrote: a z.ai-served agent called itself Claude
|
||||
# Opus 5 (memory: glm-microvm-backend). The proxy allow-lists one provider host
|
||||
# per backend and logs every dial and every denial, so "dialled the right host,
|
||||
# never denied it, dialled no other" is the whole proof — and it is what the
|
||||
# 2.1.276 rollout had to establish for glm and kimi, whose turns ride
|
||||
# ANTHROPIC_BASE_URL through a CLI that broke that path twice between 2.1.226
|
||||
# and 2.1.276.
|
||||
|
||||
placed_node() { # <mission> → ssh target of the node that ran it, or 1
|
||||
local node
|
||||
node=$(ssh "$HOST" "docker exec clawmates_postgres_1 psql -U postgres -d clawmates -tAc \
|
||||
\"select coalesce(n.name,'') from missions m left join nodes n on n.id = m.target_node_id where m.id='$1';\"" \
|
||||
| head -1 | tr -d '[:space:]')
|
||||
case "$node" in '') return 1 ;; tank) echo osobh@tank ;; *) echo "$node" ;; esac
|
||||
}
|
||||
|
||||
vm_journal() { # <ssh-target> <mission> → this mission's VM lines only
|
||||
# The VM id is m-<first 12 hex of the phase id>-<iteration>
|
||||
# (microvm_executor::vm_id_for), so a grep on that prefix cannot pick up a
|
||||
# neighbour's VM the way a time window can.
|
||||
local p12
|
||||
p12=$(ssh "$HOST" "docker exec clawmates_postgres_1 psql -U postgres -d clawmates -tAc \
|
||||
\"select substr(replace(id::text,'-',''),1,12) from mission_phases where mission_id='$2' order by order_idx limit 1;\"" \
|
||||
| head -1 | tr -d '[:space:]')
|
||||
[ -n "$p12" ] || return 1
|
||||
ssh "$1" "journalctl -u clawmates-node --since '-3 hours' --no-pager 2>/dev/null | grep -F 'microvm m-$p12-'"
|
||||
}
|
||||
|
||||
assert_provider_egress() { # <label> <mission> <provider-host>
|
||||
local label="$1" mission="$2" want="$3" target lines hits denied other
|
||||
target=$(placed_node "$mission") || { norun "$label: could not tell which node ran it — provider UNPROVEN"; return 1; }
|
||||
lines=$(vm_journal "$target" "$mission") || { norun "$label: no journal lines for this mission's VM on $target"; return 1; }
|
||||
hits=$(printf '%s\n' "$lines" | grep -c "egress -> $want")
|
||||
denied=$(printf '%s\n' "$lines" | grep -c "egress DENIED $want")
|
||||
if [ "$hits" -gt 0 ] && [ "$denied" -eq 0 ]; then
|
||||
pass "$label: the VM dialled $want ${hits}x and was never denied it"
|
||||
else
|
||||
fail "$label: egress -> $want ${hits}x, DENIED ${denied}x — provider NOT proven"
|
||||
fi
|
||||
# Dials only. A DENIED api.anthropic.com on a glm/kimi VM is Claude Code's
|
||||
# telemetry being refused, which is the proxy working, not a second provider.
|
||||
other=$(printf '%s\n' "$lines" | grep 'egress -> ' | grep -v -e "egress -> $want" -e 'git.redclaw.dev' \
|
||||
| sed 's/.*egress -> //' | sort -u | tr '\n' ' ')
|
||||
if [ -z "$other" ]; then
|
||||
pass "$label: nothing but the provider and the forge was reached"
|
||||
else
|
||||
fail "$label: other hosts reached: $other"
|
||||
fi
|
||||
}
|
||||
|
||||
assert_cli_version() { # <label> <mission> — needs checkpoint.vm (phase_runner, 2026-09-18)
|
||||
local label="$1" mission="$2" got want="${CLAWMATES_EXPECT_CLI:-2.1.276}"
|
||||
got=$(ssh "$HOST" "docker exec clawmates_postgres_1 psql -U postgres -d clawmates -tAc \
|
||||
\"select coalesce(checkpoint->'vm'->>'cli_version','') from topology_runs where mission_id='$mission' limit 1;\"" \
|
||||
| head -1 | tr -d '\r')
|
||||
case "$got" in
|
||||
"$want"*) pass "$label: the guest ran Claude Code $got" ;;
|
||||
'') fail "$label: no cli_version recorded on the run (server predates checkpoint.vm, or the probe failed)" ;;
|
||||
*) fail "$label: guest CLI was '$got', expected $want" ;;
|
||||
esac
|
||||
got=$(ssh "$HOST" "docker exec clawmates_postgres_1 psql -U postgres -d clawmates -tAc \
|
||||
\"select coalesce(checkpoint->'vm'->>'rootfs','') from topology_runs where mission_id='$mission' limit 1;\"" \
|
||||
| head -1 | tr -d '\r')
|
||||
[ -n "$got" ] && pass "$label: booted $got" || fail "$label: no rootfs recorded on the run"
|
||||
}
|
||||
|
||||
assert_claude_vm() { assert_microvm "$@"; assert_provider_egress microvm "$2" api.anthropic.com; assert_cli_version microvm "$2"; }
|
||||
assert_glm() { assert_microvm "$@"; assert_provider_egress glm "$2" api.z.ai; assert_cli_version glm "$2"; }
|
||||
assert_kimi() { assert_microvm "$@"; assert_provider_egress kimi "$2" api.kimi.com; assert_cli_version kimi "$2"; }
|
||||
|
||||
# ── Scenario: a burst larger than the fleet QUEUES, and spreads ───
|
||||
#
|
||||
# Phase 1 of the fleet-intelligence plan shipped placement-at-phase-launch and
|
||||
@@ -1506,9 +1585,23 @@ case "${1:-all}" in
|
||||
assert_microvm
|
||||
;;
|
||||
microvm)
|
||||
run_scenario microvm "$(echo "$MICROVM_BODY" | tr -d '\n')" assert_microvm
|
||||
run_scenario microvm "$(echo "$MICROVM_BODY" | tr -d '\n')" assert_claude_vm
|
||||
scenario_microvm_unavailable_backend
|
||||
;;
|
||||
glm)
|
||||
# The microvm scenario on the z.ai backend. The provider assertion is the
|
||||
# point: this CLI reaches z.ai through ANTHROPIC_BASE_URL, a path 2.1.265 and
|
||||
# 2.1.275 both broke, and "the mission completed" alone cannot say who
|
||||
# answered.
|
||||
run_scenario glm \
|
||||
"$(echo "$MICROVM_BODY" | sed 's/"backend":"claude"/"backend":"glm"/' | tr -d '\n')" \
|
||||
assert_glm
|
||||
;;
|
||||
kimi)
|
||||
run_scenario kimi \
|
||||
"$(echo "$MICROVM_BODY" | sed 's/"backend":"claude"/"backend":"kimi"/' | tr -d '\n')" \
|
||||
assert_kimi
|
||||
;;
|
||||
gatecap)
|
||||
run_scenario gatecap "$(echo "$GATECAP_BODY" | tr -d '\n')" assert_gate_cap
|
||||
;;
|
||||
@@ -1548,8 +1641,10 @@ case "${1:-all}" in
|
||||
body=${MULTIROLE_BODY//__TEAM__/$TEAM_TEMPLATE}
|
||||
run_scenario multirole "${body//__REPO__/$REPO_ID}" assert_multirole
|
||||
run_scenario noop "$NOOP_BODY" assert_noop
|
||||
run_scenario microvm "$(echo "$MICROVM_BODY" | tr -d '\n')" assert_microvm
|
||||
run_scenario microvm "$(echo "$MICROVM_BODY" | tr -d '\n')" assert_claude_vm
|
||||
scenario_microvm_unavailable_backend
|
||||
run_scenario glm "$(echo "$MICROVM_BODY" | sed 's/"backend":"claude"/"backend":"glm"/' | tr -d '\n')" assert_glm
|
||||
run_scenario kimi "$(echo "$MICROVM_BODY" | sed 's/"backend":"claude"/"backend":"kimi"/' | tr -d '\n')" assert_kimi
|
||||
run_scenario gatecap "$(echo "$GATECAP_BODY" | tr -d '\n')" assert_gate_cap
|
||||
run_scenario research-only "$(echo "$RESEARCH_ONLY_BODY" | tr -d '\n')" assert_research_only no-checkout
|
||||
run_scenario research-vm "$(echo "$RESEARCH_VM_BODY" | tr -d '\n')" assert_research_only no-checkout
|
||||
@@ -1563,7 +1658,7 @@ case "${1:-all}" in
|
||||
scenario_drain_midmission
|
||||
;;
|
||||
*)
|
||||
die "unknown scenario: $1 (selftest|uids|chain|multirole|noop|microvm|canary|gatecap|research-only|research-vm|benchmark|security|refactor|composed|roster|local-ornith|capacity|drain-midmission|all)"
|
||||
die "unknown scenario: $1 (selftest|uids|chain|multirole|noop|microvm|canary|glm|kimi|gatecap|research-only|research-vm|benchmark|security|refactor|composed|roster|local-ornith|capacity|drain-midmission|all)"
|
||||
;;
|
||||
esac
|
||||
|
||||
|
||||
Reference in New Issue
Block a user