Commit Graph
6 Commits
Author SHA1 Message Date
Omar SobhandClaude Opus 5 f7f3dfe495 feat(fleet): GLM as a real microVM backend, and per-role models for claws
Three threads, all of which end at the same place: a mission whose verifier does
not share a model with the coder it reviews.

**GLM has a credential contract now.** `microvm_credential_for` returned one env
var name, which quietly assumed every provider reads its secret from the same
place Anthropic does. It returns a `Credential { source, target }` instead —
z.ai's key lives in the server's `ZAI_API_KEY` and Claude Code reads it as
`ANTHROPIC_AUTH_TOKEN`, and collapsing those two names is what forces a guess at
the other end. A wrong guess here sends one provider's credential to another
provider's endpoint.

`images/agent-glm` is the same CLI at the same pinned version as `agent-claude`
with `ANTHROPIC_BASE_URL` baked in. The split is deliberate: the ENDPOINT is a
property of the image, the CREDENTIAL is a property of the turn. That makes the
dangerous mix-up unrepresentable — a GLM VM cannot be handed an Anthropic
subscription token, and a claude VM cannot be pointed at z.ai. Asserted both
ways, because "the GLM VM must not carry CLAUDE_CODE_OAUTH_TOKEN" is the
property that costs a credential if it ever stops holding.

Kimi stays refused. `KIMI_API_KEY` is set and Moonshot serves an
Anthropic-compatible API, but I have not verified its base URL against the
running service, and this function is precisely where guessing a URL is
expensive. It becomes an arm the day someone measures it.

`api.z.ai` joins the node's default egress allow-list. A default that cannot
run the images we ship is a trap rather than a policy — the alternative is an
operator discovering it as a hung agent with no model access.

**Per-role models for claws** (migration 0071). `template_roles` had no model
column, so `mint_team_from_template` bound every role of every mission team to
one literal — a template whose whole point is an independent reviewer minted a
reviewer sharing a model with the coder. A role may now name its own; roles that
say nothing still take the mint's default, so every template written before this
behaves exactly as it did. The literal is now that default rather than a
hardcode.

**A harness scenario for the roster flow.** `verify-mission-delivery.sh roster`
runs the whole Slice 5 loop — planner proposes, human approves, mission runs —
and asserts the roster LANDED on the mission row rather than trusting the API's
answer. That distinction is not theoretical: the first live approval returned an
error while leaving the proposal marked approved.

Built and proven on tank ahead of the deploy: `clawmates/agent-glm:dev` reports
`2.1.223` and `BASE=https://api.z.ai/api/anthropic`, and
`fc-build-rootfs.sh … glm 8G` boots a VM from it that has git, can write
/mission, and answers `claude --version`.

533 tests pass, clippy clean. Migration 0071.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-06 17:14:53 -07:00
Omar SobhandClaude Opus 5 abb97e6f03 test(harness): a composed scenario, and the stop gate asserted in a real VM
`verify-mission-delivery.sh composed` runs a `team_engine=composed` mission and
checks the one property that cannot be checked any other way: a VM is inject →
run → collect → destroy, so unless the tree is carried node to node, node 2 boots
from the original checkout, sees nothing of node 1's work, and still reports
success. The task makes each node append ONE line to STAGES.md, so the delivered
file IS the evidence — a run that lost the handoff delivers one line, and no
amount of agent confidence can fabricate the missing ones.

It also asserts the run's tier is `microvm_graph`. A composed mission that
quietly fell back to the solo path would deliver a one-line file and look exactly
like a graph that ran one node.

`assert_stop_gate` reads the count `phase_runner` reports and distinguishes three
outcomes that matter: a number (installed, fired that often), `0` (installed,
never needed), and `-` (could NOT be installed — usually a CLI in the image with
no `--settings`). Wired into the microvm scenario rather than its own, because it
applies to every coding phase on that path.

Both ran against the deployed stack:

  composed — 4/4. STAGES.md carried 5 stage lines through 5 separate VMs
  (planner → coder → tester → reviewer → committer), each stamped with the guest
  kernel 6.1.128 rather than the gateway's 6.8.0 or the node's 7.0.0. The run
  checkpointed 5 steps on the worker, and `updated_at` stayed ~2s old mid-turn,
  which is the keepalive doing its job — without it `requeue_stale` flips a live
  run at 180 seconds.

  microvm — 6/6, including the gate installed in a real VM (`blocks: 0`), one
  subagent, the GLM judge, and the unavailable-backend negative control.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-06 15:17:24 -07:00
Omar Sobh b17e18aa67 fix(harness): the verdict check matched psql's display form, not the query's
`select met || ' ' || independent` casts the booleans to `true`/`false`, but the
pattern matched `t`/`f` — psql's *column display* form. So the check reported "no
verdict recorded for the phase" while the row sat in the table saying met=true,
independent=true, glm-4.7.

A check that fails for a reason unrelated to what it checks is worse than no check:
it trains you to ignore the output. The booleans are cast explicitly now so the
shape cannot drift again, and the failure message prints what it actually got.

`verify-mission-delivery.sh microvm` now passes 5/5 against production:
  - the agent ran under guest kernel 6.1.128, not the gateway's 6.8.0-124 or the
    node's 7.0.0-28 — the one assertion that cannot pass by accident
  - the lead delegated to 1 subagent
  - the condition was met and judged INDEPENDENTLY by glm-4.7
  - the checkout has exactly one writer (uid 65532)
  - negative control: a backend no node can run is refused at launch
2026-08-05 22:44:25 -07:00
Omar Sobh 9aed20b6d0 fix(missions): capture a failed phase's work; harness gains a microvm scenario (#51)
A REGRESSION I INTRODUCED ONE COMMIT AGO. `capture_finished_coding_phases`
selects on `mp.status = 'completed'`, so the moment an unmet phase correctly began
reporting `failed`, its diff stopped being captured, committed or pushed — the work
was silently discarded. Found by the new harness scenario, whose phase legitimately
missed its condition and then had no artifact at all.

What was produced, and whether the goal was met, are different facts. The artifact
records the first; `mp.status` records the second. Capture now covers terminal
phases (`completed`, `failed`), so a phase that did real work and missed its goal
still delivers a reviewable diff — which is exactly what the next pass needs.

`scripts/verify-mission-delivery.sh microvm` — the regression net this session was
missing. Everything the microVM track proved by hand was guarded by nothing:

  - THE KERNEL LINE is the assertion that cannot pass by accident. Every other
    check would also pass if the phase had quietly run in a container on the
    gateway; only the kernel says WHERE it ran. Compared against the real gateway
    and node kernels read at start-up rather than pinned to a version, so
    upgrading vmlinux does not manufacture a failure.
  - subagent count > 0, from the server's own count of Claude Code's per-subagent
    transcripts. Before `Agent` was in the allowlist this was structurally
    impossible and nothing said so. A probe that could not run reports "?" and
    FAILS the check rather than reading as zero.
  - the verdict's judge and whether it was independent.
  - negative control, observed passing: a mission whose backend no node can run is
    refused at launch and stays draft. Without it the positive scenario would pass
    just as well against a scheduler that ignored `backend` entirely — which is
    what it did until the first real microvm mission landed on a node with no such
    rootfs.

Also fixed in the harness: `api` now sends the JSON body on STDIN (`curl -d @-`)
instead of interpolating it into a single-quoted argument inside a double-quoted
ssh command. A task description containing "the crate's test suite" ended the
quoting and killed the remote shell; two attempts to escape it were themselves
wrong, because the backslashes must survive bash AND sed AND sh. Removing the
interpolation removes the class, and the next author does not need to know that
apostrophes were forbidden.

475 tests pass, clippy clean.
2026-08-05 22:35:46 -07:00
Omar SobhandClaude Opus 5 da3731d753 fix(missions): a coding phase that delivers nothing is a failure
The last open item in the silent-success class: a coding phase that
changed no files reported `completed` — the same status a phase gets for
delivering tested, reviewed, pushed work. Mission `019fcf62` completed
that way with its agents silently unpinned from the repo, and nothing in
the platform disagreed; it was found by a script diffing the forge.

The verdict is applied at capture rather than at completion, because
capture selects on `status = 'completed'` — the platform does not know
whether a phase produced anything until after it has already finished.

Three conditions must hold before failing a phase, because a false
positive here fails honest work: the phase is a coding phase (research
phases legitimately write nothing to the tree), the diff was actually
computed (an uncomputable diff also reports zero files — blaming the
agent for a platform fault is the same defect wearing different
clothes), and `allow_empty` is not set. Only an explicit `true` opts
out, so a typo leaves the check armed. Registered in phase_config with
its reader named, per the seam-2 rule.

Also closes an ordering hazard this exposed: capture is batched and runs
after a phase completes, so a backlogged mission could close as
'completed' and only then have capture discover an empty phase — leaving
a 'completed' mission holding a 'failed' phase, unfixable because the
mission-close CASE only touches 'running' rows. A repo-bearing mission
now waits for its work to be captured before closing.

Adds a `noop` scenario to the harness: a phase told to change nothing,
which PASSES only when the phase comes back `failed`. Same discipline as
the uid self-test — a check that has never been seen to fire has not
been shown to work.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-04 19:39:14 -07:00
Omar SobhandClaude Opus 5 1253595ba7 fix(missions): stop three launch failures from passing as success
A verification run against the deployed stack found a chain mission whose
phase 0 reported `completed` with zero files, no commit error and no push
error — indistinguishable from a phase that correctly had nothing to do.
Three separate defects had to line up, each of them the same shape: a
failure sharing its representation with a legitimate negative result.

1. `pin_agent_workspaces` embedded the whole config in one `sh -c` argv.
   That works until the file grows — config gains a block per provisioned
   claw — then fails with `argument list too long`. Now written through
   the tar upload API, which has no argv limit, so the failure mode is
   gone rather than merely further away.

2. A failed pin was logged "(continuing)". Without the pin, agents write
   to their sandboxes and the committer finds nothing in /mission/repo —
   the mission cannot deliver, so the launch now fails where someone is
   still looking. The restart that applies the pin is fatal for the same
   reason.

3. `capture_phase_diff_at` swallowed `git diff` failures with
   `unwrap_or_default`, so an unreadable base landed `empty: true,
   files_changed: 0` — byte-identical to an honest no-op. The error is now
   recorded as `diff_error`, and an empty patch that came from a failed
   diff is no longer trusted to mean an unchanged tree.

Adds scripts/verify-mission-delivery.sh, which found #1 and #2 on its
first real run. Its probes are fail-closed: no placeholder values, a
self-test that proves the uid probe can detect the split it looks for,
and FAIL-NORUN for a scenario that never executed. Its own first version
had this bug too — a `die` inside `$(...)` exited the subshell, so a run
that could not authenticate printed "all checks passed" and exited 0.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-04 18:01:02 -07:00