28090d1de0cd676e7f640fff991d506d846a8b8b
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar Sobh
|
a8b8efba6a |
fix(delivery): the on_green_tests gate ran the suite in the live checkout
Fourth instance of the same defect, and the last of the three commands that run as root against a mission tree. `verify_tests` execs the project's test command with `workdir = repo` — the live checkout — inside a container running as ROOT. `cargo test` writes `target/`, so the checkout ends up owned by two uids and the next phase's cargo hits permission-denied. The harness reported `uids=0,65532` the first time this gate ever ran end to end. It survived because it had never run. Every one of the ten harness fixtures used `commit_policy: "always"`; `on_green_tests` and `on_reviewer_approval` were parsed, implemented, and never exercised — and `Gate`'s own doc already records that three recipes carried this policy while it "did precisely nothing" for want of a reader. A policy that is never exercised is indistinguishable from one that is ignored. Consolidated rather than fixed a third time. `root_copy` now owns the pattern — copy through `mission_fs::pack_dir` into a SIBLING of the mission dir, run there, and purge FROM INSIDE THE CONTAINER, because the copy's `target/` is root-owned and the server (uid 65532) cannot delete it. `benchmark_runner` moved onto it; `evaluator_tools::Sandbox` keeps its own copy logic for now (it carries an allow-list and a judge-facing API, so folding it in is a larger change than this moment warrants — noted, not done). The gate fails CLOSED if the copy cannot be made: an unverifiable suite must not license a push. Also adds the `refactor` scenario, which is what found this. I had written it off as "structurally identical to four existing scenarios" — wrong: it is the only recipe carrying `on_green_tests`, and that made it the only one testing this code path at all. 245 lib tests, 20 test binaries. |
||
|
Omar Sobh
|
2a9a62c784 |
test(harness): cover security_hardening — 4 of 5 recipes now run end to end
The third recipe whose defining phase is not `coding`, and so the third that nothing could fail before `PRODUCING_KINDS` widened: a `security_scan` phase that ran no scanner and wrote nothing reported success. One phase, not the recipe's full scan->research->code chain — what is under test is the phase KIND, and the other two kinds are already covered. Two assertions, because the first alone is weak. "Delivered a file" is satisfied by an agent that writes "I scanned it, all clear" and runs nothing — the letter-not-purpose shape this codebase keeps paying for. So the delivered patch must also carry the scanner's OWN output. Verified against the real run: the agent produced gitleaks' banner, INF/ERR lines, byte counts and exit code, not a claim about them. Only `refactor` is now uncovered, and deliberately: its single phase is `coding`, structurally identical to chain/multirole/microvm/noop. It would add runtime and no new signal. security 4/4 against the live fleet. |
||
|
Omar Sobh
|
6dd7937ece |
test(harness): cover the two recipes that had none — research_only and benchmark
The portal offers five workflow recipes. Every one of the harness's seven
fixtures was `research_and_code`, so four recipes had never run end to end —
and that is not a theoretical gap. `research_only` DESTROYED its output for as
long as it existed: `requires_repo = false`, so the capture query's
`AND m.repo_id IS NOT NULL` skipped it, the container was reaped unread, and
eight ClawHDF5 research documents were lost while the mission reported
`completed`. Nothing in 550+ tests could see it, because nothing ran the recipe.
`research-only` asserts the whole chain the loss ran through, not just the
happy end of it:
- the phase completes
- document artifacts exist AT ALL (the missing thing)
- the agent's seven identity files (SOUL.md, MEMORY.md, …) are NOT published
— the first live capture published all seven, because `.git/info/exclude`
cannot protect a mission with no `.git`
- the captured text reads back through the content endpoint, since an
artifact row pointing at nothing is a 404 with no explanation
`benchmark` covers the other half: a benchmark mission is ONE benchmark phase,
and while `empty_delivery_is_a_failure` tested `kind == "coding"` that phase was
exempt — nothing in the platform could fail it. The scenario asserts it both
completes AND delivers files.
Also: `run_scenario` takes an optional `no-checkout`. The single-writer uid probe
is a property OF A CHECKOUT, and a repo-less mission has none by design, so
probing reports a platform fault that is really a category error. It is declared
per scenario rather than inferred from a missing directory — that inference would
silently excuse a repo-BACKED mission whose checkout was reaped early, which is
the exact condition the probe exists to catch.
research-only 4/4, benchmark 3/3 against the live fleet.
|
||
|
Omar Sobh
|
bcf4866abc |
test(harness): a gate that gives up, proven against a real VM
The unit tests prove the plumbing GIVEN `released_at_cap: Some(true)`. They cannot prove the guest writes the marker, that the probe reads it back across the vsock, or that the phase lands `failed` for the right reason — and every one of those is where this class of bug has actually lived. The check is `exit 1`: impossible by construction, so the run exercises the release path rather than hoping to catch it. `blocks` reaching the cap is deliberately NOT the assertion. A healthy agent blocked three times and succeeding on the fourth reports the same 3. The phase STATUS is the assertion; the block count and the failure reason are corroborating checks, so a phase that failed for some unrelated reason cannot pass this. Measured on gw-04 against |
||
|
Omar Sobh
|
cd4d76a8c3 |
test(harness): pick the done_when wording by measuring the judge, not arguing with it
The microvm scenario's judge assertion failed four runs straight. I blamed the
wording twice and rewrote it twice; the second rewrite made it worse. That was
guessing.
With scripts/judge-eval.sh in place the question is cheap to settle. Three
candidate conditions, three draws each, same evidence and same system prompt:
"its second line is …" MET UNMET MET flaky
"records the kernel version …" MET MET UNMET flaky
"contains both … and …" MET MET MET stable
So it was never noise in general — it is a reproducible weakness with
POSITIONAL and EXCLUSIVE phrasings. "its second line is X and nothing else"
invites this judge to invent requirements about the other lines, which is
exactly the reason it kept citing ("the first line contains 'test result: ok'").
Both fixtures now state what the file CONTAINS. The composed one was checked in
both directions — 3/3 MET on good evidence, 3/3 UNMET when the versions are
missing — because a wording that always answers MET would look stable and prove
nothing.
The eval keeps `kernel-ok` failing on purpose; it is the case production hit,
and tuning it green would turn a measurement into a decoration.
Harness: 24/24, including the assertion that had failed four times.
|
||
|
Omar SobhandClaude Opus 5
|
72f8bdc87c |
test(harness): a done_when naming a COMMAND invites the judge to run it
My previous attempt at this made it worse, which is the useful part. The condition said "a Linux kernel release string" and the judge rejected `6.1.128` as "not a Linux kernel release string such as 'Linux 6.1.128'". I rewrote it as "the exact output of `uname -r`" — and the next verdict was that line 2 should be `27.0.0`. The judge has a sandbox and allow-listed commands, so naming a command told it to RUN that command, in ITS OWN container, and compare the file against the answer it got there. The file records a microVM's kernel; the judge was comparing it against the machine the judge runs on. Those are different machines by design — that is the entire point of the assertion. So a `done_when` for a tool-using judge must describe the VALUE's shape, never a command that produces it: "a bare kernel version of the form MAJOR.MINOR.PATCH (for example 6.1.128) and nothing else", plus an explicit instruction not to run uname and not to compare against the local machine, because the file records a different one. The general rule, worth carrying into how `done_when` is written anywhere: a condition phrased as "the output of X" is ambiguous about WHERE X runs, and a judge with tools resolves that ambiguity by running X where it stands. Conditions about a remote or past environment must be stated as properties of the recorded value. The scenario's real proof that the agent ran in a guest is unchanged: a separate comparison of that line against the actual gateway and node kernels, which has passed on every run including the two where the judge disagreed. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
1b556c5849 |
test(harness): say what the condition means, after the judge read it strictly
The restored GLM judge failed a phase that had done the work: MICROVM.md existed with two lines and the second was `6.1.128`, and the verdict was "a kernel version number, not a Linux kernel release string such as 'Linux 6.1.128'". The judge is wrong on the fact — `6.1.128` is exactly what `uname -r` prints, and "release" is the term for it — but the CONDITION was ambiguous, and it is our fixture. "A Linux kernel release string" can be read as either `uname -r` output or `Linux x.y.z`, and a stricter reader is entitled to the second. Both scenarios now say what they mean: the exact output of `uname -r`, a bare version, no prefix. This is not weakening the assertion. The scenario's own kernel check — the one that proves the agent ran in a guest rather than on a host — is a separate, unchanged comparison against the real host kernels, and it PASSED on the same run. What changed is only that the mission-level `done_when` now describes an observable fact precisely, which is what this codebase's own plan-authoring prompt tells models to do. Worth recording rather than papering over: an over-strict independent judge is a much safer failure mode than an over-lenient one, and this is evidence the judge READS the tree instead of rubber-stamping it — the Goodhart incident that motivated cross-provider validation was the opposite failure. But it does mean a vague `done_when` can now cost a phase, which raises the value of `done_when_check` (a shell command, judged by exit status) for anything mechanical. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
f7f3dfe495 |
feat(fleet): GLM as a real microVM backend, and per-role models for claws
Three threads, all of which end at the same place: a mission whose verifier does
not share a model with the coder it reviews.
**GLM has a credential contract now.** `microvm_credential_for` returned one env
var name, which quietly assumed every provider reads its secret from the same
place Anthropic does. It returns a `Credential { source, target }` instead —
z.ai's key lives in the server's `ZAI_API_KEY` and Claude Code reads it as
`ANTHROPIC_AUTH_TOKEN`, and collapsing those two names is what forces a guess at
the other end. A wrong guess here sends one provider's credential to another
provider's endpoint.
`images/agent-glm` is the same CLI at the same pinned version as `agent-claude`
with `ANTHROPIC_BASE_URL` baked in. The split is deliberate: the ENDPOINT is a
property of the image, the CREDENTIAL is a property of the turn. That makes the
dangerous mix-up unrepresentable — a GLM VM cannot be handed an Anthropic
subscription token, and a claude VM cannot be pointed at z.ai. Asserted both
ways, because "the GLM VM must not carry CLAUDE_CODE_OAUTH_TOKEN" is the
property that costs a credential if it ever stops holding.
Kimi stays refused. `KIMI_API_KEY` is set and Moonshot serves an
Anthropic-compatible API, but I have not verified its base URL against the
running service, and this function is precisely where guessing a URL is
expensive. It becomes an arm the day someone measures it.
`api.z.ai` joins the node's default egress allow-list. A default that cannot
run the images we ship is a trap rather than a policy — the alternative is an
operator discovering it as a hung agent with no model access.
**Per-role models for claws** (migration 0071). `template_roles` had no model
column, so `mint_team_from_template` bound every role of every mission team to
one literal — a template whose whole point is an independent reviewer minted a
reviewer sharing a model with the coder. A role may now name its own; roles that
say nothing still take the mint's default, so every template written before this
behaves exactly as it did. The literal is now that default rather than a
hardcode.
**A harness scenario for the roster flow.** `verify-mission-delivery.sh roster`
runs the whole Slice 5 loop — planner proposes, human approves, mission runs —
and asserts the roster LANDED on the mission row rather than trusting the API's
answer. That distinction is not theoretical: the first live approval returned an
error while leaving the proposal marked approved.
Built and proven on tank ahead of the deploy: `clawmates/agent-glm:dev` reports
`2.1.223` and `BASE=https://api.z.ai/api/anthropic`, and
`fc-build-rootfs.sh … glm 8G` boots a VM from it that has git, can write
/mission, and answers `claude --version`.
533 tests pass, clippy clean. Migration 0071.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
abb97e6f03 |
test(harness): a composed scenario, and the stop gate asserted in a real VM
`verify-mission-delivery.sh composed` runs a `team_engine=composed` mission and checks the one property that cannot be checked any other way: a VM is inject → run → collect → destroy, so unless the tree is carried node to node, node 2 boots from the original checkout, sees nothing of node 1's work, and still reports success. The task makes each node append ONE line to STAGES.md, so the delivered file IS the evidence — a run that lost the handoff delivers one line, and no amount of agent confidence can fabricate the missing ones. It also asserts the run's tier is `microvm_graph`. A composed mission that quietly fell back to the solo path would deliver a one-line file and look exactly like a graph that ran one node. `assert_stop_gate` reads the count `phase_runner` reports and distinguishes three outcomes that matter: a number (installed, fired that often), `0` (installed, never needed), and `-` (could NOT be installed — usually a CLI in the image with no `--settings`). Wired into the microvm scenario rather than its own, because it applies to every coding phase on that path. Both ran against the deployed stack: composed — 4/4. STAGES.md carried 5 stage lines through 5 separate VMs (planner → coder → tester → reviewer → committer), each stamped with the guest kernel 6.1.128 rather than the gateway's 6.8.0 or the node's 7.0.0. The run checkpointed 5 steps on the worker, and `updated_at` stayed ~2s old mid-turn, which is the keepalive doing its job — without it `requeue_stale` flips a live run at 180 seconds. microvm — 6/6, including the gate installed in a real VM (`blocks: 0`), one subagent, the GLM judge, and the unavailable-backend negative control. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar Sobh
|
b17e18aa67 |
fix(harness): the verdict check matched psql's display form, not the query's
`select met || ' ' || independent` casts the booleans to `true`/`false`, but the
pattern matched `t`/`f` — psql's *column display* form. So the check reported "no
verdict recorded for the phase" while the row sat in the table saying met=true,
independent=true, glm-4.7.
A check that fails for a reason unrelated to what it checks is worse than no check:
it trains you to ignore the output. The booleans are cast explicitly now so the
shape cannot drift again, and the failure message prints what it actually got.
`verify-mission-delivery.sh microvm` now passes 5/5 against production:
- the agent ran under guest kernel 6.1.128, not the gateway's 6.8.0-124 or the
node's 7.0.0-28 — the one assertion that cannot pass by accident
- the lead delegated to 1 subagent
- the condition was met and judged INDEPENDENTLY by glm-4.7
- the checkout has exactly one writer (uid 65532)
- negative control: a backend no node can run is refused at launch
|
||
|
Omar Sobh
|
9aed20b6d0 |
fix(missions): capture a failed phase's work; harness gains a microvm scenario (#51)
A REGRESSION I INTRODUCED ONE COMMIT AGO. `capture_finished_coding_phases`
selects on `mp.status = 'completed'`, so the moment an unmet phase correctly began
reporting `failed`, its diff stopped being captured, committed or pushed — the work
was silently discarded. Found by the new harness scenario, whose phase legitimately
missed its condition and then had no artifact at all.
What was produced, and whether the goal was met, are different facts. The artifact
records the first; `mp.status` records the second. Capture now covers terminal
phases (`completed`, `failed`), so a phase that did real work and missed its goal
still delivers a reviewable diff — which is exactly what the next pass needs.
`scripts/verify-mission-delivery.sh microvm` — the regression net this session was
missing. Everything the microVM track proved by hand was guarded by nothing:
- THE KERNEL LINE is the assertion that cannot pass by accident. Every other
check would also pass if the phase had quietly run in a container on the
gateway; only the kernel says WHERE it ran. Compared against the real gateway
and node kernels read at start-up rather than pinned to a version, so
upgrading vmlinux does not manufacture a failure.
- subagent count > 0, from the server's own count of Claude Code's per-subagent
transcripts. Before `Agent` was in the allowlist this was structurally
impossible and nothing said so. A probe that could not run reports "?" and
FAILS the check rather than reading as zero.
- the verdict's judge and whether it was independent.
- negative control, observed passing: a mission whose backend no node can run is
refused at launch and stays draft. Without it the positive scenario would pass
just as well against a scheduler that ignored `backend` entirely — which is
what it did until the first real microvm mission landed on a node with no such
rootfs.
Also fixed in the harness: `api` now sends the JSON body on STDIN (`curl -d @-`)
instead of interpolating it into a single-quoted argument inside a double-quoted
ssh command. A task description containing "the crate's test suite" ended the
quoting and killed the remote shell; two attempts to escape it were themselves
wrong, because the backslashes must survive bash AND sed AND sh. Removing the
interpolation removes the class, and the next author does not need to know that
apostrophes were forbidden.
475 tests pass, clippy clean.
|
||
|
Omar SobhandClaude Opus 5
|
da3731d753 |
fix(missions): a coding phase that delivers nothing is a failure
The last open item in the silent-success class: a coding phase that changed no files reported `completed` — the same status a phase gets for delivering tested, reviewed, pushed work. Mission `019fcf62` completed that way with its agents silently unpinned from the repo, and nothing in the platform disagreed; it was found by a script diffing the forge. The verdict is applied at capture rather than at completion, because capture selects on `status = 'completed'` — the platform does not know whether a phase produced anything until after it has already finished. Three conditions must hold before failing a phase, because a false positive here fails honest work: the phase is a coding phase (research phases legitimately write nothing to the tree), the diff was actually computed (an uncomputable diff also reports zero files — blaming the agent for a platform fault is the same defect wearing different clothes), and `allow_empty` is not set. Only an explicit `true` opts out, so a typo leaves the check armed. Registered in phase_config with its reader named, per the seam-2 rule. Also closes an ordering hazard this exposed: capture is batched and runs after a phase completes, so a backlogged mission could close as 'completed' and only then have capture discover an empty phase — leaving a 'completed' mission holding a 'failed' phase, unfixable because the mission-close CASE only touches 'running' rows. A repo-bearing mission now waits for its work to be captured before closing. Adds a `noop` scenario to the harness: a phase told to change nothing, which PASSES only when the phase comes back `failed`. Same discipline as the uid self-test — a check that has never been seen to fire has not been shown to work. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
1253595ba7 |
fix(missions): stop three launch failures from passing as success
A verification run against the deployed stack found a chain mission whose phase 0 reported `completed` with zero files, no commit error and no push error — indistinguishable from a phase that correctly had nothing to do. Three separate defects had to line up, each of them the same shape: a failure sharing its representation with a legitimate negative result. 1. `pin_agent_workspaces` embedded the whole config in one `sh -c` argv. That works until the file grows — config gains a block per provisioned claw — then fails with `argument list too long`. Now written through the tar upload API, which has no argv limit, so the failure mode is gone rather than merely further away. 2. A failed pin was logged "(continuing)". Without the pin, agents write to their sandboxes and the committer finds nothing in /mission/repo — the mission cannot deliver, so the launch now fails where someone is still looking. The restart that applies the pin is fatal for the same reason. 3. `capture_phase_diff_at` swallowed `git diff` failures with `unwrap_or_default`, so an unreadable base landed `empty: true, files_changed: 0` — byte-identical to an honest no-op. The error is now recorded as `diff_error`, and an empty patch that came from a failed diff is no longer trusted to mean an unchanged tree. Adds scripts/verify-mission-delivery.sh, which found #1 and #2 on its first real run. Its probes are fail-closed: no placeholder values, a self-test that proves the uid probe can detect the split it looks for, and FAIL-NORUN for a scenario that never executed. Its own first version had this bug too — a `die` inside `$(...)` exited the subshell, so a run that could not authenticate printed "all checks passed" and exited 0. Co-Authored-By: Claude Opus 5 <[email protected]> |