1678452a93aa9a306da23638c22b98b41d52139f
909
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar SobhandClaude Opus 5
|
deed591da6 |
fix(roster): a rate-limited subscription is a 503 with a reason, not a 500
The retry landed and still failed: all four attempts returned 429. A bare
16-token probe with the same token, straight from gw-04, also returned 429
with `x-should-retry: true` — the Claude Code subscription itself is limited
right now, and no amount of backoff inside one HTTP request will outlast it.
So stop pretending it is a server bug. New `ApiError::Unavailable` → 503,
carrying the one sentence the operator can act on ("clears on its own; try
again shortly"), instead of an opaque `internal error` that sends them into
the logs. The harness now prints the response body rather than the generic
"the planner produced no usable proposal", which is what hid both walls —
first the credit balance, now this.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
c3c4447810 |
fix(planner): wait out a rate limit instead of failing the whole proposal
Moving the roster and planner onto the subscription removed the credit wall and revealed the next one: the harness went from `400 credit balance too low` to `429 rate_limit_error`. A one-shot proposal call had no retry — there is no retry convention anywhere in cm-llm — so a limit that clears in seconds killed the "propose a team" button outright. Four attempts, 2/8/20s backoff, and only for errors that can actually clear: 429/5xx/transport. A 400, 401 or 404 returns immediately, because retrying those is a 30s hang ending in the identical message, which reads to an operator as a stall rather than a bad request. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar Sobh
|
72046e7985 |
fix(planner): server-side model calls run on the subscription, not the metered key
The roster planner died with `400 — "Your credit balance is too low to access the Anthropic API"` while every mission on the same machine kept running. Two Anthropic credentials reach this server and they bill differently: `ANTHROPIC_API_KEY` (sk-ant-api, metered, runs out) and the Claude Code subscription token (sk-ant-oat) that every VM already uses. `Runtime::complete` with a BARE model name — "claude-opus-4-8" — resolves to the default provider, which is the metered key. Three server-side callers did that: the roster planner, the Master Planner, and the claw enhancer. Missions were never affected because `mission_runtime` deliberately sends only the subscription token into a guest; the server had no equivalent rule. `subscription::complete_or` is now that rule, and it is the ONE place a subscription token becomes a provider — `evaluator::subscription_judge` had its own copy, and two of them is how one ends up with a prefix check the other lacks. The `sk-ant-oat` prefix is checked rather than the variable name trusted: an API key pasted into the OAuth slot would authenticate, work, and bill the metered account — the same failure again, discovered weeks later. `web_search` is carried explicitly rather than defaulted. The Master Planner and the claw enhancer both pass `true`, and a helper that quietly dropped it would have taken web search away from two features while every test still passed. `validator_preflight` deliberately keeps `Runtime::complete`: it probes whatever validator spec is configured (today `glm:glm-4.7`), and forcing it onto Anthropic would make it prove the wrong thing. A test pins both halves — no other server-side caller may regress to the metered key, and preflight must keep probing the configured spec. 258 lib tests. |
||
|
Omar Sobh
|
d84d17207f |
feat(placement): place per phase, and let a full fleet queue
Phase 1b: wires the capacity model from
|
||
|
Omar Sobh
|
3a2d76aa43 |
feat(placement): capacity model for the fleet — observed memory is not capacity
Phase 1a of the fleet-intelligence plan: the arithmetic and the inputs. Nothing is wired to it yet; the launch path still picks `capable.first()`. Placement has been `ORDER BY last_seen DESC` + `.first()` — the most recently heartbeated node. Among healthy nodes all heartbeating every 5s that is arbitrary, and it consults nothing about load, so two missions launched together land on the same machine. It did not matter while tank held the only rootfs image. All three nodes serve `claude` as of today. THE correctness point, and the reason this is not a sort change: a VM that booted 30 seconds ago holds a fraction of its 8 GiB claim, so `mem_pct` reports a sold-out node as nearly idle. `capacity_of` takes the WORSE of observed usage and committed usage. The negative control pins it with the measured case — tank at 60 GiB total / 12 GiB observed / 5 VMs booted: utilisation alone says 5 more fit, the node has room for 1. Booking those five is a node in swap, which slows every VM on it together. Commitments are unioned BY IDENTITY, never added: `vm_list` reports booted VMs, `nodes::pinned_microvm_phases` reports phases chosen but not yet booted (a window of seconds in which a real 8 GiB claim exists that no node can report). The deterministic `vm_id_for` is what lets the same phase be recognised in both — counting it twice would shrink the fleet by the number of phases starting. `EvalRow::headroom()` finally gets a caller. It was written with the doc comment "for placement ranking" and has had zero callers since. It is a TIEBREAK, not a gate: ranking is slots first (spread, don't stack), then live headroom, then node id so the same fleet state yields the same answer twice — which `last_seen DESC` could never promise. Fail-closed per house convention: draining, stale health (>30s, tuned just above the 20s offline sweeper), and an unanswerable `vm_list` are all INELIGIBLE rather than low-scoring. Stale Beszel metrics are the one exception — they demote a node to zero headroom instead of excluding it, because they only ever break ties. `FleetAtCapacity` and `FleetUnreadable` are separate variants with a test asserting the second never says "at capacity": an operator sent hunting a load problem that is really a dead daemon wastes the outage. Also names the two nodes that were both called "New node" (tank, morpheus) — a capacity report naming two machines identically is one nobody can act on. 257 lib tests. |
||
|
Omar Sobh
|
5c5f1ced33 |
refactor: the judge's sandbox joins the other three copy sites on root_copy
Four places copy a mission checkout so a ROOT command can run against it without touching the live tree: the judge, the benchmark runner, the on_green_tests gate, and — until now — the judge again, with its own implementation predating the shared one. `evaluator_tools::Sandbox::for_checkout` now builds through `root_copy::RootCopy`. Same packer, same exclusion list, same reasoning in one place. It needs the copy to OUTLIVE the handle, because the judge has not run when `for_checkout` returns and a firing `Drop` would delete the tree out from under it. That is `into_workdir`, a method rather than a `mem::forget` at the call site: the transfer of cleanup responsibility is then visible in the type instead of implied by a leak. `Sandbox::purge` remains what actually clears it, since only a container running as root can remove the root-owned `target/`. What did NOT move: `pack_dir` in the microVM inject/collect path. That marshals a tree to and from a guest over vsock — a transport, not a host-side copy — and folding it in would merge two things that only look alike. 249 lib tests. |
||
|
Omar Sobh
|
8b12245e79 |
chore(image): promote Claude Code 2.1.226 after the canary passed
Canary first, production second — the point of pinning is that the upgrade is a decision, and the point of `canary-claude` is that the decision has evidence. Taken for 2.1.225's fix to a transient 401 that replaced a long-lived CLAUDE_CODE_OAUTH_TOKEN with a short-lived one and broke HEADLESS sessions until restart. Our sessions are headless and our VMs are per-mission, so "until restart" reads as a failed phase. The 2.1.225 workspace-trust prompt does NOT apply: `--help` states the dialog is skipped in non-interactive mode (`-p`, or stdout not a TTY) and we satisfy both. Read from the CLI in a booted 2.1.226 VM rather than inferred from the changelog. Verified on the production path after the rebuild: guest kernel 6.1.128, delegation to a subagent, `--settings` stop gate installed, judge independent (glm-4.7), single writer. 6/6. `rootfs-canary-claude.ext4` is left on tank as the mechanism for the next candidate, not as a leftover. |
||
|
Omar Sobh
|
099a716bfd |
fix(egress): a backend is defined in two maps, and the canary only had one
First canary run failed: phase failed, nothing delivered, and the streamed log said exactly why — "Failed to authenticate. API Error: 403 api.anthropic.com is not on the egress allow-list". Not a 2.1.226 regression. `canary-claude` was added to the server's credential map and not to the node's `provider_hosts`, so the VM booted with a valid subscription token and a door that only opened onto the forge. The fail-closed branch was working correctly: a backend nobody taught that function about reaches no model API, deliberately, so it cannot silently borrow another provider's door. Both maps now name it, each pointing at the other, with a test asserting the canary reaches the same provider as `claude` AND that unknown backends still resolve to nothing. Worth noting what made this a five-second diagnosis instead of an afternoon: the live log streaming built earlier today. The failure was a 403 inside a microVM that no longer exists, and its reason was sitting in the run's checkpoint. |
||
|
Omar Sobh
|
4193ae2cda |
feat(missions): a canary backend for testing a CLI version on the real path
Claude Code 2.1.223 -> 2.1.226 is worth taking (2.1.225 fixes a transient 401
that replaced a long-lived CLAUDE_CODE_OAUTH_TOKEN with a short-lived one and
broke HEADLESS sessions until restart — which for us means a failed phase). But
the image every mission uses is not the place to find out whether a new CLI
still delegates, still accepts `--settings`, and still finishes.
`canary-claude` is a real rootfs built from the candidate version, credentialed
identically to `claude`, so a mission can exercise it through the production
path: egress, stop gate, delegation, delivery, streaming. Testing a new CLI
against a different provider would not be testing the thing we are about to ship.
Named explicitly rather than matched on a prefix. An unrecognised backend must
still be refused at launch — that is what `backend_can_run_a_mission` and the
harness's `microvm-negctl` scenario assert — and loosening the credential map is
exactly how that guard gets softened by accident. A test pins both halves.
Already cleared by direct measurement in a booted 2.1.226 VM, before this:
- `--settings` and `--agents` still exist
- the workspace trust prompt added in 2.1.225 does NOT apply: `--help` states
the dialog is skipped in non-interactive mode (`-p`, or stdout not a TTY).
We use both.
|
||
|
Omar Sobh
|
8c93cd8569 |
fix(runs): the composed worker's checkpoint wiped the live log on every node
Composed missions streamed ZERO bytes while solo missions streamed fine. Same executor, same command, same guest — `HubVms::run` is a straight passthrough — and the node logged a tail starting for all five graph nodes against the correct outer run id, with no errors. The bytes simply were not there at the end. Two writers, one column. `fleet.rs` appends live output under `checkpoint.log`; `topology_runs::checkpoint` wrote `SET checkpoint = $2`, replacing the whole object. A composed run checkpoints after EVERY graph node, so each node's progress silently erased the log written during it. A solo run has no second writer, which is exactly why it looked like it worked. Now merged with `||`. The keys are disjoint, so the progress object still wins for everything it owns. I was wrong about the cause twice before finding this. First I blamed the guest agent's serial accept loop — real, fixed, and not this. Then I blamed pipe buffering racing the abort at turn end — plausible, and the drain fix is right on its own merits, but composed still streamed zero afterwards, which is what ruled it out. The thing that actually located it was noticing solo and composed differ by a WRITER, not by a code path. |
||
|
Omar Sobh
|
09afa7e7ff |
fix(node): aborting the tail at turn end raced the flush that matters most
Composed runs streamed NOTHING while solo runs streamed fine — same code path, `HubVms::run` is a straight passthrough, and the node logged a tail starting for all five graph nodes with the correct outer run id. The difference was timing. `claude -p ... | tee` makes stdout a PIPE, so the CLI block-buffers and flushes at EXIT. The most valuable output — the agent's summary of what it did — arrives in the instant the turn ends. The node aborted the tail the moment `handle_op` returned, so that flush was a race: a solo turn (minutes long, output already flushed by size) won it and streamed 337 bytes; each node of a composed run (~20s) lost it and streamed zero. The tail now DRAINS. A flag is set when the turn returns, and the loop exits only after a pass that read nothing new — checked AFTER a read, never before one, because exiting on the flag alone would drop exactly the bytes this exists to capture. Bounded by a 20s timeout with the abort kept as a backstop rather than the mechanism, so a VM that stopped answering cannot hold the task open. Worth naming: 5 tails started, 5 logged cleanly, 0 bytes arrived. Every individual step reported success and the feature did nothing — the same shape as the empty Live tab this whole thread began with, one layer down. |
||
|
Omar Sobh
|
5b49d5a1a8 |
feat(merge): gate publication on the merged tree's own tests
The other half of the merge button. Merging told you the branch went in; nothing
checked that what came out still worked.
Verified BEFORE publishing, not reverted after. `merge_locally` and
`push_merged` are separate functions so the caller can run the project's tests
between them, which means a merge that breaks the base is simply never pushed —
`main` is not broken for however long it takes someone to notice. A test asserts
`merge_locally` contains no push, because the moment it does, verification
becomes after-the-fact and the guarantee is gone.
Outcomes, all reported to the operator rather than swallowed:
Passed -> published
NoSuite -> published, and SAID so; a repo with no tests is a fact about the
repo, not a pass
Failed -> not published, exit code reported, branch untouched so it can be
fixed and merged again
CouldNotRun -> not published. Fail closed: a suite that could not run has not
passed, and publishing on "we could not check" is how a green
main stops meaning anything.
`verify_tests` runs `cargo test` as ROOT in a container, so the merge workdir
ends up holding a root-owned `target/` the server (uid 65532) cannot delete —
the same leak found three times today. Purged through the container before the
ordinary cleanup.
248 lib tests.
|
||
|
Omar Sobh
|
28090d1de0 |
fix(node): the log tail gave up before the turn wrote its first byte
First live test of the streaming path: mission passed 6/6, `checkpoint.log` was 0 bytes, and the node logged nothing at all. `stream_vm_log` treated "no progress" as "the turn finished writing". But the guest's `tail` reports EOF after every idle window, and the FIRST idle window is always the one before any output exists — the VM is still booting and the CLI still starting. So the tail returned `at == 0`, the node concluded the turn was done, and it stopped seconds into a run that then went on for minutes. The abort is the terminator, not idleness: the caller already aborts this task when the exec returns, so waiting cannot outlive the turn. No-progress now sleeps and retries instead of returning. Also logs when a tail STARTS. The bug was invisible in exactly the way this session keeps finding: silence on the success path, silence on the give-up path, and an empty Live tab that looked identical to a feature nobody had wired. Method note, since it cost time: I tried to confirm the deployed binary by grepping it for `vm_out` and found zero — then found zero for `pty_out` and `vm_exec` too, in a binary whose PTY streaming demonstrably works. Binary-grep is not a reliable presence test for these literals; `stream_vm_log` and `tail of` being present is what actually showed the code had shipped. |
||
|
Omar Sobh
|
0b89b8316c |
feat(observability): stream a microVM turn's stdout/stderr to the platform live
The Live tab showed nothing while a turn ran, and the agent's own account of it
went to stderr on the node and nowhere a user could reach. This is the path that
carries it.
The blocker was the guest agent. `fcagent` handled one connection at a time,
inline, so during an hour-long turn the VM accepted nothing — which is why every
existing probe (subagents, stop-gate blocks, cap) runs AFTER the turn rather than
during it. It now spawns a thread per connection, wrapped in `catch_unwind`
because this process is pid 1: a panic used to take the accept loop with it, and
an unbootable VM is a far worse outcome than a missing log. A failed spawn logs
and keeps accepting rather than dropping the listener.
PROVED against a live VM before building on it, since "sound reasoning about this
system" and "measurement" have diverged repeatedly today. Patched rootfs, booted
under Firecracker, ran an 8s exec and a concurrent tail:
exec took 8.0s ok=True
+0.0s 'line1\nline2\n' +1.2s 'line4\n' +3.2s 'line6\n' +6.0s 'DONE\n'
VERDICT: CONCURRENT — tail returned data before exec finished
The rest is the pattern the terminal already uses. New `tail` op streams a file
by OFFSET (so a dropped link resumes instead of replaying, and the tail always
terminates — one that never returns pins a thread for the life of the VM). The
node follows the log alongside the turn and pushes `Uplink::VmOut { run_id, at,
data }` over the WebSocket it already holds, mirroring `PtyOut`. The server does
what `PtyOut` deliberately does not: it APPENDS to the run's checkpoint as well
as fanning out, because a terminal has no history worth keeping and a mission log
is the record of what the agent did. `run_events_sse` emits the new bytes as
`step` events, which the live pane already renders — no frontend change.
The turn is `tee`d, not redirected: the file feeds the live stream and stdout
still becomes `VmOutcome::summary`. A redirect would have produced a live view
and an empty summary, which is the same green-and-empty shape as the bug this
fixes. Tested, along with the log living outside the collected tree so it never
lands in a user's delivered diff.
246 lib tests, 20 binaries; node and fcagent build clean.
|
||
|
Omar Sobh
|
62509a5090 |
fix(missions): a solo microVM run showed the operator an empty Live and Output tab
Found by a frontend wiring sweep, then confirmed in the database.
Everything the UI shows of a run's CONTENT reads
`topology_runs.checkpoint.records`: `/api/missions/{id}/documents` behind the
output reader, and `/api/topology-runs/{id}/events` behind the live pane. The
`team` and `microvm_graph` tiers write those records. The SOLO microVM path
never did — it updated `status` and nothing else:
tier | checkpoint_null | records
microvm_graph | f | 2-5
team | f | 5
microvm | t | 0 <-- every one
So a single-phase microVM mission ran real work, delivered a real branch, and
showed an empty Live tab and an empty Output tab. The agent's own account of the
turn went to stderr via eprintln and nowhere a user could reach.
Note what was NOT broken, since that was the initial suspicion: the SSE path
matches (`/api/topology-runs/{id}/events` on both sides), and a sweep of all 130
frontend `/api/` calls against the 164 registered routes found zero genuinely
missing endpoints. The wiring was fine; the data was absent.
The run now persists its turn as one record shaped exactly like the ones those
two readers already parse — `node_id`, `role` (the phase kind), `phase`,
`output` — so no reader changes. Written with `checkpoint || $3::jsonb` so a
future writer of other checkpoint keys is not clobbered.
246 lib tests.
|
||
|
Omar Sobh
|
3616bc4733 |
feat(missions): an operator button to merge a mission's branch into main
`MergePolicy::Never` — the default for anything touching code — has always meant
"do not merge on your own", deferring to a human. There was no way for that human
to say yes: `auto_merge` was reachable only from the paper-harvest path, no
workflow template declares `merge_policy`, and every mission ended at a branch.
`POST /api/missions/{id}/merge` is that yes, with a button on the artifacts tab.
The additive-only gate does NOT apply here, deliberately: an operator reading a
code change is exactly the judgement the policy was holding out for.
What is not waived:
- the branch comes from the artifact delivery RECORDED, not rebuilt from the
mission id, and must have `pushed: true`. A phase that never pushed shows no
button instead of one that cannot work.
- an empty branch is refused. A button reporting success for merging nothing
is worse than no button.
- a conflict refuses, aborts, and leaves the repo clean rather than forcing.
It works in a FRESH CLONE under `_merge/<mission>`, never the mission checkout:
that directory is reaped on a timer after a mission ends, so a merge using it
would succeed right after a run and fail inexplicably an hour later. The clone is
made by the server process, so nothing runs as root and ordinary cleanup works —
unlike the copies in `root_copy`.
`merge_and_push` is split out so the operator path and the automatic path run the
SAME git commands; only the gates differ. A test asserts both call it, that the
operator path does not re-apply the additive gate it exists to bypass, and that
it still refuses an empty branch.
Harness 43/43 across all five recipes before this change, with `_gate`, `_bench`
and `_verify` all at zero.
246 lib tests, 20 binaries, 89 frontend tests, clean build.
|
||
|
Omar Sobh
|
a8b8efba6a |
fix(delivery): the on_green_tests gate ran the suite in the live checkout
Fourth instance of the same defect, and the last of the three commands that run as root against a mission tree. `verify_tests` execs the project's test command with `workdir = repo` — the live checkout — inside a container running as ROOT. `cargo test` writes `target/`, so the checkout ends up owned by two uids and the next phase's cargo hits permission-denied. The harness reported `uids=0,65532` the first time this gate ever ran end to end. It survived because it had never run. Every one of the ten harness fixtures used `commit_policy: "always"`; `on_green_tests` and `on_reviewer_approval` were parsed, implemented, and never exercised — and `Gate`'s own doc already records that three recipes carried this policy while it "did precisely nothing" for want of a reader. A policy that is never exercised is indistinguishable from one that is ignored. Consolidated rather than fixed a third time. `root_copy` now owns the pattern — copy through `mission_fs::pack_dir` into a SIBLING of the mission dir, run there, and purge FROM INSIDE THE CONTAINER, because the copy's `target/` is root-owned and the server (uid 65532) cannot delete it. `benchmark_runner` moved onto it; `evaluator_tools::Sandbox` keeps its own copy logic for now (it carries an allow-list and a judge-facing API, so folding it in is a larger change than this moment warrants — noted, not done). The gate fails CLOSED if the copy cannot be made: an unverifiable suite must not license a push. Also adds the `refactor` scenario, which is what found this. I had written it off as "structurally identical to four existing scenarios" — wrong: it is the only recipe carrying `on_green_tests`, and that made it the only one testing this code path at all. 245 lib tests, 20 test binaries. |
||
|
Omar Sobh
|
a4b4d05b8d |
fix(evaluator): the verification sandbox leaked for the same reason the bench copy did
Found by checking `_verify` after fixing the identical bug in `_bench`: 16 MB
stranded across two copies, the oldest hours old.
`Sandbox::Drop` calls `std::fs::remove_dir_all` as uid 65532. The judge runs
`cargo test` in a container as ROOT — that is the entire point of the sandbox —
so the copy's `target/` is root-owned and the removal fails on it, leaving the
whole tree. The error was logged to a stream nobody reads, so the sandbox that
exists to protect the checkout quietly filled the disk instead.
Its doc comment also claimed "the copy lives under `_verify/<mission>`, which
the next pass clears anyway". That was wrong for exactly the same reason:
`for_checkout` removes a stale root before copying, with the same uid, and fails
the same way. A leaked copy was permanent, not transient.
`Sandbox::purge` removes it from inside the container, as root, where it was
written. `evaluate` now wraps its body so the purge runs on EVERY exit — that
function returns from several branches, and cleanup only some paths reach is the
same as no cleanup on the others. `Drop` stays as a fallback for the early paths
where nothing has run as root yet, and its comment no longer claims otherwise.
This is the third instance today of the same shape: cleanup that cannot clean up,
invisible because the failure was swallowed. The others were the leaked agent
containers in the runtime tests and the bench copy in
|
||
|
Omar Sobh
|
e89a32ffef |
fix(benchmark): the bench copy leaked because only root could delete it
The copy fix in
|
||
|
Omar Sobh
|
a93a4111e1 |
fix(benchmark): the baseline runner was writing root-owned files into the checkout
Caught by the harness: `benchmark: checkout has multiple writers (uids=0,65532)`.
The previous full run passed that same check, so this was introduced by wiring
`benchmark_runner` into the sweep one commit ago.
`docker_exec` enters a container running as ROOT with the missions root
bind-mounted, and `cargo bench` writes `target/`. Run in the live tree it leaves
root-owned build output in a checkout owned by uid 65532 — the single-writer
invariant broken, and the next phase's cargo hitting permission-denied on a
directory it cannot write.
This is the SAME defect `evaluator_tools::Sandbox` was written for, found by the
same probe, and fixed the same way: benchmark a COPY. `BenchCopy` packs the
checkout through `mission_fs::pack_dir` (so it excludes exactly what the
delivered diff excludes — one exclusion list, now four consumers) into
`<missions_root>/_bench/<mission>`, a SIBLING of the per-mission dirs like
`_verify` and `_outputs`, so a mission reap cannot race a running bench. Removed
on drop, including on error paths.
The operator-triggered path (POST /api/missions/{id}/benchmark) had this bug
from the start and is fixed by the same change — it shares `run`.
Worth naming the pattern: measurement must not mutate what it measures. It
applies to the judge, to the `verifier` subagent that has no Edit or Write, and
now to the benchmark runner.
243 lib tests, zero warnings.
|
||
|
Omar Sobh
|
0d8db7ff0b |
fix: close the three remaining gaps, and repair a test I silently disabled
FIRST, the self-inflicted one. My edit in |
||
|
Omar Sobh
|
2a9a62c784 |
test(harness): cover security_hardening — 4 of 5 recipes now run end to end
The third recipe whose defining phase is not `coding`, and so the third that nothing could fail before `PRODUCING_KINDS` widened: a `security_scan` phase that ran no scanner and wrote nothing reported success. One phase, not the recipe's full scan->research->code chain — what is under test is the phase KIND, and the other two kinds are already covered. Two assertions, because the first alone is weak. "Delivered a file" is satisfied by an agent that writes "I scanned it, all clear" and runs nothing — the letter-not-purpose shape this codebase keeps paying for. So the delivered patch must also carry the scanner's OWN output. Verified against the real run: the agent produced gitleaks' banner, INF/ERR lines, byte counts and exit code, not a claim about them. Only `refactor` is now uncovered, and deliberately: its single phase is `coding`, structurally identical to chain/multirole/microvm/noop. It would add runtime and no new signal. security 4/4 against the live fleet. |
||
|
Omar Sobh
|
6dd7937ece |
test(harness): cover the two recipes that had none — research_only and benchmark
The portal offers five workflow recipes. Every one of the harness's seven
fixtures was `research_and_code`, so four recipes had never run end to end —
and that is not a theoretical gap. `research_only` DESTROYED its output for as
long as it existed: `requires_repo = false`, so the capture query's
`AND m.repo_id IS NOT NULL` skipped it, the container was reaped unread, and
eight ClawHDF5 research documents were lost while the mission reported
`completed`. Nothing in 550+ tests could see it, because nothing ran the recipe.
`research-only` asserts the whole chain the loss ran through, not just the
happy end of it:
- the phase completes
- document artifacts exist AT ALL (the missing thing)
- the agent's seven identity files (SOUL.md, MEMORY.md, …) are NOT published
— the first live capture published all seven, because `.git/info/exclude`
cannot protect a mission with no `.git`
- the captured text reads back through the content endpoint, since an
artifact row pointing at nothing is a 404 with no explanation
`benchmark` covers the other half: a benchmark mission is ONE benchmark phase,
and while `empty_delivery_is_a_failure` tested `kind == "coding"` that phase was
exempt — nothing in the platform could fail it. The scenario asserts it both
completes AND delivers files.
Also: `run_scenario` takes an optional `no-checkout`. The single-writer uid probe
is a property OF A CHECKOUT, and a repo-less mission has none by design, so
probing reports a platform fault that is really a category error. It is declared
per scenario rather than inferred from a missing directory — that inference would
silently excuse a repo-BACKED mission whose checkout was reaped early, which is
the exact condition the probe exists to catch.
research-only 4/4, benchmark 3/3 against the live fleet.
|
||
|
Omar Sobh
|
a20702d55b |
fix(evaluator): a bare validator model name claimed independence it never had
`CLAWMATES_VALIDATOR_MODEL=gemini-2.5-flash` (or any bare model name) produced
an Anthropic judge grading Anthropic work, recorded `independent = true`.
The chain:
- `provider_family` reads the SPEC. A bare `gemini-2.5-flash` matches none of
the known needles, so it returns "unknown" — deliberately NOT "anthropic",
so it passes the `family == IMPLEMENTER_FAMILY` guard.
- `Runtime::resolve_provider` (runtime.rs:224) falls back to the DEFAULT
provider for any spec it cannot route. A bare name has no `provider:` to
route on, so it silently returns the house Anthropic provider.
- The existing "no provider registered" guard checks `model.contains(':')`.
That works for `glm:glm-4.7` — an unrouted colon-spec comes back carrying
its colon — and can NEVER fire for a bare name.
So the one guarantee this path exists to make (the judge is not the implementer)
was reported as satisfied while being violated. That is the same shape as the
Goodhart incident the independent judge was built after: not a wrong answer, a
wrongly-trusted one.
A validator spec must now name its provider. `names_a_provider` is a named
predicate rather than an inline `contains(':')` so the rule is testable and the
reasoning has somewhere to live.
Found while auditing my own Gemini removal — which turned out to be
behaviour-neutral here (a gemini spec went from family "gemini" to "unknown",
both non-anthropic, same verdict). The bug is pre-existing and independent of
it; removing Gemini only made the bare `gemini-*` spelling more likely to be
left behind in someone's env.
Live config is `glm:glm-4.7`, a proper registry spec, so production behaviour is
unchanged. Negative control: make `names_a_provider` return true unconditionally
and `a_validator_spec_must_name_its_provider` fails.
241 lib tests pass.
|
||
|
Omar Sobh
|
87f188ae73 |
refactor: strip Gemini from the platform, and level up the architecture_mapper
Two things.
1. The architecture_mapper proposal, applied AND made durable.
The GLM proposal (019fddd9) was accepted in full: the agent's system_prompt now
carries the Mermaid-first constraint and its brain was rewritten. Both verified
against the live row and the .h5 file.
But `apply_identity` writes `UPDATE agents SET system_prompt` and
`apply_brain_consolidation` writes that agent's brain — neither touches the team
TEMPLATE. That agent is mission-scoped, so the improvement would have died with
the mission. The model's actual insight was sharp and worth keeping: "Mermaid
diagrams beat prose" lived in the brain SEED and not in the system PROMPT, so it
only applied when the agent happened to consult its brain. That constraint is
now in templates/teams/codebase_research.toml, where every future Codebase
Research team inherits it.
(The proposal's second item mostly restated anti-patterns the seed already
lists, so the seed is unchanged. Applying an LLM's suggestion is not the same as
agreeing with all of it.)
2. Gemini is gone.
Removed: the `gemini.default` provider alias and its `is_exact_provider_match`
prefix, GEMINI_API_KEY forwarding to agent containers, the evaluator's
gemini->gemini family row, the model selectors in claws/teams/planner and in
TeamWizard + AgentComputer, and the commented provider block in the runtime
config example (whose ZEROCLAW_AGENT_MAP example still mapped a worker_gemini
that no longer existed).
`provider_alias_for("gemini")` now returns claude_cli.default via the
unrecognised-model branch, which LOGS. A stray gemini binding degrades visibly
rather than resolving to a provider row we no longer ship. A test pins that, and
another pins that GEMINI_API_KEY is forwarded in NEITHER auth mode, so adding it
back to the list is a visible change rather than an accident.
Avatar generation is DELETED, not disabled — it called Gemini's image model, and
there is no alternative: Claude and Kimi are text-only, and z.ai answers
"Unknown Model" for cogview-3-flash and cogview-4 on our plan (measured, not
assumed). AvatarModal keeps UPLOAD, which never needed a provider; only the
prompt-generation half is gone.
240 backend lib tests, 89 frontend tests, clean tsc + eslint, build succeeds.
|
||
|
Omar Sobh
|
f6c3ddbf81 |
refactor: no feature depends on Gemini any more
Depleted Gemini prepayment credits took out PDF rendering. The same key was the
only thing standing between level-up proposals and the same fate, so both are
off it.
- `pdf_renderer` is DELETED, not disabled. Nothing sets `render_pdf: true` since
markdown became the deliverable (
|
||
|
Omar Sobh
|
821cbb8622 |
feat(missions): hold every producing phase to delivering, and read markdown instead of PDFs
Two changes the portal review asked for.
1. `benchmark` and `security_hardening` had no delivery guarantee.
`empty_delivery_is_a_failure` tested `kind == "coding"`, on the reasoning that
"research phases legitimately write nothing to the tree" — which the research
directive three modules over contradicts, since it tells the agent to save
findings under /mission/repo/research/. The cost: a `benchmark` mission is ONE
benchmark phase, and with that phase exempt nothing in the platform could fail
it. Same for `security_hardening`, whose first two phases are security_scan and
research.
Now keyed on PRODUCING_KINDS = coding, research, benchmark, security_scan.
`review` stays exempt — a reviewing phase that changes nothing has done its job,
the same distinction `vm_stop_gate::per_node` makes. The test that encoded the
old rule is rewritten rather than deleted, with the reasoning that replaced it.
All 8 harness fixtures are coding phases, so harness behaviour is unchanged.
2. PDFs are dropped; markdown is the deliverable.
Rendering a PDF meant asking an LLM to convert markdown to HTML — a paid API
call per document, on the critical path of "let me read my research", which
failed on depleted Gemini credits and left every artifact unreadable. Styling at
render time is free, offline, instant and cannot 429.
- `mission_outputs` no longer requests a render.
- New `GET /api/missions/{id}/artifacts/{artifact_id}/content`. The frontend had
no way to READ an artifact at all: it listed paths and offered a PDF preview
that never rendered (and whose `rendered_pdf_path` had no route serving it).
Two containment rules, both enforced: the artifact must belong to a mission in
the caller's workspace, and the CANONICALISED path must stay under `_outputs`
— canonicalise first, because checking the string before resolving `..` is the
classic hole.
- `MarkdownBlock` now uses react-markdown + remark-gfm + rehype-slug. It was a
deliberate zero-dep renderer for "the subset the refiner emits", and that
subset stopped matching reality: agent briefs are largely GFM pipe tables,
which it showed as literal pipes. MissionOutputReader and RefineDiffModal use
the same component and gain tables for free.
- Heading ids come from rehype-slug and `outlineOf` slugs with the same
GithubSlugger, so the outline rail's anchors still resolve. A test pins that
invariant, including duplicate headings.
Styles live in globals.css under `.md-view`: the markup is generated so there
are no class hooks, and this project has no styled-jsx registry — the app-router
requirement is documented in next/dist/docs/01-app/02-guides/css-in-js.md, which
frontend/AGENTS.md exists to make me read.
The artifacts tab moved to `MissionArtifacts.tsx`. MissionCanvas was 1341 lines
against a 1250 limit BEFORE this change — already failing lint; it is now 1248.
238 backend lib tests, 20 backend test binaries, 89 frontend tests, clean tsc,
clean eslint on every file touched, production build succeeds.
|
||
|
Omar Sobh
|
da889f83ab |
fix(missions): an empty repo-less phase was re-processed on every tick forever
The guard added in
|
||
|
Omar Sobh
|
c28c7a148f |
fix(pdf): the renderer resolved every artifact path against the wrong root
`render_one` joined `missions_root()/<mission_id>/` before the artifact path, producing `<root>/<mission>/_outputs/<mission>/<phase>/...` — the mission id twice, and no such file. Artifact paths are relative to the MISSIONS ROOT. All three registration sites write `_outputs/<mission>/<phase>/...`, and `_outputs` is deliberately a sibling of the per-mission directories so it survives their reaping; joining the mission id first put the lookup inside the very directory `_outputs` exists to escape. It went unnoticed because until now the only artifacts on the system were `code_diff` rows registered with `render_pdf: false`, which this worker never reads. `produces = ["md","pdf"]` was inert, so nothing ever asked for a render. The first artifacts to ask were the first to find it — both failed with ENOENT on the doubled path. Negative control: restore the extra join and `an_artifact_path_resolves_against_the_missions_root` fails. The worker's error handling is sound and needed no change: it recorded `render_pdf_status = 'failed'` with the full path in `render_pdf_error`, which is how this was diagnosed in one read. 237 lib tests pass. |
||
|
Omar Sobh
|
89bc53b53d |
fix(missions): repo-less capture was publishing the agent's own identity files
First live run of `capture_repo_less_phases`: 9 artifacts, of which 2 were the user's research. The other 7 were AGENTS.md, HEARTBEAT.md, IDENTITY.md, MEMORY.md, SOUL.md, TOOLS.md and USER.md — the agent runtime's identity scaffolding, seeded into the workspace root because that root is pinned to the repository root. The codebase already knew about these files and already had the list. What it did not have is a defence that works without a repo: `ignore_agent_scaffolding` writes them to `.git/info/exclude`, and a mission with no repository has no `.git`. So the exact files that once got committed into a user's repo and pushed (the reason that list exists) came back through a new channel. `AGENT_SCAFFOLDING` is now `pub(crate)` and `mission_outputs` filters on it directly — one list, two consumers, so the next file the runtime starts seeding is excluded from both at once rather than from whichever was remembered. Negative control: replace the filter with `&& true` and `research_documents_are_kept_and_scaffolding_is_not` fails. Found by running it against a live mission, not by reading it. The unit tests passed the whole time — they seeded a tree that did not contain the scaffolding, because I did not know it would be there. |
||
|
Omar Sobh
|
ceab28b902 |
fix(missions): a repo-less mission threw away everything its agents wrote
`capture_finished_coding_phases` selects `AND m.repo_id IS NOT NULL`. Every
`research_only` mission is repo-less by design (`requires_repo = false`), so the
whole capture path — including the `sync_out` that copies the agent's work OUT
of the container — never ran, and the container was reaped unread.
Measured on the real mission `019fdc35` ("ClawHDF5 Research"): four agents, 9.5
minutes, EIGHT research documents — an HDF5 parser design, a Rust ecosystem
survey, a seven-crate dependency map, tracing and fuzzing strategy. Result:
`mission_artifacts` = 0, mission `completed`. Not recoverable: no container, no
volume, nothing under the missions root.
The platform did not merely fail to save the work — it INSTRUCTED it. The task
preamble tells every agent "/mission/repo ... is the mission's git checkout",
whether or not one exists, and the research directive says to save findings
there. One agent recorded the contradiction verbatim: "No git repo — file is
written." It looked, saw no repo, complied anyway.
Three changes, one per link in that chain:
1. `mission_outputs::capture_repo_less_phases` — copies `/mission/repo` out of
the container and registers each file as an artifact under `_outputs/`,
which is a SIBLING of the mission dir and survives `teardown_container`.
This is also the code that finally reads `produces`, until now an inert key:
`produces = ["md","pdf"]` now drives `render_pdf` into the existing
pdf_renderer worker.
2. The preamble is conditional. A repo-less mission is told its workspace is
scratch, that git_operations has nothing to act on, and — the part that
matters — that files left there ARE collected and published. An agent told
only "there is no repo" has no reason to write anything to disk.
3. A repo-less phase that produced no files is FAILED, unless it declares
`allow_empty`. The same rule `empty_delivery_is_a_failure` applies to coding,
for the only channel these phases have. Note this is NOT that guard widened:
it keys on `files_changed`, which is meaningless with no checkout, and would
not have saved the ClawHDF5 documents.
Negative controls, each ablated and confirmed failing: ignore `has_repo` and the
preamble test fails; empty the skip-list and the capture test keeps `.git` and
`node_modules`; write artifacts inside the mission dir and the survives-the-reap
test fails.
236 lib tests pass.
|
||
|
Omar Sobh
|
bcf4866abc |
test(harness): a gate that gives up, proven against a real VM
The unit tests prove the plumbing GIVEN `released_at_cap: Some(true)`. They cannot prove the guest writes the marker, that the probe reads it back across the vsock, or that the phase lands `failed` for the right reason — and every one of those is where this class of bug has actually lived. The check is `exit 1`: impossible by construction, so the run exercises the release path rather than hoping to catch it. `blocks` reaching the cap is deliberately NOT the assertion. A healthy agent blocked three times and succeeding on the fourth reports the same 3. The phase STATUS is the assertion; the block count and the failure reason are corroborating checks, so a phase that failed for some unrelated reason cannot pass this. Measured on gw-04 against |
||
|
Omar Sobh
|
b36ae00ea5 |
fix(missions): a gate that gave up completed the phase green
`done_when_check` is run in exactly one place: the Stop hook inside the guest. Nothing outside it has ever re-run the command — not the evaluator (which judges the PROSE `done_when`), not capture, not delivery. The hook is capped at MAX_BLOCKS so a stuck agent cannot wedge the turn. At the cap it logs `cap: <reason>` and exits 0, releasing the agent with its check still failing. That release was invisible: rc was 0 and the work collected, so both signals the run status was decided from said "fine", and the phase completed. Green phase, unmet condition, no error anywhere — the same silent-success shape this project keeps paying for. The block COUNT cannot fix it. Three blocks then a stop that finally passed and three blocks then a surrender both report `blocks: 3`, and they are opposite outcomes. So the gate now writes a `capped` marker file, probed back out of the guest alongside the block count, and `Some(true)` fails the run on BOTH paths — solo (phase_runner) and composed (microvm_turn_executor). A marker file rather than grepping the log: a block reason embeds the check's own output, so an output line starting `cap:` would read as a release that never happened. Also corrects the comment in `per_node` that sent me looking. It claimed "the phase-level check still runs post-hoc", conflating two mechanisms — that is true of `require_changes` (via `empty_delivery_is_a_failure`) and was never true of `check`. Negative controls, both ablated and confirmed failing: drop the enforcement and `a_node_whose_gate_gave_up_fails_the_run` fails; stop writing the marker and `a_gate_that_gives_up_records_that_it_gave_up` fails. And the control against over-strictness — `a_node_that_was_blocked_and_then_succeeded_passes` — is why this keys on the marker instead of the count. 231 lib tests pass. |
||
|
Omar Sobh
|
cd4d76a8c3 |
test(harness): pick the done_when wording by measuring the judge, not arguing with it
The microvm scenario's judge assertion failed four runs straight. I blamed the
wording twice and rewrote it twice; the second rewrite made it worse. That was
guessing.
With scripts/judge-eval.sh in place the question is cheap to settle. Three
candidate conditions, three draws each, same evidence and same system prompt:
"its second line is …" MET UNMET MET flaky
"records the kernel version …" MET MET UNMET flaky
"contains both … and …" MET MET MET stable
So it was never noise in general — it is a reproducible weakness with
POSITIONAL and EXCLUSIVE phrasings. "its second line is X and nothing else"
invites this judge to invent requirements about the other lines, which is
exactly the reason it kept citing ("the first line contains 'test result: ok'").
Both fixtures now state what the file CONTAINS. The composed one was checked in
both directions — 3/3 MET on good evidence, 3/3 UNMET when the versions are
missing — because a wording that always answers MET would look stable and prove
nothing.
The eval keeps `kernel-ok` failing on purpose; it is the case production hit,
and tuning it green would turn a measurement into a decoration.
Harness: 24/24, including the assertion that had failed four times.
|
||
|
Omar SobhandClaude Opus 5
|
5c066afa7b |
test: stop leaking a container per run, and add the project's first eval
TWO FINDINGS, one from cleaning up and one from refusing to keep guessing. THE LEAK. `./scripts/test.sh` left three containers running every time — 289 had accumulated. The cause was a comment that lied: `warm_pool.rs` said "Shutdown destroys assigned AND pooled sandboxes", while `SandboxManager::shutdown` drains the POOL only. Its own doc says why — assigned sandboxes persist deliberately so a redeploy can reuse them, and production reaps the strays with `reconcile_orphans` at boot. A test has no next boot, so each one that assigned a sandbox simply left it running. The three tests now call the `release_agent` that already existed, and the comment says what the code does. Verified: 0 leaked, where the same run leaked 3 before. THE EVAL. The independent judge failed the same correct phase FOUR times, each time citing a different invented requirement. I blamed the condition's wording twice and rewrote it twice — the second rewrite made it worse, by naming a command a tool-using judge then ran in its own container. Then a control showed the same model answering MET to the same question asked directly, and a third wording test showed a STRICTER phrasing scoring UNMET. Prose wording was not the variable. Continuing to iterate would have been fitting the fixture to noise. `scripts/judge-eval.sh` measures the thing instead: five cases drawn from real incidents, each with an answer a careful human would agree with. This project has 557 tests and had zero evals, which is backwards — a test pins OUR code, an eval pins the MODEL, and the model changes without us touching anything. The result is why it was worth building: glm-4.7 4/5 — wrong on kernel-ok: says UNMET when MET kimi-for-coding 4/5 — wrong on goodhart: says MET when UNMET Identical scores, opposite failure modes. GLM fails good work; KIMI passes work where 14 assertions were deleted and the failing module removed to make a suite "pass" — the exact incident the verifying judge was built after. Swapping the validator to Kimi because it passes our failing case would have installed a rubber stamp. Keep GLM: a judge that is too strict costs a re-run, a judge that is too lenient costs the guarantee. The eval also caught a bug in itself before I trusted it: Kimi answers with a `thinking` block first, and a 160-token budget was consumed entirely by it, which the harness scored as NO-ANSWER. An eval that misreads a model is worse than no eval, so it now reads thinking blocks as a fallback and has room to answer. 557 tests pass, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
d24823b6f3 |
fix(missions): a failed phase stranded its mission at running forever
Found by counting containers during a cleanup, not by a test. gw-04 was holding
a per-mission runtime container for a mission whose only topology run had failed
three days earlier — phases `pending,failed`, mission still `running`.
The interaction, which lived entirely between two queries' predicates:
`start_pending_phases` launches a phase only when EVERY lower-order phase is
`completed`, so once one fails the phases after it can never run. They stayed
`pending`. `close_finished_missions` closes a mission only when NO phase is
outside ('completed','failed','skipped') — so a `pending` phase that would never
run kept the mission `running` indefinitely. And `mission_runtime`'s sweeper
fires N minutes after a TERMINAL state, so the container was never reaped.
One leaked container per failed multi-phase mission, accumulating silently, with
nothing in any log saying so. Neither query is wrong alone; the bug is that
nothing marked the phases the failure had made unreachable.
`skip_unreachable_phases` says it: a `pending` phase with a `failed` phase at a
LOWER order_idx becomes `skipped` — strictly earlier, because order is what makes
a phase unreachable, and a failure later in the list says nothing about one still
queued ahead of it. `skipped` is not a new concept: `close_finished_missions`
already treats it as terminal, and it is the honest word for a phase that was
never run, as distinct from one that failed.
RETRY HAD TO MOVE WITH IT, or this trades one bug for another. `retry_phase`
required the mission to be `running`, so closing failed missions would have made
the one outcome you would actually want to retry the one you could not. It now
accepts `failed` too, and in one transaction: resets the phase, REOPENS the
phases its failure had skipped (without that, a retry runs the phase and stops,
because everything after it is terminal-by-skip), and puts the mission back to
`running` — every launcher and closer keys off that status. `completed` and
`cancelled` stay refused; reopening those is a different decision.
557 tests pass, clippy clean. Three DB tests against real SQL, including that a
phase queued BEFORE the failure is untouched and that a draft's phases are never
swept.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
bf2055e725 |
fix(evaluator): the anti-Goodhart clause was failing work that RECORDS a value
Three consecutive production verdicts failed a phase that had done exactly what
its condition asked, each time with a different invented reason: "6.1.128 is not
a kernel release string like 'Linux 6.1.128'", then "line 2 should be 27.0.0",
then "line 1 must be empty or unrelated". I reworded the condition twice, and the
second rewording made it worse.
THE CONTROL THAT SETTLED IT: asked the same question with the same file and the
same condition — but WITHOUT our system prompt — glm-4.7 answered MET, citing the
exact line. The model judges this correctly. Our prompt does not.
The cause is a clause we wrote on purpose. `EVAL_SYSTEM_VERIFYING` is
deliberately adversarial because an earlier evidence-only judge was gamed by an
agent that emitted the string the judge had asked for, and it says to fail "a
required string or value hard-coded, stubbed, or printed rather than produced by
working code". A condition asking for a kernel version to be written into a file
IS that shape, read literally. The judge was obeying us.
Two clauses now, because each without the other is a known failure:
- the trap stays: work that satisfies the letter and not the purpose — tests
weakened, assertions fitted to wrong output, values stubbed — is not met.
- some conditions are satisfied BY a recorded value, and for those, writing the
value IS the work: a measured baseline, a scan report, a recorded environment
fact. Hard-coding is cheating only when the condition is about behaviour code
must produce.
And the other failure from those three verdicts: "judge the condition AS WRITTEN;
do not re-derive the expected value yourself" — a condition may describe a
DIFFERENT machine, an earlier run, or a remote environment, and the value the
judge would measure where it stands is not the one under judgement. That is
exactly what produced "line 2 should be 27.0.0": a tool-using judge ran `uname`
in its own container and compared.
This is not a niche fixture problem. The model-authored plans shipped today write
BASELINE.md and security-findings.md and gate on them — every one of those is a
recorded-value condition, and every one would have been rejected.
555 tests pass, clippy clean. A test pins both clauses, since removing either
reintroduces a failure this project has already paid for.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
72f8bdc87c |
test(harness): a done_when naming a COMMAND invites the judge to run it
My previous attempt at this made it worse, which is the useful part. The condition said "a Linux kernel release string" and the judge rejected `6.1.128` as "not a Linux kernel release string such as 'Linux 6.1.128'". I rewrote it as "the exact output of `uname -r`" — and the next verdict was that line 2 should be `27.0.0`. The judge has a sandbox and allow-listed commands, so naming a command told it to RUN that command, in ITS OWN container, and compare the file against the answer it got there. The file records a microVM's kernel; the judge was comparing it against the machine the judge runs on. Those are different machines by design — that is the entire point of the assertion. So a `done_when` for a tool-using judge must describe the VALUE's shape, never a command that produces it: "a bare kernel version of the form MAJOR.MINOR.PATCH (for example 6.1.128) and nothing else", plus an explicit instruction not to run uname and not to compare against the local machine, because the file records a different one. The general rule, worth carrying into how `done_when` is written anywhere: a condition phrased as "the output of X" is ambiguous about WHERE X runs, and a judge with tools resolves that ambiguity by running X where it stands. Conditions about a remote or past environment must be stated as properties of the recorded value. The scenario's real proof that the agent ran in a guest is unchanged: a separate comparison of that line against the actual gateway and node kernels, which has passed on every run including the two where the judge disagreed. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
e2f576ec02 |
fix(evaluator): the judge verifies a COPY, never the mission's own tree
The harness's uid probe caught this the moment cross-provider validation came back: `checkout has multiple writers (uids=0,65532)`. All 63 root-owned files were under `repo/target/`. The mechanism, confirmed rather than guessed: the judge's verification sandbox execs into `clawmates-runtime`, which runs as ROOT with the missions root bind-mounted, and its workdir was the mission's LIVE checkout. So when the judge ran `cargo test` to check a condition — which is the entire point of the verifying evaluator — cargo wrote `target/` into the checkout as uid 0, in a tree otherwise owned by the server. The next phase's `cargo` would then hit permission-denied on a directory it cannot write, which is the uid-split failure class copy mode exists to eliminate. IT WAS LATENT ALL DAY. While the z.ai credential was dead the judge never ran a single check, so the uid probe kept passing; restoring the credential surfaced it on the first gated mission. A guard that only holds while a dependency is broken is not a guard, and this one was only visible because the harness measures the invariant rather than the feature. Running the checks as the checkout's uid was the obvious fix and is the wrong one: `CARGO_HOME` is root-owned 0755 in that image, so a non-root uid fails, and the evaluator treats "could not run" as unverified — trading a polluted tree for phases that fail closed for a reason unrelated to their work. So the sandbox verifies a copy, made through `mission_fs::pack_dir` so it carries exactly what the delivered diff carries (no `target/`, no `node_modules/`) — one exclusion list, three consumers. The copy lives at `_verify/<mission>`, a sibling of the swept per-mission directories, and is removed on drop. This is the rule the codebase already applies to the `verifier` subagent, which has no Edit and no Write, stated for the judge: verification must not mutate what it verifies. A judge that can change the tree it is judging can make its own verdict true. NEGATIVE CONTROL, run: pointing the sandbox back at the live checkout fails `the_judge_verifies_a_copy_and_never_the_mission_tree`. The test seam (`Sandbox::at`) is never `owned` and never deletes, so a destructive constructor cannot masquerade as a plain one. 553 tests pass, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
1b556c5849 |
test(harness): say what the condition means, after the judge read it strictly
The restored GLM judge failed a phase that had done the work: MICROVM.md existed with two lines and the second was `6.1.128`, and the verdict was "a kernel version number, not a Linux kernel release string such as 'Linux 6.1.128'". The judge is wrong on the fact — `6.1.128` is exactly what `uname -r` prints, and "release" is the term for it — but the CONDITION was ambiguous, and it is our fixture. "A Linux kernel release string" can be read as either `uname -r` output or `Linux x.y.z`, and a stricter reader is entitled to the second. Both scenarios now say what they mean: the exact output of `uname -r`, a bare version, no prefix. This is not weakening the assertion. The scenario's own kernel check — the one that proves the agent ran in a guest rather than on a host — is a separate, unchanged comparison against the real host kernels, and it PASSED on the same run. What changed is only that the mission-level `done_when` now describes an observable fact precisely, which is what this codebase's own plan-authoring prompt tells models to do. Worth recording rather than papering over: an over-strict independent judge is a much safer failure mode than an over-lenient one, and this is evidence the judge READS the tree instead of rubber-stamping it — the Goodhart incident that motivated cross-provider validation was the opposite failure. But it does mean a vague `done_when` can now cost a phase, which raises the value of `done_when_check` (a shell command, judged by exit status) for anything mechanical. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
521da9feb9 |
fix(deploy): the verify step is the authority, not the recreate
`scripts/deploy.sh` reported failure twice this afternoon for deploys that had succeeded. Both times the 60-second rolling timer rolled the stack onto the same `:latest` first, and the script's own `docker-compose up` then hit a container-name conflict — "already in use" once, "Renaming a container with the same name" the other — for a container the timer had already recreated correctly. A deploy signal an operator has to second-guess is precisely what this script exists to prevent. Its original reason for being was a green edge on a stale image; crying wolf trains people to ignore the alarm, which gets you the same outcome by a different route. The recreate is now best-effort and says so when it fails, and the VERIFY step decides — it compares the RUNNING image id against the resolved `:latest`, which is the only question that matters and is unaffected by which process did the roll. A genuinely failed deploy still fails there, because that check never depended on the recreate succeeding. Both false alarms were settled by hand with the binary grep (`docker exec … grep -a -c "<string only in the new code>"`), which remains the strongest check when the image id is in doubt. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
3300c9d149 |
feat(missions): say at boot whether the independent judge can be reached
The z.ai credential expired mid-session and the first symptom was a two-phase mission failing after BOTH its VMs had run — the phase completed, delivered, pushed, and then one evaluation row said "the independent validator could not be reached this pass". `cross_provider_judge` refusing to fall back to the agent's own provider is correct: a verdict from the same family is not an independent check, and producing one quietly would claim a property the verdict does not have. The cost of that refusal is that a dead validator makes EVERY `done_when` phase unmeetable — and the information needed to know that existed from the moment the server booted. Nobody was told until it was expensive. The sibling of `runtime_preflight`, and the same stance: a report, not a gate. The server must still boot with a broken validator — refusing to start turns a degraded deployment into a dead one, and a mission that opts out (`validator_model = ''`) is unaffected. Two faults, kept distinguishable because they send an operator to different places: `Unregistered` (no provider by that name — the evaluator will refuse it rather than judge with the default, so register one) versus `Unreachable` (it resolved and the call failed — fix the credential). Collapsing them into "the validator is broken" is the kind of merge that costs an hour. The probe is a real completion through `Runtime::complete` — the same resolve-then-stream path the judge itself takes. A models-list or a HEAD would pass for an expired key, a revoked key, and a key with no quota, which are exactly the cases worth catching; and a probe that dialled the provider its own way could pass while the real call fails. `NotConfigured` is reported too, and not as an error: a deployment may choose the house model. It is still worth saying out loud that the check running is not an independent one. 551 tests pass, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
08dd227a45 |
feat(missions): give the planner the repository's contents, not just its names
The root listing was not enough. Given names alone the planner wrote "optimise the hot path" for a crate whose hot path is `add(a: i64, b: i64) -> i64` — a mission that was unachievable from the moment it was written, and that nothing discovered until an agent had built a benchmark harness in a VM to measure an integer addition, honestly reported no improvement was possible, and the judge correctly failed the phase. `repo_digest` fetches the whole tree (so "does this have benches/" is a fact, not an inference) and then file CONTENTS in priority order: manifests first — they say what the project is — then the README, then source ascending by size, since a planner learns more from twenty small files than from one large one. Lockfiles and build output are dropped: enormous, and they say nothing a manifest does not. THE RULE THIS ENFORCES, and the reason the rendering is its own tested module: a digest of any repository worth planning against is partial, and a model shown a partial view without being told it is partial plans as though it saw everything. So every omission is stated — how many files exist, how many were shown, what was cut from each, and "anything not shown you have NOT seen". Same distinction as `Option<u32>` for the subagent probe: "we did not look" and "there is nothing there" are different facts. Failures degrade to a stated absence rather than an empty string, and the three cases stay distinguishable: no repository, a tree that could not be read, and a tree read but no contents fetched. An unreadable tree is never rendered as an empty repository. Two more things the prompt now says, both learned from that run: plan for the repository as it IS rather than as the description implies (and if the description asks for something the code cannot support, say so in the task and plan the phase that establishes the truth, rather than a phase that must fail); and a mission agent has NO package-registry access. The agent discovered the second one mid-run and wrote a dependency-free `std::time::Instant` harness after Criterion could not be added — good adaptation, but nothing had warned it. 549 tests pass, clippy clean. The budget/priority/truncation logic is pure and tested; only the fetching touches the network. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
0aeae07db2 |
fix(missions): the planner was planning blind — show it the repository
The first real plan opened with "Identify the crate's hottest code path and run its benchmark harness". This crate has no benchmark harness. The phase ran, found nothing to baseline, delivered zero files, and the plan's second phase was left with nothing to optimise against. The planner saw the mission title, the description, and a boolean for whether a repository was bound. It never saw the repository. A plan about a codebase written without looking at the codebase is a guess that reads like a plan — and the failure surfaces two phases and one VM boot later, as an agent reporting that the thing it was told to run does not exist. The prompt now carries the repository's root listing, read from the FORGE rather than a checkout: at proposal time the mission is still a draft and `ensure_checkout` has not run, so there is nothing on disk to list. It also says outright that a phase needing something absent must CREATE it and say so in its task — the failure was not only ignorance of the tree but the assumption that missing tooling is someone else's problem. A listing that cannot be fetched degrades to "(the repository listing could not be read)" in the prompt rather than to an empty string. A model told the listing is unavailable can hedge; a model told nothing assumes — which is the same distinction as `Option<u32>` for the subagent probe, in a prompt instead of a struct. Found by running the thing end to end rather than by testing it: every unit test here passes with a planner that has never seen a repository. 543 tests pass, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
a33dbdcdc3 |
feat(missions): W1/#13 — let a model author the mission's phases
The last unstarted item from the missions-as-workflows plan, and the other half
of Slice 5: that one lets a model size the TEAM, this lets it decide what the
work IS.
Every mission's phases come from one of five hand-written recipes in
`templates/workflows/*.toml`, chosen by `template_kind` before anyone saw the
mission. That is the "do it this way: 1, 2, 3" over-specification that makes a
capable model follow a worse plan than it would have chosen. The recipes stay —
they are still the default for a mission nobody proposes a plan for, and the
fallback when a proposal is refused.
Same three verbs and the same review gate as the roster, deliberately: propose
and decide are separate because only the second changes a mission, and a second
shape would be a second thing to get right. Approving REPLACES the phases (a
plan is an answer to "what is this mission", not an addition to one), draft-only.
GROUNDED IN WHAT THE PLATFORM ACTUALLY READS, which is the part that makes this
more than a copy. `phase_config::KNOWN_KEYS` already names every phase-config key
and the code that reads it — the registry built after `task` sat unread through
every mission. A plan is validated against it, so a model cannot propose a phase
whose settings nothing will act on: the failure that registry exists to EXPOSE is
one this path cannot create. Phase kinds are checked the same way, because an
unknown kind does not error — it falls through to the catch-all purpose and runs
as a generic phase that looks like it worked.
TWO THINGS THE WORK ITSELF FOUND, both the same shape:
- `done_when_check` — the stop-gate key added earlier today — was never
registered in `phase_config`, so every mission that set it has been logging
it as an unknown key. Found by a test written for a different purpose, which
is the registry doing exactly its job. Now registered with its reader.
- `done_when` and `max_iterations` are COLUMNS promoted out of config by
`missions::create`; the evaluator sweep filters on the column in SQL every
tick. My first insert wrote the config blob alone, which would have stored a
plan's completion condition where nothing judges it. NEGATIVE CONTROL run:
binding NULL instead of the promoted value fails
`an_approved_plan_replaces_the_missions_phases`.
`order_idx` comes from the array's own order rather than a field the model sets:
two sources for one fact is how a plan ends up with two phase 0s, and order_idx
is what `start_pending_phases` sequences on.
MAX_PHASES is 4 and the prompt argues for one. Each phase is a full agent run in
sequence, and splitting one change into plan → implement → test is the documented
anti-pattern — a single agent doing all three keeps the context that makes the
later steps good.
543 tests pass, clippy clean. Migration 0072.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
a48d78f8eb |
test(missions): a composed node is offered the same help as a solo one
Every composed run so far reports `subagents: 0`, and the honest question is whether that is the tasks being small or the capability being absent. It is the former, and this is what says so: a composed node's task text is built by `microvm_turn_executor` and then wrapped by the SAME `vm_prompt` inside `run_inside`, so one prompt builder serves both paths and both carry the `Agent` tool offer and the `verifier` / `explorer` roles. Asserted rather than left to code reading, because if someone gave composed nodes their own prompt without the offer, the difference would show up only as a count nobody was watching — and "the graph fanned out but no node did" is indistinguishable from "no node needed to". The roles are read from `agent_definitions()` rather than spelled out, so adding a role without mentioning it in the prompt fails here instead of shipping a role the lead is never told about. 535 tests pass, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
742724e53c |
feat(fleet): Kimi as a microVM backend — the URL settled by measurement
The base URL took three measurements to find, and the first two were wrong in instructive ways. `api.moonshot.ai/anthropic/v1/messages` EXISTS and speaks the protocol — it answers with Moonshot's own structured error rather than a 404. It also rejects an `sk-kimi-` key, because it belongs to the platform.moonshot.ai account namespace. Two endpoints that both "work" for different accounts is precisely the shape that makes a guessed URL look like a broken key, and it is why this was refused rather than guessed for as long as it was. The Kimi CODE service is the one an `sk-kimi-` key belongs to: `POST https://api.kimi.com/coding/v1/messages` returns a real Anthropic Messages body — `msg_` id, `content` blocks, a `thinking` block with a signature. So `ANTHROPIC_BASE_URL=https://api.kimi.com/coding`, WITHOUT the `/v1`: Claude Code appends `/v1/messages` itself, and `/v1/v1/messages` would 404 in a way that reads as a broken image rather than a bad URL. Two more measured, each otherwise a silent failure at the first turn: `Authorization: Bearer` is accepted (so ANTHROPIC_AUTH_TOKEN is the right injection channel), and a `claude-*` model id is ACCEPTED AND ANSWERED — Kimi maps it onto `kimi-for-coding` exactly as z.ai does, so no ANTHROPIC_MODEL override is needed. Claude Code rather than Moonshot's own `kimi` CLI, deliberately. The mission harness is Claude-Code-shaped throughout: `--agents` JSON roles, the verifier's tool allowlist, the `Stop` hook behind the completion gate, the per-subagent transcripts counted as delegation evidence. `kimi` has none of those flags — its equivalents are TOML files and markdown agent dirs — so using it would mean a second executor with its own untested failure modes. TWO STALE MAPS, caught by the rootfs harness refusing to bless the image: both `fc-build-rootfs.sh` and the node's `required_cli` expected backend `kimi` to contain Moonshot's `kimi` binary. That assumption predates the measurement, and it failed a rootfs that was correct. Both now say `claude` for glm and kimi alike — the binary is the same in all three images; only the endpoint differs. Egress for `kimi` is `api.kimi.com` alone: not moonshot.ai (wrong namespace), not z.ai, not Anthropic. Asserted both ways, like the other two. The image and rootfs are built on tank and the rootfs passes all four checks (boots, git, writable /mission, `claude --version`). 534 tests, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
d3a53e7bf1 |
fix(fleet): a VM reaches its OWN provider and no other, measured not assumed
The GLM backend works — and proving it produced a better boundary than the one I shipped an hour ago. WHAT THE FIRST GLM MISSION SHOWED. It completed, and the delivered file said the model was "claude-opus-5". The node's egress log said the VM had dialled `api.anthropic.com` five times before `api.z.ai`. Either reading alone is consistent with a "GLM backend" that silently runs Anthropic — the exact silent-success shape this project keeps closing — so I did not accept either. THE ABLATION, run on tank rather than reasoned about: deny `anthropic.com` at the proxy and run the same mission again. It **completed**, dialling only `api.z.ai`. So the completions genuinely come from z.ai; Claude Code's calls to anthropic.com are its own telemetry, not its model traffic. And that same agent — served exclusively by z.ai, with Anthropic unreachable — still described itself as "Claude Opus 5 (1M context)". **A model's account of which model it is has no evidential value.** The proxy's log of which host it dialled does. This is the `uname -r` lesson again in a new place: ask the infrastructure, not the agent. So the allow-list is now PER BACKEND rather than a union: a `claude` VM reaches Anthropic and the forge, a `glm` VM reaches z.ai and the forge, and neither can reach the other's endpoint. A union was defensible when it was one host; once the measurement showed a GLM VM never needs Anthropic, keeping it would mean a credential mix-up upstream could still put one provider's secret on another provider's wire. Now it fails at a closed door instead. An unknown backend gets the forge and NO model API — it cannot run anyway, and borrowing somebody else's door is the failure this split prevents. An explicit `CLAWMATES_FC_EGRESS_ALLOW` still wins outright: an operator who set it drew a boundary on purpose. `DEFAULT_ALLOW` is deleted rather than left beside the new function, so there is one answer to "what may a mission reach" and not two. 534 tests pass, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
f7f3dfe495 |
feat(fleet): GLM as a real microVM backend, and per-role models for claws
Three threads, all of which end at the same place: a mission whose verifier does
not share a model with the coder it reviews.
**GLM has a credential contract now.** `microvm_credential_for` returned one env
var name, which quietly assumed every provider reads its secret from the same
place Anthropic does. It returns a `Credential { source, target }` instead —
z.ai's key lives in the server's `ZAI_API_KEY` and Claude Code reads it as
`ANTHROPIC_AUTH_TOKEN`, and collapsing those two names is what forces a guess at
the other end. A wrong guess here sends one provider's credential to another
provider's endpoint.
`images/agent-glm` is the same CLI at the same pinned version as `agent-claude`
with `ANTHROPIC_BASE_URL` baked in. The split is deliberate: the ENDPOINT is a
property of the image, the CREDENTIAL is a property of the turn. That makes the
dangerous mix-up unrepresentable — a GLM VM cannot be handed an Anthropic
subscription token, and a claude VM cannot be pointed at z.ai. Asserted both
ways, because "the GLM VM must not carry CLAUDE_CODE_OAUTH_TOKEN" is the
property that costs a credential if it ever stops holding.
Kimi stays refused. `KIMI_API_KEY` is set and Moonshot serves an
Anthropic-compatible API, but I have not verified its base URL against the
running service, and this function is precisely where guessing a URL is
expensive. It becomes an arm the day someone measures it.
`api.z.ai` joins the node's default egress allow-list. A default that cannot
run the images we ship is a trap rather than a policy — the alternative is an
operator discovering it as a hung agent with no model access.
**Per-role models for claws** (migration 0071). `template_roles` had no model
column, so `mint_team_from_template` bound every role of every mission team to
one literal — a template whose whole point is an independent reviewer minted a
reviewer sharing a model with the coder. A role may now name its own; roles that
say nothing still take the mint's default, so every template written before this
behaves exactly as it did. The literal is now that default rather than a
hardcode.
**A harness scenario for the roster flow.** `verify-mission-delivery.sh roster`
runs the whole Slice 5 loop — planner proposes, human approves, mission runs —
and asserts the roster LANDED on the mission row rather than trusting the API's
answer. That distinction is not theoretical: the first live approval returned an
error while leaving the proposal marked approved.
Built and proven on tank ahead of the deploy: `clawmates/agent-glm:dev` reports
`2.1.223` and `BASE=https://api.z.ai/api/anthropic`, and
`fc-build-rootfs.sh … glm 8G` boots a VM from it that has git, can write
/mission, and answers `claude --version`.
533 tests pass, clippy clean. Migration 0071.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
75d09241fb |
fix(missions): the first real approval found two bugs the tests could not
Deploying Slice 5 and approving one roster in production broke it twice, in ways
528 green tests had nothing to say about.
**1. `jsonb_set` refuses a scalar.** A mission created through the API without a
`config` stores jsonb `null` — a scalar — and `jsonb_set` fails on it with
"cannot set path in scalar". The guard was `coalesce(config, '{}')`, which
protects against SQL NULL; this is a perfectly good JSON null of the wrong shape,
and coalesce passes it straight through. Every test wrote `'{}'::jsonb` because
that is what a test author types. Production types nothing at all.
**2. The approval was not atomic, and failing halfway is permanent.** The claim
and the mission write were two statements, claim first, so when the write failed
the proposal stood `approved` with nothing applied — and the partial unique index
then makes that state unrecoverable: no other proposal for that mission can ever
be approved. The mission ran solo with `team_engine` still NULL while its
proposal said otherwise.
`approve_and_apply` is now one transaction: claim, write, commit or roll back.
The type guard is `CASE WHEN jsonb_typeof(config) = 'object' THEN config ELSE
'{}'::jsonb END`, which answers the question that was actually being asked.
Both regressions are tested in the shape production had, and both NEGATIVE
CONTROLS were run rather than assumed:
- restore `coalesce` → `a_roster_applies_to_a_mission_whose_config_is_json_null`
FAILS with Postgres's own "cannot set path in scalar", the exact production
error.
- commit instead of roll back on a failed apply →
`a_failed_apply_leaves_the_proposal_undecided` FAILS with the proposal stuck
`approved`.
Worth stating plainly: the API returned 500 for that approval, so this was not
silent to the caller — but the row it left behind claimed the mission had a
roster it never received, and the mission then ran and delivered, which is the
shape that gets believed.
530 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|