529497febbf3a099388b472b5ef211c37b0cff53
455
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar SobhandClaude Opus 5
|
529497febb |
fix(placement): a composed graph needs every backend its nodes name
The full harness found it — 12 of 13 scenarios green, `roster` red:
roster: the planner sized this mission at 2 member(s) PASS
roster: the approved roster is on the mission (2 nodes, composed) PASS
roster: this run added 1 line(s) for a 2-member roster FAIL
topology_runs.error: turn executor failed: node n1 in a microVM:
vm_create failed: no rootfs for backend "canary-claude" on this node
The roster proposed `verifier@canary-claude`. Placement asked
`online_for_backend` about the MISSION's backend — `claude` — and architect
answered, holding `claude` and `local-ornith`. The graph's first node ran and
delivered, the second could not boot, and the mission finished half-done. The
question placement asked was true and insufficient.
A composed graph runs on ONE node, so that node needs every image its nodes ask
for. `required_backends` collects the mission's plus each
`config.roster.nodes[].attrs.backend`, and `online_for_backends` passes the
whole set to the same jsonb `@>` — containment already means "contains ALL of
these", so the query shape did not have to change, only what it was asked.
This is the failure mode the roster feature creates by existing: its entire
purpose is putting a verifier on a different provider, which is exactly what
makes one node insufficient. Nothing before the full suite had a reason to
exercise it — the composed scenario uses one backend for all five nodes.
`NoCapableNode` now names the set and says why one node must hold all of them.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
c66c3c6377 |
feat(ui): the microVM path is reachable from the mission wizard
Everything built today — Firecracker missions, the four backends, the local GPU one — was unreachable from the dashboard. The wizard offered `zeroclaw` and `local_herdr` and nothing else, so a mission created in the UI could not be a microVM mission at all, and `local-ornith`/`glm`/`kimi` were API-only. Testing "our workflows in the UI" would have exercised none of it. Adds the runtime option and a backend picker, fed by a new `GET /api/fleet/backends` that returns `mission_roster::available_backends` verbatim — the SAME list the roster planner is handed, not a second one. Its two rules are both load-bearing and neither is visible from a node's capabilities alone: the image must be built on an online node, and the backend must have a credential contract. `agent-terminal` passes the first and fails the second — bootable, with nothing for the agent inside to authenticate with — so offering it would produce a mission that validates, launches, and dies at the agent turn. Ids are deployment vocabulary, so the picker labels them: a user choosing between `local-ornith` and `canary-claude` should not have to know which company each one bills. An empty list says why (no rootfs built) instead of showing an empty dropdown, and no node is chosen for a microVM mission because `vm_placement` picks it per phase. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
1f6108f769 |
feat(gc): reclaim the mission tree on the gateway
`cleanup_sweeper` prunes ROWS. Deleting a row has never deleted a directory, and `teardown_container` only runs while a mission still exists to tear down — so a mission removed by any path that skipped teardown left its tree behind permanently, on the smallest disk in the fleet (150 GB, shared with postgres and every checkout). 106 mission directories are sitting there now. Filesystem-first, deliberately: the DB is the PREDICATE, never the enumerator. Enumerating from the database is exactly how these became invisible — a directory whose row is gone is the one a row-driven sweep cannot see. Three reapers, one deletion path. Orphan mission dirs (no row, past a 2h grace), scratch trees (_bench/_gate/_verify/_merge past 6h — all four have leaked before), and _outputs past 90d, whose artifact rows are marked only AFTER the files are gone, because the other order claims artifacts are reaped while they are still on disk. The single removal path escalates: the server is uid 65532 and cannot delete what the per-mission daemon leaves as root, so PermissionDenied falls back to `root_copy::purge` and shouts if the tree survives even that. A GC that cannot collect is the thing being fixed, so failures are counted and reported, never swallowed. Guards worth naming: `_cargo` is a SHARED cache every mission writes to and lives under the same root, so an underscore-prefixed sibling treated as an orphan mission would delete it out from under running work and look like a slow cargo build. Only a well-formed mission id is ever a candidate — a directory whose name is not an id can have no row by construction, so without that gate every unrecognised directory looks orphaned. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
3c3d01c8d1 |
fix(llm): the chain preflight printed nothing at all
Deployed, and the report simply did not appear — from the tool built to stop
things failing silently. Two causes, both worth keeping:
There was no timeout anywhere in the probe, so one slow provider swallowed the
entire report. Each link is now bounded at 60s (generous: `complete_or` spends
up to 30s in its own backoff, so a tighter cap would report a merely throttled
link as hung) with `TimedOut` as its own state, and every line is emitted AS IT
RESOLVES rather than collected and printed at the end — a later link that hangs
must not be able to hide the ones already checked.
The first attempt at the timeout awaited the probe and then wrapped the result:
let probe = complete_or(...).await;
timeout(PROBE_TIMEOUT, async { probe }).await
That compiles, reads correctly, and bounds nothing. The timeout has to wrap the
future.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
c7c3eeab46 |
fix(test): the colon-vs-spec test did not compile
Committed and deployed while its test compile was failing: the verify step was `cargo test | grep -E "^error|test result" && git commit`, and grep exits 0 when it MATCHES, so finding the error is what let the commit proceed. The library built fine, so the deploy was sound, but the check that was supposed to gate it did the opposite of gating. The error itself was a borrow in a test closure; a plain fn fixes it. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
d9c5300859 |
fix(llm): the preflight found two broken links on its first live run, one its own
fallback chain (6 link(s), 4 usable):
claude-opus-4-8 ok
claude-sonnet-4-6 ok
claude-haiku-4-5-20251001 ok
kimi:kimi-k2.7-code BROKEN: 400 ... role 'system' must not be empty
glm:glm-4.7 ok
local:ornith-fleet:9b UNREGISTERED — resolves to the DEFAULT provider
Neither link was actually broken.
The probe sent an EMPTY system prompt, which Kimi rejects outright. A probe has
to look like the traffic it stands in for, or it measures itself.
The second is the one worth keeping. `resolve_provider` returns a spec unchanged
when it does not recognise the provider, and the part after the FIRST colon when
it does — so the obvious test, "does the model half still contain a colon",
reads correctly and is wrong the moment a model id has one. `ornith-fleet:9b`
has one. The probe reported a provider the server had just finished registering
as UNREGISTERED.
`evaluator::cross_provider_judge` had the identical check, and would therefore
have refused a local judge as "not independent" — silently falling back to a
same-family one, which is the exact claim that path exists to make honestly.
Both now compare against the whole spec.
That bug was written into the codebase before a model name with a colon existed,
was correct at the time, and became wrong when one arrived. Nothing would have
reported it; a boot-time probe of every link did, on its first run.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
c3ad5672fc |
feat(llm): six-link fallback chain, and a preflight that proves it
opus -> sonnet -> haiku -> kimi -> glm -> local. The order is capability first, then independence: three Anthropic tiers on one account (a throttle usually hits a tier, so stepping down often clears it), then two separately funded accounts (now an outage, not just a throttle, is survivable), then our own GPU (nothing left to be down). Every id was probed on this deployment and answered 200. The preflight is the more important half. Configured is not working, and this chain has a specific way of lying: `resolve_provider` falls back to the DEFAULT provider when it does not recognise a provider name, so a typo in `kimi:` does not error — it quietly runs on Anthropic, and a chain that reads as three accounts is really one. A reachability-only probe calls that link green. So `preflight` checks resolution and reachability separately, eight tokens per link through the REAL call path, and reports four states. `Throttled` is deliberately not a failure: a 429 means the spec resolved, the credential authenticated, and there was no capacity this second — the exact condition the chain exists to route around, and painting it red would train an operator to ignore red. `Unregistered` and `Broken` are failures, and they get different words because they need different fixes. It runs at boot alongside validator_preflight and runtime_preflight, spawned so it cannot delay startup. A chain is the one piece of infrastructure nobody looks at until the day it has to work, so it is now checked on the days it does not. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
b18e62041b |
feat(llm): the fallback chain's last link runs on our own hardware
`local:ornith-fleet:9b` joins opus -> haiku -> glm as the final link. Every entry above it depends on somebody else's account staying funded and unthrottled; this one depends on a GPU in the next room. It is last because it is the weakest model, and present because a chain whose every link is external is not a fallback chain, it is one outage in a trench coat. Three small changes make it work: - `build_provider_registry` accepts a provider with an empty `api_key_env`. A model on our own hardware has nothing to authenticate to, and the old behaviour SKIPPED a keyless provider — leaving the chain quietly one link shorter than it reads, which is the failure mode this whole area keeps producing. - `provider_family` learns `ornith`/`ollama` for BARE names. A qualified `local:` spec was already answered by the split, but a bare one fell through to "unknown", and `cross_provider_judge` would then refuse a judge that is genuinely a different family from the Anthropic implementer. - A test pins that the last link survives `resolve_provider`'s split-on-FIRST- colon: `local:ornith-fleet:9b` is provider `local`, model `ornith-fleet:9b`. Splitting on the last colon would ask for a provider named `local:ornith-fleet`, and the symptom would be a silent fall back to the default provider. Infra: Ollama on tank and architect now binds 0.0.0.0 so the gateway (which has no GPU) can reach it. `tailscale serve` cannot — Ollama rejects a non-local Host header as a DNS-rebinding guard and OLLAMA_ORIGINS is CORS-only, so it 403s. 0.0.0.0 still includes loopback, so the microVM vsock pipe is unaffected; verified on both nodes. This is an explicit trade: Ollama has no auth and its API can pull and delete models, so it is now reachable from the LAN as well as the tailnet. The drop-in carries the ufw one-liner to close the LAN side. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
f56d41f5b7 |
feat(backend): local-ornith — a mission backend served by the node's own GPU
Claude Code pointed at the Ollama already installed on every GPU node. Ollama has served a native Anthropic-compatible /v1/messages since v0.14, so this is an env contract rather than a translation layer — the fourth variation on the same idea as agent-glm and agent-kimi. The route is NOT the egress proxy, and that is the design. `egress` speaks CONNECT, takes a destination from the guest, resolves it and decides; every one of those powers is a liability, which is why it refuses non-443 ports and IP literals after a unit test caught them being bypassed. Routing a local model through it would have meant relaxing both. `local_model` is the opposite shape: there is no destination in the protocol. fcagent listens on guest 127.0.0.1:11434 and pumps to vsock 9003; the node splices that onto its own 127.0.0.1:11434 and copies bytes. A compromised guest cannot redirect it because there is nothing to redirect — it is a pipe, not a proxy, and strictly narrower than anything an allow-list could express. The bytes never touch a network, so there is no wire for TLS to protect, and Ollama stays bound to loopback rather than being exposed on the tailnet. The socket is bound only for a backend declared to use a local model, so a `local-ornith` VM reaches the forge through egress and nothing else, while every other backend's guest port simply refuses. Both halves have negative controls. `scripts/fleet-model-setup.sh` exists because of one measurement: stock ornith:9b reported input_tokens=2050 for a 48000-word prompt and answered as though nothing had been dropped. Ollama's default window is ~2K whatever the model card says, and it truncates silently — the exact failure an agent turn would hit and never report. The script pins num_ctx=131072 into a derived tag and then PROVES both the window and tool calling before declaring success. Verified on architect: ~65536 words -> 65604 input tokens, stop_reason=tool_use. Placement needs no new capability key: building the rootfs only on GPU nodes means `nodes::online_for_backend`'s existing `rootfs @> ["local-ornith"]` predicate does the affinity, so morpheus never offers the backend. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
f68fc019e4 |
fix(teardown): a mission dir with root-owned files is now actually removed
The server runs as uid 65532, so `remove_dir_all` on a mission directory returns PermissionDenied the moment anything root-owned is left in it — and the old code logged that at the same level as "file not found" and moved on. The directory then lived forever. After the seed-copy fix a mission holds 3281 files owned by 65532 and 26 owned by root: `.claude.json` and the session jsonl the per-mission ZeroClaw daemon writes itself, after the copy has been chowned. Twenty-six files is small enough to keep every mission directory alive without anyone noticing why. PermissionDenied now falls back to `root_copy::purge`, which deletes from inside the container as root — the same escape hatch `container_exec` keeps for exactly this, clearing debris a root process created. And if the directory survives even that, it says so, because a cleanup that silently failed is the thing being fixed. Removing the last 26 properly means running the per-mission daemon as 65532, which needs `/mission` pre-created in the image with that ownership — the daemon creates it at boot today and cannot at a lower uid. That is an image change, deliberately not bundled here. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
4967b9b8fd |
fix(runtime): the seed copy reads as root and hands the result to 65532
Running the seed copier as 65532 broke mission launch, and broke it quietly. The seed dir is root-owned with parts at mode 0600 (`.claude.json`, `clawmates-mcp.json`), so uid 65532 cannot READ them: `cp` failed on the first unreadable entry, `set -e` abandoned the rest, and the mission came up with a runtime-data holding `.zeroclaw` and nothing else — no Claude credentials, no door config. The daemon then never created its agents' workspace, and the phase failed 200 lines later on "Could not find the file /mission in container", which points nowhere near the cause. It was quiet because `seed_runtime_data` polled for the container to STOP and returned Ok without ever reading its exit code. A copier that died on a permission error and one that finished cleanly were indistinguishable. It now reads the status and says what went wrong. So: root for the read, `chown -R 65532:65532 /dst` for the result. Both halves matter and they pull opposite ways — root is needed to read the seed, and 65532 is needed because everything else in the missions tree is 65532 and a GC running as 65532 cannot delete what root left behind. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
8ddea454d1 |
fix(runtime): the seed copy ran as root too, ~3200 files per mission
The uid fix landed and the CHECKOUT came back completely clean — 0 non-65532 files under `repo/` after a benchmark run that builds and tests Rust. But the same mission still held 3247 root-owned files, all under `runtime-data/`. `seed_runtime_data` spawns a throwaway container to `cp -a` the runtime seed into the mission's directory and never set `user`, so it ran as root — the identical absent-`user` omission `container_exec` had, in a container create instead of an exec. The seed source is 65532-owned and the destination is created by the server (which itself runs as 65532), so the copy never had a reason to out-rank either. This is the tree a gateway GC has to be able to delete, and a GC running as 65532 cannot remove root-owned files — the cleanup-that-cannot-clean-up shape, found before writing the GC rather than after. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
dcd9514622 |
fix(exec): mission work runs as uid 65532, so it stops creating debris it cannot delete
`CreateExecOptions` never set `user`. Not a wrong value — an ABSENT one: the daemon defaults to root, and twelve callers inherited that without any of them choosing it. That single omission is the origin of four separate patches — root-owned `target/` directories inside a checkout owned by 65532, `root_copy` existing at all, and a cleanup that had to re-enter the container as root to undo its own mess. The rule is positional and lives in ONE place: an exec whose workdir is inside `missions_root()` runs as 65532; anything else (preflight probes, image checks) keeps the daemon default so unrelated call sites cannot break. Twelve callers each remembering to pass a uid is twelve chances to forget, and the one that forgets leaves debris the other eleven cannot remove. Non-root needs an environment the image does not provide. Measured in the deployed image: uid 65532's HOME (/zeroclaw-data) and /usr/local/cargo are both root-owned and unwritable, so this would otherwise break every cargo call — the benchmark runner, the judge's sandbox, the delivery test gate — far more quietly than the leak it fixes. The missions root IS bind-mounted and writable by 65532, so HOME/CARGO_HOME move there and the cargo cache is shared across missions rather than re-fetched per mission. Verified on gw-04: a clean `cargo build` as 65532 with those three variables produces output owned entirely by 65532. Root remains reachable only through `exec_as_root`, whose name says so, and which exists solely to clear debris earlier root execs left. `runtime_preflight` now probes the whole policy at boot, so an image that moves or tightens that mount fails loudly instead of failing every cargo call for a reason no error message would connect to a uid. evaluator_tools' inlined fourth copy of the purge is replaced by `root_copy::purge`. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
42108c840d |
docs(placement): the drain half of that fix was never the broken half
drain-midmission passed 3/3 twice, but the "re-placing this phase" line the last commit added never appeared in the log. It cannot: `online_for_backend` filters on `status = 'online'`, so a draining node is not a candidate, never reaches `unfit`, and the pin simply falls through to ranking — on the old code as well as the new. So the scenario passes either way and proves the affinity decision, not the `TargetUnfit` bug. The path that genuinely used to fail a phase is "the previous phase's node has since FILLED UP": that puts it in `unfit`, which returned a non-transient error, which never reached the queue. That is what the unit test now says, in place of a claim about draining the harness does not support. The accidental mission-to-node affinity was real and unconditional either way. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
13a35138e9 |
fix(placement): a drained previous node re-places the phase instead of failing it
`drain-midmission` found this. `choose` treated its `want` argument as a hard
requirement, and the only caller passes `missions.target_node_id` — which is
not an operator's choice, only where the PREVIOUS phase happened to run. Two
consequences, both wrong:
- A node drained or filled between phases produced `TargetUnfit`, which
`is_transient()` says false to, so `phase_runner` FAILED the phase rather
than queueing or moving it. The queue silently did not apply to the second
phase of any mission.
- While the node stayed fit, every later phase went straight back to it
regardless of ranking — accidental mission-to-node affinity, which this
module's own header says must not exist.
Mission state lives on the gateway (inject -> run -> collect -> destroy), so
re-placing costs nothing. The pin is now advisory: preferred while it fits,
and when it does not, the reason is logged and ranking proceeds. `TargetUnfit`
is deleted rather than left unconstructed, so it cannot come back as a
non-transient failure by accident.
The scenario had its own race: it waited for phase 0 to COMPLETE before
draining, but warm phases finish in ~80s against a 10s placement sweep, so
phase 1 was often already placed — and the run then blamed the platform for
running on a node that was not yet drained. It now drains while phase 0 is
still running, which does not disturb a live VM and is the more faithful test.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
91fbd2dc88 |
refactor: the missions root has one definition, not five
`mission_workspace::missions_root()` is now the only place that answers "where
does mission state live". It had fragmented into five: this function, private
`env::var("CLAWMATES_MISSIONS_ROOT")` copies in security_scan, benchmark_runner
and mission_outputs, and a hardcoded `MISSIONS_HOST_ROOT` const in
mission_runtime that read no env at all.
They agree on the deployed value, so nothing has broken. The risk is entirely
in what comes next: anything that sweeps or reclaims this tree has to be
sweeping the same tree the writers use, and five definitions cannot promise
that — a reaper written against one would silently leave the others' directories
behind forever, which is how the orphans got there in the first place.
A source-walk test fails any module outside `mission_workspace` that reads the
env var itself.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
4fedfcec30 |
fix(placement): a young VM's unconsumed memory was handed out twice
The capacity harness scenario, on its first full run, caught what it was written to catch: capacity: architect peaked at 6 of 6 slot(s) FAIL capacity: 'morpheus' peaked at 3 concurrent VM(s) with only 2 slot(s) capacity: tank peaked at 6 of 6 slot(s) PASS capacity: the over-capacity missions QUEUED PASS capacity: all 16 queued/placed missions completed `capacity_of` inferred the host's own footprint by subtracting the VMs' FULL 8 GiB claim from observed usage — which assumes they have already consumed it. A VM booted seconds ago holds about an eighth. On morpheus (31757 MiB total, 4314 MiB idle, 2 slots) with 2 young VMs at ~6314 MiB observed, the inference 6314 - 16384 goes negative, clamps to the 2048 floor, and invents 2266 MiB — exactly enough for a third VM on a two-slot node. The footprint is only honestly MEASURABLE when nothing is committed, so remember it then: `nodes.mem_baseline_mib`, sampled by `survey` whenever it observes an idle node with fresh health. When VMs are committed, take the LARGER of the remembered reading and the old inference — a node that was once idle at 4 GiB and is now running a 20 GiB build must not be scored as idle, which would be the same over-commit arrived at from the other direction. Both directions have a test; the second is the one that would otherwise rot. Raising HOST_BASELINE_FLOOR_MIB would have made this one node's numbers pass and drifted the moment the fleet changed shape. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
dc0443de34 |
feat(fleet): GET /api/fleet/capacity returns the scheduler's own survey
Pulled forward from the observability phase because the capacity harness scenario needs it. A test that recomputed the slot arithmetic in bash would drift from `vm_placement` and then agree with itself while the scheduler did something else — the same shape as every silent-success bug in this codebase. Returns `survey()` + `rank()` unmodified, and keeps `unfit` as its own list: "the fleet is full" and "we could not read the fleet" send an operator to different places, so they must not be summed into one number. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
d48bdbc9a7 |
fix(llm): two modules were posting to Anthropic behind the providers' back
The research scenario passed 4/4 and the log underneath it said: phase_summarizer: ... failed: anthropic 400 Bad Request: "Your credit balance is too low to access the Anthropic API" `phase_summarizer` and `mission_refiner` each built their own reqwest POST to the Messages API with `x-api-key: $ANTHROPIC_API_KEY`. No audit of `.complete(` call sites could have found them — they never touched a provider — so every phase summary and every mission-brief refinement on this deployment had been failing against an empty account while the phases themselves ran fine. The summarizer even persisted an error row per phase, which is why nothing ever retried loudly enough to notice. Both now go through `subscription::complete_with_fallback`, so they inherit the subscription-first credential choice, the 429 backoff, and the opus -> haiku -> glm chain. The summarizer records the model that ANSWERED in mission_phase_summaries.model rather than the one it asked for. The guard is a source WALK, not a file list: any .rs under cm-api/src that mentions the Messages API host or `x-api-key` fails the test. A hand-listed set of files is exactly what let these two hide. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
52500a689c |
fix(door): say so when the security governor is failing open
`Runtime::judge` returns "governor unreachable (fail-open)" whenever the provider never answers, and the door caller drops `reason` on every allow — so a judge model that is rate limited or uncredited turns the governor into a rubber stamp with nothing anywhere saying so. Fail-open stays (a governor outage must not halt agents), but it is now loud. Found while removing the metered key as a dependency: the governor reads CLAWMATES_JUDGE_MODEL, which was `claude-opus-4-8` — a model that is 429 on this deployment's subscription. gw-04's .env now points it at `glm:glm-4.7`, matching CLAWMATES_VALIDATOR_MODEL: funded separately, uncapped, and a different family from the agent it judges. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
9c9439a271 |
feat(llm): the subscription is the default provider, with a recorded fallback chain
Two changes so an empty metered account stops being a platform outage. 1. `build_provider` prefers the subscription token over ANTHROPIC_API_KEY. A bare model name resolves to whatever this returns, so making it the subscription means no server-side call can reach the metered key by construction — rather than by a source-grep test that already missed four call sites once. The metered key remains a fallback and now warns loudly when it is the one in use; boot no longer requires it at all. 2. `complete_with_fallback` walks a declared chain when a model has no capacity: opus -> haiku -> glm:glm-4.7 by default, overridable via CLAWMATES_MODEL_FALLBACK, empty to disable. Measured on gw-04 today: opus and sonnet return 429 on the subscription while haiku, GLM and Kimi all return 200, so a capped window no longer means "the planner is gone". The chain returns the model that ANSWERED, and every caller persists it — mission_plan_proposals.author_model, mission_team_proposals.author_model, and the swarm's step role. A plan drafted by the third link and filed as an opus plan is a silent quality change, which is the failure shape this project keeps paying for. Two negative controls hold the design: the chain never retries the model that just failed as its own fallback, and it steps down ONLY for a capacity failure — walking it on a malformed prompt would ask three models the same bad question and report the third one's confusion. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
ee5a939ce6 |
fix(planner): the other four server-side calls were still on the metered key
The test that was supposed to prevent this grepped for the literal `runtime.complete(` and passed while the phase planner (`mission_plan.rs`), both swarm calls, and a second enhance path in `claws.rs` still billed the pay-as-you-go account. They spell the receiver `state.runtime` or wrap the call across lines, so the receiver name was never the thing to match. The test now matches the METHOD, and covers all five files. `complete_or` gains the rule that makes it safe to apply everywhere: a `name:model` spec is an operator's explicit provider choice — the swarm worker model is configured exactly that way — and is passed straight to `Runtime::resolve_provider` untouched. Only a bare name is ambiguous, and a bare name is precisely what resolves to the default provider. Hijacking a chosen Kimi or GLM model onto Anthropic would be the same silent-substitution bug pointed the other way. `validator_preflight` and the evaluator judge keep calling the runtime directly, on purpose: both exist to exercise the CONFIGURED spec. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
deed591da6 |
fix(roster): a rate-limited subscription is a 503 with a reason, not a 500
The retry landed and still failed: all four attempts returned 429. A bare
16-token probe with the same token, straight from gw-04, also returned 429
with `x-should-retry: true` — the Claude Code subscription itself is limited
right now, and no amount of backoff inside one HTTP request will outlast it.
So stop pretending it is a server bug. New `ApiError::Unavailable` → 503,
carrying the one sentence the operator can act on ("clears on its own; try
again shortly"), instead of an opaque `internal error` that sends them into
the logs. The harness now prints the response body rather than the generic
"the planner produced no usable proposal", which is what hid both walls —
first the credit balance, now this.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
c3c4447810 |
fix(planner): wait out a rate limit instead of failing the whole proposal
Moving the roster and planner onto the subscription removed the credit wall and revealed the next one: the harness went from `400 credit balance too low` to `429 rate_limit_error`. A one-shot proposal call had no retry — there is no retry convention anywhere in cm-llm — so a limit that clears in seconds killed the "propose a team" button outright. Four attempts, 2/8/20s backoff, and only for errors that can actually clear: 429/5xx/transport. A 400, 401 or 404 returns immediately, because retrying those is a 30s hang ending in the identical message, which reads to an operator as a stall rather than a bad request. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar Sobh
|
72046e7985 |
fix(planner): server-side model calls run on the subscription, not the metered key
The roster planner died with `400 — "Your credit balance is too low to access the Anthropic API"` while every mission on the same machine kept running. Two Anthropic credentials reach this server and they bill differently: `ANTHROPIC_API_KEY` (sk-ant-api, metered, runs out) and the Claude Code subscription token (sk-ant-oat) that every VM already uses. `Runtime::complete` with a BARE model name — "claude-opus-4-8" — resolves to the default provider, which is the metered key. Three server-side callers did that: the roster planner, the Master Planner, and the claw enhancer. Missions were never affected because `mission_runtime` deliberately sends only the subscription token into a guest; the server had no equivalent rule. `subscription::complete_or` is now that rule, and it is the ONE place a subscription token becomes a provider — `evaluator::subscription_judge` had its own copy, and two of them is how one ends up with a prefix check the other lacks. The `sk-ant-oat` prefix is checked rather than the variable name trusted: an API key pasted into the OAuth slot would authenticate, work, and bill the metered account — the same failure again, discovered weeks later. `web_search` is carried explicitly rather than defaulted. The Master Planner and the claw enhancer both pass `true`, and a helper that quietly dropped it would have taken web search away from two features while every test still passed. `validator_preflight` deliberately keeps `Runtime::complete`: it probes whatever validator spec is configured (today `glm:glm-4.7`), and forcing it onto Anthropic would make it prove the wrong thing. A test pins both halves — no other server-side caller may regress to the metered key, and preflight must keep probing the configured spec. 258 lib tests. |
||
|
Omar Sobh
|
d84d17207f |
feat(placement): place per phase, and let a full fleet queue
Phase 1b: wires the capacity model from
|
||
|
Omar Sobh
|
3a2d76aa43 |
feat(placement): capacity model for the fleet — observed memory is not capacity
Phase 1a of the fleet-intelligence plan: the arithmetic and the inputs. Nothing is wired to it yet; the launch path still picks `capable.first()`. Placement has been `ORDER BY last_seen DESC` + `.first()` — the most recently heartbeated node. Among healthy nodes all heartbeating every 5s that is arbitrary, and it consults nothing about load, so two missions launched together land on the same machine. It did not matter while tank held the only rootfs image. All three nodes serve `claude` as of today. THE correctness point, and the reason this is not a sort change: a VM that booted 30 seconds ago holds a fraction of its 8 GiB claim, so `mem_pct` reports a sold-out node as nearly idle. `capacity_of` takes the WORSE of observed usage and committed usage. The negative control pins it with the measured case — tank at 60 GiB total / 12 GiB observed / 5 VMs booted: utilisation alone says 5 more fit, the node has room for 1. Booking those five is a node in swap, which slows every VM on it together. Commitments are unioned BY IDENTITY, never added: `vm_list` reports booted VMs, `nodes::pinned_microvm_phases` reports phases chosen but not yet booted (a window of seconds in which a real 8 GiB claim exists that no node can report). The deterministic `vm_id_for` is what lets the same phase be recognised in both — counting it twice would shrink the fleet by the number of phases starting. `EvalRow::headroom()` finally gets a caller. It was written with the doc comment "for placement ranking" and has had zero callers since. It is a TIEBREAK, not a gate: ranking is slots first (spread, don't stack), then live headroom, then node id so the same fleet state yields the same answer twice — which `last_seen DESC` could never promise. Fail-closed per house convention: draining, stale health (>30s, tuned just above the 20s offline sweeper), and an unanswerable `vm_list` are all INELIGIBLE rather than low-scoring. Stale Beszel metrics are the one exception — they demote a node to zero headroom instead of excluding it, because they only ever break ties. `FleetAtCapacity` and `FleetUnreadable` are separate variants with a test asserting the second never says "at capacity": an operator sent hunting a load problem that is really a dead daemon wastes the outage. Also names the two nodes that were both called "New node" (tank, morpheus) — a capacity report naming two machines identically is one nobody can act on. 257 lib tests. |
||
|
Omar Sobh
|
5c5f1ced33 |
refactor: the judge's sandbox joins the other three copy sites on root_copy
Four places copy a mission checkout so a ROOT command can run against it without touching the live tree: the judge, the benchmark runner, the on_green_tests gate, and — until now — the judge again, with its own implementation predating the shared one. `evaluator_tools::Sandbox::for_checkout` now builds through `root_copy::RootCopy`. Same packer, same exclusion list, same reasoning in one place. It needs the copy to OUTLIVE the handle, because the judge has not run when `for_checkout` returns and a firing `Drop` would delete the tree out from under it. That is `into_workdir`, a method rather than a `mem::forget` at the call site: the transfer of cleanup responsibility is then visible in the type instead of implied by a leak. `Sandbox::purge` remains what actually clears it, since only a container running as root can remove the root-owned `target/`. What did NOT move: `pack_dir` in the microVM inject/collect path. That marshals a tree to and from a guest over vsock — a transport, not a host-side copy — and folding it in would merge two things that only look alike. 249 lib tests. |
||
|
Omar Sobh
|
099a716bfd |
fix(egress): a backend is defined in two maps, and the canary only had one
First canary run failed: phase failed, nothing delivered, and the streamed log said exactly why — "Failed to authenticate. API Error: 403 api.anthropic.com is not on the egress allow-list". Not a 2.1.226 regression. `canary-claude` was added to the server's credential map and not to the node's `provider_hosts`, so the VM booted with a valid subscription token and a door that only opened onto the forge. The fail-closed branch was working correctly: a backend nobody taught that function about reaches no model API, deliberately, so it cannot silently borrow another provider's door. Both maps now name it, each pointing at the other, with a test asserting the canary reaches the same provider as `claude` AND that unknown backends still resolve to nothing. Worth noting what made this a five-second diagnosis instead of an afternoon: the live log streaming built earlier today. The failure was a 403 inside a microVM that no longer exists, and its reason was sitting in the run's checkpoint. |
||
|
Omar Sobh
|
4193ae2cda |
feat(missions): a canary backend for testing a CLI version on the real path
Claude Code 2.1.223 -> 2.1.226 is worth taking (2.1.225 fixes a transient 401
that replaced a long-lived CLAUDE_CODE_OAUTH_TOKEN with a short-lived one and
broke HEADLESS sessions until restart — which for us means a failed phase). But
the image every mission uses is not the place to find out whether a new CLI
still delegates, still accepts `--settings`, and still finishes.
`canary-claude` is a real rootfs built from the candidate version, credentialed
identically to `claude`, so a mission can exercise it through the production
path: egress, stop gate, delegation, delivery, streaming. Testing a new CLI
against a different provider would not be testing the thing we are about to ship.
Named explicitly rather than matched on a prefix. An unrecognised backend must
still be refused at launch — that is what `backend_can_run_a_mission` and the
harness's `microvm-negctl` scenario assert — and loosening the credential map is
exactly how that guard gets softened by accident. A test pins both halves.
Already cleared by direct measurement in a booted 2.1.226 VM, before this:
- `--settings` and `--agents` still exist
- the workspace trust prompt added in 2.1.225 does NOT apply: `--help` states
the dialog is skipped in non-interactive mode (`-p`, or stdout not a TTY).
We use both.
|
||
|
Omar Sobh
|
8c93cd8569 |
fix(runs): the composed worker's checkpoint wiped the live log on every node
Composed missions streamed ZERO bytes while solo missions streamed fine. Same executor, same command, same guest — `HubVms::run` is a straight passthrough — and the node logged a tail starting for all five graph nodes against the correct outer run id, with no errors. The bytes simply were not there at the end. Two writers, one column. `fleet.rs` appends live output under `checkpoint.log`; `topology_runs::checkpoint` wrote `SET checkpoint = $2`, replacing the whole object. A composed run checkpoints after EVERY graph node, so each node's progress silently erased the log written during it. A solo run has no second writer, which is exactly why it looked like it worked. Now merged with `||`. The keys are disjoint, so the progress object still wins for everything it owns. I was wrong about the cause twice before finding this. First I blamed the guest agent's serial accept loop — real, fixed, and not this. Then I blamed pipe buffering racing the abort at turn end — plausible, and the drain fix is right on its own merits, but composed still streamed zero afterwards, which is what ruled it out. The thing that actually located it was noticing solo and composed differ by a WRITER, not by a code path. |
||
|
Omar Sobh
|
09afa7e7ff |
fix(node): aborting the tail at turn end raced the flush that matters most
Composed runs streamed NOTHING while solo runs streamed fine — same code path, `HubVms::run` is a straight passthrough, and the node logged a tail starting for all five graph nodes with the correct outer run id. The difference was timing. `claude -p ... | tee` makes stdout a PIPE, so the CLI block-buffers and flushes at EXIT. The most valuable output — the agent's summary of what it did — arrives in the instant the turn ends. The node aborted the tail the moment `handle_op` returned, so that flush was a race: a solo turn (minutes long, output already flushed by size) won it and streamed 337 bytes; each node of a composed run (~20s) lost it and streamed zero. The tail now DRAINS. A flag is set when the turn returns, and the loop exits only after a pass that read nothing new — checked AFTER a read, never before one, because exiting on the flag alone would drop exactly the bytes this exists to capture. Bounded by a 20s timeout with the abort kept as a backstop rather than the mechanism, so a VM that stopped answering cannot hold the task open. Worth naming: 5 tails started, 5 logged cleanly, 0 bytes arrived. Every individual step reported success and the feature did nothing — the same shape as the empty Live tab this whole thread began with, one layer down. |
||
|
Omar Sobh
|
5b49d5a1a8 |
feat(merge): gate publication on the merged tree's own tests
The other half of the merge button. Merging told you the branch went in; nothing
checked that what came out still worked.
Verified BEFORE publishing, not reverted after. `merge_locally` and
`push_merged` are separate functions so the caller can run the project's tests
between them, which means a merge that breaks the base is simply never pushed —
`main` is not broken for however long it takes someone to notice. A test asserts
`merge_locally` contains no push, because the moment it does, verification
becomes after-the-fact and the guarantee is gone.
Outcomes, all reported to the operator rather than swallowed:
Passed -> published
NoSuite -> published, and SAID so; a repo with no tests is a fact about the
repo, not a pass
Failed -> not published, exit code reported, branch untouched so it can be
fixed and merged again
CouldNotRun -> not published. Fail closed: a suite that could not run has not
passed, and publishing on "we could not check" is how a green
main stops meaning anything.
`verify_tests` runs `cargo test` as ROOT in a container, so the merge workdir
ends up holding a root-owned `target/` the server (uid 65532) cannot delete —
the same leak found three times today. Purged through the container before the
ordinary cleanup.
248 lib tests.
|
||
|
Omar Sobh
|
28090d1de0 |
fix(node): the log tail gave up before the turn wrote its first byte
First live test of the streaming path: mission passed 6/6, `checkpoint.log` was 0 bytes, and the node logged nothing at all. `stream_vm_log` treated "no progress" as "the turn finished writing". But the guest's `tail` reports EOF after every idle window, and the FIRST idle window is always the one before any output exists — the VM is still booting and the CLI still starting. So the tail returned `at == 0`, the node concluded the turn was done, and it stopped seconds into a run that then went on for minutes. The abort is the terminator, not idleness: the caller already aborts this task when the exec returns, so waiting cannot outlive the turn. No-progress now sleeps and retries instead of returning. Also logs when a tail STARTS. The bug was invisible in exactly the way this session keeps finding: silence on the success path, silence on the give-up path, and an empty Live tab that looked identical to a feature nobody had wired. Method note, since it cost time: I tried to confirm the deployed binary by grepping it for `vm_out` and found zero — then found zero for `pty_out` and `vm_exec` too, in a binary whose PTY streaming demonstrably works. Binary-grep is not a reliable presence test for these literals; `stream_vm_log` and `tail of` being present is what actually showed the code had shipped. |
||
|
Omar Sobh
|
0b89b8316c |
feat(observability): stream a microVM turn's stdout/stderr to the platform live
The Live tab showed nothing while a turn ran, and the agent's own account of it
went to stderr on the node and nowhere a user could reach. This is the path that
carries it.
The blocker was the guest agent. `fcagent` handled one connection at a time,
inline, so during an hour-long turn the VM accepted nothing — which is why every
existing probe (subagents, stop-gate blocks, cap) runs AFTER the turn rather than
during it. It now spawns a thread per connection, wrapped in `catch_unwind`
because this process is pid 1: a panic used to take the accept loop with it, and
an unbootable VM is a far worse outcome than a missing log. A failed spawn logs
and keeps accepting rather than dropping the listener.
PROVED against a live VM before building on it, since "sound reasoning about this
system" and "measurement" have diverged repeatedly today. Patched rootfs, booted
under Firecracker, ran an 8s exec and a concurrent tail:
exec took 8.0s ok=True
+0.0s 'line1\nline2\n' +1.2s 'line4\n' +3.2s 'line6\n' +6.0s 'DONE\n'
VERDICT: CONCURRENT — tail returned data before exec finished
The rest is the pattern the terminal already uses. New `tail` op streams a file
by OFFSET (so a dropped link resumes instead of replaying, and the tail always
terminates — one that never returns pins a thread for the life of the VM). The
node follows the log alongside the turn and pushes `Uplink::VmOut { run_id, at,
data }` over the WebSocket it already holds, mirroring `PtyOut`. The server does
what `PtyOut` deliberately does not: it APPENDS to the run's checkpoint as well
as fanning out, because a terminal has no history worth keeping and a mission log
is the record of what the agent did. `run_events_sse` emits the new bytes as
`step` events, which the live pane already renders — no frontend change.
The turn is `tee`d, not redirected: the file feeds the live stream and stdout
still becomes `VmOutcome::summary`. A redirect would have produced a live view
and an empty summary, which is the same green-and-empty shape as the bug this
fixes. Tested, along with the log living outside the collected tree so it never
lands in a user's delivered diff.
246 lib tests, 20 binaries; node and fcagent build clean.
|
||
|
Omar Sobh
|
62509a5090 |
fix(missions): a solo microVM run showed the operator an empty Live and Output tab
Found by a frontend wiring sweep, then confirmed in the database.
Everything the UI shows of a run's CONTENT reads
`topology_runs.checkpoint.records`: `/api/missions/{id}/documents` behind the
output reader, and `/api/topology-runs/{id}/events` behind the live pane. The
`team` and `microvm_graph` tiers write those records. The SOLO microVM path
never did — it updated `status` and nothing else:
tier | checkpoint_null | records
microvm_graph | f | 2-5
team | f | 5
microvm | t | 0 <-- every one
So a single-phase microVM mission ran real work, delivered a real branch, and
showed an empty Live tab and an empty Output tab. The agent's own account of the
turn went to stderr via eprintln and nowhere a user could reach.
Note what was NOT broken, since that was the initial suspicion: the SSE path
matches (`/api/topology-runs/{id}/events` on both sides), and a sweep of all 130
frontend `/api/` calls against the 164 registered routes found zero genuinely
missing endpoints. The wiring was fine; the data was absent.
The run now persists its turn as one record shaped exactly like the ones those
two readers already parse — `node_id`, `role` (the phase kind), `phase`,
`output` — so no reader changes. Written with `checkpoint || $3::jsonb` so a
future writer of other checkpoint keys is not clobbered.
246 lib tests.
|
||
|
Omar Sobh
|
3616bc4733 |
feat(missions): an operator button to merge a mission's branch into main
`MergePolicy::Never` — the default for anything touching code — has always meant
"do not merge on your own", deferring to a human. There was no way for that human
to say yes: `auto_merge` was reachable only from the paper-harvest path, no
workflow template declares `merge_policy`, and every mission ended at a branch.
`POST /api/missions/{id}/merge` is that yes, with a button on the artifacts tab.
The additive-only gate does NOT apply here, deliberately: an operator reading a
code change is exactly the judgement the policy was holding out for.
What is not waived:
- the branch comes from the artifact delivery RECORDED, not rebuilt from the
mission id, and must have `pushed: true`. A phase that never pushed shows no
button instead of one that cannot work.
- an empty branch is refused. A button reporting success for merging nothing
is worse than no button.
- a conflict refuses, aborts, and leaves the repo clean rather than forcing.
It works in a FRESH CLONE under `_merge/<mission>`, never the mission checkout:
that directory is reaped on a timer after a mission ends, so a merge using it
would succeed right after a run and fail inexplicably an hour later. The clone is
made by the server process, so nothing runs as root and ordinary cleanup works —
unlike the copies in `root_copy`.
`merge_and_push` is split out so the operator path and the automatic path run the
SAME git commands; only the gates differ. A test asserts both call it, that the
operator path does not re-apply the additive gate it exists to bypass, and that
it still refuses an empty branch.
Harness 43/43 across all five recipes before this change, with `_gate`, `_bench`
and `_verify` all at zero.
246 lib tests, 20 binaries, 89 frontend tests, clean build.
|
||
|
Omar Sobh
|
a8b8efba6a |
fix(delivery): the on_green_tests gate ran the suite in the live checkout
Fourth instance of the same defect, and the last of the three commands that run as root against a mission tree. `verify_tests` execs the project's test command with `workdir = repo` — the live checkout — inside a container running as ROOT. `cargo test` writes `target/`, so the checkout ends up owned by two uids and the next phase's cargo hits permission-denied. The harness reported `uids=0,65532` the first time this gate ever ran end to end. It survived because it had never run. Every one of the ten harness fixtures used `commit_policy: "always"`; `on_green_tests` and `on_reviewer_approval` were parsed, implemented, and never exercised — and `Gate`'s own doc already records that three recipes carried this policy while it "did precisely nothing" for want of a reader. A policy that is never exercised is indistinguishable from one that is ignored. Consolidated rather than fixed a third time. `root_copy` now owns the pattern — copy through `mission_fs::pack_dir` into a SIBLING of the mission dir, run there, and purge FROM INSIDE THE CONTAINER, because the copy's `target/` is root-owned and the server (uid 65532) cannot delete it. `benchmark_runner` moved onto it; `evaluator_tools::Sandbox` keeps its own copy logic for now (it carries an allow-list and a judge-facing API, so folding it in is a larger change than this moment warrants — noted, not done). The gate fails CLOSED if the copy cannot be made: an unverifiable suite must not license a push. Also adds the `refactor` scenario, which is what found this. I had written it off as "structurally identical to four existing scenarios" — wrong: it is the only recipe carrying `on_green_tests`, and that made it the only one testing this code path at all. 245 lib tests, 20 test binaries. |
||
|
Omar Sobh
|
a4b4d05b8d |
fix(evaluator): the verification sandbox leaked for the same reason the bench copy did
Found by checking `_verify` after fixing the identical bug in `_bench`: 16 MB
stranded across two copies, the oldest hours old.
`Sandbox::Drop` calls `std::fs::remove_dir_all` as uid 65532. The judge runs
`cargo test` in a container as ROOT — that is the entire point of the sandbox —
so the copy's `target/` is root-owned and the removal fails on it, leaving the
whole tree. The error was logged to a stream nobody reads, so the sandbox that
exists to protect the checkout quietly filled the disk instead.
Its doc comment also claimed "the copy lives under `_verify/<mission>`, which
the next pass clears anyway". That was wrong for exactly the same reason:
`for_checkout` removes a stale root before copying, with the same uid, and fails
the same way. A leaked copy was permanent, not transient.
`Sandbox::purge` removes it from inside the container, as root, where it was
written. `evaluate` now wraps its body so the purge runs on EVERY exit — that
function returns from several branches, and cleanup only some paths reach is the
same as no cleanup on the others. `Drop` stays as a fallback for the early paths
where nothing has run as root yet, and its comment no longer claims otherwise.
This is the third instance today of the same shape: cleanup that cannot clean up,
invisible because the failure was swallowed. The others were the leaked agent
containers in the runtime tests and the bench copy in
|
||
|
Omar Sobh
|
e89a32ffef |
fix(benchmark): the bench copy leaked because only root could delete it
The copy fix in
|
||
|
Omar Sobh
|
a93a4111e1 |
fix(benchmark): the baseline runner was writing root-owned files into the checkout
Caught by the harness: `benchmark: checkout has multiple writers (uids=0,65532)`.
The previous full run passed that same check, so this was introduced by wiring
`benchmark_runner` into the sweep one commit ago.
`docker_exec` enters a container running as ROOT with the missions root
bind-mounted, and `cargo bench` writes `target/`. Run in the live tree it leaves
root-owned build output in a checkout owned by uid 65532 — the single-writer
invariant broken, and the next phase's cargo hitting permission-denied on a
directory it cannot write.
This is the SAME defect `evaluator_tools::Sandbox` was written for, found by the
same probe, and fixed the same way: benchmark a COPY. `BenchCopy` packs the
checkout through `mission_fs::pack_dir` (so it excludes exactly what the
delivered diff excludes — one exclusion list, now four consumers) into
`<missions_root>/_bench/<mission>`, a SIBLING of the per-mission dirs like
`_verify` and `_outputs`, so a mission reap cannot race a running bench. Removed
on drop, including on error paths.
The operator-triggered path (POST /api/missions/{id}/benchmark) had this bug
from the start and is fixed by the same change — it shares `run`.
Worth naming the pattern: measurement must not mutate what it measures. It
applies to the judge, to the `verifier` subagent that has no Edit or Write, and
now to the benchmark runner.
243 lib tests, zero warnings.
|
||
|
Omar Sobh
|
0d8db7ff0b |
fix: close the three remaining gaps, and repair a test I silently disabled
FIRST, the self-inflicted one. My edit in |
||
|
Omar Sobh
|
a20702d55b |
fix(evaluator): a bare validator model name claimed independence it never had
`CLAWMATES_VALIDATOR_MODEL=gemini-2.5-flash` (or any bare model name) produced
an Anthropic judge grading Anthropic work, recorded `independent = true`.
The chain:
- `provider_family` reads the SPEC. A bare `gemini-2.5-flash` matches none of
the known needles, so it returns "unknown" — deliberately NOT "anthropic",
so it passes the `family == IMPLEMENTER_FAMILY` guard.
- `Runtime::resolve_provider` (runtime.rs:224) falls back to the DEFAULT
provider for any spec it cannot route. A bare name has no `provider:` to
route on, so it silently returns the house Anthropic provider.
- The existing "no provider registered" guard checks `model.contains(':')`.
That works for `glm:glm-4.7` — an unrouted colon-spec comes back carrying
its colon — and can NEVER fire for a bare name.
So the one guarantee this path exists to make (the judge is not the implementer)
was reported as satisfied while being violated. That is the same shape as the
Goodhart incident the independent judge was built after: not a wrong answer, a
wrongly-trusted one.
A validator spec must now name its provider. `names_a_provider` is a named
predicate rather than an inline `contains(':')` so the rule is testable and the
reasoning has somewhere to live.
Found while auditing my own Gemini removal — which turned out to be
behaviour-neutral here (a gemini spec went from family "gemini" to "unknown",
both non-anthropic, same verdict). The bug is pre-existing and independent of
it; removing Gemini only made the bare `gemini-*` spelling more likely to be
left behind in someone's env.
Live config is `glm:glm-4.7`, a proper registry spec, so production behaviour is
unchanged. Negative control: make `names_a_provider` return true unconditionally
and `a_validator_spec_must_name_its_provider` fails.
241 lib tests pass.
|
||
|
Omar Sobh
|
87f188ae73 |
refactor: strip Gemini from the platform, and level up the architecture_mapper
Two things.
1. The architecture_mapper proposal, applied AND made durable.
The GLM proposal (019fddd9) was accepted in full: the agent's system_prompt now
carries the Mermaid-first constraint and its brain was rewritten. Both verified
against the live row and the .h5 file.
But `apply_identity` writes `UPDATE agents SET system_prompt` and
`apply_brain_consolidation` writes that agent's brain — neither touches the team
TEMPLATE. That agent is mission-scoped, so the improvement would have died with
the mission. The model's actual insight was sharp and worth keeping: "Mermaid
diagrams beat prose" lived in the brain SEED and not in the system PROMPT, so it
only applied when the agent happened to consult its brain. That constraint is
now in templates/teams/codebase_research.toml, where every future Codebase
Research team inherits it.
(The proposal's second item mostly restated anti-patterns the seed already
lists, so the seed is unchanged. Applying an LLM's suggestion is not the same as
agreeing with all of it.)
2. Gemini is gone.
Removed: the `gemini.default` provider alias and its `is_exact_provider_match`
prefix, GEMINI_API_KEY forwarding to agent containers, the evaluator's
gemini->gemini family row, the model selectors in claws/teams/planner and in
TeamWizard + AgentComputer, and the commented provider block in the runtime
config example (whose ZEROCLAW_AGENT_MAP example still mapped a worker_gemini
that no longer existed).
`provider_alias_for("gemini")` now returns claude_cli.default via the
unrecognised-model branch, which LOGS. A stray gemini binding degrades visibly
rather than resolving to a provider row we no longer ship. A test pins that, and
another pins that GEMINI_API_KEY is forwarded in NEITHER auth mode, so adding it
back to the list is a visible change rather than an accident.
Avatar generation is DELETED, not disabled — it called Gemini's image model, and
there is no alternative: Claude and Kimi are text-only, and z.ai answers
"Unknown Model" for cogview-3-flash and cogview-4 on our plan (measured, not
assumed). AvatarModal keeps UPLOAD, which never needed a provider; only the
prompt-generation half is gone.
240 backend lib tests, 89 frontend tests, clean tsc + eslint, build succeeds.
|
||
|
Omar Sobh
|
f6c3ddbf81 |
refactor: no feature depends on Gemini any more
Depleted Gemini prepayment credits took out PDF rendering. The same key was the
only thing standing between level-up proposals and the same fate, so both are
off it.
- `pdf_renderer` is DELETED, not disabled. Nothing sets `render_pdf: true` since
markdown became the deliverable (
|
||
|
Omar Sobh
|
821cbb8622 |
feat(missions): hold every producing phase to delivering, and read markdown instead of PDFs
Two changes the portal review asked for.
1. `benchmark` and `security_hardening` had no delivery guarantee.
`empty_delivery_is_a_failure` tested `kind == "coding"`, on the reasoning that
"research phases legitimately write nothing to the tree" — which the research
directive three modules over contradicts, since it tells the agent to save
findings under /mission/repo/research/. The cost: a `benchmark` mission is ONE
benchmark phase, and with that phase exempt nothing in the platform could fail
it. Same for `security_hardening`, whose first two phases are security_scan and
research.
Now keyed on PRODUCING_KINDS = coding, research, benchmark, security_scan.
`review` stays exempt — a reviewing phase that changes nothing has done its job,
the same distinction `vm_stop_gate::per_node` makes. The test that encoded the
old rule is rewritten rather than deleted, with the reasoning that replaced it.
All 8 harness fixtures are coding phases, so harness behaviour is unchanged.
2. PDFs are dropped; markdown is the deliverable.
Rendering a PDF meant asking an LLM to convert markdown to HTML — a paid API
call per document, on the critical path of "let me read my research", which
failed on depleted Gemini credits and left every artifact unreadable. Styling at
render time is free, offline, instant and cannot 429.
- `mission_outputs` no longer requests a render.
- New `GET /api/missions/{id}/artifacts/{artifact_id}/content`. The frontend had
no way to READ an artifact at all: it listed paths and offered a PDF preview
that never rendered (and whose `rendered_pdf_path` had no route serving it).
Two containment rules, both enforced: the artifact must belong to a mission in
the caller's workspace, and the CANONICALISED path must stay under `_outputs`
— canonicalise first, because checking the string before resolving `..` is the
classic hole.
- `MarkdownBlock` now uses react-markdown + remark-gfm + rehype-slug. It was a
deliberate zero-dep renderer for "the subset the refiner emits", and that
subset stopped matching reality: agent briefs are largely GFM pipe tables,
which it showed as literal pipes. MissionOutputReader and RefineDiffModal use
the same component and gain tables for free.
- Heading ids come from rehype-slug and `outlineOf` slugs with the same
GithubSlugger, so the outline rail's anchors still resolve. A test pins that
invariant, including duplicate headings.
Styles live in globals.css under `.md-view`: the markup is generated so there
are no class hooks, and this project has no styled-jsx registry — the app-router
requirement is documented in next/dist/docs/01-app/02-guides/css-in-js.md, which
frontend/AGENTS.md exists to make me read.
The artifacts tab moved to `MissionArtifacts.tsx`. MissionCanvas was 1341 lines
against a 1250 limit BEFORE this change — already failing lint; it is now 1248.
238 backend lib tests, 20 backend test binaries, 89 frontend tests, clean tsc,
clean eslint on every file touched, production build succeeds.
|
||
|
Omar Sobh
|
da889f83ab |
fix(missions): an empty repo-less phase was re-processed on every tick forever
The guard added in
|
||
|
Omar Sobh
|
c28c7a148f |
fix(pdf): the renderer resolved every artifact path against the wrong root
`render_one` joined `missions_root()/<mission_id>/` before the artifact path, producing `<root>/<mission>/_outputs/<mission>/<phase>/...` — the mission id twice, and no such file. Artifact paths are relative to the MISSIONS ROOT. All three registration sites write `_outputs/<mission>/<phase>/...`, and `_outputs` is deliberately a sibling of the per-mission directories so it survives their reaping; joining the mission id first put the lookup inside the very directory `_outputs` exists to escape. It went unnoticed because until now the only artifacts on the system were `code_diff` rows registered with `render_pdf: false`, which this worker never reads. `produces = ["md","pdf"]` was inert, so nothing ever asked for a render. The first artifacts to ask were the first to find it — both failed with ENOENT on the doubled path. Negative control: restore the extra join and `an_artifact_path_resolves_against_the_missions_root` fails. The worker's error handling is sound and needed no change: it recorded `render_pdf_status = 'failed'` with the full path in `render_pdf_error`, which is how this was diagnosed in one read. 237 lib tests pass. |
||
|
Omar Sobh
|
89bc53b53d |
fix(missions): repo-less capture was publishing the agent's own identity files
First live run of `capture_repo_less_phases`: 9 artifacts, of which 2 were the user's research. The other 7 were AGENTS.md, HEARTBEAT.md, IDENTITY.md, MEMORY.md, SOUL.md, TOOLS.md and USER.md — the agent runtime's identity scaffolding, seeded into the workspace root because that root is pinned to the repository root. The codebase already knew about these files and already had the list. What it did not have is a defence that works without a repo: `ignore_agent_scaffolding` writes them to `.git/info/exclude`, and a mission with no repository has no `.git`. So the exact files that once got committed into a user's repo and pushed (the reason that list exists) came back through a new channel. `AGENT_SCAFFOLDING` is now `pub(crate)` and `mission_outputs` filters on it directly — one list, two consumers, so the next file the runtime starts seeding is excluded from both at once rather than from whichever was remembered. Negative control: replace the filter with `&& true` and `research_documents_are_kept_and_scaffolding_is_not` fails. Found by running it against a live mission, not by reading it. The unit tests passed the whole time — they seeded a tree that did not contain the scaffolding, because I did not know it would be there. |
||
|
Omar Sobh
|
ceab28b902 |
fix(missions): a repo-less mission threw away everything its agents wrote
`capture_finished_coding_phases` selects `AND m.repo_id IS NOT NULL`. Every
`research_only` mission is repo-less by design (`requires_repo = false`), so the
whole capture path — including the `sync_out` that copies the agent's work OUT
of the container — never ran, and the container was reaped unread.
Measured on the real mission `019fdc35` ("ClawHDF5 Research"): four agents, 9.5
minutes, EIGHT research documents — an HDF5 parser design, a Rust ecosystem
survey, a seven-crate dependency map, tracing and fuzzing strategy. Result:
`mission_artifacts` = 0, mission `completed`. Not recoverable: no container, no
volume, nothing under the missions root.
The platform did not merely fail to save the work — it INSTRUCTED it. The task
preamble tells every agent "/mission/repo ... is the mission's git checkout",
whether or not one exists, and the research directive says to save findings
there. One agent recorded the contradiction verbatim: "No git repo — file is
written." It looked, saw no repo, complied anyway.
Three changes, one per link in that chain:
1. `mission_outputs::capture_repo_less_phases` — copies `/mission/repo` out of
the container and registers each file as an artifact under `_outputs/`,
which is a SIBLING of the mission dir and survives `teardown_container`.
This is also the code that finally reads `produces`, until now an inert key:
`produces = ["md","pdf"]` now drives `render_pdf` into the existing
pdf_renderer worker.
2. The preamble is conditional. A repo-less mission is told its workspace is
scratch, that git_operations has nothing to act on, and — the part that
matters — that files left there ARE collected and published. An agent told
only "there is no repo" has no reason to write anything to disk.
3. A repo-less phase that produced no files is FAILED, unless it declares
`allow_empty`. The same rule `empty_delivery_is_a_failure` applies to coding,
for the only channel these phases have. Note this is NOT that guard widened:
it keys on `files_changed`, which is meaningless with no checkout, and would
not have saved the ClawHDF5 documents.
Negative controls, each ablated and confirmed failing: ignore `has_repo` and the
preamble test fails; empty the skip-list and the capture test keeps `.git` and
`node_modules`; write artifacts inside the mission dir and the survives-the-reap
test fails.
236 lib tests pass.
|