736b6a9a82cad72c1a7a2716d34c25fcc4ce4e42
68
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar SobhandClaude Opus 5
|
7dd3aa0965 |
feat(skills): files is the default delivery arm
The A/B has its answer. Across four matched production runs — same recipe, same task, same three offered skills — the MCP-door arm retrieved 1 in 9 and the file arm retrieved 3 of 3, with the judge loop closing on the same run (01a098dd). A signal, not a rate; but 0, 1, 0 → 3 on an otherwise identical task is not noise, and the mechanism is explained rather than guessed: the door is a deferred tool the agents never load, and Read is not. A code default and not CLAWMATES_SKILL_DELIVERY on one server, for the reason always_inject moved into the skill files: a setting that exists only in one deployment is a setting nobody can find. The env var still overrides, and `index` and `inline` stay selectable per mission so the comparison remains runnable against one binary. Garbage in the env var still falls to `inline`, not to the default — an unreadable value must not silently select an arm that needs something installed. A test pins the default so the next change to it is a decision made with the numbers in front of you, not a slip. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz |
||
|
Omar SobhandClaude Opus 5
|
00160739de |
feat(skills): a files arm — progressive disclosure through Read, not a deferred tool
The `index` arm retrieves through `ReadMcpResourceTool`, which is DEFERRED:
absent from the agent's default tool list until `ToolSearch` loads it. Across
three matched production runs (same recipe, same task, same three offered
uris) it retrieved 1 skill in 9 chances:
01a07812 delegation forced no instruction 0/3
01a0842e no delegation no instruction 1/3
01a09877 no delegation told to load it 0/3
The third run is the decisive one. The preamble said in plain words to run
ToolSearch first; all three prompts carried it; zero ToolSearch calls, and the
three reasoning narratives never mention skills at all. The section was not
declined, it was never engaged with. Instruction is not the lever.
`Read` is a core tool. Never deferred, and every one of those agents used it.
So this arm keeps progressive disclosure exactly as `index` has it — a name, a
`when_to_use`, and a pointer the agent has to follow — and changes only what
the pointer is: a path under /mission/skills instead of an MCP uri. The bodies
are written into the container at launch (every visible skill, one tar upload;
bindings resolve per agent at turn time so a per-mission subset is not knowable
here) and a `Read` of that path is a tapped tool call, so Trigger is exactly as
observable as before.
A third arm and not a replacement, selected per mission like the others, so
the comparison runs against one binary. `resolve` falls back to `inline` when
the files were not written, for the reason `index` does: a pointer to nothing
reads as an agent ignoring its skills.
The writer and reader of a path are one pair of functions
(`skill_file_path` / `skill_from_file_path`), matched by the scorer through
the same seam `parse_uri` uses, and the end-to-end test fails when the matcher
is broken. `Mode::is_retrieval` exists so the next arm cannot silently inherit
`inline`'s "not observable" for what is a miss.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
|
||
|
Omar SobhandClaude Opus 5
|
f52cff3e04 |
feat(skill-use): progressive disclosure, as an arm and not a switch
Trigger — did the agent reach for the skill when it applied? — cannot be measured while every body is inlined into the prompt. Nothing was reached for. `skill_use` has been reporting `NotObservable` for that reason, and it was right to. The skills door made retrieval possible; this makes it a delivery arm. `index` sends each pinned skill's name, description, `when_to_use` and the uri that returns its body, and the agent fetches what it judges relevant. `inline` is unchanged and stays the default. An A/B rather than a switch, because `index` can only cost Compliance: under `inline` the procedure sits in front of the model whether or not it noticed it applied. Trading a measured axis for an unmeasured regression in another is not an improvement, so both arms stay runnable and the arm is recorded on the mission row. Three things the mechanism refuses to do: - `index` without a door falls back to `inline`. An index names bodies and says how to fetch them; with no `clawmates_skills` server reachable that is a list of dead ends, and it fails as an agent ignoring its skills rather than as a missing config. `install_skills_door` now returns whether it installed, because the caller needs the answer and not just the log line. - The scorer reads the arm off the recorded PROMPT, not off the mission row. The row says what the mission is configured to do now; the score is being computed against a turn that ran then. - Under `index`, a skill that was offered and never read is a Fail, not the inline arm's `NotObservable` — but only where the skill had a checkable consequence in that phase. Reusing the inline text would have said "this skill was inlined into the prompt" about a skill whose body was never sent, and scoring a real miss as a structural blind spot is the failure this measurement already made once. The arm is per mission (`config.skill_delivery`), not only per deployment. Both arms run against one server process; restarting between them would put a confound in the comparison that the numbers would not show. 829 tests, 108 binaries, green. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
c209e654d9 |
fix(skill-use): a research phase writing markdown is not a TDD failure
The first live scoring of run 3 reported `cargo-test-driven-development` and `tdd-red-green-refactor` as compliance=FAIL: files were written and no test ever ran. Wrong, and wrong in the way this module exists to prevent. The phase wrote fifteen markdown notes and a helper script; there was no code to test-drive. Reporting it as an agent failure is a system defect wearing an agent's name — and it would have buried the actual finding, which is that a repo-less `research_only` mission is staffed with a Rust SDLC crew whose coder, tester, reviewer and committer have nothing to do. The check is now scoped to files with a source extension in the languages the skill itself names. Shell is deliberately excluded: a helper script written during a research turn is not behaviour-adding code, and the false failure costs more than the missed one. Recorded in SKILL-USE-BASELINE.md as finding 8 rather than quietly corrected. A measurement that hides its own false positives cannot be trusted about anyone else's. Also in the doc: the Trigger reason is half false now (the transport can surface a tool call; we simply still inline), and the architecture doc's observe/gate table said the container tier was ungated and unobserved, which shipped work has made wrong. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9 |
||
|
Omar SobhandClaude Opus 5
|
4a6d0dfe01 |
test(skill-use): keep the harness that runs the measurement
The first baseline was produced by a throwaway script that no longer exists, so the second measurement could not be run the same way as the first — which is most of what makes two numbers comparable. Local stack only, because production auth is Clerk and a mission cannot be launched from a terminal there. `--score <id>` re-scores a finished run without spending another one, and every run is held for 90 days so it stays re-scorable when the scorer changes again. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9 |
||
|
Omar SobhandClaude Opus 5
|
769e002bb3 |
feat(skills): deliver on every tier, record what agents receive, let them self-author
Three phases of the approved plan, plus a correction to what the last one
claimed.
CORRECTION: skills reached ONE tier, not all of them
The previous commit said "skills can now reach a mission agent". That was
true only for the container/ZeroClaw tier — the fall-through that queues a
topology_runs row for topology_worker, which drives the executor that was
patched. compose_turn_prompt/pinned_skills_text had exactly one production
caller, and phase_runner's three other paths (composed microVM, solo
microVM, direct session) never called it. CAPABILITY-REVIEW.md said the
broad thing too; both are corrected.
Those three tiers share one task string and have no per-turn alias, so
their skills resolve per PHASE from the mission's crew and are appended
there. The container tier deliberately still injects per turn, with the
running node's own role — appending in both places would put every crew
member's skills in every turn twice.
The behavioural tests prove phase_skills_text and compose_turn_prompt work.
They cannot prove the three launch_* calls pass the composed string, and
that substitution is a one-word edit that would silently return all three
tiers to delivering nothing with every test still green. So there is also a
source-level assertion on the call sites, following the precedent in
mission_events::the_cap_is_enforced_in_one_statement. Its negative control
names the exact tier.
PROVENANCE: what an agent received, and what it said it did
Both were unanswerable. The prompt was never stored anywhere on any tier —
re-deriving it later re-runs the skill lookup against a catalogue that has
since changed, and once agents author their own skills it certainly will
have. The reasoning rows were durably write-only: pushed live once, then
never read from the database again by anything except the GC that deletes
them.
- prompt.composed records the exact bytes, on all four tiers
- the session tier writes its checkpoint record and a reasoning row,
instead of eprintln! and nothing — the same defect the solo microVM
path was fixed for, in the last tier that still had it
- narrative_for_mission reads both back
Found while doing it: the 400-event per-phase cap counted EVERY kind, so a
busy phase could push out its own phase.completed and its own provenance.
The cap now counts only the two unbounded kinds it was written for.
Negative control confirms the old behaviour dropped the prompt.
Retention is now a per-mission hold (0080) rather than a raised global —
with a test asserting unheld missions are still reaped, because an
exemption that applies to everything is not an exemption.
SELF-AUTHORING: agents apply their own skill drafts, no human click
By operator decision. level_up has generated complete drafts from a model
since it shipped; only a checkbox stood between propose and apply.
What replaces the gate is not another gate but four properties, each held
by a test:
- workspace-scoped, so a hand-authored skill can never be modified
- a draft cannot take a hand-authored skill's name. Ids are scoped and
bindings resolve by skill_id, so it could not overwrite or shadow one
anyway — but two procedures under one name means nobody reading a
transcript can tell which the agent followed, and that ambiguity is
fatal in a system where the skill is the standard being graded against
- every revision appends a skill_versions row, so it can be reverted and
a past run can be read against the text it was actually judged under
- approved_by = NULL. An agent's decision is never attributed to a person
who did not make it
Only skill_candidate applies autonomously. identity_refinement and
brain_consolidation still wait for a human: they change what an agent IS
rather than adding a procedure it can consult. State is announced at boot,
because a safety gate that changes silently is one nobody notices changed.
CLAWMATES_SKILL_SELF_AUTHORING=0 restores it.
Also: the test Postgres ran out of /dev/shm mid-suite (Docker's 64MB
default) and surfaced it during MIGRATIONS, which reads like a schema fault
and is not one. --shm-size=1g, and a pointer to the `clean` subcommand that
already existed for the 779 leaked test databases.
Full workspace suite green: 106 binaries, no failures.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
875ff948f8 |
fix(rehearsal): probe the port the bundle actually publishes
The health check polled 127.0.0.1:18080, but deploy/compose/docker-compose.yml publishes "8080:8080" and deploy/airgapped/install.sh does not rewrite ports. Nothing was ever listening on 18080, so the rehearsal always ended in "platform never became healthy" — regardless of whether the install worked. Visible now only because the earlier failures (no cargo, compose v1, project collision) all stopped the script before it got this far. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
e7d2fc9696 |
fix(rehearsal): never adopt the production compose project
INCIDENT: the release rehearsal destroyed production data on gw-04. deploy/compose/docker-compose.yml declares `name: clawmates` at the top level, and that beats --project-directory. So `compose up` from a temp directory did not create an isolated stack — it ADOPTED the running production stack of the same name, recreated its containers, and then the cleanup trap's `down -v` deleted its volumes, including clawmates_pgdata. Prod came back with an empty database: 177 repos, all missions and all agents gone. There were no backups. The fix is `-p rehearse-$$` on every invocation, plus an assertion that refuses to run under the production project name. Isolation here was implicit and therefore not isolation at all. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
41854c70e1 |
ci: the install rehearsal needs compose v2, and says so
The v1 fallback I added a commit ago cannot work: deploy/compose/docker-compose.yml
uses v2-only syntax — a top-level `name:` and long-form
`env_file: {path, required}` — so docker-compose 1.29 rejects the file outright
("'name' does not match any of the regexes"). A fallback that always fails is
worse than no fallback, so the script now requires v2 and fails immediately with
what to do about it.
$COMPOSE overrides the detection. gw-04 is deliberately left WITHOUT a
`docker compose` plugin: installing one system-wide would flip the production
rolling deploy (clawmates-deploy.sh prefers v2 when present) off docker-compose
v1 as an invisible side effect of a release change. The runner gets a standalone
v2 binary at /opt/act-runner/bin/docker-compose and the workflow passes it in,
so prod keeps rolling exactly as it did.
Verified on gw-04: standalone v2.32.4 runs, and `docker compose` still resolves
to nothing, so clawmates-deploy.sh takes its v1 branch unchanged.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
9441cf401c |
ci: make the install rehearsal work with compose v1
The rehearsal reached "First boot" — bundle assembled, signed, verified offline, images loaded, install staged — and then died with `unknown flag: --project-directory`. That message is misleading: gw-04 has no docker compose v2 plugin at all, only docker-compose 1.29.2, so `docker compose` is parsed as `docker` with a bogus flag rather than reported as a missing plugin. Use the same v2-then-v1 fallback deploy/gw-04/clawmates-deploy.sh already needs. v1.29.2 supports --project-directory, so the invocations are otherwise unchanged. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
bd1c970577 |
ci: let the install rehearsal use a pre-built bundler
The rehearsal hardcoded `cargo build -p clawmates-bundler`, so it died with "cargo: command not found" on the release runner — gw-04 builds Rust inside a container and has no toolchain of its own. The release job had already built the bundler two steps earlier, so it was also redundant work. CLAWMATES_BUNDLER now short-circuits that build when it points at an executable, falling back to cargo otherwise, so running the script by hand is unchanged. Everything before this step already passed on the runner: images built, SBOMs generated, bundle assembled and signed, and "bundle OK: 94 artifacts verified offline" inside a --network none container. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
8470534e33 |
chore(fleet): drop morpheus from the deploy loop
morpheus is packed for a move. It is drained in the `nodes` table — heartbeats preserve `draining`, so it stays out of placement when it comes back — and removed from the agent-image loop here. An unreachable host in NODES does not merely skip it. The image loop fails the whole script BEFORE its verify stage, so four deploys in a row rolled the server and frontend correctly and then reported nothing at all; every one had to be confirmed by hand. Keep this list to hosts that answer. The name is left in a comment rather than deleted: putting it back is one word, and the next person will want to know where it went. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
768e106614 |
fix(microvm): a mission with no repository can run in a VM, and its work comes back
Two halves, and the first was worse than the plan assumed. `run_phase_in_vm`
packed `<missions_root>/<mission>/repo` unconditionally — a directory a
repo-less mission does not have — and then required `/mission/repo/.git` inside
the guest before spending a turn. So a repo-less microVM phase did not merely
go uncaptured: it failed before the agent ran.
A repo-less mission now gets an EMPTY workspace at the same guest path, created
host-side so the collect unpacks back over it with no special case, and the
readiness probe asks for what was actually sent — the directory rather than a
`.git` that was never going to be there.
`mission_outputs` then drops its `runtime_kind <> 'microvm'` exclusion, whose
stated reason ("a microVM mission always has a checkout") is exactly what
stopped being true. Where the files come from now depends on the runtime, and
the difference is not cosmetic: a container mission's output is still inside a
running container, while a VM's has already been unpacked onto the host by the
end-of-turn collect. Asking docker for a VM mission's files would query a
container that never existed.
The recursive copy skips symlinks rather than following them — a link out of
the tree would publish whatever it points at.
`research-vm` is the proof, added to the suite as well as the dispatch: the same
assertions as `research-only` with `runtime_kind: microvm`. A scenario nobody
runs is a scenario that does not exist.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
f27d2605eb |
fix(agents): a soft-deleted agent could never be purged
Clearing the fleet's four leftover agents returned 404 on every one. They had
been soft-deleted back in June — correctly invisible in the UI ever since — and
`agents::get` filters `deleted_at IS NULL`, so `workspace_agent` could not find
them. Every route uses it, including `batch-delete`, the one that exists to
HARD-purge. So a soft-deleted agent was unreachable from the application
entirely and its row stayed forever.
`get_any` sees them, and only the purge path uses it: hiding soft-deleted rows
is right for every read, and wrong for the one operation whose whole job is
removing them. Written with `query_as` rather than the checked macro so it does
not force an offline-cache regeneration on every machine that builds this.
`fleet-reset.sh` now uses `batch-delete` for agents rather than
`DELETE /api/claws/{id}`. The latter is a SOFT delete, so pointing a reset
script at it would have quietly added to the pile it was meant to clear.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
171f901bcd |
feat(ops): fleet-reset — delete every mission and PROVE the disk came back
For a clean slate before a UI session, and for the thing that keeps being true here: deleting a row has never deleted a directory. A full harness run leaves ~35 missions, each with a repo checkout and a runtime-data tree, on the smallest disk in the fleet. There are 125 rows and 117 directories right now. Deletes through the API, never with SQL. `missions::delete` tears down the per-mission runtime container, hard-purges the FK graph in order, and removes the workspace directory — falling back to a root purge for the files the per-mission daemon leaves as root. A `DELETE FROM missions` skips all three and orphans every one of them, which is how the orphans got there. Then it checks, because rows gone is not bytes back and every incarnation of this cleanup has managed the first while silently failing the second: it names each directory left without a row, and counts root-owned residue separately because that is the specific way it fails. Refuses outright while any mission is RUNNING. Yanking a live mission's checkout leaves a VM writing into a directory that no longer exists, and the symptom is a phase that hangs rather than one that fails. Verified: it stopped exactly there against the in-flight harness. Dry by default; `--yes` to act; `KEEP=<substring>` to spare some. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
e7b412d578 |
test(harness): local-ornith was missing from the all suite
Added to the case dispatch when it was written, and not to `all` — so the newest backend, and the only one that runs on hardware we own, was excluded from the one run that claims to check everything. A scenario nobody runs is a scenario that does not exist. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
5afcf63324 |
fix(harness): count what the roster run ADDED, not what the file holds
Forcing the planner onto the local link produced a green chain and a red assertion: roster: the planner sized this mission at 1 member(s) PASS roster: ROSTER.md has 3 line(s) for a 1-member roster FAIL The model was right and the check was wrong. ROSTER.md does not start empty — the auto-merge work put an earlier run's two lines onto main — so a 1-member roster that correctly appended one line delivered three, and the scenario reported a model that had ignored its own proposal. It now measures the DELTA against main. Any assertion against a scratch repo that accumulates has to, or it decays into a test of how many times it has been run before. Proven on the local model end to end: opus 429 -> local:ornith-fleet:9b answered -> `mission_roster: ... local:ornith-fleet:9b proposed 1 member(s)` -> the composed graph ran -> the branch added exactly one line. 5/5. CLAWMATES_MODEL_FALLBACK is removed from gw-04's .env again; it was set only to force the last link for this test, and the deployed default is the full chain. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
774f17d194 |
test(fleet): a mission served entirely by the node's own GPU
`local-ornith` scenario, green on its first real run against tank: local-ornith: a locally-served model delivered a guest kernel (6.1.128) local-ornith: no Anthropic egress from a locally-served mission local-ornith: the node bound its local-model socket for this VM local-ornith: checkout has exactly one writer (uid=65532) Three things had to be true at once and only a real run shows all three: the agent reached a model at all (a pipe to a closed port produces a turn that HANGS rather than errors, which is why this is a scenario and not a unit test), the work came back and landed on a branch, and the VM still could not reach api.anthropic.com. That last one is not theoretical. The node log for this VM is a column of `egress DENIED api.anthropic.com` — Claude Code's own telemetry, correctly refused — while the model traffic went through the vsock pipe and Ollama logged loading ornith-fleet:9b at 100% GPU with CONTEXT 131072. A local backend that quietly kept Anthropic egress would be a credential path nobody asked for. The egress check asks the NODE's proxy log rather than the agent, for the same reason the GLM measurement did: a model's account of where its tokens came from has no evidential value, and the proxy's record of what it dialled does. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
f56d41f5b7 |
feat(backend): local-ornith — a mission backend served by the node's own GPU
Claude Code pointed at the Ollama already installed on every GPU node. Ollama has served a native Anthropic-compatible /v1/messages since v0.14, so this is an env contract rather than a translation layer — the fourth variation on the same idea as agent-glm and agent-kimi. The route is NOT the egress proxy, and that is the design. `egress` speaks CONNECT, takes a destination from the guest, resolves it and decides; every one of those powers is a liability, which is why it refuses non-443 ports and IP literals after a unit test caught them being bypassed. Routing a local model through it would have meant relaxing both. `local_model` is the opposite shape: there is no destination in the protocol. fcagent listens on guest 127.0.0.1:11434 and pumps to vsock 9003; the node splices that onto its own 127.0.0.1:11434 and copies bytes. A compromised guest cannot redirect it because there is nothing to redirect — it is a pipe, not a proxy, and strictly narrower than anything an allow-list could express. The bytes never touch a network, so there is no wire for TLS to protect, and Ollama stays bound to loopback rather than being exposed on the tailnet. The socket is bound only for a backend declared to use a local model, so a `local-ornith` VM reaches the forge through egress and nothing else, while every other backend's guest port simply refuses. Both halves have negative controls. `scripts/fleet-model-setup.sh` exists because of one measurement: stock ornith:9b reported input_tokens=2050 for a 48000-word prompt and answered as though nothing had been dropped. Ollama's default window is ~2K whatever the model card says, and it truncates silently — the exact failure an agent turn would hit and never report. The script pins num_ctx=131072 into a derived tag and then PROVES both the window and tool calling before declaring success. Verified on architect: ~65536 words -> 65604 input tokens, stop_reason=tool_use. Placement needs no new capability key: building the rootfs only on GPU nodes means `nodes::online_for_backend`'s existing `rootfs @> ["local-ornith"]` predicate does the affinity, so morpheus never offers the backend. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
e96c5143bc |
test(eval): a local judge, and the 2K context window that would have hidden it
Phase 1 of the local-model plan: prove the model before writing any plumbing. `JUDGE=local` runs the existing done_when eval against Ollama on a GPU node. Requests originate on that node rather than the gateway, because the model is bound to 127.0.0.1 deliberately — it has no network exposure at all — and the gateway has no GPU. MEASURED on tank, 3 draws per case, against the incumbent on the same cases: local (ornith-fleet:9b) 14/15 — one UNPARSED, never a wrong verdict glm (glm-4.7) 13/15 — two WRONG verdicts on kernel-ok kernel-ok is the case production actually hit and the one this script's header says is expected to fail on glm-4.7. A 5.6 GB model on hardware we already own did not get it wrong once in three draws. The tag is `ornith-fleet:9b`, not `ornith:9b`, and that is the finding worth keeping. Ollama defaults to a ~2K window whatever the model claims: stock ornith:9b reported input_tokens=2050 for a 48000-word prompt and answered as though nothing had been dropped — silent truncation, confidently. The fleet tag pins num_ctx=131072, which measures 9.3 GB resident of a 16 GB card (the full 262144 also fits, at 13.6 GB, 100% GPU). These eval cases are a few hundred tokens, so this eval would have passed either way; that is exactly why the tag under test has to be the one production would use. Also measured: Anthropic /v1/messages returns well-formed tool_use with stop_reason=tool_use on both nodes; the reported count_tokens?beta=true hang is absent in 0.31.1 (clean 404, server unaffected); ~60 tok/s generate, ~2800 tok/s prefill, 120072-token prompts accepted end to end. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
13a35138e9 |
fix(placement): a drained previous node re-places the phase instead of failing it
`drain-midmission` found this. `choose` treated its `want` argument as a hard
requirement, and the only caller passes `missions.target_node_id` — which is
not an operator's choice, only where the PREVIOUS phase happened to run. Two
consequences, both wrong:
- A node drained or filled between phases produced `TargetUnfit`, which
`is_transient()` says false to, so `phase_runner` FAILED the phase rather
than queueing or moving it. The queue silently did not apply to the second
phase of any mission.
- While the node stayed fit, every later phase went straight back to it
regardless of ranking — accidental mission-to-node affinity, which this
module's own header says must not exist.
Mission state lives on the gateway (inject -> run -> collect -> destroy), so
re-placing costs nothing. The pin is now advisory: preferred while it fits,
and when it does not, the reason is logged and ranking proceeds. `TargetUnfit`
is deleted rather than left unconstructed, so it cannot come back as a
non-transient failure by accident.
The scenario had its own race: it waited for phase 0 to COMPLETE before
draining, but warm phases finish in ~80s against a 10s placement sweep, so
phase 1 was often already placed — and the run then blamed the platform for
running on a node that was not yet drained. It now drains while phase 0 is
still running, which does not disturb a live VM and is the more faithful test.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
d4af58be85 |
fix(harness): the capacity burst got faster than the thing watching it
The first bursts took 25 minutes because every VM paid a cold 2.4 GB rootfs copy. Warm, the same 16 missions finish in 70-140s each and the whole burst is over in about two minutes — so a sampler that waited ~90s for its launch check and then ticked every 15s caught three samples of the tail and reported "architect peaked at 1 of 6" for a run that sat at 6/6/2. Sampling now starts at the first tick, runs every 5s, and folds the launch check into the same query so verifying the launches costs no observation window. The 10-sample floor that produced the last NORUN is gone; it was measuring how long the burst took, not how well it was watched. And "nothing queued" no longer has one verdict for two causes. If the fleet never actually filled — a slot can free before the sweep reaches the 15th mission — the queue was not reached and this scenario did not test it: NORUN, naming the high-water mark. Only a burst that DID saturate can call an absent queue a failure. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
eacd3ee085 |
fix(harness): a blind sampler must not report an idle fleet
The burst re-run printed "architect peaked at 1 of 6" and "nothing ever queued" for a run I could watch sitting at architect=6 tank=6 morpheus=2 with 2 phases queued. The fleet was right; the sampler was blind. Three separate ssh+psql calls per 15s tick, each with stderr to /dev/null, and under the load of 16 concurrent missions most came back empty. Empty was then read as "nothing running" — absence encoded as a legitimate value, which is the exact seam the header of this file was written about, reproduced in a scenario added to catch it. One query per tick now, returning done/blocked/per-node in a single row, and unreadable samples are COUNTED rather than silently treated as zeroes. Fewer than ten usable samples is NORUN: a sampler that barely looked must not be able to describe itself as a fleet that was idle. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
e5f097c291 |
fix(harness): the capacity burst verifies its own launches
The re-run reported FAIL-NORUN "the burst did not finish in 1800s". The fleet was fine — 3 of the 16 missions were still in `draft`. Each PATCH-to-running is an ssh plus a `docker run curl`, and 16 at once does not reliably land; the response was going to /dev/null, so a launch that never happened spent the full timeout looking like a platform stall. That is precisely the swallowed-error shape this file was written to catch, committed inside the file itself. Launches are now verified against the mission rows, retried once for the stragglers, and reported as "the burst never happened" rather than as a timeout — a scenario that did not run must not be able to describe itself as a slow one. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
2056bb1d9e |
test(fleet): prove the queue and the spread under a real burst
Phase 1 shipped placement-at-phase-launch and a queue made of `start_pending_phases` leaving a phase `pending`, both deployed unproven under load — the exact condition this project keeps getting burned by: the code is right, the system is wrong, and nothing errors. `capacity` launches `slots + 2` microVM missions simultaneously and asserts two things. That no node ever exceeds the slots `vm_placement` gave it — overcommit does not fail loudly, it swaps, and every mission on that node gets slow rather than dead. And that the excess QUEUES: a burst that drops the extras and one that wedges them both look identical to any check that only reads the end state. `capacity_blocked_since` is cleared the instant a phase is placed, so the evidence only exists mid-flight; the scenario samples while it runs. Capacity comes from `/api/fleet/capacity`, never recomputed here — a bash copy of the slot arithmetic would drift from the scheduler and then agree with itself. A burst that does not exceed capacity is reported NORUN, per rule 3. `drain-midmission` drains the node phase 0 ran on, before phase 1 is placed, and asserts phase 1 lands elsewhere AND still reads phase 0's file. That is the test of the affinity decision: mission state lives on the gateway, so re-placement is free — if it were not, this would either strand the mission or silently lose the earlier work, and "silently lose" is what a status-only check calls success. The node is restored before any assertion runs, so a failure cannot leave the fleet permanently one node smaller. Smoke-checked at CAPACITY_BURST=2: sampling, spread and completion all report, and the queue check correctly returned NORUN rather than a green tick. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
deed591da6 |
fix(roster): a rate-limited subscription is a 503 with a reason, not a 500
The retry landed and still failed: all four attempts returned 429. A bare
16-token probe with the same token, straight from gw-04, also returned 429
with `x-should-retry: true` — the Claude Code subscription itself is limited
right now, and no amount of backoff inside one HTTP request will outlast it.
So stop pretending it is a server bug. New `ApiError::Unavailable` → 503,
carrying the one sentence the operator can act on ("clears on its own; try
again shortly"), instead of an opaque `internal error` that sends them into
the logs. The harness now prints the response body rather than the generic
"the planner produced no usable proposal", which is what hid both walls —
first the credit balance, now this.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar Sobh
|
099a716bfd |
fix(egress): a backend is defined in two maps, and the canary only had one
First canary run failed: phase failed, nothing delivered, and the streamed log said exactly why — "Failed to authenticate. API Error: 403 api.anthropic.com is not on the egress allow-list". Not a 2.1.226 regression. `canary-claude` was added to the server's credential map and not to the node's `provider_hosts`, so the VM booted with a valid subscription token and a door that only opened onto the forge. The fail-closed branch was working correctly: a backend nobody taught that function about reaches no model API, deliberately, so it cannot silently borrow another provider's door. Both maps now name it, each pointing at the other, with a test asserting the canary reaches the same provider as `claude` AND that unknown backends still resolve to nothing. Worth noting what made this a five-second diagnosis instead of an afternoon: the live log streaming built earlier today. The failure was a 403 inside a microVM that no longer exists, and its reason was sitting in the run's checkpoint. |
||
|
Omar Sobh
|
a8b8efba6a |
fix(delivery): the on_green_tests gate ran the suite in the live checkout
Fourth instance of the same defect, and the last of the three commands that run as root against a mission tree. `verify_tests` execs the project's test command with `workdir = repo` — the live checkout — inside a container running as ROOT. `cargo test` writes `target/`, so the checkout ends up owned by two uids and the next phase's cargo hits permission-denied. The harness reported `uids=0,65532` the first time this gate ever ran end to end. It survived because it had never run. Every one of the ten harness fixtures used `commit_policy: "always"`; `on_green_tests` and `on_reviewer_approval` were parsed, implemented, and never exercised — and `Gate`'s own doc already records that three recipes carried this policy while it "did precisely nothing" for want of a reader. A policy that is never exercised is indistinguishable from one that is ignored. Consolidated rather than fixed a third time. `root_copy` now owns the pattern — copy through `mission_fs::pack_dir` into a SIBLING of the mission dir, run there, and purge FROM INSIDE THE CONTAINER, because the copy's `target/` is root-owned and the server (uid 65532) cannot delete it. `benchmark_runner` moved onto it; `evaluator_tools::Sandbox` keeps its own copy logic for now (it carries an allow-list and a judge-facing API, so folding it in is a larger change than this moment warrants — noted, not done). The gate fails CLOSED if the copy cannot be made: an unverifiable suite must not license a push. Also adds the `refactor` scenario, which is what found this. I had written it off as "structurally identical to four existing scenarios" — wrong: it is the only recipe carrying `on_green_tests`, and that made it the only one testing this code path at all. 245 lib tests, 20 test binaries. |
||
|
Omar Sobh
|
2a9a62c784 |
test(harness): cover security_hardening — 4 of 5 recipes now run end to end
The third recipe whose defining phase is not `coding`, and so the third that nothing could fail before `PRODUCING_KINDS` widened: a `security_scan` phase that ran no scanner and wrote nothing reported success. One phase, not the recipe's full scan->research->code chain — what is under test is the phase KIND, and the other two kinds are already covered. Two assertions, because the first alone is weak. "Delivered a file" is satisfied by an agent that writes "I scanned it, all clear" and runs nothing — the letter-not-purpose shape this codebase keeps paying for. So the delivered patch must also carry the scanner's OWN output. Verified against the real run: the agent produced gitleaks' banner, INF/ERR lines, byte counts and exit code, not a claim about them. Only `refactor` is now uncovered, and deliberately: its single phase is `coding`, structurally identical to chain/multirole/microvm/noop. It would add runtime and no new signal. security 4/4 against the live fleet. |
||
|
Omar Sobh
|
6dd7937ece |
test(harness): cover the two recipes that had none — research_only and benchmark
The portal offers five workflow recipes. Every one of the harness's seven
fixtures was `research_and_code`, so four recipes had never run end to end —
and that is not a theoretical gap. `research_only` DESTROYED its output for as
long as it existed: `requires_repo = false`, so the capture query's
`AND m.repo_id IS NOT NULL` skipped it, the container was reaped unread, and
eight ClawHDF5 research documents were lost while the mission reported
`completed`. Nothing in 550+ tests could see it, because nothing ran the recipe.
`research-only` asserts the whole chain the loss ran through, not just the
happy end of it:
- the phase completes
- document artifacts exist AT ALL (the missing thing)
- the agent's seven identity files (SOUL.md, MEMORY.md, …) are NOT published
— the first live capture published all seven, because `.git/info/exclude`
cannot protect a mission with no `.git`
- the captured text reads back through the content endpoint, since an
artifact row pointing at nothing is a 404 with no explanation
`benchmark` covers the other half: a benchmark mission is ONE benchmark phase,
and while `empty_delivery_is_a_failure` tested `kind == "coding"` that phase was
exempt — nothing in the platform could fail it. The scenario asserts it both
completes AND delivers files.
Also: `run_scenario` takes an optional `no-checkout`. The single-writer uid probe
is a property OF A CHECKOUT, and a repo-less mission has none by design, so
probing reports a platform fault that is really a category error. It is declared
per scenario rather than inferred from a missing directory — that inference would
silently excuse a repo-BACKED mission whose checkout was reaped early, which is
the exact condition the probe exists to catch.
research-only 4/4, benchmark 3/3 against the live fleet.
|
||
|
Omar Sobh
|
bcf4866abc |
test(harness): a gate that gives up, proven against a real VM
The unit tests prove the plumbing GIVEN `released_at_cap: Some(true)`. They cannot prove the guest writes the marker, that the probe reads it back across the vsock, or that the phase lands `failed` for the right reason — and every one of those is where this class of bug has actually lived. The check is `exit 1`: impossible by construction, so the run exercises the release path rather than hoping to catch it. `blocks` reaching the cap is deliberately NOT the assertion. A healthy agent blocked three times and succeeding on the fourth reports the same 3. The phase STATUS is the assertion; the block count and the failure reason are corroborating checks, so a phase that failed for some unrelated reason cannot pass this. Measured on gw-04 against |
||
|
Omar Sobh
|
cd4d76a8c3 |
test(harness): pick the done_when wording by measuring the judge, not arguing with it
The microvm scenario's judge assertion failed four runs straight. I blamed the
wording twice and rewrote it twice; the second rewrite made it worse. That was
guessing.
With scripts/judge-eval.sh in place the question is cheap to settle. Three
candidate conditions, three draws each, same evidence and same system prompt:
"its second line is …" MET UNMET MET flaky
"records the kernel version …" MET MET UNMET flaky
"contains both … and …" MET MET MET stable
So it was never noise in general — it is a reproducible weakness with
POSITIONAL and EXCLUSIVE phrasings. "its second line is X and nothing else"
invites this judge to invent requirements about the other lines, which is
exactly the reason it kept citing ("the first line contains 'test result: ok'").
Both fixtures now state what the file CONTAINS. The composed one was checked in
both directions — 3/3 MET on good evidence, 3/3 UNMET when the versions are
missing — because a wording that always answers MET would look stable and prove
nothing.
The eval keeps `kernel-ok` failing on purpose; it is the case production hit,
and tuning it green would turn a measurement into a decoration.
Harness: 24/24, including the assertion that had failed four times.
|
||
|
Omar SobhandClaude Opus 5
|
5c066afa7b |
test: stop leaking a container per run, and add the project's first eval
TWO FINDINGS, one from cleaning up and one from refusing to keep guessing. THE LEAK. `./scripts/test.sh` left three containers running every time — 289 had accumulated. The cause was a comment that lied: `warm_pool.rs` said "Shutdown destroys assigned AND pooled sandboxes", while `SandboxManager::shutdown` drains the POOL only. Its own doc says why — assigned sandboxes persist deliberately so a redeploy can reuse them, and production reaps the strays with `reconcile_orphans` at boot. A test has no next boot, so each one that assigned a sandbox simply left it running. The three tests now call the `release_agent` that already existed, and the comment says what the code does. Verified: 0 leaked, where the same run leaked 3 before. THE EVAL. The independent judge failed the same correct phase FOUR times, each time citing a different invented requirement. I blamed the condition's wording twice and rewrote it twice — the second rewrite made it worse, by naming a command a tool-using judge then ran in its own container. Then a control showed the same model answering MET to the same question asked directly, and a third wording test showed a STRICTER phrasing scoring UNMET. Prose wording was not the variable. Continuing to iterate would have been fitting the fixture to noise. `scripts/judge-eval.sh` measures the thing instead: five cases drawn from real incidents, each with an answer a careful human would agree with. This project has 557 tests and had zero evals, which is backwards — a test pins OUR code, an eval pins the MODEL, and the model changes without us touching anything. The result is why it was worth building: glm-4.7 4/5 — wrong on kernel-ok: says UNMET when MET kimi-for-coding 4/5 — wrong on goodhart: says MET when UNMET Identical scores, opposite failure modes. GLM fails good work; KIMI passes work where 14 assertions were deleted and the failing module removed to make a suite "pass" — the exact incident the verifying judge was built after. Swapping the validator to Kimi because it passes our failing case would have installed a rubber stamp. Keep GLM: a judge that is too strict costs a re-run, a judge that is too lenient costs the guarantee. The eval also caught a bug in itself before I trusted it: Kimi answers with a `thinking` block first, and a 160-token budget was consumed entirely by it, which the harness scored as NO-ANSWER. An eval that misreads a model is worse than no eval, so it now reads thinking blocks as a fallback and has room to answer. 557 tests pass, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
72f8bdc87c |
test(harness): a done_when naming a COMMAND invites the judge to run it
My previous attempt at this made it worse, which is the useful part. The condition said "a Linux kernel release string" and the judge rejected `6.1.128` as "not a Linux kernel release string such as 'Linux 6.1.128'". I rewrote it as "the exact output of `uname -r`" — and the next verdict was that line 2 should be `27.0.0`. The judge has a sandbox and allow-listed commands, so naming a command told it to RUN that command, in ITS OWN container, and compare the file against the answer it got there. The file records a microVM's kernel; the judge was comparing it against the machine the judge runs on. Those are different machines by design — that is the entire point of the assertion. So a `done_when` for a tool-using judge must describe the VALUE's shape, never a command that produces it: "a bare kernel version of the form MAJOR.MINOR.PATCH (for example 6.1.128) and nothing else", plus an explicit instruction not to run uname and not to compare against the local machine, because the file records a different one. The general rule, worth carrying into how `done_when` is written anywhere: a condition phrased as "the output of X" is ambiguous about WHERE X runs, and a judge with tools resolves that ambiguity by running X where it stands. Conditions about a remote or past environment must be stated as properties of the recorded value. The scenario's real proof that the agent ran in a guest is unchanged: a separate comparison of that line against the actual gateway and node kernels, which has passed on every run including the two where the judge disagreed. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
1b556c5849 |
test(harness): say what the condition means, after the judge read it strictly
The restored GLM judge failed a phase that had done the work: MICROVM.md existed with two lines and the second was `6.1.128`, and the verdict was "a kernel version number, not a Linux kernel release string such as 'Linux 6.1.128'". The judge is wrong on the fact — `6.1.128` is exactly what `uname -r` prints, and "release" is the term for it — but the CONDITION was ambiguous, and it is our fixture. "A Linux kernel release string" can be read as either `uname -r` output or `Linux x.y.z`, and a stricter reader is entitled to the second. Both scenarios now say what they mean: the exact output of `uname -r`, a bare version, no prefix. This is not weakening the assertion. The scenario's own kernel check — the one that proves the agent ran in a guest rather than on a host — is a separate, unchanged comparison against the real host kernels, and it PASSED on the same run. What changed is only that the mission-level `done_when` now describes an observable fact precisely, which is what this codebase's own plan-authoring prompt tells models to do. Worth recording rather than papering over: an over-strict independent judge is a much safer failure mode than an over-lenient one, and this is evidence the judge READS the tree instead of rubber-stamping it — the Goodhart incident that motivated cross-provider validation was the opposite failure. But it does mean a vague `done_when` can now cost a phase, which raises the value of `done_when_check` (a shell command, judged by exit status) for anything mechanical. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
521da9feb9 |
fix(deploy): the verify step is the authority, not the recreate
`scripts/deploy.sh` reported failure twice this afternoon for deploys that had succeeded. Both times the 60-second rolling timer rolled the stack onto the same `:latest` first, and the script's own `docker-compose up` then hit a container-name conflict — "already in use" once, "Renaming a container with the same name" the other — for a container the timer had already recreated correctly. A deploy signal an operator has to second-guess is precisely what this script exists to prevent. Its original reason for being was a green edge on a stale image; crying wolf trains people to ignore the alarm, which gets you the same outcome by a different route. The recreate is now best-effort and says so when it fails, and the VERIFY step decides — it compares the RUNNING image id against the resolved `:latest`, which is the only question that matters and is unaffected by which process did the roll. A genuinely failed deploy still fails there, because that check never depended on the recreate succeeding. Both false alarms were settled by hand with the binary grep (`docker exec … grep -a -c "<string only in the new code>"`), which remains the strongest check when the image id is in doubt. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
742724e53c |
feat(fleet): Kimi as a microVM backend — the URL settled by measurement
The base URL took three measurements to find, and the first two were wrong in instructive ways. `api.moonshot.ai/anthropic/v1/messages` EXISTS and speaks the protocol — it answers with Moonshot's own structured error rather than a 404. It also rejects an `sk-kimi-` key, because it belongs to the platform.moonshot.ai account namespace. Two endpoints that both "work" for different accounts is precisely the shape that makes a guessed URL look like a broken key, and it is why this was refused rather than guessed for as long as it was. The Kimi CODE service is the one an `sk-kimi-` key belongs to: `POST https://api.kimi.com/coding/v1/messages` returns a real Anthropic Messages body — `msg_` id, `content` blocks, a `thinking` block with a signature. So `ANTHROPIC_BASE_URL=https://api.kimi.com/coding`, WITHOUT the `/v1`: Claude Code appends `/v1/messages` itself, and `/v1/v1/messages` would 404 in a way that reads as a broken image rather than a bad URL. Two more measured, each otherwise a silent failure at the first turn: `Authorization: Bearer` is accepted (so ANTHROPIC_AUTH_TOKEN is the right injection channel), and a `claude-*` model id is ACCEPTED AND ANSWERED — Kimi maps it onto `kimi-for-coding` exactly as z.ai does, so no ANTHROPIC_MODEL override is needed. Claude Code rather than Moonshot's own `kimi` CLI, deliberately. The mission harness is Claude-Code-shaped throughout: `--agents` JSON roles, the verifier's tool allowlist, the `Stop` hook behind the completion gate, the per-subagent transcripts counted as delegation evidence. `kimi` has none of those flags — its equivalents are TOML files and markdown agent dirs — so using it would mean a second executor with its own untested failure modes. TWO STALE MAPS, caught by the rootfs harness refusing to bless the image: both `fc-build-rootfs.sh` and the node's `required_cli` expected backend `kimi` to contain Moonshot's `kimi` binary. That assumption predates the measurement, and it failed a rootfs that was correct. Both now say `claude` for glm and kimi alike — the binary is the same in all three images; only the endpoint differs. Egress for `kimi` is `api.kimi.com` alone: not moonshot.ai (wrong namespace), not z.ai, not Anthropic. Asserted both ways, like the other two. The image and rootfs are built on tank and the rootfs passes all four checks (boots, git, writable /mission, `claude --version`). 534 tests, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
f7f3dfe495 |
feat(fleet): GLM as a real microVM backend, and per-role models for claws
Three threads, all of which end at the same place: a mission whose verifier does
not share a model with the coder it reviews.
**GLM has a credential contract now.** `microvm_credential_for` returned one env
var name, which quietly assumed every provider reads its secret from the same
place Anthropic does. It returns a `Credential { source, target }` instead —
z.ai's key lives in the server's `ZAI_API_KEY` and Claude Code reads it as
`ANTHROPIC_AUTH_TOKEN`, and collapsing those two names is what forces a guess at
the other end. A wrong guess here sends one provider's credential to another
provider's endpoint.
`images/agent-glm` is the same CLI at the same pinned version as `agent-claude`
with `ANTHROPIC_BASE_URL` baked in. The split is deliberate: the ENDPOINT is a
property of the image, the CREDENTIAL is a property of the turn. That makes the
dangerous mix-up unrepresentable — a GLM VM cannot be handed an Anthropic
subscription token, and a claude VM cannot be pointed at z.ai. Asserted both
ways, because "the GLM VM must not carry CLAUDE_CODE_OAUTH_TOKEN" is the
property that costs a credential if it ever stops holding.
Kimi stays refused. `KIMI_API_KEY` is set and Moonshot serves an
Anthropic-compatible API, but I have not verified its base URL against the
running service, and this function is precisely where guessing a URL is
expensive. It becomes an arm the day someone measures it.
`api.z.ai` joins the node's default egress allow-list. A default that cannot
run the images we ship is a trap rather than a policy — the alternative is an
operator discovering it as a hung agent with no model access.
**Per-role models for claws** (migration 0071). `template_roles` had no model
column, so `mint_team_from_template` bound every role of every mission team to
one literal — a template whose whole point is an independent reviewer minted a
reviewer sharing a model with the coder. A role may now name its own; roles that
say nothing still take the mint's default, so every template written before this
behaves exactly as it did. The literal is now that default rather than a
hardcode.
**A harness scenario for the roster flow.** `verify-mission-delivery.sh roster`
runs the whole Slice 5 loop — planner proposes, human approves, mission runs —
and asserts the roster LANDED on the mission row rather than trusting the API's
answer. That distinction is not theoretical: the first live approval returned an
error while leaving the proposal marked approved.
Built and proven on tank ahead of the deploy: `clawmates/agent-glm:dev` reports
`2.1.223` and `BASE=https://api.z.ai/api/anthropic`, and
`fc-build-rootfs.sh … glm 8G` boots a VM from it that has git, can write
/mission, and answers `claude --version`.
533 tests pass, clippy clean. Migration 0071.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
abb97e6f03 |
test(harness): a composed scenario, and the stop gate asserted in a real VM
`verify-mission-delivery.sh composed` runs a `team_engine=composed` mission and checks the one property that cannot be checked any other way: a VM is inject → run → collect → destroy, so unless the tree is carried node to node, node 2 boots from the original checkout, sees nothing of node 1's work, and still reports success. The task makes each node append ONE line to STAGES.md, so the delivered file IS the evidence — a run that lost the handoff delivers one line, and no amount of agent confidence can fabricate the missing ones. It also asserts the run's tier is `microvm_graph`. A composed mission that quietly fell back to the solo path would deliver a one-line file and look exactly like a graph that ran one node. `assert_stop_gate` reads the count `phase_runner` reports and distinguishes three outcomes that matter: a number (installed, fired that often), `0` (installed, never needed), and `-` (could NOT be installed — usually a CLI in the image with no `--settings`). Wired into the microvm scenario rather than its own, because it applies to every coding phase on that path. Both ran against the deployed stack: composed — 4/4. STAGES.md carried 5 stage lines through 5 separate VMs (planner → coder → tester → reviewer → committer), each stamped with the guest kernel 6.1.128 rather than the gateway's 6.8.0 or the node's 7.0.0. The run checkpointed 5 steps on the worker, and `updated_at` stayed ~2s old mid-turn, which is the keepalive doing its job — without it `requeue_stale` flips a live run at 180 seconds. microvm — 6/6, including the gate installed in a real VM (`blocks: 0`), one subagent, the GLM judge, and the unavailable-backend negative control. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar Sobh
|
b17e18aa67 |
fix(harness): the verdict check matched psql's display form, not the query's
`select met || ' ' || independent` casts the booleans to `true`/`false`, but the
pattern matched `t`/`f` — psql's *column display* form. So the check reported "no
verdict recorded for the phase" while the row sat in the table saying met=true,
independent=true, glm-4.7.
A check that fails for a reason unrelated to what it checks is worse than no check:
it trains you to ignore the output. The booleans are cast explicitly now so the
shape cannot drift again, and the failure message prints what it actually got.
`verify-mission-delivery.sh microvm` now passes 5/5 against production:
- the agent ran under guest kernel 6.1.128, not the gateway's 6.8.0-124 or the
node's 7.0.0-28 — the one assertion that cannot pass by accident
- the lead delegated to 1 subagent
- the condition was met and judged INDEPENDENTLY by glm-4.7
- the checkout has exactly one writer (uid 65532)
- negative control: a backend no node can run is refused at launch
|
||
|
Omar Sobh
|
9aed20b6d0 |
fix(missions): capture a failed phase's work; harness gains a microvm scenario (#51)
A REGRESSION I INTRODUCED ONE COMMIT AGO. `capture_finished_coding_phases`
selects on `mp.status = 'completed'`, so the moment an unmet phase correctly began
reporting `failed`, its diff stopped being captured, committed or pushed — the work
was silently discarded. Found by the new harness scenario, whose phase legitimately
missed its condition and then had no artifact at all.
What was produced, and whether the goal was met, are different facts. The artifact
records the first; `mp.status` records the second. Capture now covers terminal
phases (`completed`, `failed`), so a phase that did real work and missed its goal
still delivers a reviewable diff — which is exactly what the next pass needs.
`scripts/verify-mission-delivery.sh microvm` — the regression net this session was
missing. Everything the microVM track proved by hand was guarded by nothing:
- THE KERNEL LINE is the assertion that cannot pass by accident. Every other
check would also pass if the phase had quietly run in a container on the
gateway; only the kernel says WHERE it ran. Compared against the real gateway
and node kernels read at start-up rather than pinned to a version, so
upgrading vmlinux does not manufacture a failure.
- subagent count > 0, from the server's own count of Claude Code's per-subagent
transcripts. Before `Agent` was in the allowlist this was structurally
impossible and nothing said so. A probe that could not run reports "?" and
FAILS the check rather than reading as zero.
- the verdict's judge and whether it was independent.
- negative control, observed passing: a mission whose backend no node can run is
refused at launch and stays draft. Without it the positive scenario would pass
just as well against a scheduler that ignored `backend` entirely — which is
what it did until the first real microvm mission landed on a node with no such
rootfs.
Also fixed in the harness: `api` now sends the JSON body on STDIN (`curl -d @-`)
instead of interpolating it into a single-quoted argument inside a double-quoted
ssh command. A task description containing "the crate's test suite" ended the
quoting and killed the remote shell; two attempts to escape it were themselves
wrong, because the backslashes must survive bash AND sed AND sh. Removing the
interpolation removes the class, and the next author does not need to know that
apostrophes were forbidden.
475 tests pass, clippy clean.
|
||
|
Omar SobhandClaude Opus 5
|
0c291ed1bb |
feat(fleet): B4.4a — a real agent-claude microVM image, and a check that it has an agent in it
The only rootfs on this track came from clawmates/agent-terminal:dev. Mounted,
it held git and nothing else: no claude, no node, no cargo. A VM booted from it
looks perfect and cannot run a mission, so B4.5 could have been written and
never verified.
images/agent-toolchain — the shared mission toolchain (node 22, git, rust +
cargo-audit, gitleaks/trivy/semgrep, tea/gitea-mcp), lifted from the proven
deploy/clawmates-runtime image minus the zeroclaw daemon: a microVM mission runs
the direct-session model, so there is no daemon to host. A base image rather
than three self-contained Dockerfiles because this layer is ~3 GB and the real
risk is scanner and toolchain versions drifting between per-CLI images — the
evaluator runs the project's own suite to check a claim, so `cargo` present in
one image and absent in another makes the same mission pass or fail by backend
with nothing saying why.
images/agent-claude — plan A6, first of three: the pinned CLI and its env
contract only, so bumping Claude Code does not rebuild the toolchain and cannot
disturb agent-kimi / agent-glm. HOME=/root with an empty .claude for B4.4 to
inject into; no ANTHROPIC_API_KEY, since it silently overrides the subscription
OAuth we already pay for.
Both the builder and the node selftest now ASK the guest for the CLI the image
is named for, instead of trusting the name. `required_cli` maps claude/kimi/glm
to a probe; an unrecognised backend reports unchecked and prints SKIP rather
than passing quietly.
Verified on tank:
- rootfs-claude.ext4 boots; git, node, cargo, a real git commit all work
- `claude --version` → 2.1.220 over vsock, in both the builder and
`--vm-selftest` (11/11, create 1498 ms)
- negative control: the same builder run against agent-terminal with
FC_CLI forced reports `cli rc=127 claude: not found` and exits 1, so the
green result above is a measurement and not a default
- `claude -p hello` fails with "Not logged in · Please run /login" — the CLI
runs headless in the VM, and B4.4 only has to supply the credential
- no leaked firecracker processes or vm dirs afterwards
437 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
08847e6a63 |
feat(fleet): B4.2 — static Rust guest agent replaces the python one
The python guest agent only ever worked because Firecracker's CI Ubuntu image happens to ship python3. NONE of our images do — agent-base has neither python nor git, agent-terminal has git but no python — so it could never have run in a real mission rootfs. An agent that dictates what must be installed in the image has the dependency backwards. crates/bins/fcagent is a 905K static x86_64-unknown-linux-musl binary that needs nothing from the rootfs it is dropped into. The wire is unchanged on purpose — 4-byte BE length + JSON, ops ping/exec/put/get — so microvm.rs and microvm_client.rs needed no edit at all. std has no AF_VSOCK and the workspace denies `unsafe`, so it uses the `vsock` crate. `process_group(0)` gives each command its own group without unsafe, so a command that spawns background children can be killed wholesale rather than outliving the run. A unit test caught a bug that would have broken EVERY exec: sourcing the image-env file with `. env.sh 2>/dev/null; cmd` returns rc=1 WITHOUT running cmd, because `.` on a missing file makes a non-interactive POSIX shell exit immediately. On any rootfs lacking that file every command would have failed while looking like an ordinary non-zero exit. Guarded with `if [ -f ]` now. Other places a failure must not borrow an outcome's representation: a killed command reports ok:false with no rc (not rc=124, which would read as a build failure); `get` on a missing path is an error, not an empty archive; a signalled process reports 128+signal rather than success. Verified on tank: --vm-selftest still 8/8 with the agent swapped (create 949ms, wire identical), fc-node-setup 8/8, and — the point of the change — a rootfs built from clawmates/agent-terminal:dev, which has NO python3, boots and reports `git version 2.39.5` from inside the VM. Also fixes a shell bug in fc-build-rootfs.sh: $HOME in a double-quoted default expanded on this Mac, so it looked for the node's binary under /Users/quantum on a Linux host. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
dec59764b1 |
feat(fleet): B4.1 — build a Firecracker rootfs from a Docker image
Until now microVMs booted Firecracker's CI Ubuntu image with a python
guest agent bolted on: no git, no toolchain, no CLI. Fine for proving
vsock, useless for running a mission.
Builds FROM a Docker image rather than debootstrapping, because the
per-CLI images (agent-claude / agent-kimi / agent-glm, per A6) are
already Dockerfiles with a tested env contract. Rebuilding that as a VM
image by hand would mean maintaining the same facts twice and finding the
drift in production.
Two things the obvious version gets wrong and this does not:
- `docker export` gives the filesystem with NONE of the image metadata:
no ENV, no ENTRYPOINT, no WORKDIR. A CLI relying on ENV PATH or HOME
would silently behave differently in the VM. The env is extracted
separately and written to /etc/profile.d.
- the ext4 is filled through a mount, not `mkfs -d`, which cannot
handle the device nodes and hard links a container image may contain
and fails late and cryptically when it hits one.
The guest agent is copied from the golden rootfs rather than re-emitted,
so there is ONE copy of the protocol on the node instead of two that can
drift.
It boots what it builds and asks the image for what a mission needs —
git, the profile env, a writable /mission — rather than assuming. An
image that builds and cannot boot is worse than no image, because it
looks finished.
FINDING, and it blocks B4.2: NONE of our images ship python3, so the
python guest agent cannot run in any of them. agent-terminal has git but
no python; agent-base has neither. The guest agent must not dictate the
image's contents — it needs to be a static binary. This script correctly
refuses to build an image whose agent cannot run, so the failure is
visible rather than a VM that boots into nothing.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
2d04c5e257 |
feat(fleet): B2 — vm_* node ops for Firecracker microVMs
create / inject / exec / collect / destroy / list, riding the node's
existing frame dispatch ({t, id, …} -> {t:"result", id, ok, output}), so
no protocol change was needed. Control is length-prefixed JSON over
vsock; the serial console stays a log, because feeding a guest over stdin
races its startup and arrives half-consumed.
DEVIATION FROM THE PLAN, deliberately: this does NOT implement
cm_sandbox::SandboxDriver. That trait is container-shaped —
attach_pty/resize_pty/argv exec — while missions need
create -> inject -> run -> collect -> destroy. Conforming would mean
building PTY-over-vsock and window-resize semantics that no mission path
calls, purely to satisfy a signature. We give up automatic RemoteDriver
marshalling; orphan reaping is a label/id sweep either way.
Three traps from the B0 spike are handled in code rather than remembered:
- Firecracker does NOT unlink its vsock UDS on exit, so destroy unlinks
it explicitly, and the selftest ASSERTS it is gone. Assuming the VM
tidies up after itself is how the mission checkout accumulated four
uid bugs.
- firecracker is spawned via setsid and killed as a process GROUP, so a
background child cannot outlive the VM holding its workdir open.
- create does not return until the guest agent has answered a ping. A
VM that booted but serves nothing is worse than one that failed, so a
half-created VM is destroyed rather than left registered.
A vm id becomes a path component, so ids are restricted to [A-Za-z0-9_-]
and REJECTED rather than sanitised — a caller that sent `../../etc`
wanted something we should not guess at.
Verified on tank through the real Rust path, as the daemon user, with no
sudo: `clawmates-node --vm-selftest` -> 8/8, create in 986ms, and the
host left with zero firecracker processes and zero VM directories. The
selftest asserts every step, including that a destroyed VM can no longer
be exec'd; a test that only reports the steps it completed cannot
distinguish "passed" from "stopped early".
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
67c56ce19b |
fix(fleet): /dev/kvm present is not /dev/kvm usable
The capability probe reported `kvm: false` on tank and morpheus while the device sat right there: /dev/kvm is `crw-rw---- root:kvm` and the kvm group was EMPTY, so the daemon — an ordinary user — could not open it. The B0 spike missed this entirely because it ran everything under sudo. This is exactly why the probe opens the device rather than stat-ing it; a stat-based check would have reported both nodes capable and every microvm mission would have failed at launch instead of at placement. fc-node-setup.sh now fixes the group itself, or says precisely what to run when it cannot. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
e65be19a45 |
feat(fleet): Firecracker node setup, proven by booting a microVM
Phase B step 0. Before writing any driver, establish that Firecracker
works on this hardware — the plan called it greenfield, and an
orchestrator built against an unproven runtime is a lot of code betting
on an assumption.
It works, and comfortably: a microVM boots, runs our init, writes a
file and shuts down in ~650-910ms wall clock, with the kernel reaching
our init at 234ms. Host->guest RPC over vsock (AF_VSOCK port 9001, no
network stack) round-trips in 27ms.
The script installs and then PROVES, because installing is not working.
It reports success only after a VM has actually booted and run our code.
Four findings from the spike that the driver must account for:
- Firecracker does NOT unlink its vsock UDS on exit, and leaves it
owned by whoever ran the VM. A driver running as anyone else cannot
clean it up — the same uid trap that cost this codebase four bugs on
the mission checkout. The driver owns the socket path lifecycle.
- tank's FORWARD policy is DROP (Tailscale/Docker), confirming the
article's warning: VM networking rules must be inserted at position
1, not appended, or return traffic dies silently.
- Feeding commands to the guest over the serial console races the
shell's startup and arrives half-consumed (`# ho FC-GUEST-ALIVE`).
The guest runs an init script; stdin is not a control channel.
- `sha256sum -c` compares by filename, so a download saved under any
other name fails for a reason unrelated to integrity. A check that
fails for the wrong reason teaches you to ignore it — compare the
hashes directly.
tank and morpheus are ready. architect requires interactive sudo, so it
is deliberately not provisioned rather than worked around.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
da3731d753 |
fix(missions): a coding phase that delivers nothing is a failure
The last open item in the silent-success class: a coding phase that changed no files reported `completed` — the same status a phase gets for delivering tested, reviewed, pushed work. Mission `019fcf62` completed that way with its agents silently unpinned from the repo, and nothing in the platform disagreed; it was found by a script diffing the forge. The verdict is applied at capture rather than at completion, because capture selects on `status = 'completed'` — the platform does not know whether a phase produced anything until after it has already finished. Three conditions must hold before failing a phase, because a false positive here fails honest work: the phase is a coding phase (research phases legitimately write nothing to the tree), the diff was actually computed (an uncomputable diff also reports zero files — blaming the agent for a platform fault is the same defect wearing different clothes), and `allow_empty` is not set. Only an explicit `true` opts out, so a typo leaves the check armed. Registered in phase_config with its reader named, per the seam-2 rule. Also closes an ordering hazard this exposed: capture is batched and runs after a phase completes, so a backlogged mission could close as 'completed' and only then have capture discover an empty phase — leaving a 'completed' mission holding a 'failed' phase, unfixable because the mission-close CASE only touches 'running' rows. A repo-bearing mission now waits for its work to be captured before closing. Adds a `noop` scenario to the harness: a phase told to change nothing, which PASSES only when the phase comes back `failed`. Same discipline as the uid self-test — a check that has never been seen to fire has not been shown to work. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
1253595ba7 |
fix(missions): stop three launch failures from passing as success
A verification run against the deployed stack found a chain mission whose phase 0 reported `completed` with zero files, no commit error and no push error — indistinguishable from a phase that correctly had nothing to do. Three separate defects had to line up, each of them the same shape: a failure sharing its representation with a legitimate negative result. 1. `pin_agent_workspaces` embedded the whole config in one `sh -c` argv. That works until the file grows — config gains a block per provisioned claw — then fails with `argument list too long`. Now written through the tar upload API, which has no argv limit, so the failure mode is gone rather than merely further away. 2. A failed pin was logged "(continuing)". Without the pin, agents write to their sandboxes and the committer finds nothing in /mission/repo — the mission cannot deliver, so the launch now fails where someone is still looking. The restart that applies the pin is fatal for the same reason. 3. `capture_phase_diff_at` swallowed `git diff` failures with `unwrap_or_default`, so an unreadable base landed `empty: true, files_changed: 0` — byte-identical to an honest no-op. The error is now recorded as `diff_error`, and an empty patch that came from a failed diff is no longer trusted to mean an unchanged tree. Adds scripts/verify-mission-delivery.sh, which found #1 and #2 on its first real run. Its probes are fail-closed: no placeholder values, a self-test that proves the uid probe can detect the split it looks for, and FAIL-NORUN for a scenario that never executed. Its own first version had this bug too — a `die` inside `$(...)` exited the subshell, so a run that could not authenticate printed "all checks passed" and exited 0. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
2380c2cb0b |
fix(deploy): identify agent images by build stamp, not image ID
The post-transfer verification added in
|