1a44405308cbe61c543157ec83a0c0a1b2f463d0
66
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar SobhandClaude Opus 5
|
9fc904a056 |
feat(microvm): the tool gate's denials and inert marker reach the mission record
vm_tool_gate writes denied.jsonl for every call it refuses and an `inert` marker each time it cannot parse its input and lets the call through. The guest has written both since the gate existed; nothing read them out of a VM. A denial, or a gate that had quietly stopped checking, left no trace — the same shape the container tier closed with drain_inert on 09-14. The executor probes both files (one exec, while /root still exists) into VmOutcome.tool_gate; launch_microvm_phase records them on the mission as the container tier's `gate.inert` (with the count) and `gate.denied` (one event per refused call, the gate's own JSON as the detail). Absent gate is None, not zero — "no gate" and "a gate that refused nothing" are different facts. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz |
||
|
Omar SobhandClaude Opus 5
|
794f2124bc |
feat(microvm): Claude Code 2.1.276 rootfs pins, and a VM run that says what it ran
Every rootfs on the fleet had sat on Claude Code 2.1.223–2.1.226 since August while the container tier moved to 2.1.276, and nothing recorded either. GLM and Kimi exist only as microVM backends, so "have we upgraded GLM and Kimi" is this change and the rebuild it drives. Pins. All four agent-* images pin 2.1.276 — as separate ARGs, since Docker has no include and each file has to stay reproducible alone — and scripts/fc-build-rootfs.sh refuses to build if they disagree, naming the odd one out. They had already drifted (claude 226, the rest 223) under comments saying "same version on purpose". Between 2.1.226 and 2.1.276, 2.1.265 and 2.1.275 each broke every turn on ANTHROPIC_BASE_URL endpoints, which is how glm and kimi reach `claude` inside a VM; the container-tier verification never exercised that path, so the VM runs on those backends are the real test. Provenance. `VmOutcome` carries the rootfs the node reported booting and the guest's own `claude --version`; `launch_microvm_phase` persists both as `checkpoint.vm` beside `records` (the two readers parse only `records`) and names them in its log line. "Which image and CLI did this mission run on" is a query now. Independence. `evaluator` derived the implementer family from a constant `"anthropic"`, true while every backend was Claude on Anthropic. With glm and kimi rootfs it made a glm mission judged by glm:glm-5.3 read as `independent = true` — the one claim that path exists to make honestly. `implementer_family(missions.backend)` mirrors `microvm_credential_for`; the subscription judge is now independent exactly when the agent did NOT run on Anthropic. Harness. `verify-mission-delivery.sh glm|kimi` run the microvm scenario on each backend and add the proof the mission itself cannot give: the placed node's journal must show the VM dialling that provider's host, never being denied it, and dialling nothing else but the forge — a model's self-report is measured worthless here. `assert_cli_version` reads checkpoint.vm. The stale scratch-repo default (dead since the 09-14 wipe) is the re-synced id. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz |
||
|
Omar SobhandClaude Opus 5
|
248948cc84 |
fix: three things that were known and written nowhere
All three have the same shape — the system learns something and only stderr
hears it — and each was flagged in the handoff as a silent-discard defect.
The gate's install outcome. `container_tool_hooks::install` returned Some or
None and both call sites wrote `let _ =`. A mission whose gate never installed
left a record indistinguishable from one whose gate stood there and matched
nothing. `EnsuredContainer` now carries the outcome to the callers that have a
pool, and they record `gate.installed` (with the settings path) or
`gate.absent` on the mission, so "was this mission gated?" is answerable from
the mission.
The inert marker. `vm_tool_gate` writes an `inert` file when it cannot parse
its input and allows everything, precisely so an inert gate does not look like
a permissive one. The only reader was a unit test. `drain_inert` now reads and
clears it at every tap drain, and a `gate.inert` event with the occurrence count
lands beside the calls that ran unchecked.
The judge's spend. `LlmEvent::Usage` arrived on every judge call and was
matched by `Ok(_) => {}`. Two plan exhaustions (2026-08-29, 2026-09-09) with
no row anywhere saying a judge token had been spent; `usage_events` had no
provider or model column. The loop now accumulates requests and tokens onto the
Verdict — counting a request BEFORE the stream opens, so a 429 the provider
refused still counts, because the retry storm was made of those — and
`record` writes a `kind = 'judge'` row with provider, model, mission and
request count. Migration 0085 adds the columns, all nullable, so the two
existing writers are untouched.
Tests: a scripted-provider verdict records one request and nonzero tokens; a
provider that refuses still records the request and zero tokens.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
|
||
|
Omar SobhandClaude Opus 5
|
8d6310f126 |
fix(judge): stop asking an exhausted plan the same question 180 times
The retry ran on the sweep's own 10s tick for a 30-minute window, so a phase whose judge was unreachable re-judged up to 180 times. A verdict is not one request either: `evaluator` is agentic and loops up to `MAX_TOOL_CALLS + 1` rounds, resending the whole growing history each time, against evidence the code's own comment sizes at ~120 KB. One unjudgeable phase could therefore issue on the order of 2,000 model requests. That is most of why the z.ai weekly plan kept emptying with no mission having visibly done anything expensive — twice now, 2026-08-29 and 2026-09-09. Nothing recorded it, because `usage_events` carries no provider or model column. Two changes: Read the error before retrying. z.ai answers an exhausted plan with a 429 carrying code 1310 and its own reset timestamp. Retrying that is arithmetic, not optimism: the reset was two days out and the phase spent its whole window asking anyway. It now fails immediately and says which problem this is — "the judge provider's plan limit is exhausted until 2026-09-11 10:01:33" sends you to the plan, where "the independent validator could not be reached" sent you into the mission. The classifier is deliberately conservative; anything that does not positively identify itself as an exhausted plan stays retryable, because giving up on a transport blip costs a phase that did nothing wrong — which is how mission 01a011bf lost its script phase. Back off. Waiting as long as we have already waited doubles total elapsed per attempt, so the schedule is exponential with no attempt counter to store: 10, 20, 40, 80, 160, 300, 300 … — about ten attempts in the same window instead of a hundred and eighty. `judge_retry_after` holds the clock and the sweep's SELECT honours it; a landed verdict clears it alongside `judge_blocked_since`. Verified rather than asserted: the migration applies and rolls back against a real postgres, and replacing the backoff with the old fixed tick makes `the_backoff_is_exponential_and_capped` fail (181 attempts, not ~10). Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz |
||
|
Omar SobhandClaude Opus 5
|
fe5c7d2c87 |
fix(tap): a subagent's tool calls are no longer credited to its parent
The previous commit's message ended "mission agents are spawning subagents and
nothing in our design accounts for it." Twelve spawns across the two production
missions, all of them used as a fetch mechanism — container missions have no
`WebFetch` or `WebSearch`, so they reach the network through `Bash` + `curl`,
and 151 of 158 Bash calls are exactly that.
Measured against the real claude 2.1.246 binary rather than reasoned about,
because the containers were reaped and the question had three possible answers:
1. A subagent's tool calls DO fire both hooks. `PostToolUse` records them, and
`PreToolUse` blocked a subagent's denied curl and got the reason back to
it. `Agent` is not a gate bypass — worth knowing before shipping the rule
in the previous commit.
2. They carry the PARENT's session_id. One parent plus one subagent produced
three events on one id. This is why attribution resolved 119/119: a
subagent never adds a session, so attribute_sessions' exact count holds.
3. Only `agent_type` / `agent_id` tell them apart — present on a subagent's
payload, absent on the parent's own.
`hook_script` appends the raw payload, so both fields were already on disk in
every production run. `parse()` read past them. The guest was never the lossy
half, so nothing container-side changes and no redeploy of the image is needed.
`Observed.subagent` / `.subagent_id` now carry them into `mission_events.detail`.
A blank `agent_type` reads as "the turn's own agent", because absence IS the
signal here and a subagent named "" is not a thing.
Same defect class as the tap discarding tool ARGUMENTS until 2026-08-21: the
record looked complete while being wrong about who did the work.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
|
||
|
Omar SobhandClaude Opus 5
|
f52cff3e04 |
feat(skill-use): progressive disclosure, as an arm and not a switch
Trigger — did the agent reach for the skill when it applied? — cannot be measured while every body is inlined into the prompt. Nothing was reached for. `skill_use` has been reporting `NotObservable` for that reason, and it was right to. The skills door made retrieval possible; this makes it a delivery arm. `index` sends each pinned skill's name, description, `when_to_use` and the uri that returns its body, and the agent fetches what it judges relevant. `inline` is unchanged and stays the default. An A/B rather than a switch, because `index` can only cost Compliance: under `inline` the procedure sits in front of the model whether or not it noticed it applied. Trading a measured axis for an unmeasured regression in another is not an improvement, so both arms stay runnable and the arm is recorded on the mission row. Three things the mechanism refuses to do: - `index` without a door falls back to `inline`. An index names bodies and says how to fetch them; with no `clawmates_skills` server reachable that is a list of dead ends, and it fails as an agent ignoring its skills rather than as a missing config. `install_skills_door` now returns whether it installed, because the caller needs the answer and not just the log line. - The scorer reads the arm off the recorded PROMPT, not off the mission row. The row says what the mission is configured to do now; the score is being computed against a turn that ran then. - Under `index`, a skill that was offered and never read is a Fail, not the inline arm's `NotObservable` — but only where the skill had a checkable consequence in that phase. Reusing the inline text would have said "this skill was inlined into the prompt" about a skill whose body was never sent, and scoring a real miss as a structural blind spot is the failure this measurement already made once. The arm is per mission (`config.skill_delivery`), not only per deployment. Both arms run against one server process; restarting between them would put a confound in the comparison that the numbers would not show. 829 tests, 108 binaries, green. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
5220f3bfea |
feat(skill-use): red-first is observable from what the RUNS reported
The open item said this needed the repository diff rather than tool
order. That was wrong, and the skill says why: "Commit the RED-to-GREEN
pair as one commit." The failing test and its fix land together by
instruction, so the diff and the commit history are as blind as the tool
ordering already was — in Rust one `Edit` adds the implementation and its
`#[cfg(test)] mod tests` in the same call.
The only remaining witness is what each test run itself printed, and the
tap was throwing it away. Claude Code's PostToolUse payload carries
`tool_response` — verified against the real binary, keys
stdout/stderr/interrupted, plus `duration_ms` and `tool_use_id`.
So `Observed.response` now keeps it, for COMMANDS only: a `Read`'s
response is the file it just read and a `Write`'s restates its own
argument — both already knowable, both large, and storing them would
double the biggest write path in the system for nothing.
`bounded_response` keeps the **end** of the output, which is the opposite
of `bounded_input` and deliberately so. An argument's meaning is its verb,
at the start. A command's meaning is its verdict, at the end: `cargo test`
prints hundreds of lines and then `test result: ok` or `FAILED`. A
head-biased truncation would keep the noise and discard the only thing
being stored for — negative-controlled with a 400-line fixture.
`red_before_green` now falls through to the run outcomes:
failing run, then a passing one → Pass, red then green observed
every run failed → Fail, the loop ends on green
every run passed → NotObservable, and the reason says
why: a test that never failed is
equally what a correct implementation
written first looks like
no outputs recorded → NotObservable (pre-capture missions)
Read from the runner's verdict line, not an exit code — the payload
carries none.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9
|
||
|
Omar SobhandClaude Opus 5
|
8591585e60 |
feat(missions): attribute a phase's tool calls to the agent that made them
`record_vm_tools` wrote `agent_id: None` for every call. The container tap is per-CONTAINER and every role in a phase shares one, so a phase arrived as one undifferentiated stream: every Skill-Use score was per-mission rather than per-role, and the World's per-agent view got nothing from this tier. One `claude -p` invocation is one turn is one agent, and Claude Code stamps each invocation with a `session_id` the tap was discarding. So the distinct sessions, in order of first appearance, are the phase's turns in the order they ran — and `prompt.composed` already records the agent of each turn in that same order, written by the tier as it sends each turn, so it IS the running order rather than a reconstruction of it. **It attributes nothing rather than guessing.** Only when the counts match exactly. A phase whose sessions and turns differ has something this correlation does not model — a retry, a turn that called no tool, two genuinely concurrent agents — and a plausible-looking wrong attribution is worse than none here: it puts one agent's `git push` on another agent's record, and a person later reasons from that. One call missing a session id refuses the whole batch, because a hole shifts every later session onto the wrong turn. The microVM call sites pass no turn agents and so keep today's behaviour exactly. Resolving a graph node to an agent uuid is the fix there, it cannot be tested while the fleet is offline, and guessing would put one node's actions on another node's record. Also restores the `#[cfg(test)]` gate on `repo_less_text_tests`, which my own insertion had taken — those tests would have compiled into release builds. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9 |
||
|
Omar SobhandClaude Opus 5
|
8cb38d1320 |
feat(missions): keep the tool's arguments, not just its name
The container tier's first measured mission recorded `Bash × 6` and not one of them said what it ran. Every behavioural question about the phase — did it run the tests, did it commit, did it call an API a skill forbids — was unanswerable from a record that looked complete. `vm_tool_tap::parse` already read `tool_input` to pull the path out of it, then dropped the rest on the floor. It now keeps it, bounded: file bodies (`content`, `new_string`, `old_string`, `edits`) become a byte count, and any other over-long string is truncated with a marker saying so. Bounded rather than whitelisted, because a whitelist silently loses the one argument that matters the first time a tool grows a field. `file.touch` keeps the absolute path in `detail.abs` alongside the repo-relative `target`. Normalising is what the map needs and exactly what destroys "did this write land outside the checkout". `tool.call` also gains `detail.path`, which the World's SSE has been reading and getting a null from on every container-tier call. `mission_events::tool_evidence_for_mission` is the reader — the counterpart to `narrative_for_mission`, and the reason it exists: the narrative is what an agent SAID it did. Host-side only. No image rebuild: the arguments were always in the tap file, the first parse threw them away. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9 |
||
|
Omar SobhandClaude Opus 5
|
e84413d437 |
fix(missions): the container tier now records its tool calls — verified live
Ran it end to end on a real mission. First time the container tier has ever
been observable:
tool.call 10 Bash 6, Read 3, Write 1
file.touch 4 research/tapproof.md
reasoning 5
prompt.composed 5
Three defects found by running it, each of which left every other link
looking correct:
1. The settings document pointed PostToolUse at {TAP_DIR}/tap.sh while the
installer wrote {HOOK_DIR}/tap.sh. Claude Code does not complain about a
hook command that does not exist — it records nothing. Asserting the
script "mentions tap.sh" had passed; the PATHS have to be compared, and
a test now does that for every hook the document names.
2. The mission container runs CLAWMATES_RUNTIME_IMAGE, not the shared
runtime container I had swapped. It was still on an image whose daemon
schema has no `settings` field, so set_claude_cli_settings returned
404 path_not_found — which the error message said plainly, and which is
the only reason this was quick to spot.
3. The sweep used connect_with_local_defaults(). The server reaches Docker
through a socket proxy (DOCKER_HOST), so that connector fails there — and
my code returned Ok(()) on the error, silently. The tap filled up, the
query matched rows, and nothing ran. Now uses container_exec::connect and
logs the failure; a test pins the choice.
All three are the same shape as the bug they were chasing: installed,
inert, indistinguishable from working. The tests added for each compare the
two ends rather than asserting a string appears somewhere.
Full workspace suite green: 107 binaries, 412 lib tests.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
cd59e4798d |
feat(missions): collect the container tier's tool calls
The hooks from the previous commit write a tap file that nothing reads — which is the same shape as the gate that is installed and inert: everything looks wired and no evidence ever appears. The microVM tier records its tools from inside the loop watching the VM. A container turn is driven asynchronously by topology_worker, so there is no such loop and something has to come and collect the file. `drain_finished_container_phases` does, on the same tick as the benchmark baseline and the security scan, reusing `record_vm_tools` so container tool calls land as the same TOOL_CALL / FILE_TOUCH events the World already renders. One shape, two tiers. Idempotent by TRUNCATION, not a marker or a cursor column: `drain` clears the file it read, so a second pass finds nothing. Read-then-clear happens in one exec, and only for phases that have FINISHED — the agent is no longer appending, so the gap between read and clear cannot lose an event. A cursor would have needed a migration and a column that means nothing to anyone else. Two tests exist because the failure is silent either way: the drain must clear what it read (otherwise every tick re-records the same calls and a phase's early files end up weighted by how long the sweep ran), and the tick must actually call the sweep (otherwise the hooks write a file nobody collects). Full workspace suite green: 107 binaries. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
91a6b4e304 |
feat(skills): the first Skill-Use measurement, and the three defects it found
Scored on the paper's three axes against two real missions on the local
stack. docs/SKILL-USE-BASELINE.md has the numbers, the method, and the
limits.
Trigger is reported as NOT OBSERVABLE, never zero
The paper measures progressive disclosure: the agent sees a name and
description and must retrieve the body, and that retrieval is the Trigger
event. We inline full bodies, because mission claws run on claude_cli which
cannot surface a tool call — there is nothing to retrieve with. So the
agent never reaches for a skill, it simply holds one.
Scoring that zero would report a delivery-model property as an agent
failure, which is the same confusion that kept 55 empty bindings invisible
for months. The verdict type carries NotObservable(reason) as a distinct
case from Fail for exactly this.
Compliance is checked by running the REAL task_card_parser rather than a
copy of its rules — a second implementation would drift, and then the score
would pass while the mission loop still stalled. Skills without a
machine-checkable consequence score not_applicable rather than a guess.
WHAT THE MEASUREMENT FOUND
1. The prompt format made its own record unparseable. Skills were
introduced with `## <name>` and skill bodies are markdown full of `##`
headings, so run 1 scored "Sizing heuristic" and "The output shape" —
subheadings inside decompose-int-items — as skills with no catalogue
row. Now an unambiguous `--- SKILL: <name> ---` marker, with both
writers sharing one renderer so the reader cannot drift from the writer.
2. A prompt was recorded that was never sent. My own Phase 1 work recorded
the phase prompt at the dispatch fork, before the tier was chosen — and
the container tier does not send that text, it sends the bare task and
appends skills per turn. Every container mission logged a `solo` prompt
that reached no agent. A provenance record of something that did not
happen is worse than no record: it is the wrong answer, delivered
confidently. Recording now happens inside each tier, with a test that
every launcher records the prompt it actually sends.
3. int-xx-marker-protocol documents a marker the platform never
implemented. PLAN_COMPLETE is in the skill's ladder and task_card_parser
has no such kind and never has, so an agent following the skill exactly
emits a marker that is silently ignored. Observed live: run 2's planner
emitted `PLAN_COMPLETE: INT-01..02`, which is also the range form — on
the kinds that ARE parsed that yields the id `INT-01..02`, a task card
for an item that does not exist while the two real items stay open.
This is a skill/implementation mismatch, not an agent failure, and it is
exactly what the measurement exists to find: the agent did what it was
told and what it was told was wrong. Both shapes now score as failures.
The reconciliation — implement PLAN_COMPLETE or drop it from the skill —
is left as a decision rather than guessed at.
The boot log now shows what the plan asked for: 53 skills, 11 templates,
every one `N role skills bound` with NO unresolved clause. Live missions
confirm per-role delivery — the planner receives decompose-int-items, the
coder receives write-rust-current-edition.
GET /api/missions/{id}/skill-use exposes the scores, and says in its
payload whether an empty result means "nothing delivered" or "the evidence
was reaped" — those have very different causes and must not look the same.
n = 2. No spread is reported because two runs cannot establish one, and the
document says so rather than letting the number be quoted as a baseline it
is not.
Full workspace suite green: 106 binaries.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
769e002bb3 |
feat(skills): deliver on every tier, record what agents receive, let them self-author
Three phases of the approved plan, plus a correction to what the last one
claimed.
CORRECTION: skills reached ONE tier, not all of them
The previous commit said "skills can now reach a mission agent". That was
true only for the container/ZeroClaw tier — the fall-through that queues a
topology_runs row for topology_worker, which drives the executor that was
patched. compose_turn_prompt/pinned_skills_text had exactly one production
caller, and phase_runner's three other paths (composed microVM, solo
microVM, direct session) never called it. CAPABILITY-REVIEW.md said the
broad thing too; both are corrected.
Those three tiers share one task string and have no per-turn alias, so
their skills resolve per PHASE from the mission's crew and are appended
there. The container tier deliberately still injects per turn, with the
running node's own role — appending in both places would put every crew
member's skills in every turn twice.
The behavioural tests prove phase_skills_text and compose_turn_prompt work.
They cannot prove the three launch_* calls pass the composed string, and
that substitution is a one-word edit that would silently return all three
tiers to delivering nothing with every test still green. So there is also a
source-level assertion on the call sites, following the precedent in
mission_events::the_cap_is_enforced_in_one_statement. Its negative control
names the exact tier.
PROVENANCE: what an agent received, and what it said it did
Both were unanswerable. The prompt was never stored anywhere on any tier —
re-deriving it later re-runs the skill lookup against a catalogue that has
since changed, and once agents author their own skills it certainly will
have. The reasoning rows were durably write-only: pushed live once, then
never read from the database again by anything except the GC that deletes
them.
- prompt.composed records the exact bytes, on all four tiers
- the session tier writes its checkpoint record and a reasoning row,
instead of eprintln! and nothing — the same defect the solo microVM
path was fixed for, in the last tier that still had it
- narrative_for_mission reads both back
Found while doing it: the 400-event per-phase cap counted EVERY kind, so a
busy phase could push out its own phase.completed and its own provenance.
The cap now counts only the two unbounded kinds it was written for.
Negative control confirms the old behaviour dropped the prompt.
Retention is now a per-mission hold (0080) rather than a raised global —
with a test asserting unheld missions are still reaped, because an
exemption that applies to everything is not an exemption.
SELF-AUTHORING: agents apply their own skill drafts, no human click
By operator decision. level_up has generated complete drafts from a model
since it shipped; only a checkbox stood between propose and apply.
What replaces the gate is not another gate but four properties, each held
by a test:
- workspace-scoped, so a hand-authored skill can never be modified
- a draft cannot take a hand-authored skill's name. Ids are scoped and
bindings resolve by skill_id, so it could not overwrite or shadow one
anyway — but two procedures under one name means nobody reading a
transcript can tell which the agent followed, and that ambiguity is
fatal in a system where the skill is the standard being graded against
- every revision appends a skill_versions row, so it can be reverted and
a past run can be read against the text it was actually judged under
- approved_by = NULL. An agent's decision is never attributed to a person
who did not make it
Only skill_candidate applies autonomously. identity_refinement and
brain_consolidation still wait for a human: they change what an agent IS
rather than adding a procedure it can consult. State is announced at boot,
because a safety gate that changes silently is one nobody notices changed.
CLAWMATES_SKILL_SELF_AUTHORING=0 restores it.
Also: the test Postgres ran out of /dev/shm mid-suite (Docker's 64MB
default) and surfaced it during MIGRATIONS, which reads like a schema fault
and is not one. --shm-size=1g, and a pointer to the `clean` subcommand that
already existed for the 779 leaked test databases.
Full workspace suite green: 106 binaries, no failures.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
e4942ce985 |
fix(missions): skills can now reach a mission agent at all
Repairing the 55 broken skill bindings made the catalogue correct. This
makes it reachable, which it was not — for any skill, on any mission, since
the catalogue was built.
The skills had exactly ONE delivery channel: the `clawmates_skills` MCP
server. A mission claw could not reach it for three independent reasons:
1. `provision_claw` wrote the constant `["clawmates_door"]` and ignored
the template's mcp_bundles — which mission_orchestrator had already
resolved and stored on the team row.
2. The runtime config defines no `clawmates_skills` bundle. The live
local config defines no bundles at all, not even the door.
3. Mission claws run on `claude_cli`, which the runtime's own config
comments document as text-only: it cannot surface a tool call, so no
MCP server is reachable from a mission turn regardless of bundles.
And a mission turn's whole system context is two sentences synthesised from
the role slot in topology_exec::build_prompt. The template's role prose is
not used either — mission_orchestrator documents this, and it means the
role prompts describing which procedures to follow were never read.
Two doc comments in cm-runtime describe the mission path as already having
the summary-and-fetch contract. It never did. The belief was written down
twice and checked zero times, which is why nobody looked — and it is why
the Skill-Use measurement this review planned could only ever have returned
a trigger rate of zero. That would have read as a finding about the agents.
- provision_claw takes the bundles, with clawmates_door always added: a
template that forgets to list it must not get an ungated agent
- all 11 templates now request clawmates_skills; web_fetch removed, since
a list that is honoured must not name a bundle that does not exist
- the re-provision sweep re-asserts the team's own stored bundles rather
than a constant, which would have silently stripped a capability
mid-mission
- pinned skill BODIES are injected into the mission prompt, bounded and
with truncation stated. Bodies, not an index: there is no `skills.read`
tool on this path, so an index would advertise a capability that does
not exist — the exact failure this whole change is about
Three tests: the body reaches the prompt, an agent with no skills adds no
heading (an empty "Your skills" section announces skills the agent does not
have), and the composition is exercised separately from the lookup, because
`pinned_skills_text` working and `run_turn` calling it are different claims
and the second is the one that was false.
Also adds the three review documents: CAPABILITY-REVIEW (inventory, what
was repaired, what is deferred and why), PROVENANCE-ASSESSMENT (assess
only, per decision — what each store answers and the two candidate paths),
and RESEARCH-SWEEP (the fortnight's papers and what we did about each,
including the ones we deliberately did nothing about).
Full workspace suite green.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
18dc0b964b |
fix(missions): the security scan phase now scans, and task upserts work
Four defects, found by checking the audit's claims instead of trusting them. Two of the audit's own findings turned out to be wrong, and the registry that exists to record which config keys are read was itself inaccurate — so the corrections are part of the change. upsert_task raised 42P10 on every call, for every caller `mission_tasks_external_uniq` is a PARTIAL unique index (WHERE external_id IS NOT NULL). Postgres will not match a partial index to an ON CONFLICT target unless the statement repeats the predicate, so the upsert failed on its first row. Both callers — the task-card parser that turns INT markers into tasks, and the security scanner — map the error to a string their caller logs. Two features were broken and nothing was red. Regression test in cm-db with a negative control: reverting the WHERE reproduces 42P10 exactly. the security scan never ran `security_scan::run` was reachable only from an operator button, so security_hardening.toml — a workflow whose entire first phase is a scan — ran an agent that was never told to scan and never fired the scanner either. phase_runner now sweeps finished security_scan phases, mirroring the benchmark baseline sweep that was added for the identical defect. Guarded on a new completion marker rather than on findings: a clean scan writes no findings, so a findings-guard would rescan forever. The marker also answers the question an operator actually asks, which is not "how many findings" but "was this looked at, by what, and when". two recipes could not fail security_hardening.toml and benchmark.toml carried no `task` and no `done_when` on any phase. A phase without done_when never enters evaluating, is never judged, and reports completed whatever it did — so a security mission could scan nothing and go green, and a benchmark mission could record no baseline that the next refactor would then compare against. Both now state the work and the condition, with inert keys annotated inline rather than deleted, so the gap between what a recipe asks for and what a phase receives stays visible. the config registry was wrong in both directions `harness` was listed NOT IMPLEMENTED while benchmark_runner reads it and phase_runner runs a baseline through it. `tools` was listed NOT IMPLEMENTED while security_scan::run reads it. A registry that exists so an operator can trust what a recipe does is worse than useless when it is inaccurate. Both corrected, `bench_name` and `cmd` added, and `test_command` deleted — it had neither a reader nor a writer, so it described a situation that could not arise. Also: CLAWMATES_JUDGE_MODEL had two different defaults (opus-4-8 in routes/topology.rs vs opus-5 in cm_runtime::judge_model) and a doc comment naming a third; topology now calls the one function. GITEA_TOKEN's absence in mission_plan is stated rather than degrading to the same "could not be read" string a private repo produces. BRAINHUB_API_KEY needed no change — hub::push already rejects an unset key with a named error. That half of the finding was overstated. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
d524107b37 |
fix(missions): harvest before the checkout, and stop an unreachable judge failing done work
Two bugs from the first real Continuous Research run, both found by running it. **1. The harvest ran AFTER the checkout.** `on_launch` cloned the vault and then harvested, so the mission's working copy predated the manifest push. The reader agent found no `harvest.jsonl` and — being resourceful — queried arXiv itself and wrote its own. That is exactly what `skills/research/arxiv-daily.md` forbids: the papers it found are not checked off in `corpus_items`, so the next run re-offers them, while the 13 the real harvest DID shelve went unread. The harvest now runs first, so the clone contains the manifest. The analysis it produced was otherwise very good — it named `crates/clawhdf5-ann/src/hnsw.rs`, cited the ROADMAP's serial insert loop and proposed a concrete pre-build probe — which is the behaviour the whole design is for. It was reading the wrong papers. **2. An unreachable judge consumed a pass.** `Verdict.error` exists to distinguish "could not judge" from "judged incomplete" and nothing acted on it. glm-5.3 returned "transport error: error decoding response body", the phase counted it as a failed pass, and with two budgeted that single outage failed a phase whose work was done and committed. The evaluator was right to refuse a same-family fallback — that would trade independence for availability — so the fix belongs here: an unreachable judge no longer spends an iteration. Retrying forever would trade a wrong failure for an invisible hang, so the wait is bounded by `judge_blocked_since` (migration 0078), mirroring how `capacity_blocked_since` bounds a phase waiting on a VM slot. Thirty minutes is many sweep ticks, so a blip recovers inside it; past that the phase FAILS with the transport reason rather than requeueing, because re-running spends a container re-doing work that was never the problem. 346 tests pass. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
b5032a732a |
fix(phase_runner): collect the agent's work BEFORE judging it
`Sandbox::for_mission` builds the judge's verification copy from the HOST
checkout. In copy mode the agents write inside the container, and their work
only reached the host when `sync_out` ran — in the capture sweep, AFTER the
phase closed. So every phase was judged against a tree that did not yet contain
the pass being judged, and the judge truthfully reported nothing there.
Mission 01a00cfa is the proof. Research pass 2 wrote a 434-line
IMPLEMENTATION_BRIEF.md, `cargo test` passed, and it was pushed to a clean
branch (clawmates/mission-01a00cfa-c69f39fd-i2 at 563cdd21). Its verdict:
failed after 2 pass(es) — met=false — research/IMPLEMENTATION_BRIEF.md
does not exist anywhere
logged one line BEFORE `captured (+434/-0 across 1 file(s))`. A phase that
succeeded was failed because the evidence had not been collected yet.
This hid because it only bites a phase judged on its OWN pass. The v2 coding
verdict cited real commits (339a5bd, 167671f) — research had already synced
that work to the host in an earlier phase.
`evaluate_finished_phases` now runs `sync_out` first, and on failure leaves the
phase `evaluating` for the next sweep rather than recording a verdict nobody
could stand behind — the same policy the capture sweep already applies, for the
same reason. microVM keeps its carve-out: `microvm_executor` collects out of
the guest over this same path before the VM is destroyed.
Research goes to 3 passes. On 01a00cfa it got no real attempts out of two: one
spent on a fabricated commit claim the judge correctly rejected, one on this
bug.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
fd5e71ccfe |
fix(missions): re-assert a mission's crew when its container is recreated
Claws are provisioned exactly once, at on_launch. `ensure_container`
RECREATES a container that is not running, and recreation reseeds
.zeroclaw from the seed directory — which does not hold this mission's
claws. The agents still exist in Postgres and the crew query looks
perfect, so nothing reads as broken; the alias is simply gone from the
daemon and /ws/chat answers 400 Bad Request. A retried mission could
therefore never connect again.
Re-assert the crew after ensure_container. provision_claw is idempotent,
so this costs one call per claw on the happy path and is the difference
between a resumable mission and a dead one.
Two things this has to get right, both of which fail silently:
- Aim at the per-mission daemon via for_gateway(ec.endpoint), never
from_env() — that targets the shared global gateway and leaves this
container with no claws at all, exactly as for_gateway's own doc
comment warns.
- Provisioning creates the agent but cannot set workspace.path (the
config prop-schema has no way to express it), so follow with
pin_agent_workspaces or every claw runs in its own sandbox and
delivers nothing.
Verified on mission 01a00538: research and coding phases both completed
after four straight failures, with all five graph aliases present and
ten claws pinned to /mission/repo.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
548f977212 |
style: rustfmt the four files the repo-less mission fix touched
Found while removing .github/workflows/ci.yml: three of the four files in that change were unformatted, and four of the diffs were newly introduced (the new prompt tests and the tool_preamble format! call). Formatting only the files that change already touched — a repo-wide `cargo fmt` would be 63 files of unrelated churn and belongs in its own commit. Mechanical; `cargo test -p cm-api --lib` stays at 322 passed. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
4dec77ae6d |
fix(missions): give repo-less container missions the workspace they are promised
Every agent on a research_only mission refused to work, each reporting it was "in Claude Code", had no /mission/repo, and only had Read/Edit/Bash. All three statements were true. The run still recorded completed — 5 turns, 7.4k tokens, 0 artifacts, no error. The machinery is correct when a repo IS bound (verified on a live prod per-mission container: /mission/repo present, all 5 agents pinned). Only the repo-less path was broken, in three layers that disagreed by construction: - sync_in no-oped without a host checkout and copy mode does not bind /mission, so NOTHING created /mission/repo. The microVM tier already creates it, for the stated reason that "the guest needs the workspace to exist before the agent writes into it". Creating it host-side also un-breaks sync_out, equally a no-op before, so work survives across phases instead of being wiped. - pin_agent_workspaces returned Ok after pinning ZERO agents, so the deliberately-fatal guard in mission_orchestrator could never fire. Its error text already described the exact outcome we got. - The prompt advertised ZeroClaw tool names and explicitly denied `bash`, while every executor ends in `claude -p`: microVM passes Read/Edit/Write/Bash/Agent, session passes Read/Edit/Write/Bash, and claude_cli agents get Claude Code's native toolset — ZeroClaw's gating never reaches the subprocess. It was telling agents to use missing tools and avoid present ones. And it went green because mission_outputs logged the failed collect and continued — with the fail-empty rule and the NO-OUTPUT marker both BELOW that continue, so the phase was retried forever and never failed. The retry is now bounded by a grace window off completed_at. Verified end to end: mission completed, agent wrote /mission/repo/research/firecracker_vs_docker.md, collected and registered as a document artifact (6.6 kB of real content). Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
31158467f4 |
feat(viz): microVM tool motion is live, and needs no fleet-node change
The plan deferred this as "the only fleet-node binary change". It is not one. `fcagent` is thread-per-connection — its own comment says so, and the live log tail has relied on exactly that for the whole length of a turn, on a second connection. So the host can drain the tap WHILE the turn's exec is in flight, from the server alone. The turn and a 20s drain loop now run concurrently. A coding phase shows its files being touched as it works rather than an hour later, all at once, and the drain is bounded by a cursor so a repeated poll returns only what is new. The cursor counts LINES, not parsed events, and that distinction is the bug this commit would otherwise have shipped. The hook appends the event and then a newline of its own, so a two-event tap is four lines; advancing by event count leaves the cursor two lines short, `tail -n +N` hands back events already recorded, and the live drain re-records everything it has already written — worse the longer the turn runs, and silent throughout. Caught while writing the test, not by it. `tap_sink` and `VmOutcome::tools` are mutually exclusive by contract: with a sink, the sink owns recording including the final batch and `tools` comes back empty. Handing the same calls back on both would double every file orb's weight with no way for the caller to tell which it was looking at. The sink is an unbounded channel to a recorder task, so the VM executor stays free of the database: it observes, phase_runner records. The task ends when the sender drops with the phase. Verified before this change: the microVM tap is real. The `microvm` scenario passed 6/6 and left ten `tool.call` rows and a `file.touch` on MICROVM.md, repo-relative, from Claude Code's own PostToolUse hook. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
9e61e3ba35 |
feat(viz): what the agents actually did, as structured events
The World could draw a mission's shape but nothing about the work. The
detail existed only as prose in checkpoint.log and model output, where a
tool name is indistinguishable from an agent *talking about* a tool — so
it was never parsed, deliberately. `mission_events` is the structured
channel that replaces it.
Three taps, one table:
- Container tier: the `_ => {}` at the end of topology_exec's typed frame
stream now matches `tool_call` and reads the tool's JSON ARGUMENTS for a
path. Never the prose summary — a path scraped from a sentence would put
files on the map that no agent opened, and the test proves a Grep whose
summary says "src/main.rs" produces no file touch. The frame name itself
is unverified, so the same commit ships an unmatched-frame-type
histogram: a tap that matches nothing looks exactly like a mission that
used no tools, and this is how one gw-04 run names the real frame.
- microVM tier: a `PostToolUse` hook, the seam vm_stop_gate already proved
fires under `claude -p`. It copies stdin to /root/tap and exits 0
unconditionally — a non-zero PostToolUse hook talks back to the model,
which would turn the observer into a participant. Drained before collect,
since the VM is destroyed moments later.
- Phase transitions: five identical copies of the pending→running UPDATE
became one `mark_phase_running`, and `close_finished_phases` grew
RETURNING. Its CASE decides each phase's status inside SQL from rows the
statement does not change, so it cannot be re-derived afterwards without
writing that CASE twice — without RETURNING it emits zero phase.completed
and reports success.
The settings.json hazard the plan called out: the stop gate wrote the
WHOLE document, so a second hook writer would have silently erased it and
a coding phase would then complete having written nothing — the exact
failure the gate exists to catch. There is now one composer,
`vm_tool_tap::guest_settings`, one writer, and a source-walk test that
fails if anything else writes a settings document.
`mission_events.run_id` carries no FK on purpose: phase_runner DELETEs
topology_runs on retry, and a cascade would erase a phase's whole history
the moment it retried — silently, since a cascade is not an error.
world.rs streams it with a cursor that separates backfill from motion.
Everything already in the table when a subscriber arrives is drawn as
settled history; only what lands afterwards animates. Otherwise opening a
finished mission replays an hour of tool calls as a burst storm.
Bounded twice: 400 events per phase (enforced inside the INSERT, since
two concurrent taps would each read a count below the cap) and a 7-day
retention sweep in mission_gc.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
5f85dbb718 |
fix(world): missions were never really on the wire
Three bugs in one query, each hiding the next, plus one that made the whole rich layer dead code. `active_missions` joined `team_members` on `missions.team_id` — the LEGACY pointer at the first minted team, superseded by the `mission_teams` junction in 0056. It was an INNER JOIN, and `mission_orchestrator::on_launch` deliberately mints no team for a microVM mission, so the platform's primary execution tier was dropped by a join and the World has been showing nothing at all for it. And it selected only `status='running'`, while missions finish in minutes, so the scene was empty almost always. Now: join `mission_teams`, LEFT so teamless missions survive (their `agent_id` is NULL and no pawn beams at them, which is the truth — nothing on this platform ran that phase except a VM), and include missions finished in the last 24h carrying `status`/`template_kind` so the client can draw a finished map instead of animating a corpse. `?mission=` scopes the feed server-side. The whole phase plan now ships as `mission.phase`, including phases that have not started: a phase list that appeared only as phases began made a five-phase mission look like a one-phase mission until it was nearly over. Attribution reuses `phase_runner::purposes_for` rather than copying it — two copies would let the picture disagree with the machine about who is working on what, which presents as a rendering bug and is really a lie. Deleted the checkpoint tail. It read `topology_runs` keyed by an `agent_runs` id; mission phases live in `topology_runs` under independently generated ids, so it ran every poll and matched nothing, for every mission, forever. That is why no mission has ever shown tool or file activity. Not repointed at `topology_runs`: its only per-step content is the agent's own prose, and a tool name in prose cannot be told from an agent talking about a tool. The a2a `run_events` tail is kept — it genuinely works for the path that writes it. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
dd80b69992 |
fix(microvm): the third place that assumed a VM phase has a repository
The research-vm scenario failed on its first run, and said exactly why:
mission has no checkout at /var/lib/clawmates-missions/<id>/repo
— a microvm phase needs a repository
`phase_runner` refuses upstream of both places the last commit fixed. Three
guards, written independently, all encoding "a microVM phase implies a git
checkout" — which is why the capture filter could cite it as settled fact.
A repo-BACKED mission with no checkout is still a real fault and still refused;
booting a VM to hand the agent an empty directory would turn a setup failure
into a confusing agent report. A repo-LESS one now gets the empty workspace
made here, so the executor's inject has something to pack.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
768e106614 |
fix(microvm): a mission with no repository can run in a VM, and its work comes back
Two halves, and the first was worse than the plan assumed. `run_phase_in_vm`
packed `<missions_root>/<mission>/repo` unconditionally — a directory a
repo-less mission does not have — and then required `/mission/repo/.git` inside
the guest before spending a turn. So a repo-less microVM phase did not merely
go uncaptured: it failed before the agent ran.
A repo-less mission now gets an EMPTY workspace at the same guest path, created
host-side so the collect unpacks back over it with no special case, and the
readiness probe asks for what was actually sent — the directory rather than a
`.git` that was never going to be there.
`mission_outputs` then drops its `runtime_kind <> 'microvm'` exclusion, whose
stated reason ("a microVM mission always has a checkout") is exactly what
stopped being true. Where the files come from now depends on the runtime, and
the difference is not cosmetic: a container mission's output is still inside a
running container, while a VM's has already been unpacked onto the host by the
end-of-turn collect. Asking docker for a VM mission's files would query a
container that never existed.
The recursive copy skips symlinks rather than following them — a link out of
the tree would publish whatever it points at.
`research-vm` is the proof, added to the suite as well as the dispatch: the same
assertions as `research-only` with `runtime_kind: microvm`. A scenario nobody
runs is a scenario that does not exist.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
529497febb |
fix(placement): a composed graph needs every backend its nodes name
The full harness found it — 12 of 13 scenarios green, `roster` red:
roster: the planner sized this mission at 2 member(s) PASS
roster: the approved roster is on the mission (2 nodes, composed) PASS
roster: this run added 1 line(s) for a 2-member roster FAIL
topology_runs.error: turn executor failed: node n1 in a microVM:
vm_create failed: no rootfs for backend "canary-claude" on this node
The roster proposed `verifier@canary-claude`. Placement asked
`online_for_backend` about the MISSION's backend — `claude` — and architect
answered, holding `claude` and `local-ornith`. The graph's first node ran and
delivered, the second could not boot, and the mission finished half-done. The
question placement asked was true and insufficient.
A composed graph runs on ONE node, so that node needs every image its nodes ask
for. `required_backends` collects the mission's plus each
`config.roster.nodes[].attrs.backend`, and `online_for_backends` passes the
whole set to the same jsonb `@>` — containment already means "contains ALL of
these", so the query shape did not have to change, only what it was asked.
This is the failure mode the roster feature creates by existing: its entire
purpose is putting a verifier on a different provider, which is exactly what
makes one node insufficient. Nothing before the full suite had a reason to
exercise it — the composed scenario uses one backend for all five nodes.
`NoCapableNode` now names the set and says why one node must hold all of them.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar Sobh
|
d84d17207f |
feat(placement): place per phase, and let a full fleet queue
Phase 1b: wires the capacity model from
|
||
|
Omar Sobh
|
0b89b8316c |
feat(observability): stream a microVM turn's stdout/stderr to the platform live
The Live tab showed nothing while a turn ran, and the agent's own account of it
went to stderr on the node and nowhere a user could reach. This is the path that
carries it.
The blocker was the guest agent. `fcagent` handled one connection at a time,
inline, so during an hour-long turn the VM accepted nothing — which is why every
existing probe (subagents, stop-gate blocks, cap) runs AFTER the turn rather than
during it. It now spawns a thread per connection, wrapped in `catch_unwind`
because this process is pid 1: a panic used to take the accept loop with it, and
an unbootable VM is a far worse outcome than a missing log. A failed spawn logs
and keeps accepting rather than dropping the listener.
PROVED against a live VM before building on it, since "sound reasoning about this
system" and "measurement" have diverged repeatedly today. Patched rootfs, booted
under Firecracker, ran an 8s exec and a concurrent tail:
exec took 8.0s ok=True
+0.0s 'line1\nline2\n' +1.2s 'line4\n' +3.2s 'line6\n' +6.0s 'DONE\n'
VERDICT: CONCURRENT — tail returned data before exec finished
The rest is the pattern the terminal already uses. New `tail` op streams a file
by OFFSET (so a dropped link resumes instead of replaying, and the tail always
terminates — one that never returns pins a thread for the life of the VM). The
node follows the log alongside the turn and pushes `Uplink::VmOut { run_id, at,
data }` over the WebSocket it already holds, mirroring `PtyOut`. The server does
what `PtyOut` deliberately does not: it APPENDS to the run's checkpoint as well
as fanning out, because a terminal has no history worth keeping and a mission log
is the record of what the agent did. `run_events_sse` emits the new bytes as
`step` events, which the live pane already renders — no frontend change.
The turn is `tee`d, not redirected: the file feeds the live stream and stdout
still becomes `VmOutcome::summary`. A redirect would have produced a live view
and an empty summary, which is the same green-and-empty shape as the bug this
fixes. Tested, along with the log living outside the collected tree so it never
lands in a user's delivered diff.
246 lib tests, 20 binaries; node and fcagent build clean.
|
||
|
Omar Sobh
|
62509a5090 |
fix(missions): a solo microVM run showed the operator an empty Live and Output tab
Found by a frontend wiring sweep, then confirmed in the database.
Everything the UI shows of a run's CONTENT reads
`topology_runs.checkpoint.records`: `/api/missions/{id}/documents` behind the
output reader, and `/api/topology-runs/{id}/events` behind the live pane. The
`team` and `microvm_graph` tiers write those records. The SOLO microVM path
never did — it updated `status` and nothing else:
tier | checkpoint_null | records
microvm_graph | f | 2-5
team | f | 5
microvm | t | 0 <-- every one
So a single-phase microVM mission ran real work, delivered a real branch, and
showed an empty Live tab and an empty Output tab. The agent's own account of the
turn went to stderr via eprintln and nowhere a user could reach.
Note what was NOT broken, since that was the initial suspicion: the SSE path
matches (`/api/topology-runs/{id}/events` on both sides), and a sweep of all 130
frontend `/api/` calls against the 164 registered routes found zero genuinely
missing endpoints. The wiring was fine; the data was absent.
The run now persists its turn as one record shaped exactly like the ones those
two readers already parse — `node_id`, `role` (the phase kind), `phase`,
`output` — so no reader changes. Written with `checkpoint || $3::jsonb` so a
future writer of other checkpoint keys is not clobbered.
246 lib tests.
|
||
|
Omar Sobh
|
0d8db7ff0b |
fix: close the three remaining gaps, and repair a test I silently disabled
FIRST, the self-inflicted one. My edit in |
||
|
Omar Sobh
|
821cbb8622 |
feat(missions): hold every producing phase to delivering, and read markdown instead of PDFs
Two changes the portal review asked for.
1. `benchmark` and `security_hardening` had no delivery guarantee.
`empty_delivery_is_a_failure` tested `kind == "coding"`, on the reasoning that
"research phases legitimately write nothing to the tree" — which the research
directive three modules over contradicts, since it tells the agent to save
findings under /mission/repo/research/. The cost: a `benchmark` mission is ONE
benchmark phase, and with that phase exempt nothing in the platform could fail
it. Same for `security_hardening`, whose first two phases are security_scan and
research.
Now keyed on PRODUCING_KINDS = coding, research, benchmark, security_scan.
`review` stays exempt — a reviewing phase that changes nothing has done its job,
the same distinction `vm_stop_gate::per_node` makes. The test that encoded the
old rule is rewritten rather than deleted, with the reasoning that replaced it.
All 8 harness fixtures are coding phases, so harness behaviour is unchanged.
2. PDFs are dropped; markdown is the deliverable.
Rendering a PDF meant asking an LLM to convert markdown to HTML — a paid API
call per document, on the critical path of "let me read my research", which
failed on depleted Gemini credits and left every artifact unreadable. Styling at
render time is free, offline, instant and cannot 429.
- `mission_outputs` no longer requests a render.
- New `GET /api/missions/{id}/artifacts/{artifact_id}/content`. The frontend had
no way to READ an artifact at all: it listed paths and offered a PDF preview
that never rendered (and whose `rendered_pdf_path` had no route serving it).
Two containment rules, both enforced: the artifact must belong to a mission in
the caller's workspace, and the CANONICALISED path must stay under `_outputs`
— canonicalise first, because checking the string before resolving `..` is the
classic hole.
- `MarkdownBlock` now uses react-markdown + remark-gfm + rehype-slug. It was a
deliberate zero-dep renderer for "the subset the refiner emits", and that
subset stopped matching reality: agent briefs are largely GFM pipe tables,
which it showed as literal pipes. MissionOutputReader and RefineDiffModal use
the same component and gain tables for free.
- Heading ids come from rehype-slug and `outlineOf` slugs with the same
GithubSlugger, so the outline rail's anchors still resolve. A test pins that
invariant, including duplicate headings.
Styles live in globals.css under `.md-view`: the markup is generated so there
are no class hooks, and this project has no styled-jsx registry — the app-router
requirement is documented in next/dist/docs/01-app/02-guides/css-in-js.md, which
frontend/AGENTS.md exists to make me read.
The artifacts tab moved to `MissionArtifacts.tsx`. MissionCanvas was 1341 lines
against a 1250 limit BEFORE this change — already failing lint; it is now 1248.
238 backend lib tests, 20 backend test binaries, 89 frontend tests, clean tsc,
clean eslint on every file touched, production build succeeds.
|
||
|
Omar Sobh
|
ceab28b902 |
fix(missions): a repo-less mission threw away everything its agents wrote
`capture_finished_coding_phases` selects `AND m.repo_id IS NOT NULL`. Every
`research_only` mission is repo-less by design (`requires_repo = false`), so the
whole capture path — including the `sync_out` that copies the agent's work OUT
of the container — never ran, and the container was reaped unread.
Measured on the real mission `019fdc35` ("ClawHDF5 Research"): four agents, 9.5
minutes, EIGHT research documents — an HDF5 parser design, a Rust ecosystem
survey, a seven-crate dependency map, tracing and fuzzing strategy. Result:
`mission_artifacts` = 0, mission `completed`. Not recoverable: no container, no
volume, nothing under the missions root.
The platform did not merely fail to save the work — it INSTRUCTED it. The task
preamble tells every agent "/mission/repo ... is the mission's git checkout",
whether or not one exists, and the research directive says to save findings
there. One agent recorded the contradiction verbatim: "No git repo — file is
written." It looked, saw no repo, complied anyway.
Three changes, one per link in that chain:
1. `mission_outputs::capture_repo_less_phases` — copies `/mission/repo` out of
the container and registers each file as an artifact under `_outputs/`,
which is a SIBLING of the mission dir and survives `teardown_container`.
This is also the code that finally reads `produces`, until now an inert key:
`produces = ["md","pdf"]` now drives `render_pdf` into the existing
pdf_renderer worker.
2. The preamble is conditional. A repo-less mission is told its workspace is
scratch, that git_operations has nothing to act on, and — the part that
matters — that files left there ARE collected and published. An agent told
only "there is no repo" has no reason to write anything to disk.
3. A repo-less phase that produced no files is FAILED, unless it declares
`allow_empty`. The same rule `empty_delivery_is_a_failure` applies to coding,
for the only channel these phases have. Note this is NOT that guard widened:
it keys on `files_changed`, which is meaningless with no checkout, and would
not have saved the ClawHDF5 documents.
Negative controls, each ablated and confirmed failing: ignore `has_repo` and the
preamble test fails; empty the skip-list and the capture test keeps `.git` and
`node_modules`; write artifacts inside the mission dir and the survives-the-reap
test fails.
236 lib tests pass.
|
||
|
Omar Sobh
|
b36ae00ea5 |
fix(missions): a gate that gave up completed the phase green
`done_when_check` is run in exactly one place: the Stop hook inside the guest. Nothing outside it has ever re-run the command — not the evaluator (which judges the PROSE `done_when`), not capture, not delivery. The hook is capped at MAX_BLOCKS so a stuck agent cannot wedge the turn. At the cap it logs `cap: <reason>` and exits 0, releasing the agent with its check still failing. That release was invisible: rc was 0 and the work collected, so both signals the run status was decided from said "fine", and the phase completed. Green phase, unmet condition, no error anywhere — the same silent-success shape this project keeps paying for. The block COUNT cannot fix it. Three blocks then a stop that finally passed and three blocks then a surrender both report `blocks: 3`, and they are opposite outcomes. So the gate now writes a `capped` marker file, probed back out of the guest alongside the block count, and `Some(true)` fails the run on BOTH paths — solo (phase_runner) and composed (microvm_turn_executor). A marker file rather than grepping the log: a block reason embeds the check's own output, so an output line starting `cap:` would read as a release that never happened. Also corrects the comment in `per_node` that sent me looking. It claimed "the phase-level check still runs post-hoc", conflating two mechanisms — that is true of `require_changes` (via `empty_delivery_is_a_failure`) and was never true of `check`. Negative controls, both ablated and confirmed failing: drop the enforcement and `a_node_whose_gate_gave_up_fails_the_run` fails; stop writing the marker and `a_gate_that_gives_up_records_that_it_gave_up` fails. And the control against over-strictness — `a_node_that_was_blocked_and_then_succeeded_passes` — is why this keys on the marker instead of the count. 231 lib tests pass. |
||
|
Omar SobhandClaude Opus 5
|
d24823b6f3 |
fix(missions): a failed phase stranded its mission at running forever
Found by counting containers during a cleanup, not by a test. gw-04 was holding
a per-mission runtime container for a mission whose only topology run had failed
three days earlier — phases `pending,failed`, mission still `running`.
The interaction, which lived entirely between two queries' predicates:
`start_pending_phases` launches a phase only when EVERY lower-order phase is
`completed`, so once one fails the phases after it can never run. They stayed
`pending`. `close_finished_missions` closes a mission only when NO phase is
outside ('completed','failed','skipped') — so a `pending` phase that would never
run kept the mission `running` indefinitely. And `mission_runtime`'s sweeper
fires N minutes after a TERMINAL state, so the container was never reaped.
One leaked container per failed multi-phase mission, accumulating silently, with
nothing in any log saying so. Neither query is wrong alone; the bug is that
nothing marked the phases the failure had made unreachable.
`skip_unreachable_phases` says it: a `pending` phase with a `failed` phase at a
LOWER order_idx becomes `skipped` — strictly earlier, because order is what makes
a phase unreachable, and a failure later in the list says nothing about one still
queued ahead of it. `skipped` is not a new concept: `close_finished_missions`
already treats it as terminal, and it is the honest word for a phase that was
never run, as distinct from one that failed.
RETRY HAD TO MOVE WITH IT, or this trades one bug for another. `retry_phase`
required the mission to be `running`, so closing failed missions would have made
the one outcome you would actually want to retry the one you could not. It now
accepts `failed` too, and in one transaction: resets the phase, REOPENS the
phases its failure had skipped (without that, a retry runs the phase and stops,
because everything after it is terminal-by-skip), and puts the mission back to
`running` — every launcher and closer keys off that status. `completed` and
`cancelled` stay refused; reopening those is a different decision.
557 tests pass, clippy clean. Three DB tests against real SQL, including that a
phase queued BEFORE the failure is untouched and that a draft's phases are never
swept.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
6991e21f94 |
feat(missions): the completion gate, moved into the agent's own loop
Every check this platform makes on a phase runs AFTER the agent has stopped: the
evaluator judges `done_when`, capture notices a coding phase delivered nothing,
and either verdict costs a whole new VM — a fresh boot, a fresh inject, and an
agent starting over with none of the context that got it that far. Meanwhile the
documented failure mode of a long-running agent is that it stops too early.
MEASURED FIRST, because the plan's chosen seam does not exist here. Probing every
hook name under `claude -p` (2.1.222, hermetic `--settings` file): `SessionStart`,
`UserPromptSubmit`, `PreToolUse`, `PostToolUse`, `SubagentStop` and `Stop` fire;
`TaskCreated`, `TaskCompleted`, `TeammateIdle`, `SessionEnd`, `Notification` and
`PreCompact` do not. The agent-teams hooks Slice 3 deferred are inert on our path
BY CONSTRUCTION — no team forms in print mode at all — so `done_when` could never
have been wired through `TaskCompleted` exit 2. `Stop` is the seam.
`vm_stop_gate` generates a POSIX `sh` hook installed via `--settings`, under
`/root/gate` and never under `/mission/repo` (anything there is collected and
arrives in the user's delivered patch). It refuses a stop when:
- the phase must deliver and the repository is untouched — asked as TWO
questions, since an agent that committed leaves a clean tree and an agent
that did not leaves HEAD alone; only both together mean nothing happened;
- `config.done_when_check` — a command the phase author wrote — exits nonzero,
in which case its OUTPUT is the feedback, not just "the check failed".
Deliberately mechanical. NOT the `done_when` verdict: that is an LLM judgement
made host-side by a different provider on purpose, and re-running it inside the
VM would put the agent's own environment in charge of grading the agent — the
correlated failure the independent judge exists to break.
THE CAP IS LOAD-BEARING. Without a ceiling a stuck agent is blocked, retries, is
blocked again, and burns the hour-long turn budget instead of failing visibly.
After 3 blocks the gate lets it stop, records that it gave up, and leaves the
verdict to the existing post-hoc path, which is unchanged.
PROVEN AGAINST A LIVE AGENT with the REAL generated artifacts, not a paraphrase:
- a read-only task → blocked 3 times with our exact message, released at
exactly the cap, and the agent took the escape hatch the message offers
("if the task genuinely requires no code change, say so explicitly") rather
than touching a file to satisfy the gate. It did not Goodhart it.
- a task that needs an edit → `blocks: 0`, log says `pass`. No false positives.
Two things that could fail silently, both closed. `--settings` is PROBED in the
image before use (`claude --help | grep`), because an unknown option is a hard
CLI error that would turn every gated phase into a failed one; a build without
it degrades to ungated and says so, since losing a check is better than losing
the work. And `stop_blocks` is reported out of the guest — `None` for no gate,
`0` for got-it-right-first-time — so a gate that never fires is distinguishable
from one that was never installed.
`require_changes` does NOT apply per node on a composed run: a graph's verifier
node is SUPPOSED to leave the tree alone, and a per-node gate would refuse its
stop three times for doing its job. `StopGate::per_node` drops it and keeps the
declared check. The phase-level rule still runs post-hoc against what the last
node collected.
An ungated phase's command is byte-identical to before, asserted by test — most
phases are gated, so the ungated path is the one nobody would notice breaking.
517 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
12147a1e01 |
feat(missions): Slice 4 — the two engines composed, with the file handoff proven
`team_engine='composed'` (the third name migration 0069 anticipated) runs a
mission as a durable ZeroClaw graph whose every node is a whole
Claude-Code-in-a-microVM session. Engine Z owns checkpoint/resume, cancellation
and per-node heterogeneity; Engine C owns shared context and cheap fan-out;
neither has the other's asset, which is why this is a composition and not a
compromise.
`MicroVmTurnExecutor` implements the existing `TurnExecutor`, so it inherits the
planners, the checkpoint, the stale-run recovery, `close_finished_phases`, the
evaluator, capture and delivery unchanged — the same trick `SubTopologyExecutor`
already plays with a heavy `run_turn`. Producer side emits ONE `queued` row
carrying the real graph and lets the worker claim it: the durability IS being
worker-driven, and the solo path's `tokio::spawn` has none of it. Still exactly
one `topology_runs` row per unit of work and one completion path — `finish()` is
now that one place, shared by every tier.
THE TRAP, solved and proven. A VM is inject → run → collect → destroy, so a
per-node VM with text-only handoff silently loses every file an earlier node
wrote: node 2 boots from the original checkout, sees nothing, and still reports
success. The mission's host checkout is the medium — every node injects from it
and collects back over it — and two properties make that safe rather than lucky:
`execute_resumable` is strictly sequential, so two VMs never write one directory;
and the vm id is deterministic per (phase, iteration, step), so a duplicate is
refused by the node ("vm already exists") instead of becoming a second writer.
NEGATIVE CONTROL, run rather than assumed: with `repo` swapped for a private
per-node workspace, `a_later_node_sees_an_earlier_nodes_files` FAILS with
`saw:[]`; restored, it passes. The `PhaseVm` seam exists for exactly this — it
models inject/collect through the real `mission_fs` tar path in milliseconds.
Two durability traps this tier walks into, both closed:
- `requeue_stale` fires at 180s on `updated_at`, and one node here can run for
an hour. `SubTopologyExecutor` keeps its parent alive from each leaf step;
there is nothing between the start and end of a VM turn, so the turn holds a
ticker that touches `updated_at` every 30s and aborts on drop. Without it a
healthy composed run is requeued mid-node and boots a second VM.
- the 15-minute stuck-run reaper asks "any step records since it was CREATED?",
which describes a healthy composed run as readily as a wedged one. Hence
`REAPABLE_TIERS` — worker-driven minus this tier. Reaping it would be #54 in
a different costume.
`on_launch` mints no team for a microVM mission, deliberately: claws in
containers are what a VM mission does not use. So `mission_orchestrator::
composed_graph` builds the shape from the team template directly — nodes, roles
and pattern, zero claws provisioned. Per-node `attrs["backend"]` and
`attrs["node_id"]` override the mission's, which is what makes a validator node
on another provider's image a first-class graph node; a malformed `node_id`
fails the node rather than quietly running it where the graph did not ask.
Refusals are recorded as a failed run, not returned as an error: `launch_phase`
is swept every ten seconds, so a returned error is a phase that retries forever
while the log repeats itself.
501 tests pass, clippy clean. NOT yet proven end to end: no composed mission has
run on the fleet, so the resume-after-a-killed-worker leg is argued from the DB
test and the step-numbering test, not from a real two-node run.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar Sobh
|
4efcde9d4f |
fix(missions): #54 — the worker was killing live microVM runs at 180 seconds
My hypothesis in #54 was WRONG, and it was wrong because I built it on a bad measurement: `grep -c 'microvm phase'` returned 0, so I concluded the completion log never printed and blamed the 15-minute reaper. The line was there all along, at 14:17:45. The real cause is worse. `requeue_stale` has NO TIER FILTER. A microvm run's `updated_at` is written once at insert and never again — it is driven by a `tokio::spawn` that owns it start to finish, and nothing in `microvm_executor` writes `topology_runs`. So at 180s the sweeper declared a perfectly healthy run stale and flipped it to `queued`; `claim_next_queued` (no tier filter either) handed it to the worker; `run_job` tried to parse the microvm graph placeholder, which `TopologyGraph` cannot deserialize; and it failed the run with "missing or invalid graph". Mission 019fd43e: run created 14:11:16, mission failed ~14:14:46. 210 seconds — the 180s window plus a tick. The agent went on working and finished at 14:17:45 with three modules written, by which time the phase was already dead and the VM was orphaned. A firecracker process was still alive 1h37m later. THE UNCOMFORTABLE PART: every microVM mission that appeared to work this session did so only by finishing inside three minutes. The 90-second ones dodged this. The harness scenario dodges it. Nothing about that was visible. `WORKER_DRIVEN_TIERS` (team, company, org, swarm, compare) is now the allowlist for all three sweep paths — claim, requeue, reap. An allowlist rather than a denylist so the next self-driven tier is safe by default instead of exposed until someone remembers the file. `tier='session'` had exactly the same exposure and is covered too. A unit test asserts microvm and session are NOT in it, next to the code that inserts them. Two more fixes from the same wreckage: - `destroy` reported `killed: pgid.is_some()` — true whenever there was a pgid to signal, whether or not anything died. It now sends the signal, polls /proc for the group leader, retries, and reports what it OBSERVED; `signalled` keeps the old meaning so "nothing to kill" is distinguishable from "it would not die". - the run-status update is now guarded with `AND status <> 'cancelled'`. An operator cancelling is a decision; this task reporting an outcome minutes later is an observation, and it must not overwrite one with the other. And the root cause of the collect timeout itself: `mission_fs::pack_dir` shipped `target/` in both directions. `mission_delivery` has excluded build output from the DIFF since day one; the TRANSPORT never knew. The host checkout was 9.4 MB of which 8.9 MB was `target/`, tarred and base64'd over vsock each way. `EXCLUDED_PATHS` is now one list shared by both layers, matched on directory name at any depth so a workspace's per-crate `target/` dirs are all covered. 483 tests pass, clippy clean. |
||
|
Omar Sobh
|
cb48f7ff3b |
feat(missions): Slice 3 — agent teams behind a per-mission switch, solo by default
`missions.team_engine` (0069): NULL = solo, `'claude_code'` = Claude Code agent teams inside the mission's VM. Solo stays the default deliberately — Anthropic measure multi-agent at 3-10x the tokens with wall-clock often LONGER, since the benefit is thoroughness rather than speed — so a mission that said nothing does not get a team. In-process teammates live in the lead's process, so ONE VM hosts the whole team. That is why this is a prompt-and-env change rather than an orchestration one: no N-VM fan-out, no placement per teammate, no new completion path. The lead decides its own team size and there is no flag that limits it, so the cap (4) is stated in the prompt. The addendum also carries the two anti-patterns from Anthropic's guidance, because they are exactly the shapes our pipeline templates have: teammates must own DIFFERENT FILES (two in one file overwrite each other), and one change must not be split into stages across teammates (a handoff loses context at every step). And: wait for your teammates — a summary written before they report is the lead's own guess. A solo mission's prompt and env are byte-identical to before this change. That is enforced by test, not by intention: the comparison between solo and team is only meaningful if the solo side did not also move. Evidence, because a team mission that forms no team is silently just a solo run that looked fine and spent fewer tokens: a second probe counts members in `~/.claude/teams/*/config.json` (minus the lead), reported separately from the subagent count, and a team mission with zero teammates logs loudly with the two likely causes. The teammate path is DOCUMENTED BUT NOT YET VERIFIED in our image, unlike the subagent transcript path which was measured — so a zero there means "no evidence found", and the first real team mission is what turns it into a fact. `Option<u32>`: None means no team was asked for or the probe could not run. Hooks (`TaskCompleted` / `TeammateIdle` exit 2, which would move `done_when` from post-hoc into the agent's own loop) are the highest-value part of this slice and are deliberately NOT here — they deserve their own pass rather than a rushed tail. 482 tests pass, clippy clean. |
||
|
Omar Sobh
|
9aed20b6d0 |
fix(missions): capture a failed phase's work; harness gains a microvm scenario (#51)
A REGRESSION I INTRODUCED ONE COMMIT AGO. `capture_finished_coding_phases`
selects on `mp.status = 'completed'`, so the moment an unmet phase correctly began
reporting `failed`, its diff stopped being captured, committed or pushed — the work
was silently discarded. Found by the new harness scenario, whose phase legitimately
missed its condition and then had no artifact at all.
What was produced, and whether the goal was met, are different facts. The artifact
records the first; `mp.status` records the second. Capture now covers terminal
phases (`completed`, `failed`), so a phase that did real work and missed its goal
still delivers a reviewable diff — which is exactly what the next pass needs.
`scripts/verify-mission-delivery.sh microvm` — the regression net this session was
missing. Everything the microVM track proved by hand was guarded by nothing:
- THE KERNEL LINE is the assertion that cannot pass by accident. Every other
check would also pass if the phase had quietly run in a container on the
gateway; only the kernel says WHERE it ran. Compared against the real gateway
and node kernels read at start-up rather than pinned to a version, so
upgrading vmlinux does not manufacture a failure.
- subagent count > 0, from the server's own count of Claude Code's per-subagent
transcripts. Before `Agent` was in the allowlist this was structurally
impossible and nothing said so. A probe that could not run reports "?" and
FAILS the check rather than reading as zero.
- the verdict's judge and whether it was independent.
- negative control, observed passing: a mission whose backend no node can run is
refused at launch and stays draft. Without it the positive scenario would pass
just as well against a scheduler that ignored `backend` entirely — which is
what it did until the first real microvm mission landed on a node with no such
rootfs.
Also fixed in the harness: `api` now sends the JSON body on STDIN (`curl -d @-`)
instead of interpolating it into a single-quoted argument inside a double-quoted
ssh command. A task description containing "the crate's test suite" ended the
quoting and killed the remote shell; two attempts to escape it were themselves
wrong, because the backslashes must survive bash AND sed AND sh. Removing the
interpolation removes the class, and the next author does not need to know that
apostrophes were forbidden.
475 tests pass, clippy clean.
|
||
|
Omar Sobh
|
bb807c2f3a |
fix(missions): an unmet goal condition is no longer reported as success
Found by the Goodhart test for the independent judge, which is exactly what it was built to find. The test: a phase whose `done_when` demanded a passing suite, and a task that deliberately left a failing test. glm-4.7 judged it, ran `cargo test` itself, saw `parity_is_wrong_on_purpose ... FAILED` (exit 101), and returned met=false quoting the assertion — while the agent's own summary said "All three steps are implemented exactly as specified and independently verified". The verdict and the agent's account diverged, which is the whole point of an independent judge. And then the mission closed `completed`. `if verdict.met || last_pass` marked BOTH outcomes completed, so a phase that ran out of passes without ever meeting its condition reported success — and through `close_finished_missions`, so did the mission. The verdict said met=false in a column nobody reads before believing a green status. Anything consuming mission status rather than digging into the verdict saw a goal that was never reached as a goal achieved. Exhausted-and-unmet is now `failed`, and the log names the judge and whether it was independent. This changes observable behaviour: missions that would previously have finished green with an unmet condition now finish failed. That is the correction, not a regression — but it is worth knowing before the next scheduled run. Also: `Verdict.independent` had no column. The field existed in the struct and in the logs, so the audit question the mechanism exists to answer — was this checked by something other than the model that wrote it? — could not be asked of the database. Migration 0067 adds it, defaulting to false, which is the truth about every row written before now. Verified in production before the fix: glm-4.7, 4 checks all executed, the real cargo failure quoted, met=false. 475 tests pass, clippy clean. Note for whoever rebases: `sqlx::migrate!` embeds migrations at COMPILE time, so a new migration needs cm-db rebuilt (`touch crates/cm-db/src/lib.rs`) or the integration tests fail on a column that exists in the file and not in the binary. |
||
|
Omar Sobh
|
2dee941080 |
feat(missions): Slice 1 — a microVM agent can delegate, and we can see that it did
`microvm_executor` passed `--allowedTools Read Edit Write Bash`, which omits the
`Agent` tool, so Claude Code could not spawn a single subagent in any of our VMs.
The tool existed, the model knew how to use it, and the allowlist quietly removed
the ability. Nothing in any output said so.
Now: `Agent` in the allowlist, two roles supplied as `--agents` JSON, and a probe
that counts what actually ran.
Roles are JSON on the command line, not files, because `/mission/repo` is
collected and diffed — a role definition written into the checkout would arrive in
the delivered patch as if the agent had authored it.
Two roles only, and the choice is the research talking:
- `verifier` — the one multi-agent pattern Anthropic endorses for coding work.
It gets Read/Grep/Glob/Bash and deliberately NOT Edit or Write: an agent that
can fix what it is checking will fix it and report success, and the report is
then about a tree nobody reviewed. Its prompt demands the COMPLETE suite,
which is the counter to the "early victory problem" — the same failure as our
own Goodhart incident.
- `explorer` — context protection, read-only.
Roles like "tester" or "committer" are absent on purpose: splitting sequential
phases of the same work is a named anti-pattern, and it is the shape our pipeline
templates already have.
THREE THINGS THE IMAGE CORRECTED, none of which review would have caught:
1. `CLAUDE_AGENT_SDK_DISABLE_BUILTIN_AGENTS=1` (in the plan) removes EVERY agent
type, including the ones `--agents` defines. Measured: the lead reported "an
empty available-agents list" after trying four role names and — to its credit —
refused to fabricate a subagent result. Worse, the unit test asserting
"builtins off is paired with our own roles" PASSED throughout, because the
pairing holds in our code and not in the CLI. Dropped, and the test rewritten
to assert only what a unit test can speak to.
2. `--forward-subagent-text` refuses to run without `--output-format=stream-json`,
which would change how this module reads output. Dropped.
3. `--append-subagent-system-prompt` does not exist in 2.1.220 despite being
documented. The anti-shortcut rule is inlined per role instead — better anyway,
since a verifier and an explorer need different wording.
Evidence instead of assumption: Claude Code writes a per-subagent transcript at
`<session>/subagents/agent-*.jsonl`, so the guest is asked to count them before
collection (they live in /root, outside the collected tree). `VmOutcome.subagents`
is `Option<u32>` and the phase log prints it: `None`/"?" means the probe could not
run, which is a different fact from "delegated to nobody" and only one of those is
about the agent.
Verified in a container against the real CLI on tank before any of this shipped:
`FANOUT-OK`, a subagent transcript on disk, and zero errored Agent calls.
469 tests pass, clippy clean.
|
||
|
Omar Sobh
|
0a9747091f |
fix(missions): a microvm mission needs no team, and two checks required one
Found by running one: the mission was created with runtime_kind='microvm' and then refused to launch with a bare 400, because draft→running requires a materializable team. Past that, `launch_phase` returns early when a phase has no matching teams — so even with the launch allowed, the phase would have sat `pending` forever while the log said only "no matching teams", and the executor would never have been reached. Neither check applies to this path: microvm_executor runs the agent CLI directly in the VM, so there is no claw graph to materialise. Satisfying the checks by attaching a team template would have provisioned claws that never run. The repo checkout still happens — the VM needs the repository. 461 tests pass, clippy clean. |
||
|
Omar SobhandClaude Opus 5
|
4f07430e92 |
feat(missions): B4.5 — phase_runner runs a microvm mission in a VM
`runtime_kind='microvm'` placed a mission on a KVM-capable node and then nothing
executed it: config accepted without a reader, one of the four seams this project
keeps closing. This is the reader.
`microvm_executor` — inject → run → collect → destroy, the shape copy mode
already proved for containers with a VM boundary instead of a namespace one. The
checkout goes in as a tar, the work comes back as a tar over the SAME host path,
so `mission_delivery::capture_phase_diff_at` needs no change at all.
The agent is told NOT to push, unlike the container path's session prompt. Two
reasons: delivery is already host-side and diffs the collected tree against the
recorded clone point (covering committed, staged and unstaged work in one pass),
so pushing would add a second untested way for work to arrive; and pushing would
mean forge credentials inside the VM, when the point of collecting is that the
guest never holds them.
Exactly ONE topology_runs row (tier='microvm'), mirroring launch_direct_session:
close_finished_phases, evaluation, capture and delivery all key off those rows,
and a second completion path would be a second way for a phase to finish with one
of them untested. The row and the phase flip happen BEFORE any fallible VM work,
so a missing token or a node that lost its capability shows up as a failed run an
operator can see — not a phase that stays pending and retries every ten seconds.
Fail-closed points, each the reader for a guarantee built earlier:
- credentials resolve BEFORE the VM boots, so a missing subscription token
fails the phase instead of booting a VM whose agent sits unauthenticated
- a VM reporting egress:false is REFUSED, which is what makes create's
egress/egress_host/egress_guest fields more than decoration — a turn without
egress does not fail, it hangs
- the injected checkout is PROVEN present in the guest before an agent turn is
spent on it; an inject that reports success while landing nothing would
otherwise become an agent reporting an empty repository
- work is collected even when the agent exits non-zero — a turn that failed
partway still wrote files, and a retry needs to see them
- a turn that ran but could not be collected is a FAILED phase, not a happy one
- destroy runs on every exit path, or an 8 GB sparse rootfs leaks
Two integration gaps found while wiring, both of which would have produced a
mission that completed having delivered nothing:
- `capture_finished_coding_phases` pulls work out of a CONTAINER before
capturing. A microvm mission has none, so the docker connect would fail, the
loop would `continue`, and capture would be skipped forever while the phase
sat marked completed. Its work is already collected by the executor.
- `launch_phase` provisioned a runtime container, copied the checkout into it
and wrote a runtime binding + pairing code describing a runtime nothing uses;
and the orchestrator's workspace pin — deliberately FATAL — would have failed
a microVM launch on a container it was never going to use.
461 tests pass, clippy clean.
NOT YET PROVEN END TO END: no mission has run through this path. The pieces under
it are each verified on tank (image, credentials, egress, a real agent turn), but
this executor has only been compiled and unit-tested. Deploy + one real microvm
mission is the remaining step.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
da3731d753 |
fix(missions): a coding phase that delivers nothing is a failure
The last open item in the silent-success class: a coding phase that changed no files reported `completed` — the same status a phase gets for delivering tested, reviewed, pushed work. Mission `019fcf62` completed that way with its agents silently unpinned from the repo, and nothing in the platform disagreed; it was found by a script diffing the forge. The verdict is applied at capture rather than at completion, because capture selects on `status = 'completed'` — the platform does not know whether a phase produced anything until after it has already finished. Three conditions must hold before failing a phase, because a false positive here fails honest work: the phase is a coding phase (research phases legitimately write nothing to the tree), the diff was actually computed (an uncomputable diff also reports zero files — blaming the agent for a platform fault is the same defect wearing different clothes), and `allow_empty` is not set. Only an explicit `true` opts out, so a typo leaves the check armed. Registered in phase_config with its reader named, per the seam-2 rule. Also closes an ordering hazard this exposed: capture is batched and runs after a phase completes, so a backlogged mission could close as 'completed' and only then have capture discover an empty phase — leaving a 'completed' mission holding a 'failed' phase, unfixable because the mission-close CASE only touches 'running' rows. A repo-bearing mission now waits for its work to be captured before closing. Adds a `noop` scenario to the harness: a phase told to change nothing, which PASSES only when the phase comes back `failed`. Same discipline as the uid self-test — a check that has never been seen to fire has not been shown to work. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
7e07c389c6 |
feat(missions): wire copy-in/copy-out behind CLAWMATES_MISSION_FS=copy
With the flag set, ensure_container omits the /mission bind, the checkout is pushed into the container at phase launch, and the agent's work is pulled back before capture. The simplification that makes this small: sync_out unpacks over the SAME host path the checkout came from. The host directory stays a server-owned staging area with exactly one writer, and capture_phase_diff_at needs no change at all — it still finds a normal checkout exactly where it always has. Delivery, gating, commit and push are untouched. Two failures are deliberately loud rather than silent: - copy-IN failure fails the phase launch. Continuing would start a phase against an empty directory, and the agent would cheerfully report having done work on a repo that was not there. - copy-OUT failure SKIPS capture. Capturing anyway would diff a stale host tree and record "no changes" for work that exists — success reported for nothing, which is the exact failure mode this codebase keeps paying for. Opt-in: the bind path is what production has run since the beginning, and the test asserts a near-miss value leaves it there rather than silently switching every mission. 414 tests, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
758b2dbd96 |
feat(missions): run a phase as one direct session, behind a flag
CLAWMATES_MISSION_EXECUTOR=session makes launch_phase run the whole phase as a single `claude -p` against /mission/repo instead of driving turns through ZeroClaw. Opt-in, because silently changing how every mission executes is exactly the sort of change that should require someone to have typed it. It still writes ONE topology_runs row. The entire downstream lifecycle -- close_finished_phases, evaluation, capture, commit, gate, publish -- keys off those rows, and inventing a second completion path would mean two ways for a phase to finish with one of them untested. The session is simply a run with tier='session' and an empty graph. Spawned rather than awaited: launch_phase runs inside the sweep loop, and blocking it for the length of a coding session would stall every other mission. The session's own summary is logged as diagnostics only. Whether the phase actually did anything is still decided downstream by capture and delivery against the repository -- a 0-exit session that pushed nothing was measured at ~5%, so the agent's account can never be the verdict. 406 tests, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
ec85f6c8da |
fix(missions): close the three seams behind this run of failures
Seam 1 — delivery inferred checkout state from the tree, so whether work survived depended on what the agent happened to do. 019fc444 committed and left a clean tree; 019fc476 had its base advanced to match HEAD; 019fc450 survived only because a phase FAILED to commit and left the tree dirty. Same code, opposite outcomes, decided by the agent. mark_phase_started records the fact at phase launch, before the agent acts, so every one of those states answers identically. The tree checks remain as a second line of defence for pre-existing checkouts. Seam 2 — phase config was accepted, stored and read by nobody. That was `task`: every phase of every mission got identical instructions. The new phase_config registry names the reader for each live key and lists the eight that are declared-but-unimplemented, reporting both at mission creation so an author sees what will not happen. Its CI test found one I had missed: security_hardening.toml sets phase-level mcp_bundles asking for gitea_forge + security_scan, but bundles come from the TEAM template and the phase gets neither. Seam 4 — push_url_for collapsed a failed query, an unbound repo and a missing clone_url into one None, so a database fault was recorded as "nothing to push to" and metadata read `pushed: null, push_error: null` — the same ambiguity commit_error already fixed. Each case now carries its reason into the artifact, and a local git failure during publish is recorded rather than dropped by .ok(). Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
8bad869248 |
fix(missions): give each phase its own task and its own capture base
The first push run against a scratch repo (mission 019fc42b) delivered two branches correctly but exposed two bugs behind them. Per-phase instructions were inert. `phase_task_text` took only (kind, title, description), so `mission_phases.config.task` was accepted by the API, stored, and read by nothing. Every phase of a mission received byte-identical text differing only by the kind directive — so both coding phases did the whole mission instead of their slice, producing the same two files. The task now reaches the agent as a trailing THIS PHASE'S TASK block, scoped against the shared brief. The capture base never advanced. `.git/clawmates-base` is written once at clone time, so phase two diffed against the original clone point and reported the union of both phases' files as its own. It now moves to each phase's committed head after the patch is on disk; the pushed branch stays cumulative because it is built from HEAD. Both regression tests were confirmed to fail with their fix disabled. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
e089360ac8 |
fix(missions): unblock the capture batch, and restore fetch auth
Two defects, both found by running a second real coding mission (019fc3ba)
after the first round of fixes. The agent created the file correctly this
time — `file_write` did its job — and capture still produced nothing.
**Head-of-line blocking.** `capture_phase_diff` returns `Ok(None)` when the
checkout is gone, and the caller treated that as success without recording
anything. The phase therefore stayed eligible forever, and because the batch
is bounded at five, five reaped phases from earlier test missions occupied
every slot permanently. A freshly finished coding phase, with its checkout
still on disk, was never reached — and nothing was logged, because nothing had
failed.
Fixed on both axes: an unreachable checkout now writes a `code_diff` marker
recording `captured: false` and why, so the row stops being selected; and the
batch orders newest-first, so live work is captured before archaeology. The
marker also distinguishes "this phase changed nothing" from "we lost the
checkout before looking", which an operator reading the mission needs to be
able to tell apart.
**Fetch lost its credentials.** `scrub_remote_credentials` (P1.1) strips the
token from `.git/config` so agents running as root cannot read it — but
`fetch_and_reset` fetched from the stored remote, which is now anonymous:
git fetch origin <branch> → exit 128:
fatal: could not read Username for 'https://git.redclaw.dev'
I accounted for push building a fresh authenticated URL and overlooked that
fetch needs one too. `fetch_and_reset` now takes the authenticated URL the
caller already computes, as does the `--unshallow` deepen. Stderr stays
redacted.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
409ca65ee7 |
fix(missions): capture from the clone point, and let agents create files
Two defects found by running a real coding mission (019fc372) rather than a test. Both made a coding phase look like it produced nothing. **Capture measured the wrong baseline.** It diffed the working tree against HEAD, which is correct only while work stays uncommitted. `rust_sdlc` has a *committer* role, so committing is the intended path — meaning a mission that did its job properly leaves a clean tree and captured nothing. That is exactly what happened: the agent created `DELIVERY_PROBE.md`, committed it as `aa3be95`, and the artifact recorded `empty: true` beside a commit that plainly contained the work. `mission_workspace` now records the clone point in `.git/clawmates-base` (in `.git/`, so it travels with the checkout, stays invisible to the repository, and cannot be reached by an agent through its pinned workspace), refreshed whenever `fetch_and_reset` moves HEAD. Capture diffs from there, covering committed, staged and unstaged changes in one pass. Checkouts predating the marker fall back to HEAD and say so via `base_recorded: false`. **Agents could not create files.** `coding_readwrite` granted `file_edit` but not `file_write`. `file_edit` replaces an exact existing string and rejects an empty `old_string`, so creating a new file was impossible. The mission transcript is unambiguous: "the tool rejected empty old_string... the shell is restricted", after which the agent worked around it through `shell`. The comment above that profile has claimed it grants file_write since the day it was written; the list never contained it. Also broadens capture from coding/benchmark/security_scan to every phase kind of a repo-bearing mission: `phase_task_text` tells research phases to "save findings under /mission/repo/research/", so filtering by kind would have discarded every research brief such a mission produced. Regression tests cover committed-only and committed-plus-uncommitted work against a real git repo. Co-Authored-By: Claude Opus 5 <[email protected]> |