0d46f892db65afd75b0fe8cfe7ac880fffbd8db1
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar SobhandClaude Opus 5
|
9fc904a056 |
feat(microvm): the tool gate's denials and inert marker reach the mission record
vm_tool_gate writes denied.jsonl for every call it refuses and an `inert` marker each time it cannot parse its input and lets the call through. The guest has written both since the gate existed; nothing read them out of a VM. A denial, or a gate that had quietly stopped checking, left no trace — the same shape the container tier closed with drain_inert on 09-14. The executor probes both files (one exec, while /root still exists) into VmOutcome.tool_gate; launch_microvm_phase records them on the mission as the container tier's `gate.inert` (with the count) and `gate.denied` (one event per refused call, the gate's own JSON as the detail). Absent gate is None, not zero — "no gate" and "a gate that refused nothing" are different facts. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz |
||
|
Omar SobhandClaude Opus 5
|
794f2124bc |
feat(microvm): Claude Code 2.1.276 rootfs pins, and a VM run that says what it ran
Every rootfs on the fleet had sat on Claude Code 2.1.223–2.1.226 since August while the container tier moved to 2.1.276, and nothing recorded either. GLM and Kimi exist only as microVM backends, so "have we upgraded GLM and Kimi" is this change and the rebuild it drives. Pins. All four agent-* images pin 2.1.276 — as separate ARGs, since Docker has no include and each file has to stay reproducible alone — and scripts/fc-build-rootfs.sh refuses to build if they disagree, naming the odd one out. They had already drifted (claude 226, the rest 223) under comments saying "same version on purpose". Between 2.1.226 and 2.1.276, 2.1.265 and 2.1.275 each broke every turn on ANTHROPIC_BASE_URL endpoints, which is how glm and kimi reach `claude` inside a VM; the container-tier verification never exercised that path, so the VM runs on those backends are the real test. Provenance. `VmOutcome` carries the rootfs the node reported booting and the guest's own `claude --version`; `launch_microvm_phase` persists both as `checkpoint.vm` beside `records` (the two readers parse only `records`) and names them in its log line. "Which image and CLI did this mission run on" is a query now. Independence. `evaluator` derived the implementer family from a constant `"anthropic"`, true while every backend was Claude on Anthropic. With glm and kimi rootfs it made a glm mission judged by glm:glm-5.3 read as `independent = true` — the one claim that path exists to make honestly. `implementer_family(missions.backend)` mirrors `microvm_credential_for`; the subscription judge is now independent exactly when the agent did NOT run on Anthropic. Harness. `verify-mission-delivery.sh glm|kimi` run the microvm scenario on each backend and add the proof the mission itself cannot give: the placed node's journal must show the VM dialling that provider's host, never being denied it, and dialling nothing else but the forge — a model's self-report is measured worthless here. `assert_cli_version` reads checkpoint.vm. The stale scratch-repo default (dead since the 09-14 wipe) is the re-synced id. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz |
||
|
Omar SobhandClaude Opus 5
|
483de9f88a |
feat(billing): agent-side spend records who was paid
Judge spend gained provider, model and mission on 2026-09-14; agent spend — the larger half — did not. The runtime's `done` frame has always carried `model` and `provider` beside the two token counts, and `topology_exec` read only the counts, summed them, and charged the sum as output with no record of which provider served the turn. `TurnOutcome` and `StepRecord` carry a `Spend` now (input/output split, provider, model), the worker passes it through `cm_billing::charge` along with the mission id, and the chat runtime records the model it requested — that loop drives one provider with no chain, so requested is answered. A bare model name is recorded without a guessed family. `StepRecord.spend` is `serde(default)` so journaled checkpoints from before this field still load, and `tokens` stays as the total every reader keys on. `charge` moved from `query!` to `query`: the macro pins the statement to offline metadata that a schema change then has to regenerate against a live database, for columns that are nullable text and uuid. The done-frame test now asserts the split and the provider survive, not just the sum. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz |
||
|
Omar SobhandClaude Opus 5
|
8591585e60 |
feat(missions): attribute a phase's tool calls to the agent that made them
`record_vm_tools` wrote `agent_id: None` for every call. The container tap is per-CONTAINER and every role in a phase shares one, so a phase arrived as one undifferentiated stream: every Skill-Use score was per-mission rather than per-role, and the World's per-agent view got nothing from this tier. One `claude -p` invocation is one turn is one agent, and Claude Code stamps each invocation with a `session_id` the tap was discarding. So the distinct sessions, in order of first appearance, are the phase's turns in the order they ran — and `prompt.composed` already records the agent of each turn in that same order, written by the tier as it sends each turn, so it IS the running order rather than a reconstruction of it. **It attributes nothing rather than guessing.** Only when the counts match exactly. A phase whose sessions and turns differ has something this correlation does not model — a retry, a turn that called no tool, two genuinely concurrent agents — and a plausible-looking wrong attribution is worse than none here: it puts one agent's `git push` on another agent's record, and a person later reasons from that. One call missing a session id refuses the whole batch, because a hole shifts every later session onto the wrong turn. The microVM call sites pass no turn agents and so keep today's behaviour exactly. Resolving a graph node to an agent uuid is the fix there, it cannot be tested while the fleet is offline, and guessing would put one node's actions on another node's record. Also restores the `#[cfg(test)]` gate on `repo_less_text_tests`, which my own insertion had taken — those tests would have compiled into release builds. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9 |
||
|
Omar SobhandClaude Opus 5
|
31158467f4 |
feat(viz): microVM tool motion is live, and needs no fleet-node change
The plan deferred this as "the only fleet-node binary change". It is not one. `fcagent` is thread-per-connection — its own comment says so, and the live log tail has relied on exactly that for the whole length of a turn, on a second connection. So the host can drain the tap WHILE the turn's exec is in flight, from the server alone. The turn and a 20s drain loop now run concurrently. A coding phase shows its files being touched as it works rather than an hour later, all at once, and the drain is bounded by a cursor so a repeated poll returns only what is new. The cursor counts LINES, not parsed events, and that distinction is the bug this commit would otherwise have shipped. The hook appends the event and then a newline of its own, so a two-event tap is four lines; advancing by event count leaves the cursor two lines short, `tail -n +N` hands back events already recorded, and the live drain re-records everything it has already written — worse the longer the turn runs, and silent throughout. Caught while writing the test, not by it. `tap_sink` and `VmOutcome::tools` are mutually exclusive by contract: with a sink, the sink owns recording including the final batch and `tools` comes back empty. Handing the same calls back on both would double every file orb's weight with no way for the caller to tell which it was looking at. The sink is an unbounded channel to a recorder task, so the VM executor stays free of the database: it observes, phase_runner records. The task ends when the sender drops with the phase. Verified before this change: the microVM tap is real. The `microvm` scenario passed 6/6 and left ten `tool.call` rows and a `file.touch` on MICROVM.md, repo-relative, from Claude Code's own PostToolUse hook. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
9e61e3ba35 |
feat(viz): what the agents actually did, as structured events
The World could draw a mission's shape but nothing about the work. The
detail existed only as prose in checkpoint.log and model output, where a
tool name is indistinguishable from an agent *talking about* a tool — so
it was never parsed, deliberately. `mission_events` is the structured
channel that replaces it.
Three taps, one table:
- Container tier: the `_ => {}` at the end of topology_exec's typed frame
stream now matches `tool_call` and reads the tool's JSON ARGUMENTS for a
path. Never the prose summary — a path scraped from a sentence would put
files on the map that no agent opened, and the test proves a Grep whose
summary says "src/main.rs" produces no file touch. The frame name itself
is unverified, so the same commit ships an unmatched-frame-type
histogram: a tap that matches nothing looks exactly like a mission that
used no tools, and this is how one gw-04 run names the real frame.
- microVM tier: a `PostToolUse` hook, the seam vm_stop_gate already proved
fires under `claude -p`. It copies stdin to /root/tap and exits 0
unconditionally — a non-zero PostToolUse hook talks back to the model,
which would turn the observer into a participant. Drained before collect,
since the VM is destroyed moments later.
- Phase transitions: five identical copies of the pending→running UPDATE
became one `mark_phase_running`, and `close_finished_phases` grew
RETURNING. Its CASE decides each phase's status inside SQL from rows the
statement does not change, so it cannot be re-derived afterwards without
writing that CASE twice — without RETURNING it emits zero phase.completed
and reports success.
The settings.json hazard the plan called out: the stop gate wrote the
WHOLE document, so a second hook writer would have silently erased it and
a coding phase would then complete having written nothing — the exact
failure the gate exists to catch. There is now one composer,
`vm_tool_tap::guest_settings`, one writer, and a source-walk test that
fails if anything else writes a settings document.
`mission_events.run_id` carries no FK on purpose: phase_runner DELETEs
topology_runs on retry, and a cascade would erase a phase's whole history
the moment it retried — silently, since a cascade is not an error.
world.rs streams it with a cursor that separates backfill from motion.
Everything already in the table when a subscriber arrives is drawn as
settled history; only what lands afterwards animates. Otherwise opening a
finished mission replays an hour of tool calls as a burst storm.
Bounded twice: 400 events per phase (enforced inside the INSERT, since
two concurrent taps would each read a count below the cap) and a 7-day
retention sweep in mission_gc.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
768e106614 |
fix(microvm): a mission with no repository can run in a VM, and its work comes back
Two halves, and the first was worse than the plan assumed. `run_phase_in_vm`
packed `<missions_root>/<mission>/repo` unconditionally — a directory a
repo-less mission does not have — and then required `/mission/repo/.git` inside
the guest before spending a turn. So a repo-less microVM phase did not merely
go uncaptured: it failed before the agent ran.
A repo-less mission now gets an EMPTY workspace at the same guest path, created
host-side so the collect unpacks back over it with no special case, and the
readiness probe asks for what was actually sent — the directory rather than a
`.git` that was never going to be there.
`mission_outputs` then drops its `runtime_kind <> 'microvm'` exclusion, whose
stated reason ("a microVM mission always has a checkout") is exactly what
stopped being true. Where the files come from now depends on the runtime, and
the difference is not cosmetic: a container mission's output is still inside a
running container, while a VM's has already been unpacked onto the host by the
end-of-turn collect. Asking docker for a VM mission's files would query a
container that never existed.
The recursive copy skips symlinks rather than following them — a link out of
the tree would publish whatever it points at.
`research-vm` is the proof, added to the suite as well as the dispatch: the same
assertions as `research-only` with `runtime_kind: microvm`. A scenario nobody
runs is a scenario that does not exist.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar Sobh
|
0b89b8316c |
feat(observability): stream a microVM turn's stdout/stderr to the platform live
The Live tab showed nothing while a turn ran, and the agent's own account of it
went to stderr on the node and nowhere a user could reach. This is the path that
carries it.
The blocker was the guest agent. `fcagent` handled one connection at a time,
inline, so during an hour-long turn the VM accepted nothing — which is why every
existing probe (subagents, stop-gate blocks, cap) runs AFTER the turn rather than
during it. It now spawns a thread per connection, wrapped in `catch_unwind`
because this process is pid 1: a panic used to take the accept loop with it, and
an unbootable VM is a far worse outcome than a missing log. A failed spawn logs
and keeps accepting rather than dropping the listener.
PROVED against a live VM before building on it, since "sound reasoning about this
system" and "measurement" have diverged repeatedly today. Patched rootfs, booted
under Firecracker, ran an 8s exec and a concurrent tail:
exec took 8.0s ok=True
+0.0s 'line1\nline2\n' +1.2s 'line4\n' +3.2s 'line6\n' +6.0s 'DONE\n'
VERDICT: CONCURRENT — tail returned data before exec finished
The rest is the pattern the terminal already uses. New `tail` op streams a file
by OFFSET (so a dropped link resumes instead of replaying, and the tail always
terminates — one that never returns pins a thread for the life of the VM). The
node follows the log alongside the turn and pushes `Uplink::VmOut { run_id, at,
data }` over the WebSocket it already holds, mirroring `PtyOut`. The server does
what `PtyOut` deliberately does not: it APPENDS to the run's checkpoint as well
as fanning out, because a terminal has no history worth keeping and a mission log
is the record of what the agent did. `run_events_sse` emits the new bytes as
`step` events, which the live pane already renders — no frontend change.
The turn is `tee`d, not redirected: the file feeds the live stream and stdout
still becomes `VmOutcome::summary`. A redirect would have produced a live view
and an empty summary, which is the same green-and-empty shape as the bug this
fixes. Tested, along with the log living outside the collected tree so it never
lands in a user's delivered diff.
246 lib tests, 20 binaries; node and fcagent build clean.
|
||
|
Omar Sobh
|
b36ae00ea5 |
fix(missions): a gate that gave up completed the phase green
`done_when_check` is run in exactly one place: the Stop hook inside the guest. Nothing outside it has ever re-run the command — not the evaluator (which judges the PROSE `done_when`), not capture, not delivery. The hook is capped at MAX_BLOCKS so a stuck agent cannot wedge the turn. At the cap it logs `cap: <reason>` and exits 0, releasing the agent with its check still failing. That release was invisible: rc was 0 and the work collected, so both signals the run status was decided from said "fine", and the phase completed. Green phase, unmet condition, no error anywhere — the same silent-success shape this project keeps paying for. The block COUNT cannot fix it. Three blocks then a stop that finally passed and three blocks then a surrender both report `blocks: 3`, and they are opposite outcomes. So the gate now writes a `capped` marker file, probed back out of the guest alongside the block count, and `Some(true)` fails the run on BOTH paths — solo (phase_runner) and composed (microvm_turn_executor). A marker file rather than grepping the log: a block reason embeds the check's own output, so an output line starting `cap:` would read as a release that never happened. Also corrects the comment in `per_node` that sent me looking. It claimed "the phase-level check still runs post-hoc", conflating two mechanisms — that is true of `require_changes` (via `empty_delivery_is_a_failure`) and was never true of `check`. Negative controls, both ablated and confirmed failing: drop the enforcement and `a_node_whose_gate_gave_up_fails_the_run` fails; stop writing the marker and `a_gate_that_gives_up_records_that_it_gave_up` fails. And the control against over-strictness — `a_node_that_was_blocked_and_then_succeeded_passes` — is why this keys on the marker instead of the count. 231 lib tests pass. |
||
|
Omar SobhandClaude Opus 5
|
6991e21f94 |
feat(missions): the completion gate, moved into the agent's own loop
Every check this platform makes on a phase runs AFTER the agent has stopped: the
evaluator judges `done_when`, capture notices a coding phase delivered nothing,
and either verdict costs a whole new VM — a fresh boot, a fresh inject, and an
agent starting over with none of the context that got it that far. Meanwhile the
documented failure mode of a long-running agent is that it stops too early.
MEASURED FIRST, because the plan's chosen seam does not exist here. Probing every
hook name under `claude -p` (2.1.222, hermetic `--settings` file): `SessionStart`,
`UserPromptSubmit`, `PreToolUse`, `PostToolUse`, `SubagentStop` and `Stop` fire;
`TaskCreated`, `TaskCompleted`, `TeammateIdle`, `SessionEnd`, `Notification` and
`PreCompact` do not. The agent-teams hooks Slice 3 deferred are inert on our path
BY CONSTRUCTION — no team forms in print mode at all — so `done_when` could never
have been wired through `TaskCompleted` exit 2. `Stop` is the seam.
`vm_stop_gate` generates a POSIX `sh` hook installed via `--settings`, under
`/root/gate` and never under `/mission/repo` (anything there is collected and
arrives in the user's delivered patch). It refuses a stop when:
- the phase must deliver and the repository is untouched — asked as TWO
questions, since an agent that committed leaves a clean tree and an agent
that did not leaves HEAD alone; only both together mean nothing happened;
- `config.done_when_check` — a command the phase author wrote — exits nonzero,
in which case its OUTPUT is the feedback, not just "the check failed".
Deliberately mechanical. NOT the `done_when` verdict: that is an LLM judgement
made host-side by a different provider on purpose, and re-running it inside the
VM would put the agent's own environment in charge of grading the agent — the
correlated failure the independent judge exists to break.
THE CAP IS LOAD-BEARING. Without a ceiling a stuck agent is blocked, retries, is
blocked again, and burns the hour-long turn budget instead of failing visibly.
After 3 blocks the gate lets it stop, records that it gave up, and leaves the
verdict to the existing post-hoc path, which is unchanged.
PROVEN AGAINST A LIVE AGENT with the REAL generated artifacts, not a paraphrase:
- a read-only task → blocked 3 times with our exact message, released at
exactly the cap, and the agent took the escape hatch the message offers
("if the task genuinely requires no code change, say so explicitly") rather
than touching a file to satisfy the gate. It did not Goodhart it.
- a task that needs an edit → `blocks: 0`, log says `pass`. No false positives.
Two things that could fail silently, both closed. `--settings` is PROBED in the
image before use (`claude --help | grep`), because an unknown option is a hard
CLI error that would turn every gated phase into a failed one; a build without
it degrades to ungated and says so, since losing a check is better than losing
the work. And `stop_blocks` is reported out of the guest — `None` for no gate,
`0` for got-it-right-first-time — so a gate that never fires is distinguishable
from one that was never installed.
`require_changes` does NOT apply per node on a composed run: a graph's verifier
node is SUPPOSED to leave the tree alone, and a per-node gate would refuse its
stop three times for doing its job. `StopGate::per_node` drops it and keeps the
declared check. The phase-level rule still runs post-hoc against what the last
node collected.
An ungated phase's command is byte-identical to before, asserted by test — most
phases are gated, so the ungated path is the one nobody would notice breaking.
517 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
12147a1e01 |
feat(missions): Slice 4 — the two engines composed, with the file handoff proven
`team_engine='composed'` (the third name migration 0069 anticipated) runs a
mission as a durable ZeroClaw graph whose every node is a whole
Claude-Code-in-a-microVM session. Engine Z owns checkpoint/resume, cancellation
and per-node heterogeneity; Engine C owns shared context and cheap fan-out;
neither has the other's asset, which is why this is a composition and not a
compromise.
`MicroVmTurnExecutor` implements the existing `TurnExecutor`, so it inherits the
planners, the checkpoint, the stale-run recovery, `close_finished_phases`, the
evaluator, capture and delivery unchanged — the same trick `SubTopologyExecutor`
already plays with a heavy `run_turn`. Producer side emits ONE `queued` row
carrying the real graph and lets the worker claim it: the durability IS being
worker-driven, and the solo path's `tokio::spawn` has none of it. Still exactly
one `topology_runs` row per unit of work and one completion path — `finish()` is
now that one place, shared by every tier.
THE TRAP, solved and proven. A VM is inject → run → collect → destroy, so a
per-node VM with text-only handoff silently loses every file an earlier node
wrote: node 2 boots from the original checkout, sees nothing, and still reports
success. The mission's host checkout is the medium — every node injects from it
and collects back over it — and two properties make that safe rather than lucky:
`execute_resumable` is strictly sequential, so two VMs never write one directory;
and the vm id is deterministic per (phase, iteration, step), so a duplicate is
refused by the node ("vm already exists") instead of becoming a second writer.
NEGATIVE CONTROL, run rather than assumed: with `repo` swapped for a private
per-node workspace, `a_later_node_sees_an_earlier_nodes_files` FAILS with
`saw:[]`; restored, it passes. The `PhaseVm` seam exists for exactly this — it
models inject/collect through the real `mission_fs` tar path in milliseconds.
Two durability traps this tier walks into, both closed:
- `requeue_stale` fires at 180s on `updated_at`, and one node here can run for
an hour. `SubTopologyExecutor` keeps its parent alive from each leaf step;
there is nothing between the start and end of a VM turn, so the turn holds a
ticker that touches `updated_at` every 30s and aborts on drop. Without it a
healthy composed run is requeued mid-node and boots a second VM.
- the 15-minute stuck-run reaper asks "any step records since it was CREATED?",
which describes a healthy composed run as readily as a wedged one. Hence
`REAPABLE_TIERS` — worker-driven minus this tier. Reaping it would be #54 in
a different costume.
`on_launch` mints no team for a microVM mission, deliberately: claws in
containers are what a VM mission does not use. So `mission_orchestrator::
composed_graph` builds the shape from the team template directly — nodes, roles
and pattern, zero claws provisioned. Per-node `attrs["backend"]` and
`attrs["node_id"]` override the mission's, which is what makes a validator node
on another provider's image a first-class graph node; a malformed `node_id`
fails the node rather than quietly running it where the graph did not ask.
Refusals are recorded as a failed run, not returned as an error: `launch_phase`
is swept every ten seconds, so a returned error is a phase that retries forever
while the log repeats itself.
501 tests pass, clippy clean. NOT yet proven end to end: no composed mission has
run on the fleet, so the resume-after-a-killed-worker leg is argued from the DB
test and the step-numbering test, not from a real two-node run.
Co-Authored-By: Claude Opus 5 <[email protected]>
|