650a556029080ce735b9b505bc300aeca1a778f4
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar SobhandClaude Opus 5
|
547b5d9987 |
feat(missions): a pre-execution gate on mission tool calls
The second half of the tool-call research. Until now a mission agent's
Bash call was gated by nothing, anywhere.
WHY THERE WAS NO GATE
vm_tool_tap is a PostToolUse hook: it fires after the tool has already run
and exit-0s unconditionally, because a non-zero PostToolUse talks back to
the model. It is telemetry and says so. GatePolicy — the §15 door — has one
enforcement site, the chat loop, and its approvals key on
(session_id, message_id), which no mission phase can produce. Meanwhile the
solo tiers run `claude -p --permission-mode acceptEdits` with Read, Edit,
Write and Bash pre-approved.
PreToolUse fires under `claude -p` in this image — measured by vm_stop_gate,
which also proved the exit-2-plus-stderr contract — and had zero callers.
This is that hook.
WHAT IT IS, AND IS NOT
A deterministic policy gate: a short deny list of actions with no legitimate
form inside a mission, blocked before they run, with the reason handed back
so the model can choose differently.
It is NOT the §15 human approval gate, and the module says so. A hook blocks
the agent's process while it runs and a human decision takes minutes to
hours; waiting inside the hook would wedge the turn. This closes the gap
between nothing and something.
The deny list is short on purpose. A gate that blocks legitimate work is
worse than none: the agent cannot ask a human, so it either works around the
block — doing something stranger than what was denied — or burns the turn.
FOUR BUGS THE TESTS FOUND, IN ORDER
1. Substring matching denied `grep -rn 'rm -rf /' docs/`. Searching for a
string is not running it. Now rules anchor to the start of a shell
segment, with a separate flag-style match that exempts text tools.
2. The `case` patterns were unquoted, so a needle containing a space made
the whole script a SYNTAX ERROR — which as a PreToolUse hook exits
non-zero and denies EVERY call. Every text assertion passed while the
script was in that state; only running it under a real `sh` found it.
3. The hook receives JSON, not a command, so "starts with" could never
match — `case` saw `{"tool_name":"bash",...` every time. Now extracts
tool_name and tool_input.command with `node` (no jq in the image; node is
guaranteed because Claude Code is a node program).
4. `IFS='\n'` in POSIX sh sets IFS to backslash and the letter n, not a
newline. Nothing split, so only commands with no separator were ever
tested and `cd /tmp && rm -rf /` sailed through. Now a literal newline.
Every failure path allows. A gate that fails closed on a parse error blocks
the whole phase, which is exactly what bug 2 did.
Wired through vm_tool_tap::guest_settings, still the single writer of the
guest settings document — a third hook makes the clobber it prevents more
likely, not less, and a test asserts all three survive one document and that
PreToolUse points at the gate's own script rather than the tap's.
Honest limit, stated in the module: a determined agent defeats any
string-matching gate. This is aimed at accidents and obvious cases; the real
isolation is the container and microVM boundary.
Full workspace suite green: 106 binaries, zero build errors.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
9e61e3ba35 |
feat(viz): what the agents actually did, as structured events
The World could draw a mission's shape but nothing about the work. The
detail existed only as prose in checkpoint.log and model output, where a
tool name is indistinguishable from an agent *talking about* a tool — so
it was never parsed, deliberately. `mission_events` is the structured
channel that replaces it.
Three taps, one table:
- Container tier: the `_ => {}` at the end of topology_exec's typed frame
stream now matches `tool_call` and reads the tool's JSON ARGUMENTS for a
path. Never the prose summary — a path scraped from a sentence would put
files on the map that no agent opened, and the test proves a Grep whose
summary says "src/main.rs" produces no file touch. The frame name itself
is unverified, so the same commit ships an unmatched-frame-type
histogram: a tap that matches nothing looks exactly like a mission that
used no tools, and this is how one gw-04 run names the real frame.
- microVM tier: a `PostToolUse` hook, the seam vm_stop_gate already proved
fires under `claude -p`. It copies stdin to /root/tap and exits 0
unconditionally — a non-zero PostToolUse hook talks back to the model,
which would turn the observer into a participant. Drained before collect,
since the VM is destroyed moments later.
- Phase transitions: five identical copies of the pending→running UPDATE
became one `mark_phase_running`, and `close_finished_phases` grew
RETURNING. Its CASE decides each phase's status inside SQL from rows the
statement does not change, so it cannot be re-derived afterwards without
writing that CASE twice — without RETURNING it emits zero phase.completed
and reports success.
The settings.json hazard the plan called out: the stop gate wrote the
WHOLE document, so a second hook writer would have silently erased it and
a coding phase would then complete having written nothing — the exact
failure the gate exists to catch. There is now one composer,
`vm_tool_tap::guest_settings`, one writer, and a source-walk test that
fails if anything else writes a settings document.
`mission_events.run_id` carries no FK on purpose: phase_runner DELETEs
topology_runs on retry, and a cascade would erase a phase's whole history
the moment it retried — silently, since a cascade is not an error.
world.rs streams it with a cursor that separates backfill from motion.
Everything already in the table when a subscriber arrives is drawn as
settled history; only what lands afterwards animates. Otherwise opening a
finished mission replays an hour of tool calls as a burst storm.
Bounded twice: 400 events per phase (enforced inside the INSERT, since
two concurrent taps would each read a count below the cap) and a 7-day
retention sweep in mission_gc.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar Sobh
|
b36ae00ea5 |
fix(missions): a gate that gave up completed the phase green
`done_when_check` is run in exactly one place: the Stop hook inside the guest. Nothing outside it has ever re-run the command — not the evaluator (which judges the PROSE `done_when`), not capture, not delivery. The hook is capped at MAX_BLOCKS so a stuck agent cannot wedge the turn. At the cap it logs `cap: <reason>` and exits 0, releasing the agent with its check still failing. That release was invisible: rc was 0 and the work collected, so both signals the run status was decided from said "fine", and the phase completed. Green phase, unmet condition, no error anywhere — the same silent-success shape this project keeps paying for. The block COUNT cannot fix it. Three blocks then a stop that finally passed and three blocks then a surrender both report `blocks: 3`, and they are opposite outcomes. So the gate now writes a `capped` marker file, probed back out of the guest alongside the block count, and `Some(true)` fails the run on BOTH paths — solo (phase_runner) and composed (microvm_turn_executor). A marker file rather than grepping the log: a block reason embeds the check's own output, so an output line starting `cap:` would read as a release that never happened. Also corrects the comment in `per_node` that sent me looking. It claimed "the phase-level check still runs post-hoc", conflating two mechanisms — that is true of `require_changes` (via `empty_delivery_is_a_failure`) and was never true of `check`. Negative controls, both ablated and confirmed failing: drop the enforcement and `a_node_whose_gate_gave_up_fails_the_run` fails; stop writing the marker and `a_gate_that_gives_up_records_that_it_gave_up` fails. And the control against over-strictness — `a_node_that_was_blocked_and_then_succeeded_passes` — is why this keys on the marker instead of the count. 231 lib tests pass. |
||
|
Omar SobhandClaude Opus 5
|
6991e21f94 |
feat(missions): the completion gate, moved into the agent's own loop
Every check this platform makes on a phase runs AFTER the agent has stopped: the
evaluator judges `done_when`, capture notices a coding phase delivered nothing,
and either verdict costs a whole new VM — a fresh boot, a fresh inject, and an
agent starting over with none of the context that got it that far. Meanwhile the
documented failure mode of a long-running agent is that it stops too early.
MEASURED FIRST, because the plan's chosen seam does not exist here. Probing every
hook name under `claude -p` (2.1.222, hermetic `--settings` file): `SessionStart`,
`UserPromptSubmit`, `PreToolUse`, `PostToolUse`, `SubagentStop` and `Stop` fire;
`TaskCreated`, `TaskCompleted`, `TeammateIdle`, `SessionEnd`, `Notification` and
`PreCompact` do not. The agent-teams hooks Slice 3 deferred are inert on our path
BY CONSTRUCTION — no team forms in print mode at all — so `done_when` could never
have been wired through `TaskCompleted` exit 2. `Stop` is the seam.
`vm_stop_gate` generates a POSIX `sh` hook installed via `--settings`, under
`/root/gate` and never under `/mission/repo` (anything there is collected and
arrives in the user's delivered patch). It refuses a stop when:
- the phase must deliver and the repository is untouched — asked as TWO
questions, since an agent that committed leaves a clean tree and an agent
that did not leaves HEAD alone; only both together mean nothing happened;
- `config.done_when_check` — a command the phase author wrote — exits nonzero,
in which case its OUTPUT is the feedback, not just "the check failed".
Deliberately mechanical. NOT the `done_when` verdict: that is an LLM judgement
made host-side by a different provider on purpose, and re-running it inside the
VM would put the agent's own environment in charge of grading the agent — the
correlated failure the independent judge exists to break.
THE CAP IS LOAD-BEARING. Without a ceiling a stuck agent is blocked, retries, is
blocked again, and burns the hour-long turn budget instead of failing visibly.
After 3 blocks the gate lets it stop, records that it gave up, and leaves the
verdict to the existing post-hoc path, which is unchanged.
PROVEN AGAINST A LIVE AGENT with the REAL generated artifacts, not a paraphrase:
- a read-only task → blocked 3 times with our exact message, released at
exactly the cap, and the agent took the escape hatch the message offers
("if the task genuinely requires no code change, say so explicitly") rather
than touching a file to satisfy the gate. It did not Goodhart it.
- a task that needs an edit → `blocks: 0`, log says `pass`. No false positives.
Two things that could fail silently, both closed. `--settings` is PROBED in the
image before use (`claude --help | grep`), because an unknown option is a hard
CLI error that would turn every gated phase into a failed one; a build without
it degrades to ungated and says so, since losing a check is better than losing
the work. And `stop_blocks` is reported out of the guest — `None` for no gate,
`0` for got-it-right-first-time — so a gate that never fires is distinguishable
from one that was never installed.
`require_changes` does NOT apply per node on a composed run: a graph's verifier
node is SUPPOSED to leave the tree alone, and a per-node gate would refuse its
stop three times for doing its job. `StopGate::per_node` drops it and keeps the
declared check. The phase-level rule still runs post-hoc against what the last
node collected.
An ungated phase's command is byte-identical to before, asserted by test — most
phases are gated, so the ungated path is the one nobody would notice breaking.
517 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|