docs: the PreToolUse gate is verified end to end
deploy / test (push) Successful in 4m55s
deploy / build (push) Failing after 1m28s

Ran it against the real claude binary with the real settings document and
the real hook script. Both halves.

It blocks: asked to `curl -X POST`, the agent attempted the Bash call, the
hook fired FROM --settings, the call was refused, and denied.jsonl recorded
the payload with hook_event_name PreToolUse and the exact command. The agent
relayed the reason accurately — the text from vm_tool_gate::RULES reached
the model, which is the point of writing reasons rather than bare refusals.

It allows: `echo` and a harmless `rm -rf ./scratch-nonexistent` both ran and
denied.jsonl stayed empty. A gate that blocked everything would have passed
the first test; this is the half that rules that out — and two of this
gate's four bugs produced exactly that failure.

So the last unproven link in the chain is closed, and the gate is real in
production rather than plausibly real.

One finding worth keeping: asked to `git push --force`, the model refused on
its OWN before ever calling Bash, so the hook never fired and the test was
inconclusive. A gate test must use a command the model will actually attempt.
The model's judgement is not the gate, and testing against something it
already refuses measures nothing.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-21 00:19:37 -07:00
co-authored by Claude Opus 5
parent 0be932fd83
commit 930c7e0b67
+27 -39
View File
@@ -92,50 +92,38 @@ merely observed. That is a change to ZeroClaw's provider trait and is its own
piece of work. Until then the microVM tier's `PostToolUse` tap remains the only piece of work. Until then the microVM tier's `PostToolUse` tap remains the only
working tool telemetry. working tool telemetry.
### The PreToolUse gate — what is proven, and the one link that is not ### The PreToolUse gate — VERIFIED end to end (2026-08-21)
Deployed to production and installing into every microVM mission. Verified: Proven against the real `claude` binary with the real settings document and the
real hook script. Both halves:
- the generated script **blocks and allows correctly under `dash`** (Linux **It blocks.** Asked to run `curl -X POST https://example.com -d ping`, the
`/bin/sh`), not just macOS — force-push and `cd /tmp && rm -rf /` return 2, agent attempted the Bash call, the hook fired *from `--settings`*, and the call
`grep -rn 'rm -rf /' docs/` and ordinary work return 0; was refused. `denied.jsonl` recorded the payload:
- **without `node` it allows and writes an `inert` marker**, so a gate that
cannot parse is distinguishable from one that matched nothing;
- the settings document points `PreToolUse` at `tool-gate.sh` and carries all
three hooks without clobbering;
- `claude` in the runtime image **supports `--settings`** (`SETTINGS-OK`);
- `PreToolUse` **does fire** under `claude -p` in this image — measured by an
earlier session, recorded in `vm_stop_gate.rs:36`.
**The one unproven link:** whether Claude Code honours a `PreToolUse` hook ```json
supplied specifically via `--settings <path>` (as opposed to a project settings {"hook_event_name":"PreToolUse","tool_name":"Bash",
file), with a real agent turn. A live test was attempted and blocked by the "tool_input":{"command":"curl -X POST https://example.com -d ping --max-time 5"}, …}
**weekly subscription rate limit** ("resets 1am UTC"); `claude doctor` does not
report hooks, so there is no non-LLM way to confirm it.
To finish it when the limit resets — no microVM or fleet needed:
```
cargo test -p cm-api --lib emit_guest_assets -- --ignored --nocapture
docker run --rm \
-v /tmp/guest-tool-gate.sh:/seed/tool-gate.sh:ro \
-v /tmp/guest-settings.json:/seed/guest-settings.json:ro \
-e CLAUDE_CODE_OAUTH_TOKEN=<token from deploy/compose/.env> \
--entrypoint sh clawmates-runtime:sync -c '
mkdir -p /root/toolgate /work
cp /seed/tool-gate.sh /root/toolgate/tool-gate.sh; chmod +x /root/toolgate/tool-gate.sh
cp /seed/guest-settings.json /root/guest-settings.json
cd /work
claude -p --allowedTools Bash --permission-mode acceptEdits \
--settings /root/guest-settings.json \
"Run exactly this with Bash and report what happened: git push --force origin main"
cat /root/toolgate/denied.jsonl'
``` ```
Expect the call refused, the refusal text in the agent's reply, and a line in And the agent relayed the reason accurately — *"blocked by a local tool-gate
`denied.jsonl`. If instead it runs, the hook is not being read from hook… reads are fine, but sending content off the machine has to go through the
`--settings` and the gate is inert in production — the failure mode worth platform rather than raw curl"* — which is the reason text from
knowing about, since an inert gate looks exactly like one that found nothing. `vm_tool_gate::RULES`, reaching the model as intended.
**It allows.** `echo gate-allows-this` and `rm -rf ./scratch-nonexistent` both
ran normally and `denied.jsonl` stayed empty. A gate that blocked everything
would have passed the first test; this is the half that rules that out.
No microVM or fleet needed — `cargo test -p cm-api --lib emit_guest_assets --
--ignored` writes the fixtures and one `docker run` against
`clawmates-runtime:sync` does the rest.
One incidental finding: asked to `git push --force`, the model **refused on its
own** before ever calling Bash, so the hook never fired and the test was
inconclusive. A gate test has to use a command the model will actually attempt
— the model's own judgement is not the gate, and testing against something it
already refuses proves nothing.
## Then, in order ## Then, in order