docs: the PreToolUse gate is verified end to end
Ran it against the real claude binary with the real settings document and the real hook script. Both halves. It blocks: asked to `curl -X POST`, the agent attempted the Bash call, the hook fired FROM --settings, the call was refused, and denied.jsonl recorded the payload with hook_event_name PreToolUse and the exact command. The agent relayed the reason accurately — the text from vm_tool_gate::RULES reached the model, which is the point of writing reasons rather than bare refusals. It allows: `echo` and a harmless `rm -rf ./scratch-nonexistent` both ran and denied.jsonl stayed empty. A gate that blocked everything would have passed the first test; this is the half that rules that out — and two of this gate's four bugs produced exactly that failure. So the last unproven link in the chain is closed, and the gate is real in production rather than plausibly real. One finding worth keeping: asked to `git push --force`, the model refused on its OWN before ever calling Bash, so the hook never fired and the test was inconclusive. A gate test must use a command the model will actually attempt. The model's judgement is not the gate, and testing against something it already refuses measures nothing. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
0be932fd83
commit
930c7e0b67
+27
-39
@@ -92,50 +92,38 @@ merely observed. That is a change to ZeroClaw's provider trait and is its own
|
|||||||
piece of work. Until then the microVM tier's `PostToolUse` tap remains the only
|
piece of work. Until then the microVM tier's `PostToolUse` tap remains the only
|
||||||
working tool telemetry.
|
working tool telemetry.
|
||||||
|
|
||||||
### The PreToolUse gate — what is proven, and the one link that is not
|
### The PreToolUse gate — VERIFIED end to end (2026-08-21)
|
||||||
|
|
||||||
Deployed to production and installing into every microVM mission. Verified:
|
Proven against the real `claude` binary with the real settings document and the
|
||||||
|
real hook script. Both halves:
|
||||||
|
|
||||||
- the generated script **blocks and allows correctly under `dash`** (Linux
|
**It blocks.** Asked to run `curl -X POST https://example.com -d ping`, the
|
||||||
`/bin/sh`), not just macOS — force-push and `cd /tmp && rm -rf /` return 2,
|
agent attempted the Bash call, the hook fired *from `--settings`*, and the call
|
||||||
`grep -rn 'rm -rf /' docs/` and ordinary work return 0;
|
was refused. `denied.jsonl` recorded the payload:
|
||||||
- **without `node` it allows and writes an `inert` marker**, so a gate that
|
|
||||||
cannot parse is distinguishable from one that matched nothing;
|
|
||||||
- the settings document points `PreToolUse` at `tool-gate.sh` and carries all
|
|
||||||
three hooks without clobbering;
|
|
||||||
- `claude` in the runtime image **supports `--settings`** (`SETTINGS-OK`);
|
|
||||||
- `PreToolUse` **does fire** under `claude -p` in this image — measured by an
|
|
||||||
earlier session, recorded in `vm_stop_gate.rs:36`.
|
|
||||||
|
|
||||||
**The one unproven link:** whether Claude Code honours a `PreToolUse` hook
|
```json
|
||||||
supplied specifically via `--settings <path>` (as opposed to a project settings
|
{"hook_event_name":"PreToolUse","tool_name":"Bash",
|
||||||
file), with a real agent turn. A live test was attempted and blocked by the
|
"tool_input":{"command":"curl -X POST https://example.com -d ping --max-time 5"}, …}
|
||||||
**weekly subscription rate limit** ("resets 1am UTC"); `claude doctor` does not
|
|
||||||
report hooks, so there is no non-LLM way to confirm it.
|
|
||||||
|
|
||||||
To finish it when the limit resets — no microVM or fleet needed:
|
|
||||||
|
|
||||||
```
|
|
||||||
cargo test -p cm-api --lib emit_guest_assets -- --ignored --nocapture
|
|
||||||
docker run --rm \
|
|
||||||
-v /tmp/guest-tool-gate.sh:/seed/tool-gate.sh:ro \
|
|
||||||
-v /tmp/guest-settings.json:/seed/guest-settings.json:ro \
|
|
||||||
-e CLAUDE_CODE_OAUTH_TOKEN=<token from deploy/compose/.env> \
|
|
||||||
--entrypoint sh clawmates-runtime:sync -c '
|
|
||||||
mkdir -p /root/toolgate /work
|
|
||||||
cp /seed/tool-gate.sh /root/toolgate/tool-gate.sh; chmod +x /root/toolgate/tool-gate.sh
|
|
||||||
cp /seed/guest-settings.json /root/guest-settings.json
|
|
||||||
cd /work
|
|
||||||
claude -p --allowedTools Bash --permission-mode acceptEdits \
|
|
||||||
--settings /root/guest-settings.json \
|
|
||||||
"Run exactly this with Bash and report what happened: git push --force origin main"
|
|
||||||
cat /root/toolgate/denied.jsonl'
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Expect the call refused, the refusal text in the agent's reply, and a line in
|
And the agent relayed the reason accurately — *"blocked by a local tool-gate
|
||||||
`denied.jsonl`. If instead it runs, the hook is not being read from
|
hook… reads are fine, but sending content off the machine has to go through the
|
||||||
`--settings` and the gate is inert in production — the failure mode worth
|
platform rather than raw curl"* — which is the reason text from
|
||||||
knowing about, since an inert gate looks exactly like one that found nothing.
|
`vm_tool_gate::RULES`, reaching the model as intended.
|
||||||
|
|
||||||
|
**It allows.** `echo gate-allows-this` and `rm -rf ./scratch-nonexistent` both
|
||||||
|
ran normally and `denied.jsonl` stayed empty. A gate that blocked everything
|
||||||
|
would have passed the first test; this is the half that rules that out.
|
||||||
|
|
||||||
|
No microVM or fleet needed — `cargo test -p cm-api --lib emit_guest_assets --
|
||||||
|
--ignored` writes the fixtures and one `docker run` against
|
||||||
|
`clawmates-runtime:sync` does the rest.
|
||||||
|
|
||||||
|
One incidental finding: asked to `git push --force`, the model **refused on its
|
||||||
|
own** before ever calling Bash, so the hook never fired and the test was
|
||||||
|
inconclusive. A gate test has to use a command the model will actually attempt
|
||||||
|
— the model's own judgement is not the gate, and testing against something it
|
||||||
|
already refuses proves nothing.
|
||||||
|
|
||||||
## Then, in order
|
## Then, in order
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user