docs: container-tier gate and telemetry shipped; CI failures were disk
Records the verified result (10 tool.call, 4 file.touch on a real mission), how hooks succeed where stream-json could not, the production state and its rollback, and the three same-shaped bugs the live test found. Also records that CI's build failures were disk pressure from my own manual runtime builds on gw-04 — not code — and that a docs-only commit was the first casualty, which made it look like a regression. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
co-authored by
Claude Opus 5
Claude Opus 5
parent
f6e6037aa0
commit
5a11fae0d6
+48
-99
@@ -10,120 +10,69 @@ Read `CAPABILITY-REVIEW.md` for the system picture,
|
|||||||
`merge/upstream-v0.8.4` is at `db1c50966`. The local suite is green — 106 test
|
`merge/upstream-v0.8.4` is at `db1c50966`. The local suite is green — 106 test
|
||||||
binaries, zero build errors, frontend builds.
|
binaries, zero build errors, frontend builds.
|
||||||
|
|
||||||
## CI green, deployed to production 2026-08-20
|
## Container-tier tool gate + telemetry — SHIPPED 2026-08-21
|
||||||
|
|
||||||
Run **498 passed and deployed**. Verified on gw-04 after the recreate:
|
The tier that actually runs missions now has both, verified end to end on a
|
||||||
|
real mission (locally first, then deployed):
|
||||||
|
|
||||||
```
|
```
|
||||||
skill_self_authoring: ENABLED — agents apply their own skill drafts without human approval
|
tool.call 10 Bash 6, Read 3, Write 1
|
||||||
skills_loader: loaded 53 builtin skill(s)
|
file.touch 4 research/tapproof.md
|
||||||
team_template_loader: loaded 11 builtin team template(s) # 0 unresolved
|
|
||||||
gateway_preflight: http://clawmates-runtime:42617 answered
|
|
||||||
missions.retain_events_until # migration 0080 applied
|
|
||||||
```
|
```
|
||||||
|
|
||||||
107 test binaries green in CI. So everything in this block is now **live**,
|
That is the first time the container tier has ever been observable.
|
||||||
including self-authoring with no human click and the PreToolUse gate installing
|
|
||||||
into every microVM mission.
|
|
||||||
|
|
||||||
### Why runs 490–496 failed (two causes, both fixed)
|
**How, after `stream-json` failed.** Claude Code runs its tools inside its own
|
||||||
|
subprocess, so they never reach ZeroClaw's executor and never become a
|
||||||
|
`TurnEvent::ToolCall`. Hooks bypass that entirely: `claude -p --settings <doc>`
|
||||||
|
honours `PreToolUse` and `PostToolUse`, so the gate blocks and the tap records
|
||||||
|
without ZeroClaw being involved at all.
|
||||||
|
|
||||||
- **490** — two runs overlapped. Every run began by removing the *shared*
|
Pieces: `--settings` on `claude_cli` (fork `be9c34b1c`),
|
||||||
container `cm-ci-pg`, so run 491 deleted run 490's database mid-suite. Fixed
|
`container_tool_hooks` writes both hook scripts and one settings document into
|
||||||
with a `concurrency` group and a run-scoped name. (497 was later *cancelled*
|
the mission container, `set_claude_cli_settings` points the provider at it, and
|
||||||
by that guard, which is correct behaviour, not a failure.)
|
`phase_runner::drain_finished_container_phases` collects the tap into
|
||||||
- **491–496** — a **single apostrophe**. The Rust step is
|
`mission_events` (idempotent by truncation — no cursor column).
|
||||||
`docker run … sh -c '…'`, and a comment inside it read `cm-api's
|
|
||||||
vm_tool_gate`. The quote closed; bash died with *"unexpected EOF"* **before
|
|
||||||
running anything** — hence an empty log, a breadcrumb showing the step
|
|
||||||
entered and produced nothing, and three wrong theories. I introduced it in
|
|
||||||
the commit that fixed 490.
|
|
||||||
|
|
||||||
`tests/workflow_shell_syntax.rs` now `bash -n`s every `run:` block, so this
|
**Production state:** server on `f6e6037`;
|
||||||
class fails locally rather than six runs later. Negative-controlled.
|
`CLAWMATES_RUNTIME_IMAGE=clawmates-runtime:hooks` in `/opt/clawmates/.env`
|
||||||
|
(backup at `.env.bak.prehooks`; rollback = restore it and recreate). Note that
|
||||||
|
host uses **legacy `docker-compose`**, not the v2 plugin.
|
||||||
|
|
||||||
### Still worth having
|
**Not yet observed in production** — no prod mission has run since the flip.
|
||||||
|
Prod auth is Clerk, so a mission cannot be launched from here by password. The
|
||||||
|
check when one runs:
|
||||||
|
|
||||||
The Gitea actions-log API returns **403** for the token in
|
```
|
||||||
`deploy/compose/.env`, and the server is behind Cloudflare on a host not on the
|
ssh gw-04 'docker logs clawmates_server_1 2>&1 | grep -E "per-mission runtime image|drained .* tool call"'
|
||||||
tailnet. Every step now writes to `/tmp/ci-logs` on the runner host as a
|
|
||||||
workaround (`ssh gw-04 'cat /tmp/ci-logs/STEP; tail -40 /tmp/ci-logs/rust-step.log'`),
|
|
||||||
but **a token with the `actions` scope** would have made the whole detour
|
|
||||||
unnecessary.
|
|
||||||
|
|
||||||
### stream-json: tested live, and it does NOT deliver observability
|
|
||||||
|
|
||||||
Built for amd64 on gw-04, deployed, and driven with a real mission. Result:
|
|
||||||
|
|
||||||
- **The agent used tools** — it ran `echo hello-from-tools` via Bash and
|
|
||||||
exercised Read/Edit/Write/Glob/Grep, confirming again that mission agents
|
|
||||||
are not tool-free.
|
|
||||||
- **No `tool.call` events appeared.** The gateway's unmatched-frame histogram
|
|
||||||
showed only `session_start`, exactly as before.
|
|
||||||
|
|
||||||
Why: `TurnEvent::ToolCall` — the thing the gateway turns into a `tool_call`
|
|
||||||
frame — is emitted from `tool_execution.rs`, i.e. **only for tools ZeroClaw
|
|
||||||
itself executes**. Claude Code runs its tools inside its own subprocess, so
|
|
||||||
ZeroClaw never emits the event and ClawMates never sees it. The provider
|
|
||||||
knowing about the calls changes nothing on its own.
|
|
||||||
|
|
||||||
**Worse, the first version was actively harmful.** It put the recovered calls
|
|
||||||
into `ChatResponse.tool_calls`, which means "execute these" — so the agent loop
|
|
||||||
tried to run `Bash`, got `Unknown tool: Bash` (`tool_execution.rs:175`), and fed
|
|
||||||
that back to the model as a tool result: an extra LLM round-trip per call,
|
|
||||||
telling the agent its own work failed. The mission still completed with correct
|
|
||||||
output, which is precisely why it would have gone unnoticed.
|
|
||||||
|
|
||||||
Fixed in fork commit `295d6d3b8`: the calls are logged, never returned. **Both
|
|
||||||
runtimes were rolled back to the known-good image** (`:v084` on gw-04,
|
|
||||||
`:toolchain` locally), and the harmful v1 image was deleted from both hosts so
|
|
||||||
it cannot be redeployed by accident.
|
|
||||||
|
|
||||||
A corrected image **`clawmates-runtime:streamjson2`** (from `295d6d3`) is built
|
|
||||||
and sitting on gw-04, **deliberately not deployed**. It delivers no
|
|
||||||
observability until the event channel exists, so shipping it would be a
|
|
||||||
provider output-format change with real risk and no current benefit.
|
|
||||||
Production stays on `:v084`. Deploy `streamjson2` only as part of the
|
|
||||||
event-channel work below — not on its own.
|
|
||||||
|
|
||||||
**The real fix for mission-tier tool observability** is to plumb an event
|
|
||||||
channel into the provider so it can emit `TurnEvent::ToolCall` for calls it
|
|
||||||
merely observed. That is a change to ZeroClaw's provider trait and is its own
|
|
||||||
piece of work. Until then the microVM tier's `PostToolUse` tap remains the only
|
|
||||||
working tool telemetry.
|
|
||||||
|
|
||||||
### The PreToolUse gate — VERIFIED end to end (2026-08-21)
|
|
||||||
|
|
||||||
Proven against the real `claude` binary with the real settings document and the
|
|
||||||
real hook script. Both halves:
|
|
||||||
|
|
||||||
**It blocks.** Asked to run `curl -X POST https://example.com -d ping`, the
|
|
||||||
agent attempted the Bash call, the hook fired *from `--settings`*, and the call
|
|
||||||
was refused. `denied.jsonl` recorded the payload:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"hook_event_name":"PreToolUse","tool_name":"Bash",
|
|
||||||
"tool_input":{"command":"curl -X POST https://example.com -d ping --max-time 5"}, …}
|
|
||||||
```
|
```
|
||||||
|
|
||||||
And the agent relayed the reason accurately — *"blocked by a local tool-gate
|
### Three bugs the live test found, all the same shape
|
||||||
hook… reads are fine, but sending content off the machine has to go through the
|
|
||||||
platform rather than raw curl"* — which is the reason text from
|
|
||||||
`vm_tool_gate::RULES`, reaching the model as intended.
|
|
||||||
|
|
||||||
**It allows.** `echo gate-allows-this` and `rm -rf ./scratch-nonexistent` both
|
Each left every other link looking correct:
|
||||||
ran normally and `denied.jsonl` stayed empty. A gate that blocked everything
|
|
||||||
would have passed the first test; this is the half that rules that out.
|
|
||||||
|
|
||||||
No microVM or fleet needed — `cargo test -p cm-api --lib emit_guest_assets --
|
1. The settings document pointed `PostToolUse` at a path the installer never
|
||||||
--ignored` writes the fixtures and one `docker run` against
|
wrote. Claude Code does not complain about a missing hook command — it
|
||||||
`clawmates-runtime:sync` does the rest.
|
records nothing. A test now compares the document's commands against the
|
||||||
|
files the installer creates.
|
||||||
|
2. The mission container uses `CLAWMATES_RUNTIME_IMAGE`, not the shared
|
||||||
|
`clawmates-runtime` container — it was on an older image whose daemon schema
|
||||||
|
had no `settings` field, so the prop write returned `404 path_not_found`.
|
||||||
|
3. The drain used `connect_with_local_defaults()`; the server reaches Docker
|
||||||
|
through a **socket proxy**, so it failed — and returned `Ok(())` silently.
|
||||||
|
|
||||||
One incidental finding: asked to `git push --force`, the model **refused on its
|
### CI: build failures were disk, not code
|
||||||
own** before ever calling Bash, so the hook never fired and the test was
|
|
||||||
inconclusive. A gate test has to use a command the model will actually attempt
|
Runs 503–506 failed at `build` with an unreadable log, and the first casualty
|
||||||
— the model's own judgement is not the gate, and testing against something it
|
was a **docs-only** commit. Cause: building runtime images by hand on gw-04
|
||||||
already refuses proves nothing.
|
competes with CI for the same 150G volume; the frontend image build lost.
|
||||||
|
`docker builder prune` reclaimed 34GB (22G → 57G free) and the next run went
|
||||||
|
green. The build job now writes breadcrumbs, a `df -h` snapshot, and which
|
||||||
|
services actually pushed — the failing runs had pushed `server`, aborted on
|
||||||
|
`frontend`, and left `:latest` unmoved, which surfaced three steps later as
|
||||||
|
"the deploy did not happen".
|
||||||
|
|
||||||
|
**Operational note:** do not build images by hand on gw-04 while CI may run.
|
||||||
|
|
||||||
## Then, in order
|
## Then, in order
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user