Files
clawmates/docs/NEXT-SESSION.md
T
Omar SobhandClaude Opus 5 0b4d91889a
deploy / test (push) Successful in 4m46s
deploy / build (push) Successful in 1m1s
docs: hand-off refresh — container-tier work shipped, stale guidance corrected
TOOL-CALL-ARCHITECTURE.md said "switch claude_cli to stream-json" as the
cheapest fix. That was wrong and is now marked so, with what actually
happened: zero tool.call events with the parser working perfectly, because
TurnEvent::ToolCall only fires for tools ZeroClaw itself executes. Hooks
sidestep that entirely, and the doc now leads with the resolution rather
than the theory. A fresh session is pointed at this file, so leaving the
wrong recommendation on top would have sent it down the same path.

NEXT-SESSION.md: state header, and the ordered list rewritten — items 1-3
are done or superseded. "Give the direct-session tier a tap" is dropped with
its reason: that tier is dormant (CLAWMATES_MISSION_EXECUTOR unset), and
checking before building saved the work. New top item is watching the first
production mission, since the gate and tap are proven locally and unproven
in prod.

Added an operational section for the things that cost the most time: the
403 actions-log API, gw-04's legacy docker-compose, the socket proxy, disk
contention between manual builds and CI, and Clerk-only prod auth.

Also flagged that SKILL-USE-BASELINE.md's Trigger column is now stale in a
good way — tool calls are observable on the container tier, so Trigger can
be scored from behaviour instead of prose. That is the highest-value
follow-up.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-21 07:22:01 -07:00

178 lines
8.9 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Where this left off — 2026-08-21
Read `CAPABILITY-REVIEW.md` for the system picture,
`TOOL-CALL-ARCHITECTURE.md` for how mission tools actually work (and the two
wrong theories that preceded it), and `SKILL-USE-BASELINE.md` for the
measurement — noting that its Trigger column is now out of date in a good way
(see "Next, in order" §2).
## State of the tree
Everything is pushed. `main` is at `5a11fae`; the fork's
`merge/upstream-v0.8.4` is at `be9c34b1c`. Local suite green: **107 test
binaries, 412 lib tests**, frontend builds.
**CI is green and production is current.** Run 507 passed `test` and `build`,
and gw-04 rolled to it. Production also runs `clawmates-runtime:hooks` for
per-mission containers (see below).
## Container-tier tool gate + telemetry — SHIPPED 2026-08-21
The tier that actually runs missions now has both, verified end to end on a
real mission (locally first, then deployed):
```
tool.call 10 Bash 6, Read 3, Write 1
file.touch 4 research/tapproof.md
```
That is the first time the container tier has ever been observable.
**How, after `stream-json` failed.** Claude Code runs its tools inside its own
subprocess, so they never reach ZeroClaw's executor and never become a
`TurnEvent::ToolCall`. Hooks bypass that entirely: `claude -p --settings <doc>`
honours `PreToolUse` and `PostToolUse`, so the gate blocks and the tap records
without ZeroClaw being involved at all.
Pieces: `--settings` on `claude_cli` (fork `be9c34b1c`),
`container_tool_hooks` writes both hook scripts and one settings document into
the mission container, `set_claude_cli_settings` points the provider at it, and
`phase_runner::drain_finished_container_phases` collects the tap into
`mission_events` (idempotent by truncation — no cursor column).
**Production state:** server on `f6e6037`;
`CLAWMATES_RUNTIME_IMAGE=clawmates-runtime:hooks` in `/opt/clawmates/.env`
(backup at `.env.bak.prehooks`; rollback = restore it and recreate). Note that
host uses **legacy `docker-compose`**, not the v2 plugin.
**Not yet observed in production** — no prod mission has run since the flip.
Prod auth is Clerk, so a mission cannot be launched from here by password. The
check when one runs:
```
ssh gw-04 'docker logs clawmates_server_1 2>&1 | grep -E "per-mission runtime image|drained .* tool call"'
```
### Three bugs the live test found, all the same shape
Each left every other link looking correct:
1. The settings document pointed `PostToolUse` at a path the installer never
wrote. Claude Code does not complain about a missing hook command — it
records nothing. A test now compares the document's commands against the
files the installer creates.
2. The mission container uses `CLAWMATES_RUNTIME_IMAGE`, not the shared
`clawmates-runtime` container — it was on an older image whose daemon schema
had no `settings` field, so the prop write returned `404 path_not_found`.
3. The drain used `connect_with_local_defaults()`; the server reaches Docker
through a **socket proxy**, so it failed — and returned `Ok(())` silently.
### CI: build failures were disk, not code
Runs 503506 failed at `build` with an unreadable log, and the first casualty
was a **docs-only** commit. Cause: building runtime images by hand on gw-04
competes with CI for the same 150G volume; the frontend image build lost.
`docker builder prune` reclaimed 34GB (22G → 57G free) and the next run went
green. The build job now writes breadcrumbs, a `df -h` snapshot, and which
services actually pushed — the failing runs had pushed `server`, aborted on
`frontend`, and left `:latest` unmoved, which surfaced three steps later as
"the deploy did not happen".
**Operational note:** do not build images by hand on gw-04 while CI may run.
## Next, in order
1. **Watch the first production mission.** Nothing has run since the runtime
flip, so the container-tier gate and tap are proven locally and unproven in
prod. Prod auth is Clerk, so a mission cannot be launched from here.
```
ssh gw-04 'docker logs clawmates_server_1 2>&1 | grep -E "per-mission runtime image|drained .* tool call"'
ssh gw-04 'docker exec <cm-runtime-mission-…> cat /root/toolhooks/tap/tools.jsonl | head'
```
If tools appear in `mission_events`, the loop is closed. If not, check the
three failure shapes listed above — each looked correct from every other
angle.
2. **Now that tool calls are observable, redo the Skill-Use measurement.**
`docs/SKILL-USE-BASELINE.md` reports Trigger as `not_observable` on the
container tier because there was no tool evidence. There is now.
`mission_events` carries `tool.call` and `file.touch` per phase, so
Trigger can be scored from behaviour instead of prose — which is what the
paper actually measures. This is the single highest-value follow-up: it
turns the baseline from "the first honest number" into a real one.
3. **Give the microVM tier the same treatment, or retire the difference.** It
has the gate and a tap already, but by a different route (`vm_tool_tap`
installs into the guest, `microvm_executor` drains inside the turn). Two
mechanisms for one job is how they drift. Worth folding onto
`container_tool_hooks` once the fleet is back — tank and morpheus have been
offline for over a week, so the microVM tier cannot be tested at all today.
4. **Deploy the door** (`docs/TOOL-CALL-ARCHITECTURE.md` §3). Config, not code:
`/zeroclaw-data/clawmates-mcp.json` plus a door-shaped provider alias. Now
less urgent than it looked — the gate no longer depends on it — but it is
still the precondition for `clawmates_skills` being reachable, and therefore
for skills moving from inlined bodies to progressive disclosure.
5. **Pull upstream's egress policy** — `0db7d999a feat(plugins): add shared
egress policy foundation (#9137)`. We are ~220 commits behind; this is the
one item identified as worth taking, and it is defence for a problem we have
not solved.
**Dropped from this list:** "give the direct-session tier a tap". That tier is
dormant — `CLAWMATES_MISSION_EXECUTOR` is unset in production, so it never
runs. Checking that before building for it saved the work.
## Open decisions that are yours
- **Self-authoring scope.** Agents now apply their own `skill_candidate` items
with no human click (`CLAWMATES_SKILL_SELF_AUTHORING=0` restores the gate).
`identity_refinement` and `brain_consolidation` still wait for a human,
because they change what an agent IS rather than adding a procedure it can
consult. Say if you want those autonomous too.
- **Skill-Use Compliance coverage.** Most skills still score `not_applicable` —
we cannot tell whether they changed anything. `small-focused-commits` and
`tdd-red-green-refactor` are the next candidates and both need the repository
diff rather than the turn text.
## Deliberately not done
- **The mission executor swap** (running turns through `ProviderExecutor` or the
chat `Runtime`). The blockers are structural, not wiring: `cm-runtime`'s
`files` tool rejects absolute paths *by construction*, `shell` runs in a
per-agent sandbox with no mission mount, `ToolContext` carries no path or VM
handle, and approvals key on `(session_id, message_id)`. The cheap fixes
deliver what it was wanted for.
- **`cm-brain` offline tests** — 6 of 9 need live `clawbrainhub.com`. Stubbing
means reproducing an external registry protocol we have no spec for.
- **Graph memory / `clawhdf5-agent`** — in the workspace manifest, used by no
crate. Measure against a baseline before migrating.
## Operational facts that cost time to learn
- The Gitea **actions-log API returns 403** for the token in
`deploy/compose/.env`. Every CI failure this week was debugged blind because
of it. Steps now write to `/tmp/ci-logs` on the runner host as a workaround;
**a token with the `actions` scope** remains the highest-value thing to
obtain.
- **gw-04 uses legacy `docker-compose`**, not the v2 plugin. `docker compose`
fails there.
- **Do not build images by hand on gw-04 while CI may run** — same 150G volume,
and the frontend image build is what loses.
- The server reaches Docker through a **socket proxy** (`DOCKER_HOST`). Use
`container_exec::connect()`, never `connect_with_local_defaults()`.
- Prod auth is **Clerk**; the bootstrap password in `deploy/compose/.env` works
only against the local stack.
## Two corrections made this session, worth remembering
- **"Missions can't call tools at all" was wrong.** They call `Bash` and `Write`
with permissions pre-accepted. The gap was observing and gating, not having.
- **Raw test counts are a bad coverage metric.** They pointed at `cm-safety`,
whose seven tests already covered its critical paths, and missed a Slack
replay hole that let one captured request authenticate forever.
The recurring shape, now seven times over: **a claim in a comment or a doc,
believed and never checked.** Every significant finding this session came from
running the thing rather than reading about it.