clawmates/docs/NEXT-SESSION.md

# Where this left off — 2026-09-14 (addendum 2026-09-18 at the end)

Nine days, ~20 commits, and every item on the last handoff's open list is
closed or explained. The platform is in the best-measured state it has been
in. Read the first section, then the open list; the middle is the record.

## Read this first — retrieval works now, and we know why it did not

Mission agents were not fetching their skills. Five matched production runs
— same recipe, same task, same three skills on offer — said this precisely:

| arm | mechanism | fetched |
|---|---|---|
| `index` | `ReadMcpResourceTool` via the MCP door | **1 of 9** |
| `files` | `Read` of `/mission/skills/<name>.md` | **7 of 9** |

The door tool is **deferred** in Claude Code: absent from the agent's default
list until `ToolSearch` loads it. Naming it in the prompt did nothing; telling
the agent to load it first did nothing (verified, `01a09877`: zero
`ToolSearch`, three narratives that never mention skills). `Read` is core,
never deferred, used in every run. So the `files` arm writes every visible
skill into the container at launch and the index points at paths.

**`files` is the code default now** (`skill_delivery::DEFAULT`). `index` and
`inline` stay selectable per mission (`config.skill_delivery`) so the
comparison remains runnable against one binary. The rule that came out of it:
a capability that depends on the model guessing a tool is loadable is not
delivered.

Two more things about skills:

- `always_inject` lives in the skill's frontmatter and the loader restores it
  on boot (proved by forcing the DB column false and watching it come back).
  `workspace-repo-commit-protocol` is the only one marked, and should stay the
  only one: a marked skill leaves the Trigger sample.
- `web-search-triage` has a compliance check now (URLs fetched vs. a primary /
  aggregator host list). All five runs pass it — including the three that
  never opened the skill. The check catches the violation; it cannot tell
  "followed the skill" from "would have done this anyway", and nothing
  mechanical could on this skill.

## What shipped this pass

### The judge's cost, three ways

The z.ai plan for `glm-5.3` emptied twice (08-29, 09-09) and nothing recorded
a single judge token. Three commits, each measured:

1. **The retry storm** (`8d6310f`). A blocked phase re-judged on the 10s sweep
   for 30 minutes — 180 attempts, each up to 13 requests. Now exponential
   backoff (~10 attempts) via `mission_phases.judge_retry_after`, and a 429
   that names its own reset time fails immediately, naming it.
2. **Accounting** (`248948c`, `736b6a9`). `usage_events` gained `provider,
   model, mission_id, requests`; every judge attempt writes a `kind='judge'`
   row, refused requests included. The first rows read `tokens_in = 0`: z.ai
   reports input in `message_delta`, Anthropic in `message_start`. Fixed.
3. **The quadratic term** (this pass). 7 of 9 verdicts ran to the 12-check
   cap, and every round resent every earlier check's output (≤12 KB each)
   whole. Earlier results now compact to an 800-byte head before the next
   round; the round that just ran stays in full. Checks per verdict unchanged.

Ask the plan before it tells you:

```sql
select provider, date_trunc('day', created_at), sum(requests),
       sum(tokens_in), sum(tokens_out)
  from usage_events where provider is not null group by 1, 2 order by 2;
```

### Agent-side spend is visible too (this pass)

The runtime's `done` frame always carried `model` and `provider`; the
executor read only the two token counts. `TurnOutcome` and `StepRecord` now
carry a `Spend` (split + provider + model), `cm_billing::charge` writes it,
and the chat runtime records its requested model (it drives one provider,
no chain, so requested is answered). Bare model names are recorded without a
guessed family.

### Three things that were known and written nowhere (`248948c`)

- `gate.installed` / `gate.absent` mission events — the hook install outcome
  used to go to stderr in a container that is later deleted.
- `gate.inert` — the marker the gate writes when it cannot parse now has a
  production reader (`drain_inert`), not only a unit test.
- Judge `LlmEvent::Usage` was `Ok(_) => {}`.

### Infra

- **ZeroClaw v0.8.5** merged into the fork and deployed — to the persistent
  runtime only, it turned out; see the 09-18 addendum. Missions reached it on
  2026-09-18.
- **Prod was off the tailnet for a day.** Tailscale node-key expiry on
  gw-01/02/04 — staggered by enrolment date, which is the tell. Re-authed,
  key expiry disabled on all five Hetzner nodes, `<node>-pub` aliases in
  `~/.ssh/config` on the public IPs, vault corrected, runbook written
  (`Valhalla/20 Infrastructure/30 Runbooks/tailscale-key-expiry-2026-09.md`).
- **Fleet re-enrolled**: tank + architect online. morpheus reappeared.
- `CLAWMATES_API_ORIGIN` set explicitly; `worker_glm`/`worker_glm5`/
  `worker_kimi` removed from the prod runtime template (byte-identical to
  `worker`, names that promised providers they never used); map routes
  `researcher`/`analyst` to `worker` directly.

## Open, in the order I would take them

1. **The judge runs to its check cap almost every time.** Compaction made
   that cheaper; it did not ask why a research verdict needs 12 commands.
   Watch `requests` per verdict on the next few missions. If it stays at the
   cap, the lever is the judge prompt, not the budget.
2. **microVM tier is unexercised since v0.8.5.** Fleet is online; every run
   this pass was container-tier. Run one microVM mission before assuming the
   upgrade left that path alone.
3. **`files` is n=3.** 7 of 9 is a signal. `structured-paper-summary` went
   unread in 2 of the last 3 runs; the skills section sits 87–90% into the
   prompt. Position is the untested lever.
4. **Compliance checks exist for 6 of 53 skills.** The rest score
   `not_applicable` on that axis forever.
5. **The persistent `clawmates-runtime` container's own config** still names
   `worker_glm`/`worker_kimi` (3 mentions). Not on the mission path; editing
   it restarts the paired runtime.
6. **Prod holds our 7 test missions** with 90-day retention. They are the
   evidence for everything above; wipe them when they stop being.

## State you should know about

- **Prod: 7 missions, 6 completed** (the 7th was the quota casualty). All
  research_only, all the same task — that sameness is what made the arm
  comparison mean anything.
- **Judge quota** resets weekly (last: 2026-09-11 10:01 UTC). With backoff
  and accounting in place a blocked phase can no longer empty it alone; a
  week of missions still can. Check the query above before a batch.
- **Postgres is named differently on each stack.** Locally
  `clawmates-postgres-1` (dashes); on gw-04 `clawmates_postgres_1`
  (underscores). Same for `server`/`frontend`.
- **`target/` is a symlink to `/Volumes/NVMeRAID`**, and that volume went
  away entirely on 2026-09-14 (SIGBUS mid-compile, then "failed to create
  directory target"). Build with `CARGO_TARGET_DIR=$HOME/cargo-target-clawmates`
  until it is back. It is the drive, not the code.
- **The Mac kills background processes under memory pressure** — four
  watchers and Tailscale this pass. Long polls belong on gw-04 (`nohup`), not
  here.
- **gw-04 is reachable two ways**: `ssh gw-04` (Tailscale) and `ssh gw-04-pub`
  (public IP, `204.168.133.187`). It is NOT on the Hetzner private net; the
  web-01 back door cannot reach it.

## Deliberately not done

- Routing any agent role to GLM. The z.ai plan is the judge's, and the judge
  is the one consumer whose spend is now measured. The design for a real
  `claude_cli.glm` route is in `deploy/clawmates-runtime/agent.config.example.toml`,
  commented out, with the reason it does not work as a TOML sub-table.
- Marking more skills `always_inject`. See above.
- The mission executor swap; a tap for the direct-session tier; `cm-brain`
  offline tests — unchanged from prior handoffs.

## Addendum — 2026-09-18

**Every version claim above was about the wrong container.** Missions are
created from `CLAWMATES_RUNTIME_IMAGE`, which pointed at
`clawmates-runtime:hooks` (zeroclaw 0.8.4, Claude Code 2.1.237, built 08-21)
on both stacks until today. The v0.8.5 image only ever ran the persistent
`clawmates-runtime`, which container-tier missions do not drive turns through.
Every measured mission this month ran on `:hooks`. The comparisons stand — one
image throughout — but "no regressions from v0.8.5" and "attribution survives
2.1.263" described a container missions never touched. Memory corrected.

Now: prod missions run `clawmates-runtime:v085-cc276` (zeroclaw 0.8.5 /
Claude Code 2.1.276 / Kimi 0.41.0), set in `/opt/clawmates/.env`; local runs
`:toolchain` from the same lineage. The Dockerfile pins both CLIs as ARGs —
2.1.265 and 2.1.275 each broke every turn on `ANTHROPIC_BASE_URL` endpoints,
so floating was never safe. Verified by two local canaries and prod mission
`01a0b58e`: arguments on 82/82 calls, 3 spawns → 52 attributed subagent calls,
gate, skill reads, judge independent first pass. Rollback is one line in
`.env` back to `:hooks` (backed up beside it) and a server recreate.

**Egress is closed.** Missions egress from `clawmates_missions` (172.25/16,
`869c3ad`); `clawmates-egress.sh` on gw-04 drops tailnet/private/link-local/ssh
for that subnet in `mangle PREROUTING --ctstate NEW`, survives docker and
tailscaled restarts, and was verified from inside a real mission container.
The server keeps the tailnet. `MISSION-EGRESS.md` has the two wrong turns.

**Also:** `docker-compose.override.yml` is tracked; `DELETE /api/missions`
removes `_outputs/<id>`; prod and local were wiped to zero on 09-14 (the runs
above are the only missions since, all ours).

**Check the MISSION container's binaries** (`docker exec cm-runtime-mission-…
claude --version`), never the persistent runtime's, before attaching a version
to a measurement.