clawmates/docs/NEXT-SESSION.md

# Where this left off — 2026-09-14 (addendum 2026-09-18 at the end)

Nine days, ~20 commits, and every item on the last handoff's open list is
closed or explained. The platform is in the best-measured state it has been
in. Read the first section, then the open list; the middle is the record.

## Read this first — retrieval works now, and we know why it did not

Mission agents were not fetching their skills. Five matched production runs
— same recipe, same task, same three skills on offer — said this precisely:

| arm | mechanism | fetched |
|---|---|---|
| `index` | `ReadMcpResourceTool` via the MCP door | **1 of 9** |
| `files` | `Read` of `/mission/skills/<name>.md` | **7 of 9** |

The door tool is **deferred** in Claude Code: absent from the agent's default
list until `ToolSearch` loads it. Naming it in the prompt did nothing; telling
the agent to load it first did nothing (verified, `01a09877`: zero
`ToolSearch`, three narratives that never mention skills). `Read` is core,
never deferred, used in every run. So the `files` arm writes every visible
skill into the container at launch and the index points at paths.

**`files` is the code default now** (`skill_delivery::DEFAULT`). `index` and
`inline` stay selectable per mission (`config.skill_delivery`) so the
comparison remains runnable against one binary. The rule that came out of it:
a capability that depends on the model guessing a tool is loadable is not
delivered.

Two more things about skills:

- `always_inject` lives in the skill's frontmatter and the loader restores it
  on boot (proved by forcing the DB column false and watching it come back).
  `workspace-repo-commit-protocol` is the only one marked, and should stay the
  only one: a marked skill leaves the Trigger sample.
- `web-search-triage` has a compliance check now (URLs fetched vs. a primary /
  aggregator host list). All five runs pass it — including the three that
  never opened the skill. The check catches the violation; it cannot tell
  "followed the skill" from "would have done this anyway", and nothing
  mechanical could on this skill.

## What shipped this pass

### The judge's cost, three ways

The z.ai plan for `glm-5.3` emptied twice (08-29, 09-09) and nothing recorded
a single judge token. Three commits, each measured:

1. **The retry storm** (`8d6310f`). A blocked phase re-judged on the 10s sweep
   for 30 minutes — 180 attempts, each up to 13 requests. Now exponential
   backoff (~10 attempts) via `mission_phases.judge_retry_after`, and a 429
   that names its own reset time fails immediately, naming it.
2. **Accounting** (`248948c`, `736b6a9`). `usage_events` gained `provider,
   model, mission_id, requests`; every judge attempt writes a `kind='judge'`
   row, refused requests included. The first rows read `tokens_in = 0`: z.ai
   reports input in `message_delta`, Anthropic in `message_start`. Fixed.
3. **The quadratic term** (this pass). 7 of 9 verdicts ran to the 12-check
   cap, and every round resent every earlier check's output (≤12 KB each)
   whole. Earlier results now compact to an 800-byte head before the next
   round; the round that just ran stays in full. Checks per verdict unchanged.

Ask the plan before it tells you:

```sql
select provider, date_trunc('day', created_at), sum(requests),
       sum(tokens_in), sum(tokens_out)
  from usage_events where provider is not null group by 1, 2 order by 2;
```

### Agent-side spend is visible too (this pass)

The runtime's `done` frame always carried `model` and `provider`; the
executor read only the two token counts. `TurnOutcome` and `StepRecord` now
carry a `Spend` (split + provider + model), `cm_billing::charge` writes it,
and the chat runtime records its requested model (it drives one provider,
no chain, so requested is answered). Bare model names are recorded without a
guessed family.

### Three things that were known and written nowhere (`248948c`)

- `gate.installed` / `gate.absent` mission events — the hook install outcome
  used to go to stderr in a container that is later deleted.
- `gate.inert` — the marker the gate writes when it cannot parse now has a
  production reader (`drain_inert`), not only a unit test.
- Judge `LlmEvent::Usage` was `Ok(_) => {}`.

### Infra

- **ZeroClaw v0.8.5** merged into the fork and deployed — to the persistent
  runtime only, it turned out; see the 09-18 addendum. Missions reached it on
  2026-09-18.
- **Prod was off the tailnet for a day.** Tailscale node-key expiry on
  gw-01/02/04 — staggered by enrolment date, which is the tell. Re-authed,
  key expiry disabled on all five Hetzner nodes, `<node>-pub` aliases in
  `~/.ssh/config` on the public IPs, vault corrected, runbook written
  (`Valhalla/20 Infrastructure/30 Runbooks/tailscale-key-expiry-2026-09.md`).
- **Fleet re-enrolled**: tank + architect online. morpheus reappeared.
- `CLAWMATES_API_ORIGIN` set explicitly; `worker_glm`/`worker_glm5`/
  `worker_kimi` removed from the prod runtime template (byte-identical to
  `worker`, names that promised providers they never used); map routes
  `researcher`/`analyst` to `worker` directly.

## Open, in the order I would take them (rewritten 2026-09-20)

Everything on the 09-14 list is closed or explained; see the addenda. What is
genuinely left:

1. **The judge costs ~9 requests / ~20 K input per verdict, and that is now
   legitimate work.** Three measured passes (09-19): the 12 KB output window
   truncated an 18 KB deliverable → 64 KB; then my compaction erased the read
   before the judge could use it → the latest round stays whole; then the
   prompt says a cat is complete, decide first, read once. Result: four cats
   of the deliverables plus cheap greps that VERIFY (placeholders, URL count,
   sources, structure), zero re-reads. The only remaining lever is getting
   the model to batch independent greps in one turn — a behaviour bet.
2. **`files` arm: 8 of 12 across four runs; the same skill is skipped every
   time.** With the section moved to 2% of the prompt (a50c41a) the evidence
   checker still does not open `structured-paper-summary`, whose
   `when_to_use` is "summarising a research paper". That is triage, not a
   miss. Nothing to fix; the scorer already says `not_applicable`.
3. **Compliance checks: 11 of 53.** The remaining 42 are content judgement.
   Adding a check for one means finding a rule that leaves a mark in tool
   arguments or delivered files — the module's own bar, and the right one.
4. **Follow-ups deliberately not taken:** `Spend{provider}` for VM turns
   (requested-not-answered; `charge()` bills a credit for zero tokens — the
   node egress log is the honest source and the harness asserts it); the
   persistent runtime's own config still naming `worker_glm`/`worker_kimi`
   (cosmetic; editing it restarts the paired runtime).
5. **Product/scaling work that was always scoped later:** missions-as-workflows
   #14 (plan viewer + planner mode) and #15 (scheduling + loop-progress
   events); scaling Phase 2/3 (node bring-up automation, Postgres HA,
   autoscale); node-placed terminal cross-container writes.

## State you should know about

- **Prod: 7 missions, 6 completed** (the 7th was the quota casualty). All
  research_only, all the same task — that sameness is what made the arm
  comparison mean anything.
- **Judge quota** resets weekly (last: 2026-09-11 10:01 UTC). With backoff
  and accounting in place a blocked phase can no longer empty it alone; a
  week of missions still can. Check the query above before a batch.
- **Postgres is named differently on each stack.** Locally
  `clawmates-postgres-1` (dashes); on gw-04 `clawmates_postgres_1`
  (underscores). Same for `server`/`frontend`.
- **`target/` is a symlink to `/Volumes/NVMeRAID`**, and that volume went
  away entirely on 2026-09-14 (SIGBUS mid-compile, then "failed to create
  directory target"). Build with `CARGO_TARGET_DIR=$HOME/cargo-target-clawmates`
  until it is back. It is the drive, not the code.
- **The Mac kills background processes under memory pressure** — four
  watchers and Tailscale this pass. Long polls belong on gw-04 (`nohup`), not
  here.
- **gw-04 is reachable two ways**: `ssh gw-04` (Tailscale) and `ssh gw-04-pub`
  (public IP, `204.168.133.187`). It is NOT on the Hetzner private net; the
  web-01 back door cannot reach it.

## Deliberately not done

- Routing any agent role to GLM. The z.ai plan is the judge's, and the judge
  is the one consumer whose spend is now measured. The design for a real
  `claude_cli.glm` route is in `deploy/clawmates-runtime/agent.config.example.toml`,
  commented out, with the reason it does not work as a TOML sub-table.
- Marking more skills `always_inject`. See above.
- The mission executor swap; a tap for the direct-session tier; `cm-brain`
  offline tests — unchanged from prior handoffs.

## Addendum — 2026-09-18

**Every version claim above was about the wrong container.** Missions are
created from `CLAWMATES_RUNTIME_IMAGE`, which pointed at
`clawmates-runtime:hooks` (zeroclaw 0.8.4, Claude Code 2.1.237, built 08-21)
on both stacks until today. The v0.8.5 image only ever ran the persistent
`clawmates-runtime`, which container-tier missions do not drive turns through.
Every measured mission this month ran on `:hooks`. The comparisons stand — one
image throughout — but "no regressions from v0.8.5" and "attribution survives
2.1.263" described a container missions never touched. Memory corrected.

Now: prod missions run `clawmates-runtime:v085-cc276` (zeroclaw 0.8.5 /
Claude Code 2.1.276 / Kimi 0.41.0), set in `/opt/clawmates/.env`; local runs
`:toolchain` from the same lineage. The Dockerfile pins both CLIs as ARGs —
2.1.265 and 2.1.275 each broke every turn on `ANTHROPIC_BASE_URL` endpoints,
so floating was never safe. Verified by two local canaries and prod mission
`01a0b58e`: arguments on 82/82 calls, 3 spawns → 52 attributed subagent calls,
gate, skill reads, judge independent first pass. Rollback is one line in
`.env` back to `:hooks` (backed up beside it) and a server recreate.

**Egress is closed.** Missions egress from `clawmates_missions` (172.25/16,
`869c3ad`); `clawmates-egress.sh` on gw-04 drops tailnet/private/link-local/ssh
for that subnet in `mangle PREROUTING --ctstate NEW`, survives docker and
tailscaled restarts, and was verified from inside a real mission container.
The server keeps the tailnet. `MISSION-EGRESS.md` has the two wrong turns.

**Also:** `docker-compose.override.yml` is tracked; `DELETE /api/missions`
removes `_outputs/<id>`; prod and local were wiped to zero on 09-14 (the runs
above are the only missions since, all ours).

**Check the MISSION container's binaries** (`docker exec cm-runtime-mission-…
claude --version`), never the persistent runtime's, before attaching a version
to a measurement.

## Addendum 2 — 2026-09-18, microVM tier on 2.1.276

GLM and Kimi are microVM-only backends, so "did we upgrade GLM and Kimi" was
this: every rootfs on the fleet had sat on Claude Code 2.1.223–2.1.226 since
August. Now: `images/agent-*` pin **2.1.276** (drift guard in
`fc-build-rootfs.sh`); tank's `claude/glm/kimi/local-ornith` rootfs rebuilt;
`verify-mission-delivery.sh microvm|glm|kimi` all pass with the new
`assert_provider_egress` (the placed node's journal must show dials to that
provider's host, none denied, nothing else reached) and `assert_cli_version`
(`checkpoint.vm` on `topology_runs`, new: rootfs + guest `claude --version`).

The canary caught a real break before promotion: `--agents` `tools` must be a
JSON **array**. We sent a comma string; 2.1.243 turned silently-ignored into a
hard error. So on every earlier CLI the definition was dropped and the
verifier's tool restriction was plausibly never applied. Fixed (`cbc9c2d`).

`evaluator::implementer_family(missions.backend)` replaces the hardcoded
`"anthropic"`: a glm mission is now judged by the subscription and honestly
`independent=true` (`claude-opus-5`); before, glm would have judged glm.

Fleet facts: both nodes on 2.1.276 (architect rebuilt later the same day —
its cargo builds into `/hot/targets/_default`, so pass `FC_AGENT_BIN`);
`rootfs-*.ext4.pre276` backups on tank (~7 GB) and architect (~4 GB), delete
after sign-off;
`rootfs-canary-claude.ext4.promoted-2.1.276` kept as the next candidate slot.
Judge spend this week after the runs: 85 z.ai requests. Prod holds 8 missions,
all ours.

Rebuild trap: rsync copies the Mac's dead `target` symlink to the node —
`rm -f ~/clawmates/target` before `cargo build`.

## Addendum 3 — 2026-09-19/20

- `--agents` `tools` must be a JSON array (cbc9c2d); every CLI before 2.1.243
  dropped the string form silently, so the verifier's tool restriction had
  plausibly never applied on a VM. Caught by the canary.
- `evaluator::implementer_family(missions.backend)` — a glm mission is judged
  by the subscription, honestly independent (794f212).
- `checkpoint.vm` on `topology_runs`: rootfs + guest `claude --version` per
  VM run; `verify-mission-delivery.sh glm|kimi` with `assert_provider_egress`.
- VM tool-gate `denied.jsonl`/`inert` → `gate.denied`/`gate.inert` (9fc904a).
- Judge: 64 KB window (4e342d1), latest round whole + budget prompt (1a44405).
- Skills section second in the prompt (a50c41a); 4 more compliance checks
  (507d744).
- `cm-files` MinIO test pinned to quay.io — Docker Hub deleted `minio/minio`;
  CI passed only on gw-04's year-old cache (5d9edd6).
- **CI trap:** a newer push cancels the in-progress run (Gitea concurrency).
  Four commits in ten minutes looked like three failures and one real one;
  the one real one was environmental (gw-04 at 85% disk, 51 GB build cache —
  reclaimed 44 GB, re-run green). Push, then wait.
- Both stacks wiped to zero 09-19 via the API; `_outputs` came with the
  missions this time. Judge ledger kept (15 rows prod, 3 local).
- Fleet: tank claude/glm/kimi/local-ornith and architect claude/local-ornith
  all on 2.1.276; backups deleted after sign-off.
- **The Mac's Tailscale switched to another tailnet** (`taila5f63e.ts.net`,
  `noc@alucio.net.id`) at some point on 09-19/20 — zero fleet peers from the
  laptop. `ssh gw-04-pub` etc. are the route until it is switched back.