Files
clawmates/docs/NEXT-SESSION.md
T
Omar SobhandClaude Opus 5 93a386e706
deploy / test (push) Successful in 5m26s
deploy / build (push) Successful in 5m32s
perf(judge): earlier check outputs shrink to a reminder before the next round
Measured on prod: 7 of 9 verdicts ran to the 12-check cap. Every round
resends the whole history, and each check's output is bounded at 12 KB — so
by the last round the judge was paying for ~144 KB of outputs it had already
read, on top of up to 120 KB of evidence, and it paid that on every round.
That is the quadratic term in a verdict's cost, and the reason a single
blocked phase could empty a weekly plan.

Before this round's results go in, every earlier tool result compacts to an
800-byte head plus a marker saying the rest was shown when the check ran.
The round that just ran stays whole; a result already carrying the marker is
left alone. The budget of checks is unchanged — each one is cheaper to
remember, not fewer to run.

Also: docs/NEXT-SESSION.md rewritten for the state as of today.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
2026-09-14 08:17:20 -05:00

7.7 KiB
Raw Blame History

Where this left off — 2026-09-14

Nine days, ~20 commits, and every item on the last handoff's open list is closed or explained. The platform is in the best-measured state it has been in. Read the first section, then the open list; the middle is the record.

Read this first — retrieval works now, and we know why it did not

Mission agents were not fetching their skills. Five matched production runs — same recipe, same task, same three skills on offer — said this precisely:

arm mechanism fetched
index ReadMcpResourceTool via the MCP door 1 of 9
files Read of /mission/skills/<name>.md 7 of 9

The door tool is deferred in Claude Code: absent from the agent's default list until ToolSearch loads it. Naming it in the prompt did nothing; telling the agent to load it first did nothing (verified, 01a09877: zero ToolSearch, three narratives that never mention skills). Read is core, never deferred, used in every run. So the files arm writes every visible skill into the container at launch and the index points at paths.

files is the code default now (skill_delivery::DEFAULT). index and inline stay selectable per mission (config.skill_delivery) so the comparison remains runnable against one binary. The rule that came out of it: a capability that depends on the model guessing a tool is loadable is not delivered.

Two more things about skills:

  • always_inject lives in the skill's frontmatter and the loader restores it on boot (proved by forcing the DB column false and watching it come back). workspace-repo-commit-protocol is the only one marked, and should stay the only one: a marked skill leaves the Trigger sample.
  • web-search-triage has a compliance check now (URLs fetched vs. a primary / aggregator host list). All five runs pass it — including the three that never opened the skill. The check catches the violation; it cannot tell "followed the skill" from "would have done this anyway", and nothing mechanical could on this skill.

What shipped this pass

The judge's cost, three ways

The z.ai plan for glm-5.3 emptied twice (08-29, 09-09) and nothing recorded a single judge token. Three commits, each measured:

  1. The retry storm (8d6310f). A blocked phase re-judged on the 10s sweep for 30 minutes — 180 attempts, each up to 13 requests. Now exponential backoff (~10 attempts) via mission_phases.judge_retry_after, and a 429 that names its own reset time fails immediately, naming it.
  2. Accounting (248948c, 736b6a9). usage_events gained provider, model, mission_id, requests; every judge attempt writes a kind='judge' row, refused requests included. The first rows read tokens_in = 0: z.ai reports input in message_delta, Anthropic in message_start. Fixed.
  3. The quadratic term (this pass). 7 of 9 verdicts ran to the 12-check cap, and every round resent every earlier check's output (≤12 KB each) whole. Earlier results now compact to an 800-byte head before the next round; the round that just ran stays in full. Checks per verdict unchanged.

Ask the plan before it tells you:

select provider, date_trunc('day', created_at), sum(requests),
       sum(tokens_in), sum(tokens_out)
  from usage_events where provider is not null group by 1, 2 order by 2;

Agent-side spend is visible too (this pass)

The runtime's done frame always carried model and provider; the executor read only the two token counts. TurnOutcome and StepRecord now carry a Spend (split + provider + model), cm_billing::charge writes it, and the chat runtime records its requested model (it drives one provider, no chain, so requested is answered). Bare model names are recorded without a guessed family.

Three things that were known and written nowhere (248948c)

  • gate.installed / gate.absent mission events — the hook install outcome used to go to stderr in a container that is later deleted.
  • gate.inert — the marker the gate writes when it cannot parse now has a production reader (drain_inert), not only a unit test.
  • Judge LlmEvent::Usage was Ok(_) => {}.

Infra

  • ZeroClaw v0.8.5 merged into the fork and deployed. No regressions.
  • Prod was off the tailnet for a day. Tailscale node-key expiry on gw-01/02/04 — staggered by enrolment date, which is the tell. Re-authed, key expiry disabled on all five Hetzner nodes, <node>-pub aliases in ~/.ssh/config on the public IPs, vault corrected, runbook written (Valhalla/20 Infrastructure/30 Runbooks/tailscale-key-expiry-2026-09.md).
  • Fleet re-enrolled: tank + architect online. morpheus reappeared.
  • CLAWMATES_API_ORIGIN set explicitly; worker_glm/worker_glm5/ worker_kimi removed from the prod runtime template (byte-identical to worker, names that promised providers they never used); map routes researcher/analyst to worker directly.

Open, in the order I would take them

  1. The judge runs to its check cap almost every time. Compaction made that cheaper; it did not ask why a research verdict needs 12 commands. Watch requests per verdict on the next few missions. If it stays at the cap, the lever is the judge prompt, not the budget.
  2. microVM tier is unexercised since v0.8.5. Fleet is online; every run this pass was container-tier. Run one microVM mission before assuming the upgrade left that path alone.
  3. files is n=3. 7 of 9 is a signal. structured-paper-summary went unread in 2 of the last 3 runs; the skills section sits 8790% into the prompt. Position is the untested lever.
  4. Compliance checks exist for 6 of 53 skills. The rest score not_applicable on that axis forever.
  5. The persistent clawmates-runtime container's own config still names worker_glm/worker_kimi (3 mentions). Not on the mission path; editing it restarts the paired runtime.
  6. Prod holds our 7 test missions with 90-day retention. They are the evidence for everything above; wipe them when they stop being.

State you should know about

  • Prod: 7 missions, 6 completed (the 7th was the quota casualty). All research_only, all the same task — that sameness is what made the arm comparison mean anything.
  • Judge quota resets weekly (last: 2026-09-11 10:01 UTC). With backoff and accounting in place a blocked phase can no longer empty it alone; a week of missions still can. Check the query above before a batch.
  • Postgres is named differently on each stack. Locally clawmates-postgres-1 (dashes); on gw-04 clawmates_postgres_1 (underscores). Same for server/frontend.
  • target/ is a symlink to /Volumes/NVMeRAID, and that volume went away entirely on 2026-09-14 (SIGBUS mid-compile, then "failed to create directory target"). Build with CARGO_TARGET_DIR=$HOME/cargo-target-clawmates until it is back. It is the drive, not the code.
  • The Mac kills background processes under memory pressure — four watchers and Tailscale this pass. Long polls belong on gw-04 (nohup), not here.
  • gw-04 is reachable two ways: ssh gw-04 (Tailscale) and ssh gw-04-pub (public IP, 204.168.133.187). It is NOT on the Hetzner private net; the web-01 back door cannot reach it.

Deliberately not done

  • Routing any agent role to GLM. The z.ai plan is the judge's, and the judge is the one consumer whose spend is now measured. The design for a real claude_cli.glm route is in deploy/clawmates-runtime/agent.config.example.toml, commented out, with the reason it does not work as a TOML sub-table.
  • Marking more skills always_inject. See above.
  • The mission executor swap; a tap for the direct-session tier; cm-brain offline tests — unchanged from prior handoffs.