docs: record the CI investigation honestly, including what is still unknown

Establishes what is verified (the code passes on the runner host, with
cargo's real exit code), what is narrowed (493/494 die inside the Rust step
before cargo starts; 495 died before step 1), the three theories that were
wrong, and the cheapest next experiment.

Also records the two things that made this expensive: the actions-log API
returns 403 for our token, and my first reproduction piped cargo into `tail`
and reported tail's exit code.

Co-Authored-By: Claude Opus 5 <[email protected]>

docs/NEXT-SESSION.md

@@ -12,80 +12,72 @@ binaries, zero build errors, frontend builds.
 
 ## ⚠ CI IS RED — READ THIS FIRST
 
-Pushing `main` auto-deploys (`.gitea/workflows/deploy.yml`: push → test → build
+Pushing `main` auto-deploys (`.gitea/workflows/deploy.yml`: push → test →
-→ registry → gw-04's 60s timer). **The `test` job is failing, so the `build` job
+build → registry → gw-04's 60s timer). **The `test` job is failing, so `build`
-has not run and NOTHING FROM THESE COMMITS IS DEPLOYED.** Production is still
+never runs and NOTHING FROM THESE COMMITS IS DEPLOYED.** Production still serves
-serving the previous image. That is a safe state, not a broken one — but it
+the previous image. That is a safe state, not a broken one.
-means the work is pushed and not live.
 
-Two runs failed: 490 (`b653dbf`) and 491 (`128b423`).
+### What is established (verified, not inferred)
 
-**Known and fixed:** run 490 failed because `vm_tool_gate`'s shell tests execute
+- **The code is fine.** On gw-04 itself, in the CI container, against a
-the generated hook, which parses its payload with `node`, and CI's
+  CI-shaped Postgres, `cargo test --workspace` exits **0** — captured as an
-`rust:1.96-slim` has no node. `deploy.yml` now installs `nodejs`, and the gate
+  explicit `CARGO_RC`, not through a pipe. `npm ci`, `typecheck` and `vitest`
-records an `inert` marker when node is missing rather than silently allowing.
+  all pass on that host too.
-Verified in a `rust:1.96-slim` container: without node the force-push payload
+- **The job dies in the "Rust tests" step**, before `cargo` starts: the
-returns 0, with node it returns 2.
+  breadcrumb `/tmp/ci-logs/STEP` read `2-rust` and no `rust.log` was ever
+  created. So the failure is `apt-get`, `git config`, or `docker run` itself.
+- Run 490 used the **same workflow file** as the last green run (489), so the
+  original breakage was not a workflow change.
 
-**Still unexplained:** run 491 carried that fix and failed anyway. The cause was
+### What is NOT yet known
-NOT determined. Do not guess at it — that is what produced two bad diagnoses
-already this session.
 
-To get the actual failure, reproduce the CI job locally (the Gitea logs API
+Which command in that step fails, and why. The Gitea actions-log API returns
-returns 403 with the token in `deploy/compose/.env`, which lacks the actions
+**403** for the token in `deploy/compose/.env`, so the job log is unreadable;
-scope):
+that is why the workflow now writes everything to `/tmp/ci-logs` **on the
+runner host**, which outlives the container:
 
 ```
-docker network create cm-ci-net
+ssh gw-04 'cat /tmp/ci-logs/STEP; cat /tmp/ci-logs/rust-step.log'
-docker run -d --name cm-ci-pg-local --network cm-ci-net \
-  -e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres --shm-size=1g postgres:16-alpine
-docker run --rm --network cm-ci-net -v "$PWD":/w -w /w \
-  -e SQLX_OFFLINE=true -e CARGO_NET_GIT_FETCH_WITH_CLI=true \
-  -e FORGE_TOKEN=<GITEA_TOKEN from deploy/compose/.env> \
-  -e CM_TEST_DATABASE_URL=postgres://postgres:postgres@cm-ci-pg-local:5432/postgres \
-  rust:1.96-slim sh -c 'apt-get update -qq &&
-    apt-get install -y -qq pkg-config libssl-dev cmake git nodejs &&
-    git config --global url."https://oauth2:[email protected]/".insteadOf "https://git.redclaw.dev/" &&
-    cargo test --workspace --no-fail-fast'
 ```
 
-A run of exactly this was in flight when the session ended; its output was going
+`rust-step.log` (added in `c02dbe2`) captures the docker run's own stdout and
-to `/tmp/cilocal.log`.
+stderr. **Read it first.**
 
-Candidates worth checking first, none confirmed:
+**New and important:** run 495 (`c02dbe2`) failed **without touching
-- something in the new tests that is Linux- or dash-specific beyond the node
+`/tmp/ci-logs/STEP` at all** — its mtime stayed at run 494's. Runs 493 and 494
-  issue (the gate script itself IS verified under dash);
+did write it. So 495 died *before step 1*, which is a different failure from
-- `--shm-size` — the local test Postgres needed 1g because the suite exhausts
+493/494 dying inside the Rust step. Two distinct failure modes are in play, or
-  Docker's 64MB default mid-run, and CI's `cm-ci-pg` sets no `--shm-size`. This
+495 was cancelled by the new `concurrency: cancel-in-progress` guard (494 and
-  surfaces as `could not resize shared memory segment … No space left on device`
+495 were pushed minutes apart) and Gitea reports a cancellation as `failure`.
-  **during migrations**, which reads like a schema fault and is not one. If that
-  is the cause, the fix is `--shm-size=1g` on CI's postgres, not on our code.
 
-## The thing to do first
+**Cheapest next experiment, no code change:** trigger a run via
+`workflow_dispatch` with nothing else queued, then watch
+`ssh gw-04 'cat /tmp/ci-logs/STEP'`. If STEP updates, the pre-step theory is
+dead and the Rust step is the target. If it does not, the failure is in
+checkout/job setup and `concurrency` is the first thing to remove.
 
-**A runtime image with the `stream-json` fix is built and NOT deployed.**
+Note the Gitea server is behind Cloudflare on a host not on the tailnet (it is
+not web-01 and not gw-04), so reading the job log from its disk was not
+possible from here. A token with the `actions` scope would make all of this
+unnecessary — that is probably the single highest-value thing to obtain.
 
-```
+### Three wrong theories, recorded so they are not re-tried
-clawmates-runtime:streamjson     # built 2026-08-19, never run
-```
 
-Nothing has yet confirmed end to end that real `tool.call` rows land in
+1. **node missing in `rust:1.96-slim`** — real (the `vm_tool_gate` shell tests
-`mission_events` from a live mission. That confirmation is the whole point of
+   need it) and fixed, but not the cause; run 491 carried the fix and failed.
-the change and it is the one step not taken. Until it runs, treat the fix as
+2. **dash vs bash** — the generated gate script is verified correct under dash.
-plausible rather than proven — the parser is unit-tested against a captured
+3. **Two runs stomping each other** — a real defect (they shared the container
-real stream, but no mission has driven it.
+   name `cm-ci-pg`, and a new run `docker rm -fv`s it), fixed with a
+   concurrency group and a run-scoped name. But run 492 ran alone and failed.
 
-To do it: deploy the image as the mission runtime, launch a mission, then
+Each of those was inferred from a red build without reading the failure. Do not
+add a fourth. Read `rust-step.log`.
 
-```sql
+### One mistake worth not repeating
-SELECT kind, target, count(*) FROM mission_events
- WHERE mission_id = '<id>' GROUP BY 1,2;
-```
 
-Expect `tool.call` rows on the container tier for the first time. If they do not
+My first "reproduction" ran `cargo test ... | tail -80` and reported `exit=0`
-appear, the next suspects are `topology_exec`'s frame parser and whether the
+— which was **tail's** exit code. It proved nothing and sent me down two of the
-gateway forwards the provider's `tool_calls` at all — neither has been exercised
+theories above. The repo already documents this trap.
-with a non-empty list.
 
 ## Then, in order