fix(ci): two runs stomped each other, and the logs blamed the tests

Runs 490 and 491 both failed `test`. Neither failure was in the code.

Runs 490 and 491 started 16 minutes apart and a full suite takes longer
than that, so they overlapped. The first thing a run does is
`docker rm -fv cm-ci-pg` — a name every run shared — so the newer run
deleted the older run's database mid-suite. Both failed, and the failures
read as test failures.

Verified before changing anything: the exact CI command, on gw-04, against
the same warm cargo volumes and a Postgres started exactly as CI starts it,
passes on 128b423 — as do `npm ci`, `typecheck` and `vitest` on that host.
The code was never the problem.

  - `concurrency: deploy-${{ gitea.ref }}` with cancel-in-progress, so runs
    on a ref serialize. A superseded run tests a commit that is no longer
    the tip; finishing it costs 20 minutes to learn something that no longer
    matters.
  - the test Postgres is named per run, so overlap cannot corrupt a run even
    if the concurrency guard is later removed. Impossible rather than
    unlikely.
  - `--shm-size=1g` on it. Docker defaults /dev/shm to 64MB and cm-testkit
    creates a database per test; Postgres exhausts its parallel-query
    segments mid-run and reports `could not resize shared memory segment`
    DURING MIGRATIONS, which reads like a schema fault. Hit locally on
    2026-08-19; scripts/test-server.sh already carries the same flag.

The lesson is the session's own: I twice inferred a cause from a red build
without reading the failure — first node, then dash — and both were wrong.
The answer came from running the job on the runner's own host.

Co-Authored-By: Claude Opus 5 <[email protected]>

.gitea/workflows/deploy.yml

@@ -21,6 +21,19 @@ on:
   # Lets you re-run a deploy without an empty commit.
   workflow_dispatch:
 
+# Two pushes close together used to STOMP each other. Runs 490 and 491 started
+# 16 minutes apart, a full suite takes longer than that, and the first thing a
+# run does is `docker rm -fv` the shared test Postgres — so the newer run
+# deleted the older run's database mid-suite and both failed. Nothing in the
+# code was wrong; the logs blamed the tests.
+#
+# `cancel-in-progress` because a superseded run is testing a commit that is no
+# longer the tip: finishing it costs 20 minutes to learn something that no
+# longer matters.
+concurrency:
+  group: deploy-${{ gitea.ref }}
+  cancel-in-progress: true
+
 env:
   REGISTRY: 100.94.185.103:5000
   NAMESPACE: clawmates
@@ -28,6 +41,9 @@ env:
 jobs:
   test:
     runs-on: gw04
+    env:
+      # Shared by the start and stop steps.
+      PG: cm-ci-pg-${{ gitea.run_id }}
     steps:
       - uses: actions/checkout@v4
 
@@ -37,15 +53,27 @@ jobs:
       # only means "no database here".
       - name: Start test Postgres
         run: |
-          docker rm -fv cm-ci-pg 2>/dev/null || true
+          # Run-scoped name. `cm-ci-pg` was shared by every run, so a second
-          docker run -d --name cm-ci-pg \
+          # run removed the first one's database while it was still being used.
+          # The concurrency group above should prevent overlap; this makes the
+          # failure impossible rather than merely unlikely.
+          docker rm -fv "$PG" 2>/dev/null || true
+          # --shm-size: Docker defaults /dev/shm to 64MB. cm-testkit creates a
+          # database per test and the suite runs many at once, so Postgres
+          # exhausts its parallel-query segments mid-run. It surfaces as
+          # `could not resize shared memory segment ... No space left on device`
+          # during MIGRATIONS, which reads like a schema fault and is not one.
+          # Hit locally on 2026-08-19; scripts/test-server.sh carries the same
+          # flag for the same reason.
+          docker run -d --name "$PG" \
+            --shm-size=1g \
             -e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres \
             -p 127.0.0.1:55432:5432 postgres:16-alpine
           for i in $(seq 1 30); do
-            docker exec cm-ci-pg pg_isready -U postgres >/dev/null 2>&1 && break
+            docker exec "$PG" pg_isready -U postgres >/dev/null 2>&1 && break
             sleep 2
           done
-          docker exec cm-ci-pg pg_isready -U postgres
+          docker exec "$PG" pg_isready -U postgres
 
       # Rust lives in a container because gw-04 has no cargo. The named volumes
       # are the whole reason this is not painfully slow: without them every run
@@ -105,7 +133,7 @@ jobs:
       # 38 GB of leaked volumes before anyone noticed.
       - name: Stop test Postgres
         if: always()
-        run: docker rm -fv cm-ci-pg 2>/dev/null || true
+        run: docker rm -fv "$PG" 2>/dev/null || true
 
       # node 22 is on the host, so these run directly.
       - name: Frontend checks

docs/NEXT-SESSION.md

@@ -6,12 +6,60 @@ Read `CAPABILITY-REVIEW.md` for the system picture,
 
 ## State of the tree
 
-**11 unpushed commits on `main`** (`origin/main` is 11 behind). Nothing has been
+**Everything is pushed.** `main` is at `128b423`; the fork's
-pushed and nothing is deployed to gw-04. Full suite green: 106 test binaries,
+`merge/upstream-v0.8.4` is at `db1c50966`. The local suite is green — 106 test
-zero build errors, frontend builds.
+binaries, zero build errors, frontend builds.
 
-**1 unpushed commit in the fork** at `/Users/quantum/projects/zeroclaw`, branch
+## ⚠ CI IS RED — READ THIS FIRST
-`merge/upstream-v0.8.4`: `db1c50966 feat(claude_cli): stream-json …`.
+
+Pushing `main` auto-deploys (`.gitea/workflows/deploy.yml`: push → test → build
+→ registry → gw-04's 60s timer). **The `test` job is failing, so the `build` job
+has not run and NOTHING FROM THESE COMMITS IS DEPLOYED.** Production is still
+serving the previous image. That is a safe state, not a broken one — but it
+means the work is pushed and not live.
+
+Two runs failed: 490 (`b653dbf`) and 491 (`128b423`).
+
+**Known and fixed:** run 490 failed because `vm_tool_gate`'s shell tests execute
+the generated hook, which parses its payload with `node`, and CI's
+`rust:1.96-slim` has no node. `deploy.yml` now installs `nodejs`, and the gate
+records an `inert` marker when node is missing rather than silently allowing.
+Verified in a `rust:1.96-slim` container: without node the force-push payload
+returns 0, with node it returns 2.
+
+**Still unexplained:** run 491 carried that fix and failed anyway. The cause was
+NOT determined. Do not guess at it — that is what produced two bad diagnoses
+already this session.
+
+To get the actual failure, reproduce the CI job locally (the Gitea logs API
+returns 403 with the token in `deploy/compose/.env`, which lacks the actions
+scope):
+
+```
+docker network create cm-ci-net
+docker run -d --name cm-ci-pg-local --network cm-ci-net \
+  -e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres --shm-size=1g postgres:16-alpine
+docker run --rm --network cm-ci-net -v "$PWD":/w -w /w \
+  -e SQLX_OFFLINE=true -e CARGO_NET_GIT_FETCH_WITH_CLI=true \
+  -e FORGE_TOKEN=<GITEA_TOKEN from deploy/compose/.env> \
+  -e CM_TEST_DATABASE_URL=postgres://postgres:postgres@cm-ci-pg-local:5432/postgres \
+  rust:1.96-slim sh -c 'apt-get update -qq &&
+    apt-get install -y -qq pkg-config libssl-dev cmake git nodejs &&
+    git config --global url."https://oauth2:[email protected]/".insteadOf "https://git.redclaw.dev/" &&
+    cargo test --workspace --no-fail-fast'
+```
+
+A run of exactly this was in flight when the session ended; its output was going
+to `/tmp/cilocal.log`.
+
+Candidates worth checking first, none confirmed:
+- something in the new tests that is Linux- or dash-specific beyond the node
+  issue (the gate script itself IS verified under dash);
+- `--shm-size` — the local test Postgres needed 1g because the suite exhausts
+  Docker's 64MB default mid-run, and CI's `cm-ci-pg` sets no `--shm-size`. This
+  surfaces as `could not resize shared memory segment … No space left on device`
+  **during migrations**, which reads like a schema fault and is not one. If that
+  is the cause, the fix is `--shm-size=1g` on CI's postgres, not on our code.
 
 ## The thing to do first