fix(ci): two runs stomped each other, and the logs blamed the tests
Runs 490 and 491 both failed `test`. Neither failure was in the code.
Runs 490 and 491 started 16 minutes apart and a full suite takes longer
than that, so they overlapped. The first thing a run does is
`docker rm -fv cm-ci-pg` — a name every run shared — so the newer run
deleted the older run's database mid-suite. Both failed, and the failures
read as test failures.
Verified before changing anything: the exact CI command, on gw-04, against
the same warm cargo volumes and a Postgres started exactly as CI starts it,
passes on 128b423 — as do `npm ci`, `typecheck` and `vitest` on that host.
The code was never the problem.
- `concurrency: deploy-${{ gitea.ref }}` with cancel-in-progress, so runs
on a ref serialize. A superseded run tests a commit that is no longer
the tip; finishing it costs 20 minutes to learn something that no longer
matters.
- the test Postgres is named per run, so overlap cannot corrupt a run even
if the concurrency guard is later removed. Impossible rather than
unlikely.
- `--shm-size=1g` on it. Docker defaults /dev/shm to 64MB and cm-testkit
creates a database per test; Postgres exhausts its parallel-query
segments mid-run and reports `could not resize shared memory segment`
DURING MIGRATIONS, which reads like a schema fault. Hit locally on
2026-08-19; scripts/test-server.sh already carries the same flag.
The lesson is the session's own: I twice inferred a cause from a red build
without reading the failure — first node, then dash — and both were wrong.
The answer came from running the job on the runner's own host.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
128b423205
commit
72ba4ba523
@@ -21,6 +21,19 @@ on:
|
||||
# Lets you re-run a deploy without an empty commit.
|
||||
workflow_dispatch:
|
||||
|
||||
# Two pushes close together used to STOMP each other. Runs 490 and 491 started
|
||||
# 16 minutes apart, a full suite takes longer than that, and the first thing a
|
||||
# run does is `docker rm -fv` the shared test Postgres — so the newer run
|
||||
# deleted the older run's database mid-suite and both failed. Nothing in the
|
||||
# code was wrong; the logs blamed the tests.
|
||||
#
|
||||
# `cancel-in-progress` because a superseded run is testing a commit that is no
|
||||
# longer the tip: finishing it costs 20 minutes to learn something that no
|
||||
# longer matters.
|
||||
concurrency:
|
||||
group: deploy-${{ gitea.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
REGISTRY: 100.94.185.103:5000
|
||||
NAMESPACE: clawmates
|
||||
@@ -28,6 +41,9 @@ env:
|
||||
jobs:
|
||||
test:
|
||||
runs-on: gw04
|
||||
env:
|
||||
# Shared by the start and stop steps.
|
||||
PG: cm-ci-pg-${{ gitea.run_id }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
@@ -37,15 +53,27 @@ jobs:
|
||||
# only means "no database here".
|
||||
- name: Start test Postgres
|
||||
run: |
|
||||
docker rm -fv cm-ci-pg 2>/dev/null || true
|
||||
docker run -d --name cm-ci-pg \
|
||||
# Run-scoped name. `cm-ci-pg` was shared by every run, so a second
|
||||
# run removed the first one's database while it was still being used.
|
||||
# The concurrency group above should prevent overlap; this makes the
|
||||
# failure impossible rather than merely unlikely.
|
||||
docker rm -fv "$PG" 2>/dev/null || true
|
||||
# --shm-size: Docker defaults /dev/shm to 64MB. cm-testkit creates a
|
||||
# database per test and the suite runs many at once, so Postgres
|
||||
# exhausts its parallel-query segments mid-run. It surfaces as
|
||||
# `could not resize shared memory segment ... No space left on device`
|
||||
# during MIGRATIONS, which reads like a schema fault and is not one.
|
||||
# Hit locally on 2026-08-19; scripts/test-server.sh carries the same
|
||||
# flag for the same reason.
|
||||
docker run -d --name "$PG" \
|
||||
--shm-size=1g \
|
||||
-e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres \
|
||||
-p 127.0.0.1:55432:5432 postgres:16-alpine
|
||||
for i in $(seq 1 30); do
|
||||
docker exec cm-ci-pg pg_isready -U postgres >/dev/null 2>&1 && break
|
||||
docker exec "$PG" pg_isready -U postgres >/dev/null 2>&1 && break
|
||||
sleep 2
|
||||
done
|
||||
docker exec cm-ci-pg pg_isready -U postgres
|
||||
docker exec "$PG" pg_isready -U postgres
|
||||
|
||||
# Rust lives in a container because gw-04 has no cargo. The named volumes
|
||||
# are the whole reason this is not painfully slow: without them every run
|
||||
@@ -105,7 +133,7 @@ jobs:
|
||||
# 38 GB of leaked volumes before anyone noticed.
|
||||
- name: Stop test Postgres
|
||||
if: always()
|
||||
run: docker rm -fv cm-ci-pg 2>/dev/null || true
|
||||
run: docker rm -fv "$PG" 2>/dev/null || true
|
||||
|
||||
# node 22 is on the host, so these run directly.
|
||||
- name: Frontend checks
|
||||
|
||||
Reference in New Issue
Block a user