Four runs failed at `build` with nothing readable — the actions-log API
returns 403 for our token, so "failure" was the whole message. The first
casualty was a DOCS-ONLY commit, which made it look like a code regression
and cost a cycle chasing one.
It was disk. I had been building runtime images on gw-04 while CI ran on the
same host; the frontend image build lost the race. Reproduced afterwards
with space free and it builds clean, and `docker builder prune` reclaimed
34GB (22G free → 57G).
The build job now writes its breadcrumb and a `df -h` snapshot to
/tmp/ci-logs on the runner host, and records which services actually got
pushed. That last one matters: the failing runs had built and pushed
`server` and then aborted on `frontend`, so the registry held a partial set
and `:latest` never moved — which presented as "the deploy did not happen"
three steps later, nowhere near the cause.
Operational note for the next person, me included: building images by hand
on gw-04 competes with CI for disk on the same 150G volume.
Co-Authored-By: Claude Opus 5 <[email protected]>
Runs 491 through 496 failed on one character.
The Rust step is a `docker run … sh -c '…'`. A comment inside that
single-quoted block read `cm-api's vm_tool_gate`, and the apostrophe closed
the quote. Bash died with "unexpected EOF while looking for matching quote"
BEFORE running anything — which is why no log ever appeared, why the
breadcrumb showed the step entered and produced nothing, and why three
separate theories were floated to explain an empty failure.
I introduced it in the commit that installed nodejs, so the fix for run 490
broke every run after it.
Run 490 itself was the stomping: it overlapped run 491, which began by
removing the shared `cm-ci-pg` container out from under it. That is fixed
too, and was a real defect — it was simply not the cause of 491+.
`bash -n` answers this in milliseconds and nothing was running it: a
workflow is not compiled, not linted, and its only feedback is a red build
with a log this deployment cannot read. `tests/workflow_shell_syntax.rs`
now extracts every `run:` block and syntax-checks it, so the failure shows
up before the push rather than six runs later. Gitea's `${{ … }}` is
replaced with a placeholder first — the point is to check OUR quoting, not
to evaluate their templating. Negative control: restoring the apostrophe
fails the test with the file and line.
The block also carries a standing NO APOSTROPHES warning, because the next
person to write a comment there will not be thinking about quoting.
Co-Authored-By: Claude Opus 5 <[email protected]>
The breadcrumb narrowed run 494 to the Rust step, and rust.log did not
exist — so cargo never started. Whatever failed (apt-get, git config, or
docker itself) wrote to the job log, which the actions-log API will not
give us.
The docker run's stdout and stderr now land on the host too, and the step
exits with docker's status.
Co-Authored-By: Claude Opus 5 <[email protected]>
The host-log change proved the job never reaches `cargo test` — rust.log is
absent while /tmp/ci-logs exists. But TWO steps create that directory, so
"the directory exists" does not say how far the job got, and that ambiguity
cost a debugging cycle on its own.
Each step now overwrites /tmp/ci-logs/STEP on entry, so the last value names
the step that died. The postgres step also runs under `set -x`.
Verified manually on gw-04 in the meantime: the postgres step's exact
commands succeed there (STEP_RC=0), as does the whole Rust container
command with cargo's real exit code, as do the three frontend commands. So
the failure is something the job does that running its steps by hand does
not reproduce — which is precisely what a breadcrumb answers and guessing
does not.
Co-Authored-By: Claude Opus 5 <[email protected]>
Three runs failed and I debugged all three blind: Gitea's actions-log API
returns 403 for the token we have, so the only evidence was the word
"failure". I twice inferred a cause from that and was twice wrong — first
node, then dash — and a third theory (two runs stomping each other) was
right about a real defect but not about these failures.
Worse, my own reproduction lied. It ran `cargo test ... | tail -80`, so the
reported exit code was TAIL's. A green pipeline over a red suite is exactly
the trap this repo already documents, and I walked into it while hunting a
red build.
- every step writes its full output to /tmp/ci-logs on the RUNNER HOST,
which outlives the container, so a failure can be read afterwards
- the Rust step captures cargo's status in a variable and exits with it,
with the grep and tail in between — no pipe anywhere near the status
- the frontend step runs npm ci / typecheck / vitest separately, keeps
each status, prints all three tails, and fails if any is non-zero.
Previously a `set -e` abort meant later steps produced no output at all
What is now known, verified on the runner host itself with cargo's real
exit code: `cargo test --workspace` PASSES on gw-04 in the CI container
against a CI-shaped Postgres (CARGO_RC=0), and `npm ci`, `typecheck` and
`vitest` all pass there too. So the failing step is not one of those, and
the next run will say which it is instead of leaving it to be guessed.
Co-Authored-By: Claude Opus 5 <[email protected]>
Runs 490 and 491 both failed `test`. Neither failure was in the code.
Runs 490 and 491 started 16 minutes apart and a full suite takes longer
than that, so they overlapped. The first thing a run does is
`docker rm -fv cm-ci-pg` — a name every run shared — so the newer run
deleted the older run's database mid-suite. Both failed, and the failures
read as test failures.
Verified before changing anything: the exact CI command, on gw-04, against
the same warm cargo volumes and a Postgres started exactly as CI starts it,
passes on 128b423 — as do `npm ci`, `typecheck` and `vitest` on that host.
The code was never the problem.
- `concurrency: deploy-${{ gitea.ref }}` with cancel-in-progress, so runs
on a ref serialize. A superseded run tests a commit that is no longer
the tip; finishing it costs 20 minutes to learn something that no longer
matters.
- the test Postgres is named per run, so overlap cannot corrupt a run even
if the concurrency guard is later removed. Impossible rather than
unlikely.
- `--shm-size=1g` on it. Docker defaults /dev/shm to 64MB and cm-testkit
creates a database per test; Postgres exhausts its parallel-query
segments mid-run and reports `could not resize shared memory segment`
DURING MIGRATIONS, which reads like a schema fault. Hit locally on
2026-08-19; scripts/test-server.sh already carries the same flag.
The lesson is the session's own: I twice inferred a cause from a red build
without reading the failure — first node, then dash — and both were wrong.
The answer came from running the job on the runner's own host.
Co-Authored-By: Claude Opus 5 <[email protected]>
The first push of the PreToolUse gate failed CI, and the reason is a
property of the gate worth fixing rather than a CI quirk.
The hook parses its JSON payload with `node` — no jq in the runtime image,
and node is guaranteed there because Claude Code is a node program. CI runs
`cargo test --workspace` inside `rust:1.96-slim`, which has no node. The
extraction returned nothing, the gate allowed everything, and the two
"blocks" tests failed.
That is correct behaviour with a dangerous appearance. A gate that cannot
read its input must not block the phase — failing closed on a parse error
denies every tool call, which is what an earlier `case`-syntax bug did. But
allowing silently makes an INERT gate indistinguishable from one that simply
matched nothing, which is this codebase's recurring defect exactly.
So the gate now records `inert` when node is absent, still allowing, and a
test pins both halves: exit 0, and the marker written. The host can check
for that file rather than infer a working gate from an absence of denials.
CI installs nodejs so the shell tests exercise the gate instead of its inert
path. Verified in a rust:1.96-slim container: without node the force-push
payload returns 0, with node it returns 2.
Also confirmed the generated script behaves under dash — Linux /bin/sh —
not only under macOS sh. An earlier apparent dash failure was invalid JSON
in the probe command, not the gate.
Full workspace suite green: 106 binaries.
Co-Authored-By: Claude Opus 5 <[email protected]>
`docker rm -f` without -v orphans the anonymous volume the postgres image
declares. cm-testkit creates a database per test, so each CI run left ~2.8 GB
behind: 38 GB of dangling volumes had accumulated on gw-04, most of the 99 GB
-> 23 GB drop in free space over one day.
Note for anyone reaching for `docker volume prune` to clean this up: don't. On
gw-04 the dangling set also contained traefik-acme (Let's Encrypt certificates)
and all three CI cargo caches. Only the anonymous 64-hex volumes were safe to
remove.
Co-Authored-By: Claude Opus 5 <[email protected]>
cm-runtime/cm-sandbox tests shell out to `docker` via std::process, so the
mounted socket alone was not enough — browser_tool failed with
`docker available: NotFound`. Mount the host binary rather than apt-installing
docker.io: the container is fresh every run, so an install would re-download
~100 MB each time and cache nothing.
Verified on gw-04 that a mounted /usr/bin/docker talks to the host daemon
(client=29.1.3 server=29.1.3), and that the agent-*:dev images these tests need
are already present there.
Co-Authored-By: Claude Opus 5 <[email protected]>
cm-files' s3_store test starts a real MinIO via testcontainers. Without the
socket it does not skip — it fails with
`Client(Init(SocketNotFoundError("/var/run/docker.sock")))`, which looks like a
broken test rather than a missing capability. It passed locally only because the
Mac's docker socket was visible to the test process.
Sibling containers testcontainers starts are reachable because the test
container already shares the host network.
Co-Authored-By: Claude Opus 5 <[email protected]>
First run failed in `cargo test --workspace`: "failed to load source for
dependency clawhdf5", preceded by three "spurious network error: invalid packet
line" retries. Two separate causes, both needed:
- libgit2 cannot fetch from Gitea's smart-HTTP. images/server.Dockerfile already
sets CARGO_NET_GIT_FETCH_WITH_CLI for exactly this; the test step did not.
- quantumclaw/clawhdf5 is private (401 anonymous), so the CLI fetch needs a
credential. Supplied via an insteadOf rewrite from a repo secret, so the token
is masked in logs and never committed.
The server image build does not hit this — it builds only clawmates-server,
which does not pull cm-brain's clawhdf5 path.
Co-Authored-By: Claude Opus 5 <[email protected]>
Closes the one manual step left in the pipeline. gw-04 has run
clawmates-deploy.timer every minute since July, pulling :latest and rolling on
drift — the CD half already worked. What was missing was anything that moved
:latest, since the old build host (tank) is packed for the move.
The runner lives on gw-04 because it is the only reachable x86_64 host and prod
images must be linux/amd64: web-01 is aarch64 and the fleet build boxes are
offline. Host executor, capacity 1, so builds serialize rather than competing
with production traffic.
Three details that are not obvious:
- `docker push :latest` does NOT move the tag on this registry once the manifest
exists under another tag. The PUT-the-manifest step is what actually moves it,
and its absence is how a "successful" deploy could leave prod on a stale image.
- The final step verifies the image prod is RUNNING, not the one we pushed. A
green edge on the old image is the failure this pipeline exists to prevent.
- broker is built here too. It had no :latest tag at all, so gw-04's deploy loop
logged a pull failure every single cycle since 2026-08-11.
Also ignore the local env backups: `.env` was ignored but `.env.bak.*` was not,
and those copies hold real credentials.
Co-Authored-By: Claude Opus 5 <[email protected]>
