docs: CI root cause was an apostrophe, not any of the three theories
deploy / test (push) Successful in 4m31s
deploy / build (push) Successful in 6m4s

Records both real causes (run 490 stomped by an overlapping run; 491-496
killed by an apostrophe closing a single-quoted sh -c block), the guard that
now catches the second class locally, and what to check when the in-flight
run settles — including that a successful build is the FIRST time these
commits reach production.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-20 07:45:02 -07:00
co-authored by Claude Opus 5
parent 8f988739ec
commit 689a5e14a3
+27 -58
View File
@@ -10,74 +10,43 @@ Read `CAPABILITY-REVIEW.md` for the system picture,
`merge/upstream-v0.8.4` is at `db1c50966`. The local suite is green — 106 test `merge/upstream-v0.8.4` is at `db1c50966`. The local suite is green — 106 test
binaries, zero build errors, frontend builds. binaries, zero build errors, frontend builds.
## ⚠ CI IS RED — READ THIS FIRST ## CI: root cause found — an apostrophe
Pushing `main` auto-deploys (`.gitea/workflows/deploy.yml`: push → test → Runs 490496 all failed. Two distinct causes, both now fixed:
build → registry → gw-04's 60s timer). **The `test` job is failing, so `build`
never runs and NOTHING FROM THESE COMMITS IS DEPLOYED.** Production still serves
the previous image. That is a safe state, not a broken one.
### What is established (verified, not inferred) - **490** — two runs overlapped. Every run began by removing the *shared*
container `cm-ci-pg`, so run 491 deleted run 490's database mid-suite. Fixed
with a `concurrency` group and a run-scoped container name.
- **491496** — a **single apostrophe**. The Rust step is
`docker run … sh -c '…'`, and a comment inside that single-quoted block read
`cm-api's vm_tool_gate`. The quote closed, bash died with *"unexpected EOF
while looking for matching quote"* **before running anything** — hence an
empty log, a breadcrumb showing the step entered and produced nothing, and
three wrong theories. I introduced it in the commit that fixed 490.
- **The code is fine.** On gw-04 itself, in the CI container, against a `tests/workflow_shell_syntax.rs` now `bash -n`s every `run:` block, so this
CI-shaped Postgres, `cargo test --workspace` exits **0** — captured as an class fails locally instead of six runs later. Negative-controlled.
explicit `CARGO_RC`, not through a pipe. `npm ci`, `typecheck` and `vitest`
all pass on that host too.
- **The job dies in the "Rust tests" step**, before `cargo` starts: the
breadcrumb `/tmp/ci-logs/STEP` read `2-rust` and no `rust.log` was ever
created. So the failure is `apt-get`, `git config`, or `docker run` itself.
- Run 490 used the **same workflow file** as the last green run (489), so the
original breakage was not a workflow change.
### What is NOT yet known **Verify the deploy landed.** A run was in flight at session end:
Which command in that step fails, and why. The Gitea actions-log API returns
**403** for the token in `deploy/compose/.env`, so the job log is unreadable;
that is why the workflow now writes everything to `/tmp/ci-logs` **on the
runner host**, which outlives the container:
``` ```
ssh gw-04 'cat /tmp/ci-logs/STEP; cat /tmp/ci-logs/rust-step.log' ssh gw-04 'curl -s "https://git.redclaw.dev/api/v1/repos/osobh/clawmates/actions/tasks?limit=4" | python3 -m json.tool | grep -E "run_number|status|name"'
ssh gw-04 'cat /tmp/ci-logs/STEP; tail -40 /tmp/ci-logs/rust-step.log'
``` ```
`rust-step.log` (added in `c02dbe2`) captures the docker run's own stdout and If `build` succeeds, gw-04's 60s timer deploys within a minute — **that is the
stderr. **Read it first.** first time these 12 commits reach production**, including self-authoring ON by
default and the PreToolUse gate installing into every microVM mission. Watch
the first mission after it.
**New and important:** run 495 (`c02dbe2`) failed **without touching If `test` fails again, read `/tmp/ci-logs/rust-step.log` — it now captures the
`/tmp/ci-logs/STEP` at all** — its mtime stayed at run 494's. Runs 493 and 494 docker command's own stdout and stderr on the host, which is what was missing
did write it. So 495 died *before step 1*, which is a different failure from the whole time. Do not infer a cause without it.
493/494 dying inside the Rust step. Two distinct failure modes are in play, or
495 was cancelled by the new `concurrency: cancel-in-progress` guard (494 and
495 were pushed minutes apart) and Gitea reports a cancellation as `failure`.
**Cheapest next experiment, no code change:** trigger a run via Note the Gitea actions-log API returns **403** for the token in
`workflow_dispatch` with nothing else queued, then watch `deploy/compose/.env`, and the Gitea server is behind Cloudflare on a host not
`ssh gw-04 'cat /tmp/ci-logs/STEP'`. If STEP updates, the pre-step theory is on the tailnet. **A token with the `actions` scope would have made this entire
dead and the Rust step is the target. If it does not, the failure is in detour unnecessary** — it is the single highest-value thing to obtain.
checkout/job setup and `concurrency` is the first thing to remove.
Note the Gitea server is behind Cloudflare on a host not on the tailnet (it is
not web-01 and not gw-04), so reading the job log from its disk was not
possible from here. A token with the `actions` scope would make all of this
unnecessary — that is probably the single highest-value thing to obtain.
### Three wrong theories, recorded so they are not re-tried
1. **node missing in `rust:1.96-slim`** — real (the `vm_tool_gate` shell tests
need it) and fixed, but not the cause; run 491 carried the fix and failed.
2. **dash vs bash** — the generated gate script is verified correct under dash.
3. **Two runs stomping each other** — a real defect (they shared the container
name `cm-ci-pg`, and a new run `docker rm -fv`s it), fixed with a
concurrency group and a run-scoped name. But run 492 ran alone and failed.
Each of those was inferred from a red build without reading the failure. Do not
add a fourth. Read `rust-step.log`.
### One mistake worth not repeating
My first "reproduction" ran `cargo test ... | tail -80` and reported `exit=0`
— which was **tail's** exit code. It proved nothing and sent me down two of the
theories above. The repo already documents this trap.
## Then, in order ## Then, in order