docs: CI root cause was an apostrophe, not any of the three theories
Records both real causes (run 490 stomped by an overlapping run; 491-496 killed by an apostrophe closing a single-quoted sh -c block), the guard that now catches the second class locally, and what to check when the in-flight run settles — including that a successful build is the FIRST time these commits reach production. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
8f988739ec
commit
689a5e14a3
+27
-58
@@ -10,74 +10,43 @@ Read `CAPABILITY-REVIEW.md` for the system picture,
|
|||||||
`merge/upstream-v0.8.4` is at `db1c50966`. The local suite is green — 106 test
|
`merge/upstream-v0.8.4` is at `db1c50966`. The local suite is green — 106 test
|
||||||
binaries, zero build errors, frontend builds.
|
binaries, zero build errors, frontend builds.
|
||||||
|
|
||||||
## ⚠ CI IS RED — READ THIS FIRST
|
## CI: root cause found — an apostrophe
|
||||||
|
|
||||||
Pushing `main` auto-deploys (`.gitea/workflows/deploy.yml`: push → test →
|
Runs 490–496 all failed. Two distinct causes, both now fixed:
|
||||||
build → registry → gw-04's 60s timer). **The `test` job is failing, so `build`
|
|
||||||
never runs and NOTHING FROM THESE COMMITS IS DEPLOYED.** Production still serves
|
|
||||||
the previous image. That is a safe state, not a broken one.
|
|
||||||
|
|
||||||
### What is established (verified, not inferred)
|
- **490** — two runs overlapped. Every run began by removing the *shared*
|
||||||
|
container `cm-ci-pg`, so run 491 deleted run 490's database mid-suite. Fixed
|
||||||
|
with a `concurrency` group and a run-scoped container name.
|
||||||
|
- **491–496** — a **single apostrophe**. The Rust step is
|
||||||
|
`docker run … sh -c '…'`, and a comment inside that single-quoted block read
|
||||||
|
`cm-api's vm_tool_gate`. The quote closed, bash died with *"unexpected EOF
|
||||||
|
while looking for matching quote"* **before running anything** — hence an
|
||||||
|
empty log, a breadcrumb showing the step entered and produced nothing, and
|
||||||
|
three wrong theories. I introduced it in the commit that fixed 490.
|
||||||
|
|
||||||
- **The code is fine.** On gw-04 itself, in the CI container, against a
|
`tests/workflow_shell_syntax.rs` now `bash -n`s every `run:` block, so this
|
||||||
CI-shaped Postgres, `cargo test --workspace` exits **0** — captured as an
|
class fails locally instead of six runs later. Negative-controlled.
|
||||||
explicit `CARGO_RC`, not through a pipe. `npm ci`, `typecheck` and `vitest`
|
|
||||||
all pass on that host too.
|
|
||||||
- **The job dies in the "Rust tests" step**, before `cargo` starts: the
|
|
||||||
breadcrumb `/tmp/ci-logs/STEP` read `2-rust` and no `rust.log` was ever
|
|
||||||
created. So the failure is `apt-get`, `git config`, or `docker run` itself.
|
|
||||||
- Run 490 used the **same workflow file** as the last green run (489), so the
|
|
||||||
original breakage was not a workflow change.
|
|
||||||
|
|
||||||
### What is NOT yet known
|
**Verify the deploy landed.** A run was in flight at session end:
|
||||||
|
|
||||||
Which command in that step fails, and why. The Gitea actions-log API returns
|
|
||||||
**403** for the token in `deploy/compose/.env`, so the job log is unreadable;
|
|
||||||
that is why the workflow now writes everything to `/tmp/ci-logs` **on the
|
|
||||||
runner host**, which outlives the container:
|
|
||||||
|
|
||||||
```
|
```
|
||||||
ssh gw-04 'cat /tmp/ci-logs/STEP; cat /tmp/ci-logs/rust-step.log'
|
ssh gw-04 'curl -s "https://git.redclaw.dev/api/v1/repos/osobh/clawmates/actions/tasks?limit=4" | python3 -m json.tool | grep -E "run_number|status|name"'
|
||||||
|
ssh gw-04 'cat /tmp/ci-logs/STEP; tail -40 /tmp/ci-logs/rust-step.log'
|
||||||
```
|
```
|
||||||
|
|
||||||
`rust-step.log` (added in `c02dbe2`) captures the docker run's own stdout and
|
If `build` succeeds, gw-04's 60s timer deploys within a minute — **that is the
|
||||||
stderr. **Read it first.**
|
first time these 12 commits reach production**, including self-authoring ON by
|
||||||
|
default and the PreToolUse gate installing into every microVM mission. Watch
|
||||||
|
the first mission after it.
|
||||||
|
|
||||||
**New and important:** run 495 (`c02dbe2`) failed **without touching
|
If `test` fails again, read `/tmp/ci-logs/rust-step.log` — it now captures the
|
||||||
`/tmp/ci-logs/STEP` at all** — its mtime stayed at run 494's. Runs 493 and 494
|
docker command's own stdout and stderr on the host, which is what was missing
|
||||||
did write it. So 495 died *before step 1*, which is a different failure from
|
the whole time. Do not infer a cause without it.
|
||||||
493/494 dying inside the Rust step. Two distinct failure modes are in play, or
|
|
||||||
495 was cancelled by the new `concurrency: cancel-in-progress` guard (494 and
|
|
||||||
495 were pushed minutes apart) and Gitea reports a cancellation as `failure`.
|
|
||||||
|
|
||||||
**Cheapest next experiment, no code change:** trigger a run via
|
Note the Gitea actions-log API returns **403** for the token in
|
||||||
`workflow_dispatch` with nothing else queued, then watch
|
`deploy/compose/.env`, and the Gitea server is behind Cloudflare on a host not
|
||||||
`ssh gw-04 'cat /tmp/ci-logs/STEP'`. If STEP updates, the pre-step theory is
|
on the tailnet. **A token with the `actions` scope would have made this entire
|
||||||
dead and the Rust step is the target. If it does not, the failure is in
|
detour unnecessary** — it is the single highest-value thing to obtain.
|
||||||
checkout/job setup and `concurrency` is the first thing to remove.
|
|
||||||
|
|
||||||
Note the Gitea server is behind Cloudflare on a host not on the tailnet (it is
|
|
||||||
not web-01 and not gw-04), so reading the job log from its disk was not
|
|
||||||
possible from here. A token with the `actions` scope would make all of this
|
|
||||||
unnecessary — that is probably the single highest-value thing to obtain.
|
|
||||||
|
|
||||||
### Three wrong theories, recorded so they are not re-tried
|
|
||||||
|
|
||||||
1. **node missing in `rust:1.96-slim`** — real (the `vm_tool_gate` shell tests
|
|
||||||
need it) and fixed, but not the cause; run 491 carried the fix and failed.
|
|
||||||
2. **dash vs bash** — the generated gate script is verified correct under dash.
|
|
||||||
3. **Two runs stomping each other** — a real defect (they shared the container
|
|
||||||
name `cm-ci-pg`, and a new run `docker rm -fv`s it), fixed with a
|
|
||||||
concurrency group and a run-scoped name. But run 492 ran alone and failed.
|
|
||||||
|
|
||||||
Each of those was inferred from a red build without reading the failure. Do not
|
|
||||||
add a fourth. Read `rust-step.log`.
|
|
||||||
|
|
||||||
### One mistake worth not repeating
|
|
||||||
|
|
||||||
My first "reproduction" ran `cargo test ... | tail -80` and reported `exit=0`
|
|
||||||
— which was **tail's** exit code. It proved nothing and sent me down two of the
|
|
||||||
theories above. The repo already documents this trap.
|
|
||||||
|
|
||||||
## Then, in order
|
## Then, in order
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user