ci: make the build job readable too, and reclaim the disk that broke it
deploy / test (push) Successful in 4m45s
deploy / build (push) Successful in 2m6s

Four runs failed at `build` with nothing readable — the actions-log API
returns 403 for our token, so "failure" was the whole message. The first
casualty was a DOCS-ONLY commit, which made it look like a code regression
and cost a cycle chasing one.

It was disk. I had been building runtime images on gw-04 while CI ran on the
same host; the frontend image build lost the race. Reproduced afterwards
with space free and it builds clean, and `docker builder prune` reclaimed
34GB (22G free → 57G).

The build job now writes its breadcrumb and a `df -h` snapshot to
/tmp/ci-logs on the runner host, and records which services actually got
pushed. That last one matters: the failing runs had built and pushed
`server` and then aborted on `frontend`, so the registry held a partial set
and `:latest` never moved — which presented as "the deploy did not happen"
three steps later, nowhere near the cause.

Operational note for the next person, me included: building images by hand
on gw-04 competes with CI for disk on the same 150G volume.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-21 07:00:43 -07:00
co-authored by Claude Opus 5
parent e84413d437
commit f6e6037aa0
+12
View File
@@ -203,6 +203,15 @@ jobs:
- name: Build + push images
run: |
# Same host-log treatment as the test job. The build job failed four
# runs in a row with nothing readable: the actions-log API returns
# 403 for our token, so "failure" was the entire message. It turned
# out to be transient disk pressure — a runtime image being built on
# this same host at the same time — and a docs-only commit was the
# first casualty, which made it look like a code regression.
mkdir -p /tmp/ci-logs
echo "5-build" > /tmp/ci-logs/STEP
df -h / > /tmp/ci-logs/build-disk.log 2>&1
set -eu
SHA=$(git rev-parse --short HEAD)
echo "SHA=$SHA" >> "$GITHUB_ENV"
@@ -231,6 +240,7 @@ jobs:
-t "$REGISTRY/$NAMESPACE/$svc:latest" .
docker push "$REGISTRY/$NAMESPACE/$svc:main-$SHA"
docker push "$REGISTRY/$NAMESPACE/$svc:latest"
echo "$svc built+pushed" >> /tmp/ci-logs/build-progress.log
done
# `docker push :latest` does NOT reliably move the tag on this registry:
@@ -242,6 +252,7 @@ jobs:
# previously leave prod on a stale image.
- name: Repoint :latest
run: |
echo "6-repoint" > /tmp/ci-logs/STEP
set -eu
for svc in server frontend broker; do
ct=$(curl -s -o /tmp/m.json -D- \
@@ -260,6 +271,7 @@ jobs:
# pipeline exists to prevent.
- name: Wait for the rolling deploy
run: |
echo "7-wait-deploy" > /tmp/ci-logs/STEP
set -eu
want=$(docker image inspect -f '{{.Id}}' "$REGISTRY/$NAMESPACE/server:latest")
for i in $(seq 1 30); do