ci: make the build job readable too, and reclaim the disk that broke it
Four runs failed at `build` with nothing readable — the actions-log API returns 403 for our token, so "failure" was the whole message. The first casualty was a DOCS-ONLY commit, which made it look like a code regression and cost a cycle chasing one. It was disk. I had been building runtime images on gw-04 while CI ran on the same host; the frontend image build lost the race. Reproduced afterwards with space free and it builds clean, and `docker builder prune` reclaimed 34GB (22G free → 57G). The build job now writes its breadcrumb and a `df -h` snapshot to /tmp/ci-logs on the runner host, and records which services actually got pushed. That last one matters: the failing runs had built and pushed `server` and then aborted on `frontend`, so the registry held a partial set and `:latest` never moved — which presented as "the deploy did not happen" three steps later, nowhere near the cause. Operational note for the next person, me included: building images by hand on gw-04 competes with CI for disk on the same 150G volume. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
e84413d437
commit
f6e6037aa0
@@ -203,6 +203,15 @@ jobs:
|
||||
|
||||
- name: Build + push images
|
||||
run: |
|
||||
# Same host-log treatment as the test job. The build job failed four
|
||||
# runs in a row with nothing readable: the actions-log API returns
|
||||
# 403 for our token, so "failure" was the entire message. It turned
|
||||
# out to be transient disk pressure — a runtime image being built on
|
||||
# this same host at the same time — and a docs-only commit was the
|
||||
# first casualty, which made it look like a code regression.
|
||||
mkdir -p /tmp/ci-logs
|
||||
echo "5-build" > /tmp/ci-logs/STEP
|
||||
df -h / > /tmp/ci-logs/build-disk.log 2>&1
|
||||
set -eu
|
||||
SHA=$(git rev-parse --short HEAD)
|
||||
echo "SHA=$SHA" >> "$GITHUB_ENV"
|
||||
@@ -231,6 +240,7 @@ jobs:
|
||||
-t "$REGISTRY/$NAMESPACE/$svc:latest" .
|
||||
docker push "$REGISTRY/$NAMESPACE/$svc:main-$SHA"
|
||||
docker push "$REGISTRY/$NAMESPACE/$svc:latest"
|
||||
echo "$svc built+pushed" >> /tmp/ci-logs/build-progress.log
|
||||
done
|
||||
|
||||
# `docker push :latest` does NOT reliably move the tag on this registry:
|
||||
@@ -242,6 +252,7 @@ jobs:
|
||||
# previously leave prod on a stale image.
|
||||
- name: Repoint :latest
|
||||
run: |
|
||||
echo "6-repoint" > /tmp/ci-logs/STEP
|
||||
set -eu
|
||||
for svc in server frontend broker; do
|
||||
ct=$(curl -s -o /tmp/m.json -D- \
|
||||
@@ -260,6 +271,7 @@ jobs:
|
||||
# pipeline exists to prevent.
|
||||
- name: Wait for the rolling deploy
|
||||
run: |
|
||||
echo "7-wait-deploy" > /tmp/ci-logs/STEP
|
||||
set -eu
|
||||
want=$(docker image inspect -f '{{.Id}}' "$REGISTRY/$NAMESPACE/server:latest")
|
||||
for i in $(seq 1 30); do
|
||||
|
||||
Reference in New Issue
Block a user