ci: build and deploy to production from a push to main
Closes the one manual step left in the pipeline. gw-04 has run clawmates-deploy.timer every minute since July, pulling :latest and rolling on drift — the CD half already worked. What was missing was anything that moved :latest, since the old build host (tank) is packed for the move. The runner lives on gw-04 because it is the only reachable x86_64 host and prod images must be linux/amd64: web-01 is aarch64 and the fleet build boxes are offline. Host executor, capacity 1, so builds serialize rather than competing with production traffic. Three details that are not obvious: - `docker push :latest` does NOT move the tag on this registry once the manifest exists under another tag. The PUT-the-manifest step is what actually moves it, and its absence is how a "successful" deploy could leave prod on a stale image. - The final step verifies the image prod is RUNNING, not the one we pushed. A green edge on the old image is the failure this pipeline exists to prevent. - broker is built here too. It had no :latest tag at all, so gw-04's deploy loop logged a pull failure every single cycle since 2026-08-11. Also ignore the local env backups: `.env` was ignored but `.env.bak.*` was not, and those copies hold real credentials. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
co-authored by
Claude Opus 5
Claude Opus 5
parent
dc8f65fc64
commit
b1bf50160a
@@ -0,0 +1,152 @@
|
||||
# Local → production pipeline.
|
||||
#
|
||||
# push to main → test → build amd64 images → push to the fleet registry
|
||||
# → move :latest → gw-04's existing 60s rolling timer picks it up.
|
||||
#
|
||||
# The last hop is NOT in this file and does not need to be: gw-04 already runs
|
||||
# `clawmates-deploy.timer` every minute, which pulls
|
||||
# `$REGISTRY/clawmates/<svc>:latest`, compares it to the running image id, and
|
||||
# recreates on drift. This workflow's job is to make `:latest` mean the newest
|
||||
# green commit. See deploy/gw-04/clawmates-deploy.sh.
|
||||
#
|
||||
# Runs on the `gw04` runner (host executor, systemd unit act-runner). gw-04 is
|
||||
# the only reachable x86_64 host — web-01 is aarch64 and the fleet build boxes
|
||||
# are packed — and prod images must be linux/amd64, so builds are native here
|
||||
# rather than emulated.
|
||||
name: deploy
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
# Lets you re-run a deploy without an empty commit.
|
||||
workflow_dispatch:
|
||||
|
||||
env:
|
||||
REGISTRY: 100.94.185.103:5000
|
||||
NAMESPACE: clawmates
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: gw04
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
# A throwaway Postgres so the integration tests actually run. Without
|
||||
# CM_TEST_DATABASE_URL, cm-testkit tries a default admin URL and the
|
||||
# approvals_api tests die on PoolTimedOut — which looks like a failure but
|
||||
# only means "no database here".
|
||||
- name: Start test Postgres
|
||||
run: |
|
||||
docker rm -f cm-ci-pg 2>/dev/null || true
|
||||
docker run -d --name cm-ci-pg \
|
||||
-e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres \
|
||||
-p 127.0.0.1:55432:5432 postgres:16-alpine
|
||||
for i in $(seq 1 30); do
|
||||
docker exec cm-ci-pg pg_isready -U postgres >/dev/null 2>&1 && break
|
||||
sleep 2
|
||||
done
|
||||
docker exec cm-ci-pg pg_isready -U postgres
|
||||
|
||||
# Rust lives in a container because gw-04 has no cargo. The named volumes
|
||||
# are the whole reason this is not painfully slow: without them every run
|
||||
# recompiles the world.
|
||||
- name: Rust tests
|
||||
run: |
|
||||
docker run --rm --network host \
|
||||
-v "$PWD":/w -w /w \
|
||||
-v cm-ci-cargo-registry:/usr/local/cargo/registry \
|
||||
-v cm-ci-cargo-git:/usr/local/cargo/git \
|
||||
-v cm-ci-target:/w/target \
|
||||
-e SQLX_OFFLINE=true \
|
||||
-e CM_TEST_DATABASE_URL=postgres://postgres:[email protected]:55432/postgres \
|
||||
rust:1.96-slim \
|
||||
sh -c 'apt-get update -qq && apt-get install -y -qq pkg-config libssl-dev cmake git >/dev/null && cargo test --workspace'
|
||||
|
||||
- name: Stop test Postgres
|
||||
if: always()
|
||||
run: docker rm -f cm-ci-pg 2>/dev/null || true
|
||||
|
||||
# node 22 is on the host, so these run directly.
|
||||
- name: Frontend checks
|
||||
working-directory: frontend
|
||||
run: |
|
||||
npm ci --no-audit --no-fund
|
||||
npm run typecheck
|
||||
npm run test
|
||||
# Lint is advisory: the repo currently has pre-existing max-lines and
|
||||
# set-state-in-effect errors that predate this pipeline. Failing the
|
||||
# deploy on them would mean nothing could ship until they are cleared.
|
||||
npm run lint || echo "::warning::lint reported problems (advisory)"
|
||||
|
||||
build:
|
||||
runs-on: gw04
|
||||
needs: test
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Build + push images
|
||||
run: |
|
||||
set -eu
|
||||
SHA=$(git rev-parse --short HEAD)
|
||||
echo "SHA=$SHA" >> "$GITHUB_ENV"
|
||||
# The daemon binary the frontend serves at /dl. images/frontend.Dockerfile
|
||||
# expects it staged; rsync-based deploys create it out of band, so build
|
||||
# it here or the image ships without the node installer.
|
||||
mkdir -p frontend/public/dl
|
||||
docker run --rm \
|
||||
-v "$PWD":/w -w /w \
|
||||
-v cm-ci-cargo-registry:/usr/local/cargo/registry \
|
||||
-v cm-ci-cargo-git:/usr/local/cargo/git \
|
||||
-v cm-ci-target:/w/target \
|
||||
-e SQLX_OFFLINE=true -e CARGO_NET_GIT_FETCH_WITH_CLI=true \
|
||||
rust:1.96-slim \
|
||||
sh -c 'apt-get update -qq && apt-get install -y -qq pkg-config libssl-dev cmake git >/dev/null && cargo build --release -p clawmates-node && cp target/release/clawmates-node frontend/public/dl/clawmates-node-linux-amd64'
|
||||
|
||||
for svc in server frontend broker; do
|
||||
docker build -f "images/$svc.Dockerfile" \
|
||||
-t "$REGISTRY/$NAMESPACE/$svc:main-$SHA" \
|
||||
-t "$REGISTRY/$NAMESPACE/$svc:latest" .
|
||||
docker push "$REGISTRY/$NAMESPACE/$svc:main-$SHA"
|
||||
docker push "$REGISTRY/$NAMESPACE/$svc:latest"
|
||||
done
|
||||
|
||||
# `docker push :latest` does NOT reliably move the tag on this registry:
|
||||
# when the manifest already exists under another tag (it does — we just
|
||||
# pushed main-$SHA), the push reports a digest but `:latest` keeps
|
||||
# resolving to the OLD image. Writing the manifest to the tag over the
|
||||
# HTTP API is what actually moves it. This is the same trick
|
||||
# scripts/deploy.sh uses, and the reason a "successful" deploy could
|
||||
# previously leave prod on a stale image.
|
||||
- name: Repoint :latest
|
||||
run: |
|
||||
set -eu
|
||||
for svc in server frontend broker; do
|
||||
ct=$(curl -s -o /tmp/m.json -D- \
|
||||
-H 'Accept: application/vnd.oci.image.index.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json' \
|
||||
"http://$REGISTRY/v2/$NAMESPACE/$svc/manifests/main-$SHA" \
|
||||
| awk -F': ' '/^[Cc]ontent-[Tt]ype/{print $2}' | tr -d '\r')
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' -X PUT \
|
||||
-H "Content-Type: $ct" --data-binary @/tmp/m.json \
|
||||
"http://$REGISTRY/v2/$NAMESPACE/$svc/manifests/latest")
|
||||
echo "$svc :latest → main-$SHA (HTTP $code)"
|
||||
case "$code" in 20*) ;; *) echo "tag write failed"; exit 1 ;; esac
|
||||
done
|
||||
|
||||
# Verify the thing that actually matters: what prod is RUNNING, not what
|
||||
# we pushed. A green edge on a stale image is the failure mode this whole
|
||||
# pipeline exists to prevent.
|
||||
- name: Wait for the rolling deploy
|
||||
run: |
|
||||
set -eu
|
||||
want=$(docker image inspect -f '{{.Id}}' "$REGISTRY/$NAMESPACE/server:latest")
|
||||
for i in $(seq 1 30); do
|
||||
got=$(docker inspect -f '{{.Image}}' clawmates_server_1 2>/dev/null || echo none)
|
||||
if [ "$got" = "$want" ]; then
|
||||
echo "prod is running main-$SHA"
|
||||
curl -s -o /dev/null -w "edge HTTP %{http_code}\n" -m 10 https://clawmates.work/ || true
|
||||
exit 0
|
||||
fi
|
||||
sleep 10
|
||||
done
|
||||
echo "prod did not roll onto main-$SHA within 5m — check clawmates-deploy.timer"
|
||||
exit 1
|
||||
Reference in New Issue
Block a user