Omar SobhandClaude Opus 5
af89020dfd
cm-runtime/cm-sandbox tests shell out to `docker` via std::process, so the mounted socket alone was not enough — browser_tool failed with `docker available: NotFound`. Mount the host binary rather than apt-installing docker.io: the container is fresh every run, so an install would re-download ~100 MB each time and cache nothing. Verified on gw-04 that a mounted /usr/bin/docker talks to the host daemon (client=29.1.3 server=29.1.3), and that the agent-*:dev images these tests need are already present there. Co-Authored-By: Claude Opus 5 <[email protected]>
190 lines
8.8 KiB
YAML
190 lines
8.8 KiB
YAML
# Local → production pipeline.
|
|
#
|
|
# push to main → test → build amd64 images → push to the fleet registry
|
|
# → move :latest → gw-04's existing 60s rolling timer picks it up.
|
|
#
|
|
# The last hop is NOT in this file and does not need to be: gw-04 already runs
|
|
# `clawmates-deploy.timer` every minute, which pulls
|
|
# `$REGISTRY/clawmates/<svc>:latest`, compares it to the running image id, and
|
|
# recreates on drift. This workflow's job is to make `:latest` mean the newest
|
|
# green commit. See deploy/gw-04/clawmates-deploy.sh.
|
|
#
|
|
# Runs on the `gw04` runner (host executor, systemd unit act-runner). gw-04 is
|
|
# the only reachable x86_64 host — web-01 is aarch64 and the fleet build boxes
|
|
# are packed — and prod images must be linux/amd64, so builds are native here
|
|
# rather than emulated.
|
|
name: deploy
|
|
|
|
on:
|
|
push:
|
|
branches: [main]
|
|
# Lets you re-run a deploy without an empty commit.
|
|
workflow_dispatch:
|
|
|
|
env:
|
|
REGISTRY: 100.94.185.103:5000
|
|
NAMESPACE: clawmates
|
|
|
|
jobs:
|
|
test:
|
|
runs-on: gw04
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
# A throwaway Postgres so the integration tests actually run. Without
|
|
# CM_TEST_DATABASE_URL, cm-testkit tries a default admin URL and the
|
|
# approvals_api tests die on PoolTimedOut — which looks like a failure but
|
|
# only means "no database here".
|
|
- name: Start test Postgres
|
|
run: |
|
|
docker rm -f cm-ci-pg 2>/dev/null || true
|
|
docker run -d --name cm-ci-pg \
|
|
-e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres \
|
|
-p 127.0.0.1:55432:5432 postgres:16-alpine
|
|
for i in $(seq 1 30); do
|
|
docker exec cm-ci-pg pg_isready -U postgres >/dev/null 2>&1 && break
|
|
sleep 2
|
|
done
|
|
docker exec cm-ci-pg pg_isready -U postgres
|
|
|
|
# Rust lives in a container because gw-04 has no cargo. The named volumes
|
|
# are the whole reason this is not painfully slow: without them every run
|
|
# recompiles the world.
|
|
# Docker socket AND the host's docker binary are mounted:
|
|
# - cm-files' s3_store test uses testcontainers (socket only).
|
|
# - cm-runtime/cm-sandbox tests (browser_tool, shell_exec, warm_pool,
|
|
# security, socket_proxy) shell out to `docker` via std::process, so
|
|
# they need the CLI on PATH too. Mounting the host binary beats
|
|
# apt-installing docker.io on every run — that is ~100 MB of download
|
|
# per job, and the container is fresh each time so nothing caches it.
|
|
# These tests do NOT skip when the capability is missing; they fail in a
|
|
# way that reads like broken code (SocketNotFoundError / NotFound), which
|
|
# is why they are worth wiring up rather than excluding.
|
|
#
|
|
# They also need clawmates/agent-{base,browser,terminal}:dev, which are
|
|
# locally-built images present on gw-04 but in no registry. If this job
|
|
# ever moves hosts, those images must move with it.
|
|
#
|
|
# `cargo test --workspace` builds cm-brain, which pulls clawhdf5 from
|
|
# git.redclaw.dev — a PRIVATE repo. Two things are needed and neither is
|
|
# optional:
|
|
# CARGO_NET_GIT_FETCH_WITH_CLI — libgit2 fails against Gitea's smart-HTTP
|
|
# with "invalid packet line" (the server Dockerfile sets it for the
|
|
# same reason). Note it is _GIT_FETCH_WITH_CLI, not _NET_FETCH_.
|
|
# the insteadOf rewrite — supplies the credential to that CLI fetch.
|
|
# The token is a repo secret, so it is masked in logs and never in git.
|
|
- name: Rust tests
|
|
run: |
|
|
docker run --rm --network host \
|
|
-v "$PWD":/w -w /w \
|
|
-v cm-ci-cargo-registry:/usr/local/cargo/registry \
|
|
-v cm-ci-cargo-git:/usr/local/cargo/git \
|
|
-v cm-ci-target:/w/target \
|
|
-v /var/run/docker.sock:/var/run/docker.sock \
|
|
-v /usr/bin/docker:/usr/bin/docker:ro \
|
|
-e SQLX_OFFLINE=true \
|
|
-e CARGO_NET_GIT_FETCH_WITH_CLI=true \
|
|
-e FORGE_TOKEN='${{ secrets.FORGE_TOKEN }}' \
|
|
-e CM_TEST_DATABASE_URL=postgres://postgres:[email protected]:55432/postgres \
|
|
rust:1.96-slim \
|
|
sh -c 'set -e
|
|
apt-get update -qq
|
|
apt-get install -y -qq pkg-config libssl-dev cmake git >/dev/null
|
|
git config --global url."https://oauth2:[email protected]/".insteadOf "https://git.redclaw.dev/"
|
|
cargo test --workspace'
|
|
|
|
- name: Stop test Postgres
|
|
if: always()
|
|
run: docker rm -f cm-ci-pg 2>/dev/null || true
|
|
|
|
# node 22 is on the host, so these run directly.
|
|
- name: Frontend checks
|
|
working-directory: frontend
|
|
run: |
|
|
npm ci --no-audit --no-fund
|
|
npm run typecheck
|
|
npm run test
|
|
# Lint is advisory: the repo currently has pre-existing max-lines and
|
|
# set-state-in-effect errors that predate this pipeline. Failing the
|
|
# deploy on them would mean nothing could ship until they are cleared.
|
|
npm run lint || echo "::warning::lint reported problems (advisory)"
|
|
|
|
build:
|
|
runs-on: gw04
|
|
needs: test
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- name: Build + push images
|
|
run: |
|
|
set -eu
|
|
SHA=$(git rev-parse --short HEAD)
|
|
echo "SHA=$SHA" >> "$GITHUB_ENV"
|
|
# The daemon binary the frontend serves at /dl. images/frontend.Dockerfile
|
|
# expects it staged; rsync-based deploys create it out of band, so build
|
|
# it here or the image ships without the node installer.
|
|
mkdir -p frontend/public/dl
|
|
docker run --rm \
|
|
-v "$PWD":/w -w /w \
|
|
-v cm-ci-cargo-registry:/usr/local/cargo/registry \
|
|
-v cm-ci-cargo-git:/usr/local/cargo/git \
|
|
-v cm-ci-target:/w/target \
|
|
-e SQLX_OFFLINE=true -e CARGO_NET_GIT_FETCH_WITH_CLI=true \
|
|
-e FORGE_TOKEN='${{ secrets.FORGE_TOKEN }}' \
|
|
rust:1.96-slim \
|
|
sh -c 'set -e
|
|
apt-get update -qq
|
|
apt-get install -y -qq pkg-config libssl-dev cmake git >/dev/null
|
|
git config --global url."https://oauth2:[email protected]/".insteadOf "https://git.redclaw.dev/"
|
|
cargo build --release -p clawmates-node
|
|
cp target/release/clawmates-node frontend/public/dl/clawmates-node-linux-amd64'
|
|
|
|
for svc in server frontend broker; do
|
|
docker build -f "images/$svc.Dockerfile" \
|
|
-t "$REGISTRY/$NAMESPACE/$svc:main-$SHA" \
|
|
-t "$REGISTRY/$NAMESPACE/$svc:latest" .
|
|
docker push "$REGISTRY/$NAMESPACE/$svc:main-$SHA"
|
|
docker push "$REGISTRY/$NAMESPACE/$svc:latest"
|
|
done
|
|
|
|
# `docker push :latest` does NOT reliably move the tag on this registry:
|
|
# when the manifest already exists under another tag (it does — we just
|
|
# pushed main-$SHA), the push reports a digest but `:latest` keeps
|
|
# resolving to the OLD image. Writing the manifest to the tag over the
|
|
# HTTP API is what actually moves it. This is the same trick
|
|
# scripts/deploy.sh uses, and the reason a "successful" deploy could
|
|
# previously leave prod on a stale image.
|
|
- name: Repoint :latest
|
|
run: |
|
|
set -eu
|
|
for svc in server frontend broker; do
|
|
ct=$(curl -s -o /tmp/m.json -D- \
|
|
-H 'Accept: application/vnd.oci.image.index.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json' \
|
|
"http://$REGISTRY/v2/$NAMESPACE/$svc/manifests/main-$SHA" \
|
|
| awk -F': ' '/^[Cc]ontent-[Tt]ype/{print $2}' | tr -d '\r')
|
|
code=$(curl -s -o /dev/null -w '%{http_code}' -X PUT \
|
|
-H "Content-Type: $ct" --data-binary @/tmp/m.json \
|
|
"http://$REGISTRY/v2/$NAMESPACE/$svc/manifests/latest")
|
|
echo "$svc :latest → main-$SHA (HTTP $code)"
|
|
case "$code" in 20*) ;; *) echo "tag write failed"; exit 1 ;; esac
|
|
done
|
|
|
|
# Verify the thing that actually matters: what prod is RUNNING, not what
|
|
# we pushed. A green edge on a stale image is the failure mode this whole
|
|
# pipeline exists to prevent.
|
|
- name: Wait for the rolling deploy
|
|
run: |
|
|
set -eu
|
|
want=$(docker image inspect -f '{{.Id}}' "$REGISTRY/$NAMESPACE/server:latest")
|
|
for i in $(seq 1 30); do
|
|
got=$(docker inspect -f '{{.Image}}' clawmates_server_1 2>/dev/null || echo none)
|
|
if [ "$got" = "$want" ]; then
|
|
echo "prod is running main-$SHA"
|
|
curl -s -o /dev/null -w "edge HTTP %{http_code}\n" -m 10 https://clawmates.work/ || true
|
|
exit 0
|
|
fi
|
|
sleep 10
|
|
done
|
|
echo "prod did not roll onto main-$SHA within 5m — check clawmates-deploy.timer"
|
|
exit 1
|