`docker rm -f` without -v orphans the anonymous volume the postgres image
declares. cm-testkit creates a database per test, so each CI run left ~2.8 GB
behind: 38 GB of dangling volumes had accumulated on gw-04, most of the 99 GB
-> 23 GB drop in free space over one day.
Note for anyone reaching for `docker volume prune` to clean this up: don't. On
gw-04 the dangling set also contained traefik-acme (Let's Encrypt certificates)
and all three CI cargo caches. Only the anonymous 64-hex volumes were safe to
remove.
Co-Authored-By: Claude Opus 5 <[email protected]>
On workflow_dispatch GITHUB_REF_NAME is the BRANCH, so the upload step created a
Gitea release AND a git tag both named "main" — a tag sharing the branch name,
from a run that was only meant to be a smoke test. Both have been deleted.
Gated on github.ref_type == 'tag'. A dispatch now exercises build, SBOM, sign
and offline verify, and stops there.
Co-Authored-By: Claude Opus 5 <[email protected]>
Every other step in the release job is inert with respect to prod — build,
SBOM, sign, offline verify. The rehearsal is the only one whose purpose is to
stand a full stack up and tear it down with `down -v`, and it was doing that on
the machine serving production. On 2026-08-13 it adopted the live compose
project and destroyed clawmates_pgdata.
The script itself is now safe (unique -p, a guard against the production project
name, and a health probe pointing at the port the bundle actually publishes) and
is kept for use on a build box or throwaway VM. What changes here is only WHERE
it runs, which was the real problem: a destructive verification step does not
belong on the host it can destroy.
Releases still build, sign, verify offline in a --network none container, and
upload to Gitea.
Co-Authored-By: Claude Opus 5 <[email protected]>
The v1 fallback I added a commit ago cannot work: deploy/compose/docker-compose.yml
uses v2-only syntax — a top-level `name:` and long-form
`env_file: {path, required}` — so docker-compose 1.29 rejects the file outright
("'name' does not match any of the regexes"). A fallback that always fails is
worse than no fallback, so the script now requires v2 and fails immediately with
what to do about it.
$COMPOSE overrides the detection. gw-04 is deliberately left WITHOUT a
`docker compose` plugin: installing one system-wide would flip the production
rolling deploy (clawmates-deploy.sh prefers v2 when present) off docker-compose
v1 as an invisible side effect of a release change. The runner gets a standalone
v2 binary at /opt/act-runner/bin/docker-compose and the workflow passes it in,
so prod keeps rolling exactly as it did.
Verified on gw-04: standalone v2.32.4 runs, and `docker compose` still resolves
to nothing, so clawmates-deploy.sh takes its v1 branch unchanged.
Co-Authored-By: Claude Opus 5 <[email protected]>
The rehearsal hardcoded `cargo build -p clawmates-bundler`, so it died with
"cargo: command not found" on the release runner — gw-04 builds Rust inside a
container and has no toolchain of its own. The release job had already built the
bundler two steps earlier, so it was also redundant work.
CLAWMATES_BUNDLER now short-circuits that build when it points at an executable,
falling back to cargo otherwise, so running the script by hand is unchanged.
Everything before this step already passed on the runner: images built, SBOMs
generated, bundle assembled and signed, and "bundle OK: 94 artifacts verified
offline" inside a --network none container.
Co-Authored-By: Claude Opus 5 <[email protected]>
First dispatch failed at exit 127, "target/release/clawmates-bundler: No such
file or directory". The bundler builds inside a container where /w/target is a
NAMED VOLUME, so the binary was written somewhere no later host step can see —
the workspace's target/ stays empty. Copy it to .tools/ (bind-mounted) and
assert it landed, so the next occurrence fails at the build step with a clear
message instead of two steps later as a missing file.
deploy.yml does not hit this because it copies clawmates-node into
frontend/public/dl/ from inside the same container.
Co-Authored-By: Claude Opus 5 <[email protected]>
It could never have run as written: `runs-on: ubuntu-latest` matches no runner
on this forge, and `softprops/action-gh-release` talks to GitHub's API. There
are zero tags and zero releases, which is consistent with it never having fired.
Rewritten for this runner:
- runs-on: gw04 (the only reachable x86_64 host; prod artifacts must be amd64)
- the bundler builds in a rust container with the shared cargo cache volumes —
gw-04 has no cargo, and installing a toolchain onto the production gateway to
build a release is the wrong trade
- release creation + asset upload go to Gitea's own API, create-or-reuse so a
re-run of a tag updates rather than 409s
- syft installs into the workspace, not /usr/local/bin: the host executor runs
as root on the gateway and a release should leave nothing behind
- a disk-reclaim step, because the artifacts are GBs of image tarballs on a box
that is also serving production. It removes only the versioned images it
created — never a blanket prune, since clawmates/agent-*:dev exist in no
registry and are the source of the microVM rootfs files
- workflow_dispatch added so the pipeline can be exercised without minting a tag
BUNDLE_SIGNING_KEY now exists as a repo secret (fresh ed25519 keypair; nothing
depended on a previous one). The signing and offline-verify steps are unchanged:
verification still runs inside a --network none container, which is the whole
air-gapped contract.
.github/ is now empty and removed.
Co-Authored-By: Claude Opus 5 <[email protected]>
cm-runtime/cm-sandbox tests shell out to `docker` via std::process, so the
mounted socket alone was not enough — browser_tool failed with
`docker available: NotFound`. Mount the host binary rather than apt-installing
docker.io: the container is fresh every run, so an install would re-download
~100 MB each time and cache nothing.
Verified on gw-04 that a mounted /usr/bin/docker talks to the host daemon
(client=29.1.3 server=29.1.3), and that the agent-*:dev images these tests need
are already present there.
Co-Authored-By: Claude Opus 5 <[email protected]>
cm-files' s3_store test starts a real MinIO via testcontainers. Without the
socket it does not skip — it fails with
`Client(Init(SocketNotFoundError("/var/run/docker.sock")))`, which looks like a
broken test rather than a missing capability. It passed locally only because the
Mac's docker socket was visible to the test process.
Sibling containers testcontainers starts are reachable because the test
container already shares the host network.
Co-Authored-By: Claude Opus 5 <[email protected]>
First run failed in `cargo test --workspace`: "failed to load source for
dependency clawhdf5", preceded by three "spurious network error: invalid packet
line" retries. Two separate causes, both needed:
- libgit2 cannot fetch from Gitea's smart-HTTP. images/server.Dockerfile already
sets CARGO_NET_GIT_FETCH_WITH_CLI for exactly this; the test step did not.
- quantumclaw/clawhdf5 is private (401 anonymous), so the CLI fetch needs a
credential. Supplied via an insteadOf rewrite from a repo secret, so the token
is masked in logs and never committed.
The server image build does not hit this — it builds only clawmates-server,
which does not pull cm-brain's clawhdf5 path.
Co-Authored-By: Claude Opus 5 <[email protected]>
Closes the one manual step left in the pipeline. gw-04 has run
clawmates-deploy.timer every minute since July, pulling :latest and rolling on
drift — the CD half already worked. What was missing was anything that moved
:latest, since the old build host (tank) is packed for the move.
The runner lives on gw-04 because it is the only reachable x86_64 host and prod
images must be linux/amd64: web-01 is aarch64 and the fleet build boxes are
offline. Host executor, capacity 1, so builds serialize rather than competing
with production traffic.
Three details that are not obvious:
- `docker push :latest` does NOT move the tag on this registry once the manifest
exists under another tag. The PUT-the-manifest step is what actually moves it,
and its absence is how a "successful" deploy could leave prod on a stale image.
- The final step verifies the image prod is RUNNING, not the one we pushed. A
green edge on the old image is the failure this pipeline exists to prevent.
- broker is built here too. It had no :latest tag at all, so gw-04's deploy loop
logged a pull failure every single cycle since 2026-08-11.
Also ignore the local env backups: `.env` was ignored but `.env.bak.*` was not,
and those copies hold real credentials.
Co-Authored-By: Claude Opus 5 <[email protected]>
