fix(deploy): move registry :latest by manifest PUT — prod follows a 60s rolling timer

Deploys were verifying green and then silently reverting minutes later.
Cause: gw-04 does not deploy from this script's recreate at all.
`clawmates-deploy.timer` runs every 60s, pulls
`$REGISTRY/clawmates/<svc>:latest`, and rolls the stack onto it whenever
the running image differs — so the local `docker tag` + `--force-recreate`
this script did was reverted within the minute. Its own log shows it:

  server drift: running=<the new image> target=<the old :latest>
  rolling: server frontend

The registry's `:latest` is therefore the only thing that decides what
prod runs — and `docker push …:latest` does NOT reliably move it here.
When the manifest already exists under another tag (the `main-<sha>` we
push immediately before), the push reports a digest but `:latest` keeps
resolving to the old image. Pushing a brand-new tag works, so it is
specific to overwriting an existing one.

Writing the manifest to the tag over the registry HTTP API does move it
(GET the main-<sha> manifest, PUT that body to :latest → 201), after
which the timer converges prod on its own. So:

- repoint :latest via manifest PUT from the build host, failing loudly on
  a non-2xx instead of assuming the push landed
- roll gw-04 immediately rather than waiting up to 60s for the timer
- verify against the resolved :latest (what compose and the timer both
  deploy from) instead of a main-<sha> tag that is never pulled there

Note for future debugging: image IDs differ per host for the same tag
(buildx OCI index — tank holds the index digest, gw-04 the resolved
platform image), so the trustworthy check is grepping the deployed binary
for a string only the new code contains.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

scripts/deploy.sh

@@ -79,25 +79,43 @@ ssh "$BUILD_HOST" 'set -e; cd ~/clawmates
   done'
 
 if [ -z "${IMAGES_ONLY:-}" ]; then
-  echo "→ pull + recreate server + frontend on $GW ($GW_DIR)"
+  echo "→ repoint registry :latest → main-$SHA"
-  # Snapshot the currently-deployed images as :rollback (a repoint, cheap) so a
+  # gw-04 does NOT deploy from this script's push alone. A systemd timer
-  # bad deploy can be reverted without a rebuild, then pull the freshly-pushed
+  # (clawmates-deploy.timer, every 60s, /usr/local/bin/clawmates-deploy.sh)
-  # images and recreate.
+  # pulls `$REGISTRY/clawmates/<svc>:latest` and rolls the stack onto it
-  # Pull the IMMUTABLE main-<sha> tag and retag it to :latest locally, then
+  # whenever the running image differs. So ANY local `docker tag`/recreate on
-  # recreate WITHOUT a compose pull. Pulling `:latest` here is not reliable —
+  # the gateway is reverted within a minute — the registry's `:latest` is the
-  # the registry has served a stale manifest for that mutable tag (a deploy
+  # single source of truth for what prod runs.
-  # pushed main-9bc5f6a fine, but `pull :latest` reported "up to date" and
+  #
-  # left the OLD image running). Immutable tags always resolve correctly, so
+  # And `docker push …:latest` does NOT reliably move that tag here: when the
-  # the sha tag is the source of truth and `:latest` is just a local alias
+  # manifest already exists in the registry under another tag (which it does,
-  # for the compose file's image reference.
+  # we just pushed main-$SHA), the push reports a digest but `:latest` keeps
+  # resolving to the old image. Writing the manifest to the tag directly over
+  # the HTTP API is what actually moves it. Verified: PUT → 201, and the
+  # timer then rolls prod on its own.
+  ssh "$BUILD_HOST" "set -e
+    for svc in server frontend; do
+      ct=\$(curl -s -o /tmp/cm-manifest.json -D- \
+        -H 'Accept: application/vnd.oci.image.index.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json' \
+        http://$REGISTRY/v2/clawmates/\$svc/manifests/main-$SHA \
+        | awk -F': ' '/^[Cc]ontent-[Tt]ype/{print \$2}' | tr -d '\r')
+      code=\$(curl -s -o /dev/null -w '%{http_code}' -X PUT \
+        -H \"Content-Type: \$ct\" --data-binary @/tmp/cm-manifest.json \
+        http://$REGISTRY/v2/clawmates/\$svc/manifests/latest)
+      echo \"   \$svc :latest → main-$SHA (HTTP \$code)\"
+      case \"\$code\" in 20*) ;; *) echo \"   ✗ tag write failed\"; exit 1 ;; esac
+    done"
+
+  echo "→ roll $GW onto main-$SHA"
+  # Roll immediately rather than waiting up to 60s for the timer. Snapshot the
+  # outgoing image as :rollback first so a revert is a repoint, not a rebuild.
   ssh "$GW" "set -e
     for svc in server frontend; do
       docker tag $REGISTRY/clawmates/\$svc:$TAG $REGISTRY/clawmates/\$svc:rollback 2>/dev/null || true
-      docker pull $REGISTRY/clawmates/\$svc:main-$SHA
+      docker pull -q $REGISTRY/clawmates/\$svc:$TAG >/dev/null
-      docker tag $REGISTRY/clawmates/\$svc:main-$SHA $REGISTRY/clawmates/\$svc:$TAG
     done
     cd $GW_DIR
-    docker-compose -p clawmates up -d --force-recreate --no-deps server frontend"
+    docker-compose -p clawmates up -d --no-deps server frontend"
 fi
 
 echo "→ load agent runtime images onto $GW + every fleet node"
@@ -110,16 +128,17 @@ done
 
 if [ -z "${IMAGES_ONLY:-}" ]; then
   echo "→ verify"
-  # Verify the RUNNING image matches what we just pushed — not just that the
+  # Verify the RUNNING image is the one we just published — not just that the
-  # edge is up. A green edge on the OLD image is the silent-revert failure mode
+  # edge is up. A green edge on the OLD image is the silent-revert failure
-  # this check exists to catch.
+  # mode this check exists to catch. Compare against the resolved :latest,
-  want=$(ssh "$GW" "docker image inspect -f '{{.Id}}' $REGISTRY/clawmates/server:main-$SHA 2>/dev/null" || true)
+  # which is what both compose and the rolling timer deploy from.
+  want=$(ssh "$GW" "docker image inspect -f '{{.Id}}' $REGISTRY/clawmates/server:$TAG 2>/dev/null" || true)
   got=$(ssh "$GW" "docker inspect -f '{{.Image}}' clawmates_server_1 2>/dev/null" || true)
   if [ -n "$want" ] && [ "$want" = "$got" ]; then
     echo "   server running expected image ($SHA): ${got:7:12}"
   else
-    echo "   ✗ server image MISMATCH — running ${got:7:12}, expected main-$SHA (${want:7:12})"
+    echo "   ✗ server image MISMATCH — running ${got:7:12}, expected ${want:7:12}"
-    echo "     the deploy did NOT take effect; check the registry pull on $GW"
+    echo "     check that :latest was repointed and the roll succeeded on $GW"
     exit 1
   fi
   ssh "$GW" 'curl -s -o /dev/null -w "   edge HTTP %{http_code}\n" -m 10 https://clawmates.work/ || true'