osobh/clawmates
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(deploy): the verify step is the authority, not the recreate

Browse Source 
`scripts/deploy.sh` reported failure twice this afternoon for deploys that had
succeeded. Both times the 60-second rolling timer rolled the stack onto the same
`:latest` first, and the script's own `docker-compose up` then hit a
container-name conflict — "already in use" once, "Renaming a container with the
same name" the other — for a container the timer had already recreated correctly.

A deploy signal an operator has to second-guess is precisely what this script
exists to prevent. Its original reason for being was a green edge on a stale
image; crying wolf trains people to ignore the alarm, which gets you the same
outcome by a different route.

The recreate is now best-effort and says so when it fails, and the VERIFY step
decides — it compares the RUNNING image id against the resolved `:latest`, which
is the only question that matters and is unaffected by which process did the
roll. A genuinely failed deploy still fails there, because that check never
depended on the recreate succeeding.

Both false alarms were settled by hand with the binary grep
(`docker exec … grep -a -c "<string only in the new code>"`), which remains the
strongest check when the image id is in doubt.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-07 05:20:37 -07:00
co-authored by Claude Opus 5
parent 3300c9d149
commit 521da9feb9
1 changed files with 19 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/deploy.sh
+19 -2
View File
@@ -141,13 +141,30 @@ if [ -z "${IMAGES_ONLY:-}" ]; then
echo "→ roll $GW onto main-$SHA" echo "→ roll $GW onto main-$SHA"
# Roll immediately rather than waiting up to 60s for the timer. Snapshot the # Roll immediately rather than waiting up to 60s for the timer. Snapshot the
# outgoing image as :rollback first so a revert is a repoint, not a rebuild. # outgoing image as :rollback first so a revert is a repoint, not a rebuild.
ssh "$GW" "set -e #
# NOT fatal on its own, and that is the point. The 60s rolling timer
# (clawmates-deploy.timer) rolls the same stack onto the same `:latest`, so the
# two race — and the loser reports a container-name conflict ("already in use",
# "Renaming a container with the same name") for a container the winner has
# already recreated CORRECTLY. That happened twice in one afternoon, and a
# deploy signal an operator has to second-guess is exactly what this script
# exists to prevent: the original failure mode was a green edge on a stale
# image, and crying wolf trains people to ignore the alarm.
#
# So the roll is best-effort and the VERIFY below is the authority — it
# compares the RUNNING image id against the resolved `:latest`, which is the
# only question that matters and is unaffected by which process did the roll.
if ! ssh "$GW" "set -e
for svc in server frontend; do for svc in server frontend; do
docker tag $REGISTRY/clawmates/\$svc:$TAG $REGISTRY/clawmates/\$svc:rollback 2>/dev/null || true docker tag $REGISTRY/clawmates/\$svc:$TAG $REGISTRY/clawmates/\$svc:rollback 2>/dev/null || true
docker pull -q $REGISTRY/clawmates/\$svc:$TAG >/dev/null docker pull -q $REGISTRY/clawmates/\$svc:$TAG >/dev/null
done done
cd $GW_DIR cd $GW_DIR
docker-compose -p clawmates up -d --no-deps server frontend" docker-compose -p clawmates up -d --no-deps server frontend"; then
echo " (the recreate reported an error — most often the 60s rolling timer"
echo " got there first. Verifying the running image rather than assuming"
echo " either outcome.)"
fi
fi fi
echo "→ load agent runtime images onto $GW + every fleet node" echo "→ load agent runtime images onto $GW + every fleet node"