morpheus is packed for a move. It is drained in the `nodes` table —
heartbeats preserve `draining`, so it stays out of placement when it comes
back — and removed from the agent-image loop here.
An unreachable host in NODES does not merely skip it. The image loop
fails the whole script BEFORE its verify stage, so four deploys in a row
rolled the server and frontend correctly and then reported nothing at all;
every one had to be confirmed by hand. Keep this list to hosts that answer.
The name is left in a comment rather than deleted: putting it back is one
word, and the next person will want to know where it went.
Co-Authored-By: Claude Opus 5 <[email protected]>
`scripts/deploy.sh` reported failure twice this afternoon for deploys that had
succeeded. Both times the 60-second rolling timer rolled the stack onto the same
`:latest` first, and the script's own `docker-compose up` then hit a
container-name conflict — "already in use" once, "Renaming a container with the
same name" the other — for a container the timer had already recreated correctly.
A deploy signal an operator has to second-guess is precisely what this script
exists to prevent. Its original reason for being was a green edge on a stale
image; crying wolf trains people to ignore the alarm, which gets you the same
outcome by a different route.
The recreate is now best-effort and says so when it fails, and the VERIFY step
decides — it compares the RUNNING image id against the resolved `:latest`, which
is the only question that matters and is unaffected by which process did the
roll. A genuinely failed deploy still fails there, because that check never
depended on the recreate succeeding.
Both false alarms were settled by hand with the binary grep
(`docker exec … grep -a -c "<string only in the new code>"`), which remains the
strongest check when the image id is in doubt.
Co-Authored-By: Claude Opus 5 <[email protected]>
The post-transfer verification added in bb34ef1 failed every deploy: it
compared `.Id` between build host and target, and a BuildKit image on the
build host carries attestation manifests that `docker save | docker load`
does not reproduce. The same build legitimately arrives with a different
Id and a different reported Size — tank had agent-base:dev at 28 MB /
363f23b7, gw-04 at 74 MB / edd46f95, both from the identical build.
`.Created` comes from the config blob, survives the round trip unchanged,
and is what actually answers "is the new build here". Both hosts reported
2026-08-02T16:43:00.599937575-07:00, which is how the false positive was
identified rather than assumed.
The verification itself stays — the truncation it guards against is real.
This corrects what it compares.
Co-Authored-By: Claude Opus 5 <[email protected]>
`docker save | docker load` across two SSH connections spliced through a
workstation truncates when either side stalls — observed as `unexpected
EOF` mid-deploy. Nothing checked afterwards, and `docker load` can exit 0
on a short stream, so a partially-populated image could ship to every
fleet node and look like a success.
Now compressed, pipefail-guarded, and verified by comparing image IDs on
the target after the transfer, with one retry for the transient stall.
A failed transfer fails the deploy rather than passing quietly.
The runtime image no longer travels this path at all: it is registry-
hosted now (100.94.185.103:5000/clawmates-runtime:v083-toolchain), built
from deploy/clawmates-runtime/Dockerfile on tank. Only agent-base /
agent-browser / agent-terminal still need save|load, because they exist
in no registry.
Co-Authored-By: Claude Opus 5 <[email protected]>
Deploys were verifying green and then silently reverting minutes later.
Cause: gw-04 does not deploy from this script's recreate at all.
`clawmates-deploy.timer` runs every 60s, pulls
`$REGISTRY/clawmates/<svc>:latest`, and rolls the stack onto it whenever
the running image differs — so the local `docker tag` + `--force-recreate`
this script did was reverted within the minute. Its own log shows it:
server drift: running=<the new image> target=<the old :latest>
rolling: server frontend
The registry's `:latest` is therefore the only thing that decides what
prod runs — and `docker push …:latest` does NOT reliably move it here.
When the manifest already exists under another tag (the `main-<sha>` we
push immediately before), the push reports a digest but `:latest` keeps
resolving to the old image. Pushing a brand-new tag works, so it is
specific to overwriting an existing one.
Writing the manifest to the tag over the registry HTTP API does move it
(GET the main-<sha> manifest, PUT that body to :latest → 201), after
which the timer converges prod on its own. So:
- repoint :latest via manifest PUT from the build host, failing loudly on
a non-2xx instead of assuming the push landed
- roll gw-04 immediately rather than waiting up to 60s for the timer
- verify against the resolved :latest (what compose and the timer both
deploy from) instead of a main-<sha> tag that is never pulled there
Note for future debugging: image IDs differ per host for the same tag
(buildx OCI index — tank holds the index digest, gw-04 the resolved
platform image), so the trustworthy check is grepping the deployed binary
for a string only the new code contains.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`docker-compose pull server frontend` pulls `:latest`, and the registry
served a STALE manifest for that mutable tag: a deploy pushed
`main-9bc5f6a` correctly, but the gateway's `pull :latest` reported
"image is up to date" and left the previous image running. The verify
step caught it (running 9f2349 = main-0a647c0, expected bbf19f7e), so
the deploy failed loudly rather than silently — but it still could not
ship.
Immutable tags always resolve correctly, so pull `main-<sha>` and retag
it to `:latest` locally on the gateway, then recreate with `--no-deps`
and no compose pull. `:latest` is now just a local alias satisfying the
compose file's image reference; the sha tag is the source of truth.
Also switch the recreate to `--no-deps` (compose v1 has no
`--no-recreate-deps`) so a server/frontend deploy stops recreating
postgres.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
rsync --delete excludes frontend/public/dl/, so the directory does not exist
on the build host and the `cp` of clawmates-node into it aborted the deploy
(set -e) before anything was pushed.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The gateway compose pulls server/frontend from the web-01 registry
(100.94.185.103:5000, tags main-<sha> + :latest), so `docker save | docker
load` + a local retag does NOT stick — the next `docker-compose up` pulls
`:latest` and silently reverts to the last-pushed image (a green edge on the
old image hid this). Rewrite the server/frontend path to: build on the build
host → tag :latest + :main-<sha> → push to the registry → `compose pull +
up --force-recreate` in /opt/clawmates (the real project dir, not the stale
/root/clawmates) → verify the RUNNING image id equals the pushed one (fail
loudly on mismatch instead of trusting HTTP 200). Agent :dev images stay on
the save|load path (not in any registry).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- frontend/.../terminal/core.ts: `useResilientTerminal` owns the xterm lifecycle
(init, fit, debounced resize, reconnect, refit-on-visible) with a pluggable
transport. Two connectors: `wsConnector` (agent terminal → container PTY over WS)
and `nodeWebrtcConnector` (node terminal → host PTY, direct WebRTC DataChannel
with WS-relay fallback, input buffered during the race). The agent terminal now
gets the node terminal's robust reconnect for free; both share one xterm setup.
- TerminalApp (agent) + NodeTerminalApp (node) reduced to thin wrappers over the
core — net ~330 lines of duplicated transport/reconnect/xterm code removed.
- scripts/deploy.sh: the deploys were manual, so the locally-built agent images
(agent-base/browser/terminal — not in any registry) were never shipped and 404'd
on provision. The script always (re)builds + loads them onto gw-04 AND every
fleet node, with a skip-if-identical guard so unchanged images aren't re-transferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
