Two durability fixes closing recurring flakes:
CI flake wrapper (broker + server Dockerfiles):
Wrapped the cargo build step in a 3-attempt retry loop with
linear backoff (10s / 20s). Directly targets the crates.io
transient network errors that keep hitting CI on the runners
('curl failed: SSL_ERROR_SYSCALL, errno 0'). Each build only
loses time on transient failures; a real compile error still
fails all 3 attempts and surfaces the last error normally.
Runtime systemd unit (deploy/clawmates-runtime/):
Replaces the manual 'docker run' that had been starting the
ZeroClaw runtime with no persistence for its network topology.
Ephemeral prod fixes at 09:30 PDT 2026-07-21 (task #38) were:
- anthropic.default provider block added to
/root/clawmates-runtime/data/.zeroclaw/config.toml (already
durable — bind-mounted from host)
- docker network connect clawmates_edge clawmates-runtime
(NOT durable — vanishes on container recreate)
New systemd unit clawmates-runtime.service (installed +
enabled on gw-04):
- ExecStart docker-runs the container attached to
clawmates_core, then connects clawmates_edge in the same
shell command, then docker waits.
- Bind-mounts both /root/clawmates-runtime/data and
/var/lib/clawmates-missions (for security_scan +
benchmark_runner).
- --rm so upgrading is just docker pull + systemctl restart.
- Restart=on-failure with 5s backoff.
Closes task #38 and preemptively closes the CI flake pattern.
Ordering matters. `rust:1.96-slim` ships with toolchain 1.96 already
installed under one identifier, but rust-toolchain.toml (channel = 1.96.0)
can prompt rustup to resolve to a distinct pinned toolchain. `rustup
target add` runs BEFORE rust-toolchain.toml lands, so it adds musl to the
wrong toolchain — cargo's later build picks up the workspace pin and can't
find core for the target. Move the COPY of rust-toolchain.toml before the
rustup target add so the target lands on the toolchain cargo actually
uses. Matches server.Dockerfile's known-working structure.
rust-toolchain.toml pins the workspace to 1.96.0. broker.Dockerfile used
rust:1-bookworm — some newer 1.x version — and `rustup target add` inside
that image installs the musl target under the container's default
toolchain. When cargo then reads rust-toolchain.toml and activates 1.96.0,
the target isn't there for that toolchain, so the build fails with
`E0463: can't find crate for core`. server.Dockerfile already uses
rust:1.96-slim; align broker to match.
server.Dockerfile already has CARGO_NET_GIT_FETCH_WITH_CLI=true plus git,
cmake, make, pkg-config, and the musl-gcc CC vars needed for the clawhdf5
git dep (libgit2 fails against Gitea smart-HTTP with "invalid packet line";
the git CLI works fine). broker.Dockerfile was missing all of it, so the
CI publish job failed at the very first `cargo build --release -p
clawmates-broker` inside the Docker builder. Copy the pattern.
Full-depth rename per the approved plan; the 'claw' product vocabulary
(claws, /claws routes, clawId, Claw Chat) stays — it is now the brand.
- Display brand: Clawmates (manifest, titles, hero, login/rail logo
'clawmates'); default host app.clawmates.work; registry
ghcr.io/clawmates
- Crates tc-* -> cm-* (16 crates + all imports); binaries
clawmates-server/broker/bundler; images clawmates/*; env prefix
CLAWMATES_* (+ CM_TEST_DATABASE_URL / CM_LIVE_LLM); config
clawmates.toml; helm chart deploy/helm/clawmates with clawmates-*
resources; db names clawmates*; sockets /run/clawmates; cookie
cm_session; kind cluster clawmates-test; seccomp node profile
clawmates-agent-profile.json
- All 9 Playwright brand assertions updated in lockstep; historical
spec document left untouched as the only remaining 'TeamClaw'
- Local env migrated: dev pg clawmates-dev-pg/clawmates_dev, shared
test server clawmates-test-pg, kind cluster recreated with image +
profile, compose images rebuilt under clawmates/*
Verified end to end: 161 Rust + 68 frontend tests, 29 Playwright
journeys, 4 live kind tests, helm/install/LOC/placeholder gates, and
the clean-room install rehearsal serving the clawmates login page from
a signed bundle of the rebuilt images.
Co-Authored-By: Claude Fable 5 <[email protected]>
- tc-telemetry: fmt subscriber always; with [telemetry] otlp_endpoint
set, spans batch-export over OTLP/HTTP. Tested against a REAL OTLP
receiver decoding the actual protobuf (official proto types): the
emitted span and service.name arrive on the wire. No endpoint = no
export = no network (air-gap stance). tower-http TraceLayer gives
every API request a span
- The broker finally has its own image (images/broker.Dockerfile,
9.5MB from scratch) — the Helm chart referenced one that never
existed — and the compose deployment now RUNS the broker, sharing a
socket volume with the server (the unix-socket equivalent of the K8s
sidecar). Compose secret flows were silently dead before this
- server.Dockerfile fixes surfaced by the rehearsal: the workspace
build needs tools/ (bundler joined the workspace) and
images/seccomp/ (include_str! profile) in the build context
- scripts/rehearse-install.sh (plan: clean-VM rehearsal): assembles a
REAL signed bundle from the built images (server/frontend/broker/
postgres/socket-proxy), runs the customer path — offline verify,
docker load, compose up — and asserts /healthz plus the served login
page before teardown. Passing locally; wired as a release.yml step,
which also builds/ships the broker + socket-proxy images now
161 Rust tests + 29 journeys; clean-room rehearsal green.
Co-Authored-By: Claude Fable 5 <[email protected]>
Omar Sobh