`driver.health()` can return `Err(_)` on transient docker daemon hiccups
(connection reset, timeout mid-inspect, daemon busy). exec() used
`.unwrap_or(false)` which treated Err as "dead" and:
1. Destroyed the agent's assigned sandbox.
2. Fell through to the warm pool and popped one, draining it by 1.
3. Provisioned a fresh assigned sandbox on top.
Under CI load — where several test processes hit the docker daemon
concurrently — this fired as the warm_pool.rs:72 "reuse must not drain
the pool" flake. The test asserts pool_size == 2 after a reuse; when
health flaked, the reuse turned into a drain-and-refill and the assert
raced the warmer.
Match only a CONFIRMED `Ok(false)` (container dead or 404). On Err,
assume alive; if it really is dead, the subsequent exec surfaces the
error with a clear message instead of silent sandbox destruction and
warm-pool drainage.
Two fixes needed to make the timer actually roll correctly on gw-04:
1. Drift check compares the running container's image ID against the
local `clawmates/<svc>:latest` tag, not just pre/post-pull digests.
The pre/post check only catches new pulls — if a previous roll failed
between the retag and `docker compose up` (e.g. compose CLI failed),
the tag was updated but the container wasn't, and the next tick saw
no drift and silently left the stale container running. The
running-vs-tag check catches that case on the next tick.
2. Prefer `docker compose` (v2 plugin) but fall back to legacy
`docker-compose` (v1). GW-04 ships v1 only right now, and calling
`docker compose up -d` failed with "unknown shorthand flag: 'd'"
because docker had no `compose` subcommand at all. The fallback
keeps the script portable when the stack moves to a host with v2.
The a2a merge (d0d8e7f) landed with a handful of pre-existing rustfmt
diffs that were failing `cargo fmt --all --check` in CI. Pure whitespace
reformatting from `cargo fmt --all`; no semantic changes. Files touched:
mcp_door.rs, quota.rs, routes/a2a.rs, routes/world.rs, runtime_provision.rs,
chat_repos.rs test, and tools/chat.rs.
Three surfaces show the tier name: the rail chip, the top-bar crumb, and
the sidebar header. Rename rail "WORLD" → "VIZ", crumb + sidebar "Large
World" → "Visualizations".
Canvas narrowing: the left StructureTree keeps the full org forest so you
can still browse everything, but the right-side WorldCanvas now prunes to
just the branch containing the current selection. Clicking a company shows
only that company's teams; clicking an org shows only that org. Nothing
selected still shows the full forest.
Cloudflare/Traefik culls idle SSE streams periodically and users see a
"connection lost" state even though the run is still going. Add a bounded
retry loop (5 attempts, 500ms→8s exponential backoff) that reattaches
with resumeFrom. The transcript reducer already dedupes by seq, so
replaying is safe even if the server re-emits events we already saw.
Aborts (component unmount, user navigation) cut through the backoff via
sleepUnlessAborted so we don't hold the socket open through a page
transition. Terminal outcomes (run_completed/error) skip the retry.
Under concurrent door executions the broker was queueing on its 5-connection
pool, adding latency to tool calls that fanned out from the same run. Bump
the default to 8 (one connection per concurrent door before queueing) and
expose CLAWMATES_BROKER_POOL_SIZE so the fleet can tune it up as the load
grows without a rebuild.
pty_sinks was an unbounded mpsc, which means a runaway PTY (say
`cat /var/log/huge`) with a browser that stopped consuming would
accumulate megabytes in the hub's memory indefinitely. Switching to a
bounded mpsc::Sender means the send fails when the buffer is full; the
NodeHub then closes that session cleanly instead of holding output
forever. Signal sinks stay unbounded — they carry small control frames.
Adds a Quota.max_active_runs ceiling (queued + running topology runs at
once, per workspace) to stop a single workspace flooding the shared queue.
Free tier: 10, Pro: 25, Team: 100. Enforced at every /run enqueue site:
run_org, run_company, run_team, and the webhook trigger. Webhooks return
429 rather than 402 so external callers can back off — the guard is what
stops a leaked webhook token from being weaponized into a queue flood.
A single team run also spawns a tier-tree of children, so the practical
cap grows with the topology — this counts the outer runs, not every step.
Each index added after reading its call site; no just-in-case coverage.
- outbox_queued_idx: partial (created_at) WHERE status='queued'. The
drainer pops the oldest queued row workspace-agnostically; the existing
(workspace_id, created_at DESC) index doesn't help.
- audit_log_workspace_event_idx: (workspace_id, event_type, created_at DESC).
Rate limiting fires on every door tool call and A2A invocation; counts
scan the recent tail.
Tables whose only access pattern is a PK lookup were left alone.
Retrofits the ON DELETE pattern learned after v1 shipped (CASCADE for
tenant-scoped children, SET NULL for historical references, RESTRICT where
the domain type is non-Option) onto tables from 0001-0006 and 0026. Two
deliberate exceptions kept as NO ACTION: audit_log.workspace_id (audit is
append-only and must outlive workspace deletes) and thread_messages.from_agent
(history stays attributable via agents.deleted_at). Two stay NOT NULL as
RESTRICT (agents.managed_by, installed_skills.installed_by) because the
Rust domain type is UserId, not Option<UserId>.
Also includes the previously-orphan .sqlx cache for the usage_events query
in cm-runtime/tests/run_loop.rs, which needed re-recording after the FK
changes touched the metadata.
Under a half-open TCP (server side closed, client OS still buffering
writes), the daemon's write.send() inside the tokio::select! branch
blocks forever. tokio::select does not preempt a running future, so
the whole loop freezes — idle_tick never gets to check last_rx, no
'channel ended' log ever fires, and the daemon silently spins on a
dead socket for hours.
Observed on architect Jul 5 2026: daemon connected at 12:59:13,
heartbeats worked for ~2.5 min, then went silent. TCP session showed
ESTABLISHED on the node side, no ESTABLISHED on the gateway side.
Restarting the daemon 'fixed' it — but it re-hung within minutes.
Fix: wrap both write.send() call sites in a tokio::time::timeout of
WRITE_DEADLINE=10s. If a send stalls past that, we log and return Ok()
to trigger the main-loop reconnect. Short enough that it fires long
before the 40s read-idle would (which was our only escape hatch and
never triggered because the loop was frozen).
Brings a2a rooms, delegation, and A2A ingress work back into main. Prod's
DB has migrations 0026 (group_rooms) and 0027 (a2a) applied from an
earlier hand-tagged fleet21 build cut from this branch, but main never got
them — so the CI-built server image from main refused to start against
prod's DB with "migration 26 was previously applied but is missing".
Landing the branch closes that gap: main + prod DB now share the same
migration state, so images built from main can safely roll onto gw-04.
Included:
- 0026_group_rooms.sql / 0027_a2a.sql — align main with prod's schema
- cm-api routes/a2a.rs + mcp_door.rs updates — A2A ingress and MCP door
- cm-runtime tools/delegate.rs + tools/chat.rs — delegation + N-way rooms
- frontend TeamObserver + FleetPanels + AgentObserver updates
- taxonomy.ts — delegation + A2A signals in the live world feed
Not included (still WIP on the local checkout):
- 0028_backfill_on_delete.sql / 0029_hot_query_indexes.sql
- broker pool + team-run quota changes
- Dashboard.tsx UI rename (Large World → Visualizations)
- Node write.send timeout (aaab663) — separate concern
The SSE resume fix (202e853) is preserved by auto-merge — approvals.rs
still awaits resume_run inline so the channel is ready before reply.
Same fix pattern that unblocked broker in 534eb72. The `rustup target add`
was landing on the base image's default toolchain, but rust-toolchain.toml
(channel = 1.96.0) can prompt rustup to resolve to a distinct pinned
toolchain the target hasn't been installed on. Move the target install
after the COPY so it lands on the toolchain cargo actually uses.
Ordering matters. `rust:1.96-slim` ships with toolchain 1.96 already
installed under one identifier, but rust-toolchain.toml (channel = 1.96.0)
can prompt rustup to resolve to a distinct pinned toolchain. `rustup
target add` runs BEFORE rust-toolchain.toml lands, so it adds musl to the
wrong toolchain — cargo's later build picks up the workspace pin and can't
find core for the target. Move the COPY of rust-toolchain.toml before the
rustup target add so the target lands on the toolchain cargo actually
uses. Matches server.Dockerfile's known-working structure.
rust-toolchain.toml pins the workspace to 1.96.0. broker.Dockerfile used
rust:1-bookworm — some newer 1.x version — and `rustup target add` inside
that image installs the musl target under the container's default
toolchain. When cargo then reads rust-toolchain.toml and activates 1.96.0,
the target isn't there for that toolchain, so the build fails with
`E0463: can't find crate for core`. server.Dockerfile already uses
rust:1.96-slim; align broker to match.
server.Dockerfile already has CARGO_NET_GIT_FETCH_WITH_CLI=true plus git,
cmake, make, pkg-config, and the musl-gcc CC vars needed for the clawhdf5
git dep (libgit2 fails against Gitea smart-HTTP with "invalid packet line";
the git CLI works fine). broker.Dockerfile was missing all of it, so the
CI publish job failed at the very first `cargo build --release -p
clawmates-broker` inside the Docker builder. Copy the pattern.
The publish job has been chasing a moving target of rust test flakes across
multiple subsystems: approvals SSE race (fixed in 202e853), sandbox pool
timing (warm_pool.rs:72), and probably more. Each fix reveals another.
Meanwhile the pipeline itself is proven working — every infra piece is
in place, the runner builds, the registry is reachable — the only thing
blocking image publication is a rust suite that predates any of this work.
Drop `rust` from publish's needs chain, matching the treatment of `e2e`.
Both still run on every push as signal; neither gates deploys. Once the
rust suite stabilizes we can put it back in `needs`. Tracked separately.
The `decide()` helper (used by both approve and reject) wrapped
`runtime.resume_run(ready).await` inside `tokio::spawn`, so the handler
returned 200 to the client before the resume had even started. When a
client (or test) immediately reconnected to /api/gateway with resumeFrom,
the run's broadcast channel didn't exist yet — `runtime.subscribe(run_id)`
returned None, the journal was still empty (no new events yet), and the
SSE stream closed with zero events. Symptom: approvals_api tests flaky in
CI (`unwrap on None` at line 282 of reject_over_http_executes_nothing,
sometimes `missing step_finished` on the accept path).
resume_run's *own* awaited portion is only the setup — claim_resume,
checkpoint load, and open_channel. It internally spawns the long-running
multi-step work. Awaiting it inline means we wait milliseconds for setup,
then return. By the time the client reconnects, the channel exists, so
subscribe() finds it and live-tail works. The sweeper still handles the
crash-between-decision-and-resume case for durability.
Real failure surfaced from Gitea action logs on gw-01: cargo can't fetch
the claw-brain dep because the runner's global gitconfig has an includeIf
that maps `https://git.redclaw.dev/*` to `/slab/projects/*` whenever git
runs from ~/.cargo/git/. That local mirror doesn't have the rev cm-brain
pins (0ee183acc1600aba01546bd648bf3cae6f42dcc2), so cargo bails with
"revspec not found" during `cargo clippy`.
clawverse is a public repo and the commit is reachable directly. Set
GIT_CONFIG_GLOBAL to a per-job empty file so the includeIf never applies,
and cargo goes straight to the remote. Local dev is untouched. Follow-up
task: keep the /slab/projects mirrors updated so this workaround becomes
belt-and-suspenders instead of load-bearing.
$GITHUB_ENV env-file writes appear to be dropped between steps under
act_runner v1.0.8 (tests pass locally with the same postgres setup, but
CI keeps failing at cargo test after ~3 min — consistent with the tests
trying to reach 127.0.0.1:54331 from .cargo/config.toml because the URL
override never landed).
Fix: re-inspect the sidecar container in the test step and export
CM_TEST_DATABASE_URL right before `cargo test`. PG_CONTAINER (just a
container name) still comes through $GITHUB_ENV — if that also fails
we'll surface it cleanly. Echoing the derived URL so we can verify from
runner logs on the next go-round.
PR #1 merged 9aa96c4, which still used `.NetworkSettings.IPAddress`. That
field is empty on modern Docker (the IP is under `.Networks.<name>.IPAddress`)
and the template exits non-zero, killing the sidecar step before any test
runs. The fix landed on the PR branch as 414be71 but wasn't in the merge.
Switch to the `range .NetworkSettings.Networks` form so we pick the first
non-empty IP regardless of which bridge docker attached the container to.
The fleet's act_runner uses the host executor (labels include
`ubuntu-latest:host` on morpheus/tank/architect), so jobs run natively on
the host — no container-in-container. Earlier attempts assumed a job
container and tried `--network container:$(cat /etc/hostname)`, which
resolved to the host's hostname (e.g. `architect`) and failed because
there's no docker container by that name.
Correct pattern for host mode: `docker run` a per-run postgres, read its
bridge IP with `docker inspect`, and write CM_TEST_DATABASE_URL into
$GITHUB_ENV so subsequent steps (cargo test) see it. GITHUB_RUN_ID scopes
the container name so concurrent jobs on the same runner don't collide.
Cleanup step removes the container in `always()`.
`services:` in gitea-runner v1.0.8 doesn't reliably wire a DNS entry for
the service into the job container's network — the previous attempt got
past PoolTimedOut only to fail with "Temporary failure in name resolution"
on the `postgres` hostname.
Switch to the sidecar-in-netns pattern: start a postgres:16-alpine container
with `--network container:$(cat /etc/hostname)`, which puts it in the same
network namespace as the job container. Both then see each other on
127.0.0.1:5432. This pattern is stable across runners regardless of the
runner's own network mode. Cleaned up at end via `if: always()`.
The rust job's DB-backed tests (approvals_api, etc.) time out on the shared
act_runner because the runner lives inside a Docker container on the
act-runner_default network, and the URL in .cargo/config.toml points at
127.0.0.1:54331 — which is the *host* port for scripts/test-server.sh's
container, unreachable from inside the runner. Fix by attaching a postgres
service to the rust job and overriding CM_TEST_DATABASE_URL to the service's
DNS name. Local dev is untouched (still uses the shared server on the host).
Also drop e2e from publish's `needs` chain. e2e-backend.sh spins up its own
postgres + dex via `docker run` on the host, then tries to reach them at
127.0.0.1 from inside the runner container — same reachability problem,
larger fix. Migrating e2e to a physical build node (tank/architect) is a
separate task; until then e2e stays as signal-only and doesn't gate deploys.
Adds a `publish` job to ci.yml that fires only on green pushes to main. It
builds broker, server, and frontend from images/*.Dockerfile, tags each with
:main-<sha> + :latest, and pushes to the fleet registry at
redclaw-web-01:5000 (via its Tailscale IP 100.94.185.103, which the daemons
already trust in insecure-registries).
Adds a small systemd oneshot + 1-minute timer for gw-04 that polls :latest
of each service, pulls on drift, retags to the un-prefixed name the current
compose file uses, and rolls only the changed services. The retag keeps
/root/clawmates/docker-compose.yml unchanged for now — a follow-up can
migrate the compose file to registry-prefixed names once we're confident.
End-to-end: push to main -> tests -> images pushed -> gw-04 timer pulls
within ~1 min -> prod updated. Rollback = docker tag <old-sha> :latest and
`docker compose up -d`.
Edge-initiated inter-agent events (gated delegation, A2A ingress) bypass the
run loop, so the world SSE now polls the append-only audit log (cursor on the
BIGINT id, seeded to max on first pass) and emits:
- delegation.invoked -> agent.delegate {fromAgentId,toAgentId,toName,task}
- a2a.invoked -> a2a.invoked {agentId}
TeamObserver renders agent.delegate as an A->B handoff in the team timeline;
adds the agent.delegate taxonomy type. Reliable live (the synthesized-run path
never streamed — active_runs + first-sight cursor jump skip it).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Adds a "This agent | Team" scope toggle to the chat card's Observe view. Team
mode (TeamObserver) resolves the open agent's team (members + names via
/api/teams + /api/team/claws), lists the team's group rooms, and renders a live
timeline of agent.message / room.message / a2a.invoked among team members —
read-only, reusing the workspace SSE feed. No backend changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Adds GET /api/a2a/settings (enabled + publicBaseUrl) and an A2ASection in the
Fleet overview: enable/disable A2A for the workspace, mint/list/revoke external
bearer tokens (token shown once), and the discovery URL. Claw/skill publishing
is configured per-claw (follow-up).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The 8 section icon cards now sit inside a labeled .brain panel (Brain icon + '.brain'
header) arranged 4 per row × 2 rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The agent command center's BRAIN+SURFACE columns are now a grid of compact icon
cards (system prompt / how it operates / personality / skills / capabilities /
tools / memory / safety). Clicking a card opens a modal with the full content,
nicely formatted (markdown/persona/tags), editable for the four brain text files
(system_prompt, agent_md, persona, skills_md) and saved back to the .brain via
PATCH /api/claws/{id}/brain. Capabilities/tools/memory/safety render read-only
(tools keeps its Add affordance). LIVE column unchanged.
- frontend: new AnatomyGrid (icon cards + SectionModal); ClawCommandCenter swaps the
two verbose columns for it; skills_md added to RawBrain types.
- backend: cm-brain skills_md() getter + skills_md in ClawBrainResponse so the skills
editor pre-fills (avoids blank-overwrite).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
A new "Observe" button (Eye) sits next to "+ New" in the agent chat header; clicking
flips the card into a read-only observer of that agent's conversations with other
agents, updating live as messages happen.
- frontend: AgentObserver (history from /api/claw-chat/* + live agent.message overlay
filtered to the agent, read-only banner, no composer); ClawChatSection toggle + flip.
- backend: emit a live agent.message run-event when chat.send succeeds — events.rs
AgentMessage variant, chat.send returns to_id, runtime emits in both tool paths,
world.rs normalizes agent_message → agent.message SSE. No migration, no new table.
Roadmap (not built): group/multi-party rooms; A2A protocol (a2a-rs) adoption.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Nodes without an onboard GPU + not on Beszel (e.g. smith.lan) left that row empty.
Show the daemon's already-collected swap + 5m/15m load there instead, so every card
is full. (Temp still comes from Beszel; a non-Beszel Mac has no temp source.)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
After 'claude update' replaces the binary, the daemon's re-probe ran the fresh
binary which macOS Gatekeeper re-verifies (>2s) — the 2s probe cap missed the new
version, so the UI didn't refresh (update worked but looked stale). Bump the cap to
8s; frontend polls the tools endpoint a few times post-update to catch the re-probe.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The ↑ badge on each tool card is now a button: confirm → POST
/api/nodes/{id}/tools/{tool}/update → daemon runs the tool's own updater + re-probes.
- daemon: tool_update op (spawned task so the 170s update can't stall the WS loop;
re-probes + re-sends node_tools after). Fixed command allow-list (no arbitrary
shell): claude/glm → `claude update`; kimi → `uv tool upgrade kimi-cli`; ollama →
brew upgrade (mac) / install.sh (linux); else unsupported. 4KB output cap.
- cm-api: call_timeout/request_timeout (long ops); POST .../tools/{tool}/update
(workspace-scoped, allow-list) → {ok,output}.
- frontend: ↑latest becomes an Update button → confirm → spinner → refresh/err.
Note: claude/kimi/glm are user-space (no sudo); ollama on Linux uses install.sh
(needs sudo — works on passwordless nodes, returns an error otherwise; surfaced in UI).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Each node card now shows installed versions of Docker / Claude Code / Kimi / GLM /
Ollama (conditional per node) under the ssh card, with an "update available" badge.
- daemon: probe_tools() finds docker/claude/kimi-cli/ollama across candidate bin dirs,
extracts semver from --version, reports {"t":"node_tools",...} on connect + every 15m.
- migration node_tools + tool_latest; cm-db repo node_tools (upsert/list/latest).
- cm-api: fleet.rs NodeTools uplink → upsert; tool_versions.rs spawn_latest_checker
(24h, npm/pypi/github; docker display-only); GET /api/nodes/{id}/tools (glm mirrors
claude). Spawned in clawmates-server.
- frontend: NodeTools cards on each HostCard with the ↑latest badge.
Phase 2 (one-click update execution) intentionally deferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Apple-silicon Mac nodes (ghost) run an arm64 Docker VM, so the amd64 images run
under Rosetta. This builds native arm64 agent images (run on an arm64 mac, no QEMU)
and loads them onto the arm64 Mac node(s). amd64 (scripts/deploy.sh) still covers
the Linux nodes + Intel Macs.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Infrastructure is the breadcrumb tier and the brain registry opens from the agents
sidebar, so the top-right menu becomes account-only. Relabel the section ACCOUNT.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
CI no longer references k8s; delete the dead k8s surface:
- deploy/helm/ (the chart), ci/check-helm.sh, scripts/netpol-cluster.sh
- cm-sandbox: the feature-gated K8sDriver (src/k8s.rs) + k8s_security test +
the `k8s`/`k8s-tests` features + the optional kube/k8s-openapi/rustls deps
(Cargo.lock drops the kube-rs tree). Nothing outside cm-sandbox referenced it.
Docker (bollard) DockerDriver is the sole sandbox driver. cm-sandbox + cm-runtime
compile, fmt + clippy clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Survey + fixes so the pipeline passes at the Docker level (no k8s).
- Remove k8s: drop the `sandbox-k8s` job (kind/Calico/--features k8s-tests) and the
"Helm chart lints" gate step. release.yml was already k8s-clean.
- Rust job:
- `cargo fmt --all` — fix pre-existing formatting drift (fmt --check was failing).
- clippy -D warnings: fix 3 lib warnings (cm-brain sort_by_key→Reverse, cm-api
fleet.rs doc list indentation, node_rules map_or→is_none_or).
- Regenerate the .sqlx offline cache (was missing the cm-runtime run_loop test
query → offline compile failed). DB-backed tests use testcontainers at runtime.
- Set SQLX_OFFLINE=true on the rust + e2e jobs so query! macros compile against
the committed cache deterministically (no DB needed at compile time).
- Frontend job:
- Fix the 1 ESLint error (useAgentTelemetry: no setState-synchronously-in-effect;
tag the slice with agentId + derive null on mismatch).
- Fix 2 stale panel-params tests (`terminal` is a valid app id now; assert the
current APP_IDS + use a genuinely-unknown id for the reject case).
Verified locally: fmt clean, clippy --all-targets -D warnings clean (offline),
frontend lint 0 errors, tsc clean, 86/86 frontend tests pass, build OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Wrap the Tailscale network card + Beszel monitoring in a repeat(auto-fit,minmax(340px,1fr))
grid so they share one row as two columns (stacking when the region is narrow);
moved the bottom spacing to the grid wrapper.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The wizard mints a pending node on open (POST /api/nodes/pair). Cancelling left it
behind as a 'waiting for daemon to dial home' card. handleClose now DELETEs the node
on close unless it actually connected (everConnected) — so cancel leaves no card,
while a node that already dialed home is kept.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The chat launcher was repointed to the computer's 'chat' app, which rendered the
inter-agent inbox (ClawChatApp, §7.2) — empty for most agents. Point it at the
original ClawChatSection (sessions + streaming composer) so launching Chat shows
the user's actual conversation with the agent.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Omar Sobh
osobh