The prior flow was ambiguous: after hitting Start research, status
flipped to "processing" and a "Submit for review" button appeared
immediately with no indication that anything was actually running.
Users had to guess whether the pipeline was working or stalled.
Backend surfaces the truth as a signal:
- new topology_runs::active_runs_for_research_topic counts
queued+running runs whose research_topic_id matches
- TopicDetail includes runs_in_flight: i64 alongside the existing
status field, so the canvas can distinguish "pipeline still
working" from "runner stalled".
ResearchCanvas is now honest about state:
- while runs_in_flight > 0, the header status pill grows a cyan
"N runs in flight" badge with an inline SVG spinner
- the stage-explainer card turns cyan-bordered and shows a
"pipeline is running" hint, plus copy pointing the user at the
Agents tier where each teammate's activity streams live
- the "Submit for review (manual)" button is HIDDEN while any run
is in flight — it's an escape hatch for stalled runs only, not
the happy-path action. It reappears if runs_in_flight drops to
zero but the topic is still marked processing, so a stalled
runner can still be nudged along.
- the canvas polls getTopic every 4s while status is processing/
publishing or runs_in_flight > 0, so the spinner + outcome swap
in automatically when the pipeline completes.
ResearchList sidebar:
- each row's status dot becomes a spinner when the topic's status
is processing or publishing, matching the canvas at a glance
- the list also polls every 6s while ANY topic is active, so
transitions land in the sidebar without waiting on a parent bump.
The poll is gated on a derived boolean to avoid effect thrash.
Follow-up: same pattern belongs on LoopsList / LoopsCanvas for
loop iterations in flight — same signal (queued+running runs per
loop) but not wired here.
Mirror the row-level delete affordance the loops sidebar already
has. Loops was wired earlier; research had a bare title-only card
with no way to remove a stale topic.
- cm-db: research_topics::delete cascades via existing FK rules
(research_topic_agents, research_publish_approvals, and the new
research_outcomes all CASCADE on topic_id; topology_runs's
research_topic_id back-ref is SET NULL so historical runs stay).
- cm-api: DELETE /api/research/{id} → 204. Idempotent.
- Frontend: deleteTopic helper. ResearchList row is now a card
with the existing title/status/outcome header plus a trash icon
that flips the card into an inline "Delete topic + all outcomes?"
confirm strip. Confirm → red Delete / gray Cancel. If the
deleted topic was selected, selection clears; local counter
bumps the list refetch without waiting on a parent.
Three connected changes that turn "Start research" from a status
flip into a real pipeline that produces a reviewable artifact:
- Migration 0036: adds research_topics.topology_kind (default
'hub_spoke') and a new research_outcomes table
(id, topic_id, version DESC, body_md, produced_by_run_id, created_at)
so each run's final synthesis is versioned and persistent.
- Wizard now has a topology picker in the Outcome step —
hub_spoke / pipeline / hierarchical / star_moe — with copy that
steers users to the right shape (Pipeline for research → distill
→ analyze → implement rosters, hub_spoke for the coordinator-
and-specialists default).
- start_topic reads the chosen topology_kind, parses it into a
cm_topology::TopologyKind, and dispatches a per-shape coordinator
prompt via build_coordinator_task. Pipeline explicitly tells
stage 1 not to write the final artifact and propagates a
"final stage MUST emit a complete markdown document with
measurable acceptance criteria" instruction downstream. The
graph builder is called with the topology the user actually
picked instead of hard-coded HubSpoke.
- topology_worker::freeze_research_outcome fires after every
successful complete(). It looks up research_topic_id on the run;
if set and final_output is non-empty, it inserts a new
research_outcomes row (version auto-derived server-side via
coalesce(max(version), 0) + 1). Best-effort — a DB hiccup logs
but doesn't fail the run.
- TopicDetail now includes topology_kind and latest_outcome.
ResearchCanvas swaps in the outcome's body_md (rendered as
pre-wrap markdown, versioned header, produced-at timestamp)
whenever an outcome exists; the original prompt collapses into
an "Original prompt" <details> below so it's still one click
away. Pre-run topics still show the description as before.
Follow-ups still open: reject-with-revision loop feeding the
coordinator, publishing → published transition + real artifact
export (md / pdf), and an approvals inbox surface for reviewers.
The prior start_topic only flipped the status column — no work was
enqueued. Now clicking "Start research" actually launches the
assigned agents through the orchestrator.
- start_topic loads the topic + its research_topic_agents, picks a
coordinator (first slot with role_slot containing "coordinator";
else the first slot), and swaps it to index 0.
- Builds a hub_spoke topology graph via cm_topology::build with
roles = [coordinator, spoke1, spoke2, …]. hub_spoke wires edges
from the hub to every spoke and back, so the coordinator can
address any specialist per turn.
- Assembles a coordinator prompt from the topic's title,
description, outcome_kind, and a roster line for each teammate —
so the coordinator knows who's on the team and what each does.
- Enqueues via a new topology_runs helper
enqueue_run_for_research_topic that stores research_topic_id on
the run row. `topology_worker::maybe_transition_research_topic`
→ `notify_run_completed` already picks up on that back-ref and
flips the topic processing → reviewing when the last run
terminates — that path was dead code until now.
- Per-agent activity streams into each claw's card for free:
the orchestrator journals turn events into run_events; the
existing /api/world/live SSE normalizer emits
agent.reasoning.delta / agent.tool.call / agent.task.update
keyed by agent id, which ClawCommandCenter is already
subscribed to.
The `published` terminal state is still unreached (that's the
"publishing → published + artifact" step from the earlier
walkthrough — separate follow-up).
Three connected fixes from a single session's feedback:
- Structure rail widened from 60→76 px, tabs 42→58 px wide with a
bit of left padding so labels ("Research", "Visualizations") no
longer bump against the active-tab indicator strip.
- Research canvas: each stage now shows a small "STAGE · <status>"
card explaining what state the topic is in and a "Next → …" hint
describing what the primary button will do. No more guessing
which of standby/processing/reviewing/publishing means what.
- Request-publish 409 fix:
- Backend TopicDetail now includes has_pending_publish_request
(SELECTs pending_for_topic when the topic loads). Frontend
TopicDetail interface + ResearchCanvas honor the flag: when
an approval is already pending the "Request publish" button
is replaced with an amber "Awaiting reviewer approval" pill,
so double-clicks can't 409 in the first place.
- runAction() also catches 409 as a signal-of-success (the
user's intent — "queue for approval" — is satisfied by the
first attempt), refetches the topic, and lets the new
awaiting-approval card render instead of surfacing a scary
error to the user.
The frontend already renamed the "Specialists" tab to "Agent" and
rewrote the intro to ask for one specialist, but the backend
prompt was still telling Opus to propose "2 or 3 domain
specialists". Two-facing message got confusing outputs.
- SPECIALISTS_NOTE now says "SINGLE agent … `members` MUST contain
EXACTLY ONE entry … topology_kind='flat' … team_name reads like
a personal handle." Removes the ambiguity.
- TEAM_NOTE range aligned to "4 to 6 agents" (matches the UI hint
and intro).
- Frontend right-panel now reads "PROPOSED AGENT" for specialists
mode (was "PROPOSED TEAM") and pluralization matches the count
("1 agent" vs "3 agents"). Build button switches to "Build agent"
in that mode.
Selecting a team/company/org in the Agents sidebar used to only
delete the grouping row; the agents inside survived, ungrouped.
Not what the user wanted, and it left a trail of orphaned runtime
state (containers, .brain files, DB rows) behind.
Backend — POST /api/claws/batch-delete is now a universal cascade
reaper. Body accepts { ids?, teams?, companies?, orgs? } in any
combination. The server walks org → companies_of_org →
teams_of_company → agents_of_team, dedupes against explicit ids,
and hard-purges every unique agent (deprovision ZeroClaw runtime,
tear down sandbox container, unlink .brain/.onion files,
transactional agents::hard_purge). Group rows are deleted last; FK
cascades on team_members, company_teams, org_companies, and
loop_agents/teams/orgs clean up the join tables. Every stage
streams SSE.
Three new cm-db helpers wire the walk: agents_of_team,
teams_of_company, companies_of_org — all DISTINCT selects on the
existing join tables.
Frontend — Dashboard's Agents-tier StructureTree now sets
selectLevel="*" (was "claw"), so the Wrench → checkbox affordance
appears on org/company/team/agent nodes alike; the same-level
invariant in onToggleSelect still prevents mixed batches.
ReapProgressModal collapses to a single POST regardless of kind —
body key derived from kind — and its subtitle is honest:
"Cascading through every agent inside — permanent."
Adds the missing pieces the wizard needed and the sidebar controls
around it:
- LoopsWizard is now a 6-step flow (identity → repo → task/topology
→ triggers → repeat → assign agents) plus the existing secrets
card. ResearchWizard picks up the same repo step and a hard gate
when the workspace has zero agents.
- New LoopStaffingStep with three tabs — Individual / Team /
Organization — that mix freely per loop; selections persist via
new loop_agents / loop_teams / loop_orgs join tables (0035
migration), each cascading on loop_id so hard-delete stays a
single-row DELETE.
- Backend CreateLoopRequest / UpdateLoopRequest accept the three
lists and apply_staffing does a transactional replace-all;
list_loops / get_loop hydrate the lists via a flattened
LoopWithStaffing response.
- LoopsList sidebar gains per-row enable/disable, edit (reopens the
wizard prefilled with the current loop, PATCHes on submit), and
delete with an inline confirm.
- NoAgentsGate blocks launching a loop or research topic from a
workspace with no roster; the sidebar `+` buttons also disable
with a tooltip pointing at the TEAM tier.
Not yet wired: the run driver still fills role slots from the
workspace-wide pool; teaching enqueue_iteration to prefer
loop_agents/loop_teams/loop_orgs is a follow-up.
Sidebar:
- Each connection header now has three inline icon buttons: Sync now
(spins while in flight), Edit (opens the modal), Remove (opens an
inline confirm strip). Removes cascade repos via ON DELETE CASCADE.
- The connection's last_sync_error surfaces as a red inline banner
under the header — no more 'error status with nowhere to see why'.
- Sync is POST /api/repos/connections/:id/sync (already existed);
after either sync or delete the sidebar re-fetches so state stays
consistent.
Edit modal (RepoConnectionEditModal):
- Loads GET /api/repos/connections/:id, pre-fills owner/base_url/label
- PATCHes only the fields that actually changed; empty string on a
Some(&str) field sends explicit null so the backend clears it
- Sync-now + Remove reachable from inside the modal too
- Rotating the token is out of scope: the modal says as much and
points the user at delete + re-create through the wizard (the
broker doesn't expose an update path, and rotating in place would
require duplicating the whole broker->store_secret flow here)
Backend:
- GET /api/repos/connections/:id — same ConnectionSummary shape
- PATCH /api/repos/connections/:id — owner/base_url use Option<Option<T>>
double-nesting so 'omit = leave alone' and 'null = clear' round-trip
distinctly through serde
- repo_connections::update with COALESCE-per-field so the SQL matches
the double-Option semantics without an OR-chain per field
Fixes 500 on any broker-touching route (POST /api/repos/connections,
POST /api/apps, the OAuth callback) when broker + server run under
different UIDs — which is exactly the prod topology on gw-04 (broker
uid 10001, clawmates-server distroless nonroot uid 65532). Linux Unix
socket connect(2) requires read+write on the socket file, and the
default bind mode 0755 gives 'others' r-x only.
Widen to 0666 after bind. The broker socket only lives inside the
shared broker_run volume — two containers mount it, nothing else on
the host can see it — so widening is safe. If set_permissions is a
no-op on the target filesystem (abstract sockets on some kernels),
we log and continue instead of failing serve().
Fleet's Gitea (git.redclaw.dev) hosts most of this workspace's repos,
so Gitea gets the same inline sync treatment GitHub already had.
Backend sync_gitea:
- base_url is required — Gitea has no shared 'gitea.com'; we accept
either the instance root (auto-appends /api/v1) or the fully-formed
API base if the user already included the suffix
- /orgs/{owner}/repos when owner set, /repos/search when not (with the
{data: [...], ok: bool} envelope Gitea wraps that endpoint in)
- 404 with an owner surfaces as 'org not found or PAT lacks access',
same UX as GitHub
- 50/page, capped at 20 pages (~1000 repos); short page terminates
- upsert_gitea_repo tolerates the small field-name differences
(stars_count vs stargazers_count, owner.login vs owner.username on
older versions)
Frontend wizard:
- Gitea listed first — matches the workspace's actual usage
- Default provider selection is now gitea
- Token-input placeholder tailored per provider (Gitea's is
'Settings → Applications → Generate New Token (repo)')
GitLab still returns 'not yet supported' — that's the next follow-up.
Migration 0034: two tables. repo_connections carries the workspace's
per-provider config (owner, base_url, label, last_synced_at,
last_sync_error) and points at an app_connections row for the PAT.
repos is the per-connection cache with (connection_id, external_id)
unique so upsert is idempotent across re-syncs. Cascading deletes clean
up cleanly on connection removal.
cm-secrets grows a FetchAuthorized op — GET with the stored PAT injected
as bearer, returns status + JSON body without ever exposing the
credential to cm-api. This is the least-privilege door for read-only
provider APIs (list repos), distinct from the InvokeHttp path that still
requires a single-use approval grant for outbound writes.
cm-api::routes::repos wires:
- POST /api/repos/connections (broker store_secret + insert both rows +
initial sync + mark_synced)
- GET /api/repos/connections
- DELETE /api/repos/connections/:id
- POST /api/repos/connections/:id/sync
- GET /api/repos (500 cap, newest provider_updated first)
- GET /api/repos/:id (full detail incl. clone_url + html_url)
GitHub provider inline for v1 — paginated pull of /orgs/:owner/repos
(when owner set) or /user/repos (when absent), 100/page, capped at 20
pages (~2k repos) to keep first-sync latency bounded. Non-2xx surface
back to the caller as sync_error; parse failures are best-effort per
repo (skipped, logged, don't abort the batch).
Gitea + GitLab providers land in a follow-up — mostly URL swap +
response-shape adapter.
Migration 0033: adds teams.lifecycle ('permanent' | 'ephemeral') and a
topology_runs.team_id back-ref with a partial index for the sibling-in-
flight check.
cm-db repo:
- teams::insert_team_with_lifecycle (insert_team keeps the permanent default)
- topology_runs::enqueue_run_for_team (populates team_id)
- topology_runs::check_ephemeral_teardown — atomic SELECT that only
returns Some when the team is ephemeral AND no siblings are still
queued/running; carries the workspace + bound claw ids for cleanup.
cm-api:
- topology_worker post-terminal hook maybe_teardown_ephemeral_team
runs deprovision_claw on each bound claw (best-effort; failures log
but don't block Postgres deletion), then hard_purge each agent row,
then delete_team.
- routes::teams::build_team_with_lifecycle (build_team keeps default);
run_team enqueues with team_id.
- planner ScaffoldRequest gains mode; lifecycle_for(mode) sets the team
to ephemeral for scheduled + triggered, permanent otherwise.
Frontend MasterPlannerModal passes mode in the scaffold payload so the
backend can derive lifecycle without duplicating the mode taxonomy.
Tests: 3 new (returns claws when no siblings, holds when siblings queued,
ignores permanent teams). 10/10 topology_jobs green; workspace clippy
--tests clean.
Frontend: send topologyKind to /api/planner/chat so the planner's user
prompt gets a USER-LOCKED TOPOLOGY block telling Opus to use it verbatim.
On buildTeam, override proposal.topology_kind with the user's pick
(belt-and-braces — if the planner ignored the lock, we still ship the
right shape). Proposal chip renders the effective kind in a lavender
tint when it was overridden, with a hover title showing what was
replaced.
Backend PlannerChatRequest gains an optional topology_kind. Empty /
absent = planner picks. Not honored for 'swarm' mode (swarm planner
doesn't take a topology kind).
Hooks the topology_worker's post-terminal path into a new
notify_run_completed repo helper that atomically transitions the topic
processing → reviewing when the completed run has research_topic_id set
AND no siblings for that topic are still queued or running. Guarded on
status='processing' so a retry, a re-fire, or a topic already past
processing are all no-ops. Best-effort at the worker; DB hiccups are
logged and never fail the run.
The manual /submit-review endpoint stays as an escape hatch for topics
that end up parked in processing with nothing to complete (updated the
doc comment).
Extend the topology-runs list route with an optional loop_id filter that
returns iterations for a single loop, newest-iteration-first. Adds the
iteration and finished_at columns to the summary (skip-null on the JSON
so compares stay compact). Backed by list_by_loop in the repo, which uses
the existing topology_runs_loop_idx partial index.
LoopsCanvas fetches the runs in parallel with the loop detail and renders
an iteration timeline card (iteration #, status pill, start time, duration,
run id prefix) between the graph section and the actions row.
Fourth commit of the Research + Loops arc. Completes the state machine
for research topics with the publish approval gate the spec asked for.
Migration 0032 — research_publish_approvals
Dedicated small table (id, workspace_id, topic_id, requested_by,
status, decided_by/at, created_at). Keeping it separate from the
existing `approvals` table (0001) because that one is tightly coupled
to gated tool calls inside an agent run — session_key + run_id +
action_type + category + payload + preview + requested_by_agent, all
NOT NULL. Forcing those nullable would ripple through cm_safety;
cleaner to give publish approvals their own two-transition state
machine.
New endpoints
POST /api/research/:id/submit-review processing → reviewing
(v1 caller-driven; the
orchestrator hook comes
when we wire actual runs)
POST /api/research/:id/request-publish creates a pending
approval. Rejects with
409 if the topic already
has one open.
GET /api/research/publish-approvals list workspace's pending
POST /api/research/publish-approvals/:id/approve flips approval to
approved + transitions
the topic
reviewing → publishing
(which stamps
published_at)
POST /api/research/publish-approvals/:id/reject stays in reviewing; new
requests allowed
The approve/reject write is an atomic UPDATE ... WHERE status = 'pending';
the decide() repo function returns whether the caller won the race so
concurrent double-approves collapse to a single topic transition.
State machine after this commit:
standby ─POST /start─▶ processing ─POST /submit-review─▶ reviewing
─POST /request-publish + approve─▶ publishing ─(future: artifact
assembly)─▶ published
Third commit of the Research + Loops arc. Lights up loops as durable
recurring topology executions:
GET /api/loops list workspace's loops
POST /api/loops create — returns webhook_token +
signing_key ONCE when webhook trigger
is enabled; never exposed again
GET /api/loops/:id detail
PATCH /api/loops/:id update definition
DELETE /api/loops/:id delete
POST /api/loops/:id/run trigger one iteration NOW
POST /api/loops/:id/enable set enabled=true
POST /api/loops/:id/disable set enabled=false
POST /webhooks/loops/:token public; HMAC-SHA256-verified
Scheduler (cm_runtime::spawn_loop_scheduler) wakes every 10s, queries the
partial index on (next_fire_at) for due loops, enqueues one topology_runs
row per fire with loop_id + iteration + parent_run_id chained back to the
previous iteration. Uses croner via the existing scheduling::next_occurrence
helper. Missed windows fire ONCE and skip the backlog — next_fire_at is
always computed strictly AFTER now(), so a late scheduler doesn't drain a
buildup.
Webhook signatures follow the same pattern as the Stripe billing webhook
(HMAC-SHA256 with constant-time hex compare). Token + signing key are
24-byte OS-RNG values; the URL uses base64-url for the token, and the
signing key is base64-std. Both surface exactly once at create time.
All three fire paths (scheduler, immediate-run, webhook) funnel through
`cm_db::repo::loops::enqueue_iteration` so the invariants stay in one
place. `iters` repeat policy is enforced by the scheduler tick; `until`
and `on_completion` land with the orchestrator hook in commit 4.
Adds cm-llm as a direct cm-api dep, getrandom for the webhook material
generator, and wires the scheduler spawn into the server binary alongside
the resume sweeper and outbox drainer.
Second commit of the Research + Loops arc. Builds on 0030 by lighting up
the container CRUD, the agent-slot attach/detach, the standby→processing
transition, and the wizard's one-shot LLM refine.
GET /api/research list workspace's topics
POST /api/research create (accepts wizard output)
GET /api/research/:id detail (topic + attached agents)
PATCH /api/research/:id update non-status fields
POST /api/research/:id/agents attach agent (idempotent)
DELETE /api/research/:id/agents/:agent detach
POST /api/research/:id/start standby → processing
POST /api/research/wizard/refine one-shot LLM refine
The refine endpoint accumulates the workspace's default LLM provider's
stream into a JSON object (`{title, description}`) with the tight system
prompt at the top of the module. Same provider routing as agent runs
(via runtime.provider()), so a workspace already using GLM/Kimi gets it
for free.
State machine's remaining transitions (processing → reviewing on last
run_completed; reviewing → publishing via approvals gate) land with the
orchestrator hookup + approvals extension. Publish approval and loops
are separate commits still to come.
Adds cm-llm as a direct cm-api dep (previously only pulled transitively
via cm-runtime) so the refine endpoint can build a ChatRequest. Uses
sqlx::query! for compile-time verification; .sqlx cache generated on
morpheus against a fresh migrated DB.
Root cause of the "reuse must not drain the pool" flake: the test called
`manager.exec(AgentId::new(), ...)` with a random UUID that had no agents
row. `agent_containers::upsert` uses `INSERT ... FROM agents WHERE a.id = $1`,
which silently inserts zero rows when no agent matches — so the "assigned"
sandbox never persisted to the DB. The next exec's reuse lookup returned
None, fell through to the provision branch, and popped from the warm pool
instead of reusing the assigned sandbox. When the warmer hadn't refilled by
the time we asserted, pool_size == 1 instead of 2.
Seed a workspace + owner + agent up front (mirrors soak.rs's setup). Now
upsert commits a real row, the second exec hits the reuse branch, and the
pool stays whole — the test asserts what its name claims.
The tolerant-health-check change in cm-runtime (15cffba) stays as a
defensive improvement for prod under docker daemon load, but the real fix
for the test is here.
Inherited from the a2a merge. Iterating `&[AgentId]` directly yields
`&AgentId`; `AgentId::as_uuid(&self)` works fine on that, so the extra
`.iter().copied()` was pure noise and tripped clippy's
`unnecessary_to_owned` under `-D warnings`.
`driver.health()` can return `Err(_)` on transient docker daemon hiccups
(connection reset, timeout mid-inspect, daemon busy). exec() used
`.unwrap_or(false)` which treated Err as "dead" and:
1. Destroyed the agent's assigned sandbox.
2. Fell through to the warm pool and popped one, draining it by 1.
3. Provisioned a fresh assigned sandbox on top.
Under CI load — where several test processes hit the docker daemon
concurrently — this fired as the warm_pool.rs:72 "reuse must not drain
the pool" flake. The test asserts pool_size == 2 after a reuse; when
health flaked, the reuse turned into a drain-and-refill and the assert
raced the warmer.
Match only a CONFIRMED `Ok(false)` (container dead or 404). On Err,
assume alive; if it really is dead, the subsequent exec surfaces the
error with a clear message instead of silent sandbox destruction and
warm-pool drainage.
The a2a merge (d0d8e7f) landed with a handful of pre-existing rustfmt
diffs that were failing `cargo fmt --all --check` in CI. Pure whitespace
reformatting from `cargo fmt --all`; no semantic changes. Files touched:
mcp_door.rs, quota.rs, routes/a2a.rs, routes/world.rs, runtime_provision.rs,
chat_repos.rs test, and tools/chat.rs.
Under concurrent door executions the broker was queueing on its 5-connection
pool, adding latency to tool calls that fanned out from the same run. Bump
the default to 8 (one connection per concurrent door before queueing) and
expose CLAWMATES_BROKER_POOL_SIZE so the fleet can tune it up as the load
grows without a rebuild.
pty_sinks was an unbounded mpsc, which means a runaway PTY (say
`cat /var/log/huge`) with a browser that stopped consuming would
accumulate megabytes in the hub's memory indefinitely. Switching to a
bounded mpsc::Sender means the send fails when the buffer is full; the
NodeHub then closes that session cleanly instead of holding output
forever. Signal sinks stay unbounded — they carry small control frames.
Adds a Quota.max_active_runs ceiling (queued + running topology runs at
once, per workspace) to stop a single workspace flooding the shared queue.
Free tier: 10, Pro: 25, Team: 100. Enforced at every /run enqueue site:
run_org, run_company, run_team, and the webhook trigger. Webhooks return
429 rather than 402 so external callers can back off — the guard is what
stops a leaked webhook token from being weaponized into a queue flood.
A single team run also spawns a tier-tree of children, so the practical
cap grows with the topology — this counts the outer runs, not every step.
Under a half-open TCP (server side closed, client OS still buffering
writes), the daemon's write.send() inside the tokio::select! branch
blocks forever. tokio::select does not preempt a running future, so
the whole loop freezes — idle_tick never gets to check last_rx, no
'channel ended' log ever fires, and the daemon silently spins on a
dead socket for hours.
Observed on architect Jul 5 2026: daemon connected at 12:59:13,
heartbeats worked for ~2.5 min, then went silent. TCP session showed
ESTABLISHED on the node side, no ESTABLISHED on the gateway side.
Restarting the daemon 'fixed' it — but it re-hung within minutes.
Fix: wrap both write.send() call sites in a tokio::time::timeout of
WRITE_DEADLINE=10s. If a send stalls past that, we log and return Ok()
to trigger the main-loop reconnect. Short enough that it fires long
before the 40s read-idle would (which was our only escape hatch and
never triggered because the loop was frozen).
Brings a2a rooms, delegation, and A2A ingress work back into main. Prod's
DB has migrations 0026 (group_rooms) and 0027 (a2a) applied from an
earlier hand-tagged fleet21 build cut from this branch, but main never got
them — so the CI-built server image from main refused to start against
prod's DB with "migration 26 was previously applied but is missing".
Landing the branch closes that gap: main + prod DB now share the same
migration state, so images built from main can safely roll onto gw-04.
Included:
- 0026_group_rooms.sql / 0027_a2a.sql — align main with prod's schema
- cm-api routes/a2a.rs + mcp_door.rs updates — A2A ingress and MCP door
- cm-runtime tools/delegate.rs + tools/chat.rs — delegation + N-way rooms
- frontend TeamObserver + FleetPanels + AgentObserver updates
- taxonomy.ts — delegation + A2A signals in the live world feed
Not included (still WIP on the local checkout):
- 0028_backfill_on_delete.sql / 0029_hot_query_indexes.sql
- broker pool + team-run quota changes
- Dashboard.tsx UI rename (Large World → Visualizations)
- Node write.send timeout (aaab663) — separate concern
The SSE resume fix (202e853) is preserved by auto-merge — approvals.rs
still awaits resume_run inline so the channel is ready before reply.
The `decide()` helper (used by both approve and reject) wrapped
`runtime.resume_run(ready).await` inside `tokio::spawn`, so the handler
returned 200 to the client before the resume had even started. When a
client (or test) immediately reconnected to /api/gateway with resumeFrom,
the run's broadcast channel didn't exist yet — `runtime.subscribe(run_id)`
returned None, the journal was still empty (no new events yet), and the
SSE stream closed with zero events. Symptom: approvals_api tests flaky in
CI (`unwrap on None` at line 282 of reject_over_http_executes_nothing,
sometimes `missing step_finished` on the accept path).
resume_run's *own* awaited portion is only the setup — claim_resume,
checkpoint load, and open_channel. It internally spawns the long-running
multi-step work. Awaiting it inline means we wait milliseconds for setup,
then return. By the time the client reconnects, the channel exists, so
subscribe() finds it and live-tail works. The sweeper still handles the
crash-between-decision-and-resume case for durability.
Edge-initiated inter-agent events (gated delegation, A2A ingress) bypass the
run loop, so the world SSE now polls the append-only audit log (cursor on the
BIGINT id, seeded to max on first pass) and emits:
- delegation.invoked -> agent.delegate {fromAgentId,toAgentId,toName,task}
- a2a.invoked -> a2a.invoked {agentId}
TeamObserver renders agent.delegate as an A->B handoff in the team timeline;
adds the agent.delegate taxonomy type. Reliable live (the synthesized-run path
never streamed — active_runs + first-sight cursor jump skip it).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Adds GET /api/a2a/settings (enabled + publicBaseUrl) and an A2ASection in the
Fleet overview: enable/disable A2A for the workspace, mint/list/revoke external
bearer tokens (token shown once), and the discovery URL. Claw/skill publishing
is configured per-claw (follow-up).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The agent command center's BRAIN+SURFACE columns are now a grid of compact icon
cards (system prompt / how it operates / personality / skills / capabilities /
tools / memory / safety). Clicking a card opens a modal with the full content,
nicely formatted (markdown/persona/tags), editable for the four brain text files
(system_prompt, agent_md, persona, skills_md) and saved back to the .brain via
PATCH /api/claws/{id}/brain. Capabilities/tools/memory/safety render read-only
(tools keeps its Add affordance). LIVE column unchanged.
- frontend: new AnatomyGrid (icon cards + SectionModal); ClawCommandCenter swaps the
two verbose columns for it; skills_md added to RawBrain types.
- backend: cm-brain skills_md() getter + skills_md in ClawBrainResponse so the skills
editor pre-fills (avoids blank-overwrite).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
A new "Observe" button (Eye) sits next to "+ New" in the agent chat header; clicking
flips the card into a read-only observer of that agent's conversations with other
agents, updating live as messages happen.
- frontend: AgentObserver (history from /api/claw-chat/* + live agent.message overlay
filtered to the agent, read-only banner, no composer); ClawChatSection toggle + flip.
- backend: emit a live agent.message run-event when chat.send succeeds — events.rs
AgentMessage variant, chat.send returns to_id, runtime emits in both tool paths,
world.rs normalizes agent_message → agent.message SSE. No migration, no new table.
Roadmap (not built): group/multi-party rooms; A2A protocol (a2a-rs) adoption.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
After 'claude update' replaces the binary, the daemon's re-probe ran the fresh
binary which macOS Gatekeeper re-verifies (>2s) — the 2s probe cap missed the new
version, so the UI didn't refresh (update worked but looked stale). Bump the cap to
8s; frontend polls the tools endpoint a few times post-update to catch the re-probe.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The ↑ badge on each tool card is now a button: confirm → POST
/api/nodes/{id}/tools/{tool}/update → daemon runs the tool's own updater + re-probes.
- daemon: tool_update op (spawned task so the 170s update can't stall the WS loop;
re-probes + re-sends node_tools after). Fixed command allow-list (no arbitrary
shell): claude/glm → `claude update`; kimi → `uv tool upgrade kimi-cli`; ollama →
brew upgrade (mac) / install.sh (linux); else unsupported. 4KB output cap.
- cm-api: call_timeout/request_timeout (long ops); POST .../tools/{tool}/update
(workspace-scoped, allow-list) → {ok,output}.
- frontend: ↑latest becomes an Update button → confirm → spinner → refresh/err.
Note: claude/kimi/glm are user-space (no sudo); ollama on Linux uses install.sh
(needs sudo — works on passwordless nodes, returns an error otherwise; surfaced in UI).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Each node card now shows installed versions of Docker / Claude Code / Kimi / GLM /
Ollama (conditional per node) under the ssh card, with an "update available" badge.
- daemon: probe_tools() finds docker/claude/kimi-cli/ollama across candidate bin dirs,
extracts semver from --version, reports {"t":"node_tools",...} on connect + every 15m.
- migration node_tools + tool_latest; cm-db repo node_tools (upsert/list/latest).
- cm-api: fleet.rs NodeTools uplink → upsert; tool_versions.rs spawn_latest_checker
(24h, npm/pypi/github; docker display-only); GET /api/nodes/{id}/tools (glm mirrors
claude). Spawned in clawmates-server.
- frontend: NodeTools cards on each HostCard with the ↑latest badge.
Phase 2 (one-click update execution) intentionally deferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
CI no longer references k8s; delete the dead k8s surface:
- deploy/helm/ (the chart), ci/check-helm.sh, scripts/netpol-cluster.sh
- cm-sandbox: the feature-gated K8sDriver (src/k8s.rs) + k8s_security test +
the `k8s`/`k8s-tests` features + the optional kube/k8s-openapi/rustls deps
(Cargo.lock drops the kube-rs tree). Nothing outside cm-sandbox referenced it.
Docker (bollard) DockerDriver is the sole sandbox driver. cm-sandbox + cm-runtime
compile, fmt + clippy clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Survey + fixes so the pipeline passes at the Docker level (no k8s).
- Remove k8s: drop the `sandbox-k8s` job (kind/Calico/--features k8s-tests) and the
"Helm chart lints" gate step. release.yml was already k8s-clean.
- Rust job:
- `cargo fmt --all` — fix pre-existing formatting drift (fmt --check was failing).
- clippy -D warnings: fix 3 lib warnings (cm-brain sort_by_key→Reverse, cm-api
fleet.rs doc list indentation, node_rules map_or→is_none_or).
- Regenerate the .sqlx offline cache (was missing the cm-runtime run_loop test
query → offline compile failed). DB-backed tests use testcontainers at runtime.
- Set SQLX_OFFLINE=true on the rust + e2e jobs so query! macros compile against
the committed cache deterministically (no DB needed at compile time).
- Frontend job:
- Fix the 1 ESLint error (useAgentTelemetry: no setState-synchronously-in-effect;
tag the slice with agentId + derive null on mismatch).
- Fix 2 stale panel-params tests (`terminal` is a valid app id now; assert the
current APP_IDS + use a genuinely-unknown id for the reject case).
Verified locally: fmt clean, clippy --all-targets -D warnings clean (offline),
frontend lint 0 errors, tsc clean, 86/86 frontend tests pass, build OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add PATCH /api/claws/{id}/brain (edit_brain): writes any of system_prompt / agent_md
/ persona / skills_md into the claw's .brain (best-effort) + persists system_prompt
to Postgres (authoritative) + commits a ClawSync revision.
Frontend: a reusable EditableSection (pencil → textarea → Save/Cancel → PATCH →
re-fetch brain). The SYSTEM PROMPT, HOW I OPERATE (AGENTS.md), and PERSONALITY cards
in the command center are now editable inline; saving writes back to the mapped
brain section. AGENTS.md card now always shows (so it can be authored when empty).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Gap 1 — the brain's identity sections were stored but UI-only. cm-runtime/brain.rs
`compose_system` now folds the brain's AGENTS.md ("## How I operate") + personality
into the live system prompt (after the Postgres-authoritative base, before skills +
memory; falls back to the brain's soul_md when the base is empty). Mirrors the
OpenClaw/ZeroClaw render order.
Mapping — expose `agent_md` on `GET /api/claws/{id}/brain` (ClawBrainResponse) and
render it as a collapsible "HOW I OPERATE · AGENTS.md" card in the command center's
BRAIN column (RawBrain gains agent_md; richBrain passes it through). So the section
that's now in the prompt is also visible in the UI.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Replace the single centered "anatomy profile" + resizable chat split with an
operator command center: compact 62px identity strip → 5-tile per-agent metrics
band → three independently-scrolling LIVE · BRAIN · SURFACE columns. Chat moves to
the computer's Chat app; the right computer pullout (DevicePanel) is untouched.
- backend: cm-api/routes/world.rs emits per-agent `telemetry{agentId,tokensPerMin,
costPerHr,loops,doorsPending}` in the SSE loop, from 4 batched GROUP BY queries
(usage_events tokens/min + credits/hr, active routines, pending approvals) — all
real, no migration. taxonomy `telemetry` gains optional agentId; stateKey now
keys it per-agent so slices don't clobber.
- frontend: new ClawCommandCenter + anatomy-cards (shared cards extracted from
Dashboard); useAgentTelemetry(agentId) feeds the metric band (Doors amber>0/
green=0); LIVE column streams the agent's task.update / reasoning.delta /
tool.call (replaces the mocked VitalsCard heatmap with a live activity chart).
- Dashboard: left region → full-height ClawCommandCenter; chat launcher opens the
computer Chat app; removed the dead anatomy cluster + unused imports.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Completes "agent on a node" (single-node): when an agent's placement points at a
fleet node, its terminal container runs there and the browser reaches it over a
direct WebRTC DataChannel (LAN speed), sharing a node-local volume with the
sandbox. gw-04-local agents are byte-identical to before.
- cm-sandbox/docker.rs: empty drive subpath → mount the whole volume at the target
(volume_options None), so a per-agent node-local volume auto-creates at ~/drives.
- cm-api/fleet.rs: NodeHub.open_pty/webrtc_offer carry optional container+session
(injected only when Some); node-terminal caller passes None (host shell unchanged).
- cm-runtime/terminals.rs: TerminalManager gains node_provider + placement
(mirrors SandboxManager, draining-aware); node_local_drive_mount(agent) =
clawmates_agent_<id> at ~/drives; placement_for() ensures + locates the container;
attach uses driver_for(node) (local byte-identical).
- cm-runtime/sandboxes.rs: a node-placed agent sandbox mounts the same per-agent
volume → shares files with the terminal on that node.
- cm-api/routes/terminal.rs: ticket response gains `node`; ws() bridges node-placed
agents through the NodeHub relay (WebRTC + fallback) execing into the container;
local path unchanged. server main wires with_node_provider.
- frontend: agentTerminalConnector mints the ticket then picks WebRTC (node-placed,
⚡ direct / relayed badge) vs WS (local); webrtcConnector generalized to be
endpoint-agnostic (node terminal reuses it).
Known follow-up: terminal (uid 65532) and sandbox (uid 10001) share the volume but
differ in uid — cross-container writes need an aligned uid/gid (group-writable).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Generalize the daemon's PTY spawn so a session can target either the node's host
shell (today) or `docker exec -it <container> tmux …` (the node-placed agent
terminal, which shares the container's node-local ~/drives). A `container` (+
optional `session`) field on pty_open/webrtc_offer selects the container path;
absent it, the host shell path is byte-identical to before — so the Infra node
terminal is unaffected. Threads PtyTarget through open_pty + rtc handle_offer →
build_peer → bridge_pty. Additive; nothing emits `container` yet.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Turn the Beszel-tapped metrics into a self-managing loop.
- migration node_rules (workspace/node-scoped: metric op threshold, for_seconds,
action JSONB, last_fired).
- cm-db: repo/node_rules.rs (CRUD + list_enabled); node_metrics::eval_all merges
Beszel + heartbeat scalars per node + a headroom() heuristic; nodes::status_of;
heartbeat now PRESERVES a `draining` status across heartbeats (so a cordon sticks).
- cm-api: node_rules.rs evaluator (spawn_evaluator, 20s) — when a metric condition
holds for the rule's window it fires drain / undrain / alert (in-memory sustained
+ cooldown tracking, modeled on the node sweeper); routes/beszel.rs rules CRUD
(GET/POST/PATCH/DELETE /api/fleet/rules); spawned in clawmates-server.
- cm-runtime: placement_node() is metrics-aware — a `draining` node stops receiving
new agent sandboxes (falls back to local), so the drain rule is actionable.
- frontend: FleetRules section in the Local view — build rules (node · metric · op ·
threshold · duration → action), toggle/delete, with fired-history.
The loop: hot/overloaded node → rule drains it → placement avoids it → recovers →
undrain rule brings it back. Deployed; node_rules migration applied.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Omar Sobh