Within a provider connection, repos now group under a foldable org card
(the repo's owner login). Each org shows chevron + owner + count, with
repos indented under a subtle left rail so the tree reads visually.
Sorting: orgs alphabetical, repos within an org alphabetical — makes
scanning stable when a re-sync reorders provider output.
Collapsed state lives per (connection_id, owner) so the same org name
appearing under two providers folds independently. Default is expanded
so the first pass after connecting shows everything.
Sidebar:
- Each connection header now has three inline icon buttons: Sync now
(spins while in flight), Edit (opens the modal), Remove (opens an
inline confirm strip). Removes cascade repos via ON DELETE CASCADE.
- The connection's last_sync_error surfaces as a red inline banner
under the header — no more 'error status with nowhere to see why'.
- Sync is POST /api/repos/connections/:id/sync (already existed);
after either sync or delete the sidebar re-fetches so state stays
consistent.
Edit modal (RepoConnectionEditModal):
- Loads GET /api/repos/connections/:id, pre-fills owner/base_url/label
- PATCHes only the fields that actually changed; empty string on a
Some(&str) field sends explicit null so the backend clears it
- Sync-now + Remove reachable from inside the modal too
- Rotating the token is out of scope: the modal says as much and
points the user at delete + re-create through the wizard (the
broker doesn't expose an update path, and rotating in place would
require duplicating the whole broker->store_secret flow here)
Backend:
- GET /api/repos/connections/:id — same ConnectionSummary shape
- PATCH /api/repos/connections/:id — owner/base_url use Option<Option<T>>
double-nesting so 'omit = leave alone' and 'null = clear' round-trip
distinctly through serde
- repo_connections::update with COALESCE-per-field so the SQL matches
the double-Option semantics without an OR-chain per field
Fleet's Gitea (git.redclaw.dev) hosts most of this workspace's repos,
so Gitea gets the same inline sync treatment GitHub already had.
Backend sync_gitea:
- base_url is required — Gitea has no shared 'gitea.com'; we accept
either the instance root (auto-appends /api/v1) or the fully-formed
API base if the user already included the suffix
- /orgs/{owner}/repos when owner set, /repos/search when not (with the
{data: [...], ok: bool} envelope Gitea wraps that endpoint in)
- 404 with an owner surfaces as 'org not found or PAT lacks access',
same UX as GitHub
- 50/page, capped at 20 pages (~1000 repos); short page terminates
- upsert_gitea_repo tolerates the small field-name differences
(stars_count vs stargazers_count, owner.login vs owner.username on
older versions)
Frontend wizard:
- Gitea listed first — matches the workspace's actual usage
- Default provider selection is now gitea
- Token-input placeholder tailored per provider (Gitea's is
'Settings → Applications → Generate New Token (repo)')
GitLab still returns 'not yet supported' — that's the next follow-up.
Replaces the earlier placeholder inside RepoConnectionWizardStub with the
actual flow (kept the filename so the Dashboard import doesn't churn).
- Provider picker (github / gitea / gitlab) as three inline cards
- PAT input (password field, never rendered back)
- Optional owner override (org or user)
- Optional label (defaults to <provider>/<owner>)
- Optional base URL — shown only for Gitea / GitLab, hidden for GitHub
- POST /api/repos/connections + immediate result card:
green when the initial sync succeeded (shows # repos synced), red
when the connection persisted but the sync failed (shows the message
the backend recorded on repo_connections.last_sync_error). The
sidebar refresh fires on both paths so the new row appears either way.
Sidebar and detail view already fetch the right endpoints from task
#11 — end-to-end works locally on this build.
Inserts a 6th tier tab between AGENT and INFRA (Tier type + TIER_TABS +
rail icon). Wires two new sidebar/canvas components with the same
list+detail pattern as loops/research:
- RepoList: header (provider count · repo count), + button opens the
connection wizard, groups repos by provider connection (empty state
prompts the user to connect the first). Fetches /api/repos/connections
and /api/repos — those routes land in tasks #12-14.
- RepoCanvas: repo detail (name, owner, private badge, description,
stars/forks/branch/updated, clone URL with copy, open-on-provider
link, last-synced footer). Empty + loading + error placeholders.
- RepoConnectionWizardStub: minimal 'coming next' modal so the + button
is wired end-to-end; real wizard replaces it in task #15.
Sidebar header branch updated so the tier renders its own header. Build
is clean; the sidebar is fully functional once the backend endpoints
respond.
Migration 0033: adds teams.lifecycle ('permanent' | 'ephemeral') and a
topology_runs.team_id back-ref with a partial index for the sibling-in-
flight check.
cm-db repo:
- teams::insert_team_with_lifecycle (insert_team keeps the permanent default)
- topology_runs::enqueue_run_for_team (populates team_id)
- topology_runs::check_ephemeral_teardown — atomic SELECT that only
returns Some when the team is ephemeral AND no siblings are still
queued/running; carries the workspace + bound claw ids for cleanup.
cm-api:
- topology_worker post-terminal hook maybe_teardown_ephemeral_team
runs deprovision_claw on each bound claw (best-effort; failures log
but don't block Postgres deletion), then hard_purge each agent row,
then delete_team.
- routes::teams::build_team_with_lifecycle (build_team keeps default);
run_team enqueues with team_id.
- planner ScaffoldRequest gains mode; lifecycle_for(mode) sets the team
to ephemeral for scheduled + triggered, permanent otherwise.
Frontend MasterPlannerModal passes mode in the scaffold payload so the
backend can derive lifecycle without duplicating the mode taxonomy.
Tests: 3 new (returns claws when no siblings, holds when siblings queued,
ignores permanent teams). 10/10 topology_jobs green; workspace clippy
--tests clean.
Frontend: send topologyKind to /api/planner/chat so the planner's user
prompt gets a USER-LOCKED TOPOLOGY block telling Opus to use it verbatim.
On buildTeam, override proposal.topology_kind with the user's pick
(belt-and-braces — if the planner ignored the lock, we still ship the
right shape). Proposal chip renders the effective kind in a lavender
tint when it was overridden, with a hover title showing what was
replaced.
Backend PlannerChatRequest gains an optional topology_kind. Empty /
absent = planner picks. Not honored for 'swarm' mode (swarm planner
doesn't take a topology kind).
Selecting a card in the strip expands into an info panel below with:
- Coordinator badge (green when the topology has a lead, gray when
leaderless — mirrors what cm-orchestrator's planner reads off the graph)
- Communication pattern one-liner
- 'When to use' guidance
- Auto-staffed role distribution chips (from the catalog's
role_distribution — same numbers TeamWizard apportions)
- ASCII sketch of the shape
Cheat sheet is client-side (TOPO_DETAIL) so the expansion is instant on
selection — mirrors the exec-plan semantics baked into cm-orchestrator.
New horizontally-scrollable card strip appears between the mode selector
and the split pane when the mode is 'team' or 'swarm'. Cards are fetched
from GET /api/topologies (the full 12-kind catalog), rendered as pill
cards with name + description. Clicking selects; 'planner picks' clears.
Selection persists per-mode via the ModeSlice snapshot, so switching
modes doesn't forget the choice. Scaffold wire-through comes next
(task #9).
switchMode was resetting messages/proposal/swarm/runSteps/etc. every
time. Now it snapshots the current mode's slice into a ref before
loading the target mode's snapshot (or a fresh slice if the target
was never visited). Switching specialists → swarm → specialists keeps
the specialists chat intact.
Also route async /planner/chat responses to the mode the message was
sent in, not the currently-active mode — user can switch modes while
the reply is in flight without the assistant response landing in the
wrong chat.
@/lib/api/topology's apiFetch pulls in bearer.ts which imports
@clerk/nextjs/server — the whole chain gets tagged as client-side
by Next when LoopsWizard imports it, and 'server-only' breaks the
production build. Switch to plain fetch(/api/topologies) with
type-only imports (mirrors what TeamWizard does). Local pnpm build
now compiles clean; typecheck/lint already pass. This unblocks the
publish job on the topology-builder push.
Step 2 now defaults to a Builder pane: kind picker (from /api/topologies)
+ team size + role distribution preview, with /api/topologies/build
rendering the canonical graph and node/edge counts. Advanced JSON stays
as a toggle for hand-crafted graphs — same shape lands in the payload
either way, so downstream code is unchanged.
Debounced build effect wraps the async load in an inner function to
avoid the setState-in-effect cascading-renders lint. Uses the same
largest-remainder role apportionment as TeamWizard so builder output
matches team-wizard output for the same kind + size.
Third radio option on the repeat step. Two inputs: event name (defaults
to 'ok') and a within_iters cap (defaults to 20). Ships the full
{kind: 'until', event, within_iters} payload the schema already accepts.
LoopsCanvas repeat summary now formats iters/until/infinite via one
helper — the previous inline expression was rendering 'until' as a bare
label with no event context.
Extend the topology-runs list route with an optional loop_id filter that
returns iterations for a single loop, newest-iteration-first. Adds the
iteration and finished_at columns to the summary (skip-null on the JSON
so compares stay compact). Backed by list_by_loop in the repo, which uses
the existing topology_runs_loop_idx partial index.
LoopsCanvas fetches the runs in parallel with the loop detail and renders
an iteration timeline card (iteration #, status pill, start time, duration,
run id prefix) between the graph section and the actions row.
Seventh and final commit of the Research + Loops arc. Replaces the
placeholder Loops surface from commit 5 with the full working UI.
New:
- lib/api/loops.ts — thin TypeScript client for every /api/loops
endpoint (list, get, create, patch, delete, run-now, enable, disable).
- dashboard/LoopsWizard.tsx — 4-step modal:
1. title + description
2. task template + topology graph JSON (defaults to empty graph; a
visual builder is future work)
3. triggers — any combination of cron / on-completion / webhook,
with the cron pattern input revealed inline when cron is on
4. repeat policy — infinite or fixed-iterations
When the webhook trigger is enabled, submit lands on a fifth SECRETS
screen showing webhook_token + signing_key exactly once (never shown
again by the backend), a copy-to-clipboard row for each, and the
X-Loop-Signature: sha256=<HMAC-SHA256(key, body)> usage snippet.
- dashboard/LoopsList.tsx — replaces the stub. Loop cards with an
enabled/disabled dot + a next-fire chip. Re-fetches on refreshKey.
- dashboard/LoopsCanvas.tsx — replaces the stub. Selected loop detail:
header (title, enabled state, next-fire), triggers card, repeat
policy, task template preview, topology graph JSON, and an action row
(Run now / Enable-or-Disable / Delete with confirm).
Wired into Dashboard.tsx via a `loopsSel` / `loopsRefresh` state pair
matching the Research pattern. onDeleted clears the selection and bumps
the refresh key so the list drops the deleted card and the canvas
returns to its placeholder.
Closes the arc: all 5 tiers active, all state machines driveable from
the UI, and the loop scheduler + cron + webhook plumbing wired end-to-
end. Future work per the roadmap: visual topology builder, iteration
timeline in the canvas (via GET /api/topology-runs?loop_id=X), and the
"until" repeat policy UI.
Second lint pass caught the standalone setTopic(null) in the early-return
branch of the effect — still synchronous at the top of the useEffect. Fold
that guard into the async load() so every setState landing on this hook
runs inside the async wrapper.
Push right after ec1ddc6 tripped three eslint errors on the same-commit
lint pass. All three are cosmetic in behavior:
- ResearchList.tsx:48 + ResearchCanvas.tsx:64 — setState was called
synchronously at the top of a useEffect. Wrap the body in an inner
async load() and call setState inside it; effect still returns the
cleanup for the alive-flag guard.
- ResearchWizard.tsx:207 — bare apostrophe in JSX text ("workspace's").
Replace with ' to satisfy react/no-unescaped-entities.
Sixth commit of the Research + Loops arc. Replaces the placeholder
Research surface from commit 5 with the full working UI.
New:
- lib/api/research.ts — thin TypeScript client for every /api/research
endpoint (list, get, create, attach/detach agent, start, submit-review,
request-publish, list/approve/reject publish approvals, wizard refine).
- dashboard/ResearchWizard.tsx — 4-step modal (topic prompt → LLM refine
→ outcome kind → agents). The refine step calls
POST /api/research/wizard/refine which streams the workspace's default
LLM and returns {title, description}. Users can accept, edit, or refine
again. Manual fallback if the LLM call errors.
- dashboard/ResearchList.tsx — replaces the stub. Real topic cards with
status pill (standby / processing / reviewing / publishing / published),
outcome-kind chip, and the + button that opens the wizard. Re-fetches
when the parent bumps refreshKey.
- dashboard/ResearchCanvas.tsx — replaces the stub. Selected topic
detail: title, status header, outcome + published_at meta, assigned
agents grid (matched to workspace claws by id), the full description
in a monospace preformatted block, and a state-appropriate primary
action button (Start research → Submit for review → Request publish).
Wired into Dashboard.tsx via a `researchSel` / `researchRefresh` state
pair: selecting a card sets the id, mutations bump the counter so both
list + canvas re-fetch.
All API calls go through the existing /api/[...path] catch-all Next
proxy — no new server routes needed.
Fifth commit of the Research + Loops arc. Lights up the tier rail (both
the chip column and the top-bar crumbs) with two new peer surfaces:
[VIZ] [RESEARCH] [LOOPS] [AGENT] [INFRA]
The stubs (ResearchList / ResearchCanvas / LoopsList / LoopsCanvas) are
self-contained placeholder components — each renders a header with a
"0 topics" / "0 loops" chip and a "coming soon" body plus a canvas hero
with the tier motif. Real functionality (topic cards, wizard, canvas
timelines, publish gate UI) arrives in the next two commits, and slots
in by replacing the stub files without touching Dashboard.tsx again.
Dashboard.tsx changes:
- `Tier` type widened to include `"research" | "loops"`.
- railIcon record grew two SVG entries — book+lens for Research, orbit
arrows for Loops. Matches the placeholder canvas hero.
- TIER_TABS reordered as Viz → Research → Loops → Agent → Infra to
match the mental grouping (conceptual work first, machinery second).
- Top-bar crumbs mirror the rail order.
- New isResearch / isLoops selectors used everywhere the existing tiers
were tested; the Agents-header conditional now also skips for the new
tiers so ResearchList and LoopsList can supply their own headers.
- Sidebar swap short-circuits to ResearchList / LoopsList; canvas swap
short-circuits to ResearchCanvas / LoopsCanvas.
Three surfaces show the tier name: the rail chip, the top-bar crumb, and
the sidebar header. Rename rail "WORLD" → "VIZ", crumb + sidebar "Large
World" → "Visualizations".
Canvas narrowing: the left StructureTree keeps the full org forest so you
can still browse everything, but the right-side WorldCanvas now prunes to
just the branch containing the current selection. Clicking a company shows
only that company's teams; clicking an org shows only that org. Nothing
selected still shows the full forest.
Edge-initiated inter-agent events (gated delegation, A2A ingress) bypass the
run loop, so the world SSE now polls the append-only audit log (cursor on the
BIGINT id, seeded to max on first pass) and emits:
- delegation.invoked -> agent.delegate {fromAgentId,toAgentId,toName,task}
- a2a.invoked -> a2a.invoked {agentId}
TeamObserver renders agent.delegate as an A->B handoff in the team timeline;
adds the agent.delegate taxonomy type. Reliable live (the synthesized-run path
never streamed — active_runs + first-sight cursor jump skip it).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Adds a "This agent | Team" scope toggle to the chat card's Observe view. Team
mode (TeamObserver) resolves the open agent's team (members + names via
/api/teams + /api/team/claws), lists the team's group rooms, and renders a live
timeline of agent.message / room.message / a2a.invoked among team members —
read-only, reusing the workspace SSE feed. No backend changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Adds GET /api/a2a/settings (enabled + publicBaseUrl) and an A2ASection in the
Fleet overview: enable/disable A2A for the workspace, mint/list/revoke external
bearer tokens (token shown once), and the discovery URL. Claw/skill publishing
is configured per-claw (follow-up).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The 8 section icon cards now sit inside a labeled .brain panel (Brain icon + '.brain'
header) arranged 4 per row × 2 rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The agent command center's BRAIN+SURFACE columns are now a grid of compact icon
cards (system prompt / how it operates / personality / skills / capabilities /
tools / memory / safety). Clicking a card opens a modal with the full content,
nicely formatted (markdown/persona/tags), editable for the four brain text files
(system_prompt, agent_md, persona, skills_md) and saved back to the .brain via
PATCH /api/claws/{id}/brain. Capabilities/tools/memory/safety render read-only
(tools keeps its Add affordance). LIVE column unchanged.
- frontend: new AnatomyGrid (icon cards + SectionModal); ClawCommandCenter swaps the
two verbose columns for it; skills_md added to RawBrain types.
- backend: cm-brain skills_md() getter + skills_md in ClawBrainResponse so the skills
editor pre-fills (avoids blank-overwrite).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
A new "Observe" button (Eye) sits next to "+ New" in the agent chat header; clicking
flips the card into a read-only observer of that agent's conversations with other
agents, updating live as messages happen.
- frontend: AgentObserver (history from /api/claw-chat/* + live agent.message overlay
filtered to the agent, read-only banner, no composer); ClawChatSection toggle + flip.
- backend: emit a live agent.message run-event when chat.send succeeds — events.rs
AgentMessage variant, chat.send returns to_id, runtime emits in both tool paths,
world.rs normalizes agent_message → agent.message SSE. No migration, no new table.
Roadmap (not built): group/multi-party rooms; A2A protocol (a2a-rs) adoption.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Nodes without an onboard GPU + not on Beszel (e.g. smith.lan) left that row empty.
Show the daemon's already-collected swap + 5m/15m load there instead, so every card
is full. (Temp still comes from Beszel; a non-Beszel Mac has no temp source.)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
After 'claude update' replaces the binary, the daemon's re-probe ran the fresh
binary which macOS Gatekeeper re-verifies (>2s) — the 2s probe cap missed the new
version, so the UI didn't refresh (update worked but looked stale). Bump the cap to
8s; frontend polls the tools endpoint a few times post-update to catch the re-probe.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The ↑ badge on each tool card is now a button: confirm → POST
/api/nodes/{id}/tools/{tool}/update → daemon runs the tool's own updater + re-probes.
- daemon: tool_update op (spawned task so the 170s update can't stall the WS loop;
re-probes + re-sends node_tools after). Fixed command allow-list (no arbitrary
shell): claude/glm → `claude update`; kimi → `uv tool upgrade kimi-cli`; ollama →
brew upgrade (mac) / install.sh (linux); else unsupported. 4KB output cap.
- cm-api: call_timeout/request_timeout (long ops); POST .../tools/{tool}/update
(workspace-scoped, allow-list) → {ok,output}.
- frontend: ↑latest becomes an Update button → confirm → spinner → refresh/err.
Note: claude/kimi/glm are user-space (no sudo); ollama on Linux uses install.sh
(needs sudo — works on passwordless nodes, returns an error otherwise; surfaced in UI).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Each node card now shows installed versions of Docker / Claude Code / Kimi / GLM /
Ollama (conditional per node) under the ssh card, with an "update available" badge.
- daemon: probe_tools() finds docker/claude/kimi-cli/ollama across candidate bin dirs,
extracts semver from --version, reports {"t":"node_tools",...} on connect + every 15m.
- migration node_tools + tool_latest; cm-db repo node_tools (upsert/list/latest).
- cm-api: fleet.rs NodeTools uplink → upsert; tool_versions.rs spawn_latest_checker
(24h, npm/pypi/github; docker display-only); GET /api/nodes/{id}/tools (glm mirrors
claude). Spawned in clawmates-server.
- frontend: NodeTools cards on each HostCard with the ↑latest badge.
Phase 2 (one-click update execution) intentionally deferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Infrastructure is the breadcrumb tier and the brain registry opens from the agents
sidebar, so the top-right menu becomes account-only. Relabel the section ACCOUNT.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Wrap the Tailscale network card + Beszel monitoring in a repeat(auto-fit,minmax(340px,1fr))
grid so they share one row as two columns (stacking when the region is narrow);
moved the bottom spacing to the grid wrapper.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The wizard mints a pending node on open (POST /api/nodes/pair). Cancelling left it
behind as a 'waiting for daemon to dial home' card. handleClose now DELETEs the node
on close unless it actually connected (everConnected) — so cancel leaves no card,
while a node that already dialed home is kept.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Inline card editing + the sidebar brain registry cover editing now; drop the
redundant top-right model chip and Edit-brain button (and the unused onEditBrain prop).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add PATCH /api/claws/{id}/brain (edit_brain): writes any of system_prompt / agent_md
/ persona / skills_md into the claw's .brain (best-effort) + persists system_prompt
to Postgres (authoritative) + commits a ClawSync revision.
Frontend: a reusable EditableSection (pencil → textarea → Save/Cancel → PATCH →
re-fetch brain). The SYSTEM PROMPT, HOW I OPERATE (AGENTS.md), and PERSONALITY cards
in the command center are now editable inline; saving writes back to the mapped
brain section. AGENTS.md card now always shows (so it can be authored when empty).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Gap 1 — the brain's identity sections were stored but UI-only. cm-runtime/brain.rs
`compose_system` now folds the brain's AGENTS.md ("## How I operate") + personality
into the live system prompt (after the Postgres-authoritative base, before skills +
memory; falls back to the brain's soul_md when the base is empty). Mirrors the
OpenClaw/ZeroClaw render order.
Mapping — expose `agent_md` on `GET /api/claws/{id}/brain` (ClawBrainResponse) and
render it as a collapsible "HOW I OPERATE · AGENTS.md" card in the command center's
BRAIN column (RawBrain gains agent_md; richBrain passes it through). So the section
that's now in the prompt is also visible in the UI.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Replace the single centered "anatomy profile" + resizable chat split with an
operator command center: compact 62px identity strip → 5-tile per-agent metrics
band → three independently-scrolling LIVE · BRAIN · SURFACE columns. Chat moves to
the computer's Chat app; the right computer pullout (DevicePanel) is untouched.
- backend: cm-api/routes/world.rs emits per-agent `telemetry{agentId,tokensPerMin,
costPerHr,loops,doorsPending}` in the SSE loop, from 4 batched GROUP BY queries
(usage_events tokens/min + credits/hr, active routines, pending approvals) — all
real, no migration. taxonomy `telemetry` gains optional agentId; stateKey now
keys it per-agent so slices don't clobber.
- frontend: new ClawCommandCenter + anatomy-cards (shared cards extracted from
Dashboard); useAgentTelemetry(agentId) feeds the metric band (Doors amber>0/
green=0); LIVE column streams the agent's task.update / reasoning.delta /
tool.call (replaces the mocked VitalsCard heatmap with a live activity chart).
- Dashboard: left region → full-height ClawCommandCenter; chat launcher opens the
computer Chat app; removed the dead anatomy cluster + unused imports.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The claw anatomy stage capped its content to an 820px column with margin:0 auto,
so on a wide left region the whole profile (incl. Morpheus's avatar) floated in the
center with large empty space toward the left sidebar. Left-align it (margin:0) and
widen to 920 so the profile hugs the left.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Turn the Beszel-tapped metrics into a self-managing loop.
- migration node_rules (workspace/node-scoped: metric op threshold, for_seconds,
action JSONB, last_fired).
- cm-db: repo/node_rules.rs (CRUD + list_enabled); node_metrics::eval_all merges
Beszel + heartbeat scalars per node + a headroom() heuristic; nodes::status_of;
heartbeat now PRESERVES a `draining` status across heartbeats (so a cordon sticks).
- cm-api: node_rules.rs evaluator (spawn_evaluator, 20s) — when a metric condition
holds for the rule's window it fires drain / undrain / alert (in-memory sustained
+ cooldown tracking, modeled on the node sweeper); routes/beszel.rs rules CRUD
(GET/POST/PATCH/DELETE /api/fleet/rules); spawned in clawmates-server.
- cm-runtime: placement_node() is metrics-aware — a `draining` node stops receiving
new agent sandboxes (falls back to local), so the drain rule is actionable.
- frontend: FleetRules section in the Local view — build rules (node · metric · op ·
threshold · duration → action), toggle/delete, with fired-history.
The loop: hot/overloaded node → rule drains it → placement avoids it → recovers →
undrain rule brings it back. Deployed; node_rules migration applied.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Tap each node's Beszel metrics (GPU/temps/disk-IO/network/per-container — beyond
our basic heartbeat) by reading the workspace's Beszel hub. The agents run in
WS-only mode with no locally-readable socket, so (per the de-risk) the server taps
the hub's PocketBase API instead of the daemon reading agents — no daemon changes.
- migrations: workspace_beszel (BYO hub URL + login, server-side only, mirrors the
Tailscale BYO pattern) + node_metrics (latest scalar columns + JSONB blob).
- cm-db: repo/fleet_beszel.rs, repo/node_metrics.rs; nodes SELECT joins node_metrics
(gpu_pct/temp_max surfaced on node_json for the live cards).
- cm-api: beszel.rs client (auth-with-password, poll `systems`, map to nodes by
hostname, upsert metrics) + a 15s spawn_poller; routes/beszel.rs (connect/status/
disconnect + GET /api/nodes/{id}/metrics with history proxied live from the hub).
- frontend: HostCard gains a GPU/temp readout + a Monitor button; NodeMonitor is a
full-width per-node page (current panel + CPU/mem/GPU/temp/net/disk charts from the
hub's 1m history); a "Beszel monitoring" connect form in the Local view.
Reachability confirmed: gw-04 → the hub over the tailnet (100.123.224.84:8090). Needs
the user to connect their hub login to activate the poller.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Nodes flapped online/offline and the terminal died on the first blip. WebSockets
are the right transport (outbound, NAT-friendly); the fixes harden around it.
Server (cm-api):
- Anti-clobber connection epoch: a reconnecting daemon gets a fresh epoch; a stale
run_channel's teardown only clears the hub + sets offline if it still owns the
slot — so a lingering old channel can't flip a live reconnection offline (the
main false-offline cause).
- WS keepalive: run_channel now pings every 15s and tears down if no inbound
frame (incl. pong) for 35s — dead links detected in seconds, not minutes.
- Staleness sweeper backstop: spawn_node_sweeper (8s tick / 20s window) wired in
clawmates-server, so a vanished node goes offline within ~28s even if its
channel hangs (mark_stale_offline was defined but never called).
Daemon (clawmates-node v0.3.0):
- Heartbeats off the select thread (dedicated thread owns System + blocking
docker/tailscale/disk CLIs) so a slow op never starves heartbeats/pongs.
- Each handle_frame runs on its own task; added a 40s inbound idle deadline so a
half-open socket triggers a reconnect.
Frontend:
- useNodes streams /api/nodes/live (SSE push) instead of a 3s poll; isLive()
derives online from lastSeen freshness (<15s) so a transient column flip never
shows a healthy node down.
- Node terminal: clean auto-reconnect loop (re-mint ticket -> reconnect -> tmux
re-attaches and redraws the live screen = mosh-style snap-to-state over TCP),
replacing the [disconnected] dead-end.
Mosh evaluated: harvest principles (session/transport decoupling, snap-to-state,
already given by tmux), don't adopt — UDP is incompatible with our browser+CF+NAT
topology and it's GPLv3. Removed temporary terminal debug traces + /api/debug route.
Verified: node holds steadily online (heartbeat 1-3s, no flap) and goes cleanly
offline when the daemon stops.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Refactor the Infrastructure tier into the dark operator console from the design
package, wired to the real fleet backend (no new plumbing):
- InfraNav: page-scoped left nav grouped RUN HERE FIRST (Local & Tailscale, with
online count + sub-items) / NEXT (Cloud providers, SOON) + coral "Connect a
host" button; coral spine on the active view.
- FleetConsole: center reads top-to-bottom — kicker → "Your fleet" → 5 stat tiles
(HOSTS/ONLINE/MEMORY/STORAGE/CONTAINERS) → Tailscale device card → LOCAL HOSTS
host cards + add-host dashed tile. Cloud view = how-it-works + SOON.
- Host cards (design system): status dot (blink when online) + hostname, IP/version
line, CPU/RAM meter bars (cyan→amber→coral by load), disk/load/ctrs mini-row, and
a footer that's a copyable `ssh <node>` target (online) OR a "waiting for daemon"
spinner (pairing) — PLUS a Terminal button that opens the in-app shell.
- FleetPill (top bar, N/M hosts online) + FleetStatusBar (ambient: daemon · tailnet
· WSS · sandbox).
Per the chosen reconciliation: kept the cloud-apps computer pull-out as the right
column, the connect-host wizard as a modal, and the in-app terminal. Dashboard
infra branch rewired (InfraNav + FleetConsole + status bar; dropped the old
InfraStage/InfraConsole split). cm-blink/spin/cm-fade keyframes already existed.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Hostname/IP:
- Daemon reports the machine's hostname (sysinfo) + primary outbound IPv4 on each
heartbeat. migrations/0021 adds nodes.hostname/local_ip; cm-db heartbeat stores
them; node JSON exposes them. Cards now title on the real hostname (falling back
to name) + show the IP, instead of the "New node" placeholder. `name` stays
user-overridable (rename).
Terminal moved into the pull-out computer (no more per-card modal):
- New infra computer app NodeTerminalApp (computer/apps/infra) — xterm bridged to
a node's host shell over the node control channel, filling the app window
(mirrors the agent Terminal's layout + ResizeObserver). Added "terminal" to the
INFRA_CATALOG grid; a ?node= panel param targets a specific node (picker when
unset). Clicking Terminal on a node card now opens the infra computer to that
node's shell instead of a separate full-screen window. Deleted NodeTerminal.tsx.
Rebuilt + re-hosted both daemon binaries (hostname change).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Agents can now provision their sandbox on a connected fleet node instead of the
gateway host. Local stays the strict default, so existing agents are byte-for-
byte unaffected until explicitly placed elsewhere.
Security parity: the daemon links the REAL cm-sandbox DockerDriver and runs the
typed container ops (sb_provision/sb_exec/sb_destroy/sb_health/sb_list) through
it — identical hardening (cap-drop ALL, seccomp, no-net, read-only, non-root) to
local sandboxes. cm-sandbox spec types are now Serialize/Deserialize so the spec
crosses the channel.
- cm-api: RemoteDriver (impl SandboxDriver over the node channel) + HubDriverProvider
(impl cm_runtime::NodeDriverProvider, hands out a driver only for connected
nodes via a sync online set) + NodeHub.call/is_connected. AppState.with_node_hub
so the hub is shared with the placement provider.
- cm-runtime SandboxManager: driver_for(node_id) routes by the recorded
agent_containers.node_id (local default = existing driver, identical path);
placement_node() reads the workspace setting and falls back to local if the
node is offline; exec/release route accordingly. NodeDriverProvider trait.
- DB: 0020_workspace_placement + repo (for_agent/get/set/clear).
- main.rs: build the NodeHub first; inject HubDriverProvider into the agent
manager + share the hub with AppState.
- API+UI: GET/PUT /api/fleet/placement + a "Run agents on: Local / <node>"
selector in the Fleet overview.
Note: a node must be able to pull the agent image (the daemon docker-pulls it);
interactive PTY for agent containers on remote nodes is not wired (Terminal app
stays local) — the in-dashboard node shell already covers host access.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Omar Sobh