fetch(url, { cache: no-store }) was hanging forever through the
edge proxy on the mission API endpoints — requests never reached
postgres and the client-side loading state was stuck true, making
Refresh appear broken. Regressed in d42398d.
Switch to a per-request _t=Date.now() query param on GETs — same
cache-defeat effect, doesn't change fetch semantics.
The refresh button was suspected of being inert when loading is
somehow stuck true. Removes disabled and renders a bulletproof
click counter + loading state next to the icon:
clicks:0 · updated 14:05:12 · idle
- clicks bumps SYNCHRONOUSLY in onClick before any await, so a
non-zero counter proves the click event reaches the handler
- console.log fires alongside for devtools verification
- disabled={loading} removed; if load happens to hang, at least
the user can click again to retry
Temporary scaffolding — will collapse once the root cause is clear.
- api client: cache: no-store so manual Refresh guarantees a fresh
server response (was potentially hitting stale HTTP cache).
- MissionCanvas: renders "updated HH:MM:SS" next to the refresh
button; the timestamp bumps on every successful load so the click
is visibly acknowledged even when nothing else on the page changed.
UUIDv7 encodes time in the leading bytes so 12-hex prefixes are
NOT unique across missions minted in the same second. Docker
accepts up to 253 chars; use the full uuid.
- mission_runtime::spawn_sweeper: force-removes runtime containers
for missions terminal for >=30 min, clears runtime_endpoint. Wired
into clawmates-server main().
- docker-compose socket-proxy: NETWORKS=1 so bollard.connect_network
can attach containers to clawmates_edge for provider egress.
- phase_runner ordering: ensure_checkout BEFORE ensure_container so
the mission dir exists before docker mounts it.
- provisioner: mkdir_p the mission dir defensively for research-only
missions that skip checkout entirely.
When a topology_run is bound to a mission whose runtime_endpoint is
set, the worker constructs ZeroClawDriveExecutor against that URL
instead of the env-derived shared gateway. Falls back to shared for
non-mission runs and pre-C3 missions.
With slices 1-3 combined, a mission launched after this deploy will:
1. get its per-mission container spawned during on_launch
2. have its checkout dropped into /var/lib/clawmates-missions/<id>
which is bind-mounted to /mission inside that container
3. run its agents against ZEROCLAW_WORKSPACE=/mission/repo — so
they can see and edit only this missions repo, no bleed-over.
- mission_orchestrator::on_launch now calls ensure_container after
the repo checkout, persists the container_name + endpoint on the
missions row. Non-fatal — logs and continues on docker errors so
dev-mode + tests keep working.
- phase_runner::launch_phase does the same as a fallback for any
mission whose runtime_endpoint is null (pre-C3 or torn down).
Nothing reads the endpoint yet; slice 3 swaps topology_worker over.
- migration 0058: adds missions.runtime_container_name + runtime_endpoint
- new mission_runtime module (bollard): ensure_container /
teardown_container. Container is spawned on clawmates_core +
clawmates_edge networks with just /var/lib/clawmates-missions/{id}
bind-mounted so agents scoped to /mission/repo can only see this
missions repo.
- provider API keys forwarded from the server envs so per-mission
runtimes inherit them.
- Mission struct + repo helpers updated for the two new columns +
set_runtime_binding().
- Unit tests cover container naming determinism + entropy.
Not wired to the orchestrator yet — that lands in slice 2.
Moves ensure_checkout into launch_phase so retries + new phase
launches all trigger the clone/fetch. mission_orchestrator still
does its own checkout at initial launch time, so first-launch
timing is unchanged; this covers the retry + additional-phase
paths.
- mission_orchestrator: run ensure_checkout BEFORE the team_id
short-circuit. Previously, a re-launched or retried mission bailed
out at the team_id=already-bound guard and skipped repo checkout
entirely, so agents ran against an empty workspace.
- mission_workspace: inject GITEA_TOKEN into git.redclaw.dev URLs so
clone auth works from the server container. Redact any token
echoed back on failure.
- refresh buttons on MissionCanvas + MissionsList now spin the icon
while loading so clicks are visibly acknowledged.
- refresh-spinner keyframe added to motion.css.
Requires operator on gw-04: sudo chown 65532:65532 /var/lib/clawmates-missions
(applied 2026-07-21 pre-commit).
Adds GET /api/topology-runs/{id}/output — trimmed view of the
runs checkpoint (totals + per-turn output previews, capped at
12 turns × 6kB each). The full checkpoint blob can be hundreds
of KB so it was never viable to send through mission polling.
Phase card run rows now expose a "show output" toggle for any
terminal run (completed/failed/cancelled), rendering turns,
tokens, records count, and per-turn agent text. Running rows
still get the live activity stream from the prior slice.
Diagnostic value: on a mission that "completed" without visible
work, this immediately shows whether the agents produced real
output (workspace missing / instructions vague / etc.) or
whether nothing ran at all.
Adds a "show activity" toggle to any running topology_run row on
the phase card. Expanded rows mount a compact SSE tail from
/api/topology-runs/{id}/events, rendering step/reasoning/tool
events as they arrive — same stream the LIVE tab consumes, just
scoped to one run.
Extracted the phase-runs list into PhaseRunsList to keep
MissionCanvas under the 1250-line budget.
Every re-attempted phase now starts with a clean slate:
- phase_runner::launch_phase DELETEs prior status IN ('failed',
'cancelled') topology_runs for the phase before enqueuing the
new ones. Completed runs are kept for audit; only the failure
noise from earlier attempts goes.
- POST /api/missions/{id}/phases/{phase_id}/retry — resets a
failed/cancelled phase to 'pending' (auth-scoped to the calling
workspace + guarded on mission.status='running'). phase_runner
picks it up on the next 10s tick.
- MissionCanvas phase card grows a coral 'Retry' button, visible
only when phase.status='failed' and mission.status='running'.
Click → resets + refreshes; the prior failed run rows disappear
from the card as soon as phase_runner enqueues the new attempt.
Design: auto-purge in phase_runner rather than a separate 'clear
failed runs' endpoint. Users don't have to manually clean up before
retrying; the runner does it as part of the natural work of firing
a fresh attempt.
Verified: cargo check + tsc + eslint --quiet all green.
Two durability fixes closing recurring flakes:
CI flake wrapper (broker + server Dockerfiles):
Wrapped the cargo build step in a 3-attempt retry loop with
linear backoff (10s / 20s). Directly targets the crates.io
transient network errors that keep hitting CI on the runners
('curl failed: SSL_ERROR_SYSCALL, errno 0'). Each build only
loses time on transient failures; a real compile error still
fails all 3 attempts and surfaces the last error normally.
Runtime systemd unit (deploy/clawmates-runtime/):
Replaces the manual 'docker run' that had been starting the
ZeroClaw runtime with no persistence for its network topology.
Ephemeral prod fixes at 09:30 PDT 2026-07-21 (task #38) were:
- anthropic.default provider block added to
/root/clawmates-runtime/data/.zeroclaw/config.toml (already
durable — bind-mounted from host)
- docker network connect clawmates_edge clawmates-runtime
(NOT durable — vanishes on container recreate)
New systemd unit clawmates-runtime.service (installed +
enabled on gw-04):
- ExecStart docker-runs the container attached to
clawmates_core, then connects clawmates_edge in the same
shell command, then docker waits.
- Bind-mounts both /root/clawmates-runtime/data and
/var/lib/clawmates-missions (for security_scan +
benchmark_runner).
- --rm so upgrading is just docker pull + systemctl restart.
- Restart=on-failure with 5s backoff.
Closes task #38 and preemptively closes the CI flake pattern.
Adds inline failure debugging to the Phases tab. When you see a
phase card marked FAILED, click the collapsed error summary and the
full topology_run.error text expands under it — the exact stack
trace / provider error / whatever the worker recorded.
Backend:
- TopologyRunSummary gains mission_phase_id + team_id + error
fields. list_by_mission SELECT extended; other constructor
(list_recent) explicitly passes None for the new fields.
- GET /api/missions/{id}/runs response now carries all of the
above so the frontend can attribute failures per phase.
Frontend:
- MissionRunSummary type mirrors backend additions.
- MissionCanvas fetches runs alongside mission on load +
auto-refresh; indexes by mission_phase_id in a memoized Map.
- Each phase card renders a per-run row: colored status pill
(running / completed / failed), short run id, finished_at
timestamp. For failed runs, a <details> collapses the error
text — first line as summary, full 4kB in a monospace <pre> on
expand.
Directly unblocks the "phase says Failed but there's no info to
debug" report. Both research and coding phases get this — the code
path is phase-kind-agnostic.
Root-cause fix for "we hit launch, waited overnight, nothing ran."
mission_orchestrator materialized teams + agents fine, but nothing
enqueued the actual work — mission_phases stayed 'pending' forever
and topology_runs count for the mission was 0.
New crates/cm-api/src/phase_runner.rs — background worker on 10s
poll that does three things:
1. start_pending_phases — for every mission_phase with
status='pending' AND parent mission.status='running' AND all
lower-order phases already 'completed', enqueue one
topology_runs row per team whose (mission_id, purpose) matches
the phase kind:
phase=research → teams with purpose='research'
phase=coding → teams with purpose='coding'
phase=benchmark → teams with purpose='coding' (fallback)
phase=security_scan → teams with purpose 'security' | 'coding'
Each run gets a phase-kind-specific task text combining the
mission title/description + a directive for that phase.
Flips phase to 'running' after enqueue.
2. close_finished_phases — SQL sweep that flips phases whose
topology_runs are all terminal to 'completed' (or 'failed' if
any run failed).
3. close_finished_missions — same shape for missions whose phases
are all terminal.
Spawned alongside task_card_worker in clawmates-server main.rs.
Ordering enforced by mission_phases.order_idx — a coding phase
doesn't fire until its research phase completes.
Idempotent: every state transition is guarded so double-firing on a
race is safe. When a mission has no matching teams for a phase (bad
wizard state), the phase stays pending and the runner logs a skip
rather than getting stuck in a fail loop.
Existing topology_worker picks up the queued runs and drives them
through the ZeroClaw executor as usual.
Missions created via the new multi-team wizard have neither team_id
nor team_template_id set — they carry config.phase_teams. Both the
frontend Launch button gate and the backend set_status precondition
were checking only the old two fields, disabling launch for every
new wizard-created mission with a "No team" tooltip.
- MissionCanvas: hasTeam now also returns true when
mission.config.phase_teams has at least one non-empty list.
- routes::missions::set_status: same check on the server so a
direct API caller with only config.phase_teams also gets past
the gate.
Directly unblocks the "we just finished the wizard, Launch is greyed
out" report. Agents materialize AFTER Launch — the button is the
trigger, not a post-condition of creation.
Adds the operator's five categorized research team archetypes:
1. codebase_research — code archeologist, architecture mapper,
flow tracer, vault scribe. Produces Obsidian vault entries
under Codebases/<repo>/ that make future missions faster.
2. papers_research — domain scout, paper reader, library curator.
Pulls arXiv / Semantic Scholar / conference proceedings, keeps
a structured local library under Papers/<topic>/.
3. insight_research — implementation tracker, novelty hunter,
publication drafter. Bidirectional loop that spots
publication-worthy novelty in our own implementations of
external papers.
4. continuous_research — signal harvester, ranker, digest writer.
Standing sweep of RSS + arXiv daily + GitHub trending; produces
a rolling ContinuousResearch/<date>/digest.md.
5. continuous_improvement — brain inspector, improvement proposer,
improvement evaluator. Standing self-audit that files level-up
proposals for the operator to review + measures the outcome.
Each template ships with role system_prompts + brain_seeds authored
in the same voice as the existing backend/frontend/etc templates —
evidence-first, redlines called out, no invention.
Schema + code:
- 0057_team_templates_category.sql — new column with
CHECK (research | development | security | ops). Existing rows
default to 'development'.
- team_templates::UpsertBuiltin + TeamTemplate carry category
(with default_category = 'development' fallback for
Serialize/Deserialize compatibility).
- team_template_loader reads `category = "..."` from the TOML;
absent defaults to 'development' so old templates keep working.
- Wizard step 3 filters:
Research teams panel → templates.filter(t.category==='research')
Development teams panel → templates.filter(t.category==='development')
Operator can no longer accidentally pick backend as their
"research team".
Test fixture updated with category="development".
The templates ship in the server image via the existing
`COPY templates /etc/clawmates/templates` line — no Dockerfile
change needed.
The README was last touched 2026-06-18 and had drifted badly:
- Crate table missed cm-brain, cm-testkit, cm-tools, and the
clawmates-node bin (the herdr daemon)
- Missions system (slices 1-9) was not mentioned
- Herdr phases 0-3 (persistent daemon, dispatch, live pane, INFRA-tier
sessions) were not mentioned
- Prod deploy path on gw-04 was undocumented — the three real gotchas
(non-compose-managed runtime, UID 65532 bind-mount, ZeroClaw provider
env inheritance) are what bit us on 2026-07-09 and 2026-07-12
- CI 1500-LoC hard budget was not called out
- Refine still said "Gemini" — refine switched to Opus 4.8 in 1cbbbbd
- Broker master-key backup mentioned only via cross-link
Splits crates into workspace crates + bins tables, adds a Production
deployment section for gw-04, adds a CI budgets section, promotes the
broker key warning inline, and rewrites Shipped to reflect what actually
landed since 2026-06-18.
Multi-team refactor changed the error text from
'no team_id and no team_template_id' to
'no team_template_id and no config.phase_teams'. Assertion now
just checks both key phrases.
Directly addresses "we want to pick one or more teams to assign to a
mission, first screen research teams, next screen dev teams." A
mission now materializes N teams, each tagged with a phase purpose.
Backend:
- 0056_mission_teams.sql — new join table
mission_teams(mission_id, team_id, purpose). team_id PK because a
team belongs to one mission-purpose. missions.team_id kept as
legacy pointer to the first minted team for single-team surfaces.
- mission_orchestrator::on_launch — reads mission.config.phase_teams
(JSONB shape { research: [tid,...], coding: [tid,...] }), mints
one team per (purpose, template) pair, records each in
mission_teams, binds the first to mission.team_id. Legacy fallback:
if config.phase_teams is absent, uses missions.team_template_id.
Hard error if both are absent.
- GET /api/missions/{id}/teams — returns
[{ team_id, purpose, team_name }], sorted by created_at asc.
Frontend wizard (step 3 rewrite):
- researchTeamIds / devTeamIds — Set<string> multi-selects
- Reusable TeamMultiSelect component (checkbox-style cards)
- Panels rendered conditionally by preset:
hasResearchPhase → "Research teams" panel
hasCodingPhase → "Development teams" panel
neither → "Teams" panel (bench/security-only missions)
- canNext enforces at least one pick in every visible panel
- submit builds config.phase_teams and passes it via CreateMissionRequest
- Review step shows both selections by name
MissionTeamTab:
- Fetches /api/missions/{id}/teams and groups by purpose
- Each purpose renders a section with per-team cards
- Falls back to a single "mission" pseudo-row for legacy missions
that only have missions.team_id (no mission_teams rows)
CreateMissionRequest no longer sends team_template_id from the wizard
— the multi-team config.phase_teams path supersedes it. The backend
still accepts team_template_id for API callers.
Verified: cargo check --workspace + tsc + eslint --quiet all green.
Root-cause fix for the "mission runs with zero agents" bug. Three
enforcement layers now guarantee a launched mission has a team:
1. mission_orchestrator::on_launch — the previous
\`return Ok(None)\` when both team_id and team_template_id are
None is now \`return Err(...)\`. That branch was never a real
"auto-provision later" path; it was a silent no-op that let
the mission flip to running with nothing to run.
2. routes::missions::set_status — the draft→running transition
now (a) rejects with 400 when team_id + team_template_id are
both null, and (b) runs on_launch BEFORE flipping status +
returns 500 on failure. No more orphan "running" missions
with no materialization.
3. MissionWizard step 3 — removed the misleading "LLM
auto-provision" tile (fake code path). First real template is
pre-selected on mount; canNext requires teamTemplateId set;
empty state surfaces a red warning if no templates loaded.
4. MissionCanvas Launch button — disabled with a "No team" label
and explanatory tooltip when the mission has neither team_id
nor team_template_id (defense-in-depth for legacy rows or
direct-API missions).
Also flipped the mission_orchestrator test that expected
Ok(None) → now expects a specific error message.
Prod cleanup: reset the stuck mission
019f814c-d36f-7d60-8915-1ce100683133 (running with team_id=NULL) back
to draft so the operator can delete or attach a template.
Verified: cargo check --workspace + tsc + eslint all green;
mission_orchestrator test updated to match new contract.
Fills the biggest UX gap surfaced during the deploy walk: hosted
missions had no live-progress surface at all. Now they do.
Auto-refresh:
- MissionCanvas grows a second useEffect that polls getMission
every 3s while mission.status === 'running'. Stops immediately
on terminal state (completed / failed / cancelled). Phases,
Tasks, Artifacts, Benchmarks all update without a manual click.
Team tab (new):
- MissionTeamTab.tsx — fetches /api/teams/{id} + /api/team/claws,
shows a card per member with role slot + an "Open" pill that
calls onOpenClaw(clawId) → Dashboard flips to AGENT tier with
that claw selected, dropping the operator into the existing
ClawCommandCenter surface (WorkingOnNow, ReasoningStream, etc).
Live events tab (new):
- MissionLiveEvents.tsx — polls /api/missions/{id}/runs every 5s
for the topology_runs bound to this mission, opens one
EventSource per active run against /api/topology-runs/{id}/events,
renders as a chronological scrolling feed with per-event kind
pills + per-run short-id badges. Auto-scrolls unless the
operator scrolled up. New runs auto-attach; terminal runs
close cleanly.
Backend:
- cm-db::repo::topology_runs::list_by_mission — SELECT ... FROM
topology_runs WHERE mission_id = $1 ORDER BY created_at DESC.
Uses runtime sqlx::query (not the macro) to avoid a sqlx cache
regen just for this route.
- TopologyRunSummary gains #[derive(Serialize)] + rfc3339 codecs.
- GET /api/missions/{id}/runs — workspace-scoped, returns
{ runs: [...] }.
Dashboard wires onOpenClaw on MissionCanvas → setAgentId + setTier("claw").
Verified: cargo check --workspace + tsc --noEmit + eslint --quiet
all green.
Long refined descriptions (Opus tends to emit full section spines)
pushed the tabs + toolbar past the viewport with no way to reach
them. Cap the description block at 38vh with its own overflow-y so
the header stays reachable no matter how long the brief gets.
Claude Opus 4.8 rejects `temperature` — 'deprecated for this model'.
Newer models manage their own sampling; the parameter is only legal
on older Claude generations. Dropping it wholesale rather than
version-gating since we default to Opus 4.8.
Verified: cargo check clean.
Prod's Gemini prepayment credits are depleted (429 on every refine
attempt). Switching to Anthropic Claude Opus 4.8 for the mission
Refine flow — ANTHROPIC_API_KEY is already set in prod for ZeroClaw's
provider config, so no new secret plumbing.
- crates/cm-api/src/mission_refiner.rs:
* DEFAULT_MODEL: gemini-2.5-flash → claude-opus-4-8
* call_gemini → call_anthropic against
https://api.anthropic.com/v1/messages with the standard
x-api-key + anthropic-version headers
* Response parser reads content[type='text'].text (Messages API
block shape) instead of Gemini's candidates path
* Timeout raised 60s → 90s (Opus can be slower than Flash on
long briefs; still bounded so a stuck call fails fast)
- deploy/compose/.env.example: doc block rewritten. Refine now
reuses ANTHROPIC_API_KEY; Level-Up keeps GEMINI_API_KEY because
it needs JSON-mode structured output.
Level-Up is NOT switched in this commit — it uses Gemini's JSON mode
which has no drop-in Anthropic equivalent (needs tool-use rewrite).
Filed as a separate concern; Refine is what was actively broken.
Verified: SQLX_OFFLINE=true cargo check -p cm-api clean;
cargo fmt --all clean.
CI on 6ffbe97 failed on two auto-fixable gates. Both fixed:
* cargo fmt --all — rustfmt applied across the surface touched
by the last ~20 commits (world.rs, security_scan.rs,
routes/{missions,nodes,terminal}.rs, fleet_herdr.rs,
mission_workspace.rs, benchmark_runner.rs, mission_refiner.rs,
lib.rs, tests/mission_orchestrator.rs, cm-db/repo/{missions,teams}.rs,
bins/clawmates-node/src/main.rs)
* eslint apostrophe escapes in HerdrSessions + MissionWizard
* eslint max-lines: extracted EditMissionModal + RefineDiffModal
(each ~200 LoC) into their own files. MissionCanvas drops from
1424 to 1026, comfortably under both the 1250 eslint cap and the
1500 CI budget.
New files:
frontend/src/components/dashboard/EditMissionModal.tsx (211 LoC)
frontend/src/components/dashboard/RefineDiffModal.tsx (208 LoC)
Verified locally: cargo fmt --check clean, cargo check clean,
mission_orchestrator test 3/3 pass, tsc + eslint --quiet both silent.
Previous push hit the gates job's file-size gate — MissionCanvas.tsx
was 1517 lines (17 over). The Live Pane xterm subcomponent is
cleanly separable (no shared state with the parent, just takes
nodeId + visible props), so it lifts into its own file at zero
behavior cost.
frontend/src/components/dashboard/MissionLivePane.tsx (new, 106 LoC)
frontend/src/components/dashboard/MissionCanvas.tsx (1517 → 1424)
Also drops the xterm.css + useResilientTerminal imports from
MissionCanvas since only MissionLivePane needs them now.
Matches the AGENT tier's manage-mode pattern. Three states on the
sidebar toolbar:
- default: [Wrench] [Refresh] [+]
- select mode active: [Wrench] (highlighted) [Refresh] [+] + rows
grow a checkbox on the left and click-toggles selection instead
of opening the mission
- selection made: [Wrench] [Trash · N] [Refresh] [+] where the
trash pill shows the count and fires window.confirm → bulk
DELETE /api/missions/{id} loop
On success: exits select mode, calls onDeleted with the ids, parent
Dashboard clears missionsSel if it was in the batch. Failures stay
selected + a red error line surfaces the count.
Reuses the CRUD backend added earlier (a1c… PATCH/DELETE) — no new
server work.
- mission_orchestrator: replace hardcoded cli=\"claude\" with real
precedence chain: mission.config.cli → template.config.default_cli
→ \"claude\". Missions can now A/B by CLI without a schema change.
- fleet.rs: scope clippy::too_many_arguments allow on
NodeHub::open_pty — 8 params (session address x4 + target address
x3) is the actual dimensionality; a struct would be ceremony.
- security_scan.rs: delete future_artifact_root — pure hint-stub,
no callers, nothing depends on it.
- routes/world.rs: delete the empty normalize() seam + its 30-line
comment block. normalize_run_event higher up does the real work;
the empty stub was pre-cleanup scaffolding.
Post-sweep validation across the whole workspace:
* cargo check --workspace → clean
* cargo test --workspace --no-run → all bins build
* cargo test -p cm-api --test mission_orchestrator → 3/3 pass
* cargo clippy -p cm-api -p clawmates-node --tests → 0 warnings
* tsc --noEmit → 0 errors
Wiring audit: every recent route handler is registered in lib.rs.
Every recent frontend component has at least one importer.
Remaining #[allow(dead_code)] entries are deprovision_claw +
unpublish_claw on RuntimeProvisioner — real rollback paths waiting
on the team-delete route. Documented future hooks, not stubs.
New INFRA category "Herdr sessions" (purple sparkles icon between
Fleet and Local hardware). Shows a card per online fleet node with:
- Node name + hostname + IP
- Per-workspace agent state pills (working / blocked / done /
idle / unknown), colored dots + pane count
- "Open" button → renders that node's full Herdr TUI inline via
xterm.js (same nodeHerdrConnector + WebRTC-with-fallback the
MissionCanvas Live Pane uses)
Backend:
- node daemon: herdr_workspaces + herdr_snapshot ops
(`herdr workspace list`, `herdr api snapshot`)
- fleet_herdr::snapshot helper on top of hub.call_timeout
- GET /api/nodes/{id}/herdr/session route
Fetch flow: /api/nodes filtered to status='online' → for each,
/api/nodes/{id}/herdr/session in parallel. Snapshot errors surface
per-card without failing the whole grid.
The "Open" xterm is separate from the MissionCanvas Live Pane —
this one is scoped to the whole node's Herdr TUI (any workspace),
not a specific mission's pane. Operator toggles between nodes via
the buttons.
Verified: cargo check --workspace + tsc --noEmit both green.
Phase 0 started Herdr via nohup — died on reboot / logout. Replaces
that with proper service management:
- deploy/fleet/herdr-persistence/herdr.service — systemd user unit
(Linux). Restart=on-failure with an 8-in-24h burst cap so a
broken binary doesn't hot-loop. Requires linger enabled so the
user's systemd manager runs without a login session; install.sh
does that via loginctl.
- deploy/fleet/herdr-persistence/dev.herdr.plist — launchd
LaunchAgent (macOS). ProgramArguments + PATH templated so the
install script substitutes actual paths at deploy time.
- deploy/fleet/herdr-persistence/install.sh — idempotent installer
that autodetects OS, drops the unit/plist in the right place,
enables + starts, prints status.
Rolled to tank + architect + morpheus (systemd) + smith + macbook
(launchd). All 5 nodes confirmed `status: running` post-install.
Phase 0 gap closed: Herdr now survives node reboots and logouts,
which is the prerequisite for the fleet_herdr dispatch path to be
reliable across mission_orchestrator restarts.
The killer UX feature: click a mission's Live Pane tab and watch the
actual Herdr TUI on the target node in the browser — cursor, colors,
tool output, all live. WebRTC DataChannel direct where the browser
can reach the node peer-to-peer, WS-relayed fallback otherwise
(same auto-negotiation the INFRA node terminal already uses).
Zero new deployment infra — reuses the existing terminal_ticket +
terminal_ws + PTY-over-control-channel machinery. The one primitive
we grew: PtyTarget::Command variant so the node can spawn an
arbitrary program (\`herdr\`) in the PTY instead of the login shell.
Node daemon (clawmates-node):
- PtyTarget grows a Command { argv } variant
- spawn_command_pty resolves bare names against user + system bin
dirs (matches how tool_update finds claude/kimi)
- PtyTarget::from_frame reads the `command` array from the pty_open
frame; precedence Command > Container > Host
cm-api:
- NodeHub::open_pty grows an optional command argv; when set, the
frame carries it and the daemon spawns the program directly.
- routes::nodes::TermCtrl gains a `command: Vec<String>`; the
fallback branch threads it through.
Frontend:
- core.ts::webrtcConnector takes an optional commandOverride
that ships inside the fallback frame
- nodeHerdrConnector(nodeId) — mints the standard ticket + WS URL
but overrides command to ["herdr"]
- MissionCanvas grows a "pane" tab, visible only when
runtime_kind='local_herdr'. LivePane subcomponent uses xterm.js
(already a workspace dep) via useResilientTerminal, shows a
connecting/relayed/direct pill in the corner.
To watch a mission live: pick "On a fleet node (Herdr)" + target
node in the wizard, launch, click Pane tab → node's Herdr TUI
appears. Navigate to the mission workspace in the Herdr sidebar
(mouse or prefix+w) to zoom into the mission's pane.
Focus-a-specific-pane-directly is a later enhancement — Herdr has
no CLI arg for it yet, so operator navigates the sidebar for now.
Verified: cargo check --workspace + tsc --noEmit both green.
Closes the operator loop for the second-runtime path. Missions
created with runtime='local_herdr' now spawn a Herdr pane on their
target_node automatically on draft→running.
Frontend (MissionWizard):
- Step 4 grows a "Runtime" section above Schedule
- Radio: "Hosted (ZeroClaw)" default | "On a fleet node (Herdr)"
- Local-Herdr shows a dropdown of ONLINE nodes only (from
/api/nodes filtered by status='online')
- canNext blocks Next when local_herdr picked without a node
- Review step shows "Runtime: Herdr on <node-name>" or "Hosted"
- Empty-online-nodes state hints "Connect one from INFRA first"
Backend:
- mission_orchestrator::on_launch grows a NodeHub param; when
mission.runtime_kind='local_herdr' + target_node_id set +
hub present → calls fleet_herdr::dispatch(). Non-fatal:
logs and continues so a research_only mission with a Herdr
runtime chosen accidentally still boots the team.
- routes::missions::set_status passes state.node_hub through.
- Test call sites updated to pass None for the new param
(integration tests don't drive real fleet nodes).
CLI stub: on_launch currently hard-codes cli="claude" for the
Herdr pane. Phase 4 will read that from the team template so a
research team → kimi, gpu team → claude, etc.
Verified: cargo check --workspace + cargo test
-p cm-api --test mission_orchestrator + tsc --noEmit all green.
The second-runtime path uses the existing NodeHub control channel —
NOT SSH. Node daemons already accept typed ops over their outbound
websocket; adding three herdr_* ops keeps everything on the auth
model that already works fleet-wide (control-channel token, no new
SSH key management, no server-container-mounted keys).
Node daemon (clawmates-node):
- New herdr_op handler in main.rs dispatching:
* herdr_dispatch — workspace create + pane split + rename + run
* herdr_status — pane get JSON (agent, agent_status, cwd)
* herdr_read — recent-unwrapped scrollback, N lines
- Herdr binary resolved from ~/.local/bin, brew, /usr/local/bin.
Missing binary returns clean error so cm-api can distinguish
"node not set up for Herdr yet" from "Herdr op failed".
cm-api:
- crates/cm-api/src/fleet_herdr.rs — dispatch / status /
read_transcript / wait_for_completion helpers on top of
hub.call_timeout(). wait_for_completion polls until agent_status
hits 'done' or an idle-after-working state, matching the SKILL
file's "either idle or done is completed" semantic.
- routes::missions::herdr_dispatch — POST /api/missions/{id}/
herdr-dispatch { cli, prompt }. Requires runtime_kind = 'local_herdr'
and target_node_id set. Manual trigger so Phase 1b is exercisable
end-to-end before Phase 1c wires the wizard + orchestrator.
Not yet wired: mission_orchestrator::on_launch still ignores
runtime_kind. Phase 1c adds the wizard picker AND the on_launch
branch that auto-dispatches on draft→running for local_herdr
missions. This commit only adds the primitives.
Verified: SQLX_OFFLINE=true cargo check --workspace green.
Phase 0 (Herdr install on fleet nodes) is the blocker to actually
exercising this end-to-end.
First slice of the second-runtime path. Missions now carry
runtime_kind ('zeroclaw' | 'local_herdr') + target_node_id (FK to
nodes) so the mission_orchestrator + phase executors can dispatch
differently depending on where the operator wants execution.
Migration:
- 0055_missions_runtime_kind.sql — adds runtime_kind (NOT NULL
DEFAULT 'zeroclaw' + CHECK), target_node_id (nullable FK ON
DELETE SET NULL). All existing missions backfill to 'zeroclaw'
so behavior is unchanged.
- topology_runs also grows herdr_workspace_id / herdr_tab_id /
herdr_pane_id text columns so a resumed run can reattach to the
same Herdr pane instead of spawning a duplicate.
Code:
- cm-db::repo::missions — Mission + NewMission carry the two new
fields; all SELECTs updated; INSERT COALESCE-defaults
runtime_kind to 'zeroclaw' when unspecified.
- routes::missions::create — validates runtime_kind and requires
target_node_id when kind='local_herdr' (400 otherwise).
- lib/api/missions.ts — RuntimeKind type; Mission carries both;
CreateMissionRequest optional fields.
Behavior is opt-in: no path exists yet to actually create a
local_herdr mission — that lands in Phase 1c (wizard picker). This
commit just makes the schema + validation in place so Phase 1b's
fleet_herdr dispatch module can key on it.
Tests: mission_orchestrator integration test still green.
Top-right toolbar grows three CRUD controls per your request:
- Plus (always visible) — opens MissionWizard, selects the new
mission on create
- Pencil (draft-only) — opens EditMissionModal for title +
description; PATCHes /api/missions/{id}
- Trash (always visible) — window.confirm then DELETEs; sidebar
selection clears via new onDeleted callback
Backend:
- cm-db::repo::missions::update_meta(id, ws, title?, description?)
— COALESCE-based partial patch
- cm-db::repo::missions::delete(id, ws) — hard delete, cascades
via FKs on phases/tasks/artifacts/benchmark_snapshots
- PATCH /api/missions/{id} (draft-only) + DELETE /api/missions/{id}
Frontend:
- lib/api/missions — updateMission + deleteMission clients
- MissionCanvas — three toolbar buttons, EditMissionModal
(title + textarea for description), local wizard state
- Dashboard — passes onSelect + onDeleted so sidebar reacts to
create + delete without stale selection
Edit is draft-only (backend enforces + button hidden past draft) so
in-flight missions can't have their brief mutated out from under
running agents. Delete is unconditional — operator responsibility to
Cancel first if a run is live.
YAML choked on the leading backtick in the description value (line 2
column 14 — the `cargo-audit` inline-code span). Quoting the whole
string makes it a scalar, not a tag/anchor. Surfaced by prod server
logs after today's compose refresh.
Named docker volume required the separately-managed clawmates-runtime
container to know the volume's on-disk path (varies by docker root).
A predictable host path (/var/lib/clawmates-missions) means the
runtime's spawn command can bind-mount the same path directly.
Closes the follow-up gap flagged when task #23 landed. security_scan
and benchmark_runner now exec against $CLAWMATES_MISSIONS_ROOT/
{mission_id}/repo — this commit is what actually puts a repo there.
- crates/cm-api/src/mission_workspace.rs — new module.
ensure_checkout(pool, workspace_id, mission_id):
* mission with no repo_id → Ok(None), no-op
* repo cloned into $ROOT/{id}/repo (--depth 1)
* dir already a git repo → fetch + reset --hard origin/{branch}
(idempotent — every launch brings the tree in sync with the
remote default_branch)
Auth uses the process's ambient git credential setup (SSH agent /
.netrc / helper). Tokens deliberately not embedded in URLs.
- crates/cm-api/src/mission_orchestrator.rs — on_launch calls
ensure_checkout after team materialization + team_id bind.
Non-fatal: clone failures log and continue so research_only
missions (no repo needed) don't get blocked.
- deploy/compose/docker-compose.yml — new named volume
missions_workspaces mounted at /var/lib/clawmates-missions on
both the server (writer) and where the clawmates-runtime
container will mount it (reader for docker exec). CLAWMATES_
MISSIONS_ROOT + CLAWMATES_RUNTIME_CONTAINER env vars set on
the server so mission_workspace + exec_target read the same
canonical values.
The scan/bench trigger buttons now actually produce findings once
you (a) run a mission whose repo_id is set, (b) have the
clawmates-runtime container bind-mounting missions_workspaces at
/var/lib/clawmates-missions.
Verified: SQLX_OFFLINE=true cargo check -p cm-api +
cargo test -p cm-api --test mission_orchestrator both green.
Missions never populated teams.zeroclaw_container /
teams.zeroclaw_gateway_url — those were research/loops-era columns
for long-lived per-team containers. Every mission-materialized team
runs inside the SHARED runtime as claws-as-agents provisioned via
RuntimeProvisioner. Reading zeroclaw_container on a mission row
always came up NULL, making security_scan + benchmark_runner
silently fail with "mission has no team container yet."
Changes:
- migrations/0054_drop_teams_zeroclaw_columns.sql — DROP both
columns.
- cm-db/src/repo/teams.rs — delete dead helpers
team_container_coords + set_team_container_coords.
- cm-api/src/security_scan.rs — replace team_container_for_mission
with exec_target(pool, mission_id): container from env
CLAWMATES_RUNTIME_CONTAINER (default clawmates-runtime); workdir
from env CLAWMATES_MISSIONS_ROOT + /{mission_id}/repo
(same convention pdf_renderer uses); precondition that mission
must have repo_id bound.
- cm-api/src/benchmark_runner.rs — same shape.
Follow-up (not in this commit): mission_orchestrator + compose stack
still need to wire a per-mission repo checkout under
CLAWMATES_MISSIONS_ROOT before scan/bench actually produce findings.
Columns cleanup here removes the misleading silent-fail; the
missing-checkout gap is now surfaced with a clear error.
Verified: SQLX_OFFLINE=true cargo check --workspace + cargo test
-p cm-api --test mission_orchestrator both green.
Closes task #23.
Real-Postgres end-to-end test locking in the missions arc's
draft→running orchestration contract. Three scenarios:
- on_launch_materializes_team_from_template — a mission with a
team_template_id and no team_id materializes exactly one member
per role, stamps template lineage onto the team row (template_id,
template_version, risk_profile, mcp_bundles), records
agent_template_link per claw, wires team_members, and binds
team_id back onto the mission.
- on_launch_is_idempotent — second invocation returns the same
team_id, no duplicate agents.
- on_launch_no_template_returns_none — mission with no template
and no team leaves the mission untouched (returns Ok(None)).
Uses cm-testkit's per-process Postgres testcontainer, so runs in CI
without any external infrastructure. Brain seeding is expected to
warn-and-continue in the sandbox (read-only FS for the HDF5 mkdir);
the mark_seeded assertion was intentionally dropped — the contract
the test locks in is "link row exists," not "seeding succeeded on
this specific filesystem."
Closes task #24.
1. Dashboard.tsx — mount LevelUpInbox as a bottom panel on the
MISSIONS tier sidebar (below MissionsList, above the tier rail).
Capped at 38% height so it never crowds the mission list; scrolls
independently when the proposal count grows.
2. deploy/compose/.env.example — document GEMINI_API_KEY +
CLAWMATES_REFINER_MODEL + CLAWMATES_LEVEL_UP_MODEL. Refine + Level-
Up silently 500 without the key; the model overrides default to
gemini-2.5-flash so setting the key is the only required step.
Closes the two loose ends I called out last turn (inbox not mounted,
env vars undocumented). The full flow — Refine → mission launch →
security scan / benchmark → level-up review — is now wireable
end-to-end on a fresh deploy just by copying .env.example → .env
and setting POSTGRES_PASSWORD + GEMINI_API_KEY.
Slice 9 cleanup removed active_research_topics + active_loops and the
repo:{topic_id} / loop:{id} project orbs they emitted. The World SSE
loop was left with only agents + active runs — no persistent pin for
"this is what the team is working on right now."
Replaces those with a mission-era equivalent: one mission:{id} orb per
running mission, plus world.touch beams from every assigned team
member. Missions outlive individual runs, so the orb persists even
when no run is claimed — matches the UX intent of the legacy
landmarks without the retired research/loops plumbing.
Query: missions ⋈ team_members where m.status='running', grouped by
mission_id for the orb + fanned out per member for the touches.
Closes task #21.
Omar Sobh