3f77370a6e1df0ab67d995e59099d74f18d0a128
163
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar Sobh
|
ee675b976a | fmt: apply cargo fmt to team template + loader | ||
|
Omar SobhandClaude Opus 4.7
|
9ba5c06a1a |
slice 3: 6 team templates seeded from TOML recipes
Team templates are the canonical rosters + tool bundles that mint
concrete teams for a mission. Every builtin ships as a TOML recipe
under templates/teams/*.toml, loaded into the DB at server boot.
Migration 0048 adds:
- team_templates (id, key, name, stack, default_topology,
risk_profile, mcp_bundles, version, source,
workspace_id)
- template_roles (m2m: template_id + slot; system_prompt,
skills[], brain_seed)
- teams gets template_id + template_version for level-up lineage
Ships 6 builtins:
- rust_sdlc — planner/coder/tester/reviewer/committer for Rust
- backend — api_designer/db_engineer/coder/tester/committer
(Postgres, DuckDB, graph DBs, wire protocols)
- frontend — designer/coder/tester/committer (React + Tailwind + ShadCN)
- mobile — designer/coder/tester/committer (Expo, RN, iOS, Android)
- gpu — arch_analyst/kernel_author/bench_engineer/coder/committer
(CUDA, Metal, ROCm from Rust)
- threejs — scene_designer/coder/shader_author/perf_engineer/
committer (three.js, WebGL, WebGPU)
Each role has a versioned system_prompt + skill list + brain_seed
markdown. Skills column is a name array today; Slice 3.5a promotes it
to a typed m2m join with the real skills catalog.
Server boot:
- team_template_loader::load_builtins reads TOML from
/etc/clawmates/templates/teams (container) or templates/teams (dev),
upserts idempotently. Deterministic uuid per template key (sha256
of a fixed namespace + key) so ids are stable across boots.
- Dockerfile copies templates/ to /etc/clawmates/templates.
Read API:
- GET /api/team-templates — list all
- GET /api/team-templates/{id} — detail with roles
Wizard:
- Step 3 rewired from a raw team_id text field to a template picker
with "LLM auto-provision" as the default option + one card per
builtin, showing stack, topology, risk profile, and description.
- Mission create now passes team_template_id (not team_id) so phase
execution knows which template to mint from.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|
||
|
Omar Sobh
|
a72da9dff0 | fmt: apply cargo fmt to missions | ||
|
Omar SobhandClaude Opus 4.7
|
fbefc67878 |
slice 1: missions data model + migration
Introduce the unified `missions` tier that will replace the current
research_topics + loops split. This slice ships the data model +
backfill + skeleton REST surface; the old wizards keep working in
parallel until Slice 9's big-bang cutover.
Migration 0047 adds:
- missions (top-level workflow: template_kind + team +
schedule + status + config)
- mission_phases (ordered {research|coding|benchmark|
security_scan} phases per mission)
- mission_tasks (typed units of work, e.g. INT-XX cards,
UPSERT-keyed on (phase_id, external_id))
- mission_artifacts (MD/PDF/benchmark/security/diff files with
a pending queue for the PDF renderer worker)
- benchmark_snapshots (before/after pairs per iteration)
Backfill copies existing research_topics + loops rows into the new
tables as one-shot missions with the appropriate template_kind, so
Slice 2's UI can render the full history immediately.
New Rust surface:
- cm_domain: MissionId, MissionPhaseId, MissionTaskId, MissionArtifactId
- cm_db::repo::missions: Mission/MissionPhase/MissionTask/
MissionArtifact structs + insert (txn-wrapped)/get/list/set_status/
phases_for/set_phase_status/upsert_task/tasks_for/register_artifact/
artifacts_for/next_pdf_pending/set_pdf_result
- cm_api::routes::missions: skeleton list/create/get/set_status
routes registered at /api/missions/*
Follow-up slices layer richer behavior (template dispatch, phase
execution, task parsing, artifact rendering) on this foundation.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|
||
|
Omar Sobh
|
68d84bf1ad | fmt: apply cargo fmt to topology_worker.rs | ||
|
Omar SobhandClaude Opus 4.7
|
2a99bba6c1 |
topology worker: fail runs when team container spawn fails, don't silently degrade
Coding loops with team_id set MUST run inside their team container so
agents have /workspace/repo bind-mounted. When try_team_gateway_url
silently returned None on error (mkdir perm denied, spawn_team failed,
etc.), the executor fell through to the shared runtime — which has no
repo mount — and the coding agents narrated for 6 minutes without
touching a file. Runs completed "green" with zero commits, hiding real
infra breakage.
Change try_team_gateway_url from Option<String> to Result<Option<String>, String>:
- Ok(Some) — team spawned/reattached; drive it.
- Ok(None) — no team binding on this run; existing per-topic/per-loop
fallback resolvers still apply.
- Err — team was expected but spawn failed; caller fails the run
with a descriptive error instead of silently degrading.
Each failure path in try_team_gateway_url now formats a specific
diagnostic string (docker connect, spawn_team, missing repo_workspace_path,
etc.) that surfaces into topology_runs.error.
The MCP-bearer mint failure is deliberately kept as a warning + best-
effort: some tools will 401 but the container still boots. Only failures
that would prevent code from being touched escalate to Err.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.7
|
5bbab870fc |
topology/log-sse: resolve team container name for coding-loop runs
The log SSE endpoint was always resolving the container name from a research topic id (research-<topic_id>-team). For paired coding loops that own a team_id, the actual container the topology worker spawns is team-<team_id>-container (spawn_team) — not the research topic's one. That mismatch produced a stream of 404s from Docker: Docker responded with status code 404: No such container: research-<topic_id>-team Mirror topology_worker::try_team_gateway_url's precedence: 1. run's loop has team_id → team-<team_id>-container 2. run has research_topic_id → research-<topic_id>-team 3. loop.source_research_topic_id (legacy paired flow) → same as 2 4. else → clear error, no more 404 spam Co-Authored-By: Claude Opus 4.7 <[email protected]> |
||
|
Omar SobhandClaude Opus 4.7
|
3444859e11 |
wizard/repo/ensure: degrade to skipped-fanout on clawstor errors
The wizard was hard-failing (500) when clawstor was unreachable — env unset, network error, non-JSON HTML response from a fallback proxy, or non-2xx status all mapped to ApiError::Internal, which blocked the wizard from advancing. Clawstor fan-out is a warmup optimization, not a prerequisite. Real fleet materialization happens later at spawn time. When the aggregator is absent, return a skipped FanoutReply (all_ok=true, empty peers) so the wizard proceeds, and log the reason server-side. Unblocks: picking clawhdf5 in the ResearchWizard when the tank/ architect clawstor daemons are running but the HTTP aggregator (`claw-store serve`) isn't deployed yet. Co-Authored-By: Claude Opus 4.7 <[email protected]> |
||
|
Omar Sobh
|
65b19b0fc5 | fmt: trim trailing blank line in topology_worker.rs | ||
|
Omar SobhandClaude Opus 4.7
|
657b666219 |
mcp door: extend service-session bearer to per-topic + per-loop spawns
Per-team runtimes had their MCP bearer swap wired in the prior slice, but per-topic (research_container::spawn) and per-loop (spawn_loop) containers still baked the stale template bearer and 401'd every tools/list. Same fix, extended: mint a workspace-owner service session via runtime_provision::mint_workspace_service_token and inject via prewrite_daemon_config_with_risk. Move mint_workspace_service_token from topology_worker into runtime_provision so all three spawn call sites share the helper. Callers updated: routes/research.rs (start_topic), routes/research_setup.rs (prepare_topic_runtime), routes/loops.rs (ensure_loop_container). Co-Authored-By: Claude Opus 4.7 <[email protected]> |
||
|
Omar Sobh
|
6504ed6062 | fmt: single-line if condition in mcp bearer rewriter | ||
|
Omar SobhandClaude Opus 4.7
|
fd3ebb628a |
clippy: replace is_some+unwrap with if let in mcp bearer rewriter
Co-Authored-By: Claude Opus 4.7 <[email protected]> |
||
|
Omar SobhandClaude Opus 4.7
|
cbb709e31d |
fix: avoid let-chain (needs edition 2024) in mcp bearer rewriter
CI's fmt check runs on stable toolchain that rejects let chains in if conditions. Nest the `if let` inside the block instead. Co-Authored-By: Claude Opus 4.7 <[email protected]> |
||
|
Omar SobhandClaude Opus 4.7
|
49f94a5360 |
mcp door: mint workspace-owner service session for team runtime bearer
The runtime template's static clawmates_door bearer is rejected by cm_auth::authenticate() (needs an auth_sessions row). Every per-team agent was getting `unauthorized: missing or invalid bearer token` and `0 tool(s) registered from 0 server(s)`. Add AuthService::mint_service_session + users::owner_of_workspace and mint a 30d service session in try_team_gateway_url; inject it into the freshly-spawned team container's config.toml [[mcp.servers]] clawmates Authorization header via prewrite_daemon_config_with_risk (bearer arg). Follow-up: apply the same pattern to research::spawn (per-topic) and per-loop spawn paths. Co-Authored-By: Claude Opus 4.7 <[email protected]> |
||
|
Omar Sobh
|
8cd0c7bd01 |
runtime_provision: migrate provider_alias_for to v0.8.3 families
Auto-provision (and every other build_team caller) hit 'dangling_reference: claude_cli is not a known provider family' because provider_alias_for still returned v0.7.x aliases — claude_cli.default / claude_cli.glm / claude_cli.glm5 / kimi_cli.default — all deleted upstream when providers.models schema was restructured. Now the mapping resolves to real configured aliases: - 'claude'* prefix → anthropic.default - 'gemini'* prefix → gemini.default - 'llama'* / 'groq' → groq.default - glm* + kimi* → anthropic.default (fallback until glm.default / moonshot.default provider tables land in the runtime template) - unknown → anthropic.default Covers the model picker's full roster (claude-sonnet-5 / opus-4-8 / haiku-4-5 / sonnet-4-6 / glm-4.6 / glm-5.2 / kimi-k2 / gemini-2.0-flash / llama-3.3-70b-versatile) plus every legacy shorthand. Tests updated to assert the new mappings including the model picker's ids. |
||
|
Omar Sobh
|
6e37802f73 |
claws: PATCH /api/claws/{id}/model — swap the runtime-bound model
Auto-provisioned agents (via the research + loops team wizard) show
up on the Agents page (same agents::insert → agents::roster path),
and PATCH /api/claws/{id} already handled name/job_title/system_prompt/
avatar/accent/wallpaper. The one thing that was NOT reachable from the
Agents page: swapping the model.
Add a dedicated endpoint:
- PATCH /api/claws/{id}/model { model: string }
- Persists via agents::set_model_binding (DB)
- Best-effort runtime rebind via RuntimeProvisioner::provision_claw
(idempotent — overwrites agents.<alias>.model_provider on the
shared ZeroClaw config)
- Audit-logged as 'agent.model_changed' with the new model in payload
Now an operator can open the Agents page, click a claw that was
auto-provisioned by the team wizard, and swap its model
(claude-sonnet-5 → glm-5.2 → whatever) without recreating the team.
Same DB row, same claw_id, same brain — just a new provider on the
next turn.
Frontend affordance not shipped in this commit — the endpoint is
usable via curl/psql/scripts today; a UI 'Model' picker on the claw
detail card can land in the next Agents-page pass.
|
||
|
Omar Sobh
|
ac8c689f50 |
logs + team parity: pretty step/container renderers; quota + audit on team creation
Two bundled changes:
── LiveRunLogs prettification ──────────────────────────────────
The Steps + Container tabs were plain mono lines with a single
color per event. Now they get structured layout:
Steps:
- Color-hashed actor pill (stable palette so [Distiller] and
[Novelty Analyst] each get their own hue across the session).
- Phase pill (plan=cyan, work=green, synth=amber, aggregate=purple).
- Token count pill formatted 1.2k / 14.3k / etc.
- Gated-action warning pill in amber when > 0.
- Left-border color strip keyed to the actor for at-a-glance
visual grouping.
- Long outputs collapse to their first 300 chars with a '+ N more'
toggle to expand the full text.
- 'done' events get a green (or red for error) border strip +
pill instead of blending into the stream.
Container:
- Splits '[actor] action (outcome) · msg' into colored spans —
actor pill (deterministic color), action in dim, outcome pill
green/red/dim by state.
- Non-line events (info/error/done) get their own left-border
strip so bash echoes and stack traces don't drown in the daemon
chatter.
- Timestamps switch to HH:MM:SS.mmm — dense but scannable.
Small palette (LOG constants) keeps the color budget bounded — no
new UI vocabulary, just cleaner reads of what was already there.
── Team-wizard governance parity ───────────────────────────────
build_team_with_lifecycle now matches POST /api/claws' governance:
- enforce_new_agent quota check per member (previously bypassed
workspace agent quotas entirely for team/auto-provision paths).
- audit::append('agent.created', ..., {source: 'team_wizard'}) per
member so team-created claws appear in the same audit trail as
individually-created ones. Adding a 'source' key distinguishes
provenance without changing consumers.
.brain (h5) handling was already consistent between the two paths —
both use the lazy on-first-access load_brain hook seeded from
agents.system_prompt. No change there.
|
||
|
Omar Sobh
|
956be2cf4f |
wizard: in-place auto-provision team from topic (LLM-derived, sonnet-5)
Step 5 of ResearchWizard was 'assign agents from workspace roster'.
When the roster was empty, the wizard body was hard-swapped for
NoAgentsGate — you couldn't reach step 5 at all.
Now step 5 shows a 'Team' panel:
- Big cyan card: 'Auto-provision team from this topic'. One click
runs an LLM plan pass, gets 3-5 role slots + system prompts back,
materializes claws via the existing build_team pipeline, stamps
runtime posture, returns a shape that drops straight into the
submit body's agents[]. Card flips green with the derived roster.
- Below that: the classic roster picker, but only when the workspace
actually has ≥1 claw AND auto-provision hasn't landed. Otherwise
hidden — no dead empty-state affordance.
Every gate that required agents.length > 0 to render the wizard body
or the footer is gone. canNext gains a step-5 clause: allow Next when
EITHER auto-team is ready OR the user handpicked from a non-empty
roster.
Backend
- POST /api/teams/auto-provision — accepts {title, description,
outcome_kind, topology_kind?, model?, risk_profile?, mcp_bundles?}.
Derives topology from outcome_kind (integrations → pipeline; else
hub_spoke). LLM plan pass yields a JSON roster of 3-5 roles
(role_slot, name, system_prompt). Materializes team + claws via
build_team, stamps risk_profile (default research_web_readonly) +
mcp_bundles (default [clawmates_door, gitea_forge]). Response
carries team_id + agents[] in the shape /api/research already
expects.
- Every provisioned claw runs on claude-sonnet-5 by default;
overridable via the model field.
Follow-ups (not in this slice):
- Same picker in LoopsWizard (slice C — parallel change, same API).
- Post-create 'Team' section on ResearchCanvas / LoopsCanvas so
users can rebind after the fact (slice D).
- Full Teams tier UI + Agents-page deprecation (slice E).
|
||
|
Omar Sobh
|
cde196dbdc |
runtime: bake tea + gitea-mcp binaries + let GITEA_TOKEN pass through
Prep for the SDLC coding team. Adds two upstream Gitea binaries to the clawmates-runtime image so team-scoped agents can drive git.redclaw.dev without custom MCP code: - tea v0.14.2 (shell CLI: clone/push/pr checkout, git remote helper) - gitea-mcp v1.3.0 (native MCP server: list_repo_pull_requests, create_pull_request, create_file, update_file, create_branch, create_issue, get_file_content, etc.) Both installed by arch (amd64 / arm64) at image build time; also adds git to the runtime layer since it was missing (needed by tea's shell delegates). research_container::inherited_env now propagates GITEA_ prefixed vars so a team container inherits GITEA_TOKEN (+ optional GITEA_HOST) from the server env. Team-scoped daemons can then start gitea-mcp via stdio with --token-env GITEA_TOKEN. Followup on gw-04: 1. rebuild clawmates-runtime image against the new Dockerfile 2. add [mcp_bundles.gitea_forge] to the runtime template 3. set GITEA_TOKEN in the server compose env |
||
|
Omar Sobh
|
0d0bb5ffaa |
teams: runtime spawn hookup for per-team containers (slice 3b)
Wires the per-loop-team arc end-to-end. When a loop with team_id fires an iteration, the worker now spawns/reattaches a dedicated team container, mounts the paired research topic's repo at /workspace/repo (rw), and stamps the team's risk_profile into every [agents.*] binding on the freshly-written config.toml. Legacy loops with team_id = NULL keep taking the per-topic / per-loop path unchanged. research_container.rs - team_container_name_for(team_id) = 'team-<uuid>-container' - team_state_root(team_id) — /var/lib/clawmates-team-state/<uuid>/state (overridable via CLAWMATES_TEAM_STATE_ROOT) - prewrite_daemon_config_with_risk: line-based sed that swaps only the risk_profile line inside each [agents.<name>] block. Avoids the regex-eats-array-literal trap that bricked the shared runtime config on the earlier out-of-band edit. - spawn_team: full-shape idempotent spawner. Same mount + env + label pattern as spawn/spawn_loop; additionally supports Claude settings bind-mount + external-bridge attach. topology_worker.rs - try_team_gateway_url resolver runs BEFORE the existing per-topic and per-loop lookups. Cold path: reads team runtime config + paired research topic repo path, spawns the container, persists coords back to teams.zeroclaw_container/zeroclaw_gateway_url. Any failure logs + returns None so the caller falls through to the legacy shared-container path — team spawn can never brick a run that could otherwise complete. Not shipped in this slice: - Wizard 'existing team' picker (currently just fresh vs reuse) - Teams tier UI to list/edit/delete teams - Auto-teardown for stale team containers (piggyback on existing reaper is a follow-up) |
||
|
Omar Sobh
|
0b7f247b0e |
wizard: 'fresh coding team' picker for paired coding loop
Second slice of the per-loop-team arc. The paired-coding-loop checkbox
in ResearchWizard step 6 now exposes a two-option picker:
⦿ Provision a dedicated coding team (default when the loop is on)
— fresh 'Coding · <topic>' team row, risk_profile =
coding_readwrite, clawmates_door in mcp_bundles. Loop's
team_id is bound at wizard-submit time.
○ Reuse the research team (legacy) — no team_id bound; coding
iterations spawn against the research topic's container.
Frontend
- New codingTeamMode state, radio picker rendered under the checkbox.
- research.ts createTopic body gains paired_coding_team_mode?: 'fresh'|
'reuse'.
Backend
- CreateTopicRequest gains paired_coding_team_mode: Option<String>.
- materialize_topic_loops takes it through and, when 'fresh', calls
the new provision_fresh_coding_team helper — inserts a teams row
via the existing insert_team_with_lifecycle (pipeline kind, same
graph as the loop), sets its runtime-config via
set_team_runtime_config, then binds loop.team_id.
- All operations best-effort with stderr logging — a team-provision
failure leaves the loop functional under the legacy fallback.
Not shipped in this slice (deferred to runtime hookup slice):
- research_container::spawn keyed on team_id → per-team container
- Config template rewrite injecting the team's risk_profile
- Migration of existing paired loops onto their own teams
The plumbing lands now so the wizard's intent is recorded; the
runtime honors it in the next PR.
|
||
|
Omar Sobh
|
6066e93889 |
teams: per-team runtime posture (risk_profile + mcp_bundles) + FK from loops/topics
Foundation slice for letting a coding loop bring its own team instead
of reusing the paired research topic's team. Turns out the teams
table already exists (0010_teams.sql) with full CRUD — this scales
back to the minimal missing bits:
Schema (0045_teams.sql)
- ALTER TABLE teams ADD risk_profile TEXT (NULL = template default)
- ALTER TABLE teams ADD mcp_bundles JSONB DEFAULT '[]'
- ALTER TABLE loops ADD team_id UUID REFERENCES teams ON DELETE SET NULL
- ALTER TABLE research_topics ADD team_id UUID REFERENCES teams
- Two partial indexes (team_id NOT NULL) for the future cascade queries
cm-db (dynamic sqlx::query so the existing get_team's compile-time
cache doesn't need regenerating):
- TeamRuntimeConfig struct
- get_team_runtime_config / set_team_runtime_config
- team_for_loop / team_for_research_topic (resolvers)
- set_team_for_loop / set_team_for_research_topic (binders)
cm-api
- GET /api/teams/{id} now surfaces risk_profile + mcp_bundles
- PATCH /api/teams/{id}/runtime-config sets them
Not touched (comes in follow-up slices):
- Wizard picker exposing 'reuse research team' vs 'fresh coding team'
- Runtime container spawn keyed on team_id
- Migration of existing paired coding loops onto their own team
|
||
|
Omar Sobh
|
2ba40b03b7 |
loops: per-iteration live logs (Steps + Container tabs), collapse graph JSON
The Loops canvas' IterationsTimeline was a flat status-pill list —
no way to see WHAT an iteration was actually doing. Meanwhile the
Research canvas had full LiveRunLogs with Steps + Container tabs
against the same underlying topology_run SSE endpoints. This
factors LiveRunLogs so both canvases share it.
Frontend
- LiveRunLogs gains a directRunId?: string prop. In this mode it
skips the topic-scoped active-runs + pipeline-state polls and
pins activeRun to the given id. topicId stays optional (topic
mode unchanged from ResearchCanvas' perspective).
- LoopsCanvas IterationsTimeline: each row is now a click-to-
expand card. On expand, renders <LiveRunLogs directRunId={...}/>
right below the header — Steps + Container tabs, full SSE tail,
same 320px terminal.
- Auto-opens the newest running/queued iteration so a click on
'Run now' immediately exposes the live pane.
- New CollapsibleSection helper wraps the graph JSON block so it
starts closed. Reference material stays one click away without
cluttering the canvas.
Backend
- run_container_log_sse now resolves the tail target via
loop.source_research_topic_id when the run itself has no
research_topic_id. Paired coding loops (kind=exec with a
source_research_topic_id) reuse the paired topic's team
container, so we tail its docker logs. Pure loop runs still
error with a clearer message.
|
||
|
Omar Sobh
|
e3ef3fd056 |
research: fix Draft 'Invalid Date' + stale-error leak in pipeline card
Two cosmetic bugs surfaced by the successful v0.8.3 pipeline run: 1. **'produced Invalid Date'** — research_outcomes.created_at was an OffsetDateTime serialized by time's default array format (`[y, ordinal, hh, mm, ss, ns, tz]`), which browser's `new Date(...)` can't parse. Add `#[serde(with = "time::serde::rfc3339")]` matching the pattern already in threads.rs / routine_runs.rs. 2. **Stale error text on pipeline card** — the runs stage's `latest_error` walked every run by `created_at DESC` and returned the first non-empty error, so a topic with an earlier failed run + a later completed run kept displaying the old error next to '1 completed'. Now the error only surfaces when the MOST RECENT run itself failed. Historical failures stay in the run count but don't leak their message. |
||
|
Omar Sobh
|
ae113dd319 |
live-run-logs: add Container tab that tails filtered daemon logs
Steps summaries only fire AFTER each topology step completes — so a
stalled first turn was completely dark. Add a second tab that
streams the team runtime container's daemon log live via a new SSE
endpoint.
Backend
- GET /api/topology-runs/:id/container-log — workspace-scoped SSE
around bollard's docker.logs(follow=true, tail=200). Buffers on
newline so partial mux chunks don't truncate a log line.
- compact_container_log: parse a zeroclaw daemon line
('[actor] ... zc_action=X zc_outcome=Y ... msg') into
'[actor] action (outcome) · msg'. Framing-only continuations are
dropped; non-zc lines (bash echoes, backtraces) pass through as-is
so nothing interesting is lost. ANSI escapes stripped.
- Everything funnels through one async_stream! so early exits
(workspace check / docker connect / no bound topic) yield an
'error' event and return without breaking Sse::new's single stream
type.
Frontend
- LiveRunLogs gets a sub-tabs strip: Steps · Container.
- New useContainerLog(runId, active) hook — gated by tab so we don't
hold two open SSE streams when the operator isn't looking.
- Same terminal widget renders each container line with a level
color (info/done grey, line default, error red). Sub-tab pill
shows count + status live.
|
||
|
Omar Sobh
|
36a9fbe81f |
research/start: allow rerun on loop-owned topics too
D1-fold loop guard 409'd even the rerun path if the topic was scheduled — a failed iteration couldn't be restarted until the loop's next scheduled fire. Now the loop guard only fires for the fresh 'standby' start; the rerun path (status=='processing' + orphan cancel) works whether or not a loop owns the topic. Loop binding is preserved either way. |
||
|
Omar Sobh
|
6af9d509b0 |
research: rerun cancels orphan runs first so 409 doesn't block restart
The previous rerun guard (standby OR (processing AND in_flight==0)) still 409'd when a lingering queued/running row existed — typically an orphan from a server restart mid-pipeline, before the reaper or stale-checkpoint requeuer had marked it failed. Now: allow rerun on any 'processing' topic; before enqueuing the fresh run, batch-cancel every in-flight run for the topic so the new run isn't racing them. Terminal states (reviewing / publishing / published) still 409 as before. |
||
|
Omar Sobh
|
671d3ac4f0 | clippy: fix doc_lazy_continuation on TopicListItem.runs_failed | ||
|
Omar Sobh
|
f70f6c679e |
research: errored-state card + one-click rerun (no wizard re-entry)
When a topic ends up parked in 'processing' with all runs failed and nothing in flight, the sidebar card was still spinning as if progress were happening. Now: Backend - topology_runs::run_counts_by_research_topic — batch query that returns (in_flight, failed-since-last-success) per topic. Used by the list endpoint; dynamic sqlx::query() so no prepare needed. - TopicListItem DTO gains runs_in_flight + runs_failed. - start_topic status guard relaxed: allow (standby) OR (processing AND runs_in_flight == 0). Blocks accidental double-fires on a live pipeline; permits rerun on a failed one. Same request body, same behavior once accepted, so the frontend just POSTs /research/:id/start on the RotateCw click. Frontend - ResearchList detects errored: status===processing && !in_flight && failed>0. Swaps the MiniSpinner for a red AlertTriangle and changes the status text to 'error · N failed'. - New RotateCw icon button next to the delete Trash — same button cluster, one click, no wizard re-entry required. Disables while a request is in flight; error surfaces in the sidebar's shared error banner. |
||
|
Omar Sobh
|
067bebd071 |
wedged-run fix: pre-approve claude perms + verbose step log + reaper
Root cause: claude CLI in the per-topic research container runs as
uid=0(root). ZeroClaw's claude_cli provider passes
--dangerously-skip-permissions which Claude CLI rejects under root
for security — so the CLI hangs waiting for interactive permission
approval that never arrives, hitting the 600s provider timeout with
zero step records journaled.
Three-part fix:
1. Claude settings bind-mount (research_container.rs):
Optional CLAWMATES_CLAUDE_SETTINGS_PATH env — when set, mount the
host file at /root/.claude/settings.json (read-only) in every
spawned team container. deploy/claude-settings.json ships the
canonical config (permissions.defaultMode = bypassPermissions +
hasCompletedOnboarding). CLI accepts requests immediately with no
--dangerously-skip-permissions flag needed.
2. Verbose per-step log line (topology_worker.rs):
Every checkpoint now writes to stderr:
topology_worker::step run_id=X step=N node=Y role=Z phase=W
output_bytes=B tokens=T gated=G
Visible in docker logs clawmates_server_1 — gives us live
'topology is flowing' signal without opening the canvas, and
makes it obvious when a topology_kind is skipping stages it
shouldn't.
3. Stuck-container reaper (topology_worker.rs):
New 60s-tick loop reap_stuck_runs: for any research topology_run
older than 15 min with zero checkpoint.records, docker-stop its
container and mark the run failed with a diagnostic error. Only
reaps research-bound runs (non-research runs don't own a
container). The existing 180s stale-checkpoint requeuer stays
in place for other failure modes.
Deploy: gw-04 needs
ln -sf /path/to/repo/deploy/claude-settings.json /opt/clawmates/claude-settings.json
CLAWMATES_CLAUDE_SETTINGS_PATH=/opt/clawmates/claude-settings.json
in the server env, plus the timeout lowered from 600 -> 120 in
compose. Both handled in the deploy step outside this commit.
|
||
 osobh
|
1d69866bcf | research canvas: collapsible sidebar + live topology-run logs (#6) | ||
|
Omar Sobh
|
13ead5cb9f | fmt: sort routes/mod.rs pub mod entries | ||
|
osobh
|
149ad992bb | wizard: materialize picked repo across clawstor fleet at step-2-next (#5) | ||
|
osobh
|
1cb643142c | research/publish: gate approve+reject on Owner role (#4) | ||
|
Omar Sobh
|
dd791061ee |
test(topology_jobs): seed research_outcome so transition test matches new invariant
Commit
|
||
|
Omar Sobh
|
212e68f6b1 |
research/probe: end-to-end smoke test endpoint (skip topics/loops/spawn)
New: POST /api/research/probe — one-shot pipeline probe against the
workspace's shared ZeroClaw gateway. Drives a trivial 'reply OK'
turn via ZeroClawDriveExecutor::drive and reports per-step timings.
Purpose: stop guessing what's broken in the wizard flow by exercising
JUST the executor→daemon→claude→response path. If probe succeeds
within a few seconds, we know:
- ZEROCLAW_TOKEN + gateway URL config is correct
- Daemon can reach and authenticate against claude
- The full ws round-trip works
…and every other failure we've been chasing (turn timed out, LLM
request failed, ws connect DNS error, etc.) is spawn-config or
prompt-size specific.
Request body (both optional):
{ "prompt": "Reply OK", "agent": "coordinator" }
Response:
{
"verdict": "ok" | "fail",
"total_duration_ms": …,
"prompt_len": …,
"response_preview": "OK",
"steps": [
{ name: "build_executor", duration_ms: …, status: "ok" },
{ name: "drive_turn", duration_ms: …, status: "ok", detail: "tokens=N, output_len=M" }
]
}
Wire-up:
- new routes/probe.rs
- pub mod probe in routes/mod.rs
- POST /api/research/probe registered in lib.rs router
- ZeroClawDriveExecutor::drive promoted from private to pub so the
probe handler can call it (behavior unchanged, other callers were
all inside the same struct).
Usage from anywhere (curl, browser dev tools, etc.):
curl -X POST https://clawmates.work/api/research/probe \
-H "Content-Type: application/json" \
-H "Cookie: <session cookie>" \
-d '{}'
Follow-up: a small frontend button (e.g. bottom of ResearchList) that
POSTs this and renders the response inline, so users don't need to
curl. Skipping in this commit to ship the useful part first.
|
||
|
Omar Sobh
|
3373f57da0 |
topology_exec: bump TURN_TIMEOUT to 700s to outlast daemon claude_cli
Verbose daemon logs revealed the real turn failure: the daemon's own claude_cli provider was timing out at 180s while running a 10-15k token coordinator prompt. The daemon reads its per-provider timeout from ZEROCLAW_providers__models__claude_cli__default__timeout_ms — we set that to 600000 (10 min) on gw-04 via a compose patch. Now the executor's TURN_TIMEOUT (was 300s) must exceed the daemon's own limit, or we kill the ws before the daemon can reply. New sequence: daemon has up to 600s to invoke claude and return a response; executor waits up to 700s (100s headroom) for the ws event stream to drain. If claude takes 500s, both survive. If the daemon really does hang past 600s, its own timeout fires first and we get a proper "provider timed out" error instead of a phantom executor timeout. Compose env applied on gw-04 in the same session: - ZEROCLAW_providers__models__claude_cli__default__timeout_ms=600000 - ZEROCLAW_providers__models__claude_cli__door__timeout_ms=600000 (backup: /opt/clawmates/docker-compose.yml.bak-timeout) Server container recreated to pick them up; env verified. Follow-up: the coordinator prompt is legitimately huge (autonomy contract + roster + description + repo tree + integration-plan template + operator notes = 10-15k tokens). We should consider either shrinking it or breaking the work into multiple smaller turns so the daemon isn't gambling on a single call taking 3-8 minutes. |
||
|
Omar Sobh
|
7e0620fd08 |
research: don't advance to reviewing without an outcome + guard-before-write
Two bugs that combine to produce the 409 you get when clicking Approve:
1) notify_run_completed (topology_worker post-hook) was advancing
topic status processing → reviewing whenever the last sibling run
terminated — success OR fail. A failed run with 0 outcomes still
pushed the topic to `reviewing`, the canvas rendered the "Request
publish" affordance, and the reviewer clicked Approve on nothing.
Fixed by adding an EXISTS(research_outcomes …) clause to the
UPDATE. Topic stays in `processing` when no outcome exists; the
loop's next iteration still has a chance to produce one.
2) decide_publish was calling
research_publish_approvals::decide(approve=true)
FIRST (which flips the row to `status='approved'`) and then
running the "no outcome? 409" guard SECOND. On the 409 return,
the DB was left half-flipped: approval says approved, topic still
in reviewing, no outcome exists, and every future click to the
same approval returns 409 on the "already decided" guard —
leaving reviewers with no way forward.
Fixed by moving the outcome-existence check BEFORE the decide()
call. On 409 now nothing was written, so the reviewer can try
again cleanly once an outcome is produced.
Also unstuck the current stuck row out-of-band (SQL UPDATE to reset
the approval to pending + topic to processing) so the user isn't
forced to delete the topic to escape the 409 loop.
sqlx dynamic query — the new EXISTS clause wasn't in the offline
cache so I switched notify_run_completed to plain `sqlx::query`.
|
||
|
Omar Sobh
|
ee05037095 |
research-pipeline-diag: state-aware statuses (waiting != failing)
The diagnostic was calling any 0-outcome + 0-completed-runs state a
FAIL — red dot + 'No outcome produced yet — check the runs stage for
the failure reason'. That fires the moment a wizard-materialized
topic lands, before its very first turn even completes, and stays
red for the whole 2-3 min a legitimate coordinator turn runs. Result:
users see 'FAIL' on every fresh topic and can't tell a real failure
from a normal in-flight state.
Backend fix — introduce a `waiting` status (blue/pulsing in UI):
Runs stage:
0 runs -> skip ('No runs yet — pipeline hasn't fired')
any running/queued -> waiting ('N in flight, M completed')
all failed -> fail (with error text)
some failed -> warn
all completed no fails -> ok
Outcomes stage:
outcome_count > 0 -> ok
0 outcomes + 0 runs -> skip ('No outcome yet (pipeline hasn't fired)')
0 outcomes + any running -> waiting ('Waiting for the current run to finish…')
0 outcomes + any failed -> fail (the actual silent-bug case)
0 outcomes + all done ok -> warn (weird — completed but wrote nothing)
Also suppresses the run stage's `latest_error` detail when the run
status is `waiting` or `skip` — reporting a stale error next to an
actively-running job is what made users think the current run had
failed.
Frontend:
- PipelineStage['status'] union grows a 'waiting' arm.
- Pill color: cyan (#5ec8d8) with a pulsing scale/opacity animation
(new cm-pulse keyframe in motion.css).
- Strip summary line: 'Pipeline in flight — waiting for run to finish…'
when there's any waiting stage and no failures.
- Border tint: cyan border when waiting, coral when failing, neutral
otherwise.
Zero backend semantic changes to the outcome-write path — this is
purely UI truth-telling.
|
||
|
Omar Sobh
|
6e06bcf136 |
research_container: spawn team daemons with --verbose
Without --verbose the daemon only emits boot + fatal-crash lines to stderr, so 'LLM request failed' 500s (webhook, /ws/chat) surface as opaque errors with no way to diagnose from `docker logs`. --verbose turns on per-request traces via the runtime's structured logger. Immediately paid off: recreating the current stuck team container with --verbose revealed 'agents.actor.model_provider resolves to a model_provider entry with no `model` set' which points at a missing config override (the shared runtime injects model = ... via ZEROCLAW_providers__models__groq__default__model on its own env; per-team containers don't get that env because it lives on the shared runtime container, not on clawmates_server_1). That's the next fix. Same --verbose is added to spawn_loop's per-loop cmd for symmetry. |
||
|
Omar Sobh
|
27b74e13d4 |
research_container: propagate ZAI_ + KIMI_ env into per-team runtime
Shared runtime's config points several providers at Z.AI's Anthropic proxy (`ANTHROPIC_AUTH_TOKEN = \"\$ZAI_API_KEY\"`) and the Kimi provider block needs KIMI_API_KEY. inherited_env only propagated ZEROCLAW_/OPENAI_/ANTHROPIC_/GEMINI_/GROQ_ prefixes — ZAI_ and KIMI_ were silently dropped, so per-team daemons booted with empty substitutions and every Z.AI-routed call died with 'LLM request failed' (500 from the daemon, executor timeout at 300s). Adds both to the prefix allowlist. Server container already has ZAI_API_KEY set from the compose env_file; KIMI_ will flow through too when we add it. Doesn't fully close today's incident — the current stuck run's webhook still 500s even with ZAI_API_KEY injected, so there's a second daemon-internal issue (need to boot the team daemon with --verbose to surface it). But the propagation fix is a real bug on its own and would silently break the moment the workspace's configured role calls into a ZAI-routed provider. |
||
|
Omar Sobh
|
bfcdca0583 |
research-canvas: managed-by-loop UI + start_topic guard (fold cleanup)
Closes the UX gap the fold introduced: the topic canvas was still showing "Start research" for standby-state topics even when a scheduled loop already owned the runs. Clicking it would 409 (or worse: race the loop into a duplicate run). Topic status stayed at standby forever because the loop path bypassed start_topic's set_status transition. Four changes: 1. **Backend status transition** — compose_and_enqueue_iteration for kind='research' now calls set_status_if(standby, processing) on the topic before the run is enqueued. New DB helper set_status_if only advances when the current status matches the "from" arg — safe against races and re-invocations. Later iterations no-op since the topic is already past standby. 2. **has_managed_loop on TopicDetail** — get_topic hydrates a new ManagedLoop struct (loop_id, title, enabled, next_fire_at, last_run_id, schedule_summary) when a kind='research' loop is bound to the topic. summarize_schedule() derives a human string from the loop's triggers jsonb (e.g. "cron: 0 3 * * * · on new artifact", "one-shot", "manual"). New DB helper loops::research_loop_for_topic returns the row. 3. **Canvas branch** — nextAction takes a managedByLoop flag; when set + status=standby, returns null (no button). The canvas renders a "MANAGED BY LOOP" strip below the topic title showing loop name, schedule summary, next fire time, and enabled dot. Reviewer buttons (Request publish / Approve / Reject) still show normally in later states — reviewers should still promote outcomes even when a loop is producing them. 4. **start_topic guard** — refuses with 409 when a research loop already owns the topic. Closes the direct-POST hole for anyone bypassing the frontend. TS type + summarize_schedule live in the same commit so an old client hitting a new backend just ignores the extra field (no breakage), and a new client hitting an old backend renders the classic buttons (managed_by_loop is optional). |
||
|
Omar Sobh
|
f910771bbb |
research-prompt: add AUTONOMY CONTRACT + relax citation rigidity
Inspection of a stuck 500s+ research run showed both agents were
correctly picking up the topic AND producing rich, structured plans —
then stalling at 'Should I proceed?' and 'Which approach?'. No human
to answer = infinite spin until stale-sweep + retry, forever.
Two prompt changes to compose_research_iteration_task:
1. AUTONOMY CONTRACT at the top:
- Explicit "no human will answer you"
- Explicit "do NOT ask for confirmation"
- Explicit "you MUST emit the completed artifact"
- Framed as a contract, at the very top, before the task itself.
2. Softer citation rule:
- Was: 'Cite every claim; never fabricate sources or repo paths.'
- Now: 'Cite what you can verify. Use [claim needs verification]
inline when you can't. A written v(N) with rough citations
beats a blocked v(N) waiting for approval. Do not fabricate
concrete titles/authors/DOIs.'
The absolute anti-fabrication rule made agents refuse to write
anything unless they could be sure. Coupled with no user available,
the whole loop stalled — turns completed successfully but no output
converged.
Follow-up if this repeats:
- Even shorter prompt (the current one is 300+ words)
- Explicit output-shape enforcement ('respond with only the markdown
artifact, no preamble')
- Cap iteration count so a broken prompt doesn't burn tokens forever
|
||
|
Omar Sobh
|
63305689be |
topology_exec: bump TURN_TIMEOUT to 300s
Turn timeout was 90s. Research coordinator turns run 60-120s routinely (long prompt + cold tool selection) — 90s was tripping legitimate turns while the daemon was still working, throwing away completed inference. 300s gives real turns room while still bounding worst-case at a walk-away limit. Latest wizard-created topic hit this precisely: bridge attached, handshake worked, claude auth verified in the container, but the run's status went to failed with 'turn executor failed: turn timed out' after exactly ~90s. The topology worker's stale-run sweep at 180s covers the case where the worker itself dies mid-turn — a 300s turn still checkpoints every step so a stuck worker will get requeued. |
||
|
Omar Sobh
|
a0fb64ac47 |
clippy: use bollard::models::NetworkConnectRequest (deprecation)
CI's clippy step failed on commit
|
||
|
Omar Sobh
|
194d63ef36 |
research_container: attach bridge network so per-team egress works
Per-team containers were spawned onto clawmates_core only. That
network is Internal=true on gw-04 (no default gateway to the host's
default route), so any egress attempt — including the daemon's
own claude/gemini/groq API calls — fails with FailedToOpenSocket
and the turn times out.
The shared clawmates-runtime is attached to BOTH clawmates_core AND
the default bridge (that's how it can hit api.anthropic.com); the
per-team containers were missing the second network.
Fix: after start_container succeeds, best-effort connect the
container to `bridge` too. Both spawn() and spawn_loop() call the
same helper. Idempotent — a 403 from Docker on repeat-attach ("already
on network") is silently ignored.
Verified out-of-band on the current stuck research team container:
- Manually `docker network connect bridge research-<id>-team`
- `docker exec ... claude --print "reply only: ok"` → returned "ok"
- Auth + egress both working, so next iteration should complete.
Sequence of pipeline fixes finally converging:
1. materialize_topic_loops didn't fire burst → fixed by
fire_initial_burst_if_set
2. research iteration skipped clone/spawn → fixed by
prepare_topic_runtime
3. graph parse failed → fixed by build_topic_graph_json
4. bind-mount perms wrong → chown 65532:65532
5. server image missing git → debian:12-slim base
6. daemon required pairing → prewrite_daemon_config
7. daemon rejected unknown agents → template config from shared runtime
8. THIS: no external egress → attach bridge post-start
|
||
|
Omar Sobh
|
fd82fe6762 |
research_container: template daemon config from shared runtime
Per-team daemon boots with require_pairing=false but no
`[agents.*]` sections. Server's ws connect authenticates fine and
then trips over "Unknown agent `coordinator` — no [agents.coordinator]
entry configured." (400).
Fix: prewrite_daemon_config now reads a template config from
CLAWMATES_RUNTIME_TEMPLATE_CONFIG (default
/var/lib/clawmates-runtime-template/config.toml) which mirrors the
shared clawmates-runtime container's config with all agent + provider
sections. We strip the template's [gateway] block (its paired_tokens
list is encrypted with the shared runtime's key and un-decryptable
per-team) and prepend a fresh [gateway] require_pairing = false.
Falls back to a minimal pairing-off config with a loud eprintln when
the template isn't readable — the log line makes the misconfig
visible instead of silently 400-ing.
Verified out-of-band on the current stuck team container:
- Restarted with the shared config + rewritten [gateway] section
- Daemon boots cleanly, logs "Pairing: DISABLED (all requests accepted)"
- ws /ws/chat handshake returns 101 Switching Protocols + session_start
message (auth working end-to-end)
gw-04 deploy step (already applied):
1. sudo mkdir -p /var/lib/clawmates-runtime-template
2. sudo cp /root/clawmates-runtime/data/.zeroclaw/config.toml \
/var/lib/clawmates-runtime-template/config.toml
3. sudo chown -R 65532:65532 /var/lib/clawmates-runtime-template
4. Compose: added
/var/lib/clawmates-runtime-template:/var/lib/clawmates-runtime-template:ro
+ CLAWMATES_RUNTIME_TEMPLATE_CONFIG env
to /opt/clawmates/docker-compose.yml
(backup: docker-compose.yml.bak-template)
Follow-up: expose a "reload template" endpoint or re-copy the shared
config on each server boot so we don't drift when the shared runtime
adds a new agent.
|
||
|
Omar Sobh
|
4be3e43f6f |
research_container: prewrite daemon config with require_pairing=false
Per-topic and per-loop team containers spawn from clawmates-runtime with an empty daemon config, so they boot with require_pairing=true and an empty paired_tokens store. Every incoming ws connect from the API server got 401 Unauthorized because the server's ZEROCLAW_TOKEN wasn't in that store. The shared clawmates-runtime container has a paired_tokens list maintained out-of-band (encrypted enc2:… entries in /root/clawmates-runtime/data/.zeroclaw/config.toml on gw-04). That list isn't portable to freshly-spawned per-team containers — the tokens are encrypted with a key we don't share, and pairing new tokens requires a pairing code we don't generate. Simplest correct answer: per-team containers are ephemeral, live on the private clawmates_core docker network, and only accept traffic from the API server. Disabling pairing there closes zero security holes. New helper prewrite_daemon_config(state_root): - creates <state_root>/.zeroclaw/ - writes config.toml with schema_version=3 + [gateway] require_pairing=false - skips write when config.toml already exists so a manually-paired team container survives re-spawn Called from both spawn() (per-topic) and spawn_loop() (per-loop) right after ensuring the host state dir. Existing paired containers are unaffected; new ones come up open-network to the compose stack. Verified out-of-band by writing the same config into the current stuck team container and restarting it — daemon health flipped require_pairing from true to false. |
||
|
Omar Sobh
|
2df9dd04df |
research: build real topology graph in materialize_topic_loops
The wizard-created research loop's first iteration failed with
"missing or invalid graph" — materialize_topic_loops was writing
`{nodes: [], edges: []}` as a placeholder, which the topology worker
rejects. Also explains why prepare_topic_runtime hadn't cloned the
repo or spawned the container: the run failed before
compose_and_enqueue_iteration got to call it.
Fix: extract build_topic_graph_json() into research_setup.rs — same
shape start_topic uses (hydrate roster, promote a role_slot-tagged
coordinator to index 0 for hub_spoke/hierarchical/star_moe, build
via cm_topology::build, serialize via cm_topology::to_json). Called
from materialize_topic_loops instead of the empty placeholder.
Fully best-effort. Any DB/topology failure falls back to a
single-node hub graph so the loop still runs (degraded, but not
silently broken).
The next wizard-created topic should now:
1. Materialize the research loop with a valid graph
2. Fire the initial burst
3. compose_and_enqueue_iteration calls prepare_topic_runtime →
clones the repo, spawns the container
4. Enqueues a run whose graph parse succeeds
5. Run drives to completion, produces an outcome
|
||
|
Omar Sobh
|
51756d0e68 |
clippy: struct-bundle enqueue_iteration_with_topic + fix doc list warnings
CI's clippy stage failed with -D warnings on three classes of lint:
1. map_clone in loops::recent_reorders — .map(|a| a.clone()) is the
pattern clippy wants replaced by .cloned(). Trivial swap.
2. too_many_arguments on enqueue_iteration_with_topic (8 args, ceiling
7). Refactored the caller-side surface into a new
IterationEnqueue<'a> struct with fields for each column. Matches
the pattern research::NewTopic + loops::NewLoop already use for
the same clippy ceiling. Callers in loops.rs (routes) + the
enqueue_iteration wrapper updated to build the struct literal.
3. doc_lazy_continuation + doc_list_indentation — two doc comment
blocks used ambiguous list-like layouts:
- compose_iteration_task's ASCII-art prepended-block preview:
wrapped in a ```text fence so clippy stops parsing "RESEARCH
ARTIFACT:" etc as list continuation.
- fire_initial_burst_if_set's mention of `burst - 1`: rewrote so
"- 1" doesn't start a line and get misread as list marker.
Purely a refactor + doc pass; no behavior change.
|