956be2cf4fb6d21052cb0a93512439adb76ebd39
31
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar Sobh
|
0b7f247b0e |
wizard: 'fresh coding team' picker for paired coding loop
Second slice of the per-loop-team arc. The paired-coding-loop checkbox
in ResearchWizard step 6 now exposes a two-option picker:
⦿ Provision a dedicated coding team (default when the loop is on)
— fresh 'Coding · <topic>' team row, risk_profile =
coding_readwrite, clawmates_door in mcp_bundles. Loop's
team_id is bound at wizard-submit time.
○ Reuse the research team (legacy) — no team_id bound; coding
iterations spawn against the research topic's container.
Frontend
- New codingTeamMode state, radio picker rendered under the checkbox.
- research.ts createTopic body gains paired_coding_team_mode?: 'fresh'|
'reuse'.
Backend
- CreateTopicRequest gains paired_coding_team_mode: Option<String>.
- materialize_topic_loops takes it through and, when 'fresh', calls
the new provision_fresh_coding_team helper — inserts a teams row
via the existing insert_team_with_lifecycle (pipeline kind, same
graph as the loop), sets its runtime-config via
set_team_runtime_config, then binds loop.team_id.
- All operations best-effort with stderr logging — a team-provision
failure leaves the loop functional under the legacy fallback.
Not shipped in this slice (deferred to runtime hookup slice):
- research_container::spawn keyed on team_id → per-team container
- Config template rewrite injecting the team's risk_profile
- Migration of existing paired loops onto their own teams
The plumbing lands now so the wizard's intent is recorded; the
runtime honors it in the next PR.
|
||
|
Omar Sobh
|
36a9fbe81f |
research/start: allow rerun on loop-owned topics too
D1-fold loop guard 409'd even the rerun path if the topic was scheduled — a failed iteration couldn't be restarted until the loop's next scheduled fire. Now the loop guard only fires for the fresh 'standby' start; the rerun path (status=='processing' + orphan cancel) works whether or not a loop owns the topic. Loop binding is preserved either way. |
||
|
Omar Sobh
|
6af9d509b0 |
research: rerun cancels orphan runs first so 409 doesn't block restart
The previous rerun guard (standby OR (processing AND in_flight==0)) still 409'd when a lingering queued/running row existed — typically an orphan from a server restart mid-pipeline, before the reaper or stale-checkpoint requeuer had marked it failed. Now: allow rerun on any 'processing' topic; before enqueuing the fresh run, batch-cancel every in-flight run for the topic so the new run isn't racing them. Terminal states (reviewing / publishing / published) still 409 as before. |
||
|
Omar Sobh
|
671d3ac4f0 | clippy: fix doc_lazy_continuation on TopicListItem.runs_failed | ||
|
Omar Sobh
|
f70f6c679e |
research: errored-state card + one-click rerun (no wizard re-entry)
When a topic ends up parked in 'processing' with all runs failed and nothing in flight, the sidebar card was still spinning as if progress were happening. Now: Backend - topology_runs::run_counts_by_research_topic — batch query that returns (in_flight, failed-since-last-success) per topic. Used by the list endpoint; dynamic sqlx::query() so no prepare needed. - TopicListItem DTO gains runs_in_flight + runs_failed. - start_topic status guard relaxed: allow (standby) OR (processing AND runs_in_flight == 0). Blocks accidental double-fires on a live pipeline; permits rerun on a failed one. Same request body, same behavior once accepted, so the frontend just POSTs /research/:id/start on the RotateCw click. Frontend - ResearchList detects errored: status===processing && !in_flight && failed>0. Swaps the MiniSpinner for a red AlertTriangle and changes the status text to 'error · N failed'. - New RotateCw icon button next to the delete Trash — same button cluster, one click, no wizard re-entry required. Disables while a request is in flight; error surfaces in the sidebar's shared error banner. |
||
 osobh
|
1cb643142c | research/publish: gate approve+reject on Owner role (#4) | ||
|
Omar Sobh
|
7e0620fd08 |
research: don't advance to reviewing without an outcome + guard-before-write
Two bugs that combine to produce the 409 you get when clicking Approve:
1) notify_run_completed (topology_worker post-hook) was advancing
topic status processing → reviewing whenever the last sibling run
terminated — success OR fail. A failed run with 0 outcomes still
pushed the topic to `reviewing`, the canvas rendered the "Request
publish" affordance, and the reviewer clicked Approve on nothing.
Fixed by adding an EXISTS(research_outcomes …) clause to the
UPDATE. Topic stays in `processing` when no outcome exists; the
loop's next iteration still has a chance to produce one.
2) decide_publish was calling
research_publish_approvals::decide(approve=true)
FIRST (which flips the row to `status='approved'`) and then
running the "no outcome? 409" guard SECOND. On the 409 return,
the DB was left half-flipped: approval says approved, topic still
in reviewing, no outcome exists, and every future click to the
same approval returns 409 on the "already decided" guard —
leaving reviewers with no way forward.
Fixed by moving the outcome-existence check BEFORE the decide()
call. On 409 now nothing was written, so the reviewer can try
again cleanly once an outcome is produced.
Also unstuck the current stuck row out-of-band (SQL UPDATE to reset
the approval to pending + topic to processing) so the user isn't
forced to delete the topic to escape the 409 loop.
sqlx dynamic query — the new EXISTS clause wasn't in the offline
cache so I switched notify_run_completed to plain `sqlx::query`.
|
||
|
Omar Sobh
|
bfcdca0583 |
research-canvas: managed-by-loop UI + start_topic guard (fold cleanup)
Closes the UX gap the fold introduced: the topic canvas was still showing "Start research" for standby-state topics even when a scheduled loop already owned the runs. Clicking it would 409 (or worse: race the loop into a duplicate run). Topic status stayed at standby forever because the loop path bypassed start_topic's set_status transition. Four changes: 1. **Backend status transition** — compose_and_enqueue_iteration for kind='research' now calls set_status_if(standby, processing) on the topic before the run is enqueued. New DB helper set_status_if only advances when the current status matches the "from" arg — safe against races and re-invocations. Later iterations no-op since the topic is already past standby. 2. **has_managed_loop on TopicDetail** — get_topic hydrates a new ManagedLoop struct (loop_id, title, enabled, next_fire_at, last_run_id, schedule_summary) when a kind='research' loop is bound to the topic. summarize_schedule() derives a human string from the loop's triggers jsonb (e.g. "cron: 0 3 * * * · on new artifact", "one-shot", "manual"). New DB helper loops::research_loop_for_topic returns the row. 3. **Canvas branch** — nextAction takes a managedByLoop flag; when set + status=standby, returns null (no button). The canvas renders a "MANAGED BY LOOP" strip below the topic title showing loop name, schedule summary, next fire time, and enabled dot. Reviewer buttons (Request publish / Approve / Reject) still show normally in later states — reviewers should still promote outcomes even when a loop is producing them. 4. **start_topic guard** — refuses with 409 when a research loop already owns the topic. Closes the direct-POST hole for anyone bypassing the frontend. TS type + summarize_schedule live in the same commit so an old client hitting a new backend just ignores the extra field (no breakage), and a new client hitting an old backend renders the classic buttons (managed_by_loop is optional). |
||
|
Omar Sobh
|
c0c5fd7104 |
research: extract setup helpers into research_setup.rs (fixes file-size gate)
research.rs hit 1446 lines with the wizard-fold + regression-fix
commits, tripping the CI file-size gate (limit 1250). Pure extraction
into a sibling module; no behavior change.
Moved to routes/research_setup.rs:
- RepoContext (now pub — used by build_coordinator_task)
- TopicSchedule (now pub — request body sub-struct)
- research_workspace_root (now pub)
- prepare_topic_runtime (already pub — used by loops iteration path)
- ensure_repo_workspace (now pub — used by start_topic + prepare)
- materialize_topic_loops (now pub — used by create_topic)
research.rs re-imports them via `use crate::routes::research_setup::{...}`
so the calling code reads identically.
New line counts:
research.rs 1127 lines (was 1446, limit 1250)
research_setup.rs 321 lines (new)
Also updated the loop-iteration callsite in routes/loops.rs to point
at the new path (crate::routes::research_setup::prepare_topic_runtime).
No API changes; migrations unaffected.
|
||
|
Omar Sobh
|
80c23e57ed |
research: fix wizard loops never firing + missing clone/spawn (regression)
Two bugs surfaced by the first end-to-end wizard run — the pipeline diagnostic showed "0 run(s)", "Repo bound but never cloned", and "Container not spawned" for a topic that had been created with a nightly research loop. Bug 1: materialize_topic_loops bypassed initial_burst firing. The wizard-materialized loops path calls cm_db::repo::loops::create directly (a plain INSERT). The initial_burst-fires-first-iteration logic lived inside the create_loop HTTP handler, so wizard loops landed in the DB but never fired their initial iteration. Fix: extract routes::loops::fire_initial_burst_if_set as a pub helper that read triggers, ensures the loop container, and calls the kind-aware compose_and_enqueue_iteration. Both create_loop and materialize_topic_loops now call it. Bug 2: research-kind loop iterations skipped clone/spawn. compose_research_iteration_task only built the coordinator prompt; the repo clone and topic container spawn lived only in start_topic. So the first research iteration ran against a nonexistent clone directory and a stale gateway, and every run failed. Fix: extract routes::research::prepare_topic_runtime as a pub helper that runs ensure_repo_workspace + research_container::spawn. Idempotent — second iteration reattaches. Called from compose_and_enqueue_iteration before enqueuing a research iteration. Topics without a repo bound are a no-op. Both fixes ship as one commit because they surface together on the same user path (wizard → research loop → first iteration) — you can't hit one without the other manifesting. Follow-up: start_topic still runs its own inline clone/spawn code (now duplicated with prepare_topic_runtime). Next commit collapses start_topic to just call prepare_topic_runtime + build_task like the loop path does, so the one-shot and loop paths agree on setup. |
||
|
Omar Sobh
|
3e42b1ea39 |
research-wizard: schedule step (once/nightly/manual) + paired coding loop
Completes the research/loop fold. The wizard now closes with a
"How should this research run?" step; picking any mode materializes a
kind='research' loop bound to the topic, and an optional checkbox
adds a paired kind='exec' loop that consumes each new artifact.
Frontend:
- ResearchWizard grows from 5 to 6 steps. Step 6 is the schedule
picker:
· Just once — initial_burst=1, no other triggers
· Nightly — initial_burst=1 + cron "0 3 * * *"
· Manual — webhook_enabled=true
Below the radios, an optional card offers "Also create a coding
loop that consumes each new artifact" — creates a paired
kind='exec' loop with on_artifact_update=true + initial_burst=1
bound to the same topic.
- createTopic API type extended with `schedule` + `create_paired_coding_loop`.
- Both fields ride the existing POST /api/research call; back-compat
is preserved when the wizard omits them.
Backend:
- CreateTopicRequest gains TopicSchedule + create_paired_coding_loop.
- After topic + agent attach, create_topic calls
materialize_topic_loops which:
1. Creates a research-kind loop titled "Research · <topic>" bound
to the topic. Triggers vary by schedule mode; next_fire_at
computed from cron for nightly. Falls back silently if
loop-create errors so the topic still lands.
2. Flips kind to 'research' via loops::set_kind (NewLoop doesn't
take kind directly — default is 'exec' for backward compat).
3. Optionally creates a coding loop titled "Coding · <topic>"
with on_artifact_update=true + initial_burst=1.
- cm_db::repo::loops::set_kind — trivial UPDATE helper used by the
materialize path.
D-answer callouts:
- D1 (fold): every runnable thing is now a loop. "Just once" is a
research loop with initial_burst=1 and no other triggers.
- D2 (inherit): both paired loops carry the same source_research_topic_id
— repo binding lives on the topic, not duplicated.
- D3 (coordinator resolves): the research iteration prompt (from the
earlier commit) instructs the team to preserve stable INT ids and
mark deprecations; coding loops' consumed lists stay valid across
versions.
Follow-ups queued:
- Kind pill on LoopsList cards (research=purple, exec=cyan) so users
can tell them apart at a glance.
- Extract start_topic's task-build so kind='research' iterations
reuse the same coordinator prompt shape as one-shot runs (they
currently use a simpler refresh-oriented prompt; that's fine for
MVP but a rich shared build would give better parity).
- Research topic sidebar shows "linked to N loops" badge.
|
||
|
Omar Sobh
|
3ed1d03d2b |
research: integrations outcome + rich wizard cards + coordinator template
Adds a fifth outcome kind ('integrations') tuned for the "audit repo,
survey papers, propose a menu of concrete integrations" use case.
Every INT-XX item is self-contained (what, how, where, prereqs, effort,
risk, testing, rollback, acceptance) so a downstream loop can execute
one per iteration.
Backend:
- Migration 0041 drops + re-adds the outcome_kind CHECK constraint
with 'integrations' allowed. Existing rows unaffected.
- VALID_OUTCOMES gains 'integrations'.
- New deliverable_template(kind) returns the canonical section shape
for each outcome — spec, prod_plan, roadmap, paper, integrations all
get first-class treatment (prior: all shared a bare label).
- build_coordinator_task injects an ARTIFACT SHAPE block from the
template into the coordinator prompt, so the final synthesis
actually matches the promise the wizard made.
Frontend:
- OutcomeKind gains 'integrations'.
- ResearchWizard OUTCOMES list carries a `sections` array per kind.
- Selected card renders an "ARTIFACT WILL CONTAIN" preview so users
pick by seeing what they'll get, not by reading a one-line hint.
- Integrations card gets the fullest preview (executive summary +
INT-XX card shape) since it's the most structured deliverable.
Follow-ups queued (next commit): loop wizard "Import from research
artifact" bridge + one-INT-per-iteration mode.
|
||
|
Omar Sobh
|
3bcd18bf50 |
research: split pipeline diagnostics into research_pipeline.rs
Gates step failed on run#211 because research.rs grew to 1313 lines (over the 1250 budget) after adding the pipeline_state handler in the previous commit. Move the diagnostic into its own module — pure extraction, no behavior change. research.rs: 1313 → 1126 lines research_pipeline.rs: new, 187 lines lib.rs: route now points at routes::research_pipeline::pipeline_state |
||
|
Omar Sobh
|
cca6e2be20 |
research: pipeline diagnostics + refuse publish without outcome
Two fixes surfaced by the first prod run of the pipeline:
1) Silent skip-to-published bug (R1 gap):
approve_publish transitioned reviewing → publishing → published
without checking that an outcome existed. Result: pipeline could
fail silently (LLM auth error, network, etc), no outcome would be
written, but state advanced to 'published' and the download endpoint
returned 404 with no user-visible error. Now refuses with 409
Conflict when no outcome exists so the frontend can surface WHY.
2) No end-to-end visibility:
Users had no way to see where a run failed until they clicked
Download and got nothing. Adds
GET /api/research/:id/pipeline-state — a read-only per-stage report
walking:
- staffing (agents assigned)
- repo (bound + cloned)
- container (per-topic team runtime spawned)
- runs (count + failed count + latest error text)
- outcomes (count — the artifact rows get_artifact reads)
- approval (pending flag)
Each stage returns ok / warn / fail / skip plus optional detail text
so the failure reason surfaces at the diagnostic level.
Frontend:
ResearchCanvas shows a compact PIPELINE strip below the topic title,
green/amber/red dots per stage, click to expand a full checklist
with per-stage detail (including the LLM error from the last run
attempt). Polls every 6s while the topic is processing/publishing.
Follow-up:
- Root cause of the specific failure just observed: Claude CLI in
the clawmates-runtime container isn't authenticated. Deploy-side
config sweep (CLAUDE_CODE_OAUTH_TOKEN or ANTHROPIC_API_KEY into
the runtime image env), not a code fix.
- Structured event stream on top of run_events for real per-step
replay in the diagnostic panel.
|
||
|
Omar Sobh
|
44d6e95022 |
fmt: apply rustfmt across the P2 arc
CI's rustfmt check flagged the multi-line sqlx::query() calls I introduced in P2 (loop_id_for_run, zeroclaw_gateway_url, etc.). No behavior change — pure formatting. |
||
|
Omar Sobh
|
e3011ed025 |
research: reject-with-revision loop (R2)
Before: reviewer rejected a publish → audit log flipped, topic stayed in reviewing, no way to feed the critique back into the run pipeline. Reviewers with revision notes had to eat them or hand-message the coordinator. Now: reject accepts an optional `notes` field. When present: - Persisted on the research_publish_approvals row (migration 0039). - Topic flips `reviewing → standby` so the next `start_topic` is legal. - `start_topic` reads the most recent rejected-approval notes for the topic and prepends "PRIOR REVIEW NOTES (address these in this revision):\n<notes>\n---" to the coordinator task. Loop closes through the same run pipeline — no new spawn code path, which means the reviewer's guidance flows through the same topology_worker, run_events, outcome-writer chain and lands as a fresh research_outcomes row (versioned, prior drafts preserved). No notes on reject = legacy behavior (topic stays in reviewing, publish requests still allowed). Migration 0039 adds nullable `notes TEXT` to research_publish_approvals. `decide()` gains a `notes: Option<&str>` parameter (only one caller, updated inline). New `latest_rejection_notes(pool, topic_id)` helper for start_topic. Frontend: - rejectPublish(id, notes?) now sends a JSON body when notes are provided. - ResearchCanvas reject button opens an inline form with a textarea + Cancel/"Send back for revision" pair. Empty notes → plain reject. - Button label switches: "Send back for revision" when notes present, "Reject without notes" when empty. Follow-up: - Notes shown in the review UI on the resulting draft so the next reviewer sees what changed. - Multiple rejection rounds — currently only the LATEST rejection's notes surface. Accumulating history is a schema-only tweak. |
||
|
Omar Sobh
|
f9e8d8d779 |
research: publishing → published + artifact download (R1)
Before this commit, approve_publish left topics stuck in 'publishing' forever — the sidebar 'published' bucket was always empty and nothing surfaced the artifact. Two-part fix: State machine — approve_publish now transitions reviewing → publishing → published in one API call. Real async packaging isn't a thing yet because the artifact IS the markdown already written to research_outcomes when the last run completed (topology_worker). The intermediate 'publishing' state is preserved (schema-level trigger stamps published_at on landing there) so we keep the option to detour through it later for pdf render / mirror-to-store / etc. Download endpoint — GET /api/research/:id/artifact returns the latest outcome as text/markdown with Content-Disposition: attachment. Filename sanitizes the topic title to ascii-alnum + dash and appends the outcome version so accumulated revision drafts (post R2) don't clobber. Workspace ownership check via research_topics::get; 404 if no outcome yet (reviewers browsing before a run completes). Frontend — ResearchList shows a green Download icon-button next to Delete when status === 'published'. Clicks trigger a plain anchor download of the .md — no JS blob dance needed since the response is already an attachment. Follow-up queued: pdf render (server-side or client-side of the md), mirror-to-store (S3-ish or Obsidian vault) as an async step during the publishing→published detour. |
||
|
Omar Sobh
|
f60df36717 |
research: teardown per-topic container on publish + delete (P1)
Path B spawned a per-topic team container on start_topic (research runtime) but nothing ever stopped it. Containers accumulated on gw-04 across the lifetime of every topic — the earlier cleanup pass reclaimed 6.72 GB of these. Now the container is stopped + removed automatically at the two terminal transitions: - delete_topic — the topic row is gone, the container is meaningless. - approve_publish (reviewing → publishing) — the research work is done. Artifact writing (queued as R1) reads from the durable run_events log, so it doesn't need a live runtime. Wiring is a fire-and-forget teardown() helper in research_container.rs: - Connects Docker via the same socket-proxy shim as spawn(). - Calls the existing stop() which is idempotent (404 = already gone, 304 = already stopped, both treated as ok). - Errors log with eprintln! and don't block the API response — a topic is done whether Docker is reachable or not. Dev machines without Docker running just log a debug line and return. Path A (repo-only mount + shared clawmates_core network) is untouched so there's no Traefik dynamic-file cleanup needed. If we ever add per-topic Traefik routing, extend teardown() with the corresponding file remove. Follow-up: gw-04 will still have any legacy containers from before this commit. One-shot `docker ps -a --filter "name=research-" -q | xargs -r docker rm -f` on gw-04 is the manual sweep. |
||
|
Omar Sobh
|
763b95a253 |
api: gate create_loop + create_topic on empty workspace roster (W2)
Frontend already disables the create button when the roster is empty, but nothing stopped a direct POST from materializing an orphan loop / topic with nothing to staff it. Both handlers now check the workspace agent count up front and return 409 Conflict when it's zero. - crates/cm-api/src/routes/loops.rs: gate at top of create_loop - crates/cm-api/src/routes/research.rs: gate at top of create_topic Uses the existing cm_db::repo::agents::count_active(pool, ws) helper (count of workspace agents where deleted_at IS NULL). 409 is the right mapping: state-of-the-workspace-prevents-this, not client-input-bad. |
||
|
Omar Sobh
|
21ac35c8d4 |
research: spawn per-topic ZeroClaw team container on start (commit 1/3)
Commit 1 of the path-B (real per-topic isolation) plan. The
container spawns and its coordinates persist — nothing talks to
it yet; commit 2 wires ZeroClawDriveExecutor to prefer the topic's
URL when populated. This split keeps each landing verifiable.
Backend
- Migration 0038: research_topics gets zeroclaw_container_name +
zeroclaw_gateway_url columns. Both nullable so a topic can exist
before a spawn and teardown just NULLs them out.
- cm-db: ResearchTopic struct extended; get/list SELECTs updated;
new set_zeroclaw_container(id, workspace_id, name, url) helper
used both for spawn (Some/Some) and teardown (None/None).
- cm-api: bollard added as a workspace dep (matches cm-sandbox's
version). New research_container module:
· connect() → uses DOCKER_HOST when set (prod's socket-proxy
at tcp://socket-proxy:2375) else the local socket. Same
pattern cm-sandbox already uses.
· container_name_for(topic_id) → "research-<uuid>-team"
(deterministic so a re-start reattaches to the same
container instead of orphaning it).
· inherited_env() → propagates ZEROCLAW_*, OPENAI_*,
ANTHROPIC_*, GEMINI_*, GROQ_* from the parent server env
(provider config + tokens), stripping the server's own
ZEROCLAW_GATEWAY_URL/WORKSPACE so the team runtime doesn't
loop back on itself. Appends ZEROCLAW_GATEWAY_PORT=42617
and ZEROCLAW_WORKSPACE=/zeroclaw-data/workspace for the
team's own listener.
· spawn(docker, topic_id, repo_host_path, state_host_path):
- inspect: if the container already exists, start it if
stopped and return its coordinates (idempotent restart).
- else create with:
image = CLAWMATES_RESEARCH_TEAM_IMAGE or
clawmates-runtime:latest
cmd = [daemon, --host, 0.0.0.0]
env = inherited_env()
mounts = repo_host_path → /workspace/repo (rw)
state_host_path → /zeroclaw-data (rw)
network = CLAWMATES_RESEARCH_TEAM_NETWORK or
clawmates_core
labels = clawmates.role=research-team,
clawmates.research.topic_id=<uuid>
- creates state_host_path first so bind doesn't ENOENT.
· stop(docker, name) → stop + remove. Idempotent on 404/304.
- start_topic wires spawn after the clone completes:
· state root = CLAWMATES_RESEARCH_WORKSPACE_ROOT / <topic> /
state
· on success, persists (name, url) on the topic row so commit
2 can look them up when constructing the executor
· every failure (docker connect, docker create/start, DB
persist) is best-effort: logs and continues. A missing team
container leaves the topic pointing at the workspace-wide
gateway URL (env), preserving prior behavior.
Deploy prerequisites (not in this commit)
- The compose stack's clawmates_server service needs bind-mounts
of CLAWMATES_RESEARCH_WORKSPACE_ROOT (e.g.
/var/lib/clawmates-research:/var/lib/clawmates-research) so
paths the server writes to are visible on the host and the
spawned team container mounts the same underlying data.
- socket-proxy's ACL must allow POST + DELETE on /containers
(already the case in prod per the audited compose file).
|
||
|
Omar Sobh
|
7984e65174 |
research_topics::create: bag 8 args into a NewTopic struct (fixes clippy)
The prior signature took (pool, workspace_id, title, description, outcome_kind, topology_kind, repo_id, created_by) — 8 args, one over the clippy::too_many_arguments ceiling and blocking CI. Refactor to a NewTopic<'a> input struct mirroring the NewLoop / NewSubTopology pattern the codebase already uses. Wizard-side additions like a repo commit branch land as struct fields instead of cascading into every call site. |
||
|
Omar Sobh
|
3465bb7a6d |
research: persist bound repo + shallow-clone on start_topic
This is the minimum viable version of the "agents actually work on
a repo" architecture. Full vision (isolated ZeroClaw container per
topic, dynamic agent provisioning inside, pause/resume, commit
gate) is real weeks of work — this closes the first, most-visible
gap so the ClawHDF5 topic can actually run against its codebase.
Backend
- Migration 0037: research_topics gets repo_id UUID (nullable, FK
to repos ON DELETE SET NULL) and repo_workspace_path TEXT for
the on-disk checkout location. Index on repo_id when set.
- research_topics::create takes repo_id: Option<Uuid>. get + list
select it and repo_workspace_path. set_repo_workspace_path
persists the path once the first clone lands.
- CreateTopicRequest accepts `repo: Option<TopicRepoRef>` — the
same denormalized shape the wizard already sends. Only repo_id
is authoritative; other fields are ignored (dead_code-allowed
so serde still deserializes the full body).
- start_topic branches on topic.repo_id. When set, it calls
ensure_repo_workspace:
· resolves repo.clone_url + repo.default_branch
· target path = CLAWMATES_RESEARCH_WORKSPACE_ROOT
// <topic_id> // repo (defaults under $TMPDIR)
· runs `git clone --depth 1 --single-branch --branch <b>` via
tokio::process. Reuses the checkout if .git already exists.
· persists the path so re-starts skip the clone
· runs `git ls-files` to sample the tree (first 60 entries,
total count reported honestly so the prompt doesn't lie
about coverage)
All best-effort — a clone failure logs but still starts the run
without repo context rather than aborting.
- build_coordinator_task takes Option<&RepoContext>. When present,
the framing gets a REPO block (slug / path / branch / file
sample) and a USING THE REPO section instructing the coordinator
to ground every recommendation in a concrete file reference and
never fabricate paths. The per-topology bodies are unchanged —
the repo guidance sits above them so it applies to every shape.
What this unblocks / doesn't unblock
Unblocks: The coordinator prompt now knows the repo exists, where
it lives on disk, and what's in it. Even without file-editing
tools wired to the checkout, the coordinator can point spokes at
concrete modules and the final artifact can reference real files.
For a spec-shaped outcome like ClawHDF5's, that's the difference
between abstract advice and a spec grounded in the actual crates.
Does NOT unblock: The agents themselves editing files, running
tests, or committing. That requires either mounting the checkout
into the ZeroClaw sandbox or exposing a new MCP tool for
repo-scoped file ops — separate follow-up.
|
||
|
Omar Sobh
|
7c1af2e070 |
research: pipeline-running signal + spinners so users aren't guessing
The prior flow was ambiguous: after hitting Start research, status flipped to "processing" and a "Submit for review" button appeared immediately with no indication that anything was actually running. Users had to guess whether the pipeline was working or stalled. Backend surfaces the truth as a signal: - new topology_runs::active_runs_for_research_topic counts queued+running runs whose research_topic_id matches - TopicDetail includes runs_in_flight: i64 alongside the existing status field, so the canvas can distinguish "pipeline still working" from "runner stalled". ResearchCanvas is now honest about state: - while runs_in_flight > 0, the header status pill grows a cyan "N runs in flight" badge with an inline SVG spinner - the stage-explainer card turns cyan-bordered and shows a "pipeline is running" hint, plus copy pointing the user at the Agents tier where each teammate's activity streams live - the "Submit for review (manual)" button is HIDDEN while any run is in flight — it's an escape hatch for stalled runs only, not the happy-path action. It reappears if runs_in_flight drops to zero but the topic is still marked processing, so a stalled runner can still be nudged along. - the canvas polls getTopic every 4s while status is processing/ publishing or runs_in_flight > 0, so the spinner + outcome swap in automatically when the pipeline completes. ResearchList sidebar: - each row's status dot becomes a spinner when the topic's status is processing or publishing, matching the canvas at a glance - the list also polls every 6s while ANY topic is active, so transitions land in the sidebar without waiting on a parent bump. The poll is gated on a derived boolean to avoid effect thrash. Follow-up: same pattern belongs on LoopsList / LoopsCanvas for loop iterations in flight — same signal (queued+running runs per loop) but not wired here. |
||
|
Omar Sobh
|
316cdbf929 |
research sidebar: delete-with-confirm per topic
Mirror the row-level delete affordance the loops sidebar already
has. Loops was wired earlier; research had a bare title-only card
with no way to remove a stale topic.
- cm-db: research_topics::delete cascades via existing FK rules
(research_topic_agents, research_publish_approvals, and the new
research_outcomes all CASCADE on topic_id; topology_runs's
research_topic_id back-ref is SET NULL so historical runs stay).
- cm-api: DELETE /api/research/{id} → 204. Idempotent.
- Frontend: deleteTopic helper. ResearchList row is now a card
with the existing title/status/outcome header plus a trash icon
that flips the card into an inline "Delete topic + all outcomes?"
confirm strip. Confirm → red Delete / gray Cancel. If the
deleted topic was selected, selection clears; local counter
bumps the list refetch without waiting on a parent.
|
||
|
Omar Sobh
|
a2d3d85ebe |
research pipeline v2: topology-aware start + persisted draft
Three connected changes that turn "Start research" from a status flip into a real pipeline that produces a reviewable artifact: - Migration 0036: adds research_topics.topology_kind (default 'hub_spoke') and a new research_outcomes table (id, topic_id, version DESC, body_md, produced_by_run_id, created_at) so each run's final synthesis is versioned and persistent. - Wizard now has a topology picker in the Outcome step — hub_spoke / pipeline / hierarchical / star_moe — with copy that steers users to the right shape (Pipeline for research → distill → analyze → implement rosters, hub_spoke for the coordinator- and-specialists default). - start_topic reads the chosen topology_kind, parses it into a cm_topology::TopologyKind, and dispatches a per-shape coordinator prompt via build_coordinator_task. Pipeline explicitly tells stage 1 not to write the final artifact and propagates a "final stage MUST emit a complete markdown document with measurable acceptance criteria" instruction downstream. The graph builder is called with the topology the user actually picked instead of hard-coded HubSpoke. - topology_worker::freeze_research_outcome fires after every successful complete(). It looks up research_topic_id on the run; if set and final_output is non-empty, it inserts a new research_outcomes row (version auto-derived server-side via coalesce(max(version), 0) + 1). Best-effort — a DB hiccup logs but doesn't fail the run. - TopicDetail now includes topology_kind and latest_outcome. ResearchCanvas swaps in the outcome's body_md (rendered as pre-wrap markdown, versioned header, produced-at timestamp) whenever an outcome exists; the original prompt collapses into an "Original prompt" <details> below so it's still one click away. Pre-run topics still show the description as before. Follow-ups still open: reject-with-revision loop feeding the coordinator, publishing → published transition + real artifact export (md / pdf), and an approvals inbox surface for reviewers. |
||
|
Omar Sobh
|
7e2b02d8bb |
research: wire start_topic to actually run the pipeline
The prior start_topic only flipped the status column — no work was enqueued. Now clicking "Start research" actually launches the assigned agents through the orchestrator. - start_topic loads the topic + its research_topic_agents, picks a coordinator (first slot with role_slot containing "coordinator"; else the first slot), and swaps it to index 0. - Builds a hub_spoke topology graph via cm_topology::build with roles = [coordinator, spoke1, spoke2, …]. hub_spoke wires edges from the hub to every spoke and back, so the coordinator can address any specialist per turn. - Assembles a coordinator prompt from the topic's title, description, outcome_kind, and a roster line for each teammate — so the coordinator knows who's on the team and what each does. - Enqueues via a new topology_runs helper enqueue_run_for_research_topic that stores research_topic_id on the run row. `topology_worker::maybe_transition_research_topic` → `notify_run_completed` already picks up on that back-ref and flips the topic processing → reviewing when the last run terminates — that path was dead code until now. - Per-agent activity streams into each claw's card for free: the orchestrator journals turn events into run_events; the existing /api/world/live SSE normalizer emits agent.reasoning.delta / agent.tool.call / agent.task.update keyed by agent id, which ClawCommandCenter is already subscribed to. The `published` terminal state is still unreached (that's the "publishing → published + artifact" step from the earlier walkthrough — separate follow-up). |
||
|
Omar Sobh
|
81a436d221 |
research canvas: wider rail + stage explainer + 409 fix
Three connected fixes from a single session's feedback:
- Structure rail widened from 60→76 px, tabs 42→58 px wide with a
bit of left padding so labels ("Research", "Visualizations") no
longer bump against the active-tab indicator strip.
- Research canvas: each stage now shows a small "STAGE · <status>"
card explaining what state the topic is in and a "Next → …" hint
describing what the primary button will do. No more guessing
which of standby/processing/reviewing/publishing means what.
- Request-publish 409 fix:
- Backend TopicDetail now includes has_pending_publish_request
(SELECTs pending_for_topic when the topic loads). Frontend
TopicDetail interface + ResearchCanvas honor the flag: when
an approval is already pending the "Request publish" button
is replaced with an amber "Awaiting reviewer approval" pill,
so double-clicks can't 409 in the first place.
- runAction() also catches 409 as a signal-of-success (the
user's intent — "queue for approval" — is satisfied by the
first attempt), refetches the topic, and lets the new
awaiting-approval card render instead of surfacing a scary
error to the user.
|
||
|
Omar Sobh
|
8a4e222aec |
research: auto-transition processing → reviewing on last run
Hooks the topology_worker's post-terminal path into a new notify_run_completed repo helper that atomically transitions the topic processing → reviewing when the completed run has research_topic_id set AND no siblings for that topic are still queued or running. Guarded on status='processing' so a retry, a re-fire, or a topic already past processing are all no-ops. Best-effort at the worker; DB hiccups are logged and never fail the run. The manual /submit-review endpoint stays as an escape hatch for topics that end up parked in processing with nothing to complete (updated the doc comment). |
||
|
Omar Sobh
|
973eeb272e |
research: publish approval gate + explicit state transitions
Fourth commit of the Research + Loops arc. Completes the state machine
for research topics with the publish approval gate the spec asked for.
Migration 0032 — research_publish_approvals
Dedicated small table (id, workspace_id, topic_id, requested_by,
status, decided_by/at, created_at). Keeping it separate from the
existing `approvals` table (0001) because that one is tightly coupled
to gated tool calls inside an agent run — session_key + run_id +
action_type + category + payload + preview + requested_by_agent, all
NOT NULL. Forcing those nullable would ripple through cm_safety;
cleaner to give publish approvals their own two-transition state
machine.
New endpoints
POST /api/research/:id/submit-review processing → reviewing
(v1 caller-driven; the
orchestrator hook comes
when we wire actual runs)
POST /api/research/:id/request-publish creates a pending
approval. Rejects with
409 if the topic already
has one open.
GET /api/research/publish-approvals list workspace's pending
POST /api/research/publish-approvals/:id/approve flips approval to
approved + transitions
the topic
reviewing → publishing
(which stamps
published_at)
POST /api/research/publish-approvals/:id/reject stays in reviewing; new
requests allowed
The approve/reject write is an atomic UPDATE ... WHERE status = 'pending';
the decide() repo function returns whether the caller won the race so
concurrent double-approves collapse to a single topic transition.
State machine after this commit:
standby ─POST /start─▶ processing ─POST /submit-review─▶ reviewing
─POST /request-publish + approve─▶ publishing ─(future: artifact
assembly)─▶ published
|
||
|
Omar Sobh
|
4ffcb3d652 | chore: cargo fmt --all — clean up research.rs formatting | ||
|
Omar Sobh
|
fb047879c2 |
research: backend routes + repo + wizard refine (behind /api/research)
Second commit of the Research + Loops arc. Builds on 0030 by lighting up
the container CRUD, the agent-slot attach/detach, the standby→processing
transition, and the wizard's one-shot LLM refine.
GET /api/research list workspace's topics
POST /api/research create (accepts wizard output)
GET /api/research/:id detail (topic + attached agents)
PATCH /api/research/:id update non-status fields
POST /api/research/:id/agents attach agent (idempotent)
DELETE /api/research/:id/agents/:agent detach
POST /api/research/:id/start standby → processing
POST /api/research/wizard/refine one-shot LLM refine
The refine endpoint accumulates the workspace's default LLM provider's
stream into a JSON object (`{title, description}`) with the tight system
prompt at the top of the module. Same provider routing as agent runs
(via runtime.provider()), so a workspace already using GLM/Kimi gets it
for free.
State machine's remaining transitions (processing → reviewing on last
run_completed; reviewing → publishing via approvals gate) land with the
orchestrator hookup + approvals extension. Publish approval and loops
are separate commits still to come.
Adds cm-llm as a direct cm-api dep (previously only pulled transitively
via cm-runtime) so the refine endpoint can build a ChatRequest. Uses
sqlx::query! for compile-time verification; .sqlx cache generated on
morpheus against a fresh migrated DB.
|