163 Commits

Include renames

Author	SHA1	Message	Date
Omar Sobh	c0c5fd7104	research: extract setup helpers into research_setup.rs (fixes file-size gate) ... research.rs hit 1446 lines with the wizard-fold + regression-fix commits, tripping the CI file-size gate (limit 1250). Pure extraction into a sibling module; no behavior change. Moved to routes/research_setup.rs: - RepoContext (now pub — used by build_coordinator_task) - TopicSchedule (now pub — request body sub-struct) - research_workspace_root (now pub) - prepare_topic_runtime (already pub — used by loops iteration path) - ensure_repo_workspace (now pub — used by start_topic + prepare) - materialize_topic_loops (now pub — used by create_topic) research.rs re-imports them via `use crate::routes::research_setup::{...}` so the calling code reads identically. New line counts: research.rs 1127 lines (was 1446, limit 1250) research_setup.rs 321 lines (new) Also updated the loop-iteration callsite in routes/loops.rs to point at the new path (crate::routes::research_setup::prepare_topic_runtime). No API changes; migrations unaffected.	2026-07-10 09:15:25 -07:00
Omar Sobh	80c23e57ed	research: fix wizard loops never firing + missing clone/spawn (regression) ... Two bugs surfaced by the first end-to-end wizard run — the pipeline diagnostic showed "0 run(s)", "Repo bound but never cloned", and "Container not spawned" for a topic that had been created with a nightly research loop. Bug 1: materialize_topic_loops bypassed initial_burst firing. The wizard-materialized loops path calls cm_db::repo::loops::create directly (a plain INSERT). The initial_burst-fires-first-iteration logic lived inside the create_loop HTTP handler, so wizard loops landed in the DB but never fired their initial iteration. Fix: extract routes::loops::fire_initial_burst_if_set as a pub helper that read triggers, ensures the loop container, and calls the kind-aware compose_and_enqueue_iteration. Both create_loop and materialize_topic_loops now call it. Bug 2: research-kind loop iterations skipped clone/spawn. compose_research_iteration_task only built the coordinator prompt; the repo clone and topic container spawn lived only in start_topic. So the first research iteration ran against a nonexistent clone directory and a stale gateway, and every run failed. Fix: extract routes::research::prepare_topic_runtime as a pub helper that runs ensure_repo_workspace + research_container::spawn. Idempotent — second iteration reattaches. Called from compose_and_enqueue_iteration before enqueuing a research iteration. Topics without a repo bound are a no-op. Both fixes ship as one commit because they surface together on the same user path (wizard → research loop → first iteration) — you can't hit one without the other manifesting. Follow-up: start_topic still runs its own inline clone/spawn code (now duplicated with prepare_topic_runtime). Next commit collapses start_topic to just call prepare_topic_runtime + build_task like the loop path does, so the one-shot and loop paths agree on setup.	2026-07-10 07:38:52 -07:00
Omar Sobh	3e42b1ea39	research-wizard: schedule step (once/nightly/manual) + paired coding loop ... Completes the research/loop fold. The wizard now closes with a "How should this research run?" step; picking any mode materializes a kind='research' loop bound to the topic, and an optional checkbox adds a paired kind='exec' loop that consumes each new artifact. Frontend: - ResearchWizard grows from 5 to 6 steps. Step 6 is the schedule picker: · Just once — initial_burst=1, no other triggers · Nightly — initial_burst=1 + cron "0 3 * * *" · Manual — webhook_enabled=true Below the radios, an optional card offers "Also create a coding loop that consumes each new artifact" — creates a paired kind='exec' loop with on_artifact_update=true + initial_burst=1 bound to the same topic. - createTopic API type extended with `schedule` + `create_paired_coding_loop`. - Both fields ride the existing POST /api/research call; back-compat is preserved when the wizard omits them. Backend: - CreateTopicRequest gains TopicSchedule + create_paired_coding_loop. - After topic + agent attach, create_topic calls materialize_topic_loops which: 1. Creates a research-kind loop titled "Research · <topic>" bound to the topic. Triggers vary by schedule mode; next_fire_at computed from cron for nightly. Falls back silently if loop-create errors so the topic still lands. 2. Flips kind to 'research' via loops::set_kind (NewLoop doesn't take kind directly — default is 'exec' for backward compat). 3. Optionally creates a coding loop titled "Coding · <topic>" with on_artifact_update=true + initial_burst=1. - cm_db::repo::loops::set_kind — trivial UPDATE helper used by the materialize path. D-answer callouts: - D1 (fold): every runnable thing is now a loop. "Just once" is a research loop with initial_burst=1 and no other triggers. - D2 (inherit): both paired loops carry the same source_research_topic_id — repo binding lives on the topic, not duplicated. - D3 (coordinator resolves): the research iteration prompt (from the earlier commit) instructs the team to preserve stable INT ids and mark deprecations; coding loops' consumed lists stay valid across versions. Follow-ups queued: - Kind pill on LoopsList cards (research=purple, exec=cyan) so users can tell them apart at a glance. - Extract start_topic's task-build so kind='research' iterations reuse the same coordinator prompt shape as one-shot runs (they currently use a simpler refresh-oriented prompt; that's fine for MVP but a rich shared build would give better parity). - Research topic sidebar shows "linked to N loops" badge.	2026-07-09 23:00:04 -07:00
Omar Sobh	91a51dce11	loops: kind='research' dispatch — research runs as loops ... Delivers the research/loop fold: kind='research' loops run the research pipeline each iteration, appending a new research_outcomes version. The paired on_artifact_update fan-out then wakes up any kind='exec' loops bound to the same topic to consume new INTs. Every runnable thing is now a loop (D1). Backend — DB helpers: - cm_db::repo::loops::kind_and_binding — reads (kind, source_topic, task_template) so callers can dispatch without hydrating the whole Loop struct. - cm_db::repo::loops::enqueue_iteration_with_topic — new variant that sets research_topic_id on topology_runs alongside loop_id, so the completion hook's freeze_research_outcome writes a new outcome version for research-kind iterations. - cm_db::repo::research_topics::get_any_workspace — cross-workspace fetch used by the research task builder (the loop row is authoritative for the workspace binding via kind_and_binding). Backend — dispatch: - routes::loops::compose_research_iteration_task — builds the coordinator prompt for a research iteration: topic title + description + outcome_kind + prior version pointer + refresh instructions (survey new sources, preserve stable INT ids, mark superseded items as deprecated rather than delete). The completion hook writes the resulting synthesis as research_outcomes v(prior+1). - routes::loops::compose_and_enqueue_iteration — one-shot dispatch: reads the kind, picks compose_iteration_task (exec) or compose_research_iteration_task (research), enqueues with or without research_topic_id set. All four enqueue callsites now route through compose_and_enqueue: - create_loop (initial_burst) - run_now - webhook_receive - topology_worker::continue_initial_burst - topology_worker::freeze_research_outcome (on_artifact_update fan-out) Research-kind loops naturally form the "nightly refresh" side of a paired research + coding loop: research writes a fresh outcome version → fan-out wakes exec loops with on_artifact_update → coding loops consume the next INT (which the research loop may have just added). D2 answer (inherit repo binding): repo lives on the topic; both loops sharing the source topic id read from the same context, no duplication. D3 answer (coordinator resolves): the research iteration prompt tells the team to preserve stable INT ids and mark deprecations rather than delete, so coding loops' consumed lists stay valid across versions. Follow-up (next commit): ResearchWizard schedule step — "Just once / Nightly / Manual" that creates the paired research-kind loop with initial_burst=1 (just once) or cron 0 3 * * * (nightly) + optional paired coding loop with on_artifact_update.	2026-07-09 22:56:19 -07:00
Omar Sobh	4ada5557f2	loops: kind column + initial_burst + on_artifact_update trigger fan-out ... Foundation for folding research into loops as a first-class kind. This commit ships the plumbing; the research-kind dispatch itself lands next. Behavior for existing exec-kind loops is unchanged unless they opt into the new trigger fields. Migration 0044: - kind TEXT NOT NULL DEFAULT 'exec' CHECK ('exec' | 'research'). New research-kind will run the research pipeline each iteration (next commit); 'exec' preserves today's behavior. - initial_burst_remaining INT NOT NULL DEFAULT 0 — countdown for the triggers.initial_burst quota. Decremented CAS-safely on each completion until it hits 0. - Two partial indexes: (kind, source_research_topic_id) for kind- aware lookups, and (source_research_topic_id) filtered on on_artifact_update=true + enabled=true for the fan-out hook. Trigger schema extended with two optional fields: - initial_burst: N — fire N iterations back-to-back at create time. create_loop enqueues the first iteration inline (subject to empty-roster gate), sets remaining=N-1, and the completion hook continues the chain until exhausted. - on_artifact_update: true — when a bound research_outcomes row is inserted for the source topic, wake up one iteration of this loop. Coalesced against has_active_run so a burst of rapid revisions doesn't queue duplicates. Backend: - cm_db::repo::loops helpers (all dynamic sqlx, no offline cache regen needed): - set_initial_burst_remaining - take_initial_burst_slot (CAS UPDATE returning prev value; 0 on exhausted or race loss) - loops_awaiting_topic (fan-out query: kind=exec + enabled + on_artifact_update=true bound to the given topic) - has_active_run (queued|running iteration existence check) - get_any_workspace (bypasses the workspace scope guard; used by the completion hook where the run row is authoritative) - routes/loops::compose_iteration_task made pub so the completion hook can build the same enriched task string as run_now. - topology_worker::freeze_research_outcome now fans out to awakened loops after the outcome insert, using compose_iteration_task and coalescing on has_active_run. - topology_worker::continue_initial_burst runs on every completion: · take_initial_burst_slot (CAS) — no-op if already exhausted · has_active_run coalesce guard · re-fetches the loop via get_any_workspace + compose_iteration_task · enqueues via loops::enqueue_iteration with parent_run_id set Follow-ups already queued: - kind='research' dispatch in run_job — build the research coordinator task from the topic config, run the research pipeline each iteration. Requires factoring start_topic's task-build. - ResearchWizard "When should this run?" step (Just once / Nightly / Manual) creating the topic + paired research-kind loop. - LoopsWizard trigger UI matching the design proposal (burst count, cron, on-artifact checkbox).	2026-07-09 22:50:01 -07:00
Omar Sobh	ce111273bb	loops: reorder history collapsed under the progress pill ... Completes the reorder rationale loop — the previous commit captured REORDER: markers server-side but nothing surfaced them. Now the loops sidebar renders a small "N reorders ▸" button under the progress pill whenever the loop has any reorder events. Click expands to a newest-first list of "iter <n> · <rationale>" lines so a reviewer can see, at a glance, when the plan was adjusted and why. Backend: - LoopProgress DTO gains recent_reorders: Vec<Value> — newest-first, capped at 5 so the card stays compact. Full history remains on the loop row's reorder_events column. - cm_db::repo::loops::recent_reorders — reads the jsonb array, returns the last N in newest-first order. - list_progress populates it per loop. Frontend: - LoopReorderEvent + recent_reorders on LoopProgress type. - LoopsList tracks openHistoryId per-loop (one open at a time). - Card renders history button + expanded panel styled to match the progress pill above. Notes: - The event object schema is {run_id, iteration, text, ts}. Fields are optional in the TS type so future schema tweaks don't break the render. - 5-item cap chosen so the sidebar card doesn't grow unbounded. If a loop accumulates a lot of reorders, follow-up UI can render the full history on the loop detail page.	2026-07-09 19:00:39 -07:00
Omar Sobh	0c17de52dd	loops: reorder rationale extraction — REORDER: markers logged per iteration ... Coordinator can now log WHY it worked on an INT-XX out of order ("REORDER: INT-05 before INT-04 because prereq X is unmet") and the completion hook captures each rationale as an append-only event on the loop. Sets up a reviewable timeline of when the plan was adjusted, independent of the underlying `consumed_int_ids` advance. Migration 0043: - loops.reorder_events JSONB NOT NULL DEFAULT '[]'::jsonb — append- only array of {run_id, iteration, text, ts}. Kept on the loop row (rather than a dedicated table) so the mini-timeline is one read away from the loop card. Backend: - topology_worker::parse_reorder_rationale — line matcher symmetric with parse_completed_int_ids. Tolerates list dashes / prefixes / markdown emphasis; case-insensitive marker match, preserves case of the rationale text. - cm_db::repo::loops::append_reorder_event — one INSERT-like append per rationale, uses jsonb_build_object with postgres now() so ts is wall-clock canonical (no client-clock skew). - topology_runs::iteration_for_run — new helper so events carry the iteration index. - routes::loops::compose_iteration_task — coordinator prompt now explicitly asks for `REORDER: <one-sentence>` at the top of the first substantive turn when working out of order, AND spells out that both markers must appear literally with colons (no bold, no code fence) so the line parser doesn't miss them. Non-loop and standalone-loop runs are unaffected — the hook only fires when the run belongs to a source-bound loop. Follow-up: expose reorder_events on the loops list endpoint + render a small collapsed timeline on the LoopsList card.	2026-07-09 18:58:55 -07:00
Omar Sobh	a34be33261	loops: N/M INTs progress pill on source-bound loop cards ... Users couldn't see how a research-bound loop was progressing through its integration plan — the consumed_int_ids state existed in the DB but nothing surfaced it. Now each loop card in the sidebar shows a cyan pill "3/8 INTs" + source topic title + a thin progress bar, when the loop is bound to a research topic. Backend: - GET /api/loops/progress — bulk read for every loop with a source topic. Returns {loop_id, source_topic_id, source_topic_title, source_outcome_version, consumed_count, total_int_count, current_int_index} per loop. Standalone loops are omitted. - count_int_ids parses unique INT-<number> ids out of the source outcome's markdown — same permissive matcher as the completion hook, so what the pill counts matches what the loop can advance. - Memoized by topic_id inside the endpoint so N loops sharing 1 source topic only fetch the outcome once. Frontend: - listLoopProgress helper + LoopProgress type in the loops API. - LoopsList fetches loops + progress in parallel on mount. - Each card looks up progress by loop_id and, when found, renders under the schedule line: pill "3/8 INTs · Topic title" with a 3px cyan progress bar. Title hover shows artifact version. Follow-ups: - Refresh button on the card — re-snapshot artifact into task_template (cosmetic; the enqueue path already reads latest). - Reorder rationale extraction — parse "REORDER: <text>" out of run output, index as a per-loop event log for a mini-timeline UI.	2026-07-09 18:57:09 -07:00
Omar Sobh	4a140cb7db	loops: parse COMPLETED: INT-XX markers to advance loop pointer (option b) ... Closes the loop bridge — the missing piece from the previous commit. Without this, `current_int_index` stayed at 0 forever and every iteration re-worked INT-01. Now the topology_worker's completion hook parses the run's final output for `COMPLETED: INT-<NN>` markers and atomically advances the loop's consumed_int_ids + current_int_index. Backend: - topology_worker::advance_loop_after_completion — new post-terminal hook that fires alongside freeze_research_outcome. Reads loop_id_for_run (skips non-loop runs) + source_research_context (skips standalone loops without a bound source topic). - parse_completed_int_ids — forgiving parser: matches `COMPLETED: INT-01`, `- COMPLETED: `INT-01``, `COMPLETED: INT-01, INT-02`, case-insensitive, tolerates list dashes / backticks / markdown emphasis. De-dupes within a single output. - cm_db::repo::loops::advance_after_completion — atomic UPDATE that: · appends only NEW ids to consumed_int_ids (idempotent on re-runs) · bumps current_int_index by the count of new ids landed Set semantics via `SELECT DISTINCT unnest(...)` so ordering-based bugs can't accumulate duplicates. Behavior end-to-end: 1. Loop wizard imports an integrations artifact (previous commit). 2. run_now / webhook_receive → compose_iteration_task prepends artifact + focus instruction ("address INT-<current+1>, log COMPLETED at end"). 3. Coordinator run does the work, emits `COMPLETED: INT-<NN>`. 4. topology_worker completion hook parses the marker, advances the loop, and the NEXT iteration sees an updated `consumed:` list + incremented `current_int_index` in its focus instruction. Follow-ups still queued: - Loop card refresh button — pull latest artifact after reject-with- revision on the source topic (right now the prepend uses the LATEST outcome automatically, so refresh is UX only, not correctness). - Reorder rationale extraction — coordinator emits "REORDER: INT-05 before INT-04 because prereq X is unmet"; today that's just prose in the output, not indexed.	2026-07-09 18:48:34 -07:00
Omar Sobh	ce73abe5ab	loops: bridge research artifact into loop iterations (option C + b) ... The bridge lets a coding loop "consume" an integrations research artifact one INT-XX item per iteration. Options b (order-sequential iteration) and C (snapshot in task_template + save the pointer for future refresh) from the design discussion. Migration 0042 — three new loops columns: - source_research_topic_id — nullable pointer to research_topics. - consumed_int_ids TEXT[] — INT-XX ids the loop has completed. Advances when topology_worker parses "COMPLETED: INT-<NN>" markers from the run's final output (wired in a follow-up commit). - current_int_index INT — monotonic pointer for order-sequential iteration. Coordinator addresses INT-<current+1> unless prereqs are unmet, in which case it works on the smallest unblocking INT-XX and logs the reorder rationale. Backend: - cm_db::repo::loops::set_source_research_topic — bind/unbind pointer. - cm_db::repo::loops::source_research_context — read pointer + state. - routes::loops::compose_iteration_task — new caller-side helper that reads the pointer, fetches the topic's latest research_outcome, and prepends the artifact + focus instruction to task_template. - run_now + webhook_receive both pass task_template through compose_iteration_task before enqueue. Standalone loops (no pointer) behave identically to before. - CreateLoopRequest accepts `source_research_topic_id`, ownership- checked via research_topics::get before persist. Frontend: - New ResearchArtifactPicker modal — lists published topics, fetches the artifact on pick, returns (topic_id, markdown) to caller. - LoopsWizard task_template step gains "Import from research artifact" button (right-aligned). Click opens the picker. On pick: task populates with the artifact markdown, pointer saved, textarea expands to 8 rows, small info strip shows "Loop is bound to topic <id>. Each iteration will focus on the next unconsumed INT-XX." - Unlink button reverts to standalone loop mode. Follow-up (next commit): - topology_worker completion hook — parse "COMPLETED: INT-<NN>" out of the run's final output + update consumed_int_ids + current_int_index atomically. Without this, current_int_index stays at 0 forever and every iteration works on the same INT. - Loop card refresh button — re-read source topic's latest outcome (useful after a reject-with-revision cycle on the source topic).	2026-07-09 18:47:17 -07:00
Omar Sobh	3ed1d03d2b	research: integrations outcome + rich wizard cards + coordinator template ... Adds a fifth outcome kind ('integrations') tuned for the "audit repo, survey papers, propose a menu of concrete integrations" use case. Every INT-XX item is self-contained (what, how, where, prereqs, effort, risk, testing, rollback, acceptance) so a downstream loop can execute one per iteration. Backend: - Migration 0041 drops + re-adds the outcome_kind CHECK constraint with 'integrations' allowed. Existing rows unaffected. - VALID_OUTCOMES gains 'integrations'. - New deliverable_template(kind) returns the canonical section shape for each outcome — spec, prod_plan, roadmap, paper, integrations all get first-class treatment (prior: all shared a bare label). - build_coordinator_task injects an ARTIFACT SHAPE block from the template into the coordinator prompt, so the final synthesis actually matches the promise the wizard made. Frontend: - OutcomeKind gains 'integrations'. - ResearchWizard OUTCOMES list carries a `sections` array per kind. - Selected card renders an "ARTIFACT WILL CONTAIN" preview so users pick by seeing what they'll get, not by reading a one-line hint. - Integrations card gets the fullest preview (executive summary + INT-XX card shape) since it's the most structured deliverable. Follow-ups queued (next commit): loop wizard "Import from research artifact" bridge + one-INT-per-iteration mode.	2026-07-09 18:42:46 -07:00
Omar Sobh	3bcd18bf50	research: split pipeline diagnostics into research_pipeline.rs ... Gates step failed on run#211 because research.rs grew to 1313 lines (over the 1250 budget) after adding the pipeline_state handler in the previous commit. Move the diagnostic into its own module — pure extraction, no behavior change. research.rs: 1313 → 1126 lines research_pipeline.rs: new, 187 lines lib.rs: route now points at routes::research_pipeline::pipeline_state	2026-07-09 17:05:28 -07:00
Omar Sobh	cca6e2be20	research: pipeline diagnostics + refuse publish without outcome ... Two fixes surfaced by the first prod run of the pipeline: 1) Silent skip-to-published bug (R1 gap): approve_publish transitioned reviewing → publishing → published without checking that an outcome existed. Result: pipeline could fail silently (LLM auth error, network, etc), no outcome would be written, but state advanced to 'published' and the download endpoint returned 404 with no user-visible error. Now refuses with 409 Conflict when no outcome exists so the frontend can surface WHY. 2) No end-to-end visibility: Users had no way to see where a run failed until they clicked Download and got nothing. Adds GET /api/research/:id/pipeline-state — a read-only per-stage report walking: - staffing (agents assigned) - repo (bound + cloned) - container (per-topic team runtime spawned) - runs (count + failed count + latest error text) - outcomes (count — the artifact rows get_artifact reads) - approval (pending flag) Each stage returns ok / warn / fail / skip plus optional detail text so the failure reason surfaces at the diagnostic level. Frontend: ResearchCanvas shows a compact PIPELINE strip below the topic title, green/amber/red dots per stage, click to expand a full checklist with per-stage detail (including the LLM error from the last run attempt). Polls every 6s while the topic is processing/publishing. Follow-up: - Root cause of the specific failure just observed: Claude CLI in the clawmates-runtime container isn't authenticated. Deploy-side config sweep (CLAUDE_CODE_OAUTH_TOKEN or ANTHROPIC_API_KEY into the runtime image env), not a code fix. - Structured event stream on top of run_events for real per-step replay in the diagnostic panel.	2026-07-09 16:59:59 -07:00
Omar Sobh	44d6e95022	fmt: apply rustfmt across the P2 arc ... CI's rustfmt check flagged the multi-line sqlx::query() calls I introduced in P2 (loop_id_for_run, zeroclaw_gateway_url, etc.). No behavior change — pure formatting.	2026-07-09 16:43:05 -07:00
Omar Sobh	03f1830d1f	loops: Path B container isolation (P2) ... Symmetric with the research pipeline: every enabled loop can now have its own per-loop team container so scheduled runs don't share state with other loops or with research. Same daemon image, same clawmates network, deterministic name loop-<id>-team. Backend surface: - Migration 0040 adds nullable `zeroclaw_container` + `zeroclaw_gateway_url` columns to loops (parallel to research_topics). - research_container.rs grows loop_container_name_for(), spawn_loop() (state-only mount, no repo), and teardown_loop(). Kept in the same module to share the docker connect() + inherited_env() plumbing; each pattern gets its own labels (clawmates.role=loop-team) so ps filters can tell them apart. - cm_db::repo::loops gains set_zeroclaw_container() + zeroclaw_gateway_url() (dynamic sqlx queries — no offline cache regen needed). - cm_db::repo::topology_runs gets loop_id_for_run(): mirror of research_topic_id, used by the worker. Wiring: - routes/loops::run_now + webhook_receive call ensure_loop_container() before enqueuing an iteration. Idempotent: an already-running container is just reattached. Failures are logged and do NOT block the enqueue — topology_worker falls back to the workspace gateway when the URL isn't set on the loop. - routes/loops::disable_loop + delete_loop both fire teardown_loop() so paused / deleted loops don't hold a docker slot. - topology_worker's per-run URL resolution: existing research fast path unchanged; when it doesn't hit, the worker now looks up loop_id and reads the loop's gateway URL. Deploy step (required on gw-04 for state to persist across container restarts): add a `/var/lib/clawmates-loops:/var/lib/clawmates-loops` bind mount + `CLAWMATES_LOOPS_STATE_ROOT=/var/lib/clawmates-loops` env var to clawmates_server_1 in the compose. Without it, loops still run — the state dir lives inside the API container's filesystem so persistence is limited to that container's lifetime. Follow-up: - Scheduler-tick fires (cron-driven, not run_now) — they call enqueue_iteration in cm-scheduler and don't yet go through ensure_loop_container. Add a symmetric spawn there so cron fires also land on the isolated daemon. - Compose file reconciliation — deploy/compose/docker-compose.yml in the repo has drifted from prod; when we sync it, add the loops mount at the same time.	2026-07-09 16:02:08 -07:00
Omar Sobh	e3011ed025	research: reject-with-revision loop (R2) ... Before: reviewer rejected a publish → audit log flipped, topic stayed in reviewing, no way to feed the critique back into the run pipeline. Reviewers with revision notes had to eat them or hand-message the coordinator. Now: reject accepts an optional `notes` field. When present: - Persisted on the research_publish_approvals row (migration 0039). - Topic flips `reviewing → standby` so the next `start_topic` is legal. - `start_topic` reads the most recent rejected-approval notes for the topic and prepends "PRIOR REVIEW NOTES (address these in this revision):\n<notes>\n---" to the coordinator task. Loop closes through the same run pipeline — no new spawn code path, which means the reviewer's guidance flows through the same topology_worker, run_events, outcome-writer chain and lands as a fresh research_outcomes row (versioned, prior drafts preserved). No notes on reject = legacy behavior (topic stays in reviewing, publish requests still allowed). Migration 0039 adds nullable `notes TEXT` to research_publish_approvals. `decide()` gains a `notes: Option<&str>` parameter (only one caller, updated inline). New `latest_rejection_notes(pool, topic_id)` helper for start_topic. Frontend: - rejectPublish(id, notes?) now sends a JSON body when notes are provided. - ResearchCanvas reject button opens an inline form with a textarea + Cancel/"Send back for revision" pair. Empty notes → plain reject. - Button label switches: "Send back for revision" when notes present, "Reject without notes" when empty. Follow-up: - Notes shown in the review UI on the resulting draft so the next reviewer sees what changed. - Multiple rejection rounds — currently only the LATEST rejection's notes surface. Accumulating history is a schema-only tweak.	2026-07-09 15:56:16 -07:00
Omar Sobh	a49cf86623	world: pre-seed repo tree on clone (V3) ... Repo focus mode used to open into an empty tree — the file/dir nodes only synthesized as agents touched files via tool calls. Cold-start users saw a lonely repo:<id> orb with nothing under it. Now the tree pre-seeds from the actual cloned repo the moment a client subscribes. Backend: - active_research_topics also returns repo_workspace_path. - preseed_repo_paths(clone_path) runs `git ls-files` (bounded to top 200) against the on-disk clone. Silently returns empty on any failure so a missing clone / git-off-PATH / empty repo just degrades to the pre-V3 behavior (tree still builds on touch). - SSE loop, on first sight of a topic per client, emits one node.activity per pre-seeded path (label = leaf name, heat 0) so the tree is quiet-solid at rest. Frontend: - engine.onNodeActivity now synthesizes the dir:<partial> chain for file: nodeIds the same way onTouch does — otherwise the pre-seed would render as flat leaves under ROOT. - Same 5-line synthesis extracted from onTouch; both paths now agree on the layout. Cap of 200 keeps the SSE payload bounded on huge repos; the tail fills in as agents actually touch files. When we later add per-file heat map (V4), the 200 already-known files get first-class treatment out of the gate.	2026-07-09 15:49:43 -07:00
Omar Sobh	9e5034cb53	wizards: ensure-chain preflight in AddToTeam + AddToCompany (W1) ... TeamWizard already ran ensure-chain before /api/teams; the two "AddTo" modals didn't, so teams/companies created from them landed as structural orphans (no company/org parent). Same treatment now applied to both modals + their backend endpoints. Backend — two symmetric `attach_to_*_id` fields (mirrors what create_team already exposes): - ComposeTeamRequest gains `attach_to_company_id`. After team insert, create_team_from_claws binds it via companies::add_team with a fresh `n{count}` node id. - CreateCompanyRequest gains `attach_to_org_id`. After company insert, create_company binds it via orgs::add_company the same way. Both bindings are optional — plain POSTs from tools/tests still work. Ownership is re-checked via `<parent>::get(pool, id, workspace_id)` so the endpoints can't be tricked into parenting into another workspace. Frontend — both modals now: 1. POST /api/structure/ensure-chain (empty body → server picks "My Workspace" / "General" fallbacks when nothing exists yet). 2. Include the returned parent id in the create request. AddToOrgModal untouched — orgs are top-level, no parent needed. MasterPlannerModal untouched — it posts to /webhooks, doesn't create structural rows. Follow-up already queued in the original list: same treatment for the Company/Org "wizard"-flavored surfaces (as opposed to the compose modals). Currently those don't exist as distinct wizards.	2026-07-09 15:47:46 -07:00
Omar Sobh	f9e8d8d779	research: publishing → published + artifact download (R1) ... Before this commit, approve_publish left topics stuck in 'publishing' forever — the sidebar 'published' bucket was always empty and nothing surfaced the artifact. Two-part fix: State machine — approve_publish now transitions reviewing → publishing → published in one API call. Real async packaging isn't a thing yet because the artifact IS the markdown already written to research_outcomes when the last run completed (topology_worker). The intermediate 'publishing' state is preserved (schema-level trigger stamps published_at on landing there) so we keep the option to detour through it later for pdf render / mirror-to-store / etc. Download endpoint — GET /api/research/:id/artifact returns the latest outcome as text/markdown with Content-Disposition: attachment. Filename sanitizes the topic title to ascii-alnum + dash and appends the outcome version so accumulated revision drafts (post R2) don't clobber. Workspace ownership check via research_topics::get; 404 if no outcome yet (reviewers browsing before a run completes). Frontend — ResearchList shows a green Download icon-button next to Delete when status === 'published'. Clicks trigger a plain anchor download of the .md — no JS blob dance needed since the response is already an attachment. Follow-up queued: pdf render (server-side or client-side of the md), mirror-to-store (S3-ish or Obsidian vault) as an async step during the publishing→published detour.	2026-07-09 14:05:21 -07:00
Omar Sobh	f60df36717	research: teardown per-topic container on publish + delete (P1) ... Path B spawned a per-topic team container on start_topic (research runtime) but nothing ever stopped it. Containers accumulated on gw-04 across the lifetime of every topic — the earlier cleanup pass reclaimed 6.72 GB of these. Now the container is stopped + removed automatically at the two terminal transitions: - delete_topic — the topic row is gone, the container is meaningless. - approve_publish (reviewing → publishing) — the research work is done. Artifact writing (queued as R1) reads from the durable run_events log, so it doesn't need a live runtime. Wiring is a fire-and-forget teardown() helper in research_container.rs: - Connects Docker via the same socket-proxy shim as spawn(). - Calls the existing stop() which is idempotent (404 = already gone, 304 = already stopped, both treated as ok). - Errors log with eprintln! and don't block the API response — a topic is done whether Docker is reachable or not. Dev machines without Docker running just log a debug line and return. Path A (repo-only mount + shared clawmates_core network) is untouched so there's no Traefik dynamic-file cleanup needed. If we ever add per-topic Traefik routing, extend teardown() with the corresponding file remove. Follow-up: gw-04 will still have any legacy containers from before this commit. One-shot `docker ps -a --filter "name=research-" -q | xargs -r docker rm -f` on gw-04 is the manual sweep.	2026-07-09 14:02:50 -07:00
Omar Sobh	763b95a253	api: gate create_loop + create_topic on empty workspace roster (W2) ... Frontend already disables the create button when the roster is empty, but nothing stopped a direct POST from materializing an orphan loop / topic with nothing to staff it. Both handlers now check the workspace agent count up front and return 409 Conflict when it's zero. - crates/cm-api/src/routes/loops.rs: gate at top of create_loop - crates/cm-api/src/routes/research.rs: gate at top of create_topic Uses the existing cm_db::repo::agents::count_active(pool, ws) helper (count of workspace agents where deleted_at IS NULL). 409 is the right mapping: state-of-the-workspace-prevents-this, not client-input-bad.	2026-07-09 14:00:54 -07:00
Omar Sobh	5a2340587e	world: loop:<id> landmark orbs (V1) ... Mirror the repo: landmark pattern for scheduled loops. Every enabled loop in the workspace gets a labeled amber orb in the World, whether it's currently running or between fires. Assigned agents converge on it with a soft 0.15 touch — the loop is a persistent landmark, not a transient run. Backend: - active_loops(pool, ws) query joins loops + loop_agents where enabled, returning (loop_id, title, agent_id) — one row per (loop, agent). - SSE loop emits node.activity + world.touch symmetric to the research block. Seen-once set dedupes the label emission across agents. Frontend: - New "loop" tier in the Tier alias, LEVEL_COLOR (#f0b866 warm amber), and ensureNode radius (11 — same landmark size as repo). - engine.onTouch / onNodeActivity preserve the tier from the loop: prefix (previously would have collapsed to service). - Pawn fireColor tinted amber for loop: touches. - WorldCanvas: loop tier joins the struct group for solid-at-rest glow + always-on labels. Focus mode recognizes loop: prefix (click → focused subtree, Esc to exit). Focus pill switches to "LOOP FOCUS" in amber when the selected id is a loop. Contrast with repo: (transient — only appears when a topic is in processing/reviewing/publishing). Loops are persistent because their whole point is recurrence.	2026-07-09 13:58:47 -07:00
Omar Sobh	708f45d09b	world: solid team + project orbs, hide ROOT, per-topic repo landmarks ... Three fixes to the Live viz per user feedback: 1) Hide the ROOT sentinel — it was rendering at the origin as a red disc with no label. It's a physics anchor, not a real orb. Skipping it in nodes/edges/labels loops removes the mystery red circle. 2) Emit `repo:<topic_id>` project orbs for active research topics from the world SSE loop. Labeled with the topic title, tier "repo" gets its own soft sky-blue palette and a landmark radius (r=11 between company and team). Assigned agents get a low-weight (0.15) convergence touch so pawns cluster around their project's orb even at rest — no wait for a file op to see the affiliation. 3) Solid at rest, bloom on interaction. Two engine changes: - Structural + repo orbs now have a soft 0.08 glow at heat=0 (down from 0.22 ambient) so the disc reads as solid until agents heat it. - `world.touch` heat is now weight-scaled (`+w*0.6`) instead of a flat `+0.5` regardless of intent. Soft convergence stays soft; file ops still explode. Click a `repo:` orb → the existing Commit F focus mode already treats that prefix as a subtree root, so users drop straight into the Gource-style repo detail view with only the files their agents are touching. Follow-ups queued: teardown of repo orbs when a topic reaches 'published' (currently they persist until the SSE loop's status filter drops them, which is correct); loops equivalent (loop:<id> landmark orbs).	2026-07-09 13:22:27 -07:00
Omar Sobh	f88e8642d9	world viz: repo focus mode — click a file/dir/run to enter the tree ... MVP of the repo-detail sub-view. Click any file/dir/run/repo node in Live and the viz culls to just that node's subtree — you're now watching agents crawl the repo instead of the whole workspace. Esc (or the pill button that appears) exits. Backend - routes/world.rs: file-op tool events now emit a `file:<path>` world.touch IN ADDITION to the existing `tool:<name>` touch. The path is pulled from the tool input's path / target / file / filename / url keys (same lookup summarize_input uses, but we keep the full string so the client can build a real hierarchy). Non-file tools are unchanged — they still hit tool:<name> nodes as before. Engine - onTouch synthesizes a directory hierarchy when the id starts with `file:`. Each intermediate path segment gets a `dir:<acc>` node (label = the segment), parented at the previous dir; the file itself parents at the innermost dir. Ensures the layout spring-simulates as a tree naturally, no separate render mode needed. - New pawn fireColor for file: touches: coral #ff8a7a. Reads as "file work" vs #5ec8d8 (tool convergence) vs #5fd08a (run activity). WorldCanvas - Render pass now takes a `visibleNodes: Set<string> | null`. When the selected id starts with file: / dir: / run: / repo:, we BFS descendants via parentId and hide every non-descendant node. Node meshes, glow sprites, hierarchy edges, and labels all gate on the set. Pawns stay visible (agents still dart to the focused files). - ESC handler on window: clears the selection by calling onSelect("") when a repo-focus id is set. - Small "REPO FOCUS · <path>" pill lands at top-center with an Esc button so the exit is discoverable at a glance without learning the shortcut. - Dashboard.onWorldSelect now treats empty string as "clear focus" (setWorldSel(null)) so the same callback handles ESC. Not yet: the always-on repo:<topic_id> node emitted at run start when a research topic has a repo bound. Today the focus works off run:<id> nodes; a repo:<id> anchor would let users click without waiting for a first file touch. Also skipped: per-file heat map / call-count visualization tied to touch weight over time. Both are natural follow-ups on this bones.	2026-07-09 12:16:05 -07:00
Omar Sobh	aa941b72a1	world viz: agents grow with their brain (log-curve dot size) ... Fresh agents start at "size of their letters" — a small dot in the live viz — and visibly bloom out as their .brain file fills. Turns "which of these agents is heavily loaded" into a glance instead of a menu dive. Taxonomy - New agent.memory event: { agentId, bytes?, count? }. STATEFUL, so a late subscriber sees the last value replayed and pawns arrive pre-sized. count is included in the schema for a future combo metric but not emitted yet — bytes carries the visual today. Backend (routes/world.rs SSE loop) - Per-agent, per-tick std::fs::metadata() on brain_dir()/claw_<uuid>.h5. Just the inode stat — no HDF5 open, no memory count, sub-ms per agent. Emit agent.memory { bytes } only when the value has changed (or on first sight). - Tracks last_bytes: HashMap<String, u64> in the SSE-stream scope alongside the existing status HashMap. - Missing file (agent never provisioned a brain) reads as 0 bytes and yields scale = 1.0 downstream — pawn stays small. Engine - GPawn gains memoryScale (visible) + memoryScaleTarget (chased). Base is 1.0; ensurePawn initializes both. - memoryScaleFromBytes(bytes): 1 + log10(1 + bytes/1MB) * 0.6, cap MAX_MEMORY_SCALE = 3.5. So 10MB ~ 1.6x, 100MB ~ 2.2x, 1GB ~ 2.8x. Log curve keeps a heavy brain readable without a lite one being invisible. - onMemory(e) sets the target. stepPawns eases the visible scale toward it at ~3/sec — a big incoming snapshot doesn't pop the sphere; it swells in like it's inhaling. Renderer (WorldCanvas) - Live subscription registers agent.memory alongside the existing status/touch/reasoning listeners. - Pawn sphere scale = 5 * p.memoryScale (was hardcoded 8). Halo scales in proportion (max(24, 4.25 * s)) so a memory-heavy agent reads as a bigger presence, not a small dot with a huge halo. - AABB bounds for the frame-camera math updated to use s instead of 8 so the camera actually frames a big agent when it's the outlier. Not yet wired: comm lines between pawns when agents talk to each other (Commit E next), and the topology-edge overlay that renders the graph shape dimly at rest. Both build on top of this — bigger dots make comm beams more visible.	2026-07-09 11:52:45 -07:00
Omar Sobh	acd2a0f287	structure polish: post-reify nav + ensure-chain + TeamWizard auto-parent ... Two small quality-of-life fixes on top of the reify commit: Post-reify navigation OrphanMigrationDialog already returned team_id in its result; Dashboard now pushes /?team=<team_id> before router.refresh() so the user lands on the freshly-materialized team and sees exactly where their agents just moved. Previously they had to hunt for it in the newly-rebuilt sidebar. Wizard auto-materialize (POST /api/structure/ensure-chain) cm-db: ensure_chain(pool, ws, fallback_org, fallback_company) — fast path returns coordinates of the first org+company already bound in this workspace (workspace's oldest org, oldest company under it). Slow path inserts a new org+company with the fallback names ("My Workspace" / "General") + binds them via org_companies. Returns { org_id, company_id, created }. Small txn — leaves the workspace consistent whether it was already wired or not. cm-api: POST /api/structure/ensure-chain accepts optional fallback_org_name and fallback_company_name in the body (trimmed, else default). Returns the ids. CreateTeamRequest gains an optional attach_to_company_id. When set, after build_team() completes, we look up the company (workspace ownership check enforced by companies::get), count its existing teams for a stable n_i node id, and insert a company_teams binding — so the team lands under the parent atomically instead of a follow-up round-trip. TeamWizard now calls ensure-chain before POST /api/teams and passes the returned company_id in attach_to_company_id. Both calls are best-effort — if ensure-chain fails (network etc.) we still try to create the team, and the migration dialog stays available as the fallback UX. Wizard flow now: fresh workspace's first team is fully wired from the moment it appears in the tree — no synthetic "My Workspace" scaffolding ever gets rendered around it. The Team/Company create paths not touched here (create_team_from_claws, company create, org create, MasterPlannerModal scaffold) still work as before — they just won't auto-parent yet. Later commits can wire them the same way.	2026-07-09 11:41:27 -07:00
Omar Sobh	8b789beec0	structure: reify-orphans endpoint + "give these a home" dialog ... Turns the four synthetic tree containers into a real migration path. Clicking any of them ("My Workspace", "Teams", "Direct", "Ungrouped") opens a dialog that creates a real org → company → team chain and re-parents every orphan into it, all in one DB transaction. cm-db (new module structure_reify) - orphan_agents / orphan_teams / orphan_companies: workspace-scoped SELECTs of entities without a parent binding in team_members / company_teams / org_companies. Used both by the dialog's counter and internally by the migration. - count_orphans: cheap combined-count via three subqueries in a single SELECT so the dialog only round-trips once for the header. - reify_orphans(pool, ws, org_name, company_name, team_name): 1. begins a tx 2. inserts a new org + company + team (all `flat`, empty graphs — user can shape them later via the existing PATCH endpoints) 3. binds company under org (org_companies "n0") 4. binds team under company (company_teams "n0") 5. inserts team_members rows for every orphan agent (n1, n2, …) 6. inserts company_teams rows for every orphan team 7. inserts org_companies rows for every orphan company 8. commits, returns the created ids + moved counts cm-api (routes/structure) - GET /api/structure/orphan-counts → { agents, teams, companies } - POST /api/structure/reify-orphans → { org_id, company_id, team_id, moved_* }. Trims + rejects any empty name; validates before starting the transaction so a 400 never rolls anything back. Frontend - New OrphanMigrationDialog: fetches counts on open, three name fields (defaults: Organization "My Workspace", Company "General", Team "Everyone"), POSTs on save. "Nothing to migrate" state disables the save button when the workspace is already fully wired. Copy explicitly notes that everything is renameable in the sidebar afterward. - Dashboard: onTreeSelect now branches on SYNTHETIC_TREE_IDS — clicking a synthetic node opens the dialog instead of falling through to the (nonexistent) selection. On successful reify, router.refresh() so the sidebar + world viz reflect the new real chain. What this doesn't do yet (next commit) - Wizard auto-materialize: when creating a team/company via wizard, auto-create parent placeholders if they don't exist. Deferred so this commit stays focused.	2026-07-09 11:26:07 -07:00
Omar Sobh	99e5207e69	sidebar: click-to-rename org/company/team + strip synthetics from world viz ... Two related pieces of the "kill My Workspace" cleanup, landed together because they share the same file: Backend - Three tiny inline-rename endpoints: PATCH /api/orgs/{id}/name PATCH /api/companies/{id}/name PATCH /api/teams/{id}/name Each takes { name: string }, trims + rejects empty, returns 204. Backed by rename_org / rename_company / rename_team in cm-db — single-row UPDATEs scoped to the caller's workspace, NotFound if the id isn't visible. - Registered next to the existing PATCH /:id (topology) routes so they don't collide. Frontend - StructureTree accepts an optional onRename and canRename. TreeRow: click on the label text of a renamable node → the span becomes an <input>, focus + select-all, save on Enter or blur, cancel on Escape. The rest of the row (row chevron / row body) still navigates + selects as before, so single-click behaviour is preserved for everything except the name text itself. react-hooks/set-state-in-effect avoided by resetting the draft in the enterEdit() click handler instead of inside a useEffect. - Dashboard passes canRename={item.level !== "claw" && !synthetic} (claws don't have a rename endpoint yet; synthetic scaffolding gets reified into real rows in the next commit — the wizard auto-materialize + orphan-migration dialog). onRename fires the corresponding PATCH and calls router.refresh() so the label lands in every consumer of the tree. - World viz seed: new stripSynthetics(roots) helper walks the tree and lifts children of any synthetic container up to their grandparent's level. worldCanvasRoots feeds through this before narrowRoots(). Result: the Live viz no longer shows "My Workspace" or "Teams" nodes — real agents orbit the world root directly (which is what you were asking for). Sidebar tree still shows them so orphaned agents remain visible until the migration lands.	2026-07-09 11:18:52 -07:00
Omar Sobh	88c78bd16e	research: topology_worker points executor at per-topic gateway (commit 2/3) ... Commit 2 of the path-B plan. The container that commit 1 spawns now actually receives the run's turns — up until now it was started but unused. This is the payoff commit: research runs are truly isolated per topic. Backend - topology_exec.rs: from_env() refactored to a thin wrapper over a new from_env_for_gateway(url) helper. Same shape (env-derived aliases + default + token) but the caller supplies the URL. The auth token/pairing code still comes from ZEROCLAW_TOKEN / ZEROCLAW_PAIRING_CODE on the server; research_container's inherited_env propagates those into the team container so the same credentials work at both endpoints. - research_container.rs: new wait_ready(url, deadline) that polls <url>/health with a 1.5s per-request timeout every 500ms until it 200s or the deadline passes. reqwest-based so it doesn't need bollard. Called by the worker after claim, before pair, to bridge the "container is starting, gateway not yet listening" gap. - topology_worker.rs run_job: 1. Look up research_topic_id for the claimed run. 2. If Some, load the topic and read zeroclaw_gateway_url. 3. If a URL is present: - best-effort wait_ready(url, 30s); a timeout logs but doesn't abort — the pair call below will just fail faster than pinging forever - build the leaf via from_env_for_gateway(url) Else fall back to from_env() (workspace-wide gateway). 4. The rest of run_job is unchanged — the leaf drops into either SubTopologyExecutor (org/company) or direct drive (team tier) as before. What now works end-to-end Starting a research topic with a bound repo: 1. clone-shallow into per-topic workspace 2. docker create + start the clawmates-runtime container, name = research-<topic>-team, joined to clawmates_core so the server reaches it by name 3. persist container name + gateway URL on the topic row 4. enqueue the topology run tagged with research_topic_id 5. worker claims → looks up the topic → waits for the team gateway's /health → constructs a from_env_for_gateway executor pointed at http://research-<topic>-team:42617 6. every turn's `/ws/chat?agent=…` hits the isolated container; agents inside see the repo at /workspace/repo (rw); each topic's memory/state lives under its own /zeroclaw-data mount Deploy prereqs (unchanged from commit 1) - clawmates_server compose service needs a bind-mount of CLAWMATES_RESEARCH_WORKSPACE_ROOT so the paths spawn() writes to are visible on the host and the spawned team container mounts the same underlying data. - socket-proxy ACL needs POST + DELETE on /containers (prod ✓).	2026-07-09 04:32:40 -07:00
Omar Sobh	21ac35c8d4	research: spawn per-topic ZeroClaw team container on start (commit 1/3) ... Commit 1 of the path-B (real per-topic isolation) plan. The container spawns and its coordinates persist — nothing talks to it yet; commit 2 wires ZeroClawDriveExecutor to prefer the topic's URL when populated. This split keeps each landing verifiable. Backend - Migration 0038: research_topics gets zeroclaw_container_name + zeroclaw_gateway_url columns. Both nullable so a topic can exist before a spawn and teardown just NULLs them out. - cm-db: ResearchTopic struct extended; get/list SELECTs updated; new set_zeroclaw_container(id, workspace_id, name, url) helper used both for spawn (Some/Some) and teardown (None/None). - cm-api: bollard added as a workspace dep (matches cm-sandbox's version). New research_container module: · connect() → uses DOCKER_HOST when set (prod's socket-proxy at tcp://socket-proxy:2375) else the local socket. Same pattern cm-sandbox already uses. · container_name_for(topic_id) → "research-<uuid>-team" (deterministic so a re-start reattaches to the same container instead of orphaning it). · inherited_env() → propagates ZEROCLAW_*, OPENAI_*, ANTHROPIC_*, GEMINI_*, GROQ_* from the parent server env (provider config + tokens), stripping the server's own ZEROCLAW_GATEWAY_URL/WORKSPACE so the team runtime doesn't loop back on itself. Appends ZEROCLAW_GATEWAY_PORT=42617 and ZEROCLAW_WORKSPACE=/zeroclaw-data/workspace for the team's own listener. · spawn(docker, topic_id, repo_host_path, state_host_path): - inspect: if the container already exists, start it if stopped and return its coordinates (idempotent restart). - else create with: image = CLAWMATES_RESEARCH_TEAM_IMAGE or clawmates-runtime:latest cmd = [daemon, --host, 0.0.0.0] env = inherited_env() mounts = repo_host_path → /workspace/repo (rw) state_host_path → /zeroclaw-data (rw) network = CLAWMATES_RESEARCH_TEAM_NETWORK or clawmates_core labels = clawmates.role=research-team, clawmates.research.topic_id=<uuid> - creates state_host_path first so bind doesn't ENOENT. · stop(docker, name) → stop + remove. Idempotent on 404/304. - start_topic wires spawn after the clone completes: · state root = CLAWMATES_RESEARCH_WORKSPACE_ROOT / <topic> / state · on success, persists (name, url) on the topic row so commit 2 can look them up when constructing the executor · every failure (docker connect, docker create/start, DB persist) is best-effort: logs and continues. A missing team container leaves the topic pointing at the workspace-wide gateway URL (env), preserving prior behavior. Deploy prerequisites (not in this commit) - The compose stack's clawmates_server service needs bind-mounts of CLAWMATES_RESEARCH_WORKSPACE_ROOT (e.g. /var/lib/clawmates-research:/var/lib/clawmates-research) so paths the server writes to are visible on the host and the spawned team container mounts the same underlying data. - socket-proxy's ACL must allow POST + DELETE on /containers (already the case in prod per the audited compose file).	2026-07-09 04:14:26 -07:00
Omar Sobh	7984e65174	research_topics::create: bag 8 args into a NewTopic struct (fixes clippy) ... The prior signature took (pool, workspace_id, title, description, outcome_kind, topology_kind, repo_id, created_by) — 8 args, one over the clippy::too_many_arguments ceiling and blocking CI. Refactor to a NewTopic<'a> input struct mirroring the NewLoop / NewSubTopology pattern the codebase already uses. Wizard-side additions like a repo commit branch land as struct fields instead of cascading into every call site.	2026-07-09 03:52:59 -07:00
Omar Sobh	3465bb7a6d	research: persist bound repo + shallow-clone on start_topic ... This is the minimum viable version of the "agents actually work on a repo" architecture. Full vision (isolated ZeroClaw container per topic, dynamic agent provisioning inside, pause/resume, commit gate) is real weeks of work — this closes the first, most-visible gap so the ClawHDF5 topic can actually run against its codebase. Backend - Migration 0037: research_topics gets repo_id UUID (nullable, FK to repos ON DELETE SET NULL) and repo_workspace_path TEXT for the on-disk checkout location. Index on repo_id when set. - research_topics::create takes repo_id: Option<Uuid>. get + list select it and repo_workspace_path. set_repo_workspace_path persists the path once the first clone lands. - CreateTopicRequest accepts `repo: Option<TopicRepoRef>` — the same denormalized shape the wizard already sends. Only repo_id is authoritative; other fields are ignored (dead_code-allowed so serde still deserializes the full body). - start_topic branches on topic.repo_id. When set, it calls ensure_repo_workspace: · resolves repo.clone_url + repo.default_branch · target path = CLAWMATES_RESEARCH_WORKSPACE_ROOT // <topic_id> // repo (defaults under $TMPDIR) · runs `git clone --depth 1 --single-branch --branch <b>` via tokio::process. Reuses the checkout if .git already exists. · persists the path so re-starts skip the clone · runs `git ls-files` to sample the tree (first 60 entries, total count reported honestly so the prompt doesn't lie about coverage) All best-effort — a clone failure logs but still starts the run without repo context rather than aborting. - build_coordinator_task takes Option<&RepoContext>. When present, the framing gets a REPO block (slug / path / branch / file sample) and a USING THE REPO section instructing the coordinator to ground every recommendation in a concrete file reference and never fabricate paths. The per-topology bodies are unchanged — the repo guidance sits above them so it applies to every shape. What this unblocks / doesn't unblock Unblocks: The coordinator prompt now knows the repo exists, where it lives on disk, and what's in it. Even without file-editing tools wired to the checkout, the coordinator can point spokes at concrete modules and the final artifact can reference real files. For a spec-shaped outcome like ClawHDF5's, that's the difference between abstract advice and a spec grounded in the actual crates. Does NOT unblock: The agents themselves editing files, running tests, or committing. That requires either mounting the checkout into the ZeroClaw sandbox or exposing a new MCP tool for repo-scoped file ops — separate follow-up.	2026-07-08 21:38:00 -07:00
Omar Sobh	7c1af2e070	research: pipeline-running signal + spinners so users aren't guessing ... The prior flow was ambiguous: after hitting Start research, status flipped to "processing" and a "Submit for review" button appeared immediately with no indication that anything was actually running. Users had to guess whether the pipeline was working or stalled. Backend surfaces the truth as a signal: - new topology_runs::active_runs_for_research_topic counts queued+running runs whose research_topic_id matches - TopicDetail includes runs_in_flight: i64 alongside the existing status field, so the canvas can distinguish "pipeline still working" from "runner stalled". ResearchCanvas is now honest about state: - while runs_in_flight > 0, the header status pill grows a cyan "N runs in flight" badge with an inline SVG spinner - the stage-explainer card turns cyan-bordered and shows a "pipeline is running" hint, plus copy pointing the user at the Agents tier where each teammate's activity streams live - the "Submit for review (manual)" button is HIDDEN while any run is in flight — it's an escape hatch for stalled runs only, not the happy-path action. It reappears if runs_in_flight drops to zero but the topic is still marked processing, so a stalled runner can still be nudged along. - the canvas polls getTopic every 4s while status is processing/ publishing or runs_in_flight > 0, so the spinner + outcome swap in automatically when the pipeline completes. ResearchList sidebar: - each row's status dot becomes a spinner when the topic's status is processing or publishing, matching the canvas at a glance - the list also polls every 6s while ANY topic is active, so transitions land in the sidebar without waiting on a parent bump. The poll is gated on a derived boolean to avoid effect thrash. Follow-up: same pattern belongs on LoopsList / LoopsCanvas for loop iterations in flight — same signal (queued+running runs per loop) but not wired here.	2026-07-08 18:36:34 -07:00
Omar Sobh	316cdbf929	research sidebar: delete-with-confirm per topic ... Mirror the row-level delete affordance the loops sidebar already has. Loops was wired earlier; research had a bare title-only card with no way to remove a stale topic. - cm-db: research_topics::delete cascades via existing FK rules (research_topic_agents, research_publish_approvals, and the new research_outcomes all CASCADE on topic_id; topology_runs's research_topic_id back-ref is SET NULL so historical runs stay). - cm-api: DELETE /api/research/{id} → 204. Idempotent. - Frontend: deleteTopic helper. ResearchList row is now a card with the existing title/status/outcome header plus a trash icon that flips the card into an inline "Delete topic + all outcomes?" confirm strip. Confirm → red Delete / gray Cancel. If the deleted topic was selected, selection clears; local counter bumps the list refetch without waiting on a parent.	2026-07-08 17:21:51 -07:00
Omar Sobh	a2d3d85ebe	research pipeline v2: topology-aware start + persisted draft ... Three connected changes that turn "Start research" from a status flip into a real pipeline that produces a reviewable artifact: - Migration 0036: adds research_topics.topology_kind (default 'hub_spoke') and a new research_outcomes table (id, topic_id, version DESC, body_md, produced_by_run_id, created_at) so each run's final synthesis is versioned and persistent. - Wizard now has a topology picker in the Outcome step — hub_spoke / pipeline / hierarchical / star_moe — with copy that steers users to the right shape (Pipeline for research → distill → analyze → implement rosters, hub_spoke for the coordinator- and-specialists default). - start_topic reads the chosen topology_kind, parses it into a cm_topology::TopologyKind, and dispatches a per-shape coordinator prompt via build_coordinator_task. Pipeline explicitly tells stage 1 not to write the final artifact and propagates a "final stage MUST emit a complete markdown document with measurable acceptance criteria" instruction downstream. The graph builder is called with the topology the user actually picked instead of hard-coded HubSpoke. - topology_worker::freeze_research_outcome fires after every successful complete(). It looks up research_topic_id on the run; if set and final_output is non-empty, it inserts a new research_outcomes row (version auto-derived server-side via coalesce(max(version), 0) + 1). Best-effort — a DB hiccup logs but doesn't fail the run. - TopicDetail now includes topology_kind and latest_outcome. ResearchCanvas swaps in the outcome's body_md (rendered as pre-wrap markdown, versioned header, produced-at timestamp) whenever an outcome exists; the original prompt collapses into an "Original prompt" <details> below so it's still one click away. Pre-run topics still show the description as before. Follow-ups still open: reject-with-revision loop feeding the coordinator, publishing → published transition + real artifact export (md / pdf), and an approvals inbox surface for reviewers.	2026-07-08 17:13:46 -07:00
Omar Sobh	7e2b02d8bb	research: wire start_topic to actually run the pipeline ... The prior start_topic only flipped the status column — no work was enqueued. Now clicking "Start research" actually launches the assigned agents through the orchestrator. - start_topic loads the topic + its research_topic_agents, picks a coordinator (first slot with role_slot containing "coordinator"; else the first slot), and swaps it to index 0. - Builds a hub_spoke topology graph via cm_topology::build with roles = [coordinator, spoke1, spoke2, …]. hub_spoke wires edges from the hub to every spoke and back, so the coordinator can address any specialist per turn. - Assembles a coordinator prompt from the topic's title, description, outcome_kind, and a roster line for each teammate — so the coordinator knows who's on the team and what each does. - Enqueues via a new topology_runs helper enqueue_run_for_research_topic that stores research_topic_id on the run row. `topology_worker::maybe_transition_research_topic` → `notify_run_completed` already picks up on that back-ref and flips the topic processing → reviewing when the last run terminates — that path was dead code until now. - Per-agent activity streams into each claw's card for free: the orchestrator journals turn events into run_events; the existing /api/world/live SSE normalizer emits agent.reasoning.delta / agent.tool.call / agent.task.update keyed by agent id, which ClawCommandCenter is already subscribed to. The `published` terminal state is still unreached (that's the "publishing → published + artifact" step from the earlier walkthrough — separate follow-up).	2026-07-08 16:25:24 -07:00
Omar Sobh	81a436d221	research canvas: wider rail + stage explainer + 409 fix ... Three connected fixes from a single session's feedback: - Structure rail widened from 60→76 px, tabs 42→58 px wide with a bit of left padding so labels ("Research", "Visualizations") no longer bump against the active-tab indicator strip. - Research canvas: each stage now shows a small "STAGE · <status>" card explaining what state the topic is in and a "Next → …" hint describing what the primary button will do. No more guessing which of standby/processing/reviewing/publishing means what. - Request-publish 409 fix: - Backend TopicDetail now includes has_pending_publish_request (SELECTs pending_for_topic when the topic loads). Frontend TopicDetail interface + ResearchCanvas honor the flag: when an approval is already pending the "Request publish" button is replaced with an amber "Awaiting reviewer approval" pill, so double-clicks can't 409 in the first place. - runAction() also catches 409 as a signal-of-success (the user's intent — "queue for approval" — is satisfied by the first attempt), refetches the topic, and lets the new awaiting-approval card render instead of surfacing a scary error to the user.	2026-07-08 15:59:04 -07:00
Omar Sobh	92e923e585	planner: specialists mode → single agent (backend + UI polish) ... The frontend already renamed the "Specialists" tab to "Agent" and rewrote the intro to ask for one specialist, but the backend prompt was still telling Opus to propose "2 or 3 domain specialists". Two-facing message got confusing outputs. - SPECIALISTS_NOTE now says "SINGLE agent … `members` MUST contain EXACTLY ONE entry … topology_kind='flat' … team_name reads like a personal handle." Removes the ambiguity. - TEAM_NOTE range aligned to "4 to 6 agents" (matches the UI hint and intro). - Frontend right-panel now reads "PROPOSED AGENT" for specialists mode (was "PROPOSED TEAM") and pluralization matches the count ("1 agent" vs "3 agents"). Build button switches to "Build agent" in that mode.	2026-07-08 15:04:00 -07:00
Omar Sobh	984d9a1274	reap: cascade orgs → companies → teams → agents ... Selecting a team/company/org in the Agents sidebar used to only delete the grouping row; the agents inside survived, ungrouped. Not what the user wanted, and it left a trail of orphaned runtime state (containers, .brain files, DB rows) behind. Backend — POST /api/claws/batch-delete is now a universal cascade reaper. Body accepts { ids?, teams?, companies?, orgs? } in any combination. The server walks org → companies_of_org → teams_of_company → agents_of_team, dedupes against explicit ids, and hard-purges every unique agent (deprovision ZeroClaw runtime, tear down sandbox container, unlink .brain/.onion files, transactional agents::hard_purge). Group rows are deleted last; FK cascades on team_members, company_teams, org_companies, and loop_agents/teams/orgs clean up the join tables. Every stage streams SSE. Three new cm-db helpers wire the walk: agents_of_team, teams_of_company, companies_of_org — all DISTINCT selects on the existing join tables. Frontend — Dashboard's Agents-tier StructureTree now sets selectLevel="*" (was "claw"), so the Wrench → checkbox affordance appears on org/company/team/agent nodes alike; the same-level invariant in onToggleSelect still prevents mixed batches. ReapProgressModal collapses to a single POST regardless of kind — body key derived from kind — and its subtitle is honest: "Cascading through every agent inside — permanent."	2026-07-08 10:36:52 -07:00
Omar Sobh	a6da19430f	loops: repo picker + agent/team/org staffing + sidebar edit/delete ... Adds the missing pieces the wizard needed and the sidebar controls around it: - LoopsWizard is now a 6-step flow (identity → repo → task/topology → triggers → repeat → assign agents) plus the existing secrets card. ResearchWizard picks up the same repo step and a hard gate when the workspace has zero agents. - New LoopStaffingStep with three tabs — Individual / Team / Organization — that mix freely per loop; selections persist via new loop_agents / loop_teams / loop_orgs join tables (0035 migration), each cascading on loop_id so hard-delete stays a single-row DELETE. - Backend CreateLoopRequest / UpdateLoopRequest accept the three lists and apply_staffing does a transactional replace-all; list_loops / get_loop hydrate the lists via a flattened LoopWithStaffing response. - LoopsList sidebar gains per-row enable/disable, edit (reopens the wizard prefilled with the current loop, PATCHes on submit), and delete with an inline confirm. - NoAgentsGate blocks launching a loop or research topic from a workspace with no roster; the sidebar `+` buttons also disable with a tooltip pointing at the TEAM tier. Not yet wired: the run driver still fills role slots from the workspace-wide pool; teaching enqueue_iteration to prefer loop_agents/loop_teams/loop_orgs is a follow-up.	2026-07-07 22:09:06 -07:00
Omar Sobh	637e1bdd69	repos: sidebar actions (sync/edit/remove) + edit modal ... Sidebar: - Each connection header now has three inline icon buttons: Sync now (spins while in flight), Edit (opens the modal), Remove (opens an inline confirm strip). Removes cascade repos via ON DELETE CASCADE. - The connection's last_sync_error surfaces as a red inline banner under the header — no more 'error status with nowhere to see why'. - Sync is POST /api/repos/connections/:id/sync (already existed); after either sync or delete the sidebar re-fetches so state stays consistent. Edit modal (RepoConnectionEditModal): - Loads GET /api/repos/connections/:id, pre-fills owner/base_url/label - PATCHes only the fields that actually changed; empty string on a Some(&str) field sends explicit null so the backend clears it - Sync-now + Remove reachable from inside the modal too - Rotating the token is out of scope: the modal says as much and points the user at delete + re-create through the wizard (the broker doesn't expose an update path, and rotating in place would require duplicating the whole broker->store_secret flow here) Backend: - GET /api/repos/connections/:id — same ConnectionSummary shape - PATCH /api/repos/connections/:id — owner/base_url use Option<Option<T>> double-nesting so 'omit = leave alone' and 'null = clear' round-trip distinctly through serde - repo_connections::update with COALESCE-per-field so the SQL matches the double-Option semantics without an OR-chain per field	2026-07-07 17:55:20 -07:00
Omar Sobh	e858a7f92f	repos: Gitea provider (first-class) — sync + wizard default ... Fleet's Gitea (git.redclaw.dev) hosts most of this workspace's repos, so Gitea gets the same inline sync treatment GitHub already had. Backend sync_gitea: - base_url is required — Gitea has no shared 'gitea.com'; we accept either the instance root (auto-appends /api/v1) or the fully-formed API base if the user already included the suffix - /orgs/{owner}/repos when owner set, /repos/search when not (with the {data: [...], ok: bool} envelope Gitea wraps that endpoint in) - 404 with an owner surfaces as 'org not found or PAT lacks access', same UX as GitHub - 50/page, capped at 20 pages (~1000 repos); short page terminates - upsert_gitea_repo tolerates the small field-name differences (stars_count vs stargazers_count, owner.login vs owner.username on older versions) Frontend wizard: - Gitea listed first — matches the workspace's actual usage - Default provider selection is now gitea - Token-input placeholder tailored per provider (Gitea's is 'Settings → Applications → Generate New Token (repo)') GitLab still returns 'not yet supported' — that's the next follow-up.	2026-07-07 15:11:30 -07:00
Omar Sobh	6d087bf537	repos: backend — schema, /api/repos routes + GitHub sync provider ... Migration 0034: two tables. repo_connections carries the workspace's per-provider config (owner, base_url, label, last_synced_at, last_sync_error) and points at an app_connections row for the PAT. repos is the per-connection cache with (connection_id, external_id) unique so upsert is idempotent across re-syncs. Cascading deletes clean up cleanly on connection removal. cm-secrets grows a FetchAuthorized op — GET with the stored PAT injected as bearer, returns status + JSON body without ever exposing the credential to cm-api. This is the least-privilege door for read-only provider APIs (list repos), distinct from the InvokeHttp path that still requires a single-use approval grant for outbound writes. cm-api::routes::repos wires: - POST /api/repos/connections (broker store_secret + insert both rows + initial sync + mark_synced) - GET /api/repos/connections - DELETE /api/repos/connections/:id - POST /api/repos/connections/:id/sync - GET /api/repos (500 cap, newest provider_updated first) - GET /api/repos/:id (full detail incl. clone_url + html_url) GitHub provider inline for v1 — paginated pull of /orgs/:owner/repos (when owner set) or /user/repos (when absent), 100/page, capped at 20 pages (~2k repos) to keep first-sync latency bounded. Non-2xx surface back to the caller as sync_error; parse failures are best-effort per repo (skipped, logged, don't abort the batch). Gitea + GitLab providers land in a follow-up — mostly URL swap + response-shape adapter.	2026-07-07 14:52:47 -07:00
Omar Sobh	806ba869e5	teams: ephemeral lifecycle for Scheduled + Triggered planner modes ... Migration 0033: adds teams.lifecycle ('permanent' | 'ephemeral') and a topology_runs.team_id back-ref with a partial index for the sibling-in- flight check. cm-db repo: - teams::insert_team_with_lifecycle (insert_team keeps the permanent default) - topology_runs::enqueue_run_for_team (populates team_id) - topology_runs::check_ephemeral_teardown — atomic SELECT that only returns Some when the team is ephemeral AND no siblings are still queued/running; carries the workspace + bound claw ids for cleanup. cm-api: - topology_worker post-terminal hook maybe_teardown_ephemeral_team runs deprovision_claw on each bound claw (best-effort; failures log but don't block Postgres deletion), then hard_purge each agent row, then delete_team. - routes::teams::build_team_with_lifecycle (build_team keeps default); run_team enqueues with team_id. - planner ScaffoldRequest gains mode; lifecycle_for(mode) sets the team to ephemeral for scheduled + triggered, permanent otherwise. Frontend MasterPlannerModal passes mode in the scaffold payload so the backend can derive lifecycle without duplicating the mode taxonomy. Tests: 3 new (returns claws when no siblings, holds when siblings queued, ignores permanent teams). 10/10 topology_jobs green; workspace clippy --tests clean.	2026-07-07 04:30:07 -07:00
Omar Sobh	b0acdfd987	master planner: wire user-locked topology_kind through chat + scaffold ... Frontend: send topologyKind to /api/planner/chat so the planner's user prompt gets a USER-LOCKED TOPOLOGY block telling Opus to use it verbatim. On buildTeam, override proposal.topology_kind with the user's pick (belt-and-braces — if the planner ignored the lock, we still ship the right shape). Proposal chip renders the effective kind in a lavender tint when it was overridden, with a hover title showing what was replaced. Backend PlannerChatRequest gains an optional topology_kind. Empty / absent = planner picks. Not honored for 'swarm' mode (swarm planner doesn't take a topology kind).	2026-07-07 04:23:00 -07:00
Omar Sobh	987f4f0e84	master planner: add 'Team' mode + size bands per mode ... Modes now: specialists (2–3 domain experts, deep prompts) · team (4–8 balanced roles, coordinator + complements) · swarm (10+ workers, self- verifying loop) · scheduled (ephemeral, cron/one-shot) · triggered (ephemeral, webhook). Backend planner_system_for() gains a TEAM_NOTE using PLANNER_SYSTEM; specialists / scheduled / triggered notes are rewritten to bake in the size + ephemeral guidance. Swarm's system prompt now targets task_count>=10 explicitly. Frontend MODES / INTRO copy match. Chat-preserving switchMode from the prior commit handles the new mode transparently — no state-plumbing changes needed.	2026-07-07 04:18:29 -07:00
Omar Sobh	8a4e222aec	research: auto-transition processing → reviewing on last run ... Hooks the topology_worker's post-terminal path into a new notify_run_completed repo helper that atomically transitions the topic processing → reviewing when the completed run has research_topic_id set AND no siblings for that topic are still queued or running. Guarded on status='processing' so a retry, a re-fire, or a topic already past processing are all no-ops. Best-effort at the worker; DB hiccups are logged and never fail the run. The manual /submit-review endpoint stays as an escape hatch for topics that end up parked in processing with nothing to complete (updated the doc comment).	2026-07-06 12:32:18 -07:00
Omar Sobh	6d1dda6197	loops: iteration timeline — GET /api/topology-runs?loop_id=X ... Extend the topology-runs list route with an optional loop_id filter that returns iterations for a single loop, newest-iteration-first. Adds the iteration and finished_at columns to the summary (skip-null on the JSON so compares stay compact). Backed by list_by_loop in the repo, which uses the existing topology_runs_loop_idx partial index. LoopsCanvas fetches the runs in parallel with the loop detail and renders an iteration timeline card (iteration #, status pill, start time, duration, run id prefix) between the graph section and the actions row.	2026-07-06 12:28:34 -07:00
Omar Sobh	973eeb272e	research: publish approval gate + explicit state transitions ... Fourth commit of the Research + Loops arc. Completes the state machine for research topics with the publish approval gate the spec asked for. Migration 0032 — research_publish_approvals Dedicated small table (id, workspace_id, topic_id, requested_by, status, decided_by/at, created_at). Keeping it separate from the existing `approvals` table (0001) because that one is tightly coupled to gated tool calls inside an agent run — session_key + run_id + action_type + category + payload + preview + requested_by_agent, all NOT NULL. Forcing those nullable would ripple through cm_safety; cleaner to give publish approvals their own two-transition state machine. New endpoints POST /api/research/:id/submit-review processing → reviewing (v1 caller-driven; the orchestrator hook comes when we wire actual runs) POST /api/research/:id/request-publish creates a pending approval. Rejects with 409 if the topic already has one open. GET /api/research/publish-approvals list workspace's pending POST /api/research/publish-approvals/:id/approve flips approval to approved + transitions the topic reviewing → publishing (which stamps published_at) POST /api/research/publish-approvals/:id/reject stays in reviewing; new requests allowed The approve/reject write is an atomic UPDATE ... WHERE status = 'pending'; the decide() repo function returns whether the caller won the race so concurrent double-approves collapse to a single topic transition. State machine after this commit: standby ─POST /start─▶ processing ─POST /submit-review─▶ reviewing ─POST /request-publish + approve─▶ publishing ─(future: artifact assembly)─▶ published	2026-07-06 06:21:29 -07:00
Omar Sobh	4b48c521eb	loops: backend routes + repo + cron scheduler + HMAC webhook ... Third commit of the Research + Loops arc. Lights up loops as durable recurring topology executions: GET /api/loops list workspace's loops POST /api/loops create — returns webhook_token + signing_key ONCE when webhook trigger is enabled; never exposed again GET /api/loops/:id detail PATCH /api/loops/:id update definition DELETE /api/loops/:id delete POST /api/loops/:id/run trigger one iteration NOW POST /api/loops/:id/enable set enabled=true POST /api/loops/:id/disable set enabled=false POST /webhooks/loops/:token public; HMAC-SHA256-verified Scheduler (cm_runtime::spawn_loop_scheduler) wakes every 10s, queries the partial index on (next_fire_at) for due loops, enqueues one topology_runs row per fire with loop_id + iteration + parent_run_id chained back to the previous iteration. Uses croner via the existing scheduling::next_occurrence helper. Missed windows fire ONCE and skip the backlog — next_fire_at is always computed strictly AFTER now(), so a late scheduler doesn't drain a buildup. Webhook signatures follow the same pattern as the Stripe billing webhook (HMAC-SHA256 with constant-time hex compare). Token + signing key are 24-byte OS-RNG values; the URL uses base64-url for the token, and the signing key is base64-std. Both surface exactly once at create time. All three fire paths (scheduler, immediate-run, webhook) funnel through `cm_db::repo::loops::enqueue_iteration` so the invariants stay in one place. `iters` repeat policy is enforced by the scheduler tick; `until` and `on_completion` land with the orchestrator hook in commit 4. Adds cm-llm as a direct cm-api dep, getrandom for the webhook material generator, and wires the scheduler spawn into the server binary alongside the resume sweeper and outbox drainer.	2026-07-06 04:39:27 -07:00