98037f9b3e9358f7c6bd8379e0187ef3b53f2a41
114
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar Sobh
|
d687a00524 |
teams: zeroclaw container coords + coding_readwrite risk profile
Slice 3a of the per-loop-team arc — prerequisites for the runtime spawn hookup that lands in 3b: 0046 migration - ALTER TABLE teams ADD zeroclaw_container TEXT - ALTER TABLE teams ADD zeroclaw_gateway_url TEXT Both NULL until the runtime's spawn_team fn (3b) provisions the container and persists its coordinates. Mirrors the shape already on research_topics (0038) so the resolver code path can generalize. cm-db - team_container_coords / set_team_container_coords: dynamic sqlx::query() readers/writers for the new columns. Runtime template (gw-04, out-of-band edit on /var/lib/clawmates-runtime-template/config.toml + shared runtime /root/clawmates-runtime/data/.zeroclaw/config.toml) - New [risk_profiles.coding_readwrite]: adds file_write + shell on top of the research_readonly baseline. Still excludes http_request / browser / composio (egress stays behind the MCP door). Slice 3b will add spawn_team (bind-mounts paired-topic repo, uses team-scoped state dir, injects team.risk_profile into the config template) and rewire topology_worker to resolve gateway URL through team_id when the loop has one. |
||
|
Omar Sobh
|
6066e93889 |
teams: per-team runtime posture (risk_profile + mcp_bundles) + FK from loops/topics
Foundation slice for letting a coding loop bring its own team instead
of reusing the paired research topic's team. Turns out the teams
table already exists (0010_teams.sql) with full CRUD — this scales
back to the minimal missing bits:
Schema (0045_teams.sql)
- ALTER TABLE teams ADD risk_profile TEXT (NULL = template default)
- ALTER TABLE teams ADD mcp_bundles JSONB DEFAULT '[]'
- ALTER TABLE loops ADD team_id UUID REFERENCES teams ON DELETE SET NULL
- ALTER TABLE research_topics ADD team_id UUID REFERENCES teams
- Two partial indexes (team_id NOT NULL) for the future cascade queries
cm-db (dynamic sqlx::query so the existing get_team's compile-time
cache doesn't need regenerating):
- TeamRuntimeConfig struct
- get_team_runtime_config / set_team_runtime_config
- team_for_loop / team_for_research_topic (resolvers)
- set_team_for_loop / set_team_for_research_topic (binders)
cm-api
- GET /api/teams/{id} now surfaces risk_profile + mcp_bundles
- PATCH /api/teams/{id}/runtime-config sets them
Not touched (comes in follow-up slices):
- Wizard picker exposing 'reuse research team' vs 'fresh coding team'
- Runtime container spawn keyed on team_id
- Migration of existing paired coding loops onto their own team
|
||
|
Omar Sobh
|
e3ef3fd056 |
research: fix Draft 'Invalid Date' + stale-error leak in pipeline card
Two cosmetic bugs surfaced by the successful v0.8.3 pipeline run: 1. **'produced Invalid Date'** — research_outcomes.created_at was an OffsetDateTime serialized by time's default array format (`[y, ordinal, hh, mm, ss, ns, tz]`), which browser's `new Date(...)` can't parse. Add `#[serde(with = "time::serde::rfc3339")]` matching the pattern already in threads.rs / routine_runs.rs. 2. **Stale error text on pipeline card** — the runs stage's `latest_error` walked every run by `created_at DESC` and returned the first non-empty error, so a topic with an earlier failed run + a later completed run kept displaying the old error next to '1 completed'. Now the error only surfaces when the MOST RECENT run itself failed. Historical failures stay in the run count but don't leak their message. |
||
|
Omar Sobh
|
f70f6c679e |
research: errored-state card + one-click rerun (no wizard re-entry)
When a topic ends up parked in 'processing' with all runs failed and nothing in flight, the sidebar card was still spinning as if progress were happening. Now: Backend - topology_runs::run_counts_by_research_topic — batch query that returns (in_flight, failed-since-last-success) per topic. Used by the list endpoint; dynamic sqlx::query() so no prepare needed. - TopicListItem DTO gains runs_in_flight + runs_failed. - start_topic status guard relaxed: allow (standby) OR (processing AND runs_in_flight == 0). Blocks accidental double-fires on a live pipeline; permits rerun on a failed one. Same request body, same behavior once accepted, so the frontend just POSTs /research/:id/start on the RotateCw click. Frontend - ResearchList detects errored: status===processing && !in_flight && failed>0. Swaps the MiniSpinner for a red AlertTriangle and changes the status text to 'error · N failed'. - New RotateCw icon button next to the delete Trash — same button cluster, one click, no wizard re-entry required. Disables while a request is in flight; error surfaces in the sidebar's shared error banner. |
||
 osobh
|
1d69866bcf | research canvas: collapsible sidebar + live topology-run logs (#6) | ||
|
Omar Sobh
|
7e0620fd08 |
research: don't advance to reviewing without an outcome + guard-before-write
Two bugs that combine to produce the 409 you get when clicking Approve:
1) notify_run_completed (topology_worker post-hook) was advancing
topic status processing → reviewing whenever the last sibling run
terminated — success OR fail. A failed run with 0 outcomes still
pushed the topic to `reviewing`, the canvas rendered the "Request
publish" affordance, and the reviewer clicked Approve on nothing.
Fixed by adding an EXISTS(research_outcomes …) clause to the
UPDATE. Topic stays in `processing` when no outcome exists; the
loop's next iteration still has a chance to produce one.
2) decide_publish was calling
research_publish_approvals::decide(approve=true)
FIRST (which flips the row to `status='approved'`) and then
running the "no outcome? 409" guard SECOND. On the 409 return,
the DB was left half-flipped: approval says approved, topic still
in reviewing, no outcome exists, and every future click to the
same approval returns 409 on the "already decided" guard —
leaving reviewers with no way forward.
Fixed by moving the outcome-existence check BEFORE the decide()
call. On 409 now nothing was written, so the reviewer can try
again cleanly once an outcome is produced.
Also unstuck the current stuck row out-of-band (SQL UPDATE to reset
the approval to pending + topic to processing) so the user isn't
forced to delete the topic to escape the 409 loop.
sqlx dynamic query — the new EXISTS clause wasn't in the offline
cache so I switched notify_run_completed to plain `sqlx::query`.
|
||
|
Omar Sobh
|
bfcdca0583 |
research-canvas: managed-by-loop UI + start_topic guard (fold cleanup)
Closes the UX gap the fold introduced: the topic canvas was still showing "Start research" for standby-state topics even when a scheduled loop already owned the runs. Clicking it would 409 (or worse: race the loop into a duplicate run). Topic status stayed at standby forever because the loop path bypassed start_topic's set_status transition. Four changes: 1. **Backend status transition** — compose_and_enqueue_iteration for kind='research' now calls set_status_if(standby, processing) on the topic before the run is enqueued. New DB helper set_status_if only advances when the current status matches the "from" arg — safe against races and re-invocations. Later iterations no-op since the topic is already past standby. 2. **has_managed_loop on TopicDetail** — get_topic hydrates a new ManagedLoop struct (loop_id, title, enabled, next_fire_at, last_run_id, schedule_summary) when a kind='research' loop is bound to the topic. summarize_schedule() derives a human string from the loop's triggers jsonb (e.g. "cron: 0 3 * * * · on new artifact", "one-shot", "manual"). New DB helper loops::research_loop_for_topic returns the row. 3. **Canvas branch** — nextAction takes a managedByLoop flag; when set + status=standby, returns null (no button). The canvas renders a "MANAGED BY LOOP" strip below the topic title showing loop name, schedule summary, next fire time, and enabled dot. Reviewer buttons (Request publish / Approve / Reject) still show normally in later states — reviewers should still promote outcomes even when a loop is producing them. 4. **start_topic guard** — refuses with 409 when a research loop already owns the topic. Closes the direct-POST hole for anyone bypassing the frontend. TS type + summarize_schedule live in the same commit so an old client hitting a new backend just ignores the extra field (no breakage), and a new client hitting an old backend renders the classic buttons (managed_by_loop is optional). |
||
|
Omar Sobh
|
51756d0e68 |
clippy: struct-bundle enqueue_iteration_with_topic + fix doc list warnings
CI's clippy stage failed with -D warnings on three classes of lint:
1. map_clone in loops::recent_reorders — .map(|a| a.clone()) is the
pattern clippy wants replaced by .cloned(). Trivial swap.
2. too_many_arguments on enqueue_iteration_with_topic (8 args, ceiling
7). Refactored the caller-side surface into a new
IterationEnqueue<'a> struct with fields for each column. Matches
the pattern research::NewTopic + loops::NewLoop already use for
the same clippy ceiling. Callers in loops.rs (routes) + the
enqueue_iteration wrapper updated to build the struct literal.
3. doc_lazy_continuation + doc_list_indentation — two doc comment
blocks used ambiguous list-like layouts:
- compose_iteration_task's ASCII-art prepended-block preview:
wrapped in a ```text fence so clippy stops parsing "RESEARCH
ARTIFACT:" etc as list continuation.
- fire_initial_burst_if_set's mention of `burst - 1`: rewrote so
"- 1" doesn't start a line and get misread as list marker.
Purely a refactor + doc pass; no behavior change.
|
||
|
Omar Sobh
|
3e42b1ea39 |
research-wizard: schedule step (once/nightly/manual) + paired coding loop
Completes the research/loop fold. The wizard now closes with a
"How should this research run?" step; picking any mode materializes a
kind='research' loop bound to the topic, and an optional checkbox
adds a paired kind='exec' loop that consumes each new artifact.
Frontend:
- ResearchWizard grows from 5 to 6 steps. Step 6 is the schedule
picker:
· Just once — initial_burst=1, no other triggers
· Nightly — initial_burst=1 + cron "0 3 * * *"
· Manual — webhook_enabled=true
Below the radios, an optional card offers "Also create a coding
loop that consumes each new artifact" — creates a paired
kind='exec' loop with on_artifact_update=true + initial_burst=1
bound to the same topic.
- createTopic API type extended with `schedule` + `create_paired_coding_loop`.
- Both fields ride the existing POST /api/research call; back-compat
is preserved when the wizard omits them.
Backend:
- CreateTopicRequest gains TopicSchedule + create_paired_coding_loop.
- After topic + agent attach, create_topic calls
materialize_topic_loops which:
1. Creates a research-kind loop titled "Research · <topic>" bound
to the topic. Triggers vary by schedule mode; next_fire_at
computed from cron for nightly. Falls back silently if
loop-create errors so the topic still lands.
2. Flips kind to 'research' via loops::set_kind (NewLoop doesn't
take kind directly — default is 'exec' for backward compat).
3. Optionally creates a coding loop titled "Coding · <topic>"
with on_artifact_update=true + initial_burst=1.
- cm_db::repo::loops::set_kind — trivial UPDATE helper used by the
materialize path.
D-answer callouts:
- D1 (fold): every runnable thing is now a loop. "Just once" is a
research loop with initial_burst=1 and no other triggers.
- D2 (inherit): both paired loops carry the same source_research_topic_id
— repo binding lives on the topic, not duplicated.
- D3 (coordinator resolves): the research iteration prompt (from the
earlier commit) instructs the team to preserve stable INT ids and
mark deprecations; coding loops' consumed lists stay valid across
versions.
Follow-ups queued:
- Kind pill on LoopsList cards (research=purple, exec=cyan) so users
can tell them apart at a glance.
- Extract start_topic's task-build so kind='research' iterations
reuse the same coordinator prompt shape as one-shot runs (they
currently use a simpler refresh-oriented prompt; that's fine for
MVP but a rich shared build would give better parity).
- Research topic sidebar shows "linked to N loops" badge.
|
||
|
Omar Sobh
|
91a51dce11 |
loops: kind='research' dispatch — research runs as loops
Delivers the research/loop fold: kind='research' loops run the research pipeline each iteration, appending a new research_outcomes version. The paired on_artifact_update fan-out then wakes up any kind='exec' loops bound to the same topic to consume new INTs. Every runnable thing is now a loop (D1). Backend — DB helpers: - cm_db::repo::loops::kind_and_binding — reads (kind, source_topic, task_template) so callers can dispatch without hydrating the whole Loop struct. - cm_db::repo::loops::enqueue_iteration_with_topic — new variant that sets research_topic_id on topology_runs alongside loop_id, so the completion hook's freeze_research_outcome writes a new outcome version for research-kind iterations. - cm_db::repo::research_topics::get_any_workspace — cross-workspace fetch used by the research task builder (the loop row is authoritative for the workspace binding via kind_and_binding). Backend — dispatch: - routes::loops::compose_research_iteration_task — builds the coordinator prompt for a research iteration: topic title + description + outcome_kind + prior version pointer + refresh instructions (survey new sources, preserve stable INT ids, mark superseded items as deprecated rather than delete). The completion hook writes the resulting synthesis as research_outcomes v(prior+1). - routes::loops::compose_and_enqueue_iteration — one-shot dispatch: reads the kind, picks compose_iteration_task (exec) or compose_research_iteration_task (research), enqueues with or without research_topic_id set. All four enqueue callsites now route through compose_and_enqueue: - create_loop (initial_burst) - run_now - webhook_receive - topology_worker::continue_initial_burst - topology_worker::freeze_research_outcome (on_artifact_update fan-out) Research-kind loops naturally form the "nightly refresh" side of a paired research + coding loop: research writes a fresh outcome version → fan-out wakes exec loops with on_artifact_update → coding loops consume the next INT (which the research loop may have just added). D2 answer (inherit repo binding): repo lives on the topic; both loops sharing the source topic id read from the same context, no duplication. D3 answer (coordinator resolves): the research iteration prompt tells the team to preserve stable INT ids and mark deprecations rather than delete, so coding loops' consumed lists stay valid across versions. Follow-up (next commit): ResearchWizard schedule step — "Just once / Nightly / Manual" that creates the paired research-kind loop with initial_burst=1 (just once) or cron 0 3 * * * (nightly) + optional paired coding loop with on_artifact_update. |
||
|
Omar Sobh
|
4ada5557f2 |
loops: kind column + initial_burst + on_artifact_update trigger fan-out
Foundation for folding research into loops as a first-class kind.
This commit ships the plumbing; the research-kind dispatch itself
lands next. Behavior for existing exec-kind loops is unchanged unless
they opt into the new trigger fields.
Migration 0044:
- kind TEXT NOT NULL DEFAULT 'exec' CHECK ('exec' | 'research'). New
research-kind will run the research pipeline each iteration (next
commit); 'exec' preserves today's behavior.
- initial_burst_remaining INT NOT NULL DEFAULT 0 — countdown for the
triggers.initial_burst quota. Decremented CAS-safely on each
completion until it hits 0.
- Two partial indexes: (kind, source_research_topic_id) for kind-
aware lookups, and (source_research_topic_id) filtered on
on_artifact_update=true + enabled=true for the fan-out hook.
Trigger schema extended with two optional fields:
- initial_burst: N — fire N iterations back-to-back at create time.
create_loop enqueues the first iteration inline (subject to
empty-roster gate), sets remaining=N-1, and the completion hook
continues the chain until exhausted.
- on_artifact_update: true — when a bound research_outcomes row is
inserted for the source topic, wake up one iteration of this loop.
Coalesced against has_active_run so a burst of rapid revisions
doesn't queue duplicates.
Backend:
- cm_db::repo::loops helpers (all dynamic sqlx, no offline cache
regen needed):
- set_initial_burst_remaining
- take_initial_burst_slot (CAS UPDATE returning prev value; 0 on
exhausted or race loss)
- loops_awaiting_topic (fan-out query: kind=exec + enabled +
on_artifact_update=true bound to the given topic)
- has_active_run (queued|running iteration existence check)
- get_any_workspace (bypasses the workspace scope guard; used by
the completion hook where the run row is authoritative)
- routes/loops::compose_iteration_task made pub so the completion
hook can build the same enriched task string as run_now.
- topology_worker::freeze_research_outcome now fans out to awakened
loops after the outcome insert, using compose_iteration_task and
coalescing on has_active_run.
- topology_worker::continue_initial_burst runs on every completion:
· take_initial_burst_slot (CAS) — no-op if already exhausted
· has_active_run coalesce guard
· re-fetches the loop via get_any_workspace + compose_iteration_task
· enqueues via loops::enqueue_iteration with parent_run_id set
Follow-ups already queued:
- kind='research' dispatch in run_job — build the research
coordinator task from the topic config, run the research pipeline
each iteration. Requires factoring start_topic's task-build.
- ResearchWizard "When should this run?" step (Just once / Nightly /
Manual) creating the topic + paired research-kind loop.
- LoopsWizard trigger UI matching the design proposal (burst count,
cron, on-artifact checkbox).
|
||
|
Omar Sobh
|
ce111273bb |
loops: reorder history collapsed under the progress pill
Completes the reorder rationale loop — the previous commit captured
REORDER: markers server-side but nothing surfaced them. Now the loops
sidebar renders a small "N reorders ▸" button under the progress pill
whenever the loop has any reorder events. Click expands to a
newest-first list of "iter <n> · <rationale>" lines so a reviewer can
see, at a glance, when the plan was adjusted and why.
Backend:
- LoopProgress DTO gains recent_reorders: Vec<Value> — newest-first,
capped at 5 so the card stays compact. Full history remains on the
loop row's reorder_events column.
- cm_db::repo::loops::recent_reorders — reads the jsonb array, returns
the last N in newest-first order.
- list_progress populates it per loop.
Frontend:
- LoopReorderEvent + recent_reorders on LoopProgress type.
- LoopsList tracks openHistoryId per-loop (one open at a time).
- Card renders history button + expanded panel styled to match the
progress pill above.
Notes:
- The event object schema is {run_id, iteration, text, ts}. Fields are
optional in the TS type so future schema tweaks don't break the
render.
- 5-item cap chosen so the sidebar card doesn't grow unbounded. If a
loop accumulates a lot of reorders, follow-up UI can render the full
history on the loop detail page.
|
||
|
Omar Sobh
|
0c17de52dd |
loops: reorder rationale extraction — REORDER: markers logged per iteration
Coordinator can now log WHY it worked on an INT-XX out of order
("REORDER: INT-05 before INT-04 because prereq X is unmet") and the
completion hook captures each rationale as an append-only event on
the loop. Sets up a reviewable timeline of when the plan was
adjusted, independent of the underlying `consumed_int_ids` advance.
Migration 0043:
- loops.reorder_events JSONB NOT NULL DEFAULT '[]'::jsonb — append-
only array of {run_id, iteration, text, ts}. Kept on the loop row
(rather than a dedicated table) so the mini-timeline is one read
away from the loop card.
Backend:
- topology_worker::parse_reorder_rationale — line matcher symmetric
with parse_completed_int_ids. Tolerates list dashes / prefixes /
markdown emphasis; case-insensitive marker match, preserves case of
the rationale text.
- cm_db::repo::loops::append_reorder_event — one INSERT-like append
per rationale, uses jsonb_build_object with postgres now() so ts is
wall-clock canonical (no client-clock skew).
- topology_runs::iteration_for_run — new helper so events carry the
iteration index.
- routes::loops::compose_iteration_task — coordinator prompt now
explicitly asks for `REORDER: <one-sentence>` at the top of the
first substantive turn when working out of order, AND spells out
that both markers must appear literally with colons (no bold, no
code fence) so the line parser doesn't miss them.
Non-loop and standalone-loop runs are unaffected — the hook only
fires when the run belongs to a source-bound loop.
Follow-up: expose reorder_events on the loops list endpoint + render
a small collapsed timeline on the LoopsList card.
|
||
|
Omar Sobh
|
4a140cb7db |
loops: parse COMPLETED: INT-XX markers to advance loop pointer (option b)
Closes the loop bridge — the missing piece from the previous commit.
Without this, `current_int_index` stayed at 0 forever and every
iteration re-worked INT-01. Now the topology_worker's completion hook
parses the run's final output for `COMPLETED: INT-<NN>` markers and
atomically advances the loop's consumed_int_ids + current_int_index.
Backend:
- topology_worker::advance_loop_after_completion — new post-terminal
hook that fires alongside freeze_research_outcome. Reads loop_id_for_run
(skips non-loop runs) + source_research_context (skips standalone
loops without a bound source topic).
- parse_completed_int_ids — forgiving parser: matches `COMPLETED: INT-01`,
`- COMPLETED: `INT-01``, `COMPLETED: INT-01, INT-02`, case-insensitive,
tolerates list dashes / backticks / markdown emphasis. De-dupes within
a single output.
- cm_db::repo::loops::advance_after_completion — atomic UPDATE that:
· appends only NEW ids to consumed_int_ids (idempotent on re-runs)
· bumps current_int_index by the count of new ids landed
Set semantics via `SELECT DISTINCT unnest(...)` so ordering-based
bugs can't accumulate duplicates.
Behavior end-to-end:
1. Loop wizard imports an integrations artifact (previous commit).
2. run_now / webhook_receive → compose_iteration_task prepends artifact
+ focus instruction ("address INT-<current+1>, log COMPLETED at end").
3. Coordinator run does the work, emits `COMPLETED: INT-<NN>`.
4. topology_worker completion hook parses the marker, advances the
loop, and the NEXT iteration sees an updated `consumed:` list +
incremented `current_int_index` in its focus instruction.
Follow-ups still queued:
- Loop card refresh button — pull latest artifact after reject-with-
revision on the source topic (right now the prepend uses the LATEST
outcome automatically, so refresh is UX only, not correctness).
- Reorder rationale extraction — coordinator emits "REORDER: INT-05
before INT-04 because prereq X is unmet"; today that's just prose
in the output, not indexed.
|
||
|
Omar Sobh
|
ce73abe5ab |
loops: bridge research artifact into loop iterations (option C + b)
The bridge lets a coding loop "consume" an integrations research artifact one INT-XX item per iteration. Options b (order-sequential iteration) and C (snapshot in task_template + save the pointer for future refresh) from the design discussion. Migration 0042 — three new loops columns: - source_research_topic_id — nullable pointer to research_topics. - consumed_int_ids TEXT[] — INT-XX ids the loop has completed. Advances when topology_worker parses "COMPLETED: INT-<NN>" markers from the run's final output (wired in a follow-up commit). - current_int_index INT — monotonic pointer for order-sequential iteration. Coordinator addresses INT-<current+1> unless prereqs are unmet, in which case it works on the smallest unblocking INT-XX and logs the reorder rationale. Backend: - cm_db::repo::loops::set_source_research_topic — bind/unbind pointer. - cm_db::repo::loops::source_research_context — read pointer + state. - routes::loops::compose_iteration_task — new caller-side helper that reads the pointer, fetches the topic's latest research_outcome, and prepends the artifact + focus instruction to task_template. - run_now + webhook_receive both pass task_template through compose_iteration_task before enqueue. Standalone loops (no pointer) behave identically to before. - CreateLoopRequest accepts `source_research_topic_id`, ownership- checked via research_topics::get before persist. Frontend: - New ResearchArtifactPicker modal — lists published topics, fetches the artifact on pick, returns (topic_id, markdown) to caller. - LoopsWizard task_template step gains "Import from research artifact" button (right-aligned). Click opens the picker. On pick: task populates with the artifact markdown, pointer saved, textarea expands to 8 rows, small info strip shows "Loop is bound to topic <id>. Each iteration will focus on the next unconsumed INT-XX." - Unlink button reverts to standalone loop mode. Follow-up (next commit): - topology_worker completion hook — parse "COMPLETED: INT-<NN>" out of the run's final output + update consumed_int_ids + current_int_index atomically. Without this, current_int_index stays at 0 forever and every iteration works on the same INT. - Loop card refresh button — re-read source topic's latest outcome (useful after a reject-with-revision cycle on the source topic). |
||
|
Omar Sobh
|
44d6e95022 |
fmt: apply rustfmt across the P2 arc
CI's rustfmt check flagged the multi-line sqlx::query() calls I introduced in P2 (loop_id_for_run, zeroclaw_gateway_url, etc.). No behavior change — pure formatting. |
||
|
Omar Sobh
|
03f1830d1f |
loops: Path B container isolation (P2)
Symmetric with the research pipeline: every enabled loop can now have its own per-loop team container so scheduled runs don't share state with other loops or with research. Same daemon image, same clawmates network, deterministic name loop-<id>-team. Backend surface: - Migration 0040 adds nullable `zeroclaw_container` + `zeroclaw_gateway_url` columns to loops (parallel to research_topics). - research_container.rs grows loop_container_name_for(), spawn_loop() (state-only mount, no repo), and teardown_loop(). Kept in the same module to share the docker connect() + inherited_env() plumbing; each pattern gets its own labels (clawmates.role=loop-team) so ps filters can tell them apart. - cm_db::repo::loops gains set_zeroclaw_container() + zeroclaw_gateway_url() (dynamic sqlx queries — no offline cache regen needed). - cm_db::repo::topology_runs gets loop_id_for_run(): mirror of research_topic_id, used by the worker. Wiring: - routes/loops::run_now + webhook_receive call ensure_loop_container() before enqueuing an iteration. Idempotent: an already-running container is just reattached. Failures are logged and do NOT block the enqueue — topology_worker falls back to the workspace gateway when the URL isn't set on the loop. - routes/loops::disable_loop + delete_loop both fire teardown_loop() so paused / deleted loops don't hold a docker slot. - topology_worker's per-run URL resolution: existing research fast path unchanged; when it doesn't hit, the worker now looks up loop_id and reads the loop's gateway URL. Deploy step (required on gw-04 for state to persist across container restarts): add a `/var/lib/clawmates-loops:/var/lib/clawmates-loops` bind mount + `CLAWMATES_LOOPS_STATE_ROOT=/var/lib/clawmates-loops` env var to clawmates_server_1 in the compose. Without it, loops still run — the state dir lives inside the API container's filesystem so persistence is limited to that container's lifetime. Follow-up: - Scheduler-tick fires (cron-driven, not run_now) — they call enqueue_iteration in cm-scheduler and don't yet go through ensure_loop_container. Add a symmetric spawn there so cron fires also land on the isolated daemon. - Compose file reconciliation — deploy/compose/docker-compose.yml in the repo has drifted from prod; when we sync it, add the loops mount at the same time. |
||
|
Omar Sobh
|
e3011ed025 |
research: reject-with-revision loop (R2)
Before: reviewer rejected a publish → audit log flipped, topic stayed in reviewing, no way to feed the critique back into the run pipeline. Reviewers with revision notes had to eat them or hand-message the coordinator. Now: reject accepts an optional `notes` field. When present: - Persisted on the research_publish_approvals row (migration 0039). - Topic flips `reviewing → standby` so the next `start_topic` is legal. - `start_topic` reads the most recent rejected-approval notes for the topic and prepends "PRIOR REVIEW NOTES (address these in this revision):\n<notes>\n---" to the coordinator task. Loop closes through the same run pipeline — no new spawn code path, which means the reviewer's guidance flows through the same topology_worker, run_events, outcome-writer chain and lands as a fresh research_outcomes row (versioned, prior drafts preserved). No notes on reject = legacy behavior (topic stays in reviewing, publish requests still allowed). Migration 0039 adds nullable `notes TEXT` to research_publish_approvals. `decide()` gains a `notes: Option<&str>` parameter (only one caller, updated inline). New `latest_rejection_notes(pool, topic_id)` helper for start_topic. Frontend: - rejectPublish(id, notes?) now sends a JSON body when notes are provided. - ResearchCanvas reject button opens an inline form with a textarea + Cancel/"Send back for revision" pair. Empty notes → plain reject. - Button label switches: "Send back for revision" when notes present, "Reject without notes" when empty. Follow-up: - Notes shown in the review UI on the resulting draft so the next reviewer sees what changed. - Multiple rejection rounds — currently only the LATEST rejection's notes surface. Accumulating history is a schema-only tweak. |
||
|
Omar Sobh
|
acd2a0f287 |
structure polish: post-reify nav + ensure-chain + TeamWizard auto-parent
Two small quality-of-life fixes on top of the reify commit:
Post-reify navigation
OrphanMigrationDialog already returned team_id in its result;
Dashboard now pushes /?team=<team_id> before router.refresh() so
the user lands on the freshly-materialized team and sees exactly
where their agents just moved. Previously they had to hunt for it
in the newly-rebuilt sidebar.
Wizard auto-materialize (POST /api/structure/ensure-chain)
cm-db: ensure_chain(pool, ws, fallback_org, fallback_company) —
fast path returns coordinates of the first org+company already
bound in this workspace (workspace's oldest org, oldest company
under it). Slow path inserts a new org+company with the
fallback names ("My Workspace" / "General") + binds them via
org_companies. Returns { org_id, company_id, created }. Small
txn — leaves the workspace consistent whether it was already
wired or not.
cm-api: POST /api/structure/ensure-chain accepts optional
fallback_org_name and fallback_company_name in the body (trimmed,
else default). Returns the ids.
CreateTeamRequest gains an optional attach_to_company_id. When
set, after build_team() completes, we look up the company
(workspace ownership check enforced by companies::get), count
its existing teams for a stable n_i node id, and insert a
company_teams binding — so the team lands under the parent
atomically instead of a follow-up round-trip.
TeamWizard now calls ensure-chain before POST /api/teams and
passes the returned company_id in attach_to_company_id. Both
calls are best-effort — if ensure-chain fails (network etc.)
we still try to create the team, and the migration dialog stays
available as the fallback UX. Wizard flow now: fresh workspace's
first team is fully wired from the moment it appears in the
tree — no synthetic "My Workspace" scaffolding ever gets
rendered around it.
The Team/Company create paths not touched here (create_team_from_claws,
company create, org create, MasterPlannerModal scaffold) still
work as before — they just won't auto-parent yet. Later commits
can wire them the same way.
|
||
|
Omar Sobh
|
8b789beec0 |
structure: reify-orphans endpoint + "give these a home" dialog
Turns the four synthetic tree containers into a real migration path.
Clicking any of them ("My Workspace", "Teams", "Direct",
"Ungrouped") opens a dialog that creates a real
org → company → team chain and re-parents every orphan into it, all
in one DB transaction.
cm-db (new module structure_reify)
- orphan_agents / orphan_teams / orphan_companies: workspace-scoped
SELECTs of entities without a parent binding in team_members /
company_teams / org_companies. Used both by the dialog's counter
and internally by the migration.
- count_orphans: cheap combined-count via three subqueries in a
single SELECT so the dialog only round-trips once for the header.
- reify_orphans(pool, ws, org_name, company_name, team_name):
1. begins a tx
2. inserts a new org + company + team (all `flat`, empty graphs
— user can shape them later via the existing PATCH endpoints)
3. binds company under org (org_companies "n0")
4. binds team under company (company_teams "n0")
5. inserts team_members rows for every orphan agent (n1, n2, …)
6. inserts company_teams rows for every orphan team
7. inserts org_companies rows for every orphan company
8. commits, returns the created ids + moved counts
cm-api (routes/structure)
- GET /api/structure/orphan-counts → { agents, teams, companies }
- POST /api/structure/reify-orphans → { org_id, company_id, team_id,
moved_* }. Trims + rejects any empty name; validates before
starting the transaction so a 400 never rolls anything back.
Frontend
- New OrphanMigrationDialog: fetches counts on open, three name
fields (defaults: Organization "My Workspace", Company "General",
Team "Everyone"), POSTs on save. "Nothing to migrate" state
disables the save button when the workspace is already fully
wired. Copy explicitly notes that everything is renameable in the
sidebar afterward.
- Dashboard: onTreeSelect now branches on SYNTHETIC_TREE_IDS —
clicking a synthetic node opens the dialog instead of falling
through to the (nonexistent) selection. On successful reify,
router.refresh() so the sidebar + world viz reflect the new real
chain.
What this doesn't do yet (next commit)
- Wizard auto-materialize: when creating a team/company via wizard,
auto-create parent placeholders if they don't exist. Deferred so
this commit stays focused.
|
||
|
Omar Sobh
|
99e5207e69 |
sidebar: click-to-rename org/company/team + strip synthetics from world viz
Two related pieces of the "kill My Workspace" cleanup, landed
together because they share the same file:
Backend
- Three tiny inline-rename endpoints:
PATCH /api/orgs/{id}/name
PATCH /api/companies/{id}/name
PATCH /api/teams/{id}/name
Each takes { name: string }, trims + rejects empty, returns 204.
Backed by rename_org / rename_company / rename_team in cm-db —
single-row UPDATEs scoped to the caller's workspace, NotFound if
the id isn't visible.
- Registered next to the existing PATCH /:id (topology) routes so
they don't collide.
Frontend
- StructureTree accepts an optional onRename and canRename.
TreeRow: click on the label text of a renamable node → the span
becomes an <input>, focus + select-all, save on Enter or blur,
cancel on Escape. The rest of the row (row chevron / row body)
still navigates + selects as before, so single-click behaviour
is preserved for everything except the name text itself.
react-hooks/set-state-in-effect avoided by resetting the draft
in the enterEdit() click handler instead of inside a useEffect.
- Dashboard passes canRename={item.level !== "claw" && !synthetic}
(claws don't have a rename endpoint yet; synthetic scaffolding
gets reified into real rows in the next commit — the wizard
auto-materialize + orphan-migration dialog).
onRename fires the corresponding PATCH and calls router.refresh()
so the label lands in every consumer of the tree.
- World viz seed: new stripSynthetics(roots) helper walks the tree
and lifts children of any synthetic container up to their
grandparent's level. worldCanvasRoots feeds through this before
narrowRoots(). Result: the Live viz no longer shows "My Workspace"
or "Teams" nodes — real agents orbit the world root directly
(which is what you were asking for). Sidebar tree still shows
them so orphaned agents remain visible until the migration lands.
|
||
|
Omar Sobh
|
21ac35c8d4 |
research: spawn per-topic ZeroClaw team container on start (commit 1/3)
Commit 1 of the path-B (real per-topic isolation) plan. The
container spawns and its coordinates persist — nothing talks to
it yet; commit 2 wires ZeroClawDriveExecutor to prefer the topic's
URL when populated. This split keeps each landing verifiable.
Backend
- Migration 0038: research_topics gets zeroclaw_container_name +
zeroclaw_gateway_url columns. Both nullable so a topic can exist
before a spawn and teardown just NULLs them out.
- cm-db: ResearchTopic struct extended; get/list SELECTs updated;
new set_zeroclaw_container(id, workspace_id, name, url) helper
used both for spawn (Some/Some) and teardown (None/None).
- cm-api: bollard added as a workspace dep (matches cm-sandbox's
version). New research_container module:
· connect() → uses DOCKER_HOST when set (prod's socket-proxy
at tcp://socket-proxy:2375) else the local socket. Same
pattern cm-sandbox already uses.
· container_name_for(topic_id) → "research-<uuid>-team"
(deterministic so a re-start reattaches to the same
container instead of orphaning it).
· inherited_env() → propagates ZEROCLAW_*, OPENAI_*,
ANTHROPIC_*, GEMINI_*, GROQ_* from the parent server env
(provider config + tokens), stripping the server's own
ZEROCLAW_GATEWAY_URL/WORKSPACE so the team runtime doesn't
loop back on itself. Appends ZEROCLAW_GATEWAY_PORT=42617
and ZEROCLAW_WORKSPACE=/zeroclaw-data/workspace for the
team's own listener.
· spawn(docker, topic_id, repo_host_path, state_host_path):
- inspect: if the container already exists, start it if
stopped and return its coordinates (idempotent restart).
- else create with:
image = CLAWMATES_RESEARCH_TEAM_IMAGE or
clawmates-runtime:latest
cmd = [daemon, --host, 0.0.0.0]
env = inherited_env()
mounts = repo_host_path → /workspace/repo (rw)
state_host_path → /zeroclaw-data (rw)
network = CLAWMATES_RESEARCH_TEAM_NETWORK or
clawmates_core
labels = clawmates.role=research-team,
clawmates.research.topic_id=<uuid>
- creates state_host_path first so bind doesn't ENOENT.
· stop(docker, name) → stop + remove. Idempotent on 404/304.
- start_topic wires spawn after the clone completes:
· state root = CLAWMATES_RESEARCH_WORKSPACE_ROOT / <topic> /
state
· on success, persists (name, url) on the topic row so commit
2 can look them up when constructing the executor
· every failure (docker connect, docker create/start, DB
persist) is best-effort: logs and continues. A missing team
container leaves the topic pointing at the workspace-wide
gateway URL (env), preserving prior behavior.
Deploy prerequisites (not in this commit)
- The compose stack's clawmates_server service needs bind-mounts
of CLAWMATES_RESEARCH_WORKSPACE_ROOT (e.g.
/var/lib/clawmates-research:/var/lib/clawmates-research) so
paths the server writes to are visible on the host and the
spawned team container mounts the same underlying data.
- socket-proxy's ACL must allow POST + DELETE on /containers
(already the case in prod per the audited compose file).
|
||
|
Omar Sobh
|
7984e65174 |
research_topics::create: bag 8 args into a NewTopic struct (fixes clippy)
The prior signature took (pool, workspace_id, title, description, outcome_kind, topology_kind, repo_id, created_by) — 8 args, one over the clippy::too_many_arguments ceiling and blocking CI. Refactor to a NewTopic<'a> input struct mirroring the NewLoop / NewSubTopology pattern the codebase already uses. Wizard-side additions like a repo commit branch land as struct fields instead of cascading into every call site. |
||
|
Omar Sobh
|
3465bb7a6d |
research: persist bound repo + shallow-clone on start_topic
This is the minimum viable version of the "agents actually work on
a repo" architecture. Full vision (isolated ZeroClaw container per
topic, dynamic agent provisioning inside, pause/resume, commit
gate) is real weeks of work — this closes the first, most-visible
gap so the ClawHDF5 topic can actually run against its codebase.
Backend
- Migration 0037: research_topics gets repo_id UUID (nullable, FK
to repos ON DELETE SET NULL) and repo_workspace_path TEXT for
the on-disk checkout location. Index on repo_id when set.
- research_topics::create takes repo_id: Option<Uuid>. get + list
select it and repo_workspace_path. set_repo_workspace_path
persists the path once the first clone lands.
- CreateTopicRequest accepts `repo: Option<TopicRepoRef>` — the
same denormalized shape the wizard already sends. Only repo_id
is authoritative; other fields are ignored (dead_code-allowed
so serde still deserializes the full body).
- start_topic branches on topic.repo_id. When set, it calls
ensure_repo_workspace:
· resolves repo.clone_url + repo.default_branch
· target path = CLAWMATES_RESEARCH_WORKSPACE_ROOT
// <topic_id> // repo (defaults under $TMPDIR)
· runs `git clone --depth 1 --single-branch --branch <b>` via
tokio::process. Reuses the checkout if .git already exists.
· persists the path so re-starts skip the clone
· runs `git ls-files` to sample the tree (first 60 entries,
total count reported honestly so the prompt doesn't lie
about coverage)
All best-effort — a clone failure logs but still starts the run
without repo context rather than aborting.
- build_coordinator_task takes Option<&RepoContext>. When present,
the framing gets a REPO block (slug / path / branch / file
sample) and a USING THE REPO section instructing the coordinator
to ground every recommendation in a concrete file reference and
never fabricate paths. The per-topology bodies are unchanged —
the repo guidance sits above them so it applies to every shape.
What this unblocks / doesn't unblock
Unblocks: The coordinator prompt now knows the repo exists, where
it lives on disk, and what's in it. Even without file-editing
tools wired to the checkout, the coordinator can point spokes at
concrete modules and the final artifact can reference real files.
For a spec-shaped outcome like ClawHDF5's, that's the difference
between abstract advice and a spec grounded in the actual crates.
Does NOT unblock: The agents themselves editing files, running
tests, or committing. That requires either mounting the checkout
into the ZeroClaw sandbox or exposing a new MCP tool for
repo-scoped file ops — separate follow-up.
|
||
|
Omar Sobh
|
7c1af2e070 |
research: pipeline-running signal + spinners so users aren't guessing
The prior flow was ambiguous: after hitting Start research, status flipped to "processing" and a "Submit for review" button appeared immediately with no indication that anything was actually running. Users had to guess whether the pipeline was working or stalled. Backend surfaces the truth as a signal: - new topology_runs::active_runs_for_research_topic counts queued+running runs whose research_topic_id matches - TopicDetail includes runs_in_flight: i64 alongside the existing status field, so the canvas can distinguish "pipeline still working" from "runner stalled". ResearchCanvas is now honest about state: - while runs_in_flight > 0, the header status pill grows a cyan "N runs in flight" badge with an inline SVG spinner - the stage-explainer card turns cyan-bordered and shows a "pipeline is running" hint, plus copy pointing the user at the Agents tier where each teammate's activity streams live - the "Submit for review (manual)" button is HIDDEN while any run is in flight — it's an escape hatch for stalled runs only, not the happy-path action. It reappears if runs_in_flight drops to zero but the topic is still marked processing, so a stalled runner can still be nudged along. - the canvas polls getTopic every 4s while status is processing/ publishing or runs_in_flight > 0, so the spinner + outcome swap in automatically when the pipeline completes. ResearchList sidebar: - each row's status dot becomes a spinner when the topic's status is processing or publishing, matching the canvas at a glance - the list also polls every 6s while ANY topic is active, so transitions land in the sidebar without waiting on a parent bump. The poll is gated on a derived boolean to avoid effect thrash. Follow-up: same pattern belongs on LoopsList / LoopsCanvas for loop iterations in flight — same signal (queued+running runs per loop) but not wired here. |
||
|
Omar Sobh
|
316cdbf929 |
research sidebar: delete-with-confirm per topic
Mirror the row-level delete affordance the loops sidebar already
has. Loops was wired earlier; research had a bare title-only card
with no way to remove a stale topic.
- cm-db: research_topics::delete cascades via existing FK rules
(research_topic_agents, research_publish_approvals, and the new
research_outcomes all CASCADE on topic_id; topology_runs's
research_topic_id back-ref is SET NULL so historical runs stay).
- cm-api: DELETE /api/research/{id} → 204. Idempotent.
- Frontend: deleteTopic helper. ResearchList row is now a card
with the existing title/status/outcome header plus a trash icon
that flips the card into an inline "Delete topic + all outcomes?"
confirm strip. Confirm → red Delete / gray Cancel. If the
deleted topic was selected, selection clears; local counter
bumps the list refetch without waiting on a parent.
|
||
|
Omar Sobh
|
a2d3d85ebe |
research pipeline v2: topology-aware start + persisted draft
Three connected changes that turn "Start research" from a status flip into a real pipeline that produces a reviewable artifact: - Migration 0036: adds research_topics.topology_kind (default 'hub_spoke') and a new research_outcomes table (id, topic_id, version DESC, body_md, produced_by_run_id, created_at) so each run's final synthesis is versioned and persistent. - Wizard now has a topology picker in the Outcome step — hub_spoke / pipeline / hierarchical / star_moe — with copy that steers users to the right shape (Pipeline for research → distill → analyze → implement rosters, hub_spoke for the coordinator- and-specialists default). - start_topic reads the chosen topology_kind, parses it into a cm_topology::TopologyKind, and dispatches a per-shape coordinator prompt via build_coordinator_task. Pipeline explicitly tells stage 1 not to write the final artifact and propagates a "final stage MUST emit a complete markdown document with measurable acceptance criteria" instruction downstream. The graph builder is called with the topology the user actually picked instead of hard-coded HubSpoke. - topology_worker::freeze_research_outcome fires after every successful complete(). It looks up research_topic_id on the run; if set and final_output is non-empty, it inserts a new research_outcomes row (version auto-derived server-side via coalesce(max(version), 0) + 1). Best-effort — a DB hiccup logs but doesn't fail the run. - TopicDetail now includes topology_kind and latest_outcome. ResearchCanvas swaps in the outcome's body_md (rendered as pre-wrap markdown, versioned header, produced-at timestamp) whenever an outcome exists; the original prompt collapses into an "Original prompt" <details> below so it's still one click away. Pre-run topics still show the description as before. Follow-ups still open: reject-with-revision loop feeding the coordinator, publishing → published transition + real artifact export (md / pdf), and an approvals inbox surface for reviewers. |
||
|
Omar Sobh
|
7e2b02d8bb |
research: wire start_topic to actually run the pipeline
The prior start_topic only flipped the status column — no work was enqueued. Now clicking "Start research" actually launches the assigned agents through the orchestrator. - start_topic loads the topic + its research_topic_agents, picks a coordinator (first slot with role_slot containing "coordinator"; else the first slot), and swaps it to index 0. - Builds a hub_spoke topology graph via cm_topology::build with roles = [coordinator, spoke1, spoke2, …]. hub_spoke wires edges from the hub to every spoke and back, so the coordinator can address any specialist per turn. - Assembles a coordinator prompt from the topic's title, description, outcome_kind, and a roster line for each teammate — so the coordinator knows who's on the team and what each does. - Enqueues via a new topology_runs helper enqueue_run_for_research_topic that stores research_topic_id on the run row. `topology_worker::maybe_transition_research_topic` → `notify_run_completed` already picks up on that back-ref and flips the topic processing → reviewing when the last run terminates — that path was dead code until now. - Per-agent activity streams into each claw's card for free: the orchestrator journals turn events into run_events; the existing /api/world/live SSE normalizer emits agent.reasoning.delta / agent.tool.call / agent.task.update keyed by agent id, which ClawCommandCenter is already subscribed to. The `published` terminal state is still unreached (that's the "publishing → published + artifact" step from the earlier walkthrough — separate follow-up). |
||
|
Omar Sobh
|
984d9a1274 |
reap: cascade orgs → companies → teams → agents
Selecting a team/company/org in the Agents sidebar used to only
delete the grouping row; the agents inside survived, ungrouped.
Not what the user wanted, and it left a trail of orphaned runtime
state (containers, .brain files, DB rows) behind.
Backend — POST /api/claws/batch-delete is now a universal cascade
reaper. Body accepts { ids?, teams?, companies?, orgs? } in any
combination. The server walks org → companies_of_org →
teams_of_company → agents_of_team, dedupes against explicit ids,
and hard-purges every unique agent (deprovision ZeroClaw runtime,
tear down sandbox container, unlink .brain/.onion files,
transactional agents::hard_purge). Group rows are deleted last; FK
cascades on team_members, company_teams, org_companies, and
loop_agents/teams/orgs clean up the join tables. Every stage
streams SSE.
Three new cm-db helpers wire the walk: agents_of_team,
teams_of_company, companies_of_org — all DISTINCT selects on the
existing join tables.
Frontend — Dashboard's Agents-tier StructureTree now sets
selectLevel="*" (was "claw"), so the Wrench → checkbox affordance
appears on org/company/team/agent nodes alike; the same-level
invariant in onToggleSelect still prevents mixed batches.
ReapProgressModal collapses to a single POST regardless of kind —
body key derived from kind — and its subtitle is honest:
"Cascading through every agent inside — permanent."
|
||
|
Omar Sobh
|
c063d4bbdd | loops: cargo fmt --all (unblock CI) | ||
|
Omar Sobh
|
a6da19430f |
loops: repo picker + agent/team/org staffing + sidebar edit/delete
Adds the missing pieces the wizard needed and the sidebar controls around it: - LoopsWizard is now a 6-step flow (identity → repo → task/topology → triggers → repeat → assign agents) plus the existing secrets card. ResearchWizard picks up the same repo step and a hard gate when the workspace has zero agents. - New LoopStaffingStep with three tabs — Individual / Team / Organization — that mix freely per loop; selections persist via new loop_agents / loop_teams / loop_orgs join tables (0035 migration), each cascading on loop_id so hard-delete stays a single-row DELETE. - Backend CreateLoopRequest / UpdateLoopRequest accept the three lists and apply_staffing does a transactional replace-all; list_loops / get_loop hydrate the lists via a flattened LoopWithStaffing response. - LoopsList sidebar gains per-row enable/disable, edit (reopens the wizard prefilled with the current loop, PATCHes on submit), and delete with an inline confirm. - NoAgentsGate blocks launching a loop or research topic from a workspace with no roster; the sidebar `+` buttons also disable with a tooltip pointing at the TEAM tier. Not yet wired: the run driver still fills role slots from the workspace-wide pool; teaching enqueue_iteration to prefer loop_agents/loop_teams/loop_orgs is a follow-up. |
||
|
Omar Sobh
|
637e1bdd69 |
repos: sidebar actions (sync/edit/remove) + edit modal
Sidebar: - Each connection header now has three inline icon buttons: Sync now (spins while in flight), Edit (opens the modal), Remove (opens an inline confirm strip). Removes cascade repos via ON DELETE CASCADE. - The connection's last_sync_error surfaces as a red inline banner under the header — no more 'error status with nowhere to see why'. - Sync is POST /api/repos/connections/:id/sync (already existed); after either sync or delete the sidebar re-fetches so state stays consistent. Edit modal (RepoConnectionEditModal): - Loads GET /api/repos/connections/:id, pre-fills owner/base_url/label - PATCHes only the fields that actually changed; empty string on a Some(&str) field sends explicit null so the backend clears it - Sync-now + Remove reachable from inside the modal too - Rotating the token is out of scope: the modal says as much and points the user at delete + re-create through the wizard (the broker doesn't expose an update path, and rotating in place would require duplicating the whole broker->store_secret flow here) Backend: - GET /api/repos/connections/:id — same ConnectionSummary shape - PATCH /api/repos/connections/:id — owner/base_url use Option<Option<T>> double-nesting so 'omit = leave alone' and 'null = clear' round-trip distinctly through serde - repo_connections::update with COALESCE-per-field so the SQL matches the double-Option semantics without an OR-chain per field |
||
|
Omar Sobh
|
6d087bf537 |
repos: backend — schema, /api/repos routes + GitHub sync provider
Migration 0034: two tables. repo_connections carries the workspace's per-provider config (owner, base_url, label, last_synced_at, last_sync_error) and points at an app_connections row for the PAT. repos is the per-connection cache with (connection_id, external_id) unique so upsert is idempotent across re-syncs. Cascading deletes clean up cleanly on connection removal. cm-secrets grows a FetchAuthorized op — GET with the stored PAT injected as bearer, returns status + JSON body without ever exposing the credential to cm-api. This is the least-privilege door for read-only provider APIs (list repos), distinct from the InvokeHttp path that still requires a single-use approval grant for outbound writes. cm-api::routes::repos wires: - POST /api/repos/connections (broker store_secret + insert both rows + initial sync + mark_synced) - GET /api/repos/connections - DELETE /api/repos/connections/:id - POST /api/repos/connections/:id/sync - GET /api/repos (500 cap, newest provider_updated first) - GET /api/repos/:id (full detail incl. clone_url + html_url) GitHub provider inline for v1 — paginated pull of /orgs/:owner/repos (when owner set) or /user/repos (when absent), 100/page, capped at 20 pages (~2k repos) to keep first-sync latency bounded. Non-2xx surface back to the caller as sync_error; parse failures are best-effort per repo (skipped, logged, don't abort the batch). Gitea + GitLab providers land in a follow-up — mostly URL swap + response-shape adapter. |
||
|
Omar Sobh
|
806ba869e5 |
teams: ephemeral lifecycle for Scheduled + Triggered planner modes
Migration 0033: adds teams.lifecycle ('permanent' | 'ephemeral') and a
topology_runs.team_id back-ref with a partial index for the sibling-in-
flight check.
cm-db repo:
- teams::insert_team_with_lifecycle (insert_team keeps the permanent default)
- topology_runs::enqueue_run_for_team (populates team_id)
- topology_runs::check_ephemeral_teardown — atomic SELECT that only
returns Some when the team is ephemeral AND no siblings are still
queued/running; carries the workspace + bound claw ids for cleanup.
cm-api:
- topology_worker post-terminal hook maybe_teardown_ephemeral_team
runs deprovision_claw on each bound claw (best-effort; failures log
but don't block Postgres deletion), then hard_purge each agent row,
then delete_team.
- routes::teams::build_team_with_lifecycle (build_team keeps default);
run_team enqueues with team_id.
- planner ScaffoldRequest gains mode; lifecycle_for(mode) sets the team
to ephemeral for scheduled + triggered, permanent otherwise.
Frontend MasterPlannerModal passes mode in the scaffold payload so the
backend can derive lifecycle without duplicating the mode taxonomy.
Tests: 3 new (returns claws when no siblings, holds when siblings queued,
ignores permanent teams). 10/10 topology_jobs green; workspace clippy
--tests clean.
|
||
|
Omar Sobh
|
8a4e222aec |
research: auto-transition processing → reviewing on last run
Hooks the topology_worker's post-terminal path into a new notify_run_completed repo helper that atomically transitions the topic processing → reviewing when the completed run has research_topic_id set AND no siblings for that topic are still queued or running. Guarded on status='processing' so a retry, a re-fire, or a topic already past processing are all no-ops. Best-effort at the worker; DB hiccups are logged and never fail the run. The manual /submit-review endpoint stays as an escape hatch for topics that end up parked in processing with nothing to complete (updated the doc comment). |
||
|
Omar Sobh
|
6d1dda6197 |
loops: iteration timeline — GET /api/topology-runs?loop_id=X
Extend the topology-runs list route with an optional loop_id filter that returns iterations for a single loop, newest-iteration-first. Adds the iteration and finished_at columns to the summary (skip-null on the JSON so compares stay compact). Backed by list_by_loop in the repo, which uses the existing topology_runs_loop_idx partial index. LoopsCanvas fetches the runs in parallel with the loop detail and renders an iteration timeline card (iteration #, status pill, start time, duration, run id prefix) between the graph section and the actions row. |
||
|
Omar Sobh
|
973eeb272e |
research: publish approval gate + explicit state transitions
Fourth commit of the Research + Loops arc. Completes the state machine
for research topics with the publish approval gate the spec asked for.
Migration 0032 — research_publish_approvals
Dedicated small table (id, workspace_id, topic_id, requested_by,
status, decided_by/at, created_at). Keeping it separate from the
existing `approvals` table (0001) because that one is tightly coupled
to gated tool calls inside an agent run — session_key + run_id +
action_type + category + payload + preview + requested_by_agent, all
NOT NULL. Forcing those nullable would ripple through cm_safety;
cleaner to give publish approvals their own two-transition state
machine.
New endpoints
POST /api/research/:id/submit-review processing → reviewing
(v1 caller-driven; the
orchestrator hook comes
when we wire actual runs)
POST /api/research/:id/request-publish creates a pending
approval. Rejects with
409 if the topic already
has one open.
GET /api/research/publish-approvals list workspace's pending
POST /api/research/publish-approvals/:id/approve flips approval to
approved + transitions
the topic
reviewing → publishing
(which stamps
published_at)
POST /api/research/publish-approvals/:id/reject stays in reviewing; new
requests allowed
The approve/reject write is an atomic UPDATE ... WHERE status = 'pending';
the decide() repo function returns whether the caller won the race so
concurrent double-approves collapse to a single topic transition.
State machine after this commit:
standby ─POST /start─▶ processing ─POST /submit-review─▶ reviewing
─POST /request-publish + approve─▶ publishing ─(future: artifact
assembly)─▶ published
|
||
|
Omar Sobh
|
4b48c521eb |
loops: backend routes + repo + cron scheduler + HMAC webhook
Third commit of the Research + Loops arc. Lights up loops as durable
recurring topology executions:
GET /api/loops list workspace's loops
POST /api/loops create — returns webhook_token +
signing_key ONCE when webhook trigger
is enabled; never exposed again
GET /api/loops/:id detail
PATCH /api/loops/:id update definition
DELETE /api/loops/:id delete
POST /api/loops/:id/run trigger one iteration NOW
POST /api/loops/:id/enable set enabled=true
POST /api/loops/:id/disable set enabled=false
POST /webhooks/loops/:token public; HMAC-SHA256-verified
Scheduler (cm_runtime::spawn_loop_scheduler) wakes every 10s, queries the
partial index on (next_fire_at) for due loops, enqueues one topology_runs
row per fire with loop_id + iteration + parent_run_id chained back to the
previous iteration. Uses croner via the existing scheduling::next_occurrence
helper. Missed windows fire ONCE and skip the backlog — next_fire_at is
always computed strictly AFTER now(), so a late scheduler doesn't drain a
buildup.
Webhook signatures follow the same pattern as the Stripe billing webhook
(HMAC-SHA256 with constant-time hex compare). Token + signing key are
24-byte OS-RNG values; the URL uses base64-url for the token, and the
signing key is base64-std. Both surface exactly once at create time.
All three fire paths (scheduler, immediate-run, webhook) funnel through
`cm_db::repo::loops::enqueue_iteration` so the invariants stay in one
place. `iters` repeat policy is enforced by the scheduler tick; `until`
and `on_completion` land with the orchestrator hook in commit 4.
Adds cm-llm as a direct cm-api dep, getrandom for the webhook material
generator, and wires the scheduler spawn into the server binary alongside
the resume sweeper and outbox drainer.
|
||
|
Omar Sobh
|
fb047879c2 |
research: backend routes + repo + wizard refine (behind /api/research)
Second commit of the Research + Loops arc. Builds on 0030 by lighting up
the container CRUD, the agent-slot attach/detach, the standby→processing
transition, and the wizard's one-shot LLM refine.
GET /api/research list workspace's topics
POST /api/research create (accepts wizard output)
GET /api/research/:id detail (topic + attached agents)
PATCH /api/research/:id update non-status fields
POST /api/research/:id/agents attach agent (idempotent)
DELETE /api/research/:id/agents/:agent detach
POST /api/research/:id/start standby → processing
POST /api/research/wizard/refine one-shot LLM refine
The refine endpoint accumulates the workspace's default LLM provider's
stream into a JSON object (`{title, description}`) with the tight system
prompt at the top of the module. Same provider routing as agent runs
(via runtime.provider()), so a workspace already using GLM/Kimi gets it
for free.
State machine's remaining transitions (processing → reviewing on last
run_completed; reviewing → publishing via approvals gate) land with the
orchestrator hookup + approvals extension. Publish approval and loops
are separate commits still to come.
Adds cm-llm as a direct cm-api dep (previously only pulled transitively
via cm-runtime) so the refine endpoint can build a ChatRequest. Uses
sqlx::query! for compile-time verification; .sqlx cache generated on
morpheus against a fresh migrated DB.
|
||
|
Omar Sobh
|
04f8302871 |
cm-db: threads — drop unnecessary .iter().copied() on participants slice
Inherited from the a2a merge. Iterating `&[AgentId]` directly yields `&AgentId`; `AgentId::as_uuid(&self)` works fine on that, so the extra `.iter().copied()` was pure noise and tripped clippy's `unnecessary_to_owned` under `-D warnings`. |
||
|
Omar Sobh
|
ebb5b5780b |
chore: cargo fmt --all — clean up a2a merge's fmt violations
The a2a merge (
|
||
|
Omar SobhandClaude Opus 4.8
|
cbfa0ff24f |
feat: agent-to-agent platform on ZeroClaw 0.8.2 — rooms, delegation, A2A ingress
Builds on the v0.8.2 runtime. Four workstreams, all behind the §15 MCP door:
- Group rooms (Phase 1): migration 0026; N-way threads repo with a DM/room
count-guard; chat.send {room} + room.create/invite/leave tools; RoomMessage
-> room.message SSE; /api/claw-chat/rooms* APIs; Observer room badge.
- Per-claw door identity: door caller_agent resolves the X-ZeroClaw-Agent
header (set by the fork) to the specific claw, falling back to roster[0].
- Gated delegation bridge (Phase 3): clawmates__delegate door tool drives a
sibling via the existing /ws/chat ZeroClawDriveExecutor (not A2A); self-deny,
per-workspace hourly budget, audit trail, untrusted-banner result. Native
in-daemon delegation stays off (it would bypass the door).
- A2A tenant ingress (Phase 2): migration 0027 (workspace_a2a + a2a_tokens);
runtime_provision enable_a2a_server/publish_claw; routes/a2a.rs tenant-aware
proxy (per-workspace tokens, injected internal bearer, daemon stays internal,
cards URL-rewritten to the cm-api edge); a2a.invoked taxonomy.
Tests: cm-db room repos, cm-runtime chat tools, door units. sqlx cache updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.8
|
96d1409a23 |
style: cargo fmt --all
Apply rustfmt (toolchain 1.96.0) to satisfy the CI Format check. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
9263418fcb |
Fleet: per-node dev-tool version cards + nightly latest-check (Phase 1, read-only)
Each node card now shows installed versions of Docker / Claude Code / Kimi / GLM /
Ollama (conditional per node) under the ssh card, with an "update available" badge.
- daemon: probe_tools() finds docker/claude/kimi-cli/ollama across candidate bin dirs,
extracts semver from --version, reports {"t":"node_tools",...} on connect + every 15m.
- migration node_tools + tool_latest; cm-db repo node_tools (upsert/list/latest).
- cm-api: fleet.rs NodeTools uplink → upsert; tool_versions.rs spawn_latest_checker
(24h, npm/pypi/github; docker display-only); GET /api/nodes/{id}/tools (glm mirrors
claude). Spawned in clawmates-server.
- frontend: NodeTools cards on each HostCard with the ↑latest badge.
Phase 2 (one-click update execution) intentionally deferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
3554a3aaf2 |
CI: remove k8s stages, fix the Docker-level pipeline green
Survey + fixes so the pipeline passes at the Docker level (no k8s).
- Remove k8s: drop the `sandbox-k8s` job (kind/Calico/--features k8s-tests) and the
"Helm chart lints" gate step. release.yml was already k8s-clean.
- Rust job:
- `cargo fmt --all` — fix pre-existing formatting drift (fmt --check was failing).
- clippy -D warnings: fix 3 lib warnings (cm-brain sort_by_key→Reverse, cm-api
fleet.rs doc list indentation, node_rules map_or→is_none_or).
- Regenerate the .sqlx offline cache (was missing the cm-runtime run_loop test
query → offline compile failed). DB-backed tests use testcontainers at runtime.
- Set SQLX_OFFLINE=true on the rust + e2e jobs so query! macros compile against
the committed cache deterministically (no DB needed at compile time).
- Frontend job:
- Fix the 1 ESLint error (useAgentTelemetry: no setState-synchronously-in-effect;
tag the slice with agentId + derive null on mismatch).
- Fix 2 stale panel-params tests (`terminal` is a valid app id now; assert the
current APP_IDS + use a genuinely-unknown id for the reject case).
Verified locally: fmt clean, clippy --all-targets -D warnings clean (offline),
frontend lint 0 errors, tsc clean, 86/86 frontend tests pass, build OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
c94784bab2 |
Fleet: actionable executions — rules engine + metrics-aware placement (Phase 2)
Turn the Beszel-tapped metrics into a self-managing loop. - migration node_rules (workspace/node-scoped: metric op threshold, for_seconds, action JSONB, last_fired). - cm-db: repo/node_rules.rs (CRUD + list_enabled); node_metrics::eval_all merges Beszel + heartbeat scalars per node + a headroom() heuristic; nodes::status_of; heartbeat now PRESERVES a `draining` status across heartbeats (so a cordon sticks). - cm-api: node_rules.rs evaluator (spawn_evaluator, 20s) — when a metric condition holds for the rule's window it fires drain / undrain / alert (in-memory sustained + cooldown tracking, modeled on the node sweeper); routes/beszel.rs rules CRUD (GET/POST/PATCH/DELETE /api/fleet/rules); spawned in clawmates-server. - cm-runtime: placement_node() is metrics-aware — a `draining` node stops receiving new agent sandboxes (falls back to local), so the drain rule is actionable. - frontend: FleetRules section in the Local view — build rules (node · metric · op · threshold · duration → action), toggle/delete, with fired-history. The loop: hot/overloaded node → rule drains it → placement avoids it → recovers → undrain rule brings it back. Deployed; node_rules migration applied. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
36a227566b |
Fleet: Beszel hub integration — rich per-node metrics + per-node monitor (Phase 1)
Tap each node's Beszel metrics (GPU/temps/disk-IO/network/per-container — beyond
our basic heartbeat) by reading the workspace's Beszel hub. The agents run in
WS-only mode with no locally-readable socket, so (per the de-risk) the server taps
the hub's PocketBase API instead of the daemon reading agents — no daemon changes.
- migrations: workspace_beszel (BYO hub URL + login, server-side only, mirrors the
Tailscale BYO pattern) + node_metrics (latest scalar columns + JSONB blob).
- cm-db: repo/fleet_beszel.rs, repo/node_metrics.rs; nodes SELECT joins node_metrics
(gpu_pct/temp_max surfaced on node_json for the live cards).
- cm-api: beszel.rs client (auth-with-password, poll `systems`, map to nodes by
hostname, upsert metrics) + a 15s spawn_poller; routes/beszel.rs (connect/status/
disconnect + GET /api/nodes/{id}/metrics with history proxied live from the hub).
- frontend: HostCard gains a GPU/temp readout + a Monitor button; NodeMonitor is a
full-width per-node page (current panel + CPU/mem/GPU/temp/net/disk charts from the
hub's 1m history); a "Beszel monitoring" connect form in the Local view.
Reachability confirmed: gw-04 → the hub over the tailnet (100.123.224.84:8090). Needs
the user to connect their hub login to activate the poller.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
cf6c331b02 |
Fleet: node hostname/IP on register + node terminal in the infra computer
Hostname/IP: - Daemon reports the machine's hostname (sysinfo) + primary outbound IPv4 on each heartbeat. migrations/0021 adds nodes.hostname/local_ip; cm-db heartbeat stores them; node JSON exposes them. Cards now title on the real hostname (falling back to name) + show the IP, instead of the "New node" placeholder. `name` stays user-overridable (rename). Terminal moved into the pull-out computer (no more per-card modal): - New infra computer app NodeTerminalApp (computer/apps/infra) — xterm bridged to a node's host shell over the node control channel, filling the app window (mirrors the agent Terminal's layout + ResizeObserver). Added "terminal" to the INFRA_CATALOG grid; a ?node= panel param targets a specific node (picker when unset). Clicking Terminal on a node card now opens the infra computer to that node's shell instead of a separate full-screen window. Deleted NodeTerminal.tsx. Rebuilt + re-hosted both daemon binaries (hostname change). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
fb59378aa2 |
Fleet P2b: run agent sandboxes on connected nodes (RemoteDriver + placement)
Agents can now provision their sandbox on a connected fleet node instead of the gateway host. Local stays the strict default, so existing agents are byte-for- byte unaffected until explicitly placed elsewhere. Security parity: the daemon links the REAL cm-sandbox DockerDriver and runs the typed container ops (sb_provision/sb_exec/sb_destroy/sb_health/sb_list) through it — identical hardening (cap-drop ALL, seccomp, no-net, read-only, non-root) to local sandboxes. cm-sandbox spec types are now Serialize/Deserialize so the spec crosses the channel. - cm-api: RemoteDriver (impl SandboxDriver over the node channel) + HubDriverProvider (impl cm_runtime::NodeDriverProvider, hands out a driver only for connected nodes via a sync online set) + NodeHub.call/is_connected. AppState.with_node_hub so the hub is shared with the placement provider. - cm-runtime SandboxManager: driver_for(node_id) routes by the recorded agent_containers.node_id (local default = existing driver, identical path); placement_node() reads the workspace setting and falls back to local if the node is offline; exec/release route accordingly. NodeDriverProvider trait. - DB: 0020_workspace_placement + repo (for_agent/get/set/clear). - main.rs: build the NodeHub first; inject HubDriverProvider into the agent manager + share the hub with AppState. - API+UI: GET/PUT /api/fleet/placement + a "Run agents on: Local / <node>" selector in the Fleet overview. Note: a node must be able to pull the agent image (the daemon docker-pulls it); interactive PTY for agent containers on remote nodes is not wired (Terminal app stays local) — the in-dashboard node shell already covers host access. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
7332d69f8a |
Fleet P1: BYO Tailscale + network metrics, Tailscale SSH, exec hardening
Security hardening: - The gateway no longer sends arbitrary shell to nodes. The WSS exec op is replaced by a typed `verify` op the daemon runs itself (fixed host+docker check); future container ops are typed too. cm-api NodeHub.verify() + the daemon's handle_command only dispatches vetted ops. BYO Tailscale: - migrations/0019_workspace_tailscale.sql + cm-db fleet_tailscale repo (store the user's Tailscale API key + tailnet, server-side only). - cm-api routes/tailscale.rs: POST/GET/DELETE /api/fleet/tailscale + GET /api/fleet/tailscale/devices (proxies api.tailscale.com device list). - Daemon: --tailscale-authkey → `tailscale up --authkey … --ssh` (enables Tailscale SSH for keyless user access); else `tailscale set --ssh=true`. Reports its tailscale IP (already). UI: - Fleet overview gains a Tailscale section: connect (key+tailnet) + live tailnet device status (online/last-seen/IP/os). Node cards show a copyable Tailscale SSH target (ssh <ip>). Remaining: P2 — RemoteDriver + placement (run agents on nodes) and the in-UI remote terminal (PTY proxied over the WSS channel). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |