osobh/clawmates
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

slice 9 cleanup: drop legacy research/loops backend + tables

Browse Source 
Retires the legacy research/loops backend after the missions arc
(slices 1-9) fully replaced it. Frontend cutover was 4663348; this
commit finishes the job on the backend + database.

Migration:
  - 0053_drop_legacy_research_loops.sql — drops the 8 legacy tables
    (research_topics, research_topic_agents, research_outcomes,
    research_publish_approvals, loops, loop_agents, loop_orgs,
    loop_teams) and the 3 topology_runs FK columns
    (research_topic_id, loop_id, iteration). parent_run_id stays;
    recursive_exec still uses it.

Files deleted (11):
  - crates/cm-api/src/routes/{research,loops,research_setup,
    research_pipeline,wizard_repo,probe}.rs
  - crates/cm-api/src/research_container.rs
  - crates/cm-db/src/repo/{research_topics,research_outcomes,
    research_publish_approvals,loops}.rs
  - crates/cm-runtime/src/loops.rs
  - crates/cm-api/tests/research_publish_role.rs

Files edited:
  - crates/cm-api/src/lib.rs — dropped 20 legacy route registrations
    (all /api/research/* + /api/loops/* + /webhooks/loops + probe)
    and module decls
  - crates/cm-api/src/topology_worker.rs — deleted legacy dispatch
    (freeze_research_outcome, advance_loop_after_completion,
    continue_initial_burst, maybe_transition_research_topic,
    parse_reorder_rationale, per-topic/loop gateway resolver).
    reap_stuck_runs now keys on mission_id (not topic_id).
    Executor path unconditionally uses ZeroClawDriveExecutor::from_env
    — mission_orchestrator provisions each claw as an agent inside
    the shared runtime via RuntimeProvisioner, so per-team gateway
    resolution is no longer applicable.
  - crates/cm-api/src/routes/topology.rs — deleted container-log SSE
    endpoint (research/loop-specific), dropped loop_id filter and
    iteration field from ListRunsQuery/RunSummary
  - crates/cm-api/src/routes/world.rs — removed
    active_research_topics/active_loops/preseed_repo_paths;
    World SSE no longer emits repo:{topic}/loop:{id} landmark orbs
    (follow-up task #21 tracks adding mission:{id} equivalents)
  - crates/cm-api/src/runtime_provision.rs — removed now-unused
    mint_workspace_service_token
  - crates/cm-db/src/repo/topology_runs.rs — removed 9 legacy
    helpers (research_topic_id lookup, loop_id_for_run,
    iteration_for_run, active_runs_for_research_topic, etc.)
  - crates/cm-db/src/repo/teams.rs — removed 4 dead helpers
    (team_for_loop, team_for_research_topic + setters)
  - crates/cm-api/tests/topology_jobs.rs — removed loop/topic
    tests, dropped enqueue_run_with_topic helper
  - crates/bins/clawmates-server/src/main.rs — removed
    spawn_loop_scheduler call
  - crates/cm-api/src/routes/mod.rs, crates/cm-db/src/repo/mod.rs,
    crates/cm-runtime/src/lib.rs — module decls stripped

sqlx cache: regenerated against post-migration schema
  (71 files changed, ~+70 / -8896 net)

Test/build: SQLX_OFFLINE=true cargo check --workspace clean;
cargo test --workspace --no-run clean.

Follow-up (task #21): World view lost the in-flight-work landmarks
when repo:{topic} / loop:{id} orbs disappeared. Add mission:{id}
orbs as the missions-era replacement.
This commit is contained in:
Omar Sobh
2026-07-19 18:37:24 -07:00
parent 56201a6985
commit fdb8cfeecc
71 changed files with 70 additions and 8896 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/cm-api/src/routes/loops.rs
-898
View File
@@ -1,898 +0,0 @@
//! Loop endpoints — CRUD, enable/disable, immediate-run, and the public
//! webhook receiver.
//!
//! GET /api/loops list workspace's loops
//! POST /api/loops create
//! GET /api/loops/:id detail
//! PATCH /api/loops/:id update definition
//! DELETE /api/loops/:id delete
//! POST /api/loops/:id/run trigger one iteration NOW (bypass schedule)
//! POST /api/loops/:id/enable set enabled=true; recomputes next_fire_at
//! POST /api/loops/:id/disable set enabled=false
//! POST /webhooks/loops/:token public; HMAC-SHA256-verified via
//! X-Loop-Signature: sha256=<hex>
use axum::body::Bytes;
use axum::extract::{Path, State};
use axum::http::{HeaderMap, StatusCode};
use axum::Json;
use base64::Engine;
use cm_runtime::scheduling::next_occurrence;
use hmac::{Hmac, Mac};
use serde::{Deserialize, Serialize};
use serde_json::Value;
use sqlx::PgPool;
use time::OffsetDateTime;
use uuid::Uuid;
/// Root of per-loop state dirs on the host. Same overridable env pattern
/// as research_workspace_root — prod points at the bind-mounted volume
/// `/var/lib/clawmates-loops` on gw-04.
fn loop_state_root() -> std::path::PathBuf {
std::env::var("CLAWMATES_LOOPS_STATE_ROOT")
.map(std::path::PathBuf::from)
.unwrap_or_else(|_| std::path::PathBuf::from("/var/lib/clawmates-loops"))
}
/// Best-effort spawn of the per-loop team container before an iteration
/// is enqueued. Idempotent — an already-running container is just
/// reattached. Failures (docker unreachable, image missing) log and
/// return without blocking the run; the topology_worker will fall back
/// to the workspace-wide gateway. Records the container name + URL on
/// the loop row on first success so subsequent fires skip re-writing.
/// Build the task string an iteration will actually run.
///
/// - Standalone loops (no source research topic bound): returns
/// `task_template` verbatim, matching legacy behavior.
/// - Loops bound to a research topic: fetches the topic's latest
/// research_outcome and prepends a block of the shape:
///
/// ```text
/// RESEARCH ARTIFACT (integration plan you're executing):
/// <markdown>
/// ITERATION FOCUS: next unconsumed INT-XX in order. If prereqs are
/// unmet, work on the smallest unblocking INT-XX. Log
/// COMPLETED: INT-<NN> at the end so the loop can advance.
/// ORIGINAL TASK:
/// <task_template>
/// ```
///
/// The topology_worker's completion hook (P3) parses the COMPLETED
/// marker to update `consumed_int_ids`.
/// One-shot compose + enqueue for a loop iteration. Reads the loop's
/// kind from the DB and dispatches: kind='exec' uses
/// compose_iteration_task (INT-consumption prepend); kind='research'
/// uses compose_research_iteration_task AND sets research_topic_id on
/// the topology_run so freeze_research_outcome writes a new outcome
/// version at completion. Returns the run id. Standalone exec loops
/// (no source topic) still work — the compose helper returns the
/// task_template verbatim.
pub async fn compose_and_enqueue_iteration(
pool: &sqlx::PgPool,
loop_id: Uuid,
workspace_id: Uuid,
graph: &Value,
parent_run_id: Option<Uuid>,
task_template_override: Option<&str>,
) -> Result<Uuid, cm_db::DbError> {
// Kind is the source of truth — task_template alone isn't enough
// to know whether to write to research_outcomes.
let (kind, source_topic, template) =
match cm_db::repo::loops::kind_and_binding(pool, loop_id).await? {
Some(t) => t,
None => return Err(cm_db::DbError::NotFound),
};
let template_ref = task_template_override.unwrap_or(&template);
let iter = cm_db::repo::loops::next_iteration(pool, loop_id).await?;
if kind == "research" {
// Research-kind requires a bound topic (schema-level constraint
// isn't enforced yet — surface the misconfiguration explicitly).
let Some(topic_id) = source_topic else {
return Err(cm_db::DbError::NotFound);
};
// Clone + spawn container BEFORE enqueuing so the run has real
// repo files + an isolated daemon to hit. Idempotent — the
// second iteration reattaches to the existing container. Runs
// even when the topic has no repo (harmless no-op).
crate::routes::research_setup::prepare_topic_runtime(pool, workspace_id, topic_id).await;
// D1 fold — advance the topic's status column when a fresh
// research iteration goes out so the canvas's classic state-
// machine card reflects reality. Only fire the standby →
// processing transition; later iterations already sit in
// processing/reviewing/publishing and set_status is a no-op
// when the status is already the target.
let _ = cm_db::repo::research_topics::set_status_if(
pool,
topic_id,
workspace_id,
"standby",
"processing",
)
.await;
let task = compose_research_iteration_task(pool, topic_id, template_ref).await;
cm_db::repo::loops::enqueue_iteration_with_topic(
pool,
cm_db::repo::loops::IterationEnqueue {
loop_id,
workspace_id,
task: &task,
graph,
iteration: iter,
parent_run_id,
research_topic_id: Some(topic_id),
},
)
.await
} else {
let task = compose_iteration_task(pool, loop_id, template_ref).await;
cm_db::repo::loops::enqueue_iteration(
pool,
loop_id,
workspace_id,
&task,
graph,
iter,
parent_run_id,
)
.await
}
}
/// Build the coordinator prompt for a kind='research' loop iteration.
/// Wraps the topic's description + outcome_kind + prior artifact
/// version pointer into an instruction that asks the team to refresh
/// the plan (survey new sources, revise existing INTs, add new ones)
/// and emit the updated artifact using the same section shape. The
/// completion hook's `freeze_research_outcome` will insert a new
/// versioned row automatically because the topology_run carries
/// research_topic_id.
pub async fn compose_research_iteration_task(
pool: &PgPool,
topic_id: Uuid,
task_template: &str,
) -> String {
let (title, description, outcome_kind, prior_version) =
match cm_db::repo::research_topics::get_any_workspace(pool, topic_id).await {
Ok(Some(t)) => {
let prior = cm_db::repo::research_outcomes::latest(pool, topic_id)
.await
.unwrap_or(None)
.map(|o| o.version)
.unwrap_or(0);
(t.title, t.description, t.outcome_kind, prior)
}
_ => return task_template.to_string(),
};
format!(
"RESEARCH LOOP ITERATION\n\
=======================\n\
Topic: {title}\n\
Outcome kind: {outcome_kind}\n\
Prior artifact version: v{prior_version} (0 = fresh)\n\n\
DESCRIPTION:\n{description}\n\n\
AUTONOMY CONTRACT (READ FIRST):\n\
- This is a scheduled autonomous run. NO HUMAN WILL ANSWER YOU.\n\
- Do NOT ask 'Should I proceed?' or 'Which approach?' — proceed with\n\
your best judgment and produce the artifact.\n\
- You MUST emit the completed artifact as your final message.\n\
Failure to emit = the entire loop iteration is wasted.\n\n\
YOUR JOB THIS ITERATION:\n\
- Refresh the research — pull in any new papers / findings since v{prior_version}.\n\
- Update the artifact using the SAME section structure the outcome_kind\n\
requires (e.g. integrations kind = executive summary + INT-XX cards).\n\
- Preserve stable ids (INT-01 stays INT-01 across versions). If an item\n\
is superseded, mark it {{deprecated: <reason>}} rather than deleting so\n\
downstream coding loops that already consumed it don't lose context.\n\
- Add NEW items with new ids continuing from the last used number.\n\
- Cite what you can verify. When you can't cite a specific paper or\n\
benchmark, write `[claim needs verification]` inline and MOVE ON — do\n\
not stall the loop asking a human for permission. The next iteration\n\
can strengthen citations; a written v{} with rough citations beats a\n\
blocked v{} waiting for approval.\n\
- Do NOT fabricate concrete paper titles, author names, or DOIs.\n\
Vague-but-honest ('a 2024 HNSW improvement paper') beats invented specifics.\n\n\
The workspace's final synthesis is captured as research_outcomes v{}. \
Downstream on_artifact_update loops will wake up on this write.\n\n\
LOOP OPERATOR NOTES:\n{task_template}\n",
prior_version + 1,
prior_version + 1,
prior_version + 1
)
}
pub async fn compose_iteration_task(pool: &PgPool, loop_id: Uuid, task_template: &str) -> String {
let ctx = cm_db::repo::loops::source_research_context(pool, loop_id)
.await
.unwrap_or(None);
let Some((topic_id, consumed, current_idx)) = ctx else {
return task_template.to_string();
};
let outcome = match cm_db::repo::research_outcomes::latest(pool, topic_id).await {
Ok(Some(o)) => o,
_ => return task_template.to_string(),
};
let consumed_list = if consumed.is_empty() {
"(none yet)".to_string()
} else {
consumed.join(", ")
};
format!(
"RESEARCH ARTIFACT (integration plan you're executing, v{}):\n\
--- BEGIN ARTIFACT ---\n{}\n--- END ARTIFACT ---\n\n\
ITERATION FOCUS:\n\
- You are on iteration index {}.\n\
- Already completed: {}.\n\
- Address the NEXT unconsumed INT-XX item in the artifact, in order.\n\
- If the next item has unmet prerequisites, work on the smallest\n\
unblocking INT-XX instead. When you reorder, emit a line\n\
`REORDER: <one-sentence rationale>` at the top of your first\n\
substantive turn — the loop indexes these for a review timeline.\n\
- Emit `COMPLETED: INT-<NN>` on its own line at the end of the run\n\
when the item is done — the loop advances on that marker.\n\
- Both markers must appear literally with the colon (no bold, no\n\
code fence); the parser is line-based.\n\n\
ORIGINAL TASK TEMPLATE:\n{}\n",
outcome.version, outcome.body_md, current_idx, consumed_list, task_template
)
}
async fn ensure_loop_container(pool: &PgPool, workspace_id: Uuid, loop_id: Uuid) {
let docker = match crate::research_container::connect() {
Ok(d) => d,
Err(e) => {
eprintln!("loops::ensure_loop_container({loop_id}): docker connect failed: {e}");
return;
}
};
let state_root = loop_state_root().join(loop_id.to_string()).join("state");
let mcp_bearer = crate::runtime_provision::mint_workspace_service_token(
pool,
cm_domain::WorkspaceId::from(workspace_id),
)
.await
.map_err(|e| {
eprintln!("loops::ensure_loop_container({loop_id}): mint MCP bearer failed: {e}");
e
})
.ok();
let spawned = match crate::research_container::spawn_loop(
&docker,
loop_id,
&state_root,
mcp_bearer.as_deref(),
)
.await
{
Ok(s) => s,
Err(e) => {
eprintln!("loops::ensure_loop_container({loop_id}): spawn failed: {e}");
return;
}
};
if let Err(e) = cm_db::repo::loops::set_zeroclaw_container(
pool,
loop_id,
workspace_id,
&spawned.name,
&spawned.gateway_url,
)
.await
{
eprintln!("loops::ensure_loop_container({loop_id}): persist failed: {e:?}");
}
}
use crate::{ApiError, AppState, Authed};
#[derive(Deserialize)]
pub struct CreateLoopRequest {
pub title: String,
pub description: String,
pub graph: Value,
pub task_template: String,
/// {cron?: '0 */6 * * *', on_completion?: bool, webhook_enabled?: bool}
#[serde(default)]
pub triggers: Value,
/// {kind: 'infinite' | 'iters' | 'until', n?: int}
#[serde(default = "default_repeat")]
pub repeat_policy: Value,
#[serde(default)]
pub agents: Vec<AgentSlotInput>,
#[serde(default)]
pub teams: Vec<Uuid>,
#[serde(default)]
pub orgs: Vec<Uuid>,
/// Optional research topic id. When set, each iteration prepends the
/// topic's latest research_outcome markdown + a "focus on next
/// unconsumed INT" instruction to the coordinator task. Migration
/// 0042 added the pointer column + consumed_int_ids tracking.
#[serde(default)]
pub source_research_topic_id: Option<Uuid>,
}
fn default_repeat() -> Value {
serde_json::json!({"kind": "infinite"})
}
#[derive(Deserialize)]
pub struct AgentSlotInput {
pub agent_id: Uuid,
#[serde(default)]
pub role_slot: Option<String>,
}
#[derive(Serialize)]
pub struct LoopCreated {
pub id: Uuid,
/// Set when `triggers.webhook_enabled == true`. The full URL is
/// `<origin>/webhooks/loops/<webhook_token>`; the signing key is
/// returned exactly once at creation and never surfaced again.
#[serde(skip_serializing_if = "Option::is_none")]
pub webhook_token: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub webhook_signing_key: Option<String>,
}
fn parse_triggers(v: &Value) -> Option<Triggers> {
serde_json::from_value(v.clone()).ok()
}
#[derive(Deserialize)]
struct Triggers {
#[serde(default)]
cron: Option<String>,
#[serde(default)]
#[allow(dead_code)]
on_completion: bool,
#[serde(default)]
webhook_enabled: bool,
/// NEW — number of iterations to fire back-to-back at loop-create
/// time. Enqueue path: fire once immediately, then chain each
/// subsequent one via on_completion until the burst quota is
/// exhausted (tracked in run metadata). Defaults to 0 for existing
/// loops (no auto-fire); new wizards typically set 1 (D1: every
/// runnable thing runs at least once).
#[serde(default)]
initial_burst: u32,
/// NEW — when this loop is bound to a source_research_topic and
/// that topic gets a fresh research_outcomes row (via
/// freeze_research_outcome), enqueue one iteration on this loop.
/// Coalesced with any in-flight run (D3: coordinator resolves;
/// no race, just one wake per artifact update). Read directly
/// from the loops.triggers jsonb by loops_awaiting_topic — no
/// need for the Rust parser to hold it after the fact.
#[serde(default)]
#[allow(dead_code)]
on_artifact_update: bool,
}
fn make_webhook_material() -> (String, String) {
// 24 bytes ≈ 192 bits of entropy each; URL-safe base64 for the token,
// standard base64 for the signing key.
let mut token_buf = [0u8; 24];
let mut key_buf = [0u8; 24];
// getrandom is already in the dep tree via base64/hmac/etc; failure
// (broken kernel RNG) is fatal enough that unwrapping is fine here.
getrandom::getrandom(&mut token_buf).expect("OS RNG");
getrandom::getrandom(&mut key_buf).expect("OS RNG");
let token = base64::engine::general_purpose::URL_SAFE_NO_PAD.encode(token_buf);
let key = base64::engine::general_purpose::STANDARD_NO_PAD.encode(key_buf);
(token, key)
}
fn compute_next_fire(triggers: &Value) -> Option<OffsetDateTime> {
let t = parse_triggers(triggers)?;
let pattern = t.cron?;
if pattern.trim().is_empty() {
return None;
}
next_occurrence(pattern.trim(), OffsetDateTime::now_utc()).ok()
}
pub async fn create_loop(
State(state): State<AppState>,
Authed(user): Authed,
Json(body): Json<CreateLoopRequest>,
) -> Result<(StatusCode, Json<LoopCreated>), ApiError> {
if body.title.trim().is_empty() || body.task_template.trim().is_empty() {
return Err(ApiError::BadRequest);
}
// Empty-roster gate: a workspace with zero agents has nothing to staff
// the loop with — refuse before any DB writes. Frontend already
// disables the create button in this state; this closes the direct-POST
// hole so we don't materialize orphan loops that never fire.
if cm_db::repo::agents::count_active(&state.pool, user.workspace_id).await? == 0 {
return Err(ApiError::Conflict);
}
let webhook_enabled = parse_triggers(&body.triggers)
.map(|t| t.webhook_enabled)
.unwrap_or(false);
let (webhook_token, webhook_signing_key) = if webhook_enabled {
let (t, k) = make_webhook_material();
(Some(t), Some(k))
} else {
(None, None)
};
let next_fire_at = compute_next_fire(&body.triggers);
let id = cm_db::repo::loops::create(
&state.pool,
cm_db::repo::loops::NewLoop {
workspace_id: user.workspace_id.as_uuid(),
title: body.title.trim(),
description: body.description.trim(),
graph: &body.graph,
task_template: body.task_template.trim(),
triggers: &body.triggers,
repeat_policy: &body.repeat_policy,
enabled: true,
next_fire_at,
webhook_token: webhook_token.as_deref(),
webhook_signing_key: webhook_signing_key.as_deref(),
created_by: user.user_id.as_uuid(),
},
)
.await?;
apply_staffing(&state.pool, id, &body.agents, &body.teams, &body.orgs).await?;
// Bridge to research (option C — snapshot in task_template + save
// pointer so a refresh can pull latest artifact into subsequent
// iterations). Ownership-checked via research_topics::get so we
// can't be tricked into pointing at another workspace's topic.
if let Some(topic_id) = body.source_research_topic_id {
let topic =
cm_db::repo::research_topics::get(&state.pool, topic_id, user.workspace_id.as_uuid())
.await?
.ok_or(ApiError::NotFound)?;
if let Err(e) = cm_db::repo::loops::set_source_research_topic(
&state.pool,
id,
user.workspace_id.as_uuid(),
Some(topic.id),
)
.await
{
eprintln!("loops::create: bind source research topic failed: {e:?}");
}
}
// Fire the initial burst if the triggers request it. Extracted so
// materialize_topic_loops (the wizard-materialized loops path) can
// reuse the same logic — previously the burst logic lived only in
// this handler and wizard-created loops never fired their first
// iteration.
fire_initial_burst_if_set(
&state.pool,
user.workspace_id.as_uuid(),
id,
&body.triggers,
body.task_template.trim(),
&body.graph,
next_fire_at,
)
.await;
Ok((
StatusCode::CREATED,
Json(LoopCreated {
id,
webhook_token,
webhook_signing_key,
}),
))
}
/// Fire the initial_burst if the loop's triggers request one. On the
/// first fire, ensures the per-loop container is spawned and (for
/// kind='research' loops) that the topic's repo is cloned and the
/// topic container is up. Sets `initial_burst_remaining` to
/// `burst - 1` so the completion hook can continue the chain.
/// Best-effort: a docker or DB hiccup on the FIRST fire logs but the
/// loop row still lives — cron / on_artifact_update / webhook can
/// still fire it later.
pub async fn fire_initial_burst_if_set(
pool: &sqlx::PgPool,
workspace_id: Uuid,
loop_id: Uuid,
triggers: &Value,
task_template: &str,
graph: &Value,
next_fire_at: Option<OffsetDateTime>,
) {
let parsed = parse_triggers(triggers);
let initial_burst = parsed.as_ref().map(|t| t.initial_burst).unwrap_or(0);
let chain_on_completion = parsed.as_ref().map(|t| t.on_completion).unwrap_or(false);
if initial_burst == 0 {
return;
}
ensure_loop_container(pool, workspace_id, loop_id).await;
match compose_and_enqueue_iteration(
pool,
loop_id,
workspace_id,
graph,
None,
Some(task_template),
)
.await
{
Ok(run_id) => {
let _ = cm_db::repo::loops::mark_fired(pool, loop_id, run_id, next_fire_at).await;
let remaining = initial_burst.saturating_sub(1) as i32;
if remaining > 0 || chain_on_completion {
let _ =
cm_db::repo::loops::set_initial_burst_remaining(pool, loop_id, remaining).await;
}
}
Err(e) => eprintln!("fire_initial_burst_if_set({loop_id}): enqueue failed: {e:?}"),
}
}
async fn apply_staffing(
pool: &sqlx::PgPool,
loop_id: Uuid,
agents: &[AgentSlotInput],
teams: &[Uuid],
orgs: &[Uuid],
) -> Result<(), ApiError> {
let slots: Vec<cm_db::repo::loops::AgentSlot> = agents
.iter()
.map(|a| cm_db::repo::loops::AgentSlot {
agent_id: a.agent_id,
role_slot: a.role_slot.clone(),
})
.collect();
cm_db::repo::loops::set_agents(pool, loop_id, &slots).await?;
cm_db::repo::loops::set_teams(pool, loop_id, teams).await?;
cm_db::repo::loops::set_orgs(pool, loop_id, orgs).await?;
Ok(())
}
#[derive(Serialize)]
pub struct LoopWithStaffing {
#[serde(flatten)]
pub inner: cm_db::repo::loops::Loop,
pub agents: Vec<cm_db::repo::loops::AgentSlot>,
pub teams: Vec<Uuid>,
pub orgs: Vec<Uuid>,
}
async fn hydrate_staffing(
pool: &sqlx::PgPool,
inner: cm_db::repo::loops::Loop,
) -> Result<LoopWithStaffing, ApiError> {
let id = inner.id;
let agents = cm_db::repo::loops::agents(pool, id).await?;
let teams = cm_db::repo::loops::teams(pool, id).await?;
let orgs = cm_db::repo::loops::orgs(pool, id).await?;
Ok(LoopWithStaffing {
inner,
agents,
teams,
orgs,
})
}
pub async fn list_loops(
State(state): State<AppState>,
Authed(user): Authed,
) -> Result<Json<Vec<LoopWithStaffing>>, ApiError> {
let loops = cm_db::repo::loops::list(&state.pool, user.workspace_id.as_uuid()).await?;
let mut out = Vec::with_capacity(loops.len());
for l in loops {
out.push(hydrate_staffing(&state.pool, l).await?);
}
Ok(Json(out))
}
pub async fn get_loop(
State(state): State<AppState>,
Authed(user): Authed,
Path(id): Path<Uuid>,
) -> Result<Json<LoopWithStaffing>, ApiError> {
let inner = cm_db::repo::loops::get(&state.pool, id, user.workspace_id.as_uuid())
.await?
.ok_or(ApiError::NotFound)?;
Ok(Json(hydrate_staffing(&state.pool, inner).await?))
}
#[derive(Serialize)]
pub struct LoopProgress {
pub loop_id: Uuid,
pub source_topic_id: Uuid,
pub source_topic_title: String,
pub source_outcome_version: i32,
pub consumed_count: usize,
pub total_int_count: usize,
pub current_int_index: i32,
/// Recent coordinator-issued reorders on this loop — newest first,
/// capped at 5 so the sidebar card stays compact. Full history is
/// on the loop row's reorder_events column.
pub recent_reorders: Vec<serde_json::Value>,
}
/// `GET /api/loops/progress` — bulk progress read for every loop in the
/// workspace that's bound to a research topic. Skips standalone loops
/// entirely (empty entry). Parses INT-XX ids from the source outcome's
/// markdown to compute the total; consumed count comes straight from
/// `consumed_int_ids`. Used by the loops sidebar to render an
/// "N/M INTs" pill on each source-bound card.
///
/// Cost: one query for the loops list + one outcome fetch per unique
/// source topic (memoized in the loop below). No N+1 on the topic
/// lookup when many loops share a source.
pub async fn list_progress(
State(state): State<AppState>,
Authed(user): Authed,
) -> Result<Json<Vec<LoopProgress>>, ApiError> {
let loops = cm_db::repo::loops::list(&state.pool, user.workspace_id.as_uuid()).await?;
let mut by_topic: std::collections::HashMap<Uuid, (String, i32, usize)> =
std::collections::HashMap::new();
let mut out = Vec::new();
for l in &loops {
let ctx = match cm_db::repo::loops::source_research_context(&state.pool, l.id).await {
Ok(Some(c)) => c,
_ => continue,
};
let (topic_id, consumed, current_idx) = ctx;
let (title, version, total) = match by_topic.get(&topic_id) {
Some(cached) => cached.clone(),
None => {
// Ownership check via get + then count INTs in the latest
// outcome. Any failure downgrades to (title, 0, 0) so the
// pill still renders — showing 3/0 is better than 500ing
// the whole list.
let topic = match cm_db::repo::research_topics::get(
&state.pool,
topic_id,
user.workspace_id.as_uuid(),
)
.await
{
Ok(Some(t)) => t,
_ => continue,
};
let outcome = cm_db::repo::research_outcomes::latest(&state.pool, topic_id)
.await
.unwrap_or(None);
let (version, total) = match &outcome {
Some(o) => (o.version, count_int_ids(&o.body_md)),
None => (0, 0),
};
let cached = (topic.title.clone(), version, total);
by_topic.insert(topic_id, cached.clone());
cached
}
};
let recent = cm_db::repo::loops::recent_reorders(&state.pool, l.id, 5)
.await
.unwrap_or_default();
out.push(LoopProgress {
loop_id: l.id,
source_topic_id: topic_id,
source_topic_title: title,
source_outcome_version: version,
consumed_count: consumed.len(),
total_int_count: total,
current_int_index: current_idx,
recent_reorders: recent,
});
}
Ok(Json(out))
}
/// Count unique INT-<number> ids in a markdown blob. Case-insensitive,
/// tolerates prefixes like `### INT-01` and inline references. Same
/// permissive matcher used by the completion-marker parser, so what the
/// pill counts matches what the completion path can advance against.
fn count_int_ids(text: &str) -> usize {
let upper = text.to_ascii_uppercase();
let mut seen = std::collections::HashSet::new();
let mut i = 0;
while let Some(pos) = upper[i..].find("INT-") {
let start = i + pos + 4;
let end = start
+ upper[start..]
.chars()
.take_while(|c| c.is_ascii_digit())
.count();
if end > start {
seen.insert(upper[start..end].parse::<u32>().ok());
}
i = end.max(i + pos + 4);
}
seen.into_iter().flatten().count()
}
#[derive(Deserialize)]
pub struct UpdateLoopRequest {
pub title: String,
pub description: String,
pub graph: Value,
pub task_template: String,
pub triggers: Value,
pub repeat_policy: Value,
#[serde(default)]
pub agents: Vec<AgentSlotInput>,
#[serde(default)]
pub teams: Vec<Uuid>,
#[serde(default)]
pub orgs: Vec<Uuid>,
}
pub async fn patch_loop(
State(state): State<AppState>,
Authed(user): Authed,
Path(id): Path<Uuid>,
Json(body): Json<UpdateLoopRequest>,
) -> Result<StatusCode, ApiError> {
if body.title.trim().is_empty() || body.task_template.trim().is_empty() {
return Err(ApiError::BadRequest);
}
cm_db::repo::loops::get(&state.pool, id, user.workspace_id.as_uuid())
.await?
.ok_or(ApiError::NotFound)?;
let next_fire_at = compute_next_fire(&body.triggers);
cm_db::repo::loops::update(
&state.pool,
id,
user.workspace_id.as_uuid(),
cm_db::repo::loops::UpdateLoop {
title: body.title.trim(),
description: body.description.trim(),
graph: &body.graph,
task_template: body.task_template.trim(),
triggers: &body.triggers,
repeat_policy: &body.repeat_policy,
next_fire_at,
},
)
.await?;
apply_staffing(&state.pool, id, &body.agents, &body.teams, &body.orgs).await?;
Ok(StatusCode::NO_CONTENT)
}
pub async fn delete_loop(
State(state): State<AppState>,
Authed(user): Authed,
Path(id): Path<Uuid>,
) -> Result<StatusCode, ApiError> {
cm_db::repo::loops::delete(&state.pool, id, user.workspace_id.as_uuid()).await?;
// Tear down the per-loop container (P2). Fire-and-forget: the row
// is gone, so any Docker failure is a log-line, not an API failure.
crate::research_container::teardown_loop(id).await;
Ok(StatusCode::NO_CONTENT)
}
pub async fn enable_loop(
State(state): State<AppState>,
Authed(user): Authed,
Path(id): Path<Uuid>,
) -> Result<StatusCode, ApiError> {
cm_db::repo::loops::set_enabled(&state.pool, id, user.workspace_id.as_uuid(), true).await?;
Ok(StatusCode::NO_CONTENT)
}
pub async fn disable_loop(
State(state): State<AppState>,
Authed(user): Authed,
Path(id): Path<Uuid>,
) -> Result<StatusCode, ApiError> {
cm_db::repo::loops::set_enabled(&state.pool, id, user.workspace_id.as_uuid(), false).await?;
// Stop the per-loop container while disabled — re-enabling later will
// spawn a fresh one on the next `run_now` / webhook fire. Keeps
// paused loops from holding a docker slot.
crate::research_container::teardown_loop(id).await;
Ok(StatusCode::NO_CONTENT)
}
#[derive(Serialize)]
pub struct RunTriggered {
pub run_id: Uuid,
pub iteration: i32,
}
/// `POST /api/loops/:id/run` — enqueue one iteration NOW, bypassing the
/// scheduler and any trigger config. Iteration counter continues from
/// wherever it was; parent_run_id chains to whatever last_run_id points at.
pub async fn run_now(
State(state): State<AppState>,
Authed(user): Authed,
Path(id): Path<Uuid>,
) -> Result<Json<RunTriggered>, ApiError> {
let l = cm_db::repo::loops::get(&state.pool, id, user.workspace_id.as_uuid())
.await?
.ok_or(ApiError::NotFound)?;
// P2: spawn the per-loop container before enqueue so topology_worker
// resolves its gateway URL when it picks up the run. Best-effort;
// never blocks the enqueue on Docker being unreachable.
ensure_loop_container(&state.pool, l.workspace_id, l.id).await;
// Kind-aware — research loops write to research_outcomes.
let iter = cm_db::repo::loops::next_iteration(&state.pool, l.id).await?;
let run_id = compose_and_enqueue_iteration(
&state.pool,
l.id,
l.workspace_id,
&l.graph,
l.last_run_id,
Some(&l.task_template),
)
.await
.map_err(|_| ApiError::Internal)?;
cm_db::repo::loops::mark_fired(&state.pool, l.id, run_id, l.next_fire_at).await?;
Ok(Json(RunTriggered {
run_id,
iteration: iter,
}))
}
/// `POST /webhooks/loops/:token` — public, HMAC-verified. Enqueues one
/// iteration on the loop that owns `token`. Returns 202 + `{run_id}` on
/// success, 401 on missing/bad signature, 404 on unknown token.
pub async fn webhook_receive(
State(state): State<AppState>,
Path(token): Path<String>,
headers: HeaderMap,
body: Bytes,
) -> (StatusCode, Json<Value>) {
let Ok(Some((_id, _ws, key, l))) =
cm_db::repo::loops::get_by_webhook_token(&state.pool, &token).await
else {
return (StatusCode::NOT_FOUND, Json(Value::Null));
};
let Some(sig_header) = headers
.get("X-Loop-Signature")
.and_then(|v| v.to_str().ok())
else {
return (StatusCode::UNAUTHORIZED, Json(Value::Null));
};
let Some(provided) = sig_header.strip_prefix("sha256=") else {
return (StatusCode::UNAUTHORIZED, Json(Value::Null));
};
if !verify_hmac(&key, &body, provided) {
return (StatusCode::UNAUTHORIZED, Json(Value::Null));
}
ensure_loop_container(&state.pool, l.workspace_id, l.id).await;
let iter = match cm_db::repo::loops::next_iteration(&state.pool, l.id).await {
Ok(n) => n,
Err(_) => return (StatusCode::INTERNAL_SERVER_ERROR, Json(Value::Null)),
};
let run_id = match compose_and_enqueue_iteration(
&state.pool,
l.id,
l.workspace_id,
&l.graph,
l.last_run_id,
Some(&l.task_template),
)
.await
{
Ok(r) => r,
Err(_) => return (StatusCode::INTERNAL_SERVER_ERROR, Json(Value::Null)),
};
let _ = cm_db::repo::loops::mark_fired(&state.pool, l.id, run_id, None).await;
(
StatusCode::ACCEPTED,
Json(serde_json::json!({"run_id": run_id, "iteration": iter})),
)
}
fn verify_hmac(key: &str, body: &[u8], provided_hex: &str) -> bool {
let Ok(mut mac) = Hmac::<sha2::Sha256>::new_from_slice(key.as_bytes()) else {
return false;
};
mac.update(body);
let expected = hex::encode(mac.finalize().into_bytes());
if expected.len() != provided_hex.len() {
return false;
}
// Constant-time compare.
expected
.bytes()
.zip(provided_hex.bytes())
.fold(0u8, |acc, (a, b)| acc | (a ^ b))
== 0
}
crates/cm-api/src/routes/mod.rs
-6
View File
@@ -13,17 +13,12 @@ pub mod gateway;
pub mod health;
pub mod identity;
pub mod level_up;
pub mod loops;
pub mod missions;
pub mod nodes;
pub mod oauth;
pub mod orgs;
pub mod planner;
pub mod probe;
pub mod repos;
pub mod research;
pub mod research_pipeline;
pub mod research_setup;
pub mod routines;
pub mod sessions;
pub mod skills;
@@ -37,5 +32,4 @@ pub mod teams;
pub mod terminal;
pub mod topology;
pub mod webhooks;
pub mod wizard_repo;
pub mod world;
crates/cm-api/src/routes/probe.rs
-147
View File
@@ -1,147 +0,0 @@
//! One-shot end-to-end pipeline probe.
//!
//! `POST /api/research/probe` bypasses the wizard / topics / loops /
//! per-team spawn machinery and drives a single trivial turn against
//! the workspace's shared ZeroClaw gateway with a minimal prompt.
//! Purpose: distinguish "pipeline is broken" from "the coordinator
//! prompt is too big for the current daemon timeouts". If this
//! succeeds, every failure we've been chasing is spawn-config or
//! prompt-size specific.
//!
//! Body: `{ "prompt": "…", "agent": "…" }` — both optional; defaults are
//! a two-letter reply prompt and the daemon's default agent alias.
//! Returns per-step timings + verdict.
use axum::extract::State;
use axum::Json;
use serde::{Deserialize, Serialize};
use std::time::Instant;
use crate::topology_exec::ZeroClawDriveExecutor;
use crate::{ApiError, AppState, Authed};
#[derive(Deserialize, Default)]
pub struct ProbeRequest {
/// The prompt to send. Defaults to a two-letter reply prompt so
/// the daemon returns fast and we can measure baseline latency.
#[serde(default)]
pub prompt: Option<String>,
/// Which agent alias to drive. Defaults to the daemon's
/// ZEROCLAW_DEFAULT_AGENT (currently `coordinator`).
#[serde(default)]
pub agent: Option<String>,
}
#[derive(Serialize)]
pub struct ProbeStep {
pub name: &'static str,
pub duration_ms: u128,
pub status: &'static str,
#[serde(skip_serializing_if = "Option::is_none")]
pub detail: Option<String>,
}
#[derive(Serialize)]
pub struct ProbeResponse {
pub verdict: &'static str,
pub total_duration_ms: u128,
pub prompt_len: usize,
#[serde(skip_serializing_if = "Option::is_none")]
pub response_preview: Option<String>,
pub steps: Vec<ProbeStep>,
}
/// `POST /api/research/probe`.
pub async fn probe(
State(_state): State<AppState>,
Authed(_user): Authed,
Json(body): Json<ProbeRequest>,
) -> Result<Json<ProbeResponse>, ApiError> {
let prompt = body.prompt.unwrap_or_else(|| {
"Respond with only these two letters (nothing else, no explanation): OK".to_string()
});
let agent_override = body.agent;
let started = Instant::now();
let mut steps: Vec<ProbeStep> = Vec::new();
// ── Step 1: build the executor from env (parses ZEROCLAW_TOKEN,
// ZEROCLAW_GATEWAY_URL, ZEROCLAW_AGENT_MAP). Anything wrong with
// the workspace config surfaces here.
let s1 = Instant::now();
let executor = match ZeroClawDriveExecutor::from_env() {
Ok(e) => e,
Err(e) => {
steps.push(ProbeStep {
name: "build_executor",
duration_ms: s1.elapsed().as_millis(),
status: "fail",
detail: Some(e.clone()),
});
return Ok(Json(ProbeResponse {
verdict: "fail",
total_duration_ms: started.elapsed().as_millis(),
prompt_len: prompt.len(),
response_preview: None,
steps,
}));
}
};
steps.push(ProbeStep {
name: "build_executor",
duration_ms: s1.elapsed().as_millis(),
status: "ok",
detail: None,
});
// ── Step 2: drive one turn end-to-end (opens ws, sends message,
// drains events until terminal). All of "handshake / auth /
// daemon spawn claude / claude call / response stream" collapse
// into this single measurement because ZeroClawDriveExecutor
// doesn't expose finer-grained hooks. But: if this succeeds
// within a few seconds, EVERY layer works and the coordinator
// failures we've been chasing are prompt-size specific.
let agent = agent_override.unwrap_or_else(|| "coordinator".to_string());
let s2 = Instant::now();
match executor.drive(&agent, &prompt).await {
Ok(outcome) => {
let out_ms = s2.elapsed().as_millis();
steps.push(ProbeStep {
name: "drive_turn",
duration_ms: out_ms,
status: "ok",
detail: Some(format!(
"tokens={}, output_len={}",
outcome.tokens,
outcome.output.len()
)),
});
let preview = if outcome.output.len() > 200 {
format!("{}…", &outcome.output[..200])
} else {
outcome.output.clone()
};
Ok(Json(ProbeResponse {
verdict: "ok",
total_duration_ms: started.elapsed().as_millis(),
prompt_len: prompt.len(),
response_preview: Some(preview),
steps,
}))
}
Err(e) => {
steps.push(ProbeStep {
name: "drive_turn",
duration_ms: s2.elapsed().as_millis(),
status: "fail",
detail: Some(format!("{e}")),
});
Ok(Json(ProbeResponse {
verdict: "fail",
total_duration_ms: started.elapsed().as_millis(),
prompt_len: prompt.len(),
response_preview: None,
steps,
}))
}
}
}
crates/cm-api/src/routes/research.rs
-1293
View File
File diff suppressed because it is too large Load Diff
crates/cm-api/src/routes/research_pipeline.rs
-298
View File
@@ -1,298 +0,0 @@
//! Pipeline diagnostics for a research topic.
//!
//! Walks the pipeline stages (staffing, repo, container, runs, outcomes,
//! approval) and returns a per-stage report. Read-only — every stage is
//! evaluated in isolation and any lookup failure downgrades to warn/skip
//! rather than failing the endpoint. Purpose: give users end-to-end
//! visibility so silent failures (a run that dies before writing an
//! outcome) are surfaced instead of buried in an empty artifact
//! download.
use axum::extract::{Path, State};
use axum::Json;
use serde::Serialize;
use sqlx::Row;
use uuid::Uuid;
use crate::{ApiError, AppState, Authed};
#[derive(Serialize)]
pub struct PipelineStage {
/// Machine-readable stage id: staffing / repo / container / runs /
/// outcomes / approval. Frontend uses this to key the checklist.
pub key: String,
/// User-facing one-line summary.
pub label: String,
/// ok | warn | fail | skip — drives the pill color in the UI.
pub status: &'static str,
/// Optional error text (last-known failure reason from the underlying
/// row) so the user can see WHY a stage failed instead of a green tick
/// with no artifact behind it.
#[serde(skip_serializing_if = "Option::is_none")]
pub detail: Option<String>,
}
#[derive(Serialize)]
pub struct PipelineState {
pub topic_id: Uuid,
pub status: String,
pub stages: Vec<PipelineStage>,
}
#[derive(Serialize)]
pub struct ActiveRun {
pub id: Uuid,
}
#[derive(Serialize)]
pub struct ActiveRuns {
pub topic_id: Uuid,
pub runs: Vec<ActiveRun>,
}
/// `GET /api/research/:id/active-runs` — queued + running topology_run
/// ids for this topic, newest first. Feeds the wizard's live-log panel
/// (SSE per run at `/api/topology-runs/:id/events`).
pub async fn active_runs(
State(state): State<AppState>,
Authed(user): Authed,
Path(id): Path<Uuid>,
) -> Result<Json<ActiveRuns>, ApiError> {
// Workspace-scope: 404 rather than leak run ids for a topic the
// caller can't see.
let _topic = cm_db::repo::research_topics::get(&state.pool, id, user.workspace_id.as_uuid())
.await?
.ok_or(ApiError::NotFound)?;
let ids =
cm_db::repo::topology_runs::active_run_ids_for_research_topic(&state.pool, id).await?;
Ok(Json(ActiveRuns {
topic_id: id,
runs: ids.into_iter().map(|id| ActiveRun { id }).collect(),
}))
}
/// `GET /api/research/:id/pipeline-state`.
pub async fn pipeline_state(
State(state): State<AppState>,
Authed(user): Authed,
Path(id): Path<Uuid>,
) -> Result<Json<PipelineState>, ApiError> {
let topic = cm_db::repo::research_topics::get(&state.pool, id, user.workspace_id.as_uuid())
.await?
.ok_or(ApiError::NotFound)?;
let mut stages = Vec::new();
// 1. staffing.
let agents = cm_db::repo::research_topics::agents(&state.pool, id)
.await
.unwrap_or_default();
stages.push(PipelineStage {
key: "staffing".into(),
label: format!("{} agent(s) assigned", agents.len()),
status: if agents.is_empty() { "fail" } else { "ok" },
detail: None,
});
// 2. repo — optional. When bound, we check the clone actually landed.
if topic.repo_id.is_some() {
let cloned = topic
.repo_workspace_path
.as_ref()
.is_some_and(|p| !p.is_empty());
stages.push(PipelineStage {
key: "repo".into(),
label: if cloned {
format!(
"Repo cloned at {}",
topic.repo_workspace_path.as_deref().unwrap_or("")
)
} else {
"Repo bound but never cloned".into()
},
status: if cloned { "ok" } else { "fail" },
detail: None,
});
} else {
stages.push(PipelineStage {
key: "repo".into(),
label: "No repo bound (optional)".into(),
status: "skip",
detail: None,
});
}
// 3. container — per-topic team runtime.
let container_ok =
topic.zeroclaw_container_name.is_some() && topic.zeroclaw_gateway_url.is_some();
stages.push(PipelineStage {
key: "container".into(),
label: if container_ok {
format!(
"Container: {}",
topic.zeroclaw_container_name.as_deref().unwrap_or("")
)
} else {
"Container not spawned (falling back to shared gateway)".into()
},
status: if container_ok { "ok" } else { "warn" },
detail: None,
});
// 4. runs — catches the failure with the actual error text.
let run_rows = sqlx::query(
"SELECT id, status, error, created_at
FROM topology_runs
WHERE research_topic_id = $1
ORDER BY created_at DESC",
)
.bind(id)
.fetch_all(&state.pool)
.await
.unwrap_or_default();
let n_runs = run_rows.len();
let n_failed = run_rows
.iter()
.filter(|r| r.try_get::<String, _>("status").ok().as_deref() == Some("failed"))
.count();
let n_running = run_rows
.iter()
.filter(|r| {
matches!(
r.try_get::<String, _>("status").ok().as_deref(),
Some("running") | Some("queued")
)
})
.count();
let n_completed = run_rows
.iter()
.filter(|r| r.try_get::<String, _>("status").ok().as_deref() == Some("completed"))
.count();
// 2026-07-16: only surface the error from the MOST RECENT run and
// only if that run itself failed. Previously we walked every run
// and returned the first non-empty error, so a pre-migration
// failed run's stale error kept showing next to a fresh successful
// run — reading like "everything is still broken" when it wasn't.
let latest_error = run_rows.first().and_then(|r| {
let status = r.try_get::<String, _>("status").ok();
if status.as_deref() == Some("failed") {
r.try_get::<Option<String>, _>("error")
.ok()
.flatten()
.filter(|s| !s.is_empty())
} else {
None
}
});
// Status rules:
// - 0 runs → skip (nothing to see yet — natural pre-fire state,
// NOT a failure)
// - any running → waiting (blue/spinner in UI — legitimate in-flight
// state)
// - all failed → fail (nothing succeeded)
// - some failed → warn (mixed history)
// - all completed → ok
let run_status = if n_runs == 0 {
"skip"
} else if n_running > 0 {
"waiting"
} else if n_failed == n_runs {
"fail"
} else if n_failed > 0 {
"warn"
} else {
"ok"
};
let run_label = if n_runs == 0 {
"No runs yet — pipeline hasn't fired".to_string()
} else if n_running > 0 && n_failed == 0 {
format!("{n_running} in flight, {n_completed} completed")
} else if n_running > 0 {
format!("{n_running} in flight, {n_completed} completed, {n_failed} failed")
} else {
format!("{n_runs} run(s), {n_failed} failed, {n_completed} completed")
};
stages.push(PipelineStage {
key: "runs".into(),
label: run_label,
// Suppress the "failure" detail line while runs are still in flight —
// reporting a prior turn's stale error text next to an actively-running
// job reads like the current run failed, which is what triggered the
// "everything looks broken" impression.
status: run_status,
detail: if run_status == "waiting" || run_status == "skip" {
None
} else {
latest_error
},
});
// 5. outcomes — the artifact rows get_artifact reads. Status is
// state-aware: an outcome-less topic with an in-flight run is a
// NORMAL waiting state, not a failure. Only flag `fail` when all
// runs have terminated AND none produced an outcome — the actual
// silent-bug case this diagnostic was designed to catch.
let outcome_count: i64 =
sqlx::query_scalar("SELECT count(*) FROM research_outcomes WHERE topic_id = $1")
.bind(id)
.fetch_one(&state.pool)
.await
.unwrap_or(0);
let outcome_status = if outcome_count > 0 {
"ok"
} else if n_runs == 0 {
"skip"
} else if n_running > 0 {
"waiting"
} else if n_failed > 0 {
"fail"
} else {
"warn"
};
let outcome_label = if outcome_count > 0 {
format!("{outcome_count} outcome(s) written")
} else if n_running > 0 {
"Waiting for the current run to finish…".to_string()
} else if n_runs == 0 {
"No outcome yet (pipeline hasn't fired)".to_string()
} else if n_failed > 0 {
"No outcome — all runs failed".to_string()
} else {
"No outcome yet".to_string()
};
let outcome_detail = if outcome_status == "fail" {
Some("No outcome produced — check the runs stage for the failure reason.".into())
} else {
None
};
stages.push(PipelineStage {
key: "outcomes".into(),
label: outcome_label,
status: outcome_status,
detail: outcome_detail,
});
// 6. approval — pending publish-approval, if any.
let pending = cm_db::repo::research_publish_approvals::pending_for_topic(&state.pool, id)
.await
.ok()
.flatten();
if let Some(a) = pending {
stages.push(PipelineStage {
key: "approval".into(),
label: format!(
"Approval pending (requested {})",
a.created_at
.format(&time::format_description::well_known::Rfc3339)
.unwrap_or_default()
),
status: "warn",
detail: None,
});
}
Ok(Json(PipelineState {
topic_id: id,
status: topic.status,
stages,
}))
}
crates/cm-api/src/routes/research_setup.rs
-506
View File
@@ -1,506 +0,0 @@
//! Research-topic runtime setup + wizard-driven loop materialization.
//!
//! Extracted from `research.rs` to keep that file under the 1250-line
//! budget. Two responsibilities:
//!
//! 1. Runtime setup — `prepare_topic_runtime` clones the bound repo
//! (idempotent) + spawns the per-topic ZeroClaw team container.
//! Called from both the one-shot `start_topic` handler and from
//! `routes::loops::compose_and_enqueue_iteration` before every
//! research-kind loop iteration.
//! 2. Wizard loop materialization — `materialize_topic_loops` creates
//! the paired research + optional coding loops when the wizard
//! picks a schedule mode.
use serde::Deserialize;
use sqlx::PgPool;
use uuid::Uuid;
/// A checked-out repo bundle for a research topic — populated by
/// `ensure_repo_workspace`; consumed by `build_coordinator_task` to
/// give the coordinator a concrete on-disk starting point for the team.
pub struct RepoContext {
/// Human-readable "owner/name".
pub slug: String,
/// Absolute path on the API host where the checkout lives.
pub path: String,
/// Branch we cloned (repo.default_branch → "main" fallback).
pub branch: String,
/// Line-per-entry preview of the working tree (relative paths).
pub tree_preview: String,
/// Files shown vs. total, so the prompt is honest about truncation.
pub shown: usize,
pub total_files: usize,
}
#[derive(Deserialize)]
pub struct TopicSchedule {
/// "once" | "nightly" | "manual".
pub mode: String,
}
/// Root directory under which `start_topic` clones per-topic checkouts.
/// Overridable via `CLAWMATES_RESEARCH_WORKSPACE_ROOT` for prod deploys
/// that want a mounted volume; defaults to a subdir of the system
/// tmpdir so dev + tests just work without setup.
pub fn research_workspace_root() -> std::path::PathBuf {
if let Ok(root) = std::env::var("CLAWMATES_RESEARCH_WORKSPACE_ROOT") {
return std::path::PathBuf::from(root);
}
std::env::temp_dir().join("clawmates-research")
}
/// Set up the on-disk workspace + container for a research topic —
/// clone repo (idempotent) + spawn ZeroClaw team container (idempotent).
/// Callable from both the one-shot `start_topic` handler and the
/// kind='research' loop iteration path in routes::loops. Fully
/// best-effort: any failure (docker unreachable, no clone_url) logs
/// and returns, letting the caller enqueue the run against the
/// workspace-wide gateway instead.
pub async fn prepare_topic_runtime(pool: &PgPool, workspace_id: Uuid, topic_id: Uuid) {
let topic = match cm_db::repo::research_topics::get_any_workspace(pool, topic_id).await {
Ok(Some(t)) => t,
_ => return,
};
let Some(repo_id) = topic.repo_id else {
return;
};
let repo =
match cm_db::repo::repos::get(pool, repo_id, cm_domain::WorkspaceId::from(workspace_id))
.await
{
Ok(r) => r,
Err(e) => {
eprintln!("prepare_topic_runtime({topic_id}): repo fetch failed: {e:?}");
return;
}
};
let ctx = match ensure_repo_workspace(pool, topic_id, workspace_id, &repo, &topic).await {
Ok(c) => c,
Err(e) => {
eprintln!("prepare_topic_runtime({topic_id}): clone failed: {e}");
return;
}
};
let repo_path = std::path::PathBuf::from(&ctx.path);
let state_root = research_workspace_root()
.join(topic_id.to_string())
.join("state");
let docker = match crate::research_container::connect() {
Ok(d) => d,
Err(e) => {
eprintln!("prepare_topic_runtime({topic_id}): docker connect failed: {e}");
return;
}
};
let mcp_bearer = crate::runtime_provision::mint_workspace_service_token(
pool,
cm_domain::WorkspaceId::from(workspace_id),
)
.await
.map_err(|e| {
eprintln!("prepare_topic_runtime({topic_id}): mint MCP bearer failed: {e}");
e
})
.ok();
match crate::research_container::spawn(
&docker,
topic_id,
&repo_path,
&state_root,
mcp_bearer.as_deref(),
)
.await
{
Ok(spawned) => {
if let Err(e) = cm_db::repo::research_topics::set_zeroclaw_container(
pool,
topic_id,
workspace_id,
Some(&spawned.name),
Some(&spawned.gateway_url),
)
.await
{
eprintln!("prepare_topic_runtime({topic_id}): persist container failed: {e}");
}
}
Err(e) => eprintln!("prepare_topic_runtime({topic_id}): spawn failed: {e}"),
}
}
/// Clone the bound repo (shallow, single branch) into a per-topic
/// workspace and gather a tree preview for the coordinator prompt.
/// Persists the clone path on the topic so a re-start reuses it
/// instead of re-cloning. Best-effort — callers treat failures as
/// "start without repo context" rather than aborting the run.
pub async fn ensure_repo_workspace(
pool: &PgPool,
topic_id: Uuid,
workspace_id: Uuid,
repo: &cm_db::repo::repos::Repo,
topic: &cm_db::repo::research_topics::ResearchTopic,
) -> Result<RepoContext, String> {
let clone_url = repo
.clone_url
.as_deref()
.ok_or_else(|| "repo has no clone_url".to_string())?;
let branch = repo
.default_branch
.as_deref()
.filter(|s| !s.is_empty())
.unwrap_or("main")
.to_string();
let target = topic.repo_workspace_path.clone().unwrap_or_else(|| {
research_workspace_root()
.join(topic_id.to_string())
.join("repo")
.to_string_lossy()
.into_owned()
});
let target_path = std::path::PathBuf::from(&target);
let should_clone = !target_path.join(".git").exists();
if should_clone {
if let Some(parent) = target_path.parent() {
std::fs::create_dir_all(parent).map_err(|e| format!("mkdir parent: {e}"))?;
}
let out = tokio::process::Command::new("git")
.arg("clone")
.arg("--depth")
.arg("1")
.arg("--single-branch")
.arg("--branch")
.arg(&branch)
.arg(clone_url)
.arg(&target_path)
.output()
.await
.map_err(|e| format!("spawn git clone: {e}"))?;
if !out.status.success() {
return Err(format!(
"git clone exit {:?}: {}",
out.status.code(),
String::from_utf8_lossy(&out.stderr).trim()
));
}
cm_db::repo::research_topics::set_repo_workspace_path(
pool,
topic_id,
workspace_id,
&target,
)
.await
.map_err(|e| format!("persist clone path: {e}"))?;
}
const MAX_TREE_LINES: usize = 60;
let ls = tokio::process::Command::new("git")
.arg("-C")
.arg(&target_path)
.arg("ls-files")
.output()
.await
.map_err(|e| format!("spawn git ls-files: {e}"))?;
let all = String::from_utf8_lossy(&ls.stdout);
let entries: Vec<&str> = all.lines().filter(|l| !l.is_empty()).collect();
let shown = entries.len().min(MAX_TREE_LINES);
let preview = entries
.iter()
.take(shown)
.map(|e| format!(" {e}"))
.collect::<Vec<_>>()
.join("\n");
Ok(RepoContext {
slug: format!("{}/{}", repo.owner, repo.name),
path: target,
branch,
tree_preview: if preview.is_empty() {
" (empty)".to_string()
} else {
preview
},
shown,
total_files: entries.len(),
})
}
/// Build a topology graph JSON for a research topic — same shape
/// start_topic uses (roster with coordinator promotion, topology-kind
/// aware role labeling, cm_topology::build). Called from
/// materialize_topic_loops so wizard-created research loops carry a
/// valid graph on their topology_run rows; without this the topology
/// worker rejects the run with `missing or invalid graph`.
///
/// Best-effort — returns a minimal fallback (single-node hub) on any
/// DB / topology-build failure so the loop still runs (degraded, but
/// not silently broken).
pub async fn build_topic_graph_json(pool: &PgPool, topic_id: Uuid) -> serde_json::Value {
use serde_json::json;
let topic = match cm_db::repo::research_topics::get_any_workspace(pool, topic_id).await {
Ok(Some(t)) => t,
_ => {
return json!({ "nodes": [{"id": "hub", "role": "coordinator", "attrs": {}}], "edges": [] })
}
};
let slots = cm_db::repo::research_topics::agents(pool, topic_id)
.await
.unwrap_or_default();
if slots.is_empty() {
return json!({ "nodes": [{"id": "hub", "role": "coordinator", "attrs": {}}], "edges": [] });
}
let mut roster: Vec<(cm_db::repo::research_topics::AgentSlot, cm_domain::Agent)> = Vec::new();
for s in &slots {
if let Ok(agent) =
cm_db::repo::agents::get(pool, cm_domain::AgentId::from(s.agent_id)).await
{
roster.push((s.clone(), agent));
}
}
if roster.is_empty() {
return json!({ "nodes": [{"id": "hub", "role": "coordinator", "attrs": {}}], "edges": [] });
}
let topo: cm_topology::TopologyKind =
serde_json::from_value(json!(topic.topology_kind.as_str()))
.unwrap_or(cm_topology::TopologyKind::HubSpoke);
let is_pipeline = matches!(topo, cm_topology::TopologyKind::Pipeline);
if !is_pipeline {
let coord_ix = roster
.iter()
.position(|(s, _)| {
s.role_slot
.as_deref()
.map(|r| r.to_ascii_lowercase().contains("coordinator"))
.unwrap_or(false)
})
.unwrap_or(0);
if coord_ix != 0 {
roster.swap(0, coord_ix);
}
}
let head_label = if is_pipeline {
"stage 1"
} else {
"coordinator"
};
let roles: Vec<String> = roster
.iter()
.enumerate()
.map(|(i, (s, a))| {
if i == 0 {
head_label.to_string()
} else if let Some(r) = &s.role_slot {
r.clone()
} else if !a.job_title.is_empty() {
a.job_title.clone()
} else if is_pipeline {
format!("stage {}", i + 1)
} else {
"specialist".to_string()
}
})
.collect();
let role_refs: Vec<&str> = roles.iter().map(|s| s.as_str()).collect();
let graph = match cm_topology::build(topo, &role_refs) {
Ok(g) => g,
Err(_) => {
return json!({ "nodes": [{"id": "hub", "role": "coordinator", "attrs": {}}], "edges": [] })
}
};
match cm_topology::to_json(&graph)
.ok()
.and_then(|s| serde_json::from_str::<serde_json::Value>(&s).ok())
{
Some(v) => v,
None => {
json!({ "nodes": [{"id": "hub", "role": "coordinator", "attrs": {}}], "edges": [] })
}
}
}
/// Creates the paired research + optional coding loops for a topic
/// (D1 fold). Fails soft — logs and returns, letting the topic land
/// even if loop creation stumbles. Skips the empty-roster gate
/// because `create_topic` already verified the workspace has agents.
#[allow(clippy::too_many_arguments)]
pub async fn materialize_topic_loops(
pool: &PgPool,
workspace_id: Uuid,
created_by: Uuid,
topic_id: Uuid,
topic_title: &str,
mode: &str,
also_coding: bool,
coding_team_mode: Option<&str>,
) {
use serde_json::json;
// Build a valid topology graph up front — an empty {nodes: [],
// edges: []} placeholder was rejected by the topology worker with
// "missing or invalid graph".
let graph = build_topic_graph_json(pool, topic_id).await;
let (r_triggers, next_fire_at) = match mode {
"nightly" => (
json!({ "initial_burst": 1, "cron": "0 3 * * *" }),
cm_runtime::scheduling::next_occurrence("0 3 * * *", time::OffsetDateTime::now_utc())
.ok(),
),
"manual" => (json!({ "webhook_enabled": true }), None),
_ => (json!({ "initial_burst": 1 }), None),
};
let r_title = format!("Research · {topic_title}");
match cm_db::repo::loops::create(
pool,
cm_db::repo::loops::NewLoop {
workspace_id,
title: &r_title,
description: "Auto-created by the research wizard. Kind=research; each iteration \
appends a new research_outcomes version for the bound topic.",
graph: &graph,
task_template: "Refresh the topic's research per the outcome kind.",
triggers: &r_triggers,
repeat_policy: &json!({ "kind": "infinite" }),
enabled: true,
next_fire_at,
webhook_token: None,
webhook_signing_key: None,
created_by,
},
)
.await
{
Ok(loop_id) => {
let _ = cm_db::repo::loops::set_source_research_topic(
pool,
loop_id,
workspace_id,
Some(topic_id),
)
.await;
let _ = cm_db::repo::loops::set_kind(pool, loop_id, "research").await;
crate::routes::loops::fire_initial_burst_if_set(
pool,
workspace_id,
loop_id,
&r_triggers,
"Refresh the topic's research per the outcome kind.",
&graph,
next_fire_at,
)
.await;
}
Err(e) => eprintln!("materialize_topic_loops: research loop create failed: {e:?}"),
}
if also_coding {
let c_title = format!("Coding · {topic_title}");
let c_triggers = json!({ "on_artifact_update": true, "initial_burst": 1 });
match cm_db::repo::loops::create(
pool,
cm_db::repo::loops::NewLoop {
workspace_id,
title: &c_title,
description: "Auto-created by the research wizard. Consumes one INT-XX per \
iteration from the paired research topic's artifact.",
graph: &graph,
task_template: "Execute the next unconsumed INT-XX from the artifact.",
triggers: &c_triggers,
repeat_policy: &json!({ "kind": "infinite" }),
enabled: true,
next_fire_at: None,
webhook_token: None,
webhook_signing_key: None,
created_by,
},
)
.await
{
Ok(loop_id) => {
let _ = cm_db::repo::loops::set_source_research_topic(
pool,
loop_id,
workspace_id,
Some(topic_id),
)
.await;
// 0045 fold — when the wizard picked "fresh" for the
// coding team, provision a dedicated team row with a
// coding_readwrite risk profile and bind it. Runtime
// spawn hookup (per-team container + config write)
// ships in a follow-up slice; the binding here ensures
// the loop already carries its intended team by the
// time that lands.
if coding_team_mode == Some("fresh") {
provision_fresh_coding_team(pool, workspace_id, loop_id, topic_title, &graph)
.await;
}
crate::routes::loops::fire_initial_burst_if_set(
pool,
workspace_id,
loop_id,
&c_triggers,
"Execute the next unconsumed INT-XX from the artifact.",
&graph,
None,
)
.await;
}
Err(e) => eprintln!("materialize_topic_loops: coding loop create failed: {e:?}"),
}
}
}
/// Create a placeholder `teams` row + set the loop's `team_id`. The
/// team is deliberately member-less at this stage — the graph is
/// carried on the loop itself, and the runtime hookup slice will
/// either back-fill members lazily on first spawn or wire the loop's
/// existing agents against the team via `add_member`.
///
/// Best-effort throughout: any failure logs to stderr but the loop
/// itself stays intact and functional under the legacy shared-team
/// fallback.
async fn provision_fresh_coding_team(
pool: &PgPool,
workspace_id: Uuid,
loop_id: Uuid,
topic_title: &str,
graph: &serde_json::Value,
) {
let team_id = Uuid::now_v7();
let team_name = format!("Coding · {topic_title}");
// insert_team_with_lifecycle keeps the topology graph so the
// runtime can reproduce the roster without a second lookup.
let ws = cm_domain::WorkspaceId::from(workspace_id);
if let Err(e) = cm_db::repo::teams::insert_team_with_lifecycle(
pool,
team_id,
ws,
&team_name,
"pipeline",
graph,
"permanent",
)
.await
{
eprintln!("provision_fresh_coding_team: insert_team failed for loop {loop_id}: {e:?}");
return;
}
if let Err(e) = cm_db::repo::teams::set_team_runtime_config(
pool,
team_id,
ws,
&cm_db::repo::teams::TeamRuntimeConfig {
risk_profile: Some("coding_readwrite".to_string()),
mcp_bundles: vec!["clawmates_door".to_string()],
},
)
.await
{
eprintln!("provision_fresh_coding_team: set_runtime_config failed: {e:?}");
}
if let Err(e) = cm_db::repo::teams::set_team_for_loop(pool, loop_id, Some(team_id)).await {
eprintln!("provision_fresh_coding_team: set_team_for_loop failed: {e:?}");
}
}
crates/cm-api/src/routes/topology.rs
+4 -228
View File
@@ -226,39 +226,26 @@ pub struct RunSummary {
pub kind: String,
pub created_at: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub iteration: Option<i32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub finished_at: Option<String>,
}
/// Query params for `GET /api/topology-runs`. `loop_id` filters to a single
/// loop's iterations, ordered newest-iteration-first.
/// Query params for `GET /api/topology-runs`.
#[derive(Deserialize)]
pub struct ListRunsQuery {
#[serde(default)]
pub loop_id: Option<Uuid>,
#[serde(default)]
pub limit: Option<i64>,
}
/// `GET /api/topology-runs` — recent runs for the workspace (compares + durable
/// run jobs), newest first. `?loop_id=X` filters to iterations of one loop,
/// ordered by iteration DESC (uses `topology_runs_loop_idx`).
/// run jobs), newest first.
pub async fn list_runs(
State(state): State<AppState>,
Authed(user): Authed,
Query(q): Query<ListRunsQuery>,
) -> Result<Json<Vec<RunSummary>>, ApiError> {
let limit = q.limit.filter(|n| *n > 0 && *n <= 200).unwrap_or(20);
let rows = match q.loop_id {
Some(loop_id) => {
cm_db::repo::topology_runs::list_by_loop(&state.pool, user.workspace_id, loop_id, limit)
.await?
}
None => {
cm_db::repo::topology_runs::list_recent(&state.pool, user.workspace_id, limit).await?
}
};
let rows =
cm_db::repo::topology_runs::list_recent(&state.pool, user.workspace_id, limit).await?;
let out = rows
.into_iter()
.map(|r| RunSummary {
@@ -267,7 +254,6 @@ pub async fn list_runs(
status: r.status,
kind: r.kind,
created_at: r.created_at.format(&Rfc3339).unwrap_or_default(),
iteration: r.iteration,
finished_at: r.finished_at.and_then(|t| t.format(&Rfc3339).ok()),
})
.collect();
@@ -388,213 +374,3 @@ pub async fn get_run(
checkpoint: run.checkpoint,
}))
}
// ── Phase: live container log tail ─────────────────────────────────
/// Strip ANSI escape sequences from a line so the browser terminal
/// renders it cleanly. Cheap and allocation-only when a match hits.
fn strip_ansi(input: &str) -> String {
let mut out = String::with_capacity(input.len());
let bytes = input.as_bytes();
let mut i = 0;
while i < bytes.len() {
if bytes[i] == 0x1b && i + 1 < bytes.len() && bytes[i + 1] == b'[' {
// Skip until final byte in @-~ range.
i += 2;
while i < bytes.len() && !(bytes[i] >= 0x40 && bytes[i] <= 0x7e) {
i += 1;
}
i += 1;
} else {
out.push(bytes[i] as char);
i += 1;
}
}
out
}
/// Squeeze a zeroclaw daemon log line into `[bracket] action outcome
/// · trailing message`. Falls back to the ANSI-stripped raw line when
/// the shape isn't recognised so we never lose an interesting line.
fn compact_container_log(line: &str) -> Option<String> {
let stripped = strip_ansi(line);
let trimmed = stripped.trim_end();
if trimmed.is_empty() {
return None;
}
// Drop pure framing noise: `zeroclaw_scope{...}` continuations
// that carry no zc_action.
let has_action = trimmed.contains("zc_action=");
if !has_action {
// Non-daemon lines (bash echoes, container startup banners,
// panic backtraces) — keep as-is; those are useful too.
if trimmed.contains("zc_") {
return None; // structural framing without action, drop
}
return Some(trimmed.to_string());
}
let bracket = trimmed
.split_once(']')
.and_then(|(before, _)| before.strip_prefix('['))
.unwrap_or("");
let action = trimmed
.split("zc_action=")
.nth(1)
.and_then(|s| s.split_whitespace().next())
.unwrap_or("?");
let outcome = trimmed
.split("zc_outcome=")
.nth(1)
.and_then(|s| s.split_whitespace().next())
.unwrap_or("");
let msg = trimmed
.rsplit(':')
.next()
.map(str::trim)
.unwrap_or("")
.to_string();
let tag = if bracket.is_empty() {
"system"
} else {
bracket
};
Some(if outcome.is_empty() || outcome == "unknown" {
format!("[{tag}] {action} · {msg}")
} else {
format!("[{tag}] {action} ({outcome}) · {msg}")
})
}
/// `GET /api/topology-runs/{id}/container-log` — SSE stream of the
/// per-topic team container's daemon log, filtered from the ZeroClaw
/// structural noise into `[actor] action (outcome) · message` lines.
/// Emits a `line` event per surviving line, plus periodic keep-alives.
/// Ends when the container's log stream closes or the client
/// disconnects. Auth: workspace-scoped like `run_events_sse`.
pub async fn run_container_log_sse(
State(state): State<AppState>,
Authed(user): Authed,
Path(id): Path<Uuid>,
) -> impl IntoResponse {
// All early exits + the live tail funnel through one stream! so
// Sse::new sees a single concrete stream type.
let pool = state.pool.clone();
let ws = user.workspace_id;
let stream = async_stream::stream! {
use futures::StreamExt;
// 1) Workspace scope + resolve the topic id whose container we'll
// tail. Two paths:
// a) run.research_topic_id set → research pipeline; use it
// directly (existing behavior).
// b) research_topic_id NULL + run belongs to a loop whose
// source_research_topic_id is set → paired coding loop;
// the loop reuses the research topic's team container.
// Anything else (raw topology runs, pure loop with no paired
// topic) errors out with a clear message.
if cm_db::repo::topology_runs::status(&pool, id, ws).await.is_err() {
yield Ok::<Event, Infallible>(
Event::default().event("error").data("run not found"),
);
return;
}
// Precedence — must mirror topology_worker::try_team_gateway_url,
// which is what actually spawns the container:
// a) run's loop has team_id set → the team runtime spawned
// `team-<team_id>-container` (matches spawn_team). This is
// the paired-coding-loop path when the wizard picked
// "fresh coding team". Loops with a team_id do NOT reuse
// the research topic's container.
// b) run.research_topic_id set → per-topic research container
// `research-<topic_id>-team` (matches spawn).
// c) run's loop has source_research_topic_id (legacy paired
// flow, no team_id) → same as (b) via the topic.
// d) anything else → error with a clear message.
let loop_id = cm_db::repo::topology_runs::loop_id_for_run(&pool, id).await.ok().flatten();
let team_id = match loop_id {
Some(lid) => cm_db::repo::teams::team_for_loop(&pool, lid).await.ok().flatten(),
None => None,
};
let direct = cm_db::repo::topology_runs::research_topic_id(&pool, id).await.ok().flatten();
let via_loop = if team_id.is_none() && direct.is_none() {
match loop_id {
Some(lid) => {
use sqlx::Row;
sqlx::query(
"SELECT source_research_topic_id FROM loops WHERE id = $1"
)
.bind(lid)
.fetch_optional(&pool)
.await
.ok()
.flatten()
.and_then(|r| r.try_get::<Option<Uuid>, _>("source_research_topic_id").ok().flatten())
}
None => None,
}
} else { None };
let container = if let Some(tid) = team_id {
crate::research_container::team_container_name_for(tid)
} else {
match direct.or(via_loop) {
Some(t) => crate::research_container::container_name_for(t),
None => {
yield Ok::<Event, Infallible>(
Event::default().event("error").data(
"run has no bound team, research topic, or paired-loop topic; container log unavailable",
),
);
return;
}
}
};
// 2) Docker handle.
let docker = match crate::research_container::connect() {
Ok(d) => d,
Err(e) => {
yield Ok(Event::default()
.event("error")
.data(format!("docker connect failed: {e}")));
return;
}
};
// 3) Tail.
let opts = bollard::query_parameters::LogsOptionsBuilder::default()
.stdout(true)
.stderr(true)
.follow(true)
.tail("200")
.timestamps(false)
.build();
yield Ok(Event::default()
.event("info")
.data(format!("tailing {container}")));
let mut log_stream = docker.logs(&container, Some(opts));
// Line-accumulator so partial chunks don't truncate a log line.
let mut buf = String::new();
while let Some(chunk) = log_stream.next().await {
let bytes = match chunk {
Ok(bollard::container::LogOutput::StdOut { message })
| Ok(bollard::container::LogOutput::StdErr { message })
| Ok(bollard::container::LogOutput::Console { message }) => message,
Ok(_) => continue,
Err(e) => {
yield Ok(Event::default().event("error").data(e.to_string()));
break;
}
};
let s = String::from_utf8_lossy(&bytes);
buf.push_str(&s);
while let Some(nl) = buf.find('\n') {
let line: String = buf.drain(..=nl).collect();
if let Some(compact) = compact_container_log(&line) {
yield Ok(Event::default().event("line").data(compact));
}
}
}
yield Ok(Event::default().event("done").data("stream closed"));
};
Sse::new(stream).keep_alive(KeepAlive::default())
}
crates/cm-api/src/routes/wizard_repo.rs
-153
View File
@@ -1,153 +0,0 @@
//! Wizard-driven clawstor repo materialization.
//!
//! Bridges the research wizard (frontend) to clawstor's fleet-wide
//! `POST /api/v2/repos/{ensure,release}` primitives so a picked repo
//! is checked out on every clawstor peer at step-2-next, and released
//! if the user backs out.
//!
//! The clawstor bearer token is server-side only; the frontend never
//! sees it. Endpoints require the standard `Authed` extractor and
//! resolve the picked `repo_id` against the caller's workspace so a
//! user cannot ensure a repo they can't see.
//!
//! Configured via env:
//! CLAWSTOR_URL — aggregator base, e.g. https://quantum.taila4f562.ts.net/clawstor
//! CLAWSTOR_TOKEN — bearer token whose namespace scopes the writes
//!
//! Both missing = disabled (500). Not-configured is a deploy-time
//! decision; runtime callers get a plain error.
use axum::{extract::State, Json};
use serde::{Deserialize, Serialize};
use uuid::Uuid;
use crate::{ApiError, AppState, Authed};
#[derive(Deserialize)]
pub struct RepoBody {
pub repo_id: Uuid,
/// Git ref (branch, tag, or SHA the remote will accept via
/// `git clone --branch`). When omitted, falls back to the repo's
/// recorded `default_branch`.
#[serde(default)]
pub git_ref: Option<String>,
}
#[derive(Serialize, Deserialize)]
pub struct PeerResult {
pub peer: String,
pub ok: bool,
#[serde(default)]
pub path: Option<String>,
#[serde(default)]
pub head_sha: Option<String>,
#[serde(default)]
pub cached: Option<bool>,
#[serde(default)]
pub removed: Option<bool>,
#[serde(default)]
pub error: Option<String>,
}
#[derive(Serialize, Deserialize)]
pub struct FanoutReply {
pub url: String,
pub git_ref: String,
pub workspace: String,
pub peers: Vec<PeerResult>,
pub all_ok: bool,
}
/// `POST /api/research/wizard/repo/ensure` — materialize the picked
/// repo across the clawstor fleet. Returns the aggregator's per-peer
/// reply so the wizard can render which nodes succeeded.
pub async fn ensure_repo(
State(state): State<AppState>,
Authed(user): Authed,
Json(body): Json<RepoBody>,
) -> Result<Json<FanoutReply>, ApiError> {
proxy(&state, &user, body, "ensure").await
}
/// `POST /api/research/wizard/repo/release` — inverse of ensure.
/// Called by the wizard on cancel (modal close before submit).
pub async fn release_repo(
State(state): State<AppState>,
Authed(user): Authed,
Json(body): Json<RepoBody>,
) -> Result<Json<FanoutReply>, ApiError> {
proxy(&state, &user, body, "release").await
}
async fn proxy(
state: &AppState,
user: &cm_auth::AuthedUser,
body: RepoBody,
action: &str,
) -> Result<Json<FanoutReply>, ApiError> {
// Workspace-scoped lookup — a caller can't touch repos outside
// their own workspace even if they know the id.
let repo = cm_db::repo::repos::get(&state.pool, body.repo_id, user.workspace_id).await?;
let url = repo.clone_url.ok_or(ApiError::BadRequest)?;
let git_ref = body
.git_ref
.as_deref()
.map(str::to_string)
.or(repo.default_branch)
.ok_or(ApiError::BadRequest)?;
if url.trim().is_empty() || git_ref.trim().is_empty() {
return Err(ApiError::BadRequest);
}
// Clawstor fan-out is best-effort — the aggregator may not be
// deployed in every environment. When it's absent (env unset,
// network error, non-JSON HTML from a fallback proxy, non-2xx),
// degrade to a "skipped" reply so the wizard doesn't block. Real
// fleet materialization happens later at spawn time; ensure was
// only a warmup.
let skipped = |reason: &str| -> Json<FanoutReply> {
eprintln!("wizard_repo::{action}: skipping fleet fan-out ({reason})");
Json(FanoutReply {
url: url.clone(),
git_ref: git_ref.clone(),
workspace: String::new(),
peers: Vec::new(),
all_ok: true,
})
};
let Ok(base) = std::env::var("CLAWSTOR_URL") else {
return Ok(skipped("CLAWSTOR_URL unset"));
};
let Ok(token) = std::env::var("CLAWSTOR_TOKEN") else {
return Ok(skipped("CLAWSTOR_TOKEN unset"));
};
let endpoint = format!("{}/api/v2/repos/{}", base.trim_end_matches('/'), action);
let Ok(client) = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(360))
.build()
else {
return Ok(skipped("http client build failed"));
};
let resp = match client
.post(&endpoint)
.bearer_auth(token)
.json(&serde_json::json!({
"url": url,
"git_ref": git_ref,
}))
.send()
.await
{
Ok(r) => r,
Err(e) => return Ok(skipped(&format!("send failed: {e}"))),
};
let status = resp.status();
if !status.is_success() {
return Ok(skipped(&format!("aggregator returned {status}")));
}
match resp.json::<FanoutReply>().await {
Ok(reply) => Ok(Json(reply)),
Err(e) => Ok(skipped(&format!("non-JSON response: {e}"))),
}
}
crates/cm-api/src/routes/world.rs
-165
View File
@@ -61,106 +61,6 @@ async fn active_runs(pool: &PgPool, ws: WorkspaceId) -> Vec<(String, String)> {
.collect()
}
/// Active research topics + their assigned agents. Each returned row is a
/// `(topic_id, title, agent_id, repo_workspace_path)` — one row per
/// (topic, agent) pair. Emitted from the SSE loop as `repo:<topic_id>`
/// project orbs so the World shows a clickable, labeled landmark for
/// every in-flight R&D initiative — no need for a file touch to land
/// first. `repo_workspace_path` (when non-null) is the on-disk clone
/// location; the SSE loop uses it to pre-seed the repo tree.
async fn active_research_topics(
pool: &PgPool,
ws: WorkspaceId,
) -> Vec<(String, String, String, Option<String>)> {
let rows = sqlx::query(
"SELECT t.id::text AS topic_id,
t.title AS title,
t.repo_workspace_path AS repo_path,
ra.agent_id::text AS agent_id
FROM research_topics t
JOIN research_topic_agents ra ON ra.topic_id = t.id
WHERE t.workspace_id = $1
AND t.status IN ('processing', 'reviewing', 'publishing')",
)
.bind(ws.as_uuid())
.fetch_all(pool)
.await
.unwrap_or_default();
rows.into_iter()
.map(|r| {
(
r.get::<String, _>("topic_id"),
r.get::<String, _>("title"),
r.get::<String, _>("agent_id"),
r.try_get::<Option<String>, _>("repo_path").unwrap_or(None),
)
})
.collect()
}
/// Cap on pre-seeded file entries per repo. Large repos surface only the
/// top N so the SSE payload stays bounded — a subsequent tool call
/// exercising a specific path will fill in additional nodes on demand.
const REPO_PRESEED_CAP: usize = 200;
/// Read the top-level file list of a topic's cloned repo via `git ls-files`
/// so the SSE loop can pre-seed dir:/file: nodes in the client engine.
/// Bounded by `REPO_PRESEED_CAP`. Returns an empty vec on any failure
/// (missing clone, git not on PATH, empty repo) — a missing pre-seed
/// degrades gracefully to the pre-V3 behavior (tree builds as agents
/// touch files).
async fn preseed_repo_paths(clone_path: &str) -> Vec<String> {
let path = std::path::Path::new(clone_path);
if !path.join(".git").exists() {
return Vec::new();
}
let out = tokio::process::Command::new("git")
.arg("-C")
.arg(path)
.arg("ls-files")
.output()
.await;
let Ok(out) = out else { return Vec::new() };
if !out.status.success() {
return Vec::new();
}
String::from_utf8_lossy(&out.stdout)
.lines()
.filter(|l| !l.trim().is_empty())
.take(REPO_PRESEED_CAP)
.map(|s| s.to_string())
.collect()
}
/// Enabled scheduled loops + their assigned agents. Same shape as
/// `active_research_topics` — `(loop_id, title, agent_id)` per (loop, agent).
/// Emitted as `loop:<loop_id>` landmark orbs so recurring/scheduled work is
/// visible in the World at all times, not just while a run is mid-flight.
/// Contrast with research topics (transient statuses processing/reviewing/
/// publishing) — loops are persistent landmarks the user can click.
async fn active_loops(pool: &PgPool, ws: WorkspaceId) -> Vec<(String, String, String)> {
let rows = sqlx::query(
"SELECT l.id::text AS loop_id, l.title AS title, la.agent_id::text AS agent_id
FROM loops l
JOIN loop_agents la ON la.loop_id = l.id
WHERE l.workspace_id = $1
AND l.enabled = TRUE",
)
.bind(ws.as_uuid())
.fetch_all(pool)
.await
.unwrap_or_default();
rows.into_iter()
.map(|r| {
(
r.get::<String, _>("loop_id"),
r.get::<String, _>("title"),
r.get::<String, _>("agent_id"),
)
})
.collect()
}
/// A short human label for a tool's input (for the tool-call target).
fn summarize_input(input: &Value) -> String {
for k in ["target", "path", "url", "query", "name", "file", "command"] {
@@ -461,71 +361,6 @@ pub async fn world_live(State(state): State<AppState>, Authed(user): Authed) ->
}
}
// Active research topics → landmark project orbs. One `repo:<id>`
// per topic, labeled with the topic title so users can click it
// and drop into the repo-focus (Gource) view before any files are
// touched. Assigned agents gently converge on their topic's orb
// so the affinity is visible even in idle windows.
let research = active_research_topics(&pool, ws).await;
let mut seen_topics = std::collections::HashSet::new();
for (topic_id, title, agent_id, repo_path) in &research {
let node_id = format!("repo:{topic_id}");
if seen_topics.insert(topic_id.clone()) {
yield sse(
"node.activity",
json!({ "nodeId": node_id, "label": title, "kind": "service", "heat": 0.0 }),
);
// Pre-seed the repo tree (V3). One-shot on first sight
// of the topic per SSE client. Each file emits with
// heat=0 so the tree is quiet-solid at rest — activity
// still hot-swaps as agents touch files. Bounded to
// REPO_PRESEED_CAP so payload stays reasonable.
if let Some(clone_path) = repo_path {
for p in preseed_repo_paths(clone_path).await {
let leaf = std::path::Path::new(&p)
.file_name()
.and_then(|s| s.to_str())
.unwrap_or(&p)
.to_string();
yield sse(
"node.activity",
json!({
"nodeId": format!("file:{p}"),
"label": leaf,
"kind": "service",
"heat": 0.0,
}),
);
}
}
}
yield sse(
"world.touch",
json!({ "agentId": agent_id, "nodeId": node_id, "kind": "service", "weight": 0.15 }),
);
}
// Scheduled loops → landmark orbs, symmetric to research topics.
// Persistent landmarks: emitted whenever a loop is enabled, so a
// loop between fires still reads as an in-flight project. When
// a loop actually runs, the topology_worker journals events
// which the run-cursor block below picks up and heats the orb.
let loops = active_loops(&pool, ws).await;
let mut seen_loops = std::collections::HashSet::new();
for (loop_id, title, agent_id) in &loops {
let node_id = format!("loop:{loop_id}");
if seen_loops.insert(loop_id.clone()) {
yield sse(
"node.activity",
json!({ "nodeId": node_id, "label": title, "kind": "service", "heat": 0.0 }),
);
}
yield sse(
"world.touch",
json!({ "agentId": agent_id, "nodeId": node_id, "kind": "service", "weight": 0.15 }),
);
}
// Real convergence: each running agent beams toward its active-run node.
for (run_id, agent_id) in &runs {
let node_id = format!("run:{}", &run_id[..run_id.len().min(8)]);