loops: kind='research' dispatch — research runs as loops
ci / gates (push) Successful in 5s
ci / frontend (push) Successful in 30s
ci / rust (push) Failing after 48s
ci / e2e (push) Has been skipped
ci / publish (push) Has been skipped

Delivers the research/loop fold: kind='research' loops run the
research pipeline each iteration, appending a new research_outcomes
version. The paired on_artifact_update fan-out then wakes up any
kind='exec' loops bound to the same topic to consume new INTs. Every
runnable thing is now a loop (D1).

Backend — DB helpers:
- cm_db::repo::loops::kind_and_binding — reads (kind, source_topic,
  task_template) so callers can dispatch without hydrating the whole
  Loop struct.
- cm_db::repo::loops::enqueue_iteration_with_topic — new variant that
  sets research_topic_id on topology_runs alongside loop_id, so the
  completion hook's freeze_research_outcome writes a new outcome
  version for research-kind iterations.
- cm_db::repo::research_topics::get_any_workspace — cross-workspace
  fetch used by the research task builder (the loop row is
  authoritative for the workspace binding via kind_and_binding).

Backend — dispatch:
- routes::loops::compose_research_iteration_task — builds the
  coordinator prompt for a research iteration: topic title +
  description + outcome_kind + prior version pointer + refresh
  instructions (survey new sources, preserve stable INT ids, mark
  superseded items as deprecated rather than delete). The completion
  hook writes the resulting synthesis as research_outcomes v(prior+1).
- routes::loops::compose_and_enqueue_iteration — one-shot dispatch:
  reads the kind, picks compose_iteration_task (exec) or
  compose_research_iteration_task (research), enqueues with or
  without research_topic_id set.

All four enqueue callsites now route through compose_and_enqueue:
- create_loop (initial_burst)
- run_now
- webhook_receive
- topology_worker::continue_initial_burst
- topology_worker::freeze_research_outcome (on_artifact_update fan-out)

Research-kind loops naturally form the "nightly refresh" side of a
paired research + coding loop: research writes a fresh outcome
version → fan-out wakes exec loops with on_artifact_update →
coding loops consume the next INT (which the research loop may have
just added). D2 answer (inherit repo binding): repo lives on the
topic; both loops sharing the source topic id read from the same
context, no duplication. D3 answer (coordinator resolves): the
research iteration prompt tells the team to preserve stable INT ids
and mark deprecations rather than delete, so coding loops' consumed
lists stay valid across versions.

Follow-up (next commit): ResearchWizard schedule step — "Just once /
Nightly / Manual" that creates the paired research-kind loop with
initial_burst=1 (just once) or cron 0 3 * * * (nightly) + optional
paired coding loop with on_artifact_update.
This commit is contained in:
Omar Sobh
2026-07-09 22:56:19 -07:00
parent 5aa2de7f30
commit 91a51dce11
4 changed files with 239 additions and 49 deletions
+118 -18
View File
@@ -55,6 +55,112 @@ fn loop_state_root() -> std::path::PathBuf {
/// <task_template>
/// The topology_worker's completion hook (P3) parses the COMPLETED
/// marker to update `consumed_int_ids`.
/// One-shot compose + enqueue for a loop iteration. Reads the loop's
/// kind from the DB and dispatches: kind='exec' uses
/// compose_iteration_task (INT-consumption prepend); kind='research'
/// uses compose_research_iteration_task AND sets research_topic_id on
/// the topology_run so freeze_research_outcome writes a new outcome
/// version at completion. Returns the run id. Standalone exec loops
/// (no source topic) still work — the compose helper returns the
/// task_template verbatim.
pub async fn compose_and_enqueue_iteration(
pool: &sqlx::PgPool,
loop_id: Uuid,
workspace_id: Uuid,
graph: &Value,
parent_run_id: Option<Uuid>,
task_template_override: Option<&str>,
) -> Result<Uuid, cm_db::DbError> {
// Kind is the source of truth — task_template alone isn't enough
// to know whether to write to research_outcomes.
let (kind, source_topic, template) =
match cm_db::repo::loops::kind_and_binding(pool, loop_id).await? {
Some(t) => t,
None => return Err(cm_db::DbError::NotFound),
};
let template_ref = task_template_override.unwrap_or(&template);
let iter = cm_db::repo::loops::next_iteration(pool, loop_id).await?;
if kind == "research" {
// Research-kind requires a bound topic (schema-level constraint
// isn't enforced yet — surface the misconfiguration explicitly).
let Some(topic_id) = source_topic else {
return Err(cm_db::DbError::NotFound);
};
let task = compose_research_iteration_task(pool, topic_id, template_ref).await;
cm_db::repo::loops::enqueue_iteration_with_topic(
pool,
loop_id,
workspace_id,
&task,
graph,
iter,
parent_run_id,
Some(topic_id),
)
.await
} else {
let task = compose_iteration_task(pool, loop_id, template_ref).await;
cm_db::repo::loops::enqueue_iteration(
pool,
loop_id,
workspace_id,
&task,
graph,
iter,
parent_run_id,
)
.await
}
}
/// Build the coordinator prompt for a kind='research' loop iteration.
/// Wraps the topic's description + outcome_kind + prior artifact
/// version pointer into an instruction that asks the team to refresh
/// the plan (survey new sources, revise existing INTs, add new ones)
/// and emit the updated artifact using the same section shape. The
/// completion hook's `freeze_research_outcome` will insert a new
/// versioned row automatically because the topology_run carries
/// research_topic_id.
pub async fn compose_research_iteration_task(
pool: &PgPool,
topic_id: Uuid,
task_template: &str,
) -> String {
let (title, description, outcome_kind, prior_version) =
match cm_db::repo::research_topics::get_any_workspace(pool, topic_id).await {
Ok(Some(t)) => {
let prior = cm_db::repo::research_outcomes::latest(pool, topic_id)
.await
.unwrap_or(None)
.map(|o| o.version)
.unwrap_or(0);
(t.title, t.description, t.outcome_kind, prior)
}
_ => return task_template.to_string(),
};
format!(
"RESEARCH LOOP ITERATION\n\
=======================\n\
Topic: {title}\n\
Outcome kind: {outcome_kind}\n\
Prior artifact version: v{prior_version} (0 = fresh)\n\n\
DESCRIPTION:\n{description}\n\n\
YOUR JOB THIS ITERATION:\n\
- Refresh the research — pull in any new papers / findings since v{prior_version}.\n\
- Update the artifact using the SAME section structure the outcome_kind\n\
requires (e.g. integrations kind = executive summary + INT-XX cards).\n\
- Preserve stable ids (INT-01 stays INT-01 across versions). If an item\n\
is superseded, mark it {{deprecated: <reason>}} rather than deleting so\n\
downstream coding loops that already consumed it don't lose context.\n\
- Add NEW items with new ids continuing from the last used number.\n\
- Cite every claim; never fabricate sources or repo paths.\n\n\
The workspace's final synthesis is captured as research_outcomes v{}. \
Downstream on_artifact_update loops will wake up on this write.\n\n\
LOOP OPERATOR NOTES:\n{task_template}\n",
prior_version + 1
)
}
pub async fn compose_iteration_task(pool: &PgPool, loop_id: Uuid, task_template: &str) -> String {
let ctx = cm_db::repo::loops::source_research_context(pool, loop_id)
.await
@@ -306,20 +412,16 @@ pub async fn create_loop(
let chain_on_completion = parsed.as_ref().map(|t| t.on_completion).unwrap_or(false);
if initial_burst > 0 {
ensure_loop_container(&state.pool, user.workspace_id.as_uuid(), id).await;
let iter = cm_db::repo::loops::next_iteration(&state.pool, id)
.await
.unwrap_or(0);
let task = compose_iteration_task(&state.pool, id, body.task_template.trim()).await;
// Best-effort — we don't want to abort the loop-create response
// just because Docker didn't answer the daemon health check.
match cm_db::repo::loops::enqueue_iteration(
// Kind-aware compose + enqueue — research-kind loops get a
// research prompt and research_topic_id set on the run.
let template = body.task_template.trim();
match compose_and_enqueue_iteration(
&state.pool,
id,
user.workspace_id.as_uuid(),
&task,
&body.graph,
iter,
None,
Some(template),
)
.await
{
@@ -629,18 +731,18 @@ pub async fn run_now(
// resolves its gateway URL when it picks up the run. Best-effort;
// never blocks the enqueue on Docker being unreachable.
ensure_loop_container(&state.pool, l.workspace_id, l.id).await;
// Kind-aware — research loops write to research_outcomes.
let iter = cm_db::repo::loops::next_iteration(&state.pool, l.id).await?;
let task = compose_iteration_task(&state.pool, l.id, &l.task_template).await;
let run_id = cm_db::repo::loops::enqueue_iteration(
let run_id = compose_and_enqueue_iteration(
&state.pool,
l.id,
l.workspace_id,
&task,
&l.graph,
iter,
l.last_run_id,
Some(&l.task_template),
)
.await?;
.await
.map_err(|_| ApiError::Internal)?;
cm_db::repo::loops::mark_fired(&state.pool, l.id, run_id, l.next_fire_at).await?;
Ok(Json(RunTriggered {
run_id,
@@ -681,15 +783,13 @@ pub async fn webhook_receive(
Ok(n) => n,
Err(_) => return (StatusCode::INTERNAL_SERVER_ERROR, Json(Value::Null)),
};
let task = compose_iteration_task(&state.pool, l.id, &l.task_template).await;
let run_id = match cm_db::repo::loops::enqueue_iteration(
let run_id = match compose_and_enqueue_iteration(
&state.pool,
l.id,
l.workspace_id,
&task,
&l.graph,
iter,
l.last_run_id,
Some(&l.task_template),
)
.await
{