loops: kind column + initial_burst + on_artifact_update trigger fan-out
ci / gates (push) Successful in 6s
ci / frontend (push) Successful in 28s
ci / rust (push) Failing after 1m0s
ci / e2e (push) Has been skipped
ci / publish (push) Has been skipped

Foundation for folding research into loops as a first-class kind.
This commit ships the plumbing; the research-kind dispatch itself
lands next. Behavior for existing exec-kind loops is unchanged unless
they opt into the new trigger fields.

Migration 0044:
- kind TEXT NOT NULL DEFAULT 'exec' CHECK ('exec' | 'research'). New
  research-kind will run the research pipeline each iteration (next
  commit); 'exec' preserves today's behavior.
- initial_burst_remaining INT NOT NULL DEFAULT 0 — countdown for the
  triggers.initial_burst quota. Decremented CAS-safely on each
  completion until it hits 0.
- Two partial indexes: (kind, source_research_topic_id) for kind-
  aware lookups, and (source_research_topic_id) filtered on
  on_artifact_update=true + enabled=true for the fan-out hook.

Trigger schema extended with two optional fields:
- initial_burst: N — fire N iterations back-to-back at create time.
  create_loop enqueues the first iteration inline (subject to
  empty-roster gate), sets remaining=N-1, and the completion hook
  continues the chain until exhausted.
- on_artifact_update: true — when a bound research_outcomes row is
  inserted for the source topic, wake up one iteration of this loop.
  Coalesced against has_active_run so a burst of rapid revisions
  doesn't queue duplicates.

Backend:
- cm_db::repo::loops helpers (all dynamic sqlx, no offline cache
  regen needed):
  - set_initial_burst_remaining
  - take_initial_burst_slot (CAS UPDATE returning prev value; 0 on
    exhausted or race loss)
  - loops_awaiting_topic (fan-out query: kind=exec + enabled +
    on_artifact_update=true bound to the given topic)
  - has_active_run (queued|running iteration existence check)
  - get_any_workspace (bypasses the workspace scope guard; used by
    the completion hook where the run row is authoritative)
- routes/loops::compose_iteration_task made pub so the completion
  hook can build the same enriched task string as run_now.
- topology_worker::freeze_research_outcome now fans out to awakened
  loops after the outcome insert, using compose_iteration_task and
  coalescing on has_active_run.
- topology_worker::continue_initial_burst runs on every completion:
  · take_initial_burst_slot (CAS) — no-op if already exhausted
  · has_active_run coalesce guard
  · re-fetches the loop via get_any_workspace + compose_iteration_task
  · enqueues via loops::enqueue_iteration with parent_run_id set

Follow-ups already queued:
- kind='research' dispatch in run_job — build the research
  coordinator task from the topic config, run the research pipeline
  each iteration. Requires factoring start_topic's task-build.
- ResearchWizard "When should this run?" step (Just once / Nightly /
  Manual) creating the topic + paired research-kind loop.
- LoopsWizard trigger UI matching the design proposal (burst count,
  cron, on-artifact checkbox).
This commit is contained in:
Omar Sobh
2026-07-09 22:50:01 -07:00
parent ce111273bb
commit 4ada5557f2
4 changed files with 321 additions and 1 deletions
+65 -1
View File
@@ -55,7 +55,7 @@ fn loop_state_root() -> std::path::PathBuf {
/// <task_template>
/// The topology_worker's completion hook (P3) parses the COMPLETED
/// marker to update `consumed_int_ids`.
async fn compose_iteration_task(pool: &PgPool, loop_id: Uuid, task_template: &str) -> String {
pub async fn compose_iteration_task(pool: &PgPool, loop_id: Uuid, task_template: &str) -> String {
let ctx = cm_db::repo::loops::source_research_context(pool, loop_id)
.await
.unwrap_or(None);
@@ -183,6 +183,24 @@ struct Triggers {
on_completion: bool,
#[serde(default)]
webhook_enabled: bool,
/// NEW — number of iterations to fire back-to-back at loop-create
/// time. Enqueue path: fire once immediately, then chain each
/// subsequent one via on_completion until the burst quota is
/// exhausted (tracked in run metadata). Defaults to 0 for existing
/// loops (no auto-fire); new wizards typically set 1 (D1: every
/// runnable thing runs at least once).
#[serde(default)]
initial_burst: u32,
/// NEW — when this loop is bound to a source_research_topic and
/// that topic gets a fresh research_outcomes row (via
/// freeze_research_outcome), enqueue one iteration on this loop.
/// Coalesced with any in-flight run (D3: coordinator resolves;
/// no race, just one wake per artifact update). Read directly
/// from the loops.triggers jsonb by loops_awaiting_topic — no
/// need for the Rust parser to hold it after the fact.
#[serde(default)]
#[allow(dead_code)]
on_artifact_update: bool,
}
fn make_webhook_material() -> (String, String) {
@@ -276,6 +294,52 @@ pub async fn create_loop(
}
}
// initial_burst — fire the first iteration inline, so every loop
// runs at least once when the trigger requests it (D1). The burst
// continues via the on_completion trigger up to `initial_burst`
// (tracked in loops.initial_burst_remaining, decremented on each
// completion). Best-effort: a docker/DB failure on the FIRST fire
// logs but the loop still lives with next_fire_at set for its
// cron.
let parsed = parse_triggers(&body.triggers);
let initial_burst = parsed.as_ref().map(|t| t.initial_burst).unwrap_or(0);
let chain_on_completion = parsed.as_ref().map(|t| t.on_completion).unwrap_or(false);
if initial_burst > 0 {
ensure_loop_container(&state.pool, user.workspace_id.as_uuid(), id).await;
let iter = cm_db::repo::loops::next_iteration(&state.pool, id)
.await
.unwrap_or(0);
let task = compose_iteration_task(&state.pool, id, body.task_template.trim()).await;
// Best-effort — we don't want to abort the loop-create response
// just because Docker didn't answer the daemon health check.
match cm_db::repo::loops::enqueue_iteration(
&state.pool,
id,
user.workspace_id.as_uuid(),
&task,
&body.graph,
iter,
None,
)
.await
{
Ok(run_id) => {
let _ = cm_db::repo::loops::mark_fired(&state.pool, id, run_id, next_fire_at).await;
// Log the burst remaining so on_completion can advance it.
// Stored on loops.initial_burst_remaining (added by the
// same migration as loops.kind). Falls back silently on
// schema absence so we don't break dev DBs mid-rollout.
let remaining = initial_burst.saturating_sub(1) as i32;
if remaining > 0 || chain_on_completion {
let _ =
cm_db::repo::loops::set_initial_burst_remaining(&state.pool, id, remaining)
.await;
}
}
Err(e) => eprintln!("loops::create: initial_burst enqueue failed: {e:?}"),
}
}
Ok((
StatusCode::CREATED,
Json(LoopCreated {