loops: kind column + initial_burst + on_artifact_update trigger fan-out
Foundation for folding research into loops as a first-class kind.
This commit ships the plumbing; the research-kind dispatch itself
lands next. Behavior for existing exec-kind loops is unchanged unless
they opt into the new trigger fields.
Migration 0044:
- kind TEXT NOT NULL DEFAULT 'exec' CHECK ('exec' | 'research'). New
research-kind will run the research pipeline each iteration (next
commit); 'exec' preserves today's behavior.
- initial_burst_remaining INT NOT NULL DEFAULT 0 — countdown for the
triggers.initial_burst quota. Decremented CAS-safely on each
completion until it hits 0.
- Two partial indexes: (kind, source_research_topic_id) for kind-
aware lookups, and (source_research_topic_id) filtered on
on_artifact_update=true + enabled=true for the fan-out hook.
Trigger schema extended with two optional fields:
- initial_burst: N — fire N iterations back-to-back at create time.
create_loop enqueues the first iteration inline (subject to
empty-roster gate), sets remaining=N-1, and the completion hook
continues the chain until exhausted.
- on_artifact_update: true — when a bound research_outcomes row is
inserted for the source topic, wake up one iteration of this loop.
Coalesced against has_active_run so a burst of rapid revisions
doesn't queue duplicates.
Backend:
- cm_db::repo::loops helpers (all dynamic sqlx, no offline cache
regen needed):
- set_initial_burst_remaining
- take_initial_burst_slot (CAS UPDATE returning prev value; 0 on
exhausted or race loss)
- loops_awaiting_topic (fan-out query: kind=exec + enabled +
on_artifact_update=true bound to the given topic)
- has_active_run (queued|running iteration existence check)
- get_any_workspace (bypasses the workspace scope guard; used by
the completion hook where the run row is authoritative)
- routes/loops::compose_iteration_task made pub so the completion
hook can build the same enriched task string as run_now.
- topology_worker::freeze_research_outcome now fans out to awakened
loops after the outcome insert, using compose_iteration_task and
coalescing on has_active_run.
- topology_worker::continue_initial_burst runs on every completion:
· take_initial_burst_slot (CAS) — no-op if already exhausted
· has_active_run coalesce guard
· re-fetches the loop via get_any_workspace + compose_iteration_task
· enqueues via loops::enqueue_iteration with parent_run_id set
Follow-ups already queued:
- kind='research' dispatch in run_job — build the research
coordinator task from the topic config, run the research pipeline
each iteration. Requires factoring start_topic's task-build.
- ResearchWizard "When should this run?" step (Just once / Nightly /
Manual) creating the topic + paired research-kind loop.
- LoopsWizard trigger UI matching the design proposal (burst count,
cron, on-artifact checkbox).
This commit is contained in:
@@ -55,7 +55,7 @@ fn loop_state_root() -> std::path::PathBuf {
|
||||
/// <task_template>
|
||||
/// The topology_worker's completion hook (P3) parses the COMPLETED
|
||||
/// marker to update `consumed_int_ids`.
|
||||
async fn compose_iteration_task(pool: &PgPool, loop_id: Uuid, task_template: &str) -> String {
|
||||
pub async fn compose_iteration_task(pool: &PgPool, loop_id: Uuid, task_template: &str) -> String {
|
||||
let ctx = cm_db::repo::loops::source_research_context(pool, loop_id)
|
||||
.await
|
||||
.unwrap_or(None);
|
||||
@@ -183,6 +183,24 @@ struct Triggers {
|
||||
on_completion: bool,
|
||||
#[serde(default)]
|
||||
webhook_enabled: bool,
|
||||
/// NEW — number of iterations to fire back-to-back at loop-create
|
||||
/// time. Enqueue path: fire once immediately, then chain each
|
||||
/// subsequent one via on_completion until the burst quota is
|
||||
/// exhausted (tracked in run metadata). Defaults to 0 for existing
|
||||
/// loops (no auto-fire); new wizards typically set 1 (D1: every
|
||||
/// runnable thing runs at least once).
|
||||
#[serde(default)]
|
||||
initial_burst: u32,
|
||||
/// NEW — when this loop is bound to a source_research_topic and
|
||||
/// that topic gets a fresh research_outcomes row (via
|
||||
/// freeze_research_outcome), enqueue one iteration on this loop.
|
||||
/// Coalesced with any in-flight run (D3: coordinator resolves;
|
||||
/// no race, just one wake per artifact update). Read directly
|
||||
/// from the loops.triggers jsonb by loops_awaiting_topic — no
|
||||
/// need for the Rust parser to hold it after the fact.
|
||||
#[serde(default)]
|
||||
#[allow(dead_code)]
|
||||
on_artifact_update: bool,
|
||||
}
|
||||
|
||||
fn make_webhook_material() -> (String, String) {
|
||||
@@ -276,6 +294,52 @@ pub async fn create_loop(
|
||||
}
|
||||
}
|
||||
|
||||
// initial_burst — fire the first iteration inline, so every loop
|
||||
// runs at least once when the trigger requests it (D1). The burst
|
||||
// continues via the on_completion trigger up to `initial_burst`
|
||||
// (tracked in loops.initial_burst_remaining, decremented on each
|
||||
// completion). Best-effort: a docker/DB failure on the FIRST fire
|
||||
// logs but the loop still lives with next_fire_at set for its
|
||||
// cron.
|
||||
let parsed = parse_triggers(&body.triggers);
|
||||
let initial_burst = parsed.as_ref().map(|t| t.initial_burst).unwrap_or(0);
|
||||
let chain_on_completion = parsed.as_ref().map(|t| t.on_completion).unwrap_or(false);
|
||||
if initial_burst > 0 {
|
||||
ensure_loop_container(&state.pool, user.workspace_id.as_uuid(), id).await;
|
||||
let iter = cm_db::repo::loops::next_iteration(&state.pool, id)
|
||||
.await
|
||||
.unwrap_or(0);
|
||||
let task = compose_iteration_task(&state.pool, id, body.task_template.trim()).await;
|
||||
// Best-effort — we don't want to abort the loop-create response
|
||||
// just because Docker didn't answer the daemon health check.
|
||||
match cm_db::repo::loops::enqueue_iteration(
|
||||
&state.pool,
|
||||
id,
|
||||
user.workspace_id.as_uuid(),
|
||||
&task,
|
||||
&body.graph,
|
||||
iter,
|
||||
None,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(run_id) => {
|
||||
let _ = cm_db::repo::loops::mark_fired(&state.pool, id, run_id, next_fire_at).await;
|
||||
// Log the burst remaining so on_completion can advance it.
|
||||
// Stored on loops.initial_burst_remaining (added by the
|
||||
// same migration as loops.kind). Falls back silently on
|
||||
// schema absence so we don't break dev DBs mid-rollout.
|
||||
let remaining = initial_burst.saturating_sub(1) as i32;
|
||||
if remaining > 0 || chain_on_completion {
|
||||
let _ =
|
||||
cm_db::repo::loops::set_initial_burst_remaining(&state.pool, id, remaining)
|
||||
.await;
|
||||
}
|
||||
}
|
||||
Err(e) => eprintln!("loops::create: initial_burst enqueue failed: {e:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
Ok((
|
||||
StatusCode::CREATED,
|
||||
Json(LoopCreated {
|
||||
|
||||
Reference in New Issue
Block a user