loops: bridge research artifact into loop iterations (option C + b)
ci / gates (push) Successful in 7s
ci / frontend (push) Successful in 37s
ci / rust (push) Failing after 13m41s
ci / e2e (push) Has been skipped
ci / publish (push) Has been skipped

The bridge lets a coding loop "consume" an integrations research
artifact one INT-XX item per iteration. Options b (order-sequential
iteration) and C (snapshot in task_template + save the pointer for
future refresh) from the design discussion.

Migration 0042 — three new loops columns:
- source_research_topic_id — nullable pointer to research_topics.
- consumed_int_ids TEXT[]   — INT-XX ids the loop has completed.
  Advances when topology_worker parses "COMPLETED: INT-<NN>" markers
  from the run's final output (wired in a follow-up commit).
- current_int_index INT     — monotonic pointer for order-sequential
  iteration. Coordinator addresses INT-<current+1> unless prereqs are
  unmet, in which case it works on the smallest unblocking INT-XX and
  logs the reorder rationale.

Backend:
- cm_db::repo::loops::set_source_research_topic — bind/unbind pointer.
- cm_db::repo::loops::source_research_context   — read pointer + state.
- routes::loops::compose_iteration_task         — new caller-side helper
  that reads the pointer, fetches the topic's latest research_outcome,
  and prepends the artifact + focus instruction to task_template.
- run_now + webhook_receive both pass task_template through
  compose_iteration_task before enqueue. Standalone loops (no pointer)
  behave identically to before.
- CreateLoopRequest accepts `source_research_topic_id`, ownership-
  checked via research_topics::get before persist.

Frontend:
- New ResearchArtifactPicker modal — lists published topics, fetches
  the artifact on pick, returns (topic_id, markdown) to caller.
- LoopsWizard task_template step gains "Import from research artifact"
  button (right-aligned). Click opens the picker. On pick: task
  populates with the artifact markdown, pointer saved, textarea
  expands to 8 rows, small info strip shows "Loop is bound to topic
  <id>. Each iteration will focus on the next unconsumed INT-XX."
- Unlink button reverts to standalone loop mode.

Follow-up (next commit):
- topology_worker completion hook — parse "COMPLETED: INT-<NN>" out
  of the run's final output + update consumed_int_ids +
  current_int_index atomically. Without this, current_int_index stays
  at 0 forever and every iteration works on the same INT.
- Loop card refresh button — re-read source topic's latest outcome
  (useful after a reject-with-revision cycle on the source topic).
This commit is contained in:
Omar Sobh
2026-07-09 18:47:17 -07:00
parent 3ed1d03d2b
commit ce73abe5ab
6 changed files with 428 additions and 4 deletions
+79 -2
View File
@@ -40,6 +40,54 @@ fn loop_state_root() -> std::path::PathBuf {
/// return without blocking the run; the topology_worker will fall back /// return without blocking the run; the topology_worker will fall back
/// to the workspace-wide gateway. Records the container name + URL on /// to the workspace-wide gateway. Records the container name + URL on
/// the loop row on first success so subsequent fires skip re-writing. /// the loop row on first success so subsequent fires skip re-writing.
/// Build the task string an iteration will actually run.
///
/// - Standalone loops (no source research topic bound): returns
/// `task_template` verbatim, matching legacy behavior.
/// - Loops bound to a research topic: fetches the topic's latest
/// research_outcome and prepends
/// RESEARCH ARTIFACT (integration plan you're executing):
/// <markdown>
/// ITERATION FOCUS: next unconsumed INT-XX in order. If prereqs are
/// unmet, work on the smallest unblocking INT-XX. Log
/// `COMPLETED: INT-<NN>` at the end so the loop can advance.
/// ORIGINAL TASK:
/// <task_template>
/// The topology_worker's completion hook (P3) parses the COMPLETED
/// marker to update `consumed_int_ids`.
async fn compose_iteration_task(pool: &PgPool, loop_id: Uuid, task_template: &str) -> String {
let ctx = cm_db::repo::loops::source_research_context(pool, loop_id)
.await
.unwrap_or(None);
let Some((topic_id, consumed, current_idx)) = ctx else {
return task_template.to_string();
};
let outcome = match cm_db::repo::research_outcomes::latest(pool, topic_id).await {
Ok(Some(o)) => o,
_ => return task_template.to_string(),
};
let consumed_list = if consumed.is_empty() {
"(none yet)".to_string()
} else {
consumed.join(", ")
};
format!(
"RESEARCH ARTIFACT (integration plan you're executing, v{}):\n\
--- BEGIN ARTIFACT ---\n{}\n--- END ARTIFACT ---\n\n\
ITERATION FOCUS:\n\
- You are on iteration index {}.\n\
- Already completed: {}.\n\
- Address the NEXT unconsumed INT-XX item in the artifact, in order.\n\
- If the next item has unmet prerequisites, work on the smallest\n\
unblocking INT-XX instead AND log the reorder rationale in your\n\
opening turn so the reviewer can trace it.\n\
- Emit `COMPLETED: INT-<NN>` on its own line at the end of the run\n\
when the item is done — the loop advances on that marker.\n\n\
ORIGINAL TASK TEMPLATE:\n{}\n",
outcome.version, outcome.body_md, current_idx, consumed_list, task_template
)
}
async fn ensure_loop_container(pool: &PgPool, workspace_id: Uuid, loop_id: Uuid) { async fn ensure_loop_container(pool: &PgPool, workspace_id: Uuid, loop_id: Uuid) {
let docker = match crate::research_container::connect() { let docker = match crate::research_container::connect() {
Ok(d) => d, Ok(d) => d,
@@ -89,6 +137,12 @@ pub struct CreateLoopRequest {
pub teams: Vec<Uuid>, pub teams: Vec<Uuid>,
#[serde(default)] #[serde(default)]
pub orgs: Vec<Uuid>, pub orgs: Vec<Uuid>,
/// Optional research topic id. When set, each iteration prepends the
/// topic's latest research_outcome markdown + a "focus on next
/// unconsumed INT" instruction to the coordinator task. Migration
/// 0042 added the pointer column + consumed_int_ids tracking.
#[serde(default)]
pub source_research_topic_id: Option<Uuid>,
} }
fn default_repeat() -> Value { fn default_repeat() -> Value {
serde_json::json!({"kind": "infinite"}) serde_json::json!({"kind": "infinite"})
@@ -198,6 +252,27 @@ pub async fn create_loop(
apply_staffing(&state.pool, id, &body.agents, &body.teams, &body.orgs).await?; apply_staffing(&state.pool, id, &body.agents, &body.teams, &body.orgs).await?;
// Bridge to research (option C — snapshot in task_template + save
// pointer so a refresh can pull latest artifact into subsequent
// iterations). Ownership-checked via research_topics::get so we
// can't be tricked into pointing at another workspace's topic.
if let Some(topic_id) = body.source_research_topic_id {
let topic =
cm_db::repo::research_topics::get(&state.pool, topic_id, user.workspace_id.as_uuid())
.await?
.ok_or(ApiError::NotFound)?;
if let Err(e) = cm_db::repo::loops::set_source_research_topic(
&state.pool,
id,
user.workspace_id.as_uuid(),
Some(topic.id),
)
.await
{
eprintln!("loops::create: bind source research topic failed: {e:?}");
}
}
Ok(( Ok((
StatusCode::CREATED, StatusCode::CREATED,
Json(LoopCreated { Json(LoopCreated {
@@ -380,11 +455,12 @@ pub async fn run_now(
// never blocks the enqueue on Docker being unreachable. // never blocks the enqueue on Docker being unreachable.
ensure_loop_container(&state.pool, l.workspace_id, l.id).await; ensure_loop_container(&state.pool, l.workspace_id, l.id).await;
let iter = cm_db::repo::loops::next_iteration(&state.pool, l.id).await?; let iter = cm_db::repo::loops::next_iteration(&state.pool, l.id).await?;
let task = compose_iteration_task(&state.pool, l.id, &l.task_template).await;
let run_id = cm_db::repo::loops::enqueue_iteration( let run_id = cm_db::repo::loops::enqueue_iteration(
&state.pool, &state.pool,
l.id, l.id,
l.workspace_id, l.workspace_id,
&l.task_template, &task,
&l.graph, &l.graph,
iter, iter,
l.last_run_id, l.last_run_id,
@@ -430,11 +506,12 @@ pub async fn webhook_receive(
Ok(n) => n, Ok(n) => n,
Err(_) => return (StatusCode::INTERNAL_SERVER_ERROR, Json(Value::Null)), Err(_) => return (StatusCode::INTERNAL_SERVER_ERROR, Json(Value::Null)),
}; };
let task = compose_iteration_task(&state.pool, l.id, &l.task_template).await;
let run_id = match cm_db::repo::loops::enqueue_iteration( let run_id = match cm_db::repo::loops::enqueue_iteration(
&state.pool, &state.pool,
l.id, l.id,
l.workspace_id, l.workspace_id,
&l.task_template, &task,
&l.graph, &l.graph,
iter, iter,
l.last_run_id, l.last_run_id,
+53
View File
@@ -247,6 +247,59 @@ pub async fn set_zeroclaw_container(
Ok(()) Ok(())
} }
/// Bind (or unbind) a loop's source research topic. When set, the loop's
/// enqueue path prepends the topic's latest artifact + a "focus on the
/// next unconsumed INT" instruction to the coordinator task (option b,
/// order-sequential iteration).
pub async fn set_source_research_topic(
pool: &PgPool,
loop_id: Uuid,
workspace_id: Uuid,
source: Option<Uuid>,
) -> Result<(), DbError> {
sqlx::query(
"UPDATE loops
SET source_research_topic_id = $3, updated_at = now()
WHERE id = $1 AND workspace_id = $2",
)
.bind(loop_id)
.bind(workspace_id)
.bind(source)
.execute(pool)
.await?;
Ok(())
}
/// Read a loop's source research topic id + consumed INT ids +
/// current index. Used by the enqueue path when building the
/// coordinator task string. Missing rows / NULL columns return None
/// so the caller can fall back to the plain task_template.
pub async fn source_research_context(
pool: &PgPool,
loop_id: Uuid,
) -> Result<Option<(Uuid, Vec<String>, i32)>, DbError> {
use sqlx::Row;
let row = sqlx::query(
"SELECT source_research_topic_id, consumed_int_ids, current_int_index
FROM loops
WHERE id = $1",
)
.bind(loop_id)
.fetch_optional(pool)
.await?;
Ok(row.and_then(|r| {
let topic = r
.try_get::<Option<Uuid>, _>("source_research_topic_id")
.ok()
.flatten()?;
let consumed = r
.try_get::<Vec<String>, _>("consumed_int_ids")
.unwrap_or_default();
let idx = r.try_get::<i32, _>("current_int_index").unwrap_or(0);
Some((topic, consumed, idx))
}))
}
/// Read the per-loop gateway URL (or None if the loop hasn't spawned a /// Read the per-loop gateway URL (or None if the loop hasn't spawned a
/// container yet). Used by `topology_worker` to prefer the isolated /// container yet). Used by `topology_worker` to prefer the isolated
/// daemon over the workspace-wide one. /// daemon over the workspace-wide one.
@@ -32,6 +32,7 @@ import {
LoopStaffingStep, LoopStaffingStep,
type AgentSelection, type AgentSelection,
} from "./LoopStaffingStep"; } from "./LoopStaffingStep";
import { ResearchArtifactPicker } from "./ResearchArtifactPicker";
const mono = const mono =
"ui-monospace, SFMono-Regular, SF Mono, Menlo, Monaco, Consolas, monospace"; "ui-monospace, SFMono-Regular, SF Mono, Menlo, Monaco, Consolas, monospace";
@@ -75,6 +76,8 @@ export function LoopsWizard({
const [description, setDescription] = useState(initial?.description ?? ""); const [description, setDescription] = useState(initial?.description ?? "");
const [repo, setRepo] = useState<PickedRepo | null>(null); const [repo, setRepo] = useState<PickedRepo | null>(null);
const [task, setTask] = useState(initial?.task_template ?? ""); const [task, setTask] = useState(initial?.task_template ?? "");
const [sourceTopicId, setSourceTopicId] = useState<string | null>(null);
const [pickerOpen, setPickerOpen] = useState(false);
// When editing an existing loop, start in advanced mode so the caller sees // When editing an existing loop, start in advanced mode so the caller sees
// the exact graph they saved. Builder mode would rebuild + overwrite it. // the exact graph they saved. Builder mode would rebuild + overwrite it.
const [topologyMode, setTopologyMode] = useState<"builder" | "advanced">( const [topologyMode, setTopologyMode] = useState<"builder" | "advanced">(
@@ -262,6 +265,7 @@ export function LoopsWizard({
}, },
repeat_policy, repeat_policy,
...(repo ? { repo } : {}), ...(repo ? { repo } : {}),
...(sourceTopicId ? { source_research_topic_id: sourceTopicId } : {}),
...staffing, ...staffing,
}; };
if (editing && initial) { if (editing && initial) {
@@ -376,15 +380,58 @@ export function LoopsWizard({
{!noAgents && step === 3 && ( {!noAgents && step === 3 && (
<div style={{ display: "flex", flexDirection: "column", gap: 10 }}> <div style={{ display: "flex", flexDirection: "column", gap: 10 }}>
<div style={{ display: "flex", alignItems: "center", justifyContent: "space-between" }}>
<label style={labelStyle} htmlFor="loop-task">Task template</label> <label style={labelStyle} htmlFor="loop-task">Task template</label>
<button
type="button"
onClick={() => setPickerOpen(true)}
style={{
padding: "6px 12px",
borderRadius: 6,
border: `1px solid ${sourceTopicId ? "rgba(94,200,216,.5)" : "rgba(255,255,255,.15)"}`,
background: sourceTopicId ? "rgba(94,200,216,.08)" : "transparent",
color: sourceTopicId ? "#5ec8d8" : "#c3c3c8",
fontSize: 11,
fontFamily: mono,
cursor: "pointer",
}}
title="Populate task_template from a published research artifact"
>
{sourceTopicId ? "✓ From research" : "Import from research artifact"}
</button>
</div>
<textarea <textarea
id="loop-task" id="loop-task"
value={task} value={task}
onChange={(e) => setTask(e.target.value)} onChange={(e) => setTask(e.target.value)}
rows={4} rows={sourceTopicId ? 8 : 4}
placeholder="What should the loop's agents do each iteration?" placeholder="What should the loop's agents do each iteration?"
style={fieldStyle} style={fieldStyle}
/> />
{sourceTopicId ? (
<div style={{ display: "flex", alignItems: "center", gap: 8, fontSize: 10.5, fontFamily: mono, color: "#8a8a92" }}>
<span>
Loop is bound to research topic {sourceTopicId.slice(0, 8)}. Each
iteration will prepend the topic&apos;s latest artifact + focus on
the next unconsumed INT-XX item.
</span>
<button
type="button"
onClick={() => setSourceTopicId(null)}
style={{
padding: "2px 8px",
borderRadius: 4,
border: "1px solid rgba(255,138,122,.4)",
background: "transparent",
color: "#ff8a7a",
fontSize: 10,
cursor: "pointer",
}}
>
Unlink
</button>
</div>
) : null}
<div <div
style={{ style={{
@@ -661,6 +708,16 @@ export function LoopsWizard({
</div> </div>
)} )}
</div> </div>
{pickerOpen ? (
<ResearchArtifactPicker
onPick={({ topicId, artifactMd }) => {
setSourceTopicId(topicId);
setTask(artifactMd);
setPickerOpen(false);
}}
onClose={() => setPickerOpen(false)}
/>
) : null}
</div> </div>
); );
} }
@@ -0,0 +1,206 @@
"use client";
// Modal for the loop wizard's "Import from research artifact" flow.
// Lists published research topics that have at least one outcome row,
// user picks one → fetches the artifact markdown → returns
// (topic_id, markdown) to the parent, which snapshots it into the
// loop's task_template AND records the pointer so future iterations
// can auto-refresh from the source topic.
import { useEffect, useState } from "react";
import { X } from "lucide-react";
import { listTopics, type TopicListItem } from "@/lib/api/research";
const mono =
"ui-monospace, SFMono-Regular, SF Mono, Menlo, Monaco, Consolas, monospace";
export interface PickedArtifact {
topicId: string;
artifactMd: string;
}
export function ResearchArtifactPicker({
onPick,
onClose,
}: {
onPick: (a: PickedArtifact) => void;
onClose: () => void;
}) {
const [topics, setTopics] = useState<TopicListItem[]>([]);
const [loading, setLoading] = useState(true);
const [busyId, setBusyId] = useState<string | null>(null);
const [error, setError] = useState<string | null>(null);
useEffect(() => {
let alive = true;
(async () => {
try {
const rows = await listTopics();
if (alive) {
// Only topics that reached 'published' — they have an artifact
// downstream loops can consume.
setTopics(rows.filter((t) => t.status === "published"));
}
} catch {
if (alive) setError("Could not load research topics");
} finally {
if (alive) setLoading(false);
}
})();
return () => {
alive = false;
};
}, []);
useEffect(() => {
const onKey = (e: KeyboardEvent) => {
if (e.key === "Escape") onClose();
};
window.addEventListener("keydown", onKey);
return () => window.removeEventListener("keydown", onKey);
}, [onClose]);
async function pick(t: TopicListItem) {
if (busyId) return;
setBusyId(t.id);
setError(null);
try {
const res = await fetch(`/api/research/${t.id}/artifact`);
if (!res.ok) {
setError(`Artifact not available (${res.status})`);
setBusyId(null);
return;
}
const artifactMd = await res.text();
onPick({ topicId: t.id, artifactMd });
} catch {
setError("Could not fetch artifact");
setBusyId(null);
}
}
return (
<div
onClick={onClose}
role="presentation"
style={{
position: "fixed",
inset: 0,
zIndex: 120,
background: "rgba(0,0,0,.68)",
backdropFilter: "blur(4px)",
display: "flex",
alignItems: "center",
justifyContent: "center",
padding: 24,
}}
>
<div
onClick={(e) => e.stopPropagation()}
role="dialog"
aria-modal="true"
aria-label="Import research artifact"
style={{
width: "100%",
maxWidth: 560,
maxHeight: "80vh",
display: "flex",
flexDirection: "column",
borderRadius: 16,
background: "#0d0d10",
border: "1px solid rgba(255,255,255,.1)",
boxShadow: "0 30px 90px rgba(0,0,0,.6)",
padding: 22,
}}
>
<div style={{ display: "flex", alignItems: "flex-start", gap: 12, marginBottom: 12 }}>
<div style={{ flex: 1 }}>
<div style={{ fontSize: 17, fontWeight: 700, color: "#f3f3f5" }}>
Import from research artifact
</div>
<div style={{ fontSize: 12, color: "#8a8a92", marginTop: 2 }}>
Pick a published topic. Its artifact will fill the task template,
and each iteration will focus on the next unconsumed INT-XX item.
</div>
</div>
<button
type="button"
onClick={onClose}
aria-label="Close"
style={{
width: 30,
height: 30,
flex: "none",
borderRadius: 8,
border: "1px solid rgba(255,255,255,.12)",
background: "transparent",
color: "#9a9aa2",
cursor: "pointer",
display: "flex",
alignItems: "center",
justifyContent: "center",
}}
>
<X size={15} />
</button>
</div>
<div style={{ flex: 1, minHeight: 0, overflowY: "auto", display: "flex", flexDirection: "column", gap: 6 }}>
{loading ? (
<div style={{ padding: 24, textAlign: "center", fontFamily: mono, fontSize: 11, color: "#8a8a92" }}>
Loading…
</div>
) : topics.length === 0 ? (
<div style={{ padding: 24, textAlign: "center", fontFamily: mono, fontSize: 11, color: "#8a8a92", lineHeight: 1.6 }}>
No published topics yet.
<br />
Publish one from the Research tab first.
</div>
) : (
topics.map((t) => (
<button
key={t.id}
type="button"
onClick={() => pick(t)}
disabled={busyId === t.id}
style={{
display: "flex",
flexDirection: "column",
alignItems: "flex-start",
gap: 4,
padding: "10px 12px",
borderRadius: 10,
border: "1px solid rgba(255,255,255,.08)",
background: "#101014",
color: "#eaeaee",
cursor: busyId === t.id ? "wait" : "pointer",
textAlign: "left",
}}
>
<div style={{ fontWeight: 600, fontSize: 13 }}>{t.title}</div>
<div style={{ fontFamily: mono, fontSize: 10, color: "#8a8a92" }}>
{t.outcome_kind.replace("_", " ")} · published
</div>
</button>
))
)}
</div>
{error ? (
<div
role="alert"
style={{
marginTop: 12,
fontFamily: mono,
fontSize: 10.5,
color: "#ff8a7a",
}}
>
{error}
</div>
) : null}
</div>
</div>
);
}
+4
View File
@@ -79,6 +79,10 @@ export const createLoop = (body: {
agents?: { agent_id: string; role_slot?: string }[]; agents?: { agent_id: string; role_slot?: string }[];
teams?: string[]; teams?: string[];
orgs?: string[]; orgs?: string[];
/** Optional research topic this loop is executing. When set, each
* iteration prepends the topic's latest artifact + a "focus on next
* unconsumed INT" instruction (option b, order-sequential). */
source_research_topic_id?: string;
}) => }) =>
api<LoopCreated>("/api/loops", { api<LoopCreated>("/api/loops", {
method: "POST", method: "POST",
+27
View File
@@ -0,0 +1,27 @@
-- Bridges research artifacts into loops. A loop can be "based on" a
-- published research topic; each iteration prepends the artifact
-- markdown to the coordinator prompt so the team executes ONE
-- integration item per iteration (INT-XX from the integrations outcome).
--
-- source_research_topic_id — nullable pointer. When set, the loop's
-- enqueue path re-reads the topic's latest outcome and prepends it
-- as context. NULL means "standalone loop" (legacy behavior).
--
-- consumed_int_ids — free-form list of INT-XX ids the loop has already
-- completed. Advances when topology_worker parses "COMPLETED: INT-XX"
-- markers out of the run's final output. Coordinator uses this to
-- pick the NEXT unconsumed item.
--
-- current_int_index — monotonic pointer for order-sequential iteration
-- ("option b" in the design discussion). Coordinator addresses
-- INT-<current+1> unless it has unmet prereqs, in which case it
-- works on the smallest unblocking INT-XX and logs the reorder
-- rationale to run_events.
ALTER TABLE loops
ADD COLUMN source_research_topic_id UUID REFERENCES research_topics (id) ON DELETE SET NULL,
ADD COLUMN consumed_int_ids TEXT[] NOT NULL DEFAULT ARRAY[]::TEXT[],
ADD COLUMN current_int_index INT NOT NULL DEFAULT 0;
CREATE INDEX loops_source_research_idx
ON loops (source_research_topic_id)
WHERE source_research_topic_id IS NOT NULL;