loops: Path B container isolation (P2)
ci / gates (push) Successful in 6s
ci / rust (push) Failing after 11s
ci / frontend (push) Successful in 28s
ci / e2e (push) Has been skipped
ci / publish (push) Has been skipped

Symmetric with the research pipeline: every enabled loop can now have
its own per-loop team container so scheduled runs don't share state
with other loops or with research. Same daemon image, same clawmates
network, deterministic name loop-<id>-team.

Backend surface:
- Migration 0040 adds nullable `zeroclaw_container` +
  `zeroclaw_gateway_url` columns to loops (parallel to
  research_topics).
- research_container.rs grows loop_container_name_for(), spawn_loop()
  (state-only mount, no repo), and teardown_loop(). Kept in the same
  module to share the docker connect() + inherited_env() plumbing;
  each pattern gets its own labels (clawmates.role=loop-team) so ps
  filters can tell them apart.
- cm_db::repo::loops gains set_zeroclaw_container() +
  zeroclaw_gateway_url() (dynamic sqlx queries — no offline cache
  regen needed).
- cm_db::repo::topology_runs gets loop_id_for_run(): mirror of
  research_topic_id, used by the worker.

Wiring:
- routes/loops::run_now + webhook_receive call ensure_loop_container()
  before enqueuing an iteration. Idempotent: an already-running
  container is just reattached. Failures are logged and do NOT block
  the enqueue — topology_worker falls back to the workspace gateway
  when the URL isn't set on the loop.
- routes/loops::disable_loop + delete_loop both fire teardown_loop()
  so paused / deleted loops don't hold a docker slot.
- topology_worker's per-run URL resolution: existing research fast
  path unchanged; when it doesn't hit, the worker now looks up
  loop_id and reads the loop's gateway URL.

Deploy step (required on gw-04 for state to persist across container
restarts): add a `/var/lib/clawmates-loops:/var/lib/clawmates-loops`
bind mount + `CLAWMATES_LOOPS_STATE_ROOT=/var/lib/clawmates-loops`
env var to clawmates_server_1 in the compose. Without it, loops still
run — the state dir lives inside the API container's filesystem so
persistence is limited to that container's lifetime.

Follow-up:
- Scheduler-tick fires (cron-driven, not run_now) — they call
  enqueue_iteration in cm-scheduler and don't yet go through
  ensure_loop_container. Add a symmetric spawn there so cron fires
  also land on the isolated daemon.
- Compose file reconciliation — deploy/compose/docker-compose.yml in
  the repo has drifted from prod; when we sync it, add the loops mount
  at the same time.
This commit is contained in:
Omar Sobh
2026-07-09 16:02:08 -07:00
parent 1896b78cb7
commit 03f1830d1f
6 changed files with 263 additions and 6 deletions
+109
View File
@@ -266,3 +266,112 @@ pub async fn teardown(topic_id: Uuid) {
Err(e) => eprintln!("research_container::teardown({topic_id}): stop {name} failed: {e}"), Err(e) => eprintln!("research_container::teardown({topic_id}): stop {name} failed: {e}"),
} }
} }
// ── Loop container isolation (P2) ─────────────────────────────────────────
//
// Loops don't have a repo like research does, so their container has ONE
// bind mount (per-loop state at /zeroclaw-data) instead of two. Same daemon
// image, same network, same env — different name and label so we can tell
// research-team containers apart from loop-team ones at a glance.
/// Deterministic container name for a loop.
pub fn loop_container_name_for(loop_id: Uuid) -> String {
format!("loop-{loop_id}-team")
}
/// Spawn (or reattach to) the per-loop team container. Same idempotent
/// pattern as `spawn` — if the container exists it's just (re)started.
pub async fn spawn_loop(
docker: &Docker,
loop_id: Uuid,
state_host_path: &Path,
) -> Result<SpawnedContainer, String> {
let name = loop_container_name_for(loop_id);
let gateway_url = format!("http://{name}:42617");
match docker
.inspect_container(&name, None::<InspectContainerOptions>)
.await
{
Ok(info) => {
let running = info.state.as_ref().and_then(|s| s.running).unwrap_or(false);
if !running {
docker
.start_container(&name, None::<StartContainerOptions>)
.await
.map_err(|e| format!("start existing {name}: {e}"))?;
}
return Ok(SpawnedContainer { name, gateway_url });
}
Err(bollard::errors::Error::DockerResponseServerError {
status_code: 404, ..
}) => { /* create below */ }
Err(e) => return Err(format!("inspect {name}: {e}")),
}
std::fs::create_dir_all(state_host_path)
.map_err(|e| format!("mkdir {}: {e}", state_host_path.display()))?;
let mounts = vec![Mount {
target: Some("/zeroclaw-data".into()),
source: Some(state_host_path.to_string_lossy().into_owned()),
typ: Some(MountTypeEnum::BIND),
read_only: Some(false),
..Default::default()
}];
let host_config = HostConfig {
mounts: Some(mounts),
network_mode: Some(team_network()),
..Default::default()
};
let body = ContainerCreateBody {
image: Some(team_image()),
cmd: Some(vec!["daemon".into(), "--host".into(), "0.0.0.0".into()]),
env: Some(inherited_env()),
host_config: Some(host_config),
labels: Some(HashMap::from([
("clawmates.role".into(), "loop-team".into()),
("clawmates.loop.id".into(), loop_id.to_string()),
])),
..Default::default()
};
docker
.create_container(
Some(CreateContainerOptions {
name: Some(name.clone()),
..Default::default()
}),
body,
)
.await
.map_err(|e| format!("create {name}: {e}"))?;
docker
.start_container(&name, None::<StartContainerOptions>)
.await
.map_err(|e| format!("start {name}: {e}"))?;
Ok(SpawnedContainer { name, gateway_url })
}
/// Fire-and-forget teardown for a loop's runtime — called from
/// `disable_loop`, `delete_loop`, and any terminal transition. Same
/// non-fatal semantics as `teardown` (Docker unreachable and container
/// already-gone both log and return so the API contract is unaffected).
pub async fn teardown_loop(loop_id: Uuid) {
let name = loop_container_name_for(loop_id);
let docker = match connect() {
Ok(d) => d,
Err(e) => {
eprintln!("research_container::teardown_loop({loop_id}): docker connect failed: {e}");
return;
}
};
match stop(&docker, &name).await {
Ok(_) => eprintln!("research_container::teardown_loop({loop_id}): removed {name}"),
Err(e) => eprintln!("research_container::teardown_loop({loop_id}): stop {name} failed: {e}"),
}
}
+58
View File
@@ -21,9 +21,55 @@ use cm_runtime::scheduling::next_occurrence;
use hmac::{Hmac, Mac}; use hmac::{Hmac, Mac};
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use serde_json::Value; use serde_json::Value;
use sqlx::PgPool;
use time::OffsetDateTime; use time::OffsetDateTime;
use uuid::Uuid; use uuid::Uuid;
/// Root of per-loop state dirs on the host. Same overridable env pattern
/// as research_workspace_root — prod points at the bind-mounted volume
/// `/var/lib/clawmates-loops` on gw-04.
fn loop_state_root() -> std::path::PathBuf {
std::env::var("CLAWMATES_LOOPS_STATE_ROOT")
.map(std::path::PathBuf::from)
.unwrap_or_else(|_| std::path::PathBuf::from("/var/lib/clawmates-loops"))
}
/// Best-effort spawn of the per-loop team container before an iteration
/// is enqueued. Idempotent — an already-running container is just
/// reattached. Failures (docker unreachable, image missing) log and
/// return without blocking the run; the topology_worker will fall back
/// to the workspace-wide gateway. Records the container name + URL on
/// the loop row on first success so subsequent fires skip re-writing.
async fn ensure_loop_container(pool: &PgPool, workspace_id: Uuid, loop_id: Uuid) {
let docker = match crate::research_container::connect() {
Ok(d) => d,
Err(e) => {
eprintln!("loops::ensure_loop_container({loop_id}): docker connect failed: {e}");
return;
}
};
let state_root = loop_state_root().join(loop_id.to_string()).join("state");
let spawned =
match crate::research_container::spawn_loop(&docker, loop_id, &state_root).await {
Ok(s) => s,
Err(e) => {
eprintln!("loops::ensure_loop_container({loop_id}): spawn failed: {e}");
return;
}
};
if let Err(e) = cm_db::repo::loops::set_zeroclaw_container(
pool,
loop_id,
workspace_id,
&spawned.name,
&spawned.gateway_url,
)
.await
{
eprintln!("loops::ensure_loop_container({loop_id}): persist failed: {e:?}");
}
}
use crate::{ApiError, AppState, Authed}; use crate::{ApiError, AppState, Authed};
#[derive(Deserialize)] #[derive(Deserialize)]
@@ -285,6 +331,9 @@ pub async fn delete_loop(
Path(id): Path<Uuid>, Path(id): Path<Uuid>,
) -> Result<StatusCode, ApiError> { ) -> Result<StatusCode, ApiError> {
cm_db::repo::loops::delete(&state.pool, id, user.workspace_id.as_uuid()).await?; cm_db::repo::loops::delete(&state.pool, id, user.workspace_id.as_uuid()).await?;
// Tear down the per-loop container (P2). Fire-and-forget: the row
// is gone, so any Docker failure is a log-line, not an API failure.
crate::research_container::teardown_loop(id).await;
Ok(StatusCode::NO_CONTENT) Ok(StatusCode::NO_CONTENT)
} }
@@ -303,6 +352,10 @@ pub async fn disable_loop(
Path(id): Path<Uuid>, Path(id): Path<Uuid>,
) -> Result<StatusCode, ApiError> { ) -> Result<StatusCode, ApiError> {
cm_db::repo::loops::set_enabled(&state.pool, id, user.workspace_id.as_uuid(), false).await?; cm_db::repo::loops::set_enabled(&state.pool, id, user.workspace_id.as_uuid(), false).await?;
// Stop the per-loop container while disabled — re-enabling later will
// spawn a fresh one on the next `run_now` / webhook fire. Keeps
// paused loops from holding a docker slot.
crate::research_container::teardown_loop(id).await;
Ok(StatusCode::NO_CONTENT) Ok(StatusCode::NO_CONTENT)
} }
@@ -323,6 +376,10 @@ pub async fn run_now(
let l = cm_db::repo::loops::get(&state.pool, id, user.workspace_id.as_uuid()) let l = cm_db::repo::loops::get(&state.pool, id, user.workspace_id.as_uuid())
.await? .await?
.ok_or(ApiError::NotFound)?; .ok_or(ApiError::NotFound)?;
// P2: spawn the per-loop container before enqueue so topology_worker
// resolves its gateway URL when it picks up the run. Best-effort;
// never blocks the enqueue on Docker being unreachable.
ensure_loop_container(&state.pool, l.workspace_id, l.id).await;
let iter = cm_db::repo::loops::next_iteration(&state.pool, l.id).await?; let iter = cm_db::repo::loops::next_iteration(&state.pool, l.id).await?;
let run_id = cm_db::repo::loops::enqueue_iteration( let run_id = cm_db::repo::loops::enqueue_iteration(
&state.pool, &state.pool,
@@ -369,6 +426,7 @@ pub async fn webhook_receive(
return (StatusCode::UNAUTHORIZED, Json(Value::Null)); return (StatusCode::UNAUTHORIZED, Json(Value::Null));
} }
ensure_loop_container(&state.pool, l.workspace_id, l.id).await;
let iter = match cm_db::repo::loops::next_iteration(&state.pool, l.id).await { let iter = match cm_db::repo::loops::next_iteration(&state.pool, l.id).await {
Ok(n) => n, Ok(n) => n,
Err(_) => return (StatusCode::INTERNAL_SERVER_ERROR, Json(Value::Null)), Err(_) => return (StatusCode::INTERNAL_SERVER_ERROR, Json(Value::Null)),
+21 -6
View File
@@ -91,12 +91,12 @@ async fn run_job(
.and_then(|c| serde_json::from_value(c).ok()) .and_then(|c| serde_json::from_value(c).ok())
.unwrap_or_default(); .unwrap_or_default();
// If this run belongs to a research topic and that topic has a per-team // If this run belongs to a research topic OR a loop with a per-team
// ZeroClaw container spawned (see research_container::spawn), point the // ZeroClaw container spawned, point the executor at THAT container's
// executor at THAT container's gateway URL so the run's turns hit its // gateway URL so the run's turns hit its isolated daemon instead of
// isolated daemon instead of the workspace-wide one. Falls back to the // the workspace-wide one. Falls back to the env-derived executor when
// env-derived executor when there's no per-topic container (non-research // there's no per-topic/loop container (chat sessions, or research/loop
// runs, or research runs where spawn failed and we recorded no URL). // runs where spawn failed and we recorded no URL).
let per_topic_url = match cm_db::repo::topology_runs::research_topic_id(pool, id).await { let per_topic_url = match cm_db::repo::topology_runs::research_topic_id(pool, id).await {
Ok(Some(topic_id)) => { Ok(Some(topic_id)) => {
match cm_db::repo::research_topics::get(pool, topic_id, job.workspace_id).await { match cm_db::repo::research_topics::get(pool, topic_id, job.workspace_id).await {
@@ -106,6 +106,21 @@ async fn run_job(
} }
_ => None, _ => None,
}; };
// Loop lookup runs only when the research lookup didn't hit — a run
// is bound to at most one of {topic, loop}. This preserves the
// existing research fast path unchanged.
let per_topic_url = if per_topic_url.is_some() {
per_topic_url
} else {
match cm_db::repo::topology_runs::loop_id_for_run(pool, id).await {
Ok(Some(loop_id)) => {
cm_db::repo::loops::zeroclaw_gateway_url(pool, loop_id)
.await
.unwrap_or(None)
}
_ => None,
}
};
if let Some(url) = &per_topic_url { if let Some(url) = &per_topic_url {
// Best-effort readiness gate — a freshly-spawned team container may // Best-effort readiness gate — a freshly-spawned team container may
// still be starting when the worker claims the run. Cap the wait so // still be starting when the worker claims the run. Cap the wait so
+40
View File
@@ -221,6 +221,46 @@ pub async fn delete(pool: &PgPool, id: Uuid, workspace_id: Uuid) -> Result<(), D
Ok(()) Ok(())
} }
/// Persist the per-loop team container name + gateway URL after a
/// successful `spawn_loop` (P2). Dynamic query so the new columns don't
/// need a fresh .sqlx offline cache entry.
pub async fn set_zeroclaw_container(
pool: &PgPool,
loop_id: Uuid,
workspace_id: Uuid,
container: &str,
gateway_url: &str,
) -> Result<(), DbError> {
sqlx::query(
"UPDATE loops
SET zeroclaw_container = $3,
zeroclaw_gateway_url = $4,
updated_at = now()
WHERE id = $1 AND workspace_id = $2",
)
.bind(loop_id)
.bind(workspace_id)
.bind(container)
.bind(gateway_url)
.execute(pool)
.await?;
Ok(())
}
/// Read the per-loop gateway URL (or None if the loop hasn't spawned a
/// container yet). Used by `topology_worker` to prefer the isolated
/// daemon over the workspace-wide one.
pub async fn zeroclaw_gateway_url(pool: &PgPool, loop_id: Uuid) -> Result<Option<String>, DbError> {
use sqlx::Row;
let row = sqlx::query(
"SELECT zeroclaw_gateway_url FROM loops WHERE id = $1",
)
.bind(loop_id)
.fetch_optional(pool)
.await?;
Ok(row.and_then(|r| r.try_get::<Option<String>, _>("zeroclaw_gateway_url").ok().flatten()))
}
// --- Staffing --------------------------------------------------------------- // --- Staffing ---------------------------------------------------------------
// //
// Loops attach agents, teams, and/or orgs. The three join tables are // Loops attach agents, teams, and/or orgs. The three join tables are
+15
View File
@@ -180,6 +180,21 @@ pub async fn research_topic_id(pool: &PgPool, id: Uuid) -> Result<Option<Uuid>,
Ok(row.and_then(|r| r.research_topic_id)) Ok(row.and_then(|r| r.research_topic_id))
} }
/// The loop this run belongs to, if any. Mirror of `research_topic_id`.
/// Used by `topology_worker` to look up the per-loop gateway URL so a
/// loop's runs land on its isolated daemon (P2). Non-loop runs return
/// None.
pub async fn loop_id_for_run(pool: &PgPool, id: Uuid) -> Result<Option<Uuid>, DbError> {
use sqlx::Row;
let row: Option<sqlx::postgres::PgRow> = sqlx::query(
"SELECT loop_id FROM topology_runs WHERE id = $1",
)
.bind(id)
.fetch_optional(pool)
.await?;
Ok(row.and_then(|r| r.try_get::<Option<Uuid>, _>("loop_id").ok().flatten()))
}
/// Enqueue a durable run bound to a research topic. `research_topic_id` is /// Enqueue a durable run bound to a research topic. `research_topic_id` is
/// stored so `notify_run_completed` can flip the owning topic /// stored so `notify_run_completed` can flip the owning topic
/// `processing → reviewing` when its last run terminates (see /// `processing → reviewing` when its last run terminates (see
+20
View File
@@ -0,0 +1,20 @@
-- Path B for loops (P2): give every enabled loop its own per-loop team
-- container so scheduled runs don't share state / memory with other loops
-- or with the research pipeline. Two nullable columns:
--
-- zeroclaw_container — deterministic docker container name; NULL
-- until the first `run_now` / `webhook_receive`
-- / scheduler tick spawns one, set + reused
-- on subsequent fires.
-- zeroclaw_gateway_url — http://<container>:42617; topology_worker
-- prefers this over the workspace-wide URL when
-- the run belongs to a loop with a spawned
-- container.
--
-- Fallback semantics match research: NULL means "no per-loop container
-- yet, use the env-derived executor". Never-spawned loops (e.g. from
-- workspaces where docker isn't reachable) keep firing on the shared
-- gateway — degraded, not broken.
ALTER TABLE loops
ADD COLUMN zeroclaw_container TEXT,
ADD COLUMN zeroclaw_gateway_url TEXT;