loops: Path B container isolation (P2)
Symmetric with the research pipeline: every enabled loop can now have its own per-loop team container so scheduled runs don't share state with other loops or with research. Same daemon image, same clawmates network, deterministic name loop-<id>-team. Backend surface: - Migration 0040 adds nullable `zeroclaw_container` + `zeroclaw_gateway_url` columns to loops (parallel to research_topics). - research_container.rs grows loop_container_name_for(), spawn_loop() (state-only mount, no repo), and teardown_loop(). Kept in the same module to share the docker connect() + inherited_env() plumbing; each pattern gets its own labels (clawmates.role=loop-team) so ps filters can tell them apart. - cm_db::repo::loops gains set_zeroclaw_container() + zeroclaw_gateway_url() (dynamic sqlx queries — no offline cache regen needed). - cm_db::repo::topology_runs gets loop_id_for_run(): mirror of research_topic_id, used by the worker. Wiring: - routes/loops::run_now + webhook_receive call ensure_loop_container() before enqueuing an iteration. Idempotent: an already-running container is just reattached. Failures are logged and do NOT block the enqueue — topology_worker falls back to the workspace gateway when the URL isn't set on the loop. - routes/loops::disable_loop + delete_loop both fire teardown_loop() so paused / deleted loops don't hold a docker slot. - topology_worker's per-run URL resolution: existing research fast path unchanged; when it doesn't hit, the worker now looks up loop_id and reads the loop's gateway URL. Deploy step (required on gw-04 for state to persist across container restarts): add a `/var/lib/clawmates-loops:/var/lib/clawmates-loops` bind mount + `CLAWMATES_LOOPS_STATE_ROOT=/var/lib/clawmates-loops` env var to clawmates_server_1 in the compose. Without it, loops still run — the state dir lives inside the API container's filesystem so persistence is limited to that container's lifetime. Follow-up: - Scheduler-tick fires (cron-driven, not run_now) — they call enqueue_iteration in cm-scheduler and don't yet go through ensure_loop_container. Add a symmetric spawn there so cron fires also land on the isolated daemon. - Compose file reconciliation — deploy/compose/docker-compose.yml in the repo has drifted from prod; when we sync it, add the loops mount at the same time.
This commit is contained in:
@@ -266,3 +266,112 @@ pub async fn teardown(topic_id: Uuid) {
|
|||||||
Err(e) => eprintln!("research_container::teardown({topic_id}): stop {name} failed: {e}"),
|
Err(e) => eprintln!("research_container::teardown({topic_id}): stop {name} failed: {e}"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── Loop container isolation (P2) ─────────────────────────────────────────
|
||||||
|
//
|
||||||
|
// Loops don't have a repo like research does, so their container has ONE
|
||||||
|
// bind mount (per-loop state at /zeroclaw-data) instead of two. Same daemon
|
||||||
|
// image, same network, same env — different name and label so we can tell
|
||||||
|
// research-team containers apart from loop-team ones at a glance.
|
||||||
|
|
||||||
|
/// Deterministic container name for a loop.
|
||||||
|
pub fn loop_container_name_for(loop_id: Uuid) -> String {
|
||||||
|
format!("loop-{loop_id}-team")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Spawn (or reattach to) the per-loop team container. Same idempotent
|
||||||
|
/// pattern as `spawn` — if the container exists it's just (re)started.
|
||||||
|
pub async fn spawn_loop(
|
||||||
|
docker: &Docker,
|
||||||
|
loop_id: Uuid,
|
||||||
|
state_host_path: &Path,
|
||||||
|
) -> Result<SpawnedContainer, String> {
|
||||||
|
let name = loop_container_name_for(loop_id);
|
||||||
|
let gateway_url = format!("http://{name}:42617");
|
||||||
|
|
||||||
|
match docker
|
||||||
|
.inspect_container(&name, None::<InspectContainerOptions>)
|
||||||
|
.await
|
||||||
|
{
|
||||||
|
Ok(info) => {
|
||||||
|
let running = info.state.as_ref().and_then(|s| s.running).unwrap_or(false);
|
||||||
|
if !running {
|
||||||
|
docker
|
||||||
|
.start_container(&name, None::<StartContainerOptions>)
|
||||||
|
.await
|
||||||
|
.map_err(|e| format!("start existing {name}: {e}"))?;
|
||||||
|
}
|
||||||
|
return Ok(SpawnedContainer { name, gateway_url });
|
||||||
|
}
|
||||||
|
Err(bollard::errors::Error::DockerResponseServerError {
|
||||||
|
status_code: 404, ..
|
||||||
|
}) => { /* create below */ }
|
||||||
|
Err(e) => return Err(format!("inspect {name}: {e}")),
|
||||||
|
}
|
||||||
|
|
||||||
|
std::fs::create_dir_all(state_host_path)
|
||||||
|
.map_err(|e| format!("mkdir {}: {e}", state_host_path.display()))?;
|
||||||
|
|
||||||
|
let mounts = vec![Mount {
|
||||||
|
target: Some("/zeroclaw-data".into()),
|
||||||
|
source: Some(state_host_path.to_string_lossy().into_owned()),
|
||||||
|
typ: Some(MountTypeEnum::BIND),
|
||||||
|
read_only: Some(false),
|
||||||
|
..Default::default()
|
||||||
|
}];
|
||||||
|
|
||||||
|
let host_config = HostConfig {
|
||||||
|
mounts: Some(mounts),
|
||||||
|
network_mode: Some(team_network()),
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
|
||||||
|
let body = ContainerCreateBody {
|
||||||
|
image: Some(team_image()),
|
||||||
|
cmd: Some(vec!["daemon".into(), "--host".into(), "0.0.0.0".into()]),
|
||||||
|
env: Some(inherited_env()),
|
||||||
|
host_config: Some(host_config),
|
||||||
|
labels: Some(HashMap::from([
|
||||||
|
("clawmates.role".into(), "loop-team".into()),
|
||||||
|
("clawmates.loop.id".into(), loop_id.to_string()),
|
||||||
|
])),
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
|
||||||
|
docker
|
||||||
|
.create_container(
|
||||||
|
Some(CreateContainerOptions {
|
||||||
|
name: Some(name.clone()),
|
||||||
|
..Default::default()
|
||||||
|
}),
|
||||||
|
body,
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.map_err(|e| format!("create {name}: {e}"))?;
|
||||||
|
|
||||||
|
docker
|
||||||
|
.start_container(&name, None::<StartContainerOptions>)
|
||||||
|
.await
|
||||||
|
.map_err(|e| format!("start {name}: {e}"))?;
|
||||||
|
|
||||||
|
Ok(SpawnedContainer { name, gateway_url })
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Fire-and-forget teardown for a loop's runtime — called from
|
||||||
|
/// `disable_loop`, `delete_loop`, and any terminal transition. Same
|
||||||
|
/// non-fatal semantics as `teardown` (Docker unreachable and container
|
||||||
|
/// already-gone both log and return so the API contract is unaffected).
|
||||||
|
pub async fn teardown_loop(loop_id: Uuid) {
|
||||||
|
let name = loop_container_name_for(loop_id);
|
||||||
|
let docker = match connect() {
|
||||||
|
Ok(d) => d,
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!("research_container::teardown_loop({loop_id}): docker connect failed: {e}");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
match stop(&docker, &name).await {
|
||||||
|
Ok(_) => eprintln!("research_container::teardown_loop({loop_id}): removed {name}"),
|
||||||
|
Err(e) => eprintln!("research_container::teardown_loop({loop_id}): stop {name} failed: {e}"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -21,9 +21,55 @@ use cm_runtime::scheduling::next_occurrence;
|
|||||||
use hmac::{Hmac, Mac};
|
use hmac::{Hmac, Mac};
|
||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
use serde_json::Value;
|
use serde_json::Value;
|
||||||
|
use sqlx::PgPool;
|
||||||
use time::OffsetDateTime;
|
use time::OffsetDateTime;
|
||||||
use uuid::Uuid;
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
/// Root of per-loop state dirs on the host. Same overridable env pattern
|
||||||
|
/// as research_workspace_root — prod points at the bind-mounted volume
|
||||||
|
/// `/var/lib/clawmates-loops` on gw-04.
|
||||||
|
fn loop_state_root() -> std::path::PathBuf {
|
||||||
|
std::env::var("CLAWMATES_LOOPS_STATE_ROOT")
|
||||||
|
.map(std::path::PathBuf::from)
|
||||||
|
.unwrap_or_else(|_| std::path::PathBuf::from("/var/lib/clawmates-loops"))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Best-effort spawn of the per-loop team container before an iteration
|
||||||
|
/// is enqueued. Idempotent — an already-running container is just
|
||||||
|
/// reattached. Failures (docker unreachable, image missing) log and
|
||||||
|
/// return without blocking the run; the topology_worker will fall back
|
||||||
|
/// to the workspace-wide gateway. Records the container name + URL on
|
||||||
|
/// the loop row on first success so subsequent fires skip re-writing.
|
||||||
|
async fn ensure_loop_container(pool: &PgPool, workspace_id: Uuid, loop_id: Uuid) {
|
||||||
|
let docker = match crate::research_container::connect() {
|
||||||
|
Ok(d) => d,
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!("loops::ensure_loop_container({loop_id}): docker connect failed: {e}");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
let state_root = loop_state_root().join(loop_id.to_string()).join("state");
|
||||||
|
let spawned =
|
||||||
|
match crate::research_container::spawn_loop(&docker, loop_id, &state_root).await {
|
||||||
|
Ok(s) => s,
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!("loops::ensure_loop_container({loop_id}): spawn failed: {e}");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
if let Err(e) = cm_db::repo::loops::set_zeroclaw_container(
|
||||||
|
pool,
|
||||||
|
loop_id,
|
||||||
|
workspace_id,
|
||||||
|
&spawned.name,
|
||||||
|
&spawned.gateway_url,
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
{
|
||||||
|
eprintln!("loops::ensure_loop_container({loop_id}): persist failed: {e:?}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
use crate::{ApiError, AppState, Authed};
|
use crate::{ApiError, AppState, Authed};
|
||||||
|
|
||||||
#[derive(Deserialize)]
|
#[derive(Deserialize)]
|
||||||
@@ -285,6 +331,9 @@ pub async fn delete_loop(
|
|||||||
Path(id): Path<Uuid>,
|
Path(id): Path<Uuid>,
|
||||||
) -> Result<StatusCode, ApiError> {
|
) -> Result<StatusCode, ApiError> {
|
||||||
cm_db::repo::loops::delete(&state.pool, id, user.workspace_id.as_uuid()).await?;
|
cm_db::repo::loops::delete(&state.pool, id, user.workspace_id.as_uuid()).await?;
|
||||||
|
// Tear down the per-loop container (P2). Fire-and-forget: the row
|
||||||
|
// is gone, so any Docker failure is a log-line, not an API failure.
|
||||||
|
crate::research_container::teardown_loop(id).await;
|
||||||
Ok(StatusCode::NO_CONTENT)
|
Ok(StatusCode::NO_CONTENT)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -303,6 +352,10 @@ pub async fn disable_loop(
|
|||||||
Path(id): Path<Uuid>,
|
Path(id): Path<Uuid>,
|
||||||
) -> Result<StatusCode, ApiError> {
|
) -> Result<StatusCode, ApiError> {
|
||||||
cm_db::repo::loops::set_enabled(&state.pool, id, user.workspace_id.as_uuid(), false).await?;
|
cm_db::repo::loops::set_enabled(&state.pool, id, user.workspace_id.as_uuid(), false).await?;
|
||||||
|
// Stop the per-loop container while disabled — re-enabling later will
|
||||||
|
// spawn a fresh one on the next `run_now` / webhook fire. Keeps
|
||||||
|
// paused loops from holding a docker slot.
|
||||||
|
crate::research_container::teardown_loop(id).await;
|
||||||
Ok(StatusCode::NO_CONTENT)
|
Ok(StatusCode::NO_CONTENT)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -323,6 +376,10 @@ pub async fn run_now(
|
|||||||
let l = cm_db::repo::loops::get(&state.pool, id, user.workspace_id.as_uuid())
|
let l = cm_db::repo::loops::get(&state.pool, id, user.workspace_id.as_uuid())
|
||||||
.await?
|
.await?
|
||||||
.ok_or(ApiError::NotFound)?;
|
.ok_or(ApiError::NotFound)?;
|
||||||
|
// P2: spawn the per-loop container before enqueue so topology_worker
|
||||||
|
// resolves its gateway URL when it picks up the run. Best-effort;
|
||||||
|
// never blocks the enqueue on Docker being unreachable.
|
||||||
|
ensure_loop_container(&state.pool, l.workspace_id, l.id).await;
|
||||||
let iter = cm_db::repo::loops::next_iteration(&state.pool, l.id).await?;
|
let iter = cm_db::repo::loops::next_iteration(&state.pool, l.id).await?;
|
||||||
let run_id = cm_db::repo::loops::enqueue_iteration(
|
let run_id = cm_db::repo::loops::enqueue_iteration(
|
||||||
&state.pool,
|
&state.pool,
|
||||||
@@ -369,6 +426,7 @@ pub async fn webhook_receive(
|
|||||||
return (StatusCode::UNAUTHORIZED, Json(Value::Null));
|
return (StatusCode::UNAUTHORIZED, Json(Value::Null));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
ensure_loop_container(&state.pool, l.workspace_id, l.id).await;
|
||||||
let iter = match cm_db::repo::loops::next_iteration(&state.pool, l.id).await {
|
let iter = match cm_db::repo::loops::next_iteration(&state.pool, l.id).await {
|
||||||
Ok(n) => n,
|
Ok(n) => n,
|
||||||
Err(_) => return (StatusCode::INTERNAL_SERVER_ERROR, Json(Value::Null)),
|
Err(_) => return (StatusCode::INTERNAL_SERVER_ERROR, Json(Value::Null)),
|
||||||
|
|||||||
@@ -91,12 +91,12 @@ async fn run_job(
|
|||||||
.and_then(|c| serde_json::from_value(c).ok())
|
.and_then(|c| serde_json::from_value(c).ok())
|
||||||
.unwrap_or_default();
|
.unwrap_or_default();
|
||||||
|
|
||||||
// If this run belongs to a research topic and that topic has a per-team
|
// If this run belongs to a research topic OR a loop with a per-team
|
||||||
// ZeroClaw container spawned (see research_container::spawn), point the
|
// ZeroClaw container spawned, point the executor at THAT container's
|
||||||
// executor at THAT container's gateway URL so the run's turns hit its
|
// gateway URL so the run's turns hit its isolated daemon instead of
|
||||||
// isolated daemon instead of the workspace-wide one. Falls back to the
|
// the workspace-wide one. Falls back to the env-derived executor when
|
||||||
// env-derived executor when there's no per-topic container (non-research
|
// there's no per-topic/loop container (chat sessions, or research/loop
|
||||||
// runs, or research runs where spawn failed and we recorded no URL).
|
// runs where spawn failed and we recorded no URL).
|
||||||
let per_topic_url = match cm_db::repo::topology_runs::research_topic_id(pool, id).await {
|
let per_topic_url = match cm_db::repo::topology_runs::research_topic_id(pool, id).await {
|
||||||
Ok(Some(topic_id)) => {
|
Ok(Some(topic_id)) => {
|
||||||
match cm_db::repo::research_topics::get(pool, topic_id, job.workspace_id).await {
|
match cm_db::repo::research_topics::get(pool, topic_id, job.workspace_id).await {
|
||||||
@@ -106,6 +106,21 @@ async fn run_job(
|
|||||||
}
|
}
|
||||||
_ => None,
|
_ => None,
|
||||||
};
|
};
|
||||||
|
// Loop lookup runs only when the research lookup didn't hit — a run
|
||||||
|
// is bound to at most one of {topic, loop}. This preserves the
|
||||||
|
// existing research fast path unchanged.
|
||||||
|
let per_topic_url = if per_topic_url.is_some() {
|
||||||
|
per_topic_url
|
||||||
|
} else {
|
||||||
|
match cm_db::repo::topology_runs::loop_id_for_run(pool, id).await {
|
||||||
|
Ok(Some(loop_id)) => {
|
||||||
|
cm_db::repo::loops::zeroclaw_gateway_url(pool, loop_id)
|
||||||
|
.await
|
||||||
|
.unwrap_or(None)
|
||||||
|
}
|
||||||
|
_ => None,
|
||||||
|
}
|
||||||
|
};
|
||||||
if let Some(url) = &per_topic_url {
|
if let Some(url) = &per_topic_url {
|
||||||
// Best-effort readiness gate — a freshly-spawned team container may
|
// Best-effort readiness gate — a freshly-spawned team container may
|
||||||
// still be starting when the worker claims the run. Cap the wait so
|
// still be starting when the worker claims the run. Cap the wait so
|
||||||
|
|||||||
@@ -221,6 +221,46 @@ pub async fn delete(pool: &PgPool, id: Uuid, workspace_id: Uuid) -> Result<(), D
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Persist the per-loop team container name + gateway URL after a
|
||||||
|
/// successful `spawn_loop` (P2). Dynamic query so the new columns don't
|
||||||
|
/// need a fresh .sqlx offline cache entry.
|
||||||
|
pub async fn set_zeroclaw_container(
|
||||||
|
pool: &PgPool,
|
||||||
|
loop_id: Uuid,
|
||||||
|
workspace_id: Uuid,
|
||||||
|
container: &str,
|
||||||
|
gateway_url: &str,
|
||||||
|
) -> Result<(), DbError> {
|
||||||
|
sqlx::query(
|
||||||
|
"UPDATE loops
|
||||||
|
SET zeroclaw_container = $3,
|
||||||
|
zeroclaw_gateway_url = $4,
|
||||||
|
updated_at = now()
|
||||||
|
WHERE id = $1 AND workspace_id = $2",
|
||||||
|
)
|
||||||
|
.bind(loop_id)
|
||||||
|
.bind(workspace_id)
|
||||||
|
.bind(container)
|
||||||
|
.bind(gateway_url)
|
||||||
|
.execute(pool)
|
||||||
|
.await?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Read the per-loop gateway URL (or None if the loop hasn't spawned a
|
||||||
|
/// container yet). Used by `topology_worker` to prefer the isolated
|
||||||
|
/// daemon over the workspace-wide one.
|
||||||
|
pub async fn zeroclaw_gateway_url(pool: &PgPool, loop_id: Uuid) -> Result<Option<String>, DbError> {
|
||||||
|
use sqlx::Row;
|
||||||
|
let row = sqlx::query(
|
||||||
|
"SELECT zeroclaw_gateway_url FROM loops WHERE id = $1",
|
||||||
|
)
|
||||||
|
.bind(loop_id)
|
||||||
|
.fetch_optional(pool)
|
||||||
|
.await?;
|
||||||
|
Ok(row.and_then(|r| r.try_get::<Option<String>, _>("zeroclaw_gateway_url").ok().flatten()))
|
||||||
|
}
|
||||||
|
|
||||||
// --- Staffing ---------------------------------------------------------------
|
// --- Staffing ---------------------------------------------------------------
|
||||||
//
|
//
|
||||||
// Loops attach agents, teams, and/or orgs. The three join tables are
|
// Loops attach agents, teams, and/or orgs. The three join tables are
|
||||||
|
|||||||
@@ -180,6 +180,21 @@ pub async fn research_topic_id(pool: &PgPool, id: Uuid) -> Result<Option<Uuid>,
|
|||||||
Ok(row.and_then(|r| r.research_topic_id))
|
Ok(row.and_then(|r| r.research_topic_id))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The loop this run belongs to, if any. Mirror of `research_topic_id`.
|
||||||
|
/// Used by `topology_worker` to look up the per-loop gateway URL so a
|
||||||
|
/// loop's runs land on its isolated daemon (P2). Non-loop runs return
|
||||||
|
/// None.
|
||||||
|
pub async fn loop_id_for_run(pool: &PgPool, id: Uuid) -> Result<Option<Uuid>, DbError> {
|
||||||
|
use sqlx::Row;
|
||||||
|
let row: Option<sqlx::postgres::PgRow> = sqlx::query(
|
||||||
|
"SELECT loop_id FROM topology_runs WHERE id = $1",
|
||||||
|
)
|
||||||
|
.bind(id)
|
||||||
|
.fetch_optional(pool)
|
||||||
|
.await?;
|
||||||
|
Ok(row.and_then(|r| r.try_get::<Option<Uuid>, _>("loop_id").ok().flatten()))
|
||||||
|
}
|
||||||
|
|
||||||
/// Enqueue a durable run bound to a research topic. `research_topic_id` is
|
/// Enqueue a durable run bound to a research topic. `research_topic_id` is
|
||||||
/// stored so `notify_run_completed` can flip the owning topic
|
/// stored so `notify_run_completed` can flip the owning topic
|
||||||
/// `processing → reviewing` when its last run terminates (see
|
/// `processing → reviewing` when its last run terminates (see
|
||||||
|
|||||||
@@ -0,0 +1,20 @@
|
|||||||
|
-- Path B for loops (P2): give every enabled loop its own per-loop team
|
||||||
|
-- container so scheduled runs don't share state / memory with other loops
|
||||||
|
-- or with the research pipeline. Two nullable columns:
|
||||||
|
--
|
||||||
|
-- zeroclaw_container — deterministic docker container name; NULL
|
||||||
|
-- until the first `run_now` / `webhook_receive`
|
||||||
|
-- / scheduler tick spawns one, set + reused
|
||||||
|
-- on subsequent fires.
|
||||||
|
-- zeroclaw_gateway_url — http://<container>:42617; topology_worker
|
||||||
|
-- prefers this over the workspace-wide URL when
|
||||||
|
-- the run belongs to a loop with a spawned
|
||||||
|
-- container.
|
||||||
|
--
|
||||||
|
-- Fallback semantics match research: NULL means "no per-loop container
|
||||||
|
-- yet, use the env-derived executor". Never-spawned loops (e.g. from
|
||||||
|
-- workspaces where docker isn't reachable) keep firing on the shared
|
||||||
|
-- gateway — degraded, not broken.
|
||||||
|
ALTER TABLE loops
|
||||||
|
ADD COLUMN zeroclaw_container TEXT,
|
||||||
|
ADD COLUMN zeroclaw_gateway_url TEXT;
|
||||||
Reference in New Issue
Block a user