osobh/clawmates
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(runtime): funnel every reap path through purge_agent; sweep node-placed orphans

Browse Source 
Agent containers leaked two independent ways.

1. The four-step teardown (deprovision ZeroClaw -> reap_sandbox -> unlink
   .brain/.onion -> hard_purge) was inlined at three call sites and two had
   drifted. missions.rs::reap_mission_resources skipped reap_sandbox;
   topology_worker::maybe_teardown_ephemeral_team skipped it and the brain
   unlink; DELETE /api/claws/{id} (soft delete) released nothing at all, so an
   offline claw that can never run again kept its container and bind mount
   forever. All four now funnel through claws::purge_agent, with
   release_claw_resources for the soft-delete case (containers gone, rows kept).

2. Both orphan reapers listed only the local driver, so a container placed on a
   fleet node was invisible to the only backstop that could find it -- this is
   what accumulated 144 tc-agent-* orphans on one node. NodeDriverProvider gains
   node_ids() (backed by NodeHub::online_ids) and both reapers now sweep every
   connected node. The remote sweep is TTL-only on purpose: the boot pass runs
   with Duration::ZERO and would otherwise kill a container another instance is
   mid-provision on.

Why it was invisible: agent_containers.agent_id is ON DELETE CASCADE, so
hard_purge took the registry row with the agent and left the container
permanently unreferenceable.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Omar Sobh
2026-07-30 10:41:22 -07:00
co-authored by Claude Opus 5
parent a78f308eea
commit c573480955
6 changed files with 213 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/cm-api/src/topology_worker.rs
+16 -12
View File
@@ -129,7 +129,7 @@ async fn run_job(
let _ = cm_db::repo::topology_runs::fail(pool, id, &e).await;
}
}
maybe_teardown_ephemeral_team(pool, id).await;
maybe_teardown_ephemeral_team(pool, runtime, id).await;
return;
}
@@ -219,14 +219,14 @@ async fn run_job(
}
}
}
maybe_teardown_ephemeral_team(pool, id).await;
maybe_teardown_ephemeral_team(pool, runtime, id).await;
}
/// Post-terminal hook: if this run's team is `ephemeral` and no siblings are
/// still in flight, deprovision every bound claw on the ZeroClaw daemon,
/// delete the claw rows, and delete the team row. Best-effort — a failure to
/// tear down leaves the team intact and logs; a future sweep can retry.
async fn maybe_teardown_ephemeral_team(pool: &PgPool, id: Uuid) {
async fn maybe_teardown_ephemeral_team(pool: &PgPool, runtime: &cm_runtime::Runtime, id: Uuid) {
let teardown = match cm_db::repo::topology_runs::check_ephemeral_teardown(pool, id).await {
Ok(Some(t)) => t,
Ok(None) => return,
@@ -239,16 +239,20 @@ async fn maybe_teardown_ephemeral_team(pool: &PgPool, id: Uuid) {
// side fails we still delete our rows (the daemon can be swept for orphans
// by the fleet-reconcile timer). This is the trade cm-api owns everywhere:
// Postgres is authoritative, the daemon config is a cache.
if let Some(prov) = crate::runtime_provision::RuntimeProvisioner::from_env() {
for cid in &teardown.claw_ids {
if let Err(e) = prov.deprovision_claw(*cid).await {
eprintln!("topology_worker: deprovision_claw({cid}) failed: {e}");
}
}
}
//
// Goes through the shared reaper so an ephemeral team's claws also get
// their sandbox containers and `.brain` files removed — this path used to
// do the daemon + DB halves only, leaking a container per ephemeral run.
let provisioner = crate::runtime_provision::RuntimeProvisioner::from_env();
for cid in &teardown.claw_ids {
if let Err(e) = cm_db::repo::agents::hard_purge(pool, cm_domain::AgentId::from(*cid)).await
{
let report = crate::routes::claws::purge_agent(
pool,
runtime,
provisioner.as_ref(),
cm_domain::AgentId::from(*cid),
)
.await;
if let Err(e) = report.counts {
eprintln!("topology_worker: agents::hard_purge({cid}) failed: {e}");
}
}