fix(runtime): funnel every reap path through purge_agent; sweep node-placed orphans

Agent containers leaked two independent ways.

1. The four-step teardown (deprovision ZeroClaw -> reap_sandbox -> unlink
   .brain/.onion -> hard_purge) was inlined at three call sites and two had
   drifted. missions.rs::reap_mission_resources skipped reap_sandbox;
   topology_worker::maybe_teardown_ephemeral_team skipped it and the brain
   unlink; DELETE /api/claws/{id} (soft delete) released nothing at all, so an
   offline claw that can never run again kept its container and bind mount
   forever. All four now funnel through claws::purge_agent, with
   release_claw_resources for the soft-delete case (containers gone, rows kept).

2. Both orphan reapers listed only the local driver, so a container placed on a
   fleet node was invisible to the only backstop that could find it -- this is
   what accumulated 144 tc-agent-* orphans on one node. NodeDriverProvider gains
   node_ids() (backed by NodeHub::online_ids) and both reapers now sweep every
   connected node. The remote sweep is TTL-only on purpose: the boot pass runs
   with Duration::ZERO and would otherwise kill a container another instance is
   mid-provision on.

Why it was invisible: agent_containers.agent_id is ON DELETE CASCADE, so
hard_purge took the registry row with the agent and left the container
permanently unreferenceable.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Omar Sobh
2026-07-30 10:41:22 -07:00
co-authored by Claude Opus 5
parent a78f308eea
commit c573480955
6 changed files with 213 additions and 37 deletions
+11 -10
View File
@@ -437,18 +437,19 @@ async fn reap_mission_resources(state: &AppState, mission_id: Uuid) {
.unwrap_or_default()
};
// 2. Reap each claw: ZeroClaw config → .brain files → all DB rows.
// 2. Reap each claw: ZeroClaw config → sandbox container → .brain files →
// all DB rows. Shared with the batch-delete reaper so this path cannot
// drift back into skipping the container teardown.
let provisioner = crate::runtime_provision::RuntimeProvisioner::from_env();
for cid in &claw_ids {
if let Some(p) = &provisioner {
let _ = p.deprovision_claw(*cid).await;
}
let brain = crate::routes::claws::brain_dir();
let _ = std::fs::remove_file(brain.join(format!("claw_{cid}.h5")));
let _ = std::fs::remove_file(brain.join(format!("claw_{cid}.h5.onion")));
if let Err(e) =
cm_db::repo::agents::hard_purge(&state.pool, cm_domain::AgentId::from(*cid)).await
{
let report = crate::routes::claws::purge_agent(
&state.pool,
&state.runtime,
provisioner.as_ref(),
cm_domain::AgentId::from(*cid),
)
.await;
if let Err(e) = report.counts {
eprintln!("missions::delete: hard_purge claw {cid} failed (continuing): {e}");
}
}