fix(runtime): funnel every reap path through purge_agent; sweep node-placed orphans

Agent containers leaked two independent ways.

1. The four-step teardown (deprovision ZeroClaw -> reap_sandbox -> unlink
   .brain/.onion -> hard_purge) was inlined at three call sites and two had
   drifted. missions.rs::reap_mission_resources skipped reap_sandbox;
   topology_worker::maybe_teardown_ephemeral_team skipped it and the brain
   unlink; DELETE /api/claws/{id} (soft delete) released nothing at all, so an
   offline claw that can never run again kept its container and bind mount
   forever. All four now funnel through claws::purge_agent, with
   release_claw_resources for the soft-delete case (containers gone, rows kept).

2. Both orphan reapers listed only the local driver, so a container placed on a
   fleet node was invisible to the only backstop that could find it -- this is
   what accumulated 144 tc-agent-* orphans on one node. NodeDriverProvider gains
   node_ids() (backed by NodeHub::online_ids) and both reapers now sweep every
   connected node. The remote sweep is TTL-only on purpose: the boot pass runs
   with Duration::ZERO and would otherwise kill a container another instance is
   mid-provision on.

Why it was invisible: agent_containers.agent_id is ON DELETE CASCADE, so
hard_purge took the registry row with the agent and left the container
permanently unreferenceable.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Omar Sobh
2026-07-30 10:41:22 -07:00
co-authored by Claude Opus 5
parent a78f308eea
commit c573480955
6 changed files with 213 additions and 37 deletions
+45
View File
@@ -404,6 +404,51 @@ impl TerminalManager {
Err(e) => eprintln!("terminal reaper: failed to remove {}: {e}", m.id),
}
}
// An agent placed on a fleet node runs its terminal there too, so sweep
// each connected node as well — otherwise a node-placed terminal whose
// registry row is gone can never be found again. TTL-only for the same
// reason as the sandbox reaper: the boot pass uses ZERO and must not
// touch containers on a shared node.
if min > 0 {
for node_id in self
.node_provider
.as_ref()
.map(|p| p.node_ids())
.unwrap_or_default()
{
if node_id == self.node_id {
continue;
}
let Some(driver) = self.node_provider.as_ref().and_then(|p| p.driver(&node_id))
else {
continue;
};
let remote = match driver.list_managed(SandboxKind::Terminal.label()).await {
Ok(m) => m,
Err(e) => {
eprintln!("terminal reaper: list on node {node_id} failed: {e}");
continue;
}
};
for m in remote {
if tracked.contains(&m.id) || now_unix - m.created_unix < min {
continue;
}
let handle = SandboxHandle {
id: m.id.clone(),
name: m.id.clone(),
};
match driver.destroy(&handle).await {
Ok(()) => reaped += 1,
Err(e) => eprintln!(
"terminal reaper: failed to remove {} on node {node_id}: {e}",
m.id
),
}
}
}
}
reaped
}