fix(runtime): funnel every reap path through purge_agent; sweep node-placed orphans
Agent containers leaked two independent ways.
1. The four-step teardown (deprovision ZeroClaw -> reap_sandbox -> unlink
.brain/.onion -> hard_purge) was inlined at three call sites and two had
drifted. missions.rs::reap_mission_resources skipped reap_sandbox;
topology_worker::maybe_teardown_ephemeral_team skipped it and the brain
unlink; DELETE /api/claws/{id} (soft delete) released nothing at all, so an
offline claw that can never run again kept its container and bind mount
forever. All four now funnel through claws::purge_agent, with
release_claw_resources for the soft-delete case (containers gone, rows kept).
2. Both orphan reapers listed only the local driver, so a container placed on a
fleet node was invisible to the only backstop that could find it -- this is
what accumulated 144 tc-agent-* orphans on one node. NodeDriverProvider gains
node_ids() (backed by NodeHub::online_ids) and both reapers now sweep every
connected node. The remote sweep is TTL-only on purpose: the boot pass runs
with Duration::ZERO and would otherwise kill a container another instance is
mid-provision on.
Why it was invisible: agent_containers.agent_id is ON DELETE CASCADE, so
hard_purge took the registry row with the agent and left the container
permanently unreferenceable.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
a78f308eea
commit
c573480955
@@ -404,6 +404,51 @@ impl TerminalManager {
|
||||
Err(e) => eprintln!("terminal reaper: failed to remove {}: {e}", m.id),
|
||||
}
|
||||
}
|
||||
|
||||
// An agent placed on a fleet node runs its terminal there too, so sweep
|
||||
// each connected node as well — otherwise a node-placed terminal whose
|
||||
// registry row is gone can never be found again. TTL-only for the same
|
||||
// reason as the sandbox reaper: the boot pass uses ZERO and must not
|
||||
// touch containers on a shared node.
|
||||
if min > 0 {
|
||||
for node_id in self
|
||||
.node_provider
|
||||
.as_ref()
|
||||
.map(|p| p.node_ids())
|
||||
.unwrap_or_default()
|
||||
{
|
||||
if node_id == self.node_id {
|
||||
continue;
|
||||
}
|
||||
let Some(driver) = self.node_provider.as_ref().and_then(|p| p.driver(&node_id))
|
||||
else {
|
||||
continue;
|
||||
};
|
||||
let remote = match driver.list_managed(SandboxKind::Terminal.label()).await {
|
||||
Ok(m) => m,
|
||||
Err(e) => {
|
||||
eprintln!("terminal reaper: list on node {node_id} failed: {e}");
|
||||
continue;
|
||||
}
|
||||
};
|
||||
for m in remote {
|
||||
if tracked.contains(&m.id) || now_unix - m.created_unix < min {
|
||||
continue;
|
||||
}
|
||||
let handle = SandboxHandle {
|
||||
id: m.id.clone(),
|
||||
name: m.id.clone(),
|
||||
};
|
||||
match driver.destroy(&handle).await {
|
||||
Ok(()) => reaped += 1,
|
||||
Err(e) => eprintln!(
|
||||
"terminal reaper: failed to remove {} on node {node_id}: {e}",
|
||||
m.id
|
||||
),
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
reaped
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user