fix(runtime): funnel every reap path through purge_agent; sweep node-placed orphans
Agent containers leaked two independent ways.
1. The four-step teardown (deprovision ZeroClaw -> reap_sandbox -> unlink
.brain/.onion -> hard_purge) was inlined at three call sites and two had
drifted. missions.rs::reap_mission_resources skipped reap_sandbox;
topology_worker::maybe_teardown_ephemeral_team skipped it and the brain
unlink; DELETE /api/claws/{id} (soft delete) released nothing at all, so an
offline claw that can never run again kept its container and bind mount
forever. All four now funnel through claws::purge_agent, with
release_claw_resources for the soft-delete case (containers gone, rows kept).
2. Both orphan reapers listed only the local driver, so a container placed on a
fleet node was invisible to the only backstop that could find it -- this is
what accumulated 144 tc-agent-* orphans on one node. NodeDriverProvider gains
node_ids() (backed by NodeHub::online_ids) and both reapers now sweep every
connected node. The remote sweep is TTL-only on purpose: the boot pass runs
with Duration::ZERO and would otherwise kill a container another instance is
mid-provision on.
Why it was invisible: agent_containers.agent_id is ON DELETE CASCADE, so
hard_purge took the registry row with the agent and left the container
permanently unreferenceable.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
a78f308eea
commit
c573480955
@@ -437,18 +437,19 @@ async fn reap_mission_resources(state: &AppState, mission_id: Uuid) {
|
||||
.unwrap_or_default()
|
||||
};
|
||||
|
||||
// 2. Reap each claw: ZeroClaw config → .brain files → all DB rows.
|
||||
// 2. Reap each claw: ZeroClaw config → sandbox container → .brain files →
|
||||
// all DB rows. Shared with the batch-delete reaper so this path cannot
|
||||
// drift back into skipping the container teardown.
|
||||
let provisioner = crate::runtime_provision::RuntimeProvisioner::from_env();
|
||||
for cid in &claw_ids {
|
||||
if let Some(p) = &provisioner {
|
||||
let _ = p.deprovision_claw(*cid).await;
|
||||
}
|
||||
let brain = crate::routes::claws::brain_dir();
|
||||
let _ = std::fs::remove_file(brain.join(format!("claw_{cid}.h5")));
|
||||
let _ = std::fs::remove_file(brain.join(format!("claw_{cid}.h5.onion")));
|
||||
if let Err(e) =
|
||||
cm_db::repo::agents::hard_purge(&state.pool, cm_domain::AgentId::from(*cid)).await
|
||||
{
|
||||
let report = crate::routes::claws::purge_agent(
|
||||
&state.pool,
|
||||
&state.runtime,
|
||||
provisioner.as_ref(),
|
||||
cm_domain::AgentId::from(*cid),
|
||||
)
|
||||
.await;
|
||||
if let Err(e) = report.counts {
|
||||
eprintln!("missions::delete: hard_purge claw {cid} failed (continuing): {e}");
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user