feat(gc): reclaim the mission tree on the gateway
`cleanup_sweeper` prunes ROWS. Deleting a row has never deleted a directory, and `teardown_container` only runs while a mission still exists to tear down — so a mission removed by any path that skipped teardown left its tree behind permanently, on the smallest disk in the fleet (150 GB, shared with postgres and every checkout). 106 mission directories are sitting there now. Filesystem-first, deliberately: the DB is the PREDICATE, never the enumerator. Enumerating from the database is exactly how these became invisible — a directory whose row is gone is the one a row-driven sweep cannot see. Three reapers, one deletion path. Orphan mission dirs (no row, past a 2h grace), scratch trees (_bench/_gate/_verify/_merge past 6h — all four have leaked before), and _outputs past 90d, whose artifact rows are marked only AFTER the files are gone, because the other order claims artifacts are reaped while they are still on disk. The single removal path escalates: the server is uid 65532 and cannot delete what the per-mission daemon leaves as root, so PermissionDenied falls back to `root_copy::purge` and shouts if the tree survives even that. A GC that cannot collect is the thing being fixed, so failures are counted and reported, never swallowed. Guards worth naming: `_cargo` is a SHARED cache every mission writes to and lives under the same root, so an underscore-prefixed sibling treated as an orphan mission would delete it out from under running work and look like a slow cargo build. Only a well-formed mission id is ever a candidate — a directory whose name is not an id can have no row by construction, so without that gate every unrecognised directory looks orphaned. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
3c3d01c8d1
commit
1f6108f769
@@ -397,6 +397,10 @@ async fn run() -> Result<(), String> {
|
||||
// Expiry/retention sweep: expires stale auth/oauth rows and prunes old
|
||||
// journal/audit rows hourly so unbounded tables don't accumulate.
|
||||
cm_api::cleanup_sweeper::spawn(pool.clone(), std::time::Duration::from_secs(3600));
|
||||
// Its filesystem counterpart. `cleanup_sweeper` prunes ROWS, and deleting a
|
||||
// row has never deleted a directory — which is why the gateway, the smallest
|
||||
// disk in the fleet, accumulates mission trees that nothing reclaims.
|
||||
cm_api::mission_gc::spawn(pool.clone(), std::time::Duration::from_secs(3600));
|
||||
// Fleet backstop: a node whose heartbeats stop (without a clean channel
|
||||
// close) goes offline within ~28s even if its control channel hangs.
|
||||
cm_api::fleet::spawn_node_sweeper(pool.clone(), std::time::Duration::from_secs(8), 20);
|
||||
|
||||
Reference in New Issue
Block a user