feat(gc): reclaim the mission tree on the gateway

`cleanup_sweeper` prunes ROWS. Deleting a row has never deleted a directory,
and `teardown_container` only runs while a mission still exists to tear down —
so a mission removed by any path that skipped teardown left its tree behind
permanently, on the smallest disk in the fleet (150 GB, shared with postgres and
every checkout). 106 mission directories are sitting there now.

Filesystem-first, deliberately: the DB is the PREDICATE, never the enumerator.
Enumerating from the database is exactly how these became invisible — a
directory whose row is gone is the one a row-driven sweep cannot see.

Three reapers, one deletion path. Orphan mission dirs (no row, past a 2h grace),
scratch trees (_bench/_gate/_verify/_merge past 6h — all four have leaked
before), and _outputs past 90d, whose artifact rows are marked only AFTER the
files are gone, because the other order claims artifacts are reaped while they
are still on disk.

The single removal path escalates: the server is uid 65532 and cannot delete
what the per-mission daemon leaves as root, so PermissionDenied falls back to
`root_copy::purge` and shouts if the tree survives even that. A GC that cannot
collect is the thing being fixed, so failures are counted and reported, never
swallowed.

Guards worth naming: `_cargo` is a SHARED cache every mission writes to and
lives under the same root, so an underscore-prefixed sibling treated as an
orphan mission would delete it out from under running work and look like a slow
cargo build. Only a well-formed mission id is ever a candidate — a directory
whose name is not an id can have no row by construction, so without that gate
every unrecognised directory looks orphaned.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-09 15:02:25 -07:00
co-authored by Claude Opus 5
parent 3c3d01c8d1
commit 1f6108f769
3 changed files with 383 additions and 0 deletions
+4
View File
@@ -397,6 +397,10 @@ async fn run() -> Result<(), String> {
// Expiry/retention sweep: expires stale auth/oauth rows and prunes old
// journal/audit rows hourly so unbounded tables don't accumulate.
cm_api::cleanup_sweeper::spawn(pool.clone(), std::time::Duration::from_secs(3600));
// Its filesystem counterpart. `cleanup_sweeper` prunes ROWS, and deleting a
// row has never deleted a directory — which is why the gateway, the smallest
// disk in the fleet, accumulates mission trees that nothing reclaims.
cm_api::mission_gc::spawn(pool.clone(), std::time::Duration::from_secs(3600));
// Fleet backstop: a node whose heartbeats stop (without a clean channel
// close) goes offline within ~28s even if its control channel hangs.
cm_api::fleet::spawn_node_sweeper(pool.clone(), std::time::Duration::from_secs(8), 20);