Reaping: sandbox orphan reaper + DB expiry/retention sweeper + volume-init retry
ci / gates (push) Has been cancelled
ci / rust (push) Has been cancelled
ci / sandbox-k8s (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / e2e (push) Has been cancelled

Closes the cleanup gaps found in review (latent today; bites under load/crashes).

Sandbox containers (cm-sandbox / cm-runtime):
- label every sandbox `clawmates.sandbox={agent|browser}` at create
- SandboxDriver::list_managed(kind) (Docker label filter + K8s label selector)
- SandboxManager::reconcile_orphans(ttl) + spawn_reaper: removes engine
  containers no live handle owns (ZERO = all)
- boot reconciliation (every pre-existing sandbox is an orphan from a dead
  process) + periodic reaper (5m interval / 10m TTL)
- SIGTERM graceful drain: serve().with_graceful_shutdown → shutdown() both
  managers so a redeploy can't leak; destroy errors now logged not swallowed

DB expiry/retention (cm-db cleanup.rs + cm-api cleanup_sweeper, hourly):
- expire auth_sessions + oauth_states past expires_at (security)
- prune sent outbox(7d), run_events(14d), routine_runs(30d), terminal
  topology_runs(90d), consumed execution_grants(7d)

Compose: volume-init restart "no" → on-failure:5 (retry instead of wedging boot).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Omar Sobh
2026-06-19 04:55:27 -07:00
co-authored by Claude Opus 4.8
parent 3511c3ca10
commit 148705769f
21 changed files with 473 additions and 26 deletions
+90
View File
@@ -0,0 +1,90 @@
//! Expiry + retention sweeps for tables that would otherwise grow unbounded.
//! Security-relevant rows (expired auth sessions, stale oauth states) are
//! deleted as soon as they're overdue; audit/journal tables get a generous
//! retention window. Each fn returns the number of rows removed.
//!
//! This is the row-level analogue of the durable-run `requeue_stale` sweep and
//! the sandbox reaper — driven on an interval by `cm_api::cleanup_sweeper`.
use sqlx::PgPool;
use crate::DbError;
/// Delete auth sessions past their `expires_at` (a revoked/expired bearer token
/// must not linger). Security-relevant — no grace window.
pub async fn expire_auth_sessions(pool: &PgPool) -> Result<u64, DbError> {
let r = sqlx::query!("DELETE FROM auth_sessions WHERE expires_at < now()")
.execute(pool)
.await?;
Ok(r.rows_affected())
}
/// Delete OAuth CSRF states past their `expires_at` (abandoned flows).
pub async fn expire_oauth_states(pool: &PgPool) -> Result<u64, DbError> {
let r = sqlx::query!("DELETE FROM oauth_states WHERE expires_at < now()")
.execute(pool)
.await?;
Ok(r.rows_affected())
}
/// Prune successfully-sent outbox rows older than `days` (failed rows are kept
/// for inspection).
pub async fn prune_sent_outbox(pool: &PgPool, days: i32) -> Result<u64, DbError> {
let r = sqlx::query!(
"DELETE FROM outbox WHERE status = 'sent' AND created_at < now() - make_interval(days => $1)",
days,
)
.execute(pool)
.await?;
Ok(r.rows_affected())
}
/// Prune the SSE run-event journal older than `days` (beyond the live-replay
/// window). Removes events for long-finished runs.
pub async fn prune_run_events(pool: &PgPool, days: i32) -> Result<u64, DbError> {
let r = sqlx::query!(
"DELETE FROM run_events WHERE created_at < now() - make_interval(days => $1)",
days,
)
.execute(pool)
.await?;
Ok(r.rows_affected())
}
/// Prune routine-run history older than `days`.
pub async fn prune_routine_runs(pool: &PgPool, days: i32) -> Result<u64, DbError> {
let r = sqlx::query!(
"DELETE FROM routine_runs WHERE started_at < now() - make_interval(days => $1)",
days,
)
.execute(pool)
.await?;
Ok(r.rows_affected())
}
/// Prune terminal topology runs older than `days` (user-visible run history —
/// keep a generous window).
pub async fn prune_topology_runs(pool: &PgPool, days: i32) -> Result<u64, DbError> {
let r = sqlx::query!(
"DELETE FROM topology_runs
WHERE status IN ('completed', 'failed', 'cancelled')
AND finished_at IS NOT NULL
AND finished_at < now() - make_interval(days => $1)",
days,
)
.execute(pool)
.await?;
Ok(r.rows_affected())
}
/// Prune consumed single-use execution grants older than `days`.
pub async fn prune_consumed_grants(pool: &PgPool, days: i32) -> Result<u64, DbError> {
let r = sqlx::query!(
"DELETE FROM execution_grants
WHERE consumed = true AND consumed_at < now() - make_interval(days => $1)",
days,
)
.execute(pool)
.await?;
Ok(r.rows_affected())
}