Reaping: sandbox orphan reaper + DB expiry/retention sweeper + volume-init retry
Closes the cleanup gaps found in review (latent today; bites under load/crashes).
Sandbox containers (cm-sandbox / cm-runtime):
- label every sandbox `clawmates.sandbox={agent|browser}` at create
- SandboxDriver::list_managed(kind) (Docker label filter + K8s label selector)
- SandboxManager::reconcile_orphans(ttl) + spawn_reaper: removes engine
containers no live handle owns (ZERO = all)
- boot reconciliation (every pre-existing sandbox is an orphan from a dead
process) + periodic reaper (5m interval / 10m TTL)
- SIGTERM graceful drain: serve().with_graceful_shutdown → shutdown() both
managers so a redeploy can't leak; destroy errors now logged not swallowed
DB expiry/retention (cm-db cleanup.rs + cm-api cleanup_sweeper, hourly):
- expire auth_sessions + oauth_states past expires_at (security)
- prune sent outbox(7d), run_events(14d), routine_runs(30d), terminal
topology_runs(90d), consumed execution_grants(7d)
Compose: volume-init restart "no" → on-failure:5 (retry instead of wedging boot).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
3511c3ca10
commit
148705769f
@@ -0,0 +1,90 @@
|
||||
//! Expiry + retention sweeps for tables that would otherwise grow unbounded.
|
||||
//! Security-relevant rows (expired auth sessions, stale oauth states) are
|
||||
//! deleted as soon as they're overdue; audit/journal tables get a generous
|
||||
//! retention window. Each fn returns the number of rows removed.
|
||||
//!
|
||||
//! This is the row-level analogue of the durable-run `requeue_stale` sweep and
|
||||
//! the sandbox reaper — driven on an interval by `cm_api::cleanup_sweeper`.
|
||||
|
||||
use sqlx::PgPool;
|
||||
|
||||
use crate::DbError;
|
||||
|
||||
/// Delete auth sessions past their `expires_at` (a revoked/expired bearer token
|
||||
/// must not linger). Security-relevant — no grace window.
|
||||
pub async fn expire_auth_sessions(pool: &PgPool) -> Result<u64, DbError> {
|
||||
let r = sqlx::query!("DELETE FROM auth_sessions WHERE expires_at < now()")
|
||||
.execute(pool)
|
||||
.await?;
|
||||
Ok(r.rows_affected())
|
||||
}
|
||||
|
||||
/// Delete OAuth CSRF states past their `expires_at` (abandoned flows).
|
||||
pub async fn expire_oauth_states(pool: &PgPool) -> Result<u64, DbError> {
|
||||
let r = sqlx::query!("DELETE FROM oauth_states WHERE expires_at < now()")
|
||||
.execute(pool)
|
||||
.await?;
|
||||
Ok(r.rows_affected())
|
||||
}
|
||||
|
||||
/// Prune successfully-sent outbox rows older than `days` (failed rows are kept
|
||||
/// for inspection).
|
||||
pub async fn prune_sent_outbox(pool: &PgPool, days: i32) -> Result<u64, DbError> {
|
||||
let r = sqlx::query!(
|
||||
"DELETE FROM outbox WHERE status = 'sent' AND created_at < now() - make_interval(days => $1)",
|
||||
days,
|
||||
)
|
||||
.execute(pool)
|
||||
.await?;
|
||||
Ok(r.rows_affected())
|
||||
}
|
||||
|
||||
/// Prune the SSE run-event journal older than `days` (beyond the live-replay
|
||||
/// window). Removes events for long-finished runs.
|
||||
pub async fn prune_run_events(pool: &PgPool, days: i32) -> Result<u64, DbError> {
|
||||
let r = sqlx::query!(
|
||||
"DELETE FROM run_events WHERE created_at < now() - make_interval(days => $1)",
|
||||
days,
|
||||
)
|
||||
.execute(pool)
|
||||
.await?;
|
||||
Ok(r.rows_affected())
|
||||
}
|
||||
|
||||
/// Prune routine-run history older than `days`.
|
||||
pub async fn prune_routine_runs(pool: &PgPool, days: i32) -> Result<u64, DbError> {
|
||||
let r = sqlx::query!(
|
||||
"DELETE FROM routine_runs WHERE started_at < now() - make_interval(days => $1)",
|
||||
days,
|
||||
)
|
||||
.execute(pool)
|
||||
.await?;
|
||||
Ok(r.rows_affected())
|
||||
}
|
||||
|
||||
/// Prune terminal topology runs older than `days` (user-visible run history —
|
||||
/// keep a generous window).
|
||||
pub async fn prune_topology_runs(pool: &PgPool, days: i32) -> Result<u64, DbError> {
|
||||
let r = sqlx::query!(
|
||||
"DELETE FROM topology_runs
|
||||
WHERE status IN ('completed', 'failed', 'cancelled')
|
||||
AND finished_at IS NOT NULL
|
||||
AND finished_at < now() - make_interval(days => $1)",
|
||||
days,
|
||||
)
|
||||
.execute(pool)
|
||||
.await?;
|
||||
Ok(r.rows_affected())
|
||||
}
|
||||
|
||||
/// Prune consumed single-use execution grants older than `days`.
|
||||
pub async fn prune_consumed_grants(pool: &PgPool, days: i32) -> Result<u64, DbError> {
|
||||
let r = sqlx::query!(
|
||||
"DELETE FROM execution_grants
|
||||
WHERE consumed = true AND consumed_at < now() - make_interval(days => $1)",
|
||||
days,
|
||||
)
|
||||
.execute(pool)
|
||||
.await?;
|
||||
Ok(r.rows_affected())
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
pub mod agents;
|
||||
pub mod audit;
|
||||
pub mod cleanup;
|
||||
pub mod companies;
|
||||
pub mod connections;
|
||||
pub mod credits;
|
||||
|
||||
Reference in New Issue
Block a user