Files
clawmates/crates/cm-db/src/repo/routines.rs
T
Omar SobhandClaude Opus 5 2c7d619cf0
ci / gates (push) Failing after 6s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
fix(scheduler): a firing could be lost between rescheduling and dispatch
`tick` advanced `next_run_at` before dispatching the work, with nothing
recording that the occurrence was owed. A process that died between the two
dropped it silently.

The window is narrower than it first looks — `claim_due` sets `last_run_at`
but does not clear `next_run_at`, so a crash *before* `set_next_run` leaves
the routine due and it re-fires on the next tick. The loss is specifically
between the reschedule and the dispatch. That is tolerable for a message
routine and not tolerable for a scheduled mission, which is why this lands
before mission scheduling does.

`routine_fires` holds one row per (routine, occurrence), claimed before
dispatch and settled after:

- Fresh   — nobody has it; fire.
- Retry   — claimed, never settled: a crash mid-fire. Safe to fire again, as
            no completion was recorded and nothing downstream saw a result.
- Settled — already dispatched; advance the clock and do not run the work.
            This is what keeps a scheduled mission to one container across
            restarts.

A failed dispatch settles terminally rather than staying retryable. Retrying
a persistently failing action every tick is how a broken routine becomes a
denial-of-service against whatever it talks to; the error is kept on the row.

The claim uses `xmax = 0` to distinguish a real insert from a no-op update in
a single statement — `ON CONFLICT DO NOTHING` returns no row at all, so two
schedulers racing one occurrence could both read it as unclaimed.

Also: fan-out capped at 25 per tick with the remainder logged and deferred (a
clock jump or an accidental every-minute cron would otherwise dispatch every
missed occurrence at once — one container each for topology routines), and
`spawn` no longer discards tick errors, so a scheduler that has stopped firing
no longer looks identical to one with nothing to do.

The pre-existing exactly-once test still passes: the claim changes
recoverability, not firing semantics.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-01 18:50:11 -07:00

184 lines
5.7 KiB
Rust

use cm_domain::AgentId;
use sqlx::PgPool;
use time::OffsetDateTime;
use uuid::Uuid;
use crate::DbError;
/// A scheduled task an agent runs on a cron cadence (§7.6).
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct Routine {
pub id: Uuid,
pub agent_id: Uuid,
pub name: String,
pub schedule_cron: String,
/// `{"message": "..."}` — the prompt sent into the routine's session.
pub action: serde_json::Value,
pub status: String,
#[serde(with = "time::serde::rfc3339::option")]
pub next_run_at: Option<OffsetDateTime>,
#[serde(with = "time::serde::rfc3339::option")]
pub last_run_at: Option<OffsetDateTime>,
}
pub async fn create(
pool: &PgPool,
agent_id: AgentId,
name: &str,
schedule_cron: &str,
action: serde_json::Value,
next_run_at: OffsetDateTime,
) -> Result<Routine, DbError> {
let id = Uuid::now_v7();
sqlx::query!(
"INSERT INTO routines (id, agent_id, name, schedule_cron, action, next_run_at)
VALUES ($1, $2, $3, $4, $5, $6)",
id,
agent_id.as_uuid(),
name,
schedule_cron,
action,
next_run_at,
)
.execute(pool)
.await?;
Ok(Routine {
id,
agent_id: agent_id.as_uuid(),
name: name.to_owned(),
schedule_cron: schedule_cron.to_owned(),
action,
status: "active".into(),
next_run_at: Some(next_run_at),
last_run_at: None,
})
}
pub async fn list_by_agent(pool: &PgPool, agent_id: AgentId) -> Result<Vec<Routine>, DbError> {
let rows = sqlx::query_as!(
Routine,
r#"SELECT id, agent_id, name, schedule_cron, action, status,
next_run_at, last_run_at
FROM routines WHERE agent_id = $1 ORDER BY created_at"#,
agent_id.as_uuid(),
)
.fetch_all(pool)
.await?;
Ok(rows)
}
/// Claims every due routine atomically (SKIP LOCKED: one firing per
/// routine even with multiple scheduler replicas) and advances its clock.
/// `next_runs` computes the following occurrence per claimed routine.
pub async fn claim_due(pool: &PgPool, now: OffsetDateTime) -> Result<Vec<Routine>, DbError> {
let rows = sqlx::query_as!(
Routine,
r#"UPDATE routines SET last_run_at = $1
WHERE id IN (
SELECT id FROM routines
WHERE status = 'active' AND next_run_at IS NOT NULL
AND next_run_at <= $1
FOR UPDATE SKIP LOCKED
)
RETURNING id, agent_id, name, schedule_cron, action, status,
next_run_at, last_run_at"#,
now,
)
.fetch_all(pool)
.await?;
Ok(rows)
}
/// Schedules the next firing after a claim.
pub async fn set_next_run(
pool: &PgPool,
id: Uuid,
next_run_at: Option<OffsetDateTime>,
) -> Result<(), DbError> {
sqlx::query!(
"UPDATE routines SET next_run_at = $2 WHERE id = $1",
id,
next_run_at,
)
.execute(pool)
.await?;
Ok(())
}
/// What claiming an occurrence found.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum FireClaim {
/// Nobody has taken this occurrence. Fire it.
Fresh,
/// A previous attempt took it and never recorded an outcome — a crash
/// between claim and dispatch. Safe to fire again: no completion was ever
/// written, so nothing downstream saw a result.
Retry,
/// Already dispatched (or already failed). Do not fire; just advance the
/// clock. This is the branch that makes a scheduled mission cost one
/// container instead of one per restart.
Settled,
}
/// Take ownership of one occurrence before dispatching it.
///
/// `scheduled_at` is the occurrence's own timestamp — the `next_run_at` that
/// came due — not the wall clock at claim time. That is what makes the claim
/// idempotent across restarts: the same occurrence always maps to the same
/// row.
pub async fn claim_fire(
pool: &PgPool,
routine_id: Uuid,
scheduled_at: OffsetDateTime,
) -> Result<FireClaim, DbError> {
use sqlx::Row;
// Insert-or-look-at-what's-there in one statement, so two schedulers
// racing the same occurrence cannot both see "fresh".
let row = sqlx::query(
"INSERT INTO routine_fires (routine_id, scheduled_at)
VALUES ($1, $2)
ON CONFLICT (routine_id, scheduled_at) DO UPDATE
SET routine_id = routine_fires.routine_id
RETURNING status, (xmax = 0) AS inserted",
)
.bind(routine_id)
.bind(scheduled_at)
.fetch_one(pool)
.await?;
// `xmax = 0` distinguishes a genuine insert from a no-op update — the
// usual Postgres trick, and the reason for the otherwise pointless
// self-assignment in DO UPDATE (a bare DO NOTHING returns no row at all).
let inserted: bool = row.try_get("inserted").unwrap_or(false);
if inserted {
return Ok(FireClaim::Fresh);
}
let status: String = row.try_get("status").unwrap_or_default();
Ok(match status.as_str() {
"claimed" => FireClaim::Retry,
_ => FireClaim::Settled,
})
}
/// Record how a dispatched occurrence ended. Called after the work is handed
/// off, so a crash before this leaves the row `claimed` and retryable.
pub async fn complete_fire(
pool: &PgPool,
routine_id: Uuid,
scheduled_at: OffsetDateTime,
error: Option<&str>,
) -> Result<(), DbError> {
sqlx::query(
"UPDATE routine_fires
SET status = $3, completed_at = now(), error = $4
WHERE routine_id = $1 AND scheduled_at = $2",
)
.bind(routine_id)
.bind(scheduled_at)
.bind(if error.is_some() { "failed" } else { "fired" })
.bind(error)
.execute(pool)
.await?;
Ok(())
}