Omar SobhandClaude Opus 5
2c7d619cf0
`tick` advanced `next_run_at` before dispatching the work, with nothing
recording that the occurrence was owed. A process that died between the two
dropped it silently.
The window is narrower than it first looks — `claim_due` sets `last_run_at`
but does not clear `next_run_at`, so a crash *before* `set_next_run` leaves
the routine due and it re-fires on the next tick. The loss is specifically
between the reschedule and the dispatch. That is tolerable for a message
routine and not tolerable for a scheduled mission, which is why this lands
before mission scheduling does.
`routine_fires` holds one row per (routine, occurrence), claimed before
dispatch and settled after:
- Fresh — nobody has it; fire.
- Retry — claimed, never settled: a crash mid-fire. Safe to fire again, as
no completion was recorded and nothing downstream saw a result.
- Settled — already dispatched; advance the clock and do not run the work.
This is what keeps a scheduled mission to one container across
restarts.
A failed dispatch settles terminally rather than staying retryable. Retrying
a persistently failing action every tick is how a broken routine becomes a
denial-of-service against whatever it talks to; the error is kept on the row.
The claim uses `xmax = 0` to distinguish a real insert from a no-op update in
a single statement — `ON CONFLICT DO NOTHING` returns no row at all, so two
schedulers racing one occurrence could both read it as unclaimed.
Also: fan-out capped at 25 per tick with the remainder logged and deferred (a
clock jump or an accidental every-minute cron would otherwise dispatch every
missed occurrence at once — one container each for topology routines), and
`spawn` no longer discards tick errors, so a scheduler that has stopped firing
no longer looks identical to one with nothing to do.
The pre-existing exactly-once test still passes: the claim changes
recoverability, not firing semantics.
Co-Authored-By: Claude Opus 5 <[email protected]>
37 lines
2.0 KiB
SQL
37 lines
2.0 KiB
SQL
-- One row per (routine, scheduled occurrence), so a firing is idempotent.
|
|
--
|
|
-- The scheduler advanced `next_run_at` *before* dispatching the work
|
|
-- (`cm-scheduler/src/lib.rs`, "Reschedule first: a firing failure must not
|
|
-- stall the clock"). That trade is defensible on its own terms, but it has no
|
|
-- record of the attempt: a crash between the reschedule and the dispatch drops
|
|
-- the occurrence with nothing anywhere to say it was owed. For a message
|
|
-- routine that costs a lost reply. For a scheduled *mission* it costs a
|
|
-- container, a repo checkout, and real money — which is why this lands before
|
|
-- mission scheduling does.
|
|
--
|
|
-- `scheduled_at` is the occurrence's own timestamp, not the claim time, so the
|
|
-- primary key is what makes a retry idempotent: re-claiming the same slot
|
|
-- finds the existing row instead of firing twice.
|
|
CREATE TABLE routine_fires (
|
|
routine_id UUID NOT NULL REFERENCES routines (id) ON DELETE CASCADE,
|
|
-- The occurrence this row accounts for (the `next_run_at` that came due).
|
|
scheduled_at TIMESTAMPTZ NOT NULL,
|
|
claimed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
completed_at TIMESTAMPTZ,
|
|
-- `claimed` — taken, dispatch not yet known to have finished. A row stuck
|
|
-- here is a crash mid-fire and is safe to retry.
|
|
-- `fired` — dispatch completed; never fire this slot again.
|
|
-- `failed` — dispatch returned an error. Terminal: the clock has already
|
|
-- moved on, and silently retrying a failing action every tick
|
|
-- is how a broken routine becomes a denial-of-service.
|
|
status TEXT NOT NULL DEFAULT 'claimed'
|
|
CHECK (status IN ('claimed', 'fired', 'failed')),
|
|
error TEXT,
|
|
PRIMARY KEY (routine_id, scheduled_at)
|
|
);
|
|
|
|
-- The reaper's query: rows still `claimed` past a grace period are crashes.
|
|
CREATE INDEX routine_fires_stuck_idx
|
|
ON routine_fires (status, claimed_at)
|
|
WHERE status = 'claimed';
|