Omar SobhandClaude Opus 5
f87853ecf9
The wizard has collected a cron since `0047_missions.sql` ("schedule JSONB
carries the trigger config (cron | one_shot | on_event)"), the frontend posts
`{kind:"cron", cron}`, and the API persists it faithfully. Nothing has ever read
it back: the only due-work enumerator in the codebase was `routines::claim_due`.
So every scheduled mission ever created sat in `draft` forever while the UI
reported it was on a schedule.
Proven before fixing, on the shipped build: a mission with `* * * * *` sat in
`draft` for 4m34s and started ZERO topology runs. After this change the same
mission launched on its next occurrence and recorded one `fired` row.
Two pieces were missing, and they are the two `routines` already had:
- `missions.next_run_at` — schedule STATE. `schedule` is user intent and stays
untouched; without somewhere to record which occurrence is owed there is
nothing to put a `<= now()` predicate on, which is why no enumerator could
be written against the JSONB alone.
- `mission_fires` — one row per (mission, occurrence). 0063_routine_fires.sql
called this exact case: "For a scheduled *mission* it costs a container, a
repo checkout, and real money — which is why this lands before mission
scheduling does."
`mission_schedule.rs` deliberately mirrors `cm-scheduler`'s shape rather than
inventing a second one: atomic `FOR UPDATE SKIP LOCKED` claim, reschedule
BEFORE dispatch so a failing launch cannot stall the clock, claim the slot
before launching so a crash mid-launch is retried rather than dropped, and a
fan-out cap. The cap is 5, not the scheduler's 25, because a mission firing is
a container and a checkout where a routine firing may be one turn.
The claim skips `status = 'running'`: a daily cron on a mission that takes
longer than a day must skip the occurrence, not stack a second crew on the same
workspace. Launch goes through `mission_orchestrator::on_launch` +
`missions::set_status`, the same path as the draft→running transition, so one
code path mints a crew. An unattended launch acts as the workspace owner
(`users::owner_of_workspace`) since missions carry no creator column; a
workspace without one settles the occurrence `failed` with the reason rather
than dropping it silently.
Backfill blast radius was MEASURED, not assumed: prod has zero missions with a
cron, this workstation had exactly one — the control created to prove the bug.
Co-Authored-By: Claude Opus 5 <[email protected]>
77 lines
4.1 KiB
SQL
77 lines
4.1 KiB
SQL
-- Make `missions.schedule` mean something.
|
|
--
|
|
-- The wizard has collected a cron since the schedule column landed
|
|
-- (0047_missions.sql: "schedule JSONB carries the trigger config (cron |
|
|
-- one_shot | on_event)"), the frontend posts `{kind:"cron", cron}`, and the API
|
|
-- persists it faithfully. Nothing has ever read it back. The only due-work
|
|
-- enumerator in the codebase is `routines::claim_due`, so every scheduled
|
|
-- mission ever created has sat in `draft` forever while the UI reported that it
|
|
-- was on a schedule. Measured before writing this: a mission with
|
|
-- `* * * * *` did not move for four minutes and started no runs.
|
|
--
|
|
-- Two pieces are missing, and they are the two `routines` already has:
|
|
--
|
|
-- 1. `next_run_at` — schedule STATE. `schedule` is the user's intent and must
|
|
-- stay untouched; a scheduler needs somewhere to record which occurrence is
|
|
-- owed next. Without it there is nothing to put a `<= now()` predicate on,
|
|
-- which is why no enumerator could be written against the JSONB alone.
|
|
--
|
|
-- 2. `mission_fires` — one row per (mission, occurrence). The scheduler
|
|
-- advances the clock BEFORE dispatching, so that a failing launch cannot
|
|
-- stall the schedule; the claim row is what remembers the occurrence was
|
|
-- owed, so a crash between the two is retried rather than silently dropped.
|
|
-- 0063_routine_fires.sql called this exact case: "For a scheduled *mission*
|
|
-- it costs a container, a repo checkout, and real money — which is why this
|
|
-- lands before mission scheduling does."
|
|
ALTER TABLE missions
|
|
ADD COLUMN IF NOT EXISTS next_run_at TIMESTAMPTZ;
|
|
|
|
COMMENT ON COLUMN missions.next_run_at IS
|
|
'When this mission is next due to launch. NULL for one_shot/on_event or a finished schedule. Derived from schedule->>cron; the schedule column stays the user intent.';
|
|
|
|
-- Partial: only scheduled missions carry a value, and the sweep asks exactly
|
|
-- "which are due now".
|
|
CREATE INDEX IF NOT EXISTS missions_next_run_at_idx
|
|
ON missions (next_run_at)
|
|
WHERE next_run_at IS NOT NULL;
|
|
|
|
CREATE TABLE IF NOT EXISTS mission_fires (
|
|
mission_id UUID NOT NULL REFERENCES missions (id) ON DELETE CASCADE,
|
|
-- The occurrence this row accounts for — the `next_run_at` that came due,
|
|
-- NOT the claim time. That is what makes a retry idempotent: re-claiming
|
|
-- the same slot finds this row instead of launching a second container.
|
|
scheduled_at TIMESTAMPTZ NOT NULL,
|
|
claimed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
completed_at TIMESTAMPTZ,
|
|
-- 'claimed' — taken, launch not yet known to have finished. Stuck here is a
|
|
-- crash mid-launch and is safe to retry.
|
|
-- 'fired' — the mission was launched; never launch this slot again.
|
|
-- 'failed' — the launch errored. Terminal: the clock has already moved on,
|
|
-- and `detail` is the only record of why.
|
|
status TEXT NOT NULL DEFAULT 'claimed',
|
|
detail TEXT,
|
|
PRIMARY KEY (mission_id, scheduled_at)
|
|
);
|
|
|
|
COMMENT ON TABLE mission_fires IS
|
|
'One row per (mission, scheduled occurrence). Makes a scheduled launch idempotent across replicas and restarts.';
|
|
|
|
-- Backfill: every mission already carrying a cron has been silently dead. Give
|
|
-- it a due time so it starts firing rather than needing to be recreated.
|
|
-- `now()` rather than the true next occurrence — Postgres has no cron parser,
|
|
-- and the scheduler recomputes from the cron on its first claim anyway. The
|
|
-- effect is one catch-up launch per already-scheduled mission, which is the
|
|
-- honest outcome: they were supposed to have been running all along.
|
|
--
|
|
-- Blast radius MEASURED before writing this, because a backfill that launches
|
|
-- containers is not something to guess at: prod (gw-04) has **zero** missions
|
|
-- with `schedule->>'kind' = 'cron'`, and this workstation has exactly one — the
|
|
-- negative-control mission created to prove the bug. So this UPDATE starts
|
|
-- nothing unexpected today; the guard exists for deployments that do have them.
|
|
UPDATE missions
|
|
SET next_run_at = now()
|
|
WHERE schedule ->> 'kind' = 'cron'
|
|
AND schedule ->> 'cron' IS NOT NULL
|
|
AND next_run_at IS NULL
|
|
AND status IN ('draft', 'completed', 'failed');
|