Omar Sobh
d84d17207f
Phase 1b: wires the capacity model from 3a2d76a into the launch path, and turns
the existing pending-phase loop into the queue.
Placement moves from mission launch to PHASE launch. A node chosen at launch is
chosen once, minutes before the first VM boots and hours before the last — and
re-placing between phases is free, because mission state lives on the gateway
checkout and every VM is inject -> run -> collect -> destroy. Pinning early
bought nothing and cost the ability to react to a node filling or draining
mid-mission. One call site serves both the solo and composed paths so they cannot
disagree; the composed worker reads `missions.target_node_id`, which placement
writes before dispatch.
QUEUEING, with no new machinery: a phase with no admissible node keeps its
`pending` status and creates no `topology_runs` row. `start_pending_phases`
retries every 10s — that loop already was a queue; nothing downstream ever sees a
run that did not happen.
The risk that creates is the one this codebase keeps paying for: a phase waiting
for capacity looks exactly like a phase nothing is working on. So the wait is
RECORDED, not merely logged — migration 0073 adds `capacity_blocked_since` and
`capacity_note`, stamped once and preserved across retries so the wait is
measured from the first refusal. It is bounded at two full turns: a fleet that
frees will free within one, and a phase that waited two hours must say so rather
than sit pending forever looking like a bug.
Mission launch still fails when NO node could ever run the backend — that is not
transient, waiting will not fix it, and `microvm-negctl` asserts such a mission
stays `draft`. Capacity refusals are transient and queue; capability refusals are
not and fail. The two are separate variants precisely so they cannot be confused.
257 lib tests, 20 binaries.
16 lines
886 B
SQL
16 lines
886 B
SQL
-- Why a phase is waiting, so a queue never looks like a stall.
|
|
--
|
|
-- Placement moves from mission launch to PHASE launch, which makes the existing
|
|
-- `start_pending_phases` loop the queue: a phase with no admissible node simply
|
|
-- stays `pending` and is retried every 10s. That is the whole queue mechanism —
|
|
-- no new status, no sweeper, no migration for a `queued` state.
|
|
--
|
|
-- The risk it creates is the one this codebase keeps paying for: a phase waiting
|
|
-- for capacity is indistinguishable from a phase nothing is working on. These two
|
|
-- columns are what make the difference visible. `capacity_blocked_since` also
|
|
-- bounds the wait, so a fleet that never frees fails the phase with a reason
|
|
-- instead of holding it forever.
|
|
ALTER TABLE mission_phases
|
|
ADD COLUMN IF NOT EXISTS capacity_blocked_since timestamptz,
|
|
ADD COLUMN IF NOT EXISTS capacity_note text;
|