The full harness found it — 12 of 13 scenarios green, `roster` red:
roster: the planner sized this mission at 2 member(s) PASS
roster: the approved roster is on the mission (2 nodes, composed) PASS
roster: this run added 1 line(s) for a 2-member roster FAIL
topology_runs.error: turn executor failed: node n1 in a microVM:
vm_create failed: no rootfs for backend "canary-claude" on this node
The roster proposed `verifier@canary-claude`. Placement asked
`online_for_backend` about the MISSION's backend — `claude` — and architect
answered, holding `claude` and `local-ornith`. The graph's first node ran and
delivered, the second could not boot, and the mission finished half-done. The
question placement asked was true and insufficient.
A composed graph runs on ONE node, so that node needs every image its nodes ask
for. `required_backends` collects the mission's plus each
`config.roster.nodes[].attrs.backend`, and `online_for_backends` passes the
whole set to the same jsonb `@>` — containment already means "contains ALL of
these", so the query shape did not have to change, only what it was asked.
This is the failure mode the roster feature creates by existing: its entire
purpose is putting a verifier on a different provider, which is exactly what
makes one node insufficient. Nothing before the full suite had a reason to
exercise it — the composed scenario uses one backend for all five nodes.
`NoCapableNode` now names the set and says why one node must hold all of them.
Co-Authored-By: Claude Opus 5 <[email protected]>
drain-midmission passed 3/3 twice, but the "re-placing this phase" line the
last commit added never appeared in the log. It cannot: `online_for_backend`
filters on `status = 'online'`, so a draining node is not a candidate, never
reaches `unfit`, and the pin simply falls through to ranking — on the old code
as well as the new.
So the scenario passes either way and proves the affinity decision, not the
`TargetUnfit` bug. The path that genuinely used to fail a phase is "the
previous phase's node has since FILLED UP": that puts it in `unfit`, which
returned a non-transient error, which never reached the queue. That is what the
unit test now says, in place of a claim about draining the harness does not
support.
The accidental mission-to-node affinity was real and unconditional either way.
Co-Authored-By: Claude Opus 5 <[email protected]>
`drain-midmission` found this. `choose` treated its `want` argument as a hard
requirement, and the only caller passes `missions.target_node_id` — which is
not an operator's choice, only where the PREVIOUS phase happened to run. Two
consequences, both wrong:
- A node drained or filled between phases produced `TargetUnfit`, which
`is_transient()` says false to, so `phase_runner` FAILED the phase rather
than queueing or moving it. The queue silently did not apply to the second
phase of any mission.
- While the node stayed fit, every later phase went straight back to it
regardless of ranking — accidental mission-to-node affinity, which this
module's own header says must not exist.
Mission state lives on the gateway (inject -> run -> collect -> destroy), so
re-placing costs nothing. The pin is now advisory: preferred while it fits,
and when it does not, the reason is logged and ranking proceeds. `TargetUnfit`
is deleted rather than left unconstructed, so it cannot come back as a
non-transient failure by accident.
The scenario had its own race: it waited for phase 0 to COMPLETE before
draining, but warm phases finish in ~80s against a 10s placement sweep, so
phase 1 was often already placed — and the run then blamed the platform for
running on a node that was not yet drained. It now drains while phase 0 is
still running, which does not disturb a live VM and is the more faithful test.
Co-Authored-By: Claude Opus 5 <[email protected]>
The capacity harness scenario, on its first full run, caught what it was
written to catch:
capacity: architect peaked at 6 of 6 slot(s)
FAIL capacity: 'morpheus' peaked at 3 concurrent VM(s) with only 2 slot(s)
capacity: tank peaked at 6 of 6 slot(s)
PASS capacity: the over-capacity missions QUEUED
PASS capacity: all 16 queued/placed missions completed
`capacity_of` inferred the host's own footprint by subtracting the VMs' FULL
8 GiB claim from observed usage — which assumes they have already consumed it.
A VM booted seconds ago holds about an eighth. On morpheus (31757 MiB total,
4314 MiB idle, 2 slots) with 2 young VMs at ~6314 MiB observed, the inference
6314 - 16384 goes negative, clamps to the 2048 floor, and invents 2266 MiB —
exactly enough for a third VM on a two-slot node.
The footprint is only honestly MEASURABLE when nothing is committed, so
remember it then: `nodes.mem_baseline_mib`, sampled by `survey` whenever it
observes an idle node with fresh health. When VMs are committed, take the
LARGER of the remembered reading and the old inference — a node that was once
idle at 4 GiB and is now running a 20 GiB build must not be scored as idle,
which would be the same over-commit arrived at from the other direction. Both
directions have a test; the second is the one that would otherwise rot.
Raising HOST_BASELINE_FLOOR_MIB would have made this one node's numbers pass
and drifted the moment the fleet changed shape.
Co-Authored-By: Claude Opus 5 <[email protected]>
Phase 1b: wires the capacity model from 3a2d76a into the launch path, and turns
the existing pending-phase loop into the queue.
Placement moves from mission launch to PHASE launch. A node chosen at launch is
chosen once, minutes before the first VM boots and hours before the last — and
re-placing between phases is free, because mission state lives on the gateway
checkout and every VM is inject -> run -> collect -> destroy. Pinning early
bought nothing and cost the ability to react to a node filling or draining
mid-mission. One call site serves both the solo and composed paths so they cannot
disagree; the composed worker reads `missions.target_node_id`, which placement
writes before dispatch.
QUEUEING, with no new machinery: a phase with no admissible node keeps its
`pending` status and creates no `topology_runs` row. `start_pending_phases`
retries every 10s — that loop already was a queue; nothing downstream ever sees a
run that did not happen.
The risk that creates is the one this codebase keeps paying for: a phase waiting
for capacity looks exactly like a phase nothing is working on. So the wait is
RECORDED, not merely logged — migration 0073 adds `capacity_blocked_since` and
`capacity_note`, stamped once and preserved across retries so the wait is
measured from the first refusal. It is bounded at two full turns: a fleet that
frees will free within one, and a phase that waited two hours must say so rather
than sit pending forever looking like a bug.
Mission launch still fails when NO node could ever run the backend — that is not
transient, waiting will not fix it, and `microvm-negctl` asserts such a mission
stays `draft`. Capacity refusals are transient and queue; capability refusals are
not and fail. The two are separate variants precisely so they cannot be confused.
257 lib tests, 20 binaries.
Phase 1a of the fleet-intelligence plan: the arithmetic and the inputs. Nothing
is wired to it yet; the launch path still picks `capable.first()`.
Placement has been `ORDER BY last_seen DESC` + `.first()` — the most recently
heartbeated node. Among healthy nodes all heartbeating every 5s that is
arbitrary, and it consults nothing about load, so two missions launched together
land on the same machine. It did not matter while tank held the only rootfs
image. All three nodes serve `claude` as of today.
THE correctness point, and the reason this is not a sort change: a VM that booted
30 seconds ago holds a fraction of its 8 GiB claim, so `mem_pct` reports a
sold-out node as nearly idle. `capacity_of` takes the WORSE of observed usage and
committed usage. The negative control pins it with the measured case — tank at
60 GiB total / 12 GiB observed / 5 VMs booted: utilisation alone says 5 more fit,
the node has room for 1. Booking those five is a node in swap, which slows every
VM on it together.
Commitments are unioned BY IDENTITY, never added: `vm_list` reports booted VMs,
`nodes::pinned_microvm_phases` reports phases chosen but not yet booted (a
window of seconds in which a real 8 GiB claim exists that no node can report).
The deterministic `vm_id_for` is what lets the same phase be recognised in both —
counting it twice would shrink the fleet by the number of phases starting.
`EvalRow::headroom()` finally gets a caller. It was written with the doc comment
"for placement ranking" and has had zero callers since. It is a TIEBREAK, not a
gate: ranking is slots first (spread, don't stack), then live headroom, then node
id so the same fleet state yields the same answer twice — which `last_seen DESC`
could never promise.
Fail-closed per house convention: draining, stale health (>30s, tuned just above
the 20s offline sweeper), and an unanswerable `vm_list` are all INELIGIBLE rather
than low-scoring. Stale Beszel metrics are the one exception — they demote a node
to zero headroom instead of excluding it, because they only ever break ties.
`FleetAtCapacity` and `FleetUnreadable` are separate variants with a test
asserting the second never says "at capacity": an operator sent hunting a load
problem that is really a dead daemon wastes the outage.
Also names the two nodes that were both called "New node" (tank, morpheus) — a
capacity report naming two machines identically is one nobody can act on.
257 lib tests.
