Omar SobhandClaude Opus 5
4fedfcec30
The capacity harness scenario, on its first full run, caught what it was written to catch: capacity: architect peaked at 6 of 6 slot(s) FAIL capacity: 'morpheus' peaked at 3 concurrent VM(s) with only 2 slot(s) capacity: tank peaked at 6 of 6 slot(s) PASS capacity: the over-capacity missions QUEUED PASS capacity: all 16 queued/placed missions completed `capacity_of` inferred the host's own footprint by subtracting the VMs' FULL 8 GiB claim from observed usage — which assumes they have already consumed it. A VM booted seconds ago holds about an eighth. On morpheus (31757 MiB total, 4314 MiB idle, 2 slots) with 2 young VMs at ~6314 MiB observed, the inference 6314 - 16384 goes negative, clamps to the 2048 floor, and invents 2266 MiB — exactly enough for a third VM on a two-slot node. The footprint is only honestly MEASURABLE when nothing is committed, so remember it then: `nodes.mem_baseline_mib`, sampled by `survey` whenever it observes an idle node with fresh health. When VMs are committed, take the LARGER of the remembered reading and the old inference — a node that was once idle at 4 GiB and is now running a 20 GiB build must not be scored as idle, which would be the same over-commit arrived at from the other direction. Both directions have a test; the second is the one that would otherwise rot. Raising HOST_BASELINE_FLOOR_MIB would have made this one node's numbers pass and drifted the moment the fleet changed shape. Co-Authored-By: Claude Opus 5 <[email protected]>
21 lines
1.1 KiB
SQL
21 lines
1.1 KiB
SQL
-- What a node's memory looks like with NO phase VMs on it.
|
|
--
|
|
-- `vm_placement::capacity_of` used to infer the host's own footprint by
|
|
-- subtracting the VMs' full claim from observed usage. That is right only when
|
|
-- the VMs have actually consumed what they claimed. A VM booted seconds ago
|
|
-- holds ~1 GiB of its 8 GiB, so the subtraction goes negative, hits the 2 GiB
|
|
-- floor, and hands back memory the host is really using.
|
|
--
|
|
-- Measured on morpheus (31757 MiB total, 4314 MiB idle) with 2 VMs committed:
|
|
-- the inferred baseline collapsed from 4314 to 2048, which freed 2266 MiB —
|
|
-- just enough to fit a 3rd VM into a 2-slot node. The capacity harness caught
|
|
-- it as "morpheus peaked at 3 concurrent VM(s) with only 2 slot(s)".
|
|
--
|
|
-- So remember the baseline instead of re-deriving it: it is only honestly
|
|
-- observable when the node has nothing committed, and that is exactly when it
|
|
-- is recorded.
|
|
ALTER TABLE nodes ADD COLUMN IF NOT EXISTS mem_baseline_mib BIGINT;
|
|
|
|
COMMENT ON COLUMN nodes.mem_baseline_mib IS
|
|
'Memory in use with zero phase VMs committed, sampled by vm_placement::survey. NULL until first observed idle.';
|