fix(placement): a young VM's unconsumed memory was handed out twice

The capacity harness scenario, on its first full run, caught what it was
written to catch:

  capacity:   architect peaked at 6 of 6 slot(s)
  FAIL       capacity: 'morpheus' peaked at 3 concurrent VM(s) with only 2 slot(s)
  capacity:   tank peaked at 6 of 6 slot(s)
  PASS       capacity: the over-capacity missions QUEUED
  PASS       capacity: all 16 queued/placed missions completed

`capacity_of` inferred the host's own footprint by subtracting the VMs' FULL
8 GiB claim from observed usage — which assumes they have already consumed it.
A VM booted seconds ago holds about an eighth. On morpheus (31757 MiB total,
4314 MiB idle, 2 slots) with 2 young VMs at ~6314 MiB observed, the inference
6314 - 16384 goes negative, clamps to the 2048 floor, and invents 2266 MiB —
exactly enough for a third VM on a two-slot node.

The footprint is only honestly MEASURABLE when nothing is committed, so
remember it then: `nodes.mem_baseline_mib`, sampled by `survey` whenever it
observes an idle node with fresh health. When VMs are committed, take the
LARGER of the remembered reading and the old inference — a node that was once
idle at 4 GiB and is now running a 20 GiB build must not be scored as idle,
which would be the same over-commit arrived at from the other direction. Both
directions have a test; the second is the one that would otherwise rot.

Raising HOST_BASELINE_FLOOR_MIB would have made this one node's numbers pass
and drifted the moment the fleet changed shape.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-09 05:25:28 -07:00
co-authored by Claude Opus 5
parent 2056bb1d9e
commit 4fedfcec30
4 changed files with 151 additions and 17 deletions
+14
View File
@@ -409,3 +409,17 @@ mod tests {
assert_eq!(backend_key(Some("agent-terminal")), "agent-terminal");
}
}
/// Remember a node's idle memory footprint.
///
/// Only ever called with a reading taken while the node had ZERO phase VMs
/// committed — that is the one moment the number is honestly observable.
/// Writing it at any other time would record the VMs as part of the host.
pub async fn set_mem_baseline(pool: &PgPool, node_id: NodeId, mib: i64) -> Result<(), DbError> {
sqlx::query("UPDATE nodes SET mem_baseline_mib = $2 WHERE id = $1")
.bind(node_id.as_uuid())
.bind(mib)
.execute(pool)
.await?;
Ok(())
}