fix(placement): a young VM's unconsumed memory was handed out twice

The capacity harness scenario, on its first full run, caught what it was
written to catch:

  capacity:   architect peaked at 6 of 6 slot(s)
  FAIL       capacity: 'morpheus' peaked at 3 concurrent VM(s) with only 2 slot(s)
  capacity:   tank peaked at 6 of 6 slot(s)
  PASS       capacity: the over-capacity missions QUEUED
  PASS       capacity: all 16 queued/placed missions completed

`capacity_of` inferred the host's own footprint by subtracting the VMs' FULL
8 GiB claim from observed usage — which assumes they have already consumed it.
A VM booted seconds ago holds about an eighth. On morpheus (31757 MiB total,
4314 MiB idle, 2 slots) with 2 young VMs at ~6314 MiB observed, the inference
6314 - 16384 goes negative, clamps to the 2048 floor, and invents 2266 MiB —
exactly enough for a third VM on a two-slot node.

The footprint is only honestly MEASURABLE when nothing is committed, so
remember it then: `nodes.mem_baseline_mib`, sampled by `survey` whenever it
observes an idle node with fresh health. When VMs are committed, take the
LARGER of the remembered reading and the old inference — a node that was once
idle at 4 GiB and is now running a 20 GiB build must not be scored as idle,
which would be the same over-commit arrived at from the other direction. Both
directions have a test; the second is the one that would otherwise rot.

Raising HOST_BASELINE_FLOOR_MIB would have made this one node's numbers pass
and drifted the moment the fleet changed shape.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-09 05:25:28 -07:00
co-authored by Claude Opus 5
parent 2056bb1d9e
commit 4fedfcec30
4 changed files with 151 additions and 17 deletions
+6
View File
@@ -78,6 +78,10 @@ pub struct EvalRow {
pub mem_total_bytes: Option<i64>,
pub mem_used_bytes: Option<i64>,
pub disk_free_bytes: Option<i64>,
/// Memory in use with no phase VMs committed, remembered from the last time
/// this node was observed idle. `None` until then, which makes placement
/// fall back to inferring it — the behaviour that over-committed morpheus.
pub mem_baseline_mib: Option<i64>,
/// Age of each source. Placement is fail-closed on stale health (a node whose
/// RAM we cannot read is one we are guessing at), and demotes rather than
/// excludes on stale Beszel metrics, which only ever break ties.
@@ -133,6 +137,7 @@ pub async fn eval_all(pool: &PgPool) -> Result<Vec<EvalRow>, DbError> {
h.mem_total AS mem_total_bytes,
h.mem_used AS mem_used_bytes,
h.disk_free AS disk_free_bytes,
n.mem_baseline_mib,
EXTRACT(EPOCH FROM now() - h.captured_at)::float8 AS health_age_secs,
EXTRACT(EPOCH FROM now() - m.updated_at)::float8 AS metrics_age_secs
FROM nodes n
@@ -156,6 +161,7 @@ pub async fn eval_all(pool: &PgPool) -> Result<Vec<EvalRow>, DbError> {
mem_total_bytes: r.get("mem_total_bytes"),
mem_used_bytes: r.get("mem_used_bytes"),
disk_free_bytes: r.get("disk_free_bytes"),
mem_baseline_mib: r.get("mem_baseline_mib"),
health_age_secs: r.get("health_age_secs"),
metrics_age_secs: r.get("metrics_age_secs"),
})
+14
View File
@@ -409,3 +409,17 @@ mod tests {
assert_eq!(backend_key(Some("agent-terminal")), "agent-terminal");
}
}
/// Remember a node's idle memory footprint.
///
/// Only ever called with a reading taken while the node had ZERO phase VMs
/// committed — that is the one moment the number is honestly observable.
/// Writing it at any other time would record the VMs as part of the host.
pub async fn set_mem_baseline(pool: &PgPool, node_id: NodeId, mib: i64) -> Result<(), DbError> {
sqlx::query("UPDATE nodes SET mem_baseline_mib = $2 WHERE id = $1")
.bind(node_id.as_uuid())
.bind(mib)
.execute(pool)
.await?;
Ok(())
}