fix(placement): a young VM's unconsumed memory was handed out twice
The capacity harness scenario, on its first full run, caught what it was written to catch: capacity: architect peaked at 6 of 6 slot(s) FAIL capacity: 'morpheus' peaked at 3 concurrent VM(s) with only 2 slot(s) capacity: tank peaked at 6 of 6 slot(s) PASS capacity: the over-capacity missions QUEUED PASS capacity: all 16 queued/placed missions completed `capacity_of` inferred the host's own footprint by subtracting the VMs' FULL 8 GiB claim from observed usage — which assumes they have already consumed it. A VM booted seconds ago holds about an eighth. On morpheus (31757 MiB total, 4314 MiB idle, 2 slots) with 2 young VMs at ~6314 MiB observed, the inference 6314 - 16384 goes negative, clamps to the 2048 floor, and invents 2266 MiB — exactly enough for a third VM on a two-slot node. The footprint is only honestly MEASURABLE when nothing is committed, so remember it then: `nodes.mem_baseline_mib`, sampled by `survey` whenever it observes an idle node with fresh health. When VMs are committed, take the LARGER of the remembered reading and the old inference — a node that was once idle at 4 GiB and is now running a 20 GiB build must not be scored as idle, which would be the same over-commit arrived at from the other direction. Both directions have a test; the second is the one that would otherwise rot. Raising HOST_BASELINE_FLOOR_MIB would have made this one node's numbers pass and drifted the moment the fleet changed shape. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
2056bb1d9e
commit
4fedfcec30
@@ -78,6 +78,10 @@ pub struct EvalRow {
|
||||
pub mem_total_bytes: Option<i64>,
|
||||
pub mem_used_bytes: Option<i64>,
|
||||
pub disk_free_bytes: Option<i64>,
|
||||
/// Memory in use with no phase VMs committed, remembered from the last time
|
||||
/// this node was observed idle. `None` until then, which makes placement
|
||||
/// fall back to inferring it — the behaviour that over-committed morpheus.
|
||||
pub mem_baseline_mib: Option<i64>,
|
||||
/// Age of each source. Placement is fail-closed on stale health (a node whose
|
||||
/// RAM we cannot read is one we are guessing at), and demotes rather than
|
||||
/// excludes on stale Beszel metrics, which only ever break ties.
|
||||
@@ -133,6 +137,7 @@ pub async fn eval_all(pool: &PgPool) -> Result<Vec<EvalRow>, DbError> {
|
||||
h.mem_total AS mem_total_bytes,
|
||||
h.mem_used AS mem_used_bytes,
|
||||
h.disk_free AS disk_free_bytes,
|
||||
n.mem_baseline_mib,
|
||||
EXTRACT(EPOCH FROM now() - h.captured_at)::float8 AS health_age_secs,
|
||||
EXTRACT(EPOCH FROM now() - m.updated_at)::float8 AS metrics_age_secs
|
||||
FROM nodes n
|
||||
@@ -156,6 +161,7 @@ pub async fn eval_all(pool: &PgPool) -> Result<Vec<EvalRow>, DbError> {
|
||||
mem_total_bytes: r.get("mem_total_bytes"),
|
||||
mem_used_bytes: r.get("mem_used_bytes"),
|
||||
disk_free_bytes: r.get("disk_free_bytes"),
|
||||
mem_baseline_mib: r.get("mem_baseline_mib"),
|
||||
health_age_secs: r.get("health_age_secs"),
|
||||
metrics_age_secs: r.get("metrics_age_secs"),
|
||||
})
|
||||
|
||||
@@ -409,3 +409,17 @@ mod tests {
|
||||
assert_eq!(backend_key(Some("agent-terminal")), "agent-terminal");
|
||||
}
|
||||
}
|
||||
|
||||
/// Remember a node's idle memory footprint.
|
||||
///
|
||||
/// Only ever called with a reading taken while the node had ZERO phase VMs
|
||||
/// committed — that is the one moment the number is honestly observable.
|
||||
/// Writing it at any other time would record the VMs as part of the host.
|
||||
pub async fn set_mem_baseline(pool: &PgPool, node_id: NodeId, mib: i64) -> Result<(), DbError> {
|
||||
sqlx::query("UPDATE nodes SET mem_baseline_mib = $2 WHERE id = $1")
|
||||
.bind(node_id.as_uuid())
|
||||
.bind(mib)
|
||||
.execute(pool)
|
||||
.await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user