feat(fleet): B1 — microvm runtime kind and KVM placement predicate
Phase B step 1, on top of the B0 spike that proved microVMs boot here.
KVM is a HARD predicate, not a preference. gw-04 — where every mission
runs today — is itself a VM without nested virtualisation and has no
/dev/kvm, so a microvm mission landing there cannot start at all. The
scheduler therefore has to be able to tell nodes apart, which means the
node has to report what it can host.
Nodes gain a `capabilities` jsonb, populated from a probe on the node
rather than from configuration: /dev/kvm either exists there or it does
not, and nothing on the server can make it appear. The probe OPENS the
device rather than stat-ing it, because it can exist while being
unopenable (wrong group, or a container without the device passed
through) — which is precisely how firecracker will fail.
`microvm` requires BOTH kvm and a firecracker binary. A node with KVM
but no binary looks capable by the obvious test and fails at launch; a
node with the binary but no KVM is gw-04.
Placement fails the launch when no capable node exists, rather than
letting a mission sit in 'running' with nowhere to run. An explicit
target_node_id is treated as a request, not a guarantee — it is honoured
only if that node actually reports the capability.
`capabilities` defaults to '{}' NOT NULL so a node that has never
reported fails every predicate: an unqueried node and an incapable node
must be indistinguishable to the scheduler, because scheduling onto a
node whose abilities are unknown is how you get a mission that cannot
start and does not say why. The report replaces rather than merges, so a
capability the node has LOST disappears instead of leaving a stale true.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
4454a1cfd9
commit
0f7fa31f86
@@ -178,6 +178,54 @@ pub async fn set_status(pool: &PgPool, id: NodeId, status: &str) -> Result<(), D
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Record what a node reports it can host, for placement predicates.
|
||||
///
|
||||
/// Replaces rather than merges: the node sends its complete view on every
|
||||
/// report, so a capability it has *stopped* having (firecracker uninstalled,
|
||||
/// `/dev/kvm` gone after a reboot into a non-virt kernel) must disappear here
|
||||
/// too. Merging would let a stale `true` survive forever.
|
||||
pub async fn set_capabilities(
|
||||
pool: &PgPool,
|
||||
id: NodeId,
|
||||
capabilities: &serde_json::Value,
|
||||
) -> Result<(), DbError> {
|
||||
sqlx::query("UPDATE nodes SET capabilities = $2 WHERE id = $1")
|
||||
.bind(id.as_uuid())
|
||||
.bind(capabilities)
|
||||
.execute(pool)
|
||||
.await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Online nodes that report every one of `required` as `true`.
|
||||
///
|
||||
/// The predicate side of placement. Nothing is assumed: a node that has never
|
||||
/// reported has `capabilities = '{}'`, which fails every requirement — an
|
||||
/// unqueried node and an incapable node are treated identically, because
|
||||
/// scheduling work onto a node whose abilities are unknown is how you get a
|
||||
/// mission that cannot start and does not say why.
|
||||
pub async fn online_with_capabilities(
|
||||
pool: &PgPool,
|
||||
workspace_id: uuid::Uuid,
|
||||
required: &[&str],
|
||||
) -> Result<Vec<NodeId>, DbError> {
|
||||
let needed: serde_json::Value = required
|
||||
.iter()
|
||||
.map(|k| ((*k).to_string(), serde_json::Value::Bool(true)))
|
||||
.collect::<serde_json::Map<_, _>>()
|
||||
.into();
|
||||
let rows: Vec<(uuid::Uuid,)> = sqlx::query_as(
|
||||
"SELECT id FROM nodes
|
||||
WHERE workspace_id = $1 AND status = 'online' AND capabilities @> $2
|
||||
ORDER BY last_seen DESC NULLS LAST",
|
||||
)
|
||||
.bind(workspace_id)
|
||||
.bind(&needed)
|
||||
.fetch_all(pool)
|
||||
.await?;
|
||||
Ok(rows.into_iter().map(|(id,)| NodeId::from(id)).collect())
|
||||
}
|
||||
|
||||
/// Mark online nodes whose last heartbeat is older than `secs` as offline.
|
||||
pub async fn mark_stale_offline(pool: &PgPool, secs: i64) -> Result<(), DbError> {
|
||||
sqlx::query(
|
||||
|
||||
Reference in New Issue
Block a user