fix(fleet): placement requires the backend's rootfs image, not just KVM

The first real microVM mission was placed on morpheus because it reports
{"microvm": true}, while only tank had rootfs-claude.ext4. It failed by name
rather than booting the wrong image — but whether a mission ran came down to
which capable node was listed first, which is a coin flip dressed as scheduling.
`missions.backend` was invisible to the scheduler.

The node now enumerates the images on its disk and reports them as a `rootfs`
ARRAY. `microvm::available_backends` lives beside `rootfs_for`, its inverse,
because the two must agree on what a backend name means; split apart, one drifts
and the scheduler starts promising images the booter cannot find. It only
advertises names `rootfs_for` would accept, and reports an empty array rather than
omitting the key — set_capabilities REPLACES, so a deleted image stops being
advertised instead of leaving a stale claim.

`nodes::online_for_backend` requires microvm AND that the node's list contains the
mission's backend. A node on an older daemon has no `rootfs` key and matches
nothing: unknown is not permission, the same treatment every other capability
gets. `backend_key` maps the three spellings of "the default image" to the one
name the node advertises, and is tested — a mismatch there would reject every node
for an ordinary mission with no backend set.

The launch error now names both halves of the fix, since "no capable node" was
true but unhelpful when the node was capable and merely lacked the image.

Mission gains `backend` on the domain struct; it was a column the executor read
from the phase query while the struct that placement uses could not see it.

464 tests pass, clippy clean.
This commit is contained in:
Omar Sobh
2026-08-05 17:22:12 -07:00
parent 1cd81a8b2a
commit d9f53a3f96
5 changed files with 171 additions and 15 deletions
+39 -7
View File
@@ -299,7 +299,11 @@ fn probe_capabilities() -> Value {
.map(|l| l.trim().to_string())
});
capabilities_from(kvm, firecracker.as_deref())
// Which rootfs images are actually on this node's disk. Reported so
// placement can require the mission's backend rather than assuming any
// KVM-capable node can boot any image — see microvm::available_backends.
let backends = microvm::available_backends();
capabilities_from(kvm, firecracker.as_deref(), &backends)
}
/// Shape the capability report from probe results.
@@ -307,10 +311,17 @@ fn probe_capabilities() -> Value {
/// Split from [`probe_capabilities`] so the rule can be tested without a
/// `/dev/kvm` to open — the machine running the tests is usually the one that
/// cannot host a microVM.
fn capabilities_from(kvm: bool, firecracker: Option<&str>) -> Value {
fn capabilities_from(kvm: bool, firecracker: Option<&str>, backends: &[String]) -> Value {
json!({
"kvm": kvm,
"firecracker": firecracker,
// The backends this node can boot. An ARRAY, and empty when there are
// none: `set_capabilities` REPLACES, so an image that was deleted stops
// being advertised on the next report instead of leaving a stale claim.
//
// Reported even when `microvm` is false, because it is a fact about the
// disk rather than a promise — placement requires both.
"rootfs": backends,
// BOTH must hold. A node with KVM but no firecracker binary looks
// capable by the obvious test and fails at launch; a node with the
// binary but no KVM is gw-04. Computed here rather than in the
@@ -1378,20 +1389,20 @@ mod capability_tests {
#[test]
fn microvm_needs_both_kvm_and_firecracker() {
assert_eq!(
capabilities_from(true, Some("Firecracker v1.16.1"))["microvm"],
capabilities_from(true, Some("Firecracker v1.16.1"), &[])["microvm"],
json!(true)
);
assert_eq!(
capabilities_from(true, None)["microvm"],
capabilities_from(true, None, &[])["microvm"],
json!(false),
"KVM without firecracker cannot host a microVM"
);
assert_eq!(
capabilities_from(false, Some("Firecracker v1.16.1"))["microvm"],
capabilities_from(false, Some("Firecracker v1.16.1"), &[])["microvm"],
json!(false),
"firecracker without KVM is gw-04 — it can never host one"
);
assert_eq!(capabilities_from(false, None)["microvm"], json!(false));
assert_eq!(capabilities_from(false, None, &[])["microvm"], json!(false));
}
/// The report replaces rather than merges server-side, so a node that has
@@ -1399,11 +1410,32 @@ mod capability_tests {
/// key and a false one must not be distinguishable to the predicate.
#[test]
fn a_lost_capability_is_reported_false_not_omitted() {
let caps = capabilities_from(false, None);
let caps = capabilities_from(false, None, &[]);
assert!(caps.get("kvm").is_some(), "kvm must always be present");
assert!(
caps.get("microvm").is_some(),
"microvm must always be present"
);
// Same reasoning for the image list: a node that deleted its last rootfs
// must report an empty ARRAY, not omit the key. Placement asks "does this
// node have backend X"; against a missing key that question has no
// answer, and a scheduler with no answer picks something.
assert_eq!(
caps.get("rootfs"),
Some(&json!([])),
"rootfs must always be present, empty when there are no images"
);
}
/// The list is what placement matches a mission's `backend` against, so it
/// must carry the names verbatim.
#[test]
fn reported_backends_are_the_names_placement_will_ask_for() {
let caps = capabilities_from(
true,
Some("Firecracker v1.16.1"),
&["claude".to_string(), "default".to_string()],
);
assert_eq!(caps["rootfs"], json!(["claude", "default"]));
}
}