osobh/clawmates
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(fleet): placement requires the backend's rootfs image, not just KVM

Browse Source 
The first real microVM mission was placed on morpheus because it reports
{"microvm": true}, while only tank had rootfs-claude.ext4. It failed by name
rather than booting the wrong image — but whether a mission ran came down to
which capable node was listed first, which is a coin flip dressed as scheduling.
`missions.backend` was invisible to the scheduler.

The node now enumerates the images on its disk and reports them as a `rootfs`
ARRAY. `microvm::available_backends` lives beside `rootfs_for`, its inverse,
because the two must agree on what a backend name means; split apart, one drifts
and the scheduler starts promising images the booter cannot find. It only
advertises names `rootfs_for` would accept, and reports an empty array rather than
omitting the key — set_capabilities REPLACES, so a deleted image stops being
advertised instead of leaving a stale claim.

`nodes::online_for_backend` requires microvm AND that the node's list contains the
mission's backend. A node on an older daemon has no `rootfs` key and matches
nothing: unknown is not permission, the same treatment every other capability
gets. `backend_key` maps the three spellings of "the default image" to the one
name the node advertises, and is tested — a mismatch there would reject every node
for an ordinary mission with no backend set.

The launch error now names both halves of the fix, since "no capable node" was
true but unhelpful when the node was capable and merely lacked the image.

Mission gains `backend` on the domain struct; it was a column the executor read
from the phase query while the struct that placement uses could not see it.

464 tests pass, clippy clean.
This commit is contained in:
Omar Sobh
2026-08-05 17:22:12 -07:00
parent 1cd81a8b2a
commit d9f53a3f96
5 changed files with 171 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/cm-api/src/mission_orchestrator.rs
+15 -6
View File
@@ -131,22 +131,31 @@ pub async fn on_launch(
// none, because the alternative is a mission that sits in 'running' having
// never had anywhere to run.
if mission.runtime_kind == "microvm" {
// Capable means BOTH: it can host a microVM, and it holds the image this
// mission's backend names. Asking only for `microvm` sent the first real
// microVM mission to a node without `rootfs-claude.ext4`.
let backend = mission.backend.as_deref();
let capable =
cm_db::repo::nodes::online_with_capabilities(pool, mission.workspace_id, &["microvm"])
cm_db::repo::nodes::online_for_backend(pool, mission.workspace_id, backend)
.await
.map_err(|e| format!("looking up microvm-capable nodes: {e}"))?;
const HOW_TO_FIX: &str = "needs /dev/kvm and firecracker installed — \
see scripts/fc-node-setup.sh";
.map_err(|e| format!("looking up nodes for backend {backend:?}: {e}"))?;
let how_to_fix = format!(
"needs /dev/kvm + firecracker (scripts/fc-node-setup.sh) AND the {} rootfs \
built on that node (scripts/fc-build-rootfs.sh <host> <image> {})",
backend.unwrap_or("default"),
backend.unwrap_or("<name>")
);
let how_to_fix = how_to_fix.as_str();
let chosen = match mission.target_node_id {
// An explicit target is a request, not a guarantee. Honour it only
// if the node actually reports the capability.
Some(want) => *capable
.iter()
.find(|n| n.as_uuid() == want)
.ok_or_else(|| format!("mission targets node {want}, which is not an online node reporting microvm capability ({HOW_TO_FIX})"))?,
.ok_or_else(|| format!("mission targets node {want}, which is not an online node reporting microvm capability ({how_to_fix})"))?,
None => *capable
.first()
.ok_or_else(|| format!("no online node reports microvm capability ({HOW_TO_FIX})"))?,
.ok_or_else(|| format!("no online node reports microvm capability ({how_to_fix})"))?,
};
sqlx::query("UPDATE missions SET target_node_id = $1, updated_at = now() WHERE id = $2")
.bind(chosen.as_uuid())