osobh/clawmates
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(placement): place per phase, and let a full fleet queue

Browse Source 
Phase 1b: wires the capacity model from 3a2d76a into the launch path, and turns
the existing pending-phase loop into the queue.

Placement moves from mission launch to PHASE launch. A node chosen at launch is
chosen once, minutes before the first VM boots and hours before the last — and
re-placing between phases is free, because mission state lives on the gateway
checkout and every VM is inject -> run -> collect -> destroy. Pinning early
bought nothing and cost the ability to react to a node filling or draining
mid-mission. One call site serves both the solo and composed paths so they cannot
disagree; the composed worker reads `missions.target_node_id`, which placement
writes before dispatch.

QUEUEING, with no new machinery: a phase with no admissible node keeps its
`pending` status and creates no `topology_runs` row. `start_pending_phases`
retries every 10s — that loop already was a queue; nothing downstream ever sees a
run that did not happen.

The risk that creates is the one this codebase keeps paying for: a phase waiting
for capacity looks exactly like a phase nothing is working on. So the wait is
RECORDED, not merely logged — migration 0073 adds `capacity_blocked_since` and
`capacity_note`, stamped once and preserved across retries so the wait is
measured from the first refusal. It is bounded at two full turns: a fleet that
frees will free within one, and a phase that waited two hours must say so rather
than sit pending forever looking like a bug.

Mission launch still fails when NO node could ever run the backend — that is not
transient, waiting will not fix it, and `microvm-negctl` asserts such a mission
stays `draft`. Capacity refusals are transient and queue; capability refusals are
not and fail. The two are separate variants precisely so they cannot be confused.

257 lib tests, 20 binaries.
This commit is contained in:
Omar Sobh
2026-08-08 08:45:42 -07:00
parent 3a2d76aa43
commit d84d17207f
4 changed files with 143 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/cm-api/src/mission_orchestrator.rs
+22 -25
View File
@@ -146,33 +146,30 @@ pub async fn on_launch(
backend.unwrap_or("<name>")
);
let how_to_fix = how_to_fix.as_str();
let chosen = match mission.target_node_id {
// An explicit target is a request, not a guarantee. Honour it only
// if the node actually reports the capability.
Some(want) => *capable
.iter()
.find(|n| n.as_uuid() == want)
.ok_or_else(|| format!(
"mission targets node {want}, which is not an online node that can \
run backend {:?} — {how_to_fix}",
backend.unwrap_or("default")
))?,
None => *capable.first().ok_or_else(|| format!(
// Names the BACKEND, not just "microvm capability". Both nodes
// report that capability; what one of them lacked was the image.
// The first version of this message would have sent an operator to
// reinstall firecracker on a node that already had it.
// CAPABILITY is checked here; CAPACITY is not, and no node is pinned.
//
// Placement moved to phase launch (`phase_runner`). A node chosen now
// would be chosen once, minutes before the first VM boots and hours
// before the last — and re-placing between phases is free, because
// mission state lives on the gateway checkout and every VM is
// inject → run → collect → destroy. Pinning early bought nothing and
// cost the ability to react to a node filling or draining mid-mission.
//
// Launching still FAILS here when no node could ever run this backend:
// that is not transient, waiting will not fix it, and the harness's
// `microvm-negctl` scenario asserts such a mission stays `draft`.
if capable.is_empty() {
return Err(format!(
"no online node can run backend {:?} — {how_to_fix}",
backend.unwrap_or("default")
))?,
};
sqlx::query("UPDATE missions SET target_node_id = $1, updated_at = now() WHERE id = $2")
.bind(chosen.as_uuid())
.bind(mission_id)
.execute(pool)
.await
.map_err(|e| format!("pin mission {mission_id} to node {chosen:?}: {e}"))?;
eprintln!("mission_orchestrator: mission {mission_id} placed on microvm node {chosen:?}");
));
}
eprintln!(
"mission_orchestrator: mission {mission_id} has {} node(s) able to run \
backend {:?}; placement happens per phase",
capable.len(),
backend.unwrap_or("default")
);
}
// A microVM mission materialises no team. Its phases run as one `claude -p`