Fleet: actionable executions — rules engine + metrics-aware placement (Phase 2)
ci / gates (push) Failing after 16s
ci / rust (push) Has been skipped
ci / sandbox-k8s (push) Has been skipped
ci / frontend (push) Has been skipped
ci / e2e (push) Has been skipped

Turn the Beszel-tapped metrics into a self-managing loop.

- migration node_rules (workspace/node-scoped: metric op threshold, for_seconds,
  action JSONB, last_fired).
- cm-db: repo/node_rules.rs (CRUD + list_enabled); node_metrics::eval_all merges
  Beszel + heartbeat scalars per node + a headroom() heuristic; nodes::status_of;
  heartbeat now PRESERVES a `draining` status across heartbeats (so a cordon sticks).
- cm-api: node_rules.rs evaluator (spawn_evaluator, 20s) — when a metric condition
  holds for the rule's window it fires drain / undrain / alert (in-memory sustained
  + cooldown tracking, modeled on the node sweeper); routes/beszel.rs rules CRUD
  (GET/POST/PATCH/DELETE /api/fleet/rules); spawned in clawmates-server.
- cm-runtime: placement_node() is metrics-aware — a `draining` node stops receiving
  new agent sandboxes (falls back to local), so the drain rule is actionable.
- frontend: FleetRules section in the Local view — build rules (node · metric · op ·
  threshold · duration → action), toggle/delete, with fired-history.

The loop: hot/overloaded node → rule drains it → placement avoids it → recovers →
undrain rule brings it back. Deployed; node_rules migration applied.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
Omar Sobh
2026-06-25 23:43:49 -07:00
co-authored by Claude Opus 4.8
parent 36a227566b
commit c94784bab2
11 changed files with 546 additions and 3 deletions
+14 -1
View File
@@ -121,7 +121,10 @@ pub async fn heartbeat(
h: &NodeHealth,
) -> Result<(), DbError> {
sqlx::query(
"UPDATE nodes SET status = 'online', last_seen = now(),
// Preserve a rule-/operator-set `draining` state across heartbeats; a node
// is only un-drained by an explicit set_status.
"UPDATE nodes SET status = CASE WHEN status = 'draining' THEN 'draining' ELSE 'online' END,
last_seen = now(),
agent_version = COALESCE($2, agent_version),
tailscale_ip = COALESCE($3, tailscale_ip),
hostname = COALESCE($4, hostname),
@@ -198,6 +201,16 @@ pub async fn delete(pool: &PgPool, id: NodeId, workspace_id: WorkspaceId) -> Res
Ok(())
}
/// A node's current status (for metrics-aware placement: skip 'draining').
pub async fn status_of(pool: &PgPool, id: NodeId) -> Result<Option<String>, DbError> {
Ok(
sqlx::query_scalar::<_, String>("SELECT status FROM nodes WHERE id = $1")
.bind(id.as_uuid())
.fetch_optional(pool)
.await?,
)
}
fn map_node(r: sqlx::postgres::PgRow) -> NodeRow {
let health = r
.get::<Option<uuid::Uuid>, _>("health_node")