Fleet: actionable executions — rules engine + metrics-aware placement (Phase 2)
Turn the Beszel-tapped metrics into a self-managing loop. - migration node_rules (workspace/node-scoped: metric op threshold, for_seconds, action JSONB, last_fired). - cm-db: repo/node_rules.rs (CRUD + list_enabled); node_metrics::eval_all merges Beszel + heartbeat scalars per node + a headroom() heuristic; nodes::status_of; heartbeat now PRESERVES a `draining` status across heartbeats (so a cordon sticks). - cm-api: node_rules.rs evaluator (spawn_evaluator, 20s) — when a metric condition holds for the rule's window it fires drain / undrain / alert (in-memory sustained + cooldown tracking, modeled on the node sweeper); routes/beszel.rs rules CRUD (GET/POST/PATCH/DELETE /api/fleet/rules); spawned in clawmates-server. - cm-runtime: placement_node() is metrics-aware — a `draining` node stops receiving new agent sandboxes (falls back to local), so the drain rule is actionable. - frontend: FleetRules section in the Local view — build rules (node · metric · op · threshold · duration → action), toggle/delete, with fired-history. The loop: hot/overloaded node → rule drains it → placement avoids it → recovers → undrain rule brings it back. Deployed; node_rules migration applied. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
36a227566b
commit
c94784bab2
@@ -121,7 +121,10 @@ pub async fn heartbeat(
|
||||
h: &NodeHealth,
|
||||
) -> Result<(), DbError> {
|
||||
sqlx::query(
|
||||
"UPDATE nodes SET status = 'online', last_seen = now(),
|
||||
// Preserve a rule-/operator-set `draining` state across heartbeats; a node
|
||||
// is only un-drained by an explicit set_status.
|
||||
"UPDATE nodes SET status = CASE WHEN status = 'draining' THEN 'draining' ELSE 'online' END,
|
||||
last_seen = now(),
|
||||
agent_version = COALESCE($2, agent_version),
|
||||
tailscale_ip = COALESCE($3, tailscale_ip),
|
||||
hostname = COALESCE($4, hostname),
|
||||
@@ -198,6 +201,16 @@ pub async fn delete(pool: &PgPool, id: NodeId, workspace_id: WorkspaceId) -> Res
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// A node's current status (for metrics-aware placement: skip 'draining').
|
||||
pub async fn status_of(pool: &PgPool, id: NodeId) -> Result<Option<String>, DbError> {
|
||||
Ok(
|
||||
sqlx::query_scalar::<_, String>("SELECT status FROM nodes WHERE id = $1")
|
||||
.bind(id.as_uuid())
|
||||
.fetch_optional(pool)
|
||||
.await?,
|
||||
)
|
||||
}
|
||||
|
||||
fn map_node(r: sqlx::postgres::PgRow) -> NodeRow {
|
||||
let health = r
|
||||
.get::<Option<uuid::Uuid>, _>("health_node")
|
||||
|
||||
Reference in New Issue
Block a user