Fleet: actionable executions — rules engine + metrics-aware placement (Phase 2)
Turn the Beszel-tapped metrics into a self-managing loop. - migration node_rules (workspace/node-scoped: metric op threshold, for_seconds, action JSONB, last_fired). - cm-db: repo/node_rules.rs (CRUD + list_enabled); node_metrics::eval_all merges Beszel + heartbeat scalars per node + a headroom() heuristic; nodes::status_of; heartbeat now PRESERVES a `draining` status across heartbeats (so a cordon sticks). - cm-api: node_rules.rs evaluator (spawn_evaluator, 20s) — when a metric condition holds for the rule's window it fires drain / undrain / alert (in-memory sustained + cooldown tracking, modeled on the node sweeper); routes/beszel.rs rules CRUD (GET/POST/PATCH/DELETE /api/fleet/rules); spawned in clawmates-server. - cm-runtime: placement_node() is metrics-aware — a `draining` node stops receiving new agent sandboxes (falls back to local), so the drain rule is actionable. - frontend: FleetRules section in the Local view — build rules (node · metric · op · threshold · duration → action), toggle/delete, with fired-history. The loop: hot/overloaded node → rule drains it → placement avoids it → recovers → undrain rule brings it back. Deployed; node_rules migration applied. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
36a227566b
commit
c94784bab2
@@ -271,6 +271,8 @@ async fn run() -> Result<(), String> {
|
||||
cm_api::fleet::spawn_node_sweeper(pool.clone(), std::time::Duration::from_secs(8), 20);
|
||||
// Beszel: poll each workspace's monitoring hub for rich per-node metrics.
|
||||
cm_api::beszel::spawn_poller(pool.clone(), std::time::Duration::from_secs(15));
|
||||
// Fleet automation: evaluate metric-threshold rules → drain/undrain/alert.
|
||||
cm_api::node_rules::spawn_evaluator(pool.clone(), std::time::Duration::from_secs(20));
|
||||
|
||||
// Hosted identity (Clerk / OIDC): pin the issuer and load its JWKS.
|
||||
let auth_verifier = match config.auth.mode {
|
||||
|
||||
Reference in New Issue
Block a user