Fleet: actionable executions — rules engine + metrics-aware placement (Phase 2)
ci / gates (push) Failing after 16s
ci / rust (push) Has been skipped
ci / sandbox-k8s (push) Has been skipped
ci / frontend (push) Has been skipped
ci / e2e (push) Has been skipped

Turn the Beszel-tapped metrics into a self-managing loop.

- migration node_rules (workspace/node-scoped: metric op threshold, for_seconds,
  action JSONB, last_fired).
- cm-db: repo/node_rules.rs (CRUD + list_enabled); node_metrics::eval_all merges
  Beszel + heartbeat scalars per node + a headroom() heuristic; nodes::status_of;
  heartbeat now PRESERVES a `draining` status across heartbeats (so a cordon sticks).
- cm-api: node_rules.rs evaluator (spawn_evaluator, 20s) — when a metric condition
  holds for the rule's window it fires drain / undrain / alert (in-memory sustained
  + cooldown tracking, modeled on the node sweeper); routes/beszel.rs rules CRUD
  (GET/POST/PATCH/DELETE /api/fleet/rules); spawned in clawmates-server.
- cm-runtime: placement_node() is metrics-aware — a `draining` node stops receiving
  new agent sandboxes (falls back to local), so the drain rule is actionable.
- frontend: FleetRules section in the Local view — build rules (node · metric · op ·
  threshold · duration → action), toggle/delete, with fired-history.

The loop: hot/overloaded node → rule drains it → placement avoids it → recovers →
undrain rule brings it back. Deployed; node_rules migration applied.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
Omar Sobh
2026-06-25 23:43:49 -07:00
co-authored by Claude Opus 4.8
parent 36a227566b
commit c94784bab2
11 changed files with 546 additions and 3 deletions
+2
View File
@@ -271,6 +271,8 @@ async fn run() -> Result<(), String> {
cm_api::fleet::spawn_node_sweeper(pool.clone(), std::time::Duration::from_secs(8), 20);
// Beszel: poll each workspace's monitoring hub for rich per-node metrics.
cm_api::beszel::spawn_poller(pool.clone(), std::time::Duration::from_secs(15));
// Fleet automation: evaluate metric-threshold rules → drain/undrain/alert.
cm_api::node_rules::spawn_evaluator(pool.clone(), std::time::Duration::from_secs(20));
// Hosted identity (Clerk / OIDC): pin the issuer and load its JWKS.
let auth_verifier = match config.auth.mode {