Omar SobhandClaude Opus 4.8
c94784bab2
Turn the Beszel-tapped metrics into a self-managing loop. - migration node_rules (workspace/node-scoped: metric op threshold, for_seconds, action JSONB, last_fired). - cm-db: repo/node_rules.rs (CRUD + list_enabled); node_metrics::eval_all merges Beszel + heartbeat scalars per node + a headroom() heuristic; nodes::status_of; heartbeat now PRESERVES a `draining` status across heartbeats (so a cordon sticks). - cm-api: node_rules.rs evaluator (spawn_evaluator, 20s) — when a metric condition holds for the rule's window it fires drain / undrain / alert (in-memory sustained + cooldown tracking, modeled on the node sweeper); routes/beszel.rs rules CRUD (GET/POST/PATCH/DELETE /api/fleet/rules); spawned in clawmates-server. - cm-runtime: placement_node() is metrics-aware — a `draining` node stops receiving new agent sandboxes (falls back to local), so the drain rule is actionable. - frontend: FleetRules section in the Local view — build rules (node · metric · op · threshold · duration → action), toggle/delete, with fired-history. The loop: hot/overloaded node → rule drains it → placement avoids it → recovers → undrain rule brings it back. Deployed; node_rules migration applied. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
20 lines
1.1 KiB
SQL
20 lines
1.1 KiB
SQL
-- Fleet automation: metric-driven rules. When a node's metric crosses a threshold
|
|
-- for a sustained window, fire an action (drain a hot node, bring one back, alert).
|
|
-- A null node_id applies the rule to every node in the workspace. Paired with the
|
|
-- metrics-aware placement (draining nodes stop receiving new agent workloads).
|
|
CREATE TABLE node_rules (
|
|
id UUID PRIMARY KEY,
|
|
workspace_id UUID NOT NULL REFERENCES workspaces (id) ON DELETE CASCADE,
|
|
node_id UUID REFERENCES nodes (id) ON DELETE CASCADE,
|
|
name TEXT NOT NULL,
|
|
enabled BOOLEAN NOT NULL DEFAULT true,
|
|
metric TEXT NOT NULL, -- cpu_pct | mem_pct | disk_pct | gpu_pct | temp_max | load1
|
|
op TEXT NOT NULL, -- '>' | '<' | '>=' | '<='
|
|
threshold DOUBLE PRECISION NOT NULL,
|
|
for_seconds INTEGER NOT NULL DEFAULT 60,
|
|
action JSONB NOT NULL, -- {"type":"drain"|"undrain"|"alert"}
|
|
last_fired_at TIMESTAMPTZ,
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
|
);
|
|
CREATE INDEX node_rules_ws ON node_rules (workspace_id);
|