af70c5416ea8228b60795d468c8f49b4a95b6110
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar SobhandClaude Opus 5
|
4fedfcec30 |
fix(placement): a young VM's unconsumed memory was handed out twice
The capacity harness scenario, on its first full run, caught what it was written to catch: capacity: architect peaked at 6 of 6 slot(s) FAIL capacity: 'morpheus' peaked at 3 concurrent VM(s) with only 2 slot(s) capacity: tank peaked at 6 of 6 slot(s) PASS capacity: the over-capacity missions QUEUED PASS capacity: all 16 queued/placed missions completed `capacity_of` inferred the host's own footprint by subtracting the VMs' FULL 8 GiB claim from observed usage — which assumes they have already consumed it. A VM booted seconds ago holds about an eighth. On morpheus (31757 MiB total, 4314 MiB idle, 2 slots) with 2 young VMs at ~6314 MiB observed, the inference 6314 - 16384 goes negative, clamps to the 2048 floor, and invents 2266 MiB — exactly enough for a third VM on a two-slot node. The footprint is only honestly MEASURABLE when nothing is committed, so remember it then: `nodes.mem_baseline_mib`, sampled by `survey` whenever it observes an idle node with fresh health. When VMs are committed, take the LARGER of the remembered reading and the old inference — a node that was once idle at 4 GiB and is now running a 20 GiB build must not be scored as idle, which would be the same over-commit arrived at from the other direction. Both directions have a test; the second is the one that would otherwise rot. Raising HOST_BASELINE_FLOOR_MIB would have made this one node's numbers pass and drifted the moment the fleet changed shape. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar Sobh
|
3a2d76aa43 |
feat(placement): capacity model for the fleet — observed memory is not capacity
Phase 1a of the fleet-intelligence plan: the arithmetic and the inputs. Nothing is wired to it yet; the launch path still picks `capable.first()`. Placement has been `ORDER BY last_seen DESC` + `.first()` — the most recently heartbeated node. Among healthy nodes all heartbeating every 5s that is arbitrary, and it consults nothing about load, so two missions launched together land on the same machine. It did not matter while tank held the only rootfs image. All three nodes serve `claude` as of today. THE correctness point, and the reason this is not a sort change: a VM that booted 30 seconds ago holds a fraction of its 8 GiB claim, so `mem_pct` reports a sold-out node as nearly idle. `capacity_of` takes the WORSE of observed usage and committed usage. The negative control pins it with the measured case — tank at 60 GiB total / 12 GiB observed / 5 VMs booted: utilisation alone says 5 more fit, the node has room for 1. Booking those five is a node in swap, which slows every VM on it together. Commitments are unioned BY IDENTITY, never added: `vm_list` reports booted VMs, `nodes::pinned_microvm_phases` reports phases chosen but not yet booted (a window of seconds in which a real 8 GiB claim exists that no node can report). The deterministic `vm_id_for` is what lets the same phase be recognised in both — counting it twice would shrink the fleet by the number of phases starting. `EvalRow::headroom()` finally gets a caller. It was written with the doc comment "for placement ranking" and has had zero callers since. It is a TIEBREAK, not a gate: ranking is slots first (spread, don't stack), then live headroom, then node id so the same fleet state yields the same answer twice — which `last_seen DESC` could never promise. Fail-closed per house convention: draining, stale health (>30s, tuned just above the 20s offline sweeper), and an unanswerable `vm_list` are all INELIGIBLE rather than low-scoring. Stale Beszel metrics are the one exception — they demote a node to zero headroom instead of excluding it, because they only ever break ties. `FleetAtCapacity` and `FleetUnreadable` are separate variants with a test asserting the second never says "at capacity": an operator sent hunting a load problem that is really a dead daemon wastes the outage. Also names the two nodes that were both called "New node" (tank, morpheus) — a capacity report naming two machines identically is one nobody can act on. 257 lib tests. |
||
|
Omar SobhandClaude Opus 4.8
|
3554a3aaf2 |
CI: remove k8s stages, fix the Docker-level pipeline green
Survey + fixes so the pipeline passes at the Docker level (no k8s).
- Remove k8s: drop the `sandbox-k8s` job (kind/Calico/--features k8s-tests) and the
"Helm chart lints" gate step. release.yml was already k8s-clean.
- Rust job:
- `cargo fmt --all` — fix pre-existing formatting drift (fmt --check was failing).
- clippy -D warnings: fix 3 lib warnings (cm-brain sort_by_key→Reverse, cm-api
fleet.rs doc list indentation, node_rules map_or→is_none_or).
- Regenerate the .sqlx offline cache (was missing the cm-runtime run_loop test
query → offline compile failed). DB-backed tests use testcontainers at runtime.
- Set SQLX_OFFLINE=true on the rust + e2e jobs so query! macros compile against
the committed cache deterministically (no DB needed at compile time).
- Frontend job:
- Fix the 1 ESLint error (useAgentTelemetry: no setState-synchronously-in-effect;
tag the slice with agentId + derive null on mismatch).
- Fix 2 stale panel-params tests (`terminal` is a valid app id now; assert the
current APP_IDS + use a genuinely-unknown id for the reject case).
Verified locally: fmt clean, clippy --all-targets -D warnings clean (offline),
frontend lint 0 errors, tsc clean, 86/86 frontend tests pass, build OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
c94784bab2 |
Fleet: actionable executions — rules engine + metrics-aware placement (Phase 2)
Turn the Beszel-tapped metrics into a self-managing loop. - migration node_rules (workspace/node-scoped: metric op threshold, for_seconds, action JSONB, last_fired). - cm-db: repo/node_rules.rs (CRUD + list_enabled); node_metrics::eval_all merges Beszel + heartbeat scalars per node + a headroom() heuristic; nodes::status_of; heartbeat now PRESERVES a `draining` status across heartbeats (so a cordon sticks). - cm-api: node_rules.rs evaluator (spawn_evaluator, 20s) — when a metric condition holds for the rule's window it fires drain / undrain / alert (in-memory sustained + cooldown tracking, modeled on the node sweeper); routes/beszel.rs rules CRUD (GET/POST/PATCH/DELETE /api/fleet/rules); spawned in clawmates-server. - cm-runtime: placement_node() is metrics-aware — a `draining` node stops receiving new agent sandboxes (falls back to local), so the drain rule is actionable. - frontend: FleetRules section in the Local view — build rules (node · metric · op · threshold · duration → action), toggle/delete, with fired-history. The loop: hot/overloaded node → rule drains it → placement avoids it → recovers → undrain rule brings it back. Deployed; node_rules migration applied. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
36a227566b |
Fleet: Beszel hub integration — rich per-node metrics + per-node monitor (Phase 1)
Tap each node's Beszel metrics (GPU/temps/disk-IO/network/per-container — beyond
our basic heartbeat) by reading the workspace's Beszel hub. The agents run in
WS-only mode with no locally-readable socket, so (per the de-risk) the server taps
the hub's PocketBase API instead of the daemon reading agents — no daemon changes.
- migrations: workspace_beszel (BYO hub URL + login, server-side only, mirrors the
Tailscale BYO pattern) + node_metrics (latest scalar columns + JSONB blob).
- cm-db: repo/fleet_beszel.rs, repo/node_metrics.rs; nodes SELECT joins node_metrics
(gpu_pct/temp_max surfaced on node_json for the live cards).
- cm-api: beszel.rs client (auth-with-password, poll `systems`, map to nodes by
hostname, upsert metrics) + a 15s spawn_poller; routes/beszel.rs (connect/status/
disconnect + GET /api/nodes/{id}/metrics with history proxied live from the hub).
- frontend: HostCard gains a GPU/temp readout + a Monitor button; NodeMonitor is a
full-width per-node page (current panel + CPU/mem/GPU/temp/net/disk charts from the
hub's 1m history); a "Beszel monitoring" connect form in the Local view.
Reachability confirmed: gw-04 → the hub over the tailnet (100.123.224.84:8090). Needs
the user to connect their hub login to activate the poller.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|