Commit Graph
2 Commits
Author SHA1 Message Date
Omar SobhandClaude Opus 5 dc8f65fc64 fix(metrics): GPU, network and disk IO were arriving and being dropped
ci / gates (push) Failing after 6s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped
`gpu_pct` read `info.g` as a scalar. Beszel 0.18 puts GPU in a different
collection entirely, as a MAP keyed by GPU index —
`{"0":{"n":"GeForce RTX 5060 Ti","u":0,"p":4.38}}` in `system_stats.stats`
— and `systems.info` carries no `g` at all. So every NVIDIA node reported
null while the data sat one request away. Null and "no GPU" are
indistinguishable downstream, so the fleet card showed nothing and a
`gpu_pct` drain rule could never fire, both without an error.

`net_sent_ps`, `net_recv_ps`, `disk_read_ps` and `disk_write_ps` were
columns nothing ever wrote. They come from the same sample.

The two array orders were MEASURED, not read off a schema, because
inverting one does not fail — it reports upload as download forever:

  b   = [sent, recv]. `stats.ni` gives per-interface [sent_ps, recv_ps,
        total_sent, total_recv]; indices 2 and 3 matched /proc/net/dev
        tx_bytes and rx_bytes on all four of tank's interfaces, and `b` is
        the sum of the per-second pair across them.
  dio = [read, write]. An 800 MB dd on tank moved index 1 from 7441 to
        23688 while index 0 stayed near zero.

`info.ct` is deliberately NOT mapped to container_count. It reads 1 on
tank, which runs 1 container, and also 1 on architect, which runs 4 —
right exactly often enough to pass a spot check.

One extra request per poll, not one per node: the newest 1m sample for
every system arrives in a single sorted page. A hub that cannot answer it
falls back to the info snapshot rather than losing the CPU and memory
readings that still work.

GPU is the busiest card, not the mean — placement asks whether there is a
free GPU, and averaging a saturated card with an idle one answers a
question nobody asked.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-11 17:53:16 -07:00
Omar SobhandClaude Opus 4.8 36a227566b Fleet: Beszel hub integration — rich per-node metrics + per-node monitor (Phase 1)
ci / gates (push) Failing after 5s
ci / rust (push) Has been skipped
ci / sandbox-k8s (push) Has been skipped
ci / frontend (push) Has been skipped
ci / e2e (push) Has been skipped
Tap each node's Beszel metrics (GPU/temps/disk-IO/network/per-container — beyond
our basic heartbeat) by reading the workspace's Beszel hub. The agents run in
WS-only mode with no locally-readable socket, so (per the de-risk) the server taps
the hub's PocketBase API instead of the daemon reading agents — no daemon changes.

- migrations: workspace_beszel (BYO hub URL + login, server-side only, mirrors the
  Tailscale BYO pattern) + node_metrics (latest scalar columns + JSONB blob).
- cm-db: repo/fleet_beszel.rs, repo/node_metrics.rs; nodes SELECT joins node_metrics
  (gpu_pct/temp_max surfaced on node_json for the live cards).
- cm-api: beszel.rs client (auth-with-password, poll `systems`, map to nodes by
  hostname, upsert metrics) + a 15s spawn_poller; routes/beszel.rs (connect/status/
  disconnect + GET /api/nodes/{id}/metrics with history proxied live from the hub).
- frontend: HostCard gains a GPU/temp readout + a Monitor button; NodeMonitor is a
  full-width per-node page (current panel + CPU/mem/GPU/temp/net/disk charts from the
  hub's 1m history); a "Beszel monitoring" connect form in the Local view.

Reachability confirmed: gw-04 → the hub over the tailnet (100.123.224.84:8090). Needs
the user to connect their hub login to activate the poller.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-25 23:28:29 -07:00