7 Commits
Author SHA1 Message Date
osobh f30ea04ab6 Merge pull request 'Phase 9 R1: repo-ensure RPC + fleet-health fixes' (#107) from phase-9-r1a-repo-ensure-rpc into main
Build with clawstor cache / Cargo build (clawstor-cached) (push) Failing after 3s
2026-07-31 20:05:11 +00:00
osobhandClaude Sonnet 5 bb24c77676 Include self in dashboard-v2 fleet aggregation
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Failing after 3s
V2State::from_config built its peer fan-out list solely from
[[cluster.peers]], which by definition never includes the local
node. Result: hitting a given node's /api/v2/fleet directly always
omitted that node from its own fleet view, even when perfectly
healthy -- looked like "node X is missing" from the dashboard when
X just never queried itself.

Fix: synthesize a self PeerEntry from the node's own gossip bind
address and include it in the fan-out, same as any other peer.
peer_rpc_addr()'s existing +1 port convention resolves it to the
same bind_rpc_lan/bind_rpc_tailscale the daemon already listens on.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-07-31 13:01:02 -07:00
osobhandClaude Sonnet 5 75d822d0f2 Fail fast when cluster gossip bootstrap fails
Bind failures at startup are almost always a boot-time race against
DHCP/network-online (bind address not yet assigned to the interface).
Previously the daemon caught the error and kept running in a degraded
state with no gossip, RPC, or Prometheus endpoint and no visible
failure signal. Now it propagates the error so the process exits and
systemd's Restart=on-failure retries once the network is actually up.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-07-31 11:48:40 -07:00
Omar Sobh dd2b90872a Phase 9 R1c: daemon wiring + integration test
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Failing after 3s
- Wire RpcRouter::with_repo_root at daemon startup in services.rs
  using <blob_store_root>/repos. Nodes with no blob_store_root
  still return NotConfigured (unchanged).
- Add end-to-end ensure→cached→release integration test that seeds
  a bare git repo in a tempdir and exercises the real git-clone
  path. Marked #[ignore] so CI runners without git skip silently;
  runs green locally.
2026-07-15 04:11:58 -07:00
Omar Sobh 5c9bc7eb9c Phase 9 R1b: aggregator fan-out for RepoEnsure/RepoRelease
Layers HTTP over the R1a per-peer primitive so external callers
(clawmates, gitea runners, ops tooling) speak one URL to the
aggregator instead of dialing every peer.

Endpoints (require v2 auth, same middleware as tags/sessions):
  POST /api/v2/repos/ensure   {url, git_ref, workspace?}
  POST /api/v2/repos/release  {url, git_ref, workspace?}

Namespaced tokens are pinned to their own workspace (workspace omitted
in body → derived from token; explicit mismatch → 403). Admin/open
callers must supply workspace explicitly.

Reply shape mirrors the tag fan-out (FanoutReply) with per-peer
{peer, ok, path?, head_sha?, cached?, removed?, error?}. all_ok is
true iff every peer succeeded.

Also adds client wrappers call_repo_ensure/call_repo_release in
cluster/rpc/client.rs used by the aggregator's fan-out.
2026-07-15 04:08:35 -07:00
Omar Sobh 5c1d962bf2 Phase 9 R1a fixup: drop dead if-let wrapping around mkdir 2026-07-15 03:48:24 -07:00
Omar Sobh a3fe1d147c Phase 9 R1a: peer RepoEnsure/RepoRelease RPC (0x1e/0x1f)
New per-peer RPCs to shallow-clone a (url, git_ref) under a caller-
provided workspace namespace, and to release the checkout. Fleet
fan-out via the aggregator ships separately in R1b.

- src/cluster/repo_ensure.rs: request/reply types + derive_path
  (traversal-safe, blake3-hashed url segment), ensure_repo (cache
  hit → rev-parse HEAD, else remove-and-reclone with 5-min timeout),
  release_repo (idempotent rm)
- src/cluster/rpc.rs: Method::RepoEnsure=0x1e, RepoRelease=0x1f,
  RpcRouter::with_repo_root builder, two dispatch arms returning
  NotConfigured when repo_root is unset
- src/cluster.rs: pub mod repo_ensure
- src/actions.rs: fix pre-existing test-only Config init missing the
  aggregator field (unblocks lib tests)

Tests: 6 unit tests covering path derivation determinism, ref/url
independence, traversal safety, and sanitizer edge cases. All pass.
2026-07-15 03:42:27 -07:00
2 changed files with 39 additions and 18 deletions
+10 -10
View File
@@ -7,7 +7,7 @@ use crate::manifest::Manifest;
use crate::snapshot; use crate::snapshot;
use crate::sync::{SyncQueue, drain_sync_queue}; use crate::sync::{SyncQueue, drain_sync_queue};
use crate::zfs::SystemZfs; use crate::zfs::SystemZfs;
use anyhow::Result; use anyhow::{Context, Result};
use chrono::Utc; use chrono::Utc;
use sysinfo::{ProcessRefreshKind, RefreshKind, System}; use sysinfo::{ProcessRefreshKind, RefreshKind, System};
use tokio::time::{interval, Duration}; use tokio::time::{interval, Duration};
@@ -36,7 +36,14 @@ pub async fn run(cfg: Config, mut manifest: Manifest) -> Result<()> {
let hot_dir = cfg.hot.path.clone(); let hot_dir = cfg.hot.path.clone();
let hot_max_bytes = cfg.hot.max_gb.saturating_mul(1024 * 1024 * 1024); let hot_max_bytes = cfg.hot.max_gb.saturating_mul(1024 * 1024 * 1024);
let blob_root = cluster_cfg.blob_store_root.clone(); let blob_root = cluster_cfg.blob_store_root.clone();
match ClusterServices::start( // Fail fast rather than degrade silently: a bind failure here is
// almost always a boot-time race against DHCP/network-online
// (the bind address isn't assigned to the interface yet). The
// systemd unit has `Restart=on-failure`; exiting lets it retry
// a few seconds later once the network is actually up, instead
// of leaving the daemon running indefinitely with no gossip,
// RPC, or Prometheus endpoint and no visible failure state.
let svc = ClusterServices::start(
cluster_cfg, cluster_cfg,
cfg.node.name.clone(), cfg.node.name.clone(),
hot_dir, hot_dir,
@@ -44,8 +51,7 @@ pub async fn run(cfg: Config, mut manifest: Manifest) -> Result<()> {
blob_root, blob_root,
) )
.await .await
{ .context("starting cluster services")?;
Ok(svc) => {
tracing::info!( tracing::info!(
rpc_enabled = svc.rpc_enabled(), rpc_enabled = svc.rpc_enabled(),
blob_store_enabled = svc.blob_store_enabled(), blob_store_enabled = svc.blob_store_enabled(),
@@ -54,12 +60,6 @@ pub async fn run(cfg: Config, mut manifest: Manifest) -> Result<()> {
); );
Some(svc) Some(svc)
} }
Err(e) => {
tracing::error!(error = %e, "cluster services failed to start; continuing without cluster");
None
}
}
}
None => { None => {
tracing::info!("no [cluster] section in config; running standalone"); tracing::info!("no [cluster] section in config; running standalone");
None None
+22 -1
View File
@@ -91,9 +91,30 @@ impl V2State {
.join("aggregator-sessions.json"); .join("aggregator-sessions.json");
let sessions = SessionStore::load(sessions_path) let sessions = SessionStore::load(sessions_path)
.map_err(|e| anyhow::anyhow!("loading session store: {e}"))?; .map_err(|e| anyhow::anyhow!("loading session store: {e}"))?;
// Bug fix 2026-07-31: the fleet view previously never included
// the node actually serving the dashboard — `cluster.peers` is
// by definition every *other* node, so hitting a given node's
// `/api/v2/fleet` directly silently dropped that node from its
// own view (looked like "node X is missing" from the UI, even
// though X was perfectly healthy — it just never queried
// itself). Fix: synthesize a self `PeerEntry` from our own
// gossip bind address and include it in the fan-out list, same
// as any other peer. `peer_rpc_addr` derives the RPC port from
// `lan_addr`/`tailscale_addr` via the fleet's +1 convention, so
// this resolves to the same `bind_rpc_lan`/`bind_rpc_tailscale`
// the daemon actually listens on.
let self_peer = PeerEntry {
name: cfg.node.name.clone(),
zone: cluster.zone.clone(),
lan_addr: cluster.bind_lan,
tailscale_addr: cluster.bind_tailscale,
};
let mut peers = cluster.peers.clone();
peers.push(self_peer);
Ok(Self { Ok(Self {
aggregator_name: cfg.node.name.clone(), aggregator_name: cfg.node.name.clone(),
peers: cluster.peers.clone(), peers,
client: std::sync::Arc::new(client), client: std::sync::Arc::new(client),
default_rpc_port_offset: 1, default_rpc_port_offset: 1,
api_token: cfg.api_token.clone(), api_token: cfg.api_token.clone(),