Phase 1a: cluster module + LAN-first peer probe
First cut of the v2 distributed FS. Captures the architecture design in ARCHITECTURE-v2.md and lands the smallest useful new capability: probing cluster peers with a LAN-first policy so subsequent transport + gossip layers (Phase 1b, 1c) can build on a real routing decision. New: - ARCHITECTURE-v2.md: zones (fabric-10g/lan-1g/roaming), tier lifecycle (hot/warm/cold), fingerprint-keyed build cache design, smart-clean policy, phase plan, explicit non-goals. - claw-store/src/cluster.rs: RouteKind, RouteWinner, LanFirstProbe. LAN 200ms timeout, Tailscale 500ms fallback. 8 tests use real TCP listeners on 127.0.0.1 (no mocks); cover happy path, fall-through, both-fail, single-address, and elapsed reporting. - claw-store/src/config.rs: ClusterConfig + PeerEntry with validation (bind-address presence, no duplicate peer names, per-peer reachable address required). Optional at top level so pre-v2 configs still load unchanged. 6 new tests. - claw-store/src/main.rs: `claw-store cluster-probe <peer>` CLI subcommand that reads config, resolves the peer, probes, prints the winning route + elapsed time. All 16 new tests pass. Existing 45 pass. Sole failure (hot::tests::test_project_target_size_bytes) is a pre-existing macOS-only issue with `du -sb`; Linux CI unaffected. Follow-on Phase 1 cuts (subsequent sessions): - 1b: chitchat SWIM gossip for live membership state - 1c: quinn QUIC transport with fleet-CA mTLS - 1d: `claw-store cluster status` — live membership view Every file well under the 1300-line ceiling (cluster.rs 268, config.rs 428, main.rs 450).
This commit is contained in:
@@ -0,0 +1,253 @@
|
|||||||
|
# clawstor v2 — Distributed Filesystem Architecture
|
||||||
|
|
||||||
|
Purpose-built distributed FS for a small fleet (3–10 nodes, single trust domain).
|
||||||
|
Not a Ceph-scale general-purpose system: opinionated, workload-specific, self-hosted.
|
||||||
|
|
||||||
|
## Non-goals
|
||||||
|
|
||||||
|
Explicit choices we do NOT make:
|
||||||
|
|
||||||
|
- **Not POSIX-strict.** No byte-range locks, `mmap` semantics, `flock`. FUSE mount is
|
||||||
|
read-mostly for git worktrees; cargo target dirs never traverse it.
|
||||||
|
- **Not erasure coding.** Simple replication (`N=2` warm, `N=3` for pinned-critical).
|
||||||
|
- **Not encryption at our layer.** ZFS native encryption at rest; Tailscale (WireGuard)
|
||||||
|
+ QUIC TLS 1.3 in transit. Adding a third layer is theater.
|
||||||
|
- **Not multi-tenant.** Single trust domain. Auth piggybacks on Tailscale identity for
|
||||||
|
roaming clients; fleet-CA-signed mTLS for LAN peers.
|
||||||
|
- **Not our own consensus algorithm.** Vendor `openraft` (only where strong consistency is
|
||||||
|
required) or `chitchat` (SWIM gossip for eventually-consistent state).
|
||||||
|
- **Not massive scale.** 3–10 nodes. Membership can be gossip; placement can be config.
|
||||||
|
|
||||||
|
## Workload shapes we serve
|
||||||
|
|
||||||
|
Five, and no others:
|
||||||
|
|
||||||
|
1. Source trees / git worktrees (~10–100 MB, moderate churn)
|
||||||
|
2. Cargo build artifacts (deps trees, 1–15 GB compressed, content-addressed)
|
||||||
|
3. ZFS state / snapshots (large, sequential)
|
||||||
|
4. ML model weights / cold blobs (multi-GB, WORM)
|
||||||
|
5. Small config/state (tiny, frequent reads, low writes)
|
||||||
|
|
||||||
|
## Fleet topology
|
||||||
|
|
||||||
|
Three zones defined by physical network:
|
||||||
|
|
||||||
|
| Zone | Members (typical) | Link | Purpose |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `fabric-10g` | tank, architect | 10G direct-connect | primary + warm-tier replication |
|
||||||
|
| `lan-1g` | morpheus (+ any 1G LAN node) | 1G switched | build node, warm-tier consumer |
|
||||||
|
| `roaming` | laptop, remote nodes | Tailscale WireGuard | read/write client with offline queue |
|
||||||
|
|
||||||
|
Every node has both a LAN IP (or none, for roaming) and a Tailscale IP. Peer connections
|
||||||
|
probe LAN first (200ms timeout) with Tailscale as fallback. Direct LAN sockets skip
|
||||||
|
WireGuard framing entirely — real perf win on the 10G fabric.
|
||||||
|
|
||||||
|
## Data tiers
|
||||||
|
|
||||||
|
```
|
||||||
|
┌───────┐ activate ┌────────┐ evict ┌───────┐ age-out ┌──────┐
|
||||||
|
│ new │ ───────────▶ │ HOT │ ─────────▶ │ WARM │ ──────────▶ │ COLD │
|
||||||
|
│(miss) │ │(local │ │(fleet │ │(arch │
|
||||||
|
│ │ ◀── build ── │target) │ ◀── get ── │blob) │ ◀─ promote │ zfs) │
|
||||||
|
└───────┘ └────────┘ └───────┘ └──────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Hot** — local NVMe cargo target dir. LRU + stale-hours eviction. Symlinked into the
|
||||||
|
workspace on `activate`. Never accessed remotely.
|
||||||
|
- **Warm** — content-addressed chunked blobs in `/slab/build-cache/by-hash/<fingerprint>/`
|
||||||
|
on ZFS-owning nodes. Placement per zone rules; replicated within `fabric-10g`.
|
||||||
|
- **Cold** — same content-addressed layout but zstd-19 compressed, single copy on
|
||||||
|
architect's `data` pool (7.27 TB). Rehydrates to warm on demand.
|
||||||
|
|
||||||
|
## The killer feature: fingerprint-keyed build cache
|
||||||
|
|
||||||
|
Cargo target dirs are captured post-build and stored fleet-wide, keyed by:
|
||||||
|
|
||||||
|
```
|
||||||
|
BLAKE3(
|
||||||
|
Cargo.lock
|
||||||
|
rust-toolchain.toml (if present)
|
||||||
|
.cargo/config.toml (if present)
|
||||||
|
active_rustc_version + host_triple + target_triple
|
||||||
|
RUSTFLAGS (or CARGO_ENCODED_RUSTFLAGS)
|
||||||
|
enabled_features (sorted)
|
||||||
|
profile ("dev" / "release")
|
||||||
|
CC / CXX versions (crates with build.rs)
|
||||||
|
glibc_version (Linux binary compat)
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Captured subset: `deps/ + .fingerprint/ + build/ + examples/ + small top-level`. Explicitly
|
||||||
|
NOT captured: `incremental/` (not portable across machines; rebuilt locally in ~1 min).
|
||||||
|
|
||||||
|
Chunk-level dedup via BLAKE3 4 MB chunks — two fingerprints sharing 95% of files share 95%
|
||||||
|
of storage.
|
||||||
|
|
||||||
|
Client integration is a thin `claw-cargo build` wrapper that:
|
||||||
|
1. Computes fingerprint.
|
||||||
|
2. `cache-check` → hit: streams chunks, extracts; miss: continues.
|
||||||
|
3. Runs `cargo build`.
|
||||||
|
4. `cache-put` on success.
|
||||||
|
|
||||||
|
The workspace-level `.cargo/config.toml` gets an `[alias] clean = "!claw-store clean --soft"`
|
||||||
|
so `cargo clean` invokes the smart-clean layer transparently.
|
||||||
|
|
||||||
|
## Smart clean — four modes
|
||||||
|
|
||||||
|
- `--incremental-only` — nuke `incremental/`, keep everything else. ~20-30% reclaim.
|
||||||
|
- `--soft` — remove local hot target dir, keep fleet blob. Default GC action.
|
||||||
|
- `--medium` — soft + demote fleet blob to cold if unreferenced + 14d old.
|
||||||
|
- `--hard` — full purge. Requires `--force` when live refs exist.
|
||||||
|
|
||||||
|
Reference tracking: every `cache-put` records the git refs that produced it. Nightly sweep
|
||||||
|
queries Gitea `/api/v1/repos/.../branches` and `/tags`. Fingerprints with zero live refs
|
||||||
|
+ age > retention → eligible for deletion.
|
||||||
|
|
||||||
|
## Membership + transport (Phase 1)
|
||||||
|
|
||||||
|
- **Membership** via `chitchat` SWIM gossip. 3-node cluster; heartbeat state, zones,
|
||||||
|
hot-tier occupancy.
|
||||||
|
- **Transport** via `quinn` (QUIC/TLS 1.3). LAN-first probe: 200 ms LAN attempt, then
|
||||||
|
Tailscale fallback. Winning route cached per peer per session; re-probe every 5 min
|
||||||
|
or on transport error.
|
||||||
|
- **Identity**: fleet CA signs peer mTLS certs on LAN. Tailscale-node-identity certs used
|
||||||
|
for roaming clients.
|
||||||
|
|
||||||
|
## Consistency model
|
||||||
|
|
||||||
|
- **Content-addressed blobs** — immutable, self-verifying. No consistency problem.
|
||||||
|
- **Metadata pointers** (HEAD, tag mappings, pin lists) — vector clocks + last-writer-wins
|
||||||
|
with human-readable warning on conflict. Small fleet + typically single-user means
|
||||||
|
conflicts are rare.
|
||||||
|
- **CRDT for pin sets** — grow-only set with tombstones for unpin.
|
||||||
|
- **Strong consistency** only where required (e.g. leader election for the eviction
|
||||||
|
coordinator) via a tiny `openraft` group across `fabric-10g` nodes only.
|
||||||
|
|
||||||
|
## Roaming client (full R/W, offline queue)
|
||||||
|
|
||||||
|
Same daemon binary, same primitives, adds:
|
||||||
|
|
||||||
|
- **Local WAL** (redb) — mutating ops appended before network. Crash-safe.
|
||||||
|
- **Bandwidth-aware push loop** — throttles to protect tether/coffee-shop wifi.
|
||||||
|
- **Reconnect handshake** — pull metadata delta, reconcile, push blobs (dedup automatic),
|
||||||
|
push metadata updates.
|
||||||
|
- **Pin classes** — `--local` (full offline), `--metadata-only`, `--hot` (with `--stale`).
|
||||||
|
- **LRU + budget** (`--max-local-gb=100`) — prevents disk-fill.
|
||||||
|
|
||||||
|
## FUSE mount (Phase 6)
|
||||||
|
|
||||||
|
Single mount point (`/shared/projects/` with per-user `~/projects` symlink) exposing
|
||||||
|
warm-tier git worktrees fleet-wide. Read-mostly, small-file-friendly. Cargo target dirs
|
||||||
|
NEVER traverse this mount — they use the CLI-managed hot tier on local NVMe.
|
||||||
|
|
||||||
|
## CI runner integration
|
||||||
|
|
||||||
|
Runners advertise per-repo affinity as Gitea labels:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
labels:
|
||||||
|
- "ubuntu-latest:host"
|
||||||
|
- "clawstor:tank"
|
||||||
|
- "clawstor:has:osobh/zeroclaw,osobh/clawmates"
|
||||||
|
```
|
||||||
|
|
||||||
|
Workflows request affinity via `runs-on: [ubuntu-latest, "clawstor:has:${{ github.repository }}"]`.
|
||||||
|
`claw-store daemon` rewrites its labels dynamically as warm-tier state changes.
|
||||||
|
|
||||||
|
Two Gitea Actions replace stock steps:
|
||||||
|
- `clawstor-checkout@v1` — bind-mount from mount instead of git clone.
|
||||||
|
- `clawstor-cargo-cache@v1` — fingerprint-keyed cache-check / cache-put.
|
||||||
|
|
||||||
|
## Phase plan
|
||||||
|
|
||||||
|
| Phase | Chunk | Weeks |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 | Membership + LAN-first transport (chitchat + quinn) | 2 |
|
||||||
|
| 2 | Content-addressed blob store (BLAKE3 chunking, put/get) | 2 |
|
||||||
|
| 3 | Namespace + metadata + vector clocks + CRDTs | 3 |
|
||||||
|
| 4 | Client mode + WAL + reconnect + pin semantics | 3 |
|
||||||
|
| 5 | Cargo build-artifact cache integration | 3 |
|
||||||
|
| 6 | FUSE mount for git worktrees | 2 |
|
||||||
|
| 7 | Snapshot + repair + scrub + smart-clean + reference tracking | 3-4 |
|
||||||
|
| 8 | Fleet CA + Tailscale identity for roaming | 1 |
|
||||||
|
|
||||||
|
Roughly **18-20 weeks focused work**, ~5-7 months elapsed with other priorities.
|
||||||
|
|
||||||
|
## Placement policy
|
||||||
|
|
||||||
|
Rule-based, config-driven. Defaults sensible:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[[placement.rules]]
|
||||||
|
match = "size > 1GB and hot"
|
||||||
|
primary = "fabric-10g-any"
|
||||||
|
replicas = 2
|
||||||
|
|
||||||
|
[[placement.rules]]
|
||||||
|
match = "type = cold"
|
||||||
|
primary = "architect"
|
||||||
|
replicas = 1
|
||||||
|
|
||||||
|
[[placement.rules]]
|
||||||
|
default = true
|
||||||
|
primary = "tank"
|
||||||
|
replicas = ["architect"]
|
||||||
|
|
||||||
|
[placement.override."osobh/clawverse"]
|
||||||
|
primary = "architect"
|
||||||
|
replicas = ["tank"]
|
||||||
|
```
|
||||||
|
|
||||||
|
Read prioritization (fixed order, no config):
|
||||||
|
1. Local cache
|
||||||
|
2. Local warm tier
|
||||||
|
3. `fabric-10g` peer
|
||||||
|
4. `lan-1g` peer
|
||||||
|
5. `roaming` peer
|
||||||
|
|
||||||
|
Write path: local WAL → primary sync-replicate → replicas async-replicate.
|
||||||
|
|
||||||
|
## Cleanup policy
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[cleanup.hot]
|
||||||
|
default_max_age_hours = 168
|
||||||
|
default_max_total_gb = 500
|
||||||
|
default_min_free_gb = 100
|
||||||
|
soft_evict_if_fleet_cached = true
|
||||||
|
|
||||||
|
[cleanup.warm]
|
||||||
|
retain_referenced_days = 90
|
||||||
|
retain_unreferenced_days = 14
|
||||||
|
demote_to_cold_after_days = 30
|
||||||
|
min_replicas_fabric_10g = 2
|
||||||
|
|
||||||
|
[cleanup.cold]
|
||||||
|
retain_referenced_days = 365
|
||||||
|
retain_unreferenced_days = 60
|
||||||
|
|
||||||
|
[cleanup.triggers]
|
||||||
|
run_gc_nightly_at = "03:00"
|
||||||
|
run_gc_on_disk_pressure = true
|
||||||
|
disk_pressure_threshold_pct = 85
|
||||||
|
```
|
||||||
|
|
||||||
|
## Delivery constraints (per project rules)
|
||||||
|
|
||||||
|
- Strict TDD — tests before code, tests must fail before implementation.
|
||||||
|
- Every file under 1300 lines.
|
||||||
|
- No mocks, stubs, or TODOs — full implementations only.
|
||||||
|
- Real sockets in tests where network behavior matters.
|
||||||
|
|
||||||
|
## What Phase 1a delivers (this cut)
|
||||||
|
|
||||||
|
- `PeerAddress`, `Zone`, `NodeIdentity` types.
|
||||||
|
- `LanFirstProbe` — races LAN endpoint then Tailscale, returns winning route.
|
||||||
|
- `ClusterConfig` — optional config section listing peers with their two addresses.
|
||||||
|
- CLI: `claw-store cluster probe <peer-name>` — visible sanity check.
|
||||||
|
- Real TCP-listener tests, no mocks.
|
||||||
|
|
||||||
|
Follow-on Phase 1 cuts (subsequent sessions):
|
||||||
|
- 1b: chitchat gossip integration + peer state broadcast.
|
||||||
|
- 1c: `quinn` QUIC transport with fleet-CA mTLS.
|
||||||
|
- 1d: `claw-store cluster status` — live membership view.
|
||||||
Generated
+1
-1
@@ -256,7 +256,7 @@ checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9"
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "claw-store"
|
name = "claw-store"
|
||||||
version = "0.2.0"
|
version = "0.3.0"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"anyhow",
|
"anyhow",
|
||||||
"axum",
|
"axum",
|
||||||
|
|||||||
@@ -126,6 +126,7 @@ mod tests {
|
|||||||
cold: None,
|
cold: None,
|
||||||
replication: None,
|
replication: None,
|
||||||
peer: None,
|
peer: None,
|
||||||
|
cluster: None,
|
||||||
api_token: None,
|
api_token: None,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,268 @@
|
|||||||
|
//! Cluster peer routing.
|
||||||
|
//!
|
||||||
|
//! Phase 1a of the v2 distributed FS. Every fleet node has two potential
|
||||||
|
//! endpoints: a LAN socket (fast, direct, no WireGuard framing) and a
|
||||||
|
//! Tailscale socket (fallback, always reachable when the tailnet is up).
|
||||||
|
//! This module races LAN first and falls through to Tailscale, returning
|
||||||
|
//! the winning route so the caller can cache it for the session.
|
||||||
|
//!
|
||||||
|
//! Cache the returned [`RouteWinner`] and re-probe on transport error or
|
||||||
|
//! after ~5 min so route changes (node moved networks, LAN NIC came back)
|
||||||
|
//! propagate without a daemon restart.
|
||||||
|
|
||||||
|
use crate::config::PeerEntry;
|
||||||
|
use anyhow::{bail, Result};
|
||||||
|
use std::net::SocketAddr;
|
||||||
|
use std::time::{Duration, Instant};
|
||||||
|
use tokio::net::TcpStream;
|
||||||
|
use tokio::time::timeout;
|
||||||
|
|
||||||
|
/// Which network route reached a peer.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub enum RouteKind {
|
||||||
|
/// Direct LAN socket. No WireGuard overhead.
|
||||||
|
Lan,
|
||||||
|
/// Tailscale-provided socket. WireGuard-encapsulated.
|
||||||
|
Tailscale,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl RouteKind {
|
||||||
|
/// Short lowercase label suitable for logs, metrics, config.
|
||||||
|
pub fn as_str(self) -> &'static str {
|
||||||
|
match self {
|
||||||
|
RouteKind::Lan => "lan",
|
||||||
|
RouteKind::Tailscale => "tailscale",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A successful probe result: which address answered, which route kind it was,
|
||||||
|
/// and how long the probe took from start to first-success (useful for logs).
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct RouteWinner {
|
||||||
|
pub addr: SocketAddr,
|
||||||
|
pub kind: RouteKind,
|
||||||
|
pub elapsed: Duration,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// LAN-first probe.
|
||||||
|
///
|
||||||
|
/// Attempts the peer's LAN address with a tight timeout; on any error or
|
||||||
|
/// timeout, falls through to the Tailscale address with a longer timeout
|
||||||
|
/// (Tailscale may need to set up a direct WireGuard tunnel on first probe).
|
||||||
|
/// Errors only when neither route succeeds.
|
||||||
|
pub struct LanFirstProbe {
|
||||||
|
lan_timeout: Duration,
|
||||||
|
tailscale_timeout: Duration,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for LanFirstProbe {
|
||||||
|
/// LAN 200ms, Tailscale 500ms. Rationale: on a healthy LAN the TCP SYN
|
||||||
|
/// round-trip is sub-millisecond; anything slower than 200ms means the
|
||||||
|
/// LAN route is not usable and we should fall through immediately.
|
||||||
|
/// Tailscale needs headroom for WireGuard handshakes and DERP fallback.
|
||||||
|
fn default() -> Self {
|
||||||
|
Self {
|
||||||
|
lan_timeout: Duration::from_millis(200),
|
||||||
|
tailscale_timeout: Duration::from_millis(500),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl LanFirstProbe {
|
||||||
|
/// Probe using default timeouts (200ms LAN, 500ms Tailscale).
|
||||||
|
pub fn new() -> Self {
|
||||||
|
Self::default()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Probe with custom timeouts. Useful for tests and for tuning per
|
||||||
|
/// deployment shape.
|
||||||
|
pub fn with_timeouts(lan: Duration, tailscale: Duration) -> Self {
|
||||||
|
Self {
|
||||||
|
lan_timeout: lan,
|
||||||
|
tailscale_timeout: tailscale,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Probe a peer. Returns the first route (LAN preferred) that accepts a
|
||||||
|
/// TCP connection within its per-route timeout. Errors when neither
|
||||||
|
/// route succeeds, or when the peer has no addresses configured.
|
||||||
|
pub async fn probe(&self, peer: &PeerEntry) -> Result<RouteWinner> {
|
||||||
|
let start = Instant::now();
|
||||||
|
if let Some(lan) = peer.lan_addr {
|
||||||
|
if try_connect(lan, self.lan_timeout).await.is_ok() {
|
||||||
|
return Ok(RouteWinner {
|
||||||
|
addr: lan,
|
||||||
|
kind: RouteKind::Lan,
|
||||||
|
elapsed: start.elapsed(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(ts) = peer.tailscale_addr {
|
||||||
|
if try_connect(ts, self.tailscale_timeout).await.is_ok() {
|
||||||
|
return Ok(RouteWinner {
|
||||||
|
addr: ts,
|
||||||
|
kind: RouteKind::Tailscale,
|
||||||
|
elapsed: start.elapsed(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
bail!(
|
||||||
|
"peer {} has no reachable address (lan={:?}, tailscale={:?})",
|
||||||
|
peer.name,
|
||||||
|
peer.lan_addr,
|
||||||
|
peer.tailscale_addr,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Attempt a TCP connect with a deadline. Returns Ok on successful handshake,
|
||||||
|
/// Err on connection error or timeout.
|
||||||
|
async fn try_connect(addr: SocketAddr, deadline: Duration) -> Result<()> {
|
||||||
|
match timeout(deadline, TcpStream::connect(addr)).await {
|
||||||
|
Ok(Ok(_stream)) => Ok(()),
|
||||||
|
Ok(Err(e)) => Err(anyhow::Error::from(e)),
|
||||||
|
Err(_) => bail!("connect to {} timed out after {:?}", addr, deadline),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use tokio::net::TcpListener;
|
||||||
|
|
||||||
|
/// Bind a TCP listener on an OS-assigned port, then drop it. The returned
|
||||||
|
/// address points at a port that's very likely unbound. Tiny race window
|
||||||
|
/// between drop and the caller's probe (microseconds); in practice tests
|
||||||
|
/// pass reliably. If ever flaky, we upgrade to a strict SO_LINGER=0 trick.
|
||||||
|
async fn free_but_unbound_addr() -> SocketAddr {
|
||||||
|
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let addr = listener.local_addr().unwrap();
|
||||||
|
drop(listener);
|
||||||
|
addr
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Bind a listener and spawn an accept loop so it stays alive across probes.
|
||||||
|
/// Returns the listener's local address.
|
||||||
|
async fn spawn_accepting_listener() -> SocketAddr {
|
||||||
|
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let addr = listener.local_addr().unwrap();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
loop {
|
||||||
|
if listener.accept().await.is_err() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
addr
|
||||||
|
}
|
||||||
|
|
||||||
|
fn peer_with(lan: Option<SocketAddr>, ts: Option<SocketAddr>) -> PeerEntry {
|
||||||
|
PeerEntry {
|
||||||
|
name: "test-peer".into(),
|
||||||
|
zone: "test-zone".into(),
|
||||||
|
lan_addr: lan,
|
||||||
|
tailscale_addr: ts,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn probe_prefers_lan_when_available() {
|
||||||
|
let lan = spawn_accepting_listener().await;
|
||||||
|
// Tailscale unreachable — must not be attempted when LAN succeeds.
|
||||||
|
let ts = free_but_unbound_addr().await;
|
||||||
|
let probe = LanFirstProbe::new();
|
||||||
|
let win = probe.probe(&peer_with(Some(lan), Some(ts))).await.unwrap();
|
||||||
|
assert_eq!(win.kind, RouteKind::Lan);
|
||||||
|
assert_eq!(win.addr, lan);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn probe_falls_through_to_tailscale_when_lan_fails() {
|
||||||
|
let lan_dead = free_but_unbound_addr().await;
|
||||||
|
let ts = spawn_accepting_listener().await;
|
||||||
|
// Short LAN timeout so the test runs fast even if the OS holds the
|
||||||
|
// failed connect in a queue rather than refusing immediately.
|
||||||
|
let probe = LanFirstProbe::with_timeouts(
|
||||||
|
Duration::from_millis(50),
|
||||||
|
Duration::from_millis(500),
|
||||||
|
);
|
||||||
|
let win = probe
|
||||||
|
.probe(&peer_with(Some(lan_dead), Some(ts)))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(win.kind, RouteKind::Tailscale);
|
||||||
|
assert_eq!(win.addr, ts);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn probe_errors_when_both_routes_fail() {
|
||||||
|
let lan_dead = free_but_unbound_addr().await;
|
||||||
|
let ts_dead = free_but_unbound_addr().await;
|
||||||
|
let probe = LanFirstProbe::with_timeouts(
|
||||||
|
Duration::from_millis(50),
|
||||||
|
Duration::from_millis(50),
|
||||||
|
);
|
||||||
|
let err = probe
|
||||||
|
.probe(&peer_with(Some(lan_dead), Some(ts_dead)))
|
||||||
|
.await
|
||||||
|
.unwrap_err()
|
||||||
|
.to_string();
|
||||||
|
assert!(
|
||||||
|
err.contains("no reachable address"),
|
||||||
|
"unexpected error: {err}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn probe_uses_only_tailscale_when_lan_absent() {
|
||||||
|
let ts = spawn_accepting_listener().await;
|
||||||
|
let probe = LanFirstProbe::new();
|
||||||
|
let win = probe.probe(&peer_with(None, Some(ts))).await.unwrap();
|
||||||
|
assert_eq!(win.kind, RouteKind::Tailscale);
|
||||||
|
assert_eq!(win.addr, ts);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn probe_uses_only_lan_when_tailscale_absent() {
|
||||||
|
let lan = spawn_accepting_listener().await;
|
||||||
|
let probe = LanFirstProbe::new();
|
||||||
|
let win = probe.probe(&peer_with(Some(lan), None)).await.unwrap();
|
||||||
|
assert_eq!(win.kind, RouteKind::Lan);
|
||||||
|
assert_eq!(win.addr, lan);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn probe_errors_when_no_addresses_configured() {
|
||||||
|
// PeerEntry::validate would reject this at config-load time, but the
|
||||||
|
// probe must still handle it gracefully (defense in depth).
|
||||||
|
let probe = LanFirstProbe::new();
|
||||||
|
let err = probe
|
||||||
|
.probe(&peer_with(None, None))
|
||||||
|
.await
|
||||||
|
.unwrap_err()
|
||||||
|
.to_string();
|
||||||
|
assert!(
|
||||||
|
err.contains("no reachable address"),
|
||||||
|
"unexpected error: {err}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn probe_records_elapsed_time_under_timeout() {
|
||||||
|
let lan = spawn_accepting_listener().await;
|
||||||
|
let probe = LanFirstProbe::new();
|
||||||
|
let win = probe.probe(&peer_with(Some(lan), None)).await.unwrap();
|
||||||
|
assert!(
|
||||||
|
win.elapsed < Duration::from_millis(500),
|
||||||
|
"probe should complete quickly on a live local listener; took {:?}",
|
||||||
|
win.elapsed
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn route_kind_as_str_matches_variants() {
|
||||||
|
assert_eq!(RouteKind::Lan.as_str(), "lan");
|
||||||
|
assert_eq!(RouteKind::Tailscale.as_str(), "tailscale");
|
||||||
|
}
|
||||||
|
}
|
||||||
+278
-3
@@ -1,5 +1,7 @@
|
|||||||
use anyhow::{Context, Result};
|
use anyhow::{bail, Context, Result};
|
||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
|
use std::collections::HashSet;
|
||||||
|
use std::net::SocketAddr;
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
#[derive(Debug, Clone, Deserialize, Serialize, PartialEq)]
|
#[derive(Debug, Clone, Deserialize, Serialize, PartialEq)]
|
||||||
@@ -51,6 +53,88 @@ pub struct PeerConfig {
|
|||||||
pub user: String,
|
pub user: String,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// One peer node the cluster knows about. Every peer carries up to two reachable
|
||||||
|
/// endpoints: a LAN socket (fast path, tried first) and a Tailscale socket
|
||||||
|
/// (fallback, always reachable when the tailnet is up). At least one must be set.
|
||||||
|
#[derive(Debug, Clone, Deserialize, Serialize, PartialEq)]
|
||||||
|
pub struct PeerEntry {
|
||||||
|
/// Stable human-facing name (e.g. `"architect"`). Must be unique in the config.
|
||||||
|
pub name: String,
|
||||||
|
/// Zone tag — one of `"fabric-10g"`, `"lan-1g"`, `"roaming"`, or a custom label.
|
||||||
|
pub zone: String,
|
||||||
|
/// LAN socket. Omit when this peer has no LAN presence from our perspective
|
||||||
|
/// (e.g. a roaming laptop reachable only via Tailscale).
|
||||||
|
#[serde(default)]
|
||||||
|
pub lan_addr: Option<SocketAddr>,
|
||||||
|
/// Tailscale socket. Omit only if the peer is LAN-only.
|
||||||
|
#[serde(default)]
|
||||||
|
pub tailscale_addr: Option<SocketAddr>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl PeerEntry {
|
||||||
|
/// Sanity check: at least one address must be reachable.
|
||||||
|
pub fn validate(&self) -> Result<()> {
|
||||||
|
if self.name.is_empty() {
|
||||||
|
bail!("peer entry with empty name");
|
||||||
|
}
|
||||||
|
if self.zone.is_empty() {
|
||||||
|
bail!("peer {} has empty zone", self.name);
|
||||||
|
}
|
||||||
|
if self.lan_addr.is_none() && self.tailscale_addr.is_none() {
|
||||||
|
bail!(
|
||||||
|
"peer {} has no reachable address (both lan_addr and tailscale_addr unset)",
|
||||||
|
self.name
|
||||||
|
);
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Cluster membership configuration. Optional at the top level so existing
|
||||||
|
/// single-node deployments (pre-v2) keep loading. Once present, describes the
|
||||||
|
/// local node's zone + bind addresses, and enumerates known peers.
|
||||||
|
#[derive(Debug, Clone, Deserialize, Serialize, PartialEq)]
|
||||||
|
pub struct ClusterConfig {
|
||||||
|
/// This node's zone tag.
|
||||||
|
pub zone: String,
|
||||||
|
/// LAN listen socket (typically `0.0.0.0:7701`). Omit on roaming nodes.
|
||||||
|
#[serde(default)]
|
||||||
|
pub bind_lan: Option<SocketAddr>,
|
||||||
|
/// Tailscale listen socket (Tailscale IP + port). Omit on strictly-LAN nodes.
|
||||||
|
#[serde(default)]
|
||||||
|
pub bind_tailscale: Option<SocketAddr>,
|
||||||
|
/// Static seed list of peers. Runtime membership (Phase 1b) will extend this
|
||||||
|
/// via gossip; the config list bootstraps discovery.
|
||||||
|
#[serde(default)]
|
||||||
|
pub peers: Vec<PeerEntry>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ClusterConfig {
|
||||||
|
/// Sanity check the cluster config: at least one bind address, no duplicate
|
||||||
|
/// peer names, every peer has at least one address.
|
||||||
|
pub fn validate(&self) -> Result<()> {
|
||||||
|
if self.zone.is_empty() {
|
||||||
|
bail!("cluster.zone is empty");
|
||||||
|
}
|
||||||
|
if self.bind_lan.is_none() && self.bind_tailscale.is_none() {
|
||||||
|
bail!("cluster has no bind address (both bind_lan and bind_tailscale unset)");
|
||||||
|
}
|
||||||
|
let mut names = HashSet::new();
|
||||||
|
for peer in &self.peers {
|
||||||
|
peer.validate()?;
|
||||||
|
if !names.insert(peer.name.as_str()) {
|
||||||
|
bail!("duplicate peer name in cluster.peers: {}", peer.name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Look up a peer by name.
|
||||||
|
pub fn peer(&self, name: &str) -> Option<&PeerEntry> {
|
||||||
|
self.peers.iter().find(|p| p.name == name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
#[derive(Debug, Clone, Deserialize, Serialize)]
|
#[derive(Debug, Clone, Deserialize, Serialize)]
|
||||||
pub struct Config {
|
pub struct Config {
|
||||||
pub node: NodeConfig,
|
pub node: NodeConfig,
|
||||||
@@ -59,6 +143,10 @@ pub struct Config {
|
|||||||
pub cold: Option<ColdConfig>,
|
pub cold: Option<ColdConfig>,
|
||||||
pub replication: Option<ReplicationConfig>,
|
pub replication: Option<ReplicationConfig>,
|
||||||
pub peer: Option<PeerConfig>,
|
pub peer: Option<PeerConfig>,
|
||||||
|
/// v2 cluster membership (peers, zones, bind addresses). Optional so pre-v2
|
||||||
|
/// configs still load; once populated, `Config::load` calls `validate()`.
|
||||||
|
#[serde(default)]
|
||||||
|
pub cluster: Option<ClusterConfig>,
|
||||||
/// Optional Bearer token required on all HTTP POST endpoints.
|
/// Optional Bearer token required on all HTTP POST endpoints.
|
||||||
/// Set to a long random string, e.g. `openssl rand -hex 32`.
|
/// Set to a long random string, e.g. `openssl rand -hex 32`.
|
||||||
/// If absent, POST endpoints are unauthenticated (internal-network use only).
|
/// If absent, POST endpoints are unauthenticated (internal-network use only).
|
||||||
@@ -70,8 +158,14 @@ impl Config {
|
|||||||
pub fn load(path: &Path) -> Result<Self> {
|
pub fn load(path: &Path) -> Result<Self> {
|
||||||
let content = std::fs::read_to_string(path)
|
let content = std::fs::read_to_string(path)
|
||||||
.with_context(|| format!("reading config at {}", path.display()))?;
|
.with_context(|| format!("reading config at {}", path.display()))?;
|
||||||
toml::from_str(&content)
|
let cfg: Config = toml::from_str(&content)
|
||||||
.with_context(|| format!("parsing config at {}", path.display()))
|
.with_context(|| format!("parsing config at {}", path.display()))?;
|
||||||
|
if let Some(cluster) = cfg.cluster.as_ref() {
|
||||||
|
cluster
|
||||||
|
.validate()
|
||||||
|
.with_context(|| format!("validating cluster config in {}", path.display()))?;
|
||||||
|
}
|
||||||
|
Ok(cfg)
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn default_path() -> PathBuf {
|
pub fn default_path() -> PathBuf {
|
||||||
@@ -119,6 +213,187 @@ peer_user = "osobh"
|
|||||||
assert_eq!(cfg.cold.unwrap().retain_weeks, 12);
|
assert_eq!(cfg.cold.unwrap().retain_weeks, 12);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_pre_v2_config_still_loads_without_cluster_section() {
|
||||||
|
// Backwards compatibility: any config that worked before v2 must still
|
||||||
|
// parse cleanly with `cluster` absent.
|
||||||
|
let toml = r#"
|
||||||
|
[node]
|
||||||
|
name = "tank"
|
||||||
|
role = "secondary"
|
||||||
|
|
||||||
|
[hot]
|
||||||
|
path = "/hot/targets"
|
||||||
|
max_gb = 200
|
||||||
|
stale_hours = 48
|
||||||
|
|
||||||
|
[warm]
|
||||||
|
projects_path = "/slab/projects"
|
||||||
|
zfs_dataset = "slab/projects"
|
||||||
|
snapshot_retain_hours = 24
|
||||||
|
snapshot_retain_days = 7
|
||||||
|
snapshot_retain_weeks = 4
|
||||||
|
"#;
|
||||||
|
let cfg: Config = toml::from_str(toml).unwrap();
|
||||||
|
assert!(cfg.cluster.is_none(), "cluster must be optional");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_cluster_config_parses_with_peers() {
|
||||||
|
let toml = r#"
|
||||||
|
[node]
|
||||||
|
name = "tank"
|
||||||
|
role = "secondary"
|
||||||
|
|
||||||
|
[hot]
|
||||||
|
path = "/hot/targets"
|
||||||
|
max_gb = 200
|
||||||
|
stale_hours = 48
|
||||||
|
|
||||||
|
[warm]
|
||||||
|
projects_path = "/slab/projects"
|
||||||
|
zfs_dataset = "slab/projects"
|
||||||
|
snapshot_retain_hours = 24
|
||||||
|
snapshot_retain_days = 7
|
||||||
|
snapshot_retain_weeks = 4
|
||||||
|
|
||||||
|
[cluster]
|
||||||
|
zone = "fabric-10g"
|
||||||
|
bind_lan = "10.0.0.14:7701"
|
||||||
|
bind_tailscale = "100.64.1.2:7701"
|
||||||
|
|
||||||
|
[[cluster.peers]]
|
||||||
|
name = "architect"
|
||||||
|
zone = "fabric-10g"
|
||||||
|
lan_addr = "10.0.0.13:7701"
|
||||||
|
tailscale_addr = "100.64.1.3:7701"
|
||||||
|
|
||||||
|
[[cluster.peers]]
|
||||||
|
name = "morpheus"
|
||||||
|
zone = "lan-1g"
|
||||||
|
lan_addr = "192.168.1.50:7701"
|
||||||
|
tailscale_addr = "100.64.1.4:7701"
|
||||||
|
|
||||||
|
[[cluster.peers]]
|
||||||
|
name = "laptop"
|
||||||
|
zone = "roaming"
|
||||||
|
tailscale_addr = "100.64.1.5:7701"
|
||||||
|
"#;
|
||||||
|
let cfg: Config = toml::from_str(toml).unwrap();
|
||||||
|
let cluster = cfg.cluster.expect("cluster section present");
|
||||||
|
assert_eq!(cluster.zone, "fabric-10g");
|
||||||
|
assert_eq!(cluster.peers.len(), 3);
|
||||||
|
|
||||||
|
let architect = cluster.peer("architect").expect("architect peer");
|
||||||
|
assert_eq!(architect.zone, "fabric-10g");
|
||||||
|
assert_eq!(
|
||||||
|
architect.lan_addr.unwrap(),
|
||||||
|
"10.0.0.13:7701".parse::<SocketAddr>().unwrap()
|
||||||
|
);
|
||||||
|
|
||||||
|
let laptop = cluster.peer("laptop").expect("laptop peer");
|
||||||
|
assert!(
|
||||||
|
laptop.lan_addr.is_none(),
|
||||||
|
"roaming peer has no LAN address"
|
||||||
|
);
|
||||||
|
assert!(laptop.tailscale_addr.is_some());
|
||||||
|
|
||||||
|
cluster.validate().expect("valid cluster config");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_peer_without_addresses_fails_validation() {
|
||||||
|
let peer = PeerEntry {
|
||||||
|
name: "ghost".into(),
|
||||||
|
zone: "fabric-10g".into(),
|
||||||
|
lan_addr: None,
|
||||||
|
tailscale_addr: None,
|
||||||
|
};
|
||||||
|
let err = peer.validate().unwrap_err().to_string();
|
||||||
|
assert!(
|
||||||
|
err.contains("ghost") && err.contains("no reachable address"),
|
||||||
|
"unexpected error: {err}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_cluster_without_bind_addresses_fails_validation() {
|
||||||
|
let cluster = ClusterConfig {
|
||||||
|
zone: "fabric-10g".into(),
|
||||||
|
bind_lan: None,
|
||||||
|
bind_tailscale: None,
|
||||||
|
peers: vec![],
|
||||||
|
};
|
||||||
|
let err = cluster.validate().unwrap_err().to_string();
|
||||||
|
assert!(
|
||||||
|
err.contains("no bind address"),
|
||||||
|
"unexpected error: {err}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_cluster_duplicate_peer_names_fails_validation() {
|
||||||
|
let cluster = ClusterConfig {
|
||||||
|
zone: "fabric-10g".into(),
|
||||||
|
bind_lan: Some("10.0.0.14:7701".parse().unwrap()),
|
||||||
|
bind_tailscale: None,
|
||||||
|
peers: vec![
|
||||||
|
PeerEntry {
|
||||||
|
name: "architect".into(),
|
||||||
|
zone: "fabric-10g".into(),
|
||||||
|
lan_addr: Some("10.0.0.13:7701".parse().unwrap()),
|
||||||
|
tailscale_addr: None,
|
||||||
|
},
|
||||||
|
PeerEntry {
|
||||||
|
name: "architect".into(),
|
||||||
|
zone: "fabric-10g".into(),
|
||||||
|
lan_addr: Some("10.0.0.14:7701".parse().unwrap()),
|
||||||
|
tailscale_addr: None,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
|
let err = cluster.validate().unwrap_err().to_string();
|
||||||
|
assert!(
|
||||||
|
err.contains("duplicate peer name"),
|
||||||
|
"unexpected error: {err}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_cluster_config_load_invokes_validate() {
|
||||||
|
// Same shape as a real config file but written to a tempfile so the
|
||||||
|
// full path — read, parse, validate — is exercised.
|
||||||
|
let toml = r#"
|
||||||
|
[node]
|
||||||
|
name = "tank"
|
||||||
|
role = "secondary"
|
||||||
|
|
||||||
|
[hot]
|
||||||
|
path = "/hot/targets"
|
||||||
|
max_gb = 200
|
||||||
|
stale_hours = 48
|
||||||
|
|
||||||
|
[warm]
|
||||||
|
projects_path = "/slab/projects"
|
||||||
|
zfs_dataset = "slab/projects"
|
||||||
|
snapshot_retain_hours = 24
|
||||||
|
snapshot_retain_days = 7
|
||||||
|
snapshot_retain_weeks = 4
|
||||||
|
|
||||||
|
[cluster]
|
||||||
|
zone = "fabric-10g"
|
||||||
|
|
||||||
|
[[cluster.peers]]
|
||||||
|
name = "architect"
|
||||||
|
zone = "fabric-10g"
|
||||||
|
"#;
|
||||||
|
let tmp = tempfile::NamedTempFile::new().unwrap();
|
||||||
|
std::fs::write(tmp.path(), toml).unwrap();
|
||||||
|
let err = Config::load(tmp.path()).unwrap_err().to_string();
|
||||||
|
// Fails because cluster has no bind address AND peer has no reachable address.
|
||||||
|
assert!(err.contains("validating cluster config"), "err: {err}");
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_load_secondary_config() {
|
fn test_load_secondary_config() {
|
||||||
let toml = r#"
|
let toml = r#"
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
mod actions;
|
mod actions;
|
||||||
mod cargo_init;
|
mod cargo_init;
|
||||||
|
mod cluster;
|
||||||
mod config;
|
mod config;
|
||||||
mod daemon;
|
mod daemon;
|
||||||
mod head_watch;
|
mod head_watch;
|
||||||
@@ -68,6 +69,11 @@ enum Cmd {
|
|||||||
Pin { project: String },
|
Pin { project: String },
|
||||||
/// Unmark a project — it's now a normal GC candidate again
|
/// Unmark a project — it's now a normal GC candidate again
|
||||||
Unpin { project: String },
|
Unpin { project: String },
|
||||||
|
/// Probe a cluster peer's LAN and Tailscale endpoints; print the winning route
|
||||||
|
ClusterProbe {
|
||||||
|
/// Peer name as listed in `[[cluster.peers]]` in the config
|
||||||
|
peer: String,
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::main]
|
#[tokio::main]
|
||||||
@@ -102,10 +108,31 @@ async fn main() -> Result<()> {
|
|||||||
serve::run_server(cfg, manifest, port, static_dir).await?,
|
serve::run_server(cfg, manifest, port, static_dir).await?,
|
||||||
Cmd::Pin { project } => cmd_set_pin(&manifest_path, &project, true)?,
|
Cmd::Pin { project } => cmd_set_pin(&manifest_path, &project, true)?,
|
||||||
Cmd::Unpin { project } => cmd_set_pin(&manifest_path, &project, false)?,
|
Cmd::Unpin { project } => cmd_set_pin(&manifest_path, &project, false)?,
|
||||||
|
Cmd::ClusterProbe { peer } => cmd_cluster_probe(&cfg, &peer).await?,
|
||||||
}
|
}
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── cluster probe ────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
async fn cmd_cluster_probe(cfg: &Config, peer_name: &str) -> Result<()> {
|
||||||
|
let cluster_cfg = cfg
|
||||||
|
.cluster
|
||||||
|
.as_ref()
|
||||||
|
.context("no [cluster] section in config; cannot probe peers")?;
|
||||||
|
let peer = cluster_cfg
|
||||||
|
.peer(peer_name)
|
||||||
|
.with_context(|| format!("peer {peer_name:?} not found in cluster.peers"))?;
|
||||||
|
let probe = cluster::LanFirstProbe::new();
|
||||||
|
let winner = probe.probe(peer).await?;
|
||||||
|
println!("peer: {}", peer.name);
|
||||||
|
println!("zone: {}", peer.zone);
|
||||||
|
println!("route: {}", winner.kind.as_str());
|
||||||
|
println!("addr: {}", winner.addr);
|
||||||
|
println!("elapsed: {:?}", winner.elapsed);
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
// ── pin / unpin ──────────────────────────────────────────────────────────────
|
// ── pin / unpin ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
fn cmd_set_pin(manifest_path: &std::path::Path, project: &str, pinned: bool) -> Result<()> {
|
fn cmd_set_pin(manifest_path: &std::path::Path, project: &str, pinned: bool) -> Result<()> {
|
||||||
|
|||||||
Reference in New Issue
Block a user