Phase 1a: cluster module + LAN-first peer probe

First cut of the v2 distributed FS. Captures the architecture design
in ARCHITECTURE-v2.md and lands the smallest useful new capability:
probing cluster peers with a LAN-first policy so subsequent transport
+ gossip layers (Phase 1b, 1c) can build on a real routing decision.

New:
- ARCHITECTURE-v2.md: zones (fabric-10g/lan-1g/roaming), tier
  lifecycle (hot/warm/cold), fingerprint-keyed build cache design,
  smart-clean policy, phase plan, explicit non-goals.
- claw-store/src/cluster.rs: RouteKind, RouteWinner, LanFirstProbe.
  LAN 200ms timeout, Tailscale 500ms fallback. 8 tests use real
  TCP listeners on 127.0.0.1 (no mocks); cover happy path,
  fall-through, both-fail, single-address, and elapsed reporting.
- claw-store/src/config.rs: ClusterConfig + PeerEntry with
  validation (bind-address presence, no duplicate peer names,
  per-peer reachable address required). Optional at top level so
  pre-v2 configs still load unchanged. 6 new tests.
- claw-store/src/main.rs: `claw-store cluster-probe <peer>` CLI
  subcommand that reads config, resolves the peer, probes, prints
  the winning route + elapsed time.

All 16 new tests pass. Existing 45 pass. Sole failure
(hot::tests::test_project_target_size_bytes) is a pre-existing
macOS-only issue with `du -sb`; Linux CI unaffected.

Follow-on Phase 1 cuts (subsequent sessions):
- 1b: chitchat SWIM gossip for live membership state
- 1c: quinn QUIC transport with fleet-CA mTLS
- 1d: `claw-store cluster status` — live membership view

Every file well under the 1300-line ceiling
(cluster.rs 268, config.rs 428, main.rs 450).
This commit is contained in:
Omar Sobh
2026-07-11 21:30:50 -07:00
parent aefa1cce58
commit 5c19c60292
6 changed files with 828 additions and 4 deletions
+253
View File
@@ -0,0 +1,253 @@
# clawstor v2 — Distributed Filesystem Architecture
Purpose-built distributed FS for a small fleet (310 nodes, single trust domain).
Not a Ceph-scale general-purpose system: opinionated, workload-specific, self-hosted.
## Non-goals
Explicit choices we do NOT make:
- **Not POSIX-strict.** No byte-range locks, `mmap` semantics, `flock`. FUSE mount is
read-mostly for git worktrees; cargo target dirs never traverse it.
- **Not erasure coding.** Simple replication (`N=2` warm, `N=3` for pinned-critical).
- **Not encryption at our layer.** ZFS native encryption at rest; Tailscale (WireGuard)
+ QUIC TLS 1.3 in transit. Adding a third layer is theater.
- **Not multi-tenant.** Single trust domain. Auth piggybacks on Tailscale identity for
roaming clients; fleet-CA-signed mTLS for LAN peers.
- **Not our own consensus algorithm.** Vendor `openraft` (only where strong consistency is
required) or `chitchat` (SWIM gossip for eventually-consistent state).
- **Not massive scale.** 310 nodes. Membership can be gossip; placement can be config.
## Workload shapes we serve
Five, and no others:
1. Source trees / git worktrees (~10100 MB, moderate churn)
2. Cargo build artifacts (deps trees, 115 GB compressed, content-addressed)
3. ZFS state / snapshots (large, sequential)
4. ML model weights / cold blobs (multi-GB, WORM)
5. Small config/state (tiny, frequent reads, low writes)
## Fleet topology
Three zones defined by physical network:
| Zone | Members (typical) | Link | Purpose |
|---|---|---|---|
| `fabric-10g` | tank, architect | 10G direct-connect | primary + warm-tier replication |
| `lan-1g` | morpheus (+ any 1G LAN node) | 1G switched | build node, warm-tier consumer |
| `roaming` | laptop, remote nodes | Tailscale WireGuard | read/write client with offline queue |
Every node has both a LAN IP (or none, for roaming) and a Tailscale IP. Peer connections
probe LAN first (200ms timeout) with Tailscale as fallback. Direct LAN sockets skip
WireGuard framing entirely — real perf win on the 10G fabric.
## Data tiers
```
┌───────┐ activate ┌────────┐ evict ┌───────┐ age-out ┌──────┐
│ new │ ───────────▶ │ HOT │ ─────────▶ │ WARM │ ──────────▶ │ COLD │
│(miss) │ │(local │ │(fleet │ │(arch │
│ │ ◀── build ── │target) │ ◀── get ── │blob) │ ◀─ promote │ zfs) │
└───────┘ └────────┘ └───────┘ └──────┘
```
- **Hot** — local NVMe cargo target dir. LRU + stale-hours eviction. Symlinked into the
workspace on `activate`. Never accessed remotely.
- **Warm** — content-addressed chunked blobs in `/slab/build-cache/by-hash/<fingerprint>/`
on ZFS-owning nodes. Placement per zone rules; replicated within `fabric-10g`.
- **Cold** — same content-addressed layout but zstd-19 compressed, single copy on
architect's `data` pool (7.27 TB). Rehydrates to warm on demand.
## The killer feature: fingerprint-keyed build cache
Cargo target dirs are captured post-build and stored fleet-wide, keyed by:
```
BLAKE3(
Cargo.lock
rust-toolchain.toml (if present)
.cargo/config.toml (if present)
active_rustc_version + host_triple + target_triple
RUSTFLAGS (or CARGO_ENCODED_RUSTFLAGS)
enabled_features (sorted)
profile ("dev" / "release")
CC / CXX versions (crates with build.rs)
glibc_version (Linux binary compat)
)
```
Captured subset: `deps/ + .fingerprint/ + build/ + examples/ + small top-level`. Explicitly
NOT captured: `incremental/` (not portable across machines; rebuilt locally in ~1 min).
Chunk-level dedup via BLAKE3 4 MB chunks — two fingerprints sharing 95% of files share 95%
of storage.
Client integration is a thin `claw-cargo build` wrapper that:
1. Computes fingerprint.
2. `cache-check` → hit: streams chunks, extracts; miss: continues.
3. Runs `cargo build`.
4. `cache-put` on success.
The workspace-level `.cargo/config.toml` gets an `[alias] clean = "!claw-store clean --soft"`
so `cargo clean` invokes the smart-clean layer transparently.
## Smart clean — four modes
- `--incremental-only` — nuke `incremental/`, keep everything else. ~20-30% reclaim.
- `--soft` — remove local hot target dir, keep fleet blob. Default GC action.
- `--medium` — soft + demote fleet blob to cold if unreferenced + 14d old.
- `--hard` — full purge. Requires `--force` when live refs exist.
Reference tracking: every `cache-put` records the git refs that produced it. Nightly sweep
queries Gitea `/api/v1/repos/.../branches` and `/tags`. Fingerprints with zero live refs
+ age > retention → eligible for deletion.
## Membership + transport (Phase 1)
- **Membership** via `chitchat` SWIM gossip. 3-node cluster; heartbeat state, zones,
hot-tier occupancy.
- **Transport** via `quinn` (QUIC/TLS 1.3). LAN-first probe: 200 ms LAN attempt, then
Tailscale fallback. Winning route cached per peer per session; re-probe every 5 min
or on transport error.
- **Identity**: fleet CA signs peer mTLS certs on LAN. Tailscale-node-identity certs used
for roaming clients.
## Consistency model
- **Content-addressed blobs** — immutable, self-verifying. No consistency problem.
- **Metadata pointers** (HEAD, tag mappings, pin lists) — vector clocks + last-writer-wins
with human-readable warning on conflict. Small fleet + typically single-user means
conflicts are rare.
- **CRDT for pin sets** — grow-only set with tombstones for unpin.
- **Strong consistency** only where required (e.g. leader election for the eviction
coordinator) via a tiny `openraft` group across `fabric-10g` nodes only.
## Roaming client (full R/W, offline queue)
Same daemon binary, same primitives, adds:
- **Local WAL** (redb) — mutating ops appended before network. Crash-safe.
- **Bandwidth-aware push loop** — throttles to protect tether/coffee-shop wifi.
- **Reconnect handshake** — pull metadata delta, reconcile, push blobs (dedup automatic),
push metadata updates.
- **Pin classes** — `--local` (full offline), `--metadata-only`, `--hot` (with `--stale`).
- **LRU + budget** (`--max-local-gb=100`) — prevents disk-fill.
## FUSE mount (Phase 6)
Single mount point (`/shared/projects/` with per-user `~/projects` symlink) exposing
warm-tier git worktrees fleet-wide. Read-mostly, small-file-friendly. Cargo target dirs
NEVER traverse this mount — they use the CLI-managed hot tier on local NVMe.
## CI runner integration
Runners advertise per-repo affinity as Gitea labels:
```yaml
labels:
- "ubuntu-latest:host"
- "clawstor:tank"
- "clawstor:has:osobh/zeroclaw,osobh/clawmates"
```
Workflows request affinity via `runs-on: [ubuntu-latest, "clawstor:has:${{ github.repository }}"]`.
`claw-store daemon` rewrites its labels dynamically as warm-tier state changes.
Two Gitea Actions replace stock steps:
- `clawstor-checkout@v1` — bind-mount from mount instead of git clone.
- `clawstor-cargo-cache@v1` — fingerprint-keyed cache-check / cache-put.
## Phase plan
| Phase | Chunk | Weeks |
|---|---|---|
| 1 | Membership + LAN-first transport (chitchat + quinn) | 2 |
| 2 | Content-addressed blob store (BLAKE3 chunking, put/get) | 2 |
| 3 | Namespace + metadata + vector clocks + CRDTs | 3 |
| 4 | Client mode + WAL + reconnect + pin semantics | 3 |
| 5 | Cargo build-artifact cache integration | 3 |
| 6 | FUSE mount for git worktrees | 2 |
| 7 | Snapshot + repair + scrub + smart-clean + reference tracking | 3-4 |
| 8 | Fleet CA + Tailscale identity for roaming | 1 |
Roughly **18-20 weeks focused work**, ~5-7 months elapsed with other priorities.
## Placement policy
Rule-based, config-driven. Defaults sensible:
```toml
[[placement.rules]]
match = "size > 1GB and hot"
primary = "fabric-10g-any"
replicas = 2
[[placement.rules]]
match = "type = cold"
primary = "architect"
replicas = 1
[[placement.rules]]
default = true
primary = "tank"
replicas = ["architect"]
[placement.override."osobh/clawverse"]
primary = "architect"
replicas = ["tank"]
```
Read prioritization (fixed order, no config):
1. Local cache
2. Local warm tier
3. `fabric-10g` peer
4. `lan-1g` peer
5. `roaming` peer
Write path: local WAL → primary sync-replicate → replicas async-replicate.
## Cleanup policy
```toml
[cleanup.hot]
default_max_age_hours = 168
default_max_total_gb = 500
default_min_free_gb = 100
soft_evict_if_fleet_cached = true
[cleanup.warm]
retain_referenced_days = 90
retain_unreferenced_days = 14
demote_to_cold_after_days = 30
min_replicas_fabric_10g = 2
[cleanup.cold]
retain_referenced_days = 365
retain_unreferenced_days = 60
[cleanup.triggers]
run_gc_nightly_at = "03:00"
run_gc_on_disk_pressure = true
disk_pressure_threshold_pct = 85
```
## Delivery constraints (per project rules)
- Strict TDD — tests before code, tests must fail before implementation.
- Every file under 1300 lines.
- No mocks, stubs, or TODOs — full implementations only.
- Real sockets in tests where network behavior matters.
## What Phase 1a delivers (this cut)
- `PeerAddress`, `Zone`, `NodeIdentity` types.
- `LanFirstProbe` — races LAN endpoint then Tailscale, returns winning route.
- `ClusterConfig` — optional config section listing peers with their two addresses.
- CLI: `claw-store cluster probe <peer-name>` — visible sanity check.
- Real TCP-listener tests, no mocks.
Follow-on Phase 1 cuts (subsequent sessions):
- 1b: chitchat gossip integration + peer state broadcast.
- 1c: `quinn` QUIC transport with fleet-CA mTLS.
- 1d: `claw-store cluster status` — live membership view.
Generated
+1 -1
View File
@@ -256,7 +256,7 @@ checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9"
[[package]] [[package]]
name = "claw-store" name = "claw-store"
version = "0.2.0" version = "0.3.0"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"axum", "axum",
+1
View File
@@ -126,6 +126,7 @@ mod tests {
cold: None, cold: None,
replication: None, replication: None,
peer: None, peer: None,
cluster: None,
api_token: None, api_token: None,
} }
} }
+268
View File
@@ -0,0 +1,268 @@
//! Cluster peer routing.
//!
//! Phase 1a of the v2 distributed FS. Every fleet node has two potential
//! endpoints: a LAN socket (fast, direct, no WireGuard framing) and a
//! Tailscale socket (fallback, always reachable when the tailnet is up).
//! This module races LAN first and falls through to Tailscale, returning
//! the winning route so the caller can cache it for the session.
//!
//! Cache the returned [`RouteWinner`] and re-probe on transport error or
//! after ~5 min so route changes (node moved networks, LAN NIC came back)
//! propagate without a daemon restart.
use crate::config::PeerEntry;
use anyhow::{bail, Result};
use std::net::SocketAddr;
use std::time::{Duration, Instant};
use tokio::net::TcpStream;
use tokio::time::timeout;
/// Which network route reached a peer.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum RouteKind {
/// Direct LAN socket. No WireGuard overhead.
Lan,
/// Tailscale-provided socket. WireGuard-encapsulated.
Tailscale,
}
impl RouteKind {
/// Short lowercase label suitable for logs, metrics, config.
pub fn as_str(self) -> &'static str {
match self {
RouteKind::Lan => "lan",
RouteKind::Tailscale => "tailscale",
}
}
}
/// A successful probe result: which address answered, which route kind it was,
/// and how long the probe took from start to first-success (useful for logs).
#[derive(Debug, Clone)]
pub struct RouteWinner {
pub addr: SocketAddr,
pub kind: RouteKind,
pub elapsed: Duration,
}
/// LAN-first probe.
///
/// Attempts the peer's LAN address with a tight timeout; on any error or
/// timeout, falls through to the Tailscale address with a longer timeout
/// (Tailscale may need to set up a direct WireGuard tunnel on first probe).
/// Errors only when neither route succeeds.
pub struct LanFirstProbe {
lan_timeout: Duration,
tailscale_timeout: Duration,
}
impl Default for LanFirstProbe {
/// LAN 200ms, Tailscale 500ms. Rationale: on a healthy LAN the TCP SYN
/// round-trip is sub-millisecond; anything slower than 200ms means the
/// LAN route is not usable and we should fall through immediately.
/// Tailscale needs headroom for WireGuard handshakes and DERP fallback.
fn default() -> Self {
Self {
lan_timeout: Duration::from_millis(200),
tailscale_timeout: Duration::from_millis(500),
}
}
}
impl LanFirstProbe {
/// Probe using default timeouts (200ms LAN, 500ms Tailscale).
pub fn new() -> Self {
Self::default()
}
/// Probe with custom timeouts. Useful for tests and for tuning per
/// deployment shape.
pub fn with_timeouts(lan: Duration, tailscale: Duration) -> Self {
Self {
lan_timeout: lan,
tailscale_timeout: tailscale,
}
}
/// Probe a peer. Returns the first route (LAN preferred) that accepts a
/// TCP connection within its per-route timeout. Errors when neither
/// route succeeds, or when the peer has no addresses configured.
pub async fn probe(&self, peer: &PeerEntry) -> Result<RouteWinner> {
let start = Instant::now();
if let Some(lan) = peer.lan_addr {
if try_connect(lan, self.lan_timeout).await.is_ok() {
return Ok(RouteWinner {
addr: lan,
kind: RouteKind::Lan,
elapsed: start.elapsed(),
});
}
}
if let Some(ts) = peer.tailscale_addr {
if try_connect(ts, self.tailscale_timeout).await.is_ok() {
return Ok(RouteWinner {
addr: ts,
kind: RouteKind::Tailscale,
elapsed: start.elapsed(),
});
}
}
bail!(
"peer {} has no reachable address (lan={:?}, tailscale={:?})",
peer.name,
peer.lan_addr,
peer.tailscale_addr,
);
}
}
/// Attempt a TCP connect with a deadline. Returns Ok on successful handshake,
/// Err on connection error or timeout.
async fn try_connect(addr: SocketAddr, deadline: Duration) -> Result<()> {
match timeout(deadline, TcpStream::connect(addr)).await {
Ok(Ok(_stream)) => Ok(()),
Ok(Err(e)) => Err(anyhow::Error::from(e)),
Err(_) => bail!("connect to {} timed out after {:?}", addr, deadline),
}
}
#[cfg(test)]
mod tests {
use super::*;
use tokio::net::TcpListener;
/// Bind a TCP listener on an OS-assigned port, then drop it. The returned
/// address points at a port that's very likely unbound. Tiny race window
/// between drop and the caller's probe (microseconds); in practice tests
/// pass reliably. If ever flaky, we upgrade to a strict SO_LINGER=0 trick.
async fn free_but_unbound_addr() -> SocketAddr {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
drop(listener);
addr
}
/// Bind a listener and spawn an accept loop so it stays alive across probes.
/// Returns the listener's local address.
async fn spawn_accepting_listener() -> SocketAddr {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
loop {
if listener.accept().await.is_err() {
return;
}
}
});
addr
}
fn peer_with(lan: Option<SocketAddr>, ts: Option<SocketAddr>) -> PeerEntry {
PeerEntry {
name: "test-peer".into(),
zone: "test-zone".into(),
lan_addr: lan,
tailscale_addr: ts,
}
}
#[tokio::test]
async fn probe_prefers_lan_when_available() {
let lan = spawn_accepting_listener().await;
// Tailscale unreachable — must not be attempted when LAN succeeds.
let ts = free_but_unbound_addr().await;
let probe = LanFirstProbe::new();
let win = probe.probe(&peer_with(Some(lan), Some(ts))).await.unwrap();
assert_eq!(win.kind, RouteKind::Lan);
assert_eq!(win.addr, lan);
}
#[tokio::test]
async fn probe_falls_through_to_tailscale_when_lan_fails() {
let lan_dead = free_but_unbound_addr().await;
let ts = spawn_accepting_listener().await;
// Short LAN timeout so the test runs fast even if the OS holds the
// failed connect in a queue rather than refusing immediately.
let probe = LanFirstProbe::with_timeouts(
Duration::from_millis(50),
Duration::from_millis(500),
);
let win = probe
.probe(&peer_with(Some(lan_dead), Some(ts)))
.await
.unwrap();
assert_eq!(win.kind, RouteKind::Tailscale);
assert_eq!(win.addr, ts);
}
#[tokio::test]
async fn probe_errors_when_both_routes_fail() {
let lan_dead = free_but_unbound_addr().await;
let ts_dead = free_but_unbound_addr().await;
let probe = LanFirstProbe::with_timeouts(
Duration::from_millis(50),
Duration::from_millis(50),
);
let err = probe
.probe(&peer_with(Some(lan_dead), Some(ts_dead)))
.await
.unwrap_err()
.to_string();
assert!(
err.contains("no reachable address"),
"unexpected error: {err}"
);
}
#[tokio::test]
async fn probe_uses_only_tailscale_when_lan_absent() {
let ts = spawn_accepting_listener().await;
let probe = LanFirstProbe::new();
let win = probe.probe(&peer_with(None, Some(ts))).await.unwrap();
assert_eq!(win.kind, RouteKind::Tailscale);
assert_eq!(win.addr, ts);
}
#[tokio::test]
async fn probe_uses_only_lan_when_tailscale_absent() {
let lan = spawn_accepting_listener().await;
let probe = LanFirstProbe::new();
let win = probe.probe(&peer_with(Some(lan), None)).await.unwrap();
assert_eq!(win.kind, RouteKind::Lan);
assert_eq!(win.addr, lan);
}
#[tokio::test]
async fn probe_errors_when_no_addresses_configured() {
// PeerEntry::validate would reject this at config-load time, but the
// probe must still handle it gracefully (defense in depth).
let probe = LanFirstProbe::new();
let err = probe
.probe(&peer_with(None, None))
.await
.unwrap_err()
.to_string();
assert!(
err.contains("no reachable address"),
"unexpected error: {err}"
);
}
#[tokio::test]
async fn probe_records_elapsed_time_under_timeout() {
let lan = spawn_accepting_listener().await;
let probe = LanFirstProbe::new();
let win = probe.probe(&peer_with(Some(lan), None)).await.unwrap();
assert!(
win.elapsed < Duration::from_millis(500),
"probe should complete quickly on a live local listener; took {:?}",
win.elapsed
);
}
#[test]
fn route_kind_as_str_matches_variants() {
assert_eq!(RouteKind::Lan.as_str(), "lan");
assert_eq!(RouteKind::Tailscale.as_str(), "tailscale");
}
}
+278 -3
View File
@@ -1,5 +1,7 @@
use anyhow::{Context, Result}; use anyhow::{bail, Context, Result};
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use std::collections::HashSet;
use std::net::SocketAddr;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
#[derive(Debug, Clone, Deserialize, Serialize, PartialEq)] #[derive(Debug, Clone, Deserialize, Serialize, PartialEq)]
@@ -51,6 +53,88 @@ pub struct PeerConfig {
pub user: String, pub user: String,
} }
/// One peer node the cluster knows about. Every peer carries up to two reachable
/// endpoints: a LAN socket (fast path, tried first) and a Tailscale socket
/// (fallback, always reachable when the tailnet is up). At least one must be set.
#[derive(Debug, Clone, Deserialize, Serialize, PartialEq)]
pub struct PeerEntry {
/// Stable human-facing name (e.g. `"architect"`). Must be unique in the config.
pub name: String,
/// Zone tag — one of `"fabric-10g"`, `"lan-1g"`, `"roaming"`, or a custom label.
pub zone: String,
/// LAN socket. Omit when this peer has no LAN presence from our perspective
/// (e.g. a roaming laptop reachable only via Tailscale).
#[serde(default)]
pub lan_addr: Option<SocketAddr>,
/// Tailscale socket. Omit only if the peer is LAN-only.
#[serde(default)]
pub tailscale_addr: Option<SocketAddr>,
}
impl PeerEntry {
/// Sanity check: at least one address must be reachable.
pub fn validate(&self) -> Result<()> {
if self.name.is_empty() {
bail!("peer entry with empty name");
}
if self.zone.is_empty() {
bail!("peer {} has empty zone", self.name);
}
if self.lan_addr.is_none() && self.tailscale_addr.is_none() {
bail!(
"peer {} has no reachable address (both lan_addr and tailscale_addr unset)",
self.name
);
}
Ok(())
}
}
/// Cluster membership configuration. Optional at the top level so existing
/// single-node deployments (pre-v2) keep loading. Once present, describes the
/// local node's zone + bind addresses, and enumerates known peers.
#[derive(Debug, Clone, Deserialize, Serialize, PartialEq)]
pub struct ClusterConfig {
/// This node's zone tag.
pub zone: String,
/// LAN listen socket (typically `0.0.0.0:7701`). Omit on roaming nodes.
#[serde(default)]
pub bind_lan: Option<SocketAddr>,
/// Tailscale listen socket (Tailscale IP + port). Omit on strictly-LAN nodes.
#[serde(default)]
pub bind_tailscale: Option<SocketAddr>,
/// Static seed list of peers. Runtime membership (Phase 1b) will extend this
/// via gossip; the config list bootstraps discovery.
#[serde(default)]
pub peers: Vec<PeerEntry>,
}
impl ClusterConfig {
/// Sanity check the cluster config: at least one bind address, no duplicate
/// peer names, every peer has at least one address.
pub fn validate(&self) -> Result<()> {
if self.zone.is_empty() {
bail!("cluster.zone is empty");
}
if self.bind_lan.is_none() && self.bind_tailscale.is_none() {
bail!("cluster has no bind address (both bind_lan and bind_tailscale unset)");
}
let mut names = HashSet::new();
for peer in &self.peers {
peer.validate()?;
if !names.insert(peer.name.as_str()) {
bail!("duplicate peer name in cluster.peers: {}", peer.name);
}
}
Ok(())
}
/// Look up a peer by name.
pub fn peer(&self, name: &str) -> Option<&PeerEntry> {
self.peers.iter().find(|p| p.name == name)
}
}
#[derive(Debug, Clone, Deserialize, Serialize)] #[derive(Debug, Clone, Deserialize, Serialize)]
pub struct Config { pub struct Config {
pub node: NodeConfig, pub node: NodeConfig,
@@ -59,6 +143,10 @@ pub struct Config {
pub cold: Option<ColdConfig>, pub cold: Option<ColdConfig>,
pub replication: Option<ReplicationConfig>, pub replication: Option<ReplicationConfig>,
pub peer: Option<PeerConfig>, pub peer: Option<PeerConfig>,
/// v2 cluster membership (peers, zones, bind addresses). Optional so pre-v2
/// configs still load; once populated, `Config::load` calls `validate()`.
#[serde(default)]
pub cluster: Option<ClusterConfig>,
/// Optional Bearer token required on all HTTP POST endpoints. /// Optional Bearer token required on all HTTP POST endpoints.
/// Set to a long random string, e.g. `openssl rand -hex 32`. /// Set to a long random string, e.g. `openssl rand -hex 32`.
/// If absent, POST endpoints are unauthenticated (internal-network use only). /// If absent, POST endpoints are unauthenticated (internal-network use only).
@@ -70,8 +158,14 @@ impl Config {
pub fn load(path: &Path) -> Result<Self> { pub fn load(path: &Path) -> Result<Self> {
let content = std::fs::read_to_string(path) let content = std::fs::read_to_string(path)
.with_context(|| format!("reading config at {}", path.display()))?; .with_context(|| format!("reading config at {}", path.display()))?;
toml::from_str(&content) let cfg: Config = toml::from_str(&content)
.with_context(|| format!("parsing config at {}", path.display())) .with_context(|| format!("parsing config at {}", path.display()))?;
if let Some(cluster) = cfg.cluster.as_ref() {
cluster
.validate()
.with_context(|| format!("validating cluster config in {}", path.display()))?;
}
Ok(cfg)
} }
pub fn default_path() -> PathBuf { pub fn default_path() -> PathBuf {
@@ -119,6 +213,187 @@ peer_user = "osobh"
assert_eq!(cfg.cold.unwrap().retain_weeks, 12); assert_eq!(cfg.cold.unwrap().retain_weeks, 12);
} }
#[test]
fn test_pre_v2_config_still_loads_without_cluster_section() {
// Backwards compatibility: any config that worked before v2 must still
// parse cleanly with `cluster` absent.
let toml = r#"
[node]
name = "tank"
role = "secondary"
[hot]
path = "/hot/targets"
max_gb = 200
stale_hours = 48
[warm]
projects_path = "/slab/projects"
zfs_dataset = "slab/projects"
snapshot_retain_hours = 24
snapshot_retain_days = 7
snapshot_retain_weeks = 4
"#;
let cfg: Config = toml::from_str(toml).unwrap();
assert!(cfg.cluster.is_none(), "cluster must be optional");
}
#[test]
fn test_cluster_config_parses_with_peers() {
let toml = r#"
[node]
name = "tank"
role = "secondary"
[hot]
path = "/hot/targets"
max_gb = 200
stale_hours = 48
[warm]
projects_path = "/slab/projects"
zfs_dataset = "slab/projects"
snapshot_retain_hours = 24
snapshot_retain_days = 7
snapshot_retain_weeks = 4
[cluster]
zone = "fabric-10g"
bind_lan = "10.0.0.14:7701"
bind_tailscale = "100.64.1.2:7701"
[[cluster.peers]]
name = "architect"
zone = "fabric-10g"
lan_addr = "10.0.0.13:7701"
tailscale_addr = "100.64.1.3:7701"
[[cluster.peers]]
name = "morpheus"
zone = "lan-1g"
lan_addr = "192.168.1.50:7701"
tailscale_addr = "100.64.1.4:7701"
[[cluster.peers]]
name = "laptop"
zone = "roaming"
tailscale_addr = "100.64.1.5:7701"
"#;
let cfg: Config = toml::from_str(toml).unwrap();
let cluster = cfg.cluster.expect("cluster section present");
assert_eq!(cluster.zone, "fabric-10g");
assert_eq!(cluster.peers.len(), 3);
let architect = cluster.peer("architect").expect("architect peer");
assert_eq!(architect.zone, "fabric-10g");
assert_eq!(
architect.lan_addr.unwrap(),
"10.0.0.13:7701".parse::<SocketAddr>().unwrap()
);
let laptop = cluster.peer("laptop").expect("laptop peer");
assert!(
laptop.lan_addr.is_none(),
"roaming peer has no LAN address"
);
assert!(laptop.tailscale_addr.is_some());
cluster.validate().expect("valid cluster config");
}
#[test]
fn test_peer_without_addresses_fails_validation() {
let peer = PeerEntry {
name: "ghost".into(),
zone: "fabric-10g".into(),
lan_addr: None,
tailscale_addr: None,
};
let err = peer.validate().unwrap_err().to_string();
assert!(
err.contains("ghost") && err.contains("no reachable address"),
"unexpected error: {err}"
);
}
#[test]
fn test_cluster_without_bind_addresses_fails_validation() {
let cluster = ClusterConfig {
zone: "fabric-10g".into(),
bind_lan: None,
bind_tailscale: None,
peers: vec![],
};
let err = cluster.validate().unwrap_err().to_string();
assert!(
err.contains("no bind address"),
"unexpected error: {err}"
);
}
#[test]
fn test_cluster_duplicate_peer_names_fails_validation() {
let cluster = ClusterConfig {
zone: "fabric-10g".into(),
bind_lan: Some("10.0.0.14:7701".parse().unwrap()),
bind_tailscale: None,
peers: vec![
PeerEntry {
name: "architect".into(),
zone: "fabric-10g".into(),
lan_addr: Some("10.0.0.13:7701".parse().unwrap()),
tailscale_addr: None,
},
PeerEntry {
name: "architect".into(),
zone: "fabric-10g".into(),
lan_addr: Some("10.0.0.14:7701".parse().unwrap()),
tailscale_addr: None,
},
],
};
let err = cluster.validate().unwrap_err().to_string();
assert!(
err.contains("duplicate peer name"),
"unexpected error: {err}"
);
}
#[test]
fn test_cluster_config_load_invokes_validate() {
// Same shape as a real config file but written to a tempfile so the
// full path — read, parse, validate — is exercised.
let toml = r#"
[node]
name = "tank"
role = "secondary"
[hot]
path = "/hot/targets"
max_gb = 200
stale_hours = 48
[warm]
projects_path = "/slab/projects"
zfs_dataset = "slab/projects"
snapshot_retain_hours = 24
snapshot_retain_days = 7
snapshot_retain_weeks = 4
[cluster]
zone = "fabric-10g"
[[cluster.peers]]
name = "architect"
zone = "fabric-10g"
"#;
let tmp = tempfile::NamedTempFile::new().unwrap();
std::fs::write(tmp.path(), toml).unwrap();
let err = Config::load(tmp.path()).unwrap_err().to_string();
// Fails because cluster has no bind address AND peer has no reachable address.
assert!(err.contains("validating cluster config"), "err: {err}");
}
#[test] #[test]
fn test_load_secondary_config() { fn test_load_secondary_config() {
let toml = r#" let toml = r#"
+27
View File
@@ -1,5 +1,6 @@
mod actions; mod actions;
mod cargo_init; mod cargo_init;
mod cluster;
mod config; mod config;
mod daemon; mod daemon;
mod head_watch; mod head_watch;
@@ -68,6 +69,11 @@ enum Cmd {
Pin { project: String }, Pin { project: String },
/// Unmark a project — it's now a normal GC candidate again /// Unmark a project — it's now a normal GC candidate again
Unpin { project: String }, Unpin { project: String },
/// Probe a cluster peer's LAN and Tailscale endpoints; print the winning route
ClusterProbe {
/// Peer name as listed in `[[cluster.peers]]` in the config
peer: String,
},
} }
#[tokio::main] #[tokio::main]
@@ -102,10 +108,31 @@ async fn main() -> Result<()> {
serve::run_server(cfg, manifest, port, static_dir).await?, serve::run_server(cfg, manifest, port, static_dir).await?,
Cmd::Pin { project } => cmd_set_pin(&manifest_path, &project, true)?, Cmd::Pin { project } => cmd_set_pin(&manifest_path, &project, true)?,
Cmd::Unpin { project } => cmd_set_pin(&manifest_path, &project, false)?, Cmd::Unpin { project } => cmd_set_pin(&manifest_path, &project, false)?,
Cmd::ClusterProbe { peer } => cmd_cluster_probe(&cfg, &peer).await?,
} }
Ok(()) Ok(())
} }
// ── cluster probe ────────────────────────────────────────────────────────────
async fn cmd_cluster_probe(cfg: &Config, peer_name: &str) -> Result<()> {
let cluster_cfg = cfg
.cluster
.as_ref()
.context("no [cluster] section in config; cannot probe peers")?;
let peer = cluster_cfg
.peer(peer_name)
.with_context(|| format!("peer {peer_name:?} not found in cluster.peers"))?;
let probe = cluster::LanFirstProbe::new();
let winner = probe.probe(peer).await?;
println!("peer: {}", peer.name);
println!("zone: {}", peer.zone);
println!("route: {}", winner.kind.as_str());
println!("addr: {}", winner.addr);
println!("elapsed: {:?}", winner.elapsed);
Ok(())
}
// ── pin / unpin ────────────────────────────────────────────────────────────── // ── pin / unpin ──────────────────────────────────────────────────────────────
fn cmd_set_pin(manifest_path: &std::path::Path, project: &str, pinned: bool) -> Result<()> { fn cmd_set_pin(manifest_path: &std::path::Path, project: &str, pinned: bool) -> Result<()> {