Author SHA1 Message Date
osobhandClaude Sonnet 5 bb24c77676 Include self in dashboard-v2 fleet aggregation
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Failing after 3s
V2State::from_config built its peer fan-out list solely from
[[cluster.peers]], which by definition never includes the local
node. Result: hitting a given node's /api/v2/fleet directly always
omitted that node from its own fleet view, even when perfectly
healthy -- looked like "node X is missing" from the dashboard when
X just never queried itself.

Fix: synthesize a self PeerEntry from the node's own gossip bind
address and include it in the fan-out, same as any other peer.
peer_rpc_addr()'s existing +1 port convention resolves it to the
same bind_rpc_lan/bind_rpc_tailscale the daemon already listens on.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-07-31 13:01:02 -07:00
osobhandClaude Sonnet 5 75d822d0f2 Fail fast when cluster gossip bootstrap fails
Bind failures at startup are almost always a boot-time race against
DHCP/network-online (bind address not yet assigned to the interface).
Previously the daemon caught the error and kept running in a degraded
state with no gossip, RPC, or Prometheus endpoint and no visible
failure signal. Now it propagates the error so the process exits and
systemd's Restart=on-failure retries once the network is actually up.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-07-31 11:48:40 -07:00
Omar Sobh dd2b90872a Phase 9 R1c: daemon wiring + integration test
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Failing after 3s
- Wire RpcRouter::with_repo_root at daemon startup in services.rs
  using <blob_store_root>/repos. Nodes with no blob_store_root
  still return NotConfigured (unchanged).
- Add end-to-end ensure→cached→release integration test that seeds
  a bare git repo in a tempdir and exercises the real git-clone
  path. Marked #[ignore] so CI runners without git skip silently;
  runs green locally.
2026-07-15 04:11:58 -07:00
Omar Sobh 5c9bc7eb9c Phase 9 R1b: aggregator fan-out for RepoEnsure/RepoRelease
Layers HTTP over the R1a per-peer primitive so external callers
(clawmates, gitea runners, ops tooling) speak one URL to the
aggregator instead of dialing every peer.

Endpoints (require v2 auth, same middleware as tags/sessions):
  POST /api/v2/repos/ensure   {url, git_ref, workspace?}
  POST /api/v2/repos/release  {url, git_ref, workspace?}

Namespaced tokens are pinned to their own workspace (workspace omitted
in body → derived from token; explicit mismatch → 403). Admin/open
callers must supply workspace explicitly.

Reply shape mirrors the tag fan-out (FanoutReply) with per-peer
{peer, ok, path?, head_sha?, cached?, removed?, error?}. all_ok is
true iff every peer succeeded.

Also adds client wrappers call_repo_ensure/call_repo_release in
cluster/rpc/client.rs used by the aggregator's fan-out.
2026-07-15 04:08:35 -07:00
Omar Sobh 5c1d962bf2 Phase 9 R1a fixup: drop dead if-let wrapping around mkdir 2026-07-15 03:48:24 -07:00
Omar Sobh a3fe1d147c Phase 9 R1a: peer RepoEnsure/RepoRelease RPC (0x1e/0x1f)
New per-peer RPCs to shallow-clone a (url, git_ref) under a caller-
provided workspace namespace, and to release the checkout. Fleet
fan-out via the aggregator ships separately in R1b.

- src/cluster/repo_ensure.rs: request/reply types + derive_path
  (traversal-safe, blake3-hashed url segment), ensure_repo (cache
  hit → rev-parse HEAD, else remove-and-reclone with 5-min timeout),
  release_repo (idempotent rm)
- src/cluster/rpc.rs: Method::RepoEnsure=0x1e, RepoRelease=0x1f,
  RpcRouter::with_repo_root builder, two dispatch arms returning
  NotConfigured when repo_root is unset
- src/cluster.rs: pub mod repo_ensure
- src/actions.rs: fix pre-existing test-only Config init missing the
  aggregator field (unblocks lib tests)

Tests: 6 unit tests covering path derivation determinism, ref/url
independence, traversal safety, and sanitizer edge cases. All pass.
2026-07-15 03:42:27 -07:00
10 changed files with 52 additions and 205 deletions
+1 -1
View File
@@ -32,7 +32,7 @@ install-systemd:
install-dashboard: install-dashboard:
cd dashboard && npm ci && npm run build cd dashboard && npm ci && npm run build
install -dm755 $(INSTALL_STATIC) install -dm755 $(INSTALL_STATIC)
cp -r claw-store/static/. $(INSTALL_STATIC)/ cp -r dashboard/dist/. $(INSTALL_STATIC)/
@echo "Dashboard installed to $(INSTALL_STATIC)" @echo "Dashboard installed to $(INSTALL_STATIC)"
## Install node-specific config (NODE=architect|tank) ## Install node-specific config (NODE=architect|tank)
+1 -62
View File
@@ -21,14 +21,8 @@ pub fn write_cargo_config(warm_path: &Path, hot_target_path: &Path) -> Result<()
String::new() String::new()
}; };
// Drop any leading copies of our own marker comment before stripping the
// [build] section — it sits above the [build] header, outside the range
// strip_section tracks, so without this it would survive every
// regenerate cycle and duplicate one more time.
let existing = strip_leading_marker(&existing);
// Remove old [build] block (from its header to the next section or EOF). // Remove old [build] block (from its header to the next section or EOF).
let stripped = strip_section(existing, "build"); let stripped = strip_section(&existing, "build");
let new_content = format!( let new_content = format!(
"# claw-store managed — do not edit manually\n\ "# claw-store managed — do not edit manually\n\
@@ -43,21 +37,6 @@ pub fn write_cargo_config(warm_path: &Path, hot_target_path: &Path) -> Result<()
.with_context(|| format!("writing {}", config_path.display())) .with_context(|| format!("writing {}", config_path.display()))
} }
const MANAGED_MARKER: &str = "# claw-store managed — do not edit manually";
/// Strip leading copies of `MANAGED_MARKER`, one per line, from the start of `src`.
fn strip_leading_marker(src: &str) -> &str {
let mut rest = src;
while let Some(line_end) = rest.find('\n') {
if rest[..line_end].trim() == MANAGED_MARKER {
rest = &rest[line_end + 1..];
} else {
break;
}
}
rest
}
/// Remove a TOML section `[name]` and all its key=value lines from `src`, /// Remove a TOML section `[name]` and all its key=value lines from `src`,
/// stopping at the next `[section]` header or EOF. /// stopping at the next `[section]` header or EOF.
fn strip_section(src: &str, name: &str) -> String { fn strip_section(src: &str, name: &str) -> String {
@@ -123,46 +102,6 @@ mod tests {
assert!(verify_cargo_config(&warm, &hot).unwrap()); assert!(verify_cargo_config(&warm, &hot).unwrap());
} }
#[test]
fn test_write_cargo_config_idempotent_no_duplicate_marker() {
let dir = TempDir::new().unwrap();
let warm = dir.path().join("proj");
std::fs::create_dir_all(&warm).unwrap();
let hot: std::path::PathBuf = "/hot/targets/proj".into();
write_cargo_config(&warm, &hot).unwrap();
write_cargo_config(&warm, &hot).unwrap();
write_cargo_config(&warm, &hot).unwrap();
let content = std::fs::read_to_string(warm.join(".cargo/config.toml")).unwrap();
assert_eq!(content.matches("claw-store managed").count(), 1);
}
#[test]
fn test_write_cargo_config_heals_existing_duplicate_marker() {
let dir = TempDir::new().unwrap();
let warm = dir.path().join("proj");
let cargo_dir = warm.join(".cargo");
std::fs::create_dir_all(&cargo_dir).unwrap();
std::fs::write(
cargo_dir.join("config.toml"),
"# claw-store managed — do not edit manually\n\
[build]\n\
target-dir = \"/hot/targets/proj\"\n\
# claw-store managed — do not edit manually\n\
[env]\n\
FOO = \"bar\"\n",
)
.unwrap();
let hot: std::path::PathBuf = "/hot/targets/proj".into();
write_cargo_config(&warm, &hot).unwrap();
let content = std::fs::read_to_string(cargo_dir.join("config.toml")).unwrap();
assert_eq!(content.matches("claw-store managed").count(), 1);
assert!(content.contains("FOO = \"bar\""));
}
#[test] #[test]
fn test_verify_cargo_config_detects_missing() { fn test_verify_cargo_config_detects_missing() {
let dir = TempDir::new().unwrap(); let dir = TempDir::new().unwrap();
+17 -17
View File
@@ -7,7 +7,7 @@ use crate::manifest::Manifest;
use crate::snapshot; use crate::snapshot;
use crate::sync::{SyncQueue, drain_sync_queue}; use crate::sync::{SyncQueue, drain_sync_queue};
use crate::zfs::SystemZfs; use crate::zfs::SystemZfs;
use anyhow::Result; use anyhow::{Context, Result};
use chrono::Utc; use chrono::Utc;
use sysinfo::{ProcessRefreshKind, RefreshKind, System}; use sysinfo::{ProcessRefreshKind, RefreshKind, System};
use tokio::time::{interval, Duration}; use tokio::time::{interval, Duration};
@@ -36,7 +36,14 @@ pub async fn run(cfg: Config, mut manifest: Manifest) -> Result<()> {
let hot_dir = cfg.hot.path.clone(); let hot_dir = cfg.hot.path.clone();
let hot_max_bytes = cfg.hot.max_gb.saturating_mul(1024 * 1024 * 1024); let hot_max_bytes = cfg.hot.max_gb.saturating_mul(1024 * 1024 * 1024);
let blob_root = cluster_cfg.blob_store_root.clone(); let blob_root = cluster_cfg.blob_store_root.clone();
match ClusterServices::start( // Fail fast rather than degrade silently: a bind failure here is
// almost always a boot-time race against DHCP/network-online
// (the bind address isn't assigned to the interface yet). The
// systemd unit has `Restart=on-failure`; exiting lets it retry
// a few seconds later once the network is actually up, instead
// of leaving the daemon running indefinitely with no gossip,
// RPC, or Prometheus endpoint and no visible failure state.
let svc = ClusterServices::start(
cluster_cfg, cluster_cfg,
cfg.node.name.clone(), cfg.node.name.clone(),
hot_dir, hot_dir,
@@ -44,21 +51,14 @@ pub async fn run(cfg: Config, mut manifest: Manifest) -> Result<()> {
blob_root, blob_root,
) )
.await .await
{ .context("starting cluster services")?;
Ok(svc) => { tracing::info!(
tracing::info!( rpc_enabled = svc.rpc_enabled(),
rpc_enabled = svc.rpc_enabled(), blob_store_enabled = svc.blob_store_enabled(),
blob_store_enabled = svc.blob_store_enabled(), zone = %cluster_cfg.zone,
zone = %cluster_cfg.zone, "cluster services online"
"cluster services online" );
); Some(svc)
Some(svc)
}
Err(e) => {
tracing::error!(error = %e, "cluster services failed to start; continuing without cluster");
None
}
}
} }
None => { None => {
tracing::info!("no [cluster] section in config; running standalone"); tracing::info!("no [cluster] section in config; running standalone");
+22 -1
View File
@@ -91,9 +91,30 @@ impl V2State {
.join("aggregator-sessions.json"); .join("aggregator-sessions.json");
let sessions = SessionStore::load(sessions_path) let sessions = SessionStore::load(sessions_path)
.map_err(|e| anyhow::anyhow!("loading session store: {e}"))?; .map_err(|e| anyhow::anyhow!("loading session store: {e}"))?;
// Bug fix 2026-07-31: the fleet view previously never included
// the node actually serving the dashboard — `cluster.peers` is
// by definition every *other* node, so hitting a given node's
// `/api/v2/fleet` directly silently dropped that node from its
// own view (looked like "node X is missing" from the UI, even
// though X was perfectly healthy — it just never queried
// itself). Fix: synthesize a self `PeerEntry` from our own
// gossip bind address and include it in the fan-out list, same
// as any other peer. `peer_rpc_addr` derives the RPC port from
// `lan_addr`/`tailscale_addr` via the fleet's +1 convention, so
// this resolves to the same `bind_rpc_lan`/`bind_rpc_tailscale`
// the daemon actually listens on.
let self_peer = PeerEntry {
name: cfg.node.name.clone(),
zone: cluster.zone.clone(),
lan_addr: cluster.bind_lan,
tailscale_addr: cluster.bind_tailscale,
};
let mut peers = cluster.peers.clone();
peers.push(self_peer);
Ok(Self { Ok(Self {
aggregator_name: cfg.node.name.clone(), aggregator_name: cfg.node.name.clone(),
peers: cluster.peers.clone(), peers,
client: std::sync::Arc::new(client), client: std::sync::Arc::new(client),
default_rpc_port_offset: 1, default_rpc_port_offset: 1,
api_token: cfg.api_token.clone(), api_token: cfg.api_token.clone(),
-27
View File
@@ -19,33 +19,6 @@ archive_path = "/data/archive"
zfs_dataset = "data/archive" zfs_dataset = "data/archive"
retain_weeks = 12 retain_weeks = 12
[cluster]
zone = "fabric-10g"
bind_lan = "10.0.0.13:7701"
prom_bind = "0.0.0.0:7703"
bind_rpc_lan = "10.0.0.13:7702"
bind_rpc_tailscale = "100.104.171.32:7702"
blob_store_root = "/home/osobh/clawstor-deploy/data"
[[cluster.peers]]
name = "tank"
zone = "fabric-10g"
lan_addr = "10.0.0.14:7701"
rpc_lan_addr = "10.10.0.10:7702"
tailscale_addr = "100.108.129.81:7702"
[[cluster.peers]]
name = "morpheus"
zone = "lan-1g"
lan_addr = "10.0.0.5:7701"
rpc_lan_addr = "10.0.0.5:7702"
tailscale_addr = "100.123.224.84:7702"
[cluster.tls]
ca_cert = "/home/osobh/clawstor-deploy/tls/ca.crt"
node_cert = "/home/osobh/clawstor-deploy/tls/node.crt"
node_key = "/home/osobh/clawstor-deploy/tls/node.key"
[replication] [replication]
receive_from_peer = true receive_from_peer = true
peer_user = "osobh" peer_user = "osobh"
-43
View File
@@ -1,43 +0,0 @@
[node]
name = "morpheus"
role = "secondary"
[hot]
path = "/hot/targets"
max_gb = 80
stale_hours = 48
[warm]
projects_path = "/slab/projects"
zfs_dataset = "none"
snapshot_retain_hours = 24
snapshot_retain_days = 7
snapshot_retain_weeks = 4
[cluster]
zone = "lan-1g"
# Morpheus has no direct 10G to Architect/Tank — use main LAN for all traffic
bind_lan = "10.0.0.5:7701"
prom_bind = "0.0.0.0:7703"
bind_rpc_lan = "10.0.0.5:7702"
bind_rpc_tailscale = "100.123.224.84:7702"
blob_store_root = "/home/osobh/clawstor-deploy/data"
[[cluster.peers]]
name = "architect"
zone = "fabric-10g"
lan_addr = "10.0.0.13:7701"
rpc_lan_addr = "10.0.0.13:7702"
tailscale_addr = "100.104.171.32:7702"
[[cluster.peers]]
name = "tank"
zone = "fabric-10g"
lan_addr = "10.0.0.14:7701"
rpc_lan_addr = "10.0.0.14:7702"
tailscale_addr = "100.108.129.81:7702"
[cluster.tls]
ca_cert = "/home/osobh/clawstor-deploy/tls/ca.crt"
node_cert = "/home/osobh/clawstor-deploy/tls/node.crt"
node_key = "/home/osobh/clawstor-deploy/tls/node.key"
+5 -27
View File
@@ -14,35 +14,13 @@ snapshot_retain_hours = 24
snapshot_retain_days = 7 snapshot_retain_days = 7
snapshot_retain_weeks = 4 snapshot_retain_weeks = 4
[cluster]
zone = "fabric-10g"
bind_lan = "10.0.0.14:7701"
prom_bind = "0.0.0.0:7703"
bind_rpc_lan = "10.0.0.14:7702"
bind_rpc_tailscale = "100.108.129.81:7702"
blob_store_root = "/home/osobh/clawstor-deploy/data"
[[cluster.peers]]
name = "architect"
zone = "fabric-10g"
lan_addr = "10.0.0.13:7701"
rpc_lan_addr = "10.10.0.9:7702"
tailscale_addr = "100.104.171.32:7702"
[[cluster.peers]]
name = "morpheus"
zone = "lan-1g"
lan_addr = "10.0.0.5:7701"
rpc_lan_addr = "10.0.0.5:7702"
tailscale_addr = "100.123.224.84:7702"
[cluster.tls]
ca_cert = "/home/osobh/clawstor-deploy/tls/ca.crt"
node_cert = "/home/osobh/clawstor-deploy/tls/node.crt"
node_key = "/home/osobh/clawstor-deploy/tls/node.key"
[replication] [replication]
# 10.10.0.9 is architect-fab-tank — the dedicated 10G fabric link between
# the two nodes. Intentionally used for replication to maximise bandwidth;
# architect's primary LAN address is 10.0.0.13.
send_to_host = "10.10.0.9" send_to_host = "10.10.0.9"
send_to_user = "osobh" send_to_user = "osobh"
cold_dataset_on_peer = "data/archive/tank-projects" cold_dataset_on_peer = "data/archive/tank-projects"
# nightly_at is reserved for future use; replication schedule is currently
# controlled by the claw-store-replicate.timer systemd unit.
nightly_at = "03:30" nightly_at = "03:30"
-17
View File
@@ -1,17 +0,0 @@
[Unit]
Description=clawstor FUSE mount
After=claw-store.service
Requires=claw-store.service
[Service]
Type=simple
User=osobh
Environment=PATH=/home/osobh/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ExecStartPre=/bin/mkdir -p /home/osobh/clawstor-mount
ExecStart=/usr/local/bin/claw-fuse --data-dir /home/osobh/clawstor-deploy/data --mount /home/osobh/clawstor-mount
ExecStop=/bin/fusermount -u /home/osobh/clawstor-mount
Restart=on-failure
RestartSec=30
[Install]
WantedBy=multi-user.target
+1 -1
View File
@@ -11,7 +11,7 @@ Wants=network-online.target
[Service] [Service]
Type=simple Type=simple
User=osobh User=osobh
ExecStart=/usr/local/bin/claw-store serve --port 7700 --static-dir /usr/share/claw-store/static --v2-static-dir /usr/share/claw-store/v2 ExecStart=/usr/local/bin/claw-store serve --port 7700 --static-dir /usr/share/claw-store/static
Restart=on-failure Restart=on-failure
RestartSec=15 RestartSec=15
Environment=RUST_LOG=info Environment=RUST_LOG=info
+5 -9
View File
@@ -1,19 +1,15 @@
[Unit] [Unit]
Description=clawstor cluster daemon (gossip + QUIC blob store + ZFS snapshots) Description=claw-store fleet storage daemon
After=network-online.target After=zfs-mount.service network.target
Wants=network-online.target Wants=zfs-mount.service
[Service] [Service]
Type=simple Type=simple
User=osobh User=osobh
Environment=PATH=/home/osobh/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Environment=RUST_LOG=info
ExecStart=/usr/local/bin/claw-store daemon ExecStart=/usr/local/bin/claw-store daemon
Restart=on-failure Restart=on-failure
RestartSec=15 RestartSec=30
TimeoutStopSec=60 Environment=RUST_LOG=info
ProtectSystem=strict
ReadWritePaths=/var/lib/claw-store /hot/targets /slab/projects /home/osobh/clawstor-deploy
[Install] [Install]
WantedBy=multi-user.target WantedBy=multi-user.target