Node-local script an operator runs before powering a node off for
parts replacement. Guards against shutting down mid-build or with
un-pushed sync jobs, takes a final snapshot + replicates to cold,
stops the maintenance timers and dashboard, gracefully stops the
daemon (giving gossip its TimeoutStopSec=60 window to announce
departure to peers instead of relying on the 10s failure-detector
timeout), and cleanly unmounts FUSE before declaring the node safe
to power off. --dry-run runs every check for real but only prints
what the stop/unmount steps would do.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Console errors on the live dashboard (architect:7700) turned out to
be two separate problems layered together:
1. The deployed static bundle wasn't built from this repo at all --
it called /api/v2/hot-refs, /api/v2/anomalies, and
/api/v2/node/*/metrics-history, none of which exist anywhere in
this codebase on any branch (checked via `git log --all -S`).
Someone built and shipped a frontend straight to
/usr/share/claw-store/v2 without ever committing the source.
2. Separately, this repo's own committed vite.config.ts had a latent
bug: `base: '/clawstor/'`, contradicting its own comment ("served
by claw-store serve under /v2/*") and the actual backend mount in
serve.rs (`nest_service("/v2", ...)`). Checked `tailscale serve
status` on tank + architect -- neither has ever proxied a
/clawstor path, so that base would have 404'd every asset the
moment anyone rebuilt and redeployed from source.
Fix: base = '/v2/', matching the real mount. Rebuilt and redeployed
to tank + architect (orphaned build backed up to
/usr/share/claw-store/v2.bak-orphaned on both).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
V2State::from_config built its peer fan-out list solely from
[[cluster.peers]], which by definition never includes the local
node. Result: hitting a given node's /api/v2/fleet directly always
omitted that node from its own fleet view, even when perfectly
healthy -- looked like "node X is missing" from the dashboard when
X just never queried itself.
Fix: synthesize a self PeerEntry from the node's own gossip bind
address and include it in the fan-out, same as any other peer.
peer_rpc_addr()'s existing +1 port convention resolves it to the
same bind_rpc_lan/bind_rpc_tailscale the daemon already listens on.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Bind failures at startup are almost always a boot-time race against
DHCP/network-online (bind address not yet assigned to the interface).
Previously the daemon caught the error and kept running in a degraded
state with no gossip, RPC, or Prometheus endpoint and no visible
failure signal. Now it propagates the error so the process exits and
systemd's Restart=on-failure retries once the network is actually up.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
serve_v2::peer_rpc_addr derives each peer's RPC socket as lan_addr.ip():port+1.
Architect and Tank were bound on their 10G fabric NICs (10.10.0.9/10.10.0.10)
but the peer entries only carry the main-LAN gossip address, causing the
aggregator to time out when probing Tank and present a stale cert to Morpheus.
Changing bind_rpc_lan to the main-LAN addresses (10.0.0.13/10.0.0.14) fixes both.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- All three node configs (architect/tank/morpheus) now include [cluster]
section with correct 10G fabric IPs, mTLS TLS paths, and blob store root.
Architect binds gossip on 10.0.0.13:7701 and RPC on 10.10.0.9:7702 (10G
to Tank); Morpheus uses LAN 10.0.0.5 (no direct 10G).
- claw-store.service: updated description, adds clawstor-deploy to
ReadWritePaths, removes NoNewPrivileges (needed for sudo zfs snapshot).
- claw-store-serve.service: adds --v2-static-dir /usr/share/claw-store/v2
so the aggregator serves dashboard-v2 ("clawstor · command center") at /v2/.
- claw-fuse.service: new unit, uses correct --data-dir + --mount flags.
Deployed to Architect, Tank, Morpheus. Fleet CA re-initialized; new certs
signed for all three nodes and distributed. Dashboard accessible at
http://100.104.171.32:7700/v2/ aggregating Tank + Morpheus via /api/v2/fleet.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Tertiary dev node (100.123.224.84) — no ZFS, no dedicated NVMe.
Hot tier on root filesystem, snapshot/replicate timers not enabled.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
vite.config.ts sets outDir to ../claw-store/static (relative to dashboard/),
so the built assets land in claw-store/static/, not the standard dashboard/dist/.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
strip_section now strips only the [build] block before rewriting it,
so existing [alias] tables and other custom sections survive repeated
activate calls. Adds strip_leading_marker to prevent the managed-marker
comment from accumulating on each re-activation.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- Wire RpcRouter::with_repo_root at daemon startup in services.rs
using <blob_store_root>/repos. Nodes with no blob_store_root
still return NotConfigured (unchanged).
- Add end-to-end ensure→cached→release integration test that seeds
a bare git repo in a tempdir and exercises the real git-clone
path. Marked #[ignore] so CI runners without git skip silently;
runs green locally.
Layers HTTP over the R1a per-peer primitive so external callers
(clawmates, gitea runners, ops tooling) speak one URL to the
aggregator instead of dialing every peer.
Endpoints (require v2 auth, same middleware as tags/sessions):
POST /api/v2/repos/ensure {url, git_ref, workspace?}
POST /api/v2/repos/release {url, git_ref, workspace?}
Namespaced tokens are pinned to their own workspace (workspace omitted
in body → derived from token; explicit mismatch → 403). Admin/open
callers must supply workspace explicitly.
Reply shape mirrors the tag fan-out (FanoutReply) with per-peer
{peer, ok, path?, head_sha?, cached?, removed?, error?}. all_ok is
true iff every peer succeeded.
Also adds client wrappers call_repo_ensure/call_repo_release in
cluster/rpc/client.rs used by the aggregator's fan-out.
New per-peer RPCs to shallow-clone a (url, git_ref) under a caller-
provided workspace namespace, and to release the checkout. Fleet
fan-out via the aggregator ships separately in R1b.
- src/cluster/repo_ensure.rs: request/reply types + derive_path
(traversal-safe, blake3-hashed url segment), ensure_repo (cache
hit → rev-parse HEAD, else remove-and-reclone with 5-min timeout),
release_repo (idempotent rm)
- src/cluster/rpc.rs: Method::RepoEnsure=0x1e, RepoRelease=0x1f,
RpcRouter::with_repo_root builder, two dispatch arms returning
NotConfigured when repo_root is unset
- src/cluster.rs: pub mod repo_ensure
- src/actions.rs: fix pre-existing test-only Config init missing the
aggregator field (unblocks lib tests)
Tests: 6 unit tests covering path derivation determinism, ref/url
independence, traversal safety, and sanitizer edge cases. All pass.
Backend:
* New DashboardProject { repo, cache_bytes, fingerprint_count,
refs, first_seen_unix, last_seen_unix, tier }.
* Handler build_projects() joins ref-tracking with the ref-store
and blob-store: for each recorded fp, resolve fp → blob-id via
RefStore::{get_stamped, get} then sum blob sizes per repo.
* Tier is a wall-clock function of last_seen_unix:
active < 24h, recent < 7d, else idle.
* DashboardStorageReply gains `projects: Vec<DashboardProject>`
(serde-default so older clients still parse).
Aggregator:
* New /api/v2/projects endpoint. Fans out DashboardStorage to
every peer, tags each project row with its originating node,
returns hottest-first.
Frontend:
* New ProjectsPanel component appended to the FleetHealth
landing. Groups by repo, one row per project with tier badge
(active/recent/idle), per-node pill badges, cache-size sum,
ref list, last-activity age.
* Empty state explains how to populate: claw-cargo build with
--repo + --git-ref (or CLAWSTOR_REPO/CLAWSTOR_GIT_REF env
vars in CI).
Data populates automatically as each cache-put runs. Existing
demo entry on tank (clawverse/clawstor · main · c384a4...) will
surface after redeploy.
Reworks the SPA around the new DashboardStatus fields. The
landing is now a fleet-health dashboard aimed at a layperson —
disk gauges, mount ✓/✗, cache hit rate, next scheduled job.
No hex, no primitives.
Nav restructure:
* Primary: 'Fleet health' (just the landing).
* 'View Advanced ▾' dropdown reveals: Blobs / Tags / Refs /
Snapshots / Ref-tracking. Routes moved under /advanced/*.
New components:
* StorageBar — horizontal used/total bar with pinned/evictable
split, health-color threshold at 60/85%.
* NodeCard — traffic-light dot + disk + hot tier bars + mount
state + cache hit rate + next-timer countdown. Whole card
is a link into node detail.
New CommandCenter:
* Fleet-wide storage roll-up card (sum of every node's disk).
* Grid of NodeCards.
* 10s poll cadence retained from previous version.
Existing StorageBrowser + RefTrackingPage moved behind
/advanced/* routes; internal component code untouched.
Extends DashboardStatus with the primitives the FleetHealth
landing needs. All new fields serde-default to None/[]/false so
old clients (aggregator running an older build) still parse the
reply.
Added fields:
* filesystem — statvfs on blob_store_root: total / used / avail.
* hot — {used_bytes, max_bytes, pinned_bytes?}, read from
gossip (already published every 10s).
* mount — probes /proc/mounts for ~/clawstor-mount by
convention. { path, active }.
* cache — router.metrics().snapshot() summarised as
{hits, misses, bytes_served, bytes_ingested,
hit_rate}. hits = get_ref+get_tag+get_chunk hits,
misses similarly — the counters that matter for
claw-cargo build outcomes.
* timers — well-known set (scrub / gc / ref-sweep /
snapshot-rotate) queried via `systemctl --user
show`. next_fire_unix + last_result per timer.
Aggregator NodeStatusV2 mirrors the same fields verbatim so the
new frontend can consume them without further backend hops.
Follow-on PRs:
* PR 2 — new FleetHealth landing page + node-detail rework
* PR 3 — polish (pinned_bytes derivation, activity timeline,
per-repo grouping)
Bare <Link href='/'> was navigating to tailnet root =
OpenClaw (which mounts /). Wrap App in <Router base=...>
detected from window.location so links + route matching
prefix the mount path. Falls back to no-base for local
:7700/v2/ dev too.
All storage rows now carry a 'node' field from the aggregator.
Adds a small green-pill component that renders the source node
name; click drills into that node's detail page.
Also fixes the TS interfaces to match the new backend shape
(node field added to Blob/Tag/Ref/Snapshot/RefTracking types).
Storage tabs 404'd because the aggregator only had /fleet + /node/:name
after the pivot. Adds:
* New RPC method DashboardStorage = 0x1d — one round trip returns
tags (full), snapshots (full), ref-tracking (full), blobs
(first 200 by id), refs (first 200 by fp) for the responding
daemon.
* Client wrapper call_dashboard_storage.
* Aggregator fans out to every peer, tags each row with the
originating node, sorts + returns:
GET /api/v2/storage/blobs
GET /api/v2/storage/tags
GET /api/v2/storage/refs
GET /api/v2/storage/snapshots
GET /api/v2/storage/ref-tracking
QuicClient promoted to Arc<QuicClient> inside V2State so the
per-peer JoinSet can hand it to spawned tasks without recreating
the endpoint.
Prior ./ (fully-relative) fix broke when browser hit /clawstor
without trailing slash — ./assets/… resolved to /assets/… at
tailnet root, 404. Absolute /clawstor/ is deploy-path-coupled
but no-slash-safe. Verified: no-slash HTML 200, CSS 200,
JS 200, /clawstor/api/v2/fleet 200.
Browser hit /clawstor/ then requested /v2/assets/index-XXX.css
(absolute path baked in by vite base '/v2/') which 404'd because
Tailscale Serve only mounted /clawstor and /clawstor/api. Switch
vite base to './' — assets resolve relative to whatever URL the
SPA loaded from. Works for local (:7700/v2/) + Tailscale
(/clawstor) with no config coupling.
Pivot from per-node dashboard to single-pane-of-glass. The
aggregator (typically the operator's laptop) holds a fleet-CA
leaf cert + the peer list; each dashboard request fans out to
every peer over the existing QUIC/mTLS cluster port and issues
the new DashboardStatus RPC. Peers don't need to run any HTTP
server of their own.
Backend:
* New RPC method DashboardStatus = 0x1c
* Server handler reads BlobStore / TagStore / RefStore /
SnapshotStore / RefTracking counts + on-disk bytes + rustc
release. Cheap: 5 filesystem walks per request.
* Client wrapper call_dashboard_status
* serve_v2 rewritten as aggregator: V2State holds a QuicClient +
peer list from `[[cluster.peers]]`. Endpoints:
GET /api/v2/fleet fan-out to every peer, parallel
GET /api/v2/node/:name/status one peer, on-demand
Failed peers surface as { online: false, error: "..." } cards
instead of dropping.
serve.rs graceful degrade: v2 aggregator routes only mount when
[cluster.tls] is set. Static SPA still serves at /v2/* even
without an aggregator config so operators see the SPA's built-in
"config missing" error.
Deployment model (this session):
* Aggregator runs on quantum (Mac) with a signed leaf.
* Fleet daemons run cluster-only — no HTTP dashboard anywhere
on tank/architect/morpheus. The clawstor-dashboard.service
systemd units on the fleet are being retired.
Frontend rework to consume /api/v2/fleet ships in the next PR.
React 19 + Vite + Tailwind + wouter (tiny router, no external
state library). Consumes the /api/v2/* endpoints shipped in PR 1.
Serves under /v2/* so the legacy dashboard at / stays live.
Pages:
* CommandCenter (/) — fleet strip + this-node stat tiles
* NodeDetail (/nodes/:name) — per-node deep dive
* StorageBrowser (/storage/{blobs,tags,refs,snapshots}) — tables
with prefix filter
* RefTrackingPage (/refs/tracking) — grouped by repo
Backend changes:
* claw-store serve grows --v2-static-dir <path>
* build_app split into build_app_with_v2 for the extra static
mount
* /v2/* falls through to index.html so wouter client routing works
New systemd unit: clawstor-dashboard.service. Points at both
static dirs; installs on any node.
dashboard/ (legacy) untouched. dashboard-v2/ built to
target/dashboard-v2/dist for deploy.
Deploy sequence per node:
1. cp target/release/claw-store ~/clawstor-deploy/
2. rsync dashboard-v2/dist/ ~/clawstor-deploy/dashboard-v2/
3. cp deploy/systemd/clawstor-dashboard.service ~/.config/systemd/user/
4. systemctl --user daemon-reload && enable --now clawstor-dashboard.service
Cross-node fan-out for /api/v2/node/:name/status is PR 3.
Action POSTs (scrub/gc/snapshot/pin) are PR 4.
deploy/scripts/fleet-status.sh iterates NODES (default 'tank
architect morpheus') and prints per-host: daemon state, FUSE
mount + layer listing, blob-store byte count, and each timer's
next-fire + last-result.
Fits in half a screen per node; useful as a smoke check before
+ after any fleet-wide change. Read-only over ssh — safe to
run from any workstation with keys.
Live smoke on morpheus with the previous unit failed:
Error: mounting FUSE at /home/osobh/clawstor-mount
Caused by: Operation not permitted (os error 1)
But 'systemd-run --user' with the exact same binary worked, and
the manual invocation worked. The delta was our ExecStartPre
chain (mkdir + fusermount3 -u -z) — on some Ubuntu 24.04 builds
that combination poisons the subsequent mount syscall even with
the -z lazy flag + ignored exit code.
Fix: drop the pre-mkdir + pre-unmount. Operator creates the
mount dir manually once (documented in README). Restart handling
falls to systemd's Restart=on-failure + ExecStop unmount.
Verified live on morpheus with the minimal unit — active +
mount visible in <3s.
Cargo.toml: fuser dep now target-gated to Linux + macOS. macOS
build requires macFUSE (brew install --cask macfuse) + pkg-config
before 'cargo build --features fuse' works.
deploy/macos/README.md — one-time prereqs, build steps, mount /
umount, known differences (no AllowOther, unmount is 'umount'
not fusermount3 -u).
deploy/macos/claw-fuse.plist — launchd agent template
(RunAtLoad + KeepAlive) so ghost / macbook / smith can run the
mount the same way tank/architect do under systemd.
Default 'cargo build' (no --features fuse) still works on macOS
with no macFUSE installed — feature gate keeps the dep opt-in.
osobh
