New per-peer RPCs to shallow-clone a (url, git_ref) under a caller-
provided workspace namespace, and to release the checkout. Fleet
fan-out via the aggregator ships separately in R1b.
- src/cluster/repo_ensure.rs: request/reply types + derive_path
(traversal-safe, blake3-hashed url segment), ensure_repo (cache
hit → rev-parse HEAD, else remove-and-reclone with 5-min timeout),
release_repo (idempotent rm)
- src/cluster/rpc.rs: Method::RepoEnsure=0x1e, RepoRelease=0x1f,
RpcRouter::with_repo_root builder, two dispatch arms returning
NotConfigured when repo_root is unset
- src/cluster.rs: pub mod repo_ensure
- src/actions.rs: fix pre-existing test-only Config init missing the
aggregator field (unblocks lib tests)
Tests: 6 unit tests covering path derivation determinism, ref/url
independence, traversal safety, and sanitizer edge cases. All pass.
Backend:
* New DashboardProject { repo, cache_bytes, fingerprint_count,
refs, first_seen_unix, last_seen_unix, tier }.
* Handler build_projects() joins ref-tracking with the ref-store
and blob-store: for each recorded fp, resolve fp → blob-id via
RefStore::{get_stamped, get} then sum blob sizes per repo.
* Tier is a wall-clock function of last_seen_unix:
active < 24h, recent < 7d, else idle.
* DashboardStorageReply gains `projects: Vec<DashboardProject>`
(serde-default so older clients still parse).
Aggregator:
* New /api/v2/projects endpoint. Fans out DashboardStorage to
every peer, tags each project row with its originating node,
returns hottest-first.
Frontend:
* New ProjectsPanel component appended to the FleetHealth
landing. Groups by repo, one row per project with tier badge
(active/recent/idle), per-node pill badges, cache-size sum,
ref list, last-activity age.
* Empty state explains how to populate: claw-cargo build with
--repo + --git-ref (or CLAWSTOR_REPO/CLAWSTOR_GIT_REF env
vars in CI).
Data populates automatically as each cache-put runs. Existing
demo entry on tank (clawverse/clawstor · main · c384a4...) will
surface after redeploy.
Reworks the SPA around the new DashboardStatus fields. The
landing is now a fleet-health dashboard aimed at a layperson —
disk gauges, mount ✓/✗, cache hit rate, next scheduled job.
No hex, no primitives.
Nav restructure:
* Primary: 'Fleet health' (just the landing).
* 'View Advanced ▾' dropdown reveals: Blobs / Tags / Refs /
Snapshots / Ref-tracking. Routes moved under /advanced/*.
New components:
* StorageBar — horizontal used/total bar with pinned/evictable
split, health-color threshold at 60/85%.
* NodeCard — traffic-light dot + disk + hot tier bars + mount
state + cache hit rate + next-timer countdown. Whole card
is a link into node detail.
New CommandCenter:
* Fleet-wide storage roll-up card (sum of every node's disk).
* Grid of NodeCards.
* 10s poll cadence retained from previous version.
Existing StorageBrowser + RefTrackingPage moved behind
/advanced/* routes; internal component code untouched.
Extends DashboardStatus with the primitives the FleetHealth
landing needs. All new fields serde-default to None/[]/false so
old clients (aggregator running an older build) still parse the
reply.
Added fields:
* filesystem — statvfs on blob_store_root: total / used / avail.
* hot — {used_bytes, max_bytes, pinned_bytes?}, read from
gossip (already published every 10s).
* mount — probes /proc/mounts for ~/clawstor-mount by
convention. { path, active }.
* cache — router.metrics().snapshot() summarised as
{hits, misses, bytes_served, bytes_ingested,
hit_rate}. hits = get_ref+get_tag+get_chunk hits,
misses similarly — the counters that matter for
claw-cargo build outcomes.
* timers — well-known set (scrub / gc / ref-sweep /
snapshot-rotate) queried via `systemctl --user
show`. next_fire_unix + last_result per timer.
Aggregator NodeStatusV2 mirrors the same fields verbatim so the
new frontend can consume them without further backend hops.
Follow-on PRs:
* PR 2 — new FleetHealth landing page + node-detail rework
* PR 3 — polish (pinned_bytes derivation, activity timeline,
per-repo grouping)
Bare <Link href='/'> was navigating to tailnet root =
OpenClaw (which mounts /). Wrap App in <Router base=...>
detected from window.location so links + route matching
prefix the mount path. Falls back to no-base for local
:7700/v2/ dev too.
All storage rows now carry a 'node' field from the aggregator.
Adds a small green-pill component that renders the source node
name; click drills into that node's detail page.
Also fixes the TS interfaces to match the new backend shape
(node field added to Blob/Tag/Ref/Snapshot/RefTracking types).
Storage tabs 404'd because the aggregator only had /fleet + /node/:name
after the pivot. Adds:
* New RPC method DashboardStorage = 0x1d — one round trip returns
tags (full), snapshots (full), ref-tracking (full), blobs
(first 200 by id), refs (first 200 by fp) for the responding
daemon.
* Client wrapper call_dashboard_storage.
* Aggregator fans out to every peer, tags each row with the
originating node, sorts + returns:
GET /api/v2/storage/blobs
GET /api/v2/storage/tags
GET /api/v2/storage/refs
GET /api/v2/storage/snapshots
GET /api/v2/storage/ref-tracking
QuicClient promoted to Arc<QuicClient> inside V2State so the
per-peer JoinSet can hand it to spawned tasks without recreating
the endpoint.
Prior ./ (fully-relative) fix broke when browser hit /clawstor
without trailing slash — ./assets/… resolved to /assets/… at
tailnet root, 404. Absolute /clawstor/ is deploy-path-coupled
but no-slash-safe. Verified: no-slash HTML 200, CSS 200,
JS 200, /clawstor/api/v2/fleet 200.
Browser hit /clawstor/ then requested /v2/assets/index-XXX.css
(absolute path baked in by vite base '/v2/') which 404'd because
Tailscale Serve only mounted /clawstor and /clawstor/api. Switch
vite base to './' — assets resolve relative to whatever URL the
SPA loaded from. Works for local (:7700/v2/) + Tailscale
(/clawstor) with no config coupling.
Pivot from per-node dashboard to single-pane-of-glass. The
aggregator (typically the operator's laptop) holds a fleet-CA
leaf cert + the peer list; each dashboard request fans out to
every peer over the existing QUIC/mTLS cluster port and issues
the new DashboardStatus RPC. Peers don't need to run any HTTP
server of their own.
Backend:
* New RPC method DashboardStatus = 0x1c
* Server handler reads BlobStore / TagStore / RefStore /
SnapshotStore / RefTracking counts + on-disk bytes + rustc
release. Cheap: 5 filesystem walks per request.
* Client wrapper call_dashboard_status
* serve_v2 rewritten as aggregator: V2State holds a QuicClient +
peer list from `[[cluster.peers]]`. Endpoints:
GET /api/v2/fleet fan-out to every peer, parallel
GET /api/v2/node/:name/status one peer, on-demand
Failed peers surface as { online: false, error: "..." } cards
instead of dropping.
serve.rs graceful degrade: v2 aggregator routes only mount when
[cluster.tls] is set. Static SPA still serves at /v2/* even
without an aggregator config so operators see the SPA's built-in
"config missing" error.
Deployment model (this session):
* Aggregator runs on quantum (Mac) with a signed leaf.
* Fleet daemons run cluster-only — no HTTP dashboard anywhere
on tank/architect/morpheus. The clawstor-dashboard.service
systemd units on the fleet are being retired.
Frontend rework to consume /api/v2/fleet ships in the next PR.
React 19 + Vite + Tailwind + wouter (tiny router, no external
state library). Consumes the /api/v2/* endpoints shipped in PR 1.
Serves under /v2/* so the legacy dashboard at / stays live.
Pages:
* CommandCenter (/) — fleet strip + this-node stat tiles
* NodeDetail (/nodes/:name) — per-node deep dive
* StorageBrowser (/storage/{blobs,tags,refs,snapshots}) — tables
with prefix filter
* RefTrackingPage (/refs/tracking) — grouped by repo
Backend changes:
* claw-store serve grows --v2-static-dir <path>
* build_app split into build_app_with_v2 for the extra static
mount
* /v2/* falls through to index.html so wouter client routing works
New systemd unit: clawstor-dashboard.service. Points at both
static dirs; installs on any node.
dashboard/ (legacy) untouched. dashboard-v2/ built to
target/dashboard-v2/dist for deploy.
Deploy sequence per node:
1. cp target/release/claw-store ~/clawstor-deploy/
2. rsync dashboard-v2/dist/ ~/clawstor-deploy/dashboard-v2/
3. cp deploy/systemd/clawstor-dashboard.service ~/.config/systemd/user/
4. systemctl --user daemon-reload && enable --now clawstor-dashboard.service
Cross-node fan-out for /api/v2/node/:name/status is PR 3.
Action POSTs (scrub/gc/snapshot/pin) are PR 4.
deploy/scripts/fleet-status.sh iterates NODES (default 'tank
architect morpheus') and prints per-host: daemon state, FUSE
mount + layer listing, blob-store byte count, and each timer's
next-fire + last-result.
Fits in half a screen per node; useful as a smoke check before
+ after any fleet-wide change. Read-only over ssh — safe to
run from any workstation with keys.
Live smoke on morpheus with the previous unit failed:
Error: mounting FUSE at /home/osobh/clawstor-mount
Caused by: Operation not permitted (os error 1)
But 'systemd-run --user' with the exact same binary worked, and
the manual invocation worked. The delta was our ExecStartPre
chain (mkdir + fusermount3 -u -z) — on some Ubuntu 24.04 builds
that combination poisons the subsequent mount syscall even with
the -z lazy flag + ignored exit code.
Fix: drop the pre-mkdir + pre-unmount. Operator creates the
mount dir manually once (documented in README). Restart handling
falls to systemd's Restart=on-failure + ExecStop unmount.
Verified live on morpheus with the minimal unit — active +
mount visible in <3s.
Cargo.toml: fuser dep now target-gated to Linux + macOS. macOS
build requires macFUSE (brew install --cask macfuse) + pkg-config
before 'cargo build --features fuse' works.
deploy/macos/README.md — one-time prereqs, build steps, mount /
umount, known differences (no AllowOther, unmount is 'umount'
not fusermount3 -u).
deploy/macos/claw-fuse.plist — launchd agent template
(RunAtLoad + KeepAlive) so ghost / macbook / smith can run the
mount the same way tank/architect do under systemd.
Default 'cargo build' (no --features fuse) still works on macOS
with no macFUSE installed — feature gate keeps the dep opt-in.
Historically every bin (claw-store, claw-cargo, claw-fuse)
re-declared the same 13-line 'mod ...' block at its own root.
Working but fragile — a new bin (or a new module) needed edits
in N+1 places and stayed one edit-slip away from silently
dropping something.
Now: src/lib.rs owns the shared module tree. Cargo.toml gets a
[lib] entry so cargo picks it up. claw_fuse.rs migrated as the
proof-of-concept — 13 mod lines → 4 use lines.
claw-store + claw-cargo bins still use their internal 'mod ...'
blocks and 'crate::...' paths — migrating them is mechanical
but noisy; deferred to a follow-on so this PR stays reviewable.
Both bins + tests continue to build unchanged.
387 tests pass.
deploy/scripts/rotate-snapshots.sh creates daily-YYYY-MM-DD
snapshot + prunes daily-* older than RETAIN_DAYS. Only touches
its own daily-* namespace so hand-created snapshots (release
anchors etc) never get reaped.
02:00 timer runs ahead of the 03:15 ref-sweep + 03:30 gc so
tonight's fresh snapshot pins protect its blobs from eviction.
DRY_RUN=1 for preview.
Default remains dry-run. --apply iterates the stale set and calls
RefTracking::forget per fp. Blob eviction stays a separate step
(next cluster-gc). Errors are surfaced per-fp, batch continues.
03:15 local, Persistent=true. Runs ahead of the 03:30 GC so
operators see the stale-fingerprint report before eviction lands.
Environment= drives the Gitea URL; token expected in a drop-in
(clawstor-ref-sweep.service.d/token.conf) so it doesn't sit in
the unit file. README + install recipe updated.
Omar Sobh
osobh