All storage rows now carry a 'node' field from the aggregator.
Adds a small green-pill component that renders the source node
name; click drills into that node's detail page.
Also fixes the TS interfaces to match the new backend shape
(node field added to Blob/Tag/Ref/Snapshot/RefTracking types).
Storage tabs 404'd because the aggregator only had /fleet + /node/:name
after the pivot. Adds:
* New RPC method DashboardStorage = 0x1d — one round trip returns
tags (full), snapshots (full), ref-tracking (full), blobs
(first 200 by id), refs (first 200 by fp) for the responding
daemon.
* Client wrapper call_dashboard_storage.
* Aggregator fans out to every peer, tags each row with the
originating node, sorts + returns:
GET /api/v2/storage/blobs
GET /api/v2/storage/tags
GET /api/v2/storage/refs
GET /api/v2/storage/snapshots
GET /api/v2/storage/ref-tracking
QuicClient promoted to Arc<QuicClient> inside V2State so the
per-peer JoinSet can hand it to spawned tasks without recreating
the endpoint.
Prior ./ (fully-relative) fix broke when browser hit /clawstor
without trailing slash — ./assets/… resolved to /assets/… at
tailnet root, 404. Absolute /clawstor/ is deploy-path-coupled
but no-slash-safe. Verified: no-slash HTML 200, CSS 200,
JS 200, /clawstor/api/v2/fleet 200.
Browser hit /clawstor/ then requested /v2/assets/index-XXX.css
(absolute path baked in by vite base '/v2/') which 404'd because
Tailscale Serve only mounted /clawstor and /clawstor/api. Switch
vite base to './' — assets resolve relative to whatever URL the
SPA loaded from. Works for local (:7700/v2/) + Tailscale
(/clawstor) with no config coupling.
Pivot from per-node dashboard to single-pane-of-glass. The
aggregator (typically the operator's laptop) holds a fleet-CA
leaf cert + the peer list; each dashboard request fans out to
every peer over the existing QUIC/mTLS cluster port and issues
the new DashboardStatus RPC. Peers don't need to run any HTTP
server of their own.
Backend:
* New RPC method DashboardStatus = 0x1c
* Server handler reads BlobStore / TagStore / RefStore /
SnapshotStore / RefTracking counts + on-disk bytes + rustc
release. Cheap: 5 filesystem walks per request.
* Client wrapper call_dashboard_status
* serve_v2 rewritten as aggregator: V2State holds a QuicClient +
peer list from `[[cluster.peers]]`. Endpoints:
GET /api/v2/fleet fan-out to every peer, parallel
GET /api/v2/node/:name/status one peer, on-demand
Failed peers surface as { online: false, error: "..." } cards
instead of dropping.
serve.rs graceful degrade: v2 aggregator routes only mount when
[cluster.tls] is set. Static SPA still serves at /v2/* even
without an aggregator config so operators see the SPA's built-in
"config missing" error.
Deployment model (this session):
* Aggregator runs on quantum (Mac) with a signed leaf.
* Fleet daemons run cluster-only — no HTTP dashboard anywhere
on tank/architect/morpheus. The clawstor-dashboard.service
systemd units on the fleet are being retired.
Frontend rework to consume /api/v2/fleet ships in the next PR.
React 19 + Vite + Tailwind + wouter (tiny router, no external
state library). Consumes the /api/v2/* endpoints shipped in PR 1.
Serves under /v2/* so the legacy dashboard at / stays live.
Pages:
* CommandCenter (/) — fleet strip + this-node stat tiles
* NodeDetail (/nodes/:name) — per-node deep dive
* StorageBrowser (/storage/{blobs,tags,refs,snapshots}) — tables
with prefix filter
* RefTrackingPage (/refs/tracking) — grouped by repo
Backend changes:
* claw-store serve grows --v2-static-dir <path>
* build_app split into build_app_with_v2 for the extra static
mount
* /v2/* falls through to index.html so wouter client routing works
New systemd unit: clawstor-dashboard.service. Points at both
static dirs; installs on any node.
dashboard/ (legacy) untouched. dashboard-v2/ built to
target/dashboard-v2/dist for deploy.
Deploy sequence per node:
1. cp target/release/claw-store ~/clawstor-deploy/
2. rsync dashboard-v2/dist/ ~/clawstor-deploy/dashboard-v2/
3. cp deploy/systemd/clawstor-dashboard.service ~/.config/systemd/user/
4. systemctl --user daemon-reload && enable --now clawstor-dashboard.service
Cross-node fan-out for /api/v2/node/:name/status is PR 3.
Action POSTs (scrub/gc/snapshot/pin) are PR 4.
deploy/scripts/fleet-status.sh iterates NODES (default 'tank
architect morpheus') and prints per-host: daemon state, FUSE
mount + layer listing, blob-store byte count, and each timer's
next-fire + last-result.
Fits in half a screen per node; useful as a smoke check before
+ after any fleet-wide change. Read-only over ssh — safe to
run from any workstation with keys.
Live smoke on morpheus with the previous unit failed:
Error: mounting FUSE at /home/osobh/clawstor-mount
Caused by: Operation not permitted (os error 1)
But 'systemd-run --user' with the exact same binary worked, and
the manual invocation worked. The delta was our ExecStartPre
chain (mkdir + fusermount3 -u -z) — on some Ubuntu 24.04 builds
that combination poisons the subsequent mount syscall even with
the -z lazy flag + ignored exit code.
Fix: drop the pre-mkdir + pre-unmount. Operator creates the
mount dir manually once (documented in README). Restart handling
falls to systemd's Restart=on-failure + ExecStop unmount.
Verified live on morpheus with the minimal unit — active +
mount visible in <3s.
Cargo.toml: fuser dep now target-gated to Linux + macOS. macOS
build requires macFUSE (brew install --cask macfuse) + pkg-config
before 'cargo build --features fuse' works.
deploy/macos/README.md — one-time prereqs, build steps, mount /
umount, known differences (no AllowOther, unmount is 'umount'
not fusermount3 -u).
deploy/macos/claw-fuse.plist — launchd agent template
(RunAtLoad + KeepAlive) so ghost / macbook / smith can run the
mount the same way tank/architect do under systemd.
Default 'cargo build' (no --features fuse) still works on macOS
with no macFUSE installed — feature gate keeps the dep opt-in.
Historically every bin (claw-store, claw-cargo, claw-fuse)
re-declared the same 13-line 'mod ...' block at its own root.
Working but fragile — a new bin (or a new module) needed edits
in N+1 places and stayed one edit-slip away from silently
dropping something.
Now: src/lib.rs owns the shared module tree. Cargo.toml gets a
[lib] entry so cargo picks it up. claw_fuse.rs migrated as the
proof-of-concept — 13 mod lines → 4 use lines.
claw-store + claw-cargo bins still use their internal 'mod ...'
blocks and 'crate::...' paths — migrating them is mechanical
but noisy; deferred to a follow-on so this PR stays reviewable.
Both bins + tests continue to build unchanged.
387 tests pass.
deploy/scripts/rotate-snapshots.sh creates daily-YYYY-MM-DD
snapshot + prunes daily-* older than RETAIN_DAYS. Only touches
its own daily-* namespace so hand-created snapshots (release
anchors etc) never get reaped.
02:00 timer runs ahead of the 03:15 ref-sweep + 03:30 gc so
tonight's fresh snapshot pins protect its blobs from eviction.
DRY_RUN=1 for preview.
Default remains dry-run. --apply iterates the stale set and calls
RefTracking::forget per fp. Blob eviction stays a separate step
(next cluster-gc). Errors are surfaced per-fp, batch continues.
03:15 local, Persistent=true. Runs ahead of the 03:30 GC so
operators see the stale-fingerprint report before eviction lands.
Environment= drives the Gitea URL; token expected in a drop-in
(clawstor-ref-sweep.service.d/token.conf) so it doesn't sit in
the unit file. README + install recipe updated.
03:30 local, Persistent=true. Ordered before the Sun 04:00
scrub so scrub reads a fresh post-GC layout.
Default ExecStart is orphan-chunk sweep only (safe on any
node). Fleets that want LRU size-cap eviction add a drop-in:
systemctl --user edit clawstor-gc.service
[Service]
ExecStart=
ExecStart=%h/clawstor-deploy/claw-store --config ... cluster-gc --evict-to-gb 200
README updated.
Deploy artifact. Sunday 04:00 local, Persistent=true so a
box that missed the wall-clock moment fires on next boot.
Timer + Service pair — both go under
~/.config/systemd/user/. The service is Type=oneshot; timer
drives it. Non-zero exit from cluster-scrub (integrity issue)
surfaces via systemd failed state; journalctl has the details.
README updated with install recipe.
<mount>/refs/<fp-hex> file, content = blob that fp points at
Adds RefStore::list() unioning legacy refs/ and stamped refs-v2/.
FUSE wires the new dir. On lookup, tries get_stamped first then
legacy — matches the resolution order used by claw-cargo build.
Ref files piggy-back the blob inode (same content), so a hex
readable via /blobs/<hex> and /refs/<fp> shares the inode. cheap.
status=203/EXEC means the exec path didn't resolve. systemd
expands %h (home) at parse time but does NOT expand ${VAR}
in the ExecStart executable position — only in ExecStart args.
Rewrote to use %h/clawstor-deploy/claw-fuse directly.
Deploy artifact — not built into the binary tree. `deploy/systemd/
clawstor-fuse.service` + README with the install recipe.
Unit shape:
* Type=simple, blocks on the FUSE binary (unmount = SIGTERM).
* ExecStartPre = mkdir -p mount, best-effort lazy unmount of any
stale prior mount (guards against a hard SIGKILL leaving the
kernel with a dangling mount).
* Restart=on-failure, RestartSec=5 — recover from transient IO
errors without operator involvement.
* Environment= for CLAWSTOR_FUSE_BIN / DATA / MOUNT so a
`systemctl edit` drop-in retargets without editing the unit file.
Dependency chain: clawstor-fuse.service After= + Wants=
clawstor-cluster.service. FUSE reads only the on-disk state so
this is technically not needed for correctness — but keeps the
mount from spinning up on a node where the daemon is broken.
Companion to the previous list/get fix. Same layer-split problem:
* delete() only unlinked tags/, leaving stamped tags-v2/ behind.
Result: `unpin` prints \"no such tag\" for pins created via
Phase 3c+ RPC even though the tag is right there on disk.
* contains() only checked tags/. Same false-negative.
Fix: both APIs now inspect BOTH layers. delete() unlinks
whichever files exist (either or both) AND removes the TTL
expiry sidecar if present. Returns true when anything was
actually removed.
Legacy behavior preserved: tests unchanged, 381 tests pass.
osobh
Omar Sobh