- services.rs: add prefetch_task that wakes every 10 min, checks if
hot-tier fill is below 60%, then pulls up to 5 blobs per live peer
that this node doesn't already have locally; sets the ref locally
after each successful pull so future GetRef calls are pure local hits
- rpc.rs: make pull_blob_locally pub(crate) so services.rs can call it;
add outbound_client() accessor on RpcRouter for the prefetch task
The prefetch task is a no-op when hot-tier fill >= 60% or when the node
has no outbound QUIC client configured (i.e. in test environments).
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Parse Cargo.lock TOML and emit a canonical `name@version#checksum` line
per package (sorted) instead of hashing the raw text. This makes the
fingerprint stable across:
- Cargo-generated header comments and blank-line drift
- TOML whitespace reformatting between cargo versions
The hash domain changes from v1 → v2, which invalidates old cached
blobs intentionally — existing caches will miss once then rebuild under
the new, stable fingerprint.
Adds three new unit tests: canonicalize_lock_is_stable_across_comments,
canonicalize_lock_sorts_packages, canonicalize_lock_fallback_on_invalid_toml.
385 tests pass.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- gossip.rs: add clawstor.health_score key constant, health_score field on
PeerView, set_health_score() method on ClusterGossip, extraction in
peer_view_from_state(); fix two test initializers for the new field
- rpc.rs: make read_cpu_load_1m() pub(crate); add cpu_load_1m + health_score
to DashboardStatusReply with Lifeguard-style composite scoring
- services.rs: compute and publish health_score each metrics_gossip tick so
peers can observe each other's stress without an RPC round-trip
- dashboard: propagate health_score and cpu_load_1m through api.ts/NodeCard;
add NodeHistorySparklines (recharts sparklines) and drift-adaptive anomaly
banner in CommandCenter
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
serve_v2: Welford online stats per node (hot_pct, hit_rate, fs_pct,
chunk_miss_rate) updated on every poller tick. Anomaly score = sum of z²;
warn at 9, alert at 16. Exposed via GET /api/v2/anomalies. Score embedded
in MetricSample so sparklines can also surface it.
hot: Replace pure-LRU gc_by_space with size×age eviction scoring.
Candidates are ranked by ln(size) × ln(age_secs); pinned projects immune.
Evicts the most space with the least cost rather than just the oldest entry.
dashboard: Fleet anomaly banner in CommandCenter (red/amber) + per-node
"anomaly"/"drift" badge on NodeCard. Anomalies polled every 10s alongside
the fleet endpoint.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Add a 24h per-node metrics ring buffer to the aggregator (1440 samples
at 1-min resolution via background poller), expose via
GET /api/v2/node/:name/metrics-history, and wire up hot-tier usage and
cache hit-rate sparklines in NodeCard using Recharts AreaChart/LineChart.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Backend (rpc.rs):
- DashboardStatusReply gains daemon_started_unix (proxy for daemon restart time)
- CacheSummary gains per-type breakdown: get_ref_hits/misses, get_tag_hits/misses,
has_chunk_hits/misses — dedup efficiency is now visible in the API response
Frontend (api.ts, NodeCard.tsx):
- NodeStatusV2 type carries daemon_started_unix and expanded CacheSummary
- fmtUptime() helper renders "up 3d 14h" from a unix timestamp
- NodeCard now shows: uptime, available disk bytes in the bar label,
dedup efficiency % (has_chunk hit rate), timer last-result text for
failed timers
Infrastructure (T3.1):
- FQ qdisc applied on all 3 nodes (Architect enp11s0+enp5s0f1, Tank same,
Morpheus eno1) for precise QUIC packet pacing per QUIC Steps paper
- GSO already on on Architect and Tank 10G NICs
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
serve_v2::peer_rpc_addr derives each peer's RPC socket as lan_addr.ip():port+1.
Architect and Tank were bound on their 10G fabric NICs (10.10.0.9/10.10.0.10)
but the peer entries only carry the main-LAN gossip address, causing the
aggregator to time out when probing Tank and present a stale cert to Morpheus.
Changing bind_rpc_lan to the main-LAN addresses (10.0.0.13/10.0.0.14) fixes both.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- All three node configs (architect/tank/morpheus) now include [cluster]
section with correct 10G fabric IPs, mTLS TLS paths, and blob store root.
Architect binds gossip on 10.0.0.13:7701 and RPC on 10.10.0.9:7702 (10G
to Tank); Morpheus uses LAN 10.0.0.5 (no direct 10G).
- claw-store.service: updated description, adds clawstor-deploy to
ReadWritePaths, removes NoNewPrivileges (needed for sudo zfs snapshot).
- claw-store-serve.service: adds --v2-static-dir /usr/share/claw-store/v2
so the aggregator serves dashboard-v2 ("clawstor · command center") at /v2/.
- claw-fuse.service: new unit, uses correct --data-dir + --mount flags.
Deployed to Architect, Tank, Morpheus. Fleet CA re-initialized; new certs
signed for all three nodes and distributed. Dashboard accessible at
http://100.104.171.32:7700/v2/ aggregating Tank + Morpheus via /api/v2/fleet.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Tertiary dev node (100.123.224.84) — no ZFS, no dedicated NVMe.
Hot tier on root filesystem, snapshot/replicate timers not enabled.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
vite.config.ts sets outDir to ../claw-store/static (relative to dashboard/),
so the built assets land in claw-store/static/, not the standard dashboard/dist/.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
strip_section now strips only the [build] block before rewriting it,
so existing [alias] tables and other custom sections survive repeated
activate calls. Adds strip_leading_marker to prevent the managed-marker
comment from accumulating on each re-activation.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Backend:
* New DashboardProject { repo, cache_bytes, fingerprint_count,
refs, first_seen_unix, last_seen_unix, tier }.
* Handler build_projects() joins ref-tracking with the ref-store
and blob-store: for each recorded fp, resolve fp → blob-id via
RefStore::{get_stamped, get} then sum blob sizes per repo.
* Tier is a wall-clock function of last_seen_unix:
active < 24h, recent < 7d, else idle.
* DashboardStorageReply gains `projects: Vec<DashboardProject>`
(serde-default so older clients still parse).
Aggregator:
* New /api/v2/projects endpoint. Fans out DashboardStorage to
every peer, tags each project row with its originating node,
returns hottest-first.
Frontend:
* New ProjectsPanel component appended to the FleetHealth
landing. Groups by repo, one row per project with tier badge
(active/recent/idle), per-node pill badges, cache-size sum,
ref list, last-activity age.
* Empty state explains how to populate: claw-cargo build with
--repo + --git-ref (or CLAWSTOR_REPO/CLAWSTOR_GIT_REF env
vars in CI).
Data populates automatically as each cache-put runs. Existing
demo entry on tank (clawverse/clawstor · main · c384a4...) will
surface after redeploy.
Reworks the SPA around the new DashboardStatus fields. The
landing is now a fleet-health dashboard aimed at a layperson —
disk gauges, mount ✓/✗, cache hit rate, next scheduled job.
No hex, no primitives.
Nav restructure:
* Primary: 'Fleet health' (just the landing).
* 'View Advanced ▾' dropdown reveals: Blobs / Tags / Refs /
Snapshots / Ref-tracking. Routes moved under /advanced/*.
New components:
* StorageBar — horizontal used/total bar with pinned/evictable
split, health-color threshold at 60/85%.
* NodeCard — traffic-light dot + disk + hot tier bars + mount
state + cache hit rate + next-timer countdown. Whole card
is a link into node detail.
New CommandCenter:
* Fleet-wide storage roll-up card (sum of every node's disk).
* Grid of NodeCards.
* 10s poll cadence retained from previous version.
Existing StorageBrowser + RefTrackingPage moved behind
/advanced/* routes; internal component code untouched.
Extends DashboardStatus with the primitives the FleetHealth
landing needs. All new fields serde-default to None/[]/false so
old clients (aggregator running an older build) still parse the
reply.
Added fields:
* filesystem — statvfs on blob_store_root: total / used / avail.
* hot — {used_bytes, max_bytes, pinned_bytes?}, read from
gossip (already published every 10s).
* mount — probes /proc/mounts for ~/clawstor-mount by
convention. { path, active }.
* cache — router.metrics().snapshot() summarised as
{hits, misses, bytes_served, bytes_ingested,
hit_rate}. hits = get_ref+get_tag+get_chunk hits,
misses similarly — the counters that matter for
claw-cargo build outcomes.
* timers — well-known set (scrub / gc / ref-sweep /
snapshot-rotate) queried via `systemctl --user
show`. next_fire_unix + last_result per timer.
Aggregator NodeStatusV2 mirrors the same fields verbatim so the
new frontend can consume them without further backend hops.
Follow-on PRs:
* PR 2 — new FleetHealth landing page + node-detail rework
* PR 3 — polish (pinned_bytes derivation, activity timeline,
per-repo grouping)
Bare <Link href='/'> was navigating to tailnet root =
OpenClaw (which mounts /). Wrap App in <Router base=...>
detected from window.location so links + route matching
prefix the mount path. Falls back to no-base for local
:7700/v2/ dev too.
All storage rows now carry a 'node' field from the aggregator.
Adds a small green-pill component that renders the source node
name; click drills into that node's detail page.
Also fixes the TS interfaces to match the new backend shape
(node field added to Blob/Tag/Ref/Snapshot/RefTracking types).
Storage tabs 404'd because the aggregator only had /fleet + /node/:name
after the pivot. Adds:
* New RPC method DashboardStorage = 0x1d — one round trip returns
tags (full), snapshots (full), ref-tracking (full), blobs
(first 200 by id), refs (first 200 by fp) for the responding
daemon.
* Client wrapper call_dashboard_storage.
* Aggregator fans out to every peer, tags each row with the
originating node, sorts + returns:
GET /api/v2/storage/blobs
GET /api/v2/storage/tags
GET /api/v2/storage/refs
GET /api/v2/storage/snapshots
GET /api/v2/storage/ref-tracking
QuicClient promoted to Arc<QuicClient> inside V2State so the
per-peer JoinSet can hand it to spawned tasks without recreating
the endpoint.
Prior ./ (fully-relative) fix broke when browser hit /clawstor
without trailing slash — ./assets/… resolved to /assets/… at
tailnet root, 404. Absolute /clawstor/ is deploy-path-coupled
but no-slash-safe. Verified: no-slash HTML 200, CSS 200,
JS 200, /clawstor/api/v2/fleet 200.
Browser hit /clawstor/ then requested /v2/assets/index-XXX.css
(absolute path baked in by vite base '/v2/') which 404'd because
Tailscale Serve only mounted /clawstor and /clawstor/api. Switch
vite base to './' — assets resolve relative to whatever URL the
SPA loaded from. Works for local (:7700/v2/) + Tailscale
(/clawstor) with no config coupling.
Pivot from per-node dashboard to single-pane-of-glass. The
aggregator (typically the operator's laptop) holds a fleet-CA
leaf cert + the peer list; each dashboard request fans out to
every peer over the existing QUIC/mTLS cluster port and issues
the new DashboardStatus RPC. Peers don't need to run any HTTP
server of their own.
Backend:
* New RPC method DashboardStatus = 0x1c
* Server handler reads BlobStore / TagStore / RefStore /
SnapshotStore / RefTracking counts + on-disk bytes + rustc
release. Cheap: 5 filesystem walks per request.
* Client wrapper call_dashboard_status
* serve_v2 rewritten as aggregator: V2State holds a QuicClient +
peer list from `[[cluster.peers]]`. Endpoints:
GET /api/v2/fleet fan-out to every peer, parallel
GET /api/v2/node/:name/status one peer, on-demand
Failed peers surface as { online: false, error: "..." } cards
instead of dropping.
serve.rs graceful degrade: v2 aggregator routes only mount when
[cluster.tls] is set. Static SPA still serves at /v2/* even
without an aggregator config so operators see the SPA's built-in
"config missing" error.
Deployment model (this session):
* Aggregator runs on quantum (Mac) with a signed leaf.
* Fleet daemons run cluster-only — no HTTP dashboard anywhere
on tank/architect/morpheus. The clawstor-dashboard.service
systemd units on the fleet are being retired.
Frontend rework to consume /api/v2/fleet ships in the next PR.
React 19 + Vite + Tailwind + wouter (tiny router, no external
state library). Consumes the /api/v2/* endpoints shipped in PR 1.
Serves under /v2/* so the legacy dashboard at / stays live.
Pages:
* CommandCenter (/) — fleet strip + this-node stat tiles
* NodeDetail (/nodes/:name) — per-node deep dive
* StorageBrowser (/storage/{blobs,tags,refs,snapshots}) — tables
with prefix filter
* RefTrackingPage (/refs/tracking) — grouped by repo
Backend changes:
* claw-store serve grows --v2-static-dir <path>
* build_app split into build_app_with_v2 for the extra static
mount
* /v2/* falls through to index.html so wouter client routing works
New systemd unit: clawstor-dashboard.service. Points at both
static dirs; installs on any node.
dashboard/ (legacy) untouched. dashboard-v2/ built to
target/dashboard-v2/dist for deploy.
Deploy sequence per node:
1. cp target/release/claw-store ~/clawstor-deploy/
2. rsync dashboard-v2/dist/ ~/clawstor-deploy/dashboard-v2/
3. cp deploy/systemd/clawstor-dashboard.service ~/.config/systemd/user/
4. systemctl --user daemon-reload && enable --now clawstor-dashboard.service
Cross-node fan-out for /api/v2/node/:name/status is PR 3.
Action POSTs (scrub/gc/snapshot/pin) are PR 4.
deploy/scripts/fleet-status.sh iterates NODES (default 'tank
architect morpheus') and prints per-host: daemon state, FUSE
mount + layer listing, blob-store byte count, and each timer's
next-fire + last-result.
Fits in half a screen per node; useful as a smoke check before
+ after any fleet-wide change. Read-only over ssh — safe to
run from any workstation with keys.
Live smoke on morpheus with the previous unit failed:
Error: mounting FUSE at /home/osobh/clawstor-mount
Caused by: Operation not permitted (os error 1)
But 'systemd-run --user' with the exact same binary worked, and
the manual invocation worked. The delta was our ExecStartPre
chain (mkdir + fusermount3 -u -z) — on some Ubuntu 24.04 builds
that combination poisons the subsequent mount syscall even with
the -z lazy flag + ignored exit code.
Fix: drop the pre-mkdir + pre-unmount. Operator creates the
mount dir manually once (documented in README). Restart handling
falls to systemd's Restart=on-failure + ExecStop unmount.
Verified live on morpheus with the minimal unit — active +
mount visible in <3s.
Cargo.toml: fuser dep now target-gated to Linux + macOS. macOS
build requires macFUSE (brew install --cask macfuse) + pkg-config
before 'cargo build --features fuse' works.
deploy/macos/README.md — one-time prereqs, build steps, mount /
umount, known differences (no AllowOther, unmount is 'umount'
not fusermount3 -u).
deploy/macos/claw-fuse.plist — launchd agent template
(RunAtLoad + KeepAlive) so ghost / macbook / smith can run the
mount the same way tank/architect do under systemd.
Default 'cargo build' (no --features fuse) still works on macOS
with no macFUSE installed — feature gate keeps the dep opt-in.
Historically every bin (claw-store, claw-cargo, claw-fuse)
re-declared the same 13-line 'mod ...' block at its own root.
Working but fragile — a new bin (or a new module) needed edits
in N+1 places and stayed one edit-slip away from silently
dropping something.
Now: src/lib.rs owns the shared module tree. Cargo.toml gets a
[lib] entry so cargo picks it up. claw_fuse.rs migrated as the
proof-of-concept — 13 mod lines → 4 use lines.
claw-store + claw-cargo bins still use their internal 'mod ...'
blocks and 'crate::...' paths — migrating them is mechanical
but noisy; deferred to a follow-on so this PR stays reviewable.
Both bins + tests continue to build unchanged.
387 tests pass.
deploy/scripts/rotate-snapshots.sh creates daily-YYYY-MM-DD
snapshot + prunes daily-* older than RETAIN_DAYS. Only touches
its own daily-* namespace so hand-created snapshots (release
anchors etc) never get reaped.
02:00 timer runs ahead of the 03:15 ref-sweep + 03:30 gc so
tonight's fresh snapshot pins protect its blobs from eviction.
DRY_RUN=1 for preview.
osobh