Command::output() captures stdout and stderr as two separate
buffers. check() was concatenating stdout-then-stderr, which throws
away chronological order entirely -- every stderr line (e.g.
"Error: send_to_host not set" from `claw-store replicate` on a node
with no downstream replication target, which is normal and expected
on architect) landed at the very end of the report regardless of
when it actually printed, making a mid-script, already-handled
condition look like a failure that happened after "DRY RUN
COMPLETE".
Fix: invoke via `bash -c "script --dry-run 2>&1"` so stderr merges
into stdout inside the shell, before either stream reaches us --
true chronological order preserved, single buffer to read.
Verified on architect: the send_to_host message now appears exactly
where it happens, inside the "taking final snapshot + replicating"
step, with DRY RUN COMPLETE correctly last.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Two problems surfaced while checking on the fleet after the shutdown-
prep button PR:
1. morpheus is configured with zfs_dataset = "none" (it has no ZFS
pool -- warm tier is a plain directory on the LVM root volume),
but nothing in the code actually implemented that as a sentinel.
cmd_snapshot/cmd_replicate and the daemon's periodic snap/repl
ticks always tried real zfs/zpool calls regardless, producing
"zfs: command not found" errors on every hourly tick and in the
shutdown-prep report. WarmConfig::zfs_enabled() now gates all four
call sites; a non-ZFS node gets a clean "nothing to
snapshot/replicate" instead of a raw shell error.
2. safe-shutdown-prep.sh's zpool-health step now checks `command -v
zpool` first instead of leaking "zpool: command not found" into
the report.
3. "we don't see the button" turned out to be page confusion: the
shutdown-prep panel lives on the per-node detail page
(/v2/nodes/<name>), not the root Fleet Health landing page. Added
a small "view detail · maintenance & shutdown prep →" hint to the
bottom of every NodeCard so it's discoverable without already
knowing to click through.
Verified against tank, architect, and morpheus -- morpheus's
shutdown-prep --dry-run report is now clean (no "command not found"
lines) both when run locally and via cross-node RPC from tank.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Wires safe-shutdown-prep.sh into the dashboard so an operator can
prep a node for hardware maintenance from a browser instead of SSH.
New RPC methods (0x20/0x21):
- ShutdownPrepCheck runs `--dry-run` to completion and returns the
full report. Never stops anything, safe to call repeatedly.
- ShutdownPrepExecute starts the real run detached (`systemd-run
--user --scope --collect`), placing it in a cgroup outside
claw-store.service's own -- the script's own step 6 stops that
service, i.e. the process that would otherwise be running it, so
it has to survive its own parent dying. Returns immediately with
a "started" message; full output lands in
/var/lib/claw-store/shutdown-prep.log for whoever's at the machine
once it's gone dark, since there's no way to stream a live result
past the point the daemon stops itself.
- Execute double-checks confirm_node_name against the peer's own
configured name server-side, on top of the aggregator's own path
match -- defense in depth for a highly consequential action.
Aggregator endpoints (admin-token gated, AuthedCaller::require_admin):
POST /api/v2/node/:name/shutdown-prep/check
POST /api/v2/node/:name/shutdown-prep/execute
Frontend: ShutdownPrepPanel on NodeDetail. Check button always
enabled; the real "stop services" button only unlocks after a ready
check, and additionally requires typing the exact node name to
confirm before it's clickable.
Also fixes a script bug found while testing this against the live
daemon process (not caught in manual interactive-shell testing): the
zpool-detection line parsed raw `mount` output positionally, which
returned the wrong field under the daemon's process context for
reasons that didn't reproduce interactively. Switched to
`df --output=source`, which is stable across both.
Verified end-to-end against tank, architect, and morpheus, including
cross-node targeting (tank's dashboard successfully triggered a
check on morpheus over the fleet RPC layer).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Node-local script an operator runs before powering a node off for
parts replacement. Guards against shutting down mid-build or with
un-pushed sync jobs, takes a final snapshot + replicates to cold,
stops the maintenance timers and dashboard, gracefully stops the
daemon (giving gossip its TimeoutStopSec=60 window to announce
departure to peers instead of relying on the 10s failure-detector
timeout), and cleanly unmounts FUSE before declaring the node safe
to power off. --dry-run runs every check for real but only prints
what the stop/unmount steps would do.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Console errors on the live dashboard (architect:7700) turned out to
be two separate problems layered together:
1. The deployed static bundle wasn't built from this repo at all --
it called /api/v2/hot-refs, /api/v2/anomalies, and
/api/v2/node/*/metrics-history, none of which exist anywhere in
this codebase on any branch (checked via `git log --all -S`).
Someone built and shipped a frontend straight to
/usr/share/claw-store/v2 without ever committing the source.
2. Separately, this repo's own committed vite.config.ts had a latent
bug: `base: '/clawstor/'`, contradicting its own comment ("served
by claw-store serve under /v2/*") and the actual backend mount in
serve.rs (`nest_service("/v2", ...)`). Checked `tailscale serve
status` on tank + architect -- neither has ever proxied a
/clawstor path, so that base would have 404'd every asset the
moment anyone rebuilt and redeployed from source.
Fix: base = '/v2/', matching the real mount. Rebuilt and redeployed
to tank + architect (orphaned build backed up to
/usr/share/claw-store/v2.bak-orphaned on both).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
V2State::from_config built its peer fan-out list solely from
[[cluster.peers]], which by definition never includes the local
node. Result: hitting a given node's /api/v2/fleet directly always
omitted that node from its own fleet view, even when perfectly
healthy -- looked like "node X is missing" from the dashboard when
X just never queried itself.
Fix: synthesize a self PeerEntry from the node's own gossip bind
address and include it in the fan-out, same as any other peer.
peer_rpc_addr()'s existing +1 port convention resolves it to the
same bind_rpc_lan/bind_rpc_tailscale the daemon already listens on.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Bind failures at startup are almost always a boot-time race against
DHCP/network-online (bind address not yet assigned to the interface).
Previously the daemon caught the error and kept running in a degraded
state with no gossip, RPC, or Prometheus endpoint and no visible
failure signal. Now it propagates the error so the process exits and
systemd's Restart=on-failure retries once the network is actually up.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
serve_v2::peer_rpc_addr derives each peer's RPC socket as lan_addr.ip():port+1.
Architect and Tank were bound on their 10G fabric NICs (10.10.0.9/10.10.0.10)
but the peer entries only carry the main-LAN gossip address, causing the
aggregator to time out when probing Tank and present a stale cert to Morpheus.
Changing bind_rpc_lan to the main-LAN addresses (10.0.0.13/10.0.0.14) fixes both.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- All three node configs (architect/tank/morpheus) now include [cluster]
section with correct 10G fabric IPs, mTLS TLS paths, and blob store root.
Architect binds gossip on 10.0.0.13:7701 and RPC on 10.10.0.9:7702 (10G
to Tank); Morpheus uses LAN 10.0.0.5 (no direct 10G).
- claw-store.service: updated description, adds clawstor-deploy to
ReadWritePaths, removes NoNewPrivileges (needed for sudo zfs snapshot).
- claw-store-serve.service: adds --v2-static-dir /usr/share/claw-store/v2
so the aggregator serves dashboard-v2 ("clawstor · command center") at /v2/.
- claw-fuse.service: new unit, uses correct --data-dir + --mount flags.
Deployed to Architect, Tank, Morpheus. Fleet CA re-initialized; new certs
signed for all three nodes and distributed. Dashboard accessible at
http://100.104.171.32:7700/v2/ aggregating Tank + Morpheus via /api/v2/fleet.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Tertiary dev node (100.123.224.84) — no ZFS, no dedicated NVMe.
Hot tier on root filesystem, snapshot/replicate timers not enabled.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
vite.config.ts sets outDir to ../claw-store/static (relative to dashboard/),
so the built assets land in claw-store/static/, not the standard dashboard/dist/.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
strip_section now strips only the [build] block before rewriting it,
so existing [alias] tables and other custom sections survive repeated
activate calls. Adds strip_leading_marker to prevent the managed-marker
comment from accumulating on each re-activation.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- Wire RpcRouter::with_repo_root at daemon startup in services.rs
using <blob_store_root>/repos. Nodes with no blob_store_root
still return NotConfigured (unchanged).
- Add end-to-end ensure→cached→release integration test that seeds
a bare git repo in a tempdir and exercises the real git-clone
path. Marked #[ignore] so CI runners without git skip silently;
runs green locally.
Layers HTTP over the R1a per-peer primitive so external callers
(clawmates, gitea runners, ops tooling) speak one URL to the
aggregator instead of dialing every peer.
Endpoints (require v2 auth, same middleware as tags/sessions):
POST /api/v2/repos/ensure {url, git_ref, workspace?}
POST /api/v2/repos/release {url, git_ref, workspace?}
Namespaced tokens are pinned to their own workspace (workspace omitted
in body → derived from token; explicit mismatch → 403). Admin/open
callers must supply workspace explicitly.
Reply shape mirrors the tag fan-out (FanoutReply) with per-peer
{peer, ok, path?, head_sha?, cached?, removed?, error?}. all_ok is
true iff every peer succeeded.
Also adds client wrappers call_repo_ensure/call_repo_release in
cluster/rpc/client.rs used by the aggregator's fan-out.
New per-peer RPCs to shallow-clone a (url, git_ref) under a caller-
provided workspace namespace, and to release the checkout. Fleet
fan-out via the aggregator ships separately in R1b.
- src/cluster/repo_ensure.rs: request/reply types + derive_path
(traversal-safe, blake3-hashed url segment), ensure_repo (cache
hit → rev-parse HEAD, else remove-and-reclone with 5-min timeout),
release_repo (idempotent rm)
- src/cluster/rpc.rs: Method::RepoEnsure=0x1e, RepoRelease=0x1f,
RpcRouter::with_repo_root builder, two dispatch arms returning
NotConfigured when repo_root is unset
- src/cluster.rs: pub mod repo_ensure
- src/actions.rs: fix pre-existing test-only Config init missing the
aggregator field (unblocks lib tests)
Tests: 6 unit tests covering path derivation determinism, ref/url
independence, traversal safety, and sanitizer edge cases. All pass.
Backend:
* New DashboardProject { repo, cache_bytes, fingerprint_count,
refs, first_seen_unix, last_seen_unix, tier }.
* Handler build_projects() joins ref-tracking with the ref-store
and blob-store: for each recorded fp, resolve fp → blob-id via
RefStore::{get_stamped, get} then sum blob sizes per repo.
* Tier is a wall-clock function of last_seen_unix:
active < 24h, recent < 7d, else idle.
* DashboardStorageReply gains `projects: Vec<DashboardProject>`
(serde-default so older clients still parse).
Aggregator:
* New /api/v2/projects endpoint. Fans out DashboardStorage to
every peer, tags each project row with its originating node,
returns hottest-first.
Frontend:
* New ProjectsPanel component appended to the FleetHealth
landing. Groups by repo, one row per project with tier badge
(active/recent/idle), per-node pill badges, cache-size sum,
ref list, last-activity age.
* Empty state explains how to populate: claw-cargo build with
--repo + --git-ref (or CLAWSTOR_REPO/CLAWSTOR_GIT_REF env
vars in CI).
Data populates automatically as each cache-put runs. Existing
demo entry on tank (clawverse/clawstor · main · c384a4...) will
surface after redeploy.
Reworks the SPA around the new DashboardStatus fields. The
landing is now a fleet-health dashboard aimed at a layperson —
disk gauges, mount ✓/✗, cache hit rate, next scheduled job.
No hex, no primitives.
Nav restructure:
* Primary: 'Fleet health' (just the landing).
* 'View Advanced ▾' dropdown reveals: Blobs / Tags / Refs /
Snapshots / Ref-tracking. Routes moved under /advanced/*.
New components:
* StorageBar — horizontal used/total bar with pinned/evictable
split, health-color threshold at 60/85%.
* NodeCard — traffic-light dot + disk + hot tier bars + mount
state + cache hit rate + next-timer countdown. Whole card
is a link into node detail.
New CommandCenter:
* Fleet-wide storage roll-up card (sum of every node's disk).
* Grid of NodeCards.
* 10s poll cadence retained from previous version.
Existing StorageBrowser + RefTrackingPage moved behind
/advanced/* routes; internal component code untouched.
Extends DashboardStatus with the primitives the FleetHealth
landing needs. All new fields serde-default to None/[]/false so
old clients (aggregator running an older build) still parse the
reply.
Added fields:
* filesystem — statvfs on blob_store_root: total / used / avail.
* hot — {used_bytes, max_bytes, pinned_bytes?}, read from
gossip (already published every 10s).
* mount — probes /proc/mounts for ~/clawstor-mount by
convention. { path, active }.
* cache — router.metrics().snapshot() summarised as
{hits, misses, bytes_served, bytes_ingested,
hit_rate}. hits = get_ref+get_tag+get_chunk hits,
misses similarly — the counters that matter for
claw-cargo build outcomes.
* timers — well-known set (scrub / gc / ref-sweep /
snapshot-rotate) queried via `systemctl --user
show`. next_fire_unix + last_result per timer.
Aggregator NodeStatusV2 mirrors the same fields verbatim so the
new frontend can consume them without further backend hops.
Follow-on PRs:
* PR 2 — new FleetHealth landing page + node-detail rework
* PR 3 — polish (pinned_bytes derivation, activity timeline,
per-repo grouping)
Bare <Link href='/'> was navigating to tailnet root =
OpenClaw (which mounts /). Wrap App in <Router base=...>
detected from window.location so links + route matching
prefix the mount path. Falls back to no-base for local
:7700/v2/ dev too.
All storage rows now carry a 'node' field from the aggregator.
Adds a small green-pill component that renders the source node
name; click drills into that node's detail page.
Also fixes the TS interfaces to match the new backend shape
(node field added to Blob/Tag/Ref/Snapshot/RefTracking types).
Storage tabs 404'd because the aggregator only had /fleet + /node/:name
after the pivot. Adds:
* New RPC method DashboardStorage = 0x1d — one round trip returns
tags (full), snapshots (full), ref-tracking (full), blobs
(first 200 by id), refs (first 200 by fp) for the responding
daemon.
* Client wrapper call_dashboard_storage.
* Aggregator fans out to every peer, tags each row with the
originating node, sorts + returns:
GET /api/v2/storage/blobs
GET /api/v2/storage/tags
GET /api/v2/storage/refs
GET /api/v2/storage/snapshots
GET /api/v2/storage/ref-tracking
QuicClient promoted to Arc<QuicClient> inside V2State so the
per-peer JoinSet can hand it to spawned tasks without recreating
the endpoint.
Prior ./ (fully-relative) fix broke when browser hit /clawstor
without trailing slash — ./assets/… resolved to /assets/… at
tailnet root, 404. Absolute /clawstor/ is deploy-path-coupled
but no-slash-safe. Verified: no-slash HTML 200, CSS 200,
JS 200, /clawstor/api/v2/fleet 200.
Browser hit /clawstor/ then requested /v2/assets/index-XXX.css
(absolute path baked in by vite base '/v2/') which 404'd because
Tailscale Serve only mounted /clawstor and /clawstor/api. Switch
vite base to './' — assets resolve relative to whatever URL the
SPA loaded from. Works for local (:7700/v2/) + Tailscale
(/clawstor) with no config coupling.
Pivot from per-node dashboard to single-pane-of-glass. The
aggregator (typically the operator's laptop) holds a fleet-CA
leaf cert + the peer list; each dashboard request fans out to
every peer over the existing QUIC/mTLS cluster port and issues
the new DashboardStatus RPC. Peers don't need to run any HTTP
server of their own.
Backend:
* New RPC method DashboardStatus = 0x1c
* Server handler reads BlobStore / TagStore / RefStore /
SnapshotStore / RefTracking counts + on-disk bytes + rustc
release. Cheap: 5 filesystem walks per request.
* Client wrapper call_dashboard_status
* serve_v2 rewritten as aggregator: V2State holds a QuicClient +
peer list from `[[cluster.peers]]`. Endpoints:
GET /api/v2/fleet fan-out to every peer, parallel
GET /api/v2/node/:name/status one peer, on-demand
Failed peers surface as { online: false, error: "..." } cards
instead of dropping.
serve.rs graceful degrade: v2 aggregator routes only mount when
[cluster.tls] is set. Static SPA still serves at /v2/* even
without an aggregator config so operators see the SPA's built-in
"config missing" error.
Deployment model (this session):
* Aggregator runs on quantum (Mac) with a signed leaf.
* Fleet daemons run cluster-only — no HTTP dashboard anywhere
on tank/architect/morpheus. The clawstor-dashboard.service
systemd units on the fleet are being retired.
Frontend rework to consume /api/v2/fleet ships in the next PR.
React 19 + Vite + Tailwind + wouter (tiny router, no external
state library). Consumes the /api/v2/* endpoints shipped in PR 1.
Serves under /v2/* so the legacy dashboard at / stays live.
Pages:
* CommandCenter (/) — fleet strip + this-node stat tiles
* NodeDetail (/nodes/:name) — per-node deep dive
* StorageBrowser (/storage/{blobs,tags,refs,snapshots}) — tables
with prefix filter
* RefTrackingPage (/refs/tracking) — grouped by repo
Backend changes:
* claw-store serve grows --v2-static-dir <path>
* build_app split into build_app_with_v2 for the extra static
mount
* /v2/* falls through to index.html so wouter client routing works
New systemd unit: clawstor-dashboard.service. Points at both
static dirs; installs on any node.
dashboard/ (legacy) untouched. dashboard-v2/ built to
target/dashboard-v2/dist for deploy.
Deploy sequence per node:
1. cp target/release/claw-store ~/clawstor-deploy/
2. rsync dashboard-v2/dist/ ~/clawstor-deploy/dashboard-v2/
3. cp deploy/systemd/clawstor-dashboard.service ~/.config/systemd/user/
4. systemctl --user daemon-reload && enable --now clawstor-dashboard.service
Cross-node fan-out for /api/v2/node/:name/status is PR 3.
Action POSTs (scrub/gc/snapshot/pin) are PR 4.
osobh