React 19 + Vite + Tailwind + wouter (tiny router, no external
state library). Consumes the /api/v2/* endpoints shipped in PR 1.
Serves under /v2/* so the legacy dashboard at / stays live.
Pages:
* CommandCenter (/) — fleet strip + this-node stat tiles
* NodeDetail (/nodes/:name) — per-node deep dive
* StorageBrowser (/storage/{blobs,tags,refs,snapshots}) — tables
with prefix filter
* RefTrackingPage (/refs/tracking) — grouped by repo
Backend changes:
* claw-store serve grows --v2-static-dir <path>
* build_app split into build_app_with_v2 for the extra static
mount
* /v2/* falls through to index.html so wouter client routing works
New systemd unit: clawstor-dashboard.service. Points at both
static dirs; installs on any node.
dashboard/ (legacy) untouched. dashboard-v2/ built to
target/dashboard-v2/dist for deploy.
Deploy sequence per node:
1. cp target/release/claw-store ~/clawstor-deploy/
2. rsync dashboard-v2/dist/ ~/clawstor-deploy/dashboard-v2/
3. cp deploy/systemd/clawstor-dashboard.service ~/.config/systemd/user/
4. systemctl --user daemon-reload && enable --now clawstor-dashboard.service
Cross-node fan-out for /api/v2/node/:name/status is PR 3.
Action POSTs (scrub/gc/snapshot/pin) are PR 4.
Default remains dry-run. --apply iterates the stale set and calls
RefTracking::forget per fp. Blob eviction stays a separate step
(next cluster-gc). Errors are surfaced per-fp, batch continues.
Same shape as Phase 8c did for cluster-peer-status + cluster-repair.
New flags: --tailscale-addr (optional) + --lan-probe-ms (default 200).
Route (LAN vs tailnet) printed on the output. Zero flag = identical
to pre-8 single-addr behavior.
Fourth of four operator-facing CLIs now routing-aware
(cluster-peer-status, cluster-repair, cluster-ping done; cluster-ping
was the last outstanding one).
No new tests: pure glue over connect_lan_first, which has its own
unit coverage.
Wires the operator CLIs to the Phase 8b connect_lan_first primitive.
Roaming ops (laptop on LTE, coffee-shop wifi) can now pass a
tailnet address alongside the usual --rpc-addr and get the
LAN-first-with-fallback behavior automatically.
New flags on both cluster-peer-status and cluster-repair:
* --tailscale-addr <addr> — optional tailnet RPC socket. When
set, --rpc-addr is tried first with
a short deadline, then this on
failure/timeout.
* --lan-probe-ms <ms> — LAN probe deadline. Default 200
matches the arch doc.
Zero flag → byte-identical to pre-8c behavior (single-addr dial).
Both flags → chosen route printed in the output header so
operators can see whether LAN or tailnet won.
No new tests: this is thin glue over connect_lan_first, which
already has its own unit coverage. Smoke test live on tank
against architect (LAN), and against fake unroutable + real
tailnet exercises both branches.
First slice of Phase 8 (roaming client identity). Adds a helper
that mints a leaf cert whose SANs include this node's Tailscale
identity — MagicDNS name (laptop.taila4f562.ts.net) + all tailnet
IPs — alongside the primary node name.
Closes the "how does a laptop join the fleet without hand-editing
SANs" gap: on a machine that's on Tailscale, one command produces
a leaf that peers can dial by MagicDNS from anywhere on the
tailnet.
New CLI:
claw-store fleet-ca-tailscale-sign \
--ca-dir /etc/claw-store/ca \
[--node <name>] # defaults to Tailscale HostName
--out-dir /etc/claw-store/tls
Reads identity by shelling to `tailscale status --json` (already
present on any node that's on the tailnet; no extra dep). If
tailscale isn't running or installed, exits cleanly with a real
error.
New module cluster::tailscale:
* TailscaleSelf { magicdns_name, tailscale_ips, short_hostname }
* read_self() — runs the CLI, returns identity
* parse_status() — pure decoder, unit-tested
* suggested_sans() — MagicDNS + IPs ordered for the CA sign flow
FleetCa additions:
* sign_leaf_to_pem_with_sans(node_name, extra_sans, out_dir) —
Sans-extended variant of sign_leaf_to_pem. Empty entries dropped.
Existing sign_leaf_to_pem now delegates with empty extras (100%
backward compat).
* mint_leaf_with_sans — internal shared helper.
+5 tests: parse full identity, parse missing MagicDNS, error on
no Self record, suggested_sans ordering, suggested_sans skips
missing MagicDNS.
372 tests pass (+5). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
Next Phase 8 slices: (a) tailnet-preferring peer probe with a
config-selectable auth mode, (b) documented "roaming client"
config template.
Wires the Phase 7f ref-tracking primitives to a real Gitea. New
CLI `claw-store cluster-ref-sweep --gitea-url <> [--gitea-token]
[--retention-days N]` queries every distinct repo we've recorded
against, fetches its live branches + tags, computes the stale set
via RefTracking::stale_at, and prints the stale fingerprints
grouped by repo.
Dry-run only in this cut. Deletion is separate — the operator
decides whether to call `forget` per fp, and whether to also
prune the corresponding blob/tag. Blob eviction happens via
cluster-gc as usual (dead refs no longer contribute to any pin).
New module cluster:
* GiteaClient::new(base_url, token) — reqwest with 15s timeout,
rustls-tls (reuses the rustls stack quinn already pulls in).
* live_refs(repo) — fetches /branches + /tags concurrently,
paginated (page 200 hard cap for safety), returns HashSet.
* 404 on either endpoint returns empty set — deleted repos then
flow through stale_at as "all refs dead", the correct default.
Deps:
* reqwest 0.12 with rustls-tls + json, default-features off (no
native-tls / openssl chain).
* clap 4 + "env" feature so --gitea-token can read GITEA_TOKEN.
+2 tests (validate_repo shape, client trims trailing slash).
Full test suite: 367 pass (+2). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
Closes the retention loop between snapshots and pin-aware LRU
eviction. A snapshot is not just a "list of blobs at time T" any
more — it's a *retention pin* on every blob it captures.
Operators can guarantee a build stays on disk for N days by
snapshotting it and pruning the snapshot when the window is up.
Additions:
* SnapshotStore::pinned_blob_ids() → union of blob_ids across all
live snapshots. Cheap: one JSON read per snapshot.
* cmd_cluster_gc extends the tag-pin set with snapshot pins
before handing it to evict_to_size_cap_with_pins. Output line
now reads "pinned blobs: N (M from snapshots)".
* ClusterServices auto-GC ticker does the same on every tick;
log fields include snapshot_pins so ops see the retention set
size at a glance.
+2 tests:
- pinned_blob_ids_unions_all_snapshots (overlap dedupe)
- pinned_blob_ids_empty_when_no_snapshots
355 tests pass (+2). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
A snapshot is a named, immutable point-in-time record of every blob
live in the store. It's NOT a data copy — blobs are content-addressed
and already live under blobs/. A snapshot is a JSON reference set at
<root>/snapshots/<name>.json.
Why:
* Rollback anchor before risky migrations.
* Retention pin: combined with the Phase 4a pin-aware LRU eviction,
operators can guarantee "these blobs stay on disk N days".
* Audit: "which blobs existed at release time?"
New module cluster::snapshot:
* SnapshotStore::create(name, blob_store, created_at)
* SnapshotStore::get(name) / list() / delete(name)
* SnapshotManifest { name, created_at_unix, blob_ids }
* SnapshotSummary for cheap list rendering (no blob-list slurp).
BlobStore gains list_blob_ids() — walks blobs/**/*.manifest.json
and returns the blob id set. Manifests only, no chunk reads.
New CLI commands:
* claw-store cluster-snapshot-create --name <>
* claw-store cluster-snapshot-list
* claw-store cluster-snapshot-show --name <>
* claw-store cluster-snapshot-delete --name <>
Semantics:
* Snapshots are immutable: create with existing name errors, does
not clobber. Delete-then-create if you really want to overwrite.
* delete() removes only the reference file. Never touches blob
data — protects against operators nuking live data by pruning
snapshots.
* list() sorts by created_at_unix ascending — oldest first so
triage picks pruning candidates quickly.
* blob_ids are sorted at write time so the same content on two
nodes yields byte-identical snapshot files.
* Names validated: no /, \\, NUL, control chars; max 512 bytes.
+8 tests covering create+capture, immutability, get-missing,
list-ordering, delete truth-values, delete-doesn't-touch-blobs,
name-validation, and sorted round-trip.
353 tests pass (+8). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
New command: `claw-store cluster-repair --peer <name> --rpc-addr <host:port>
--tls-dir <dir> [--dry-run]`.
Flow:
1. Local scrub identifies bad chunks (missing + corrupt).
2. Deduplicate to unique chunk-hashes (scrub emits per-reference,
fetcher work is per-chunk).
3. Connect to peer over QUIC + mTLS.
4. For each unique chunk: HasChunk probe → GetChunk on hit →
put locally (re-hashed by put_chunk, so a lying peer can't
corrupt us further).
5. Report attempted/repaired/unrecoverable/errors.
--dry-run stops after the dedup step: prints the plan without
touching the peer or disk.
Behavior details:
* Zero bad chunks → clean exit with no peer contact.
* Any unrecoverable or per-chunk error → non-zero exit so cron/CI
notice. Message names counts.
* HasChunk-first means a peer that lacks the chunk is one cheap
round-trip, not a full GetChunk attempt.
Companion piece for the Phase 7b repair library (already merged).
No new tests here — logic is thin glue over `repair_chunks` +
`call_has_chunk`/`call_get_chunk`, all of which have their own
unit + integration coverage. Behavior gets its real workout in
live smoke on tank+architect.
New primitive: BlobStore::scrub_all() → ScrubReport.
Walks every .manifest.json under blobs/, for each referenced chunk
reads the file from disk and recomputes BLAKE3. Verdict per chunk:
* file absent → missing
* hash mismatch → corrupt
* match → ok
Design points:
* Read-only. Never touches disk state. Safe against a live daemon
— worst case a chunk lands mid-scrub and is skipped this pass.
* Per-reference counting: a bad chunk that N manifests depend on
shows up as N corrupt entries so operators see the full blast
radius. But each unique chunk is hashed exactly once via an
in-memory verdict cache.
* Report holds explicit (blob_id, chunk_hash) pairs for every
bad chunk so the fix path (repair in Phase 7b) has enough
info to act.
CLI: `claw-store cluster-scrub [--verbose]`. Non-zero exit when
integrity issues exist so cron / CI notice.
+4 tests:
- scrub_reports_all_ok_when_store_is_healthy
- scrub_detects_corrupt_chunk (owner blob id preserved)
- scrub_detects_missing_chunk (owner blob id preserved)
- scrub_dedups_shared_chunk_hashing_once (shared chunk, 2 owners
reported, single disk read)
341 tests pass (+4). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
Adds the on-disk mechanism for time-scoped pins. No RPC or CLI yet
— a follow-on will expose \`pin --ttl <duration>\`. This PR is
purely library + eviction wiring.
Layout addition: alongside each stamped tag at
\`tags-v2/<hh>/<hash>.svtag\`, an optional sidecar
\`tags-v2/<hh>/<hash>.svtag.exp\` holds an 8-byte LE unix
\`expires_at\`. Absence of the sidecar = never expires (current
behavior).
New TagStore methods:
* set_stamped_expiry(key, expires_at_unix) — writes sidecar;
passing 0 removes it. Idempotent.
* get_stamped_expiry(key) — reads sidecar; None when absent.
* pinned_blob_values_at(now_unix) — same union as
pinned_blob_values, but skips stamped tags whose sidecar shows
expires_at ≤ now. Legacy tags/ entries never expire.
* prune_expired_stamped_at(now_unix) — deletes stamped tags AND
their sidecars where expires_at ≤ now. Returns count.
* pinned_blob_values() — now a shim that calls _at(u64::MAX) for
100% backward compat.
Wired the two existing gc call sites:
* ClusterServices auto-GC ticker prunes-then-collects at
SystemTime::now(). One pass per tick.
* \`claw-store cluster-gc --evict-to-gb N\` CLI same pattern.
Report now includes \"expired pins pruned: N\".
+1 test (expiry_gates_pin_set_and_prune_removes_expired):
covers live/expired/no-ttl mix, sidecar round-trip, prune
removes only expired, expires_at=0 clears sidecar, dropped
tag stops filtering.
286 tests pass (+3 from 283). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
A `claw-cargo pin` used to be silently vulnerable to the size-cap
eviction ticker — the tag existed but the underlying blob could get
LRU'd out, leaving a dangling reference. Now tags act as
retention markers: any blob referenced by any tag (stamped or
legacy) is protected from `evict_to_size_cap`.
* `BlobStore::evict_to_size_cap_with_pins(max_bytes, pinned_set)` —
same LRU-by-mtime pass, but pinned blob IDs skip the eviction
loop. Existing `evict_to_size_cap` is now a thin wrapper with an
empty pin set (100% backward compat).
* `TagStore::pinned_blob_values()` — unions every 32-byte value
referenced by any tag across `tags/` (legacy) and `tags-v2/`
(Phase 3c stamped). Dedupes naturally.
* Auto-GC ticker in `ClusterServices` now collects the pin set on
every eviction pass and passes it in. Log fields include
`pinned_blobs = N` so operators can see the retention set size.
* `claw-store cluster-gc --evict-to-gb N` CLI opens the tag store
the same way, prints `pinned blobs: N` in the report.
+3 tests:
- evict_with_pins_protects_pinned_blobs_from_eviction — 3 blobs
ordered oldest→newest, pin the oldest; without pins LRU would
evict it; with pins the next-oldest goes instead. Guards the
main semantic.
- evict_with_pins_stops_when_pinned_footprint_dominates —
everything pinned + cap = 0 → no-op. Guards the "operator asked
for the impossible" case.
- pinned_blob_values_unions_both_stores — legacy tag with value V1,
stamped tag with value V2, second stamped tag also referencing
V1 → set contains {V1, V2}. Dedupe check.
283 tests pass (baseline +3). Pre-existing macOS failure unchanged.
Orphan-chunk GC alone doesn't stop unbounded growth: as long as
fingerprint→blob refs keep getting PutRef'd, the manifest set keeps
growing and no chunk is ever an orphan.
* `BlobStore::evict_to_size_cap(max_bytes)` — walks manifests oldest
first by mtime, deletes them, refcount-decrements each chunk they
used, unlinks + reclaims size for any chunk whose refcount hits
zero. Shared chunks stay put until the last blob referencing them
is evicted.
* `ManifestSummary` internal type keeps the diff-set bookkeeping
cheap (one HashMap<ChunkHash, u32>, no repeated tree walks).
* `claw-store cluster-gc --evict-to-gb <N>` extends the CLI: still
runs the orphan sweep first, then optionally caps the store.
* Config: `cluster.blob_max_gb: Option<u64>`. The auto-GC ticker
runs eviction after every orphan sweep when this is set. Silent
when the store is already under cap; INFO log when it evicts.
+3 tests:
- evict_to_size_cap_reclaims_oldest_blobs_first: 3 blobs with
distinct mtimes, cap below combined size → oldest evicted,
newer blobs survive
- evict_keeps_shared_chunks_when_still_referenced: guards the
refcount decrement path (content-addressed dedup keeps identical
content as one blob → chunk survives until manifest deleted)
- evict_on_empty_store_is_a_noop: sanity
257 tests pass (baseline +3). Pre-existing macOS failure unchanged.
Both surfaced by the 2026-07-12 pilot as real operator concerns:
## rustc drift warning at build time
Runners silently silo their cache when rustc versions differ across
peers (fingerprint depends on rustc verbose output). The pilot's
first flow burned a full cold+upload before we realized the silo.
- `PeerStatusReply.local_rustc_release` — new field, populated from
the peer's own gossip `RUSTC_RELEASE` key via a new
`ClusterGossip::self_kv(key)` accessor.
- `claw-cargo build`: on cache MISS, calls `PeerStatus`; if the
peer's rustc release ≠ our local `rustc --version`, emits a WARN
with both versions + hint to add `rust-toolchain.toml`.
- Best-effort: absence of either release string is a shrug, not
an error.
## blob GC
Blob store grows unbounded on a runner; disk-full is a real
incident. `gc_orphan_chunks` already existed but wasn't exposed.
- New CLI: `claw-store cluster-gc` — runs `gc_orphan_chunks`,
prints report. Safe to run any time, safe to interrupt.
- New config: `cluster.gc_interval_hours: Option<u64>`. When set to
a positive integer, the daemon spawns a periodic ticker that
invokes GC in-process. Skips the first tick (nothing to reclaim
on boot). Errors are logged and retried next tick.
- Shutdown aborts the ticker cleanly.
254 tests pass (baseline unchanged). Pre-existing macOS failure
untouched.
Bundles the profile→dir bug (PR #20 supersede) with four new fixes
discovered by running clawstor against itself + across the fabric:
* target_subdir_for: `dev`/`test` → `debug/`, `release`/`bench` →
`release/`, custom passes through. Was silently skipping upload.
* rustc release via gossip: daemon probes `rustc --version` at start,
publishes the release string as `clawstor.rustc.release`. PeerView
carries it; `cluster-peer-status` prints it in a new column and
emits a warning line when the fleet has mixed versions. Would have
surfaced the tank/architect 1.96.1 vs 1.95.0 drift instantly.
* prewarm publishes fingerprint→blob ref downstream: `pin` now writes
a companion tag `<name>.fingerprint` holding the fingerprint bytes.
`prewarm` reads the companion, PutTag's it downstream, then
PutRef(fp→blob) so a subsequent fingerprint-based `build` HITS.
Without this, prewarm was almost useless for the runner path
(build always missed even with matching source + rustc).
* streaming byte counters: BlobPutStream + BlobGetStream now record
the transferred bytes via `record_blob_{put,get}_bytes`. Metric
used to stay at 0 no matter how much you moved.
* capture determinism: replaced `tar::Builder::append_dir_all` (uses
`read_dir`'s native order) with `append_dir_sorted` that walks the
tree recursively and sorts by filename bytes at every level. Two
byte-identical trees now produce byte-identical tars regardless of
filesystem ordering.
+3 tests:
- target_subdir_matches_cargo_layout (from #20)
- fingerprint_companion_tag_uses_dotted_suffix
- capture_is_order_independent_of_filesystem_readdir (guard against
the exact bug we saw in the field)
252 tests pass (+1 from Phase 5h's 251). Pre-existing macOS `du -sb`
failure unchanged.
Supersedes #20 (also included here). Ready for re-deploy to
tank + architect for the retest run.
Ties Phase 1a-1d together into a live subsystem the daemon actually
runs. When `[cluster]` is present in config, `claw-store daemon` now:
1. Starts chitchat gossip via ClusterServices::start.
2. Publishes hot.max_gb immediately and re-measures the hot dir
every 30s, updating `clawstor.hot.used`.
3. If `[cluster.tls]` is also configured, loads NodeIdentity from
PEM files and binds a QUIC RPC server that accepts + dispatches
incoming Ping / PeerStatus requests via RpcRouter.
New modules:
- cluster/rpc.rs (510 lines):
- Method enum (Ping=0x01, PeerStatus=0x02)
- ErrorCode enum (EmptyRequest / UnknownMethod / HandlerFailure)
- PeerStatusReply { local_name, local_zone, peers: Vec<PeerView> }
- RpcRouter — dispatch state (holds Arc<ClusterGossip> + local
name/zone)
- rpc_call / call_ping / call_peer_status — client helpers
- serve_connection — server accept-bidi loop
- Wire format: `method:u8 || payload:bytes` → `reply:bytes` or
single-byte ErrorCode
- cluster/services.rs (418 lines):
- ClusterServices { gossip, accept_task, metric_task }
- start(cluster_cfg, name, hot_dir, hot_max_bytes) → bootstraps
everything above
- shutdown() aborts background tasks cleanly
- Recursive dir-walker (spawn_blocking) for hot-tier metric
PeerView now derives Serialize/Deserialize so it round-trips through
JSON over the RPC.
daemon.rs integration (~35 lines added):
- Bootstraps ClusterServices before entering the select loop
- Held for daemon lifetime
- Shutdown on SIGTERM
- Gossip-less config still runs standalone (backwards compat)
CLI: `cluster-peer-status --peer <name> --rpc-addr <addr>
--tls-dir <dir>`
Loads a persistent identity, dials the peer, calls PeerStatus,
prints the peer's local view as a table.
Tests (15 new, all real — no mocks, real UDP + TLS + JSON round trip):
RPC (9):
- method_round_trips_byte_encoding
- dispatch_returns_pong_for_ping
- dispatch_returns_json_for_peer_status
- dispatch_returns_empty_request_error
- dispatch_returns_unknown_method_error
- rpc_call_rejects_oversize_payload (with real quinn connection)
- end_to_end_ping_and_peer_status_over_real_quic — full 2-node quinn
with mTLS + both RPCs
- peer_status_reflects_peer_gossip_state — 2 gossip instances converge,
RPC caller from a third identity sees the converged view
- error_code_describe_covers_all_variants
Services (6):
- dir_bytes_sync_returns_zero_for_missing_path
- dir_bytes_sync_sums_recursive_file_sizes (3-level nesting)
- services_start_without_tls_leaves_rpc_disabled
- services_start_with_tls_serves_rpc_end_to_end — full stack: cut CA on
disk, sign leaves, start ClusterServices for A, dial from B via
persisted mTLS, run ping + PeerStatus over the wire
- services_publish_hot_used_metric_periodically
- services_gossip_sees_peer_after_convergence
95 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/rpc.rs: 510
- cluster/services.rs: 418
- cluster/gossip.rs: 579
- cluster/transport.rs: 916
- daemon.rs: 262
- main.rs: 749
Phase 1 done end-to-end. `claw-store daemon` now boots a real distributed
cluster stack when config asks for one; peers can call each other's
PeerStatus RPC and see live membership. Next: Phase 2 (content-addressed
blob store) can hook new RPC methods into the same RpcRouter with a
one-line dispatch arm.
Closes out Phase 1. A production operator can now cut a fleet CA,
sign per-node leaves, drop the resulting PEMs at
/etc/claw-store/tls/, point [cluster.tls] at them, and the daemon
loads real mTLS material on startup — no more ephemeral in-process
CA hack.
New public API in cluster::transport:
- FleetCa::generate(cn) — new self-signed root CA
- FleetCa::save(dir) / FleetCa::load(dir) — round-trip PEM
- FleetCa::sign_leaf(name) — mint an in-memory NodeIdentity
- FleetCa::sign_leaf_to_pem(name, out_dir) — write ca.crt + node.crt +
node.key (node.key at 0o600 on Unix)
- NodeIdentity::from_pem_files(ca, cert, key) — production load path
- NodeIdentity::from_pem_dir(dir) — canonical filename layout
- NodeIdentity::from_cluster_config(cfg) — pick up [cluster.tls] paths
Manual Debug for FleetCa redacts the private key.
Config extension:
- [cluster.tls] ca_cert / node_cert / node_key (all PathBuf).
- Optional at the top level; callers that need mTLS surface a clear
error when it's absent.
Deps:
- rcgen features += "x509-parser" (for FleetCa::load's from_ca_cert_pem).
- rustls-pemfile 2 (parse PEM back into DER for rustls).
CLI (new commands; short-circuit config load so they run on fresh
boxes without /etc/claw-store/config.toml):
- fleet-ca-init --dir <dir> [--cn <name>]
generates ca.crt + ca.key (both 0o600).
- fleet-ca-sign --ca-dir <dir> --node <name> --out-dir <dir>
writes ca.crt + node.crt + node.key (node.key at 0o600).
- cluster-ping now accepts --tls-dir <dir> to load persistent
NodeIdentity from disk (produced by fleet-ca-sign).
Tests (8 new, all real — no mocks, real filesystem, real TLS handshake):
- fleet_ca_rejects_empty_common_name
- fleet_ca_save_and_load_round_trip_preserves_signing (asserts 0o600
on ca.key)
- fleet_ca_load_errors_when_files_missing
- sign_leaf_to_pem_writes_all_three_files_with_correct_permissions
(asserts 0o600 on node.key)
- persistent_identity_round_trips_through_disk_and_pings — end-to-end:
cut CA on disk, reload it, sign two leaves via sign_leaf_to_pem,
reload them via from_pem_dir, run real QUIC ping/pong. This is the
operator flow.
- node_identity_from_cluster_config_errors_without_tls_section
- node_identity_from_cluster_config_loads_pem_paths
- from_pem_files_errors_on_missing_ca_file
80 tests pass. Pre-existing macOS-only hot test unchanged.
Also verified live CLI smoke test:
fleet-ca-init → ca.crt + ca.key at 0o600
fleet-ca-sign → ca.crt + node.crt + node.key at 0o600
Files parse as valid X.509.
File sizes (all under 1300-line ceiling):
- cluster/transport.rs: 916
- cluster/gossip.rs: 576
- cluster.rs: 275
- config.rs: 503
- main.rs: 662
Phase 1 complete. Next up:
- Phase 1e (daemon integration): gossip + QUIC RPC server wired into
claw-store daemon; hot-tier metrics periodically pushed; a real
PeerStatus RPC alongside ping.
- Phase 2: content-addressed blob store (BLAKE3 chunking, put/get).
Wraps quinn 0.11 in cluster/transport.rs with rustls 0.23 + rcgen 0.13
for identity management. Every peer connection is mTLS: both sides must
present certs signed by the shared fleet CA, and rustls verifies the
peer's cert SAN matches the requested server name.
Public API on `cluster::transport`:
- `NodeIdentity` (cert chain + private key + trusted CA)
- `NodeIdentity::generate_test_pair(a, b)` — ephemeral CA + two signed
leaves; shape matches the production PEM-file loader (Phase 1d)
- `QuicServer::bind(addr, identity)` — bind mTLS-enforced listener
- `QuicServer::accept()` — accept one connection (Option<Result<_>>)
- `QuicClient::new(local_addr, identity)` — build client endpoint
- `QuicClient::connect(peer_addr, expected_name)` — outbound with SAN check
- `ping(&conn, payload)` — bidi stream RPC; server echoes as `pong:<payload>`
- `ping_handler_loop(conn)` — server-side accept/echo forever
- `CLAWSTOR_RPC_ALPN` constant, single ALPN "clawstor-rpc/1"
Config extension:
- `bind_rpc_lan`, `bind_rpc_tailscale` on `ClusterConfig` (both optional).
- Defaults: gossip port + 1 (so gossip UDP and QUIC UDP don't collide).
- Helper: `ClusterConfig::rpc_lan()`, `rpc_tailscale()`,
`PeerEntry::rpc_lan()`, `rpc_tailscale()`.
Gossip.rs now advertises the RPC address, not the gossip address, on
the well-known `clawstor.rpc.lan` / `clawstor.rpc.tailscale` keys.
CLI:
- `cluster-ping --name <me> --peer <you> --rpc-addr <addr> [--payload X]`
Runs a full mTLS handshake and single ping. For dev/loopback use
today (both sides need to share a CA); persistent-identity ping
lands in Phase 1d.
Tests (6 new, real UDP + TLS handshake, no mocks):
- generate_test_pair produces two distinct leaves that share a CA
- ping_pong_between_two_mtls_peers: real 2-node QUIC round trip with
full mTLS chain verification, `open_bi()`/`accept_bi()`, byte-exact
response check
- client_rejects_peer_with_wrong_ca: TLS chain verification failure
when the server presents a cert signed by a different CA
- client_rejects_wrong_server_name: SAN mismatch is enforced
- server_binds_wildcard_and_reports_concrete_local_addr: port 0 → real
- ping_rejects_oversize_payload: MAX_MESSAGE_BYTES cap enforced client-side
Fixed-port allocator range 42000+ so transport tests don't conflict
with gossip tests (41000+).
72 tests pass. Pre-existing `hot::test_project_target_size_bytes`
macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/transport.rs: 485
- cluster/gossip.rs: 570
- cluster.rs: 275
- config.rs: 480
- main.rs: 564
Follow-on Phase 1 cut (1d):
- Load NodeIdentity from persistent PEM files at /etc/claw-store/tls/
- Fleet CA bootstrap ceremony (rcgen → write CA cert; per-node leaf CSR)
- Daemon-level RPC server that runs alongside gossip + serves real
operations (blob get/put, metadata sync)
- cluster-status reads from live daemon via API instead of standalone
Wraps chitchat 0.11 in cluster/gossip.rs. Each node publishes its
zone, RPC endpoints, hot-tier usage, warm-tier project list, and
uptime under well-known kv keys; peers propagate this via gossip and
are classified alive/dead by phi-accrual failure detection.
Bootstraps from static [[cluster.peers]] seed nodes in the config.
Membership extends dynamically as nodes join or leave.
Public API on `ClusterGossip`:
- bootstrap(cluster_cfg, local_name) — start UDP gossip service
- set(key, value), set_hot_used, set_hot_max, set_warm_projects
- peers() — all known peers with typed PeerView + liveness
- peer(name) — one peer by advertised name
- peers_in_zone(zone) — live peers filtered by zone
- shutdown() — abort gossip task
CLI:
- `claw-store cluster-status --name <me> [--wait-secs N]`
bootstraps gossip, waits for convergence, prints a peer table
with zone, alive/dead, RPC LAN/Tailscale addresses, hot usage.
Tests (7 new):
- bootstrap_publishes_our_own_state — self-ID + advertised address
- bootstrap_fails_when_local_name_empty
- bootstrap_fails_when_no_bind_address — cluster.validate error chain
- two_node_cluster_converges_and_shares_state — REAL two-node
in-process UDP gossip. Publish state on A, wait for phi-accrual
liveness on B, verify all fields propagated (zone, rpc_lan, hot
bytes, warm_projects list, fill ratio, zone filtering).
- peers_excludes_self — solo cluster reports zero peers
- peer_view_hot_fill_ratio_handles_missing_or_zero
No mocks. Real UDP transport, real chitchat runtime, real gossip
protocol. Ports allocated from a fixed range (41001+) via atomic
counter — deterministic and race-free within the test binary.
66 tests pass. Pre-existing `hot::test_project_target_size_bytes`
macOS-only failure unchanged.
File sizes (under 1300-line ceiling):
- cluster/gossip.rs: 549
- cluster.rs: 274
- main.rs: 506
Follow-on Phase 1 cuts:
- 1c: `quinn` QUIC transport + fleet-CA mTLS for RPC (separate UDP
port so gossip and QUIC don't collide)
- 1d: daemon wire-up — gossip runs in background, hot-tier usage
pushed periodically, `cluster-status` reads from live daemon
First cut of the v2 distributed FS. Captures the architecture design
in ARCHITECTURE-v2.md and lands the smallest useful new capability:
probing cluster peers with a LAN-first policy so subsequent transport
+ gossip layers (Phase 1b, 1c) can build on a real routing decision.
New:
- ARCHITECTURE-v2.md: zones (fabric-10g/lan-1g/roaming), tier
lifecycle (hot/warm/cold), fingerprint-keyed build cache design,
smart-clean policy, phase plan, explicit non-goals.
- claw-store/src/cluster.rs: RouteKind, RouteWinner, LanFirstProbe.
LAN 200ms timeout, Tailscale 500ms fallback. 8 tests use real
TCP listeners on 127.0.0.1 (no mocks); cover happy path,
fall-through, both-fail, single-address, and elapsed reporting.
- claw-store/src/config.rs: ClusterConfig + PeerEntry with
validation (bind-address presence, no duplicate peer names,
per-peer reachable address required). Optional at top level so
pre-v2 configs still load unchanged. 6 new tests.
- claw-store/src/main.rs: `claw-store cluster-probe <peer>` CLI
subcommand that reads config, resolves the peer, probes, prints
the winning route + elapsed time.
All 16 new tests pass. Existing 45 pass. Sole failure
(hot::tests::test_project_target_size_bytes) is a pre-existing
macOS-only issue with `du -sb`; Linux CI unaffected.
Follow-on Phase 1 cuts (subsequent sessions):
- 1b: chitchat SWIM gossip for live membership state
- 1c: quinn QUIC transport with fleet-CA mTLS
- 1d: `claw-store cluster status` — live membership view
Every file well under the 1300-line ceiling
(cluster.rs 268, config.rs 428, main.rs 450).
Answers a real operational question — 'why does /hot/targets stay near
full even when I haven't touched most of these projects for weeks?'
Previously the stale sweep was gated behind 'hot tier > 90% full', so
an idle project held NVMe until the operator manually deactivated it
or space ran out and LRU came for it.
The change:
* new claw-store/src/actions.rs — shared deactivate_project(cfg,
manifest_path, project) with the FULL flow: sync-to-peer +
hot rm + .cargo/config.toml removal + manifest update, plus a
DeactivateOutcome { synced, sync_error, freed_bytes } return.
Extracted from main.rs::cmd_deactivate so both CLI and daemon
run the same code — no drift between manual and auto semantics.
* daemon.rs poll_tick now runs the stale sweep on EVERY tick,
independent of space pressure. Each project idle > stale_hours
(48 by config) goes through the full deactivate, logs
'stale-gc: X freed N MB (synced=Y)'.
* gc_by_space (LRU) still gates on >90% full — it's the emergency
'even after the stale sweep we're still tight' path.
* main.rs cmd_deactivate is now a thin CLI wrapper that adds
println! feedback + reports freed MB in the terminal output.
Space-pressure LRU keeps its original .cargo/config.toml-preserving
semantics (rm hot only, not the shim) so a project marked 'active'
by manual activation but hard-evicted for space can still be
re-activated cheaply. Stale sweep is the aggressive one because the
project genuinely hasn't been touched.
Tests: two new unit tests in actions.rs cover the full flow +
idempotency; the freed_bytes assertion is Linux-only-safe (BSD du
returns 0 for -sb, same limitation as the existing hot test).
Removes the biggest workflow footgun in the architect↔tank flow: the
operator no longer has to remember to run 'claw-store sync <project>'.
An ordinary 'git commit' anywhere under /slab/projects/*/* is now
picked up automatically on the next 5-min poll tick and the peer is
notified — same code path deactivate/sync used to trigger by hand.
New module claw-store/src/head_watch.rs owns the state:
* HeadCache — {project → last-seen HEAD sha}, persisted at
/var/lib/claw-store/head-cache.toml (same directory as the
existing sync queue and manifest).
* scan_and_enqueue walks warm_root/<org>/<repo>, calls
'git rev-parse HEAD' on each, compares to cache, enqueues on
change. Non-git dirs, symlinks, and unreadable entries are
skipped (matches the project-list walker's behavior).
First-scan policy: if the cache is empty on startup we populate it
silently instead of enqueueing every project — otherwise a fresh
install with 55 warm projects would flood the peer with 55 syncs on
the first tick. Existing state gets baselined; only movement from
that point on triggers work.
daemon.rs poll_tick was one 'retry the queue' block; it's now
'scan for HEAD changes → save cache → drain queue' in that order,
so a commit from between ticks gets enqueued AND drained the
same tick.
Four unit tests cover the invariants: silent first scan, enqueue
on real HEAD move (only for the moved repo), cache roundtrip,
non-git directory skip.
The two biggest pain points coming out of the architecture review:
1. The manifest at /var/lib/claw-store/projects.toml was the only
piece of writable state but had no locking, no atomic writes,
and three concurrent writers (daemon poll tick, every CLI verb,
and the dashboard shelling out via /api/activate). Two writers
interleaving silently dropped one of them; a crash mid-write
left a corrupt half-written TOML that the next reader parsed
as an empty manifest.
2. Reboot survival: the dashboard had no systemd unit and was a
stray hand-launched process. Architect lost its dashboard on
todays reboot.
This commit lands:
- Manifest::update(path, FnOnce(&mut Manifest)) — locked-atomic
load-mutate-save in one transaction. Uses libc::flock(LOCK_EX) on
a sidecar .lock file (so the data file can be replaced by rename
without invalidating the lock) and tempfile + persist for the
rename. Concurrent writers serialise; readers see the previous
state or the new state, never a torn write. Manifest::load uses
LOCK_SH so it never races a mid-rename.
- Project.pinned: bool with #[serde(default)] so legacy manifests
parse cleanly. hot::gc_stale_targets and hot::gc_by_space both
skip pinned projects, with a WARN log when every remaining
project is pinned but were still over budget — operator intent
beats space pressure.
- claw-store pin <project> / unpin <project> CLI verbs.
status command surfaces pin marker (📌).
activate preserves an existing rows pinned flag so re-activating
doesnt silently unpin.
- claw-store-serve.service systemd unit. Type=simple, Restart=
on-failure, RestartSec=15, ProtectSystem=strict + ReadWritePaths
=/var/lib/claw-store, ProtectHome, NoNewPrivileges, PrivateTmp.
- daemon poll tick reloads the manifest from disk at the start of
each cycle (so CLI activations between ticks are visible) and
routes its GC write through Manifest::update (so it cant race a
concurrent CLI pin).
- libc + tempfile move from dev-deps into runtime deps.
- empty-manifest fallthrough on load (treat "" as default) so a
half-written tempfile crashed pre-rename doesnt hard-fail the
daemon next boot.
- 25 tests passing incl. new ones: legacy-toml-parses, update-
serializes-two-sequential-writers, pinned-survives-stale-gc,
pinned-survives-space-gc-even-when-lru.
Version bumped 0.1.0 → 0.2.0.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Adds the serve subcommand: an axum 0.7 + tower-http server on
:7700 backing the React dashboard. Routes:
GET /api/status /api/projects /api/snapshots
/api/sync-queue /api/hot /api/events (SSE 5s tick)
POST /api/activate /api/deactivate /api/sync
/api/gc /api/snapshot
Mutating endpoints shell out to /usr/local/bin/claw-store so
all CLI logic stays single-sourced. claw-store/static/ holds
the prebuilt dashboard Vite bundle for --static-dir.
cargo_init.rs gets a minor wiring tweak so the API can read
back the managed cargo config marker.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Skip non-git directories inside org folders so flat layout dirs like
claw-store/target or claw-store/.cargo don't appear as fake repos.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- activate <org/repo>: wire hot tier, write .cargo/config.toml, register
- deactivate <org/repo>: auto-sync to peer, evict hot tier, unregister
- list: show all warm repos with activation status and last-active time
- sync <org/repo>: git push origin then SSH peer to pull
- pull <org/repo>: git pull from origin, stamp last_sync in manifest
- SyncQueue: file-based retry queue (/var/lib/claw-store/sync-queue.toml)
- daemon: drains sync queue on every 5-min poll tick
- config: [peer] host/user for cross-node notification
- manifest: Project.name is now org/repo, adds last_sync field
- hot tier paths are org/repo-namespaced to avoid collisions
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
osobh
Omar Sobh
