9debd84e957dc5210c2c3b80d355eb5baf7e6603
21
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 osobhandClaude Sonnet 5
|
4ea1cbed2e |
Add shutdown-prep button to dashboard-v2 NodeDetail
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Failing after 3s
Wires safe-shutdown-prep.sh into the dashboard so an operator can prep a node for hardware maintenance from a browser instead of SSH. New RPC methods (0x20/0x21): - ShutdownPrepCheck runs `--dry-run` to completion and returns the full report. Never stops anything, safe to call repeatedly. - ShutdownPrepExecute starts the real run detached (`systemd-run --user --scope --collect`), placing it in a cgroup outside claw-store.service's own -- the script's own step 6 stops that service, i.e. the process that would otherwise be running it, so it has to survive its own parent dying. Returns immediately with a "started" message; full output lands in /var/lib/claw-store/shutdown-prep.log for whoever's at the machine once it's gone dark, since there's no way to stream a live result past the point the daemon stops itself. - Execute double-checks confirm_node_name against the peer's own configured name server-side, on top of the aggregator's own path match -- defense in depth for a highly consequential action. Aggregator endpoints (admin-token gated, AuthedCaller::require_admin): POST /api/v2/node/:name/shutdown-prep/check POST /api/v2/node/:name/shutdown-prep/execute Frontend: ShutdownPrepPanel on NodeDetail. Check button always enabled; the real "stop services" button only unlocks after a ready check, and additionally requires typing the exact node name to confirm before it's clickable. Also fixes a script bug found while testing this against the live daemon process (not caught in manual interactive-shell testing): the zpool-detection line parsed raw `mount` output positionally, which returned the wrong field under the daemon's process context for reasons that didn't reproduce interactively. Switched to `df --output=source`, which is stable across both. Verified end-to-end against tank, architect, and morpheus, including cross-node targeting (tank's dashboard successfully triggered a check on morpheus over the fleet RPC layer). Co-Authored-By: Claude Sonnet 5 <[email protected]> |
||
 osobh
|
c3f6b500fc |
Phase 9 R1: RepoEnsure — peer RPC + aggregator fan-out (#106)
Build with clawstor cache / Cargo build (clawstor-cached) (push) Failing after 25s
|
||
|
Omar Sobh
|
f31c94607a |
FleetHealth PR 3: 'Projects' panel — which repos live where
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 34s
Backend:
* New DashboardProject { repo, cache_bytes, fingerprint_count,
refs, first_seen_unix, last_seen_unix, tier }.
* Handler build_projects() joins ref-tracking with the ref-store
and blob-store: for each recorded fp, resolve fp → blob-id via
RefStore::{get_stamped, get} then sum blob sizes per repo.
* Tier is a wall-clock function of last_seen_unix:
active < 24h, recent < 7d, else idle.
* DashboardStorageReply gains `projects: Vec<DashboardProject>`
(serde-default so older clients still parse).
Aggregator:
* New /api/v2/projects endpoint. Fans out DashboardStorage to
every peer, tags each project row with its originating node,
returns hottest-first.
Frontend:
* New ProjectsPanel component appended to the FleetHealth
landing. Groups by repo, one row per project with tier badge
(active/recent/idle), per-node pill badges, cache-size sum,
ref list, last-activity age.
* Empty state explains how to populate: claw-cargo build with
--repo + --git-ref (or CLAWSTOR_REPO/CLAWSTOR_GIT_REF env
vars in CI).
Data populates automatically as each cache-put runs. Existing
demo entry on tank (clawverse/clawstor · main · c384a4...) will
surface after redeploy.
|
||
|
Omar Sobh
|
9d6badf2aa |
FleetHealth PR 1 (backend): human-oriented DashboardStatus fields
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 17s
Extends DashboardStatus with the primitives the FleetHealth
landing needs. All new fields serde-default to None/[]/false so
old clients (aggregator running an older build) still parse the
reply.
Added fields:
* filesystem — statvfs on blob_store_root: total / used / avail.
* hot — {used_bytes, max_bytes, pinned_bytes?}, read from
gossip (already published every 10s).
* mount — probes /proc/mounts for ~/clawstor-mount by
convention. { path, active }.
* cache — router.metrics().snapshot() summarised as
{hits, misses, bytes_served, bytes_ingested,
hit_rate}. hits = get_ref+get_tag+get_chunk hits,
misses similarly — the counters that matter for
claw-cargo build outcomes.
* timers — well-known set (scrub / gc / ref-sweep /
snapshot-rotate) queried via `systemctl --user
show`. next_fire_unix + last_result per timer.
Aggregator NodeStatusV2 mirrors the same fields verbatim so the
new frontend can consume them without further backend hops.
Follow-on PRs:
* PR 2 — new FleetHealth landing page + node-detail rework
* PR 3 — polish (pinned_bytes derivation, activity timeline,
per-repo grouping)
|
||
|
Omar Sobh
|
1211f0d891 |
dashboard-v2: DashboardStorage RPC + aggregated /storage/* endpoints
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 18s
Storage tabs 404'd because the aggregator only had /fleet + /node/:name
after the pivot. Adds:
* New RPC method DashboardStorage = 0x1d — one round trip returns
tags (full), snapshots (full), ref-tracking (full), blobs
(first 200 by id), refs (first 200 by fp) for the responding
daemon.
* Client wrapper call_dashboard_storage.
* Aggregator fans out to every peer, tags each row with the
originating node, sorts + returns:
GET /api/v2/storage/blobs
GET /api/v2/storage/tags
GET /api/v2/storage/refs
GET /api/v2/storage/snapshots
GET /api/v2/storage/ref-tracking
QuicClient promoted to Arc<QuicClient> inside V2State so the
per-peer JoinSet can hand it to spawned tasks without recreating
the endpoint.
|
||
|
Omar Sobh
|
a2114b918d |
dashboard-v2 PR 3: fleet aggregator via DashboardStatus RPC
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 15s
Pivot from per-node dashboard to single-pane-of-glass. The
aggregator (typically the operator's laptop) holds a fleet-CA
leaf cert + the peer list; each dashboard request fans out to
every peer over the existing QUIC/mTLS cluster port and issues
the new DashboardStatus RPC. Peers don't need to run any HTTP
server of their own.
Backend:
* New RPC method DashboardStatus = 0x1c
* Server handler reads BlobStore / TagStore / RefStore /
SnapshotStore / RefTracking counts + on-disk bytes + rustc
release. Cheap: 5 filesystem walks per request.
* Client wrapper call_dashboard_status
* serve_v2 rewritten as aggregator: V2State holds a QuicClient +
peer list from `[[cluster.peers]]`. Endpoints:
GET /api/v2/fleet fan-out to every peer, parallel
GET /api/v2/node/:name/status one peer, on-demand
Failed peers surface as { online: false, error: "..." } cards
instead of dropping.
serve.rs graceful degrade: v2 aggregator routes only mount when
[cluster.tls] is set. Static SPA still serves at /v2/* even
without an aggregator config so operators see the SPA's built-in
"config missing" error.
Deployment model (this session):
* Aggregator runs on quantum (Mac) with a signed leaf.
* Fleet daemons run cluster-only — no HTTP dashboard anywhere
on tank/architect/morpheus. The clawstor-dashboard.service
systemd units on the fleet are being retired.
Frontend rework to consume /api/v2/fleet ships in the next PR.
|
||
|
Omar SobhandClaude Opus 4.7
|
4630925040 |
Phase 4b follow-on: pin --ttl RPC + CLI
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 10s
Closes Phase 4b by exposing the TTL sidecar written by Phase 4b primitives on the wire and via `claw-cargo pin`. Wire additions: * Method::SetTagExpiry (0x1a) — payload `key_len:u16 || key || expires_at:u64 (LE)`. Reply single-byte OK. `expires_at == 0` clears the sidecar. * Method::GetTagExpiry (0x1b) — payload raw key bytes. Reply 8 bytes (u64 LE) on hit; NotFound when no sidecar is present. Both accept writes even when the stamped tag itself is absent, matching `TagStore::set_stamped_expiry` semantics — the sidecar takes effect the moment the tag lands. CLI: * `claw-cargo pin --ttl <duration>` — humantime-style duration (`30d`, `1h30m`, `2w`, ...). Applied to both the primary tag and its `.fingerprint` companion so eviction treats them as one lifetime. `--ttl 0` / `clear` / `none` clears an existing sidecar without touching the value. Tests: encode/decode roundtrip + malformed-input rejection for `encode_expiry_record`, method-byte stability, NotConfigured without a tag store, end-to-end set/get/overwrite/clear over QUIC, and a real-pin flow that publishes a stamped tag then attaches TTL. Duration parser is unit-tested for single/compound forms, case-insensitive units, bad input, and clock alignment. No new deps — the humantime-style parser is 60 lines in-tree. Co-Authored-By: Claude Opus 4.7 <[email protected]> |
||
|
Omar Sobh
|
2c3cd2ab38 |
Phase 3c + 3e: stamped tags + namespaced ref keys — closes Phase 3
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 20s
Ships the last two pieces from the arch doc's Phase 3 scope for the
cargo-cache use case:
## 3c: Stamped tags (CRDT-merge on PutTag)
Mirror of Phase 3a/3b for TagStore. Two concurrent `claw-cargo pin`
calls on the same tag now race deterministically instead of silently
clobbering.
* `StampedTagValue` — same 48-byte (value, clock, node) tuple as
StampedRef.
* `TagStore::put_stamped(key, StampedTagValue) -> TagPutOutcome`
and `TagStore::get_stamped(key)` — data lives under `tags-v2/`
(separate from `tags/` for cutover safety).
* New wire methods `PutTagVersioned = 0x18` +
`GetTagVersioned = 0x19`.
* `call_put_tag_versioned` / `call_get_tag_versioned` client
helpers.
* `claw-cargo pin` now writes stamped tags. Concurrent pin gets
AlreadyExists and moves on (blob content is content-addressed so
both winners agree on the payload).
## 3e: Namespaced ref keys
Opt-in `--namespace <slug>` on peer-facing subcommands. When set,
the ref key becomes `blake3("clawstor.ns.v1" || namespace || fp)`
so two runners on different namespaces (`clawverse/main` vs
`clawverse/pr-42`) don't collide on the same fingerprint. Empty
namespace = pre-3e behavior, so this is 100% backward compat.
* `refs::namespaced_ref_key(namespace, fingerprint) -> RefKey`
primitive.
* `PeerArgs::namespace: Option<String>` CLI flag flows through to
`cmd_status`, `cmd_prefetch`, `cmd_build`.
* `peer_lookup` now takes a `RefKey` directly (was `&Fingerprint`)
so the namespace resolution stays in the caller — the daemon
never sees "namespace" as a concept.
## 3d: Deferred
Full vector clocks per namespace are noted in the arch doc as a
Phase-3 goal; scalar wall-clock (clock + node stamp) is sufficient
for the cargo-cache use case (single-key LWW merge). NTP-synced
runners see monotonic ordering; skewed runners lose an ordering
but the CRDT semantics still guarantee no data corruption. Full VC
is deferred to a future phase.
+9 tests, 280 total (baseline +8: 7 unit + 1 e2e over real QUIC).
|
||
|
Omar Sobh
|
ac51e3e9b5 |
Phase 3b: thread stamped refs through claw-cargo + forwarding
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 25s
Runner + prewarm use PutRefVersioned/GetRefVersioned; daemon GetRefVersioned forwards on miss + pulls blob transparently. New GetRefVersionedLocal (0x17) prevents recursion. Backward-compat: existing GetRef/PutRef path unchanged; two on-disk namespaces coexist (refs/ and refs-v2/). +1 test, 272 total (unchanged from 3a because we reused existing scaffolding). |
||
|
Omar Sobh
|
af5350ac17 |
Phase 3a: Lamport-stamped refs with CRDT-merge on PutRef
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 16s
Concurrent PutRef safety via (clock, node) total order. New wire methods PutRefVersioned (0x15) + GetRefVersioned (0x16). Existing PutRef/GetRef unchanged for backward compat. Data in refs-v2/ namespace so the two coexist during cutover. +8 tests, 272 total (baseline +8). |
||
|
Omar Sobh
|
58c5bc341b |
GetRef: transparent ref-forwarding on local miss
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 10s
Cross-runner cache silos (tank + architect measured on 2026-07-13): same fingerprint, same rustc, but each runner's daemon only knows about the refs its own runner uploaded. Every runner that lands on a peer that isn't tank re-uploads a duplicate blob. Fix: on `GetRef` miss the daemon fans out to alive gossip peers via a strict-local `GetRefLocal` variant, and the FIRST peer that has the ref triggers a transparent pull — chunks + manifest into the local blob store, then `PutRef` locally — before returning the value to the caller. Subsequent lookups are pure-local hits. * `Method::GetRefLocal = 0x14` — new wire method, identical shape to GetRef but the peer MUST NOT recurse. Loop prevention: our forwarding only calls `GetRefLocal` on peers, so chain depth is always 1. * `RpcRouter::with_outbound_client(Arc<QuicClient>)` — dependency injection point for the forwarding dial path. `None` disables forwarding entirely (GetRef becomes GetRefLocal-equivalent). * `RpcRouter::forward_get_ref(key)` — concurrent peer probes via `JoinSet`, 3s timeout per dial, first successful pull wins, remaining tasks aborted. * `pull_blob_locally` — walks manifest, fetches only chunks the local store lacks (`has_chunk`), commits via `put_manifest_verified`. Bounded memory: one 4 MiB chunk at a time. * `ClusterServices::start` loads NodeIdentity twice — server takes ownership; outbound client gets its own copy for TLS presentation on peer dials. Wires the outbound client into the router when TLS material is available. * `call_get_ref_local(conn, key)` client helper (used by daemon forwarding + available to any RPC consumer that wants the no-recursion semantics). +3 tests in `rpc/tests_forwarding.rs`: - Local hit works without forwarding; local miss with no peers returns None. Guards the base cases. - GetRefLocal never forwards even when outbound is configured (no peers reachable → miss returns None immediately, no attempted fan-out). - Method byte 0x14 encoding is stable across releases. Full end-to-end forwarding is exercised in the pilot deploy: two daemons on the fleet-CA, tank populates a ref, architect's runner GetRef → tank forwards → architect pulls → HIT locally next time. 264 tests pass (baseline +3). Pre-existing macOS failure unchanged. |
||
|
Omar Sobh
|
84aa758fd4 |
Two pilot follow-ons: rustc drift warning + blob GC
Both surfaced by the 2026-07-12 pilot as real operator concerns: ## rustc drift warning at build time Runners silently silo their cache when rustc versions differ across peers (fingerprint depends on rustc verbose output). The pilot's first flow burned a full cold+upload before we realized the silo. - `PeerStatusReply.local_rustc_release` — new field, populated from the peer's own gossip `RUSTC_RELEASE` key via a new `ClusterGossip::self_kv(key)` accessor. - `claw-cargo build`: on cache MISS, calls `PeerStatus`; if the peer's rustc release ≠ our local `rustc --version`, emits a WARN with both versions + hint to add `rust-toolchain.toml`. - Best-effort: absence of either release string is a shrug, not an error. ## blob GC Blob store grows unbounded on a runner; disk-full is a real incident. `gc_orphan_chunks` already existed but wasn't exposed. - New CLI: `claw-store cluster-gc` — runs `gc_orphan_chunks`, prints report. Safe to run any time, safe to interrupt. - New config: `cluster.gc_interval_hours: Option<u64>`. When set to a positive integer, the daemon spawns a periodic ticker that invokes GC in-process. Skips the first tick (nothing to reclaim on boot). Errors are logged and retried next tick. - Shutdown aborts the ticker cleanly. 254 tests pass (baseline unchanged). Pre-existing macOS failure untouched. |
||
|
Omar Sobh
|
e70f5d74e0 |
Pilot findings: 5 real-world fixes from the 2026-07-12 deploy
Bundles the profile→dir bug (PR #20 supersede) with four new fixes discovered by running clawstor against itself + across the fabric: * target_subdir_for: `dev`/`test` → `debug/`, `release`/`bench` → `release/`, custom passes through. Was silently skipping upload. * rustc release via gossip: daemon probes `rustc --version` at start, publishes the release string as `clawstor.rustc.release`. PeerView carries it; `cluster-peer-status` prints it in a new column and emits a warning line when the fleet has mixed versions. Would have surfaced the tank/architect 1.96.1 vs 1.95.0 drift instantly. * prewarm publishes fingerprint→blob ref downstream: `pin` now writes a companion tag `<name>.fingerprint` holding the fingerprint bytes. `prewarm` reads the companion, PutTag's it downstream, then PutRef(fp→blob) so a subsequent fingerprint-based `build` HITS. Without this, prewarm was almost useless for the runner path (build always missed even with matching source + rustc). * streaming byte counters: BlobPutStream + BlobGetStream now record the transferred bytes via `record_blob_{put,get}_bytes`. Metric used to stay at 0 no matter how much you moved. * capture determinism: replaced `tar::Builder::append_dir_all` (uses `read_dir`'s native order) with `append_dir_sorted` that walks the tree recursively and sorts by filename bytes at every level. Two byte-identical trees now produce byte-identical tars regardless of filesystem ordering. +3 tests: - target_subdir_matches_cargo_layout (from #20) - fingerprint_companion_tag_uses_dotted_suffix - capture_is_order_independent_of_filesystem_readdir (guard against the exact bug we saw in the field) 252 tests pass (+1 from Phase 5h's 251). Pre-existing macOS `du -sb` failure unchanged. Supersedes #20 (also included here). Ready for re-deploy to tank + architect for the retest run. |
||
|
Omar Sobh
|
d36cec11a6 |
Phase 5g: cache metrics + GetMetrics RPC + peer-metrics CLI
Every RPC handler that answers a hit-or-miss question now increments
lock-free atomic counters. The GetMetrics RPC (0x13) returns a JSON
snapshot of every counter; the new claw-cargo peer-metrics CLI
prints hit rates, byte volumes, and counter uptime.
Placement engines can now poll these across the fleet to bias runner
scheduling toward whichever node has the warmest cache for a given
repo/tag combination.
## New module: cluster/metrics.rs (268 lines)
Types:
- CacheMetrics — atomic counters, all AtomicU64, Relaxed ordering
(metrics are advisory, not consistency-critical)
- MetricsReply — JSON snapshot returned by GetMetrics
Public API:
- CacheMetrics::new() — timestamped start, all counters at 0
- record_get_ref_hit / _miss
- record_get_tag_hit / _miss
- record_blob_get_bytes / record_blob_put_bytes
- record_get_chunk_hit / _miss
- record_has_chunk_hit / _miss
- snapshot() — atomic-load every field into a MetricsReply
MetricsReply derived helpers:
- get_ref_hit_rate() / get_tag_hit_rate() / has_chunk_hit_rate() —
Option<f64> so 0/0 returns None instead of NaN
## RPC method
- GetMetrics (0x13): payload = empty; reply = JSON MetricsReply
Wire-level instrumentation added to RpcRouter dispatch:
- GetRef → record_get_ref_hit / _miss
- GetTag → record_get_tag_hit / _miss
- BlobGet → record_blob_get_bytes (on hit)
- BlobPut → record_blob_put_bytes
- HasChunk → record_has_chunk_hit / _miss
- GetChunk → record_get_chunk_hit + record_blob_get_bytes on hit
/ record_get_chunk_miss
RpcRouter grows Arc<CacheMetrics> unconditionally — every router has
metrics, so a fresh node with no traffic still returns a valid
snapshot with all zeros + started_unix.
Streaming variants (BlobPutStream / BlobGetStream) don't yet track
byte counts — they'd require plumbing the count out of put_stream /
stream_to. Follow-on if it turns out to matter for placement.
## Client helper + CLI
- call_get_metrics(&conn) → Result<MetricsReply>
- claw-cargo peer-metrics [--peer ...] [--peer-addr ...] [--tls-dir ...]
Fetches + pretty-prints:
counter uptime: 42s
GetRef hits/misses: 123 / 45
hit rate: 73.21%
GetTag hits/misses: 8 / 2
hit rate: 80.00%
HasChunk hits/miss: 512 / 88
hit rate: 85.33%
GetChunk hits/miss: 47 / 12
Blob GET bytes: 1.23 GiB
Blob PUT bytes: 3.45 GiB
human_bytes() helper picks GiB / MiB / KiB / B based on magnitude.
Subcommand count now 9: build / prefetch / status / fingerprint /
pin / unpin / list-tags / prewarm / peer-metrics.
## Tests (13 new, all real filesystem / real QUIC — no mocks)
CacheMetrics (6):
- new_starts_all_counters_at_zero_except_timestamp
- recorders_increment_the_right_field (every recorder × 1-2 counts)
- hit_rates_none_when_zero_events (avoids 0/0 NaN)
- hit_rates_compute_correctly (3 hits / 1 miss → 75%)
- snapshot_round_trips_through_json
- snapshots_across_threads_are_consistent_up_to_relaxed_ordering
(8 threads × 1000 increments → exactly 8000)
RPC integration (7):
- phase_5g_method_byte_encoding
- get_metrics_returns_empty_snapshot_before_any_activity
- get_ref_records_hit_and_miss_counters (2 hits + 1 miss)
- get_tag_records_hit_and_miss_counters
- blob_get_and_blob_put_record_byte_counts
- has_chunk_and_get_chunk_record_hit_miss_counters
- **end_to_end_get_metrics_over_real_quic** — seed activity locally,
fire GetRef/GetTag/BlobGet dispatches to move the counters, then
fetch metrics through real QUIC + mTLS and verify each field
including the 0.5 hit rate calculation
238 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/metrics.rs: 268
- cluster/rpc.rs: 818
- cluster/rpc/tests_phase5.rs: 905
- claw_cargo.rs: 894
## What this enables
Fleet-wide visibility into which peer is actually serving traffic:
# From anywhere with connectivity + fleet mTLS
claw-cargo peer-metrics --peer tank
claw-cargo peer-metrics --peer architect
claw-cargo peer-metrics --peer morpheus
Compare hit rates side by side to see which node's cache is warmest.
A placement engine can automate this — poll every 30s, feed the
scheduler.
## Follow-on
- 5h: streaming variant of prewarm (fixed-memory ceiling for many-GB
blobs)
- 5i: metrics also published via gossip so PeerView carries hit rate
without a per-peer GetMetrics roundtrip
- 5j: prometheus /metrics endpoint on the daemon for existing dash
integrations
- 6: FUSE mount for warm-tier git worktrees
|
||
|
Omar Sobh
|
db55903311 |
Phase 5f: claw-cargo prewarm — cross-peer cache copy
The last piece before "Gitea webhook triggers a cache-warm for the
CI runner before its build starts." Adds a `prewarm` subcommand that
copies a tagged cache from one peer (upstream) to another (downstream)
in one shot — same tag, same BlobId, both sides serve it after.
## New subcommand
```
claw-cargo prewarm \
--from-peer tank --from-addr 10.0.0.14:7702 \
--to-peer morpheus --to-addr 10.0.0.15:7702 \
--tls-dir /etc/claw-store/tls \
--pin clawverse:main:latest
```
Flow:
1. Connect to upstream with local mTLS identity
2. `GetTag(tag)` → BlobId; `BlobStat(BlobId)` → size + chunk count
3. `BlobGetStream(BlobId)` → download bytes
4. Connect to downstream (second QUIC endpoint, same identity)
5. `BlobPutStream(bytes)` → returns BlobId; verified equal to upstream's
6. `PutTag(tag → BlobId)` on downstream
Summary output shows tag, blob id, both endpoints, byte count,
download/upload timings, total wall clock.
Assumes upstream + downstream share the same fleet CA (the common
case). Mixed-fleet variant with distinct identities is a follow-on.
## Integrity check
`assigned_id != blob_id` after the downstream upload triggers a
bail — the two BlobIds must match because content is BLAKE3-hashed
end-to-end. If they don't, the wire path corrupted bytes and the
whole prewarm fails loud rather than silently pinning a bad blob.
## Buffered vs streamed
Current implementation buffers the whole blob in memory between
download and upload. Fine for cargo target dirs (~1-5 GB compressed);
would break for a 20 GB blob. A follow-on will pipe upstream → tokio
duplex → downstream to run at fixed memory.
## Tests (1 new, real 2-peer QUIC)
- **`end_to_end_prewarm_copies_tagged_blob_between_two_peers`**
Two full RpcRouters serving in-process (A upstream + C downstream),
each on a distinct port. Seeds A with a blob + tag, then runs the
exact sequence prewarm runs internally: `GetTag → BlobGetStream`
against A, then `BlobPutStream → PutTag` against C. Verifies that
C's blob store returns byte-equal payload and C's tag store now
points at the same BlobId. Proves the composition works.
## Housekeeping
`rpc/tests.rs` hit 1408 lines with the new prewarm test. Phase 5
tests (5b refs + 5d tags + 5e restore + 5f prewarm) split to
`rpc/tests_phase5.rs` via a second `#[path]` module in rpc.rs.
Result:
- rpc/tests.rs: 727 (phase 1-2d tests)
- rpc/tests_phase5.rs: 722 (phase 5 tests)
- rpc.rs: 777
- rpc/client.rs: 589
- claw_cargo.rs: 818
- All under ceiling.
225 tests pass. Pre-existing macOS-only failure unchanged.
## What this enables
The complete CI runner flow now works end-to-end:
```
Primary (e.g. tank):
claw-cargo build # first ever build — MISS, uploads
claw-cargo pin --name clawverse:main:latest
Fleet control plane on PR open:
gitea webhook → shell hook → claw-cargo prewarm \
--from-peer tank --to-peer $RUNNER_LOCAL \
--pin clawverse:main:latest
# runner's local daemon now serves the tag + blob
Runner picks up job:
claw-cargo build --peer 127.0.0.1:7702
# local daemon is warm → prefetch returns HIT
# cargo build runs against restored deps → workspace-crates only
# 50 min → 3 min
```
Every subcommand claw-cargo needs for this pipeline now exists:
build / prefetch / prefetch --pin / status / fingerprint /
pin / unpin / list-tags / prewarm.
## What's next
- 5g: cache hit/miss metrics into gossip so placement engines can
bias runner scheduling toward warm nodes
- 5h: streaming variant of prewarm (tokio duplex) for many-GB blobs
- 6: FUSE mount for warm-tier git worktrees
- 3: full CRDT metadata if the plain-tag model shows conflict problems
|
||
|
Omar Sobh
|
b7904b59a5 |
Phase 5d: named tags + pin/unpin/list-tags CLI
Human-readable pins on top of the raw 32-byte ref layer. Operators
publish `clawverse:main:latest-cache` → BlobId once, then everything
downstream (CI runners, dev laptops) references the tag instead of
passing 64-char hex hashes around.
## Module: cluster/tags.rs (433 lines)
TagStore for string-key → 32-byte-value:
- open(root) — creates layout, safe on existing stores
- put(key, value) / get(key) / delete(key) / contains(key)
- list() — sorted by key
- Atomic writes via tempfile + rename
- Key length capped at MAX_TAG_KEY_BYTES (4 KiB); empty keys rejected
On-disk record: `key_len:u16 (LE) || key_bytes || value:32bytes`.
Filename is `blake3(key)` hex so arbitrary UTF-8 keys land at
deterministic paths without shell escaping.
TagEntry type (public, serde) for list results:
`{ key, value_hex }`. Includes `decode_value() → Result<[u8;32]>`.
## RPC methods
- PutTag (0x0f): payload = encoded record → STREAM_STATUS_OK / err
- GetTag (0x10): payload = key bytes → 32-byte value / NotFound
- DeleteTag (0x11): payload = key bytes → STREAM_STATUS_OK / NotFound
- ListTags (0x12): payload = empty → JSON Vec<TagEntry>
RpcRouter grows optional Arc<TagStore> via `.with_tag_store(store)`.
## Services + config
ClusterServices auto-opens a TagStore at `<blob_store_root>/tags-db`
alongside the ref store. `tag_store` field on ClusterServices, same
enable-with-blob-store semantics.
## claw-cargo new subcommands
- `claw-cargo pin --name clawverse:main:latest`
Compute current fingerprint → look up its BlobId via GetRef →
publish TagStore mapping. Errors cleanly if the fingerprint
hasn't been built yet (nothing to point at).
- `claw-cargo unpin --name clawverse:main:latest`
Delete the tag. Prints "no such tag" if it wasn't set.
- `claw-cargo list-tags`
Print every tag with its 32-byte hex value.
Total subcommand count now 7: build / prefetch / status / fingerprint
/ pin / unpin / list-tags. All share the layered config from Phase 5c.
## Client helpers
- call_put_tag / call_get_tag / call_delete_tag / call_list_tags
- All follow the same error-mapping conventions as prior client helpers
(NotFound → Ok(None) or Ok(false), everything else → Err)
## Housekeeping
rpc.rs was pushing past the 1300-line ceiling with the tag methods
added. Client helpers moved to `cluster/rpc/client.rs` with a
re-export (`pub use client::*;`) so external callers still write
`cluster::rpc::call_*`. Result:
- rpc.rs: 773 (was 1343)
- rpc/client.rs: 593 (new)
- rpc/tests.rs: 1235
- All under ceiling.
## Tests (33 new, all real filesystem / real QUIC — no mocks)
TagStore (16 in cluster/tags.rs):
- open_creates_layout
- get_returns_none_for_missing (+ contains false)
- put_and_get_round_trip
- put_overwrites_prior_value
- delete_returns_true_for_existing_and_false_for_missing
- put_rejects_empty_key
- put_rejects_oversize_key
- list_returns_all_tags_sorted
- list_is_empty_on_fresh_store
- keys_with_slashes_and_colons_round_trip (real-world tag shape)
- encode_and_decode_round_trip (raw wire format)
- decode_rejects_short_record
- decode_rejects_length_mismatch
- decode_rejects_non_utf8_key
- tag_entry_decode_value_round_trip
- tag_entry_decode_value_rejects_bad_hex
RPC dispatch (7 new):
- phase_5d_method_byte_encoding
- tag_rpcs_return_not_configured_without_store
- put_tag_stores_and_get_tag_reads_back
- get_tag_returns_not_found_for_missing
- get_tag_rejects_empty_key
- delete_tag_removes_and_returns_not_found_after
- list_tags_returns_json_sorted
End-to-end over real QUIC (1):
- **end_to_end_pin_lookup_delete_over_real_quic** — publish tag →
look up → list → delete → confirm gone. Full round trip through
the wire layer including JSON deserialization of the list.
Also 8 downstream tests continued passing after the client.rs split
(no test moved, they were untouched).
223 tests pass. Pre-existing macOS-only failure unchanged.
## What this enables
Operator flow:
# Build once on the primary
$ claw-cargo build
→ cache MISS → cargo build (50 min) → capture + upload
→ summary: fingerprint 4a3b…, blob 8c2f…, uploaded 3.2 GiB
# Publish a friendly name
$ claw-cargo pin --name clawverse:main:2026-07-12
pinned: clawverse:main:2026-07-12
fingerprint: 4a3b2c…
blob: 8c2f1a…
# Anyone else can now find it via list-tags
$ claw-cargo list-tags
clawverse:main:2026-07-12 8c2f1a…
clawverse:main:latest 8c2f1a…
# CI runner sees the same fingerprint in its workspace state, hits
# the ref directly via GetRef — the tag is for operator visibility
## Follow-on
- 5e: prefetch --pin <tag> — bypass fingerprint compute, download
the tagged BlobId directly (useful when you want an old cache to
test regression scenarios)
- 5f: gitea webhook pre-fetch — daemon pre-warms cache for known
fingerprints before CI runner starts
- 3: full CRDT metadata layer (namespaces, versioned pointers,
vector clocks) if the plain-tag model turns out to have
real-world conflict scenarios
|
||
|
Omar Sobh
|
2d09b4687c |
Phase 5b: KV refs + claw-cargo CLI (the killer feature, live)
Ships the actual user-facing cargo build cache. Combined with Phase 5a
(fingerprint + capture + restore) + the whole Phase 2 blob substrate,
`claw-cargo build` now runs `cargo build` with a peer-cache lookup:
hit → download+restore, miss → build+capture+upload.
## What ships
### cluster/refs.rs (243 lines)
A dumb 32-byte-key → 32-byte-value directory-backed store. Used to map
fingerprints → BlobIds. Layout mirrors BlobStore:
<root>/
refs/<kk>/<key_hex>.ref — 32 raw bytes
.tmp/ — atomic-rename staging
Public API: RefStore::open / get / put / delete / contains. All writes
atomic via tempfile + rename. Deliberately no versioning or CRDT
semantics — that's Phase 3. Every real cargo-cache lookup is a
single-key-single-value shape.
### New RPC methods
- GetRef (0x0d): payload = 32-byte RefKey; reply = 32 bytes / NotFound
- PutRef (0x0e): payload = 32-byte RefKey || 32-byte RefValue;
reply = STREAM_STATUS_OK / error
### RpcRouter + services
- RpcRouter grows optional Arc<RefStore> via `with_ref_store`
- ClusterServices opens a RefStore alongside the BlobStore when
`blob_store_root` is configured (co-located at `<blob_root>/refs-db`)
- `blob_store_enabled()` / `ref_store_enabled()` introspection
### claw-cargo binary (319 lines)
New bin target `claw-cargo` — thin CLI wrapping the whole stack:
claw-cargo fingerprint --profile release --features "a,b"
→ prints the workspace fingerprint (no network)
claw-cargo build \
--peer <name> --peer-addr <ip:port> --tls-dir <dir> \
--profile release --features "a,b" \
-- --workspace=x --frozen ...
→ 1. compute fingerprint
2. QUIC + mTLS connect to peer
3. GetRef(fingerprint) → BlobId?
HIT: BlobStat → BlobGetStream → restore_target → cargo build
MISS: cargo build → capture_target → BlobPutStream → PutRef
4. Print summary: fingerprint, hit/miss, bytes, cargo elapsed
## Live smoke test
Ran claw-cargo fingerprint on this workspace with three profile/feature
combos — got three distinct 32-byte fingerprints. Same profile+features
on the same workspace state → same fingerprint (Phase 5a's guarantee
carried through the CLI).
## Tests (14 new, all real — no mocks)
Refs store (7):
- open creates layout
- get returns None for missing
- put + get round-trips
- put overwrites prior value
- delete removes ref + reports (false on second delete)
- distinct keys produce distinct on-disk files (bucket fan-out proof)
- rejects_wrong_length_on_disk (corruption detection)
RPC (7):
- phase_5b_method_byte_encoding
- get_ref_returns_not_found_for_missing
- put_ref_stores_and_get_ref_reads_back
- put_ref_rejects_wrong_length_payload
- get_ref_rejects_wrong_length_payload
- ref_rpcs_return_not_configured_without_store
- end_to_end_put_ref_get_ref_over_real_quic — full 2-node QUIC + mTLS
round trip proving PutRef/GetRef work at the wire level
188 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/refs.rs: 243
- cluster/rpc.rs: 1169
- cluster/rpc/tests.rs: 1073
- cluster/services.rs: 565
- claw_cargo.rs: 319
## Where this leaves us
The distributed FS + cargo cache is functionally complete for the
happy path:
Node A builds clawverse for the first time
→ cargo build (50 min cold)
→ capture_target (a few seconds)
→ push to node B via BlobPutStream (network-bound)
→ PutRef(fingerprint → BlobId)
Node B on the same workspace state runs `claw-cargo build …`
→ compute_fingerprint (ms)
→ GetRef → hit
→ BlobGetStream (network-bound)
→ restore_target (a few seconds)
→ cargo build → sees valid deps/.fingerprint, builds only
workspace crates (~3 min instead of 50)
Same workspace state on a third machine? Same fingerprint → same
cache hit. That's the whole design.
## Follow-on
- Phase 5c: pre-fetch on Gitea webhook so CI runners never wait
- Phase 5d: metric ticker publishes cache hit rate into gossip so
the placement engine can bias runner scheduling toward warm nodes
- Phase 3: CRDT metadata for human-readable pins on top of raw
32-byte refs (`clawverse:main:latest-cache` → fingerprint hex)
- Phase 6+: FUSE mount for the warm-tier git worktrees
|
||
|
Omar Sobh
|
2e984b924d |
Phase 2d: chunk-level RPC (HasChunk / PutChunk / GetChunk / PutManifest)
Unlocks partial-sync replication — a peer that already has some
chunks of a blob (typical when two nodes share overlapping cargo
build caches) only receives the chunks it's missing.
## New methods
| Byte | Method | Payload | Reply |
|---|---|---|---|
| 0x09 | HasChunk | 32-byte ChunkHash | STREAM_STATUS_OK / NotFound |
| 0x0a | PutChunk | ChunkHash \|\| bytes | STREAM_STATUS_OK / error |
| 0x0b | GetChunk | ChunkHash | STREAM_STATUS_OK \|\| bytes / NotFound |
| 0x0c | PutManifest | JSON BlobManifest | JSON PutManifestReply |
`PutManifestReply { blob_id, missing: Vec<ChunkHash> }`: empty
`missing` means the manifest was written; non-empty tells the
client which chunks to upload before retrying.
Server verifies bytes hash to claimed hash on PutChunk; a
mismatch surfaces as InvalidRequest and the store is untouched.
## BlobStore additions
- `has_chunk(&ChunkHash) → bool`
- `read_chunk(&ChunkHash) → Option<Vec<u8>>` — verifies hash on read
- `put_chunk(&ChunkHash, bytes) → Result<()>` — verifies bytes-vs-hash
- `put_manifest_verified(&manifest) → Result<Vec<ChunkHash>>` —
returns the list of chunks missing on disk (empty on success)
- `chunk_path` promoted to `pub` for advanced callers
## Client helpers
- `call_has_chunk` / `call_put_chunk` / `call_get_chunk` / `call_put_manifest`
- `push_blob_missing_chunks(conn, local_store, blob_id) →
Result<(uploaded, total)>` — high-level partial-sync helper
`push_blob_missing_chunks` loads the local manifest, calls HasChunk
for each chunk, uploads only the missing ones via PutChunk, then
commits via PutManifest. On a fully-overlapping cache the uploaded
count is 0 and only the ~small manifest crosses the wire.
## Tests (17 new, all real filesystem + real QUIC — no mocks)
Blob store (6):
- has_chunk_is_false_before_put_and_true_after
- read_chunk_returns_bytes_and_none_when_missing
- put_chunk_rejects_hash_mismatch (nothing written)
- read_chunk_detects_corruption (bit-flip → mismatch error)
- put_manifest_verified_reports_missing_chunks
- put_manifest_verified_writes_when_all_chunks_present
Router dispatch (7):
- phase_2d_method_byte_encoding
- method_reports_streaming_variants — extended for 4 new methods
- has_chunk_returns_ok_for_present_and_not_found_for_missing
- put_chunk_stores_and_returns_status_ok
- put_chunk_rejects_hash_mismatch_over_wire
- get_chunk_returns_content_prefixed_with_status_ok
- get_chunk_returns_not_found_for_missing
- put_manifest_reports_missing_chunks_when_incomplete
- put_manifest_writes_when_chunks_present
- chunk_rpcs_return_not_configured_without_store
End-to-end (2):
- **end_to_end_push_blob_missing_chunks_replicates_only_needed_bytes**:
Peer A pre-seeded with chunk 0 of a 2-chunk (8 MiB) blob;
`push_blob_missing_chunks` reports `(uploaded=1, total=2)`,
only chunk 1 crosses the wire, A's store then contains the
complete blob and `get_bytes` returns byte-equal content.
- **call_get_chunk_verifies_returned_hash**: real 2-node fetch,
client hashes received bytes and compares to requested hash.
156 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/rpc.rs: 1053
- cluster/rpc/tests.rs: 940
- cluster/blob.rs: 1186
## Where this fits
With Phase 2c whole-blob streaming + Phase 2d partial-chunk sync,
the storage substrate is now genuinely bandwidth-efficient in the
distributed setting:
- First-ever push of a blob: `push_blob_missing_chunks` uploads
everything (all chunks missing).
- Second push of a similar blob (95% chunk overlap with prior
contents): only the 5% new chunks cross the wire, plus a tiny
manifest.
- Whole-blob download: BlobGetStream, bounded by network bandwidth.
## Follow-on
- Phase 3: CRDT metadata for human-readable namespaces on top of
content hashes.
- Phase 5: the killer feature. Fingerprint cargo target dir → tar
→ hash → PutBlobStream (or push_blob_missing_chunks if a similar
build already lives on the peer). Same fingerprint on the next
node → BlobGetStream. This is the whole cargo-cache design in
one line and it now sits on a substrate that handles all the
hard cases (dedup, verification, resumability, partial sync).
|
||
|
Omar Sobh
|
1fd1027da4 |
Phase 2c: streaming Blob RPC (BlobPutStream / BlobGetStream)
Removes the 16 MiB message cap for blob transfers. The bounded Blob* methods from Phase 2b still exist; the streaming variants let a peer push or pull a many-GB blob without either side holding it in memory. ## Wire format Streaming methods use a slightly different reply shape so the client can route on the first byte alone: Reply : status:u8 || payload:bytes... Where `status` is either `STREAM_STATUS_OK` (0x00, content follows) or a single-byte ErrorCode. `serve_connection` now peeks at the method tag byte via read_exact and hands streaming methods the raw send/recv streams; bounded methods still use the old read_to_end path. ## Method additions - BlobPutStream (0x07): client streams bytes → server pipes into BlobStore::put_stream → reply is 0x00 || 32-byte BlobId - BlobGetStream (0x08): client sends 32-byte BlobId → server verifies existence, writes 0x00 status, then streams chunks from disk into the send stream Method::is_streaming() introspection so callers can decide which wire variant to use. ## BlobStore additions - put_stream<R: AsyncRead + Unpin>(reader) -> BlobId Memory ceiling: one CHUNK_SIZE (4 MiB) buffer regardless of blob size. Handles short-reads correctly (loops until CHUNK_SIZE bytes are available or EOF), including the empty-reader case (produces the empty-blob BlobId, zero chunks). - stream_to<W: AsyncWrite + Unpin>(id, writer) -> bool Ok(false) on NotFound (writer untouched). Verifies each chunk hash before emitting; corruption halts mid-stream with Err. ## Client helpers - call_blob_put_stream(conn, reader) -> Result<BlobId> Uses tokio::io::copy directly onto quinn's SendStream. - call_blob_get_stream(conn, id, writer) -> Result<bool> Ok(false) on NotFound; other errors surface as Err. ## Tests (11 new, all real — no mocks) Blob store (6): - put_stream_produces_same_hash_as_put_bytes (3-chunk blob via Cursor) - put_stream_handles_empty_reader (produces empty-blob BlobId) - put_stream_handles_short_reads (custom Trickle reader that only serves 100 bytes per read call — must still assemble full chunks) - stream_to_writes_full_blob (2-chunk write to Vec<u8>) - stream_to_returns_false_when_missing (writer untouched) - stream_to_detects_chunk_corruption (bit-flip a chunk → Err with "chunk hash mismatch") RPC (5): - method_reports_streaming_variants - end_to_end_stream_put_and_get_over_real_quic — 12 MiB + 777 bytes → 4 chunks, real 2-node QUIC + mTLS + stream round-trip - stream_get_returns_false_for_missing_blob - stream_methods_return_not_configured_without_store - stream_put_deduplicates_with_prior_put_bytes — verify streaming put produces the same BlobId as a prior bounded put on identical content, and the manifest chunk count didn't fork ## Housekeeping rpc.rs was tipping over the 1300-line ceiling with the streaming handlers + helpers + tests. Tests split into `cluster/rpc/tests.rs` via `#[path = "rpc/tests.rs"] mod tests;`. Result: - rpc.rs: 748 lines - rpc/tests.rs: 694 lines - blob.rs: 1002 lines - All under ceiling. 139 tests pass. Pre-existing macOS-only failure unchanged. ## What's next - Phase 2d: chunk-level RPC (BlobPutChunk / BlobGetChunk) so a receiver can `LoadManifest` then request only the chunks it's missing — big bandwidth win on partially-overlapping caches. - Phase 3: CRDT metadata for human-readable namespaces on top of content hashes. - Phase 5: the killer feature — fingerprint the cargo target dir, BlobPutStream it, next node BlobGetStream by the same fingerprint. Now buildable directly on Phase 2c since target dirs run 100 MB to a few GB and the previous 16 MiB cap would have blocked us. |
||
|
Omar Sobh
|
f43b34ad15 |
Phase 2b: Blob RPC (Stat/Get/Put/LoadManifest)
Wires the Phase 2 content-addressed blob store onto the network via
four new methods on the existing RpcRouter. Combined with the mTLS +
gossip stack from Phase 1c-1e, peers can now exchange content-addressed
blobs over real QUIC. This is the substrate the fingerprint-keyed cargo
cache (Phase 5) sits on directly.
New methods:
- BlobStat (0x03): payload = 32-byte BlobId, reply = JSON BlobStat
- BlobGet (0x04): payload = 32-byte BlobId, reply = raw bytes
- BlobPut (0x05): payload = raw bytes, reply = 32-byte BlobId
- BlobLoadManifest (0x06): payload = 32-byte BlobId, reply = JSON manifest
New error codes:
- NotFound (0xf3) — the requested BlobId isn't in the local store
- InvalidRequest (0xf4) — e.g. non-32-byte payload for a hash-keyed method
- NotConfigured (0xf5) — Blob* called on a router without an attached store
Wire-format bump: MAX_MESSAGE_BYTES 16 KiB → 16 MiB so a single 4 MiB
chunk (plus JSON overhead) fits comfortably. Anything above 16 MiB
needs the streaming variants coming in Phase 2c.
Client helpers:
- call_blob_stat / call_blob_get / call_blob_put / call_blob_load_manifest
- All map ErrorCode::NotFound to Ok(None), other codes to Err.
- call_blob_put verifies the peer-assigned BlobId matches local blake3
hash of the payload — corruption or protocol drift surfaces
immediately instead of silently accepting a mismatched receipt.
Router changes:
- RpcRouter grows an Option<Arc<BlobStore>> via with_blob_store(store).
- handle() split into handle_outcome() → HandlerOutcome enum
{Reply(bytes) | Error(ErrorCode)} for cleaner control flow across
the growing method set.
Services / daemon:
- ClusterServices::start gains blob_store_root: Option<PathBuf>.
- ClusterConfig gains blob_store_root: Option<PathBuf>.
- daemon.rs reads it from cluster_cfg + passes through.
- New ClusterServices::blob_store_enabled() introspection.
Tests (11 new, all real — no mocks):
Router (7 new):
- method_round_trips_byte_encoding — updated for 6 methods
- error_code_describe_covers_all_variants — updated for 6 codes
- decode_error_covers_all_known_codes
- blob_rpcs_return_not_configured_without_store — all four Blob*
methods return NotConfigured when the router lacks a store
- blob_stat_returns_not_found_for_missing
- blob_stat_returns_json_for_existing
- blob_stat_returns_invalid_request_for_bad_length
- blob_get_returns_content_bytes
- blob_put_stores_bytes_and_returns_hash — verifies BlobId matches
independent local hash
- blob_load_manifest_returns_json_for_existing (2-chunk case)
- blob_load_manifest_returns_not_found_for_missing
End-to-end over real QUIC (2 new):
- end_to_end_blob_put_stat_get_over_real_quic — full 4-method loop
(Put → Stat → Get → LoadManifest) + NotFound path
- end_to_end_multi_chunk_blob_over_real_quic — 6 MiB blob → 2 chunks,
proves MAX_MESSAGE_BYTES bump took effect
Services (1 new):
- services_with_blob_store_serves_blob_rpc_end_to_end — cut CA, sign
leaves, config includes blob_store_root, start ClusterServices,
dial from B over persisted mTLS, put + get through the router,
then independently verify bytes landed on A's on-disk store
Also fixed a parallel-test port collision: services `next_port()`
now increments by 2 so `port + 1` (the RPC bind) is reserved
alongside `port` (the gossip bind).
128 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/rpc.rs: 1006
- cluster/services.rs: 535
- cluster/blob.rs: 802
- config.rs: 511
- daemon.rs: 265
Follow-on:
- 2c: streaming variants (AsyncRead/AsyncWrite) so a many-GB blob
transfers without holding it in memory
- 2d: chunk-level RPC (BlobPutChunk / BlobGetChunk) so a receiver
can request only chunks it's missing after LoadManifest
- Phase 5 (the killer feature) can now build on Phase 2b directly —
fingerprint the target dir, PutBlob the compressed tarball, and
next node calls GetBlob keyed by the same fingerprint hash.
|
||
|
Omar Sobh
|
2ae079fbd4 |
Phase 1e: daemon-integrated cluster services + PeerStatus RPC
Ties Phase 1a-1d together into a live subsystem the daemon actually
runs. When `[cluster]` is present in config, `claw-store daemon` now:
1. Starts chitchat gossip via ClusterServices::start.
2. Publishes hot.max_gb immediately and re-measures the hot dir
every 30s, updating `clawstor.hot.used`.
3. If `[cluster.tls]` is also configured, loads NodeIdentity from
PEM files and binds a QUIC RPC server that accepts + dispatches
incoming Ping / PeerStatus requests via RpcRouter.
New modules:
- cluster/rpc.rs (510 lines):
- Method enum (Ping=0x01, PeerStatus=0x02)
- ErrorCode enum (EmptyRequest / UnknownMethod / HandlerFailure)
- PeerStatusReply { local_name, local_zone, peers: Vec<PeerView> }
- RpcRouter — dispatch state (holds Arc<ClusterGossip> + local
name/zone)
- rpc_call / call_ping / call_peer_status — client helpers
- serve_connection — server accept-bidi loop
- Wire format: `method:u8 || payload:bytes` → `reply:bytes` or
single-byte ErrorCode
- cluster/services.rs (418 lines):
- ClusterServices { gossip, accept_task, metric_task }
- start(cluster_cfg, name, hot_dir, hot_max_bytes) → bootstraps
everything above
- shutdown() aborts background tasks cleanly
- Recursive dir-walker (spawn_blocking) for hot-tier metric
PeerView now derives Serialize/Deserialize so it round-trips through
JSON over the RPC.
daemon.rs integration (~35 lines added):
- Bootstraps ClusterServices before entering the select loop
- Held for daemon lifetime
- Shutdown on SIGTERM
- Gossip-less config still runs standalone (backwards compat)
CLI: `cluster-peer-status --peer <name> --rpc-addr <addr>
--tls-dir <dir>`
Loads a persistent identity, dials the peer, calls PeerStatus,
prints the peer's local view as a table.
Tests (15 new, all real — no mocks, real UDP + TLS + JSON round trip):
RPC (9):
- method_round_trips_byte_encoding
- dispatch_returns_pong_for_ping
- dispatch_returns_json_for_peer_status
- dispatch_returns_empty_request_error
- dispatch_returns_unknown_method_error
- rpc_call_rejects_oversize_payload (with real quinn connection)
- end_to_end_ping_and_peer_status_over_real_quic — full 2-node quinn
with mTLS + both RPCs
- peer_status_reflects_peer_gossip_state — 2 gossip instances converge,
RPC caller from a third identity sees the converged view
- error_code_describe_covers_all_variants
Services (6):
- dir_bytes_sync_returns_zero_for_missing_path
- dir_bytes_sync_sums_recursive_file_sizes (3-level nesting)
- services_start_without_tls_leaves_rpc_disabled
- services_start_with_tls_serves_rpc_end_to_end — full stack: cut CA on
disk, sign leaves, start ClusterServices for A, dial from B via
persisted mTLS, run ping + PeerStatus over the wire
- services_publish_hot_used_metric_periodically
- services_gossip_sees_peer_after_convergence
95 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/rpc.rs: 510
- cluster/services.rs: 418
- cluster/gossip.rs: 579
- cluster/transport.rs: 916
- daemon.rs: 262
- main.rs: 749
Phase 1 done end-to-end. `claw-store daemon` now boots a real distributed
cluster stack when config asks for one; peers can call each other's
PeerStatus RPC and see live membership. Next: Phase 2 (content-addressed
blob store) can hook new RPC methods into the same RpcRouter with a
one-line dispatch arm.
|