15 Commits

Include renames

Author	SHA1	Message	Date
osobh	c3f6b500fc	Phase 9 R1: RepoEnsure — peer RPC + aggregator fan-out (#106)	2026-07-15 11:19:35 +00:00
Omar Sobh	dbc1587bcb	Phase 8d: daemon binds a second QuicServer on the tailnet interface ... Live smoke on tank↔architect exposed the gap: bind_rpc_tailscale was being *advertised* via gossip so peers learned to dial it, but the daemon never actually LISTENED there. Tailnet dials hit a closed port. Fix: when both bind_rpc_lan and bind_rpc_tailscale are set (and differ), spawn a second QuicServer on the tailnet address. Shares the same fleet-CA identity + RpcRouter as the LAN listener — requests from either side hit the same handlers. If the second bind fails (e.g. tailnet interface not up), we log a warning and keep the LAN listener alive rather than aborting daemon startup. Standard graceful-degrade shape. No new tests here — a live integration test would need two network interfaces + a running tailscale, which the CI runners don't have. Coverage happens on the tank+architect deployment: `ss -lunp` on architect must show TWO clawstor UDP listeners after this change (10.0.0.13:7702 + 100.104.171.32:7702). Follow-on: cert SAN for the tailnet address. The current fleet-CA-signed leaf only has the node name as SAN, so rustls verification on the client side still checks against --peer <name> which passes because CN == node name. But a belt-and-suspenders leaf using fleet-ca-tailscale-sign (Phase 8a) would be more correct.	2026-07-14 12:00:06 -07:00
Omar Sobh	eebc62d87b	Phase 7d follow-on: snapshots pin blobs against LRU eviction ... Closes the retention loop between snapshots and pin-aware LRU eviction. A snapshot is not just a "list of blobs at time T" any more — it's a *retention pin* on every blob it captures. Operators can guarantee a build stays on disk for N days by snapshotting it and pruning the snapshot when the window is up. Additions: * SnapshotStore::pinned_blob_ids() → union of blob_ids across all live snapshots. Cheap: one JSON read per snapshot. * cmd_cluster_gc extends the tag-pin set with snapshot pins before handing it to evict_to_size_cap_with_pins. Output line now reads "pinned blobs: N (M from snapshots)". * ClusterServices auto-GC ticker does the same on every tick; log fields include snapshot_pins so ops see the retention set size at a glance. +2 tests: - pinned_blob_ids_unions_all_snapshots (overlap dedupe) - pinned_blob_ids_empty_when_no_snapshots 355 tests pass (+2). Pre-existing macOS hot::tests::test_project_target_size_bytes failure unchanged.	2026-07-14 09:30:31 -07:00
Omar Sobh	1418d35487	Phase 4b: TagStore expiry primitives (pin TTL groundwork) ... Adds the on-disk mechanism for time-scoped pins. No RPC or CLI yet — a follow-on will expose \`pin --ttl <duration>\`. This PR is purely library + eviction wiring. Layout addition: alongside each stamped tag at \`tags-v2/<hh>/<hash>.svtag\`, an optional sidecar \`tags-v2/<hh>/<hash>.svtag.exp\` holds an 8-byte LE unix \`expires_at\`. Absence of the sidecar = never expires (current behavior). New TagStore methods: * set_stamped_expiry(key, expires_at_unix) — writes sidecar; passing 0 removes it. Idempotent. * get_stamped_expiry(key) — reads sidecar; None when absent. * pinned_blob_values_at(now_unix) — same union as pinned_blob_values, but skips stamped tags whose sidecar shows expires_at ≤ now. Legacy tags/ entries never expire. * prune_expired_stamped_at(now_unix) — deletes stamped tags AND their sidecars where expires_at ≤ now. Returns count. * pinned_blob_values() — now a shim that calls _at(u64::MAX) for 100% backward compat. Wired the two existing gc call sites: * ClusterServices auto-GC ticker prunes-then-collects at SystemTime::now(). One pass per tick. * \`claw-store cluster-gc --evict-to-gb N\` CLI same pattern. Report now includes \"expired pins pruned: N\". +1 test (expiry_gates_pin_set_and_prune_removes_expired): covers live/expired/no-ttl mix, sidecar round-trip, prune removes only expired, expires_at=0 clears sidecar, dropped tag stops filtering. 286 tests pass (+3 from 283). Pre-existing macOS hot::tests::test_project_target_size_bytes failure unchanged.	2026-07-13 15:25:41 -07:00
Omar Sobh	5be11a11b0	Phase 4a: pin-aware LRU eviction ... A `claw-cargo pin` used to be silently vulnerable to the size-cap eviction ticker — the tag existed but the underlying blob could get LRU'd out, leaving a dangling reference. Now tags act as retention markers: any blob referenced by any tag (stamped or legacy) is protected from `evict_to_size_cap`. * `BlobStore::evict_to_size_cap_with_pins(max_bytes, pinned_set)` — same LRU-by-mtime pass, but pinned blob IDs skip the eviction loop. Existing `evict_to_size_cap` is now a thin wrapper with an empty pin set (100% backward compat). * `TagStore::pinned_blob_values()` — unions every 32-byte value referenced by any tag across `tags/` (legacy) and `tags-v2/` (Phase 3c stamped). Dedupes naturally. * Auto-GC ticker in `ClusterServices` now collects the pin set on every eviction pass and passes it in. Log fields include `pinned_blobs = N` so operators can see the retention set size. * `claw-store cluster-gc --evict-to-gb N` CLI opens the tag store the same way, prints `pinned blobs: N` in the report. +3 tests: - evict_with_pins_protects_pinned_blobs_from_eviction — 3 blobs ordered oldest→newest, pin the oldest; without pins LRU would evict it; with pins the next-oldest goes instead. Guards the main semantic. - evict_with_pins_stops_when_pinned_footprint_dominates — everything pinned + cap = 0 → no-op. Guards the "operator asked for the impossible" case. - pinned_blob_values_unions_both_stores — legacy tag with value V1, stamped tag with value V2, second stamped tag also referencing V1 → set contains {V1, V2}. Dedupe check. 283 tests pass (baseline +3). Pre-existing macOS failure unchanged.	2026-07-13 13:54:47 -07:00
Omar Sobh	58c5bc341b	GetRef: transparent ref-forwarding on local miss ... Cross-runner cache silos (tank + architect measured on 2026-07-13): same fingerprint, same rustc, but each runner's daemon only knows about the refs its own runner uploaded. Every runner that lands on a peer that isn't tank re-uploads a duplicate blob. Fix: on `GetRef` miss the daemon fans out to alive gossip peers via a strict-local `GetRefLocal` variant, and the FIRST peer that has the ref triggers a transparent pull — chunks + manifest into the local blob store, then `PutRef` locally — before returning the value to the caller. Subsequent lookups are pure-local hits. * `Method::GetRefLocal = 0x14` — new wire method, identical shape to GetRef but the peer MUST NOT recurse. Loop prevention: our forwarding only calls `GetRefLocal` on peers, so chain depth is always 1. * `RpcRouter::with_outbound_client(Arc<QuicClient>)` — dependency injection point for the forwarding dial path. `None` disables forwarding entirely (GetRef becomes GetRefLocal-equivalent). * `RpcRouter::forward_get_ref(key)` — concurrent peer probes via `JoinSet`, 3s timeout per dial, first successful pull wins, remaining tasks aborted. * `pull_blob_locally` — walks manifest, fetches only chunks the local store lacks (`has_chunk`), commits via `put_manifest_verified`. Bounded memory: one 4 MiB chunk at a time. * `ClusterServices::start` loads NodeIdentity twice — server takes ownership; outbound client gets its own copy for TLS presentation on peer dials. Wires the outbound client into the router when TLS material is available. * `call_get_ref_local(conn, key)` client helper (used by daemon forwarding + available to any RPC consumer that wants the no-recursion semantics). +3 tests in `rpc/tests_forwarding.rs`: - Local hit works without forwarding; local miss with no peers returns None. Guards the base cases. - GetRefLocal never forwards even when outbound is configured (no peers reachable → miss returns None immediately, no attempted fan-out). - Method byte 0x14 encoding is stable across releases. Full end-to-end forwarding is exercised in the pilot deploy: two daemons on the fleet-CA, tank populates a ref, architect's runner GetRef → tank forwards → architect pulls → HIT locally next time. 264 tests pass (baseline +3). Pre-existing macOS failure unchanged.	2026-07-13 06:50:53 -07:00
Omar Sobh	2f3055a3aa	blob: size-based LRU eviction + auto-cap in the GC ticker ... Orphan-chunk GC alone doesn't stop unbounded growth: as long as fingerprint→blob refs keep getting PutRef'd, the manifest set keeps growing and no chunk is ever an orphan. * `BlobStore::evict_to_size_cap(max_bytes)` — walks manifests oldest first by mtime, deletes them, refcount-decrements each chunk they used, unlinks + reclaims size for any chunk whose refcount hits zero. Shared chunks stay put until the last blob referencing them is evicted. * `ManifestSummary` internal type keeps the diff-set bookkeeping cheap (one HashMap<ChunkHash, u32>, no repeated tree walks). * `claw-store cluster-gc --evict-to-gb <N>` extends the CLI: still runs the orphan sweep first, then optionally caps the store. * Config: `cluster.blob_max_gb: Option<u64>`. The auto-GC ticker runs eviction after every orphan sweep when this is set. Silent when the store is already under cap; INFO log when it evicts. +3 tests: - evict_to_size_cap_reclaims_oldest_blobs_first: 3 blobs with distinct mtimes, cap below combined size → oldest evicted, newer blobs survive - evict_keeps_shared_chunks_when_still_referenced: guards the refcount decrement path (content-addressed dedup keeps identical content as one blob → chunk survives until manifest deleted) - evict_on_empty_store_is_a_noop: sanity 257 tests pass (baseline +3). Pre-existing macOS failure unchanged.	2026-07-12 06:35:17 -07:00
Omar Sobh	84aa758fd4	Two pilot follow-ons: rustc drift warning + blob GC ... Both surfaced by the 2026-07-12 pilot as real operator concerns: ## rustc drift warning at build time Runners silently silo their cache when rustc versions differ across peers (fingerprint depends on rustc verbose output). The pilot's first flow burned a full cold+upload before we realized the silo. - `PeerStatusReply.local_rustc_release` — new field, populated from the peer's own gossip `RUSTC_RELEASE` key via a new `ClusterGossip::self_kv(key)` accessor. - `claw-cargo build`: on cache MISS, calls `PeerStatus`; if the peer's rustc release ≠ our local `rustc --version`, emits a WARN with both versions + hint to add `rust-toolchain.toml`. - Best-effort: absence of either release string is a shrug, not an error. ## blob GC Blob store grows unbounded on a runner; disk-full is a real incident. `gc_orphan_chunks` already existed but wasn't exposed. - New CLI: `claw-store cluster-gc` — runs `gc_orphan_chunks`, prints report. Safe to run any time, safe to interrupt. - New config: `cluster.gc_interval_hours: Option<u64>`. When set to a positive integer, the daemon spawns a periodic ticker that invokes GC in-process. Skips the first tick (nothing to reclaim on boot). Errors are logged and retried next tick. - Shutdown aborts the ticker cleanly. 254 tests pass (baseline unchanged). Pre-existing macOS failure untouched.	2026-07-12 06:23:46 -07:00
Omar Sobh	e70f5d74e0	Pilot findings: 5 real-world fixes from the 2026-07-12 deploy ... Bundles the profile→dir bug (PR #20 supersede) with four new fixes discovered by running clawstor against itself + across the fabric: * target_subdir_for: `dev`/`test` → `debug/`, `release`/`bench` → `release/`, custom passes through. Was silently skipping upload. * rustc release via gossip: daemon probes `rustc --version` at start, publishes the release string as `clawstor.rustc.release`. PeerView carries it; `cluster-peer-status` prints it in a new column and emits a warning line when the fleet has mixed versions. Would have surfaced the tank/architect 1.96.1 vs 1.95.0 drift instantly. * prewarm publishes fingerprint→blob ref downstream: `pin` now writes a companion tag `<name>.fingerprint` holding the fingerprint bytes. `prewarm` reads the companion, PutTag's it downstream, then PutRef(fp→blob) so a subsequent fingerprint-based `build` HITS. Without this, prewarm was almost useless for the runner path (build always missed even with matching source + rustc). * streaming byte counters: BlobPutStream + BlobGetStream now record the transferred bytes via `record_blob_{put,get}_bytes`. Metric used to stay at 0 no matter how much you moved. * capture determinism: replaced `tar::Builder::append_dir_all` (uses `read_dir`'s native order) with `append_dir_sorted` that walks the tree recursively and sorts by filename bytes at every level. Two byte-identical trees now produce byte-identical tars regardless of filesystem ordering. +3 tests: - target_subdir_matches_cargo_layout (from #20) - fingerprint_companion_tag_uses_dotted_suffix - capture_is_order_independent_of_filesystem_readdir (guard against the exact bug we saw in the field) 252 tests pass (+1 from Phase 5h's 251). Pre-existing macOS `du -sb` failure unchanged. Supersedes #20 (also included here). Ready for re-deploy to tank + architect for the retest run.	2026-07-12 05:34:36 -07:00
Omar Sobh	523b22f148	Phase 5j: Prometheus /metrics endpoint ... Adds a tiny axum-served HTTP endpoint that exposes the same CacheMetrics counters that back GetMetrics + gossip, in Prometheus text exposition format (v0.0.4). Enable per node by setting `cluster.prom_bind` in the daemon config. * metrics.rs: MetricsReply::to_prometheus() emits one HELP + TYPE + sample line per counter. started_unix is a gauge; everything else is a counter. Preallocates ~1 KiB so no reallocs mid-format. * prom.rs (new): PromServer::bind spins up axum on a TcpListener, serves GET /metrics, returns 404 elsewhere. Graceful shutdown via oneshot channel; abort() variant for the sync-drop path in ClusterServices. Snapshots on every scrape (no cache) — Relaxed atomic loads are cheap enough that even 1 Hz is sub-microsecond. * config.rs: new optional `cluster.prom_bind: SocketAddr` field. Default None means no server; typical value is 127.0.0.1:7702. * services.rs: wires PromServer into ClusterServices when both a router and prom_bind exist. Warns (doesn't fail) if prom_bind is set without RPC — nothing would ever change on a scrape. +8 tests: - metrics: to_prometheus emits every counter with correct type; zeros still produce valid exposition (fresh daemon scrape) - prom: content-type is text/plain; version=0.0.4; live updates between requests (no cache); unknown paths 404; shutdown stops serving - services: end-to-end scrape returns router-driven counters; prom_addr() is None when unconfigured (no accidentally-leaked port) Raw-TCP HTTP client in tests instead of pulling in reqwest — 30 lines of tokio::net + string split for GET / read-to-close is small enough to justify not adding a dep. 249 tests pass (+8 from Phase 5i). Pre-existing macOS `du -sb` failure unchanged.	2026-07-12 04:36:34 -07:00
Omar Sobh	29be728089	Phase 5i: publish cache metrics via gossip ... ClusterServices now periodically snapshots the router's CacheMetrics and republishes four raw counters — GetRef hits/misses and blob GET/PUT byte totals — through chitchat as clawstor.cache.* keys. Peers derive the hit rate locally via PeerView::cache_get_ref_hit_rate, eliminating a per-peer GetMetrics roundtrip for placement decisions. * gossip.rs: 4 new well-known keys, PeerView carries the counters + a saturating-add derived hit rate that returns None on 0/0 or when either counter is missing (guards against half-writes reading as 100% hits). * services.rs: router hoisted out of the TLS branch so a cache_metric_task can hold Arc<RpcRouter>. Publishes once at start (initial zeros so peers don't wait 60s for first read) then every CACHE_METRIC_INTERVAL. Ticker skipped when RPC isn't up — counters only fire inside dispatch. +3 tests: - gossip: two-node convergence with cache counters + hit-rate math - gossip: half-write / 0-0 / normal PeerView cases return correct rates - services: fresh cluster sees Some(0) for all four keys, then after in-process router counters + explicit set_cache_metrics the peer sees the updated values with hit rate 0.8 241 tests pass (+3 from Phase 5g). Pre-existing macOS `du -sb` failure in hot::tests unchanged.	2026-07-12 04:20:45 -07:00
Omar Sobh	b7904b59a5	Phase 5d: named tags + pin/unpin/list-tags CLI ... Human-readable pins on top of the raw 32-byte ref layer. Operators publish `clawverse:main:latest-cache` → BlobId once, then everything downstream (CI runners, dev laptops) references the tag instead of passing 64-char hex hashes around. ## Module: cluster/tags.rs (433 lines) TagStore for string-key → 32-byte-value: - open(root) — creates layout, safe on existing stores - put(key, value) / get(key) / delete(key) / contains(key) - list() — sorted by key - Atomic writes via tempfile + rename - Key length capped at MAX_TAG_KEY_BYTES (4 KiB); empty keys rejected On-disk record: `key_len:u16 (LE) || key_bytes || value:32bytes`. Filename is `blake3(key)` hex so arbitrary UTF-8 keys land at deterministic paths without shell escaping. TagEntry type (public, serde) for list results: `{ key, value_hex }`. Includes `decode_value() → Result<[u8;32]>`. ## RPC methods - PutTag (0x0f): payload = encoded record → STREAM_STATUS_OK / err - GetTag (0x10): payload = key bytes → 32-byte value / NotFound - DeleteTag (0x11): payload = key bytes → STREAM_STATUS_OK / NotFound - ListTags (0x12): payload = empty → JSON Vec<TagEntry> RpcRouter grows optional Arc<TagStore> via `.with_tag_store(store)`. ## Services + config ClusterServices auto-opens a TagStore at `<blob_store_root>/tags-db` alongside the ref store. `tag_store` field on ClusterServices, same enable-with-blob-store semantics. ## claw-cargo new subcommands - `claw-cargo pin --name clawverse:main:latest` Compute current fingerprint → look up its BlobId via GetRef → publish TagStore mapping. Errors cleanly if the fingerprint hasn't been built yet (nothing to point at). - `claw-cargo unpin --name clawverse:main:latest` Delete the tag. Prints "no such tag" if it wasn't set. - `claw-cargo list-tags` Print every tag with its 32-byte hex value. Total subcommand count now 7: build / prefetch / status / fingerprint / pin / unpin / list-tags. All share the layered config from Phase 5c. ## Client helpers - call_put_tag / call_get_tag / call_delete_tag / call_list_tags - All follow the same error-mapping conventions as prior client helpers (NotFound → Ok(None) or Ok(false), everything else → Err) ## Housekeeping rpc.rs was pushing past the 1300-line ceiling with the tag methods added. Client helpers moved to `cluster/rpc/client.rs` with a re-export (`pub use client::*;`) so external callers still write `cluster::rpc::call_*`. Result: - rpc.rs: 773 (was 1343) - rpc/client.rs: 593 (new) - rpc/tests.rs: 1235 - All under ceiling. ## Tests (33 new, all real filesystem / real QUIC — no mocks) TagStore (16 in cluster/tags.rs): - open_creates_layout - get_returns_none_for_missing (+ contains false) - put_and_get_round_trip - put_overwrites_prior_value - delete_returns_true_for_existing_and_false_for_missing - put_rejects_empty_key - put_rejects_oversize_key - list_returns_all_tags_sorted - list_is_empty_on_fresh_store - keys_with_slashes_and_colons_round_trip (real-world tag shape) - encode_and_decode_round_trip (raw wire format) - decode_rejects_short_record - decode_rejects_length_mismatch - decode_rejects_non_utf8_key - tag_entry_decode_value_round_trip - tag_entry_decode_value_rejects_bad_hex RPC dispatch (7 new): - phase_5d_method_byte_encoding - tag_rpcs_return_not_configured_without_store - put_tag_stores_and_get_tag_reads_back - get_tag_returns_not_found_for_missing - get_tag_rejects_empty_key - delete_tag_removes_and_returns_not_found_after - list_tags_returns_json_sorted End-to-end over real QUIC (1): - **end_to_end_pin_lookup_delete_over_real_quic** — publish tag → look up → list → delete → confirm gone. Full round trip through the wire layer including JSON deserialization of the list. Also 8 downstream tests continued passing after the client.rs split (no test moved, they were untouched). 223 tests pass. Pre-existing macOS-only failure unchanged. ## What this enables Operator flow: # Build once on the primary $ claw-cargo build → cache MISS → cargo build (50 min) → capture + upload → summary: fingerprint 4a3b…, blob 8c2f…, uploaded 3.2 GiB # Publish a friendly name $ claw-cargo pin --name clawverse:main:2026-07-12 pinned: clawverse:main:2026-07-12 fingerprint: 4a3b2c… blob: 8c2f1a… # Anyone else can now find it via list-tags $ claw-cargo list-tags clawverse:main:2026-07-12 8c2f1a… clawverse:main:latest 8c2f1a… # CI runner sees the same fingerprint in its workspace state, hits # the ref directly via GetRef — the tag is for operator visibility ## Follow-on - 5e: prefetch --pin <tag> — bypass fingerprint compute, download the tagged BlobId directly (useful when you want an old cache to test regression scenarios) - 5f: gitea webhook pre-fetch — daemon pre-warms cache for known fingerprints before CI runner starts - 3: full CRDT metadata layer (namespaces, versioned pointers, vector clocks) if the plain-tag model turns out to have real-world conflict scenarios	2026-07-12 00:07:33 -07:00
Omar Sobh	2d09b4687c	Phase 5b: KV refs + claw-cargo CLI (the killer feature, live) ... Ships the actual user-facing cargo build cache. Combined with Phase 5a (fingerprint + capture + restore) + the whole Phase 2 blob substrate, `claw-cargo build` now runs `cargo build` with a peer-cache lookup: hit → download+restore, miss → build+capture+upload. ## What ships ### cluster/refs.rs (243 lines) A dumb 32-byte-key → 32-byte-value directory-backed store. Used to map fingerprints → BlobIds. Layout mirrors BlobStore: <root>/ refs/<kk>/<key_hex>.ref — 32 raw bytes .tmp/ — atomic-rename staging Public API: RefStore::open / get / put / delete / contains. All writes atomic via tempfile + rename. Deliberately no versioning or CRDT semantics — that's Phase 3. Every real cargo-cache lookup is a single-key-single-value shape. ### New RPC methods - GetRef (0x0d): payload = 32-byte RefKey; reply = 32 bytes / NotFound - PutRef (0x0e): payload = 32-byte RefKey || 32-byte RefValue; reply = STREAM_STATUS_OK / error ### RpcRouter + services - RpcRouter grows optional Arc<RefStore> via `with_ref_store` - ClusterServices opens a RefStore alongside the BlobStore when `blob_store_root` is configured (co-located at `<blob_root>/refs-db`) - `blob_store_enabled()` / `ref_store_enabled()` introspection ### claw-cargo binary (319 lines) New bin target `claw-cargo` — thin CLI wrapping the whole stack: claw-cargo fingerprint --profile release --features "a,b" → prints the workspace fingerprint (no network) claw-cargo build \ --peer <name> --peer-addr <ip:port> --tls-dir <dir> \ --profile release --features "a,b" \ -- --workspace=x --frozen ... → 1. compute fingerprint 2. QUIC + mTLS connect to peer 3. GetRef(fingerprint) → BlobId? HIT: BlobStat → BlobGetStream → restore_target → cargo build MISS: cargo build → capture_target → BlobPutStream → PutRef 4. Print summary: fingerprint, hit/miss, bytes, cargo elapsed ## Live smoke test Ran claw-cargo fingerprint on this workspace with three profile/feature combos — got three distinct 32-byte fingerprints. Same profile+features on the same workspace state → same fingerprint (Phase 5a's guarantee carried through the CLI). ## Tests (14 new, all real — no mocks) Refs store (7): - open creates layout - get returns None for missing - put + get round-trips - put overwrites prior value - delete removes ref + reports (false on second delete) - distinct keys produce distinct on-disk files (bucket fan-out proof) - rejects_wrong_length_on_disk (corruption detection) RPC (7): - phase_5b_method_byte_encoding - get_ref_returns_not_found_for_missing - put_ref_stores_and_get_ref_reads_back - put_ref_rejects_wrong_length_payload - get_ref_rejects_wrong_length_payload - ref_rpcs_return_not_configured_without_store - end_to_end_put_ref_get_ref_over_real_quic — full 2-node QUIC + mTLS round trip proving PutRef/GetRef work at the wire level 188 tests pass. Pre-existing macOS-only failure unchanged. File sizes (all under 1300-line ceiling): - cluster/refs.rs: 243 - cluster/rpc.rs: 1169 - cluster/rpc/tests.rs: 1073 - cluster/services.rs: 565 - claw_cargo.rs: 319 ## Where this leaves us The distributed FS + cargo cache is functionally complete for the happy path: Node A builds clawverse for the first time → cargo build (50 min cold) → capture_target (a few seconds) → push to node B via BlobPutStream (network-bound) → PutRef(fingerprint → BlobId) Node B on the same workspace state runs `claw-cargo build …` → compute_fingerprint (ms) → GetRef → hit → BlobGetStream (network-bound) → restore_target (a few seconds) → cargo build → sees valid deps/.fingerprint, builds only workspace crates (~3 min instead of 50) Same workspace state on a third machine? Same fingerprint → same cache hit. That's the whole design. ## Follow-on - Phase 5c: pre-fetch on Gitea webhook so CI runners never wait - Phase 5d: metric ticker publishes cache hit rate into gossip so the placement engine can bias runner scheduling toward warm nodes - Phase 3: CRDT metadata for human-readable pins on top of raw 32-byte refs (`clawverse:main:latest-cache` → fingerprint hex) - Phase 6+: FUSE mount for the warm-tier git worktrees	2026-07-11 23:36:37 -07:00
Omar Sobh	f43b34ad15	Phase 2b: Blob RPC (Stat/Get/Put/LoadManifest) ... Wires the Phase 2 content-addressed blob store onto the network via four new methods on the existing RpcRouter. Combined with the mTLS + gossip stack from Phase 1c-1e, peers can now exchange content-addressed blobs over real QUIC. This is the substrate the fingerprint-keyed cargo cache (Phase 5) sits on directly. New methods: - BlobStat (0x03): payload = 32-byte BlobId, reply = JSON BlobStat - BlobGet (0x04): payload = 32-byte BlobId, reply = raw bytes - BlobPut (0x05): payload = raw bytes, reply = 32-byte BlobId - BlobLoadManifest (0x06): payload = 32-byte BlobId, reply = JSON manifest New error codes: - NotFound (0xf3) — the requested BlobId isn't in the local store - InvalidRequest (0xf4) — e.g. non-32-byte payload for a hash-keyed method - NotConfigured (0xf5) — Blob* called on a router without an attached store Wire-format bump: MAX_MESSAGE_BYTES 16 KiB → 16 MiB so a single 4 MiB chunk (plus JSON overhead) fits comfortably. Anything above 16 MiB needs the streaming variants coming in Phase 2c. Client helpers: - call_blob_stat / call_blob_get / call_blob_put / call_blob_load_manifest - All map ErrorCode::NotFound to Ok(None), other codes to Err. - call_blob_put verifies the peer-assigned BlobId matches local blake3 hash of the payload — corruption or protocol drift surfaces immediately instead of silently accepting a mismatched receipt. Router changes: - RpcRouter grows an Option<Arc<BlobStore>> via with_blob_store(store). - handle() split into handle_outcome() → HandlerOutcome enum {Reply(bytes) | Error(ErrorCode)} for cleaner control flow across the growing method set. Services / daemon: - ClusterServices::start gains blob_store_root: Option<PathBuf>. - ClusterConfig gains blob_store_root: Option<PathBuf>. - daemon.rs reads it from cluster_cfg + passes through. - New ClusterServices::blob_store_enabled() introspection. Tests (11 new, all real — no mocks): Router (7 new): - method_round_trips_byte_encoding — updated for 6 methods - error_code_describe_covers_all_variants — updated for 6 codes - decode_error_covers_all_known_codes - blob_rpcs_return_not_configured_without_store — all four Blob* methods return NotConfigured when the router lacks a store - blob_stat_returns_not_found_for_missing - blob_stat_returns_json_for_existing - blob_stat_returns_invalid_request_for_bad_length - blob_get_returns_content_bytes - blob_put_stores_bytes_and_returns_hash — verifies BlobId matches independent local hash - blob_load_manifest_returns_json_for_existing (2-chunk case) - blob_load_manifest_returns_not_found_for_missing End-to-end over real QUIC (2 new): - end_to_end_blob_put_stat_get_over_real_quic — full 4-method loop (Put → Stat → Get → LoadManifest) + NotFound path - end_to_end_multi_chunk_blob_over_real_quic — 6 MiB blob → 2 chunks, proves MAX_MESSAGE_BYTES bump took effect Services (1 new): - services_with_blob_store_serves_blob_rpc_end_to_end — cut CA, sign leaves, config includes blob_store_root, start ClusterServices, dial from B over persisted mTLS, put + get through the router, then independently verify bytes landed on A's on-disk store Also fixed a parallel-test port collision: services `next_port()` now increments by 2 so `port + 1` (the RPC bind) is reserved alongside `port` (the gossip bind). 128 tests pass. Pre-existing macOS-only failure unchanged. File sizes (all under 1300-line ceiling): - cluster/rpc.rs: 1006 - cluster/services.rs: 535 - cluster/blob.rs: 802 - config.rs: 511 - daemon.rs: 265 Follow-on: - 2c: streaming variants (AsyncRead/AsyncWrite) so a many-GB blob transfers without holding it in memory - 2d: chunk-level RPC (BlobPutChunk / BlobGetChunk) so a receiver can request only chunks it's missing after LoadManifest - Phase 5 (the killer feature) can now build on Phase 2b directly — fingerprint the target dir, PutBlob the compressed tarball, and next node calls GetBlob keyed by the same fingerprint hash.	2026-07-11 22:42:17 -07:00
Omar Sobh	2ae079fbd4	Phase 1e: daemon-integrated cluster services + PeerStatus RPC ... Ties Phase 1a-1d together into a live subsystem the daemon actually runs. When `[cluster]` is present in config, `claw-store daemon` now: 1. Starts chitchat gossip via ClusterServices::start. 2. Publishes hot.max_gb immediately and re-measures the hot dir every 30s, updating `clawstor.hot.used`. 3. If `[cluster.tls]` is also configured, loads NodeIdentity from PEM files and binds a QUIC RPC server that accepts + dispatches incoming Ping / PeerStatus requests via RpcRouter. New modules: - cluster/rpc.rs (510 lines): - Method enum (Ping=0x01, PeerStatus=0x02) - ErrorCode enum (EmptyRequest / UnknownMethod / HandlerFailure) - PeerStatusReply { local_name, local_zone, peers: Vec<PeerView> } - RpcRouter — dispatch state (holds Arc<ClusterGossip> + local name/zone) - rpc_call / call_ping / call_peer_status — client helpers - serve_connection — server accept-bidi loop - Wire format: `method:u8 || payload:bytes` → `reply:bytes` or single-byte ErrorCode - cluster/services.rs (418 lines): - ClusterServices { gossip, accept_task, metric_task } - start(cluster_cfg, name, hot_dir, hot_max_bytes) → bootstraps everything above - shutdown() aborts background tasks cleanly - Recursive dir-walker (spawn_blocking) for hot-tier metric PeerView now derives Serialize/Deserialize so it round-trips through JSON over the RPC. daemon.rs integration (~35 lines added): - Bootstraps ClusterServices before entering the select loop - Held for daemon lifetime - Shutdown on SIGTERM - Gossip-less config still runs standalone (backwards compat) CLI: `cluster-peer-status --peer <name> --rpc-addr <addr> --tls-dir <dir>` Loads a persistent identity, dials the peer, calls PeerStatus, prints the peer's local view as a table. Tests (15 new, all real — no mocks, real UDP + TLS + JSON round trip): RPC (9): - method_round_trips_byte_encoding - dispatch_returns_pong_for_ping - dispatch_returns_json_for_peer_status - dispatch_returns_empty_request_error - dispatch_returns_unknown_method_error - rpc_call_rejects_oversize_payload (with real quinn connection) - end_to_end_ping_and_peer_status_over_real_quic — full 2-node quinn with mTLS + both RPCs - peer_status_reflects_peer_gossip_state — 2 gossip instances converge, RPC caller from a third identity sees the converged view - error_code_describe_covers_all_variants Services (6): - dir_bytes_sync_returns_zero_for_missing_path - dir_bytes_sync_sums_recursive_file_sizes (3-level nesting) - services_start_without_tls_leaves_rpc_disabled - services_start_with_tls_serves_rpc_end_to_end — full stack: cut CA on disk, sign leaves, start ClusterServices for A, dial from B via persisted mTLS, run ping + PeerStatus over the wire - services_publish_hot_used_metric_periodically - services_gossip_sees_peer_after_convergence 95 tests pass. Pre-existing macOS-only failure unchanged. File sizes (all under 1300-line ceiling): - cluster/rpc.rs: 510 - cluster/services.rs: 418 - cluster/gossip.rs: 579 - cluster/transport.rs: 916 - daemon.rs: 262 - main.rs: 749 Phase 1 done end-to-end. `claw-store daemon` now boots a real distributed cluster stack when config asks for one; peers can call each other's PeerStatus RPC and see live membership. Next: Phase 2 (content-addressed blob store) can hook new RPC methods into the same RpcRouter with a one-line dispatch arm.	2026-07-11 22:15:41 -07:00