Omar Sobh
58c5bc341b
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 10s
Cross-runner cache silos (tank + architect measured on 2026-07-13): same fingerprint, same rustc, but each runner's daemon only knows about the refs its own runner uploaded. Every runner that lands on a peer that isn't tank re-uploads a duplicate blob. Fix: on `GetRef` miss the daemon fans out to alive gossip peers via a strict-local `GetRefLocal` variant, and the FIRST peer that has the ref triggers a transparent pull — chunks + manifest into the local blob store, then `PutRef` locally — before returning the value to the caller. Subsequent lookups are pure-local hits. * `Method::GetRefLocal = 0x14` — new wire method, identical shape to GetRef but the peer MUST NOT recurse. Loop prevention: our forwarding only calls `GetRefLocal` on peers, so chain depth is always 1. * `RpcRouter::with_outbound_client(Arc<QuicClient>)` — dependency injection point for the forwarding dial path. `None` disables forwarding entirely (GetRef becomes GetRefLocal-equivalent). * `RpcRouter::forward_get_ref(key)` — concurrent peer probes via `JoinSet`, 3s timeout per dial, first successful pull wins, remaining tasks aborted. * `pull_blob_locally` — walks manifest, fetches only chunks the local store lacks (`has_chunk`), commits via `put_manifest_verified`. Bounded memory: one 4 MiB chunk at a time. * `ClusterServices::start` loads NodeIdentity twice — server takes ownership; outbound client gets its own copy for TLS presentation on peer dials. Wires the outbound client into the router when TLS material is available. * `call_get_ref_local(conn, key)` client helper (used by daemon forwarding + available to any RPC consumer that wants the no-recursion semantics). +3 tests in `rpc/tests_forwarding.rs`: - Local hit works without forwarding; local miss with no peers returns None. Guards the base cases. - GetRefLocal never forwards even when outbound is configured (no peers reachable → miss returns None immediately, no attempted fan-out). - Method byte 0x14 encoding is stable across releases. Full end-to-end forwarding is exercised in the pilot deploy: two daemons on the fleet-CA, tank populates a ref, architect's runner GetRef → tank forwards → architect pulls → HIT locally next time. 264 tests pass (baseline +3). Pre-existing macOS failure unchanged.