Commit Graph
1 Commits
Author SHA1 Message Date
Omar Sobh 58c5bc341b GetRef: transparent ref-forwarding on local miss
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 10s
Cross-runner cache silos (tank + architect measured on 2026-07-13):
same fingerprint, same rustc, but each runner's daemon only knows
about the refs its own runner uploaded. Every runner that lands on
a peer that isn't tank re-uploads a duplicate blob.

Fix: on `GetRef` miss the daemon fans out to alive gossip peers
via a strict-local `GetRefLocal` variant, and the FIRST peer that
has the ref triggers a transparent pull — chunks + manifest into
the local blob store, then `PutRef` locally — before returning the
value to the caller. Subsequent lookups are pure-local hits.

* `Method::GetRefLocal = 0x14` — new wire method, identical shape
  to GetRef but the peer MUST NOT recurse. Loop prevention: our
  forwarding only calls `GetRefLocal` on peers, so chain depth is
  always 1.
* `RpcRouter::with_outbound_client(Arc<QuicClient>)` — dependency
  injection point for the forwarding dial path. `None` disables
  forwarding entirely (GetRef becomes GetRefLocal-equivalent).
* `RpcRouter::forward_get_ref(key)` — concurrent peer probes via
  `JoinSet`, 3s timeout per dial, first successful pull wins,
  remaining tasks aborted.
* `pull_blob_locally` — walks manifest, fetches only chunks the
  local store lacks (`has_chunk`), commits via
  `put_manifest_verified`. Bounded memory: one 4 MiB chunk at a
  time.
* `ClusterServices::start` loads NodeIdentity twice — server takes
  ownership; outbound client gets its own copy for TLS presentation
  on peer dials. Wires the outbound client into the router when
  TLS material is available.
* `call_get_ref_local(conn, key)` client helper (used by daemon
  forwarding + available to any RPC consumer that wants the
  no-recursion semantics).

+3 tests in `rpc/tests_forwarding.rs`:
- Local hit works without forwarding; local miss with no peers
  returns None. Guards the base cases.
- GetRefLocal never forwards even when outbound is configured (no
  peers reachable → miss returns None immediately, no attempted
  fan-out).
- Method byte 0x14 encoding is stable across releases.

Full end-to-end forwarding is exercised in the pilot deploy: two
daemons on the fleet-CA, tank populates a ref, architect's runner
GetRef → tank forwards → architect pulls → HIT locally next time.

264 tests pass (baseline +3). Pre-existing macOS failure unchanged.
2026-07-13 06:50:53 -07:00