GetRef: transparent ref-forwarding on local miss
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 10s
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 10s
Cross-runner cache silos (tank + architect measured on 2026-07-13): same fingerprint, same rustc, but each runner's daemon only knows about the refs its own runner uploaded. Every runner that lands on a peer that isn't tank re-uploads a duplicate blob. Fix: on `GetRef` miss the daemon fans out to alive gossip peers via a strict-local `GetRefLocal` variant, and the FIRST peer that has the ref triggers a transparent pull — chunks + manifest into the local blob store, then `PutRef` locally — before returning the value to the caller. Subsequent lookups are pure-local hits. * `Method::GetRefLocal = 0x14` — new wire method, identical shape to GetRef but the peer MUST NOT recurse. Loop prevention: our forwarding only calls `GetRefLocal` on peers, so chain depth is always 1. * `RpcRouter::with_outbound_client(Arc<QuicClient>)` — dependency injection point for the forwarding dial path. `None` disables forwarding entirely (GetRef becomes GetRefLocal-equivalent). * `RpcRouter::forward_get_ref(key)` — concurrent peer probes via `JoinSet`, 3s timeout per dial, first successful pull wins, remaining tasks aborted. * `pull_blob_locally` — walks manifest, fetches only chunks the local store lacks (`has_chunk`), commits via `put_manifest_verified`. Bounded memory: one 4 MiB chunk at a time. * `ClusterServices::start` loads NodeIdentity twice — server takes ownership; outbound client gets its own copy for TLS presentation on peer dials. Wires the outbound client into the router when TLS material is available. * `call_get_ref_local(conn, key)` client helper (used by daemon forwarding + available to any RPC consumer that wants the no-recursion semantics). +3 tests in `rpc/tests_forwarding.rs`: - Local hit works without forwarding; local miss with no peers returns None. Guards the base cases. - GetRefLocal never forwards even when outbound is configured (no peers reachable → miss returns None immediately, no attempted fan-out). - Method byte 0x14 encoding is stable across releases. Full end-to-end forwarding is exercised in the pilot deploy: two daemons on the fleet-CA, tank populates a ref, architect's runner GetRef → tank forwards → architect pulls → HIT locally next time. 264 tests pass (baseline +3). Pre-existing macOS failure unchanged.
This commit is contained in:
@@ -413,7 +413,25 @@ pub async fn call_get_ref(
|
||||
conn: &Connection,
|
||||
key: &RefKey,
|
||||
) -> Result<Option<RefValue>> {
|
||||
let reply = rpc_call(conn, Method::GetRef, key).await?;
|
||||
call_get_ref_inner(conn, key, Method::GetRef).await
|
||||
}
|
||||
|
||||
/// Ref-forwarding (2026-07-13): strict local-only variant. The peer
|
||||
/// MUST NOT recurse to its own peers; used by daemons doing ref
|
||||
/// forwarding to prevent loops.
|
||||
pub async fn call_get_ref_local(
|
||||
conn: &Connection,
|
||||
key: &RefKey,
|
||||
) -> Result<Option<RefValue>> {
|
||||
call_get_ref_inner(conn, key, Method::GetRefLocal).await
|
||||
}
|
||||
|
||||
async fn call_get_ref_inner(
|
||||
conn: &Connection,
|
||||
key: &RefKey,
|
||||
method: Method,
|
||||
) -> Result<Option<RefValue>> {
|
||||
let reply = rpc_call(conn, method, key).await?;
|
||||
if reply.len() == 1 {
|
||||
match decode_error(reply[0]) {
|
||||
Some(ErrorCode::NotFound) => return Ok(None),
|
||||
|
||||
Reference in New Issue
Block a user