Phase 5h: streaming chunk-level prewarm

Bounded-memory cross-peer prewarm: instead of buffering the whole
blob in RAM (previous 5f path), iterate the upstream manifest chunk
by chunk, ask downstream `HasChunk`, stream missing chunks one at a
time. Memory ceiling is 1 chunk (4 MiB) regardless of blob size — a
5 GiB target dir no longer needs 5 GiB of mediator RAM.

* rpc/client.rs: new `prewarm_missing_chunks_between(upstream,
  downstream, blob_id)` helper. Returns `(uploaded, total)` — the
  difference is the dedup save. Retries once if the downstream
  `PutManifest` reports missing chunks after our push (guards a
  narrow eviction race); a second failure surfaces as `Err`.
* claw_cargo.rs: `prewarm` now streams by default; new `--buffered`
  flag for the old whole-blob path (kept for diagnostic
  comparability during rollout). Human-readable output shows the
  mode + dedup count.

+2 tests:
- cold downstream: 3-chunk payload (with a partial tail chunk) is
  copied exactly, reassembly is byte-equal to source
- partial dedup: pre-seed 1 of 3 chunks on downstream → uploaded=2;
  rerun is a full no-op (uploaded=0), proving idempotence

251 tests pass (+2 from Phase 5j). Pre-existing macOS failure
unchanged.

Follow-ons: (1) parallel chunk transfer (uploaded chunks in fan-out)
would speed multi-GB prewarms further; (2) exposing an accurate
transferred-bytes counter needs router-side accounting instead of
the current chunk-count × CHUNK_SIZE approximation.
This commit is contained in:
Omar Sobh
2026-07-12 04:48:55 -07:00
parent 523b22f148
commit 6fb16286dd
3 changed files with 362 additions and 48 deletions
+76
View File
@@ -600,3 +600,79 @@ pub async fn push_blob_missing_chunks(
}
Ok((uploaded, total))
}
/// Phase 5h: chunk-level cross-peer prewarm with bounded memory.
///
/// Unlike [`push_blob_missing_chunks`] this doesn't need a local
/// `BlobStore` — it forwards chunk data directly from `upstream` to
/// `downstream` one chunk at a time. Memory ceiling is one chunk
/// (4 MiB by default), independent of blob size, so multi-GB blobs
/// don't blow up RAM on the mediator.
///
/// Flow:
/// 1. Load the manifest from `upstream`.
/// 2. For each chunk, ask `downstream` if it already has it.
/// 3. For missing chunks: `GetChunk` upstream → `PutChunk` downstream.
/// Bytes are held only for the duration of the pair of calls.
/// 4. Commit the manifest downstream via `PutManifest`. If the peer
/// still reports missing chunks (concurrent eviction, storage
/// failure), we retry each once; a second failure surfaces as
/// `Err`.
///
/// Returns `(uploaded_chunks, total_chunks)` — the difference is the
/// dedup save. When `downstream` is already fully warm the returned
/// `uploaded` is 0 and no chunk bytes crossed the wire.
///
/// Note: this streams chunks sequentially. Fan-out (N concurrent
/// chunk transfers) would speed multi-GB prewarms but complicates
/// error handling; a future revision can layer parallelism on top
/// without changing the memory story per in-flight chunk.
pub async fn prewarm_missing_chunks_between(
upstream: &Connection,
downstream: &Connection,
id: &BlobId,
) -> Result<(usize, usize)> {
let manifest = call_blob_load_manifest(upstream, id)
.await?
.with_context(|| format!("upstream missing blob {}", id.to_hex()))?;
let total = manifest.chunks.len();
let mut uploaded = 0usize;
for hash in &manifest.chunks {
if call_has_chunk(downstream, hash).await? {
continue;
}
let bytes = call_get_chunk(upstream, hash)
.await?
.with_context(|| {
format!(
"upstream manifest referenced chunk {} but GetChunk returned NotFound",
hash.to_hex()
)
})?;
call_put_chunk(downstream, hash, &bytes).await?;
uploaded += 1;
// `bytes` is dropped here — memory ceiling is one chunk at a
// time. Reassigned on the next iteration.
}
let still_missing = call_put_manifest(downstream, &manifest).await?;
if !still_missing.is_empty() {
// One retry pass: fetch + push each chunk the peer still lacks.
// Guards against a narrow race where the peer evicted a chunk
// between our HasChunk and its PutManifest verification pass.
for hash in &still_missing {
let bytes = call_get_chunk(upstream, hash).await?.with_context(|| {
format!("retry fetch of chunk {} failed", hash.to_hex())
})?;
call_put_chunk(downstream, hash, &bytes).await?;
uploaded += 1;
}
let final_missing = call_put_manifest(downstream, &manifest).await?;
if !final_missing.is_empty() {
bail!(
"downstream still missing {} chunks after retry; storage may be failing",
final_missing.len()
);
}
}
Ok((uploaded, total))
}