Phase 5h: streaming chunk-level prewarm
Bounded-memory cross-peer prewarm: instead of buffering the whole blob in RAM (previous 5f path), iterate the upstream manifest chunk by chunk, ask downstream `HasChunk`, stream missing chunks one at a time. Memory ceiling is 1 chunk (4 MiB) regardless of blob size — a 5 GiB target dir no longer needs 5 GiB of mediator RAM. * rpc/client.rs: new `prewarm_missing_chunks_between(upstream, downstream, blob_id)` helper. Returns `(uploaded, total)` — the difference is the dedup save. Retries once if the downstream `PutManifest` reports missing chunks after our push (guards a narrow eviction race); a second failure surfaces as `Err`. * claw_cargo.rs: `prewarm` now streams by default; new `--buffered` flag for the old whole-blob path (kept for diagnostic comparability during rollout). Human-readable output shows the mode + dedup count. +2 tests: - cold downstream: 3-chunk payload (with a partial tail chunk) is copied exactly, reassembly is byte-equal to source - partial dedup: pre-seed 1 of 3 chunks on downstream → uploaded=2; rerun is a full no-op (uploaded=0), proving idempotence 251 tests pass (+2 from Phase 5j). Pre-existing macOS failure unchanged. Follow-ons: (1) parallel chunk transfer (uploaded chunks in fan-out) would speed multi-GB prewarms further; (2) exposing an accurate transferred-bytes counter needs router-side accounting instead of the current chunk-count × CHUNK_SIZE approximation.
This commit is contained in:
@@ -600,3 +600,79 @@ pub async fn push_blob_missing_chunks(
|
||||
}
|
||||
Ok((uploaded, total))
|
||||
}
|
||||
|
||||
/// Phase 5h: chunk-level cross-peer prewarm with bounded memory.
|
||||
///
|
||||
/// Unlike [`push_blob_missing_chunks`] this doesn't need a local
|
||||
/// `BlobStore` — it forwards chunk data directly from `upstream` to
|
||||
/// `downstream` one chunk at a time. Memory ceiling is one chunk
|
||||
/// (4 MiB by default), independent of blob size, so multi-GB blobs
|
||||
/// don't blow up RAM on the mediator.
|
||||
///
|
||||
/// Flow:
|
||||
/// 1. Load the manifest from `upstream`.
|
||||
/// 2. For each chunk, ask `downstream` if it already has it.
|
||||
/// 3. For missing chunks: `GetChunk` upstream → `PutChunk` downstream.
|
||||
/// Bytes are held only for the duration of the pair of calls.
|
||||
/// 4. Commit the manifest downstream via `PutManifest`. If the peer
|
||||
/// still reports missing chunks (concurrent eviction, storage
|
||||
/// failure), we retry each once; a second failure surfaces as
|
||||
/// `Err`.
|
||||
///
|
||||
/// Returns `(uploaded_chunks, total_chunks)` — the difference is the
|
||||
/// dedup save. When `downstream` is already fully warm the returned
|
||||
/// `uploaded` is 0 and no chunk bytes crossed the wire.
|
||||
///
|
||||
/// Note: this streams chunks sequentially. Fan-out (N concurrent
|
||||
/// chunk transfers) would speed multi-GB prewarms but complicates
|
||||
/// error handling; a future revision can layer parallelism on top
|
||||
/// without changing the memory story per in-flight chunk.
|
||||
pub async fn prewarm_missing_chunks_between(
|
||||
upstream: &Connection,
|
||||
downstream: &Connection,
|
||||
id: &BlobId,
|
||||
) -> Result<(usize, usize)> {
|
||||
let manifest = call_blob_load_manifest(upstream, id)
|
||||
.await?
|
||||
.with_context(|| format!("upstream missing blob {}", id.to_hex()))?;
|
||||
let total = manifest.chunks.len();
|
||||
let mut uploaded = 0usize;
|
||||
for hash in &manifest.chunks {
|
||||
if call_has_chunk(downstream, hash).await? {
|
||||
continue;
|
||||
}
|
||||
let bytes = call_get_chunk(upstream, hash)
|
||||
.await?
|
||||
.with_context(|| {
|
||||
format!(
|
||||
"upstream manifest referenced chunk {} but GetChunk returned NotFound",
|
||||
hash.to_hex()
|
||||
)
|
||||
})?;
|
||||
call_put_chunk(downstream, hash, &bytes).await?;
|
||||
uploaded += 1;
|
||||
// `bytes` is dropped here — memory ceiling is one chunk at a
|
||||
// time. Reassigned on the next iteration.
|
||||
}
|
||||
let still_missing = call_put_manifest(downstream, &manifest).await?;
|
||||
if !still_missing.is_empty() {
|
||||
// One retry pass: fetch + push each chunk the peer still lacks.
|
||||
// Guards against a narrow race where the peer evicted a chunk
|
||||
// between our HasChunk and its PutManifest verification pass.
|
||||
for hash in &still_missing {
|
||||
let bytes = call_get_chunk(upstream, hash).await?.with_context(|| {
|
||||
format!("retry fetch of chunk {} failed", hash.to_hex())
|
||||
})?;
|
||||
call_put_chunk(downstream, hash, &bytes).await?;
|
||||
uploaded += 1;
|
||||
}
|
||||
let final_missing = call_put_manifest(downstream, &manifest).await?;
|
||||
if !final_missing.is_empty() {
|
||||
bail!(
|
||||
"downstream still missing {} chunks after retry; storage may be failing",
|
||||
final_missing.len()
|
||||
);
|
||||
}
|
||||
}
|
||||
Ok((uploaded, total))
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user