Phase 2d: chunk-level RPC (HasChunk / PutChunk / GetChunk / PutManifest)
Unlocks partial-sync replication — a peer that already has some
chunks of a blob (typical when two nodes share overlapping cargo
build caches) only receives the chunks it's missing.
## New methods
| Byte | Method | Payload | Reply |
|---|---|---|---|
| 0x09 | HasChunk | 32-byte ChunkHash | STREAM_STATUS_OK / NotFound |
| 0x0a | PutChunk | ChunkHash \|\| bytes | STREAM_STATUS_OK / error |
| 0x0b | GetChunk | ChunkHash | STREAM_STATUS_OK \|\| bytes / NotFound |
| 0x0c | PutManifest | JSON BlobManifest | JSON PutManifestReply |
`PutManifestReply { blob_id, missing: Vec<ChunkHash> }`: empty
`missing` means the manifest was written; non-empty tells the
client which chunks to upload before retrying.
Server verifies bytes hash to claimed hash on PutChunk; a
mismatch surfaces as InvalidRequest and the store is untouched.
## BlobStore additions
- `has_chunk(&ChunkHash) → bool`
- `read_chunk(&ChunkHash) → Option<Vec<u8>>` — verifies hash on read
- `put_chunk(&ChunkHash, bytes) → Result<()>` — verifies bytes-vs-hash
- `put_manifest_verified(&manifest) → Result<Vec<ChunkHash>>` —
returns the list of chunks missing on disk (empty on success)
- `chunk_path` promoted to `pub` for advanced callers
## Client helpers
- `call_has_chunk` / `call_put_chunk` / `call_get_chunk` / `call_put_manifest`
- `push_blob_missing_chunks(conn, local_store, blob_id) →
Result<(uploaded, total)>` — high-level partial-sync helper
`push_blob_missing_chunks` loads the local manifest, calls HasChunk
for each chunk, uploads only the missing ones via PutChunk, then
commits via PutManifest. On a fully-overlapping cache the uploaded
count is 0 and only the ~small manifest crosses the wire.
## Tests (17 new, all real filesystem + real QUIC — no mocks)
Blob store (6):
- has_chunk_is_false_before_put_and_true_after
- read_chunk_returns_bytes_and_none_when_missing
- put_chunk_rejects_hash_mismatch (nothing written)
- read_chunk_detects_corruption (bit-flip → mismatch error)
- put_manifest_verified_reports_missing_chunks
- put_manifest_verified_writes_when_all_chunks_present
Router dispatch (7):
- phase_2d_method_byte_encoding
- method_reports_streaming_variants — extended for 4 new methods
- has_chunk_returns_ok_for_present_and_not_found_for_missing
- put_chunk_stores_and_returns_status_ok
- put_chunk_rejects_hash_mismatch_over_wire
- get_chunk_returns_content_prefixed_with_status_ok
- get_chunk_returns_not_found_for_missing
- put_manifest_reports_missing_chunks_when_incomplete
- put_manifest_writes_when_chunks_present
- chunk_rpcs_return_not_configured_without_store
End-to-end (2):
- **end_to_end_push_blob_missing_chunks_replicates_only_needed_bytes**:
Peer A pre-seeded with chunk 0 of a 2-chunk (8 MiB) blob;
`push_blob_missing_chunks` reports `(uploaded=1, total=2)`,
only chunk 1 crosses the wire, A's store then contains the
complete blob and `get_bytes` returns byte-equal content.
- **call_get_chunk_verifies_returned_hash**: real 2-node fetch,
client hashes received bytes and compares to requested hash.
156 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/rpc.rs: 1053
- cluster/rpc/tests.rs: 940
- cluster/blob.rs: 1186
## Where this fits
With Phase 2c whole-blob streaming + Phase 2d partial-chunk sync,
the storage substrate is now genuinely bandwidth-efficient in the
distributed setting:
- First-ever push of a blob: `push_blob_missing_chunks` uploads
everything (all chunks missing).
- Second push of a similar blob (95% chunk overlap with prior
contents): only the 5% new chunks cross the wire, plus a tiny
manifest.
- Whole-blob download: BlobGetStream, bounded by network bandwidth.
## Follow-on
- Phase 3: CRDT metadata for human-readable namespaces on top of
content hashes.
- Phase 5: the killer feature. Fingerprint cargo target dir → tar
→ hash → PutBlobStream (or push_blob_missing_chunks if a similar
build already lives on the peer). Same fingerprint on the next
node → BlobGetStream. This is the whole cargo-cache design in
one line and it now sits on a substrate that handles all the
hard cases (dedup, verification, resumability, partial sync).
This commit is contained in:
@@ -513,6 +513,252 @@ fn method_reports_streaming_variants() {
|
||||
assert!(!Method::BlobLoadManifest.is_streaming());
|
||||
assert!(Method::BlobPutStream.is_streaming());
|
||||
assert!(Method::BlobGetStream.is_streaming());
|
||||
assert!(!Method::HasChunk.is_streaming());
|
||||
assert!(!Method::PutChunk.is_streaming());
|
||||
assert!(!Method::GetChunk.is_streaming());
|
||||
assert!(!Method::PutManifest.is_streaming());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn phase_2d_method_byte_encoding() {
|
||||
assert_eq!(Method::HasChunk.as_byte(), 0x09);
|
||||
assert_eq!(Method::PutChunk.as_byte(), 0x0a);
|
||||
assert_eq!(Method::GetChunk.as_byte(), 0x0b);
|
||||
assert_eq!(Method::PutManifest.as_byte(), 0x0c);
|
||||
assert_eq!(Method::from_byte(0x09), Some(Method::HasChunk));
|
||||
assert_eq!(Method::from_byte(0x0a), Some(Method::PutChunk));
|
||||
assert_eq!(Method::from_byte(0x0b), Some(Method::GetChunk));
|
||||
assert_eq!(Method::from_byte(0x0c), Some(Method::PutManifest));
|
||||
}
|
||||
|
||||
// ── Phase 2d: chunk-level RPC ────────────────────────────────────────
|
||||
|
||||
#[tokio::test]
|
||||
async fn has_chunk_returns_ok_for_present_and_not_found_for_missing() {
|
||||
let (_tmp, router) = router_with_blobs("solo", next_port()).await;
|
||||
let bytes = b"chunky content";
|
||||
let hash = ChunkHash::from_bytes(blake3::hash(bytes).into());
|
||||
router
|
||||
.blob_store()
|
||||
.unwrap()
|
||||
.put_chunk(&hash, bytes)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let mut req = vec![Method::HasChunk.as_byte()];
|
||||
req.extend_from_slice(hash.as_bytes());
|
||||
assert_eq!(dispatch(&router, &req).await, vec![STREAM_STATUS_OK]);
|
||||
|
||||
let mut req = vec![Method::HasChunk.as_byte()];
|
||||
req.extend_from_slice(&[0u8; 32]);
|
||||
assert_eq!(dispatch(&router, &req).await, vec![ErrorCode::NotFound.as_byte()]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn put_chunk_stores_and_returns_status_ok() {
|
||||
let (_tmp, router) = router_with_blobs("solo", next_port()).await;
|
||||
let bytes = b"payload for put_chunk";
|
||||
let hash = ChunkHash::from_bytes(blake3::hash(bytes).into());
|
||||
let mut req = vec![Method::PutChunk.as_byte()];
|
||||
req.extend_from_slice(hash.as_bytes());
|
||||
req.extend_from_slice(bytes);
|
||||
assert_eq!(dispatch(&router, &req).await, vec![STREAM_STATUS_OK]);
|
||||
assert!(router.blob_store().unwrap().has_chunk(&hash).await.unwrap());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn put_chunk_rejects_hash_mismatch_over_wire() {
|
||||
let (_tmp, router) = router_with_blobs("solo", next_port()).await;
|
||||
let bytes = b"real";
|
||||
let bogus = ChunkHash::from_bytes([0xffu8; 32]);
|
||||
let mut req = vec![Method::PutChunk.as_byte()];
|
||||
req.extend_from_slice(bogus.as_bytes());
|
||||
req.extend_from_slice(bytes);
|
||||
let reply = dispatch(&router, &req).await;
|
||||
assert_eq!(reply, vec![ErrorCode::InvalidRequest.as_byte()]);
|
||||
assert!(!router.blob_store().unwrap().has_chunk(&bogus).await.unwrap());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn get_chunk_returns_content_prefixed_with_status_ok() {
|
||||
let (_tmp, router) = router_with_blobs("solo", next_port()).await;
|
||||
let bytes = b"fetch me by hash";
|
||||
let hash = ChunkHash::from_bytes(blake3::hash(bytes).into());
|
||||
router
|
||||
.blob_store()
|
||||
.unwrap()
|
||||
.put_chunk(&hash, bytes)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let mut req = vec![Method::GetChunk.as_byte()];
|
||||
req.extend_from_slice(hash.as_bytes());
|
||||
let reply = dispatch(&router, &req).await;
|
||||
assert_eq!(reply[0], STREAM_STATUS_OK);
|
||||
assert_eq!(&reply[1..], bytes);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn get_chunk_returns_not_found_for_missing() {
|
||||
let (_tmp, router) = router_with_blobs("solo", next_port()).await;
|
||||
let missing = ChunkHash::from_bytes([0u8; 32]);
|
||||
let mut req = vec![Method::GetChunk.as_byte()];
|
||||
req.extend_from_slice(missing.as_bytes());
|
||||
let reply = dispatch(&router, &req).await;
|
||||
assert_eq!(reply, vec![ErrorCode::NotFound.as_byte()]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn put_manifest_reports_missing_chunks_when_incomplete() {
|
||||
let (_tmp, router) = router_with_blobs("solo", next_port()).await;
|
||||
let phantom = ChunkHash::from_bytes([0xaau8; 32]);
|
||||
let manifest = BlobManifest {
|
||||
blob_id: BlobId::from_bytes([0x11u8; 32]),
|
||||
total_size: 42,
|
||||
chunks: vec![phantom],
|
||||
};
|
||||
let mut req = vec![Method::PutManifest.as_byte()];
|
||||
req.extend_from_slice(&serde_json::to_vec(&manifest).unwrap());
|
||||
let reply = dispatch(&router, &req).await;
|
||||
let decoded: PutManifestReply = serde_json::from_slice(&reply).unwrap();
|
||||
assert_eq!(decoded.blob_id, manifest.blob_id);
|
||||
assert_eq!(decoded.missing, vec![phantom]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn put_manifest_writes_when_chunks_present() {
|
||||
let (_tmp, router) = router_with_blobs("solo", next_port()).await;
|
||||
let store = router.blob_store().unwrap().clone();
|
||||
let a = b"first chunk";
|
||||
let b = b"second chunk";
|
||||
let hash_a = ChunkHash::from_bytes(blake3::hash(a).into());
|
||||
let hash_b = ChunkHash::from_bytes(blake3::hash(b).into());
|
||||
store.put_chunk(&hash_a, a).await.unwrap();
|
||||
store.put_chunk(&hash_b, b).await.unwrap();
|
||||
|
||||
let manifest = BlobManifest {
|
||||
blob_id: BlobId::from_bytes([0x22u8; 32]),
|
||||
total_size: (a.len() + b.len()) as u64,
|
||||
chunks: vec![hash_a, hash_b],
|
||||
};
|
||||
let mut req = vec![Method::PutManifest.as_byte()];
|
||||
req.extend_from_slice(&serde_json::to_vec(&manifest).unwrap());
|
||||
let reply = dispatch(&router, &req).await;
|
||||
let decoded: PutManifestReply = serde_json::from_slice(&reply).unwrap();
|
||||
assert!(decoded.missing.is_empty());
|
||||
assert!(store.contains(&manifest.blob_id).await.unwrap());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn chunk_rpcs_return_not_configured_without_store() {
|
||||
let gossip = bootstrap_gossip("solo", next_port()).await;
|
||||
let router = RpcRouter::new(gossip, "solo".into(), "z".into());
|
||||
for method in [
|
||||
Method::HasChunk,
|
||||
Method::PutChunk,
|
||||
Method::GetChunk,
|
||||
Method::PutManifest,
|
||||
] {
|
||||
let mut req = vec![method.as_byte()];
|
||||
req.extend_from_slice(&[0u8; 32]);
|
||||
let reply = dispatch(&router, &req).await;
|
||||
assert_eq!(
|
||||
reply,
|
||||
vec![ErrorCode::NotConfigured.as_byte()],
|
||||
"method {method:?} should be NotConfigured"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn end_to_end_push_blob_missing_chunks_replicates_only_needed_bytes() {
|
||||
// Peer A already has one of the two chunks of a blob;
|
||||
// push_blob_missing_chunks uploads only the missing one and
|
||||
// commits the manifest. Verified via the uploaded/total counters.
|
||||
let (id_a, id_b) = NodeIdentity::generate_test_pair("a", "b").unwrap();
|
||||
let (_tmp_a, router_a) = router_with_blobs("a", next_port()).await;
|
||||
let store_a = router_a.blob_store().unwrap().clone();
|
||||
|
||||
let server = QuicServer::bind(loopback(0), id_a).unwrap();
|
||||
let server_addr = server.local_addr().unwrap();
|
||||
let router_srv = router_a.clone();
|
||||
let accept_task = tokio::spawn(async move {
|
||||
if let Some(Ok(conn)) = server.accept().await {
|
||||
let _ = serve_connection(conn, router_srv).await;
|
||||
}
|
||||
});
|
||||
|
||||
let tmp_b = tempfile::TempDir::new().unwrap();
|
||||
let store_b = crate::cluster::blob::BlobStore::open(tmp_b.path().to_path_buf()).unwrap();
|
||||
let payload = {
|
||||
let mut v = Vec::with_capacity(2 * crate::cluster::blob::CHUNK_SIZE);
|
||||
v.extend(std::iter::repeat(0x11u8).take(crate::cluster::blob::CHUNK_SIZE));
|
||||
v.extend(std::iter::repeat(0x22u8).take(crate::cluster::blob::CHUNK_SIZE));
|
||||
v
|
||||
};
|
||||
let blob_id = store_b.put_bytes(&payload).await.unwrap();
|
||||
let manifest_b = store_b.load_manifest(&blob_id).await.unwrap().unwrap();
|
||||
assert_eq!(manifest_b.chunks.len(), 2);
|
||||
|
||||
// Pre-seed A with the FIRST chunk only.
|
||||
let first_bytes = &payload[..crate::cluster::blob::CHUNK_SIZE];
|
||||
store_a
|
||||
.put_chunk(&manifest_b.chunks[0], first_bytes)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let client = QuicClient::new(loopback(0), id_b).unwrap();
|
||||
let conn = client.connect(server_addr, "a").await.unwrap();
|
||||
|
||||
let (uploaded, total) = push_blob_missing_chunks(&conn, &store_b, &blob_id)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(total, 2);
|
||||
assert_eq!(uploaded, 1, "only the missing chunk should have crossed");
|
||||
|
||||
let round = store_a.get_bytes(&blob_id).await.unwrap().unwrap();
|
||||
assert_eq!(round, payload);
|
||||
|
||||
conn.close(quinn::VarInt::from_u32(0), b"done");
|
||||
client.shutdown().await;
|
||||
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||
accept_task.abort();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn call_get_chunk_verifies_returned_hash() {
|
||||
let (id_a, id_b) = NodeIdentity::generate_test_pair("a", "b").unwrap();
|
||||
let (_tmp, router) = router_with_blobs("a", next_port()).await;
|
||||
let bytes = b"chunk to fetch";
|
||||
let hash = ChunkHash::from_bytes(blake3::hash(bytes).into());
|
||||
router
|
||||
.blob_store()
|
||||
.unwrap()
|
||||
.put_chunk(&hash, bytes)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let server = QuicServer::bind(loopback(0), id_a).unwrap();
|
||||
let server_addr = server.local_addr().unwrap();
|
||||
let router_srv = router.clone();
|
||||
let accept_task = tokio::spawn(async move {
|
||||
if let Some(Ok(conn)) = server.accept().await {
|
||||
let _ = serve_connection(conn, router_srv).await;
|
||||
}
|
||||
});
|
||||
|
||||
let client = QuicClient::new(loopback(0), id_b).unwrap();
|
||||
let conn = client.connect(server_addr, "a").await.unwrap();
|
||||
let got = call_get_chunk(&conn, &hash).await.unwrap().unwrap();
|
||||
assert_eq!(got, bytes);
|
||||
|
||||
let missing = ChunkHash::from_bytes([0u8; 32]);
|
||||
assert!(call_get_chunk(&conn, &missing).await.unwrap().is_none());
|
||||
|
||||
conn.close(quinn::VarInt::from_u32(0), b"done");
|
||||
client.shutdown().await;
|
||||
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||
accept_task.abort();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
Reference in New Issue
Block a user