Phase 2d: chunk-level RPC (HasChunk / PutChunk / GetChunk / PutManifest)

Unlocks partial-sync replication — a peer that already has some
chunks of a blob (typical when two nodes share overlapping cargo
build caches) only receives the chunks it's missing.

## New methods

| Byte | Method | Payload | Reply |
|---|---|---|---|
| 0x09 | HasChunk | 32-byte ChunkHash | STREAM_STATUS_OK / NotFound |
| 0x0a | PutChunk | ChunkHash \|\| bytes | STREAM_STATUS_OK / error |
| 0x0b | GetChunk | ChunkHash | STREAM_STATUS_OK \|\| bytes / NotFound |
| 0x0c | PutManifest | JSON BlobManifest | JSON PutManifestReply |

`PutManifestReply { blob_id, missing: Vec<ChunkHash> }`: empty
`missing` means the manifest was written; non-empty tells the
client which chunks to upload before retrying.

Server verifies bytes hash to claimed hash on PutChunk; a
mismatch surfaces as InvalidRequest and the store is untouched.

## BlobStore additions

- `has_chunk(&ChunkHash) → bool`
- `read_chunk(&ChunkHash) → Option<Vec<u8>>` — verifies hash on read
- `put_chunk(&ChunkHash, bytes) → Result<()>` — verifies bytes-vs-hash
- `put_manifest_verified(&manifest) → Result<Vec<ChunkHash>>` —
  returns the list of chunks missing on disk (empty on success)
- `chunk_path` promoted to `pub` for advanced callers

## Client helpers

- `call_has_chunk` / `call_put_chunk` / `call_get_chunk` / `call_put_manifest`
- `push_blob_missing_chunks(conn, local_store, blob_id) →
   Result<(uploaded, total)>` — high-level partial-sync helper

`push_blob_missing_chunks` loads the local manifest, calls HasChunk
for each chunk, uploads only the missing ones via PutChunk, then
commits via PutManifest. On a fully-overlapping cache the uploaded
count is 0 and only the ~small manifest crosses the wire.

## Tests (17 new, all real filesystem + real QUIC — no mocks)

Blob store (6):
- has_chunk_is_false_before_put_and_true_after
- read_chunk_returns_bytes_and_none_when_missing
- put_chunk_rejects_hash_mismatch (nothing written)
- read_chunk_detects_corruption (bit-flip → mismatch error)
- put_manifest_verified_reports_missing_chunks
- put_manifest_verified_writes_when_all_chunks_present

Router dispatch (7):
- phase_2d_method_byte_encoding
- method_reports_streaming_variants — extended for 4 new methods
- has_chunk_returns_ok_for_present_and_not_found_for_missing
- put_chunk_stores_and_returns_status_ok
- put_chunk_rejects_hash_mismatch_over_wire
- get_chunk_returns_content_prefixed_with_status_ok
- get_chunk_returns_not_found_for_missing
- put_manifest_reports_missing_chunks_when_incomplete
- put_manifest_writes_when_chunks_present
- chunk_rpcs_return_not_configured_without_store

End-to-end (2):
- **end_to_end_push_blob_missing_chunks_replicates_only_needed_bytes**:
  Peer A pre-seeded with chunk 0 of a 2-chunk (8 MiB) blob;
  `push_blob_missing_chunks` reports `(uploaded=1, total=2)`,
  only chunk 1 crosses the wire, A's store then contains the
  complete blob and `get_bytes` returns byte-equal content.
- **call_get_chunk_verifies_returned_hash**: real 2-node fetch,
  client hashes received bytes and compares to requested hash.

156 tests pass. Pre-existing macOS-only failure unchanged.

File sizes (all under 1300-line ceiling):
- cluster/rpc.rs: 1053
- cluster/rpc/tests.rs: 940
- cluster/blob.rs: 1186

## Where this fits

With Phase 2c whole-blob streaming + Phase 2d partial-chunk sync,
the storage substrate is now genuinely bandwidth-efficient in the
distributed setting:

- First-ever push of a blob: `push_blob_missing_chunks` uploads
  everything (all chunks missing).
- Second push of a similar blob (95% chunk overlap with prior
  contents): only the 5% new chunks cross the wire, plus a tiny
  manifest.
- Whole-blob download: BlobGetStream, bounded by network bandwidth.

## Follow-on

- Phase 3: CRDT metadata for human-readable namespaces on top of
  content hashes.
- Phase 5: the killer feature. Fingerprint cargo target dir → tar
  → hash → PutBlobStream (or push_blob_missing_chunks if a similar
  build already lives on the peer). Same fingerprint on the next
  node → BlobGetStream. This is the whole cargo-cache design in
  one line and it now sits on a substrate that handles all the
  hard cases (dedup, verification, resumability, partial sync).
This commit is contained in:
Omar Sobh
2026-07-11 23:22:25 -07:00
parent 1fd1027da4
commit 2e984b924d
3 changed files with 737 additions and 2 deletions
+246
View File
@@ -513,6 +513,252 @@ fn method_reports_streaming_variants() {
assert!(!Method::BlobLoadManifest.is_streaming());
assert!(Method::BlobPutStream.is_streaming());
assert!(Method::BlobGetStream.is_streaming());
assert!(!Method::HasChunk.is_streaming());
assert!(!Method::PutChunk.is_streaming());
assert!(!Method::GetChunk.is_streaming());
assert!(!Method::PutManifest.is_streaming());
}
#[test]
fn phase_2d_method_byte_encoding() {
assert_eq!(Method::HasChunk.as_byte(), 0x09);
assert_eq!(Method::PutChunk.as_byte(), 0x0a);
assert_eq!(Method::GetChunk.as_byte(), 0x0b);
assert_eq!(Method::PutManifest.as_byte(), 0x0c);
assert_eq!(Method::from_byte(0x09), Some(Method::HasChunk));
assert_eq!(Method::from_byte(0x0a), Some(Method::PutChunk));
assert_eq!(Method::from_byte(0x0b), Some(Method::GetChunk));
assert_eq!(Method::from_byte(0x0c), Some(Method::PutManifest));
}
// ── Phase 2d: chunk-level RPC ────────────────────────────────────────
#[tokio::test]
async fn has_chunk_returns_ok_for_present_and_not_found_for_missing() {
let (_tmp, router) = router_with_blobs("solo", next_port()).await;
let bytes = b"chunky content";
let hash = ChunkHash::from_bytes(blake3::hash(bytes).into());
router
.blob_store()
.unwrap()
.put_chunk(&hash, bytes)
.await
.unwrap();
let mut req = vec![Method::HasChunk.as_byte()];
req.extend_from_slice(hash.as_bytes());
assert_eq!(dispatch(&router, &req).await, vec![STREAM_STATUS_OK]);
let mut req = vec![Method::HasChunk.as_byte()];
req.extend_from_slice(&[0u8; 32]);
assert_eq!(dispatch(&router, &req).await, vec![ErrorCode::NotFound.as_byte()]);
}
#[tokio::test]
async fn put_chunk_stores_and_returns_status_ok() {
let (_tmp, router) = router_with_blobs("solo", next_port()).await;
let bytes = b"payload for put_chunk";
let hash = ChunkHash::from_bytes(blake3::hash(bytes).into());
let mut req = vec![Method::PutChunk.as_byte()];
req.extend_from_slice(hash.as_bytes());
req.extend_from_slice(bytes);
assert_eq!(dispatch(&router, &req).await, vec![STREAM_STATUS_OK]);
assert!(router.blob_store().unwrap().has_chunk(&hash).await.unwrap());
}
#[tokio::test]
async fn put_chunk_rejects_hash_mismatch_over_wire() {
let (_tmp, router) = router_with_blobs("solo", next_port()).await;
let bytes = b"real";
let bogus = ChunkHash::from_bytes([0xffu8; 32]);
let mut req = vec![Method::PutChunk.as_byte()];
req.extend_from_slice(bogus.as_bytes());
req.extend_from_slice(bytes);
let reply = dispatch(&router, &req).await;
assert_eq!(reply, vec![ErrorCode::InvalidRequest.as_byte()]);
assert!(!router.blob_store().unwrap().has_chunk(&bogus).await.unwrap());
}
#[tokio::test]
async fn get_chunk_returns_content_prefixed_with_status_ok() {
let (_tmp, router) = router_with_blobs("solo", next_port()).await;
let bytes = b"fetch me by hash";
let hash = ChunkHash::from_bytes(blake3::hash(bytes).into());
router
.blob_store()
.unwrap()
.put_chunk(&hash, bytes)
.await
.unwrap();
let mut req = vec![Method::GetChunk.as_byte()];
req.extend_from_slice(hash.as_bytes());
let reply = dispatch(&router, &req).await;
assert_eq!(reply[0], STREAM_STATUS_OK);
assert_eq!(&reply[1..], bytes);
}
#[tokio::test]
async fn get_chunk_returns_not_found_for_missing() {
let (_tmp, router) = router_with_blobs("solo", next_port()).await;
let missing = ChunkHash::from_bytes([0u8; 32]);
let mut req = vec![Method::GetChunk.as_byte()];
req.extend_from_slice(missing.as_bytes());
let reply = dispatch(&router, &req).await;
assert_eq!(reply, vec![ErrorCode::NotFound.as_byte()]);
}
#[tokio::test]
async fn put_manifest_reports_missing_chunks_when_incomplete() {
let (_tmp, router) = router_with_blobs("solo", next_port()).await;
let phantom = ChunkHash::from_bytes([0xaau8; 32]);
let manifest = BlobManifest {
blob_id: BlobId::from_bytes([0x11u8; 32]),
total_size: 42,
chunks: vec![phantom],
};
let mut req = vec![Method::PutManifest.as_byte()];
req.extend_from_slice(&serde_json::to_vec(&manifest).unwrap());
let reply = dispatch(&router, &req).await;
let decoded: PutManifestReply = serde_json::from_slice(&reply).unwrap();
assert_eq!(decoded.blob_id, manifest.blob_id);
assert_eq!(decoded.missing, vec![phantom]);
}
#[tokio::test]
async fn put_manifest_writes_when_chunks_present() {
let (_tmp, router) = router_with_blobs("solo", next_port()).await;
let store = router.blob_store().unwrap().clone();
let a = b"first chunk";
let b = b"second chunk";
let hash_a = ChunkHash::from_bytes(blake3::hash(a).into());
let hash_b = ChunkHash::from_bytes(blake3::hash(b).into());
store.put_chunk(&hash_a, a).await.unwrap();
store.put_chunk(&hash_b, b).await.unwrap();
let manifest = BlobManifest {
blob_id: BlobId::from_bytes([0x22u8; 32]),
total_size: (a.len() + b.len()) as u64,
chunks: vec![hash_a, hash_b],
};
let mut req = vec![Method::PutManifest.as_byte()];
req.extend_from_slice(&serde_json::to_vec(&manifest).unwrap());
let reply = dispatch(&router, &req).await;
let decoded: PutManifestReply = serde_json::from_slice(&reply).unwrap();
assert!(decoded.missing.is_empty());
assert!(store.contains(&manifest.blob_id).await.unwrap());
}
#[tokio::test]
async fn chunk_rpcs_return_not_configured_without_store() {
let gossip = bootstrap_gossip("solo", next_port()).await;
let router = RpcRouter::new(gossip, "solo".into(), "z".into());
for method in [
Method::HasChunk,
Method::PutChunk,
Method::GetChunk,
Method::PutManifest,
] {
let mut req = vec![method.as_byte()];
req.extend_from_slice(&[0u8; 32]);
let reply = dispatch(&router, &req).await;
assert_eq!(
reply,
vec![ErrorCode::NotConfigured.as_byte()],
"method {method:?} should be NotConfigured"
);
}
}
#[tokio::test]
async fn end_to_end_push_blob_missing_chunks_replicates_only_needed_bytes() {
// Peer A already has one of the two chunks of a blob;
// push_blob_missing_chunks uploads only the missing one and
// commits the manifest. Verified via the uploaded/total counters.
let (id_a, id_b) = NodeIdentity::generate_test_pair("a", "b").unwrap();
let (_tmp_a, router_a) = router_with_blobs("a", next_port()).await;
let store_a = router_a.blob_store().unwrap().clone();
let server = QuicServer::bind(loopback(0), id_a).unwrap();
let server_addr = server.local_addr().unwrap();
let router_srv = router_a.clone();
let accept_task = tokio::spawn(async move {
if let Some(Ok(conn)) = server.accept().await {
let _ = serve_connection(conn, router_srv).await;
}
});
let tmp_b = tempfile::TempDir::new().unwrap();
let store_b = crate::cluster::blob::BlobStore::open(tmp_b.path().to_path_buf()).unwrap();
let payload = {
let mut v = Vec::with_capacity(2 * crate::cluster::blob::CHUNK_SIZE);
v.extend(std::iter::repeat(0x11u8).take(crate::cluster::blob::CHUNK_SIZE));
v.extend(std::iter::repeat(0x22u8).take(crate::cluster::blob::CHUNK_SIZE));
v
};
let blob_id = store_b.put_bytes(&payload).await.unwrap();
let manifest_b = store_b.load_manifest(&blob_id).await.unwrap().unwrap();
assert_eq!(manifest_b.chunks.len(), 2);
// Pre-seed A with the FIRST chunk only.
let first_bytes = &payload[..crate::cluster::blob::CHUNK_SIZE];
store_a
.put_chunk(&manifest_b.chunks[0], first_bytes)
.await
.unwrap();
let client = QuicClient::new(loopback(0), id_b).unwrap();
let conn = client.connect(server_addr, "a").await.unwrap();
let (uploaded, total) = push_blob_missing_chunks(&conn, &store_b, &blob_id)
.await
.unwrap();
assert_eq!(total, 2);
assert_eq!(uploaded, 1, "only the missing chunk should have crossed");
let round = store_a.get_bytes(&blob_id).await.unwrap().unwrap();
assert_eq!(round, payload);
conn.close(quinn::VarInt::from_u32(0), b"done");
client.shutdown().await;
tokio::time::sleep(Duration::from_millis(50)).await;
accept_task.abort();
}
#[tokio::test]
async fn call_get_chunk_verifies_returned_hash() {
let (id_a, id_b) = NodeIdentity::generate_test_pair("a", "b").unwrap();
let (_tmp, router) = router_with_blobs("a", next_port()).await;
let bytes = b"chunk to fetch";
let hash = ChunkHash::from_bytes(blake3::hash(bytes).into());
router
.blob_store()
.unwrap()
.put_chunk(&hash, bytes)
.await
.unwrap();
let server = QuicServer::bind(loopback(0), id_a).unwrap();
let server_addr = server.local_addr().unwrap();
let router_srv = router.clone();
let accept_task = tokio::spawn(async move {
if let Some(Ok(conn)) = server.accept().await {
let _ = serve_connection(conn, router_srv).await;
}
});
let client = QuicClient::new(loopback(0), id_b).unwrap();
let conn = client.connect(server_addr, "a").await.unwrap();
let got = call_get_chunk(&conn, &hash).await.unwrap().unwrap();
assert_eq!(got, bytes);
let missing = ChunkHash::from_bytes([0u8; 32]);
assert!(call_get_chunk(&conn, &missing).await.unwrap().is_none());
conn.close(quinn::VarInt::from_u32(0), b"done");
client.shutdown().await;
tokio::time::sleep(Duration::from_millis(50)).await;
accept_task.abort();
}
#[tokio::test]