clawverse/clawstor
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 2d: chunk-level RPC (HasChunk / PutChunk / GetChunk / PutManifest)

Browse Source 
Unlocks partial-sync replication — a peer that already has some
chunks of a blob (typical when two nodes share overlapping cargo
build caches) only receives the chunks it's missing.

## New methods

| Byte | Method | Payload | Reply |
|---|---|---|---|
| 0x09 | HasChunk | 32-byte ChunkHash | STREAM_STATUS_OK / NotFound |
| 0x0a | PutChunk | ChunkHash \|\| bytes | STREAM_STATUS_OK / error |
| 0x0b | GetChunk | ChunkHash | STREAM_STATUS_OK \|\| bytes / NotFound |
| 0x0c | PutManifest | JSON BlobManifest | JSON PutManifestReply |

`PutManifestReply { blob_id, missing: Vec<ChunkHash> }`: empty
`missing` means the manifest was written; non-empty tells the
client which chunks to upload before retrying.

Server verifies bytes hash to claimed hash on PutChunk; a
mismatch surfaces as InvalidRequest and the store is untouched.

## BlobStore additions

- `has_chunk(&ChunkHash) → bool`
- `read_chunk(&ChunkHash) → Option<Vec<u8>>` — verifies hash on read
- `put_chunk(&ChunkHash, bytes) → Result<()>` — verifies bytes-vs-hash
- `put_manifest_verified(&manifest) → Result<Vec<ChunkHash>>` —
  returns the list of chunks missing on disk (empty on success)
- `chunk_path` promoted to `pub` for advanced callers

## Client helpers

- `call_has_chunk` / `call_put_chunk` / `call_get_chunk` / `call_put_manifest`
- `push_blob_missing_chunks(conn, local_store, blob_id) →
   Result<(uploaded, total)>` — high-level partial-sync helper

`push_blob_missing_chunks` loads the local manifest, calls HasChunk
for each chunk, uploads only the missing ones via PutChunk, then
commits via PutManifest. On a fully-overlapping cache the uploaded
count is 0 and only the ~small manifest crosses the wire.

## Tests (17 new, all real filesystem + real QUIC — no mocks)

Blob store (6):
- has_chunk_is_false_before_put_and_true_after
- read_chunk_returns_bytes_and_none_when_missing
- put_chunk_rejects_hash_mismatch (nothing written)
- read_chunk_detects_corruption (bit-flip → mismatch error)
- put_manifest_verified_reports_missing_chunks
- put_manifest_verified_writes_when_all_chunks_present

Router dispatch (7):
- phase_2d_method_byte_encoding
- method_reports_streaming_variants — extended for 4 new methods
- has_chunk_returns_ok_for_present_and_not_found_for_missing
- put_chunk_stores_and_returns_status_ok
- put_chunk_rejects_hash_mismatch_over_wire
- get_chunk_returns_content_prefixed_with_status_ok
- get_chunk_returns_not_found_for_missing
- put_manifest_reports_missing_chunks_when_incomplete
- put_manifest_writes_when_chunks_present
- chunk_rpcs_return_not_configured_without_store

End-to-end (2):
- **end_to_end_push_blob_missing_chunks_replicates_only_needed_bytes**:
  Peer A pre-seeded with chunk 0 of a 2-chunk (8 MiB) blob;
  `push_blob_missing_chunks` reports `(uploaded=1, total=2)`,
  only chunk 1 crosses the wire, A's store then contains the
  complete blob and `get_bytes` returns byte-equal content.
- **call_get_chunk_verifies_returned_hash**: real 2-node fetch,
  client hashes received bytes and compares to requested hash.

156 tests pass. Pre-existing macOS-only failure unchanged.

File sizes (all under 1300-line ceiling):
- cluster/rpc.rs: 1053
- cluster/rpc/tests.rs: 940
- cluster/blob.rs: 1186

## Where this fits

With Phase 2c whole-blob streaming + Phase 2d partial-chunk sync,
the storage substrate is now genuinely bandwidth-efficient in the
distributed setting:

- First-ever push of a blob: `push_blob_missing_chunks` uploads
  everything (all chunks missing).
- Second push of a similar blob (95% chunk overlap with prior
  contents): only the 5% new chunks cross the wire, plus a tiny
  manifest.
- Whole-blob download: BlobGetStream, bounded by network bandwidth.

## Follow-on

- Phase 3: CRDT metadata for human-readable namespaces on top of
  content hashes.
- Phase 5: the killer feature. Fingerprint cargo target dir → tar
  → hash → PutBlobStream (or push_blob_missing_chunks if a similar
  build already lives on the peer). Same fingerprint on the next
  node → BlobGetStream. This is the whole cargo-cache design in
  one line and it now sits on a substrate that handles all the
  hard cases (dedup, verification, resumability, partial sync).
This commit is contained in:
Omar Sobh
2026-07-11 23:22:25 -07:00
parent 1fd1027da4
commit 2e984b924d
3 changed files with 737 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
claw-store/src/cluster/blob.rs
+185 -1
View File
@@ -353,6 +353,90 @@ impl BlobStore {
Ok(true)
}
// ── Chunk-level API (Phase 2d) ───────────────────────────────────
//
// These operations expose the store's chunk substrate directly so a
// sender can query + upload one chunk at a time. Combined with
// gossip-derived affinity + the `HasChunk` RPC, they let a peer
// replicate a blob by transferring only chunks the receiver is
// missing — the "big win when caches overlap" case.
/// Whether a chunk file exists on disk. No hash verification —
/// callers who want cryptographic proof should follow with
/// [`read_chunk`] and check the returned hash themselves.
pub async fn has_chunk(&self, hash: &ChunkHash) -> Result<bool> {
match tokio::fs::metadata(self.chunk_path(hash)).await {
Ok(_) => Ok(true),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(false),
Err(e) => Err(anyhow::Error::from(e)),
}
}
/// Read a single chunk's raw bytes. Returns `None` when the chunk
/// isn't on disk. Verifies the chunk hashes to the requested
/// `hash` (defense in depth against silent corruption); a mismatch
/// returns `Err`.
pub async fn read_chunk(&self, hash: &ChunkHash) -> Result<Option<Vec<u8>>> {
let path = self.chunk_path(hash);
let bytes = match tokio::fs::read(&path).await {
Ok(b) => b,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(None),
Err(e) => return Err(anyhow::Error::from(e)),
};
let recomputed = ChunkHash(blake3::hash(&bytes).into());
if recomputed != *hash {
bail!(
"chunk hash mismatch at {}: requested {}, disk hashes to {}",
path.display(),
hash.to_hex(),
recomputed.to_hex()
);
}
Ok(Some(bytes))
}
/// Store one chunk. Verifies that `bytes` actually hash to `hash`
/// before writing — the caller can't accidentally (or maliciously)
/// stash unrelated bytes under a hash it doesn't own. Idempotent:
/// writing the same chunk twice is a no-op after the first.
pub async fn put_chunk(&self, hash: &ChunkHash, bytes: &[u8]) -> Result<()> {
let recomputed = ChunkHash(blake3::hash(bytes).into());
if recomputed != *hash {
bail!(
"chunk hash mismatch: claimed {}, bytes hash to {}",
hash.to_hex(),
recomputed.to_hex()
);
}
self.write_chunk_if_absent(hash, bytes).await
}
/// Persist a manifest whose referenced chunks are already on disk.
/// Verifies that every chunk in the manifest exists before writing —
/// callers can't accidentally register a manifest that references
/// missing chunks (which would break subsequent `get_bytes`).
///
/// Returns the list of chunk hashes that were missing when the
/// call started; if empty, the manifest was written. Otherwise
/// nothing was written and the client is expected to upload the
/// missing chunks (typically via [`put_chunk`]) and retry.
pub async fn put_manifest_verified(
&self,
manifest: &BlobManifest,
) -> Result<Vec<ChunkHash>> {
let mut missing = Vec::new();
for chunk_hash in &manifest.chunks {
if !self.has_chunk(chunk_hash).await? {
missing.push(*chunk_hash);
}
}
if !missing.is_empty() {
return Ok(missing);
}
self.write_manifest_if_absent(manifest).await?;
Ok(Vec::new())
}
/// Whether a manifest exists for `id`. Cheap — no chunk reads.
pub async fn contains(&self, id: &BlobId) -> Result<bool> {
match tokio::fs::metadata(self.manifest_path(id)).await {
@@ -471,7 +555,10 @@ impl BlobStore {
.join(format!("{hex}.manifest.json"))
}
fn chunk_path(&self, hash: &ChunkHash) -> PathBuf {
/// On-disk path for a chunk. Public so tests + advanced callers
/// (e.g. custom migration tools) can address individual chunks
/// without duplicating the layout convention.
pub fn chunk_path(&self, hash: &ChunkHash) -> PathBuf {
let hex = hash.to_hex();
self.root.join("chunks").join(&hex[..2]).join(&hex)
}
@@ -959,6 +1046,103 @@ mod tests {
assert!(sink.is_empty());
}
// ── Phase 2d: chunk-level API ────────────────────────────────────
#[tokio::test]
async fn has_chunk_is_false_before_put_and_true_after() {
let (_tmp, store) = open_store();
let bytes = b"chunk-content";
let hash = ChunkHash::from_bytes(blake3::hash(bytes).into());
assert!(!store.has_chunk(&hash).await.unwrap());
store.put_chunk(&hash, bytes).await.unwrap();
assert!(store.has_chunk(&hash).await.unwrap());
}
#[tokio::test]
async fn read_chunk_returns_bytes_and_none_when_missing() {
let (_tmp, store) = open_store();
let bytes = b"read-me";
let hash = ChunkHash::from_bytes(blake3::hash(bytes).into());
assert!(store.read_chunk(&hash).await.unwrap().is_none());
store.put_chunk(&hash, bytes).await.unwrap();
let round = store.read_chunk(&hash).await.unwrap().unwrap();
assert_eq!(round, bytes);
}
#[tokio::test]
async fn put_chunk_rejects_hash_mismatch() {
let (_tmp, store) = open_store();
// Claim a hash that doesn't match the bytes.
let bogus = ChunkHash::from_bytes([0xffu8; 32]);
let err = store
.put_chunk(&bogus, b"real content")
.await
.unwrap_err()
.to_string();
assert!(
err.contains("chunk hash mismatch"),
"unexpected error: {err}"
);
// Nothing should have been written.
assert!(!store.has_chunk(&bogus).await.unwrap());
}
#[tokio::test]
async fn read_chunk_detects_corruption() {
let (_tmp, store) = open_store();
let bytes = b"tamperproof";
let hash = ChunkHash::from_bytes(blake3::hash(bytes).into());
store.put_chunk(&hash, bytes).await.unwrap();
// Corrupt the on-disk file.
std::fs::write(store.chunk_path(&hash), b"tampered").unwrap();
let err = store.read_chunk(&hash).await.unwrap_err().to_string();
assert!(err.contains("chunk hash mismatch"));
}
#[tokio::test]
async fn put_manifest_verified_reports_missing_chunks() {
let (_tmp, store) = open_store();
// Cook a manifest whose chunks aren't on disk.
let phantom_chunks = vec![
ChunkHash::from_bytes([0x01u8; 32]),
ChunkHash::from_bytes([0x02u8; 32]),
];
let manifest = BlobManifest {
blob_id: BlobId::from_bytes([0x03u8; 32]),
total_size: 100,
chunks: phantom_chunks.clone(),
};
let missing = store.put_manifest_verified(&manifest).await.unwrap();
assert_eq!(missing, phantom_chunks);
// Manifest must NOT have been persisted since chunks are missing.
assert!(!store.contains(&manifest.blob_id).await.unwrap());
}
#[tokio::test]
async fn put_manifest_verified_writes_when_all_chunks_present() {
let (_tmp, store) = open_store();
// Upload two chunks, then commit a manifest referencing them.
let a = b"chunk-a";
let b = b"chunk-b";
let hash_a = ChunkHash::from_bytes(blake3::hash(a).into());
let hash_b = ChunkHash::from_bytes(blake3::hash(b).into());
store.put_chunk(&hash_a, a).await.unwrap();
store.put_chunk(&hash_b, b).await.unwrap();
// BlobId here is arbitrary — chunk-level API doesn't verify
// that blob_id = blake3(concat(chunks)); that's the sender's
// responsibility (production callers get it right via the
// top-level put_bytes / put_stream paths).
let manifest = BlobManifest {
blob_id: BlobId::from_bytes([0x99u8; 32]),
total_size: (a.len() + b.len()) as u64,
chunks: vec![hash_a, hash_b],
};
let missing = store.put_manifest_verified(&manifest).await.unwrap();
assert!(missing.is_empty());
assert!(store.contains(&manifest.blob_id).await.unwrap());
}
#[tokio::test]
async fn stream_to_detects_chunk_corruption() {
let (_tmp, store) = open_store();