osobhandClaude Sonnet 4.6
6d2eb5a1db
- Add QuicSyncBackend + quic_scheduler; fix push stream/close race by waiting for peer close instead of calling close_and_drain() on the client side (SyncComplete stream was racing CONNECTION_CLOSE) - Add PipeWriteHalf::shutdown_push() for client-sends-last QUIC paths; use it in cmd_push so the server can process SyncComplete before the connection tears down - Add SshSyncBackend + ssh_scheduler with subprocess integration tests - Add clawsync diff command with --porcelain flag and subprocess tests - Add QUIC subprocess push/pull integration tests - Fix clippy --tests violations across 5 crates - Fix broken intra-doc links (reader.rs, lib.rs, scheduler.rs) - Rewrite crate READMEs; update BENCHMARKS.md with clawsync-fs CDC row 653 tests, 0 failures. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
70 lines
2.3 KiB
Markdown
70 lines
2.3 KiB
Markdown
# clawsync-onion
|
||
|
||
ClawSync onion-aware revision diff, layer packaging, IBLT pre-flight, and
|
||
manifest protocol.
|
||
|
||
## Overview
|
||
|
||
`clawsync-onion` sits between the onion storage layer (`clawhdf5-onion`) and the
|
||
transport layer (`clawsync-transport`). It knows how to:
|
||
|
||
1. Compute which revisions a remote node is missing (`diff_revisions`)
|
||
2. Package those revisions as `OnionLayerPacket` wire messages
|
||
3. Apply incoming packets to a local `OnionFile` (`merge_packets`)
|
||
4. Build and exchange `IbltManifest` for sub-linear pre-flight negotiation
|
||
5. Filter revisions by branch, ceiling, or selector (`SyncSelector`)
|
||
|
||
## Key types
|
||
|
||
### `diff_revisions(local, remote_head) -> Vec<OnionLayerPacket>`
|
||
|
||
Returns the minimal set of packets needed to bring the remote up to date —
|
||
ascending revision order, ready to send.
|
||
|
||
### `merge_packets(onion, packets, flush) -> MergeStats`
|
||
|
||
Applies incoming packets to a local `OnionFile`. Verifies BLAKE3 integrity,
|
||
skips already-present revisions (idempotent), and optionally flushes.
|
||
|
||
### `IbltManifest` / `IbltSketch`
|
||
|
||
Rateless IBLT (Invertible Bloom Lookup Table) for sub-linear pre-flight
|
||
comparison. At N=1K revisions, IBLT pre-flight requires ~4 ms and ~4 KB of wire
|
||
data vs ~2 s and ~4 MB for a flat manifest exchange — roughly 500× faster.
|
||
|
||
```rust
|
||
// Client side: encode local revisions into a sketch
|
||
let sketch = IbltSketch::from_keys(&local_rev_numbers, seed);
|
||
let manifest = IbltManifest { sketch: sketch.to_bytes(), .. };
|
||
|
||
// Server side: compute what the client is missing
|
||
let diff = client_manifest.diff_against(&server_revs)?;
|
||
// diff.only_in_b = revisions client has that server lacks → send these
|
||
```
|
||
|
||
### `ClawSyncManifest`
|
||
|
||
Flat manifest exchanged during pull-direction negotiation. Contains revision
|
||
count, HEAD revision, and per-revision BLAKE3 hashes. Used when the remote
|
||
has more data than the local node (pull pre-flight).
|
||
|
||
### `SyncSelector`
|
||
|
||
Filter revisions before pushing:
|
||
|
||
```rust
|
||
SyncSelector::All // push every revision
|
||
SyncSelector::Branch(id) // push only revisions on branch `id`
|
||
SyncSelector::UpTo(rev) // push revisions 0..=rev
|
||
```
|
||
|
||
### `OnionLayerPacket`
|
||
|
||
Wire representation of a single revision: revision number, branch ID, parent
|
||
revision, BLAKE3 root, and a list of `OnionPage` (h5_offset + page bytes).
|
||
Serialized with rkyv for zero-copy deserialization.
|
||
|
||
## License
|
||
|
||
MIT — see repository root.
|