Initial commit: ClawSync v0.1.0
8-crate pure-Rust workspace for revision-aware HDF5 sync. ## Crates - clawhdf5-onion: ClawOnion VFD — page-level versioned HDF5 storage, binary format, writer/reader, branch DAG, GC, snapshots, provenance - clawsync-core: BLAKE3, xxHash3, FastCDC (+ SIMD NEON), zstd/lz4 - clawsync-onion: IBLT sketch, Merkle tree differ, packet differ/merger, ClawSyncManifest, SyncSelector - clawsync-hdf5: dataset-level manifest, differ, patcher, wire payload reconstruction (apply_received_payloads) - clawsync-transport: TCP, QUIC (quinn 0.11/TLS 1.3), SyncPeer abstraction, length-prefixed rkyv wire protocol (21 SyncMessage variants) - clawsync-agent: OnionMemory, SyncScheduler, TcpSyncBackend, PeerCapabilities negotiation - clawsync-fs: CDC-based delta sync for any file type; FsSyncClient/Server, W=16 pipelining, atomic writes - clawsync-cli: push/pull/serve/hdf5-sync/serve-hdf5/sync/serve-fs + all local management commands; --quic on all network commands ## Key features - IBLT pre-flight: O(revision count) vs rsync's O(file size) - W=16 sliding-window push: 13–15x speedup over stop-and-wait at WAN RTT - Dataset-granular HDF5 sync: only modified datasets transferred - CDC delta for any file type: insertion-stable chunk boundaries - Full revision DAG: branch, merge, rollback, export, snapshot, GC - QUIC transport: TLS 1.3, per-message streams via quinn 0.11 ## Tests ~573 passing (default features); ~589 with --features simd-cdc ## Performance (Apple Silicon) - Reconstruct rev=100: 68 µs (target ≤ 1 ms) - BLAKE3 Rayon 1 MB: 10.3 GiB/s (target ≥ 5 GB/s) - GC 500 revisions: 20.6 µs (target ≤ 2 s) - W=16 vs W=1 at 5 ms RTT: 14.8x speedup - No-op pre-flight at 16 MB: 4 ms vs rsync 35 ms (7.8x) Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
This commit is contained in:
@@ -0,0 +1,55 @@
|
||||
# clawsync-transport
|
||||
|
||||
ClawSync transport backends: TCP, QUIC, and mmap.
|
||||
|
||||
## Overview
|
||||
|
||||
`clawsync-transport` provides the network and IPC layer for ClawSync. All
|
||||
backends share the same `SyncMessage` wire protocol and length-prefixed framing.
|
||||
|
||||
## Backends
|
||||
|
||||
### TCP (`tcp`)
|
||||
|
||||
Async tokio TCP transport. Each `SyncMessage` is framed as a 4-byte LE length
|
||||
prefix followed by rkyv-serialized body. Suitable for LAN and WAN sync.
|
||||
|
||||
```rust
|
||||
// Server
|
||||
let server = TcpServer::bind("0.0.0.0:9999".parse()?).await?;
|
||||
let (mut conn, _addr) = server.accept().await?;
|
||||
let msg = conn.recv().await?;
|
||||
|
||||
// Client
|
||||
let mut client = TcpConnection::connect("10.0.0.1:9999".parse()?).await?;
|
||||
client.send(&SyncMessage::ManifestRequest { .. }).await?;
|
||||
```
|
||||
|
||||
### QUIC (`quic`)
|
||||
|
||||
Quinn-based QUIC transport with TLS 1.3, 0-RTT, and certificate pinning. One
|
||||
stream per `LayerPacket`. Preferred for WAN connections with packet loss.
|
||||
|
||||
### mmap (`mmap`)
|
||||
|
||||
Zero-copy same-node sync via a shared memory-mapped file ring buffer. For
|
||||
co-located processes on the same machine.
|
||||
|
||||
## Protocol
|
||||
|
||||
```text
|
||||
Client Server
|
||||
│── ManifestRequest ─────────▶│
|
||||
│◀─ ManifestResponse ─────────│
|
||||
│── LayerPacket (rev N+1) ───▶│
|
||||
│◀─ Ack { revision: N+1 } ────│
|
||||
│ (repeat for each revision)│
|
||||
│── SyncComplete ─────────────▶│
|
||||
```
|
||||
|
||||
The `SyncMessage` enum covers: `ManifestRequest`, `ManifestResponse`,
|
||||
`LayerPacket`, `Ack`, `RetryRequest`, `SyncComplete`, `Error`.
|
||||
|
||||
## License
|
||||
|
||||
MIT — see repository root.
|
||||
Reference in New Issue
Block a user