Initial commit: ClawSync v0.1.0

8-crate pure-Rust workspace for revision-aware HDF5 sync.

## Crates
- clawhdf5-onion: ClawOnion VFD — page-level versioned HDF5 storage,
  binary format, writer/reader, branch DAG, GC, snapshots, provenance
- clawsync-core: BLAKE3, xxHash3, FastCDC (+ SIMD NEON), zstd/lz4
- clawsync-onion: IBLT sketch, Merkle tree differ, packet differ/merger,
  ClawSyncManifest, SyncSelector
- clawsync-hdf5: dataset-level manifest, differ, patcher, wire payload
  reconstruction (apply_received_payloads)
- clawsync-transport: TCP, QUIC (quinn 0.11/TLS 1.3), SyncPeer abstraction,
  length-prefixed rkyv wire protocol (21 SyncMessage variants)
- clawsync-agent: OnionMemory, SyncScheduler, TcpSyncBackend,
  PeerCapabilities negotiation
- clawsync-fs: CDC-based delta sync for any file type; FsSyncClient/Server,
  W=16 pipelining, atomic writes
- clawsync-cli: push/pull/serve/hdf5-sync/serve-hdf5/sync/serve-fs +
  all local management commands; --quic on all network commands

## Key features
- IBLT pre-flight: O(revision count) vs rsync's O(file size)
- W=16 sliding-window push: 13–15x speedup over stop-and-wait at WAN RTT
- Dataset-granular HDF5 sync: only modified datasets transferred
- CDC delta for any file type: insertion-stable chunk boundaries
- Full revision DAG: branch, merge, rollback, export, snapshot, GC
- QUIC transport: TLS 1.3, per-message streams via quinn 0.11

## Tests
~573 passing (default features); ~589 with --features simd-cdc

## Performance (Apple Silicon)
- Reconstruct rev=100: 68 µs (target ≤ 1 ms)
- BLAKE3 Rayon 1 MB: 10.3 GiB/s (target ≥ 5 GB/s)
- GC 500 revisions: 20.6 µs (target ≤ 2 s)
- W=16 vs W=1 at 5 ms RTT: 14.8x speedup
- No-op pre-flight at 16 MB: 4 ms vs rsync 35 ms (7.8x)

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
This commit is contained in:
osobh
2026-04-04 18:41:22 -05:00
co-authored by Claude Sonnet 4.6
commit 260e15f5b6
102 changed files with 30098 additions and 0 deletions
+55
View File
@@ -0,0 +1,55 @@
# clawsync-transport
ClawSync transport backends: TCP, QUIC, and mmap.
## Overview
`clawsync-transport` provides the network and IPC layer for ClawSync. All
backends share the same `SyncMessage` wire protocol and length-prefixed framing.
## Backends
### TCP (`tcp`)
Async tokio TCP transport. Each `SyncMessage` is framed as a 4-byte LE length
prefix followed by rkyv-serialized body. Suitable for LAN and WAN sync.
```rust
// Server
let server = TcpServer::bind("0.0.0.0:9999".parse()?).await?;
let (mut conn, _addr) = server.accept().await?;
let msg = conn.recv().await?;
// Client
let mut client = TcpConnection::connect("10.0.0.1:9999".parse()?).await?;
client.send(&SyncMessage::ManifestRequest { .. }).await?;
```
### QUIC (`quic`)
Quinn-based QUIC transport with TLS 1.3, 0-RTT, and certificate pinning. One
stream per `LayerPacket`. Preferred for WAN connections with packet loss.
### mmap (`mmap`)
Zero-copy same-node sync via a shared memory-mapped file ring buffer. For
co-located processes on the same machine.
## Protocol
```text
Client Server
│── ManifestRequest ─────────▶│
│◀─ ManifestResponse ─────────│
│── LayerPacket (rev N+1) ───▶│
│◀─ Ack { revision: N+1 } ────│
│ (repeat for each revision)│
│── SyncComplete ─────────────▶│
```
The `SyncMessage` enum covers: `ManifestRequest`, `ManifestResponse`,
`LayerPacket`, `Ack`, `RetryRequest`, `SyncComplete`, `Error`.
## License
MIT — see repository root.