Commit Graph
2 Commits
Author SHA1 Message Date
rust-refactor 6c419e0ad9 research: two-stage RBF pre-flight for divergent replicas [arxiv:2510.27614]
Adds a Rateless Bloom Filter + residual IBLT hybrid for the sync
pre-flight, implementing the divergent-replica reconciliation scheme
from arXiv 2510.27614 (Silva Gomes & Baquero, 2025).

When the local revision count exceeds RBF_REV_COUNT_THRESHOLD (200) —
the heuristic for initial clones and post-partition reconnects — the
client now ships a small RBF + small residual IBLT instead of an
over-provisioned single-stage IBLT.  The receiver partitions its key
set by the Bloom filter, builds the candidate-intersection residual
IBLT, and subtracts the client bundle to recover missing_from_remote
in one round trip.

Wire-format change is strictly additive: two new SyncMessage variants
(RbfRequest, RbfResponse) are appended after SealedFileAck; all
existing discriminants are preserved (regression test added).  Onion
diff and transport framing are unchanged.

Files modified / added:
- crates/clawsync-onion/src/rbf.rs (new module, 30 unit tests + 3 proptests)
- crates/clawsync-onion/src/manifest.rs (RbfManifest bundle type + 6 tests)
- crates/clawsync-onion/src/lib.rs (re-exports)
- crates/clawsync-transport/src/protocol.rs (RbfRequest/RbfResponse variants + 3 roundtrip tests)
- crates/clawsync-cli/src/main.rs (heuristic switch in client push path + server dispatch)
- crates/clawsync-agent/src/backend.rs (TCP/QUIC/SSH push paths opt into RBF)
- CHANGELOG.md (Unreleased entry)
- crates/clawsync-onion/proptest-regressions/rbf.txt (seed file)

Gates:
- cargo check --workspace: clean
- cargo clippy -p (affected crates) --lib --bins --tests -- -D warnings: clean
- cargo test --workspace: 704 passed / 0 failed (87 in clawsync-onion incl. all RBF tests)
2026-05-19 07:46:56 -07:00
osobhandClaude Sonnet 4.6 260e15f5b6 Initial commit: ClawSync v0.1.0
8-crate pure-Rust workspace for revision-aware HDF5 sync.

## Crates
- clawhdf5-onion: ClawOnion VFD — page-level versioned HDF5 storage,
  binary format, writer/reader, branch DAG, GC, snapshots, provenance
- clawsync-core: BLAKE3, xxHash3, FastCDC (+ SIMD NEON), zstd/lz4
- clawsync-onion: IBLT sketch, Merkle tree differ, packet differ/merger,
  ClawSyncManifest, SyncSelector
- clawsync-hdf5: dataset-level manifest, differ, patcher, wire payload
  reconstruction (apply_received_payloads)
- clawsync-transport: TCP, QUIC (quinn 0.11/TLS 1.3), SyncPeer abstraction,
  length-prefixed rkyv wire protocol (21 SyncMessage variants)
- clawsync-agent: OnionMemory, SyncScheduler, TcpSyncBackend,
  PeerCapabilities negotiation
- clawsync-fs: CDC-based delta sync for any file type; FsSyncClient/Server,
  W=16 pipelining, atomic writes
- clawsync-cli: push/pull/serve/hdf5-sync/serve-hdf5/sync/serve-fs +
  all local management commands; --quic on all network commands

## Key features
- IBLT pre-flight: O(revision count) vs rsync's O(file size)
- W=16 sliding-window push: 13–15x speedup over stop-and-wait at WAN RTT
- Dataset-granular HDF5 sync: only modified datasets transferred
- CDC delta for any file type: insertion-stable chunk boundaries
- Full revision DAG: branch, merge, rollback, export, snapshot, GC
- QUIC transport: TLS 1.3, per-message streams via quinn 0.11

## Tests
~573 passing (default features); ~589 with --features simd-cdc

## Performance (Apple Silicon)
- Reconstruct rev=100: 68 µs (target ≤ 1 ms)
- BLAKE3 Rayon 1 MB: 10.3 GiB/s (target ≥ 5 GB/s)
- GC 500 revisions: 20.6 µs (target ≤ 2 s)
- W=16 vs W=1 at 5 ms RTT: 14.8x speedup
- No-op pre-flight at 16 MB: 4 ms vs rsync 35 ms (7.8x)

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-04-04 18:41:22 -05:00