Files

T

osobhandClaude Sonnet 4.6 3d524e2d63 Fix intra-doc link warnings; add clawsync-fs README and metadata

Doc fixes (20 warnings → 1 benign name-collision warning):
- clawsync-transport/quic.rs: QuicConfig::with_cert doesn't exist → backtick
- clawsync-onion/iblt.rs: bare `insert` link → backtick
- clawsync-core/lib.rs: simd_cdc is feature-gated → backtick
- clawhdf5-onion/annotation.rs: RevisionEntry/BranchEntry are in external
  clawhdf5-format crate, not re-exported → backtick
- clawhdf5-onion/gc.rs: compact_dead_epoch_revisions is private; flush links
  broken → backtick
- clawhdf5-onion/provenance.rs: RevisionEntry from external crate → backtick
- clawhdf5-onion/reader.rs: reconstruct_revision/revision_pages → Self:: prefix
- clawhdf5-onion/writer.rs: REV_FLAG_SNAPSHOT → crate::format:: path;
  reconstruct_revision → Self:: prefix
Remaining warning is `format` module/macro name collision — not a broken link.

clawsync-fs crate metadata:
- Add readme, keywords, categories to Cargo.toml
- Write README.md (CDC protocol diagram, module overview, usage examples)

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

2026-04-04 20:29:28 -05:00

4.0 KiB

Raw Permalink Blame History

clawsync-fs

CDC-based delta sync for any file type — the general-purpose sync engine behind clawsync sync and clawsync serve-fs.

Overview

clawsync-fs closes the gap between ClawSync's HDF5-specific revision sync and rsync's general-purpose file sync. It handles arbitrary file types and directory trees using Content-Defined Chunking (FastCDC) so chunk boundaries are insertion-stable: a 1-byte prefix insertion does not invalidate the rest of the file's chunks.

Architecture

FsSyncClient                          FsSyncServer
  │                                         │
  ├── FsManifest::build (rayon parallel)    │
  │   BLAKE3 every file in local tree       │
  │                                         │
  ├─── FsDirManifest ──────────────────────▶│
  │                                         │ FsManifest::build (server tree)
  │                                         │ diff_manifests → Added/Modified/Removed
  │◀── FsDirNeed ───────────────────────────│
  │    (needed_files with server chunks,    │
  │     to_delete if allow_delete)          │
  │                                         │
  │  [W=16 pipelined — into_pipe_halves()]  │
  ├─── FsCdcData (/path, chunk_order, ─────▶│ reconstruct_file
  │              literal_chunks)            │ atomic write (.tmp → rename)
  │◀── FsFileAck ───────────────────────────│
  │    (repeat for each modified/added      │
  │     file in any order)                  │
  │◀── FsDirComplete ───────────────────────│

Protocol cost:

Scenario	RTTs
Warm no-op	1
Cold copy	2
Incremental	2

Modules

`manifest`

FsManifest::build(root, excludes) — parallel BLAKE3 walk of a directory tree using walkdir + Rayon. Returns a sorted manifest of all regular files with their BLAKE3 hash, size, and mtime.

let excludes = GlobSet::empty();
let manifest = FsManifest::build(Path::new("./data"), &excludes)?;
for entry in &manifest.entries {
    println!("{} {} bytes", entry.rel_path, entry.size);
}

`differ`

diff_manifests(local, remote) — compare two manifest entry lists by path and BLAKE3 hash. Returns Vec<FileDiff> with Added, Modified, Removed, and Unchanged variants.

`delta`

CDC chunking, transfer-need computation, and file reconstruction.

chunk_file_for_request(data) — chunk bytes and return Vec<FsChunkHash>
compute_needed_indices(server_existing, client_request) — which chunks the server lacks (by xxHash3-64)
build_chunk_data(data, chunks, needed_indices) — pack literal data for transfer; zstd-compresses each chunk, falls back to raw if compressed ≥ raw
reconstruct_file(server_file, client_order, server_map, literals, expected_blake3) — rebuild the target file from server-local chunks + received literals; verifies BLAKE3 on output

`session`

FsSyncClient and FsSyncServer — async protocol orchestration over a SyncPeer (TCP or QUIC).

// Client
let stats = FsSyncClient::new(peer, local_root, excludes, delete)
    .run()
    .await?;
println!("{} added, {} modified, {} removed", stats.files_added, stats.files_modified, stats.files_removed);

// Server (called per accepted connection)
FsSyncServer::new(peer, serve_root, excludes, allow_delete)
    .handle()
    .await?;

Performance

Manifest build: parallelised with Rayon — scales with core count
Chunk transfer: W=16 pipelined send/recv via SyncPeer::into_pipe_halves()
CDC boundaries: FastCDC, immune to insertion staircase that breaks rsync's rolling checksum
Compression: per-chunk zstd; falls back to raw for already-compressed data

License

MIT — see repository root.

4.0 KiB Raw Permalink Blame History