Fix intra-doc link warnings; add clawsync-fs README and metadata

Doc fixes (20 warnings → 1 benign name-collision warning): - clawsync-transport/quic.rs: QuicConfig::with_cert doesn't exist → backtick - clawsync-onion/iblt.rs: bare `insert` link → backtick - clawsync-core/lib.rs: simd_cdc is feature-gated → backtick - clawhdf5-onion/annotation.rs: RevisionEntry/BranchEntry are in external clawhdf5-format crate, not re-exported → backtick - clawhdf5-onion/gc.rs: compact_dead_epoch_revisions is private; flush links broken → backtick - clawhdf5-onion/provenance.rs: RevisionEntry from external crate → backtick - clawhdf5-onion/reader.rs: reconstruct_revision/revision_pages → Self:: prefix - clawhdf5-onion/writer.rs: REV_FLAG_SNAPSHOT → crate::format:: path; reconstruct_revision → Self:: prefix Remaining warning is `format` module/macro name collision — not a broken link. clawsync-fs crate metadata: - Add readme, keywords, categories to Cargo.toml - Write README.md (CDC protocol diagram, module overview, usage examples) Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-04-04 20:29:28 -05:00
parent 2e423d2682
commit 3d524e2d63
10 changed files with 126 additions and 16 deletions
@@ -0,0 +1,107 @@
+# clawsync-fs
+
+CDC-based delta sync for any file type — the general-purpose sync engine behind
+`clawsync sync` and `clawsync serve-fs`.
+
+## Overview
+
+`clawsync-fs` closes the gap between ClawSync's HDF5-specific revision sync and
+rsync's general-purpose file sync. It handles arbitrary file types and directory
+trees using **Content-Defined Chunking** (FastCDC) so chunk boundaries are
+insertion-stable: a 1-byte prefix insertion does not invalidate the rest of the
+file's chunks.
+
+## Architecture
+
+```
+FsSyncClient                          FsSyncServer
+  │                                         │
+  ├── FsManifest::build (rayon parallel)    │
+  │   BLAKE3 every file in local tree       │
+  │                                         │
+  ├─── FsDirManifest ──────────────────────▶│
+  │                                         │ FsManifest::build (server tree)
+  │                                         │ diff_manifests → Added/Modified/Removed
+  │◀── FsDirNeed ───────────────────────────│
+  │    (needed_files with server chunks,    │
+  │     to_delete if allow_delete)          │
+  │                                         │
+  │  [W=16 pipelined — into_pipe_halves()]  │
+  ├─── FsCdcData (/path, chunk_order, ─────▶│ reconstruct_file
+  │              literal_chunks)            │ atomic write (.tmp → rename)
+  │◀── FsFileAck ───────────────────────────│
+  │    (repeat for each modified/added      │
+  │     file in any order)                  │
+  │◀── FsDirComplete ───────────────────────│
+```
+
+**Protocol cost:**
+| Scenario | RTTs |
+|----------|------|
+| Warm no-op | 1 |
+| Cold copy | 2 |
+| Incremental | 2 |
+
+## Modules
+
+### `manifest`
+
+`FsManifest::build(root, excludes)` — parallel BLAKE3 walk of a directory tree
+using `walkdir` + Rayon. Returns a sorted manifest of all regular files with
+their BLAKE3 hash, size, and mtime.
+
+```rust
+let excludes = GlobSet::empty();
+let manifest = FsManifest::build(Path::new("./data"), &excludes)?;
+for entry in &manifest.entries {
+    println!("{} {} bytes", entry.rel_path, entry.size);
+}
+```
+
+### `differ`
+
+`diff_manifests(local, remote)` — compare two manifest entry lists by path and
+BLAKE3 hash. Returns `Vec<FileDiff>` with `Added`, `Modified`, `Removed`, and
+`Unchanged` variants.
+
+### `delta`
+
+CDC chunking, transfer-need computation, and file reconstruction.
+
+- `chunk_file_for_request(data)` — chunk bytes and return `Vec<FsChunkHash>`
+- `compute_needed_indices(server_existing, client_request)` — which chunks the
+  server lacks (by xxHash3-64)
+- `build_chunk_data(data, chunks, needed_indices)` — pack literal data for
+  transfer; zstd-compresses each chunk, falls back to raw if compressed ≥ raw
+- `reconstruct_file(server_file, client_order, server_map, literals, expected_blake3)` —
+  rebuild the target file from server-local chunks + received literals; verifies
+  BLAKE3 on output
+
+### `session`
+
+`FsSyncClient` and `FsSyncServer` — async protocol orchestration over a
+`SyncPeer` (TCP or QUIC).
+
+```rust
+// Client
+let stats = FsSyncClient::new(peer, local_root, excludes, delete)
+    .run()
+    .await?;
+println!("{} added, {} modified, {} removed", stats.files_added, stats.files_modified, stats.files_removed);
+
+// Server (called per accepted connection)
+FsSyncServer::new(peer, serve_root, excludes, allow_delete)
+    .handle()
+    .await?;
+```
+
+## Performance
+
+- **Manifest build:** parallelised with Rayon — scales with core count
+- **Chunk transfer:** W=16 pipelined send/recv via `SyncPeer::into_pipe_halves()`
+- **CDC boundaries:** FastCDC, immune to insertion staircase that breaks rsync's rolling checksum
+- **Compression:** per-chunk zstd; falls back to raw for already-compressed data
+
+## License
+
+MIT — see repository root.