osobhandClaude Sonnet 4.6
3d524e2d63
Doc fixes (20 warnings → 1 benign name-collision warning): - clawsync-transport/quic.rs: QuicConfig::with_cert doesn't exist → backtick - clawsync-onion/iblt.rs: bare `insert` link → backtick - clawsync-core/lib.rs: simd_cdc is feature-gated → backtick - clawhdf5-onion/annotation.rs: RevisionEntry/BranchEntry are in external clawhdf5-format crate, not re-exported → backtick - clawhdf5-onion/gc.rs: compact_dead_epoch_revisions is private; flush links broken → backtick - clawhdf5-onion/provenance.rs: RevisionEntry from external crate → backtick - clawhdf5-onion/reader.rs: reconstruct_revision/revision_pages → Self:: prefix - clawhdf5-onion/writer.rs: REV_FLAG_SNAPSHOT → crate::format:: path; reconstruct_revision → Self:: prefix Remaining warning is `format` module/macro name collision — not a broken link. clawsync-fs crate metadata: - Add readme, keywords, categories to Cargo.toml - Write README.md (CDC protocol diagram, module overview, usage examples) Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
108 lines
4.0 KiB
Markdown
108 lines
4.0 KiB
Markdown
# clawsync-fs
|
|
|
|
CDC-based delta sync for any file type — the general-purpose sync engine behind
|
|
`clawsync sync` and `clawsync serve-fs`.
|
|
|
|
## Overview
|
|
|
|
`clawsync-fs` closes the gap between ClawSync's HDF5-specific revision sync and
|
|
rsync's general-purpose file sync. It handles arbitrary file types and directory
|
|
trees using **Content-Defined Chunking** (FastCDC) so chunk boundaries are
|
|
insertion-stable: a 1-byte prefix insertion does not invalidate the rest of the
|
|
file's chunks.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
FsSyncClient FsSyncServer
|
|
│ │
|
|
├── FsManifest::build (rayon parallel) │
|
|
│ BLAKE3 every file in local tree │
|
|
│ │
|
|
├─── FsDirManifest ──────────────────────▶│
|
|
│ │ FsManifest::build (server tree)
|
|
│ │ diff_manifests → Added/Modified/Removed
|
|
│◀── FsDirNeed ───────────────────────────│
|
|
│ (needed_files with server chunks, │
|
|
│ to_delete if allow_delete) │
|
|
│ │
|
|
│ [W=16 pipelined — into_pipe_halves()] │
|
|
├─── FsCdcData (/path, chunk_order, ─────▶│ reconstruct_file
|
|
│ literal_chunks) │ atomic write (.tmp → rename)
|
|
│◀── FsFileAck ───────────────────────────│
|
|
│ (repeat for each modified/added │
|
|
│ file in any order) │
|
|
│◀── FsDirComplete ───────────────────────│
|
|
```
|
|
|
|
**Protocol cost:**
|
|
| Scenario | RTTs |
|
|
|----------|------|
|
|
| Warm no-op | 1 |
|
|
| Cold copy | 2 |
|
|
| Incremental | 2 |
|
|
|
|
## Modules
|
|
|
|
### `manifest`
|
|
|
|
`FsManifest::build(root, excludes)` — parallel BLAKE3 walk of a directory tree
|
|
using `walkdir` + Rayon. Returns a sorted manifest of all regular files with
|
|
their BLAKE3 hash, size, and mtime.
|
|
|
|
```rust
|
|
let excludes = GlobSet::empty();
|
|
let manifest = FsManifest::build(Path::new("./data"), &excludes)?;
|
|
for entry in &manifest.entries {
|
|
println!("{} {} bytes", entry.rel_path, entry.size);
|
|
}
|
|
```
|
|
|
|
### `differ`
|
|
|
|
`diff_manifests(local, remote)` — compare two manifest entry lists by path and
|
|
BLAKE3 hash. Returns `Vec<FileDiff>` with `Added`, `Modified`, `Removed`, and
|
|
`Unchanged` variants.
|
|
|
|
### `delta`
|
|
|
|
CDC chunking, transfer-need computation, and file reconstruction.
|
|
|
|
- `chunk_file_for_request(data)` — chunk bytes and return `Vec<FsChunkHash>`
|
|
- `compute_needed_indices(server_existing, client_request)` — which chunks the
|
|
server lacks (by xxHash3-64)
|
|
- `build_chunk_data(data, chunks, needed_indices)` — pack literal data for
|
|
transfer; zstd-compresses each chunk, falls back to raw if compressed ≥ raw
|
|
- `reconstruct_file(server_file, client_order, server_map, literals, expected_blake3)` —
|
|
rebuild the target file from server-local chunks + received literals; verifies
|
|
BLAKE3 on output
|
|
|
|
### `session`
|
|
|
|
`FsSyncClient` and `FsSyncServer` — async protocol orchestration over a
|
|
`SyncPeer` (TCP or QUIC).
|
|
|
|
```rust
|
|
// Client
|
|
let stats = FsSyncClient::new(peer, local_root, excludes, delete)
|
|
.run()
|
|
.await?;
|
|
println!("{} added, {} modified, {} removed", stats.files_added, stats.files_modified, stats.files_removed);
|
|
|
|
// Server (called per accepted connection)
|
|
FsSyncServer::new(peer, serve_root, excludes, allow_delete)
|
|
.handle()
|
|
.await?;
|
|
```
|
|
|
|
## Performance
|
|
|
|
- **Manifest build:** parallelised with Rayon — scales with core count
|
|
- **Chunk transfer:** W=16 pipelined send/recv via `SyncPeer::into_pipe_halves()`
|
|
- **CDC boundaries:** FastCDC, immune to insertion staircase that breaks rsync's rolling checksum
|
|
- **Compression:** per-chunk zstd; falls back to raw for already-compressed data
|
|
|
|
## License
|
|
|
|
MIT — see repository root.
|