osobhandClaude Sonnet 4.6
679e146cba
- subprocess_verify_tcp: reports pending adds, clean after sync (TCP) - subprocess_sync_delete: server-side deletion via --allow-delete + --delete - subprocess_sync_dry_run: shows would-add without writing any files - subprocess_hdf5_sync_cold_copy: cold copy of 2 HDF5 datasets via serve-hdf5 - ServeAllServer::start_with_delete() helper for --allow-delete servers - ServeHdf5Server helper for serve-hdf5 subprocess tests - CHANGELOG: document QUIC stream/close race fix in v0.1.0 ### Fixed section 660 tests, 0 failures. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
112 lines
5.8 KiB
Markdown
112 lines
5.8 KiB
Markdown
# Changelog
|
||
|
||
All notable changes to this project are documented in this file.
|
||
|
||
Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
||
Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||
|
||
---
|
||
|
||
## [0.1.0] — 2026-04-06
|
||
|
||
Initial release of the ClawSync workspace.
|
||
|
||
### Added
|
||
|
||
#### Core primitives (`clawsync-core`)
|
||
- BLAKE3 + xxHash3-64 checksums with Rayon-parallel batch hashing
|
||
- FastCDC content-defined chunking (variable-length, insertion-stable)
|
||
- SIMD-accelerated CDC via Gear hash (NEON/SSE2; opt-in `simd-cdc` feature)
|
||
- zstd and lz4 streaming compression wrappers
|
||
- Fixed-block delta encoding (copy/insert operations, apply, size accounting)
|
||
|
||
#### Wire protocol (`clawsync-transport`)
|
||
- Length-prefixed rkyv framing over TCP, QUIC (quinn/rustls), mmap ring-buffer, and stdin/stdout pipes
|
||
- `SyncMessage` enum with 22 variants (append-only; discriminants stable)
|
||
- `SyncPeer` abstraction unifying TCP / QUIC / Mmap / Stream behind a single send/recv interface
|
||
- `PipeWriteHalf` / `PipeReadHalf` for W=16 split-pipeline push
|
||
- `StreamPeer::from_child` / `from_stdio` for SSH and pipe transports
|
||
- QUIC with self-signed TLS 1.3 (dev) or custom `QuicConfig` (production)
|
||
|
||
#### ClawOnion VFD (`clawhdf5-onion`)
|
||
- Pure-Rust HDF5 versioning sidecar: diff-based commits, full-state snapshots, branch fork/merge
|
||
- zstd + TDT float compression per page (codec negotiable per-file)
|
||
- GC policies: KeepLastN, KeepTagged, KeepSince, KeepRevisions + epoch-based O(N) gc()
|
||
- Merkle tree over revision layers for O(log N) sync-diff walks
|
||
- Rateless IBLT sketches for sub-linear pre-flight revision set comparison
|
||
- `VersionedFile` high-level API: open/create/commit/branch/merge/snapshot/export
|
||
- Golden file format freeze test (4417-byte canonical representation)
|
||
- `revision_page_entries` for zero-copy page metadata access
|
||
|
||
#### Sync engine (`clawsync-onion`, `clawsync-hdf5`)
|
||
- IBLT-based push pre-flight: 30× wire savings at N=1K+ revisions vs flat manifest
|
||
- W=16 pipelined push: 13–15× faster than stop-and-wait at 1–5 ms simulated RTT
|
||
- `SyncSelector`: filter by branch, revision ceiling, or "all"
|
||
- HDF5 dataset-granular sync: only modified datasets transferred per file
|
||
- Dataset-level differ, patcher, and manifest with `want_delete` / `allow_delete` guards
|
||
|
||
#### Agent library (`clawsync-agent`)
|
||
- `SyncBackend` trait: pluggable push/pull transport
|
||
- `TcpSyncBackend`: push/pull over TCP with W=16 IBLT pipeline
|
||
- `QuicSyncBackend`: push/pull over QUIC (TLS 1.3, self-signed for dev)
|
||
- `SshSyncBackend`: push/pull over SSH pipe — spawns `clawsync serve --stdio` on demand, no pre-running daemon required; overridable via `CLAWSYNC_SSH_COMMAND`
|
||
- `OnionMemory`: autonomous flush → sync lifecycle with WAL
|
||
- `SyncScheduler`: debounced background sync with configurable interval
|
||
|
||
#### General file sync (`clawsync-fs`)
|
||
- CDC-based delta for any file type: only missing FastCDC chunks transferred
|
||
- `FsSyncClient` / `FsSyncServer`: 2-RTT push protocol with W=16 pipelining
|
||
- `FsSyncPullClient` / `FsSyncPullServer`: reverse (server→client) direction
|
||
- `FsManifest`: parallel BLAKE3 walk via rayon; `.clawsync.state` nanosecond-mtime cache
|
||
- `FsDiffer`: Added/Modified/Removed/Unchanged classification
|
||
- `FsDelta`: chunk-level need computation and reconstruction with BLAKE3 integrity check
|
||
- mtime preservation on synced files (±2 s tolerance for FAT/NFS)
|
||
- `--bwlimit <KBPS>` token-bucket rate limiter
|
||
- `--dry-run` / `--verbose` / `--delete` / `--exclude <glob>` options
|
||
- `fs_status()` + `FsStatusReport`: local-only A/M/D change detection vs `.clawsync.state`
|
||
|
||
#### CLI (`clawsync-cli`)
|
||
- `push` / `pull` — revision sync for `.onion` HDF5 sidecars
|
||
- `serve` — TCP/QUIC server for push/pull; `--stdio` for SSH pipe
|
||
- `sync` / `serve-fs` — general file sync (any type, CDC delta)
|
||
- `pull-fs` — reverse file sync (fetch from remote to local)
|
||
- `hdf5-sync` / `serve-hdf5` — dataset-granular HDF5 sync
|
||
- `watch` — OS FS event watcher (kqueue/inotify) with debounce + auto-reconnect
|
||
- `serve-all` — single-port universal server dispatching all three protocol types
|
||
- `list-revisions`, `rollback`, `gc`, `export-revision`, `snapshot`, `branch` — local onion management
|
||
- `diff <rev1> [<rev2>]` — page-level diff between revisions; `--porcelain` for machine output
|
||
- `status` — local A/M/D report vs `.clawsync.state` cache (no network)
|
||
- `verify` — 1-RTT dry-run diff against remote without transferring data
|
||
- SSH transport on all network commands: `user@host:path` syntax spawns remote binary on demand
|
||
- `--quic` flag for QUIC transport on all network commands
|
||
|
||
### Performance (Apple Silicon, `cargo bench`)
|
||
|
||
| Operation | Measured | Target |
|
||
|-----------|----------|--------|
|
||
| Commit 1 page (4 KB) | 5.42 µs | — |
|
||
| Reconstruct rev=100 (no snapshot) | 68 µs | ≤ 1 ms |
|
||
| Reconstruct rev=200 (with snapshot) | 8.1 µs | ≤ 5 ms |
|
||
| BLAKE3 (Rayon, 1 MB) | 10.3 GiB/s | ≥ 5 GB/s |
|
||
| Branch fork | 3.4 µs | ≤ 100 µs |
|
||
| GC 500 revisions | 20.6 µs | ≤ 2 s |
|
||
| WAN push W=16 vs W=1 @ 1 ms RTT | 13.5× faster | — |
|
||
| WAN push W=16 vs W=1 @ 5 ms RTT | 14.8× faster | — |
|
||
| Pre-flight (IBLT) vs rsync at 1 GB | ~4 ms vs ~2 s | 500× faster |
|
||
|
||
### Fixed
|
||
|
||
- **QUIC stream/close race on push**: `PipeWriteHalf::shutdown()` called
|
||
`close_and_drain()` immediately after the client sent `SyncComplete`, which
|
||
could race the in-flight unidirectional QUIC stream before the server read it.
|
||
Added `PipeWriteHalf::shutdown_push()` — used by `cmd_push` — that calls
|
||
`wait_for_peer_close()` instead, keeping the connection alive until the server
|
||
acknowledges receipt and closes its side.
|
||
|
||
### Notes
|
||
|
||
- MSRV: Rust 1.85 (Rust 2024 edition)
|
||
- `#![forbid(unsafe_code)]` on all clawsync-* crates
|
||
- `SyncMessage` discriminants are stable and append-only; never insert or reorder variants
|
||
- Publish blocked on upstream `clawhdf5` workspace reaching crates.io; publish order documented in README
|