feat: clawsync diff command + revision_page_entries API + CHANGELOG

diff command:
  clawsync diff <file.h5> <rev1> [rev2]
  - compares page-level metadata between two revisions (or rev1 vs HEAD)
  - ~modified / +added / -removed output per page
  - --porcelain: tab-separated machine-readable output
  - detection via data_offset equality (append-only sidecar invariant)

clawhdf5-onion:
  OnionFile::revision_page_entries(rev) -> &[PageTableEntry]
  - zero-copy access to page table entries for diff and analytics

4 subprocess tests: identical/changed/head-default/porcelain

CHANGELOG.md: Keep-a-Changelog v0.1.0 entry covering all 8 crates,
all CLI commands, and benchmark numbers.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

CHANGELOG.md

@@ -0,0 +1,102 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
+Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+---
+
+## [0.1.0] — 2026-04-06
+
+Initial release of the ClawSync workspace.
+
+### Added
+
+#### Core primitives (`clawsync-core`)
+- BLAKE3 + xxHash3-64 checksums with Rayon-parallel batch hashing
+- FastCDC content-defined chunking (variable-length, insertion-stable)
+- SIMD-accelerated CDC via Gear hash (NEON/SSE2; opt-in `simd-cdc` feature)
+- zstd and lz4 streaming compression wrappers
+- Fixed-block delta encoding (copy/insert operations, apply, size accounting)
+
+#### Wire protocol (`clawsync-transport`)
+- Length-prefixed rkyv framing over TCP, QUIC (quinn/rustls), mmap ring-buffer, and stdin/stdout pipes
+- `SyncMessage` enum with 22 variants (append-only; discriminants stable)
+- `SyncPeer` abstraction unifying TCP / QUIC / Mmap / Stream behind a single send/recv interface
+- `PipeWriteHalf` / `PipeReadHalf` for W=16 split-pipeline push
+- `StreamPeer::from_child` / `from_stdio` for SSH and pipe transports
+- QUIC with self-signed TLS 1.3 (dev) or custom `QuicConfig` (production)
+
+#### ClawOnion VFD (`clawhdf5-onion`)
+- Pure-Rust HDF5 versioning sidecar: diff-based commits, full-state snapshots, branch fork/merge
+- zstd + TDT float compression per page (codec negotiable per-file)
+- GC policies: KeepLastN, KeepTagged, KeepSince, KeepRevisions + epoch-based O(N) gc()
+- Merkle tree over revision layers for O(log N) sync-diff walks
+- Rateless IBLT sketches for sub-linear pre-flight revision set comparison
+- `VersionedFile` high-level API: open/create/commit/branch/merge/snapshot/export
+- Golden file format freeze test (4417-byte canonical representation)
+- `revision_page_entries` for zero-copy page metadata access
+
+#### Sync engine (`clawsync-onion`, `clawsync-hdf5`)
+- IBLT-based push pre-flight: 30× wire savings at N=1K+ revisions vs flat manifest
+- W=16 pipelined push: 13–15× faster than stop-and-wait at 1–5 ms simulated RTT
+- `SyncSelector`: filter by branch, revision ceiling, or "all"
+- HDF5 dataset-granular sync: only modified datasets transferred per file
+- Dataset-level differ, patcher, and manifest with `want_delete` / `allow_delete` guards
+
+#### Agent library (`clawsync-agent`)
+- `SyncBackend` trait: pluggable push/pull transport
+- `TcpSyncBackend`: push/pull over TCP with W=16 IBLT pipeline
+- `QuicSyncBackend`: push/pull over QUIC (TLS 1.3, self-signed for dev)
+- `SshSyncBackend`: push/pull over SSH pipe — spawns `clawsync serve --stdio` on demand, no pre-running daemon required; overridable via `CLAWSYNC_SSH_COMMAND`
+- `OnionMemory`: autonomous flush → sync lifecycle with WAL
+- `SyncScheduler`: debounced background sync with configurable interval
+
+#### General file sync (`clawsync-fs`)
+- CDC-based delta for any file type: only missing FastCDC chunks transferred
+- `FsSyncClient` / `FsSyncServer`: 2-RTT push protocol with W=16 pipelining
+- `FsSyncPullClient` / `FsSyncPullServer`: reverse (server→client) direction
+- `FsManifest`: parallel BLAKE3 walk via rayon; `.clawsync.state` nanosecond-mtime cache
+- `FsDiffer`: Added/Modified/Removed/Unchanged classification
+- `FsDelta`: chunk-level need computation and reconstruction with BLAKE3 integrity check
+- mtime preservation on synced files (±2 s tolerance for FAT/NFS)
+- `--bwlimit <KBPS>` token-bucket rate limiter
+- `--dry-run` / `--verbose` / `--delete` / `--exclude <glob>` options
+- `fs_status()` + `FsStatusReport`: local-only A/M/D change detection vs `.clawsync.state`
+
+#### CLI (`clawsync-cli`)
+- `push` / `pull` — revision sync for `.onion` HDF5 sidecars
+- `serve` — TCP/QUIC server for push/pull; `--stdio` for SSH pipe
+- `sync` / `serve-fs` — general file sync (any type, CDC delta)
+- `pull-fs` — reverse file sync (fetch from remote to local)
+- `hdf5-sync` / `serve-hdf5` — dataset-granular HDF5 sync
+- `watch` — OS FS event watcher (kqueue/inotify) with debounce + auto-reconnect
+- `serve-all` — single-port universal server dispatching all three protocol types
+- `list-revisions`, `rollback`, `gc`, `export-revision`, `snapshot`, `branch` — local onion management
+- `diff <rev1> [<rev2>]` — page-level diff between revisions; `--porcelain` for machine output
+- `status` — local A/M/D report vs `.clawsync.state` cache (no network)
+- `verify` — 1-RTT dry-run diff against remote without transferring data
+- SSH transport on all network commands: `user@host:path` syntax spawns remote binary on demand
+- `--quic` flag for QUIC transport on all network commands
+
+### Performance (Apple Silicon, `cargo bench`)
+
+| Operation | Measured | Target |
+|-----------|----------|--------|
+| Commit 1 page (4 KB) | 5.42 µs | — |
+| Reconstruct rev=100 (no snapshot) | 68 µs | ≤ 1 ms |
+| Reconstruct rev=200 (with snapshot) | 8.1 µs | ≤ 5 ms |
+| BLAKE3 (Rayon, 1 MB) | 10.3 GiB/s | ≥ 5 GB/s |
+| Branch fork | 3.4 µs | ≤ 100 µs |
+| GC 500 revisions | 20.6 µs | ≤ 2 s |
+| WAN push W=16 vs W=1 @ 1 ms RTT | 13.5× faster | — |
+| WAN push W=16 vs W=1 @ 5 ms RTT | 14.8× faster | — |
+| Pre-flight (IBLT) vs rsync at 1 GB | ~4 ms vs ~2 s | 500× faster |
+
+### Notes
+
+- MSRV: Rust 1.85 (Rust 2024 edition)
+- `#![forbid(unsafe_code)]` on all clawsync-* crates
+- `SyncMessage` discriminants are stable and append-only; never insert or reorder variants
+- Publish blocked on upstream `clawhdf5` workspace reaching crates.io; publish order documented in README

crates/clawhdf5-onion/src/reader.rs

@@ -199,6 +199,22 @@ impl OnionFile {
     /// callers that intend to send the data over the network can ship the
     /// compressed bytes directly and let the receiver decompress.
     ///
+    /// Returns the raw [`PageTableEntry`] slice for a revision.
+    ///
+    /// Useful for comparing page metadata across revisions without loading the
+    /// actual page data.  The `data_offset` field uniquely identifies the
+    /// stored content: two entries with the same `data_offset` contain identical
+    /// (compressed) bytes.
+    pub fn revision_page_entries(
+        &self,
+        rev: u64,
+    ) -> Result<&[crate::format::PageTableEntry], OnionError> {
+        self.page_tables
+            .get(rev as usize)
+            .map(|v| v.as_slice())
+            .ok_or(OnionError::RevisionNotFound(rev))
+    }
+
     /// Returns `(h5_offset, compressed_data, codec_byte, orig_size)`.
     pub fn revision_pages_raw(&self, rev: u64) -> Result<Vec<RawPage>, OnionError> {
         let table = self

crates/clawsync-cli/src/main.rs

@@ -195,6 +195,35 @@ enum Commands {
         message: Option<String>,
     },
 
+    /// Show page-level differences between two revisions of a local .onion sidecar.
+    ///
+    /// If REV2 is omitted, compares REV1 against the current HEAD revision.
+    ///
+    /// Output:
+    ///   `~  page N  (changed)`
+    ///   `+  page N  (added)`
+    ///   `-  page N  (removed)`
+    ///
+    /// Pass `--porcelain` for machine-readable tab-separated output.
+    Diff {
+        /// Local HDF5 file path (must have an .onion sidecar).
+        #[arg(value_name = "LOCAL.h5")]
+        h5_path: PathBuf,
+
+        /// First revision to compare from.
+        #[arg(value_name = "REV1")]
+        rev1: u64,
+
+        /// Second revision to compare against (default: HEAD).
+        #[arg(value_name = "REV2")]
+        rev2: Option<u64>,
+
+        /// Machine-readable output: one entry per line, tab-separated
+        /// `<status>\t<page_index>\t<size_bytes>`.
+        #[arg(long)]
+        porcelain: bool,
+    },
+
     /// Sync a local file or directory to a remote path (any file type).
     ///
     /// Uses CDC-based delta transfer: only changed chunks are sent,
@@ -1296,6 +1325,91 @@ async fn cmd_snapshot(h5_path: PathBuf, message: Option<String>) -> Result<()> {
     Ok(())
 }
 
+async fn cmd_diff(h5_path: PathBuf, rev1: u64, rev2: Option<u64>, porcelain: bool) -> Result<()> {
+    use std::collections::HashMap;
+
+    let onion = OnionFile::open(&h5_path)
+        .with_context(|| format!("cannot open {}", h5_path.display()))?;
+
+    let head = onion
+        .list_revisions()
+        .last()
+        .map(|s| s.revision)
+        .with_context(|| "no revisions in sidecar")?;
+    let rev2 = rev2.unwrap_or(head);
+
+    // Build h5_offset → data_offset maps.  The data_offset uniquely identifies
+    // the stored content: same data_offset = same bytes (sidecar is append-only).
+    let pts1 = onion
+        .revision_page_entries(rev1)
+        .with_context(|| format!("cannot read page entries for rev {rev1}"))?;
+    let pts2 = onion
+        .revision_page_entries(rev2)
+        .with_context(|| format!("cannot read page entries for rev {rev2}"))?;
+
+    let map1: HashMap<u64, u64> = pts1.iter().map(|e| (e.h5_offset, e.data_offset)).collect();
+    let map2: HashMap<u64, u64> = pts2.iter().map(|e| (e.h5_offset, e.data_offset)).collect();
+
+    let mut all_h5: Vec<u64> = map1.keys().chain(map2.keys()).copied().collect();
+    all_h5.sort_unstable();
+    all_h5.dedup();
+
+    let page_size = onion.page_size() as u64;
+    let mut n_changed = 0u64;
+    let mut n_added = 0u64;
+    let mut n_removed = 0u64;
+    let mut lines: Vec<String> = Vec::new();
+
+    for &h5_off in &all_h5 {
+        let page_idx = h5_off / page_size;
+        match (map1.get(&h5_off), map2.get(&h5_off)) {
+            (Some(d1), Some(d2)) if d1 == d2 => { /* unchanged */ }
+            (Some(_), Some(_)) => {
+                // Same page location, different data_offset — content changed.
+                n_changed += 1;
+                if porcelain {
+                    lines.push(format!("~\t{page_idx}\t{page_size}"));
+                } else {
+                    lines.push(format!("  ~  page {page_idx}  (changed, {page_size} B)"));
+                }
+            }
+            (None, Some(_)) => {
+                n_added += 1;
+                if porcelain {
+                    lines.push(format!("+\t{page_idx}\t{page_size}"));
+                } else {
+                    lines.push(format!("  +  page {page_idx}  (added, {page_size} B)"));
+                }
+            }
+            (Some(_), None) => {
+                n_removed += 1;
+                if porcelain {
+                    lines.push(format!("-\t{page_idx}", ));
+                } else {
+                    lines.push(format!("  -  page {page_idx}  (removed)"));
+                }
+            }
+            (None, None) => unreachable!(),
+        }
+    }
+
+    if !porcelain {
+        println!("Diff rev {rev1} → rev {rev2}:");
+    }
+    for line in &lines {
+        println!("{line}");
+    }
+    if !porcelain {
+        if lines.is_empty() {
+            println!("No differences.");
+        } else {
+            println!("{n_changed} page(s) changed, {n_added} added, {n_removed} removed.");
+        }
+    }
+
+    Ok(())
+}
+
 async fn cmd_branch_merge(
     h5_path: PathBuf,
     source: String,
@@ -2406,6 +2520,12 @@ fn main() -> Result<()> {
                 .await
             }
             Commands::Snapshot { h5_path, message } => cmd_snapshot(h5_path, message).await,
+            Commands::Diff {
+                h5_path,
+                rev1,
+                rev2,
+                porcelain,
+            } => cmd_diff(h5_path, rev1, rev2, porcelain).await,
             Commands::Sync {
                 local,
                 remote,

crates/clawsync-cli/tests/subprocess.rs

@@ -955,3 +955,135 @@ fn serve_all_mixed_protocol_push_then_fs_sync() {
     let written = await_revisions(&root.path().join("memory.claws"), 3);
     assert_eq!(written, 3, "onion revisions lost after FS sync on same server");
 }
+
+// ─────────────────────────────────────────────────────────────────────────────
+// clawsync diff tests
+// ─────────────────────────────────────────────────────────────────────────────
+
+fn run_diff(h5: &Path, rev1: u64, rev2: Option<u64>) -> (bool, String) {
+    let mut cmd = std::process::Command::new(BIN);
+    cmd.arg("diff").arg(h5).arg(rev1.to_string());
+    if let Some(r2) = rev2 {
+        cmd.arg(r2.to_string());
+    }
+    let out = cmd.output().expect("failed to spawn clawsync diff");
+    (
+        out.status.success(),
+        String::from_utf8_lossy(&out.stdout).into_owned(),
+    )
+}
+
+fn run_diff_porcelain(h5: &Path, rev1: u64, rev2: Option<u64>) -> (bool, String) {
+    let mut cmd = std::process::Command::new(BIN);
+    cmd.arg("diff").arg(h5).arg(rev1.to_string());
+    if let Some(r2) = rev2 {
+        cmd.arg(r2.to_string());
+    }
+    cmd.arg("--porcelain");
+    let out = cmd.output().expect("failed to spawn clawsync diff --porcelain");
+    (
+        out.status.success(),
+        String::from_utf8_lossy(&out.stdout).into_owned(),
+    )
+}
+
+/// Identical revisions → "No differences."
+#[test]
+fn diff_identical_revisions_shows_no_changes() {
+    let (_dir, h5, _) = make_versioned_h5(3);
+
+    // Rev 0 compared against itself.
+    let (ok, out) = run_diff(&h5, 0, Some(0));
+    assert!(ok, "diff failed: {out}");
+    assert!(out.contains("No differences."), "expected no differences; got: {out}");
+}
+
+/// When a page changes between revisions, diff reports it as modified.
+#[test]
+fn diff_changed_page_shows_modified() {
+    let dir = TempDir::new().unwrap();
+    let h5 = dir.path().join("data.h5");
+    std::fs::write(&h5, H5_MAGIC).unwrap();
+    let mut onion = OnionFile::create(&h5, 4096).unwrap();
+
+    // Rev 0: page 0 = all 0xAA.
+    {
+        let mut s = onion.begin_session(None).unwrap();
+        s.record_page(0, &vec![0xAAu8; 4096]);
+        onion.commit_session(s, None).unwrap();
+    }
+    // Rev 1: page 0 = all 0xBB (changed).
+    {
+        let mut s = onion.begin_session(None).unwrap();
+        s.record_page(0, &vec![0xBBu8; 4096]);
+        onion.commit_session(s, None).unwrap();
+    }
+    onion.flush().unwrap();
+
+    let (ok, out) = run_diff(&h5, 0, Some(1));
+    assert!(ok, "diff failed: {out}");
+    assert!(
+        out.contains("changed"),
+        "expected 'changed' in diff output; got: {out}"
+    );
+    assert!(
+        out.contains("1 page(s) changed"),
+        "expected summary; got: {out}"
+    );
+}
+
+/// Omitting rev2 diffs against HEAD.
+#[test]
+fn diff_default_rev2_is_head() {
+    let (_dir, h5, _) = make_versioned_h5(5);
+
+    // Rev 0 vs HEAD (rev 4): make_versioned_h5 writes the same page index with
+    // different fill bytes each revision, so page 0 will have changed.
+    let (ok, out) = run_diff(&h5, 0, None);
+    assert!(ok, "diff failed: {out}");
+    // Either changed or the summary line must appear.
+    assert!(
+        out.contains("Diff rev 0 →"),
+        "expected diff header; got: {out}"
+    );
+}
+
+/// --porcelain outputs tab-separated lines without human text.
+#[test]
+fn diff_porcelain_output_format() {
+    let dir = TempDir::new().unwrap();
+    let h5 = dir.path().join("data.h5");
+    std::fs::write(&h5, H5_MAGIC).unwrap();
+    let mut onion = OnionFile::create(&h5, 4096).unwrap();
+
+    // Rev 0: page 0.
+    {
+        let mut s = onion.begin_session(None).unwrap();
+        s.record_page(0, &vec![0x11u8; 4096]);
+        onion.commit_session(s, None).unwrap();
+    }
+    // Rev 1: page 0 changed, page 1 added.
+    {
+        let mut s = onion.begin_session(None).unwrap();
+        s.record_page(0, &vec![0x22u8; 4096]);
+        s.record_page(4096, &vec![0x33u8; 4096]);
+        onion.commit_session(s, None).unwrap();
+    }
+    onion.flush().unwrap();
+
+    let (ok, out) = run_diff_porcelain(&h5, 0, Some(1));
+    assert!(ok, "porcelain diff failed: {out}");
+
+    let lines: Vec<&str> = out.lines().collect();
+    // Each line must start with ~, +, or -.
+    for line in &lines {
+        let ch = line.chars().next().expect("empty line");
+        assert!(
+            matches!(ch, '~' | '+' | '-'),
+            "unexpected porcelain line: {line}"
+        );
+        // Must have at least 2 tab-separated fields.
+        assert!(line.contains('\t'), "missing tab in porcelain line: {line}");
+    }
+    assert!(!lines.is_empty(), "expected at least one diff line");
+}