osobhandClaude Sonnet 4.6
260e15f5b6
8-crate pure-Rust workspace for revision-aware HDF5 sync. ## Crates - clawhdf5-onion: ClawOnion VFD — page-level versioned HDF5 storage, binary format, writer/reader, branch DAG, GC, snapshots, provenance - clawsync-core: BLAKE3, xxHash3, FastCDC (+ SIMD NEON), zstd/lz4 - clawsync-onion: IBLT sketch, Merkle tree differ, packet differ/merger, ClawSyncManifest, SyncSelector - clawsync-hdf5: dataset-level manifest, differ, patcher, wire payload reconstruction (apply_received_payloads) - clawsync-transport: TCP, QUIC (quinn 0.11/TLS 1.3), SyncPeer abstraction, length-prefixed rkyv wire protocol (21 SyncMessage variants) - clawsync-agent: OnionMemory, SyncScheduler, TcpSyncBackend, PeerCapabilities negotiation - clawsync-fs: CDC-based delta sync for any file type; FsSyncClient/Server, W=16 pipelining, atomic writes - clawsync-cli: push/pull/serve/hdf5-sync/serve-hdf5/sync/serve-fs + all local management commands; --quic on all network commands ## Key features - IBLT pre-flight: O(revision count) vs rsync's O(file size) - W=16 sliding-window push: 13–15x speedup over stop-and-wait at WAN RTT - Dataset-granular HDF5 sync: only modified datasets transferred - CDC delta for any file type: insertion-stable chunk boundaries - Full revision DAG: branch, merge, rollback, export, snapshot, GC - QUIC transport: TLS 1.3, per-message streams via quinn 0.11 ## Tests ~573 passing (default features); ~589 with --features simd-cdc ## Performance (Apple Silicon) - Reconstruct rev=100: 68 µs (target ≤ 1 ms) - BLAKE3 Rayon 1 MB: 10.3 GiB/s (target ≥ 5 GB/s) - GC 500 revisions: 20.6 µs (target ≤ 2 s) - W=16 vs W=1 at 5 ms RTT: 14.8x speedup - No-op pre-flight at 16 MB: 4 ms vs rsync 35 ms (7.8x) Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
38 lines
1.0 KiB
Markdown
38 lines
1.0 KiB
Markdown
# clawsync-hdf5
|
|
|
|
Dataset-granular HDF5 manifest, diff, and patch for ClawSync.
|
|
|
|
## Overview
|
|
|
|
`clawsync-hdf5` provides HDF5-aware diffing that operates above the onion page
|
|
layer. Instead of treating an HDF5 file as an opaque byte stream, it understands
|
|
individual datasets and generates per-dataset patches.
|
|
|
|
This allows ClawSync to report *which datasets changed* rather than just *which
|
|
pages changed*, and to apply patches with dataset-level granularity.
|
|
|
|
## Modules
|
|
|
|
### `manifest`
|
|
|
|
`DatasetManifest` — a compact summary of every dataset in an HDF5 file:
|
|
name, shape, dtype, chunk layout, and BLAKE3 hash of the dataset bytes.
|
|
|
|
```rust
|
|
let manifest = DatasetManifest::from_file(&h5_path)?;
|
|
```
|
|
|
|
### `differ`
|
|
|
|
`diff_manifests(local, remote) -> Vec<DatasetPatch>` — compare two manifests
|
|
and return patches only for datasets that differ.
|
|
|
|
### `patcher`
|
|
|
|
`apply_patches(h5_path, patches)` — apply a `Vec<DatasetPatch>` to an HDF5
|
|
file on disk, updating only the changed datasets.
|
|
|
|
## License
|
|
|
|
MIT — see repository root.
|