osobhandClaude Sonnet 4.6
260e15f5b6
8-crate pure-Rust workspace for revision-aware HDF5 sync. ## Crates - clawhdf5-onion: ClawOnion VFD — page-level versioned HDF5 storage, binary format, writer/reader, branch DAG, GC, snapshots, provenance - clawsync-core: BLAKE3, xxHash3, FastCDC (+ SIMD NEON), zstd/lz4 - clawsync-onion: IBLT sketch, Merkle tree differ, packet differ/merger, ClawSyncManifest, SyncSelector - clawsync-hdf5: dataset-level manifest, differ, patcher, wire payload reconstruction (apply_received_payloads) - clawsync-transport: TCP, QUIC (quinn 0.11/TLS 1.3), SyncPeer abstraction, length-prefixed rkyv wire protocol (21 SyncMessage variants) - clawsync-agent: OnionMemory, SyncScheduler, TcpSyncBackend, PeerCapabilities negotiation - clawsync-fs: CDC-based delta sync for any file type; FsSyncClient/Server, W=16 pipelining, atomic writes - clawsync-cli: push/pull/serve/hdf5-sync/serve-hdf5/sync/serve-fs + all local management commands; --quic on all network commands ## Key features - IBLT pre-flight: O(revision count) vs rsync's O(file size) - W=16 sliding-window push: 13–15x speedup over stop-and-wait at WAN RTT - Dataset-granular HDF5 sync: only modified datasets transferred - CDC delta for any file type: insertion-stable chunk boundaries - Full revision DAG: branch, merge, rollback, export, snapshot, GC - QUIC transport: TLS 1.3, per-message streams via quinn 0.11 ## Tests ~573 passing (default features); ~589 with --features simd-cdc ## Performance (Apple Silicon) - Reconstruct rev=100: 68 µs (target ≤ 1 ms) - BLAKE3 Rayon 1 MB: 10.3 GiB/s (target ≥ 5 GB/s) - GC 500 revisions: 20.6 µs (target ≤ 2 s) - W=16 vs W=1 at 5 ms RTT: 14.8x speedup - No-op pre-flight at 16 MB: 4 ms vs rsync 35 ms (7.8x) Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
77 lines
2.5 KiB
Markdown
77 lines
2.5 KiB
Markdown
# clawhdf5-onion
|
||
|
||
Pure-Rust ClawOnion VFD — revision-layered HDF5 versioning with DAG branching.
|
||
|
||
## Overview
|
||
|
||
`clawhdf5-onion` implements the **ClawOnion Virtual File Driver**: a zero-C-dependency
|
||
versioning layer for HDF5 files that stores write sessions as page-level diffs in a
|
||
`.onion` sidecar file. Any historical revision can be reconstructed without modifying
|
||
the original `.h5` file.
|
||
|
||
The metaphor: the original file state is the core; each write session adds a new skin
|
||
of page changes on top.
|
||
|
||
## Features
|
||
|
||
- **Revision history** — every write session becomes an immutable revision; open any past state read-only
|
||
- **DAG branching** — fork, write, and merge branches with three merge strategies (last-write-wins, dataset-level resolver, three-way)
|
||
- **BLAKE3 provenance** — per-revision content hashing for tamper detection
|
||
- **Per-page compression** — zstd or lz4 with configurable codec per session
|
||
- **Snapshot checkpoints** — bounds reconstruction cost at deep revision history
|
||
- **GC policies** — keep-last-N, keep-tagged, keep-since to reclaim space
|
||
- **No C dependencies** — pure Rust
|
||
|
||
## Usage
|
||
|
||
```rust
|
||
use clawhdf5_onion::writer::OnionFile;
|
||
|
||
// Create a new versioned file pair (.h5 + .h5.onion)
|
||
let mut onion = OnionFile::create("data.h5", 4096)?;
|
||
|
||
// Write a session
|
||
let mut session = onion.begin_session(None)?;
|
||
session.record_page(0, &page_bytes);
|
||
onion.commit_session(session, Some("initial import"))?;
|
||
|
||
// Open a historical revision
|
||
let bytes_at_rev0 = onion.open_revision(0)?;
|
||
```
|
||
|
||
The `VersionedFile` wrapper integrates with `clawhdf5::File` for high-level access:
|
||
|
||
```rust
|
||
use clawhdf5_onion::versioned_file::VersionedFile;
|
||
|
||
let vf = VersionedFile::open("data.h5")?;
|
||
// ... use the clawhdf5 File API on the HEAD revision
|
||
let historical = clawhdf5_onion::open_at_revision("data.h5", 3)?;
|
||
```
|
||
|
||
## Format
|
||
|
||
The `.onion` sidecar uses a fixed binary layout (ClawOnion v1):
|
||
|
||
| Section | Size |
|
||
|---|---|
|
||
| `OnionHeader` | 128 bytes |
|
||
| `RevisionIndex` | 112 bytes × N revisions |
|
||
| `BranchManifest` | 40 bytes × B branches |
|
||
| `PageTable` entries | 32 bytes × P pages |
|
||
| `PageData` | compressed page bytes |
|
||
| `AnnotationHeap` | length-prefixed UTF-8 strings |
|
||
|
||
## Feature Flags
|
||
|
||
| Feature | Default | Description |
|
||
|---|---|---|
|
||
| `provenance` | on | BLAKE3 per-revision page hashing |
|
||
| `compress` | on | per-page zstd/lz4 compression |
|
||
| `parallel` | off | Rayon parallel page hashing |
|
||
| `async` | off | Tokio async flush |
|
||
|
||
## License
|
||
|
||
MIT — see repository root.
|