58c5bc341be7566374b46f8d319f6d7f1059071e
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar Sobh
|
58c5bc341b |
GetRef: transparent ref-forwarding on local miss
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 10s
Cross-runner cache silos (tank + architect measured on 2026-07-13): same fingerprint, same rustc, but each runner's daemon only knows about the refs its own runner uploaded. Every runner that lands on a peer that isn't tank re-uploads a duplicate blob. Fix: on `GetRef` miss the daemon fans out to alive gossip peers via a strict-local `GetRefLocal` variant, and the FIRST peer that has the ref triggers a transparent pull — chunks + manifest into the local blob store, then `PutRef` locally — before returning the value to the caller. Subsequent lookups are pure-local hits. * `Method::GetRefLocal = 0x14` — new wire method, identical shape to GetRef but the peer MUST NOT recurse. Loop prevention: our forwarding only calls `GetRefLocal` on peers, so chain depth is always 1. * `RpcRouter::with_outbound_client(Arc<QuicClient>)` — dependency injection point for the forwarding dial path. `None` disables forwarding entirely (GetRef becomes GetRefLocal-equivalent). * `RpcRouter::forward_get_ref(key)` — concurrent peer probes via `JoinSet`, 3s timeout per dial, first successful pull wins, remaining tasks aborted. * `pull_blob_locally` — walks manifest, fetches only chunks the local store lacks (`has_chunk`), commits via `put_manifest_verified`. Bounded memory: one 4 MiB chunk at a time. * `ClusterServices::start` loads NodeIdentity twice — server takes ownership; outbound client gets its own copy for TLS presentation on peer dials. Wires the outbound client into the router when TLS material is available. * `call_get_ref_local(conn, key)` client helper (used by daemon forwarding + available to any RPC consumer that wants the no-recursion semantics). +3 tests in `rpc/tests_forwarding.rs`: - Local hit works without forwarding; local miss with no peers returns None. Guards the base cases. - GetRefLocal never forwards even when outbound is configured (no peers reachable → miss returns None immediately, no attempted fan-out). - Method byte 0x14 encoding is stable across releases. Full end-to-end forwarding is exercised in the pilot deploy: two daemons on the fleet-CA, tank populates a ref, architect's runner GetRef → tank forwards → architect pulls → HIT locally next time. 264 tests pass (baseline +3). Pre-existing macOS failure unchanged. |
||
|
Omar Sobh
|
846ecffe10 |
Pi deploy follow-ups: XDG default_path + parallel restore
Two fixes surfaced by the vision-02 Pi 5 measurement: ## XDG default_path `Manifest::default_path` was hardcoded to `/var/lib/claw-store/projects.toml`. That path is read-only under the user-mode systemd unit's `ProtectSystem=strict`, and creating it needs root — awful for a runner install. Precedence, matching XDG Base Directory: 1. `$XDG_STATE_HOME/claw-store/projects.toml` 2. `$HOME/.local/state/claw-store/projects.toml` 3. `/var/lib/claw-store/projects.toml` (system fallback) User-mode installs now write in $HOME by default; system installs (root, no HOME set) still land in /var/lib. +1 test: `default_path_honours_xdg_state_home` — covers all three precedence branches. Env mutation is process-global so the test saves + restores. ## Parallel restore on cache HIT Pi restore of 947 MiB via `BlobGetStream` took ~18s (~53 MiB/s) single-stream. Per-stream throughput ceilings on the connection type cap sequential fetches; parallel chunk fetches stack their contributions. - New `call_blob_get_parallel(conn, blob_id, concurrency) -> Option<Vec<u8>>` in `rpc/client.rs`. `JoinSet` + `Semaphore`, reassembles by chunk index at manifest-known offsets so out-of-order arrival is fine. - `claw-cargo build --parallel-restore N` (default 8). `N <= 1` falls through to `BlobGetStream` for parity. - Memory: `total_size + 4 MiB × in-flight` — dominated by the reassembly buffer, not the fanout. +1 test: `parallel_blob_get_reassembles_multi_chunk_blob_byte_equal` covers roundtrip byte-equality vs BlobGetStream, tail-chunk offset, concurrency=1 correctness, and NotFound → None. 259 tests pass (+2). Pre-existing macOS failure unchanged. |
||
|
Omar Sobh
|
54e9da4d62 |
Phase 5k: parallel-fanout chunk transfer for prewarm
Pilot 2026-07-12 measured 109 MiB/s on the sequential prewarm path — ~11% of a 10G fabric. `quinn::Connection` is cheap-Clone (internal Arc), so we can run the has→get→put pipeline per chunk in concurrent tasks under a bounded semaphore. - `prewarm_missing_chunks_between_parallel(up, down, id, concurrency)` in `rpc/client.rs`. `concurrency <= 1` degrades to the sequential path (kept for diagnostic parity). - `claw-cargo prewarm --parallel N` (default 8). Ignored with `--buffered`. Memory ceiling: 4 MiB × in-flight = 32 MiB @ 8, 128 MiB @ 32. - Uses `tokio::task::JoinSet` + `Arc<Semaphore>`; permit held for the whole per-chunk pipeline so we never over-commit. - Retry pass on `put_manifest` mismatch stays sequential — small, correctness-critical. - Errors: JoinSet drains completely + returns first task error so a mid-fanout failure doesn't leave zombie tasks. +1 test: `end_to_end_parallel_prewarm_copies_chunks_and_matches_sequential` runs 5-chunk payload with concurrency=3, verifies byte-equal restore, then reruns with concurrency=8 → 0 uploads (has_chunk dedup), then concurrency=0 → 0 uploads (sequential fallback path). 254 tests pass (+1 from previous). Pre-existing macOS failure unchanged. |
||
|
Omar Sobh
|
6fb16286dd |
Phase 5h: streaming chunk-level prewarm
Bounded-memory cross-peer prewarm: instead of buffering the whole blob in RAM (previous 5f path), iterate the upstream manifest chunk by chunk, ask downstream `HasChunk`, stream missing chunks one at a time. Memory ceiling is 1 chunk (4 MiB) regardless of blob size — a 5 GiB target dir no longer needs 5 GiB of mediator RAM. * rpc/client.rs: new `prewarm_missing_chunks_between(upstream, downstream, blob_id)` helper. Returns `(uploaded, total)` — the difference is the dedup save. Retries once if the downstream `PutManifest` reports missing chunks after our push (guards a narrow eviction race); a second failure surfaces as `Err`. * claw_cargo.rs: `prewarm` now streams by default; new `--buffered` flag for the old whole-blob path (kept for diagnostic comparability during rollout). Human-readable output shows the mode + dedup count. +2 tests: - cold downstream: 3-chunk payload (with a partial tail chunk) is copied exactly, reassembly is byte-equal to source - partial dedup: pre-seed 1 of 3 chunks on downstream → uploaded=2; rerun is a full no-op (uploaded=0), proving idempotence 251 tests pass (+2 from Phase 5j). Pre-existing macOS failure unchanged. Follow-ons: (1) parallel chunk transfer (uploaded chunks in fan-out) would speed multi-GB prewarms further; (2) exposing an accurate transferred-bytes counter needs router-side accounting instead of the current chunk-count × CHUNK_SIZE approximation. |
||
|
Omar Sobh
|
d36cec11a6 |
Phase 5g: cache metrics + GetMetrics RPC + peer-metrics CLI
Every RPC handler that answers a hit-or-miss question now increments
lock-free atomic counters. The GetMetrics RPC (0x13) returns a JSON
snapshot of every counter; the new claw-cargo peer-metrics CLI
prints hit rates, byte volumes, and counter uptime.
Placement engines can now poll these across the fleet to bias runner
scheduling toward whichever node has the warmest cache for a given
repo/tag combination.
## New module: cluster/metrics.rs (268 lines)
Types:
- CacheMetrics — atomic counters, all AtomicU64, Relaxed ordering
(metrics are advisory, not consistency-critical)
- MetricsReply — JSON snapshot returned by GetMetrics
Public API:
- CacheMetrics::new() — timestamped start, all counters at 0
- record_get_ref_hit / _miss
- record_get_tag_hit / _miss
- record_blob_get_bytes / record_blob_put_bytes
- record_get_chunk_hit / _miss
- record_has_chunk_hit / _miss
- snapshot() — atomic-load every field into a MetricsReply
MetricsReply derived helpers:
- get_ref_hit_rate() / get_tag_hit_rate() / has_chunk_hit_rate() —
Option<f64> so 0/0 returns None instead of NaN
## RPC method
- GetMetrics (0x13): payload = empty; reply = JSON MetricsReply
Wire-level instrumentation added to RpcRouter dispatch:
- GetRef → record_get_ref_hit / _miss
- GetTag → record_get_tag_hit / _miss
- BlobGet → record_blob_get_bytes (on hit)
- BlobPut → record_blob_put_bytes
- HasChunk → record_has_chunk_hit / _miss
- GetChunk → record_get_chunk_hit + record_blob_get_bytes on hit
/ record_get_chunk_miss
RpcRouter grows Arc<CacheMetrics> unconditionally — every router has
metrics, so a fresh node with no traffic still returns a valid
snapshot with all zeros + started_unix.
Streaming variants (BlobPutStream / BlobGetStream) don't yet track
byte counts — they'd require plumbing the count out of put_stream /
stream_to. Follow-on if it turns out to matter for placement.
## Client helper + CLI
- call_get_metrics(&conn) → Result<MetricsReply>
- claw-cargo peer-metrics [--peer ...] [--peer-addr ...] [--tls-dir ...]
Fetches + pretty-prints:
counter uptime: 42s
GetRef hits/misses: 123 / 45
hit rate: 73.21%
GetTag hits/misses: 8 / 2
hit rate: 80.00%
HasChunk hits/miss: 512 / 88
hit rate: 85.33%
GetChunk hits/miss: 47 / 12
Blob GET bytes: 1.23 GiB
Blob PUT bytes: 3.45 GiB
human_bytes() helper picks GiB / MiB / KiB / B based on magnitude.
Subcommand count now 9: build / prefetch / status / fingerprint /
pin / unpin / list-tags / prewarm / peer-metrics.
## Tests (13 new, all real filesystem / real QUIC — no mocks)
CacheMetrics (6):
- new_starts_all_counters_at_zero_except_timestamp
- recorders_increment_the_right_field (every recorder × 1-2 counts)
- hit_rates_none_when_zero_events (avoids 0/0 NaN)
- hit_rates_compute_correctly (3 hits / 1 miss → 75%)
- snapshot_round_trips_through_json
- snapshots_across_threads_are_consistent_up_to_relaxed_ordering
(8 threads × 1000 increments → exactly 8000)
RPC integration (7):
- phase_5g_method_byte_encoding
- get_metrics_returns_empty_snapshot_before_any_activity
- get_ref_records_hit_and_miss_counters (2 hits + 1 miss)
- get_tag_records_hit_and_miss_counters
- blob_get_and_blob_put_record_byte_counts
- has_chunk_and_get_chunk_record_hit_miss_counters
- **end_to_end_get_metrics_over_real_quic** — seed activity locally,
fire GetRef/GetTag/BlobGet dispatches to move the counters, then
fetch metrics through real QUIC + mTLS and verify each field
including the 0.5 hit rate calculation
238 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/metrics.rs: 268
- cluster/rpc.rs: 818
- cluster/rpc/tests_phase5.rs: 905
- claw_cargo.rs: 894
## What this enables
Fleet-wide visibility into which peer is actually serving traffic:
# From anywhere with connectivity + fleet mTLS
claw-cargo peer-metrics --peer tank
claw-cargo peer-metrics --peer architect
claw-cargo peer-metrics --peer morpheus
Compare hit rates side by side to see which node's cache is warmest.
A placement engine can automate this — poll every 30s, feed the
scheduler.
## Follow-on
- 5h: streaming variant of prewarm (fixed-memory ceiling for many-GB
blobs)
- 5i: metrics also published via gossip so PeerView carries hit rate
without a per-peer GetMetrics roundtrip
- 5j: prometheus /metrics endpoint on the daemon for existing dash
integrations
- 6: FUSE mount for warm-tier git worktrees
|
||
|
Omar Sobh
|
db55903311 |
Phase 5f: claw-cargo prewarm — cross-peer cache copy
The last piece before "Gitea webhook triggers a cache-warm for the
CI runner before its build starts." Adds a `prewarm` subcommand that
copies a tagged cache from one peer (upstream) to another (downstream)
in one shot — same tag, same BlobId, both sides serve it after.
## New subcommand
```
claw-cargo prewarm \
--from-peer tank --from-addr 10.0.0.14:7702 \
--to-peer morpheus --to-addr 10.0.0.15:7702 \
--tls-dir /etc/claw-store/tls \
--pin clawverse:main:latest
```
Flow:
1. Connect to upstream with local mTLS identity
2. `GetTag(tag)` → BlobId; `BlobStat(BlobId)` → size + chunk count
3. `BlobGetStream(BlobId)` → download bytes
4. Connect to downstream (second QUIC endpoint, same identity)
5. `BlobPutStream(bytes)` → returns BlobId; verified equal to upstream's
6. `PutTag(tag → BlobId)` on downstream
Summary output shows tag, blob id, both endpoints, byte count,
download/upload timings, total wall clock.
Assumes upstream + downstream share the same fleet CA (the common
case). Mixed-fleet variant with distinct identities is a follow-on.
## Integrity check
`assigned_id != blob_id` after the downstream upload triggers a
bail — the two BlobIds must match because content is BLAKE3-hashed
end-to-end. If they don't, the wire path corrupted bytes and the
whole prewarm fails loud rather than silently pinning a bad blob.
## Buffered vs streamed
Current implementation buffers the whole blob in memory between
download and upload. Fine for cargo target dirs (~1-5 GB compressed);
would break for a 20 GB blob. A follow-on will pipe upstream → tokio
duplex → downstream to run at fixed memory.
## Tests (1 new, real 2-peer QUIC)
- **`end_to_end_prewarm_copies_tagged_blob_between_two_peers`**
Two full RpcRouters serving in-process (A upstream + C downstream),
each on a distinct port. Seeds A with a blob + tag, then runs the
exact sequence prewarm runs internally: `GetTag → BlobGetStream`
against A, then `BlobPutStream → PutTag` against C. Verifies that
C's blob store returns byte-equal payload and C's tag store now
points at the same BlobId. Proves the composition works.
## Housekeeping
`rpc/tests.rs` hit 1408 lines with the new prewarm test. Phase 5
tests (5b refs + 5d tags + 5e restore + 5f prewarm) split to
`rpc/tests_phase5.rs` via a second `#[path]` module in rpc.rs.
Result:
- rpc/tests.rs: 727 (phase 1-2d tests)
- rpc/tests_phase5.rs: 722 (phase 5 tests)
- rpc.rs: 777
- rpc/client.rs: 589
- claw_cargo.rs: 818
- All under ceiling.
225 tests pass. Pre-existing macOS-only failure unchanged.
## What this enables
The complete CI runner flow now works end-to-end:
```
Primary (e.g. tank):
claw-cargo build # first ever build — MISS, uploads
claw-cargo pin --name clawverse:main:latest
Fleet control plane on PR open:
gitea webhook → shell hook → claw-cargo prewarm \
--from-peer tank --to-peer $RUNNER_LOCAL \
--pin clawverse:main:latest
# runner's local daemon now serves the tag + blob
Runner picks up job:
claw-cargo build --peer 127.0.0.1:7702
# local daemon is warm → prefetch returns HIT
# cargo build runs against restored deps → workspace-crates only
# 50 min → 3 min
```
Every subcommand claw-cargo needs for this pipeline now exists:
build / prefetch / prefetch --pin / status / fingerprint /
pin / unpin / list-tags / prewarm.
## What's next
- 5g: cache hit/miss metrics into gossip so placement engines can
bias runner scheduling toward warm nodes
- 5h: streaming variant of prewarm (tokio duplex) for many-GB blobs
- 6: FUSE mount for warm-tier git worktrees
- 3: full CRDT metadata if the plain-tag model shows conflict problems
|
||
|
Omar Sobh
|
05ba800d01 |
Phase 5e: prefetch --pin <tag>
Small, focused extension to Phase 5c's prefetch: an optional
`--pin <tag-name>` flag that skips fingerprint compute entirely
and resolves the tag → BlobId via GetTag, then downloads that.
## Use case
Restore an old cache into a fresh checkout for regression testing:
$ claw-cargo prefetch --pin clawverse:main:2026-07-12
cache HIT — downloading 3221225472 bytes (768 chunks) to /path/target/dev
...
── claw-cargo prefetch ─────────────────────────────
source: --pin clawverse:main:2026-07-12
blob: 8c2f1a…
downloaded: 3221225472 bytes in 12.3s
restored to: /path/target/dev
────────────────────────────────────────────────────
Or diagnose a "why does this build fail against the pinned cache"
question by prefetching the tagged cache and then running cargo
against your current source. Cargo will detect the mismatched
.fingerprint state and rebuild affected crates — that's the point,
you're diffing behaviour between two known-good cache snapshots.
## Changes
- New PrefetchArgs struct (was reusing PeerArgs) with an optional
`pin: Option<String>` field
- resolve_pin(conn, tag) — internal helper that does
GetTag → BlobStat, returning None on either NotFound
- cmd_prefetch branches at the top: --pin → resolve_pin(); default
→ fingerprint-based peer_lookup()
- Rest of the flow is unchanged: BlobStat → BlobGetStream →
restore_target
- Summary output shows `source: --pin <tag>` instead of
`fingerprint: <hex>` when the pinned path was taken
`peer_lookup` (fingerprint path) and `resolve_pin` (tag path) return
the same `Option<(BlobId, BlobStat)>` shape so the downstream code
is identical.
## Live smoke test
`prefetch --help` now advertises --pin with full description.
Missing-tag path prints "no such tag: <name>" and exits 0
(consistent with the fingerprint-miss path).
## Tests (1 new, real QUIC)
- **`end_to_end_tag_resolve_and_stream_restore_over_real_quic`** —
seeds blob store with a 2 MiB "captured target" payload, publishes
a tag pointing at its BlobId, then runs the exact client
sequence `prefetch --pin <tag>` runs internally:
GetTag → BlobStat → BlobGetStream
Verifies bytes reassemble byte-equal to source. Also covers the
missing-tag path.
The pin flow uses the same underlying calls tested separately in
Phase 5b/5c/5d, so the new test proves the composition works rather
than re-verifying primitives.
224 tests pass. Pre-existing macOS-only failure unchanged.
## What's next
- 5f: Gitea webhook pre-fetch — daemon receives PR-open hints and
warms cache for the predicted fingerprint before CI runner starts
- 6: FUSE mount for warm-tier git worktrees so `~/projects/clawverse`
is transparently fleet-shared
- 3: full CRDT metadata layer (only if real conflicts emerge in the
simple tag model)
|
||
|
Omar Sobh
|
b7904b59a5 |
Phase 5d: named tags + pin/unpin/list-tags CLI
Human-readable pins on top of the raw 32-byte ref layer. Operators
publish `clawverse:main:latest-cache` → BlobId once, then everything
downstream (CI runners, dev laptops) references the tag instead of
passing 64-char hex hashes around.
## Module: cluster/tags.rs (433 lines)
TagStore for string-key → 32-byte-value:
- open(root) — creates layout, safe on existing stores
- put(key, value) / get(key) / delete(key) / contains(key)
- list() — sorted by key
- Atomic writes via tempfile + rename
- Key length capped at MAX_TAG_KEY_BYTES (4 KiB); empty keys rejected
On-disk record: `key_len:u16 (LE) || key_bytes || value:32bytes`.
Filename is `blake3(key)` hex so arbitrary UTF-8 keys land at
deterministic paths without shell escaping.
TagEntry type (public, serde) for list results:
`{ key, value_hex }`. Includes `decode_value() → Result<[u8;32]>`.
## RPC methods
- PutTag (0x0f): payload = encoded record → STREAM_STATUS_OK / err
- GetTag (0x10): payload = key bytes → 32-byte value / NotFound
- DeleteTag (0x11): payload = key bytes → STREAM_STATUS_OK / NotFound
- ListTags (0x12): payload = empty → JSON Vec<TagEntry>
RpcRouter grows optional Arc<TagStore> via `.with_tag_store(store)`.
## Services + config
ClusterServices auto-opens a TagStore at `<blob_store_root>/tags-db`
alongside the ref store. `tag_store` field on ClusterServices, same
enable-with-blob-store semantics.
## claw-cargo new subcommands
- `claw-cargo pin --name clawverse:main:latest`
Compute current fingerprint → look up its BlobId via GetRef →
publish TagStore mapping. Errors cleanly if the fingerprint
hasn't been built yet (nothing to point at).
- `claw-cargo unpin --name clawverse:main:latest`
Delete the tag. Prints "no such tag" if it wasn't set.
- `claw-cargo list-tags`
Print every tag with its 32-byte hex value.
Total subcommand count now 7: build / prefetch / status / fingerprint
/ pin / unpin / list-tags. All share the layered config from Phase 5c.
## Client helpers
- call_put_tag / call_get_tag / call_delete_tag / call_list_tags
- All follow the same error-mapping conventions as prior client helpers
(NotFound → Ok(None) or Ok(false), everything else → Err)
## Housekeeping
rpc.rs was pushing past the 1300-line ceiling with the tag methods
added. Client helpers moved to `cluster/rpc/client.rs` with a
re-export (`pub use client::*;`) so external callers still write
`cluster::rpc::call_*`. Result:
- rpc.rs: 773 (was 1343)
- rpc/client.rs: 593 (new)
- rpc/tests.rs: 1235
- All under ceiling.
## Tests (33 new, all real filesystem / real QUIC — no mocks)
TagStore (16 in cluster/tags.rs):
- open_creates_layout
- get_returns_none_for_missing (+ contains false)
- put_and_get_round_trip
- put_overwrites_prior_value
- delete_returns_true_for_existing_and_false_for_missing
- put_rejects_empty_key
- put_rejects_oversize_key
- list_returns_all_tags_sorted
- list_is_empty_on_fresh_store
- keys_with_slashes_and_colons_round_trip (real-world tag shape)
- encode_and_decode_round_trip (raw wire format)
- decode_rejects_short_record
- decode_rejects_length_mismatch
- decode_rejects_non_utf8_key
- tag_entry_decode_value_round_trip
- tag_entry_decode_value_rejects_bad_hex
RPC dispatch (7 new):
- phase_5d_method_byte_encoding
- tag_rpcs_return_not_configured_without_store
- put_tag_stores_and_get_tag_reads_back
- get_tag_returns_not_found_for_missing
- get_tag_rejects_empty_key
- delete_tag_removes_and_returns_not_found_after
- list_tags_returns_json_sorted
End-to-end over real QUIC (1):
- **end_to_end_pin_lookup_delete_over_real_quic** — publish tag →
look up → list → delete → confirm gone. Full round trip through
the wire layer including JSON deserialization of the list.
Also 8 downstream tests continued passing after the client.rs split
(no test moved, they were untouched).
223 tests pass. Pre-existing macOS-only failure unchanged.
## What this enables
Operator flow:
# Build once on the primary
$ claw-cargo build
→ cache MISS → cargo build (50 min) → capture + upload
→ summary: fingerprint 4a3b…, blob 8c2f…, uploaded 3.2 GiB
# Publish a friendly name
$ claw-cargo pin --name clawverse:main:2026-07-12
pinned: clawverse:main:2026-07-12
fingerprint: 4a3b2c…
blob: 8c2f1a…
# Anyone else can now find it via list-tags
$ claw-cargo list-tags
clawverse:main:2026-07-12 8c2f1a…
clawverse:main:latest 8c2f1a…
# CI runner sees the same fingerprint in its workspace state, hits
# the ref directly via GetRef — the tag is for operator visibility
## Follow-on
- 5e: prefetch --pin <tag> — bypass fingerprint compute, download
the tagged BlobId directly (useful when you want an old cache to
test regression scenarios)
- 5f: gitea webhook pre-fetch — daemon pre-warms cache for known
fingerprints before CI runner starts
- 3: full CRDT metadata layer (namespaces, versioned pointers,
vector clocks) if the plain-tag model turns out to have
real-world conflict scenarios
|
||
|
Omar Sobh
|
2d09b4687c |
Phase 5b: KV refs + claw-cargo CLI (the killer feature, live)
Ships the actual user-facing cargo build cache. Combined with Phase 5a
(fingerprint + capture + restore) + the whole Phase 2 blob substrate,
`claw-cargo build` now runs `cargo build` with a peer-cache lookup:
hit → download+restore, miss → build+capture+upload.
## What ships
### cluster/refs.rs (243 lines)
A dumb 32-byte-key → 32-byte-value directory-backed store. Used to map
fingerprints → BlobIds. Layout mirrors BlobStore:
<root>/
refs/<kk>/<key_hex>.ref — 32 raw bytes
.tmp/ — atomic-rename staging
Public API: RefStore::open / get / put / delete / contains. All writes
atomic via tempfile + rename. Deliberately no versioning or CRDT
semantics — that's Phase 3. Every real cargo-cache lookup is a
single-key-single-value shape.
### New RPC methods
- GetRef (0x0d): payload = 32-byte RefKey; reply = 32 bytes / NotFound
- PutRef (0x0e): payload = 32-byte RefKey || 32-byte RefValue;
reply = STREAM_STATUS_OK / error
### RpcRouter + services
- RpcRouter grows optional Arc<RefStore> via `with_ref_store`
- ClusterServices opens a RefStore alongside the BlobStore when
`blob_store_root` is configured (co-located at `<blob_root>/refs-db`)
- `blob_store_enabled()` / `ref_store_enabled()` introspection
### claw-cargo binary (319 lines)
New bin target `claw-cargo` — thin CLI wrapping the whole stack:
claw-cargo fingerprint --profile release --features "a,b"
→ prints the workspace fingerprint (no network)
claw-cargo build \
--peer <name> --peer-addr <ip:port> --tls-dir <dir> \
--profile release --features "a,b" \
-- --workspace=x --frozen ...
→ 1. compute fingerprint
2. QUIC + mTLS connect to peer
3. GetRef(fingerprint) → BlobId?
HIT: BlobStat → BlobGetStream → restore_target → cargo build
MISS: cargo build → capture_target → BlobPutStream → PutRef
4. Print summary: fingerprint, hit/miss, bytes, cargo elapsed
## Live smoke test
Ran claw-cargo fingerprint on this workspace with three profile/feature
combos — got three distinct 32-byte fingerprints. Same profile+features
on the same workspace state → same fingerprint (Phase 5a's guarantee
carried through the CLI).
## Tests (14 new, all real — no mocks)
Refs store (7):
- open creates layout
- get returns None for missing
- put + get round-trips
- put overwrites prior value
- delete removes ref + reports (false on second delete)
- distinct keys produce distinct on-disk files (bucket fan-out proof)
- rejects_wrong_length_on_disk (corruption detection)
RPC (7):
- phase_5b_method_byte_encoding
- get_ref_returns_not_found_for_missing
- put_ref_stores_and_get_ref_reads_back
- put_ref_rejects_wrong_length_payload
- get_ref_rejects_wrong_length_payload
- ref_rpcs_return_not_configured_without_store
- end_to_end_put_ref_get_ref_over_real_quic — full 2-node QUIC + mTLS
round trip proving PutRef/GetRef work at the wire level
188 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/refs.rs: 243
- cluster/rpc.rs: 1169
- cluster/rpc/tests.rs: 1073
- cluster/services.rs: 565
- claw_cargo.rs: 319
## Where this leaves us
The distributed FS + cargo cache is functionally complete for the
happy path:
Node A builds clawverse for the first time
→ cargo build (50 min cold)
→ capture_target (a few seconds)
→ push to node B via BlobPutStream (network-bound)
→ PutRef(fingerprint → BlobId)
Node B on the same workspace state runs `claw-cargo build …`
→ compute_fingerprint (ms)
→ GetRef → hit
→ BlobGetStream (network-bound)
→ restore_target (a few seconds)
→ cargo build → sees valid deps/.fingerprint, builds only
workspace crates (~3 min instead of 50)
Same workspace state on a third machine? Same fingerprint → same
cache hit. That's the whole design.
## Follow-on
- Phase 5c: pre-fetch on Gitea webhook so CI runners never wait
- Phase 5d: metric ticker publishes cache hit rate into gossip so
the placement engine can bias runner scheduling toward warm nodes
- Phase 3: CRDT metadata for human-readable pins on top of raw
32-byte refs (`clawverse:main:latest-cache` → fingerprint hex)
- Phase 6+: FUSE mount for the warm-tier git worktrees
|
||
|
Omar Sobh
|
2e984b924d |
Phase 2d: chunk-level RPC (HasChunk / PutChunk / GetChunk / PutManifest)
Unlocks partial-sync replication — a peer that already has some
chunks of a blob (typical when two nodes share overlapping cargo
build caches) only receives the chunks it's missing.
## New methods
| Byte | Method | Payload | Reply |
|---|---|---|---|
| 0x09 | HasChunk | 32-byte ChunkHash | STREAM_STATUS_OK / NotFound |
| 0x0a | PutChunk | ChunkHash \|\| bytes | STREAM_STATUS_OK / error |
| 0x0b | GetChunk | ChunkHash | STREAM_STATUS_OK \|\| bytes / NotFound |
| 0x0c | PutManifest | JSON BlobManifest | JSON PutManifestReply |
`PutManifestReply { blob_id, missing: Vec<ChunkHash> }`: empty
`missing` means the manifest was written; non-empty tells the
client which chunks to upload before retrying.
Server verifies bytes hash to claimed hash on PutChunk; a
mismatch surfaces as InvalidRequest and the store is untouched.
## BlobStore additions
- `has_chunk(&ChunkHash) → bool`
- `read_chunk(&ChunkHash) → Option<Vec<u8>>` — verifies hash on read
- `put_chunk(&ChunkHash, bytes) → Result<()>` — verifies bytes-vs-hash
- `put_manifest_verified(&manifest) → Result<Vec<ChunkHash>>` —
returns the list of chunks missing on disk (empty on success)
- `chunk_path` promoted to `pub` for advanced callers
## Client helpers
- `call_has_chunk` / `call_put_chunk` / `call_get_chunk` / `call_put_manifest`
- `push_blob_missing_chunks(conn, local_store, blob_id) →
Result<(uploaded, total)>` — high-level partial-sync helper
`push_blob_missing_chunks` loads the local manifest, calls HasChunk
for each chunk, uploads only the missing ones via PutChunk, then
commits via PutManifest. On a fully-overlapping cache the uploaded
count is 0 and only the ~small manifest crosses the wire.
## Tests (17 new, all real filesystem + real QUIC — no mocks)
Blob store (6):
- has_chunk_is_false_before_put_and_true_after
- read_chunk_returns_bytes_and_none_when_missing
- put_chunk_rejects_hash_mismatch (nothing written)
- read_chunk_detects_corruption (bit-flip → mismatch error)
- put_manifest_verified_reports_missing_chunks
- put_manifest_verified_writes_when_all_chunks_present
Router dispatch (7):
- phase_2d_method_byte_encoding
- method_reports_streaming_variants — extended for 4 new methods
- has_chunk_returns_ok_for_present_and_not_found_for_missing
- put_chunk_stores_and_returns_status_ok
- put_chunk_rejects_hash_mismatch_over_wire
- get_chunk_returns_content_prefixed_with_status_ok
- get_chunk_returns_not_found_for_missing
- put_manifest_reports_missing_chunks_when_incomplete
- put_manifest_writes_when_chunks_present
- chunk_rpcs_return_not_configured_without_store
End-to-end (2):
- **end_to_end_push_blob_missing_chunks_replicates_only_needed_bytes**:
Peer A pre-seeded with chunk 0 of a 2-chunk (8 MiB) blob;
`push_blob_missing_chunks` reports `(uploaded=1, total=2)`,
only chunk 1 crosses the wire, A's store then contains the
complete blob and `get_bytes` returns byte-equal content.
- **call_get_chunk_verifies_returned_hash**: real 2-node fetch,
client hashes received bytes and compares to requested hash.
156 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/rpc.rs: 1053
- cluster/rpc/tests.rs: 940
- cluster/blob.rs: 1186
## Where this fits
With Phase 2c whole-blob streaming + Phase 2d partial-chunk sync,
the storage substrate is now genuinely bandwidth-efficient in the
distributed setting:
- First-ever push of a blob: `push_blob_missing_chunks` uploads
everything (all chunks missing).
- Second push of a similar blob (95% chunk overlap with prior
contents): only the 5% new chunks cross the wire, plus a tiny
manifest.
- Whole-blob download: BlobGetStream, bounded by network bandwidth.
## Follow-on
- Phase 3: CRDT metadata for human-readable namespaces on top of
content hashes.
- Phase 5: the killer feature. Fingerprint cargo target dir → tar
→ hash → PutBlobStream (or push_blob_missing_chunks if a similar
build already lives on the peer). Same fingerprint on the next
node → BlobGetStream. This is the whole cargo-cache design in
one line and it now sits on a substrate that handles all the
hard cases (dedup, verification, resumability, partial sync).
|
||
|
Omar Sobh
|
1fd1027da4 |
Phase 2c: streaming Blob RPC (BlobPutStream / BlobGetStream)
Removes the 16 MiB message cap for blob transfers. The bounded Blob* methods from Phase 2b still exist; the streaming variants let a peer push or pull a many-GB blob without either side holding it in memory. ## Wire format Streaming methods use a slightly different reply shape so the client can route on the first byte alone: Reply : status:u8 || payload:bytes... Where `status` is either `STREAM_STATUS_OK` (0x00, content follows) or a single-byte ErrorCode. `serve_connection` now peeks at the method tag byte via read_exact and hands streaming methods the raw send/recv streams; bounded methods still use the old read_to_end path. ## Method additions - BlobPutStream (0x07): client streams bytes → server pipes into BlobStore::put_stream → reply is 0x00 || 32-byte BlobId - BlobGetStream (0x08): client sends 32-byte BlobId → server verifies existence, writes 0x00 status, then streams chunks from disk into the send stream Method::is_streaming() introspection so callers can decide which wire variant to use. ## BlobStore additions - put_stream<R: AsyncRead + Unpin>(reader) -> BlobId Memory ceiling: one CHUNK_SIZE (4 MiB) buffer regardless of blob size. Handles short-reads correctly (loops until CHUNK_SIZE bytes are available or EOF), including the empty-reader case (produces the empty-blob BlobId, zero chunks). - stream_to<W: AsyncWrite + Unpin>(id, writer) -> bool Ok(false) on NotFound (writer untouched). Verifies each chunk hash before emitting; corruption halts mid-stream with Err. ## Client helpers - call_blob_put_stream(conn, reader) -> Result<BlobId> Uses tokio::io::copy directly onto quinn's SendStream. - call_blob_get_stream(conn, id, writer) -> Result<bool> Ok(false) on NotFound; other errors surface as Err. ## Tests (11 new, all real — no mocks) Blob store (6): - put_stream_produces_same_hash_as_put_bytes (3-chunk blob via Cursor) - put_stream_handles_empty_reader (produces empty-blob BlobId) - put_stream_handles_short_reads (custom Trickle reader that only serves 100 bytes per read call — must still assemble full chunks) - stream_to_writes_full_blob (2-chunk write to Vec<u8>) - stream_to_returns_false_when_missing (writer untouched) - stream_to_detects_chunk_corruption (bit-flip a chunk → Err with "chunk hash mismatch") RPC (5): - method_reports_streaming_variants - end_to_end_stream_put_and_get_over_real_quic — 12 MiB + 777 bytes → 4 chunks, real 2-node QUIC + mTLS + stream round-trip - stream_get_returns_false_for_missing_blob - stream_methods_return_not_configured_without_store - stream_put_deduplicates_with_prior_put_bytes — verify streaming put produces the same BlobId as a prior bounded put on identical content, and the manifest chunk count didn't fork ## Housekeeping rpc.rs was tipping over the 1300-line ceiling with the streaming handlers + helpers + tests. Tests split into `cluster/rpc/tests.rs` via `#[path = "rpc/tests.rs"] mod tests;`. Result: - rpc.rs: 748 lines - rpc/tests.rs: 694 lines - blob.rs: 1002 lines - All under ceiling. 139 tests pass. Pre-existing macOS-only failure unchanged. ## What's next - Phase 2d: chunk-level RPC (BlobPutChunk / BlobGetChunk) so a receiver can `LoadManifest` then request only the chunks it's missing — big bandwidth win on partially-overlapping caches. - Phase 3: CRDT metadata for human-readable namespaces on top of content hashes. - Phase 5: the killer feature — fingerprint the cargo target dir, BlobPutStream it, next node BlobGetStream by the same fingerprint. Now buildable directly on Phase 2c since target dirs run 100 MB to a few GB and the previous 16 MiB cap would have blocked us. |