b7904b59a5501d924b6c847755b59838f18fccdc
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar Sobh
|
b7904b59a5 |
Phase 5d: named tags + pin/unpin/list-tags CLI
Human-readable pins on top of the raw 32-byte ref layer. Operators
publish `clawverse:main:latest-cache` → BlobId once, then everything
downstream (CI runners, dev laptops) references the tag instead of
passing 64-char hex hashes around.
## Module: cluster/tags.rs (433 lines)
TagStore for string-key → 32-byte-value:
- open(root) — creates layout, safe on existing stores
- put(key, value) / get(key) / delete(key) / contains(key)
- list() — sorted by key
- Atomic writes via tempfile + rename
- Key length capped at MAX_TAG_KEY_BYTES (4 KiB); empty keys rejected
On-disk record: `key_len:u16 (LE) || key_bytes || value:32bytes`.
Filename is `blake3(key)` hex so arbitrary UTF-8 keys land at
deterministic paths without shell escaping.
TagEntry type (public, serde) for list results:
`{ key, value_hex }`. Includes `decode_value() → Result<[u8;32]>`.
## RPC methods
- PutTag (0x0f): payload = encoded record → STREAM_STATUS_OK / err
- GetTag (0x10): payload = key bytes → 32-byte value / NotFound
- DeleteTag (0x11): payload = key bytes → STREAM_STATUS_OK / NotFound
- ListTags (0x12): payload = empty → JSON Vec<TagEntry>
RpcRouter grows optional Arc<TagStore> via `.with_tag_store(store)`.
## Services + config
ClusterServices auto-opens a TagStore at `<blob_store_root>/tags-db`
alongside the ref store. `tag_store` field on ClusterServices, same
enable-with-blob-store semantics.
## claw-cargo new subcommands
- `claw-cargo pin --name clawverse:main:latest`
Compute current fingerprint → look up its BlobId via GetRef →
publish TagStore mapping. Errors cleanly if the fingerprint
hasn't been built yet (nothing to point at).
- `claw-cargo unpin --name clawverse:main:latest`
Delete the tag. Prints "no such tag" if it wasn't set.
- `claw-cargo list-tags`
Print every tag with its 32-byte hex value.
Total subcommand count now 7: build / prefetch / status / fingerprint
/ pin / unpin / list-tags. All share the layered config from Phase 5c.
## Client helpers
- call_put_tag / call_get_tag / call_delete_tag / call_list_tags
- All follow the same error-mapping conventions as prior client helpers
(NotFound → Ok(None) or Ok(false), everything else → Err)
## Housekeeping
rpc.rs was pushing past the 1300-line ceiling with the tag methods
added. Client helpers moved to `cluster/rpc/client.rs` with a
re-export (`pub use client::*;`) so external callers still write
`cluster::rpc::call_*`. Result:
- rpc.rs: 773 (was 1343)
- rpc/client.rs: 593 (new)
- rpc/tests.rs: 1235
- All under ceiling.
## Tests (33 new, all real filesystem / real QUIC — no mocks)
TagStore (16 in cluster/tags.rs):
- open_creates_layout
- get_returns_none_for_missing (+ contains false)
- put_and_get_round_trip
- put_overwrites_prior_value
- delete_returns_true_for_existing_and_false_for_missing
- put_rejects_empty_key
- put_rejects_oversize_key
- list_returns_all_tags_sorted
- list_is_empty_on_fresh_store
- keys_with_slashes_and_colons_round_trip (real-world tag shape)
- encode_and_decode_round_trip (raw wire format)
- decode_rejects_short_record
- decode_rejects_length_mismatch
- decode_rejects_non_utf8_key
- tag_entry_decode_value_round_trip
- tag_entry_decode_value_rejects_bad_hex
RPC dispatch (7 new):
- phase_5d_method_byte_encoding
- tag_rpcs_return_not_configured_without_store
- put_tag_stores_and_get_tag_reads_back
- get_tag_returns_not_found_for_missing
- get_tag_rejects_empty_key
- delete_tag_removes_and_returns_not_found_after
- list_tags_returns_json_sorted
End-to-end over real QUIC (1):
- **end_to_end_pin_lookup_delete_over_real_quic** — publish tag →
look up → list → delete → confirm gone. Full round trip through
the wire layer including JSON deserialization of the list.
Also 8 downstream tests continued passing after the client.rs split
(no test moved, they were untouched).
223 tests pass. Pre-existing macOS-only failure unchanged.
## What this enables
Operator flow:
# Build once on the primary
$ claw-cargo build
→ cache MISS → cargo build (50 min) → capture + upload
→ summary: fingerprint 4a3b…, blob 8c2f…, uploaded 3.2 GiB
# Publish a friendly name
$ claw-cargo pin --name clawverse:main:2026-07-12
pinned: clawverse:main:2026-07-12
fingerprint: 4a3b2c…
blob: 8c2f1a…
# Anyone else can now find it via list-tags
$ claw-cargo list-tags
clawverse:main:2026-07-12 8c2f1a…
clawverse:main:latest 8c2f1a…
# CI runner sees the same fingerprint in its workspace state, hits
# the ref directly via GetRef — the tag is for operator visibility
## Follow-on
- 5e: prefetch --pin <tag> — bypass fingerprint compute, download
the tagged BlobId directly (useful when you want an old cache to
test regression scenarios)
- 5f: gitea webhook pre-fetch — daemon pre-warms cache for known
fingerprints before CI runner starts
- 3: full CRDT metadata layer (namespaces, versioned pointers,
vector clocks) if the plain-tag model turns out to have
real-world conflict scenarios
|
||
|
Omar Sobh
|
2d09b4687c |
Phase 5b: KV refs + claw-cargo CLI (the killer feature, live)
Ships the actual user-facing cargo build cache. Combined with Phase 5a
(fingerprint + capture + restore) + the whole Phase 2 blob substrate,
`claw-cargo build` now runs `cargo build` with a peer-cache lookup:
hit → download+restore, miss → build+capture+upload.
## What ships
### cluster/refs.rs (243 lines)
A dumb 32-byte-key → 32-byte-value directory-backed store. Used to map
fingerprints → BlobIds. Layout mirrors BlobStore:
<root>/
refs/<kk>/<key_hex>.ref — 32 raw bytes
.tmp/ — atomic-rename staging
Public API: RefStore::open / get / put / delete / contains. All writes
atomic via tempfile + rename. Deliberately no versioning or CRDT
semantics — that's Phase 3. Every real cargo-cache lookup is a
single-key-single-value shape.
### New RPC methods
- GetRef (0x0d): payload = 32-byte RefKey; reply = 32 bytes / NotFound
- PutRef (0x0e): payload = 32-byte RefKey || 32-byte RefValue;
reply = STREAM_STATUS_OK / error
### RpcRouter + services
- RpcRouter grows optional Arc<RefStore> via `with_ref_store`
- ClusterServices opens a RefStore alongside the BlobStore when
`blob_store_root` is configured (co-located at `<blob_root>/refs-db`)
- `blob_store_enabled()` / `ref_store_enabled()` introspection
### claw-cargo binary (319 lines)
New bin target `claw-cargo` — thin CLI wrapping the whole stack:
claw-cargo fingerprint --profile release --features "a,b"
→ prints the workspace fingerprint (no network)
claw-cargo build \
--peer <name> --peer-addr <ip:port> --tls-dir <dir> \
--profile release --features "a,b" \
-- --workspace=x --frozen ...
→ 1. compute fingerprint
2. QUIC + mTLS connect to peer
3. GetRef(fingerprint) → BlobId?
HIT: BlobStat → BlobGetStream → restore_target → cargo build
MISS: cargo build → capture_target → BlobPutStream → PutRef
4. Print summary: fingerprint, hit/miss, bytes, cargo elapsed
## Live smoke test
Ran claw-cargo fingerprint on this workspace with three profile/feature
combos — got three distinct 32-byte fingerprints. Same profile+features
on the same workspace state → same fingerprint (Phase 5a's guarantee
carried through the CLI).
## Tests (14 new, all real — no mocks)
Refs store (7):
- open creates layout
- get returns None for missing
- put + get round-trips
- put overwrites prior value
- delete removes ref + reports (false on second delete)
- distinct keys produce distinct on-disk files (bucket fan-out proof)
- rejects_wrong_length_on_disk (corruption detection)
RPC (7):
- phase_5b_method_byte_encoding
- get_ref_returns_not_found_for_missing
- put_ref_stores_and_get_ref_reads_back
- put_ref_rejects_wrong_length_payload
- get_ref_rejects_wrong_length_payload
- ref_rpcs_return_not_configured_without_store
- end_to_end_put_ref_get_ref_over_real_quic — full 2-node QUIC + mTLS
round trip proving PutRef/GetRef work at the wire level
188 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/refs.rs: 243
- cluster/rpc.rs: 1169
- cluster/rpc/tests.rs: 1073
- cluster/services.rs: 565
- claw_cargo.rs: 319
## Where this leaves us
The distributed FS + cargo cache is functionally complete for the
happy path:
Node A builds clawverse for the first time
→ cargo build (50 min cold)
→ capture_target (a few seconds)
→ push to node B via BlobPutStream (network-bound)
→ PutRef(fingerprint → BlobId)
Node B on the same workspace state runs `claw-cargo build …`
→ compute_fingerprint (ms)
→ GetRef → hit
→ BlobGetStream (network-bound)
→ restore_target (a few seconds)
→ cargo build → sees valid deps/.fingerprint, builds only
workspace crates (~3 min instead of 50)
Same workspace state on a third machine? Same fingerprint → same
cache hit. That's the whole design.
## Follow-on
- Phase 5c: pre-fetch on Gitea webhook so CI runners never wait
- Phase 5d: metric ticker publishes cache hit rate into gossip so
the placement engine can bias runner scheduling toward warm nodes
- Phase 3: CRDT metadata for human-readable pins on top of raw
32-byte refs (`clawverse:main:latest-cache` → fingerprint hex)
- Phase 6+: FUSE mount for the warm-tier git worktrees
|
||
|
Omar Sobh
|
2e984b924d |
Phase 2d: chunk-level RPC (HasChunk / PutChunk / GetChunk / PutManifest)
Unlocks partial-sync replication — a peer that already has some
chunks of a blob (typical when two nodes share overlapping cargo
build caches) only receives the chunks it's missing.
## New methods
| Byte | Method | Payload | Reply |
|---|---|---|---|
| 0x09 | HasChunk | 32-byte ChunkHash | STREAM_STATUS_OK / NotFound |
| 0x0a | PutChunk | ChunkHash \|\| bytes | STREAM_STATUS_OK / error |
| 0x0b | GetChunk | ChunkHash | STREAM_STATUS_OK \|\| bytes / NotFound |
| 0x0c | PutManifest | JSON BlobManifest | JSON PutManifestReply |
`PutManifestReply { blob_id, missing: Vec<ChunkHash> }`: empty
`missing` means the manifest was written; non-empty tells the
client which chunks to upload before retrying.
Server verifies bytes hash to claimed hash on PutChunk; a
mismatch surfaces as InvalidRequest and the store is untouched.
## BlobStore additions
- `has_chunk(&ChunkHash) → bool`
- `read_chunk(&ChunkHash) → Option<Vec<u8>>` — verifies hash on read
- `put_chunk(&ChunkHash, bytes) → Result<()>` — verifies bytes-vs-hash
- `put_manifest_verified(&manifest) → Result<Vec<ChunkHash>>` —
returns the list of chunks missing on disk (empty on success)
- `chunk_path` promoted to `pub` for advanced callers
## Client helpers
- `call_has_chunk` / `call_put_chunk` / `call_get_chunk` / `call_put_manifest`
- `push_blob_missing_chunks(conn, local_store, blob_id) →
Result<(uploaded, total)>` — high-level partial-sync helper
`push_blob_missing_chunks` loads the local manifest, calls HasChunk
for each chunk, uploads only the missing ones via PutChunk, then
commits via PutManifest. On a fully-overlapping cache the uploaded
count is 0 and only the ~small manifest crosses the wire.
## Tests (17 new, all real filesystem + real QUIC — no mocks)
Blob store (6):
- has_chunk_is_false_before_put_and_true_after
- read_chunk_returns_bytes_and_none_when_missing
- put_chunk_rejects_hash_mismatch (nothing written)
- read_chunk_detects_corruption (bit-flip → mismatch error)
- put_manifest_verified_reports_missing_chunks
- put_manifest_verified_writes_when_all_chunks_present
Router dispatch (7):
- phase_2d_method_byte_encoding
- method_reports_streaming_variants — extended for 4 new methods
- has_chunk_returns_ok_for_present_and_not_found_for_missing
- put_chunk_stores_and_returns_status_ok
- put_chunk_rejects_hash_mismatch_over_wire
- get_chunk_returns_content_prefixed_with_status_ok
- get_chunk_returns_not_found_for_missing
- put_manifest_reports_missing_chunks_when_incomplete
- put_manifest_writes_when_chunks_present
- chunk_rpcs_return_not_configured_without_store
End-to-end (2):
- **end_to_end_push_blob_missing_chunks_replicates_only_needed_bytes**:
Peer A pre-seeded with chunk 0 of a 2-chunk (8 MiB) blob;
`push_blob_missing_chunks` reports `(uploaded=1, total=2)`,
only chunk 1 crosses the wire, A's store then contains the
complete blob and `get_bytes` returns byte-equal content.
- **call_get_chunk_verifies_returned_hash**: real 2-node fetch,
client hashes received bytes and compares to requested hash.
156 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/rpc.rs: 1053
- cluster/rpc/tests.rs: 940
- cluster/blob.rs: 1186
## Where this fits
With Phase 2c whole-blob streaming + Phase 2d partial-chunk sync,
the storage substrate is now genuinely bandwidth-efficient in the
distributed setting:
- First-ever push of a blob: `push_blob_missing_chunks` uploads
everything (all chunks missing).
- Second push of a similar blob (95% chunk overlap with prior
contents): only the 5% new chunks cross the wire, plus a tiny
manifest.
- Whole-blob download: BlobGetStream, bounded by network bandwidth.
## Follow-on
- Phase 3: CRDT metadata for human-readable namespaces on top of
content hashes.
- Phase 5: the killer feature. Fingerprint cargo target dir → tar
→ hash → PutBlobStream (or push_blob_missing_chunks if a similar
build already lives on the peer). Same fingerprint on the next
node → BlobGetStream. This is the whole cargo-cache design in
one line and it now sits on a substrate that handles all the
hard cases (dedup, verification, resumability, partial sync).
|
||
|
Omar Sobh
|
1fd1027da4 |
Phase 2c: streaming Blob RPC (BlobPutStream / BlobGetStream)
Removes the 16 MiB message cap for blob transfers. The bounded Blob* methods from Phase 2b still exist; the streaming variants let a peer push or pull a many-GB blob without either side holding it in memory. ## Wire format Streaming methods use a slightly different reply shape so the client can route on the first byte alone: Reply : status:u8 || payload:bytes... Where `status` is either `STREAM_STATUS_OK` (0x00, content follows) or a single-byte ErrorCode. `serve_connection` now peeks at the method tag byte via read_exact and hands streaming methods the raw send/recv streams; bounded methods still use the old read_to_end path. ## Method additions - BlobPutStream (0x07): client streams bytes → server pipes into BlobStore::put_stream → reply is 0x00 || 32-byte BlobId - BlobGetStream (0x08): client sends 32-byte BlobId → server verifies existence, writes 0x00 status, then streams chunks from disk into the send stream Method::is_streaming() introspection so callers can decide which wire variant to use. ## BlobStore additions - put_stream<R: AsyncRead + Unpin>(reader) -> BlobId Memory ceiling: one CHUNK_SIZE (4 MiB) buffer regardless of blob size. Handles short-reads correctly (loops until CHUNK_SIZE bytes are available or EOF), including the empty-reader case (produces the empty-blob BlobId, zero chunks). - stream_to<W: AsyncWrite + Unpin>(id, writer) -> bool Ok(false) on NotFound (writer untouched). Verifies each chunk hash before emitting; corruption halts mid-stream with Err. ## Client helpers - call_blob_put_stream(conn, reader) -> Result<BlobId> Uses tokio::io::copy directly onto quinn's SendStream. - call_blob_get_stream(conn, id, writer) -> Result<bool> Ok(false) on NotFound; other errors surface as Err. ## Tests (11 new, all real — no mocks) Blob store (6): - put_stream_produces_same_hash_as_put_bytes (3-chunk blob via Cursor) - put_stream_handles_empty_reader (produces empty-blob BlobId) - put_stream_handles_short_reads (custom Trickle reader that only serves 100 bytes per read call — must still assemble full chunks) - stream_to_writes_full_blob (2-chunk write to Vec<u8>) - stream_to_returns_false_when_missing (writer untouched) - stream_to_detects_chunk_corruption (bit-flip a chunk → Err with "chunk hash mismatch") RPC (5): - method_reports_streaming_variants - end_to_end_stream_put_and_get_over_real_quic — 12 MiB + 777 bytes → 4 chunks, real 2-node QUIC + mTLS + stream round-trip - stream_get_returns_false_for_missing_blob - stream_methods_return_not_configured_without_store - stream_put_deduplicates_with_prior_put_bytes — verify streaming put produces the same BlobId as a prior bounded put on identical content, and the manifest chunk count didn't fork ## Housekeeping rpc.rs was tipping over the 1300-line ceiling with the streaming handlers + helpers + tests. Tests split into `cluster/rpc/tests.rs` via `#[path = "rpc/tests.rs"] mod tests;`. Result: - rpc.rs: 748 lines - rpc/tests.rs: 694 lines - blob.rs: 1002 lines - All under ceiling. 139 tests pass. Pre-existing macOS-only failure unchanged. ## What's next - Phase 2d: chunk-level RPC (BlobPutChunk / BlobGetChunk) so a receiver can `LoadManifest` then request only the chunks it's missing — big bandwidth win on partially-overlapping caches. - Phase 3: CRDT metadata for human-readable namespaces on top of content hashes. - Phase 5: the killer feature — fingerprint the cargo target dir, BlobPutStream it, next node BlobGetStream by the same fingerprint. Now buildable directly on Phase 2c since target dirs run 100 MB to a few GB and the previous 16 MiB cap would have blocked us. |