Omar Sobh
2d09b4687c
Ships the actual user-facing cargo build cache. Combined with Phase 5a
(fingerprint + capture + restore) + the whole Phase 2 blob substrate,
`claw-cargo build` now runs `cargo build` with a peer-cache lookup:
hit → download+restore, miss → build+capture+upload.
## What ships
### cluster/refs.rs (243 lines)
A dumb 32-byte-key → 32-byte-value directory-backed store. Used to map
fingerprints → BlobIds. Layout mirrors BlobStore:
<root>/
refs/<kk>/<key_hex>.ref — 32 raw bytes
.tmp/ — atomic-rename staging
Public API: RefStore::open / get / put / delete / contains. All writes
atomic via tempfile + rename. Deliberately no versioning or CRDT
semantics — that's Phase 3. Every real cargo-cache lookup is a
single-key-single-value shape.
### New RPC methods
- GetRef (0x0d): payload = 32-byte RefKey; reply = 32 bytes / NotFound
- PutRef (0x0e): payload = 32-byte RefKey || 32-byte RefValue;
reply = STREAM_STATUS_OK / error
### RpcRouter + services
- RpcRouter grows optional Arc<RefStore> via `with_ref_store`
- ClusterServices opens a RefStore alongside the BlobStore when
`blob_store_root` is configured (co-located at `<blob_root>/refs-db`)
- `blob_store_enabled()` / `ref_store_enabled()` introspection
### claw-cargo binary (319 lines)
New bin target `claw-cargo` — thin CLI wrapping the whole stack:
claw-cargo fingerprint --profile release --features "a,b"
→ prints the workspace fingerprint (no network)
claw-cargo build \
--peer <name> --peer-addr <ip:port> --tls-dir <dir> \
--profile release --features "a,b" \
-- --workspace=x --frozen ...
→ 1. compute fingerprint
2. QUIC + mTLS connect to peer
3. GetRef(fingerprint) → BlobId?
HIT: BlobStat → BlobGetStream → restore_target → cargo build
MISS: cargo build → capture_target → BlobPutStream → PutRef
4. Print summary: fingerprint, hit/miss, bytes, cargo elapsed
## Live smoke test
Ran claw-cargo fingerprint on this workspace with three profile/feature
combos — got three distinct 32-byte fingerprints. Same profile+features
on the same workspace state → same fingerprint (Phase 5a's guarantee
carried through the CLI).
## Tests (14 new, all real — no mocks)
Refs store (7):
- open creates layout
- get returns None for missing
- put + get round-trips
- put overwrites prior value
- delete removes ref + reports (false on second delete)
- distinct keys produce distinct on-disk files (bucket fan-out proof)
- rejects_wrong_length_on_disk (corruption detection)
RPC (7):
- phase_5b_method_byte_encoding
- get_ref_returns_not_found_for_missing
- put_ref_stores_and_get_ref_reads_back
- put_ref_rejects_wrong_length_payload
- get_ref_rejects_wrong_length_payload
- ref_rpcs_return_not_configured_without_store
- end_to_end_put_ref_get_ref_over_real_quic — full 2-node QUIC + mTLS
round trip proving PutRef/GetRef work at the wire level
188 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/refs.rs: 243
- cluster/rpc.rs: 1169
- cluster/rpc/tests.rs: 1073
- cluster/services.rs: 565
- claw_cargo.rs: 319
## Where this leaves us
The distributed FS + cargo cache is functionally complete for the
happy path:
Node A builds clawverse for the first time
→ cargo build (50 min cold)
→ capture_target (a few seconds)
→ push to node B via BlobPutStream (network-bound)
→ PutRef(fingerprint → BlobId)
Node B on the same workspace state runs `claw-cargo build …`
→ compute_fingerprint (ms)
→ GetRef → hit
→ BlobGetStream (network-bound)
→ restore_target (a few seconds)
→ cargo build → sees valid deps/.fingerprint, builds only
workspace crates (~3 min instead of 50)
Same workspace state on a third machine? Same fingerprint → same
cache hit. That's the whole design.
## Follow-on
- Phase 5c: pre-fetch on Gitea webhook so CI runners never wait
- Phase 5d: metric ticker publishes cache hit rate into gossip so
the placement engine can bias runner scheduling toward warm nodes
- Phase 3: CRDT metadata for human-readable pins on top of raw
32-byte refs (`clawverse:main:latest-cache` → fingerprint hex)
- Phase 6+: FUSE mount for the warm-tier git worktrees
75 lines
3.1 KiB
TOML
75 lines
3.1 KiB
TOML
[package]
|
||
name = "claw-store"
|
||
version = "0.3.0"
|
||
edition = "2021"
|
||
|
||
[[bin]]
|
||
name = "claw-store"
|
||
path = "src/main.rs"
|
||
|
||
# Phase 5b: fingerprint-keyed cargo build cache CLI. Wraps `cargo build`
|
||
# with a peer-cache lookup: hit → download+restore, miss → build+capture+upload.
|
||
# Uses the same identity/config surface as claw-store daemon.
|
||
[[bin]]
|
||
name = "claw-cargo"
|
||
path = "src/claw_cargo.rs"
|
||
|
||
[dependencies]
|
||
clap = { version = "4", features = ["derive"] }
|
||
serde = { version = "1", features = ["derive"] }
|
||
toml = "0.8"
|
||
tokio = { version = "1", features = ["full"] }
|
||
sysinfo = "0.30"
|
||
anyhow = "1"
|
||
tracing = "0.1"
|
||
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
||
chrono = { version = "0.4", features = ["serde"] }
|
||
axum = { version = "0.7", features = ["macros"] }
|
||
tower-http = { version = "0.5", features = ["cors", "fs"] }
|
||
tokio-stream = "0.1"
|
||
serde_json = "1"
|
||
# v0.2.0 — flock(2) wrapper for atomic+locked manifest writes
|
||
# (manifest.rs). Already a transitive dep; declaring it directly
|
||
# makes the call site obvious.
|
||
libc = "0.2"
|
||
# v0.2.0 — sibling-tempfile + rename for atomic manifest persistence.
|
||
# Was previously dev-dep only; promoted to main.
|
||
tempfile = "3"
|
||
# v0.11.1 — cluster membership via scuttlebutt gossip + phi-accrual failure
|
||
# detection. Wraps in cluster/gossip.rs. UDP transport, keyed KV state per
|
||
# node, seed_nodes bootstrap from [[cluster.peers]] config.
|
||
chitchat = "0.11"
|
||
# v0.11 — QUIC transport for peer RPC (Phase 1c). UDP-based; runs on a
|
||
# separate port from chitchat gossip. TLS 1.3 by default; we wire mTLS
|
||
# against a fleet root CA in cluster/transport.rs.
|
||
quinn = { version = "0.11", default-features = false, features = ["runtime-tokio", "rustls-ring"] }
|
||
# v0.23 — TLS 1.3 provider driving quinn's crypto. Pinned to the ring
|
||
# provider so a single CryptoProvider is installed process-wide.
|
||
rustls = { version = "0.23", default-features = false, features = ["ring"] }
|
||
# v0.13 — X.509 cert generation for the fleet CA + per-node leaf certs.
|
||
# Used both by production bootstrap (writes PEM to /etc/claw-store/tls/)
|
||
# and by tests (in-memory ephemeral CA).
|
||
rcgen = { version = "0.13", features = ["pem", "x509-parser"] }
|
||
# v2 — parse PEM files into DER for rustls consumption. Used in
|
||
# NodeIdentity::from_pem_files (Phase 1d) so persistent identity round-trips
|
||
# through the file system on daemon restart.
|
||
rustls-pemfile = "2"
|
||
# v1 — content-addressed hashing for the Phase 2 blob store. BLAKE3 is
|
||
# used both for whole-blob addressing (BlobId) and for per-chunk
|
||
# addressing (dedup). Fast enough that a full-blob rehash on read
|
||
# verification stays cheap.
|
||
blake3 = "1"
|
||
# v0.4 — tar archive writer/reader for the Phase 5 build-artifact
|
||
# capture flow (cargo target/deps → tar → BlobPutStream). Preserves
|
||
# file metadata (mtimes, perms) which cargo relies on for its own
|
||
# incremental-rebuild fingerprint checks.
|
||
tar = "0.4"
|
||
# v0.13 — zstd wrapping around the tar stream. Level 3 is the default;
|
||
# gets 5-10× compression on cargo .rlib without noticeable CPU cost.
|
||
zstd = "0.13"
|
||
|
||
[dev-dependencies]
|
||
tempfile = "3"
|
||
tower = { version = "0.5", features = ["util"] }
|
||
http-body-util = "0.1"
|