7 Commits

Include renames

Author	SHA1	Message	Date
Omar Sobh	2c3cd2ab38	Phase 3c + 3e: stamped tags + namespaced ref keys — closes Phase 3 ... Ships the last two pieces from the arch doc's Phase 3 scope for the cargo-cache use case: ## 3c: Stamped tags (CRDT-merge on PutTag) Mirror of Phase 3a/3b for TagStore. Two concurrent `claw-cargo pin` calls on the same tag now race deterministically instead of silently clobbering. * `StampedTagValue` — same 48-byte (value, clock, node) tuple as StampedRef. * `TagStore::put_stamped(key, StampedTagValue) -> TagPutOutcome` and `TagStore::get_stamped(key)` — data lives under `tags-v2/` (separate from `tags/` for cutover safety). * New wire methods `PutTagVersioned = 0x18` + `GetTagVersioned = 0x19`. * `call_put_tag_versioned` / `call_get_tag_versioned` client helpers. * `claw-cargo pin` now writes stamped tags. Concurrent pin gets AlreadyExists and moves on (blob content is content-addressed so both winners agree on the payload). ## 3e: Namespaced ref keys Opt-in `--namespace <slug>` on peer-facing subcommands. When set, the ref key becomes `blake3("clawstor.ns.v1" || namespace || fp)` so two runners on different namespaces (`clawverse/main` vs `clawverse/pr-42`) don't collide on the same fingerprint. Empty namespace = pre-3e behavior, so this is 100% backward compat. * `refs::namespaced_ref_key(namespace, fingerprint) -> RefKey` primitive. * `PeerArgs::namespace: Option<String>` CLI flag flows through to `cmd_status`, `cmd_prefetch`, `cmd_build`. * `peer_lookup` now takes a `RefKey` directly (was `&Fingerprint`) so the namespace resolution stays in the caller — the daemon never sees "namespace" as a concept. ## 3d: Deferred Full vector clocks per namespace are noted in the arch doc as a Phase-3 goal; scalar wall-clock (clock + node stamp) is sufficient for the cargo-cache use case (single-key LWW merge). NTP-synced runners see monotonic ordering; skewed runners lose an ordering but the CRDT semantics still guarantee no data corruption. Full VC is deferred to a future phase. +9 tests, 280 total (baseline +8: 7 unit + 1 e2e over real QUIC).	2026-07-13 12:32:46 -07:00
Omar Sobh	af5350ac17	Phase 3a: Lamport-stamped refs with CRDT-merge on PutRef ... Concurrent PutRef safety via (clock, node) total order. New wire methods PutRefVersioned (0x15) + GetRefVersioned (0x16). Existing PutRef/GetRef unchanged for backward compat. Data in refs-v2/ namespace so the two coexist during cutover. +8 tests, 272 total (baseline +8).	2026-07-13 07:34:28 -07:00
Omar Sobh	846ecffe10	Pi deploy follow-ups: XDG default_path + parallel restore ... Two fixes surfaced by the vision-02 Pi 5 measurement: ## XDG default_path `Manifest::default_path` was hardcoded to `/var/lib/claw-store/projects.toml`. That path is read-only under the user-mode systemd unit's `ProtectSystem=strict`, and creating it needs root — awful for a runner install. Precedence, matching XDG Base Directory: 1. `$XDG_STATE_HOME/claw-store/projects.toml` 2. `$HOME/.local/state/claw-store/projects.toml` 3. `/var/lib/claw-store/projects.toml` (system fallback) User-mode installs now write in $HOME by default; system installs (root, no HOME set) still land in /var/lib. +1 test: `default_path_honours_xdg_state_home` — covers all three precedence branches. Env mutation is process-global so the test saves + restores. ## Parallel restore on cache HIT Pi restore of 947 MiB via `BlobGetStream` took ~18s (~53 MiB/s) single-stream. Per-stream throughput ceilings on the connection type cap sequential fetches; parallel chunk fetches stack their contributions. - New `call_blob_get_parallel(conn, blob_id, concurrency) -> Option<Vec<u8>>` in `rpc/client.rs`. `JoinSet` + `Semaphore`, reassembles by chunk index at manifest-known offsets so out-of-order arrival is fine. - `claw-cargo build --parallel-restore N` (default 8). `N <= 1` falls through to `BlobGetStream` for parity. - Memory: `total_size + 4 MiB × in-flight` — dominated by the reassembly buffer, not the fanout. +1 test: `parallel_blob_get_reassembles_multi_chunk_blob_byte_equal` covers roundtrip byte-equality vs BlobGetStream, tail-chunk offset, concurrency=1 correctness, and NotFound → None. 259 tests pass (+2). Pre-existing macOS failure unchanged.	2026-07-12 07:42:57 -07:00
Omar Sobh	54e9da4d62	Phase 5k: parallel-fanout chunk transfer for prewarm ... Pilot 2026-07-12 measured 109 MiB/s on the sequential prewarm path — ~11% of a 10G fabric. `quinn::Connection` is cheap-Clone (internal Arc), so we can run the has→get→put pipeline per chunk in concurrent tasks under a bounded semaphore. - `prewarm_missing_chunks_between_parallel(up, down, id, concurrency)` in `rpc/client.rs`. `concurrency <= 1` degrades to the sequential path (kept for diagnostic parity). - `claw-cargo prewarm --parallel N` (default 8). Ignored with `--buffered`. Memory ceiling: 4 MiB × in-flight = 32 MiB @ 8, 128 MiB @ 32. - Uses `tokio::task::JoinSet` + `Arc<Semaphore>`; permit held for the whole per-chunk pipeline so we never over-commit. - Retry pass on `put_manifest` mismatch stays sequential — small, correctness-critical. - Errors: JoinSet drains completely + returns first task error so a mid-fanout failure doesn't leave zombie tasks. +1 test: `end_to_end_parallel_prewarm_copies_chunks_and_matches_sequential` runs 5-chunk payload with concurrency=3, verifies byte-equal restore, then reruns with concurrency=8 → 0 uploads (has_chunk dedup), then concurrency=0 → 0 uploads (sequential fallback path). 254 tests pass (+1 from previous). Pre-existing macOS failure unchanged.	2026-07-12 06:13:28 -07:00
Omar Sobh	6fb16286dd	Phase 5h: streaming chunk-level prewarm ... Bounded-memory cross-peer prewarm: instead of buffering the whole blob in RAM (previous 5f path), iterate the upstream manifest chunk by chunk, ask downstream `HasChunk`, stream missing chunks one at a time. Memory ceiling is 1 chunk (4 MiB) regardless of blob size — a 5 GiB target dir no longer needs 5 GiB of mediator RAM. * rpc/client.rs: new `prewarm_missing_chunks_between(upstream, downstream, blob_id)` helper. Returns `(uploaded, total)` — the difference is the dedup save. Retries once if the downstream `PutManifest` reports missing chunks after our push (guards a narrow eviction race); a second failure surfaces as `Err`. * claw_cargo.rs: `prewarm` now streams by default; new `--buffered` flag for the old whole-blob path (kept for diagnostic comparability during rollout). Human-readable output shows the mode + dedup count. +2 tests: - cold downstream: 3-chunk payload (with a partial tail chunk) is copied exactly, reassembly is byte-equal to source - partial dedup: pre-seed 1 of 3 chunks on downstream → uploaded=2; rerun is a full no-op (uploaded=0), proving idempotence 251 tests pass (+2 from Phase 5j). Pre-existing macOS failure unchanged. Follow-ons: (1) parallel chunk transfer (uploaded chunks in fan-out) would speed multi-GB prewarms further; (2) exposing an accurate transferred-bytes counter needs router-side accounting instead of the current chunk-count × CHUNK_SIZE approximation.	2026-07-12 04:48:55 -07:00
Omar Sobh	d36cec11a6	Phase 5g: cache metrics + GetMetrics RPC + peer-metrics CLI ... Every RPC handler that answers a hit-or-miss question now increments lock-free atomic counters. The GetMetrics RPC (0x13) returns a JSON snapshot of every counter; the new claw-cargo peer-metrics CLI prints hit rates, byte volumes, and counter uptime. Placement engines can now poll these across the fleet to bias runner scheduling toward whichever node has the warmest cache for a given repo/tag combination. ## New module: cluster/metrics.rs (268 lines) Types: - CacheMetrics — atomic counters, all AtomicU64, Relaxed ordering (metrics are advisory, not consistency-critical) - MetricsReply — JSON snapshot returned by GetMetrics Public API: - CacheMetrics::new() — timestamped start, all counters at 0 - record_get_ref_hit / _miss - record_get_tag_hit / _miss - record_blob_get_bytes / record_blob_put_bytes - record_get_chunk_hit / _miss - record_has_chunk_hit / _miss - snapshot() — atomic-load every field into a MetricsReply MetricsReply derived helpers: - get_ref_hit_rate() / get_tag_hit_rate() / has_chunk_hit_rate() — Option<f64> so 0/0 returns None instead of NaN ## RPC method - GetMetrics (0x13): payload = empty; reply = JSON MetricsReply Wire-level instrumentation added to RpcRouter dispatch: - GetRef → record_get_ref_hit / _miss - GetTag → record_get_tag_hit / _miss - BlobGet → record_blob_get_bytes (on hit) - BlobPut → record_blob_put_bytes - HasChunk → record_has_chunk_hit / _miss - GetChunk → record_get_chunk_hit + record_blob_get_bytes on hit / record_get_chunk_miss RpcRouter grows Arc<CacheMetrics> unconditionally — every router has metrics, so a fresh node with no traffic still returns a valid snapshot with all zeros + started_unix. Streaming variants (BlobPutStream / BlobGetStream) don't yet track byte counts — they'd require plumbing the count out of put_stream / stream_to. Follow-on if it turns out to matter for placement. ## Client helper + CLI - call_get_metrics(&conn) → Result<MetricsReply> - claw-cargo peer-metrics [--peer ...] [--peer-addr ...] [--tls-dir ...] Fetches + pretty-prints: counter uptime: 42s GetRef hits/misses: 123 / 45 hit rate: 73.21% GetTag hits/misses: 8 / 2 hit rate: 80.00% HasChunk hits/miss: 512 / 88 hit rate: 85.33% GetChunk hits/miss: 47 / 12 Blob GET bytes: 1.23 GiB Blob PUT bytes: 3.45 GiB human_bytes() helper picks GiB / MiB / KiB / B based on magnitude. Subcommand count now 9: build / prefetch / status / fingerprint / pin / unpin / list-tags / prewarm / peer-metrics. ## Tests (13 new, all real filesystem / real QUIC — no mocks) CacheMetrics (6): - new_starts_all_counters_at_zero_except_timestamp - recorders_increment_the_right_field (every recorder × 1-2 counts) - hit_rates_none_when_zero_events (avoids 0/0 NaN) - hit_rates_compute_correctly (3 hits / 1 miss → 75%) - snapshot_round_trips_through_json - snapshots_across_threads_are_consistent_up_to_relaxed_ordering (8 threads × 1000 increments → exactly 8000) RPC integration (7): - phase_5g_method_byte_encoding - get_metrics_returns_empty_snapshot_before_any_activity - get_ref_records_hit_and_miss_counters (2 hits + 1 miss) - get_tag_records_hit_and_miss_counters - blob_get_and_blob_put_record_byte_counts - has_chunk_and_get_chunk_record_hit_miss_counters - **end_to_end_get_metrics_over_real_quic** — seed activity locally, fire GetRef/GetTag/BlobGet dispatches to move the counters, then fetch metrics through real QUIC + mTLS and verify each field including the 0.5 hit rate calculation 238 tests pass. Pre-existing macOS-only failure unchanged. File sizes (all under 1300-line ceiling): - cluster/metrics.rs: 268 - cluster/rpc.rs: 818 - cluster/rpc/tests_phase5.rs: 905 - claw_cargo.rs: 894 ## What this enables Fleet-wide visibility into which peer is actually serving traffic: # From anywhere with connectivity + fleet mTLS claw-cargo peer-metrics --peer tank claw-cargo peer-metrics --peer architect claw-cargo peer-metrics --peer morpheus Compare hit rates side by side to see which node's cache is warmest. A placement engine can automate this — poll every 30s, feed the scheduler. ## Follow-on - 5h: streaming variant of prewarm (fixed-memory ceiling for many-GB blobs) - 5i: metrics also published via gossip so PeerView carries hit rate without a per-peer GetMetrics roundtrip - 5j: prometheus /metrics endpoint on the daemon for existing dash integrations - 6: FUSE mount for warm-tier git worktrees	2026-07-12 04:06:17 -07:00
Omar Sobh	db55903311	Phase 5f: claw-cargo prewarm — cross-peer cache copy ... The last piece before "Gitea webhook triggers a cache-warm for the CI runner before its build starts." Adds a `prewarm` subcommand that copies a tagged cache from one peer (upstream) to another (downstream) in one shot — same tag, same BlobId, both sides serve it after. ## New subcommand ``` claw-cargo prewarm \ --from-peer tank --from-addr 10.0.0.14:7702 \ --to-peer morpheus --to-addr 10.0.0.15:7702 \ --tls-dir /etc/claw-store/tls \ --pin clawverse:main:latest ``` Flow: 1. Connect to upstream with local mTLS identity 2. `GetTag(tag)` → BlobId; `BlobStat(BlobId)` → size + chunk count 3. `BlobGetStream(BlobId)` → download bytes 4. Connect to downstream (second QUIC endpoint, same identity) 5. `BlobPutStream(bytes)` → returns BlobId; verified equal to upstream's 6. `PutTag(tag → BlobId)` on downstream Summary output shows tag, blob id, both endpoints, byte count, download/upload timings, total wall clock. Assumes upstream + downstream share the same fleet CA (the common case). Mixed-fleet variant with distinct identities is a follow-on. ## Integrity check `assigned_id != blob_id` after the downstream upload triggers a bail — the two BlobIds must match because content is BLAKE3-hashed end-to-end. If they don't, the wire path corrupted bytes and the whole prewarm fails loud rather than silently pinning a bad blob. ## Buffered vs streamed Current implementation buffers the whole blob in memory between download and upload. Fine for cargo target dirs (~1-5 GB compressed); would break for a 20 GB blob. A follow-on will pipe upstream → tokio duplex → downstream to run at fixed memory. ## Tests (1 new, real 2-peer QUIC) - **`end_to_end_prewarm_copies_tagged_blob_between_two_peers`** Two full RpcRouters serving in-process (A upstream + C downstream), each on a distinct port. Seeds A with a blob + tag, then runs the exact sequence prewarm runs internally: `GetTag → BlobGetStream` against A, then `BlobPutStream → PutTag` against C. Verifies that C's blob store returns byte-equal payload and C's tag store now points at the same BlobId. Proves the composition works. ## Housekeeping `rpc/tests.rs` hit 1408 lines with the new prewarm test. Phase 5 tests (5b refs + 5d tags + 5e restore + 5f prewarm) split to `rpc/tests_phase5.rs` via a second `#[path]` module in rpc.rs. Result: - rpc/tests.rs: 727 (phase 1-2d tests) - rpc/tests_phase5.rs: 722 (phase 5 tests) - rpc.rs: 777 - rpc/client.rs: 589 - claw_cargo.rs: 818 - All under ceiling. 225 tests pass. Pre-existing macOS-only failure unchanged. ## What this enables The complete CI runner flow now works end-to-end: ``` Primary (e.g. tank): claw-cargo build # first ever build — MISS, uploads claw-cargo pin --name clawverse:main:latest Fleet control plane on PR open: gitea webhook → shell hook → claw-cargo prewarm \ --from-peer tank --to-peer $RUNNER_LOCAL \ --pin clawverse:main:latest # runner's local daemon now serves the tag + blob Runner picks up job: claw-cargo build --peer 127.0.0.1:7702 # local daemon is warm → prefetch returns HIT # cargo build runs against restored deps → workspace-crates only # 50 min → 3 min ``` Every subcommand claw-cargo needs for this pipeline now exists: build / prefetch / prefetch --pin / status / fingerprint / pin / unpin / list-tags / prewarm. ## What's next - 5g: cache hit/miss metrics into gossip so placement engines can bias runner scheduling toward warm nodes - 5h: streaming variant of prewarm (tokio duplex) for many-GB blobs - 6: FUSE mount for warm-tier git worktrees - 3: full CRDT metadata if the plain-tag model shows conflict problems	2026-07-12 03:57:10 -07:00