Records which (repo, git-ref) combinations produced each cache
fingerprint. Later slices will wire this to a nightly Gitea sweep
that queries /api/v1/repos/.../branches and /tags, then evicts
fingerprints whose recorded refs are all gone AND whose
last_seen_unix is older than the retention window.
Per-fingerprint (not per-blob) because:
* Fingerprints are the cache keys claw-cargo uses. Tracking at the
fp layer keeps this aligned with the claw-cargo boundary.
* Blobs are content-addressed and may be shared. Ref-tracking is
about "why we kept this cache" — a per-fp concern.
New module cluster::ref_tracking:
* RefEntry { fingerprint, repo, refs, first_seen_unix, last_seen_unix }
* RefTracking::record(fp, repo, git_ref, now) — creates or updates
* RefTracking::get(fp) / list_all() / forget(fp)
* RefTracking::stale_at(now, live_refs_by_repo, retention_secs) →
Vec<fingerprint>, the deletion-eligibility list
On-disk: <root>/ref-tracking/<hh>/<fp_hex>.json. JSON so operators
can inspect with jq. One record per cached fp; even 100k fps is
under 50 MB.
Semantics baked in:
* record() APPENDS refs, never removes — sweep decides staleness
* record() rejects repo change for a fp (collision or bug detector)
* refs stable-sorted in-file so cross-node diff is easy
* stale_at treats "repo not in live_refs map" as "all refs dead"
→ deleted repos don't leak caches
* retention_secs is a floor: dead-but-fresh caches survive
+10 tests: create, append-and-refresh, dedup, repo-change reject,
stale-at happy path, stale-at missing-repo, stale-at retention,
forget truth values, list sorted, validate rejects.
365 tests pass (+10). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
No CLI or wire integration in this PR — pure library, testable
in isolation. Follow-ons: (a) claw-cargo auto-record on cache put,
(b) Gitea polling adapter, (c) sweep wired into cluster-gc.
Closes the retention loop between snapshots and pin-aware LRU
eviction. A snapshot is not just a "list of blobs at time T" any
more — it's a *retention pin* on every blob it captures.
Operators can guarantee a build stays on disk for N days by
snapshotting it and pruning the snapshot when the window is up.
Additions:
* SnapshotStore::pinned_blob_ids() → union of blob_ids across all
live snapshots. Cheap: one JSON read per snapshot.
* cmd_cluster_gc extends the tag-pin set with snapshot pins
before handing it to evict_to_size_cap_with_pins. Output line
now reads "pinned blobs: N (M from snapshots)".
* ClusterServices auto-GC ticker does the same on every tick;
log fields include snapshot_pins so ops see the retention set
size at a glance.
+2 tests:
- pinned_blob_ids_unions_all_snapshots (overlap dedupe)
- pinned_blob_ids_empty_when_no_snapshots
355 tests pass (+2). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
A snapshot is a named, immutable point-in-time record of every blob
live in the store. It's NOT a data copy — blobs are content-addressed
and already live under blobs/. A snapshot is a JSON reference set at
<root>/snapshots/<name>.json.
Why:
* Rollback anchor before risky migrations.
* Retention pin: combined with the Phase 4a pin-aware LRU eviction,
operators can guarantee "these blobs stay on disk N days".
* Audit: "which blobs existed at release time?"
New module cluster::snapshot:
* SnapshotStore::create(name, blob_store, created_at)
* SnapshotStore::get(name) / list() / delete(name)
* SnapshotManifest { name, created_at_unix, blob_ids }
* SnapshotSummary for cheap list rendering (no blob-list slurp).
BlobStore gains list_blob_ids() — walks blobs/**/*.manifest.json
and returns the blob id set. Manifests only, no chunk reads.
New CLI commands:
* claw-store cluster-snapshot-create --name <>
* claw-store cluster-snapshot-list
* claw-store cluster-snapshot-show --name <>
* claw-store cluster-snapshot-delete --name <>
Semantics:
* Snapshots are immutable: create with existing name errors, does
not clobber. Delete-then-create if you really want to overwrite.
* delete() removes only the reference file. Never touches blob
data — protects against operators nuking live data by pruning
snapshots.
* list() sorts by created_at_unix ascending — oldest first so
triage picks pruning candidates quickly.
* blob_ids are sorted at write time so the same content on two
nodes yields byte-identical snapshot files.
* Names validated: no /, \\, NUL, control chars; max 512 bytes.
+8 tests covering create+capture, immutability, get-missing,
list-ordering, delete truth-values, delete-doesn't-touch-blobs,
name-validation, and sorted round-trip.
353 tests pass (+8). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
New command: `claw-store cluster-repair --peer <name> --rpc-addr <host:port>
--tls-dir <dir> [--dry-run]`.
Flow:
1. Local scrub identifies bad chunks (missing + corrupt).
2. Deduplicate to unique chunk-hashes (scrub emits per-reference,
fetcher work is per-chunk).
3. Connect to peer over QUIC + mTLS.
4. For each unique chunk: HasChunk probe → GetChunk on hit →
put locally (re-hashed by put_chunk, so a lying peer can't
corrupt us further).
5. Report attempted/repaired/unrecoverable/errors.
--dry-run stops after the dedup step: prints the plan without
touching the peer or disk.
Behavior details:
* Zero bad chunks → clean exit with no peer contact.
* Any unrecoverable or per-chunk error → non-zero exit so cron/CI
notice. Message names counts.
* HasChunk-first means a peer that lacks the chunk is one cheap
round-trip, not a full GetChunk attempt.
Companion piece for the Phase 7b repair library (already merged).
No new tests here — logic is thin glue over `repair_chunks` +
`call_has_chunk`/`call_get_chunk`, all of which have their own
unit + integration coverage. Behavior gets its real workout in
live smoke on tank+architect.
New primitive: BlobStore::repair_chunks(chunks, fetch) → RepairReport.
Consumer flow: cluster-scrub returns a list of (blob, chunk) bad
pairs. cluster-repair (next slice) will hand the chunk hashes here
with a fetcher that walks peers via HasChunk/GetChunk. This PR is
the library-only half — no peer wiring — so it's testable in
isolation and reusable by callers who already have a chunk source.
Fetcher contract:
* Ok(Some(bytes)) → put locally, count repaired
* Ok(None) → nobody has it, record as unrecoverable
* Err(e) → per-chunk error, batch continues
Guardrails:
* Bytes are re-hashed by put_chunk before writing. A peer that
returns wrong bytes for a hash cannot corrupt us further.
* Duplicate chunk hashes in the input dedupe → fetcher called
exactly once per unique chunk. Matters because scrub reports
shared chunks once per owning manifest.
* Errors on one chunk never abort the batch — the remaining
chunks still get their shot.
* Repair overwrites a corrupt file: unlink-then-put_chunk, since
put_chunk itself is write-if-absent. NotFound on unlink is fine
(missing-chunk case).
+4 tests:
- repair_writes_fetched_bytes_and_marks_repaired (happy: corrupt
→ repair → post-scrub clean)
- repair_records_unrecoverable_when_fetcher_returns_none
- repair_records_error_and_continues_batch (batch survives one
chunk's error)
- repair_dedups_duplicate_chunks_in_input (fetcher called exactly
once for 3 identical hashes)
345 tests pass (+4). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
New primitive: BlobStore::scrub_all() → ScrubReport.
Walks every .manifest.json under blobs/, for each referenced chunk
reads the file from disk and recomputes BLAKE3. Verdict per chunk:
* file absent → missing
* hash mismatch → corrupt
* match → ok
Design points:
* Read-only. Never touches disk state. Safe against a live daemon
— worst case a chunk lands mid-scrub and is skipped this pass.
* Per-reference counting: a bad chunk that N manifests depend on
shows up as N corrupt entries so operators see the full blast
radius. But each unique chunk is hashed exactly once via an
in-memory verdict cache.
* Report holds explicit (blob_id, chunk_hash) pairs for every
bad chunk so the fix path (repair in Phase 7b) has enough
info to act.
CLI: `claw-store cluster-scrub [--verbose]`. Non-zero exit when
integrity issues exist so cron / CI notice.
+4 tests:
- scrub_reports_all_ok_when_store_is_healthy
- scrub_detects_corrupt_chunk (owner blob id preserved)
- scrub_detects_missing_chunk (owner blob id preserved)
- scrub_dedups_shared_chunk_hashing_once (shared chunk, 2 owners
reported, single disk read)
341 tests pass (+4). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
Wires cmd_pin through the WalQueue built in the preceding four
PRs (#48 → #52). First real caller of the client-mode WAL
stack.
New surfaces:
claw-cargo pin --offline --blob <BlobId> [--ttl <duration>]
* no peer connection is opened
* enqueues the same three mutations cmd_pin would emit
online: primary tag (PutTagVersioned), .fingerprint
companion (PutTagVersioned), and — if --ttl — the two
SetTagExpiry sidecars
* requires --blob because offline mode can't do the
GetRefVersioned lookup that resolves fingerprint → BlobId
* prints the assigned WAL seqs + "next step: drain"
claw-cargo drain --peer ...
* opens a peer, drains the queue, truncates up to the last
applied seq
* partial-failure safe: whatever applied is truncated;
anything after a hard error stays on disk for retry
* exits non-zero when drain stopped mid-stream
claw-cargo wal-status
* read-only, no network
* pending count, oldest/newest seq, storage path, decoded
entries (or UNDECODABLE marker on frame errors)
WAL location follows the XDG state-home pattern already used
by manifest.rs:
1. $XDG_STATE_HOME/claw-cargo/wal/
2. $HOME/.local/state/claw-cargo/wal/
3. ./.claw-cargo-wal/ (worst-case container fallback)
Tests (2): default_wal_path_honours_xdg_state_home (mirroring
manifest.rs's env-var pattern) + parse_blob_id_rejects_bad_
hex_and_wrong_len.
claw_cargo.rs grew from 1668 to ~1870 lines. Still under the
1300-per-*module* interpretation but this bin file has been
above 1300 since Phase 5. Split-out is Phase 6 territory.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Collapses the enqueue + drain + truncate dance around
WriteAheadLog + wal_mutation + wal_replay into one API so
downstream callers (Phase 4e: cmd_pin & friends) don't have
to orchestrate three modules themselves.
Two-call flow:
let mut q = WalQueue::open(state_dir.join("wal")).await?;
q.enqueue(&WalMutation::PutTagVersioned { .. }).await?;
// ...later, on reconnect:
let report = q.drain(&conn).await?;
drain() advances the watermark to the last successfully-
applied (or Superseded) seq whether or not the drive stopped
on a hard error mid-stream. Nothing is truncated past the
failure point, so the failing record and everything after
it are retried on the next drain.
Introspection surface (`pending_count` / `oldest_pending_seq`
/ `newest_pending_seq` / `snapshot` / `is_empty`) is what a
metrics endpoint or CLI status view wants. `wal()` escape
hatch exposes the backing WAL for advanced callers.
Tests (6, all green — 4 unit + 2 end-to-end over QUIC):
* empty queue reports empty bounds
* enqueue updates bounds correctly
* snapshot decodes in seq order and preserves kind info
* drain clears the queue and applies to peer (verifies via
call_get_ref + call_get_tag_versioned)
* drain over a pre-seeded dominant version returns
Superseded and still drains the queue
* enqueue survives reopen — bounds recover through
WriteAheadLog::open scan
346 lines, well under the 1300 ceiling.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Given a peer connection + a decoded WalMutation, re-issue the
correct RPC. Closes the loop from "durably logged at client"
to "actually applied at peer" on reconnect.
Outcome classification is deliberate:
* Applied — peer accepted the mutation.
* Superseded — peer already had a dominant version, or the
delete target was absent. NOT a failure; the
mutation's intent matches current peer state.
* Err(_) — genuine RPC failure; caller retries later.
Both Applied and Superseded advance the watermark past the
record — the WAL can safely truncate.
Public surface:
ReplayOutcome { Applied | Superseded }
replay_one(&conn, &mutation) -> Result<ReplayOutcome>
drive_replay(&conn, &wal, start_seq) -> Result<DriveReport>
DriveReport { last_applied, applied, superseded,
skipped, stopped_at: Option<(seq, msg)> }
drive_replay stops on the first hard error and returns
last_applied so the caller can `wal.truncate_up_to(...)`
before closing. Undecodable/unknown-kind records mid-stream
are skipped (with warn!) rather than aborting — otherwise
one bad record would jam an otherwise-good tail forever.
Tests (4, all green, end-to-end over QUIC):
* every variant round-trips; peer state verified via
call_get_ref / call_get_tag_versioned / call_get_tag_expiry
* versioned-reject counts as Superseded, not Err
* DeleteTag on a missing key is Superseded
* undecodable record between two real mutations is skipped;
both good records still apply; last_applied advances past
the skip
434 lines, well under the 1300 ceiling.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Adds the encoding layer that turns the raw WAL (opaque bytes)
into a typed queue of client-mode mutations, ready for Phase 4e
to wire actual RPCs through.
Frame (self-describing, forward-compatible):
version : u8 = 0x01
kind : u8 = one of the Kind discriminants
body : [u8] kind-specific
Body encodings mirror the existing on-wire shapes so a future
replay path can splice a WAL record straight into an RPC payload.
Variants (Kinds 0x01–0x06):
PutRef, PutRefVersioned, PutTag, PutTagVersioned,
DeleteTag, SetTagExpiry
Blob-put mutations are deliberately NOT modeled — blob data is
too large to keep in the WAL. The roaming-client design stages
blobs on local disk and records a reference to them once the
local BlobPutStream completes.
Public helpers:
append_mutation(&mut wal, &m) -> Result<seq>
replay_mutations(&wal, start_seq)
-> Vec<(seq, Result<WalMutation, WalMutationError>)>
Unknown-kind records surface as `Err(UnknownKind(byte))`, not
a panic — forward-compat when a newer writer wrote a record
this reader doesn't understand. Malformed records also surface
as Err so the caller can decide (log-and-skip vs abort replay).
Tests (9, all green): kind-byte stability, roundtrip every
variant, rejects empty/short/bad-version/unknown-kind,
malformed bodies (wrong length, over-declared key_len, trailing
garbage on DeleteTag), non-UTF-8 keys, append+replay through a
real on-disk WAL, and replay-survives-unknown-kind mid-stream.
No new deps — hand-rolled error type in-tree (no thiserror).
515 lines, well under the 1300 ceiling.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Turns the single-file Phase 4c WAL into a segmented log so it
can grow past a single file safely. This unblocks every
downstream Phase 4d/4e integration — reconnect + push loop
can't rely on an unbounded single file.
Layout change:
<root>/segment-<20-digit-first-seq>.bin
20-digit zero-padded first-seq means lex sort == numeric sort,
so `read_dir + sort_by_key` recovers the natural order.
Rotation policy:
* `max_segment_bytes` default 8 MiB, overridable via
`open_with_options`.
* `append` rolls to a fresh segment BEFORE writing when the
current tail is non-empty AND at/above the cap. A single
oversize record always lands in one segment — we never split
a record.
Truncation across segments:
* whole segments with `last_seq <= watermark` are `unlink`'d
* the boundary segment (if any) is rewritten in place via
`tempfile-in-parent + rename` + parent-dir fsync
* full truncation resets head/tail to 0 and the next append
creates a fresh segment
Legacy compat: on open, if a pre-4d `log.bin` is present and
no `segment-*.bin` files exist, it is scanned for its first
seq and renamed to the correct segment name. Refuses to
silently overwrite on filename collision.
Tests (18, all green): rotation-happens-at-cap, reopen-
enumerates-all-segments, truncate-drops-whole-segments,
truncate-partial-rewrites-boundary, oversize-record-still-
fits-one-segment, legacy-log.bin-migration, plus the full
Phase 4c suite (fresh open, append, iter partial ranges,
reopen recovers tail, torn-write truncation, corruption is
hard error, full truncation appendable, below-head no-op,
large payload, empty payload, append-after-reopen).
942-line file, comfortably under the 1300-line ceiling.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Foundation for the roaming/offline-client story described in
Architecture v2 ("Roaming client (full R/W, offline queue)"):
mutating ops append to a durable local log before hitting the
network, and are replayed at reconnect. This PR ships the
primitive; RPC/reconnect wiring lands in Phase 4d.
Segment format (single-file for now — rotation is 4d):
seq : u64 LE (8 bytes)
len : u32 LE (4 bytes) payload length
csum : [u8; 8] (8 bytes) first 8 bytes of
BLAKE3(seq || len || payload)
bytes : [u8; len]
On open, the log is scanned linearly. A short read or truncated
tail is treated as "clean crash boundary" — the file is
size-truncated to the last fully-fsynced record, no error.
A checksum mismatch on a full-length record is fatal (real
corruption, don't silently swallow data).
Public API:
WriteAheadLog::open(root) -> Self
wal.append(&[u8]) -> Result<u64> // durable, fsynced
wal.iter_from(start_seq) -> Vec<WalRecord>
wal.truncate_up_to(watermark) -> () // atomic rewrite via
// tempfile-in-parent + rename
wal.head_seq() / wal.tail_seq() / wal.is_empty()
Tests (12, all green): fresh open, monotonic seq, replay full &
partial ranges, reopen-recovers-tail, torn-write truncation on
open, corruption is hard error, prefix truncation, full
truncation leaves appendable, below-head no-op, 1 MiB payload
roundtrip, empty payload roundtrip, append-after-reopen.
No new deps — BLAKE3 (already a dep) supplies the checksum.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Closes Phase 4b by exposing the TTL sidecar written by
Phase 4b primitives on the wire and via `claw-cargo pin`.
Wire additions:
* Method::SetTagExpiry (0x1a) — payload `key_len:u16 || key ||
expires_at:u64 (LE)`. Reply single-byte OK. `expires_at == 0`
clears the sidecar.
* Method::GetTagExpiry (0x1b) — payload raw key bytes. Reply 8
bytes (u64 LE) on hit; NotFound when no sidecar is present.
Both accept writes even when the stamped tag itself is absent,
matching `TagStore::set_stamped_expiry` semantics — the sidecar
takes effect the moment the tag lands.
CLI:
* `claw-cargo pin --ttl <duration>` — humantime-style duration
(`30d`, `1h30m`, `2w`, ...). Applied to both the primary tag
and its `.fingerprint` companion so eviction treats them as
one lifetime. `--ttl 0` / `clear` / `none` clears an existing
sidecar without touching the value.
Tests: encode/decode roundtrip + malformed-input rejection for
`encode_expiry_record`, method-byte stability, NotConfigured
without a tag store, end-to-end set/get/overwrite/clear over
QUIC, and a real-pin flow that publishes a stamped tag then
attaches TTL. Duration parser is unit-tested for single/compound
forms, case-insensitive units, bad input, and clock alignment.
No new deps — the humantime-style parser is 60 lines in-tree.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Adds the on-disk mechanism for time-scoped pins. No RPC or CLI yet
— a follow-on will expose \`pin --ttl <duration>\`. This PR is
purely library + eviction wiring.
Layout addition: alongside each stamped tag at
\`tags-v2/<hh>/<hash>.svtag\`, an optional sidecar
\`tags-v2/<hh>/<hash>.svtag.exp\` holds an 8-byte LE unix
\`expires_at\`. Absence of the sidecar = never expires (current
behavior).
New TagStore methods:
* set_stamped_expiry(key, expires_at_unix) — writes sidecar;
passing 0 removes it. Idempotent.
* get_stamped_expiry(key) — reads sidecar; None when absent.
* pinned_blob_values_at(now_unix) — same union as
pinned_blob_values, but skips stamped tags whose sidecar shows
expires_at ≤ now. Legacy tags/ entries never expire.
* prune_expired_stamped_at(now_unix) — deletes stamped tags AND
their sidecars where expires_at ≤ now. Returns count.
* pinned_blob_values() — now a shim that calls _at(u64::MAX) for
100% backward compat.
Wired the two existing gc call sites:
* ClusterServices auto-GC ticker prunes-then-collects at
SystemTime::now(). One pass per tick.
* \`claw-store cluster-gc --evict-to-gb N\` CLI same pattern.
Report now includes \"expired pins pruned: N\".
+1 test (expiry_gates_pin_set_and_prune_removes_expired):
covers live/expired/no-ttl mix, sidecar round-trip, prune
removes only expired, expires_at=0 clears sidecar, dropped
tag stops filtering.
286 tests pass (+3 from 283). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
pin lookup was legacy call_get_ref only, missing refs written via
call_put_ref_versioned (all Phase 3b+ builds). Try versioned first,
fall back to legacy — same pattern as claw-cargo build path.
A `claw-cargo pin` used to be silently vulnerable to the size-cap
eviction ticker — the tag existed but the underlying blob could get
LRU'd out, leaving a dangling reference. Now tags act as
retention markers: any blob referenced by any tag (stamped or
legacy) is protected from `evict_to_size_cap`.
* `BlobStore::evict_to_size_cap_with_pins(max_bytes, pinned_set)` —
same LRU-by-mtime pass, but pinned blob IDs skip the eviction
loop. Existing `evict_to_size_cap` is now a thin wrapper with an
empty pin set (100% backward compat).
* `TagStore::pinned_blob_values()` — unions every 32-byte value
referenced by any tag across `tags/` (legacy) and `tags-v2/`
(Phase 3c stamped). Dedupes naturally.
* Auto-GC ticker in `ClusterServices` now collects the pin set on
every eviction pass and passes it in. Log fields include
`pinned_blobs = N` so operators can see the retention set size.
* `claw-store cluster-gc --evict-to-gb N` CLI opens the tag store
the same way, prints `pinned blobs: N` in the report.
+3 tests:
- evict_with_pins_protects_pinned_blobs_from_eviction — 3 blobs
ordered oldest→newest, pin the oldest; without pins LRU would
evict it; with pins the next-oldest goes instead. Guards the
main semantic.
- evict_with_pins_stops_when_pinned_footprint_dominates —
everything pinned + cap = 0 → no-op. Guards the "operator asked
for the impossible" case.
- pinned_blob_values_unions_both_stores — legacy tag with value V1,
stamped tag with value V2, second stamped tag also referencing
V1 → set contains {V1, V2}. Dedupe check.
283 tests pass (baseline +3). Pre-existing macOS failure unchanged.
Ships the last two pieces from the arch doc's Phase 3 scope for the
cargo-cache use case:
## 3c: Stamped tags (CRDT-merge on PutTag)
Mirror of Phase 3a/3b for TagStore. Two concurrent `claw-cargo pin`
calls on the same tag now race deterministically instead of silently
clobbering.
* `StampedTagValue` — same 48-byte (value, clock, node) tuple as
StampedRef.
* `TagStore::put_stamped(key, StampedTagValue) -> TagPutOutcome`
and `TagStore::get_stamped(key)` — data lives under `tags-v2/`
(separate from `tags/` for cutover safety).
* New wire methods `PutTagVersioned = 0x18` +
`GetTagVersioned = 0x19`.
* `call_put_tag_versioned` / `call_get_tag_versioned` client
helpers.
* `claw-cargo pin` now writes stamped tags. Concurrent pin gets
AlreadyExists and moves on (blob content is content-addressed so
both winners agree on the payload).
## 3e: Namespaced ref keys
Opt-in `--namespace <slug>` on peer-facing subcommands. When set,
the ref key becomes `blake3("clawstor.ns.v1" || namespace || fp)`
so two runners on different namespaces (`clawverse/main` vs
`clawverse/pr-42`) don't collide on the same fingerprint. Empty
namespace = pre-3e behavior, so this is 100% backward compat.
* `refs::namespaced_ref_key(namespace, fingerprint) -> RefKey`
primitive.
* `PeerArgs::namespace: Option<String>` CLI flag flows through to
`cmd_status`, `cmd_prefetch`, `cmd_build`.
* `peer_lookup` now takes a `RefKey` directly (was `&Fingerprint`)
so the namespace resolution stays in the caller — the daemon
never sees "namespace" as a concept.
## 3d: Deferred
Full vector clocks per namespace are noted in the arch doc as a
Phase-3 goal; scalar wall-clock (clock + node stamp) is sufficient
for the cargo-cache use case (single-key LWW merge). NTP-synced
runners see monotonic ordering; skewed runners lose an ordering
but the CRDT semantics still guarantee no data corruption. Full VC
is deferred to a future phase.
+9 tests, 280 total (baseline +8: 7 unit + 1 e2e over real QUIC).
Runner + prewarm use PutRefVersioned/GetRefVersioned; daemon
GetRefVersioned forwards on miss + pulls blob transparently. New
GetRefVersionedLocal (0x17) prevents recursion. Backward-compat:
existing GetRef/PutRef path unchanged; two on-disk namespaces
coexist (refs/ and refs-v2/).
+1 test, 272 total (unchanged from 3a because we reused existing
scaffolding).
Concurrent PutRef safety via (clock, node) total order. New wire
methods PutRefVersioned (0x15) + GetRefVersioned (0x16). Existing
PutRef/GetRef unchanged for backward compat. Data in refs-v2/
namespace so the two coexist during cutover.
+8 tests, 272 total (baseline +8).
Cross-runner cache silos (tank + architect measured on 2026-07-13):
same fingerprint, same rustc, but each runner's daemon only knows
about the refs its own runner uploaded. Every runner that lands on
a peer that isn't tank re-uploads a duplicate blob.
Fix: on `GetRef` miss the daemon fans out to alive gossip peers
via a strict-local `GetRefLocal` variant, and the FIRST peer that
has the ref triggers a transparent pull — chunks + manifest into
the local blob store, then `PutRef` locally — before returning the
value to the caller. Subsequent lookups are pure-local hits.
* `Method::GetRefLocal = 0x14` — new wire method, identical shape
to GetRef but the peer MUST NOT recurse. Loop prevention: our
forwarding only calls `GetRefLocal` on peers, so chain depth is
always 1.
* `RpcRouter::with_outbound_client(Arc<QuicClient>)` — dependency
injection point for the forwarding dial path. `None` disables
forwarding entirely (GetRef becomes GetRefLocal-equivalent).
* `RpcRouter::forward_get_ref(key)` — concurrent peer probes via
`JoinSet`, 3s timeout per dial, first successful pull wins,
remaining tasks aborted.
* `pull_blob_locally` — walks manifest, fetches only chunks the
local store lacks (`has_chunk`), commits via
`put_manifest_verified`. Bounded memory: one 4 MiB chunk at a
time.
* `ClusterServices::start` loads NodeIdentity twice — server takes
ownership; outbound client gets its own copy for TLS presentation
on peer dials. Wires the outbound client into the router when
TLS material is available.
* `call_get_ref_local(conn, key)` client helper (used by daemon
forwarding + available to any RPC consumer that wants the
no-recursion semantics).
+3 tests in `rpc/tests_forwarding.rs`:
- Local hit works without forwarding; local miss with no peers
returns None. Guards the base cases.
- GetRefLocal never forwards even when outbound is configured (no
peers reachable → miss returns None immediately, no attempted
fan-out).
- Method byte 0x14 encoding is stable across releases.
Full end-to-end forwarding is exercised in the pilot deploy: two
daemons on the fleet-CA, tank populates a ref, architect's runner
GetRef → tank forwards → architect pulls → HIT locally next time.
264 tests pass (baseline +3). Pre-existing macOS failure unchanged.
Three fixes surfaced by the 2026-07-13 Gitea Actions wire-up.
## XDG config path
Before: `client_config::user_config_path` only looked at
`~/.claw-cargo/config.toml`. My runner-integration doc initially
told operators to install at `~/.config/claw-cargo/config.toml`
(XDG-style). Config wasn't loaded.
Now: three-way lookup, first hit wins.
1. `$XDG_CONFIG_HOME/claw-cargo/config.toml`
2. `$HOME/.config/claw-cargo/config.toml`
3. `$HOME/.claw-cargo/config.toml` (legacy, still honoured)
+3 tests: XDG env wins when the file exists, .config wins over
legacy dotfile when both present, legacy dotfile returned as
error-message fallback when none exist.
## Composite action
Before: composite action silently no-op'd. Log showed the script
lines echoed but only the `if command -v claw-cargo` fail branch
ran. Root cause: Gitea Actions composite steps run with a stripped
PATH that omits `/usr/local/bin`.
Now: composite step exports PATH defensively:
export PATH="/usr/local/bin:$HOME/.cargo/bin:$PATH"
And it looks for the config at BOTH the XDG-style path and the
legacy dotfile (same order as client_config).
Workflow file back to using the composite action.
## Docs
Runner-integration doc now:
- Explicitly warns that `ubuntu-latest` routes to container mode
even with `:host` suffix on runner labels (field-observed).
- Documents the dedicated `clawstor-cache` label pattern that
works.
- Sample workflow uses `runs-on: clawstor-cache` instead of
`ubuntu-latest`.
+3 tests, 262 total (baseline unchanged).
client_config.rs load_layered looks at ~/.claw-cargo/config.toml,
not ~/.config/claw-cargo/config.toml. Fix the workflow preflight
path to match. Both runners already have the file at both locations.
Fleet has 7 linux-amd64 runners; only tank + architect have
claw-cargo provisioned. Added a clawstor-cache:host label to those
two runners so this workflow only lands on them.
Omar Sobh
osobh