New primitive: BlobStore::scrub_all() → ScrubReport.
Walks every .manifest.json under blobs/, for each referenced chunk
reads the file from disk and recomputes BLAKE3. Verdict per chunk:
* file absent → missing
* hash mismatch → corrupt
* match → ok
Design points:
* Read-only. Never touches disk state. Safe against a live daemon
— worst case a chunk lands mid-scrub and is skipped this pass.
* Per-reference counting: a bad chunk that N manifests depend on
shows up as N corrupt entries so operators see the full blast
radius. But each unique chunk is hashed exactly once via an
in-memory verdict cache.
* Report holds explicit (blob_id, chunk_hash) pairs for every
bad chunk so the fix path (repair in Phase 7b) has enough
info to act.
CLI: `claw-store cluster-scrub [--verbose]`. Non-zero exit when
integrity issues exist so cron / CI notice.
+4 tests:
- scrub_reports_all_ok_when_store_is_healthy
- scrub_detects_corrupt_chunk (owner blob id preserved)
- scrub_detects_missing_chunk (owner blob id preserved)
- scrub_dedups_shared_chunk_hashing_once (shared chunk, 2 owners
reported, single disk read)
341 tests pass (+4). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
Wires cmd_pin through the WalQueue built in the preceding four
PRs (#48 → #52). First real caller of the client-mode WAL
stack.
New surfaces:
claw-cargo pin --offline --blob <BlobId> [--ttl <duration>]
* no peer connection is opened
* enqueues the same three mutations cmd_pin would emit
online: primary tag (PutTagVersioned), .fingerprint
companion (PutTagVersioned), and — if --ttl — the two
SetTagExpiry sidecars
* requires --blob because offline mode can't do the
GetRefVersioned lookup that resolves fingerprint → BlobId
* prints the assigned WAL seqs + "next step: drain"
claw-cargo drain --peer ...
* opens a peer, drains the queue, truncates up to the last
applied seq
* partial-failure safe: whatever applied is truncated;
anything after a hard error stays on disk for retry
* exits non-zero when drain stopped mid-stream
claw-cargo wal-status
* read-only, no network
* pending count, oldest/newest seq, storage path, decoded
entries (or UNDECODABLE marker on frame errors)
WAL location follows the XDG state-home pattern already used
by manifest.rs:
1. $XDG_STATE_HOME/claw-cargo/wal/
2. $HOME/.local/state/claw-cargo/wal/
3. ./.claw-cargo-wal/ (worst-case container fallback)
Tests (2): default_wal_path_honours_xdg_state_home (mirroring
manifest.rs's env-var pattern) + parse_blob_id_rejects_bad_
hex_and_wrong_len.
claw_cargo.rs grew from 1668 to ~1870 lines. Still under the
1300-per-*module* interpretation but this bin file has been
above 1300 since Phase 5. Split-out is Phase 6 territory.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Collapses the enqueue + drain + truncate dance around
WriteAheadLog + wal_mutation + wal_replay into one API so
downstream callers (Phase 4e: cmd_pin & friends) don't have
to orchestrate three modules themselves.
Two-call flow:
let mut q = WalQueue::open(state_dir.join("wal")).await?;
q.enqueue(&WalMutation::PutTagVersioned { .. }).await?;
// ...later, on reconnect:
let report = q.drain(&conn).await?;
drain() advances the watermark to the last successfully-
applied (or Superseded) seq whether or not the drive stopped
on a hard error mid-stream. Nothing is truncated past the
failure point, so the failing record and everything after
it are retried on the next drain.
Introspection surface (`pending_count` / `oldest_pending_seq`
/ `newest_pending_seq` / `snapshot` / `is_empty`) is what a
metrics endpoint or CLI status view wants. `wal()` escape
hatch exposes the backing WAL for advanced callers.
Tests (6, all green — 4 unit + 2 end-to-end over QUIC):
* empty queue reports empty bounds
* enqueue updates bounds correctly
* snapshot decodes in seq order and preserves kind info
* drain clears the queue and applies to peer (verifies via
call_get_ref + call_get_tag_versioned)
* drain over a pre-seeded dominant version returns
Superseded and still drains the queue
* enqueue survives reopen — bounds recover through
WriteAheadLog::open scan
346 lines, well under the 1300 ceiling.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Given a peer connection + a decoded WalMutation, re-issue the
correct RPC. Closes the loop from "durably logged at client"
to "actually applied at peer" on reconnect.
Outcome classification is deliberate:
* Applied — peer accepted the mutation.
* Superseded — peer already had a dominant version, or the
delete target was absent. NOT a failure; the
mutation's intent matches current peer state.
* Err(_) — genuine RPC failure; caller retries later.
Both Applied and Superseded advance the watermark past the
record — the WAL can safely truncate.
Public surface:
ReplayOutcome { Applied | Superseded }
replay_one(&conn, &mutation) -> Result<ReplayOutcome>
drive_replay(&conn, &wal, start_seq) -> Result<DriveReport>
DriveReport { last_applied, applied, superseded,
skipped, stopped_at: Option<(seq, msg)> }
drive_replay stops on the first hard error and returns
last_applied so the caller can `wal.truncate_up_to(...)`
before closing. Undecodable/unknown-kind records mid-stream
are skipped (with warn!) rather than aborting — otherwise
one bad record would jam an otherwise-good tail forever.
Tests (4, all green, end-to-end over QUIC):
* every variant round-trips; peer state verified via
call_get_ref / call_get_tag_versioned / call_get_tag_expiry
* versioned-reject counts as Superseded, not Err
* DeleteTag on a missing key is Superseded
* undecodable record between two real mutations is skipped;
both good records still apply; last_applied advances past
the skip
434 lines, well under the 1300 ceiling.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Adds the encoding layer that turns the raw WAL (opaque bytes)
into a typed queue of client-mode mutations, ready for Phase 4e
to wire actual RPCs through.
Frame (self-describing, forward-compatible):
version : u8 = 0x01
kind : u8 = one of the Kind discriminants
body : [u8] kind-specific
Body encodings mirror the existing on-wire shapes so a future
replay path can splice a WAL record straight into an RPC payload.
Variants (Kinds 0x01–0x06):
PutRef, PutRefVersioned, PutTag, PutTagVersioned,
DeleteTag, SetTagExpiry
Blob-put mutations are deliberately NOT modeled — blob data is
too large to keep in the WAL. The roaming-client design stages
blobs on local disk and records a reference to them once the
local BlobPutStream completes.
Public helpers:
append_mutation(&mut wal, &m) -> Result<seq>
replay_mutations(&wal, start_seq)
-> Vec<(seq, Result<WalMutation, WalMutationError>)>
Unknown-kind records surface as `Err(UnknownKind(byte))`, not
a panic — forward-compat when a newer writer wrote a record
this reader doesn't understand. Malformed records also surface
as Err so the caller can decide (log-and-skip vs abort replay).
Tests (9, all green): kind-byte stability, roundtrip every
variant, rejects empty/short/bad-version/unknown-kind,
malformed bodies (wrong length, over-declared key_len, trailing
garbage on DeleteTag), non-UTF-8 keys, append+replay through a
real on-disk WAL, and replay-survives-unknown-kind mid-stream.
No new deps — hand-rolled error type in-tree (no thiserror).
515 lines, well under the 1300 ceiling.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Turns the single-file Phase 4c WAL into a segmented log so it
can grow past a single file safely. This unblocks every
downstream Phase 4d/4e integration — reconnect + push loop
can't rely on an unbounded single file.
Layout change:
<root>/segment-<20-digit-first-seq>.bin
20-digit zero-padded first-seq means lex sort == numeric sort,
so `read_dir + sort_by_key` recovers the natural order.
Rotation policy:
* `max_segment_bytes` default 8 MiB, overridable via
`open_with_options`.
* `append` rolls to a fresh segment BEFORE writing when the
current tail is non-empty AND at/above the cap. A single
oversize record always lands in one segment — we never split
a record.
Truncation across segments:
* whole segments with `last_seq <= watermark` are `unlink`'d
* the boundary segment (if any) is rewritten in place via
`tempfile-in-parent + rename` + parent-dir fsync
* full truncation resets head/tail to 0 and the next append
creates a fresh segment
Legacy compat: on open, if a pre-4d `log.bin` is present and
no `segment-*.bin` files exist, it is scanned for its first
seq and renamed to the correct segment name. Refuses to
silently overwrite on filename collision.
Tests (18, all green): rotation-happens-at-cap, reopen-
enumerates-all-segments, truncate-drops-whole-segments,
truncate-partial-rewrites-boundary, oversize-record-still-
fits-one-segment, legacy-log.bin-migration, plus the full
Phase 4c suite (fresh open, append, iter partial ranges,
reopen recovers tail, torn-write truncation, corruption is
hard error, full truncation appendable, below-head no-op,
large payload, empty payload, append-after-reopen).
942-line file, comfortably under the 1300-line ceiling.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Foundation for the roaming/offline-client story described in
Architecture v2 ("Roaming client (full R/W, offline queue)"):
mutating ops append to a durable local log before hitting the
network, and are replayed at reconnect. This PR ships the
primitive; RPC/reconnect wiring lands in Phase 4d.
Segment format (single-file for now — rotation is 4d):
seq : u64 LE (8 bytes)
len : u32 LE (4 bytes) payload length
csum : [u8; 8] (8 bytes) first 8 bytes of
BLAKE3(seq || len || payload)
bytes : [u8; len]
On open, the log is scanned linearly. A short read or truncated
tail is treated as "clean crash boundary" — the file is
size-truncated to the last fully-fsynced record, no error.
A checksum mismatch on a full-length record is fatal (real
corruption, don't silently swallow data).
Public API:
WriteAheadLog::open(root) -> Self
wal.append(&[u8]) -> Result<u64> // durable, fsynced
wal.iter_from(start_seq) -> Vec<WalRecord>
wal.truncate_up_to(watermark) -> () // atomic rewrite via
// tempfile-in-parent + rename
wal.head_seq() / wal.tail_seq() / wal.is_empty()
Tests (12, all green): fresh open, monotonic seq, replay full &
partial ranges, reopen-recovers-tail, torn-write truncation on
open, corruption is hard error, prefix truncation, full
truncation leaves appendable, below-head no-op, 1 MiB payload
roundtrip, empty payload roundtrip, append-after-reopen.
No new deps — BLAKE3 (already a dep) supplies the checksum.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Closes Phase 4b by exposing the TTL sidecar written by
Phase 4b primitives on the wire and via `claw-cargo pin`.
Wire additions:
* Method::SetTagExpiry (0x1a) — payload `key_len:u16 || key ||
expires_at:u64 (LE)`. Reply single-byte OK. `expires_at == 0`
clears the sidecar.
* Method::GetTagExpiry (0x1b) — payload raw key bytes. Reply 8
bytes (u64 LE) on hit; NotFound when no sidecar is present.
Both accept writes even when the stamped tag itself is absent,
matching `TagStore::set_stamped_expiry` semantics — the sidecar
takes effect the moment the tag lands.
CLI:
* `claw-cargo pin --ttl <duration>` — humantime-style duration
(`30d`, `1h30m`, `2w`, ...). Applied to both the primary tag
and its `.fingerprint` companion so eviction treats them as
one lifetime. `--ttl 0` / `clear` / `none` clears an existing
sidecar without touching the value.
Tests: encode/decode roundtrip + malformed-input rejection for
`encode_expiry_record`, method-byte stability, NotConfigured
without a tag store, end-to-end set/get/overwrite/clear over
QUIC, and a real-pin flow that publishes a stamped tag then
attaches TTL. Duration parser is unit-tested for single/compound
forms, case-insensitive units, bad input, and clock alignment.
No new deps — the humantime-style parser is 60 lines in-tree.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Adds the on-disk mechanism for time-scoped pins. No RPC or CLI yet
— a follow-on will expose \`pin --ttl <duration>\`. This PR is
purely library + eviction wiring.
Layout addition: alongside each stamped tag at
\`tags-v2/<hh>/<hash>.svtag\`, an optional sidecar
\`tags-v2/<hh>/<hash>.svtag.exp\` holds an 8-byte LE unix
\`expires_at\`. Absence of the sidecar = never expires (current
behavior).
New TagStore methods:
* set_stamped_expiry(key, expires_at_unix) — writes sidecar;
passing 0 removes it. Idempotent.
* get_stamped_expiry(key) — reads sidecar; None when absent.
* pinned_blob_values_at(now_unix) — same union as
pinned_blob_values, but skips stamped tags whose sidecar shows
expires_at ≤ now. Legacy tags/ entries never expire.
* prune_expired_stamped_at(now_unix) — deletes stamped tags AND
their sidecars where expires_at ≤ now. Returns count.
* pinned_blob_values() — now a shim that calls _at(u64::MAX) for
100% backward compat.
Wired the two existing gc call sites:
* ClusterServices auto-GC ticker prunes-then-collects at
SystemTime::now(). One pass per tick.
* \`claw-store cluster-gc --evict-to-gb N\` CLI same pattern.
Report now includes \"expired pins pruned: N\".
+1 test (expiry_gates_pin_set_and_prune_removes_expired):
covers live/expired/no-ttl mix, sidecar round-trip, prune
removes only expired, expires_at=0 clears sidecar, dropped
tag stops filtering.
286 tests pass (+3 from 283). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
pin lookup was legacy call_get_ref only, missing refs written via
call_put_ref_versioned (all Phase 3b+ builds). Try versioned first,
fall back to legacy — same pattern as claw-cargo build path.
A `claw-cargo pin` used to be silently vulnerable to the size-cap
eviction ticker — the tag existed but the underlying blob could get
LRU'd out, leaving a dangling reference. Now tags act as
retention markers: any blob referenced by any tag (stamped or
legacy) is protected from `evict_to_size_cap`.
* `BlobStore::evict_to_size_cap_with_pins(max_bytes, pinned_set)` —
same LRU-by-mtime pass, but pinned blob IDs skip the eviction
loop. Existing `evict_to_size_cap` is now a thin wrapper with an
empty pin set (100% backward compat).
* `TagStore::pinned_blob_values()` — unions every 32-byte value
referenced by any tag across `tags/` (legacy) and `tags-v2/`
(Phase 3c stamped). Dedupes naturally.
* Auto-GC ticker in `ClusterServices` now collects the pin set on
every eviction pass and passes it in. Log fields include
`pinned_blobs = N` so operators can see the retention set size.
* `claw-store cluster-gc --evict-to-gb N` CLI opens the tag store
the same way, prints `pinned blobs: N` in the report.
+3 tests:
- evict_with_pins_protects_pinned_blobs_from_eviction — 3 blobs
ordered oldest→newest, pin the oldest; without pins LRU would
evict it; with pins the next-oldest goes instead. Guards the
main semantic.
- evict_with_pins_stops_when_pinned_footprint_dominates —
everything pinned + cap = 0 → no-op. Guards the "operator asked
for the impossible" case.
- pinned_blob_values_unions_both_stores — legacy tag with value V1,
stamped tag with value V2, second stamped tag also referencing
V1 → set contains {V1, V2}. Dedupe check.
283 tests pass (baseline +3). Pre-existing macOS failure unchanged.
Ships the last two pieces from the arch doc's Phase 3 scope for the
cargo-cache use case:
## 3c: Stamped tags (CRDT-merge on PutTag)
Mirror of Phase 3a/3b for TagStore. Two concurrent `claw-cargo pin`
calls on the same tag now race deterministically instead of silently
clobbering.
* `StampedTagValue` — same 48-byte (value, clock, node) tuple as
StampedRef.
* `TagStore::put_stamped(key, StampedTagValue) -> TagPutOutcome`
and `TagStore::get_stamped(key)` — data lives under `tags-v2/`
(separate from `tags/` for cutover safety).
* New wire methods `PutTagVersioned = 0x18` +
`GetTagVersioned = 0x19`.
* `call_put_tag_versioned` / `call_get_tag_versioned` client
helpers.
* `claw-cargo pin` now writes stamped tags. Concurrent pin gets
AlreadyExists and moves on (blob content is content-addressed so
both winners agree on the payload).
## 3e: Namespaced ref keys
Opt-in `--namespace <slug>` on peer-facing subcommands. When set,
the ref key becomes `blake3("clawstor.ns.v1" || namespace || fp)`
so two runners on different namespaces (`clawverse/main` vs
`clawverse/pr-42`) don't collide on the same fingerprint. Empty
namespace = pre-3e behavior, so this is 100% backward compat.
* `refs::namespaced_ref_key(namespace, fingerprint) -> RefKey`
primitive.
* `PeerArgs::namespace: Option<String>` CLI flag flows through to
`cmd_status`, `cmd_prefetch`, `cmd_build`.
* `peer_lookup` now takes a `RefKey` directly (was `&Fingerprint`)
so the namespace resolution stays in the caller — the daemon
never sees "namespace" as a concept.
## 3d: Deferred
Full vector clocks per namespace are noted in the arch doc as a
Phase-3 goal; scalar wall-clock (clock + node stamp) is sufficient
for the cargo-cache use case (single-key LWW merge). NTP-synced
runners see monotonic ordering; skewed runners lose an ordering
but the CRDT semantics still guarantee no data corruption. Full VC
is deferred to a future phase.
+9 tests, 280 total (baseline +8: 7 unit + 1 e2e over real QUIC).
Runner + prewarm use PutRefVersioned/GetRefVersioned; daemon
GetRefVersioned forwards on miss + pulls blob transparently. New
GetRefVersionedLocal (0x17) prevents recursion. Backward-compat:
existing GetRef/PutRef path unchanged; two on-disk namespaces
coexist (refs/ and refs-v2/).
+1 test, 272 total (unchanged from 3a because we reused existing
scaffolding).
Concurrent PutRef safety via (clock, node) total order. New wire
methods PutRefVersioned (0x15) + GetRefVersioned (0x16). Existing
PutRef/GetRef unchanged for backward compat. Data in refs-v2/
namespace so the two coexist during cutover.
+8 tests, 272 total (baseline +8).
Cross-runner cache silos (tank + architect measured on 2026-07-13):
same fingerprint, same rustc, but each runner's daemon only knows
about the refs its own runner uploaded. Every runner that lands on
a peer that isn't tank re-uploads a duplicate blob.
Fix: on `GetRef` miss the daemon fans out to alive gossip peers
via a strict-local `GetRefLocal` variant, and the FIRST peer that
has the ref triggers a transparent pull — chunks + manifest into
the local blob store, then `PutRef` locally — before returning the
value to the caller. Subsequent lookups are pure-local hits.
* `Method::GetRefLocal = 0x14` — new wire method, identical shape
to GetRef but the peer MUST NOT recurse. Loop prevention: our
forwarding only calls `GetRefLocal` on peers, so chain depth is
always 1.
* `RpcRouter::with_outbound_client(Arc<QuicClient>)` — dependency
injection point for the forwarding dial path. `None` disables
forwarding entirely (GetRef becomes GetRefLocal-equivalent).
* `RpcRouter::forward_get_ref(key)` — concurrent peer probes via
`JoinSet`, 3s timeout per dial, first successful pull wins,
remaining tasks aborted.
* `pull_blob_locally` — walks manifest, fetches only chunks the
local store lacks (`has_chunk`), commits via
`put_manifest_verified`. Bounded memory: one 4 MiB chunk at a
time.
* `ClusterServices::start` loads NodeIdentity twice — server takes
ownership; outbound client gets its own copy for TLS presentation
on peer dials. Wires the outbound client into the router when
TLS material is available.
* `call_get_ref_local(conn, key)` client helper (used by daemon
forwarding + available to any RPC consumer that wants the
no-recursion semantics).
+3 tests in `rpc/tests_forwarding.rs`:
- Local hit works without forwarding; local miss with no peers
returns None. Guards the base cases.
- GetRefLocal never forwards even when outbound is configured (no
peers reachable → miss returns None immediately, no attempted
fan-out).
- Method byte 0x14 encoding is stable across releases.
Full end-to-end forwarding is exercised in the pilot deploy: two
daemons on the fleet-CA, tank populates a ref, architect's runner
GetRef → tank forwards → architect pulls → HIT locally next time.
264 tests pass (baseline +3). Pre-existing macOS failure unchanged.
Three fixes surfaced by the 2026-07-13 Gitea Actions wire-up.
## XDG config path
Before: `client_config::user_config_path` only looked at
`~/.claw-cargo/config.toml`. My runner-integration doc initially
told operators to install at `~/.config/claw-cargo/config.toml`
(XDG-style). Config wasn't loaded.
Now: three-way lookup, first hit wins.
1. `$XDG_CONFIG_HOME/claw-cargo/config.toml`
2. `$HOME/.config/claw-cargo/config.toml`
3. `$HOME/.claw-cargo/config.toml` (legacy, still honoured)
+3 tests: XDG env wins when the file exists, .config wins over
legacy dotfile when both present, legacy dotfile returned as
error-message fallback when none exist.
## Composite action
Before: composite action silently no-op'd. Log showed the script
lines echoed but only the `if command -v claw-cargo` fail branch
ran. Root cause: Gitea Actions composite steps run with a stripped
PATH that omits `/usr/local/bin`.
Now: composite step exports PATH defensively:
export PATH="/usr/local/bin:$HOME/.cargo/bin:$PATH"
And it looks for the config at BOTH the XDG-style path and the
legacy dotfile (same order as client_config).
Workflow file back to using the composite action.
## Docs
Runner-integration doc now:
- Explicitly warns that `ubuntu-latest` routes to container mode
even with `:host` suffix on runner labels (field-observed).
- Documents the dedicated `clawstor-cache` label pattern that
works.
- Sample workflow uses `runs-on: clawstor-cache` instead of
`ubuntu-latest`.
+3 tests, 262 total (baseline unchanged).
client_config.rs load_layered looks at ~/.claw-cargo/config.toml,
not ~/.config/claw-cargo/config.toml. Fix the workflow preflight
path to match. Both runners already have the file at both locations.
Fleet has 7 linux-amd64 runners; only tank + architect have
claw-cargo provisioned. Added a clawstor-cache:host label to those
two runners so this workflow only lands on them.
ubuntu-latest routes to container mode in act_runner even with the
:host suffix on the runner labels. Explicit self-hosted forces
host-mode where /usr/local/bin/claw-cargo + per-runner tls_dir are
visible.
Omar Sobh
osobh