Adds the encoding layer that turns the raw WAL (opaque bytes)
into a typed queue of client-mode mutations, ready for Phase 4e
to wire actual RPCs through.
Frame (self-describing, forward-compatible):
version : u8 = 0x01
kind : u8 = one of the Kind discriminants
body : [u8] kind-specific
Body encodings mirror the existing on-wire shapes so a future
replay path can splice a WAL record straight into an RPC payload.
Variants (Kinds 0x01–0x06):
PutRef, PutRefVersioned, PutTag, PutTagVersioned,
DeleteTag, SetTagExpiry
Blob-put mutations are deliberately NOT modeled — blob data is
too large to keep in the WAL. The roaming-client design stages
blobs on local disk and records a reference to them once the
local BlobPutStream completes.
Public helpers:
append_mutation(&mut wal, &m) -> Result<seq>
replay_mutations(&wal, start_seq)
-> Vec<(seq, Result<WalMutation, WalMutationError>)>
Unknown-kind records surface as `Err(UnknownKind(byte))`, not
a panic — forward-compat when a newer writer wrote a record
this reader doesn't understand. Malformed records also surface
as Err so the caller can decide (log-and-skip vs abort replay).
Tests (9, all green): kind-byte stability, roundtrip every
variant, rejects empty/short/bad-version/unknown-kind,
malformed bodies (wrong length, over-declared key_len, trailing
garbage on DeleteTag), non-UTF-8 keys, append+replay through a
real on-disk WAL, and replay-survives-unknown-kind mid-stream.
No new deps — hand-rolled error type in-tree (no thiserror).
515 lines, well under the 1300 ceiling.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Turns the single-file Phase 4c WAL into a segmented log so it
can grow past a single file safely. This unblocks every
downstream Phase 4d/4e integration — reconnect + push loop
can't rely on an unbounded single file.
Layout change:
<root>/segment-<20-digit-first-seq>.bin
20-digit zero-padded first-seq means lex sort == numeric sort,
so `read_dir + sort_by_key` recovers the natural order.
Rotation policy:
* `max_segment_bytes` default 8 MiB, overridable via
`open_with_options`.
* `append` rolls to a fresh segment BEFORE writing when the
current tail is non-empty AND at/above the cap. A single
oversize record always lands in one segment — we never split
a record.
Truncation across segments:
* whole segments with `last_seq <= watermark` are `unlink`'d
* the boundary segment (if any) is rewritten in place via
`tempfile-in-parent + rename` + parent-dir fsync
* full truncation resets head/tail to 0 and the next append
creates a fresh segment
Legacy compat: on open, if a pre-4d `log.bin` is present and
no `segment-*.bin` files exist, it is scanned for its first
seq and renamed to the correct segment name. Refuses to
silently overwrite on filename collision.
Tests (18, all green): rotation-happens-at-cap, reopen-
enumerates-all-segments, truncate-drops-whole-segments,
truncate-partial-rewrites-boundary, oversize-record-still-
fits-one-segment, legacy-log.bin-migration, plus the full
Phase 4c suite (fresh open, append, iter partial ranges,
reopen recovers tail, torn-write truncation, corruption is
hard error, full truncation appendable, below-head no-op,
large payload, empty payload, append-after-reopen).
942-line file, comfortably under the 1300-line ceiling.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Foundation for the roaming/offline-client story described in
Architecture v2 ("Roaming client (full R/W, offline queue)"):
mutating ops append to a durable local log before hitting the
network, and are replayed at reconnect. This PR ships the
primitive; RPC/reconnect wiring lands in Phase 4d.
Segment format (single-file for now — rotation is 4d):
seq : u64 LE (8 bytes)
len : u32 LE (4 bytes) payload length
csum : [u8; 8] (8 bytes) first 8 bytes of
BLAKE3(seq || len || payload)
bytes : [u8; len]
On open, the log is scanned linearly. A short read or truncated
tail is treated as "clean crash boundary" — the file is
size-truncated to the last fully-fsynced record, no error.
A checksum mismatch on a full-length record is fatal (real
corruption, don't silently swallow data).
Public API:
WriteAheadLog::open(root) -> Self
wal.append(&[u8]) -> Result<u64> // durable, fsynced
wal.iter_from(start_seq) -> Vec<WalRecord>
wal.truncate_up_to(watermark) -> () // atomic rewrite via
// tempfile-in-parent + rename
wal.head_seq() / wal.tail_seq() / wal.is_empty()
Tests (12, all green): fresh open, monotonic seq, replay full &
partial ranges, reopen-recovers-tail, torn-write truncation on
open, corruption is hard error, prefix truncation, full
truncation leaves appendable, below-head no-op, 1 MiB payload
roundtrip, empty payload roundtrip, append-after-reopen.
No new deps — BLAKE3 (already a dep) supplies the checksum.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Closes Phase 4b by exposing the TTL sidecar written by
Phase 4b primitives on the wire and via `claw-cargo pin`.
Wire additions:
* Method::SetTagExpiry (0x1a) — payload `key_len:u16 || key ||
expires_at:u64 (LE)`. Reply single-byte OK. `expires_at == 0`
clears the sidecar.
* Method::GetTagExpiry (0x1b) — payload raw key bytes. Reply 8
bytes (u64 LE) on hit; NotFound when no sidecar is present.
Both accept writes even when the stamped tag itself is absent,
matching `TagStore::set_stamped_expiry` semantics — the sidecar
takes effect the moment the tag lands.
CLI:
* `claw-cargo pin --ttl <duration>` — humantime-style duration
(`30d`, `1h30m`, `2w`, ...). Applied to both the primary tag
and its `.fingerprint` companion so eviction treats them as
one lifetime. `--ttl 0` / `clear` / `none` clears an existing
sidecar without touching the value.
Tests: encode/decode roundtrip + malformed-input rejection for
`encode_expiry_record`, method-byte stability, NotConfigured
without a tag store, end-to-end set/get/overwrite/clear over
QUIC, and a real-pin flow that publishes a stamped tag then
attaches TTL. Duration parser is unit-tested for single/compound
forms, case-insensitive units, bad input, and clock alignment.
No new deps — the humantime-style parser is 60 lines in-tree.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Adds the on-disk mechanism for time-scoped pins. No RPC or CLI yet
— a follow-on will expose \`pin --ttl <duration>\`. This PR is
purely library + eviction wiring.
Layout addition: alongside each stamped tag at
\`tags-v2/<hh>/<hash>.svtag\`, an optional sidecar
\`tags-v2/<hh>/<hash>.svtag.exp\` holds an 8-byte LE unix
\`expires_at\`. Absence of the sidecar = never expires (current
behavior).
New TagStore methods:
* set_stamped_expiry(key, expires_at_unix) — writes sidecar;
passing 0 removes it. Idempotent.
* get_stamped_expiry(key) — reads sidecar; None when absent.
* pinned_blob_values_at(now_unix) — same union as
pinned_blob_values, but skips stamped tags whose sidecar shows
expires_at ≤ now. Legacy tags/ entries never expire.
* prune_expired_stamped_at(now_unix) — deletes stamped tags AND
their sidecars where expires_at ≤ now. Returns count.
* pinned_blob_values() — now a shim that calls _at(u64::MAX) for
100% backward compat.
Wired the two existing gc call sites:
* ClusterServices auto-GC ticker prunes-then-collects at
SystemTime::now(). One pass per tick.
* \`claw-store cluster-gc --evict-to-gb N\` CLI same pattern.
Report now includes \"expired pins pruned: N\".
+1 test (expiry_gates_pin_set_and_prune_removes_expired):
covers live/expired/no-ttl mix, sidecar round-trip, prune
removes only expired, expires_at=0 clears sidecar, dropped
tag stops filtering.
286 tests pass (+3 from 283). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
pin lookup was legacy call_get_ref only, missing refs written via
call_put_ref_versioned (all Phase 3b+ builds). Try versioned first,
fall back to legacy — same pattern as claw-cargo build path.
A `claw-cargo pin` used to be silently vulnerable to the size-cap
eviction ticker — the tag existed but the underlying blob could get
LRU'd out, leaving a dangling reference. Now tags act as
retention markers: any blob referenced by any tag (stamped or
legacy) is protected from `evict_to_size_cap`.
* `BlobStore::evict_to_size_cap_with_pins(max_bytes, pinned_set)` —
same LRU-by-mtime pass, but pinned blob IDs skip the eviction
loop. Existing `evict_to_size_cap` is now a thin wrapper with an
empty pin set (100% backward compat).
* `TagStore::pinned_blob_values()` — unions every 32-byte value
referenced by any tag across `tags/` (legacy) and `tags-v2/`
(Phase 3c stamped). Dedupes naturally.
* Auto-GC ticker in `ClusterServices` now collects the pin set on
every eviction pass and passes it in. Log fields include
`pinned_blobs = N` so operators can see the retention set size.
* `claw-store cluster-gc --evict-to-gb N` CLI opens the tag store
the same way, prints `pinned blobs: N` in the report.
+3 tests:
- evict_with_pins_protects_pinned_blobs_from_eviction — 3 blobs
ordered oldest→newest, pin the oldest; without pins LRU would
evict it; with pins the next-oldest goes instead. Guards the
main semantic.
- evict_with_pins_stops_when_pinned_footprint_dominates —
everything pinned + cap = 0 → no-op. Guards the "operator asked
for the impossible" case.
- pinned_blob_values_unions_both_stores — legacy tag with value V1,
stamped tag with value V2, second stamped tag also referencing
V1 → set contains {V1, V2}. Dedupe check.
283 tests pass (baseline +3). Pre-existing macOS failure unchanged.
Ships the last two pieces from the arch doc's Phase 3 scope for the
cargo-cache use case:
## 3c: Stamped tags (CRDT-merge on PutTag)
Mirror of Phase 3a/3b for TagStore. Two concurrent `claw-cargo pin`
calls on the same tag now race deterministically instead of silently
clobbering.
* `StampedTagValue` — same 48-byte (value, clock, node) tuple as
StampedRef.
* `TagStore::put_stamped(key, StampedTagValue) -> TagPutOutcome`
and `TagStore::get_stamped(key)` — data lives under `tags-v2/`
(separate from `tags/` for cutover safety).
* New wire methods `PutTagVersioned = 0x18` +
`GetTagVersioned = 0x19`.
* `call_put_tag_versioned` / `call_get_tag_versioned` client
helpers.
* `claw-cargo pin` now writes stamped tags. Concurrent pin gets
AlreadyExists and moves on (blob content is content-addressed so
both winners agree on the payload).
## 3e: Namespaced ref keys
Opt-in `--namespace <slug>` on peer-facing subcommands. When set,
the ref key becomes `blake3("clawstor.ns.v1" || namespace || fp)`
so two runners on different namespaces (`clawverse/main` vs
`clawverse/pr-42`) don't collide on the same fingerprint. Empty
namespace = pre-3e behavior, so this is 100% backward compat.
* `refs::namespaced_ref_key(namespace, fingerprint) -> RefKey`
primitive.
* `PeerArgs::namespace: Option<String>` CLI flag flows through to
`cmd_status`, `cmd_prefetch`, `cmd_build`.
* `peer_lookup` now takes a `RefKey` directly (was `&Fingerprint`)
so the namespace resolution stays in the caller — the daemon
never sees "namespace" as a concept.
## 3d: Deferred
Full vector clocks per namespace are noted in the arch doc as a
Phase-3 goal; scalar wall-clock (clock + node stamp) is sufficient
for the cargo-cache use case (single-key LWW merge). NTP-synced
runners see monotonic ordering; skewed runners lose an ordering
but the CRDT semantics still guarantee no data corruption. Full VC
is deferred to a future phase.
+9 tests, 280 total (baseline +8: 7 unit + 1 e2e over real QUIC).
Runner + prewarm use PutRefVersioned/GetRefVersioned; daemon
GetRefVersioned forwards on miss + pulls blob transparently. New
GetRefVersionedLocal (0x17) prevents recursion. Backward-compat:
existing GetRef/PutRef path unchanged; two on-disk namespaces
coexist (refs/ and refs-v2/).
+1 test, 272 total (unchanged from 3a because we reused existing
scaffolding).
Concurrent PutRef safety via (clock, node) total order. New wire
methods PutRefVersioned (0x15) + GetRefVersioned (0x16). Existing
PutRef/GetRef unchanged for backward compat. Data in refs-v2/
namespace so the two coexist during cutover.
+8 tests, 272 total (baseline +8).
Cross-runner cache silos (tank + architect measured on 2026-07-13):
same fingerprint, same rustc, but each runner's daemon only knows
about the refs its own runner uploaded. Every runner that lands on
a peer that isn't tank re-uploads a duplicate blob.
Fix: on `GetRef` miss the daemon fans out to alive gossip peers
via a strict-local `GetRefLocal` variant, and the FIRST peer that
has the ref triggers a transparent pull — chunks + manifest into
the local blob store, then `PutRef` locally — before returning the
value to the caller. Subsequent lookups are pure-local hits.
* `Method::GetRefLocal = 0x14` — new wire method, identical shape
to GetRef but the peer MUST NOT recurse. Loop prevention: our
forwarding only calls `GetRefLocal` on peers, so chain depth is
always 1.
* `RpcRouter::with_outbound_client(Arc<QuicClient>)` — dependency
injection point for the forwarding dial path. `None` disables
forwarding entirely (GetRef becomes GetRefLocal-equivalent).
* `RpcRouter::forward_get_ref(key)` — concurrent peer probes via
`JoinSet`, 3s timeout per dial, first successful pull wins,
remaining tasks aborted.
* `pull_blob_locally` — walks manifest, fetches only chunks the
local store lacks (`has_chunk`), commits via
`put_manifest_verified`. Bounded memory: one 4 MiB chunk at a
time.
* `ClusterServices::start` loads NodeIdentity twice — server takes
ownership; outbound client gets its own copy for TLS presentation
on peer dials. Wires the outbound client into the router when
TLS material is available.
* `call_get_ref_local(conn, key)` client helper (used by daemon
forwarding + available to any RPC consumer that wants the
no-recursion semantics).
+3 tests in `rpc/tests_forwarding.rs`:
- Local hit works without forwarding; local miss with no peers
returns None. Guards the base cases.
- GetRefLocal never forwards even when outbound is configured (no
peers reachable → miss returns None immediately, no attempted
fan-out).
- Method byte 0x14 encoding is stable across releases.
Full end-to-end forwarding is exercised in the pilot deploy: two
daemons on the fleet-CA, tank populates a ref, architect's runner
GetRef → tank forwards → architect pulls → HIT locally next time.
264 tests pass (baseline +3). Pre-existing macOS failure unchanged.
Three fixes surfaced by the 2026-07-13 Gitea Actions wire-up.
## XDG config path
Before: `client_config::user_config_path` only looked at
`~/.claw-cargo/config.toml`. My runner-integration doc initially
told operators to install at `~/.config/claw-cargo/config.toml`
(XDG-style). Config wasn't loaded.
Now: three-way lookup, first hit wins.
1. `$XDG_CONFIG_HOME/claw-cargo/config.toml`
2. `$HOME/.config/claw-cargo/config.toml`
3. `$HOME/.claw-cargo/config.toml` (legacy, still honoured)
+3 tests: XDG env wins when the file exists, .config wins over
legacy dotfile when both present, legacy dotfile returned as
error-message fallback when none exist.
## Composite action
Before: composite action silently no-op'd. Log showed the script
lines echoed but only the `if command -v claw-cargo` fail branch
ran. Root cause: Gitea Actions composite steps run with a stripped
PATH that omits `/usr/local/bin`.
Now: composite step exports PATH defensively:
export PATH="/usr/local/bin:$HOME/.cargo/bin:$PATH"
And it looks for the config at BOTH the XDG-style path and the
legacy dotfile (same order as client_config).
Workflow file back to using the composite action.
## Docs
Runner-integration doc now:
- Explicitly warns that `ubuntu-latest` routes to container mode
even with `:host` suffix on runner labels (field-observed).
- Documents the dedicated `clawstor-cache` label pattern that
works.
- Sample workflow uses `runs-on: clawstor-cache` instead of
`ubuntu-latest`.
+3 tests, 262 total (baseline unchanged).
client_config.rs load_layered looks at ~/.claw-cargo/config.toml,
not ~/.config/claw-cargo/config.toml. Fix the workflow preflight
path to match. Both runners already have the file at both locations.
Fleet has 7 linux-amd64 runners; only tank + architect have
claw-cargo provisioned. Added a clawstor-cache:host label to those
two runners so this workflow only lands on them.
ubuntu-latest routes to container mode in act_runner even with the
:host suffix on the runner labels. Explicit self-hosted forces
host-mode where /usr/local/bin/claw-cargo + per-runner tls_dir are
visible.
Host runners have cmake/gcc/pkg-config from the OS and cargo/rustup
in the act_runner user's ~/.cargo/bin. apt-get needs root — the
runner isn't. Replace with a preflight that fails fast when any
tool is missing.
Ships the wire-up piece for real CI: a composite Gitea Action that
wraps `claw-cargo build` with cache-outcome reporting, plus a
matching workflow file that opts the clawstor repo itself into
being cache-hit-tested on every push. Also docs the one-time
per-runner provisioning (leaf cert, PATH install, config.toml).
* `.gitea/actions/cargo-cache/action.yml` — composite Action.
Inputs: workspace, profile, no-upload, parallel-restore. Outputs:
cache-outcome (HIT|MISS|POPULATED|SKIPPED), fingerprint,
elapsed-seconds. Runner-side config lives in
`~/.config/claw-cargo/config.toml` (not in the workflow — no
secrets shipped from repos).
* `.gitea/workflows/build-with-cache.yml` — dogfoods the action on
clawstor's own repo. `no-upload` set from event_name so PRs from
forks can't poison the cache.
* `docs/runner-integration.md` — one-time setup steps, sample
workflow snippet, expected numbers (Pi 5: 2.79× wall, tank:
2.18×), and troubleshooting for the failures I hit in the tank
and Pi pilots (bind_lan on fabric-only, missing CLI/config,
rustc drift warn).
Test protocol: push this branch → main triggers the workflow → the
runner on tank has claw-cargo + tls + config provisioned already
(2026-07-13 pilot setup) → first build should MISS + populate,
subsequent build on same fingerprint should HIT.
Pi 5 loopback measurement 2026-07-13:
--parallel-restore 1 : wall 2m52s, restore 20s
--parallel-restore 8 : wall 3m06s, restore 34s
Sequential is 70% faster on loopback. N-way stream contention costs
more than a single stream's congestion-control amortization. Same
shape as Phase 5k prewarm — fanout only wins when per-stream
throughput has a ceiling (WAN, tunneled links).
--parallel-restore N remains as opt-in.
Follow-up to PR #28. Fresh Pi deploy 2026-07-12 hit a `ProtectHome=
read-only` block on the daemon's XDG-driven
$HOME/.local/state/claw-store/projects.toml write. Adding the path
to `ReadWritePaths` in the shipped unit means future deployers
don't need a drop-in.
Two fixes surfaced by the vision-02 Pi 5 measurement:
## XDG default_path
`Manifest::default_path` was hardcoded to
`/var/lib/claw-store/projects.toml`. That path is read-only under
the user-mode systemd unit's `ProtectSystem=strict`, and creating
it needs root — awful for a runner install.
Precedence, matching XDG Base Directory:
1. `$XDG_STATE_HOME/claw-store/projects.toml`
2. `$HOME/.local/state/claw-store/projects.toml`
3. `/var/lib/claw-store/projects.toml` (system fallback)
User-mode installs now write in $HOME by default; system installs
(root, no HOME set) still land in /var/lib.
+1 test: `default_path_honours_xdg_state_home` — covers all three
precedence branches. Env mutation is process-global so the test
saves + restores.
## Parallel restore on cache HIT
Pi restore of 947 MiB via `BlobGetStream` took ~18s (~53 MiB/s)
single-stream. Per-stream throughput ceilings on the connection
type cap sequential fetches; parallel chunk fetches stack their
contributions.
- New `call_blob_get_parallel(conn, blob_id, concurrency) ->
Option<Vec<u8>>` in `rpc/client.rs`. `JoinSet` + `Semaphore`,
reassembles by chunk index at manifest-known offsets so
out-of-order arrival is fine.
- `claw-cargo build --parallel-restore N` (default 8). `N <= 1`
falls through to `BlobGetStream` for parity.
- Memory: `total_size + 4 MiB × in-flight` — dominated by the
reassembly buffer, not the fanout.
+1 test: `parallel_blob_get_reassembles_multi_chunk_blob_byte_equal`
covers roundtrip byte-equality vs BlobGetStream, tail-chunk offset,
concurrency=1 correctness, and NotFound → None.
259 tests pass (+2). Pre-existing macOS failure unchanged.
Orphan-chunk GC alone doesn't stop unbounded growth: as long as
fingerprint→blob refs keep getting PutRef'd, the manifest set keeps
growing and no chunk is ever an orphan.
* `BlobStore::evict_to_size_cap(max_bytes)` — walks manifests oldest
first by mtime, deletes them, refcount-decrements each chunk they
used, unlinks + reclaims size for any chunk whose refcount hits
zero. Shared chunks stay put until the last blob referencing them
is evicted.
* `ManifestSummary` internal type keeps the diff-set bookkeeping
cheap (one HashMap<ChunkHash, u32>, no repeated tree walks).
* `claw-store cluster-gc --evict-to-gb <N>` extends the CLI: still
runs the orphan sweep first, then optionally caps the store.
* Config: `cluster.blob_max_gb: Option<u64>`. The auto-GC ticker
runs eviction after every orphan sweep when this is set. Silent
when the store is already under cap; INFO log when it evicts.
+3 tests:
- evict_to_size_cap_reclaims_oldest_blobs_first: 3 blobs with
distinct mtimes, cap below combined size → oldest evicted,
newer blobs survive
- evict_keeps_shared_chunks_when_still_referenced: guards the
refcount decrement path (content-addressed dedup keeps identical
content as one blob → chunk survives until manifest deleted)
- evict_on_empty_store_is_a_noop: sanity
257 tests pass (baseline +3). Pre-existing macOS failure unchanged.
osobh