First slice of Phase 6. Ships a minimal, feature-gated `claw-fuse`
binary that mounts the local blob store read-only as a POSIX
filesystem:
<mount>/blobs/<blob-id-hex> ← file, content = assembled blob
<mount>/blobs/ ← dir, ls shows all blob-ids
<mount>/ ← dir, contains `blobs`
Lets an operator `tar -tvzf`, `md5sum`, or grep at a cached
tarball without wiring a client. Debug + audit tool for now;
warm-tier git-worktrees + write path come in later slices.
Feature-gated so my macOS dev box doesn't need macFUSE headers
to build the rest of the tree:
* Cargo.toml declares `[[bin]] name = "claw-fuse"` with
`required-features = ["fuse"]`.
* Feature `fuse` pulls in `fuser = "0.15"`, target-restricted
to `cfg(target_os = "linux")` — dep resolution never
considers fuser on other platforms.
* `cargo build` (default) leaves claw-fuse out entirely.
`cargo build --features fuse --bin claw-fuse` on Linux builds it.
Design notes baked into the impl:
* Inode allocation is lazy — first `lookup` for a hex assigns an
inode. Avoids pre-indexing the full blob store at mount time
which would be O(blobs) fs walk before FUSE is even ready.
* getattr / read validate that the manifest exists on every
call — no stale-inode reads if a blob is GC'd out from under
us mid-mount. Extra read cost is negligible against the
per-request FUSE overhead.
* size = manifest.total_size (bytes reported without touching
chunk files) so `ls -l` is cheap.
* runtime = current-thread tokio, block_on per callback. fuser
is sync; a full tokio worker pool would just add scheduling
overhead when callbacks are already serialized by the kernel.
No new tests here — Filesystem impls are integration-heavy and
the underlying BlobStore methods are already covered. The
`fuse` feature build itself will be smoke-tested on tank.
381 tests pass unchanged (feature-gated bin doesn't affect the
existing test surface).
Same shape as Phase 8c did for cluster-peer-status + cluster-repair.
New flags: --tailscale-addr (optional) + --lan-probe-ms (default 200).
Route (LAN vs tailnet) printed on the output. Zero flag = identical
to pre-8 single-addr behavior.
Fourth of four operator-facing CLIs now routing-aware
(cluster-peer-status, cluster-repair, cluster-ping done; cluster-ping
was the last outstanding one).
No new tests: pure glue over connect_lan_first, which has its own
unit coverage.
Reclaims local target-dir disk in increasing bluntness. All
modes are LOCAL only — the fleet blob cache is untouched, so
`claw-cargo build` after smart-clean restores from peer.
Modes:
* incremental-only — remove target/*/incremental/ across all
profiles. Safest; keeps final artifacts + deps.
* soft (default) — remove target/ entirely. Blob still on peer.
* hard — soft, but requires --force. Reserved for operators who
know their build is transient. Rejected without --force even
in --dry-run so the safety belt can't be trained away.
--dry-run reports paths + byte count without touching disk.
New helpers (unit-tested in isolation):
* find_incremental_dirs(target) — walks target/*/incremental,
returns only existing entries.
* dir_size_bytes(root) — recursive byte count, silent on read
errors (used only for reporting, not correctness).
+5 tests: incremental discovery (existing only), missing target
empty, byte sum recursive, missing dir returns 0, hard-without-
force rejects.
381 tests pass (+5). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
Live smoke on tank↔architect exposed the gap: bind_rpc_tailscale
was being *advertised* via gossip so peers learned to dial it,
but the daemon never actually LISTENED there. Tailnet dials hit
a closed port.
Fix: when both bind_rpc_lan and bind_rpc_tailscale are set (and
differ), spawn a second QuicServer on the tailnet address. Shares
the same fleet-CA identity + RpcRouter as the LAN listener —
requests from either side hit the same handlers.
If the second bind fails (e.g. tailnet interface not up), we log
a warning and keep the LAN listener alive rather than aborting
daemon startup. Standard graceful-degrade shape.
No new tests here — a live integration test would need two
network interfaces + a running tailscale, which the CI runners
don't have. Coverage happens on the tank+architect deployment:
`ss -lunp` on architect must show TWO clawstor UDP listeners
after this change (10.0.0.13:7702 + 100.104.171.32:7702).
Follow-on: cert SAN for the tailnet address. The current
fleet-CA-signed leaf only has the node name as SAN, so rustls
verification on the client side still checks against
--peer <name> which passes because CN == node name. But a
belt-and-suspenders leaf using fleet-ca-tailscale-sign (Phase
8a) would be more correct.
Live smoke on tank↔architect (both LAN) failed with 200ms probe:
LAN handshake takes longer than that in the wild (TLS 1.3 with
full cert chain + rustls startup on fresh endpoint). The old
single-addr .connect() had no deadline, so pre-8c callers never
noticed.
Fix: when `tailscale` is `None`, treat LAN as unlimited — the
probe deadline only matters as a fall-through trigger, and
there's nothing to fall through to. Callers with a real fallback
addr still get the fast-path routing behavior unchanged.
+1 test (connect_lan_first_lan_only_ignores_probe_deadline)
using a 1-nanosecond probe budget that a real handshake could
never meet — must succeed anyway because no fallback exists.
376 tests pass (+1).
Wires the operator CLIs to the Phase 8b connect_lan_first primitive.
Roaming ops (laptop on LTE, coffee-shop wifi) can now pass a
tailnet address alongside the usual --rpc-addr and get the
LAN-first-with-fallback behavior automatically.
New flags on both cluster-peer-status and cluster-repair:
* --tailscale-addr <addr> — optional tailnet RPC socket. When
set, --rpc-addr is tried first with
a short deadline, then this on
failure/timeout.
* --lan-probe-ms <ms> — LAN probe deadline. Default 200
matches the arch doc.
Zero flag → byte-identical to pre-8c behavior (single-addr dial).
Both flags → chosen route printed in the output header so
operators can see whether LAN or tailnet won.
No new tests: this is thin glue over connect_lan_first, which
already has its own unit coverage. Smoke test live on tank
against architect (LAN), and against fake unroutable + real
tailnet exercises both branches.
Second Phase 8 slice. Prior transport.connect() took a single
address; the LAN-first-then-Tailscale routing the arch doc calls
out was implicit ("pick lan_addr OR tailscale_addr from gossip
state") and never actually raced or fell through.
New: QuicClient::connect_lan_first(name, lan, tailscale, lan_probe)
* Try LAN first with `lan_probe` deadline (fleet default ~200ms).
* If LAN handshake fails OR the deadline fires → fall back to
the Tailscale address.
* Both slots None → error immediately (no hang).
Returns (connection, ConnectRoute) so callers + telemetry see
which side won. New enum ConnectRoute::{Lan(addr), Tailscale(addr)}.
+3 tests exercising the three shapes:
- lan-first when LAN reachable (never dials fake tailscale addr)
- fallback when LAN black-holes (240.0.0.1 SYN gets no response;
probe deadline fires, tailscale server wins)
- errors cleanly when both addrs absent
375 tests pass (+3). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
Follow-ons for Phase 8 completion:
- Wire the peer-connect call sites (RPC forwarding, PeerStatus,
build-cache) through connect_lan_first with per-peer
lan/tailscale addrs from gossip state.
- Document the roaming-client config template.
First slice of Phase 8 (roaming client identity). Adds a helper
that mints a leaf cert whose SANs include this node's Tailscale
identity — MagicDNS name (laptop.taila4f562.ts.net) + all tailnet
IPs — alongside the primary node name.
Closes the "how does a laptop join the fleet without hand-editing
SANs" gap: on a machine that's on Tailscale, one command produces
a leaf that peers can dial by MagicDNS from anywhere on the
tailnet.
New CLI:
claw-store fleet-ca-tailscale-sign \
--ca-dir /etc/claw-store/ca \
[--node <name>] # defaults to Tailscale HostName
--out-dir /etc/claw-store/tls
Reads identity by shelling to `tailscale status --json` (already
present on any node that's on the tailnet; no extra dep). If
tailscale isn't running or installed, exits cleanly with a real
error.
New module cluster::tailscale:
* TailscaleSelf { magicdns_name, tailscale_ips, short_hostname }
* read_self() — runs the CLI, returns identity
* parse_status() — pure decoder, unit-tested
* suggested_sans() — MagicDNS + IPs ordered for the CA sign flow
FleetCa additions:
* sign_leaf_to_pem_with_sans(node_name, extra_sans, out_dir) —
Sans-extended variant of sign_leaf_to_pem. Empty entries dropped.
Existing sign_leaf_to_pem now delegates with empty extras (100%
backward compat).
* mint_leaf_with_sans — internal shared helper.
+5 tests: parse full identity, parse missing MagicDNS, error on
no Self record, suggested_sans ordering, suggested_sans skips
missing MagicDNS.
372 tests pass (+5). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
Next Phase 8 slices: (a) tailnet-preferring peer probe with a
config-selectable auth mode, (b) documented "roaming client"
config template.
Closes the ref-tracking loop. claw-cargo build now records the
producing (repo, git_ref) alongside every cache-put fingerprint,
so cluster-ref-sweep can identify stale entries later without
operator bookkeeping.
New BuildArgs flags:
* --repo <owner/name> (env CLAWSTOR_REPO, or GITEA_REPOSITORY /
GITHUB_REPOSITORY when the CI runner sets them via that name
in workflow env)
* --git-ref <branch-or-tag> (env CLAWSTOR_GIT_REF)
* --ref-tracking-dir <path> (env CLAWSTOR_DATA_DIR, typically
/var/lib/claw-store/data — same root as cluster.blob_store_root)
Semantics:
* All three unset → silently skipped. Existing cache flows are
unchanged.
* dir doesn't exist or open() fails → logs warn, cache still valid.
* record() call fails → logs warn, cache still valid.
The tracking store is co-located with the daemon's data dir so
cluster-ref-sweep on that host sees the annotations. Runners
mount /var/lib/claw-store/data via bind-mount today.
No new tests here — the primitive (RefTracking::record) already
has full coverage. This is thin glue.
Wires the Phase 7f ref-tracking primitives to a real Gitea. New
CLI `claw-store cluster-ref-sweep --gitea-url <> [--gitea-token]
[--retention-days N]` queries every distinct repo we've recorded
against, fetches its live branches + tags, computes the stale set
via RefTracking::stale_at, and prints the stale fingerprints
grouped by repo.
Dry-run only in this cut. Deletion is separate — the operator
decides whether to call `forget` per fp, and whether to also
prune the corresponding blob/tag. Blob eviction happens via
cluster-gc as usual (dead refs no longer contribute to any pin).
New module cluster:
* GiteaClient::new(base_url, token) — reqwest with 15s timeout,
rustls-tls (reuses the rustls stack quinn already pulls in).
* live_refs(repo) — fetches /branches + /tags concurrently,
paginated (page 200 hard cap for safety), returns HashSet.
* 404 on either endpoint returns empty set — deleted repos then
flow through stale_at as "all refs dead", the correct default.
Deps:
* reqwest 0.12 with rustls-tls + json, default-features off (no
native-tls / openssl chain).
* clap 4 + "env" feature so --gitea-token can read GITEA_TOKEN.
+2 tests (validate_repo shape, client trims trailing slash).
Full test suite: 367 pass (+2). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
Records which (repo, git-ref) combinations produced each cache
fingerprint. Later slices will wire this to a nightly Gitea sweep
that queries /api/v1/repos/.../branches and /tags, then evicts
fingerprints whose recorded refs are all gone AND whose
last_seen_unix is older than the retention window.
Per-fingerprint (not per-blob) because:
* Fingerprints are the cache keys claw-cargo uses. Tracking at the
fp layer keeps this aligned with the claw-cargo boundary.
* Blobs are content-addressed and may be shared. Ref-tracking is
about "why we kept this cache" — a per-fp concern.
New module cluster::ref_tracking:
* RefEntry { fingerprint, repo, refs, first_seen_unix, last_seen_unix }
* RefTracking::record(fp, repo, git_ref, now) — creates or updates
* RefTracking::get(fp) / list_all() / forget(fp)
* RefTracking::stale_at(now, live_refs_by_repo, retention_secs) →
Vec<fingerprint>, the deletion-eligibility list
On-disk: <root>/ref-tracking/<hh>/<fp_hex>.json. JSON so operators
can inspect with jq. One record per cached fp; even 100k fps is
under 50 MB.
Semantics baked in:
* record() APPENDS refs, never removes — sweep decides staleness
* record() rejects repo change for a fp (collision or bug detector)
* refs stable-sorted in-file so cross-node diff is easy
* stale_at treats "repo not in live_refs map" as "all refs dead"
→ deleted repos don't leak caches
* retention_secs is a floor: dead-but-fresh caches survive
+10 tests: create, append-and-refresh, dedup, repo-change reject,
stale-at happy path, stale-at missing-repo, stale-at retention,
forget truth values, list sorted, validate rejects.
365 tests pass (+10). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
No CLI or wire integration in this PR — pure library, testable
in isolation. Follow-ons: (a) claw-cargo auto-record on cache put,
(b) Gitea polling adapter, (c) sweep wired into cluster-gc.
Closes the retention loop between snapshots and pin-aware LRU
eviction. A snapshot is not just a "list of blobs at time T" any
more — it's a *retention pin* on every blob it captures.
Operators can guarantee a build stays on disk for N days by
snapshotting it and pruning the snapshot when the window is up.
Additions:
* SnapshotStore::pinned_blob_ids() → union of blob_ids across all
live snapshots. Cheap: one JSON read per snapshot.
* cmd_cluster_gc extends the tag-pin set with snapshot pins
before handing it to evict_to_size_cap_with_pins. Output line
now reads "pinned blobs: N (M from snapshots)".
* ClusterServices auto-GC ticker does the same on every tick;
log fields include snapshot_pins so ops see the retention set
size at a glance.
+2 tests:
- pinned_blob_ids_unions_all_snapshots (overlap dedupe)
- pinned_blob_ids_empty_when_no_snapshots
355 tests pass (+2). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
A snapshot is a named, immutable point-in-time record of every blob
live in the store. It's NOT a data copy — blobs are content-addressed
and already live under blobs/. A snapshot is a JSON reference set at
<root>/snapshots/<name>.json.
Why:
* Rollback anchor before risky migrations.
* Retention pin: combined with the Phase 4a pin-aware LRU eviction,
operators can guarantee "these blobs stay on disk N days".
* Audit: "which blobs existed at release time?"
New module cluster::snapshot:
* SnapshotStore::create(name, blob_store, created_at)
* SnapshotStore::get(name) / list() / delete(name)
* SnapshotManifest { name, created_at_unix, blob_ids }
* SnapshotSummary for cheap list rendering (no blob-list slurp).
BlobStore gains list_blob_ids() — walks blobs/**/*.manifest.json
and returns the blob id set. Manifests only, no chunk reads.
New CLI commands:
* claw-store cluster-snapshot-create --name <>
* claw-store cluster-snapshot-list
* claw-store cluster-snapshot-show --name <>
* claw-store cluster-snapshot-delete --name <>
Semantics:
* Snapshots are immutable: create with existing name errors, does
not clobber. Delete-then-create if you really want to overwrite.
* delete() removes only the reference file. Never touches blob
data — protects against operators nuking live data by pruning
snapshots.
* list() sorts by created_at_unix ascending — oldest first so
triage picks pruning candidates quickly.
* blob_ids are sorted at write time so the same content on two
nodes yields byte-identical snapshot files.
* Names validated: no /, \\, NUL, control chars; max 512 bytes.
+8 tests covering create+capture, immutability, get-missing,
list-ordering, delete truth-values, delete-doesn't-touch-blobs,
name-validation, and sorted round-trip.
353 tests pass (+8). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
New command: `claw-store cluster-repair --peer <name> --rpc-addr <host:port>
--tls-dir <dir> [--dry-run]`.
Flow:
1. Local scrub identifies bad chunks (missing + corrupt).
2. Deduplicate to unique chunk-hashes (scrub emits per-reference,
fetcher work is per-chunk).
3. Connect to peer over QUIC + mTLS.
4. For each unique chunk: HasChunk probe → GetChunk on hit →
put locally (re-hashed by put_chunk, so a lying peer can't
corrupt us further).
5. Report attempted/repaired/unrecoverable/errors.
--dry-run stops after the dedup step: prints the plan without
touching the peer or disk.
Behavior details:
* Zero bad chunks → clean exit with no peer contact.
* Any unrecoverable or per-chunk error → non-zero exit so cron/CI
notice. Message names counts.
* HasChunk-first means a peer that lacks the chunk is one cheap
round-trip, not a full GetChunk attempt.
Companion piece for the Phase 7b repair library (already merged).
No new tests here — logic is thin glue over `repair_chunks` +
`call_has_chunk`/`call_get_chunk`, all of which have their own
unit + integration coverage. Behavior gets its real workout in
live smoke on tank+architect.
New primitive: BlobStore::repair_chunks(chunks, fetch) → RepairReport.
Consumer flow: cluster-scrub returns a list of (blob, chunk) bad
pairs. cluster-repair (next slice) will hand the chunk hashes here
with a fetcher that walks peers via HasChunk/GetChunk. This PR is
the library-only half — no peer wiring — so it's testable in
isolation and reusable by callers who already have a chunk source.
Fetcher contract:
* Ok(Some(bytes)) → put locally, count repaired
* Ok(None) → nobody has it, record as unrecoverable
* Err(e) → per-chunk error, batch continues
Guardrails:
* Bytes are re-hashed by put_chunk before writing. A peer that
returns wrong bytes for a hash cannot corrupt us further.
* Duplicate chunk hashes in the input dedupe → fetcher called
exactly once per unique chunk. Matters because scrub reports
shared chunks once per owning manifest.
* Errors on one chunk never abort the batch — the remaining
chunks still get their shot.
* Repair overwrites a corrupt file: unlink-then-put_chunk, since
put_chunk itself is write-if-absent. NotFound on unlink is fine
(missing-chunk case).
+4 tests:
- repair_writes_fetched_bytes_and_marks_repaired (happy: corrupt
→ repair → post-scrub clean)
- repair_records_unrecoverable_when_fetcher_returns_none
- repair_records_error_and_continues_batch (batch survives one
chunk's error)
- repair_dedups_duplicate_chunks_in_input (fetcher called exactly
once for 3 identical hashes)
345 tests pass (+4). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
New primitive: BlobStore::scrub_all() → ScrubReport.
Walks every .manifest.json under blobs/, for each referenced chunk
reads the file from disk and recomputes BLAKE3. Verdict per chunk:
* file absent → missing
* hash mismatch → corrupt
* match → ok
Design points:
* Read-only. Never touches disk state. Safe against a live daemon
— worst case a chunk lands mid-scrub and is skipped this pass.
* Per-reference counting: a bad chunk that N manifests depend on
shows up as N corrupt entries so operators see the full blast
radius. But each unique chunk is hashed exactly once via an
in-memory verdict cache.
* Report holds explicit (blob_id, chunk_hash) pairs for every
bad chunk so the fix path (repair in Phase 7b) has enough
info to act.
CLI: `claw-store cluster-scrub [--verbose]`. Non-zero exit when
integrity issues exist so cron / CI notice.
+4 tests:
- scrub_reports_all_ok_when_store_is_healthy
- scrub_detects_corrupt_chunk (owner blob id preserved)
- scrub_detects_missing_chunk (owner blob id preserved)
- scrub_dedups_shared_chunk_hashing_once (shared chunk, 2 owners
reported, single disk read)
341 tests pass (+4). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
Wires cmd_pin through the WalQueue built in the preceding four
PRs (#48 → #52). First real caller of the client-mode WAL
stack.
New surfaces:
claw-cargo pin --offline --blob <BlobId> [--ttl <duration>]
* no peer connection is opened
* enqueues the same three mutations cmd_pin would emit
online: primary tag (PutTagVersioned), .fingerprint
companion (PutTagVersioned), and — if --ttl — the two
SetTagExpiry sidecars
* requires --blob because offline mode can't do the
GetRefVersioned lookup that resolves fingerprint → BlobId
* prints the assigned WAL seqs + "next step: drain"
claw-cargo drain --peer ...
* opens a peer, drains the queue, truncates up to the last
applied seq
* partial-failure safe: whatever applied is truncated;
anything after a hard error stays on disk for retry
* exits non-zero when drain stopped mid-stream
claw-cargo wal-status
* read-only, no network
* pending count, oldest/newest seq, storage path, decoded
entries (or UNDECODABLE marker on frame errors)
WAL location follows the XDG state-home pattern already used
by manifest.rs:
1. $XDG_STATE_HOME/claw-cargo/wal/
2. $HOME/.local/state/claw-cargo/wal/
3. ./.claw-cargo-wal/ (worst-case container fallback)
Tests (2): default_wal_path_honours_xdg_state_home (mirroring
manifest.rs's env-var pattern) + parse_blob_id_rejects_bad_
hex_and_wrong_len.
claw_cargo.rs grew from 1668 to ~1870 lines. Still under the
1300-per-*module* interpretation but this bin file has been
above 1300 since Phase 5. Split-out is Phase 6 territory.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Collapses the enqueue + drain + truncate dance around
WriteAheadLog + wal_mutation + wal_replay into one API so
downstream callers (Phase 4e: cmd_pin & friends) don't have
to orchestrate three modules themselves.
Two-call flow:
let mut q = WalQueue::open(state_dir.join("wal")).await?;
q.enqueue(&WalMutation::PutTagVersioned { .. }).await?;
// ...later, on reconnect:
let report = q.drain(&conn).await?;
drain() advances the watermark to the last successfully-
applied (or Superseded) seq whether or not the drive stopped
on a hard error mid-stream. Nothing is truncated past the
failure point, so the failing record and everything after
it are retried on the next drain.
Introspection surface (`pending_count` / `oldest_pending_seq`
/ `newest_pending_seq` / `snapshot` / `is_empty`) is what a
metrics endpoint or CLI status view wants. `wal()` escape
hatch exposes the backing WAL for advanced callers.
Tests (6, all green — 4 unit + 2 end-to-end over QUIC):
* empty queue reports empty bounds
* enqueue updates bounds correctly
* snapshot decodes in seq order and preserves kind info
* drain clears the queue and applies to peer (verifies via
call_get_ref + call_get_tag_versioned)
* drain over a pre-seeded dominant version returns
Superseded and still drains the queue
* enqueue survives reopen — bounds recover through
WriteAheadLog::open scan
346 lines, well under the 1300 ceiling.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Given a peer connection + a decoded WalMutation, re-issue the
correct RPC. Closes the loop from "durably logged at client"
to "actually applied at peer" on reconnect.
Outcome classification is deliberate:
* Applied — peer accepted the mutation.
* Superseded — peer already had a dominant version, or the
delete target was absent. NOT a failure; the
mutation's intent matches current peer state.
* Err(_) — genuine RPC failure; caller retries later.
Both Applied and Superseded advance the watermark past the
record — the WAL can safely truncate.
Public surface:
ReplayOutcome { Applied | Superseded }
replay_one(&conn, &mutation) -> Result<ReplayOutcome>
drive_replay(&conn, &wal, start_seq) -> Result<DriveReport>
DriveReport { last_applied, applied, superseded,
skipped, stopped_at: Option<(seq, msg)> }
drive_replay stops on the first hard error and returns
last_applied so the caller can `wal.truncate_up_to(...)`
before closing. Undecodable/unknown-kind records mid-stream
are skipped (with warn!) rather than aborting — otherwise
one bad record would jam an otherwise-good tail forever.
Tests (4, all green, end-to-end over QUIC):
* every variant round-trips; peer state verified via
call_get_ref / call_get_tag_versioned / call_get_tag_expiry
* versioned-reject counts as Superseded, not Err
* DeleteTag on a missing key is Superseded
* undecodable record between two real mutations is skipped;
both good records still apply; last_applied advances past
the skip
434 lines, well under the 1300 ceiling.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Adds the encoding layer that turns the raw WAL (opaque bytes)
into a typed queue of client-mode mutations, ready for Phase 4e
to wire actual RPCs through.
Frame (self-describing, forward-compatible):
version : u8 = 0x01
kind : u8 = one of the Kind discriminants
body : [u8] kind-specific
Body encodings mirror the existing on-wire shapes so a future
replay path can splice a WAL record straight into an RPC payload.
Variants (Kinds 0x01–0x06):
PutRef, PutRefVersioned, PutTag, PutTagVersioned,
DeleteTag, SetTagExpiry
Blob-put mutations are deliberately NOT modeled — blob data is
too large to keep in the WAL. The roaming-client design stages
blobs on local disk and records a reference to them once the
local BlobPutStream completes.
Public helpers:
append_mutation(&mut wal, &m) -> Result<seq>
replay_mutations(&wal, start_seq)
-> Vec<(seq, Result<WalMutation, WalMutationError>)>
Unknown-kind records surface as `Err(UnknownKind(byte))`, not
a panic — forward-compat when a newer writer wrote a record
this reader doesn't understand. Malformed records also surface
as Err so the caller can decide (log-and-skip vs abort replay).
Tests (9, all green): kind-byte stability, roundtrip every
variant, rejects empty/short/bad-version/unknown-kind,
malformed bodies (wrong length, over-declared key_len, trailing
garbage on DeleteTag), non-UTF-8 keys, append+replay through a
real on-disk WAL, and replay-survives-unknown-kind mid-stream.
No new deps — hand-rolled error type in-tree (no thiserror).
515 lines, well under the 1300 ceiling.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Turns the single-file Phase 4c WAL into a segmented log so it
can grow past a single file safely. This unblocks every
downstream Phase 4d/4e integration — reconnect + push loop
can't rely on an unbounded single file.
Layout change:
<root>/segment-<20-digit-first-seq>.bin
20-digit zero-padded first-seq means lex sort == numeric sort,
so `read_dir + sort_by_key` recovers the natural order.
Rotation policy:
* `max_segment_bytes` default 8 MiB, overridable via
`open_with_options`.
* `append` rolls to a fresh segment BEFORE writing when the
current tail is non-empty AND at/above the cap. A single
oversize record always lands in one segment — we never split
a record.
Truncation across segments:
* whole segments with `last_seq <= watermark` are `unlink`'d
* the boundary segment (if any) is rewritten in place via
`tempfile-in-parent + rename` + parent-dir fsync
* full truncation resets head/tail to 0 and the next append
creates a fresh segment
Legacy compat: on open, if a pre-4d `log.bin` is present and
no `segment-*.bin` files exist, it is scanned for its first
seq and renamed to the correct segment name. Refuses to
silently overwrite on filename collision.
Tests (18, all green): rotation-happens-at-cap, reopen-
enumerates-all-segments, truncate-drops-whole-segments,
truncate-partial-rewrites-boundary, oversize-record-still-
fits-one-segment, legacy-log.bin-migration, plus the full
Phase 4c suite (fresh open, append, iter partial ranges,
reopen recovers tail, torn-write truncation, corruption is
hard error, full truncation appendable, below-head no-op,
large payload, empty payload, append-after-reopen).
942-line file, comfortably under the 1300-line ceiling.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Foundation for the roaming/offline-client story described in
Architecture v2 ("Roaming client (full R/W, offline queue)"):
mutating ops append to a durable local log before hitting the
network, and are replayed at reconnect. This PR ships the
primitive; RPC/reconnect wiring lands in Phase 4d.
Segment format (single-file for now — rotation is 4d):
seq : u64 LE (8 bytes)
len : u32 LE (4 bytes) payload length
csum : [u8; 8] (8 bytes) first 8 bytes of
BLAKE3(seq || len || payload)
bytes : [u8; len]
On open, the log is scanned linearly. A short read or truncated
tail is treated as "clean crash boundary" — the file is
size-truncated to the last fully-fsynced record, no error.
A checksum mismatch on a full-length record is fatal (real
corruption, don't silently swallow data).
Public API:
WriteAheadLog::open(root) -> Self
wal.append(&[u8]) -> Result<u64> // durable, fsynced
wal.iter_from(start_seq) -> Vec<WalRecord>
wal.truncate_up_to(watermark) -> () // atomic rewrite via
// tempfile-in-parent + rename
wal.head_seq() / wal.tail_seq() / wal.is_empty()
Tests (12, all green): fresh open, monotonic seq, replay full &
partial ranges, reopen-recovers-tail, torn-write truncation on
open, corruption is hard error, prefix truncation, full
truncation leaves appendable, below-head no-op, 1 MiB payload
roundtrip, empty payload roundtrip, append-after-reopen.
No new deps — BLAKE3 (already a dep) supplies the checksum.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Closes Phase 4b by exposing the TTL sidecar written by
Phase 4b primitives on the wire and via `claw-cargo pin`.
Wire additions:
* Method::SetTagExpiry (0x1a) — payload `key_len:u16 || key ||
expires_at:u64 (LE)`. Reply single-byte OK. `expires_at == 0`
clears the sidecar.
* Method::GetTagExpiry (0x1b) — payload raw key bytes. Reply 8
bytes (u64 LE) on hit; NotFound when no sidecar is present.
Both accept writes even when the stamped tag itself is absent,
matching `TagStore::set_stamped_expiry` semantics — the sidecar
takes effect the moment the tag lands.
CLI:
* `claw-cargo pin --ttl <duration>` — humantime-style duration
(`30d`, `1h30m`, `2w`, ...). Applied to both the primary tag
and its `.fingerprint` companion so eviction treats them as
one lifetime. `--ttl 0` / `clear` / `none` clears an existing
sidecar without touching the value.
Tests: encode/decode roundtrip + malformed-input rejection for
`encode_expiry_record`, method-byte stability, NotConfigured
without a tag store, end-to-end set/get/overwrite/clear over
QUIC, and a real-pin flow that publishes a stamped tag then
attaches TTL. Duration parser is unit-tested for single/compound
forms, case-insensitive units, bad input, and clock alignment.
No new deps — the humantime-style parser is 60 lines in-tree.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Adds the on-disk mechanism for time-scoped pins. No RPC or CLI yet
— a follow-on will expose \`pin --ttl <duration>\`. This PR is
purely library + eviction wiring.
Layout addition: alongside each stamped tag at
\`tags-v2/<hh>/<hash>.svtag\`, an optional sidecar
\`tags-v2/<hh>/<hash>.svtag.exp\` holds an 8-byte LE unix
\`expires_at\`. Absence of the sidecar = never expires (current
behavior).
New TagStore methods:
* set_stamped_expiry(key, expires_at_unix) — writes sidecar;
passing 0 removes it. Idempotent.
* get_stamped_expiry(key) — reads sidecar; None when absent.
* pinned_blob_values_at(now_unix) — same union as
pinned_blob_values, but skips stamped tags whose sidecar shows
expires_at ≤ now. Legacy tags/ entries never expire.
* prune_expired_stamped_at(now_unix) — deletes stamped tags AND
their sidecars where expires_at ≤ now. Returns count.
* pinned_blob_values() — now a shim that calls _at(u64::MAX) for
100% backward compat.
Wired the two existing gc call sites:
* ClusterServices auto-GC ticker prunes-then-collects at
SystemTime::now(). One pass per tick.
* \`claw-store cluster-gc --evict-to-gb N\` CLI same pattern.
Report now includes \"expired pins pruned: N\".
+1 test (expiry_gates_pin_set_and_prune_removes_expired):
covers live/expired/no-ttl mix, sidecar round-trip, prune
removes only expired, expires_at=0 clears sidecar, dropped
tag stops filtering.
286 tests pass (+3 from 283). Pre-existing macOS
hot::tests::test_project_target_size_bytes failure unchanged.
pin lookup was legacy call_get_ref only, missing refs written via
call_put_ref_versioned (all Phase 3b+ builds). Try versioned first,
fall back to legacy — same pattern as claw-cargo build path.
osobh
Omar Sobh
