Concurrent PutRef safety via (clock, node) total order. New wire
methods PutRefVersioned (0x15) + GetRefVersioned (0x16). Existing
PutRef/GetRef unchanged for backward compat. Data in refs-v2/
namespace so the two coexist during cutover.
+8 tests, 272 total (baseline +8).
Cross-runner cache silos (tank + architect measured on 2026-07-13):
same fingerprint, same rustc, but each runner's daemon only knows
about the refs its own runner uploaded. Every runner that lands on
a peer that isn't tank re-uploads a duplicate blob.
Fix: on `GetRef` miss the daemon fans out to alive gossip peers
via a strict-local `GetRefLocal` variant, and the FIRST peer that
has the ref triggers a transparent pull — chunks + manifest into
the local blob store, then `PutRef` locally — before returning the
value to the caller. Subsequent lookups are pure-local hits.
* `Method::GetRefLocal = 0x14` — new wire method, identical shape
to GetRef but the peer MUST NOT recurse. Loop prevention: our
forwarding only calls `GetRefLocal` on peers, so chain depth is
always 1.
* `RpcRouter::with_outbound_client(Arc<QuicClient>)` — dependency
injection point for the forwarding dial path. `None` disables
forwarding entirely (GetRef becomes GetRefLocal-equivalent).
* `RpcRouter::forward_get_ref(key)` — concurrent peer probes via
`JoinSet`, 3s timeout per dial, first successful pull wins,
remaining tasks aborted.
* `pull_blob_locally` — walks manifest, fetches only chunks the
local store lacks (`has_chunk`), commits via
`put_manifest_verified`. Bounded memory: one 4 MiB chunk at a
time.
* `ClusterServices::start` loads NodeIdentity twice — server takes
ownership; outbound client gets its own copy for TLS presentation
on peer dials. Wires the outbound client into the router when
TLS material is available.
* `call_get_ref_local(conn, key)` client helper (used by daemon
forwarding + available to any RPC consumer that wants the
no-recursion semantics).
+3 tests in `rpc/tests_forwarding.rs`:
- Local hit works without forwarding; local miss with no peers
returns None. Guards the base cases.
- GetRefLocal never forwards even when outbound is configured (no
peers reachable → miss returns None immediately, no attempted
fan-out).
- Method byte 0x14 encoding is stable across releases.
Full end-to-end forwarding is exercised in the pilot deploy: two
daemons on the fleet-CA, tank populates a ref, architect's runner
GetRef → tank forwards → architect pulls → HIT locally next time.
264 tests pass (baseline +3). Pre-existing macOS failure unchanged.
Three fixes surfaced by the 2026-07-13 Gitea Actions wire-up.
## XDG config path
Before: `client_config::user_config_path` only looked at
`~/.claw-cargo/config.toml`. My runner-integration doc initially
told operators to install at `~/.config/claw-cargo/config.toml`
(XDG-style). Config wasn't loaded.
Now: three-way lookup, first hit wins.
1. `$XDG_CONFIG_HOME/claw-cargo/config.toml`
2. `$HOME/.config/claw-cargo/config.toml`
3. `$HOME/.claw-cargo/config.toml` (legacy, still honoured)
+3 tests: XDG env wins when the file exists, .config wins over
legacy dotfile when both present, legacy dotfile returned as
error-message fallback when none exist.
## Composite action
Before: composite action silently no-op'd. Log showed the script
lines echoed but only the `if command -v claw-cargo` fail branch
ran. Root cause: Gitea Actions composite steps run with a stripped
PATH that omits `/usr/local/bin`.
Now: composite step exports PATH defensively:
export PATH="/usr/local/bin:$HOME/.cargo/bin:$PATH"
And it looks for the config at BOTH the XDG-style path and the
legacy dotfile (same order as client_config).
Workflow file back to using the composite action.
## Docs
Runner-integration doc now:
- Explicitly warns that `ubuntu-latest` routes to container mode
even with `:host` suffix on runner labels (field-observed).
- Documents the dedicated `clawstor-cache` label pattern that
works.
- Sample workflow uses `runs-on: clawstor-cache` instead of
`ubuntu-latest`.
+3 tests, 262 total (baseline unchanged).
client_config.rs load_layered looks at ~/.claw-cargo/config.toml,
not ~/.config/claw-cargo/config.toml. Fix the workflow preflight
path to match. Both runners already have the file at both locations.
Fleet has 7 linux-amd64 runners; only tank + architect have
claw-cargo provisioned. Added a clawstor-cache:host label to those
two runners so this workflow only lands on them.
ubuntu-latest routes to container mode in act_runner even with the
:host suffix on the runner labels. Explicit self-hosted forces
host-mode where /usr/local/bin/claw-cargo + per-runner tls_dir are
visible.
Host runners have cmake/gcc/pkg-config from the OS and cargo/rustup
in the act_runner user's ~/.cargo/bin. apt-get needs root — the
runner isn't. Replace with a preflight that fails fast when any
tool is missing.
Ships the wire-up piece for real CI: a composite Gitea Action that
wraps `claw-cargo build` with cache-outcome reporting, plus a
matching workflow file that opts the clawstor repo itself into
being cache-hit-tested on every push. Also docs the one-time
per-runner provisioning (leaf cert, PATH install, config.toml).
* `.gitea/actions/cargo-cache/action.yml` — composite Action.
Inputs: workspace, profile, no-upload, parallel-restore. Outputs:
cache-outcome (HIT|MISS|POPULATED|SKIPPED), fingerprint,
elapsed-seconds. Runner-side config lives in
`~/.config/claw-cargo/config.toml` (not in the workflow — no
secrets shipped from repos).
* `.gitea/workflows/build-with-cache.yml` — dogfoods the action on
clawstor's own repo. `no-upload` set from event_name so PRs from
forks can't poison the cache.
* `docs/runner-integration.md` — one-time setup steps, sample
workflow snippet, expected numbers (Pi 5: 2.79× wall, tank:
2.18×), and troubleshooting for the failures I hit in the tank
and Pi pilots (bind_lan on fabric-only, missing CLI/config,
rustc drift warn).
Test protocol: push this branch → main triggers the workflow → the
runner on tank has claw-cargo + tls + config provisioned already
(2026-07-13 pilot setup) → first build should MISS + populate,
subsequent build on same fingerprint should HIT.
Pi 5 loopback measurement 2026-07-13:
--parallel-restore 1 : wall 2m52s, restore 20s
--parallel-restore 8 : wall 3m06s, restore 34s
Sequential is 70% faster on loopback. N-way stream contention costs
more than a single stream's congestion-control amortization. Same
shape as Phase 5k prewarm — fanout only wins when per-stream
throughput has a ceiling (WAN, tunneled links).
--parallel-restore N remains as opt-in.
Follow-up to PR #28. Fresh Pi deploy 2026-07-12 hit a `ProtectHome=
read-only` block on the daemon's XDG-driven
$HOME/.local/state/claw-store/projects.toml write. Adding the path
to `ReadWritePaths` in the shipped unit means future deployers
don't need a drop-in.
Two fixes surfaced by the vision-02 Pi 5 measurement:
## XDG default_path
`Manifest::default_path` was hardcoded to
`/var/lib/claw-store/projects.toml`. That path is read-only under
the user-mode systemd unit's `ProtectSystem=strict`, and creating
it needs root — awful for a runner install.
Precedence, matching XDG Base Directory:
1. `$XDG_STATE_HOME/claw-store/projects.toml`
2. `$HOME/.local/state/claw-store/projects.toml`
3. `/var/lib/claw-store/projects.toml` (system fallback)
User-mode installs now write in $HOME by default; system installs
(root, no HOME set) still land in /var/lib.
+1 test: `default_path_honours_xdg_state_home` — covers all three
precedence branches. Env mutation is process-global so the test
saves + restores.
## Parallel restore on cache HIT
Pi restore of 947 MiB via `BlobGetStream` took ~18s (~53 MiB/s)
single-stream. Per-stream throughput ceilings on the connection
type cap sequential fetches; parallel chunk fetches stack their
contributions.
- New `call_blob_get_parallel(conn, blob_id, concurrency) ->
Option<Vec<u8>>` in `rpc/client.rs`. `JoinSet` + `Semaphore`,
reassembles by chunk index at manifest-known offsets so
out-of-order arrival is fine.
- `claw-cargo build --parallel-restore N` (default 8). `N <= 1`
falls through to `BlobGetStream` for parity.
- Memory: `total_size + 4 MiB × in-flight` — dominated by the
reassembly buffer, not the fanout.
+1 test: `parallel_blob_get_reassembles_multi_chunk_blob_byte_equal`
covers roundtrip byte-equality vs BlobGetStream, tail-chunk offset,
concurrency=1 correctness, and NotFound → None.
259 tests pass (+2). Pre-existing macOS failure unchanged.
Orphan-chunk GC alone doesn't stop unbounded growth: as long as
fingerprint→blob refs keep getting PutRef'd, the manifest set keeps
growing and no chunk is ever an orphan.
* `BlobStore::evict_to_size_cap(max_bytes)` — walks manifests oldest
first by mtime, deletes them, refcount-decrements each chunk they
used, unlinks + reclaims size for any chunk whose refcount hits
zero. Shared chunks stay put until the last blob referencing them
is evicted.
* `ManifestSummary` internal type keeps the diff-set bookkeeping
cheap (one HashMap<ChunkHash, u32>, no repeated tree walks).
* `claw-store cluster-gc --evict-to-gb <N>` extends the CLI: still
runs the orphan sweep first, then optionally caps the store.
* Config: `cluster.blob_max_gb: Option<u64>`. The auto-GC ticker
runs eviction after every orphan sweep when this is set. Silent
when the store is already under cap; INFO log when it evicts.
+3 tests:
- evict_to_size_cap_reclaims_oldest_blobs_first: 3 blobs with
distinct mtimes, cap below combined size → oldest evicted,
newer blobs survive
- evict_keeps_shared_chunks_when_still_referenced: guards the
refcount decrement path (content-addressed dedup keeps identical
content as one blob → chunk survives until manifest deleted)
- evict_on_empty_store_is_a_noop: sanity
257 tests pass (baseline +3). Pre-existing macOS failure unchanged.
Both surfaced by the 2026-07-12 pilot as real operator concerns:
## rustc drift warning at build time
Runners silently silo their cache when rustc versions differ across
peers (fingerprint depends on rustc verbose output). The pilot's
first flow burned a full cold+upload before we realized the silo.
- `PeerStatusReply.local_rustc_release` — new field, populated from
the peer's own gossip `RUSTC_RELEASE` key via a new
`ClusterGossip::self_kv(key)` accessor.
- `claw-cargo build`: on cache MISS, calls `PeerStatus`; if the
peer's rustc release ≠ our local `rustc --version`, emits a WARN
with both versions + hint to add `rust-toolchain.toml`.
- Best-effort: absence of either release string is a shrug, not
an error.
## blob GC
Blob store grows unbounded on a runner; disk-full is a real
incident. `gc_orphan_chunks` already existed but wasn't exposed.
- New CLI: `claw-store cluster-gc` — runs `gc_orphan_chunks`,
prints report. Safe to run any time, safe to interrupt.
- New config: `cluster.gc_interval_hours: Option<u64>`. When set to
a positive integer, the daemon spawns a periodic ticker that
invokes GC in-process. Skips the first tick (nothing to reclaim
on boot). Errors are logged and retried next tick.
- Shutdown aborts the ticker cleanly.
254 tests pass (baseline unchanged). Pre-existing macOS failure
untouched.
Pilot 2026-07-12 measured 109 MiB/s on the sequential prewarm path —
~11% of a 10G fabric. `quinn::Connection` is cheap-Clone (internal
Arc), so we can run the has→get→put pipeline per chunk in concurrent
tasks under a bounded semaphore.
- `prewarm_missing_chunks_between_parallel(up, down, id, concurrency)`
in `rpc/client.rs`. `concurrency <= 1` degrades to the sequential
path (kept for diagnostic parity).
- `claw-cargo prewarm --parallel N` (default 8). Ignored with
`--buffered`. Memory ceiling: 4 MiB × in-flight = 32 MiB @ 8,
128 MiB @ 32.
- Uses `tokio::task::JoinSet` + `Arc<Semaphore>`; permit held for
the whole per-chunk pipeline so we never over-commit.
- Retry pass on `put_manifest` mismatch stays sequential — small,
correctness-critical.
- Errors: JoinSet drains completely + returns first task error so a
mid-fanout failure doesn't leave zombie tasks.
+1 test: `end_to_end_parallel_prewarm_copies_chunks_and_matches_sequential`
runs 5-chunk payload with concurrency=3, verifies byte-equal restore,
then reruns with concurrency=8 → 0 uploads (has_chunk dedup), then
concurrency=0 → 0 uploads (sequential fallback path).
254 tests pass (+1 from previous). Pre-existing macOS failure unchanged.
Pilot ran the cluster daemon under nohup; production needs proper
restart-on-failure + clean PATH inheritance (rustc gossip probe
needs `~/.cargo/bin` on PATH, which nohup's env didn't get).
The unit is user-scoped (`~/.config/systemd/user/`) so it works
without root on the pilot nodes:
cp systemd/clawstor-cluster.service ~/.config/systemd/user/
loginctl enable-linger $USER
systemctl --user daemon-reload
systemctl --user enable --now clawstor-cluster.service
Defaults:
- CLAWSTOR_BIN = ~/clawstor-deploy/claw-store
- CLAWSTOR_CONFIG = ~/clawstor-deploy/config.toml
- PATH prefixed with ~/.cargo/bin so rustc is found
Override any of those via `systemctl --user edit
clawstor-cluster.service`.
Security hardening:
- NoNewPrivileges=yes
- ProtectSystem=strict (system dirs read-only)
- ProtectHome=read-only (home dir read-only)
- ReadWritePaths=%h/clawstor-deploy (only the deploy tree is
writable)
- PrivateTmp=yes
Restart semantics:
- Restart=on-failure with RestartSec=10 — pilot-verified: kill -9
the daemon PID and the service comes back within ~10s
- TimeoutStopSec=60 so a slow gossip departure can complete
Deployed to tank + architect 2026-07-12 as part of the pilot
retest.
Field finding 2026-07-12 (clawverse measurement): the buffered
`capture_target -> Vec<u8>` path peaked at 2.8 GB RAM to capture a
6.1 GB target/debug into a 995 MiB compressed tar. Every byte
crossed RAM before touching the network.
* `capture_target_to_writer(target_dir, writer) -> u64` — new
streaming variant. Walks the tree + writes tar+zstd straight into
the caller's Writer via a small ByteCounter wrapper. Peak memory
stays at ~zstd sliding window size (few MB).
* `capture_target -> Vec<u8>` kept as a thin wrapper for the tests
+ smaller callers that don't care.
* `cmd_build`: capture into a tempfile under `target/`, then open
it with `tokio::fs::File` (AsyncRead + Unpin) and hand that to
`call_blob_put_stream`. Same-filesystem tempfile means no cross-
mount concerns; auto-unlinks on drop.
+1 test: `capture_streaming_matches_buffered_and_restores_correctly`
proves the streamed bytes match the buffered variant, the reported
byte count agrees with the written length, and roundtrip restore
from the streamed file works.
Combined with PR #22 (QUIC idle timeout), this closes the two RAM/
timeout blockers surfaced by the clawverse pilot. Expected memory
ceiling on a runner drops from GBs to MBs, unlocking small-runner
deployments (the actual pitch use case).
Field finding 2026-07-12 (clawverse cold on tank):
Compiling claw-cli v0.1.0 (...)
Finished `dev` profile ... in 45.08s
cargo build finished in 45.135491979s
Error: opening bidi stream for BlobPutStream
Caused by: timed out
Cargo took 45s → QUIC's 30s idle timeout killed the connection between
the initial peer-lookup connect and the follow-up capture+upload path.
The RPC never got a chance to open its stream.
Fix: two belt-and-braces changes:
1. IDLE_TIMEOUT 30s → 600s. The timeout is there to detect crashed
peers, not to enforce build pacing.
2. Client applies a `keep_alive_interval` of 15s so the connection
stays warm across cargo runs even shorter than the idle window.
quinn's keep-alive fires from an internal runtime task, not the app
thread, so a fully-CPU-pinned cargo build doesn't suppress it.
Bundles the profile→dir bug (PR #20 supersede) with four new fixes
discovered by running clawstor against itself + across the fabric:
* target_subdir_for: `dev`/`test` → `debug/`, `release`/`bench` →
`release/`, custom passes through. Was silently skipping upload.
* rustc release via gossip: daemon probes `rustc --version` at start,
publishes the release string as `clawstor.rustc.release`. PeerView
carries it; `cluster-peer-status` prints it in a new column and
emits a warning line when the fleet has mixed versions. Would have
surfaced the tank/architect 1.96.1 vs 1.95.0 drift instantly.
* prewarm publishes fingerprint→blob ref downstream: `pin` now writes
a companion tag `<name>.fingerprint` holding the fingerprint bytes.
`prewarm` reads the companion, PutTag's it downstream, then
PutRef(fp→blob) so a subsequent fingerprint-based `build` HITS.
Without this, prewarm was almost useless for the runner path
(build always missed even with matching source + rustc).
* streaming byte counters: BlobPutStream + BlobGetStream now record
the transferred bytes via `record_blob_{put,get}_bytes`. Metric
used to stay at 0 no matter how much you moved.
* capture determinism: replaced `tar::Builder::append_dir_all` (uses
`read_dir`'s native order) with `append_dir_sorted` that walks the
tree recursively and sorts by filename bytes at every level. Two
byte-identical trees now produce byte-identical tars regardless of
filesystem ordering.
+3 tests:
- target_subdir_matches_cargo_layout (from #20)
- fingerprint_companion_tag_uses_dotted_suffix
- capture_is_order_independent_of_filesystem_readdir (guard against
the exact bug we saw in the field)
252 tests pass (+1 from Phase 5h's 251). Pre-existing macOS `du -sb`
failure unchanged.
Supersedes #20 (also included here). Ready for re-deploy to
tank + architect for the retest run.
Bounded-memory cross-peer prewarm: instead of buffering the whole
blob in RAM (previous 5f path), iterate the upstream manifest chunk
by chunk, ask downstream `HasChunk`, stream missing chunks one at a
time. Memory ceiling is 1 chunk (4 MiB) regardless of blob size — a
5 GiB target dir no longer needs 5 GiB of mediator RAM.
* rpc/client.rs: new `prewarm_missing_chunks_between(upstream,
downstream, blob_id)` helper. Returns `(uploaded, total)` — the
difference is the dedup save. Retries once if the downstream
`PutManifest` reports missing chunks after our push (guards a
narrow eviction race); a second failure surfaces as `Err`.
* claw_cargo.rs: `prewarm` now streams by default; new `--buffered`
flag for the old whole-blob path (kept for diagnostic
comparability during rollout). Human-readable output shows the
mode + dedup count.
+2 tests:
- cold downstream: 3-chunk payload (with a partial tail chunk) is
copied exactly, reassembly is byte-equal to source
- partial dedup: pre-seed 1 of 3 chunks on downstream → uploaded=2;
rerun is a full no-op (uploaded=0), proving idempotence
251 tests pass (+2 from Phase 5j). Pre-existing macOS failure
unchanged.
Follow-ons: (1) parallel chunk transfer (uploaded chunks in fan-out)
would speed multi-GB prewarms further; (2) exposing an accurate
transferred-bytes counter needs router-side accounting instead of
the current chunk-count × CHUNK_SIZE approximation.
osobh
Omar Sobh