client_config.rs load_layered looks at ~/.claw-cargo/config.toml,
not ~/.config/claw-cargo/config.toml. Fix the workflow preflight
path to match. Both runners already have the file at both locations.
Fleet has 7 linux-amd64 runners; only tank + architect have
claw-cargo provisioned. Added a clawstor-cache:host label to those
two runners so this workflow only lands on them.
ubuntu-latest routes to container mode in act_runner even with the
:host suffix on the runner labels. Explicit self-hosted forces
host-mode where /usr/local/bin/claw-cargo + per-runner tls_dir are
visible.
Host runners have cmake/gcc/pkg-config from the OS and cargo/rustup
in the act_runner user's ~/.cargo/bin. apt-get needs root — the
runner isn't. Replace with a preflight that fails fast when any
tool is missing.
Ships the wire-up piece for real CI: a composite Gitea Action that
wraps `claw-cargo build` with cache-outcome reporting, plus a
matching workflow file that opts the clawstor repo itself into
being cache-hit-tested on every push. Also docs the one-time
per-runner provisioning (leaf cert, PATH install, config.toml).
* `.gitea/actions/cargo-cache/action.yml` — composite Action.
Inputs: workspace, profile, no-upload, parallel-restore. Outputs:
cache-outcome (HIT|MISS|POPULATED|SKIPPED), fingerprint,
elapsed-seconds. Runner-side config lives in
`~/.config/claw-cargo/config.toml` (not in the workflow — no
secrets shipped from repos).
* `.gitea/workflows/build-with-cache.yml` — dogfoods the action on
clawstor's own repo. `no-upload` set from event_name so PRs from
forks can't poison the cache.
* `docs/runner-integration.md` — one-time setup steps, sample
workflow snippet, expected numbers (Pi 5: 2.79× wall, tank:
2.18×), and troubleshooting for the failures I hit in the tank
and Pi pilots (bind_lan on fabric-only, missing CLI/config,
rustc drift warn).
Test protocol: push this branch → main triggers the workflow → the
runner on tank has claw-cargo + tls + config provisioned already
(2026-07-13 pilot setup) → first build should MISS + populate,
subsequent build on same fingerprint should HIT.
Pi 5 loopback measurement 2026-07-13:
--parallel-restore 1 : wall 2m52s, restore 20s
--parallel-restore 8 : wall 3m06s, restore 34s
Sequential is 70% faster on loopback. N-way stream contention costs
more than a single stream's congestion-control amortization. Same
shape as Phase 5k prewarm — fanout only wins when per-stream
throughput has a ceiling (WAN, tunneled links).
--parallel-restore N remains as opt-in.
Follow-up to PR #28. Fresh Pi deploy 2026-07-12 hit a `ProtectHome=
read-only` block on the daemon's XDG-driven
$HOME/.local/state/claw-store/projects.toml write. Adding the path
to `ReadWritePaths` in the shipped unit means future deployers
don't need a drop-in.
Two fixes surfaced by the vision-02 Pi 5 measurement:
## XDG default_path
`Manifest::default_path` was hardcoded to
`/var/lib/claw-store/projects.toml`. That path is read-only under
the user-mode systemd unit's `ProtectSystem=strict`, and creating
it needs root — awful for a runner install.
Precedence, matching XDG Base Directory:
1. `$XDG_STATE_HOME/claw-store/projects.toml`
2. `$HOME/.local/state/claw-store/projects.toml`
3. `/var/lib/claw-store/projects.toml` (system fallback)
User-mode installs now write in $HOME by default; system installs
(root, no HOME set) still land in /var/lib.
+1 test: `default_path_honours_xdg_state_home` — covers all three
precedence branches. Env mutation is process-global so the test
saves + restores.
## Parallel restore on cache HIT
Pi restore of 947 MiB via `BlobGetStream` took ~18s (~53 MiB/s)
single-stream. Per-stream throughput ceilings on the connection
type cap sequential fetches; parallel chunk fetches stack their
contributions.
- New `call_blob_get_parallel(conn, blob_id, concurrency) ->
Option<Vec<u8>>` in `rpc/client.rs`. `JoinSet` + `Semaphore`,
reassembles by chunk index at manifest-known offsets so
out-of-order arrival is fine.
- `claw-cargo build --parallel-restore N` (default 8). `N <= 1`
falls through to `BlobGetStream` for parity.
- Memory: `total_size + 4 MiB × in-flight` — dominated by the
reassembly buffer, not the fanout.
+1 test: `parallel_blob_get_reassembles_multi_chunk_blob_byte_equal`
covers roundtrip byte-equality vs BlobGetStream, tail-chunk offset,
concurrency=1 correctness, and NotFound → None.
259 tests pass (+2). Pre-existing macOS failure unchanged.
Orphan-chunk GC alone doesn't stop unbounded growth: as long as
fingerprint→blob refs keep getting PutRef'd, the manifest set keeps
growing and no chunk is ever an orphan.
* `BlobStore::evict_to_size_cap(max_bytes)` — walks manifests oldest
first by mtime, deletes them, refcount-decrements each chunk they
used, unlinks + reclaims size for any chunk whose refcount hits
zero. Shared chunks stay put until the last blob referencing them
is evicted.
* `ManifestSummary` internal type keeps the diff-set bookkeeping
cheap (one HashMap<ChunkHash, u32>, no repeated tree walks).
* `claw-store cluster-gc --evict-to-gb <N>` extends the CLI: still
runs the orphan sweep first, then optionally caps the store.
* Config: `cluster.blob_max_gb: Option<u64>`. The auto-GC ticker
runs eviction after every orphan sweep when this is set. Silent
when the store is already under cap; INFO log when it evicts.
+3 tests:
- evict_to_size_cap_reclaims_oldest_blobs_first: 3 blobs with
distinct mtimes, cap below combined size → oldest evicted,
newer blobs survive
- evict_keeps_shared_chunks_when_still_referenced: guards the
refcount decrement path (content-addressed dedup keeps identical
content as one blob → chunk survives until manifest deleted)
- evict_on_empty_store_is_a_noop: sanity
257 tests pass (baseline +3). Pre-existing macOS failure unchanged.
Both surfaced by the 2026-07-12 pilot as real operator concerns:
## rustc drift warning at build time
Runners silently silo their cache when rustc versions differ across
peers (fingerprint depends on rustc verbose output). The pilot's
first flow burned a full cold+upload before we realized the silo.
- `PeerStatusReply.local_rustc_release` — new field, populated from
the peer's own gossip `RUSTC_RELEASE` key via a new
`ClusterGossip::self_kv(key)` accessor.
- `claw-cargo build`: on cache MISS, calls `PeerStatus`; if the
peer's rustc release ≠ our local `rustc --version`, emits a WARN
with both versions + hint to add `rust-toolchain.toml`.
- Best-effort: absence of either release string is a shrug, not
an error.
## blob GC
Blob store grows unbounded on a runner; disk-full is a real
incident. `gc_orphan_chunks` already existed but wasn't exposed.
- New CLI: `claw-store cluster-gc` — runs `gc_orphan_chunks`,
prints report. Safe to run any time, safe to interrupt.
- New config: `cluster.gc_interval_hours: Option<u64>`. When set to
a positive integer, the daemon spawns a periodic ticker that
invokes GC in-process. Skips the first tick (nothing to reclaim
on boot). Errors are logged and retried next tick.
- Shutdown aborts the ticker cleanly.
254 tests pass (baseline unchanged). Pre-existing macOS failure
untouched.
Pilot 2026-07-12 measured 109 MiB/s on the sequential prewarm path —
~11% of a 10G fabric. `quinn::Connection` is cheap-Clone (internal
Arc), so we can run the has→get→put pipeline per chunk in concurrent
tasks under a bounded semaphore.
- `prewarm_missing_chunks_between_parallel(up, down, id, concurrency)`
in `rpc/client.rs`. `concurrency <= 1` degrades to the sequential
path (kept for diagnostic parity).
- `claw-cargo prewarm --parallel N` (default 8). Ignored with
`--buffered`. Memory ceiling: 4 MiB × in-flight = 32 MiB @ 8,
128 MiB @ 32.
- Uses `tokio::task::JoinSet` + `Arc<Semaphore>`; permit held for
the whole per-chunk pipeline so we never over-commit.
- Retry pass on `put_manifest` mismatch stays sequential — small,
correctness-critical.
- Errors: JoinSet drains completely + returns first task error so a
mid-fanout failure doesn't leave zombie tasks.
+1 test: `end_to_end_parallel_prewarm_copies_chunks_and_matches_sequential`
runs 5-chunk payload with concurrency=3, verifies byte-equal restore,
then reruns with concurrency=8 → 0 uploads (has_chunk dedup), then
concurrency=0 → 0 uploads (sequential fallback path).
254 tests pass (+1 from previous). Pre-existing macOS failure unchanged.
Pilot ran the cluster daemon under nohup; production needs proper
restart-on-failure + clean PATH inheritance (rustc gossip probe
needs `~/.cargo/bin` on PATH, which nohup's env didn't get).
The unit is user-scoped (`~/.config/systemd/user/`) so it works
without root on the pilot nodes:
cp systemd/clawstor-cluster.service ~/.config/systemd/user/
loginctl enable-linger $USER
systemctl --user daemon-reload
systemctl --user enable --now clawstor-cluster.service
Defaults:
- CLAWSTOR_BIN = ~/clawstor-deploy/claw-store
- CLAWSTOR_CONFIG = ~/clawstor-deploy/config.toml
- PATH prefixed with ~/.cargo/bin so rustc is found
Override any of those via `systemctl --user edit
clawstor-cluster.service`.
Security hardening:
- NoNewPrivileges=yes
- ProtectSystem=strict (system dirs read-only)
- ProtectHome=read-only (home dir read-only)
- ReadWritePaths=%h/clawstor-deploy (only the deploy tree is
writable)
- PrivateTmp=yes
Restart semantics:
- Restart=on-failure with RestartSec=10 — pilot-verified: kill -9
the daemon PID and the service comes back within ~10s
- TimeoutStopSec=60 so a slow gossip departure can complete
Deployed to tank + architect 2026-07-12 as part of the pilot
retest.
Field finding 2026-07-12 (clawverse measurement): the buffered
`capture_target -> Vec<u8>` path peaked at 2.8 GB RAM to capture a
6.1 GB target/debug into a 995 MiB compressed tar. Every byte
crossed RAM before touching the network.
* `capture_target_to_writer(target_dir, writer) -> u64` — new
streaming variant. Walks the tree + writes tar+zstd straight into
the caller's Writer via a small ByteCounter wrapper. Peak memory
stays at ~zstd sliding window size (few MB).
* `capture_target -> Vec<u8>` kept as a thin wrapper for the tests
+ smaller callers that don't care.
* `cmd_build`: capture into a tempfile under `target/`, then open
it with `tokio::fs::File` (AsyncRead + Unpin) and hand that to
`call_blob_put_stream`. Same-filesystem tempfile means no cross-
mount concerns; auto-unlinks on drop.
+1 test: `capture_streaming_matches_buffered_and_restores_correctly`
proves the streamed bytes match the buffered variant, the reported
byte count agrees with the written length, and roundtrip restore
from the streamed file works.
Combined with PR #22 (QUIC idle timeout), this closes the two RAM/
timeout blockers surfaced by the clawverse pilot. Expected memory
ceiling on a runner drops from GBs to MBs, unlocking small-runner
deployments (the actual pitch use case).
Field finding 2026-07-12 (clawverse cold on tank):
Compiling claw-cli v0.1.0 (...)
Finished `dev` profile ... in 45.08s
cargo build finished in 45.135491979s
Error: opening bidi stream for BlobPutStream
Caused by: timed out
Cargo took 45s → QUIC's 30s idle timeout killed the connection between
the initial peer-lookup connect and the follow-up capture+upload path.
The RPC never got a chance to open its stream.
Fix: two belt-and-braces changes:
1. IDLE_TIMEOUT 30s → 600s. The timeout is there to detect crashed
peers, not to enforce build pacing.
2. Client applies a `keep_alive_interval` of 15s so the connection
stays warm across cargo runs even shorter than the idle window.
quinn's keep-alive fires from an internal runtime task, not the app
thread, so a fully-CPU-pinned cargo build doesn't suppress it.
Bundles the profile→dir bug (PR #20 supersede) with four new fixes
discovered by running clawstor against itself + across the fabric:
* target_subdir_for: `dev`/`test` → `debug/`, `release`/`bench` →
`release/`, custom passes through. Was silently skipping upload.
* rustc release via gossip: daemon probes `rustc --version` at start,
publishes the release string as `clawstor.rustc.release`. PeerView
carries it; `cluster-peer-status` prints it in a new column and
emits a warning line when the fleet has mixed versions. Would have
surfaced the tank/architect 1.96.1 vs 1.95.0 drift instantly.
* prewarm publishes fingerprint→blob ref downstream: `pin` now writes
a companion tag `<name>.fingerprint` holding the fingerprint bytes.
`prewarm` reads the companion, PutTag's it downstream, then
PutRef(fp→blob) so a subsequent fingerprint-based `build` HITS.
Without this, prewarm was almost useless for the runner path
(build always missed even with matching source + rustc).
* streaming byte counters: BlobPutStream + BlobGetStream now record
the transferred bytes via `record_blob_{put,get}_bytes`. Metric
used to stay at 0 no matter how much you moved.
* capture determinism: replaced `tar::Builder::append_dir_all` (uses
`read_dir`'s native order) with `append_dir_sorted` that walks the
tree recursively and sorts by filename bytes at every level. Two
byte-identical trees now produce byte-identical tars regardless of
filesystem ordering.
+3 tests:
- target_subdir_matches_cargo_layout (from #20)
- fingerprint_companion_tag_uses_dotted_suffix
- capture_is_order_independent_of_filesystem_readdir (guard against
the exact bug we saw in the field)
252 tests pass (+1 from Phase 5h's 251). Pre-existing macOS `du -sb`
failure unchanged.
Supersedes #20 (also included here). Ready for re-deploy to
tank + architect for the retest run.
Bounded-memory cross-peer prewarm: instead of buffering the whole
blob in RAM (previous 5f path), iterate the upstream manifest chunk
by chunk, ask downstream `HasChunk`, stream missing chunks one at a
time. Memory ceiling is 1 chunk (4 MiB) regardless of blob size — a
5 GiB target dir no longer needs 5 GiB of mediator RAM.
* rpc/client.rs: new `prewarm_missing_chunks_between(upstream,
downstream, blob_id)` helper. Returns `(uploaded, total)` — the
difference is the dedup save. Retries once if the downstream
`PutManifest` reports missing chunks after our push (guards a
narrow eviction race); a second failure surfaces as `Err`.
* claw_cargo.rs: `prewarm` now streams by default; new `--buffered`
flag for the old whole-blob path (kept for diagnostic
comparability during rollout). Human-readable output shows the
mode + dedup count.
+2 tests:
- cold downstream: 3-chunk payload (with a partial tail chunk) is
copied exactly, reassembly is byte-equal to source
- partial dedup: pre-seed 1 of 3 chunks on downstream → uploaded=2;
rerun is a full no-op (uploaded=0), proving idempotence
251 tests pass (+2 from Phase 5j). Pre-existing macOS failure
unchanged.
Follow-ons: (1) parallel chunk transfer (uploaded chunks in fan-out)
would speed multi-GB prewarms further; (2) exposing an accurate
transferred-bytes counter needs router-side accounting instead of
the current chunk-count × CHUNK_SIZE approximation.
Adds a tiny axum-served HTTP endpoint that exposes the same
CacheMetrics counters that back GetMetrics + gossip, in Prometheus
text exposition format (v0.0.4). Enable per node by setting
`cluster.prom_bind` in the daemon config.
* metrics.rs: MetricsReply::to_prometheus() emits one HELP + TYPE +
sample line per counter. started_unix is a gauge; everything else is
a counter. Preallocates ~1 KiB so no reallocs mid-format.
* prom.rs (new): PromServer::bind spins up axum on a TcpListener,
serves GET /metrics, returns 404 elsewhere. Graceful shutdown via
oneshot channel; abort() variant for the sync-drop path in
ClusterServices. Snapshots on every scrape (no cache) — Relaxed
atomic loads are cheap enough that even 1 Hz is sub-microsecond.
* config.rs: new optional `cluster.prom_bind: SocketAddr` field.
Default None means no server; typical value is 127.0.0.1:7702.
* services.rs: wires PromServer into ClusterServices when both a
router and prom_bind exist. Warns (doesn't fail) if prom_bind is set
without RPC — nothing would ever change on a scrape.
+8 tests:
- metrics: to_prometheus emits every counter with correct type; zeros
still produce valid exposition (fresh daemon scrape)
- prom: content-type is text/plain; version=0.0.4; live updates
between requests (no cache); unknown paths 404; shutdown stops
serving
- services: end-to-end scrape returns router-driven counters;
prom_addr() is None when unconfigured (no accidentally-leaked port)
Raw-TCP HTTP client in tests instead of pulling in reqwest — 30 lines
of tokio::net + string split for GET / read-to-close is small enough
to justify not adding a dep.
249 tests pass (+8 from Phase 5i). Pre-existing macOS `du -sb` failure
unchanged.
ClusterServices now periodically snapshots the router's CacheMetrics
and republishes four raw counters — GetRef hits/misses and blob
GET/PUT byte totals — through chitchat as clawstor.cache.* keys.
Peers derive the hit rate locally via PeerView::cache_get_ref_hit_rate,
eliminating a per-peer GetMetrics roundtrip for placement decisions.
* gossip.rs: 4 new well-known keys, PeerView carries the counters +
a saturating-add derived hit rate that returns None on 0/0 or when
either counter is missing (guards against half-writes reading as
100% hits).
* services.rs: router hoisted out of the TLS branch so a
cache_metric_task can hold Arc<RpcRouter>. Publishes once at start
(initial zeros so peers don't wait 60s for first read) then every
CACHE_METRIC_INTERVAL. Ticker skipped when RPC isn't up — counters
only fire inside dispatch.
+3 tests:
- gossip: two-node convergence with cache counters + hit-rate math
- gossip: half-write / 0-0 / normal PeerView cases return correct rates
- services: fresh cluster sees Some(0) for all four keys, then after
in-process router counters + explicit set_cache_metrics the peer sees
the updated values with hit rate 0.8
241 tests pass (+3 from Phase 5g). Pre-existing macOS `du -sb` failure
in hot::tests unchanged.
Omar Sobh
osobh