client_config.rs load_layered looks at ~/.claw-cargo/config.toml,
not ~/.config/claw-cargo/config.toml. Fix the workflow preflight
path to match. Both runners already have the file at both locations.
Fleet has 7 linux-amd64 runners; only tank + architect have
claw-cargo provisioned. Added a clawstor-cache:host label to those
two runners so this workflow only lands on them.
ubuntu-latest routes to container mode in act_runner even with the
:host suffix on the runner labels. Explicit self-hosted forces
host-mode where /usr/local/bin/claw-cargo + per-runner tls_dir are
visible.
Host runners have cmake/gcc/pkg-config from the OS and cargo/rustup
in the act_runner user's ~/.cargo/bin. apt-get needs root — the
runner isn't. Replace with a preflight that fails fast when any
tool is missing.
Ships the wire-up piece for real CI: a composite Gitea Action that
wraps `claw-cargo build` with cache-outcome reporting, plus a
matching workflow file that opts the clawstor repo itself into
being cache-hit-tested on every push. Also docs the one-time
per-runner provisioning (leaf cert, PATH install, config.toml).
* `.gitea/actions/cargo-cache/action.yml` — composite Action.
Inputs: workspace, profile, no-upload, parallel-restore. Outputs:
cache-outcome (HIT|MISS|POPULATED|SKIPPED), fingerprint,
elapsed-seconds. Runner-side config lives in
`~/.config/claw-cargo/config.toml` (not in the workflow — no
secrets shipped from repos).
* `.gitea/workflows/build-with-cache.yml` — dogfoods the action on
clawstor's own repo. `no-upload` set from event_name so PRs from
forks can't poison the cache.
* `docs/runner-integration.md` — one-time setup steps, sample
workflow snippet, expected numbers (Pi 5: 2.79× wall, tank:
2.18×), and troubleshooting for the failures I hit in the tank
and Pi pilots (bind_lan on fabric-only, missing CLI/config,
rustc drift warn).
Test protocol: push this branch → main triggers the workflow → the
runner on tank has claw-cargo + tls + config provisioned already
(2026-07-13 pilot setup) → first build should MISS + populate,
subsequent build on same fingerprint should HIT.
Pi 5 loopback measurement 2026-07-13:
--parallel-restore 1 : wall 2m52s, restore 20s
--parallel-restore 8 : wall 3m06s, restore 34s
Sequential is 70% faster on loopback. N-way stream contention costs
more than a single stream's congestion-control amortization. Same
shape as Phase 5k prewarm — fanout only wins when per-stream
throughput has a ceiling (WAN, tunneled links).
--parallel-restore N remains as opt-in.
Follow-up to PR #28. Fresh Pi deploy 2026-07-12 hit a `ProtectHome=
read-only` block on the daemon's XDG-driven
$HOME/.local/state/claw-store/projects.toml write. Adding the path
to `ReadWritePaths` in the shipped unit means future deployers
don't need a drop-in.
Two fixes surfaced by the vision-02 Pi 5 measurement:
## XDG default_path
`Manifest::default_path` was hardcoded to
`/var/lib/claw-store/projects.toml`. That path is read-only under
the user-mode systemd unit's `ProtectSystem=strict`, and creating
it needs root — awful for a runner install.
Precedence, matching XDG Base Directory:
1. `$XDG_STATE_HOME/claw-store/projects.toml`
2. `$HOME/.local/state/claw-store/projects.toml`
3. `/var/lib/claw-store/projects.toml` (system fallback)
User-mode installs now write in $HOME by default; system installs
(root, no HOME set) still land in /var/lib.
+1 test: `default_path_honours_xdg_state_home` — covers all three
precedence branches. Env mutation is process-global so the test
saves + restores.
## Parallel restore on cache HIT
Pi restore of 947 MiB via `BlobGetStream` took ~18s (~53 MiB/s)
single-stream. Per-stream throughput ceilings on the connection
type cap sequential fetches; parallel chunk fetches stack their
contributions.
- New `call_blob_get_parallel(conn, blob_id, concurrency) ->
Option<Vec<u8>>` in `rpc/client.rs`. `JoinSet` + `Semaphore`,
reassembles by chunk index at manifest-known offsets so
out-of-order arrival is fine.
- `claw-cargo build --parallel-restore N` (default 8). `N <= 1`
falls through to `BlobGetStream` for parity.
- Memory: `total_size + 4 MiB × in-flight` — dominated by the
reassembly buffer, not the fanout.
+1 test: `parallel_blob_get_reassembles_multi_chunk_blob_byte_equal`
covers roundtrip byte-equality vs BlobGetStream, tail-chunk offset,
concurrency=1 correctness, and NotFound → None.
259 tests pass (+2). Pre-existing macOS failure unchanged.
Orphan-chunk GC alone doesn't stop unbounded growth: as long as
fingerprint→blob refs keep getting PutRef'd, the manifest set keeps
growing and no chunk is ever an orphan.
* `BlobStore::evict_to_size_cap(max_bytes)` — walks manifests oldest
first by mtime, deletes them, refcount-decrements each chunk they
used, unlinks + reclaims size for any chunk whose refcount hits
zero. Shared chunks stay put until the last blob referencing them
is evicted.
* `ManifestSummary` internal type keeps the diff-set bookkeeping
cheap (one HashMap<ChunkHash, u32>, no repeated tree walks).
* `claw-store cluster-gc --evict-to-gb <N>` extends the CLI: still
runs the orphan sweep first, then optionally caps the store.
* Config: `cluster.blob_max_gb: Option<u64>`. The auto-GC ticker
runs eviction after every orphan sweep when this is set. Silent
when the store is already under cap; INFO log when it evicts.
+3 tests:
- evict_to_size_cap_reclaims_oldest_blobs_first: 3 blobs with
distinct mtimes, cap below combined size → oldest evicted,
newer blobs survive
- evict_keeps_shared_chunks_when_still_referenced: guards the
refcount decrement path (content-addressed dedup keeps identical
content as one blob → chunk survives until manifest deleted)
- evict_on_empty_store_is_a_noop: sanity
257 tests pass (baseline +3). Pre-existing macOS failure unchanged.
Both surfaced by the 2026-07-12 pilot as real operator concerns:
## rustc drift warning at build time
Runners silently silo their cache when rustc versions differ across
peers (fingerprint depends on rustc verbose output). The pilot's
first flow burned a full cold+upload before we realized the silo.
- `PeerStatusReply.local_rustc_release` — new field, populated from
the peer's own gossip `RUSTC_RELEASE` key via a new
`ClusterGossip::self_kv(key)` accessor.
- `claw-cargo build`: on cache MISS, calls `PeerStatus`; if the
peer's rustc release ≠ our local `rustc --version`, emits a WARN
with both versions + hint to add `rust-toolchain.toml`.
- Best-effort: absence of either release string is a shrug, not
an error.
## blob GC
Blob store grows unbounded on a runner; disk-full is a real
incident. `gc_orphan_chunks` already existed but wasn't exposed.
- New CLI: `claw-store cluster-gc` — runs `gc_orphan_chunks`,
prints report. Safe to run any time, safe to interrupt.
- New config: `cluster.gc_interval_hours: Option<u64>`. When set to
a positive integer, the daemon spawns a periodic ticker that
invokes GC in-process. Skips the first tick (nothing to reclaim
on boot). Errors are logged and retried next tick.
- Shutdown aborts the ticker cleanly.
254 tests pass (baseline unchanged). Pre-existing macOS failure
untouched.
Pilot 2026-07-12 measured 109 MiB/s on the sequential prewarm path —
~11% of a 10G fabric. `quinn::Connection` is cheap-Clone (internal
Arc), so we can run the has→get→put pipeline per chunk in concurrent
tasks under a bounded semaphore.
- `prewarm_missing_chunks_between_parallel(up, down, id, concurrency)`
in `rpc/client.rs`. `concurrency <= 1` degrades to the sequential
path (kept for diagnostic parity).
- `claw-cargo prewarm --parallel N` (default 8). Ignored with
`--buffered`. Memory ceiling: 4 MiB × in-flight = 32 MiB @ 8,
128 MiB @ 32.
- Uses `tokio::task::JoinSet` + `Arc<Semaphore>`; permit held for
the whole per-chunk pipeline so we never over-commit.
- Retry pass on `put_manifest` mismatch stays sequential — small,
correctness-critical.
- Errors: JoinSet drains completely + returns first task error so a
mid-fanout failure doesn't leave zombie tasks.
+1 test: `end_to_end_parallel_prewarm_copies_chunks_and_matches_sequential`
runs 5-chunk payload with concurrency=3, verifies byte-equal restore,
then reruns with concurrency=8 → 0 uploads (has_chunk dedup), then
concurrency=0 → 0 uploads (sequential fallback path).
254 tests pass (+1 from previous). Pre-existing macOS failure unchanged.
Pilot ran the cluster daemon under nohup; production needs proper
restart-on-failure + clean PATH inheritance (rustc gossip probe
needs `~/.cargo/bin` on PATH, which nohup's env didn't get).
The unit is user-scoped (`~/.config/systemd/user/`) so it works
without root on the pilot nodes:
cp systemd/clawstor-cluster.service ~/.config/systemd/user/
loginctl enable-linger $USER
systemctl --user daemon-reload
systemctl --user enable --now clawstor-cluster.service
Defaults:
- CLAWSTOR_BIN = ~/clawstor-deploy/claw-store
- CLAWSTOR_CONFIG = ~/clawstor-deploy/config.toml
- PATH prefixed with ~/.cargo/bin so rustc is found
Override any of those via `systemctl --user edit
clawstor-cluster.service`.
Security hardening:
- NoNewPrivileges=yes
- ProtectSystem=strict (system dirs read-only)
- ProtectHome=read-only (home dir read-only)
- ReadWritePaths=%h/clawstor-deploy (only the deploy tree is
writable)
- PrivateTmp=yes
Restart semantics:
- Restart=on-failure with RestartSec=10 — pilot-verified: kill -9
the daemon PID and the service comes back within ~10s
- TimeoutStopSec=60 so a slow gossip departure can complete
Deployed to tank + architect 2026-07-12 as part of the pilot
retest.
Field finding 2026-07-12 (clawverse measurement): the buffered
`capture_target -> Vec<u8>` path peaked at 2.8 GB RAM to capture a
6.1 GB target/debug into a 995 MiB compressed tar. Every byte
crossed RAM before touching the network.
* `capture_target_to_writer(target_dir, writer) -> u64` — new
streaming variant. Walks the tree + writes tar+zstd straight into
the caller's Writer via a small ByteCounter wrapper. Peak memory
stays at ~zstd sliding window size (few MB).
* `capture_target -> Vec<u8>` kept as a thin wrapper for the tests
+ smaller callers that don't care.
* `cmd_build`: capture into a tempfile under `target/`, then open
it with `tokio::fs::File` (AsyncRead + Unpin) and hand that to
`call_blob_put_stream`. Same-filesystem tempfile means no cross-
mount concerns; auto-unlinks on drop.
+1 test: `capture_streaming_matches_buffered_and_restores_correctly`
proves the streamed bytes match the buffered variant, the reported
byte count agrees with the written length, and roundtrip restore
from the streamed file works.
Combined with PR #22 (QUIC idle timeout), this closes the two RAM/
timeout blockers surfaced by the clawverse pilot. Expected memory
ceiling on a runner drops from GBs to MBs, unlocking small-runner
deployments (the actual pitch use case).
Field finding 2026-07-12 (clawverse cold on tank):
Compiling claw-cli v0.1.0 (...)
Finished `dev` profile ... in 45.08s
cargo build finished in 45.135491979s
Error: opening bidi stream for BlobPutStream
Caused by: timed out
Cargo took 45s → QUIC's 30s idle timeout killed the connection between
the initial peer-lookup connect and the follow-up capture+upload path.
The RPC never got a chance to open its stream.
Fix: two belt-and-braces changes:
1. IDLE_TIMEOUT 30s → 600s. The timeout is there to detect crashed
peers, not to enforce build pacing.
2. Client applies a `keep_alive_interval` of 15s so the connection
stays warm across cargo runs even shorter than the idle window.
quinn's keep-alive fires from an internal runtime task, not the app
thread, so a fully-CPU-pinned cargo build doesn't suppress it.
Bundles the profile→dir bug (PR #20 supersede) with four new fixes
discovered by running clawstor against itself + across the fabric:
* target_subdir_for: `dev`/`test` → `debug/`, `release`/`bench` →
`release/`, custom passes through. Was silently skipping upload.
* rustc release via gossip: daemon probes `rustc --version` at start,
publishes the release string as `clawstor.rustc.release`. PeerView
carries it; `cluster-peer-status` prints it in a new column and
emits a warning line when the fleet has mixed versions. Would have
surfaced the tank/architect 1.96.1 vs 1.95.0 drift instantly.
* prewarm publishes fingerprint→blob ref downstream: `pin` now writes
a companion tag `<name>.fingerprint` holding the fingerprint bytes.
`prewarm` reads the companion, PutTag's it downstream, then
PutRef(fp→blob) so a subsequent fingerprint-based `build` HITS.
Without this, prewarm was almost useless for the runner path
(build always missed even with matching source + rustc).
* streaming byte counters: BlobPutStream + BlobGetStream now record
the transferred bytes via `record_blob_{put,get}_bytes`. Metric
used to stay at 0 no matter how much you moved.
* capture determinism: replaced `tar::Builder::append_dir_all` (uses
`read_dir`'s native order) with `append_dir_sorted` that walks the
tree recursively and sorts by filename bytes at every level. Two
byte-identical trees now produce byte-identical tars regardless of
filesystem ordering.
+3 tests:
- target_subdir_matches_cargo_layout (from #20)
- fingerprint_companion_tag_uses_dotted_suffix
- capture_is_order_independent_of_filesystem_readdir (guard against
the exact bug we saw in the field)
252 tests pass (+1 from Phase 5h's 251). Pre-existing macOS `du -sb`
failure unchanged.
Supersedes #20 (also included here). Ready for re-deploy to
tank + architect for the retest run.
Bounded-memory cross-peer prewarm: instead of buffering the whole
blob in RAM (previous 5f path), iterate the upstream manifest chunk
by chunk, ask downstream `HasChunk`, stream missing chunks one at a
time. Memory ceiling is 1 chunk (4 MiB) regardless of blob size — a
5 GiB target dir no longer needs 5 GiB of mediator RAM.
* rpc/client.rs: new `prewarm_missing_chunks_between(upstream,
downstream, blob_id)` helper. Returns `(uploaded, total)` — the
difference is the dedup save. Retries once if the downstream
`PutManifest` reports missing chunks after our push (guards a
narrow eviction race); a second failure surfaces as `Err`.
* claw_cargo.rs: `prewarm` now streams by default; new `--buffered`
flag for the old whole-blob path (kept for diagnostic
comparability during rollout). Human-readable output shows the
mode + dedup count.
+2 tests:
- cold downstream: 3-chunk payload (with a partial tail chunk) is
copied exactly, reassembly is byte-equal to source
- partial dedup: pre-seed 1 of 3 chunks on downstream → uploaded=2;
rerun is a full no-op (uploaded=0), proving idempotence
251 tests pass (+2 from Phase 5j). Pre-existing macOS failure
unchanged.
Follow-ons: (1) parallel chunk transfer (uploaded chunks in fan-out)
would speed multi-GB prewarms further; (2) exposing an accurate
transferred-bytes counter needs router-side accounting instead of
the current chunk-count × CHUNK_SIZE approximation.
Adds a tiny axum-served HTTP endpoint that exposes the same
CacheMetrics counters that back GetMetrics + gossip, in Prometheus
text exposition format (v0.0.4). Enable per node by setting
`cluster.prom_bind` in the daemon config.
* metrics.rs: MetricsReply::to_prometheus() emits one HELP + TYPE +
sample line per counter. started_unix is a gauge; everything else is
a counter. Preallocates ~1 KiB so no reallocs mid-format.
* prom.rs (new): PromServer::bind spins up axum on a TcpListener,
serves GET /metrics, returns 404 elsewhere. Graceful shutdown via
oneshot channel; abort() variant for the sync-drop path in
ClusterServices. Snapshots on every scrape (no cache) — Relaxed
atomic loads are cheap enough that even 1 Hz is sub-microsecond.
* config.rs: new optional `cluster.prom_bind: SocketAddr` field.
Default None means no server; typical value is 127.0.0.1:7702.
* services.rs: wires PromServer into ClusterServices when both a
router and prom_bind exist. Warns (doesn't fail) if prom_bind is set
without RPC — nothing would ever change on a scrape.
+8 tests:
- metrics: to_prometheus emits every counter with correct type; zeros
still produce valid exposition (fresh daemon scrape)
- prom: content-type is text/plain; version=0.0.4; live updates
between requests (no cache); unknown paths 404; shutdown stops
serving
- services: end-to-end scrape returns router-driven counters;
prom_addr() is None when unconfigured (no accidentally-leaked port)
Raw-TCP HTTP client in tests instead of pulling in reqwest — 30 lines
of tokio::net + string split for GET / read-to-close is small enough
to justify not adding a dep.
249 tests pass (+8 from Phase 5i). Pre-existing macOS `du -sb` failure
unchanged.
ClusterServices now periodically snapshots the router's CacheMetrics
and republishes four raw counters — GetRef hits/misses and blob
GET/PUT byte totals — through chitchat as clawstor.cache.* keys.
Peers derive the hit rate locally via PeerView::cache_get_ref_hit_rate,
eliminating a per-peer GetMetrics roundtrip for placement decisions.
* gossip.rs: 4 new well-known keys, PeerView carries the counters +
a saturating-add derived hit rate that returns None on 0/0 or when
either counter is missing (guards against half-writes reading as
100% hits).
* services.rs: router hoisted out of the TLS branch so a
cache_metric_task can hold Arc<RpcRouter>. Publishes once at start
(initial zeros so peers don't wait 60s for first read) then every
CACHE_METRIC_INTERVAL. Ticker skipped when RPC isn't up — counters
only fire inside dispatch.
+3 tests:
- gossip: two-node convergence with cache counters + hit-rate math
- gossip: half-write / 0-0 / normal PeerView cases return correct rates
- services: fresh cluster sees Some(0) for all four keys, then after
in-process router counters + explicit set_cache_metrics the peer sees
the updated values with hit rate 0.8
241 tests pass (+3 from Phase 5g). Pre-existing macOS `du -sb` failure
in hot::tests unchanged.
Every RPC handler that answers a hit-or-miss question now increments
lock-free atomic counters. The GetMetrics RPC (0x13) returns a JSON
snapshot of every counter; the new claw-cargo peer-metrics CLI
prints hit rates, byte volumes, and counter uptime.
Placement engines can now poll these across the fleet to bias runner
scheduling toward whichever node has the warmest cache for a given
repo/tag combination.
## New module: cluster/metrics.rs (268 lines)
Types:
- CacheMetrics — atomic counters, all AtomicU64, Relaxed ordering
(metrics are advisory, not consistency-critical)
- MetricsReply — JSON snapshot returned by GetMetrics
Public API:
- CacheMetrics::new() — timestamped start, all counters at 0
- record_get_ref_hit / _miss
- record_get_tag_hit / _miss
- record_blob_get_bytes / record_blob_put_bytes
- record_get_chunk_hit / _miss
- record_has_chunk_hit / _miss
- snapshot() — atomic-load every field into a MetricsReply
MetricsReply derived helpers:
- get_ref_hit_rate() / get_tag_hit_rate() / has_chunk_hit_rate() —
Option<f64> so 0/0 returns None instead of NaN
## RPC method
- GetMetrics (0x13): payload = empty; reply = JSON MetricsReply
Wire-level instrumentation added to RpcRouter dispatch:
- GetRef → record_get_ref_hit / _miss
- GetTag → record_get_tag_hit / _miss
- BlobGet → record_blob_get_bytes (on hit)
- BlobPut → record_blob_put_bytes
- HasChunk → record_has_chunk_hit / _miss
- GetChunk → record_get_chunk_hit + record_blob_get_bytes on hit
/ record_get_chunk_miss
RpcRouter grows Arc<CacheMetrics> unconditionally — every router has
metrics, so a fresh node with no traffic still returns a valid
snapshot with all zeros + started_unix.
Streaming variants (BlobPutStream / BlobGetStream) don't yet track
byte counts — they'd require plumbing the count out of put_stream /
stream_to. Follow-on if it turns out to matter for placement.
## Client helper + CLI
- call_get_metrics(&conn) → Result<MetricsReply>
- claw-cargo peer-metrics [--peer ...] [--peer-addr ...] [--tls-dir ...]
Fetches + pretty-prints:
counter uptime: 42s
GetRef hits/misses: 123 / 45
hit rate: 73.21%
GetTag hits/misses: 8 / 2
hit rate: 80.00%
HasChunk hits/miss: 512 / 88
hit rate: 85.33%
GetChunk hits/miss: 47 / 12
Blob GET bytes: 1.23 GiB
Blob PUT bytes: 3.45 GiB
human_bytes() helper picks GiB / MiB / KiB / B based on magnitude.
Subcommand count now 9: build / prefetch / status / fingerprint /
pin / unpin / list-tags / prewarm / peer-metrics.
## Tests (13 new, all real filesystem / real QUIC — no mocks)
CacheMetrics (6):
- new_starts_all_counters_at_zero_except_timestamp
- recorders_increment_the_right_field (every recorder × 1-2 counts)
- hit_rates_none_when_zero_events (avoids 0/0 NaN)
- hit_rates_compute_correctly (3 hits / 1 miss → 75%)
- snapshot_round_trips_through_json
- snapshots_across_threads_are_consistent_up_to_relaxed_ordering
(8 threads × 1000 increments → exactly 8000)
RPC integration (7):
- phase_5g_method_byte_encoding
- get_metrics_returns_empty_snapshot_before_any_activity
- get_ref_records_hit_and_miss_counters (2 hits + 1 miss)
- get_tag_records_hit_and_miss_counters
- blob_get_and_blob_put_record_byte_counts
- has_chunk_and_get_chunk_record_hit_miss_counters
- **end_to_end_get_metrics_over_real_quic** — seed activity locally,
fire GetRef/GetTag/BlobGet dispatches to move the counters, then
fetch metrics through real QUIC + mTLS and verify each field
including the 0.5 hit rate calculation
238 tests pass. Pre-existing macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/metrics.rs: 268
- cluster/rpc.rs: 818
- cluster/rpc/tests_phase5.rs: 905
- claw_cargo.rs: 894
## What this enables
Fleet-wide visibility into which peer is actually serving traffic:
# From anywhere with connectivity + fleet mTLS
claw-cargo peer-metrics --peer tank
claw-cargo peer-metrics --peer architect
claw-cargo peer-metrics --peer morpheus
Compare hit rates side by side to see which node's cache is warmest.
A placement engine can automate this — poll every 30s, feed the
scheduler.
## Follow-on
- 5h: streaming variant of prewarm (fixed-memory ceiling for many-GB
blobs)
- 5i: metrics also published via gossip so PeerView carries hit rate
without a per-peer GetMetrics roundtrip
- 5j: prometheus /metrics endpoint on the daemon for existing dash
integrations
- 6: FUSE mount for warm-tier git worktrees
Omar Sobh
osobh