Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 15s
README.md was still describing the pre-distributed ZFS-only
architecture (activate/deactivate cargo target dirs, SSH-based
tank→architect sync). Nothing about the distributed content-
addressed store, chunk-level dedup, QUIC/mTLS, snapshots, tags,
refs, FUSE mount, ref-tracking, Tailscale identity, or the
systemd timer set — i.e. everything shipped in Phases 1-8.
Full rewrite:
* What you get: 6 feature sections spanning storage/distribution,
fingerprint-keyed cargo cache, human primitives, operations,
roaming, reliability.
* One-paragraph architecture + link to ARCHITECTURE-v2.md.
* Fleet layout table (tank/architect/morpheus current state).
* Install: prereqs (Linux + macOS macFUSE), build, first-time CA
bootstrap, config template with tailnet bind, systemd install
recipe.
* Daily usage: claw-cargo build (env-driven for CI), pins,
snapshots, integrity+repair, fleet-status.sh, FUSE mount layout.
* CLI reference table (28 commands across claw-store + claw-cargo
+ claw-fuse).
* Prometheus metrics list.
* Ongoing operations: 4 timers table with sequencing rationale.
* Testing: 389 tests, known pre-existing macOS failure.
* Design-docs pointer list.
* Contributing + license note.
Also:
* ARCHITECTURE-v2.md phase plan table: replaced weeks-estimates
with ship-state (all ✅ shipped, dated).
* docs/runner-integration.md: added ref-tracking env-var section
so CI wiring populates the nightly sweep.
* dashboard/README.md: replaced the Vite template boilerplate
with actual dashboard context + note that it predates the
current architecture.
153 lines
5.2 KiB
Markdown
153 lines
5.2 KiB
Markdown
# Wiring a Gitea runner into clawstor
|
||
|
||
One-time per-runner setup, then workflows opt in with
|
||
|
||
```yaml
|
||
- uses: ./.gitea/actions/cargo-cache
|
||
```
|
||
|
||
## Prerequisites on the runner host
|
||
|
||
1. A running clawstor cluster daemon on the host. See
|
||
`systemd/clawstor-cluster.service`.
|
||
2. The runner uses `runs-on: ubuntu-latest:host` (or equivalent host
|
||
runner label). Container-only runners need to bind-mount the CLI
|
||
+ tls dir into the job container; that's a future extension.
|
||
|
||
## Provisioning claw-cargo for the runner user
|
||
|
||
Run these on the runner host as an operator with sudo. The runner
|
||
user (act_runner on stock Gitea Actions installs) needs the CLI on
|
||
`PATH`, a leaf cert signed by the fleet CA, and a
|
||
`~/.config/claw-cargo/config.toml` pointing at the local daemon.
|
||
|
||
```sh
|
||
# 1. Sign a leaf cert for this runner. `--node` becomes the SAN.
|
||
claw-store fleet-ca-sign \
|
||
--ca-dir /path/to/fleet-ca \
|
||
--node act-runner-$(hostname -s) \
|
||
--out-dir /tmp/act-runner-tls
|
||
|
||
# 2. Install the CLI + TLS material where act_runner can read them.
|
||
sudo install -m 755 /path/to/claw-cargo /usr/local/bin/claw-cargo
|
||
|
||
sudo install -d -o act_runner -g act_runner -m 700 \
|
||
/var/lib/act_runner/.config/claw-cargo \
|
||
/var/lib/act_runner/clawstor-tls
|
||
|
||
sudo install -o act_runner -g act_runner -m 644 \
|
||
/tmp/act-runner-tls/ca.crt /var/lib/act_runner/clawstor-tls/ca.crt
|
||
sudo install -o act_runner -g act_runner -m 644 \
|
||
/tmp/act-runner-tls/node.crt /var/lib/act_runner/clawstor-tls/node.crt
|
||
sudo install -o act_runner -g act_runner -m 600 \
|
||
/tmp/act-runner-tls/node.key /var/lib/act_runner/clawstor-tls/node.key
|
||
|
||
# 3. Write the runner's claw-cargo config. `peer.name` MUST match the
|
||
# daemon's cert SAN. `peer.addr` is where act_runner reaches the
|
||
# daemon — 127.0.0.1:7702 for host-mode runners.
|
||
sudo tee /var/lib/act_runner/.config/claw-cargo/config.toml >/dev/null <<'EOF'
|
||
[peer]
|
||
name = "tank" # <-- match this to your host
|
||
addr = "127.0.0.1:7702"
|
||
tls_dir = "/var/lib/act_runner/clawstor-tls"
|
||
|
||
[build]
|
||
profile = "dev"
|
||
EOF
|
||
sudo chown act_runner:act_runner /var/lib/act_runner/.config/claw-cargo/config.toml
|
||
sudo chmod 600 /var/lib/act_runner/.config/claw-cargo/config.toml
|
||
|
||
# 4. Smoke test as the runner user.
|
||
sudo -u act_runner claw-cargo peer-metrics \
|
||
--peer tank --peer-addr 127.0.0.1:7702 \
|
||
--tls-dir /var/lib/act_runner/clawstor-tls
|
||
```
|
||
|
||
## Runner label
|
||
|
||
Every provisioned runner gets a dedicated label so workflows only
|
||
land on hosts that actually have `claw-cargo` installed. Add to
|
||
`/etc/act_runner/config.yaml`:
|
||
|
||
```yaml
|
||
runner:
|
||
labels:
|
||
- "clawstor-cache:host" # add this line
|
||
- "ubuntu-latest:host" # keep any existing labels
|
||
```
|
||
|
||
Then `sudo systemctl restart act_runner.service` so the label
|
||
re-registers.
|
||
|
||
## Using the action in a workflow
|
||
|
||
Field finding: `ubuntu-latest` on Gitea Actions routes to
|
||
container mode by default (even with the `:host` suffix on the
|
||
runner label). Container mode hides `/usr/local/bin/claw-cargo` and
|
||
the per-runner tls_dir. Use the dedicated label instead:
|
||
|
||
```yaml
|
||
jobs:
|
||
build:
|
||
runs-on: clawstor-cache # dedicated host runner
|
||
steps:
|
||
- uses: actions/checkout@v4
|
||
- uses: ./.gitea/actions/cargo-cache
|
||
with:
|
||
workspace: .
|
||
# Read-only from forks so a malicious PR can't poison the cache.
|
||
no-upload: ${{ github.event_name == 'pull_request' && 'true' || 'false' }}
|
||
```
|
||
|
||
Outputs available on the step: `cache-outcome` (HIT | MISS |
|
||
POPULATED | SKIPPED), `fingerprint`, `elapsed-seconds`.
|
||
|
||
### Ref-tracking (Phase 7f)
|
||
|
||
For the nightly `cluster-ref-sweep` timer to identify stale cache
|
||
entries, `claw-cargo build` needs to record which `(repo, git-ref)`
|
||
produced each fingerprint. Set these in the workflow env:
|
||
|
||
```yaml
|
||
jobs:
|
||
build:
|
||
runs-on: clawstor-cache
|
||
env:
|
||
CLAWSTOR_REPO: ${{ gitea.repository }}
|
||
CLAWSTOR_GIT_REF: ${{ gitea.ref_name }}
|
||
CLAWSTOR_DATA_DIR: /var/lib/claw-store/data
|
||
steps:
|
||
- uses: actions/checkout@v4
|
||
- uses: ./.gitea/actions/cargo-cache
|
||
```
|
||
|
||
`claw-cargo` reads these via clap `env=` fallbacks — no flag
|
||
plumbing needed. If unset, the build still works, but the
|
||
fingerprint won't gain a ref-tracking annotation and won't be
|
||
eligible for automatic stale-cache reaping.
|
||
|
||
## What to expect
|
||
|
||
- **First push on main** → cache MISS → the runner rebuilds cold,
|
||
captures + uploads the target dir. Wall clock ≈ cold-build time.
|
||
- **Second push touching just workspace crates** → cache HIT → deps
|
||
restored from the daemon, cargo only rebuilds changed workspace
|
||
crates. Ratio measured on real workloads:
|
||
|
||
| Runner | Cold | Warm HIT | Speedup |
|
||
|---|---|---|---|
|
||
| Pi 5 (4 core, 8 GB, SSD) | 7m 54s | 2m 50s | **2.79× wall, 5m 4s saved** |
|
||
| Tank (48 core Threadripper) | 59s | 27s | 2.18× wall |
|
||
|
||
## Troubleshooting
|
||
|
||
- `claw-cargo not on PATH` → step 2 missed.
|
||
- `missing $HOME/.config/claw-cargo/config.toml` → step 3 missed.
|
||
- `Error: dialing 127.0.0.1:7702 for name tank` → daemon isn't running
|
||
or `bind_lan` still points at the fabric-only address. Change to
|
||
`0.0.0.0:7701` and restart.
|
||
- `rustc release drift: local_rustc=X peer_rustc=Y` → your
|
||
workspace's `rust-toolchain.toml` doesn't match the runner's
|
||
system rustc. The cache will still be valid but siloed per rustc.
|
||
Fix by pinning `rust-toolchain.toml`.
|