Files
clawstor/docs/runner-integration.md
Omar Sobh 24ff3ec130
Build with clawstor cache / Cargo build (clawstor-cached) (pull_request) Successful in 15s
Polish: comprehensive docs overhaul
README.md was still describing the pre-distributed ZFS-only
architecture (activate/deactivate cargo target dirs, SSH-based
tank→architect sync). Nothing about the distributed content-
addressed store, chunk-level dedup, QUIC/mTLS, snapshots, tags,
refs, FUSE mount, ref-tracking, Tailscale identity, or the
systemd timer set — i.e. everything shipped in Phases 1-8.

Full rewrite:
* What you get: 6 feature sections spanning storage/distribution,
  fingerprint-keyed cargo cache, human primitives, operations,
  roaming, reliability.
* One-paragraph architecture + link to ARCHITECTURE-v2.md.
* Fleet layout table (tank/architect/morpheus current state).
* Install: prereqs (Linux + macOS macFUSE), build, first-time CA
  bootstrap, config template with tailnet bind, systemd install
  recipe.
* Daily usage: claw-cargo build (env-driven for CI), pins,
  snapshots, integrity+repair, fleet-status.sh, FUSE mount layout.
* CLI reference table (28 commands across claw-store + claw-cargo
  + claw-fuse).
* Prometheus metrics list.
* Ongoing operations: 4 timers table with sequencing rationale.
* Testing: 389 tests, known pre-existing macOS failure.
* Design-docs pointer list.
* Contributing + license note.

Also:
* ARCHITECTURE-v2.md phase plan table: replaced weeks-estimates
  with ship-state (all ✅ shipped, dated).
* docs/runner-integration.md: added ref-tracking env-var section
  so CI wiring populates the nightly sweep.
* dashboard/README.md: replaced the Vite template boilerplate
  with actual dashboard context + note that it predates the
  current architecture.
2026-07-14 15:16:50 -07:00

153 lines
5.2 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Wiring a Gitea runner into clawstor
One-time per-runner setup, then workflows opt in with
```yaml
- uses: ./.gitea/actions/cargo-cache
```
## Prerequisites on the runner host
1. A running clawstor cluster daemon on the host. See
`systemd/clawstor-cluster.service`.
2. The runner uses `runs-on: ubuntu-latest:host` (or equivalent host
runner label). Container-only runners need to bind-mount the CLI
+ tls dir into the job container; that's a future extension.
## Provisioning claw-cargo for the runner user
Run these on the runner host as an operator with sudo. The runner
user (act_runner on stock Gitea Actions installs) needs the CLI on
`PATH`, a leaf cert signed by the fleet CA, and a
`~/.config/claw-cargo/config.toml` pointing at the local daemon.
```sh
# 1. Sign a leaf cert for this runner. `--node` becomes the SAN.
claw-store fleet-ca-sign \
--ca-dir /path/to/fleet-ca \
--node act-runner-$(hostname -s) \
--out-dir /tmp/act-runner-tls
# 2. Install the CLI + TLS material where act_runner can read them.
sudo install -m 755 /path/to/claw-cargo /usr/local/bin/claw-cargo
sudo install -d -o act_runner -g act_runner -m 700 \
/var/lib/act_runner/.config/claw-cargo \
/var/lib/act_runner/clawstor-tls
sudo install -o act_runner -g act_runner -m 644 \
/tmp/act-runner-tls/ca.crt /var/lib/act_runner/clawstor-tls/ca.crt
sudo install -o act_runner -g act_runner -m 644 \
/tmp/act-runner-tls/node.crt /var/lib/act_runner/clawstor-tls/node.crt
sudo install -o act_runner -g act_runner -m 600 \
/tmp/act-runner-tls/node.key /var/lib/act_runner/clawstor-tls/node.key
# 3. Write the runner's claw-cargo config. `peer.name` MUST match the
# daemon's cert SAN. `peer.addr` is where act_runner reaches the
# daemon — 127.0.0.1:7702 for host-mode runners.
sudo tee /var/lib/act_runner/.config/claw-cargo/config.toml >/dev/null <<'EOF'
[peer]
name = "tank" # <-- match this to your host
addr = "127.0.0.1:7702"
tls_dir = "/var/lib/act_runner/clawstor-tls"
[build]
profile = "dev"
EOF
sudo chown act_runner:act_runner /var/lib/act_runner/.config/claw-cargo/config.toml
sudo chmod 600 /var/lib/act_runner/.config/claw-cargo/config.toml
# 4. Smoke test as the runner user.
sudo -u act_runner claw-cargo peer-metrics \
--peer tank --peer-addr 127.0.0.1:7702 \
--tls-dir /var/lib/act_runner/clawstor-tls
```
## Runner label
Every provisioned runner gets a dedicated label so workflows only
land on hosts that actually have `claw-cargo` installed. Add to
`/etc/act_runner/config.yaml`:
```yaml
runner:
labels:
- "clawstor-cache:host" # add this line
- "ubuntu-latest:host" # keep any existing labels
```
Then `sudo systemctl restart act_runner.service` so the label
re-registers.
## Using the action in a workflow
Field finding: `ubuntu-latest` on Gitea Actions routes to
container mode by default (even with the `:host` suffix on the
runner label). Container mode hides `/usr/local/bin/claw-cargo` and
the per-runner tls_dir. Use the dedicated label instead:
```yaml
jobs:
build:
runs-on: clawstor-cache # dedicated host runner
steps:
- uses: actions/checkout@v4
- uses: ./.gitea/actions/cargo-cache
with:
workspace: .
# Read-only from forks so a malicious PR can't poison the cache.
no-upload: ${{ github.event_name == 'pull_request' && 'true' || 'false' }}
```
Outputs available on the step: `cache-outcome` (HIT | MISS |
POPULATED | SKIPPED), `fingerprint`, `elapsed-seconds`.
### Ref-tracking (Phase 7f)
For the nightly `cluster-ref-sweep` timer to identify stale cache
entries, `claw-cargo build` needs to record which `(repo, git-ref)`
produced each fingerprint. Set these in the workflow env:
```yaml
jobs:
build:
runs-on: clawstor-cache
env:
CLAWSTOR_REPO: ${{ gitea.repository }}
CLAWSTOR_GIT_REF: ${{ gitea.ref_name }}
CLAWSTOR_DATA_DIR: /var/lib/claw-store/data
steps:
- uses: actions/checkout@v4
- uses: ./.gitea/actions/cargo-cache
```
`claw-cargo` reads these via clap `env=` fallbacks — no flag
plumbing needed. If unset, the build still works, but the
fingerprint won't gain a ref-tracking annotation and won't be
eligible for automatic stale-cache reaping.
## What to expect
- **First push on main** → cache MISS → the runner rebuilds cold,
captures + uploads the target dir. Wall clock ≈ cold-build time.
- **Second push touching just workspace crates** → cache HIT → deps
restored from the daemon, cargo only rebuilds changed workspace
crates. Ratio measured on real workloads:
| Runner | Cold | Warm HIT | Speedup |
|---|---|---|---|
| Pi 5 (4 core, 8 GB, SSD) | 7m 54s | 2m 50s | **2.79× wall, 5m 4s saved** |
| Tank (48 core Threadripper) | 59s | 27s | 2.18× wall |
## Troubleshooting
- `claw-cargo not on PATH` → step 2 missed.
- `missing $HOME/.config/claw-cargo/config.toml` → step 3 missed.
- `Error: dialing 127.0.0.1:7702 for name tank` → daemon isn't running
or `bind_lan` still points at the fabric-only address. Change to
`0.0.0.0:7701` and restart.
- `rustc release drift: local_rustc=X peer_rustc=Y` → your
workspace's `rust-toolchain.toml` doesn't match the runner's
system rustc. The cache will still be valid but siloed per rustc.
Fix by pinning `rust-toolchain.toml`.