Wraps quinn 0.11 in cluster/transport.rs with rustls 0.23 + rcgen 0.13
for identity management. Every peer connection is mTLS: both sides must
present certs signed by the shared fleet CA, and rustls verifies the
peer's cert SAN matches the requested server name.
Public API on `cluster::transport`:
- `NodeIdentity` (cert chain + private key + trusted CA)
- `NodeIdentity::generate_test_pair(a, b)` — ephemeral CA + two signed
leaves; shape matches the production PEM-file loader (Phase 1d)
- `QuicServer::bind(addr, identity)` — bind mTLS-enforced listener
- `QuicServer::accept()` — accept one connection (Option<Result<_>>)
- `QuicClient::new(local_addr, identity)` — build client endpoint
- `QuicClient::connect(peer_addr, expected_name)` — outbound with SAN check
- `ping(&conn, payload)` — bidi stream RPC; server echoes as `pong:<payload>`
- `ping_handler_loop(conn)` — server-side accept/echo forever
- `CLAWSTOR_RPC_ALPN` constant, single ALPN "clawstor-rpc/1"
Config extension:
- `bind_rpc_lan`, `bind_rpc_tailscale` on `ClusterConfig` (both optional).
- Defaults: gossip port + 1 (so gossip UDP and QUIC UDP don't collide).
- Helper: `ClusterConfig::rpc_lan()`, `rpc_tailscale()`,
`PeerEntry::rpc_lan()`, `rpc_tailscale()`.
Gossip.rs now advertises the RPC address, not the gossip address, on
the well-known `clawstor.rpc.lan` / `clawstor.rpc.tailscale` keys.
CLI:
- `cluster-ping --name <me> --peer <you> --rpc-addr <addr> [--payload X]`
Runs a full mTLS handshake and single ping. For dev/loopback use
today (both sides need to share a CA); persistent-identity ping
lands in Phase 1d.
Tests (6 new, real UDP + TLS handshake, no mocks):
- generate_test_pair produces two distinct leaves that share a CA
- ping_pong_between_two_mtls_peers: real 2-node QUIC round trip with
full mTLS chain verification, `open_bi()`/`accept_bi()`, byte-exact
response check
- client_rejects_peer_with_wrong_ca: TLS chain verification failure
when the server presents a cert signed by a different CA
- client_rejects_wrong_server_name: SAN mismatch is enforced
- server_binds_wildcard_and_reports_concrete_local_addr: port 0 → real
- ping_rejects_oversize_payload: MAX_MESSAGE_BYTES cap enforced client-side
Fixed-port allocator range 42000+ so transport tests don't conflict
with gossip tests (41000+).
72 tests pass. Pre-existing `hot::test_project_target_size_bytes`
macOS-only failure unchanged.
File sizes (all under 1300-line ceiling):
- cluster/transport.rs: 485
- cluster/gossip.rs: 570
- cluster.rs: 275
- config.rs: 480
- main.rs: 564
Follow-on Phase 1 cut (1d):
- Load NodeIdentity from persistent PEM files at /etc/claw-store/tls/
- Fleet CA bootstrap ceremony (rcgen → write CA cert; per-node leaf CSR)
- Daemon-level RPC server that runs alongside gossip + serves real
operations (blob get/put, metadata sync)
- cluster-status reads from live daemon via API instead of standalone
Wraps chitchat 0.11 in cluster/gossip.rs. Each node publishes its
zone, RPC endpoints, hot-tier usage, warm-tier project list, and
uptime under well-known kv keys; peers propagate this via gossip and
are classified alive/dead by phi-accrual failure detection.
Bootstraps from static [[cluster.peers]] seed nodes in the config.
Membership extends dynamically as nodes join or leave.
Public API on `ClusterGossip`:
- bootstrap(cluster_cfg, local_name) — start UDP gossip service
- set(key, value), set_hot_used, set_hot_max, set_warm_projects
- peers() — all known peers with typed PeerView + liveness
- peer(name) — one peer by advertised name
- peers_in_zone(zone) — live peers filtered by zone
- shutdown() — abort gossip task
CLI:
- `claw-store cluster-status --name <me> [--wait-secs N]`
bootstraps gossip, waits for convergence, prints a peer table
with zone, alive/dead, RPC LAN/Tailscale addresses, hot usage.
Tests (7 new):
- bootstrap_publishes_our_own_state — self-ID + advertised address
- bootstrap_fails_when_local_name_empty
- bootstrap_fails_when_no_bind_address — cluster.validate error chain
- two_node_cluster_converges_and_shares_state — REAL two-node
in-process UDP gossip. Publish state on A, wait for phi-accrual
liveness on B, verify all fields propagated (zone, rpc_lan, hot
bytes, warm_projects list, fill ratio, zone filtering).
- peers_excludes_self — solo cluster reports zero peers
- peer_view_hot_fill_ratio_handles_missing_or_zero
No mocks. Real UDP transport, real chitchat runtime, real gossip
protocol. Ports allocated from a fixed range (41001+) via atomic
counter — deterministic and race-free within the test binary.
66 tests pass. Pre-existing `hot::test_project_target_size_bytes`
macOS-only failure unchanged.
File sizes (under 1300-line ceiling):
- cluster/gossip.rs: 549
- cluster.rs: 274
- main.rs: 506
Follow-on Phase 1 cuts:
- 1c: `quinn` QUIC transport + fleet-CA mTLS for RPC (separate UDP
port so gossip and QUIC don't collide)
- 1d: daemon wire-up — gossip runs in background, hot-tier usage
pushed periodically, `cluster-status` reads from live daemon
- Extract build_app() from run_server() so tests can construct the
router without binding a port (tower::ServiceExt::oneshot pattern)
- Add 10 new tests in serve::tests: validate_project_name (valid +
invalid slugs), daemon_uptime when file absent, GET /api/status,
GET /api/projects, POST /api/activate with invalid name, auth
middleware (no header → 401, wrong token → 401, correct token →
pass-through, GET bypasses auth entirely)
- Add tower + http-body-util to dev-dependencies
- Add project README covering architecture, install (Makefile),
config options, SSH setup, snapshot schedule, CLI reference,
HTTP API table, pinning, and troubleshooting guide
- Bump version to 0.3.0
Test suite: 39 tests, 0 failures
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
The two biggest pain points coming out of the architecture review:
1. The manifest at /var/lib/claw-store/projects.toml was the only
piece of writable state but had no locking, no atomic writes,
and three concurrent writers (daemon poll tick, every CLI verb,
and the dashboard shelling out via /api/activate). Two writers
interleaving silently dropped one of them; a crash mid-write
left a corrupt half-written TOML that the next reader parsed
as an empty manifest.
2. Reboot survival: the dashboard had no systemd unit and was a
stray hand-launched process. Architect lost its dashboard on
todays reboot.
This commit lands:
- Manifest::update(path, FnOnce(&mut Manifest)) — locked-atomic
load-mutate-save in one transaction. Uses libc::flock(LOCK_EX) on
a sidecar .lock file (so the data file can be replaced by rename
without invalidating the lock) and tempfile + persist for the
rename. Concurrent writers serialise; readers see the previous
state or the new state, never a torn write. Manifest::load uses
LOCK_SH so it never races a mid-rename.
- Project.pinned: bool with #[serde(default)] so legacy manifests
parse cleanly. hot::gc_stale_targets and hot::gc_by_space both
skip pinned projects, with a WARN log when every remaining
project is pinned but were still over budget — operator intent
beats space pressure.
- claw-store pin <project> / unpin <project> CLI verbs.
status command surfaces pin marker (📌).
activate preserves an existing rows pinned flag so re-activating
doesnt silently unpin.
- claw-store-serve.service systemd unit. Type=simple, Restart=
on-failure, RestartSec=15, ProtectSystem=strict + ReadWritePaths
=/var/lib/claw-store, ProtectHome, NoNewPrivileges, PrivateTmp.
- daemon poll tick reloads the manifest from disk at the start of
each cycle (so CLI activations between ticks are visible) and
routes its GC write through Manifest::update (so it cant race a
concurrent CLI pin).
- libc + tempfile move from dev-deps into runtime deps.
- empty-manifest fallthrough on load (treat "" as default) so a
half-written tempfile crashed pre-rename doesnt hard-fail the
daemon next boot.
- 25 tests passing incl. new ones: legacy-toml-parses, update-
serializes-two-sequential-writers, pinned-survives-stale-gc,
pinned-survives-space-gc-even-when-lru.
Version bumped 0.1.0 → 0.2.0.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Adds the serve subcommand: an axum 0.7 + tower-http server on
:7700 backing the React dashboard. Routes:
GET /api/status /api/projects /api/snapshots
/api/sync-queue /api/hot /api/events (SSE 5s tick)
POST /api/activate /api/deactivate /api/sync
/api/gc /api/snapshot
Mutating endpoints shell out to /usr/local/bin/claw-store so
all CLI logic stays single-sourced. claw-store/static/ holds
the prebuilt dashboard Vite bundle for --static-dir.
cargo_init.rs gets a minor wiring tweak so the API can read
back the managed cargo config marker.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Omar Sobh