Phase 1c: QUIC transport with fleet-CA mTLS + cluster-ping

Wraps quinn 0.11 in cluster/transport.rs with rustls 0.23 + rcgen 0.13
for identity management. Every peer connection is mTLS: both sides must
present certs signed by the shared fleet CA, and rustls verifies the
peer's cert SAN matches the requested server name.

Public API on `cluster::transport`:
- `NodeIdentity` (cert chain + private key + trusted CA)
- `NodeIdentity::generate_test_pair(a, b)` — ephemeral CA + two signed
  leaves; shape matches the production PEM-file loader (Phase 1d)
- `QuicServer::bind(addr, identity)` — bind mTLS-enforced listener
- `QuicServer::accept()` — accept one connection (Option<Result<_>>)
- `QuicClient::new(local_addr, identity)` — build client endpoint
- `QuicClient::connect(peer_addr, expected_name)` — outbound with SAN check
- `ping(&conn, payload)` — bidi stream RPC; server echoes as `pong:<payload>`
- `ping_handler_loop(conn)` — server-side accept/echo forever
- `CLAWSTOR_RPC_ALPN` constant, single ALPN "clawstor-rpc/1"

Config extension:
- `bind_rpc_lan`, `bind_rpc_tailscale` on `ClusterConfig` (both optional).
- Defaults: gossip port + 1 (so gossip UDP and QUIC UDP don't collide).
- Helper: `ClusterConfig::rpc_lan()`, `rpc_tailscale()`,
  `PeerEntry::rpc_lan()`, `rpc_tailscale()`.

Gossip.rs now advertises the RPC address, not the gossip address, on
the well-known `clawstor.rpc.lan` / `clawstor.rpc.tailscale` keys.

CLI:
- `cluster-ping --name <me> --peer <you> --rpc-addr <addr> [--payload X]`
  Runs a full mTLS handshake and single ping. For dev/loopback use
  today (both sides need to share a CA); persistent-identity ping
  lands in Phase 1d.

Tests (6 new, real UDP + TLS handshake, no mocks):
- generate_test_pair produces two distinct leaves that share a CA
- ping_pong_between_two_mtls_peers: real 2-node QUIC round trip with
  full mTLS chain verification, `open_bi()`/`accept_bi()`, byte-exact
  response check
- client_rejects_peer_with_wrong_ca: TLS chain verification failure
  when the server presents a cert signed by a different CA
- client_rejects_wrong_server_name: SAN mismatch is enforced
- server_binds_wildcard_and_reports_concrete_local_addr: port 0 → real
- ping_rejects_oversize_payload: MAX_MESSAGE_BYTES cap enforced client-side

Fixed-port allocator range 42000+ so transport tests don't conflict
with gossip tests (41000+).

72 tests pass. Pre-existing `hot::test_project_target_size_bytes`
macOS-only failure unchanged.

File sizes (all under 1300-line ceiling):
- cluster/transport.rs: 485
- cluster/gossip.rs: 570
- cluster.rs: 275
- config.rs: 480
- main.rs: 564

Follow-on Phase 1 cut (1d):
- Load NodeIdentity from persistent PEM files at /etc/claw-store/tls/
- Fleet CA bootstrap ceremony (rcgen → write CA cert; per-node leaf CSR)
- Daemon-level RPC server that runs alongside gossip + serves real
  operations (blob get/put, metadata sync)
- cluster-status reads from live daemon via API instead of standalone
This commit is contained in:
Omar Sobh
2026-07-11 21:50:28 -07:00
parent 5283ab8655
commit 4d93f955c5
7 changed files with 947 additions and 22 deletions
+485
View File
@@ -0,0 +1,485 @@
//! QUIC transport for peer RPC with fleet-CA mTLS.
//!
//! Runs on a UDP port distinct from chitchat gossip (see
//! [`ClusterConfig::rpc_lan`](crate::config::ClusterConfig::rpc_lan) —
//! defaults to gossip port + 1) so the two UDP-based protocols don't
//! collide.
//!
//! # Trust model
//!
//! A single fleet root CA signs every node's leaf certificate. Each
//! node loads its own cert + private key + the CA public cert into a
//! [`NodeIdentity`]. Both server and client verifiers require peer
//! certs to chain to the fleet CA — that's mTLS. A node without a
//! signed cert cannot join RPC.
//!
//! # Test model
//!
//! The [`NodeIdentity::generate_test_pair`] helper cuts an ephemeral
//! CA + two leaf certs in-memory for two named nodes. Both nodes share
//! the CA (so they trust each other) but have distinct leaves. This is
//! the same code path used in production; there is no test-only side
//! door in the transport itself.
//!
//! # ALPN
//!
//! The single protocol identifier `clawstor-rpc/1` is offered on every
//! connection; peers advertising anything else are rejected during the
//! TLS handshake.
use anyhow::{bail, Context, Result};
use quinn::{ClientConfig, Endpoint, ServerConfig, VarInt};
use rustls::pki_types::{CertificateDer, PrivateKeyDer};
use std::net::SocketAddr;
use std::sync::Arc;
use std::time::Duration;
/// ALPN identifier all fleet RPC endpoints advertise + require.
pub const CLAWSTOR_RPC_ALPN: &[u8] = b"clawstor-rpc/1";
/// Idle timeout on connection — if no data for this long the connection
/// dies. Short enough to notice partitions, long enough to survive a
/// paused laptop.
const IDLE_TIMEOUT: Duration = Duration::from_secs(30);
/// Cap on any single RPC message payload. Ping/pong is tiny; other RPCs
/// stream larger payloads via streams-of-many-messages. Prevents an
/// adversary from allocating unbounded memory on a single message.
const MAX_MESSAGE_BYTES: usize = 16 * 1024;
/// Cryptographic identity a node presents on both sides of the RPC
/// endpoint. Fully specifies the mTLS setup: our own cert chain + key,
/// plus the CA cert we require the peer's cert to chain to.
///
/// [`PrivateKeyDer`] is not `Clone`, so `NodeIdentity` isn't derive-Clone
/// either. Use [`NodeIdentity::clone_id`] when a second owner is required.
#[derive(Debug)]
pub struct NodeIdentity {
/// This node's certificate chain (leaf + optional intermediates).
pub cert_chain: Vec<CertificateDer<'static>>,
/// This node's private key matching the leaf cert above.
pub key: PrivateKeyDer<'static>,
/// The fleet root CA. Both server verifier and client verifier
/// require the peer's chain to end at this cert.
pub trusted_ca: CertificateDer<'static>,
}
impl NodeIdentity {
/// Build a fresh fleet CA and a pair of signed leaf certs named
/// after `node_a` and `node_b`. Returns the two independent
/// identities — each carries its own leaf but shares the CA so
/// they trust each other's certs.
///
/// Used by tests + as a starting point for the production
/// bootstrap flow (write PEMs to `/etc/claw-store/tls/`).
pub fn generate_test_pair(node_a: &str, node_b: &str) -> Result<(Self, Self)> {
let ca_key = rcgen::KeyPair::generate().context("generating CA key")?;
let mut ca_params = rcgen::CertificateParams::new(vec![])
.context("building CA params")?;
ca_params.is_ca = rcgen::IsCa::Ca(rcgen::BasicConstraints::Unconstrained);
ca_params
.distinguished_name
.push(rcgen::DnType::CommonName, "clawstor fleet CA");
let ca_cert = ca_params
.self_signed(&ca_key)
.context("self-signing CA cert")?;
let ca_der = ca_cert.der().clone();
let id_a = build_leaf(node_a, &ca_cert, &ca_key, ca_der.clone())?;
let id_b = build_leaf(node_b, &ca_cert, &ca_key, ca_der)?;
Ok((id_a, id_b))
}
}
/// Sign a single leaf cert for `name` under the given CA and package it
/// with the CA cert into a [`NodeIdentity`].
fn build_leaf(
name: &str,
ca_cert: &rcgen::Certificate,
ca_key: &rcgen::KeyPair,
ca_der: CertificateDer<'static>,
) -> Result<NodeIdentity> {
let leaf_key = rcgen::KeyPair::generate().context("generating leaf key")?;
// Include the node name as a DNS SAN so a client connecting with
// server_name=name passes rustls's cert-name verification.
let mut leaf_params = rcgen::CertificateParams::new(vec![name.to_string()])
.context("building leaf params")?;
leaf_params
.distinguished_name
.push(rcgen::DnType::CommonName, name);
let leaf_cert = leaf_params
.signed_by(&leaf_key, ca_cert, ca_key)
.context("signing leaf cert with CA")?;
let leaf_der = leaf_cert.der().clone();
// rustls wants the private key in DER form.
let key_der = PrivateKeyDer::try_from(leaf_key.serialize_der())
.map_err(|e| anyhow::anyhow!("converting leaf key to rustls form: {e}"))?;
Ok(NodeIdentity {
cert_chain: vec![leaf_der],
key: key_der,
trusted_ca: ca_der,
})
}
/// A running QUIC RPC server. Owns the quinn endpoint; drop stops it.
pub struct QuicServer {
endpoint: Endpoint,
}
impl QuicServer {
/// Bind a QUIC listener on `bind` with mTLS enforced against the
/// fleet CA in `identity`. `bind` accepts port 0 to let the OS
/// assign one (useful in tests).
pub fn bind(bind: SocketAddr, identity: NodeIdentity) -> Result<Self> {
install_default_crypto_provider();
let server_crypto = build_server_crypto(&identity)?;
let quic_crypto = quinn::crypto::rustls::QuicServerConfig::try_from(server_crypto)
.context("wrapping rustls ServerConfig for quinn")?;
let mut server_config = ServerConfig::with_crypto(Arc::new(quic_crypto));
Arc::get_mut(&mut server_config.transport)
.expect("fresh transport config is unique")
.max_concurrent_uni_streams(0u8.into())
.max_idle_timeout(Some(
VarInt::from_u64(IDLE_TIMEOUT.as_millis() as u64)
.expect("idle timeout fits u64")
.into(),
));
let endpoint =
Endpoint::server(server_config, bind).context("binding quinn server endpoint")?;
Ok(Self { endpoint })
}
/// The actual bound socket address (resolves port 0 to the assigned port).
pub fn local_addr(&self) -> Result<SocketAddr> {
self.endpoint.local_addr().map_err(Into::into)
}
/// Accept the next incoming connection. Returns `None` when the
/// endpoint is closed. The returned [`quinn::Connection`] is
/// already handshaked; call `ping_handler_loop` (or any other
/// server-side handler) to serve requests on it.
pub async fn accept(&self) -> Option<Result<quinn::Connection>> {
let incoming = self.endpoint.accept().await?;
// Two failure modes matter here:
// 1. `incoming.accept()` returns Err → convert to a Result inside
// the returned Option so callers see the accept failure.
// 2. The Connecting future fails → same shape.
let connecting = match incoming.accept() {
Ok(c) => c,
Err(e) => return Some(Err(anyhow::Error::from(e))),
};
Some(connecting.await.map_err(anyhow::Error::from))
}
/// Graceful shutdown.
pub async fn shutdown(&self) {
self.endpoint
.close(VarInt::from_u32(0), b"server shutdown");
self.endpoint.wait_idle().await;
}
}
/// A QUIC client for opening outbound peer connections.
pub struct QuicClient {
endpoint: Endpoint,
client_config: ClientConfig,
}
impl QuicClient {
/// Build a client bound to `bind_addr` (usually `0.0.0.0:0` — OS
/// picks the port) that verifies peer certs against the fleet CA
/// carried in `identity`, and presents `identity`'s leaf on mTLS
/// challenge.
pub fn new(bind_addr: SocketAddr, identity: NodeIdentity) -> Result<Self> {
install_default_crypto_provider();
let client_crypto = build_client_crypto(&identity)?;
let quic_crypto = quinn::crypto::rustls::QuicClientConfig::try_from(client_crypto)
.context("wrapping rustls ClientConfig for quinn")?;
let client_config = ClientConfig::new(Arc::new(quic_crypto));
let endpoint = Endpoint::client(bind_addr).context("binding quinn client endpoint")?;
Ok(Self {
endpoint,
client_config,
})
}
/// Connect to `addr` and validate the peer cert's SAN matches
/// `expected_server_name`. Returns the completed connection.
pub async fn connect(
&self,
addr: SocketAddr,
expected_server_name: &str,
) -> Result<quinn::Connection> {
let connecting = self
.endpoint
.connect_with(self.client_config.clone(), addr, expected_server_name)
.with_context(|| format!("dialing {addr} for name {expected_server_name}"))?;
connecting
.await
.with_context(|| format!("completing handshake with {addr}"))
}
/// Graceful shutdown.
pub async fn shutdown(&self) {
self.endpoint
.close(VarInt::from_u32(0), b"client shutdown");
self.endpoint.wait_idle().await;
}
}
/// Open a bidi stream on `conn`, send `payload`, read back the peer's
/// response (bounded by [`MAX_MESSAGE_BYTES`]). This is the client
/// side of the ping RPC.
pub async fn ping(conn: &quinn::Connection, payload: &[u8]) -> Result<Vec<u8>> {
if payload.len() > MAX_MESSAGE_BYTES {
bail!(
"ping payload {} bytes exceeds cap {}",
payload.len(),
MAX_MESSAGE_BYTES
);
}
let (mut send, mut recv) = conn
.open_bi()
.await
.context("opening bidi stream for ping")?;
send.write_all(payload).await.context("writing ping payload")?;
send.finish().context("finishing ping send stream")?;
let response = recv
.read_to_end(MAX_MESSAGE_BYTES)
.await
.context("reading ping response")?;
Ok(response)
}
/// Server side of the ping RPC. Loops accepting bidi streams on `conn`
/// and echoing every payload back prefixed with `b"pong:"`. Runs until
/// the connection closes.
pub async fn ping_handler_loop(conn: quinn::Connection) -> Result<()> {
loop {
let (mut send, mut recv) = match conn.accept_bi().await {
Ok(pair) => pair,
Err(quinn::ConnectionError::ApplicationClosed(_))
| Err(quinn::ConnectionError::ConnectionClosed(_))
| Err(quinn::ConnectionError::LocallyClosed) => return Ok(()),
Err(e) => return Err(anyhow::Error::from(e)),
};
let request = recv
.read_to_end(MAX_MESSAGE_BYTES)
.await
.context("reading ping request")?;
let mut response = b"pong:".to_vec();
response.extend_from_slice(&request);
send.write_all(&response)
.await
.context("writing pong response")?;
send.finish().context("finishing pong send stream")?;
}
}
/// Build the server-side rustls config: require client certs, chain to
/// our fleet CA, present our own leaf on handshake.
fn build_server_crypto(identity: &NodeIdentity) -> Result<rustls::ServerConfig> {
let mut ca_store = rustls::RootCertStore::empty();
ca_store
.add(identity.trusted_ca.clone())
.context("installing fleet CA in server root store")?;
let client_verifier =
rustls::server::WebPkiClientVerifier::builder(Arc::new(ca_store))
.build()
.context("building mTLS client verifier")?;
let mut cfg = rustls::ServerConfig::builder()
.with_client_cert_verifier(client_verifier)
.with_single_cert(identity.cert_chain.clone(), identity.key.clone_key())
.context("installing server leaf cert + key")?;
cfg.alpn_protocols = vec![CLAWSTOR_RPC_ALPN.to_vec()];
Ok(cfg)
}
/// Build the client-side rustls config: verify server cert chains to
/// our fleet CA, present our own leaf when the server challenges.
fn build_client_crypto(identity: &NodeIdentity) -> Result<rustls::ClientConfig> {
let mut ca_store = rustls::RootCertStore::empty();
ca_store
.add(identity.trusted_ca.clone())
.context("installing fleet CA in client root store")?;
let mut cfg = rustls::ClientConfig::builder()
.with_root_certificates(ca_store)
.with_client_auth_cert(identity.cert_chain.clone(), identity.key.clone_key())
.context("installing client leaf cert + key")?;
cfg.alpn_protocols = vec![CLAWSTOR_RPC_ALPN.to_vec()];
Ok(cfg)
}
/// Install the ring-backed CryptoProvider once per process. Called from
/// both server and client bootstrap. Idempotent — ignores the "already
/// installed" error so multiple endpoints in the same process (or tests)
/// coexist.
fn install_default_crypto_provider() {
// `install_default` returns Err if a provider is already set — that's
// fine; we just want SOME provider present when rustls builders run.
let _ = rustls::crypto::ring::default_provider().install_default();
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::atomic::{AtomicU16, Ordering};
/// Port allocator separate from the gossip tests (which use 41000+)
/// so parallel test execution never conflicts.
static NEXT_PORT: AtomicU16 = AtomicU16::new(42001);
fn next_port() -> u16 {
NEXT_PORT.fetch_add(1, Ordering::Relaxed)
}
fn loopback(port: u16) -> SocketAddr {
format!("127.0.0.1:{port}").parse().unwrap()
}
fn wildcard(port: u16) -> SocketAddr {
format!("0.0.0.0:{port}").parse().unwrap()
}
#[test]
fn generate_test_pair_produces_two_distinct_leaves_sharing_a_ca() {
let (a, b) = NodeIdentity::generate_test_pair("a", "b").unwrap();
assert_ne!(
a.cert_chain[0], b.cert_chain[0],
"each node must have a distinct leaf"
);
assert_eq!(
a.trusted_ca, b.trusted_ca,
"both nodes must trust the same CA to talk"
);
assert!(!a.cert_chain.is_empty());
assert!(!b.cert_chain.is_empty());
}
#[tokio::test]
async fn ping_pong_between_two_mtls_peers() {
let (id_a, id_b) = NodeIdentity::generate_test_pair("a", "b").unwrap();
// B binds a server on an OS-assigned port.
let server = QuicServer::bind(loopback(0), id_b).unwrap();
let server_addr = server.local_addr().unwrap();
// Spawn the accept loop: for every incoming connection, serve
// the ping RPC.
let accept_task = tokio::spawn(async move {
if let Some(res) = server.accept().await {
let conn = res.expect("accept succeeded");
let _ = ping_handler_loop(conn).await;
}
server.shutdown().await;
});
// A connects as a client and calls ping.
let client = QuicClient::new(loopback(0), id_a).unwrap();
let conn = client
.connect(server_addr, "b")
.await
.expect("A→B connect");
let response = ping(&conn, b"hello").await.expect("ping");
assert_eq!(response, b"pong:hello");
// Cleanup.
conn.close(VarInt::from_u32(0), b"done");
client.shutdown().await;
// Give the accept task a moment to finish, then abort to be safe.
tokio::time::sleep(Duration::from_millis(50)).await;
accept_task.abort();
}
#[tokio::test]
async fn client_rejects_peer_with_wrong_ca() {
// Two CAs, A and X. A pair (id_a, id_b) share CA_A; a rogue id_x
// has its own CA_X. When A tries to talk to X, TLS handshake
// must fail — X's leaf doesn't chain to CA_A.
let (id_a, _id_b) = NodeIdentity::generate_test_pair("a", "b").unwrap();
let (_id_x_peer, id_x) = NodeIdentity::generate_test_pair("x-peer", "x").unwrap();
let server = QuicServer::bind(loopback(0), id_x).unwrap();
let server_addr = server.local_addr().unwrap();
let _accept_task = tokio::spawn(async move {
// Server will fail the handshake because the client's cert
// doesn't chain to CA_X. We just want to be here to accept
// the SYN; the handshake failure is what the client asserts.
let _ = server.accept().await;
});
let client = QuicClient::new(loopback(0), id_a).unwrap();
let err = client
.connect(server_addr, "x")
.await
.err()
.expect("connect must fail — wrong CA");
let msg = format!("{err:#}");
// The exact error text varies by rustls version; look for the
// token that always appears in a chain-verification failure.
assert!(
msg.to_lowercase().contains("certificate")
|| msg.to_lowercase().contains("verify")
|| msg.to_lowercase().contains("closed")
|| msg.to_lowercase().contains("handshake"),
"unexpected error text: {msg}"
);
}
#[tokio::test]
async fn client_rejects_wrong_server_name() {
// A and B share a CA; B's cert has SAN=["b"]. A connects
// asking for server_name="c" — rustls must reject on SAN
// mismatch even though the cert chain is valid.
let (id_a, id_b) = NodeIdentity::generate_test_pair("a", "b").unwrap();
let server = QuicServer::bind(loopback(0), id_b).unwrap();
let server_addr = server.local_addr().unwrap();
let _accept_task = tokio::spawn(async move {
let _ = server.accept().await;
});
let client = QuicClient::new(loopback(0), id_a).unwrap();
let err = client
.connect(server_addr, "c")
.await
.err()
.expect("connect must fail — SAN mismatch");
let msg = format!("{err:#}").to_lowercase();
assert!(
msg.contains("name") || msg.contains("certificate") || msg.contains("handshake"),
"unexpected error text: {msg}"
);
}
#[tokio::test]
async fn server_binds_wildcard_and_reports_concrete_local_addr() {
let (_id_a, id_b) = NodeIdentity::generate_test_pair("a", "b").unwrap();
let server = QuicServer::bind(wildcard(0), id_b).unwrap();
let addr = server.local_addr().unwrap();
assert_ne!(addr.port(), 0, "OS must assign a real port");
server.shutdown().await;
}
#[tokio::test]
async fn ping_rejects_oversize_payload() {
let (id_a, id_b) = NodeIdentity::generate_test_pair("a", "b").unwrap();
let server = QuicServer::bind(loopback(0), id_b).unwrap();
let server_addr = server.local_addr().unwrap();
let _accept = tokio::spawn(async move {
if let Some(Ok(conn)) = server.accept().await {
let _ = ping_handler_loop(conn).await;
}
});
let client = QuicClient::new(loopback(0), id_a).unwrap();
let conn = client.connect(server_addr, "b").await.unwrap();
let oversized = vec![0u8; MAX_MESSAGE_BYTES + 1];
let err = ping(&conn, &oversized)
.await
.err()
.expect("must reject oversize payload before send");
assert!(err.to_string().contains("exceeds cap"));
conn.close(VarInt::from_u32(0), b"done");
client.shutdown().await;
}
}