Flesh out deployment runbook with copy-pasteable VPS commands

Make the production runbook executable on a fresh Ubuntu host instead of a
high-level checklist, and fix gaps that would block a first deploy.

- README: concrete apt installs (Node, Caddy, coturn, certbot), git clone,
  ufw rules, certbot issuance for the TURN cert, secret injection, coturn
  enable, signaling and web steps. Documents the single-IP vs two-IP
  TURN-over-TLS 443 trade-off and automatic cert renewal.
- deploy/coturn-cert-deploy-hook.sh: certbot deploy hook that makes the cert
  readable by the unprivileged coturn user and restarts coturn on renewal.
- turnserver.conf: read cert/key from /etc/coturn (set by the hook); annotate
  the 443 listener conflict with Caddy.
- Caddyfile: drop the misconfigured on-demand turn.redclaw.dev block; coturn
  terminates its own TLS.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

README.md

@@ -29,7 +29,7 @@ through the TURN relay only when a direct path cannot be established.
 ```
 server/   Node + ws signaling server (/ws, /ice, /health)
 web/      React + Vite SPA (/broadcast, /listen)
-deploy/   Caddyfile, coturn turnserver.conf, systemd unit
+deploy/   Caddyfile, coturn turnserver.conf, systemd unit, certbot deploy hook
 ```
 
 ## Quick start (local)
@@ -87,35 +87,129 @@ Web (`web/.env`, see `.env.example`):
 
 ## Production deployment runbook
 
-All on a single small VPS (Hetzner or Vultr). Follow the PRD roadmap phases.
+A single small VPS (Hetzner or Vultr) runs everything: Caddy (TLS + static SPA
++ reverse proxy), the Node signaling server, and coturn. Commands below assume
+a fresh Ubuntu 22.04/24.04 host and `sudo` access. Replace `redclaw.dev` hosts
+and the certbot email if they differ.
 
 ### Phase 0, provisioning
 
-- Provision the VPS. Install Caddy, Node LTS, and coturn.
+DNS first, so cert issuance works later: point `event.redclaw.dev` and
-- DNS: `event.redclaw.dev` and `turn.redclaw.dev` A records to the host.
+`turn.redclaw.dev` A records at the host's public IP.
-- Open firewall ports: TCP 80, 443; TCP/UDP 3478, 5349, 443 (TURN); UDP 49152-65535 (relay range).
+
+Install Node LTS, Caddy, coturn, certbot, and git:
+
+```
+# Node LTS (NodeSource)
+curl -fsSL https://deb.nodesource.com/setup_lts.x | sudo -E bash -
+sudo apt-get install -y nodejs
+
+# Caddy (official apt repo)
+sudo apt-get install -y debian-keyring debian-archive-keyring apt-transport-https curl
+curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' \
+  | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
+curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' \
+  | sudo tee /etc/apt/sources.list.d/caddy-stable.list
+sudo apt-get update
+sudo apt-get install -y caddy coturn certbot git
+```
+
+Get the code onto the box:
+
+```
+sudo git clone https://git.redclaw.dev/clawverse/livecast.git /opt/livecast
+```
+
+Open the firewall (`ufw`):
+
+```
+sudo ufw allow 80/tcp        # HTTP (Let's Encrypt, redirect to HTTPS)
+sudo ufw allow 443           # HTTPS (Caddy)
+sudo ufw allow 3478          # STUN/TURN (UDP + TCP)
+sudo ufw allow 5349          # TURN over TLS
+sudo ufw allow 49152:65535/udp   # TURN relay range
+sudo ufw enable
+```
 
 ### Phase 1, TURN/STUN
 
-- Generate the shared secret: `openssl rand -hex 32`.
+Generate the shared secret, install the config, and inject the secret into the
-- Put it in `deploy/turnserver.conf` (`static-auth-secret`) and in `server/.env` (`TURN_SECRET`). The two must match.
+live config (not the repo copy). **Save the printed secret** for the signaling
-- Install the config: `sudo cp deploy/turnserver.conf /etc/turnserver.conf`, then start coturn.
+server's `.env` in Phase 2; the two must match.
-- Verify with a Trickle ICE test against `turn.redclaw.dev` that both `srflx` (STUN) and `relay` (TURN) candidates appear (AC-10).
+
-- Confirm TURN-over-TLS on 443 works from a locked-down network.
+```
+SECRET=$(openssl rand -hex 32)
+sudo cp /opt/livecast/deploy/turnserver.conf /etc/turnserver.conf
+sudo sed -i "s/REPLACE_WITH_LONG_RANDOM_SECRET/$SECRET/" /etc/turnserver.conf
+echo "TURN_SECRET=$SECRET"   # copy this value into server/.env
+
+# coturn ships disabled on Ubuntu; enable the daemon
+sudo sed -i 's/^#TURNSERVER_ENABLED/TURNSERVER_ENABLED/' /etc/default/coturn
+```
+
+**TURN-over-TLS on 443 (decision point).** On a single public IP, Caddy owns
+TCP 443 for HTTPS, so coturn cannot also bind 443. Two options:
+
+- *Single IP (default, zero cost):* keep TURN-over-TLS on 5349 only. Comment
+  the 443 listener out: `sudo sed -i 's/^alt-tls-listening-port=443/#&/' /etc/turnserver.conf`.
+  Slightly weaker traversal through firewalls that block everything except 443.
+- *Two IPs (full PRD behavior):* attach a second IP to the host, set
+  `listening-ip` / `relay-ip` to it in `/etc/turnserver.conf`, and leave
+  `alt-tls-listening-port=443` enabled so TURN-over-TLS is reachable on 443.
+
+Obtain the TURN certificate. The deploy hook copies it into `/etc/coturn` (so
+the unprivileged coturn user can read it) and restarts coturn, now and on every
+future renewal. certbot needs port 80, so stop Caddy briefly:
+
+```
+sudo chmod +x /opt/livecast/deploy/coturn-cert-deploy-hook.sh
+sudo systemctl stop caddy
+sudo certbot certonly --standalone -d turn.redclaw.dev \
+  --non-interactive --agree-tos -m [email protected] \
+  --deploy-hook /opt/livecast/deploy/coturn-cert-deploy-hook.sh
+sudo systemctl start caddy
+sudo systemctl enable coturn
+```
+
+Verify with a Trickle ICE test against `turn.redclaw.dev` that both `srflx`
+(STUN) and `relay` (TURN) candidates appear (AC-10), then confirm TURN-over-TLS
+works from a locked-down network.
 
 ### Phase 2, signaling
 
-- Copy `server/` to `/opt/livecast/server`, run `npm install --omit=dev`.
+```
-- Create `/opt/livecast/server/.env` from `.env.example` with the production values.
+cd /opt/livecast/server
-- Install the systemd unit: `sudo cp deploy/livecast-signaling.service /etc/systemd/system/`, then `sudo systemctl enable --now livecast-signaling`.
+sudo npm install --omit=dev
-- Verify `/health` returns 200 (AC-7).
+sudo cp .env.example .env
+sudo nano .env        # set TURN_SECRET (the value from Phase 1), TURN_HOST,
+                      # TURN_REALM = turn.redclaw.dev. Leave PORT=8080.
+
+sudo cp /opt/livecast/deploy/livecast-signaling.service /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl enable --now livecast-signaling
+
+curl localhost:8080/health     # expect 200 with status JSON (AC-7)
+```
 
 ### Phase 3, web app
 
-- Build: `cd web && VITE_SIGNAL_URL=wss://event.redclaw.dev/ws npm run build` (or leave it blank to derive from the origin, which also works behind the proxy).
+Build on the box (Node is installed) and let Caddy serve the static output.
-- Copy `web/dist` to `/var/www/livecast`.
+`VITE_SIGNAL_URL` is left blank so the client derives `wss://event.redclaw.dev/ws`
-- Install `deploy/Caddyfile` at `/etc/caddy/Caddyfile`, then `sudo systemctl reload caddy`. Caddy auto-provisions TLS.
+from the origin, which is correct behind the proxy.
-- Verify HTTPS loads with no mixed-content or insecure-WebSocket warnings (AC-9).
+
+```
+cd /opt/livecast/web
+npm install
+npm run build
+sudo mkdir -p /var/www/livecast
+sudo cp -r dist/* /var/www/livecast/
+
+sudo cp /opt/livecast/deploy/Caddyfile /etc/caddy/Caddyfile
+sudo systemctl reload caddy     # Caddy auto-provisions TLS for event.redclaw.dev
+```
+
+Load `https://event.redclaw.dev/listen` and `/broadcast`; confirm valid TLS and
+no mixed-content or insecure-WebSocket warnings (AC-9).
 
 ### Phase 4, cross-network validation
 
@@ -130,6 +224,9 @@ All on a single small VPS (Hetzner or Vultr). Follow the PRD roadmap phases.
 - Confirm systemd restart policy and add log rotation.
 - Add health monitoring on `/health` and on coturn.
 - Confirm TURN credentials delivered to the browser are time-limited and expire (AC-8).
+- Certificate renewal is automatic: certbot's systemd timer renews the
+  `turn.redclaw.dev` cert and the deploy hook restarts coturn; Caddy renews its
+  own cert. Verify the timer is active with `systemctl status certbot.timer`.
 - Sign off against the acceptance criteria.
 
 ## Acceptance criteria checklist

deploy/Caddyfile

@@ -21,12 +21,8 @@ event.redclaw.dev {
 	file_server
 }
 
-# Optional: if TURN-over-TLS terminates with a cert Caddy manages, having this
+# Note: turn.redclaw.dev is NOT served by Caddy. coturn terminates its own TLS
-# block makes Caddy obtain and renew the certificate for turn.redclaw.dev.
+# using a certificate obtained by certbot (see the runbook and the certbot
-# coturn then reads the cert/key files directly (see turnserver.conf).
+# deploy hook in deploy/coturn-cert-deploy-hook.sh). On a single-IP host Caddy
-turn.redclaw.dev {
+# owns TCP 443 for HTTPS, so coturn's TURN-over-TLS uses 5349 there; see the
-	tls {
+# runbook for the second-IP option that restores TURN-over-TLS on 443.
-		on_demand
-	}
-	respond "LiveCast TURN host" 200
-}

deploy/coturn-cert-deploy-hook.sh

@@ -0,0 +1,22 @@
+#!/bin/sh
+# certbot deploy hook for coturn.
+#
+# Runs as root after every successful certificate issuance or renewal for
+# turn.redclaw.dev. coturn runs as the unprivileged "turnserver" user and
+# cannot read the root-only privkey under /etc/letsencrypt, so this copies the
+# cert and key into /etc/coturn, hands them to turnserver, and restarts coturn
+# so it picks up the fresh certificate.
+#
+# Wired up via:  certbot certonly ... --deploy-hook /opt/livecast/deploy/coturn-cert-deploy-hook.sh
+set -eu
+
+LINEAGE="${RENEWED_LINEAGE:-/etc/letsencrypt/live/turn.redclaw.dev}"
+DEST=/etc/coturn
+
+mkdir -p "$DEST"
+cp "$LINEAGE/fullchain.pem" "$DEST/fullchain.pem"
+cp "$LINEAGE/privkey.pem" "$DEST/privkey.pem"
+chown turnserver:turnserver "$DEST/fullchain.pem" "$DEST/privkey.pem"
+chmod 640 "$DEST/privkey.pem"
+
+systemctl restart coturn

deploy/turnserver.conf

@@ -7,7 +7,10 @@
 
 listening-port=3478
 tls-listening-port=5349
-# Also listen on 443 for TURN-over-TLS to defeat strict firewalls:
+# Also listen on 443 for TURN-over-TLS to defeat strict firewalls.
+# IMPORTANT: on a single-IP host Caddy already owns TCP 443 for HTTPS, so this
+# line must be commented out there (TURN-over-TLS then uses 5349). Keep it only
+# if coturn is bound to a second IP. See the runbook "TURN-over-TLS on 443".
 alt-tls-listening-port=443
 
 fingerprint
@@ -18,8 +21,12 @@ realm=turn.redclaw.dev
 total-quota=100
 stale-nonce=600
 
-cert=/etc/letsencrypt/live/turn.redclaw.dev/fullchain.pem
+# coturn runs as the unprivileged "turnserver" user and cannot read the
-pkey=/etc/letsencrypt/live/turn.redclaw.dev/privkey.pem
+# root-only key under /etc/letsencrypt. The certbot deploy hook
+# (deploy/coturn-cert-deploy-hook.sh) copies the cert/key to /etc/coturn and
+# chowns them to turnserver on every issuance and renewal.
+cert=/etc/coturn/fullchain.pem
+pkey=/etc/coturn/privkey.pem
 no-tlsv1
 no-tlsv1_1