clawverse/livecast
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Hetzner fleet deployment guide and nginx site config

Browse Source 
Fleet-specific overlay for deploying LiveCast to the Redclaw Hetzner
infrastructure, grounded in the real conventions (hel1, Cloudflare DNS, nginx
on the web host, systemd, Infisical).

- DEPLOY-HETZNER.md: chosen topology (reuse redclaw-web-01 as the event host,
  dedicated coturn node for full TURN-over-TLS on 443), Hetzner API server
  provisioning, Cloudflare DNS records (event proxied, turn DNS-only with the
  rationale), per-node setup, end-to-end verification, ops and rollback.
- deploy/nginx/event.redclaw.dev.conf: nginx site for hosts that use nginx
  instead of Caddy; serves the SPA, proxies /ws + /ice + /health to the
  signaling server, uses the existing *.redclaw.dev wildcard cert.
- Signaling port moved to 8088 on web-01 (8080 is taken by MindBridge there).
- README: pointers to both companion deploy guides.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
Omar Sobh
2026-06-04 17:51:45 -05:00
co-authored by Claude Opus 4.8
parent 6bb4cf1c63
commit 52fa3189e7
3 changed files with 395 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DEPLOY-HETZNER.md
+321
View File
@@ -0,0 +1,321 @@
# LiveCast on the Redclaw Hetzner Fleet
A production-style deployment guide for standing up LiveCast on the existing
Redclaw Hetzner infrastructure, with `redclaw.dev` domains mapped through
Cloudflare. This is grounded in the actual fleet conventions (Hetzner Cloud in
`hel1`, Cloudflare DNS, nginx on the web host, systemd, Infisical for secrets).
For the provider-neutral version of the steps see `DEPLOY.md`. This document is
the fleet-specific overlay.
---
## Chosen topology
Two nodes, full TURN-over-TLS-on-443 parity, reusing the primary web host:
| Role | Host | What runs there | Cloudflare |
| --- | --- | --- | --- |
| event | **redclaw-web-01** (reuse, `65.108.250.65`) | nginx site + Node signaling (systemd) + static SPA | `event.redclaw.dev` A record, **proxied (orange)** |
| turn | **livecast-turn-01** (new CX22, `hel1`) | coturn only, owns 443 | `turn.redclaw.dev` A record, **DNS-only (grey)** |
Why a separate TURN node: `redclaw-web-01` already runs nginx on 443 and has its
own coturn on 5349. Full TURN-over-TLS on 443 (the strict-firewall path the PRD
calls for) needs a host where coturn owns 443 outright, so coturn gets its own
small box. Everything else reuses the existing web host.
**Why `turn.redclaw.dev` must be grey-cloud (DNS-only):** Cloudflare's proxy
only handles HTTP/HTTPS. TURN and STUN (3478, 5349, 443 TLS, UDP relay range)
are not HTTP, so a proxied record would break them. The TURN record must point
straight at the node IP. `event.redclaw.dev` stays proxied like the rest of the
fleet (Cloudflare supports WebSockets through the proxy).
> Zero-new-node alternative: if you would rather not add a box, point
> `turn.redclaw.dev` (grey) at `redclaw-web-01` and reuse its existing
> `coturn.service`. You then lose TURN-over-TLS on 443 (coturn there can only use
> 5349, since nginx owns 443), which is the single-node trade-off. The rest of
> this guide is unchanged except you skip Part 1 and Part 3's install step.
---
## 0. Prerequisites (gather before starting)
Access:
- SSH to `redclaw-web-01` as `redclaw` (public `65.108.250.65` or Tailscale `100.94.185.103`).
- Hetzner Cloud API token (read-write): Infisical `cloud-providers/prod` (see `Valhalla/20 Infrastructure/30 Runbooks/Hetzner API.md`).
- Cloudflare Global API Key + email: Infisical `cloud-providers/prod/CLOUDFLARE_API_KEY`.
- Git access to `https://git.redclaw.dev/clawverse/livecast.git`.
Load the cloud credentials into your shell (fleet convention):
```
infisical run --projectId=$INFISICAL_PROJ_CLOUD_PROVIDERS --env=prod -- bash
# now $HCLOUD_TOKEN (or the Hetzner RW token), $CLOUDFLARE_EMAIL, $CLOUDFLARE_API_KEY are set
export ZONE_ID=05ee86579bbf2049dc35c8dff8702958 # redclaw.dev
export WEB01_IP=65.108.250.65
```
Local tools: `curl`, `jq`, and the Infisical CLI.
---
## 1. Provision the TURN node (Hetzner API)
Look up the SSH keys and the existing firewall (which already opens the STUN/TURN
ports), then create a CX22 in `hel1`.
```
# Firewall id for "redclaw-web-01" (already allows 22/80/443/3478/5349 + relay range)
FW_ID=$(curl -s https://api.hetzner.cloud/v1/firewalls \
-H "Authorization: Bearer $HCLOUD_TOKEN" | jq '.firewalls[] | select(.name=="redclaw-web-01") | .id')
echo "firewall: $FW_ID"
# Create the TURN node
curl -s -X POST https://api.hetzner.cloud/v1/servers \
-H "Authorization: Bearer $HCLOUD_TOKEN" -H "Content-Type: application/json" \
-d "{
\"name\": \"livecast-turn-01\",
\"server_type\": \"cx22\",
\"image\": \"ubuntu-24.04\",
\"location\": \"hel1\",
\"ssh_keys\": [\"[email protected]\", \"[email protected]\", \"[email protected]\"],
\"firewalls\": [{\"firewall\": $FW_ID}],
\"labels\": {\"role\": \"turn\", \"project\": \"livecast\"}
}" | jq '.server | {id, name, status, ip: .public_net.ipv4.ip}'
# Capture the public IP it reports, then:
export TURN_IP=<the ipv4 from the response>
```
The `redclaw-web-01` firewall is sufficient for the TURN node, so no new firewall
is needed. Because that firewall is the fleet's network-level control, ufw on the
node is optional; this guide relies on the Hetzner Cloud Firewall.
Optional, to match fleet management: install Tailscale on the node
(`curl -fsSL https://tailscale.com/install.sh | sh && sudo tailscale up`).
---
## 2. DNS records (Cloudflare API)
```
# event.redclaw.dev -> web-01, PROXIED (orange), matches the fleet
curl -s -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
-H "X-Auth-Email: $CLOUDFLARE_EMAIL" -H "X-Auth-Key: $CLOUDFLARE_API_KEY" \
-H "Content-Type: application/json" \
--data "{\"type\":\"A\",\"name\":\"event\",\"content\":\"$WEB01_IP\",\"proxied\":true,\"ttl\":1}" | jq '.success,.errors'
# turn.redclaw.dev -> TURN node, DNS-ONLY (grey). MUST NOT be proxied.
curl -s -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
-H "X-Auth-Email: $CLOUDFLARE_EMAIL" -H "X-Auth-Key: $CLOUDFLARE_API_KEY" \
-H "Content-Type: application/json" \
--data "{\"type\":\"A\",\"name\":\"turn\",\"content\":\"$TURN_IP\",\"proxied\":false,\"ttl\":300}" | jq '.success,.errors'
```
Verify before continuing (the turn record must return the node IP directly):
```
dig +short event.redclaw.dev # Cloudflare anycast IPs (proxied) is expected
dig +short turn.redclaw.dev # must return $TURN_IP exactly (grey cloud)
```
---
## 3. TURN node setup (run on livecast-turn-01)
SSH in as root (or the default user) and configure coturn. This mirrors
`DEPLOY.md` Phase A.
```
sudo apt-get update
sudo apt-get install -y coturn certbot git openssl
sudo git clone https://git.redclaw.dev/clawverse/livecast.git /opt/livecast
# Shared secret (store it in Infisical, e.g. livecast/prod/TURN_SECRET)
SECRET=$(openssl rand -hex 32)
sudo cp /opt/livecast/deploy/turnserver.conf /etc/turnserver.conf
sudo sed -i "s/REPLACE_WITH_LONG_RANDOM_SECRET/$SECRET/" /etc/turnserver.conf
sudo sed -i 's/^#TURNSERVER_ENABLED/TURNSERVER_ENABLED/' /etc/default/coturn
printf 'TURN_SECRET=%s\n' "$SECRET" # record for Part 4 and Infisical
# TLS cert for turn.redclaw.dev (grey-cloud, so certbot standalone on port 80 works)
sudo chmod +x /opt/livecast/deploy/coturn-cert-deploy-hook.sh
sudo certbot certonly --standalone -d turn.redclaw.dev \
--non-interactive --agree-tos -m [email protected] \
--deploy-hook /opt/livecast/deploy/coturn-cert-deploy-hook.sh
sudo systemctl enable --now coturn
```
`alt-tls-listening-port=443` stays enabled in `turnserver.conf`: coturn owns
this node, so 443 is free. Verify:
```
systemctl is-active coturn
echo | openssl s_client -connect turn.redclaw.dev:443 -servername turn.redclaw.dev 2>/dev/null \
| openssl x509 -noout -subject -dates
# Headless relay test with a minted ephemeral credential:
U=$(( $(date +%s) + 3600 ))
C=$(printf '%s' "$U" | openssl dgst -sha1 -hmac "$SECRET" -binary | openssl base64)
turnutils_uclient -v -u "$U" -w "$C" -p 3478 turn.redclaw.dev # expect non-zero transmit/receive
```
---
## 4. Event setup (run on redclaw-web-01)
This host serves 19+ production domains. Only **add** to it. Do not overwrite
the existing nginx config or touch the existing `coturn.service`.
### 4a. Pre-flight checks
```
# Confirm the signaling port we will use (8088) is free; 8080 is taken by MindBridge.
ss -tulpn | grep -E ':8088|:8080' || echo "8088 is free"
# Note the nginx site directory convention in use
ls /etc/nginx/conf.d/ /etc/nginx/sites-enabled/ 2>/dev/null
```
### 4b. Code, signaling service
```
sudo git clone https://git.redclaw.dev/clawverse/livecast.git /opt/livecast || \
sudo git -C /opt/livecast pull --ff-only
cd /opt/livecast/server
sudo npm install --omit=dev
sudo cp -n .env.example .env
SECRET='<paste the TURN_SECRET from Part 3>'
sudo sed -i \
-e 's/^PORT=.*/PORT=8088/' \
-e 's#^TURN_HOST=.*#TURN_HOST=turn.redclaw.dev#' \
-e 's#^TURN_REALM=.*#TURN_REALM=turn.redclaw.dev#' \
-e "s#^TURN_SECRET=.*#TURN_SECRET=${SECRET}#" \
-e 's#^TURN_TTL_SECONDS=.*#TURN_TTL_SECONDS=43200#' \
/opt/livecast/server/.env
sudo cp /opt/livecast/deploy/livecast-signaling.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now livecast-signaling
systemctl is-active livecast-signaling
curl -s localhost:8088/health # expect status JSON, broadcasterOnline:false
curl -s localhost:8088/ice # expect a turns:turn.redclaw.dev:443 entry with username/credential
```
### 4c. Web build and nginx site
```
cd /opt/livecast/web
npm install
npm run build
sudo mkdir -p /var/www/livecast
sudo rm -rf /var/www/livecast/*
sudo cp -r dist/* /var/www/livecast/
# Add the nginx site (uses the existing *.redclaw.dev wildcard cert, proxies to :8088)
sudo cp /opt/livecast/deploy/nginx/event.redclaw.dev.conf /etc/nginx/conf.d/event.redclaw.dev.conf
sudo nginx -t && sudo systemctl reload nginx
```
If `/etc/nginx/conf.d/` is not where this host keeps sites (check 4a output),
place the file in the directory the other domains use and reload.
---
## 5. End-to-end verification
```
# Through Cloudflare to web-01 nginx to the signaling server
curl -s https://event.redclaw.dev/health # 200 + status JSON (AC-7)
curl -s https://event.redclaw.dev/ice | grep -o 'turns:[^"]*' # turns:turn.redclaw.dev:443 (AC-8)
# TLS is valid (Cloudflare edge cert for the proxied host)
curl -sI https://event.redclaw.dev/ | head -n1 # HTTP/2 200 (AC-9)
```
Then the real test, on two devices on different networks (this is the point of
the deployment, and it cannot be verified headlessly):
1. Phone A: open `https://event.redclaw.dev/broadcast`, tap Go Live, grant the mic.
2. Phone B (different network, ideally cellular): open `https://event.redclaw.dev/listen`,
tap Connect and Enable Audio. Confirm the LIVE indicator, the go-live
notification, audible audio, and the waveform.
3. Force-relay check: in the browser console on the listener, or via the WebRTC
Trickle ICE tool against `turn.redclaw.dev`, confirm a `relay` candidate
appears (proves the TURN path, AC-3 / AC-10).
---
## 6. Secrets, operations, rollback
Secrets:
- Store `TURN_SECRET` in Infisical (`livecast/prod/TURN_SECRET`). It exists in two
places that must match: `/etc/turnserver.conf` on the TURN node and
`/opt/livecast/server/.env` on web-01. It is never sent to a browser.
Operations:
| Action | Where | Command |
| --- | --- | --- |
| Signaling logs | web-01 | `journalctl -u livecast-signaling -f` |
| Restart signaling | web-01 | `sudo systemctl restart livecast-signaling` |
| coturn logs | turn node | `journalctl -u coturn -f` |
| Reload nginx | web-01 | `sudo nginx -t && sudo systemctl reload nginx` |
| Cert renewal (turn) | turn node | automatic via certbot timer + deploy hook |
Redeploy after a code change:
```
# web-01
sudo git -C /opt/livecast pull --ff-only
cd /opt/livecast/server && sudo npm install --omit=dev && sudo systemctl restart livecast-signaling
cd /opt/livecast/web && npm install && npm run build && sudo rm -rf /var/www/livecast/* && sudo cp -r dist/* /var/www/livecast/
sudo systemctl reload nginx
```
Rollback (clean removal from web-01, leaves the 19 other domains untouched):
```
sudo rm /etc/nginx/conf.d/event.redclaw.dev.conf
sudo systemctl reload nginx
sudo systemctl disable --now livecast-signaling
sudo rm /etc/systemd/system/livecast-signaling.service
sudo systemctl daemon-reload
# optionally remove /opt/livecast and /var/www/livecast
```
To tear down the TURN node, delete `livecast-turn-01` in the Hetzner console or
via API, and remove the `turn` DNS record.
---
## 7. Acceptance criteria
Headless (an operator or agent can confirm these from the commands above):
- AC-7 `/health` 200, AC-8 `/ice` ephemeral creds, AC-9 valid TLS,
AC-3 / AC-10 relay allocation via `turnutils_uclient`.
Requires real devices and networks (schedule a human tester):
- AC-1 iOS Safari and Android Chrome over cellular and Wi-Fi, AC-2 sub-second
latency cross-network, AC-4 notification timing, AC-5 stop timing,
AC-6 reconnect.
---
## 8. Notes specific to this fleet
- `redclaw-web-01` is ARM (cax21). Node 22 and the Vite build run fine on ARM.
- Port 8080 on web-01 is MindBridge; LiveCast signaling uses 8088. If 8088 is
also taken (check in 4a), pick another free port and update both `.env`
(`PORT=`) and the nginx upstream in `event.redclaw.dev.conf`.
- `event.redclaw.dev` is Cloudflare-proxied with SSL mode Full, like the rest of
the fleet; WebSockets pass through the proxy. The origin uses the existing
`*.redclaw.dev` wildcard cert at `/etc/letsencrypt/live/redclaw.dev/`.
- Leave the existing `coturn.service` on web-01 alone; LiveCast's TURN runs on
the dedicated node.
README.md
+9 -1
View File
@@ -87,7 +87,15 @@ Web (`web/.env`, see `.env.example`):
## Production deployment runbook ## Production deployment runbook
LiveCast deploys across two nodes: Companion guides:
- `DEPLOY.md` is the provider-neutral, agent-executable runbook (verification at
each step, what an agent can and cannot confirm).
- `DEPLOY-HETZNER.md` is the Redclaw fleet overlay: provision on Hetzner `hel1`,
map Cloudflare DNS for `redclaw.dev`, reuse `redclaw-web-01` (nginx) as the
event host with a dedicated coturn node.
The generic two-node model follows. LiveCast deploys across two nodes:
- **event node** (`event.redclaw.dev`): Caddy (TLS, static SPA, reverse proxy) - **event node** (`event.redclaw.dev`): Caddy (TLS, static SPA, reverse proxy)
and the Node signaling server. and the Node signaling server.
deploy/nginx/event.redclaw.dev.conf
+65
View File
@@ -0,0 +1,65 @@
# nginx site for event.redclaw.dev (LiveCast web app + signaling).
#
# For hosts that already standardize on nginx (e.g. redclaw-web-01). It serves
# the static SPA build from /var/www/livecast and reverse-proxies the signaling
# endpoints to the Node server on 127.0.0.1. Uses the existing wildcard
# *.redclaw.dev Let's Encrypt certificate.
#
# IMPORTANT: the upstream port below (8088) must match PORT in
# /opt/livecast/server/.env. On redclaw-web-01 the default 8080 is already used
# by MindBridge, so LiveCast uses 8088.
#
# Install: /etc/nginx/conf.d/event.redclaw.dev.conf, then `nginx -t && systemctl reload nginx`.
upstream livecast_signaling {
server 127.0.0.1:8088;
}
server {
listen 80;
listen [::]:80;
server_name event.redclaw.dev;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl;
listen [::]:443 ssl;
http2 on;
server_name event.redclaw.dev;
# Existing fleet wildcard cert (already present on redclaw-web-01).
ssl_certificate /etc/letsencrypt/live/redclaw.dev/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/redclaw.dev/privkey.pem;
root /var/www/livecast;
index index.html;
# Signaling WebSocket: requires HTTP/1.1 upgrade headers and a long timeout.
location /ws {
proxy_pass http://livecast_signaling;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
}
# Ephemeral TURN credentials and health, proxied to the signaling server.
location = /ice {
proxy_pass http://livecast_signaling;
proxy_set_header Host $host;
}
location = /health {
proxy_pass http://livecast_signaling;
proxy_set_header Host $host;
}
# Static SPA with client-side routing (/broadcast, /listen).
location / {
try_files $uri /index.html;
}
}