docs(egress): the remediation was applied, failed twice, and what holds

MISSION-EGRESS.md said the fix was deliberately not applied and prescribed
DOCKER-USER. Both would now mislead. Appended what happened on 2026-09-18:
missions shared the server's subnet, DOCKER-USER loses to ts-forward on every
tailscaled restart, raw PREROUTING dropped the server's replies — and the
design that holds, verified from inside a real mission container.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz

docs/MISSION-EGRESS.md

@@ -1,5 +1,11 @@
 # What a mission container can reach — 2026-08-27
 
+> **Applied 2026-09-18.** The lateral path this document measured is closed.
+> The remediation below was applied, failed twice in ways worth reading, and
+> was replaced by a different design. See [Applied — what actually
+> worked](#applied--what-actually-worked-2026-09-18) at the end before
+> acting on anything above it.
+
 Measured, not modelled. Nothing in this document changes production; it exists
 because plan item 4 named a fix that cannot reach the problem, and the problem
 turned out to be larger than the one it was written for.
@@ -99,7 +105,7 @@ Stated because a report that lists only the bad half is not a measurement:
 - `169.254.169.254` is link-local on bare metal, not a cloud metadata service.
   Reachable, but there is no credential endpoint behind it on gw-04.
 
-## Remediation — described, deliberately NOT applied
+## Remediation — described, deliberately NOT applied (superseded — see the end)
 
 The operator's call was to measure and report. The smallest change that closes
 the lateral path, for whenever that decision is made:
@@ -150,3 +156,83 @@ says what it means. The container's networks are the fact, not the return code.
 - `architect` and the fleet nodes were offline, so they were not probed.
 - The probe container ran `alpine`, not `clawmates-runtime`. Same two networks
   and the same route table; a different image cannot have more access.
+
+## Applied — what actually worked (2026-09-18)
+
+Everything measured above was re-measured first, from a container on the
+mission egress network, and was still true: tank's SSH, web-01's registry,
+this host's SSH, link-local — all open. Then the rules above were applied, and
+the positive/negative control pair this document asked for found three things
+the rules could not have known.
+
+### 1. Missions shared a subnet with the server
+
+Missions egressed from `clawmates_edge` (172.23/16). So does
+`clawmates_server_1`, and the server **needs** the tailnet: the Beszel hub on
+architect (`:8090`), Ollama for the local-model backend (`:11434`), and the
+node daemons (`:8088`) for exec-test and node-placed terminals. The tailnet
+drop scoped to 172.23/16 cut the server off from architect:8090 within the
+minute it was applied. No rule on that subnet could ever be right for both.
+
+**Fix:** missions egress from a network of their own, `clawmates_missions`,
+pinned at **172.25.0.0/16** (`mission_runtime.rs` `EDGE_NETWORK`, commit
+`869c3ad`; declared in both compose files). `core` is unchanged — the skills
+door and the API are still reached over 172.20. Compose v1 will not create a
+network no service uses, so on gw-04 it was created by hand with compose's own
+labels (`com.docker.compose.network=missions`, `…project=clawmates`), which
+compose then accepts as its own.
+
+### 2. `DOCKER-USER` loses to Tailscale
+
+`tailscaled` inserts `-j ts-forward` at the head of FORWARD on every restart,
+above `-j DOCKER-USER`, and `ts-forward` ends in `-o tailscale0 -j ACCEPT`. A
+tailnet drop in `DOCKER-USER` worked until `systemctl restart tailscaled`, and
+then the tailnet was open again — verified by doing exactly that. Re-asserting
+the rule from an `ExecStartPost` on `tailscaled` ran before Tailscale had
+installed its chains and lost the same race. Link-local, which does not go
+through `tailscale0`, stayed blocked throughout; that is what made the cause
+legible.
+
+### 3. `raw PREROUTING` cannot see connection state
+
+Moving the drops to the raw table put them ahead of every chain Tailscale
+touches — and ahead of conntrack. A drop on `-d 100.64.0.0/10` then matched
+the server's **replies** to tailnet clients (tank's node daemon, the operator's
+`curl`) and took the API off `100.102.112.85:8088` while every container
+reported healthy. The public site kept serving, because Traefik reaches the
+server over the Docker network.
+
+### What holds
+
+`/usr/local/sbin/clawmates-egress.sh` on gw-04, run by
+`clawmates-egress.service` at boot and by `ExecStartPost` drop-ins on
+`docker.service` and `tailscaled.service`. Idempotent (`-C` before `-I`). The
+rules live in **`mangle PREROUTING` with `-m conntrack --ctstate NEW`**: after
+conntrack, so replies pass; before FORWARD and INPUT, so ordering against
+`ts-forward` is moot; and neither Docker nor Tailscale writes to that table.
+
+```
+missions (172.25/16): DROP NEW → 100.64/10, 10/8, 192.168/16, 169.254/16, tcp/22 anywhere
+edge     (172.23/16): DROP NEW → 10/8, 192.168/16, 169.254/16, tcp/22 anywhere   (tailnet allowed)
+```
+
+Verified three ways, then verified again after restarting both daemons:
+
+| from | tailnet | host ssh | link-local | public | core door |
+|---|---|---|---|---|---|
+| mission subnet (throwaway container) | blocked | blocked | blocked | open | 200 |
+| **inside a real mission container** (`01a0b550`) | blocked | blocked | blocked | open | 200 |
+| edge (the server) | **open** | blocked | blocked | open | — |
+
+Inbound to the API on the tailnet port: 200 from the host and from tank. The
+mission itself ran normally behind the policy — 19 `curl` fetches, 2 skill
+reads through the door, judge `met=true`, independent.
+
+### Limits
+
+- gw-04 only. The laptop has the network (`clawmates_missions`, same subnet)
+  and no firewall; a local mission can still reach the LAN and tailnet.
+- Reachability of a port, as before. A mission can still `curl` any public host.
+- The microVM tier has its own egress model (per-backend rootfs, vsock) and is
+  not covered by any of this.
+