From a0f6cd71752e77d862aff16d0dda902e4c829234 Mon Sep 17 00:00:00 2001 From: Omar Sobh Date: Fri, 18 Sep 2026 11:50:54 -0500 Subject: [PATCH] docs(egress): the remediation was applied, failed twice, and what holds MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MISSION-EGRESS.md said the fix was deliberately not applied and prescribed DOCKER-USER. Both would now mislead. Appended what happened on 2026-09-18: missions shared the server's subnet, DOCKER-USER loses to ts-forward on every tailscaled restart, raw PREROUTING dropped the server's replies — and the design that holds, verified from inside a real mission container. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz --- docs/MISSION-EGRESS.md | 88 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 87 insertions(+), 1 deletion(-) diff --git a/docs/MISSION-EGRESS.md b/docs/MISSION-EGRESS.md index 87a096e..6d1e308 100644 --- a/docs/MISSION-EGRESS.md +++ b/docs/MISSION-EGRESS.md @@ -1,5 +1,11 @@ # What a mission container can reach — 2026-08-27 +> **Applied 2026-09-18.** The lateral path this document measured is closed. +> The remediation below was applied, failed twice in ways worth reading, and +> was replaced by a different design. See [Applied — what actually +> worked](#applied--what-actually-worked-2026-09-18) at the end before +> acting on anything above it. + Measured, not modelled. Nothing in this document changes production; it exists because plan item 4 named a fix that cannot reach the problem, and the problem turned out to be larger than the one it was written for. @@ -99,7 +105,7 @@ Stated because a report that lists only the bad half is not a measurement: - `169.254.169.254` is link-local on bare metal, not a cloud metadata service. Reachable, but there is no credential endpoint behind it on gw-04. -## Remediation — described, deliberately NOT applied +## Remediation — described, deliberately NOT applied (superseded — see the end) The operator's call was to measure and report. The smallest change that closes the lateral path, for whenever that decision is made: @@ -150,3 +156,83 @@ says what it means. The container's networks are the fact, not the return code. - `architect` and the fleet nodes were offline, so they were not probed. - The probe container ran `alpine`, not `clawmates-runtime`. Same two networks and the same route table; a different image cannot have more access. + +## Applied — what actually worked (2026-09-18) + +Everything measured above was re-measured first, from a container on the +mission egress network, and was still true: tank's SSH, web-01's registry, +this host's SSH, link-local — all open. Then the rules above were applied, and +the positive/negative control pair this document asked for found three things +the rules could not have known. + +### 1. Missions shared a subnet with the server + +Missions egressed from `clawmates_edge` (172.23/16). So does +`clawmates_server_1`, and the server **needs** the tailnet: the Beszel hub on +architect (`:8090`), Ollama for the local-model backend (`:11434`), and the +node daemons (`:8088`) for exec-test and node-placed terminals. The tailnet +drop scoped to 172.23/16 cut the server off from architect:8090 within the +minute it was applied. No rule on that subnet could ever be right for both. + +**Fix:** missions egress from a network of their own, `clawmates_missions`, +pinned at **172.25.0.0/16** (`mission_runtime.rs` `EDGE_NETWORK`, commit +`869c3ad`; declared in both compose files). `core` is unchanged — the skills +door and the API are still reached over 172.20. Compose v1 will not create a +network no service uses, so on gw-04 it was created by hand with compose's own +labels (`com.docker.compose.network=missions`, `…project=clawmates`), which +compose then accepts as its own. + +### 2. `DOCKER-USER` loses to Tailscale + +`tailscaled` inserts `-j ts-forward` at the head of FORWARD on every restart, +above `-j DOCKER-USER`, and `ts-forward` ends in `-o tailscale0 -j ACCEPT`. A +tailnet drop in `DOCKER-USER` worked until `systemctl restart tailscaled`, and +then the tailnet was open again — verified by doing exactly that. Re-asserting +the rule from an `ExecStartPost` on `tailscaled` ran before Tailscale had +installed its chains and lost the same race. Link-local, which does not go +through `tailscale0`, stayed blocked throughout; that is what made the cause +legible. + +### 3. `raw PREROUTING` cannot see connection state + +Moving the drops to the raw table put them ahead of every chain Tailscale +touches — and ahead of conntrack. A drop on `-d 100.64.0.0/10` then matched +the server's **replies** to tailnet clients (tank's node daemon, the operator's +`curl`) and took the API off `100.102.112.85:8088` while every container +reported healthy. The public site kept serving, because Traefik reaches the +server over the Docker network. + +### What holds + +`/usr/local/sbin/clawmates-egress.sh` on gw-04, run by +`clawmates-egress.service` at boot and by `ExecStartPost` drop-ins on +`docker.service` and `tailscaled.service`. Idempotent (`-C` before `-I`). The +rules live in **`mangle PREROUTING` with `-m conntrack --ctstate NEW`**: after +conntrack, so replies pass; before FORWARD and INPUT, so ordering against +`ts-forward` is moot; and neither Docker nor Tailscale writes to that table. + +``` +missions (172.25/16): DROP NEW → 100.64/10, 10/8, 192.168/16, 169.254/16, tcp/22 anywhere +edge (172.23/16): DROP NEW → 10/8, 192.168/16, 169.254/16, tcp/22 anywhere (tailnet allowed) +``` + +Verified three ways, then verified again after restarting both daemons: + +| from | tailnet | host ssh | link-local | public | core door | +|---|---|---|---|---|---| +| mission subnet (throwaway container) | blocked | blocked | blocked | open | 200 | +| **inside a real mission container** (`01a0b550`) | blocked | blocked | blocked | open | 200 | +| edge (the server) | **open** | blocked | blocked | open | — | + +Inbound to the API on the tailnet port: 200 from the host and from tank. The +mission itself ran normally behind the policy — 19 `curl` fetches, 2 skill +reads through the door, judge `met=true`, independent. + +### Limits + +- gw-04 only. The laptop has the network (`clawmates_missions`, same subnet) + and no firewall; a local mission can still reach the LAN and tailnet. +- Reachability of a port, as before. A mission can still `curl` any public host. +- The microVM tier has its own egress model (per-backend rootfs, vsock) and is + not covered by any of this. +