clawmates/docs/MISSION-EGRESS.md

# What a mission container can reach — 2026-08-27

> **Applied 2026-09-18.** The lateral path this document measured is closed.
> The remediation below was applied, failed twice in ways worth reading, and
> was replaced by a different design. See [Applied — what actually
> worked](#applied--what-actually-worked-2026-09-18) at the end before
> acting on anything above it.

Measured, not modelled. Nothing in this document changes production; it exists
because plan item 4 named a fix that cannot reach the problem, and the problem
turned out to be larger than the one it was written for.

## Item 4 was mis-scoped

The plan said to pull upstream's `0db7d999a feat(plugins): add shared egress
policy foundation (#9137)` — "defence for the egress problem we have not
solved". It is good work (DNS pinning, IPv4-mapped metadata blocking,
proxy-conflict surfacing, ~2,400 lines across three crates) and it **cannot
observe a single mission tool call.**

Two facts settle it:

- `claude_cli` runs the claude binary as a **subprocess** —
  `Command::new(&self.binary_path)` … `.spawn()` in
  `crates/zeroclaw-providers/src/claude_cli.rs:345`. Every tool the mission
  agent runs happens inside that child process.
- `net_guard`'s only call sites upstream are `link_enricher.rs`,
  `helpers/domain_guard.rs` and `plugins/egress.rs` — ZeroClaw's own Rust HTTP
  paths.

A mission agent's `curl` is spawned by claude, not by ZeroClaw, so it never
touches the guarded stack. Pulling the commit would harden the **chat** tier's
link previews and native web tools. It would leave mission egress exactly as it
is. Worth doing on its own merits; not worth doing under the belief that it
closes this.

## The topology

Mission containers are attached to two docker networks
(`mission_runtime.rs:327`):

| network           | `internal` | subnet         | what it is for                |
|-------------------|-----------|----------------|-------------------------------|
| `clawmates_core`  | **true**  | 172.20.0.0/16  | server, database, skills door |
| `clawmates_edge`  | false     | 172.23.0.0/16  | outbound provider egress      |

`core` has no default route at all — confirmed from a container on it, where
every external address is unreachable and even the positive control fails.
`edge` supplies the default route, and with it everything below.

## The measurement

A throwaway `alpine:3.20` container attached to **both** networks, exactly as a
mission container is. Controls in both directions, because a probe whose
positive control fails proves nothing — the first run of this probe was
`core`-only, reported "no internet", and was discarded for that reason.

```
--- routes ---
default via 172.23.0.1 dev eth1
172.20.0.0/16 dev eth0 scope link  src 172.20.0.6
172.23.0.0/16 dev eth1 scope link  src 172.23.0.5

positive control  1.1.1.1:443  raw IP              REACHABLE   (expected)
positive control  arxiv.org:443 DNS+connect        REACHABLE   (expected)
negative control  192.0.2.1:80  TEST-NET           blocked     (expected)

tailnet     gw-02 100.84.218.70:22                 REACHABLE   <--
host SSH    docker gateway 172.23.0.1:22           REACHABLE   <--
link-local  169.254.169.254:80                     REACHABLE   <--
database    clawmates_postgres_1:5432              REACHABLE   (authenticated)
internet    unrestricted                           REACHABLE
LAN         192.168.1.1:80                         blocked
```

`architect:8090` read blocked only because that host has been offline 15 days;
it is not evidence of a control.

## What this means

A mission agent reaches the **entire tailnet** and **SSH on its own host**. The
container's default route is the docker gateway, the host runs tailscale, and
NAT forwards the rest. Nothing between the agent and 100.64.0.0/10.

That matters more here than it would for ordinary software, because a mission
agent runs **model-generated shell commands over content it fetched from the
open web**. Two production missions made 158 `Bash` calls, 151 of them
`curl`/`wget` against 12+ hosts, and delegated 12 more fetches to subagents. The
instruction stream and the data stream are the same stream.

Related: gw-01/02/04 are key-only with fail2ban,
but the hosts have no firewall, and gw-02's databases are bound to localhost
only. Reachability is not compromise. It is the precondition for it.

### What is *not* exposed

Stated because a report that lists only the bad half is not a measurement:

- **Postgres requires a password over TCP.** Attempted from the mission network
  as both `postgres` and `clawmates`: `fe_sendauth: no password supplied`.
- **No database credentials are forwarded into mission containers.** The env is
  `ZEROCLAW_GATEWAY_PORT`, `CM_MISSION_ID`, `GIT_CONFIG_*` and provider keys —
  nothing else (`mission_runtime.rs:735`).
- **The LAN is not reachable**, only the tailnet.
- `169.254.169.254` is link-local on bare metal, not a cloud metadata service.
  Reachable, but there is no credential endpoint behind it on gw-04.

## Remediation — described, deliberately NOT applied (superseded — see the end)

The operator's call was to measure and report. The smallest change that closes
the lateral path, for whenever that decision is made:

```sh
# exempt first: missions legitimately need the core network (skills door, API)
iptables -I DOCKER-USER -s 172.23.0.0/16 -d 172.20.0.0/16 -j ACCEPT
# then deny the private world
iptables -I DOCKER-USER -s 172.23.0.0/16 -d 100.64.0.0/10  -j DROP   # tailnet
iptables -I DOCKER-USER -s 172.23.0.0/16 -d 169.254.0.0/16 -j DROP   # link-local
iptables -I DOCKER-USER -s 172.23.0.0/16 -d 10.0.0.0/8     -j DROP
iptables -I DOCKER-USER -s 172.23.0.0/16 -d 192.168.0.0/16 -j DROP
```

Public egress is untouched, so research missions keep working — which is the
constraint that rules out a host allow-list as the first move. `JEPA Research`
alone fetched arxiv, api.github.com, raw.githubusercontent.com, arrow.apache.org,
pytables, clawpack, h5py, lancedb, netcdf4, paperswithcode and more. No
pre-approved list would have contained them, and a mission that cannot read
cannot do research.

Order matters (`-I` prepends, so the ACCEPT must be inserted last to sit first),
the rules are not persistent across reboot as written, and they should be
verified with the same positive/negative control pair used above rather than
assumed.

## One fix that was applied

`mission_runtime.rs` discarded the result of the edge-network attach:

```rust
let _ = self.docker.connect_network(EDGE_NETWORK, …).await;
```

`core` is internal, so a failed attach leaves a mission with **no egress at
all** — no provider call, no fetch — while the launch reports success and the
phase can still complete. The green-with-nothing shape this codebase keeps
meeting.

Now: on error, the container's own network list decides. Already attached is
benign and logged; genuinely not attached fails the launch with a message that
says what it means. The container's networks are the fact, not the return code.

## Limits of this measurement

- One host (gw-04), one moment. Other deployments may differ.
- Reachability of a **port**, not exploitability of a service.
- `architect` and the fleet nodes were offline, so they were not probed.
- The probe container ran `alpine`, not `clawmates-runtime`. Same two networks
  and the same route table; a different image cannot have more access.

## Applied — what actually worked (2026-09-18)

Everything measured above was re-measured first, from a container on the
mission egress network, and was still true: tank's SSH, web-01's registry,
this host's SSH, link-local — all open. Then the rules above were applied, and
the positive/negative control pair this document asked for found three things
the rules could not have known.

### 1. Missions shared a subnet with the server

Missions egressed from `clawmates_edge` (172.23/16). So does
`clawmates_server_1`, and the server **needs** the tailnet: the Beszel hub on
architect (`:8090`), Ollama for the local-model backend (`:11434`), and the
node daemons (`:8088`) for exec-test and node-placed terminals. The tailnet
drop scoped to 172.23/16 cut the server off from architect:8090 within the
minute it was applied. No rule on that subnet could ever be right for both.

**Fix:** missions egress from a network of their own, `clawmates_missions`,
pinned at **172.25.0.0/16** (`mission_runtime.rs` `EDGE_NETWORK`, commit
`869c3ad`; declared in both compose files). `core` is unchanged — the skills
door and the API are still reached over 172.20. Compose v1 will not create a
network no service uses, so on gw-04 it was created by hand with compose's own
labels (`com.docker.compose.network=missions`, `…project=clawmates`), which
compose then accepts as its own.

### 2. `DOCKER-USER` loses to Tailscale

`tailscaled` inserts `-j ts-forward` at the head of FORWARD on every restart,
above `-j DOCKER-USER`, and `ts-forward` ends in `-o tailscale0 -j ACCEPT`. A
tailnet drop in `DOCKER-USER` worked until `systemctl restart tailscaled`, and
then the tailnet was open again — verified by doing exactly that. Re-asserting
the rule from an `ExecStartPost` on `tailscaled` ran before Tailscale had
installed its chains and lost the same race. Link-local, which does not go
through `tailscale0`, stayed blocked throughout; that is what made the cause
legible.

### 3. `raw PREROUTING` cannot see connection state

Moving the drops to the raw table put them ahead of every chain Tailscale
touches — and ahead of conntrack. A drop on `-d 100.64.0.0/10` then matched
the server's **replies** to tailnet clients (tank's node daemon, the operator's
`curl`) and took the API off `100.102.112.85:8088` while every container
reported healthy. The public site kept serving, because Traefik reaches the
server over the Docker network.

### What holds

`/usr/local/sbin/clawmates-egress.sh` on gw-04, run by
`clawmates-egress.service` at boot and by `ExecStartPost` drop-ins on
`docker.service` and `tailscaled.service`. Idempotent (`-C` before `-I`). The
rules live in **`mangle PREROUTING` with `-m conntrack --ctstate NEW`**: after
conntrack, so replies pass; before FORWARD and INPUT, so ordering against
`ts-forward` is moot; and neither Docker nor Tailscale writes to that table.

```
missions (172.25/16): DROP NEW → 100.64/10, 10/8, 192.168/16, 169.254/16, tcp/22 anywhere
edge     (172.23/16): DROP NEW → 10/8, 192.168/16, 169.254/16, tcp/22 anywhere   (tailnet allowed)
```

Verified three ways, then verified again after restarting both daemons:

| from | tailnet | host ssh | link-local | public | core door |
|---|---|---|---|---|---|
| mission subnet (throwaway container) | blocked | blocked | blocked | open | 200 |
| **inside a real mission container** (`01a0b550`) | blocked | blocked | blocked | open | 200 |
| edge (the server) | **open** | blocked | blocked | open | — |

Inbound to the API on the tailnet port: 200 from the host and from tank. The
mission itself ran normally behind the policy — 19 `curl` fetches, 2 skill
reads through the door, judge `met=true`, independent.

### Limits

- gw-04 only. The laptop has the network (`clawmates_missions`, same subnet)
  and no firewall; a local mission can still reach the LAN and tailnet.
- Reachability of a port, as before. A mission can still `curl` any public host.
- The microVM tier has its own egress model (per-backend rootfs, vsock) and is
  not covered by any of this.