fix(missions): egress from a subnet of their own, so the host can police it
deploy / test (push) Successful in 5m39s
deploy / build (push) Successful in 7m28s

docs/MISSION-EGRESS.md measured that a mission container reaches the entire
tailnet and SSH on its own host, and left the remediation unapplied. Applying
it on 2026-09-18 found why five iptables lines were never going to be enough:
missions egressed from clawmates_edge, the SERVER's network, and the server
needs the tailnet — Beszel on architect, Ollama for the local backend, the
node daemons for exec-test and node-placed terminals. A tailnet drop scoped to
172.23/16 cut the server off from architect:8090 inside a minute.

Missions now egress from clawmates_missions, 172.25.0.0/16, pinned so the
firewall can name it and declared in both compose files with the same shape
edge has. Compose v1 does not create a network no service uses, so on gw-04
it was created by hand with compose's own labels; the server's attach failure
message now says to check for it. core is unchanged: the door and API are
still reached over 172.20.

The policy itself (/usr/local/sbin/clawmates-egress.sh on gw-04, systemd unit
+ drop-ins on docker and tailscaled) lives in mangle/PREROUTING with
--ctstate NEW. Two earlier placements failed measurably: filter/FORWARD loses
to tailscaled re-inserting ts-forward above it on every restart, and
raw/PREROUTING runs before conntrack, so it dropped the server's replies to
tailnet clients and took the API off 100.102.112.85:8088. Verified from the
mission subnet (tailnet, host ssh, link-local blocked; public and core open),
from edge (tailnet open, ssh blocked), and inbound from tank; and proved to
survive restarting both daemons.

Also: deploy/compose/docker-compose.override.yml is tracked now. It holds the
fixes for the five local bring-up gaps and every credential in it is a
${VAR:?} reference, and it had lived on one laptop that lost a volume this
week.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
This commit is contained in:
Omar Sobh
2026-09-18 11:06:03 -05:00
co-authored by Claude Opus 5
parent 1678452a93
commit 869c3adcb7
4 changed files with 205 additions and 6 deletions
+3 -1
View File
@@ -28,4 +28,6 @@ deploy/compose/.env.*
# runtime at MacBook-specific paths. docker-compose picks this file up
# automatically, so committing it would silently reconfigure anyone who runs
# deploy/compose.
deploy/compose/docker-compose.override.yml
# deploy/compose/docker-compose.override.yml is TRACKED as of 2026-09-18: it
# holds the fixes for the five local bring-up gaps and every credential in it is
# a ${VAR:?} reference into .env. It lived only on one laptop until then.
+19 -5
View File
@@ -323,11 +323,24 @@ pub fn runtime_auth_mode() -> RuntimeAuth {
}
}
/// Docker networks the runtime container must be attached to.
/// - `clawmates_core`: talks to the server + database
/// - `clawmates_edge`: has egress for outbound provider calls
/// Docker networks the mission runtime container must be attached to.
/// - `clawmates_core`: talks to the server (skills door, API) + database
/// - `clawmates_missions`: has egress for outbound provider calls and fetches
///
/// Missions used to egress from `clawmates_edge`, the same network as the
/// SERVER. That made the host's egress policy impossible to scope: a mission
/// agent runs model-generated shell over content it fetched from the open web
/// and must not reach the tailnet, but the server on the same subnet must —
/// Beszel, Ollama, the node daemons. Measured 2026-09-18: the tailnet rule
/// written for missions cut the server off from architect:8090 within the
/// minute. A subnet of their own is what lets `clawmates-egress.sh` on gw-04
/// drop tailnet/private/link-local/ssh for missions and nothing else.
///
/// The network is declared by compose (prod: /opt/clawmates/docker-compose.yml;
/// local: deploy/compose/docker-compose.yml) with the same shape `edge` has —
/// not internal, so it routes off the host.
const CORE_NETWORK: &str = "clawmates_core";
const EDGE_NETWORK: &str = "clawmates_edge";
const EDGE_NETWORK: &str = "clawmates_missions";
/// Host path (as seen by the docker engine, NOT the server container)
/// where the mission's checkouts live. Matches the mount source used
@@ -832,7 +845,8 @@ impl MissionRuntimeProvisioner {
{CORE_NETWORK} is internal and has no route off the host, so \
this mission would run with no egress at all: every provider \
call and every fetch would fail while the phase still \
reported completion"
reported completion. Is the `missions` network declared in \
the compose file and created (`docker network ls`)?"
));
}
}
+174
View File
@@ -0,0 +1,174 @@
# LOCAL-ONLY override for the MacBook stack (2026-08-12). NOT for prod —
# gw-04 gets this wiring from its host .env and a hand-managed runtime.
#
# Why this file exists: docker-compose.yml deliberately leaves the zeroclaw
# runtime "separately managed" (see its volumes comment), so a fresh local stack
# comes up with NO runtime and every mission dies on "ZEROCLAW_GATEWAY_URL not
# set". This override closes that gap so `docker compose up -d` is enough.
services:
# The agent runtime. Built locally for arm64:
# cd ~/projects/zeroclaw && docker build --platform linux/arm64 \
# -f ~/projects/clawmates/deploy/clawmates-runtime/Dockerfile \
# -t clawmates-runtime:sync .
# (The web-01 registry copy is amd64-only and would run under emulation.)
runtime:
# :toolchain = :sync + cmake + build-essential, via
# images/runtime-toolchain.Dockerfile. Without them `cargo test` on a repo
# whose deps drive a cmake build (clawhdf5 → libz-ng-sys) exits 101 in 13
# seconds, and the delivery gate reads that as a RED SUITE rather than as a
# missing toolchain. The permanent fix is in
# deploy/clawmates-runtime/Dockerfile; this tag is what lets the laptop run
# today without recompiling zeroclaw from the fork.
image: clawmates-runtime:toolchain
# Pinned: the server addresses the shared runtime by this exact name via
# CLAWMATES_RUNTIME_CONTAINER, whose default is `clawmates-runtime`.
container_name: clawmates-runtime
command: ["daemon"]
restart: unless-stopped
# `edge` (not internal) because the runtime needs egress to Anthropic.
networks: [edge]
volumes:
- runtime_data:/zeroclaw-data
# Same host path on both sides so the runtime sees the trees the server
# wrote — the reason docker-compose.yml bind-mounts rather than uses a volume.
- /var/lib/clawmates-missions:/var/lib/clawmates-missions
environment:
ZEROCLAW_GATEWAY_PORT: "42617"
# The daemon defaults to 127.0.0.1, which is unreachable from the server
# container and surfaces only as "ws connect failed: Connection refused".
# No host port is published, so this stays on the Docker bridge, and the
# bearer token still gates every request.
ZEROCLAW_gateway__host: "0.0.0.0"
ZEROCLAW_gateway__allow_public_bind: "true"
ZEROCLAW_WORKSPACE: /zeroclaw-data/workspace
# MUST be env, not a TOML sub-table: `[providers.models.claude_cli.default]`
# in config.toml parses to an EMPTY entry (the documented resolve_default_model
# gotcha in deploy/clawmates-runtime/README.md).
# MODEL POLICY (operator, 2026-08-16): haiku ONLY for yes/no questions;
# anything requiring thinking is opus-5; CODING is sonnet-5.
#
# This value is what the mission AGENTS run on, and it was `haiku`.
# `provider_alias_for` maps every `claude-*` binding to the single alias
# `claude_cli.default`, so a crew whose `model_binding` reads
# `claude-sonnet-5` still ran haiku — the binding is cosmetic and this
# line is the truth. Measured consequence on mission 01a00bbb: the coding
# agents CLAIMED six INT items and delivered three, and the judge caught
# the discrepancy against git history.
ZEROCLAW_providers__models__claude_cli__default__model: claude-sonnet-5
# The subscription. claude_cli spawns `claude -p` authed by this.
CLAUDE_CODE_OAUTH_TOKEN: ${ANTHROPIC_OAUTH_TOKEN:?set in .env}
frontend:
# Bind the published port to loopback ONLY. The base compose publishes
# "3000:3000", i.e. 0.0.0.0 — which on a tailnet machine means every peer
# can reach the app over plain HTTP at http://quantum:3000, bypassing the
# HTTPS front door entirely. `tailscale serve` proxies from 127.0.0.1, so
# loopback is all it needs.
ports: !override
- "127.0.0.1:3000:3000"
environment:
# LOCAL-ONLY: skip the login form and land on the dashboard as this user.
# Both vars are required — the /auth/autologin route 404s without them,
# so prod (which sets neither) is unaffected. This performs a REAL login
# against the backend; it does not weaken API auth.
LOCAL_AUTOLOGIN_EMAIL: ${CLAWMATES_BOOTSTRAP_OWNER_EMAIL:?set in .env}
LOCAL_AUTOLOGIN_PASSWORD: ${CLAWMATES_BOOTSTRAP_OWNER_PASSWORD:?set in .env}
server:
volumes:
# Shelf for harvested PDFs. Separate from /var/lib/clawmates-missions on
# purpose: that tree is SWEPT, and a paper collected there would be
# deleted out from under its own catalogue note.
- blobs:/var/lib/clawmates-blobs
# Same reasoning as the frontend: the API does not need to be on 0.0.0.0.
ports: !override
- "127.0.0.1:8080:8080"
environment:
ZEROCLAW_GATEWAY_URL: http://clawmates-runtime:42617
# Durable bearer token: pair ONCE, then runs are repeatable.
# code=$(docker logs clawmates-runtime 2>&1 | grep -oE '[0-9]{6}' | head -1)
# docker exec clawmates-runtime curl -s -X POST \
# http://127.0.0.1:42617/pair -H "X-Pairing-Code: $code"
ZEROCLAW_TOKEN: ${ZEROCLAW_TOKEN:?pair the runtime and set this in .env}
# brain_seed defaults to /data/brains, which the nonroot (65532) server
# cannot create. Park it under the missions root, which is chowned to 65532.
CLAWMATES_BRAIN_DIR: /var/lib/clawmates-missions/_brains
# Ambient forge credential for git.redclaw.dev clones (mission_workspace
# FORGE_HOST). Separate from the repo *connection* token in the broker:
# the connection lists repos for the UI, this one lets a mission actually
# clone one. Without it a private clone hangs on git's /dev/tty prompt.
GITEA_TOKEN: ${GITEA_TOKEN:?set in .env}
# Level-up defaults to `glm:glm-4.7` (level_up.rs DEFAULT_MODEL), which is
# right on gw-04 but has no credential here — the call 404s with
# "model: glm:glm-4.7" and the button 500s. A BARE model name routes to the
# Claude subscription this stack already authenticates with.
# Level-up proposes new skills for an agent — composition, not
# classification, so it takes the thinking tier.
CLAWMATES_LEVEL_UP_MODEL: claude-opus-5
# The done_when judge. It reads a phase's evidence, audits it against git
# history and writes both a reason and guidance for the next pass — the
# opposite of a yes/no lookup, whatever the shape of its final verdict.
# It is also the ONE component whose failure mode is silently passing work
# that was never done, which is the failure this project keeps paying for.
CLAWMATES_EVALUATOR_SUBSCRIPTION_MODEL: claude-opus-5
# The independent judge. `evaluator.rs` prefers a CROSS-PROVIDER judge and
# refuses to call a same-family one `independent`; with this set, an
# Anthropic implementer is graded by GLM instead of by itself.
#
# glm-5.3 is the newest z.ai publishes (4.5, 4.5-air, 4.6, 4.7, 5,
# 5-turbo, 5.1, 5.2, 5.3 as of 2026-08-17). gw-04 still runs glm-4.7.
# Measured on a realistic phase-evidence prompt: 5.3 emits a `thinking`
# block before its JSON (819 output tokens against the evaluator's 1024
# budget), and our SSE parser ignores `thinking_delta` and keeps the text,
# so the shape is compatible — but the headroom is why the evaluator's
# max_tokens was raised alongside this.
CLAWMATES_VALIDATOR_MODEL: glm:glm-5.3
CLAWMATES_JUDGE_MODEL: glm:glm-5.3
# Read by the `glm` provider's api_key_env in clawmates.toml.
ZAI_API_KEY: ${ZAI_API_KEY:?set in .env}
# ElevenLabs GenFM, for rendering the Continuous Research episode.
#
# SERVER-SIDE ONLY, deliberately. It is NOT in
# `mission_runtime::forwarded_provider_keys`, so it never reaches an agent
# container: the audio is rendered from the script the agents committed,
# by the server, after the phase is done. An agent needs the key for
# nothing, and a key that never enters a container cannot leak from one.
ELEVENLABS_API_KEY: ${ELEVENLABS_API_KEY:?set in .env}
# The origin a PODCAST APP will fetch from. Not localhost: the feed is
# opened by a phone, not by the browser that created it, and a feed that
# only resolves on this machine silently never syncs. The tailnet name is
# reachable from every device already on the tailnet.
CLAWMATES_PUBLIC_URL: ${CLAWMATES_PUBLIC_URL:-https://quantum.taila4f562.ts.net:8443}
# Per-mission runtime containers are seeded from this dir. The built-in
# default is /root/clawmates-runtime/data, which cannot exist here: macOS
# has no /root and Docker Desktop refuses to share it ("path is not shared
# from the host"). Without the override every mission silently falls back
# to the SHARED runtime — which is never workspace-pinned, so agents run in
# their own sandboxes and deliver nothing.
# Blob storage root. `storage.data_dir` defaults to "./data" and the
# container's cwd is `/`, so the server tried to create `/data` as uid
# 65532 and every shelve failed with "storage io: Permission denied".
# It surfaced honestly — the harvest reported `0 shelved, 5 failed` with
# the reason per paper — but nothing could ever be stored.
CLAWMATES_STORAGE__DATA_DIR: /var/lib/clawmates-blobs
CLAWMATES_RUNTIME_SEED_DIR: /var/lib/clawmates-runtime-seed
# PER-MISSION containers must carry the same toolchain as the shared one:
# the agents build the user's repo inside them. The built-in default is
# `clawmates-runtime:sync` (mission_runtime.rs DEFAULT_IMAGE), which has
# no cmake, so a coding phase could not compile clawhdf5 at all.
CLAWMATES_RUNTIME_IMAGE: clawmates-runtime:hooks
# Per-mission containers default to forwarding ANTHROPIC_API_KEY, which is
# dead here ("credit balance is too low"). Claude Code ranks that key ABOVE
# the subscription token, so forwarding it doesn't just fail — it WINS over
# a working credential. Subscription mode forwards CLAUDE_CODE_OAUTH_TOKEN
# instead, which is why the token is also set under that exact name below.
CLAWMATES_RUNTIME_AUTH: subscription
CLAUDE_CODE_OAUTH_TOKEN: ${ANTHROPIC_OAUTH_TOKEN:?set in .env}
volumes:
blobs: {}
runtime_data:
# Pre-existing volume from the manual bring-up; keeps the paired token and
# the risk_profiles/onboard_state already written into config.toml.
name: clawmates-runtime-data
external: true
+9
View File
@@ -11,6 +11,15 @@ name: clawmates
networks:
edge: {}
# Mission containers egress from here, NOT from edge, so the host's egress
# policy (tailnet/private/link-local/ssh dropped for missions, allowed for
# the server) can be scoped by subnet. See mission_runtime.rs EDGE_NETWORK.
missions:
# Pinned so the host firewall can name the subnet. 172.25/16 is free on
# gw-04 and the laptop; docker's auto-assignment is not stable across hosts.
ipam:
config:
- subnet: 172.25.0.0/16
core:
internal: true
sandbox_net: