clawmates/docs/agent-engine-architecture.md

# Agent Engine Architecture — per-tenant containerized runtime

_Status: accepted direction (Phase 0). Supersedes the assumption that Clawmates' own
`cm-runtime` is the long-term execution engine. This document is the source of truth for the
backend agent-execution strategy; implementation proceeds in the phases at the end._

## Problem

Clawmates must let each customer build their own multi-agent "company/team" and run it at
scale: many agents, across machines, with full lifecycle management (provision → run →
suspend → resume → retire) — without weakening the §15 safety contract (per-tool human
approval, taint, secret broker, single audited journal) that is the product's moat and is
acceptance-blocking.

Clawmates' current `cm-runtime` (`crates/cm-runtime/src/runtime.rs`) is a real agent loop, but
it is single-host: one server process drives all agents. It does not scale horizontally or
isolate tenants at the infrastructure layer.

## Options considered

| Engine | Verdict |
|---|---|
| **clawverse** (Rust, distributed mission orchestrator) | Closest to "distributed agent fleet," but mission/repo-shaped, partly WIP, and re-implements node scheduling/allocation/health that Kubernetes already provides. **Shelved** — revisit only if one tenant must span many nodes/GPUs, or for its mission/self-improvement IP. |
| **openclaw** (Node agent OS) | Most complete *single-host* agent OS, but Node (our stack is Rust), single-tenant by design, and we lack confirmed source + license to fork/ship it. **Not the base.** |
| **zeroclaw** (`~/projects/zeroclaw`, Rust, MIT/Apache) | Multi-agent-per-daemon runtime; ownable, single-binary, embeddable, ~80 providers, 6-layer sandbox, gateway/RPC drive API. **Chosen as the per-tenant runtime.** |

## Decision

**Run one containerized ZeroClaw daemon per workspace/tenant.** The unit of scaling becomes the
*tenant container*, placed and lifecycle-managed by a standard orchestrator (Kubernetes or Fly
Machines) — which replaces a bespoke node/allocation layer. Clawmates remains the **control
plane**: tenancy, auth, billing, the audited SSE gateway, and the **§15 safety engine**. The
tenant's agents reach the outside world **only** through Clawmates-gated capabilities.

ZeroClaw **v0.8.0** makes this viable nearly out of the box:
- **Multi-agent per daemon** — isolated workspace/memory/model/policy/persona per agent → one
  container = one tenant's whole team.
- **RPC transport** (local socket / remote WSS + token) with **restart-surviving, tmux-style
  sessions** → clean drive channel and clean scale-to-zero/resume.
- **Schema-V3 config CRUD** over the gateway (`/api/config/*`) → declarative provisioning.
- **Per-agent tool allowlists + MCP client + private-host allowlists** → the hook to make
  Clawmates the only outbound door.
- **Attribution-aware structured logs** + **per-agent/per-model cost tracking** → feed the
  audited journal and billing.
- **Lean channel bundle as Cargo features** (`crates/zeroclaw-channels/Cargo.toml`:
  `default-channels = acp-server, email, telegram, webhook`) → tiny, fast-cold-start image.
- **MIT/Apache license** → we may fork, containerize, and ship it.

## Existing infrastructure & what we reuse (this is already proven here)

The per-tenant ZeroClaw-in-a-container model is **already running in production** in this fleet
(ClawBooks, EasyA, gr33t). Clawmates reuses that proven stack rather than introducing K8s/Fly:

- **Substrate:** Docker + Nginx/Traefik + a per-app **orchestrator** on Hetzner VMs (Tailscale mesh).
  The orchestrator (`:3500`) provisions/deprovisions a container per user; a webhook bridge (`:3501`)
  reacts to Clerk `user.created`; Nginx routes `/ws/agent/{userId}` → the container.
- **Per-user container:** ZeroClaw binary + a small domain sidecar + plugins/skills.
- **Per-user volume:** `/data/users/{id}/` = `config.toml`, `domain.db` (SQLite), `brain.h5`
  (ZeroClaw's HDF5/EdgeHDF5 memory) , `workspace/`, `channels/`. Backup = **2 portable files**
  (`domain.db` + `brain.h5`) rsynced offsite — trivial tenant move/restore.
- **Image registry:** private registry at `web-01:5000` (Tailscale-only).
- **Auth + billing:** Clerk + Stripe (Clawmates already supports Clerk auth mode).
- **Bootstrap scripts (reuse):** `redclawsystems/zeroclaw → deploy/clawbooks/bootstrap/`
  (`scale-up.sh` = Hetzner provision + bootstrap + deploy in ~5 min; `bootstrap-node.sh`;
  `deploy-to-node.sh`).
- **Secrets:** Infisical at `icarus.lan:8443` (`HETZNER_API_TOKEN_RW/RO`, `ANTHROPIC/OPENAI/…`,
  Clerk, Stripe, Cloudflare, Tailscale). ⚠️ Some are still plaintext in the vault and flagged for
  rotation — rotate before production.

### Resources / deploy target (live check)

The 6-server Hetzner fleet (Helsinki) has ample headroom; **no new server is needed**. Dev/test
target: **gw-05** (8c / 15Gi, ~188G free, lightly loaded) — or gw-01 (~234G free). Estimated
capacity ~50–100 concurrent per-tenant containers across the fleet without new hardware.

## Target architecture

```
┌──────────────────────────────────────────────────────────────────────┐
│ Clawmates control plane (Rust SaaS) — responsibilities UNCHANGED       │
│  • tenancy, auth, billing, product UI, the single audited SSE gateway  │
│  • §15 engine: approvals · taint · gated categories · single-use       │
│    grants · secret broker            (cm-safety, cm-secrets, cm-api)    │
│  • Clawmates MCP server: the ONLY outbound door (email/slack/pay/…)     │
│  • tenant-runtime manager: provision/suspend/resume one container/tenant│
└───────────────┬────────────────────────────────────────────────────────┘
                │ (a) provision agents via /api/config
                │ (b) run turns via sessions/prompt (SSE) or RPC
                │ (c) pull cost + logs
                │ (d) agent's only egress = our gated MCP tools
                ▼
┌──────────────────────────────────────────────────────────────────────┐
│ Per-tenant ZeroClaw daemon (slim container, our fork)                  │
│  • N named agents (= N clawmates): per-agent workspace/memory/model    │
│  • NO native outbound channels/secrets; tools.allow locked            │
│  • MCP client → Clawmates MCP server for every sensitive capability    │
└───────────────┬────────────────────────────────────────────────────────┘
                │ spawns sandboxes for shell/browser
                ▼
   Per-agent sandboxes — sibling pods / gVisor, brokered via socket-proxy
                ▲
┌──────────────────────────────────────────────────────────────────────┐
│ Substrate: Hetzner VMs (Tailscale) · Docker · Nginx/Traefik ·          │
│  per-app orchestrator (:3500) + Clerk webhook bridge (:3501)           │
│  — the EXISTING proven ClawBooks/EasyA stack (replaces clawverse).     │
│  K8s/Fly only if scale later demands it.                               │
└──────────────────────────────────────────────────────────────────────┘
```

## Concept mapping

| Clawmates | ZeroClaw |
|---|---|
| workspace / tenant | one daemon (one container) |
| claw (agent) | one named agent (isolated workspace/memory/model/policy) |
| chat session / turn | `POST /api/sessions/new` + `POST /api/session/{id}/prompt` (SSE), or an RPC session |
| the claw's "computer" / files | `/api/agents/{alias}/workspace/*` + a sibling sandbox for shell/browser |
| routine | a zeroclaw cron job bound to that agent |
| credits / billing | `/api/cost` (per-agent / per-model splits, cached-input tokens) |
| audit journal (`run_events`) | gateway attribution-aware logs, mirrored into `run_events` |
| §15 outbound action | a **Clawmates MCP tool** the agent calls — we gate + broker + journal |

## §15 — the safety contract stays in Clawmates

Two independent layers:

1. **No native edges.** Each tenant agent is provisioned with `tools.allow` excluding native
   outbound/dangerous tools and with **no real channel or secret config**. It physically cannot
   email, post, pay, or exfiltrate on its own.
2. **One gated door.** Clawmates exposes an **MCP server** of the sensitive capabilities (send
   email, post to Slack, move money, delete data, change access, grant infra access). When an
   agent invokes one:
   - the call lands in Clawmates → classify `Effect` + `TaintSource` (reuse `cm-tools` policy),
   - if gated → create an `approvals` row, render a preview, **suspend**, await a human decision,
   - on approve → consume the single-use `execution_grants` row, **broker-execute with the
     secret** (`cm-secrets`, never exposing the credential), and journal to `run_events`.

   This reuses the existing approval state machine (`cm-safety`), secret broker (`cm-secrets`),
   and the audited gateway (`cm-api`) verbatim. Defense-in-depth: run agents at ZeroClaw's
   **Supervised** risk profile so its own per-dispatch allowlist + `ask_operator` back-stop us.

The §15 properties (taint-by-default, gated categories, single audited channel, single-use
grants, broker isolation) are therefore **unchanged** — they move from gating `cm-runtime`'s
in-process tools to gating the MCP boundary.

## Slim per-tenant image

- Fork/vendor zeroclaw; build with only needed features
  (`--no-default-features --features channel-webhook` + MCP client), no native chat channels.
  Push to the private registry at `web-01:5000`.
- Gateway bound to an in-pod socket / WSS with a per-tenant token; Nginx routes
  `/ws/agent/{userId}` → the container (existing convention).
- Per-tenant **persistent volume** at `/data/users/{id}/` (`config.toml`, `domain.db`,
  `brain.h5`, `workspace/`) — the proven 2-file (`domain.db` + `brain.h5`) backup/restore applies.
- Provider keys injected at start via env (`ZEROCLAW_*`) from Infisical / the Clawmates broker,
  not baked into the image.

## Risks / open questions

- **Idle economics** — a container per user is costly; the existing orchestrator can **stop idle
  containers** and start them on next request (per-tenant volume persists; v0.8 restart-surviving
  sessions make resume clean). Tune idle TTL / wake latency; K8s+KEDA or Fly auto-stop only if we
  outgrow the orchestrator.
- **Nested sandboxing** — tool sandboxes inside a tenant container; prefer **sibling** sandbox
  pods via the existing `socket-proxy` pattern over privileged DinD; evaluate gVisor/sysbox.
- **§15 coverage** — must guarantee *no* un-gated path to the outside; audit zeroclaw's native
  tool set per build and lock `tools.allow`.
- **Fork maintenance** — zeroclaw is v0.8 **beta**, workspace `publish=false`; maintain a thin
  fork/vendor and track upstream.
- **Provisioning contract** — Schema-V3 config CRUD is the provisioning API; pin its behavior.

## Phased delivery

- **Phase 0 — this document.** Decision + architecture + §15-via-MCP + risks. ✅
- **Phase 1 — Spike (on gw-05).** Build a slim zeroclaw container (push to `web-01:5000`); from a
  small Clawmates-side client, provision one agent via `/api/config`, run one turn via
  `sessions/prompt`, stream events back, and prove a §15 gate by routing one outbound action
  through a stub Clawmates MCP tool. Reuse the ClawBooks `deploy/clawbooks/bootstrap/` scripts as
  the starting point.
- **Phase 2 — Provisioning + lifecycle.** Adapt the existing orchestrator (`:3500`) + Clerk
  webhook (`:3501`) + Nginx pattern to provision/stop/resume a Clawmates tenant container +
  per-claw agent on workspace/claw create.
- **Phase 3 — §15 MCP server + journal/cost bridge.** Real gated MCP capabilities (`cm-safety` /
  `cm-secrets`); map zeroclaw attribution logs + `/api/cost` into `run_events` / billing.
- **Phase 4 — Idle stop/resume + per-agent sandboxes (sibling containers via socket-proxy).**
- **Phase 5 — Cutover** from `cm-runtime` to the containerized runtime behind a feature flag.

> Reuse source-of-truth: `redclawsystems/zeroclaw → deploy/clawbooks/bootstrap/` and the
> ClawBooks/EasyA orchestrator. Rotate the plaintext Hetzner/Cloudflare/Clerk tokens (vault
> `Infrastructure Overview.md`) into Infisical before production.

## Verification

- Phase 1: end-to-end on a dev box — one claw answers a turn on a containerized zeroclaw via
  Clawmates, events journaled, one approval gate fires through the Clawmates MCP door.
- Keep Clawmates' E2E + §15 suites (`p0`–`p7`, approval/secret-broker tests) green against the
  new backend behind the flag before cutover.
- Confirm slim-image cold-start and resume-from-sleep preserve sessions.

## References

- Clawmates runtime + safety: `crates/cm-runtime/src/runtime.rs`, `crates/cm-safety/`,
  `crates/cm-secrets/`, `crates/cm-api/`, `crates/cm-tools/`, `migrations/0001_init.sql`,
  spec `docs/spec.md` §15.
- ZeroClaw: `~/projects/zeroclaw` — `crates/zeroclaw-channels/Cargo.toml` (channel features),
  `crates/zeroclaw-gateway/src` (drive API: config / sessions / workspace / cost / cron),
  multi-agent runtime + RPC transport (v0.8.0).
- Infra & proven pattern (Obsidian vault `~/projects/Valhalla`): `20 Infrastructure/Infrastructure
  Overview.md`, `20 Infrastructure/Tailscale Network Map.md`, `20 Infrastructure/10 Architecture/
  SaaS Platform Template.md`, `20 Infrastructure/30 Runbooks/Credentials Inventory.md`.
- Reuse: `redclawsystems/zeroclaw → deploy/clawbooks/bootstrap/`; private registry `web-01:5000`;
  Infisical `icarus.lan:8443`; dev/test box `gw-05`.