clawmates/docs/agent-engine-architecture.md

Agent Engine Architecture — per-tenant containerized runtime

Status: accepted direction (Phase 0). Supersedes the assumption that Clawmates' own cm-runtime is the long-term execution engine. This document is the source of truth for the backend agent-execution strategy; implementation proceeds in the phases at the end.

Problem

Clawmates must let each customer build their own multi-agent "company/team" and run it at scale: many agents, across machines, with full lifecycle management (provision → run → suspend → resume → retire) — without weakening the §15 safety contract (per-tool human approval, taint, secret broker, single audited journal) that is the product's moat and is acceptance-blocking.

Clawmates' current cm-runtime (crates/cm-runtime/src/runtime.rs) is a real agent loop, but it is single-host: one server process drives all agents. It does not scale horizontally or isolate tenants at the infrastructure layer.

Options considered

Engine	Verdict
clawverse (Rust, distributed mission orchestrator)	Closest to "distributed agent fleet," but mission/repo-shaped, partly WIP, and re-implements node scheduling/allocation/health that Kubernetes already provides. Shelved — revisit only if one tenant must span many nodes/GPUs, or for its mission/self-improvement IP.
openclaw (Node agent OS)	Most complete single-host agent OS, but Node (our stack is Rust), single-tenant by design, and we lack confirmed source + license to fork/ship it. Not the base.
zeroclaw (~/projects/zeroclaw, Rust, MIT/Apache)	Multi-agent-per-daemon runtime; ownable, single-binary, embeddable, ~80 providers, 6-layer sandbox, gateway/RPC drive API. Chosen as the per-tenant runtime.

Decision

Run one containerized ZeroClaw daemon per workspace/tenant. The unit of scaling becomes the tenant container, placed and lifecycle-managed by a standard orchestrator (Kubernetes or Fly Machines) — which replaces a bespoke node/allocation layer. Clawmates remains the control plane: tenancy, auth, billing, the audited SSE gateway, and the §15 safety engine. The tenant's agents reach the outside world only through Clawmates-gated capabilities.

ZeroClaw v0.8.0 makes this viable nearly out of the box:

• Multi-agent per daemon — isolated workspace/memory/model/policy/persona per agent → one container = one tenant's whole team.
• RPC transport (local socket / remote WSS + token) with restart-surviving, tmux-style sessions → clean drive channel and clean scale-to-zero/resume.
• Schema-V3 config CRUD over the gateway (/api/config/*) → declarative provisioning.
• Per-agent tool allowlists + MCP client + private-host allowlists → the hook to make Clawmates the only outbound door.
• Attribution-aware structured logs + per-agent/per-model cost tracking → feed the audited journal and billing.
• Lean channel bundle as Cargo features (crates/zeroclaw-channels/Cargo.toml: default-channels = acp-server, email, telegram, webhook) → tiny, fast-cold-start image.
• MIT/Apache license → we may fork, containerize, and ship it.

Existing infrastructure & what we reuse (this is already proven here)

The per-tenant ZeroClaw-in-a-container model is already running in production in this fleet (ClawBooks, EasyA, gr33t). Clawmates reuses that proven stack rather than introducing K8s/Fly:

• Substrate: Docker + Nginx/Traefik + a per-app orchestrator on Hetzner VMs (Tailscale mesh). The orchestrator (:3500) provisions/deprovisions a container per user; a webhook bridge (:3501) reacts to Clerk user.created; Nginx routes /ws/agent/{userId} → the container.
• Per-user container: ZeroClaw binary + a small domain sidecar + plugins/skills.
• Per-user volume: /data/users/{id}/ = config.toml, domain.db (SQLite), brain.h5 (ZeroClaw's HDF5/EdgeHDF5 memory) , workspace/, channels/. Backup = 2 portable files (domain.db + brain.h5) rsynced offsite — trivial tenant move/restore.
• Image registry: private registry at web-01:5000 (Tailscale-only).
• Auth + billing: Clerk + Stripe (Clawmates already supports Clerk auth mode).
• Bootstrap scripts (reuse): redclawsystems/zeroclaw → deploy/clawbooks/bootstrap/ (scale-up.sh = Hetzner provision + bootstrap + deploy in ~5 min; bootstrap-node.sh; deploy-to-node.sh).
• Secrets: Infisical at icarus.lan:8443 (HETZNER_API_TOKEN_RW/RO, ANTHROPIC/OPENAI/…, Clerk, Stripe, Cloudflare, Tailscale). ⚠️ Some are still plaintext in the vault and flagged for rotation — rotate before production.

Resources / deploy target (live check)

The 6-server Hetzner fleet (Helsinki) has ample headroom; no new server is needed. Dev/test target: gw-05 (8c / 15Gi, ~188G free, lightly loaded) — or gw-01 (~234G free). Estimated capacity ~50–100 concurrent per-tenant containers across the fleet without new hardware.

Target architecture

┌──────────────────────────────────────────────────────────────────────┐
│ Clawmates control plane (Rust SaaS) — responsibilities UNCHANGED       │
│  • tenancy, auth, billing, product UI, the single audited SSE gateway  │
│  • §15 engine: approvals · taint · gated categories · single-use       │
│    grants · secret broker            (cm-safety, cm-secrets, cm-api)    │
│  • Clawmates MCP server: the ONLY outbound door (email/slack/pay/…)     │
│  • tenant-runtime manager: provision/suspend/resume one container/tenant│
└───────────────┬────────────────────────────────────────────────────────┘
                │ (a) provision agents via /api/config
                │ (b) run turns via sessions/prompt (SSE) or RPC
                │ (c) pull cost + logs
                │ (d) agent's only egress = our gated MCP tools
                ▼
┌──────────────────────────────────────────────────────────────────────┐
│ Per-tenant ZeroClaw daemon (slim container, our fork)                  │
│  • N named agents (= N clawmates): per-agent workspace/memory/model    │
│  • NO native outbound channels/secrets; tools.allow locked            │
│  • MCP client → Clawmates MCP server for every sensitive capability    │
└───────────────┬────────────────────────────────────────────────────────┘
                │ spawns sandboxes for shell/browser
                ▼
   Per-agent sandboxes — sibling pods / gVisor, brokered via socket-proxy
                ▲
┌──────────────────────────────────────────────────────────────────────┐
│ Substrate: Hetzner VMs (Tailscale) · Docker · Nginx/Traefik ·          │
│  per-app orchestrator (:3500) + Clerk webhook bridge (:3501)           │
│  — the EXISTING proven ClawBooks/EasyA stack (replaces clawverse).     │
│  K8s/Fly only if scale later demands it.                               │
└──────────────────────────────────────────────────────────────────────┘

Concept mapping

Clawmates	ZeroClaw
workspace / tenant	one daemon (one container)
claw (agent)	one named agent (isolated workspace/memory/model/policy)
chat session / turn	POST /api/sessions/new + POST /api/session/{id}/prompt (SSE), or an RPC session
the claw's "computer" / files	/api/agents/{alias}/workspace/* + a sibling sandbox for shell/browser
routine	a zeroclaw cron job bound to that agent
credits / billing	/api/cost (per-agent / per-model splits, cached-input tokens)
audit journal (run_events)	gateway attribution-aware logs, mirrored into run_events
§15 outbound action	a Clawmates MCP tool the agent calls — we gate + broker + journal

§15 — the safety contract stays in Clawmates

Two independent layers:

1. No native edges. Each tenant agent is provisioned with tools.allow excluding native outbound/dangerous tools and with no real channel or secret config. It physically cannot email, post, pay, or exfiltrate on its own.

2. One gated door. Clawmates exposes an MCP server of the sensitive capabilities (send email, post to Slack, move money, delete data, change access, grant infra access). When an agent invokes one:

   • the call lands in Clawmates → classify Effect + TaintSource (reuse cm-tools policy),
   • if gated → create an approvals row, render a preview, suspend, await a human decision,
   • on approve → consume the single-use execution_grants row, broker-execute with the secret (cm-secrets, never exposing the credential), and journal to run_events.

   This reuses the existing approval state machine (cm-safety), secret broker (cm-secrets), and the audited gateway (cm-api) verbatim. Defense-in-depth: run agents at ZeroClaw's Supervised risk profile so its own per-dispatch allowlist + ask_operator back-stop us.

The §15 properties (taint-by-default, gated categories, single audited channel, single-use grants, broker isolation) are therefore unchanged — they move from gating cm-runtime's in-process tools to gating the MCP boundary.

Slim per-tenant image

• Fork/vendor zeroclaw; build with only needed features (--no-default-features --features channel-webhook + MCP client), no native chat channels. Push to the private registry at web-01:5000.
• Gateway bound to an in-pod socket / WSS with a per-tenant token; Nginx routes /ws/agent/{userId} → the container (existing convention).
• Per-tenant persistent volume at /data/users/{id}/ (config.toml, domain.db, brain.h5, workspace/) — the proven 2-file (domain.db + brain.h5) backup/restore applies.
• Provider keys injected at start via env (ZEROCLAW_*) from Infisical / the Clawmates broker, not baked into the image.

Risks / open questions

• Idle economics — a container per user is costly; the existing orchestrator can stop idle containers and start them on next request (per-tenant volume persists; v0.8 restart-surviving sessions make resume clean). Tune idle TTL / wake latency; K8s+KEDA or Fly auto-stop only if we outgrow the orchestrator.
• Nested sandboxing — tool sandboxes inside a tenant container; prefer sibling sandbox pods via the existing socket-proxy pattern over privileged DinD; evaluate gVisor/sysbox.
• §15 coverage — must guarantee no un-gated path to the outside; audit zeroclaw's native tool set per build and lock tools.allow.
• Fork maintenance — zeroclaw is v0.8 beta, workspace publish=false; maintain a thin fork/vendor and track upstream.
• Provisioning contract — Schema-V3 config CRUD is the provisioning API; pin its behavior.

Phased delivery

• Phase 0 — this document. Decision + architecture + §15-via-MCP + risks. ✅
• Phase 1 — Spike (on gw-05). Build a slim zeroclaw container (push to web-01:5000); from a small Clawmates-side client, provision one agent via /api/config, run one turn via sessions/prompt, stream events back, and prove a §15 gate by routing one outbound action through a stub Clawmates MCP tool. Reuse the ClawBooks deploy/clawbooks/bootstrap/ scripts as the starting point.
• Phase 2 — Provisioning + lifecycle. Adapt the existing orchestrator (:3500) + Clerk webhook (:3501) + Nginx pattern to provision/stop/resume a Clawmates tenant container + per-claw agent on workspace/claw create.
• Phase 3 — §15 MCP server + journal/cost bridge. Real gated MCP capabilities (cm-safety / cm-secrets); map zeroclaw attribution logs + /api/cost into run_events / billing.
• Phase 4 — Idle stop/resume + per-agent sandboxes (sibling containers via socket-proxy).
• Phase 5 — Cutover from cm-runtime to the containerized runtime behind a feature flag.

Reuse source-of-truth: redclawsystems/zeroclaw → deploy/clawbooks/bootstrap/ and the ClawBooks/EasyA orchestrator. Rotate the plaintext Hetzner/Cloudflare/Clerk tokens (vault Infrastructure Overview.md) into Infisical before production.

Verification

• Phase 1: end-to-end on a dev box — one claw answers a turn on a containerized zeroclaw via Clawmates, events journaled, one approval gate fires through the Clawmates MCP door.
• Keep Clawmates' E2E + §15 suites (p0–p7, approval/secret-broker tests) green against the new backend behind the flag before cutover.
• Confirm slim-image cold-start and resume-from-sleep preserve sessions.

References

• Clawmates runtime + safety: crates/cm-runtime/src/runtime.rs, crates/cm-safety/, crates/cm-secrets/, crates/cm-api/, crates/cm-tools/, migrations/0001_init.sql, spec docs/spec.md §15.
• ZeroClaw: ~/projects/zeroclaw — crates/zeroclaw-channels/Cargo.toml (channel features), crates/zeroclaw-gateway/src (drive API: config / sessions / workspace / cost / cron), multi-agent runtime + RPC transport (v0.8.0).
• Infra & proven pattern (Obsidian vault ~/projects/Valhalla): 20 Infrastructure/Infrastructure Overview.md, 20 Infrastructure/Tailscale Network Map.md, 20 Infrastructure/10 Architecture/ SaaS Platform Template.md, 20 Infrastructure/30 Runbooks/Credentials Inventory.md.
• Reuse: redclawsystems/zeroclaw → deploy/clawbooks/bootstrap/; private registry web-01:5000; Infisical icarus.lan:8443; dev/test box gw-05.