clawmates/README.md

# Clawmates

**Deploy agents at any scale — a single claw, a team, a company, or a whole org — and run a mission across
the organizational *topology* that fits it.**

Clawmates is a multi-agent platform where every unit of work is a **topology**: a graph of role-slots
bound to real AI agents ("claws"). The same model nests recursively — a team is a topology of claws, a
company is a topology of teams, an org is a topology of companies — so you compose and run agentic
systems from one agent up to an entire organization. The work itself runs as **missions**: recipes of
phases, staffed by team templates, executed in isolated containers or Firecracker microVMs, and checked
by an independent judge before anything is called done.

Live at **[clawmates.work](https://clawmates.work)**.

---

## Features

- **One dashboard.** The workspace home is a single live dashboard with a tier rail — **World, Agents,
  Missions, Repos, Infra, Podcast**. World is a live Gource-style graph of what agents are doing; the
  org → company → team structure is one expandable graph; each agent opens a slide-out "computer"
  (chat, terminal, files, brain).
- **Missions.** A mission is a **recipe** (`templates/workflows/*.toml`) of ordered phases — research,
  coding, security scan, benchmark — each with a task brief and a `done_when` completion condition, staffed
  by a **team template** (`templates/teams/*.toml`). Seven recipes ship: `research_only`,
  `research_and_code`, `security_hardening`, `refactor`, `benchmark`, `continuous_research`, `self_audit`.
- **Two execution tiers.** *Container tier*: each mission gets its own ZeroClaw runtime container on the
  gateway, running Claude Code (`claude_cli`) with fallback to Kimi and GLM. *MicroVM tier*: phases run
  in Firecracker microVMs on fleet nodes (`clawmates-node` + the `fcagent` guest), with per-backend
  egress (Claude, GLM, Kimi, or a local model over vsock).
- **An independent judge.** Every conditioned phase is judged. When a cross-provider judge is configured
  (prod: GLM, with Kimi as automatic fallback) it is a model from a *different* provider family than the
  agents; without one (the self-host default) the phase is judged by Claude and recorded as **not**
  independent. The judge runs its own allow-listed
  checks — tests, `rg`, git — against a copy of the work, commits to a verification plan before reading
  the evidence, and installs npm dependencies offline from the lockfile. A phase that fails is retried
  with the judge's guidance, up to `max_iterations`. Measured with `scripts/judge-eval.sh` (15 known-answer
  cases).
- **Judge quota watchdog.** Polls the GLM and Kimi usage APIs, warns at 80% of any window, and switches
  judging to the fallback at 95% (`GET /api/judge/quota`).
- **Delivery to git.** Each phase's work is committed and pushed to a mission branch on the repo's forge;
  `continuous_research` auto-merges additive-only changes into the vault.
- **Tool gates on every agent call.** A `PreToolUse` gate (both tiers) refuses destructive and exfiltrating
  commands, protects its own hook files, enforces per-role policy (e.g. a read-only verifier), and records
  task-permission and argument-provenance ("taint") violations in shadow mode. A stop gate sends a
  microVM agent back to work while its `done_when_check` fails, up to 3 times.
- **Skills.** A catalog of skills bound to team roles, delivered to mission agents through the MCP
  skills door; per-mission skill triage and skill-use measurement.
- **Project memory.** Each repository keeps a `.brain` (ClawhDF5) of every judge verdict; missions recall
  relevant past verdicts, and the `self_audit` recipe audits the whole record for failure patterns.
- **Continuous research + podcast.** Harvests new arXiv papers into an Obsidian vault, triages them,
  writes an analysis and a two-host script, and renders audio with ElevenLabs.
- **Decision tier (`cm-decide`).** Cheap calibrated classifiers (Jev) for gut-check decisions: the §15
  door governor (allow / deny / hold for approval), skill triage, paper triage, memory rerank.
- **Master Planner.** The "+" deploy is a chat that proposes and scaffolds a team for a goal
  (specialists, swarm, scheduled, triggered).
- **12 organizational topologies + evolution.** Hierarchical, pipeline, swarm, mesh, debate, hub-spoke,
  star-MoE, market, ring, flat, holacratic, blackboard; multi-topology comparison with a quality/cost
  **Pareto front**, and MAP-Elites evolution over (kind × team size).
- **Fleet.** Multi-user workspaces with quotas; a node pool (`clawmates-node` over Tailscale) with
  capacity-aware placement, drain, per-node tool versions and one-click updates, and Beszel metrics
  feeding a rules engine.
- **Agent-to-agent comms.** Chat rooms, a gated delegation bridge, per-claw door identity and A2A ingress.
- **Self-hostable.** A single-node Docker Compose deployment runs the whole platform.

---

## Architecture

A Rust workspace (the platform) + a Next.js app (the web UI).

### Backend — Rust workspace (`crates/`)

| Crate | Role |
|---|---|
| `cm-domain` | Shared types: ids, roles, workspaces, users |
| `cm-topology` | Topology data model: 12-kind taxonomy, graph, classifier, per-kind builders + heuristics |
| `cm-orchestrator` | Execution engine: async control-flow over a generic `TurnExecutor`; planners, comparison harness, MAP-Elites evolution |
| `cm-runtime` | The §15-safe per-tenant agent runtime and the LLM provider registry |
| `cm-brain` | Facade over the canonical `.brain` (ClawhDF5 brain-pack): one HDF5 file per agent or repo holding definition + memory |
| `cm-decide` | Typed, calibrated decisions (Jev classifiers) for the platform's code to branch on |
| `cm-api` | REST/SSE API, missions + phase runner, judge, tool gates, delivery, the MCP doors, fleet, podcast |
| `cm-db` | Postgres persistence (sqlx, offline-checked) |
| `cm-llm` | Provider abstraction (Anthropic-format and OpenAI-compatible backends) |
| `cm-secrets` | Secret storage behind the broker |
| `cm-sandbox` / `cm-safety` | Sandbox provisioning + the §15 approval/gating model |
| `cm-tools` | Tool contract + registry surfaced through the door |
| `cm-testkit` | Shared test utilities (scripted providers, fixture builders) |
| `cm-auth`, `cm-billing`, `cm-files`, `cm-scheduler`, `cm-config`, `cm-telemetry` | Supporting services |

### Binaries (`crates/bins/`)

| Bin | Role |
|---|---|
| `clawmates-server` | The single server binary (API + gateway + runtime + background workers) |
| `clawmates-broker` | Out-of-process secret broker over a private unix socket |
| `clawmates-node` | Fleet node daemon: registers with the server, runs microVMs, terminals and tool updates on its node |
| `fcagent` | PID 1 inside each Firecracker microVM; answers the host over vsock |

### Frontend (`frontend/`)

Next.js 16, React 19, Tailwind v4. The dashboard at `/` (tier rail, World graph, agent computer),
plus agent, team, skills, approvals and team-management pages. Talks to the backend through a
same-origin `/api` proxy that swaps the session for a bearer token and streams SSE.

### Content (`templates/`)

`templates/teams/` (12 team templates: roles, prompts, brain seeds, skill bindings) and
`templates/workflows/` (7 recipes), loaded at boot. How mature each one is — which have been run and
what they delivered — is tracked in [`docs/TEMPLATE-MATURITY.md`](docs/TEMPLATE-MATURITY.md).

### Data plane

Postgres, with the server self-migrating on boot from `migrations/` (`0001`–`0087`).

---

## Quickstart

The fastest path is the single-node Docker Compose deployment.

```bash
cd deploy/compose
cp .env.example .env
# Edit .env: set POSTGRES_PASSWORD and CLAWMATES_BOOTSTRAP_OWNER_PASSWORD.

docker compose build      # or load a signed release bundle (deploy/airgapped/install.sh)
docker compose up -d
```

The server self-migrates and, if `CLAWMATES_BOOTSTRAP_OWNER_PASSWORD` is set, provisions the first
Owner + workspace on first boot. Then:

- **App** → http://localhost:3000 (sign in with the bootstrap owner)
- **API health** → http://localhost:8080/healthz

Backends: `openai_compat` by default (point `[llm].base_url` at vLLM/Ollama/llama.cpp); set
`provider = "anthropic"` + `ANTHROPIC_API_KEY` to use Claude. Extra named providers (GLM, Kimi, a local
model) go in `[[llm.providers]]` and are selected as `name:model`. Auth is `local` by default or `clerk` at
runtime. See [`deploy/compose/README.md`](deploy/compose/README.md) for all knobs and the broker
master-key backup step.

**Broker master key.** The broker's master key lives in the `broker_key` named volume and is
generated on first boot. **Back this up** before running the stack for anything real — losing it
un-decrypts every stored secret.

**Local development:**

```bash
# Backend (needs a Postgres; sqlx is offline-checked against .sqlx/)
cargo build
cargo test
cargo clippy --all-targets

# Regenerate the sqlx cache after changing any query:
#   DATABASE_URL=… cargo sqlx prepare --workspace

# Frontend
cd frontend
npm install
npm run dev        # also: npm run lint / npm run typecheck / npm run build
```

Run the reproducible topology benchmark (offline-deterministic; real models with `ANTHROPIC_API_KEY`):

```bash
cargo run -p cm-orchestrator --example topology_bench --features provider
```

**End-to-end verification against a deployment:**

```bash
scripts/verify-mission-delivery.sh <scenario>   # launches real missions, asserts delivery, gates, judge
scripts/judge-eval.sh                            # the judge's 15 known-answer cases (JUDGE=kimi to compare)
```

---

## Production deployment (clawmates.work on gw-04)

- **CI:** a push to `main` runs [`.gitea/workflows/deploy.yml`](.gitea/workflows/deploy.yml) on the
  gw-04 Gitea runner: `cargo test --workspace`, then builds the images and pushes
  `main-<sha>` and `:latest` to the fleet registry at `100.94.185.103:5000`. The workflow then waits
  up to 5 minutes for prod to report the new commit.
- **Roll-out:** [`deploy/gw-04/clawmates-deploy.sh`](deploy/gw-04/clawmates-deploy.sh), run every minute
  by `clawmates-deploy.timer`, drift-checks each running image against `:latest` and recreates the
  service on drift. Logs: `/var/log/clawmates-deploy.log`.
- **Host config** lives outside git on gw-04: `/opt/clawmates/.env` and `/opt/clawmates/clawmates.toml`
  (provider registry, including the judge's GLM and Kimi providers).

**gw-04-specific gotchas:**

- `clawmates-runtime` on gw-04 is **not compose-managed** — it's a standalone `docker run` invocation.
  Provider env (`ZEROCLAW_providers__*`) must be set on that container.
- The server container runs as **UID 65532** (distroless nonroot). Any bind-mount host path must be
  `chown 65532:65532` before boot or the server can't write.
- Per-mission ZeroClaw containers get their provider keys forwarded from the server; those envs must live
  on the compose `server` block, not just on the shared runtime.
- The judge's container (`clawmates-runtime` on `clawmates_core`) has **no route to the internet** by
  design; dependencies it needs are installed offline.

---

## Code size budget

[`ci/check-loc.sh`](ci/check-loc.sh) defines a soft limit of 1100 lines and a hard limit of 1500 per
source file. It is **not currently run in CI**, and 12 files exceed the hard limit (the largest,
`crates/cm-api/src/phase_runner.rs`, is ~3,450 lines). Split files when you touch them.

---

## Roadmap

**Shipped**
- ✅ Pure-Rust topology engine — model → classify → build (all 12 kinds) → execute → compare (Pareto) →
  evolve; durable, crash-resumable topology runs with live SSE.
- ✅ The deploy ladder (single → team → company → org) and the single live dashboard.
- ✅ Missions as recipes of judged phases, on container and microVM tiers, delivered to git.
- ✅ Independent cross-provider judge with its own checks, verification plans, offline npm installs,
  Kimi fallback and a quota watchdog.
- ✅ `PreToolUse` gate on both tiers (floor rules, role policy, protected hook files) with task-permission
  and argument-provenance (taint) in shadow; microVM stop gate.
- ✅ Skills catalog + MCP skills door; skill triage and skill-use measurement.
- ✅ Per-repo project memory (`.brain`) and the `self_audit` recipe over it.
- ✅ Continuous research → vault → podcast, end to end.
- ✅ Decision tier (`cm-decide`): calibrated door governor with a held band for approval.
- ✅ Multi-user workspaces, node pool with capacity-aware placement, Beszel metrics + rules.
- ✅ Self-host via Docker Compose; CI → registry → timer-driven roll-out on gw-04.

**Next**
- Enforce task permission and argument provenance (both still in shadow, gathering evidence).
- Evidence the remaining team templates (4 of 12 still need a target stack: mobile, gpu, threejs, and
  `insight_research`).
- A dedicated judge key, so no other consumer of a shared provider plan can starve the judge.
- Wire `ci/check-loc.sh` into CI once the oversized files are split.

**Research**
- The accompanying paper, *Large Dynamic Agentic Topologies* (`papers/dynamic-agentic-topologies.md`):
  the topology-invariant authority result, the comparison/Pareto methodology, and benchmark results.

---

## Safety

The design goal is **authority that no topology can widen** (spec §15): a structure, or a switch between
structures, never gives an agent more reach than its sandbox. What that means today, per tier:

- **Chat / §15 agents** run tool-free; every action that leaves the sandbox goes through the MCP door,
  where a calibrated governor allows, denies, or **holds for human approval**.
- **Mission agents** do run tools (Bash, file edits) inside their own container or microVM, behind the
  `PreToolUse` gate. MicroVMs have no NIC and reach only an allow-listed set of hosts through a proxy.
  **Container-tier missions currently have open public-internet egress** (tailnet and host SSH are
  blocked); see [`docs/MISSION-EGRESS.md`](docs/MISSION-EGRESS.md) and
  [`docs/TASK-PERMISSION-AND-TAINT.md`](docs/TASK-PERMISSION-AND-TAINT.md) for the measurements and the
  controls being built on top.
- Platform credentials are held by the secret broker behind a private socket, and a mission container
  gets only a narrowly scoped skills token, never a ClawMates session. **Model-provider keys never enter
  a container-tier mission** when the LLM proxy is on (`CLAWMATES_LLM_PROXY=1`, as on prod): the
  container holds a per-mission token, Claude Code's base URL points at the server's proxy on an
  unpublished port, and the proxy adds the real credential — honouring the token only while its mission
  is running. Behind that, delivery refuses to push any change containing a server key, and every
  recorded event, judge verdict and judge input is redacted. **MicroVM guests hold no provider key
  either** (node daemon 0.5.0+): the guest's CLI talks to its own loopback, fcagent pipes that to the
  node, and the node relays it to the proxy on the server's tailnet-only port — no key on the node or in
  the guest. The server reaches Docker through an allow-listed socket proxy.
- The gate is a guardrail against accidents and obvious exfiltration, not a boundary against a
  determined agent (indirection defeats string matching). The boundaries are the VM, the network policy
  and the broker.