clawmates/README.md

# Clawmates

**Deploy agents at any scale — a single claw, a team, a company, or a whole org — and run a mission across
the organizational *topology* that fits it.**

Clawmates is a multi-agent platform where every unit of work is a **topology**: a graph of role-slots
bound to real AI agents ("claws"). The same model nests recursively — a team is a topology of claws, a
company is a topology of teams, an org is a topology of companies — so you compose and run agentic
systems from one agent up to an entire organization. The work itself runs as **missions**: recipes of
phases, staffed by team templates, executed in isolated containers or Firecracker microVMs, and checked
by an independent judge before anything is called done.

Live at **[clawmates.work](https://clawmates.work)**.

---

## Features

- **One dashboard.** The workspace home is a single live dashboard with a tier rail — **World, Agents,
  Missions, Repos, Infra, Podcast**. World is a live Gource-style graph of what agents are doing; the
  org → company → team structure is one expandable graph; each agent opens a slide-out "computer"
  (chat, terminal, files, brain).
- **Missions.** A mission is a **recipe** (`templates/workflows/*.toml`) of ordered phases — research,
  coding, security scan, benchmark — each with a task brief and a `done_when` completion condition, staffed
  by a **team template** (`templates/teams/*.toml`). Seven recipes ship: `research_only`,
  `research_and_code`, `security_hardening`, `refactor`, `benchmark`, `continuous_research`, `self_audit`.
- **Two execution tiers.** *Container tier*: each mission gets its own ZeroClaw runtime container on the
  gateway, running Claude Code (`claude_cli`) with fallback to Kimi and GLM. *MicroVM tier*: phases run
  in Firecracker microVMs on fleet nodes (`clawmates-node` + the `fcagent` guest), with per-backend
  egress (Claude, GLM, Kimi, or a local model over vsock).
- **An independent judge.** Every conditioned phase is judged. When a cross-provider judge is configured
  (prod: GLM, with Kimi as automatic fallback) it is a model from a *different* provider family than the
  agents; without one (the self-host default) the phase is judged by Claude and recorded as **not**
  independent. The judge runs its own allow-listed
  checks — tests, `rg`, git — against a copy of the work, commits to a verification plan before reading
  the evidence, and installs npm dependencies offline from the lockfile. A phase that fails is retried
  with the judge's guidance, up to `max_iterations`. Measured with `scripts/judge-eval.sh` (15 known-answer
  cases).
- **Judge quota watchdog.** Polls the GLM and Kimi usage APIs, warns at 80% of any window, and switches
  judging to the fallback at 95% (`GET /api/judge/quota`).
- **Delivery to git.** Each phase's work is committed and pushed to a mission branch on the repo's forge;
  `continuous_research` auto-merges additive-only changes into the vault.
- **Tool gates on every agent call.** A `PreToolUse` gate (both tiers) refuses destructive and exfiltrating
  commands, protects its own hook files, enforces per-role policy (e.g. a read-only verifier), and records
  task-permission and argument-provenance ("taint") violations in shadow mode. A stop gate sends a
  microVM agent back to work while its `done_when_check` fails, up to 3 times.
- **Skills.** A catalog of skills bound to team roles, delivered to mission agents through the MCP
  skills door; per-mission skill triage and skill-use measurement.
- **Project memory.** Each repository keeps a `.brain` (ClawhDF5) of every judge verdict; missions recall
  relevant past verdicts, and the `self_audit` recipe audits the whole record for failure patterns.
- **Continuous research + podcast.** Harvests new arXiv papers into an Obsidian vault, triages them,
  writes an analysis and a two-host script, and renders audio with ElevenLabs.
- **Decision tier (`cm-decide`).** Cheap calibrated classifiers (Jev) for gut-check decisions: the §15
  door governor (allow / deny / hold for approval), skill triage, paper triage, memory rerank.
- **Master Planner.** The "+" deploy is a chat that proposes and scaffolds a team for a goal
  (specialists, swarm, scheduled, triggered).
- **12 organizational topologies + evolution.** Hierarchical, pipeline, swarm, mesh, debate, hub-spoke,
  star-MoE, market, ring, flat, holacratic, blackboard; multi-topology comparison with a quality/cost
  **Pareto front**, and MAP-Elites evolution over (kind × team size).
- **Fleet.** Multi-user workspaces with quotas; a node pool (`clawmates-node` over Tailscale) with
  capacity-aware placement, drain, per-node tool versions and one-click updates, and Beszel metrics
  feeding a rules engine.
- **Agent-to-agent comms.** Chat rooms, a gated delegation bridge, per-claw door identity and A2A ingress.
- **Self-hostable.** A single-node Docker Compose deployment runs the whole platform.

---

## Architecture

A Rust workspace (the platform) + a Next.js app (the web UI).

### Backend — Rust workspace (`crates/`)

| Crate | Role |
|---|---|
| `cm-domain` | Shared types: ids, roles, workspaces, users |
| `cm-topology` | Topology data model: 12-kind taxonomy, graph, classifier, per-kind builders + heuristics |
| `cm-orchestrator` | Execution engine: async control-flow over a generic `TurnExecutor`; planners, comparison harness, MAP-Elites evolution |
| `cm-runtime` | The §15-safe per-tenant agent runtime and the LLM provider registry |
| `cm-brain` | Facade over the canonical `.brain` (ClawhDF5 brain-pack): one HDF5 file per agent or repo holding definition + memory |
| `cm-decide` | Typed, calibrated decisions (Jev classifiers) for the platform's code to branch on |
| `cm-api` | REST/SSE API, missions + phase runner, judge, tool gates, delivery, the MCP doors, fleet, podcast |
| `cm-db` | Postgres persistence (sqlx, offline-checked) |
| `cm-llm` | Provider abstraction (Anthropic-format and OpenAI-compatible backends) |
| `cm-secrets` | Secret storage behind the broker |
| `cm-sandbox` / `cm-safety` | Sandbox provisioning + the §15 approval/gating model |
| `cm-tools` | Tool contract + registry surfaced through the door |
| `cm-testkit` | Shared test utilities (scripted providers, fixture builders) |
| `cm-auth`, `cm-billing`, `cm-files`, `cm-scheduler`, `cm-config`, `cm-telemetry` | Supporting services |

### Binaries (`crates/bins/`)

| Bin | Role |
|---|---|
| `clawmates-server` | The single server binary (API + gateway + runtime + background workers) |
| `clawmates-broker` | Out-of-process secret broker over a private unix socket |
| `clawmates-node` | Fleet node daemon: registers with the server, runs microVMs, terminals and tool updates on its node |
| `fcagent` | PID 1 inside each Firecracker microVM; answers the host over vsock |

### Frontend (`frontend/`)

Next.js 16, React 19, Tailwind v4. The dashboard at `/` (tier rail, World graph, agent computer),
plus agent, team, skills, approvals and team-management pages. Talks to the backend through a
same-origin `/api` proxy that swaps the session for a bearer token and streams SSE.

### Content (`templates/`)

`templates/teams/` (12 team templates: roles, prompts, brain seeds, skill bindings) and
`templates/workflows/` (7 recipes), loaded at boot. How mature each one is — which have been run and
what they delivered — is tracked in [`docs/TEMPLATE-MATURITY.md`](docs/TEMPLATE-MATURITY.md).

### Data plane

Postgres, with the server self-migrating on boot from `migrations/` (`0001`–`0087`).

---

## Quickstart

The fastest path is the single-node Docker Compose deployment.

```bash
cd deploy/compose
cp .env.example .env
# Edit .env: set POSTGRES_PASSWORD and CLAWMATES_BOOTSTRAP_OWNER_PASSWORD.

docker compose build      # or load a signed release bundle (deploy/airgapped/install.sh)
docker compose up -d
```

The server self-migrates and, if `CLAWMATES_BOOTSTRAP_OWNER_PASSWORD` is set, provisions the first
Owner + workspace on first boot. Then:

- **App** → http://localhost:3000 (sign in with the bootstrap owner)
- **API health** → http://localhost:8080/healthz

Backends: `openai_compat` by default (point `[llm].base_url` at vLLM/Ollama/llama.cpp); set
`provider = "anthropic"` + `ANTHROPIC_API_KEY` to use Claude. Extra named providers (GLM, Kimi, a local
model) go in `[[llm.providers]]` and are selected as `name:model`. Auth is `local` by default or `clerk` at
runtime. See [`deploy/compose/README.md`](deploy/compose/README.md) for all knobs and the broker
master-key backup step.

**Broker master key.** The broker's master key lives in the `broker_key` named volume and is
generated on first boot. **Back this up** before running the stack for anything real — losing it
un-decrypts every stored secret.

**Local development:**

```bash
# Backend (needs a Postgres; sqlx is offline-checked against .sqlx/)
cargo build
cargo test
cargo clippy --all-targets

# Regenerate the sqlx cache after changing any query:
#   DATABASE_URL=… cargo sqlx prepare --workspace

# Frontend
cd frontend
npm install
npm run dev        # also: npm run lint / npm run typecheck / npm run build
```

Run the reproducible topology benchmark (offline-deterministic; real models with `ANTHROPIC_API_KEY`):

```bash
cargo run -p cm-orchestrator --example topology_bench --features provider
```

**End-to-end verification against a deployment:**

```bash
scripts/verify-mission-delivery.sh <scenario>   # launches real missions, asserts delivery, gates, judge
scripts/judge-eval.sh                            # the judge's 15 known-answer cases (JUDGE=kimi to compare)
```

---

## Production deployment (clawmates.work on gw-04)

- **CI:** a push to `main` runs [`.gitea/workflows/deploy.yml`](.gitea/workflows/deploy.yml) on the
  gw-04 Gitea runner: `cargo test --workspace`, then builds the images and pushes
  `main-<sha>` and `:latest` to the fleet registry at `100.94.185.103:5000`. The workflow then waits
  up to 5 minutes for prod to report the new commit.
- **Roll-out:** [`deploy/gw-04/clawmates-deploy.sh`](deploy/gw-04/clawmates-deploy.sh), run every minute
  by `clawmates-deploy.timer`, drift-checks each running image against `:latest` and recreates the
  service on drift. Logs: `/var/log/clawmates-deploy.log`.
- **Host config** lives outside git on gw-04: `/opt/clawmates/.env` and `/opt/clawmates/clawmates.toml`
  (provider registry, including the judge's GLM and Kimi providers).

**gw-04-specific gotchas:**

- `clawmates-runtime` on gw-04 is **not compose-managed** — it's a standalone `docker run` invocation.
  Provider env (`ZEROCLAW_providers__*`) must be set on that container.
- The server container runs as **UID 65532** (distroless nonroot). Any bind-mount host path must be
  `chown 65532:65532` before boot or the server can't write.
- Per-mission ZeroClaw containers get their provider keys forwarded from the server; those envs must live
  on the compose `server` block, not just on the shared runtime.
- The judge's container (`clawmates-runtime` on `clawmates_core`) has **no route to the internet** by
  design; dependencies it needs are installed offline.

---

## Code size budget

[`ci/check-loc.sh`](ci/check-loc.sh) defines a soft limit of 1100 lines and a hard limit of 1500 per
source file. It is **not currently run in CI**, and 12 files exceed the hard limit (the largest,
`crates/cm-api/src/phase_runner.rs`, is ~3,450 lines). Split files when you touch them.

---

## Roadmap

**Shipped**
- ✅ Pure-Rust topology engine — model → classify → build (all 12 kinds) → execute → compare (Pareto) →
  evolve; durable, crash-resumable topology runs with live SSE.
- ✅ The deploy ladder (single → team → company → org) and the single live dashboard.
- ✅ Missions as recipes of judged phases, on container and microVM tiers, delivered to git.
- ✅ Independent cross-provider judge with its own checks, verification plans, offline npm installs,
  Kimi fallback and a quota watchdog.
- ✅ `PreToolUse` gate on both tiers (floor rules, role policy, protected hook files) with task-permission
  and argument-provenance (taint) in shadow; microVM stop gate.
- ✅ Skills catalog + MCP skills door; skill triage and skill-use measurement.
- ✅ Per-repo project memory (`.brain`) and the `self_audit` recipe over it.
- ✅ Continuous research → vault → podcast, end to end.
- ✅ Decision tier (`cm-decide`): calibrated door governor with a held band for approval.
- ✅ Multi-user workspaces, node pool with capacity-aware placement, Beszel metrics + rules.
- ✅ Self-host via Docker Compose; CI → registry → timer-driven roll-out on gw-04.

**Next**
- Enforce task permission and argument provenance (both still in shadow, gathering evidence).
- Evidence the remaining team templates (4 of 12 still need a target stack: mobile, gpu, threejs, and
  `insight_research`).
- A dedicated judge key, so no other consumer of a shared provider plan can starve the judge.
- Wire `ci/check-loc.sh` into CI once the oversized files are split.

**Research**
- The accompanying paper, *Large Dynamic Agentic Topologies* (`papers/dynamic-agentic-topologies.md`):
  the topology-invariant authority result, the comparison/Pareto methodology, and benchmark results.

---

## Safety

The design goal is **authority that no topology can widen** (spec §15): a structure, or a switch between
structures, never gives an agent more reach than its sandbox. What that means today, per tier:

- **Chat / §15 agents** run tool-free; every action that leaves the sandbox goes through the MCP door,
  where a calibrated governor allows, denies, or **holds for human approval**.
- **Mission agents** do run tools (Bash, file edits) inside their own container or microVM, behind the
  `PreToolUse` gate. MicroVMs have no NIC and reach only an allow-listed set of hosts through a proxy.
  **Container-tier missions currently have open public-internet egress** (tailnet and host SSH are
  blocked); see [`docs/MISSION-EGRESS.md`](docs/MISSION-EGRESS.md) and
  [`docs/TASK-PERMISSION-AND-TAINT.md`](docs/TASK-PERMISSION-AND-TAINT.md) for the measurements and the
  controls being built on top.
- Platform credentials are held by the secret broker behind a private socket, and a mission container
  gets only a narrowly scoped skills token, never a ClawMates session. **Model-provider keys are the
  exception:** Claude Code inside a mission needs its own credential, so container-tier missions carry
  the provider keys (`CLAUDE_CODE_OAUTH_TOKEN`, and `ZAI_API_KEY` / `KIMI_API_KEY` for the fallback
  chain) in their environment, readable by the agent. What limits the damage: delivery refuses to push
  any change containing one of those keys (exact values, verbatim or base64), and every recorded event,
  judge verdict and judge input is redacted before it is stored or sent. A key sent straight to a host
  over the network is not covered yet. The server reaches Docker through an allow-listed socket proxy.
- The gate is a guardrail against accidents and obvious exfiltration, not a boundary against a
  determined agent (indirection defeats string matching). The boundaries are the VM, the network policy
  and the broker.