diff --git a/README.md b/README.md index 67b040f..da332c7 100644 --- a/README.md +++ b/README.md @@ -6,9 +6,9 @@ the organizational *topology* that fits it.** Clawmates is a multi-agent platform where every unit of work is a **topology**: a graph of role-slots bound to real AI agents ("claws"). The same model nests recursively — a team is a topology of claws, a company is a topology of teams, an org is a topology of companies — so you compose and run agentic -systems from one agent up to an entire organization. A safety invariant runs through all of it: -**authority is topology-invariant** — no choice of structure can let an agent exceed its sandbox (spec -§15). +systems from one agent up to an entire organization. The work itself runs as **missions**: recipes of +phases, staffed by team templates, executed in isolated containers or Firecracker microVMs, and checked +by an independent judge before anything is called done. Live at **[clawmates.work](https://clawmates.work)**. @@ -16,34 +16,50 @@ Live at **[clawmates.work](https://clawmates.work)**. ## Features -- **The deploy ladder — single → team → company → org.** Pick a scale; each rung instantiates a baseline - topology and binds it to real, individually-chattable claws. Higher rungs *compose* the rung below: - a company is staffed with teams, an org with companies. -- **Missions.** The primary user-facing unit: a scoped multi-team workload with team templates, live - progress, bulk operations, and a canvas view. Missions are hard-required to include a team template - and can span research + development teams. -- **Recursive execution.** Running a parent runs each child's whole sub-topology, all the way down to - the leaf claws — on a **durable, crash-resumable** runner (checkpointed per step, with cancellation). -- **INFRA tier via Herdr.** Every fleet node runs a persistent `clawmates-node` daemon (herdr) reachable - from the platform: mission wizard picks a target runtime, `fleet_herdr` dispatches on launch, and - a Live Pane surfaces each node's herdr TUI via xterm.js. -- **12 organizational topologies.** Hierarchical, pipeline, swarm, mesh, debate, hub-spoke, star-MoE, - market, ring, flat, holacratic, blackboard — over five execution patterns. -- **Multi-topology comparison + evolution.** Run one task across many topologies and get a quality/cost - **Pareto front**; a MAP-Elites search evolves better (kind × team-size) configurations using the - comparison harness as fitness. -- **Recursive zoom canvas.** One view for every tier: click a node to drill down (org→company→team→claw), - breadcrumb to zoom back up. -- **§15 safety by construction.** Agents run tool-free in network-isolated sandboxes; every - sandbox-leaving action is a gated, human-approvable "door" tool. A secret broker holds credentials that - never reach agent code, and an allow-listed Docker socket caps blast radius. -- **Heterogeneous models.** Bind any node to a different backend; supported providers include Claude - (default for refine), GLM, Kimi, Groq. Configured per-node in the wizard. -- **Level-Up.** Per-claw and per-team improvement proposals with an inbox + review drawer. -- **Beszel + Tailscale integration.** First-class routes to the fleet's monitoring hub and mesh. -- **Self-hostable.** A single-node Docker Compose deployment runs the whole platform with the same - network-segmented security model as the Kubernetes path; a separate rolling-deploy path serves - clawmates.work from gw-04 against the fleet registry. +- **One dashboard.** The workspace home is a single live dashboard with a tier rail — **World, Agents, + Missions, Repos, Infra, Podcast**. World is a live Gource-style graph of what agents are doing; the + org → company → team structure is one expandable graph; each agent opens a slide-out "computer" + (chat, terminal, files, brain). +- **Missions.** A mission is a **recipe** (`templates/workflows/*.toml`) of ordered phases — research, + coding, security scan, benchmark — each with a task brief and a `done_when` completion condition, staffed + by a **team template** (`templates/teams/*.toml`). Seven recipes ship: `research_only`, + `research_and_code`, `security_hardening`, `refactor`, `benchmark`, `continuous_research`, `self_audit`. +- **Two execution tiers.** *Container tier*: each mission gets its own ZeroClaw runtime container on the + gateway, running Claude Code (`claude_cli`) with fallback to Kimi and GLM. *MicroVM tier*: phases run + in Firecracker microVMs on fleet nodes (`clawmates-node` + the `fcagent` guest), with per-backend + egress (Claude, GLM, Kimi, or a local model over vsock). +- **An independent judge.** Every conditioned phase is judged by a model from a *different* provider + family than the agents (GLM by default, Kimi as automatic fallback), which runs its own allow-listed + checks — tests, `rg`, git — against a copy of the work, commits to a verification plan before reading + the evidence, and installs npm dependencies offline from the lockfile. A phase that fails is retried + with the judge's guidance, up to `max_iterations`. Measured with `scripts/judge-eval.sh` (15 known-answer + cases). +- **Judge quota watchdog.** Polls the GLM and Kimi usage APIs, warns at 80% of any window, and switches + judging to the fallback at 95% (`GET /api/judge/quota`). +- **Delivery to git.** Each phase's work is committed and pushed to a mission branch on the repo's forge; + `continuous_research` auto-merges additive-only changes into the vault. +- **Tool gates on every agent call.** A `PreToolUse` gate (both tiers) refuses destructive and exfiltrating + commands, protects its own hook files, enforces per-role policy (e.g. a read-only verifier), and records + task-permission and argument-provenance ("taint") violations in shadow mode. A stop gate keeps a + microVM agent working until its `done_when_check` passes. +- **Skills.** A catalog of skills bound to team roles, delivered to mission agents through the MCP + skills door; per-mission skill triage and skill-use measurement. +- **Project memory.** Each repository keeps a `.brain` (ClawhDF5) of every judge verdict; missions recall + relevant past verdicts, and the `self_audit` recipe audits the whole record for failure patterns. +- **Continuous research + podcast.** Harvests new arXiv papers into an Obsidian vault, triages them, + writes an analysis and a two-host script, and renders audio with ElevenLabs. +- **Decision tier (`cm-decide`).** Cheap calibrated classifiers (Jev) for gut-check decisions: the §15 + door governor (allow / deny / hold for approval), skill triage, paper triage, memory rerank. +- **Master Planner.** The "+" deploy is a chat that proposes and scaffolds a team for a goal + (specialists, swarm, scheduled, triggered). +- **12 organizational topologies + evolution.** Hierarchical, pipeline, swarm, mesh, debate, hub-spoke, + star-MoE, market, ring, flat, holacratic, blackboard; multi-topology comparison with a quality/cost + **Pareto front**, and MAP-Elites evolution over (kind × team size). +- **Fleet.** Multi-user workspaces with quotas; a node pool (`clawmates-node` over Tailscale) with + capacity-aware placement, drain, per-node tool versions and one-click updates, and Beszel metrics + feeding a rules engine. +- **Agent-to-agent comms.** Chat rooms, a gated delegation bridge, per-claw door identity and A2A ingress. +- **Self-hostable.** A single-node Docker Compose deployment runs the whole platform. --- @@ -58,12 +74,13 @@ A Rust workspace (the platform) + a Next.js app (the web UI). | `cm-domain` | Shared types: ids, roles, workspaces, users | | `cm-topology` | Topology data model: 12-kind taxonomy, graph, classifier, per-kind builders + heuristics | | `cm-orchestrator` | Execution engine: async control-flow over a generic `TurnExecutor`; planners, comparison harness, MAP-Elites evolution | -| `cm-runtime` | The §15-safe per-tenant agent runtime | -| `cm-brain` | Shared LLM planning + reasoning primitives used by orchestrator and refine | -| `cm-api` | REST/SSE API + streaming gateway + recursive tier execution + the MCP "door" + missions + fleet_herdr | +| `cm-runtime` | The §15-safe per-tenant agent runtime and the LLM provider registry | +| `cm-brain` | Facade over the canonical `.brain` (ClawhDF5 brain-pack): one HDF5 file per agent or repo holding definition + memory | +| `cm-decide` | Typed, calibrated decisions (Jev classifiers) for the platform's code to branch on | +| `cm-api` | REST/SSE API, missions + phase runner, judge, tool gates, delivery, the MCP doors, fleet, podcast | | `cm-db` | Postgres persistence (sqlx, offline-checked) | -| `cm-llm` | Provider abstraction over the model backends | -| `cm-secrets` / `clawmates-broker` | The secret broker — credentials never leave it | +| `cm-llm` | Provider abstraction (Anthropic-format and OpenAI-compatible backends) | +| `cm-secrets` | Secret storage behind the broker | | `cm-sandbox` / `cm-safety` | Sandbox provisioning + the §15 approval/gating model | | `cm-tools` | Tool contract + registry surfaced through the door | | `cm-testkit` | Shared test utilities (scripted providers, fixture builders) | @@ -73,21 +90,26 @@ A Rust workspace (the platform) + a Next.js app (the web UI). | Bin | Role | |---|---| -| `clawmates-server` | The single server binary (API + gateway + runtime + scheduler) | +| `clawmates-server` | The single server binary (API + gateway + runtime + background workers) | | `clawmates-broker` | Out-of-process secret broker over a private unix socket | -| `clawmates-node` | The **herdr** daemon: runs on every fleet node, dispatches missions to that node, exposes a TUI streamed into the Live Pane | +| `clawmates-node` | Fleet node daemon: registers with the server, runs microVMs, terminals and tool updates on its node | +| `fcagent` | PID 1 inside each Firecracker microVM; answers the host over vsock | ### Frontend (`frontend/`) -Next.js 16, React 19, Tailwind v4. Two-tier rail (structure + context), the recursive zoom canvas, and -the deploy wizards. Missions surface (canvas + list + wizard + live pane + team tab + live events), -Herdr sessions UI, Level-Up inbox + review drawer. Talks to the backend through a same-origin `/api` -proxy that swaps the session for a bearer token and streams SSE. +Next.js 16, React 19, Tailwind v4. The dashboard at `/` (tier rail, World graph, agent computer), +plus agent, team, skills, approvals and team-management pages. Talks to the backend through a +same-origin `/api` proxy that swaps the session for a bearer token and streams SSE. + +### Content (`templates/`) + +`templates/teams/` (12 team templates: roles, prompts, brain seeds, skill bindings) and +`templates/workflows/` (7 recipes), loaded at boot. How mature each one is — which have been run and +what they delivered — is tracked in [`docs/TEMPLATE-MATURITY.md`](docs/TEMPLATE-MATURITY.md). ### Data plane -Postgres, with the server self-migrating on boot. Migration series `0001–0057+`; slice-9 cleanup -(`0053`) retired the legacy research/loops path after missions replaced it. +Postgres, with the server self-migrating on boot from `migrations/` (`0001`–`0087`). --- @@ -111,7 +133,8 @@ Owner + workspace on first boot. Then: - **API health** → http://localhost:8080/healthz Backends: `openai_compat` by default (point `[llm].base_url` at vLLM/Ollama/llama.cpp); set -`provider = "anthropic"` + `ANTHROPIC_API_KEY` to use Claude. Auth is `local` by default or `clerk` at +`provider = "anthropic"` + `ANTHROPIC_API_KEY` to use Claude. Extra named providers (GLM, Kimi, a local +model) go in `[[llm.providers]]` and are selected as `name:model`. Auth is `local` by default or `clerk` at runtime. See [`deploy/compose/README.md`](deploy/compose/README.md) for all knobs and the broker master-key backup step. @@ -142,40 +165,45 @@ Run the reproducible topology benchmark (offline-deterministic; real models with cargo run -p cm-orchestrator --example topology_bench --features provider ``` +**End-to-end verification against a deployment:** + +```bash +scripts/verify-mission-delivery.sh # launches real missions, asserts delivery, gates, judge +scripts/judge-eval.sh # the judge's 15 known-answer cases (JUDGE=kimi to compare) +``` + --- ## Production deployment (clawmates.work on gw-04) -The public site runs a different path than the airgapped compose. Gitea Actions builds and pushes -`broker`, `server`, and `frontend` images to the fleet registry at -`100.94.185.103:5000/clawmates/:latest`. gw-04 runs a systemd-timer-driven rolling deploy: +- **CI:** a push to `main` runs [`.gitea/workflows/deploy.yml`](.gitea/workflows/deploy.yml) on the + gw-04 Gitea runner: `cargo test --workspace`, then builds the images and pushes + `main-` and `:latest` to the fleet registry at `100.94.185.103:5000`. The workflow then waits + up to 5 minutes for prod to report the new commit. +- **Roll-out:** [`deploy/gw-04/clawmates-deploy.sh`](deploy/gw-04/clawmates-deploy.sh), run every minute + by `clawmates-deploy.timer`, drift-checks each running image against `:latest` and recreates the + service on drift. Logs: `/var/log/clawmates-deploy.log`. +- **Host config** lives outside git on gw-04: `/opt/clawmates/.env` and `/opt/clawmates/clawmates.toml` + (provider registry, including the judge's GLM and Kimi providers). -- **Deploy script:** [`deploy/gw-04/clawmates-deploy.sh`](deploy/gw-04/clawmates-deploy.sh) — polls the - registry, drift-checks each service's running image ID against `:latest`, and calls - `docker compose up -d ` on drift. Portable across `docker compose` v2 and legacy `docker-compose` v1. -- **Timer + unit:** installed alongside the script under `/etc/systemd/system/`. -- **Logs:** `/var/log/clawmates-deploy.log`. -- **Compose file:** references registry-prefixed images directly — no retag bridging. - -**gw-04-specific gotchas (bit us on 2026-07-09 and 2026-07-12):** +**gw-04-specific gotchas:** - `clawmates-runtime` on gw-04 is **not compose-managed** — it's a standalone `docker run` invocation. - Provider env (`ANTHROPIC_API_KEY`, `ZEROCLAW_providers__*`) must be set on that container. + Provider env (`ZEROCLAW_providers__*`) must be set on that container. - The server container runs as **UID 65532** (distroless nonroot). Any bind-mount host path must be `chown 65532:65532` before boot or the server can't write. -- Per-team ZeroClaw containers inherit `ZEROCLAW_providers__*` from the server; those envs must live - on the compose `server` block, not just on shared runtime. +- Per-mission ZeroClaw containers get their provider keys forwarded from the server; those envs must live + on the compose `server` block, not just on the shared runtime. +- The judge's container (`clawmates-runtime` on `clawmates_core`) has **no route to the internet** by + design; dependencies it needs are installed offline. --- -## CI budgets +## Code size budget -- **Hard limit:** 1500 lines per source file. CI fails. -- **Soft limit:** 1100 lines. CI warns — split before it hurts. -- Enforced by [`ci/check-loc.sh`](ci/check-loc.sh). - -Common split pattern: extract sub-components (`MissionLivePane`, `AutoProvisionCard`) into their own -file when the parent creeps past the soft limit. +[`ci/check-loc.sh`](ci/check-loc.sh) defines a soft limit of 1100 lines and a hard limit of 1500 per +source file. It is **not currently run in CI**, and 12 files exceed the hard limit (the largest, +`crates/cm-api/src/phase_runner.rs`, is ~3,450 lines). Split files when you touch them. --- @@ -183,32 +211,26 @@ file when the parent creeps past the soft limit. **Shipped** - ✅ Pure-Rust topology engine — model → classify → build (all 12 kinds) → execute → compare (Pareto) → - workflow → LLM-judge → evolve. -- ✅ Topologies UI: catalog browser, builder/visualizer, multi-topology comparison with a Pareto scatter. -- ✅ Durable topology runs — crash-resumable, checkpointed per step, cancellable, with live SSE. -- ✅ **The full deploy ladder** — single → team → company → org, with recursive execution down to the - leaf claws and a recursive zoom canvas + two-tier navigation. -- ✅ **Missions (slices 1–9)** — multi-team model, hard-required team templates, canvas + wizard + list, - auto-refresh + Team tab + Live events tab, add/edit/delete toolbar, bulk delete, security-scan + - benchmark trigger buttons, per-mission repo checkout on launch, LevelUpInbox mounted. -- ✅ **Herdr (phases 0–3)** — `clawmates-node` daemon (systemd/launchd persistence), `fleet_herdr` - dispatch module + node daemon ops, missions `runtime_kind` + `target_node` schema, wizard runtime - picker with `on_launch` auto-dispatch, Live Pane (xterm.js → node's herdr TUI), INFRA-tier Herdr - sessions surface. -- ✅ **Refine on Opus 4.8** — before/after diff view, accept/cancel/restore controls. -- ✅ **Level-Up** — per-claw/per-team improvement proposals, inbox + review drawer. -- ✅ §15 safety: tool-free sandboxes, the gated MCP "door" (with real email/Slack delivery), the secret - broker, allow-listed Docker socket. -- ✅ Self-host: single-node Docker Compose with full network segmentation. -- ✅ Prod path on gw-04: registry-driven rolling deploy via systemd timer. + evolve; durable, crash-resumable topology runs with live SSE. +- ✅ The deploy ladder (single → team → company → org) and the single live dashboard. +- ✅ Missions as recipes of judged phases, on container and microVM tiers, delivered to git. +- ✅ Independent cross-provider judge with its own checks, verification plans, offline npm installs, + Kimi fallback and a quota watchdog. +- ✅ `PreToolUse` gate on both tiers (floor rules, role policy, protected hook files) with task-permission + and argument-provenance (taint) in shadow; microVM stop gate. +- ✅ Skills catalog + MCP skills door; skill triage and skill-use measurement. +- ✅ Per-repo project memory (`.brain`) and the `self_audit` recipe over it. +- ✅ Continuous research → vault → podcast, end to end. +- ✅ Decision tier (`cm-decide`): calibrated door governor with a held band for approval. +- ✅ Multi-user workspaces, node pool with capacity-aware placement, Beszel metrics + rules. +- ✅ Self-host via Docker Compose; CI → registry → timer-driven roll-out on gw-04. **Next** -- Team / company templates as first-class saved catalogs (compose orgs from reusable building blocks). -- Per-leaf nested checkpoint resume (today the recursive runner resumes at parent-node granularity). -- Persona injection into runtime turns (beyond role-driven prompting). -- Richer per-tier dashboards (company coordination, org portfolio/governance metrics). -- Group lifecycle management (delete/edit a deployed team/company/org; deprovision its agents) — - bulk delete shipped for missions, extending to teams/companies/orgs next. +- Enforce task permission and argument provenance (both still in shadow, gathering evidence). +- Evidence the remaining team templates (4 of 12 still need a target stack: mobile, gpu, threejs, and + `insight_research`). +- A dedicated judge key, so no other consumer of a shared provider plan can starve the judge. +- Wire `ci/check-loc.sh` into CI once the oversized files are split. **Research** - The accompanying paper, *Large Dynamic Agentic Topologies* (`papers/dynamic-agentic-topologies.md`): @@ -218,8 +240,20 @@ file when the parent creeps past the soft limit. ## Safety -The network segmentation **is** the security model. Agent sandboxes run with no network at all; only the -browser container has egress. The secret broker is reachable only over a private socket and credentials -never enter agent code. The server reaches Docker through an allow-listed socket proxy that can manage -sandbox containers and nothing else. Every sandbox-leaving action is gated behind a human approval. No -topology — and no switch between topologies — can bypass any of this. +The design goal is **authority that no topology can widen** (spec §15): a structure, or a switch between +structures, never gives an agent more reach than its sandbox. What that means today, per tier: + +- **Chat / §15 agents** run tool-free; every action that leaves the sandbox goes through the MCP door, + where a calibrated governor allows, denies, or **holds for human approval**. +- **Mission agents** do run tools (Bash, file edits) inside their own container or microVM, behind the + `PreToolUse` gate. MicroVMs have no NIC and reach only an allow-listed set of hosts through a proxy. + **Container-tier missions currently have open public-internet egress** (tailnet and host SSH are + blocked); see [`docs/MISSION-EGRESS.md`](docs/MISSION-EGRESS.md) and + [`docs/TASK-PERMISSION-AND-TAINT.md`](docs/TASK-PERMISSION-AND-TAINT.md) for the measurements and the + controls being built on top. +- Credentials are held by the secret broker behind a private socket. A mission container gets only a + narrowly scoped skills token, never a full session. The server reaches Docker through an allow-listed + socket proxy. +- The gate is a guardrail against accidents and obvious exfiltration, not a boundary against a + determined agent (indirection defeats string matching). The boundaries are the VM, the network policy + and the broker.