Omar SobhandClaude Opus 5 483de9f88a feat(billing): agent-side spend records who was paid
Judge spend gained provider, model and mission on 2026-09-14; agent spend —
the larger half — did not. The runtime's `done` frame has always carried
`model` and `provider` beside the two token counts, and `topology_exec` read
only the counts, summed them, and charged the sum as output with no record of
which provider served the turn.

`TurnOutcome` and `StepRecord` carry a `Spend` now (input/output split,
provider, model), the worker passes it through `cm_billing::charge` along with
the mission id, and the chat runtime records the model it requested — that
loop drives one provider with no chain, so requested is answered. A bare
model name is recorded without a guessed family. `StepRecord.spend` is
`serde(default)` so journaled checkpoints from before this field still load,
and `tokens` stays as the total every reader keys on.

`charge` moved from `query!` to `query`: the macro pins the statement to
offline metadata that a schema change then has to regenerate against a live
database, for columns that are nullable text and uuid.

The done-frame test now asserts the split and the provider survive, not just
the sum.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
2026-09-14 08:17:20 -05:00

Clawmates

Deploy agents at any scale — a single claw, a team, a company, or a whole org — and run a mission across the organizational topology that fits it.

Clawmates is a multi-agent platform where every unit of work is a topology: a graph of role-slots bound to real AI agents ("claws"). The same model nests recursively — a team is a topology of claws, a company is a topology of teams, an org is a topology of companies — so you compose and run agentic systems from one agent up to an entire organization. A safety invariant runs through all of it: authority is topology-invariant — no choice of structure can let an agent exceed its sandbox (spec §15).

Live at clawmates.work.


Features

  • The deploy ladder — single → team → company → org. Pick a scale; each rung instantiates a baseline topology and binds it to real, individually-chattable claws. Higher rungs compose the rung below: a company is staffed with teams, an org with companies.
  • Missions. The primary user-facing unit: a scoped multi-team workload with team templates, live progress, bulk operations, and a canvas view. Missions are hard-required to include a team template and can span research + development teams.
  • Recursive execution. Running a parent runs each child's whole sub-topology, all the way down to the leaf claws — on a durable, crash-resumable runner (checkpointed per step, with cancellation).
  • INFRA tier via Herdr. Every fleet node runs a persistent clawmates-node daemon (herdr) reachable from the platform: mission wizard picks a target runtime, fleet_herdr dispatches on launch, and a Live Pane surfaces each node's herdr TUI via xterm.js.
  • 12 organizational topologies. Hierarchical, pipeline, swarm, mesh, debate, hub-spoke, star-MoE, market, ring, flat, holacratic, blackboard — over five execution patterns.
  • Multi-topology comparison + evolution. Run one task across many topologies and get a quality/cost Pareto front; a MAP-Elites search evolves better (kind × team-size) configurations using the comparison harness as fitness.
  • Recursive zoom canvas. One view for every tier: click a node to drill down (org→company→team→claw), breadcrumb to zoom back up.
  • §15 safety by construction. Agents run tool-free in network-isolated sandboxes; every sandbox-leaving action is a gated, human-approvable "door" tool. A secret broker holds credentials that never reach agent code, and an allow-listed Docker socket caps blast radius.
  • Heterogeneous models. Bind any node to a different backend; supported providers include Claude (default for refine), GLM, Kimi, Groq. Configured per-node in the wizard.
  • Level-Up. Per-claw and per-team improvement proposals with an inbox + review drawer.
  • Beszel + Tailscale integration. First-class routes to the fleet's monitoring hub and mesh.
  • Self-hostable. A single-node Docker Compose deployment runs the whole platform with the same network-segmented security model as the Kubernetes path; a separate rolling-deploy path serves clawmates.work from gw-04 against the fleet registry.

Architecture

A Rust workspace (the platform) + a Next.js app (the web UI).

Backend — Rust workspace (crates/)

Crate Role
cm-domain Shared types: ids, roles, workspaces, users
cm-topology Topology data model: 12-kind taxonomy, graph, classifier, per-kind builders + heuristics
cm-orchestrator Execution engine: async control-flow over a generic TurnExecutor; planners, comparison harness, MAP-Elites evolution
cm-runtime The §15-safe per-tenant agent runtime
cm-brain Shared LLM planning + reasoning primitives used by orchestrator and refine
cm-api REST/SSE API + streaming gateway + recursive tier execution + the MCP "door" + missions + fleet_herdr
cm-db Postgres persistence (sqlx, offline-checked)
cm-llm Provider abstraction over the model backends
cm-secrets / clawmates-broker The secret broker — credentials never leave it
cm-sandbox / cm-safety Sandbox provisioning + the §15 approval/gating model
cm-tools Tool contract + registry surfaced through the door
cm-testkit Shared test utilities (scripted providers, fixture builders)
cm-auth, cm-billing, cm-files, cm-scheduler, cm-config, cm-telemetry Supporting services

Binaries (crates/bins/)

Bin Role
clawmates-server The single server binary (API + gateway + runtime + scheduler)
clawmates-broker Out-of-process secret broker over a private unix socket
clawmates-node The herdr daemon: runs on every fleet node, dispatches missions to that node, exposes a TUI streamed into the Live Pane

Frontend (frontend/)

Next.js 16, React 19, Tailwind v4. Two-tier rail (structure + context), the recursive zoom canvas, and the deploy wizards. Missions surface (canvas + list + wizard + live pane + team tab + live events), Herdr sessions UI, Level-Up inbox + review drawer. Talks to the backend through a same-origin /api proxy that swaps the session for a bearer token and streams SSE.

Data plane

Postgres, with the server self-migrating on boot. Migration series 00010057+; slice-9 cleanup (0053) retired the legacy research/loops path after missions replaced it.


Quickstart

The fastest path is the single-node Docker Compose deployment.

cd deploy/compose
cp .env.example .env
# Edit .env: set POSTGRES_PASSWORD and CLAWMATES_BOOTSTRAP_OWNER_PASSWORD.

docker compose build      # or load a signed release bundle (deploy/airgapped/install.sh)
docker compose up -d

The server self-migrates and, if CLAWMATES_BOOTSTRAP_OWNER_PASSWORD is set, provisions the first Owner + workspace on first boot. Then:

Backends: openai_compat by default (point [llm].base_url at vLLM/Ollama/llama.cpp); set provider = "anthropic" + ANTHROPIC_API_KEY to use Claude. Auth is local by default or clerk at runtime. See deploy/compose/README.md for all knobs and the broker master-key backup step.

Broker master key. The broker's master key lives in the broker_key named volume and is generated on first boot. Back this up before running the stack for anything real — losing it un-decrypts every stored secret.

Local development:

# Backend (needs a Postgres; sqlx is offline-checked against .sqlx/)
cargo build
cargo test
cargo clippy --all-targets

# Regenerate the sqlx cache after changing any query:
#   DATABASE_URL=… cargo sqlx prepare --workspace

# Frontend
cd frontend
npm install
npm run dev        # also: npm run lint / npm run typecheck / npm run build

Run the reproducible topology benchmark (offline-deterministic; real models with ANTHROPIC_API_KEY):

cargo run -p cm-orchestrator --example topology_bench --features provider

Production deployment (clawmates.work on gw-04)

The public site runs a different path than the airgapped compose. Gitea Actions builds and pushes broker, server, and frontend images to the fleet registry at 100.94.185.103:5000/clawmates/<svc>:latest. gw-04 runs a systemd-timer-driven rolling deploy:

  • Deploy script: deploy/gw-04/clawmates-deploy.sh — polls the registry, drift-checks each service's running image ID against :latest, and calls docker compose up -d <svc> on drift. Portable across docker compose v2 and legacy docker-compose v1.
  • Timer + unit: installed alongside the script under /etc/systemd/system/.
  • Logs: /var/log/clawmates-deploy.log.
  • Compose file: references registry-prefixed images directly — no retag bridging.

gw-04-specific gotchas (bit us on 2026-07-09 and 2026-07-12):

  • clawmates-runtime on gw-04 is not compose-managed — it's a standalone docker run invocation. Provider env (ANTHROPIC_API_KEY, ZEROCLAW_providers__*) must be set on that container.
  • The server container runs as UID 65532 (distroless nonroot). Any bind-mount host path must be chown 65532:65532 before boot or the server can't write.
  • Per-team ZeroClaw containers inherit ZEROCLAW_providers__* from the server; those envs must live on the compose server block, not just on shared runtime.

CI budgets

  • Hard limit: 1500 lines per source file. CI fails.
  • Soft limit: 1100 lines. CI warns — split before it hurts.
  • Enforced by ci/check-loc.sh.

Common split pattern: extract sub-components (MissionLivePane, AutoProvisionCard) into their own file when the parent creeps past the soft limit.


Roadmap

Shipped

  • Pure-Rust topology engine — model → classify → build (all 12 kinds) → execute → compare (Pareto) → workflow → LLM-judge → evolve.
  • Topologies UI: catalog browser, builder/visualizer, multi-topology comparison with a Pareto scatter.
  • Durable topology runs — crash-resumable, checkpointed per step, cancellable, with live SSE.
  • The full deploy ladder — single → team → company → org, with recursive execution down to the leaf claws and a recursive zoom canvas + two-tier navigation.
  • Missions (slices 19) — multi-team model, hard-required team templates, canvas + wizard + list, auto-refresh + Team tab + Live events tab, add/edit/delete toolbar, bulk delete, security-scan + benchmark trigger buttons, per-mission repo checkout on launch, LevelUpInbox mounted.
  • Herdr (phases 03)clawmates-node daemon (systemd/launchd persistence), fleet_herdr dispatch module + node daemon ops, missions runtime_kind + target_node schema, wizard runtime picker with on_launch auto-dispatch, Live Pane (xterm.js → node's herdr TUI), INFRA-tier Herdr sessions surface.
  • Refine on Opus 4.8 — before/after diff view, accept/cancel/restore controls.
  • Level-Up — per-claw/per-team improvement proposals, inbox + review drawer.
  • §15 safety: tool-free sandboxes, the gated MCP "door" (with real email/Slack delivery), the secret broker, allow-listed Docker socket.
  • Self-host: single-node Docker Compose with full network segmentation.
  • Prod path on gw-04: registry-driven rolling deploy via systemd timer.

Next

  • Team / company templates as first-class saved catalogs (compose orgs from reusable building blocks).
  • Per-leaf nested checkpoint resume (today the recursive runner resumes at parent-node granularity).
  • Persona injection into runtime turns (beyond role-driven prompting).
  • Richer per-tier dashboards (company coordination, org portfolio/governance metrics).
  • Group lifecycle management (delete/edit a deployed team/company/org; deprovision its agents) — bulk delete shipped for missions, extending to teams/companies/orgs next.

Research

  • The accompanying paper, Large Dynamic Agentic Topologies (papers/dynamic-agentic-topologies.md): the topology-invariant authority result, the comparison/Pareto methodology, and benchmark results.

Safety

The network segmentation is the security model. Agent sandboxes run with no network at all; only the browser container has egress. The secret broker is reachable only over a private socket and credentials never enter agent code. The server reaches Docker through an allow-listed socket proxy that can manage sandbox containers and nothing else. Every sandbox-leaving action is gated behind a human approval. No topology — and no switch between topologies — can bypass any of this.

S
Description
No description provided
Readme
43 MiB
Languages
Rust 57.9%
TypeScript 24.8%
HTML 8.9%
JavaScript 4.3%
Shell 2.8%
Other 1.3%