`vm_create` takes a backend name and boots `rootfs-<backend>.ext4`; NULL
or "default" boots the golden image. Makes the per-CLI images from B4.1
actually reachable (one image per CLI, per A6).
A missing image is an ERROR naming the file and how to build it, never a
quiet fall back to the default. That fallback is the tempting version and
the wrong one: it would run a claude mission in a kimi VM, or in a rootfs
with no CLI at all, and report success for whatever came out. Verified on
real hardware, not just in a unit test — the selftest asks for an image
that does not exist and FAILS if it boots.
`create` now reports the rootfs that actually booted, not the one that was
requested, so a mission artifact can show the wrong VM ran.
The migration adds no CHECK constraint listing the CLIs. Which images
exist is a property of the NODES, not the schema; a constraint would need
migrating for every new image while still not guaranteeing the image
exists anywhere. The node validates and names what is missing. Backend
names are `[A-Za-z0-9_-]` and rejected rather than sanitised, since they
become filenames.
Verified on tank: default backend 8/8; `CLAWMATES_FC_BACKEND=agent-terminal`
9/9 including the absent-image check, create in 910ms on a rootfs built
from a real Docker image. 435 tests green, no leaked processes or VM dirs.
Co-Authored-By: Claude Opus 5 <[email protected]>
The python guest agent only ever worked because Firecracker's CI Ubuntu
image happens to ship python3. NONE of our images do — agent-base has
neither python nor git, agent-terminal has git but no python — so it
could never have run in a real mission rootfs. An agent that dictates
what must be installed in the image has the dependency backwards.
crates/bins/fcagent is a 905K static x86_64-unknown-linux-musl binary
that needs nothing from the rootfs it is dropped into. The wire is
unchanged on purpose — 4-byte BE length + JSON, ops ping/exec/put/get —
so microvm.rs and microvm_client.rs needed no edit at all.
std has no AF_VSOCK and the workspace denies `unsafe`, so it uses the
`vsock` crate. `process_group(0)` gives each command its own group without
unsafe, so a command that spawns background children can be killed
wholesale rather than outliving the run.
A unit test caught a bug that would have broken EVERY exec: sourcing the
image-env file with `. env.sh 2>/dev/null; cmd` returns rc=1 WITHOUT
running cmd, because `.` on a missing file makes a non-interactive POSIX
shell exit immediately. On any rootfs lacking that file every command
would have failed while looking like an ordinary non-zero exit. Guarded
with `if [ -f ]` now.
Other places a failure must not borrow an outcome's representation: a
killed command reports ok:false with no rc (not rc=124, which would read
as a build failure); `get` on a missing path is an error, not an empty
archive; a signalled process reports 128+signal rather than success.
Verified on tank: --vm-selftest still 8/8 with the agent swapped
(create 949ms, wire identical), fc-node-setup 8/8, and — the point of the
change — a rootfs built from clawmates/agent-terminal:dev, which has NO
python3, boots and reports `git version 2.39.5` from inside the VM.
Also fixes a shell bug in fc-build-rootfs.sh: $HOME in a double-quoted
default expanded on this Mac, so it looked for the node's binary under
/Users/quantum on a Linux host.
Co-Authored-By: Claude Opus 5 <[email protected]>
create / inject / exec / collect / destroy / list, riding the node's
existing frame dispatch ({t, id, …} -> {t:"result", id, ok, output}), so
no protocol change was needed. Control is length-prefixed JSON over
vsock; the serial console stays a log, because feeding a guest over stdin
races its startup and arrives half-consumed.
DEVIATION FROM THE PLAN, deliberately: this does NOT implement
cm_sandbox::SandboxDriver. That trait is container-shaped —
attach_pty/resize_pty/argv exec — while missions need
create -> inject -> run -> collect -> destroy. Conforming would mean
building PTY-over-vsock and window-resize semantics that no mission path
calls, purely to satisfy a signature. We give up automatic RemoteDriver
marshalling; orphan reaping is a label/id sweep either way.
Three traps from the B0 spike are handled in code rather than remembered:
- Firecracker does NOT unlink its vsock UDS on exit, so destroy unlinks
it explicitly, and the selftest ASSERTS it is gone. Assuming the VM
tidies up after itself is how the mission checkout accumulated four
uid bugs.
- firecracker is spawned via setsid and killed as a process GROUP, so a
background child cannot outlive the VM holding its workdir open.
- create does not return until the guest agent has answered a ping. A
VM that booted but serves nothing is worse than one that failed, so a
half-created VM is destroyed rather than left registered.
A vm id becomes a path component, so ids are restricted to [A-Za-z0-9_-]
and REJECTED rather than sanitised — a caller that sent `../../etc`
wanted something we should not guess at.
Verified on tank through the real Rust path, as the daemon user, with no
sudo: `clawmates-node --vm-selftest` -> 8/8, create in 986ms, and the
host left with zero firecracker processes and zero VM directories. The
selftest asserts every step, including that a destroyed VM can no longer
be exec'd; a test that only reports the steps it completed cannot
distinguish "passed" from "stopped early".
Co-Authored-By: Claude Opus 5 <[email protected]>
The first half of removing the shared bind mount. Not wired yet — this
adds the mechanism and its tests.
One cause, four fixes so far: .git/objects permission denied
(core.sharedRepository), the capture base being overwritten each phase,
COMMIT_EDITMSG root-owned, and reset --hard deleting a prior phase's work
(.git/clawmates-in-use). core.sharedRepository was never a general
solution — it covers objects and refs, and every OTHER file git touches
is a fresh opportunity. Copy-in/copy-out removes the cause instead: the
agent owns its filesystem with no second writer.
Measured before building, because the plan named copy cost as the open
risk: a real 65 MB checkout of this repo copies in 0.23s and out 0.18s on
gw-04. Not a risk at this size; re-measure an order of magnitude larger.
No compression — the payload crosses a local socket, so gzip would spend
CPU to save nothing.
Two safety properties, both tested:
- The archive comes back from a container the agent controls as ROOT, so
it is untrusted input. A `../ESCAPED` entry must not write outside the
destination. The test writes the tar header bytes by hand because the
tar crate refuses to BUILD such an entry through its safe API — which
is reassuring, but means the hostile case has to be constructed the way
an attacker would.
- Symlinks are packed as links, never dereferenced. Following them on
copy-IN would smuggle host files into the container; the test plants a
host secret behind a symlink and asserts its contents never appear in
the archive.
Ownership is deliberately not preserved on unpack: the archive's uids are
the container's root, and re-applying them on the host would recreate the
exact uid split this exists to remove.
413 tests, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
Turns the parts into a job. Order is the point: the checkmark list is
consulted BEFORE anything downloads. Checking afterwards would still
dedupe the catalogue while re-downloading every paper we already have,
every week, forever.
Two properties the tests pin down, both learned the hard way this week:
- A quiet week is not a failure. `shelved == 0` with no errors is a
healthy run against a mature library; `shelved == 0` with errors is
broken. Harvest::healthy() and ::added_anything() keep those apart
rather than collapsing them into one ambiguous "did nothing".
- A failed download leaves the paper UNSEEN. Checking it off before the
PDF is safely shelved would mean one transient network error retires
that paper permanently. The checkmark is written last, after the bytes
and the note are both on disk.
The skip test gives every candidate a pdf_url pointing at a closed port,
so if the skip ever regresses the test fails loudly instead of quietly
re-fetching.
Live end-to-end against arXiv, run twice:
RUN1 3 candidates, 0 already held, 3 shelved, 0 failed
RUN2 3 candidates, 3 already held, 0 shelved, 0 failed
Library<'_> groups the five values that always describe one library;
passing them loose is how a run shelves into one place and catalogues
into another (also silences clippy::too_many_arguments honestly rather
than by allow).
391 tests, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
Validating P0 against production found one bug in each of the three pieces,
none of which any test would have caught.
**The scanners were installed but not allow-listed.** Mission 019fc058's
condition asked for a gitleaks result; `gitleaks detect` came back
`ran=false`, and the judge said it could not verify. P0.3 put the binaries in
the image and never added them to `evaluator_tools::ALLOWED_PROGRAMS`, so the
judge could not invoke the tools installed for it. Adds gitleaks, trivy,
semgrep and `which`.
**Every `continue` after a fire claim leaked the claim.** Introduced by the
scheduler fix itself: the orphan-agent and empty-action paths skipped
`complete_fire`, so the row stayed `claimed` — which reads as a crash
mid-fire, meaning the routine is re-claimed forever and the table grows one
stuck row per occurrence. Observed in production: five `claimed` rows, no
dispatch, no `routine_runs`. Both paths now settle with a reason, and log it.
**The agent writes its own identity files into the user's repository.**
`workspace.path` is pinned to the repo root, so the runtime drops AGENTS.md,
HEARTBEAT.md, IDENTITY.md, MEMORY.md, SOUL.md, TOOLS.md and USER.md into the
checkout — SOUL.md opens "Who You Are / You're not a chatbot." Two
consequences: every mission's tree is permanently dirty, so a `done_when`
about a clean tree can never pass; and P1's `git add -A` would have committed
the agent's SOUL.md into someone's repository and pushed it. The P1 deny-list
covered build artifacts and would not have caught this.
Fixed by writing the names to `.git/info/exclude` after clone — local to the
checkout, never itself a change, and it suppresses only *untracked* files, so
a repo that genuinely tracks its own AGENTS.md still reports modifications to
it. Idempotent, and preserves any pre-existing exclude.
Co-Authored-By: Claude Opus 5 <[email protected]>
Mission agents were burning ~275K tokens producing nothing: the coder had
only file_read and its workspace was the empty ephemeral sandbox, so it
dumped a full spec inline instead of writing files. Two root causes:
1. Risk-profile allowlists used pre-0.8 tool names. `coding_readwrite`
allow-listed `file_write` (renamed to `file_edit` in ZeroClaw 0.8, and
`file_write` now refuses on ephemeral workspaces) and omitted file_edit
/ content_search / glob_search / git_operations — the exact tools the
phase prompt tells agents to use. Since allowed_tools is a strict
allowlist, agents were effectively read-only. Documents the correct
profiles in agent.config.example.toml (they only lived in host config;
the live runtime profiles were corrected via its config API).
2. workspace.path never got set. `agents.<alias>.workspace.path` is an
Option<PathBuf> the ZeroClaw Configurable macro skips from prop
enumeration, so provision_claw's set_prop always 404'd and the whole
call errored into a swallowed eprintln. Removes the dead set_prop and
pins the workspace out-of-band: MissionRuntimeProvisioner::
pin_agent_workspaces patches the shared config file on the per-mission
container (format-preserving via toml_edit, atomic temp+mv); the daemon
applies it on the same reload that surfaces the freshly-provisioned
claws. Covered by unit tests for the TOML stamp.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adds a Refine button to the left of Refresh + Launch on the mission
detail toolbar (draft-only). Clicking it POSTs to a new endpoint that
calls Gemini 2.5 Flash to rewrite the user's freeform description into
a coherent, sectioned Markdown brief (Objective / Context / Scope /
Constraints / Acceptance Criteria / Open Questions) ready for the
research + coding agents to ingest cleanly.
Backend:
- crates/cm-api/src/mission_refiner.rs — Gemini call with a
system prompt that preserves user-provided facts, avoids
invention, and emits raw markdown (not JSON).
- POST /api/missions/{id}/refine — draft-only, 400 on empty
description or non-draft state.
- cm-db::repo::missions::set_description helper.
Frontend:
- MarkdownBlock — tiny zero-dep renderer for h1/h2/h3, bullet +
numbered lists, **bold**, `code`, paragraphs. Deliberately
small; the refiner emits a bounded subset.
- MissionCanvas — Refine button (Sparkles icon, secondary style)
to the left of Refresh; description now renders through
MarkdownBlock instead of a single <p>. Disabled while
description is empty or a refine is in flight.
- lib/api/missions — refineMission client.
Adds the missing pieces the wizard needed and the sidebar controls
around it:
- LoopsWizard is now a 6-step flow (identity → repo → task/topology
→ triggers → repeat → assign agents) plus the existing secrets
card. ResearchWizard picks up the same repo step and a hard gate
when the workspace has zero agents.
- New LoopStaffingStep with three tabs — Individual / Team /
Organization — that mix freely per loop; selections persist via
new loop_agents / loop_teams / loop_orgs join tables (0035
migration), each cascading on loop_id so hard-delete stays a
single-row DELETE.
- Backend CreateLoopRequest / UpdateLoopRequest accept the three
lists and apply_staffing does a transactional replace-all;
list_loops / get_loop hydrate the lists via a flattened
LoopWithStaffing response.
- LoopsList sidebar gains per-row enable/disable, edit (reopens the
wizard prefilled with the current loop, PATCHes on submit), and
delete with an inline confirm.
- NoAgentsGate blocks launching a loop or research topic from a
workspace with no roster; the sidebar `+` buttons also disable
with a tooltip pointing at the TEAM tier.
Not yet wired: the run driver still fills role slots from the
workspace-wide pool; teaching enqueue_iteration to prefer
loop_agents/loop_teams/loop_orgs is a follow-up.
CI no longer references k8s; delete the dead k8s surface:
- deploy/helm/ (the chart), ci/check-helm.sh, scripts/netpol-cluster.sh
- cm-sandbox: the feature-gated K8sDriver (src/k8s.rs) + k8s_security test +
the `k8s`/`k8s-tests` features + the optional kube/k8s-openapi/rustls deps
(Cargo.lock drops the kube-rs tree). Nothing outside cm-sandbox referenced it.
Docker (bollard) DockerDriver is the sole sandbox driver. cm-sandbox + cm-runtime
compile, fmt + clippy clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Tap each node's Beszel metrics (GPU/temps/disk-IO/network/per-container — beyond
our basic heartbeat) by reading the workspace's Beszel hub. The agents run in
WS-only mode with no locally-readable socket, so (per the de-risk) the server taps
the hub's PocketBase API instead of the daemon reading agents — no daemon changes.
- migrations: workspace_beszel (BYO hub URL + login, server-side only, mirrors the
Tailscale BYO pattern) + node_metrics (latest scalar columns + JSONB blob).
- cm-db: repo/fleet_beszel.rs, repo/node_metrics.rs; nodes SELECT joins node_metrics
(gpu_pct/temp_max surfaced on node_json for the live cards).
- cm-api: beszel.rs client (auth-with-password, poll `systems`, map to nodes by
hostname, upsert metrics) + a 15s spawn_poller; routes/beszel.rs (connect/status/
disconnect + GET /api/nodes/{id}/metrics with history proxied live from the hub).
- frontend: HostCard gains a GPU/temp readout + a Monitor button; NodeMonitor is a
full-width per-node page (current panel + CPU/mem/GPU/temp/net/disk charts from the
hub's 1m history); a "Beszel monitoring" connect form in the Local view.
Reachability confirmed: gw-04 → the hub over the tailnet (100.123.224.84:8090). Needs
the user to connect their hub login to activate the poller.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Terminal keystrokes were ~400ms because every byte relayed browser→Cloudflare→
gw-04 (Europe)→tailscale→node, even when the node is on the user's own LAN. Add a
direct browser↔node WebRTC DataChannel so co-located terminals run at LAN speed;
the gateway is reduced to signaling; the WebSocket relay stays as the automatic
fallback (graceful degradation — never worse than before).
Daemon (clawmates-node v0.4.0, new src/rtc.rs):
- Add the `webrtc` crate (reuses the ring crypto provider we already install — no
conflict). Browser is the offerer; we answer, trickle ICE back over the control
channel, and on DataChannel open spawn a host PTY (tmux) bridged DIRECTLY to the
channel. Refactor open_pty → spawn_terminal_pty shared by both transports.
iceServers: STUN + auto host/tailnet candidates (direct, no relay, for LAN/tailnet).
Server (cm-api):
- NodeConn.signal_sinks; Uplink WebRtcAnswer/WebRtcIce/WebRtcFailed routed to the
browser; NodeHub webrtc_offer/ice/close + open_session/open_pty (open_terminal
split so the PTY opens only once the transport is chosen). bridge_terminal relays
signaling over the existing ticket-authed WS and opens the relay PTY on
{type:"fallback"}.
Browser (NodeTerminalApp):
- RTCPeerConnection + reliable/ordered DataChannel; offer/answer/ICE over the WS;
2.5s race → use the DataChannel if it opens, else fall back to the WS relay.
Reconnect wraps both. A direct⚡/relayed indicator shows the live transport.
Deployed; both nodes (morpheus, tank) updated to v0.4.0 and steady online. Direct-
path proof is a browser action (the ⚡ indicator + latency); confirmable from the
daemon's [rtc] logs.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Nodes flapped online/offline and the terminal died on the first blip. WebSockets
are the right transport (outbound, NAT-friendly); the fixes harden around it.
Server (cm-api):
- Anti-clobber connection epoch: a reconnecting daemon gets a fresh epoch; a stale
run_channel's teardown only clears the hub + sets offline if it still owns the
slot — so a lingering old channel can't flip a live reconnection offline (the
main false-offline cause).
- WS keepalive: run_channel now pings every 15s and tears down if no inbound
frame (incl. pong) for 35s — dead links detected in seconds, not minutes.
- Staleness sweeper backstop: spawn_node_sweeper (8s tick / 20s window) wired in
clawmates-server, so a vanished node goes offline within ~28s even if its
channel hangs (mark_stale_offline was defined but never called).
Daemon (clawmates-node v0.3.0):
- Heartbeats off the select thread (dedicated thread owns System + blocking
docker/tailscale/disk CLIs) so a slow op never starves heartbeats/pongs.
- Each handle_frame runs on its own task; added a 40s inbound idle deadline so a
half-open socket triggers a reconnect.
Frontend:
- useNodes streams /api/nodes/live (SSE push) instead of a 3s poll; isLive()
derives online from lastSeen freshness (<15s) so a transient column flip never
shows a healthy node down.
- Node terminal: clean auto-reconnect loop (re-mint ticket -> reconnect -> tmux
re-attaches and redraws the live screen = mosh-style snap-to-state over TCP),
replacing the [disconnected] dead-end.
Mosh evaluated: harvest principles (session/transport decoupling, snap-to-state,
already given by tmux), don't adopt — UDP is incompatible with our browser+CF+NAT
topology and it's GPLv3. Removed temporary terminal debug traces + /api/debug route.
Verified: node holds steadily online (heartbeat 1-3s, no flap) and goes cleanly
offline when the daemon stops.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Root cause (from the server trace: 69 banner bytes then immediate "to_browser
ended" = pty_exit): tmux spawned and exited instantly, writing its "sessions
should be nested with care" warning to stderr (not the PTY). The operator runs
the daemon inside their own tmux, so $TMUX was inherited and the spawned tmux
refused to nest → browser saw nothing.
- daemon (v0.2.2): spawn tmux on a DEDICATED socket (`tmux -L clawmates
new-session -A -s main`) and `env_remove("TMUX")`, so it can never collide with
or be refused by the operator's tmux.
- NodeTerminalApp: debounce the ResizeObserver (150ms). The pull-out animates
open, firing the observer on every pixel — previously ~80 resize frames per
open, each fit()+SIGWINCH. Now one resize after layout settles.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Two bugs surfaced running the daemon on a real node:
- Linking cm-sandbox (bollard) brought a second rustls provider into the graph,
so rustls couldn't auto-pick one and panicked at the WSS TLS handshake. Install
the ring provider explicitly at startup (rustls dep + install_default()).
- The daemon auto-ran `tailscale set --ssh`, which tries to reroute the user's
live SSH session and aborts ("will result in your session disconnecting"). Now
Tailscale is only touched when an auth key is explicitly passed (opt-in), with
--accept-risk=lose-ssh to avoid the interactive abort.
Rebuilt + re-hosted both binaries (linux-amd64, darwin-arm64).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Agents can now provision their sandbox on a connected fleet node instead of the
gateway host. Local stays the strict default, so existing agents are byte-for-
byte unaffected until explicitly placed elsewhere.
Security parity: the daemon links the REAL cm-sandbox DockerDriver and runs the
typed container ops (sb_provision/sb_exec/sb_destroy/sb_health/sb_list) through
it — identical hardening (cap-drop ALL, seccomp, no-net, read-only, non-root) to
local sandboxes. cm-sandbox spec types are now Serialize/Deserialize so the spec
crosses the channel.
- cm-api: RemoteDriver (impl SandboxDriver over the node channel) + HubDriverProvider
(impl cm_runtime::NodeDriverProvider, hands out a driver only for connected
nodes via a sync online set) + NodeHub.call/is_connected. AppState.with_node_hub
so the hub is shared with the placement provider.
- cm-runtime SandboxManager: driver_for(node_id) routes by the recorded
agent_containers.node_id (local default = existing driver, identical path);
placement_node() reads the workspace setting and falls back to local if the
node is offline; exec/release route accordingly. NodeDriverProvider trait.
- DB: 0020_workspace_placement + repo (for_agent/get/set/clear).
- main.rs: build the NodeHub first; inject HubDriverProvider into the agent
manager + share the hub with AppState.
- API+UI: GET/PUT /api/fleet/placement + a "Run agents on: Local / <node>"
selector in the Fleet overview.
Note: a node must be able to pull the agent image (the daemon docker-pulls it);
interactive PTY for agent containers on remote nodes is not wired (Terminal app
stays local) — the in-dashboard node shell already covers host access.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
You can now open a real shell on any connected node from the dashboard — the
daemon spawns a host PTY and streams it over the existing outbound control
channel (no inbound port, no Tailscale brokering needed).
Daemon:
- portable-pty host shell sessions: pty_open/pty_in/pty_resize/pty_close ops; a
reader thread streams base64 pty_out frames. Outbound frames now funnel through
one mpsc channel so PTY output and heartbeats interleave.
cm-api NodeHub:
- per-connection pty_sinks + sid multiplexing; open_terminal/terminal_input/
terminal_resize/terminal_close; in-memory single-use terminal tickets (the
browser WS can't carry a bearer, and the session is instance-local anyway).
- routes/nodes.rs: POST /api/nodes/{id}/terminal/ticket + GET .../terminal/ws
(bridges browser xterm <-> node PTY: binary = keystrokes, text = resize).
Frontend:
- NodeTerminal xterm modal (reuses the agent Terminal's xterm setup); a Terminal
button on each online node card opens a shell.
This proves the bidirectional streaming-over-channel mechanism the RemoteDriver
will reuse. Remaining P2: RemoteDriver + placement (run agent workloads on nodes).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The §15 door's email_send queues to `outbox` but nothing delivered it. Add a
real transport: cm-db outbox repo (list_queued/mark_sent/mark_failed) + a
cm-runtime drainer — an EmailSender trait (testable), a lettre STARTTLS
LettreSender, drain_once (queued -> sent/failed), and spawn_drainer wired into
the server beside the scheduler/sweeper/topology-worker.
Config-gated: inert (logs "outbox delivery DISABLED") until CLAWMATES_SMTP_*
is set, so it ships safely before credentials exist. The agent never holds the
SMTP credential — it only writes to outbox through the gated door; the server
owns the transport.
NOTE: live delivery is still credential-blocked — Migadu's API can't send
(SMTP-only) and the admin token is invalid; no SMTP creds exist. The transport
is built + tested (drain_marks_sent_and_failed via a mock sender); set
CLAWMATES_SMTP_* to go live with zero further code. clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
cm-orchestrator owns the topology graph; each turn now drives a real ZeroClaw
role-agent in a container via the proven gateway drive recipe, instead of a
tool-free cm-llm call.
- crates/cm-api/src/topology_exec.rs: ZeroClawDriveExecutor impl TurnExecutor —
pair (POST /pair + X-Pairing-Code, token cached) -> ws /ws/chat?agent=<alias>
-> send {type:message,content} -> drain chunk/done/approval_request/error.
approval_request is recorded as a BLOCKED GatedAction, never auto-approved (§15).
Role->alias via ZEROCLAW_AGENT_MAP, fallback ZEROCLAW_DEFAULT_AGENT (scout).
- POST /api/topologies/run {task,graph} -> execute() -> RunRecord, persisted
best-effort to the existing topology_runs table (no migration). compare stays
tool-free. from_env() is read in-handler so cm-api still boots unset.
- deploy/clawmates-runtime: example config now declares a tool-free multi-agent
role-cast; README documents the ZEROCLAW_* knobs + run endpoint.
tokio-tungstenite 0.26 (already in lock) + dev axum `ws` for the hermetic test.
3 lib tests green, clippy clean, SQLX_OFFLINE build clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Run a task across topologies via the API and return a leaderboard + quality/cost
Pareto front. The handler builds a ProviderExecutor + JudgeScorer over the
runtime's configured provider/model, then calls cm_orchestrator::compare.
- cm-runtime: expose provider()/model()/max_tokens() accessors on Runtime.
- cm-api: depend on cm-orchestrator (provider feature); add the compare route.
- Integration test runs a 2-topology comparison through the real server
(scripted provider) → 200 with results + leaderboard. 4 topology tests green;
offline build + clippy clean.
The topology endpoints (catalog/classify/build/compare) ship to gw-04 with the
upcoming ReactFlow UI in one server+frontend redeploy.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Stateless, auth-gated routes backing the topology builder UI:
- GET /api/topologies — catalog of the 12 kinds + descriptions + role mix
- POST /api/topologies/classify — infer a kind + metrics from a posted graph
- POST /api/topologies/build — build a canonical graph from {kind, roles}
Add Serialize to cm-topology Classification/GraphMetrics; add ApiError::BadRequest
(400) for invalid build input; add cm-topology dep. 3 integration tests
(catalog/build/auth) green against Postgres; offline build + clippy clean.
No DB or provider yet — running/comparing topologies is a later provider-backed
endpoint. Server redeploy will batch with the ReactFlow UI that consumes these.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add a real TurnExecutor (behind the `provider` feature) that runs each
topology turn as a single tool-free LLM call via any cm-llm provider
(Anthropic / OpenAI-compat / scripted). Topologies now execute real model
calls and produce real metrics (tokens), feeding the comparison harness.
- ProviderExecutor builds a per-role system prompt + threads upstream context
into the user message; collects TextDelta → output, Usage → tokens.
- Tool-free reasoning turns take no sandbox-leaving actions (gated = []); §15
remains satisfied. Tool-using turns will route through a cm-runtime adapter.
- Core crate stays dependency-light; cm-llm/futures are optional (feature).
- Tests: pipeline + hierarchical run over the deterministic scripted provider
(7 tests with --features provider). Clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A pure async control-flow engine that executes a task across a TopologyGraph
by sequencing safe agent turns. Safety by construction: the engine can only
invoke turns via a generic TurnExecutor — it performs no side effects itself,
so §15 gating (inside each turn) is inherited and switching topology cannot
escalate authority.
- TurnExecutor trait + TurnRequest/TurnOutcome (real impl will wrap
cm-runtime::Runtime; tests use a scripted Echo executor).
- Pure planners (plan.rs): hierarchical (delegate down / synthesize up),
pipeline (topo-ordered threading), swarm (parallel attempts + aggregate).
- RunRecord journal (per-step + RunMetrics: tokens, gated actions, approvals
granted/blocked, turns) — feeds the Phase 4 comparison harness/paper.
- Unsupported kinds return an error (no panic). 5 tests, clippy clean.
Next (Phase 2b): a real TurnExecutor adapter over cm-runtime::send_message.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Last open item from the roadmap + post-1.0 list. Run against the live
Clerk instance closing-seasnail-39.clerk.accounts.dev.
- Backend (crates/cm-auth/tests/live_clerk.rs, CM_LIVE_CLERK=1): pulls
REAL discovery + JWKS from the live instance, mints a REAL session JWT
via Clerk's Backend API (create user -> open session -> session token),
and runs it through AuthService::authenticate — verify + JIT provision
(keyed on the real sub), duplicate-subject suppression, tamper
rejection against the live JWKS. Decodes the instance domain from the
publishable key; cleans up the test user after. PASSING
- Frontend: built with AUTH_MODE=clerk + real keys, next start serves
Clerk's <SignIn /> at /login wired to the instance (instance domain +
data-clerk attributes present in the HTML). Both halves confirmed
end to end against production Clerk
- docs/clerk.md: documented the smoke procedure for both halves
166 Rust tests (+6 live, key-gated). Keys used via env only, never
stored — rotate them (they passed through chat).
Co-Authored-By: Claude Fable 5 <[email protected]>
Full-depth rename per the approved plan; the 'claw' product vocabulary
(claws, /claws routes, clawId, Claw Chat) stays — it is now the brand.
- Display brand: Clawmates (manifest, titles, hero, login/rail logo
'clawmates'); default host app.clawmates.work; registry
ghcr.io/clawmates
- Crates tc-* -> cm-* (16 crates + all imports); binaries
clawmates-server/broker/bundler; images clawmates/*; env prefix
CLAWMATES_* (+ CM_TEST_DATABASE_URL / CM_LIVE_LLM); config
clawmates.toml; helm chart deploy/helm/clawmates with clawmates-*
resources; db names clawmates*; sockets /run/clawmates; cookie
cm_session; kind cluster clawmates-test; seccomp node profile
clawmates-agent-profile.json
- All 9 Playwright brand assertions updated in lockstep; historical
spec document left untouched as the only remaining 'TeamClaw'
- Local env migrated: dev pg clawmates-dev-pg/clawmates_dev, shared
test server clawmates-test-pg, kind cluster recreated with image +
profile, compose images rebuilt under clawmates/*
Verified end to end: 161 Rust + 68 frontend tests, 29 Playwright
journeys, 4 live kind tests, helm/install/LOC/placeholder gates, and
the clean-room install rehearsal serving the clawmates login page from
a signed bundle of the rebuilt images.
Co-Authored-By: Claude Fable 5 <[email protected]>
- tc-telemetry: fmt subscriber always; with [telemetry] otlp_endpoint
set, spans batch-export over OTLP/HTTP. Tested against a REAL OTLP
receiver decoding the actual protobuf (official proto types): the
emitted span and service.name arrive on the wire. No endpoint = no
export = no network (air-gap stance). tower-http TraceLayer gives
every API request a span
- The broker finally has its own image (images/broker.Dockerfile,
9.5MB from scratch) — the Helm chart referenced one that never
existed — and the compose deployment now RUNS the broker, sharing a
socket volume with the server (the unix-socket equivalent of the K8s
sidecar). Compose secret flows were silently dead before this
- server.Dockerfile fixes surfaced by the rehearsal: the workspace
build needs tools/ (bundler joined the workspace) and
images/seccomp/ (include_str! profile) in the build context
- scripts/rehearse-install.sh (plan: clean-VM rehearsal): assembles a
REAL signed bundle from the built images (server/frontend/broker/
postgres/socket-proxy), runs the customer path — offline verify,
docker load, compose up — and asserts /healthz plus the served login
page before teardown. Passing locally; wired as a release.yml step,
which also builds/ships the broker + socket-proxy images now
161 Rust tests + 29 journeys; clean-room rehearsal green.
Co-Authored-By: Claude Fable 5 <[email protected]>
- The compose server NEVER sees the raw Docker socket (plan risk #5):
tecnativa/docker-socket-proxy on an isolated engine_net with exactly
CONTAINERS/POST/EXEC/DELETE/VERSION allowed; server reaches it via
DOCKER_HOST. DockerDriver honors DOCKER_HOST (connect_to). Proven by a
REAL proxy test: full sandbox lifecycle works through the allowlist
while /networks, /secrets, and /images all 403 — the blast-radius cap
if the server is ever owned. (This also fixes compose deployments,
where sandbox provisioning previously had no engine access at all.)
- Gateway load test (plan P6): 40 concurrent SSE streams against one
server — every run completes with the full §13 event vocabulary,
every journal strictly monotonic, every resumeFrom=0 replay byte-equal
to its live stream
- release.yml: SBOMs (syft, spdx-json) for all four images shipped
INSIDE the signed bundle; final verification now runs in a
--network none container — proving the customer's verify path needs
no internet, not just claiming it
159 Rust tests.
Co-Authored-By: Claude Fable 5 <[email protected]>
- tc-auth JwtVerifier: OIDC discovery -> JWKS, RS256 with the issuer
pinned, 5s leeway (the crate's default 60s would double the life of
Clerk's 60s session tokens), key cache with one refresh on unknown kid
(Clerk rotates). Serves auth.mode = clerk AND generic oidc — a Clerk
instance IS an OIDC issuer, so one verifier covers both
- AuthService.authenticate dispatches: JWT-shaped bearers take the
hosted-identity path, everything else stays a local opaque session.
External users JIT-provision keyed by the stable sub claim
(users.auth_subject, unique partial index in migration 0007); an
existing local account with the same email is LINKED, not duplicated;
role tracks the issuer claim every request (org:admin -> Owner)
- Config auth.mode = "clerk" (requires issuer_url; validated), server
pins the issuer at boot, Helm values/configmap accept mode=clerk
- Tests with REAL crypto, no mocks: fresh RSA keypairs, a live local
issuer publishing real discovery + JWKS docs, Clerk-shaped tokens —
JIT + role mapping, repeat-subject no-dup, expired refused (leeway
regression), wrong-key forgery refused, foreign issuer refused, and
the full router round trip with Authorization: Bearer <session JWT>
- docs/clerk.md: dashboard session-token customization (email + org
role claims), config, @clerk/nextjs getToken() wiring, what CI proves
157 Rust + 63 frontend tests + 29 journeys. Air-gapped installs keep
local auth — Clerk is a cloud-only alternative, not a replacement.
Co-Authored-By: Claude Fable 5 <[email protected]>
- SandboxManager (tc-runtime): one container per agent, provisioned
lazily on first use, reused for the manager's lifetime, replaced
transparently if dead, destroyed on shutdown
- shell.exec tool: sh -lc inside the agent's sandbox; stdout/stderr/
exit_code return to the model as the step output. No external effects
declared — the sandbox boundary (uid 10001, no caps, seccomp
allowlist, read-only rootfs, zero egress) is the §15 control here,
not an approval gate
- RuntimeConfig.sandboxes (+ with_sandboxes builder); [sandbox] config
{image, enabled}; the server connects the Docker driver at boot and
tolerates an absent engine (shell.exec reports it per-call)
- Tests with the REAL DockerDriver: a scripted run executes two
commands — output proves uid 10001 from inside, and /home/agent state
written by the first call is read by the second (same sandbox); a
deployment without a sandbox runtime records honest error steps and
the run still completes
151 Rust tests + 27 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- images/seccomp/agent-profile.json is now a TRUE ALLOWLIST: Docker's
default profile (vendored from moby v27.5.1, defaultAction ERRNO) with
18 syscalls an agent never needs stripped from the allow groups
(unshare, ptrace, bpf, mount family, setns, module loading,
perf_event_open, process_vm_*, reboot, quotactl, ...); arch map trimmed
to x86_64 + aarch64. All 6 Docker kernel assertions still green.
- K8sDriver (tc-sandbox feature 'k8s', kube-rs): one hardened pod per
sandbox — runAsUser 10001, cap-drop ALL, no-new-privs via
allowPrivilegeEscalation=false, RuntimeDefault seccomp, read-only
rootfs with emptyDir /tmp + /home/agent, resource limits, no service
account token — in a PSS-restricted namespace carrying a default-deny
NetworkPolicy (applied server-side apply, idempotent). Exec via the API
server attach channel with exit codes parsed from v1.Status.
- Live suite (feature 'k8s-tests') against a REAL kind cluster: uid /
CapEff==0 / NoNewPrivs / rootfs probes from inside pods, PSS label +
deny-all policy asserted via the API, lifecycle. Honest limits in the
rustdoc: Localhost seccomp profile and CNI-enforced egress are
per-cluster provisioning (kindnet does not enforce NetworkPolicy).
- rustls 0.23 process provider pinned to ring at driver connect.
- CI: dedicated sandbox-k8s job (helm/kind-action) running the suite.
149 Rust tests + 3 live kind tests; clippy clean including the k8s feature.
Co-Authored-By: Claude Fable 5 <[email protected]>
- S3BlobStore (object_store, path-style) behind the same BlobStore trait,
tested against a REAL MinIO container (round trip, overwrite, NotFound
on get and delete, nested keys); [storage] backend=local|s3 config with
validation + server-side selection (S3 creds via env overlay)
- Helm chart: server pod with the secret broker as a SIDECAR sharing a
private emptyDir unix socket (no network hop carries credentials),
frontend, optional local PVC vs S3, OIDC/oauth values, unbuffered-SSE
ingress annotations, NetworkPolicies (frontend->server only), hardened
securityContexts; ci/check-helm.sh lints AND asserts the rendered
topology properties
- deploy/airgapped/install.sh: offline signature+checksum verification via
the bundled teamclaw-bundler BEFORE any docker load; --verify-only mode;
ci/test-install.sh rehearses clean/tampered/wrong-key paths with the
real binary
- CI: helm gate + installer rehearsal wired in
149 Rust tests; helm lint + rendered assertions green; installer
verify-path rehearsal green.
Co-Authored-By: Claude Fable 5 <[email protected]>
- Concurrency soak (exit criterion): 12 concurrent gated runs, every
decision attempted twice concurrently, explicit resumes racing the
durable sweeper — exactly one execution per approval, grants consumed
at most once, every decision audited, zero stuck runs, zero unaudited
executions. (Testkit pool raised to 20 connections; the 5-connection
pool starved the storm.)
- axe a11y sweep (exit criterion): serious+critical violations fail CI on
login, shell, chat, computer home, settings app, all global pages, and
the wizard. Two real violations found and fixed: aria-label on a plain
div (wizard progress -> role=group) and a button directly inside a <dl>
(settings -> plain bordered list).
- tools/bundler (exit criterion): keygen / assemble / verify CLI — copies
artifacts, writes manifest.json + sha256 checksums.txt + a detached
ed25519 signature; verification is fully offline (keyless signing is
internet-dependent and disqualified). Tests: round trip, tampered
artifact caught by hash, tampered checksum list caught by signature,
wrong key refused, missing artifact reported.
147 Rust + 63 frontend tests + 27 Playwright journeys (incl. 4 a11y).
Co-Authored-By: Claude Fable 5 <[email protected]>
- LlmEvent::Usage across all three providers (Scripted deterministic
word-count accounting; Anthropic message_start/delta usage; OpenAI-compat
stream_options include_usage)
- tc-billing: ceil(tokens/1000) min 1 credit; lots drain oldest-first under
FOR UPDATE; balance clamps at zero while the usage ledger records the
full obligation; promo codes redeem exactly once via CAS (migration 0006)
- Runtime charges every completed run (billing failure never fails a run);
proven: 1 token in + 3 out -> 1 credit deducted
- API: GET /api/team/usage, POST /api/credits/redeem (409 on reuse, audited)
- Credits page: balance, 7-day usage meter with runway estimate, PromoRedeem
- /claws/new is the full §9 wizard: ?step=identity|access|slack deep-linked
progress, accent swatches + name randomizer, access toggles, optional
Slack step, explicit review-and-confirm (creation = live agent), animated
provisioning state -> straight into chat
- E2E: chat decrements the visible balance and fills the usage meter;
WELCOME500 adds exactly 500 once then refuses; wizard round trip
140 Rust + 63 frontend tests + 23 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- migration 0005 oauth_states: one-time states (10-min TTL), consumed by a
CAS DELETE on callback — replays and forgeries both 404
- POST /api/apps/oauth/start: OIDC discovery on the configured issuer (or
the custom MCP issuer for authType=mcp_oauth), state row, authorize URL
- GET /api/apps/oauth/callback: code exchanged at the REAL token endpoint
(client id+secret form POST); the access token goes straight to the
broker (test proves it never appears unencrypted in Postgres); connection
row + audit; redirects to the claw's Add Apps panel
- [oauth] config (issuer/client/redirect_base) wired through AppState
- Tests against a real local IdP server (discovery + validating token
endpoint): full round trip, broker-held token, replay/forged state
refused, bad code fails exchange, mcp_oauth uses the custom issuer while
plain oauth refuses without a configured IdP
- AddAppsApp: live connection badges + inline API-key connect per app
(E2E: connect Notion by key from the directory)
136 Rust + 63 frontend tests + 21 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- Broker op VerifySlackSignature: v0 HMAC-SHA256 computed INSIDE the broker
(constant-time compare); the signing secret never crosses the socket.
Slack secrets are one JSON credential {bot_token, signing_secret}; the
broker extracts the right field per operation
- Public POST /api/slack/events: signature verified against connected slack
connections via the broker; forged signatures 401; url_verification
handshake echoed only when signed; app_mention starts a real run in the
agent's dedicated '💬 Slack' session — and the agent's reply is itself a
gated outbound post
- SlackApp Connection tab captures bot token + signing secret
- Integration test: forged 401, signed challenge, signed mention -> run ->
slack.post pending in the approval queue
- E2E: full loop — connect, gated outbound (sink empty -> exactly one post),
then a node-crypto-signed mention -> approval card -> approve -> 'On it!'
lands in the sink
134 Rust + 63 frontend tests + 21 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- app_connections repo; POST /api/apps/connect (keys/basic): the credential
goes to the secret broker over its socket and only the encrypted ref lands
in the row; disconnect endpoint; /api/apps directory merged with live
connection status; audit rows for connect/disconnect
- Broker protocol: InvokeHttp carries a JSON body
- slack.post tool (SendsExternally -> gated): marked broker_executed — the
runtime skips its own grant consumption and the BROKER independently
verifies + consumes the single-use grant, then calls Slack with the bot
token injected; the runtime never sees the credential
- Config: [broker] socket_path + [slack] base_url; e2e harness spawns the
real teamclaw-broker daemon and the server hosts an e2e-only /__slack sink
- SlackApp: Connection tab stores the token via the broker; connected state
- Integration test: blocked while pending -> approved -> sink received
exactly one post with 'Bearer xoxb-test-token' -> grant replay refused
- E2E journey: connect Slack in the panel -> gated post card with preview ->
sink empty while pending -> approve -> exactly one post, queue clear
133 Rust + 63 frontend tests + 21 Playwright journeys.
Co-Authored-By: Claude Fable 5 <[email protected]>
- tc-db: sessions/messages/steps/runs/run_events repos (atomic seq assignment,
history with ordered step traces, journal replay-from-offset); migration 0003
- tc-llm: provider-neutral ChatRequest/LlmEvent; ScriptedProvider (scenario
TOML, word-level deltas, multi-turn tool legs — ships in production for
e2e/air-gap smoke), AnthropicProvider (Messages SSE), OpenAiCompatProvider
(vLLM/Ollama/llama.cpp); opt-in live tests via TC_LIVE_LLM=1
- tc-runtime: run loop with persist-before-emit event journal, real built-in
clock.now tool, step rows on the reply message, tool-error resilience,
broadcast channels for live attach
- tc-api: agent CRUD + settings/full (tenant-isolated, RBAC'd, audited),
sessions create/list/history?tools=true, POST /api/gateway SSE with
monotonic ids and exact resumeFrom journal replay (tested equal to live)
- teamclaw-server: config-driven provider factory
83 Rust tests green, all against real Postgres / real TCP.
Co-Authored-By: Claude Fable 5 <[email protected]>
