`team_engine='composed'` (the third name migration 0069 anticipated) runs a
mission as a durable ZeroClaw graph whose every node is a whole
Claude-Code-in-a-microVM session. Engine Z owns checkpoint/resume, cancellation
and per-node heterogeneity; Engine C owns shared context and cheap fan-out;
neither has the other's asset, which is why this is a composition and not a
compromise.
`MicroVmTurnExecutor` implements the existing `TurnExecutor`, so it inherits the
planners, the checkpoint, the stale-run recovery, `close_finished_phases`, the
evaluator, capture and delivery unchanged — the same trick `SubTopologyExecutor`
already plays with a heavy `run_turn`. Producer side emits ONE `queued` row
carrying the real graph and lets the worker claim it: the durability IS being
worker-driven, and the solo path's `tokio::spawn` has none of it. Still exactly
one `topology_runs` row per unit of work and one completion path — `finish()` is
now that one place, shared by every tier.
THE TRAP, solved and proven. A VM is inject → run → collect → destroy, so a
per-node VM with text-only handoff silently loses every file an earlier node
wrote: node 2 boots from the original checkout, sees nothing, and still reports
success. The mission's host checkout is the medium — every node injects from it
and collects back over it — and two properties make that safe rather than lucky:
`execute_resumable` is strictly sequential, so two VMs never write one directory;
and the vm id is deterministic per (phase, iteration, step), so a duplicate is
refused by the node ("vm already exists") instead of becoming a second writer.
NEGATIVE CONTROL, run rather than assumed: with `repo` swapped for a private
per-node workspace, `a_later_node_sees_an_earlier_nodes_files` FAILS with
`saw:[]`; restored, it passes. The `PhaseVm` seam exists for exactly this — it
models inject/collect through the real `mission_fs` tar path in milliseconds.
Two durability traps this tier walks into, both closed:
- `requeue_stale` fires at 180s on `updated_at`, and one node here can run for
an hour. `SubTopologyExecutor` keeps its parent alive from each leaf step;
there is nothing between the start and end of a VM turn, so the turn holds a
ticker that touches `updated_at` every 30s and aborts on drop. Without it a
healthy composed run is requeued mid-node and boots a second VM.
- the 15-minute stuck-run reaper asks "any step records since it was CREATED?",
which describes a healthy composed run as readily as a wedged one. Hence
`REAPABLE_TIERS` — worker-driven minus this tier. Reaping it would be #54 in
a different costume.
`on_launch` mints no team for a microVM mission, deliberately: claws in
containers are what a VM mission does not use. So `mission_orchestrator::
composed_graph` builds the shape from the team template directly — nodes, roles
and pattern, zero claws provisioned. Per-node `attrs["backend"]` and
`attrs["node_id"]` override the mission's, which is what makes a validator node
on another provider's image a first-class graph node; a malformed `node_id`
fails the node rather than quietly running it where the graph did not ask.
Refusals are recorded as a failed run, not returned as an error: `launch_phase`
is swept every ten seconds, so a returned error is a phase that retries forever
while the log repeats itself.
501 tests pass, clippy clean. NOT yet proven end to end: no composed mission has
run on the fleet, so the resume-after-a-killed-worker leg is argued from the DB
test and the step-numbering test, not from a real two-node run.
Co-Authored-By: Claude Opus 5 <[email protected]>
The #54 fix had no test that could see it. Its defining property is that it only
appears past 180 seconds, and `verify-mission-delivery.sh microvm` runs a
90-second mission — so the end-to-end harness written to catch silent failure was
structurally blind to this one. A unit test asserting the allowlist's membership
helps, but would not notice a NEW sweeper added without the filter.
`crates/cm-db/tests/self_driven_runs.rs` tests the real SQL against a migrated
database, in milliseconds instead of eight minutes:
- a `microvm` and a `session` run, 30 minutes idle and still `running`, must be
left alone by `requeue_stale` — that is the bug, in one assertion
- a `team` run in the SAME state must still be requeued, so the fix is "sweep the
right rows" and not "stop sweeping"
- the worker must not CLAIM a queued self-driven row, which is what turned a
healthy run into "missing or invalid graph"
- the allowlist names only worker-driven tiers
NEGATIVE CONTROL, run rather than assumed: with the tier filter removed from
`requeue_stale`, `requeue_stale_leaves_self_driven_runs_alone` FAILS; restored, it
passes. A guard that cannot detect the bug it was written for is decoration, and
this project has shipped one of those before.
489 tests pass, clippy clean.
`upsert_builtin` replaced the role set with DELETE + reinsert. That looks
equivalent to an upsert and is not: `agent_template_link` carries a plain FK
on (template_id, role_slot), so the delete is rejected as soon as one agent
has been minted from the template, rolling back the whole transaction.
The failure mode was silent and self-targeting. The loader logs the error and
continues, so the on-disk TOML and the DB drifted apart — and only for the
templates someone had actually used. Running the smoke mission against
insight_research is what put it on the boot log:
failed to load insight_research.toml: violates foreign key constraint
"agent_template_link_template_id_role_slot_fkey"
which also means that template never received the skill-name fix.
- Upsert each role in place via ON CONFLICT (template_id, slot), the table's
primary key.
- Prune only slots the TOML dropped, and skip a slot still referenced by a
live agent with a log line. Keeping one stale role row is a smaller failure
than discarding every edit to the template.
- Regression test drives the real sequence — upsert, mint an agent, link it,
upsert again — and asserts both the prompt and skill edits land. Verified to
fail without the fix with the same 23503 the server logged.
Co-Authored-By: Claude Opus 5 <[email protected]>
The a2a merge (d0d8e7f) landed with a handful of pre-existing rustfmt
diffs that were failing `cargo fmt --all --check` in CI. Pure whitespace
reformatting from `cargo fmt --all`; no semantic changes. Files touched:
mcp_door.rs, quota.rs, routes/a2a.rs, routes/world.rs, runtime_provision.rs,
chat_repos.rs test, and tools/chat.rs.
Full-depth rename per the approved plan; the 'claw' product vocabulary
(claws, /claws routes, clawId, Claw Chat) stays — it is now the brand.
- Display brand: Clawmates (manifest, titles, hero, login/rail logo
'clawmates'); default host app.clawmates.work; registry
ghcr.io/clawmates
- Crates tc-* -> cm-* (16 crates + all imports); binaries
clawmates-server/broker/bundler; images clawmates/*; env prefix
CLAWMATES_* (+ CM_TEST_DATABASE_URL / CM_LIVE_LLM); config
clawmates.toml; helm chart deploy/helm/clawmates with clawmates-*
resources; db names clawmates*; sockets /run/clawmates; cookie
cm_session; kind cluster clawmates-test; seccomp node profile
clawmates-agent-profile.json
- All 9 Playwright brand assertions updated in lockstep; historical
spec document left untouched as the only remaining 'TeamClaw'
- Local env migrated: dev pg clawmates-dev-pg/clawmates_dev, shared
test server clawmates-test-pg, kind cluster recreated with image +
profile, compose images rebuilt under clawmates/*
Verified end to end: 161 Rust + 68 frontend tests, 29 Playwright
journeys, 4 live kind tests, helm/install/LOC/placeholder gates, and
the clean-room install rehearsal serving the clawmates login page from
a signed bundle of the rebuilt images.
Co-Authored-By: Claude Fable 5 <[email protected]>
