248948cc847b4d229291fa65785d940b02fd36ca
107
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar SobhandClaude Opus 5
|
113de610ec |
fix(security): a signed Slack request could be replayed forever
Phase 5. The headline is not the coverage work — it is what looking for coverage found. A CAPTURED SLACK REQUEST AUTHENTICATED INDEFINITELY `slack_signature_valid` verified the HMAC correctly, and nothing anywhere checked how old the timestamp was. The timestamp is an input to the basestring, so an old request's signature verifies exactly as well as a fresh one — meaning anyone holding a single captured signed request (a proxy log, a mirrored packet, a leaked webhook body) could replay it forever, and every replay would authenticate. Slack's documented 5-minute window is now enforced IN THE BROKER, not the caller: the broker does not trust its caller (§15), and a check the caller can forget to make is one that will eventually be forgotten. Symmetric, so a far-future timestamp cannot mint a request valid for as long as the attacker chooses. Seven unit tests over the pure function with the clock injected, and the HTTP-level test now asserts an hour-old but validly signed request is refused. Negative control: removing the window fails the stale and future cases specifically. The existing slack_inbound test used the literal timestamp "12345" — a 1970 date — which passed only because nothing checked freshness. That is the shape of the whole finding: the fixture could not have failed, so it never told us anything. COVERAGE, RE-EXAMINED The review ranked crates by raw test count. That metric was misleading and found the wrong crates: cm-safety's seven tests already cover the decide CAS, grant double-consume, expiry and the approved/rejected split, and the audit_log immutability trigger is tested over in cm-db. Reading the API surface against the tests found the real gaps — verify_slack_signature above, and `credits_for_tokens`, pure pricing arithmetic that every existing billing test went through the database to reach without ever checking directly. Now pinned: the round-up contract, the deliberate one-credit floor, and that an absurd token count cannot wrap into a negative charge (a refund granted by an overflow). Still genuinely thin: cm-brain, where 6 of 9 tests need live clawbrainhub.com. Stubbing it means reproducing an external registry protocol we have no spec for — its own piece of work, not a coverage chore. Recorded rather than faked. GATEWAY PREFLIGHT ZEROCLAW_GATEWAY_URL and ZEROCLAW_TOKEN have no defaults and are read at FIRST USE, so a deployment missing them boots clean, serves every page, and fails the first time someone presses run. Third sibling of runtime_preflight and validator_preflight, same stance: a report, not a gate. The message names the consequence — "container-tier missions cannot run" — rather than only the unset variable. One process note: `cargo test -p cm-secrets` passed while the LIBRARY build was broken, because `time` is a dev-dependency there and my reference to it only resolved under cfg(test). Switched to std. Checking `cargo build --workspace` as well as the test profile is the guard. Full workspace suite green: 106 binaries, zero build errors. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
769e002bb3 |
feat(skills): deliver on every tier, record what agents receive, let them self-author
Three phases of the approved plan, plus a correction to what the last one
claimed.
CORRECTION: skills reached ONE tier, not all of them
The previous commit said "skills can now reach a mission agent". That was
true only for the container/ZeroClaw tier — the fall-through that queues a
topology_runs row for topology_worker, which drives the executor that was
patched. compose_turn_prompt/pinned_skills_text had exactly one production
caller, and phase_runner's three other paths (composed microVM, solo
microVM, direct session) never called it. CAPABILITY-REVIEW.md said the
broad thing too; both are corrected.
Those three tiers share one task string and have no per-turn alias, so
their skills resolve per PHASE from the mission's crew and are appended
there. The container tier deliberately still injects per turn, with the
running node's own role — appending in both places would put every crew
member's skills in every turn twice.
The behavioural tests prove phase_skills_text and compose_turn_prompt work.
They cannot prove the three launch_* calls pass the composed string, and
that substitution is a one-word edit that would silently return all three
tiers to delivering nothing with every test still green. So there is also a
source-level assertion on the call sites, following the precedent in
mission_events::the_cap_is_enforced_in_one_statement. Its negative control
names the exact tier.
PROVENANCE: what an agent received, and what it said it did
Both were unanswerable. The prompt was never stored anywhere on any tier —
re-deriving it later re-runs the skill lookup against a catalogue that has
since changed, and once agents author their own skills it certainly will
have. The reasoning rows were durably write-only: pushed live once, then
never read from the database again by anything except the GC that deletes
them.
- prompt.composed records the exact bytes, on all four tiers
- the session tier writes its checkpoint record and a reasoning row,
instead of eprintln! and nothing — the same defect the solo microVM
path was fixed for, in the last tier that still had it
- narrative_for_mission reads both back
Found while doing it: the 400-event per-phase cap counted EVERY kind, so a
busy phase could push out its own phase.completed and its own provenance.
The cap now counts only the two unbounded kinds it was written for.
Negative control confirms the old behaviour dropped the prompt.
Retention is now a per-mission hold (0080) rather than a raised global —
with a test asserting unheld missions are still reaped, because an
exemption that applies to everything is not an exemption.
SELF-AUTHORING: agents apply their own skill drafts, no human click
By operator decision. level_up has generated complete drafts from a model
since it shipped; only a checkbox stood between propose and apply.
What replaces the gate is not another gate but four properties, each held
by a test:
- workspace-scoped, so a hand-authored skill can never be modified
- a draft cannot take a hand-authored skill's name. Ids are scoped and
bindings resolve by skill_id, so it could not overwrite or shadow one
anyway — but two procedures under one name means nobody reading a
transcript can tell which the agent followed, and that ambiguity is
fatal in a system where the skill is the standard being graded against
- every revision appends a skill_versions row, so it can be reverted and
a past run can be read against the text it was actually judged under
- approved_by = NULL. An agent's decision is never attributed to a person
who did not make it
Only skill_candidate applies autonomously. identity_refinement and
brain_consolidation still wait for a human: they change what an agent IS
rather than adding a procedure it can consult. State is announced at boot,
because a safety gate that changes silently is one nobody notices changed.
CLAWMATES_SKILL_SELF_AUTHORING=0 restores it.
Also: the test Postgres ran out of /dev/shm mid-suite (Docker's 64MB
default) and surfaced it during MIGRATIONS, which reads like a schema fault
and is not one. --shm-size=1g, and a pointer to the `clean` subcommand that
already existed for the 779 leaked test databases.
Full workspace suite green: 106 binaries, no failures.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
1f39f642a3 |
feat(podcast): render finished missions into episodes, and serve them as a feed
The renderer existed but nothing called it. This wires it to the missions and puts the result somewhere a phone can reach. **A sweep, not a phase step.** Rendering is not the agents' work and must not be able to fail a phase that succeeded; a transient API error simply retries next tick, and a mission already rendered is skipped because its episode row exists. `podcast_episodes` is that record — without it the sweep would re-render on every pass and re-bill for it, the same lesson `corpus_items` taught for papers. **It is racing a reaper.** script.md lives in the mission checkout, and `mission_runtime`'s sweeper deletes that tree 30 minutes after the mission reaches a terminal state. So the sweep runs every 2 minutes, leaving ~15 attempts inside the window. When it does lose — as it did for three missions that had completed hours before this shipped — it now SAYS so and records a marker rather than skipping in silence, which is how a feed ends up quietly missing a day. The feed filters those markers out: a zero-byte enclosure shows a broken episode in a podcast app, where showing nothing is honest. **Duration is read from the audio, not estimated from the script.** The feed advertises a length and that length should be the real one — and it is the check that catches a 6 MB file playing for six seconds. **The feed authenticates by query-string token**, because no podcast app can set headers. That is a real trade: the token lands in the app's database and any proxy log. It reuses `AuthService::authenticate`, so revoking the session revokes the feed with it rather than creating a second secret to forget to rotate. Titles are XML-escaped — one raw ampersand makes a client reject the WHOLE feed, not one episode. 363 tests pass. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
a02e0cba69 |
feat(missions): Continuous Research harvests at launch, and cards launch by clicking
The card shipped in
|
||
|
Omar SobhandClaude Opus 5
|
f87853ecf9 |
fix(missions): scheduled missions never fired — nothing read missions.schedule
The wizard has collected a cron since `0047_missions.sql` ("schedule JSONB
carries the trigger config (cron | one_shot | on_event)"), the frontend posts
`{kind:"cron", cron}`, and the API persists it faithfully. Nothing has ever read
it back: the only due-work enumerator in the codebase was `routines::claim_due`.
So every scheduled mission ever created sat in `draft` forever while the UI
reported it was on a schedule.
Proven before fixing, on the shipped build: a mission with `* * * * *` sat in
`draft` for 4m34s and started ZERO topology runs. After this change the same
mission launched on its next occurrence and recorded one `fired` row.
Two pieces were missing, and they are the two `routines` already had:
- `missions.next_run_at` — schedule STATE. `schedule` is user intent and stays
untouched; without somewhere to record which occurrence is owed there is
nothing to put a `<= now()` predicate on, which is why no enumerator could
be written against the JSONB alone.
- `mission_fires` — one row per (mission, occurrence). 0063_routine_fires.sql
called this exact case: "For a scheduled *mission* it costs a container, a
repo checkout, and real money — which is why this lands before mission
scheduling does."
`mission_schedule.rs` deliberately mirrors `cm-scheduler`'s shape rather than
inventing a second one: atomic `FOR UPDATE SKIP LOCKED` claim, reschedule
BEFORE dispatch so a failing launch cannot stall the clock, claim the slot
before launching so a crash mid-launch is retried rather than dropped, and a
fan-out cap. The cap is 5, not the scheduler's 25, because a mission firing is
a container and a checkout where a routine firing may be one turn.
The claim skips `status = 'running'`: a daily cron on a mission that takes
longer than a day must skip the occurrence, not stack a second crew on the same
workspace. Launch goes through `mission_orchestrator::on_launch` +
`missions::set_status`, the same path as the draft→running transition, so one
code path mints a crew. An unattended launch acts as the workspace owner
(`users::owner_of_workspace`) since missions carry no creator column; a
workspace without one settles the occurrence `failed` with the reason rather
than dropping it silently.
Backfill blast radius was MEASURED, not assumed: prod has zero missions with a
cron, this workstation had exactly one — the control created to prove the bug.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
a494634f81 |
feat(agents): classify agents by lifecycle and reap the finished and orphaned
A mission mints a crew, and the only thing that reaped one was DELETING the mission. A mission that merely completed left its agents in the roster forever, and a crew whose reap was skipped or failed left agents bound to nothing — indistinguishable in the UI from the operator's own staff. Four states, from one query: owned no agent_template_link row → hand-created. NEVER reaped. active on a running/draft mission → working right now. Kept. completed every mission terminal → reaped after a 24h grace. orphaned minted, bound to nothing → reaped. The discriminator is `agent_template_link`, which mission_orchestrator writes per minted claw. This matters more than it looks: verified on live data, a hand-created agent and an orphaned crew member both have ZERO team links and are structurally identical by binding alone. Judging orphanhood by "no team" would delete the user's workforce. Provenance is the only honest signal. The grace window exists because the results view, the World's 24h replay and "who did this work?" all read the crew AFTER the run ends; reaping on the terminal transition deletes the answer exactly when the question gets asked. A completed crew with no usable timestamp is KEPT — a missing date must never read as "old enough to delete". Also fixes the delete summary, which reported how many claws were FOUND rather than purged: "reaped 4 claw(s)" was printed by a delete that purged none, which is precisely the log you would read while wondering why the agents are still there. It now reports purged / kept / FAILED, and failed > 0 is the orphan case. Verified against live data — all four states observed, including the two that look alike. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
1f6108f769 |
feat(gc): reclaim the mission tree on the gateway
`cleanup_sweeper` prunes ROWS. Deleting a row has never deleted a directory, and `teardown_container` only runs while a mission still exists to tear down — so a mission removed by any path that skipped teardown left its tree behind permanently, on the smallest disk in the fleet (150 GB, shared with postgres and every checkout). 106 mission directories are sitting there now. Filesystem-first, deliberately: the DB is the PREDICATE, never the enumerator. Enumerating from the database is exactly how these became invisible — a directory whose row is gone is the one a row-driven sweep cannot see. Three reapers, one deletion path. Orphan mission dirs (no row, past a 2h grace), scratch trees (_bench/_gate/_verify/_merge past 6h — all four have leaked before), and _outputs past 90d, whose artifact rows are marked only AFTER the files are gone, because the other order claims artifacts are reaped while they are still on disk. The single removal path escalates: the server is uid 65532 and cannot delete what the per-mission daemon leaves as root, so PermissionDenied falls back to `root_copy::purge` and shouts if the tree survives even that. A GC that cannot collect is the thing being fixed, so failures are counted and reported, never swallowed. Guards worth naming: `_cargo` is a SHARED cache every mission writes to and lives under the same root, so an underscore-prefixed sibling treated as an orphan mission would delete it out from under running work and look like a slow cargo build. Only a well-formed mission id is ever a candidate — a directory whose name is not an id can have no row by construction, so without that gate every unrecognised directory looks orphaned. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
c3ad5672fc |
feat(llm): six-link fallback chain, and a preflight that proves it
opus -> sonnet -> haiku -> kimi -> glm -> local. The order is capability first, then independence: three Anthropic tiers on one account (a throttle usually hits a tier, so stepping down often clears it), then two separately funded accounts (now an outage, not just a throttle, is survivable), then our own GPU (nothing left to be down). Every id was probed on this deployment and answered 200. The preflight is the more important half. Configured is not working, and this chain has a specific way of lying: `resolve_provider` falls back to the DEFAULT provider when it does not recognise a provider name, so a typo in `kimi:` does not error — it quietly runs on Anthropic, and a chain that reads as three accounts is really one. A reachability-only probe calls that link green. So `preflight` checks resolution and reachability separately, eight tokens per link through the REAL call path, and reports four states. `Throttled` is deliberately not a failure: a 429 means the spec resolved, the credential authenticated, and there was no capacity this second — the exact condition the chain exists to route around, and painting it red would train an operator to ignore red. `Unregistered` and `Broken` are failures, and they get different words because they need different fixes. It runs at boot alongside validator_preflight and runtime_preflight, spawned so it cannot delay startup. A chain is the one piece of infrastructure nobody looks at until the day it has to work, so it is now checked on the days it does not. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
b18e62041b |
feat(llm): the fallback chain's last link runs on our own hardware
`local:ornith-fleet:9b` joins opus -> haiku -> glm as the final link. Every entry above it depends on somebody else's account staying funded and unthrottled; this one depends on a GPU in the next room. It is last because it is the weakest model, and present because a chain whose every link is external is not a fallback chain, it is one outage in a trench coat. Three small changes make it work: - `build_provider_registry` accepts a provider with an empty `api_key_env`. A model on our own hardware has nothing to authenticate to, and the old behaviour SKIPPED a keyless provider — leaving the chain quietly one link shorter than it reads, which is the failure mode this whole area keeps producing. - `provider_family` learns `ornith`/`ollama` for BARE names. A qualified `local:` spec was already answered by the split, but a bare one fell through to "unknown", and `cross_provider_judge` would then refuse a judge that is genuinely a different family from the Anthropic implementer. - A test pins that the last link survives `resolve_provider`'s split-on-FIRST- colon: `local:ornith-fleet:9b` is provider `local`, model `ornith-fleet:9b`. Splitting on the last colon would ask for a provider named `local:ornith-fleet`, and the symptom would be a silent fall back to the default provider. Infra: Ollama on tank and architect now binds 0.0.0.0 so the gateway (which has no GPU) can reach it. `tailscale serve` cannot — Ollama rejects a non-local Host header as a DNS-rebinding guard and OLLAMA_ORIGINS is CORS-only, so it 403s. 0.0.0.0 still includes loopback, so the microVM vsock pipe is unaffected; verified on both nodes. This is an explicit trade: Ollama has no auth and its API can pull and delete models, so it is now reachable from the LAN as well as the tailnet. The drop-in carries the ufw one-liner to close the LAN side. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
f56d41f5b7 |
feat(backend): local-ornith — a mission backend served by the node's own GPU
Claude Code pointed at the Ollama already installed on every GPU node. Ollama has served a native Anthropic-compatible /v1/messages since v0.14, so this is an env contract rather than a translation layer — the fourth variation on the same idea as agent-glm and agent-kimi. The route is NOT the egress proxy, and that is the design. `egress` speaks CONNECT, takes a destination from the guest, resolves it and decides; every one of those powers is a liability, which is why it refuses non-443 ports and IP literals after a unit test caught them being bypassed. Routing a local model through it would have meant relaxing both. `local_model` is the opposite shape: there is no destination in the protocol. fcagent listens on guest 127.0.0.1:11434 and pumps to vsock 9003; the node splices that onto its own 127.0.0.1:11434 and copies bytes. A compromised guest cannot redirect it because there is nothing to redirect — it is a pipe, not a proxy, and strictly narrower than anything an allow-list could express. The bytes never touch a network, so there is no wire for TLS to protect, and Ollama stays bound to loopback rather than being exposed on the tailnet. The socket is bound only for a backend declared to use a local model, so a `local-ornith` VM reaches the forge through egress and nothing else, while every other backend's guest port simply refuses. Both halves have negative controls. `scripts/fleet-model-setup.sh` exists because of one measurement: stock ornith:9b reported input_tokens=2050 for a 48000-word prompt and answered as though nothing had been dropped. Ollama's default window is ~2K whatever the model card says, and it truncates silently — the exact failure an agent turn would hit and never report. The script pins num_ctx=131072 into a derived tag and then PROVES both the window and tool calling before declaring success. Verified on architect: ~65536 words -> 65604 input tokens, stop_reason=tool_use. Placement needs no new capability key: building the rootfs only on GPU nodes means `nodes::online_for_backend`'s existing `rootfs @> ["local-ornith"]` predicate does the affinity, so morpheus never offers the backend. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
d48bdbc9a7 |
fix(llm): two modules were posting to Anthropic behind the providers' back
The research scenario passed 4/4 and the log underneath it said: phase_summarizer: ... failed: anthropic 400 Bad Request: "Your credit balance is too low to access the Anthropic API" `phase_summarizer` and `mission_refiner` each built their own reqwest POST to the Messages API with `x-api-key: $ANTHROPIC_API_KEY`. No audit of `.complete(` call sites could have found them — they never touched a provider — so every phase summary and every mission-brief refinement on this deployment had been failing against an empty account while the phases themselves ran fine. The summarizer even persisted an error row per phase, which is why nothing ever retried loudly enough to notice. Both now go through `subscription::complete_with_fallback`, so they inherit the subscription-first credential choice, the 429 backoff, and the opus -> haiku -> glm chain. The summarizer records the model that ANSWERED in mission_phase_summaries.model rather than the one it asked for. The guard is a source WALK, not a file list: any .rs under cm-api/src that mentions the Messages API host or `x-api-key` fails the test. A hand-listed set of files is exactly what let these two hide. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
9c9439a271 |
feat(llm): the subscription is the default provider, with a recorded fallback chain
Two changes so an empty metered account stops being a platform outage. 1. `build_provider` prefers the subscription token over ANTHROPIC_API_KEY. A bare model name resolves to whatever this returns, so making it the subscription means no server-side call can reach the metered key by construction — rather than by a source-grep test that already missed four call sites once. The metered key remains a fallback and now warns loudly when it is the one in use; boot no longer requires it at all. 2. `complete_with_fallback` walks a declared chain when a model has no capacity: opus -> haiku -> glm:glm-4.7 by default, overridable via CLAWMATES_MODEL_FALLBACK, empty to disable. Measured on gw-04 today: opus and sonnet return 429 on the subscription while haiku, GLM and Kimi all return 200, so a capped window no longer means "the planner is gone". The chain returns the model that ANSWERED, and every caller persists it — mission_plan_proposals.author_model, mission_team_proposals.author_model, and the swarm's step role. A plan drafted by the third link and filed as an opus plan is a silent quality change, which is the failure shape this project keeps paying for. Two negative controls hold the design: the chain never retries the model that just failed as its own fallback, and it steps down ONLY for a capacity failure — walking it on a malformed prompt would ask three models the same bad question and report the third one's confusion. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar Sobh
|
099a716bfd |
fix(egress): a backend is defined in two maps, and the canary only had one
First canary run failed: phase failed, nothing delivered, and the streamed log said exactly why — "Failed to authenticate. API Error: 403 api.anthropic.com is not on the egress allow-list". Not a 2.1.226 regression. `canary-claude` was added to the server's credential map and not to the node's `provider_hosts`, so the VM booted with a valid subscription token and a door that only opened onto the forge. The fail-closed branch was working correctly: a backend nobody taught that function about reaches no model API, deliberately, so it cannot silently borrow another provider's door. Both maps now name it, each pointing at the other, with a test asserting the canary reaches the same provider as `claude` AND that unknown backends still resolve to nothing. Worth noting what made this a five-second diagnosis instead of an afternoon: the live log streaming built earlier today. The failure was a 403 inside a microVM that no longer exists, and its reason was sitting in the run's checkpoint. |
||
|
Omar Sobh
|
09afa7e7ff |
fix(node): aborting the tail at turn end raced the flush that matters most
Composed runs streamed NOTHING while solo runs streamed fine — same code path, `HubVms::run` is a straight passthrough, and the node logged a tail starting for all five graph nodes with the correct outer run id. The difference was timing. `claude -p ... | tee` makes stdout a PIPE, so the CLI block-buffers and flushes at EXIT. The most valuable output — the agent's summary of what it did — arrives in the instant the turn ends. The node aborted the tail the moment `handle_op` returned, so that flush was a race: a solo turn (minutes long, output already flushed by size) won it and streamed 337 bytes; each node of a composed run (~20s) lost it and streamed zero. The tail now DRAINS. A flag is set when the turn returns, and the loop exits only after a pass that read nothing new — checked AFTER a read, never before one, because exiting on the flag alone would drop exactly the bytes this exists to capture. Bounded by a 20s timeout with the abort kept as a backstop rather than the mechanism, so a VM that stopped answering cannot hold the task open. Worth naming: 5 tails started, 5 logged cleanly, 0 bytes arrived. Every individual step reported success and the feature did nothing — the same shape as the empty Live tab this whole thread began with, one layer down. |
||
|
Omar Sobh
|
28090d1de0 |
fix(node): the log tail gave up before the turn wrote its first byte
First live test of the streaming path: mission passed 6/6, `checkpoint.log` was 0 bytes, and the node logged nothing at all. `stream_vm_log` treated "no progress" as "the turn finished writing". But the guest's `tail` reports EOF after every idle window, and the FIRST idle window is always the one before any output exists — the VM is still booting and the CLI still starting. So the tail returned `at == 0`, the node concluded the turn was done, and it stopped seconds into a run that then went on for minutes. The abort is the terminator, not idleness: the caller already aborts this task when the exec returns, so waiting cannot outlive the turn. No-progress now sleeps and retries instead of returning. Also logs when a tail STARTS. The bug was invisible in exactly the way this session keeps finding: silence on the success path, silence on the give-up path, and an empty Live tab that looked identical to a feature nobody had wired. Method note, since it cost time: I tried to confirm the deployed binary by grepping it for `vm_out` and found zero — then found zero for `pty_out` and `vm_exec` too, in a binary whose PTY streaming demonstrably works. Binary-grep is not a reliable presence test for these literals; `stream_vm_log` and `tail of` being present is what actually showed the code had shipped. |
||
|
Omar Sobh
|
0b89b8316c |
feat(observability): stream a microVM turn's stdout/stderr to the platform live
The Live tab showed nothing while a turn ran, and the agent's own account of it
went to stderr on the node and nowhere a user could reach. This is the path that
carries it.
The blocker was the guest agent. `fcagent` handled one connection at a time,
inline, so during an hour-long turn the VM accepted nothing — which is why every
existing probe (subagents, stop-gate blocks, cap) runs AFTER the turn rather than
during it. It now spawns a thread per connection, wrapped in `catch_unwind`
because this process is pid 1: a panic used to take the accept loop with it, and
an unbootable VM is a far worse outcome than a missing log. A failed spawn logs
and keeps accepting rather than dropping the listener.
PROVED against a live VM before building on it, since "sound reasoning about this
system" and "measurement" have diverged repeatedly today. Patched rootfs, booted
under Firecracker, ran an 8s exec and a concurrent tail:
exec took 8.0s ok=True
+0.0s 'line1\nline2\n' +1.2s 'line4\n' +3.2s 'line6\n' +6.0s 'DONE\n'
VERDICT: CONCURRENT — tail returned data before exec finished
The rest is the pattern the terminal already uses. New `tail` op streams a file
by OFFSET (so a dropped link resumes instead of replaying, and the tail always
terminates — one that never returns pins a thread for the life of the VM). The
node follows the log alongside the turn and pushes `Uplink::VmOut { run_id, at,
data }` over the WebSocket it already holds, mirroring `PtyOut`. The server does
what `PtyOut` deliberately does not: it APPENDS to the run's checkpoint as well
as fanning out, because a terminal has no history worth keeping and a mission log
is the record of what the agent did. `run_events_sse` emits the new bytes as
`step` events, which the live pane already renders — no frontend change.
The turn is `tee`d, not redirected: the file feeds the live stream and stdout
still becomes `VmOutcome::summary`. A redirect would have produced a live view
and an empty summary, which is the same green-and-empty shape as the bug this
fixes. Tested, along with the log living outside the collected tree so it never
lands in a user's delivered diff.
246 lib tests, 20 binaries; node and fcagent build clean.
|
||
|
Omar Sobh
|
f6c3ddbf81 |
refactor: no feature depends on Gemini any more
Depleted Gemini prepayment credits took out PDF rendering. The same key was the
only thing standing between level-up proposals and the same fate, so both are
off it.
- `pdf_renderer` is DELETED, not disabled. Nothing sets `render_pdf: true` since
markdown became the deliverable (
|
||
|
Omar SobhandClaude Opus 5
|
3300c9d149 |
feat(missions): say at boot whether the independent judge can be reached
The z.ai credential expired mid-session and the first symptom was a two-phase mission failing after BOTH its VMs had run — the phase completed, delivered, pushed, and then one evaluation row said "the independent validator could not be reached this pass". `cross_provider_judge` refusing to fall back to the agent's own provider is correct: a verdict from the same family is not an independent check, and producing one quietly would claim a property the verdict does not have. The cost of that refusal is that a dead validator makes EVERY `done_when` phase unmeetable — and the information needed to know that existed from the moment the server booted. Nobody was told until it was expensive. The sibling of `runtime_preflight`, and the same stance: a report, not a gate. The server must still boot with a broken validator — refusing to start turns a degraded deployment into a dead one, and a mission that opts out (`validator_model = ''`) is unaffected. Two faults, kept distinguishable because they send an operator to different places: `Unregistered` (no provider by that name — the evaluator will refuse it rather than judge with the default, so register one) versus `Unreachable` (it resolved and the call failed — fix the credential). Collapsing them into "the validator is broken" is the kind of merge that costs an hour. The probe is a real completion through `Runtime::complete` — the same resolve-then-stream path the judge itself takes. A models-list or a HEAD would pass for an expired key, a revoked key, and a key with no quota, which are exactly the cases worth catching; and a probe that dialled the provider its own way could pass while the real call fails. `NotConfigured` is reported too, and not as an error: a deployment may choose the house model. It is still worth saying out loud that the check running is not an independent one. 551 tests pass, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
742724e53c |
feat(fleet): Kimi as a microVM backend — the URL settled by measurement
The base URL took three measurements to find, and the first two were wrong in instructive ways. `api.moonshot.ai/anthropic/v1/messages` EXISTS and speaks the protocol — it answers with Moonshot's own structured error rather than a 404. It also rejects an `sk-kimi-` key, because it belongs to the platform.moonshot.ai account namespace. Two endpoints that both "work" for different accounts is precisely the shape that makes a guessed URL look like a broken key, and it is why this was refused rather than guessed for as long as it was. The Kimi CODE service is the one an `sk-kimi-` key belongs to: `POST https://api.kimi.com/coding/v1/messages` returns a real Anthropic Messages body — `msg_` id, `content` blocks, a `thinking` block with a signature. So `ANTHROPIC_BASE_URL=https://api.kimi.com/coding`, WITHOUT the `/v1`: Claude Code appends `/v1/messages` itself, and `/v1/v1/messages` would 404 in a way that reads as a broken image rather than a bad URL. Two more measured, each otherwise a silent failure at the first turn: `Authorization: Bearer` is accepted (so ANTHROPIC_AUTH_TOKEN is the right injection channel), and a `claude-*` model id is ACCEPTED AND ANSWERED — Kimi maps it onto `kimi-for-coding` exactly as z.ai does, so no ANTHROPIC_MODEL override is needed. Claude Code rather than Moonshot's own `kimi` CLI, deliberately. The mission harness is Claude-Code-shaped throughout: `--agents` JSON roles, the verifier's tool allowlist, the `Stop` hook behind the completion gate, the per-subagent transcripts counted as delegation evidence. `kimi` has none of those flags — its equivalents are TOML files and markdown agent dirs — so using it would mean a second executor with its own untested failure modes. TWO STALE MAPS, caught by the rootfs harness refusing to bless the image: both `fc-build-rootfs.sh` and the node's `required_cli` expected backend `kimi` to contain Moonshot's `kimi` binary. That assumption predates the measurement, and it failed a rootfs that was correct. Both now say `claude` for glm and kimi alike — the binary is the same in all three images; only the endpoint differs. Egress for `kimi` is `api.kimi.com` alone: not moonshot.ai (wrong namespace), not z.ai, not Anthropic. Asserted both ways, like the other two. The image and rootfs are built on tank and the rootfs passes all four checks (boots, git, writable /mission, `claude --version`). 534 tests, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
d3a53e7bf1 |
fix(fleet): a VM reaches its OWN provider and no other, measured not assumed
The GLM backend works — and proving it produced a better boundary than the one I shipped an hour ago. WHAT THE FIRST GLM MISSION SHOWED. It completed, and the delivered file said the model was "claude-opus-5". The node's egress log said the VM had dialled `api.anthropic.com` five times before `api.z.ai`. Either reading alone is consistent with a "GLM backend" that silently runs Anthropic — the exact silent-success shape this project keeps closing — so I did not accept either. THE ABLATION, run on tank rather than reasoned about: deny `anthropic.com` at the proxy and run the same mission again. It **completed**, dialling only `api.z.ai`. So the completions genuinely come from z.ai; Claude Code's calls to anthropic.com are its own telemetry, not its model traffic. And that same agent — served exclusively by z.ai, with Anthropic unreachable — still described itself as "Claude Opus 5 (1M context)". **A model's account of which model it is has no evidential value.** The proxy's log of which host it dialled does. This is the `uname -r` lesson again in a new place: ask the infrastructure, not the agent. So the allow-list is now PER BACKEND rather than a union: a `claude` VM reaches Anthropic and the forge, a `glm` VM reaches z.ai and the forge, and neither can reach the other's endpoint. A union was defensible when it was one host; once the measurement showed a GLM VM never needs Anthropic, keeping it would mean a credential mix-up upstream could still put one provider's secret on another provider's wire. Now it fails at a closed door instead. An unknown backend gets the forge and NO model API — it cannot run anyway, and borrowing somebody else's door is the failure this split prevents. An explicit `CLAWMATES_FC_EGRESS_ALLOW` still wins outright: an operator who set it drew a boundary on purpose. `DEFAULT_ALLOW` is deleted rather than left beside the new function, so there is one answer to "what may a mission reach" and not two. 534 tests pass, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
f7f3dfe495 |
feat(fleet): GLM as a real microVM backend, and per-role models for claws
Three threads, all of which end at the same place: a mission whose verifier does
not share a model with the coder it reviews.
**GLM has a credential contract now.** `microvm_credential_for` returned one env
var name, which quietly assumed every provider reads its secret from the same
place Anthropic does. It returns a `Credential { source, target }` instead —
z.ai's key lives in the server's `ZAI_API_KEY` and Claude Code reads it as
`ANTHROPIC_AUTH_TOKEN`, and collapsing those two names is what forces a guess at
the other end. A wrong guess here sends one provider's credential to another
provider's endpoint.
`images/agent-glm` is the same CLI at the same pinned version as `agent-claude`
with `ANTHROPIC_BASE_URL` baked in. The split is deliberate: the ENDPOINT is a
property of the image, the CREDENTIAL is a property of the turn. That makes the
dangerous mix-up unrepresentable — a GLM VM cannot be handed an Anthropic
subscription token, and a claude VM cannot be pointed at z.ai. Asserted both
ways, because "the GLM VM must not carry CLAUDE_CODE_OAUTH_TOKEN" is the
property that costs a credential if it ever stops holding.
Kimi stays refused. `KIMI_API_KEY` is set and Moonshot serves an
Anthropic-compatible API, but I have not verified its base URL against the
running service, and this function is precisely where guessing a URL is
expensive. It becomes an arm the day someone measures it.
`api.z.ai` joins the node's default egress allow-list. A default that cannot
run the images we ship is a trap rather than a policy — the alternative is an
operator discovering it as a hung agent with no model access.
**Per-role models for claws** (migration 0071). `template_roles` had no model
column, so `mint_team_from_template` bound every role of every mission team to
one literal — a template whose whole point is an independent reviewer minted a
reviewer sharing a model with the coder. A role may now name its own; roles that
say nothing still take the mint's default, so every template written before this
behaves exactly as it did. The literal is now that default rather than a
hardcode.
**A harness scenario for the roster flow.** `verify-mission-delivery.sh roster`
runs the whole Slice 5 loop — planner proposes, human approves, mission runs —
and asserts the roster LANDED on the mission row rather than trusting the API's
answer. That distinction is not theoretical: the first live approval returned an
error while leaving the proposal marked approved.
Built and proven on tank ahead of the deploy: `clawmates/agent-glm:dev` reports
`2.1.223` and `BASE=https://api.z.ai/api/anthropic`, and
`fc-build-rootfs.sh … glm 8G` boots a VM from it that has git, can write
/mission, and answers `claude --version`.
533 tests pass, clippy clean. Migration 0071.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
12147a1e01 |
feat(missions): Slice 4 — the two engines composed, with the file handoff proven
`team_engine='composed'` (the third name migration 0069 anticipated) runs a
mission as a durable ZeroClaw graph whose every node is a whole
Claude-Code-in-a-microVM session. Engine Z owns checkpoint/resume, cancellation
and per-node heterogeneity; Engine C owns shared context and cheap fan-out;
neither has the other's asset, which is why this is a composition and not a
compromise.
`MicroVmTurnExecutor` implements the existing `TurnExecutor`, so it inherits the
planners, the checkpoint, the stale-run recovery, `close_finished_phases`, the
evaluator, capture and delivery unchanged — the same trick `SubTopologyExecutor`
already plays with a heavy `run_turn`. Producer side emits ONE `queued` row
carrying the real graph and lets the worker claim it: the durability IS being
worker-driven, and the solo path's `tokio::spawn` has none of it. Still exactly
one `topology_runs` row per unit of work and one completion path — `finish()` is
now that one place, shared by every tier.
THE TRAP, solved and proven. A VM is inject → run → collect → destroy, so a
per-node VM with text-only handoff silently loses every file an earlier node
wrote: node 2 boots from the original checkout, sees nothing, and still reports
success. The mission's host checkout is the medium — every node injects from it
and collects back over it — and two properties make that safe rather than lucky:
`execute_resumable` is strictly sequential, so two VMs never write one directory;
and the vm id is deterministic per (phase, iteration, step), so a duplicate is
refused by the node ("vm already exists") instead of becoming a second writer.
NEGATIVE CONTROL, run rather than assumed: with `repo` swapped for a private
per-node workspace, `a_later_node_sees_an_earlier_nodes_files` FAILS with
`saw:[]`; restored, it passes. The `PhaseVm` seam exists for exactly this — it
models inject/collect through the real `mission_fs` tar path in milliseconds.
Two durability traps this tier walks into, both closed:
- `requeue_stale` fires at 180s on `updated_at`, and one node here can run for
an hour. `SubTopologyExecutor` keeps its parent alive from each leaf step;
there is nothing between the start and end of a VM turn, so the turn holds a
ticker that touches `updated_at` every 30s and aborts on drop. Without it a
healthy composed run is requeued mid-node and boots a second VM.
- the 15-minute stuck-run reaper asks "any step records since it was CREATED?",
which describes a healthy composed run as readily as a wedged one. Hence
`REAPABLE_TIERS` — worker-driven minus this tier. Reaping it would be #54 in
a different costume.
`on_launch` mints no team for a microVM mission, deliberately: claws in
containers are what a VM mission does not use. So `mission_orchestrator::
composed_graph` builds the shape from the team template directly — nodes, roles
and pattern, zero claws provisioned. Per-node `attrs["backend"]` and
`attrs["node_id"]` override the mission's, which is what makes a validator node
on another provider's image a first-class graph node; a malformed `node_id`
fails the node rather than quietly running it where the graph did not ask.
Refusals are recorded as a failed run, not returned as an error: `launch_phase`
is swept every ten seconds, so a returned error is a phase that retries forever
while the log repeats itself.
501 tests pass, clippy clean. NOT yet proven end to end: no composed mission has
run on the fleet, so the resume-after-a-killed-worker leg is argued from the DB
test and the step-numbering test, not from a real two-node run.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar Sobh
|
d49acaed5e |
fix(missions): exclude build output from COLLECT too, not just inject
The other half of the same bug. The previous commit filtered `mission_fs::pack_dir` (the inject side) and left the guest's `op_get` tarring everything, so the re-run that proved the #54 fix — it survived 480s where it used to die at 210 — still lost its work to `vm_collect ... node timed out`. Two modules written, four subagents used, nothing delivered. `op_get` now takes an `exclude` list, sent by the host from `mission_fs::transport_excludes()` — the same list `mission_delivery` uses for the diff. Policy in one place, applied at both ends of the wire. Matched on directory NAME at any depth, so a workspace's per-crate `target/` dirs are all covered, with a test that plants a nested one and asserts it does not come along. Also proven by that run: the worker no longer kills a live microVM run. It ran 480 seconds straight through the 180s requeue window and the 210s mark where mission 019fd43e died, untouched. And `subagents: 4` — the team addendum did drive real fan-out this time, which is the first evidence the Slice 3 switch does anything. 483 tests pass, clippy clean. Still to prove: a >3-minute mission that actually DELIVERS. The collect fix is tested in isolation but has not yet carried a real mission's work back, and the guest agent needs rebuilding into the rootfs before it can. |
||
|
Omar Sobh
|
4efcde9d4f |
fix(missions): #54 — the worker was killing live microVM runs at 180 seconds
My hypothesis in #54 was WRONG, and it was wrong because I built it on a bad measurement: `grep -c 'microvm phase'` returned 0, so I concluded the completion log never printed and blamed the 15-minute reaper. The line was there all along, at 14:17:45. The real cause is worse. `requeue_stale` has NO TIER FILTER. A microvm run's `updated_at` is written once at insert and never again — it is driven by a `tokio::spawn` that owns it start to finish, and nothing in `microvm_executor` writes `topology_runs`. So at 180s the sweeper declared a perfectly healthy run stale and flipped it to `queued`; `claim_next_queued` (no tier filter either) handed it to the worker; `run_job` tried to parse the microvm graph placeholder, which `TopologyGraph` cannot deserialize; and it failed the run with "missing or invalid graph". Mission 019fd43e: run created 14:11:16, mission failed ~14:14:46. 210 seconds — the 180s window plus a tick. The agent went on working and finished at 14:17:45 with three modules written, by which time the phase was already dead and the VM was orphaned. A firecracker process was still alive 1h37m later. THE UNCOMFORTABLE PART: every microVM mission that appeared to work this session did so only by finishing inside three minutes. The 90-second ones dodged this. The harness scenario dodges it. Nothing about that was visible. `WORKER_DRIVEN_TIERS` (team, company, org, swarm, compare) is now the allowlist for all three sweep paths — claim, requeue, reap. An allowlist rather than a denylist so the next self-driven tier is safe by default instead of exposed until someone remembers the file. `tier='session'` had exactly the same exposure and is covered too. A unit test asserts microvm and session are NOT in it, next to the code that inserts them. Two more fixes from the same wreckage: - `destroy` reported `killed: pgid.is_some()` — true whenever there was a pgid to signal, whether or not anything died. It now sends the signal, polls /proc for the group leader, retries, and reports what it OBSERVED; `signalled` keeps the old meaning so "nothing to kill" is distinguishable from "it would not die". - the run-status update is now guarded with `AND status <> 'cancelled'`. An operator cancelling is a decision; this task reporting an outcome minutes later is an observation, and it must not overwrite one with the other. And the root cause of the collect timeout itself: `mission_fs::pack_dir` shipped `target/` in both directions. `mission_delivery` has excluded build output from the DIFF since day one; the TRANSPORT never knew. The host checkout was 9.4 MB of which 8.9 MB was `target/`, tarred and base64'd over vsock each way. `EXCLUDED_PATHS` is now one list shared by both layers, matched on directory name at any depth so a workspace's per-crate `target/` dirs are all covered. 483 tests pass, clippy clean. |
||
|
Omar Sobh
|
d9f53a3f96 |
fix(fleet): placement requires the backend's rootfs image, not just KVM
The first real microVM mission was placed on morpheus because it reports
{"microvm": true}, while only tank had rootfs-claude.ext4. It failed by name
rather than booting the wrong image — but whether a mission ran came down to
which capable node was listed first, which is a coin flip dressed as scheduling.
`missions.backend` was invisible to the scheduler.
The node now enumerates the images on its disk and reports them as a `rootfs`
ARRAY. `microvm::available_backends` lives beside `rootfs_for`, its inverse,
because the two must agree on what a backend name means; split apart, one drifts
and the scheduler starts promising images the booter cannot find. It only
advertises names `rootfs_for` would accept, and reports an empty array rather than
omitting the key — set_capabilities REPLACES, so a deleted image stops being
advertised instead of leaving a stale claim.
`nodes::online_for_backend` requires microvm AND that the node's list contains the
mission's backend. A node on an older daemon has no `rootfs` key and matches
nothing: unknown is not permission, the same treatment every other capability
gets. `backend_key` maps the three spellings of "the default image" to the one
name the node advertises, and is tested — a mismatch there would reject every node
for an ordinary mission with no backend set.
The launch error now names both halves of the fix, since "no capable node" was
true but unhelpful when the node was capable and merely lacked the image.
Mission gains `backend` on the domain struct; it was a column the executor read
from the phase query while the struct that placement uses could not see it.
464 tests pass, clippy clean.
|
||
|
Omar SobhandClaude Opus 5
|
4f07430e92 |
feat(missions): B4.5 — phase_runner runs a microvm mission in a VM
`runtime_kind='microvm'` placed a mission on a KVM-capable node and then nothing
executed it: config accepted without a reader, one of the four seams this project
keeps closing. This is the reader.
`microvm_executor` — inject → run → collect → destroy, the shape copy mode
already proved for containers with a VM boundary instead of a namespace one. The
checkout goes in as a tar, the work comes back as a tar over the SAME host path,
so `mission_delivery::capture_phase_diff_at` needs no change at all.
The agent is told NOT to push, unlike the container path's session prompt. Two
reasons: delivery is already host-side and diffs the collected tree against the
recorded clone point (covering committed, staged and unstaged work in one pass),
so pushing would add a second untested way for work to arrive; and pushing would
mean forge credentials inside the VM, when the point of collecting is that the
guest never holds them.
Exactly ONE topology_runs row (tier='microvm'), mirroring launch_direct_session:
close_finished_phases, evaluation, capture and delivery all key off those rows,
and a second completion path would be a second way for a phase to finish with one
of them untested. The row and the phase flip happen BEFORE any fallible VM work,
so a missing token or a node that lost its capability shows up as a failed run an
operator can see — not a phase that stays pending and retries every ten seconds.
Fail-closed points, each the reader for a guarantee built earlier:
- credentials resolve BEFORE the VM boots, so a missing subscription token
fails the phase instead of booting a VM whose agent sits unauthenticated
- a VM reporting egress:false is REFUSED, which is what makes create's
egress/egress_host/egress_guest fields more than decoration — a turn without
egress does not fail, it hangs
- the injected checkout is PROVEN present in the guest before an agent turn is
spent on it; an inject that reports success while landing nothing would
otherwise become an agent reporting an empty repository
- work is collected even when the agent exits non-zero — a turn that failed
partway still wrote files, and a retry needs to see them
- a turn that ran but could not be collected is a FAILED phase, not a happy one
- destroy runs on every exit path, or an 8 GB sparse rootfs leaks
Two integration gaps found while wiring, both of which would have produced a
mission that completed having delivered nothing:
- `capture_finished_coding_phases` pulls work out of a CONTAINER before
capturing. A microvm mission has none, so the docker connect would fail, the
loop would `continue`, and capture would be skipped forever while the phase
sat marked completed. Its work is already collected by the executor.
- `launch_phase` provisioned a runtime container, copied the checkout into it
and wrote a runtime binding + pairing code describing a runtime nothing uses;
and the orchestrator's workspace pin — deliberately FATAL — would have failed
a microVM launch on a container it was never going to use.
461 tests pass, clippy clean.
NOT YET PROVEN END TO END: no mission has run through this path. The pieces under
it are each verified on tank (image, credentials, egress, a real agent turn), but
this executor has only been compiled and unit-tested. Deploy + one real microvm
mission is the remaining step.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
ebdba34da6 |
feat(fleet): B4.6 — a microVM reaches the API through a vsock CONNECT proxy, with an allow-list
The guest still has no network interface, and now that is the design rather than
a gap. Its only route out is an HTTP CONNECT proxy: agent CLI -> 127.0.0.1:3128
in the guest -> vsock 9002 -> a per-VM Unix socket on the host -> TLS to an
allow-listed host.
Why not TAP + iptables, which is what the Firecracker write-ups do — measured,
not argued:
- `ip tuntap add` is DENIED to the daemon user (needs CAP_NET_ADMIN), so TAP
would need root to pre-provision devices, the same privilege detour the
loop-mounted rootfs already forced.
- tank's FORWARD policy is DROP with Docker and Tailscale chains, so rules
would have to be inserted at position 1; appended ones die silently.
- a leaked TAP is a new class of host litter to reap.
CONNECT needs no privilege at all and is better on the merits: the client hands
us the HOSTNAME, so resolution happens host-side and the guest needs no DNS or
resolv.conf; the allow-list is by name, not address; and nothing in the guest can
reach the network except through one function. The guest end parses nothing and
enforces nothing, so a compromised agent cannot argue with the policy.
Rests on one measured fact: `claude` honours HTTPS_PROXY. With the proxy at a
closed port, `claude -p` fails ConnectionRefused instead of answering.
THE RESULT: a real agent turn now completes inside a VM with no network card, on
subscription auth — `claude -p` replies VM-OK. The selftest asks for it whenever
CLAUDE_CODE_OAUTH_TOKEN is present and SKIPS loudly when it is not, since it
spends a little of the plan.
The audit log earns its keep immediately: during that turn the proxy logged
`egress DENIED http-intake.logs.us5.datadoghq.com` — the CLI's telemetry, which
the mission container permits today without anyone deciding to.
Three bugs found by the checks rather than by review:
- `env_pairs` returned early when a caller sent no env, so the proxy address
was never added and `curl` in a VM with a working tunnel reported "Could not
resolve host". Absent env means "the caller sent none", not "this command
needs no environment".
- the deny check PASSED for the wrong reason — DNS was failing, so nothing was
refused by the allow-list at all. It now requires a 403 from the proxy, so it
cannot go green on a broken tunnel.
- `host_allowed` accepted `evil.test/api.anthropic.com`, which ends with an
allowed suffix. Hostnames are now validated against a character class, which
also refuses IP literals so an address cannot sidestep a list of names.
- `BufReader::into_inner()` discards buffered bytes: wrapping the stream twice
would have dropped the start of the TLS handshake and stalled a tunnel that
looked established. One reader now spans the request, and anything buffered
past the headers is forwarded as payload.
`iproute2` is in agent-toolchain because it is load-bearing: the guest's `lo`
starts DOWN, and while it is down a listener on loopback BINDS and then refuses
every connection with ENETUNREACH. fcagent finds `ip` by absolute path — as pid 1
its PATH comes from the kernel, and execvp's fallback excludes /usr/sbin, where
Debian puts it.
Egress needs both ends up, so `create` reports `egress` and the guest's `ping`
reports its own half. A VM without it is legal but never silent.
Verified on tank: 16/16 with backend=claude (create 1428 ms), 12/12 on the
default rootfs, no leaked processes, VM dirs or proxy sockets. 457 tests pass,
clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
c3297b86cf |
feat(fleet): B4.4 — credentials reach the microVM guest as exec env, and a bad entry refuses the exec
`claude -p` in the VM failed with "Not logged in". The credential now travels on
the exec op: `env` on `vm_exec` → fcagent → the command's environment. An env var
rather than a file because the per-VM rootfs dies with the VM but an env var
never touches the guest disk at all.
**Every problem in an env entry fails the exec.** The tempting alternative —
skip the entry we cannot use and run anyway — produces a `claude -p` with no
credential, and that does not error, it HANGS. A phase stuck at `running` for
ten minutes with nothing in the logs is exactly what a missing token looked like
on the container path. Names are validated ('=' or NUL would define a different
variable than the one asked for via putenv semantics), values must be strings,
and errors name the key and never the value — an error string travels back over
the wire and into logs.
One list of which credentials travel: `forwarded_provider_env` reuses
`forwarded_provider_keys`, and the container path now reads it too. If the two
execution paths diverged, a mission would behave differently depending on where
it landed — including the expensive way, where one path forwards
ANTHROPIC_API_KEY and bills it while the other uses the subscription. A blank
value is omitted rather than forwarded empty, so `claude` reports having no
credential instead of failing authentication with one.
Verified on tank (`--vm-selftest` backend=claude, 13/13, create 1532 ms): an
injected var reaches the guest command over the real vsock wire, and an
unusable entry comes back ok:false with no rc.
FINDING — the CLI leg remains UNPROVEN, and deliberately so. The guest has no
network interface: `create` writes boot-source, drives, machine-config and vsock
and no `network-interfaces` key, and a booted guest has no routes, no
resolv.conf, no DNS and no TCP. So `claude -p` cannot reach the API whatever
credential it holds. Injecting the real token would have proven nothing, because
the failure would have been network and not auth. Filed as B4.6 (task #49) with
the TAP-vs-vsock-proxy trade-off; B4.5 is now blocked on it.
444 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
0c291ed1bb |
feat(fleet): B4.4a — a real agent-claude microVM image, and a check that it has an agent in it
The only rootfs on this track came from clawmates/agent-terminal:dev. Mounted,
it held git and nothing else: no claude, no node, no cargo. A VM booted from it
looks perfect and cannot run a mission, so B4.5 could have been written and
never verified.
images/agent-toolchain — the shared mission toolchain (node 22, git, rust +
cargo-audit, gitleaks/trivy/semgrep, tea/gitea-mcp), lifted from the proven
deploy/clawmates-runtime image minus the zeroclaw daemon: a microVM mission runs
the direct-session model, so there is no daemon to host. A base image rather
than three self-contained Dockerfiles because this layer is ~3 GB and the real
risk is scanner and toolchain versions drifting between per-CLI images — the
evaluator runs the project's own suite to check a claim, so `cargo` present in
one image and absent in another makes the same mission pass or fail by backend
with nothing saying why.
images/agent-claude — plan A6, first of three: the pinned CLI and its env
contract only, so bumping Claude Code does not rebuild the toolchain and cannot
disturb agent-kimi / agent-glm. HOME=/root with an empty .claude for B4.4 to
inject into; no ANTHROPIC_API_KEY, since it silently overrides the subscription
OAuth we already pay for.
Both the builder and the node selftest now ASK the guest for the CLI the image
is named for, instead of trusting the name. `required_cli` maps claude/kimi/glm
to a probe; an unrecognised backend reports unchecked and prints SKIP rather
than passing quietly.
Verified on tank:
- rootfs-claude.ext4 boots; git, node, cargo, a real git commit all work
- `claude --version` → 2.1.220 over vsock, in both the builder and
`--vm-selftest` (11/11, create 1498 ms)
- negative control: the same builder run against agent-terminal with
FC_CLI forced reports `cli rc=127 claude: not found` and exits 1, so the
green result above is a measurement and not a default
- `claude -p hello` fails with "Not logged in · Please run /login" — the CLI
runs headless in the VM, and B4.4 only has to supply the credential
- no leaked firecracker processes or vm dirs afterwards
437 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
6687f8b808 |
feat(fleet): B4.3 — per-mission rootfs selection (missions.backend)
`vm_create` takes a backend name and boots `rootfs-<backend>.ext4`; NULL or "default" boots the golden image. Makes the per-CLI images from B4.1 actually reachable (one image per CLI, per A6). A missing image is an ERROR naming the file and how to build it, never a quiet fall back to the default. That fallback is the tempting version and the wrong one: it would run a claude mission in a kimi VM, or in a rootfs with no CLI at all, and report success for whatever came out. Verified on real hardware, not just in a unit test — the selftest asks for an image that does not exist and FAILS if it boots. `create` now reports the rootfs that actually booted, not the one that was requested, so a mission artifact can show the wrong VM ran. The migration adds no CHECK constraint listing the CLIs. Which images exist is a property of the NODES, not the schema; a constraint would need migrating for every new image while still not guaranteeing the image exists anywhere. The node validates and names what is missing. Backend names are `[A-Za-z0-9_-]` and rejected rather than sanitised, since they become filenames. Verified on tank: default backend 8/8; `CLAWMATES_FC_BACKEND=agent-terminal` 9/9 including the absent-image check, create in 910ms on a rootfs built from a real Docker image. 435 tests green, no leaked processes or VM dirs. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
08847e6a63 |
feat(fleet): B4.2 — static Rust guest agent replaces the python one
The python guest agent only ever worked because Firecracker's CI Ubuntu image happens to ship python3. NONE of our images do — agent-base has neither python nor git, agent-terminal has git but no python — so it could never have run in a real mission rootfs. An agent that dictates what must be installed in the image has the dependency backwards. crates/bins/fcagent is a 905K static x86_64-unknown-linux-musl binary that needs nothing from the rootfs it is dropped into. The wire is unchanged on purpose — 4-byte BE length + JSON, ops ping/exec/put/get — so microvm.rs and microvm_client.rs needed no edit at all. std has no AF_VSOCK and the workspace denies `unsafe`, so it uses the `vsock` crate. `process_group(0)` gives each command its own group without unsafe, so a command that spawns background children can be killed wholesale rather than outliving the run. A unit test caught a bug that would have broken EVERY exec: sourcing the image-env file with `. env.sh 2>/dev/null; cmd` returns rc=1 WITHOUT running cmd, because `.` on a missing file makes a non-interactive POSIX shell exit immediately. On any rootfs lacking that file every command would have failed while looking like an ordinary non-zero exit. Guarded with `if [ -f ]` now. Other places a failure must not borrow an outcome's representation: a killed command reports ok:false with no rc (not rc=124, which would read as a build failure); `get` on a missing path is an error, not an empty archive; a signalled process reports 128+signal rather than success. Verified on tank: --vm-selftest still 8/8 with the agent swapped (create 949ms, wire identical), fc-node-setup 8/8, and — the point of the change — a rootfs built from clawmates/agent-terminal:dev, which has NO python3, boots and reports `git version 2.39.5` from inside the VM. Also fixes a shell bug in fc-build-rootfs.sh: $HOME in a double-quoted default expanded on this Mac, so it looked for the node's binary under /Users/quantum on a Linux host. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
02ba557c3e |
feat(fleet): B3 — server-side microVM client over NodeHub
cm_api::microvm_client::MicroVm wraps the node's vm_* ops as typed calls
over the existing hub request/response channel: create / inject / exec /
collect / destroy, plus list() for reaping. No new transport.
Fixes a wire-contract mismatch B2 would have shipped. `Uplink::Result`
declares `output: String`, but the node's vm_* handler returned a JSON
object. The frame then failed to deserialize and hit the uplink match's
`Err(_) => {}` arm, so the reply VANISHED and every vm_* call would have
timed out after 20s with nothing anywhere explaining why. The node now
sends a string, matching the contract rather than what looked tidier.
That silent arm is fixed too: an unparseable frame now logs the node, the
parse error and the frame head, and says explicitly that the request it
was answering will time out. It is the arm that would have hidden this.
Two more places where a failure must not borrow a legitimate outcome's
representation:
- vm_exec returning no `rc` is an error, not a zero. A missing exit code
means the guest did not report one; reading it as success is how a
failed command becomes a passing phase.
- vm_collect on a missing path is an error, not an empty archive — an
empty tar looks exactly like a run that produced nothing.
Timeouts: the hub's deadline is the guest's plus 30s, saturating. A
caller passing a huge budget would otherwise wrap to a tiny timeout and
turn a long agent turn into a spurious transport failure. clippy caught
the tautological assertion in the first version of that test, which is
what surfaced the overflow.
Verified: `--vm-selftest` on tank still 8/8 after the output-type change
(create 950ms), 427 tests green.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
2d04c5e257 |
feat(fleet): B2 — vm_* node ops for Firecracker microVMs
create / inject / exec / collect / destroy / list, riding the node's
existing frame dispatch ({t, id, …} -> {t:"result", id, ok, output}), so
no protocol change was needed. Control is length-prefixed JSON over
vsock; the serial console stays a log, because feeding a guest over stdin
races its startup and arrives half-consumed.
DEVIATION FROM THE PLAN, deliberately: this does NOT implement
cm_sandbox::SandboxDriver. That trait is container-shaped —
attach_pty/resize_pty/argv exec — while missions need
create -> inject -> run -> collect -> destroy. Conforming would mean
building PTY-over-vsock and window-resize semantics that no mission path
calls, purely to satisfy a signature. We give up automatic RemoteDriver
marshalling; orphan reaping is a label/id sweep either way.
Three traps from the B0 spike are handled in code rather than remembered:
- Firecracker does NOT unlink its vsock UDS on exit, so destroy unlinks
it explicitly, and the selftest ASSERTS it is gone. Assuming the VM
tidies up after itself is how the mission checkout accumulated four
uid bugs.
- firecracker is spawned via setsid and killed as a process GROUP, so a
background child cannot outlive the VM holding its workdir open.
- create does not return until the guest agent has answered a ping. A
VM that booted but serves nothing is worse than one that failed, so a
half-created VM is destroyed rather than left registered.
A vm id becomes a path component, so ids are restricted to [A-Za-z0-9_-]
and REJECTED rather than sanitised — a caller that sent `../../etc`
wanted something we should not guess at.
Verified on tank through the real Rust path, as the daemon user, with no
sudo: `clawmates-node --vm-selftest` -> 8/8, create in 986ms, and the
host left with zero firecracker processes and zero VM directories. The
selftest asserts every step, including that a destroyed VM can no longer
be exec'd; a test that only reports the steps it completed cannot
distinguish "passed" from "stopped early".
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
0f7fa31f86 |
feat(fleet): B1 — microvm runtime kind and KVM placement predicate
Phase B step 1, on top of the B0 spike that proved microVMs boot here.
KVM is a HARD predicate, not a preference. gw-04 — where every mission
runs today — is itself a VM without nested virtualisation and has no
/dev/kvm, so a microvm mission landing there cannot start at all. The
scheduler therefore has to be able to tell nodes apart, which means the
node has to report what it can host.
Nodes gain a `capabilities` jsonb, populated from a probe on the node
rather than from configuration: /dev/kvm either exists there or it does
not, and nothing on the server can make it appear. The probe OPENS the
device rather than stat-ing it, because it can exist while being
unopenable (wrong group, or a container without the device passed
through) — which is precisely how firecracker will fail.
`microvm` requires BOTH kvm and a firecracker binary. A node with KVM
but no binary looks capable by the obvious test and fails at launch; a
node with the binary but no KVM is gw-04.
Placement fails the launch when no capable node exists, rather than
letting a mission sit in 'running' with nowhere to run. An explicit
target_node_id is treated as a request, not a guarantee — it is honoured
only if that node actually reports the capability.
`capabilities` defaults to '{}' NOT NULL so a node that has never
reported fails every predicate: an unqueried node and an incapable node
must be indistinguishable to the scheduler, because scheduling onto a
node whose abilities are unknown is how you get a mission that cannot
start and does not say why. The report replaces rather than merges, so a
capability the node has LOST disappears instead of leaving a stale true.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
107f0dbced |
feat(library): expose the library over the API
POST /api/library/runs harvests now; GET /api/library/items lists what the library holds. Thin wrappers — the work stays in crate::library — so a run can be started by a person, a schedule or the UI rather than only from an integration test. The response reports `healthy` explicitly rather than leaving a caller to infer it from an empty `shelved` list. A quiet week and a broken run both shelve zero papers, and collapsing those two is the exact ambiguity that cost most of this week. Failure reasons go to the log, not the response body: they can carry the remote URL and raw git stderr. AppState gains an optional blob store (the shelf), wired from the server binary where storage is already constructed. Optional because AppState::new is used by tests that never touch blobs; a route that needs it fails loudly rather than the constructor demanding it everywhere. 393 tests, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
f7e336ff5f |
fix(missions): make an unrunnable test suite legible, and check the runtime at boot
Two changes against the same defect: the platform could not tell a missing capability from a legitimate negative result. verify_tests returned Option<bool>, collapsing four outcomes into None: no suite found, docker unreachable, exec failed, and no exit status. When clawmates-runtime shipped without cargo, every on_green_tests phase returned None and landed on -wip — identical to the reading for "this repo has no tests", which is the conclusion I drew and reported. The gate was correct throughout; it simply could not say why it was unproven. TestOutcome now names the four cases. Gating is unchanged (only Passed clears, unproven is never a pass), and tests_verified keeps its tri-state meaning for existing readers. tests_status and tests_detail are new, so an artifact distinguishes no_suite from could_not_run, and a CouldNotRun is logged as the infrastructure fault it is rather than passing quietly. runtime_preflight probes the runtime container at boot for every tool the platform invokes inside it and names what each absence disables. This is the check that was missing: the Dockerfile gained a toolchain, the image was never built, gw-04 ran the old one for days, and the only symptoms were an ungated suite and a security scan that scanned nothing. A report, not a gate — a missing scanner should stop us believing a scan, not stop the server. Its test guards the probes themselves, since a typo would produce a permanent false "missing" and train operators to ignore it. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
09486ec759 |
perf(evaluator): judge with a bare API call instead of an agent (156x fewer tokens)
A phase verdict is a classification: fixed prompt, no tools, no memory, one JSON answer. Routing it through a ZeroClaw agent charged 17,772 input tokens to produce a 20-token reply, and at the runtime's 32k context that scaffolding — role prompt, tool descriptors, memory, identity — consumed over half the window before the judge read any evidence. The same verdict as a direct Messages API call costs 114 input tokens, with the real system prompt and evidence. Measured through the production seam via `cargo run -p cm-llm --example oauth_probe`. - cm-llm: teach AnthropicProvider subscription auth. A `sk-ant-oat…` credential switches to bearer auth, adds the Claude Code beta set, and prepends the identity line the API requires as the first system block — idempotently, so re-wrapping can't stack it or waste tokens. - evaluator: prefer a direct provider call whenever ANTHROPIC_OAUTH_TOKEN is set, falling back to the configured spec (including `runtime:<alias>`) otherwise. Fail-closed parsing is untouched and still governs every path. - The ANTHROPIC_API_KEY shape guard now points at the slot that understands bearer auth rather than only saying no. Deleting the agent from this path is the ablation applied to our own harness: the scaffolding was there because a judge was built like every other agent, not because a judge needs it. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
ca45597c79 |
feat(credentials): make provider substitution and runtime auth mode visible
Three guardrails around which credential pays for what. 1. Boot announces the mission-runtime auth mode, and warns when subscription auth is configured on a deployment with more than one user. A consumer subscription credential may only run the account holder's own work, and that condition is otherwise invisible -- it holds today and quietly stops holding the first time someone else signs up. Adds users::count_all (dynamic query, so the offline cache needs no regeneration). 2. Reject an ANTHROPIC_API_KEY shaped like a subscription OAuth token (sk-ant-oat...) at boot rather than failing on the first model call far from the mistake. Both credentials start sk-ant-, so the confusion is easy to make and hard to spot. 3. provider_alias_for's GLM/Kimi -> anthropic.default fallback was documented as deliberate but was silent in effect: a user picking "kimi" in the UI got an agent spending the Anthropic key, with nothing saying so. It now logs the substitution, and is_exact_provider_match() lets callers tell a real family match from a substitution so a UI can say which model will actually run. Behaviour is unchanged -- only the silence is. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
f848248fac |
feat(missions): goal conditions and phase iteration, judged on the subscription model
A phase used to complete when its topology_runs reached a terminal state --
purely structural. It marked itself done whether the agents produced the
artifact or wrote nothing at all, and it ran exactly once: execute_resumable's
skip(start) is resume, not repeat, and the only re-run path was a human
hitting the retry endpoint.
A phase can now carry `done_when`, a completion condition judged after each
pass against the evidence the agents actually surfaced. Not met and passes
remain -> the phase goes back to pending with iteration bumped, and the
verdict's reason is appended to the next pass's task text. That feedback is
what makes iteration converge rather than repeat -- the same mechanism /goal
uses, and that swarm.rs already uses for rejected work.
The evaluator runs on the SUBSCRIPTION model. CLAWMATES_EVALUATOR_MODEL
defaults to judge_model(), and a `runtime:<alias>` spec routes through
ZeroClawDriveExecutor -- a container agent on claude_cli, i.e. Claude Code on
the OAuth subscription, needing no platform API key. Same routing the door
governor uses.
Two deliberate departures from the governor's contract, both required:
- FAIL-CLOSED. Runtime::judge is fail-open and reads a verdict by
!contains("DENY"), so a model explaining why it *would* deny reads as
approval and an empty reply reads as approval. For completion that is
backwards: unsure must mean not done. The contract is swarm.rs's strict
JSON {"met","reason"} with .unwrap_or(false). Six tests cover the closed
paths -- prose, empty, missing field, non-boolean, transport error.
- judge_raw returns the raw reply; judge collapses to a bool too early to
carry a structured verdict.
Iteration scoping is the subtle part and has its own test: on pass 2 the
phase's own iteration is 1 but pass 1's completed run is still in the table,
so "are this phase's runs all finished?" must ask about the CURRENT pass or
that stale row closes out pass 2 the instant it is enqueued.
Evidence comes from phase_summarizer::collect_evidence, extracted from the
existing collect_material so the evaluator and the summary card cannot
disagree about what a phase produced.
done_when/max_iterations are promoted from phase config into columns (the
sweep filters on them every tick) and max_iterations is clamped to 20 at
insert -- the UI limits it too, but a runaway loop must not be one crafted
request away.
A phase with no condition completes exactly as before; that regression guard
is the first test in the file.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
49bcf53b84 |
feat(missions): wire the workflow registry so phase config reaches the database
workflow_registry.rs had zero call sites -- lib.rs declared the module and
nothing ever called load() or get(). So templates/workflows/*.toml was never
read, and because the client's TEMPLATE_PRESETS carries only {kind, order_idx}
with no config, PhaseSpec.config defaulted to Value::Null and every
wizard-created mission stored a null mission_phases.config.
Every per-phase setting was therefore inert. `loop = "until_no_more_int_items"`
and `commit_policy = "on_green_tests"` described a scheduler that does not
exist AND had no path to the database. benchmark_runner and security_scan
already read phase_config(); they were reading from null.
- Mission create derives phases from the recipe when none are sent, and
backfills config per phase (matched on kind+order_idx, then kind) when the
caller sends shape without config. An explicit config always wins.
- phases_for_create takes Option<&WorkflowRecipe> rather than reaching for the
global, because the registry resolves its directory relative to the process
cwd -- which under cargo test is the crate root, not the repo root.
- GET /api/workflows serves the catalog; the wizard fetches it and falls back
to TEMPLATE_PRESETS. Adding a TOML now adds a template with no FE change.
- load() runs at boot so a malformed recipe appears in the boot log instead of
silently producing a mission with no phase config.
Also fixes a latent bug in all five recipes: `default_team_template` was
written below the first [[phases]] block, and TOML scopes a bare key after a
table header INTO that table -- so it parsed as
phases[last].config.default_team_template and the real field was always None.
Invisible while the registry was dead code. Moved above the phases, with a
test asserting it neither returns None nor leaks into a phase config.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
Omar Sobh
|
5c63ef0ed3 |
missions: phase-completion summary card (Claude Opus 4.8 synthesized)
New phase_summarizer background worker fires on any mission_phase
transition to a terminal state (completed/failed). Aggregates every
topology_runs.checkpoint.outputs[] + mission_tasks + mission_artifacts
bound to that phase and asks Claude Opus 4.8 to produce a structured
JSON card:
{ narrative, metrics, sources, tooling, next_actions }
Rendered inline on the mission page under each completed phase via
new PhaseSummaryCard component. Metrics grid is kind-specific:
research surfaces insights/sources/int_cards/artifacts, coding
surfaces cards_picked_up/commits/tests/issues, benchmark surfaces
regressions/improvements, security surfaces findings-by-severity.
New table: mission_phase_summaries (migration 0060), unique per
phase_id — regenerates on retry.
New endpoint: GET /api/missions/{id}/phases/{phase_id}/summary.
Model overridable via CLAWMATES_SUMMARIZER_MODEL. Reuses the
ANTHROPIC_API_KEY prod already carries for mission_refiner.
|
||
|
Omar Sobh
|
69a6e4e7f2 |
missions: sweeper + socket-proxy NETWORKS grant + mount ordering (C3 slice 4-5)
- mission_runtime::spawn_sweeper: force-removes runtime containers for missions terminal for >=30 min, clears runtime_endpoint. Wired into clawmates-server main(). - docker-compose socket-proxy: NETWORKS=1 so bollard.connect_network can attach containers to clawmates_edge for provider egress. - phase_runner ordering: ensure_checkout BEFORE ensure_container so the mission dir exists before docker mounts it. - provisioner: mkdir_p the mission dir defensively for research-only missions that skip checkout entirely. |
||
|
Omar Sobh
|
277189ea9b |
missions: phase_runner — actually execute mission phases
Root-cause fix for "we hit launch, waited overnight, nothing ran."
mission_orchestrator materialized teams + agents fine, but nothing
enqueued the actual work — mission_phases stayed 'pending' forever
and topology_runs count for the mission was 0.
New crates/cm-api/src/phase_runner.rs — background worker on 10s
poll that does three things:
1. start_pending_phases — for every mission_phase with
status='pending' AND parent mission.status='running' AND all
lower-order phases already 'completed', enqueue one
topology_runs row per team whose (mission_id, purpose) matches
the phase kind:
phase=research → teams with purpose='research'
phase=coding → teams with purpose='coding'
phase=benchmark → teams with purpose='coding' (fallback)
phase=security_scan → teams with purpose 'security' | 'coding'
Each run gets a phase-kind-specific task text combining the
mission title/description + a directive for that phase.
Flips phase to 'running' after enqueue.
2. close_finished_phases — SQL sweep that flips phases whose
topology_runs are all terminal to 'completed' (or 'failed' if
any run failed).
3. close_finished_missions — same shape for missions whose phases
are all terminal.
Spawned alongside task_card_worker in clawmates-server main.rs.
Ordering enforced by mission_phases.order_idx — a coding phase
doesn't fire until its research phase completes.
Idempotent: every state transition is guarded so double-firing on a
race is safe. When a mission has no matching teams for a phase (bad
wizard state), the phase stays pending and the runner logs a skip
rather than getting stuck in a fail loop.
Existing topology_worker picks up the queued runs and drives them
through the ZeroClaw executor as usual.
|
||
|
Omar Sobh
|
d8c8793c4a |
ci fixes: cargo fmt, eslint entities, max-lines split
CI on
|
||
|
Omar Sobh
|
bf4af48c80 |
herdr phase 3: INFRA tier Herdr sessions surface
New INFRA category "Herdr sessions" (purple sparkles icon between
Fleet and Local hardware). Shows a card per online fleet node with:
- Node name + hostname + IP
- Per-workspace agent state pills (working / blocked / done /
idle / unknown), colored dots + pane count
- "Open" button → renders that node's full Herdr TUI inline via
xterm.js (same nodeHerdrConnector + WebRTC-with-fallback the
MissionCanvas Live Pane uses)
Backend:
- node daemon: herdr_workspaces + herdr_snapshot ops
(`herdr workspace list`, `herdr api snapshot`)
- fleet_herdr::snapshot helper on top of hub.call_timeout
- GET /api/nodes/{id}/herdr/session route
Fetch flow: /api/nodes filtered to status='online' → for each,
/api/nodes/{id}/herdr/session in parallel. Snapshot errors surface
per-card without failing the whole grid.
The "Open" xterm is separate from the MissionCanvas Live Pane —
this one is scoped to the whole node's Herdr TUI (any workspace),
not a specific mission's pane. Operator toggles between nodes via
the buttons.
Verified: cargo check --workspace + tsc --noEmit both green.
|
||
|
Omar Sobh
|
a5588b0289 |
herdr phase 2: Live Pane tab (xterm.js → node's herdr TUI)
The killer UX feature: click a mission's Live Pane tab and watch the
actual Herdr TUI on the target node in the browser — cursor, colors,
tool output, all live. WebRTC DataChannel direct where the browser
can reach the node peer-to-peer, WS-relayed fallback otherwise
(same auto-negotiation the INFRA node terminal already uses).
Zero new deployment infra — reuses the existing terminal_ticket +
terminal_ws + PTY-over-control-channel machinery. The one primitive
we grew: PtyTarget::Command variant so the node can spawn an
arbitrary program (\`herdr\`) in the PTY instead of the login shell.
Node daemon (clawmates-node):
- PtyTarget grows a Command { argv } variant
- spawn_command_pty resolves bare names against user + system bin
dirs (matches how tool_update finds claude/kimi)
- PtyTarget::from_frame reads the `command` array from the pty_open
frame; precedence Command > Container > Host
cm-api:
- NodeHub::open_pty grows an optional command argv; when set, the
frame carries it and the daemon spawns the program directly.
- routes::nodes::TermCtrl gains a `command: Vec<String>`; the
fallback branch threads it through.
Frontend:
- core.ts::webrtcConnector takes an optional commandOverride
that ships inside the fallback frame
- nodeHerdrConnector(nodeId) — mints the standard ticket + WS URL
but overrides command to ["herdr"]
- MissionCanvas grows a "pane" tab, visible only when
runtime_kind='local_herdr'. LivePane subcomponent uses xterm.js
(already a workspace dep) via useResilientTerminal, shows a
connecting/relayed/direct pill in the corner.
To watch a mission live: pick "On a fleet node (Herdr)" + target
node in the wizard, launch, click Pane tab → node's Herdr TUI
appears. Navigate to the mission workspace in the Herdr sidebar
(mouse or prefix+w) to zoom into the mission's pane.
Focus-a-specific-pane-directly is a later enhancement — Herdr has
no CLI arg for it yet, so operator navigates the sidebar for now.
Verified: cargo check --workspace + tsc --noEmit both green.
|
||
|
Omar Sobh
|
47f986257f |
herdr phase 1b: fleet_herdr dispatch module + node daemon ops
The second-runtime path uses the existing NodeHub control channel —
NOT SSH. Node daemons already accept typed ops over their outbound
websocket; adding three herdr_* ops keeps everything on the auth
model that already works fleet-wide (control-channel token, no new
SSH key management, no server-container-mounted keys).
Node daemon (clawmates-node):
- New herdr_op handler in main.rs dispatching:
* herdr_dispatch — workspace create + pane split + rename + run
* herdr_status — pane get JSON (agent, agent_status, cwd)
* herdr_read — recent-unwrapped scrollback, N lines
- Herdr binary resolved from ~/.local/bin, brew, /usr/local/bin.
Missing binary returns clean error so cm-api can distinguish
"node not set up for Herdr yet" from "Herdr op failed".
cm-api:
- crates/cm-api/src/fleet_herdr.rs — dispatch / status /
read_transcript / wait_for_completion helpers on top of
hub.call_timeout(). wait_for_completion polls until agent_status
hits 'done' or an idle-after-working state, matching the SKILL
file's "either idle or done is completed" semantic.
- routes::missions::herdr_dispatch — POST /api/missions/{id}/
herdr-dispatch { cli, prompt }. Requires runtime_kind = 'local_herdr'
and target_node_id set. Manual trigger so Phase 1b is exercisable
end-to-end before Phase 1c wires the wizard + orchestrator.
Not yet wired: mission_orchestrator::on_launch still ignores
runtime_kind. Phase 1c adds the wizard picker AND the on_launch
branch that auto-dispatches on draft→running for local_herdr
missions. This commit only adds the primitives.
Verified: SQLX_OFFLINE=true cargo check --workspace green.
Phase 0 (Herdr install on fleet nodes) is the blocker to actually
exercising this end-to-end.
|
||
|
Omar Sobh
|
fdb8cfeecc |
slice 9 cleanup: drop legacy research/loops backend + tables
Retires the legacy research/loops backend after the missions arc
(slices 1-9) fully replaced it. Frontend cutover was 4663348; this
commit finishes the job on the backend + database.
Migration:
- 0053_drop_legacy_research_loops.sql — drops the 8 legacy tables
(research_topics, research_topic_agents, research_outcomes,
research_publish_approvals, loops, loop_agents, loop_orgs,
loop_teams) and the 3 topology_runs FK columns
(research_topic_id, loop_id, iteration). parent_run_id stays;
recursive_exec still uses it.
Files deleted (11):
- crates/cm-api/src/routes/{research,loops,research_setup,
research_pipeline,wizard_repo,probe}.rs
- crates/cm-api/src/research_container.rs
- crates/cm-db/src/repo/{research_topics,research_outcomes,
research_publish_approvals,loops}.rs
- crates/cm-runtime/src/loops.rs
- crates/cm-api/tests/research_publish_role.rs
Files edited:
- crates/cm-api/src/lib.rs — dropped 20 legacy route registrations
(all /api/research/* + /api/loops/* + /webhooks/loops + probe)
and module decls
- crates/cm-api/src/topology_worker.rs — deleted legacy dispatch
(freeze_research_outcome, advance_loop_after_completion,
continue_initial_burst, maybe_transition_research_topic,
parse_reorder_rationale, per-topic/loop gateway resolver).
reap_stuck_runs now keys on mission_id (not topic_id).
Executor path unconditionally uses ZeroClawDriveExecutor::from_env
— mission_orchestrator provisions each claw as an agent inside
the shared runtime via RuntimeProvisioner, so per-team gateway
resolution is no longer applicable.
- crates/cm-api/src/routes/topology.rs — deleted container-log SSE
endpoint (research/loop-specific), dropped loop_id filter and
iteration field from ListRunsQuery/RunSummary
- crates/cm-api/src/routes/world.rs — removed
active_research_topics/active_loops/preseed_repo_paths;
World SSE no longer emits repo:{topic}/loop:{id} landmark orbs
(follow-up task #21 tracks adding mission:{id} equivalents)
- crates/cm-api/src/runtime_provision.rs — removed now-unused
mint_workspace_service_token
- crates/cm-db/src/repo/topology_runs.rs — removed 9 legacy
helpers (research_topic_id lookup, loop_id_for_run,
iteration_for_run, active_runs_for_research_topic, etc.)
- crates/cm-db/src/repo/teams.rs — removed 4 dead helpers
(team_for_loop, team_for_research_topic + setters)
- crates/cm-api/tests/topology_jobs.rs — removed loop/topic
tests, dropped enqueue_run_with_topic helper
- crates/bins/clawmates-server/src/main.rs — removed
spawn_loop_scheduler call
- crates/cm-api/src/routes/mod.rs, crates/cm-db/src/repo/mod.rs,
crates/cm-runtime/src/lib.rs — module decls stripped
sqlx cache: regenerated against post-migration schema
(71 files changed, ~+70 / -8896 net)
Test/build: SQLX_OFFLINE=true cargo check --workspace clean;
cargo test --workspace --no-run clean.
Follow-up (task #21): World view lost the in-flight-work landmarks
when repo:{topic} / loop:{id} orbs disappeared. Add mission:{id}
orbs as the missions-era replacement.
|
||
|
Omar SobhandClaude Opus 4.7
|
3ac3d53da7 |
slice 6: LLM + Chromium PDF renderer worker
Watches mission_artifacts for MD entries with render_pdf_status='pending'
and turns them into styled PDFs via:
1. Read source MD from <mission_root>/<path>
2. Call configured LLM (default gemini-2.5-flash) with a document-
typesetter system prompt that constrains style to a self-contained
HTML doc with inline CSS + our color palette
3. Print to PDF via `chromium --headless=new --print-to-pdf`
4. Save alongside source MD (foo.md → foo.pdf) + update
mission_artifacts.rendered_pdf_path + render_pdf_status='done'
Graceful degradation: GEMINI_API_KEY unset OR chromium missing =
row marked failed with a descriptive error, worker keeps ticking.
The frontend's "Open PDF" affordance (Slice 2) light up automatically
when render succeeds.
Boot ordering: PDF worker spawns after task_card_worker. Poll every
30s over up to MAX_PARALLEL=2 rows at a time — respects LLM rate
limits and keeps chromium's peak RAM under control.
Env knobs:
GEMINI_API_KEY — required for LLM step
CLAWMATES_PDF_RENDERER_MODEL — model id, default gemini-2.5-flash
CHROMIUM_BIN — chromium binary, default `chromium`
CLAWMATES_MISSIONS_ROOT — artifact dir root, default /var/lib/clawmates-missions
Dockerfile now installs chromium + fonts-liberation and sets
CHROMIUM_BIN=/usr/bin/chromium so the container image has everything
the renderer needs.
Also bumps workspace tokio deps to include the `process` feature
(required for tokio::process::Command).
Follow-ups:
- Anthropic + OpenAI provider variants (only Gemini in this slice)
- SSE stream on /api/missions/{id}/artifacts for the "PDF ready"
notification instead of poll-via-mission-GET
- Per-template PDF style overrides (currently one house style
for all missions)
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.7
|
f40ec075a5 |
slice 5: task-card parser + background worker
Watches topology_runs' event stream for the INT-XX marker protocol
(see skills/foundation/int-xx-marker-protocol.md) and materializes
mission_tasks rows with typed status so the canvas Tasks tab renders
a live timeline instead of raw agent chatter.
Migration 0051 adds mission_id + mission_phase_id columns to
topology_runs (nullable) so runs enqueued by a mission phase can be
attributed. Populated by future phase executors; NULL for legacy
research/loops runs (parser skips them cleanly).
New Rust surface:
- task_card_parser::parse(text) — line-scanner over TASK/WORK/
HANDOFF/TEST_PASS/TEST_FAIL/REVIEW_APPROVE/REVIEW_BLOCK/COMPLETED
markers. Strict: exact kind + colon + INT- prefix, no in-prose
matches, no bold/code-fence wrappers.
- task_card_parser::apply_for_run(pool, run_id) — reads the run's
mission binding, walks its event payloads, extracts text/output/
content/message string fields (matching every ZeroClaw event
shape we see), parses markers, UPSERTs mission_tasks via the
(phase_id, external_id) unique key from Slice 1.
- task_card_worker::spawn — 15s poller over runs updated in the
last 5 minutes. Idempotent + generous window survives server
restarts + task-scheduling jitter.
Boot wires the worker after the content loaders. Silent no-op when
mission wiring isn't populated yet.
MarkerKind → status mapping (monotonic-forward):
TASK → created
WORK → working
HANDOFF → validating
TEST_PASS → validating
TEST_FAIL → failed
REVIEW_APPROVE → validating
REVIEW_BLOCK → failed
COMPLETED → complete
Follow-ups:
- Wire phase executor to populate topology_runs.mission_id +
mission_phase_id (Slice 6/7/8 work)
- Assign assigned_agent_id via the event's producing agent alias
(currently always None)
- SSE stream on /api/missions/{id}/tasks for live canvas updates
(currently the canvas polls via mission GET)
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|