650a556029080ce735b9b505bc300aeca1a778f4
53
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar SobhandClaude Opus 5
|
869c3adcb7 |
fix(missions): egress from a subnet of their own, so the host can police it
docs/MISSION-EGRESS.md measured that a mission container reaches the entire
tailnet and SSH on its own host, and left the remediation unapplied. Applying
it on 2026-09-18 found why five iptables lines were never going to be enough:
missions egressed from clawmates_edge, the SERVER's network, and the server
needs the tailnet — Beszel on architect, Ollama for the local backend, the
node daemons for exec-test and node-placed terminals. A tailnet drop scoped to
172.23/16 cut the server off from architect:8090 inside a minute.
Missions now egress from clawmates_missions, 172.25.0.0/16, pinned so the
firewall can name it and declared in both compose files with the same shape
edge has. Compose v1 does not create a network no service uses, so on gw-04
it was created by hand with compose's own labels; the server's attach failure
message now says to check for it. core is unchanged: the door and API are
still reached over 172.20.
The policy itself (/usr/local/sbin/clawmates-egress.sh on gw-04, systemd unit
+ drop-ins on docker and tailscaled) lives in mangle/PREROUTING with
--ctstate NEW. Two earlier placements failed measurably: filter/FORWARD loses
to tailscaled re-inserting ts-forward above it on every restart, and
raw/PREROUTING runs before conntrack, so it dropped the server's replies to
tailnet clients and took the API off 100.102.112.85:8088. Verified from the
mission subnet (tailnet, host ssh, link-local blocked; public and core open),
from edge (tailnet open, ssh blocked), and inbound from tank; and proved to
survive restarting both daemons.
Also: deploy/compose/docker-compose.override.yml is tracked now. It holds the
fixes for the five local bring-up gaps and every credential in it is a
${VAR:?} reference, and it had lived on one laptop that lost a volume this
week.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
|
||
|
Omar SobhandClaude Opus 5
|
248948cc84 |
fix: three things that were known and written nowhere
All three have the same shape — the system learns something and only stderr
hears it — and each was flagged in the handoff as a silent-discard defect.
The gate's install outcome. `container_tool_hooks::install` returned Some or
None and both call sites wrote `let _ =`. A mission whose gate never installed
left a record indistinguishable from one whose gate stood there and matched
nothing. `EnsuredContainer` now carries the outcome to the callers that have a
pool, and they record `gate.installed` (with the settings path) or
`gate.absent` on the mission, so "was this mission gated?" is answerable from
the mission.
The inert marker. `vm_tool_gate` writes an `inert` file when it cannot parse
its input and allows everything, precisely so an inert gate does not look like
a permissive one. The only reader was a unit test. `drain_inert` now reads and
clears it at every tap drain, and a `gate.inert` event with the occurrence count
lands beside the calls that ran unchecked.
The judge's spend. `LlmEvent::Usage` arrived on every judge call and was
matched by `Ok(_) => {}`. Two plan exhaustions (2026-08-29, 2026-09-09) with
no row anywhere saying a judge token had been spent; `usage_events` had no
provider or model column. The loop now accumulates requests and tokens onto the
Verdict — counting a request BEFORE the stream opens, so a 429 the provider
refused still counts, because the retry storm was made of those — and
`record` writes a `kind = 'judge'` row with provider, model, mission and
request count. Migration 0085 adds the columns, all nullable, so the two
existing writers are untouched.
Tests: a scripted-provider verdict records one request and nonzero tokens; a
provider that refuses still records the request and zero tokens.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
|
||
|
Omar SobhandClaude Opus 5
|
bd7fd46305 |
fix(mission-runtime): a mission with no egress no longer launches
`clawmates_core` is `internal: true`. Verified from a container attached to it
and nothing else: no default route, and every external address unreachable —
the positive control fails, which is what makes the network's own configuration
visible rather than inferred.
So the attach to `clawmates_edge` is not an optimisation. Without it a mission
has no route off the host: no provider call, no fetch, no work. The result was
discarded:
let _ = self.docker.connect_network(EDGE_NETWORK, …).await;
which makes a failure here indistinguishable from success. The mission starts,
the phase runs, every tool call fails for a reason nothing reports, and the
phase can still reach `completed`. Green-with-nothing, again.
Not fatal on the error alone: re-attaching an already-connected container is
also an error, and a benign one on any relaunch path. So the container's own
network list settles it rather than the return code — already attached is
logged and continues, genuinely not attached fails the launch with a message
that says what it means. `inspect` failing counts as NOT attached, because the
whole point is to stop guessing that egress is present.
Behaviour change worth stating plainly: a mission that would previously have
run blind now refuses to start. That is the intended trade — a mission which
cannot reach anything cannot do the work it reports having done.
Suite: 108 binaries, 838 tests, green.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
|
||
|
Omar SobhandClaude Opus 5
|
4b160c5a1b |
test(orphans): prove the sweep against real containers, both directions
`sweep_orphans` force-removes containers and had never run against a daemon — only its pure decision logic was covered. The two Docker-touching seams are exactly the ones worth exercising for real: what it can see, and whether a checkout holds work no remote has. Three fixtures, three outcomes, one sweep: - unpushed commits, no remote ref → SURVIVES - every commit on a remote ref → reaped - inside the grace window → survives anyway Negative-controlled: making `unpushed_commits` return `None` for a dirty checkout fails with "the probe said a checkout with an unpushed commit holds nothing — this is the exact answer that destroys work". Two real hazards the test surfaced, neither of them in the sweep: 1. **The tests raced each other.** The sweep is global — it reaps every orphaned mission container on the daemon, including fixtures another test in this file just started. A `FIXTURES` mutex serialises them. Found the honest way: the reap test deleted the listing test's fixture and the listing test reported a container it could not see. 2. **The test destroyed real local state.** The sweep asks the DATABASE whether a container is known, and `test_pool()` knows nothing — so on a developer machine it classified the live stack's mission containers as orphans and reaped two of them on the first run. `adopt_existing` now gives every pre-existing mission container a row before sweeping, which makes the test safe AND covers the one case the other assertions missed: a container the platform still knows about is never touched. Negative-controlled both ways with a bystander container: without adoption REAPED, with adoption SURVIVED. Skips cleanly with no Docker, so a runner without one reports "not run" rather than failing — the placeholder-as-result shape `scripts/verify-mission-delivery.sh` was written to avoid. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9 |
||
|
Omar SobhandClaude Opus 5
|
6af1149e45 |
feat(missions): reap orphaned runtime containers — unless they hold work
`sweep_once` selects `FROM missions`, and `teardown_container` is only ever called with an id from that query. So a container whose row is gone is invisible to every reaper: nothing enumerates docker, nothing errors, and the only symptom is disk. Found on gw-04 today — `cm-runtime-mission-019ff5b1…`, Up nine days, 2.5G, against a `missions` table with zero rows. `list_mission_containers` is the piece that never existed: without it "which containers exist" is a question the platform cannot ask, and a container the database has forgotten is not merely unreaped, it is unseeable. **The sweep refuses to reap work that exists nowhere else.** That container's checkout held ten commits on a branch that had never been pushed — +3451/-30 across 30 files, eighteen INT items including AES-256-GCM, Ed25519 signing and HNSW batch insert. A reaper that deleted on sight would have destroyed all of it silently, as its designed behaviour. `unpushed_commits` asks the checkout (`git rev-list --all --not --remotes`) and leaves the container alone, loudly, every tick, when the answer is not zero. Every failure path returns `SomeOrUnknown`: a container we cannot question is not a container we may delete. Same for one docker will not date — including a future `Created` from clock skew, which would otherwise underflow into an age past any grace period. Grace is 24h, long on purpose. The row-driven sweep already handles everything the platform knows about, so anything reaching this path is already unexpected. The container above was handled by hand first: bundled, verified, branch pushed to git.redclaw.dev, confirmed on the remote at the branch tip, then removed. 59G free, up from 57G. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9 |
||
|
Omar SobhandClaude Opus 5
|
b89606fcf1 |
feat(missions): gate and observe tools on the container tier
The container tier is the one that actually runs missions in production, and it had neither a tool gate nor tool telemetry. The microVM tier has had both since yesterday; the tier that matters had neither. Both gaps have one cause. `claude_cli` runs claude as a subprocess, claude runs its tools inside that subprocess, and those calls never pass through ZeroClaw's executor — the only thing that emits TurnEvent::ToolCall and therefore the only thing the gateway turns into a frame ClawMates can see. Recovering the calls from the CLI's stream-json output did not help: a real mission produced zero tool.call events with the parser working perfectly. The transport was never the problem. Hooks are the way in, and they are proven. Claude Code reads hooks.PreToolUse / PostToolUse from the document given to `--settings` and honours them under `-p` — measured yesterday against the real binary, where the gate blocked a Bash call, recorded the payload, and got its refusal reason back to the model. So the same hook scripts the microVM tier uses are now written into the mission's container, and the provider is pointed at the settings document (`--settings` added to claude_cli in the fork, be9c34b1c). Composed in ONE script for one document: two writers of one settings.json is a silent clobber, and the microVM tier already learned that expensively. Installed on BOTH container paths — created and reused. A hook that exists only on first creation quietly disappears after a server redeploy, and the container outlives the server process. Everything degrades to "no hooks", never to a failed mission: a phase that runs unobserved still delivers; one that fails to start because telemetry could not be installed delivers nothing. Four tests, including two that exist because the halves are inert alone: the installer and the provider prop must both be wired (hooks nobody reads, or a document nobody wrote), and nothing may be written under /mission/repo, where it would arrive as part of the agent's delivered diff. Full workspace suite green: 107 binaries. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
548f977212 |
style: rustfmt the four files the repo-less mission fix touched
Found while removing .github/workflows/ci.yml: three of the four files in that change were unformatted, and four of the diffs were newly introduced (the new prompt tests and the tool_preamble format! call). Formatting only the files that change already touched — a repo-wide `cargo fmt` would be 63 files of unrelated churn and belongs in its own commit. Mechanical; `cargo test -p cm-api --lib` stays at 322 passed. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
4dec77ae6d |
fix(missions): give repo-less container missions the workspace they are promised
Every agent on a research_only mission refused to work, each reporting it was "in Claude Code", had no /mission/repo, and only had Read/Edit/Bash. All three statements were true. The run still recorded completed — 5 turns, 7.4k tokens, 0 artifacts, no error. The machinery is correct when a repo IS bound (verified on a live prod per-mission container: /mission/repo present, all 5 agents pinned). Only the repo-less path was broken, in three layers that disagreed by construction: - sync_in no-oped without a host checkout and copy mode does not bind /mission, so NOTHING created /mission/repo. The microVM tier already creates it, for the stated reason that "the guest needs the workspace to exist before the agent writes into it". Creating it host-side also un-breaks sync_out, equally a no-op before, so work survives across phases instead of being wiped. - pin_agent_workspaces returned Ok after pinning ZERO agents, so the deliberately-fatal guard in mission_orchestrator could never fire. Its error text already described the exact outcome we got. - The prompt advertised ZeroClaw tool names and explicitly denied `bash`, while every executor ends in `claude -p`: microVM passes Read/Edit/Write/Bash/Agent, session passes Read/Edit/Write/Bash, and claude_cli agents get Claude Code's native toolset — ZeroClaw's gating never reaches the subprocess. It was telling agents to use missing tools and avoid present ones. And it went green because mission_outputs logged the failed collect and continued — with the fail-empty rule and the NO-OUTPUT marker both BELOW that continue, so the phase was retried forever and never failed. The retry is now bounded by a grace window off completed_at. Verified end to end: mission completed, agent wrote /mission/repo/research/firecracker_vs_docker.md, collected and registered as a document artifact (6.6 kB of real content). Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
9f76f0915b |
fix(runtime): announce the runtime image once, not on every sweep tick
`MissionRuntimeProvisioner::from_env` is called per use — on every mission
launch and from the terminal-mission reaper sweep — so the line added in
|
||
|
Omar SobhandClaude Opus 5
|
cdc45bd082 |
chore(runtime): promote v0.8.4 from canary to the default image
Every mission on gw-04 was already running v0.8.4 — pinned by CLAWMATES_RUNTIME_IMAGE in .env. The canary is retired: the default tag `clawmates-runtime:sync` now IS that image, the override is commented out, and the built-in default is the single source of truth again. Promoting it exposed why the pin was load-bearing in the first place. The default tag resolved to zeroclaw 0.8.3 — two releases behind what was actually running — and the REGISTRY copy of the same tag was a different image again, 849MB against 2.31GB, without the Rust toolchain. A host that pulled `sync` rather than retagging it would have lost the on-green test gate with every probe still reporting success. A moving tag pointing somewhere old resolves perfectly, starts perfectly, and runs old code. Nothing anywhere said which image a mission got, so two things now do: - mission_runtime logs the image it resolved and whether that came from the env override or the built-in default, once at startup. - runtime_preflight probes `zeroclaw --version` alongside the other tools and prints every tool's VERSION, not just that it is present. A presence check passes happily on an image two releases behind, which is exactly what happened here and was found by running the binary by hand. Rollback is a retag: `clawmates-runtime:pre-v084-default` on gw-04 holds the previous default, and .env.pre-v084-default holds the previous pin. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
98037f9b3e |
feat(workforce): every mission hires its own crew
Reverses the reuse added earlier, by operator decision. Reuse hired the existing claw for a (template, slot) so the roster stayed at one team — but it also meant every mission was staffed by the same five names, and the workforce view showed one crew repeated down the page with nothing to tell the missions apart. Distinct crews read better than a bounded roster. The cost is the one reuse existed to avoid: claws are lifecycle='permanent' and nothing reaps them until their MISSION is deleted, so the roster now grows by the team size per mission. `agent_names::pick` keeps names unique workspace-wide and degrades to a numeric suffix rather than colliding, and the pool grew from 70 to 200+ given names so a workspace runs ~35 missions before the first repeat. `reusable_claw` is kept in cm-db with its tests: this policy has now flipped twice and the query is the hard part. Also revives a test that had silently stopped running. An edit stranded `runtime_data_is_scoped_to_one_mission`'s `#[test]` above its neighbour, leaving two attributes there and none here — so the neighbour ran TWICE and this one never ran at all. The total test count was unchanged by the fix (291 before and after), which is exactly why a count is not evidence: rustc had said "duplicated attribute" and "function is never used" all along, and both read as ordinary warnings. The test guards per-mission `/zeroclaw-data` isolation, i.e. one mission reading another's door token. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
0ad53da49c |
feat(workforce): missions group the roster, and agents get human names
Three things, all visible on the agents page.
**The roster looked like it was multiplying.** The sidebar flattened
orgs → companies → teams → agents, which renders a claw once per TEAM it
belongs to. Claws are reused across missions now, so a crew of five that had
run five missions appeared as twenty-five rows of the same five people. The
data was right and the view was lying. `GET /api/workforce` returns the roster
grouped by mission, and the tree renders each mission as a collapsible group,
so the repetition means something: the same colleague under each mission they
staffed. Claws on no mission come back under "Not on a mission" rather than
vanishing. The root now counts DISTINCT people, not rows.
**Agents were named after their jobs.** A team came back as planner, coder,
tester, reviewer, committer — the UI showed the same word twice (name on top,
role beneath) and the roster read as a stack of job tickets. New claws get a
given name from a deliberately wide pool (Amara, Vijay, Tomasz, Meredith…),
unique against the workspace roster AND within the team being minted. The role
is untouched in `job_title`, which is what the mission machinery binds on:
team_members.role_slot and the topology node carry the slot, so nothing
downstream keys off the display name. A reused claw keeps the name it had.
**Two latent reap bugs found while investigating a leak that was not one.**
Containers of completed missions are removed by `spawn_sweeper` after a
30-minute grace, and it works — an earlier report of leaking containers was me
reading that deliberate grace as a bug. But:
- the sweeper cleared the runtime binding even when teardown FAILED, and it
selects on `runtime_endpoint IS NOT NULL`. One transient docker error would
therefore hide a surviving container from the only thing that would retry
it, permanently. It now asks docker whether the container actually
survived: gone means clear, still there means keep the binding and retry —
which closes the orphan path without reintroducing the infinite retry the
original comment was guarding against.
- `set_runtime_binding` discarded rows_affected, so a mismatched workspace
updated nothing and returned Ok. The binding is how the sweeper finds a
container; a silent no-op there leaks one with no record of anything wrong.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
104e3ef27c |
fix(runtime): the kimi fallback hop spawned a binary that 401s
A throttled subscription had nowhere to go. `claude_cli.default` carried no
`fallback`, and neither target alias was declared — they existed only as
commented-out examples. Forcing a 429 with a shimmed `claude` surfaced four
defects that all read as correct config and do nothing:
- a `[providers.models.<f>.<a>.env]` SUB-TABLE is parsed then silently
ignored ("fields must live directly under ..."). This block was already
live for claude_cli.default, so the token injection has been inert. For
the glm alias it would have dropped the z.ai routing AND the clearing of
CLAUDE_CODE_OAUTH_TOKEN — credentials crossing between providers.
- an empty `[providers.models.kimi_cli.default]` is skipped at runtime.
- a claude_cli alias used as a fallback needs a non-empty `api_key` to pass
FamilyProviderFactory's default readiness gate, even though the provider
ignores the key and authenticates through `env`. Absent it the agent dies
at STARTUP, which takes out every mission, not just throttled ones.
- timeout_secs=600 capped every turn under the 3600s TURN_TIMEOUT from
|
||
|
Omar SobhandClaude Opus 5
|
f56d41f5b7 |
feat(backend): local-ornith — a mission backend served by the node's own GPU
Claude Code pointed at the Ollama already installed on every GPU node. Ollama has served a native Anthropic-compatible /v1/messages since v0.14, so this is an env contract rather than a translation layer — the fourth variation on the same idea as agent-glm and agent-kimi. The route is NOT the egress proxy, and that is the design. `egress` speaks CONNECT, takes a destination from the guest, resolves it and decides; every one of those powers is a liability, which is why it refuses non-443 ports and IP literals after a unit test caught them being bypassed. Routing a local model through it would have meant relaxing both. `local_model` is the opposite shape: there is no destination in the protocol. fcagent listens on guest 127.0.0.1:11434 and pumps to vsock 9003; the node splices that onto its own 127.0.0.1:11434 and copies bytes. A compromised guest cannot redirect it because there is nothing to redirect — it is a pipe, not a proxy, and strictly narrower than anything an allow-list could express. The bytes never touch a network, so there is no wire for TLS to protect, and Ollama stays bound to loopback rather than being exposed on the tailnet. The socket is bound only for a backend declared to use a local model, so a `local-ornith` VM reaches the forge through egress and nothing else, while every other backend's guest port simply refuses. Both halves have negative controls. `scripts/fleet-model-setup.sh` exists because of one measurement: stock ornith:9b reported input_tokens=2050 for a 48000-word prompt and answered as though nothing had been dropped. Ollama's default window is ~2K whatever the model card says, and it truncates silently — the exact failure an agent turn would hit and never report. The script pins num_ctx=131072 into a derived tag and then PROVES both the window and tool calling before declaring success. Verified on architect: ~65536 words -> 65604 input tokens, stop_reason=tool_use. Placement needs no new capability key: building the rootfs only on GPU nodes means `nodes::online_for_backend`'s existing `rootfs @> ["local-ornith"]` predicate does the affinity, so morpheus never offers the backend. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
f68fc019e4 |
fix(teardown): a mission dir with root-owned files is now actually removed
The server runs as uid 65532, so `remove_dir_all` on a mission directory returns PermissionDenied the moment anything root-owned is left in it — and the old code logged that at the same level as "file not found" and moved on. The directory then lived forever. After the seed-copy fix a mission holds 3281 files owned by 65532 and 26 owned by root: `.claude.json` and the session jsonl the per-mission ZeroClaw daemon writes itself, after the copy has been chowned. Twenty-six files is small enough to keep every mission directory alive without anyone noticing why. PermissionDenied now falls back to `root_copy::purge`, which deletes from inside the container as root — the same escape hatch `container_exec` keeps for exactly this, clearing debris a root process created. And if the directory survives even that, it says so, because a cleanup that silently failed is the thing being fixed. Removing the last 26 properly means running the per-mission daemon as 65532, which needs `/mission` pre-created in the image with that ownership — the daemon creates it at boot today and cannot at a lower uid. That is an image change, deliberately not bundled here. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
4967b9b8fd |
fix(runtime): the seed copy reads as root and hands the result to 65532
Running the seed copier as 65532 broke mission launch, and broke it quietly. The seed dir is root-owned with parts at mode 0600 (`.claude.json`, `clawmates-mcp.json`), so uid 65532 cannot READ them: `cp` failed on the first unreadable entry, `set -e` abandoned the rest, and the mission came up with a runtime-data holding `.zeroclaw` and nothing else — no Claude credentials, no door config. The daemon then never created its agents' workspace, and the phase failed 200 lines later on "Could not find the file /mission in container", which points nowhere near the cause. It was quiet because `seed_runtime_data` polled for the container to STOP and returned Ok without ever reading its exit code. A copier that died on a permission error and one that finished cleanly were indistinguishable. It now reads the status and says what went wrong. So: root for the read, `chown -R 65532:65532 /dst` for the result. Both halves matter and they pull opposite ways — root is needed to read the seed, and 65532 is needed because everything else in the missions tree is 65532 and a GC running as 65532 cannot delete what root left behind. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
8ddea454d1 |
fix(runtime): the seed copy ran as root too, ~3200 files per mission
The uid fix landed and the CHECKOUT came back completely clean — 0 non-65532 files under `repo/` after a benchmark run that builds and tests Rust. But the same mission still held 3247 root-owned files, all under `runtime-data/`. `seed_runtime_data` spawns a throwaway container to `cp -a` the runtime seed into the mission's directory and never set `user`, so it ran as root — the identical absent-`user` omission `container_exec` had, in a container create instead of an exec. The seed source is 65532-owned and the destination is created by the server (which itself runs as 65532), so the copy never had a reason to out-rank either. This is the tree a gateway GC has to be able to delete, and a GC running as 65532 cannot remove root-owned files — the cleanup-that-cannot-clean-up shape, found before writing the GC rather than after. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
91fbd2dc88 |
refactor: the missions root has one definition, not five
`mission_workspace::missions_root()` is now the only place that answers "where
does mission state live". It had fragmented into five: this function, private
`env::var("CLAWMATES_MISSIONS_ROOT")` copies in security_scan, benchmark_runner
and mission_outputs, and a hardcoded `MISSIONS_HOST_ROOT` const in
mission_runtime that read no env at all.
They agree on the deployed value, so nothing has broken. The risk is entirely
in what comes next: anything that sweeps or reclaims this tree has to be
sweeping the same tree the writers use, and five definitions cannot promise
that — a reaper written against one would silently leave the others' directories
behind forever, which is how the orphans got there in the first place.
A source-walk test fails any module outside `mission_workspace` that reads the
env var itself.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar Sobh
|
4193ae2cda |
feat(missions): a canary backend for testing a CLI version on the real path
Claude Code 2.1.223 -> 2.1.226 is worth taking (2.1.225 fixes a transient 401
that replaced a long-lived CLAUDE_CODE_OAUTH_TOKEN with a short-lived one and
broke HEADLESS sessions until restart — which for us means a failed phase). But
the image every mission uses is not the place to find out whether a new CLI
still delegates, still accepts `--settings`, and still finishes.
`canary-claude` is a real rootfs built from the candidate version, credentialed
identically to `claude`, so a mission can exercise it through the production
path: egress, stop gate, delegation, delivery, streaming. Testing a new CLI
against a different provider would not be testing the thing we are about to ship.
Named explicitly rather than matched on a prefix. An unrecognised backend must
still be refused at launch — that is what `backend_can_run_a_mission` and the
harness's `microvm-negctl` scenario assert — and loosening the credential map is
exactly how that guard gets softened by accident. A test pins both halves.
Already cleared by direct measurement in a booted 2.1.226 VM, before this:
- `--settings` and `--agents` still exist
- the workspace trust prompt added in 2.1.225 does NOT apply: `--help` states
the dialog is skipped in non-interactive mode (`-p`, or stdout not a TTY).
We use both.
|
||
|
Omar Sobh
|
87f188ae73 |
refactor: strip Gemini from the platform, and level up the architecture_mapper
Two things.
1. The architecture_mapper proposal, applied AND made durable.
The GLM proposal (019fddd9) was accepted in full: the agent's system_prompt now
carries the Mermaid-first constraint and its brain was rewritten. Both verified
against the live row and the .h5 file.
But `apply_identity` writes `UPDATE agents SET system_prompt` and
`apply_brain_consolidation` writes that agent's brain — neither touches the team
TEMPLATE. That agent is mission-scoped, so the improvement would have died with
the mission. The model's actual insight was sharp and worth keeping: "Mermaid
diagrams beat prose" lived in the brain SEED and not in the system PROMPT, so it
only applied when the agent happened to consult its brain. That constraint is
now in templates/teams/codebase_research.toml, where every future Codebase
Research team inherits it.
(The proposal's second item mostly restated anti-patterns the seed already
lists, so the seed is unchanged. Applying an LLM's suggestion is not the same as
agreeing with all of it.)
2. Gemini is gone.
Removed: the `gemini.default` provider alias and its `is_exact_provider_match`
prefix, GEMINI_API_KEY forwarding to agent containers, the evaluator's
gemini->gemini family row, the model selectors in claws/teams/planner and in
TeamWizard + AgentComputer, and the commented provider block in the runtime
config example (whose ZEROCLAW_AGENT_MAP example still mapped a worker_gemini
that no longer existed).
`provider_alias_for("gemini")` now returns claude_cli.default via the
unrecognised-model branch, which LOGS. A stray gemini binding degrades visibly
rather than resolving to a provider row we no longer ship. A test pins that, and
another pins that GEMINI_API_KEY is forwarded in NEITHER auth mode, so adding it
back to the list is a visible change rather than an accident.
Avatar generation is DELETED, not disabled — it called Gemini's image model, and
there is no alternative: Claude and Kimi are text-only, and z.ai answers
"Unknown Model" for cogview-3-flash and cogview-4 on our plan (measured, not
assumed). AvatarModal keeps UPLOAD, which never needed a provider; only the
prompt-generation half is gone.
240 backend lib tests, 89 frontend tests, clean tsc + eslint, build succeeds.
|
||
|
Omar SobhandClaude Opus 5
|
742724e53c |
feat(fleet): Kimi as a microVM backend — the URL settled by measurement
The base URL took three measurements to find, and the first two were wrong in instructive ways. `api.moonshot.ai/anthropic/v1/messages` EXISTS and speaks the protocol — it answers with Moonshot's own structured error rather than a 404. It also rejects an `sk-kimi-` key, because it belongs to the platform.moonshot.ai account namespace. Two endpoints that both "work" for different accounts is precisely the shape that makes a guessed URL look like a broken key, and it is why this was refused rather than guessed for as long as it was. The Kimi CODE service is the one an `sk-kimi-` key belongs to: `POST https://api.kimi.com/coding/v1/messages` returns a real Anthropic Messages body — `msg_` id, `content` blocks, a `thinking` block with a signature. So `ANTHROPIC_BASE_URL=https://api.kimi.com/coding`, WITHOUT the `/v1`: Claude Code appends `/v1/messages` itself, and `/v1/v1/messages` would 404 in a way that reads as a broken image rather than a bad URL. Two more measured, each otherwise a silent failure at the first turn: `Authorization: Bearer` is accepted (so ANTHROPIC_AUTH_TOKEN is the right injection channel), and a `claude-*` model id is ACCEPTED AND ANSWERED — Kimi maps it onto `kimi-for-coding` exactly as z.ai does, so no ANTHROPIC_MODEL override is needed. Claude Code rather than Moonshot's own `kimi` CLI, deliberately. The mission harness is Claude-Code-shaped throughout: `--agents` JSON roles, the verifier's tool allowlist, the `Stop` hook behind the completion gate, the per-subagent transcripts counted as delegation evidence. `kimi` has none of those flags — its equivalents are TOML files and markdown agent dirs — so using it would mean a second executor with its own untested failure modes. TWO STALE MAPS, caught by the rootfs harness refusing to bless the image: both `fc-build-rootfs.sh` and the node's `required_cli` expected backend `kimi` to contain Moonshot's `kimi` binary. That assumption predates the measurement, and it failed a rootfs that was correct. Both now say `claude` for glm and kimi alike — the binary is the same in all three images; only the endpoint differs. Egress for `kimi` is `api.kimi.com` alone: not moonshot.ai (wrong namespace), not z.ai, not Anthropic. Asserted both ways, like the other two. The image and rootfs are built on tank and the rootfs passes all four checks (boots, git, writable /mission, `claude --version`). 534 tests, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
f7f3dfe495 |
feat(fleet): GLM as a real microVM backend, and per-role models for claws
Three threads, all of which end at the same place: a mission whose verifier does
not share a model with the coder it reviews.
**GLM has a credential contract now.** `microvm_credential_for` returned one env
var name, which quietly assumed every provider reads its secret from the same
place Anthropic does. It returns a `Credential { source, target }` instead —
z.ai's key lives in the server's `ZAI_API_KEY` and Claude Code reads it as
`ANTHROPIC_AUTH_TOKEN`, and collapsing those two names is what forces a guess at
the other end. A wrong guess here sends one provider's credential to another
provider's endpoint.
`images/agent-glm` is the same CLI at the same pinned version as `agent-claude`
with `ANTHROPIC_BASE_URL` baked in. The split is deliberate: the ENDPOINT is a
property of the image, the CREDENTIAL is a property of the turn. That makes the
dangerous mix-up unrepresentable — a GLM VM cannot be handed an Anthropic
subscription token, and a claude VM cannot be pointed at z.ai. Asserted both
ways, because "the GLM VM must not carry CLAUDE_CODE_OAUTH_TOKEN" is the
property that costs a credential if it ever stops holding.
Kimi stays refused. `KIMI_API_KEY` is set and Moonshot serves an
Anthropic-compatible API, but I have not verified its base URL against the
running service, and this function is precisely where guessing a URL is
expensive. It becomes an arm the day someone measures it.
`api.z.ai` joins the node's default egress allow-list. A default that cannot
run the images we ship is a trap rather than a policy — the alternative is an
operator discovering it as a hung agent with no model access.
**Per-role models for claws** (migration 0071). `template_roles` had no model
column, so `mint_team_from_template` bound every role of every mission team to
one literal — a template whose whole point is an independent reviewer minted a
reviewer sharing a model with the coder. A role may now name its own; roles that
say nothing still take the mint's default, so every template written before this
behaves exactly as it did. The literal is now that default rather than a
hardcode.
**A harness scenario for the roster flow.** `verify-mission-delivery.sh roster`
runs the whole Slice 5 loop — planner proposes, human approves, mission runs —
and asserts the roster LANDED on the mission row rather than trusting the API's
answer. That distinction is not theoretical: the first live approval returned an
error while leaving the proposal marked approved.
Built and proven on tank ahead of the deploy: `clawmates/agent-glm:dev` reports
`2.1.223` and `BASE=https://api.z.ai/api/anthropic`, and
`fc-build-rootfs.sh … glm 8G` boots a VM from it that has git, can write
/mission, and answers `claude --version`.
533 tests pass, clippy clean. Migration 0071.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
aa470091aa |
fix(missions): a bootable rootfs is not a runnable one
Found by looking at what the fleet actually reports, not by reasoning about it: tank's `capabilities.rootfs` is `["agent-terminal", "claude", "default"]`. Slice 5 offered that list to the planner as the menu of backends and validated proposals against it — so a roster naming `agent-terminal` would have been proposed, validated, approved and launched, and then failed at the agent turn, because `microvm_credential_for` has no contract for it and refuses rather than forward an Anthropic subscription token to an unknown endpoint. Refusing at boot is correct and is exactly the wrong PLACE: it is three steps and one human approval after the point where the answer was already knowable. The menu is now the intersection of "a node can boot it" and "a mission agent can authenticate in it", which is what the question meant all along. `backend_can_run_a_mission` derives from the credential contract rather than restating it, so a backend gaining one (GLM and Kimi, when B4.6's base-URL contract is settled) becomes proposable in the same commit that makes it runnable — instead of in a second list someone has to remember. 528 tests pass, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
2edafdaf0d |
fix(fleet): a microVM authenticates by subscription only — never with an API key
B4.4 had the microVM path share `forwarded_provider_env(auth)` with the
container path, on the reasoning that the two must not diverge. That was wrong
in the one direction that costs money: gw-04 has CLAWMATES_RUNTIME_AUTH unset,
so the container path forwards ANTHROPIC_API_KEY today — and a VM would have
received it. Claude Code ranks the API key ABOVE the subscription's OAuth token,
so the VM would have worked perfectly while billing per-token against a plan we
already pay for. No error, no symptom but the invoice.
`microvm_provider_env` is subscription-only BY CONSTRUCTION: it does not take
the auth mode as an argument and does not read CLAWMATES_RUNTIME_AUTH at all.
Taking the mode as a parameter would mean one unset variable on a new host
silently turns the API key back on. The container path is unchanged and still
honours the operator's mode — the divergence is now deliberate, with the reason
at the definition.
Two other fail-closed rules fall out of it:
- A missing or blank subscription token REFUSES the launch rather than
returning an empty environment. A VM with no credential does not error;
`claude -p` hangs, which reads as a phase stuck at `running` with nothing in
the logs. The refusal names the variable.
- An unrecognised backend is refused rather than handed the Anthropic token.
GLM and Kimi reach their own endpoints via ANTHROPIC_BASE_URL and that
contract is not settled yet; guessing it would send a subscription
credential to z.ai.
Measured on tank, and this is the end-to-end proof B4.4 could not give:
`claude -p` in the agent-claude image with the real subscription token replies
"OK". Injecting the token in a VM moves the failure from "Not logged in" to a
network error, so the credential channel is accepted by the CLI — the VM's
remaining problem is egress (#49), not auth.
447 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
c3297b86cf |
feat(fleet): B4.4 — credentials reach the microVM guest as exec env, and a bad entry refuses the exec
`claude -p` in the VM failed with "Not logged in". The credential now travels on
the exec op: `env` on `vm_exec` → fcagent → the command's environment. An env var
rather than a file because the per-VM rootfs dies with the VM but an env var
never touches the guest disk at all.
**Every problem in an env entry fails the exec.** The tempting alternative —
skip the entry we cannot use and run anyway — produces a `claude -p` with no
credential, and that does not error, it HANGS. A phase stuck at `running` for
ten minutes with nothing in the logs is exactly what a missing token looked like
on the container path. Names are validated ('=' or NUL would define a different
variable than the one asked for via putenv semantics), values must be strings,
and errors name the key and never the value — an error string travels back over
the wire and into logs.
One list of which credentials travel: `forwarded_provider_env` reuses
`forwarded_provider_keys`, and the container path now reads it too. If the two
execution paths diverged, a mission would behave differently depending on where
it landed — including the expensive way, where one path forwards
ANTHROPIC_API_KEY and bills it while the other uses the subscription. A blank
value is omitted rather than forwarded empty, so `claude` reports having no
credential instead of failing authentication with one.
Verified on tank (`--vm-selftest` backend=claude, 13/13, create 1532 ms): an
injected var reaches the guest command over the real vsock wire, and an
unusable entry comes back ok:false with no rc.
FINDING — the CLI leg remains UNPROVEN, and deliberately so. The guest has no
network interface: `create` writes boot-source, drives, machine-config and vsock
and no `network-interfaces` key, and a booted guest has no routes, no
resolv.conf, no DNS and no TCP. So `claude -p` cannot reach the API whatever
credential it holds. Injecting the real token would have proven nothing, because
the failure would have been network and not auth. Filed as B4.6 (task #49) with
the TAP-vs-vsock-proxy trade-off; B4.5 is now blocked on it.
444 tests pass, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
1253595ba7 |
fix(missions): stop three launch failures from passing as success
A verification run against the deployed stack found a chain mission whose phase 0 reported `completed` with zero files, no commit error and no push error — indistinguishable from a phase that correctly had nothing to do. Three separate defects had to line up, each of them the same shape: a failure sharing its representation with a legitimate negative result. 1. `pin_agent_workspaces` embedded the whole config in one `sh -c` argv. That works until the file grows — config gains a block per provisioned claw — then fails with `argument list too long`. Now written through the tar upload API, which has no argv limit, so the failure mode is gone rather than merely further away. 2. A failed pin was logged "(continuing)". Without the pin, agents write to their sandboxes and the committer finds nothing in /mission/repo — the mission cannot deliver, so the launch now fails where someone is still looking. The restart that applies the pin is fatal for the same reason. 3. `capture_phase_diff_at` swallowed `git diff` failures with `unwrap_or_default`, so an unreadable base landed `empty: true, files_changed: 0` — byte-identical to an honest no-op. The error is now recorded as `diff_error`, and an empty patch that came from a failed diff is no longer trusted to mean an unchanged tree. Adds scripts/verify-mission-delivery.sh, which found #1 and #2 on its first real run. Its probes are fail-closed: no placeholder values, a self-test that proves the uid probe can detect the split it looks for, and FAIL-NORUN for a scenario that never executed. Its own first version had this bug too — a `die` inside `$(...)` exited the subshell, so a run that could not authenticate printed "all checks passed" and exited 0. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
7e07c389c6 |
feat(missions): wire copy-in/copy-out behind CLAWMATES_MISSION_FS=copy
With the flag set, ensure_container omits the /mission bind, the checkout is pushed into the container at phase launch, and the agent's work is pulled back before capture. The simplification that makes this small: sync_out unpacks over the SAME host path the checkout came from. The host directory stays a server-owned staging area with exactly one writer, and capture_phase_diff_at needs no change at all — it still finds a normal checkout exactly where it always has. Delivery, gating, commit and push are untouched. Two failures are deliberately loud rather than silent: - copy-IN failure fails the phase launch. Continuing would start a phase against an empty directory, and the agent would cheerfully report having done work on a repo that was not there. - copy-OUT failure SKIPS capture. Capturing anyway would diff a stale host tree and record "no changes" for work that exists — success reported for nothing, which is the exact failure mode this codebase keeps paying for. Opt-in: the bind path is what production has run since the beginning, and the test asserts a near-miss value leaves it there rather than silently switching every mission. 414 tests, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
0d9498ec6e |
fix(missions): copy an allow-list, not the whole 1.7GB seed dir
Checking before deploying caught a mistake in the previous commit. The seed dir on gw-04 is 1.7 GB and the first version copied all of it per mission — tens of seconds each, and ~17 GB across ten concurrent missions. 1.5 GB of that is .rustup: a Rust toolchain that installed itself into the data dir back when HOME=/zeroclaw-data and the image had no toolchain. The image now ships Rust at /usr/local/cargo, which is what the container's PATH actually resolves — verified live. The data-dir copy is dead weight and is not even reachable. SEEDED_PATHS now copies only what carries per-mission identity or secrets: .zeroclaw (config.toml with the door token, sessions.db, devices.db), clawmates-mcp.json, .claude + .claude.json, .kimi-code, glm-home, agents. Roughly 46 MB instead of 1.7 GB — about 37x smaller. Caches and toolchains are deliberately excluded: .rustup, .npm, .cargo, .cache, .local. They hold no secrets and a mission reads the image's. Absent paths are tolerated: a fresh deployment has no .kimi-code until Kimi is first used, and that must not fail container creation. The test asserts both directions — the token-bearing paths ARE copied and the caches are NOT — because either mistake is silent: copying everything just makes missions slow, and copying nothing quietly restores the credential sharing. 409 tests, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
15e7608e4a |
fix(missions): give each mission its own runtime data
Every per-mission container bind-mounted the SAME host seed dir as
/zeroclaw-data — shared with each other AND with the singleton runtime.
That directory holds config.toml, which carries the §15 door bearer
token, plus sessions.db and devices.db.
So one mission could read another mission's credential, and anything it
wrote there was inherited by every later mission. teardown_container
only removes /var/lib/clawmates-missions/{id}, so the shared directory
was never cleaned — the contamination was permanent.
The code already knew. The comment on DEFAULT_SEED_DIR names the sqlite
race and calls copy-on-write per mission the long-term fix. This is that
fix: seed_runtime_data copies the seed into
<missions_root>/<mission>/runtime-data at container create, and the
mount points there. Cleanup is free — teardown already removes that tree.
The copy runs in a throwaway container because cm-api cannot see the seed
dir: it hands that host path to Docker but never mounts it itself. The
runtime image is reused so nothing extra is pulled, and `cp -a /seed/.`
copies dotfiles — `/seed/*` would silently skip .zeroclaw/ and produce a
runtime with no config at all.
A copy failure is fatal to container creation on purpose. Falling back to
the shared mount would silently restore the credential sharing this
removes, and silent fallback to a weaker posture is the failure mode this
codebase keeps paying for.
The test asserts path shape rather than behaviour: an edit that points
the mount back at the seed dir restores credential sharing with no other
visible symptom, so the path IS the invariant.
408 tests, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
5d98fcf44a |
feat(missions): forward ZAI/KIMI keys so one binary serves three backends
All three providers run through the SAME `claude` binary, verified live: Anthropic CLAUDE_CODE_OAUTH_TOKEN -> ANTHROPIC-OK GLM ANTHROPIC_BASE_URL=https://api.z.ai/api/anthropic -> GLM-OK Kimi ANTHROPIC_BASE_URL=https://api.kimi.com/coding/ -> KIMI-OK That is a stronger multi-provider story than a provider-per-implementation: skills, subagents, MCP, hooks and tool policy are identical across all three because it is literally the same harness. The `kimi` CLI (0.31.1, shipped in the image) 401s on this key and is not needed -- the claude binary reaches Kimi's Anthropic-compatible endpoint directly. Worth knowing before someone debugs the CLI. forwarded_provider_keys now ships ZAI_API_KEY and KIMI_API_KEY into mission containers in BOTH auth modes: they are unrelated to the Anthropic credential, so the api_key/subscription split does not apply to them. A mission that selects a backend without its key present would otherwise fail at the first turn. Keys persisted in /opt/clawmates/.env and passed through compose. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
5232175c88 |
fix(missions): forward the subscription token into mission containers
Switching agents to claude_cli left missions hanging: the per-mission container had claude_cli configured but no credential, so `claude -p` waited forever. A phase sat at `running` for ten minutes with nothing in the logs — no error, because there is nothing to error on. The original subscription design assumed a persisted `claude /login` under a bind-mounted $HOME. That holds for the shared runtime and NOT for a mission container, which gets its own data dir and therefore no login. So subscription mode now forwards CLAUDE_CODE_OAUTH_TOKEN. The two Anthropic credentials remain mutually exclusive, and there is now a test asserting it in both directions: Claude Code ranks ANTHROPIC_API_KEY above the OAuth token, so shipping both bills the API while the deployment believes it is on the subscription — visible only on the invoice. Deployment: CLAWMATES_RUNTIME_AUTH=subscription and CLAUDE_CODE_OAUTH_TOKEN added to compose + .env on gw-04. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
a0e6b16abc |
fix(missions): stop agents having to work around git ownership
The captured diff from mission 019fc3ba contained the deliverable and, beside
it, a file the agent had invented:
+++ b/.gitconfig_temp
+[safe]
+ directory = /mission/repo
The server clones as uid 65532 and the mission container runs as root, so
every `git` an agent runs is refused with "detected dubious ownership". Agents
do not surface that as a failure — they improvise around it, and the
improvisation lands in the repository. Left alone it would have been committed
and pushed to the user's repo alongside the real work.
The judge got `GIT_CONFIG_*` for this in dd8dad2; the mission containers never
did. They do now — git's environment form of `-c`, inherited by subprocesses,
so it covers the agent's own git, the `git_operations` tool, and anything that
shells out. Scoped to the checkout, never `--global`.
`.gitconfig_temp` is also added to the capture exclusions. The cause is fixed,
but a stray workaround from some future agent should not reach a user's
repository, and the exclusion costs nothing.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
322c1be89c |
feat(missions): capture runs automatically, and once more before teardown
Wires diff capture into the two sweeps that matter. `phase_runner::sweep_once` gains `capture_finished_coding_phases`, guarded by `NOT EXISTS (code_diff for this phase)`. Deliberately a separate step rather than a hook on `close_finished_phases` or `evaluate_finished_phases`: a phase reaches `completed` through one or the other depending on whether it declared a `done_when`, so hanging capture off either would silently skip half the missions. The guard also makes it retryable — a capture that errors is simply re-selected next tick. `mission_runtime::sweep_once` captures anything still outstanding immediately before `teardown_container`, which deletes the checkout. This covers what the phase sweep structurally cannot: a mission that ended `failed` mid-coding still has real work on disk, and reaping it unexamined destroys the only evidence of what the agents actually did. Applies to coding, benchmark and security_scan phases — all three operate on a repo. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
c812b714f4 |
fix(evaluator): the verification sandbox never ran a command
`evaluator_tools::Sandbox::run` shelled out to `tokio::process::Command::new
("docker")`. The server image installs `git ca-certificates chromium
fonts-liberation` and nothing else, so in production every verification
command failed to spawn.
The failure was invisible in the worst way. `Sandbox::run` deliberately turns
execution failures into evidence text rather than errors, so a judge reasons
about "that command did not run" instead of the pass collapsing. With no
`docker` binary every command returned COULD NOT RUN, the judge correctly
concluded it could not verify, and fail-closed returned "not met". The
verdicts were right. The verification never happened — and the adversarial
validation that appeared to prove the feature working proved fail-closed
working instead.
The second defect made it worse: `checks` recorded the *attempt*, pushed
before the command ran, so a verdict reached with a dead sandbox reported
"verified by 10 checks" — a stronger claim than "no checks at all", made on
weaker evidence.
- New `container_exec` routes execution through the Docker API via bollard,
which was already a dependency and already reaches the daemon through the
socket proxy. Captures the exit code (absent from the old helper) and keeps
stdout and stderr apart (`LogOutput`'s Display merged them, which is why
nothing downstream could tell JSON from a progress bar). `security_scan`
parses stdout alone; `benchmark_runner` needs both.
- `ExecOutput::success()` requires `Some(0)`. An unreadable status is not
success — `commit_policy = "on_green_tests"` will gate on this, and
"unknown" reading as "green" would push untested work.
- `Sandbox::run` returns a `CheckOutcome` carrying `ran`/`refused`/
`exit_code`. `Verdict::verified_checks()` counts executions, not attempts.
- The UI gains a third state: "could not verify (N attempted, 0 ran)" —
precisely the case that used to render as verified.
- Regression tests reproduce the production shape: two checks recorded,
neither executed, `was_verified() == false`; plus a failing suite (exit 101)
still counting as verification, because that is something the judge learned
rather than was told.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
af44c92dd6 |
feat(runtime): let the mission runtime authenticate by subscription instead of API key
Claude Code resolves credentials in a fixed priority order and ranks ANTHROPIC_API_KEY ABOVE the subscription's CLAUDE_CODE_OAUTH_TOKEN. mission_runtime forwarded that key into every per-mission container unconditionally, so on a runtime authenticated with `claude /login` the key would silently win: `claude` still works, agents still run, and every mission bills the API while appearing to use the subscription. There is no error to observe -- the only symptom is the invoice. CLAWMATES_RUNTIME_AUTH = subscription | api_key now gates the forward list. In subscription mode ANTHROPIC_API_KEY is withheld; Gemini/Groq/OpenAI still forward in both modes since they have no subscription equivalent. The mode is logged per container so it is visible in the deploy log rather than inferred. Default is api_key -- today's behaviour exactly. An unset or misspelled value falls back to it too, because defaulting to subscription on a typo would strip the key and leave missions with no credential at all. forwarded_provider_keys() is the single source for the list, called by both ensure_container and the tests, so the two cannot drift -- the failure mode here is invisible, which is precisely when duplicated knowledge is worst. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
06ae608d0c |
fix(missions): provision claws into the mission's own daemon + reload the pin
Mission turns execute against the per-mission runtime container, but claws were provisioned via RuntimeProvisioner::from_env() — i.e. the GLOBAL gateway. That daemon loads config once at boot and never re-reads the file, so the per-mission daemon had no claw_* agents at all: querying it for a mission claw's risk_profile returned 404 while the global daemon returned 200. With the alias unresolvable, the daemon silently fell back to the default `scout` agent, which is jailed to the global workspace — agents reported "the scout agent workspace" and "/mission/repo isn't accessible", produced no files, and burned tokens. This is the deeper cause behind the empty-output runs; the tool-allowlist and workspace-pin fixes were necessary but not sufficient. - RuntimeProvisioner::for_gateway(url) — aim the provisioner at a specific gateway (mirrors ZeroClawDriveExecutor::from_env_for_gateway); from_env now delegates to it. - mission_orchestrator captures the per-mission endpoint from ensure_container and provisions every claw there, falling back to the global gateway only when there is no per-mission runtime (dev/no-docker). - workspace.path is file-only (the config prop API cannot set a PathBuf), and the daemon never re-reads the file, so pin_agent_workspaces is now followed by restart_container(): restart + wait for /health to answer. Agents created through the daemon's own config API are already persisted to that file, so they survive; the pairing code is re-minted on every launch. The readiness probe inspects the /health BODY — exec_capture only fails on docker errors, so a curl that cannot connect still "succeeds". Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
Omar SobhandClaude Opus 4.8
|
bf4ef4c4bf |
fix(missions): reap all mission resources on delete (no hanging claws/files)
DELETE /api/missions/{id} was a bare `DELETE FROM missions` relying on FK
cascades that only cover mission-owned tables. Everything the mission
provisioned leaked: per-mission runtime container, host workspace dir,
teams (created lifecycle=permanent, so no cascade + skipped by the
ephemeral-teardown path), and every claw's ZeroClaw config, .brain files,
and DB rows. Observed live with 0 missions in the DB: 174 orphaned gateway
claw configs, 7 orphaned teams, 31 agents, 39 .brain files, 6 workspace
dirs, a 4-day-old orphaned container, and 123 detached topology_runs.
delete() now calls reap_mission_resources() before the row delete:
- resolve the mission's teams (mission_teams) → claws (team_members)
- per claw: deprovision_claw (gateway) + rm .brain files + hard_purge (DB),
reusing the manual agent-reap pattern in routes/claws.rs
- delete the permanent-lifecycle teams (team_members cascades)
- delete the mission's topology_runs (else they linger with mission_id
nulled by the cascade and accumulate)
- teardown_container(), now extended to also rm the /mission/repo workspace
dir and tolerate an already-gone container (idempotent for the sweeper +
delete paths)
Runtime-side steps are best-effort (Postgres authoritative; fleet sweeper
reconciles daemon config); DB purges are logged on failure but never block.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.8
|
34409bca0c |
fix(missions): grant coding tools + pin claw workspace to /mission/repo
Mission agents were burning ~275K tokens producing nothing: the coder had only file_read and its workspace was the empty ephemeral sandbox, so it dumped a full spec inline instead of writing files. Two root causes: 1. Risk-profile allowlists used pre-0.8 tool names. `coding_readwrite` allow-listed `file_write` (renamed to `file_edit` in ZeroClaw 0.8, and `file_write` now refuses on ephemeral workspaces) and omitted file_edit / content_search / glob_search / git_operations — the exact tools the phase prompt tells agents to use. Since allowed_tools is a strict allowlist, agents were effectively read-only. Documents the correct profiles in agent.config.example.toml (they only lived in host config; the live runtime profiles were corrected via its config API). 2. workspace.path never got set. `agents.<alias>.workspace.path` is an Option<PathBuf> the ZeroClaw Configurable macro skips from prop enumeration, so provision_claw's set_prop always 404'd and the whole call errored into a swallowed eprintln. Removes the dead set_prop and pins the workspace out-of-band: MissionRuntimeProvisioner:: pin_agent_workspaces patches the shared config file on the per-mission container (format-preserving via toml_edit, atomic temp+mv); the daemon applies it on the same reload that surfaces the freshly-provisioned claws. Covered by unit tests for the TOML stamp. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
Omar Sobh
|
1be3430bf2 |
fix(mission_runtime): remove ZEROCLAW_WORKSPACE env — it was hijacking config-dir
Deprecated ZEROCLAW_WORKSPACE env var (schema.rs:17467) is used by the daemon as a legacy config-dir pointer that overrides everything else. Setting it to /mission/repo made the mission daemon compute its config dir as /mission/repo/.zeroclaw (empty) and fall back to defaults — zero agents loaded. This is the actual root cause of Unknown agent errors on WS. The seed-mount + admin/paircode/new + per-node-agent-injection fixes we shipped earlier were correct but couldnt take effect because the daemon wasnt reading our bind-mounted config at all. Per-agent workspace pinning belongs in config.toml as agents.<alias>.workspace, not env. |
||
|
Omar Sobh
|
8ba9bf0c1c |
fix(mission_runtime): re-add shared /zeroclaw-data mount for agent library
Fresh runtimes had zero agents in their config so WS handshake with ?agent=scout returned 400. Bind-mount the shared runtimes data dir so per-mission gateways inherit the seeded claw_* agents. Per-mission pairing (minted via /admin/paircode/new) still works against the shared devices.db — each mission gets its own accepted token. Concurrency caveat on sqlite sessions.db documented in the const doc comment. |
||
|
Omar Sobh
|
70e7ab3ad6 | fix: rename remaining scrape_pairing_code call site | ||
|
Omar Sobh
|
54bba1e113 |
fix(mission_runtime): mint pairing code via admin endpoint, not log scrape
Fresh gateways sometimes boot claim-ing already paired (no pairing_code in the log banner), which broke the log-scrape approach. Instead, docker exec into the container and hit the localhost /admin/paircode/new endpoint that always mints a fresh one-time code and returns JSON we can parse. |
||
|
Omar Sobh
|
5f4407e889 | fmt: import ordering | ||
|
Omar Sobh
|
37f3f5abfd | fix: mission_runtime_pairing_code in single-row mapping + fmt | ||
|
Omar Sobh
|
b569688e04 |
fix(mission_runtime): per-mission auto-pair via container log scrape (C3 auth)
The seed-mount approach didnt work: even with the shared runtimes data dir bind-mounted, a fresh gateway instance mints a new pairing key and requires re-pairing. The topology_worker connect returned 401 forever. New approach — per-mission gateways self-pair: - Provisioner tails container logs after start, extracts the X-Pairing-Code from the boot banner - Persists it on missions.runtime_pairing_code (migration 0059) - topology_worker constructs ZeroClawDriveExecutor with THAT code via from_env_for_gateway_with_code, which triggers the lazy /pair handshake on first turn and caches the returned bearer Drops the shared-runtime data-dir mount — each per-mission gateway now owns its own state, restoring the C3 isolation guarantee. |
||
|
Omar Sobh
|
033fcb98f1 | fmt: mission_runtime seed_dir | ||
|
Omar Sobh
|
c4e7ca8aa4 |
fix(mission_runtime): seed per-mission gateway with shared pairing state
Fresh mission runtime containers had no ZEROCLAW pairing token so the topology_worker got 401 Unauthorized on WS connect. Mount the shared runtimes /root/clawmates-runtime/data as /zeroclaw-data so the gateway boots pre-paired and accepts the servers ZEROCLAW_TOKEN. Seed dir overridable via CLAWMATES_RUNTIME_SEED_DIR. Known caveat: sqlite sessions dir is shared across concurrent mission runtimes. Fine while topology_worker runs sequentially per mission; next iteration should copy-on-write per-mission. |
||
|
Omar Sobh
|
5a1fcba403 |
fix(mission_runtime): full-uuid container names + assertion fix
UUIDv7 encodes time in the leading bytes so 12-hex prefixes are NOT unique across missions minted in the same second. Docker accepts up to 253 chars; use the full uuid. |
||
|
Omar Sobh
|
bb5cfc1519 | fmt: mission_runtime sweeper | ||
|
Omar Sobh
|
69a6e4e7f2 |
missions: sweeper + socket-proxy NETWORKS grant + mount ordering (C3 slice 4-5)
- mission_runtime::spawn_sweeper: force-removes runtime containers for missions terminal for >=30 min, clears runtime_endpoint. Wired into clawmates-server main(). - docker-compose socket-proxy: NETWORKS=1 so bollard.connect_network can attach containers to clawmates_edge for provider egress. - phase_runner ordering: ensure_checkout BEFORE ensure_container so the mission dir exists before docker mounts it. - provisioner: mkdir_p the mission dir defensively for research-only missions that skip checkout entirely. |