19c4de36e4459a65a9c96f410f5adf9b550c914a
870
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar SobhandClaude Opus 5
|
9f76f0915b |
fix(runtime): announce the runtime image once, not on every sweep tick
`MissionRuntimeProvisioner::from_env` is called per use — on every mission
launch and from the terminal-mission reaper sweep — so the line added in
|
||
|
Omar SobhandClaude Opus 5
|
cdc45bd082 |
chore(runtime): promote v0.8.4 from canary to the default image
Every mission on gw-04 was already running v0.8.4 — pinned by CLAWMATES_RUNTIME_IMAGE in .env. The canary is retired: the default tag `clawmates-runtime:sync` now IS that image, the override is commented out, and the built-in default is the single source of truth again. Promoting it exposed why the pin was load-bearing in the first place. The default tag resolved to zeroclaw 0.8.3 — two releases behind what was actually running — and the REGISTRY copy of the same tag was a different image again, 849MB against 2.31GB, without the Rust toolchain. A host that pulled `sync` rather than retagging it would have lost the on-green test gate with every probe still reporting success. A moving tag pointing somewhere old resolves perfectly, starts perfectly, and runs old code. Nothing anywhere said which image a mission got, so two things now do: - mission_runtime logs the image it resolved and whether that came from the env override or the built-in default, once at startup. - runtime_preflight probes `zeroclaw --version` alongside the other tools and prints every tool's VERSION, not just that it is present. A presence check passes happily on an image two releases behind, which is exactly what happened here and was found by running the binary by hand. Rollback is a retag: `clawmates-runtime:pre-v084-default` on gw-04 holds the previous default, and .env.pre-v084-default holds the previous pin. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
d810fc0a86 |
fix(viz): read the gateway's real tool_call keys, and correct the record
The frame is `{"type":"tool_call","id","name","args"}` — zeroclaw-gateway
/src/ws.rs. The tap read `tool` then `name`, and `arguments` then `input`.
`name` happened to be in the fallback chain; `args` was not in it at all,
so a container-tier tool call would have been recorded with its name and
NO path — a tool that reads as having touched nothing. `tool` and
`arguments` belong to `approval_request`, which is where they came from.
Also corrects what the histogram was read as saying. A mission turn on
gw-04 carried only chunk/done/session_start, and the first reading was
"there is no tool_call frame". Wrong: `grep -c tool_call` on the deployed
0.8.3 binary returns 46. Container-tier agents are provisioned tool-free
behind the MCP door (§15), so they call nothing — there is nothing to
observe on that tier, and nothing is broken.
That distinction is exactly what the histogram was shipped to make
possible: a tap matching no frame is otherwise indistinguishable from a
mission that used no tools.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
31158467f4 |
feat(viz): microVM tool motion is live, and needs no fleet-node change
The plan deferred this as "the only fleet-node binary change". It is not one. `fcagent` is thread-per-connection — its own comment says so, and the live log tail has relied on exactly that for the whole length of a turn, on a second connection. So the host can drain the tap WHILE the turn's exec is in flight, from the server alone. The turn and a 20s drain loop now run concurrently. A coding phase shows its files being touched as it works rather than an hour later, all at once, and the drain is bounded by a cursor so a repeated poll returns only what is new. The cursor counts LINES, not parsed events, and that distinction is the bug this commit would otherwise have shipped. The hook appends the event and then a newline of its own, so a two-event tap is four lines; advancing by event count leaves the cursor two lines short, `tail -n +N` hands back events already recorded, and the live drain re-records everything it has already written — worse the longer the turn runs, and silent throughout. Caught while writing the test, not by it. `tap_sink` and `VmOutcome::tools` are mutually exclusive by contract: with a sink, the sink owns recording including the final batch and `tools` comes back empty. Handing the same calls back on both would double every file orb's weight with no way for the caller to tell which it was looking at. The sink is an unbounded channel to a recorder task, so the VM executor stays free of the database: it observes, phase_runner records. The task ends when the sender drops with the phase. Verified before this change: the microVM tap is real. The `microvm` scenario passed 6/6 and left ten `tool.call` rows and a `file.touch` on MICROVM.md, repo-relative, from Claude Code's own PostToolUse hook. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
f8438c32ea |
feat(viz): kind-specific choreography and a finished mission you can read
Security: the pawns already orbited their destination, so homing them at
the security station gave circling for free. This adds the radial
press-and-retreat — an agent closing on the target and backing off reads
as probing it, where a fixed radius reads as waiting — and holds the
stochastic target release while probing, or the circling breaks up into
stray trips that look like distraction rather than a scan.
Findings are `mission_tasks` rows, one orb each, popped once. There is
deliberately no severity anywhere in the path: the scanner keeps
severity, file and line as substrings inside `title`, so a severity
parsed out of prose and rendered as an orb's RADIUS would be the picture
asserting a measurement the data never contained. Count only.
Benchmarks annotate the station, as text. `delta` has no schema —
compute_delta emits `{kind:"opaque"}` whenever the before/after metrics
were not structurally comparable, which is most drivers. The server
formats the shape it can parse and COUNTS the rest; an unparseable driver
reports "3 sample(s)" rather than an invented improvement, and an opaque
delta says nothing at all.
The finished map: the live label rule gates service/event nodes on
`heat > 0.12`, which is exactly backwards once everything has cooled — a
static map would be unlabelled dots. Frozen, the 25 most-touched nodes
label regardless of heat, phase stations carry a second line counting
what they produced, and the camera is released ONCE so it frames the
result even if the user panned during the run.
Every count in a caption is read off the drawn scene rather than a
parallel tally, so the words and the picture cannot disagree.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
9e61e3ba35 |
feat(viz): what the agents actually did, as structured events
The World could draw a mission's shape but nothing about the work. The
detail existed only as prose in checkpoint.log and model output, where a
tool name is indistinguishable from an agent *talking about* a tool — so
it was never parsed, deliberately. `mission_events` is the structured
channel that replaces it.
Three taps, one table:
- Container tier: the `_ => {}` at the end of topology_exec's typed frame
stream now matches `tool_call` and reads the tool's JSON ARGUMENTS for a
path. Never the prose summary — a path scraped from a sentence would put
files on the map that no agent opened, and the test proves a Grep whose
summary says "src/main.rs" produces no file touch. The frame name itself
is unverified, so the same commit ships an unmatched-frame-type
histogram: a tap that matches nothing looks exactly like a mission that
used no tools, and this is how one gw-04 run names the real frame.
- microVM tier: a `PostToolUse` hook, the seam vm_stop_gate already proved
fires under `claude -p`. It copies stdin to /root/tap and exits 0
unconditionally — a non-zero PostToolUse hook talks back to the model,
which would turn the observer into a participant. Drained before collect,
since the VM is destroyed moments later.
- Phase transitions: five identical copies of the pending→running UPDATE
became one `mark_phase_running`, and `close_finished_phases` grew
RETURNING. Its CASE decides each phase's status inside SQL from rows the
statement does not change, so it cannot be re-derived afterwards without
writing that CASE twice — without RETURNING it emits zero phase.completed
and reports success.
The settings.json hazard the plan called out: the stop gate wrote the
WHOLE document, so a second hook writer would have silently erased it and
a coding phase would then complete having written nothing — the exact
failure the gate exists to catch. There is now one composer,
`vm_tool_tap::guest_settings`, one writer, and a source-walk test that
fails if anything else writes a settings document.
`mission_events.run_id` carries no FK on purpose: phase_runner DELETEs
topology_runs on retry, and a cascade would erase a phase's whole history
the moment it retried — silently, since a cascade is not an error.
world.rs streams it with a cursor that separates backfill from motion.
Everything already in the table when a subscriber arrives is drawn as
settled history; only what lands afterwards animates. Otherwise opening a
finished mission replays an hour of tool calls as a burst storm.
Bounded twice: 400 events per phase (enforced inside the INSERT, since
two concurrent taps would each read a count below the cap) and a 7-day
retention sweep in mission_gc.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
c2fa8067e1 |
feat(viz): the coding station fractures into the files it worked on
The engine already turned `file:src/lib/a.ts` into a real dir chain, but both copies of that loop rooted it at the origin — so a mission's files floated beside the map instead of belonging to the work that produced them. One `fileParent` helper now serves both call sites; splitting them was how half the files could end up nesting correctly and half not, decided by whichever code path saw the file first. Files hang under the coding station when there is exactly one, else the single running phase, else the origin. `world.touch` carries no phase id, so with two coding phases any attribution is invented — the fallback is the honest answer. Two ordering hazards, both silent: - the server emitted files BEFORE phases, so on the first pass a file arrived with no station to hang under and first-write-wins pinned its tree at the origin. Loops reordered, with a source-walk guard. - `setFileHome` re-parents trees rooted before the plan landed, for the reconnect case the ordering alone cannot cover. `mission.file` with `source: "tool"` is treated as motion (burst, pawn beams); `"diff"` is end-of-phase truth and only marks the file present and warm — bursting every file of a captured diff would set the whole map alight at once on reconnect. Directories taper in radius and opacity by path depth, so `src` and `src/lib/live` no longer render as identical dots. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
cb8184e784 |
feat(delivery): record WHICH files a phase touched, not just how many
`capture_phase_diff_at` parsed `git diff --stat` down to three integers and
threw the filenames away. Nothing downstream could name a single file a coding
phase changed: the World can draw a coding station but nothing underneath it,
and an operator reading a mission sees "11 files" with no way to learn which.
A second `--name-status` call now records the paths into the code_diff metadata
and into `names.txt` beside `diffstat.txt`, so raw evidence survives
independently of the JSONB.
Three ways this could have been wrong, each guarded:
- Different revision or excludes from the `--stat` call would make
`files_changed` and the path list describe different diffs, with no way to
tell which lied. A source-walk test pins both to the same `base_sha` and
the same `excludes`.
- Running after `git reset --quiet` would drop newly CREATED files, since
`--intent-to-add` is what makes them visible to diff at all — and the stat
would still count them, so the list would look merely incomplete rather
than wrong. A test asserts the ordering.
- A rename is `R100\told\tnew` — three fields. Taking field two records where
the file USED to be, naming a path nobody can open, and the bug is
invisible in any repo where nothing was renamed. `changed_paths` is now
shared with auto_merge (which had the same parse) and takes the NEW path,
with tests for renames and copies.
The list is capped at 500 paths with `files_truncated` beside it: a cap that
silently clips is worse than no cap, because "touched 12 files" and "touched at
least 500" would look identical.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
f37c6b92d8 |
feat(viz): a station shows whether it is pending, working, or done
Three independent questions get three independent channels, because encoding them all as brightness makes "not started" and "finished" identical: stateAlpha presence — a pending station is faint; it has not happened yet heatFloor life — a running station stays lit between events settledColor settlement — a terminal station wears a ring (green/red/grey) heatFloor is one line in the decay (`max(floor, heat - dt*0.5)`) and it lights the whole existing treatment, since emissive, radius, glow and sparks are all already heat-driven. The ring is the only new primitive and it earns its place. The part that matters most is the staleness decay. A phase is drawn lit because `mission_phases.status` says `running` — and that column keeps saying `running` long after the agents behind it have died. Drawing that confidently lit is the exact failure this codebase keeps hitting: something that looks alive because a status field says so. After 90s with no real event landing on the station, its floor sinks to a dim ember and the HUD counts it as "quiet", so a busy station and an abandoned one cannot look the same. That is also why `applyMissionPlan` only sets a running phase's floor ONCE, on first sight. Re-applying it on every plan refresh would relight a dead station every few seconds — the poll would silently undo the decay. Rings are removed as well as added: status moves backwards when a phase re-enters `running` on a retry, and they are swept with the mesh they orbit or they leak one per phase and keep drawing at a stale position. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
006432c2dc |
feat(viz): the mission becomes a map — centre, stations, and agents at theirs
The clump was structural, not cosmetic. Four causes, each fixed here. `homes` (an agent's resting node) was written only by `seed()`, so every agent homed to the mission centre and orbited the same dot regardless of which phase it was on. `setHome` points each agent at its CURRENT phase, and the existing physics does the rest for free: the pawn rests at its station, the pawn→home line tethers it there, and a touch becomes a visible departure and return. No new motion code. The mission node was seeded as `level: "team"` (my own bug from the focus work). `seed()` casts that straight to a Tier and `ensureNode` is first-write-wins, so it was created as a small teal team dot that the later `node.activity` could never upgrade. That dot at the centre of the scene was one line. `mission`/`phase` replace the retired `repo`/`loop` tiers rather than adding a parallel set. The backend stopped emitting repo:/loop: ids, which left their whole landmark treatment — bigger radius, distinct colour, always-labelled, 60s fade instead of 22s — orphaned on prefixes nothing sends. Missions and phases need exactly that treatment. The two separately-written prefix→tier ternaries in onTouch and onNodeActivity are now one `tierFor`: they agreed only by luck, and whichever path saw a node first fixed its tier forever. Phase stations spring out at 190 rather than the shared 64, or they pack into a rosette around the centre and the point — agents moving BETWEEN stations — is invisible. Idle roam is off under a mission scope: wandering to a random node keeps an idle workspace alive, but inside one mission it sends agents to files nobody opened, which reads as work and isn't. Palette is injected at construction and keyed on template_kind, so a benchmark run and a security sweep no longer render identically to a research mission. It also collapses two uncoordinated kind→colour maps that had drifted: LEVEL_COLOR by tier, and the fireColor if-chain by id prefix. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
5f85dbb718 |
fix(world): missions were never really on the wire
Three bugs in one query, each hiding the next, plus one that made the whole rich layer dead code. `active_missions` joined `team_members` on `missions.team_id` — the LEGACY pointer at the first minted team, superseded by the `mission_teams` junction in 0056. It was an INNER JOIN, and `mission_orchestrator::on_launch` deliberately mints no team for a microVM mission, so the platform's primary execution tier was dropped by a join and the World has been showing nothing at all for it. And it selected only `status='running'`, while missions finish in minutes, so the scene was empty almost always. Now: join `mission_teams`, LEFT so teamless missions survive (their `agent_id` is NULL and no pawn beams at them, which is the truth — nothing on this platform ran that phase except a VM), and include missions finished in the last 24h carrying `status`/`template_kind` so the client can draw a finished map instead of animating a corpse. `?mission=` scopes the feed server-side. The whole phase plan now ships as `mission.phase`, including phases that have not started: a phase list that appeared only as phases began made a five-phase mission look like a one-phase mission until it was nearly over. Attribution reuses `phase_runner::purposes_for` rather than copying it — two copies would let the picture disagree with the machine about who is working on what, which presents as a rendering bug and is really a lie. Deleted the checkpoint tail. It read `topology_runs` keyed by an `agent_runs` id; mission phases live in `topology_runs` under independently generated ids, so it ran every poll and matched nothing, for every mission, forever. That is why no mission has ever shown tool or file activity. Not repointed at `topology_runs`: its only per-step content is the agent's own prose, and a tool name in prose cannot be told from an agent talking about a tool. The a2a `run_events` tail is kept — it genuinely works for the path that writes it. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
b210acf3c2 |
feat(viz): pin the World to one mission by default
Two leftovers from the sidebar swap. The header still read "N ORGS · N AGENTS", describing the org->company->team forest this tier stopped rendering; it now counts the missions and the distinct people the sidebar actually lists. And with nothing selected the World still fell back to every agent of every mission in one space. That is not a picture of anything that happens — missions do not share a stage, and past a few dozen agents the scene says less the more it shows. It now pins to the most recent mission (the workforce feed is ordered newest-first) and stays on whatever mission is pinned when an agent is selected from the graph rather than from a mission group. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
44079eb8b4 |
feat(viz): the World shows one mission, not every mission at once
The visualization page carried the org -> company -> team -> agent forest in its sidebar — the hierarchy the agents page stopped rendering — so the two pages disagreed about the shape of the workspace, and there was no way to ask the World to show a single mission. Everything ran together in one clump. Same sidebar as the agents page now: My Workforce, missions under it, agents under those. Selecting a mission scopes the scene to that mission's crew. Scoping had to happen at the FEED, not the seed. `WorldEngine.ensurePawn` materialises a pawn for any agentId an event mentions, so seeding the engine with one crew would have left every other mission's agents streaming in anyway — the view would have looked filtered for a frame and then re-clumped. `focusAgents` gates every agent-bearing event, `focusMissionId` keeps other missions' landmark orbs out, and comm beams require BOTH ends in focus or a delegation would drag an outside agent onto the stage. The engine also re-seeds when the focus changes. It was seeded once on mount, which was right when the World only ever showed everything; now a stale engine would keep the previous mission's pawns on stage, and the feed filter cannot remove what is already there. Keyed on focusMissionId rather than on `roots` identity — `roots` is rebuilt every Dashboard render, so depending on it would throw the scene away continuously. The HUD says which mission is being shown when scoped. A filtered world and an idle world look identical otherwise, and that difference is the whole question a viewer is asking. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
d3a398716b |
fix(workforce): a crew should not read as an alphabetical run
Seeding the name pick with the role index (0..n) started every crew at the top of the pool and took the next free names, so the first mission after the switch to per-mission crews hired Aarav, Abebe, Adaora, Adrian, Agnieszka. Unique and correct, and transparently generated. Seed from the claw's own uuid instead. UUIDv7 puts its random bytes LAST — the leading bytes are a timestamp, which would cluster the same way — so the tail is what spreads five picks across the whole pool. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
98037f9b3e |
feat(workforce): every mission hires its own crew
Reverses the reuse added earlier, by operator decision. Reuse hired the existing claw for a (template, slot) so the roster stayed at one team — but it also meant every mission was staffed by the same five names, and the workforce view showed one crew repeated down the page with nothing to tell the missions apart. Distinct crews read better than a bounded roster. The cost is the one reuse existed to avoid: claws are lifecycle='permanent' and nothing reaps them until their MISSION is deleted, so the roster now grows by the team size per mission. `agent_names::pick` keeps names unique workspace-wide and degrades to a numeric suffix rather than colliding, and the pool grew from 70 to 200+ given names so a workspace runs ~35 missions before the first repeat. `reusable_claw` is kept in cm-db with its tests: this policy has now flipped twice and the query is the hard part. Also revives a test that had silently stopped running. An edit stranded `runtime_data_is_scoped_to_one_mission`'s `#[test]` above its neighbour, leaving two attributes there and none here — so the neighbour ran TWICE and this one never ran at all. The total test count was unchanged by the fix (291 before and after), which is exactly why a count is not evidence: rustc had said "duplicated attribute" and "function is never used" all along, and both read as ordinary warnings. The test guards per-mission `/zeroclaw-data` isolation, i.e. one mission reading another's door token. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
c85027c83a |
fix(workforce): team_members.role, not role_slot
The roster query named tm.role_slot. That column is on agent_template_link; team_members calls it plain `role`. These queries use untyped sqlx::query(), so nothing caught it at compile time and the endpoint 500'd on its first real request — the 401 an unauthed probe returns looks identical whether the SQL is valid or not, which is why the payload had to be fetched with a real session before believing the route worked. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
0ad53da49c |
feat(workforce): missions group the roster, and agents get human names
Three things, all visible on the agents page.
**The roster looked like it was multiplying.** The sidebar flattened
orgs → companies → teams → agents, which renders a claw once per TEAM it
belongs to. Claws are reused across missions now, so a crew of five that had
run five missions appeared as twenty-five rows of the same five people. The
data was right and the view was lying. `GET /api/workforce` returns the roster
grouped by mission, and the tree renders each mission as a collapsible group,
so the repetition means something: the same colleague under each mission they
staffed. Claws on no mission come back under "Not on a mission" rather than
vanishing. The root now counts DISTINCT people, not rows.
**Agents were named after their jobs.** A team came back as planner, coder,
tester, reviewer, committer — the UI showed the same word twice (name on top,
role beneath) and the roster read as a stack of job tickets. New claws get a
given name from a deliberately wide pool (Amara, Vijay, Tomasz, Meredith…),
unique against the workspace roster AND within the team being minted. The role
is untouched in `job_title`, which is what the mission machinery binds on:
team_members.role_slot and the topology node carry the slot, so nothing
downstream keys off the display name. A reused claw keeps the name it had.
**Two latent reap bugs found while investigating a leak that was not one.**
Containers of completed missions are removed by `spawn_sweeper` after a
30-minute grace, and it works — an earlier report of leaking containers was me
reading that deliberate grace as a bug. But:
- the sweeper cleared the runtime binding even when teardown FAILED, and it
selects on `runtime_endpoint IS NOT NULL`. One transient docker error would
therefore hide a surviving container from the only thing that would retry
it, permanently. It now asks docker whether the container actually
survived: gone means clear, still there means keep the binding and retry —
which closes the orphan path without reintroducing the infinite retry the
original comment was guarding against.
- `set_runtime_binding` discarded rows_affected, so a mismatched workspace
updated nothing and returned Ok. The binding is how the sweeper finds a
container; a silent no-op there leaks one with no record of anything wrong.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
e2c312b728 |
docs(runtime): the judge no longer runs on a dead credential
`provider_alias_for` still documented the judge as deliberately sitting on `anthropic.judge`/API key, so a subscription throttle would degrade missions while verification kept working. That credential is an account with a zero balance — driving the real path returns 400 "Your credit balance is too low" — so the comment pointed the next reader at something that cannot answer. `anthropic.default` and `anthropic.judge` are retired from the runtime config and every agent that named them was repointed onto a live credential. The independence argument that put the judge there still holds; it is now served by a different FAMILY rather than a different key — CLAWMATES_VALIDATOR_MODEL is glm:glm-4.7 on gw-04, and cross_provider_judge already refuses a validator in the implementer's own family. Config-side (gw-04, not in this repo): 755 -> 246 lines, 128 -> 14 agent blocks, after sweeping 19 [agents.claw_<uuid>] corpses — every one verified against agents WHERE deleted_at IS NULL. Nothing reaped those, and the file is byte-copied into every mission. Verified: scout/judge/worker_kimi/worker_glm each answer on their new provider, and multirole passes 4/4 against the reorganized config. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
104e3ef27c |
fix(runtime): the kimi fallback hop spawned a binary that 401s
A throttled subscription had nowhere to go. `claude_cli.default` carried no
`fallback`, and neither target alias was declared — they existed only as
commented-out examples. Forcing a 429 with a shimmed `claude` surfaced four
defects that all read as correct config and do nothing:
- a `[providers.models.<f>.<a>.env]` SUB-TABLE is parsed then silently
ignored ("fields must live directly under ..."). This block was already
live for claude_cli.default, so the token injection has been inert. For
the glm alias it would have dropped the z.ai routing AND the clearing of
CLAUDE_CODE_OAUTH_TOKEN — credentials crossing between providers.
- an empty `[providers.models.kimi_cli.default]` is skipped at runtime.
- a claude_cli alias used as a fallback needs a non-empty `api_key` to pass
FamilyProviderFactory's default readiness gate, even though the provider
ignores the key and authenticates through `env`. Absent it the agent dies
at STARTUP, which takes out every mission, not just throttled ones.
- timeout_secs=600 capped every turn under the 3600s TURN_TIMEOUT from
|
||
|
Omar SobhandClaude Opus 5
|
4c418f7d9b |
fix(runtime): a turn gets the phase's budget, not 100s more than one call
A turn is an agent LOOP, not one model call. Each call inside it is bounded separately by the daemon — `claude_cli`'s `timeout_secs`, 600s on gw-04, verified in the live config — so TURN_TIMEOUT has to cover however many calls the loop makes, not one of them. It was 700s. MEASURED: a healthy research turn is ~157s. A throttled one blew the budget with one slow call plus a second, and the executor killed it at 11m43s with no error from the daemon, because nothing had failed yet. The operator got "turn executor failed: turn timed out" and the container holding the reason was torn down minutes later. An hour matches the phase's own budget. A genuinely stuck CALL is still caught at 600s by the daemon and surfaces as a real error; this only stops us killing turns that are working, slowly. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
b2e2735583 |
fix(diag): "turn timed out" now says what the agent was doing
A research_and_code mission failed with: turn executor failed: turn timed out turn executor failed: turn timed out and that is the entire record. Investigating it found: the run produced zero steps and zero output, it died at exactly 700s (TURN_TIMEOUT), the node→claw aliases were bound correctly, and the same zeroclaw team path passes in the `multirole` scenario. So the platform path is fine and the agent simply never finished a turn — but the one place the reason lived, the per-mission runtime container, is torn down after the phase and takes its log with it. By the time anyone looks, all that survives is the string. The timeout now reads the last 40 lines out of that container while it still exists, and reports which agent alias and which gateway it was driving. Best-effort by construction: it runs on a path that is ALREADY failing, so a docker error there degrades to a note rather than replacing the real failure with a second one. `container_name` derives the container from the gateway URL and returns None rather than guessing, because this feeds a diagnostic — a wrong name would put a different container's log under a failure and send the reader somewhere else entirely. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
e417247e7e |
fix(ui): clicking "My Workforce" offered to rebuild the hierarchy it replaced
Clicking the root opened the orphan-migration dialog: "You have some entities that never got parented into a real org → company → team chain. Naming the three below will materialize the chain and move everything under it in one transaction." which is an offer to reconstruct exactly the structure that root exists to replace. `SYNTHETIC_TREE_IDS` was doing three jobs at once — "not a database row, so cannot be renamed or selected for reap" AND "is a placeholder for unparented entities, so clicking it offers the migration" — and adding `my-workforce` to it inherited the second along with the first. Split by what each set is FOR. `ORPHAN_CONTAINER_IDS` are the placeholders the migration applies to and the nodes the world visualisation strips; `SYNTHETIC_TREE_IDS` is that set plus the workforce root, and still guards rename and reap. Clicking the root now just toggles the branch, which the row handler in `StructureTree` was already doing before `onSelectNode` ran. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
fe2451fd60 |
feat(workforce): missions hire the agents you already have, and name them by role
Every zeroclaw mission minted a fresh team of claws. They are created
`lifecycle = 'permanent'` and nothing reaps them until the MISSION is deleted,
so the roster grew by a whole team per mission while each member worked exactly
once — "My Workforce" was a list of strangers, and upskilling had nothing
durable to act on.
A mission now hires the claw that already does the job, matched on
`agent_template_link (template_id, role_slot)`, minting only what is missing.
Oldest first, so reuse concentrates on the same few claws and their brains
actually accumulate rather than spreading thinly across a growing pool.
A claw on a RUNNING mission is not offered. Two missions driving the same
ZeroClaw agent and the same `.brain` at once is a data race with a model on the
other end of it, and minting a second claw is much cheaper than reasoning about
that.
A reused claw is NOT re-seeded from the template's brain_seed — that would
overwrite what it learned with its starting point, which is precisely the
accumulation this exists for.
Names are the role now (`planner`), not
`"{mission} · {purpose} · {template} · {slot}"`. That produced
"verify: a repo-less research mission keeps its output · mission · Rust SDLC ·
planner" — unreadable in the roster, the API and every log line at once. Which
mission a claw is on is context a caller can join to; it is not its name.
And the half that makes reuse safe rather than destructive: deleting a mission
now purges only claws no OTHER mission still employs. Without it, tidying up one
mission deletes staff another one holds — presenting as the roster quietly
shrinking rather than as an error. A test asserts the guard exists inside the
reaper AND runs before the purge, because a check after it is decoration.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
895413509d |
feat(ui): My Workforce — one flat list of agents, and the "+" starts a mission
The sidebar showed Organization -> Company -> Team -> Agent. On this workspace that read "My Workspace -> General -> Everyone": three levels of placeholder wrapping five agents, with five orgs and three companies named "My Workspace", "General" and "Workplace" between them. None of it was load-bearing in the UI. `agents` has no org/company/team column at all — membership is only the `team_members` join, which the mission executor uses to map graph nodes to claws — and /orgs, /companies and /teams already redirect to the dashboard. The tree survived in exactly one place. So the tree is now a single "My Workforce" root with the agents directly under it, expanded by default: a workforce collapsed behind a disclosure is one the user has to discover they own. The World tier keeps the full forest, because that visualisation is ABOUT structure and flattening it would remove its subject. Nothing is deleted — the group pages and their APIs are untouched. Both "+" affordances now open the MISSION wizard. They opened the deploy wizard, while the copy beside them said "deploy wizard" and the tooltip said "Deploy a new agent" — none of which is what someone arriving at an empty workspace wants to do first. You get a workforce BY running missions. Hand-staffing one is a real thing to want, just not the first thing, so it is demoted to "or create an agent yourself" rather than removed. "Add a new agent, team, company, or organization" becomes "Create your agent workforce". Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
dd80b69992 |
fix(microvm): the third place that assumed a VM phase has a repository
The research-vm scenario failed on its first run, and said exactly why:
mission has no checkout at /var/lib/clawmates-missions/<id>/repo
— a microvm phase needs a repository
`phase_runner` refuses upstream of both places the last commit fixed. Three
guards, written independently, all encoding "a microVM phase implies a git
checkout" — which is why the capture filter could cite it as settled fact.
A repo-BACKED mission with no checkout is still a real fault and still refused;
booting a VM to hand the agent an empty directory would turn a setup failure
into a confusing agent report. A repo-LESS one now gets the empty workspace
made here, so the executor's inject has something to pack.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
768e106614 |
fix(microvm): a mission with no repository can run in a VM, and its work comes back
Two halves, and the first was worse than the plan assumed. `run_phase_in_vm`
packed `<missions_root>/<mission>/repo` unconditionally — a directory a
repo-less mission does not have — and then required `/mission/repo/.git` inside
the guest before spending a turn. So a repo-less microVM phase did not merely
go uncaptured: it failed before the agent ran.
A repo-less mission now gets an EMPTY workspace at the same guest path, created
host-side so the collect unpacks back over it with no special case, and the
readiness probe asks for what was actually sent — the directory rather than a
`.git` that was never going to be there.
`mission_outputs` then drops its `runtime_kind <> 'microvm'` exclusion, whose
stated reason ("a microVM mission always has a checkout") is exactly what
stopped being true. Where the files come from now depends on the runtime, and
the difference is not cosmetic: a container mission's output is still inside a
running container, while a VM's has already been unpacked onto the host by the
end-of-turn collect. Asking docker for a VM mission's files would query a
container that never existed.
The recursive copy skips symlinks rather than following them — a link out of
the tree would publish whatever it points at.
`research-vm` is the proof, added to the suite as well as the dispatch: the same
assertions as `research-only` with `runtime_kind: microvm`. A scenario nobody
runs is a scenario that does not exist.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
e4bddeb1ba |
feat(ui): three-screen mission wizard that asks only what the type needs
Five fixed steps for every mission type, and getting a research document out of it meant naming a team, choosing a runtime, and writing per-phase completion conditions under a paragraph explaining what a model checker can and cannot prove. Two of those steps asked for things the mission does not use, and one of them blocked outright. 1 What do you want to do? 2 Title, a description with a Polish button, repo ONLY if the type needs one 3 Review -> Launch, plus one collapsed Advanced section Two hard defects fixed on the way: - The microVM runtime could not be selected AT ALL. Step 4 gated Next on `targetNodeId`, which microVM deliberately never sets because placement picks the node per phase. Everything shipped today, the local-GPU backend included, was unreachable from the UI. - Step 3 required a team while every workflow TOML already names one in `default_team_template` — which this file ignored. The answer was always available and the question was always asked. It is now resolved by key, with a category fallback, and shown under Advanced so an operator can see WHICH default rather than having to supply one. A failed `/api/team-templates` request and a genuinely empty list rendered the identical red banner, which sends the reader looking for missing template files when the request had 401'd. They now say different things. `phases[]` is no longer sent unless someone set a completion condition. `recipeToPreset` strips each phase's `config`, so posting the stripped list overrode the recipe's real settings — tools, commit policy, loop mode — with nothing. Omitting it lets `phases_for_create` use the recipe, which is both simpler and more correct. Launch keeps its own gate, since Advanced can still produce an unlaunchable combination — but it names what is missing instead of greying out in silence. Artifacts get a Download link. Deliberately a plain link to the streaming route rather than a Blob built from what "Read" already fetched: that content is capped at 2 MiB and UTF-8-decoded, so reusing it would silently produce a truncated or undownloadable file for exactly the artifacts worth downloading. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
25f075a8be |
feat(api): polish a description before the mission exists, and download an artifact
Two endpoints the wizard redesign needs.
`POST /api/missions/refine-draft` — the polish button fires while the user is
still typing, before anything is created, so it has no id to route on.
`refine` deliberately requires a saved draft because its Accept writes back;
this one has nothing to write back to and returns the text. Same system prompt,
same model chain. The phase list comes from the workflow recipe rather than the
caller, for the same reason `phases_for_create` prefers it: a client that
guessed would have the model write acceptance criteria for phases the mission
will not run.
`GET /api/missions/{id}/artifacts/{artifact_id}/download` — the file itself.
`artifact_content` caps at 2 MiB and reads as UTF-8, so a large or binary
artifact is unreachable by any means today; this streams the bytes with a
filename attached and no ceiling.
Both artifact routes now resolve through ONE containment check. Two copies of
"is this path under _outputs" is two chances for one of them to be the lenient
one, and the lenient one is an arbitrary read of the gateway's filesystem — so a
test asserts there is a single resolver and that both routes call it.
The download filename was chosen by an AGENT and lands in a header every browser
parses, so quotes, backslashes and control characters are stripped rather than
escaped; the test covers a header-injection attempt.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
f27d2605eb |
fix(agents): a soft-deleted agent could never be purged
Clearing the fleet's four leftover agents returned 404 on every one. They had
been soft-deleted back in June — correctly invisible in the UI ever since — and
`agents::get` filters `deleted_at IS NULL`, so `workspace_agent` could not find
them. Every route uses it, including `batch-delete`, the one that exists to
HARD-purge. So a soft-deleted agent was unreachable from the application
entirely and its row stayed forever.
`get_any` sees them, and only the purge path uses it: hiding soft-deleted rows
is right for every read, and wrong for the one operation whose whole job is
removing them. Written with `query_as` rather than the checked macro so it does
not force an offline-cache regeneration on every machine that builds this.
`fleet-reset.sh` now uses `batch-delete` for agents rather than
`DELETE /api/claws/{id}`. The latter is a SOFT delete, so pointing a reset
script at it would have quietly added to the pile it was meant to clear.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
16cfc29074 |
fix(ui): the backend picker showed two options meaning the same thing
`default` is the generic `rootfs.ext4` and `claude` is the named one, and `microvm_credential_for` gives them the identical contract — so the list came back with both under the same label, and whichever a user picked they got the same thing. Collapsed to the named one where it exists; the generic keeps a label of its own for a fleet that only has that. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
529497febb |
fix(placement): a composed graph needs every backend its nodes name
The full harness found it — 12 of 13 scenarios green, `roster` red:
roster: the planner sized this mission at 2 member(s) PASS
roster: the approved roster is on the mission (2 nodes, composed) PASS
roster: this run added 1 line(s) for a 2-member roster FAIL
topology_runs.error: turn executor failed: node n1 in a microVM:
vm_create failed: no rootfs for backend "canary-claude" on this node
The roster proposed `verifier@canary-claude`. Placement asked
`online_for_backend` about the MISSION's backend — `claude` — and architect
answered, holding `claude` and `local-ornith`. The graph's first node ran and
delivered, the second could not boot, and the mission finished half-done. The
question placement asked was true and insufficient.
A composed graph runs on ONE node, so that node needs every image its nodes ask
for. `required_backends` collects the mission's plus each
`config.roster.nodes[].attrs.backend`, and `online_for_backends` passes the
whole set to the same jsonb `@>` — containment already means "contains ALL of
these", so the query shape did not have to change, only what it was asked.
This is the failure mode the roster feature creates by existing: its entire
purpose is putting a verifier on a different provider, which is exactly what
makes one node insufficient. Nothing before the full suite had a reason to
exercise it — the composed scenario uses one backend for all five nodes.
`NoCapableNode` now names the set and says why one node must hold all of them.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
171f901bcd |
feat(ops): fleet-reset — delete every mission and PROVE the disk came back
For a clean slate before a UI session, and for the thing that keeps being true here: deleting a row has never deleted a directory. A full harness run leaves ~35 missions, each with a repo checkout and a runtime-data tree, on the smallest disk in the fleet. There are 125 rows and 117 directories right now. Deletes through the API, never with SQL. `missions::delete` tears down the per-mission runtime container, hard-purges the FK graph in order, and removes the workspace directory — falling back to a root purge for the files the per-mission daemon leaves as root. A `DELETE FROM missions` skips all three and orphans every one of them, which is how the orphans got there. Then it checks, because rows gone is not bytes back and every incarnation of this cleanup has managed the first while silently failing the second: it names each directory left without a row, and counts root-owned residue separately because that is the specific way it fails. Refuses outright while any mission is RUNNING. Yanking a live mission's checkout leaves a VM writing into a directory that no longer exists, and the symptom is a phase that hangs rather than one that fails. Verified: it stopped exactly there against the in-flight harness. Dry by default; `--yes` to act; `KEEP=<substring>` to spare some. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
e7b412d578 |
test(harness): local-ornith was missing from the all suite
Added to the case dispatch when it was written, and not to `all` — so the newest backend, and the only one that runs on hardware we own, was excluded from the one run that claims to check everything. A scenario nobody runs is a scenario that does not exist. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
c66c3c6377 |
feat(ui): the microVM path is reachable from the mission wizard
Everything built today — Firecracker missions, the four backends, the local GPU one — was unreachable from the dashboard. The wizard offered `zeroclaw` and `local_herdr` and nothing else, so a mission created in the UI could not be a microVM mission at all, and `local-ornith`/`glm`/`kimi` were API-only. Testing "our workflows in the UI" would have exercised none of it. Adds the runtime option and a backend picker, fed by a new `GET /api/fleet/backends` that returns `mission_roster::available_backends` verbatim — the SAME list the roster planner is handed, not a second one. Its two rules are both load-bearing and neither is visible from a node's capabilities alone: the image must be built on an online node, and the backend must have a credential contract. `agent-terminal` passes the first and fails the second — bootable, with nothing for the agent inside to authenticate with — so offering it would produce a mission that validates, launches, and dies at the agent turn. Ids are deployment vocabulary, so the picker labels them: a user choosing between `local-ornith` and `canary-claude` should not have to know which company each one bills. An empty list says why (no rootfs built) instead of showing an empty dropdown, and no node is chosen for a microVM mission because `vm_placement` picks it per phase. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
1f6108f769 |
feat(gc): reclaim the mission tree on the gateway
`cleanup_sweeper` prunes ROWS. Deleting a row has never deleted a directory, and `teardown_container` only runs while a mission still exists to tear down — so a mission removed by any path that skipped teardown left its tree behind permanently, on the smallest disk in the fleet (150 GB, shared with postgres and every checkout). 106 mission directories are sitting there now. Filesystem-first, deliberately: the DB is the PREDICATE, never the enumerator. Enumerating from the database is exactly how these became invisible — a directory whose row is gone is the one a row-driven sweep cannot see. Three reapers, one deletion path. Orphan mission dirs (no row, past a 2h grace), scratch trees (_bench/_gate/_verify/_merge past 6h — all four have leaked before), and _outputs past 90d, whose artifact rows are marked only AFTER the files are gone, because the other order claims artifacts are reaped while they are still on disk. The single removal path escalates: the server is uid 65532 and cannot delete what the per-mission daemon leaves as root, so PermissionDenied falls back to `root_copy::purge` and shouts if the tree survives even that. A GC that cannot collect is the thing being fixed, so failures are counted and reported, never swallowed. Guards worth naming: `_cargo` is a SHARED cache every mission writes to and lives under the same root, so an underscore-prefixed sibling treated as an orphan mission would delete it out from under running work and look like a slow cargo build. Only a well-formed mission id is ever a candidate — a directory whose name is not an id can have no row by construction, so without that gate every unrecognised directory looks orphaned. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
3c3d01c8d1 |
fix(llm): the chain preflight printed nothing at all
Deployed, and the report simply did not appear — from the tool built to stop
things failing silently. Two causes, both worth keeping:
There was no timeout anywhere in the probe, so one slow provider swallowed the
entire report. Each link is now bounded at 60s (generous: `complete_or` spends
up to 30s in its own backoff, so a tighter cap would report a merely throttled
link as hung) with `TimedOut` as its own state, and every line is emitted AS IT
RESOLVES rather than collected and printed at the end — a later link that hangs
must not be able to hide the ones already checked.
The first attempt at the timeout awaited the probe and then wrapped the result:
let probe = complete_or(...).await;
timeout(PROBE_TIMEOUT, async { probe }).await
That compiles, reads correctly, and bounds nothing. The timeout has to wrap the
future.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
c7c3eeab46 |
fix(test): the colon-vs-spec test did not compile
Committed and deployed while its test compile was failing: the verify step was `cargo test | grep -E "^error|test result" && git commit`, and grep exits 0 when it MATCHES, so finding the error is what let the commit proceed. The library built fine, so the deploy was sound, but the check that was supposed to gate it did the opposite of gating. The error itself was a borrow in a test closure; a plain fn fixes it. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
d9c5300859 |
fix(llm): the preflight found two broken links on its first live run, one its own
fallback chain (6 link(s), 4 usable):
claude-opus-4-8 ok
claude-sonnet-4-6 ok
claude-haiku-4-5-20251001 ok
kimi:kimi-k2.7-code BROKEN: 400 ... role 'system' must not be empty
glm:glm-4.7 ok
local:ornith-fleet:9b UNREGISTERED — resolves to the DEFAULT provider
Neither link was actually broken.
The probe sent an EMPTY system prompt, which Kimi rejects outright. A probe has
to look like the traffic it stands in for, or it measures itself.
The second is the one worth keeping. `resolve_provider` returns a spec unchanged
when it does not recognise the provider, and the part after the FIRST colon when
it does — so the obvious test, "does the model half still contain a colon",
reads correctly and is wrong the moment a model id has one. `ornith-fleet:9b`
has one. The probe reported a provider the server had just finished registering
as UNREGISTERED.
`evaluator::cross_provider_judge` had the identical check, and would therefore
have refused a local judge as "not independent" — silently falling back to a
same-family one, which is the exact claim that path exists to make honestly.
Both now compare against the whole spec.
That bug was written into the codebase before a model name with a colon existed,
was correct at the time, and became wrong when one arrived. Nothing would have
reported it; a boot-time probe of every link did, on its first run.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
c3ad5672fc |
feat(llm): six-link fallback chain, and a preflight that proves it
opus -> sonnet -> haiku -> kimi -> glm -> local. The order is capability first, then independence: three Anthropic tiers on one account (a throttle usually hits a tier, so stepping down often clears it), then two separately funded accounts (now an outage, not just a throttle, is survivable), then our own GPU (nothing left to be down). Every id was probed on this deployment and answered 200. The preflight is the more important half. Configured is not working, and this chain has a specific way of lying: `resolve_provider` falls back to the DEFAULT provider when it does not recognise a provider name, so a typo in `kimi:` does not error — it quietly runs on Anthropic, and a chain that reads as three accounts is really one. A reachability-only probe calls that link green. So `preflight` checks resolution and reachability separately, eight tokens per link through the REAL call path, and reports four states. `Throttled` is deliberately not a failure: a 429 means the spec resolved, the credential authenticated, and there was no capacity this second — the exact condition the chain exists to route around, and painting it red would train an operator to ignore red. `Unregistered` and `Broken` are failures, and they get different words because they need different fixes. It runs at boot alongside validator_preflight and runtime_preflight, spawned so it cannot delay startup. A chain is the one piece of infrastructure nobody looks at until the day it has to work, so it is now checked on the days it does not. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
5afcf63324 |
fix(harness): count what the roster run ADDED, not what the file holds
Forcing the planner onto the local link produced a green chain and a red assertion: roster: the planner sized this mission at 1 member(s) PASS roster: ROSTER.md has 3 line(s) for a 1-member roster FAIL The model was right and the check was wrong. ROSTER.md does not start empty — the auto-merge work put an earlier run's two lines onto main — so a 1-member roster that correctly appended one line delivered three, and the scenario reported a model that had ignored its own proposal. It now measures the DELTA against main. Any assertion against a scratch repo that accumulates has to, or it decays into a test of how many times it has been run before. Proven on the local model end to end: opus 429 -> local:ornith-fleet:9b answered -> `mission_roster: ... local:ornith-fleet:9b proposed 1 member(s)` -> the composed graph ran -> the branch added exactly one line. 5/5. CLAWMATES_MODEL_FALLBACK is removed from gw-04's .env again; it was set only to force the last link for this test, and the deployed default is the full chain. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
b18e62041b |
feat(llm): the fallback chain's last link runs on our own hardware
`local:ornith-fleet:9b` joins opus -> haiku -> glm as the final link. Every entry above it depends on somebody else's account staying funded and unthrottled; this one depends on a GPU in the next room. It is last because it is the weakest model, and present because a chain whose every link is external is not a fallback chain, it is one outage in a trench coat. Three small changes make it work: - `build_provider_registry` accepts a provider with an empty `api_key_env`. A model on our own hardware has nothing to authenticate to, and the old behaviour SKIPPED a keyless provider — leaving the chain quietly one link shorter than it reads, which is the failure mode this whole area keeps producing. - `provider_family` learns `ornith`/`ollama` for BARE names. A qualified `local:` spec was already answered by the split, but a bare one fell through to "unknown", and `cross_provider_judge` would then refuse a judge that is genuinely a different family from the Anthropic implementer. - A test pins that the last link survives `resolve_provider`'s split-on-FIRST- colon: `local:ornith-fleet:9b` is provider `local`, model `ornith-fleet:9b`. Splitting on the last colon would ask for a provider named `local:ornith-fleet`, and the symptom would be a silent fall back to the default provider. Infra: Ollama on tank and architect now binds 0.0.0.0 so the gateway (which has no GPU) can reach it. `tailscale serve` cannot — Ollama rejects a non-local Host header as a DNS-rebinding guard and OLLAMA_ORIGINS is CORS-only, so it 403s. 0.0.0.0 still includes loopback, so the microVM vsock pipe is unaffected; verified on both nodes. This is an explicit trade: Ollama has no auth and its API can pull and delete models, so it is now reachable from the LAN as well as the tailnet. The drop-in carries the ufw one-liner to close the LAN side. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
774f17d194 |
test(fleet): a mission served entirely by the node's own GPU
`local-ornith` scenario, green on its first real run against tank: local-ornith: a locally-served model delivered a guest kernel (6.1.128) local-ornith: no Anthropic egress from a locally-served mission local-ornith: the node bound its local-model socket for this VM local-ornith: checkout has exactly one writer (uid=65532) Three things had to be true at once and only a real run shows all three: the agent reached a model at all (a pipe to a closed port produces a turn that HANGS rather than errors, which is why this is a scenario and not a unit test), the work came back and landed on a branch, and the VM still could not reach api.anthropic.com. That last one is not theoretical. The node log for this VM is a column of `egress DENIED api.anthropic.com` — Claude Code's own telemetry, correctly refused — while the model traffic went through the vsock pipe and Ollama logged loading ornith-fleet:9b at 100% GPU with CONTEXT 131072. A local backend that quietly kept Anthropic egress would be a credential path nobody asked for. The egress check asks the NODE's proxy log rather than the agent, for the same reason the GLM measurement did: a model's account of where its tokens came from has no evidential value, and the proxy's record of what it dialled does. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
f56d41f5b7 |
feat(backend): local-ornith — a mission backend served by the node's own GPU
Claude Code pointed at the Ollama already installed on every GPU node. Ollama has served a native Anthropic-compatible /v1/messages since v0.14, so this is an env contract rather than a translation layer — the fourth variation on the same idea as agent-glm and agent-kimi. The route is NOT the egress proxy, and that is the design. `egress` speaks CONNECT, takes a destination from the guest, resolves it and decides; every one of those powers is a liability, which is why it refuses non-443 ports and IP literals after a unit test caught them being bypassed. Routing a local model through it would have meant relaxing both. `local_model` is the opposite shape: there is no destination in the protocol. fcagent listens on guest 127.0.0.1:11434 and pumps to vsock 9003; the node splices that onto its own 127.0.0.1:11434 and copies bytes. A compromised guest cannot redirect it because there is nothing to redirect — it is a pipe, not a proxy, and strictly narrower than anything an allow-list could express. The bytes never touch a network, so there is no wire for TLS to protect, and Ollama stays bound to loopback rather than being exposed on the tailnet. The socket is bound only for a backend declared to use a local model, so a `local-ornith` VM reaches the forge through egress and nothing else, while every other backend's guest port simply refuses. Both halves have negative controls. `scripts/fleet-model-setup.sh` exists because of one measurement: stock ornith:9b reported input_tokens=2050 for a 48000-word prompt and answered as though nothing had been dropped. Ollama's default window is ~2K whatever the model card says, and it truncates silently — the exact failure an agent turn would hit and never report. The script pins num_ctx=131072 into a derived tag and then PROVES both the window and tool calling before declaring success. Verified on architect: ~65536 words -> 65604 input tokens, stop_reason=tool_use. Placement needs no new capability key: building the rootfs only on GPU nodes means `nodes::online_for_backend`'s existing `rootfs @> ["local-ornith"]` predicate does the affinity, so morpheus never offers the backend. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
e96c5143bc |
test(eval): a local judge, and the 2K context window that would have hidden it
Phase 1 of the local-model plan: prove the model before writing any plumbing. `JUDGE=local` runs the existing done_when eval against Ollama on a GPU node. Requests originate on that node rather than the gateway, because the model is bound to 127.0.0.1 deliberately — it has no network exposure at all — and the gateway has no GPU. MEASURED on tank, 3 draws per case, against the incumbent on the same cases: local (ornith-fleet:9b) 14/15 — one UNPARSED, never a wrong verdict glm (glm-4.7) 13/15 — two WRONG verdicts on kernel-ok kernel-ok is the case production actually hit and the one this script's header says is expected to fail on glm-4.7. A 5.6 GB model on hardware we already own did not get it wrong once in three draws. The tag is `ornith-fleet:9b`, not `ornith:9b`, and that is the finding worth keeping. Ollama defaults to a ~2K window whatever the model claims: stock ornith:9b reported input_tokens=2050 for a 48000-word prompt and answered as though nothing had been dropped — silent truncation, confidently. The fleet tag pins num_ctx=131072, which measures 9.3 GB resident of a 16 GB card (the full 262144 also fits, at 13.6 GB, 100% GPU). These eval cases are a few hundred tokens, so this eval would have passed either way; that is exactly why the tag under test has to be the one production would use. Also measured: Anthropic /v1/messages returns well-formed tool_use with stop_reason=tool_use on both nodes; the reported count_tokens?beta=true hang is absent in 0.31.1 (clean 404, server unaffected); ~60 tok/s generate, ~2800 tok/s prefill, 120072-token prompts accepted end to end. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
f68fc019e4 |
fix(teardown): a mission dir with root-owned files is now actually removed
The server runs as uid 65532, so `remove_dir_all` on a mission directory returns PermissionDenied the moment anything root-owned is left in it — and the old code logged that at the same level as "file not found" and moved on. The directory then lived forever. After the seed-copy fix a mission holds 3281 files owned by 65532 and 26 owned by root: `.claude.json` and the session jsonl the per-mission ZeroClaw daemon writes itself, after the copy has been chowned. Twenty-six files is small enough to keep every mission directory alive without anyone noticing why. PermissionDenied now falls back to `root_copy::purge`, which deletes from inside the container as root — the same escape hatch `container_exec` keeps for exactly this, clearing debris a root process created. And if the directory survives even that, it says so, because a cleanup that silently failed is the thing being fixed. Removing the last 26 properly means running the per-mission daemon as 65532, which needs `/mission` pre-created in the image with that ownership — the daemon creates it at boot today and cannot at a lower uid. That is an image change, deliberately not bundled here. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
4967b9b8fd |
fix(runtime): the seed copy reads as root and hands the result to 65532
Running the seed copier as 65532 broke mission launch, and broke it quietly. The seed dir is root-owned with parts at mode 0600 (`.claude.json`, `clawmates-mcp.json`), so uid 65532 cannot READ them: `cp` failed on the first unreadable entry, `set -e` abandoned the rest, and the mission came up with a runtime-data holding `.zeroclaw` and nothing else — no Claude credentials, no door config. The daemon then never created its agents' workspace, and the phase failed 200 lines later on "Could not find the file /mission in container", which points nowhere near the cause. It was quiet because `seed_runtime_data` polled for the container to STOP and returned Ok without ever reading its exit code. A copier that died on a permission error and one that finished cleanly were indistinguishable. It now reads the status and says what went wrong. So: root for the read, `chown -R 65532:65532 /dst` for the result. Both halves matter and they pull opposite ways — root is needed to read the seed, and 65532 is needed because everything else in the missions tree is 65532 and a GC running as 65532 cannot delete what root left behind. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
8ddea454d1 |
fix(runtime): the seed copy ran as root too, ~3200 files per mission
The uid fix landed and the CHECKOUT came back completely clean — 0 non-65532 files under `repo/` after a benchmark run that builds and tests Rust. But the same mission still held 3247 root-owned files, all under `runtime-data/`. `seed_runtime_data` spawns a throwaway container to `cp -a` the runtime seed into the mission's directory and never set `user`, so it ran as root — the identical absent-`user` omission `container_exec` had, in a container create instead of an exec. The seed source is 65532-owned and the destination is created by the server (which itself runs as 65532), so the copy never had a reason to out-rank either. This is the tree a gateway GC has to be able to delete, and a GC running as 65532 cannot remove root-owned files — the cleanup-that-cannot-clean-up shape, found before writing the GC rather than after. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
dcd9514622 |
fix(exec): mission work runs as uid 65532, so it stops creating debris it cannot delete
`CreateExecOptions` never set `user`. Not a wrong value — an ABSENT one: the daemon defaults to root, and twelve callers inherited that without any of them choosing it. That single omission is the origin of four separate patches — root-owned `target/` directories inside a checkout owned by 65532, `root_copy` existing at all, and a cleanup that had to re-enter the container as root to undo its own mess. The rule is positional and lives in ONE place: an exec whose workdir is inside `missions_root()` runs as 65532; anything else (preflight probes, image checks) keeps the daemon default so unrelated call sites cannot break. Twelve callers each remembering to pass a uid is twelve chances to forget, and the one that forgets leaves debris the other eleven cannot remove. Non-root needs an environment the image does not provide. Measured in the deployed image: uid 65532's HOME (/zeroclaw-data) and /usr/local/cargo are both root-owned and unwritable, so this would otherwise break every cargo call — the benchmark runner, the judge's sandbox, the delivery test gate — far more quietly than the leak it fixes. The missions root IS bind-mounted and writable by 65532, so HOME/CARGO_HOME move there and the cargo cache is shared across missions rather than re-fetched per mission. Verified on gw-04: a clean `cargo build` as 65532 with those three variables produces output owned entirely by 65532. Root remains reachable only through `exec_as_root`, whose name says so, and which exists solely to clear debris earlier root execs left. `runtime_preflight` now probes the whole policy at boot, so an image that moves or tightens that mount fails loudly instead of failing every cargo call for a reason no error message would connect to a uid. evaluator_tools' inlined fourth copy of the purge is replaced by `root_copy::purge`. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
42108c840d |
docs(placement): the drain half of that fix was never the broken half
drain-midmission passed 3/3 twice, but the "re-placing this phase" line the last commit added never appeared in the log. It cannot: `online_for_backend` filters on `status = 'online'`, so a draining node is not a candidate, never reaches `unfit`, and the pin simply falls through to ranking — on the old code as well as the new. So the scenario passes either way and proves the affinity decision, not the `TargetUnfit` bug. The path that genuinely used to fail a phase is "the previous phase's node has since FILLED UP": that puts it in `unfit`, which returned a non-transient error, which never reached the queue. That is what the unit test now says, in place of a claim about draining the harness does not support. The accidental mission-to-node affinity was real and unconditional either way. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
13a35138e9 |
fix(placement): a drained previous node re-places the phase instead of failing it
`drain-midmission` found this. `choose` treated its `want` argument as a hard
requirement, and the only caller passes `missions.target_node_id` — which is
not an operator's choice, only where the PREVIOUS phase happened to run. Two
consequences, both wrong:
- A node drained or filled between phases produced `TargetUnfit`, which
`is_transient()` says false to, so `phase_runner` FAILED the phase rather
than queueing or moving it. The queue silently did not apply to the second
phase of any mission.
- While the node stayed fit, every later phase went straight back to it
regardless of ranking — accidental mission-to-node affinity, which this
module's own header says must not exist.
Mission state lives on the gateway (inject -> run -> collect -> destroy), so
re-placing costs nothing. The pin is now advisory: preferred while it fits,
and when it does not, the reason is logged and ranking proceeds. `TargetUnfit`
is deleted rather than left unconstructed, so it cannot come back as a
non-transient failure by accident.
The scenario had its own race: it waited for phase 0 to COMPLETE before
draining, but warm phases finish in ~80s against a 10s placement sweep, so
phase 1 was often already placed — and the run then blamed the platform for
running on a node that was not yet drained. It now drains while phase 0 is
still running, which does not disturb a live VM and is the more faithful test.
Co-Authored-By: Claude Opus 5 <[email protected]>
|