2f1a870949fe9fc964eea8ac37b53397407309ba
553
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar SobhandClaude Opus 5
|
f87853ecf9 |
fix(missions): scheduled missions never fired — nothing read missions.schedule
The wizard has collected a cron since `0047_missions.sql` ("schedule JSONB
carries the trigger config (cron | one_shot | on_event)"), the frontend posts
`{kind:"cron", cron}`, and the API persists it faithfully. Nothing has ever read
it back: the only due-work enumerator in the codebase was `routines::claim_due`.
So every scheduled mission ever created sat in `draft` forever while the UI
reported it was on a schedule.
Proven before fixing, on the shipped build: a mission with `* * * * *` sat in
`draft` for 4m34s and started ZERO topology runs. After this change the same
mission launched on its next occurrence and recorded one `fired` row.
Two pieces were missing, and they are the two `routines` already had:
- `missions.next_run_at` — schedule STATE. `schedule` is user intent and stays
untouched; without somewhere to record which occurrence is owed there is
nothing to put a `<= now()` predicate on, which is why no enumerator could
be written against the JSONB alone.
- `mission_fires` — one row per (mission, occurrence). 0063_routine_fires.sql
called this exact case: "For a scheduled *mission* it costs a container, a
repo checkout, and real money — which is why this lands before mission
scheduling does."
`mission_schedule.rs` deliberately mirrors `cm-scheduler`'s shape rather than
inventing a second one: atomic `FOR UPDATE SKIP LOCKED` claim, reschedule
BEFORE dispatch so a failing launch cannot stall the clock, claim the slot
before launching so a crash mid-launch is retried rather than dropped, and a
fan-out cap. The cap is 5, not the scheduler's 25, because a mission firing is
a container and a checkout where a routine firing may be one turn.
The claim skips `status = 'running'`: a daily cron on a mission that takes
longer than a day must skip the occurrence, not stack a second crew on the same
workspace. Launch goes through `mission_orchestrator::on_launch` +
`missions::set_status`, the same path as the draft→running transition, so one
code path mints a crew. An unattended launch acts as the workspace owner
(`users::owner_of_workspace`) since missions carry no creator column; a
workspace without one settles the occurrence `failed` with the reason rather
than dropping it silently.
Backfill blast radius was MEASURED, not assumed: prod has zero missions with a
cron, this workstation had exactly one — the control created to prove the bug.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
3cc65c22c4 |
feat(judge): room to analyse — and a panic in the evidence path
Three changes, one of them a live bug. **The bug.** `phase_summarizer` truncated agent output with `&s[..remaining]`, a BYTE slice of arbitrary UTF-8. Agent turn output routinely carries arrows, box-drawing and emoji, so a cut landing mid-character panics — taking down the evaluation sweep for that phase, triggered by nothing more than an agent writing a long enough line with a non-ASCII character at the wrong offset. Replaced with `clamp_to_char_boundary`, tested across every cut offset of a pure-4-byte string. It is precisely the bug the clawhdf5 agents found and fixed in `clawhdf5-migrate/src/validate.rs` this week — in our own code, in the path that feeds the judge. **Evidence budget** 60 KB -> 120 KB. Output headroom is worthless if the judge cannot see the work: the verdict is only as good as what reaches it. **Judge max_tokens** 2048 -> 16384. glm-5.3 is a reasoning model that spends most of its budget on a `thinking` block before writing the verdict, and running out mid-thought truncates it. A truncated verdict parses as empty and FAILS CLOSED, burning one of the phase's passes on a judge that never answered — how mission 01a00bbb lost one. Measured ceiling: z.ai accepts max_tokens up to 131072 on both glm-5.1 and glm-5.3 (131073 -> 400, "限制数值范围[1,131072]"), so 16384 is chosen for cost and latency rather than capability, and only emitted tokens are billed. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
9c4b0722e8 |
feat(judge): an independent GLM judge, on the newest model z.ai publishes
Every phase verdict this session was Anthropic grading Anthropic, and the boot
log said so on each start:
validator_preflight: no CLAWMATES_VALIDATOR_MODEL — phase verdicts are
judged by the house model, which is NOT an independent check
`evaluator.rs` already preferred a cross-provider judge and refused to call a
same-family one `independent`; the local stack simply had no non-Anthropic
credential. It now carries the same `glm` provider gw-04 has had all along —
`format = "anthropic"` is load-bearing, since z.ai's OpenAI-compatible endpoint
is ToS-throttled for raw SDK access while its Anthropic-format one is not.
Model: glm-5.3, the newest z.ai lists (4.5, 4.5-air, 4.6, 4.7, 5, 5-turbo,
5.1, 5.2, 5.3 as of 2026-08-17). gw-04 still runs glm-4.7.
glm-5.3 is a REASONING model: it emits a `thinking` block before its JSON. Our
SSE parser ignores `thinking_delta` and keeps the text, so the wire shape is
compatible — but on a realistic phase-evidence prompt it spent 819 of the
evaluator's 1024 output tokens. A longer phase would truncate the verdict, and
a truncated verdict parses as empty and FAILS CLOSED, burning one of the
phase's passes on a judge that never answered — precisely how mission 01a00bbb
lost a pass. max_tokens raised to 2048.
Measured before wiring: asked to judge 25 commits claiming INT-01..INT-25 with
tests passing, glm-5.3 returned met=false because the evidence never
established what the brief actually required. That skepticism is the point of
an independent judge.
Boot now reports: `validator_preflight: independent validator glm:glm-5.3
answered`.
The key lives in .env (gitignored), never in this file.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
b5032a732a |
fix(phase_runner): collect the agent's work BEFORE judging it
`Sandbox::for_mission` builds the judge's verification copy from the HOST
checkout. In copy mode the agents write inside the container, and their work
only reached the host when `sync_out` ran — in the capture sweep, AFTER the
phase closed. So every phase was judged against a tree that did not yet contain
the pass being judged, and the judge truthfully reported nothing there.
Mission 01a00cfa is the proof. Research pass 2 wrote a 434-line
IMPLEMENTATION_BRIEF.md, `cargo test` passed, and it was pushed to a clean
branch (clawmates/mission-01a00cfa-c69f39fd-i2 at 563cdd21). Its verdict:
failed after 2 pass(es) — met=false — research/IMPLEMENTATION_BRIEF.md
does not exist anywhere
logged one line BEFORE `captured (+434/-0 across 1 file(s))`. A phase that
succeeded was failed because the evidence had not been collected yet.
This hid because it only bites a phase judged on its OWN pass. The v2 coding
verdict cited real commits (339a5bd, 167671f) — research had already synced
that work to the host in an earlier phase.
`evaluate_finished_phases` now runs `sync_out` first, and on failure leaves the
phase `evaluating` for the next sweep rather than recording a verdict nobody
could stand behind — the same policy the capture sweep already applies, for the
same reason. microVM keeps its carve-out: `microvm_executor` collects out of
the guest over this same path before the VM is destroyed.
Research goes to 3 passes. On 01a00cfa it got no real attempts out of two: one
spent on a fabricated commit claim the judge correctly rejected, one on this
bug.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
d341640255 |
fix(mission_fs): drop build output when collecting work back from a container
`pack_dir` (host -> container) skips `transport_excludes`; `copy_out`
(container -> host) is the raw Docker archive API and carries the whole tree,
`target/` included. The asymmetry was invisible for as long as the runtime
image had no cmake — nothing could compile, so no `target/` existed.
The moment missions could actually build, every collection died on a build
artifact:
failed to unpack `…/repo/target/debug/build/ahash-…/build_script_build-…`
`phase_runner` then correctly refused to capture, rather than record a stale
tree as an empty diff — so mission 01a00c57's coding phase, which had done the
work, delivered nothing and retried forever. A fix that let missions compile
created a delivery failure one layer down.
`unpack_into` now skips excluded entries by NAME at any depth (a workspace has
a `target/` per crate) and logs how many it dropped.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
99dd29cc8a |
fix(worker): the stuck-run reaper was killing healthy sonnet-5 turns
REAP_STUCK_AFTER_SECS was 15 minutes; the runtime grants a single turn `timeout_secs = 3000` (50 minutes). A run journals its first step record when its first step COMPLETES, so a turn still legitimately in flight is indistinguishable from a wedged container — and with a window shorter than the turn timeout the reaper does not detect stuck runs, it kills slow healthy ones. The old value was calibrated on haiku, where "healthy first-step latency is typically 5-60s" held. Moving mission agents to sonnet-5 made first turns longer than the window: mission 01a00c41's research phase was reaped at 900s having already written +402/-39 across 13 files. We only know it was healthy because the delivery path captured and pushed that work anyway, to branch clawmates/mission-01a00c41-421200ee at b08df7b6. Raised to 60 minutes, above the turn timeout, with the invariant written down so the next person changing either number sees the relationship. Generalises: a liveness timeout calibrated against one model becomes a correctness bug when the model changes. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
b31a79f650 |
fix(llm): the subscription 429s were a malformed request, not a rate limit
On the OAuth path Anthropic requires the Claude Code identity to be its own
first system BLOCK. We concatenated it with the caller's prompt into a single
string, so EVERY server-side call that set a system prompt was rejected — and
the rejection arrives as `429 {"type":"rate_limit_error","message":"Error"}`,
which reads as throttling and is not.
Measured on one token, seconds apart:
"PREAMBLE" (string) -> 200
"PREAMBLE\n\nJudge the …" (string) -> 429
"PREAMBLE" (string) -> 200 (control)
["PREAMBLE"] (blocks) -> 200
["PREAMBLE", "Judge the …"] (blocks) -> 200
while the account reported `5h utilization 0.07, 7d 0.11, overage 0.0`, every
window `allowed`. A Max 20x subscription at 7% was being read as out of
capacity.
What this was breaking, silently, for as long as it has been there:
- every `done_when` verdict on the subscription judge. Mission 01a00bbb
pass 2 returned "could not evaluate the completion condition this pass"
and BURNED one of the phase's three passes on it.
- the boot preflight, which reported `claude-opus-4-8 throttled (configured,
no capacity now)` on every start — a diagnostic that was itself the bug.
- mission_refiner, phase_summarizer, swarm planning.
The `claude` CLI was unaffected throughout, because it sends its system prompt
as blocks. That divergence is what made this look like an account problem: the
agents worked while everything server-side "throttled".
After the fix the preflight reports opus-5, sonnet-5 and haiku all `ok`.
The API-key path keeps sending a plain string — it never had this constraint.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
69c294addc |
fix(models): coding runs on sonnet-5, judging on opus-5, haiku only as last resort
Operator model policy: haiku ONLY for genuine yes/no questions; anything
requiring thinking is opus-5; coding is sonnet-5.
The mission AGENTS were running haiku, and nothing in the product said so.
`provider_alias_for` maps every `claude-*` binding onto the single alias
`claude_cli.default`, so a crew whose `model_binding` reads `claude-sonnet-5`
— as this deployment's does — still ran whatever that alias pointed at, which
was `model = "haiku"` in the runtime config. The binding is cosmetic; the
alias is the truth.
Measured consequence on mission 01a00bbb: the coding agents claimed six INT
items complete and had committed three, and the done_when judge caught it by
auditing git history against the claims.
Model assignments, by what the component actually does:
evaluator (done_when judge) haiku -> opus-5 reads evidence, audits it
against the repo, writes
guidance. The verdict is a
boolean; the work is not —
and this is the one component
whose failure mode is passing
work that was never done.
judge_model 4-8 -> opus-5
mission_refiner 4-8 -> opus-5 composition
phase_summarizer 4-8 -> opus-5 composition
swarm planner 4-8 -> opus-5 planning
subscription preflight head 4-8 -> opus-5
fallback chain head 4-6 -> sonnet-5 haiku stays BELOW it as a
last-resort link, never a peer
Every value stays env-overridable; only the shipped defaults move.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
53da4d7e6d |
fix(runtime): a mission could not build the repo it was given
`clawmates-runtime` shipped with `gcc` and `make` but no `cmake`, no `g++` and no `python3-dev`. Measured on clawhdf5, three probes: no cmake → "is `cmake` not installed?" exit 101 after 13s no python3-dev → "cannot find -lpython3.11" exit 101 at link with both → cargo test PASSES exit 0 after 69s This is not only the delivery gate. The AGENTS run in this image, so a coding phase was writing Rust it had no way to compile or test — which reframes the last run's 11 agent commits as unverifiable by construction. `images/agent-toolchain/Dockerfile` (the microVM path) has had `cmake build-essential` all along, and its own header warns about precisely this: "if `cargo` is present in one image and absent in another, the same mission passes or fails depending on which backend it landed on, and nothing says why." Both images now install the same set — it was missing `python3-dev` too. `images/runtime-toolchain.Dockerfile` is a thin local overlay so the laptop can run today without recompiling zeroclaw from the fork; it is meant to be deleted once a runtime image built from the corrected deploy/ Dockerfile is published. Also: a build failure is no longer reported as a red suite. Both are cargo exit 101, and `verify_tests` mapped every non-zero to `Failed(code)` — so a missing toolchain was recorded as the USER's tests failing. It now returns `CouldNotRun` with the reason when the output shows a compile or link failure. Deliberately narrow: a failing `assert!` still reads as red, because letting broken code past `on_green_tests` is the expensive direction to be wrong in. Both directions are pinned by tests built from today's two real samples. And the coding phase finally has a loop: `research_and_code.toml` declared `loop = "until_no_more_int_items"`, which `phase_config.rs` lists as DECLARED_BUT_UNREAD. Iteration is driven by `max_iterations` + `done_when`, and with `max_iterations = 1` and no `done_when` the phase ran ONCE and was never judged — reporting `completed` whatever it produced. Now 3 passes against a stated goal, wording per the measured rule (say what the tree must CONTAIN). Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
10341cf7fe |
fix(missions): a retry's work is no longer silently destroyed
Two independent bugs, either of which loses everything a retried phase
produced, and neither of which reports a failure.
1. Capture is suppressed forever on a retry. Both
`capture_finished_coding_phases` and the sweeper's last-chance
`capture_outstanding_phases` skip any phase that already has a
`code_diff` artifact. That guard is right for a phase that ran once and
catastrophic for a retried one: the artifact from the FAILED attempt
suppresses capture of the new attempt, the container is reaped on its
normal grace, and everything the agents committed inside it is gone.
The UI keeps showing the old diff, so the mission reads as delivered.
`retry_phase` now clears the reopened phases' captures in the same
transaction that reopens them, which is what makes its own doc comment
("the phase card starts fresh on the retry") true of the artifacts too.
2. `git add` exits non-zero over a gitignored path while staging correctly.
Measured: with a populated `target/`, `git add -- . :(exclude)target`
exits 1 and stages the right files; `-c advice.addIgnoredFile=false`,
`--ignore-errors`, `-A` and `:/` all behave identically. Propagating
that with `?` aborted the commit AFTER a successful staging — no branch,
no commit, no push — for every Rust repo an agent has built in.
`capture_phase_diff_at` already treats the same command as advisory;
the commit path now does too, and the staged index decides.
Mission 01a00538 hit both: it completed research and coding on the retry,
11 agent commits and all, delivered a patch dated the previous day, and
lost the commits when the container was reaped. The remote was never
touched — its HEAD still equalled the mission's own base_sha.
Covered by a test that drives real git and asserts the files are staged
regardless of the exit code.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
fd5e71ccfe |
fix(missions): re-assert a mission's crew when its container is recreated
Claws are provisioned exactly once, at on_launch. `ensure_container`
RECREATES a container that is not running, and recreation reseeds
.zeroclaw from the seed directory — which does not hold this mission's
claws. The agents still exist in Postgres and the crew query looks
perfect, so nothing reads as broken; the alias is simply gone from the
daemon and /ws/chat answers 400 Bad Request. A retried mission could
therefore never connect again.
Re-assert the crew after ensure_container. provision_claw is idempotent,
so this costs one call per claw on the happy path and is the difference
between a resumable mission and a dead one.
Two things this has to get right, both of which fail silently:
- Aim at the per-mission daemon via for_gateway(ec.endpoint), never
from_env() — that targets the shared global gateway and leaves this
container with no claws at all, exactly as for_gateway's own doc
comment warns.
- Provisioning creates the agent but cannot set workspace.path (the
config prop-schema has no way to express it), so follow with
pin_agent_workspaces or every claw runs in its own sandbox and
delivers nothing.
Verified on mission 01a00538: research and coding phases both completed
after four straight failures, with all five graph aliases present and
ten claws pinned to /mission/repo.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
85a6038c08 |
fix(missions): create /mission before copying the checkout in
`copy_in` cannot create its own destination, so a mission whose container had no /mission directory failed its checkout sync outright. In copy mode that is how the agent gets the code at all, so the phase launched against an empty tree. Exec `mkdir -p /mission` as root first. Idempotent, and it costs one exec on a path that already shells out. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
6e8785f159 |
fix(missions): a server restart no longer kills a running mission
Mission 01a00538 ("ClawHDF5 REsearch and Refactor") failed 19 minutes and 93,762
tokens into its research phase with `pair failed: 403 Forbidden`, and its coding
phase was then correctly skipped as unreachable. The cause was not the coding
phase and not the model — it was pairing.
A per-mission runtime is authenticated with a SINGLE-USE pairing code, and the
bearer token it returns was cached in memory only. Any restart of the server
discarded that token; the next turn re-paired with a code the gateway had
already spent and got 403 — permanently, for that mission. A deploy, a crash or
an OOM would each do it. The durable-run machinery exists precisely so work
survives a restart; pairing was the one thread that did not, and it failed
closed.
`missions.runtime_token` persists the token at the moment pairing succeeds, and
the worker seeds the executor's cache from it, so a new process reuses the
credential instead of re-pairing. Persisting is best-effort: failing to save
must not fail a turn that just paired successfully.
Verified by reproducing the original failure: launched a mission, confirmed the
token was written, restarted the server MID-PHASE, and watched the mission run
to completion with no pairing failure.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
43436d7181 |
feat(telemetry): push bus for live agent frames
/api/world/live is a 2s database poll, which is right for queryable state and wrong for a token stream: reasoning only became visible after a step finished and its row was written. This adds a process-wide broadcast bus that topology_exec publishes to as the runtime's WebSocket delivers frames, and the SSE handler forwards without waiting for the next tick. Measured: the pushed frame arrived ~2.2s before the polled copy of the same text. Design notes worth keeping: - A global (OnceLock), not an AppState field. The publisher is reached through phase_runner -> topology_worker -> MissionTap, none of which hold AppState; threading a handle through all of them would put a UI concern into four layers that have no other reason to know about one. - Lossy by design. A slow subscriber lags and skips rather than applying backpressure to the agent producing. mission_events remains the durable record; this bus is the fast path, never the source of truth. - Only `claw_<uuid>` aliases are attributed. The governor, door and evaluator drive real turns under other names, and attributing their output to an agent would put words in someone's mouth. Asserted in a test. - The poll no longer emits `reasoning`: with both paths live, every turn arrived TWICE — once pushed, once polled ~2s later. The row is still written; this feed just is not its second mouth. CEILING, measured rather than assumed: turns are not token-level because the runtime is not streaming. zeroclaw's claude_cli provider runs `claude -p --output-format json`, which returns ONE result object when the turn completes — there are no incremental tokens to forward. Making this genuinely token-by-token needs `--output-format stream-json` and incremental parsing in the zeroclaw fork, not here. The bus is in place and will carry them the day it does. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
ba9d7aa185 |
feat(telemetry): WORKING ON NOW shows the mission an agent is on
The last of the three declared-but-never-emitted event types. `agent.task.update` had no producer anywhere in the backend, so the card read "idle — no active task" for an agent that was mid-turn. Derived rather than newly instrumented: an agent is working on its crew's RUNNING mission, and that mission's phases are the steps (completed/skipped → done, running/evaluating → active, else pending). Nothing is emitted for an agent with no running mission, so "idle" stays truthful rather than freezing on a stale last-known task. Verified on a live mission: 116 agent.task.update events observed on /api/world/live, carrying the mission title and phase steps, with the state advancing pending → active as the phase started. That closes the set. Of the seven cards in the command centre, five were dark: three had no emitter at all and two read a table the mission path never wrote. DOORS and LOOPS were correctly wired the whole time and were reporting an honest zero. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
bf40d10064 |
feat(telemetry): the reasoning stream actually streams
`agent.reasoning.delta` and `agent.tool.call` have been declared in the taxonomy and listened for by the command centre since it shipped — and NOTHING ever emitted them. The world feed emitted five types; neither was among them, so REASONING STREAM could not populate no matter what an agent did. The feed is a database poll, not a push bus, so a live card can only show what was persisted. The worker already holds each step's output text and the claw that produced it, so it records a `reasoning` mission_event (truncated — the card renders a tail, not a transcript, and mission_events is capped per phase), and the feed emits it forward from a cursor that starts at the current max so a page load streams rather than replaying history. `tool.call` is emitted from the same place. On the container tier it will stay empty, and that is correct rather than broken: those agents are tool-free behind the §15 door. Tool lines appear where agents actually hold tools. Verified on a live mission: agent.reasoning.delta observed on /api/world/live carrying the agent's own text, keyed by agentId. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
8be7b3c9b2 |
feat(telemetry): record per-agent usage for mission turns
The command centre's SPEND, ACTIVITY and THROUGHPUT cards read `usage_events`, and nothing on the mission path ever wrote a row: `cm_billing::charge` was called only from the agent-run path. Measured mid-mission with 14 agents live, `usage_events` was 0 while a crew had just burned 15k tokens — so an agent that had done real work reported zero cost and zero activity. The worker already knew everything needed: it logs node, role and token count per step, and the node's `attrs.agent` carries the `claw_<uuid>` binding the runtime dispatches on. This routes that to the ledger. `charge`'s run_id is now Option. `usage_events.run_id` references `agent_runs`, and a topology turn has no row there — passing its `topology_runs` id was a foreign-key violation, which is exactly what the first attempt hit. NULL is the honest value; the agent-run caller still passes its real id. The executor reports one total rather than an in/out split, so the cost is right (credits price the sum) and the columns record it as output rather than inventing a split. Verified end to end on a real mission: 4 agents, 1046-8670 tokens each, credits attributed per agent, and the SPEND/ACTIVITY queries now return real numbers. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
8ef7067467 |
fix(repos): a scoped connection can name a user, not just an org
Scoping a Gitea connection to `osobh` — the personal namespace clawmates itself
lives in — failed with "org 'osobh' not found or PAT lacks access". The sync
only ever called /orgs/{owner}/repos, and Gitea serves user namespaces from
/users/{owner}/repos. The error pointed at permissions for what was really a
wrong endpoint, which is the kind of message that sends you to rotate a token
that was fine.
Retry as a user on 404 before giving up, and say what was actually checked.
Verified: owner=osobh now syncs 7 repos, owner=redclaw 22 — 29 instead of the
182 an unscoped connection pulls.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
b290025fc4 |
feat(agents): make delete permanent, and expose the census
A soft delete marked the row and left it. The agent stayed in the table forever,
kept appearing on any surface that forgot `deleted_at IS NULL`, and deleting it
again did nothing — the decision was recorded and never honoured. Two agents on
this deployment had been in that state since June.
`deleted` is now a fifth lifecycle state, collected with NO grace window: a
human already decided, months ago. It takes usage_events with it, which is the
explicit trade — the alternative is rows that outlive the decision to delete
them.
Two endpoints, because this was previously only answerable by reading the
database by hand:
GET /api/claws/lifecycle the census: who is active, completed,
orphaned, deleted — and what is reapable
POST /api/claws/lifecycle/sweep run the reap now, rather than waiting out
the hourly timer for a decision already made
Verified end to end: census reported both as `deleted`/`reapable`, the sweep
returned {"reaped":2,"failed":0}, and agents and usage_events both went to 0.
The safety property is unchanged and re-asserted by a new test: adding `deleted`
did not make `owned` or `active` reapable.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
ccbc387f4b |
fix(agents): stop listing soft-deleted agents
Deleting an agent looked like a no-op: it disappeared from the workforce but
stayed on the Team board, and deleting it again did nothing because the row was
already marked. Two queries selected from `agents` without `deleted_at IS NULL`:
routes/team.rs the leaderboard — the surface still showing them
routes/world.rs the "working" set — a deleted agent holding a stale
agent_containers row rendered as live
Observed on this deployment: /api/workforce correctly returned nothing while
/api/team/leaderboard returned two agents soft-deleted back in June.
NOT changed: those rows still exist. Making delete permanent means hard_purge,
which also deletes usage_events — billing history, 6 credits on one of these
two. Discarding that as a side effect of tidying a roster is an explicit
decision, not something a display fix should smuggle in.
.sqlx regenerated: team.rs uses the compile-time-checked query! macro, so the
cached entry no longer matched.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
a494634f81 |
feat(agents): classify agents by lifecycle and reap the finished and orphaned
A mission mints a crew, and the only thing that reaped one was DELETING the mission. A mission that merely completed left its agents in the roster forever, and a crew whose reap was skipped or failed left agents bound to nothing — indistinguishable in the UI from the operator's own staff. Four states, from one query: owned no agent_template_link row → hand-created. NEVER reaped. active on a running/draft mission → working right now. Kept. completed every mission terminal → reaped after a 24h grace. orphaned minted, bound to nothing → reaped. The discriminator is `agent_template_link`, which mission_orchestrator writes per minted claw. This matters more than it looks: verified on live data, a hand-created agent and an orphaned crew member both have ZERO team links and are structurally identical by binding alone. Judging orphanhood by "no team" would delete the user's workforce. Provenance is the only honest signal. The grace window exists because the results view, the World's 24h replay and "who did this work?" all read the crew AFTER the run ends; reaping on the terminal transition deletes the answer exactly when the question gets asked. A completed crew with no usable timestamp is KEPT — a missing date must never read as "old enough to delete". Also fixes the delete summary, which reported how many claws were FOUND rather than purged: "reaped 4 claw(s)" was printed by a delete that purged none, which is precisely the log you would read while wondering why the agents are still there. It now reports purged / kept / FAILED, and failed > 0 is the orphan case. Verified against live data — all four states observed, including the two that look alike. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
548f977212 |
style: rustfmt the four files the repo-less mission fix touched
Found while removing .github/workflows/ci.yml: three of the four files in that change were unformatted, and four of the diffs were newly introduced (the new prompt tests and the tool_preamble format! call). Formatting only the files that change already touched — a repo-wide `cargo fmt` would be 63 files of unrelated churn and belongs in its own commit. Mechanical; `cargo test -p cm-api --lib` stays at 322 passed. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
4dec77ae6d |
fix(missions): give repo-less container missions the workspace they are promised
Every agent on a research_only mission refused to work, each reporting it was "in Claude Code", had no /mission/repo, and only had Read/Edit/Bash. All three statements were true. The run still recorded completed — 5 turns, 7.4k tokens, 0 artifacts, no error. The machinery is correct when a repo IS bound (verified on a live prod per-mission container: /mission/repo present, all 5 agents pinned). Only the repo-less path was broken, in three layers that disagreed by construction: - sync_in no-oped without a host checkout and copy mode does not bind /mission, so NOTHING created /mission/repo. The microVM tier already creates it, for the stated reason that "the guest needs the workspace to exist before the agent writes into it". Creating it host-side also un-breaks sync_out, equally a no-op before, so work survives across phases instead of being wiped. - pin_agent_workspaces returned Ok after pinning ZERO agents, so the deliberately-fatal guard in mission_orchestrator could never fire. Its error text already described the exact outcome we got. - The prompt advertised ZeroClaw tool names and explicitly denied `bash`, while every executor ends in `claude -p`: microVM passes Read/Edit/Write/Bash/Agent, session passes Read/Edit/Write/Bash, and claude_cli agents get Claude Code's native toolset — ZeroClaw's gating never reaches the subprocess. It was telling agents to use missing tools and avoid present ones. And it went green because mission_outputs logged the failed collect and continued — with the fail-empty rule and the NO-OUTPUT marker both BELOW that continue, so the phase was retried forever and never failed. The retry is now bounded by a grace window off completed_at. Verified end to end: mission completed, agent wrote /mission/repo/research/firecracker_vs_docker.md, collected and registered as a document artifact (6.6 kB of real content). Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
dc8f65fc64 |
fix(metrics): GPU, network and disk IO were arriving and being dropped
`gpu_pct` read `info.g` as a scalar. Beszel 0.18 puts GPU in a different
collection entirely, as a MAP keyed by GPU index —
`{"0":{"n":"GeForce RTX 5060 Ti","u":0,"p":4.38}}` in `system_stats.stats`
— and `systems.info` carries no `g` at all. So every NVIDIA node reported
null while the data sat one request away. Null and "no GPU" are
indistinguishable downstream, so the fleet card showed nothing and a
`gpu_pct` drain rule could never fire, both without an error.
`net_sent_ps`, `net_recv_ps`, `disk_read_ps` and `disk_write_ps` were
columns nothing ever wrote. They come from the same sample.
The two array orders were MEASURED, not read off a schema, because
inverting one does not fail — it reports upload as download forever:
b = [sent, recv]. `stats.ni` gives per-interface [sent_ps, recv_ps,
total_sent, total_recv]; indices 2 and 3 matched /proc/net/dev
tx_bytes and rx_bytes on all four of tank's interfaces, and `b` is
the sum of the per-second pair across them.
dio = [read, write]. An 800 MB dd on tank moved index 1 from 7441 to
23688 while index 0 stayed near zero.
`info.ct` is deliberately NOT mapped to container_count. It reads 1 on
tank, which runs 1 container, and also 1 on architect, which runs 4 —
right exactly often enough to pass a spot check.
One extra request per poll, not one per node: the newest 1m sample for
every system arrives in a single sorted page. A hub that cannot answer it
falls back to the info snapshot rather than losing the CPU and memory
readings that still work.
GPU is the busiest card, not the mean — placement asks whether there is a
free GPU, and averaging a saturated card with an idle one answers a
question nobody asked.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
9f76f0915b |
fix(runtime): announce the runtime image once, not on every sweep tick
`MissionRuntimeProvisioner::from_env` is called per use — on every mission
launch and from the terminal-mission reaper sweep — so the line added in
|
||
|
Omar SobhandClaude Opus 5
|
cdc45bd082 |
chore(runtime): promote v0.8.4 from canary to the default image
Every mission on gw-04 was already running v0.8.4 — pinned by CLAWMATES_RUNTIME_IMAGE in .env. The canary is retired: the default tag `clawmates-runtime:sync` now IS that image, the override is commented out, and the built-in default is the single source of truth again. Promoting it exposed why the pin was load-bearing in the first place. The default tag resolved to zeroclaw 0.8.3 — two releases behind what was actually running — and the REGISTRY copy of the same tag was a different image again, 849MB against 2.31GB, without the Rust toolchain. A host that pulled `sync` rather than retagging it would have lost the on-green test gate with every probe still reporting success. A moving tag pointing somewhere old resolves perfectly, starts perfectly, and runs old code. Nothing anywhere said which image a mission got, so two things now do: - mission_runtime logs the image it resolved and whether that came from the env override or the built-in default, once at startup. - runtime_preflight probes `zeroclaw --version` alongside the other tools and prints every tool's VERSION, not just that it is present. A presence check passes happily on an image two releases behind, which is exactly what happened here and was found by running the binary by hand. Rollback is a retag: `clawmates-runtime:pre-v084-default` on gw-04 holds the previous default, and .env.pre-v084-default holds the previous pin. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
d810fc0a86 |
fix(viz): read the gateway's real tool_call keys, and correct the record
The frame is `{"type":"tool_call","id","name","args"}` — zeroclaw-gateway
/src/ws.rs. The tap read `tool` then `name`, and `arguments` then `input`.
`name` happened to be in the fallback chain; `args` was not in it at all,
so a container-tier tool call would have been recorded with its name and
NO path — a tool that reads as having touched nothing. `tool` and
`arguments` belong to `approval_request`, which is where they came from.
Also corrects what the histogram was read as saying. A mission turn on
gw-04 carried only chunk/done/session_start, and the first reading was
"there is no tool_call frame". Wrong: `grep -c tool_call` on the deployed
0.8.3 binary returns 46. Container-tier agents are provisioned tool-free
behind the MCP door (§15), so they call nothing — there is nothing to
observe on that tier, and nothing is broken.
That distinction is exactly what the histogram was shipped to make
possible: a tap matching no frame is otherwise indistinguishable from a
mission that used no tools.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
31158467f4 |
feat(viz): microVM tool motion is live, and needs no fleet-node change
The plan deferred this as "the only fleet-node binary change". It is not one. `fcagent` is thread-per-connection — its own comment says so, and the live log tail has relied on exactly that for the whole length of a turn, on a second connection. So the host can drain the tap WHILE the turn's exec is in flight, from the server alone. The turn and a 20s drain loop now run concurrently. A coding phase shows its files being touched as it works rather than an hour later, all at once, and the drain is bounded by a cursor so a repeated poll returns only what is new. The cursor counts LINES, not parsed events, and that distinction is the bug this commit would otherwise have shipped. The hook appends the event and then a newline of its own, so a two-event tap is four lines; advancing by event count leaves the cursor two lines short, `tail -n +N` hands back events already recorded, and the live drain re-records everything it has already written — worse the longer the turn runs, and silent throughout. Caught while writing the test, not by it. `tap_sink` and `VmOutcome::tools` are mutually exclusive by contract: with a sink, the sink owns recording including the final batch and `tools` comes back empty. Handing the same calls back on both would double every file orb's weight with no way for the caller to tell which it was looking at. The sink is an unbounded channel to a recorder task, so the VM executor stays free of the database: it observes, phase_runner records. The task ends when the sender drops with the phase. Verified before this change: the microVM tap is real. The `microvm` scenario passed 6/6 and left ten `tool.call` rows and a `file.touch` on MICROVM.md, repo-relative, from Claude Code's own PostToolUse hook. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
f8438c32ea |
feat(viz): kind-specific choreography and a finished mission you can read
Security: the pawns already orbited their destination, so homing them at
the security station gave circling for free. This adds the radial
press-and-retreat — an agent closing on the target and backing off reads
as probing it, where a fixed radius reads as waiting — and holds the
stochastic target release while probing, or the circling breaks up into
stray trips that look like distraction rather than a scan.
Findings are `mission_tasks` rows, one orb each, popped once. There is
deliberately no severity anywhere in the path: the scanner keeps
severity, file and line as substrings inside `title`, so a severity
parsed out of prose and rendered as an orb's RADIUS would be the picture
asserting a measurement the data never contained. Count only.
Benchmarks annotate the station, as text. `delta` has no schema —
compute_delta emits `{kind:"opaque"}` whenever the before/after metrics
were not structurally comparable, which is most drivers. The server
formats the shape it can parse and COUNTS the rest; an unparseable driver
reports "3 sample(s)" rather than an invented improvement, and an opaque
delta says nothing at all.
The finished map: the live label rule gates service/event nodes on
`heat > 0.12`, which is exactly backwards once everything has cooled — a
static map would be unlabelled dots. Frozen, the 25 most-touched nodes
label regardless of heat, phase stations carry a second line counting
what they produced, and the camera is released ONCE so it frames the
result even if the user panned during the run.
Every count in a caption is read off the drawn scene rather than a
parallel tally, so the words and the picture cannot disagree.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
9e61e3ba35 |
feat(viz): what the agents actually did, as structured events
The World could draw a mission's shape but nothing about the work. The
detail existed only as prose in checkpoint.log and model output, where a
tool name is indistinguishable from an agent *talking about* a tool — so
it was never parsed, deliberately. `mission_events` is the structured
channel that replaces it.
Three taps, one table:
- Container tier: the `_ => {}` at the end of topology_exec's typed frame
stream now matches `tool_call` and reads the tool's JSON ARGUMENTS for a
path. Never the prose summary — a path scraped from a sentence would put
files on the map that no agent opened, and the test proves a Grep whose
summary says "src/main.rs" produces no file touch. The frame name itself
is unverified, so the same commit ships an unmatched-frame-type
histogram: a tap that matches nothing looks exactly like a mission that
used no tools, and this is how one gw-04 run names the real frame.
- microVM tier: a `PostToolUse` hook, the seam vm_stop_gate already proved
fires under `claude -p`. It copies stdin to /root/tap and exits 0
unconditionally — a non-zero PostToolUse hook talks back to the model,
which would turn the observer into a participant. Drained before collect,
since the VM is destroyed moments later.
- Phase transitions: five identical copies of the pending→running UPDATE
became one `mark_phase_running`, and `close_finished_phases` grew
RETURNING. Its CASE decides each phase's status inside SQL from rows the
statement does not change, so it cannot be re-derived afterwards without
writing that CASE twice — without RETURNING it emits zero phase.completed
and reports success.
The settings.json hazard the plan called out: the stop gate wrote the
WHOLE document, so a second hook writer would have silently erased it and
a coding phase would then complete having written nothing — the exact
failure the gate exists to catch. There is now one composer,
`vm_tool_tap::guest_settings`, one writer, and a source-walk test that
fails if anything else writes a settings document.
`mission_events.run_id` carries no FK on purpose: phase_runner DELETEs
topology_runs on retry, and a cascade would erase a phase's whole history
the moment it retried — silently, since a cascade is not an error.
world.rs streams it with a cursor that separates backfill from motion.
Everything already in the table when a subscriber arrives is drawn as
settled history; only what lands afterwards animates. Otherwise opening a
finished mission replays an hour of tool calls as a burst storm.
Bounded twice: 400 events per phase (enforced inside the INSERT, since
two concurrent taps would each read a count below the cap) and a 7-day
retention sweep in mission_gc.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
c2fa8067e1 |
feat(viz): the coding station fractures into the files it worked on
The engine already turned `file:src/lib/a.ts` into a real dir chain, but both copies of that loop rooted it at the origin — so a mission's files floated beside the map instead of belonging to the work that produced them. One `fileParent` helper now serves both call sites; splitting them was how half the files could end up nesting correctly and half not, decided by whichever code path saw the file first. Files hang under the coding station when there is exactly one, else the single running phase, else the origin. `world.touch` carries no phase id, so with two coding phases any attribution is invented — the fallback is the honest answer. Two ordering hazards, both silent: - the server emitted files BEFORE phases, so on the first pass a file arrived with no station to hang under and first-write-wins pinned its tree at the origin. Loops reordered, with a source-walk guard. - `setFileHome` re-parents trees rooted before the plan landed, for the reconnect case the ordering alone cannot cover. `mission.file` with `source: "tool"` is treated as motion (burst, pawn beams); `"diff"` is end-of-phase truth and only marks the file present and warm — bursting every file of a captured diff would set the whole map alight at once on reconnect. Directories taper in radius and opacity by path depth, so `src` and `src/lib/live` no longer render as identical dots. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
cb8184e784 |
feat(delivery): record WHICH files a phase touched, not just how many
`capture_phase_diff_at` parsed `git diff --stat` down to three integers and
threw the filenames away. Nothing downstream could name a single file a coding
phase changed: the World can draw a coding station but nothing underneath it,
and an operator reading a mission sees "11 files" with no way to learn which.
A second `--name-status` call now records the paths into the code_diff metadata
and into `names.txt` beside `diffstat.txt`, so raw evidence survives
independently of the JSONB.
Three ways this could have been wrong, each guarded:
- Different revision or excludes from the `--stat` call would make
`files_changed` and the path list describe different diffs, with no way to
tell which lied. A source-walk test pins both to the same `base_sha` and
the same `excludes`.
- Running after `git reset --quiet` would drop newly CREATED files, since
`--intent-to-add` is what makes them visible to diff at all — and the stat
would still count them, so the list would look merely incomplete rather
than wrong. A test asserts the ordering.
- A rename is `R100\told\tnew` — three fields. Taking field two records where
the file USED to be, naming a path nobody can open, and the bug is
invisible in any repo where nothing was renamed. `changed_paths` is now
shared with auto_merge (which had the same parse) and takes the NEW path,
with tests for renames and copies.
The list is capped at 500 paths with `files_truncated` beside it: a cap that
silently clips is worse than no cap, because "touched 12 files" and "touched at
least 500" would look identical.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
006432c2dc |
feat(viz): the mission becomes a map — centre, stations, and agents at theirs
The clump was structural, not cosmetic. Four causes, each fixed here. `homes` (an agent's resting node) was written only by `seed()`, so every agent homed to the mission centre and orbited the same dot regardless of which phase it was on. `setHome` points each agent at its CURRENT phase, and the existing physics does the rest for free: the pawn rests at its station, the pawn→home line tethers it there, and a touch becomes a visible departure and return. No new motion code. The mission node was seeded as `level: "team"` (my own bug from the focus work). `seed()` casts that straight to a Tier and `ensureNode` is first-write-wins, so it was created as a small teal team dot that the later `node.activity` could never upgrade. That dot at the centre of the scene was one line. `mission`/`phase` replace the retired `repo`/`loop` tiers rather than adding a parallel set. The backend stopped emitting repo:/loop: ids, which left their whole landmark treatment — bigger radius, distinct colour, always-labelled, 60s fade instead of 22s — orphaned on prefixes nothing sends. Missions and phases need exactly that treatment. The two separately-written prefix→tier ternaries in onTouch and onNodeActivity are now one `tierFor`: they agreed only by luck, and whichever path saw a node first fixed its tier forever. Phase stations spring out at 190 rather than the shared 64, or they pack into a rosette around the centre and the point — agents moving BETWEEN stations — is invisible. Idle roam is off under a mission scope: wandering to a random node keeps an idle workspace alive, but inside one mission it sends agents to files nobody opened, which reads as work and isn't. Palette is injected at construction and keyed on template_kind, so a benchmark run and a security sweep no longer render identically to a research mission. It also collapses two uncoordinated kind→colour maps that had drifted: LEVEL_COLOR by tier, and the fireColor if-chain by id prefix. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
5f85dbb718 |
fix(world): missions were never really on the wire
Three bugs in one query, each hiding the next, plus one that made the whole rich layer dead code. `active_missions` joined `team_members` on `missions.team_id` — the LEGACY pointer at the first minted team, superseded by the `mission_teams` junction in 0056. It was an INNER JOIN, and `mission_orchestrator::on_launch` deliberately mints no team for a microVM mission, so the platform's primary execution tier was dropped by a join and the World has been showing nothing at all for it. And it selected only `status='running'`, while missions finish in minutes, so the scene was empty almost always. Now: join `mission_teams`, LEFT so teamless missions survive (their `agent_id` is NULL and no pawn beams at them, which is the truth — nothing on this platform ran that phase except a VM), and include missions finished in the last 24h carrying `status`/`template_kind` so the client can draw a finished map instead of animating a corpse. `?mission=` scopes the feed server-side. The whole phase plan now ships as `mission.phase`, including phases that have not started: a phase list that appeared only as phases began made a five-phase mission look like a one-phase mission until it was nearly over. Attribution reuses `phase_runner::purposes_for` rather than copying it — two copies would let the picture disagree with the machine about who is working on what, which presents as a rendering bug and is really a lie. Deleted the checkpoint tail. It read `topology_runs` keyed by an `agent_runs` id; mission phases live in `topology_runs` under independently generated ids, so it ran every poll and matched nothing, for every mission, forever. That is why no mission has ever shown tool or file activity. Not repointed at `topology_runs`: its only per-step content is the agent's own prose, and a tool name in prose cannot be told from an agent talking about a tool. The a2a `run_events` tail is kept — it genuinely works for the path that writes it. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
d3a398716b |
fix(workforce): a crew should not read as an alphabetical run
Seeding the name pick with the role index (0..n) started every crew at the top of the pool and took the next free names, so the first mission after the switch to per-mission crews hired Aarav, Abebe, Adaora, Adrian, Agnieszka. Unique and correct, and transparently generated. Seed from the claw's own uuid instead. UUIDv7 puts its random bytes LAST — the leading bytes are a timestamp, which would cluster the same way — so the tail is what spreads five picks across the whole pool. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
98037f9b3e |
feat(workforce): every mission hires its own crew
Reverses the reuse added earlier, by operator decision. Reuse hired the existing claw for a (template, slot) so the roster stayed at one team — but it also meant every mission was staffed by the same five names, and the workforce view showed one crew repeated down the page with nothing to tell the missions apart. Distinct crews read better than a bounded roster. The cost is the one reuse existed to avoid: claws are lifecycle='permanent' and nothing reaps them until their MISSION is deleted, so the roster now grows by the team size per mission. `agent_names::pick` keeps names unique workspace-wide and degrades to a numeric suffix rather than colliding, and the pool grew from 70 to 200+ given names so a workspace runs ~35 missions before the first repeat. `reusable_claw` is kept in cm-db with its tests: this policy has now flipped twice and the query is the hard part. Also revives a test that had silently stopped running. An edit stranded `runtime_data_is_scoped_to_one_mission`'s `#[test]` above its neighbour, leaving two attributes there and none here — so the neighbour ran TWICE and this one never ran at all. The total test count was unchanged by the fix (291 before and after), which is exactly why a count is not evidence: rustc had said "duplicated attribute" and "function is never used" all along, and both read as ordinary warnings. The test guards per-mission `/zeroclaw-data` isolation, i.e. one mission reading another's door token. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
c85027c83a |
fix(workforce): team_members.role, not role_slot
The roster query named tm.role_slot. That column is on agent_template_link; team_members calls it plain `role`. These queries use untyped sqlx::query(), so nothing caught it at compile time and the endpoint 500'd on its first real request — the 401 an unauthed probe returns looks identical whether the SQL is valid or not, which is why the payload had to be fetched with a real session before believing the route worked. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
0ad53da49c |
feat(workforce): missions group the roster, and agents get human names
Three things, all visible on the agents page.
**The roster looked like it was multiplying.** The sidebar flattened
orgs → companies → teams → agents, which renders a claw once per TEAM it
belongs to. Claws are reused across missions now, so a crew of five that had
run five missions appeared as twenty-five rows of the same five people. The
data was right and the view was lying. `GET /api/workforce` returns the roster
grouped by mission, and the tree renders each mission as a collapsible group,
so the repetition means something: the same colleague under each mission they
staffed. Claws on no mission come back under "Not on a mission" rather than
vanishing. The root now counts DISTINCT people, not rows.
**Agents were named after their jobs.** A team came back as planner, coder,
tester, reviewer, committer — the UI showed the same word twice (name on top,
role beneath) and the roster read as a stack of job tickets. New claws get a
given name from a deliberately wide pool (Amara, Vijay, Tomasz, Meredith…),
unique against the workspace roster AND within the team being minted. The role
is untouched in `job_title`, which is what the mission machinery binds on:
team_members.role_slot and the topology node carry the slot, so nothing
downstream keys off the display name. A reused claw keeps the name it had.
**Two latent reap bugs found while investigating a leak that was not one.**
Containers of completed missions are removed by `spawn_sweeper` after a
30-minute grace, and it works — an earlier report of leaking containers was me
reading that deliberate grace as a bug. But:
- the sweeper cleared the runtime binding even when teardown FAILED, and it
selects on `runtime_endpoint IS NOT NULL`. One transient docker error would
therefore hide a surviving container from the only thing that would retry
it, permanently. It now asks docker whether the container actually
survived: gone means clear, still there means keep the binding and retry —
which closes the orphan path without reintroducing the infinite retry the
original comment was guarding against.
- `set_runtime_binding` discarded rows_affected, so a mismatched workspace
updated nothing and returned Ok. The binding is how the sweeper finds a
container; a silent no-op there leaks one with no record of anything wrong.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
e2c312b728 |
docs(runtime): the judge no longer runs on a dead credential
`provider_alias_for` still documented the judge as deliberately sitting on `anthropic.judge`/API key, so a subscription throttle would degrade missions while verification kept working. That credential is an account with a zero balance — driving the real path returns 400 "Your credit balance is too low" — so the comment pointed the next reader at something that cannot answer. `anthropic.default` and `anthropic.judge` are retired from the runtime config and every agent that named them was repointed onto a live credential. The independence argument that put the judge there still holds; it is now served by a different FAMILY rather than a different key — CLAWMATES_VALIDATOR_MODEL is glm:glm-4.7 on gw-04, and cross_provider_judge already refuses a validator in the implementer's own family. Config-side (gw-04, not in this repo): 755 -> 246 lines, 128 -> 14 agent blocks, after sweeping 19 [agents.claw_<uuid>] corpses — every one verified against agents WHERE deleted_at IS NULL. Nothing reaped those, and the file is byte-copied into every mission. Verified: scout/judge/worker_kimi/worker_glm each answer on their new provider, and multirole passes 4/4 against the reorganized config. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
104e3ef27c |
fix(runtime): the kimi fallback hop spawned a binary that 401s
A throttled subscription had nowhere to go. `claude_cli.default` carried no
`fallback`, and neither target alias was declared — they existed only as
commented-out examples. Forcing a 429 with a shimmed `claude` surfaced four
defects that all read as correct config and do nothing:
- a `[providers.models.<f>.<a>.env]` SUB-TABLE is parsed then silently
ignored ("fields must live directly under ..."). This block was already
live for claude_cli.default, so the token injection has been inert. For
the glm alias it would have dropped the z.ai routing AND the clearing of
CLAUDE_CODE_OAUTH_TOKEN — credentials crossing between providers.
- an empty `[providers.models.kimi_cli.default]` is skipped at runtime.
- a claude_cli alias used as a fallback needs a non-empty `api_key` to pass
FamilyProviderFactory's default readiness gate, even though the provider
ignores the key and authenticates through `env`. Absent it the agent dies
at STARTUP, which takes out every mission, not just throttled ones.
- timeout_secs=600 capped every turn under the 3600s TURN_TIMEOUT from
|
||
|
Omar SobhandClaude Opus 5
|
4c418f7d9b |
fix(runtime): a turn gets the phase's budget, not 100s more than one call
A turn is an agent LOOP, not one model call. Each call inside it is bounded separately by the daemon — `claude_cli`'s `timeout_secs`, 600s on gw-04, verified in the live config — so TURN_TIMEOUT has to cover however many calls the loop makes, not one of them. It was 700s. MEASURED: a healthy research turn is ~157s. A throttled one blew the budget with one slow call plus a second, and the executor killed it at 11m43s with no error from the daemon, because nothing had failed yet. The operator got "turn executor failed: turn timed out" and the container holding the reason was torn down minutes later. An hour matches the phase's own budget. A genuinely stuck CALL is still caught at 600s by the daemon and surfaces as a real error; this only stops us killing turns that are working, slowly. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
b2e2735583 |
fix(diag): "turn timed out" now says what the agent was doing
A research_and_code mission failed with: turn executor failed: turn timed out turn executor failed: turn timed out and that is the entire record. Investigating it found: the run produced zero steps and zero output, it died at exactly 700s (TURN_TIMEOUT), the node→claw aliases were bound correctly, and the same zeroclaw team path passes in the `multirole` scenario. So the platform path is fine and the agent simply never finished a turn — but the one place the reason lived, the per-mission runtime container, is torn down after the phase and takes its log with it. By the time anyone looks, all that survives is the string. The timeout now reads the last 40 lines out of that container while it still exists, and reports which agent alias and which gateway it was driving. Best-effort by construction: it runs on a path that is ALREADY failing, so a docker error there degrades to a note rather than replacing the real failure with a second one. `container_name` derives the container from the gateway URL and returns None rather than guessing, because this feeds a diagnostic — a wrong name would put a different container's log under a failure and send the reader somewhere else entirely. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
fe2451fd60 |
feat(workforce): missions hire the agents you already have, and name them by role
Every zeroclaw mission minted a fresh team of claws. They are created
`lifecycle = 'permanent'` and nothing reaps them until the MISSION is deleted,
so the roster grew by a whole team per mission while each member worked exactly
once — "My Workforce" was a list of strangers, and upskilling had nothing
durable to act on.
A mission now hires the claw that already does the job, matched on
`agent_template_link (template_id, role_slot)`, minting only what is missing.
Oldest first, so reuse concentrates on the same few claws and their brains
actually accumulate rather than spreading thinly across a growing pool.
A claw on a RUNNING mission is not offered. Two missions driving the same
ZeroClaw agent and the same `.brain` at once is a data race with a model on the
other end of it, and minting a second claw is much cheaper than reasoning about
that.
A reused claw is NOT re-seeded from the template's brain_seed — that would
overwrite what it learned with its starting point, which is precisely the
accumulation this exists for.
Names are the role now (`planner`), not
`"{mission} · {purpose} · {template} · {slot}"`. That produced
"verify: a repo-less research mission keeps its output · mission · Rust SDLC ·
planner" — unreadable in the roster, the API and every log line at once. Which
mission a claw is on is context a caller can join to; it is not its name.
And the half that makes reuse safe rather than destructive: deleting a mission
now purges only claws no OTHER mission still employs. Without it, tidying up one
mission deletes staff another one holds — presenting as the roster quietly
shrinking rather than as an error. A test asserts the guard exists inside the
reaper AND runs before the purge, because a check after it is decoration.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
dd80b69992 |
fix(microvm): the third place that assumed a VM phase has a repository
The research-vm scenario failed on its first run, and said exactly why:
mission has no checkout at /var/lib/clawmates-missions/<id>/repo
— a microvm phase needs a repository
`phase_runner` refuses upstream of both places the last commit fixed. Three
guards, written independently, all encoding "a microVM phase implies a git
checkout" — which is why the capture filter could cite it as settled fact.
A repo-BACKED mission with no checkout is still a real fault and still refused;
booting a VM to hand the agent an empty directory would turn a setup failure
into a confusing agent report. A repo-LESS one now gets the empty workspace
made here, so the executor's inject has something to pack.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
768e106614 |
fix(microvm): a mission with no repository can run in a VM, and its work comes back
Two halves, and the first was worse than the plan assumed. `run_phase_in_vm`
packed `<missions_root>/<mission>/repo` unconditionally — a directory a
repo-less mission does not have — and then required `/mission/repo/.git` inside
the guest before spending a turn. So a repo-less microVM phase did not merely
go uncaptured: it failed before the agent ran.
A repo-less mission now gets an EMPTY workspace at the same guest path, created
host-side so the collect unpacks back over it with no special case, and the
readiness probe asks for what was actually sent — the directory rather than a
`.git` that was never going to be there.
`mission_outputs` then drops its `runtime_kind <> 'microvm'` exclusion, whose
stated reason ("a microVM mission always has a checkout") is exactly what
stopped being true. Where the files come from now depends on the runtime, and
the difference is not cosmetic: a container mission's output is still inside a
running container, while a VM's has already been unpacked onto the host by the
end-of-turn collect. Asking docker for a VM mission's files would query a
container that never existed.
The recursive copy skips symlinks rather than following them — a link out of
the tree would publish whatever it points at.
`research-vm` is the proof, added to the suite as well as the dispatch: the same
assertions as `research-only` with `runtime_kind: microvm`. A scenario nobody
runs is a scenario that does not exist.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
25f075a8be |
feat(api): polish a description before the mission exists, and download an artifact
Two endpoints the wizard redesign needs.
`POST /api/missions/refine-draft` — the polish button fires while the user is
still typing, before anything is created, so it has no id to route on.
`refine` deliberately requires a saved draft because its Accept writes back;
this one has nothing to write back to and returns the text. Same system prompt,
same model chain. The phase list comes from the workflow recipe rather than the
caller, for the same reason `phases_for_create` prefers it: a client that
guessed would have the model write acceptance criteria for phases the mission
will not run.
`GET /api/missions/{id}/artifacts/{artifact_id}/download` — the file itself.
`artifact_content` caps at 2 MiB and reads as UTF-8, so a large or binary
artifact is unreachable by any means today; this streams the bytes with a
filename attached and no ceiling.
Both artifact routes now resolve through ONE containment check. Two copies of
"is this path under _outputs" is two chances for one of them to be the lenient
one, and the lenient one is an arbitrary read of the gateway's filesystem — so a
test asserts there is a single resolver and that both routes call it.
The download filename was chosen by an AGENT and lands in a header every browser
parses, so quotes, backslashes and control characters are stripped rather than
escaped; the test covers a header-injection attempt.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
f27d2605eb |
fix(agents): a soft-deleted agent could never be purged
Clearing the fleet's four leftover agents returned 404 on every one. They had
been soft-deleted back in June — correctly invisible in the UI ever since — and
`agents::get` filters `deleted_at IS NULL`, so `workspace_agent` could not find
them. Every route uses it, including `batch-delete`, the one that exists to
HARD-purge. So a soft-deleted agent was unreachable from the application
entirely and its row stayed forever.
`get_any` sees them, and only the purge path uses it: hiding soft-deleted rows
is right for every read, and wrong for the one operation whose whole job is
removing them. Written with `query_as` rather than the checked macro so it does
not force an offline-cache regeneration on every machine that builds this.
`fleet-reset.sh` now uses `batch-delete` for agents rather than
`DELETE /api/claws/{id}`. The latter is a SOFT delete, so pointing a reset
script at it would have quietly added to the pile it was meant to clear.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
16cfc29074 |
fix(ui): the backend picker showed two options meaning the same thing
`default` is the generic `rootfs.ext4` and `claude` is the named one, and `microvm_credential_for` gives them the identical contract — so the list came back with both under the same label, and whichever a user picked they got the same thing. Collapsed to the named one where it exists; the generic keeps a label of its own for a fleet that only has that. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
529497febb |
fix(placement): a composed graph needs every backend its nodes name
The full harness found it — 12 of 13 scenarios green, `roster` red:
roster: the planner sized this mission at 2 member(s) PASS
roster: the approved roster is on the mission (2 nodes, composed) PASS
roster: this run added 1 line(s) for a 2-member roster FAIL
topology_runs.error: turn executor failed: node n1 in a microVM:
vm_create failed: no rootfs for backend "canary-claude" on this node
The roster proposed `verifier@canary-claude`. Placement asked
`online_for_backend` about the MISSION's backend — `claude` — and architect
answered, holding `claude` and `local-ornith`. The graph's first node ran and
delivered, the second could not boot, and the mission finished half-done. The
question placement asked was true and insufficient.
A composed graph runs on ONE node, so that node needs every image its nodes ask
for. `required_backends` collects the mission's plus each
`config.roster.nodes[].attrs.backend`, and `online_for_backends` passes the
whole set to the same jsonb `@>` — containment already means "contains ALL of
these", so the query shape did not have to change, only what it was asked.
This is the failure mode the roster feature creates by existing: its entire
purpose is putting a verifier on a different provider, which is exactly what
makes one node insufficient. Nothing before the full suite had a reason to
exercise it — the composed scenario uses one backend for all five nodes.
`NoCapableNode` now names the set and says why one node must hold all of them.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
c66c3c6377 |
feat(ui): the microVM path is reachable from the mission wizard
Everything built today — Firecracker missions, the four backends, the local GPU one — was unreachable from the dashboard. The wizard offered `zeroclaw` and `local_herdr` and nothing else, so a mission created in the UI could not be a microVM mission at all, and `local-ornith`/`glm`/`kimi` were API-only. Testing "our workflows in the UI" would have exercised none of it. Adds the runtime option and a backend picker, fed by a new `GET /api/fleet/backends` that returns `mission_roster::available_backends` verbatim — the SAME list the roster planner is handed, not a second one. Its two rules are both load-bearing and neither is visible from a node's capabilities alone: the image must be built on an online node, and the backend must have a credential contract. `agent-terminal` passes the first and fails the second — bootable, with nothing for the agent inside to authenticate with — so offering it would produce a mission that validates, launches, and dies at the agent turn. Ids are deployment vocabulary, so the picker labels them: a user choosing between `local-ornith` and `canary-claude` should not have to know which company each one bills. An empty list says why (no rootfs built) instead of showing an empty dropdown, and no node is chosen for a microVM mission because `vm_placement` picks it per phase. Co-Authored-By: Claude Opus 5 <[email protected]> |