19c4de36e4459a65a9c96f410f5adf9b550c914a
870
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar SobhandClaude Opus 5
|
d524107b37 |
fix(missions): harvest before the checkout, and stop an unreachable judge failing done work
Two bugs from the first real Continuous Research run, both found by running it. **1. The harvest ran AFTER the checkout.** `on_launch` cloned the vault and then harvested, so the mission's working copy predated the manifest push. The reader agent found no `harvest.jsonl` and — being resourceful — queried arXiv itself and wrote its own. That is exactly what `skills/research/arxiv-daily.md` forbids: the papers it found are not checked off in `corpus_items`, so the next run re-offers them, while the 13 the real harvest DID shelve went unread. The harvest now runs first, so the clone contains the manifest. The analysis it produced was otherwise very good — it named `crates/clawhdf5-ann/src/hnsw.rs`, cited the ROADMAP's serial insert loop and proposed a concrete pre-build probe — which is the behaviour the whole design is for. It was reading the wrong papers. **2. An unreachable judge consumed a pass.** `Verdict.error` exists to distinguish "could not judge" from "judged incomplete" and nothing acted on it. glm-5.3 returned "transport error: error decoding response body", the phase counted it as a failed pass, and with two budgeted that single outage failed a phase whose work was done and committed. The evaluator was right to refuse a same-family fallback — that would trade independence for availability — so the fix belongs here: an unreachable judge no longer spends an iteration. Retrying forever would trade a wrong failure for an invisible hang, so the wait is bounded by `judge_blocked_since` (migration 0078), mirroring how `capacity_blocked_since` bounds a phase waiting on a VM slot. Thirty minutes is many sweep ticks, so a blip recovers inside it; past that the phase FAILS with the transport reason rather than requeueing, because re-running spends a container re-doing work that was never the problem. 346 tests pass. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
a02e0cba69 |
feat(missions): Continuous Research harvests at launch, and cards launch by clicking
The card shipped in
|
||
|
Omar SobhandClaude Opus 5
|
a2d7e3ea92 |
feat(skills): author the Continuous Research skills, and make its roles honest
The boot log has said `continuous_research — 4 role skills bound, 8 unresolved`
on every start. Those eight roles ran without the instructions their template
promised them, silently: `skills_loader` reports the miss and carries on.
Worse than the missing files was what the prompts described. v1 told the
harvester to sweep "RSS feeds, GitHub trending, HN front page, YouTube /
podcast RSS" — none of which exist. `harvest.rs` searches arXiv and nothing
else. A role prompt describing a machine that was never built is the failure
this codebase keeps paying for, so v2 describes the machine that exists.
Roles now match the pipeline: paper_reader (the harvest already ran; read the
manifest and the papers), signal_ranker (unchanged in spirit), script_writer
(the podcast half, which had no role at all).
Seven skills authored under skills/research/, kebab-case to match the loader —
team_template_loader.rs:177-181 documents the snake_case/kebab-case trap that
already unbinds skills elsewhere:
arxiv-daily what the harvest guarantees, so an agent does
NOT re-search arXiv and corrupt the seen-set
paper-to-project-relevance name a file or roadmap item, or say "no bearing"
duplicate-detection the seen-set catches identity; this catches the
same work under a different id
signal-to-noise-ranking novelty/relevance/depth, and the two biases to
resist (recency up, inconvenience down)
executive-summary-writing what it is / why it matters / what to do — decide,
do not hedge
obsidian-vault-conventions the vault is a human's live workspace; never
main, never reorganise, hash the body not the file
podcast-dialogue-writing write for someone on a treadmill; the 10-70 char
highlight bound is the API's, not a style rule
`web_fetch` dropped from mcp_bundles: runtime_provision.rs binds every mission
claw to `["clawmates_door"]` and never reads that field, so declaring it
instructed roles to use a tool that never arrived. The prompts say `curl` via
Bash, which is what they actually have.
Boot now reports `continuous_research — 11 role skills bound`, with no
unresolved clause. 344 tests pass.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
e20b321055 |
feat(missions): Continuous Research is a mission type, not just a team checkbox
`templates/teams/continuous_research.toml` has existed with three well-written roles since it was authored, but no workflow recipe pointed at it — every recipe in templates/workflows/ defaults `default_team_template = "rust_sdlc"`. So the only way to reach it was as a checkbox under Advanced. It is now a Step-1 card: the registry loads it at boot and `GET /api/workflows` serves it, with no frontend change (MissionWizard renders whatever the endpoint returns). Both phases are kind `research`, deliberately, rather than new `read`/`script` kinds. An unrecognised kind falls through `purposes_for`'s `_ => ["mission"]` and is absent from `PRODUCING_KINDS`, so it would get the generic directive AND be exempt from the empty-delivery rule — a phase that produces nothing and still passes. That is the shape this codebase keeps paying for; two `research` phases differentiated by `task` keep both guards. `commit_policy = "always"`, not `on_green_tests`: the vault is prose with no suite, so a test gate would find nothing to run and land every branch `-wip`. The harvest is NOT an agent phase. `continuous_research.rs` calls the existing `library::run_to_vault` — arXiv search, seen-set check, PDF shelf, vault note, attributed by `mission_id` — because that path is deterministic, takes seconds, and owns the `corpus_items` seen-set that is the whole reason a recurring mission knows what it already covered. An agent redoing it would be slower and would lose that. The manifest path is not invented either: the team template has told `signal_harvester` to write `ContinuousResearch/<date>/harvest.jsonl` all along. This makes the code produce what the prompt already promised, and a test pins the path and every documented key so the two cannot drift into an agent reading a file nothing writes. DEFAULT_CORPUS / DEFAULT_VAULT_URL exported rather than duplicated, so the route and the launch hook cannot disagree about which vault. 344 tests pass. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
f87853ecf9 |
fix(missions): scheduled missions never fired — nothing read missions.schedule
The wizard has collected a cron since `0047_missions.sql` ("schedule JSONB
carries the trigger config (cron | one_shot | on_event)"), the frontend posts
`{kind:"cron", cron}`, and the API persists it faithfully. Nothing has ever read
it back: the only due-work enumerator in the codebase was `routines::claim_due`.
So every scheduled mission ever created sat in `draft` forever while the UI
reported it was on a schedule.
Proven before fixing, on the shipped build: a mission with `* * * * *` sat in
`draft` for 4m34s and started ZERO topology runs. After this change the same
mission launched on its next occurrence and recorded one `fired` row.
Two pieces were missing, and they are the two `routines` already had:
- `missions.next_run_at` — schedule STATE. `schedule` is user intent and stays
untouched; without somewhere to record which occurrence is owed there is
nothing to put a `<= now()` predicate on, which is why no enumerator could
be written against the JSONB alone.
- `mission_fires` — one row per (mission, occurrence). 0063_routine_fires.sql
called this exact case: "For a scheduled *mission* it costs a container, a
repo checkout, and real money — which is why this lands before mission
scheduling does."
`mission_schedule.rs` deliberately mirrors `cm-scheduler`'s shape rather than
inventing a second one: atomic `FOR UPDATE SKIP LOCKED` claim, reschedule
BEFORE dispatch so a failing launch cannot stall the clock, claim the slot
before launching so a crash mid-launch is retried rather than dropped, and a
fan-out cap. The cap is 5, not the scheduler's 25, because a mission firing is
a container and a checkout where a routine firing may be one turn.
The claim skips `status = 'running'`: a daily cron on a mission that takes
longer than a day must skip the occurrence, not stack a second crew on the same
workspace. Launch goes through `mission_orchestrator::on_launch` +
`missions::set_status`, the same path as the draft→running transition, so one
code path mints a crew. An unattended launch acts as the workspace owner
(`users::owner_of_workspace`) since missions carry no creator column; a
workspace without one settles the occurrence `failed` with the reason rather
than dropping it silently.
Backfill blast radius was MEASURED, not assumed: prod has zero missions with a
cron, this workstation had exactly one — the control created to prove the bug.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
3cc65c22c4 |
feat(judge): room to analyse — and a panic in the evidence path
Three changes, one of them a live bug. **The bug.** `phase_summarizer` truncated agent output with `&s[..remaining]`, a BYTE slice of arbitrary UTF-8. Agent turn output routinely carries arrows, box-drawing and emoji, so a cut landing mid-character panics — taking down the evaluation sweep for that phase, triggered by nothing more than an agent writing a long enough line with a non-ASCII character at the wrong offset. Replaced with `clamp_to_char_boundary`, tested across every cut offset of a pure-4-byte string. It is precisely the bug the clawhdf5 agents found and fixed in `clawhdf5-migrate/src/validate.rs` this week — in our own code, in the path that feeds the judge. **Evidence budget** 60 KB -> 120 KB. Output headroom is worthless if the judge cannot see the work: the verdict is only as good as what reaches it. **Judge max_tokens** 2048 -> 16384. glm-5.3 is a reasoning model that spends most of its budget on a `thinking` block before writing the verdict, and running out mid-thought truncates it. A truncated verdict parses as empty and FAILS CLOSED, burning one of the phase's passes on a judge that never answered — how mission 01a00bbb lost one. Measured ceiling: z.ai accepts max_tokens up to 131072 on both glm-5.1 and glm-5.3 (131073 -> 400, "限制数值范围[1,131072]"), so 16384 is chosen for cost and latency rather than capability, and only emitted tokens are billed. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
9c4b0722e8 |
feat(judge): an independent GLM judge, on the newest model z.ai publishes
Every phase verdict this session was Anthropic grading Anthropic, and the boot
log said so on each start:
validator_preflight: no CLAWMATES_VALIDATOR_MODEL — phase verdicts are
judged by the house model, which is NOT an independent check
`evaluator.rs` already preferred a cross-provider judge and refused to call a
same-family one `independent`; the local stack simply had no non-Anthropic
credential. It now carries the same `glm` provider gw-04 has had all along —
`format = "anthropic"` is load-bearing, since z.ai's OpenAI-compatible endpoint
is ToS-throttled for raw SDK access while its Anthropic-format one is not.
Model: glm-5.3, the newest z.ai lists (4.5, 4.5-air, 4.6, 4.7, 5, 5-turbo,
5.1, 5.2, 5.3 as of 2026-08-17). gw-04 still runs glm-4.7.
glm-5.3 is a REASONING model: it emits a `thinking` block before its JSON. Our
SSE parser ignores `thinking_delta` and keeps the text, so the wire shape is
compatible — but on a realistic phase-evidence prompt it spent 819 of the
evaluator's 1024 output tokens. A longer phase would truncate the verdict, and
a truncated verdict parses as empty and FAILS CLOSED, burning one of the
phase's passes on a judge that never answered — precisely how mission 01a00bbb
lost a pass. max_tokens raised to 2048.
Measured before wiring: asked to judge 25 commits claiming INT-01..INT-25 with
tests passing, glm-5.3 returned met=false because the evidence never
established what the brief actually required. That skepticism is the point of
an independent judge.
Boot now reports: `validator_preflight: independent validator glm:glm-5.3
answered`.
The key lives in .env (gitignored), never in this file.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
b5032a732a |
fix(phase_runner): collect the agent's work BEFORE judging it
`Sandbox::for_mission` builds the judge's verification copy from the HOST
checkout. In copy mode the agents write inside the container, and their work
only reached the host when `sync_out` ran — in the capture sweep, AFTER the
phase closed. So every phase was judged against a tree that did not yet contain
the pass being judged, and the judge truthfully reported nothing there.
Mission 01a00cfa is the proof. Research pass 2 wrote a 434-line
IMPLEMENTATION_BRIEF.md, `cargo test` passed, and it was pushed to a clean
branch (clawmates/mission-01a00cfa-c69f39fd-i2 at 563cdd21). Its verdict:
failed after 2 pass(es) — met=false — research/IMPLEMENTATION_BRIEF.md
does not exist anywhere
logged one line BEFORE `captured (+434/-0 across 1 file(s))`. A phase that
succeeded was failed because the evidence had not been collected yet.
This hid because it only bites a phase judged on its OWN pass. The v2 coding
verdict cited real commits (339a5bd, 167671f) — research had already synced
that work to the host in an earlier phase.
`evaluate_finished_phases` now runs `sync_out` first, and on failure leaves the
phase `evaluating` for the next sweep rather than recording a verdict nobody
could stand behind — the same policy the capture sweep already applies, for the
same reason. microVM keeps its carve-out: `microvm_executor` collects out of
the guest over this same path before the VM is destroyed.
Research goes to 3 passes. On 01a00cfa it got no real attempts out of two: one
spent on a fabricated commit claim the judge correctly rejected, one on this
bug.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
a582dea4fc |
feat(workflow): research plans, coding builds — the split was a fiction
`rust_sdlc` gives the research team coding roles and a writable /mission/repo, so research implemented what it found and the coding phase then opened a clean tree, produced +0/-0 and failed on the empty-delivery rule. Mission 01a00c57 ended exactly that way: research shipped both INT items itself (+276/-57), coding delivered nothing. Worse than the wasted phase is WHERE the code landed. Research ran under a gate that does not check tests, so its two source changes reached a branch with `tests_status: null` — never compiled by the gate, never run. Keeping implementation in the coding phase is what puts it behind `on_green_tests`. Research now carries a `task` that scopes it to the brief and says plainly that editing crates/ is not its job this phase, plus a `done_when` describing what the brief must CONTAIN. The no-source-edits constraint deliberately lives in the prose and NOT in `done_when`: "and nothing else" phrasings measurably make a judge invent requirements it was never given. Coding gets the counterpart `task`: implement the brief's items, one commit each, tests green. Stated explicitly because a phase that finds a clean tree and no instructions has historically written a REPORT about the work instead of doing it — four documentation commits and one implementation, on the run that became the haiku baseline branch. Research also gets `commit_policy = "on_green_tests"` as a safety net, so source it writes anyway still has to pass the suite. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
d341640255 |
fix(mission_fs): drop build output when collecting work back from a container
`pack_dir` (host -> container) skips `transport_excludes`; `copy_out`
(container -> host) is the raw Docker archive API and carries the whole tree,
`target/` included. The asymmetry was invisible for as long as the runtime
image had no cmake — nothing could compile, so no `target/` existed.
The moment missions could actually build, every collection died on a build
artifact:
failed to unpack `…/repo/target/debug/build/ahash-…/build_script_build-…`
`phase_runner` then correctly refused to capture, rather than record a stale
tree as an empty diff — so mission 01a00c57's coding phase, which had done the
work, delivered nothing and retried forever. A fix that let missions compile
created a delivery failure one layer down.
`unpack_into` now skips excluded entries by NAME at any depth (a workspace has
a `target/` per crate) and logs how many it dropped.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
99dd29cc8a |
fix(worker): the stuck-run reaper was killing healthy sonnet-5 turns
REAP_STUCK_AFTER_SECS was 15 minutes; the runtime grants a single turn `timeout_secs = 3000` (50 minutes). A run journals its first step record when its first step COMPLETES, so a turn still legitimately in flight is indistinguishable from a wedged container — and with a window shorter than the turn timeout the reaper does not detect stuck runs, it kills slow healthy ones. The old value was calibrated on haiku, where "healthy first-step latency is typically 5-60s" held. Moving mission agents to sonnet-5 made first turns longer than the window: mission 01a00c41's research phase was reaped at 900s having already written +402/-39 across 13 files. We only know it was healthy because the delivery path captured and pushed that work anyway, to branch clawmates/mission-01a00c41-421200ee at b08df7b6. Raised to 60 minutes, above the turn timeout, with the invariant written down so the next person changing either number sees the relationship. Generalises: a liveness timeout calibrated against one model becomes a correctness bug when the model changes. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
b31a79f650 |
fix(llm): the subscription 429s were a malformed request, not a rate limit
On the OAuth path Anthropic requires the Claude Code identity to be its own
first system BLOCK. We concatenated it with the caller's prompt into a single
string, so EVERY server-side call that set a system prompt was rejected — and
the rejection arrives as `429 {"type":"rate_limit_error","message":"Error"}`,
which reads as throttling and is not.
Measured on one token, seconds apart:
"PREAMBLE" (string) -> 200
"PREAMBLE\n\nJudge the …" (string) -> 429
"PREAMBLE" (string) -> 200 (control)
["PREAMBLE"] (blocks) -> 200
["PREAMBLE", "Judge the …"] (blocks) -> 200
while the account reported `5h utilization 0.07, 7d 0.11, overage 0.0`, every
window `allowed`. A Max 20x subscription at 7% was being read as out of
capacity.
What this was breaking, silently, for as long as it has been there:
- every `done_when` verdict on the subscription judge. Mission 01a00bbb
pass 2 returned "could not evaluate the completion condition this pass"
and BURNED one of the phase's three passes on it.
- the boot preflight, which reported `claude-opus-4-8 throttled (configured,
no capacity now)` on every start — a diagnostic that was itself the bug.
- mission_refiner, phase_summarizer, swarm planning.
The `claude` CLI was unaffected throughout, because it sends its system prompt
as blocks. That divergence is what made this look like an account problem: the
agents worked while everything server-side "throttled".
After the fix the preflight reports opus-5, sonnet-5 and haiku all `ok`.
The API-key path keeps sending a plain string — it never had this constraint.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
69c294addc |
fix(models): coding runs on sonnet-5, judging on opus-5, haiku only as last resort
Operator model policy: haiku ONLY for genuine yes/no questions; anything
requiring thinking is opus-5; coding is sonnet-5.
The mission AGENTS were running haiku, and nothing in the product said so.
`provider_alias_for` maps every `claude-*` binding onto the single alias
`claude_cli.default`, so a crew whose `model_binding` reads `claude-sonnet-5`
— as this deployment's does — still ran whatever that alias pointed at, which
was `model = "haiku"` in the runtime config. The binding is cosmetic; the
alias is the truth.
Measured consequence on mission 01a00bbb: the coding agents claimed six INT
items complete and had committed three, and the done_when judge caught it by
auditing git history against the claims.
Model assignments, by what the component actually does:
evaluator (done_when judge) haiku -> opus-5 reads evidence, audits it
against the repo, writes
guidance. The verdict is a
boolean; the work is not —
and this is the one component
whose failure mode is passing
work that was never done.
judge_model 4-8 -> opus-5
mission_refiner 4-8 -> opus-5 composition
phase_summarizer 4-8 -> opus-5 composition
swarm planner 4-8 -> opus-5 planning
subscription preflight head 4-8 -> opus-5
fallback chain head 4-6 -> sonnet-5 haiku stays BELOW it as a
last-resort link, never a peer
Every value stays env-overridable; only the shipped defaults move.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
53da4d7e6d |
fix(runtime): a mission could not build the repo it was given
`clawmates-runtime` shipped with `gcc` and `make` but no `cmake`, no `g++` and no `python3-dev`. Measured on clawhdf5, three probes: no cmake → "is `cmake` not installed?" exit 101 after 13s no python3-dev → "cannot find -lpython3.11" exit 101 at link with both → cargo test PASSES exit 0 after 69s This is not only the delivery gate. The AGENTS run in this image, so a coding phase was writing Rust it had no way to compile or test — which reframes the last run's 11 agent commits as unverifiable by construction. `images/agent-toolchain/Dockerfile` (the microVM path) has had `cmake build-essential` all along, and its own header warns about precisely this: "if `cargo` is present in one image and absent in another, the same mission passes or fails depending on which backend it landed on, and nothing says why." Both images now install the same set — it was missing `python3-dev` too. `images/runtime-toolchain.Dockerfile` is a thin local overlay so the laptop can run today without recompiling zeroclaw from the fork; it is meant to be deleted once a runtime image built from the corrected deploy/ Dockerfile is published. Also: a build failure is no longer reported as a red suite. Both are cargo exit 101, and `verify_tests` mapped every non-zero to `Failed(code)` — so a missing toolchain was recorded as the USER's tests failing. It now returns `CouldNotRun` with the reason when the output shows a compile or link failure. Deliberately narrow: a failing `assert!` still reads as red, because letting broken code past `on_green_tests` is the expensive direction to be wrong in. Both directions are pinned by tests built from today's two real samples. And the coding phase finally has a loop: `research_and_code.toml` declared `loop = "until_no_more_int_items"`, which `phase_config.rs` lists as DECLARED_BUT_UNREAD. Iteration is driven by `max_iterations` + `done_when`, and with `max_iterations = 1` and no `done_when` the phase ran ONCE and was never judged — reporting `completed` whatever it produced. Now 3 passes against a stated goal, wording per the measured rule (say what the tree must CONTAIN). Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
10341cf7fe |
fix(missions): a retry's work is no longer silently destroyed
Two independent bugs, either of which loses everything a retried phase
produced, and neither of which reports a failure.
1. Capture is suppressed forever on a retry. Both
`capture_finished_coding_phases` and the sweeper's last-chance
`capture_outstanding_phases` skip any phase that already has a
`code_diff` artifact. That guard is right for a phase that ran once and
catastrophic for a retried one: the artifact from the FAILED attempt
suppresses capture of the new attempt, the container is reaped on its
normal grace, and everything the agents committed inside it is gone.
The UI keeps showing the old diff, so the mission reads as delivered.
`retry_phase` now clears the reopened phases' captures in the same
transaction that reopens them, which is what makes its own doc comment
("the phase card starts fresh on the retry") true of the artifacts too.
2. `git add` exits non-zero over a gitignored path while staging correctly.
Measured: with a populated `target/`, `git add -- . :(exclude)target`
exits 1 and stages the right files; `-c advice.addIgnoredFile=false`,
`--ignore-errors`, `-A` and `:/` all behave identically. Propagating
that with `?` aborted the commit AFTER a successful staging — no branch,
no commit, no push — for every Rust repo an agent has built in.
`capture_phase_diff_at` already treats the same command as advisory;
the commit path now does too, and the staged index decides.
Mission 01a00538 hit both: it completed research and coding on the retry,
11 agent commits and all, delivered a patch dated the previous day, and
lost the commits when the container was reaped. The remote was never
touched — its HEAD still equalled the mission's own base_sha.
Covered by a test that drives real git and asserts the files are staged
regardless of the exit code.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
fd5e71ccfe |
fix(missions): re-assert a mission's crew when its container is recreated
Claws are provisioned exactly once, at on_launch. `ensure_container`
RECREATES a container that is not running, and recreation reseeds
.zeroclaw from the seed directory — which does not hold this mission's
claws. The agents still exist in Postgres and the crew query looks
perfect, so nothing reads as broken; the alias is simply gone from the
daemon and /ws/chat answers 400 Bad Request. A retried mission could
therefore never connect again.
Re-assert the crew after ensure_container. provision_claw is idempotent,
so this costs one call per claw on the happy path and is the difference
between a resumable mission and a dead one.
Two things this has to get right, both of which fail silently:
- Aim at the per-mission daemon via for_gateway(ec.endpoint), never
from_env() — that targets the shared global gateway and leaves this
container with no claws at all, exactly as for_gateway's own doc
comment warns.
- Provisioning creates the agent but cannot set workspace.path (the
config prop-schema has no way to express it), so follow with
pin_agent_workspaces or every claw runs in its own sandbox and
delivers nothing.
Verified on mission 01a00538: research and coding phases both completed
after four straight failures, with all five graph aliases present and
ten claws pinned to /mission/repo.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
85a6038c08 |
fix(missions): create /mission before copying the checkout in
`copy_in` cannot create its own destination, so a mission whose container had no /mission directory failed its checkout sync outright. In copy mode that is how the agent gets the code at all, so the phase launched against an empty tree. Exec `mkdir -p /mission` as root first. Idempotent, and it costs one exec on a path that already shells out. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
6e8785f159 |
fix(missions): a server restart no longer kills a running mission
Mission 01a00538 ("ClawHDF5 REsearch and Refactor") failed 19 minutes and 93,762
tokens into its research phase with `pair failed: 403 Forbidden`, and its coding
phase was then correctly skipped as unreachable. The cause was not the coding
phase and not the model — it was pairing.
A per-mission runtime is authenticated with a SINGLE-USE pairing code, and the
bearer token it returns was cached in memory only. Any restart of the server
discarded that token; the next turn re-paired with a code the gateway had
already spent and got 403 — permanently, for that mission. A deploy, a crash or
an OOM would each do it. The durable-run machinery exists precisely so work
survives a restart; pairing was the one thread that did not, and it failed
closed.
`missions.runtime_token` persists the token at the moment pairing succeeds, and
the worker seeds the executor's cache from it, so a new process reuses the
credential instead of re-pairing. Persisting is best-effort: failing to save
must not fail a turn that just paired successfully.
Verified by reproducing the original failure: launched a mission, confirmed the
token was written, restarted the server MID-PHASE, and watched the mission run
to completion with no pairing failure.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
43436d7181 |
feat(telemetry): push bus for live agent frames
/api/world/live is a 2s database poll, which is right for queryable state and wrong for a token stream: reasoning only became visible after a step finished and its row was written. This adds a process-wide broadcast bus that topology_exec publishes to as the runtime's WebSocket delivers frames, and the SSE handler forwards without waiting for the next tick. Measured: the pushed frame arrived ~2.2s before the polled copy of the same text. Design notes worth keeping: - A global (OnceLock), not an AppState field. The publisher is reached through phase_runner -> topology_worker -> MissionTap, none of which hold AppState; threading a handle through all of them would put a UI concern into four layers that have no other reason to know about one. - Lossy by design. A slow subscriber lags and skips rather than applying backpressure to the agent producing. mission_events remains the durable record; this bus is the fast path, never the source of truth. - Only `claw_<uuid>` aliases are attributed. The governor, door and evaluator drive real turns under other names, and attributing their output to an agent would put words in someone's mouth. Asserted in a test. - The poll no longer emits `reasoning`: with both paths live, every turn arrived TWICE — once pushed, once polled ~2s later. The row is still written; this feed just is not its second mouth. CEILING, measured rather than assumed: turns are not token-level because the runtime is not streaming. zeroclaw's claude_cli provider runs `claude -p --output-format json`, which returns ONE result object when the turn completes — there are no incremental tokens to forward. Making this genuinely token-by-token needs `--output-format stream-json` and incremental parsing in the zeroclaw fork, not here. The bus is in place and will carry them the day it does. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
ba9d7aa185 |
feat(telemetry): WORKING ON NOW shows the mission an agent is on
The last of the three declared-but-never-emitted event types. `agent.task.update` had no producer anywhere in the backend, so the card read "idle — no active task" for an agent that was mid-turn. Derived rather than newly instrumented: an agent is working on its crew's RUNNING mission, and that mission's phases are the steps (completed/skipped → done, running/evaluating → active, else pending). Nothing is emitted for an agent with no running mission, so "idle" stays truthful rather than freezing on a stale last-known task. Verified on a live mission: 116 agent.task.update events observed on /api/world/live, carrying the mission title and phase steps, with the state advancing pending → active as the phase started. That closes the set. Of the seven cards in the command centre, five were dark: three had no emitter at all and two read a table the mission path never wrote. DOORS and LOOPS were correctly wired the whole time and were reporting an honest zero. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
bf40d10064 |
feat(telemetry): the reasoning stream actually streams
`agent.reasoning.delta` and `agent.tool.call` have been declared in the taxonomy and listened for by the command centre since it shipped — and NOTHING ever emitted them. The world feed emitted five types; neither was among them, so REASONING STREAM could not populate no matter what an agent did. The feed is a database poll, not a push bus, so a live card can only show what was persisted. The worker already holds each step's output text and the claw that produced it, so it records a `reasoning` mission_event (truncated — the card renders a tail, not a transcript, and mission_events is capped per phase), and the feed emits it forward from a cursor that starts at the current max so a page load streams rather than replaying history. `tool.call` is emitted from the same place. On the container tier it will stay empty, and that is correct rather than broken: those agents are tool-free behind the §15 door. Tool lines appear where agents actually hold tools. Verified on a live mission: agent.reasoning.delta observed on /api/world/live carrying the agent's own text, keyed by agentId. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
8be7b3c9b2 |
feat(telemetry): record per-agent usage for mission turns
The command centre's SPEND, ACTIVITY and THROUGHPUT cards read `usage_events`, and nothing on the mission path ever wrote a row: `cm_billing::charge` was called only from the agent-run path. Measured mid-mission with 14 agents live, `usage_events` was 0 while a crew had just burned 15k tokens — so an agent that had done real work reported zero cost and zero activity. The worker already knew everything needed: it logs node, role and token count per step, and the node's `attrs.agent` carries the `claw_<uuid>` binding the runtime dispatches on. This routes that to the ledger. `charge`'s run_id is now Option. `usage_events.run_id` references `agent_runs`, and a topology turn has no row there — passing its `topology_runs` id was a foreign-key violation, which is exactly what the first attempt hit. NULL is the honest value; the agent-run caller still passes its real id. The executor reports one total rather than an in/out split, so the cost is right (credits price the sum) and the columns record it as output rather than inventing a split. Verified end to end on a real mission: 4 agents, 1046-8670 tokens each, credits attributed per agent, and the SPEND/ACTIVITY queries now return real numbers. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
8ef7067467 |
fix(repos): a scoped connection can name a user, not just an org
Scoping a Gitea connection to `osobh` — the personal namespace clawmates itself
lives in — failed with "org 'osobh' not found or PAT lacks access". The sync
only ever called /orgs/{owner}/repos, and Gitea serves user namespaces from
/users/{owner}/repos. The error pointed at permissions for what was really a
wrong endpoint, which is the kind of message that sends you to rotate a token
that was fine.
Retry as a user on 404 before giving up, and say what was actually checked.
Verified: owner=osobh now syncs 7 repos, owner=redclaw 22 — 29 instead of the
182 an unscoped connection pulls.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
b290025fc4 |
feat(agents): make delete permanent, and expose the census
A soft delete marked the row and left it. The agent stayed in the table forever,
kept appearing on any surface that forgot `deleted_at IS NULL`, and deleting it
again did nothing — the decision was recorded and never honoured. Two agents on
this deployment had been in that state since June.
`deleted` is now a fifth lifecycle state, collected with NO grace window: a
human already decided, months ago. It takes usage_events with it, which is the
explicit trade — the alternative is rows that outlive the decision to delete
them.
Two endpoints, because this was previously only answerable by reading the
database by hand:
GET /api/claws/lifecycle the census: who is active, completed,
orphaned, deleted — and what is reapable
POST /api/claws/lifecycle/sweep run the reap now, rather than waiting out
the hourly timer for a decision already made
Verified end to end: census reported both as `deleted`/`reapable`, the sweep
returned {"reaped":2,"failed":0}, and agents and usage_events both went to 0.
The safety property is unchanged and re-asserted by a new test: adding `deleted`
did not make `owned` or `active` reapable.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
ccbc387f4b |
fix(agents): stop listing soft-deleted agents
Deleting an agent looked like a no-op: it disappeared from the workforce but
stayed on the Team board, and deleting it again did nothing because the row was
already marked. Two queries selected from `agents` without `deleted_at IS NULL`:
routes/team.rs the leaderboard — the surface still showing them
routes/world.rs the "working" set — a deleted agent holding a stale
agent_containers row rendered as live
Observed on this deployment: /api/workforce correctly returned nothing while
/api/team/leaderboard returned two agents soft-deleted back in June.
NOT changed: those rows still exist. Making delete permanent means hard_purge,
which also deletes usage_events — billing history, 6 credits on one of these
two. Discarding that as a side effect of tidying a roster is an explicit
decision, not something a display fix should smuggle in.
.sqlx regenerated: team.rs uses the compile-time-checked query! macro, so the
cached entry no longer matched.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
a494634f81 |
feat(agents): classify agents by lifecycle and reap the finished and orphaned
A mission mints a crew, and the only thing that reaped one was DELETING the mission. A mission that merely completed left its agents in the roster forever, and a crew whose reap was skipped or failed left agents bound to nothing — indistinguishable in the UI from the operator's own staff. Four states, from one query: owned no agent_template_link row → hand-created. NEVER reaped. active on a running/draft mission → working right now. Kept. completed every mission terminal → reaped after a 24h grace. orphaned minted, bound to nothing → reaped. The discriminator is `agent_template_link`, which mission_orchestrator writes per minted claw. This matters more than it looks: verified on live data, a hand-created agent and an orphaned crew member both have ZERO team links and are structurally identical by binding alone. Judging orphanhood by "no team" would delete the user's workforce. Provenance is the only honest signal. The grace window exists because the results view, the World's 24h replay and "who did this work?" all read the crew AFTER the run ends; reaping on the terminal transition deletes the answer exactly when the question gets asked. A completed crew with no usable timestamp is KEPT — a missing date must never read as "old enough to delete". Also fixes the delete summary, which reported how many claws were FOUND rather than purged: "reaped 4 claw(s)" was printed by a delete that purged none, which is precisely the log you would read while wondering why the agents are still there. It now reports purged / kept / FAILED, and failed > 0 is the orphan case. Verified against live data — all four states observed, including the two that look alike. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
eb120a10dd |
perf(image): drop chromium from the server image — 875 MB to 212 MB
Chromium and fonts-liberation were 758 MB of an 875 MB image: 87% of the server image was a browser it never launched. It was installed for the Slice 6 mission PDF renderer, which no longer exists — every call site passes `render_pdf: false` because markdown is the deliverable — and NOTHING in the workspace reads the CHROMIUM_BIN this image set. The only Chromium the platform actually uses is `browser.goto`, which runs it inside the agent's dedicated egress-enabled BROWSER container (cm-runtime/src/tools/browser.rs), never in the server. Measured on gw-04: 875 MB -> 212 MB. The remainder is debian-slim (75 MB), git and its dependencies (~95 MB) and the server binary (42 MB). git stays: research topic clones shell out to it, which is why this image left distroless in the first place. Verified in the slimmed image: git 2.39.5 present, CA bundle present, chromium absent, binary executable, templates and all 8 skills shipped. That 663 MB was paid on every deploy, every registry push, and every air-gapped bundle. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
4d07868410 |
ci: stop leaking a 2.8 GB postgres volume every run
`docker rm -f` without -v orphans the anonymous volume the postgres image declares. cm-testkit creates a database per test, so each CI run left ~2.8 GB behind: 38 GB of dangling volumes had accumulated on gw-04, most of the 99 GB -> 23 GB drop in free space over one day. Note for anyone reaching for `docker volume prune` to clean this up: don't. On gw-04 the dangling set also contained traefik-acme (Let's Encrypt certificates) and all three CI cargo caches. Only the anonymous 64-hex volumes were safe to remove. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
25a3d6902a |
ci: only publish a release for an actual tag
On workflow_dispatch GITHUB_REF_NAME is the BRANCH, so the upload step created a Gitea release AND a git tag both named "main" — a tag sharing the branch name, from a run that was only meant to be a smoke test. Both have been deleted. Gated on github.ref_type == 'tag'. A dispatch now exercises build, SBOM, sign and offline verify, and stops there. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
837a3d3ff0 |
ci: don't run the install rehearsal on the production gateway
Every other step in the release job is inert with respect to prod — build, SBOM, sign, offline verify. The rehearsal is the only one whose purpose is to stand a full stack up and tear it down with `down -v`, and it was doing that on the machine serving production. On 2026-08-13 it adopted the live compose project and destroyed clawmates_pgdata. The script itself is now safe (unique -p, a guard against the production project name, and a health probe pointing at the port the bundle actually publishes) and is kept for use on a build box or throwaway VM. What changes here is only WHERE it runs, which was the real problem: a destructive verification step does not belong on the host it can destroy. Releases still build, sign, verify offline in a --network none container, and upload to Gitea. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
875ff948f8 |
fix(rehearsal): probe the port the bundle actually publishes
The health check polled 127.0.0.1:18080, but deploy/compose/docker-compose.yml publishes "8080:8080" and deploy/airgapped/install.sh does not rewrite ports. Nothing was ever listening on 18080, so the rehearsal always ended in "platform never became healthy" — regardless of whether the install worked. Visible now only because the earlier failures (no cargo, compose v1, project collision) all stopped the script before it got this far. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
e7d2fc9696 |
fix(rehearsal): never adopt the production compose project
INCIDENT: the release rehearsal destroyed production data on gw-04. deploy/compose/docker-compose.yml declares `name: clawmates` at the top level, and that beats --project-directory. So `compose up` from a temp directory did not create an isolated stack — it ADOPTED the running production stack of the same name, recreated its containers, and then the cleanup trap's `down -v` deleted its volumes, including clawmates_pgdata. Prod came back with an empty database: 177 repos, all missions and all agents gone. There were no backups. The fix is `-p rehearse-$$` on every invocation, plus an assertion that refuses to run under the production project name. Isolation here was implicit and therefore not isolation at all. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
41854c70e1 |
ci: the install rehearsal needs compose v2, and says so
The v1 fallback I added a commit ago cannot work: deploy/compose/docker-compose.yml
uses v2-only syntax — a top-level `name:` and long-form
`env_file: {path, required}` — so docker-compose 1.29 rejects the file outright
("'name' does not match any of the regexes"). A fallback that always fails is
worse than no fallback, so the script now requires v2 and fails immediately with
what to do about it.
$COMPOSE overrides the detection. gw-04 is deliberately left WITHOUT a
`docker compose` plugin: installing one system-wide would flip the production
rolling deploy (clawmates-deploy.sh prefers v2 when present) off docker-compose
v1 as an invisible side effect of a release change. The runner gets a standalone
v2 binary at /opt/act-runner/bin/docker-compose and the workflow passes it in,
so prod keeps rolling exactly as it did.
Verified on gw-04: standalone v2.32.4 runs, and `docker compose` still resolves
to nothing, so clawmates-deploy.sh takes its v1 branch unchanged.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
9441cf401c |
ci: make the install rehearsal work with compose v1
The rehearsal reached "First boot" — bundle assembled, signed, verified offline, images loaded, install staged — and then died with `unknown flag: --project-directory`. That message is misleading: gw-04 has no docker compose v2 plugin at all, only docker-compose 1.29.2, so `docker compose` is parsed as `docker` with a bogus flag rather than reported as a missing plugin. Use the same v2-then-v1 fallback deploy/gw-04/clawmates-deploy.sh already needs. v1.29.2 supports --project-directory, so the invocations are otherwise unchanged. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
bd1c970577 |
ci: let the install rehearsal use a pre-built bundler
The rehearsal hardcoded `cargo build -p clawmates-bundler`, so it died with "cargo: command not found" on the release runner — gw-04 builds Rust inside a container and has no toolchain of its own. The release job had already built the bundler two steps earlier, so it was also redundant work. CLAWMATES_BUNDLER now short-circuits that build when it points at an executable, falling back to cargo otherwise, so running the script by hand is unchanged. Everything before this step already passed on the runner: images built, SBOMs generated, bundle assembled and signed, and "bundle OK: 94 artifacts verified offline" inside a --network none container. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
c1642a7004 |
ci: copy the bundler out of the target volume
First dispatch failed at exit 127, "target/release/clawmates-bundler: No such file or directory". The bundler builds inside a container where /w/target is a NAMED VOLUME, so the binary was written somewhere no later host step can see — the workspace's target/ stays empty. Copy it to .tools/ (bind-mounted) and assert it landed, so the next occurrence fails at the build step with a clear message instead of two steps later as a missing file. deploy.yml does not hit this because it copies clawmates-node into frontend/public/dl/ from inside the same container. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
de8736c16b |
ci: move release.yml to Gitea and make it actually runnable
It could never have run as written: `runs-on: ubuntu-latest` matches no runner on this forge, and `softprops/action-gh-release` talks to GitHub's API. There are zero tags and zero releases, which is consistent with it never having fired. Rewritten for this runner: - runs-on: gw04 (the only reachable x86_64 host; prod artifacts must be amd64) - the bundler builds in a rust container with the shared cargo cache volumes — gw-04 has no cargo, and installing a toolchain onto the production gateway to build a release is the wrong trade - release creation + asset upload go to Gitea's own API, create-or-reuse so a re-run of a tag updates rather than 409s - syft installs into the workspace, not /usr/local/bin: the host executor runs as root on the gateway and a release should leave nothing behind - a disk-reclaim step, because the artifacts are GBs of image tarballs on a box that is also serving production. It removes only the versioned images it created — never a blanket prune, since clawmates/agent-*:dev exist in no registry and are the source of the microVM rootfs files - workflow_dispatch added so the pipeline can be exercised without minting a tag BUNDLE_SIGNING_KEY now exists as a repo secret (fresh ed25519 keypair; nothing depended on a previous one). The signing and offline-verify steps are unchanged: verification still runs inside a --network none container, which is the whole air-gapped contract. .github/ is now empty and removed. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
26e571fe01 |
ci: drop .github/workflows/ci.yml
It targets `runs-on: ubuntu-latest`, which no runner on this Gitea provides, so
every push left a failed job in the Actions tab. .gitea/workflows/deploy.yml now
covers the gates that actually hold: the full `cargo test --workspace` (including
the DB- and docker-backed integration suites, which this workflow never ran) plus
frontend typecheck and tests.
What is deliberately NOT carried over, because none of it passes today and
silently keeping a red gate is worse than removing it:
cargo fmt --all --check 63 files drift
clippy -D warnings pre-existing warnings across the workspace
ci/check-loc.sh MissionWizard.tsx 1153 lines vs a 1100 soft limit
ci/check-no-placeholders.sh false positive on `vec!["rg", "TODO", "src"]`,
which is test DATA, not a placeholder
playwright e2e needs a browser toolchain on the runner
Re-adopting any of these is a cleanup project, not a workflow edit. The scripts
under ci/ are kept so that work has somewhere to start.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
548f977212 |
style: rustfmt the four files the repo-less mission fix touched
Found while removing .github/workflows/ci.yml: three of the four files in that change were unformatted, and four of the diffs were newly introduced (the new prompt tests and the tool_preamble format! call). Formatting only the files that change already touched — a repo-wide `cargo fmt` would be 63 files of unrelated churn and belongs in its own commit. Mechanical; `cargo test -p cm-api --lib` stays at 322 passed. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
022ef98e44 |
feat(auth): opt-in local auto-login for single-user deployments
Skips the login form and lands on the dashboard. It performs a REAL backend login — the API still issues and can revoke the session — so this does not weaken auth; it only removes a form for a deployment with exactly one operator. Gated on BOTH LOCAL_AUTOLOGIN_EMAIL and LOCAL_AUTOLOGIN_PASSWORD, and refuses outright in clerk mode. Prod sets neither, so the route 404s there. Two conditions rather than one flag: a single misread value should not be able to hand a session to an anonymous visitor. The route emits a RELATIVE Location — inside the container request.url is the 0.0.0.0:3000 bind, so NextResponse.redirect would send the browser to a host that only exists in Docker — and the cookie's secure flag keys on x-forwarded-proto rather than NODE_ENV. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
7cf77a9248 |
feat(ui): level up moves to the agents sidebar; repos open collapsed
Level up now sits at the bottom of the Agents sidebar, labelled with the selected agent's name, and renders only once an agent is selected — it is hidden during select mode so the reap bar stays the single footer action there. It is gone from the ClawCommandCenter header. Repos open with every org folded. Rather than seeding a "collapsed" set with all keys on load, the state tracks EXPANDED: a smaller change that also stays correct for orgs that arrive later from a sync, which a seeded set would render open. With 182 repos across 8 orgs, an all-expanded default buried the org names the list is meant to be navigated by. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
5db695460f |
fix(missions-ui): the results area could not scroll at all
The canvas host is a position:relative BLOCK, so flex:1 on MissionCanvas's root was inert and its height collapsed to its content. That starved the scroller beneath it — scrollHeight === clientHeight — so it never scrolled, and the overflow spilled past the page and was clipped by the host's overflow:hidden. Long results were rendered and then thrown away. Every sibling canvas already used position:absolute; inset:0; missions was the only one that did not. Measured after, on a brief 5x the viewport: one scroller, clientH 736 vs scrollH 3244, scrolling 0 -> 2508 (exactly scrollH - clientH, i.e. the true bottom), zero page overflow, tab strip pinned throughout. Also removed five nested scrollers (70vh on live events; maxHeight caps on run streams, phase summaries, artifact bodies and error traces). Those existed only to work around the missing height and would have become portholes onto the very content the operator is trying to read. The xterm pane keeps its bounded box — FitAddon needs one, and a terminal owning its scrollback is correct. Deleting the header's description peek reclaims 104px for results (header 256 -> 152px); the same text renders in full in Setup -> Overview, as the code's own comment noted. Streaming now follows only when already at the bottom, via a shared useStickToBottom hook replacing two byte-identical copies, plus a "jump to latest" pill neither had. Defaults collapse by mission state, and a remount key fixes scrollTop leaking between tabs — a bug that only appears once scrolling works. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
4dec77ae6d |
fix(missions): give repo-less container missions the workspace they are promised
Every agent on a research_only mission refused to work, each reporting it was "in Claude Code", had no /mission/repo, and only had Read/Edit/Bash. All three statements were true. The run still recorded completed — 5 turns, 7.4k tokens, 0 artifacts, no error. The machinery is correct when a repo IS bound (verified on a live prod per-mission container: /mission/repo present, all 5 agents pinned). Only the repo-less path was broken, in three layers that disagreed by construction: - sync_in no-oped without a host checkout and copy mode does not bind /mission, so NOTHING created /mission/repo. The microVM tier already creates it, for the stated reason that "the guest needs the workspace to exist before the agent writes into it". Creating it host-side also un-breaks sync_out, equally a no-op before, so work survives across phases instead of being wiped. - pin_agent_workspaces returned Ok after pinning ZERO agents, so the deliberately-fatal guard in mission_orchestrator could never fire. Its error text already described the exact outcome we got. - The prompt advertised ZeroClaw tool names and explicitly denied `bash`, while every executor ends in `claude -p`: microVM passes Read/Edit/Write/Bash/Agent, session passes Read/Edit/Write/Bash, and claude_cli agents get Claude Code's native toolset — ZeroClaw's gating never reaches the subprocess. It was telling agents to use missing tools and avoid present ones. And it went green because mission_outputs logged the failed collect and continued — with the fail-empty rule and the NO-OUTPUT marker both BELOW that continue, so the phase was retried forever and never failed. The retry is now bounded by a grace window off completed_at. Verified end to end: mission completed, agent wrote /mission/repo/research/firecracker_vs_docker.md, collected and registered as a document artifact (6.6 kB of real content). Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
af89020dfd |
ci: put the docker CLI on PATH for the sandbox integration tests
cm-runtime/cm-sandbox tests shell out to `docker` via std::process, so the mounted socket alone was not enough — browser_tool failed with `docker available: NotFound`. Mount the host binary rather than apt-installing docker.io: the container is fresh every run, so an install would re-download ~100 MB each time and cache nothing. Verified on gw-04 that a mounted /usr/bin/docker talks to the host daemon (client=29.1.3 server=29.1.3), and that the agent-*:dev images these tests need are already present there. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
ee1cea72d9 |
ci: mount the docker socket so the testcontainers suite can run
cm-files' s3_store test starts a real MinIO via testcontainers. Without the
socket it does not skip — it fails with
`Client(Init(SocketNotFoundError("/var/run/docker.sock")))`, which looks like a
broken test rather than a missing capability. It passed locally only because the
Mac's docker socket was visible to the test process.
Sibling containers testcontainers starts are reachable because the test
container already shares the host network.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
8129f58845 |
ci: give cargo the credential for the private clawhdf5 git dep
First run failed in `cargo test --workspace`: "failed to load source for dependency clawhdf5", preceded by three "spurious network error: invalid packet line" retries. Two separate causes, both needed: - libgit2 cannot fetch from Gitea's smart-HTTP. images/server.Dockerfile already sets CARGO_NET_GIT_FETCH_WITH_CLI for exactly this; the test step did not. - quantumclaw/clawhdf5 is private (401 anonymous), so the CLI fetch needs a credential. Supplied via an insteadOf rewrite from a repo secret, so the token is masked in logs and never committed. The server image build does not hit this — it builds only clawmates-server, which does not pull cm-brain's clawhdf5 path. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
b1bf50160a |
ci: build and deploy to production from a push to main
Closes the one manual step left in the pipeline. gw-04 has run clawmates-deploy.timer every minute since July, pulling :latest and rolling on drift — the CD half already worked. What was missing was anything that moved :latest, since the old build host (tank) is packed for the move. The runner lives on gw-04 because it is the only reachable x86_64 host and prod images must be linux/amd64: web-01 is aarch64 and the fleet build boxes are offline. Host executor, capacity 1, so builds serialize rather than competing with production traffic. Three details that are not obvious: - `docker push :latest` does NOT move the tag on this registry once the manifest exists under another tag. The PUT-the-manifest step is what actually moves it, and its absence is how a "successful" deploy could leave prod on a stale image. - The final step verifies the image prod is RUNNING, not the one we pushed. A green edge on the old image is the failure this pipeline exists to prevent. - broker is built here too. It had no :latest tag at all, so gw-04's deploy loop logged a pull failure every single cycle since 2026-08-11. Also ignore the local env backups: `.env` was ignored but `.env.bak.*` was not, and those copies hold real credentials. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
dc8f65fc64 |
fix(metrics): GPU, network and disk IO were arriving and being dropped
`gpu_pct` read `info.g` as a scalar. Beszel 0.18 puts GPU in a different
collection entirely, as a MAP keyed by GPU index —
`{"0":{"n":"GeForce RTX 5060 Ti","u":0,"p":4.38}}` in `system_stats.stats`
— and `systems.info` carries no `g` at all. So every NVIDIA node reported
null while the data sat one request away. Null and "no GPU" are
indistinguishable downstream, so the fleet card showed nothing and a
`gpu_pct` drain rule could never fire, both without an error.
`net_sent_ps`, `net_recv_ps`, `disk_read_ps` and `disk_write_ps` were
columns nothing ever wrote. They come from the same sample.
The two array orders were MEASURED, not read off a schema, because
inverting one does not fail — it reports upload as download forever:
b = [sent, recv]. `stats.ni` gives per-interface [sent_ps, recv_ps,
total_sent, total_recv]; indices 2 and 3 matched /proc/net/dev
tx_bytes and rx_bytes on all four of tank's interfaces, and `b` is
the sum of the per-second pair across them.
dio = [read, write]. An 800 MB dd on tank moved index 1 from 7441 to
23688 while index 0 stayed near zero.
`info.ct` is deliberately NOT mapped to container_count. It reads 1 on
tank, which runs 1 container, and also 1 on architect, which runs 4 —
right exactly often enough to pass a spot check.
One extra request per poll, not one per node: the newest 1m sample for
every system arrives in a single sorted page. A hub that cannot answer it
falls back to the info snapshot rather than losing the CPU and memory
readings that still work.
GPU is the busiest card, not the mean — placement asks whether there is a
free GPU, and averaging a saturated card with an idle one answers a
question nobody asked.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
8470534e33 |
chore(fleet): drop morpheus from the deploy loop
morpheus is packed for a move. It is drained in the `nodes` table — heartbeats preserve `draining`, so it stays out of placement when it comes back — and removed from the agent-image loop here. An unreachable host in NODES does not merely skip it. The image loop fails the whole script BEFORE its verify stage, so four deploys in a row rolled the server and frontend correctly and then reported nothing at all; every one had to be confirmed by hand. Keep this list to hosts that answer. The name is left in a comment rather than deleted: putting it back is one word, and the next person will want to know where it went. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
c59cd9c424 |
feat(viz): the World draws missions, never the org chart
The World seeded the Organization → Company → Team → Agent tree whenever no mission was pinned — "My Workspace → General → Everyone". That tree describes almost nothing: `agents` has no org, company or team column, real membership is the `team_members` join, and four of its containers are fabricated in the browser and exist in no table. Worse, it did not replace the mission view, it SHARED the canvas with it. The plan events are only filtered by id when a mission is pinned, so with nothing pinned a live mission was drawn on top of the org chart: two unrelated graphs, both parented at the invisible root, reading as one scene in which they somehow connected. They never did — there is no edge between them because there is no relationship in the data to draw. The World now shows exactly one mission, or none. Three parts: - the org tree is gone from the canvas seed, and `worldCanvasRoots`, `narrowRoots` and `stripSynthetics` with it. The sidebar keeps its synthetic containers so orphaned agents still have a visible home. - the default focus prefers a RUNNING mission over the newest one. Newest-first picked whatever was created last, which on a workspace with history is a finished mission — so starting a run left the World looking at an old static map while the new work went unwatched. - exactly one mission is focused whenever there is any, which is load-bearing rather than cosmetic: the plan channel keeps ONE `planRef`, so two missions on the wire overwrite each other's title and phases and the scene becomes a blend of two runs that never happened. Seeding "all missions" was the tempting middle ground and is wrong twice: /api/workforce returns every mission ever with no limit, which puts hundreds of agents back on one canvas, and the plan channel cannot hold more than one anyway. An empty stage now says so. Blank canvas and broken page look identical, and filling that silence with a hierarchy that meant nothing is how this started. Co-Authored-By: Claude Opus 5 <[email protected]> |