08dd227a450c3e74cb00b7af42953bd39852e418
184
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar SobhandClaude Opus 4.7
|
f843c9ddb1 |
slice 7: before/after benchmark runner
Executes a benchmark harness inside the mission's team container
and records the resulting metrics as a benchmark_snapshots row keyed
on (phase_id, iteration). Baseline pass (iteration=0) captures
before_metrics; each post-iteration call captures after_metrics +
computes delta vs baseline.
Rust surface:
- cm_db::repo::missions::upsert_benchmark_snapshot / benchmark_snapshots_for
- cm_api::benchmark_runner::{baseline, after_iteration, run}
- Harness enum: Auto | Criterion | CargoBench | VitestBench |
PytestBench | Shell (each with a command() vector)
- Auto detection peeks at the repo layout inside the container
(Cargo.toml → CargoBench, package.json → VitestBench, pyproject
→ PytestBench). Falls back to a Shell echo when nothing
identifiable.
- Bencher-format line parser extracts (name, ns_per_iter,
plusminus) so criterion + `cargo bench` output become structured
samples the canvas can diff.
- compute_delta pairs samples by name, emits {before_ns, after_ns,
delta_pct, direction: improved|regressed}.
API:
- POST /api/missions/{id}/benchmark { phase_id, slot, iteration? }
triggers baseline or after run and returns the mission's full
snapshot list.
- GET /api/missions/{id} now includes `benchmarks[]` in the detail
payload.
Frontend:
- New Benchmarks tab on MissionCanvas with iteration + driver
header, plus a 4-column grid (bench / before / after / Δ%) when
delta samples are present. Improved deltas render green,
regressions red.
- TS types + triggerBenchmark() helper in lib/api/missions.ts.
Wiring notes:
- team_container_for_mission reads teams.zeroclaw_container — that's
populated by topology_worker::try_team_gateway_url on first run,
so trigger baseline AFTER the mission's first phase spawns the
container.
- Not auto-fired yet by phase execution; that's the "template phase
executor" work that spans Slices 4-8. Manual API trigger works
today; automated hook is a follow-up.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.7
|
565f6cae65 |
slice 4: 5 workflow templates as TOML recipes + mission-launch orchestrator
Two things land together:
1. Ships the 5 workflow recipes as TOML files under
templates/workflows/*.toml:
- research_only (hub_spoke research → MD + PDF)
- research_and_code (research → coding loop until INT-XX drained)
- security_hardening (scan → research patches → coding with
reviewer approval + full MCP bundle)
- refactor (single-pass coding with dep audit
preamble + before/after benchmarks)
- benchmark (author + baseline benchmarks per stack)
Each declares phases[], per-phase config, default_team_template.
Loaded read-only into an in-memory registry (workflow_registry)
via OnceLock — no DB row per recipe.
2. Ships the mission-launch orchestrator that closes the loop from
Slice 3.5d's mechanics. When a mission transitions draft→running,
`mission_orchestrator::on_launch`:
- Reads mission.team_template_id (skips if unset)
- Loads the team template detail (roles + skills bindings)
- Builds a topology graph from role slots via cm_topology::build
- Inserts the teams row + stamps template_id/version/risk_profile/mcp_bundles
- For each role: agent insert, model binding, runtime provision
(opt-in via RuntimeProvisioner::from_env), brain_seed::ingest
(Slice 3.5d), agent_template_link::upsert (Slice 3.5d),
team_members bind, audit trail
- UPDATE missions SET team_id = ...
Wired into routes::missions::set_status when prior.status='draft'
and new='running'. Failures log + are non-fatal (mission still
flips to running so the user can inspect + retry).
With this, Slice 3.5d's brain-seed + link machinery actually gets
populated, and the MCP skills server's template-defaults-merge path
(Slice 3.5b/d) starts serving real bindings to real agents.
Follow-ups (Slice 5-8):
- Task-card parser watches run events for TASK/COMPLETED markers
→ mission_tasks rows
- PDF renderer worker turns MD artifacts into PDFs
- Before/after benchmark runner honors phases[].config.benchmark
- Security scan MCP bundle exposes cargo-audit/gitleaks/trivy/semgrep
- Level-up endpoints diff learned-vs-seeded via agent_template_link
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.7
|
1153d72d00 |
slice 3.5a: skills catalog — the "how to think" layer
Introduce the skills catalog: the second half of the two-layer agent
model (skills teach agents HOW to think about a problem; MCP servers
give them the ABILITY to act). Delivery via MCP resources lands in
Slice 3.5b; this slice ships the data model + API surface.
Migration 0049 extends the legacy `skills` table (from 0001_init.sql,
originally a workspace catalog of markdown snippets) with the richer
typing we need — name, when_to_use, tags, source_kind, current_version
— rather than duplicating tables. Also adds:
- skill_versions (version history for level-up promotions +
rollback; back-pointer via promoted_from
JSONB records agent_id / research artifact /
brain memory that produced it)
- template_role_skills (m2m binding skills to team-template roles
with pin_in_context + order_idx)
- agent_skills_ext (per-agent overlay: include=true adds a skill
to the bundle; include=false prunes a
template default for this specific agent)
Rust surface:
- cm_db::repo::skills_catalog with typed Skill/SkillVersion/
AgentSkillBinding structs + upsert_builtin (idempotent — bumps
version + appends to skill_versions ONLY when body changes) +
list_visible/get/get_by_name reads + template + agent binding
helpers + effective_for_agent (merges template defaults with
agent overrides, applies exclude precedence, batch-fetches skill
bodies)
- cm_api::routes::skills_catalog with:
GET /api/skills — list visible
GET /api/skills/{id} — detail
GET /api/claws/{id}/skills — effective binding (accepts
template_id + role_slot as query args to merge in template
defaults)
Follow-ups:
- Slice 3.5b: clawmates_skills MCP server exposes catalog as MCP
resources, honoring pin_in_context for auto-injection
- Slice 3.5c: seed ~40-60 builtin skills across the 6 stacks
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|
||
|
Omar Sobh
|
ee675b976a | fmt: apply cargo fmt to team template + loader | ||
|
Omar SobhandClaude Opus 4.7
|
9ba5c06a1a |
slice 3: 6 team templates seeded from TOML recipes
Team templates are the canonical rosters + tool bundles that mint
concrete teams for a mission. Every builtin ships as a TOML recipe
under templates/teams/*.toml, loaded into the DB at server boot.
Migration 0048 adds:
- team_templates (id, key, name, stack, default_topology,
risk_profile, mcp_bundles, version, source,
workspace_id)
- template_roles (m2m: template_id + slot; system_prompt,
skills[], brain_seed)
- teams gets template_id + template_version for level-up lineage
Ships 6 builtins:
- rust_sdlc — planner/coder/tester/reviewer/committer for Rust
- backend — api_designer/db_engineer/coder/tester/committer
(Postgres, DuckDB, graph DBs, wire protocols)
- frontend — designer/coder/tester/committer (React + Tailwind + ShadCN)
- mobile — designer/coder/tester/committer (Expo, RN, iOS, Android)
- gpu — arch_analyst/kernel_author/bench_engineer/coder/committer
(CUDA, Metal, ROCm from Rust)
- threejs — scene_designer/coder/shader_author/perf_engineer/
committer (three.js, WebGL, WebGPU)
Each role has a versioned system_prompt + skill list + brain_seed
markdown. Skills column is a name array today; Slice 3.5a promotes it
to a typed m2m join with the real skills catalog.
Server boot:
- team_template_loader::load_builtins reads TOML from
/etc/clawmates/templates/teams (container) or templates/teams (dev),
upserts idempotently. Deterministic uuid per template key (sha256
of a fixed namespace + key) so ids are stable across boots.
- Dockerfile copies templates/ to /etc/clawmates/templates.
Read API:
- GET /api/team-templates — list all
- GET /api/team-templates/{id} — detail with roles
Wizard:
- Step 3 rewired from a raw team_id text field to a template picker
with "LLM auto-provision" as the default option + one card per
builtin, showing stack, topology, risk profile, and description.
- Mission create now passes team_template_id (not team_id) so phase
execution knows which template to mint from.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|
||
|
Omar Sobh
|
a72da9dff0 | fmt: apply cargo fmt to missions | ||
|
Omar SobhandClaude Opus 4.7
|
fbefc67878 |
slice 1: missions data model + migration
Introduce the unified `missions` tier that will replace the current
research_topics + loops split. This slice ships the data model +
backfill + skeleton REST surface; the old wizards keep working in
parallel until Slice 9's big-bang cutover.
Migration 0047 adds:
- missions (top-level workflow: template_kind + team +
schedule + status + config)
- mission_phases (ordered {research|coding|benchmark|
security_scan} phases per mission)
- mission_tasks (typed units of work, e.g. INT-XX cards,
UPSERT-keyed on (phase_id, external_id))
- mission_artifacts (MD/PDF/benchmark/security/diff files with
a pending queue for the PDF renderer worker)
- benchmark_snapshots (before/after pairs per iteration)
Backfill copies existing research_topics + loops rows into the new
tables as one-shot missions with the appropriate template_kind, so
Slice 2's UI can render the full history immediately.
New Rust surface:
- cm_domain: MissionId, MissionPhaseId, MissionTaskId, MissionArtifactId
- cm_db::repo::missions: Mission/MissionPhase/MissionTask/
MissionArtifact structs + insert (txn-wrapped)/get/list/set_status/
phases_for/set_phase_status/upsert_task/tasks_for/register_artifact/
artifacts_for/next_pdf_pending/set_pdf_result
- cm_api::routes::missions: skeleton list/create/get/set_status
routes registered at /api/missions/*
Follow-up slices layer richer behavior (template dispatch, phase
execution, task parsing, artifact rendering) on this foundation.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|
||
|
Omar SobhandClaude Opus 4.7
|
5bbab870fc |
topology/log-sse: resolve team container name for coding-loop runs
The log SSE endpoint was always resolving the container name from a research topic id (research-<topic_id>-team). For paired coding loops that own a team_id, the actual container the topology worker spawns is team-<team_id>-container (spawn_team) — not the research topic's one. That mismatch produced a stream of 404s from Docker: Docker responded with status code 404: No such container: research-<topic_id>-team Mirror topology_worker::try_team_gateway_url's precedence: 1. run's loop has team_id → team-<team_id>-container 2. run has research_topic_id → research-<topic_id>-team 3. loop.source_research_topic_id (legacy paired flow) → same as 2 4. else → clear error, no more 404 spam Co-Authored-By: Claude Opus 4.7 <[email protected]> |
||
|
Omar SobhandClaude Opus 4.7
|
3444859e11 |
wizard/repo/ensure: degrade to skipped-fanout on clawstor errors
The wizard was hard-failing (500) when clawstor was unreachable — env unset, network error, non-JSON HTML response from a fallback proxy, or non-2xx status all mapped to ApiError::Internal, which blocked the wizard from advancing. Clawstor fan-out is a warmup optimization, not a prerequisite. Real fleet materialization happens later at spawn time. When the aggregator is absent, return a skipped FanoutReply (all_ok=true, empty peers) so the wizard proceeds, and log the reason server-side. Unblocks: picking clawhdf5 in the ResearchWizard when the tank/ architect clawstor daemons are running but the HTTP aggregator (`claw-store serve`) isn't deployed yet. Co-Authored-By: Claude Opus 4.7 <[email protected]> |
||
|
Omar SobhandClaude Opus 4.7
|
657b666219 |
mcp door: extend service-session bearer to per-topic + per-loop spawns
Per-team runtimes had their MCP bearer swap wired in the prior slice, but per-topic (research_container::spawn) and per-loop (spawn_loop) containers still baked the stale template bearer and 401'd every tools/list. Same fix, extended: mint a workspace-owner service session via runtime_provision::mint_workspace_service_token and inject via prewrite_daemon_config_with_risk. Move mint_workspace_service_token from topology_worker into runtime_provision so all three spawn call sites share the helper. Callers updated: routes/research.rs (start_topic), routes/research_setup.rs (prepare_topic_runtime), routes/loops.rs (ensure_loop_container). Co-Authored-By: Claude Opus 4.7 <[email protected]> |
||
|
Omar Sobh
|
6e37802f73 |
claws: PATCH /api/claws/{id}/model — swap the runtime-bound model
Auto-provisioned agents (via the research + loops team wizard) show
up on the Agents page (same agents::insert → agents::roster path),
and PATCH /api/claws/{id} already handled name/job_title/system_prompt/
avatar/accent/wallpaper. The one thing that was NOT reachable from the
Agents page: swapping the model.
Add a dedicated endpoint:
- PATCH /api/claws/{id}/model { model: string }
- Persists via agents::set_model_binding (DB)
- Best-effort runtime rebind via RuntimeProvisioner::provision_claw
(idempotent — overwrites agents.<alias>.model_provider on the
shared ZeroClaw config)
- Audit-logged as 'agent.model_changed' with the new model in payload
Now an operator can open the Agents page, click a claw that was
auto-provisioned by the team wizard, and swap its model
(claude-sonnet-5 → glm-5.2 → whatever) without recreating the team.
Same DB row, same claw_id, same brain — just a new provider on the
next turn.
Frontend affordance not shipped in this commit — the endpoint is
usable via curl/psql/scripts today; a UI 'Model' picker on the claw
detail card can land in the next Agents-page pass.
|
||
|
Omar Sobh
|
ac8c689f50 |
logs + team parity: pretty step/container renderers; quota + audit on team creation
Two bundled changes:
── LiveRunLogs prettification ──────────────────────────────────
The Steps + Container tabs were plain mono lines with a single
color per event. Now they get structured layout:
Steps:
- Color-hashed actor pill (stable palette so [Distiller] and
[Novelty Analyst] each get their own hue across the session).
- Phase pill (plan=cyan, work=green, synth=amber, aggregate=purple).
- Token count pill formatted 1.2k / 14.3k / etc.
- Gated-action warning pill in amber when > 0.
- Left-border color strip keyed to the actor for at-a-glance
visual grouping.
- Long outputs collapse to their first 300 chars with a '+ N more'
toggle to expand the full text.
- 'done' events get a green (or red for error) border strip +
pill instead of blending into the stream.
Container:
- Splits '[actor] action (outcome) · msg' into colored spans —
actor pill (deterministic color), action in dim, outcome pill
green/red/dim by state.
- Non-line events (info/error/done) get their own left-border
strip so bash echoes and stack traces don't drown in the daemon
chatter.
- Timestamps switch to HH:MM:SS.mmm — dense but scannable.
Small palette (LOG constants) keeps the color budget bounded — no
new UI vocabulary, just cleaner reads of what was already there.
── Team-wizard governance parity ───────────────────────────────
build_team_with_lifecycle now matches POST /api/claws' governance:
- enforce_new_agent quota check per member (previously bypassed
workspace agent quotas entirely for team/auto-provision paths).
- audit::append('agent.created', ..., {source: 'team_wizard'}) per
member so team-created claws appear in the same audit trail as
individually-created ones. Adding a 'source' key distinguishes
provenance without changing consumers.
.brain (h5) handling was already consistent between the two paths —
both use the lazy on-first-access load_brain hook seeded from
agents.system_prompt. No change there.
|
||
|
Omar Sobh
|
956be2cf4f |
wizard: in-place auto-provision team from topic (LLM-derived, sonnet-5)
Step 5 of ResearchWizard was 'assign agents from workspace roster'.
When the roster was empty, the wizard body was hard-swapped for
NoAgentsGate — you couldn't reach step 5 at all.
Now step 5 shows a 'Team' panel:
- Big cyan card: 'Auto-provision team from this topic'. One click
runs an LLM plan pass, gets 3-5 role slots + system prompts back,
materializes claws via the existing build_team pipeline, stamps
runtime posture, returns a shape that drops straight into the
submit body's agents[]. Card flips green with the derived roster.
- Below that: the classic roster picker, but only when the workspace
actually has ≥1 claw AND auto-provision hasn't landed. Otherwise
hidden — no dead empty-state affordance.
Every gate that required agents.length > 0 to render the wizard body
or the footer is gone. canNext gains a step-5 clause: allow Next when
EITHER auto-team is ready OR the user handpicked from a non-empty
roster.
Backend
- POST /api/teams/auto-provision — accepts {title, description,
outcome_kind, topology_kind?, model?, risk_profile?, mcp_bundles?}.
Derives topology from outcome_kind (integrations → pipeline; else
hub_spoke). LLM plan pass yields a JSON roster of 3-5 roles
(role_slot, name, system_prompt). Materializes team + claws via
build_team, stamps risk_profile (default research_web_readonly) +
mcp_bundles (default [clawmates_door, gitea_forge]). Response
carries team_id + agents[] in the shape /api/research already
expects.
- Every provisioned claw runs on claude-sonnet-5 by default;
overridable via the model field.
Follow-ups (not in this slice):
- Same picker in LoopsWizard (slice C — parallel change, same API).
- Post-create 'Team' section on ResearchCanvas / LoopsCanvas so
users can rebind after the fact (slice D).
- Full Teams tier UI + Agents-page deprecation (slice E).
|
||
|
Omar Sobh
|
0b7f247b0e |
wizard: 'fresh coding team' picker for paired coding loop
Second slice of the per-loop-team arc. The paired-coding-loop checkbox
in ResearchWizard step 6 now exposes a two-option picker:
⦿ Provision a dedicated coding team (default when the loop is on)
— fresh 'Coding · <topic>' team row, risk_profile =
coding_readwrite, clawmates_door in mcp_bundles. Loop's
team_id is bound at wizard-submit time.
○ Reuse the research team (legacy) — no team_id bound; coding
iterations spawn against the research topic's container.
Frontend
- New codingTeamMode state, radio picker rendered under the checkbox.
- research.ts createTopic body gains paired_coding_team_mode?: 'fresh'|
'reuse'.
Backend
- CreateTopicRequest gains paired_coding_team_mode: Option<String>.
- materialize_topic_loops takes it through and, when 'fresh', calls
the new provision_fresh_coding_team helper — inserts a teams row
via the existing insert_team_with_lifecycle (pipeline kind, same
graph as the loop), sets its runtime-config via
set_team_runtime_config, then binds loop.team_id.
- All operations best-effort with stderr logging — a team-provision
failure leaves the loop functional under the legacy fallback.
Not shipped in this slice (deferred to runtime hookup slice):
- research_container::spawn keyed on team_id → per-team container
- Config template rewrite injecting the team's risk_profile
- Migration of existing paired loops onto their own teams
The plumbing lands now so the wizard's intent is recorded; the
runtime honors it in the next PR.
|
||
|
Omar Sobh
|
6066e93889 |
teams: per-team runtime posture (risk_profile + mcp_bundles) + FK from loops/topics
Foundation slice for letting a coding loop bring its own team instead
of reusing the paired research topic's team. Turns out the teams
table already exists (0010_teams.sql) with full CRUD — this scales
back to the minimal missing bits:
Schema (0045_teams.sql)
- ALTER TABLE teams ADD risk_profile TEXT (NULL = template default)
- ALTER TABLE teams ADD mcp_bundles JSONB DEFAULT '[]'
- ALTER TABLE loops ADD team_id UUID REFERENCES teams ON DELETE SET NULL
- ALTER TABLE research_topics ADD team_id UUID REFERENCES teams
- Two partial indexes (team_id NOT NULL) for the future cascade queries
cm-db (dynamic sqlx::query so the existing get_team's compile-time
cache doesn't need regenerating):
- TeamRuntimeConfig struct
- get_team_runtime_config / set_team_runtime_config
- team_for_loop / team_for_research_topic (resolvers)
- set_team_for_loop / set_team_for_research_topic (binders)
cm-api
- GET /api/teams/{id} now surfaces risk_profile + mcp_bundles
- PATCH /api/teams/{id}/runtime-config sets them
Not touched (comes in follow-up slices):
- Wizard picker exposing 'reuse research team' vs 'fresh coding team'
- Runtime container spawn keyed on team_id
- Migration of existing paired coding loops onto their own team
|
||
|
Omar Sobh
|
2ba40b03b7 |
loops: per-iteration live logs (Steps + Container tabs), collapse graph JSON
The Loops canvas' IterationsTimeline was a flat status-pill list —
no way to see WHAT an iteration was actually doing. Meanwhile the
Research canvas had full LiveRunLogs with Steps + Container tabs
against the same underlying topology_run SSE endpoints. This
factors LiveRunLogs so both canvases share it.
Frontend
- LiveRunLogs gains a directRunId?: string prop. In this mode it
skips the topic-scoped active-runs + pipeline-state polls and
pins activeRun to the given id. topicId stays optional (topic
mode unchanged from ResearchCanvas' perspective).
- LoopsCanvas IterationsTimeline: each row is now a click-to-
expand card. On expand, renders <LiveRunLogs directRunId={...}/>
right below the header — Steps + Container tabs, full SSE tail,
same 320px terminal.
- Auto-opens the newest running/queued iteration so a click on
'Run now' immediately exposes the live pane.
- New CollapsibleSection helper wraps the graph JSON block so it
starts closed. Reference material stays one click away without
cluttering the canvas.
Backend
- run_container_log_sse now resolves the tail target via
loop.source_research_topic_id when the run itself has no
research_topic_id. Paired coding loops (kind=exec with a
source_research_topic_id) reuse the paired topic's team
container, so we tail its docker logs. Pure loop runs still
error with a clearer message.
|
||
|
Omar Sobh
|
e3ef3fd056 |
research: fix Draft 'Invalid Date' + stale-error leak in pipeline card
Two cosmetic bugs surfaced by the successful v0.8.3 pipeline run: 1. **'produced Invalid Date'** — research_outcomes.created_at was an OffsetDateTime serialized by time's default array format (`[y, ordinal, hh, mm, ss, ns, tz]`), which browser's `new Date(...)` can't parse. Add `#[serde(with = "time::serde::rfc3339")]` matching the pattern already in threads.rs / routine_runs.rs. 2. **Stale error text on pipeline card** — the runs stage's `latest_error` walked every run by `created_at DESC` and returned the first non-empty error, so a topic with an earlier failed run + a later completed run kept displaying the old error next to '1 completed'. Now the error only surfaces when the MOST RECENT run itself failed. Historical failures stay in the run count but don't leak their message. |
||
|
Omar Sobh
|
ae113dd319 |
live-run-logs: add Container tab that tails filtered daemon logs
Steps summaries only fire AFTER each topology step completes — so a
stalled first turn was completely dark. Add a second tab that
streams the team runtime container's daemon log live via a new SSE
endpoint.
Backend
- GET /api/topology-runs/:id/container-log — workspace-scoped SSE
around bollard's docker.logs(follow=true, tail=200). Buffers on
newline so partial mux chunks don't truncate a log line.
- compact_container_log: parse a zeroclaw daemon line
('[actor] ... zc_action=X zc_outcome=Y ... msg') into
'[actor] action (outcome) · msg'. Framing-only continuations are
dropped; non-zc lines (bash echoes, backtraces) pass through as-is
so nothing interesting is lost. ANSI escapes stripped.
- Everything funnels through one async_stream! so early exits
(workspace check / docker connect / no bound topic) yield an
'error' event and return without breaking Sse::new's single stream
type.
Frontend
- LiveRunLogs gets a sub-tabs strip: Steps · Container.
- New useContainerLog(runId, active) hook — gated by tab so we don't
hold two open SSE streams when the operator isn't looking.
- Same terminal widget renders each container line with a level
color (info/done grey, line default, error red). Sub-tab pill
shows count + status live.
|
||
|
Omar Sobh
|
36a9fbe81f |
research/start: allow rerun on loop-owned topics too
D1-fold loop guard 409'd even the rerun path if the topic was scheduled — a failed iteration couldn't be restarted until the loop's next scheduled fire. Now the loop guard only fires for the fresh 'standby' start; the rerun path (status=='processing' + orphan cancel) works whether or not a loop owns the topic. Loop binding is preserved either way. |
||
|
Omar Sobh
|
6af9d509b0 |
research: rerun cancels orphan runs first so 409 doesn't block restart
The previous rerun guard (standby OR (processing AND in_flight==0)) still 409'd when a lingering queued/running row existed — typically an orphan from a server restart mid-pipeline, before the reaper or stale-checkpoint requeuer had marked it failed. Now: allow rerun on any 'processing' topic; before enqueuing the fresh run, batch-cancel every in-flight run for the topic so the new run isn't racing them. Terminal states (reviewing / publishing / published) still 409 as before. |
||
|
Omar Sobh
|
671d3ac4f0 | clippy: fix doc_lazy_continuation on TopicListItem.runs_failed | ||
|
Omar Sobh
|
f70f6c679e |
research: errored-state card + one-click rerun (no wizard re-entry)
When a topic ends up parked in 'processing' with all runs failed and nothing in flight, the sidebar card was still spinning as if progress were happening. Now: Backend - topology_runs::run_counts_by_research_topic — batch query that returns (in_flight, failed-since-last-success) per topic. Used by the list endpoint; dynamic sqlx::query() so no prepare needed. - TopicListItem DTO gains runs_in_flight + runs_failed. - start_topic status guard relaxed: allow (standby) OR (processing AND runs_in_flight == 0). Blocks accidental double-fires on a live pipeline; permits rerun on a failed one. Same request body, same behavior once accepted, so the frontend just POSTs /research/:id/start on the RotateCw click. Frontend - ResearchList detects errored: status===processing && !in_flight && failed>0. Swaps the MiniSpinner for a red AlertTriangle and changes the status text to 'error · N failed'. - New RotateCw icon button next to the delete Trash — same button cluster, one click, no wizard re-entry required. Disables while a request is in flight; error surfaces in the sidebar's shared error banner. |
||
 osobh
|
1d69866bcf | research canvas: collapsible sidebar + live topology-run logs (#6) | ||
|
Omar Sobh
|
13ead5cb9f | fmt: sort routes/mod.rs pub mod entries | ||
|
osobh
|
149ad992bb | wizard: materialize picked repo across clawstor fleet at step-2-next (#5) | ||
|
osobh
|
1cb643142c | research/publish: gate approve+reject on Owner role (#4) | ||
|
Omar Sobh
|
212e68f6b1 |
research/probe: end-to-end smoke test endpoint (skip topics/loops/spawn)
New: POST /api/research/probe — one-shot pipeline probe against the
workspace's shared ZeroClaw gateway. Drives a trivial 'reply OK'
turn via ZeroClawDriveExecutor::drive and reports per-step timings.
Purpose: stop guessing what's broken in the wizard flow by exercising
JUST the executor→daemon→claude→response path. If probe succeeds
within a few seconds, we know:
- ZEROCLAW_TOKEN + gateway URL config is correct
- Daemon can reach and authenticate against claude
- The full ws round-trip works
…and every other failure we've been chasing (turn timed out, LLM
request failed, ws connect DNS error, etc.) is spawn-config or
prompt-size specific.
Request body (both optional):
{ "prompt": "Reply OK", "agent": "coordinator" }
Response:
{
"verdict": "ok" | "fail",
"total_duration_ms": …,
"prompt_len": …,
"response_preview": "OK",
"steps": [
{ name: "build_executor", duration_ms: …, status: "ok" },
{ name: "drive_turn", duration_ms: …, status: "ok", detail: "tokens=N, output_len=M" }
]
}
Wire-up:
- new routes/probe.rs
- pub mod probe in routes/mod.rs
- POST /api/research/probe registered in lib.rs router
- ZeroClawDriveExecutor::drive promoted from private to pub so the
probe handler can call it (behavior unchanged, other callers were
all inside the same struct).
Usage from anywhere (curl, browser dev tools, etc.):
curl -X POST https://clawmates.work/api/research/probe \
-H "Content-Type: application/json" \
-H "Cookie: <session cookie>" \
-d '{}'
Follow-up: a small frontend button (e.g. bottom of ResearchList) that
POSTs this and renders the response inline, so users don't need to
curl. Skipping in this commit to ship the useful part first.
|
||
|
Omar Sobh
|
7e0620fd08 |
research: don't advance to reviewing without an outcome + guard-before-write
Two bugs that combine to produce the 409 you get when clicking Approve:
1) notify_run_completed (topology_worker post-hook) was advancing
topic status processing → reviewing whenever the last sibling run
terminated — success OR fail. A failed run with 0 outcomes still
pushed the topic to `reviewing`, the canvas rendered the "Request
publish" affordance, and the reviewer clicked Approve on nothing.
Fixed by adding an EXISTS(research_outcomes …) clause to the
UPDATE. Topic stays in `processing` when no outcome exists; the
loop's next iteration still has a chance to produce one.
2) decide_publish was calling
research_publish_approvals::decide(approve=true)
FIRST (which flips the row to `status='approved'`) and then
running the "no outcome? 409" guard SECOND. On the 409 return,
the DB was left half-flipped: approval says approved, topic still
in reviewing, no outcome exists, and every future click to the
same approval returns 409 on the "already decided" guard —
leaving reviewers with no way forward.
Fixed by moving the outcome-existence check BEFORE the decide()
call. On 409 now nothing was written, so the reviewer can try
again cleanly once an outcome is produced.
Also unstuck the current stuck row out-of-band (SQL UPDATE to reset
the approval to pending + topic to processing) so the user isn't
forced to delete the topic to escape the 409 loop.
sqlx dynamic query — the new EXISTS clause wasn't in the offline
cache so I switched notify_run_completed to plain `sqlx::query`.
|
||
|
Omar Sobh
|
ee05037095 |
research-pipeline-diag: state-aware statuses (waiting != failing)
The diagnostic was calling any 0-outcome + 0-completed-runs state a
FAIL — red dot + 'No outcome produced yet — check the runs stage for
the failure reason'. That fires the moment a wizard-materialized
topic lands, before its very first turn even completes, and stays
red for the whole 2-3 min a legitimate coordinator turn runs. Result:
users see 'FAIL' on every fresh topic and can't tell a real failure
from a normal in-flight state.
Backend fix — introduce a `waiting` status (blue/pulsing in UI):
Runs stage:
0 runs -> skip ('No runs yet — pipeline hasn't fired')
any running/queued -> waiting ('N in flight, M completed')
all failed -> fail (with error text)
some failed -> warn
all completed no fails -> ok
Outcomes stage:
outcome_count > 0 -> ok
0 outcomes + 0 runs -> skip ('No outcome yet (pipeline hasn't fired)')
0 outcomes + any running -> waiting ('Waiting for the current run to finish…')
0 outcomes + any failed -> fail (the actual silent-bug case)
0 outcomes + all done ok -> warn (weird — completed but wrote nothing)
Also suppresses the run stage's `latest_error` detail when the run
status is `waiting` or `skip` — reporting a stale error next to an
actively-running job is what made users think the current run had
failed.
Frontend:
- PipelineStage['status'] union grows a 'waiting' arm.
- Pill color: cyan (#5ec8d8) with a pulsing scale/opacity animation
(new cm-pulse keyframe in motion.css).
- Strip summary line: 'Pipeline in flight — waiting for run to finish…'
when there's any waiting stage and no failures.
- Border tint: cyan border when waiting, coral when failing, neutral
otherwise.
Zero backend semantic changes to the outcome-write path — this is
purely UI truth-telling.
|
||
|
Omar Sobh
|
bfcdca0583 |
research-canvas: managed-by-loop UI + start_topic guard (fold cleanup)
Closes the UX gap the fold introduced: the topic canvas was still showing "Start research" for standby-state topics even when a scheduled loop already owned the runs. Clicking it would 409 (or worse: race the loop into a duplicate run). Topic status stayed at standby forever because the loop path bypassed start_topic's set_status transition. Four changes: 1. **Backend status transition** — compose_and_enqueue_iteration for kind='research' now calls set_status_if(standby, processing) on the topic before the run is enqueued. New DB helper set_status_if only advances when the current status matches the "from" arg — safe against races and re-invocations. Later iterations no-op since the topic is already past standby. 2. **has_managed_loop on TopicDetail** — get_topic hydrates a new ManagedLoop struct (loop_id, title, enabled, next_fire_at, last_run_id, schedule_summary) when a kind='research' loop is bound to the topic. summarize_schedule() derives a human string from the loop's triggers jsonb (e.g. "cron: 0 3 * * * · on new artifact", "one-shot", "manual"). New DB helper loops::research_loop_for_topic returns the row. 3. **Canvas branch** — nextAction takes a managedByLoop flag; when set + status=standby, returns null (no button). The canvas renders a "MANAGED BY LOOP" strip below the topic title showing loop name, schedule summary, next fire time, and enabled dot. Reviewer buttons (Request publish / Approve / Reject) still show normally in later states — reviewers should still promote outcomes even when a loop is producing them. 4. **start_topic guard** — refuses with 409 when a research loop already owns the topic. Closes the direct-POST hole for anyone bypassing the frontend. TS type + summarize_schedule live in the same commit so an old client hitting a new backend just ignores the extra field (no breakage), and a new client hitting an old backend renders the classic buttons (managed_by_loop is optional). |
||
|
Omar Sobh
|
f910771bbb |
research-prompt: add AUTONOMY CONTRACT + relax citation rigidity
Inspection of a stuck 500s+ research run showed both agents were
correctly picking up the topic AND producing rich, structured plans —
then stalling at 'Should I proceed?' and 'Which approach?'. No human
to answer = infinite spin until stale-sweep + retry, forever.
Two prompt changes to compose_research_iteration_task:
1. AUTONOMY CONTRACT at the top:
- Explicit "no human will answer you"
- Explicit "do NOT ask for confirmation"
- Explicit "you MUST emit the completed artifact"
- Framed as a contract, at the very top, before the task itself.
2. Softer citation rule:
- Was: 'Cite every claim; never fabricate sources or repo paths.'
- Now: 'Cite what you can verify. Use [claim needs verification]
inline when you can't. A written v(N) with rough citations
beats a blocked v(N) waiting for approval. Do not fabricate
concrete titles/authors/DOIs.'
The absolute anti-fabrication rule made agents refuse to write
anything unless they could be sure. Coupled with no user available,
the whole loop stalled — turns completed successfully but no output
converged.
Follow-up if this repeats:
- Even shorter prompt (the current one is 300+ words)
- Explicit output-shape enforcement ('respond with only the markdown
artifact, no preamble')
- Cap iteration count so a broken prompt doesn't burn tokens forever
|
||
|
Omar Sobh
|
2df9dd04df |
research: build real topology graph in materialize_topic_loops
The wizard-created research loop's first iteration failed with
"missing or invalid graph" — materialize_topic_loops was writing
`{nodes: [], edges: []}` as a placeholder, which the topology worker
rejects. Also explains why prepare_topic_runtime hadn't cloned the
repo or spawned the container: the run failed before
compose_and_enqueue_iteration got to call it.
Fix: extract build_topic_graph_json() into research_setup.rs — same
shape start_topic uses (hydrate roster, promote a role_slot-tagged
coordinator to index 0 for hub_spoke/hierarchical/star_moe, build
via cm_topology::build, serialize via cm_topology::to_json). Called
from materialize_topic_loops instead of the empty placeholder.
Fully best-effort. Any DB/topology failure falls back to a
single-node hub graph so the loop still runs (degraded, but not
silently broken).
The next wizard-created topic should now:
1. Materialize the research loop with a valid graph
2. Fire the initial burst
3. compose_and_enqueue_iteration calls prepare_topic_runtime →
clones the repo, spawns the container
4. Enqueues a run whose graph parse succeeds
5. Run drives to completion, produces an outcome
|
||
|
Omar Sobh
|
51756d0e68 |
clippy: struct-bundle enqueue_iteration_with_topic + fix doc list warnings
CI's clippy stage failed with -D warnings on three classes of lint:
1. map_clone in loops::recent_reorders — .map(|a| a.clone()) is the
pattern clippy wants replaced by .cloned(). Trivial swap.
2. too_many_arguments on enqueue_iteration_with_topic (8 args, ceiling
7). Refactored the caller-side surface into a new
IterationEnqueue<'a> struct with fields for each column. Matches
the pattern research::NewTopic + loops::NewLoop already use for
the same clippy ceiling. Callers in loops.rs (routes) + the
enqueue_iteration wrapper updated to build the struct literal.
3. doc_lazy_continuation + doc_list_indentation — two doc comment
blocks used ambiguous list-like layouts:
- compose_iteration_task's ASCII-art prepended-block preview:
wrapped in a ```text fence so clippy stops parsing "RESEARCH
ARTIFACT:" etc as list continuation.
- fire_initial_burst_if_set's mention of `burst - 1`: rewrote so
"- 1" doesn't start a line and get misread as list marker.
Purely a refactor + doc pass; no behavior change.
|
||
|
Omar Sobh
|
c0c5fd7104 |
research: extract setup helpers into research_setup.rs (fixes file-size gate)
research.rs hit 1446 lines with the wizard-fold + regression-fix
commits, tripping the CI file-size gate (limit 1250). Pure extraction
into a sibling module; no behavior change.
Moved to routes/research_setup.rs:
- RepoContext (now pub — used by build_coordinator_task)
- TopicSchedule (now pub — request body sub-struct)
- research_workspace_root (now pub)
- prepare_topic_runtime (already pub — used by loops iteration path)
- ensure_repo_workspace (now pub — used by start_topic + prepare)
- materialize_topic_loops (now pub — used by create_topic)
research.rs re-imports them via `use crate::routes::research_setup::{...}`
so the calling code reads identically.
New line counts:
research.rs 1127 lines (was 1446, limit 1250)
research_setup.rs 321 lines (new)
Also updated the loop-iteration callsite in routes/loops.rs to point
at the new path (crate::routes::research_setup::prepare_topic_runtime).
No API changes; migrations unaffected.
|
||
|
Omar Sobh
|
80c23e57ed |
research: fix wizard loops never firing + missing clone/spawn (regression)
Two bugs surfaced by the first end-to-end wizard run — the pipeline diagnostic showed "0 run(s)", "Repo bound but never cloned", and "Container not spawned" for a topic that had been created with a nightly research loop. Bug 1: materialize_topic_loops bypassed initial_burst firing. The wizard-materialized loops path calls cm_db::repo::loops::create directly (a plain INSERT). The initial_burst-fires-first-iteration logic lived inside the create_loop HTTP handler, so wizard loops landed in the DB but never fired their initial iteration. Fix: extract routes::loops::fire_initial_burst_if_set as a pub helper that read triggers, ensures the loop container, and calls the kind-aware compose_and_enqueue_iteration. Both create_loop and materialize_topic_loops now call it. Bug 2: research-kind loop iterations skipped clone/spawn. compose_research_iteration_task only built the coordinator prompt; the repo clone and topic container spawn lived only in start_topic. So the first research iteration ran against a nonexistent clone directory and a stale gateway, and every run failed. Fix: extract routes::research::prepare_topic_runtime as a pub helper that runs ensure_repo_workspace + research_container::spawn. Idempotent — second iteration reattaches. Called from compose_and_enqueue_iteration before enqueuing a research iteration. Topics without a repo bound are a no-op. Both fixes ship as one commit because they surface together on the same user path (wizard → research loop → first iteration) — you can't hit one without the other manifesting. Follow-up: start_topic still runs its own inline clone/spawn code (now duplicated with prepare_topic_runtime). Next commit collapses start_topic to just call prepare_topic_runtime + build_task like the loop path does, so the one-shot and loop paths agree on setup. |
||
|
Omar Sobh
|
3e42b1ea39 |
research-wizard: schedule step (once/nightly/manual) + paired coding loop
Completes the research/loop fold. The wizard now closes with a
"How should this research run?" step; picking any mode materializes a
kind='research' loop bound to the topic, and an optional checkbox
adds a paired kind='exec' loop that consumes each new artifact.
Frontend:
- ResearchWizard grows from 5 to 6 steps. Step 6 is the schedule
picker:
· Just once — initial_burst=1, no other triggers
· Nightly — initial_burst=1 + cron "0 3 * * *"
· Manual — webhook_enabled=true
Below the radios, an optional card offers "Also create a coding
loop that consumes each new artifact" — creates a paired
kind='exec' loop with on_artifact_update=true + initial_burst=1
bound to the same topic.
- createTopic API type extended with `schedule` + `create_paired_coding_loop`.
- Both fields ride the existing POST /api/research call; back-compat
is preserved when the wizard omits them.
Backend:
- CreateTopicRequest gains TopicSchedule + create_paired_coding_loop.
- After topic + agent attach, create_topic calls
materialize_topic_loops which:
1. Creates a research-kind loop titled "Research · <topic>" bound
to the topic. Triggers vary by schedule mode; next_fire_at
computed from cron for nightly. Falls back silently if
loop-create errors so the topic still lands.
2. Flips kind to 'research' via loops::set_kind (NewLoop doesn't
take kind directly — default is 'exec' for backward compat).
3. Optionally creates a coding loop titled "Coding · <topic>"
with on_artifact_update=true + initial_burst=1.
- cm_db::repo::loops::set_kind — trivial UPDATE helper used by the
materialize path.
D-answer callouts:
- D1 (fold): every runnable thing is now a loop. "Just once" is a
research loop with initial_burst=1 and no other triggers.
- D2 (inherit): both paired loops carry the same source_research_topic_id
— repo binding lives on the topic, not duplicated.
- D3 (coordinator resolves): the research iteration prompt (from the
earlier commit) instructs the team to preserve stable INT ids and
mark deprecations; coding loops' consumed lists stay valid across
versions.
Follow-ups queued:
- Kind pill on LoopsList cards (research=purple, exec=cyan) so users
can tell them apart at a glance.
- Extract start_topic's task-build so kind='research' iterations
reuse the same coordinator prompt shape as one-shot runs (they
currently use a simpler refresh-oriented prompt; that's fine for
MVP but a rich shared build would give better parity).
- Research topic sidebar shows "linked to N loops" badge.
|
||
|
Omar Sobh
|
91a51dce11 |
loops: kind='research' dispatch — research runs as loops
Delivers the research/loop fold: kind='research' loops run the research pipeline each iteration, appending a new research_outcomes version. The paired on_artifact_update fan-out then wakes up any kind='exec' loops bound to the same topic to consume new INTs. Every runnable thing is now a loop (D1). Backend — DB helpers: - cm_db::repo::loops::kind_and_binding — reads (kind, source_topic, task_template) so callers can dispatch without hydrating the whole Loop struct. - cm_db::repo::loops::enqueue_iteration_with_topic — new variant that sets research_topic_id on topology_runs alongside loop_id, so the completion hook's freeze_research_outcome writes a new outcome version for research-kind iterations. - cm_db::repo::research_topics::get_any_workspace — cross-workspace fetch used by the research task builder (the loop row is authoritative for the workspace binding via kind_and_binding). Backend — dispatch: - routes::loops::compose_research_iteration_task — builds the coordinator prompt for a research iteration: topic title + description + outcome_kind + prior version pointer + refresh instructions (survey new sources, preserve stable INT ids, mark superseded items as deprecated rather than delete). The completion hook writes the resulting synthesis as research_outcomes v(prior+1). - routes::loops::compose_and_enqueue_iteration — one-shot dispatch: reads the kind, picks compose_iteration_task (exec) or compose_research_iteration_task (research), enqueues with or without research_topic_id set. All four enqueue callsites now route through compose_and_enqueue: - create_loop (initial_burst) - run_now - webhook_receive - topology_worker::continue_initial_burst - topology_worker::freeze_research_outcome (on_artifact_update fan-out) Research-kind loops naturally form the "nightly refresh" side of a paired research + coding loop: research writes a fresh outcome version → fan-out wakes exec loops with on_artifact_update → coding loops consume the next INT (which the research loop may have just added). D2 answer (inherit repo binding): repo lives on the topic; both loops sharing the source topic id read from the same context, no duplication. D3 answer (coordinator resolves): the research iteration prompt tells the team to preserve stable INT ids and mark deprecations rather than delete, so coding loops' consumed lists stay valid across versions. Follow-up (next commit): ResearchWizard schedule step — "Just once / Nightly / Manual" that creates the paired research-kind loop with initial_burst=1 (just once) or cron 0 3 * * * (nightly) + optional paired coding loop with on_artifact_update. |
||
|
Omar Sobh
|
4ada5557f2 |
loops: kind column + initial_burst + on_artifact_update trigger fan-out
Foundation for folding research into loops as a first-class kind.
This commit ships the plumbing; the research-kind dispatch itself
lands next. Behavior for existing exec-kind loops is unchanged unless
they opt into the new trigger fields.
Migration 0044:
- kind TEXT NOT NULL DEFAULT 'exec' CHECK ('exec' | 'research'). New
research-kind will run the research pipeline each iteration (next
commit); 'exec' preserves today's behavior.
- initial_burst_remaining INT NOT NULL DEFAULT 0 — countdown for the
triggers.initial_burst quota. Decremented CAS-safely on each
completion until it hits 0.
- Two partial indexes: (kind, source_research_topic_id) for kind-
aware lookups, and (source_research_topic_id) filtered on
on_artifact_update=true + enabled=true for the fan-out hook.
Trigger schema extended with two optional fields:
- initial_burst: N — fire N iterations back-to-back at create time.
create_loop enqueues the first iteration inline (subject to
empty-roster gate), sets remaining=N-1, and the completion hook
continues the chain until exhausted.
- on_artifact_update: true — when a bound research_outcomes row is
inserted for the source topic, wake up one iteration of this loop.
Coalesced against has_active_run so a burst of rapid revisions
doesn't queue duplicates.
Backend:
- cm_db::repo::loops helpers (all dynamic sqlx, no offline cache
regen needed):
- set_initial_burst_remaining
- take_initial_burst_slot (CAS UPDATE returning prev value; 0 on
exhausted or race loss)
- loops_awaiting_topic (fan-out query: kind=exec + enabled +
on_artifact_update=true bound to the given topic)
- has_active_run (queued|running iteration existence check)
- get_any_workspace (bypasses the workspace scope guard; used by
the completion hook where the run row is authoritative)
- routes/loops::compose_iteration_task made pub so the completion
hook can build the same enriched task string as run_now.
- topology_worker::freeze_research_outcome now fans out to awakened
loops after the outcome insert, using compose_iteration_task and
coalescing on has_active_run.
- topology_worker::continue_initial_burst runs on every completion:
· take_initial_burst_slot (CAS) — no-op if already exhausted
· has_active_run coalesce guard
· re-fetches the loop via get_any_workspace + compose_iteration_task
· enqueues via loops::enqueue_iteration with parent_run_id set
Follow-ups already queued:
- kind='research' dispatch in run_job — build the research
coordinator task from the topic config, run the research pipeline
each iteration. Requires factoring start_topic's task-build.
- ResearchWizard "When should this run?" step (Just once / Nightly /
Manual) creating the topic + paired research-kind loop.
- LoopsWizard trigger UI matching the design proposal (burst count,
cron, on-artifact checkbox).
|
||
|
Omar Sobh
|
ce111273bb |
loops: reorder history collapsed under the progress pill
Completes the reorder rationale loop — the previous commit captured
REORDER: markers server-side but nothing surfaced them. Now the loops
sidebar renders a small "N reorders ▸" button under the progress pill
whenever the loop has any reorder events. Click expands to a
newest-first list of "iter <n> · <rationale>" lines so a reviewer can
see, at a glance, when the plan was adjusted and why.
Backend:
- LoopProgress DTO gains recent_reorders: Vec<Value> — newest-first,
capped at 5 so the card stays compact. Full history remains on the
loop row's reorder_events column.
- cm_db::repo::loops::recent_reorders — reads the jsonb array, returns
the last N in newest-first order.
- list_progress populates it per loop.
Frontend:
- LoopReorderEvent + recent_reorders on LoopProgress type.
- LoopsList tracks openHistoryId per-loop (one open at a time).
- Card renders history button + expanded panel styled to match the
progress pill above.
Notes:
- The event object schema is {run_id, iteration, text, ts}. Fields are
optional in the TS type so future schema tweaks don't break the
render.
- 5-item cap chosen so the sidebar card doesn't grow unbounded. If a
loop accumulates a lot of reorders, follow-up UI can render the full
history on the loop detail page.
|
||
|
Omar Sobh
|
0c17de52dd |
loops: reorder rationale extraction — REORDER: markers logged per iteration
Coordinator can now log WHY it worked on an INT-XX out of order
("REORDER: INT-05 before INT-04 because prereq X is unmet") and the
completion hook captures each rationale as an append-only event on
the loop. Sets up a reviewable timeline of when the plan was
adjusted, independent of the underlying `consumed_int_ids` advance.
Migration 0043:
- loops.reorder_events JSONB NOT NULL DEFAULT '[]'::jsonb — append-
only array of {run_id, iteration, text, ts}. Kept on the loop row
(rather than a dedicated table) so the mini-timeline is one read
away from the loop card.
Backend:
- topology_worker::parse_reorder_rationale — line matcher symmetric
with parse_completed_int_ids. Tolerates list dashes / prefixes /
markdown emphasis; case-insensitive marker match, preserves case of
the rationale text.
- cm_db::repo::loops::append_reorder_event — one INSERT-like append
per rationale, uses jsonb_build_object with postgres now() so ts is
wall-clock canonical (no client-clock skew).
- topology_runs::iteration_for_run — new helper so events carry the
iteration index.
- routes::loops::compose_iteration_task — coordinator prompt now
explicitly asks for `REORDER: <one-sentence>` at the top of the
first substantive turn when working out of order, AND spells out
that both markers must appear literally with colons (no bold, no
code fence) so the line parser doesn't miss them.
Non-loop and standalone-loop runs are unaffected — the hook only
fires when the run belongs to a source-bound loop.
Follow-up: expose reorder_events on the loops list endpoint + render
a small collapsed timeline on the LoopsList card.
|
||
|
Omar Sobh
|
a34be33261 |
loops: N/M INTs progress pill on source-bound loop cards
Users couldn't see how a research-bound loop was progressing through
its integration plan — the consumed_int_ids state existed in the DB
but nothing surfaced it. Now each loop card in the sidebar shows a
cyan pill "3/8 INTs" + source topic title + a thin progress bar, when
the loop is bound to a research topic.
Backend:
- GET /api/loops/progress — bulk read for every loop with a source
topic. Returns {loop_id, source_topic_id, source_topic_title,
source_outcome_version, consumed_count, total_int_count,
current_int_index} per loop. Standalone loops are omitted.
- count_int_ids parses unique INT-<number> ids out of the source
outcome's markdown — same permissive matcher as the completion
hook, so what the pill counts matches what the loop can advance.
- Memoized by topic_id inside the endpoint so N loops sharing 1
source topic only fetch the outcome once.
Frontend:
- listLoopProgress helper + LoopProgress type in the loops API.
- LoopsList fetches loops + progress in parallel on mount.
- Each card looks up progress by loop_id and, when found, renders
under the schedule line: pill "3/8 INTs · Topic title" with a
3px cyan progress bar. Title hover shows artifact version.
Follow-ups:
- Refresh button on the card — re-snapshot artifact into
task_template (cosmetic; the enqueue path already reads latest).
- Reorder rationale extraction — parse "REORDER: <text>" out of run
output, index as a per-loop event log for a mini-timeline UI.
|
||
|
Omar Sobh
|
ce73abe5ab |
loops: bridge research artifact into loop iterations (option C + b)
The bridge lets a coding loop "consume" an integrations research artifact one INT-XX item per iteration. Options b (order-sequential iteration) and C (snapshot in task_template + save the pointer for future refresh) from the design discussion. Migration 0042 — three new loops columns: - source_research_topic_id — nullable pointer to research_topics. - consumed_int_ids TEXT[] — INT-XX ids the loop has completed. Advances when topology_worker parses "COMPLETED: INT-<NN>" markers from the run's final output (wired in a follow-up commit). - current_int_index INT — monotonic pointer for order-sequential iteration. Coordinator addresses INT-<current+1> unless prereqs are unmet, in which case it works on the smallest unblocking INT-XX and logs the reorder rationale. Backend: - cm_db::repo::loops::set_source_research_topic — bind/unbind pointer. - cm_db::repo::loops::source_research_context — read pointer + state. - routes::loops::compose_iteration_task — new caller-side helper that reads the pointer, fetches the topic's latest research_outcome, and prepends the artifact + focus instruction to task_template. - run_now + webhook_receive both pass task_template through compose_iteration_task before enqueue. Standalone loops (no pointer) behave identically to before. - CreateLoopRequest accepts `source_research_topic_id`, ownership- checked via research_topics::get before persist. Frontend: - New ResearchArtifactPicker modal — lists published topics, fetches the artifact on pick, returns (topic_id, markdown) to caller. - LoopsWizard task_template step gains "Import from research artifact" button (right-aligned). Click opens the picker. On pick: task populates with the artifact markdown, pointer saved, textarea expands to 8 rows, small info strip shows "Loop is bound to topic <id>. Each iteration will focus on the next unconsumed INT-XX." - Unlink button reverts to standalone loop mode. Follow-up (next commit): - topology_worker completion hook — parse "COMPLETED: INT-<NN>" out of the run's final output + update consumed_int_ids + current_int_index atomically. Without this, current_int_index stays at 0 forever and every iteration works on the same INT. - Loop card refresh button — re-read source topic's latest outcome (useful after a reject-with-revision cycle on the source topic). |
||
|
Omar Sobh
|
3ed1d03d2b |
research: integrations outcome + rich wizard cards + coordinator template
Adds a fifth outcome kind ('integrations') tuned for the "audit repo,
survey papers, propose a menu of concrete integrations" use case.
Every INT-XX item is self-contained (what, how, where, prereqs, effort,
risk, testing, rollback, acceptance) so a downstream loop can execute
one per iteration.
Backend:
- Migration 0041 drops + re-adds the outcome_kind CHECK constraint
with 'integrations' allowed. Existing rows unaffected.
- VALID_OUTCOMES gains 'integrations'.
- New deliverable_template(kind) returns the canonical section shape
for each outcome — spec, prod_plan, roadmap, paper, integrations all
get first-class treatment (prior: all shared a bare label).
- build_coordinator_task injects an ARTIFACT SHAPE block from the
template into the coordinator prompt, so the final synthesis
actually matches the promise the wizard made.
Frontend:
- OutcomeKind gains 'integrations'.
- ResearchWizard OUTCOMES list carries a `sections` array per kind.
- Selected card renders an "ARTIFACT WILL CONTAIN" preview so users
pick by seeing what they'll get, not by reading a one-line hint.
- Integrations card gets the fullest preview (executive summary +
INT-XX card shape) since it's the most structured deliverable.
Follow-ups queued (next commit): loop wizard "Import from research
artifact" bridge + one-INT-per-iteration mode.
|
||
|
Omar Sobh
|
3bcd18bf50 |
research: split pipeline diagnostics into research_pipeline.rs
Gates step failed on run#211 because research.rs grew to 1313 lines (over the 1250 budget) after adding the pipeline_state handler in the previous commit. Move the diagnostic into its own module — pure extraction, no behavior change. research.rs: 1313 → 1126 lines research_pipeline.rs: new, 187 lines lib.rs: route now points at routes::research_pipeline::pipeline_state |
||
|
Omar Sobh
|
cca6e2be20 |
research: pipeline diagnostics + refuse publish without outcome
Two fixes surfaced by the first prod run of the pipeline:
1) Silent skip-to-published bug (R1 gap):
approve_publish transitioned reviewing → publishing → published
without checking that an outcome existed. Result: pipeline could
fail silently (LLM auth error, network, etc), no outcome would be
written, but state advanced to 'published' and the download endpoint
returned 404 with no user-visible error. Now refuses with 409
Conflict when no outcome exists so the frontend can surface WHY.
2) No end-to-end visibility:
Users had no way to see where a run failed until they clicked
Download and got nothing. Adds
GET /api/research/:id/pipeline-state — a read-only per-stage report
walking:
- staffing (agents assigned)
- repo (bound + cloned)
- container (per-topic team runtime spawned)
- runs (count + failed count + latest error text)
- outcomes (count — the artifact rows get_artifact reads)
- approval (pending flag)
Each stage returns ok / warn / fail / skip plus optional detail text
so the failure reason surfaces at the diagnostic level.
Frontend:
ResearchCanvas shows a compact PIPELINE strip below the topic title,
green/amber/red dots per stage, click to expand a full checklist
with per-stage detail (including the LLM error from the last run
attempt). Polls every 6s while the topic is processing/publishing.
Follow-up:
- Root cause of the specific failure just observed: Claude CLI in
the clawmates-runtime container isn't authenticated. Deploy-side
config sweep (CLAUDE_CODE_OAUTH_TOKEN or ANTHROPIC_API_KEY into
the runtime image env), not a code fix.
- Structured event stream on top of run_events for real per-step
replay in the diagnostic panel.
|
||
|
Omar Sobh
|
44d6e95022 |
fmt: apply rustfmt across the P2 arc
CI's rustfmt check flagged the multi-line sqlx::query() calls I introduced in P2 (loop_id_for_run, zeroclaw_gateway_url, etc.). No behavior change — pure formatting. |
||
|
Omar Sobh
|
03f1830d1f |
loops: Path B container isolation (P2)
Symmetric with the research pipeline: every enabled loop can now have its own per-loop team container so scheduled runs don't share state with other loops or with research. Same daemon image, same clawmates network, deterministic name loop-<id>-team. Backend surface: - Migration 0040 adds nullable `zeroclaw_container` + `zeroclaw_gateway_url` columns to loops (parallel to research_topics). - research_container.rs grows loop_container_name_for(), spawn_loop() (state-only mount, no repo), and teardown_loop(). Kept in the same module to share the docker connect() + inherited_env() plumbing; each pattern gets its own labels (clawmates.role=loop-team) so ps filters can tell them apart. - cm_db::repo::loops gains set_zeroclaw_container() + zeroclaw_gateway_url() (dynamic sqlx queries — no offline cache regen needed). - cm_db::repo::topology_runs gets loop_id_for_run(): mirror of research_topic_id, used by the worker. Wiring: - routes/loops::run_now + webhook_receive call ensure_loop_container() before enqueuing an iteration. Idempotent: an already-running container is just reattached. Failures are logged and do NOT block the enqueue — topology_worker falls back to the workspace gateway when the URL isn't set on the loop. - routes/loops::disable_loop + delete_loop both fire teardown_loop() so paused / deleted loops don't hold a docker slot. - topology_worker's per-run URL resolution: existing research fast path unchanged; when it doesn't hit, the worker now looks up loop_id and reads the loop's gateway URL. Deploy step (required on gw-04 for state to persist across container restarts): add a `/var/lib/clawmates-loops:/var/lib/clawmates-loops` bind mount + `CLAWMATES_LOOPS_STATE_ROOT=/var/lib/clawmates-loops` env var to clawmates_server_1 in the compose. Without it, loops still run — the state dir lives inside the API container's filesystem so persistence is limited to that container's lifetime. Follow-up: - Scheduler-tick fires (cron-driven, not run_now) — they call enqueue_iteration in cm-scheduler and don't yet go through ensure_loop_container. Add a symmetric spawn there so cron fires also land on the isolated daemon. - Compose file reconciliation — deploy/compose/docker-compose.yml in the repo has drifted from prod; when we sync it, add the loops mount at the same time. |
||
|
Omar Sobh
|
e3011ed025 |
research: reject-with-revision loop (R2)
Before: reviewer rejected a publish → audit log flipped, topic stayed in reviewing, no way to feed the critique back into the run pipeline. Reviewers with revision notes had to eat them or hand-message the coordinator. Now: reject accepts an optional `notes` field. When present: - Persisted on the research_publish_approvals row (migration 0039). - Topic flips `reviewing → standby` so the next `start_topic` is legal. - `start_topic` reads the most recent rejected-approval notes for the topic and prepends "PRIOR REVIEW NOTES (address these in this revision):\n<notes>\n---" to the coordinator task. Loop closes through the same run pipeline — no new spawn code path, which means the reviewer's guidance flows through the same topology_worker, run_events, outcome-writer chain and lands as a fresh research_outcomes row (versioned, prior drafts preserved). No notes on reject = legacy behavior (topic stays in reviewing, publish requests still allowed). Migration 0039 adds nullable `notes TEXT` to research_publish_approvals. `decide()` gains a `notes: Option<&str>` parameter (only one caller, updated inline). New `latest_rejection_notes(pool, topic_id)` helper for start_topic. Frontend: - rejectPublish(id, notes?) now sends a JSON body when notes are provided. - ResearchCanvas reject button opens an inline form with a textarea + Cancel/"Send back for revision" pair. Empty notes → plain reject. - Button label switches: "Send back for revision" when notes present, "Reject without notes" when empty. Follow-up: - Notes shown in the review UI on the resulting draft so the next reviewer sees what changed. - Multiple rejection rounds — currently only the LATEST rejection's notes surface. Accumulating history is a schema-only tweak. |
||
|
Omar Sobh
|
a49cf86623 |
world: pre-seed repo tree on clone (V3)
Repo focus mode used to open into an empty tree — the file/dir nodes only synthesized as agents touched files via tool calls. Cold-start users saw a lonely repo:<id> orb with nothing under it. Now the tree pre-seeds from the actual cloned repo the moment a client subscribes. Backend: - active_research_topics also returns repo_workspace_path. - preseed_repo_paths(clone_path) runs `git ls-files` (bounded to top 200) against the on-disk clone. Silently returns empty on any failure so a missing clone / git-off-PATH / empty repo just degrades to the pre-V3 behavior (tree still builds on touch). - SSE loop, on first sight of a topic per client, emits one node.activity per pre-seeded path (label = leaf name, heat 0) so the tree is quiet-solid at rest. Frontend: - engine.onNodeActivity now synthesizes the dir:<partial> chain for file: nodeIds the same way onTouch does — otherwise the pre-seed would render as flat leaves under ROOT. - Same 5-line synthesis extracted from onTouch; both paths now agree on the layout. Cap of 200 keeps the SSE payload bounded on huge repos; the tail fills in as agents actually touch files. When we later add per-file heat map (V4), the 200 already-known files get first-class treatment out of the gate. |
||
|
Omar Sobh
|
9e5034cb53 |
wizards: ensure-chain preflight in AddToTeam + AddToCompany (W1)
TeamWizard already ran ensure-chain before /api/teams; the two "AddTo"
modals didn't, so teams/companies created from them landed as
structural orphans (no company/org parent). Same treatment now applied
to both modals + their backend endpoints.
Backend — two symmetric `attach_to_*_id` fields (mirrors what
create_team already exposes):
- ComposeTeamRequest gains `attach_to_company_id`. After team insert,
create_team_from_claws binds it via companies::add_team with a fresh
`n{count}` node id.
- CreateCompanyRequest gains `attach_to_org_id`. After company insert,
create_company binds it via orgs::add_company the same way.
Both bindings are optional — plain POSTs from tools/tests still work.
Ownership is re-checked via `<parent>::get(pool, id, workspace_id)` so
the endpoints can't be tricked into parenting into another workspace.
Frontend — both modals now:
1. POST /api/structure/ensure-chain (empty body → server picks
"My Workspace" / "General" fallbacks when nothing exists yet).
2. Include the returned parent id in the create request.
AddToOrgModal untouched — orgs are top-level, no parent needed.
MasterPlannerModal untouched — it posts to /webhooks, doesn't create
structural rows.
Follow-up already queued in the original list: same treatment for the
Company/Org "wizard"-flavored surfaces (as opposed to the compose
modals). Currently those don't exist as distinct wizards.
|