Omar SobhandClaude Opus 5
f848248fac
A phase used to complete when its topology_runs reached a terminal state --
purely structural. It marked itself done whether the agents produced the
artifact or wrote nothing at all, and it ran exactly once: execute_resumable's
skip(start) is resume, not repeat, and the only re-run path was a human
hitting the retry endpoint.
A phase can now carry `done_when`, a completion condition judged after each
pass against the evidence the agents actually surfaced. Not met and passes
remain -> the phase goes back to pending with iteration bumped, and the
verdict's reason is appended to the next pass's task text. That feedback is
what makes iteration converge rather than repeat -- the same mechanism /goal
uses, and that swarm.rs already uses for rejected work.
The evaluator runs on the SUBSCRIPTION model. CLAWMATES_EVALUATOR_MODEL
defaults to judge_model(), and a `runtime:<alias>` spec routes through
ZeroClawDriveExecutor -- a container agent on claude_cli, i.e. Claude Code on
the OAuth subscription, needing no platform API key. Same routing the door
governor uses.
Two deliberate departures from the governor's contract, both required:
- FAIL-CLOSED. Runtime::judge is fail-open and reads a verdict by
!contains("DENY"), so a model explaining why it *would* deny reads as
approval and an empty reply reads as approval. For completion that is
backwards: unsure must mean not done. The contract is swarm.rs's strict
JSON {"met","reason"} with .unwrap_or(false). Six tests cover the closed
paths -- prose, empty, missing field, non-boolean, transport error.
- judge_raw returns the raw reply; judge collapses to a bool too early to
carry a structured verdict.
Iteration scoping is the subtle part and has its own test: on pass 2 the
phase's own iteration is 1 but pass 1's completed run is still in the table,
so "are this phase's runs all finished?" must ask about the CURRENT pass or
that stale row closes out pass 2 the instant it is enqueued.
Evidence comes from phase_summarizer::collect_evidence, extracted from the
existing collect_material so the evaluator and the summary card cannot
disagree about what a phase produced.
done_when/max_iterations are promoted from phase config into columns (the
sweep filters on them every tick) and max_iterations is clamped to 20 at
insert -- the UI limits it too, but a runaway loop must not be one crafted
request away.
A phase with no condition completes exactly as before; that regression guard
is the first test in the file.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
66 lines
3.1 KiB
SQL
66 lines
3.1 KiB
SQL
-- Goal conditions for mission phases.
|
|
--
|
|
-- Until now a phase completed when its topology_runs reached a terminal
|
|
-- state -- purely structural, with no notion of whether the work was any
|
|
-- good. `close_finished_phases` marked a phase `completed` whether the
|
|
-- agents produced the artifact or wrote nothing at all.
|
|
--
|
|
-- `done_when` is a natural-language completion condition, judged after each
|
|
-- pass by a model (see crates/cm-api/src/evaluator.rs). It follows the
|
|
-- constraint the evaluator operates under: the judge cannot run commands, so
|
|
-- the condition must be demonstrable from what the agents surfaced in their
|
|
-- turn output.
|
|
--
|
|
-- NULL `done_when` preserves today's behaviour exactly: terminal runs ->
|
|
-- completed, no evaluation, no extra model spend. Existing missions are
|
|
-- unaffected.
|
|
|
|
ALTER TABLE mission_phases
|
|
-- The completion condition. NULL = no evaluation (legacy behaviour).
|
|
ADD COLUMN done_when TEXT,
|
|
-- Upper bound on passes. 1 = run once, matching current behaviour.
|
|
-- Capped server-side as well; this is a backstop against a runaway loop.
|
|
ADD COLUMN max_iterations INT NOT NULL DEFAULT 1,
|
|
-- Which pass the phase is on, 0-based.
|
|
ADD COLUMN iteration INT NOT NULL DEFAULT 0;
|
|
|
|
-- One verdict per (phase, iteration). Modelled on mission_phase_summaries
|
|
-- (0060): the model emits structured JSON, we persist it with the model name
|
|
-- so a verdict can be attributed, and keep the reason because it is both the
|
|
-- explanation shown to the operator AND the guidance fed into the next pass.
|
|
CREATE TABLE mission_phase_evaluations (
|
|
id UUID PRIMARY KEY,
|
|
mission_id UUID NOT NULL REFERENCES missions(id) ON DELETE CASCADE,
|
|
phase_id UUID NOT NULL REFERENCES mission_phases(id) ON DELETE CASCADE,
|
|
iteration INT NOT NULL,
|
|
-- Whether the condition held. Fail-closed: an unparseable or missing
|
|
-- verdict is recorded as false, never as "done".
|
|
met BOOLEAN NOT NULL,
|
|
reason TEXT NOT NULL,
|
|
-- The model that judged, e.g. "runtime:coordinator" or "claude-opus-4-8".
|
|
model TEXT NOT NULL,
|
|
-- Set when the evaluator itself failed (transport, parse). `met` is false
|
|
-- in that case; this distinguishes "judged not done" from "could not judge".
|
|
error TEXT,
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
UNIQUE (phase_id, iteration)
|
|
);
|
|
|
|
CREATE INDEX mission_phase_evaluations_phase_idx
|
|
ON mission_phase_evaluations (phase_id, iteration DESC);
|
|
CREATE INDEX mission_phase_evaluations_mission_idx
|
|
ON mission_phase_evaluations (mission_id, created_at DESC);
|
|
|
|
-- Which pass produced a run. Without this, "are this phase's runs all
|
|
-- finished?" matches pass 1's completed rows forever and a second pass would
|
|
-- be declared done the instant it was enqueued.
|
|
--
|
|
-- (An `iteration` column existed on this table once and was dropped in 0053
|
|
-- along with the legacy loops backend. This one is for mission phases.)
|
|
ALTER TABLE topology_runs
|
|
ADD COLUMN iteration INT NOT NULL DEFAULT 0;
|
|
|
|
CREATE INDEX topology_runs_phase_iteration_idx
|
|
ON topology_runs (mission_phase_id, iteration)
|
|
WHERE mission_phase_id IS NOT NULL;
|