Omar Sobh
bb807c2f3a
Found by the Goodhart test for the independent judge, which is exactly what it was built to find. The test: a phase whose `done_when` demanded a passing suite, and a task that deliberately left a failing test. glm-4.7 judged it, ran `cargo test` itself, saw `parity_is_wrong_on_purpose ... FAILED` (exit 101), and returned met=false quoting the assertion — while the agent's own summary said "All three steps are implemented exactly as specified and independently verified". The verdict and the agent's account diverged, which is the whole point of an independent judge. And then the mission closed `completed`. `if verdict.met || last_pass` marked BOTH outcomes completed, so a phase that ran out of passes without ever meeting its condition reported success — and through `close_finished_missions`, so did the mission. The verdict said met=false in a column nobody reads before believing a green status. Anything consuming mission status rather than digging into the verdict saw a goal that was never reached as a goal achieved. Exhausted-and-unmet is now `failed`, and the log names the judge and whether it was independent. This changes observable behaviour: missions that would previously have finished green with an unmet condition now finish failed. That is the correction, not a regression — but it is worth knowing before the next scheduled run. Also: `Verdict.independent` had no column. The field existed in the struct and in the logs, so the audit question the mechanism exists to answer — was this checked by something other than the model that wrote it? — could not be asked of the database. Migration 0067 adds it, defaulting to false, which is the truth about every row written before now. Verified in production before the fix: glm-4.7, 4 checks all executed, the real cargo failure quoted, met=false. 475 tests pass, clippy clean. Note for whoever rebases: `sqlx::migrate!` embeds migrations at COMPILE time, so a new migration needs cm-db rebuilt (`touch crates/cm-db/src/lib.rs`) or the integration tests fail on a column that exists in the file and not in the binary.
17 lines
900 B
SQL
17 lines
900 B
SQL
-- Record whether a phase verdict came from a judge INDEPENDENT of the agent that
|
|
-- did the work — a different provider family, not merely a different model.
|
|
--
|
|
-- `Verdict.independent` existed in the struct and in the logs before this column
|
|
-- did, so the field was a claim held in memory and thrown away. The audit question
|
|
-- this whole mechanism exists to answer — "was this work checked by something other
|
|
-- than the model that produced it?" — could not be asked of the database.
|
|
--
|
|
-- Defaults to false, which is the truth about every row written before now: the
|
|
-- judge was Claude and so was the agent.
|
|
ALTER TABLE mission_phase_evaluations
|
|
ADD COLUMN IF NOT EXISTS independent boolean NOT NULL DEFAULT false;
|
|
|
|
-- Find the verdicts nobody independently checked.
|
|
CREATE INDEX IF NOT EXISTS mission_phase_evaluations_independent_idx
|
|
ON mission_phase_evaluations (independent);
|