feat(judge): the judge commits to a verification plan before it reads the evidence

One tool-free round on the condition alone: which files, strings and tests
would show MET, and which commands would settle it. The plan is placed
between the condition and the evidence in the verifying prompt and stored
as mission_phase_evaluations.expectation beside the verdict, so an operator
can see whether the checks the judge ran are the ones it said it would run.

Self-Play Reward Hacking of Reference-Free Judges (arXiv 2607.05904): a
judge's pass rate climbed 0.72 -> 0.94 across rounds while accuracy stayed
0.20; cross-family judges and ensembles did not help; the judge committing
its own answer first cut the false-positive rate 0.719 -> 0.012. Ours
commits to a plan, not a value — re-deriving values is the failure
done-when-wording measured, and the commit prompt forbids it.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
This commit is contained in:
Omar Sobh
2026-09-20 22:07:39 -05:00
co-authored by Claude Opus 5
parent ef1f21024d
commit 1fc6cb41ba
4 changed files with 190 additions and 8 deletions
+12
View File
@@ -280,6 +280,7 @@ async fn evaluations_are_unique_per_iteration_and_upsert() {
checks: Vec::new(),
independent: false,
usage: Default::default(),
expectation: None,
};
cm_api::evaluator::record(&pool, mission, phase, 0, &first)
.await
@@ -300,6 +301,7 @@ async fn evaluations_are_unique_per_iteration_and_upsert() {
}],
independent: false,
usage: Default::default(),
expectation: Some("rg 'brief' docs/".into()),
};
cm_api::evaluator::record(&pool, mission, phase, 0, &second)
.await
@@ -330,6 +332,15 @@ async fn evaluations_are_unique_per_iteration_and_upsert() {
.unwrap()
.get("reason");
assert_eq!(reason, "brief written");
// The commit-first plan is stored beside the verdict it governed.
let expectation: Option<String> =
sqlx::query("SELECT expectation FROM mission_phase_evaluations WHERE phase_id = $1")
.bind(phase)
.fetch_one(&pool)
.await
.unwrap()
.get("expectation");
assert_eq!(expectation.as_deref(), Some("rg 'brief' docs/"));
}
/// `latest` must return the newest pass, which is what feeds guidance into the
@@ -359,6 +370,7 @@ async fn latest_returns_the_most_recent_iteration() {
checks: Vec::new(),
independent: false,
usage: Default::default(),
expectation: None,
},
)
.await