feat(judge): the judge commits to a verification plan before it reads the evidence
One tool-free round on the condition alone: which files, strings and tests would show MET, and which commands would settle it. The plan is placed between the condition and the evidence in the verifying prompt and stored as mission_phase_evaluations.expectation beside the verdict, so an operator can see whether the checks the judge ran are the ones it said it would run. Self-Play Reward Hacking of Reference-Free Judges (arXiv 2607.05904): a judge's pass rate climbed 0.72 -> 0.94 across rounds while accuracy stayed 0.20; cross-family judges and ensembles did not help; the judge committing its own answer first cut the false-positive rate 0.719 -> 0.012. Ours commits to a plan, not a value — re-deriving values is the failure done-when-wording measured, and the commit prompt forbids it. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
This commit is contained in:
co-authored by
Claude Opus 5
parent
ef1f21024d
commit
1fc6cb41ba
@@ -280,6 +280,7 @@ async fn evaluations_are_unique_per_iteration_and_upsert() {
|
||||
checks: Vec::new(),
|
||||
independent: false,
|
||||
usage: Default::default(),
|
||||
expectation: None,
|
||||
};
|
||||
cm_api::evaluator::record(&pool, mission, phase, 0, &first)
|
||||
.await
|
||||
@@ -300,6 +301,7 @@ async fn evaluations_are_unique_per_iteration_and_upsert() {
|
||||
}],
|
||||
independent: false,
|
||||
usage: Default::default(),
|
||||
expectation: Some("rg 'brief' docs/".into()),
|
||||
};
|
||||
cm_api::evaluator::record(&pool, mission, phase, 0, &second)
|
||||
.await
|
||||
@@ -330,6 +332,15 @@ async fn evaluations_are_unique_per_iteration_and_upsert() {
|
||||
.unwrap()
|
||||
.get("reason");
|
||||
assert_eq!(reason, "brief written");
|
||||
// The commit-first plan is stored beside the verdict it governed.
|
||||
let expectation: Option<String> =
|
||||
sqlx::query("SELECT expectation FROM mission_phase_evaluations WHERE phase_id = $1")
|
||||
.bind(phase)
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.unwrap()
|
||||
.get("expectation");
|
||||
assert_eq!(expectation.as_deref(), Some("rg 'brief' docs/"));
|
||||
}
|
||||
|
||||
/// `latest` must return the newest pass, which is what feeds guidance into the
|
||||
@@ -359,6 +370,7 @@ async fn latest_returns_the_most_recent_iteration() {
|
||||
checks: Vec::new(),
|
||||
independent: false,
|
||||
usage: Default::default(),
|
||||
expectation: None,
|
||||
},
|
||||
)
|
||||
.await
|
||||
|
||||
Reference in New Issue
Block a user