feat(judge): the judge commits to a verification plan before it reads the evidence

One tool-free round on the condition alone: which files, strings and tests
would show MET, and which commands would settle it. The plan is placed
between the condition and the evidence in the verifying prompt and stored
as mission_phase_evaluations.expectation beside the verdict, so an operator
can see whether the checks the judge ran are the ones it said it would run.

Self-Play Reward Hacking of Reference-Free Judges (arXiv 2607.05904): a
judge's pass rate climbed 0.72 -> 0.94 across rounds while accuracy stayed
0.20; cross-family judges and ensembles did not help; the judge committing
its own answer first cut the false-positive rate 0.719 -> 0.012. Ours
commits to a plan, not a value — re-deriving values is the failure
done-when-wording measured, and the commit prompt forbids it.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
This commit is contained in:
Omar Sobh
2026-09-20 22:07:39 -05:00
co-authored by Claude Opus 5
parent ef1f21024d
commit 1fc6cb41ba
4 changed files with 190 additions and 8 deletions
+5 -1
View File
@@ -1336,7 +1336,7 @@ pub async fn list_phase_evaluations(
.ok_or(ApiError::NotFound)?;
use sqlx::Row;
let rows = sqlx::query(
"SELECT iteration, met, reason, model, error, created_at, checks
"SELECT iteration, met, reason, model, error, created_at, checks, expectation
FROM mission_phase_evaluations
WHERE mission_id = $1 AND phase_id = $2
ORDER BY iteration DESC",
@@ -1359,6 +1359,10 @@ pub async fn list_phase_evaluations(
// empty list means the verdict rests on agent claims
// alone, which an operator should be able to see.
"checks": r.get::<serde_json::Value, _>("checks"),
// What the judge said it would check BEFORE it read the
// evidence; set beside `checks` so an operator can see
// whether it kept to its plan.
"expectation": r.get::<Option<String>, _>("expectation"),
"created_at": created_at
.format(&time::format_description::well_known::Rfc3339)
.unwrap_or_default(),