feat(judge): the judge commits to a verification plan before it reads the evidence
One tool-free round on the condition alone: which files, strings and tests would show MET, and which commands would settle it. The plan is placed between the condition and the evidence in the verifying prompt and stored as mission_phase_evaluations.expectation beside the verdict, so an operator can see whether the checks the judge ran are the ones it said it would run. Self-Play Reward Hacking of Reference-Free Judges (arXiv 2607.05904): a judge's pass rate climbed 0.72 -> 0.94 across rounds while accuracy stayed 0.20; cross-family judges and ensembles did not help; the judge committing its own answer first cut the false-positive rate 0.719 -> 0.012. Ours commits to a plan, not a value — re-deriving values is the failure done-when-wording measured, and the commit prompt forbids it. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
This commit is contained in:
co-authored by
Claude Opus 5
parent
ef1f21024d
commit
1fc6cb41ba
@@ -1336,7 +1336,7 @@ pub async fn list_phase_evaluations(
|
||||
.ok_or(ApiError::NotFound)?;
|
||||
use sqlx::Row;
|
||||
let rows = sqlx::query(
|
||||
"SELECT iteration, met, reason, model, error, created_at, checks
|
||||
"SELECT iteration, met, reason, model, error, created_at, checks, expectation
|
||||
FROM mission_phase_evaluations
|
||||
WHERE mission_id = $1 AND phase_id = $2
|
||||
ORDER BY iteration DESC",
|
||||
@@ -1359,6 +1359,10 @@ pub async fn list_phase_evaluations(
|
||||
// empty list means the verdict rests on agent claims
|
||||
// alone, which an operator should be able to see.
|
||||
"checks": r.get::<serde_json::Value, _>("checks"),
|
||||
// What the judge said it would check BEFORE it read the
|
||||
// evidence; set beside `checks` so an operator can see
|
||||
// whether it kept to its plan.
|
||||
"expectation": r.get::<Option<String>, _>("expectation"),
|
||||
"created_at": created_at
|
||||
.format(&time::format_description::well_known::Rfc3339)
|
||||
.unwrap_or_default(),
|
||||
|
||||
Reference in New Issue
Block a user