Omar SobhandClaude Opus 5
1fc6cb41ba
One tool-free round on the condition alone: which files, strings and tests would show MET, and which commands would settle it. The plan is placed between the condition and the evidence in the verifying prompt and stored as mission_phase_evaluations.expectation beside the verdict, so an operator can see whether the checks the judge ran are the ones it said it would run. Self-Play Reward Hacking of Reference-Free Judges (arXiv 2607.05904): a judge's pass rate climbed 0.72 -> 0.94 across rounds while accuracy stayed 0.20; cross-family judges and ensembles did not help; the judge committing its own answer first cut the false-positive rate 0.719 -> 0.012. Ours commits to a plan, not a value — re-deriving values is the failure done-when-wording measured, and the commit prompt forbids it. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
20 lines
1.2 KiB
SQL
20 lines
1.2 KiB
SQL
-- What the judge expected to find, written down BEFORE it saw the agents' output.
|
|
--
|
|
-- Self-Play Reward Hacking of Reference-Free Judges (arXiv 2607.05904) measured
|
|
-- a judge's pass rate climbing 0.72 -> 0.94 across rounds while the answers
|
|
-- stayed 0.20 correct: a judge that reads the candidate first is argued into
|
|
-- the candidate's framing. Cross-family judges and ensembles did not help.
|
|
-- The one mitigation that did was making the judge commit to its own answer
|
|
-- before seeing the candidate (false-positive rate 0.719 -> 0.012).
|
|
--
|
|
-- Our analogue: one tool-free round on the condition alone, producing a
|
|
-- verification plan — which files, strings, tests would show MET — that the
|
|
-- verifying prompt then holds the judge to. Stored beside the verdict so an
|
|
-- operator can see whether the checks the judge ran are the ones it said it
|
|
-- would run, and NULL on every verdict written before the round existed.
|
|
ALTER TABLE mission_phase_evaluations
|
|
ADD COLUMN IF NOT EXISTS expectation TEXT;
|
|
|
|
COMMENT ON COLUMN mission_phase_evaluations.expectation IS
|
|
'The judge''s verification plan, committed from the condition alone before it read the evidence; NULL when the judge had no commit round.';
|