test(harness): pick the done_when wording by measuring the judge, not arguing with it

The microvm scenario's judge assertion failed four runs straight. I blamed the
wording twice and rewrote it twice; the second rewrite made it worse. That was
guessing.

With scripts/judge-eval.sh in place the question is cheap to settle. Three
candidate conditions, three draws each, same evidence and same system prompt:

  "its second line is …"            MET  UNMET  MET     flaky
  "records the kernel version …"    MET  MET    UNMET   flaky
  "contains both … and …"           MET  MET    MET     stable

So it was never noise in general — it is a reproducible weakness with
POSITIONAL and EXCLUSIVE phrasings. "its second line is X and nothing else"
invites this judge to invent requirements about the other lines, which is
exactly the reason it kept citing ("the first line contains 'test result: ok'").

Both fixtures now state what the file CONTAINS. The composed one was checked in
both directions — 3/3 MET on good evidence, 3/3 UNMET when the versions are
missing — because a wording that always answers MET would look stable and prove
nothing.

The eval keeps `kernel-ok` failing on purpose; it is the case production hit,
and tuning it green would turn a measurement into a decoration.

Harness: 24/24, including the assertion that had failed four times.
This commit is contained in:
Omar Sobh
2026-08-07 08:54:00 -07:00
parent 5c066afa7b
commit cd4d76a8c3
2 changed files with 16 additions and 3 deletions
+14 -1
View File
@@ -12,7 +12,20 @@
# argument and trivial to settle by measurement. # argument and trivial to settle by measurement.
# #
# Cases are drawn from real incidents, each with a known answer a careful human # Cases are drawn from real incidents, each with a known answer a careful human
# would agree with. Usage: # would agree with. `kernel-ok` is EXPECTED TO FAIL on glm-4.7 today and is kept
# failing on purpose: it is the case production hit, and tuning it away would
# turn a measurement into a decoration. A non-zero exit means "not every case
# passes", which is the truth.
#
# MEASURED while choosing a fixture wording (3 draws each, same file, same
# system prompt): a POSITIONAL condition ("its second line is …") scored
# MET/UNMET/MET and a possessive one ("records the kernel version …") scored
# MET/MET/UNMET, while a CONTENT-EXISTENCE one ("contains both a test-result
# summary and a recorded kernel version") scored MET three times out of three.
# Conditions that name a line position or say "and nothing else" invite this
# judge to invent requirements about the other lines. Prefer "contains X and Y".
#
# Usage:
# #
# scripts/judge-eval.sh # the configured validator # scripts/judge-eval.sh # the configured validator
# JUDGE=kimi scripts/judge-eval.sh # compare another # JUDGE=kimi scripts/judge-eval.sh # compare another
+2 -2
View File
@@ -268,7 +268,7 @@ MICROVM_BODY=$(cat <<JSON
"description":"Prove a coding phase executes in a Firecracker microVM and can delegate.", "description":"Prove a coding phase executes in a Firecracker microVM and can delegate.",
"phases":[ "phases":[
{"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1, {"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1,
"done_when":"A file named MICROVM.md exists at the repository root and its second line is a bare kernel version of the form MAJOR.MINOR.PATCH (for example 6.1.128) and nothing else. Judge only what the file contains — do NOT run uname yourself and do NOT compare the value against the machine you are running on; the file records a DIFFERENT machine.", "done_when":"MICROVM.md exists at the repository root and contains both a test-result summary and a recorded kernel version such as 6.1.128.",
"task":"1. Use the verifier subagent to run the COMPLETE test suite of this crate and report what it found. Do not run it yourself and report that instead — the point is an independent check.\n2. Write MICROVM.md at the repository root with exactly two lines: the first is the test-result summary line the verifier reported, the second is the kernel release from running uname -r.\nCreate no other files."}} "task":"1. Use the verifier subagent to run the COMPLETE test suite of this crate and report what it found. Do not run it yourself and report that instead — the point is an independent check.\n2. Write MICROVM.md at the repository root with exactly two lines: the first is the test-result summary line the verifier reported, the second is the kernel release from running uname -r.\nCreate no other files."}}
]} ]}
JSON JSON
@@ -389,7 +389,7 @@ COMPOSED_BODY=$(cat <<JSON
"description":"Prove a ZeroClaw graph of microVM nodes carries file work between its nodes.", "description":"Prove a ZeroClaw graph of microVM nodes carries file work between its nodes.",
"phases":[ "phases":[
{"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1, {"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1,
"done_when":"STAGES.md exists at the repository root and its last line ends with a bare kernel version of the form MAJOR.MINOR.PATCH (for example 6.1.128). Judge only what the file contains — do NOT run uname yourself; the file records a DIFFERENT machine than the one you are on.", "done_when":"STAGES.md exists at the repository root and contains one line per stage, each recording a stage name and a kernel version such as 6.1.128.",
"task":"Append EXACTLY ONE line to STAGES.md at the repository root, creating the file if it does not exist. The line is: your stage name, a space, and the output of \`uname -r\`.\n\nDo NOT rewrite, reorder or remove any line already in the file — earlier stages wrote those, and they are the record of this run. If the file already has lines, yours goes after them. Change no other file."}} "task":"Append EXACTLY ONE line to STAGES.md at the repository root, creating the file if it does not exist. The line is: your stage name, a space, and the output of \`uname -r\`.\n\nDo NOT rewrite, reorder or remove any line already in the file — earlier stages wrote those, and they are the record of this run. If the file already has lines, yours goes after them. Change no other file."}}
]} ]}
JSON JSON