test(harness): pick the done_when wording by measuring the judge, not arguing with it
The microvm scenario's judge assertion failed four runs straight. I blamed the
wording twice and rewrote it twice; the second rewrite made it worse. That was
guessing.
With scripts/judge-eval.sh in place the question is cheap to settle. Three
candidate conditions, three draws each, same evidence and same system prompt:
"its second line is …" MET UNMET MET flaky
"records the kernel version …" MET MET UNMET flaky
"contains both … and …" MET MET MET stable
So it was never noise in general — it is a reproducible weakness with
POSITIONAL and EXCLUSIVE phrasings. "its second line is X and nothing else"
invites this judge to invent requirements about the other lines, which is
exactly the reason it kept citing ("the first line contains 'test result: ok'").
Both fixtures now state what the file CONTAINS. The composed one was checked in
both directions — 3/3 MET on good evidence, 3/3 UNMET when the versions are
missing — because a wording that always answers MET would look stable and prove
nothing.
The eval keeps `kernel-ok` failing on purpose; it is the case production hit,
and tuning it green would turn a measurement into a decoration.
Harness: 24/24, including the assertion that had failed four times.
This commit is contained in:
+14
-1
@@ -12,7 +12,20 @@
|
|||||||
# argument and trivial to settle by measurement.
|
# argument and trivial to settle by measurement.
|
||||||
#
|
#
|
||||||
# Cases are drawn from real incidents, each with a known answer a careful human
|
# Cases are drawn from real incidents, each with a known answer a careful human
|
||||||
# would agree with. Usage:
|
# would agree with. `kernel-ok` is EXPECTED TO FAIL on glm-4.7 today and is kept
|
||||||
|
# failing on purpose: it is the case production hit, and tuning it away would
|
||||||
|
# turn a measurement into a decoration. A non-zero exit means "not every case
|
||||||
|
# passes", which is the truth.
|
||||||
|
#
|
||||||
|
# MEASURED while choosing a fixture wording (3 draws each, same file, same
|
||||||
|
# system prompt): a POSITIONAL condition ("its second line is …") scored
|
||||||
|
# MET/UNMET/MET and a possessive one ("records the kernel version …") scored
|
||||||
|
# MET/MET/UNMET, while a CONTENT-EXISTENCE one ("contains both a test-result
|
||||||
|
# summary and a recorded kernel version") scored MET three times out of three.
|
||||||
|
# Conditions that name a line position or say "and nothing else" invite this
|
||||||
|
# judge to invent requirements about the other lines. Prefer "contains X and Y".
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
#
|
#
|
||||||
# scripts/judge-eval.sh # the configured validator
|
# scripts/judge-eval.sh # the configured validator
|
||||||
# JUDGE=kimi scripts/judge-eval.sh # compare another
|
# JUDGE=kimi scripts/judge-eval.sh # compare another
|
||||||
|
|||||||
@@ -268,7 +268,7 @@ MICROVM_BODY=$(cat <<JSON
|
|||||||
"description":"Prove a coding phase executes in a Firecracker microVM and can delegate.",
|
"description":"Prove a coding phase executes in a Firecracker microVM and can delegate.",
|
||||||
"phases":[
|
"phases":[
|
||||||
{"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1,
|
{"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1,
|
||||||
"done_when":"A file named MICROVM.md exists at the repository root and its second line is a bare kernel version of the form MAJOR.MINOR.PATCH (for example 6.1.128) and nothing else. Judge only what the file contains — do NOT run uname yourself and do NOT compare the value against the machine you are running on; the file records a DIFFERENT machine.",
|
"done_when":"MICROVM.md exists at the repository root and contains both a test-result summary and a recorded kernel version such as 6.1.128.",
|
||||||
"task":"1. Use the verifier subagent to run the COMPLETE test suite of this crate and report what it found. Do not run it yourself and report that instead — the point is an independent check.\n2. Write MICROVM.md at the repository root with exactly two lines: the first is the test-result summary line the verifier reported, the second is the kernel release from running uname -r.\nCreate no other files."}}
|
"task":"1. Use the verifier subagent to run the COMPLETE test suite of this crate and report what it found. Do not run it yourself and report that instead — the point is an independent check.\n2. Write MICROVM.md at the repository root with exactly two lines: the first is the test-result summary line the verifier reported, the second is the kernel release from running uname -r.\nCreate no other files."}}
|
||||||
]}
|
]}
|
||||||
JSON
|
JSON
|
||||||
@@ -389,7 +389,7 @@ COMPOSED_BODY=$(cat <<JSON
|
|||||||
"description":"Prove a ZeroClaw graph of microVM nodes carries file work between its nodes.",
|
"description":"Prove a ZeroClaw graph of microVM nodes carries file work between its nodes.",
|
||||||
"phases":[
|
"phases":[
|
||||||
{"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1,
|
{"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1,
|
||||||
"done_when":"STAGES.md exists at the repository root and its last line ends with a bare kernel version of the form MAJOR.MINOR.PATCH (for example 6.1.128). Judge only what the file contains — do NOT run uname yourself; the file records a DIFFERENT machine than the one you are on.",
|
"done_when":"STAGES.md exists at the repository root and contains one line per stage, each recording a stage name and a kernel version such as 6.1.128.",
|
||||||
"task":"Append EXACTLY ONE line to STAGES.md at the repository root, creating the file if it does not exist. The line is: your stage name, a space, and the output of \`uname -r\`.\n\nDo NOT rewrite, reorder or remove any line already in the file — earlier stages wrote those, and they are the record of this run. If the file already has lines, yours goes after them. Change no other file."}}
|
"task":"Append EXACTLY ONE line to STAGES.md at the repository root, creating the file if it does not exist. The line is: your stage name, a space, and the output of \`uname -r\`.\n\nDo NOT rewrite, reorder or remove any line already in the file — earlier stages wrote those, and they are the record of this run. If the file already has lines, yours goes after them. Change no other file."}}
|
||||||
]}
|
]}
|
||||||
JSON
|
JSON
|
||||||
|
|||||||
Reference in New Issue
Block a user