test(harness): say what the condition means, after the judge read it strictly
The restored GLM judge failed a phase that had done the work: MICROVM.md existed with two lines and the second was `6.1.128`, and the verdict was "a kernel version number, not a Linux kernel release string such as 'Linux 6.1.128'". The judge is wrong on the fact — `6.1.128` is exactly what `uname -r` prints, and "release" is the term for it — but the CONDITION was ambiguous, and it is our fixture. "A Linux kernel release string" can be read as either `uname -r` output or `Linux x.y.z`, and a stricter reader is entitled to the second. Both scenarios now say what they mean: the exact output of `uname -r`, a bare version, no prefix. This is not weakening the assertion. The scenario's own kernel check — the one that proves the agent ran in a guest rather than on a host — is a separate, unchanged comparison against the real host kernels, and it PASSED on the same run. What changed is only that the mission-level `done_when` now describes an observable fact precisely, which is what this codebase's own plan-authoring prompt tells models to do. Worth recording rather than papering over: an over-strict independent judge is a much safer failure mode than an over-lenient one, and this is evidence the judge READS the tree instead of rubber-stamping it — the Goodhart incident that motivated cross-provider validation was the opposite failure. But it does mean a vague `done_when` can now cost a phase, which raises the value of `done_when_check` (a shell command, judged by exit status) for anything mechanical. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
521da9feb9
commit
1b556c5849
@@ -268,7 +268,7 @@ MICROVM_BODY=$(cat <<JSON
|
|||||||
"description":"Prove a coding phase executes in a Firecracker microVM and can delegate.",
|
"description":"Prove a coding phase executes in a Firecracker microVM and can delegate.",
|
||||||
"phases":[
|
"phases":[
|
||||||
{"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1,
|
{"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1,
|
||||||
"done_when":"A file named MICROVM.md exists at the repository root and contains at least two lines, the second of which is a Linux kernel release string.",
|
"done_when":"A file named MICROVM.md exists at the repository root and contains at least two lines, the second of which is the exact output of `uname -r` (a bare version like 6.1.128, with no `Linux` prefix and no other text).",
|
||||||
"task":"1. Use the verifier subagent to run the COMPLETE test suite of this crate and report what it found. Do not run it yourself and report that instead — the point is an independent check.\n2. Write MICROVM.md at the repository root with exactly two lines: the first is the test-result summary line the verifier reported, the second is the kernel release from running uname -r.\nCreate no other files."}}
|
"task":"1. Use the verifier subagent to run the COMPLETE test suite of this crate and report what it found. Do not run it yourself and report that instead — the point is an independent check.\n2. Write MICROVM.md at the repository root with exactly two lines: the first is the test-result summary line the verifier reported, the second is the kernel release from running uname -r.\nCreate no other files."}}
|
||||||
]}
|
]}
|
||||||
JSON
|
JSON
|
||||||
@@ -389,7 +389,7 @@ COMPOSED_BODY=$(cat <<JSON
|
|||||||
"description":"Prove a ZeroClaw graph of microVM nodes carries file work between its nodes.",
|
"description":"Prove a ZeroClaw graph of microVM nodes carries file work between its nodes.",
|
||||||
"phases":[
|
"phases":[
|
||||||
{"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1,
|
{"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1,
|
||||||
"done_when":"STAGES.md exists at the repository root and its last line is a Linux kernel release string.",
|
"done_when":"STAGES.md exists at the repository root and its last line ends with the exact output of `uname -r` (a bare version like 6.1.128, with no `Linux` prefix).",
|
||||||
"task":"Append EXACTLY ONE line to STAGES.md at the repository root, creating the file if it does not exist. The line is: your stage name, a space, and the output of \`uname -r\`.\n\nDo NOT rewrite, reorder or remove any line already in the file — earlier stages wrote those, and they are the record of this run. If the file already has lines, yours goes after them. Change no other file."}}
|
"task":"Append EXACTLY ONE line to STAGES.md at the repository root, creating the file if it does not exist. The line is: your stage name, a space, and the output of \`uname -r\`.\n\nDo NOT rewrite, reorder or remove any line already in the file — earlier stages wrote those, and they are the record of this run. If the file already has lines, yours goes after them. Change no other file."}}
|
||||||
]}
|
]}
|
||||||
JSON
|
JSON
|
||||||
|
|||||||
Reference in New Issue
Block a user