test(harness): say what the condition means, after the judge read it strictly

The restored GLM judge failed a phase that had done the work: MICROVM.md existed
with two lines and the second was `6.1.128`, and the verdict was "a kernel
version number, not a Linux kernel release string such as 'Linux 6.1.128'".

The judge is wrong on the fact — `6.1.128` is exactly what `uname -r` prints,
and "release" is the term for it — but the CONDITION was ambiguous, and it is our
fixture. "A Linux kernel release string" can be read as either `uname -r` output
or `Linux x.y.z`, and a stricter reader is entitled to the second. Both scenarios
now say what they mean: the exact output of `uname -r`, a bare version, no prefix.

This is not weakening the assertion. The scenario's own kernel check — the one
that proves the agent ran in a guest rather than on a host — is a separate,
unchanged comparison against the real host kernels, and it PASSED on the same
run. What changed is only that the mission-level `done_when` now describes an
observable fact precisely, which is what this codebase's own plan-authoring
prompt tells models to do.

Worth recording rather than papering over: an over-strict independent judge is a
much safer failure mode than an over-lenient one, and this is evidence the judge
READS the tree instead of rubber-stamping it — the Goodhart incident that
motivated cross-provider validation was the opposite failure. But it does mean a
vague `done_when` can now cost a phase, which raises the value of
`done_when_check` (a shell command, judged by exit status) for anything
mechanical.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-07 06:19:00 -07:00
co-authored by Claude Opus 5
parent 521da9feb9
commit 1b556c5849
+2 -2
View File
@@ -268,7 +268,7 @@ MICROVM_BODY=$(cat <<JSON
"description":"Prove a coding phase executes in a Firecracker microVM and can delegate.", "description":"Prove a coding phase executes in a Firecracker microVM and can delegate.",
"phases":[ "phases":[
{"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1, {"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1,
"done_when":"A file named MICROVM.md exists at the repository root and contains at least two lines, the second of which is a Linux kernel release string.", "done_when":"A file named MICROVM.md exists at the repository root and contains at least two lines, the second of which is the exact output of `uname -r` (a bare version like 6.1.128, with no `Linux` prefix and no other text).",
"task":"1. Use the verifier subagent to run the COMPLETE test suite of this crate and report what it found. Do not run it yourself and report that instead — the point is an independent check.\n2. Write MICROVM.md at the repository root with exactly two lines: the first is the test-result summary line the verifier reported, the second is the kernel release from running uname -r.\nCreate no other files."}} "task":"1. Use the verifier subagent to run the COMPLETE test suite of this crate and report what it found. Do not run it yourself and report that instead — the point is an independent check.\n2. Write MICROVM.md at the repository root with exactly two lines: the first is the test-result summary line the verifier reported, the second is the kernel release from running uname -r.\nCreate no other files."}}
]} ]}
JSON JSON
@@ -389,7 +389,7 @@ COMPOSED_BODY=$(cat <<JSON
"description":"Prove a ZeroClaw graph of microVM nodes carries file work between its nodes.", "description":"Prove a ZeroClaw graph of microVM nodes carries file work between its nodes.",
"phases":[ "phases":[
{"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1, {"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1,
"done_when":"STAGES.md exists at the repository root and its last line is a Linux kernel release string.", "done_when":"STAGES.md exists at the repository root and its last line ends with the exact output of `uname -r` (a bare version like 6.1.128, with no `Linux` prefix).",
"task":"Append EXACTLY ONE line to STAGES.md at the repository root, creating the file if it does not exist. The line is: your stage name, a space, and the output of \`uname -r\`.\n\nDo NOT rewrite, reorder or remove any line already in the file — earlier stages wrote those, and they are the record of this run. If the file already has lines, yours goes after them. Change no other file."}} "task":"Append EXACTLY ONE line to STAGES.md at the repository root, creating the file if it does not exist. The line is: your stage name, a space, and the output of \`uname -r\`.\n\nDo NOT rewrite, reorder or remove any line already in the file — earlier stages wrote those, and they are the record of this run. If the file already has lines, yours goes after them. Change no other file."}}
]} ]}
JSON JSON