test(harness): a done_when naming a COMMAND invites the judge to run it

My previous attempt at this made it worse, which is the useful part.

The condition said "a Linux kernel release string" and the judge rejected
`6.1.128` as "not a Linux kernel release string such as 'Linux 6.1.128'". I
rewrote it as "the exact output of `uname -r`" — and the next verdict was that
line 2 should be `27.0.0`. The judge has a sandbox and allow-listed commands, so
naming a command told it to RUN that command, in ITS OWN container, and compare
the file against the answer it got there. The file records a microVM's kernel;
the judge was comparing it against the machine the judge runs on. Those are
different machines by design — that is the entire point of the assertion.

So a `done_when` for a tool-using judge must describe the VALUE's shape, never a
command that produces it: "a bare kernel version of the form MAJOR.MINOR.PATCH
(for example 6.1.128) and nothing else", plus an explicit instruction not to run
uname and not to compare against the local machine, because the file records a
different one.

The general rule, worth carrying into how `done_when` is written anywhere: a
condition phrased as "the output of X" is ambiguous about WHERE X runs, and a
judge with tools resolves that ambiguity by running X where it stands. Conditions
about a remote or past environment must be stated as properties of the recorded
value.

The scenario's real proof that the agent ran in a guest is unchanged: a separate
comparison of that line against the actual gateway and node kernels, which has
passed on every run including the two where the judge disagreed.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-07 06:35:15 -07:00
co-authored by Claude Opus 5
parent e2f576ec02
commit 72f8bdc87c
+2 -2
View File
@@ -268,7 +268,7 @@ MICROVM_BODY=$(cat <<JSON
"description":"Prove a coding phase executes in a Firecracker microVM and can delegate.", "description":"Prove a coding phase executes in a Firecracker microVM and can delegate.",
"phases":[ "phases":[
{"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1, {"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1,
"done_when":"A file named MICROVM.md exists at the repository root and contains at least two lines, the second of which is the exact output of `uname -r` (a bare version like 6.1.128, with no `Linux` prefix and no other text).", "done_when":"A file named MICROVM.md exists at the repository root and its second line is a bare kernel version of the form MAJOR.MINOR.PATCH (for example 6.1.128) and nothing else. Judge only what the file contains — do NOT run uname yourself and do NOT compare the value against the machine you are running on; the file records a DIFFERENT machine.",
"task":"1. Use the verifier subagent to run the COMPLETE test suite of this crate and report what it found. Do not run it yourself and report that instead — the point is an independent check.\n2. Write MICROVM.md at the repository root with exactly two lines: the first is the test-result summary line the verifier reported, the second is the kernel release from running uname -r.\nCreate no other files."}} "task":"1. Use the verifier subagent to run the COMPLETE test suite of this crate and report what it found. Do not run it yourself and report that instead — the point is an independent check.\n2. Write MICROVM.md at the repository root with exactly two lines: the first is the test-result summary line the verifier reported, the second is the kernel release from running uname -r.\nCreate no other files."}}
]} ]}
JSON JSON
@@ -389,7 +389,7 @@ COMPOSED_BODY=$(cat <<JSON
"description":"Prove a ZeroClaw graph of microVM nodes carries file work between its nodes.", "description":"Prove a ZeroClaw graph of microVM nodes carries file work between its nodes.",
"phases":[ "phases":[
{"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1, {"kind":"coding","order_idx":0,"config":{"commit_policy":"always","max_iterations":1,
"done_when":"STAGES.md exists at the repository root and its last line ends with the exact output of `uname -r` (a bare version like 6.1.128, with no `Linux` prefix).", "done_when":"STAGES.md exists at the repository root and its last line ends with a bare kernel version of the form MAJOR.MINOR.PATCH (for example 6.1.128). Judge only what the file contains — do NOT run uname yourself; the file records a DIFFERENT machine than the one you are on.",
"task":"Append EXACTLY ONE line to STAGES.md at the repository root, creating the file if it does not exist. The line is: your stage name, a space, and the output of \`uname -r\`.\n\nDo NOT rewrite, reorder or remove any line already in the file — earlier stages wrote those, and they are the record of this run. If the file already has lines, yours goes after them. Change no other file."}} "task":"Append EXACTLY ONE line to STAGES.md at the repository root, creating the file if it does not exist. The line is: your stage name, a space, and the output of \`uname -r\`.\n\nDo NOT rewrite, reorder or remove any line already in the file — earlier stages wrote those, and they are the record of this run. If the file already has lines, yours goes after them. Change no other file."}}
]} ]}
JSON JSON