fix(evaluator): the verification sandbox never ran a command
`evaluator_tools::Sandbox::run` shelled out to `tokio::process::Command::new
("docker")`. The server image installs `git ca-certificates chromium
fonts-liberation` and nothing else, so in production every verification
command failed to spawn.
The failure was invisible in the worst way. `Sandbox::run` deliberately turns
execution failures into evidence text rather than errors, so a judge reasons
about "that command did not run" instead of the pass collapsing. With no
`docker` binary every command returned COULD NOT RUN, the judge correctly
concluded it could not verify, and fail-closed returned "not met". The
verdicts were right. The verification never happened — and the adversarial
validation that appeared to prove the feature working proved fail-closed
working instead.
The second defect made it worse: `checks` recorded the *attempt*, pushed
before the command ran, so a verdict reached with a dead sandbox reported
"verified by 10 checks" — a stronger claim than "no checks at all", made on
weaker evidence.
- New `container_exec` routes execution through the Docker API via bollard,
which was already a dependency and already reaches the daemon through the
socket proxy. Captures the exit code (absent from the old helper) and keeps
stdout and stderr apart (`LogOutput`'s Display merged them, which is why
nothing downstream could tell JSON from a progress bar). `security_scan`
parses stdout alone; `benchmark_runner` needs both.
- `ExecOutput::success()` requires `Some(0)`. An unreadable status is not
success — `commit_policy = "on_green_tests"` will gate on this, and
"unknown" reading as "green" would push untested work.
- `Sandbox::run` returns a `CheckOutcome` carrying `ran`/`refused`/
`exit_code`. `Verdict::verified_checks()` counts executions, not attempts.
- The UI gains a third state: "could not verify (N attempted, 0 ran)" —
precisely the case that used to render as verified.
- Regression tests reproduce the production shape: two checks recorded,
neither executed, `was_verified() == false`; plus a failing suite (exit 101)
still counting as verification, because that is something the judge learned
rather than was told.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
3eb89620e7
commit
c812b714f4
@@ -5,10 +5,55 @@ import { Target } from "lucide-react";
|
||||
|
||||
import {
|
||||
getPhaseEvaluations,
|
||||
type CheckOutcome,
|
||||
type MissionPhase,
|
||||
type PhaseEvaluation,
|
||||
} from "@/lib/api/missions";
|
||||
|
||||
/**
|
||||
* Whether a verdict was verified, and how honestly we can say so.
|
||||
*
|
||||
* Three states, not two. A judge that attempted ten commands and executed none
|
||||
* — because the sandbox was unreachable — is not the same as a judge that ran
|
||||
* ten, and neither is the same as a phase with no repo to check. Collapsing
|
||||
* the middle case into "verified" is what the first version of this badge did.
|
||||
*/
|
||||
function VerificationNote({ checks }: { checks?: CheckOutcome[] }) {
|
||||
const all = checks ?? [];
|
||||
const ran = all.filter((c) => c.ran);
|
||||
const failed = ran.filter((c) => c.exit_code !== 0);
|
||||
|
||||
if (ran.length > 0) {
|
||||
const detail = ran.map((c) => `${c.argv.join(" ")} → exit ${c.exit_code}`).join("\n");
|
||||
return (
|
||||
<em style={{ color: failed.length ? "#e8b465" : "#6a8ab0" }} title={detail}>
|
||||
{` — verified by ${ran.length} check${ran.length === 1 ? "" : "s"}`}
|
||||
{failed.length ? ` (${failed.length} non-zero)` : ""}
|
||||
</em>
|
||||
);
|
||||
}
|
||||
if (all.length > 0) {
|
||||
// Attempted but nothing executed: a broken sandbox, or every command
|
||||
// refused. Say so — this is the case that used to read as "verified".
|
||||
return (
|
||||
<em
|
||||
style={{ color: "#ff8a7a" }}
|
||||
title={all.map((c) => `${c.argv.join(" ")} → ${c.refused ? "refused" : "could not run"}`).join("\n")}
|
||||
>
|
||||
{` — could not verify (${all.length} attempted, 0 ran)`}
|
||||
</em>
|
||||
);
|
||||
}
|
||||
return (
|
||||
<em
|
||||
style={{ color: "#8a7a5a" }}
|
||||
title="No commands were run — this verdict rests on what the agents reported."
|
||||
>
|
||||
{" — from agent claims only"}
|
||||
</em>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* The completion condition on a phase, plus how the last pass was judged.
|
||||
*
|
||||
@@ -21,9 +66,9 @@ import {
|
||||
* were told: they get a sanitized `guidance` that withholds the acceptance
|
||||
* text, so a pass cannot be satisfied by pasting the verdict back.
|
||||
*
|
||||
* Whether the judge verified anything is shown alongside the verdict. A
|
||||
* verdict with no checks rests on the agents' own claims, and an operator
|
||||
* should not have to guess which kind they are looking at.
|
||||
* Whether the judge verified anything is shown alongside the verdict — see
|
||||
* `VerificationNote`. An operator should never have to guess whether a verdict
|
||||
* rests on executed commands or on the agents' own account of themselves.
|
||||
*/
|
||||
export function PhaseGoalStrip({
|
||||
missionId,
|
||||
@@ -117,20 +162,7 @@ export function PhaseGoalStrip({
|
||||
// an evaluator outage should not read as a verdict on the work.
|
||||
<em style={{ color: "#ff8a7a" }}> (evaluator error)</em>
|
||||
)}
|
||||
{!latest.error && (
|
||||
<em
|
||||
style={{ color: latest.checks?.length ? "#6a8ab0" : "#8a7a5a" }}
|
||||
title={
|
||||
latest.checks?.length
|
||||
? latest.checks.join("\n")
|
||||
: "No commands were run — this verdict rests on what the agents reported."
|
||||
}
|
||||
>
|
||||
{latest.checks?.length
|
||||
? ` — verified by ${latest.checks.length} check${latest.checks.length === 1 ? "" : "s"}`
|
||||
: " — from agent claims only"}
|
||||
</em>
|
||||
)}
|
||||
{!latest.error && <VerificationNote checks={latest.checks} />}
|
||||
</span>
|
||||
</div>
|
||||
)}
|
||||
|
||||
Reference in New Issue
Block a user