fix(evaluator): the verification sandbox never ran a command
ci / gates (push) Failing after 5s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped

`evaluator_tools::Sandbox::run` shelled out to `tokio::process::Command::new
("docker")`. The server image installs `git ca-certificates chromium
fonts-liberation` and nothing else, so in production every verification
command failed to spawn.

The failure was invisible in the worst way. `Sandbox::run` deliberately turns
execution failures into evidence text rather than errors, so a judge reasons
about "that command did not run" instead of the pass collapsing. With no
`docker` binary every command returned COULD NOT RUN, the judge correctly
concluded it could not verify, and fail-closed returned "not met". The
verdicts were right. The verification never happened — and the adversarial
validation that appeared to prove the feature working proved fail-closed
working instead.

The second defect made it worse: `checks` recorded the *attempt*, pushed
before the command ran, so a verdict reached with a dead sandbox reported
"verified by 10 checks" — a stronger claim than "no checks at all", made on
weaker evidence.

- New `container_exec` routes execution through the Docker API via bollard,
  which was already a dependency and already reaches the daemon through the
  socket proxy. Captures the exit code (absent from the old helper) and keeps
  stdout and stderr apart (`LogOutput`'s Display merged them, which is why
  nothing downstream could tell JSON from a progress bar). `security_scan`
  parses stdout alone; `benchmark_runner` needs both.
- `ExecOutput::success()` requires `Some(0)`. An unreadable status is not
  success — `commit_policy = "on_green_tests"` will gate on this, and
  "unknown" reading as "green" would push untested work.
- `Sandbox::run` returns a `CheckOutcome` carrying `ran`/`refused`/
  `exit_code`. `Verdict::verified_checks()` counts executions, not attempts.
- The UI gains a third state: "could not verify (N attempted, 0 ran)" —
  precisely the case that used to render as verified.
- Regression tests reproduce the production shape: two checks recorded,
  neither executed, `was_verified() == false`; plus a failing suite (exit 101)
  still counting as verification, because that is something the judge learned
  rather than was told.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-01 18:33:32 -07:00
co-authored by Claude Opus 5
parent 3eb89620e7
commit c812b714f4
11 changed files with 588 additions and 128 deletions
+5 -1
View File
@@ -226,7 +226,11 @@ async fn effective_risk_profile(
.bind(agent.workspace_id.as_uuid())
.fetch_optional(pool)
.await?;
let from_team = row.and_then(|r| r.try_get::<Option<String>, _>("risk_profile").ok().flatten());
let from_team = row.and_then(|r| {
r.try_get::<Option<String>, _>("risk_profile")
.ok()
.flatten()
});
Ok(from_team.unwrap_or_else(|| {
crate::runtime_provision::RuntimeProvisioner::default_risk_profile_for_role(
&agent.job_title,