osobh/clawmates
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(evaluator): the verification sandbox leaked for the same reason the bench copy did

Browse Source 
Found by checking `_verify` after fixing the identical bug in `_bench`: 16 MB
stranded across two copies, the oldest hours old.

`Sandbox::Drop` calls `std::fs::remove_dir_all` as uid 65532. The judge runs
`cargo test` in a container as ROOT — that is the entire point of the sandbox —
so the copy's `target/` is root-owned and the removal fails on it, leaving the
whole tree. The error was logged to a stream nobody reads, so the sandbox that
exists to protect the checkout quietly filled the disk instead.

Its doc comment also claimed "the copy lives under `_verify/<mission>`, which
the next pass clears anyway". That was wrong for exactly the same reason:
`for_checkout` removes a stale root before copying, with the same uid, and fails
the same way. A leaked copy was permanent, not transient.

`Sandbox::purge` removes it from inside the container, as root, where it was
written. `evaluate` now wraps its body so the purge runs on EVERY exit — that
function returns from several branches, and cleanup only some paths reach is the
same as no cleanup on the others. `Drop` stays as a fallback for the early paths
where nothing has run as root yet, and its comment no longer claims otherwise.

This is the third instance today of the same shape: cleanup that cannot clean up,
invisible because the failure was swallowed. The others were the leaked agent
containers in the runtime tests and the bench copy in e89a32f.

243 lib tests.
This commit is contained in:
Omar Sobh
2026-08-07 17:34:54 -07:00
parent e89a32ffef
commit a4b4d05b8d
2 changed files with 143 additions and 86 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/cm-api/src/evaluator.rs
+93 -81
View File
@@ -424,103 +424,115 @@ pub async fn evaluate(
"COMPLETION CONDITION:\n{condition}\n\nEVIDENCE (agent claims — verify them):\n{evidence}" "COMPLETION CONDITION:\n{condition}\n\nEVIDENCE (agent claims — verify them):\n{evidence}"
); );
let sandbox = crate::evaluator_tools::Sandbox::for_mission(mission_id); let sandbox = crate::evaluator_tools::Sandbox::for_mission(mission_id);
// Purged explicitly at every exit below: `Drop` runs as uid 65532 and cannot
// delete the root-owned `target/` the judge's own `cargo test` leaves behind.
// Wrapped so the purge below runs on EVERY exit: this function returns
// from several branches, and a cleanup only some paths reach is the same
// as no cleanup on the others.
let verdict = async {
// Most preferred: a judge from a DIFFERENT provider family, with the same // Most preferred: a judge from a DIFFERENT provider family, with the same
// allow-listed tool loop. Claude judging Claude's work is a correlated // allow-listed tool loop. Claude judging Claude's work is a correlated
// failure — the model that talked itself into a shortcut is the one disposed // failure — the model that talked itself into a shortcut is the one disposed
// to accept it — and the tool loop is what makes the check evidence rather // to accept it — and the tool loop is what makes the check evidence rather
// than opinion, so an independent judge must have it too. // than opinion, so an independent judge must have it too.
if let Some((provider, model)) = cross_provider_judge(runtime, mission_id).await { if let Some((provider, model)) = cross_provider_judge(runtime, mission_id).await {
let system = match &sandbox { let system = match &sandbox {
Some(_) => format!("{EVAL_SYSTEM_VERIFYING}\n\n{VERDICT_CONTRACT}"), Some(_) => format!("{EVAL_SYSTEM_VERIFYING}\n\n{VERDICT_CONTRACT}"),
None => format!("{EVAL_SYSTEM_EVIDENCE_ONLY}\n\n{VERDICT_CONTRACT}"), None => format!("{EVAL_SYSTEM_EVIDENCE_ONLY}\n\n{VERDICT_CONTRACT}"),
}; };
eprintln!( eprintln!(
"evaluator: mission {mission_id} judged independently by {} ({})", "evaluator: mission {mission_id} judged independently by {} ({})",
model, model,
provider_family(&model) provider_family(&model)
); );
match judge_with_tools(provider.as_ref(), &system, &user, &model, sandbox.as_ref()).await { match judge_with_tools(provider.as_ref(), &system, &user, &model, sandbox.as_ref()).await {
Ok((text, checks)) => { Ok((text, checks)) => {
let mut v = parse_verdict(&model, &text); let mut v = parse_verdict(&model, &text);
v.guidance = sanitize_guidance(condition, evidence, &v.guidance); v.guidance = sanitize_guidance(condition, evidence, &v.guidance);
v.checks = checks; v.checks = checks;
v.independent = true; v.independent = true;
return v; return v;
} }
// Deliberately NOT a silent fall-through to the house judge. An // Deliberately NOT a silent fall-through to the house judge. An
// independent check that failed and was quietly replaced by a // independent check that failed and was quietly replaced by a
// same-family one would leave a verdict claiming a property it does // same-family one would leave a verdict claiming a property it does
// not have. The phase stays unmet this pass and says why; the next // not have. The phase stays unmet this pass and says why; the next
// sweep retries. // sweep retries.
Err(e) => { Err(e) => {
eprintln!( eprintln!(
"evaluator: the independent judge ({model}) failed — NOT falling back to the agent's own provider: {e}" "evaluator: the independent judge ({model}) failed — NOT falling back to the agent's own provider: {e}"
); );
return Verdict::not_met( return Verdict::not_met(
&model, &model,
"the independent validator could not be reached this pass", "the independent validator could not be reached this pass",
Some(e), Some(e),
); );
}
} }
} }
}
// Preferred: a bare Messages API call on the subscription token. See // Preferred: a bare Messages API call on the subscription token. See
// `subscription_judge` for why this beats routing through an agent. // `subscription_judge` for why this beats routing through an agent.
if let Some(provider) = subscription_judge() { if let Some(provider) = subscription_judge() {
let model = subscription_model(); let model = subscription_model();
let system = match &sandbox { let system = match &sandbox {
Some(_) => format!("{EVAL_SYSTEM_VERIFYING}\n\n{VERDICT_CONTRACT}"), Some(_) => format!("{EVAL_SYSTEM_VERIFYING}\n\n{VERDICT_CONTRACT}"),
None => format!("{EVAL_SYSTEM_EVIDENCE_ONLY}\n\n{VERDICT_CONTRACT}"), None => format!("{EVAL_SYSTEM_EVIDENCE_ONLY}\n\n{VERDICT_CONTRACT}"),
};
let outcome = judge_with_tools(&provider, &system, &user, &model, sandbox.as_ref()).await;
// Same family as the agent; `independent` stays false below.
return match outcome {
Err(e) => Verdict::not_met(
&model,
"could not evaluate the completion condition this pass",
Some(e),
),
Ok((text, checks)) => {
let mut v = parse_verdict(&model, &text);
v.guidance = sanitize_guidance(condition, evidence, &v.guidance);
v.checks = checks;
v
}
};
}
// Fallback paths have no tool loop, so they judge claims only and must say so.
let system = format!("{EVAL_SYSTEM_EVIDENCE_ONLY}\n\n{VERDICT_CONTRACT}");
let (eval_system, user) = (system.as_str(), user);
let model = evaluator_model();
// Same routing as the door governor (mcp_door.rs): `runtime:<alias>` goes
// through the container agent so a subscription-only model can judge.
let raw: Result<String, String> = if let Some(alias) = model.strip_prefix("runtime:") {
match crate::topology_exec::ZeroClawDriveExecutor::from_env() {
Ok(exec) => exec.judge_raw(alias.trim(), eval_system, &user).await,
Err(e) => Err(format!("runtime executor unavailable: {e}")),
}
} else {
runtime
.complete(eval_system, &user, &model, 512, false)
.await
}; };
let outcome = judge_with_tools(&provider, &system, &user, &model, sandbox.as_ref()).await;
// Same family as the agent; `independent` stays false below. match raw {
return match outcome {
Err(e) => Verdict::not_met( Err(e) => Verdict::not_met(
&model, &model,
"could not evaluate the completion condition this pass", "could not evaluate the completion condition this pass",
Some(e), Some(e),
), ),
Ok((text, checks)) => { Ok(text) => {
let mut v = parse_verdict(&model, &text); let mut v = parse_verdict(&model, &text);
v.guidance = sanitize_guidance(condition, evidence, &v.guidance); v.guidance = sanitize_guidance(condition, evidence, &v.guidance);
v.checks = checks;
v v
} }
};
}
// Fallback paths have no tool loop, so they judge claims only and must say so.
let system = format!("{EVAL_SYSTEM_EVIDENCE_ONLY}\n\n{VERDICT_CONTRACT}");
let (eval_system, user) = (system.as_str(), user);
let model = evaluator_model();
// Same routing as the door governor (mcp_door.rs): `runtime:<alias>` goes
// through the container agent so a subscription-only model can judge.
let raw: Result<String, String> = if let Some(alias) = model.strip_prefix("runtime:") {
match crate::topology_exec::ZeroClawDriveExecutor::from_env() {
Ok(exec) => exec.judge_raw(alias.trim(), eval_system, &user).await,
Err(e) => Err(format!("runtime executor unavailable: {e}")),
}
} else {
runtime
.complete(eval_system, &user, &model, 512, false)
.await
};
match raw {
Err(e) => Verdict::not_met(
&model,
"could not evaluate the completion condition this pass",
Some(e),
),
Ok(text) => {
let mut v = parse_verdict(&model, &text);
v.guidance = sanitize_guidance(condition, evidence, &v.guidance);
v
} }
} }
.await;
if let Some(sb) = &sandbox {
sb.purge().await;
}
verdict
} }
/// Ceiling on verification commands per verdict. A judge that has run twelve /// Ceiling on verification commands per verdict. A judge that has run twelve
crates/cm-api/src/evaluator_tools.rs
+50 -5
View File
@@ -392,12 +392,57 @@ fn git_ownership_env(workdir: &str) -> Vec<String> {
] ]
} }
impl Drop for Sandbox { impl Sandbox {
/// Remove the copy when the evaluation is done. /// Remove the copy, from inside the container that wrote it.
/// ///
/// Best-effort: a leaked copy costs disk, and failing an evaluation over /// `Drop` cannot do this. The judge runs `cargo test` in a container as
/// cleanup would trade a real verdict for a housekeeping error. The copy /// ROOT, so the copy's `target/` is root-owned, and the server process is
/// lives under `_verify/<mission>`, which the next pass clears anyway. /// uid 65532 — its `remove_dir_all` fails on those files and leaves the
/// whole tree behind. Measured: 16 MB across two stranded copies, the oldest
/// hours old, while `Drop` logged nothing anyone read.
///
/// The claim that "the next pass clears anyway" was wrong for the same
/// reason: `for_checkout` removes a stale root before copying, with the same
/// uid, and fails the same way.
///
/// Still best-effort — a housekeeping error must not cost a real verdict —
/// but now attempted by something that can actually succeed.
pub async fn purge(&self) {
if !self.owned {
return;
}
let Some(root) = self.workdir.parent() else {
return;
};
let Ok(docker) = crate::container_exec::connect() else {
return;
};
let argv = vec![
"rm".to_string(),
"-rf".to_string(),
root.display().to_string(),
];
if let Err(e) = crate::container_exec::exec(
&docker,
&self.container,
Some("/"),
&argv,
COMMAND_TIMEOUT,
)
.await
{
eprintln!(
"evaluator_tools: could not remove the verification copy at {} ({e})",
root.display()
);
}
}
}
impl Drop for Sandbox {
/// Fallback only — see [`Sandbox::purge`], which is what actually clears a
/// copy the judge has run commands in. This still catches the early paths
/// where nothing has run as root yet.
fn drop(&mut self) { fn drop(&mut self) {
if !self.owned { if !self.owned {
return; return;