Omar SobhandClaude Opus 5
5c066afa7b
TWO FINDINGS, one from cleaning up and one from refusing to keep guessing. THE LEAK. `./scripts/test.sh` left three containers running every time — 289 had accumulated. The cause was a comment that lied: `warm_pool.rs` said "Shutdown destroys assigned AND pooled sandboxes", while `SandboxManager::shutdown` drains the POOL only. Its own doc says why — assigned sandboxes persist deliberately so a redeploy can reuse them, and production reaps the strays with `reconcile_orphans` at boot. A test has no next boot, so each one that assigned a sandbox simply left it running. The three tests now call the `release_agent` that already existed, and the comment says what the code does. Verified: 0 leaked, where the same run leaked 3 before. THE EVAL. The independent judge failed the same correct phase FOUR times, each time citing a different invented requirement. I blamed the condition's wording twice and rewrote it twice — the second rewrite made it worse, by naming a command a tool-using judge then ran in its own container. Then a control showed the same model answering MET to the same question asked directly, and a third wording test showed a STRICTER phrasing scoring UNMET. Prose wording was not the variable. Continuing to iterate would have been fitting the fixture to noise. `scripts/judge-eval.sh` measures the thing instead: five cases drawn from real incidents, each with an answer a careful human would agree with. This project has 557 tests and had zero evals, which is backwards — a test pins OUR code, an eval pins the MODEL, and the model changes without us touching anything. The result is why it was worth building: glm-4.7 4/5 — wrong on kernel-ok: says UNMET when MET kimi-for-coding 4/5 — wrong on goodhart: says MET when UNMET Identical scores, opposite failure modes. GLM fails good work; KIMI passes work where 14 assertions were deleted and the failing module removed to make a suite "pass" — the exact incident the verifying judge was built after. Swapping the validator to Kimi because it passes our failing case would have installed a rubber stamp. Keep GLM: a judge that is too strict costs a re-run, a judge that is too lenient costs the guarantee. The eval also caught a bug in itself before I trusted it: Kimi answers with a `thinking` block first, and a 160-token budget was consumed entirely by it, which the harness scored as NO-ANSWER. An eval that misreads a model is worse than no eval, so it now reads thinking blocks as a fallback and has room to answer. 557 tests pass, clippy clean. Co-Authored-By: Claude Opus 5 <[email protected]>
196 lines
6.4 KiB
Rust
196 lines
6.4 KiB
Rust
//! The browser tool, end to end with REAL Chromium in the egress-enabled
|
|
//! browser container: navigate to a real local page, return its text
|
|
//! tainted `web`, store a viewport screenshot — and prove the §15 chain:
|
|
//! a gated action AFTER browsing carries the web taint into its approval.
|
|
|
|
use std::process::Command;
|
|
use std::sync::Arc;
|
|
|
|
use cm_domain::{
|
|
AccessPolicy, Agent, AgentId, AgentStatus, Role, User, UserId, Workspace, WorkspaceId,
|
|
};
|
|
use cm_llm::ScriptedProvider;
|
|
use cm_runtime::{RunEventBody, Runtime, RuntimeConfig, SandboxManager};
|
|
use cm_sandbox::DockerDriver;
|
|
|
|
const BROWSER_IMAGE: &str = "clawmates/agent-browser:dev";
|
|
|
|
fn scenario(url: &str) -> String {
|
|
format!(
|
|
r#"
|
|
[[scenario]]
|
|
marker = "[[scenario:research]]"
|
|
|
|
[[scenario.turns]]
|
|
events = [
|
|
{{ type = "tool_use", name = "browser.goto", input = {{ url = "{url}" }} }},
|
|
]
|
|
|
|
[[scenario.turns]]
|
|
events = [
|
|
{{ type = "tool_use", name = "email.send", input = {{ to = "[email protected]", subject = "Findings", body = "Summary of the page." }} }},
|
|
]
|
|
|
|
[[scenario.turns]]
|
|
events = [
|
|
{{ type = "text", text = "Done." }},
|
|
]
|
|
"#
|
|
)
|
|
}
|
|
|
|
fn ensure_browser_image() {
|
|
let exists = Command::new("docker")
|
|
.args(["image", "inspect", BROWSER_IMAGE])
|
|
.output()
|
|
.expect("docker available")
|
|
.status
|
|
.success();
|
|
if !exists {
|
|
let root = env!("CARGO_MANIFEST_DIR");
|
|
let status = Command::new("docker")
|
|
.args([
|
|
"build",
|
|
"-t",
|
|
BROWSER_IMAGE,
|
|
&format!("{root}/../../images/agent-browser"),
|
|
])
|
|
.status()
|
|
.expect("docker build runs");
|
|
assert!(status.success(), "agent-browser image build failed");
|
|
}
|
|
}
|
|
|
|
/// A real page served on all interfaces so the container can reach it.
|
|
async fn spawn_page_server() -> String {
|
|
let app = axum::Router::new().route(
|
|
"/page",
|
|
axum::routing::get(|| async {
|
|
axum::response::Html(
|
|
"<html><head><title>Q2 Numbers</title></head>\
|
|
<body><h1>Quarterly report</h1><p>Revenue up 14 percent.</p></body></html>",
|
|
)
|
|
}),
|
|
);
|
|
let listener = tokio::net::TcpListener::bind("0.0.0.0:0").await.unwrap();
|
|
let port = listener.local_addr().unwrap().port();
|
|
tokio::spawn(async move {
|
|
axum::serve(listener, app).await.unwrap();
|
|
});
|
|
format!("http://host.docker.internal:{port}/page")
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn browsing_returns_web_tainted_content_and_taints_later_gated_actions() {
|
|
ensure_browser_image();
|
|
let pool = cm_testkit::test_pool().await;
|
|
let url = spawn_page_server().await;
|
|
|
|
let ws = Workspace {
|
|
id: WorkspaceId::new(),
|
|
name: "Acme".into(),
|
|
plan: "team".into(),
|
|
};
|
|
cm_db::repo::workspaces::insert(&pool, &ws).await.unwrap();
|
|
let owner = User {
|
|
id: UserId::new(),
|
|
workspace_id: ws.id,
|
|
email: format!("{}@acme.test", UserId::new()),
|
|
role: Role::Owner,
|
|
display_name: "Owner".into(),
|
|
created_at: time::OffsetDateTime::UNIX_EPOCH,
|
|
};
|
|
cm_db::repo::users::insert(&pool, &owner).await.unwrap();
|
|
let agent = Agent {
|
|
id: AgentId::new(),
|
|
workspace_id: ws.id,
|
|
name: "Scout".into(),
|
|
job_title: "Analyst".into(),
|
|
system_prompt: String::new(),
|
|
avatar: String::new(),
|
|
accent: String::new(),
|
|
wallpaper: String::new(),
|
|
managed_by: owner.id,
|
|
status: AgentStatus::Online,
|
|
};
|
|
cm_db::repo::agents::insert(&pool, &agent, &AccessPolicy::default())
|
|
.await
|
|
.unwrap();
|
|
|
|
let driver: Arc<dyn cm_sandbox::SandboxDriver> =
|
|
Arc::new(DockerDriver::connect().expect("docker reachable"));
|
|
let browser =
|
|
Arc::new(SandboxManager::new(driver, pool.clone(), "local", BROWSER_IMAGE).with_egress());
|
|
let blob = Arc::new(cm_files::LocalBlobStore::new(
|
|
std::env::temp_dir().join(format!("tc-brw-{}", uuid::Uuid::now_v7())),
|
|
));
|
|
let rt = Runtime::with_blob_store(
|
|
pool.clone(),
|
|
Arc::new(ScriptedProvider::from_toml(&scenario(&url)).unwrap()),
|
|
RuntimeConfig::basic("scripted", 1024).with_browser(browser.clone()),
|
|
blob.clone(),
|
|
);
|
|
|
|
let session = cm_db::repo::sessions::create(&pool, agent.id, ws.id, "Research")
|
|
.await
|
|
.unwrap();
|
|
let started = rt
|
|
.send_message(session.id, "research it [[scenario:research]]")
|
|
.await
|
|
.unwrap();
|
|
let mut rx = started.events;
|
|
let mut suspended = false;
|
|
while let Ok(envelope) = rx.recv().await {
|
|
match envelope.event {
|
|
RunEventBody::RunSuspended { .. } => {
|
|
suspended = true;
|
|
break;
|
|
}
|
|
RunEventBody::Error { .. } => break,
|
|
_ => {}
|
|
}
|
|
}
|
|
assert!(suspended, "email.send must gate after browsing");
|
|
|
|
// The browse step returned real page text, tainted web.
|
|
let (output, taint): (serde_json::Value, Vec<String>) = sqlx::query_as(
|
|
"SELECT s.output, s.taint FROM steps s
|
|
JOIN messages m ON m.id = s.message_id
|
|
WHERE m.session_id = $1 AND s.tool_name = 'browser.goto'",
|
|
)
|
|
.bind(session.id.as_uuid())
|
|
.fetch_one(&pool)
|
|
.await
|
|
.unwrap();
|
|
let text = output["content"].as_str().unwrap();
|
|
assert!(
|
|
text.contains("Revenue up 14 percent"),
|
|
"real chromium fetched the real page: {text}"
|
|
);
|
|
assert!(taint.contains(&"web".to_owned()), "taint: {taint:?}");
|
|
|
|
// The §15 chain: the approval for the LATER gated action carries the
|
|
// web taint — untrusted content can never quietly reach outward.
|
|
let pending = cm_safety::approvals::list_pending(&pool, ws.id)
|
|
.await
|
|
.unwrap();
|
|
assert_eq!(pending.len(), 1);
|
|
assert!(
|
|
pending[0].taint_sources.contains(&"web".to_owned()),
|
|
"approval taint: {:?}",
|
|
pending[0].taint_sources
|
|
);
|
|
|
|
// The viewport screenshot landed in the blob store as a real PNG.
|
|
use cm_files::BlobStore;
|
|
let key = format!("{}/browser/{}/viewport.png", ws.id, agent.id);
|
|
let png = blob.get(&key).await.expect("screenshot stored");
|
|
assert_eq!(&png[..8], b"\x89PNG\r\n\x1a\n", "PNG magic");
|
|
|
|
// `shutdown` drains the warm pool only; the sandbox assigned to this agent
|
|
// outlives it by design (reused across a redeploy). A test has no next
|
|
// deploy, so it must release its own or the container simply stays.
|
|
browser.release_agent(agent.id).await;
|
|
browser.shutdown().await;
|
|
}
|