cm-runtime: don't drain the warm pool on a flaky sandbox health check
ci / gates (push) Successful in 7s
ci / rust (push) Failing after 36s
ci / frontend (push) Successful in 1m7s
ci / e2e (push) Has been skipped
ci / publish (push) Successful in 2m27s

`driver.health()` can return `Err(_)` on transient docker daemon hiccups
(connection reset, timeout mid-inspect, daemon busy). exec() used
`.unwrap_or(false)` which treated Err as "dead" and:
  1. Destroyed the agent's assigned sandbox.
  2. Fell through to the warm pool and popped one, draining it by 1.
  3. Provisioned a fresh assigned sandbox on top.

Under CI load — where several test processes hit the docker daemon
concurrently — this fired as the warm_pool.rs:72 "reuse must not drain
the pool" flake. The test asserts pool_size == 2 after a reuse; when
health flaked, the reuse turned into a drain-and-refill and the assert
raced the warmer.

Match only a CONFIRMED `Ok(false)` (container dead or 404). On Err,
assume alive; if it really is dead, the subsequent exec surfaces the
error with a clear message instead of silent sandbox destruction and
warm-pool drainage.
This commit is contained in:
Omar Sobh
2026-07-05 19:00:19 -07:00
parent 3f82d9efe4
commit 15cffba2d7
+9 -1
View File
@@ -193,7 +193,15 @@ impl SandboxManager {
id: row.container_id, id: row.container_id,
name: row.name, name: row.name,
}; };
if driver.health(&handle).await.unwrap_or(false) { // Only treat the sandbox as dead on a CONFIRMED `Ok(false)`. Docker
// health checks flake under load (transient connection resets,
// daemon busy) and `.unwrap_or(false)` used to bulldoze the
// assigned sandbox AND drain a warm pool slot on those flakes —
// observed as the warm_pool.rs:72 "reuse must not drain the pool"
// flake. On ambiguous state (Err), assume alive; the subsequent
// exec surfaces any real failure with a clear error to the caller.
let confirmed_dead = matches!(driver.health(&handle).await, Ok(false));
if !confirmed_dead {
return driver return driver
.exec(&handle, &["sh", "-lc", command]) .exec(&handle, &["sh", "-lc", command])
.await .await