cm-runtime: don't drain the warm pool on a flaky sandbox health check
`driver.health()` can return `Err(_)` on transient docker daemon hiccups (connection reset, timeout mid-inspect, daemon busy). exec() used `.unwrap_or(false)` which treated Err as "dead" and: 1. Destroyed the agent's assigned sandbox. 2. Fell through to the warm pool and popped one, draining it by 1. 3. Provisioned a fresh assigned sandbox on top. Under CI load — where several test processes hit the docker daemon concurrently — this fired as the warm_pool.rs:72 "reuse must not drain the pool" flake. The test asserts pool_size == 2 after a reuse; when health flaked, the reuse turned into a drain-and-refill and the assert raced the warmer. Match only a CONFIRMED `Ok(false)` (container dead or 404). On Err, assume alive; if it really is dead, the subsequent exec surfaces the error with a clear message instead of silent sandbox destruction and warm-pool drainage.
This commit is contained in:
@@ -193,7 +193,15 @@ impl SandboxManager {
|
|||||||
id: row.container_id,
|
id: row.container_id,
|
||||||
name: row.name,
|
name: row.name,
|
||||||
};
|
};
|
||||||
if driver.health(&handle).await.unwrap_or(false) {
|
// Only treat the sandbox as dead on a CONFIRMED `Ok(false)`. Docker
|
||||||
|
// health checks flake under load (transient connection resets,
|
||||||
|
// daemon busy) and `.unwrap_or(false)` used to bulldoze the
|
||||||
|
// assigned sandbox AND drain a warm pool slot on those flakes —
|
||||||
|
// observed as the warm_pool.rs:72 "reuse must not drain the pool"
|
||||||
|
// flake. On ambiguous state (Err), assume alive; the subsequent
|
||||||
|
// exec surfaces any real failure with a clear error to the caller.
|
||||||
|
let confirmed_dead = matches!(driver.health(&handle).await, Ok(false));
|
||||||
|
if !confirmed_dead {
|
||||||
return driver
|
return driver
|
||||||
.exec(&handle, &["sh", "-lc", command])
|
.exec(&handle, &["sh", "-lc", command])
|
||||||
.await
|
.await
|
||||||
|
|||||||
Reference in New Issue
Block a user