feat(missions): reap orphaned runtime containers — unless they hold work

`sweep_once` selects `FROM missions`, and `teardown_container` is only
ever called with an id from that query. So a container whose row is gone
is invisible to every reaper: nothing enumerates docker, nothing errors,
and the only symptom is disk.

Found on gw-04 today — `cm-runtime-mission-019ff5b1…`, Up nine days,
2.5G, against a `missions` table with zero rows.

`list_mission_containers` is the piece that never existed: without it
"which containers exist" is a question the platform cannot ask, and a
container the database has forgotten is not merely unreaped, it is
unseeable.

**The sweep refuses to reap work that exists nowhere else.** That
container's checkout held ten commits on a branch that had never been
pushed — +3451/-30 across 30 files, eighteen INT items including
AES-256-GCM, Ed25519 signing and HNSW batch insert. A reaper that deleted
on sight would have destroyed all of it silently, as its designed
behaviour. `unpushed_commits` asks the checkout (`git rev-list --all
--not --remotes`) and leaves the container alone, loudly, every tick,
when the answer is not zero.

Every failure path returns `SomeOrUnknown`: a container we cannot
question is not a container we may delete. Same for one docker will not
date — including a future `Created` from clock skew, which would
otherwise underflow into an age past any grace period.

Grace is 24h, long on purpose. The row-driven sweep already handles
everything the platform knows about, so anything reaching this path is
already unexpected.

The container above was handled by hand first: bundled, verified,
branch pushed to git.redclaw.dev, confirmed on the remote at the branch
tip, then removed. 59G free, up from 57G.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9

crates/cm-api/src/mission_runtime.rs

@@ -1070,6 +1070,103 @@ impl MissionRuntimeProvisioner {
     /// this to decide whether to KEEP a binding for a retry, and answering
     /// "still there" when docker cannot be reached would pin the binding open
     /// on an unreachable daemon rather than on a real container.
+    /// Every `cm-runtime-mission-*` container on this engine, running or not.
+    ///
+    /// The piece the row-driven sweep never had. Without it "which containers
+    /// exist" is a question the platform cannot ask, and a container the
+    /// database has forgotten is not merely unreaped — it is unseeable.
+    pub async fn list_mission_containers(&self) -> Result<Vec<(String, Option<i64>)>, String> {
+        let mut filters = std::collections::HashMap::new();
+        filters.insert("name".to_string(), vec!["cm-runtime-mission-".to_string()]);
+        let opts = bollard::query_parameters::ListContainersOptionsBuilder::default()
+            .all(true)
+            .filters(&filters)
+            .build();
+        let list = self
+            .docker
+            .list_containers(Some(opts))
+            .await
+            .map_err(|e| format!("list mission containers: {e}"))?;
+        Ok(list
+            .into_iter()
+            .filter_map(|c| {
+                let name = c
+                    .names
+                    .unwrap_or_default()
+                    .into_iter()
+                    // Docker returns names with a leading slash.
+                    .map(|n| n.trim_start_matches('/').to_string())
+                    .find(|n| n.starts_with("cm-runtime-mission-"))?;
+                Some((name, c.created))
+            })
+            .collect())
+    }
+
+    /// How long ago docker says this container was created.
+    ///
+    /// Taken from the listing rather than a second `inspect`: `created` is
+    /// already a unix timestamp there, so this needs neither a date parser nor
+    /// another round-trip. `None` when docker reported none, and the caller
+    /// treats that as "do not reap" — a container we cannot date is exactly the
+    /// one worth leaving.
+    pub fn container_age(created_epoch: Option<i64>) -> Option<std::time::Duration> {
+        let created = created_epoch?;
+        let now = std::time::SystemTime::now()
+            .duration_since(std::time::UNIX_EPOCH)
+            .ok()?
+            .as_secs() as i64;
+        u64::try_from(now - created).ok().map(std::time::Duration::from_secs)
+    }
+
+    /// Does this container's checkout hold commits no remote has?
+    ///
+    /// Answered by `git` inside the container, because only it knows which
+    /// refs the remote had. `--not --remotes` lists every commit reachable
+    /// from any local ref and from no remote-tracking ref — which is exactly
+    /// "work that exists only here".
+    ///
+    /// Every failure path returns `SomeOrUnknown`. A container we cannot
+    /// question is not a container we may delete.
+    pub async fn unpushed_commits(&self, name: &str) -> UnpushedWork {
+        let script = "cd /mission/repo 2>/dev/null || exit 91; \
+                      git rev-list --all --not --remotes 2>/dev/null | wc -l";
+        let argv = vec!["sh".to_string(), "-lc".to_string(), script.to_string()];
+        let out = match crate::container_exec::exec_as_root(
+            &self.docker,
+            name,
+            None,
+            &argv,
+            std::time::Duration::from_secs(30),
+        )
+        .await
+        {
+            Ok(o) => o,
+            Err(e) => {
+                return UnpushedWork::SomeOrUnknown(format!("could not ask git ({e})"));
+            }
+        };
+        if out.exit_code == Some(91) {
+            // No checkout at all — nothing to lose.
+            return UnpushedWork::None;
+        }
+        if out.exit_code != Some(0) {
+            return UnpushedWork::SomeOrUnknown(format!(
+                "git probe exited {:?}",
+                out.exit_code
+            ));
+        }
+        match out.stdout.trim().parse::<u64>() {
+            Ok(0) => UnpushedWork::None,
+            Ok(n) => UnpushedWork::SomeOrUnknown(format!(
+                "{n} commit(s) in its checkout are on no remote"
+            )),
+            Err(_) => UnpushedWork::SomeOrUnknown(format!(
+                "unreadable git output {:?}",
+                out.stdout.trim()
+            )),
+        }
+    }
+
     pub async fn container_exists(&self, mission_id: Uuid) -> bool {
         self.docker
             .inspect_container(&container_name(mission_id), None::<InspectContainerOptions>)
@@ -1152,10 +1249,114 @@ pub fn spawn_sweeper(pool: sqlx::PgPool, grace: std::time::Duration) {
             if let Err(e) = sweep_once(&pool, grace).await {
                 eprintln!("mission_runtime::sweeper: sweep failed: {e}");
             }
+            if let Err(e) = sweep_orphans(&pool, ORPHAN_GRACE).await {
+                eprintln!("mission_runtime::sweeper: orphan sweep failed: {e}");
+            }
         }
     });
 }
 
+/// How long a container with no mission row may sit before it is reaped.
+///
+/// Long, deliberately. The row-driven sweep above handles every container the
+/// platform still knows about, so anything reaching this path is already
+/// unexpected — and the one real orphan we have seen held ten unpushed commits.
+/// A day of disk is cheaper than being wrong about that.
+const ORPHAN_GRACE: std::time::Duration = std::time::Duration::from_secs(24 * 3600);
+
+/// Reap `cm-runtime-mission-*` containers that no `missions` row points at.
+///
+/// [`sweep_once`] selects `FROM missions`, and `teardown_container` is only
+/// ever called with an id that came from that query. So a container whose row
+/// is gone is invisible to every reaper: nothing enumerates docker, nothing
+/// errors, and the only symptom is disk.
+///
+/// Found on gw-04 2026-08-21 — a container `Up` for nine days holding 2.5G,
+/// against a `missions` table with **zero rows**.
+///
+/// # It refuses to reap work that exists nowhere else
+///
+/// That container's checkout held **ten commits on a branch that had never
+/// been pushed** (+3451/-30 across 30 files). A reaper that deleted on sight
+/// would have destroyed all of it, silently, as its designed behaviour. So
+/// before removing anything this asks the checkout whether it holds commits
+/// that no remote has, and leaves the container alone — loudly, every tick —
+/// when it does.
+///
+/// The check is deliberately one-sided in the safe direction: an inspection
+/// that fails for any reason counts as "might hold work", never as "safe to
+/// delete". Losing a day of disk to an unreadable container is recoverable;
+/// the other way round is not.
+async fn sweep_orphans(pool: &sqlx::PgPool, grace: std::time::Duration) -> Result<(), String> {
+    let Some(prov) = MissionRuntimeProvisioner::from_env() else {
+        return Ok(());
+    };
+    let names = prov.list_mission_containers().await?;
+    if names.is_empty() {
+        return Ok(());
+    }
+    for (name, created) in names {
+        let Some(id) = mission_id_from_container(&name) else {
+            continue;
+        };
+        // `WHERE id = $1` across every workspace on purpose: the question is
+        // whether ANY row still points at this container, not whether one the
+        // caller can see does.
+        let known: Option<(Uuid,)> = sqlx::query_as("SELECT id FROM missions WHERE id = $1")
+            .bind(id)
+            .fetch_optional(pool)
+            .await
+            .map_err(|e| format!("look up mission {id}: {e}"))?;
+        if known.is_some() {
+            continue;
+        }
+        match MissionRuntimeProvisioner::container_age(created) {
+            Some(age) if age < grace => continue,
+            None => continue,
+            Some(_) => {}
+        }
+        match prov.unpushed_commits(&name).await {
+            // The safe answer, and the one an error also produces.
+            UnpushedWork::SomeOrUnknown(why) => {
+                eprintln!(
+                    "mission_runtime::orphans: {name} has no mission row and is older than \
+                     the grace period, but it is NOT safe to reap: {why}. Recover the work \
+                     (`git bundle create … origin/main..HEAD`, or push the branch) and then \
+                     remove it by hand."
+                );
+            }
+            UnpushedWork::None => {
+                eprintln!(
+                    "mission_runtime::orphans: reaping {name} — no mission row, older than \
+                     the grace period, and its checkout holds nothing a remote does not"
+                );
+                if let Err(e) = prov.teardown_container(id).await {
+                    eprintln!("mission_runtime::orphans: reap {name}: {e}");
+                }
+            }
+        }
+    }
+    Ok(())
+}
+
+/// Whether an orphan's checkout holds commits no remote has.
+#[derive(Debug, PartialEq, Eq)]
+pub enum UnpushedWork {
+    /// Every commit is reachable from a remote ref — nothing is lost.
+    None,
+    /// There IS unpushed work, or the question could not be answered. One
+    /// variant for both, because the reaper must treat them identically.
+    SomeOrUnknown(String),
+}
+
+/// The mission id encoded in a runtime container's name, if it is one.
+///
+/// The inverse of [`container_name`], which formats the uuid `simple` (no
+/// dashes). Anything that does not parse is not ours and is left alone.
+pub fn mission_id_from_container(name: &str) -> Option<Uuid> {
+    Uuid::parse_str(name.strip_prefix("cm-runtime-mission-")?).ok()
+}
+
 async fn sweep_once(pool: &sqlx::PgPool, grace: std::time::Duration) -> Result<(), String> {
     use sqlx::Row;
     let grace_secs = grace.as_secs() as f64;
@@ -1569,6 +1770,48 @@ mod tests {
         }
     }
 
+    /// The name→id round trip the orphan sweep depends on.
+    ///
+    /// If this is wrong the sweep either skips every orphan (harmless) or
+    /// resolves a container to the WRONG mission id and asks the database
+    /// about a mission that does exist — reading "still live, leave it" for a
+    /// container that is not. Cheap to get right, expensive to get wrong.
+    #[test]
+    fn a_container_name_round_trips_to_its_mission() {
+        let id = Uuid::now_v7();
+        assert_eq!(mission_id_from_container(&container_name(id)), Some(id));
+        // Not ours, and not a panic.
+        assert_eq!(mission_id_from_container("clawmates_server_1"), None);
+        assert_eq!(mission_id_from_container("cm-runtime-mission-nonsense"), None);
+        assert_eq!(mission_id_from_container("cm-sandbox-abc"), None);
+    }
+
+    /// A container docker will not date must not be reaped.
+    #[test]
+    fn an_undatable_container_has_no_age() {
+        assert_eq!(MissionRuntimeProvisioner::container_age(None), None);
+        let now = std::time::SystemTime::now()
+            .duration_since(std::time::UNIX_EPOCH)
+            .unwrap()
+            .as_secs() as i64;
+        let age = MissionRuntimeProvisioner::container_age(Some(now - 3600)).expect("age");
+        assert!(age.as_secs() >= 3500 && age.as_secs() <= 3700, "{age:?}");
+        // A clock skew that puts creation in the future must not underflow into
+        // a colossal age that reads as "long past the grace period".
+        assert_eq!(MissionRuntimeProvisioner::container_age(Some(now + 600)), None);
+    }
+
+    /// The grace period is long, and that is the point.
+    #[test]
+    fn the_orphan_grace_is_generous() {
+        assert!(
+            ORPHAN_GRACE >= std::time::Duration::from_secs(12 * 3600),
+            "the row-driven sweep already handles everything the platform knows \
+             about, so anything reaching the orphan path is unexpected — and the \
+             one real orphan held ten unpushed commits"
+        );
+    }
+
     const SAMPLE_CONFIG: &str = r#"# top comment
 [agents.claw_a]
 model_provider = "anthropic.default"