feat(missions): give the planner the repository's contents, not just its names

The root listing was not enough. Given names alone the planner wrote "optimise
the hot path" for a crate whose hot path is `add(a: i64, b: i64) -> i64` — a
mission that was unachievable from the moment it was written, and that nothing
discovered until an agent had built a benchmark harness in a VM to measure an
integer addition, honestly reported no improvement was possible, and the judge
correctly failed the phase.

`repo_digest` fetches the whole tree (so "does this have benches/" is a fact, not
an inference) and then file CONTENTS in priority order: manifests first — they
say what the project is — then the README, then source ascending by size, since
a planner learns more from twenty small files than from one large one. Lockfiles
and build output are dropped: enormous, and they say nothing a manifest does not.

THE RULE THIS ENFORCES, and the reason the rendering is its own tested module: a
digest of any repository worth planning against is partial, and a model shown a
partial view without being told it is partial plans as though it saw everything.
So every omission is stated — how many files exist, how many were shown, what
was cut from each, and "anything not shown you have NOT seen". Same distinction
as `Option<u32>` for the subagent probe: "we did not look" and "there is nothing
there" are different facts.

Failures degrade to a stated absence rather than an empty string, and the three
cases stay distinguishable: no repository, a tree that could not be read, and a
tree read but no contents fetched. An unreadable tree is never rendered as an
empty repository.

Two more things the prompt now says, both learned from that run: plan for the
repository as it IS rather than as the description implies (and if the
description asks for something the code cannot support, say so in the task and
plan the phase that establishes the truth, rather than a phase that must fail);
and a mission agent has NO package-registry access. The agent discovered the
second one mid-run and wrote a dependency-free `std::time::Instant` harness after
Criterion could not be added — good adaptation, but nothing had warned it.

549 tests pass, clippy clean. The budget/priority/truncation logic is pure and
tested; only the fetching touches the network.

Co-Authored-By: Claude Opus 5 <[email protected]>

crates/cm-api/src/lib.rs

@@ -29,6 +29,7 @@ pub mod mission_fs;
 pub mod papers;
 pub mod phase_config;
 pub mod session_executor;
+pub mod repo_digest;
 pub mod runtime_preflight;
 pub mod mission_plan;
 pub mod mission_roster;

crates/cm-api/src/repo_digest.rs

@@ -0,0 +1,266 @@
+//! What a repository actually contains, small enough to put in a prompt.
+//!
+//! The planner was given the root listing and planned "optimise the hot path"
+//! for a crate whose hot path is `add(a: i64, b: i64) -> i64`. Names were not
+//! enough: the mission was unachievable from the moment it was written, and
+//! nothing discovered that until an agent had built a benchmark harness to
+//! measure an integer addition.
+//!
+//! # The rule this module exists to enforce
+//!
+//! A digest is always partial for any repository worth planning against, and a
+//! model shown a partial view without being told it is partial plans as though
+//! it saw everything. So every omission is STATED — how many files were listed,
+//! how many were shown, what was cut from each. That is the same distinction as
+//! `Option<u32>` for the subagent probe: "we did not look" and "there is nothing
+//! there" are different facts, and only one of them is about the repository.
+//!
+//! # Priority
+//!
+//! Manifests first (they say what the project IS and what it may depend on),
+//! then the README, then source ascending by size — smallest-first shows the
+//! most files per byte, and a planner benefits more from seeing twenty small
+//! files than one large one.
+
+/// One file in the repository tree.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct FileEntry {
+    pub path: String,
+    pub size: usize,
+}
+
+/// Total characters of file CONTENT a digest may carry.
+///
+/// The prompt around it is ~1.5k, and the planner is a single call per mission,
+/// so this is generous by design: the cost of a too-small digest is a plan built
+/// on a guess, which costs a VM boot to discover.
+pub const CONTENT_BUDGET: usize = 12_000;
+
+/// Ceiling per file, so one large file cannot spend the whole budget.
+pub const PER_FILE_CAP: usize = 3_000;
+
+/// Files worth showing before any source.
+fn is_manifest(path: &str) -> bool {
+    matches!(
+        path,
+        "Cargo.toml"
+            | "package.json"
+            | "pyproject.toml"
+            | "setup.py"
+            | "go.mod"
+            | "Gemfile"
+            | "pom.xml"
+            | "build.gradle"
+            | "Makefile"
+    )
+}
+
+fn is_readme(path: &str) -> bool {
+    path.eq_ignore_ascii_case("README.md") || path.eq_ignore_ascii_case("README")
+}
+
+/// Paths to fetch, in the order they earn their place.
+///
+/// Directories and files a planner cannot use are dropped: lockfiles are huge
+/// and say nothing a manifest does not, and build output is not source.
+pub fn priority(entries: &[FileEntry]) -> Vec<&FileEntry> {
+    let mut useful: Vec<&FileEntry> = entries
+        .iter()
+        .filter(|e| {
+            let p = e.path.as_str();
+            !p.starts_with(".git/")
+                && !p.contains("/target/")
+                && !p.starts_with("target/")
+                && !p.contains("node_modules/")
+                && p != "Cargo.lock"
+                && p != "package-lock.json"
+                && p != "poetry.lock"
+                && e.size > 0
+        })
+        .collect();
+    useful.sort_by_key(|e| {
+        let rank = if is_manifest(&e.path) {
+            0
+        } else if is_readme(&e.path) {
+            1
+        } else {
+            2
+        };
+        (rank, e.size, e.path.clone())
+    });
+    useful
+}
+
+/// Render the digest a planner sees.
+///
+/// `contents` is `(path, text)` for the files that were actually fetched, in
+/// priority order. Anything not fetched is still LISTED, so the model knows the
+/// file exists even when it cannot read it.
+pub fn render(entries: &[FileEntry], contents: &[(String, String)]) -> String {
+    if entries.is_empty() {
+        return "(the repository is empty, or its tree could not be read)".to_string();
+    }
+    let mut out = String::new();
+    out.push_str(&format!("FILES ({} total):\n", entries.len()));
+    // The whole tree by name is cheap and is what stops "does X exist" guessing.
+    // Capped anyway: a 10k-file monorepo listing is not a prompt.
+    const MAX_LISTED: usize = 300;
+    for e in entries.iter().take(MAX_LISTED) {
+        out.push_str(&format!("  {} ({} bytes)\n", e.path, e.size));
+    }
+    if entries.len() > MAX_LISTED {
+        out.push_str(&format!(
+            "  … and {} more files NOT listed\n",
+            entries.len() - MAX_LISTED
+        ));
+    }
+
+    if contents.is_empty() {
+        out.push_str("\n(no file contents could be read — plan from the names alone, and say so if that is not enough)\n");
+        return out;
+    }
+
+    out.push_str(&format!(
+        "\nCONTENTS ({} of {} files shown; anything not shown you have NOT seen):\n",
+        contents.len(),
+        entries.len()
+    ));
+    for (path, text) in contents {
+        out.push_str(&format!("\n--- {path} ---\n{text}\n"));
+    }
+    out
+}
+
+/// Take file texts up to the budget, truncating each at [`PER_FILE_CAP`].
+///
+/// Truncation is marked in the text itself rather than silently cutting: a model
+/// that can see it is reading a fragment asks differently than one that believes
+/// it read the file.
+pub fn fit(fetched: Vec<(String, String)>) -> Vec<(String, String)> {
+    let mut out = Vec::new();
+    let mut spent = 0usize;
+    for (path, text) in fetched {
+        if spent >= CONTENT_BUDGET {
+            break;
+        }
+        let room = (CONTENT_BUDGET - spent).min(PER_FILE_CAP);
+        let text = if text.len() <= room {
+            text
+        } else {
+            let end = (0..=room)
+                .rev()
+                .find(|i| text.is_char_boundary(*i))
+                .unwrap_or(0);
+            format!(
+                "{}\n… [truncated: {} of {} bytes shown]",
+                &text[..end],
+                end,
+                text.len()
+            )
+        };
+        spent += text.len();
+        out.push((path, text));
+    }
+    out
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn f(path: &str, size: usize) -> FileEntry {
+        FileEntry {
+            path: path.into(),
+            size,
+        }
+    }
+
+    /// Manifests first, then the README, then source smallest-first. A planner
+    /// learns more from twenty small files than from one large one.
+    #[test]
+    fn the_files_that_say_what_this_is_come_first() {
+        let entries = vec![
+            f("src/big.rs", 9000),
+            f("README.md", 400),
+            f("src/lib.rs", 120),
+            f("Cargo.toml", 200),
+        ];
+        let order: Vec<&str> = priority(&entries).iter().map(|e| e.path.as_str()).collect();
+        assert_eq!(order, vec!["Cargo.toml", "README.md", "src/lib.rs", "src/big.rs"]);
+    }
+
+    /// Lockfiles and build output are dropped: enormous, and they say nothing a
+    /// manifest does not.
+    #[test]
+    fn noise_is_not_offered_to_the_planner() {
+        let entries = vec![
+            f("Cargo.lock", 50_000),
+            f("target/debug/thing", 900_000),
+            f("node_modules/x/index.js", 400),
+            f(".git/config", 100),
+            f("src/lib.rs", 100),
+            f("empty.rs", 0),
+        ];
+        let kept: Vec<&str> = priority(&entries).iter().map(|e| e.path.as_str()).collect();
+        assert_eq!(kept, vec!["src/lib.rs"]);
+    }
+
+    /// THE rule. A partial view presented as complete is planned against as
+    /// though it were complete — which is how "optimise the hot path" gets
+    /// written for a crate that adds two integers.
+    #[test]
+    fn every_omission_is_stated() {
+        let entries: Vec<FileEntry> = (0..400).map(|i| f(&format!("src/f{i}.rs"), 100)).collect();
+        let shown = vec![("src/f0.rs".to_string(), "fn a() {}".to_string())];
+        let out = render(&entries, &shown);
+
+        assert!(out.contains("FILES (400 total)"), "{out}");
+        assert!(out.contains("and 100 more files NOT listed"), "{out}");
+        assert!(out.contains("1 of 400 files shown"), "{out}");
+        assert!(
+            out.contains("you have NOT seen"),
+            "the model must be told the view is partial: {out}"
+        );
+    }
+
+    /// A file cut short says so, in the text the model reads.
+    #[test]
+    fn a_truncated_file_says_it_was_truncated() {
+        let big = "x".repeat(PER_FILE_CAP * 2);
+        let out = fit(vec![("src/big.rs".into(), big.clone())]);
+        assert_eq!(out.len(), 1);
+        assert!(out[0].1.contains("truncated"), "{}", &out[0].1[..80]);
+        assert!(out[0].1.len() < big.len());
+        // And the marker names both numbers, so "how much did I miss" is
+        // answerable rather than guessable.
+        assert!(out[0].1.contains(&big.len().to_string()));
+    }
+
+    /// The budget is a total, not per file: one large file must not starve the
+    /// rest, and the whole digest must stay promptable.
+    #[test]
+    fn the_budget_bounds_the_whole_digest() {
+        let files: Vec<(String, String)> = (0..20)
+            .map(|i| (format!("src/f{i}.rs"), "y".repeat(PER_FILE_CAP)))
+            .collect();
+        let out = fit(files);
+        let total: usize = out.iter().map(|(_, t)| t.len()).sum();
+        assert!(total <= CONTENT_BUDGET, "digest was {total} bytes");
+        assert!(!out.is_empty(), "and it still shows something");
+        assert!(out.len() < 20, "not everything fits, by construction");
+    }
+
+    /// An empty or unreadable tree is stated as such — never rendered as a
+    /// repository that happens to contain nothing.
+    #[test]
+    fn an_unreadable_tree_is_not_an_empty_repository() {
+        let out = render(&[], &[]);
+        assert!(out.contains("could not be read"), "{out}");
+
+        // A tree we CAN read but no contents we could fetch is a different
+        // fact, and says so.
+        let out = render(&[f("src/lib.rs", 100)], &[]);
+        assert!(out.contains("src/lib.rs"), "{out}");
+        assert!(out.contains("no file contents could be read"), "{out}");
+    }
+}

crates/cm-api/src/routes/mission_plan.rs

@@ -23,18 +23,17 @@ const PLANNER_MODEL: &str = "claude-opus-4-8";
 
 /// What the repository actually contains, for the planner's prompt.
 ///
-/// The planner used to see only the mission's title, description and whether a
+/// Names were not enough. Given the root listing alone, the planner wrote
-/// repo was bound — and it planned accordingly. Its first real plan opened with
+/// "optimise the hot path" for a crate whose hot path is
-/// "run its benchmark harness" against a crate that has none, so the phase found
+/// `add(a: i64, b: i64) -> i64` — a mission that was unachievable from the
-/// nothing to baseline and delivered zero files. A plan about a repository
+/// moment it was written, and that nothing discovered until an agent had built a
-/// written without seeing the repository is a guess.
+/// benchmark harness to measure an integer addition.
 ///
-/// Read from the FORGE rather than a checkout: at proposal time the mission is
+/// Read from the FORGE, not a checkout: at proposal time the mission is still a
-/// still a draft and `ensure_checkout` has not run, so there is nothing on disk
+/// draft and `ensure_checkout` has not run, so there is nothing on disk. Every
-/// to list. One API call, and a failure degrades to a stated absence rather than
+/// failure degrades to a STATED absence — a planner told "the listing could not
-/// to silence — a model told "the listing could not be read" can hedge; a model
+/// be read" can hedge; one told nothing assumes.
-/// told nothing assumes.
+async fn repo_digest(pool: &sqlx::PgPool, mission_id: uuid::Uuid) -> String {
-async fn repo_listing(pool: &sqlx::PgPool, mission_id: uuid::Uuid) -> String {
     let row: Option<(Option<String>, Option<String>, Option<String>)> = sqlx::query_as(
         "SELECT r.owner, r.name, r.default_branch
            FROM missions m JOIN repos r ON r.id = m.repo_id
@@ -48,39 +47,72 @@ async fn repo_listing(pool: &sqlx::PgPool, mission_id: uuid::Uuid) -> String {
     let Some((Some(owner), Some(name), branch)) = row else {
         return "(this mission has no repository)".to_string();
     };
+    let branch = branch.unwrap_or_else(|| "main".to_string());
     let token = std::env::var("GITEA_TOKEN").unwrap_or_default();
-    let url = format!(
+    let Ok(client) = reqwest::Client::builder()
-        "https://git.redclaw.dev/api/v1/repos/{owner}/{name}/contents?ref={}",
+        .timeout(std::time::Duration::from_secs(20))
-        branch.as_deref().unwrap_or("main")
-    );
-    let client = match reqwest::Client::builder()
-        .timeout(std::time::Duration::from_secs(15))
         .build()
-    {
+    else {
-        Ok(c) => c,
+        return "(the repository could not be read)".to_string();
-        Err(_) => return "(the repository listing could not be read)".to_string(),
     };
-    let mut req = client.get(&url);
+    let auth = |r: reqwest::RequestBuilder| {
-    if !token.trim().is_empty() {
+        if token.trim().is_empty() {
-        req = req.header("Authorization", format!("token {token}"));
+            r
+        } else {
+            r.header("Authorization", format!("token {token}"))
         }
-    let entries: Vec<serde_json::Value> = match req.send().await {
+    };
+
+    // The whole tree in one call, so "does this repo have benches/" is a fact
+    // rather than an inference from the root.
+    let tree_url = format!(
+        "https://git.redclaw.dev/api/v1/repos/{owner}/{name}/git/trees/{branch}?recursive=true&per_page=1000"
+    );
+    let tree: serde_json::Value = match auth(client.get(&tree_url)).send().await {
         Ok(r) if r.status().is_success() => r.json().await.unwrap_or_default(),
-        _ => return "(the repository listing could not be read)".to_string(),
+        _ => return "(the repository tree could not be read)".to_string(),
     };
-    if entries.is_empty() {
+    let entries: Vec<crate::repo_digest::FileEntry> = tree
-        return "(the repository root is empty)".to_string();
+        .get("tree")
-    }
+        .and_then(|t| t.as_array())
-    let mut names: Vec<String> = entries
+        .map(|items| {
+            items
                 .iter()
+                .filter(|e| e.get("type").and_then(|v| v.as_str()) == Some("blob"))
                 .filter_map(|e| {
-            let n = e.get("name")?.as_str()?;
+                    Some(crate::repo_digest::FileEntry {
-            let t = e.get("type").and_then(|v| v.as_str()).unwrap_or("file");
+                        path: e.get("path")?.as_str()?.to_string(),
-            Some(if t == "dir" { format!("{n}/") } else { n.to_string() })
+                        size: e.get("size").and_then(|v| v.as_u64()).unwrap_or(0) as usize,
                     })
-        .collect();
+                })
-    names.sort();
+                .collect()
-    names.join(", ")
+        })
+        .unwrap_or_default();
+
+    // Fetch in priority order until the budget is spent. Requested serially and
+    // capped: this runs inside one API request, and a repo with 500 useful files
+    // must not turn a proposal into 500 round trips.
+    let mut fetched: Vec<(String, String)> = Vec::new();
+    let mut spent = 0usize;
+    for e in crate::repo_digest::priority(&entries).into_iter().take(40) {
+        if spent >= crate::repo_digest::CONTENT_BUDGET {
+            break;
+        }
+        let raw = format!(
+            "https://git.redclaw.dev/api/v1/repos/{owner}/{name}/raw/{}?ref={branch}",
+            e.path
+        );
+        if let Ok(r) = auth(client.get(&raw)).send().await {
+            if r.status().is_success() {
+                if let Ok(text) = r.text().await {
+                    spent += text.len().min(crate::repo_digest::PER_FILE_CAP);
+                    fetched.push((e.path.clone(), text));
+                }
+            }
+        }
+    }
+
+    crate::repo_digest::render(&entries, &crate::repo_digest::fit(fetched))
 }
 
 const PLAN_SYSTEM: &str = "You decide what ONE software mission actually does — its phases, in order. \
@@ -125,14 +157,20 @@ pub async fn suggest(
         .ok_or(ApiError::NotFound)?;
 
     let prompt = format!(
-        "MISSION: {}\n\nDESCRIPTION:\n{}\n\nREPOSITORY ROOT: {}\nPlan for the repository as it \
+        "MISSION: {}\n\nDESCRIPTION:\n{}\n\n=== THE REPOSITORY ===\n{}\n=== END REPOSITORY \
-         ACTUALLY IS. If the work needs something the repository does not have — a benchmark \
+         ===\n\nPlan for the repository as it ACTUALLY IS, not as the description implies it \
-         harness, a test suite, a config file — the phase that needs it must CREATE it, and its \
+         might be. If the work needs something absent — a benchmark harness, a test suite, a \
-         task must say so.\n\nPHASE KINDS YOU MAY USE (nothing else runs): {}\nCEILING: \
+         config file — the phase that needs it must CREATE it, and its task must say so. If the \
+         description asks for something this code cannot support (optimising a function with \
+         nothing to optimise, testing a module that does not exist), say so in the task text and \
+         plan the phase that would establish the truth, rather than a phase that must fail.\n\n\
+         NOTE: a mission agent has NO package-registry access — it cannot add dependencies. A \
+         phase needing tooling must build it from the standard library or from what is already \
+         vendored here.\n\nPHASE KINDS YOU MAY USE (nothing else runs): {}\nCEILING: \
          {MAX_PHASES} phases.\n\nPropose the plan now (JSON only).",
         mission.title,
         mission.description.as_deref().unwrap_or("(none)"),
-        repo_listing(&state.pool, id).await,
+        repo_digest(&state.pool, id).await,
         PLANNABLE_KINDS.join(", "),
     );