//! Did a skill we delivered actually change what the agent did? //! //! Scored on the three axes from `Skill-Use` (arXiv, 2026-08-05): **Trigger** //! (did the agent reach for the skill), **Compliance** (did it follow the //! procedure), **Boundary** (did it avoid what the skill forbids). //! //! ## One axis does not survive the translation, and saying so is the finding //! //! The paper measures agents under *progressive disclosure*: the agent sees a //! name and a description, and must decide to retrieve the body. The retrieval //! is the Trigger event, and it is observable because it is a tool call. //! //! We do not deliver skills that way on the mission path. `pinned_skills_text` //! inlines full bodies into the prompt, so the agent never "reaches for" a //! skill; it is simply holding one. //! //! Trigger is therefore **not observable on the mission path**, and this module //! reports it as `NotObservable` with the reason attached rather than scoring //! it zero. A zero would read as "the agents ignore their skills" when it //! actually means "the question does not apply to how we deliver them" — the //! precise confusion that made 55 empty skill bindings invisible for months. //! //! ### The reason changed, and only half of it went away //! //! Until 2026-08-21 the stated reason was that `claude_cli` "cannot surface a //! tool call at all — there is nothing to retrieve *with*". That half is now //! false: [`crate::container_tool_hooks`] installs `PostToolUse`, mission tool //! calls land in `mission_events`, and a retrieval would be as visible as any //! other call. //! //! The other half still holds, and it is the one that decides the verdict: we //! still **inline**. Trigger is now *instrumentable* and still not //! *observable*, and the blocker has moved from the transport to the delivery //! model. Making it real is one change — serve skills through the door //! (`docs/TOOL-CALL-ARCHITECTURE.md` §3) so retrieval becomes a tool call — //! and nothing else in this module has to change to score it. //! //! ## Compliance and Boundary are scored from ACTIONS where actions exist //! //! They were scored from the concatenated narrative, which is the agent's own //! account of its turn — written by the thing being measured, and silent about //! anything it did not think worth mentioning. [`Evidence`] carries the //! recorded tool calls alongside that text, and every check prefers them: //! `Bash` arguments say what ran, `Write` paths say where it landed. //! //! Every tool-backed check is **one-sided**. It reports a violation it can see //! and never infers compliance from silence — the recorded stream is capped //! per phase ([`crate::mission_events::PER_PHASE_CAP`]), so an absent call is //! not proof of an absent action. use serde::Serialize; /// The outcome of one axis for one skill. #[derive(Debug, Clone, PartialEq, Eq, Serialize)] #[serde(rename_all = "snake_case", tag = "verdict", content = "why")] pub enum Verdict { Pass, /// A pass that can say what it saw. Same tag as [`Verdict::Pass`] on the /// wire — `{"verdict":"pass","why":…}` — so every reader that keys on the /// tag is unaffected and the evidence is there for the one that looks. /// /// Exists because a bare pass on `web-search-triage` would have hidden /// the only finding worth having: the agent's spawn prompts asked for the /// page's date, which the task never did and the skill does. #[serde(rename = "pass")] PassWith(String), Fail(String), /// The skill says nothing this axis can check. NotApplicable, /// The axis cannot be measured here, for a stated structural reason. /// /// Distinct from `Fail` on purpose. Collapsing the two is how a /// measurement reports a system defect as an agent defect. NotObservable(String), } impl Verdict { pub fn label(&self) -> &'static str { match self { Verdict::Pass | Verdict::PassWith(_) => "pass", Verdict::Fail(_) => "FAIL", Verdict::NotApplicable => "n/a", Verdict::NotObservable(_) => "not observable", } } } /// One skill's score for one phase. #[derive(Debug, Clone, Serialize)] pub struct SkillUse { pub skill: String, /// `builtin` (hand-authored) or `promoted_from_brain` (agent-authored). /// /// Carried into the report because an agent that authors its own skill can /// raise its own compliance score without changing what it does. A rising /// number on agent-authored skills has to be visible as such rather than /// averaged in with the rest. pub source_kind: String, pub trigger: Verdict, pub compliance: Verdict, pub boundary: Verdict, } /// The skills a prompt actually delivered. /// /// Delegates to the delivery layer's own parser, so the reader cannot drift /// from the writer. This function originally matched `## ` itself, and /// skill bodies are markdown full of `##` headings — a live mission duly /// scored "Sizing heuristic" and "The output shape" as skills. Parsed from the /// recorded prompt rather than re-derived from the catalogue, because the /// catalogue changes, and now that agents author their own skills it changes /// by itself. pub fn skills_in_prompt(prompt: &str) -> Vec { crate::topology_exec::skill_names_in(prompt) } /// What a mission left behind, for the checks to read. /// /// Two sources, deliberately kept apart. `text` is what the agent *said* it /// did; `tools` is what it did. Where they disagree the tools win, and a check /// that can only consult `text` says so in its verdict rather than presenting a /// narrative reading as an observation. pub struct Evidence<'a> { /// Concatenated `reasoning` events for the mission. pub text: &'a str, /// Every `tool.call` recorded for the mission, in order. pub tools: &'a [crate::mission_events::ToolEvidence], } /// Tools that modify a file. `target` on the recorded event is the tool name. const WRITE_TOOLS: [&str; 4] = ["Write", "Edit", "MultiEdit", "NotebookEdit"]; impl<'a> Evidence<'a> { pub fn new(text: &'a str, tools: &'a [crate::mission_events::ToolEvidence]) -> Self { Evidence { text, tools } } /// Narrative only — a mission whose events predate the tool tap. pub fn from_text(text: &'a str) -> Self { Evidence { text, tools: &[] } } /// Every shell command the agent ran, in order. fn commands(&self) -> impl Iterator { self.tools.iter().filter_map(|t| t.command()) } /// Test runs, in order, each with whether it went red or green. /// /// `None` where the outcome was not recorded — a mission whose events /// predate the response capture, which must not read as either. fn test_runs(&self) -> Vec> { self.tools .iter() .filter(|t| t.command().is_some_and(is_test_command)) .map(|t| test_passed(&t.response)) .collect() } /// Every file-modifying call that named a path, in order. fn writes(&self) -> impl Iterator { self.tools .iter() .filter(|t| WRITE_TOOLS.contains(&t.tool.as_str()) && t.path.is_some()) } } /// Skills the agent RETRIEVED, in the order it reached for them. /// /// This is Trigger, and it is observable the moment a skill arrives by /// retrieval instead of by inlining. Claude Code reads an MCP resource through /// `ReadMcpResourceTool`, and the recorded arguments carry the URI: /// /// ```text /// ReadMcpResourceTool {"uri":"skill:global/workspace-repo-commit-protocol", /// "server":"clawmates_skills"} /// ``` /// /// The URI is parsed by `mcp_skills::parse_uri` — the same function that /// WROTE it — rather than by a second matcher here, for the reason every other /// reader in this module delegates: two implementations of one format drift, /// and the drift shows up as a skill silently scoring nothing. pub fn retrieved_skills(ev: &Evidence<'_>) -> Vec { let mut out = Vec::new(); for t in ev.tools { let name = match t.tool.as_str() { "ReadMcpResourceTool" => t .input .get("uri") .and_then(|v| v.as_str()) .and_then(crate::mcp_skills::parse_uri) .map(|(_, name)| name), // The `files` arm: the pointer is a path and the retrieval is a // plain `Read`. Matched through `skill_from_file_path`, the reader // half of the function that wrote the path, for the same reason // the uri goes through `parse_uri`. A `Read` anywhere else is an // ordinary file read and is not a retrieval of anything. "Read" => t .input .get("file_path") .and_then(|v| v.as_str()) .and_then(crate::skill_delivery::skill_from_file_path), _ => None, }; if let Some(name) = name { if !out.contains(&name) { out.push(name); } } } out } /// Did the agent reach for this skill? /// /// The answer depends on whether it was ever given the chance, which is what /// the delivery arm decides — so this takes the arm rather than assuming one. /// Getting that wrong is not a rounding error: under `Index` the old text /// would have said "this skill was inlined into the prompt" about a skill that /// was not, and scored a real miss as a structural blind spot. fn trigger_verdict( mode: crate::skill_delivery::Mode, retrieved: bool, compliance: &Verdict, boundary: &Verdict, ) -> Verdict { if retrieved { // The agent reached for it. That is the paper's Trigger, and it is a // recorded tool call like any other. return Verdict::Pass; } match mode { // Handed over, so there was no reaching-for to observe. Not a failure // and not a pass — the axis simply does not exist in this arm. crate::skill_delivery::Mode::Inline => Verdict::NotObservable( "this skill was inlined into the prompt, not retrieved — the agent \ was handed it, so there is no reaching-for to observe. Serve it \ through the door instead and this becomes a tool call" .into(), ), // Both retrieval arms: offered by name and `when_to_use`, and never // opened. Whether that is a miss depends on whether the skill had // anything to say about this phase at all: a skill with no // machine-checkable consequence here is one an agent is right to pass // over, and scoring that as a failure would punish correct triage. crate::skill_delivery::Mode::Index | crate::skill_delivery::Mode::Files => { if matches!(compliance, Verdict::NotApplicable) && matches!(boundary, Verdict::NotApplicable) { Verdict::NotApplicable } else { Verdict::Fail( "offered in the index with its `when_to_use`, and never read — \ the agent had the entry in front of it and did not fetch the \ procedure" .into(), ) } } } } /// Score every skill a phase's prompt delivered, against what the agent produced. pub fn score( prompt: &str, evidence: &Evidence<'_>, source_kinds: &dyn Fn(&str) -> String, ) -> Vec { let retrieved = retrieved_skills(evidence); // Read off the prompt that was actually sent, not off the mission row: the // row says what the mission is configured to do now, and this is scoring a // turn that ran then. let mode = crate::skill_delivery::mode_in_prompt(prompt); // Delivered by either route. A skill that was retrieved and never inlined // is invisible to `skills_in_prompt`, and under progressive disclosure that // is EVERY skill — so scoring only the prompt would report zero for the // delivery model this axis exists to measure. let mut names = skills_in_prompt(prompt); for r in &retrieved { if !names.contains(r) { names.push(r.clone()); } } names .into_iter() .map(|skill| { let (compliance, boundary) = check(&skill, evidence); // The arm belongs to the prompt; `always_inject` belongs to the // skill. A skill whose BODY is in the prompt was handed over, so // there is no reaching-for to observe even under `Index` — scoring // it as a Trigger miss would report a failure against an agent that // was never asked to fetch anything. let delivered = match crate::skill_delivery::skill_was_indexed(prompt, &skill) { Some(false) => crate::skill_delivery::Mode::Inline, _ => mode, }; SkillUse { source_kind: source_kinds(&skill), trigger: trigger_verdict( delivered, retrieved.contains(&skill), &compliance, &boundary, ), compliance, boundary, skill, } }) .collect() } /// Per-skill mechanical checks. /// /// Only skills whose procedure has a machine-checkable consequence are checked. /// Everything else returns `NotApplicable` rather than a guess: a heuristic /// that scores prose by keyword overlap produces a number that looks like a /// measurement and is not one. fn check(skill: &str, ev: &Evidence<'_>) -> (Verdict, Verdict) { match skill { "int-xx-marker-protocol" => (marker_compliance(ev.text), marker_boundary(ev.text)), "arxiv-daily" => (Verdict::NotApplicable, arxiv_boundary(ev)), "workspace-repo-commit-protocol" => (Verdict::NotApplicable, workspace_boundary(ev)), "small-focused-commits" => (Verdict::NotApplicable, commit_subject_boundary(ev)), "web-search-triage" => (triage_compliance(ev), Verdict::NotApplicable), // One check for both: `cargo-test-driven-development` is the Rust // flavour of the same loop, and its own text says so. Scoring them by // separate rules would mean two rules for one procedure, which is how // the microVM and container taps drifted apart. "tdd-red-green-refactor" | "cargo-test-driven-development" => { (red_before_green(ev), Verdict::NotApplicable) } _ => (Verdict::NotApplicable, Verdict::NotApplicable), } } /// The markers must be ones `task_card_parser` actually parses. /// /// Checked by running the real parser rather than a copy of its rules — a /// second implementation of the contract would drift from the first, and then /// the measurement would pass while the mission loop still stalled. fn marker_compliance(output: &str) -> Verdict { if crate::task_card_parser::parse(output).is_empty() { // Only a failure if the output looks like it TRIED. A turn with no // marker-shaped line was probably not a coding turn at all. if output.lines().any(|l| looks_like_marker_attempt(l)) { return Verdict::Fail( "emitted marker-shaped lines that the parser does not accept — \ the mission loop will not advance" .into(), ); } return Verdict::NotApplicable; } Verdict::Pass } /// Rule 1: exactly one INT id per marker line. /// /// Two shapes violate it, and they fail differently: /// /// `COMPLETED: INT-05, INT-06` — the parser takes the first and drops the /// rest, so an item is silently never closed. /// `PLAN_COMPLETE: INT-01..02` — the range form. The parser now REJECTS a /// malformed id, so this is caught by the /// compliance check above as a marker the /// parser does not accept. /// /// The second was found by running this measurement against a live mission. It /// is the worse of the two, because a dropped marker leaves a gap and a /// malformed one leaves a plausible-looking row. fn marker_boundary(output: &str) -> Verdict { for line in output.lines() { let t = line.trim(); if !looks_like_marker_attempt(t) { continue; } let ids = t.matches("INT-").count(); if ids > 1 { return Verdict::Fail(format!( "{ids} INT ids on one marker line — only the first parses, so \ the rest are silently dropped: {t:?}" )); } } Verdict::NotApplicable } fn looks_like_marker_attempt(line: &str) -> bool { const KINDS: &[&str] = &[ "TASK:", "WORK:", "HANDOFF:", "TEST_PASS:", "TEST_FAIL:", "REVIEW_APPROVE:", "REVIEW_BLOCK:", "COMPLETED:", "PLAN_COMPLETE:", ]; let t = line.trim().trim_start_matches(['*', '#', '-', '`', ' ']); KINDS.iter().any(|k| t.starts_with(k)) } /// The mission checkout is `/mission/repo`. Working anywhere else is not /// delivered. /// /// Scored as a BOUNDARY rather than compliance: the skill's positive /// instruction ("cd there at the start of every substantive turn") has no /// reliable trace in the output, but writing source somewhere that is never /// collected does, and it is the failure that costs a whole phase. /// /// This check exists because the skill itself was wrong. It taught /// `/workspace/repo` — a path the platform does not mount — while the same /// prompt told the agent `/mission/repo`. An agent that obeyed the skill wrote /// into a directory nothing collects. Corrected 2026-08-19, and /// `skills_loader::contradiction_tests` now holds it. fn workspace_boundary(ev: &Evidence<'_>) -> Verdict { // The recorded writes settle this directly. Before the tool tap the only // available evidence was the narrative, which answers a different question: // an agent that wrote to the wrong root without narrating it scored a // clean pass. let stray: Vec<&str> = ev .writes() .filter_map(|w| w.path.as_deref()) .filter(|p| out_of_bounds(p)) .collect(); if let Some(first) = stray.first() { return Verdict::Fail(format!( "wrote to {first} ({} write(s) outside the checkout) — the mission \ checkout is /mission/repo, so anything written there is never \ collected and the phase delivers nothing", stray.len() )); } // `--force` is the skill's other hard prohibition, and unlike the path rule // it leaves no trace anywhere else: the push succeeds and the history it // overwrote is gone. if let Some(cmd) = ev.commands().find(|c| is_force_push(c)) { return Verdict::Fail(format!( "force-pushed ({cmd:?}) — the skill allows it only after an explicit \ `HANDOFF: safe to force-push` from the reviewer, because rewriting \ pushed history is the one thing here that cannot be undone" )); } // No violation seen. Whether that is a pass depends on whether there was // anything to see. if ev.writes().next().is_some() { return Verdict::Pass; } if ev.text.contains("/workspace/repo") { return Verdict::Fail( "the narrative describes working in /workspace/repo — the mission \ checkout is /mission/repo, so anything written there is never \ collected and the phase delivers nothing" .into(), ); } if ev.text.contains("/mission/repo") { return Verdict::Pass; } // Silence is not compliance. This returned `Pass` before, which is how a // mission with no evidence at all scored the same as one that was checked. Verdict::NotObservable( "no file-modifying tool call was recorded and the narrative names no \ checkout — there is nothing here to place either inside or outside \ /mission/repo" .into(), ) } /// Is this write outside the mission checkout? /// /// Relative paths are skipped: the working directory is not recorded, so /// `src/a.rs` could be either, and guessing would fail honest work. /// /// The scratch roots are excluded because the skill itself blesses them — /// "anything else is scratch and will not be delivered" — and the hooks, the /// stop gate and claude's own state all live under `/root` by design. fn out_of_bounds(path: &str) -> bool { const SCRATCH: [&str; 5] = ["/tmp/", "/var/tmp/", "/root/", "/dev/", "/proc/"]; path.starts_with('/') && !path.starts_with("/mission/repo") && !SCRATCH.iter().any(|s| path.starts_with(s)) } /// Matched on whole tokens, outside the commit message. /// /// Both halves are load-bearing, and each was a live false positive before the /// negative control caught it: /// /// - `cmd.contains("-f ")` fires on `--follow-tags`. /// - Tokenising the whole command fires on /// `git commit -m "document the -f flag" && git push`, where `-f` is prose. /// /// A boundary check that fails honest work gets switched off, which costs more /// than the check was worth. fn is_force_push(cmd: &str) -> bool { const FORCE: [&str; 3] = ["--force", "-f", "--force-with-lease"]; let stripped = match dash_m_value(cmd) { Some(msg) => cmd.replace(&msg, " "), None => cmd.to_string(), }; let tokens = || stripped.split_whitespace(); let pushes = stripped.contains("git") && tokens().any(|t| t == "push"); pushes && tokens().any(|t| FORCE.contains(&t)) } /// The commit subjects the skill names as never-merge. /// /// Matched exactly, after stripping an `INT-NN` prefix. A substring match would /// fail `fix the parser panic`, which is a perfectly good subject — the skill's /// objection is to `fix` as the WHOLE message, not to the word. fn commit_subject_boundary(ev: &Evidence<'_>) -> Verdict { const NEVER_MERGE: [&str; 7] = [ "wip", "fix", "update", "updates", "update stuff", "stuff", "wip commit", ]; for subject in commit_subjects(ev) { let bare = strip_int_prefix(&subject).to_ascii_lowercase(); let bare = bare.trim_matches(|c: char| !c.is_alphanumeric()).trim(); if NEVER_MERGE.contains(&bare) { return Verdict::Fail(format!( "committed {subject:?} — the skill lists this exact message as \ never-merge, because a reviewer and a bisect both need the \ subject to stand alone in `git log --oneline`" )); } } Verdict::NotApplicable } fn strip_int_prefix(subject: &str) -> &str { let s = subject.trim().trim_start_matches(['<', '[', '(']); let Some(rest) = s.strip_prefix("INT-") else { return subject.trim(); }; let rest = rest.trim_start_matches(|c: char| c.is_ascii_digit()); rest.trim_start_matches(['>', ']', ')', ':', '-', '—', ' ']) .trim() } /// Did a test ever run, and did it run first? /// /// One-sided by construction, because only one of the three answers is /// unambiguous: /// /// - **Files written, no test command anywhere** — red-green did not happen. /// There is no reading of the loop under which it did. /// - **A test ran before the first write** — that is the RED step, observed. /// - **A test ran after the first write** — undecidable, and reported as such. /// In Rust the unit test lives in `#[cfg(test)] mod tests` *inside the file /// under test, so "wrote the file, then ran the test" is what writing the /// failing test first looks like from here. Scoring that as a failure would /// fail the agents who followed the skill most precisely. /// Did this test run pass? /// /// Read from the runner's own verdict line, not from an exit code — Claude /// Code's `tool_response` carries `stdout`/`stderr`/`interrupted` and no /// status. That is why `bounded_response` keeps the END of the output: every /// one of these markers is printed last. fn test_passed(response: &serde_json::Value) -> Option { let text = format!( "{}\n{}", response.get("stdout").and_then(|v| v.as_str()).unwrap_or(""), response.get("stderr").and_then(|v| v.as_str()).unwrap_or("") ); if text.trim().is_empty() { return None; } const FAILED: [&str; 6] = [ "test result: FAILED", "FAILED (", "error: test failed", "Tests: ", "failures:", "error[E", ]; const PASSED: [&str; 4] = ["test result: ok", "ok. ", "passed", "PASS"]; // Failure first: a run that reports both a passing binary and a failing one // is a failing run, and the summary line of the failure is what matters. if FAILED.iter().any(|m| text.contains(m)) { return Some(false); } if PASSED.iter().any(|m| text.contains(m)) { return Some(true); } None } fn red_before_green(ev: &Evidence<'_>) -> Verdict { if ev.tools.is_empty() { return Verdict::NotObservable( "no tool calls were recorded for this mission — the order of test \ runs and writes is what this check reads, and neither is in the \ narrative" .into(), ); } let first_write = ev.tools.iter().position(|t| { WRITE_TOOLS.contains(&t.tool.as_str()) && t.path.as_deref().is_some_and(is_source) }); let first_test = ev .tools .iter() .position(|t| t.command().is_some_and(is_test_command)); match (first_write, first_test) { // Nothing was written, so there was no implementation to test-drive. (None, _) => Verdict::NotApplicable, (Some(_), None) => Verdict::Fail(format!( "wrote {} source file(s) and never ran a test — the loop is red, \ green, refactor, and a test that never ran cannot have been red", ev.writes() .filter(|w| w.path.as_deref().is_some_and(is_source)) .count() )), (Some(w), Some(t)) if t < w => Verdict::Pass, _ => { // Ordering is exhausted here, and so is the repository: the skill // says to "commit the RED-to-GREEN pair as one commit", so the // failing test and its fix land together by instruction. The only // remaining witness is what the runs THEMSELVES reported. let runs = ev.test_runs(); let went_red = runs.iter().position(|r| *r == Some(false)); let went_green = runs.iter().rposition(|r| *r == Some(true)); match (went_red, went_green) { (Some(red), Some(green)) if red < green => Verdict::Pass, (Some(_), None) => Verdict::Fail( "every recorded test run failed — the loop ends on green, \ and this phase never got there" .into(), ), _ if runs.iter().all(Option::is_none) => Verdict::NotObservable( "a test ran after the first write and no run recorded its \ output — in Rust the unit test lives in the file under \ test, so ordering cannot separate red-first from \ tests-added-after, and without the run's own verdict there \ is nothing else to read" .into(), ), _ => Verdict::NotObservable( "every recorded test run passed — a test that never failed \ is a test that was never red, but it is equally what a \ correct implementation written first looks like, and these \ runs cannot tell them apart" .into(), ), } } } } /// Is this file behaviour-adding code, in the languages the skill names? /// /// The check is scoped to source because the first live run scored `fail` /// against a **research** phase: its agents wrote fifteen markdown notes and /// never ran a test, which is not a violation of anything — there was no code /// to test-drive. Reporting that as an agent failure would be a system defect /// wearing an agent's name, and the real finding it obscures is that a research /// role is pinned TDD skills at all. /// /// Extensions the skill itself names — "Rust, TypeScript, Python, anywhere /// tests can run cheap" — plus their immediate neighbours. Prose, config and /// data are excluded. Shell is excluded deliberately: a helper script written /// during a research turn is not the behaviour-adding code this loop is about, /// and the false failure costs more than the missed one. fn is_source(path: &str) -> bool { const SOURCE: [&str; 10] = [ ".rs", ".ts", ".tsx", ".py", ".js", ".jsx", ".go", ".java", ".rb", ".kt", ]; let p = path.to_ascii_lowercase(); SOURCE.iter().any(|e| p.ends_with(e)) } /// Commands that run a test suite, across the languages the skills name. fn is_test_command(cmd: &str) -> bool { const RUNNERS: [&str; 9] = [ "cargo test", "cargo nextest", "cargo llvm-cov", "npm test", "yarn test", "pnpm test", "vitest", "pytest", "go test", ]; // `jest` on its own is too short to match safely — it appears inside // package names and paths — so it is matched as a command word. RUNNERS.iter().any(|r| cmd.contains(r)) || cmd.split_whitespace().any(|w| w == "jest" || w == "npx") && cmd.contains("jest") } /// The subject line of every commit the agent made. /// /// Read out of the `-m` argument. A commit written with `-F` or a heredoc /// yields nothing, and the caller must treat an empty result as "no evidence" /// rather than "no commits" — which is why every commit check here returns /// `NotApplicable` on an empty list instead of a verdict. fn commit_subjects(ev: &Evidence<'_>) -> Vec { ev.commands() .filter(|c| c.contains("git commit") || c.contains("git ci")) .filter_map(dash_m_value) .filter_map(|m| subject_line(&m)) .collect() } /// The subject out of a `-m` value. /// /// Normally the first line. But Claude Code writes a multi-line message as /// /// ```text /// git commit -m "$(cat <<'EOF' /// INT-01 Add slugify function /// /// …body… /// EOF /// )" /// ``` /// /// and the first line of that value is `$(cat <<'EOF'` — the heredoc *opener*, /// not the subject. Observed on the first live coding run; it scored no /// violation only because `$(cat <<'EOF'` happens not to be one of the /// never-merge messages, which is luck, not a check. /// /// So: if the first line opens a heredoc or a command substitution, the subject /// is the next non-empty line. fn subject_line(msg: &str) -> Option { let mut lines = msg.lines().map(str::trim).filter(|l| !l.is_empty()); let first = lines.next()?; if first.starts_with("$(") || first.contains("<<") { return lines.next().map(str::to_string); } Some(first.to_string()) } /// The value of the `-m` flag in a shell command. /// /// Hand-scanned rather than split on whitespace: the value is the one argument /// guaranteed to contain spaces, and the commit template in /// `workspace-repo-commit-protocol` spans several lines. fn dash_m_value(cmd: &str) -> Option { let b = cmd.as_bytes(); for i in 0..b.len() { // Anchor on the `m`, not on `-m`: agents write `git commit -am …` and // anchoring on the dash misses the whole cluster form. if b[i] != b'm' { continue; } // `m` must END the cluster, so `--message` and `-mtime` are not it. if !matches!( b.get(i + 1), None | Some(b' ') | Some(b'\t') | Some(b'"') | Some(b'\'') ) { continue; } // Walk back over the cluster's other short flags to its single dash. let mut j = i; while j > 0 && b[j - 1].is_ascii_alphabetic() { j -= 1; } if j == 0 || b[j - 1] != b'-' { continue; } let dash = j - 1; // One dash, starting a word — not `--m` and not the tail of a path. if dash > 0 && !b[dash - 1].is_ascii_whitespace() { continue; } let mut k = i + 1; while k < b.len() && b[k].is_ascii_whitespace() { k += 1; } if k >= b.len() { return None; } let rest = &cmd[k..]; let quote = rest.chars().next().filter(|c| *c == '"' || *c == '\''); return match quote { Some(q) => rest[1..].split(q).next().map(str::to_string), None => rest.split_whitespace().next().map(str::to_string), }; } None } /// The tools whose arguments carry a URL the agent reached for. /// /// `Agent` is here on purpose. On both `files`-arm runs the parent decomposed /// the sweep into per-source fetches and sent each to a subagent — the URLs /// live in the spawn PROMPT, and a check that only read `curl` lines would /// have scored those runs as fetching nothing. const FETCH_TOOLS: &[&str] = &["Bash", "Agent", "WebFetch", "web_fetch"]; /// Every `http(s)` URL in the arguments of a fetching tool, in call order. fn fetched_urls(ev: &Evidence<'_>) -> Vec { let mut out = Vec::new(); for t in ev.tools { if !FETCH_TOOLS.contains(&t.tool.as_str()) { continue; } // A `Bash` that never fetches is most of what agents run. if t.tool == "Bash" && !t .command() .is_some_and(|c| c.contains("curl") || c.contains("wget")) { continue; } let text = t.input.to_string(); let mut rest = text.as_str(); while let Some(i) = rest.find("http") { let cand = &rest[i..]; let end = cand .find(|c: char| { c.is_whitespace() || matches!(c, '"' | '\'' | '<' | '>' | ')' | ']' | '\\') }) .unwrap_or(cand.len()); let url = cand[..end].trim_end_matches(|c| matches!(c, '.' | ',' | ';' | ':')); if url.starts_with("http://") || url.starts_with("https://") { out.push(url.to_string()); } rest = &cand[end.max(4)..]; } } out } /// Rank 0 in `web-search-triage`'s ladder: the paper, the spec, the release. /// /// A conservative allow-list. Anything not on it is UNRANKED, not rank 3 — a /// vendor's docs, an author's own blog and a lab page are all rank 0 or 1 and /// none of them can be recognised by hostname. fn is_primary_source(url: &str) -> bool { const HOSTS: &[&str] = &[ "arxiv.org/abs/", "arxiv.org/pdf/", "doi.org/", "aclanthology.org/", "openreview.net/", "proceedings.neurips.cc/", "proceedings.mlr.press/", "dl.acm.org/doi/", "ieeexplore.ieee.org/", "github.com/", "nature.com/articles/", "science.org/doi/", ]; HOSTS.iter().any(|h| url.contains(h)) } /// Rank 3 and below: a restatement of a restatement. The skill's own words are /// "the same item and should be recorded once, if at all", and its skip signals /// — "a numbered list of tools", no date — describe these hosts. /// /// Also conservative. Substack, X and personal blogs are NOT here: an author's /// own post is rank 1 and the skill says to read it. fn is_aggregator(url: &str) -> bool { const HOSTS: &[&str] = &[ "medium.com/", "towardsdatascience.com/", "reddit.com/", "news.ycombinator.com/", "quora.com/", "dev.to/", "linkedin.com/", "wikipedia.org/", ]; HOSTS.iter().any(|h| url.contains(h)) } /// `web-search-triage`: read the primary source, and treat undated as a finding. /// /// Two of the skill's rules leave a mark in the recorded ARGUMENTS, and this /// scores exactly those two — the rest of the skill is judgement about page /// content the tap never sees, and a heuristic over it would be a number that /// looks like a measurement and is not one. /// /// - The ranking rule. Every URL a fetch was sent to is classified against a /// short allow-list of primary hosts and a short skip-list of aggregators. /// Fetching an aggregator is the visible violation; fetching primary sources /// is the visible compliance. Anything unrecognised is unranked and decides /// nothing. /// - The date rule. On mission `01a09b42` the parent's spawn prompts read /// "Return the URL, date if visible, and the key content" — the task never /// asked for a date; the skill's "undated is a finding" did. Reported as /// extra evidence on a pass, never required for one: a curl to an abstract /// page has no prompt to ask in. /// /// One-sided like every check here: it reports a violation it can see and never /// infers compliance from silence. fn triage_compliance(ev: &Evidence<'_>) -> Verdict { if ev.tools.is_empty() { return Verdict::NotObservable( "no tool calls were recorded for this mission — which URLs were \ fetched is what this check reads, and that is not in the narrative" .into(), ); } let urls = fetched_urls(ev); if urls.is_empty() { // Never swept the web, so there was nothing to triage. return Verdict::NotApplicable; } if let Some(u) = urls.iter().find(|u| is_aggregator(u)) { return Verdict::Fail(format!( "fetched {u} — an aggregator, rank 3 or below on the skill's ladder; \ the procedure is to find the primary source and record the rest as \ one item, not to read them" )); } let primary = urls.iter().filter(|u| is_primary_source(u)).count(); if primary == 0 { return Verdict::NotObservable(format!( "fetched {} URL(s), none on the primary-source list and none on the \ aggregator list — the check cannot rank them, and a rank it cannot \ see is not a violation", urls.len() )); } let asked_for_date = ev .tools .iter() .filter(|t| t.tool == "Agent") .filter(|t| { t.input .get("prompt") .and_then(serde_json::Value::as_str) .is_some_and(|p| p.to_ascii_lowercase().contains("date")) }) .count(); let mut why = format!( "fetched {primary} primary source(s) out of {} URL(s) and no aggregator", urls.len() ); if asked_for_date > 0 { why.push_str(&format!( "; {asked_for_date} fetch(es) delegated to a subagent asked for the \ page's date, which the task did not and the skill does" )); } Verdict::PassWith(why) } /// `arxiv-daily` forbids searching arXiv — the harvest already ran. /// /// This is the one boundary we have watched an agent cross in production, so it /// is checked against the query endpoint rather than the word "arxiv", which /// appears legitimately all over a research turn. fn arxiv_boundary(ev: &Evidence<'_>) -> Verdict { // The query endpoint, not the word "arxiv" — and not `arxiv.org/abs/` // either, which the same skill explicitly tells the agent to `curl`. A // check that could not tell those apart would fail agents for following // the paragraph directly under the one it is scoring. const QUERY_MARKERS: &[&str] = &[ "export.arxiv.org/api/query", "arxiv.org/api/query", "http://export.arxiv.org", ]; let hit = |s: &str| QUERY_MARKERS.iter().find(|m| s.contains(**m)).copied(); // Arguments first: a `curl` that ran is a fact, where a narrative that // mentions a URL may be the agent explaining that it did NOT fetch it. for t in ev.tools { let text = t.input.to_string(); if let Some(m) = hit(&text) { return Verdict::Fail(format!( "called {} against the arXiv API ({m}) — the harvest already \ ran, and shelving papers outside it corrupts the seen-set", t.tool )); } } if let Some(m) = hit(ev.text) { return Verdict::Fail(format!( "queried the arXiv API ({m}) — the harvest already ran, and \ shelving papers outside it corrupts the seen-set" )); } Verdict::Pass } /// Score every skill delivered during a mission, from what was recorded. /// /// Reads `prompt.composed` and `reasoning` rows. Both are per-mission and /// ordered, so the prompts say what was delivered and the narratives say what /// came back. Scoring is against the CONCATENATED output for the mission rather /// than turn-by-turn: a procedure can be followed in a later turn than the one /// that carried it, and pairing strictly by turn would score that as a failure. /// /// Returns an empty vec for a mission whose events have already been reaped — /// which is why `missions.retain_events_until` exists. An empty result means /// "no evidence", never "no compliance", and the report has to say so. pub async fn score_mission( pool: &sqlx::PgPool, mission_id: uuid::Uuid, ) -> Result, String> { let rows = crate::mission_events::narrative_for_mission(pool, mission_id) .await .map_err(|e| format!("read narrative: {e}"))?; let mut prompts = String::new(); let mut outputs = String::new(); for (kind, _, _, text) in &rows { if kind == crate::mission_events::PROMPT_COMPOSED { prompts.push_str(text); prompts.push('\n'); } else { outputs.push_str(text); outputs.push('\n'); } } // What the agent DID, alongside what it said. A mission that ran before // the tool tap existed returns an empty list, and the checks degrade to // the narrative rather than reporting a violation they cannot see. let tools = crate::mission_events::tool_evidence_for_mission(pool, mission_id) .await .map_err(|e| format!("read tool evidence: {e}"))?; // One lookup for every delivered name, so the report can separate // hand-authored skills from the ones agents wrote for themselves. let names = skills_in_prompt(&prompts); let kinds: std::collections::HashMap = sqlx::query_as::<_, (String, String)>( "SELECT name, source_kind FROM skills WHERE name = ANY($1)", ) .bind(&names) .fetch_all(pool) .await .map_err(|e| format!("read skill sources: {e}"))? .into_iter() .collect(); Ok(score(&prompts, &Evidence::new(&outputs, &tools), &|name| { kinds .get(name) .cloned() // A skill in a prompt with no catalogue row was delivered and then // deleted. Naming that explicitly beats defaulting it to builtin. .unwrap_or_else(|| "unknown (no catalogue row)".to_string()) })) } #[cfg(test)] mod tests { use super::*; use crate::mission_events::ToolEvidence; use serde_json::json; fn builtin(_: &str) -> String { "builtin".to_string() } /// Narrative-only evidence — a mission whose events predate the tool tap. fn narrative(text: &str) -> Evidence<'_> { Evidence::from_text(text) } /// Evidence from recorded ACTIONS. `(tool, path, input)`. fn acted(calls: &[(&str, Option<&str>, serde_json::Value)]) -> Vec { calls .iter() .map(|(tool, path, input)| ToolEvidence { tool: (*tool).to_string(), path: path.map(str::to_string), input: input.clone(), response: serde_json::Value::Null, }) .collect() } /// A `Bash` call that ran a test and reported an outcome. fn test_run(cmd: &str, stdout: &str) -> ToolEvidence { ToolEvidence { tool: "Bash".into(), path: None, input: json!({ "command": cmd }), response: json!({ "stdout": stdout, "stderr": "" }), } } /// One `Bash` call. fn ran(cmd: &str) -> (&'static str, Option<&'static str>, serde_json::Value) { ("Bash", None, serde_json::json!({ "command": cmd })) } /// A prompt exactly as the delivery layer renders it. fn rendered(skills: &[(&str, &str)]) -> String { let body: String = skills .iter() .map(|(n, b)| crate::topology_exec::render_pinned_skill(n, b)) .collect(); crate::topology_exec::compose_turn_prompt( "Task: do the thing", Some(&body), crate::skill_delivery::Mode::Inline, ) } /// A prompt as the delivery layer renders it under the INDEX arm. fn rendered_index(skills: &[(&str, &str)]) -> String { let body: String = skills .iter() .map(|(name, when)| { crate::topology_exec::render_pinned_skill( name, &crate::skill_delivery::index_entry( "a procedure", Some(when), &format!("skill:global/{name}"), ), ) }) .collect(); crate::topology_exec::compose_turn_prompt( "Task: do the thing", Some(&body), crate::skill_delivery::Mode::Index, ) } fn read_skill(uri: &str) -> ToolEvidence { ToolEvidence { tool: "ReadMcpResourceTool".into(), path: None, input: json!({ "server": "clawmates_skills", "uri": uri }), response: serde_json::Value::Null, } } fn rendered_files(skills: &[(&str, &str)]) -> String { let body: String = skills .iter() .map(|(name, when)| { crate::topology_exec::render_pinned_skill( name, &crate::skill_delivery::file_entry( "a procedure", Some(when), &crate::skill_delivery::skill_file_path(name), ), ) }) .collect(); format!( "Task: x\n\n# Your skills\n\n{}\n{body}", crate::skill_delivery::FILES_PREAMBLE ) } fn read_file(path: &str) -> ToolEvidence { ToolEvidence { tool: "Read".into(), path: Some(path.into()), input: json!({ "file_path": path }), response: serde_json::Value::Null, } } /// The `files` arm's loop, end to end, for the same reason as the uri /// test below: `skill_file_path` writes the path, `file_entry` puts it in /// the prompt, `skill_from_file_path` reads it back off a `Read`. #[test] fn the_path_the_files_arm_advertises_is_the_one_the_scorer_recovers() { let prompt = rendered_files(&[("workspace-repo-commit-protocol", "before committing")]); assert!( prompt.contains("Read(file_path=\"/mission/skills/workspace-repo-commit-protocol.md\")"), "the entry must name the file to read:\n{prompt}" ); assert_eq!( crate::skill_delivery::mode_in_prompt(&prompt), crate::skill_delivery::Mode::Files ); let tools = vec![read_file("/mission/skills/workspace-repo-commit-protocol.md")]; let ev = Evidence::new("", &tools); assert_eq!(retrieved_skills(&ev), vec!["workspace-repo-commit-protocol"]); let scored = score(&prompt, &ev, &builtin); assert_eq!(scored.len(), 1); assert!( matches!(scored[0].trigger, Verdict::Pass), "a Read of the advertised path IS the Trigger axis: {:?}", scored[0].trigger ); } /// Agents read files constantly. Only a `Read` INSIDE the skills directory /// is a retrieval; anything else scoring as one would make Trigger a count /// of file reads. #[test] fn an_ordinary_read_is_not_a_retrieval() { let tools = vec![ read_file("/mission/repo/research/REPORT.md"), read_file("/mission/skills"), read_file("/mission/skills/nested/x.md"), read_file("/etc/passwd"), ]; assert!(retrieved_skills(&Evidence::new("", &tools)).is_empty()); } /// Under `files`, never opened is a miss when the skill had a checkable /// consequence — the same rule as `index`, and NOT `inline`'s /// "not observable", which would report the arm's own defect as a blind spot. #[test] fn the_files_arm_scores_a_miss_like_the_index_arm() { let prompt = rendered_files(&[("int-xx-marker-protocol", "when writing markers")]); let tools: Vec = vec![]; let ev = Evidence::new("INT-01 something without the required shape", &tools); let scored = score(&prompt, &ev, &builtin); assert_eq!(scored.len(), 1); assert!( !matches!(scored[0].trigger, Verdict::NotObservable(_)), "files is a retrieval arm; a miss must not read as inline: {:?}", scored[0].trigger ); } fn bash(cmd: &str) -> ToolEvidence { ToolEvidence { tool: "Bash".into(), path: None, input: json!({ "command": cmd }), response: serde_json::Value::Null, } } fn spawn(prompt: &str) -> ToolEvidence { ToolEvidence { tool: "Agent".into(), path: None, input: json!({ "prompt": prompt, "subagent_type": "general-purpose" }), response: serde_json::Value::Null, } } /// The shape mission 01a09b42 recorded: the parent read the skill, then /// sent each primary source to a subagent and asked for the date — which /// the task never did and the skill's "undated is a finding" does. #[test] fn triage_passes_on_primary_sources_and_reports_the_date_fingerprint() { let tools = vec![ spawn( "Fetch the following URLs and return their full text content. Return the \ URL, date if visible, and the key content.\n1. https://arxiv.org/abs/2309.15217 \ (RAGAS paper)\n2. https://arxiv.org/abs/2311.09476", ), bash("curl -s \"https://arxiv.org/abs/2309.01431\" | head -200"), ]; let ev = Evidence::new("", &tools); match triage_compliance(&ev) { Verdict::PassWith(why) => { assert!(why.contains("3 primary source(s)"), "{why}"); assert!(why.contains("asked for the page's date"), "{why}"); } other => panic!("expected a pass with evidence, got {other:?}"), } } /// The shape the `index` runs recorded — inline curls, no delegation. The /// agent still read primary sources, so it still complied; the date /// fingerprint is extra evidence, never a requirement. #[test] fn triage_passes_on_inline_curls_without_the_date_clause() { let tools = vec![ bash("curl -sL https://arxiv.org/abs/2204.04745"), bash("ls -la research/"), ]; match triage_compliance(&Evidence::new("", &tools)) { Verdict::PassWith(why) => assert!(!why.contains("date"), "{why}"), other => panic!("{other:?}"), } } /// The one violation the check can see: reading a restatement of a /// restatement instead of the source it restates. #[test] fn triage_fails_on_an_aggregator() { let tools = vec![ bash("curl -s https://arxiv.org/abs/2309.15217"), spawn("Fetch https://medium.com/@someone/rag-eval-explained-2024 and summarise"), ]; match triage_compliance(&Evidence::new("", &tools)) { Verdict::Fail(why) => assert!(why.contains("medium.com"), "{why}"), other => panic!("{other:?}"), } } /// One-sided, both ways. Nothing fetched is nothing to triage; something /// fetched that the lists cannot rank is unranked, not a violation. #[test] fn triage_is_silent_where_it_cannot_see() { let none: Vec = vec![]; assert!(matches!( triage_compliance(&Evidence::new("", &none)), Verdict::NotObservable(_) )); let no_fetch = vec![bash("cargo test"), bash("git status")]; assert!(matches!( triage_compliance(&Evidence::new("", &no_fetch)), Verdict::NotApplicable )); let unranked = vec![bash("curl -s https://docs.example-vendor.io/eval/guide")]; assert!(matches!( triage_compliance(&Evidence::new("", &unranked)), Verdict::NotObservable(_) )); } /// A URL inside JSON is followed by a quote, and one at the end of a /// sentence by a full stop. Neither is part of the URL. #[test] fn fetched_urls_stop_at_the_right_character() { let tools = vec![spawn( "Fetch \"https://arxiv.org/abs/1\" then https://doi.org/10.1/x. Done.", )]; assert_eq!( fetched_urls(&Evidence::new("", &tools)), vec!["https://arxiv.org/abs/1", "https://doi.org/10.1/x"] ); } /// `PassWith` must be indistinguishable from `Pass` to a reader keyed on /// the tag, or every consumer of the report grows a fourth branch. #[test] fn a_pass_with_evidence_serialises_under_the_pass_tag() { let v = serde_json::to_value(Verdict::PassWith("saw it".into())).unwrap(); assert_eq!(v["verdict"], "pass"); assert_eq!(v["why"], "saw it"); assert_eq!(Verdict::PassWith("x".into()).label(), Verdict::Pass.label()); } /// The whole loop, end to end: the index writes a uri, the agent reads that /// exact uri back, and the scorer recovers the skill's name from it. /// /// Three components have to agree on one string — `mcp_skills::skill_uri` /// writes it, `skill_delivery::index_entry` puts it in the prompt, and /// `parse_uri` reads it. Asserting them separately would let any pair drift /// while each one's own test stayed green. #[test] fn the_uri_the_index_advertises_is_the_one_the_scorer_recovers() { let prompt = rendered_index(&[("workspace-repo-commit-protocol", "before committing")]); assert!( prompt.contains("uri=\"skill:global/workspace-repo-commit-protocol\""), "the entry must name the uri to fetch:\n{prompt}" ); let tools = vec![read_skill("skill:global/workspace-repo-commit-protocol")]; let ev = Evidence::new("", &tools); assert_eq!( retrieved_skills(&ev), vec!["workspace-repo-commit-protocol"], "the uri the prompt advertised must parse back to the skill's name" ); let scored = score(&prompt, &ev, &builtin); assert_eq!(scored.len(), 1); assert!( matches!(scored[0].trigger, Verdict::Pass), "reaching for an indexed skill IS the Trigger axis: {:?}", scored[0].trigger ); } /// The index arm must not report a miss as a blind spot. /// /// Under `Inline` "never retrieved" is `NotObservable`, and that is honest /// there. Reusing it here would say "this skill was inlined into the /// prompt" about a skill whose body was never sent — the exact shape of a /// check reporting a system defect where an agent behaviour belongs. #[test] fn an_indexed_skill_that_was_never_read_is_a_miss_not_a_blind_spot() { let prompt = rendered_index(&[("workspace-repo-commit-protocol", "before committing")]); // It wrote outside the mission checkout, so the boundary check applies // — this skill had something to say about this phase and went unread. let tools = acted(&[("Write", Some("/tmp/scratch.rs"), json!({}))]); let scored = score(&prompt, &Evidence::new("", &tools), &builtin); assert!( matches!(scored[0].trigger, Verdict::Fail(_)), "offered by name and when_to_use, never fetched: {:?}", scored[0].trigger ); } /// ...but only when the skill had a consequence to check. /// /// A skill with nothing machine-checkable in this phase is one an agent is /// right to pass over, and scoring that as a Trigger failure would punish /// correct triage — which is the behaviour progressive disclosure is /// supposed to reward. #[test] fn passing_over_a_skill_with_nothing_to_check_is_not_a_trigger_failure() { let prompt = rendered_index(&[("some-unchecked-skill", "when writing prose")]); let scored = score(&prompt, &narrative("wrote the report"), &builtin); assert!( matches!(scored[0].trigger, Verdict::NotApplicable), "{:?}", scored[0].trigger ); } /// The control arm has to be unchanged, or the A/B measures this edit too. /// The case this whole flag exists for. /// /// `workspace-repo-commit-protocol` applies to every agent that writes, /// which is exactly why no agent reads it as *theirs* — it scored /// Trigger=FAIL beside a passing boundary check on the first A/B pair. /// Marked `always_inject`, its body is in the prompt under the index arm, /// and a skill the agent was handed cannot be a reaching-for failure. #[test] fn a_skill_delivered_in_full_under_the_index_arm_is_not_a_trigger_miss() { let prompt = format!( "{}\n{}{}", crate::skill_delivery::INDEX_PREAMBLE, crate::topology_exec::render_pinned_skill( "workspace-repo-commit-protocol", "Commit only inside /workspace/repo. Never write outside it.", ), rendered_index(&[("arxiv-daily", "when sweeping arxiv")]), ); let tools = []; let ev = Evidence { text: "", tools: &tools }; let got = score(&prompt, &ev, &|_| "builtin".to_string()); let it = got .iter() .find(|u| u.skill == "workspace-repo-commit-protocol") .expect("the always-injected skill must still be scored"); assert!( matches!(it.trigger, Verdict::NotObservable(_)), "handed over, not offered — there is no retrieval to miss: {:?}", it.trigger ); } /// The flag must not leak: a skill still delivered as an index entry keeps /// being scored on whether it was fetched. #[test] fn an_indexed_skill_in_the_same_prompt_is_still_judged_on_retrieval() { let prompt = format!( "{}\n{}{}", crate::skill_delivery::INDEX_PREAMBLE, crate::topology_exec::render_pinned_skill( "workspace-repo-commit-protocol", "Commit only inside /workspace/repo.", ), rendered_index(&[("arxiv-daily", "when sweeping arxiv")]), ); for u in score(&prompt, &Evidence { text: "", tools: &[] }, &|_| "builtin".into()) { if u.skill != "workspace-repo-commit-protocol" { assert!( !matches!(u.trigger, Verdict::NotObservable(_)), "{} was offered by uri, so retrieval is observable for it", u.skill ); } } } #[test] fn the_inline_arm_still_reports_trigger_as_unobservable() { let prompt = rendered(&[("workspace-repo-commit-protocol", "body")]); let tools = acted(&[("Write", Some("/tmp/scratch.rs"), json!({}))]); let scored = score(&prompt, &Evidence::new("", &tools), &builtin); assert!( matches!(scored[0].trigger, Verdict::NotObservable(_)), "{:?}", scored[0].trigger ); } /// The prompt is the record of what was delivered, so parsing it must match /// exactly what the delivery layer writes. #[test] fn the_delivered_skills_are_read_back_out_of_the_prompt() { let prompt = rendered(&[ ("arxiv-daily", "Do not re-search."), ("int-xx-marker-protocol", "Emit markers."), ]); assert_eq!( skills_in_prompt(&prompt), vec!["arxiv-daily", "int-xx-marker-protocol"], "the scorer reads what the delivery layer wrote — if these drift, \ every score is attributed to the wrong skill" ); } /// Skill bodies are markdown and contain their own headings. /// /// The first delimiter was `## `, so every section of every body /// counted as a separate skill. Caught on a live mission, which scored /// "Sizing heuristic" and "The output shape" — both of them subheadings /// inside `decompose-int-items` — as skills with no catalogue row. #[test] fn headings_inside_a_skill_body_are_not_skills() { let body = "# Decomposing into INT-XX items\n\n\ ## Sizing heuristic\nOne unit of work.\n\n\ ## The output shape\nTASK: INT-NN — title\n"; let prompt = rendered(&[("decompose-int-items", body)]); assert_eq!( skills_in_prompt(&prompt), vec!["decompose-int-items"], "a body's own headings must not be counted as skills — every one \ would be scored against a catalogue row that does not exist" ); } /// A body that quotes the marker text must not fabricate a skill either. #[test] fn a_body_mentioning_the_marker_does_not_create_a_skill() { let body = format!( "Prompts introduce a skill with a line beginning `{}`.\n", crate::topology_exec::SKILL_MARKER.trim() ); let prompt = rendered(&[("prompt-anatomy", &body)]); assert_eq!( skills_in_prompt(&prompt), vec!["prompt-anatomy"], "the marker is matched at line start; an inline mention is prose" ); } #[test] fn a_prompt_with_no_skills_yields_no_scores() { assert!(skills_in_prompt("Task: do the thing").is_empty()); assert!(score("Task: do the thing", &narrative("did it"), &builtin).is_empty()); } #[test] fn trigger_is_reported_as_unobservable_not_as_a_failure() { let prompt = rendered(&[("arxiv-daily", "x")]); let scored = score(&prompt, &narrative("I read the manifest."), &builtin); assert!( matches!(scored[0].trigger, Verdict::NotObservable(_)), "scoring Trigger zero would report a delivery-model property as an \ agent failure — the exact confusion this module exists to avoid" ); } #[test] fn markers_the_real_parser_accepts_are_compliant() { let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]); let good = "I implemented the parser.\nCOMPLETED: INT-07 — wire the loop"; let scored = score(&prompt, &narrative(good), &builtin); assert_eq!(scored[0].compliance, Verdict::Pass); } /// The failure the skill exists to prevent: marker-shaped lines that the /// parser rejects, so the mission silently never advances. #[test] fn marker_shaped_lines_the_parser_rejects_are_a_failure() { let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]); // Bold, which the skill explicitly forbids, and the parser will not take. let bad = "**COMPLETED: INT-07**"; let scored = score(&prompt, &narrative(bad), &builtin); assert!( matches!(scored[0].compliance, Verdict::Fail(_)), "a marker the parser rejects must score as a failure — that is the \ whole consequence the skill is written to avoid. Got {:?}", scored[0].compliance ); // A turn that never tried is not a violation. let unrelated = score(&prompt, &narrative("I read three files and wrote a summary."), &builtin); assert_eq!(unrelated[0].compliance, Verdict::NotApplicable); } /// Found on a live mission: `PLAN_COMPLETE: INT-01..02`. /// /// Two defects in one line, both since fixed in the parser rather than /// worked around here: /// /// 1. `PLAN_COMPLETE` was documented in `int-xx-marker-protocol` and never /// implemented, so an agent following the skill exactly was ignored. It /// is implemented now. /// 2. The range form parsed into the id `INT-01..02`, matching no real /// item — a task card for something that did not exist. Ids are now /// strictly `INT-`, so the marker is REJECTED instead, which is /// visible where a plausible-looking row was not. #[test] fn plan_complete_is_now_a_real_marker() { let parsed = crate::task_card_parser::parse("PLAN_COMPLETE: INT-01"); assert_eq!(parsed.len(), 1, "the skill documents it; the parser must accept it"); assert_eq!(parsed[0].int_id, "INT-01"); let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]); assert_eq!( score(&prompt, &narrative("PLAN_COMPLETE: INT-01"), &builtin)[0].compliance, Verdict::Pass ); } /// The exact line a live planner emitted. #[test] fn a_range_marker_is_rejected_rather_than_creating_a_phantom_item() { assert!( crate::task_card_parser::parse("PLAN_COMPLETE: INT-01..02").is_empty(), "a range must not parse — it produced a task card for an item that \ does not exist while INT-01 and INT-02 stayed open" ); assert!(crate::task_card_parser::parse("COMPLETED: INT-01..02").is_empty()); // And the agent is told, because the marker it emitted did nothing. let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]); match &score(&prompt, &narrative("PLAN_COMPLETE: INT-01..02"), &builtin)[0].compliance { Verdict::Fail(why) => assert!(why.contains("does not accept")), other => panic!("an ignored marker must not read as success; got {other:?}"), } } #[test] fn two_int_ids_on_one_marker_line_cross_the_boundary() { let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]); let scored = score(&prompt, &narrative("COMPLETED: INT-05, INT-06"), &builtin); match &scored[0].boundary { Verdict::Fail(why) => assert!(why.contains("silently dropped")), other => panic!("the second id is silently dropped by the parser; got {other:?}"), } } #[test] fn querying_arxiv_crosses_the_boundary_but_naming_it_does_not() { let prompt = rendered(&[("arxiv-daily", "x")]); let violating = score( &prompt, &narrative("curl 'http://export.arxiv.org/api/query?search_query=all:agents'"), &builtin, ); assert!(matches!(violating[0].boundary, Verdict::Fail(_))); // The word appears legitimately in every research turn. Scoring on it // would make the metric fire constantly and mean nothing. let fine = score( &prompt, &narrative("I read the arXiv notes in the manifest and summarised three of them."), &builtin, ); assert_eq!(fine[0].boundary, Verdict::Pass); } /// Trigger, from the exact arguments a live mission recorded. /// /// Run 8, 2026-08-21: three agents each reached through the door and read a /// skill body. Copied from `mission_events`, not invented — the shape of /// this payload is the whole basis of the axis. #[test] fn a_retrieved_skill_scores_trigger_as_a_pass() { let tools = acted(&[ ("ListMcpResourcesTool", None, json!({})), ( "ReadMcpResourceTool", None, json!({"uri": "skill:global/workspace-repo-commit-protocol", "server": "clawmates_skills"}), ), ]); let ev = Evidence::new("", &tools); assert_eq!( retrieved_skills(&ev), vec!["workspace-repo-commit-protocol"] ); // Delivered by RETRIEVAL only — nothing in the prompt at all. Under // progressive disclosure this is every skill, so a scorer that reads // only the prompt would report nothing for the delivery model this // axis exists to measure. let scored = score("Task: do the thing", &ev, &builtin); assert_eq!(scored.len(), 1); assert_eq!(scored[0].skill, "workspace-repo-commit-protocol"); assert_eq!( scored[0].trigger, Verdict::Pass, "the agent reached for it; that is the paper's Trigger" ); } /// An inlined skill is still `NotObservable`, and the reason now names the /// fix rather than the transport. #[test] fn an_inlined_skill_is_still_unobservable_for_trigger() { let prompt = rendered(&[("arxiv-daily", "x")]); let scored = score(&prompt, &narrative("did it"), &builtin); match &scored[0].trigger { Verdict::NotObservable(why) => assert!( why.contains("handed it"), "the reason must say WHY it cannot be seen — being handed a \ skill is not failing to reach for one: {why}" ), other => panic!("got {other:?}"), } } /// A workspace-scoped URI resolves to the same name as a global one. #[test] fn both_uri_shapes_yield_the_skill_name() { let ws = uuid::Uuid::now_v7(); let tools = acted(&[ ( "ReadMcpResourceTool", None, json!({"uri": format!("skill:workspace/{ws}/team-local-thing")}), ), // Not a skill URI, and not a panic. ("ReadMcpResourceTool", None, json!({"uri": "file:///etc/passwd"})), // A LIST call names no skill — only reads are retrievals. ("ListMcpResourcesTool", None, json!({})), ]); assert_eq!( retrieved_skills(&Evidence::new("", &tools)), vec!["team-local-thing"], "listing the catalogue is browsing; reading a body is the reach" ); } /// Agent-authored skills must stay visible as such in the report. #[test] fn writing_outside_the_mission_checkout_crosses_the_boundary() { let prompt = rendered(&[("workspace-repo-commit-protocol", "Work in /mission/repo.")]); let wrong = score( &prompt, &narrative("cd /workspace/repo && git add -A && git commit -m 'INT-01 done'"), &builtin, ); match &wrong[0].boundary { Verdict::Fail(why) => assert!( why.contains("never collected"), "the failure must name the consequence — a phase that delivers \ nothing — not just the wrong path: {why}" ), other => panic!("the wrong checkout must be caught; got {other:?}"), } let right = score( &prompt, &narrative("cd /mission/repo && git add -A && git commit -m 'INT-01 done'"), &builtin, ); assert_eq!(right[0].boundary, Verdict::Pass); } /// The upgrade, stated as a test: the agent never said where it wrote. /// /// Before the tool tap this scored a clean `Pass`, because the only /// evidence was the narrative and the narrative did not mention a path. The /// phase had still delivered nothing. #[test] fn a_write_outside_the_checkout_is_caught_even_when_the_narrative_is_silent() { let prompt = rendered(&[("workspace-repo-commit-protocol", "Work in /mission/repo.")]); let tools = acted(&[ ("Write", Some("/app/src/main.rs"), json!({"file_path": "/app/src/main.rs"})), ]); let scored = score(&prompt, &Evidence::new("I implemented the feature.", &tools), &builtin); match &scored[0].boundary { Verdict::Fail(why) => assert!(why.contains("/app/src/main.rs"), "{why}"), other => panic!("the write is right there in the record; got {other:?}"), } } /// Scratch is not a violation — the skill blesses it by name. #[test] fn scratch_and_the_checkout_both_stay_inside_the_boundary() { let prompt = rendered(&[("workspace-repo-commit-protocol", "Work in /mission/repo.")]); let tools = acted(&[ ("Write", Some("/tmp/scratch.txt"), json!({"file_path": "/tmp/scratch.txt"})), ("Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"})), ]); let scored = score(&prompt, &Evidence::new("done", &tools), &builtin); assert_eq!(scored[0].boundary, Verdict::Pass); } /// Nothing observed is not the same as nothing done. /// /// This returned `Pass` before — a mission with no evidence scored /// identically to one that was checked and found clean, which is the exact /// collapse `Verdict::NotObservable` exists to prevent. #[test] fn silence_is_not_scored_as_compliance() { let prompt = rendered(&[("workspace-repo-commit-protocol", "Work in /mission/repo.")]); let scored = score(&prompt, &narrative("I thought about the problem."), &builtin); assert!( matches!(scored[0].boundary, Verdict::NotObservable(_)), "got {:?}", scored[0].boundary ); } #[test] fn a_force_push_crosses_the_boundary_and_a_quoted_flag_does_not() { let prompt = rendered(&[("workspace-repo-commit-protocol", "Work in /mission/repo.")]); let forced = acted(&[ ("Write", Some("/mission/repo/a.rs"), json!({"file_path": "/mission/repo/a.rs"})), ran("cd /mission/repo && git push --force origin mission"), ]); match &score(&prompt, &Evidence::new("", &forced), &builtin)[0].boundary { Verdict::Fail(why) => assert!(why.contains("force-push"), "{why}"), other => panic!("rewriting pushed history is the one unrecoverable act; got {other:?}"), } // The flag inside a commit message rewrites nothing. let innocent = acted(&[ ("Write", Some("/mission/repo/a.rs"), json!({"file_path": "/mission/repo/a.rs"})), ran(r#"git commit -m "document the -f flag" && git push origin mission"#), ]); assert_eq!( score(&prompt, &Evidence::new("", &innocent), &builtin)[0].boundary, Verdict::Pass ); } /// Writing code and never running a test cannot be red-green-refactor under /// any reading of the loop. #[test] fn code_written_with_no_test_run_fails_the_loop() { let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]); let tools = acted(&[ ("Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"})), ran("cargo build"), ]); match &score(&prompt, &Evidence::new("Implemented it.", &tools), &builtin)[0].compliance { Verdict::Fail(why) => assert!(why.contains("never ran a test"), "{why}"), other => panic!("got {other:?}"), } } #[test] fn a_test_run_before_the_first_write_is_red_first() { let prompt = rendered(&[("cargo-test-driven-development", "Red, green, refactor.")]); let tools = acted(&[ ran("cargo nextest run parses_the_range_form"), ("Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"})), ran("cargo nextest run parses_the_range_form"), ]); assert_eq!( score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance, Verdict::Pass ); } /// The ordering that cannot be read either way, reported as such. /// /// A Rust unit test lives in `#[cfg(test)] mod tests` inside the file under /// test, so writing the failing test IS a write to the implementation file. /// Scoring this as a failure would fail the agents who followed the skill /// most precisely. #[test] fn a_test_run_after_the_first_write_is_undecidable_not_a_failure() { let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]); let tools = acted(&[ ("Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"})), ran("cargo test -p cm-api"), ]); assert!( matches!( score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance, Verdict::NotObservable(_) ), "the Rust unit test lives in the file under test — this ordering \ cannot separate red-first from tests-added-after" ); } /// Red then green, read from what the runs themselves reported. /// /// This is the case ordering cannot decide and the repository cannot /// either: one `Edit` adds the implementation and its `#[cfg(test)] mod /// tests` together, and `tdd-red-green-refactor` says in so many words to /// "commit the RED-to-GREEN pair as one commit". The run outputs are the /// only witness left. #[test] fn a_failing_run_followed_by_a_passing_one_is_red_then_green() { let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]); let mut tools = acted(&[( "Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"}), )]); tools.push(test_run( "cargo test", "running 1 test\ntest slugs ... FAILED\n\ntest result: FAILED. 0 passed; 1 failed", )); tools.push(test_run( "cargo test", "running 1 test\ntest slugs ... ok\n\ntest result: ok. 1 passed; 0 failed", )); assert_eq!( score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance, Verdict::Pass ); } /// Every run green is NOT evidence of red-first. #[test] fn runs_that_only_ever_passed_cannot_prove_the_loop() { let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]); let mut tools = acted(&[( "Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"}), )]); tools.push(test_run("cargo test", "test result: ok. 3 passed; 0 failed")); match &score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance { Verdict::NotObservable(why) => assert!( why.contains("never failed"), "a test that never failed is equally what a correct \ implementation written first looks like: {why}" ), other => panic!("got {other:?}"), } } /// Ending red is a failure of the loop, and a different one from never /// running a test at all. #[test] fn a_phase_that_never_reached_green_fails() { let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]); let mut tools = acted(&[( "Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"}), )]); tools.push(test_run("cargo test", "test result: FAILED. 0 passed; 2 failed")); match &score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance { Verdict::Fail(why) => assert!(why.contains("never got there"), "{why}"), other => panic!("got {other:?}"), } } /// A mission recorded before responses were captured must not read as /// either verdict. #[test] fn runs_with_no_recorded_output_stay_unobservable() { let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]); let tools = acted(&[ ("Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"})), ("Bash", None, json!({"command": "cargo test"})), ]); assert!(matches!( score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance, Verdict::NotObservable(_) )); } /// The verdict line is at the END of a test run, which is why the response /// bound keeps the tail. #[test] fn a_truncated_response_still_carries_the_verdict() { let long = "compiling…\n".repeat(400) + "test result: FAILED. 0 passed; 1 failed"; let bounded = crate::vm_tool_tap::bounded_response( "Bash", &json!({ "stdout": long, "stderr": "" }), ); let kept = bounded["stdout"].as_str().expect("stdout kept"); assert!( kept.contains("test result: FAILED"), "a head-biased truncation would keep the noise and drop the verdict, \ which is the only reason the output is stored" ); assert!(kept.starts_with("[truncated]"), "{kept}"); } /// A `Read`'s response is the file it just read — already knowable, and /// large. #[test] fn only_commands_carry_a_response() { assert_eq!( crate::vm_tool_tap::bounded_response("Read", &json!({"stdout": "x"})), serde_json::Value::Null ); } /// A phase that wrote nothing had no implementation to test-drive. /// The verdict the first live run got wrong. /// /// A research phase wrote fifteen markdown notes and ran no test, and the /// check called it a TDD failure. There was no code to test-drive. The /// real finding is that a research role is pinned TDD skills at all, and /// scoring the agent for it would have buried that. #[test] fn writing_prose_is_not_a_tdd_failure() { let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]); let tools = acted(&[ ("Write", Some("/mission/repo/research/notes.md"), json!({"file_path": "/mission/repo/research/notes.md"})), ("Edit", Some("/mission/repo/research/notes.md"), json!({"file_path": "/mission/repo/research/notes.md"})), ("Write", Some("/mission/repo/research/check.sh"), json!({"file_path": "/mission/repo/research/check.sh"})), ]); assert_eq!( score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance, Verdict::NotApplicable, "markdown and a helper script are not the behaviour-adding code \ this loop is about" ); } #[test] fn a_phase_that_wrote_nothing_is_not_a_tdd_failure() { let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]); let tools = acted(&[ran("ls /mission/repo"), ("Read", Some("/mission/repo/a.rs"), json!({}))]); assert_eq!( score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance, Verdict::NotApplicable ); } #[test] fn the_never_merge_subjects_are_caught_and_a_real_one_is_not() { let prompt = rendered(&[("small-focused-commits", "One idea per commit.")]); let bad = acted(&[ran(r#"git commit -m "wip""#)]); match &score(&prompt, &Evidence::new("", &bad), &builtin)[0].boundary { Verdict::Fail(why) => assert!(why.contains("never-merge"), "{why}"), other => panic!("got {other:?}"), } // The objection is to `fix` as the whole message, not to the word. let good = acted(&[ran(r#"git commit -m "INT-04 fix the range form the parser rejects""#)]); assert_eq!( score(&prompt, &Evidence::new("", &good), &builtin)[0].boundary, Verdict::NotApplicable ); } /// The subject comes out of the shell command, in the shapes agents write. #[test] fn the_commit_subject_is_read_out_of_the_real_command_shapes() { assert_eq!(dash_m_value(r#"git commit -m "one line""#).as_deref(), Some("one line")); assert_eq!(dash_m_value("git commit -am 'clustered flag'").as_deref(), Some("clustered flag")); // The template in `workspace-repo-commit-protocol` spans several lines; // only the first is the subject. let multi = "cd /mission/repo && git commit -m \"INT-01 wire the loop\n\nWHY: it stalled.\n\nRefs: INT-01\n\""; assert_eq!( commit_subjects(&Evidence::new("", &acted(&[ran(multi)]))), vec!["INT-01 wire the loop"] ); // `-F` gives up nothing, and that must read as no evidence. assert!(dash_m_value("git commit -F /tmp/msg").is_none()); } /// The exact commit command the first live coding run ran. /// /// Claude Code writes a multi-line message as a heredoc inside a command /// substitution, so the first line of the `-m` value is the heredoc opener /// and the subject is the line after it. Read from /// `mission_events`, not invented. #[test] fn a_heredoc_commit_message_gives_up_its_real_subject() { let live = "git add src/lib.rs && git commit -m \"$(cat <<'EOF'\n\ INT-01 Add slugify function to src/lib.rs\n\n\ The crate exposed only add() with no string-normalization utility.\n\n\ Refs: INT-01\n\ EOF\n\ )\""; assert_eq!( commit_subjects(&Evidence::new("", &acted(&[ran(live)]))), vec!["INT-01 Add slugify function to src/lib.rs"], "the heredoc OPENER is not the subject — reading it as one puts \ `$(cat <<'EOF'` in the record, and every commit check then scores \ a string the agent never wrote" ); } /// The skill tells the agent to `curl` the abstract page in the paragraph /// under the one that forbids the query API. A check that cannot tell them /// apart fails agents for obeying the skill. #[test] fn curling_an_abstract_is_allowed_where_querying_the_api_is_not() { let prompt = rendered(&[("arxiv-daily", "The harvest already ran.")]); let allowed = acted(&[ran("curl -s https://arxiv.org/abs/2401.12345")]); assert_eq!( score(&prompt, &Evidence::new("", &allowed), &builtin)[0].boundary, Verdict::Pass ); let forbidden = acted(&[ran("curl -s 'http://export.arxiv.org/api/query?search_query=all:agents'")]); match &score(&prompt, &Evidence::new("", &forbidden), &builtin)[0].boundary { Verdict::Fail(why) => assert!(why.contains("seen-set"), "{why}"), other => panic!("got {other:?}"), } } #[test] fn the_source_of_a_skill_is_carried_into_its_score() { let prompt = rendered(&[("self-made", "x")]); let scored = score(&prompt, &narrative("done"), &|_| "promoted_from_brain".to_string()); assert_eq!( scored[0].source_kind, "promoted_from_brain", "an agent that writes its own skill can raise its own score against \ it; that has to be legible in the report rather than averaged in" ); } }