feat(skill-use): score Compliance and Boundary from actions, not narrative
The scorer read the concatenated `reasoning` text — the agent's own account of its turn, written by the thing being measured and silent about anything it did not think worth mentioning. `Evidence` now carries the recorded tool calls alongside that text and every check prefers them. What that changes, concretely: - `workspace-repo-commit-protocol` Boundary was a substring search for `/workspace/repo` in the narrative. An agent that wrote to the wrong root without narrating it scored a clean pass. It now reads the `Write` and `Edit` paths, and gained the skill's other hard prohibition — force-push — which leaves no trace anywhere else once it succeeds. - `arxiv-daily` Boundary reads the `curl` that ran rather than a URL in prose, which may be the agent explaining that it did NOT fetch it. - `tdd-red-green-refactor` and `cargo-test-driven-development` gain their first Compliance check: files written with no test command anywhere cannot have been red-green under any reading of the loop. - `small-focused-commits` gains a Boundary check on the exact subjects the skill names as never-merge, read out of `git commit -m`. Two verdicts changed for honesty rather than coverage. Silence used to score `Pass`: a mission with no evidence scored identically to one checked and found clean. It is now `NotObservable`. And a test that ran AFTER the first write is `NotObservable`, not a failure — a Rust unit test lives in the file under test, so that ordering is what following the skill most precisely looks like from here. Every tool-backed check is one-sided: it reports a violation it can see and never infers compliance from silence, because the recorded stream is capped per phase. The negative controls earned their keep — they caught `-f` inside a commit message scoring as a force-push, and `git commit -am` yielding no subject at all. Trigger stays `NotObservable`, and half its stated reason is now wrong. "`claude_cli` cannot surface a tool call" is false; we simply still inline. The blocker moved from the transport to the delivery model, and the module says so. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9
This commit is contained in:
Omar Sobh
co-authored by
Claude Opus 5
Claude Opus 5
parent
8cb38d1320
commit
1a6fdfc0e6
+598
-41
@@ -11,10 +11,8 @@
|
||||
//! is the Trigger event, and it is observable because it is a tool call.
|
||||
//!
|
||||
//! We do not deliver skills that way on the mission path. `pinned_skills_text`
|
||||
//! inlines full bodies into the prompt, because mission claws run on
|
||||
//! `claude_cli`, which cannot surface a tool call at all — there is nothing to
|
||||
//! retrieve *with*. So the agent never "reaches for" a skill; it is simply
|
||||
//! holding one.
|
||||
//! inlines full bodies into the prompt, so the agent never "reaches for" a
|
||||
//! skill; it is simply holding one.
|
||||
//!
|
||||
//! Trigger is therefore **not observable on the mission path**, and this module
|
||||
//! reports it as `NotObservable` with the reason attached rather than scoring
|
||||
@@ -22,8 +20,33 @@
|
||||
//! actually means "the question does not apply to how we deliver them" — the
|
||||
//! precise confusion that made 55 empty skill bindings invisible for months.
|
||||
//!
|
||||
//! Compliance and Boundary are observable, because they are properties of the
|
||||
//! output rather than of the retrieval.
|
||||
//! ### The reason changed, and only half of it went away
|
||||
//!
|
||||
//! Until 2026-08-21 the stated reason was that `claude_cli` "cannot surface a
|
||||
//! tool call at all — there is nothing to retrieve *with*". That half is now
|
||||
//! false: [`crate::container_tool_hooks`] installs `PostToolUse`, mission tool
|
||||
//! calls land in `mission_events`, and a retrieval would be as visible as any
|
||||
//! other call.
|
||||
//!
|
||||
//! The other half still holds, and it is the one that decides the verdict: we
|
||||
//! still **inline**. Trigger is now *instrumentable* and still not
|
||||
//! *observable*, and the blocker has moved from the transport to the delivery
|
||||
//! model. Making it real is one change — serve skills through the door
|
||||
//! (`docs/TOOL-CALL-ARCHITECTURE.md` §3) so retrieval becomes a tool call —
|
||||
//! and nothing else in this module has to change to score it.
|
||||
//!
|
||||
//! ## Compliance and Boundary are scored from ACTIONS where actions exist
|
||||
//!
|
||||
//! They were scored from the concatenated narrative, which is the agent's own
|
||||
//! account of its turn — written by the thing being measured, and silent about
|
||||
//! anything it did not think worth mentioning. [`Evidence`] carries the
|
||||
//! recorded tool calls alongside that text, and every check prefers them:
|
||||
//! `Bash` arguments say what ran, `Write` paths say where it landed.
|
||||
//!
|
||||
//! Every tool-backed check is **one-sided**. It reports a violation it can see
|
||||
//! and never infers compliance from silence — the recorded stream is capped
|
||||
//! per phase ([`crate::mission_events::PER_PHASE_CAP`]), so an absent call is
|
||||
//! not proof of an absent action.
|
||||
|
||||
use serde::Serialize;
|
||||
|
||||
@@ -82,12 +105,55 @@ pub fn skills_in_prompt(prompt: &str) -> Vec<String> {
|
||||
crate::topology_exec::skill_names_in(prompt)
|
||||
}
|
||||
|
||||
/// What a mission left behind, for the checks to read.
|
||||
///
|
||||
/// Two sources, deliberately kept apart. `text` is what the agent *said* it
|
||||
/// did; `tools` is what it did. Where they disagree the tools win, and a check
|
||||
/// that can only consult `text` says so in its verdict rather than presenting a
|
||||
/// narrative reading as an observation.
|
||||
pub struct Evidence<'a> {
|
||||
/// Concatenated `reasoning` events for the mission.
|
||||
pub text: &'a str,
|
||||
/// Every `tool.call` recorded for the mission, in order.
|
||||
pub tools: &'a [crate::mission_events::ToolEvidence],
|
||||
}
|
||||
|
||||
/// Tools that modify a file. `target` on the recorded event is the tool name.
|
||||
const WRITE_TOOLS: [&str; 4] = ["Write", "Edit", "MultiEdit", "NotebookEdit"];
|
||||
|
||||
impl<'a> Evidence<'a> {
|
||||
pub fn new(text: &'a str, tools: &'a [crate::mission_events::ToolEvidence]) -> Self {
|
||||
Evidence { text, tools }
|
||||
}
|
||||
|
||||
/// Narrative only — a mission whose events predate the tool tap.
|
||||
pub fn from_text(text: &'a str) -> Self {
|
||||
Evidence { text, tools: &[] }
|
||||
}
|
||||
|
||||
/// Every shell command the agent ran, in order.
|
||||
fn commands(&self) -> impl Iterator<Item = &str> {
|
||||
self.tools.iter().filter_map(|t| t.command())
|
||||
}
|
||||
|
||||
/// Every file-modifying call that named a path, in order.
|
||||
fn writes(&self) -> impl Iterator<Item = &crate::mission_events::ToolEvidence> {
|
||||
self.tools
|
||||
.iter()
|
||||
.filter(|t| WRITE_TOOLS.contains(&t.tool.as_str()) && t.path.is_some())
|
||||
}
|
||||
}
|
||||
|
||||
/// Score every skill a phase's prompt delivered, against what the agent produced.
|
||||
pub fn score(prompt: &str, output: &str, source_kinds: &dyn Fn(&str) -> String) -> Vec<SkillUse> {
|
||||
pub fn score(
|
||||
prompt: &str,
|
||||
evidence: &Evidence<'_>,
|
||||
source_kinds: &dyn Fn(&str) -> String,
|
||||
) -> Vec<SkillUse> {
|
||||
skills_in_prompt(prompt)
|
||||
.into_iter()
|
||||
.map(|skill| {
|
||||
let (compliance, boundary) = check(&skill, output);
|
||||
let (compliance, boundary) = check(&skill, evidence);
|
||||
SkillUse {
|
||||
source_kind: source_kinds(&skill),
|
||||
trigger: Verdict::NotObservable(
|
||||
@@ -109,11 +175,19 @@ pub fn score(prompt: &str, output: &str, source_kinds: &dyn Fn(&str) -> String)
|
||||
/// Everything else returns `NotApplicable` rather than a guess: a heuristic
|
||||
/// that scores prose by keyword overlap produces a number that looks like a
|
||||
/// measurement and is not one.
|
||||
fn check(skill: &str, output: &str) -> (Verdict, Verdict) {
|
||||
fn check(skill: &str, ev: &Evidence<'_>) -> (Verdict, Verdict) {
|
||||
match skill {
|
||||
"int-xx-marker-protocol" => (marker_compliance(output), marker_boundary(output)),
|
||||
"arxiv-daily" => (Verdict::NotApplicable, arxiv_boundary(output)),
|
||||
"workspace-repo-commit-protocol" => (Verdict::NotApplicable, workspace_boundary(output)),
|
||||
"int-xx-marker-protocol" => (marker_compliance(ev.text), marker_boundary(ev.text)),
|
||||
"arxiv-daily" => (Verdict::NotApplicable, arxiv_boundary(ev)),
|
||||
"workspace-repo-commit-protocol" => (Verdict::NotApplicable, workspace_boundary(ev)),
|
||||
"small-focused-commits" => (Verdict::NotApplicable, commit_subject_boundary(ev)),
|
||||
// One check for both: `cargo-test-driven-development` is the Rust
|
||||
// flavour of the same loop, and its own text says so. Scoring them by
|
||||
// separate rules would mean two rules for one procedure, which is how
|
||||
// the microVM and container taps drifted apart.
|
||||
"tdd-red-green-refactor" | "cargo-test-driven-development" => {
|
||||
(red_before_green(ev), Verdict::NotApplicable)
|
||||
}
|
||||
_ => (Verdict::NotApplicable, Verdict::NotApplicable),
|
||||
}
|
||||
}
|
||||
@@ -192,20 +266,269 @@ fn looks_like_marker_attempt(line: &str) -> bool {
|
||||
/// prompt told the agent `/mission/repo`. An agent that obeyed the skill wrote
|
||||
/// into a directory nothing collects. Corrected 2026-08-19, and
|
||||
/// `skills_loader::contradiction_tests` now holds it.
|
||||
fn workspace_boundary(output: &str) -> Verdict {
|
||||
// Only paths that look like a REPO root the agent chose to work in. A
|
||||
// mention of /tmp is normal; `cd /workspace/repo` is not.
|
||||
const WRONG_ROOTS: &[&str] = &["/workspace/repo", "~/workspace/repo"];
|
||||
for root in WRONG_ROOTS {
|
||||
if output.contains(root) {
|
||||
fn workspace_boundary(ev: &Evidence<'_>) -> Verdict {
|
||||
// The recorded writes settle this directly. Before the tool tap the only
|
||||
// available evidence was the narrative, which answers a different question:
|
||||
// an agent that wrote to the wrong root without narrating it scored a
|
||||
// clean pass.
|
||||
let stray: Vec<&str> = ev
|
||||
.writes()
|
||||
.filter_map(|w| w.path.as_deref())
|
||||
.filter(|p| out_of_bounds(p))
|
||||
.collect();
|
||||
if let Some(first) = stray.first() {
|
||||
return Verdict::Fail(format!(
|
||||
"wrote to {first} ({} write(s) outside the checkout) — the mission \
|
||||
checkout is /mission/repo, so anything written there is never \
|
||||
collected and the phase delivers nothing",
|
||||
stray.len()
|
||||
));
|
||||
}
|
||||
|
||||
// `--force` is the skill's other hard prohibition, and unlike the path rule
|
||||
// it leaves no trace anywhere else: the push succeeds and the history it
|
||||
// overwrote is gone.
|
||||
if let Some(cmd) = ev.commands().find(|c| is_force_push(c)) {
|
||||
return Verdict::Fail(format!(
|
||||
"force-pushed ({cmd:?}) — the skill allows it only after an explicit \
|
||||
`HANDOFF: safe to force-push` from the reviewer, because rewriting \
|
||||
pushed history is the one thing here that cannot be undone"
|
||||
));
|
||||
}
|
||||
|
||||
// No violation seen. Whether that is a pass depends on whether there was
|
||||
// anything to see.
|
||||
if ev.writes().next().is_some() {
|
||||
return Verdict::Pass;
|
||||
}
|
||||
if ev.text.contains("/workspace/repo") {
|
||||
return Verdict::Fail(
|
||||
"the narrative describes working in /workspace/repo — the mission \
|
||||
checkout is /mission/repo, so anything written there is never \
|
||||
collected and the phase delivers nothing"
|
||||
.into(),
|
||||
);
|
||||
}
|
||||
if ev.text.contains("/mission/repo") {
|
||||
return Verdict::Pass;
|
||||
}
|
||||
// Silence is not compliance. This returned `Pass` before, which is how a
|
||||
// mission with no evidence at all scored the same as one that was checked.
|
||||
Verdict::NotObservable(
|
||||
"no file-modifying tool call was recorded and the narrative names no \
|
||||
checkout — there is nothing here to place either inside or outside \
|
||||
/mission/repo"
|
||||
.into(),
|
||||
)
|
||||
}
|
||||
|
||||
/// Is this write outside the mission checkout?
|
||||
///
|
||||
/// Relative paths are skipped: the working directory is not recorded, so
|
||||
/// `src/a.rs` could be either, and guessing would fail honest work.
|
||||
///
|
||||
/// The scratch roots are excluded because the skill itself blesses them —
|
||||
/// "anything else is scratch and will not be delivered" — and the hooks, the
|
||||
/// stop gate and claude's own state all live under `/root` by design.
|
||||
fn out_of_bounds(path: &str) -> bool {
|
||||
const SCRATCH: [&str; 5] = ["/tmp/", "/var/tmp/", "/root/", "/dev/", "/proc/"];
|
||||
path.starts_with('/')
|
||||
&& !path.starts_with("/mission/repo")
|
||||
&& !SCRATCH.iter().any(|s| path.starts_with(s))
|
||||
}
|
||||
|
||||
/// Matched on whole tokens, outside the commit message.
|
||||
///
|
||||
/// Both halves are load-bearing, and each was a live false positive before the
|
||||
/// negative control caught it:
|
||||
///
|
||||
/// - `cmd.contains("-f ")` fires on `--follow-tags`.
|
||||
/// - Tokenising the whole command fires on
|
||||
/// `git commit -m "document the -f flag" && git push`, where `-f` is prose.
|
||||
///
|
||||
/// A boundary check that fails honest work gets switched off, which costs more
|
||||
/// than the check was worth.
|
||||
fn is_force_push(cmd: &str) -> bool {
|
||||
const FORCE: [&str; 3] = ["--force", "-f", "--force-with-lease"];
|
||||
let stripped = match dash_m_value(cmd) {
|
||||
Some(msg) => cmd.replace(&msg, " "),
|
||||
None => cmd.to_string(),
|
||||
};
|
||||
let tokens = || stripped.split_whitespace();
|
||||
let pushes = stripped.contains("git") && tokens().any(|t| t == "push");
|
||||
pushes && tokens().any(|t| FORCE.contains(&t))
|
||||
}
|
||||
|
||||
/// The commit subjects the skill names as never-merge.
|
||||
///
|
||||
/// Matched exactly, after stripping an `INT-NN` prefix. A substring match would
|
||||
/// fail `fix the parser panic`, which is a perfectly good subject — the skill's
|
||||
/// objection is to `fix` as the WHOLE message, not to the word.
|
||||
fn commit_subject_boundary(ev: &Evidence<'_>) -> Verdict {
|
||||
const NEVER_MERGE: [&str; 7] = [
|
||||
"wip",
|
||||
"fix",
|
||||
"update",
|
||||
"updates",
|
||||
"update stuff",
|
||||
"stuff",
|
||||
"wip commit",
|
||||
];
|
||||
for subject in commit_subjects(ev) {
|
||||
let bare = strip_int_prefix(&subject).to_ascii_lowercase();
|
||||
let bare = bare.trim_matches(|c: char| !c.is_alphanumeric()).trim();
|
||||
if NEVER_MERGE.contains(&bare) {
|
||||
return Verdict::Fail(format!(
|
||||
"worked in {root} — the mission checkout is /mission/repo, so \
|
||||
anything written there is never collected and the phase \
|
||||
delivers nothing"
|
||||
"committed {subject:?} — the skill lists this exact message as \
|
||||
never-merge, because a reviewer and a bisect both need the \
|
||||
subject to stand alone in `git log --oneline`"
|
||||
));
|
||||
}
|
||||
}
|
||||
Verdict::Pass
|
||||
Verdict::NotApplicable
|
||||
}
|
||||
|
||||
fn strip_int_prefix(subject: &str) -> &str {
|
||||
let s = subject.trim().trim_start_matches(['<', '[', '(']);
|
||||
let Some(rest) = s.strip_prefix("INT-") else {
|
||||
return subject.trim();
|
||||
};
|
||||
let rest = rest.trim_start_matches(|c: char| c.is_ascii_digit());
|
||||
rest.trim_start_matches(['>', ']', ')', ':', '-', '—', ' '])
|
||||
.trim()
|
||||
}
|
||||
|
||||
/// Did a test ever run, and did it run first?
|
||||
///
|
||||
/// One-sided by construction, because only one of the three answers is
|
||||
/// unambiguous:
|
||||
///
|
||||
/// - **Files written, no test command anywhere** — red-green did not happen.
|
||||
/// There is no reading of the loop under which it did.
|
||||
/// - **A test ran before the first write** — that is the RED step, observed.
|
||||
/// - **A test ran after the first write** — undecidable, and reported as such.
|
||||
/// In Rust the unit test lives in `#[cfg(test)] mod tests` *inside the file
|
||||
/// under test, so "wrote the file, then ran the test" is what writing the
|
||||
/// failing test first looks like from here. Scoring that as a failure would
|
||||
/// fail the agents who followed the skill most precisely.
|
||||
fn red_before_green(ev: &Evidence<'_>) -> Verdict {
|
||||
if ev.tools.is_empty() {
|
||||
return Verdict::NotObservable(
|
||||
"no tool calls were recorded for this mission — the order of test \
|
||||
runs and writes is what this check reads, and neither is in the \
|
||||
narrative"
|
||||
.into(),
|
||||
);
|
||||
}
|
||||
let first_write = ev
|
||||
.tools
|
||||
.iter()
|
||||
.position(|t| WRITE_TOOLS.contains(&t.tool.as_str()) && t.path.is_some());
|
||||
let first_test = ev
|
||||
.tools
|
||||
.iter()
|
||||
.position(|t| t.command().is_some_and(is_test_command));
|
||||
match (first_write, first_test) {
|
||||
// Nothing was written, so there was no implementation to test-drive.
|
||||
(None, _) => Verdict::NotApplicable,
|
||||
(Some(_), None) => Verdict::Fail(format!(
|
||||
"wrote {} file(s) and never ran a test — the loop is red, green, \
|
||||
refactor, and a test that never ran cannot have been red",
|
||||
ev.writes().count()
|
||||
)),
|
||||
(Some(w), Some(t)) if t < w => Verdict::Pass,
|
||||
_ => Verdict::NotObservable(
|
||||
"a test ran, but after the first write — in Rust the unit test \
|
||||
lives in the file under test, so this ordering cannot separate \
|
||||
red-first from tests-added-after"
|
||||
.into(),
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
/// Commands that run a test suite, across the languages the skills name.
|
||||
fn is_test_command(cmd: &str) -> bool {
|
||||
const RUNNERS: [&str; 9] = [
|
||||
"cargo test",
|
||||
"cargo nextest",
|
||||
"cargo llvm-cov",
|
||||
"npm test",
|
||||
"yarn test",
|
||||
"pnpm test",
|
||||
"vitest",
|
||||
"pytest",
|
||||
"go test",
|
||||
];
|
||||
// `jest` on its own is too short to match safely — it appears inside
|
||||
// package names and paths — so it is matched as a command word.
|
||||
RUNNERS.iter().any(|r| cmd.contains(r))
|
||||
|| cmd.split_whitespace().any(|w| w == "jest" || w == "npx")
|
||||
&& cmd.contains("jest")
|
||||
}
|
||||
|
||||
/// The subject line of every commit the agent made.
|
||||
///
|
||||
/// Read out of the `-m` argument. A commit written with `-F` or a heredoc
|
||||
/// yields nothing, and the caller must treat an empty result as "no evidence"
|
||||
/// rather than "no commits" — which is why every commit check here returns
|
||||
/// `NotApplicable` on an empty list instead of a verdict.
|
||||
fn commit_subjects(ev: &Evidence<'_>) -> Vec<String> {
|
||||
ev.commands()
|
||||
.filter(|c| c.contains("git commit") || c.contains("git ci"))
|
||||
.filter_map(dash_m_value)
|
||||
.filter_map(|m| m.lines().next().map(|l| l.trim().to_string()))
|
||||
.filter(|s| !s.is_empty())
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The value of the `-m` flag in a shell command.
|
||||
///
|
||||
/// Hand-scanned rather than split on whitespace: the value is the one argument
|
||||
/// guaranteed to contain spaces, and the commit template in
|
||||
/// `workspace-repo-commit-protocol` spans several lines.
|
||||
fn dash_m_value(cmd: &str) -> Option<String> {
|
||||
let b = cmd.as_bytes();
|
||||
for i in 0..b.len() {
|
||||
// Anchor on the `m`, not on `-m`: agents write `git commit -am …` and
|
||||
// anchoring on the dash misses the whole cluster form.
|
||||
if b[i] != b'm' {
|
||||
continue;
|
||||
}
|
||||
// `m` must END the cluster, so `--message` and `-mtime` are not it.
|
||||
if !matches!(
|
||||
b.get(i + 1),
|
||||
None | Some(b' ') | Some(b'\t') | Some(b'"') | Some(b'\'')
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
// Walk back over the cluster's other short flags to its single dash.
|
||||
let mut j = i;
|
||||
while j > 0 && b[j - 1].is_ascii_alphabetic() {
|
||||
j -= 1;
|
||||
}
|
||||
if j == 0 || b[j - 1] != b'-' {
|
||||
continue;
|
||||
}
|
||||
let dash = j - 1;
|
||||
// One dash, starting a word — not `--m` and not the tail of a path.
|
||||
if dash > 0 && !b[dash - 1].is_ascii_whitespace() {
|
||||
continue;
|
||||
}
|
||||
let mut k = i + 1;
|
||||
while k < b.len() && b[k].is_ascii_whitespace() {
|
||||
k += 1;
|
||||
}
|
||||
if k >= b.len() {
|
||||
return None;
|
||||
}
|
||||
let rest = &cmd[k..];
|
||||
let quote = rest.chars().next().filter(|c| *c == '"' || *c == '\'');
|
||||
return match quote {
|
||||
Some(q) => rest[1..].split(q).next().map(str::to_string),
|
||||
None => rest.split_whitespace().next().map(str::to_string),
|
||||
};
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// `arxiv-daily` forbids searching arXiv — the harvest already ran.
|
||||
@@ -213,20 +536,36 @@ fn workspace_boundary(output: &str) -> Verdict {
|
||||
/// This is the one boundary we have watched an agent cross in production, so it
|
||||
/// is checked against the query endpoint rather than the word "arxiv", which
|
||||
/// appears legitimately all over a research turn.
|
||||
fn arxiv_boundary(output: &str) -> Verdict {
|
||||
fn arxiv_boundary(ev: &Evidence<'_>) -> Verdict {
|
||||
// The query endpoint, not the word "arxiv" — and not `arxiv.org/abs/`
|
||||
// either, which the same skill explicitly tells the agent to `curl`. A
|
||||
// check that could not tell those apart would fail agents for following
|
||||
// the paragraph directly under the one it is scoring.
|
||||
const QUERY_MARKERS: &[&str] = &[
|
||||
"export.arxiv.org/api/query",
|
||||
"arxiv.org/api/query",
|
||||
"http://export.arxiv.org",
|
||||
];
|
||||
for m in QUERY_MARKERS {
|
||||
if output.contains(m) {
|
||||
let hit = |s: &str| QUERY_MARKERS.iter().find(|m| s.contains(**m)).copied();
|
||||
|
||||
// Arguments first: a `curl` that ran is a fact, where a narrative that
|
||||
// mentions a URL may be the agent explaining that it did NOT fetch it.
|
||||
for t in ev.tools {
|
||||
let text = t.input.to_string();
|
||||
if let Some(m) = hit(&text) {
|
||||
return Verdict::Fail(format!(
|
||||
"queried the arXiv API ({m}) — the harvest already ran, and \
|
||||
shelving papers outside it corrupts the seen-set"
|
||||
"called {} against the arXiv API ({m}) — the harvest already \
|
||||
ran, and shelving papers outside it corrupts the seen-set",
|
||||
t.tool
|
||||
));
|
||||
}
|
||||
}
|
||||
if let Some(m) = hit(ev.text) {
|
||||
return Verdict::Fail(format!(
|
||||
"queried the arXiv API ({m}) — the harvest already ran, and \
|
||||
shelving papers outside it corrupts the seen-set"
|
||||
));
|
||||
}
|
||||
Verdict::Pass
|
||||
}
|
||||
|
||||
@@ -261,6 +600,13 @@ pub async fn score_mission(
|
||||
}
|
||||
}
|
||||
|
||||
// What the agent DID, alongside what it said. A mission that ran before
|
||||
// the tool tap existed returns an empty list, and the checks degrade to
|
||||
// the narrative rather than reporting a violation they cannot see.
|
||||
let tools = crate::mission_events::tool_evidence_for_mission(pool, mission_id)
|
||||
.await
|
||||
.map_err(|e| format!("read tool evidence: {e}"))?;
|
||||
|
||||
// One lookup for every delivered name, so the report can separate
|
||||
// hand-authored skills from the ones agents wrote for themselves.
|
||||
let names = skills_in_prompt(&prompts);
|
||||
@@ -274,7 +620,7 @@ pub async fn score_mission(
|
||||
.into_iter()
|
||||
.collect();
|
||||
|
||||
Ok(score(&prompts, &outputs, &|name| {
|
||||
Ok(score(&prompts, &Evidence::new(&outputs, &tools), &|name| {
|
||||
kinds
|
||||
.get(name)
|
||||
.cloned()
|
||||
@@ -287,11 +633,35 @@ pub async fn score_mission(
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::mission_events::ToolEvidence;
|
||||
use serde_json::json;
|
||||
|
||||
fn builtin(_: &str) -> String {
|
||||
"builtin".to_string()
|
||||
}
|
||||
|
||||
/// Narrative-only evidence — a mission whose events predate the tool tap.
|
||||
fn narrative(text: &str) -> Evidence<'_> {
|
||||
Evidence::from_text(text)
|
||||
}
|
||||
|
||||
/// Evidence from recorded ACTIONS. `(tool, path, input)`.
|
||||
fn acted(calls: &[(&str, Option<&str>, serde_json::Value)]) -> Vec<ToolEvidence> {
|
||||
calls
|
||||
.iter()
|
||||
.map(|(tool, path, input)| ToolEvidence {
|
||||
tool: (*tool).to_string(),
|
||||
path: path.map(str::to_string),
|
||||
input: input.clone(),
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// One `Bash` call.
|
||||
fn ran(cmd: &str) -> (&'static str, Option<&'static str>, serde_json::Value) {
|
||||
("Bash", None, serde_json::json!({ "command": cmd }))
|
||||
}
|
||||
|
||||
/// A prompt exactly as the delivery layer renders it.
|
||||
fn rendered(skills: &[(&str, &str)]) -> String {
|
||||
let body: String = skills
|
||||
@@ -355,13 +725,13 @@ mod tests {
|
||||
#[test]
|
||||
fn a_prompt_with_no_skills_yields_no_scores() {
|
||||
assert!(skills_in_prompt("Task: do the thing").is_empty());
|
||||
assert!(score("Task: do the thing", "did it", &builtin).is_empty());
|
||||
assert!(score("Task: do the thing", &narrative("did it"), &builtin).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn trigger_is_reported_as_unobservable_not_as_a_failure() {
|
||||
let prompt = rendered(&[("arxiv-daily", "x")]);
|
||||
let scored = score(&prompt, "I read the manifest.", &builtin);
|
||||
let scored = score(&prompt, &narrative("I read the manifest."), &builtin);
|
||||
assert!(
|
||||
matches!(scored[0].trigger, Verdict::NotObservable(_)),
|
||||
"scoring Trigger zero would report a delivery-model property as an \
|
||||
@@ -373,7 +743,7 @@ mod tests {
|
||||
fn markers_the_real_parser_accepts_are_compliant() {
|
||||
let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]);
|
||||
let good = "I implemented the parser.\nCOMPLETED: INT-07 — wire the loop";
|
||||
let scored = score(&prompt, good, &builtin);
|
||||
let scored = score(&prompt, &narrative(good), &builtin);
|
||||
assert_eq!(scored[0].compliance, Verdict::Pass);
|
||||
}
|
||||
|
||||
@@ -384,7 +754,7 @@ mod tests {
|
||||
let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]);
|
||||
// Bold, which the skill explicitly forbids, and the parser will not take.
|
||||
let bad = "**COMPLETED: INT-07**";
|
||||
let scored = score(&prompt, bad, &builtin);
|
||||
let scored = score(&prompt, &narrative(bad), &builtin);
|
||||
assert!(
|
||||
matches!(scored[0].compliance, Verdict::Fail(_)),
|
||||
"a marker the parser rejects must score as a failure — that is the \
|
||||
@@ -393,7 +763,7 @@ mod tests {
|
||||
);
|
||||
|
||||
// A turn that never tried is not a violation.
|
||||
let unrelated = score(&prompt, "I read three files and wrote a summary.", &builtin);
|
||||
let unrelated = score(&prompt, &narrative("I read three files and wrote a summary."), &builtin);
|
||||
assert_eq!(unrelated[0].compliance, Verdict::NotApplicable);
|
||||
}
|
||||
|
||||
@@ -417,7 +787,7 @@ mod tests {
|
||||
|
||||
let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]);
|
||||
assert_eq!(
|
||||
score(&prompt, "PLAN_COMPLETE: INT-01", &builtin)[0].compliance,
|
||||
score(&prompt, &narrative("PLAN_COMPLETE: INT-01"), &builtin)[0].compliance,
|
||||
Verdict::Pass
|
||||
);
|
||||
}
|
||||
@@ -434,7 +804,7 @@ mod tests {
|
||||
|
||||
// And the agent is told, because the marker it emitted did nothing.
|
||||
let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]);
|
||||
match &score(&prompt, "PLAN_COMPLETE: INT-01..02", &builtin)[0].compliance {
|
||||
match &score(&prompt, &narrative("PLAN_COMPLETE: INT-01..02"), &builtin)[0].compliance {
|
||||
Verdict::Fail(why) => assert!(why.contains("does not accept")),
|
||||
other => panic!("an ignored marker must not read as success; got {other:?}"),
|
||||
}
|
||||
@@ -443,7 +813,7 @@ mod tests {
|
||||
#[test]
|
||||
fn two_int_ids_on_one_marker_line_cross_the_boundary() {
|
||||
let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]);
|
||||
let scored = score(&prompt, "COMPLETED: INT-05, INT-06", &builtin);
|
||||
let scored = score(&prompt, &narrative("COMPLETED: INT-05, INT-06"), &builtin);
|
||||
match &scored[0].boundary {
|
||||
Verdict::Fail(why) => assert!(why.contains("silently dropped")),
|
||||
other => panic!("the second id is silently dropped by the parser; got {other:?}"),
|
||||
@@ -456,7 +826,7 @@ mod tests {
|
||||
|
||||
let violating = score(
|
||||
&prompt,
|
||||
"curl 'http://export.arxiv.org/api/query?search_query=all:agents'",
|
||||
&narrative("curl 'http://export.arxiv.org/api/query?search_query=all:agents'"),
|
||||
&builtin,
|
||||
);
|
||||
assert!(matches!(violating[0].boundary, Verdict::Fail(_)));
|
||||
@@ -465,7 +835,7 @@ mod tests {
|
||||
// would make the metric fire constantly and mean nothing.
|
||||
let fine = score(
|
||||
&prompt,
|
||||
"I read the arXiv notes in the manifest and summarised three of them.",
|
||||
&narrative("I read the arXiv notes in the manifest and summarised three of them."),
|
||||
&builtin,
|
||||
);
|
||||
assert_eq!(fine[0].boundary, Verdict::Pass);
|
||||
@@ -478,7 +848,7 @@ mod tests {
|
||||
|
||||
let wrong = score(
|
||||
&prompt,
|
||||
"cd /workspace/repo && git add -A && git commit -m 'INT-01 done'",
|
||||
&narrative("cd /workspace/repo && git add -A && git commit -m 'INT-01 done'"),
|
||||
&builtin,
|
||||
);
|
||||
match &wrong[0].boundary {
|
||||
@@ -492,16 +862,203 @@ mod tests {
|
||||
|
||||
let right = score(
|
||||
&prompt,
|
||||
"cd /mission/repo && git add -A && git commit -m 'INT-01 done'",
|
||||
&narrative("cd /mission/repo && git add -A && git commit -m 'INT-01 done'"),
|
||||
&builtin,
|
||||
);
|
||||
assert_eq!(right[0].boundary, Verdict::Pass);
|
||||
}
|
||||
|
||||
/// The upgrade, stated as a test: the agent never said where it wrote.
|
||||
///
|
||||
/// Before the tool tap this scored a clean `Pass`, because the only
|
||||
/// evidence was the narrative and the narrative did not mention a path. The
|
||||
/// phase had still delivered nothing.
|
||||
#[test]
|
||||
fn a_write_outside_the_checkout_is_caught_even_when_the_narrative_is_silent() {
|
||||
let prompt = rendered(&[("workspace-repo-commit-protocol", "Work in /mission/repo.")]);
|
||||
let tools = acted(&[
|
||||
("Write", Some("/app/src/main.rs"), json!({"file_path": "/app/src/main.rs"})),
|
||||
]);
|
||||
let scored = score(&prompt, &Evidence::new("I implemented the feature.", &tools), &builtin);
|
||||
match &scored[0].boundary {
|
||||
Verdict::Fail(why) => assert!(why.contains("/app/src/main.rs"), "{why}"),
|
||||
other => panic!("the write is right there in the record; got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
/// Scratch is not a violation — the skill blesses it by name.
|
||||
#[test]
|
||||
fn scratch_and_the_checkout_both_stay_inside_the_boundary() {
|
||||
let prompt = rendered(&[("workspace-repo-commit-protocol", "Work in /mission/repo.")]);
|
||||
let tools = acted(&[
|
||||
("Write", Some("/tmp/scratch.txt"), json!({"file_path": "/tmp/scratch.txt"})),
|
||||
("Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"})),
|
||||
]);
|
||||
let scored = score(&prompt, &Evidence::new("done", &tools), &builtin);
|
||||
assert_eq!(scored[0].boundary, Verdict::Pass);
|
||||
}
|
||||
|
||||
/// Nothing observed is not the same as nothing done.
|
||||
///
|
||||
/// This returned `Pass` before — a mission with no evidence scored
|
||||
/// identically to one that was checked and found clean, which is the exact
|
||||
/// collapse `Verdict::NotObservable` exists to prevent.
|
||||
#[test]
|
||||
fn silence_is_not_scored_as_compliance() {
|
||||
let prompt = rendered(&[("workspace-repo-commit-protocol", "Work in /mission/repo.")]);
|
||||
let scored = score(&prompt, &narrative("I thought about the problem."), &builtin);
|
||||
assert!(
|
||||
matches!(scored[0].boundary, Verdict::NotObservable(_)),
|
||||
"got {:?}",
|
||||
scored[0].boundary
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_force_push_crosses_the_boundary_and_a_quoted_flag_does_not() {
|
||||
let prompt = rendered(&[("workspace-repo-commit-protocol", "Work in /mission/repo.")]);
|
||||
|
||||
let forced = acted(&[
|
||||
("Write", Some("/mission/repo/a.rs"), json!({"file_path": "/mission/repo/a.rs"})),
|
||||
ran("cd /mission/repo && git push --force origin mission"),
|
||||
]);
|
||||
match &score(&prompt, &Evidence::new("", &forced), &builtin)[0].boundary {
|
||||
Verdict::Fail(why) => assert!(why.contains("force-push"), "{why}"),
|
||||
other => panic!("rewriting pushed history is the one unrecoverable act; got {other:?}"),
|
||||
}
|
||||
|
||||
// The flag inside a commit message rewrites nothing.
|
||||
let innocent = acted(&[
|
||||
("Write", Some("/mission/repo/a.rs"), json!({"file_path": "/mission/repo/a.rs"})),
|
||||
ran(r#"git commit -m "document the -f flag" && git push origin mission"#),
|
||||
]);
|
||||
assert_eq!(
|
||||
score(&prompt, &Evidence::new("", &innocent), &builtin)[0].boundary,
|
||||
Verdict::Pass
|
||||
);
|
||||
}
|
||||
|
||||
/// Writing code and never running a test cannot be red-green-refactor under
|
||||
/// any reading of the loop.
|
||||
#[test]
|
||||
fn code_written_with_no_test_run_fails_the_loop() {
|
||||
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
||||
let tools = acted(&[
|
||||
("Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"})),
|
||||
ran("cargo build"),
|
||||
]);
|
||||
match &score(&prompt, &Evidence::new("Implemented it.", &tools), &builtin)[0].compliance {
|
||||
Verdict::Fail(why) => assert!(why.contains("never ran a test"), "{why}"),
|
||||
other => panic!("got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_test_run_before_the_first_write_is_red_first() {
|
||||
let prompt = rendered(&[("cargo-test-driven-development", "Red, green, refactor.")]);
|
||||
let tools = acted(&[
|
||||
ran("cargo nextest run parses_the_range_form"),
|
||||
("Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"})),
|
||||
ran("cargo nextest run parses_the_range_form"),
|
||||
]);
|
||||
assert_eq!(
|
||||
score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance,
|
||||
Verdict::Pass
|
||||
);
|
||||
}
|
||||
|
||||
/// The ordering that cannot be read either way, reported as such.
|
||||
///
|
||||
/// A Rust unit test lives in `#[cfg(test)] mod tests` inside the file under
|
||||
/// test, so writing the failing test IS a write to the implementation file.
|
||||
/// Scoring this as a failure would fail the agents who followed the skill
|
||||
/// most precisely.
|
||||
#[test]
|
||||
fn a_test_run_after_the_first_write_is_undecidable_not_a_failure() {
|
||||
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
||||
let tools = acted(&[
|
||||
("Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"})),
|
||||
ran("cargo test -p cm-api"),
|
||||
]);
|
||||
assert!(
|
||||
matches!(
|
||||
score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance,
|
||||
Verdict::NotObservable(_)
|
||||
),
|
||||
"the Rust unit test lives in the file under test — this ordering \
|
||||
cannot separate red-first from tests-added-after"
|
||||
);
|
||||
}
|
||||
|
||||
/// A phase that wrote nothing had no implementation to test-drive.
|
||||
#[test]
|
||||
fn a_phase_that_wrote_nothing_is_not_a_tdd_failure() {
|
||||
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
||||
let tools = acted(&[ran("ls /mission/repo"), ("Read", Some("/mission/repo/a.rs"), json!({}))]);
|
||||
assert_eq!(
|
||||
score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance,
|
||||
Verdict::NotApplicable
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_never_merge_subjects_are_caught_and_a_real_one_is_not() {
|
||||
let prompt = rendered(&[("small-focused-commits", "One idea per commit.")]);
|
||||
|
||||
let bad = acted(&[ran(r#"git commit -m "wip""#)]);
|
||||
match &score(&prompt, &Evidence::new("", &bad), &builtin)[0].boundary {
|
||||
Verdict::Fail(why) => assert!(why.contains("never-merge"), "{why}"),
|
||||
other => panic!("got {other:?}"),
|
||||
}
|
||||
|
||||
// The objection is to `fix` as the whole message, not to the word.
|
||||
let good = acted(&[ran(r#"git commit -m "INT-04 fix the range form the parser rejects""#)]);
|
||||
assert_eq!(
|
||||
score(&prompt, &Evidence::new("", &good), &builtin)[0].boundary,
|
||||
Verdict::NotApplicable
|
||||
);
|
||||
}
|
||||
|
||||
/// The subject comes out of the shell command, in the shapes agents write.
|
||||
#[test]
|
||||
fn the_commit_subject_is_read_out_of_the_real_command_shapes() {
|
||||
assert_eq!(dash_m_value(r#"git commit -m "one line""#).as_deref(), Some("one line"));
|
||||
assert_eq!(dash_m_value("git commit -am 'clustered flag'").as_deref(), Some("clustered flag"));
|
||||
// The template in `workspace-repo-commit-protocol` spans several lines;
|
||||
// only the first is the subject.
|
||||
let multi = "cd /mission/repo && git commit -m \"INT-01 wire the loop\n\nWHY: it stalled.\n\nRefs: INT-01\n\"";
|
||||
assert_eq!(
|
||||
commit_subjects(&Evidence::new("", &acted(&[ran(multi)]))),
|
||||
vec!["INT-01 wire the loop"]
|
||||
);
|
||||
// `-F` gives up nothing, and that must read as no evidence.
|
||||
assert!(dash_m_value("git commit -F /tmp/msg").is_none());
|
||||
}
|
||||
|
||||
/// The skill tells the agent to `curl` the abstract page in the paragraph
|
||||
/// under the one that forbids the query API. A check that cannot tell them
|
||||
/// apart fails agents for obeying the skill.
|
||||
#[test]
|
||||
fn curling_an_abstract_is_allowed_where_querying_the_api_is_not() {
|
||||
let prompt = rendered(&[("arxiv-daily", "The harvest already ran.")]);
|
||||
|
||||
let allowed = acted(&[ran("curl -s https://arxiv.org/abs/2401.12345")]);
|
||||
assert_eq!(
|
||||
score(&prompt, &Evidence::new("", &allowed), &builtin)[0].boundary,
|
||||
Verdict::Pass
|
||||
);
|
||||
|
||||
let forbidden = acted(&[ran("curl -s 'http://export.arxiv.org/api/query?search_query=all:agents'")]);
|
||||
match &score(&prompt, &Evidence::new("", &forbidden), &builtin)[0].boundary {
|
||||
Verdict::Fail(why) => assert!(why.contains("seen-set"), "{why}"),
|
||||
other => panic!("got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_source_of_a_skill_is_carried_into_its_score() {
|
||||
let prompt = rendered(&[("self-made", "x")]);
|
||||
let scored = score(&prompt, "done", &|_| "promoted_from_brain".to_string());
|
||||
let scored = score(&prompt, &narrative("done"), &|_| "promoted_from_brain".to_string());
|
||||
assert_eq!(
|
||||
scored[0].source_kind, "promoted_from_brain",
|
||||
"an agent that writes its own skill can raise its own score against \
|
||||
|
||||
Reference in New Issue
Block a user