Omar SobhandClaude Opus 5
758760cedb
The scorer could see that agents OPENED web-search-triage (trigger=pass on both files-arm runs) and nothing about whether they followed it — compliance was not_applicable because no mechanical check existed. The evidence was in the recorded arguments the whole time. On the runs that read the skill, the parent decomposed the sweep into per-source fetches and sent each to a subagent; on 01a09b42 two of those spawn prompts read "Return the URL, date if visible, and the key content". The task never asked for a date. The skill's "undated is a finding" did. On the runs that did not read it: inline curls, no subagents, no date. Two of the skill's rules leave a mark in arguments, and the check scores exactly those two. The ranking rule: every URL a fetch was sent to is classified against a short allow-list of primary hosts (rank 0) and a short skip-list of aggregators (rank 3+); fetching an aggregator is the visible violation, fetching primary sources the visible compliance, and anything unrecognised is unranked and decides nothing. The date rule: reported as extra evidence on a pass, never required for one, because a curl to an abstract page has no prompt to ask in. `Agent` is a fetching tool here on purpose. The URLs on the files-arm runs live in the spawn PROMPT; a check that only read curl lines would have scored those runs as fetching nothing. `Verdict::PassWith(String)` carries the evidence and serialises under the same "pass" tag, so no reader grows a fourth branch and the one that looks finds the date fingerprint in `why`. One-sided like every check in this module: no tools is not observable, no fetch is not applicable, an unrankable fetch is not a violation. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
2091 lines
86 KiB
Rust
2091 lines
86 KiB
Rust
//! Did a skill we delivered actually change what the agent did?
|
|
//!
|
|
//! Scored on the three axes from `Skill-Use` (arXiv, 2026-08-05): **Trigger**
|
|
//! (did the agent reach for the skill), **Compliance** (did it follow the
|
|
//! procedure), **Boundary** (did it avoid what the skill forbids).
|
|
//!
|
|
//! ## One axis does not survive the translation, and saying so is the finding
|
|
//!
|
|
//! The paper measures agents under *progressive disclosure*: the agent sees a
|
|
//! name and a description, and must decide to retrieve the body. The retrieval
|
|
//! is the Trigger event, and it is observable because it is a tool call.
|
|
//!
|
|
//! We do not deliver skills that way on the mission path. `pinned_skills_text`
|
|
//! inlines full bodies into the prompt, so the agent never "reaches for" a
|
|
//! skill; it is simply holding one.
|
|
//!
|
|
//! Trigger is therefore **not observable on the mission path**, and this module
|
|
//! reports it as `NotObservable` with the reason attached rather than scoring
|
|
//! it zero. A zero would read as "the agents ignore their skills" when it
|
|
//! actually means "the question does not apply to how we deliver them" — the
|
|
//! precise confusion that made 55 empty skill bindings invisible for months.
|
|
//!
|
|
//! ### The reason changed, and only half of it went away
|
|
//!
|
|
//! Until 2026-08-21 the stated reason was that `claude_cli` "cannot surface a
|
|
//! tool call at all — there is nothing to retrieve *with*". That half is now
|
|
//! false: [`crate::container_tool_hooks`] installs `PostToolUse`, mission tool
|
|
//! calls land in `mission_events`, and a retrieval would be as visible as any
|
|
//! other call.
|
|
//!
|
|
//! The other half still holds, and it is the one that decides the verdict: we
|
|
//! still **inline**. Trigger is now *instrumentable* and still not
|
|
//! *observable*, and the blocker has moved from the transport to the delivery
|
|
//! model. Making it real is one change — serve skills through the door
|
|
//! (`docs/TOOL-CALL-ARCHITECTURE.md` §3) so retrieval becomes a tool call —
|
|
//! and nothing else in this module has to change to score it.
|
|
//!
|
|
//! ## Compliance and Boundary are scored from ACTIONS where actions exist
|
|
//!
|
|
//! They were scored from the concatenated narrative, which is the agent's own
|
|
//! account of its turn — written by the thing being measured, and silent about
|
|
//! anything it did not think worth mentioning. [`Evidence`] carries the
|
|
//! recorded tool calls alongside that text, and every check prefers them:
|
|
//! `Bash` arguments say what ran, `Write` paths say where it landed.
|
|
//!
|
|
//! Every tool-backed check is **one-sided**. It reports a violation it can see
|
|
//! and never infers compliance from silence — the recorded stream is capped
|
|
//! per phase ([`crate::mission_events::PER_PHASE_CAP`]), so an absent call is
|
|
//! not proof of an absent action.
|
|
|
|
use serde::Serialize;
|
|
|
|
/// The outcome of one axis for one skill.
|
|
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
|
|
#[serde(rename_all = "snake_case", tag = "verdict", content = "why")]
|
|
pub enum Verdict {
|
|
Pass,
|
|
/// A pass that can say what it saw. Same tag as [`Verdict::Pass`] on the
|
|
/// wire — `{"verdict":"pass","why":…}` — so every reader that keys on the
|
|
/// tag is unaffected and the evidence is there for the one that looks.
|
|
///
|
|
/// Exists because a bare pass on `web-search-triage` would have hidden
|
|
/// the only finding worth having: the agent's spawn prompts asked for the
|
|
/// page's date, which the task never did and the skill does.
|
|
#[serde(rename = "pass")]
|
|
PassWith(String),
|
|
Fail(String),
|
|
/// The skill says nothing this axis can check.
|
|
NotApplicable,
|
|
/// The axis cannot be measured here, for a stated structural reason.
|
|
///
|
|
/// Distinct from `Fail` on purpose. Collapsing the two is how a
|
|
/// measurement reports a system defect as an agent defect.
|
|
NotObservable(String),
|
|
}
|
|
|
|
impl Verdict {
|
|
pub fn label(&self) -> &'static str {
|
|
match self {
|
|
Verdict::Pass | Verdict::PassWith(_) => "pass",
|
|
Verdict::Fail(_) => "FAIL",
|
|
Verdict::NotApplicable => "n/a",
|
|
Verdict::NotObservable(_) => "not observable",
|
|
}
|
|
}
|
|
}
|
|
|
|
/// One skill's score for one phase.
|
|
#[derive(Debug, Clone, Serialize)]
|
|
pub struct SkillUse {
|
|
pub skill: String,
|
|
/// `builtin` (hand-authored) or `promoted_from_brain` (agent-authored).
|
|
///
|
|
/// Carried into the report because an agent that authors its own skill can
|
|
/// raise its own compliance score without changing what it does. A rising
|
|
/// number on agent-authored skills has to be visible as such rather than
|
|
/// averaged in with the rest.
|
|
pub source_kind: String,
|
|
pub trigger: Verdict,
|
|
pub compliance: Verdict,
|
|
pub boundary: Verdict,
|
|
}
|
|
|
|
/// The skills a prompt actually delivered.
|
|
///
|
|
/// Delegates to the delivery layer's own parser, so the reader cannot drift
|
|
/// from the writer. This function originally matched `## <name>` itself, and
|
|
/// skill bodies are markdown full of `##` headings — a live mission duly
|
|
/// scored "Sizing heuristic" and "The output shape" as skills. Parsed from the
|
|
/// recorded prompt rather than re-derived from the catalogue, because the
|
|
/// catalogue changes, and now that agents author their own skills it changes
|
|
/// by itself.
|
|
pub fn skills_in_prompt(prompt: &str) -> Vec<String> {
|
|
crate::topology_exec::skill_names_in(prompt)
|
|
}
|
|
|
|
/// What a mission left behind, for the checks to read.
|
|
///
|
|
/// Two sources, deliberately kept apart. `text` is what the agent *said* it
|
|
/// did; `tools` is what it did. Where they disagree the tools win, and a check
|
|
/// that can only consult `text` says so in its verdict rather than presenting a
|
|
/// narrative reading as an observation.
|
|
pub struct Evidence<'a> {
|
|
/// Concatenated `reasoning` events for the mission.
|
|
pub text: &'a str,
|
|
/// Every `tool.call` recorded for the mission, in order.
|
|
pub tools: &'a [crate::mission_events::ToolEvidence],
|
|
}
|
|
|
|
/// Tools that modify a file. `target` on the recorded event is the tool name.
|
|
const WRITE_TOOLS: [&str; 4] = ["Write", "Edit", "MultiEdit", "NotebookEdit"];
|
|
|
|
impl<'a> Evidence<'a> {
|
|
pub fn new(text: &'a str, tools: &'a [crate::mission_events::ToolEvidence]) -> Self {
|
|
Evidence { text, tools }
|
|
}
|
|
|
|
/// Narrative only — a mission whose events predate the tool tap.
|
|
pub fn from_text(text: &'a str) -> Self {
|
|
Evidence { text, tools: &[] }
|
|
}
|
|
|
|
/// Every shell command the agent ran, in order.
|
|
fn commands(&self) -> impl Iterator<Item = &str> {
|
|
self.tools.iter().filter_map(|t| t.command())
|
|
}
|
|
|
|
/// Test runs, in order, each with whether it went red or green.
|
|
///
|
|
/// `None` where the outcome was not recorded — a mission whose events
|
|
/// predate the response capture, which must not read as either.
|
|
fn test_runs(&self) -> Vec<Option<bool>> {
|
|
self.tools
|
|
.iter()
|
|
.filter(|t| t.command().is_some_and(is_test_command))
|
|
.map(|t| test_passed(&t.response))
|
|
.collect()
|
|
}
|
|
|
|
/// Every file-modifying call that named a path, in order.
|
|
fn writes(&self) -> impl Iterator<Item = &crate::mission_events::ToolEvidence> {
|
|
self.tools
|
|
.iter()
|
|
.filter(|t| WRITE_TOOLS.contains(&t.tool.as_str()) && t.path.is_some())
|
|
}
|
|
}
|
|
|
|
/// Skills the agent RETRIEVED, in the order it reached for them.
|
|
///
|
|
/// This is Trigger, and it is observable the moment a skill arrives by
|
|
/// retrieval instead of by inlining. Claude Code reads an MCP resource through
|
|
/// `ReadMcpResourceTool`, and the recorded arguments carry the URI:
|
|
///
|
|
/// ```text
|
|
/// ReadMcpResourceTool {"uri":"skill:global/workspace-repo-commit-protocol",
|
|
/// "server":"clawmates_skills"}
|
|
/// ```
|
|
///
|
|
/// The URI is parsed by `mcp_skills::parse_uri` — the same function that
|
|
/// WROTE it — rather than by a second matcher here, for the reason every other
|
|
/// reader in this module delegates: two implementations of one format drift,
|
|
/// and the drift shows up as a skill silently scoring nothing.
|
|
pub fn retrieved_skills(ev: &Evidence<'_>) -> Vec<String> {
|
|
let mut out = Vec::new();
|
|
for t in ev.tools {
|
|
let name = match t.tool.as_str() {
|
|
"ReadMcpResourceTool" => t
|
|
.input
|
|
.get("uri")
|
|
.and_then(|v| v.as_str())
|
|
.and_then(crate::mcp_skills::parse_uri)
|
|
.map(|(_, name)| name),
|
|
// The `files` arm: the pointer is a path and the retrieval is a
|
|
// plain `Read`. Matched through `skill_from_file_path`, the reader
|
|
// half of the function that wrote the path, for the same reason
|
|
// the uri goes through `parse_uri`. A `Read` anywhere else is an
|
|
// ordinary file read and is not a retrieval of anything.
|
|
"Read" => t
|
|
.input
|
|
.get("file_path")
|
|
.and_then(|v| v.as_str())
|
|
.and_then(crate::skill_delivery::skill_from_file_path),
|
|
_ => None,
|
|
};
|
|
if let Some(name) = name {
|
|
if !out.contains(&name) {
|
|
out.push(name);
|
|
}
|
|
}
|
|
}
|
|
out
|
|
}
|
|
|
|
/// Did the agent reach for this skill?
|
|
///
|
|
/// The answer depends on whether it was ever given the chance, which is what
|
|
/// the delivery arm decides — so this takes the arm rather than assuming one.
|
|
/// Getting that wrong is not a rounding error: under `Index` the old text
|
|
/// would have said "this skill was inlined into the prompt" about a skill that
|
|
/// was not, and scored a real miss as a structural blind spot.
|
|
fn trigger_verdict(
|
|
mode: crate::skill_delivery::Mode,
|
|
retrieved: bool,
|
|
compliance: &Verdict,
|
|
boundary: &Verdict,
|
|
) -> Verdict {
|
|
if retrieved {
|
|
// The agent reached for it. That is the paper's Trigger, and it is a
|
|
// recorded tool call like any other.
|
|
return Verdict::Pass;
|
|
}
|
|
match mode {
|
|
// Handed over, so there was no reaching-for to observe. Not a failure
|
|
// and not a pass — the axis simply does not exist in this arm.
|
|
crate::skill_delivery::Mode::Inline => Verdict::NotObservable(
|
|
"this skill was inlined into the prompt, not retrieved — the agent \
|
|
was handed it, so there is no reaching-for to observe. Serve it \
|
|
through the door instead and this becomes a tool call"
|
|
.into(),
|
|
),
|
|
// Both retrieval arms: offered by name and `when_to_use`, and never
|
|
// opened. Whether that is a miss depends on whether the skill had
|
|
// anything to say about this phase at all: a skill with no
|
|
// machine-checkable consequence here is one an agent is right to pass
|
|
// over, and scoring that as a failure would punish correct triage.
|
|
crate::skill_delivery::Mode::Index | crate::skill_delivery::Mode::Files => {
|
|
if matches!(compliance, Verdict::NotApplicable)
|
|
&& matches!(boundary, Verdict::NotApplicable)
|
|
{
|
|
Verdict::NotApplicable
|
|
} else {
|
|
Verdict::Fail(
|
|
"offered in the index with its `when_to_use`, and never read — \
|
|
the agent had the entry in front of it and did not fetch the \
|
|
procedure"
|
|
.into(),
|
|
)
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Score every skill a phase's prompt delivered, against what the agent produced.
|
|
pub fn score(
|
|
prompt: &str,
|
|
evidence: &Evidence<'_>,
|
|
source_kinds: &dyn Fn(&str) -> String,
|
|
) -> Vec<SkillUse> {
|
|
let retrieved = retrieved_skills(evidence);
|
|
// Read off the prompt that was actually sent, not off the mission row: the
|
|
// row says what the mission is configured to do now, and this is scoring a
|
|
// turn that ran then.
|
|
let mode = crate::skill_delivery::mode_in_prompt(prompt);
|
|
// Delivered by either route. A skill that was retrieved and never inlined
|
|
// is invisible to `skills_in_prompt`, and under progressive disclosure that
|
|
// is EVERY skill — so scoring only the prompt would report zero for the
|
|
// delivery model this axis exists to measure.
|
|
let mut names = skills_in_prompt(prompt);
|
|
for r in &retrieved {
|
|
if !names.contains(r) {
|
|
names.push(r.clone());
|
|
}
|
|
}
|
|
names
|
|
.into_iter()
|
|
.map(|skill| {
|
|
let (compliance, boundary) = check(&skill, evidence);
|
|
// The arm belongs to the prompt; `always_inject` belongs to the
|
|
// skill. A skill whose BODY is in the prompt was handed over, so
|
|
// there is no reaching-for to observe even under `Index` — scoring
|
|
// it as a Trigger miss would report a failure against an agent that
|
|
// was never asked to fetch anything.
|
|
let delivered = match crate::skill_delivery::skill_was_indexed(prompt, &skill) {
|
|
Some(false) => crate::skill_delivery::Mode::Inline,
|
|
_ => mode,
|
|
};
|
|
SkillUse {
|
|
source_kind: source_kinds(&skill),
|
|
trigger: trigger_verdict(
|
|
delivered,
|
|
retrieved.contains(&skill),
|
|
&compliance,
|
|
&boundary,
|
|
),
|
|
compliance,
|
|
boundary,
|
|
skill,
|
|
}
|
|
})
|
|
.collect()
|
|
}
|
|
|
|
/// Per-skill mechanical checks.
|
|
///
|
|
/// Only skills whose procedure has a machine-checkable consequence are checked.
|
|
/// Everything else returns `NotApplicable` rather than a guess: a heuristic
|
|
/// that scores prose by keyword overlap produces a number that looks like a
|
|
/// measurement and is not one.
|
|
fn check(skill: &str, ev: &Evidence<'_>) -> (Verdict, Verdict) {
|
|
match skill {
|
|
"int-xx-marker-protocol" => (marker_compliance(ev.text), marker_boundary(ev.text)),
|
|
"arxiv-daily" => (Verdict::NotApplicable, arxiv_boundary(ev)),
|
|
"workspace-repo-commit-protocol" => (Verdict::NotApplicable, workspace_boundary(ev)),
|
|
"small-focused-commits" => (Verdict::NotApplicable, commit_subject_boundary(ev)),
|
|
"web-search-triage" => (triage_compliance(ev), Verdict::NotApplicable),
|
|
// One check for both: `cargo-test-driven-development` is the Rust
|
|
// flavour of the same loop, and its own text says so. Scoring them by
|
|
// separate rules would mean two rules for one procedure, which is how
|
|
// the microVM and container taps drifted apart.
|
|
"tdd-red-green-refactor" | "cargo-test-driven-development" => {
|
|
(red_before_green(ev), Verdict::NotApplicable)
|
|
}
|
|
_ => (Verdict::NotApplicable, Verdict::NotApplicable),
|
|
}
|
|
}
|
|
|
|
/// The markers must be ones `task_card_parser` actually parses.
|
|
///
|
|
/// Checked by running the real parser rather than a copy of its rules — a
|
|
/// second implementation of the contract would drift from the first, and then
|
|
/// the measurement would pass while the mission loop still stalled.
|
|
fn marker_compliance(output: &str) -> Verdict {
|
|
if crate::task_card_parser::parse(output).is_empty() {
|
|
// Only a failure if the output looks like it TRIED. A turn with no
|
|
// marker-shaped line was probably not a coding turn at all.
|
|
if output.lines().any(|l| looks_like_marker_attempt(l)) {
|
|
return Verdict::Fail(
|
|
"emitted marker-shaped lines that the parser does not accept — \
|
|
the mission loop will not advance"
|
|
.into(),
|
|
);
|
|
}
|
|
return Verdict::NotApplicable;
|
|
}
|
|
Verdict::Pass
|
|
}
|
|
|
|
/// Rule 1: exactly one INT id per marker line.
|
|
///
|
|
/// Two shapes violate it, and they fail differently:
|
|
///
|
|
/// `COMPLETED: INT-05, INT-06` — the parser takes the first and drops the
|
|
/// rest, so an item is silently never closed.
|
|
/// `PLAN_COMPLETE: INT-01..02` — the range form. The parser now REJECTS a
|
|
/// malformed id, so this is caught by the
|
|
/// compliance check above as a marker the
|
|
/// parser does not accept.
|
|
///
|
|
/// The second was found by running this measurement against a live mission. It
|
|
/// is the worse of the two, because a dropped marker leaves a gap and a
|
|
/// malformed one leaves a plausible-looking row.
|
|
fn marker_boundary(output: &str) -> Verdict {
|
|
for line in output.lines() {
|
|
let t = line.trim();
|
|
if !looks_like_marker_attempt(t) {
|
|
continue;
|
|
}
|
|
let ids = t.matches("INT-").count();
|
|
if ids > 1 {
|
|
return Verdict::Fail(format!(
|
|
"{ids} INT ids on one marker line — only the first parses, so \
|
|
the rest are silently dropped: {t:?}"
|
|
));
|
|
}
|
|
}
|
|
Verdict::NotApplicable
|
|
}
|
|
|
|
fn looks_like_marker_attempt(line: &str) -> bool {
|
|
const KINDS: &[&str] = &[
|
|
"TASK:", "WORK:", "HANDOFF:", "TEST_PASS:", "TEST_FAIL:", "REVIEW_APPROVE:",
|
|
"REVIEW_BLOCK:", "COMPLETED:", "PLAN_COMPLETE:",
|
|
];
|
|
let t = line.trim().trim_start_matches(['*', '#', '-', '`', ' ']);
|
|
KINDS.iter().any(|k| t.starts_with(k))
|
|
}
|
|
|
|
/// The mission checkout is `/mission/repo`. Working anywhere else is not
|
|
/// delivered.
|
|
///
|
|
/// Scored as a BOUNDARY rather than compliance: the skill's positive
|
|
/// instruction ("cd there at the start of every substantive turn") has no
|
|
/// reliable trace in the output, but writing source somewhere that is never
|
|
/// collected does, and it is the failure that costs a whole phase.
|
|
///
|
|
/// This check exists because the skill itself was wrong. It taught
|
|
/// `/workspace/repo` — a path the platform does not mount — while the same
|
|
/// prompt told the agent `/mission/repo`. An agent that obeyed the skill wrote
|
|
/// into a directory nothing collects. Corrected 2026-08-19, and
|
|
/// `skills_loader::contradiction_tests` now holds it.
|
|
fn workspace_boundary(ev: &Evidence<'_>) -> Verdict {
|
|
// The recorded writes settle this directly. Before the tool tap the only
|
|
// available evidence was the narrative, which answers a different question:
|
|
// an agent that wrote to the wrong root without narrating it scored a
|
|
// clean pass.
|
|
let stray: Vec<&str> = ev
|
|
.writes()
|
|
.filter_map(|w| w.path.as_deref())
|
|
.filter(|p| out_of_bounds(p))
|
|
.collect();
|
|
if let Some(first) = stray.first() {
|
|
return Verdict::Fail(format!(
|
|
"wrote to {first} ({} write(s) outside the checkout) — the mission \
|
|
checkout is /mission/repo, so anything written there is never \
|
|
collected and the phase delivers nothing",
|
|
stray.len()
|
|
));
|
|
}
|
|
|
|
// `--force` is the skill's other hard prohibition, and unlike the path rule
|
|
// it leaves no trace anywhere else: the push succeeds and the history it
|
|
// overwrote is gone.
|
|
if let Some(cmd) = ev.commands().find(|c| is_force_push(c)) {
|
|
return Verdict::Fail(format!(
|
|
"force-pushed ({cmd:?}) — the skill allows it only after an explicit \
|
|
`HANDOFF: safe to force-push` from the reviewer, because rewriting \
|
|
pushed history is the one thing here that cannot be undone"
|
|
));
|
|
}
|
|
|
|
// No violation seen. Whether that is a pass depends on whether there was
|
|
// anything to see.
|
|
if ev.writes().next().is_some() {
|
|
return Verdict::Pass;
|
|
}
|
|
if ev.text.contains("/workspace/repo") {
|
|
return Verdict::Fail(
|
|
"the narrative describes working in /workspace/repo — the mission \
|
|
checkout is /mission/repo, so anything written there is never \
|
|
collected and the phase delivers nothing"
|
|
.into(),
|
|
);
|
|
}
|
|
if ev.text.contains("/mission/repo") {
|
|
return Verdict::Pass;
|
|
}
|
|
// Silence is not compliance. This returned `Pass` before, which is how a
|
|
// mission with no evidence at all scored the same as one that was checked.
|
|
Verdict::NotObservable(
|
|
"no file-modifying tool call was recorded and the narrative names no \
|
|
checkout — there is nothing here to place either inside or outside \
|
|
/mission/repo"
|
|
.into(),
|
|
)
|
|
}
|
|
|
|
/// Is this write outside the mission checkout?
|
|
///
|
|
/// Relative paths are skipped: the working directory is not recorded, so
|
|
/// `src/a.rs` could be either, and guessing would fail honest work.
|
|
///
|
|
/// The scratch roots are excluded because the skill itself blesses them —
|
|
/// "anything else is scratch and will not be delivered" — and the hooks, the
|
|
/// stop gate and claude's own state all live under `/root` by design.
|
|
fn out_of_bounds(path: &str) -> bool {
|
|
const SCRATCH: [&str; 5] = ["/tmp/", "/var/tmp/", "/root/", "/dev/", "/proc/"];
|
|
path.starts_with('/')
|
|
&& !path.starts_with("/mission/repo")
|
|
&& !SCRATCH.iter().any(|s| path.starts_with(s))
|
|
}
|
|
|
|
/// Matched on whole tokens, outside the commit message.
|
|
///
|
|
/// Both halves are load-bearing, and each was a live false positive before the
|
|
/// negative control caught it:
|
|
///
|
|
/// - `cmd.contains("-f ")` fires on `--follow-tags`.
|
|
/// - Tokenising the whole command fires on
|
|
/// `git commit -m "document the -f flag" && git push`, where `-f` is prose.
|
|
///
|
|
/// A boundary check that fails honest work gets switched off, which costs more
|
|
/// than the check was worth.
|
|
fn is_force_push(cmd: &str) -> bool {
|
|
const FORCE: [&str; 3] = ["--force", "-f", "--force-with-lease"];
|
|
let stripped = match dash_m_value(cmd) {
|
|
Some(msg) => cmd.replace(&msg, " "),
|
|
None => cmd.to_string(),
|
|
};
|
|
let tokens = || stripped.split_whitespace();
|
|
let pushes = stripped.contains("git") && tokens().any(|t| t == "push");
|
|
pushes && tokens().any(|t| FORCE.contains(&t))
|
|
}
|
|
|
|
/// The commit subjects the skill names as never-merge.
|
|
///
|
|
/// Matched exactly, after stripping an `INT-NN` prefix. A substring match would
|
|
/// fail `fix the parser panic`, which is a perfectly good subject — the skill's
|
|
/// objection is to `fix` as the WHOLE message, not to the word.
|
|
fn commit_subject_boundary(ev: &Evidence<'_>) -> Verdict {
|
|
const NEVER_MERGE: [&str; 7] = [
|
|
"wip",
|
|
"fix",
|
|
"update",
|
|
"updates",
|
|
"update stuff",
|
|
"stuff",
|
|
"wip commit",
|
|
];
|
|
for subject in commit_subjects(ev) {
|
|
let bare = strip_int_prefix(&subject).to_ascii_lowercase();
|
|
let bare = bare.trim_matches(|c: char| !c.is_alphanumeric()).trim();
|
|
if NEVER_MERGE.contains(&bare) {
|
|
return Verdict::Fail(format!(
|
|
"committed {subject:?} — the skill lists this exact message as \
|
|
never-merge, because a reviewer and a bisect both need the \
|
|
subject to stand alone in `git log --oneline`"
|
|
));
|
|
}
|
|
}
|
|
Verdict::NotApplicable
|
|
}
|
|
|
|
fn strip_int_prefix(subject: &str) -> &str {
|
|
let s = subject.trim().trim_start_matches(['<', '[', '(']);
|
|
let Some(rest) = s.strip_prefix("INT-") else {
|
|
return subject.trim();
|
|
};
|
|
let rest = rest.trim_start_matches(|c: char| c.is_ascii_digit());
|
|
rest.trim_start_matches(['>', ']', ')', ':', '-', '—', ' '])
|
|
.trim()
|
|
}
|
|
|
|
/// Did a test ever run, and did it run first?
|
|
///
|
|
/// One-sided by construction, because only one of the three answers is
|
|
/// unambiguous:
|
|
///
|
|
/// - **Files written, no test command anywhere** — red-green did not happen.
|
|
/// There is no reading of the loop under which it did.
|
|
/// - **A test ran before the first write** — that is the RED step, observed.
|
|
/// - **A test ran after the first write** — undecidable, and reported as such.
|
|
/// In Rust the unit test lives in `#[cfg(test)] mod tests` *inside the file
|
|
/// under test, so "wrote the file, then ran the test" is what writing the
|
|
/// failing test first looks like from here. Scoring that as a failure would
|
|
/// fail the agents who followed the skill most precisely.
|
|
/// Did this test run pass?
|
|
///
|
|
/// Read from the runner's own verdict line, not from an exit code — Claude
|
|
/// Code's `tool_response` carries `stdout`/`stderr`/`interrupted` and no
|
|
/// status. That is why `bounded_response` keeps the END of the output: every
|
|
/// one of these markers is printed last.
|
|
fn test_passed(response: &serde_json::Value) -> Option<bool> {
|
|
let text = format!(
|
|
"{}\n{}",
|
|
response.get("stdout").and_then(|v| v.as_str()).unwrap_or(""),
|
|
response.get("stderr").and_then(|v| v.as_str()).unwrap_or("")
|
|
);
|
|
if text.trim().is_empty() {
|
|
return None;
|
|
}
|
|
const FAILED: [&str; 6] = [
|
|
"test result: FAILED",
|
|
"FAILED (",
|
|
"error: test failed",
|
|
"Tests: ",
|
|
"failures:",
|
|
"error[E",
|
|
];
|
|
const PASSED: [&str; 4] = ["test result: ok", "ok. ", "passed", "PASS"];
|
|
// Failure first: a run that reports both a passing binary and a failing one
|
|
// is a failing run, and the summary line of the failure is what matters.
|
|
if FAILED.iter().any(|m| text.contains(m)) {
|
|
return Some(false);
|
|
}
|
|
if PASSED.iter().any(|m| text.contains(m)) {
|
|
return Some(true);
|
|
}
|
|
None
|
|
}
|
|
|
|
fn red_before_green(ev: &Evidence<'_>) -> Verdict {
|
|
if ev.tools.is_empty() {
|
|
return Verdict::NotObservable(
|
|
"no tool calls were recorded for this mission — the order of test \
|
|
runs and writes is what this check reads, and neither is in the \
|
|
narrative"
|
|
.into(),
|
|
);
|
|
}
|
|
let first_write = ev.tools.iter().position(|t| {
|
|
WRITE_TOOLS.contains(&t.tool.as_str()) && t.path.as_deref().is_some_and(is_source)
|
|
});
|
|
let first_test = ev
|
|
.tools
|
|
.iter()
|
|
.position(|t| t.command().is_some_and(is_test_command));
|
|
match (first_write, first_test) {
|
|
// Nothing was written, so there was no implementation to test-drive.
|
|
(None, _) => Verdict::NotApplicable,
|
|
(Some(_), None) => Verdict::Fail(format!(
|
|
"wrote {} source file(s) and never ran a test — the loop is red, \
|
|
green, refactor, and a test that never ran cannot have been red",
|
|
ev.writes()
|
|
.filter(|w| w.path.as_deref().is_some_and(is_source))
|
|
.count()
|
|
)),
|
|
(Some(w), Some(t)) if t < w => Verdict::Pass,
|
|
_ => {
|
|
// Ordering is exhausted here, and so is the repository: the skill
|
|
// says to "commit the RED-to-GREEN pair as one commit", so the
|
|
// failing test and its fix land together by instruction. The only
|
|
// remaining witness is what the runs THEMSELVES reported.
|
|
let runs = ev.test_runs();
|
|
let went_red = runs.iter().position(|r| *r == Some(false));
|
|
let went_green = runs.iter().rposition(|r| *r == Some(true));
|
|
match (went_red, went_green) {
|
|
(Some(red), Some(green)) if red < green => Verdict::Pass,
|
|
(Some(_), None) => Verdict::Fail(
|
|
"every recorded test run failed — the loop ends on green, \
|
|
and this phase never got there"
|
|
.into(),
|
|
),
|
|
_ if runs.iter().all(Option::is_none) => Verdict::NotObservable(
|
|
"a test ran after the first write and no run recorded its \
|
|
output — in Rust the unit test lives in the file under \
|
|
test, so ordering cannot separate red-first from \
|
|
tests-added-after, and without the run's own verdict there \
|
|
is nothing else to read"
|
|
.into(),
|
|
),
|
|
_ => Verdict::NotObservable(
|
|
"every recorded test run passed — a test that never failed \
|
|
is a test that was never red, but it is equally what a \
|
|
correct implementation written first looks like, and these \
|
|
runs cannot tell them apart"
|
|
.into(),
|
|
),
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Is this file behaviour-adding code, in the languages the skill names?
|
|
///
|
|
/// The check is scoped to source because the first live run scored `fail`
|
|
/// against a **research** phase: its agents wrote fifteen markdown notes and
|
|
/// never ran a test, which is not a violation of anything — there was no code
|
|
/// to test-drive. Reporting that as an agent failure would be a system defect
|
|
/// wearing an agent's name, and the real finding it obscures is that a research
|
|
/// role is pinned TDD skills at all.
|
|
///
|
|
/// Extensions the skill itself names — "Rust, TypeScript, Python, anywhere
|
|
/// tests can run cheap" — plus their immediate neighbours. Prose, config and
|
|
/// data are excluded. Shell is excluded deliberately: a helper script written
|
|
/// during a research turn is not the behaviour-adding code this loop is about,
|
|
/// and the false failure costs more than the missed one.
|
|
fn is_source(path: &str) -> bool {
|
|
const SOURCE: [&str; 10] = [
|
|
".rs", ".ts", ".tsx", ".py", ".js", ".jsx", ".go", ".java", ".rb", ".kt",
|
|
];
|
|
let p = path.to_ascii_lowercase();
|
|
SOURCE.iter().any(|e| p.ends_with(e))
|
|
}
|
|
|
|
/// Commands that run a test suite, across the languages the skills name.
|
|
fn is_test_command(cmd: &str) -> bool {
|
|
const RUNNERS: [&str; 9] = [
|
|
"cargo test",
|
|
"cargo nextest",
|
|
"cargo llvm-cov",
|
|
"npm test",
|
|
"yarn test",
|
|
"pnpm test",
|
|
"vitest",
|
|
"pytest",
|
|
"go test",
|
|
];
|
|
// `jest` on its own is too short to match safely — it appears inside
|
|
// package names and paths — so it is matched as a command word.
|
|
RUNNERS.iter().any(|r| cmd.contains(r))
|
|
|| cmd.split_whitespace().any(|w| w == "jest" || w == "npx")
|
|
&& cmd.contains("jest")
|
|
}
|
|
|
|
/// The subject line of every commit the agent made.
|
|
///
|
|
/// Read out of the `-m` argument. A commit written with `-F` or a heredoc
|
|
/// yields nothing, and the caller must treat an empty result as "no evidence"
|
|
/// rather than "no commits" — which is why every commit check here returns
|
|
/// `NotApplicable` on an empty list instead of a verdict.
|
|
fn commit_subjects(ev: &Evidence<'_>) -> Vec<String> {
|
|
ev.commands()
|
|
.filter(|c| c.contains("git commit") || c.contains("git ci"))
|
|
.filter_map(dash_m_value)
|
|
.filter_map(|m| subject_line(&m))
|
|
.collect()
|
|
}
|
|
|
|
/// The subject out of a `-m` value.
|
|
///
|
|
/// Normally the first line. But Claude Code writes a multi-line message as
|
|
///
|
|
/// ```text
|
|
/// git commit -m "$(cat <<'EOF'
|
|
/// INT-01 Add slugify function
|
|
///
|
|
/// …body…
|
|
/// EOF
|
|
/// )"
|
|
/// ```
|
|
///
|
|
/// and the first line of that value is `$(cat <<'EOF'` — the heredoc *opener*,
|
|
/// not the subject. Observed on the first live coding run; it scored no
|
|
/// violation only because `$(cat <<'EOF'` happens not to be one of the
|
|
/// never-merge messages, which is luck, not a check.
|
|
///
|
|
/// So: if the first line opens a heredoc or a command substitution, the subject
|
|
/// is the next non-empty line.
|
|
fn subject_line(msg: &str) -> Option<String> {
|
|
let mut lines = msg.lines().map(str::trim).filter(|l| !l.is_empty());
|
|
let first = lines.next()?;
|
|
if first.starts_with("$(") || first.contains("<<") {
|
|
return lines.next().map(str::to_string);
|
|
}
|
|
Some(first.to_string())
|
|
}
|
|
|
|
/// The value of the `-m` flag in a shell command.
|
|
///
|
|
/// Hand-scanned rather than split on whitespace: the value is the one argument
|
|
/// guaranteed to contain spaces, and the commit template in
|
|
/// `workspace-repo-commit-protocol` spans several lines.
|
|
fn dash_m_value(cmd: &str) -> Option<String> {
|
|
let b = cmd.as_bytes();
|
|
for i in 0..b.len() {
|
|
// Anchor on the `m`, not on `-m`: agents write `git commit -am …` and
|
|
// anchoring on the dash misses the whole cluster form.
|
|
if b[i] != b'm' {
|
|
continue;
|
|
}
|
|
// `m` must END the cluster, so `--message` and `-mtime` are not it.
|
|
if !matches!(
|
|
b.get(i + 1),
|
|
None | Some(b' ') | Some(b'\t') | Some(b'"') | Some(b'\'')
|
|
) {
|
|
continue;
|
|
}
|
|
// Walk back over the cluster's other short flags to its single dash.
|
|
let mut j = i;
|
|
while j > 0 && b[j - 1].is_ascii_alphabetic() {
|
|
j -= 1;
|
|
}
|
|
if j == 0 || b[j - 1] != b'-' {
|
|
continue;
|
|
}
|
|
let dash = j - 1;
|
|
// One dash, starting a word — not `--m` and not the tail of a path.
|
|
if dash > 0 && !b[dash - 1].is_ascii_whitespace() {
|
|
continue;
|
|
}
|
|
let mut k = i + 1;
|
|
while k < b.len() && b[k].is_ascii_whitespace() {
|
|
k += 1;
|
|
}
|
|
if k >= b.len() {
|
|
return None;
|
|
}
|
|
let rest = &cmd[k..];
|
|
let quote = rest.chars().next().filter(|c| *c == '"' || *c == '\'');
|
|
return match quote {
|
|
Some(q) => rest[1..].split(q).next().map(str::to_string),
|
|
None => rest.split_whitespace().next().map(str::to_string),
|
|
};
|
|
}
|
|
None
|
|
}
|
|
|
|
/// The tools whose arguments carry a URL the agent reached for.
|
|
///
|
|
/// `Agent` is here on purpose. On both `files`-arm runs the parent decomposed
|
|
/// the sweep into per-source fetches and sent each to a subagent — the URLs
|
|
/// live in the spawn PROMPT, and a check that only read `curl` lines would
|
|
/// have scored those runs as fetching nothing.
|
|
const FETCH_TOOLS: &[&str] = &["Bash", "Agent", "WebFetch", "web_fetch"];
|
|
|
|
/// Every `http(s)` URL in the arguments of a fetching tool, in call order.
|
|
fn fetched_urls(ev: &Evidence<'_>) -> Vec<String> {
|
|
let mut out = Vec::new();
|
|
for t in ev.tools {
|
|
if !FETCH_TOOLS.contains(&t.tool.as_str()) {
|
|
continue;
|
|
}
|
|
// A `Bash` that never fetches is most of what agents run.
|
|
if t.tool == "Bash"
|
|
&& !t
|
|
.command()
|
|
.is_some_and(|c| c.contains("curl") || c.contains("wget"))
|
|
{
|
|
continue;
|
|
}
|
|
let text = t.input.to_string();
|
|
let mut rest = text.as_str();
|
|
while let Some(i) = rest.find("http") {
|
|
let cand = &rest[i..];
|
|
let end = cand
|
|
.find(|c: char| {
|
|
c.is_whitespace() || matches!(c, '"' | '\'' | '<' | '>' | ')' | ']' | '\\')
|
|
})
|
|
.unwrap_or(cand.len());
|
|
let url = cand[..end].trim_end_matches(|c| matches!(c, '.' | ',' | ';' | ':'));
|
|
if url.starts_with("http://") || url.starts_with("https://") {
|
|
out.push(url.to_string());
|
|
}
|
|
rest = &cand[end.max(4)..];
|
|
}
|
|
}
|
|
out
|
|
}
|
|
|
|
/// Rank 0 in `web-search-triage`'s ladder: the paper, the spec, the release.
|
|
///
|
|
/// A conservative allow-list. Anything not on it is UNRANKED, not rank 3 — a
|
|
/// vendor's docs, an author's own blog and a lab page are all rank 0 or 1 and
|
|
/// none of them can be recognised by hostname.
|
|
fn is_primary_source(url: &str) -> bool {
|
|
const HOSTS: &[&str] = &[
|
|
"arxiv.org/abs/",
|
|
"arxiv.org/pdf/",
|
|
"doi.org/",
|
|
"aclanthology.org/",
|
|
"openreview.net/",
|
|
"proceedings.neurips.cc/",
|
|
"proceedings.mlr.press/",
|
|
"dl.acm.org/doi/",
|
|
"ieeexplore.ieee.org/",
|
|
"github.com/",
|
|
"nature.com/articles/",
|
|
"science.org/doi/",
|
|
];
|
|
HOSTS.iter().any(|h| url.contains(h))
|
|
}
|
|
|
|
/// Rank 3 and below: a restatement of a restatement. The skill's own words are
|
|
/// "the same item and should be recorded once, if at all", and its skip signals
|
|
/// — "a numbered list of tools", no date — describe these hosts.
|
|
///
|
|
/// Also conservative. Substack, X and personal blogs are NOT here: an author's
|
|
/// own post is rank 1 and the skill says to read it.
|
|
fn is_aggregator(url: &str) -> bool {
|
|
const HOSTS: &[&str] = &[
|
|
"medium.com/",
|
|
"towardsdatascience.com/",
|
|
"reddit.com/",
|
|
"news.ycombinator.com/",
|
|
"quora.com/",
|
|
"dev.to/",
|
|
"linkedin.com/",
|
|
"wikipedia.org/",
|
|
];
|
|
HOSTS.iter().any(|h| url.contains(h))
|
|
}
|
|
|
|
/// `web-search-triage`: read the primary source, and treat undated as a finding.
|
|
///
|
|
/// Two of the skill's rules leave a mark in the recorded ARGUMENTS, and this
|
|
/// scores exactly those two — the rest of the skill is judgement about page
|
|
/// content the tap never sees, and a heuristic over it would be a number that
|
|
/// looks like a measurement and is not one.
|
|
///
|
|
/// - The ranking rule. Every URL a fetch was sent to is classified against a
|
|
/// short allow-list of primary hosts and a short skip-list of aggregators.
|
|
/// Fetching an aggregator is the visible violation; fetching primary sources
|
|
/// is the visible compliance. Anything unrecognised is unranked and decides
|
|
/// nothing.
|
|
/// - The date rule. On mission `01a09b42` the parent's spawn prompts read
|
|
/// "Return the URL, date if visible, and the key content" — the task never
|
|
/// asked for a date; the skill's "undated is a finding" did. Reported as
|
|
/// extra evidence on a pass, never required for one: a curl to an abstract
|
|
/// page has no prompt to ask in.
|
|
///
|
|
/// One-sided like every check here: it reports a violation it can see and never
|
|
/// infers compliance from silence.
|
|
fn triage_compliance(ev: &Evidence<'_>) -> Verdict {
|
|
if ev.tools.is_empty() {
|
|
return Verdict::NotObservable(
|
|
"no tool calls were recorded for this mission — which URLs were \
|
|
fetched is what this check reads, and that is not in the narrative"
|
|
.into(),
|
|
);
|
|
}
|
|
let urls = fetched_urls(ev);
|
|
if urls.is_empty() {
|
|
// Never swept the web, so there was nothing to triage.
|
|
return Verdict::NotApplicable;
|
|
}
|
|
if let Some(u) = urls.iter().find(|u| is_aggregator(u)) {
|
|
return Verdict::Fail(format!(
|
|
"fetched {u} — an aggregator, rank 3 or below on the skill's ladder; \
|
|
the procedure is to find the primary source and record the rest as \
|
|
one item, not to read them"
|
|
));
|
|
}
|
|
let primary = urls.iter().filter(|u| is_primary_source(u)).count();
|
|
if primary == 0 {
|
|
return Verdict::NotObservable(format!(
|
|
"fetched {} URL(s), none on the primary-source list and none on the \
|
|
aggregator list — the check cannot rank them, and a rank it cannot \
|
|
see is not a violation",
|
|
urls.len()
|
|
));
|
|
}
|
|
let asked_for_date = ev
|
|
.tools
|
|
.iter()
|
|
.filter(|t| t.tool == "Agent")
|
|
.filter(|t| {
|
|
t.input
|
|
.get("prompt")
|
|
.and_then(serde_json::Value::as_str)
|
|
.is_some_and(|p| p.to_ascii_lowercase().contains("date"))
|
|
})
|
|
.count();
|
|
let mut why = format!(
|
|
"fetched {primary} primary source(s) out of {} URL(s) and no aggregator",
|
|
urls.len()
|
|
);
|
|
if asked_for_date > 0 {
|
|
why.push_str(&format!(
|
|
"; {asked_for_date} fetch(es) delegated to a subagent asked for the \
|
|
page's date, which the task did not and the skill does"
|
|
));
|
|
}
|
|
Verdict::PassWith(why)
|
|
}
|
|
|
|
/// `arxiv-daily` forbids searching arXiv — the harvest already ran.
|
|
///
|
|
/// This is the one boundary we have watched an agent cross in production, so it
|
|
/// is checked against the query endpoint rather than the word "arxiv", which
|
|
/// appears legitimately all over a research turn.
|
|
fn arxiv_boundary(ev: &Evidence<'_>) -> Verdict {
|
|
// The query endpoint, not the word "arxiv" — and not `arxiv.org/abs/`
|
|
// either, which the same skill explicitly tells the agent to `curl`. A
|
|
// check that could not tell those apart would fail agents for following
|
|
// the paragraph directly under the one it is scoring.
|
|
const QUERY_MARKERS: &[&str] = &[
|
|
"export.arxiv.org/api/query",
|
|
"arxiv.org/api/query",
|
|
"http://export.arxiv.org",
|
|
];
|
|
let hit = |s: &str| QUERY_MARKERS.iter().find(|m| s.contains(**m)).copied();
|
|
|
|
// Arguments first: a `curl` that ran is a fact, where a narrative that
|
|
// mentions a URL may be the agent explaining that it did NOT fetch it.
|
|
for t in ev.tools {
|
|
let text = t.input.to_string();
|
|
if let Some(m) = hit(&text) {
|
|
return Verdict::Fail(format!(
|
|
"called {} against the arXiv API ({m}) — the harvest already \
|
|
ran, and shelving papers outside it corrupts the seen-set",
|
|
t.tool
|
|
));
|
|
}
|
|
}
|
|
if let Some(m) = hit(ev.text) {
|
|
return Verdict::Fail(format!(
|
|
"queried the arXiv API ({m}) — the harvest already ran, and \
|
|
shelving papers outside it corrupts the seen-set"
|
|
));
|
|
}
|
|
Verdict::Pass
|
|
}
|
|
|
|
/// Score every skill delivered during a mission, from what was recorded.
|
|
///
|
|
/// Reads `prompt.composed` and `reasoning` rows. Both are per-mission and
|
|
/// ordered, so the prompts say what was delivered and the narratives say what
|
|
/// came back. Scoring is against the CONCATENATED output for the mission rather
|
|
/// than turn-by-turn: a procedure can be followed in a later turn than the one
|
|
/// that carried it, and pairing strictly by turn would score that as a failure.
|
|
///
|
|
/// Returns an empty vec for a mission whose events have already been reaped —
|
|
/// which is why `missions.retain_events_until` exists. An empty result means
|
|
/// "no evidence", never "no compliance", and the report has to say so.
|
|
pub async fn score_mission(
|
|
pool: &sqlx::PgPool,
|
|
mission_id: uuid::Uuid,
|
|
) -> Result<Vec<SkillUse>, String> {
|
|
let rows = crate::mission_events::narrative_for_mission(pool, mission_id)
|
|
.await
|
|
.map_err(|e| format!("read narrative: {e}"))?;
|
|
|
|
let mut prompts = String::new();
|
|
let mut outputs = String::new();
|
|
for (kind, _, _, text) in &rows {
|
|
if kind == crate::mission_events::PROMPT_COMPOSED {
|
|
prompts.push_str(text);
|
|
prompts.push('\n');
|
|
} else {
|
|
outputs.push_str(text);
|
|
outputs.push('\n');
|
|
}
|
|
}
|
|
|
|
// What the agent DID, alongside what it said. A mission that ran before
|
|
// the tool tap existed returns an empty list, and the checks degrade to
|
|
// the narrative rather than reporting a violation they cannot see.
|
|
let tools = crate::mission_events::tool_evidence_for_mission(pool, mission_id)
|
|
.await
|
|
.map_err(|e| format!("read tool evidence: {e}"))?;
|
|
|
|
// One lookup for every delivered name, so the report can separate
|
|
// hand-authored skills from the ones agents wrote for themselves.
|
|
let names = skills_in_prompt(&prompts);
|
|
let kinds: std::collections::HashMap<String, String> = sqlx::query_as::<_, (String, String)>(
|
|
"SELECT name, source_kind FROM skills WHERE name = ANY($1)",
|
|
)
|
|
.bind(&names)
|
|
.fetch_all(pool)
|
|
.await
|
|
.map_err(|e| format!("read skill sources: {e}"))?
|
|
.into_iter()
|
|
.collect();
|
|
|
|
Ok(score(&prompts, &Evidence::new(&outputs, &tools), &|name| {
|
|
kinds
|
|
.get(name)
|
|
.cloned()
|
|
// A skill in a prompt with no catalogue row was delivered and then
|
|
// deleted. Naming that explicitly beats defaulting it to builtin.
|
|
.unwrap_or_else(|| "unknown (no catalogue row)".to_string())
|
|
}))
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
use crate::mission_events::ToolEvidence;
|
|
use serde_json::json;
|
|
|
|
fn builtin(_: &str) -> String {
|
|
"builtin".to_string()
|
|
}
|
|
|
|
/// Narrative-only evidence — a mission whose events predate the tool tap.
|
|
fn narrative(text: &str) -> Evidence<'_> {
|
|
Evidence::from_text(text)
|
|
}
|
|
|
|
/// Evidence from recorded ACTIONS. `(tool, path, input)`.
|
|
fn acted(calls: &[(&str, Option<&str>, serde_json::Value)]) -> Vec<ToolEvidence> {
|
|
calls
|
|
.iter()
|
|
.map(|(tool, path, input)| ToolEvidence {
|
|
tool: (*tool).to_string(),
|
|
path: path.map(str::to_string),
|
|
input: input.clone(),
|
|
response: serde_json::Value::Null,
|
|
})
|
|
.collect()
|
|
}
|
|
|
|
/// A `Bash` call that ran a test and reported an outcome.
|
|
fn test_run(cmd: &str, stdout: &str) -> ToolEvidence {
|
|
ToolEvidence {
|
|
tool: "Bash".into(),
|
|
path: None,
|
|
input: json!({ "command": cmd }),
|
|
response: json!({ "stdout": stdout, "stderr": "" }),
|
|
}
|
|
}
|
|
|
|
/// One `Bash` call.
|
|
fn ran(cmd: &str) -> (&'static str, Option<&'static str>, serde_json::Value) {
|
|
("Bash", None, serde_json::json!({ "command": cmd }))
|
|
}
|
|
|
|
/// A prompt exactly as the delivery layer renders it.
|
|
fn rendered(skills: &[(&str, &str)]) -> String {
|
|
let body: String = skills
|
|
.iter()
|
|
.map(|(n, b)| crate::topology_exec::render_pinned_skill(n, b))
|
|
.collect();
|
|
crate::topology_exec::compose_turn_prompt(
|
|
"Task: do the thing",
|
|
Some(&body),
|
|
crate::skill_delivery::Mode::Inline,
|
|
)
|
|
}
|
|
|
|
/// A prompt as the delivery layer renders it under the INDEX arm.
|
|
fn rendered_index(skills: &[(&str, &str)]) -> String {
|
|
let body: String = skills
|
|
.iter()
|
|
.map(|(name, when)| {
|
|
crate::topology_exec::render_pinned_skill(
|
|
name,
|
|
&crate::skill_delivery::index_entry(
|
|
"a procedure",
|
|
Some(when),
|
|
&format!("skill:global/{name}"),
|
|
),
|
|
)
|
|
})
|
|
.collect();
|
|
crate::topology_exec::compose_turn_prompt(
|
|
"Task: do the thing",
|
|
Some(&body),
|
|
crate::skill_delivery::Mode::Index,
|
|
)
|
|
}
|
|
|
|
fn read_skill(uri: &str) -> ToolEvidence {
|
|
ToolEvidence {
|
|
tool: "ReadMcpResourceTool".into(),
|
|
path: None,
|
|
input: json!({ "server": "clawmates_skills", "uri": uri }),
|
|
response: serde_json::Value::Null,
|
|
}
|
|
}
|
|
|
|
fn rendered_files(skills: &[(&str, &str)]) -> String {
|
|
let body: String = skills
|
|
.iter()
|
|
.map(|(name, when)| {
|
|
crate::topology_exec::render_pinned_skill(
|
|
name,
|
|
&crate::skill_delivery::file_entry(
|
|
"a procedure",
|
|
Some(when),
|
|
&crate::skill_delivery::skill_file_path(name),
|
|
),
|
|
)
|
|
})
|
|
.collect();
|
|
format!(
|
|
"Task: x\n\n# Your skills\n\n{}\n{body}",
|
|
crate::skill_delivery::FILES_PREAMBLE
|
|
)
|
|
}
|
|
|
|
fn read_file(path: &str) -> ToolEvidence {
|
|
ToolEvidence {
|
|
tool: "Read".into(),
|
|
path: Some(path.into()),
|
|
input: json!({ "file_path": path }),
|
|
response: serde_json::Value::Null,
|
|
}
|
|
}
|
|
|
|
/// The `files` arm's loop, end to end, for the same reason as the uri
|
|
/// test below: `skill_file_path` writes the path, `file_entry` puts it in
|
|
/// the prompt, `skill_from_file_path` reads it back off a `Read`.
|
|
#[test]
|
|
fn the_path_the_files_arm_advertises_is_the_one_the_scorer_recovers() {
|
|
let prompt = rendered_files(&[("workspace-repo-commit-protocol", "before committing")]);
|
|
assert!(
|
|
prompt.contains("Read(file_path=\"/mission/skills/workspace-repo-commit-protocol.md\")"),
|
|
"the entry must name the file to read:\n{prompt}"
|
|
);
|
|
assert_eq!(
|
|
crate::skill_delivery::mode_in_prompt(&prompt),
|
|
crate::skill_delivery::Mode::Files
|
|
);
|
|
let tools = vec![read_file("/mission/skills/workspace-repo-commit-protocol.md")];
|
|
let ev = Evidence::new("", &tools);
|
|
assert_eq!(retrieved_skills(&ev), vec!["workspace-repo-commit-protocol"]);
|
|
let scored = score(&prompt, &ev, &builtin);
|
|
assert_eq!(scored.len(), 1);
|
|
assert!(
|
|
matches!(scored[0].trigger, Verdict::Pass),
|
|
"a Read of the advertised path IS the Trigger axis: {:?}",
|
|
scored[0].trigger
|
|
);
|
|
}
|
|
|
|
/// Agents read files constantly. Only a `Read` INSIDE the skills directory
|
|
/// is a retrieval; anything else scoring as one would make Trigger a count
|
|
/// of file reads.
|
|
#[test]
|
|
fn an_ordinary_read_is_not_a_retrieval() {
|
|
let tools = vec![
|
|
read_file("/mission/repo/research/REPORT.md"),
|
|
read_file("/mission/skills"),
|
|
read_file("/mission/skills/nested/x.md"),
|
|
read_file("/etc/passwd"),
|
|
];
|
|
assert!(retrieved_skills(&Evidence::new("", &tools)).is_empty());
|
|
}
|
|
|
|
/// Under `files`, never opened is a miss when the skill had a checkable
|
|
/// consequence — the same rule as `index`, and NOT `inline`'s
|
|
/// "not observable", which would report the arm's own defect as a blind spot.
|
|
#[test]
|
|
fn the_files_arm_scores_a_miss_like_the_index_arm() {
|
|
let prompt = rendered_files(&[("int-xx-marker-protocol", "when writing markers")]);
|
|
let tools: Vec<ToolEvidence> = vec![];
|
|
let ev = Evidence::new("INT-01 something without the required shape", &tools);
|
|
let scored = score(&prompt, &ev, &builtin);
|
|
assert_eq!(scored.len(), 1);
|
|
assert!(
|
|
!matches!(scored[0].trigger, Verdict::NotObservable(_)),
|
|
"files is a retrieval arm; a miss must not read as inline: {:?}",
|
|
scored[0].trigger
|
|
);
|
|
}
|
|
|
|
fn bash(cmd: &str) -> ToolEvidence {
|
|
ToolEvidence {
|
|
tool: "Bash".into(),
|
|
path: None,
|
|
input: json!({ "command": cmd }),
|
|
response: serde_json::Value::Null,
|
|
}
|
|
}
|
|
|
|
fn spawn(prompt: &str) -> ToolEvidence {
|
|
ToolEvidence {
|
|
tool: "Agent".into(),
|
|
path: None,
|
|
input: json!({ "prompt": prompt, "subagent_type": "general-purpose" }),
|
|
response: serde_json::Value::Null,
|
|
}
|
|
}
|
|
|
|
/// The shape mission 01a09b42 recorded: the parent read the skill, then
|
|
/// sent each primary source to a subagent and asked for the date — which
|
|
/// the task never did and the skill's "undated is a finding" does.
|
|
#[test]
|
|
fn triage_passes_on_primary_sources_and_reports_the_date_fingerprint() {
|
|
let tools = vec![
|
|
spawn(
|
|
"Fetch the following URLs and return their full text content. Return the \
|
|
URL, date if visible, and the key content.\n1. https://arxiv.org/abs/2309.15217 \
|
|
(RAGAS paper)\n2. https://arxiv.org/abs/2311.09476",
|
|
),
|
|
bash("curl -s \"https://arxiv.org/abs/2309.01431\" | head -200"),
|
|
];
|
|
let ev = Evidence::new("", &tools);
|
|
match triage_compliance(&ev) {
|
|
Verdict::PassWith(why) => {
|
|
assert!(why.contains("3 primary source(s)"), "{why}");
|
|
assert!(why.contains("asked for the page's date"), "{why}");
|
|
}
|
|
other => panic!("expected a pass with evidence, got {other:?}"),
|
|
}
|
|
}
|
|
|
|
/// The shape the `index` runs recorded — inline curls, no delegation. The
|
|
/// agent still read primary sources, so it still complied; the date
|
|
/// fingerprint is extra evidence, never a requirement.
|
|
#[test]
|
|
fn triage_passes_on_inline_curls_without_the_date_clause() {
|
|
let tools = vec![
|
|
bash("curl -sL https://arxiv.org/abs/2204.04745"),
|
|
bash("ls -la research/"),
|
|
];
|
|
match triage_compliance(&Evidence::new("", &tools)) {
|
|
Verdict::PassWith(why) => assert!(!why.contains("date"), "{why}"),
|
|
other => panic!("{other:?}"),
|
|
}
|
|
}
|
|
|
|
/// The one violation the check can see: reading a restatement of a
|
|
/// restatement instead of the source it restates.
|
|
#[test]
|
|
fn triage_fails_on_an_aggregator() {
|
|
let tools = vec![
|
|
bash("curl -s https://arxiv.org/abs/2309.15217"),
|
|
spawn("Fetch https://medium.com/@someone/rag-eval-explained-2024 and summarise"),
|
|
];
|
|
match triage_compliance(&Evidence::new("", &tools)) {
|
|
Verdict::Fail(why) => assert!(why.contains("medium.com"), "{why}"),
|
|
other => panic!("{other:?}"),
|
|
}
|
|
}
|
|
|
|
/// One-sided, both ways. Nothing fetched is nothing to triage; something
|
|
/// fetched that the lists cannot rank is unranked, not a violation.
|
|
#[test]
|
|
fn triage_is_silent_where_it_cannot_see() {
|
|
let none: Vec<ToolEvidence> = vec![];
|
|
assert!(matches!(
|
|
triage_compliance(&Evidence::new("", &none)),
|
|
Verdict::NotObservable(_)
|
|
));
|
|
let no_fetch = vec![bash("cargo test"), bash("git status")];
|
|
assert!(matches!(
|
|
triage_compliance(&Evidence::new("", &no_fetch)),
|
|
Verdict::NotApplicable
|
|
));
|
|
let unranked = vec![bash("curl -s https://docs.example-vendor.io/eval/guide")];
|
|
assert!(matches!(
|
|
triage_compliance(&Evidence::new("", &unranked)),
|
|
Verdict::NotObservable(_)
|
|
));
|
|
}
|
|
|
|
/// A URL inside JSON is followed by a quote, and one at the end of a
|
|
/// sentence by a full stop. Neither is part of the URL.
|
|
#[test]
|
|
fn fetched_urls_stop_at_the_right_character() {
|
|
let tools = vec![spawn(
|
|
"Fetch \"https://arxiv.org/abs/1\" then https://doi.org/10.1/x. Done.",
|
|
)];
|
|
assert_eq!(
|
|
fetched_urls(&Evidence::new("", &tools)),
|
|
vec!["https://arxiv.org/abs/1", "https://doi.org/10.1/x"]
|
|
);
|
|
}
|
|
|
|
/// `PassWith` must be indistinguishable from `Pass` to a reader keyed on
|
|
/// the tag, or every consumer of the report grows a fourth branch.
|
|
#[test]
|
|
fn a_pass_with_evidence_serialises_under_the_pass_tag() {
|
|
let v = serde_json::to_value(Verdict::PassWith("saw it".into())).unwrap();
|
|
assert_eq!(v["verdict"], "pass");
|
|
assert_eq!(v["why"], "saw it");
|
|
assert_eq!(Verdict::PassWith("x".into()).label(), Verdict::Pass.label());
|
|
}
|
|
|
|
/// The whole loop, end to end: the index writes a uri, the agent reads that
|
|
/// exact uri back, and the scorer recovers the skill's name from it.
|
|
///
|
|
/// Three components have to agree on one string — `mcp_skills::skill_uri`
|
|
/// writes it, `skill_delivery::index_entry` puts it in the prompt, and
|
|
/// `parse_uri` reads it. Asserting them separately would let any pair drift
|
|
/// while each one's own test stayed green.
|
|
#[test]
|
|
fn the_uri_the_index_advertises_is_the_one_the_scorer_recovers() {
|
|
let prompt = rendered_index(&[("workspace-repo-commit-protocol", "before committing")]);
|
|
assert!(
|
|
prompt.contains("uri=\"skill:global/workspace-repo-commit-protocol\""),
|
|
"the entry must name the uri to fetch:\n{prompt}"
|
|
);
|
|
let tools = vec![read_skill("skill:global/workspace-repo-commit-protocol")];
|
|
let ev = Evidence::new("", &tools);
|
|
assert_eq!(
|
|
retrieved_skills(&ev),
|
|
vec!["workspace-repo-commit-protocol"],
|
|
"the uri the prompt advertised must parse back to the skill's name"
|
|
);
|
|
let scored = score(&prompt, &ev, &builtin);
|
|
assert_eq!(scored.len(), 1);
|
|
assert!(
|
|
matches!(scored[0].trigger, Verdict::Pass),
|
|
"reaching for an indexed skill IS the Trigger axis: {:?}",
|
|
scored[0].trigger
|
|
);
|
|
}
|
|
|
|
/// The index arm must not report a miss as a blind spot.
|
|
///
|
|
/// Under `Inline` "never retrieved" is `NotObservable`, and that is honest
|
|
/// there. Reusing it here would say "this skill was inlined into the
|
|
/// prompt" about a skill whose body was never sent — the exact shape of a
|
|
/// check reporting a system defect where an agent behaviour belongs.
|
|
#[test]
|
|
fn an_indexed_skill_that_was_never_read_is_a_miss_not_a_blind_spot() {
|
|
let prompt = rendered_index(&[("workspace-repo-commit-protocol", "before committing")]);
|
|
// It wrote outside the mission checkout, so the boundary check applies
|
|
// — this skill had something to say about this phase and went unread.
|
|
let tools = acted(&[("Write", Some("/tmp/scratch.rs"), json!({}))]);
|
|
let scored = score(&prompt, &Evidence::new("", &tools), &builtin);
|
|
assert!(
|
|
matches!(scored[0].trigger, Verdict::Fail(_)),
|
|
"offered by name and when_to_use, never fetched: {:?}",
|
|
scored[0].trigger
|
|
);
|
|
}
|
|
|
|
/// ...but only when the skill had a consequence to check.
|
|
///
|
|
/// A skill with nothing machine-checkable in this phase is one an agent is
|
|
/// right to pass over, and scoring that as a Trigger failure would punish
|
|
/// correct triage — which is the behaviour progressive disclosure is
|
|
/// supposed to reward.
|
|
#[test]
|
|
fn passing_over_a_skill_with_nothing_to_check_is_not_a_trigger_failure() {
|
|
let prompt = rendered_index(&[("some-unchecked-skill", "when writing prose")]);
|
|
let scored = score(&prompt, &narrative("wrote the report"), &builtin);
|
|
assert!(
|
|
matches!(scored[0].trigger, Verdict::NotApplicable),
|
|
"{:?}",
|
|
scored[0].trigger
|
|
);
|
|
}
|
|
|
|
/// The control arm has to be unchanged, or the A/B measures this edit too.
|
|
/// The case this whole flag exists for.
|
|
///
|
|
/// `workspace-repo-commit-protocol` applies to every agent that writes,
|
|
/// which is exactly why no agent reads it as *theirs* — it scored
|
|
/// Trigger=FAIL beside a passing boundary check on the first A/B pair.
|
|
/// Marked `always_inject`, its body is in the prompt under the index arm,
|
|
/// and a skill the agent was handed cannot be a reaching-for failure.
|
|
#[test]
|
|
fn a_skill_delivered_in_full_under_the_index_arm_is_not_a_trigger_miss() {
|
|
let prompt = format!(
|
|
"{}\n{}{}",
|
|
crate::skill_delivery::INDEX_PREAMBLE,
|
|
crate::topology_exec::render_pinned_skill(
|
|
"workspace-repo-commit-protocol",
|
|
"Commit only inside /workspace/repo. Never write outside it.",
|
|
),
|
|
rendered_index(&[("arxiv-daily", "when sweeping arxiv")]),
|
|
);
|
|
let tools = [];
|
|
let ev = Evidence { text: "", tools: &tools };
|
|
let got = score(&prompt, &ev, &|_| "builtin".to_string());
|
|
let it = got
|
|
.iter()
|
|
.find(|u| u.skill == "workspace-repo-commit-protocol")
|
|
.expect("the always-injected skill must still be scored");
|
|
assert!(
|
|
matches!(it.trigger, Verdict::NotObservable(_)),
|
|
"handed over, not offered — there is no retrieval to miss: {:?}",
|
|
it.trigger
|
|
);
|
|
}
|
|
|
|
/// The flag must not leak: a skill still delivered as an index entry keeps
|
|
/// being scored on whether it was fetched.
|
|
#[test]
|
|
fn an_indexed_skill_in_the_same_prompt_is_still_judged_on_retrieval() {
|
|
let prompt = format!(
|
|
"{}\n{}{}",
|
|
crate::skill_delivery::INDEX_PREAMBLE,
|
|
crate::topology_exec::render_pinned_skill(
|
|
"workspace-repo-commit-protocol",
|
|
"Commit only inside /workspace/repo.",
|
|
),
|
|
rendered_index(&[("arxiv-daily", "when sweeping arxiv")]),
|
|
);
|
|
for u in score(&prompt, &Evidence { text: "", tools: &[] }, &|_| "builtin".into()) {
|
|
if u.skill != "workspace-repo-commit-protocol" {
|
|
assert!(
|
|
!matches!(u.trigger, Verdict::NotObservable(_)),
|
|
"{} was offered by uri, so retrieval is observable for it",
|
|
u.skill
|
|
);
|
|
}
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn the_inline_arm_still_reports_trigger_as_unobservable() {
|
|
let prompt = rendered(&[("workspace-repo-commit-protocol", "body")]);
|
|
let tools = acted(&[("Write", Some("/tmp/scratch.rs"), json!({}))]);
|
|
let scored = score(&prompt, &Evidence::new("", &tools), &builtin);
|
|
assert!(
|
|
matches!(scored[0].trigger, Verdict::NotObservable(_)),
|
|
"{:?}",
|
|
scored[0].trigger
|
|
);
|
|
}
|
|
|
|
/// The prompt is the record of what was delivered, so parsing it must match
|
|
/// exactly what the delivery layer writes.
|
|
#[test]
|
|
fn the_delivered_skills_are_read_back_out_of_the_prompt() {
|
|
let prompt = rendered(&[
|
|
("arxiv-daily", "Do not re-search."),
|
|
("int-xx-marker-protocol", "Emit markers."),
|
|
]);
|
|
assert_eq!(
|
|
skills_in_prompt(&prompt),
|
|
vec!["arxiv-daily", "int-xx-marker-protocol"],
|
|
"the scorer reads what the delivery layer wrote — if these drift, \
|
|
every score is attributed to the wrong skill"
|
|
);
|
|
}
|
|
|
|
/// Skill bodies are markdown and contain their own headings.
|
|
///
|
|
/// The first delimiter was `## <name>`, so every section of every body
|
|
/// counted as a separate skill. Caught on a live mission, which scored
|
|
/// "Sizing heuristic" and "The output shape" — both of them subheadings
|
|
/// inside `decompose-int-items` — as skills with no catalogue row.
|
|
#[test]
|
|
fn headings_inside_a_skill_body_are_not_skills() {
|
|
let body = "# Decomposing into INT-XX items\n\n\
|
|
## Sizing heuristic\nOne unit of work.\n\n\
|
|
## The output shape\nTASK: INT-NN — title\n";
|
|
let prompt = rendered(&[("decompose-int-items", body)]);
|
|
assert_eq!(
|
|
skills_in_prompt(&prompt),
|
|
vec!["decompose-int-items"],
|
|
"a body's own headings must not be counted as skills — every one \
|
|
would be scored against a catalogue row that does not exist"
|
|
);
|
|
}
|
|
|
|
/// A body that quotes the marker text must not fabricate a skill either.
|
|
#[test]
|
|
fn a_body_mentioning_the_marker_does_not_create_a_skill() {
|
|
let body = format!(
|
|
"Prompts introduce a skill with a line beginning `{}`.\n",
|
|
crate::topology_exec::SKILL_MARKER.trim()
|
|
);
|
|
let prompt = rendered(&[("prompt-anatomy", &body)]);
|
|
assert_eq!(
|
|
skills_in_prompt(&prompt),
|
|
vec!["prompt-anatomy"],
|
|
"the marker is matched at line start; an inline mention is prose"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn a_prompt_with_no_skills_yields_no_scores() {
|
|
assert!(skills_in_prompt("Task: do the thing").is_empty());
|
|
assert!(score("Task: do the thing", &narrative("did it"), &builtin).is_empty());
|
|
}
|
|
|
|
#[test]
|
|
fn trigger_is_reported_as_unobservable_not_as_a_failure() {
|
|
let prompt = rendered(&[("arxiv-daily", "x")]);
|
|
let scored = score(&prompt, &narrative("I read the manifest."), &builtin);
|
|
assert!(
|
|
matches!(scored[0].trigger, Verdict::NotObservable(_)),
|
|
"scoring Trigger zero would report a delivery-model property as an \
|
|
agent failure — the exact confusion this module exists to avoid"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn markers_the_real_parser_accepts_are_compliant() {
|
|
let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]);
|
|
let good = "I implemented the parser.\nCOMPLETED: INT-07 — wire the loop";
|
|
let scored = score(&prompt, &narrative(good), &builtin);
|
|
assert_eq!(scored[0].compliance, Verdict::Pass);
|
|
}
|
|
|
|
/// The failure the skill exists to prevent: marker-shaped lines that the
|
|
/// parser rejects, so the mission silently never advances.
|
|
#[test]
|
|
fn marker_shaped_lines_the_parser_rejects_are_a_failure() {
|
|
let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]);
|
|
// Bold, which the skill explicitly forbids, and the parser will not take.
|
|
let bad = "**COMPLETED: INT-07**";
|
|
let scored = score(&prompt, &narrative(bad), &builtin);
|
|
assert!(
|
|
matches!(scored[0].compliance, Verdict::Fail(_)),
|
|
"a marker the parser rejects must score as a failure — that is the \
|
|
whole consequence the skill is written to avoid. Got {:?}",
|
|
scored[0].compliance
|
|
);
|
|
|
|
// A turn that never tried is not a violation.
|
|
let unrelated = score(&prompt, &narrative("I read three files and wrote a summary."), &builtin);
|
|
assert_eq!(unrelated[0].compliance, Verdict::NotApplicable);
|
|
}
|
|
|
|
/// Found on a live mission: `PLAN_COMPLETE: INT-01..02`.
|
|
///
|
|
/// Two defects in one line, both since fixed in the parser rather than
|
|
/// worked around here:
|
|
///
|
|
/// 1. `PLAN_COMPLETE` was documented in `int-xx-marker-protocol` and never
|
|
/// implemented, so an agent following the skill exactly was ignored. It
|
|
/// is implemented now.
|
|
/// 2. The range form parsed into the id `INT-01..02`, matching no real
|
|
/// item — a task card for something that did not exist. Ids are now
|
|
/// strictly `INT-<digits>`, so the marker is REJECTED instead, which is
|
|
/// visible where a plausible-looking row was not.
|
|
#[test]
|
|
fn plan_complete_is_now_a_real_marker() {
|
|
let parsed = crate::task_card_parser::parse("PLAN_COMPLETE: INT-01");
|
|
assert_eq!(parsed.len(), 1, "the skill documents it; the parser must accept it");
|
|
assert_eq!(parsed[0].int_id, "INT-01");
|
|
|
|
let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]);
|
|
assert_eq!(
|
|
score(&prompt, &narrative("PLAN_COMPLETE: INT-01"), &builtin)[0].compliance,
|
|
Verdict::Pass
|
|
);
|
|
}
|
|
|
|
/// The exact line a live planner emitted.
|
|
#[test]
|
|
fn a_range_marker_is_rejected_rather_than_creating_a_phantom_item() {
|
|
assert!(
|
|
crate::task_card_parser::parse("PLAN_COMPLETE: INT-01..02").is_empty(),
|
|
"a range must not parse — it produced a task card for an item that \
|
|
does not exist while INT-01 and INT-02 stayed open"
|
|
);
|
|
assert!(crate::task_card_parser::parse("COMPLETED: INT-01..02").is_empty());
|
|
|
|
// And the agent is told, because the marker it emitted did nothing.
|
|
let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]);
|
|
match &score(&prompt, &narrative("PLAN_COMPLETE: INT-01..02"), &builtin)[0].compliance {
|
|
Verdict::Fail(why) => assert!(why.contains("does not accept")),
|
|
other => panic!("an ignored marker must not read as success; got {other:?}"),
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn two_int_ids_on_one_marker_line_cross_the_boundary() {
|
|
let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]);
|
|
let scored = score(&prompt, &narrative("COMPLETED: INT-05, INT-06"), &builtin);
|
|
match &scored[0].boundary {
|
|
Verdict::Fail(why) => assert!(why.contains("silently dropped")),
|
|
other => panic!("the second id is silently dropped by the parser; got {other:?}"),
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn querying_arxiv_crosses_the_boundary_but_naming_it_does_not() {
|
|
let prompt = rendered(&[("arxiv-daily", "x")]);
|
|
|
|
let violating = score(
|
|
&prompt,
|
|
&narrative("curl 'http://export.arxiv.org/api/query?search_query=all:agents'"),
|
|
&builtin,
|
|
);
|
|
assert!(matches!(violating[0].boundary, Verdict::Fail(_)));
|
|
|
|
// The word appears legitimately in every research turn. Scoring on it
|
|
// would make the metric fire constantly and mean nothing.
|
|
let fine = score(
|
|
&prompt,
|
|
&narrative("I read the arXiv notes in the manifest and summarised three of them."),
|
|
&builtin,
|
|
);
|
|
assert_eq!(fine[0].boundary, Verdict::Pass);
|
|
}
|
|
|
|
/// Trigger, from the exact arguments a live mission recorded.
|
|
///
|
|
/// Run 8, 2026-08-21: three agents each reached through the door and read a
|
|
/// skill body. Copied from `mission_events`, not invented — the shape of
|
|
/// this payload is the whole basis of the axis.
|
|
#[test]
|
|
fn a_retrieved_skill_scores_trigger_as_a_pass() {
|
|
let tools = acted(&[
|
|
("ListMcpResourcesTool", None, json!({})),
|
|
(
|
|
"ReadMcpResourceTool",
|
|
None,
|
|
json!({"uri": "skill:global/workspace-repo-commit-protocol",
|
|
"server": "clawmates_skills"}),
|
|
),
|
|
]);
|
|
let ev = Evidence::new("", &tools);
|
|
assert_eq!(
|
|
retrieved_skills(&ev),
|
|
vec!["workspace-repo-commit-protocol"]
|
|
);
|
|
|
|
// Delivered by RETRIEVAL only — nothing in the prompt at all. Under
|
|
// progressive disclosure this is every skill, so a scorer that reads
|
|
// only the prompt would report nothing for the delivery model this
|
|
// axis exists to measure.
|
|
let scored = score("Task: do the thing", &ev, &builtin);
|
|
assert_eq!(scored.len(), 1);
|
|
assert_eq!(scored[0].skill, "workspace-repo-commit-protocol");
|
|
assert_eq!(
|
|
scored[0].trigger,
|
|
Verdict::Pass,
|
|
"the agent reached for it; that is the paper's Trigger"
|
|
);
|
|
}
|
|
|
|
/// An inlined skill is still `NotObservable`, and the reason now names the
|
|
/// fix rather than the transport.
|
|
#[test]
|
|
fn an_inlined_skill_is_still_unobservable_for_trigger() {
|
|
let prompt = rendered(&[("arxiv-daily", "x")]);
|
|
let scored = score(&prompt, &narrative("did it"), &builtin);
|
|
match &scored[0].trigger {
|
|
Verdict::NotObservable(why) => assert!(
|
|
why.contains("handed it"),
|
|
"the reason must say WHY it cannot be seen — being handed a \
|
|
skill is not failing to reach for one: {why}"
|
|
),
|
|
other => panic!("got {other:?}"),
|
|
}
|
|
}
|
|
|
|
/// A workspace-scoped URI resolves to the same name as a global one.
|
|
#[test]
|
|
fn both_uri_shapes_yield_the_skill_name() {
|
|
let ws = uuid::Uuid::now_v7();
|
|
let tools = acted(&[
|
|
(
|
|
"ReadMcpResourceTool",
|
|
None,
|
|
json!({"uri": format!("skill:workspace/{ws}/team-local-thing")}),
|
|
),
|
|
// Not a skill URI, and not a panic.
|
|
("ReadMcpResourceTool", None, json!({"uri": "file:///etc/passwd"})),
|
|
// A LIST call names no skill — only reads are retrievals.
|
|
("ListMcpResourcesTool", None, json!({})),
|
|
]);
|
|
assert_eq!(
|
|
retrieved_skills(&Evidence::new("", &tools)),
|
|
vec!["team-local-thing"],
|
|
"listing the catalogue is browsing; reading a body is the reach"
|
|
);
|
|
}
|
|
|
|
/// Agent-authored skills must stay visible as such in the report.
|
|
#[test]
|
|
fn writing_outside_the_mission_checkout_crosses_the_boundary() {
|
|
let prompt = rendered(&[("workspace-repo-commit-protocol", "Work in /mission/repo.")]);
|
|
|
|
let wrong = score(
|
|
&prompt,
|
|
&narrative("cd /workspace/repo && git add -A && git commit -m 'INT-01 done'"),
|
|
&builtin,
|
|
);
|
|
match &wrong[0].boundary {
|
|
Verdict::Fail(why) => assert!(
|
|
why.contains("never collected"),
|
|
"the failure must name the consequence — a phase that delivers \
|
|
nothing — not just the wrong path: {why}"
|
|
),
|
|
other => panic!("the wrong checkout must be caught; got {other:?}"),
|
|
}
|
|
|
|
let right = score(
|
|
&prompt,
|
|
&narrative("cd /mission/repo && git add -A && git commit -m 'INT-01 done'"),
|
|
&builtin,
|
|
);
|
|
assert_eq!(right[0].boundary, Verdict::Pass);
|
|
}
|
|
|
|
/// The upgrade, stated as a test: the agent never said where it wrote.
|
|
///
|
|
/// Before the tool tap this scored a clean `Pass`, because the only
|
|
/// evidence was the narrative and the narrative did not mention a path. The
|
|
/// phase had still delivered nothing.
|
|
#[test]
|
|
fn a_write_outside_the_checkout_is_caught_even_when_the_narrative_is_silent() {
|
|
let prompt = rendered(&[("workspace-repo-commit-protocol", "Work in /mission/repo.")]);
|
|
let tools = acted(&[
|
|
("Write", Some("/app/src/main.rs"), json!({"file_path": "/app/src/main.rs"})),
|
|
]);
|
|
let scored = score(&prompt, &Evidence::new("I implemented the feature.", &tools), &builtin);
|
|
match &scored[0].boundary {
|
|
Verdict::Fail(why) => assert!(why.contains("/app/src/main.rs"), "{why}"),
|
|
other => panic!("the write is right there in the record; got {other:?}"),
|
|
}
|
|
}
|
|
|
|
/// Scratch is not a violation — the skill blesses it by name.
|
|
#[test]
|
|
fn scratch_and_the_checkout_both_stay_inside_the_boundary() {
|
|
let prompt = rendered(&[("workspace-repo-commit-protocol", "Work in /mission/repo.")]);
|
|
let tools = acted(&[
|
|
("Write", Some("/tmp/scratch.txt"), json!({"file_path": "/tmp/scratch.txt"})),
|
|
("Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"})),
|
|
]);
|
|
let scored = score(&prompt, &Evidence::new("done", &tools), &builtin);
|
|
assert_eq!(scored[0].boundary, Verdict::Pass);
|
|
}
|
|
|
|
/// Nothing observed is not the same as nothing done.
|
|
///
|
|
/// This returned `Pass` before — a mission with no evidence scored
|
|
/// identically to one that was checked and found clean, which is the exact
|
|
/// collapse `Verdict::NotObservable` exists to prevent.
|
|
#[test]
|
|
fn silence_is_not_scored_as_compliance() {
|
|
let prompt = rendered(&[("workspace-repo-commit-protocol", "Work in /mission/repo.")]);
|
|
let scored = score(&prompt, &narrative("I thought about the problem."), &builtin);
|
|
assert!(
|
|
matches!(scored[0].boundary, Verdict::NotObservable(_)),
|
|
"got {:?}",
|
|
scored[0].boundary
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn a_force_push_crosses_the_boundary_and_a_quoted_flag_does_not() {
|
|
let prompt = rendered(&[("workspace-repo-commit-protocol", "Work in /mission/repo.")]);
|
|
|
|
let forced = acted(&[
|
|
("Write", Some("/mission/repo/a.rs"), json!({"file_path": "/mission/repo/a.rs"})),
|
|
ran("cd /mission/repo && git push --force origin mission"),
|
|
]);
|
|
match &score(&prompt, &Evidence::new("", &forced), &builtin)[0].boundary {
|
|
Verdict::Fail(why) => assert!(why.contains("force-push"), "{why}"),
|
|
other => panic!("rewriting pushed history is the one unrecoverable act; got {other:?}"),
|
|
}
|
|
|
|
// The flag inside a commit message rewrites nothing.
|
|
let innocent = acted(&[
|
|
("Write", Some("/mission/repo/a.rs"), json!({"file_path": "/mission/repo/a.rs"})),
|
|
ran(r#"git commit -m "document the -f flag" && git push origin mission"#),
|
|
]);
|
|
assert_eq!(
|
|
score(&prompt, &Evidence::new("", &innocent), &builtin)[0].boundary,
|
|
Verdict::Pass
|
|
);
|
|
}
|
|
|
|
/// Writing code and never running a test cannot be red-green-refactor under
|
|
/// any reading of the loop.
|
|
#[test]
|
|
fn code_written_with_no_test_run_fails_the_loop() {
|
|
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
|
let tools = acted(&[
|
|
("Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"})),
|
|
ran("cargo build"),
|
|
]);
|
|
match &score(&prompt, &Evidence::new("Implemented it.", &tools), &builtin)[0].compliance {
|
|
Verdict::Fail(why) => assert!(why.contains("never ran a test"), "{why}"),
|
|
other => panic!("got {other:?}"),
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn a_test_run_before_the_first_write_is_red_first() {
|
|
let prompt = rendered(&[("cargo-test-driven-development", "Red, green, refactor.")]);
|
|
let tools = acted(&[
|
|
ran("cargo nextest run parses_the_range_form"),
|
|
("Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"})),
|
|
ran("cargo nextest run parses_the_range_form"),
|
|
]);
|
|
assert_eq!(
|
|
score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance,
|
|
Verdict::Pass
|
|
);
|
|
}
|
|
|
|
/// The ordering that cannot be read either way, reported as such.
|
|
///
|
|
/// A Rust unit test lives in `#[cfg(test)] mod tests` inside the file under
|
|
/// test, so writing the failing test IS a write to the implementation file.
|
|
/// Scoring this as a failure would fail the agents who followed the skill
|
|
/// most precisely.
|
|
#[test]
|
|
fn a_test_run_after_the_first_write_is_undecidable_not_a_failure() {
|
|
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
|
let tools = acted(&[
|
|
("Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"})),
|
|
ran("cargo test -p cm-api"),
|
|
]);
|
|
assert!(
|
|
matches!(
|
|
score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance,
|
|
Verdict::NotObservable(_)
|
|
),
|
|
"the Rust unit test lives in the file under test — this ordering \
|
|
cannot separate red-first from tests-added-after"
|
|
);
|
|
}
|
|
|
|
/// Red then green, read from what the runs themselves reported.
|
|
///
|
|
/// This is the case ordering cannot decide and the repository cannot
|
|
/// either: one `Edit` adds the implementation and its `#[cfg(test)] mod
|
|
/// tests` together, and `tdd-red-green-refactor` says in so many words to
|
|
/// "commit the RED-to-GREEN pair as one commit". The run outputs are the
|
|
/// only witness left.
|
|
#[test]
|
|
fn a_failing_run_followed_by_a_passing_one_is_red_then_green() {
|
|
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
|
let mut tools = acted(&[(
|
|
"Write",
|
|
Some("/mission/repo/src/a.rs"),
|
|
json!({"file_path": "/mission/repo/src/a.rs"}),
|
|
)]);
|
|
tools.push(test_run(
|
|
"cargo test",
|
|
"running 1 test\ntest slugs ... FAILED\n\ntest result: FAILED. 0 passed; 1 failed",
|
|
));
|
|
tools.push(test_run(
|
|
"cargo test",
|
|
"running 1 test\ntest slugs ... ok\n\ntest result: ok. 1 passed; 0 failed",
|
|
));
|
|
assert_eq!(
|
|
score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance,
|
|
Verdict::Pass
|
|
);
|
|
}
|
|
|
|
/// Every run green is NOT evidence of red-first.
|
|
#[test]
|
|
fn runs_that_only_ever_passed_cannot_prove_the_loop() {
|
|
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
|
let mut tools = acted(&[(
|
|
"Write",
|
|
Some("/mission/repo/src/a.rs"),
|
|
json!({"file_path": "/mission/repo/src/a.rs"}),
|
|
)]);
|
|
tools.push(test_run("cargo test", "test result: ok. 3 passed; 0 failed"));
|
|
match &score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance {
|
|
Verdict::NotObservable(why) => assert!(
|
|
why.contains("never failed"),
|
|
"a test that never failed is equally what a correct \
|
|
implementation written first looks like: {why}"
|
|
),
|
|
other => panic!("got {other:?}"),
|
|
}
|
|
}
|
|
|
|
/// Ending red is a failure of the loop, and a different one from never
|
|
/// running a test at all.
|
|
#[test]
|
|
fn a_phase_that_never_reached_green_fails() {
|
|
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
|
let mut tools = acted(&[(
|
|
"Write",
|
|
Some("/mission/repo/src/a.rs"),
|
|
json!({"file_path": "/mission/repo/src/a.rs"}),
|
|
)]);
|
|
tools.push(test_run("cargo test", "test result: FAILED. 0 passed; 2 failed"));
|
|
match &score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance {
|
|
Verdict::Fail(why) => assert!(why.contains("never got there"), "{why}"),
|
|
other => panic!("got {other:?}"),
|
|
}
|
|
}
|
|
|
|
/// A mission recorded before responses were captured must not read as
|
|
/// either verdict.
|
|
#[test]
|
|
fn runs_with_no_recorded_output_stay_unobservable() {
|
|
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
|
let tools = acted(&[
|
|
("Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"})),
|
|
("Bash", None, json!({"command": "cargo test"})),
|
|
]);
|
|
assert!(matches!(
|
|
score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance,
|
|
Verdict::NotObservable(_)
|
|
));
|
|
}
|
|
|
|
/// The verdict line is at the END of a test run, which is why the response
|
|
/// bound keeps the tail.
|
|
#[test]
|
|
fn a_truncated_response_still_carries_the_verdict() {
|
|
let long = "compiling…\n".repeat(400) + "test result: FAILED. 0 passed; 1 failed";
|
|
let bounded = crate::vm_tool_tap::bounded_response(
|
|
"Bash",
|
|
&json!({ "stdout": long, "stderr": "" }),
|
|
);
|
|
let kept = bounded["stdout"].as_str().expect("stdout kept");
|
|
assert!(
|
|
kept.contains("test result: FAILED"),
|
|
"a head-biased truncation would keep the noise and drop the verdict, \
|
|
which is the only reason the output is stored"
|
|
);
|
|
assert!(kept.starts_with("[truncated]"), "{kept}");
|
|
}
|
|
|
|
/// A `Read`'s response is the file it just read — already knowable, and
|
|
/// large.
|
|
#[test]
|
|
fn only_commands_carry_a_response() {
|
|
assert_eq!(
|
|
crate::vm_tool_tap::bounded_response("Read", &json!({"stdout": "x"})),
|
|
serde_json::Value::Null
|
|
);
|
|
}
|
|
|
|
/// A phase that wrote nothing had no implementation to test-drive.
|
|
/// The verdict the first live run got wrong.
|
|
///
|
|
/// A research phase wrote fifteen markdown notes and ran no test, and the
|
|
/// check called it a TDD failure. There was no code to test-drive. The
|
|
/// real finding is that a research role is pinned TDD skills at all, and
|
|
/// scoring the agent for it would have buried that.
|
|
#[test]
|
|
fn writing_prose_is_not_a_tdd_failure() {
|
|
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
|
let tools = acted(&[
|
|
("Write", Some("/mission/repo/research/notes.md"), json!({"file_path": "/mission/repo/research/notes.md"})),
|
|
("Edit", Some("/mission/repo/research/notes.md"), json!({"file_path": "/mission/repo/research/notes.md"})),
|
|
("Write", Some("/mission/repo/research/check.sh"), json!({"file_path": "/mission/repo/research/check.sh"})),
|
|
]);
|
|
assert_eq!(
|
|
score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance,
|
|
Verdict::NotApplicable,
|
|
"markdown and a helper script are not the behaviour-adding code \
|
|
this loop is about"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn a_phase_that_wrote_nothing_is_not_a_tdd_failure() {
|
|
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
|
let tools = acted(&[ran("ls /mission/repo"), ("Read", Some("/mission/repo/a.rs"), json!({}))]);
|
|
assert_eq!(
|
|
score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance,
|
|
Verdict::NotApplicable
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn the_never_merge_subjects_are_caught_and_a_real_one_is_not() {
|
|
let prompt = rendered(&[("small-focused-commits", "One idea per commit.")]);
|
|
|
|
let bad = acted(&[ran(r#"git commit -m "wip""#)]);
|
|
match &score(&prompt, &Evidence::new("", &bad), &builtin)[0].boundary {
|
|
Verdict::Fail(why) => assert!(why.contains("never-merge"), "{why}"),
|
|
other => panic!("got {other:?}"),
|
|
}
|
|
|
|
// The objection is to `fix` as the whole message, not to the word.
|
|
let good = acted(&[ran(r#"git commit -m "INT-04 fix the range form the parser rejects""#)]);
|
|
assert_eq!(
|
|
score(&prompt, &Evidence::new("", &good), &builtin)[0].boundary,
|
|
Verdict::NotApplicable
|
|
);
|
|
}
|
|
|
|
/// The subject comes out of the shell command, in the shapes agents write.
|
|
#[test]
|
|
fn the_commit_subject_is_read_out_of_the_real_command_shapes() {
|
|
assert_eq!(dash_m_value(r#"git commit -m "one line""#).as_deref(), Some("one line"));
|
|
assert_eq!(dash_m_value("git commit -am 'clustered flag'").as_deref(), Some("clustered flag"));
|
|
// The template in `workspace-repo-commit-protocol` spans several lines;
|
|
// only the first is the subject.
|
|
let multi = "cd /mission/repo && git commit -m \"INT-01 wire the loop\n\nWHY: it stalled.\n\nRefs: INT-01\n\"";
|
|
assert_eq!(
|
|
commit_subjects(&Evidence::new("", &acted(&[ran(multi)]))),
|
|
vec!["INT-01 wire the loop"]
|
|
);
|
|
// `-F` gives up nothing, and that must read as no evidence.
|
|
assert!(dash_m_value("git commit -F /tmp/msg").is_none());
|
|
}
|
|
|
|
/// The exact commit command the first live coding run ran.
|
|
///
|
|
/// Claude Code writes a multi-line message as a heredoc inside a command
|
|
/// substitution, so the first line of the `-m` value is the heredoc opener
|
|
/// and the subject is the line after it. Read from
|
|
/// `mission_events`, not invented.
|
|
#[test]
|
|
fn a_heredoc_commit_message_gives_up_its_real_subject() {
|
|
let live = "git add src/lib.rs && git commit -m \"$(cat <<'EOF'\n\
|
|
INT-01 Add slugify function to src/lib.rs\n\n\
|
|
The crate exposed only add() with no string-normalization utility.\n\n\
|
|
Refs: INT-01\n\
|
|
EOF\n\
|
|
)\"";
|
|
assert_eq!(
|
|
commit_subjects(&Evidence::new("", &acted(&[ran(live)]))),
|
|
vec!["INT-01 Add slugify function to src/lib.rs"],
|
|
"the heredoc OPENER is not the subject — reading it as one puts \
|
|
`$(cat <<'EOF'` in the record, and every commit check then scores \
|
|
a string the agent never wrote"
|
|
);
|
|
}
|
|
|
|
/// The skill tells the agent to `curl` the abstract page in the paragraph
|
|
/// under the one that forbids the query API. A check that cannot tell them
|
|
/// apart fails agents for obeying the skill.
|
|
#[test]
|
|
fn curling_an_abstract_is_allowed_where_querying_the_api_is_not() {
|
|
let prompt = rendered(&[("arxiv-daily", "The harvest already ran.")]);
|
|
|
|
let allowed = acted(&[ran("curl -s https://arxiv.org/abs/2401.12345")]);
|
|
assert_eq!(
|
|
score(&prompt, &Evidence::new("", &allowed), &builtin)[0].boundary,
|
|
Verdict::Pass
|
|
);
|
|
|
|
let forbidden = acted(&[ran("curl -s 'http://export.arxiv.org/api/query?search_query=all:agents'")]);
|
|
match &score(&prompt, &Evidence::new("", &forbidden), &builtin)[0].boundary {
|
|
Verdict::Fail(why) => assert!(why.contains("seen-set"), "{why}"),
|
|
other => panic!("got {other:?}"),
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn the_source_of_a_skill_is_carried_into_its_score() {
|
|
let prompt = rendered(&[("self-made", "x")]);
|
|
let scored = score(&prompt, &narrative("done"), &|_| "promoted_from_brain".to_string());
|
|
assert_eq!(
|
|
scored[0].source_kind, "promoted_from_brain",
|
|
"an agent that writes its own skill can raise its own score against \
|
|
it; that has to be legible in the report rather than averaged in"
|
|
);
|
|
}
|
|
}
|