Omar SobhandClaude Opus 5
f52cff3e04
Trigger — did the agent reach for the skill when it applied? — cannot be measured while every body is inlined into the prompt. Nothing was reached for. `skill_use` has been reporting `NotObservable` for that reason, and it was right to. The skills door made retrieval possible; this makes it a delivery arm. `index` sends each pinned skill's name, description, `when_to_use` and the uri that returns its body, and the agent fetches what it judges relevant. `inline` is unchanged and stays the default. An A/B rather than a switch, because `index` can only cost Compliance: under `inline` the procedure sits in front of the model whether or not it noticed it applied. Trading a measured axis for an unmeasured regression in another is not an improvement, so both arms stay runnable and the arm is recorded on the mission row. Three things the mechanism refuses to do: - `index` without a door falls back to `inline`. An index names bodies and says how to fetch them; with no `clawmates_skills` server reachable that is a list of dead ends, and it fails as an agent ignoring its skills rather than as a missing config. `install_skills_door` now returns whether it installed, because the caller needs the answer and not just the log line. - The scorer reads the arm off the recorded PROMPT, not off the mission row. The row says what the mission is configured to do now; the score is being computed against a turn that ran then. - Under `index`, a skill that was offered and never read is a Fail, not the inline arm's `NotObservable` — but only where the skill had a checkable consequence in that phase. Reusing the inline text would have said "this skill was inlined into the prompt" about a skill whose body was never sent, and scoring a real miss as a structural blind spot is the failure this measurement already made once. The arm is per mission (`config.skill_delivery`), not only per deployment. Both arms run against one server process; restarting between them would put a confound in the comparison that the numbers would not show. 829 tests, 108 binaries, green. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1645 lines
68 KiB
Rust
1645 lines
68 KiB
Rust
//! Did a skill we delivered actually change what the agent did?
|
|
//!
|
|
//! Scored on the three axes from `Skill-Use` (arXiv, 2026-08-05): **Trigger**
|
|
//! (did the agent reach for the skill), **Compliance** (did it follow the
|
|
//! procedure), **Boundary** (did it avoid what the skill forbids).
|
|
//!
|
|
//! ## One axis does not survive the translation, and saying so is the finding
|
|
//!
|
|
//! The paper measures agents under *progressive disclosure*: the agent sees a
|
|
//! name and a description, and must decide to retrieve the body. The retrieval
|
|
//! is the Trigger event, and it is observable because it is a tool call.
|
|
//!
|
|
//! We do not deliver skills that way on the mission path. `pinned_skills_text`
|
|
//! inlines full bodies into the prompt, so the agent never "reaches for" a
|
|
//! skill; it is simply holding one.
|
|
//!
|
|
//! Trigger is therefore **not observable on the mission path**, and this module
|
|
//! reports it as `NotObservable` with the reason attached rather than scoring
|
|
//! it zero. A zero would read as "the agents ignore their skills" when it
|
|
//! actually means "the question does not apply to how we deliver them" — the
|
|
//! precise confusion that made 55 empty skill bindings invisible for months.
|
|
//!
|
|
//! ### The reason changed, and only half of it went away
|
|
//!
|
|
//! Until 2026-08-21 the stated reason was that `claude_cli` "cannot surface a
|
|
//! tool call at all — there is nothing to retrieve *with*". That half is now
|
|
//! false: [`crate::container_tool_hooks`] installs `PostToolUse`, mission tool
|
|
//! calls land in `mission_events`, and a retrieval would be as visible as any
|
|
//! other call.
|
|
//!
|
|
//! The other half still holds, and it is the one that decides the verdict: we
|
|
//! still **inline**. Trigger is now *instrumentable* and still not
|
|
//! *observable*, and the blocker has moved from the transport to the delivery
|
|
//! model. Making it real is one change — serve skills through the door
|
|
//! (`docs/TOOL-CALL-ARCHITECTURE.md` §3) so retrieval becomes a tool call —
|
|
//! and nothing else in this module has to change to score it.
|
|
//!
|
|
//! ## Compliance and Boundary are scored from ACTIONS where actions exist
|
|
//!
|
|
//! They were scored from the concatenated narrative, which is the agent's own
|
|
//! account of its turn — written by the thing being measured, and silent about
|
|
//! anything it did not think worth mentioning. [`Evidence`] carries the
|
|
//! recorded tool calls alongside that text, and every check prefers them:
|
|
//! `Bash` arguments say what ran, `Write` paths say where it landed.
|
|
//!
|
|
//! Every tool-backed check is **one-sided**. It reports a violation it can see
|
|
//! and never infers compliance from silence — the recorded stream is capped
|
|
//! per phase ([`crate::mission_events::PER_PHASE_CAP`]), so an absent call is
|
|
//! not proof of an absent action.
|
|
|
|
use serde::Serialize;
|
|
|
|
/// The outcome of one axis for one skill.
|
|
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
|
|
#[serde(rename_all = "snake_case", tag = "verdict", content = "why")]
|
|
pub enum Verdict {
|
|
Pass,
|
|
Fail(String),
|
|
/// The skill says nothing this axis can check.
|
|
NotApplicable,
|
|
/// The axis cannot be measured here, for a stated structural reason.
|
|
///
|
|
/// Distinct from `Fail` on purpose. Collapsing the two is how a
|
|
/// measurement reports a system defect as an agent defect.
|
|
NotObservable(String),
|
|
}
|
|
|
|
impl Verdict {
|
|
pub fn label(&self) -> &'static str {
|
|
match self {
|
|
Verdict::Pass => "pass",
|
|
Verdict::Fail(_) => "FAIL",
|
|
Verdict::NotApplicable => "n/a",
|
|
Verdict::NotObservable(_) => "not observable",
|
|
}
|
|
}
|
|
}
|
|
|
|
/// One skill's score for one phase.
|
|
#[derive(Debug, Clone, Serialize)]
|
|
pub struct SkillUse {
|
|
pub skill: String,
|
|
/// `builtin` (hand-authored) or `promoted_from_brain` (agent-authored).
|
|
///
|
|
/// Carried into the report because an agent that authors its own skill can
|
|
/// raise its own compliance score without changing what it does. A rising
|
|
/// number on agent-authored skills has to be visible as such rather than
|
|
/// averaged in with the rest.
|
|
pub source_kind: String,
|
|
pub trigger: Verdict,
|
|
pub compliance: Verdict,
|
|
pub boundary: Verdict,
|
|
}
|
|
|
|
/// The skills a prompt actually delivered.
|
|
///
|
|
/// Delegates to the delivery layer's own parser, so the reader cannot drift
|
|
/// from the writer. This function originally matched `## <name>` itself, and
|
|
/// skill bodies are markdown full of `##` headings — a live mission duly
|
|
/// scored "Sizing heuristic" and "The output shape" as skills. Parsed from the
|
|
/// recorded prompt rather than re-derived from the catalogue, because the
|
|
/// catalogue changes, and now that agents author their own skills it changes
|
|
/// by itself.
|
|
pub fn skills_in_prompt(prompt: &str) -> Vec<String> {
|
|
crate::topology_exec::skill_names_in(prompt)
|
|
}
|
|
|
|
/// What a mission left behind, for the checks to read.
|
|
///
|
|
/// Two sources, deliberately kept apart. `text` is what the agent *said* it
|
|
/// did; `tools` is what it did. Where they disagree the tools win, and a check
|
|
/// that can only consult `text` says so in its verdict rather than presenting a
|
|
/// narrative reading as an observation.
|
|
pub struct Evidence<'a> {
|
|
/// Concatenated `reasoning` events for the mission.
|
|
pub text: &'a str,
|
|
/// Every `tool.call` recorded for the mission, in order.
|
|
pub tools: &'a [crate::mission_events::ToolEvidence],
|
|
}
|
|
|
|
/// Tools that modify a file. `target` on the recorded event is the tool name.
|
|
const WRITE_TOOLS: [&str; 4] = ["Write", "Edit", "MultiEdit", "NotebookEdit"];
|
|
|
|
impl<'a> Evidence<'a> {
|
|
pub fn new(text: &'a str, tools: &'a [crate::mission_events::ToolEvidence]) -> Self {
|
|
Evidence { text, tools }
|
|
}
|
|
|
|
/// Narrative only — a mission whose events predate the tool tap.
|
|
pub fn from_text(text: &'a str) -> Self {
|
|
Evidence { text, tools: &[] }
|
|
}
|
|
|
|
/// Every shell command the agent ran, in order.
|
|
fn commands(&self) -> impl Iterator<Item = &str> {
|
|
self.tools.iter().filter_map(|t| t.command())
|
|
}
|
|
|
|
/// Test runs, in order, each with whether it went red or green.
|
|
///
|
|
/// `None` where the outcome was not recorded — a mission whose events
|
|
/// predate the response capture, which must not read as either.
|
|
fn test_runs(&self) -> Vec<Option<bool>> {
|
|
self.tools
|
|
.iter()
|
|
.filter(|t| t.command().is_some_and(is_test_command))
|
|
.map(|t| test_passed(&t.response))
|
|
.collect()
|
|
}
|
|
|
|
/// Every file-modifying call that named a path, in order.
|
|
fn writes(&self) -> impl Iterator<Item = &crate::mission_events::ToolEvidence> {
|
|
self.tools
|
|
.iter()
|
|
.filter(|t| WRITE_TOOLS.contains(&t.tool.as_str()) && t.path.is_some())
|
|
}
|
|
}
|
|
|
|
/// Skills the agent RETRIEVED, in the order it reached for them.
|
|
///
|
|
/// This is Trigger, and it is observable the moment a skill arrives by
|
|
/// retrieval instead of by inlining. Claude Code reads an MCP resource through
|
|
/// `ReadMcpResourceTool`, and the recorded arguments carry the URI:
|
|
///
|
|
/// ```text
|
|
/// ReadMcpResourceTool {"uri":"skill:global/workspace-repo-commit-protocol",
|
|
/// "server":"clawmates_skills"}
|
|
/// ```
|
|
///
|
|
/// The URI is parsed by `mcp_skills::parse_uri` — the same function that
|
|
/// WROTE it — rather than by a second matcher here, for the reason every other
|
|
/// reader in this module delegates: two implementations of one format drift,
|
|
/// and the drift shows up as a skill silently scoring nothing.
|
|
pub fn retrieved_skills(ev: &Evidence<'_>) -> Vec<String> {
|
|
let mut out = Vec::new();
|
|
for t in ev.tools {
|
|
if t.tool != "ReadMcpResourceTool" {
|
|
continue;
|
|
}
|
|
let Some(uri) = t.input.get("uri").and_then(|v| v.as_str()) else {
|
|
continue;
|
|
};
|
|
if let Some((_, name)) = crate::mcp_skills::parse_uri(uri) {
|
|
if !out.contains(&name) {
|
|
out.push(name);
|
|
}
|
|
}
|
|
}
|
|
out
|
|
}
|
|
|
|
/// Did the agent reach for this skill?
|
|
///
|
|
/// The answer depends on whether it was ever given the chance, which is what
|
|
/// the delivery arm decides — so this takes the arm rather than assuming one.
|
|
/// Getting that wrong is not a rounding error: under `Index` the old text
|
|
/// would have said "this skill was inlined into the prompt" about a skill that
|
|
/// was not, and scored a real miss as a structural blind spot.
|
|
fn trigger_verdict(
|
|
mode: crate::skill_delivery::Mode,
|
|
retrieved: bool,
|
|
compliance: &Verdict,
|
|
boundary: &Verdict,
|
|
) -> Verdict {
|
|
if retrieved {
|
|
// The agent reached for it. That is the paper's Trigger, and it is a
|
|
// recorded tool call like any other.
|
|
return Verdict::Pass;
|
|
}
|
|
match mode {
|
|
// Handed over, so there was no reaching-for to observe. Not a failure
|
|
// and not a pass — the axis simply does not exist in this arm.
|
|
crate::skill_delivery::Mode::Inline => Verdict::NotObservable(
|
|
"this skill was inlined into the prompt, not retrieved — the agent \
|
|
was handed it, so there is no reaching-for to observe. Serve it \
|
|
through the door instead and this becomes a tool call"
|
|
.into(),
|
|
),
|
|
crate::skill_delivery::Mode::Index => {
|
|
// Offered by name and `when_to_use`, and never opened. Whether that
|
|
// is a miss depends on whether the skill had anything to say about
|
|
// this phase at all: a skill with no machine-checkable consequence
|
|
// here is one an agent is right to pass over, and scoring that as a
|
|
// failure would punish correct triage.
|
|
if matches!(compliance, Verdict::NotApplicable)
|
|
&& matches!(boundary, Verdict::NotApplicable)
|
|
{
|
|
Verdict::NotApplicable
|
|
} else {
|
|
Verdict::Fail(
|
|
"offered in the index with its `when_to_use`, and never read — \
|
|
the agent had the entry in front of it and did not fetch the \
|
|
procedure"
|
|
.into(),
|
|
)
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Score every skill a phase's prompt delivered, against what the agent produced.
|
|
pub fn score(
|
|
prompt: &str,
|
|
evidence: &Evidence<'_>,
|
|
source_kinds: &dyn Fn(&str) -> String,
|
|
) -> Vec<SkillUse> {
|
|
let retrieved = retrieved_skills(evidence);
|
|
// Read off the prompt that was actually sent, not off the mission row: the
|
|
// row says what the mission is configured to do now, and this is scoring a
|
|
// turn that ran then.
|
|
let mode = crate::skill_delivery::mode_in_prompt(prompt);
|
|
// Delivered by either route. A skill that was retrieved and never inlined
|
|
// is invisible to `skills_in_prompt`, and under progressive disclosure that
|
|
// is EVERY skill — so scoring only the prompt would report zero for the
|
|
// delivery model this axis exists to measure.
|
|
let mut names = skills_in_prompt(prompt);
|
|
for r in &retrieved {
|
|
if !names.contains(r) {
|
|
names.push(r.clone());
|
|
}
|
|
}
|
|
names
|
|
.into_iter()
|
|
.map(|skill| {
|
|
let (compliance, boundary) = check(&skill, evidence);
|
|
SkillUse {
|
|
source_kind: source_kinds(&skill),
|
|
trigger: trigger_verdict(
|
|
mode,
|
|
retrieved.contains(&skill),
|
|
&compliance,
|
|
&boundary,
|
|
),
|
|
compliance,
|
|
boundary,
|
|
skill,
|
|
}
|
|
})
|
|
.collect()
|
|
}
|
|
|
|
/// Per-skill mechanical checks.
|
|
///
|
|
/// Only skills whose procedure has a machine-checkable consequence are checked.
|
|
/// Everything else returns `NotApplicable` rather than a guess: a heuristic
|
|
/// that scores prose by keyword overlap produces a number that looks like a
|
|
/// measurement and is not one.
|
|
fn check(skill: &str, ev: &Evidence<'_>) -> (Verdict, Verdict) {
|
|
match skill {
|
|
"int-xx-marker-protocol" => (marker_compliance(ev.text), marker_boundary(ev.text)),
|
|
"arxiv-daily" => (Verdict::NotApplicable, arxiv_boundary(ev)),
|
|
"workspace-repo-commit-protocol" => (Verdict::NotApplicable, workspace_boundary(ev)),
|
|
"small-focused-commits" => (Verdict::NotApplicable, commit_subject_boundary(ev)),
|
|
// One check for both: `cargo-test-driven-development` is the Rust
|
|
// flavour of the same loop, and its own text says so. Scoring them by
|
|
// separate rules would mean two rules for one procedure, which is how
|
|
// the microVM and container taps drifted apart.
|
|
"tdd-red-green-refactor" | "cargo-test-driven-development" => {
|
|
(red_before_green(ev), Verdict::NotApplicable)
|
|
}
|
|
_ => (Verdict::NotApplicable, Verdict::NotApplicable),
|
|
}
|
|
}
|
|
|
|
/// The markers must be ones `task_card_parser` actually parses.
|
|
///
|
|
/// Checked by running the real parser rather than a copy of its rules — a
|
|
/// second implementation of the contract would drift from the first, and then
|
|
/// the measurement would pass while the mission loop still stalled.
|
|
fn marker_compliance(output: &str) -> Verdict {
|
|
if crate::task_card_parser::parse(output).is_empty() {
|
|
// Only a failure if the output looks like it TRIED. A turn with no
|
|
// marker-shaped line was probably not a coding turn at all.
|
|
if output.lines().any(|l| looks_like_marker_attempt(l)) {
|
|
return Verdict::Fail(
|
|
"emitted marker-shaped lines that the parser does not accept — \
|
|
the mission loop will not advance"
|
|
.into(),
|
|
);
|
|
}
|
|
return Verdict::NotApplicable;
|
|
}
|
|
Verdict::Pass
|
|
}
|
|
|
|
/// Rule 1: exactly one INT id per marker line.
|
|
///
|
|
/// Two shapes violate it, and they fail differently:
|
|
///
|
|
/// `COMPLETED: INT-05, INT-06` — the parser takes the first and drops the
|
|
/// rest, so an item is silently never closed.
|
|
/// `PLAN_COMPLETE: INT-01..02` — the range form. The parser now REJECTS a
|
|
/// malformed id, so this is caught by the
|
|
/// compliance check above as a marker the
|
|
/// parser does not accept.
|
|
///
|
|
/// The second was found by running this measurement against a live mission. It
|
|
/// is the worse of the two, because a dropped marker leaves a gap and a
|
|
/// malformed one leaves a plausible-looking row.
|
|
fn marker_boundary(output: &str) -> Verdict {
|
|
for line in output.lines() {
|
|
let t = line.trim();
|
|
if !looks_like_marker_attempt(t) {
|
|
continue;
|
|
}
|
|
let ids = t.matches("INT-").count();
|
|
if ids > 1 {
|
|
return Verdict::Fail(format!(
|
|
"{ids} INT ids on one marker line — only the first parses, so \
|
|
the rest are silently dropped: {t:?}"
|
|
));
|
|
}
|
|
}
|
|
Verdict::NotApplicable
|
|
}
|
|
|
|
fn looks_like_marker_attempt(line: &str) -> bool {
|
|
const KINDS: &[&str] = &[
|
|
"TASK:", "WORK:", "HANDOFF:", "TEST_PASS:", "TEST_FAIL:", "REVIEW_APPROVE:",
|
|
"REVIEW_BLOCK:", "COMPLETED:", "PLAN_COMPLETE:",
|
|
];
|
|
let t = line.trim().trim_start_matches(['*', '#', '-', '`', ' ']);
|
|
KINDS.iter().any(|k| t.starts_with(k))
|
|
}
|
|
|
|
/// The mission checkout is `/mission/repo`. Working anywhere else is not
|
|
/// delivered.
|
|
///
|
|
/// Scored as a BOUNDARY rather than compliance: the skill's positive
|
|
/// instruction ("cd there at the start of every substantive turn") has no
|
|
/// reliable trace in the output, but writing source somewhere that is never
|
|
/// collected does, and it is the failure that costs a whole phase.
|
|
///
|
|
/// This check exists because the skill itself was wrong. It taught
|
|
/// `/workspace/repo` — a path the platform does not mount — while the same
|
|
/// prompt told the agent `/mission/repo`. An agent that obeyed the skill wrote
|
|
/// into a directory nothing collects. Corrected 2026-08-19, and
|
|
/// `skills_loader::contradiction_tests` now holds it.
|
|
fn workspace_boundary(ev: &Evidence<'_>) -> Verdict {
|
|
// The recorded writes settle this directly. Before the tool tap the only
|
|
// available evidence was the narrative, which answers a different question:
|
|
// an agent that wrote to the wrong root without narrating it scored a
|
|
// clean pass.
|
|
let stray: Vec<&str> = ev
|
|
.writes()
|
|
.filter_map(|w| w.path.as_deref())
|
|
.filter(|p| out_of_bounds(p))
|
|
.collect();
|
|
if let Some(first) = stray.first() {
|
|
return Verdict::Fail(format!(
|
|
"wrote to {first} ({} write(s) outside the checkout) — the mission \
|
|
checkout is /mission/repo, so anything written there is never \
|
|
collected and the phase delivers nothing",
|
|
stray.len()
|
|
));
|
|
}
|
|
|
|
// `--force` is the skill's other hard prohibition, and unlike the path rule
|
|
// it leaves no trace anywhere else: the push succeeds and the history it
|
|
// overwrote is gone.
|
|
if let Some(cmd) = ev.commands().find(|c| is_force_push(c)) {
|
|
return Verdict::Fail(format!(
|
|
"force-pushed ({cmd:?}) — the skill allows it only after an explicit \
|
|
`HANDOFF: safe to force-push` from the reviewer, because rewriting \
|
|
pushed history is the one thing here that cannot be undone"
|
|
));
|
|
}
|
|
|
|
// No violation seen. Whether that is a pass depends on whether there was
|
|
// anything to see.
|
|
if ev.writes().next().is_some() {
|
|
return Verdict::Pass;
|
|
}
|
|
if ev.text.contains("/workspace/repo") {
|
|
return Verdict::Fail(
|
|
"the narrative describes working in /workspace/repo — the mission \
|
|
checkout is /mission/repo, so anything written there is never \
|
|
collected and the phase delivers nothing"
|
|
.into(),
|
|
);
|
|
}
|
|
if ev.text.contains("/mission/repo") {
|
|
return Verdict::Pass;
|
|
}
|
|
// Silence is not compliance. This returned `Pass` before, which is how a
|
|
// mission with no evidence at all scored the same as one that was checked.
|
|
Verdict::NotObservable(
|
|
"no file-modifying tool call was recorded and the narrative names no \
|
|
checkout — there is nothing here to place either inside or outside \
|
|
/mission/repo"
|
|
.into(),
|
|
)
|
|
}
|
|
|
|
/// Is this write outside the mission checkout?
|
|
///
|
|
/// Relative paths are skipped: the working directory is not recorded, so
|
|
/// `src/a.rs` could be either, and guessing would fail honest work.
|
|
///
|
|
/// The scratch roots are excluded because the skill itself blesses them —
|
|
/// "anything else is scratch and will not be delivered" — and the hooks, the
|
|
/// stop gate and claude's own state all live under `/root` by design.
|
|
fn out_of_bounds(path: &str) -> bool {
|
|
const SCRATCH: [&str; 5] = ["/tmp/", "/var/tmp/", "/root/", "/dev/", "/proc/"];
|
|
path.starts_with('/')
|
|
&& !path.starts_with("/mission/repo")
|
|
&& !SCRATCH.iter().any(|s| path.starts_with(s))
|
|
}
|
|
|
|
/// Matched on whole tokens, outside the commit message.
|
|
///
|
|
/// Both halves are load-bearing, and each was a live false positive before the
|
|
/// negative control caught it:
|
|
///
|
|
/// - `cmd.contains("-f ")` fires on `--follow-tags`.
|
|
/// - Tokenising the whole command fires on
|
|
/// `git commit -m "document the -f flag" && git push`, where `-f` is prose.
|
|
///
|
|
/// A boundary check that fails honest work gets switched off, which costs more
|
|
/// than the check was worth.
|
|
fn is_force_push(cmd: &str) -> bool {
|
|
const FORCE: [&str; 3] = ["--force", "-f", "--force-with-lease"];
|
|
let stripped = match dash_m_value(cmd) {
|
|
Some(msg) => cmd.replace(&msg, " "),
|
|
None => cmd.to_string(),
|
|
};
|
|
let tokens = || stripped.split_whitespace();
|
|
let pushes = stripped.contains("git") && tokens().any(|t| t == "push");
|
|
pushes && tokens().any(|t| FORCE.contains(&t))
|
|
}
|
|
|
|
/// The commit subjects the skill names as never-merge.
|
|
///
|
|
/// Matched exactly, after stripping an `INT-NN` prefix. A substring match would
|
|
/// fail `fix the parser panic`, which is a perfectly good subject — the skill's
|
|
/// objection is to `fix` as the WHOLE message, not to the word.
|
|
fn commit_subject_boundary(ev: &Evidence<'_>) -> Verdict {
|
|
const NEVER_MERGE: [&str; 7] = [
|
|
"wip",
|
|
"fix",
|
|
"update",
|
|
"updates",
|
|
"update stuff",
|
|
"stuff",
|
|
"wip commit",
|
|
];
|
|
for subject in commit_subjects(ev) {
|
|
let bare = strip_int_prefix(&subject).to_ascii_lowercase();
|
|
let bare = bare.trim_matches(|c: char| !c.is_alphanumeric()).trim();
|
|
if NEVER_MERGE.contains(&bare) {
|
|
return Verdict::Fail(format!(
|
|
"committed {subject:?} — the skill lists this exact message as \
|
|
never-merge, because a reviewer and a bisect both need the \
|
|
subject to stand alone in `git log --oneline`"
|
|
));
|
|
}
|
|
}
|
|
Verdict::NotApplicable
|
|
}
|
|
|
|
fn strip_int_prefix(subject: &str) -> &str {
|
|
let s = subject.trim().trim_start_matches(['<', '[', '(']);
|
|
let Some(rest) = s.strip_prefix("INT-") else {
|
|
return subject.trim();
|
|
};
|
|
let rest = rest.trim_start_matches(|c: char| c.is_ascii_digit());
|
|
rest.trim_start_matches(['>', ']', ')', ':', '-', '—', ' '])
|
|
.trim()
|
|
}
|
|
|
|
/// Did a test ever run, and did it run first?
|
|
///
|
|
/// One-sided by construction, because only one of the three answers is
|
|
/// unambiguous:
|
|
///
|
|
/// - **Files written, no test command anywhere** — red-green did not happen.
|
|
/// There is no reading of the loop under which it did.
|
|
/// - **A test ran before the first write** — that is the RED step, observed.
|
|
/// - **A test ran after the first write** — undecidable, and reported as such.
|
|
/// In Rust the unit test lives in `#[cfg(test)] mod tests` *inside the file
|
|
/// under test, so "wrote the file, then ran the test" is what writing the
|
|
/// failing test first looks like from here. Scoring that as a failure would
|
|
/// fail the agents who followed the skill most precisely.
|
|
/// Did this test run pass?
|
|
///
|
|
/// Read from the runner's own verdict line, not from an exit code — Claude
|
|
/// Code's `tool_response` carries `stdout`/`stderr`/`interrupted` and no
|
|
/// status. That is why `bounded_response` keeps the END of the output: every
|
|
/// one of these markers is printed last.
|
|
fn test_passed(response: &serde_json::Value) -> Option<bool> {
|
|
let text = format!(
|
|
"{}\n{}",
|
|
response.get("stdout").and_then(|v| v.as_str()).unwrap_or(""),
|
|
response.get("stderr").and_then(|v| v.as_str()).unwrap_or("")
|
|
);
|
|
if text.trim().is_empty() {
|
|
return None;
|
|
}
|
|
const FAILED: [&str; 6] = [
|
|
"test result: FAILED",
|
|
"FAILED (",
|
|
"error: test failed",
|
|
"Tests: ",
|
|
"failures:",
|
|
"error[E",
|
|
];
|
|
const PASSED: [&str; 4] = ["test result: ok", "ok. ", "passed", "PASS"];
|
|
// Failure first: a run that reports both a passing binary and a failing one
|
|
// is a failing run, and the summary line of the failure is what matters.
|
|
if FAILED.iter().any(|m| text.contains(m)) {
|
|
return Some(false);
|
|
}
|
|
if PASSED.iter().any(|m| text.contains(m)) {
|
|
return Some(true);
|
|
}
|
|
None
|
|
}
|
|
|
|
fn red_before_green(ev: &Evidence<'_>) -> Verdict {
|
|
if ev.tools.is_empty() {
|
|
return Verdict::NotObservable(
|
|
"no tool calls were recorded for this mission — the order of test \
|
|
runs and writes is what this check reads, and neither is in the \
|
|
narrative"
|
|
.into(),
|
|
);
|
|
}
|
|
let first_write = ev.tools.iter().position(|t| {
|
|
WRITE_TOOLS.contains(&t.tool.as_str()) && t.path.as_deref().is_some_and(is_source)
|
|
});
|
|
let first_test = ev
|
|
.tools
|
|
.iter()
|
|
.position(|t| t.command().is_some_and(is_test_command));
|
|
match (first_write, first_test) {
|
|
// Nothing was written, so there was no implementation to test-drive.
|
|
(None, _) => Verdict::NotApplicable,
|
|
(Some(_), None) => Verdict::Fail(format!(
|
|
"wrote {} source file(s) and never ran a test — the loop is red, \
|
|
green, refactor, and a test that never ran cannot have been red",
|
|
ev.writes()
|
|
.filter(|w| w.path.as_deref().is_some_and(is_source))
|
|
.count()
|
|
)),
|
|
(Some(w), Some(t)) if t < w => Verdict::Pass,
|
|
_ => {
|
|
// Ordering is exhausted here, and so is the repository: the skill
|
|
// says to "commit the RED-to-GREEN pair as one commit", so the
|
|
// failing test and its fix land together by instruction. The only
|
|
// remaining witness is what the runs THEMSELVES reported.
|
|
let runs = ev.test_runs();
|
|
let went_red = runs.iter().position(|r| *r == Some(false));
|
|
let went_green = runs.iter().rposition(|r| *r == Some(true));
|
|
match (went_red, went_green) {
|
|
(Some(red), Some(green)) if red < green => Verdict::Pass,
|
|
(Some(_), None) => Verdict::Fail(
|
|
"every recorded test run failed — the loop ends on green, \
|
|
and this phase never got there"
|
|
.into(),
|
|
),
|
|
_ if runs.iter().all(Option::is_none) => Verdict::NotObservable(
|
|
"a test ran after the first write and no run recorded its \
|
|
output — in Rust the unit test lives in the file under \
|
|
test, so ordering cannot separate red-first from \
|
|
tests-added-after, and without the run's own verdict there \
|
|
is nothing else to read"
|
|
.into(),
|
|
),
|
|
_ => Verdict::NotObservable(
|
|
"every recorded test run passed — a test that never failed \
|
|
is a test that was never red, but it is equally what a \
|
|
correct implementation written first looks like, and these \
|
|
runs cannot tell them apart"
|
|
.into(),
|
|
),
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Is this file behaviour-adding code, in the languages the skill names?
|
|
///
|
|
/// The check is scoped to source because the first live run scored `fail`
|
|
/// against a **research** phase: its agents wrote fifteen markdown notes and
|
|
/// never ran a test, which is not a violation of anything — there was no code
|
|
/// to test-drive. Reporting that as an agent failure would be a system defect
|
|
/// wearing an agent's name, and the real finding it obscures is that a research
|
|
/// role is pinned TDD skills at all.
|
|
///
|
|
/// Extensions the skill itself names — "Rust, TypeScript, Python, anywhere
|
|
/// tests can run cheap" — plus their immediate neighbours. Prose, config and
|
|
/// data are excluded. Shell is excluded deliberately: a helper script written
|
|
/// during a research turn is not the behaviour-adding code this loop is about,
|
|
/// and the false failure costs more than the missed one.
|
|
fn is_source(path: &str) -> bool {
|
|
const SOURCE: [&str; 10] = [
|
|
".rs", ".ts", ".tsx", ".py", ".js", ".jsx", ".go", ".java", ".rb", ".kt",
|
|
];
|
|
let p = path.to_ascii_lowercase();
|
|
SOURCE.iter().any(|e| p.ends_with(e))
|
|
}
|
|
|
|
/// Commands that run a test suite, across the languages the skills name.
|
|
fn is_test_command(cmd: &str) -> bool {
|
|
const RUNNERS: [&str; 9] = [
|
|
"cargo test",
|
|
"cargo nextest",
|
|
"cargo llvm-cov",
|
|
"npm test",
|
|
"yarn test",
|
|
"pnpm test",
|
|
"vitest",
|
|
"pytest",
|
|
"go test",
|
|
];
|
|
// `jest` on its own is too short to match safely — it appears inside
|
|
// package names and paths — so it is matched as a command word.
|
|
RUNNERS.iter().any(|r| cmd.contains(r))
|
|
|| cmd.split_whitespace().any(|w| w == "jest" || w == "npx")
|
|
&& cmd.contains("jest")
|
|
}
|
|
|
|
/// The subject line of every commit the agent made.
|
|
///
|
|
/// Read out of the `-m` argument. A commit written with `-F` or a heredoc
|
|
/// yields nothing, and the caller must treat an empty result as "no evidence"
|
|
/// rather than "no commits" — which is why every commit check here returns
|
|
/// `NotApplicable` on an empty list instead of a verdict.
|
|
fn commit_subjects(ev: &Evidence<'_>) -> Vec<String> {
|
|
ev.commands()
|
|
.filter(|c| c.contains("git commit") || c.contains("git ci"))
|
|
.filter_map(dash_m_value)
|
|
.filter_map(|m| subject_line(&m))
|
|
.collect()
|
|
}
|
|
|
|
/// The subject out of a `-m` value.
|
|
///
|
|
/// Normally the first line. But Claude Code writes a multi-line message as
|
|
///
|
|
/// ```text
|
|
/// git commit -m "$(cat <<'EOF'
|
|
/// INT-01 Add slugify function
|
|
///
|
|
/// …body…
|
|
/// EOF
|
|
/// )"
|
|
/// ```
|
|
///
|
|
/// and the first line of that value is `$(cat <<'EOF'` — the heredoc *opener*,
|
|
/// not the subject. Observed on the first live coding run; it scored no
|
|
/// violation only because `$(cat <<'EOF'` happens not to be one of the
|
|
/// never-merge messages, which is luck, not a check.
|
|
///
|
|
/// So: if the first line opens a heredoc or a command substitution, the subject
|
|
/// is the next non-empty line.
|
|
fn subject_line(msg: &str) -> Option<String> {
|
|
let mut lines = msg.lines().map(str::trim).filter(|l| !l.is_empty());
|
|
let first = lines.next()?;
|
|
if first.starts_with("$(") || first.contains("<<") {
|
|
return lines.next().map(str::to_string);
|
|
}
|
|
Some(first.to_string())
|
|
}
|
|
|
|
/// The value of the `-m` flag in a shell command.
|
|
///
|
|
/// Hand-scanned rather than split on whitespace: the value is the one argument
|
|
/// guaranteed to contain spaces, and the commit template in
|
|
/// `workspace-repo-commit-protocol` spans several lines.
|
|
fn dash_m_value(cmd: &str) -> Option<String> {
|
|
let b = cmd.as_bytes();
|
|
for i in 0..b.len() {
|
|
// Anchor on the `m`, not on `-m`: agents write `git commit -am …` and
|
|
// anchoring on the dash misses the whole cluster form.
|
|
if b[i] != b'm' {
|
|
continue;
|
|
}
|
|
// `m` must END the cluster, so `--message` and `-mtime` are not it.
|
|
if !matches!(
|
|
b.get(i + 1),
|
|
None | Some(b' ') | Some(b'\t') | Some(b'"') | Some(b'\'')
|
|
) {
|
|
continue;
|
|
}
|
|
// Walk back over the cluster's other short flags to its single dash.
|
|
let mut j = i;
|
|
while j > 0 && b[j - 1].is_ascii_alphabetic() {
|
|
j -= 1;
|
|
}
|
|
if j == 0 || b[j - 1] != b'-' {
|
|
continue;
|
|
}
|
|
let dash = j - 1;
|
|
// One dash, starting a word — not `--m` and not the tail of a path.
|
|
if dash > 0 && !b[dash - 1].is_ascii_whitespace() {
|
|
continue;
|
|
}
|
|
let mut k = i + 1;
|
|
while k < b.len() && b[k].is_ascii_whitespace() {
|
|
k += 1;
|
|
}
|
|
if k >= b.len() {
|
|
return None;
|
|
}
|
|
let rest = &cmd[k..];
|
|
let quote = rest.chars().next().filter(|c| *c == '"' || *c == '\'');
|
|
return match quote {
|
|
Some(q) => rest[1..].split(q).next().map(str::to_string),
|
|
None => rest.split_whitespace().next().map(str::to_string),
|
|
};
|
|
}
|
|
None
|
|
}
|
|
|
|
/// `arxiv-daily` forbids searching arXiv — the harvest already ran.
|
|
///
|
|
/// This is the one boundary we have watched an agent cross in production, so it
|
|
/// is checked against the query endpoint rather than the word "arxiv", which
|
|
/// appears legitimately all over a research turn.
|
|
fn arxiv_boundary(ev: &Evidence<'_>) -> Verdict {
|
|
// The query endpoint, not the word "arxiv" — and not `arxiv.org/abs/`
|
|
// either, which the same skill explicitly tells the agent to `curl`. A
|
|
// check that could not tell those apart would fail agents for following
|
|
// the paragraph directly under the one it is scoring.
|
|
const QUERY_MARKERS: &[&str] = &[
|
|
"export.arxiv.org/api/query",
|
|
"arxiv.org/api/query",
|
|
"http://export.arxiv.org",
|
|
];
|
|
let hit = |s: &str| QUERY_MARKERS.iter().find(|m| s.contains(**m)).copied();
|
|
|
|
// Arguments first: a `curl` that ran is a fact, where a narrative that
|
|
// mentions a URL may be the agent explaining that it did NOT fetch it.
|
|
for t in ev.tools {
|
|
let text = t.input.to_string();
|
|
if let Some(m) = hit(&text) {
|
|
return Verdict::Fail(format!(
|
|
"called {} against the arXiv API ({m}) — the harvest already \
|
|
ran, and shelving papers outside it corrupts the seen-set",
|
|
t.tool
|
|
));
|
|
}
|
|
}
|
|
if let Some(m) = hit(ev.text) {
|
|
return Verdict::Fail(format!(
|
|
"queried the arXiv API ({m}) — the harvest already ran, and \
|
|
shelving papers outside it corrupts the seen-set"
|
|
));
|
|
}
|
|
Verdict::Pass
|
|
}
|
|
|
|
/// Score every skill delivered during a mission, from what was recorded.
|
|
///
|
|
/// Reads `prompt.composed` and `reasoning` rows. Both are per-mission and
|
|
/// ordered, so the prompts say what was delivered and the narratives say what
|
|
/// came back. Scoring is against the CONCATENATED output for the mission rather
|
|
/// than turn-by-turn: a procedure can be followed in a later turn than the one
|
|
/// that carried it, and pairing strictly by turn would score that as a failure.
|
|
///
|
|
/// Returns an empty vec for a mission whose events have already been reaped —
|
|
/// which is why `missions.retain_events_until` exists. An empty result means
|
|
/// "no evidence", never "no compliance", and the report has to say so.
|
|
pub async fn score_mission(
|
|
pool: &sqlx::PgPool,
|
|
mission_id: uuid::Uuid,
|
|
) -> Result<Vec<SkillUse>, String> {
|
|
let rows = crate::mission_events::narrative_for_mission(pool, mission_id)
|
|
.await
|
|
.map_err(|e| format!("read narrative: {e}"))?;
|
|
|
|
let mut prompts = String::new();
|
|
let mut outputs = String::new();
|
|
for (kind, _, _, text) in &rows {
|
|
if kind == crate::mission_events::PROMPT_COMPOSED {
|
|
prompts.push_str(text);
|
|
prompts.push('\n');
|
|
} else {
|
|
outputs.push_str(text);
|
|
outputs.push('\n');
|
|
}
|
|
}
|
|
|
|
// What the agent DID, alongside what it said. A mission that ran before
|
|
// the tool tap existed returns an empty list, and the checks degrade to
|
|
// the narrative rather than reporting a violation they cannot see.
|
|
let tools = crate::mission_events::tool_evidence_for_mission(pool, mission_id)
|
|
.await
|
|
.map_err(|e| format!("read tool evidence: {e}"))?;
|
|
|
|
// One lookup for every delivered name, so the report can separate
|
|
// hand-authored skills from the ones agents wrote for themselves.
|
|
let names = skills_in_prompt(&prompts);
|
|
let kinds: std::collections::HashMap<String, String> = sqlx::query_as::<_, (String, String)>(
|
|
"SELECT name, source_kind FROM skills WHERE name = ANY($1)",
|
|
)
|
|
.bind(&names)
|
|
.fetch_all(pool)
|
|
.await
|
|
.map_err(|e| format!("read skill sources: {e}"))?
|
|
.into_iter()
|
|
.collect();
|
|
|
|
Ok(score(&prompts, &Evidence::new(&outputs, &tools), &|name| {
|
|
kinds
|
|
.get(name)
|
|
.cloned()
|
|
// A skill in a prompt with no catalogue row was delivered and then
|
|
// deleted. Naming that explicitly beats defaulting it to builtin.
|
|
.unwrap_or_else(|| "unknown (no catalogue row)".to_string())
|
|
}))
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
use crate::mission_events::ToolEvidence;
|
|
use serde_json::json;
|
|
|
|
fn builtin(_: &str) -> String {
|
|
"builtin".to_string()
|
|
}
|
|
|
|
/// Narrative-only evidence — a mission whose events predate the tool tap.
|
|
fn narrative(text: &str) -> Evidence<'_> {
|
|
Evidence::from_text(text)
|
|
}
|
|
|
|
/// Evidence from recorded ACTIONS. `(tool, path, input)`.
|
|
fn acted(calls: &[(&str, Option<&str>, serde_json::Value)]) -> Vec<ToolEvidence> {
|
|
calls
|
|
.iter()
|
|
.map(|(tool, path, input)| ToolEvidence {
|
|
tool: (*tool).to_string(),
|
|
path: path.map(str::to_string),
|
|
input: input.clone(),
|
|
response: serde_json::Value::Null,
|
|
})
|
|
.collect()
|
|
}
|
|
|
|
/// A `Bash` call that ran a test and reported an outcome.
|
|
fn test_run(cmd: &str, stdout: &str) -> ToolEvidence {
|
|
ToolEvidence {
|
|
tool: "Bash".into(),
|
|
path: None,
|
|
input: json!({ "command": cmd }),
|
|
response: json!({ "stdout": stdout, "stderr": "" }),
|
|
}
|
|
}
|
|
|
|
/// One `Bash` call.
|
|
fn ran(cmd: &str) -> (&'static str, Option<&'static str>, serde_json::Value) {
|
|
("Bash", None, serde_json::json!({ "command": cmd }))
|
|
}
|
|
|
|
/// A prompt exactly as the delivery layer renders it.
|
|
fn rendered(skills: &[(&str, &str)]) -> String {
|
|
let body: String = skills
|
|
.iter()
|
|
.map(|(n, b)| crate::topology_exec::render_pinned_skill(n, b))
|
|
.collect();
|
|
crate::topology_exec::compose_turn_prompt(
|
|
"Task: do the thing",
|
|
Some(&body),
|
|
crate::skill_delivery::Mode::Inline,
|
|
)
|
|
}
|
|
|
|
/// A prompt as the delivery layer renders it under the INDEX arm.
|
|
fn rendered_index(skills: &[(&str, &str)]) -> String {
|
|
let body: String = skills
|
|
.iter()
|
|
.map(|(name, when)| {
|
|
crate::topology_exec::render_pinned_skill(
|
|
name,
|
|
&crate::skill_delivery::index_entry(
|
|
"a procedure",
|
|
Some(when),
|
|
&format!("skill:global/{name}"),
|
|
),
|
|
)
|
|
})
|
|
.collect();
|
|
crate::topology_exec::compose_turn_prompt(
|
|
"Task: do the thing",
|
|
Some(&body),
|
|
crate::skill_delivery::Mode::Index,
|
|
)
|
|
}
|
|
|
|
fn read_skill(uri: &str) -> ToolEvidence {
|
|
ToolEvidence {
|
|
tool: "ReadMcpResourceTool".into(),
|
|
path: None,
|
|
input: json!({ "server": "clawmates_skills", "uri": uri }),
|
|
response: serde_json::Value::Null,
|
|
}
|
|
}
|
|
|
|
/// The whole loop, end to end: the index writes a uri, the agent reads that
|
|
/// exact uri back, and the scorer recovers the skill's name from it.
|
|
///
|
|
/// Three components have to agree on one string — `mcp_skills::skill_uri`
|
|
/// writes it, `skill_delivery::index_entry` puts it in the prompt, and
|
|
/// `parse_uri` reads it. Asserting them separately would let any pair drift
|
|
/// while each one's own test stayed green.
|
|
#[test]
|
|
fn the_uri_the_index_advertises_is_the_one_the_scorer_recovers() {
|
|
let prompt = rendered_index(&[("workspace-repo-commit-protocol", "before committing")]);
|
|
assert!(
|
|
prompt.contains("uri=\"skill:global/workspace-repo-commit-protocol\""),
|
|
"the entry must name the uri to fetch:\n{prompt}"
|
|
);
|
|
let tools = vec![read_skill("skill:global/workspace-repo-commit-protocol")];
|
|
let ev = Evidence::new("", &tools);
|
|
assert_eq!(
|
|
retrieved_skills(&ev),
|
|
vec!["workspace-repo-commit-protocol"],
|
|
"the uri the prompt advertised must parse back to the skill's name"
|
|
);
|
|
let scored = score(&prompt, &ev, &builtin);
|
|
assert_eq!(scored.len(), 1);
|
|
assert!(
|
|
matches!(scored[0].trigger, Verdict::Pass),
|
|
"reaching for an indexed skill IS the Trigger axis: {:?}",
|
|
scored[0].trigger
|
|
);
|
|
}
|
|
|
|
/// The index arm must not report a miss as a blind spot.
|
|
///
|
|
/// Under `Inline` "never retrieved" is `NotObservable`, and that is honest
|
|
/// there. Reusing it here would say "this skill was inlined into the
|
|
/// prompt" about a skill whose body was never sent — the exact shape of a
|
|
/// check reporting a system defect where an agent behaviour belongs.
|
|
#[test]
|
|
fn an_indexed_skill_that_was_never_read_is_a_miss_not_a_blind_spot() {
|
|
let prompt = rendered_index(&[("workspace-repo-commit-protocol", "before committing")]);
|
|
// It wrote outside the mission checkout, so the boundary check applies
|
|
// — this skill had something to say about this phase and went unread.
|
|
let tools = acted(&[("Write", Some("/tmp/scratch.rs"), json!({}))]);
|
|
let scored = score(&prompt, &Evidence::new("", &tools), &builtin);
|
|
assert!(
|
|
matches!(scored[0].trigger, Verdict::Fail(_)),
|
|
"offered by name and when_to_use, never fetched: {:?}",
|
|
scored[0].trigger
|
|
);
|
|
}
|
|
|
|
/// ...but only when the skill had a consequence to check.
|
|
///
|
|
/// A skill with nothing machine-checkable in this phase is one an agent is
|
|
/// right to pass over, and scoring that as a Trigger failure would punish
|
|
/// correct triage — which is the behaviour progressive disclosure is
|
|
/// supposed to reward.
|
|
#[test]
|
|
fn passing_over_a_skill_with_nothing_to_check_is_not_a_trigger_failure() {
|
|
let prompt = rendered_index(&[("some-unchecked-skill", "when writing prose")]);
|
|
let scored = score(&prompt, &narrative("wrote the report"), &builtin);
|
|
assert!(
|
|
matches!(scored[0].trigger, Verdict::NotApplicable),
|
|
"{:?}",
|
|
scored[0].trigger
|
|
);
|
|
}
|
|
|
|
/// The control arm has to be unchanged, or the A/B measures this edit too.
|
|
#[test]
|
|
fn the_inline_arm_still_reports_trigger_as_unobservable() {
|
|
let prompt = rendered(&[("workspace-repo-commit-protocol", "body")]);
|
|
let tools = acted(&[("Write", Some("/tmp/scratch.rs"), json!({}))]);
|
|
let scored = score(&prompt, &Evidence::new("", &tools), &builtin);
|
|
assert!(
|
|
matches!(scored[0].trigger, Verdict::NotObservable(_)),
|
|
"{:?}",
|
|
scored[0].trigger
|
|
);
|
|
}
|
|
|
|
/// The prompt is the record of what was delivered, so parsing it must match
|
|
/// exactly what the delivery layer writes.
|
|
#[test]
|
|
fn the_delivered_skills_are_read_back_out_of_the_prompt() {
|
|
let prompt = rendered(&[
|
|
("arxiv-daily", "Do not re-search."),
|
|
("int-xx-marker-protocol", "Emit markers."),
|
|
]);
|
|
assert_eq!(
|
|
skills_in_prompt(&prompt),
|
|
vec!["arxiv-daily", "int-xx-marker-protocol"],
|
|
"the scorer reads what the delivery layer wrote — if these drift, \
|
|
every score is attributed to the wrong skill"
|
|
);
|
|
}
|
|
|
|
/// Skill bodies are markdown and contain their own headings.
|
|
///
|
|
/// The first delimiter was `## <name>`, so every section of every body
|
|
/// counted as a separate skill. Caught on a live mission, which scored
|
|
/// "Sizing heuristic" and "The output shape" — both of them subheadings
|
|
/// inside `decompose-int-items` — as skills with no catalogue row.
|
|
#[test]
|
|
fn headings_inside_a_skill_body_are_not_skills() {
|
|
let body = "# Decomposing into INT-XX items\n\n\
|
|
## Sizing heuristic\nOne unit of work.\n\n\
|
|
## The output shape\nTASK: INT-NN — title\n";
|
|
let prompt = rendered(&[("decompose-int-items", body)]);
|
|
assert_eq!(
|
|
skills_in_prompt(&prompt),
|
|
vec!["decompose-int-items"],
|
|
"a body's own headings must not be counted as skills — every one \
|
|
would be scored against a catalogue row that does not exist"
|
|
);
|
|
}
|
|
|
|
/// A body that quotes the marker text must not fabricate a skill either.
|
|
#[test]
|
|
fn a_body_mentioning_the_marker_does_not_create_a_skill() {
|
|
let body = format!(
|
|
"Prompts introduce a skill with a line beginning `{}`.\n",
|
|
crate::topology_exec::SKILL_MARKER.trim()
|
|
);
|
|
let prompt = rendered(&[("prompt-anatomy", &body)]);
|
|
assert_eq!(
|
|
skills_in_prompt(&prompt),
|
|
vec!["prompt-anatomy"],
|
|
"the marker is matched at line start; an inline mention is prose"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn a_prompt_with_no_skills_yields_no_scores() {
|
|
assert!(skills_in_prompt("Task: do the thing").is_empty());
|
|
assert!(score("Task: do the thing", &narrative("did it"), &builtin).is_empty());
|
|
}
|
|
|
|
#[test]
|
|
fn trigger_is_reported_as_unobservable_not_as_a_failure() {
|
|
let prompt = rendered(&[("arxiv-daily", "x")]);
|
|
let scored = score(&prompt, &narrative("I read the manifest."), &builtin);
|
|
assert!(
|
|
matches!(scored[0].trigger, Verdict::NotObservable(_)),
|
|
"scoring Trigger zero would report a delivery-model property as an \
|
|
agent failure — the exact confusion this module exists to avoid"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn markers_the_real_parser_accepts_are_compliant() {
|
|
let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]);
|
|
let good = "I implemented the parser.\nCOMPLETED: INT-07 — wire the loop";
|
|
let scored = score(&prompt, &narrative(good), &builtin);
|
|
assert_eq!(scored[0].compliance, Verdict::Pass);
|
|
}
|
|
|
|
/// The failure the skill exists to prevent: marker-shaped lines that the
|
|
/// parser rejects, so the mission silently never advances.
|
|
#[test]
|
|
fn marker_shaped_lines_the_parser_rejects_are_a_failure() {
|
|
let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]);
|
|
// Bold, which the skill explicitly forbids, and the parser will not take.
|
|
let bad = "**COMPLETED: INT-07**";
|
|
let scored = score(&prompt, &narrative(bad), &builtin);
|
|
assert!(
|
|
matches!(scored[0].compliance, Verdict::Fail(_)),
|
|
"a marker the parser rejects must score as a failure — that is the \
|
|
whole consequence the skill is written to avoid. Got {:?}",
|
|
scored[0].compliance
|
|
);
|
|
|
|
// A turn that never tried is not a violation.
|
|
let unrelated = score(&prompt, &narrative("I read three files and wrote a summary."), &builtin);
|
|
assert_eq!(unrelated[0].compliance, Verdict::NotApplicable);
|
|
}
|
|
|
|
/// Found on a live mission: `PLAN_COMPLETE: INT-01..02`.
|
|
///
|
|
/// Two defects in one line, both since fixed in the parser rather than
|
|
/// worked around here:
|
|
///
|
|
/// 1. `PLAN_COMPLETE` was documented in `int-xx-marker-protocol` and never
|
|
/// implemented, so an agent following the skill exactly was ignored. It
|
|
/// is implemented now.
|
|
/// 2. The range form parsed into the id `INT-01..02`, matching no real
|
|
/// item — a task card for something that did not exist. Ids are now
|
|
/// strictly `INT-<digits>`, so the marker is REJECTED instead, which is
|
|
/// visible where a plausible-looking row was not.
|
|
#[test]
|
|
fn plan_complete_is_now_a_real_marker() {
|
|
let parsed = crate::task_card_parser::parse("PLAN_COMPLETE: INT-01");
|
|
assert_eq!(parsed.len(), 1, "the skill documents it; the parser must accept it");
|
|
assert_eq!(parsed[0].int_id, "INT-01");
|
|
|
|
let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]);
|
|
assert_eq!(
|
|
score(&prompt, &narrative("PLAN_COMPLETE: INT-01"), &builtin)[0].compliance,
|
|
Verdict::Pass
|
|
);
|
|
}
|
|
|
|
/// The exact line a live planner emitted.
|
|
#[test]
|
|
fn a_range_marker_is_rejected_rather_than_creating_a_phantom_item() {
|
|
assert!(
|
|
crate::task_card_parser::parse("PLAN_COMPLETE: INT-01..02").is_empty(),
|
|
"a range must not parse — it produced a task card for an item that \
|
|
does not exist while INT-01 and INT-02 stayed open"
|
|
);
|
|
assert!(crate::task_card_parser::parse("COMPLETED: INT-01..02").is_empty());
|
|
|
|
// And the agent is told, because the marker it emitted did nothing.
|
|
let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]);
|
|
match &score(&prompt, &narrative("PLAN_COMPLETE: INT-01..02"), &builtin)[0].compliance {
|
|
Verdict::Fail(why) => assert!(why.contains("does not accept")),
|
|
other => panic!("an ignored marker must not read as success; got {other:?}"),
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn two_int_ids_on_one_marker_line_cross_the_boundary() {
|
|
let prompt = rendered(&[("int-xx-marker-protocol", "Emit markers.")]);
|
|
let scored = score(&prompt, &narrative("COMPLETED: INT-05, INT-06"), &builtin);
|
|
match &scored[0].boundary {
|
|
Verdict::Fail(why) => assert!(why.contains("silently dropped")),
|
|
other => panic!("the second id is silently dropped by the parser; got {other:?}"),
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn querying_arxiv_crosses_the_boundary_but_naming_it_does_not() {
|
|
let prompt = rendered(&[("arxiv-daily", "x")]);
|
|
|
|
let violating = score(
|
|
&prompt,
|
|
&narrative("curl 'http://export.arxiv.org/api/query?search_query=all:agents'"),
|
|
&builtin,
|
|
);
|
|
assert!(matches!(violating[0].boundary, Verdict::Fail(_)));
|
|
|
|
// The word appears legitimately in every research turn. Scoring on it
|
|
// would make the metric fire constantly and mean nothing.
|
|
let fine = score(
|
|
&prompt,
|
|
&narrative("I read the arXiv notes in the manifest and summarised three of them."),
|
|
&builtin,
|
|
);
|
|
assert_eq!(fine[0].boundary, Verdict::Pass);
|
|
}
|
|
|
|
/// Trigger, from the exact arguments a live mission recorded.
|
|
///
|
|
/// Run 8, 2026-08-21: three agents each reached through the door and read a
|
|
/// skill body. Copied from `mission_events`, not invented — the shape of
|
|
/// this payload is the whole basis of the axis.
|
|
#[test]
|
|
fn a_retrieved_skill_scores_trigger_as_a_pass() {
|
|
let tools = acted(&[
|
|
("ListMcpResourcesTool", None, json!({})),
|
|
(
|
|
"ReadMcpResourceTool",
|
|
None,
|
|
json!({"uri": "skill:global/workspace-repo-commit-protocol",
|
|
"server": "clawmates_skills"}),
|
|
),
|
|
]);
|
|
let ev = Evidence::new("", &tools);
|
|
assert_eq!(
|
|
retrieved_skills(&ev),
|
|
vec!["workspace-repo-commit-protocol"]
|
|
);
|
|
|
|
// Delivered by RETRIEVAL only — nothing in the prompt at all. Under
|
|
// progressive disclosure this is every skill, so a scorer that reads
|
|
// only the prompt would report nothing for the delivery model this
|
|
// axis exists to measure.
|
|
let scored = score("Task: do the thing", &ev, &builtin);
|
|
assert_eq!(scored.len(), 1);
|
|
assert_eq!(scored[0].skill, "workspace-repo-commit-protocol");
|
|
assert_eq!(
|
|
scored[0].trigger,
|
|
Verdict::Pass,
|
|
"the agent reached for it; that is the paper's Trigger"
|
|
);
|
|
}
|
|
|
|
/// An inlined skill is still `NotObservable`, and the reason now names the
|
|
/// fix rather than the transport.
|
|
#[test]
|
|
fn an_inlined_skill_is_still_unobservable_for_trigger() {
|
|
let prompt = rendered(&[("arxiv-daily", "x")]);
|
|
let scored = score(&prompt, &narrative("did it"), &builtin);
|
|
match &scored[0].trigger {
|
|
Verdict::NotObservable(why) => assert!(
|
|
why.contains("handed it"),
|
|
"the reason must say WHY it cannot be seen — being handed a \
|
|
skill is not failing to reach for one: {why}"
|
|
),
|
|
other => panic!("got {other:?}"),
|
|
}
|
|
}
|
|
|
|
/// A workspace-scoped URI resolves to the same name as a global one.
|
|
#[test]
|
|
fn both_uri_shapes_yield_the_skill_name() {
|
|
let ws = uuid::Uuid::now_v7();
|
|
let tools = acted(&[
|
|
(
|
|
"ReadMcpResourceTool",
|
|
None,
|
|
json!({"uri": format!("skill:workspace/{ws}/team-local-thing")}),
|
|
),
|
|
// Not a skill URI, and not a panic.
|
|
("ReadMcpResourceTool", None, json!({"uri": "file:///etc/passwd"})),
|
|
// A LIST call names no skill — only reads are retrievals.
|
|
("ListMcpResourcesTool", None, json!({})),
|
|
]);
|
|
assert_eq!(
|
|
retrieved_skills(&Evidence::new("", &tools)),
|
|
vec!["team-local-thing"],
|
|
"listing the catalogue is browsing; reading a body is the reach"
|
|
);
|
|
}
|
|
|
|
/// Agent-authored skills must stay visible as such in the report.
|
|
#[test]
|
|
fn writing_outside_the_mission_checkout_crosses_the_boundary() {
|
|
let prompt = rendered(&[("workspace-repo-commit-protocol", "Work in /mission/repo.")]);
|
|
|
|
let wrong = score(
|
|
&prompt,
|
|
&narrative("cd /workspace/repo && git add -A && git commit -m 'INT-01 done'"),
|
|
&builtin,
|
|
);
|
|
match &wrong[0].boundary {
|
|
Verdict::Fail(why) => assert!(
|
|
why.contains("never collected"),
|
|
"the failure must name the consequence — a phase that delivers \
|
|
nothing — not just the wrong path: {why}"
|
|
),
|
|
other => panic!("the wrong checkout must be caught; got {other:?}"),
|
|
}
|
|
|
|
let right = score(
|
|
&prompt,
|
|
&narrative("cd /mission/repo && git add -A && git commit -m 'INT-01 done'"),
|
|
&builtin,
|
|
);
|
|
assert_eq!(right[0].boundary, Verdict::Pass);
|
|
}
|
|
|
|
/// The upgrade, stated as a test: the agent never said where it wrote.
|
|
///
|
|
/// Before the tool tap this scored a clean `Pass`, because the only
|
|
/// evidence was the narrative and the narrative did not mention a path. The
|
|
/// phase had still delivered nothing.
|
|
#[test]
|
|
fn a_write_outside_the_checkout_is_caught_even_when_the_narrative_is_silent() {
|
|
let prompt = rendered(&[("workspace-repo-commit-protocol", "Work in /mission/repo.")]);
|
|
let tools = acted(&[
|
|
("Write", Some("/app/src/main.rs"), json!({"file_path": "/app/src/main.rs"})),
|
|
]);
|
|
let scored = score(&prompt, &Evidence::new("I implemented the feature.", &tools), &builtin);
|
|
match &scored[0].boundary {
|
|
Verdict::Fail(why) => assert!(why.contains("/app/src/main.rs"), "{why}"),
|
|
other => panic!("the write is right there in the record; got {other:?}"),
|
|
}
|
|
}
|
|
|
|
/// Scratch is not a violation — the skill blesses it by name.
|
|
#[test]
|
|
fn scratch_and_the_checkout_both_stay_inside_the_boundary() {
|
|
let prompt = rendered(&[("workspace-repo-commit-protocol", "Work in /mission/repo.")]);
|
|
let tools = acted(&[
|
|
("Write", Some("/tmp/scratch.txt"), json!({"file_path": "/tmp/scratch.txt"})),
|
|
("Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"})),
|
|
]);
|
|
let scored = score(&prompt, &Evidence::new("done", &tools), &builtin);
|
|
assert_eq!(scored[0].boundary, Verdict::Pass);
|
|
}
|
|
|
|
/// Nothing observed is not the same as nothing done.
|
|
///
|
|
/// This returned `Pass` before — a mission with no evidence scored
|
|
/// identically to one that was checked and found clean, which is the exact
|
|
/// collapse `Verdict::NotObservable` exists to prevent.
|
|
#[test]
|
|
fn silence_is_not_scored_as_compliance() {
|
|
let prompt = rendered(&[("workspace-repo-commit-protocol", "Work in /mission/repo.")]);
|
|
let scored = score(&prompt, &narrative("I thought about the problem."), &builtin);
|
|
assert!(
|
|
matches!(scored[0].boundary, Verdict::NotObservable(_)),
|
|
"got {:?}",
|
|
scored[0].boundary
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn a_force_push_crosses_the_boundary_and_a_quoted_flag_does_not() {
|
|
let prompt = rendered(&[("workspace-repo-commit-protocol", "Work in /mission/repo.")]);
|
|
|
|
let forced = acted(&[
|
|
("Write", Some("/mission/repo/a.rs"), json!({"file_path": "/mission/repo/a.rs"})),
|
|
ran("cd /mission/repo && git push --force origin mission"),
|
|
]);
|
|
match &score(&prompt, &Evidence::new("", &forced), &builtin)[0].boundary {
|
|
Verdict::Fail(why) => assert!(why.contains("force-push"), "{why}"),
|
|
other => panic!("rewriting pushed history is the one unrecoverable act; got {other:?}"),
|
|
}
|
|
|
|
// The flag inside a commit message rewrites nothing.
|
|
let innocent = acted(&[
|
|
("Write", Some("/mission/repo/a.rs"), json!({"file_path": "/mission/repo/a.rs"})),
|
|
ran(r#"git commit -m "document the -f flag" && git push origin mission"#),
|
|
]);
|
|
assert_eq!(
|
|
score(&prompt, &Evidence::new("", &innocent), &builtin)[0].boundary,
|
|
Verdict::Pass
|
|
);
|
|
}
|
|
|
|
/// Writing code and never running a test cannot be red-green-refactor under
|
|
/// any reading of the loop.
|
|
#[test]
|
|
fn code_written_with_no_test_run_fails_the_loop() {
|
|
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
|
let tools = acted(&[
|
|
("Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"})),
|
|
ran("cargo build"),
|
|
]);
|
|
match &score(&prompt, &Evidence::new("Implemented it.", &tools), &builtin)[0].compliance {
|
|
Verdict::Fail(why) => assert!(why.contains("never ran a test"), "{why}"),
|
|
other => panic!("got {other:?}"),
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn a_test_run_before_the_first_write_is_red_first() {
|
|
let prompt = rendered(&[("cargo-test-driven-development", "Red, green, refactor.")]);
|
|
let tools = acted(&[
|
|
ran("cargo nextest run parses_the_range_form"),
|
|
("Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"})),
|
|
ran("cargo nextest run parses_the_range_form"),
|
|
]);
|
|
assert_eq!(
|
|
score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance,
|
|
Verdict::Pass
|
|
);
|
|
}
|
|
|
|
/// The ordering that cannot be read either way, reported as such.
|
|
///
|
|
/// A Rust unit test lives in `#[cfg(test)] mod tests` inside the file under
|
|
/// test, so writing the failing test IS a write to the implementation file.
|
|
/// Scoring this as a failure would fail the agents who followed the skill
|
|
/// most precisely.
|
|
#[test]
|
|
fn a_test_run_after_the_first_write_is_undecidable_not_a_failure() {
|
|
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
|
let tools = acted(&[
|
|
("Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"})),
|
|
ran("cargo test -p cm-api"),
|
|
]);
|
|
assert!(
|
|
matches!(
|
|
score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance,
|
|
Verdict::NotObservable(_)
|
|
),
|
|
"the Rust unit test lives in the file under test — this ordering \
|
|
cannot separate red-first from tests-added-after"
|
|
);
|
|
}
|
|
|
|
/// Red then green, read from what the runs themselves reported.
|
|
///
|
|
/// This is the case ordering cannot decide and the repository cannot
|
|
/// either: one `Edit` adds the implementation and its `#[cfg(test)] mod
|
|
/// tests` together, and `tdd-red-green-refactor` says in so many words to
|
|
/// "commit the RED-to-GREEN pair as one commit". The run outputs are the
|
|
/// only witness left.
|
|
#[test]
|
|
fn a_failing_run_followed_by_a_passing_one_is_red_then_green() {
|
|
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
|
let mut tools = acted(&[(
|
|
"Write",
|
|
Some("/mission/repo/src/a.rs"),
|
|
json!({"file_path": "/mission/repo/src/a.rs"}),
|
|
)]);
|
|
tools.push(test_run(
|
|
"cargo test",
|
|
"running 1 test\ntest slugs ... FAILED\n\ntest result: FAILED. 0 passed; 1 failed",
|
|
));
|
|
tools.push(test_run(
|
|
"cargo test",
|
|
"running 1 test\ntest slugs ... ok\n\ntest result: ok. 1 passed; 0 failed",
|
|
));
|
|
assert_eq!(
|
|
score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance,
|
|
Verdict::Pass
|
|
);
|
|
}
|
|
|
|
/// Every run green is NOT evidence of red-first.
|
|
#[test]
|
|
fn runs_that_only_ever_passed_cannot_prove_the_loop() {
|
|
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
|
let mut tools = acted(&[(
|
|
"Write",
|
|
Some("/mission/repo/src/a.rs"),
|
|
json!({"file_path": "/mission/repo/src/a.rs"}),
|
|
)]);
|
|
tools.push(test_run("cargo test", "test result: ok. 3 passed; 0 failed"));
|
|
match &score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance {
|
|
Verdict::NotObservable(why) => assert!(
|
|
why.contains("never failed"),
|
|
"a test that never failed is equally what a correct \
|
|
implementation written first looks like: {why}"
|
|
),
|
|
other => panic!("got {other:?}"),
|
|
}
|
|
}
|
|
|
|
/// Ending red is a failure of the loop, and a different one from never
|
|
/// running a test at all.
|
|
#[test]
|
|
fn a_phase_that_never_reached_green_fails() {
|
|
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
|
let mut tools = acted(&[(
|
|
"Write",
|
|
Some("/mission/repo/src/a.rs"),
|
|
json!({"file_path": "/mission/repo/src/a.rs"}),
|
|
)]);
|
|
tools.push(test_run("cargo test", "test result: FAILED. 0 passed; 2 failed"));
|
|
match &score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance {
|
|
Verdict::Fail(why) => assert!(why.contains("never got there"), "{why}"),
|
|
other => panic!("got {other:?}"),
|
|
}
|
|
}
|
|
|
|
/// A mission recorded before responses were captured must not read as
|
|
/// either verdict.
|
|
#[test]
|
|
fn runs_with_no_recorded_output_stay_unobservable() {
|
|
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
|
let tools = acted(&[
|
|
("Write", Some("/mission/repo/src/a.rs"), json!({"file_path": "/mission/repo/src/a.rs"})),
|
|
("Bash", None, json!({"command": "cargo test"})),
|
|
]);
|
|
assert!(matches!(
|
|
score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance,
|
|
Verdict::NotObservable(_)
|
|
));
|
|
}
|
|
|
|
/// The verdict line is at the END of a test run, which is why the response
|
|
/// bound keeps the tail.
|
|
#[test]
|
|
fn a_truncated_response_still_carries_the_verdict() {
|
|
let long = "compiling…\n".repeat(400) + "test result: FAILED. 0 passed; 1 failed";
|
|
let bounded = crate::vm_tool_tap::bounded_response(
|
|
"Bash",
|
|
&json!({ "stdout": long, "stderr": "" }),
|
|
);
|
|
let kept = bounded["stdout"].as_str().expect("stdout kept");
|
|
assert!(
|
|
kept.contains("test result: FAILED"),
|
|
"a head-biased truncation would keep the noise and drop the verdict, \
|
|
which is the only reason the output is stored"
|
|
);
|
|
assert!(kept.starts_with("[truncated]"), "{kept}");
|
|
}
|
|
|
|
/// A `Read`'s response is the file it just read — already knowable, and
|
|
/// large.
|
|
#[test]
|
|
fn only_commands_carry_a_response() {
|
|
assert_eq!(
|
|
crate::vm_tool_tap::bounded_response("Read", &json!({"stdout": "x"})),
|
|
serde_json::Value::Null
|
|
);
|
|
}
|
|
|
|
/// A phase that wrote nothing had no implementation to test-drive.
|
|
/// The verdict the first live run got wrong.
|
|
///
|
|
/// A research phase wrote fifteen markdown notes and ran no test, and the
|
|
/// check called it a TDD failure. There was no code to test-drive. The
|
|
/// real finding is that a research role is pinned TDD skills at all, and
|
|
/// scoring the agent for it would have buried that.
|
|
#[test]
|
|
fn writing_prose_is_not_a_tdd_failure() {
|
|
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
|
let tools = acted(&[
|
|
("Write", Some("/mission/repo/research/notes.md"), json!({"file_path": "/mission/repo/research/notes.md"})),
|
|
("Edit", Some("/mission/repo/research/notes.md"), json!({"file_path": "/mission/repo/research/notes.md"})),
|
|
("Write", Some("/mission/repo/research/check.sh"), json!({"file_path": "/mission/repo/research/check.sh"})),
|
|
]);
|
|
assert_eq!(
|
|
score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance,
|
|
Verdict::NotApplicable,
|
|
"markdown and a helper script are not the behaviour-adding code \
|
|
this loop is about"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn a_phase_that_wrote_nothing_is_not_a_tdd_failure() {
|
|
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
|
let tools = acted(&[ran("ls /mission/repo"), ("Read", Some("/mission/repo/a.rs"), json!({}))]);
|
|
assert_eq!(
|
|
score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance,
|
|
Verdict::NotApplicable
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn the_never_merge_subjects_are_caught_and_a_real_one_is_not() {
|
|
let prompt = rendered(&[("small-focused-commits", "One idea per commit.")]);
|
|
|
|
let bad = acted(&[ran(r#"git commit -m "wip""#)]);
|
|
match &score(&prompt, &Evidence::new("", &bad), &builtin)[0].boundary {
|
|
Verdict::Fail(why) => assert!(why.contains("never-merge"), "{why}"),
|
|
other => panic!("got {other:?}"),
|
|
}
|
|
|
|
// The objection is to `fix` as the whole message, not to the word.
|
|
let good = acted(&[ran(r#"git commit -m "INT-04 fix the range form the parser rejects""#)]);
|
|
assert_eq!(
|
|
score(&prompt, &Evidence::new("", &good), &builtin)[0].boundary,
|
|
Verdict::NotApplicable
|
|
);
|
|
}
|
|
|
|
/// The subject comes out of the shell command, in the shapes agents write.
|
|
#[test]
|
|
fn the_commit_subject_is_read_out_of_the_real_command_shapes() {
|
|
assert_eq!(dash_m_value(r#"git commit -m "one line""#).as_deref(), Some("one line"));
|
|
assert_eq!(dash_m_value("git commit -am 'clustered flag'").as_deref(), Some("clustered flag"));
|
|
// The template in `workspace-repo-commit-protocol` spans several lines;
|
|
// only the first is the subject.
|
|
let multi = "cd /mission/repo && git commit -m \"INT-01 wire the loop\n\nWHY: it stalled.\n\nRefs: INT-01\n\"";
|
|
assert_eq!(
|
|
commit_subjects(&Evidence::new("", &acted(&[ran(multi)]))),
|
|
vec!["INT-01 wire the loop"]
|
|
);
|
|
// `-F` gives up nothing, and that must read as no evidence.
|
|
assert!(dash_m_value("git commit -F /tmp/msg").is_none());
|
|
}
|
|
|
|
/// The exact commit command the first live coding run ran.
|
|
///
|
|
/// Claude Code writes a multi-line message as a heredoc inside a command
|
|
/// substitution, so the first line of the `-m` value is the heredoc opener
|
|
/// and the subject is the line after it. Read from
|
|
/// `mission_events`, not invented.
|
|
#[test]
|
|
fn a_heredoc_commit_message_gives_up_its_real_subject() {
|
|
let live = "git add src/lib.rs && git commit -m \"$(cat <<'EOF'\n\
|
|
INT-01 Add slugify function to src/lib.rs\n\n\
|
|
The crate exposed only add() with no string-normalization utility.\n\n\
|
|
Refs: INT-01\n\
|
|
EOF\n\
|
|
)\"";
|
|
assert_eq!(
|
|
commit_subjects(&Evidence::new("", &acted(&[ran(live)]))),
|
|
vec!["INT-01 Add slugify function to src/lib.rs"],
|
|
"the heredoc OPENER is not the subject — reading it as one puts \
|
|
`$(cat <<'EOF'` in the record, and every commit check then scores \
|
|
a string the agent never wrote"
|
|
);
|
|
}
|
|
|
|
/// The skill tells the agent to `curl` the abstract page in the paragraph
|
|
/// under the one that forbids the query API. A check that cannot tell them
|
|
/// apart fails agents for obeying the skill.
|
|
#[test]
|
|
fn curling_an_abstract_is_allowed_where_querying_the_api_is_not() {
|
|
let prompt = rendered(&[("arxiv-daily", "The harvest already ran.")]);
|
|
|
|
let allowed = acted(&[ran("curl -s https://arxiv.org/abs/2401.12345")]);
|
|
assert_eq!(
|
|
score(&prompt, &Evidence::new("", &allowed), &builtin)[0].boundary,
|
|
Verdict::Pass
|
|
);
|
|
|
|
let forbidden = acted(&[ran("curl -s 'http://export.arxiv.org/api/query?search_query=all:agents'")]);
|
|
match &score(&prompt, &Evidence::new("", &forbidden), &builtin)[0].boundary {
|
|
Verdict::Fail(why) => assert!(why.contains("seen-set"), "{why}"),
|
|
other => panic!("got {other:?}"),
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn the_source_of_a_skill_is_carried_into_its_score() {
|
|
let prompt = rendered(&[("self-made", "x")]);
|
|
let scored = score(&prompt, &narrative("done"), &|_| "promoted_from_brain".to_string());
|
|
assert_eq!(
|
|
scored[0].source_kind, "promoted_from_brain",
|
|
"an agent that writes its own skill can raise its own score against \
|
|
it; that has to be legible in the report rather than averaged in"
|
|
);
|
|
}
|
|
}
|