fix(skill-use): a research phase writing markdown is not a TDD failure
The first live scoring of run 3 reported `cargo-test-driven-development` and `tdd-red-green-refactor` as compliance=FAIL: files were written and no test ever ran. Wrong, and wrong in the way this module exists to prevent. The phase wrote fifteen markdown notes and a helper script; there was no code to test-drive. Reporting it as an agent failure is a system defect wearing an agent's name — and it would have buried the actual finding, which is that a repo-less `research_only` mission is staffed with a Rust SDLC crew whose coder, tester, reviewer and committer have nothing to do. The check is now scoped to files with a source extension in the languages the skill itself names. Shell is deliberately excluded: a helper script written during a research turn is not behaviour-adding code, and the false failure costs more than the missed one. Recorded in SKILL-USE-BASELINE.md as finding 8 rather than quietly corrected. A measurement that hides its own false positives cannot be trusted about anyone else's. Also in the doc: the Trigger reason is half false now (the transport can surface a tool call; we simply still inline), and the architecture doc's observe/gate table said the container tier was ungated and unobserved, which shipped work has made wrong. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9
This commit is contained in:
co-authored by
Claude Opus 5
parent
4a6d0dfe01
commit
c209e654d9
@@ -420,10 +420,9 @@ fn red_before_green(ev: &Evidence<'_>) -> Verdict {
|
||||
.into(),
|
||||
);
|
||||
}
|
||||
let first_write = ev
|
||||
.tools
|
||||
.iter()
|
||||
.position(|t| WRITE_TOOLS.contains(&t.tool.as_str()) && t.path.is_some());
|
||||
let first_write = ev.tools.iter().position(|t| {
|
||||
WRITE_TOOLS.contains(&t.tool.as_str()) && t.path.as_deref().is_some_and(is_source)
|
||||
});
|
||||
let first_test = ev
|
||||
.tools
|
||||
.iter()
|
||||
@@ -432,9 +431,11 @@ fn red_before_green(ev: &Evidence<'_>) -> Verdict {
|
||||
// Nothing was written, so there was no implementation to test-drive.
|
||||
(None, _) => Verdict::NotApplicable,
|
||||
(Some(_), None) => Verdict::Fail(format!(
|
||||
"wrote {} file(s) and never ran a test — the loop is red, green, \
|
||||
refactor, and a test that never ran cannot have been red",
|
||||
ev.writes().count()
|
||||
"wrote {} source file(s) and never ran a test — the loop is red, \
|
||||
green, refactor, and a test that never ran cannot have been red",
|
||||
ev.writes()
|
||||
.filter(|w| w.path.as_deref().is_some_and(is_source))
|
||||
.count()
|
||||
)),
|
||||
(Some(w), Some(t)) if t < w => Verdict::Pass,
|
||||
_ => Verdict::NotObservable(
|
||||
@@ -446,6 +447,28 @@ fn red_before_green(ev: &Evidence<'_>) -> Verdict {
|
||||
}
|
||||
}
|
||||
|
||||
/// Is this file behaviour-adding code, in the languages the skill names?
|
||||
///
|
||||
/// The check is scoped to source because the first live run scored `fail`
|
||||
/// against a **research** phase: its agents wrote fifteen markdown notes and
|
||||
/// never ran a test, which is not a violation of anything — there was no code
|
||||
/// to test-drive. Reporting that as an agent failure would be a system defect
|
||||
/// wearing an agent's name, and the real finding it obscures is that a research
|
||||
/// role is pinned TDD skills at all.
|
||||
///
|
||||
/// Extensions the skill itself names — "Rust, TypeScript, Python, anywhere
|
||||
/// tests can run cheap" — plus their immediate neighbours. Prose, config and
|
||||
/// data are excluded. Shell is excluded deliberately: a helper script written
|
||||
/// during a research turn is not the behaviour-adding code this loop is about,
|
||||
/// and the false failure costs more than the missed one.
|
||||
fn is_source(path: &str) -> bool {
|
||||
const SOURCE: [&str; 10] = [
|
||||
".rs", ".ts", ".tsx", ".py", ".js", ".jsx", ".go", ".java", ".rb", ".kt",
|
||||
];
|
||||
let p = path.to_ascii_lowercase();
|
||||
SOURCE.iter().any(|e| p.ends_with(e))
|
||||
}
|
||||
|
||||
/// Commands that run a test suite, across the languages the skills name.
|
||||
fn is_test_command(cmd: &str) -> bool {
|
||||
const RUNNERS: [&str; 9] = [
|
||||
@@ -991,6 +1014,28 @@ mod tests {
|
||||
}
|
||||
|
||||
/// A phase that wrote nothing had no implementation to test-drive.
|
||||
/// The verdict the first live run got wrong.
|
||||
///
|
||||
/// A research phase wrote fifteen markdown notes and ran no test, and the
|
||||
/// check called it a TDD failure. There was no code to test-drive. The
|
||||
/// real finding is that a research role is pinned TDD skills at all, and
|
||||
/// scoring the agent for it would have buried that.
|
||||
#[test]
|
||||
fn writing_prose_is_not_a_tdd_failure() {
|
||||
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
||||
let tools = acted(&[
|
||||
("Write", Some("/mission/repo/research/notes.md"), json!({"file_path": "/mission/repo/research/notes.md"})),
|
||||
("Edit", Some("/mission/repo/research/notes.md"), json!({"file_path": "/mission/repo/research/notes.md"})),
|
||||
("Write", Some("/mission/repo/research/check.sh"), json!({"file_path": "/mission/repo/research/check.sh"})),
|
||||
]);
|
||||
assert_eq!(
|
||||
score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance,
|
||||
Verdict::NotApplicable,
|
||||
"markdown and a helper script are not the behaviour-adding code \
|
||||
this loop is about"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_phase_that_wrote_nothing_is_not_a_tdd_failure() {
|
||||
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
|
||||
|
||||
Reference in New Issue
Block a user