fix(skill-use): a research phase writing markdown is not a TDD failure

The first live scoring of run 3 reported `cargo-test-driven-development`
and `tdd-red-green-refactor` as compliance=FAIL: files were written and no
test ever ran.

Wrong, and wrong in the way this module exists to prevent. The phase wrote
fifteen markdown notes and a helper script; there was no code to
test-drive. Reporting it as an agent failure is a system defect wearing an
agent's name — and it would have buried the actual finding, which is that
a repo-less `research_only` mission is staffed with a Rust SDLC crew whose
coder, tester, reviewer and committer have nothing to do.

The check is now scoped to files with a source extension in the languages
the skill itself names. Shell is deliberately excluded: a helper script
written during a research turn is not behaviour-adding code, and the false
failure costs more than the missed one.

Recorded in SKILL-USE-BASELINE.md as finding 8 rather than quietly
corrected. A measurement that hides its own false positives cannot be
trusted about anyone else's.

Also in the doc: the Trigger reason is half false now (the transport can
surface a tool call; we simply still inline), and the architecture doc's
observe/gate table said the container tier was ungated and unobserved,
which shipped work has made wrong.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9
This commit is contained in:
Omar Sobh
2026-08-21 08:37:15 -07:00
co-authored by Claude Opus 5
parent 4a6d0dfe01
commit c209e654d9
4 changed files with 259 additions and 126 deletions
+52 -7
View File
@@ -420,10 +420,9 @@ fn red_before_green(ev: &Evidence<'_>) -> Verdict {
.into(),
);
}
let first_write = ev
.tools
.iter()
.position(|t| WRITE_TOOLS.contains(&t.tool.as_str()) && t.path.is_some());
let first_write = ev.tools.iter().position(|t| {
WRITE_TOOLS.contains(&t.tool.as_str()) && t.path.as_deref().is_some_and(is_source)
});
let first_test = ev
.tools
.iter()
@@ -432,9 +431,11 @@ fn red_before_green(ev: &Evidence<'_>) -> Verdict {
// Nothing was written, so there was no implementation to test-drive.
(None, _) => Verdict::NotApplicable,
(Some(_), None) => Verdict::Fail(format!(
"wrote {} file(s) and never ran a test — the loop is red, green, \
refactor, and a test that never ran cannot have been red",
ev.writes().count()
"wrote {} source file(s) and never ran a test — the loop is red, \
green, refactor, and a test that never ran cannot have been red",
ev.writes()
.filter(|w| w.path.as_deref().is_some_and(is_source))
.count()
)),
(Some(w), Some(t)) if t < w => Verdict::Pass,
_ => Verdict::NotObservable(
@@ -446,6 +447,28 @@ fn red_before_green(ev: &Evidence<'_>) -> Verdict {
}
}
/// Is this file behaviour-adding code, in the languages the skill names?
///
/// The check is scoped to source because the first live run scored `fail`
/// against a **research** phase: its agents wrote fifteen markdown notes and
/// never ran a test, which is not a violation of anything — there was no code
/// to test-drive. Reporting that as an agent failure would be a system defect
/// wearing an agent's name, and the real finding it obscures is that a research
/// role is pinned TDD skills at all.
///
/// Extensions the skill itself names — "Rust, TypeScript, Python, anywhere
/// tests can run cheap" — plus their immediate neighbours. Prose, config and
/// data are excluded. Shell is excluded deliberately: a helper script written
/// during a research turn is not the behaviour-adding code this loop is about,
/// and the false failure costs more than the missed one.
fn is_source(path: &str) -> bool {
const SOURCE: [&str; 10] = [
".rs", ".ts", ".tsx", ".py", ".js", ".jsx", ".go", ".java", ".rb", ".kt",
];
let p = path.to_ascii_lowercase();
SOURCE.iter().any(|e| p.ends_with(e))
}
/// Commands that run a test suite, across the languages the skills name.
fn is_test_command(cmd: &str) -> bool {
const RUNNERS: [&str; 9] = [
@@ -991,6 +1014,28 @@ mod tests {
}
/// A phase that wrote nothing had no implementation to test-drive.
/// The verdict the first live run got wrong.
///
/// A research phase wrote fifteen markdown notes and ran no test, and the
/// check called it a TDD failure. There was no code to test-drive. The
/// real finding is that a research role is pinned TDD skills at all, and
/// scoring the agent for it would have buried that.
#[test]
fn writing_prose_is_not_a_tdd_failure() {
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
let tools = acted(&[
("Write", Some("/mission/repo/research/notes.md"), json!({"file_path": "/mission/repo/research/notes.md"})),
("Edit", Some("/mission/repo/research/notes.md"), json!({"file_path": "/mission/repo/research/notes.md"})),
("Write", Some("/mission/repo/research/check.sh"), json!({"file_path": "/mission/repo/research/check.sh"})),
]);
assert_eq!(
score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance,
Verdict::NotApplicable,
"markdown and a helper script are not the behaviour-adding code \
this loop is about"
);
}
#[test]
fn a_phase_that_wrote_nothing_is_not_a_tdd_failure() {
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);