test(skill-use): the coding run, and the parsing bug it found

Run 4 (`research_and_code`, real repo) is the first mission that could
have violated the TDD and commit checks. It exercised both, and found a
bug in one.

Claude Code writes a multi-line commit message as a heredoc inside a
command substitution:

    git commit -m "$(cat <<'EOF'
    INT-01 Add slugify function to src/lib.rs
    …
    EOF
    )"

`commit_subjects` read the first line of the `-m` value, which is the
heredoc OPENER. Every commit check was scoring `$(cat <<'EOF'` — a string
the agent never wrote. It reported no violation only because that string
is not one of the never-merge messages, which is luck rather than a check.
Regression test built from the exact command in `mission_events`.

The TDD verdict came back `not_observable`, which is the honest answer and
also a real limit worth stating: the agents edited `src/lib.rs` once —
implementation and `#[cfg(test)] mod tests` in the same write — then ran
`cargo test` five times. In Rust the unit test lives in the file under
test, so that ordering is exactly what following the skill precisely looks
like from outside. The check detects "wrote source, never ran a test" and
cannot confirm red-first. Confirming it needs the diff, not the tool order.

Every one of run 4's 33 tool calls stayed inside /mission/repo.

Handoff and baseline updated: production has never run a mission (both
tables empty), a mission container has leaked since 2026-08-12 that no
reaper can see, and `research_only` staffs a five-role Rust SDLC crew on a
repo-less markdown mission — which is what "most skills score
not_applicable" has been measuring all along.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9
This commit is contained in:
Omar Sobh
2026-08-21 08:43:37 -07:00
co-authored by Claude Opus 5
parent c209e654d9
commit 9560aaec41
3 changed files with 264 additions and 142 deletions
+53 -2
View File
@@ -499,11 +499,39 @@ fn commit_subjects(ev: &Evidence<'_>) -> Vec<String> {
ev.commands()
.filter(|c| c.contains("git commit") || c.contains("git ci"))
.filter_map(dash_m_value)
.filter_map(|m| m.lines().next().map(|l| l.trim().to_string()))
.filter(|s| !s.is_empty())
.filter_map(|m| subject_line(&m))
.collect()
}
/// The subject out of a `-m` value.
///
/// Normally the first line. But Claude Code writes a multi-line message as
///
/// ```text
/// git commit -m "$(cat <<'EOF'
/// INT-01 Add slugify function
///
/// …body…
/// EOF
/// )"
/// ```
///
/// and the first line of that value is `$(cat <<'EOF'` — the heredoc *opener*,
/// not the subject. Observed on the first live coding run; it scored no
/// violation only because `$(cat <<'EOF'` happens not to be one of the
/// never-merge messages, which is luck, not a check.
///
/// So: if the first line opens a heredoc or a command substitution, the subject
/// is the next non-empty line.
fn subject_line(msg: &str) -> Option<String> {
let mut lines = msg.lines().map(str::trim).filter(|l| !l.is_empty());
let first = lines.next()?;
if first.starts_with("$(") || first.contains("<<") {
return lines.next().map(str::to_string);
}
Some(first.to_string())
}
/// The value of the `-m` flag in a shell command.
///
/// Hand-scanned rather than split on whitespace: the value is the one argument
@@ -1080,6 +1108,29 @@ mod tests {
assert!(dash_m_value("git commit -F /tmp/msg").is_none());
}
/// The exact commit command the first live coding run ran.
///
/// Claude Code writes a multi-line message as a heredoc inside a command
/// substitution, so the first line of the `-m` value is the heredoc opener
/// and the subject is the line after it. Read from
/// `mission_events`, not invented.
#[test]
fn a_heredoc_commit_message_gives_up_its_real_subject() {
let live = "git add src/lib.rs && git commit -m \"$(cat <<'EOF'\n\
INT-01 Add slugify function to src/lib.rs\n\n\
The crate exposed only add() with no string-normalization utility.\n\n\
Refs: INT-01\n\
EOF\n\
)\"";
assert_eq!(
commit_subjects(&Evidence::new("", &acted(&[ran(live)]))),
vec!["INT-01 Add slugify function to src/lib.rs"],
"the heredoc OPENER is not the subject — reading it as one puts \
`$(cat <<'EOF'` in the record, and every commit check then scores \
a string the agent never wrote"
);
}
/// The skill tells the agent to `curl` the abstract page in the paragraph
/// under the one that forbids the query API. A check that cannot tell them
/// apart fails agents for obeying the skill.