feat(skill-use): red-first is observable from what the RUNS reported

The open item said this needed the repository diff rather than tool
order. That was wrong, and the skill says why: "Commit the RED-to-GREEN
pair as one commit." The failing test and its fix land together by
instruction, so the diff and the commit history are as blind as the tool
ordering already was — in Rust one `Edit` adds the implementation and its
`#[cfg(test)] mod tests` in the same call.

The only remaining witness is what each test run itself printed, and the
tap was throwing it away. Claude Code's PostToolUse payload carries
`tool_response` — verified against the real binary, keys
stdout/stderr/interrupted, plus `duration_ms` and `tool_use_id`.

So `Observed.response` now keeps it, for COMMANDS only: a `Read`'s
response is the file it just read and a `Write`'s restates its own
argument — both already knowable, both large, and storing them would
double the biggest write path in the system for nothing.

`bounded_response` keeps the **end** of the output, which is the opposite
of `bounded_input` and deliberately so. An argument's meaning is its verb,
at the start. A command's meaning is its verdict, at the end: `cargo test`
prints hundreds of lines and then `test result: ok` or `FAILED`. A
head-biased truncation would keep the noise and discard the only thing
being stored for — negative-controlled with a 400-line fixture.

`red_before_green` now falls through to the run outcomes:

  failing run, then a passing one  → Pass, red then green observed
  every run failed                 → Fail, the loop ends on green
  every run passed                 → NotObservable, and the reason says
                                     why: a test that never failed is
                                     equally what a correct implementation
                                     written first looks like
  no outputs recorded              → NotObservable (pre-capture missions)

Read from the runner's verdict line, not an exit code — the payload
carries none.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9
This commit is contained in:
Omar Sobh
2026-08-21 12:45:20 -07:00
co-authored by Claude Opus 5
parent b47ae7fa6b
commit 5220f3bfea
4 changed files with 276 additions and 7 deletions
+7
View File
@@ -196,6 +196,12 @@ pub struct ToolEvidence {
pub path: Option<String>,
/// The tool's arguments, bounded by `vm_tool_tap::bounded_input`.
pub input: Value,
/// What a command produced, bounded by `vm_tool_tap::bounded_response`.
///
/// Null for every tool that is not a command. This is where a failing test
/// run is visible, and it is the only place it is — the recorded stream has
/// no exit codes.
pub response: Value,
}
impl ToolEvidence {
@@ -241,6 +247,7 @@ pub async fn tool_evidence_for_mission(
.and_then(Value::as_str)
.map(str::to_string),
input: detail.get("input").cloned().unwrap_or(Value::Null),
response: detail.get("response").cloned().unwrap_or(Value::Null),
})
.collect())
}