osobh/clawmates
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test(skill-use): the coding run, and the parsing bug it found

Browse Source 
Run 4 (`research_and_code`, real repo) is the first mission that could
have violated the TDD and commit checks. It exercised both, and found a
bug in one.

Claude Code writes a multi-line commit message as a heredoc inside a
command substitution:

    git commit -m "$(cat <<'EOF'
    INT-01 Add slugify function to src/lib.rs
    …
    EOF
    )"

`commit_subjects` read the first line of the `-m` value, which is the
heredoc OPENER. Every commit check was scoring `$(cat <<'EOF'` — a string
the agent never wrote. It reported no violation only because that string
is not one of the never-merge messages, which is luck rather than a check.
Regression test built from the exact command in `mission_events`.

The TDD verdict came back `not_observable`, which is the honest answer and
also a real limit worth stating: the agents edited `src/lib.rs` once —
implementation and `#[cfg(test)] mod tests` in the same write — then ran
`cargo test` five times. In Rust the unit test lives in the file under
test, so that ordering is exactly what following the skill precisely looks
like from outside. The check detects "wrote source, never ran a test" and
cannot confirm red-first. Confirming it needs the diff, not the tool order.

Every one of run 4's 33 tool calls stayed inside /mission/repo.

Handoff and baseline updated: production has never run a mission (both
tables empty), a mission container has leaked since 2026-08-12 that no
reaper can see, and `research_only` staffs a five-role Rust SDLC crew on a
repo-less markdown mission — which is what "most skills score
not_applicable" has been measuring all along.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9
This commit is contained in:
Omar Sobh
2026-08-21 08:43:37 -07:00
co-authored by Claude Opus 5
parent c209e654d9
commit 9560aaec41
3 changed files with 264 additions and 142 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/SKILL-USE-BASELINE.md
+72 -16
View File
@@ -73,17 +73,21 @@ drift, and then the score would pass while the mission loop still stalled.
## The runs
Three missions on the container/ZeroClaw tier, local stack, `research_only`.
Run 3 uses the **same task text as run 2**, so the only variable is the scorer.
Four missions on the container/ZeroClaw tier, local stack. Runs 1–3 are
`research_only`; run 3 uses the **same task text as run 2**, so the only
variable is the scorer. Run 4 is `research_and_code` against a real repository,
because a research mission writes no code and makes no commits — the TDD and
commit checks could never fire on one.
| | run 1 | run 2 | run 3 |
|---|---|---|---|
| date | 08-19 | 08-19 | 08-21 |
| distinct skills delivered | 3 | 9 | 9 |
| total deliveries (per role prompt) | 3 | 14 | 14 |
| phantom "skills" scored | **2** | 0 | 0 |
| tool calls recorded | 0 | 0 | **49** |
| axes scored from actions | 0 | 0 | **3 skills** |
| | run 1 | run 2 | run 3 | run 4 |
|---|---|---|---|---|
| date | 08-19 | 08-19 | 08-21 | 08-21 |
| workflow | research | research | research | **code** |
| distinct skills delivered | 3 | 9 | 9 | 9 |
| total deliveries (per role prompt) | 3 | 14 | 14 | 28 |
| phantom "skills" scored | **2** | 0 | 0 | 0 |
| tool calls recorded | 0 | 0 | **49** | **33** |
| writes outside `/mission/repo` | ? | ? | 0 | 0 |
Run 3, per skill (all `source_kind=builtin`; no agent-authored skill has been
delivered yet):
@@ -100,7 +104,7 @@ delivered yet):
| `code-review-checklist` | 1 | n/a | n/a |
| `criterion-benchmarking` | 1 | n/a | n/a |
**n = 3 runs. No spread is reported because three cannot establish one.** This
**n = 4 runs. No spread is reported because four cannot establish one.** This
is a baseline in the sense of "the first honest number", not in the sense of
`metrics-baseline-comparison.md`, which requires enough runs to see the noise
floor before any change is judged against it.
@@ -109,6 +113,57 @@ floor before any change is judged against it.
improved: it now rests on twelve recorded `Write`/`Edit` paths, every one under
`/mission/repo`, instead of on the absence of a string in prose.
### Run 4 — the first run that could have violated the new checks
`research_and_code` against `clawmates-delivery-scratch`, task: add a `slugify`
utility and commit it. The task says nothing about testing; priming it would
have measured the prompt rather than the skill.
| skill | deliveries | compliance | boundary |
|---|---|---|---|
| `int-xx-marker-protocol` | 2 | **pass** | n/a |
| `workspace-repo-commit-protocol` | 4 | n/a | **pass** |
| `cargo-test-driven-development` | 4 | **not observable** | n/a |
| `tdd-red-green-refactor` | 2 | **not observable** | n/a |
| `small-focused-commits` | 8 | n/a | n/a |
| the other four | 2 each | n/a | n/a |
The agents edited `src/lib.rs` once, ran `cargo test` five times, and committed
with `INT-01` on the subject. Every one of 33 tool calls stayed inside
`/mission/repo`.
**The TDD verdict is `not_observable`, and that is the honest answer rather
than a gap in the run.** The single `Edit` to `src/lib.rs` added the
implementation *and* a `#[cfg(test)] mod tests` block, then the tests ran. In
Rust the unit test lives in the file under test, so "wrote the file, then ran
the test" is exactly what writing the failing test first looks like from the
outside. The check therefore detects one thing only — **a phase that wrote
source and never ran a test at all** — and cannot confirm red-first. That is a
real limit of scoring TDD from tool ordering, and it applies to the most common
Rust shape, not an edge case.
#### The parsing bug run 4 found
Claude Code writes a multi-line commit message as
```
git commit -m "$(cat <<'EOF'
INT-01 Add slugify function to src/lib.rs
…
EOF
)"
```
and `commit_subjects` read the first line of the `-m` value — which is the
heredoc *opener*, `$(cat <<'EOF'`. Every commit check was scoring a string the
agent never wrote. It happened to score no violation, because `$(cat <<'EOF'`
is not one of the never-merge messages; that is luck, not a check. Fixed, with
a regression test built from the exact command in `mission_events`.
The verdicts in the table above are unchanged by the fix — the real subject,
`INT-01 Add slugify function to src/lib.rs`, is not a never-merge message
either — so the table reproduces against the shipped scorer.
## What the measurement found
### 1–4: the 2026-08-19 findings
@@ -207,12 +262,13 @@ anyone else's.
## Honest limits
- **Three runs, one tier, one workflow.** Nothing here generalises to the
- **Four runs, one tier, two workflows.** Nothing here generalises to the
microVM or session tiers.
- **The new checks are not yet exercised live.** `research_only` writes no code
and makes no commits, so the TDD and commit checks are proven by unit tests
and negative controls, not by a mission that could have violated them. A
coding run against a real repository is the next measurement.
- **The TDD check is one-sided and the common Rust case is undecidable.** It
catches "wrote source, never ran a test". It cannot confirm red-first,
because a Rust unit test lives in the file under test — see run 4.
- **Four runs, and run 4 is the only coding one.** The commit checks have been
*reached* live exactly once.
- **Tool calls carry no agent attribution.** `record_vm_tools` writes
`agent_id: None` — the container tier's tap is per-container, and all five
roles share one container. Every score above is therefore per-**mission**,