clawmates/docs/SKILL-USE-BASELINE.md

# Skill-Use baseline

*Whether ClawMates' skills change what agents do. First measured 2026-08-19;
re-measured 2026-08-21 against tool evidence rather than agent prose.*

Scored on the three axes from `Skill-Use` (arXiv, 2026-08-05): **Trigger** (did
the agent reach for the skill), **Compliance** (did it follow the procedure),
**Boundary** (did it avoid what the skill forbids).

Read the method before the numbers. A measurement whose limits are not stated
is worse than none, because it gets quoted without them.

## What changed on 2026-08-21

The 2026-08-19 measurement scored Compliance and Boundary from the
`reasoning` events — the agent's own account of its turn. Since then the
container tier records what agents actually **do**
(`container_tool_hooks`, `PostToolUse`), and `vm_tool_tap` stopped throwing the
tool's arguments away, so `Bash` commands and `Write` paths are on the record.

`skill_use` now reads those. The difference is not cosmetic:

- `workspace-repo-commit-protocol`'s Boundary was a substring search for
  `/workspace/repo` in the narrative. **An agent that wrote to the wrong root
  without narrating it scored a clean pass.** It now reads the write paths.
- `arxiv-daily`'s Boundary read a URL in prose, which may be the agent
  explaining that it did *not* fetch it. It now reads the `curl` that ran.
- `tdd-red-green-refactor`, `cargo-test-driven-development` and
  `small-focused-commits` gained their first checks at all.

Two verdicts changed for honesty rather than coverage. **Silence used to score
`Pass`** — a mission with no evidence scored identically to one checked and
found clean; it is now `NotObservable`. And a test that ran *after* the first
write is `NotObservable`, not a failure, because a Rust unit test lives in the
file under test.

### Trigger: the reason changed, and only half of it went away

The 2026-08-19 document said Trigger was unobservable because `claude_cli`
"cannot surface a tool call — there is nothing to retrieve *with*."

**That half is now false.** Mission tool calls are recorded on both tiers; a
retrieval would be as visible as any other call.

The other half still holds and is the one that decides the verdict: **we still
inline**. `pinned_skills_text` puts full skill bodies in the prompt, so the
agent never reaches for anything — it is simply holding one. Trigger is now
*instrumentable* and still not *observable*, and the blocker has moved from the
transport to the delivery model.

Making it real is one change and no scorer work: serve skills through the door
(`TOOL-CALL-ARCHITECTURE.md` §3) so retrieval becomes a tool call.

## Method, and what it cannot see

Scored by `cm_api::skill_use` from what the platform records: `prompt.composed`
(the exact bytes an agent received), `reasoning` (what it said it did), and
`tool.call` (what it did). No re-derivation from the catalogue — the catalogue
changes, and now that agents author their own skills it changes by itself.

**Every tool-backed check is one-sided.** It reports a violation it can see and
never infers compliance from silence: the recorded stream is capped per phase
(`PER_PHASE_CAP = 400`), so an absent call is not proof of an absent action.

Only skills whose procedure has a machine-checkable consequence are scored.
Everything else returns `not_applicable` rather than a guess — a heuristic that
scores prose by keyword overlap produces a number that looks like a measurement
and is not one.

Compliance for `int-xx-marker-protocol` is checked by running the **real**
`task_card_parser`, not a copy of its rules; a second implementation would
drift, and then the score would pass while the mission loop still stalled.

## The runs

Five missions on the container/ZeroClaw tier, local stack. Runs 1–3 are
`research_only`; run 3 uses the **same task text as run 2**, so the only
variable is the scorer. Run 4 is `research_and_code` against a real repository,
because a research mission writes no code and makes no commits — the TDD and
commit checks could never fire on one.

| | run 1 | run 2 | run 3 | run 4 |
|---|---|---|---|---|
| date | 08-19 | 08-19 | 08-21 | 08-21 |
| workflow | research | research | research | **code** |
| distinct skills delivered | 3 | 9 | 9 | 9 |
| total deliveries (per role prompt) | 3 | 14 | 14 | 28 |
| phantom "skills" scored | **2** | 0 | 0 | 0 |
| tool calls recorded | 0 | 0 | **49** | **33** |
| writes outside `/mission/repo` | ? | ? | 0 | 0 |

Run 3, per skill (all `source_kind=builtin`; no agent-authored skill has been
delivered yet):

| skill | deliveries | compliance | boundary |
|---|---|---|---|
| `int-xx-marker-protocol` | 1 | **pass** | n/a |
| `workspace-repo-commit-protocol` | 2 | n/a | **pass** *(from 12 write paths)* |
| `small-focused-commits` | 4 | n/a | n/a *(no commit ran)* |
| `cargo-test-driven-development` | 2 | n/a | n/a |
| `tdd-red-green-refactor` | 1 | n/a | n/a |
| `decompose-int-items` | 1 | n/a | n/a |
| `write-rust-current-edition` | 1 | n/a | n/a |
| `code-review-checklist` | 1 | n/a | n/a |
| `criterion-benchmarking` | 1 | n/a | n/a |

**n = 5 runs. No spread is reported because five cannot establish one.** This
is a baseline in the sense of "the first honest number", not in the sense of
`metrics-baseline-comparison.md`, which requires enough runs to see the noise
floor before any change is judged against it.

`workspace-repo-commit-protocol`'s pass is the one score that materially
improved: it now rests on twelve recorded `Write`/`Edit` paths, every one under
`/mission/repo`, instead of on the absence of a string in prose.

### Run 4 — the first run that could have violated the new checks

`research_and_code` against `clawmates-delivery-scratch`, task: add a `slugify`
utility and commit it. The task says nothing about testing; priming it would
have measured the prompt rather than the skill.

| skill | deliveries | compliance | boundary |
|---|---|---|---|
| `int-xx-marker-protocol` | 2 | **pass** | n/a |
| `workspace-repo-commit-protocol` | 4 | n/a | **pass** |
| `cargo-test-driven-development` | 4 | **not observable** | n/a |
| `tdd-red-green-refactor` | 2 | **not observable** | n/a |
| `small-focused-commits` | 8 | n/a | n/a |
| the other four | 2 each | n/a | n/a |

The agents edited `src/lib.rs` once, ran `cargo test` five times, and committed
with `INT-01` on the subject. Every one of 33 tool calls stayed inside
`/mission/repo`.

**The TDD verdict is `not_observable`, and that is the honest answer rather
than a gap in the run.** The single `Edit` to `src/lib.rs` added the
implementation *and* a `#[cfg(test)] mod tests` block, then the tests ran. In
Rust the unit test lives in the file under test, so "wrote the file, then ran
the test" is exactly what writing the failing test first looks like from the
outside. The check therefore detects one thing only — **a phase that wrote
source and never ran a test at all** — and cannot confirm red-first. That is a
real limit of scoring TDD from tool ordering, and it applies to the most common
Rust shape, not an edge case.

#### The parsing bug run 4 found

Claude Code writes a multi-line commit message as

```
git commit -m "$(cat <<'EOF'
INT-01 Add slugify function to src/lib.rs
…
EOF
)"
```

and `commit_subjects` read the first line of the `-m` value — which is the
heredoc *opener*, `$(cat <<'EOF'`. Every commit check was scoring a string the
agent never wrote. It happened to score no violation, because `$(cat <<'EOF'`
is not one of the never-merge messages; that is luck, not a check. Fixed, with
a regression test built from the exact command in `mission_events`.

The verdicts in the table above are unchanged by the fix — the real subject,
`INT-01 Add slugify function to src/lib.rs`, is not a never-merge message
either — so the table reproduces against the shipped scorer.

## What the measurement found

### 1–4: the 2026-08-19 findings

Four defects, none of which any test or log would have surfaced: the prompt
format made its own record unparseable (`## <name>` against markdown bodies);
a prompt was recorded that was never sent; a pinned skill taught
`/workspace/repo`, a path the platform does not mount; and
`int-xx-marker-protocol` documented a `PLAN_COMPLETE` marker the parser had
never implemented. All four are fixed, with guards in
`skills_loader::contradiction_tests` and `topology_exec`. The detail is in this
file's git history.

### 5. The tool tap recorded the name and discarded the argument

The first container-tier mission with telemetry recorded `Bash × 6` and not one
of them said what it ran. `vm_tool_tap::parse` read `tool_input` to pull the
path out of it and dropped the rest.

Every behavioural question was therefore unanswerable from a record that looked
complete — which is the recurring shape, not a new one. Fixed host-side: the
arguments were always in the tap file.

### 6. Two more skills contradicted the platform

Same class as finding 3, and both found by reading the source of truth before
writing a check against it.

- **`decompose-int-items` taught `PLAN_COMPLETE: INT-01..05`.** An id is
  strictly `INT-` plus digits, so the range form is rejected outright: the plan
  pass records nothing while every item stays open. A live planner emitted
  exactly that line.
- **`workspace-repo-commit-protocol` claimed the task-card parser advances
  mission state on the INT id in your commit subject.** Nothing in the platform
  reads commit messages. `task_card_parser::apply_for_run` reads `run_events` —
  the agent's turn output. An agent that believed this would commit with the id,
  never emit `COMPLETED: INT-NN`, and leave the mission open on an item it had
  already finished.

`no_skill_shows_a_marker_the_parser_would_reject` now runs the real parser over
every marker in every skill's fenced blocks, negative-controlled against the
range form.

### 7. A repo-less research mission is staffed with a Rust SDLC crew

This is the finding of run 3, and it explains most of the `not_applicable`
column above.

`templates/workflows/research_only.toml` declares `requires_repo = false` and a
single `research` phase — and `default_team_template = "rust_sdlc"`. So the
mission was staffed with **planner, coder, tester, reviewer, committer**, and
each received the skills its role is bound to:

```
coder     :: write-rust-current-edition, cargo-test-driven-development,
             workspace-repo-commit-protocol, small-focused-commits,
             int-xx-marker-protocol
tester    :: cargo-test-driven-development, criterion-benchmarking,
             tdd-red-green-refactor
committer :: workspace-repo-commit-protocol, small-focused-commits
reviewer  :: code-review-checklist, small-focused-commits
planner   :: decompose-int-items, small-focused-commits
```

There is no repository, nothing to test, nothing to review and nothing to
commit. Four of the five roles have no work, and 50KB of prompt (~12.6k tokens)
is spent staffing them.

**The skills are correctly bound to the roles. The roles are wrong for the
workflow.** That distinction matters: a reader who saw only "7 of 9 skills
scored not_applicable" would conclude the skills are useless, when what the
number actually measures is a staffing default.

**Fixed the same day, and measured again as run 5.** `research_only` now
defaults to a new `topic_research` team, and `default_phase_teams` lets a
recipe staff each phase *purpose* separately — `research_and_code` and
`security_hardening` give their research phases the research team and keep
`rust_sdlc` for coding. `benchmark` and `refactor` were checked and left alone:
one coding-purpose phase each, correctly staffed already.

Run 5 is run 3's task, re-run against the new staffing:

| | run 3 | run 5 |
|---|---|---|
| roles staffed | 5 | **3** |
| distinct skills delivered | 9 | **4** |
| total deliveries | 14 | **4** |
| prompt bytes across roles | 50,449 | **24,065** |
| skills applicable to the phase | 1 of 9 | **4 of 4** |

**Read the last row carefully, and not the ones above it.** The *scores* barely
moved: run 5 has one `pass` and three `not_applicable`. What changed is what
`not_applicable` now means. In run 3 it mostly meant "this skill had nothing to
do with what this phase was doing"; in run 5 it means "this skill's procedure
has no machine-checkable consequence" — which is the honest, permanent reason,
and the one this measurement was designed to report.

Cutting the prompt in half is real but incidental. The finding is that the
denominator was wrong: seven of the nine skills in run 3 were never applicable,
so any ratio computed over them measured staffing, not skill use.

The agents also produced exactly the structure the new team's task specifies —
`research/questions.md`, `research/evidence.md`, `research/REPORT.md` — with
zero writes outside `/mission/repo`.

### 8. The check that got it wrong first

Run 3's first scoring reported `cargo-test-driven-development` and
`tdd-red-green-refactor` as **compliance = fail**: files were written and no
test ever ran.

That verdict was wrong, and wrong in the way this whole document exists to
prevent. The phase wrote fifteen markdown notes and a helper script. There was
no code to test-drive. Reporting it as an agent failure would have been a system
defect wearing an agent's name — and it would have buried the real finding,
which is finding 7 above.

The check is now scoped to files with a source extension in the languages the
skill itself names. It is recorded here rather than quietly corrected, because
a measurement that hides its own false positives cannot be trusted about
anyone else's.

## Honest limits

- **Five runs, one tier, two workflows.** Nothing here generalises to the
  microVM or session tiers.
- **The TDD check is one-sided and the common Rust case is undecidable.** It
  catches "wrote source, never ran a test". It cannot confirm red-first,
  because a Rust unit test lives in the file under test — see run 4.
- **Four runs, and run 4 is the only coding one.** The commit checks have been
  *reached* live exactly once.
- **Tool calls carry no agent attribution.** `record_vm_tools` writes
  `agent_id: None` — the container tier's tap is per-container, and all five
  roles share one container. Every score above is therefore per-**mission**,
  not per-role, and the World's per-agent view gets nothing from it. Mapping
  the hook payload's `session_id` back to a turn would fix it.
- **Most skills still score `not_applicable`** on both observable axes. That is
  not a pass. See finding 7 for why the number is what it is.
- **No agent-authored skill has been measured.** `source_kind` is carried
  through the scorer specifically so a rising score on agent-authored skills is
  visible rather than averaged in.
- **Evidence expires.** Mission events are reaped after 7 days unless
  `retain_events_until` is set; `scripts/skill-use-run.sh` holds every run for
  90 days so it stays re-scorable when the scorer changes again — which is
  exactly what happened to run 3. An empty score means "no evidence", never "no
  compliance", and the API says so in its payload.

## Reproducing

```
scripts/skill-use-run.sh "<title>" "<task>"      # run and score
scripts/skill-use-run.sh --score <mission-id>    # re-score, no new run
```

Local stack only. Production auth is Clerk and a mission cannot be launched
from a terminal there — which is also why, as of 2026-08-21, **production has
never run a mission at all** (`select count(*) from missions` → 0).