clawmates/docs/SKILL-USE-BASELINE.md

# Skill-Use baseline

*First measurement of whether ClawMates' skills change what agents do.
2026-08-19.*

Scored on the three axes from `Skill-Use` (arXiv, 2026-08-05): **Trigger** (did
the agent reach for the skill), **Compliance** (did it follow the procedure),
**Boundary** (did it avoid what the skill forbids).

Read the method before the numbers. A measurement whose limits are not stated
is worse than none, because it gets quoted without them.

## Why there was no baseline before today

Not because nobody ran it. Because **it could not have returned anything but
zero**, for two structural reasons that had nothing to do with agent behaviour:

1. 55 of 85 role skill bindings resolved to skills that were never authored.
2. Even resolved skills had no delivery channel to a mission agent — the
   catalogue's only route was an MCP server that mission claws cannot reach.

Both were fixed in the two commits preceding this document. Anyone who had run
this measurement in July would have concluded "our agents ignore their skills",
which would have been false and expensive.

## Method, and what it cannot see

Scored by `cm_api::skill_use` from what the platform records: the
`prompt.composed` event (the exact bytes an agent received) and the `reasoning`
events (what it said it did). No re-derivation from the catalogue — the
catalogue changes, and now that agents author their own skills it changes by
itself.

### Trigger is not observable here, and that is a finding

The paper measures agents under **progressive disclosure**: the agent sees a
name and description and must decide to retrieve the body. That retrieval is a
tool call, which makes Trigger observable.

We do not deliver skills that way. `pinned_skills_text` inlines full bodies into
the prompt, because mission claws run on `claude_cli`, which cannot surface a
tool call — there is nothing to retrieve *with*. The agent never reaches for a
skill; it is simply holding one.

So Trigger is reported as `not_observable` with the reason attached, **never as
zero**. Scoring it zero would report a delivery-model property as an agent
failure — the same confusion that kept 55 empty bindings invisible.

### Compliance and Boundary are checked mechanically, or not at all

Only skills whose procedure has a machine-checkable consequence are scored.
Everything else returns `not_applicable` rather than a guess: a heuristic that
scores prose by keyword overlap produces a number that looks like a measurement
and is not one.

Compliance for `int-xx-marker-protocol` is checked by running the **real**
`task_card_parser`, not a copy of its rules — a second implementation would
drift, and then the score would pass while the mission loop still stalled.

## The runs

Two missions on the container/ZeroClaw tier, local stack, `research_only`.

| | run 1 | run 2 |
|---|---|---|
| distinct skills delivered | 3 | **9** |
| total deliveries (per role prompt) | 3 | 14 |
| phantom "skills" scored | **2** | 0 |

Run 1's phantom entries are the finding of the run, described below.

Run 2, per skill (all `source_kind=builtin`; no agent-authored skill has been
delivered yet):

| skill | deliveries | compliance | boundary |
|---|---|---|---|
| `int-xx-marker-protocol` | 1 | **pass** | n/a |
| `small-focused-commits` | 4 | n/a | n/a |
| `cargo-test-driven-development` | 2 | n/a | n/a |
| `workspace-repo-commit-protocol` | 2 | n/a | n/a |
| `decompose-int-items` | 1 | n/a | n/a |
| `write-rust-current-edition` | 1 | n/a | n/a |
| `code-review-checklist` | 1 | n/a | n/a |
| `criterion-benchmarking` | 1 | n/a | n/a |
| `tdd-red-green-refactor` | 1 | n/a | n/a |

**n = 2 runs. No spread is reported because two runs cannot establish one.**
This is a baseline in the sense of "the first honest number", not in the sense
of `metrics-baseline-comparison.md`, which requires enough runs to see the noise
floor before any change is judged against it. Do not compare a future number to
this one without first establishing that floor.

## What the measurement found

Four defects, none of which any test or log would have surfaced.

### 1. The prompt format made its own record unparseable

Skills were introduced with `## <name>`, and skill bodies are markdown full of
`##` headings. Run 1 duly scored **"Sizing heuristic"** and **"The output
shape"** — both subheadings inside `decompose-int-items` — as skills with no
catalogue row.

Fixed with an unambiguous `--- SKILL: <name> ---` marker, and both writers now
share one renderer so the reader cannot drift from the writer.

### 2. A prompt was recorded that was never sent

The phase prompt was recorded at the dispatch fork, before the tier was chosen.
The container tier does not send that text — it sends the bare task and appends
skills per turn. So every container mission logged a `solo` prompt that reached
no agent.

A provenance record of something that did not happen is worse than no record: it
is the wrong answer, delivered confidently. Recording now happens inside each
tier, and a test asserts every launcher records the prompt it actually sends.

### 3. A pinned skill contradicted the platform in the same prompt

`workspace-repo-commit-protocol` told agents that **`/workspace/repo`** was "the
ONLY path where source-modifying edits belong". The platform mounts and
advertises **`/mission/repo`** — 26 references in the code; `/workspace/repo`
appears in none. The skill is bound on **29 role bindings** and was delivered
twice in run 2, so agents received the real path in the tool preamble and a
skill contradicting it a few hundred tokens later.

It also instructed `file_read` / `file_write` / `shell` — ZeroClaw's tool names,
the exact ones `phase_task_text` was fixed to stop advertising after five agents
on one mission spent 7.4k tokens describing the mismatch instead of working.

An agent that obeyed this skill wrote source into a directory nothing collects,
and reached for tools its subprocess does not expose. Rewritten against what the
code actually does, with two guards in `skills_loader::contradiction_tests`: no
skill may name a repo path the platform does not mount, and none may instruct a
tool the agent does not have. Both negative-controlled.

This is the same shape as the finding below and it is worth stating as a class:
**the skills were never checked against the platform they describe.** Nothing
compared them, so a skill could contradict the prompt it ships inside and stay
that way indefinitely.

### 4. The skill documents a marker the platform never implemented

`int-xx-marker-protocol` lists `PLAN_COMPLETE: INT-NN` in its ladder.
`task_card_parser` has **no such kind** and never has. An agent following the
skill exactly emits a marker that is silently ignored.

Observed live: run 2's planner emitted `PLAN_COMPLETE: INT-01..02`, which is
also the range form — on the kinds that *are* parsed, that yields the id
`INT-01..02`, a task card for an item that does not exist while the two real
items stay open.

**This is a skill/implementation mismatch, not an agent failure**, and it is
precisely what this measurement exists to find: the agent did what it was told,
and what it was told was wrong. Both shapes are now scored as failures; the
underlying reconciliation — implement `PLAN_COMPLETE` or remove it from the
skill — is deliberately left as a decision rather than guessed at here.

## Honest limits

- **Two runs, one tier, one workflow.** Nothing here generalises to the microVM
  or session tiers yet, and this document should not be read as if it does.
- **Most skills score `not_applicable`** on both observable axes. That is not a
  pass. It means we cannot currently tell whether those skills changed anything.
  `workspace-repo-commit-protocol` now has a Boundary check (writing outside
  `/mission/repo`); `small-focused-commits` and `tdd-red-green-refactor` remain
  candidates, and both need the repository diff rather than the turn text.
- **No agent-authored skill has been measured.** Self-authoring shipped in the
  same pass; `source_kind` is carried through the scorer specifically so a
  rising score on agent-authored skills is visible rather than averaged in.
- **Evidence expires.** Mission events are reaped after 7 days unless
  `retain_events_until` is set. Both runs here are held for 90 days. An empty
  score means "no evidence", never "no compliance", and the API says so in its
  payload rather than leaving the caller to infer it.