feat(skills): the first Skill-Use measurement, and the three defects it found

Scored on the paper's three axes against two real missions on the local
stack. docs/SKILL-USE-BASELINE.md has the numbers, the method, and the
limits.

Trigger is reported as NOT OBSERVABLE, never zero

The paper measures progressive disclosure: the agent sees a name and
description and must retrieve the body, and that retrieval is the Trigger
event. We inline full bodies, because mission claws run on claude_cli which
cannot surface a tool call — there is nothing to retrieve with. So the
agent never reaches for a skill, it simply holds one.

Scoring that zero would report a delivery-model property as an agent
failure, which is the same confusion that kept 55 empty bindings invisible
for months. The verdict type carries NotObservable(reason) as a distinct
case from Fail for exactly this.

Compliance is checked by running the REAL task_card_parser rather than a
copy of its rules — a second implementation would drift, and then the score
would pass while the mission loop still stalled. Skills without a
machine-checkable consequence score not_applicable rather than a guess.

WHAT THE MEASUREMENT FOUND

1. The prompt format made its own record unparseable. Skills were
   introduced with `## <name>` and skill bodies are markdown full of `##`
   headings, so run 1 scored "Sizing heuristic" and "The output shape" —
   subheadings inside decompose-int-items — as skills with no catalogue
   row. Now an unambiguous `--- SKILL: <name> ---` marker, with both
   writers sharing one renderer so the reader cannot drift from the writer.

2. A prompt was recorded that was never sent. My own Phase 1 work recorded
   the phase prompt at the dispatch fork, before the tier was chosen — and
   the container tier does not send that text, it sends the bare task and
   appends skills per turn. Every container mission logged a `solo` prompt
   that reached no agent. A provenance record of something that did not
   happen is worse than no record: it is the wrong answer, delivered
   confidently. Recording now happens inside each tier, with a test that
   every launcher records the prompt it actually sends.

3. int-xx-marker-protocol documents a marker the platform never
   implemented. PLAN_COMPLETE is in the skill's ladder and task_card_parser
   has no such kind and never has, so an agent following the skill exactly
   emits a marker that is silently ignored. Observed live: run 2's planner
   emitted `PLAN_COMPLETE: INT-01..02`, which is also the range form — on
   the kinds that ARE parsed that yields the id `INT-01..02`, a task card
   for an item that does not exist while the two real items stay open.

   This is a skill/implementation mismatch, not an agent failure, and it is
   exactly what the measurement exists to find: the agent did what it was
   told and what it was told was wrong. Both shapes now score as failures.
   The reconciliation — implement PLAN_COMPLETE or drop it from the skill —
   is left as a decision rather than guessed at.

The boot log now shows what the plan asked for: 53 skills, 11 templates,
every one `N role skills bound` with NO unresolved clause. Live missions
confirm per-role delivery — the planner receives decompose-int-items, the
coder receives write-rust-current-edition.

GET /api/missions/{id}/skill-use exposes the scores, and says in its
payload whether an empty result means "nothing delivered" or "the evidence
was reaped" — those have very different causes and must not look the same.

n = 2. No spread is reported because two runs cannot establish one, and the
document says so rather than letting the number be quoted as a baseline it
is not.

Full workspace suite green: 106 binaries.

Co-Authored-By: Claude Opus 5 <[email protected]>

docs/SKILL-USE-BASELINE.md

@@ -0,0 +1,150 @@
+# Skill-Use baseline
+
+*First measurement of whether ClawMates' skills change what agents do.
+2026-08-19.*
+
+Scored on the three axes from `Skill-Use` (arXiv, 2026-08-05): **Trigger** (did
+the agent reach for the skill), **Compliance** (did it follow the procedure),
+**Boundary** (did it avoid what the skill forbids).
+
+Read the method before the numbers. A measurement whose limits are not stated
+is worse than none, because it gets quoted without them.
+
+## Why there was no baseline before today
+
+Not because nobody ran it. Because **it could not have returned anything but
+zero**, for two structural reasons that had nothing to do with agent behaviour:
+
+1. 55 of 85 role skill bindings resolved to skills that were never authored.
+2. Even resolved skills had no delivery channel to a mission agent — the
+   catalogue's only route was an MCP server that mission claws cannot reach.
+
+Both were fixed in the two commits preceding this document. Anyone who had run
+this measurement in July would have concluded "our agents ignore their skills",
+which would have been false and expensive.
+
+## Method, and what it cannot see
+
+Scored by `cm_api::skill_use` from what the platform records: the
+`prompt.composed` event (the exact bytes an agent received) and the `reasoning`
+events (what it said it did). No re-derivation from the catalogue — the
+catalogue changes, and now that agents author their own skills it changes by
+itself.
+
+### Trigger is not observable here, and that is a finding
+
+The paper measures agents under **progressive disclosure**: the agent sees a
+name and description and must decide to retrieve the body. That retrieval is a
+tool call, which makes Trigger observable.
+
+We do not deliver skills that way. `pinned_skills_text` inlines full bodies into
+the prompt, because mission claws run on `claude_cli`, which cannot surface a
+tool call — there is nothing to retrieve *with*. The agent never reaches for a
+skill; it is simply holding one.
+
+So Trigger is reported as `not_observable` with the reason attached, **never as
+zero**. Scoring it zero would report a delivery-model property as an agent
+failure — the same confusion that kept 55 empty bindings invisible.
+
+### Compliance and Boundary are checked mechanically, or not at all
+
+Only skills whose procedure has a machine-checkable consequence are scored.
+Everything else returns `not_applicable` rather than a guess: a heuristic that
+scores prose by keyword overlap produces a number that looks like a measurement
+and is not one.
+
+Compliance for `int-xx-marker-protocol` is checked by running the **real**
+`task_card_parser`, not a copy of its rules — a second implementation would
+drift, and then the score would pass while the mission loop still stalled.
+
+## The runs
+
+Two missions on the container/ZeroClaw tier, local stack, `research_only`.
+
+| | run 1 | run 2 |
+|---|---|---|
+| distinct skills delivered | 3 | **9** |
+| total deliveries (per role prompt) | 3 | 14 |
+| phantom "skills" scored | **2** | 0 |
+
+Run 1's phantom entries are the finding of the run, described below.
+
+Run 2, per skill (all `source_kind=builtin`; no agent-authored skill has been
+delivered yet):
+
+| skill | deliveries | compliance | boundary |
+|---|---|---|---|
+| `int-xx-marker-protocol` | 1 | **pass** | n/a |
+| `small-focused-commits` | 4 | n/a | n/a |
+| `cargo-test-driven-development` | 2 | n/a | n/a |
+| `workspace-repo-commit-protocol` | 2 | n/a | n/a |
+| `decompose-int-items` | 1 | n/a | n/a |
+| `write-rust-current-edition` | 1 | n/a | n/a |
+| `code-review-checklist` | 1 | n/a | n/a |
+| `criterion-benchmarking` | 1 | n/a | n/a |
+| `tdd-red-green-refactor` | 1 | n/a | n/a |
+
+**n = 2 runs. No spread is reported because two runs cannot establish one.**
+This is a baseline in the sense of "the first honest number", not in the sense
+of `metrics-baseline-comparison.md`, which requires enough runs to see the noise
+floor before any change is judged against it. Do not compare a future number to
+this one without first establishing that floor.
+
+## What the measurement found
+
+Three defects, none of which any test or log would have surfaced.
+
+### 1. The prompt format made its own record unparseable
+
+Skills were introduced with `## <name>`, and skill bodies are markdown full of
+`##` headings. Run 1 duly scored **"Sizing heuristic"** and **"The output
+shape"** — both subheadings inside `decompose-int-items` — as skills with no
+catalogue row.
+
+Fixed with an unambiguous `--- SKILL: <name> ---` marker, and both writers now
+share one renderer so the reader cannot drift from the writer.
+
+### 2. A prompt was recorded that was never sent
+
+The phase prompt was recorded at the dispatch fork, before the tier was chosen.
+The container tier does not send that text — it sends the bare task and appends
+skills per turn. So every container mission logged a `solo` prompt that reached
+no agent.
+
+A provenance record of something that did not happen is worse than no record: it
+is the wrong answer, delivered confidently. Recording now happens inside each
+tier, and a test asserts every launcher records the prompt it actually sends.
+
+### 3. The skill documents a marker the platform never implemented
+
+`int-xx-marker-protocol` lists `PLAN_COMPLETE: INT-NN` in its ladder.
+`task_card_parser` has **no such kind** and never has. An agent following the
+skill exactly emits a marker that is silently ignored.
+
+Observed live: run 2's planner emitted `PLAN_COMPLETE: INT-01..02`, which is
+also the range form — on the kinds that *are* parsed, that yields the id
+`INT-01..02`, a task card for an item that does not exist while the two real
+items stay open.
+
+**This is a skill/implementation mismatch, not an agent failure**, and it is
+precisely what this measurement exists to find: the agent did what it was told,
+and what it was told was wrong. Both shapes are now scored as failures; the
+underlying reconciliation — implement `PLAN_COMPLETE` or remove it from the
+skill — is deliberately left as a decision rather than guessed at here.
+
+## Honest limits
+
+- **Two runs, one tier, one workflow.** Nothing here generalises to the microVM
+  or session tiers yet, and this document should not be read as if it does.
+- **7 of 9 skills scored `not_applicable`** on both observable axes. That is not
+  a pass. It means we cannot currently tell whether those skills changed
+  anything, and the next unit of work is mechanical checks for more of them —
+  `workspace-repo-commit-protocol` and `small-focused-commits` both have
+  checkable consequences in git history.
+- **No agent-authored skill has been measured.** Self-authoring shipped in the
+  same pass; `source_kind` is carried through the scorer specifically so a
+  rising score on agent-authored skills is visible rather than averaged in.
+- **Evidence expires.** Mission events are reaped after 7 days unless
+  `retain_events_until` is set. Both runs here are held for 90 days. An empty
+  score means "no evidence", never "no compliance", and the API says so in its
+  payload rather than leaving the caller to infer it.