Omar SobhandClaude Opus 5
91a6b4e304
Scored on the paper's three axes against two real missions on the local
stack. docs/SKILL-USE-BASELINE.md has the numbers, the method, and the
limits.
Trigger is reported as NOT OBSERVABLE, never zero
The paper measures progressive disclosure: the agent sees a name and
description and must retrieve the body, and that retrieval is the Trigger
event. We inline full bodies, because mission claws run on claude_cli which
cannot surface a tool call — there is nothing to retrieve with. So the
agent never reaches for a skill, it simply holds one.
Scoring that zero would report a delivery-model property as an agent
failure, which is the same confusion that kept 55 empty bindings invisible
for months. The verdict type carries NotObservable(reason) as a distinct
case from Fail for exactly this.
Compliance is checked by running the REAL task_card_parser rather than a
copy of its rules — a second implementation would drift, and then the score
would pass while the mission loop still stalled. Skills without a
machine-checkable consequence score not_applicable rather than a guess.
WHAT THE MEASUREMENT FOUND
1. The prompt format made its own record unparseable. Skills were
introduced with `## <name>` and skill bodies are markdown full of `##`
headings, so run 1 scored "Sizing heuristic" and "The output shape" —
subheadings inside decompose-int-items — as skills with no catalogue
row. Now an unambiguous `--- SKILL: <name> ---` marker, with both
writers sharing one renderer so the reader cannot drift from the writer.
2. A prompt was recorded that was never sent. My own Phase 1 work recorded
the phase prompt at the dispatch fork, before the tier was chosen — and
the container tier does not send that text, it sends the bare task and
appends skills per turn. Every container mission logged a `solo` prompt
that reached no agent. A provenance record of something that did not
happen is worse than no record: it is the wrong answer, delivered
confidently. Recording now happens inside each tier, with a test that
every launcher records the prompt it actually sends.
3. int-xx-marker-protocol documents a marker the platform never
implemented. PLAN_COMPLETE is in the skill's ladder and task_card_parser
has no such kind and never has, so an agent following the skill exactly
emits a marker that is silently ignored. Observed live: run 2's planner
emitted `PLAN_COMPLETE: INT-01..02`, which is also the range form — on
the kinds that ARE parsed that yields the id `INT-01..02`, a task card
for an item that does not exist while the two real items stay open.
This is a skill/implementation mismatch, not an agent failure, and it is
exactly what the measurement exists to find: the agent did what it was
told and what it was told was wrong. Both shapes now score as failures.
The reconciliation — implement PLAN_COMPLETE or drop it from the skill —
is left as a decision rather than guessed at.
The boot log now shows what the plan asked for: 53 skills, 11 templates,
every one `N role skills bound` with NO unresolved clause. Live missions
confirm per-role delivery — the planner receives decompose-int-items, the
coder receives write-rust-current-edition.
GET /api/missions/{id}/skill-use exposes the scores, and says in its
payload whether an empty result means "nothing delivered" or "the evidence
was reaped" — those have very different causes and must not look the same.
n = 2. No spread is reported because two runs cannot establish one, and the
document says so rather than letting the number be quoted as a baseline it
is not.
Full workspace suite green: 106 binaries.
Co-Authored-By: Claude Opus 5 <[email protected]>
6.9 KiB
Skill-Use baseline
First measurement of whether ClawMates' skills change what agents do. 2026-08-19.
Scored on the three axes from Skill-Use (arXiv, 2026-08-05): Trigger (did
the agent reach for the skill), Compliance (did it follow the procedure),
Boundary (did it avoid what the skill forbids).
Read the method before the numbers. A measurement whose limits are not stated is worse than none, because it gets quoted without them.
Why there was no baseline before today
Not because nobody ran it. Because it could not have returned anything but zero, for two structural reasons that had nothing to do with agent behaviour:
- 55 of 85 role skill bindings resolved to skills that were never authored.
- Even resolved skills had no delivery channel to a mission agent — the catalogue's only route was an MCP server that mission claws cannot reach.
Both were fixed in the two commits preceding this document. Anyone who had run this measurement in July would have concluded "our agents ignore their skills", which would have been false and expensive.
Method, and what it cannot see
Scored by cm_api::skill_use from what the platform records: the
prompt.composed event (the exact bytes an agent received) and the reasoning
events (what it said it did). No re-derivation from the catalogue — the
catalogue changes, and now that agents author their own skills it changes by
itself.
Trigger is not observable here, and that is a finding
The paper measures agents under progressive disclosure: the agent sees a name and description and must decide to retrieve the body. That retrieval is a tool call, which makes Trigger observable.
We do not deliver skills that way. pinned_skills_text inlines full bodies into
the prompt, because mission claws run on claude_cli, which cannot surface a
tool call — there is nothing to retrieve with. The agent never reaches for a
skill; it is simply holding one.
So Trigger is reported as not_observable with the reason attached, never as
zero. Scoring it zero would report a delivery-model property as an agent
failure — the same confusion that kept 55 empty bindings invisible.
Compliance and Boundary are checked mechanically, or not at all
Only skills whose procedure has a machine-checkable consequence are scored.
Everything else returns not_applicable rather than a guess: a heuristic that
scores prose by keyword overlap produces a number that looks like a measurement
and is not one.
Compliance for int-xx-marker-protocol is checked by running the real
task_card_parser, not a copy of its rules — a second implementation would
drift, and then the score would pass while the mission loop still stalled.
The runs
Two missions on the container/ZeroClaw tier, local stack, research_only.
| run 1 | run 2 | |
|---|---|---|
| distinct skills delivered | 3 | 9 |
| total deliveries (per role prompt) | 3 | 14 |
| phantom "skills" scored | 2 | 0 |
Run 1's phantom entries are the finding of the run, described below.
Run 2, per skill (all source_kind=builtin; no agent-authored skill has been
delivered yet):
| skill | deliveries | compliance | boundary |
|---|---|---|---|
int-xx-marker-protocol |
1 | pass | n/a |
small-focused-commits |
4 | n/a | n/a |
cargo-test-driven-development |
2 | n/a | n/a |
workspace-repo-commit-protocol |
2 | n/a | n/a |
decompose-int-items |
1 | n/a | n/a |
write-rust-current-edition |
1 | n/a | n/a |
code-review-checklist |
1 | n/a | n/a |
criterion-benchmarking |
1 | n/a | n/a |
tdd-red-green-refactor |
1 | n/a | n/a |
n = 2 runs. No spread is reported because two runs cannot establish one.
This is a baseline in the sense of "the first honest number", not in the sense
of metrics-baseline-comparison.md, which requires enough runs to see the noise
floor before any change is judged against it. Do not compare a future number to
this one without first establishing that floor.
What the measurement found
Three defects, none of which any test or log would have surfaced.
1. The prompt format made its own record unparseable
Skills were introduced with ## <name>, and skill bodies are markdown full of
## headings. Run 1 duly scored "Sizing heuristic" and "The output
shape" — both subheadings inside decompose-int-items — as skills with no
catalogue row.
Fixed with an unambiguous --- SKILL: <name> --- marker, and both writers now
share one renderer so the reader cannot drift from the writer.
2. A prompt was recorded that was never sent
The phase prompt was recorded at the dispatch fork, before the tier was chosen.
The container tier does not send that text — it sends the bare task and appends
skills per turn. So every container mission logged a solo prompt that reached
no agent.
A provenance record of something that did not happen is worse than no record: it is the wrong answer, delivered confidently. Recording now happens inside each tier, and a test asserts every launcher records the prompt it actually sends.
3. The skill documents a marker the platform never implemented
int-xx-marker-protocol lists PLAN_COMPLETE: INT-NN in its ladder.
task_card_parser has no such kind and never has. An agent following the
skill exactly emits a marker that is silently ignored.
Observed live: run 2's planner emitted PLAN_COMPLETE: INT-01..02, which is
also the range form — on the kinds that are parsed, that yields the id
INT-01..02, a task card for an item that does not exist while the two real
items stay open.
This is a skill/implementation mismatch, not an agent failure, and it is
precisely what this measurement exists to find: the agent did what it was told,
and what it was told was wrong. Both shapes are now scored as failures; the
underlying reconciliation — implement PLAN_COMPLETE or remove it from the
skill — is deliberately left as a decision rather than guessed at here.
Honest limits
- Two runs, one tier, one workflow. Nothing here generalises to the microVM or session tiers yet, and this document should not be read as if it does.
- 7 of 9 skills scored
not_applicableon both observable axes. That is not a pass. It means we cannot currently tell whether those skills changed anything, and the next unit of work is mechanical checks for more of them —workspace-repo-commit-protocolandsmall-focused-commitsboth have checkable consequences in git history. - No agent-authored skill has been measured. Self-authoring shipped in the
same pass;
source_kindis carried through the scorer specifically so a rising score on agent-authored skills is visible rather than averaged in. - Evidence expires. Mission events are reaped after 7 days unless
retain_events_untilis set. Both runs here are held for 90 days. An empty score means "no evidence", never "no compliance", and the API says so in its payload rather than leaving the caller to infer it.