feat(skills): the first Skill-Use measurement, and the three defects it found

Scored on the paper's three axes against two real missions on the local
stack. docs/SKILL-USE-BASELINE.md has the numbers, the method, and the
limits.

Trigger is reported as NOT OBSERVABLE, never zero

The paper measures progressive disclosure: the agent sees a name and
description and must retrieve the body, and that retrieval is the Trigger
event. We inline full bodies, because mission claws run on claude_cli which
cannot surface a tool call — there is nothing to retrieve with. So the
agent never reaches for a skill, it simply holds one.

Scoring that zero would report a delivery-model property as an agent
failure, which is the same confusion that kept 55 empty bindings invisible
for months. The verdict type carries NotObservable(reason) as a distinct
case from Fail for exactly this.

Compliance is checked by running the REAL task_card_parser rather than a
copy of its rules — a second implementation would drift, and then the score
would pass while the mission loop still stalled. Skills without a
machine-checkable consequence score not_applicable rather than a guess.

WHAT THE MEASUREMENT FOUND

1. The prompt format made its own record unparseable. Skills were
   introduced with `## <name>` and skill bodies are markdown full of `##`
   headings, so run 1 scored "Sizing heuristic" and "The output shape" —
   subheadings inside decompose-int-items — as skills with no catalogue
   row. Now an unambiguous `--- SKILL: <name> ---` marker, with both
   writers sharing one renderer so the reader cannot drift from the writer.

2. A prompt was recorded that was never sent. My own Phase 1 work recorded
   the phase prompt at the dispatch fork, before the tier was chosen — and
   the container tier does not send that text, it sends the bare task and
   appends skills per turn. Every container mission logged a `solo` prompt
   that reached no agent. A provenance record of something that did not
   happen is worse than no record: it is the wrong answer, delivered
   confidently. Recording now happens inside each tier, with a test that
   every launcher records the prompt it actually sends.

3. int-xx-marker-protocol documents a marker the platform never
   implemented. PLAN_COMPLETE is in the skill's ladder and task_card_parser
   has no such kind and never has, so an agent following the skill exactly
   emits a marker that is silently ignored. Observed live: run 2's planner
   emitted `PLAN_COMPLETE: INT-01..02`, which is also the range form — on
   the kinds that ARE parsed that yields the id `INT-01..02`, a task card
   for an item that does not exist while the two real items stay open.

   This is a skill/implementation mismatch, not an agent failure, and it is
   exactly what the measurement exists to find: the agent did what it was
   told and what it was told was wrong. Both shapes now score as failures.
   The reconciliation — implement PLAN_COMPLETE or drop it from the skill —
   is left as a decision rather than guessed at.

The boot log now shows what the plan asked for: 53 skills, 11 templates,
every one `N role skills bound` with NO unresolved clause. Live missions
confirm per-role delivery — the planner receives decompose-int-items, the
coder receives write-rust-current-edition.

GET /api/missions/{id}/skill-use exposes the scores, and says in its
payload whether an empty result means "nothing delivered" or "the evidence
was reaped" — those have very different causes and must not look the same.

n = 2. No spread is reported because two runs cannot establish one, and the
document says so rather than letting the number be quoted as a baseline it
is not.

Full workspace suite green: 106 binaries.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-19 10:54:58 -07:00
co-authored by Claude Opus 5
parent 769e002bb3
commit 91a6b4e304
6 changed files with 772 additions and 24 deletions
+5
View File
@@ -53,6 +53,7 @@ pub mod runtime_provision;
pub mod security_scan;
pub mod session_executor;
pub mod skill_self_authoring;
pub mod skill_use;
pub mod skills_loader;
pub mod subscription;
pub mod swarm;
@@ -590,6 +591,10 @@ pub fn router(state: AppState) -> Router {
"/api/missions/{id}/phases/{phase_id}/evaluations",
get(routes::missions::list_phase_evaluations),
)
.route(
"/api/missions/{id}/skill-use",
get(routes::missions::skill_use),
)
.route(
"/api/missions/{id}/teams",
get(routes::missions::list_teams),