feat(skills): the first Skill-Use measurement, and the three defects it found

Scored on the paper's three axes against two real missions on the local
stack. docs/SKILL-USE-BASELINE.md has the numbers, the method, and the
limits.

Trigger is reported as NOT OBSERVABLE, never zero

The paper measures progressive disclosure: the agent sees a name and
description and must retrieve the body, and that retrieval is the Trigger
event. We inline full bodies, because mission claws run on claude_cli which
cannot surface a tool call — there is nothing to retrieve with. So the
agent never reaches for a skill, it simply holds one.

Scoring that zero would report a delivery-model property as an agent
failure, which is the same confusion that kept 55 empty bindings invisible
for months. The verdict type carries NotObservable(reason) as a distinct
case from Fail for exactly this.

Compliance is checked by running the REAL task_card_parser rather than a
copy of its rules — a second implementation would drift, and then the score
would pass while the mission loop still stalled. Skills without a
machine-checkable consequence score not_applicable rather than a guess.

WHAT THE MEASUREMENT FOUND

1. The prompt format made its own record unparseable. Skills were
   introduced with `## <name>` and skill bodies are markdown full of `##`
   headings, so run 1 scored "Sizing heuristic" and "The output shape" —
   subheadings inside decompose-int-items — as skills with no catalogue
   row. Now an unambiguous `--- SKILL: <name> ---` marker, with both
   writers sharing one renderer so the reader cannot drift from the writer.

2. A prompt was recorded that was never sent. My own Phase 1 work recorded
   the phase prompt at the dispatch fork, before the tier was chosen — and
   the container tier does not send that text, it sends the bare task and
   appends skills per turn. Every container mission logged a `solo` prompt
   that reached no agent. A provenance record of something that did not
   happen is worse than no record: it is the wrong answer, delivered
   confidently. Recording now happens inside each tier, with a test that
   every launcher records the prompt it actually sends.

3. int-xx-marker-protocol documents a marker the platform never
   implemented. PLAN_COMPLETE is in the skill's ladder and task_card_parser
   has no such kind and never has, so an agent following the skill exactly
   emits a marker that is silently ignored. Observed live: run 2's planner
   emitted `PLAN_COMPLETE: INT-01..02`, which is also the range form — on
   the kinds that ARE parsed that yields the id `INT-01..02`, a task card
   for an item that does not exist while the two real items stay open.

   This is a skill/implementation mismatch, not an agent failure, and it is
   exactly what the measurement exists to find: the agent did what it was
   told and what it was told was wrong. Both shapes now score as failures.
   The reconciliation — implement PLAN_COMPLETE or drop it from the skill —
   is left as a decision rather than guessed at.

The boot log now shows what the plan asked for: 53 skills, 11 templates,
every one `N role skills bound` with NO unresolved clause. Live missions
confirm per-role delivery — the planner receives decompose-int-items, the
coder receives write-rust-current-edition.

GET /api/missions/{id}/skill-use exposes the scores, and says in its
payload whether an empty result means "nothing delivered" or "the evidence
was reaped" — those have very different causes and must not look the same.

n = 2. No spread is reported because two runs cannot establish one, and the
document says so rather than letting the number be quoted as a baseline it
is not.

Full workspace suite green: 106 binaries.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-19 10:54:58 -07:00
co-authored by Claude Opus 5
parent 769e002bb3
commit 91a6b4e304
6 changed files with 772 additions and 24 deletions
+30 -5
View File
@@ -49,6 +49,35 @@ const TURN_TIMEOUT: Duration = Duration::from_secs(3600);
/// foundation set, and it is stated in the prompt when it fires.
pub(crate) const MAX_PINNED_SKILL_BYTES: usize = 24_000;
/// The line that introduces each skill in a prompt.
///
/// NOT a markdown heading. The first version used `## <name>`, and skill bodies
/// are markdown that contain their own `##` headings — so anything reading the
/// prompt back counted every section of every body as a separate skill. A live
/// mission scored "Sizing heuristic" and "The output shape" as skills, which is
/// what surfaced it.
///
/// This marker cannot occur inside a body, so the prompt stays parseable by
/// whatever reads it later. Skills are written by one function
/// ([`render_pinned_skill`]) for the same reason: two renderers would drift and
/// the reader would silently match only one.
pub const SKILL_MARKER: &str = "--- SKILL: ";
/// One skill, rendered for a prompt.
pub fn render_pinned_skill(name: &str, body: &str) -> String {
format!("\n{SKILL_MARKER}{name} ---\n{body}\n")
}
/// The skill names a rendered prompt delivered.
pub fn skill_names_in(prompt: &str) -> Vec<String> {
prompt
.lines()
.filter_map(|l| l.trim().strip_prefix(SKILL_MARKER))
.map(|rest| rest.trim_end_matches(" ---").trim().to_string())
.filter(|n| !n.is_empty())
.collect()
}
pub struct ZeroClawDriveExecutor {
/// Gateway base URL, e.g. `http://127.0.0.1:42617`.
gateway_url: String,
@@ -329,11 +358,7 @@ impl ZeroClawDriveExecutor {
));
continue;
}
out.push_str("\n## ");
out.push_str(&b.skill.name);
out.push('\n');
out.push_str(&b.skill.body);
out.push('\n');
out.push_str(&render_pinned_skill(&b.skill.name, &b.skill.body));
n += 1;
}
if n == 0 {