feat(skills): the first Skill-Use measurement, and the three defects it found
Scored on the paper's three axes against two real missions on the local
stack. docs/SKILL-USE-BASELINE.md has the numbers, the method, and the
limits.
Trigger is reported as NOT OBSERVABLE, never zero
The paper measures progressive disclosure: the agent sees a name and
description and must retrieve the body, and that retrieval is the Trigger
event. We inline full bodies, because mission claws run on claude_cli which
cannot surface a tool call — there is nothing to retrieve with. So the
agent never reaches for a skill, it simply holds one.
Scoring that zero would report a delivery-model property as an agent
failure, which is the same confusion that kept 55 empty bindings invisible
for months. The verdict type carries NotObservable(reason) as a distinct
case from Fail for exactly this.
Compliance is checked by running the REAL task_card_parser rather than a
copy of its rules — a second implementation would drift, and then the score
would pass while the mission loop still stalled. Skills without a
machine-checkable consequence score not_applicable rather than a guess.
WHAT THE MEASUREMENT FOUND
1. The prompt format made its own record unparseable. Skills were
introduced with `## <name>` and skill bodies are markdown full of `##`
headings, so run 1 scored "Sizing heuristic" and "The output shape" —
subheadings inside decompose-int-items — as skills with no catalogue
row. Now an unambiguous `--- SKILL: <name> ---` marker, with both
writers sharing one renderer so the reader cannot drift from the writer.
2. A prompt was recorded that was never sent. My own Phase 1 work recorded
the phase prompt at the dispatch fork, before the tier was chosen — and
the container tier does not send that text, it sends the bare task and
appends skills per turn. Every container mission logged a `solo` prompt
that reached no agent. A provenance record of something that did not
happen is worse than no record: it is the wrong answer, delivered
confidently. Recording now happens inside each tier, with a test that
every launcher records the prompt it actually sends.
3. int-xx-marker-protocol documents a marker the platform never
implemented. PLAN_COMPLETE is in the skill's ladder and task_card_parser
has no such kind and never has, so an agent following the skill exactly
emits a marker that is silently ignored. Observed live: run 2's planner
emitted `PLAN_COMPLETE: INT-01..02`, which is also the range form — on
the kinds that ARE parsed that yields the id `INT-01..02`, a task card
for an item that does not exist while the two real items stay open.
This is a skill/implementation mismatch, not an agent failure, and it is
exactly what the measurement exists to find: the agent did what it was
told and what it was told was wrong. Both shapes now score as failures.
The reconciliation — implement PLAN_COMPLETE or drop it from the skill —
is left as a decision rather than guessed at.
The boot log now shows what the plan asked for: 53 skills, 11 templates,
every one `N role skills bound` with NO unresolved clause. Live missions
confirm per-role delivery — the planner receives decompose-int-items, the
coder receives write-rust-current-edition.
GET /api/missions/{id}/skill-use exposes the scores, and says in its
payload whether an empty result means "nothing delivered" or "the evidence
was reaped" — those have very different causes and must not look the same.
n = 2. No spread is reported because two runs cannot establish one, and the
document says so rather than letting the number be quoted as a baseline it
is not.
Full workspace suite green: 106 binaries.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
769e002bb3
commit
91a6b4e304
@@ -49,6 +49,35 @@ const TURN_TIMEOUT: Duration = Duration::from_secs(3600);
|
||||
/// foundation set, and it is stated in the prompt when it fires.
|
||||
pub(crate) const MAX_PINNED_SKILL_BYTES: usize = 24_000;
|
||||
|
||||
/// The line that introduces each skill in a prompt.
|
||||
///
|
||||
/// NOT a markdown heading. The first version used `## <name>`, and skill bodies
|
||||
/// are markdown that contain their own `##` headings — so anything reading the
|
||||
/// prompt back counted every section of every body as a separate skill. A live
|
||||
/// mission scored "Sizing heuristic" and "The output shape" as skills, which is
|
||||
/// what surfaced it.
|
||||
///
|
||||
/// This marker cannot occur inside a body, so the prompt stays parseable by
|
||||
/// whatever reads it later. Skills are written by one function
|
||||
/// ([`render_pinned_skill`]) for the same reason: two renderers would drift and
|
||||
/// the reader would silently match only one.
|
||||
pub const SKILL_MARKER: &str = "--- SKILL: ";
|
||||
|
||||
/// One skill, rendered for a prompt.
|
||||
pub fn render_pinned_skill(name: &str, body: &str) -> String {
|
||||
format!("\n{SKILL_MARKER}{name} ---\n{body}\n")
|
||||
}
|
||||
|
||||
/// The skill names a rendered prompt delivered.
|
||||
pub fn skill_names_in(prompt: &str) -> Vec<String> {
|
||||
prompt
|
||||
.lines()
|
||||
.filter_map(|l| l.trim().strip_prefix(SKILL_MARKER))
|
||||
.map(|rest| rest.trim_end_matches(" ---").trim().to_string())
|
||||
.filter(|n| !n.is_empty())
|
||||
.collect()
|
||||
}
|
||||
|
||||
pub struct ZeroClawDriveExecutor {
|
||||
/// Gateway base URL, e.g. `http://127.0.0.1:42617`.
|
||||
gateway_url: String,
|
||||
@@ -329,11 +358,7 @@ impl ZeroClawDriveExecutor {
|
||||
));
|
||||
continue;
|
||||
}
|
||||
out.push_str("\n## ");
|
||||
out.push_str(&b.skill.name);
|
||||
out.push('\n');
|
||||
out.push_str(&b.skill.body);
|
||||
out.push('\n');
|
||||
out.push_str(&render_pinned_skill(&b.skill.name, &b.skill.body));
|
||||
n += 1;
|
||||
}
|
||||
if n == 0 {
|
||||
|
||||
Reference in New Issue
Block a user