feat(skills): the first Skill-Use measurement, and the three defects it found
Scored on the paper's three axes against two real missions on the local
stack. docs/SKILL-USE-BASELINE.md has the numbers, the method, and the
limits.
Trigger is reported as NOT OBSERVABLE, never zero
The paper measures progressive disclosure: the agent sees a name and
description and must retrieve the body, and that retrieval is the Trigger
event. We inline full bodies, because mission claws run on claude_cli which
cannot surface a tool call — there is nothing to retrieve with. So the
agent never reaches for a skill, it simply holds one.
Scoring that zero would report a delivery-model property as an agent
failure, which is the same confusion that kept 55 empty bindings invisible
for months. The verdict type carries NotObservable(reason) as a distinct
case from Fail for exactly this.
Compliance is checked by running the REAL task_card_parser rather than a
copy of its rules — a second implementation would drift, and then the score
would pass while the mission loop still stalled. Skills without a
machine-checkable consequence score not_applicable rather than a guess.
WHAT THE MEASUREMENT FOUND
1. The prompt format made its own record unparseable. Skills were
introduced with `## <name>` and skill bodies are markdown full of `##`
headings, so run 1 scored "Sizing heuristic" and "The output shape" —
subheadings inside decompose-int-items — as skills with no catalogue
row. Now an unambiguous `--- SKILL: <name> ---` marker, with both
writers sharing one renderer so the reader cannot drift from the writer.
2. A prompt was recorded that was never sent. My own Phase 1 work recorded
the phase prompt at the dispatch fork, before the tier was chosen — and
the container tier does not send that text, it sends the bare task and
appends skills per turn. Every container mission logged a `solo` prompt
that reached no agent. A provenance record of something that did not
happen is worse than no record: it is the wrong answer, delivered
confidently. Recording now happens inside each tier, with a test that
every launcher records the prompt it actually sends.
3. int-xx-marker-protocol documents a marker the platform never
implemented. PLAN_COMPLETE is in the skill's ladder and task_card_parser
has no such kind and never has, so an agent following the skill exactly
emits a marker that is silently ignored. Observed live: run 2's planner
emitted `PLAN_COMPLETE: INT-01..02`, which is also the range form — on
the kinds that ARE parsed that yields the id `INT-01..02`, a task card
for an item that does not exist while the two real items stay open.
This is a skill/implementation mismatch, not an agent failure, and it is
exactly what the measurement exists to find: the agent did what it was
told and what it was told was wrong. Both shapes now score as failures.
The reconciliation — implement PLAN_COMPLETE or drop it from the skill —
is left as a decision rather than guessed at.
The boot log now shows what the plan asked for: 53 skills, 11 templates,
every one `N role skills bound` with NO unresolved clause. Live missions
confirm per-role delivery — the planner receives decompose-int-items, the
coder receives write-rust-current-edition.
GET /api/missions/{id}/skill-use exposes the scores, and says in its
payload whether an empty result means "nothing delivered" or "the evidence
was reaped" — those have very different causes and must not look the same.
n = 2. No spread is reported because two runs cannot establish one, and the
document says so rather than letting the number be quoted as a baseline it
is not.
Full workspace suite green: 106 binaries.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
769e002bb3
commit
91a6b4e304
@@ -1223,6 +1223,42 @@ pub async fn retry_phase(
|
||||
/// One row per pass. The `reason` is the operator-facing explanation of why a
|
||||
/// phase iterated (or stopped), and is the same text fed back to the agents as
|
||||
/// guidance for the following pass.
|
||||
/// Skill-Use scores for a mission: did the skills we delivered change what the
|
||||
/// agent did?
|
||||
///
|
||||
/// Reads only what was recorded — the prompts the agent received and the
|
||||
/// narratives it returned. An empty result means the evidence is gone (events
|
||||
/// are reaped after 7 days unless `retain_events_until` is set), NOT that no
|
||||
/// skill was followed, and the caller has to present it that way.
|
||||
pub async fn skill_use(
|
||||
State(state): State<AppState>,
|
||||
Authed(user): Authed,
|
||||
Path(id): Path<Uuid>,
|
||||
) -> Result<Json<Value>, ApiError> {
|
||||
let _ = cm_db::repo::missions::get(&state.pool, id, user.workspace_id.as_uuid())
|
||||
.await?
|
||||
.ok_or(ApiError::NotFound)?;
|
||||
let scores = crate::skill_use::score_mission(&state.pool, id)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
eprintln!("skill_use: scoring mission {id} failed: {e}");
|
||||
ApiError::Internal
|
||||
})?;
|
||||
Ok(Json(serde_json::json!({
|
||||
"mission_id": id,
|
||||
"skills": scores,
|
||||
// Said in the payload rather than left for the reader to infer: an
|
||||
// empty list has two very different causes and they must not look the
|
||||
// same to whoever consumes this.
|
||||
"evidence": if scores.is_empty() {
|
||||
"no delivered skills found in the recorded prompts — either none \
|
||||
were delivered, or the events have been reaped"
|
||||
} else {
|
||||
"scored from recorded prompt.composed and reasoning events"
|
||||
},
|
||||
})))
|
||||
}
|
||||
|
||||
pub async fn list_phase_evaluations(
|
||||
State(state): State<AppState>,
|
||||
Authed(user): Authed,
|
||||
|
||||
Reference in New Issue
Block a user