fix(skills): every team-template skill binding now resolves

55 of 85 role skill bindings pointed at skills that were never authored,
so 10 of 11 team templates bound a smaller context bundle than their role
prompts assumed. Three roles bound nothing at all (gpu.bench_engineer,
threejs.shader_author, threejs.perf_engineer) while their prompts described
procedures they had no way to read.

The loader comment at team_template_loader.rs:167 already diagnosed this —
snake_case slugs in TOML against kebab-case skill files — and it was
half-fixed: the kebab names were corrected, the snake_case ones left.

It was invisible because both existing tests assert authored ⊆ referenced
(30/30, green) and the second explicitly declines to check the other
direction. So the failing half was the half nobody asserted.

Resolved every name by one of three explicit choices:

  - 23 skills authored where the role genuinely needed the procedure
    (gpu, threejs, research, analysis, frontend, mobile, backend, platform)
  - renames onto authored skills where one existed in substance, including
    the four-near-duplicate cases that collapse onto one real skill
  - 22 aspirational references deleted — a binding an agent cannot read is
    a promise, not a capability

Two tests now hold it. The unit test checks referenced ⊆ authored against
the files. The new integration test runs both loaders in boot order and
asserts the bindings survive the trip through the database, which is a
different question: resolution goes through skills_catalog rows, so a skill
file that exists but fails to ingest still leaves the role empty.

Negative controls: the unit test failed naming all 55; the integration test
fails naming the exact role when one name is reverted.

threejs.shader_author and .perf_engineer gained a second and third skill
after the collapse — pin_in_context pins idx < 2, so a role left with one
skill silently pins less than the policy intends.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-19 07:42:48 -07:00
co-authored by Claude Opus 5
parent ba98c29481
commit 4358964c05
35 changed files with 1392 additions and 28 deletions
@@ -0,0 +1,53 @@
---
name: metrics-baseline-comparison
description: Judging whether an agent or system change actually improved anything, against a baseline that existed first.
when_to_use: You are evaluating whether a change to an agent, prompt or model made things better.
tags: [platform, evaluation]
---
# Without a baseline there is no comparison, only an anecdote
The most common failure in agent evaluation is measuring after the change and
comparing against a memory of before.
## Record the baseline before changing anything
Whatever the metric — task success, tokens per task, wall clock, human
corrections — capture it on the current system first, over enough runs to see
the spread. Agent runs are high variance: two runs of the same task on the same
prompt can differ enormously, so a single before and a single after tells you
nothing.
## Compare like with like
Hold constant everything you are not testing: the task set, the model, the
runtime tier, the repository state. `MemoryLake on MemoryArena` is the shape to
copy — same framework, same model alias, same task samples, same scoring code,
with the memory backend the intentionally changed component.
If two things changed, the result attributes to neither.
## Variance first, effect second
Run the *unchanged* system several times to learn the noise floor. An
improvement smaller than the run-to-run spread has not been demonstrated,
however good the story is. This one discipline invalidates most informal agent
comparisons, including ones made in good faith.
## Report the distribution and the n
"9 of 40" is honest and comparable. "Significantly better" is neither. Give the
count, the denominator and the spread, and say how many runs each side had.
## Beware the metric becoming the target
An agent optimised against a judge learns the judge. If the measure is a model's
verdict, keep an independent check — a different provider family, or a
deterministic assertion the agent cannot talk its way past. A rising score with
flat real-world outcomes is the signal that this has happened.
## A negative result is a result
"No measurable difference" is worth recording and prevents the change being
proposed again in three months. Most changes do not help; a process that only
reports wins is not measuring.