Omar SobhandClaude Opus 5
4358964c05
55 of 85 role skill bindings pointed at skills that were never authored,
so 10 of 11 team templates bound a smaller context bundle than their role
prompts assumed. Three roles bound nothing at all (gpu.bench_engineer,
threejs.shader_author, threejs.perf_engineer) while their prompts described
procedures they had no way to read.
The loader comment at team_template_loader.rs:167 already diagnosed this —
snake_case slugs in TOML against kebab-case skill files — and it was
half-fixed: the kebab names were corrected, the snake_case ones left.
It was invisible because both existing tests assert authored ⊆ referenced
(30/30, green) and the second explicitly declines to check the other
direction. So the failing half was the half nobody asserted.
Resolved every name by one of three explicit choices:
- 23 skills authored where the role genuinely needed the procedure
(gpu, threejs, research, analysis, frontend, mobile, backend, platform)
- renames onto authored skills where one existed in substance, including
the four-near-duplicate cases that collapse onto one real skill
- 22 aspirational references deleted — a binding an agent cannot read is
a promise, not a capability
Two tests now hold it. The unit test checks referenced ⊆ authored against
the files. The new integration test runs both loaders in boot order and
asserts the bindings survive the trip through the database, which is a
different question: resolution goes through skills_catalog rows, so a skill
file that exists but fails to ingest still leaves the role empty.
Negative controls: the unit test failed naming all 55; the integration test
fails naming the exact role when one name is reverted.
threejs.shader_author and .perf_engineer gained a second and third skill
after the collapse — pin_in_context pins idx < 2, so a role left with one
skill silently pins less than the policy intends.
Co-Authored-By: Claude Opus 5 <[email protected]>
2.2 KiB
name, description, when_to_use, tags
| name | description | when_to_use | tags | ||
|---|---|---|---|---|---|
| metrics-baseline-comparison | Judging whether an agent or system change actually improved anything, against a baseline that existed first. | You are evaluating whether a change to an agent, prompt or model made things better. |
|
Without a baseline there is no comparison, only an anecdote
The most common failure in agent evaluation is measuring after the change and comparing against a memory of before.
Record the baseline before changing anything
Whatever the metric — task success, tokens per task, wall clock, human corrections — capture it on the current system first, over enough runs to see the spread. Agent runs are high variance: two runs of the same task on the same prompt can differ enormously, so a single before and a single after tells you nothing.
Compare like with like
Hold constant everything you are not testing: the task set, the model, the
runtime tier, the repository state. MemoryLake on MemoryArena is the shape to
copy — same framework, same model alias, same task samples, same scoring code,
with the memory backend the intentionally changed component.
If two things changed, the result attributes to neither.
Variance first, effect second
Run the unchanged system several times to learn the noise floor. An improvement smaller than the run-to-run spread has not been demonstrated, however good the story is. This one discipline invalidates most informal agent comparisons, including ones made in good faith.
Report the distribution and the n
"9 of 40" is honest and comparable. "Significantly better" is neither. Give the count, the denominator and the spread, and say how many runs each side had.
Beware the metric becoming the target
An agent optimised against a judge learns the judge. If the measure is a model's verdict, keep an independent check — a different provider family, or a deterministic assertion the agent cannot talk its way past. A rising score with flat real-world outcomes is the signal that this has happened.
A negative result is a result
"No measurable difference" is worth recording and prevents the change being proposed again in three months. Most changes do not help; a process that only reports wins is not measuring.