Omar SobhandClaude Opus 5
4358964c05
55 of 85 role skill bindings pointed at skills that were never authored,
so 10 of 11 team templates bound a smaller context bundle than their role
prompts assumed. Three roles bound nothing at all (gpu.bench_engineer,
threejs.shader_author, threejs.perf_engineer) while their prompts described
procedures they had no way to read.
The loader comment at team_template_loader.rs:167 already diagnosed this —
snake_case slugs in TOML against kebab-case skill files — and it was
half-fixed: the kebab names were corrected, the snake_case ones left.
It was invisible because both existing tests assert authored ⊆ referenced
(30/30, green) and the second explicitly declines to check the other
direction. So the failing half was the half nobody asserted.
Resolved every name by one of three explicit choices:
- 23 skills authored where the role genuinely needed the procedure
(gpu, threejs, research, analysis, frontend, mobile, backend, platform)
- renames onto authored skills where one existed in substance, including
the four-near-duplicate cases that collapse onto one real skill
- 22 aspirational references deleted — a binding an agent cannot read is
a promise, not a capability
Two tests now hold it. The unit test checks referenced ⊆ authored against
the files. The new integration test runs both loaders in boot order and
asserts the bindings survive the trip through the database, which is a
different question: resolution goes through skills_catalog rows, so a skill
file that exists but fails to ingest still leaves the role empty.
Negative controls: the unit test failed naming all 55; the integration test
fails naming the exact role when one name is reverted.
threejs.shader_author and .perf_engineer gained a second and third skill
after the collapse — pin_in_context pins idx < 2, so a role left with one
skill silently pins less than the policy intends.
Co-Authored-By: Claude Opus 5 <[email protected]>
54 lines
2.2 KiB
Markdown
54 lines
2.2 KiB
Markdown
---
|
|
name: metrics-baseline-comparison
|
|
description: Judging whether an agent or system change actually improved anything, against a baseline that existed first.
|
|
when_to_use: You are evaluating whether a change to an agent, prompt or model made things better.
|
|
tags: [platform, evaluation]
|
|
---
|
|
|
|
# Without a baseline there is no comparison, only an anecdote
|
|
|
|
The most common failure in agent evaluation is measuring after the change and
|
|
comparing against a memory of before.
|
|
|
|
## Record the baseline before changing anything
|
|
|
|
Whatever the metric — task success, tokens per task, wall clock, human
|
|
corrections — capture it on the current system first, over enough runs to see
|
|
the spread. Agent runs are high variance: two runs of the same task on the same
|
|
prompt can differ enormously, so a single before and a single after tells you
|
|
nothing.
|
|
|
|
## Compare like with like
|
|
|
|
Hold constant everything you are not testing: the task set, the model, the
|
|
runtime tier, the repository state. `MemoryLake on MemoryArena` is the shape to
|
|
copy — same framework, same model alias, same task samples, same scoring code,
|
|
with the memory backend the intentionally changed component.
|
|
|
|
If two things changed, the result attributes to neither.
|
|
|
|
## Variance first, effect second
|
|
|
|
Run the *unchanged* system several times to learn the noise floor. An
|
|
improvement smaller than the run-to-run spread has not been demonstrated,
|
|
however good the story is. This one discipline invalidates most informal agent
|
|
comparisons, including ones made in good faith.
|
|
|
|
## Report the distribution and the n
|
|
|
|
"9 of 40" is honest and comparable. "Significantly better" is neither. Give the
|
|
count, the denominator and the spread, and say how many runs each side had.
|
|
|
|
## Beware the metric becoming the target
|
|
|
|
An agent optimised against a judge learns the judge. If the measure is a model's
|
|
verdict, keep an independent check — a different provider family, or a
|
|
deterministic assertion the agent cannot talk its way past. A rising score with
|
|
flat real-world outcomes is the signal that this has happened.
|
|
|
|
## A negative result is a result
|
|
|
|
"No measurable difference" is worth recording and prevents the change being
|
|
proposed again in three months. Most changes do not help; a process that only
|
|
reports wins is not measuring.
|