---
name: metrics-baseline-comparison
description: Judging whether an agent or system change actually improved anything, against a baseline that existed first.
when_to_use: You are evaluating whether a change to an agent, prompt or model made things better.
tags: [platform, evaluation]
---

# Without a baseline there is no comparison, only an anecdote

The most common failure in agent evaluation is measuring after the change and
comparing against a memory of before.

## Record the baseline before changing anything

Whatever the metric — task success, tokens per task, wall clock, human
corrections — capture it on the current system first, over enough runs to see
the spread. Agent runs are high variance: two runs of the same task on the same
prompt can differ enormously, so a single before and a single after tells you
nothing.

## Compare like with like

Hold constant everything you are not testing: the task set, the model, the
runtime tier, the repository state. `MemoryLake on MemoryArena` is the shape to
copy — same framework, same model alias, same task samples, same scoring code,
with the memory backend the intentionally changed component.

If two things changed, the result attributes to neither.

## Variance first, effect second

Run the *unchanged* system several times to learn the noise floor. An
improvement smaller than the run-to-run spread has not been demonstrated,
however good the story is. This one discipline invalidates most informal agent
comparisons, including ones made in good faith.

## Report the distribution and the n

"9 of 40" is honest and comparable. "Significantly better" is neither. Give the
count, the denominator and the spread, and say how many runs each side had.

## Beware the metric becoming the target

An agent optimised against a judge learns the judge. If the measure is a model's
verdict, keep an independent check — a different provider family, or a
deterministic assertion the agent cannot talk its way past. A rising score with
flat real-world outcomes is the signal that this has happened.

## A negative result is a result

"No measurable difference" is worth recording and prevents the change being
proposed again in three months. Most changes do not help; a process that only
reports wins is not measuring.