---
name: criterion-benchmarking
description: Writing benchmarks whose numbers mean something — warmup, distributions, and the changes that are noise.
when_to_use: You are asked to benchmark a change, or to show a performance claim is real.
tags: [gpu, rust, benchmarking]
---

# A benchmark is an experiment

Most performance claims fail not because the code is slow but because the
measurement cannot support the claim.

## What criterion does for you

`criterion` runs the routine many times, discards warmup, and reports a
confidence interval rather than a single number. Take that seriously: if the
intervals for before and after overlap, **you have not measured an improvement**,
whatever the point estimates say.

```rust
fn bench_search(c: &mut Criterion) {
    let index = build_index(10_000);          // setup OUTSIDE the timed closure
    c.bench_function("search/10k", |b| {
        b.iter(|| index.search(black_box(&query), 10))
    });
}
```

Two mistakes this shape avoids:
- **Setup inside `iter`** measures the setup. If the setup must be per-iteration,
  use `iter_batched` so it is excluded.
- **A missing `black_box`** lets the optimiser delete the work entirely. A
  benchmark that got 400× faster after a refactor usually got deleted, not
  optimised.

## Report the shape, not the headline

"34.7% faster" invites a follow-up question the number cannot answer. Give the
distribution, the input size, and the machine. A change that is 30% faster at
10k elements and 5% slower at 10M is a trade-off, and only the sweep shows it.

## What is noise

On a laptop, expect 5-10% run-to-run variance from thermal state and other
processes alone. Treat anything under that as unmeasured. If a change is
genuinely small but real, prove it by increasing iterations rather than by
asserting it — and if that is too expensive, say the change was too small to
measure rather than reporting the point estimate as fact.

## Benchmarks are also regression tests

The value is in the baseline. Record it (`--save-baseline`), compare against it
(`--baseline`), and keep the numbers with the commit that produced them.
`benchmark_snapshots` exists for exactly this — a benchmark whose history is
lost measures nothing the next time.