---
name: gpu-profiling-workflow
description: Using Nsight, rocprof and Metal frame capture to find where a kernel actually spends time, instead of guessing.
when_to_use: You are the bench engineer on a GPU team and need to explain or improve a kernel's runtime.
tags: [gpu, profiling]
---

# Measure the machine, not your model of it

GPU intuition is unusually unreliable: the bottleneck is far more often memory
movement or occupancy than arithmetic. Profile before changing anything.

## Order of questions

1. **Is the GPU busy at all?** Kernel time versus wall time. A "slow kernel"
   that occupies 8% of wall time is a host-side or transfer problem, and no
   amount of kernel tuning will show up.
2. **Memory or compute bound?** Achieved bandwidth against the device peak, and
   achieved FLOPs against peak. See `roofline-model` — the roofline tells you
   which ceiling you are under, and therefore which optimisations can possibly
   help.
3. **Occupancy?** Only after 1 and 2. Occupancy is a means, not a goal: a
   kernel at 40% occupancy saturating bandwidth is finished, and raising
   occupancy will not make it faster.

## The tools

- **Nsight Compute** (`ncu`) — per-kernel counters. Start with
  `--set full` on ONE kernel invocation, not the whole run; it serialises and
  replays kernels, so a full-application profile takes minutes and changes the
  timing you were trying to measure.
- **Nsight Systems** (`nsys`) — the timeline. This is where question 1 is
  answered: gaps between kernels, host-device copies, stream serialisation.
  Use it *first*; `ncu` optimises a kernel that `nsys` may show is irrelevant.
- **rocprof** — the ROCm equivalent. `--stats` for the summary,
  `--hip-trace`/`--hsa-trace` for the timeline.
- **Metal frame capture** — Xcode's GPU capture. Per-encoder timings and the
  shader profiler's per-line cost. It is a *frame* capture: for compute work,
  bracket the dispatch in a capture scope explicitly or you get nothing.

## Warm up, and say what you measured

First-call timings include JIT compilation, allocator growth and page faults —
routinely 10-100× the steady state. Discard warmup iterations and report a
distribution, not a single number: a median with a spread tells the reader
whether the change is real. See `criterion-benchmarking` for the statistics.

Record the device, driver version and clock state alongside the number. GPUs
throttle; a measurement without its conditions cannot be compared to the one
you take next month.