New to this topic? That is completely fine. It is completely fine if you have never studied some of these topics before. We want to see how fast you can learn something new, grasp it, and teach it from first principles.
This page is a quick primer, not a script. Use it to get your bearings, then read the links below and anything else you find: other blogs, papers, videos, code. We are judging the story you build, your examples and metaphors, your first-principles thinking, and any experiments you run yourself.
A quick primer
A GPU has two hard limits. It can only do so many calculations per second, and it can only move so many bytes from memory per second. Every program you run is held back by one of the two. The roofline plot shows which one, at a glance.
The two limits (NVIDIA H100 SXM, dense BF16)
- Peak compute: about 989 TFLOP/s (trillion floating-point operations per second).
- Memory bandwidth: about 3.35 TB/s from HBM3 memory.
Both are fixed by the hardware. What you control is the arithmetic intensity: how much useful maths you do for every byte you move.
The ridge point is where the two limits meet: FLOP per byte. A kernel needs roughly 295 operations for every byte it reads to keep the GPU fully busy.
| Operation (BF16, 2 bytes per number) | FLOPs | Bytes moved | Intensity | Verdict |
|---|---|---|---|---|
| Add two vectors, | 1 per element | 6 per element | ≈ 0.17 | Memory-bound. About 0.56 TFLOP/s: under 0.1% of peak. |
| LLM decode, batch 1 (matrix × vector) | ≈ (the weights) | ≈ 1 | Memory-bound. Why generating tokens one at a time is slow. | |
| Decode with batch | ≈ | ≈ | Batching raises intensity: the weights are read once for users. | |
| Square matmul, | Compute-bound. This is what GPUs are built for. |
Questions worth exploring
- Why is the plot on log-log axes, and why does the memory roof appear as a straight diagonal line?
- Why does operator fusion (as in FlashAttention) help so much?
- Why is prefill compute-bound and decode memory-bound in LLM inference?
- Can you measure it? Time a matmul at growing sizes on a free Colab GPU and plot your own roofline.
Read
- Roofline: An Insightful Visual Performance Model · Williams, Waterman and Patterson (UC Berkeley)The original roofline paper.
- Making Deep Learning Go Brrrr From First Principles · Horace HeCompute-bound, memory-bound and overhead-bound, explained beautifully.
- GPU Performance Background User's Guide · NVIDIAWorked arithmetic-intensity examples for real layers.
- Roofline Model (GPU Glossary) · ModalA crisp definition with diagrams.
- All About Rooflines · How To Scale Your Model (Google DeepMind)Rooflines applied to matmuls and LLMs, with exercises.
Watch
- ▶ A very short intro to the Roofline model · NHR@FAU
- ▶ Lecture 8: CUDA Performance Checklist · GPU MODE
- ▶ LLM Inference: Roofline Analysis for GPU · Faradawn Yang
Go beyond this list. These are starting points. The best submissions usually bring in something we did not link: an experiment, a paper, a story of their own.