Welcome

The roofline.dev learn tab is a human written learning resource for everything GPU / AI accelerator programming. It is designed to be jumped around, feel free to skip to parts that interest you. We aim to create an engaging, interactive learning experience, something a university won't give you. Our primary goal is to teach you as much information as possible in as little time as possible, leveraging intrinsic variables like fun and interactivity to aid in the learning process.

This will be mostly focused on CUDA, but the same principles are largely applicable to ROCM and some chinese GPUs.

Introduction to GPUs

Here you will learn all about computing on the gpu. Intuitively, a GPU is a cluster of small, less efficient CPUs called threads stacked into a long array. What these small CPUs allow is parallelism, the ability to compute multiple instructions at the same time. Below is a demonstration of the computation of the Mandelbrot's set.

Both sides compute the same fractal: z = z² + c iterated up to 60 times per pixel. Each tick, the CPU finishes one pixel; the GPU advances every pixel by one iteration. Same tick rate, vastly different throughput.

The GPU was able to compute 60 passes in a fraction of the time that it took a CPU to compute one pass, how is that possible? The GPUs are made up of abstractions which we will digest incrementally in order not to overwhelm ourselves.

The Programming model

The programming model is an abstraction over the aforementioned threads. Intuitively, what you do is define some chunk of code and then set it to execute accross all threads. Use the panel below to try out and get a feel for how it works.

kernel(x, y)x, y ∈ 0..47

return;

invalid — only x, y, numbers, + - * / % ** ^ | &, sin cos sqrt abs min max PI, etc.

One expression, evaluated independently for every pixel. Hover any cell to see exactly what was computed there. That's the kernel — write the math for a single thread, the GPU runs it across every one.

This is a very simplified version of how the CUDA programming model works in reality, but it gives you a good intuitive understanding for how and what it can be used for. Apart from CUDA, there are simplified frameworks like GLSL, which works exactly by this principle. GLSL is something called a shading language which is typically used for, among other things, image processing.

Now that you're in the right headspace, we can start learning how GPUs work internally in the next section.