The SIMT model

SIMT stands for Single Insutrction Multiple Threads, to understand how it works works we need to dive into the internals of GPUs. The threads of a GPU are segmented into hierarchies, where each point of the hierarchy has unique features and limitations.

Threads are the individual computing units of GPUs. They are executed in parallel with typically the same instructions. A common confusion arises when thinking about how multiple threads can be helpful when they're executing the same instruction. That is trivially explained, every thread has its own id, which it knows and uses to offset the memory it is accessing.

kernel(tid)step 0

memory[tid] ←

x = memory[tid], tid= this thread's ID.

invalid — only x, tid, numbers, + - * / % **, sin cos sqrt abs min max PI, etc.

t10

t11

t12

t13

t14

t15

1010

1111

1212

1313

1414

1515

every thread · one tick

Same kernel runs on every thread. Each one uses its tid to write into its own slot of memory[]. Hover any thread or memory cell to trace the pair and see what that thread computes.

Threads are organized into chunks of 32 called warps, with each of them exeucting the same instruction. More often than not, you would want multiple warps executing the same instructions, so the 32 thread warp chunk is effectively a limitation, meaning if you want to do work that requires less than 32 threads you would have some of those warps inactive, unable to execute anything else.

Blocks

The abstraction that sits above warps is called blocks. Blocks are a software abstraction that represent a grid of warps. As mentioned earlier, they have unique features separate from warps.