From Threads to Tensor Cores: Understanding How GPUs Really Work

NVIDIA GH100 GPU Block Diagram

If the diagram above feels confusing, you’re in good company. When I first started looking into GPU architecture, I found myself asking the same questions: What exactly is a GPU? How is it different from a CPU? And why is it so effective at powering AI models and rendering modern games?

GPU architecture can seem intimidating at first because there are many moving parts and layers of abstraction. Over the past few days, I’ve been trying to develop a deeper understanding of how these systems work, at first I built this visualiser https://staru09.github.io/gpu_viz/ and now this blog is the result of that journey. Think of it as a gentle introduction rather than a comprehensive guide.

While it’s tempting to think of a GPU as simply a chip with thousands of cores, the real story is much more interesting. The performance of a GPU comes from how those cores are organized, how memory is structured, and how thousands of lightweight threads coordinate and execute work in parallel. Understanding these ideas is what makes GPU architecture fascinating. So, let’s break it down step by step.

Think of a GPU as a massive factory. A CPU resembles a factory staffed by a small number of highly skilled workers, each capable of handling complex, multi-step tasks independently. A GPU, in contrast, is like a factory with thousands of specialized workers performing similar operations in parallel on different pieces of data. Neither architecture is inherently better than the other they are optimized for different types of workloads.

As you begin exploring GPU architecture, you’ll encounter some familiar concepts from operating systems, such as threads, alongside GPU-specific abstractions like warps, thread blocks, grids, and streaming multiprocessors. Understanding how these components interact is key to understanding how GPUs achieve their massive parallel processing capabilities.

A modern GPU contains many interconnected components, and covering every detail would quickly turn this into a very long discussion. Instead, this blog focuses on the fundamental architectural concepts that provide the strongest foundation for understanding how GPUs work. The goal is to keep the explanation concise while covering the most important ideas. For readers interested in a deeper dive, I’ll include an appendix and a collection of recommended resources at the end of the blog that explore GPU architecture in much greater detail.

The key to understanding a GPU is understanding its three major planes i.e. Thread Hierarchy, Memory Hierarchy and Compute Units.

Thread Hierarchy for a GPU

A) The Thread Hierarchy :- It organizes the execution of parallel programs across multiple levels, from individual threads up to entire GPU devices. We’ll focus on these 4 as of now Threads, Warps, Thread Blocks and Grids.

When you launch a program on a GPU, you don’t just run one task. You run millions of tiny tasks called threads, all at once. But you can’t just throw millions of threads at a GPU and hope for the best they need to be organized. That’s where the thread hierarchy comes in.

Threads A thread is the most basic unit of work on a GPU. Each thread runs the same code but operates on different pieces of data. Imagine you need to multiply every pixel in an image by 2 instead of doing it one pixel at a time, you launch one thread per pixel, and they all run simultaneously. Each thread gets its own private set of registers (tiny, ultra-fast storage) to work with.

Warps GPUs don’t schedule individual threads instead, they group 32 threads into a warp. The hardware issues one instruction at a time to the entire warp, and all active threads execute it together. This execution model is called SIMT (Single Instruction, Multiple Threads). On older architectures (pre-Volta), all 32 threads in a warp shared a single program counter and executed the same instruction in strict lockstep. Since Volta (2017), NVIDIA introduced independent thread scheduling, giving each thread its own program counter meaning threads in a warp can technically be at different points in the program and the hardware reconverges them.

Thread Blocks Multiple warps are grouped into a thread block (also called a Cooperative Thread Array, or CTA). A block can contain up to 1,024 threads. All the threads in a block are guaranteed to land on the same hardware unit (called an SM more on that shortly), which means they can: Share a fast, programmer-controlled memory region called shared memory Synchronize with each other using __syncthreads(). Threads in different blocks, however, can’t easily talk to each other. They run independently, which is actually a feature it means the hardware can schedule them in any order without worrying about dependencies.

Grids The outermost level is the grid the entire collection of thread blocks that make up one kernel launch. A grid can be 1D, 2D, or 3D, which makes it natural to map onto data that is inherently multi-dimensional, like images (2D) or volumes (3D). The GPU’s work distributor (called the **GigaThread Engine** in NVIDIA’s architecture) is responsible for handing blocks out to the available hardware as it becomes free.

B) The Memory Hierarchy:— Memory is where GPU performance is most often won or lost. The GPU’s memory system is structured as a hierarchy faster memories are smaller and closer to the compute units, while slower memories are larger and further away. Latency here is measured in clock cycles the number of ticks of the GPU’s internal oscillator that pass from the moment a memory request is issued to the moment the data arrives. At 1.5 GHz, one cycle is about 0.67 nanoseconds.

Memory Hierarchy for a GPU

Registers The fastest memory on the GPU. Each thread gets its own private registers, allocated from the SM’s register file. They’re so fast that accessing them takes just a single clock cycle. The downside is they’re limited — if a thread uses too many, the excess “spills” into local memory, which is drastically slower.

Local Memory Despite the friendly name, local memory is not fast. It’s private to each thread, but it physically resides in the GPU’s main DRAM. It’s used as an overflow when registers run out, and it’s painfully slow. Good GPU code tries hard to avoid local memory spills.

Shared Memory / L1 Cache This is the workhorse of GPU optimization. Shared memory is an on-chip, programmer controlled scratchpad that all threads in a block can read from and write to. It’s about 20 30× faster than global memory, and smart use of it loading data into shared memory once and reusing it many times is the basis of most GPU optimization techniques like tiling. On modern NVIDIA GPUs (Ampere and later), shared memory and the L1 cache share the same physical SRAM on the SM, and the split between them can be tuned by the programmer.

L2 Cache The L2 cache sits outside the SMs but still on the chip. It’s shared by all SMs and serves as a buffer between the fast on-chip world and the slow off-chip DRAM. The H100 has a 40 MB L2, which is large enough to hold entire small models or frequently reused weight matrices.

Global Memory / HBM This is the main event the GPU’s VRAM. It’s what you’re allocating when you call cudaMalloc . It’s large (up to 192 GB on the H100), and modern GPUs use High Bandwidth Memory (HBM), which stacks DRAM dies vertically to achieve enormous bandwidth up to 3.35 TB/s on the H100. But bandwidth and latency are different things. Even with all that bandwidth, every individual memory access still takes 400–800 cycles. This is why the warp scheduler exists while one warp is waiting on a memory fetch, the scheduler switches to another warp that’s ready. With enough warps in flight, the latency gets hidden behind useful work. This is called latency hiding through occupancy, and it’s the fundamental execution model of the GPU.

Host Memory / PCIe The CPU’s system RAM, accessed over the PCIe bus (or NVLink for high-end configurations). This is by far the slowest link in the chain. Minimizing data transfers between CPU and GPU is one of the first things any GPU programmer learns to do.

C) The Compute Units :— The Machines Inside the Factory All computation on a GPU happens inside units called Streaming Multiprocessors (SMs). The SM is the fundamental processing block of an NVIDIA GPU every other compute unit lives inside it. A high-end GPU like the NVIDIA H100 has 132 SMs. Scaling a GPU up largely means adding more SMs. Inside each SM, you’ll find several types of compute units, each designed for a specific kind of work:

Inside Streaming Multiprocessor (SM)

CUDA Cores These are the most abundant units the general-purpose workhorses. Each CUDA core handles single-precision (FP32) floating point and integer arithmetic. A typical SM on an Ampere GPU has 128 CUDA cores, and an H100 has 16,896 in total. Simple, fast, and plentiful.

FP64 Cores For scientific computing, simulations, and numerical methods that require higher precision, GPUs include double-precision (FP64) cores. Data center GPUs (like the H100) have a healthy number of these. Consumer gaming GPUs, however, deliberately cripple FP64 throughput — sometimes to just 1/64th of FP32 to differentiate from the more expensive professional hardware.

Tensor Cores Tensor Cores are specialized units that perform matrix multiply-accumulate operations on entire tiles of matrices in a single instruction, rather than element by element. This is exactly the kind of math that dominates deep learning (think matrix multiplications in transformer attention layers). Each generation of Tensor Cores has gotten more capable, adding support for lower precision formats like BF16, TF32, and FP8. That’s the reason GPUs have become the engine of the AI revolution.

SFUs (Special Function Units) Not all math is simple multiplication. Functions like sin , cos , log , exp , and sqrt require dedicated hardware. That’s what SFUs are for. There are far fewer of them than CUDA cores (typically 1 SFU for every 8 CUDA cores), so if your kernel is heavy on transcendental functions, SFUs can become a bottleneck.

LSUs (Load/Store Units) Every time a thread needs to read from or write to memory, it goes through an LSU. These units handle the traffic between the registers (inside the SM) and the wider memory hierarchy. Again, there are fewer of them than CUDA cores roughly 1 per 8 cores which is why memory-bound kernels can stall even when the arithmetic units are sitting idle.

Warp Schedulers Finally, each SM contains 4 warp schedulers. These aren’t arithmetic units, but they’re the most important piece of the SM. Their job is to constantly look at all the warps assigned to the SM and pick the ones that are ready to execute. This is the mechanism that enables latency hiding, which brings us neatly to memory.

A skeleton diagram for GPU architecture

The beauty of GPU architecture lies in how its components work together as a cohesive system. The thread hierarchy organizes and distributes work efficiently, allowing massive numbers of threads to execute concurrently. The memory hierarchy ensures that data remains as close to the compute units as possible, minimizing costly memory accesses. Meanwhile, the compute units from general-purpose CUDA cores to specialized Tensor Cores are designed to handle different types of workloads with maximum efficiency.

tldr; It’s not simply about having thousands of cores; it’s about coordinating vast numbers of threads, managing data across multiple levels of memory, and taking advantage of specialized hardware designed for parallel computation all working together to deliver extraordinary performance.

Appendix:- 1. https://modal.com/gpu-glossary 2. https://docs.nvidia.com/cuda/cuda-programming-guide 3. https://cvw.cac.cornell.edu/gpu-architecture 4. https://www.educate.elsevier.com/book/details/9780443439001 5. https://www.gpumode.com/