🔥 Things to Know about Parallelizing Large Models

Photo by Rahul Mishra on Unsplash

In the last few years, we have seen a trend in which GPUs have gotten bigger, along with the models that run on them, and we are not seeing any signs of reversal.

For Non-Members: Read here!

A Challenge With Bigger Models

A billion-parameter model with 8-bit precision takes up 1 GB of VRAM on a GPU. Now think about DeepSeek-V3.1 with 671 billion parameters. Loading this model on a single B200 GPU would throw an OOM (out-of-memory) error.

You can theoretically load the deepseek-v3.1 model on 4 B200s, which in total has a capacity of 720 GB of VRAM. We are just squeezing the model here because there is no room left for KV cache, which often takes up 80 percent or more of the remaining VRAM after weights. To serve the DeepSeek model to real traffic, we must have at least 8 B200s.

How to estimate the GPU requirement?

You can estimate the minimum number of GPUs required for a model by multiplying the precision, parameter count, and expected KV cache allocation together.

bits_precision = 8
params = 671 #parameter in billions
kv_cache_allocation = 1.8

vram_required = (bits_precision / 8) * params * kv_cache_allocation

#deepseek requires 1200 GB of VRAM with this config

b200_ram_available = [180, 360, 720, 1440)

round_up(vram_required, b200_ram_available)

#serving deepseek in fp8 requires 1440 GB of VRAM (8 x b200s)

However, all of this requires that inference scales efficiently from one GPU to multiple GPUs. The limitation in scaling parallel inference is the communication overhead between the GPUs.

To overcome the communication overhead, the GPUs are interconnected using NVLink and NVSwitch within the nodes, and InfiniBand between nodes.

Though these connectors (NVLink and others) offer high bandwidth, they are a fraction of the speed of VRAM.

With LLM decoding bounded to memory bandwidth, multi-GPU inference needs to be carefully designed to avoid bottlenecks in inter-GPU communication. This field of study is called topology-aware parallelism.

How Does Model Parallelism Tackle the Bigger Model’s Challenge

During inference, we can parallelize the model in three forms:

• Pipeline Parallelism (PP): It splits the layers of the model across GPUs.
• Tensor Parallelism (TP): It splits the tensors within each layer across GPUs.
• Expert Parallelism (EP): It shards the entire experts from MoE models across different GPUs.

Each form of parallelism has its own tradeoffs:

Tensor Parallelism is generally best for low-latency model inference within a single node, while the Expert Parallelism improves throughput for MoE LLMs. Pipeline Parallelism is only used for multi-node inference.

Additionally, data parallelism strategies like Context Parallelism splits the computation across devices. These strategies are rare in LLM inference but essential for video generation.

Tensor Parallelism for Latency

This should be our default strategy for multi-GPU model inference. It supports both the dense models like LLama 405B, and MoE models, which is currently dominating the open model space.

It works by splitting apart each layer of the model (as opposed to Pipeline Parallelism, which keeps layers intact) and distributing the layer fragments across the allocated GPUs.

For each layer, the expense of reading from weights memory and executing matrix multiplication is shared across the GPUs.

However, the results of each layer need to be communicated in an all-reduce fashion into a single output before the next layer can be computed. In nodes with high-bandwidth intra-node NVLink and NVSwitch, this communication overhead is minimized.

Increasing Tensor Parallelism improves TPS on a per-user basis (assuming the model is large enough and the sequences are long enough that the communication overhead doesn’t outweigh the faster forward pass, which is the case for most frontier models).

Expert Parallelism for Higher Throughput

It is a specialized model-parallel strategy designed for Mixture-of-Experts arhitecture. Each expert layer is sharded and distributed across multiple-GPUs rather than replicating it on each GPU.

It neatly divides experts across GPUs. In a model with 128 experts served in EP8 across eight GPUs, each GPU will host 16 full experts.

EP improves total system throughput, making inference more scalable and less expensive. With individual experts processing tokens separately, each token takes just as long, but the system as a whole can handle more simultaneous tokens.

Many deployments use a mix of TP and EP to achieve both the benefits.

Expert Parallelism requires less inter-GPU communication than Tensor Parallelism. The Expert Router, which determines which experts each token activates, is replicated onto each GPU as it is a relatively small component of the model.

Inter-GPU communication is necessary for passing tokens from expert to expert, but unlike TP, it is not required to collect the results of each layer.

Thanks to this lower communication overhead, EP scales well to multi-node deployments and systems with limited interconnect bandwidth.

Pipeline Parallelism

Pipeline parallelism divides the neural network layers across multiple GPUs, enabling simultaneous computation and memory reuse. This technique contrasts sharply with sequential processing, where each GPU waits for the previous to finish before starting its task.

It reduces idle time between computation and communication steps, directly improving resource utilization.

For example, DeepSpeed’s implementation shows that pipeline parallelism enhances both memory efficiency and compute efficiency, enabling models to train faster without requiring proportional increases in hardware costs.

Thanks for reading

Digital Products

ML Interview Book: Crack Your Next ML Interview with Machine Learning Interview Playbook

Productivity Tool: ***Social Media Time Tracker: Take Back Your Time, a tool that annoys you when you log in to social media sites. Chrome Extension.***

Connect with the author

LinkedIn | YouTube | Threads | Instagram | Facebook

Reference

Inference Engineering by Philip Kiely