LLM Inference Parallelism: A Salad of Acronyms

If you’ve ever tried to serve an LLM and immediately thought “why the hell is this so much harder than training?”, welcome to the club. Training is just math. Inference is systems hell: VRAM that fills up in weird ways, interconnect that decides whether your GPUs actually work together, and the brutal fact that one “simple” chat endpoint is really two completely different jobs smashed together.

This post isn’t a deep dive into any single framework. It’s the mental model I wish I’d had the first time I tried to put a 70B model into production. We’re going to walk through the four parallelism strategies you’ll hear about constantly:

• DP — Data Parallelism
• TP — Tensor Parallelism
• PP — Pipeline Parallelism
• EP — Expert Parallelism (MoE only)

A Salad of Acronyms (image generated with Nano Banna)

The three bills you always pay

Serving LLMs comes down to three costs that never go away:

• Weights - the static memory tax. They have to live in VRAM.
• KV cache - the dynamic memory tax. Grows with context length and how many requests you’re handling at once.
• Communication - the hidden tax. Every time you split the model, something has to move between GPUs (or nodes).

KV cache is usually the one that quietly kills you. Double the context or the concurrency and your memory footprint doesn’t just grow — it explodes. If you don’t account for this, you’ll hit OOM (Out of Memory) errors long before your GPUs hit 100% utilization.

If you’re new to KV cache: don’t worry about exact formulas yet. The only thing you need right now is the scaling behavior:

• More context length → larger KV cache
• More concurrent sequences → KV cache multiplies
• More layers / heads / hidden size → KV cache per token increases

So the moment you move from “one request” to “a few users at once,” memory pressure changes dramatically.

Before we move to the main topic, let’s understand a key point in LLM Inference.

Inference is two different workloads: prefill and decode

Inference isn’t one workload. It’s two, and they hate each other:

• Prefill — you dump the entire prompt in at once. Compute-heavy, loves big batches, but still contributes to TTFT.
• Decode — you generate one token at a time. It’s memory-bandwidth bound and extremely sensitive to any communication overhead, and users feel every millisecond. Because the per-token compute is small, synchronization/collectives become a much larger fraction of total time.

That split matters because some parallelism strategies are great for throughput (prefill) and feel worse for interactive latency (decode), or vice versa. You can even see it in how we measure success:

• Prefill -> TTFT (Time to First Token)
• Decode -> TPOT / ITL (Time per Output Token / Inter Token Latency)

For now, my goal here is to get you acquainted with the formal terms. In later posts we’ll dig into why you can’t really optimize inference if you treat it as one monolithic task — and why production systems often end up treating prefill and decode as separate problems.

The Acronyms

Data Parallelism (DP): the simplest win

DP is simple: you run identical replicas of the model on different GPUs. GPU A handles Request 1; GPU B handles Request 2.

• What moves: Nothing between GPUs during the forward pass.
• The win: Perfect for high QPS (Queries Per Second) and “agentic” workloads where you have many small, independent requests.
• The catch: If the model doesn’t fit on one GPU (alongside its KV cache), DP alone won’t save you.

If your model + KV cache fits on a single card, start with DP. It has the lowest complexity and zero communication overhead.

Tensor Parallelism (TP): make the big model fit, but pay communication constantly

TP is the nuclear option for when a model is too big for one card. It shards individual weight matrices across multiple GPUs. Every GPU computes a piece of the math for every layer.

• What moves: Partial results (activations). GPUs must sync via frequent collectives (e.g., all-reduce / all-gather) throughout the forward pass.
• The win: It lowers per-GPU memory usage and can actually reduce latency for single requests because multiple GPUs are crunching the same math.
• The catch: TP is hungry for bandwidth. If you don’t have NVLink (fast GPU-to-GPU interconnect on the same node), the time spent “talking” will often exceed the time spent “calculating.” (PCIe bottleneck)

Pipeline Parallelism (PP): fewer collectives, but now you have pipeline bubbles

PP split the model by layers. GPU 0 handles layers 1–20, GPU 1 handles 21–40, and so on.

• What moves: Activations at the “stage boundaries” (where one GPU’s chunk ends and the next begins).
• The win: Much lower communication frequency than TP. It’s often the only way to scale across nodes that lack high-speed interconnects.
• The catch: “Pipeline bubbles.” While GPU 3 is working on the end of the request, GPUs 0, 1, and 2 might be sitting idle unless you have enough concurrent requests (micro-batching) to keep the pipe full.

PP can shine on throughput when you have enough traffic, but it can feel worse for single-user interactive latency.

Expert Parallelism (EP): MoE changes the game

EP is a specialized strategy for Mixture-of-Experts models (like DeepSeek R1 or GPT OSS). In an MoE model, only a fraction of the “experts” are active for any single token. Thus, reducing the active parameters per token.

• What moves: Token activations are routed to the specific GPU that holds the “expert” weights required for that token.
• The logic: It’s cheaper to move a small token activation to a GPU than it is to shard a massive “expert” matrix across the whole cluster.
• The win: It allows you to run trillion-parameter models that wouldn’t otherwise fit, provided you have enough traffic to keep the experts busy.

On small GPU counts or low traffic, EP can be underwhelming or even worse than a simpler strategy.

Note: DP/EP/TP/PP are not mutually exclusive. In practice you combine them: e.g., TP within a node, PP across nodes, and DP across replicas.

How to actually pick one

I sort workloads into three rough buckets. They’re not scientific but they can save you a lot of guessing.

Profile A: The Interactive Chatbot (Decode-Heavy)

• Focus: TTFT (interactivity) and TPOT (token smoothness). Users hate stuttering text.
• Strategy: If the model fits, use DP. If it doesn’t, use TP (if you have NVLink) to keep latency low. Avoid high PP degrees as they can add “lag” to the start of the response.

Profile B: The RAG Pipeline (Prefill-Heavy)

• Focus: Throughput and long context handling.
• Strategy: PP is often great here because you can batch many long-context requests together to fill the pipeline. You’ll need aggressive KV cache management (like PagedAttention) to keep the prefill from OOMing.

Profile C: Agentic Loops (High QPS, Small Requests)

• Focus: Cost per request and total QPS (tool loops cause bursts of requests).
• Strategy: Stick to DP replicas. You want to avoid the coordination overhead of TP/PP for tiny requests.

The decision tree (at least for starters)

1. Does the model + KV cache for my target context and concurrency fit on one GPU? -> Yes: DP replicas. Stop over-engineering.
2. Sharding time. Are your GPUs on different nodes or connected by PCIe only? Is interconnect your bottleneck? (GPUs idle, NCCL time high) - -> Lean PP. Otherwise TP is usually the first thing to try.
3. Is it MoE and you’re at real scale? -> Layer EP on top of your TP/PP baseline. EP is an optimization lever, not a first-step requirement.

The traps that eat weeks

• Treating “tokens/sec” as a single number. You can have great aggregate throughput and still have users waiting 4 seconds between tokens. Always look at TTFT, TPOT, and p95/p99 under load.
• Forgetting KV cache until it OOMs. Context × concurrency is the real limit, not the weights.
• Benchmarking with one prompt. Do at least short-prompt/long-decode, long-prompt/short-decode, and high-concurrency short requests.
• Assuming more GPUs = faster. Parallelism is not free, sometimes two 4-GPU TP groups beat one 8-GPU group because of less communication.

Quick note on frameworks

Don’t assume DP/TP/PP/EP mean identical things everywhere. Though vLLM, SGLang, TensorRT-LLM, etc. all implement these ideas slightly differently (especially around MoE and how they group DP). The Schema still holds: know what’s moving (weights, activations, tokens) and how often.

The three things that actually matter

1. Inference is two workloads. Design for both.
2. Parallelism is about what moves and how often.
3. Start simple. DP if it fits. TP or PP if it doesn’t. EP only when you’re big and MoE.

What’s next in this series (hopefully)

In the next posts I want to go deeper into what really drives cost and performance in production:

• Prefill vs Decode: why they behave differently and how to benchmark each
• KV cache: why it grows, why it kills you, and what to do about it
• Prefill/Decode disaggregation: when splitting them helps (and when it doesn’t)
• Prefix caching: KV reuse, KV transfer, and KV offload (and why “cache” doesn’t always mean “faster”)

If you’re building an inference service today, you don’t need all the tricks at once. But you do need the map, otherwise every optimization looks like a random flag.