Bandwidth, Not Compute: What Actually Limits LLM Inference Speed

TL;DR: Prefill is compute-friendly. Decode is memory-bound. Nearly everything about GPU selection, latency optimization, and throughput engineering follows from that. This matters more than ever for agentic workloads, where a single user request can trigger dozens of LLM calls chained together, and latency compounds at every step.

This post goes from first principles to a real benchmark comparing the RTX Pro 6000 Blackwell (96 GB GDDR7, 1.792 TB/s) against the H100 SXM (80 GB HBM3, 3.35 TB/s) on GPT-OSS 120B.

The Short Version

Before going deep, here is the decision frame everything builds toward:

If your workload looks like this

Use that table as a first pass. The rest of this article is about why those mappings hold up once you look at arithmetic intensity, KV-cache growth, and GPU architecture.

The Two Phases of LLM Inference

Every LLM inference request goes through exactly two stages.

Prefill

When you send a prompt, the model processes all tokens in parallel through the full forward pass. For a 512-token prompt across 80 transformer layers, that means 80 attention operations, each attending to all 512 tokens simultaneously. Huge matrix multiplications. Dense arithmetic. The GPU is doing enormous amounts of computation per byte moved.

The output of prefill is two things:

1. The KV cache — key and value tensors for every token, every layer, cached for the decode loop
2. The first token logits — the probability distribution from which the first output token is sampled

Prefill latency maps directly to TTFT (Time to First Token).

Decode

Now the model generates output one token at a time. At each step:

• Only the single newest token has its Q, K, V computed
• The new K and V vectors are appended to the KV cache
• The new Q attends against all cached K vectors — which means reading the entire KV cache from VRAM for every single token

This is the fundamental problem. To generate one token, the GPU must:

• Read the full model weights (~once)
• Read the full KV cache (~once)
• Do a tiny amount of actual computation

The ratio of memory reads to useful FLOPs is extremely low. This is what “memory-bound” means in practice.

Decode latency per token maps to TPOT (Time Per Output Token) and ITL (Inter-Token Latency).

End-to-End Latency Math

E2E latency = TTFT + (N_output_tokens × TPOT)

where:
TTFT ≈ f(prompt_length, model_size, TFLOPS)
TPOT ≈ f(model_size, KV_cache_size, bandwidth)

For a 512-token prompt generating 256 tokens on a mid-range GPU, TTFT might be 200ms and TPOT might be 50ms — meaning the total wait is 200ms + (256 × 50ms) = 13 seconds. TPOT dominates almost every real user interaction.

This compounds badly in agentic systems. A single agent task — say, a ReAct loop that calls tools, synthesizes results, and reasons over them — might involve 10–20 sequential LLM calls. If each call costs 2 seconds end-to-end, the full task takes 20–40 seconds. Every millisecond you shave off TPOT multiplies across every step in the chain. This is why inference latency is not just a UX concern anymore. It is the primary constraint on how fast an agent can operate.

The Roofline Model: Why This Matters

The Roofline model is the right mental model for understanding GPU-bound inference. It defines two performance ceilings for any given operation:

Compute ceiling:   Peak tensor throughput (e.g. ~989 dense BF16 tensor 
TFLOPS on H100 SXM)

Bandwidth ceiling: Peak bandwidth × arithmetic intensity

Arithmetic intensity is the ratio of FLOPs to bytes moved:

Arithmetic Intensity (AI) = FLOPs / Bytes

That number tells you how much compute you get out of each byte you move through memory. If the arithmetic intensity is too low, the GPU spends its time waiting for data. If it is high enough, the GPU can stay busy doing math.

Every GPU has a ridge point:

• below it, the workload is memory bound
• above it, the workload is compute bound

For a matrix multiply of weight matrix W with input X:

FLOPs = 2 × batch × d_in × d_out
Bytes = d_in × d_out × dtype_bytes (just reading the weights,dominant term)
AI    = 2 × batch / dtype_bytes

At BF16 (2 bytes): AI = batch

For batch = 1, BF16 decode has arithmetic intensity around:

AI ≈ 1 FLOP / byte

That is tiny. On modern GPUs, the ridge point is nowhere near 1 FLOP/byte. So decode at small batch sizes lives far below the ridge point, which means it is overwhelmingly memory bound.

Using a simplified dense-BF16 roofline approximation, the H100 SXM has a ridge point of roughly:

Ridge point = Peak TFLOPS / Peak bandwidth
            = 989 × 10^12 / (3.35 × 10^12)
            ≈ 295 (at BF16)

Meaning: under this simplified model, you need an effective batch around 295 before this approximation stops looking clearly bandwidth-dominated. At batch = 1, decode sits deep in the memory-bound regime.

This is why decode is memory-bound. Not as a vague intuition, but as a mathematical consequence of autoregressive generation. If the workload cannot feed the arithmetic units, the card with the bigger compute ceiling still spends its time waiting on memory.

Quantization shifts the arithmetic intensity

This is one of the cleanest ways to think about quantization. If you reduce weight precision, you reduce bytes moved per parameter.

• BF16: 2 bytes
• FP8: 1 byte
• INT4/FP4: 0.5 bytes in idealized weight-only terms

So the same approximation becomes:

AI ≈ 2 × B / dtype_bytes

Which means:

• BF16: AI ≈ B
• FP8: AI ≈ 2B
• INT4 / FP4: AI ≈ 4B

FP8 doubles arithmetic intensity. At batch=1, you go from AI=1 to AI=2 — still memory-bound, but the effective ridge point is now half of what it was.

INT4 quantization: AI ≈ 4 × batch. Under the same H100 SXM approximation, the ridge point drops from about 295 to about 74.

This is why quantization improves decode throughput beyond just “smaller weights fit in VRAM” — it structurally changes where you sit on the roofline.

Prefill is different

During prefill, the input X has shape (seq_len, d_model) — potentially thousands of tokens. If seq_len=2048:

AI ≈ 2 × 2048 / 2 = 2048 (BF16)

Well above the ridge point. Prefill is compute-bound on data-center GPUs.

Memory Breakdown: Training vs Inference

Understanding where VRAM goes is essential before talking GPUs.

Training (per parameter at BF16 with AdamW)

A 7B model needs about 56 GB for Adam optimizer state alone (m + v in fp32), or about 84 GB if you include BF16 weights and gradients before counting activations, temporary buffers, or fragmentation. That is why training a 70B model on a single H100 is impossible without offloading, activation checkpointing, ZeRO-style partitioning, and aggressive memory optimization.

Inference (per parameter at BF16)

Inference is far leaner. The KV cache becomes the variable cost. It gets brutal once context length and concurrency grow.

KV Cache Formula

For standard Multi-Head Attention (MHA):

KV_bytes = 2 × n_layers × n_heads × head_dim × seq_len × batch_size × dtype_bytes

The factor of 2 is for K and V. For Grouped Query Attention (GQA), replace n_heads with n_kv_heads:

KV cache bytes(GQA)
= batch × 2 × n_layers × n_kv_heads × head_dim × seq_len × dtype_bytes

Let’s work this out using a concrete example to build intuition. I will use gpt-oss-120b, because it is one of my favorite open-weight models.

GPT-OSS 120B (MoE, 5.1B active params/token):

From the model architecture:

• n_layers=36
• attention_heads=64
• n_kv_heads=8 (GQA)
• head_dim=64
• dtype=BF16 (2 bytes)
• context length up to 128k.

So at BF16:

per-token KV bytes
= 2 × 36 × 8 × 64 × 2
= 73,728 bytes
≈ 72 KiB per token per sequence

That means:

• 8k tokens → about 576 MiB per sequence
• 32k tokens → about 2.25 GiB per sequence
• 128k tokens → about 9.0 GiB per sequence

Now multiply by concurrent requests:

• 16 concurrent requests at 8k context → about 9 GiB
• 32 concurrent requests at 32k context → about 72 GiB
• 16 concurrent requests at 128k context → about 144 GiB

That is why long-context serving gets ugly so fast. The cache becomes the pressure point.

Agentic systems make this significantly worse. A tool-calling agent typically passes the full conversation history back as context on every call. An agent running 10 steps with 500-token tool responses per step accumulates 5,000+ tokens of context before it finishes.

KV cache dtype changes the math fast

Because KV cache size scales linearly with dtype_bytes, dropping precision buys space immediately:

• BF16 / FP16: baseline
• INT8 KV cache: ~2x smaller than BF16
• 4-bit KV cache: ~4x smaller than BF16

Using the same GPT-OSS 120B example:

• BF16: 8k → 576 MiB, 32k → 2.25 GiB, 128k → 9.0 GiB
• INT8: 8k → 288 MiB, 32k → 1.125 GiB, 128k → 4.5 GiB
• 4-bit: 8k → 144 MiB, 32k → 576 MiB, 128k → 2.25 GiB

A recent example is TurboQuant from Google Research, which pushes the quality-loss curve further out: near-lossless behavior around 3.5 bits per channel, mild degradation around 2.5 bits per channel.

GPU Architecture: Where the Numbers Come From

Memory Hierarchy

Every GPU has a memory hierarchy with very different bandwidth at each level:

HBM/GDDR7  →  L2 cache  →  L1/Shared Memory  →  Registers
~1-4 TB/s     ~10 TB/s      ~100+ TB/s           ~20 PB/s (theoretical)

Model weights and KV cache live in HBM/GDDR7. Every decode step, the full weight matrices and KV cache must traverse the HBM→SM bus. This is the bottleneck.

HBM vs GDDR7

HBM is stacked directly on the GPU die using Through-Silicon Vias (TSVs), giving extremely short signal paths and high bandwidth. GDDR7 sits off-die on the PCB, limited by the memory bus width (RTX Pro 6000: 512-bit).

NVLink vs PCIe

• NVLink gives far higher GPU-to-GPU bandwidth (900 GB/s chip-to-chip)
• PCIe works, but the communication tax arrives sooner (~64 GB/s per direction)

For multi-GPU inference: NVLink is better for tensor parallelism, expert traffic in MoE systems, and large prompt-side synchronization. If you know you are going multi-GPU, NVLink > PCIe is still the right default rule.

The Experiment: RTX Pro 6000 Blackwell vs H100 SXM on GPT-OSS 120B

Now the part that illustrates everything above concretely.

The headline result: They are nearly tied at small batch, and H100 pulls ahead as batch grows.

Actual Benchmark Results

Full benchmark code and methodology: github.com/kishan5111/gptoss-benchmark

Key Findings

1. Low-batch decode is basically a tie. At bs=1, the RTX posts 172.1 tok/s versus 166.1 tok/s on H100, while short-prompt TTFT is essentially identical.

2. The bandwidth story shows up as batch grows. At bs=8, H100 leads by about 14%. At bs=32, H100 leads by about 23%.

3. H100 looks better on longer prefill. 4K-token TTFT: 199.3 ms vs 220.8 ms, roughly 10% faster.

4. Single-agent latency is nearly the same. On the 10-step agentic loop, the difference is only 2.1%: 8.04s on H100 versus 8.21s on RTX PRO 6000.

5. VRAM headroom is still a real advantage for RTX. The extra 16 GB matters when the model barely fits or KV cache pressure grows.

GPU Selection Guide

Choose RTX Pro 6000 Blackwell when: Single-user apps/local agent setup, batch below 8, VRAM matters (96 GB vs 80 GB), budget ($8,500 vs $25,000+), workstation/local deployment.

Choose H100 SXM when: High-throughput serving with batch sizes 32+, multi-GPU tensor parallelism (NVLink), long-context workloads, datacenter budget.

The real lesson: Bandwidth theory provides a ceiling, not a floor. Always benchmark your actual workload before assuming spec-sheet ratios translate directly to real performance.

Continuous Batching: Why Throughput ≠ 1/Latency

With continuous batching, as sequences finish their slots are recycled, new requests are inserted into the active batch, and the decode loop keeps running with a moving frontier of live sequences.

The tradeoff: prefill operations are much more compute-intensive than decode. Inserting a new prefill into a batch of ongoing decode steps “steals” GPU time from the decoding sequences. This is the prefill-decode interference problem, and it’s why disaggregated prefill/decode is an active area of systems research.

For agentic workloads, this interference pattern is especially painful. Agents tend to generate short decode sequences followed by a new prefill, creating rapid prefill-decode cycles.

Why PagedAttention matters here

Speculative Decoding: Fixing the Bottleneck Directly

If decode is slow because each token requires a serial forward pass, the fix is to draft multiple tokens at once and verify them in parallel. A draft model generates K candidates quickly. The verifier (full model) runs a single forward pass that accepts or rejects each draft token. Effective speedup: 2–4x on decode.

Variants include classic draft models (small separate LM), EAGLE-1/2 (draft head trained on hidden states), and P-EAGLE (Parallel-EAGLE3 drafting with multiple draft heads in parallel). The key idea is not “more FLOPS” — it’s “fewer expensive serial large-model decode steps.”

GPU Selection Framework

• If the model barely fits, maximize VRAM first. Offloading is a survival problem, not an optimization problem.
• If the workload is decode-heavy, maximize memory bandwidth.
• If the workload is prefill-heavy, maximize compute throughput and kernel quality.
• If multi-GPU inference is unavoidable, favor NVLink over PCIe.

The rough rules: If it fits and batch is small, buy bandwidth. If it barely fits, buy VRAM. If prompts are huge, compute matters more. If generation dominates, bandwidth dominates.

Failure Modes I’ve Hit

1. Thinking TFLOPS = inference speed. More FLOPS only matters when you’re compute-bound. For batch 1–4 and large models, you’re memory-bound.
2. Not accounting for KV cache VRAM at serving scale. Local testing at bs=1 feels fine. bs=32 blows VRAM. Always calculate KV cache for your target concurrency.
3. PCIe multi-GPU for tensor parallel. All-reduce at each layer over PCIe crushes throughput. NVLink is not optional beyond TP=2.
4. Prefill-decode interference in continuous batching. ITL spikes — invisible in throughput benchmarks.
5. Trusting vendor headline numbers. Data-center and workstation parts expose different metrics at different precisions.
6. Underestimating context growth in agentic loops. By step 10, prefill is 5–10x longer than step 1. Always profile the full trajectory.

Practical Takeaway

1. Arithmetic intensity determines the bottleneck.
2. The bottleneck determines which GPU spec matters.

Decode is usually memory bound at real serving batch sizes, while prefill is much more compute-friendly. KV cache, not just weights, determines whether the deployment remains healthy under concurrency.

For agentic systems specifically: you are almost always in the low-batch, high-decode-frequency regime. Bandwidth wins. Every ms off TPOT compounds across every agent step.

The more useful question is not “which GPU is best?” It is:

Is my workload dominated by prompt ingestion, token-by-token generation, agentic loops, or memory pressure from concurrency?

Answer that first. The hardware choice gets much easier after that.