Everyone Is Scaling AI. Nobody Is Solving Inference. That’s the Real Problem

Everyone is racing to build bigger models. The real crisis is in serving them.

Here is a number that should make every AI executive pause: the cost of a single output token has fallen by roughly 280x over the last two years. And yet, the average enterprise AI budget grew from $1.2 million per year in 2024 to $7 million in 2026. Some Fortune 500 companies now report monthly AI bills in the tens of millions of dollars.

The paradox is almost too clean to believe. Intelligence is getting cheaper. Deploying intelligence is getting more expensive. And the companies that do not understand the difference between those two things will spend the next few years very confused about their burn rate.

The reason for this paradox has a name: the inference problem. And it is not a software inefficiency that a clever engineer can patch. It is a fundamental mismatch between how modern AI models are architected and how the hardware we use to run them actually works. In January 2026, Google Distinguished Engineer David Patterson — the Turing Award-winning computer architect who co-designed the Berkeley RISC processor and helped create the TPU — published a paper with colleague Xiaoyu Ma that opens with three words: “LLM inference is a crisis.”

“LLM inference is a crisis.” — Xiaoyu Ma and David Patterson, Google DeepMind, arXiv:2601.05047, January 2026

So today, we are talking about why Patterson is right, what is actually causing the bottleneck at the hardware level, how the AI industry is responding — and why the answer to this crisis will determine which companies can actually afford to run frontier AI at scale.

1. Training vs. Inference: Why They Are Completely Different Problems

When the AI industry talks about compute, it almost always means training. Entire news cycles are built around training runs: how many GPUs, how many tokens, how many dollars. But once a model is trained, you have to run it; over and over, for every user query, every agent loop, every API call. That is inference. And inference has a fundamentally different character.

To understand why, you need to know how a transformer-based language model generates text. There are two phases:

Prefill: The model reads your entire input prompt — all of it simultaneously. This phase is compute-intensive: the model is doing large matrix multiplications across all tokens in parallel. This is why GPUs, which excel at parallelism, are well-suited here.

Decode: The model generates output tokens one at a time. Each new token requires a full forward pass through the model. This is autoregressive: you cannot predict token 50 until you have token 49. There is no way to parallelize across the output sequence.

That second phase — decode — is where everything breaks down. Because generating a single token requires reading all of the model’s weights from memory for every single step, inference during the decode phase is not compute-bound. It is memory-bound.

📐 The Math That Explains Everything

A 70B parameter model in FP16 precision occupies approximately 140 GB of data. On an H100 SXM5 GPU — the current gold standard — the peak memory bandwidth is 3.35 TB/s. At batch size 1 (a single user), simply transferring the weights for one decode step takes roughly 42 milliseconds. That is a hard lower bound on latency. You cannot beat it by adding more FLOPS. You can double the computational budget of the GPU and the 42ms floor stays exactly where it is, because the bottleneck is the memory bus, not the arithmetic units.

This is what Patterson and Ma call the memory wall. GPU compute (measured in FLOPS) has improved roughly 80 times from 2012 to 2022. Memory bandwidth over the same period improved only 17 times. The gap keeps widening. We have built increasingly powerful compute engines that are, for a large fraction of inference workloads, sitting idle — waiting for data to arrive from memory.

2. The KV Cache: The Hidden Tax on Every Query

If decode latency were the only problem, it would be manageable. But there is another layer of memory pressure that grows with every query: the Key-Value (KV) Cache.

During the decode phase, the model needs to attend to all previous tokens in the context window. Computing the attention scores for each new token requires access to the Keys and Values computed for every prior token. Rather than recompute all of these from scratch at every step — which would be catastrophically expensive — transformers cache them in GPU memory after the prefill phase. This KV cache dramatically speeds up decode.

The problem is that the KV cache grows linearly with context length. For a short query, the cache is small. For a long document, a multi-turn conversation, or an agentic loop that accumulates reasoning steps across thousands of tokens, the cache becomes the dominant memory consumer, often dwarfing the model weights themselves.

And context lengths are only getting longer. Models today routinely support 128K, 512K, and 1M token context windows. Reasoning models — which think step by step before answering — generate long chains of intermediate tokens that must be cached. RAG systems inject external documents as extra context. Agentic pipelines accumulate tool call results and prior steps. Every one of these trends drives the KV cache larger.

The GPU has a fixed pool of High Bandwidth Memory (HBM). That memory has to hold both the model weights and the KV cache for all active requests. As context grows, the cache crowds out capacity for batching more users — which is the primary mechanism for amortizing the cost of each GPU across many queries. Fewer requests per GPU means higher cost per token.

💡 DeepSeek’s answer: Multi-Head Latent Attention (MLA)

DeepSeek addressed the KV cache problem at the architecture level with Multi-Head Latent Attention, introduced in DeepSeek-V2 and carried through to V3 and beyond. Instead of storing full Key and Value tensors for every layer, MLA compresses them into lower-dimensional latent vectors, reducing KV cache memory consumption by approximately 40% without significant quality loss. This is one of the key reasons DeepSeek achieves dramatically lower inference costs — it is not just efficiency theater. The model is architecturally designed for cheaper serving.

3. New Trends Making the Problem Worse

The memory wall was already a serious problem. Then the industry decided to make it harder in every conceivable way simultaneously.

Mixture of Experts (MoE)

Modern frontier models — GPT-4, DeepSeek-V3, Llama 4, Mixtral — almost universally use a Mixture of Experts (MoE) architecture. Instead of activating all parameters for every token, MoE routes each token to a subset of specialist “expert” sub-networks. DeepSeek-V3 has 256 experts but activates only a handful per token.

The training story is excellent: you get a model with enormous total parameter capacity at relatively modest compute cost. The inference story is more complicated. Yes, you activate fewer parameters per token — but you have to have ALL experts available in memory, because you do not know which ones a given token will need until routing happens. A trillion-parameter MoE model still requires enough memory to hold all its weights. And because different tokens route to different experts, serving at large batch sizes involves non-trivial communication overhead between devices that hold different expert shards.

Reasoning Models

Models like OpenAI’s o-series, DeepSeek-R1, and Claude’s extended thinking mode generate long chains of reasoning before producing a final answer. This is tremendously useful for hard problems. It is also, from an inference economics standpoint, expensive by design.

A reasoning model might generate 2,000 to 20,000 “thinking” tokens before the actual output token. Each of those tokens is a full decode step. Each requires a full weight read. Each extends the KV cache. The per-query cost of a reasoning model is not 1x of a standard model — it can be 10x to 100x, depending on how much the model thinks. This is a fundamental tension: the capabilities most valuable for complex tasks are also the most expensive to serve.

Long Contexts and RAG

Retrieval-Augmented Generation injects relevant documents into the model’s context at query time. This is enormously useful for grounding responses in current, specific knowledge. But a RAG query with five retrieved chunks of 1,000 tokens each adds 5,000 tokens to the prefill before the model even starts generating. Longer prefill means more KV cache at the start of decode. More KV cache means less memory for other users. And more memory pressure means lower throughput.

The cruel irony is that all the capabilities making AI models genuinely useful — the ability to reason deeply, to handle long documents, to augment with fresh knowledge, to maintain long conversations — all of them make inference more expensive in the same direction.

4. The Economics: Where the Numbers Get Alarming

The economics of inference in 2026 are not what anyone projected. Deloitte puts inference at roughly two-thirds of all AI compute this year, up from one-third in 2023. The market for inference-optimized chips is projected to reach $50 billion in 2026 alone. Meanwhile:

• OpenAI reportedly lost roughly $5 billion in 2024 on $3.7 billion in revenue — a significant portion attributable to inference serving costs.

• Enterprise AI budgets have ballooned from $1.2M annually in 2024 to $7M in 2026, even as token prices fell dramatically.

• Fortune 500 companies are reporting monthly AI inference bills in the tens of millions of dollars.

• HBM prices increased 35% between 2023 and 2025 for both capacity and bandwidth, even as standard DRAM prices fell by roughly half.

The token price deflation has been real — 280x in two years is not marketing spin. But usage is growing faster than prices are falling. Agents make API calls in loops. Reasoning models think for minutes before answering. Applications that once made one query now make dozens. The model that costs $0.01 per query is still expensive if it gets called 10,000 times per user session.

And here is the structural constraint that no amount of software optimization can fix: HBM supply is managed by a small number of manufacturers — SK Hynix, Samsung, Micron — and it cannot scale quickly. Building new fabs takes years and billions of dollars. SK Hynix has warned of supply chain constraints extending into 2028. Hyperscalers have reportedly locked up 40% of global DRAM supply through long-term contracts. The infrastructure race is real, and the memory supply chain is a hard ceiling.

5. What the Industry Is Actually Doing About It

The response to the inference crisis is happening on three simultaneous fronts: software-level optimizations, architecture-level redesigns, and hardware-level rethinking. Here is what is actually being deployed and what is still on the research horizon.

Software: Squeezing More Out of What We Have

Speculative Decoding: Perhaps the most elegant software-level optimization. The core observation is that some tokens are easy to predict — common words, simple continuations — and some are hard. Speculative decoding uses a small, fast “draft” model to speculatively generate several tokens ahead, then passes them all to the large model for parallel verification. If the draft tokens match what the large model would have predicted, they are all accepted in one pass — effectively turning sequential decode into a batch operation for the easy cases. Recent work, including SAGUARO (March 2026), demonstrates up to 5x speedup over standard autoregressive decoding for favorable token distributions. The technique is lossless: it does not change the output distribution of the large model at all.

FlashAttention: Attention computation is the other major bottleneck, particularly for long sequences. FlashAttention (Tri Dao, 2022, extended and iterated multiple times since) restructures how attention is computed to minimize reads from slow HBM by fusing operations and tiling data in on-chip SRAM. This is a kernel-level implementation detail, but the effect is significant: FlashAttention reduces memory usage for attention by a constant factor and speeds up long-context inference substantially. FlashAttention-3 and subsequent variants continue to push these gains further.

PagedAttention and vLLM: Traditional KV cache management pre-allocates contiguous memory blocks for each request, leading to severe fragmentation — reserved memory that cannot be used by other requests because it might be needed later. PagedAttention, the core innovation in the vLLM serving framework, borrows virtual memory management from operating systems. KV cache is stored in fixed-size, non-contiguous “pages” that are allocated on demand. This dramatically reduces waste and allows the GPU to service many more concurrent requests from the same memory budget, improving throughput by 2–4x in practice.

Quantization: Reducing the numerical precision of model weights — from FP16 to INT8, or even INT4 — cuts memory requirements roughly proportionally. A model that requires 140 GB at FP16 needs only 70 GB at INT8 and 35 GB at INT4. At INT4, even large models can run on single-GPU or consumer hardware. The tradeoff is quality: aggressive quantization can degrade outputs on complex tasks, though recent techniques like GPTQ and AWQ have pushed the quality ceiling higher. DeepSeek uses FP8 training, which reduces memory requirements at training time and eases the transition to quantized inference.

Hardware: Rethinking from the Ground Up

This is where the Patterson and Ma paper is most consequential. Their argument is that the optimization ceiling for software running on current hardware is close to being reached. The next order-of-magnitude improvement requires rethinking the memory hierarchy itself. They identify four concrete research directions:

High Bandwidth Flash (HBF): Flash storage has roughly 10x the capacity density of DRAM, at much lower cost per gigabyte — but it has always been too slow for active computation. HBF is a new memory architecture that stacks flash chips like HBM, creating parallel sub-arrays that can operate simultaneously, and achieves bandwidths in the range of 400–800 GB/s — still slower than HBM’s 3+ TB/s, but with 10x more capacity at a fraction of the cost. This makes HBF ideal for storing model weights during inference: the weights are read-only (you never write back to them), so the write endurance limitation of flash is irrelevant. Sandisk and SK Hynix announced a joint standardization effort in February 2026 under the Open Compute Project. First commercial HBF devices targeting AI inference are expected in early 2027.

Processing-Near-Memory (PNM): Instead of moving data from memory to the compute units, what if you moved some compute to where the data lives? PNM architectures attach processing logic to memory modules — not inside the memory die itself (which is PIM, and has severe area and thermal constraints) but adjacent to it, in the same package. This dramatically reduces the data movement that is the root cause of latency. PNM allows memory partitions 1,000x larger than PIM implementations, making it practical for LLM inference where memory is sharded across many locations. Samsung’s AXDIMM and Marvell’s Structera-A are early commercial attempts; the research direction is gaining significant momentum.

3D Memory-Logic Stacking: Using Through-Silicon Vias (TSVs), compute and memory dies can be vertically stacked, dramatically shortening the distance data has to travel. This is already the architecture behind HBM (which stacks DRAM dies), but the concept can be extended: compute-on-HBM designs integrate processing logic directly onto the HBM base die, enabling bandwidths significantly exceeding what is possible with off-chip connections. AMD and several research groups have demonstrated early concepts.

Low-Latency Interconnects: When inference requires distributing a model across multiple nodes — as is increasingly the case for the largest models — the interconnects between nodes become a bottleneck in their own right. Current networking infrastructure was designed for training workloads, where latency tolerance is high. Inference, particularly for reasoning models that generate long sequential chains, demands much lower latency. Specialized in-network computing and topologies optimized for inference communication patterns are an active research area.

6. DeepSeek’s Inference Efficiency: A Case Study in What Is Possible

The DeepSeek moment in January 2025 — when DeepSeek-R1 demonstrated OpenAI o1-class performance at roughly 1/27th the inference cost — was the first widely-noticed proof that the inference economics problem is not simply a matter of raw scale. It is a matter of architectural intelligence.

DeepSeek’s inference efficiency rests on several compounding innovations:

Multi-Head Latent Attention (MLA) compresses the KV cache into low-dimensional latent representations, reducing memory requirements by ~40% at comparable quality. Expert parallelism (their DeepEP library) achieves efficient all-to-all communication at over 40 GB/s, essential for MoE inference at scale. Dual micro-batch overlap intentionally overlaps communication latency with computation — the inference stack is designed, from first principles, with the memory bottleneck in mind.

The result: DeepSeek-V3/V4, with 671B to 1 trillion total parameters, can be served at costs orders of magnitude lower than equivalently-capable dense models. This is not primarily because the MoE architecture activates fewer parameters per token — though that matters. It is because the entire architecture, from attention design to expert routing to inference infrastructure, was co-designed for serving efficiency.

Western labs, optimizing for benchmark performance on training-focused hardware assumptions, are now having to play catch-up with architectural choices that were baked in years earlier.

7. The Strategic Picture: Inference Is Where the Money Is (and Is Being Lost)

This is not an academic problem. The companies that solve inference economics will have structurally lower costs than their competitors. Every dollar saved on inference is a dollar that can go to model development, go to customers as price reduction, or go to the bottom line. The companies that do not solve it will be constrained in how much frontier AI they can actually deploy.

Consider the implications by domain:

Hardware vendors: NVIDIA’s market dominance rests on training. The inference era is creating room for challengers: custom ASICs tuned for memory bandwidth over compute throughput (Groq’s LPX architecture), disaggregated inference systems (prefill and decode handled by separate specialized hardware), and the emerging HBF ecosystem. Whoever ships the first cost-competitive HBF-based inference accelerator will have a compelling proposition.

AI labs: A model that achieves 95% of frontier performance at 30% of the inference cost is commercially more interesting than a model that achieves 100% at full cost. Inference efficiency is becoming a competitive differentiator on par with benchmark performance. The labs that treat efficient inference as a first-class architectural goal — rather than an afterthought — will compound their advantage over time.

Enterprises: Model routing — using smaller, cheaper models for simple queries and reserving frontier models for complex ones — is becoming a standard architectural pattern. The “Big Model Fallacy” — assuming that all tasks require the largest model — is the most expensive mistake in enterprise AI. Quantization, caching strategies, and batching configuration are no longer just engineering concerns; they are budget decisions.

The Bottom Line

For four years, the AI industry has talked almost exclusively about training: larger models, more parameters, better benchmarks. The unexamined assumption was that if you train it, you can serve it. That assumption is now breaking down under the weight of its own success.

The autoregressive decode loop — one token at a time, one full memory read per token — is a structural constraint that no amount of FLOP improvement directly addresses. The KV cache grows with context. MoE models require vast memory even as they activate only a fraction of it. Reasoning models burn tokens to think. RAG injects documents into an already-strained context. Every capability that makes AI useful in 2026 puts more pressure on the same memory bottleneck.

The solutions exist: speculative decoding, quantization, PagedAttention, MLA, FlashAttention. And on the horizon: High Bandwidth Flash, Processing-Near-Memory, and 3D memory-logic stacking. These are real engineering responses to a real hardware constraint. But they require years to reach commercial deployment, and in the meantime, the cost curve is working against the labs and enterprises that have not taken inference efficiency seriously.

Whoever solves the inference bottleneck does not just win on cost. They decide what kinds of AI are economically viable to run at all — and therefore what kinds of AI actually get built.

The companies that win the AI race will not just be the ones that trained the best models. They will be the ones that figured out how to serve them.

Key Sources & Papers

Ma, X. & Patterson, D. “Challenges and Research Directions for Large Language Model Inference Hardware.” arXiv:2601.05047, January 2026.

Spheron Network. “AI’s Memory Wall Problem: Why More GPUs Don’t Fix Inference Latency.” April 2026.

Deloitte / HumanX 2026 Conference Reports on AI Inference Economics.

DeepSeek Technical Reports: V2, V3, R1. “DeepSeekMoE,” “Multi-Head Latent Attention,” “DeepEP.”

Tensor Economics. “MoE Inference Economics from First Principles.” September 2025.

NVIDIA Technical Blog. “Mastering LLM Techniques: Inference Optimization.”

Sandisk & SK Hynix. “HBF Standardization Kick-Off.” Open Compute Project, February 2026.

Leviathan et al. “Fast Inference from Transformers via Speculative Decoding.” ICML 2023.

Dao et al. “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.” NeurIPS 2022.

Kwon et al. “Efficient Memory Management for Large Language Model Serving with PagedAttention.” SOSP 2023.