The Empire Strikes Back: Nvidia Snapped Up Groq

Jensen just paid 3X IPO valuation for a $20B sleigh ride into low-latency inference

Christmas came early for Jensen Huang. On December 24th, Nvidia dropped a bombshell: a $20B deal with Groq. To put that in context, that’s 40× Groq’s 2025 projected revenue and 3× its proposed IPO valuation just three months ago. Crazy? Absolutely. A sleigh‑full of cash for a company whose “acquisition” is more of a strategic technology licensing and key-hire play than a classic buyout.

And the timing? Almost comically on the nose. Just a month prior, Meta had started sourcing TPUs from Google, hinting at a subtle tectonic shift in AI hardware sourcing. It’s as if the empire sensed a rival fleet moving in and decided to strike first.

Groq: The Unicorn That Plays by Different Rules

So why did Nvidia pay a king’s ransom? Because Groq’s LPU (short for Language Processing Unit) isn’t your everyday AI accelerator.

Prefill vs. Decode: Why “Low Latency” Matters

Before we dive into the hardware, a quick primer for the LLM inference:

When a large language model (LLM) generates text, there are two main stages:

1. Prefill / Time-to-First-Token (TTFT): This is when the model reads and processes the input prompt.
2. Decode / Token-by-Token Generation: After the first token is ready, the model starts generating text one token at a time, each token depending on the previous ones.

Deterministic, Low-Latency Inference

The sweet spot for Groq LPU is precisely low-latency decode. Benchmarks show 5×–10× speedup over NV H100 for single item inference — despite using a 14 nm process, lagging two to three generations behind H100’s 4 nm node.

In other words, the LPU can outperform newer silicon by knowing how to make every cycle count, it is likely beating Usian Bolt at a 100 meter dash on a pair of flip-flops.

How the LPU Does It

At the risk of repeating myself in the previous post. Let’s again start with the fundamentals. Any processor, CPU, GPU or an ASIC based AI accelerators such as Google TPU and Groq LPU, has three main unit types on the chip

In simple terms:

1. Control Unit ordinates operations of various components, the flow of instruction execution
2. Arithmetic Logic Unit (ALU) performs calculations
3. Memory unit stores data to be processed as well as resulting data

While CPU, GPU and TPU all have these three units. Their missions and priorities lead to different weighting, specialization and optimization of these units.

CPU, because of its all encompassing abilities, spends vast amounts of silicon on the control unit, which handles instruction decoding, branching and control signal generation. That leaves relatively little room for a few, though power ALUs.

In comparison with CPU, GPU doesn’t process complex logic. Hence it can afford several orders of magnitude more ALUs to perform calculations. In Nvidia’s GPUs, those ALUs are further broken into CUDA cores, which perform general floating point and integer calculations, and Tensor Cores, which specialize in matrix calculations.

ASIC AI accelerators, similar to GPU, are very light on control units. Google TPU spends even less silicon on control units than GPU.and has 128 x 128 (16K) computing units. Moreover, their ALUs are constructed specifically for matrix calculations. Google TPU names those units MXU (matrix multiplication unit).

1️⃣ LPU Architecture at a glance

• Minimal control unit: most execution decisions are made at compile time. Think of it as no “if … else …” is allowed.
• ALU-centric: one giant, wide ALU specialized for vector/matrix operations, capable of handling large size matrix at once
• SRAM-heavy: on-chip memory stores the entire context close to the ALU, behaving more like registers, thus avoiding external memory bottlenecks. Its memory size is small compared to GPU’s HBM, but it uses a different type of memory SRAM (static RAM), which is 10X faster than DRAM type HBM.

2️⃣ Compiler-first approach

• The Groq compiler statically schedules every operation, guaranteeing deterministic throughput.
• No dynamic branching, no thread contention — every token is processed exactly on time.

3️⃣ Putting it together with metaphors

• GPU: an all-terrain vehicle — versatile, can handle almost anything, reasonable performance for both training and inference.
• TPU: a truck on a paved road — high throughput on structured, batch-heavy workloads, but less agile, but more efficient (lower cost) per million tokens or per watt.
• LPU: a motorcycle on narrow alley ways — blazing fast for well-defined sequential tasks (decode), but struggles with unpredictable loads or dynamic routing.

4️⃣ Real-world impact

• Latency-critical applications like real-time AI agents, and multi-modal pipelines benefit directly: users get responses fast, every time.
• H100/TPU might have higher raw TFLOPs, but unpredictable stalls and memory latency can slow down per-token delivery.

There’s No Free Lunch

Even brilliance comes with trade-offs.

MoE Models: A Headache

• Mixture-of-Experts (MoE) models dynamically select which set of weights to use per token.
• The LPU’s static pipeline cannot handle this efficiently; dynamic branching decisions break its deterministic scheduling.
• GPUs and TPUs are better at conditional execution and dynamic sparsity, albeit at the cost of variable latency.

Tile Programming Trend

• Tile-style AI programming frameworks like OpenAI Triton, and the latest CUDA 13.1 optimize GPU/TPU workloads by slicing computations into cache-friendly tiles.
• LPU doesn’t need tiles; its streaming ALU + SRAM pipeline handles everything.
• But developers shifting toward tiles and GPUs may ignore LPU, slowing adoption.

Overall

• Dynamic models, very large batch throughput, multi-expert routing — GPUs/TPUs still rule.
• LPU sacrifices general-purpose flexibility for deterministic speed.

Strategic Implications: Why Nvidia Cares

1. Inference is king Real-time AI applications need low and predictable latency. LPU-like performance is increasingly valuable.
2. Talent & tech acquisition over competition

• Nvidia gains SRAM, compiler, and deterministic-pipeline expertise.
• Expect future Nvidia inference products to incorporate these ideas.

3. Clever architecture can beat raw nm

• Even on 14 nm, LPUs outperform in well-defined latency-critical workloads.
• Sometimes narrow cases + architecture + software > bleeding-edge silicon.

Takeaways

• LPUs excel at low-latency, deterministic inference, increasingly important for real-time AI.
• GPUs = versatile all-terrain vehicle; TPUs = high-throughput truck; LPUs = motorcycle delivering small payloads fast and on time.
• Sometimes it’s not raw transistor count, but knowing the streets, tuning the engine, and delivering on schedule.