Deep Dive into NVIDIA Groq 3 LPU: A New Choice for AI Inference

Amid the continuous evolution of AI computing systems, chip architectures are shifting from “general-purpose compute competition” toward “task-specific optimization.” At the recently held GTC 2026, Jensen Huang systematically showcased the blueprint for next-generation AI infrastructure, where the focus is no longer on a single processor but on a complete computing system built around the concept of an “AI Factory.”

The most attention-grabbing announcement this time was not a traditional GPU, but a new computational role — Language Processing Unit (LPU). The Groq 3 LPU has been integrated into the next-generation Vera Rubin platform, working alongside GPUs to form an inference acceleration ecosystem. In AI inference, GPUs are no longer the sole optimal solution; specialized architectures designed for specific workloads are becoming a key complement. This article provides a systematic analysis of the core design philosophy of the Groq 3 LPU, its collaboration with GPUs, and its impact on the AI inference market.

From Single Chip to System-Level Platform: Vera Rubin’s Architectural Shift

In recent years, NVIDIA’s product launches have typically centered on a single GPU, such as the Hopper or Blackwell architectures. Unlike these past single-chip-focused launches, Jensen Huang this time showcased a system-level platform composed of multiple key chips. He explained that when people mentioned Hopper in the past, they thought of a single chip; when mentioning Vera Rubin, it refers to an entire system.

• Vera CPU: A processor designed for the “AI agent era” and reinforcement learning scenarios. It integrates 88 self-developed Olympus cores, offering roughly 50% higher performance and approximately 2× energy efficiency compared to traditional CPUs. Its main role is system-level scheduling and task management, handling resource allocation and workload coordination in large-scale concurrent computing environments.
• Rubin GPU: Connected to the Vera CPU via NVLink-C2C technology, achieving up to 1.8 TB/s interconnect bandwidth, together forming the platform’s main compute core.
• NVLink6 Switch, ConnectX-9 SuperNIC、BlueField-4 DPU, and Spectrum-X scalable switch with integrated co-packaged optics: These components provide high-speed interconnect and network infrastructure for the AI Factory, enabling large-scale data transfer and distributed computing coordination.

Based on these components, NVIDIA demonstrated the Vera Rubin NVL72 rack system, integrating 72 Rubin GPUs and 36 Vera CPUs. Compared with the previous-generation Blackwell platform, the system requires only one-quarter of the GPUs for training large hybrid-expert models, while inference throughput per watt can increase by approximately 10×.

While introducing the Vera Rubin platform, Jensen Huang also highlighted a smaller yet strategically significant chip — the Groq 3 Language Processing Unit (LPU). Originating from NVIDIA’s 2025 acquisition of Groq’s core technologies, it is positioned as a Rubin GPU inference co-processor, becoming an essential part of the Vera Rubin platform.

Explaining the rationale for introducing the LPU, Huang pointed out that with the growth of AI agent applications, inference workloads are becoming increasingly diversified. Some tasks prioritize interactivity and ultra-low latency, where traditional GPUs may exhibit performance redundancy. To meet this demand, NVIDIA introduced an LPU architecture focused on low-latency token generation.

The following sections analyze the technical features of the Groq 3 LPU, hybrid inference architecture design, and its potential impact on AI inference infrastructure.

Core Architecture Design of Groq 3 LPU

The LPU is designed to tightly coordinate compute, storage, and communication under compiler control, achieving efficient and predictable token generation. Unlike architectures that pursue peak compute performance, the LPU emphasizes deterministic execution, on-chip memory bandwidth, and explicit data flow management. This design makes it highly suitable for decoding-heavy, latency-sensitive inference workloads.

Tensor-First Compute with Explicit Data Scheduling

In the LPU architecture, compute and data movement operate on fixed-size vectors (320 bytes) as the basic unit. Arithmetic operations, memory accesses, and cross-device communication all revolve around this granularity, simplifying scheduling logic and reducing synchronization complexity.

Different types of operations are handled by dedicated execution units:

• Matrix Execution Module (MXM): Handles tensor computation, offering high-density multiply-accumulate capability, supporting fixed data types, and ensuring stable throughput.
• Vector Execution Module (VXM): Based on channel-distributed ALU arrays, it handles element-wise operations, data type conversion, and activation function computation.
• Shuffle Execution Module (SXM): Manages structured data movement, including vector rearrangement, rotation, distribution, and transposition.

By explicitly orchestrating data movement, the LPU overlaps computation, memory access, and communication without relying on hardware-level dynamic scheduling.

High-Bandwidth On-Chip SRAM Storage

The LPU’s memory system centers around MEM modules, using a flat design with on-chip SRAM as primary working storage. Approximately 500 MB of high-speed SRAM stores critical inference data, including model weights, activations, and key-value caches.

Unlike traditional cache-based architectures, the LPU uses compiler and runtime-controlled data placement and migration, keeping active data close to compute units. This reduces unpredictable access latency and maintains consistent inference timing.

Due to limited on-chip storage, large models typically scale via multi-chip collaboration, using hierarchical partitioning and parallel strategies to distribute computation across interconnected LPU accelerators. In this mode, system performance depends more on the sustained supply of compute resources than peak performance of individual devices. The LPU offers roughly 150 TB/s on-chip memory bandwidth, combined with high-bandwidth chip-to-chip communication to support stable system output.

C2C Communication for Scalable Deployment

To enable cross-device scaling, the LPU integrates high-radix, high-speed chip-to-chip (C2C) interconnects. Each LPU features 96 links running at 112 Gbps, achieving up to ~2.5 TB/s aggregate bidirectional bandwidth with predictable communication timing.

This design is particularly suitable for distributed inference pipelines, where communication latency often determines overall performance. By ensuring deterministic data transfer, the system can reliably coordinate multiple accelerators.

The LPU uses Groq’s spatial execution model, with the compiler orchestrating computation, data transfer, and synchronization. Unlike traditional architectures relying on runtime dynamic scheduling, this model employs quasi-synchronous hardware mechanisms to minimize clock skew, allowing multiple accelerators to operate in sync.

Under this execution model:

• Memory access and compute are precisely aligned
• Instruction timing is explicitly controlled by the compiler
• Performance variation under different workloads is minimized

For real-time inference, this deterministic execution stabilizes first-token latency and per-token generation time, maintaining consistent response even with small batch sizes.

NVIDIA GPU + Groq 3 LPU Collaboration

In NVIDIA’s Vera Rubin architecture, GPUs and LPUs do not operate as independent systems; they collaborate along different stages of the inference workflow, forming a unified heterogeneous inference path.

For large-model inference, the process typically involves two stages: Prefill and Decode. Rubin GPUs continue to handle prefill operations and attention computations during decoding, which require highly parallel computation and high-bandwidth memory.

Meanwhile, the Groq 3 LPU focuses on latency-sensitive decoding operations, such as feed-forward network (FFN) computations and MoE (Mixture of Experts) execution. Using a deterministic execution model and high-bandwidth on-chip SRAM, the LPU ensures stable response times even under increased concurrency or variable request patterns.

At scale, the LPX rack system contains 256 Groq 3 LPU accelerators, interconnected via C2C links across trays and racks. This vertical interconnect architecture allows efficient workload distribution among multiple accelerators while minimizing latency variability.

When LPX is deployed alongside the Vera Rubin NVL72, the system forms a layered inference path: GPUs handle large-scale parallel computation, while LPUs accelerate key decoding stages. This design maintains high aggregated throughput while providing faster response for interactive AI applications.

Strategic Significance: Why NVIDIA Needs LPUs

As large models scale, AI computing focus is shifting from “training first” to a balance of “inference and service capability.” Introducing the LPU is not just a product-level addition but a structural adjustment in NVIDIA’s overall compute ecosystem.

While NVIDIA GPUs dominate training, their general-purpose parallel architecture is not always optimal for low-latency inference. Competitors like Cerebras and Groq leverage high-bandwidth on-chip storage and sequential execution-oriented dataflow architectures to differentiate in inference. Integrating the LPU into the Rubin platform essentially strengthens NVIDIA’s inference capabilities, covering the full compute chain from training to inference.

Groq 3 LPX racks are compatible with the existing CUDA software ecosystem. Enterprises can add LPU nodes to improve inference efficiency without modifying existing application code. This design lowers the technical barrier for new hardware deployment, allowing clients to expand inference capacity within their current infrastructure.

Conclusion

The launch of the Groq 3 LPU represents not just a new chip but a significant shift in AI computing architecture. In future AI infrastructure, GPUs will remain core, but they are no longer the sole compute engine. By introducing dedicated inference processors, AI systems can achieve a better balance of performance, efficiency, and cost. Performance gains will rely less on individual hardware advances and more on system-level optimization and collaboration. GPUs handle complex computation, LPUs provide rapid response, and together they form a solution that better matches real-world workloads.