Optimizing AI Inference Workloads: Reducing Latency, Boosting Throughput, and Cutting Costs

AI inference workloads refer to the process by which a model applies knowledge learned during training to new data to generate predictions, decisions, or outputs, such as language models generating text or image recognition models identifying objects. With the development of generative AI, large language models (LLM), and real-time intelligent applications, the demand for inference is growing rapidly.

In this context, inference systems need to simultaneously meet the requirements of low latency, high concurrency, and controllable cost. As deployment scales up, performance no longer solely depends on increased computing power; the efficiency of data transmission between nodes is becoming increasingly critical. Therefore, optimizing AI inference workloads requires not only focusing on computing resources but also building high-performance networks and efficient interconnect architectures to support overall performance and scalability.

AI Training vs AI Inference Workloads

In practical AI infrastructure design, the core differences between Training and Inference lie primarily in their computational power requirements and cost structure. For a more complete comparison, please refer to the article: Training vs Inference: Why Your AI Network Architecture Needs to Be Different

Computational Power Requirements

AI Training is a typical computationally intensive workload, requiring the processing of large-scale datasets and continuous model parameter updates, placing extremely high demands on computational power, parallelism, and throughput. Such scenarios typically rely on general-purpose GPU architectures to achieve high throughput and massively parallel computing capabilities. Simultaneously, the training process involves frequent gradient synchronization, placing high demands on network bandwidth and latency.

In contrast, AI Inference emphasizes response efficiency and service stability. While the computational scale of a single request is relatively small, the number of requests is large and continuous, making it more sensitive to end-to-end system latency. Against this backdrop, dedicated inference chips are increasingly being introduced for inference scenarios, such as Google’s TPU, Amazon’s Inferentia2, and Groq’s LPU. These hardware components, through targeted optimization, offer significant advantages in energy efficiency and response speed.

Key Takeaway：

In AI inference workloads, computational performance is not the only determining factor. Even with dedicated inference hardware, the efficiency of data transfer between nodes still impacts overall system performance. Network architecture design remains a crucial factor influencing inference performance.

Cost Differences

AI Training is typically a phased investment. Model training is often concentrated within a specific period, such as several days or weeks of intensive computation, with the core goal of achieving model convergence as quickly as possible. During the training phase, costs are primarily one-time or phased expenditures, and optimization focuses on shortening training time, improving computing power utilization, and optimizing resource scheduling efficiency.

AI Inference, however, exhibits entirely different cost characteristics. Inference systems usually need to run for extended periods to continuously respond to business requests. Every user request and every model invocation continuously consumes computing resources and energy. This means that inference costs do not stop once the model is completed but accumulate continuously as the business scales up.

Key Takeaway：

In large-scale AI inference clusters, frequent communication and data exchange are required between different nodes, and network equipment and interconnection solutions will become an important part of the overall cost.

Networking Challenges in AI Inference Workloads

As inference scales up, the network’s role in the system gradually shifts from an auxiliary function to a key component requiring focused attention. In practical deployments, AI Inference Workloads network challenges primarily manifest in three aspects: latency, throughput, and cost. For an overview of the challenges in large-scale inference systems, please refer to this further reading: Top5 Challenges in Large-Scale AI Inference Workloads

Latency

Latency is often the most intuitive and sensitive challenge. In real-time inference scenarios, user requests need to be responded to within a very short time, and the overall latency depends not only on the computation process but also on the data transmission time between different nodes. Key influencing factors include:

• Excessively long network paths or inappropriate topology design
• Inadequate congestion control mechanisms
• Retransmission overhead due to packet loss

In multi-node collaborative inference architectures (such as model parallelism or pipelined inference), each cross-node communication adds to the latency. If network stability is insufficient, latency will not only increase but may also exhibit significant jitter, thus affecting the consistency of the overall service experience.

Throughput

Throughput issues are primarily evident in high-concurrency inference scenarios. When the system needs to handle a large number of requests simultaneously, the network must have sufficient bandwidth to support the overall traffic. Key issues include:

• Insufficient bandwidth leading to request backlog in queues
• Network congestion caused by traffic bursts
• Increased overall response time under high load

In real-world environments, throughput and latency are often interrelated. When network load approaches its limit, latency usually increases simultaneously. Therefore, simply increasing bandwidth cannot completely solve the problem; optimization combining traffic scheduling mechanisms and congestion control strategies is also necessary.

Cost

Cost permeates the entire lifecycle of an AI inference system and is an unavoidable constraint in network design. It is mainly reflected in the following aspects:

• Differences in initial investment costs for network equipment and different technical solutions
• Differences in power consumption costs resulting from different interconnection solutions
• Increased maintenance costs due to increased cabling complexity

Since AI inference systems are typically long-term infrastructure, the cost differences in network solutions become significantly amplified after large-scale deployment.

From latency and throughput to cost, it is clear that the challenges of AI Inference Workloads are no longer limited to a single dimension of computing power. As the scale of inference increases, system performance increasingly depends on the efficient flow of data within the cluster. Therefore, when optimizing AI inference workloads, the focus needs to expand from “improving computing power” to “system-level optimization.” This includes not only the allocation of computing resources but also the selection of appropriate network architecture and interconnection solutions based on the specific characteristics of the business workload. In other words, the network is no longer just a supporting layer but a core component that directly affects inference efficiency and the overall cost structure.

Networking Strategies for AI Inference Workloads

Network Options: InfiniBand vs. RoCE

In AI inference clusters, the current mainstream high-performance network solutions mainly include InfiniBand and RoCE (RDMA over Converged Ethernet). InfiniBand features extremely low latency and high bandwidth, and provides a mature congestion control mechanism. Its advantages are particularly evident in AI inference workloads where performance and stability requirements are extremely high.

In contrast, RoCE implements RDMA over Ethernet, with RoCE v2 being the more widely adopted version. RoCE v2 has advantages in compatibility and cost control, but its performance is highly dependent on the design and optimization capabilities of the lossless network.

The main differences between InfiniBand and RoCE in AI inference workloads are as follows:

• Bandwidth and Latency: InfiniBand typically has advantages in latency control and bandwidth utilization, making it more suitable for inference tasks sensitive to performance limits; RoCE v2, with proper configuration, can meet the needs of most common inference scenarios.
• Compatibility and Cost: InfiniBand relies on dedicated hardware, resulting in higher overall costs; RoCEv2 is based on standard Ethernet, offering greater compatibility and flexibility in device selection and network evolution, helping to control overall costs.
• Configuration Complexity: InfiniBand’s network architecture is relatively closed, making configuration and management simpler; RoCEv2 requires fine-grained configuration to achieve a lossless network, thereby ensuring performance and stability.
• Ecosystem and Supply Chain: InfiniBand hardware is primarily dominated by a single vendor; RoCEv2 is supported by multiple vendors, offering greater flexibility in supply chain and choice.

In AI Inference Workloads, InfiniBand and RoCE are not simple substitutes, but rather network choices tailored to different performance and cost objectives. The former is more suitable for inference clusters with extremely high requirements for low latency and determinism, while the latter has advantages in compatibility with existing Ethernet infrastructure and controllable deployment costs.

Key Takeaway：

In actual deployment, optimizing AI Inference Workloads depends not only on the network type, but also on the specific business scale and architecture design, and further on matching a suitable interconnection solution.

Interconnect Options: DAC vs AOC vs Optical Transceiver

For AI inference clusters, whether it’s server-to-switch connections, intra-rack node interconnections, or backbone links between adjacent racks, choosing the right interconnect solution is crucial. Three common options are DAC, AOC, and Optical Transceiver.

DACs are typically used for short-distance connections, such as within a rack, offering lower cost but limited transmission distance. AOC provides a relatively balanced option between distance and interference immunity, suitable for inter-rack connections. Optical modules offer greater flexibility, supporting longer distances and more complex cabling environments, but at a higher overall cost.

In large-scale AI inference workloads, a more common approach is not to use a single type uniformly, but rather to deploy a combination based on different link layers to achieve a balance between performance, cabling complexity, and total cost of ownership. This layered strategy allows enterprises to adopt more appropriate cost-performance combinations at different network layers, rather than pursuing the global optimum of a single solution. For AI inference, a reasonable interconnect design should serve the overall architectural goals: meeting current workloads while reserving space for future expansion, and avoiding over-investment in unnecessary areas.

NADDOD Lossless Networking Solutions for AI Inference Workloads

NADDOD offers lossless network solutions for AI inference workloads, covering both InfiniBand and RoCE, two mainstream technology paths, and is optimized for different architectures and bandwidth requirements.

NADDOD InfiniBand Network Solutions

Infiniband XDR Solution for Rubin & Blackwell

NADDOD provides a network solution based on InfiniBand XDR for Rubin/Blackwell architectures. This solution uses NVIDIA Quantum-X800 Q3400-RA switches as a unified Spine and Leaf platform, building a two-layer Fat-Tree architecture. The entire solution consists of 32 Leaf switches, 18 Spine switches, and 32 NVIDIA DGX Rubin NVL72/GB300 compute nodes, supporting cluster deployments of up to 2304 nodes.

This solution provides high-bandwidth, low-latency inter-node communication capabilities, suitable for large-scale AI inference scenarios. Under multi-node collaborative reasoning and high-concurrency loads, it can effectively guarantee system performance and scalability.

In terms of interconnect design, this solution offers the following combinations for different connection distances and deployment scenarios:

• Switch to Switch：
• 1.6T OSFP224 2xDR4 IHS ⟷ 1.6T OSFP224 2xDR4 IHS（500m）
• 1.6T OSFP224 2xFR4 IHS ⟷ 1.6T OSFP224 2xFR4 IHS（2km）
• 1.6T OSFP224 DAC IHS（0.9m）
• 1.6T OSFP224 ACC IHS（1~2.5m）
• 1.6T OSFP224 AEC IHS （3m）
• Switch to Server：
• 1.6T OSFP224 2xDR4 IHS ⟷ 800G OSFP224 DR4 RHS (500m)
• 1.6T OSFP224 IHS to 2x800G OSFP224 RHS ACC（2.5m）

Infiniband NDR Solution for Blackwell & Hopper

NADDOD provides a network solution based on InfiniBand NDR for the Blackwell/Hopper architecture. This solution uses NVIDIA Quantum-2 switches as the core devices for the Spine and Leaf nodes, constructing a two-layer Fat-Tree architecture. The entire solution consists of 16 Spine switches, multiple Leaf switches, and 256 NVIDIA DGX H200/B200 compute nodes, supporting cluster deployments of approximately 1024 GPUs.

This solution supports flexible access of 1×400G or 2×400G per node, providing high bandwidth and low latency communication capabilities, suitable for large-scale AI inference scenarios.

Regarding interconnect design, this solution offers the following combinations for different connection distances and deployment scenarios:

• Switch to Switch：
• 800G OSFP 2xSR4 IHS ⟷ 800G OSFP 2xSR4 IHS（100m）
• 800G OSFP 2xDR4 IHS ⟷ 800G OSFP 2xDR4 IHS（500m）
• 800G OSFP 2xFR4 IHS ⟷ 800G OSFP 2xFR4 IHS（2km）
• 800G OSFP IHS DAC（1~3m）
• Switch to Server：
• 800G OSFP 2xSR4 IHS ⟷ 400G OSFP SR4 RHS /400G QSFP112 SR4 RHS（100m）
• 800G OSFP 2xDR4 IHS ⟷ 400G OSFP DR4 RHS（500m）
• 800G OSFP IHS to 2x400G 0SFP RHS DAC（1~3m）

NADDOD RoCE Networking Solutions

1.6T RoCE Solution for Rubin

NADDOD has developed a 1.6T RoCE network solution for the Rubin architecture. This solution uses NVIDIA SN6600-LD switches as the unified platform for Spine and Leaf nodes, building a two-layer Fat-Tree architecture. The entire solution consists of 32 Leaf switches, 18 Spine switches, and 32 NVIDIA DGX Rubin NVL72 compute nodes, supporting clusters up to 2304 GPUs.

In terms of link design, compute nodes access the Leaf layer via 800G RoCE, and Spine and Leaf nodes are interconnected via 2×800G RoCE links. A UFM management node is also included for network monitoring and maintenance.

This solution is suitable for ultra-large-scale AI inference scenarios, providing high-bandwidth communication capabilities while maintaining the deployment flexibility and cost advantages of Ethernet architecture under multi-node collaboration and high-concurrency loads.

For interconnect design, the following combinations are provided:

• Switch to Switch：
• 1.6T OSFP 2xDR4 RHS ⟷ 1.6T OSFP 2xDR4 RHS（500m）
• 1.6T OSFP 2xFR4 RHS ⟷ 1.6T OSFP 2xFR4 RHS（2km）
• 1.6T OSFP DAC RHS（1m）
• 1.6T OSFP ACC RHS（1.1m）
• Switch to Server：
• 1.6T OSFP 2xDR4 RHS ⟷ 800G OSFP DR4 RHS（500m）

800G RoCE Solution for Rubin & Blackwell

NADDOD provides an 800G RoCE network solution for the Rubin/Blackwell platform. This solution is based on a two-layer Fat-Tree architecture, deploying NADDOD N9500–64OC switches at the Spine and Leaf layers. The entire solution consists of 32 Leaf switches, 16 Spine switches, and 256 DGX Rubin NVL8/B300 compute nodes, supporting a cluster size of 2048 nodes.

Compute nodes access the Leaf layer via 400G RoCE, and the Spine and Leaf layers are interconnected via 2×400G RoCE links, offering a more cost-effective deployment option.

The interconnect design offers the following combinations:

• Switch to Switch：
• 800G OSFP 2xSR4 IHS ⟷ 800G OSFP 2xSR4 IHS（100m）
• 800G OSFP 2xDR4 IHS ⟷ 800G OSFP 2xDR4 IHS（500m）
• Switch to Server：
• 800G OSFP 2xSR4 IHS ⟷ 800G OSFP 2xSR4 RHS（100m）
• 800G OSFP 2xDR4 IHS ⟷ 800G OSFP 2xDR4 RHS（500m）

Conclusion

As AI applications move from the training phase to large-scale inference deployments, AI inference workloads have become one of the most critical loads in data centers. In this process, simply relying on increased computing power is no longer sufficient to meet the dual demands of performance and cost. The network is transforming from a “supporting role” to a “core infrastructure.” Latency, throughput, and cost constitute the three key dimensions of inference network design. Therefore, optimizing AI inference workloads is essentially a systemic optimization: it requires not only increasing computing power but also building a high-performance, scalable, and cost-effective network and interconnect architecture.