Tuning Proxmox for High-Speed Networking: Lessons from a 100GbE Struggle

Upgrading to a 100GbE network sounds like a straightforward way to supercharge performance. But for many, the reality is frustratingly different — bottlenecks, unexplained slowdowns, and performance that barely surpasses lower-speed setups.

One user set out to push their Proxmox environment to its full 100GbE potential, only to hit roadblock after roadblock. Their experience offers valuable lessons on what it really takes to optimize Proxmox for high-speed networking, from hardware choices to software tuning. If you’re looking to get the most out of your 100GbE setup, here’s what you need to know.

The Setup: On Paper, Everything Should Work

The test system was built to handle high-speed networking:

• A Proxmox 8.3 host
• Dual 100GbE Mellanox NICs
• A 100GbE-capable Mikrotik CRS520 switch
• Ubuntu VMs configured for testing
• Iperf3 used to measure network throughput

With this kind of hardware, you’d expect line-rate performance — or at least something close to it. Instead, initial tests maxed out at just 10–13Gbps per VM. Even with multiple VMs running simultaneous transfers, the total bandwidth barely reached 40–45Gbps.

That’s nowhere near the 100GbE target. So, what’s holding things back?

Finding the Bottlenecks: Where Performance Breaks Down

When you’re dealing with networking at this scale, the bottleneck is rarely just one thing. Instead, it’s a mix of hardware limitations, software inefficiencies, and configuration issues that all add up. Here are the key problem areas that emerged during testing.

1. Virtualization Overhead and CPU Bottlenecks

One of the biggest misconceptions about high-speed networking is that the NIC is doing all the heavy lifting. In reality, the CPU plays a massive role in processing network traffic.

With Proxmox running virtualized workloads, networking performance is tied to the efficiency of VirtIO drivers, CPU queues, and memory bandwidth. At high speeds, even small inefficiencies in how packets are processed can slow everything down.

Key discoveries:

• The CPU was hitting 30–40% utilization at lower speeds and spiking to 80–90% when pushing past 40Gbps.
• Increasing the number of parallel streams in Iperf helped, but there was a clear CPU bottleneck.
• VM-to-VM communication within the same host was slightly faster (~55Gbps) but still not near full line rate.

2. Multiqueue Networking: Unlocking More Throughput

Most modern NICs and hypervisors support multiqueue networking, which allows network traffic to be processed across multiple CPU cores instead of just one. This is essential for scaling performance beyond 10GbE speeds.

Enabling multiqueue in Proxmox significantly improved performance:

• Setting the number of queues to match the number of vCPUs in each VM led to an increase in total throughput.
• The highest recorded speed jumped to 48Gbps, a major improvement from the initial 10–13Gbps.

While this wasn’t a full 100GbE, it showed that multiqueue settings were a critical factor in unlocking more performance.

3. NUMA and CPU Affinity: Memory Matters

For systems running AMD EPYC or similar high-core-count processors, NUMA (Non-Uniform Memory Access) plays a crucial role in performance. If a VM’s CPU and memory are spread across multiple NUMA nodes, latency increases, and performance drops.

Testing showed that:

• Enabling NUMA awareness in Proxmox didn’t have a major impact on performance.
• However, ensuring that VMs were allocated CPU cores and memory within the same NUMA node helped reduce CPU load during tests.

4. SR-IOV: Direct Hardware Access vs. Flexibility

One suggestion for improving performance was to use SR-IOV (Single Root I/O Virtualization), which allows VMs to have direct access to network hardware instead of going through the Proxmox virtual network stack.

Pros of SR-IOV:

• Significantly reduces CPU overhead.
• Allows VMs to communicate at near-native speeds.

Cons of SR-IOV:

• VMs lose features like live migration.
• Requires additional configuration and driver support.

While SR-IOV is a great solution for maximizing performance, it’s not always ideal for environments where flexibility is needed.

5. Jumbo Frames and Network Offloading: Minor Gains

Another optimization was increasing MTU (Maximum Transmission Unit) to allow larger packets (jumbo frames). In theory, this reduces CPU overhead and increases throughput.

Testing with 9000-byte MTU (vs. the default 1500) showed small but noticeable improvements. However, the biggest gains came from CPU optimizations rather than just MTU changes.

Network offloading features (such as checksum offloading and GRO/LRO) were also tested, but results varied depending on the workload.

The Final Results: How Close to 100GbE Can Proxmox Get?

After extensive tuning, the Proxmox setup was able to push:

• 55Gbps for VM-to-VM communication within the same host.
• 45Gbps when using physical NICs and networking between VMs on different hosts.
• An increase from 10Gbps to nearly 50Gbps in some configurations.

While still shy of the full 100GbE, these improvements highlight just how much tuning is required to maximize high-speed networking in virtualized environments.

Key Takeaways: What You Can Do to Improve Proxmox Networking

If you’re struggling with network performance in a Proxmox setup, here are the most effective tuning steps:

1. Enable Multiqueue Networking

• Set the number of queues in Proxmox to match the vCPU count of each VM.
• Check if your NIC and VirtIO drivers are optimized for multiqueue support.

2. Optimize CPU and Memory Allocation

• Assign VMs to a single NUMA node where possible.
• Use CPU pinning to ensure network-intensive workloads aren’t jumping across cores.

3. Consider SR-IOV for Critical Workloads

• If raw performance is more important than flexibility, pass NICs directly to VMs using SR-IOV.

4. Fine-Tune Network Settings

• Increase MTU to 9000 (jumbo frames) if supported.
• Experiment with offloading features (GRO, LRO, checksum offloading).
• Use PCIe passthrough if absolute max performance is required.

5. Watch Your CPU Usage

• A 100GbE network doesn’t just need a fast NIC — it needs a powerful CPU to keep up.
• High core count and high clock speed are both critical for handling large amounts of network traffic.

Is 100GbE Worth It in Proxmox?

100GbE networking in virtualized environments is far from plug-and-play. Even with top-tier hardware, achieving full line-rate speeds requires careful tuning and an understanding of how virtualization impacts performance.

For many setups, maxing out a 100GbE link may not be realistic — but with the right optimizations, you can get close. The key is balancing performance with manageability, making smart trade-offs between raw speed and the flexibility virtualization provides.