Tuning linux network stack - with examples - Part 3

You’ve built a blazing-fast web application with a high-performance language/framework using Java, Go, or Rust. Load testing shows your application is scaling well with a large number of concurrent users and delivering content with ultra high speed. It’s deployed in a region geographically close to target user base, ensuring minimal latency. Survey shows large portion of target audience is already on high-bandwidth internet connections, and you’ve already tuned the Linux network stack to support thousands of concurrent connections (as covered in [Part 1]), with buffer sizes optimized for max throughput (see [Part 2]). Everything is perfect - yet users are still reporting slowness when downloading large files, continuously streaming media, or interacting with WebSocket-powered interfaces. What might be the root cause of this slowness?

Well, the real-world networks are messier than the networks in a cloud environments where performance tests might have been executed. Real users connect over unpredictable networks such as shared Wi-Fi, mobile data etc with varying bandwidths and reliability. These networks introduce latency, jitter, and packet loss that our servers must gracefully handle. Identifying real issue that might be causing slowness would be complicated. However we can first verify which congestion control algorithm our servers are using to deliver content.

By default many linux distributions are using cubic congestion control algorithm, which is good for general-purpose usage such as typical user computers as well as servers delivering content within stable networks like cloud/data center networks. But there is yet another modern congestion control algorithm BBR(Bottleneck Bandwidth and Round-trip propagation time).

In this post, we’ll see how switching to modern BBR congestion control algorithm can dramatically improve user experience in the real-world by efficiently delivering content over public networks. To show effect of cubic vs BBR congestion control algorithms, I am going to test different scenarios over different artificially simulated network conditions. I am going to use 2 AWS EC2 instances (in a same VPC). In each scenario - on a client host, I will simulate different network conditions using tc (traffc control) linux CLI utility which helps implementing various traffic shaping strategies. On a server host - I will just switch congestion control algorithm and observe the effect.

To view current congestion control algorithm on server, we can use following command -

sysctl net.ipv4.tcp_congestion_control

To switch congestion control algorithm on server, we can use following commands -

# To switch to cubic
sudo sysctl -w net.ipv4.tcp_congestion_control=cubic
sudo sysctl -p

# To switch to BBR
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr
sudo sysctl -p

In all scenarios I am using following commands to observe network performance -

# On server
iperf3 -s

# On client
iperf3 -R -c SERVER_IP -t 60 -P 4

Scenario 1 - Default network performance

In this scenario, we will see effect of different congestion control algorithms in default network condition in AWS cloud allowing 10 Gbps bandwidth across EC2 hosts. Since the network is very stable we can see iperf3 test report shows close to optimal download speed on client with both congestion control algorithms -

• With cubic - Bitrate: 9.94 Gbits/sec Retransmissions: 0
• With BBR - Bitrate: 9.92 Gbits/sec Retransmissions: 0

As we can see in ideal lossless network with very low latencies and jitters - cubic congestion control algorithm performs slightly better than BBR.

Scenario 2 - Simulating ingress traffic delivery latency and packet loss

In this scenario, I will add artificial latency and packet loss network condition on client host for ingress traffic flow. We can do that using following command on client host -

sudo ip link add ifb0 type ifb
sudo ip link set ifb0 up

sudo tc qdisc add dev ens5 handle ffff: ingress

sudo tc filter add dev ens5 parent ffff: \
  protocol ip u32 match u32 0 0 \
  action mirred egress redirect dev ifb0

sudo tc qdisc add dev ifb0 root \
  netem delay 80ms 20ms distribution normal loss 2%

But how do we know if this setup is properly delaying ingress data packets and adding artificial latency which we just configured?

Simplest test to check this is with ping CLI utility. In normal condition, we will observe ping timing is very small and consistent -

64 bytes from 8.8.8.8: icmp_seq=787 ttl=116 time=1.38 ms
64 bytes from 8.8.8.8: icmp_seq=788 ttl=116 time=1.31 ms
64 bytes from 8.8.8.8: icmp_seq=789 ttl=116 time=1.31 ms
64 bytes from 8.8.8.8: icmp_seq=790 ttl=116 time=1.35 ms
64 bytes from 8.8.8.8: icmp_seq=791 ttl=116 time=1.31 ms
64 bytes from 8.8.8.8: icmp_seq=792 ttl=116 time=1.30 ms
64 bytes from 8.8.8.8: icmp_seq=793 ttl=116 time=1.32 ms
64 bytes from 8.8.8.8: icmp_seq=794 ttl=116 time=1.31 ms
64 bytes from 8.8.8.8: icmp_seq=795 ttl=116 time=1.32 ms

However with added artificial network condition, we can see ping network round-trip latency is high and there is random jitter -

64 bytes from 8.8.8.8: icmp_seq=215 ttl=117 time=83.7 ms
64 bytes from 8.8.8.8: icmp_seq=216 ttl=117 time=38.1 ms
64 bytes from 8.8.8.8: icmp_seq=217 ttl=117 time=75.9 ms
64 bytes from 8.8.8.8: icmp_seq=218 ttl=117 time=117 ms
64 bytes from 8.8.8.8: icmp_seq=220 ttl=117 time=107 ms
64 bytes from 8.8.8.8: icmp_seq=221 ttl=117 time=79.3 ms
64 bytes from 8.8.8.8: icmp_seq=222 ttl=117 time=69.0 ms
64 bytes from 8.8.8.8: icmp_seq=223 ttl=117 time=110 ms

Let’s observe network performance of different congestion control algorithms in this high bandwidth, high latency and lossy network -

• With cubic - Bitrate: 24.7 Mbits/sec Retransmissions: 1086 Total Transfer: 177 MBytes
• With BBR - Bitrate: 103 Mbits/sec Retransmissions: 89154 Total Transfer: 740 MBytes

As we can see by just switching congestion control algorithm on server host to BBR, server is able to transfer significant amount of more data to client at much higher bitrate accommodating with lossy network conditions.

For a quick side-by-side view, here’s a comparison table -

Scenario 3 - Simulating Bufferbloat

Bufferbloat is a condition where excessive buffering in network devices (like routers, switches, or other network interfaces on network path) causes high latency and jitter, especially during periods of high network usage. Bufferbloat happens when packets sit too long in an oversized queues (without dropping), increasing delay for interactive traffic like websockets, video calls, gaming, or VoIP.

Bufferbloat can produce unpredictable network performance behaviors during high network usage. For simple illustrative example in home network - if 1 user starts downloading large file and the other user is on a video call, and if bufferbloat condition happens due to all available ingress bandwidth utilization and excessive network device packet queue, then the other user will experience sluggish/choppy video call due to the video call’s packets being delayed or dropped in the overstuffed buffer.

Many times when bufferbloat happen, speed test will show good bandwidth available but interactive network traffic like gaming or video calls will still perform poorly because speed tests primarily measure throughput, not latency or packet loss, which are the key metrics affected by bufferbloat.

To simulate bufferbloat we can execute following command on client host -

sudo ip link add ifb0 type ifb
sudo ip link set ifb0 up

sudo tc qdisc add dev ens5 handle ffff: ingress

sudo tc filter add dev ens5 parent ffff: \
  protocol ip u32 match u32 0 0 \
  action mirred egress redirect dev ifb0

sudo tc qdisc add dev ifb0 root handle 1: htb default 10 r2q 100
sudo tc class add dev ifb0 parent 1: classid 1:10 htb rate 100mbit ceil 100mbit
sudo tc qdisc add dev ifb0 parent 1:10 handle 10: bfifo limit 10mb

Let’s observe ping timing after this ingress bufferbloat setup -

64 bytes from 8.8.8.8: icmp_seq=171 ttl=117 time=1.07 ms
64 bytes from 8.8.8.8: icmp_seq=172 ttl=117 time=1.08 ms
64 bytes from 8.8.8.8: icmp_seq=173 ttl=117 time=1.08 ms
64 bytes from 8.8.8.8: icmp_seq=174 ttl=117 time=1.08 ms
64 bytes from 8.8.8.8: icmp_seq=175 ttl=117 time=1.08 ms
64 bytes from 8.8.8.8: icmp_seq=176 ttl=117 time=1.41 ms
64 bytes from 8.8.8.8: icmp_seq=177 ttl=117 time=1.07 ms
64 bytes from 8.8.8.8: icmp_seq=178 ttl=117 time=1.07 ms

It is showing consistent low latency round-trip durations without much jitter when network is idle. Let’s run network performance test with cubic congestion control algorithm

• Bitrate: 98.3 Mbits/sec Retransmissions: 645 Total Transfer: 703 MBytes Average Packet Backlog: 631 Average Packet Backlog Size: 9.2 MB

Packet backlog can be observed during the tests by executing and monitoring output of following command -

watch -n1 'tc -s qdisc show dev ifb0'

We can see large amount of packet backlog is accumulated. If any new packet arrives it will have to wait for all prior packets in a queue to be consumed before current packet’s turn come. This will increase latency for interactive traffic. If we observe latency in ping (which is kind of interactive traffic), we will see each ping is delayed by about 700 millis which was usually completing in ~1 millis in idle network -

64 bytes from 8.8.8.8: icmp_seq=238 ttl=117 time=781 ms
64 bytes from 8.8.8.8: icmp_seq=239 ttl=117 time=839 ms
64 bytes from 8.8.8.8: icmp_seq=240 ttl=117 time=551 ms
64 bytes from 8.8.8.8: icmp_seq=241 ttl=117 time=637 ms
64 bytes from 8.8.8.8: icmp_seq=242 ttl=117 time=692 ms
64 bytes from 8.8.8.8: icmp_seq=243 ttl=117 time=734 ms
64 bytes from 8.8.8.8: icmp_seq=244 ttl=117 time=756 ms
64 bytes from 8.8.8.8: icmp_seq=245 ttl=117 time=771 ms

Let’s see how network behaves when we are using BBR congestion control algorithm on our server -

• Bitrate: 99.2 Mbits/sec Retransmissions: 4 Total Transfer: 710 MBytes Average Packet Backlog: 38 Average Packet Backlog Size: 0.11 MB

Since bandwidth is limited to 100 Mbps, with BBRcongestion control algorithm - server is able to send similar amount of data to the client with almost same bitrate as with cubic congestion control algorithm. However as we can see there is very low packet backlog. There is no huge queue build up of pending packets in a queue. Server continuously observe and adopt to number of packets which can be sent to this congested network and will not overflood network similar to cubic congestion control algorithm was doing. Since there is no huge queue build up, interactive traffic will experience much smoother experience with BBR compared to cubic. If we observe latency in ping, we will see each ping is delayed by about 40 millis only -

64 bytes from 8.8.8.8: icmp_seq=360 ttl=117 time=43.0 ms
64 bytes from 8.8.8.8: icmp_seq=361 ttl=117 time=40.9 ms
64 bytes from 8.8.8.8: icmp_seq=362 ttl=117 time=48.3 ms
64 bytes from 8.8.8.8: icmp_seq=363 ttl=117 time=43.7 ms
64 bytes from 8.8.8.8: icmp_seq=364 ttl=117 time=41.5 ms
64 bytes from 8.8.8.8: icmp_seq=365 ttl=117 time=12.2 ms
64 bytes from 8.8.8.8: icmp_seq=366 ttl=117 time=63.1 ms
64 bytes from 8.8.8.8: icmp_seq=367 ttl=117 time=61.6 ms

For a quick side-by-side view, here’s a comparison table -

Summary In essence, while cubic works well in stable, predictable network environments, BBR is a more advanced congestion control algorithm built for the realities of the public internet. It significantly enhances user experience by adeptly handling fluctuating bandwidths, latency, and packet loss. BBR achieves this by consistently and efficiently delivering data packets without overwhelming the network, resulting in a much smoother and more responsive experience for your application’s users.