Beyond Round Robin: Achieving True Traffic Balance across Instances in GCP

In today’s cloud-driven world, ensuring your applications are always available and performing optimally is no longer a luxury, it’s a necessity. Recently for one of my clients who wants to migrate workload from Rackspace to GCP. The main issue is with how to divert traffic to GCP workload and how to effectively distribute evenly across multiple zones. Application Load Balancers (ALBs) in Google Cloud are a critical tool for achieving this, providing a powerful platform for sophisticated traffic management.

Why Traffic Management Matters

Effective traffic management is the key to utilize the full potential of your applications in the cloud. By distributing traffic across your backend instances, you can:

• Maximize uptime and availability: Prevent downtime by automatically routing traffic away from unhealthy instances.
• Enhance performance: Optimize response times by directing traffic to the most suitable backends.
• Improve scalability: Effortlessly handle traffic spikes by scaling your infrastructure dynamically.
• Strengthen security: Protect your applications from attacks with integrated security features.

Failing to implement robust traffic management can lead to slowdowns, outages, and even security vulnerabilities, impacting your users and your business.

Use Case (Round Robin Is Not Round Robin Always)

In this blog, I would like to talk about a specific use case where a client has established a regional application internal load balancer and has encountered anomalous behavior from the load balancer. During the process of migrating 50% of traffic from rackspace workloads to Google Cloud Platform (GCP), the load balancer is routing approximately 70–80% of traffic to a specific zone, while the remaining traffic is distributed across other zones. MIG utilization is less than 10% across zones. The customer has configured a Regional Application Internal Load Balancer with a Round Robin traffic policy, anticipating even traffic distribution across backend instances.

Current customer setup

Load Balancer : Regional Application Internal Load Balancer
Backend : Managed Instance Group (multi Zone)
Traffic policy : Round Robin
Balancing Mode : Utilization

Although there can be multiple reasons for unevenly distributed traffic for example, session affinity, client request origin locations from zone or depends on utilization or how healthy are VMs. Even if ALB is configured for the Round Robin and balancing Mode as Utilization then too ALB will distribute traffic unevenly and redirect traffic to specific zones.

GCP documentation has mentioned this issue to trouble shoot and possible resolution. Troubleshoot issues with internal Application Load Balancers

You might observe an imbalance in your internal Application Load Balancer traffic across zones. This can happen especially when there is low utilization (< 10%) of your backend capacity.

Such behavior can affect overall latency due to traffic being sent to only a few servers in one zone.

To even out the traffic distribution across zones, you can make the following configuration changes:

Use the RATE balancing mode with a low max-rate-per-instance target capacity.

Use the LocalityLbPolicy backend traffic policy with a load balancing algorithm of LEAST_REQUEST.

To address this issue and provide a solution, I’ve created a scenario with an example. The backend configuration has been adjusted.

Changed configuration:

Balancing mode : RATE
Traffic Policy : LEAST_REQUEST
Max-rate-per-instance : 10

Setting the value for the max-rate-per-instance target capacity requires proper configuration. This value will change as per individuals use case. Ideally, it should be less than 10% of the maximum capacity served by the instance. This will enable a smooth transition for requests diverted to other instances in the instance group.

If traffic sent exceeds the max-rate-per-instance multiplied by the number of instances, the backend service is considered full. However, traffic exceeding the configured max-rate is not dropped. The load balancer will continue to distribute traffic across all healthy backends in proportion to their configured max-rate.

Now, Lets move forward in changing ALB configuration:

Initiate parallel request using httperf : number of connections 5000 and 50 requests in parallel

CPU Utilization across zones is on an average same.

Secondly, I tried to increase the load and observe the behavior.

Number of connections :- 50000 and 50 request parallel

CPU utilization was approximately same across instances.

Conclusion

Setting the traffic policy to Round Robin doesn’t always mean traffic will be split evenly between zones and instances. GCP ALB checks the load in each zone and instance, and if it’s super low (like 10% or less), it’ll keep sending most of the traffic there, which makes things uneven.

If you really want things to be evenly distributed, check out the GCP docs as suggested in the blog for a suggested setup. If that doesn’t work, you might need to reach out to GCP support for help.