How we seamlessly transitioned our node services to k8s

At Games24x7, we receive bursts of requests that grow exponentially. At season peak, My11Circle product receives close to 200K requests per sec. This means the system handling such scale should be fast, reliable, highly scalable, and cost-effective.

In this blog, we’ll talk about our user-facing DMZ layer of services written in Node.js, acting as upstream API proxies, and serving website traffic. During peak season, we used to run 250+ instances to manage the workload which were hosted on AWS EC2. But for quick scaling and workload compaction capabilities, we decided to switch to Kubernetes (k8s) which allows us to efficiently manage hundreds of pods, optimizing resource utilization and reducing operational costs while maintaining high-performance levels during periods of increased demand.

Background and Migration Plan

In our existing infrastructure, the entire use request goes through the common load balancer that has path-based routing, directing traffic to respective destination target groups and subsequently, to the corresponding applications based on the defined paths. For example, general API traffic goes to our API proxy, finance related APIs go to our common API proxy. The default traffic lands on our Website Landing Pages service.

In our first phase, we planned to migrate Landing Pages to Kubernetes while maintaining the existing infrastructure and other applications intact.

Landing pages are currently hosted in an EC2 environment where both Nginx and Node.js operate on the same instance, each listening on distinct ports. In this setup, client requests are first directed to Nginx, which then forwards them to the Node.js application running locally with cluster.js module. The response from the Node.js process is then compressed and sent back to the client.

Why Nginx? It’s added to the system for its efficient, scale, and robust response compression capabilities, optimizing the transfer of data between the client and the Node.js application.

Why compression? With compression, the response sizes get reduced by 60–70%. It simply means less packet transmissions over the wire. Fewer packets directly means faster response times and that gives a better user experience.

Why not rely on CDN for compression?

• Our CDN, i.e. AWS CloudFront provides compression capabilities. But that’s not guaranteed.
• If the content is cachable, then compression is guaranteed by the edge nodes.
• However, if the content is dynamic, i.e. non-cachable, highly personalized API requests, and responses, then CDN does compression on a best-effort basis.
• Best-effort means? If it’s a low-traffic day for AWS in that edge location (which is used by many AWS customers), then the compression %age will be high, i.e. 50–60% of traffic gets compressed. But, on busy days, like IPL match day when all AWS edge locations PAN India is under high traffic, this compression %age drops to ~30% only.

First challenge: Packaging

The plan was to deploy two containers within the same pod. One container will run Nginx, while the other will host the Node.js application. We planned to utilize local communication between containers, which will avoid any additional network hops.

Key advantages of this approach include:

• Independent Scaling: Each container can scale independently to meet varying workload demands.
• Resource Utilization: Efficient use of resources by isolating Nginx and Node.js in separate containers.
• Independent Dependencies: Containers can maintain distinct dependencies, enhancing flexibility.
• Independent Runtime Variables: Each container can manage its runtime variables independently, providing more granular control.

​

Second Challenge: Request routing using existing ALB

The major challenge was how to leverage the existing load balancer (common load balancer) that is already placed in the system and doesn’t want to give control to any service as it’s tightly coupled with other services, for this we used TargetGroupBinding.

TargetGroupBinding is a custom resource that can expose your pods using an existing ALB TargetGroup or NLB TargetGroup.

This will allow you to provision the load balancer infrastructure completely outside of Kubernetes but still manage the targets with Kubernetes Service.

TargetGroupBinding can be used to bind pods to the target group, facilitating independent deployments that are detached from the common load balancer. The only required step is a one-time manual attachment of the target group to the common load balancer.

The next step was to test this system, below sample yaml was used to bind pods to the target group and it worked perfectly as expected.

Sample Yaml:

kind: TargetGroupBinding
metadata:
  name: my-tgb
spec:
  serviceRef:
    name: awesome-service # route traffic to the service
    port: 80
  targetGroupARN: <arn-to-targetGroup>

Third Challenge: API latency issue with Nodejs on K8s

At Games24x7, We consistently perform performance testing (PT) for each service, given our scale. During PT, we noticed that the system was not scaling as expected, and latency levels were unacceptably high. To investigate the cause, the team thoroughly examined the entire system, scrutinized multiple logs, and pinpointed the issue: DNS resolution time was significantly impacting latency.

Upon further debugging, we identified that CoreDNS latency was spiking. In the K8s setup, CoreDNS is used for service discovery, and the latency has reached around 5 seconds, which raised concerns.

CoreDNS stats showing high latency

This problem has many solutions but with their own trade-offs. The first way could be to augment the resources for CoreDNS, but this would increase costs and is not a sustainable solution. Alternatively, we could implement service-level caching for DNS or leverage dnsmasq. However, opting for service-level changes might not be ideal, considering the multitude of services planned for migration to Kubernetes.

This problem could be effectively mitigated by using NodeLocal DNSCache, which offers an efficient solution for improving DNS resolution in Kubernetes clusters. By installing this add-on, DNS requests from pods are cached locally on their respective nodes. This approach not only enhances performance but also ensures seamless integration, as it requires no modifications to node services and does not affect service restarts. As a result, NodeLocal DNSCache provides a robust method for optimizing DNS operations within Kubernetes environments.

After the installation of NodeLocal DNSCache, coreDNS latency was improved and the system scaled.

Latency stats with NodeLocal DNSCache

Some Key points to keep in mind while working with k8s

• If both containers are running, ensure the liveliness and health checks are configured for both containers within the pod. Otherwise, if the parent container (in this case, Nginx) is up and starts accepting requests, but it forwards requests to another container that is not yet in a working state, the requests will fail.
• Another issue we’ve identified is related to the k8s cluster, which comprises three availability zones. Pods can launch in any of the subnets of these zones, but the common load balancer supports only two of them. Consequently, we need to modify the load balancer to accommodate all three availability zones.

References:

https://aws.amazon.com/blogs/containers/a-deeper-look-at-ingress-sharing-and-target-group-binding-in-aws-load-balancer-controller/ https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.5/guide/targetgroupbinding/spec/

https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html#compressed-content-cloudfront-notes

About the author:

Abhay Gupta is a backend engineer at Games24x7. With over 8 years of experience in the software industry, he specializes in building distributed systems that operate at high scale and concurrency.

Find him on LinkedIn here: https://www.linkedin.com/in/abhayata/