Deploying DistilBERT on AWS EKS

From Minikube to AWS EKS: Infrastructure as Code, Load Testing, and What CPU Throttling Does to Inference Latency

Introduction

In my previous blog, I deployed DistilBERT on a local Kubernetes cluster. This time, I am taking that setup to AWS using EKS (Elastic Kubernetes Service) and adding monitoring with Prometheus and Grafana.

Running DistilBERT locally was straightforward, but moving it to the cloud introduced new challenges such as networking, IAM access, and performance monitoring. By the end of this post, you will see how I deployed an open-source NLP model on AWS, how CPU limits affected inference latency, and what to watch for when running ML workloads on EKS.

Github Repo: https://github.com/ngthecoder/distilbert_serving

Infrastructure Architecture

The infrastructure for this project consists of the following components:

• VPC with 2 public subnets and 2 private subnets in 2 availability zones
• 1 Internet Gateway (IGW)
• 1 NAT Gateway with an Elastic IP for outbound traffic from private subnets
• 1 Elastic Load Balancer (ELB) as a stable endpoint for users
• EKS Cluster (Kubernetes 1.35)
• A managed node group with 1 t3.medium EC2 instance, running 3 pods in private subnets
• Elastic Container Registry (ECR) for image storage
• IAM roles for EKS control plane and worker nodes

I created two public subnets because EKS requires subnets across at least 2 availability zones for high availability. A single-AZ setup causes the cluster creation to fail. I also created a NAT Gateway with an Elastic IP because the pods running in private subnets still need outbound internet access to pull container images and reach external AWS services.

AWS Architecture Diagram

The diagram above shows the architecture and the three main traffic flows through the VPC.

• The first is the request path: Users -> Internet -> IGW-> ELB -> Worker Node -> Pods.
• The second is the response path: Pods -> Worker Node -> ELB-> IGW-> Internet -> Users.
• The third is the image pull and outbound access path: Worker Node -> NAT Gateway -> IGW -> ECR.

Implementation by Terraform

First, what is IaC (Infrastructure as Code)? Without IaC, we need to manually manage the infrastructure by clicking buttons in AWS Console which leads to non-repeatable and unstable infrastructure setup. But with IaC, we can automate the provisioning and management of infrastructure using code instead of manual processes.

I split the Terraform configuration into multiple files by responsibility:

• main.tf: Configures Provider and Terraform
• vpc.tf: Creates VPC, subnets, NAT gateway and route tables
• eks.tf: Defines EKS cluster and managed node group
• ecr.tf: Creates ECR and attaches ECR policy
• iam.tf: Creates IAM roles and policy attachments for the cluster and node group
• variables.tf: Defines input variables
• outputs.tf: Creates outputs such as cluster name, endpoint, arn and ECR repository URL

One pitfall I ran into was that ECR rejects repository deletion when images still exist. To avoid that issue during teardown, I set force_delete = true in ecr.tf.

I also hit an error during terraform destroy because a load balancer created by the Kubernetes Service was still attached to the VPC. Since that ELB was created outside Terraform, it could not be removed by terraform destroy. The fix was to delete the Kubernetes manifests first with kubectl delete -f k8s/, and then destroy the Terraform-managed infrastructure.

Deploying and Testing the Model

After provisioning the infrastructure with terraform apply, the next step was to push the DistilBERT image to ECR. To do that, I first authenticated Docker against ECR using the following command:

aws ecr get-login-password - region us-east-1 | \
docker login - username AWS - password-stdin \
<account id>.dkr.ecr.us-east-1.amazonaws.com

After logging into ECR, we can then build the image, tag it with the URL and push the image with the following command.

docker build -t distilbert-serving .

docker tag distilbert-serving:latest \
<account id>.dkr.ecr.us-east-1.amazonaws.com/distilbert-serving-ecr-repo:latest

docker push \
<account id>.dkr.ecr.us-east-1.amazonaws.com/distilbert-serving-ecr-repo:latest

When deploying Kubernetes Manifests, I had to adjust some parts specifically for the cloud.

In deployment.yaml, I previously had image: distilbert_api:latest and imagePullPolicy: Never to tell Minikube that it needs to look for the image in the local environment. This time, I wanted the cluster to pull image from the remote ECR so I set imagePullPolicy: Always and specified the image image: <account id>.dkr.ecr.us-east-1.amazonaws.com/distilbert-serving-ecr-repo:latest

In service.yaml, I changed the Service type from NodePort to LoadBalancer. NodePort was fine for local testing with Minikube, but it depends on the node IP and port, which is not a stable public entry point in AWS. With LoadBalancer, AWS automatically creates an ELB in front of the nodes, giving the application a stable endpoint even if node IPs change underneath.

After making those changes to the Manifests, I had to run the following commands to get a new kubeconfig to connect kubectl with the remote EKS cluster.

aws eks update-kubeconfig \
 - region us-east-1 \
 - name distilbert-serving-eks-cluster

After updating kubeconfig, I still could not access the cluster from kubectl. The missing step was granting my AWS identity access to the EKS cluster by creating an access entry and associating the AmazonEKSClusterAdminPolicy.

aws eks create-access-entry \
 - cluster-name distilbert-serving-eks-cluster \
 - principal-arn $(aws sts get-caller-identity - query Arn - output text) \
 - region us-east-1

aws eks associate-access-policy \
 - cluster-name distilbert-serving-eks-cluster \
 - principal-arn $(aws sts get-caller-identity - query Arn - output text) \
 - policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
 - access-scope type=cluster \
 - region us-east-1

After giving my AWS user access to the EKS cluster, I applied the Manifests and started the node & pods. Then, I obtained the endpoint URL using kubectl get service distilbert-service.

Using the endpoint URL, I made a single request using curl to confirm the pods are successfully working as follows:

curl -X POST \
http://<ELB URL>.us-east-1.elb.amazonaws.com:8080/analyze \
-H "Content-Type: application/json" \
-d '{"text": "This movie was absolutely fantastic!"}'

{"label":"POSITIVE","score":0.999874472618103}

Load Testing with k6

After confirming the pods were working with curl, I installed k6 and wrote load_test.js. k6 is an open-source load testing tool that makes it easy to simulate concurrent requests over time. Unlike curl, which only verifies individual requests, k6 helped me observe how the system behaved under sustained load.

import http from 'k6/http';
export const options = {
  scenarios: {
    contacts: {
      executor: 'constant-vus',
      vus: 10,
      duration: '30s',
    },
  },
};

const url = 'http://<ELB URL>.us-east-1.elb.amazonaws.com:8080/analyze';

export default function () {
  let data = { text: 'I lost my wallet.' };

  let res = http.post(url, JSON.stringify(data), {
    headers: { 'Content-Type': 'application/json' },
  });

  console.log(res.json().label);
}

This JavaScript file lets 10 Virtual Users (vus) make a large number of concurrent requests for 30 seconds as specified in the options. Using this JS file, I tested the performance of the cluster with two different CPU limits (500m and 1000m).

The test output for 500m CPU limit:

k6 Test Output for 500m CPU Limit

The test output for 1000m CPU limit:

k6 Test Output for 1000m CPU Limit

The important metrics that we need to pay attention to here are http_req_duration, http_req_failed and http_reqs. The http_req_duration field indicates the latencies and p(90/95) shows the latencies 90% or 95% of the requests got. The results are organized in the table below. The http_req_failed field shows the percentage of failed requests and the http_reqs field shows how many requests were made in a given duration.

k6 Test Results

The improvement rates imply that although we doubled the CPU limit, the performance didn’t scale linearly with the CPU limit increase. This implies that even though CPU throttling is eliminated by increasing the limit, there are many more variables that affect the latency such as the network latency and the memory access and that explains why the improvements in p(90) & p(95) are lower; the outliers are caused by sources other than CPU throttling. However, we can see significant improvement in the number of requests made (throughput) and that’s because the pods spend less time waiting for new cycles and can process requests in a shorter time.

Monitoring with Prometheus + Grafana

Along with k6 load testing, I implemented monitoring with Prometheus and Grafana using Helm. First, Helm is a package manager like brew and apt but for Kubernetes. Without Helm, I would have to write all the Kubernetes manifests for Prometheus and Grafana from scratch.

For reference, in Helm, a package is called a Chart, and it typically includes files such as Chart.yaml, values.yaml, and a templates/ directory.

I ran the following commands to set up Prometheus/Grafana pods in the cluster.

brew install helm

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo update

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
 - namespace monitoring \
 - create-namespace

The helm repo add and helm repo update command locally register the Chart’s registry like brew tap does and the helm install command runs the kubectl apply command under the hood to apply the Manifests in the Chart’s template folder.

After the monitoring pods are initialized, I accessed the Grafana UI by port-forwarding to the svc/kube-prometheus-stack-grafana service using the following command. The second command gets the login password for the admin user.

kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80

kubectl get secret -n monitoring kube-prometheus-stack-grafana \
-o jsonpath="{.data.admin-password}" | base64 - decode

The Grafana Dashboard below shows the CPU usage and throttling under 500m CPU limit. The CPU usage reached 0.0546 cores and CPU throttling peaked at 52%. This 52% means that the pod was forced to wait a long time for CPU time. Linux kernel manages the CPU work as cycles of 100 milliseconds and with 500m of CPU limit, a process is only allowed to use 50 milliseconds out of the full 100ms cycle and forced to wait the rest of the cycle (50ms).

Grafana Dashboard under 500m CPU Limit

On the other hand, under 1000m CPU limit as shown below, the CPU throttling stayed at 0% even after the k6 load testing and the CPU usage increased to 0.0901 cores because now the throttling no longer stops the process to use what it actually needs to complete the operations.

Grafana Dashboard under 1000m CPU Limit

Conclusion

Deploying the Kubernetes cluster on AWS took this project beyond local experimentation and helped me better understand how cloud infrastructure changes the way ML systems are operated. Along the way, I ran into a few unexpected issues that made the project much more realistic.

The first was during the terraform destroy process, when I kept getting error messages and eventually found that the ELB was blocking the VPC deletion. The second was that kubectl could not access the nodes and pods even after I connected the EKS cluster to my local environment, because I was missing the cluster access configuration for my AWS identity. IAM is by far one of the most difficult concepts in AWS, but I can feel that I am building a deeper understanding as I work through more projects.

I also conducted more thorough testing using k6, which was a significant improvement over relying only on curl commands. One interesting finding was that although CPU throttling is one cause of latency, there are many other variables that affect it, such as network latency and Python runtime overhead. Throughput improved more significantly than the other metrics, which suggests that CPU throttling had a greater impact on throughput than on latency. To improve latency further, the next steps would be to try measures such as switching from PyTorch to ONNX Runtime, using more suitable compute instances, and improving how the inference service is tuned under load.

The next step is to explore how Horizontal Pod Autoscaling (HPA) affects latency and throughput under various load patterns.