Deploying DistilBERT on Kubernetes

What I Learned About Resource Limits and Inference Latency

Introduction

In the previous blog, I containerized DistilBERT and ended with a question: what changes when you move from a local Docker container to Kubernetes? This post is my attempt to answer that. While deploying on Kubernetes, I faced new challenges around resource management, probe configurations and how CPU limits directly affect inference latency. What I found was that the hard part wasn’t just getting the service to run, but understanding how startup behavior, probes, and CPU limits shaped real inference performance.

Github Repo: https://github.com/ngthecoder/distilbert_serving

Project Overview

As we saw in the previous blog, I have been building a DistilBERT serving infrastructure with Docker and this time, I deployed it on Kubernetes. There are two endpoints for this FastAPI-based service: /pingand /analyze. The current configurations create 3 replicas of DistilBERT pods, accessible via NodePort. When a request comes in, it is routed by the NodePort Service to one of the three replicas. The model is pre-downloaded into the Docker image at build time, which I will cover in the next section.

Why the Model Must Be Pre-downloaded

When using Docker, the model was pulled from HuggingFace on the first run and cached locally, so restarting the container was never an issue. With Kubernetes, however, pods can go down and get re-created at any time, and we cannot rely on local caching the same way. Pulling the model from HuggingFace every time a pod restarts would cause unnecessary delays before the pod is ready to serve requests.

To solve this, I baked the model into the Docker image at build time by adding the following line to the Dockerfile after installing the dependencies:

RUN python -c "from transformers import pipeline; pipeline('sentiment-analysis', model='distilbert-base-uncased-finetuned-sst-2-english')"

The -c flag lets us run Python commands directly from the shell. When this line executes during the build, the model weights are downloaded and stored in HuggingFace’s cache directory ( /root/.cache/huggingface/hub) inside the image. When main.py runs later, the pipeline() call finds the cached model and loads it directly without reaching out to HuggingFace.

Choosing the Right Probe for an Inference Service

We used LivenessProbe and ReadinessProbe this time as well, but with different reasoning compared to the previous project (Defense in Depth: Building a Resilient API with Go and Kubernetes). In that project, I used /pingfor LivenessProbe and /moviesfor ReadinessProbe, because a successful response from /moviesconfirmed that the database connection was established and the app was truly ready to serve traffic. This time, I used /pingg for both, and here is why.

First, Kubernetes only supports GET requests for probes, and since /analyzeis a POST endpoint, /pingwas the only option anyway. Second, pipeline()is called at the module level in main.py, which means the model is fully loaded before uvicorn starts accepting requests. So if /pingresponds, the model is guaranteed to be ready. Third, even if POST requests were supported, hitting /analyzeevery 15 seconds as a health check would run inference on every probe, putting unnecessary strain on compute resources. We definitely want to avoid it from both an operational and cost perspective.

How Kubernetes Limits Pod Resources

Since inference workloads can consume significant CPU and memory, if we don’t limit resources pods can use, they would use up the resources and the whole cluster would suffer from it. In response to this, this time I included resource limits in deployment.yaml. In Kubernetes, requests is the amount of resources guaranteed to a pod, while limits is the maximum it is allowed to use.

Initially, I didn’t know how many resources would be sufficient to run the inference so I gave 1Giof memory to both limits and requests. The details are shown in the YAML codeblock below:

resources:
  limits:
    memory: "1Gi"
    cpu: "500m"
  requests:
    memory: "1Gi"
    cpu: "200m"

I measured the resource usage when the pods were idle meaning they were waiting for incoming requests and at the peak after making 20 /analyze requests. The following table shows the results.

The measurements show that the memory usage doesn’t change whether the pods are idle or at the peak so I adjusted both resources/limits/memory and resources/requests/memory to 500Mi. However, there was a huge bump in CPU usage so I updated resources/requests/cpu from 200m to 50m and kept resources/limits/cpu as 500m. I kept the CPU limit at 500m so the pod could still burst during inference, but lowered the CPU request to 50m because the service spends most of its time idle and I wanted to avoid over-reserving cluster capacity. Below is the updated YAML.

resources:
  limits:
    memory: "500Mi"
    cpu: "500m"
  requests:
    memory: "500Mi"
    cpu: "50m"

How CPU Limits Affect Inference Latency

To understand how CPU limits impact inference latency, I measured the response time under different CPU constraints.

Each request was sent using the following curl command (the port number varies each time):

curl -w "\nOperation lasted: %{time_total} seconds" \
 -X POST "http://127.0.0.1:60579/analyze" \
 -H "Content-Type: application/json" \
 -d '{"text": "This is absolutely amazing!"}'

At first, I ran the test with 3 replicas but the results were all over the place. That didn’t make sense until I realized that the requests were being load balanced across the replicas with different resource usage. So, for this testing, I restarted with 1 replica.

When I dropped the CPU limit to 100m as the first test, the system started falling apart meaning the pods took forever to get ready and sometimes they even dropped from Ready to Not Ready. I first hypothesized that there was something wrong with my deployment. But after thorough investigation, I realized that the problem was that 100m CPU wasn’t enough to even load the model and the uvicorn server couldn’t start within the readiness probe’s initialDelaySeconds. Increasing initialDelaySeconds from 15 to 30 seconds fixed this issue but it unexpectedly revealed something important. That is, CPU limits don’t just affect the performance and under some circumstances, they can even break the startup.

Once the system stabilized, I measured the response time across different CPU limits. The results are shown in the table below.

Simply put, the trend shows more CPU -> lower latency. But this trend is non-linear.

Going from 100m to 300m drastically reduced the latency but once it passed 300m, the rate of change became small. The answer to this lies in how Kubernetes handles CPU limits.

CPU limits are implemented using time slicing. With full access to CPU, a process can use CPU core for the full cycle (100 milliseconds) but with 300 millicore limit, the process can only use CPU core for 30 milliseconds and it needs to wait doing nothing for 70 milliseconds. So if the process with 300 millicore limit wants to do the same thing as it did with no limit, it requires 4 cycles (30ms of working & 70ms of waiting -> 30ms of working & 70ms of waiting -> 30ms of working & 70ms of waiting -> 10ms of working). This is called CPU throttling.

Thus, once the CPU limit is high enough to complete the inference in a single or similar number of cycles, the bottleneck shifts from CPU throttling to the model’s own computation time. That’s why the gains become smaller beyond 300m.

In this setup, 300m looked like the tipping point where CPU throttling stopped being the dominant source of latency. Beyond that, adding CPU still helped, but much less dramatically.

Conclusion

Moving this service from Docker to Kubernetes taught me that deployment is not just about getting containers to run. It forced me to think about startup behavior, health checks, and how resource constraints shape real inference performance. In this experiment, CPU limits had a clear impact on latency, but only up to a point. Once throttling stopped dominating, additional CPU power brought smaller gains.

But this was still a local Minikube setup. The next question is what changes when the same service is moved into a more production-like environment. In the next sprint, I plan to deploy it on EKS with Terraform, add Prometheus and Grafana for visibility, and use k6 to see where the real bottlenecks appear under load.