🚀 LLM In Production — From Demo to Real-World Systems

Photo by Louis Reed on Unsplash

Today, anyone with a GPU can build an LLM, but this is only half the job done. The real half is far from pending.

Many engineers spend weeks optimizing the models, reducing the latency, applying quantization, tuning KV caches, and benchmarking inference engines. But once the model starts serving thousands or millions of users, a completely different set of problems starts appearing.

For Non-Members: Read here!

Questions like:

1. How do you deploy models safely?
2. What happens when traffic suddenly increases 10x?
3. How do you avoid downtime?
4. How do you monitor the GPU utilization?
5. How do you manage deployments across multiple cloud providers?
6. How do you estimate infrastructure costs?

We basically move from model optimization to operating the inference systems in the real world. Production inference is about scaling low-latency and high-throughput AI systems while maintaining reliability, security, and cost efficiency.

Let us dive into the details of LLM production

Why Production Matters

A model running on your laptop is not production, but serving the millions of requests every day is. The biggest difference is that production systems must handle:

• Request Failures
• Traffic spikes
• Software updates
• Security requirements
• Cost constraints
• Monitoring and alerting

Even if the inference itself is fast, users still experience latency caused by:

• Network communication
• Load balancers
• Queues
• Autoscaling delays
• Client-side bottlenecks

Therefore, inference optimization alone is not enough. End-to-end system design becomes critical.

Containerization

This is the fundamental unit of a software system, irrespective of what you’re deploying.

Containerization means packaging an application together with all its dependencies so that it behaves the same everywhere. Without containers, engineers often face: It works on my machine. But fails in production.

We use containers to solve this problem, and the most common container technology is Docker, where the important concepts include:

Docker Layers

Docker images are built using layers. There are three major layers:

Base Image: It is usually a:

• Ubuntu
• NVIDIA CUDA image
• vLLM image
• SGLang image

And there are additional layers which contain:

• Application code
• Python packages
• Configuration files

Container Layer: These layers are created when the container runs. Any changes made here disappear when the container stops. This keeps the deployments predictable and reproducible.

Dependency Management: Inference systems have very long dependency chains. A typical LLM deployment depends on: CUDA, cuDNN, GPU drivers, PyTorch, Transformers, vLLM, TensorRT-LLM, and Audio or video libraries.

A mismatch in even one dependency can break inference. Thus, it makes sense to preserve the known-good builds using containers because AI infrastructure changes rapidly.

Autoscaling

Once your LLM deployment works, the next challenge is how to handle the traffic. Because the traffic is never constant. For example:

Keeping the GPUs permanently provisioned for peak traffic is expensive. Autoscaling solves this problem. The system automatically:

• Adds replicas when demand rises
• Removes replicas when demand falls

This keeps the latency low while controlling the costs simultaneously.

Concurrency and Batch Sizing: Autoscaling decisions depend on the number of concurrent requests that are expected.

Concurrency refers to how many requests a replica can process simultaneously.

For example:

• Replica A handles 50 requests
• Replica B handles another 50 requests

Total capacity is 100 concurrent requests. Batch sizing is closely related.

With higher batch sizes, we have better GPU utilization, higher throughput, and lower cost. But the downside is higher latency.

A smaller batch leads to faster responses, whereas the downside is the lower throughput.

Inference engineers must balance the latency and throughput based on the application requirements.

Cold Starts Challenge: In general, software systems in production have a cold start problem. And it is more so with the LLM inference set up due to its huge size.

Imagine:

1. Autoscaler creates a new replica.
2. Model weights must load.
3. GPU memory must be initialized.
4. KV caches must warm up.

This takes from a few seconds to even minutes. During this period, many users may experience increased latency, timeout errors, and queue buildup.

Cold starts are particularly painful for large LLMs, image generation models, and video generation models.

Therefore, the production systems must carefully tune the autoscaling policies.

Routing, Load Balancing, and Queues: When there are multiple replicas present, the requests must be distributed among these replicas. This is the job of load balancing.

A load balancer decides:

• Which replica receives traffic
• Which replica is the least busy
• Which region should serve the request

Poor routing of requests leads to idle GPUs, overloaded replicas, and higher latency.

Queues also become important when all the replicas are busy because the requests start entering a queue. Queue depth becomes an important metric because growing queues often indicate insufficient capacity.

Scale to Zero: Some applications receive very little to no traffic. For example, internal tools, research systems, and demo applications.

Running GPUs on these applications continuously becomes wasteful. Scale-to-zero allows all the replicas to shut down, and leads to no GPU costs when idle.

When a request arrives:

• New replicas start
• Model loads
• Service becomes available

The tradeoff is lower cost and higher cold-start latency

Independent Component Scaling: Modern AI systems often contain multiple components:

• Multiple LLMs
• Embedding model
• Reranker
• Speech model
• Vector database

Scaling everything together is inefficient; instead, we can scale each component independently.

Example:

• Embedding service → 20 replicas
• LLM service → 5 replicas
• Reranker → 2 replicas

This reduces infrastructure waste and improves resource utilization.

Multi-Cloud Capacity Management

In production systems, a large-scale system eventually outgrows a single cloud provider. Due to various reasons:

• GPU shortages
• Regional outages
• Compliance requirements
• Cost optimization

It is highly recommended to have multi-cloud deployments.

GPU Procurement: Getting GPUs is harder than many people expect.

Popular GPUs are H100, H200, B100, and B200 from NVIDIA, often due to supply constraints. Production teams must:

• Reserve capacity
• Work with multiple providers
• Plan infrastructure months ahead

A GPU procurement becomes a strategic challenge at scale.

Geo-Aware Load Balancing:

Latency depends heavily on physical distance. A user in India should not always be routed to a US data center. Geo-aware routing sends traffic to the nearest available region.

Benefits:

• Faster responses
• Better user experience
• Lower network latency

Reliability: Once the LLM is in production, it must handle the request failures. Common failures include:

• GPU crashes
• Network outages
• Region failures
• Cloud provider downtime

Multi-cloud deployment improves reliability by redirecting traffic to available services across other regions during failures.

Security and Compliance: Security is especially important for AI systems, and three assets must be protected: user data, prompts, and outputs.

Model Weights: This often leads to valuable intellectual property.

Infrastructure: GPUs and compute resources.

Reducing Attack Surface: If possible, don’t store user inputs and outputs. Less stored data means:

• Lower compliance burden
• Smaller attack surface
• Reduced security risk

Compliance Requirements: Many industries require compliance standards, such as SOC 2, HIPAA, and Regional data residency requirements.

Multi-region deployments help satisfy these requirements by keeping data within approved geographic boundaries.

Testing and Deployment

Before releasing the updates to the larger user base, systems must be tested. Three testing approaches include:

Manual Testing: Engineers send the requests manually to the system to verify the response. It is useful for debugging and small updates to the system. But insufficient at scale.

Load Testing: It artificially generates large amounts of traffic with the intention of testing the durability of your system. To understand where it breaks.

Its purpose includes measuring the scalability, finding bottlenecks, and testing the autoscaling.

Shadow Traffic: This is a pretty cool technique; here, the production traffic is copied to a new deployment, while the users still receive responses from the original system.

Benefits:

• Real-world testing
• No user impact
• Better confidence before deployment

Zero-Downtime Deployment: In this deployment strategy, the users never notice that the updates are being done.

Let’s compare the two strategies.

Blue-Green Deployment

Two complete environments exist: Blue (current) and Green (new). The traffic switches from Blue to Green.

The problem is, if Blue uses 100 GPUs, Green needs another 100 GPUs. Making it very expensive.

Canary Deployment: A more practical solution than Blue-Green deployment. The process:

1. Deploy a new version of the system
2. Send a small traffic percentage to this system
3. Monitor performance
4. Gradually increase traffic if everything goes through.

If something breaks, rollback immediately. Canary deployments provide safety without doubling GPU requirements.

Cost Estimation

This is one of the hardest production problems, especially for an AI system, due to various components.

With API providers, Cost is easy. Cost = Tokens × Price

But dedicated inference is different; the cost depends on:

• GPU count
• Traffic patterns
• Batch sizes
• Input lengths
• Output lengths
• Utilization rates

It is highly recommended to estimate the cost over at least a week because the daily traffic fluctuates significantly.

Another important concept is the Total Cost of Ownership (TCO). TCO includes:

• GPU expenses
• Engineering time
• Maintenance
• Monitoring
• Operations

Many teams underestimate the engineering costs when moving from API-based inference to dedicated infrastructure.

Observability

You cannot fix what you cannot see. Running a large-scale inference system is great, but without monitoring the system, you’ll never know what’s working and what’s not. Monitoring involves:

• Request volume
• Input and Output size
• Response codes
• TTFT and TPS
• End-to-end latency
• Replica count
• CPU and GPU utilization
• Memory usage
• Queue depth

These metrics help answer what and why it’s happening.

For example, High latency may be caused by traffic spikes, long prompts, queue buildup, and GPU saturation.

Observability helps identify the root cause. We can integrate the inference monitoring with tools such as Grafana, Datadog, PagerDuty, and Sentry. So, the inference metrics become part of the organization’s overall monitoring stack.

Conclusion

Production is where Inference Engineering is Tested.

The core message is simple: A fast model is not enough. A production AI system must also be deployable, scalable, reliable, secure, observable, and cost-efficient.

Everything we have discussed above is a practice to transform a model from a research artifact into a production-grade AI service capable of serving real users at scale.

Digital Products

ML Interview Book: Crack Your Next ML Interview with Machine Learning Interview Playbook

Productivity Tool: ***Social Media Time Tracker: Take Back Your Time, a tool that annoys you when you log in to social media sites. Chrome Extension.***

Connect with the author

LinkedIn | YouTube | Threads | Twitter | Instagram | Facebook

Reference

Inference Engineering By Philip Kiely