From Bottlenecks to Benchmarks: How We Scale LLM Inference on Limited Hardware

When teams first put an LLM into production, the story usually sounds simple:

deploy the model, expose an API, and let users ask questions.

But once real traffic arrives, the system reveals its true nature. The model is not just answering prompts; it is competing for GPU memory, compute, batching efficiency, and queue capacity. That is why a deployment that looks fine at 40 concurrent requests may begin failing at 50 or 60.

The lesson is straightforward: with fixed hardware, LLM serving is always a trade-off between

speed (latency), capacity (how many users we can serve at once), and quality/cost (model size, memory usage, and GPU consumption).

The goal is not to maximize all three, because that is not possible on the same hardware. The goal is to find the best operating point for the workload you actually care about

The first symptom

In our case, the first signal came from load testing. JMeter showed that the service could handle around 40 concurrent users reliably, but at 50 to 60 concurrent requests, some requests started failing. That is a classic sign that the system is crossing from “healthy throughput” into “oversubscription.” At that point, failures can show up as 429s, timeouts, or 5xx responses depending on whether the limit is being hit at the gateway, the application layer, or the model engine itself.

That matters because not all failures mean the same thing. A 429 usually means the system is politely refusing more work due to a quota or concurrency ceiling. A 5xx error usually means the engine itself is under pressure, possibly from overload, long queues, or memory exhaustion. Distinguishing those two cases is the difference between tuning a request limit and resizing a GPU workload.

The layered approach

Once the failure pattern is understood, the solution becomes a layered one. The most effective teams do not jump straight to buying more hardware. They start by reducing unnecessary work at the application level, then optimize prompts, then tune the serving engine, and only then expand platform and hardware capacity.

Application layer: reduce wasted work

The application layer is where the real user value is defined, so this is the best place to start. If a workflow asks the model to generate long, free-form explanations when a short structured output is enough, the system pays for every extra token in latency and compute. A more efficient design is to break work into smaller steps when needed. For example, the application can extract key facts first, then classify, then summarize. That reduces unnecessary reasoning in a single call and can improve concurrency because each request becomes cheaper.

Prompt layer: make tokens count

The prompt is often the easiest place to win back performance. In LLM serving, tokens are the real currency: more tokens mean more prefill work, more decode work, and more latency. Shorter prompts and shorter outputs are not just nicer for users; they are directly correlated with higher throughput and lower response times. Another useful tactic is standardization. If the deployment uses the same system prompt or instruction template across many requests, the runtime can reuse shared prefix computations more effectively. That improves efficiency without changing the model. Structured outputs like JSON or compact bullet points also reduce the total token budget and make the downstream system easier to parse.

Engine layer: tune vLLM for the workload

At the engine layer, the serving runtime matters a lot. IBM watsonx.ai uses vLLM underneath, and vLLM is designed to do smart batching and scheduling so GPUs stay busy.

The main tuning question is not “How many requests can the system accept?” but “How many should it actively process at once before latency and error rates become unacceptable?”

The key controls are the concurrency and batching parameters. max-num-seqs limits how many active sequences the engine handles at once. max-num-batched-tokens controls how much prompt work is admitted into a batch. max-model-len defines how much total context each request may consume. If any of these are too aggressive for the model and GPU size, the system may start to thrash, queue badly, or fail under load.

The practical method is simple: find the concurrency point where throughput stops improving and latency starts rising sharply, then use that as the baseline for tuning. For many real workloads, the right answer is not to force 100 requests through one replica at once, but to keep each replica healthy and spread traffic across more replicas or smaller request shapes.

Platform layer: watsonx.ai and OpenShift

The platform layer is where these choices become operational. Watsonx.ai gives the team a way to package the model, route requests, and expose it as a service. OpenShift then adds scheduling, scaling, and isolation. Together, they decide which workloads get priority, how many replicas are needed, and how the GPUs are shared.

This is also where service tiers matter. Not every request needs the same treatment. A customer-facing use case may deserve lower latency and more replicas, while an internal summarization job may tolerate more batching or a slightly slower response. That separation protects critical paths from being crowded out by less urgent workloads.

OpenShift GPU configuration also matters. In an air-gapped or on-prem environment, the deployment must be reproducible and measurable without depending on external services. That is one reason GuideLLM is useful: it can benchmark the exact deployment inside the disconnected cluster, using the real model and the real serving stack.

Benchmarking as the proof step

Optimization without benchmarking is just guessing. That is where GuideLLM becomes important. It provides a way to run repeatable tests against the serving stack, measure token throughput, TTFT, inter-token latency, full request latency, concurrency scaling, and error rates, and then compare the impact of each change.

In an air-gapped OpenShift environment, this is especially valuable because it means the team can prove performance locally, without internet access, external dependencies, or uncertainty about the benchmark environment. The benchmark becomes the bridge between engineering intuition and executive confidence.

The best flow is always the same:

1. Measure the baseline.
2. Identify where throughput plateaus and latency degrades.
3. Apply one optimization at a time.
4. Re-run GuideLLM.
5. Keep the change only if the benchmark improves the target metrics.

When hardware enters the picture

Hardware should be the final lever, not the first one. Once the team has reduced waste in prompts, stabilized engine tuning, and validated results with benchmarking, it becomes much easier to justify more GPUs or larger GPU slices. At that point, the question is no longer “Can we make it work?” but “What additional capacity do we gain for a given investment?”

That framing is powerful because it turns a vague infrastructure discussion into a business case. Instead of asking for more hardware because the system is struggling, you can show how specific improvements in prompt design, batching, and model sizing changed the concurrency curve. Then you can show exactly what extra capacity a new GPU purchase would buy.