Understanding Cost in LLM Inference Systems

Why Cost Thinking Matters in LLM Systems?

When we run LLM workloads, we are not paying for requests or tokens, we are paying for time. This creates a mismatch between what we pay for and what we actually care about.

A GPU running at full capacity and a GPU sitting idle both cost the same per hour. Because of this, cost is not directly tied to how much work we do, but to how efficiently we utilize the GPU during that time.

So instead of asking:

How many requests are we serving?

We should be asking:

How much useful work are we extracting per unit time we are paying for?

A more useful way to define efficiency in this context is:

Efficiency = useful work (tokens) produced per unit cost

The goal is to connect infrastructure cost with actual output so that we can reason about efficiency in measurable terms.

Translating GPU Cost into Cost per Token

To understand cost, we need to move from infrastructure-level pricing to output-level cost. We can break it down like this:

GPU cost/hour  
↓  
cost/second  
↓  
tokens/second  
↓  
cost/token

We start with GPU cost per hour, which is typically provided by the cloud provider. Then we convert it into cost per second.

Next, we run a real workload and measure throughput in terms of tokens per second (tokens/sec).

Once we have both:

• cost per second
• tokens per second

we can express cost as:

Cost per token = GPU_cost_per_hour / (tokens_per_second × 3600 × utilization)

From this, a few things become clear:

• Higher tokens/sec → lower cost per token
• Higher utilization → lower cost per token
• Anything that reduces concurrency (for example, KV cache growth) → reduces throughput → increases cost

This gives us a practical way to evaluate how efficiently the GPU is converting time into useful output.

How Workload Shape Affects Cost?

Even with the same model and the same GPU, cost can vary significantly depending on how the system is driven.

For example:

• Low concurrency → poor batching → lower throughput → higher cost per token
• Higher concurrency → better batching → higher throughput → lower cost

Similarly:

• Longer prompts increase KV cache usage
• Increased KV cache reduces the number of concurrent requests
• Lower concurrency reduces throughput
• Reduced throughput increases cost

So what we observe is:

Cost is primarily influenced by workload characteristics such as concurrency, prompt size, and output length, rather than just infrastructure.

Because of this, changing GPUs or instance types alone often does not solve cost inefficiency if the workload itself is not optimized.

Understanding Effective Cost?

In practice, cost inefficiencies often come from factors that are not directly visible in basic metrics.

• Idle time: GPU waiting between requests still incurs full cost
• Overprovisioning: allocating more resources than the workload can utilize
• Capacity underuse: inability to fully utilize GPU due to batching or memory constraints

To reason about this, it is useful to define:

Effective cost = Actual cost / utilization

If utilization is low, effective cost increases proportionally. For example:

• 30% utilization → roughly 3× higher effective cost per unit of work

This leads to a more practical optimization goal:

Reduce the gap between actual cost and effective cost by improving throughput and utilization.