Your p99 Went to 900ms and Your Traffic Chart Is Lying to You

A model that used to answer in 180ms at p99 now takes 900ms. Requests per second are flat. The dashboards are green everywhere except the one that matters.

The reflex is to profile the GPU. Do that and you will find a GPU doing almost exactly what it was doing last week. Same kernels, same occupancy, same clocks. Hours disappear into flame graphs that show nothing wrong, because nothing is wrong down there.

Both numbers in that report are measured in units that don’t describe a generative endpoint. Fix the units and the investigation gets short.

Three quarters of that 180ms was never yours to control

Start by deriving where 180ms comes from, because a number has to be decomposable before it can be argued about.

Take an 8B-parameter model in fp16 on an H100 SXM. Weights are 16 GB. HBM3 peak bandwidth is 3.35 TB/s, and real kernels achieve roughly 65% to 75% of peak, so call it 70%:

16 GB / (3.35 TB/s × 0.70) = 6.8 ms

That is the time to stream the weights out of HBM once. Decoding is memory-bandwidth-bound, because every generated token requires reading the entire weight matrix. So 6.8ms is the floor on inter-token latency, and it barely moves with batch size. A decode iteration reads those 16 GB once and emits one token for every sequence in the batch. At batch 64 with 500-token sequences, the KV reads add about 4 GiB on top of the weights, so the iteration costs perhaps a quarter more while producing sixty-four times the tokens.

Which inverts the usual intuition: batching buys throughput, not latency. An individual request sees roughly 7ms per token whether it’s alone on the card or sharing it with sixty others.

Now assemble a request. Take 250 prompt tokens and 20 output tokens, a profile that describes an extraction or classification endpoint. Nothing else answers in 180ms.

prefill   250 tokens at ~15k tok/s   ≈   17 ms
decode    20 tokens × 6.8 ms         ≈  136 ms
network, tokenization, queue         ≈   25 ms
                                       -------
                                        178 ms

Decode is 76% of the number, and decode is output_tokens × inter_token_latency. Inter-token latency is the system's speed. Output token count is a property of what the model was asked to produce.

So before anything else: did the output length distribution move? In vLLM’s V1 metrics that’s vllm:request_decode_time_seconds against vllm:inter_token_latency_seconds. If decode time grew five times while inter-token latency stayed at 7ms, the system is exactly as fast as it always was. Going from 20 output tokens to about 125 lands you on 900ms with nothing in the serving stack having changed at all.

That is not a performance regression. It is a product change wearing one: a raised max_tokens, a deleted stop sequence, a reworded system prompt that made the model chattier, a model version bump. It costs one query to rule out, it explains this symptom more often than anything else on the list, and it is the reason end-to-end latency makes a bad SLO for a generative endpoint. You are measuring a quantity you don't control.

Requests per second is not load

The other half of the report claims traffic hasn’t changed, and that claim is denominated in requests. For an LLM the unit of work is the token, and the constrained resource is usually not compute but KV cache memory.

Per token, the KV cache costs 2 × layers × kv_heads × head_dim × bytes. For an 8B model with grouped-query attention (32 layers, 8 KV heads, head dim 128) at fp16:

2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes = 128 KiB per token

On an 80 GB card with weights and activations accounted for, roughly 60 GiB is left for the KV pool:

60 GiB / 128 KiB = 491,520 tokens of KV capacity

Divide that by sequence length and you get the number of requests the box can hold in flight. At 500 tokens per sequence, about 983. At 4,000 tokens, about 123.

Prompts grew eight times, requests per second never moved, and the machine’s concurrency capacity fell by a factor of eight. The traffic chart is flat and honest and completely uninformative.

Prompts grow for reasons that never reach a deploy log. The retriever went from k=5 to k=10. Someone doubled the chunk size. Few-shot examples got added to the system prompt. Chat history stopped being truncated. An agent framework started including tool schemas on every call. A new tenant sends longer documents at the same request rate. None of these touch the serving code, and most of them live in a config service or a prompt-management tool that nobody thought to put a change log on.

Prefix caching belongs in this section because it fails in a way that looks like magic. Automatic prefix caching fires only on an exact prefix match. Prepend a timestamp, a session UUID, a request ID, or an A/B bucket label, and every request becomes a cache miss. Moving Current time: {now} from the bottom of a system prompt to the top can take a 90% hit rate to zero and multiply prefill cost by an order of magnitude, in a change that touched no code and shipped through no pipeline. Watch it with rate(vllm:prefix_cache_hits_total[5m]) / rate(vllm:prefix_cache_queries_total[5m]), both counted in tokens.

The percentile that moved tells you how much traffic is affected

The report says p99. It does not say p50 moved. Assuming that holds up when you check it, it is the most informative sentence in the ticket, because the percentile profile measures blast radius arithmetically.

Suppose some fraction f of requests hits a degraded path, and all of them are slower than the healthy ones. Those requests occupy the top f of the distribution, so the population’s p99 falls inside that band at subset-percentile 1 − 0.01/f.

If 10% of traffic is degraded, your p99 is the p90 of the degraded subset, which is what one bad replica out of ten looks like. If 2% is degraded, p99 is the median of the affected group, and you’re looking at a rare event rather than a broken component. At 1% exactly, p99 sits on the boundary and becomes too unstable to reason about; go to p99.9.

Run it backwards and it partitions the search space before you open anything.

When p50, p90, and p99 all rise roughly five times, nearly every request is affected. That is systemic: clock throttling, a config change, quantization silently reverting to fp16. When p50 stays flat and p90 and p99 both rise, five to fifteen percent of traffic is affected, which points at one replica, one tenant, or one node pool. When p50 and p90 both stay flat and only p99 and p99.9 move, you’re down to one or two percent, and that is a rare-event signature: preemption, a cold start, a retry path.

That last case is the one matching the report, and it points somewhere specific.

Past the knee, more load buys less capacity

The textbook intuition for a rare-event tail is queueing. Take ten replicas carrying 7.2 replica-units of offered load, which is 72% utilization, then lose two of them to a scale-in event, a drained node, or reclaimed spot capacity:

10 replicas:  ρ = 0.72   mean queue wait = ρ/(1−ρ) = 2.6 service times
 8 replicas:  ρ = 0.90   mean queue wait = ρ/(1−ρ) = 9.0 service times

A 3.5× jump in mean wait, and the tail of the waiting-time distribution grows faster than the mean. Same traffic, less supply, nothing in the request path changed.

But M/M/1 is the wrong model for a continuous-batching server, and the error runs in the uncomfortable direction.

Below saturation, batching makes the real curve flatter than the formula predicts. Since a decode iteration reads the weights once and serves the whole batch, adding a concurrent request is nearly free until the KV pool fills. Utilization rises with almost no latency cost, which is exactly the regime where a capacity dashboard looks reassuring.

Above saturation it gets worse than the formula, and not by a little. When the KV pool has no room for a new sequence, the scheduler evicts a running one and recomputes its prefill later. Preempted work is work performed twice. Offered load goes up, effective capacity goes down, and the system can enter a regime where accepting more requests finishes fewer of them.

So the curve is flat, then it is a cliff. There is no gentle shoulder to notice in a weekly review, which is why “we run at 70%” means nothing unless you have measured where the knee actually sits. It also explains the p99-only signature, since preemption hits a minority of sequences by construction and deforms the tail while leaving the median untouched.

Three cheap signals confirm it: vllm:num_preemptions_total nonzero and rising, vllm:kv_cache_usage_perc approaching 1.0 (despite the name, it's a fraction rather than a percentage), and vllm:num_requests_waiting sustained above zero.

Confirming preemption is not the same as finding the cause, though. Preemption is a symptom of vanished headroom, and the headroom went somewhere: longer sequences, fewer replicas, or a smaller pool after someone changed gpu_memory_utilization or loaded more LoRA adapters. Fixing the preemption without finding which one is treating a fever.

Admit fewer requests and you will finish more of them

The obvious response to a saturated service is more GPUs. Against a genuine capacity shortfall that’s correct, and it’s a legitimate way to buy time while you investigate, provided you say out loud that that’s what it is. Against a prefix-cache regression or a single throttling card, it purchases a linear improvement against a problem that isn’t capacity-shaped, at the highest unit cost in your infrastructure, while leaving the defect in place.

The changes that actually address the mechanism each cost something specific.

Chunked prefill breaks long prefills into pieces that interleave with decode steps. Without it, one 8,000-token prefill occupies an entire scheduler iteration and stalls every streaming request on the card. The cost lands on the request that triggered it: its time-to-first-token gets worse, in exchange for the inter-token latency of everyone else staying flat. If your traffic is uniformly short prompts, this buys nothing.

Lowering max_num_seqs is the counterintuitive one. Admitting fewer concurrent sequences reduces KV pressure, which reduces preemption, which eliminates recompute. Recompute is pure waste, so you get less concurrency and more goodput. The cost is real peak throughput in the regime where you aren't saturated, which makes this a trade of the good case for the bad one. Raising it, the more common instinct, buys throughput by spending tail latency.

Autoscale on queue wait or KV utilization, never on RPS or CPU. Both of the usual signals are blind to token load, which is the entire failure mode described above. The cost is that queue-based signals are noisier, and that scaling up an LLM service is slow, since weight loading and CUDA graph capture take minutes. Gate readiness on warm-up completion or your autoscaler will add cold pods that make the tail worse at precisely the moment it’s already bad.

Prefix-aware routing hashes the prompt prefix and sends matching prefixes to the same replica, turning a set of per-replica caches into something closer to a shared one. It works directly against load balance, so it needs a fallback for when the target replica is hot, and that fallback is where the complexity lives.

One thing not to do: raise the timeout. Little’s Law says L = λW. Increasing how long each request stays resident increases how many are resident, which increases KV pressure, which increases how long each request stays resident. It is a feedback loop pointed the wrong way.

Put the SLO on the part you control

This investigation is hard because two variables moved at once and production metrics cannot separate them. “The system got slower” and “the workload got heavier” produce the same end-to-end graph. Three changes make that ambiguity impossible to have again.

Move the SLO off end-to-end latency. Put it on time-to-first-token, which is what a user feels on a streaming endpoint, and on inter-token latency, which is the system’s actual speed independent of how much it was asked to produce. With those two, the ticket would have diagnosed itself. TTFT up with flat inter-token latency is prefill or queueing. Inter-token latency up is a real slowdown. Both flat with end-to-end up means the outputs got longer, and you should go talk to whoever owns the prompt.

Measure load in prefill tokens per second and decode tokens per second. Requests per second is a vanity metric for a service whose unit of work is a token, and keeping it as the primary capacity chart is what let an eightfold load increase render as a flat line.

Then run one replica against a fixed synthetic workload, continuously. It holds the workload constant so the system is the only variable left, and it is the difference between knowing on Tuesday morning and arguing about it until Thursday.

Metric names are vLLM V1, verified against vllm/v1/metrics/loggers.py. They drift. gpu_cache_usage_perc became kv_cache_usage_perc, time_per_output_token_seconds became inter_token_latency_seconds, and the prefix-cache hit-rate gauge was replaced by hits and queries counters. On TGI, SGLang, or TensorRT-LLM the names differ but the decomposition into queue, prefill, decode, and per-token is the same; if your stack doesn't expose it, that's the first thing to fix. All figures here are derived from the stated assumptions and published hardware specs, not measured on a specific deployment. The arithmetic is shown so you can rerun it with your own numbers.