The One Free Variable

In KV-bound serving, bytes per resident token shape both the memory ceiling and the attention-traffic floor — here is the model, the measurements, and the validation protocol.

By Bhagawan Gnanapa, Co-Founder, Yantrion. Yantrion builds inference-time algorithms that lower the cost of serving large language models.

Start with the regime, not the slogan

In long-context, high-concurrency LLM serving, the bottleneck can shift away from tensor-core arithmetic and toward memory capacity and data movement. Prefill and decode are different workloads: prefill exposes substantial parallelism, while decode advances one token at a time per request and often uses the GPU’s compute units less efficiently.

Model weights remain resident in memory and are read during decode. The KV cache adds a second, per-request working set. It grows with every live token, remains resident for the life of the request, and is consulted by attention as generation continues. So the KV cache is simultaneously what fills memory and what must be moved on every decode step — that dual role is the hinge of the whole argument.

Within that regime, one software-visible quantity becomes unusually high leverage: the bytes stored per KV token, per layer, per serving rank. Call it b.

Scope. This argument concerns long-context, high-concurrency serving where the KV working set is a material capacity and bandwidth constraint — long contexts, high concurrency, agent fleets, or deployments already queuing, evicting, or running out of memory. It is not a claim that every model, batch size, accelerator, or serving stack is KV-bound.

The capacity identity

For R requests that share the same resident context length T, the idealized KV footprint is:

L is the model's layer count. b includes the stored key and value state for one token at one layer on the relevant serving rank.

Production traffic rarely has equal-length requests, so the more general form sums resident tokens across live requests:

The actual allocation is larger or differently shaped, because serving systems add block rounding, allocator metadata, reserved workspace, fragmentation, and other fixed overhead:

Illustrative, not to scale: model weights stay resident and are read during decode, while the KV cache is the per-request working set that grows with every live token — the limiting band in long-context, high-concurrency serving.

This identity exposes the business pressure. Products want longer contexts and more simultaneous sessions, and those two dimensions enter as a product. Double the mean resident context and double the number of live requests, and the ideal KV demand grows by roughly four times — multiplicatively, in exactly the directions a roadmap wants to scale.

b is not literally the only system lever. Model architecture, grouped-query attention, tensor parallelism, prefix sharing, paging, scheduling, context policies, and workload shape all matter. It is, however, the direct representation lever considered here: lowering it can preserve the selected model, retained context, and request concurrency while reducing the KV working set itself.

The latency connection — and its limit

Capacity is only half the argument. The same stored KV state also contributes to decode traffic.

A more complete first-order model for a decode step is:

Q_weights is weight traffic, Q_KV(b) is the traffic associated with the active KV state, and Q_other covers other memory movement. The remaining terms represent runtime overhead: launches, communication, scheduling, and any representation conversion.

In a configuration where the KV term is material, lowering b reduces two pressures at once:

1. Resident capacity: fewer bytes are needed for each live token.
2. Attention traffic: fewer KV bytes need to move through the memory system.

That does not guarantee lower end-to-end TPOT. A smaller representation can add conversion work, or expose launch and scheduling overhead that was previously hidden. The realized latency result must therefore be measured. We use:

A value below 1 is faster than stock; 1 is parity; above 1 is an overhead.

An illustrative decode-traffic model: step time tracks total bytes moved, Q_total = Q_weights + Q_KV(b) + Q_other; lowering b lowers the KV-dependent term, while total TPOT remains a measured runtime result.

The useful systems insight is therefore more precise than “one lever always improves everything”:

A lower KV footprint creates room to improve both resident capacity and KV-related decode traffic. The runtime decides how much of that theoretical advantage becomes measured latency or throughput.

Why common optimizations are complementary

Several familiar techniques address adjacent parts of the problem, and the compelling point is composition, not dismissal.

Weight quantization (MXFP4, INT4, and friends) reduces the memory occupied by model parameters and can reduce weight traffic. It does not change the bytes stored per KV token, but it can free memory for a larger KV pool. Weight and KV techniques compose: one shrinks the model, the other shrinks the per-token cache.

Paged allocation and prefix sharing reduce fragmentation or duplicated state. They improve how efficiently the pool is used, rather than changing the representation size of each token.

Windowing and eviction reduce retained context. They lower memory demand by changing what the model remembers, which may be acceptable for some products and unacceptable for others.

The specific target here is different: reduce the effective b while preserving the chosen model and retained context, then prove that the runtime and task quality remain acceptable.

What Yantrion claims to change

Yantrion is an inference-time serving layer that lowers the effective KV bytes per token. It is delivered as algorithm-aware optimized kernels and serving-stack integrations, and it does not retrain or modify the model weights.

This piece is deliberately empirical. It is written to be judged by what an operator can reproduce on their own hardware, not by a description of the method — so the entire case rests on quantities anyone can inspect:

• the stock and modified KV pool reported by the same serving engine;
• TPOT under the same model, workload, and runtime configuration;
• downstream task quality under a paired evaluation protocol;
• an off-path comparison that restores the unmodified server behavior;
• reproducibility across specified stacks and hardware.

The measured fact we build on is the increase in resident-token capacity under a matched KV-memory budget. Read through the capacity identity above — and after fixed overhead is accounted for — that increase corresponds to a lower effective per-token KV footprint. How that footprint is achieved does not enter the argument; the result is carried entirely by the measurements.

The results, by stack and test

The measurement set is published as a table by stack and test rather than a blended range. A row is not always a single operating point. The capacity columns (stock/modified KV and ρ) and the TPOT column (τ) are, in general, separate tests on that stack — sometimes at different serving configurations — not one co-measured configuration. So: compare stock and modified values within the capacity test; read τ as a separate TPOT measurement on the same stack, at a different operating point; and treat absolute pool sizes as within-row, not cross-row, benchmarks

The τ values are TPOT ratios against unmodified stock, and they are not uniform. Three honesty notes travel with the table. First, capacity and TPOT in a row are separate tests, not one co-measured operating point — read them independently, never as a matched (capacity, latency) pair. Second, the latency picture differs by stack: on the shipped AMD/SGLang path, at the high concurrency this serving targets (32–128 concurrent requests), TPOT is at or slightly below stock (τ ≈ 0.96–0.98×) — the 1.80× capacity gain carries no latency penalty in that regime; GB10/vLLM carries a small ~7% small-batch overhead (τ ≈ 1.07×); and ‡ the TensorRT-LLM path is still on bring-up — its measured TPOT is dominated by un-fused per-request/per-head host loops rather than the codec, so we exclude it (—) rather than headline it. Third, each row is a different model — Kimi-K2.7-Code on AMD/SGLang, GLM-4.7-Flash on GB10/vLLM, DeepSeek-V2-Lite on GB10/TensorRT-LLM — so the rows show the lever reproducing across models and stacks, not a single-model sweep.

A note on the counts, because a reader will check them: the shipped AMD/SGLang result is 3,310,444 / 1,839,135 = 1.7999, i.e. 1.80× realized after fixed overhead. The GB10/vLLM allocator ratio is 9,074,464 / 2,629,472 = 3.4510 (3.45×), and TensorRT-LLM is 3,320,704 / 969,536 = 3.4250 (3.43×) — both the live pool the engine's own KV manager will hold. These are counts an operator can read out of the serving engine, not headline figures. The counting convention (instance-level vs per-rank) is noted per row where it matters.

The AMD/SGLang row is a realized production result: 1.80× more resident KV tokens in a high-concurrency (MRR-128) configuration, after fixed overhead and shared-VRAM effects are charged against the pool. The two GB10 rows are allocator-capacity ceilings: 3.45× (vLLM) and 3.43× (TensorRT-LLM) are the resident-token ratios the engine's own KV manager will hold, stock versus modified. So across three stacks and two GPU vendors, the same lever yields a 1.80× realized production gain and 3.4–3.5× allocator ceilings — the gap is fixed overhead and box sharing, not a different lever. A conservative capacity plan sizes a budget against the 1.80×; the ceilings show the headroom on an unshared box.

On latency, measured against unmodified stock: the shipped AMD/SGLang path is at or slightly below stock at production concurrency — τ ≈ 0.96–0.98× at 32–128 concurrent requests — so the 1.80× capacity gain comes with no latency penalty in the high-concurrency regime this serving targets. (Low-concurrency performance should be evaluated separately and is not characterized by this high-concurrency result.) On GB10/vLLM the optimized kernel — tuned from 228 µs to 80 µs (a 2.85× kernel improvement) — carries a small ~7% small-batch overhead (τ ≈ 1.07×). The TensorRT-LLM path is still on bring-up: its measured TPOT is dominated by un-fused per-request/per-head host loops rather than KV bandwidth, so it is unrepresentative of the codec's cost and we exclude it from the table. Per-stack TPOT absolutes, percentiles, and the kernel-fusion status were recorded for each run.

Measurements were collected in June–July 2026, with full per-row configuration recorded — stack commits, driver and runtime versions, cache dtype, block size, parallelism, context and concurrency, decoding settings, seeds, warm-up policy, TPOT absolutes and percentiles, and allocator startup logs.

At a fixed KV budget, resident-token capacity scales approximately with 1/b; the measured GB10/vLLM endpoints (9,074,464 vs 2,629,472, 3.45×) anchor the line, which excludes differing fixed overhead. Measured endpoints, illustrative interpolation.

Boundaries that travel with every headline

Limitations belong before the derived economics, not after them.

The regime matters. The strongest fit is long-context, high-concurrency serving where KV capacity or traffic is material. A compute-bound or launch-bound single-stream configuration may not realize the same benefit.

The algebra transfers; the ratios do not. The identity and the traffic model carry across silicon, but every capacity ratio, TPOT ratio, and kernel speedup must be remeasured for the target model, stack, allocator, hardware, and workload.

Allocator capacity is not end-to-end throughput. Both are valuable measurements, but they must be named separately.

A KV-memory estimate is not a bill. The KV-memory estimate 1/ρ, introduced below, is arithmetic on a measured ratio — not a total-cost-of-ownership result. (We deliberately do not publish a ρ/τ "capacity-per-latency" figure; capacity and TPOT were not co-measured at one operating point.)

Productionization is engineering work, not polish. Batching, graph capture, fault handling, observability, and per-workload tuning can materially change both performance and risk.

What can be derived — and what still needs to be measured

At fixed overhead, the ideal capacity ratio is:

A tempting next step is to divide the capacity ratio by the TPOT ratio to get a “capacity-per-latency” figure. We deliberately do not publish that number. The capacity and TPOT results here come from different operating points — and, on a given stack, sometimes different serving configurations — so their quotient does not describe a single attainable configuration, and it should not be read as one. A combined capacity-and-latency trade-off measured at one matched operating point remains to be run.

What can be derived are two independent planning quantities. First, 1/ρ is a useful estimate of the KV-resident-memory requirement for the same token pool. It is not automatically the total dollar-cost ratio of a production service, which also includes the model footprint, CPUs, networking, power, fixed fleet granularity, utilization, and the latency target. Second, a related planning figure is the resident-session density:

where P is the resident KV-pool tokens per node. In the KV-bound regime, multiplying the resident pool by ρ lets a node hold roughly ρ× more live sessions of a given context length before it must evict and re-process — the point where memory-bound serving latency cliffs rather than gracefully degrades.

Measured capacity, independent derived quantities: the measured capacity ratio ρ and resident pool P each feed a separate planning figure — KV-memory requirement ≈ 1/ρ, and sessions/node ≈ P / tokens-per-session. TPOT is measured and reported separately; it is deliberately not combined with ρ into a single "capacity-per-latency" number, because the two were not measured at one operating point.

Correctness is the gate, not the decoration

A serving-layer optimization can look successful in a latency chart while quietly changing an answer. A corrupted cache passes every timing check and is still wrong. That is the failure mode the validation protocol is designed to catch, so the governing rule is stated as a principle:

Gate on downstream task behavior before using latency as evidence of success.

The public protocol contains four layers.

1. Exact-path checks where exactness is expected

When the flag is off, no modified code executes: the server runs the identical stock code path on the same base-server commit, build, model files, configuration, and deterministic request corpus, so it reproduces stock output exactly. Confirm that equality and, where you have the artifacts, publish the output hashes. A single flag that restores this base path is valuable because it bounds operational recovery time and helps isolate incidents — it answers “is this the modified layer or the base server?” at the coarsest possible granularity. (Whether that flag is a live toggle or a config-plus-restart is stated with the deployment notes.) It is not described as eliminating all deployment risk. A cosine printed as 1.000000 is supplementary evidence, not proof of byte identity.

2. Numerical error checks where floating-point variation is expected

Compare every supported kernel shape against an independent reference. Publish max-absolute error, max-relative error, and ULP error by dtype, head geometry, batch size, and sequence length.

For split or tiling robustness, state the exact statistic behind the tolerance. Floating-point reductions can depend on operation order; the defensible claim is that every supported tiling remains within a predeclared bound relative to the reference — in the reductions here, max-abs ≤ 6.5×10⁻⁸ across supported tilings.

3. Paired task gates

Use complementary functional tests:

• Long-context retrieval (needle-in-a-haystack, NIAH) checks whether information buried deep in the prompt remains usable.
• Multi-step reasoning (GSM8K) checks whether small numerical changes accumulate into a changed answer.

Across the disclosed runs, no task regression was observed in the disclosed sample under the stated per-integration gates: NIAH is at stock (1.0 on AMD/SGLang; 5/5 = stock on GB10/vLLM), and the GSM8K point estimate was ~0.94 for Yantrion versus approximately 0.93 for stock (n=100). We treat this as parity and do not claim an improvement in reasoning. A formal non-inferiority result — a predeclared margin, an appropriate paired analysis, and a confidence interval — is not asserted here; the paired scores, scorer, and seeds were logged, and a publication-grade version would use a larger paired sample or deterministic decoding.

4. Capture and replay for regressions

When a metric regresses, capture the real serving operands, replay them offline against the reference, and staged-diff until the first divergence is localized. This converts an intermittent server-level failure into a reproducible offline comparison with a definite answer.

The correctness-first validation protocol: exact equality where expected, otherwise published max-abs / rel / ULP bounds, then paired NIAH and GSM8K task gates (no observed regression) with n, seeds, and scorer stated; the flag-off path provides single-flag rollback and fault isolation.

Cross-stack and cross-vendor reproduction

The resident-capacity effect reproduced across three serving integrations and two GPU vendors: SGLang on AMD MI350X, and vLLM plus TensorRT-LLM on NVIDIA GB10. TPOT was measured on the SGLang and vLLM rows; the TensorRT-LLM evidence currently covers capacity and correctness (its flag-off path reproduces stock output exactly). Each integration was evaluated with the task gates shown for its row.

That diversity is meaningful. Different allocators, kernel paths, and integration surfaces reduce the chance that a result is an artifact of one serving engine. The accurate term is cross-stack, cross-vendor reproduction; “independent replication” is reserved for experiments run and controlled by a separate organization or investigator.

For vLLM, the integration is packaged through the project’s documented plugin entry-point mechanism — a supported path for user-registered, out-of-tree code — not a runtime patch. The TensorRT-LLM row (on DeepSeek-V2-Lite) is validated in Yantrion’s TensorRT-LLM integration; its flag-off path runs the identical stock code and reproduces stock output exactly. Exact stack versions, commit SHAs, and measurement dates were recorded per row.

Cross-stack, cross-vendor reproduction: the same measurement contract across three integration paths (SGLang on AMD MI350X; vLLM and TensorRT-LLM on NVIDIA GB10) and two GPU vendors, with per-integration task gates shown. Capacity reproduced on all three; TPOT was measured on SGLang and vLLM.

The stance

The strongest version of this story is not that one equation proves a product result. It is that the equation identifies a high-leverage variable, the serving engine exposes that variable, and a disciplined experiment can measure what happens when it changes. The case is carried by those measurements, not by any account of the method.

The useful question for an operator is simple: under the same model, stack, workload, and task gate, how many resident KV tokens fit, what happens to TPOT, and do the answers remain correct?

Every quantity here is one an operator can read out of the same serving engine — the stock and modified pool, the TPOT ratio, the task-gate scores. That is deliberate: the result is meant to stand on measurements you could take yourself, under a disclosed model, stack, workload, and correctness gate — not on taking Yantrion’s word for it.

References

• S. Williams, A. Waterman, D. Patterson. Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM, 2009.
• K. Cobbe et al. Training Verifiers to Solve Math Word Problems (GSM8K). arXiv:2110.14168, 2021.
• G. Kamradt. Needle In A Haystack — Pressure Testing LLMs. 2023.
• vLLM project. Plugins (documented entry-point extension mechanism). vLLM documentation.
• NVIDIA. GB10 Grace Blackwell Superchip — unified-memory bandwidth ≈273 GB/s. Product specification.

The One Free Variable — Yantrion. LLM Inference · GPU · KV Cache · Machine Learning Systems · AI Infrastructure.