Tuning MoE kernels in vLLM on consumer GPUs (2× RX 7900 XTX): +91% throughput — and proof that the gain survives production

A case study in three layers. First, a controlled synthetic A/B shows +91% throughput at 64 simultaneous requests after tuning the MoE kernels of a Qwen3.6‑35B‑A3B (AWQ int4) served on two Radeon RX 7900 XTX. Then two real production agents — a diagram generator and a ticket analyzer are used to answer the question every synthetic benchmark must face: does this hold up in real life? The answer is yes. And the most beautiful detail: the synthetic benchmark predicted a scaling collapse of the default, and production confirmed it blind.

Counterintuitive bonus: the biggest performance jump came from removing a tuning point — not from adding one.

TL;DR

Running a 35B MoE (≈3B active) on 2× AMD Radeon RX 7900 XTX — consumer cards, RDNA3, 24 GB each — vLLM’s kernel tuning delivered three levels of evidence:

Layer 1 — Synthetic benchmark (controlled A/B, 10 reps/point, 180 measurements):

• +91% throughput at 64 simultaneous requests (344 → 659 tok/s)
• +24% at 32 concurrent (411 → 511 tok/s)
• −28% to −36% on TTFT (time to first token) across the whole range
• TPOT (per‑token latency) cut in half at 64 concurrent (170 ms → 86 ms)
• The default regresses from 32 → 64 concurrent (411 → 345 tok/s): it simply does not scale on this GPU

Layer 2 — Real single‑request agent (diagram generator, 14 types, 58 runs):

• +5.8% mean throughput at concurrency ≈ 1 — modest, and exactly what theory predicts for low load. Full honesty: some diagram types tied or went slightly negative. Low concurrency is not where MoE tuning shines — and the real data proves it.

Layer 3 — Real high‑concurrency agent (491 tickets analyzed in parallel):

• +47.8% throughput and −32% wall‑clock time (718 s → 488 s; 230 seconds saved) at 64 concurrent
• And the crown jewel: the agent reproduced blind the scaling collapse seen in the synthetic test. Untuned, doubling concurrency from 32 → 64 changes nothing (721 s → 718 s, +0%). Tuned, the same tickets finish 14.5% faster.

The cross‑cutting lesson: the MoE tuning gain grows with concurrency — low load ~+6 to +10%, high load +48 to +91%. That is exactly the scenario of serving many agents in parallel. And the counterintuitive engineering lesson: fewer tuning points, but well chosen, won — removing a single mis‑generalized config (bs=32) unlocked the biggest gain.

Why this matters

Mixture‑of‑Experts (MoE) models are the dominant way to scale capacity without paying the full inference cost: Qwen3.6‑35B‑A3B has 35B total parameters but activates only ~3B per token (top‑8 of 256 experts). This makes them ideal for serving agent workloads, where you fire dozens or hundreds of concurrent requests (every reasoning turn, every tool call, every item in a batch).

But there is a detail almost nobody measures: the heart of an MoE is the fused MoE kernel, a grouped GEMM that routes tokens to experts and multiplies matrices. That kernel’s efficiency depends on a tiling configuration (block sizes, number of warps, pipeline stages) that is GPU‑specific. And all the tuning that ships in vLLM is calibrated for datacenter cards (MI300X, H100…). For a consumer GPU like the 7900 XTX, that config simply does not exist and vLLM falls back to a generic one, warning that “performance might be sub‑optimal”.

The question of this work: how much does that “sub‑optimal” cost? And does fixing it survive a real load, or is it a benchmark artifact?

The setup

These are two gamer cards running a model that, without MoE, would be prohibitive. Tuning was done with vLLM’s benchmark_moe.py (Triton kernel autotuner) for dtype=int4_w4a16, E=256, N=256 (intermediate 512 ÷ TP 2), block_shape=[0,128]. The generated artifact is a single JSON:

E=256,N=256,device_name=AMD_Radeon_RX7900XTX,dtype=int4_w4a16.json

The file vLLM was looking for and couldn’t find.

The problem: the missing config

Without the JSON above, the vLLM log leaves the note:

WARNING fused_moe.py: Using default MoE config. Performance might be sub-optimal!
Config file not found at .../E=256,N=256,device_name=AMD_Radeon_RX7900XTX,dtype=int4_w4a16.json

That “default” is a generic tiling heuristic. It works, but it leaves performance on the table, and (as we’ll see) stops scaling exactly in the high‑concurrency regime. Generating the right config is the object of study.

Methodology

Rigor matters: if the difference between “tuned” and “untuned” is smaller than measurement noise, the result is worthless. So the only variable between the two sides was the presence of the tuned config. Everything else, identical.

• Tool: own bench tool, random dataset, input 1024 / output 256 tokens, --ignore-eos (fixed output length), greedy (temperature 0), seed 0.
• Concurrency sweep: 1, 2, 4, 8, 12, 16, 24, 32, 64.
• 10 repetitions per configuration, with 3 s rest between levels.
• Prefix caching OFF on both sides. With random prompts and a fixed seed, enabling it would create spurious cache hits and contaminate the measurement.
• No concurrent real traffic during measurements.

Total: 9 levels × 10 reps × 2 configs = 180 controlled measurements in the synthetic layer alone.

Then, two real production workloads (agents already running in service) were measured with the same model, before and after tuning — not to redo the A/B in a lab, but to answer the generalization question.

Part I — Synthetic results

Generation throughput (tokens/s) — mean of 10 reps

Notice the standard deviations: almost all below 1–4 tok/s across 10 runs. Highly reproducible, the gain is orders of magnitude above the noise.

The gain grows with load

This is the signature of MoE tuning: +10% at low load, +91% at peak load. It’s not a flat improvement, it’s an improvement that accelerates as you stack requests. And there’s a critical point: untuned, throughput regresses from 32 → 64 concurrent (411 → 345). The generic kernel isn’t just slower, it hits a ceiling and falls off beyond ~32 requests on this GPU. Tuned, the curve keeps climbing.

At 64 concurrent, each individual user went from ~5.9 tok/s (untuned) to ~11.6 tok/s (tuned), double the perceived speed per person, at the highest load point. In other words: tuning not only serves more people at once, it serves each of them faster.

Why the gain grows with load (the kernel physics)

Why does MoE tuning yield +10% at concurrency 1 and +91% at 64? The answer lies in the nature of the grouped GEMM at the heart of the MoE.

The fused MoE kernel takes all the tokens in the batch, groups them by destination expert, and runs one matrix multiply per group. The relevant size is M = number of tokens reaching the kernel:

• Low concurrency / decode (small M). Each expert sees very few tokens. The GEMM is “skinny” (few rows) and the bottleneck is memory bandwidth — you spend more time moving the quantized weights than computing. There isn’t much a better tiling can do: the roofline is on the memory side. That's why the gain at conc=1 is modest.
• High concurrency / prefill (large M). Many tokens are batched together; each expert receives a fat lot. The GEMM becomes compute‑bound, and now the tiling config (BLOCK_M/N/K sizes, num_warps, num_stages, GROUP_M) decides the occupancy of the compute units (CUs) and the overlap between compute and data movement. A bad tiling under‑utilizes the GPU and chokes, exactly what we see in the default's 32 → 64 collapse. A good tiling keeps the CUs full and the curve rising.

That’s the mechanism. And it has a subtle consequence that becomes the next chapter: because vLLM picks the config by nearest neighbor on M, a single mis‑tuned point can contaminate an entire regime.

Part II — From the bench to production

Every synthetic benchmark carries a suspicion: is this real, or a pretty number that evaporates when a real load hits? To answer, we measured two agents already running in production with this model, no artificial prompts, no --ignore-eos, no fixed length. Real work.

2a. The single‑request agent: the diagram generator (honesty time)

The first agent generates documentation diagrams from a specification — 14 different types (sequence, class, c4-context, archimate, mindmap, wbs, …), one request at a time (concurrency ≈ 1). It is the least favorable scenario for MoE tuning, and it was chosen on purpose: if tuning were snake oil, this is where it would show.

And here is the scientific honesty that lends credibility to the rest: not every type improved. wbs jumped +28.8% and sequence +11.5%, but mindmap came in at −5.5% and object at −3.2%. Concurrency 1 is a regime of high noise and low headroom, decode is bounded by memory bandwidth, not by the MoE kernel, so tiling has little to contribute and natural run‑to‑run variation dominates.

This is not a disappointment, it’s a confirmation. Theory says: MoE tuning helps little at low concurrency. The synthetic test at conc=1 gave +10.6%, the real agent at conc≈1 gave +5.8%. Both modest, both in the right direction. If the real agent had given +90% at concurrency 1, that would be reason to distrust the experiment.

2b. The high‑concurrency agent: ticket analysis (impact time)

The second agent performs ticket analysis and parallelizes the work across 16, 32, or 64 concurrent calls. This is the regime MoE tuning was made for and where theory predicts the biggest gain. Two batches were measured: a small one (31 tickets) and a large one (491 tickets).

At 64 concurrent, on the real batch of 491 tickets, tuning cut 230 seconds, almost 4 minutes, off a ~12‑minute job, and raised throughput by nearly 50%. This isn’t a benchmark: it’s wall‑clock time a team waited less, on work it was already doing.

The cross‑validation: the predicted collapse is confirmed

Here is the result I’m proudest of, because it wasn’t planned, it emerged from the data.

Back in Part I, the synthetic benchmark revealed that the default does not scale: doubling concurrency from 32 → 64, throughput regresses (−16.1%). That was a lab prediction. The natural question: would this happen with a real load?

The ticket agent answered, without knowing it was being asked. Look at what happens when you double concurrency from 32 → 64 on the 491‑ticket batch:

The reading is unambiguous:

• Untuned, doubling concurrency bought absolutely nothing. In the synthetic test throughput even regressed; in production it was pinned at zero gain (718 s with 64 calls is statistically equal to 721 s with 32). The generic kernel saturated, you’re paying for double the concurrency to get the same result.
• Tuned, concurrency turned into real speed in both worlds: +28.8% synthetic, +17.6% production.

A synthetic micro‑benchmark predicted a scaling behavior, and a fully independent production job confirmed it. This is the strongest validation a performance study can have: the lab number didn’t evaporate, it showed up in the end‑of‑month bill.

One tuning, three workloads

Bringing the three layers of evidence into a single image, the same tuned config, measured across three workloads with increasing concurrency:

The message is single and consistent: the value of MoE tuning is an increasing function of concurrency. If you serve one request at a time, you gain little. If you serve a fleet of agents in parallel, which is where the world is heading, the gain is transformative.

Business translation

Throughput is abstract; let’s convert it into units that pay bills. On the 491‑ticket batch at 64 concurrent:

In other words: the same two consumer cards process ~1,160 more tickets per hour after tuning, without changing a line of the model, without buying a new GPU, without changing the prompt. It’s pure efficiency unlocked by a single kernel‑config JSON file.

And because the gain grows with load, it translates directly into serving density: more simultaneous agents per GPU before you need to scale horizontally. For anyone running self‑hosted inference, that’s the difference between buying the next machine now or a year from now.

Statistical rigor

• 180 controlled measurements in the synthetic layer (10 reps × 9 levels × 2 configs).
• Typical standard deviation < 1% of the mean → the gain sits far above the noise. At 64 concurrent, σ = 0.75–0.80 tok/s over means of 345 and 659 — the distributions don’t come close to touching.
• One contaminated repetition (accidental GPU use during measurement) was detected and redone individually, without discarding the other 9, preserving the sample size.
• For the production workloads, the comparison is paired: same agent, same model, same inputs, same ticket count; the only variable is the kernel config.

Honesty about the real workloads: they have a smaller n than the synthetic test (one to four runs per point, not ten) and were not instrumented for a lab A/B, they are production measurements. So they serve as generalization validation, not as the primary source of the gain numbers. The strength of the argument comes from the convergence of the three layers, not from any one in isolation.

Reproducibility

It’s my work! Call me and i can help you!

Limitations and scientific honesty

A performance study is only trustworthy if it also says where it proves nothing:

1. One model, one GPU, one quantization. The numbers hold for Qwen3.6‑35B‑A3B AWQ int4 on 2× RX 7900 XTX. The direction (tuning helps more under load) should generalize; the magnitudes don’t transfer to other hardware/models without re‑measuring.
2. Production workloads have small n. They are generalization validation, not a lab A/B. They confirm the synthetic signal; they don’t replace it.
3. 64 concurrent is the tested limit. We don’t know where the tuned build also saturates, probably higher, but it wasn’t measured.
4. Single‑stream decode isn’t the strong suit. At low concurrency the bottleneck is memory bandwidth + attention, not the MoE, hence the modest gain, and hence speculative decoding on the next‑steps list.
5. Per‑task variability. In the diagram agent, some types regressed at conc=1. We report this openly rather than hide it, it’s the lowest‑headroom regime.
6. This test was designed for individual users, micro, small, and at most medium-sized businesses. This is the target audience for whom the tested competitor makes sense and can handle numerous case studies!

Next steps

• Prefix caching for agent workloads (context that grows each turn), potential to drastically cut TTFT in multi‑turn conversations.
• Speculative decoding to speed up the latency of a single request (single‑stream decode is bound by memory bandwidth + attention, not by the MoE, exactly what explains the diagram agent’s modest gain).
• A dedicated prefill config (large batch) to squeeze TTFT even further.
• Extend the sweep beyond 64 concurrent to find where the tuned build also saturates.
• Repeat the controlled A/B inside production (n=10) to promote the real workloads from “validation” to “primary source”.

Appendix: full data tables

Stack: vLLM · ROCm · AMD RDNA3 · Triton · Qwen3 MoE · AWQ