MoE: Why It Won

Because 80% of a transformer’s compute lives in one place, and once you see that, MoE becomes the only rational response.

In January 2025, a Chinese AI lab, DeepSeek, released a model that sent nearly a trillion dollars of market value evaporating from American tech stocks in a single day.

The model was DeepSeek V3. What panicked investors was not just that it matched GPT-4 level performance. It was how cheaply it was trained, approximately 5 million dollars, against the hundreds of millions assumed necessary for frontier AI. The economics of the entire industry were apparently wrong.

What made that possible was not a secret. It was not even new. It was an architectural choice that every major lab had already quietly made — an idea first published in 1991 by four researchers, one of whom would later win a Nobel Prize.

That idea is Mixture of Experts. And understanding it is not optional anymore because it is now the engine running inside DeepSeek V3 and V4, Grok 3, Gemini, Mistral, and Gemma, and maybe the closed proprietary models are also using this. Every major frontier model. Open and closed. American, Chinese and European. They all made the same architectural choice. This article explains why, from the mathematics that made it inevitable to the engineering that makes it brutal to implement at scale. First, we need to go back some years, to

1991: the idea that arrived too early

The year is 1991. Neural networks are deeply unfashionable. Geoffrey Hinton, working with Robert Jacobs, Michael Jordan, and Steven Nowlan, publishes a paper that almost nobody reads at the time. The paper name is “Adaptive Mixture of Local Experts”. The core idea is borrowed from cognitive science, the observation that the human brain does not process all information through a single monolithic system. Different regions specialize. Vision happens mostly in the occipital lobe. Language in Broca’s and Wernicke’s areas. Motor control in the cerebellum. There is a division of cognitive labour, and it works.

What if you could build something like that artificially?

The 1991 paper proposed a system with multiple “expert” networks and a “gating” network. The gating network’s job was simple: look at each input and decide which expert should handle it. During training, both the experts and the gate trained together. Experts naturally specialized, they got rewarded when routed inputs they were good at, and penalized otherwise. Over time, each expert carved out its own niche in the input space without being explicitly told what that niche was.

The insight was not just efficiency. It was that specialization emerges from incentive. You do not need to design which expert handles which domain. You need only create the conditions where specialization is the optimal strategy. The rest takes care of itself.

The paper was correct and prescient and largely ignored. Neural networks were about to enter their second winter. The gating networks of 1991 were too expensive to train at any interesting scale. The idea sat in the literature, quiet, for more than two decades.

Three ideas that converged

Between 2010 and 2017, three separate research threads developed that would eventually collide to make modern MoE possible.

• 2012 — Alexnet and the GPU revolution Deep learning becomes computationally feasible on GPUs. This is the prerequisite for everything. Without cheap matrix multiplications on parallel hardware, MoE is interesting but untrainable at scale.
• 2013 — Experts as components, not whole models Eigen, Ranzato, and Ilya Sutskever publish work showing that MoE layers can be embedded inside deep networks rather than being entire standalone systems. This is the key architectural unlock. Suddenly, MoE is not a competing approach to deep learning but a component of it. A layer within a layer.
• 2013–2016: Yoshua Bengio and conditional computation Bengio publishes a series of papers on “conditional computation”, the idea that a network should dynamically activate or deactivate components based on the specific input, rather than always running everything. This is the theoretical framework that makes sparse MoE coherent: you are not skipping computation randomly, you are conditioning it on relevance.
• 2017 — Shazeer et al. and the sparsely-gated MoE layer Noam Shazeer, with co-authors including Geoffrey Hinton and Jeff Dean, publishes "Outrageously Large Neural Networks: the Sparsely-Gated Mixture-of-Experts Layer." They scale the 1991 idea to a 137 billion parameter LSTM, the dominant NLP architecture of the time, by introducing sparsity and top-k gating. This is the paper that proves MoE works at a real scale. It is also the paper that names the primary challenge: load balancing. Not all experts get used equally, and fixing that turns out to be deeply non-trivial.

Geoffrey Hinton appears in both the 1991 paper that invented the concept and the 2017 paper that made it practical. He also won the Nobel Prize in 2024 for his foundational work on neural networks. The man has a remarkable habit of being at the origin point of ideas that take thirty years to matter.

What MoE actually is

Before we get to MoE architecture, there is a misconception to clear up. It is everywhere, and it will confuse everything that follows if you carry it in.

When people first hear “Mixture of Experts,” they imagine something like a team of human specialists — one expert knows medicine, one knows law, one knows physics. The model routes your question to whichever domain matches. It sounds intuitive. I was thinking the same. But it’s not like this.

Experts in MoE models are not specialized in domains like Psychology or Biology. They specialize at a much finer level — syntactic and structural patterns in language. Some experts are good at punctuation. Some at verbs. Some at conjunctions. Some at numbers. Some at code syntax. The specialization is at the word-level, not the topic-level. Calling them “experts” has been widely criticized as misleading, and once you understand what they actually do, you will see why.

This distinction matters because it changes how you think about routing. The question the router is asking is not “is this token about chemistry or law?” It is asking something much more granular: “What kind of linguistic and structural transformation does this token need right now?” A number token in a chemistry paper and a number token in a financial report will often route to the same expert — because both need numerical processing, not because both are “science.” Now, let us build the idea from the ground up.

Instead of routing every input token through every weight in the model, a router examines each token and selects a small set of “expert” subnetworks to process it. The other experts sit idle for that token. The total model capacity, the number of experts, and the total parameter count can be enormous. The active parameter count per token remains small and constant.

The Scaling Of MoE

Here is where the story gets genuinely surprising. Most people who work with transformers assume that the attention mechanism, the famous, much-written-about mechanism, the one in the paper title “Attention Is All You Need”, is the expensive part. It’s not right. And the data proving this is what made MoE not just appealing but economically necessary.

The table above shows where computation actually goes inside a transformer as you scale from 760 million to 175 billion parameters. Read the attention column carefully: 35% at 760M, dropping steadily to 17% at 175B. Now read the FFN column: 44% at 760M, climbing relentlessly to 80% at 175B.

As you scale a transformer, the feed-forward networks eat almost all the compute. Attention shrinks to a rounding error.

The MoE insight is now inescapable: if 80% of your compute is in FFN layers, and different tokens genuinely require different types of thinking, a token in a chemistry derivation, a token in a legal argument, a token in a Python function — then why do the same FFN weights activate for all of them? The chemistry token does not need the weights that learned about legal precedent. The Python token does not need the weights that learned Mandarin grammar. Routing is not just clever. At this scale, it is the only rational response to where the compute actually lives.

From the 2017 paper to modern frontier models

Understanding modern MoE requires following one more evolution: the move from a few large experts to many small ones, and the introduction of shared experts alongside routed ones. These are not incremental tweaks. They are architectural philosophy changes with major performance implications.

The Switch Transformer (2021): k=1 works fine

Google’s Switch Transformer was the first to demonstrate that you could use k=1, meaning route each token to exactly one expert, and still train stably. This was theoretically controversial (less gradient signal for the router) but empirically clear: Switch Transformers achieved a 7x speedup in pre-training over dense T5 models of equivalent quality. The paper also formalised the auxiliary load-balancing loss, a penalty added to training that discourages the router from ignoring most experts.

Mixtral 8x7B (2023): the open-source proof point

Mistral’s Mixtral 8x7B was the moment MoE stopped being a lab curiosity and became a production pattern. It had 46 billion total parameters but activated only 12.9 billion per token, outperforming Llama 2 70B on most benchmarks at roughly six times the inference speed. The recipe was simple: eight experts per FFN layer, two activated per token. The result: dense model quality, fraction of the inference cost.

What “8x7B” actually means When you see the notation “8x7B,” it means eight experts each with approximately 7 billion parameters. The total is roughly 47B. At inference, you activate 2 of 8 experts per token, about 13B parameters. The model knows like a 47B model. It runs like a 13B model. This is the value proposition in one sentence.

DeepSeek V3 (2024): fine-grained experts and auxiliary-loss-free balancing

DeepSeek V3 pushed two innovations simultaneously. First, instead of 8 large experts, they used 256 smaller experts per layer, dramatically increasing combinatorial flexibility. With 8 large experts and top-2 routing, there are 28 possible expert combinations. With 256 small experts and top-8 routing, there are over four billion. This granularity lets the router compose expertise with far more precision.

Second, DeepSeek solved the load-balancing problem more elegantly than auxiliary losses. Instead of penalising imbalance during training (which fights against the model’s natural routing preferences), they monitored expert load at every step and adjusted each expert’s selection bias dynamically. Experts getting too many tokens got their bias decreased, making them slightly less likely to be selected. Experts getting too few got their bias increased. No quality degradation from forced routing. No training instability from competing loss terms. Just adaptive bias adjustment. Elegant.

DeepSeek V4 (2026): 1.6 trillion parameters, 49 billion active

In April 2026, DeepSeek released V4, 1.6 trillion total parameters, trained on over 32 trillion tokens, with only 49 billion parameters active per token. To put that in perspective: you get the knowledge capacity of a 1.6 trillion parameter model at the inference cost of a 49 billion parameter model. The ratio is roughly 33:1.

DeepSeek V4

V4 introduced three architectural innovations to handle trillion-parameter stability: a hybrid attention mechanism that reduces KV-cache memory by 90% — enabling native one-million-token context windows, Manifold-Constrained Hyper-Connections (to prevent signal amplification instabilities across very deep MoE stacks). One more change they have made is using the Muon optimiser.

V4 was also trained entirely on Huawei Ascend 950PR chips. Not NVIDIA. This is a signal that MoE architectures are mature enough to be hardware-agnostic; the algorithms no longer require CUDA monoculture.

The full landscape in 2026

Notice what is missing from this table: any major frontier model that is purely dense. There are none. The transition is complete. Every major AI lab, open and closed, American and Chinese and European, has converged on MoE as the architecture for frontier scale. Still, we don’t know about most of the frontier models' architecture.

MoE has Some Challenges

Everything above makes MoE sound like a straightforward win. Bigger total capacity. Lower inference cost. Emergent specialization. So why did it take until 2023 for MoE to dominate the field? Because the engineering challenges are genuinely brutal, and most of them are not obvious until you try to run these systems at scale.

Expert collapse: the training failure mode

Without intervention, MoE models catastrophically fail during training. The router starts with random weights. By random chance, some experts produce slightly better outputs in the first few batches. The router learns to prefer them. Those experts receive more gradient updates. They improve faster. The router learns to prefer them even more. The other experts receive almost no training signal. After thousands of steps, you have a model with 256 “experts” but only 3 that actually work — effectively a waste of 98% of your parameters.

Solutions include auxiliary load-balancing losses (a penalty for unequal expert utilisation), token capacity limits (hard caps on how many tokens any single expert can receive per batch), and DeepSeek’s dynamic bias adjustment (monitoring load and adjusting routing probabilities continuously). None of these is trivial to implement correctly. Getting MoE training stable is, as of 2026, still more art than science.

The memory paradox

You save compute by only activating a fraction of the model per token. But you still have to load the entire model into memory. All 256 experts. All 671 billion parameters. All 1.3 terabytes of weights.

DeepSeek V3 at full precision requires over 1.3 TB of GPU VRAM in full precision(fp32). That means multi-GPU, multi-node deployments with fast interconnects. The routing process then creates an “all-to-all” communication pattern; tokens must be physically moved across the network to reach the GPUs where their designated experts are stored. At a trillion-parameter scale, this communication overhead can exceed the actual computation time. Recent measurements suggest that in naive EP implementations, communication consumes 77% of total MoE layer processing time. The math you are doing to generate intelligence is taking less time than moving the tokens around to do it.

Training instability at scale

The hard switching of experts, a token goes to Expert A or Expert B, not a blend, creates theoretical discontinuities in the loss landscape. Small changes in routing decisions cause sudden changes in the gradient flow. At the trillion-parameter scale, these discontinuities can cause training loss spikes that are difficult to recover from. Modern solutions include selective mixed precision (keeping routing operations in float32 while running the rest in bfloat16 or FP8), gradient clipping tuned specifically for MoE dynamics, and SwiGLU activation clamping to prevent extreme values.

The real engineering lesson

DeepSeek’s achievement in training V3 for $5 million was not just an algorithmic win. It was an engineering discipline win. Their auxiliary-loss-free balancing, FP8 training throughout, and multi-head latent attention for KV-cache compression were each individually significant. Together they made frontier MoE training economically feasible without a hundred-million-dollar compute budget. The gap between “MoE theoretically works” and “MoE is deployable in production” was closed by engineering, not mathematics.

The Future Of MoE

The field is not done with MoE. It has only just started using it at scale. The research questions that are actively being explored right now point toward even more radical forms of sparsity and specialisation.

1. Hierarchical routing Instead of flat selection from 256 experts, future models may use tree-structured routing: first to a “category expert” (Math, Code, Language), then to a specialist within that category. More interpretable. Potentially faster routing. DeepSeek V4’s 384-expert architecture is pushing in this direction.
2. Cross-layer expert sharing Current MoE models have separate expert pools per layer. Emerging research explores a global pool accessible by any layer — the same expert can be called by layer 12 and layer 48. This would dramatically increase parameter reuse and allow specialised knowledge to influence computation at multiple depths.
3. MoE for multimodality Gemini’s architecture uses expert routing across modalities — different experts for vision, audio, text, and video tokens. As models become more genuinely multimodal, MoE provides a natural mechanism for modality-specific specialisation without sacrificing cross-modal reasoning.
4. Hardware co-design NVIDIA’s Blackwell and Huawei’s Ascend 950PR both include dedicated engines for sparse expert routing and all-to-all communication. The hardware is now being designed around MoE’s communication patterns rather than the other way around. This will dramatically reduce the memory and bandwidth overhead that currently limits MoE efficiency.

This post was not written to cover how the Mixtral of Experts architecture works. I will have a specific dedicated blog post for this explanation. I wanted to give an overview of what I have learned about the MoE architecture and why companies are choosing this now.

Further readings:

1. https://arxiv.org/abs/2401.04088: Mixtral Of Experts
2. https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf: DeepSeek V4 Paper
3. https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-mixture-of-experts: beautiful Blog visualizing MoE
4. https://cameronrwolfe.substack.com/p/moe-llms: Another piece of good writing on MoE.