Mixture of Experts — How One Model Can Be Many

The architecture behind GPT, Mixtral, and DeepSeek. How sparse activation lets you scale to trillions of parameters without proportional inference cost — and why every frontier lab is betting on it.

👋 Welcome back to From First Principles.

In Issue #4 we covered the six types of LLMs — and one of them was MoE: Mixture of Experts. We described it in one sentence: routes inputs to specialist sub-networks instead of activating the full model every time.

That sentence is accurate. It’s also incomplete.

MoE is one of the most consequential architectural ideas in modern AI. It’s why GPT reportedly has 1.7 trillion parameters but costs similar to a 200B dense model to run. It’s why Mixtral 8×7B outperforms Llama 2 70B at a fraction of the inference cost. It’s why DeepSeek-V3 trained a frontier-quality model for under $6 million — roughly 10–50× cheaper than comparable dense models.

Understanding MoE properly requires understanding three things: the problem it solves, the mechanism it uses, and the engineering challenges it introduces.

This issue covers all three.

The problem: scaling laws and the inference wall

To understand why MoE exists, you need to understand the tension at the heart of modern LLM scaling.

Scaling laws (Kaplan et al., 2020) showed that language model performance improves predictably as you increase parameters, data, and compute. More parameters → better models. This drove the race from GPT-2 (1.5B) to GPT-3 (175B) to GPT-4 (reportedly 1.7T).

But in a standard dense transformer, every parameter participates in every forward pass. Double the parameters, double the inference compute. Scale to 1 trillion parameters and every token prediction requires 1 trillion multiply-accumulate operations.

The economics break. A model 10× larger than GPT-3 costs 10× as much to run per token, for every query, forever. At production usage volume — billions of queries per day — the inference bill becomes structurally prohibitive.

The core question MoE answers: can you have the knowledge of a trillion-parameter model without paying the inference cost of one?

Yes — with sparse activation.

The mechanism: sparse activation

A Mixture of Experts transformer replaces the dense feed-forward network in each transformer block with a set of parallel feed-forward networks — the “experts” — plus a small routing network called the gating network.

Dense transformer block (recap from Issue #1)

Input tokens
  → Self-attention (all tokens attend to all tokens)
  → Feed-forward network (same FFN for every token, always)
  → Output

MoE transformer block

Input tokens
  → Self-attention (identical — attention is unchanged)
  → Gating network: which 2 experts should handle this token?
  → Top-k experts process the token (k=1 or k=2 typically)
  → Weighted combination of expert outputs
  → Output

The self-attention mechanism is completely unchanged. The difference is entirely in the feed-forward layer: instead of one universal FFN, there are N experts (8, 16, 64, or more), and each token is routed to only k of them.

This is sparse activation. A model with 64 experts activates 2 per token — 3% of expert capacity. The remaining 97% sit idle for that token.

The gating network — how routing works

The gating network is a small learned linear layer that takes the token’s hidden state as input and outputs a probability distribution over all experts:

gates = softmax(W_gate · h)

Where W_gate is a learned weight matrix and h is the token’s hidden state. Top-k selection picks the highest-scoring experts. The token is processed by each selected expert independently, and results are combined as a weighted sum:

output = Σ (gate_score_i × Expert_i(h))   for i in top-k experts

The key property: the gating network learns during training. Nobody programs what each expert should specialize in. Specialization emerges from the training objective.

What experts actually learn to specialize in

Without any explicit supervision, experts develop meaningful specializations:

• Some specialize in syntactic patterns (sentence structure, punctuation)
• Some specialize in domain vocabulary (code syntax, medical terms, legal language)
• Some specialize in specific languages
• Some specialize in reasoning patterns vs factual recall

This emergent specialization is why MoE works so well in practice. Routing isn’t arbitrary — it learns to direct each token to the experts most capable of handling it.

The parameter math — why MoE is efficient

Here’s the key calculation.

Replace a dense FFN (d_model × d_ff) with 8 experts of the same size, using top-2 routing.

Dense MoE (8 experts, top-2) Total parameters 1× FFN 8× FFN Active parameters per token 1× FFN 2/8 = 25% of experts = ~2× FFN Inference FLOPs per token 1× ~2× Knowledge capacity 1× 4×

You get 4× more knowledge capacity for 2× the per-token compute cost.

MoE’s value proposition: decouple model capacity (total parameters) from inference cost (active parameters per token).

Real numbers: Mixtral 8×7B

• 8 experts, each a 7B-parameter FFN
• Total parameters: ~46.7B
• Active parameters per token (top-2): ~13B
• Quality: matches or exceeds Llama 2 70B on most benchmarks
• Inference cost: similar to a 13B dense model

70B-class quality. 13B-class inference cost. That’s the MoE advantage in production.

Load balancing: the hard engineering problem

If MoE were simply “route to top-k experts,” it would have become standard much earlier. The reason it remained a research curiosity for years is a severe failure mode: expert collapse.

Without intervention, training converges to a state where a small number of experts receive nearly all the tokens. Popular experts get trained well. Ignored experts stagnate. The model degrades to a few overloaded, well-trained experts and many useless ones.

This happens because of a feedback loop:

1. Slightly better experts get slightly higher gate scores
2. Higher gate scores → more tokens → more gradient signal → better training
3. Better training → even higher gate scores
4. The rich get richer. Unused experts never learn.

The auxiliary load balancing loss

The standard fix: add a load balancing term to the training objective:

L_total = L_language_modeling + α × L_load_balance

The load balancing loss penalizes routing distributions where experts receive unequal numbers of tokens. It pushes the router toward uniform distribution.

The coefficient α is delicate. Too small → collapse still occurs. Too large → routing becomes arbitrary, destroying specialization. Getting this balance right is one of the core engineering challenges in MoE training.

Expert capacity and token dropping

Production MoE systems set an expert capacity — a maximum number of tokens each expert can process per batch. Tokens assigned to a full expert are “dropped” (passed through as-is or handled by a fallback).

This hard limit prevents any expert from being overwhelmed and forces load balance. But dropped tokens lose expert processing — too much dropping degrades quality.

Expert Choice routing

An alternative approach (Zhou et al., 2022): instead of each token choosing its experts, each expert chooses its top-k tokens. This guarantees perfect load balance by construction — every expert processes exactly k tokens per batch.

The tradeoff: some tokens may be selected by fewer experts than others, and routing is no longer purely token-driven. But training stability improves significantly. Expert Choice has become popular in recent large-scale MoE implementations.

Memory: the hidden cost

MoE solves the inference compute problem. It does not solve the memory problem.

All experts must be loaded into memory simultaneously, even though only a fraction activate per forward pass. A model with 8 experts each of size 7B has 46.7B parameters in memory at all times — not 13B.

The implication: MoE models require more GPU memory than their inference compute suggests.

• Mixtral 8×7B at FP16: ~93GB of GPU memory just for weights
• DeepSeek-V3 (671B MoE): 671B parameters in memory, ~37B active per token

This is the primary deployment constraint. You need GPUs large enough to hold the full model, even though you’re only computing with a fraction of it.

Mitigation strategies:

Expert offloading — keep infrequently used experts on CPU RAM, load to GPU on demand. Works for batch inference. Impractical for real-time applications.

Expert quantization — combine with QLoRA techniques (Issue #8) to compress experts to INT4/INT8. Significantly reduces memory footprint at modest quality cost.

Expert parallelism — distribute different experts across different GPUs. Standard approach for training and deploying very large MoE models. Requires careful orchestration of routing decisions across GPUs.

Fine-grained MoE: the frontier direction

Early MoE models used few large experts (8 × large FFN). Recent frontier models have shifted toward fine-grained MoE — many more, much smaller experts.

DeepSeek-V3: 256 experts with top-8 routing. Each expert is smaller than Mixtral’s design, but 256 experts allow much finer-grained specialization. More routing flexibility. Better quality at similar compute cost.

Shared experts: DeepSeek’s architecture introduces “shared experts” — a small number of experts always activated for every token, regardless of routing. These shared experts capture universal patterns (grammar, common reasoning) while routed experts handle specialization. This hybrid design improves quality at modest additional cost.

The direction: more experts, smaller experts, richer routing, shared expert mechanisms. As hardware improves and routing algorithms mature, the sweet spot continues moving toward finer granularity.

MoE vs dense: the honest comparison

MoE is not strictly better than dense. It’s a different set of tradeoffs.

Dense MoE Training cost Lower Higher (load balancing overhead) Inference compute Proportional to total params Decoupled — proportional to active params Memory Proportional to total params Same as total params (all experts in memory) Training stability Well-understood Expert collapse risk, load balance tuning Fine-tuning maturity Mature — LoRA works well More complex — which experts to adapt? Quality per FLOP Lower at scale Higher at scale Deployment simplicity Simple Complex — expert parallelism required

MoE wins when: you need frontier-quality at reasonable inference cost, and have the infrastructure to manage deployment complexity.

Dense wins when: you’re fine-tuning extensively, deploying on limited hardware, or need simple predictable deployment.

This is why both coexist in production. GPT-4 and DeepSeek-V3 use MoE. Llama 3 405B is dense. Different teams, different constraints, different optimization priorities.

Why DeepSeek matters for MoE’s trajectory

DeepSeek-V3’s training cost — under $6M for a frontier-quality model — sent a signal through the industry. The efficiency enablers included MoE architecture, aggressive quantization, FP8 training precision, and careful load balancing.

The takeaway isn’t just cost reduction. It’s architectural trajectory: MoE combined with modern efficiency techniques is the path toward models that are simultaneously larger in capacity and cheaper to run.

The open research questions: 1,024 experts? Hierarchical routing? Dynamic routing depth? Expert specialization signals beyond token-level hidden states?

These will define the next generation of frontier models.

3 practical takeaways

1. Parameter count is misleading for MoE models. When choosing a model for production, “number of parameters” misrepresents inference cost for MoE models. What matters is active parameters per token. A 47B MoE model with 13B active parameters has inference costs closer to a 13B dense model. Benchmark latency and throughput on your actual hardware — don’t trust the parameter headline.

2. Memory is the binding constraint, not compute. “MoE is cheaper to run” is true for compute, not for memory. A 47B MoE model requires more GPU memory than a 13B dense model, even though inference compute is similar. For memory-constrained deployments, the MoE compute advantage partially disappears. Always profile both for your specific setup.

3. Fine-tuning MoE models is more complex. LoRA on a dense model is mature and well-understood. For MoE, the decisions multiply: adapt all experts? Only the routing layer? Only the most-activated experts? Shared experts only? Research is still settling on best practices. For most production teams today, fine-tuning a dense model with LoRA is simpler and more reliable than fine-tuning an MoE model.

TL;DR

MoE replaces the dense FFN in transformer blocks with N expert networks and a gating network that routes each token to the top-k experts. This decouples model capacity (total parameters) from inference cost (active parameters per token).

Key engineering challenges: expert collapse (fixed with load balancing loss), memory requirements (all experts in memory despite sparse activation), and deployment complexity (expert parallelism at scale).

Frontier direction: fine-grained MoE with many small experts, shared mechanisms, and richer routing.

MoE is how you build a trillion-parameter model that doesn’t cost a trillion parameters to run.

🎓 That’s Arc 2 — complete.

Twelve issues. Twelve deep-dives. Two complete arcs.

Arc 2 at a glance:

Topic Core insight 7 RAG Don’t ask the model to remember — give it the information 8 LoRA and QLoRA Train 0.1% of parameters, get most of the quality 9 The Fine-Tuning Decision Prompt first. RAG second. Fine-tune third. 10 Embeddings Meaning is geometry. Similarity is nearest-neighbor search. 11 The Evaluation Problem Benchmarks lie. Build evals that tell you something true. 12 Mixture of Experts Decouple capacity from inference cost.

📬 Arc 3 — what’s coming

The series continues. Candidates:

→ AI Agents — the real architecture (beyond LLM + tools) → Prompt Engineering — what actually works (systematic, not magical) → Vector Databases — how HNSW and ANN search actually work → AI Infrastructure — GPUs, inference optimization, serving at scale → Multimodal Models — how vision and language actually fuse → The Alignment Problem — beyond RLHF, what does safe AI require?

What should Arc 3 open with? Drop it in the comments.

Thank you for reading From First Principles.

→ Subscribe for Arc 3 → Share with someone building production AI systems right now

See you in Arc 3.

— Charan