Inside MoE Pre-Training: How Sparse Routing Creates an Inductive Bias for Knowledge Storage

While Mixture-of-Experts (MoE) architectures are capturing the spotlight — with industry leaders like DeepSeek, Mixtral, and Qwen heavily…