MoE: Why It Won

Because 80% of a transformer’s compute lives in one place, and once you see that, MoE becomes the only rational response.