Layer Normalization: The Quiet Idea That Made Deep Learning Stable

By the mid-2010s, deep learning had already demonstrated remarkable success in vision and speech. Yet training deep neural networks…