When Training Takes Too Long: Distributed Data Parallelism

“When training speed limits progress, distribute the data”