Distributed Systems in Machine Learning, and foundations to frontier practice

A systems-level view of distributed training: parallelism, communication, and constraints