Optimizing Distributed Training: Understanding Data, Pipeline, and Tensor Parallelism

Introduction