DeepSpeed: Training Foundation Models Beyond a Single GPU

Ever wonder how Foundation Models — too large to fit into a single GPU — are trained?