ZeRO (Zero Redundancy Optimizer): The Secret Behind Scaling LLM Training

Introduction