PyTorch 100B Training: Memory & Parallelism Architecture

Parallelism Is Not About Speed, It’s About Memory Architecture