Fully Sharded Data Parallel (FSDP) — Memory-Efficient Training Tool

As deep learning models continue to grow in size, from millions to billions of parameters, the limitations of traditional parallel…