Understanding Distributed LLM Training: How I Built and Profiled a Multi-GPU Parallelism Sandbox

Demystifying Tensor, Data, and Pipeline Parallelism strategies through a custom-built profiling sandbox environment