FSDP vs DeepSpeed

Romeo Kienzler in collaboration with Mark Sosman — IBM Research & ETH Zürich

Training today’s foundation models is not a matter of if you distribute across GPUs — it’s how well your system handles the explosion of parameters, activations, optimizer states, and communication overhead. As model sizes surpass the memory capacity of even the latest GPUs, researchers rely on specialized sharding frameworks to squeeze every last bit of performance out of their clusters.

Two open-source frameworks dominate large-scale training workflows today:

• Meta’s Fully Sharded Data Parallel (FSDP) — now a first-class PyTorch 2.x citizen
• Microsoft’s DeepSpeed — built around the influential ZeRO optimization family

Both promise to make multi-billion-parameter training feasible — but they achieve that goal with very different trade-offs.

In collaboration with IBM Research Zurich and ETH Zürich, we benchmarked both frameworks under realistic multi-node conditions using Vision Transformers. This article presents the results, but more importantly:

it explains why the frameworks behave differently, how they scale, and what that means as we march toward trillion-parameter models.

Why Scaling Large Models Is Harder Than It Looks

Every time you double model size, you don’t just double memory — you double:

• parameters
• gradients
• optimizer states
• activation memory
• bandwidth requirements
• synchronization costs

In a transformer, activation memory alone can exceed parameter memory. And every additional GPU introduces more communication edges: latency grows, bandwidth saturates, and synchronization penalties accumulate.

This is why naive data parallelism collapses at scale.

Frameworks like FSDP and DeepSpeed exist to shard all this state across GPUs, turning a fundamentally memory-bound problem into a communication-bound one. And at scale, how you communicate determines who wins.

Experimental Setup

The experiments ran on IBM Power System AC922 nodes:

• 6× NVIDIA V100 (16GB HBM2) per node
• NVLink (intra-node)
• InfiniBand HDR100 (inter-node)
• PyTorch 2.x with NCCL backend

We trained two Vision Transformers on Flood-101:

• ViT-Large (~304M params) (https://huggingface.co/ibm-nasa-geospatial/Prithvi-EO-2.0-300M)
• ViT-Huge (~631M params) (https://huggingface.co/ibm-nasa-geospatial/Prithvi-EO-2.0-600M)

Both FP16 and FP32 were tested. We evaluated four configurations:

Experiments used weak scaling — per-GPU batch size constant as GPU count increased.

This reflects real-world usage, where throughput must scale without breaking memory limits.

Results:

1. FSDP is simply faster — often significantly faster

Across the board:

FSDP FULL_SHARD ran up to ~5× faster per iteration than DeepSpeed ZeRO-3.

Why such a gap?

• FSDP is implemented inside PyTorch, avoiding Python-level indirection
• Runtime graph transformations in PyTorch 2.x reduce context switches
• FSDP uses highly optimized parameter flattening + fused reduce-scatter paths
• DeepSpeed’s Python/CPP hybrid architecture incurs additional orchestration overhead

In summary: FSDP wastes fewer cycles “being a framework,” and more cycles executing kernels.

2. DeepSpeed scales more smoothly as model size increases

DeepSpeed’s slowdown from ViT-Large → ViT-Huge: ~1.26× FSDP’s slowdown: ~1.35×

This subtly matters. ZeRO-3’s multi-stage partitioning scales favorably as parameter count grows — the more you shard, the less you replicate.

This is why organizations training 70B+ models often favor DeepSpeed: performance flattens because larger parameter sets amortize communication better.

FSDP excels in the mid range (100M–1B). DeepSpeed opens its advantage later (10B+), where memory savings become more important than raw iteration speed.

3. Mixed precision helps both — but DeepSpeed can pay an extra tax

Mixed precision (FP16/BF16):

• reduces memory
• speeds up kernels
• reduces all-reduce payload

But DeepSpeed ZeRO-3 sometimes exhibits FP32 master weight synchronization overhead, especially when optimizer steps require upcasting.

FSDP evolved rapidly here: PyTorch 2.x’s native mixed-precision stack integrates with FSDP’s sharding logic, reducing redundant casting and synchronizations.

4. Bigger micro-batches = slower iteration (as expected)

Under weak scaling increasing micro-batch size from 4 to16 equals 25–30% longer iteration times

This reflects the usual transformer reality:

• activation memory dominates
• larger batches amplify all-reduce volume
• pipeline stalls become more visible

No framework can solve this problem — but the penalty is smaller with FSDP due to more efficient reduce-scatter patterns.

Why FSDP Is Faster (The Architecture Advantage)

FSDP benefits from several inherent design choices:

1. Native integration in PyTorch

No wrapping. No second execution engine. FSDP manipulates PyTorch’s autograd graph directly.

2. Fused communication patterns

FSDP leverages:

• reduce-scatter + all-gather fusion
• flat parameter tensors
• bucketed communication

DeepSpeed performs similar operations but with more cross-layer orchestration overhead.

3. Less runtime bookkeeping

DeepSpeed must track partition ownership, scheduler steps, state transitions, optimizer partition metadata — across Python and C++.

FSDP stores sharding logic in optimized C++ kernels and autograd hooks.

Why DeepSpeed Still Matters (Especially at >10B params)

DeepSpeed shines when you scale far beyond what these experiments tested.

1. Mature ZeRO-3 and ZeRO-Infinity

ZeRO-Infinity (elastic sharding + NVMe offloading) makes IO-bound training possible for insane model sizes — useful on low-memory GPUs or huge clusters.

2. Optimized CPU/NVMe offload

DeepSpeed pioneered this. FSDP has caught up, but DeepSpeed remains more feature-rich for hybrid CPU/GPU memory architectures.

3. Industry adoption in LLM training

Many GPT-like systems still rely on DeepSpeed:

• ZeRO-3 + pipeline parallelism
• Activation checkpointing
• Load balancing for huge layers

Btw: DeepSpeed integrates nicely with Megatron-LM and tensor-parallel libraries.

Summary

If your model is under ~1B parameters:

FSDP is the clear winner. It is faster, simpler, integrates beautifully with PyTorch 2.x, and removes layers of complexity.

If your model targets 10B–200B parameters:

DeepSpeed becomes competitive and often preferable once sharding depth, offloading, and extreme memory savings dominate the equation.

If you’re planning >200B or trillion-scale:

You will almost certainly use:

• DeepSpeed ZeRO-3 or ZeRO-Infinity
• Tensor + pipeline parallelism
• Custom fused kernels
• Model/tensor slicing frameworks

In other words: Above a certain size, no single framework is enough. You stack them.

The Future: Beyond FSDP and DeepSpeed

Several trends are reshaping large-model training:

1. GPU interconnect bandwidth is becoming the real bottleneck

On V100/NVLink/IB we already see this. On A100/H100 with NVSwitch it’s better, but:

Training speed increasingly depends on network topology, not GPU FLOPs.

2. Compiler-level sharding is coming

PyTorch is moving toward Ahead-of-Time graph partitioning: compiler decides how to shard parameters & activations across devices.

3. Activation recomputation will become universal

Large models cannot fit activations at all. FSDP & DeepSpeed already support this, but compilers will fuse it automatically.

4. Hybrid parallelism is now unavoidable

Everyone at scale uses combinations of:

• tensor parallelism
• pipeline parallelism
• sequence parallelism
• expert parallelism (Mixture of Experts, MoE)
• sharding (FSDP/ZeRO)

The future is systems that blend these automatically.

Final Verdict

FSDP wins on raw throughput and integration. DeepSpeed wins on scaling and extreme memory savings.

Full code and thesis: https://github.com/marksosman/ParallelFrameworkComparison

Tags: #PyTorch #DeepSpeed #FSDP #AIInfrastructure #FoundationModels #DistributedTraining