One Slow DDP Rank Can Hold Back Your Whole PyTorch Job

A PyTorch DDP job can be slow without looking broken. No crash. No OOM. All GPUs are doing something. The training loop just takes longer than it should.

That is what makes DDP performance issues annoying. From the outside, it looks like a cluster-wide slowdown. Inside the run, the cause is often much more specific: one rank arrives late, and the rest of the job pays for it.

The useful debugging question is not only: Why is DDP slow? It is: Which rank is slow, and which phase is slowing it down?

That framing matters because the fix changes completely depending on the answer.

The Experiment

I ran a controlled DDP experiment with TraceML on:

• 2 nodes on AWS
• 1 NVIDIA Tesla T4 GPU per node
• PyTorch DDP
• 1 process / rank per node
• TraceML summary mode
• Synthetic precomputed tensors
• Compute-heavy MLP baseline: batch size 128, hidden dim 4096, depth 3

The same training loop runs in three modes:

1. A balanced baseline
2. An input straggler, where one rank has a slower DataLoader path
3. A compute straggler, where one rank does extra optimizer-side GPU work

This is not meant to be a model benchmark. It is a debugging benchmark. The point is to change one part of the system and see whether the runtime summary points to the right rank and phase.

You do not need two nodes to try the demo. The exact measured numbers below came from the two-node T4 setup, but the same script can also run on one machine with two local DDP ranks using --nproc-per-node=2. That is the quick way to see the diagnosis print before running it on real multi-node infrastructure.

On two ranks, you can still reason about this by hand. On 64 ranks across multiple nodes, you cannot. That is where a per-rank, per-phase summary becomes necessary instead of just convenient.

Baseline: Both Ranks Look the Same

First, the balanced run:

Diagnosis: COMPUTE-BOUND

Step time: 124.6 / 124.6 ms
Input:       1.4 / 1.4 ms
Compute:   122.4 / 122.4 ms

The median / worst values are basically identical. Input loading is small. Compute dominates the step. This is the control run: both ranks are doing similar work.

Case 1: Input Rank Straggler

Then I slowed the input path on rank 0. TraceML reported:

Diagnosis: INPUT STRAGGLER

Rank r0 dataloader: 201.6 ms
Rank r1 dataloader:   1.4 ms

This is a common distributed-training trap. The model is not the first thing to inspect. CUDA kernels are not the first thing to inspect. Rank 0 is simply getting batches much later than rank 1.

In a real job, this could come from:

• uneven data shards
• slow storage on one host
• rank-local tokenization or collation
• file locality problems
• CPU contention on one node

If you only look at average GPU utilization, this stays vague. If you compare ranks and phases, it becomes specific: rank 0 has an input-path problem.

Case 2: Compute Rank Straggler

Then I made rank 0 do extra optimizer-side GPU work. TraceML reported:

Diagnosis: COMPUTE STRAGGLER

Rank r0 optimizer: 33.1 ms
Rank r1 optimizer: 14.5 ms

This is a different problem. The DataLoader path is not the first suspect anymore. The skew is coming from compute-side work on one rank.

In a real job, I would now inspect:

• uneven input shapes
• rank-local branches
• optimizer behavior
• hooks or callbacks
• model code that only runs on one rank

One subtle detail: DDP can make a slowdown on one rank show up later during synchronization on another rank. So the visible wait is not always the original cause. A useful first-pass diagnosis needs to ask: which rank had extra work, and in which phase?

Try the 60-Second Local Version

If you just want to see the pattern, run the demo on one machine with two local DDP ranks. The pip install gives you the TraceML CLI. The clone is only needed for the runnable demo script used below.

# Install the tool
python -m pip install traceml-ai

# Get the example scripts (only needed to run the demo)
git clone https://github.com/traceopt-ai/traceml
cd traceml

Balanced baseline:

traceml run examples/ddp_rank_straggler_demo.py --mode=summary --nproc-per-node=2 --run-name ddp_balanced --args --scenario balanced

Input straggler:

traceml run examples/ddp_rank_straggler_demo.py --mode=summary --nproc-per-node=2 --run-name ddp_input_straggler --args --scenario input-straggler --straggler-rank 0 --input-sleep-ms 200

Compute straggler:

traceml run examples/ddp_rank_straggler_demo.py --mode=summary --nproc-per-node=2 --run-name ddp_compute_straggler --args --scenario compute-straggler --straggler-rank 0 --compute-extra-matmuls 8This local run is not the same as a real two-node cluster, but it is enough to reproduce the rank-skew fingerprint and understand what TraceML is reporting.

This local run is not the same as a real two-node cluster, but it is enough to reproduce the rank-skew fingerprint and understand what TraceML is reporting.

Reproduce the Two-Node Setup

For the measured run above, I used two nodes with one GPU each. Use the private IP of node 0 as MASTER_ADDR. Install and clone on both nodes:

python -m pip install traceml-ai
git clone https://github.com/traceopt-ai/traceml
cd traceml

Balanced run:

# Node 0
traceml run examples/ddp_rank_straggler_demo.py --mode=summary --nnodes=2 --nproc-per-node=1 --node-rank=0 --master-addr <NODE_0_PRIVATE_IP> --master-port 29546 --run-name ddp_balanced --args --scenario balanced

# Node 1
traceml run examples/ddp_rank_straggler_demo.py --mode=summary --nnodes=2 --nproc-per-node=1 --node-rank=1 --master-addr <NODE_0_PRIVATE_IP> --master-port 29546 --run-name ddp_balanced --args --scenario balanced

Input straggler run:

# Node 0
traceml run examples/ddp_rank_straggler_demo.py --mode=summary --nnodes=2 --nproc-per-node=1 --node-rank=0 --master-addr <NODE_0_PRIVATE_IP> --master-port 29547 --run-name ddp_input_straggler --args --scenario input-straggler --straggler-rank 0 --input-sleep-ms 200

# Node 1
traceml run examples/ddp_rank_straggler_demo.py --mode=summary --nnodes=2 --nproc-per-node=1 --node-rank=1 --master-addr <NODE_0_PRIVATE_IP> --master-port 29547 --run-name ddp_input_straggler --args --scenario input-straggler --straggler-rank 0 --input-sleep-ms 200

Compute straggler run:

# Node 0
traceml run examples/ddp_rank_straggler_demo.py --mode=summary --nnodes=2 --nproc-per-node=1 --node-rank=0 --master-addr <NODE_0_PRIVATE_IP> --master-port 29548 --run-name ddp_compute_straggler --args --scenario compute-straggler --straggler-rank 0 --compute-extra-matmuls 8

# Node 1
traceml run examples/ddp_rank_straggler_demo.py --mode=summary --nnodes=2 --nproc-per-node=1 --node-rank=1 --master-addr <NODE_0_PRIVATE_IP> --master-port 29548 --run-name ddp_compute_straggler --args --scenario compute-straggler --straggler-rank 0 --compute-extra-matmuls 8

The Takeaway

Two slow DDP runs can look similar from the outside and require completely different fixes. If one rank is slow in input loading, inspect the data path. If one rank is slow in compute, inspect model-side or optimizer-side work.

If you do not compare ranks and phases, you can spend hours optimizing the wrong thing.

Full reproducible guide: https://traceopt.ai/guides/ddp-slow-training-rank-straggler/

TraceML is open source: https://github.com/traceopt-ai/traceml

If this helps, star TraceML on GitHub or share your final_summary.txt in GitHub Discussions.