Your GPUs are fast. Your paths between them might not be.

Identical GPUs, training loop, but different throughput — check the physical paths data takes between cards.

Inter GPU Communication

At the scale most people mean by LLM training, you rarely stop at one GPU, and you often do not stop at one server either: some of your ranks sit on the same node, and the rest sit on other nodes in the cluster. e.g. In a “16-GPU training,” ranks have IDs 0 … 15 and Ranks 0–7 could be on node 1 and Ranks 8–15 on node 2. Inside a node, NVIDIA GPUs can talk over NVLink (a dedicated, very high-bandwidth GPU-to-GPU link) or fall back to paths that still involve PCIe (the same kind of bus that also connects CPUs, NICs, and switches). NVLink is built for neighbor GPUs shuffling big tensors; PCIe is general-purpose and often the slower or more contended hop when NVLink does not directly connect a pair. Between nodes, traffic usually leaves the box through NICs and the datacenter network — another step, another place where latency and bandwidth bite.

Under the hood, training a large language model on multiple GPUs is not only a math problem — it is a shipping problem. Each device(gpu) holds its own memory; a backward pass produces per-GPU pieces of the gradient , but the optimizer step needs everyone to agree on the same combined update. So training constantly runs collectives — fixed patterns like “sum these tensors and give the result to every rank” (all-reduce) — not ad-hoc point-to-point chatter. Those patterns are fiddly, bandwidth-hungry, and easy to get wrong, which is exactly why frameworks do not leave them to application code.

PyTorch surfaces that work as **torch.distributed: you describe what collective you need; the backend carries it out on the wire. On NVIDIA GPUs that backend is almost always `nccl`, i.e. NCCL (NVIDIA Collective Communications Library). NCCL is the layer that turns a call like all-reduce into a real plan: which ranks exchange data, in what order, and over NVLink, PCIe, NICs,** or other paths the machine exposes (including some switch-assisted shortcuts when available).

Inter-node and messy intra-node paths are where training often stalls: one slow collective waits on one rank, and the whole training step waits with it. Fixing that is not only “buy faster chips” — it is routing and scheduling the bytes. In practice you end up with profiling tools like Nsight Systems (or similar) and NCCL logs, but those timelines only help if you know what pattern you expected on the wire!

The rest of this article builds that mental picture, then ties it to the names of communication protocol that NCCL uses when it picks a plan.

Topology awareness

Your tensor cores can be booked solid, but if the schedule for a collective (like all-reduce described above ) keeps pushing traffic through the slowest path between GPUs (a weak PCIe hop, a busy inter-switch link, a long NIC leg — not “a slow machine,” but a slow hop on the map), the step still feels “bandwidth starved.” Topology awareness means NCCL notices that map and tries to spend more bytes on the fast links.

Topology = how your machines are actually wired. Topology-aware scheduling means NCCL tries to move most data over the fastest links and only use slow or narrow hops when it has to — instead of treating every path as if it were the same speed.

Note that, if you have a big NVSwitch box (Fully NVLinked rack server) where everything talks to everything at ludicrous speed, topology still exists — it is just less painful. Whereas on a mixed PCIe / NUMA / multi-node setups, it starts to matter a lot due to large variation in network bandwidth capabilities. Same code, different box, different communication plan.

Commands to view topology

nvidia-smi topo -m to get a basic GPU interconnect view on any nvidia gpu

nvidia-smi nvlinkNVSwitch- and NVLink-based systems

Imagine a building with two fast hallways and one slow door

Picture four roommates in two pairs of connected rooms:

• GPU 0 and 1 can shout at each other through a wide open doorway (think NVLink — very fast).
• GPU 2 and 3 have the same between them.
• But any conversation between the {0,1} side and the {2,3} side has to go through a narrow hallway (PCIe — fine, but not the same league).

Fig 1: Two fast pairs, one slower bridge

The ring is fair — which is exactly the problem

One classic way to all-reduce is a ring: data travels 0 → 1 → 2 → 3 → 0 like a relay baton. Everyone does their share. Lovely — if every leg of the relay feels the same.

But in our toy building, two legs are “wide doorway” and other two legs are “narrow hallway.” The ring does not care. It still uses the narrow hallway again and again. Your average speed is not “how fast NVLink feels”; it is “how fast the worst part of the lap allows.”

Fig 2:One ring, two slow legs every lap

So you can have tons of NVLink and still feel underfed if the schedule keeps marching data through the skinny door for no good reason.

Remember, nothing is “wrong” with rings in general — they are a workhorse. The point for LLM-scale training is narrower: when the machine is not link-symmetric, a schedule that ignores the map can leave performance on the table even if the GPUs themselves are monsters.

The topology-aware instinct

Once you see the building, the fix is almost boring: do as much combining and copying as you can inside each fast pair, push only the smaller cross-pair summaries through the slow connector, then fan the full result back inside each pair again. Same four GPUs, same collective — less time begging the skinny hallway.

1. Merge work inside each fast pair (lots of chatter on NVLink — quick).
2. Send only the smaller “we already combined our side” package through the slow door once (or as little as possible).
3. Copy the final answer back inside each pair on NVLink again.

Here is the illustration in below figure. Same four GPUs. Different plan. The orange line still exists — we just stop treating it like a highway.

Fig 3: Reduce in pairs, swap summaries, fan out

Below figure shows the same thing in ring language: two tiny rings (one per pair) plus one orange bridge between them — instead of one big ring that keeps crossing the bridge for sport.

Fig 4: Two small rings, one cross-link

Reality check: real NCCL builds real graphs; your trace might not look exactly like these figures. Treat the drawings as intuition, not a promise of kernel names. If you profile and see a burst of “inside the node” work then a thinner “across the slow hop” phase, you are seeing the same idea in the wild.

From “we measured the wires” to “Ring, Tree, NVLSTree…”?

So far this has been geometry — the pictures were about where the slow hop is. Below figure is rest of the story: what NCCL does after it knows that map. NCCL’s runtime problem is: given that geometry and your tensor sizes, which implementation wins? It builds a link graph, runs internal models, and picks a communication algorithm from a toolbox. Here is that pipeline in one glance consisting of 3 steps: measure links → build a mental map → score options.

Fig 5: Topology in, algorithms out

1 — Topology graph (input). Before it can prefer “fast pairs first,” NCCL has to learn the topology: which GPUs can talk to which, and over what kind of link (NVLink vs PCIe vs NIC paths, rough speeds, how many hops).

2 — Search + models (middle box). When PyTorch asks for a collective (e.g. all-reduce) at a given size, NCCL does not run ten implementations to see which wins. It uses internal models (built from theory plus a lot of benchmarking over the years) to estimate how long Ring would take on this graph, how long Tree would take, whether NVLS is even legal on this SKU, and so on. Think of it as a spreadsheet in disguise: rows are candidate plans, columns are “cost on this topology / this message size.”

3 — Algorithms (output). The output is the label on the winning row. “Ring,” “Tree,” “NVLSTree,” “PAT,” CollNet variants — those words are names of whole game plans (who sends what, in what order) that NCCL might install for that call.

Generally you dont pick this communication algorithm by hand; NCCL does. But if you override with [[NCCL_ALGO](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html)], you are basically forcing a different label in the third step — do that only when you know why. We will cover this in more detail in a later section.

What each label in the third step roughly means in practice:

• Ring — Each GPU only talks to its two neighbors on a fixed circle. Chunks of the tensor lap the ring until everyone has the full reduced value (same idea as Fig 2, just drawn as a square loop here). The upside is simplicity and predictable bandwidth when every edge of the circle is equally fast. The downside is exactly our hallway story: one bad edge on the circle caps the whole lap, because every step still has to cross that edge eventually.
• Tree — GPUs are arranged in layers. In the “up” phase, partial results merge toward a small set of roots (fewer and fewer GPUs hold bigger partials). In the “down” phase, the final value fans out along the same edges. With many GPUs, the number of steps often grows like log(N) instead of N steps around a ring — but the tree is still drawn on top of the real PCIe/NVLink/NIC graph, so a bad root placement or a slow edge near the root still hurts.
• Collnet (chain / direct variants in NCCL) — Think multi-node first. Instead of every pair of distant GPUs trying to talk directly across many NIC hops, CollNet-style plans try to use a collective-network plugin (switch + software path) so part of the reduction or aggregation happens while data is still “in the network” between servers. Your training script still says all_reduce; NCCL swaps in a different pattern of sends when the cluster exposes that path. If you only ever train on one server, you may never touch this family.
• NVLS / NVLSTree — These require specific NVIDIA hardware where part of a collective can be offloaded into the NVSwitch / NVLink fabric (NVLink SHARP–class features in documentation). If you read “CollTree” in a blog, the spelling that actually exists in [NCCL_ALGO] is **NVLSTree**.
• PAT (Parallel Aggregated Tree, NCCL 2.23+) — Same Fig 5 pipeline, but aimed mainly at all-gather and reduce-scatter collectives, when the job is very large. See the PAT write-up for the real design.

Communication Algorithm

Setting [NCCL_ALGO](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html)

NCCL’s auto-picker is still a guess: it compares Ring, Tree, and the rest using timings it predicts from your topology and tensor size. If two plans both look like “about 0.8 ms” on paper, either one might win — tiny changes in NCCL version or message size can tip the scale. That feels random from the outside; it is really “two meals that cost the same on the menu.”

So we need to do two things here, firstly determine if we should try to experiment with changing this default value and second what value to set ?

Inspect if problem exists

• The “Communication Bar”: In a PyTorch Profiler trace, look for ncclAllReduce or ncclAllGather. If these blocks are wider than your compute kernels (like CUDNN_CONVOLUTION), you are "Communication Bound."
• The “Bus Bandwidth” Metric: Use Nsight Systems to see the actual throughput on the NVLink or PCIe bus. If you see your H100 peaking at 50GB/s when it should do 450GB/s, your topology is likely the culprit.
• NCCL Inspector Profiler Plugin : It explicitly reports Algorithmic Bandwidth (how much data your model thinks it sent) vs. Bus Bandwidth (how much raw data actually moved). If Bus BW is much higher than Algorithmic BW, your algorithm (like Ring) is likely crossing slow switches too many times.

If answer to one or more question is yes, then probably you should try to change the value of NCCL_ALGO and measure its impact.

Guidelines to select value for NCCL_ALGO

Scenario A: The “Spine Crossing” (Small Clusters)

• The Problem: On a 2-node or 4-node setup without a high-end switch (like InfiniBand), NCCL might default to Ring. The ring has to go Node 0 → Node 1 → Node 2 → Node 0. This “hairpin” turn through a standard Ethernet switch introduces massive jitter.
• The Fix: Set NCCL_ALGO=Tree. In small, high-latency networks, a Tree often reduces the number of "hops" across the switch, stabilizing your step time.

Scenario B: Massive Scale (High Node Count)

• The Problem: The Ring algorithm’s latency scales linearly ($O(N)$) with the number of nodes. On 128+ nodes, the “bucket brigade” takes too long to complete a single rotation.
• The Fix: Set NCCL_ALGO=Tree. Trees scale logarithmically ($O(\log N)$), making them far superior for latency-sensitive workloads at scale.

Scenario C: Heterogeneous Networks (Cloud & Virtualized)

• The Problem: In environments like AWS or GCP where you might have “noisy neighbors” or variable link speeds, one slow link kills a Ring (the ring is only as fast as its slowest link).
• The Fix: Use NCCL_ALGO=Tree or ensure CollNet is enabled if your hardware supports it. These are more resilient to a single "straggler" node.

Scenario D: Maximizing Throughput on Dense Nodes (The “Fat-Tree” setup)

• The Problem: You are running large-scale training (like an LLM pre-training) where the gradient sizes are massive. NCCL’s Tree algorithm is great for reducing latency, but it doesn’t always saturate the full available bandwidth of an all-to-all NVLink topology within a node. If your GPUs are sitting on a high-bandwidth “gold mine” (like an H100 8-GPU baseboard), you want to squeeze every last Gbps out of those copper traces.
• The Fix: Set NCCL_ALGO=Ring. In a Ring, every GPU is sending and receiving data simultaneously across all available links. For large data packets (massive gradients), the overhead of the "ring" is outweighed by the fact that it utilizes 100% of the aggregate bi-directional bandwidth. This is the "high-occupancy" mode—perfect for when the "pipe" is huge and the "payload" is even bigger.

Additional Considerations

Very small tensors behave differently. Once the tensor is small enough, most of the wait is starting kernels and syncing, not shipping gigabytes. In that world, arguing about NVLink vs PCIe is like arguing about which highway to use for a one-block errand — the engine warm-up dominates.

Some algorithm names only work on certain hardware (NVSwitch offload paths, collective-network plugins, and so on). If your machine does not have that path, NCCL never picks it — the third step in Fig 5 simply does not list that algorithm for your list.

If you set **NCCL_ALGO, you are telling NCCL “always** put this label in the third step,” even when the guesser would have chosen something else. That is fine for an experiment but if wall-clock time does not improve, delete the line from your job script.

Further reading (short list)

• NCCL user guide — boring name, useful answers.
• NCCL GitHub — issues and release notes when behavior changes.
• PAT paper — if you are here for huge all-gather / reduce-scatter.