#aws-neuron

Tree Communication Collective by Hand: Why the Cost Is 2 log2(N), and When That Beats the Ring

The previous blog traced ring all-reduce and showed its cost is 2(N-1): one N-1 for reduce-scatter, one N-1 for all-gather. The ring is…