Collective Communication
Every distributed training strategy is a choice of which tensor to split and which collective to pay for. There are only about six collectives, and the cost model for each is one line.
Standing
Current standardPart of the default recipe. Present in most frontier models trained today, and the thing a new design departs from rather than argues for.
The vocabulary every parallelism strategy is written in. Older than deep learning by two decades and unchanged in substance.
judged as of 2026-09 · what the labels mean
Theory
Distributed training is a small number of tensor-splitting decisions, and each one buys memory by paying for a collective. Knowing the six operations and what each costs is enough to reason about any of the strategies on this branch.
| Operation | Each rank has | Each rank ends with |
|---|---|---|
| broadcast | rank 0 has | |
| reduce | rank 0 has | |
| all-reduce | ||
| all-gather | ||
| reduce-scatter | (full size) | shard of |
| all-to-all |
The identity worth memorising:
That decomposition is not a curiosity. It is exactly how ZeRO turns data-parallel all-reduce into something cheaper: if each rank only needs its shard of the summed gradient, the all-gather half can be skipped.
Ring all-reduce
The naïve implementation sends everything to one rank and back, which makes that rank’s link the bottleneck and scales badly. The ring algorithm does not.
Arrange the ranks in a circle and split the tensor into chunks. In the reduce-scatter phase, each rank sends one chunk to its neighbour and adds what it receives — after steps each rank holds the complete sum for one chunk. The all-gather phase circulates those around, another steps.
The bandwidth term approaches and stops growing — twice the tensor size per rank however many ranks there are, which is why data parallelism scales at all. The latency term does grow with , which is why small tensors are the problem and gradient bucketing exists.
Topology
Bandwidth differs by an order of magnitude depending on which link a message crosses.
| Link | Bandwidth |
|---|---|
| NVLink, intra-node | 400–900 GB/s |
| PCIe 5 ×16 | 64 GB/s |
| InfiniBand NDR | 50 GB/s per port |
| Ethernet | 12 GB/s |
Every serious strategy places the chattiest parallelism inside a node. Tensor parallelism all-reduces twice per layer and belongs on NVLink; data parallelism reduces once per step and tolerates the network. Getting that assignment backwards is the most common way to lose half the throughput of a cluster, and it is a launch-configuration decision rather than a code change.
Implementation
import torch
import torch.distributed as dist
def gradient_all_reduce(grads: list[torch.Tensor], bucket_mb: int = 25) -> None:
"""Bucket small tensors: the latency term dominates below ~1 MB, so
one call for a hundred tensors beats a hundred calls."""
bucket, size = [], 0
for g in grads:
bucket.append(g)
size += g.numel() * g.element_size()
if size >= bucket_mb * 2**20:
_flush(bucket)
bucket, size = [], 0
if bucket:
_flush(bucket)
def _flush(bucket: list[torch.Tensor]) -> None:
flat = torch._utils._flatten_dense_tensors(bucket)
dist.all_reduce(flat, op=dist.ReduceOp.AVG)
for g, r in zip(bucket, torch._utils._unflatten_dense_tensors(flat, bucket)):
g.copy_(r)ReduceOp.AVG rather than SUM followed by a division: it is one kernel instead
of two, and it avoids a subtle overflow when many ranks contribute large
gradients in low precision.
Bucketing is what DDP does automatically, and it is why a hand-rolled data-parallel loop is usually much slower — a transformer has hundreds of parameter tensors, most of them small, and the per-call latency dominates.