Parallelism Composition
No single parallelism reaches ten thousand devices. The configuration that does is a product of four or five of them, ordered so the chattiest sits on the fastest wire.
Standing
Current standardPart of the default recipe. Present in most frontier models trained today, and the thing a new design departs from rather than argues for.
What a frontier training configuration is. The individual strategies are textbook; arranging them against a specific cluster is the part that is still craft.
judged as of 2026-09 · what the labels mean
Theory
Each strategy on this branch has a limit. Tensor parallelism saturates the interconnect past a node. Pipeline parallelism has a bubble that worsens with more stages. Data parallelism runs out at the critical batch size. Context parallelism is bounded by sequence length.
A frontier run uses all of them, and the arrangement is determined by one quantity.
Sort the strategies by intensity, sort the links by bandwidth, and match them up.
| Strategy | Per step | Placement |
|---|---|---|
| tensor | 4 collectives per layer | inside a node |
| context | hops per attention | inside a node |
| expert | 2 all-to-all per MoE layer | inside a node, or few nodes |
| pipeline | 1 activation per boundary | across nodes |
| data / ZeRO | 1 reduce per step | across everything |
Tensor parallelism is typically capped at the NVLink domain — 8 on most nodes — because the ninth device is across PCIe and the collective inherits its bandwidth.
The arithmetic
Both constraints bind at once, which is what makes the configuration awkward. Raising pipeline depth to fit a larger model reduces the data-parallel degree at fixed device count, which reduces the batch, which raises the bubble fraction because there are fewer micro-batches to fill it.
What goes wrong
Rank ordering. The mapping from global rank to (tp, cp, pp, dp) coordinates decides which group lands on which link. Get it wrong and tensor-parallel groups straddle nodes while data-parallel groups sit inside them — a configuration that trains correctly at a fraction of the throughput, with no error and no obvious symptom.
Stragglers. Every collective is a barrier, so the slowest rank sets the pace for its group. One thermally throttled GPU in sixteen thousand slows the entire job, and finding it means per-rank step timing.
Imbalance. Pipeline stages must be balanced in time; expert parallelism needs load balancing; context parallelism needs the zigzag assignment. Each is a separate mechanism for the same problem.
Implementation
import torch.distributed as dist
def build_mesh(tp: int, cp: int, pp: int, dp: int):
"""Rank layout, fastest-varying dimension first.
Global rank = ((dp_i * pp + pp_i) * cp + cp_i) * tp + tp_i
tp varies fastest, so a tensor-parallel group is contiguous ranks —
which the launcher places on one node. That single choice is what
keeps the chattiest collective on NVLink.
"""
world = tp * cp * pp * dp
assert dist.get_world_size() == world
mesh = dist.device_mesh.init_device_mesh(
"cuda", (dp, pp, cp, tp), mesh_dim_names=("dp", "pp", "cp", "tp")
)
return mesh
def sanity_check(mesh) -> None:
"""Assert the intra-node groups are actually intra-node. Cheap, and it
catches the misconfiguration that costs half the cluster's throughput."""
local = int(os.environ["LOCAL_WORLD_SIZE"])
tp_ranks = mesh["tp"].mesh.tolist()
assert max(tp_ranks) // local == min(tp_ranks) // local, "TP group spans nodes"That assertion is worth writing. The failure it catches is silent, expensive, and survives for as long as nobody compares measured throughput against a model FLOPs utilisation estimate — which on a new cluster is often the whole first month.
How to arrive at a configuration
- Tensor parallel to the NVLink domain, or until the layer fits.
- Context parallel if the sequence does not fit; multiply into the same node budget.
- Pipeline across nodes until the model fits in aggregate.
- Data parallel with everything left, up to the critical batch size.
- ZeRO or FSDP over the data-parallel group to shard what remains.
Then measure. Model FLOPs utilisation — achieved FLOPs over peak — is the number that says whether the layout is right, and anything below about 35% on a modern cluster means one of the five steps is wrong rather than that the model is demanding.