Pipeline Parallelism

Split the model by depth: layers 1–20 on device 0, 21–40 on device 1, and so on. Communication is one activation tensor per stage boundary — far less than tensor parallelism, and point-to-point rather than a collective, so it survives crossing a network.

The problem is that a pipeline with one item in it is not a pipeline.

The bubble

Device 1 cannot start until device 0 finishes. With one batch, $S$ stages and no other work, each device is busy for $1/S$ of the time.

The fix is to split the batch into micro-batches and keep several in flight. GPipe does exactly this: run $M$ micro-batches forward through the pipeline, then all their backwards.

At $S=8$ and $M=32$ that is 18% idle. Raising $M$ helps and costs activation memory, since GPipe holds every micro-batch’s activations until its backward pass runs.

1F1B

The scheduling improvement that made pipelines practical. Instead of all forwards then all backwards, alternate — once the pipeline is full, each device does one forward and one backward per slot.

The bubble is unchanged. What changes is memory: a micro-batch’s activations are freed as soon as its backward completes, so a device holds at most $S$ micro-batches’ worth rather than $M$. That decouples $M$ from memory, and $M$ is the term that shrinks the bubble.

Interleaving

Megatron’s addition. Rather than one contiguous block per device, give each device $v$ non-adjacent chunks — device 0 gets layers 1–4 and 17–20, device 1 gets 5–8 and 21–24, and so on.

Each stage is smaller, so the pipeline fills faster. The cost is $v$ times as many point-to-point transfers, which is why it is worth it on a fast interconnect and not on a slow one.

def schedule_1f1b(stage: int, stages: int, micro_batches: int) -> list[str]:
    """What this stage does, in order. Warmup, steady, drain."""
    warmup = min(stages - stage - 1, micro_batches)

    ops = ["F"] * warmup                                  # fill the pipeline
    steady = micro_batches - warmup
    ops += ["F", "B"] * steady                            # one of each per slot
    ops += ["B"] * warmup                                 # drain

    return ops


# Stage 0 of 4, 8 micro-batches:
#   F F F  FB FB FB FB FB  B B B
# Stage 3 of 4 starts its backward immediately — no warmup, no drain.

The asymmetry in that schedule is the memory story. Stage 0 holds $S$ micro-batches of activations through its warmup; the last stage holds one. In practice the first stage is given fewer layers to compensate, which is the same balancing problem as the embedding, from the other direction.

Only when you must. Pipeline parallelism adds a scheduling problem, a balancing problem and an irreducible bubble, and it interacts awkwardly with anything that changes sequence length between steps.

The rule of thumb at scale: tensor parallelism inside a node, up to the NVLink domain; pipeline parallelism across nodes, because it communicates least; data parallelism outermost. Composing them is what a frontier training configuration actually is.

Note that ZeRO stage 3 is an alternative to the middle term rather than a complement — it shards parameters across the data-parallel group and gathers them per layer, which achieves the same memory reduction with collectives instead of a schedule. Which is better depends on the interconnect, and the answer at frontier scale has been changing every couple of years.