02 — siblings

objectives/optimisers/scaling/memory/

distributed/

Training / distributed

Distributed

Once the model, its gradients and its optimiser state exceed a single accelerator, the question becomes what to shard and when to gather it again. The answers differ in how much communication they add to buy how much memory back.

8 entries.

Entries

Collective Communicationstandard

All-reduce, all-gather, reduce-scatter, all-to-all — the six primitives everything is built from.

Context Parallelismcommon

Split the sequence across devices, and pass the KV blocks around a ring.

O(n²·d/N)02.05.3

Data Parallelismstandard

Replicate the model, split the batch, average the gradients.

Fault Tolerancecommon

At ten thousand devices something fails hourly; the run has to survive it.

Parallelism Compositionstandard

Four or five strategies at once, and the topology decides the order.

Pipeline Parallelismstandard

Layers on different devices, and the bubble that schedule design exists to shrink.

Tensor Parallelismstandard

Split each matrix across devices; two all-reduces per layer buy the memory back.

ZeRO Shardingstandard

Partition optimiser state, gradients and parameters across ranks.