Training / memory
Memory
Backpropagation needs the forward pass’s activations, and at depth that dominates memory. The techniques here recompute what they refused to store, which is almost always the right trade when the alternative is not fitting at all.
8 entries.
Entries
02.04.1O(L·B·T·d)02.04.2O(L·B·T·d)02.04.3—02.04.4O(1)02.04.5O(√L)02.04.6—02.04.7O(P)02.04.8O(L)
Activation Compressionpromising
Store the saved activations in fewer bits instead of recomputing them.
Activation Memory Accountingstandard
How much a forward pass must keep, and which term dominates when.
CPU and NVMe Offloadingcommon
Park what is not in use on host memory or disk, bounded by PCIe rather than compute.
Gradient Accumulationstandard
Simulate a large batch with several small ones, and the normalisation bug that follows.
Gradient Checkpointingstandard
Trade recompute for activation memory at segment boundaries.
Memory Fragmentationcommon
The allocator that says out-of-memory with gigabytes free, and what to do about it.
Mixed-Precision Trainingstandard
Half-precision compute, full-precision accumulation, and the master copy in between.
Selective Recomputationcommon
Recompute only the cheap-and-large activations, not the whole block.