Training / optimisers
Optimisers
The optimiser decides how a gradient becomes a step. Almost everything at scale is a variant of one adaptive method plus a carefully shaped learning rate, and the shape turns out to matter about as much as the method.
8 entries, 1 of them a stub.
Entries
02.02.1O(d)02.02.2O(P)02.02.3O(1)02.02.4—02.02.5O(P)02.02.6O(P)02.02.7O(d^1.5)02.02.8O(P)
AdamWstandard
Adaptive moments with weight decay decoupled from the gradient.
Gradient Clippingstandard
Bound the update norm so one bad batch cannot destroy a run.
Learning-Rate Schedulesstandard
Warmup, cosine decay, and the warmup–stable–decay alternative.
Maximal Update Parameterisationcommon
Scale the initialisation and learning rates so the optimum transfers across width.
Memory-Efficient Optimiserscommon
Adam’s state is twice the model; factorise it, quantise it, or project it.
Schedule-Free Optimisationpromising
Averaging in place of a decay schedule, so the run has no fixed horizon.
Second-Order Preconditioningstubpromising
Shampoo, Muon and the orthogonalised-update family.
Stochastic Gradient Descentcommon
A step against the gradient, momentum on top, and why it loses to Adam on transformers.