02 — siblings

optimisers/

scaling/memory/distributed/

Training / optimisers

Optimisers

The optimiser decides how a gradient becomes a step. Almost everything at scale is a variant of one adaptive method plus a carefully shaped learning rate, and the shape turns out to matter about as much as the method.

3 entries, 1 of them a stub.

Entries

Adaptive moments with weight decay decoupled from the gradient.

Learning-Rate Schedulesstandard

Warmup, cosine decay, and the warmup–stable–decay alternative.

Second-Order Preconditioningstubpromising

Shampoo, Muon and the orthogonalised-update family.