Training / scaling
Scaling
Scaling laws are the closest thing the field has to a design equation: given a compute budget, they say how large a model to train and how much data to train it on. They are empirical, they have been revised, and they are still the first calculation anyone does.
8 entries.
Entries
02.03.1—02.03.2—02.03.3—02.03.4—02.03.5—02.03.6—02.03.7—02.03.8O(N·D)
Chinchilla Scaling Lawscommon
Compute-optimal allocation between parameters and training tokens.
Compute-Optimal versus Inference-Optimalstandard
Chinchilla minimises training cost. Nobody trains a model they only train once.
Data-Constrained Scalingcommon
What repeated tokens are worth, and how many epochs before they are worth nothing.
Emergent Abilitiescommon
Capabilities that appear abruptly with scale, and the argument that the abruptness is the metric.
Kaplan Scaling Lawsstale
The 2020 power laws, and the three choices that made them recommend the wrong model.
Scaling Law Methodologycommon
How the curves are fitted, and how the fits go wrong.
Test-Time Compute Scalingstandard
Spend more compute per question instead of more compute per model.
Training Compute Accountingstandard
Why a training run costs 6ND FLOPs, and where the six comes from.