Kaplan Scaling Laws

Before 2020 there was no principled way to decide how large a model to train. Kaplan et al. supplied one, and the form of their answer has never been seriously challenged.

with $\alpha_N \approx 0.076$ and $\alpha_D \approx 0.095$. No thresholds, no plateaus, no discontinuities — loss falls predictably and the exponents are small, which is the fact that made scaling a strategy rather than a hope.

Their allocation result was that given extra compute, most of it should go to parameters: $N \propto C^{0.73}$, $D \propto C^{0.27}$. GPT-3 was built to that recommendation — 175B parameters on 300B tokens, fewer than two tokens per parameter — and so was almost everything trained in the two years after.

The three causes

Porian et al. reproduced both papers and isolated what differed. Three choices, and each moves the exponent.

The learning-rate schedule. Kaplan et al. trained every model with a single schedule set for their longest run. A cosine schedule truncated early leaves the model mid-decay and undertrained, so every small-$D$ point was worse than it should have been — which makes data look less valuable than it is. Chinchilla tuned the schedule to each run’s length. This is the largest of the three.

Embedding parameters. Kaplan et al. excluded them from $N$; Chinchilla included them. At the small end of the sweep, embeddings are a large fraction of the model, so the two conventions disagree most exactly where the fit is most sensitive.

Warmup. Small models in the Kaplan sweep spent a significant fraction of their brief training in warmup, again penalising short runs.

Correct all three and the Kaplan setup reproduces Chinchilla’s exponent. The disagreement was never about the science.

Quite a lot, and it is worth separating from what did not.

The power-law form itself, over seven orders of magnitude in compute — this is the finding, and everything since is a refinement of the constants.

The irrelevance of architectural detail: within a wide band, depth, width and aspect ratio move the loss far less than scale does. The result that made it reasonable to stop redesigning transformers and start enlarging them.

The critical batch size, $B_{\text{crit}} \propto L^{-4.8}$ — a prediction of how large a batch can be before extra examples stop buying anything, still the starting point for that calculation.

The failure was not in the model of scaling. It was that a hyperparameter held fixed across the sweep became a confound, and the confound was correlated with the variable being studied.

That is a hazard for any scaling study, and it is why methodology is a topic in its own right. Something has to be held fixed across a sweep, and the question of whether the thing you fixed is neutral with respect to what you are measuring has no general answer — only, as here, an expensive empirical one.