Chinchilla Scaling Laws

Given a fixed compute budget, how large should the model be and how much data should it see? The answer is that both scale as the square root of compute — roughly twenty tokens per parameter.

[scaling][training]Commonly used

Standing

Commonly usedEstablished and frequently the right choice, but competing with live alternatives rather than having settled the question.

Still the first calculation anyone does, and routinely overridden: inference cost now argues for training smaller models on far more data than compute-optimality alone would.

judged as of 2026-09 · what the labels mean

Theory

Fit the loss as a function of parameter count $N$ and training tokens $D$ . The functional form has three terms: an irreducible entropy of the data, a penalty for finite model capacity, and a penalty for finite data.

L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}

Minimise eq. 1 subject to the compute constraint $C \approx 6ND$ — six FLOPs per parameter per token, two for the forward pass and four for the backward. Because $\alpha \approx \beta$ , the optimum splits the budget almost evenly between the two terms.

N_{\text{opt}} \propto C^{\frac{\beta}{\alpha+\beta}}, \qquad D_{\text{opt}} \propto C^{\frac{\alpha}{\alpha+\beta}}, \qquad \frac{D_{\text{opt}}}{N_{\text{opt}}} \approx 20

What it corrected

Kaplan’s earlier fit concluded that parameters should grow far faster than data, which is why the models of 2020–2021 were large and badly undertrained. The difference traces to two methodological points: Kaplan held the learning-rate schedule fixed rather than matching it to each run’s length, and counted non-embedding parameters. Chinchilla, at 70B trained on 1.4T tokens, beat a 280B model trained on 300B tokens using the same compute.

The replication attempt is worth reading alongside the original: it recovers the same conclusion but finds the reported confidence intervals implausibly tight, and estimates $\alpha$ and $\beta$ closer to $0.35$ each.

Implementation

python · numpy + scipy

import numpy as np
from scipy.optimize import minimize

# Hoffmann et al., Table 2
E, A, B, ALPHA, BETA = 1.69, 406.4, 410.7, 0.34, 0.28


def loss(n_params: float, n_tokens: float) -> float:
    return E + A / n_params**ALPHA + B / n_tokens**BETA


def compute_optimal(flops: float) -> tuple[float, float]:
    """Minimise loss subject to C = 6ND."""
    def objective(log_n):
        n = np.exp(log_n[0])
        return loss(n, flops / (6 * n))

    best = minimize(objective, [np.log(1e9)], method="Nelder-Mead")
    n = float(np.exp(best.x[0]))
    return n, flops / (6 * n)


n, d = compute_optimal(1e23)
print(f"{n/1e9:.1f}B params, {d/1e12:.2f}T tokens, {d/n:.0f} tokens/param")

The $6ND$ estimate ignores attention, which is fine while $n \ll 12d$ and increasingly wrong at long context — at 32k tokens and $d = 4096$ the attention term is no longer a rounding error. Fitting the constants to your own architecture on a ladder of small runs beats importing Hoffmann’s numbers wholesale; what transfers is the shape, not the coefficients.

7B model

140B tokens

70B model

1.4T tokens

Tokens / param

≈ 20

compute-optimal points on the Hoffmann fit

learning-rate-schedules adamw perplexity zero-sharding

References

[1]Hoffmann et al. — Training Compute-Optimal Large Language Models (2022)arXiv:2203.15556

[2]Kaplan et al. — Scaling Laws for Neural Language Models (2020)arXiv:2001.08361

[3]Besiroglu et al. — Chinchilla Scaling: A Replication Attempt (2024)arXiv:2404.10102