Scaling Law Methodology

Scaling laws are presented as measurements. They are regressions — a smooth function fitted to perhaps forty runs, extrapolated two or three orders of magnitude past the largest of them. The exponent is an estimate with error bars that are rarely shown.

Two of the field’s most consequential scaling results have since been found to have fitting problems. That is the case for treating the methodology as a topic.

The three designs

Fixed model, vary data. Train a few sizes to convergence and read loss along each training curve. Cheap, because intermediate checkpoints are free — and biased if the learning-rate schedule is set for the full run, since every intermediate point is then mid-decay. This is exactly what went wrong in Kaplan.

IsoFLOP. Fix a compute budget, sweep model size along it, and find the minimum of the resulting U. Repeat at several budgets and fit the locus of minima. More expensive and much more robust, because each point is a properly scheduled run.

Parametric. Fit a closed form to every point at once.

Five parameters, a non-convex objective, and — in Chinchilla’s case — a fit that disagreed with the paper’s own other two approaches without that being flagged.

The replication

Besiroglu et al. reconstructed Hoffmann’s data from the published figures and refitted. Three findings.

The approach-3 parameters cannot be recovered from the data: the reported fit is not the optimum of the stated objective. The implied confidence intervals are implausibly narrow for forty-odd points. And a correct refit gives $a \approx 0.5$, agreeing with approaches 1 and 2 — so the paper’s headline conclusion is right, and one of its three routes to it was not.

The checklist

Choshen et al. surveyed a thousand-model corpus and produced the practical version.

Fit in log space. $\log L$ against $\log C$. Raw-space least squares weights the cheap, high-loss points most, which is the wrong end for extrapolation.

Use intermediate checkpoints. They are nearly free and they improve the fit materially — but only if the schedule is per-run, or the first design’s bias returns.

Discard the first ~10 000 steps. Early training is not on the power law and including it drags the fit.

Multi-start the optimiser. For any parametric form, refit from many initialisations and report the spread. This alone would have caught the Chinchilla discrepancy.

Report intervals. A point estimate of an exponent extrapolated three orders of magnitude is not a result on its own.

Five model sizes, minimum. Below that the exponent is not identified, and the largest should be within about an order of magnitude of the target.

import numpy as np
from scipy.optimize import minimize


def fit_parametric(n: np.ndarray, d: np.ndarray, loss: np.ndarray, restarts: int = 64):
    """Fit L = E + A/N^α + B/D^β. Multi-start, in log space, Huber objective."""
    def objective(theta):
        log_e, log_a, log_b, alpha, beta = theta
        pred = np.exp(log_e) + np.exp(log_a) / n**alpha + np.exp(log_b) / d**beta
        r = np.log(pred) - np.log(loss)                       # log space
        # Huber: quadratic near zero, linear in the tails.
        return np.where(np.abs(r) < 1e-3, 0.5 * r**2, 1e-3 * (np.abs(r) - 5e-4)).sum()

    rng = np.random.default_rng(0)
    fits = []
    for _ in range(restarts):
        x0 = rng.normal([0.5, 6, 6, 0.35, 0.35], [1, 2, 2, 0.15, 0.15])
        r = minimize(objective, x0, method="L-BFGS-B")
        if r.success:
            fits.append((r.fun, r.x))

    best = min(fits)[1]
    # The spread across restarts is the diagnostic, not the decoration.
    spread = np.std([f[1][3] for f in fits if f[0] < min(fits)[0] * 1.01])
    return {"alpha": best[3], "beta": best[4], "alpha_spread": spread}

alpha_spread is the line that matters. If restarts converging to near-identical objective values disagree about the exponent, the parameter is not identified by the data and no amount of reporting precision will make it so.