Statistical Power for Evals

Accuracy on $n$ items is a mean of $n$ Bernoulli draws. Its standard error is the familiar one, and it is larger than most leaderboard gaps.

Pair the comparison

Comparing two models on the same items means the question is not “are these two proportions different” but “is the mean of the per-item difference nonzero”. Items that both models get right or both get wrong contribute nothing to the variance, and on a shared benchmark most items are like that.

The paired standard error is typically two to three times smaller than the unpaired one. McNemar’s test is the same idea in exact form: condition on the discordant pairs and test whether they split evenly.

Two things that inflate confidence

Clustering. Benchmarks with several questions per passage or per template violate independence. The effective sample size is closer to the number of clusters than the number of items, and ignoring it understates the interval — often by a factor of two.

Sampling variance. At non-zero temperature the model is a random variable too. Report over $k$ samples per item and the variance decomposes into between-item and within-item parts; only the second falls as you sample more.

With fewer than a few hundred items the normal approximation is unreliable at the tails. Use a Wilson or Clopper–Pearson interval, or bootstrap — the $\hat p \pm 1.96\,\mathrm{SE}$ reflex is the thing to unlearn first.

import numpy as np
from scipy import stats


def wilson(correct: int, n: int, z: float = 1.96) -> tuple[float, float]:
    """Better than normal approximation at small n and extreme p."""
    p = correct / n
    denom = 1 + z**2 / n
    centre = (p + z**2 / (2 * n)) / denom
    half = z * np.sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / denom
    return centre - half, centre + half


def paired_test(a: np.ndarray, b: np.ndarray) -> dict[str, float]:
    """a, b: 0/1 correctness on the SAME items, aligned."""
    only_a = int(np.sum((a == 1) & (b == 0)))
    only_b = int(np.sum((a == 0) & (b == 1)))

    # exact McNemar: how surprising is this split of the discordant pairs?
    p_value = stats.binomtest(only_a, only_a + only_b, 0.5).pvalue
    diff = float(np.mean(a - b))
    se = float(np.std(a - b, ddof=1) / np.sqrt(len(a)))

    return {"diff": diff, "se": se, "ci95": 1.96 * se, "p": p_value}

ddof=1 is not a detail — the population formula understates the standard error at the sample sizes evals actually use. If the benchmark has clusters, bootstrap by resampling whole clusters rather than items; resampling items treats correlated questions as independent evidence and returns an interval that is confidently wrong.