Elo / Bradley-Terry Scoring

Assign each model a scalar strength such that the logistic of the difference predicts who wins. The same model that underwrites DPO, fitted to human votes instead.

[metric][preference]Commonly used

Standing

Commonly usedEstablished and frequently the right choice, but competing with live alternatives rather than having settled the question.

judged as of 2026-09 · what the labels mean

Theory

Bradley–Terry posits a latent strength per competitor with the probability of a win given by a logistic in the difference. Fitting is a convex logistic regression on the pairwise outcomes.

P(i \text{ beats } j) = \sigma\!\left( \frac{s_i - s_j}{400} \ln 10 \right)

Online Elo updates are order-dependent. Fit the whole matrix by maximum likelihood instead and bootstrap for intervals. The scale has no zero — only differences are identified, which is why every leaderboard fixes an anchor.

\hat{s} = \arg\max_{s} \sum_{(i,j)} w_{ij} \log \sigma(s_i - s_j) \quad \text{subject to} \quad \textstyle\sum_i s_i = 0

Implementation

python · numpy + scipy

import numpy as np
from scipy.optimize import minimize
from scipy.special import log_expit


def bradley_terry(wins: np.ndarray, scale: float = 400.0) -> np.ndarray:
    """wins[i, j] = number of times i beat j. Returns Elo-scaled strengths."""
    n = wins.shape[0]

    def nll(s):
        diff = s[:, None] - s[None, :]
        return -(wins * log_expit(diff)).sum()

    res = minimize(nll, np.zeros(n), method="L-BFGS-B")
    s = res.x - res.x.mean()               # fix the gauge
    return 1000 + s * (scale / np.log(10))

Use log_expit rather than log(sigmoid(·)); the composed form underflows to $-\infty$ for strength gaps a fit will routinely visit, and L-BFGS then walks off. Centring the solution is not cosmetic — the likelihood is invariant to a constant shift, so without it the optimiser wanders along a flat direction and the reported numbers are not reproducible.

perplexity direct-preference-optimisation contamination-detection

References

[1]Bradley & Terry — Rank Analysis of Incomplete Block Designs (1952)Biometrika 39

[2]Chiang et al. — Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (2024)arXiv:2403.04132