Elo / Bradley-Terry Scoring
Assign each model a scalar strength such that the logistic of the difference predicts who wins. The same model that underwrites DPO, fitted to human votes instead.
Bradley–Terry posits a latent strength per competitor with the probability of a win given by a logistic in the difference. Fitting is a convex logistic regression on the pairwise outcomes.
Online Elo updates are order-dependent. Fit the whole matrix by maximum likelihood instead and bootstrap for intervals. The scale has no zero — only differences are identified, which is why every leaderboard fixes an anchor.
import numpy as np
from scipy.optimize import minimize
from scipy.special import log_expit
def bradley_terry(wins: np.ndarray, scale: float = 400.0) -> np.ndarray:
"""wins[i, j] = number of times i beat j. Returns Elo-scaled strengths."""
n = wins.shape[0]
def nll(s):
diff = s[:, None] - s[None, :]
return -(wins * log_expit(diff)).sum()
res = minimize(nll, np.zeros(n), method="L-BFGS-B")
s = res.x - res.x.mean() # fix the gauge
return 1000 + s * (scale / np.log(10))Use log_expit rather than log(sigmoid(·)); the composed form underflows to
for strength gaps a fit will routinely visit, and L-BFGS then walks off.
Centring the solution is not cosmetic — the likelihood is invariant to a constant
shift, so without it the optimiser wanders along a flat direction and the reported
numbers are not reproducible.