AI Grimoire
Tree — 07
metrics/3data/2

Evaluation

What a number on a leaderboard is actually measuring, and the two failure modes that dominate: the benchmark leaked into pretraining, or the difference between two systems is inside the noise.

Five entries across two branches, none of them stubs.

Metrics

metrics/ ›

what the numbers measure, and what they merely correlate with

07.1
Elo / Bradley-Terry Scoringcommon
Latent-strength model fitted to pairwise preference outcomes.
O(P)07.2
LLM-as-Judgecommon
Using a model to score model outputs, and the biases that come with it.
O(P)07.3
Perplexitycommon
Exponentiated mean negative log-likelihood over a held-out corpus.
O(n)

Data

data/ ›

contamination, sample size, and whether a result is real

07.4
Contamination Detectioncommon
n-gram and canary overlap tests between benchmark and pretraining data.
O(N)07.5
Statistical Power for Evalsstandard
Error bars, paired tests and how many items an eval actually needs.
O(N)