AI Grimoire
07 — siblings
metrics/
data/

Evaluation / data

Data

Before a benchmark result means anything, two questions have to be settled: whether the test set leaked into training, and whether the sample was large enough to distinguish the systems being compared. Both are routinely skipped.

2 entries.

Entries

07.02.1
Contamination Detectioncommon
n-gram and canary overlap tests between benchmark and pretraining data.
O(N)
07.02.2
Statistical Power for Evalsstandard
Error bars, paired tests and how many items an eval actually needs.
O(N)