Skip to content
←

필사 모드: How to Read Leaderboards and Benchmarks: Why SOTA Has Such a Short Shelf Life

English
0%
정확도 0%
💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction

The last part of this series is about numbers, not models. Across the previous five articles I kept repeating the same sentence: this figure is author-reported, and it changes with the harness. This part supplies the papers behind that sentence.

Paper details were verified directly from the primary sources on 2026-08-12. This field moves fast, so check the current state yourself.

Why SOTA has such a short shelf life

SOTA is a timestamp, not a state. "X is SOTA" is shorthand for "on the leaderboard snapshot I last checked, with the harness and prompt I used, on the benchmark I picked, X was above the others." Four conditions are elided in that sentence, and three of them change within days.

HELM (arxiv.org/abs/2211.09110, submitted 2022-11-16), an early attempt to systematize benchmark evaluation, names exactly this problem. The authors report that prior to HELM, models were on average evaluated on just 17.9% of the core scenarios, which means the field was ranking scores measured on different subsets. Worth noting: the HELM authors also list the gaps in their own framework, including question answering for neglected English dialects and metrics for trustworthiness.

Contamination: when the exam leaks into the textbook

The best-known problem, and still underrated.

String matching is not enough for decontamination. The paper on rephrased samples (arxiv.org/abs/2311.04850, submitted 2023-11-08) shows that paraphrasing or translating test data defeats standard decontamination, and reports that a 13B model can overfit a benchmark and reach drastically high performance. It also observes that 8 to 18% of HumanEval overlaps with real pre-training corpora.

But not every model is contaminated. The GSM1k paper (arxiv.org/abs/2405.00332, submitted 2024-05-01), which rebuilt a benchmark in the image of GSM8k, reports drops of up to 8% and systematic overfitting in several model families, while also stating that many models, especially frontier ones, show minimal signs of overfitting and that all models broadly generalize to novel problems guaranteed absent from training. Ending the contamination discussion in cynicism would misread it.

The subject as a whole is surveyed in the contamination survey (arxiv.org/abs/2406.04244, submitted 2024-06-06). One response is to keep replacing the questions. LiveBench (arxiv.org/abs/2406.19314, submitted 2024-06-27) refreshes questions monthly from recent sources and scores automatically against objective ground truth. Note that the title says contamination-limited, not contamination-free. That was the authors' word choice.

Harness and prompt: same model, different score

This is the part with the most practical bite.

The paper quantifying prompt format sensitivity (arxiv.org/abs/2310.11324, submitted 2023-10-17) reports performance differences of up to 76 accuracy points from trivial formatting changes in few-shot settings. The instability persisted across larger models, more examples and instruction tuning. The authors' conclusion is blunt: the methodological validity of comparing models with an arbitrarily chosen fixed format is itself in question, and evaluations should report a range of performance across plausible formats instead of a single-format number.

The same problem approached from the tooling side appears in the lm-eval-harness team's write-up (arxiv.org/abs/2405.14782, submitted 2024-05-23, v3 on 2026-05-31). It enumerates sensitivity to evaluation setup, difficulty of proper comparison across methods, and the lack of reproducibility and transparency, and tries to codify knowledge that had circulated only as folk practice.

Speech has its own version. The Open ASR Leaderboard (arxiv.org/abs/2510.06961, submitted 2025-10-08) compares 86 systems across 12 datasets while standardizing WER and RTFx across differing toolkits. A standardized comparison and a set of separately measured numbers are different objects.

Self-reported scores and the structure of leaderboards

Technical report scores are self-reported. But a public leaderboard is not a neutral scale either.

The paper analyzing leaderboard structure (arxiv.org/abs/2504.20879, submitted 2025-04-29) examines one large arena-style evaluation platform. It reports that private pre-testing lets a provider run many variants and disclose only the good ones, that sampling is asymmetric — a couple of large providers each received roughly 19 to 20% of arena data while 83 open-weight models received about 30% combined — and that limited additional arena data can produce relative gains of up to 112% under conservative estimates. The authors conclude that the metric ends up reflecting overfitting to arena-specific dynamics rather than general model quality.

When the judge is not human

A great deal of current evaluation is one LLM grading another. The paper that first systematized this (arxiv.org/abs/2306.05685, submitted 2023-06-09) reports that strong judges agree with human preferences over 80% of the time, while acknowledging in the same abstract position bias, verbosity bias, self-enhancement bias and limited reasoning ability. The strength and the limitation live in one paragraph.

The 2026 follow-up work is blunter. A judge reliability stress test (arxiv.org/abs/2603.05399, submitted 2026-03-05) reports that no judge they evaluated was uniformly reliable across benchmarks. Judgments shifted under simple text formatting changes, paraphrasing, verbosity changes and flipping the ground truth label. The authors describe these as preliminary experiments.

Putting error bars on the numbers

Finally, what is missing from most evaluation tables is uncertainty. The paper arguing that evals should be treated as experiments and analyzed statistically (arxiv.org/abs/2411.00640, submitted 2024-11-01) treats eval questions as drawn from an unseen super-population and supplies formulas for measuring differences between two models and for planning an evaluation experiment.

A newer problem sits on top of that. The paper on the winner's curse in adaptive benchmarking (arxiv.org/abs/2605.05973, submitted 2026-05-07) points out that reusing benchmark items inside tuning creates selection bias that inflates performance estimates. The moment you search prompts and settings and keep the best result, that score stops being an estimate of performance on fresh data.

Reading order and a checklist

[Questions to ask of any leaderboard number]
 1. Who measured it   : author self-measurement or third party
 2. Which harness     : prompt format, few-shot count, answer extraction
 3. When was it written: after the model training cutoff or before
 4. Is sampling fair  : who got evaluated how many times
 5. Who is the judge  : human or model, were biases checked
 6. Are there error bars: is the gap larger than the noise
 7. Was it used for tuning: did that benchmark pick the settings

If you cannot answer three or more of these seven, do not make a decision with that number.

How and when this was verified

Every paper cited here was verified on 2026-08-12 by opening the arXiv abstract page directly and reading the title, identifier, submission date and abstract claims. Candidates that could not be opened were not cited. Every figure quoted is a value reported by that paper's authors.

The rule this whole series tried to keep is the same. Never write that a model is best; only carry over what the authors claimed and what they limited themselves. I did not reproduce any benchmark myself.

Try it yourself

  • AI Benchmark Collection — leaderboard sites grouped by category. Take the checklist above, open any leaderboard, and try answering the seven questions.
  • Browser AI Lab — running sentiment analysis or zero-shot classification yourself lets you watch results wobble when the same sentence is worded slightly differently.

Previous in the series: Image Generation and Understanding Technical Reports: What to Read

Back to the start: Text LLM Technical Reports: What to Read

References

현재 단락 (1/46)

The last part of this series is about numbers, not models. Across the previous five articles I kept ...

작성 글자: 0자원문 글자: 9,171자작성 단락: 0/46