필사 모드: Taking Apart LLM Benchmark Tooling — Why the Same MMLU Gives Different Scores in Different Harnesses

• Introduction — Same Model, Same MMLU, Three Different Scores
• What a Harness Actually Does — Six Steps
• How You Pick the Answer Is What Makes the Score
• Comparing the Major Harnesses
• Running It Once, Start to Finish
  • Reading and Editing a Task Definition Directly
• The Minimum Checklist for Reproducibility
• The Grader Is the Benchmark
• Closing — A Number Without Conditions Isn't a Number
• References

Introduction — Same Model, Same MMLU, Three Different Scores

In June 2023, the team behind Hugging Face's Open LLM Leaderboard got flooded with strange complaints. LLaMA 65B's MMLU score was coming in far below the original paper's 63.4. The investigation turned up no bug — just three separate MMLU implementations, each grading the same dataset a different way.

The source is Hugging Face's What's going on with the Open LLM Leaderboard?. What's worth noting here isn't the size of the gap — it's that the gap differs by model. At 65B it's nearly 15 points, but at 7B it's almost nothing. In other words, swapping harnesses doesn't shift scores by a uniform amount — it flips the ranking between models. This isn't a problem you can fix with a single calibration constant.

Here's what separated the three implementations. The original compares only the probabilities of the four letters A/B/C/D. HELM has the model generate a single letter and treats that as the answer. The harness of the time compared the log-likelihood of the entire option sentence. The prompts differed too — the original included the subject-name line, HELM prepended a "Question:" prefix, and the harness dropped the subject name and added "Choices:".

This post's premise comes from right here. A benchmark score is not a property of the model — it is the result of a measurement performed under specific conditions. And yet most published numbers omit those conditions. So there's exactly one question we should keep asking: what has to be true for this number to mean what it appears to mean?

The big picture on how much to trust leaderboard numbers is already covered in Separating Signal From Noise in AI Coding Model Evals, and a map of which benchmark measures what is covered in AI Agents & LLM Benchmarks 2026. This post looks at the layer beneath both — what the tooling is actually doing.

What a Harness Actually Does — Six Steps

The sentence "we ran MMLU" compresses at least six steps. Each step has a decision to make, and if that decision isn't recorded, reproduction is impossible.

1. Load data       Which split (test/validation)? How many? Is the order fixed?
        |
2. Build few-shot   Which pool are examples drawn from, how many, what order, what seed?
        |
3. Assemble prompt  Template, delimiters, system message, whether a chat template is applied
        |
4. Call the model   Temperature, top_p, max_tokens, stop strings, retries, batch size
        |
5. Parse the answer Log-likelihood comparison? Regex extraction? Last number?
        |
6. Grade & aggregate Exact match, partial credit, model grading, and the unit you average over

Summarized in one line per step, here's how each one moves the score.

The MMLU-Pro paper (arXiv 2406.01574, NeurIPS 2024) actually measured this sensitivity. Run across 24 prompt styles, MMLU's score swing was generally 4-5 percentage points, and up to 10.98 percentage points. On MMLU-Pro, whose options were expanded to 10, that swing shrank to generally 2 percentage points, up to 3.74. Put the other way: on the original MMLU, just changing the prompt moves the score enough to cut across the leaderboard's mid-tier.

There are even more extreme measurements. Sclar et al.'s Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design (arXiv 2310.11324) reports that formatting changes with identical meaning alone produced up to a 76-point accuracy swing on LLaMA-2-13B. A 2025 paper, A Single Character can Make or Break Your LLM Evals (arXiv 2510.05152), goes a step further — it reports that changing just a single character used to separate examples can move an MMLU score by up to 23 percent, and that manipulation alone is enough to put whichever model you want in first place. Prompt format isn't an experimental setting — it's part of the measuring instrument.

How You Pick the Answer Is What Makes the Score

There are at least three ways to implement "the model picked an answer" for a multiple-choice task. The three measure different abilities.

Log-likelihood comparison. The model is never asked to generate anything. Each option string is appended after the prompt, its log probability is computed, and whichever is highest is treated as the chosen answer. With no generation, it's deterministic and fast. In exchange, it measures the ability to follow instructions not at all. A base model that ignores instructions still gets a normal score.

Letter-probability comparison. The prompt is built up to "Answer: " and only the probability of the next token being A, B, C, or D is compared. It's unaffected by option length, but it ignores the reality that the model might actually have answered like "Answer: B."

Generate, then parse. The model is asked to actually answer, and the answer is extracted from the text. This is closest to real usage. In exchange, the parsing rule becomes part of the grading.

The log-likelihood approach has one more well-known trap. A longer string has more tokens, so the sum of its log probabilities is naturally smaller. So comparing raw sums with no correction structurally favors shorter options. That's why lm-evaluation-harness reports acc_norm alongside acc — acc_norm divides the log-likelihood by option length to offset this bias. If the two numbers diverge sharply on a given task, that's a signal the task is sensitive to the distribution of option lengths.

Evaluating in Korean adds one more layer on top of this. EleutherAI's normalization explainer calls this correction byte-length normalization and explains its advantage as being independent of tokenization scheme. But issue 3278 in the repository points out that the actual code divides by the string's character count instead. In English these are effectively the same, but they differ for Korean, Japanese, and Chinese, where one character is 3 bytes in UTF-8. If your eval set mixes Korean and English options, this difference flows straight into the score. This is exactly why you should check the normalization method in the code, not the docs.

GSM8K is the representative case on the generate-then-parse side. The lm-evaluation-harness gsm8k task produces two scores from a single run.

Same run, same output, and two different numbers come out. Yet the GSM8K score cited in papers and model cards is usually just one of the two, without saying which. There's even an open issue in the repository reporting that the flexible-extract filter fails to properly normalize currency symbols or commas, marking correct answers as wrong (issue 3214). It's a structure where a bug in the parser gets recorded as the model's score.

Comparing the Major Harnesses

Here's a rundown of the tools actually used in practice and in papers as of August 2026. Versions are the latest release as of the check date.

A few things worth noting.

HELM's perspective is different. HELM doesn't ask "what's this model's MMLU score" — it asks "on this scenario, how does this model do on each of accuracy, calibration, robustness, fairness, toxicity, and efficiency." As the original paper (arXiv 2211.09110) puts it, this is a multi-metric approach that measures 7 metrics together across 16 core scenarios. You run it with helm-run, aggregate with helm-summarize, and view the results in a browser with helm-server.

You should be aware of a status change, though. HELM switched to maintenance mode on June 1, 2026. The code and leaderboard stay public, but no new features or new evaluations get added. Keep that in mind if you're adopting it as a tool now, and conversely, when citing a past HELM score, don't assume that the leaderboard from that time is still being updated today.

It's worth knowing how HELM handles multiple choice, too. There's a mode where all options are presented and the model generates a letter (joint), a mode that scores each option separately (separate), and a mode that also corrects for the options' own prior probability (separate-calibrated). As more recent APIs stop exposing token probabilities, the field has leaned toward joint — and this single choice was the axis behind the 15-point gap we saw in the table above.

Inspect AI was designed with agent evaluation as the premise. Tasks are split into three parts: Dataset, Solver, and Scorer. A Solver can be as simple as a single generation, or a full agent that uses tools over many turns; a Scorer can range from string comparison to model grading. Docker and Kubernetes sandboxes for safely running model-generated code ship built in. It's developed jointly by the UK AI Security Institute and Meridian Labs, and it's effectively the standard on the safety-evaluation side. Hugging Face's LightEval adopting it as the preferred backend is one of 2026's notable convergence signals.

OpenAI simple-evals is a reference implementation more than a tool. Its reason for existing is to disclose exactly which prompt and settings OpenAI used to produce the accuracy numbers it publishes alongside a model. It insists on 0-shot chain-of-thought, on the judgment that few-shot prompting is a holdover from the base-model era and reflects real usage less. That single choice makes it incomparable with other harnesses. There's an interesting detail too — different sampler implementations default to different temperatures: the chat-API sampler defaults to temperature 0.5 and a max of 1024 tokens, while the Claude sampler defaults to temperature 0.0 and a max of 4096 tokens. Sampling conditions differ by model even within the same repository.

Two things worth distinguishing that are easy to confuse. The GitHub repo openai/evals hasn't been archived and there's no deprecation notice, but it's effectively dormant (the PyPI package hasn't been updated since 2024). By contrast, the Evals dashboard and API that OpenAI hosted had their shutdown announced on June 3, 2026, going read-only on October 31, 2026, and shutting down entirely on November 30, 2026. The repo and the platform are different things.

promptfoo was acquired by OpenAI on March 9, 2026 (OpenAI's announcement). The open-source license was announced to remain unchanged. Worth knowing when choosing a tool — the maintaining entity has changed.

One more thing. Hugging Face's Open LLM Leaderboard shut down operations on March 13, 2025 — the very leaderboard behind the MMLU table at the top of this post. Even after a leaderboard disappears, the scores that came out of it keep getting cited in papers and model cards, so it's a good habit to check whether the source of a cited score is still alive today.

Running It Once, Start to Finish

It's faster to just run it once than to keep explaining. This is sized to finish even on a laptop CPU.

python3 -m venv .venv
source .venv/bin/activate
pip install "lm-eval[api]==0.4.12"

# First, check what tasks are available
lm_eval --tasks list | head -40

Run ARC-Easy on a small model, limited to 200 questions. Pinning the seed and the few-shot count is the key part.

lm_eval \
  --model hf \
  --model_args pretrained=EleutherAI/pythia-160m,dtype=float32 \
  --tasks arc_easy \
  --num_fewshot 5 \
  --batch_size 8 \
  --device cpu \
  --seed 1234 \
  --limit 200 \
  --output_path ./results/pythia-160m \
  --log_samples

What matters here is what each flag pins down.

• Run without --num_fewshot 5 and the task YAML's default gets used. It differs by task, and it changes when the version changes.
• --seed pins the few-shot example draw and the shuffle. Leave it out and every run builds a different prompt.
• --limit 200 cuts it off at the first 200. This is not a random sample, so don't cite this number as an overall score. It's for debugging.
• --log_samples genuinely matters. It saves the raw prompt, the model's raw output, and the parsing result — all of it. Without it, you can never find out after the fact why the score came out the way it did.

Once it finishes, JSON shows up in the results directory. The structure looks roughly like this. Exact key names can vary by version, so it's worth opening it directly to check.

{
  "results": {
    "arc_easy": {
      "alias": "arc_easy",
      "acc,none": 0.395,
      "acc_stderr,none": 0.0346,
      "acc_norm,none": 0.365,
      "acc_norm_stderr,none": 0.0341
    }
  },
  "configs": {
    "arc_easy": {
      "task": "arc_easy",
      "output_type": "multiple_choice",
      "num_fewshot": 5,
      "metric_list": [{ "metric": "acc" }, { "metric": "acc_norm" }]
    }
  },
  "config": {
    "model": "hf",
    "model_args": "pretrained=EleutherAI/pythia-160m,dtype=float32",
    "batch_size": 8,
    "random_seed": 1234,
    "limit": 200
  },
  "git_hash": "…",
  "date": 1785000000
}

What you actually need to look at in this JSON isn't results — it's configs and config. Everything you need to attach when citing a single score is right here. The fact that acc and acc_norm are 3 points apart is only visible here too.

Measuring an instruction-tuned model over an API changes the command. Stand up an OpenAI-compatible endpoint with something like vLLM or Ollama, then connect like this.

lm_eval \
  --model local-chat-completions \
  --model_args model=qwen3-8b,base_url=http://localhost:8000/v1/chat/completions,num_concurrent=8,max_retries=3,tokenized_requests=False \
  --tasks gsm8k \
  --num_fewshot 5 \
  --apply_chat_template \
  --fewshot_as_multiturn \
  --gen_kwargs temperature=0,max_gen_toks=512 \
  --output_path ./results/qwen3-8b \
  --log_samples

--apply_chat_template and --fewshot_as_multiturn are almost always needed when measuring an instruction-tuned model. The former wraps the prompt in the chat format the model was trained on; the latter feeds few-shot examples in as actual back-and-forth conversation turns instead of one long block of text. Just the presence or absence of these two flags can split an instruction-tuned model's score significantly. And yet it's rare to find this information alongside a published number.

Some APIs don't support log-likelihood at all. In that case a multiple-choice task can't be run via log-likelihood, so only the generate_until family works. In other words, whether you're measuring a model via API or via weights changes which grading methods are even available to you in the first place. This alone can make two scores on the same benchmark incomparable.

Reading and Editing a Task Definition Directly

In lm-evaluation-harness, a task is a single YAML file. If you want to use the same harness on internal data, this is the only file you need to touch. Let's look at multiple choice first.

task: internal_mcq
dataset_path: json
dataset_kwargs:
  data_files:
    test: ./data/internal_mcq.jsonl
output_type: multiple_choice
test_split: test
doc_to_text: "Answer the following question.\nQuestion: {{question}}\nAnswer:"
doc_to_choice: "{{choices}}"
doc_to_target: "{{label}}"
num_fewshot: 5
fewshot_config:
  sampler: default
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
  - metric: acc_norm
    aggregation: mean
    higher_is_better: true
should_decontaminate: false
metadata:
  version: 1.0

Change even a single character in doc_to_text here and the score moves. That's exactly what happened in the MMLU case earlier. So this file absolutely has to be under version control, and metadata's version must be bumped every time the definition changes. The reason this field exists is to confirm whether "last month's score and this month's score came from the same definition."

Generative tasks come with a filter chain attached. This is where the parsing rule becomes explicit.

task: internal_math
dataset_path: json
dataset_kwargs:
  data_files:
    test: ./data/internal_math.jsonl
output_type: generate_until
test_split: test
doc_to_text: "Problem: {{question}}\nWrite out your solution, then on the last line answer only in the form 'Answer: number'.\n"
doc_to_target: "{{answer}}"
generation_kwargs:
  until:
    - "\n\n"
  do_sample: false
  temperature: 0.0
  max_gen_toks: 512
filter_list:
  - name: strict-match
    filter:
      - function: regex
        regex_pattern: "Answer:\\s*(-?[0-9][0-9,]*(?:\\.[0-9]+)?)"
      - function: take_first
  - name: last-number
    filter:
      - function: regex
        regex_pattern: "(-?[0-9][0-9,]*(?:\\.[0-9]+)?)"
        group_select: -1
      - function: take_first
metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true
    ignore_case: true
    ignore_punctuation: false
metadata:
  version: 1.0

Two filters each produce a score from the same run. The difference between the two scores isn't the model's math ability — it's how well the model follows the output format it was instructed to use. Getting in the habit of looking at these two numbers side by side matters. If strict is low and last-number is high, the model can do the math but isn't following the format, and that's a problem you can fix with the prompt. If both are low, it can't do the math, and that's a problem you have to fix by changing the model. Report only one number and this distinction disappears.

Look at how commas are handled. The regex includes [0-9,]*. Without it, if the model answers "1,024," only "1" gets extracted and it's marked wrong. Answer-parsing rules are an accumulation of decisions like this one, and that accumulation is the score.

The Minimum Checklist for Reproducibility

Recreating a single score requires everything below. Miss even one item and what you have isn't a reproduction — it's an approximation.

The last item is often ignored, but it genuinely matters. Even with identical weights, float16 and bfloat16 can produce different answers; batch size changes padding and kernel selection; and vLLM and transformers can emit different tokens for the same prompt. Differences at this layer are usually small, but they're often larger than the gap that separates first place from third on a leaderboard.

The cheapest way to handle this list in practice is to make the eval run a file, not a command a human types.

# evals/run-2026-08-02.yaml — commit this file itself
harness: lm-eval==0.4.12
model:
  provider: local-chat-completions
  base_url: http://localhost:8000/v1/chat/completions
  name: qwen3-8b
  weights_revision: 8f2c1d9
sampling:
  temperature: 0
  max_gen_toks: 512
tasks:
  - name: gsm8k
    num_fewshot: 5
    apply_chat_template: true
    fewshot_as_multiturn: true
    report_filters: [strict-match, flexible-extract]
repeats: 3
seed: 1234

With this file, "how did you get that score" can be answered with a single commit hash. Without it, no matter how diligent someone is, they won't be able to answer three months later.

The Grader Is the Benchmark

Everything up to here converges on one conclusion. A benchmark's identity isn't its dataset — it's its grader. Put the same 14,000 questions in front of a different grader, and it's a different exam.

Grading methods break down into roughly three types, and each has a different way of silently breaking.

We already saw regex extraction's fragility with GSM8K. Model grading's bias is covered in detail in the next post. Execution-based grading's trap — weak tests letting wrong patches pass — has actually been measured in the SWE-bench family, and that's also the next post's subject.

Let me leave one practical rule here. When you adopt a new benchmark, read the grading code before the dataset. Reading 30 lines of a grading function will tell you far more precisely what a benchmark measures than eyeballing a handful of questions. And run ten of your own team's failure cases through that grader. If even one case that's obviously correct to a human gets marked zero, that benchmark's score is measuring something other than what you think it is.

Closing — A Number Without Conditions Isn't a Number

The incident where LLaMA 65B's MMLU was 63.6 and 48.8 at the same time wasn't an exceptional accident — it's a basic property of evaluation. A benchmark score is a single observation produced jointly by the model, the harness, the prompt, the parser, and the sampling settings together. Change even one of them and you get a different observation.

So in practice there are only two things to keep to. First, when citing an external score, check first whether you can verify what conditions it came from. If you can't verify it, that's not data — it's a claim. Second, when producing your own team's score, write the conditions down in a file. Flags typed on a command line vanish in three months, but a committed YAML survives.

The way to trust a number isn't to collect more numbers — it's to write down the conditions that produced it.

References

• Hugging Face — What's going on with the Open LLM Leaderboard? — the score gap between three MMLU implementations
• EleutherAI lm-evaluation-harness — task YAML, filter chains, CLI
• lm-eval on PyPI — for checking versions and release dates
• Stanford CRFM HELM — multi-metric evaluation framework
• Inspect AI (UK AI Security Institute) — the solver/scorer-separated design
• OpenAI simple-evals — the 0-shot CoT reference implementation
• MMLU-Pro (arXiv 2406.01574) — measuring prompt sensitivity
• OpenAI — promptfoo acquisition announcement (2026-03-09)
• Separating Signal From Noise in AI Coding Model Evals (related post)
• AI Agents & LLM Benchmarks 2026 (related post)