How Text, Images, and Agents Are Each Measured — Why the Three Domains Measure Fundamentally Different Things

• Introduction — Measuring Different Things With the Same Word
• Text (1) — What Multiple Choice Measures, and What It Can't
• Text (2) — Generative Tasks and LLM Judges
• Images (1) — What Generation-Quality Metrics Actually Measure
• Images (2) — Grading Comprehension Tasks, and the Caption Trap
• Agents — The Domain Hardest to Measure
  • Squeezing Partial Success Into 0 and 1
  • Environment State Is Part of the Score
  • The Same Model Produces a Different Score Every Time
  • Time and Cost Caps Make the Score
  • What You Miss by Looking Only at Success Rate — Safe Failure vs. Dangerous Failure
• The Three Domains on One Page
• Closing — Ask About Procedure, Not the Unit of the Number
• References

Introduction — Measuring Different Things With the Same Word

Model release materials put three kinds of numbers side by side. MMLU 84.2, MMMU 72.1, SWE-bench Verified 71.4. All three are percentages, and all three are labeled "higher is better." So they look like the same kind of number.

But the three have entirely different measurement structures.

• The first number is often produced without the model generating an answer at all. It may just be a comparison of the probabilities of the option strings.
• The second number has a human-defined correct answer, but the grading rule has arbitrary constants baked in, like a 5 percent error tolerance or an edit-distance threshold.
• The third number is the result of spinning up a container, running commands, and running tests. Run the same model again and you get a different value.

The moment you line the three numbers up and say "this model is strong across the board," you've mistaken three different measuring instruments for a single scale. This post keeps asking the same question — what has to be true for this number to mean what it appears to mean?

How harnesses actually produce these numbers is covered in Taking Apart LLM Benchmark Tooling. Here we look at the layer above that — the measurement logic that differs from domain to domain.

Text (1) — What Multiple Choice Measures, and What It Can't

Multiple-choice benchmarks are beloved because grading them is free. The correct answer is a single label, so you need neither a human nor a judge model. In exchange, you give up three things.

First, the correct answer itself may be wrong. Are We Done with MMLU?, a paper that re-audited MMLU, had 14 experts re-read 3,000 questions across 30 subjects. The overall error rate came out to about 6.49 percent, but the real issue was the distribution. Virology had errors in 57 percent of its sample (30 percent had a wrong answer label, 15 percent had an ambiguous question), and Logical Fallacies and College Chemistry both exceeded 20 percent. In other words, a substantial share of MMLU's total score was measuring "how well you match a wrong answer key." That fact disappears entirely the moment you average scores across subjects.

Second, the format itself creates shortcuts. With four options, guessing gets you 25 points. On top of that, models can narrow down the answer using surface cues between options — length, specificity, eliminability. That's exactly why MMLU-Pro expanded to 10 options, and as a result the score swing from prompt changes dropped from the 4-5 percentage-point range to around 2 percentage points (arXiv 2406.01574). The shrinking swing is itself evidence of how loose the original format was.

Third, the order of the options bleeds into the score. Large Language Models Are Not Robust Multiple Choice Selectors shows that models tend to prefer an option symbol (A or D) in a particular position, and that simply shuffling the option order moves scores significantly. A multiple-choice score that wasn't run multiple times with shuffled order and averaged is a value mixing the model's knowledge with the model's positional taste.

The metric is high but reality is bad (text multiple choice). It's common for a base model that scores high on MMLU graded by log-likelihood to be unusable once plugged into a service, because it doesn't follow instructions. Log-likelihood scoring never has the model generate anything, so it excludes the ability to follow instructions from what's being measured in the first place. The score didn't lie — we're the ones who read that score as the answer to a different question.

Text (2) — Generative Tasks and LLM Judges

Move to tasks with multiple valid answers — summarization, support replies, code explanations — and grading itself becomes a research problem. N-gram overlap metrics like BLEU or ROUGE dock a good answer for different phrasing, and rescue a wrong answer for matching phrasing. So practice has shifted to LLM judges.

Judge model performance itself isn't bad. In the measurements from Zheng et al. (NeurIPS 2023), the paper that built MT-Bench and Chatbot Arena, agreement between the GPT-4 judge and human experts was 85 percent excluding ties, and under the same conditions, agreement between humans themselves was 81 percent. It was 87 percent in a crowdsourced setting. The objection that judges are worse than humans loses its force in front of these numbers.

The problem is the structure of the bias. The same paper's other measurements are far heavier.

Just flipping the order of the two answers makes even GPT-4 change its verdict once in every three times. Claude-v1 gave the win to whichever answer came first, three times out of four. Run an A/B comparison just once under these conditions and conclude "the new prompt won," and a substantial share of that conclusion was really order.

Length bias is quantified too. In the same paper's "repeated list" attack — padding an answer into list form without adding any new content — Claude-v1 and GPT-3.5 were fooled 91.3 percent of the time, while GPT-4 was fooled 8.7 percent of the time. The magnitude of length bias differs by a factor of ten depending on the judge.

Here's how the mitigations line up.

Length control has actually been shown to work. AlpacaEval 2.0's length-controlled win rate is a metric that regresses out the effect of answer length, and this correction alone raised the Spearman correlation with the Chatbot Arena ranking from the low 0.9s to 0.98. Put another way, a substantial share of the pre-correction metric's disagreement was, in fact, length.

Human-preference leaderboards aren't a safe zone either. The Leaderboard Illusion (arXiv 2504.20879) reports that on Chatbot Arena, some providers could privately test multiple variants before release and keep only the one they liked. One provider tested 27 private variants before publishing one. Data allocation was uneven too — Google and OpenAI received 19.2 percent and 20.4 percent of all battle data respectively, while 83 open-weight models together received 29.7 percent, and the paper estimates that data volume alone could produce up to a 112 percent relative performance gain in the arena distribution. The Arena team accepted the criticism and announced plans to improve.

The metric is high but reality is bad (generative). A situation where the judge win rate rose from 55 percent to 62 percent, but user satisfaction stayed flat. If the only change was adding "explain in detail" to the prompt, which made answers 1.6x longer, then what went up wasn't quality — it was the judge's preference for length. This distinction is invisible unless you look alongside the length-controlled win rate.

Images (1) — What Generation-Quality Metrics Actually Measure

FID, the most widely cited metric in image-generation evaluation, does not, despite its name, measure the quality of a single image. Here's how it's computed.

1. Prepare N real images and N generated images (conventionally N = 50,000)
2. Resize each image to 299x299 and pass it through Inception-v3
3. Extract the 2048-dimensional feature vector from the pool3 layer
4. "Assume" each set of features follows a multivariate normal distribution and estimate mean and covariance
5. Compute the Frechet distance between the two normal distributions → this value is FID (lower is better)

Four assumptions are hiding here, and all four get violated.

Assumption 1: Inception-v3 features represent image quality. Inception-v3 is a model trained in 2015 to classify 1,000 ImageNet classes. That's too narrow to represent the diversity of content today's text-to-image models produce.

Assumption 2: The features follow a normal distribution. They don't. Rethinking FID (CVPR 2024, arXiv 2401.09603) shows this assumption is wrong, and lays out that FID disagrees with human raters' judgment, fails to reflect the incremental improvement of iteratively refined models, fails to capture the degree of distortion, and produces inconsistent results depending on sample size. As an alternative, it proposes CMMD, which uses CLIP embeddings and maximum mean discrepancy.

Assumption 3: Sample size has no effect. FID is a biased estimator, so the value grows larger the smaller the sample. That's exactly why the convention of 50,000 images became fixed — when papers use different sample counts, the numbers aren't comparable.

Assumption 4: Preprocessing is neutral. This is the most shocking part. clean-fid (CVPR 2022) measured that resize implementations differ across libraries, producing different FID values from the same image. Taking a correctly implemented PIL bicubic as the reference, OpenCV, PyTorch, and TensorFlow's bicubic implementations each produced an FID difference of 6 or more. Without changing the model at all — just by changing the resize function — you get a gap larger than the improvement papers boast about. JPEG compression is the same story: compression-quality differences invisible to the human eye move FID.

So recent text-to-image evaluation has shifted from distribution distance toward component-level verification. GenEval uses an object detector to check, item by item, whether the requested object is actually present, whether the count is right, whether the color is right, whether the spatial relationships are right. DPG-Bench gives dense, paragraph-length prompts and uses a VQA model to check each semantic element. Because the grading rationale is a human-readable item, this approach is far more diagnostic than FID. Of course, this approach also imports the detector's and VQA model's own errors directly into the grading.

The metric is high but reality is bad (image generation). A case where FID "improved" from 12.4 to 9.8, but human evaluation actually got worse. The cause could be a resize library swap in the preprocessing pipeline, or a JPEG quality change in the reference image set. If you report an FID value without also stating the resize implementation, sample count, and reference set, that number can't be verified by another team.

Images (2) — Grading Comprehension Tasks, and the Caption Trap

Evaluating the ability to read images, rather than generate them, looks similar to text multiple choice, but the grading rules have domain circumstances baked in. Let's look directly at the definitions of three representative metrics.

VQA Accuracy (VQAv2, TextVQA)
  score = min(number of people who gave that answer / 3, 1)
  → Full marks if 3 or more out of 10 people gave the same answer.
    Full marks is simply impossible on questions where people's answers split.

ANLS (DocVQA)
  NL = normalized edit distance(prediction, ground truth)
  score = 1 - NL   (when NL < 0.5)
        = 0        (otherwise)
  → An OCR typo of one or two characters gets partial credit; more than half wrong gets zero.
    The 0.5 threshold is a judgment call: "wrong by this much means it's no longer an OCR error."

Relaxed accuracy (ChartQA)
  A numeric answer is correct if it's within 5% relative error
  → A concession that reflects the reality of tasks reading values off a chart.
    At the same time, in domains where that 5% matters, it grades a wrong answer as correct.

All three definitions are reasonable. At the same time, all three embed an arbitrary constant. 3 people, 0.5, 5 percent. These constants are justified in their respective papers, but there's no guarantee the same value is right for your domain. If you're building a service that reads numbers off financial statements, ChartQA's 5 percent tolerance is a disaster. A model that scores 90 on ChartQA may be unusable for your task, and that's not the model's fault — it's ours, for taking the metric as-is.

The option-count problem repeats here too. MMMU-Pro (arXiv 2409.02813) expanded MMMU's options from 4 to 10 and embedded the question text inside the image itself, so the task can only be solved by actually looking at the image. Simply increasing the option count alone dropped GPT-4o(0513)'s score from 64.7 to 54.0 — a 10.7 percentage-point drop. That 10.7 points was elimination logic, not comprehension.

Caption evaluation is an even older trap. CIDEr, SPICE, and BLEU measure n-gram or graph overlap against reference captions, but there are usually only 5 reference captions, and the score drops even with a slight change in phrasing. CLIPScore (EMNLP 2021) showed that measuring only the embedding similarity between image and caption, with no reference at all, agrees better with human judgment. That said, the same paper is explicit about its limits — it's weaker than reference-based metrics when the task needs context knowledge outside the image, like captioning a news photo. There's no universal metric; you have to pick the metric to match the nature of the captioning task.

The metric is high but reality is bad (image comprehension). A document parser with a DocVQA ANLS of 0.91 got plugged into contract-amount extraction, and it caused an incident. Because ANLS is an edit-distance-based partial-credit metric, reading "1,250,000" as "125,000" still gets a score near 0.9, since the edit distance is small. Amount fields need to be graded by exact match, not partial credit. You have to check first whether the metric is one that gives partial credit for the kind of task you're using it for.

Agents — The Domain Hardest to Measure

The two domains above each map one input to one output. Agents don't. One input maps to dozens of actions, those actions change the environment, and once the environment changes, the meaning of the next action changes too. Here, most of the premises of measurement break down.

Squeezing Partial Success Into 0 and 1

Most agent tasks are multi-step. A run that does 9 out of 10 steps perfectly and fails at the last one, and a run that flounders from the first step, both get the same score of zero. The information loss is large.

Different benchmarks handle this differently. AppWorld checks final state, but reports task-level completion and scenario-level completion separately. GAIA splits into difficulty levels to show which tier things break down at. OSWorld attaches a separate execution-based verification script per task to confirm "does the result file actually look like this." Either way, cite a single success rate alone and this whole structure disappears.

Environment State Is Part of the Score

Agent benchmarks use live environments. Web page DOMs change, software gets updated, CAPTCHAs pop up, loading slows down. These changes have nothing to do with model performance, but they show up in the score anyway.

That's exactly why the OSWorld team released OSWorld-Verified on July 28, 2025. They confirmed and fixed over 300 reported problems — web structure changes, timing dependencies, task ambiguity, verification functions that didn't accept multiple valid solution paths, grading that was too strict or too loose. In other words, the OSWorld scores published before that were, to a substantial degree, measuring the state of the environment. The current human baseline is estimated at about 72 percent, with top systems in the 60s.

The SWE-bench family runs into the same problem a different way. Because its grading is execution-based, it looks semantically robust, but if the tests are weak, a wrong patch passes too. Studies auditing SWE-bench's quality report that 32.67 percent of successful patches had the answer leaked in the issue description, and 31.08 percent passed because the tests were inadequate. Even in SWE-bench Verified, more than 15 percent of instances needed test reinforcement, and estimates put the leaderboard success rate as inflated by 6-7 percentage points.

The Same Model Produces a Different Score Every Time

Even with temperature set to 0, agent execution is not deterministic. Tool-call ordering, network latency, timeouts, retries, container resources — all of it changes the outcome.

The τ-bench family built this non-determinism directly into the metric. If pass@k is "succeeds at least once in k tries," then pass^k is "succeeds all k times." For a task with success rate p, pass^k shrinks to roughly p raised to the k-th power. Even an agent with a 90 percent success rate drops to 0.9 raised to the 8th power — about 43 percent — when k is 8. Flip it around, and that means there's roughly a 57 percent chance of at least one failure somewhere in eight tries. Mixing up which of these two numbers you're citing is a common mistake, so always double-check which one you mean. In the original τ-bench paper's report, the best function-calling agents of the time couldn't solve even half the tasks, and pass^8 in the retail domain was under 25 percent.

This metric matters because the question that actually matters in production is not pass@k but pass^k. No service gets to retry a single customer request eight times and call it a success if one of the eight works.

Infrastructure configuration moves the score too. Anthropic's own measurements found that adjusting only the execution resources, without touching the model weights at all, produced a 6-percentage-point difference on Terminal-Bench 2.0. Details are laid out in the signal-vs-noise post. For reference, Terminal-Bench 2.0 was released in November 2025 with 89 tasks, and runs on the Harbor framework.

Time and Cost Caps Make the Score

Agent evaluation always has a cap: a maximum step count, a wall-clock time, a token budget. Double that cap and the score goes up; halve it and the score goes down. And yet it's rare to find that cap stated alongside a published agent score.

Comparing two scores whose caps aren't stated is meaningless. More precisely, that comparison may be a comparison of budgets, not of model ability. In practice, when choosing an agent, you have to look at "average tokens per task" and "average time per task" right alongside the score. If a success rate is 5 percentage points higher but costs three times as much, that isn't a better model — it's a more expensive configuration.

What You Miss by Looking Only at Success Rate — Safe Failure vs. Dangerous Failure

This is the least-measured part of agent evaluation. Success rate treats every failure as one lump. But in practice, the kinds of failure are worlds apart.

Safe failure                            Dangerous failure
------------------------------          ------------------------------
Says "I can't do this" and stops        Confidently returns a wrong answer
Asks a human to confirm                 Performs an irreversible action without confirmation
Only reads, then exits                  Writes to the wrong target, then exits
Aborts on exceeding budget              Aborts after a partial edit (state left broken)
Reports the test failure                Edits the test itself to make it pass

On a success-rate metric, all ten of these are the same zero. And yet the right column breaks your service, while the left column just needs a human to pick up where it left off.

Benchmarks that directly measure harmful behavior have appeared too. AgentHarm (arXiv 2410.09024) built 110 explicitly malicious agent tasks (440 with augmentation) across 11 harm categories — fraud, cybercrime, harassment, and more — and measures both refusal rate and whether capability is retained after jailbreaking. The reported figures are uncomfortable — GPT-4o completed 48-55 percent of harmful tasks even without jailbreaking, and applying a generic jailbreak template raised compliance to 73 percent while dropping the refusal rate from 49 percent to 14 percent.

Here, the direction of the metric flips. In AgentHarm, the higher the task-completion rate, the worse it is. This is the point where it becomes clear just how shallow an assumption "success rate: higher is always better" really was.

The metric is high but reality is bad (agent). A new model with a 5-percentage-point higher SWE-bench Verified score got adopted, and the review burden went up. If the new model was tampering with the tests themselves to make them pass, or reproducing an answer that leaked into the issue, the benchmark score goes up while real value goes down. This distinction is invisible unless you classify and count "how it failed when it failed" alongside the success rate.

The Three Domains on One Page

There's one pattern that repeats across this table. Every metric threw something away to make itself computable. If what it threw away is something that matters to your service, that metric is useless to you. Choosing a metric is choosing what you're allowed to throw away.

Closing — Ask About Procedure, Not the Unit of the Number

Having gone through all three domains, a common thread shows up. In no domain does a metric measure ability directly. It measures the shadow of ability, and only under a particular light at that. Multiple choice, under the light of answer-label accuracy and option construction; FID, under the light of the Inception feature space and the resize implementation; agent success rate, under the light of environment state and resource caps.

So in practice, when you encounter a new benchmark number, what you should ask isn't "what's the score" — it's three things. First, what procedure was used to compute this score? Second, is there anything among what that procedure threw away that matters to us? Third, what comes out if we apply the same procedure to our own data?

That third question ultimately leads to your own eval set. That story continues in Building an Eval Set for Your Own Service.

References

• Are We Done with MMLU? (arXiv 2406.04127) — re-auditing MMLU's error rate
• MMLU-Pro (arXiv 2406.01574) — expanded options and prompt sensitivity
• Large Language Models Are Not Robust Multiple Choice Selectors (arXiv 2309.03882)
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685) — measuring judge agreement and bias
• Length-Controlled AlpacaEval (arXiv 2404.04475) — length-controlled win rate
• The Leaderboard Illusion (arXiv 2504.20879) — structural bias in the arena leaderboard
• Rethinking FID / CMMD (arXiv 2401.09603)
• clean-fid (CVPR 2022) — the effect of resize and compression on FID
• GenEval (arXiv 2310.11513) — object-based prompt-fidelity evaluation
• MMMU-Pro (arXiv 2409.02813)
• CLIPScore (arXiv 2104.08718)
• OSWorld-Verified (2025-07-28) — the impact of environment change on scores
• τ-bench (arXiv 2406.12045) — the pass^k metric
• AgentHarm (arXiv 2410.09024) — a benchmark that directly measures harmful behavior
• Taking Apart LLM Benchmark Tooling (related post)
• Separating Signal From Noise in AI Coding Model Evals (related post)
• AI Agents That Operate Browsers and Computers (related post)