How to Stop Evaluating LLMs on Vibes — Sample Size, Judge Bias, and CI Regression Tests

Introduction — the cost of "it seems better"

You have probably edited a prompt, run a few examples, and said "that is definitely better now." So have I. The problem is what comes next.

That change made some inputs better and some worse, but the only ones you checked were the few that caught your eye. The ones that got worse quietly survive into the next release. Repeat this twenty times and a system that "seemed better" at every step may be worse than where it started, and worse still, you have no way to confirm that. The baseline for each change was a different memory each time.

An evaluation system is not a device for raising model quality. It is a device for leaving behind a point you can return to. This post covers how to build that system starting from the cheapest layer, and how to interpret the numbers that come out of it.

The four layers of evaluation

You do not have to do all of them, but you do have to know which layer catches what.

The order matters. Many teams build the LLM-as-judge layer first. It is flashy and it shows up in papers often. Yet what actually prevents incidents is, overwhelmingly often, the first row.

Cheap assertions come first

Even for tasks where you cannot define a correct answer, you can define what must never happen. This is not statistics, it is unit testing.

import re, json

def assert_output_contract(resp, ctx):
    """things that must always hold regardless of model quality"""
    checks = []

    checks.append(("파싱 가능", is_valid_json(resp.text)))
    checks.append(("스키마 준수", validate_schema(resp.text) is True))
    checks.append(("잘리지 않음", resp.finish_reason != "length"))
    checks.append(("금칙 표현 없음", not FORBIDDEN.search(resp.text)))
    checks.append(("문맥 밖 URL 없음", set(urls(resp.text)) <= set(urls(ctx))))
    checks.append(("지연 상한", resp.latency_ms < 4000))
    checks.append(("출력 길이 상한", resp.output_tokens < 800))

    return [name for name, ok in checks if not ok]

The fifth line earns its keep especially well. A link the model produced that was not in the context is almost certainly a hallucination, and one line of set arithmetic catches it. The same approach lets you check whether a cited number actually appears in the context, and whether a citation index points at a chunk that exists. These are all things you can verify without knowing the correct answer.

This layer does not raise accuracy. What it does is effectively eliminate the "we had an outage because the format broke" class of incident. A substantial share of real incidents falls into that class, which is why the return on investment here is the best of any layer.

The golden dataset — composition over size

To measure accuracy you need inputs with answers attached. Two mistakes are common here.

First, collecting only easy examples. An evaluation set filled with cases that already work starts at 0.95 accuracy and does not move for any change. As a measuring instrument, it is dead. An evaluation set only produces signal when it deliberately includes the inputs that failed, the boundary conditions and the ambiguous inputs.

Second, building it once and never updating it. Over time the system memorizes the evaluation set, because you kept editing the prompt to fit those examples. Unless you periodically pull new examples from production logs, the evaluation set slowly turns into a training set.

The loop that grows the set from logs looks like this.

def harvest_candidates(logs, judge, sample_rate=0.02):
    """automatically pick out requests worth adding to the evaluation set"""
    out = []
    for r in logs:
        reason = None
        if r.user_feedback == "negative":     reason = "사용자 부정 피드백"
        elif r.retry_count > 0:               reason = "재시도 발생"
        elif r.assertion_failures:            reason = "어서션 실패"
        elif r.judge_score is not None and r.judge_score < 3:
            reason = "심사자 저점"
        elif random.random() < sample_rate:   reason = "무작위 표본"

        if reason:
            out.append({"input": r.input, "output": r.output, "reason": reason})
    return out   # the ground-truth labels are attached by a human

Do not remove that last random sample branch. Fill the set with failure signals only and it becomes biased toward the hard end, and you stop catching regressions that appear on normal requests.

LLM-as-judge bias and mitigation

Tasks with more than one right answer — summarization, support replies and the like — cannot be measured by accuracy. This is where a judge model comes in. But unless you handle three known biases, its scores cannot be trusted.

Position bias. Show two answers side by side and ask which is better, and the one presented first is favored. Mitigation is simple: evaluate twice with the order swapped and count a win only when the same side wins both times.

Length preference. Judges tend to give higher scores to long, detailed answers. The catch is that length does correlate with real quality sometimes, so you must not correct for it unconditionally. The practical approach is to record the output lengths of both candidates and check the correlation between win rate and length difference. A strong correlation means the win rate is likely measuring length rather than quality.

Self-preference. Judges have been reported to give higher scores to their own output or that of models in the same family. If one of the candidates is from the same family as the judge, run a judge from a different family too and check whether the result flips.

def judge_pairwise(judge, question, answer_a, answer_b, rubric):
    """evaluate twice with the order swapped and count a win only when they agree"""
    def ask(first, second):
        prompt = (f"{rubric}\n\n질문: {question}\n\n"
                  f"[답변 1]\n{first}\n\n[답변 2]\n{second}\n\n"
                  "먼저 기준별 근거를 쓰고, 마지막 줄에 '승자: 1' 또는 '승자: 2' 또는 '승자: 무승부'.")
        return parse_winner(judge(prompt))

    r1 = ask(answer_a, answer_b)        # A first
    r2 = ask(answer_b, answer_a)        # B first
    if r1 == "1" and r2 == "2":  return "A"
    if r1 == "2" and r2 == "1":  return "B"
    return "무승부"                      # if swapping the order flips it, no verdict

Here, "the share of cases where swapping the order flipped the result" is itself a useful metric. A high value means either the real gap between the two candidates is small or the judge cannot discriminate on this task, and in either case you must not make a deployment decision on that judge's scores.

And the most important step is still left: validating the judge. Take around 100 human-labeled examples and measure how well the judge's verdicts agree with the humans. If agreement is low, that judge is just automated vibes. Skipping this validation and putting judge scores on a dashboard is the most common self-deception in this field.

Twenty examples tell you nothing

You got 17 out of 20 right. That is 85 percent. How much should you believe that number?

import math

def wilson(successes, n, z=1.96):
    if n == 0:
        return (0.0, 1.0)
    p = successes / n
    denom = 1 + z * z / n
    center = (p + z * z / (2 * n)) / denom
    half = (z / denom) * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (max(0.0, center - half), min(1.0, center + half))

for n in (20, 60, 200, 1000):
    lo, hi = wilson(round(0.85 * n), n)
    print(f"n={n:4}  85% 관측 → 95% 신뢰구간 [{lo:.3f}, {hi:.3f}]  폭 {hi - lo:.3f}")
# n=  20  85% 관측 → 95% 신뢰구간 [0.640, 0.948]  폭 0.308
# n=  60  85% 관측 → 95% 신뢰구간 [0.739, 0.919]  폭 0.180
# n= 200  85% 관측 → 95% 신뢰구간 [0.794, 0.893]  폭 0.099
# n=1000  85% 관측 → 95% 신뢰구간 [0.827, 0.871]  폭 0.044

The true value behind 85 percent observed on 20 examples could be 64 percent or it could be 95 percent. Every change that moves within that interval is indistinguishable from noise. If you measured two versions on 20 examples each and got 80 percent and 85 percent, that result is entirely compatible with "no difference."

So how many do you need?

Z_A, Z_B = 1.96, 0.8416   # significance level 5%, power 80%

def n_unpaired(p1, p2):
    """number of examples per group when measuring the two versions on different examples"""
    num = (Z_A + Z_B) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2))
    return math.ceil(num / (p1 - p2) ** 2)

def n_paired(win_ratio, discordant_rate):
    """number of examples needed when comparing the two versions on the same examples.
    win_ratio: share of the disagreeing examples that the new version wins"""
    num = (Z_A / 2 + Z_B * math.sqrt(win_ratio * (1 - win_ratio))) ** 2
    need_discordant = math.ceil(num / (win_ratio - 0.5) ** 2)
    return math.ceil(need_discordant / discordant_rate), need_discordant

print(f"짝짓지 않음: 그룹당 {n_unpaired(0.80, 0.85)}개 (총 {2 * n_unpaired(0.80, 0.85)}개)")
total, disc = n_paired(0.70, 0.20)
print(f"짝지은 비교: 총 {total}개 (의견 갈린 {disc}개 필요)")
# 짝짓지 않음: 그룹당 903개 (총 1806개)
# 짝지은 비교: 총 235개 (의견 갈린 47개 필요)

The number of examples needed to reach the same conclusion dropped from 1806 to 235. The reason is simple. In an unpaired comparison, "variance in example difficulty" and "difference between versions" are mixed together, whereas comparing the two versions on the same examples cancels the difficulty term. If you cannot afford a large evaluation set, at minimum run both versions on the same examples. That one change alone shrinks the minimum detectable effect size by several times.

There are caveats worth stating honestly. The calculation above assumes the examples are independent of each other. Ten questions derived from the same document are not independent, and in that case the effective sample size is smaller than ten. This formula also handles the case where a result is either correct or incorrect; a scored scale such as 1 to 5 uses different math. And comparing ten prompt candidates against the same evaluation set raises the probability that one of them looks good by chance. When you compare many times, the bar has to be raised accordingly.

Putting it in CI, and the problem that temperature 0 is not reproducible

Evaluation that only runs by hand eventually stops running. It has to go into the pipeline.

What you run into there is nondeterminism. Setting the temperature to 0 feels like it should give the same output for the same input, but in practice it does not. The batch composition changes the order in which floating-point values accumulate, and that changes which kernel path is selected. When the top two tokens have nearly equal probability, that minute difference makes a different token win, and from there on you get a completely different sentence. If you are on a provider API, unannounced server-side changes get added on top.

So a CI test that compares output strings verbatim will inevitably break. Structure it like this instead.

def ci_gate(baseline, candidate, n, threshold=0.03):
    """decide by a per-metric drop threshold. Do not fail on a single-example mismatch."""
    failures = []
    for metric in ("accuracy", "assertion_pass", "citation_valid"):
        drop = baseline[metric] - candidate[metric]
        lo, hi = wilson(round(candidate[metric] * n), n)
        if drop > threshold and baseline[metric] > hi:
            failures.append(f"{metric}: {baseline[metric]:.3f} → {candidate[metric]:.3f} "
                            f"(구간 상한 {hi:.3f})")
    return failures

It fails only when the baseline sits above the candidate's confidence interval. Without that, noise keeps the pipeline red, and a few weeks later nobody looks at the red light anymore. Training people to ignore an alarm is worse than having no alarm.

The execution tiers have to be split too. The tier that runs on every PR should finish within minutes using assertions and a small golden set, while the full evaluation with a judge runs nightly or before a release. Skip this split and evaluation gets slow enough that someone turns it off.

Last comes online evaluation. However good the offline evaluation set is, it differs from the real traffic distribution. Roll out in stages and watch a guardrail metric that must never get worse alongside the metric you are trying to improve. Latency doubling or the abstention rate spiking while chasing accuracy really does happen often. And when the failures found there flow back into the golden set, the loop closes.

Closing — if the baseline is a memory, regressions are invisible

If you take one thing from this post, let it be this. The purpose of an evaluation system is not to get good scores, it is to freeze a comparable baseline in code. When the baseline lives in a person's memory, regressions are by definition unobservable.

So the minimum configuration you can start today is about this much: ten assertions that must always hold, a 50-example golden set filled with inputs that failed, and one script that runs the new version and the old version on the same examples and compares them. When the difference that comes out is inside the confidence interval, you get to say "I do not know." Being able to say that is the real difference from evaluating on vibes.