필사 모드: In Eval-Driven Development, the First Thing to Calibrate Is the Judge

• The Judge Score Went Up and So Did the Support Tickets
• Where Eval-Driven Development Decisively Differs from TDD
• A Judge Is Not Code, It Is a Deployed Model
• Calibrating the Judge with a Golden Set of 50 to 100 Examples
• Why You Should Not Take 88 Percent Agreement at Face Value
• Three to Five Judges Beat Twenty to Thirty
• You Cannot Grade an Agent on the Final Answer Alone
• What to Gate On and What to Leave on the Dashboard
• References

The Judge Score Went Up and So Did the Support Tickets

You touch the prompt once, and the faithfulness score on the internal eval dashboard climbs from 0.82 to 0.90. You ship. The next week, escalations to human agents go up. Open the logs and the answers are longer, more polite, and more confidently wrong.

At this point most teams conclude "our evals were not thorough enough" and grow the eval set. A hundred cases become five hundred, five criteria are split into twenty. But the real cause is not the size of the eval set. The model you handed the grading to was awarding more points to long, polite answers, and nobody had ever checked that. Grow the eval set fivefold and you simply measure that bias five times more precisely.

Where Eval-Driven Development Decisively Differs from TDD

Eval-driven development, published by Airbnb Engineering on July 28, 2026, argues that evaluation should be treated as a first-class engineering discipline rather than a side task. Up to this point the slogan is the same as TDD.

The difference is the arbiter. In TDD the arbiter is assert, and this arbiter does not get things wrong. In eval-driven development a large share of the arbitration is done by an LLM, and this arbiter does get things wrong. Not randomly, either — it gets things wrong consistently in particular directions: length, formatting, confidence, a preference for models from its own family.

So one step gets added to the order. Writing the test first is not enough. You have to first confirm that the test measures with a straight ruler.

A Judge Is Not Code, It Is a Deployed Model

Once you accept this framing, the way you handle a judge changes. A judge prompt is not a config file, it is something you deploy. It needs a version, it needs a regression check when it changes, and which model and which temperature it ran with has to be stored alongside the results.

The three-layer structure the Airbnb post lays out takes on meaning here. Deterministic code checks are layer one, the judge model is layer two, humans are layer three. The point is not that the layers were separated but the relationship: layer three continuously audits layer two. Humans do not look at every output. They look at how far the judge diverges from humans.

What you put in layer one matters too. JSON schema violations, banned terms, assertions without citations, response length caps — everything expressible as a rule should be filtered out at layer one. This is not a cost question but a trust question. If you hand a failure that a rule could catch to the judge, then when the judge is wrong that failure passes. Handing something deterministically decidable to a probabilistic decider is always a losing trade.

Calibrating the Judge with a Golden Set of 50 to 100 Examples

The Airbnb post organizes judge calibration into five steps. Build a golden set, run the judge over it, measure agreement, dig into the disagreements, and recalibrate periodically. The recommended scale is "a golden set of 50 to 100," and it says to start labeling from "20 to 100 rows labeled by subject-matter experts."

There is a reason these numbers look small. A golden set is not a ruler for the model's performance, it is a ruler for the judge's performance. Checking a ruler does not take many samples. What it does take is failure cases mixed in. On a golden set assembled only from good answers, even a judge that catches nothing scores full marks.

In the real example from the same post, the first faithfulness judge agreed with humans 78 percent of the time. After the rubric was refined and a few examples were added it rose to 88 percent, and only then was it put to work at scale.

Why You Should Not Take 88 Percent Agreement at Face Value

One more step is needed here. Plain agreement inflates when the label distribution is skewed. On an eval where 90 percent pass, a judge that stamps "pass" unconditionally also earns 90 percent agreement. So you have to look at a metric with chance agreement stripped out, and the Airbnb post explicitly says to use Cohen's kappa or Krippendorff's alpha.

"""Judge calibration: never look at agreement alone, look at kappa alongside it."""
from collections import Counter
from typing import Sequence


def cohens_kappa(human: Sequence[str], judge: Sequence[str]) -> float:
    assert len(human) == len(judge) and human, "paired labels are required"
    n = len(human)
    po = sum(h == j for h, j in zip(human, judge)) / n  # observed agreement
    hc, jc = Counter(human), Counter(judge)
    pe = sum((hc[k] / n) * (jc[k] / n) for k in set(hc) | set(jc))  # chance agreement
    return 1.0 if pe == 1 else (po - pe) / (1 - pe)


def calibration_report(rows):
    """rows: [(input_id, human_label, judge_label), ...]"""
    human = [r[1] for r in rows]
    judge = [r[2] for r in rows]
    k = cohens_kappa(human, judge)
    po = sum(h == j for h, j in zip(human, judge)) / len(rows)
    # you only catch the bias if you look at the direction of the disagreement too
    lenient = sum(h == "fail" and j == "pass" for h, j in zip(human, judge))
    strict = sum(h == "pass" and j == "fail" for h, j in zip(human, judge))
    return {
        "agreement": round(po, 3),
        "kappa": round(k, 3),
        "judge_too_lenient": lenient,   # missed failures — the most expensive error
        "judge_too_strict": strict,
        "verdict": "ship" if k >= 0.6 and lenient == 0 else "recalibrate",
    }


rows = (
    [(f"g{i}", "pass", "pass") for i in range(80)]
    + [(f"b{i}", "fail", "pass") for i in range(8)]   # failures the judge missed
    + [(f"c{i}", "fail", "fail") for i in range(12)]
)
print(calibration_report(rows))
# {'agreement': 0.92, 'kappa': 0.694, 'judge_too_lenient': 8, ..., 'verdict': 'recalibrate'}

The reason the verdict is "recalibrate" despite 92 percent agreement is in the last line. Of the 20 cases humans judged as failures, the judge passed 8. Wire this judge into CI as it stands and the team will never again see that class of failure.

Three to Five Judges Beat Twenty to Thirty

To carry over the Airbnb post's phrasing directly, three to five well-calibrated judges beat twenty to thirty noisy ones. It is more accurate to read that sentence not as "have fewer criteria" but as "create only as many criteria as you can afford to calibrate."

The cost of maintaining a single judge is not one blob of prompt. It is a set: 50 to 100 golden cases, the expert hours that went into labeling, recalibration every time the failure modes shift, and a pipeline that checks for regressions when the judge prompt changes. A criterion whose set you cannot afford is better left uncreated. An uncalibrated judge is not merely worse than nothing, it is a compass pointing the wrong way.

You Cannot Grade an Agent on the Final Answer Alone

Move to agents and another axis appears. The same post frames agent evaluation as spanning three dimensions: the final output, the intermediate reasoning steps, and the tool calls with their arguments.

The reason is simple. The fact that the final answer was right tells you nothing about whether that run will reproduce. A run that spun through three fruitless searches and happened to land on a cached answer and a run that called the right tool with the right arguments on the first call get the same score but are completely different systems. The first one collapses next week.

In practice you normalize tool call names and arguments, store them as a trajectory, and count at least three things separately: the number of unnecessary tool calls, the number of retries caused by wrong arguments, and how much the trajectory changes when you feed the same input again. The third matters most. If the trajectory varies wildly for the same question every time, that agent does not yet know the method, and there is no guarantee today's success rate survives the next deploy.

When you use a judge to grade trajectories, write the rubric about procedure rather than outcome. Not "is this a good answer" but "was calling this tool at this step necessary." Procedural questions have short answers whose evidence remains in the log, so the same judge lands noticeably higher agreement with humans than it does on outcome questions.

What to Gate On and What to Leave on the Dashboard

The last piece is the connection to deployment. Gate on every metric in CI and the pipeline stays permanently red; gate on nothing and evaluation becomes decoration. Drawing the line this way is practical.

And you have to keep pulling samples from production. The Airbnb post states that it samples 5 percent of de-identified live traffic daily. An eval set starts aging the day you build it, so without a path for new failure modes to flow into the golden set, a few months later you are repeatedly measuring only the old problems.

The first task of eval-driven development is not building an eval set. It is knowing, as a number, how far the judge you are using right now diverges from humans. Without that number, every score on the dashboard means nothing yet.

References

• Eval-driven development: Lessons from evaluating GenAI at scale — Airbnb Tech Blog, 2026-07-28 — the three-layer structure, the golden set of 50 to 100, the calibration example going from 78 percent to 88 percent, and the 5 percent daily sampling all come from this post.
• Related posts on this blog: Building an Eval Set for Your Own Service, How to Stop Evaluating LLMs on Vibes
• The kappa computation above implements the standard definition directly, and the example numbers are fictional labels I put together for illustration. I recommend swapping in your own team data and running it.