Split View: 검증 설계 — 테스트를 통과 여부가 아니라 신뢰 근거로 보기

Designing Verification — Tests as Grounds for Trust, Not Pass or Fail

• Green on Its Own Means Nothing
• Presence Can Be Shown, Absence Cannot
• Attach a Trust Sentence to Every Test
• What Coverage Measures and What It Cannot
• Write Down What You Decided Not to Verify
• Domains With No Oracle
• Try It This Week
• Further Reading
• Sources

Green on Its Own Means Nothing

CI is green. So what do you now know?

Plenty of green builds cannot answer that. The passing is there; what the passing guarantees is a sentence nobody holds. In that state a test suite is a gate in the deploy pipeline, not grounds for a judgment. Gates get forgotten once cleared; grounds get pulled back out when something goes wrong.

The difference shows right after a release. When an incident lands, somebody always asks whether the tests caught this. Being able to answer "that area was deliberately outside our verification scope" lets an organization make its next decision. Being able only to say "they should have" guarantees a repeat.

Presence Can Be Shown, Absence Cannot

The starting point for this whole subject is the sentence Edsger Dijkstra left in "Notes On Structured Programming" in 1970: program testing can be used to show the presence of bugs, but never to show their absence.

That line is often misread as an argument against testing. It is the opposite. It says tests are not a proof device but an evidence-collection device. With evidence, what matters is not how much you gathered but what it is evidence for. So designing tests stops being about raising a count and becomes about choosing which claims you intend to support.

Part 1 argued that performing a judgment gets automated while designing the judgment stays. Testing is the cleanest instance of that sentence. Choosing a good test is expensive; running it ten thousand times is free.

Attach a Trust Sentence to Every Test

There is one cheap way to move this into practice: attach a sentence to each test. If this is green, what do I no longer have to worry about?

Example — trust sentences attached to tests

test_refund_exceeds_payment_is_rejected
  -> I can trust that a refund larger than the payment never lands in the database

test_order_service_returns_200
  -> (no sentence available) nothing gets less worrying when this one is green

The second kind is more common than you would guess: a test that executes without asserting anything. Those are worse than nothing, because they pad the count, produce reassurance, and judgments get made on top of that reassurance.

Writing the sentences produces a side effect. You see the same sentence attached to several tests, which is duplication, and you see important sentences with no test attached at all. Those empty slots are the tests to write next.

What Coverage Measures and What It Cannot

Coverage is useful and routinely misread. What it measures is whether execution passed through a line during the test run. Not whether the result of that line was checked.

A test with no assertions raises coverage, because it passes as long as nothing throws. So the number carries almost no information upward and only carries information downward. Twenty percent tells you something is definitely missing; ninety percent contains no statement that you are safe.

More useful signals exist. Does the code path that caused your last incident have a test. Do the irreversible behaviors have tests. And how many flaky tests are there. The last one matters most: when a test goes red occasionally, people learn to ignore red, and that lesson does not stay confined to the one test.

Write Down What You Decided Not to Verify

This is the hardest item here to practice and the most valuable. Record not what you verified but what you decided not to verify.

Since nothing can be verified exhaustively, every team is already making this choice. They just make it implicitly, which means nobody ever reviews it. Making it explicit produces three things.

First, when an incident hits, you can tell a known risk from an unknown one. Second, the decision to accept a risk moves up from one person's judgment to a team decision. Third, showing people the not-verified list draws objections, and the objections tell you which items actually mattered.

The format can stay simple. Three lines of what this change verified, three lines of what it did not, and one line of why not is enough.

Domains With No Oracle

The hardest territory right now is the one with no answer key: search result quality, whether a recommendation fits, whether generated text is useful. Tests that split into pass and fail are unavailable here.

What you need then is not a test but a designed judgment, on three axes.

First, a reference set. Collect human-judged cases and freeze them into a fixed set. It does not need to be large, but it must contain hard and contested cases. Assemble only easy cases and the score stops moving no matter what you change.

Second, proxy metrics. You cannot measure the thing you want, so you measure something adjacent. What must be written alongside it is how the proxy can be gamed. Fail to write the gaming path in advance and someday the metric climbs along that path with nobody noticing.

Third, sampled review. Where automated judgment is impossible, people look at samples on a schedule. What matters here is drawing at random rather than looking at good ones. Hand-picked review teaches nothing.

Try It This Week

Open five tests you wrote recently and attach a one-line trust sentence to each. If even one of them resists, that is the most important finding this article can give you.

• SLO and Error Budget Calculator — treat reliability as a budget rather than a target and the question of how much to verify becomes an arithmetic problem.
• Logical Reasoning Trainer — includes drills on separating necessary from sufficient conditions, the same structure as saying precisely what a test guarantees.

Where this does not apply: in an exploratory phase where requirements change weekly, a thick test suite becomes a wall against change. There it is better to attach verification only to the irreversible parts and leave the rest bare. Verification density should track the cost of undoing, not the importance of the code.

Further Reading

• Related post on this blog: Why PoCs Never Reach Production — Success Criteria, Security Review, Handover

What Stays Expensive series

• Previous: Reading What Others Wrote — Entering a Codebase and Reverse-Engineering an Undocumented System
• Next: Writing as an Engineering Skill — What Design Docs, Postmortems, and Review Comments Actually Do

Sources

• Edsger W. Dijkstra quotations — Wikiquote — the sentence that program testing shows the presence of bugs but never their absence is sourced to "Notes On Structured Programming" (EWD249), 1970. Read 2026-08-15.
• The trust sentence, the explicit not-verified list, and the three axes of judgment design are not from that source; they are the procedure assembled in this post.