Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Stopping at Fixed Versus Asking Why
Debugging Is Elimination, Not Generation
A Hypothesis Has to Be Falsifiable
Cutting the System Into Layers
Working a Bug That Will Not Reproduce
Why the Gap Does Not Track Years of Experience
Try It This Week
Further Reading

Stopping at Fixed Versus Asking Why

There is a bug. Two people take it.

One changes a suspicious spot and reloads. Nothing. Changes another spot and reloads. After several rounds, the symptom disappears. They commit and move on.

The other does not stop where the symptom disappeared. They retrace what they just changed and check whether that change explains the disappearance. If it does not explain it, the work is not finished. They also revert to see whether the symptom comes back.

The gap between these two barely shows on this bug. It shows three months later, when the same cause returns wearing a different face.

Debugging Is Elimination, Not Generation

Part 1 argued that the skills holding value are the ones expensive to verify. Debugging fits that definition almost perfectly, because the direction of the work is the reverse of writing code.

Writing code produces one of the possible things. Once produced, its existence is the deliverable. Debugging erases possible causes until one remains. Erasing leaves no artifact. What remains is a list of eliminated hypotheses, and that list usually gets recorded nowhere.

The inputs debugging needs are also not inside the code. Since when has it done this, what shipped recently, does the symptom hit only one customer, who touched which setting three weeks ago. That is not searchable text; it is state scattered across an organization. Gathering it is already half the job.

A Hypothesis Has to Be Falsifiable

Here is what separates someone who learned debugging from someone who has merely done it a long time: the shape of the hypothesis.

"Seems like a network problem" is not a hypothesis. Nothing is specified about what you would see if it were wrong, so the sentence survives every observation. A guess that survives everything narrows nothing.

A usable hypothesis looks like this. "The request is dropped between the gateway and the application. If so, the gateway access log holds a five-second timeout and the application access log has no record of this request id at all."

The difference is the second sentence. Something specific is nominated for checking, and if the check comes back differently the hypothesis dies. Only hypotheses that can die reduce the candidate set.

Example — three lines to write before you change anything

Hypothesis: the request is dropped between gateway and application
If true:    gateway log shows a 5s timeout, app log has no such request id
If false:   app log shows the request arriving and failing during processing

Observed:
Next hypothesis:

Writing those three lines takes a minute. The minute looks wasteful until you skip it and find yourself, thirty minutes later, re-checking something you already ruled out. On an incident with several responders the effect is larger: if the eliminated list is not shared, three people check the same thing three times.

Cutting the System Into Layers

The most reliable way to narrow hypotheses is bisection. Same principle as bisecting a commit history, except the objects are layers rather than commits.

First, write down in order the layers a request or a record passes through: client, DNS, load balancer, gateway, application, cache, database, external integration. Then pick the exact middle and ask whether things are still correct at that point.

The crucial part is picking a point where correctness can actually be judged. Cut where judgment is impossible and you erase nothing. That is why a good system is one with many cuttable points, and it is what observability substantively means.

Compare with walking the layers in order. Eight layers checked in sequence averages four checks; halving finishes in three. At twenty layers the gap becomes ten versus five. And in a real incident each check costs minutes, so this difference is recovery time.

Working a Bug That Will Not Reproduce

The hardest category is the one you cannot reproduce. It happens twice a day and never while you are watching.

The common failure here is making reproduction the goal. Days go into it and it ends with nothing. Better to change direction and move the goal from reproduction to the next observation.

The procedure: first list what evidence would let you identify the cause the next time the symptom occurs. Then check whether that evidence is currently being retained. Usually it is not. So add the instrumentation that retains it, and wait. Next time it happens, the evidence is in your hands.

The advantage of this approach is that failure still leaves something behind. Instrumentation you planted gets used on the next bug even if it misses this one. A failed reproduction attempt leaves nothing.

One caution. When the cause is timing or a race, the act of adding instrumentation changes the timing and hides the symptom. Then you need observation that stays out of the execution path: line up timestamps in logs already being written, sample to reduce overhead, or dump state after the fact.

Why the Gap Does Not Track Years of Experience

You would expect debugging to improve on its own with experience. It does not, for three reasons.

First, almost nobody teaches it explicitly. Languages and frameworks get taught; the narrowing procedure gets absorbed over someone's shoulder. Without a person nearby who has a procedure, ten years can pass inside the guess-and-change loop.

Second, the feedback is wired wrong. The symptom disappearing is rewarded. Not knowing why it disappeared is rewarded identically. So a lucky fix gets mistaken for competence.

Third, good debugging leaves no trace. The commit from someone who narrowed it precisely in thirty minutes looks the same as the commit from someone who flailed for three hours and got lucky. Organizations therefore have trouble seeing this capability, and what cannot be seen does not get developed.

The response is to leave a trace. Writing the eliminated hypotheses and how you checked them into the issue, even briefly, trains you and becomes a team asset at the same time.

Try It This Week

Pick one bug you hit this week and write the three lines above before you touch any code: hypothesis, what you would see if true, what you would see if false. If the third line will not come, what you have is not a hypothesis yet but a feeling.

Linux Terminal Simulator — safe repetition of checking logs, processes, and network state by hand.
FDE Career RPG — 31 missions that make you narrow a cause and commit a diagnosis under limited time and access, designed so that leveling a domain changes what you can read in the same log.

Where this advice does not apply: while the service is down, mitigation comes before diagnosis. Roll back or shift traffic to stop the bleeding, then narrow. Writing tidy hypotheses is what you do when users are not waiting.