Skip to content
Published on

The FDE Incident Diagnosis Playbook — Six Steps from Access to Report

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Why Incidents in Unfamiliar Environments Are Different

When your own team's service breaks, your body knows where to look: the dashboard URL, the log locations, the recent deploy history are all in your head. A customer-site incident starts with all of those premises gone. You do not know what the monitoring is, where the logs pile up, or what changed yesterday — and the customer is beside you asking when it will be fixed.

What this condition demands is not superior intuition but a fixed order. This post is that order written down as a six-step playbook. One case runs through the whole piece: a report that the payments API has been intermittently returning 504 since the afternoon. It is not a real incident but an example constructed for this post, and every command shown uses only public, general-purpose tools such as kubectl, curl, and grep.

Step 1 — Confirm Access and Permissions

Instinct says open the logs first, but in an unfamiliar environment the first step is confirming access. Which environments can you enter, by which path? Are your current permissions read or write? Is this production or staging? Two reasons. First, discovering mid-diagnosis that you lack a permission wastes the entire round-trip of requesting and approving it. Second, acting beyond your permissions is itself an incident. One over-privileged command in a customer's production loses more trust than the outage did.

The checklist is short. Is the access path alive? What is the scope of this account's permissions? Can the diagnosis be completed read-only? If write access becomes necessary, who approves it? Write these four lines at the top of the incident ticket and then begin.

Step 2 — Reproduce the Symptom

"It is sometimes slow" cannot be diagnosed. Only measurements can. So the second step is pinning the reported symptom down into a single command.

# Pin the symptom into one command (constructed example)
curl -sS -o /dev/null -w "%{http_code} %{time_total}s\n" \
  https://api.customer.example/v1/payments/health

# Repeat 20 times and measure the failure rate
for i in $(seq 1 20); do
  curl -s -o /dev/null -w "%{http_code}\n" https://api.customer.example/v1/payments/health
done | sort | uniq -c

If 3 out of 20 runs return 504, the incident has turned from hearsay into a number. If it does not reproduce, that too is information: the problem lives with specific users, specific hours, or specific paths, and the question just narrowed accordingly. The reproduction command gets reused at every later step as the yardstick that decides whether the fix worked.

Step 3 — Isolate the Layer

Before hunting the cause, find the neighborhood it lives in. Ask exactly one question of each of four layers — network, auth, application, data. Are DNS and TLS healthy? Are 401s or 403s mixed into the errors? Are the pods alive without restarts? Is latency in the data layer propagating up into the app?

# One question per layer (constructed example)
kubectl -n payments get pods -o wide            # app layer: status and restart counts
kubectl -n payments describe deploy/api | grep -A3 Limits   # resource limits
kubectl -n payments get endpoints api           # service-to-pod wiring
kubectl -n payments logs deploy/api --since=30m | grep -ciE "timeout|refused|pool"

In the constructed example, there are no pod restarts, no 401s, and the log grep shows timeout-family strings clustering. Network and auth drop off the suspect list, and the search narrows to somewhere between the app and the data layer. The purpose of layer isolation is not to guess the answer; it is to make certain which places you no longer need to look.

Step 4 — Form Cause Hypotheses

From here on you are fighting the temptation to rummage through logs at random. The logs of an unfamiliar environment are an ocean; enter without a hypothesis and only your time disappears. Within the remaining suspect zone, state two or three hypotheses explicitly. For the constructed example: first, DB connection pool exhaustion — it fits the pool strings in the log and the intermittency. Second, a specific query slowing down — data growth may have flipped an execution plan. Third, an afternoon deploy or config change — the timing overlap needs checking.

A good hypothesis has one requirement: it must be clear what evidence would kill it. A hypothesis with no imaginable refutation is not diagnosis, it is guessing. Write the hypothesis list straight into the ticket — later it becomes half of the report.

Step 5 — Verify the Hypotheses

For each hypothesis, go collect only the minimum evidence needed to reject or confirm it.

# Hypothesis 1: pool exhaustion — frequency and time distribution (constructed example)
kubectl -n payments logs deploy/api --since=2h \
  | grep -c "connection pool exhausted"

kubectl -n payments logs deploy/api --since=2h \
  | grep "connection pool exhausted" | cut -c1-16 | sort | uniq -c
# If it clusters only after 14:05 → ask the customer what changed at that time

The constructed example resolves like this: the errors appear only after 14:05; asked about it, the customer confirms a config deploy at that time; the deploy diff shows the pool size was reduced. After reverting, rerun the reproduction command from step 2 and confirm 0 failures out of 20. Verification points in two directions: evidence that confirms the cause, and the healthy number re-measured with the same yardstick after the fix. Without the latter, you cannot say it is fixed.

Step 6 — Write the Report

An FDE's incident response ends not in the system but in the report. Fix everything perfectly and still, if the report is late or vague, what remains in the customer's memory is anxiety. The first paragraph carries impact and current status; root-cause analysis comes after.

[Incident report — constructed example]
Impact  : payments API 5xx rate 0.1% baseline → peak 7% (14:10–15:40)
Status  : mitigation applied, error rate confirmed back in baseline range
Probable cause: 14:05 config deploy reduced the DB connection pool size
Next    : revert complete; will propose a pre-deploy config diff check

The no-names principle weighs especially heavily at a customer site. The moment you write "the customer's admin changed the setting incorrectly," that admin hides information during the next incident. Describe only how the system failed. A blameless report is not a courtesy; it is an investment that buys speed for the next diagnosis.

Practice by Doing

This playbook sticks a hundred times faster when lived than when read.

  • FDE Career RPG — the 31 missions, from "it is slow" to "we cannot connect" to "the monitoring died first," are all variations of these six steps. Practice keeping the order while time and trust drain under pressure.
  • FDE Curriculum Roadmap — check the per-domain skills that layer isolation depends on.

FDE Complete Guide series

Discuss on Twitter • View on GitHub