Split View: 새벽 3시에 깨우지 않는 알림 설계 — 증상 기반 알림과 번 레이트 실전

Alerting That Does Not Wake You at 3AM — Symptom-Based Alerts and Burn Rate in Practice

• Introduction — How Many of Last Night's Pages Led to an Action
• The Mechanism by Which Alert Fatigue Makes You Miss Real Incidents
• Page on Symptoms, Put Causes on Dashboards
• SLOs and Error Budgets — Setting Thresholds You Can Defend
• Multi-Window Burn Rate Alerts — Catching Sensitivity and False Positives Together
  • What the for Clause Actually Does
• Page, Ticket, Dashboard — Three Tiers and Mandatory Runbooks
• The Procedure for Reviewing and Deleting Alerts
• Closing — The Purpose of an Alert Is Action, Not Detection

Introduction — How Many of Last Night's Pages Led to an Action

If you cannot answer that question immediately, the alerting system is already broken. If the answer is "almost none", the breakage is severe.

The typical path by which on-call collapses looks like this. An incident happens, the retrospective concludes "we should have known about this in advance", and one alert gets added. Repeat for two years and you have 300 alert rules, 400 messages a day piling into a Slack channel, and an on-call engineer who wakes up three times a night, does nothing all three times, and goes back to sleep.

This post is not about reducing those 300. It is about re-deciding what should have been paged on in the first place.

The Mechanism by Which Alert Fatigue Makes You Miss Real Incidents

Alert fatigue is not a problem of laziness. It is a problem of probability.

If 4 out of 400 daily alerts genuinely need action, the precision is 1 percent. In that state, the optimal strategy a human learns is "ignore it for now and check later". It is more dangerous precisely because it is rational. No amount of training or resolve can reverse it.

Concretely, three things break.

First, reaction time grows. Before you even open the alert, a prior judgement of "probably that thing again" attaches itself. The first ten minutes of a real incident disappear here.

Second, the signal drowns in noise. If 4 out of 400 are real, those 4 are visually indistinguishable from the other 396. Severity labels do not change that. If everything is critical, nothing is critical.

Third, people wear out. Night-time pages degrade the next day's judgement. Waking on-call for something that requires no action is pre-emptively eating into the quality of the response to the next incident.

From this comes a practical target. Aim for an average of at most 2 pages per 12-hour shift, and an action conversion rate of at least 70 percent for pages. Cross either of those numbers and it is time to delete alerts, not add them.

Page on Symptoms, Put Causes on Dashboards

If you had to pick the single most powerful rule, it would be this. Page only on symptoms the user experiences.

CPU utilization at 90 percent is not a symptom. Users do not feel CPU. If CPU is at 90 percent and response times are normal, nothing has happened; if CPU is at 40 percent and responses take 5 seconds, that is a serious incident. Page on a cause metric and you are wrong in both cases.

What you page on is what the service promised its users.

• Availability: the fraction of requests that failed with a server error
• Latency: the fraction of requests not completed within the threshold
• Throughput collapse: traffic disappearing relative to normal (a common symptom of an outage)
• Freshness: how far behind data has fallen in a batch or stream pipeline
• Correctness: the number of discrepancies found by a reconciliation job

What you do not page on is the cause metrics. CPU, memory, disk IOPS, pod restart count, thread pool utilization, GC time, replica count. Keep these on investigation dashboards and use them to narrow down the cause once an incident is underway.

There are two categories of exception. First, things that are irreversible and need lead time — disk free space, certificate expiry, quota exhaustion. Items that are hard to recover from once reached but preventable if known in advance get a prediction-based page.

# Is the disk on a trajectory to fill within 4 hours — after it fills it is too late
predict_linear(node_filesystem_avail_bytes{mountpoint="/data"}[6h], 4 * 3600) < 0
and
node_filesystem_avail_bytes{mountpoint="/data"} / node_filesystem_size_bytes{mountpoint="/data"} < 0.2

Second, breakage of the observability system itself. A state where metrics stop arriving disables every alert, so it must be a paging target.

# A job whose scrapes have stopped — without this alert the rest die silently
up{job="checkout-api"} == 0
or
absent(up{job="checkout-api"})

SLOs and Error Budgets — Setting Thresholds You Can Defend

Where did the 5 percent in "alert if the error rate exceeds 5 percent" come from? For most teams, from nowhere at all. A number set as, say, 1.2 times the historical maximum cannot be explained, and a threshold you cannot explain creeps upward a little with every incident.

Derive it from the SLO and the threshold acquires a justification.

If the target is 99.9 percent over 30 days, the error budget is 0.1 percent. Thirty days is 43,200 minutes, so the budget is 43.2 minutes.

python3 - <<'PY'
slo = 0.999
days = 30
budget_ratio = 1 - slo
minutes = days * 24 * 60 * budget_ratio
print(f"error budget: {budget_ratio*100:.2f}% = {minutes:.1f} min / {days} days")
for br in (14.4, 6, 3, 1):
    # how long until the budget is gone if you keep burning at burn rate br
    hours = days * 24 / br
    print(f"  burn rate {br:>4}: exhausted in {hours:6.1f} hours")
PY
# error budget: 0.10% = 43.2 min / 30 days
#   burn rate 14.4: exhausted in   50.0 hours
#   burn rate    6: exhausted in  120.0 hours
#   burn rate    3: exhausted in  240.0 hours
#   burn rate    1: exhausted in  720.0 hours

The burn rate is the actual error rate divided by the budget ratio. With a 99.9 percent target, an error rate of 1.44 percent is a burn rate of 14.4. Sustain that speed for an hour and 2 percent of the 30-day budget is gone.

Precompute the SLI with recording rules. Compute a 3-day-window rate on every alert evaluation and Prometheus dies first.

# rules/slo.yml
groups:
  - name: checkout-slo-sli
    interval: 30s
    rules:
      - record: job:slo_errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{job="checkout-api", code=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total{job="checkout-api"}[5m]))

      - record: job:slo_errors:ratio_rate1h
        expr: |
          sum by (job) (rate(http_requests_total{job="checkout-api", code=~"5.."}[1h]))
          /
          sum by (job) (rate(http_requests_total{job="checkout-api"}[1h]))

      - record: job:slo_errors:ratio_rate30m
        expr: |
          sum by (job) (rate(http_requests_total{job="checkout-api", code=~"5.."}[30m]))
          /
          sum by (job) (rate(http_requests_total{job="checkout-api"}[30m]))

      - record: job:slo_errors:ratio_rate6h
        expr: |
          sum by (job) (rate(http_requests_total{job="checkout-api", code=~"5.."}[6h]))
          /
          sum by (job) (rate(http_requests_total{job="checkout-api"}[6h]))

      # Minimum traffic gate to stop the ratio swinging wildly in low-traffic periods
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total{job="checkout-api"}[5m]))

That last recording rule matters. In a period taking 3 requests every five minutes, one failure makes the error rate 33 percent and crosses every burn rate threshold at once. AND in a minimum traffic condition and the ghost alerts at dawn disappear. If traffic is exactly zero the ratio becomes 0 divided by 0, a NaN, and the alert quietly vanishes — that case has to be caught separately by a throughput collapse alert.

Multi-Window Burn Rate Alerts — Catching Sensitivity and False Positives Together

A single-window alert always gives up one side. A short window reacts to momentary fluctuation and produces false positives; a long window catches big incidents late.

The solution is to join two windows with AND. The long window decides "are we burning at this rate", and the short window decides "is it still happening right now". The key point is that the short window's role is not sensitivity but resolution speed. Without the short window, the long window keeps the alert up for its own window length even after the incident ends.

The convention is to set the short window at one twelfth of the long window. At that ratio the alert clears roughly within one short-window length after the incident ends.

# rules/slo.yml (continued)
  - name: checkout-slo-alerts
    rules:
      - alert: CheckoutErrorBudgetBurnFast
        expr: |
          job:slo_errors:ratio_rate1h{job="checkout-api"} > (14.4 * 0.001)
          and
          job:slo_errors:ratio_rate5m{job="checkout-api"} > (14.4 * 0.001)
          and
          job:http_requests:rate5m{job="checkout-api"} > 1
        for: 2m
        labels:
          severity: page
          slo: checkout-availability
        annotations:
          summary: 'checkout-api is burning error budget at 2% per hour'
          description: 'current 1h error rate {{ printf "%.2f" $value }} — check the dashboard for remaining budget'
          runbook_url: 'https://runbooks.internal/checkout/error-budget-burn'
          dashboard_url: 'https://grafana.internal/d/checkout-slo'

      - alert: CheckoutErrorBudgetBurnSlow
        expr: |
          job:slo_errors:ratio_rate6h{job="checkout-api"} > (6 * 0.001)
          and
          job:slo_errors:ratio_rate30m{job="checkout-api"} > (6 * 0.001)
        for: 15m
        labels:
          severity: ticket
          slo: checkout-availability
        annotations:
          summary: 'checkout-api error budget is burning continuously over a 6h window'
          runbook_url: 'https://runbooks.internal/checkout/error-budget-burn'

What the for Clause Actually Does

for means the expression must stay true continuously for that duration before it fires. If it becomes false even once in between, the timer resets to zero. That is the principle by which it prevents flapping.

There is a common piece of bad advice here. "We get too many false positives, let us raise for to 30 minutes." That works, but the price is high. Detection is delayed by 30 minutes, and if the incident ends in 25 minutes the alert never fires at all. And the real problem remains — the fact that the expression itself is looking at noise.

The correct order is this. First smooth the noise with a long-window rate. Then AND in a short window to confirm it is ongoing. Use for last, at only 2 to 5 minutes, to absorb one or two missing samples from something like a failed scrape. If you are already using a long window and still holding for above 15 minutes, the window design is wrong.

Always test the rules. Alert rules are production code, and production code does not ship untested.

# rules/slo_test.yml
rule_files:
  - slo.yml
evaluation_interval: 30s
tests:
  - interval: 30s
    input_series:
      # 95 successes and 5 failures per second = 5% error rate (above the 1.44% threshold)
      - series: 'http_requests_total{job="checkout-api", code="200"}'
        values: '0+2850x240'
      - series: 'http_requests_total{job="checkout-api", code="500"}'
        values: '0+150x240'
    alert_rule_test:
      - eval_time: 70m
        alertname: CheckoutErrorBudgetBurnFast
        exp_alerts:
          - exp_labels:
              severity: page
              slo: checkout-availability
              job: checkout-api

promtool check rules rules/slo.yml
# Checking rules/slo.yml
#   SUCCESS: 7 rules found

promtool test rules rules/slo_test.yml
# Unit Testing:  rules/slo_test.yml
#   SUCCESS

Page, Ticket, Dashboard — Three Tiers and Mandatory Runbooks

Every alert must be one of three things. There is no fourth.

• Page: a human has to wake up now. User impact is ongoing, it will not self-heal, and human action changes the situation. The target is at most 2 per 12-hour shift.
• Ticket: it can be handled during business hours. The budget is leaking but slowly. An issue is created automatically and an owner is assigned.
• Dashboard: do not create an alert. It is a metric you look at while investigating. The urge to attach an alert here is where 300 alerts begin.

Making that third category explicit matters. The compromise of "let us just send it to a Slack channel for now" is the worst option. Alerts grow without limit while belonging nowhere, nobody watches that channel any more, and later when a real signal flows through it, it gets missed.

Enforce runbook links on page alerts. Someone woken at 3AM is in no state to be creative. What they need is a document listing three things to check and two things to do.

# CI gate: break the build if any alert has severity=page but no runbook_url
missing=$(yq -r '
  .groups[].rules[]
  | select(.labels.severity == "page")
  | select(.annotations.runbook_url == null)
  | .alert
' rules/*.yml)

if [ -n "$missing" ]; then
  echo "page alerts with no runbook link:"
  echo "$missing"
  exit 1
fi
echo "every page alert has a runbook attached."

Four things must appear in a runbook. The user impact this alert signifies, the dashboards and queries to check right away, the known causes and the action for each, and the escalation target. A runbook that is only a link with nothing inside is worse than no link at all.

The Procedure for Reviewing and Deleting Alerts

Alerts multiply because there is a procedure for adding them and none for deleting them. You need a regular review that makes deletion the default action.

Once a month, start by pulling out the following data.

# Total firing time per alert (minutes). The evaluation interval is 30s, so multiply the sample count by 0.5.
sort_desc(
  sum by (alertname) (
    count_over_time(ALERTS{alertstate="firing", severity="page"}[30d])
  ) * 0.5
)

# What is firing now, and what is being quietly suppressed
amtool alert query --alertmanager.url=http://alertmanager:9093 --output=extended
amtool silence query --alertmanager.url=http://alertmanager:9093 --expired=false

# If a silence has been in place for months with no expiry, that is an alert that should be deleted

Fill in four things per alert. Firing count over 30 days, the fraction of those that led to an actual action, median time to resolution, and the fraction that fired at night. Then apply the following rules mechanically.

• A page that has led to no action for three consecutive months gets demoted to a ticket or deleted. To keep it, the reason for keeping it has to be written down.
• An alert that has been silenced for more than 30 days gets deleted. A silence is the most honest signal that an alert is wrong.
• A bundle of alerts that always fires together in the same incident gets merged into one or tied together with inhibition rules. A structure that sends 12 pages for one incident obstructs the response.
• When you add an alert in an incident retrospective, you must either delete one or write down why there is nothing to delete.

Inhibition rules are the cheapest tool for cutting alert count. When an entire cluster goes down, send one alert instead of firing off 40 individual service alerts.

# alertmanager.yml
inhibit_rules:
  - source_matchers: ['severity="page"', 'alertname="ClusterDown"']
    target_matchers: ['severity=~"page|ticket"']
    equal: ['cluster']

route:
  group_by: ['alertname', 'cluster', 'slo']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'ticket-queue'
  routes:
    - matchers: ['severity="page"']
      receiver: 'oncall-pager'
      repeat_interval: 1h

What you put in group_by determines the alert count. Group by instance and you get as many alerts as you have pods. Group by service or by SLO and the count becomes something a human can read.

Closing — The Purpose of an Alert Is Action, Not Detection

There is one sentence to remember. The success criterion for an alert is not "did it detect the problem" but "does a human have to do something right now".

An alert that fails that criterion is harmful even when it is accurate. If 400 accurate alerts destroy the reaction speed of your on-call, accuracy means nothing. So the order becomes this. Define SLOs from user symptoms, derive thresholds backwards from the budget, join long and short windows with AND, enforce runbooks on pages, and every month delete the alerts that led to no action.

If there is one thing you can do tomorrow, it is to pull the three page alerts with the longest firing time over the last 30 days and check whether each has ever led to an action. In most teams, those three account for half of all on-call fatigue.

Further reading.

• Google SRE Workbook: Alerting on SLOs
• Google SRE Book: Monitoring Distributed Systems
• Prometheus: Alerting rules documentation
• Prometheus: Unit testing for rules