필사 모드: Dashboards That Get Read and Alerts Worth Paging — Defining Questions, Variables, SLOs, and Alert Fatigue
English- Introduction — The Dashboard Nobody Opens
- Write Down the Questions a Dashboard Must Answer First
- Data Sources and Variables — Using One Dashboard Across Multiple Targets
- Panel Design — Layouts That Get Read and Ones That Don't
- Alert on Symptoms, Not Causes
- SLOs and Error Budgets — The Basis for Setting Thresholds
- Rules for Cutting Alert Fatigue
- Managing Dashboards and Alerts as Code
- Closing — Dashboards and Alerts Are About Designing Human Behavior
Introduction — The Dashboard Nobody Opens
I often see organizations with 60 dashboards. Of those, on-call actually opens two or three. The rest get opened once when they're built, and never again after that.
Alerts are similar. There are 200 rules, on-call has muted the notification channel, and the one that really matters is buried inside it.
Both problems have the same root cause. Dashboards and alerts got built starting from "what should we show." The starting point should be "who needs to make what decision, and when."
This post starts over from that point. Verified against the Grafana 12 line and Prometheus 3.13.0 LTS. Since Grafana's UI paths change often, everything here is explained through provisioning files and queries rather than screen operations.
Write Down the Questions a Dashboard Must Answer First
Write the question before building the panel. No question, no panel.
The questions a dashboard needs to answer differ by type, and trying to cram everything into one is where failure starts.
| Type | Audience | Question it must answer | Panel count |
|---|---|---|---|
| Service overview | On-call, first 5 minutes | Are users affected right now, and on which route | 6–8 |
| Service detail | That service's owner | Is the cause code, a dependency, or a resource | 15–25 |
| Dependencies | On-call | Is anything we call having trouble | 6–10 |
| Capacity | Weekly review | What hits its limit first next quarter | 10–15 |
| SLO | Team lead, monthly | How much error budget is left | 4–6 |
Five questions are enough for a service-overview dashboard.
- Are requests failing, and what percentage
- Has it gotten slower, and on which route
- Since when, and does it overlap with a deploy
- Has traffic itself changed
- Has anything we depend on gotten worse
Keep only the panels that answer these five, and the dashboard won't exceed 8 panels. Order matters too. Reading top to bottom should flow "is there impact → where → why." If a CPU graph sits at the top, that dashboard already has its order wrong.
Data Sources and Variables — Using One Dashboard Across Multiple Targets
Once you start copying the same dashboard per environment, management collapses. A fix made in staging doesn't reach production, and three months later you have six different copies.
The fix is to turn the data source itself into a variable.
# provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
- name: Prometheus-prod
uid: prom-prod
type: prometheus
access: proxy
url: http://prometheus.observability.svc:9090
jsonData:
httpMethod: POST
timeInterval: 15s # Scrape interval. This becomes the basis for computing the rate interval
prometheusType: Prometheus
prometheusVersion: 3.13.0
exemplarTraceIdDestinations:
- name: trace_id
datasourceUid: tempo-prod
isDefault: true
- name: Prometheus-staging
uid: prom-staging
type: prometheus
access: proxy
url: http://prometheus.staging.svc:9090
jsonData:
httpMethod: POST
timeInterval: 15s
Matching timeInterval to the scrape interval matters. Grafana's rate-interval variable computes a safe window based on this value plus the panel width. If it's left blank, it computes from a default, so when you narrow the panel or pick a short time range, the rate result comes back empty.
Variables are structured in a cascade. A choice in an earlier variable narrows the candidates for the next one.
{
"templating": {
"list": [
{
"name": "datasource",
"type": "datasource",
"query": "prometheus",
"current": { "text": "Prometheus-prod", "value": "prom-prod" }
},
{
"name": "namespace",
"type": "query",
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"query": "label_values(kube_namespace_status_phase, namespace)",
"refresh": 1,
"sort": 1
},
{
"name": "service",
"type": "query",
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"query": "label_values(http_requests_total{namespace=\"$namespace\"}, service)",
"refresh": 2,
"includeAll": true,
"multi": true
},
{
"name": "route",
"type": "query",
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"query": "label_values(http_requests_total{namespace=\"$namespace\", service=~\"$service\"}, route)",
"refresh": 2,
"includeAll": true,
"multi": true
}
]
}
}
The meaning of the refresh value gets confused often. 1 refreshes when the dashboard opens; 2 refreshes when the time range changes. In an environment where rolling deploys keep changing pod names, you need 2. Leave it at 1, and a tab left open from yesterday keeps querying pods that are already gone.
When putting a multi-select variable into a query, use regex matching.
# A multi-select variable comes in as regex. Using equality only works when there's a single value
sum by (route) (
rate(http_requests_total{namespace="$namespace", service=~"$service", route=~"$route"}[$__rate_interval])
)
# Use the rate-interval variable in dashboard panels.
# A fixed [5m] wastes resolution when you're looking at a wide time range,
# and leaves gaps in the graph from too few samples when you're looking at a narrow one
histogram_quantile(0.99,
sum by (le, route) (
rate(http_request_duration_seconds_bucket{namespace="$namespace", service=~"$service"}[$__rate_interval])
)
)
If you allow "select all," also check includeAll's custom value. If the default doesn't work as a regex, results come back empty when "all" is selected.
Panel Design — Layouts That Get Read and Ones That Don't
Even with the same data, some layouts get read and some don't.
Group questions by row. The first row is "is there impact." Error rate, p99, and traffic — three panels is enough. The second row is "where." Breakdown by route and top error types. The third row onward is candidate causes.
Fix the axes. Leave the error-rate panel's max on auto, and the graph wobbles and looks alarming even at 0.01% error. Conversely, during a real incident it can't be compared against before. A percentage axis should start at 0 and be drawn together with a threshold line.
Specify units. Leave a second-denominated latency without a unit, and the reader can't tell whether 1.4 means 1.4 seconds or 1.4 milliseconds. You should never force that judgment call at 3am.
Never put more than 20 time series in one panel. Draw all of them for a service with 120 routes, and you see nothing. Draw only the top N and merge the rest.
# Only the top 10 routes. Narrow with a variable if you want to see the rest
topk(10,
sum by (route) (
rate(http_requests_total{service=~"$service", status_class="5xx"}[$__rate_interval])
)
)
# Overlay deploy timestamps as annotations, and the "since when" question gets answered instantly
changes(kube_deployment_status_observed_generation{namespace="$namespace"}[$__rate_interval]) > 0
Deploy annotations are the highest-ROI item here. If there's a vertical line at the point where the graph bends, and it's a deploy, the investigation ends right there.
Alert on Symptoms, Not Causes
If you had to pick one core principle of alert design, it's this: alert on what users feel.
Let's look at the problem with cause-based alerts through an example.
# Cause-based — once around 40 rules like this pile up, alert fatigue starts
- alert: HighCPU
expr: rate(process_cpu_seconds_total[5m]) > 0.8
for: 5m
- alert: HighMemory
expr: process_resident_memory_bytes / container_spec_memory_limit_bytes > 0.9
for: 5m
- alert: ManyGoroutines
expr: go_goroutines > 10000
for: 5m
- alert: DBConnectionsHigh
expr: pg_stat_activity_count > 80
for: 5m
These four have three problems. First, they fire even with zero impact on users — 85% CPU might just mean resources are being used well. Second, during a real incident all four fire at once, which actually obscures what the cause is. Third, they miss the situation where users are experiencing failures while all four metrics look normal.
# Symptom-based — alert on what the user experiences
groups:
- name: symptom_alerts
rules:
- alert: HighErrorRate
expr: |
sum by (service) (rate(http_requests_total{status_class="5xx"}[5m]))
/
sum by (service) (rate(http_requests_total[5m]))
> 0.02
for: 5m
labels:
severity: page
annotations:
summary: '5xx rate has exceeded 2%'
runbook_url: https://wiki.internal/runbook/high-error-rate
- alert: HighLatency
expr: |
histogram_quantile(0.99,
sum by (le, service) (rate(http_request_duration_seconds_bucket[5m]))
) > 1.5
for: 10m
labels:
severity: page
annotations:
summary: 'p99 latency has exceeded 1.5 seconds'
- alert: RequestsStopped
expr: |
sum by (service) (rate(http_requests_total[5m])) == 0
and
sum by (service) (rate(http_requests_total[5m] offset 1h)) > 1
for: 5m
labels:
severity: page
annotations:
summary: 'A service that normally has traffic is receiving no requests'
The third rule gets left out often. Since an error rate can't be computed with a zero denominator, if a service dies completely and receives no requests at all, the error-rate alert never fires. Missing traffic has to be watched separately.
Cause metrics belong on the dashboard, not in an alert. They're the material on-call uses to narrow down the cause after being woken by a symptom alert.
Catching Alerts That Never Fire
The most dangerous alert isn't a noisy one — it's one that's silently broken.
# A problematic rule — a 5-minute average almost never crosses the threshold
avg_over_time(http_request_duration_seconds_sum[5m])
/ avg_over_time(http_request_duration_seconds_count[5m]) > 5
# Why it doesn't fire: an average hides the tail. p99 could be 8 seconds while the average is 0.2 seconds.
# This rule only fires when the entire service takes 8 seconds across the board.
Nobody notices a rule like this even months after it's deployed. Periodically pull the rules with zero firing history.
# Find rules that haven't fired even once in the last 30 days
# ALERTS_FOR_STATE only exists for active alerts, so you have to cross-check against the rule list
count by (alertname) (max_over_time(ALERTS[30d]))
# Also check whether rule evaluation itself is failing
increase(prometheus_rule_evaluation_failures_total[1h]) > 0
Putting rule-level unit tests in CI catches this before deployment.
# tests/alerts_test.yml
rule_files:
- ../rules/symptom_alerts.yml
evaluation_interval: 30s
tests:
- interval: 15s
input_series:
- series: 'http_requests_total{service="checkout", status_class="2xx"}'
values: '0+90x60'
- series: 'http_requests_total{service="checkout", status_class="5xx"}'
values: '0+10x60'
alert_rule_test:
- eval_time: 10m
alertname: HighErrorRate
exp_alerts:
- exp_labels:
service: checkout
severity: page
exp_annotations:
summary: '5xx rate has exceeded 2%'
runbook_url: https://wiki.internal/runbook/high-error-rate
SLOs and Error Budgets — The Basis for Setting Thresholds
Where did the number "2% error rate" come from? Usually, from nowhere at all. Someone picked it by gut feel, and nobody has questioned it since.
An SLO gives that threshold a basis. There are three steps.
- Define the SLI. Express it as "the ratio of good events" — the ratio of non-5xx responses, the ratio of requests that finish within 300ms.
- Set the SLO. What percentage or higher must the SLI be over 30 days.
- Compute the error budget. If the SLO is 99.9%, the budget is 0.1%. Over a 30-day window, that's 43.2 minutes.
# rules/slo.yml
groups:
- name: slo_sli
interval: 30s
rules:
# Ratio of good requests — health checks excluded
- record: service:sli_availability:ratio_rate5m
expr: |
sum by (service) (
rate(http_requests_total{status_class!="5xx", route!~"/healthz|/readyz"}[5m])
)
/
sum by (service) (
rate(http_requests_total{route!~"/healthz|/readyz"}[5m])
)
# Error ratio for each window — referenced by the burn-rate alerts
- record: service:http_error_ratio:rate5m
expr: |
1 - service:sli_availability:ratio_rate5m
- record: service:http_error_ratio:rate1h
expr: |
sum by (service) (rate(http_requests_total{status_class="5xx", route!~"/healthz|/readyz"}[1h]))
/
sum by (service) (rate(http_requests_total{route!~"/healthz|/readyz"}[1h]))
- record: service:http_error_ratio:rate6h
expr: |
sum by (service) (rate(http_requests_total{status_class="5xx", route!~"/healthz|/readyz"}[6h]))
/
sum by (service) (rate(http_requests_total{route!~"/healthz|/readyz"}[6h]))
- record: service:http_error_ratio:rate30m
expr: |
sum by (service) (rate(http_requests_total{status_class="5xx", route!~"/healthz|/readyz"}[30m]))
/
sum by (service) (rate(http_requests_total{route!~"/healthz|/readyz"}[30m]))
# Remaining error-budget ratio — the core panel on the dashboard
- record: service:error_budget_remaining:ratio30d
expr: |
1 - (
(
sum by (service) (rate(http_requests_total{status_class="5xx", route!~"/healthz|/readyz"}[30d]))
/
sum by (service) (rate(http_requests_total{route!~"/healthz|/readyz"}[30d]))
)
/ 0.001
)
Burn rate is a multiple of "how quickly the budget would be exhausted if it kept going at this speed." A burn rate of 1 is the speed that spends it all in exactly 30 days; 14.4 is the speed that spends it all in about 2 days.
| Burn rate | Time to exhaust a 30-day budget | Long window | Short window | Response |
|---|---|---|---|---|
| 14.4 | ~2 days | 1 hour | 5 minutes | Page immediately |
| 6 | 5 days | 6 hours | 30 minutes | Respond within business hours |
| 3 | 10 days | 1 day | 2 hours | Create a ticket |
| 1 | 30 days | 3 days | 6 hours | Weekly review |
The reason for using a long window and a short window together is to block two different kinds of false positives. A long window alone keeps firing for hours over an incident that's already over. A short window alone fires on a momentary spike.
groups:
- name: slo_burn_rate
rules:
- alert: ErrorBudgetBurnFast
expr: |
service:http_error_ratio:rate1h > (14.4 * 0.001)
and
service:http_error_ratio:rate5m > (14.4 * 0.001)
for: 2m
labels:
severity: page
slo: availability
annotations:
summary: 'Burning the error budget at 14.4x. It will be fully exhausted in about 2 days'
runbook_url: https://wiki.internal/runbook/slo-burn
- alert: ErrorBudgetBurnSlow
expr: |
service:http_error_ratio:rate6h > (6 * 0.001)
and
service:http_error_ratio:rate30m > (6 * 0.001)
for: 15m
labels:
severity: ticket
slo: availability
annotations:
summary: 'Burning the error budget at 6x'
Four panels are enough for an SLO dashboard: remaining budget ratio, 30-day SLI trend, current burn rate, and the top list of routes that spent the most budget. These four answer the question "is it okay to take on risk right now."
Rules for Cutting Alert Fatigue
The way alerts stop working is by getting noisy. The rules below have a big effect.
Attach a runbook link to every alert. Don't create an alert without one. This one rule naturally shrinks the number of alerts, because the process of writing the runbook is where you realize "a person doesn't actually need to look at this."
Distinguish pages from tickets. Only things that need a human to get up right now are pages. Everything else is a ticket or a channel notification. If an alert that fired overnight could have waited until morning, downgrade its severity.
Use inhibit rules. While a higher-level incident is in progress, group the downstream alerts it causes.
# alertmanager.yml
route:
group_by: ['alertname', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: slack-default
routes:
- matchers: [severity = page]
receiver: pagerduty
group_wait: 10s
repeat_interval: 1h
continue: true
- matchers: [severity = ticket]
receiver: jira
repeat_interval: 24h
inhibit_rules:
# Suppress individual service alerts while a cluster-wide outage is in progress
- source_matchers: [alertname = ClusterUnreachable]
target_matchers: [severity =~ 'page|ticket']
equal: [cluster]
# If a page is already firing for a service, suppress its tickets
- source_matchers: [severity = page]
target_matchers: [severity = ticket]
equal: [service]
receivers:
- name: pagerduty
pagerduty_configs:
- routing_key_file: /etc/alertmanager/pd_key
- name: jira
webhook_configs:
- url: http://alert-to-jira.internal/hook
- name: slack-default
slack_configs:
- channel: '#alerts'
send_resolved: true
Review alerts regularly. Once a month, classify the alerts that fired last month along two axes.
| Did it fire | Did it trigger action | Action |
|---|---|---|
| Yes | Yes | Keep |
| Yes | No | Adjust threshold or downgrade severity; delete if it recurs |
| No | N/A | Test whether the rule works; fix or delete if not |
| No firing history at all | N/A | Verify it can fire with a unit test |
The second row matters most. An alert that fired but nobody did anything about repeats the same thing next month. An alert that doesn't trigger action is information, not an alert.
Managing Dashboards and Alerts as Code
A dashboard built in the UI can't be reviewed or rolled back. Grafana 12 has provisioning and observability-as-code well sorted out, so managing it as files is clearly better.
# provisioning/dashboards/dashboards.yaml
apiVersion: 1
providers:
- name: gitops
orgId: 1
folder: Services
type: file
disableDeletion: true
updateIntervalSeconds: 60
allowUiUpdates: false # No UI edits. Changes must go through the repository
options:
path: /etc/grafana/dashboards
foldersFromFilesStructure: true
# provisioning/alerting/rules.yaml — define Grafana-managed alerts as files
apiVersion: 1
groups:
- orgId: 1
name: checkout_slo
folder: Alerts
interval: 1m
rules:
- uid: checkout-burn-fast
title: Checkout error budget burn (fast)
condition: THRESHOLD
data:
- refId: BURN
relativeTimeRange:
from: 3600
to: 0
datasourceUid: prom-prod
model:
expr: service:http_error_ratio:rate1h{service="checkout"}
instant: true
refId: BURN
- refId: THRESHOLD
datasourceUid: __expr__
model:
type: threshold
expression: BURN
conditions:
- evaluator:
type: gt
params: [0.0144]
refId: THRESHOLD
for: 2m
noDataState: NoData
execErrState: Alerting
labels:
severity: page
slo: availability
annotations:
summary: Checkout's error budget is burning at 14.4x
runbook_url: https://wiki.internal/runbook/slo-burn
What you set noDataState to is where practice diverges. Set it to Alerting, and a brief scrape gap fires it. Set it to OK, and it stays silent even if data collection stops entirely. It's usually better to leave it at NoData and create a separate alert that watches for collection itself stopping.
- alert: ScrapeTargetDown
expr: up{job=~"checkout.*|payment.*"} == 0
for: 5m
labels:
severity: page
annotations:
summary: 'A scrape target has not responded for 5 minutes'
There are two things to watch when putting dashboard JSON into the repository. First, saving from the UI produces a diff full of noise like panel coordinates and version numbers. Put a normalization script in CI that strips unnecessary fields before saving, and reviews become possible. Second, hardcoding a data source by UID breaks in other environments. Use a data-source variable and have panels reference that variable.
Closing — Dashboards and Alerts Are About Designing Human Behavior
Panels and rules are the output; what you're really designing is what a person woken at 3am sees and does. Looked at that way, the calls get easy. Delete a panel that doesn't determine the next action when opened. Downgrade or remove an alert that has nothing to do when it fires.
Here are two checks you can run right now. First, pull the list of dashboards that haven't been opened once in the last 30 days. Nobody opens them, but the maintenance cost keeps accruing anyway. Second, count the alerts that fired over the last 30 days, and compute the fraction that actually led to action. If that fraction is below half, the alerting system has already lost credibility.
Further reading.
현재 단락 (1/382)
I often see organizations with 60 dashboards. Of those, on-call actually opens two or three. The res...