Dashboards That Get Read and Alerts Worth Paging — Defining Questions, Variables, SLOs, and Alert Fatigue

Introduction — The Dashboard Nobody Opens
Write Down the Questions a Dashboard Must Answer First
Data Sources and Variables — Using One Dashboard Across Multiple Targets
Panel Design — Layouts That Get Read and Ones That Don't
Alert on Symptoms, Not Causes
- Catching Alerts That Never Fire
SLOs and Error Budgets — The Basis for Setting Thresholds
Rules for Cutting Alert Fatigue
Managing Dashboards and Alerts as Code
Closing — Dashboards and Alerts Are About Designing Human Behavior

Introduction — The Dashboard Nobody Opens

I often see organizations with 60 dashboards. Of those, on-call actually opens two or three. The rest get opened once when they're built, and never again after that.

Alerts are similar. There are 200 rules, on-call has muted the notification channel, and the one that really matters is buried inside it.

Both problems have the same root cause. Dashboards and alerts got built starting from "what should we show." The starting point should be "who needs to make what decision, and when."

This post starts over from that point. Verified against the Grafana 12 line and Prometheus 3.13.0 LTS. Since Grafana's UI paths change often, everything here is explained through provisioning files and queries rather than screen operations.

Write Down the Questions a Dashboard Must Answer First

Write the question before building the panel. No question, no panel.

The questions a dashboard needs to answer differ by type, and trying to cram everything into one is where failure starts.

Type	Audience	Question it must answer	Panel count
Service overview	On-call, first 5 minutes	Are users affected right now, and on which route	6–8
Service detail	That service's owner	Is the cause code, a dependency, or a resource	15–25
Dependencies	On-call	Is anything we call having trouble	6–10
Capacity	Weekly review	What hits its limit first next quarter	10–15
SLO	Team lead, monthly	How much error budget is left	4–6

Five questions are enough for a service-overview dashboard.

Are requests failing, and what percentage
Has it gotten slower, and on which route
Since when, and does it overlap with a deploy
Has traffic itself changed
Has anything we depend on gotten worse

Keep only the panels that answer these five, and the dashboard won't exceed 8 panels. Order matters too. Reading top to bottom should flow "is there impact → where → why." If a CPU graph sits at the top, that dashboard already has its order wrong.

Data Sources and Variables — Using One Dashboard Across Multiple Targets

Once you start copying the same dashboard per environment, management collapses. A fix made in staging doesn't reach production, and three months later you have six different copies.

The fix is to turn the data source itself into a variable.

# provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
  - name: Prometheus-prod
    uid: prom-prod
    type: prometheus
    access: proxy
    url: http://prometheus.observability.svc:9090
    jsonData:
      httpMethod: POST
      timeInterval: 15s          # Scrape interval. This becomes the basis for computing the rate interval
      prometheusType: Prometheus
      prometheusVersion: 3.13.0
      exemplarTraceIdDestinations:
        - name: trace_id
          datasourceUid: tempo-prod
    isDefault: true

  - name: Prometheus-staging
    uid: prom-staging
    type: prometheus
    access: proxy
    url: http://prometheus.staging.svc:9090
    jsonData:
      httpMethod: POST
      timeInterval: 15s

Matching timeInterval to the scrape interval matters. Grafana's rate-interval variable computes a safe window based on this value plus the panel width. If it's left blank, it computes from a default, so when you narrow the panel or pick a short time range, the rate result comes back empty.

Variables are structured in a cascade. A choice in an earlier variable narrows the candidates for the next one.

{
  "templating": {
    "list": [
      {
        "name": "datasource",
        "type": "datasource",
        "query": "prometheus",
        "current": { "text": "Prometheus-prod", "value": "prom-prod" }
      },
      {
        "name": "namespace",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "${datasource}" },
        "query": "label_values(kube_namespace_status_phase, namespace)",
        "refresh": 1,
        "sort": 1
      },
      {
        "name": "service",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "${datasource}" },
        "query": "label_values(http_requests_total{namespace=\"$namespace\"}, service)",
        "refresh": 2,
        "includeAll": true,
        "multi": true
      },
      {
        "name": "route",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "${datasource}" },
        "query": "label_values(http_requests_total{namespace=\"$namespace\", service=~\"$service\"}, route)",
        "refresh": 2,
        "includeAll": true,
        "multi": true
      }
    ]
  }
}

The meaning of the refresh value gets confused often. 1 refreshes when the dashboard opens; 2 refreshes when the time range changes. In an environment where rolling deploys keep changing pod names, you need 2. Leave it at 1, and a tab left open from yesterday keeps querying pods that are already gone.

When putting a multi-select variable into a query, use regex matching.

# A multi-select variable comes in as regex. Using equality only works when there's a single value
sum by (route) (
  rate(http_requests_total{namespace="$namespace", service=~"$service", route=~"$route"}[$__rate_interval])
)

# Use the rate-interval variable in dashboard panels.
# A fixed [5m] wastes resolution when you're looking at a wide time range,
# and leaves gaps in the graph from too few samples when you're looking at a narrow one
histogram_quantile(0.99,
  sum by (le, route) (
    rate(http_request_duration_seconds_bucket{namespace="$namespace", service=~"$service"}[$__rate_interval])
  )
)

If you allow "select all," also check includeAll's custom value. If the default doesn't work as a regex, results come back empty when "all" is selected.

Panel Design — Layouts That Get Read and Ones That Don't

Even with the same data, some layouts get read and some don't.

Group questions by row. The first row is "is there impact." Error rate, p99, and traffic — three panels is enough. The second row is "where." Breakdown by route and top error types. The third row onward is candidate causes.

Fix the axes. Leave the error-rate panel's max on auto, and the graph wobbles and looks alarming even at 0.01% error. Conversely, during a real incident it can't be compared against before. A percentage axis should start at 0 and be drawn together with a threshold line.

Specify units. Leave a second-denominated latency without a unit, and the reader can't tell whether 1.4 means 1.4 seconds or 1.4 milliseconds. You should never force that judgment call at 3am.

Never put more than 20 time series in one panel. Draw all of them for a service with 120 routes, and you see nothing. Draw only the top N and merge the rest.

# Only the top 10 routes. Narrow with a variable if you want to see the rest
topk(10,
  sum by (route) (
    rate(http_requests_total{service=~"$service", status_class="5xx"}[$__rate_interval])
  )
)

# Overlay deploy timestamps as annotations, and the "since when" question gets answered instantly
changes(kube_deployment_status_observed_generation{namespace="$namespace"}[$__rate_interval]) > 0

Deploy annotations are the highest-ROI item here. If there's a vertical line at the point where the graph bends, and it's a deploy, the investigation ends right there.

Alert on Symptoms, Not Causes

If you had to pick one core principle of alert design, it's this: alert on what users feel.

Let's look at the problem with cause-based alerts through an example.

# Cause-based — once around 40 rules like this pile up, alert fatigue starts
- alert: HighCPU
  expr: rate(process_cpu_seconds_total[5m]) > 0.8
  for: 5m
- alert: HighMemory
  expr: process_resident_memory_bytes / container_spec_memory_limit_bytes > 0.9
  for: 5m
- alert: ManyGoroutines
  expr: go_goroutines > 10000
  for: 5m
- alert: DBConnectionsHigh
  expr: pg_stat_activity_count > 80
  for: 5m

These four have three problems. First, they fire even with zero impact on users — 85% CPU might just mean resources are being used well. Second, during a real incident all four fire at once, which actually obscures what the cause is. Third, they miss the situation where users are experiencing failures while all four metrics look normal.

# Symptom-based — alert on what the user experiences
groups:
  - name: symptom_alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum by (service) (rate(http_requests_total{status_class="5xx"}[5m]))
            /
          sum by (service) (rate(http_requests_total[5m]))
            > 0.02
        for: 5m
        labels:
          severity: page
        annotations:
          summary: '5xx rate has exceeded 2%'
          runbook_url: https://wiki.internal/runbook/high-error-rate

      - alert: HighLatency
        expr: |
          histogram_quantile(0.99,
            sum by (le, service) (rate(http_request_duration_seconds_bucket[5m]))
          ) > 1.5
        for: 10m
        labels:
          severity: page
        annotations:
          summary: 'p99 latency has exceeded 1.5 seconds'

      - alert: RequestsStopped
        expr: |
          sum by (service) (rate(http_requests_total[5m])) == 0
            and
          sum by (service) (rate(http_requests_total[5m] offset 1h)) > 1
        for: 5m
        labels:
          severity: page
        annotations:
          summary: 'A service that normally has traffic is receiving no requests'

The third rule gets left out often. Since an error rate can't be computed with a zero denominator, if a service dies completely and receives no requests at all, the error-rate alert never fires. Missing traffic has to be watched separately.

Cause metrics belong on the dashboard, not in an alert. They're the material on-call uses to narrow down the cause after being woken by a symptom alert.

Catching Alerts That Never Fire

The most dangerous alert isn't a noisy one — it's one that's silently broken.

# A problematic rule — a 5-minute average almost never crosses the threshold
avg_over_time(http_request_duration_seconds_sum[5m])
  / avg_over_time(http_request_duration_seconds_count[5m]) > 5

# Why it doesn't fire: an average hides the tail. p99 could be 8 seconds while the average is 0.2 seconds.
# This rule only fires when the entire service takes 8 seconds across the board.

Nobody notices a rule like this even months after it's deployed. Periodically pull the rules with zero firing history.

# Find rules that haven't fired even once in the last 30 days
# ALERTS_FOR_STATE only exists for active alerts, so you have to cross-check against the rule list
count by (alertname) (max_over_time(ALERTS[30d]))

# Also check whether rule evaluation itself is failing
increase(prometheus_rule_evaluation_failures_total[1h]) > 0

Putting rule-level unit tests in CI catches this before deployment.

# tests/alerts_test.yml
rule_files:
  - ../rules/symptom_alerts.yml

evaluation_interval: 30s

tests:
  - interval: 15s
    input_series:
      - series: 'http_requests_total{service="checkout", status_class="2xx"}'
        values: '0+90x60'
      - series: 'http_requests_total{service="checkout", status_class="5xx"}'
        values: '0+10x60'
    alert_rule_test:
      - eval_time: 10m
        alertname: HighErrorRate
        exp_alerts:
          - exp_labels:
              service: checkout
              severity: page
            exp_annotations:
              summary: '5xx rate has exceeded 2%'
              runbook_url: https://wiki.internal/runbook/high-error-rate

SLOs and Error Budgets — The Basis for Setting Thresholds

Where did the number "2% error rate" come from? Usually, from nowhere at all. Someone picked it by gut feel, and nobody has questioned it since.

An SLO gives that threshold a basis. There are three steps.

Define the SLI. Express it as "the ratio of good events" — the ratio of non-5xx responses, the ratio of requests that finish within 300ms.
Set the SLO. What percentage or higher must the SLI be over 30 days.
Compute the error budget. If the SLO is 99.9%, the budget is 0.1%. Over a 30-day window, that's 43.2 minutes.

# rules/slo.yml
groups:
  - name: slo_sli
    interval: 30s
    rules:
      # Ratio of good requests — health checks excluded
      - record: service:sli_availability:ratio_rate5m
        expr: |
          sum by (service) (
            rate(http_requests_total{status_class!="5xx", route!~"/healthz|/readyz"}[5m])
          )
          /
          sum by (service) (
            rate(http_requests_total{route!~"/healthz|/readyz"}[5m])
          )

      # Error ratio for each window — referenced by the burn-rate alerts
      - record: service:http_error_ratio:rate5m
        expr: |
          1 - service:sli_availability:ratio_rate5m

      - record: service:http_error_ratio:rate1h
        expr: |
          sum by (service) (rate(http_requests_total{status_class="5xx", route!~"/healthz|/readyz"}[1h]))
          /
          sum by (service) (rate(http_requests_total{route!~"/healthz|/readyz"}[1h]))

      - record: service:http_error_ratio:rate6h
        expr: |
          sum by (service) (rate(http_requests_total{status_class="5xx", route!~"/healthz|/readyz"}[6h]))
          /
          sum by (service) (rate(http_requests_total{route!~"/healthz|/readyz"}[6h]))

      - record: service:http_error_ratio:rate30m
        expr: |
          sum by (service) (rate(http_requests_total{status_class="5xx", route!~"/healthz|/readyz"}[30m]))
          /
          sum by (service) (rate(http_requests_total{route!~"/healthz|/readyz"}[30m]))

      # Remaining error-budget ratio — the core panel on the dashboard
      - record: service:error_budget_remaining:ratio30d
        expr: |
          1 - (
            (
              sum by (service) (rate(http_requests_total{status_class="5xx", route!~"/healthz|/readyz"}[30d]))
              /
              sum by (service) (rate(http_requests_total{route!~"/healthz|/readyz"}[30d]))
            )
            / 0.001
          )

Burn rate is a multiple of "how quickly the budget would be exhausted if it kept going at this speed." A burn rate of 1 is the speed that spends it all in exactly 30 days; 14.4 is the speed that spends it all in about 2 days.

Burn rate	Time to exhaust a 30-day budget	Long window	Short window	Response
14.4	~2 days	1 hour	5 minutes	Page immediately
6	5 days	6 hours	30 minutes	Respond within business hours
3	10 days	1 day	2 hours	Create a ticket
1	30 days	3 days	6 hours	Weekly review

The reason for using a long window and a short window together is to block two different kinds of false positives. A long window alone keeps firing for hours over an incident that's already over. A short window alone fires on a momentary spike.

groups:
  - name: slo_burn_rate
    rules:
      - alert: ErrorBudgetBurnFast
        expr: |
          service:http_error_ratio:rate1h > (14.4 * 0.001)
            and
          service:http_error_ratio:rate5m > (14.4 * 0.001)
        for: 2m
        labels:
          severity: page
          slo: availability
        annotations:
          summary: 'Burning the error budget at 14.4x. It will be fully exhausted in about 2 days'
          runbook_url: https://wiki.internal/runbook/slo-burn

      - alert: ErrorBudgetBurnSlow
        expr: |
          service:http_error_ratio:rate6h > (6 * 0.001)
            and
          service:http_error_ratio:rate30m > (6 * 0.001)
        for: 15m
        labels:
          severity: ticket
          slo: availability
        annotations:
          summary: 'Burning the error budget at 6x'

Four panels are enough for an SLO dashboard: remaining budget ratio, 30-day SLI trend, current burn rate, and the top list of routes that spent the most budget. These four answer the question "is it okay to take on risk right now."

Rules for Cutting Alert Fatigue

The way alerts stop working is by getting noisy. The rules below have a big effect.

Attach a runbook link to every alert. Don't create an alert without one. This one rule naturally shrinks the number of alerts, because the process of writing the runbook is where you realize "a person doesn't actually need to look at this."

Distinguish pages from tickets. Only things that need a human to get up right now are pages. Everything else is a ticket or a channel notification. If an alert that fired overnight could have waited until morning, downgrade its severity.

Use inhibit rules. While a higher-level incident is in progress, group the downstream alerts it causes.

# alertmanager.yml
route:
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: slack-default
  routes:
    - matchers: [severity = page]
      receiver: pagerduty
      group_wait: 10s
      repeat_interval: 1h
      continue: true
    - matchers: [severity = ticket]
      receiver: jira
      repeat_interval: 24h

inhibit_rules:
  # Suppress individual service alerts while a cluster-wide outage is in progress
  - source_matchers: [alertname = ClusterUnreachable]
    target_matchers: [severity =~ 'page|ticket']
    equal: [cluster]
  # If a page is already firing for a service, suppress its tickets
  - source_matchers: [severity = page]
    target_matchers: [severity = ticket]
    equal: [service]

receivers:
  - name: pagerduty
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/pd_key
  - name: jira
    webhook_configs:
      - url: http://alert-to-jira.internal/hook
  - name: slack-default
    slack_configs:
      - channel: '#alerts'
        send_resolved: true

Review alerts regularly. Once a month, classify the alerts that fired last month along two axes.

Did it fire	Did it trigger action	Action
Yes	Yes	Keep
Yes	No	Adjust threshold or downgrade severity; delete if it recurs
No	N/A	Test whether the rule works; fix or delete if not
No firing history at all	N/A	Verify it can fire with a unit test

The second row matters most. An alert that fired but nobody did anything about repeats the same thing next month. An alert that doesn't trigger action is information, not an alert.

Managing Dashboards and Alerts as Code

A dashboard built in the UI can't be reviewed or rolled back. Grafana 12 has provisioning and observability-as-code well sorted out, so managing it as files is clearly better.

# provisioning/dashboards/dashboards.yaml
apiVersion: 1
providers:
  - name: gitops
    orgId: 1
    folder: Services
    type: file
    disableDeletion: true
    updateIntervalSeconds: 60
    allowUiUpdates: false        # No UI edits. Changes must go through the repository
    options:
      path: /etc/grafana/dashboards
      foldersFromFilesStructure: true

# provisioning/alerting/rules.yaml — define Grafana-managed alerts as files
apiVersion: 1
groups:
  - orgId: 1
    name: checkout_slo
    folder: Alerts
    interval: 1m
    rules:
      - uid: checkout-burn-fast
        title: Checkout error budget burn (fast)
        condition: THRESHOLD
        data:
          - refId: BURN
            relativeTimeRange:
              from: 3600
              to: 0
            datasourceUid: prom-prod
            model:
              expr: service:http_error_ratio:rate1h{service="checkout"}
              instant: true
              refId: BURN
          - refId: THRESHOLD
            datasourceUid: __expr__
            model:
              type: threshold
              expression: BURN
              conditions:
                - evaluator:
                    type: gt
                    params: [0.0144]
              refId: THRESHOLD
        for: 2m
        noDataState: NoData
        execErrState: Alerting
        labels:
          severity: page
          slo: availability
        annotations:
          summary: Checkout's error budget is burning at 14.4x
          runbook_url: https://wiki.internal/runbook/slo-burn

What you set noDataState to is where practice diverges. Set it to Alerting, and a brief scrape gap fires it. Set it to OK, and it stays silent even if data collection stops entirely. It's usually better to leave it at NoData and create a separate alert that watches for collection itself stopping.

- alert: ScrapeTargetDown
  expr: up{job=~"checkout.*|payment.*"} == 0
  for: 5m
  labels:
    severity: page
  annotations:
    summary: 'A scrape target has not responded for 5 minutes'

There are two things to watch when putting dashboard JSON into the repository. First, saving from the UI produces a diff full of noise like panel coordinates and version numbers. Put a normalization script in CI that strips unnecessary fields before saving, and reviews become possible. Second, hardcoding a data source by UID breaks in other environments. Use a data-source variable and have panels reference that variable.

Closing — Dashboards and Alerts Are About Designing Human Behavior

Panels and rules are the output; what you're really designing is what a person woken at 3am sees and does. Looked at that way, the calls get easy. Delete a panel that doesn't determine the next action when opened. Downgrade or remove an alert that has nothing to do when it fires.

Here are two checks you can run right now. First, pull the list of dashboards that haven't been opened once in the last 30 days. Nobody opens them, but the maintenance cost keeps accruing anyway. Second, count the alerts that fired over the last 30 days, and compute the fraction that actually led to action. If that fraction is below half, the alerting system has already lost credibility.