Split View: 질문에 답하는 Prometheus 메트릭 설계 — 타입 선택, 카디널리티 예산, rate 와 분위수의 함정

Designing Prometheus Metrics That Answer Questions — Choosing Types, Cardinality Budgets, and the Traps in rate and Quantiles

• Introduction — 40 Panels, and Still No Answer
• The Shape of Questions a Metric Can Answer
• Counters, Gauges, Histograms — Pick Wrong and a Calculation Becomes Impossible
• Label Cardinality Budgets
• Conditions Under Which rate Silently Gives the Wrong Answer
  • Trap 1 — The Window Is Narrow Relative to the Scrape Interval
  • Trap 2 — Applying rate After sum
  • Trap 3 — Graphing a Counter As-Is
• Conditions Under Which histogram_quantile Silently Gives the Wrong Answer
  • Trap 1 — Leaving le Out of the Aggregation
  • Trap 2 — Extrapolating Outside the Bucket Boundaries
  • Trap 3 — Averaging Quantiles
  • Native Histograms
• Recording Rules — When, and Under What Name
• Five Questions to Ask Yourself Before Putting Something on a Dashboard
• Closing — A Metric's Value Comes From Answering Questions, Not From Its Count

Introduction — 40 Panels, and Still No Answer

You open the dashboard while responding to an incident. There are 40 panels. CPU, memory, thread count, GC count, connection pool size, heap usage, requests per second. Every one of them has a graph drawn. But there's exactly one thing you want to know right now: "are users experiencing failures, and if so, what percentage."

That answer isn't in any of those 40 panels.

The problem with metric design isn't a shortage of data — it's a mismatch between the questions the collected data can answer and the questions that need answering. This post covers how to close that gap. Verified against Prometheus 3.13.0 LTS; native histograms stabilized in 3.8.0, but scraping them still has to be turned on with an explicit option.

The Shape of Questions a Metric Can Answer

A metric is an aggregate of a number that changes over time. This definition sets the shape of the questions it can answer.

The last three rows matter. The moment you add labels to try to answer those questions, a metric becomes an inferior substitute for logs or traces. Cardinality is what sets a metric's limit. Once you start putting anything that varies per request into a label, what you have isn't a metric anymore — it's an event store with a bad compression ratio.

The starting point is a list of questions. Write down the five questions on-call asks at 3am first, and build only the metrics that answer those five.

1. Are users experiencing failures, and what percentage
2. Has it gotten slower, and on which route
3. Since when, and does it overlap with a deploy time
4. Has it hit a capacity limit (queue length, connection pool, disk)
5. Is anything among the external services it depends on having trouble

Counters, Gauges, Histograms — Pick Wrong and a Calculation Becomes Impossible

Choosing a type isn't a matter of taste. Pick wrong, and a calculation you want to do later becomes impossible in principle.

A counter only ever increases monotonically. It resets to 0 when the process restarts. The value itself is meaningless — it becomes meaningful once you look at the per-second rate of change with rate. Request count, error count, bytes processed, and retry count fall into this category.

A gauge goes up and down. It represents the state at the current moment. Queue length, active connection count, memory usage, and temperature fall into this category.

A histogram records the distribution of observations as a set of bucket counters. Use it for "values you want to know the quantiles of," like latency or response size.

The most common mistake is recording latency as a gauge.

# Bad — only the last request's latency survives. You can compute neither a quantile nor an average
from prometheus_client import Gauge

last_latency = Gauge("http_request_duration_seconds", "Request processing time")

def handle(req):
    t0 = time.monotonic()
    resp = process(req)
    last_latency.set(time.monotonic() - t0)   # every prior value is simply gone
    return resp

With a 15-second scrape interval and 500 requests per second, only 1 out of 7,500 values gets stored. The rest end up as if they'd never existed. You can't compute p99 from this time series, and reprocessing the data later won't bring it back.

# Good — a histogram accumulates every observation into buckets
from prometheus_client import Counter, Histogram

REQUESTS = Counter(
    "http_requests_total", "Total request count",
    ["method", "route", "status_class"],
)
LATENCY = Histogram(
    "http_request_duration_seconds", "Request processing time",
    ["method", "route"],
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0),
)

def handle(req):
    route = req.route_template          # "/v1/orders/:id" — not the actual ID
    with LATENCY.labels(req.method, route).time():
        resp = process(req)
    REQUESTS.labels(req.method, route, f"{resp.status // 100}xx").inc()
    return resp

Notice status_class. Put the status code in as-is, and you get dozens of values, but most questions only need to know 2xx versus 5xx. The moment you need the specific code is when you're investigating, and that's when you look at the logs.

The last row is a problem that keeps recurring in batch jobs. You have to increment the counter on failure too, or you can't distinguish "ran but failed" from "never ran at all." Also keep a gauge of the last success timestamp alongside it.

JOB_RUNS = Counter("batch_job_runs_total", "Batch run count", ["job", "result"])
JOB_LAST_SUCCESS = Gauge("batch_job_last_success_timestamp_seconds", "Last success timestamp", ["job"])

def run_job(name, fn):
    try:
        fn()
        JOB_RUNS.labels(name, "success").inc()
        JOB_LAST_SUCCESS.labels(name).set(time.time())
    except Exception:
        JOB_RUNS.labels(name, "failure").inc()
        raise

Label Cardinality Budgets

Cardinality isn't a gut feeling — it's multiplication. The number of time series for one metric is the product of the label value counts, and that's further multiplied by the instance count.

http_request_duration_seconds_bucket
  route          120 (route templates)
  method           5
  le              11 (bucket boundaries + Inf)
  instance        40 (pod count)
  ------------------------------------
  = 120 * 5 * 11 * 40 = 264,000 time series

Add _sum and _count on top of this
  120 * 5 * 40 * 2 = 48,000
  ------------------------------------
Total roughly 312,000 time series — from one metric

A single Prometheus starts seeing memory and query responsiveness degrade sharply somewhere around a few million active time series. The exact limit depends on hardware and query patterns, but what matters is this: if one metric uses 300,000 time series, that one metric alone eats up a substantial chunk of the budget.

Labels you must never add are the ones whose value set is close to infinite.

• User ID, session ID, request ID, trace ID
• An unnormalized URL path — /v1/orders/A-99183
• Email, phone number, order number
• A timestamp, or a string containing one
• Raw error message text — mix in a fragment of a stack trace and the value set is effectively infinite
• A free-form query string

Some sit right on the border. pod or instance multiplies by the pod count, and since autoscaling and rolling deploys keep changing the values, it accumulates over time. A practical strategy is to keep it only on metrics that genuinely need per-instance distinction, and for everything else, aggregate with a recording rule and keep the raw data short-lived.

Here are queries to diagnose the current state.

# Top 10 metrics by time-series count
topk(10, count by (__name__)({__name__=~".+"}))

# Which labels create cardinality for a specific metric
count(count by (route) (http_requests_total))
count(count by (instance) (http_requests_total))
count(count by (status) (http_requests_total))

# Trend of total active time series — a step jump means a new label got introduced
prometheus_tsdb_head_series

# How many samples each scrape target sends (scrape gets rejected past the cap)
topk(20, scrape_samples_scraped)

prometheus_tsdb_head_series is worth alerting on. Cardinality incidents mostly show up as a step jump right after a deploy, so lining up the timing with deploys lets you find the culprit commit right away.

There's also a way to block it at collection time. Set a sample-count cap in the scrape config, and it stops a target that's exploded from bringing down all of Prometheus.

# prometheus.yml
scrape_configs:
  - job_name: checkout-api
    sample_limit: 20000              # if this target sends more than this, the scrape is treated as a failure
    label_limit: 24
    label_value_length_limit: 256
    scrape_interval: 15s
    metric_relabel_configs:
      # For incident response — strip a problem label at collection time.
      # Why replace with an empty value instead of labeldrop: labeldrop matches purely on the label
      # "name," so it would strip user_id from every metric in this job. Written as below,
      # it can be stripped from http_requests_total alone.
      # An empty-valued label is the same as a missing label in Prometheus's data model.
      - source_labels: [__name__]
        regex: 'http_requests_total'
        target_label: user_id
        replacement: ''
        action: replace
      # Drop metrics that aren't needed at all
      - source_labels: [__name__]
        regex: 'go_gc_duration_seconds.*|python_gc_.*'
        action: drop

Hitting sample_limit fails the entire scrape for that target, so set the value generously, but always set it. Without a cap, one service's mistake can halt monitoring across the board.

Conditions Under Which rate Silently Gives the Wrong Answer

rate computes the per-second rate of increase from a range vector's first and last samples, and corrects for counter resets. There are three traps.

Trap 1 — The Window Is Narrow Relative to the Scrape Interval

rate needs at least two samples inside the window. If the scrape interval is 15 seconds and the window is 20 seconds, there are moments when only one sample falls in, and the result comes back empty. This shows up as a gap on a graph, and as "the condition never holds" in an alert.

# Dangerous — with a 15s scrape interval, a [20s] window has moments with only 1 sample
rate(http_requests_total[20s])

# Safe — at least 4x the scrape interval. For alerts, [5m] or more is recommended
rate(http_requests_total[1m])
rate(http_requests_total[5m])

The rule of thumb is simple: use 5 minutes or more for alerts, and Grafana's rate-interval variable for dashboards. The 4x rule lets the result survive even if one or two scrapes get missed.

Trap 2 — Applying rate After sum

This mistake survives a long time because the result looks plausible.

# Wrong — sum the counters first, and an instance restart (a reset) goes undetected
rate(sum(http_requests_total) by (route)[5m:])

# Right — rate first, then aggregate
sum(rate(http_requests_total[5m])) by (route)

Counter-reset correction is only accurate at the level of an individual time series. When one pod restarts, that series drops to 0, but once it's already been summed, it just looks like "the total dropped a little," and doesn't get recognized as a reset. The result is a request rate that comes out lower than reality, making it look like traffic drops every time right after a deploy.

Trap 3 — Graphing a Counter As-Is

# Produces a line that just keeps rising to the upper right. No information at all
http_requests_total

# Per-second rate of increase — this is the value you actually wanted to see
sum(rate(http_requests_total[5m])) by (route)

# Total increase over a specific window — good to use in an alert message
sum(increase(http_requests_total{status_class="5xx"}[1h])) by (route)

increase is rate multiplied by the window length. So it produces a value that isn't an integer. When 3 errors actually happened but increase returns 3.4, that's not a bug — it's a result of extrapolation. Don't use it for a calculation that needs "exactly N."

irate only looks at the last two samples. It's useful for seeing an instant reaction on a dashboard, but never use it in an alert. A single burst of noise sets it off.

Conditions Under Which histogram_quantile Silently Gives the Wrong Answer

Quantile computation has even more traps.

Trap 1 — Leaving le Out of the Aggregation

# Wrong — drop le, and the buckets get mashed together into a meaningless number
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (route))

# Right — always keep le
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route)
)

This mistake returns a number instead of throwing an error. So it sits on dashboards for months.

Trap 2 — Extrapolating Outside the Bucket Boundaries

histogram_quantile linearly interpolates between buckets. If p99 sits above the last finite bucket, the function returns that last boundary value. If the real p99 is 8 seconds but the last bucket is 2.5 seconds, the result is forever 2.5 seconds.

# Diagnosis — look at the ratio between the Inf bucket and the last finite bucket
sum(rate(http_request_duration_seconds_bucket{le="+Inf"}[5m])) by (route)
-
sum(rate(http_request_duration_seconds_bucket{le="10.0"}[5m])) by (route)

If this value is greater than 0, it means there are requests beyond the last bucket, and p99 may be in an unreliable state.

Conversely, if the buckets are too sparse, interpolation error is large. If the buckets are only 0.1 and 1.0, but most requests cluster at 0.15 seconds, the p99 estimate linearly cuts between 0.1 and 1.0 and ends up far off from reality. Set buckets densely around the SLO threshold. If the target is 300ms, the buckets should include 0.2, 0.25, 0.3, 0.4, 0.5.

Trap 3 — Averaging Quantiles

# Wrong — quantiles don't support arithmetic averaging
avg(histogram_quantile(0.99, ...))

# Right — sum the bucket counters first, then compute the quantile
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

The average of per-instance p99s isn't the overall p99. This is exactly why a histogram is valuable. Bucket counters can be added together, and computing the quantile after adding them gives you an estimate over the whole distribution. Expose an application-computed p99 gauge instead, and you lose this property.

Native Histograms

Native histograms have been a stable feature since Prometheus 3.8.0. Bucket boundaries are managed automatically with an exponential schema, so you don't have to pick them yourself, and it's represented as a single time series, which cuts cardinality dramatically. Scraping still has to be turned on explicitly, though.

# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: checkout-api
    # Native histogram scraping is explicit opt-in
    scrape_native_histograms: true
    static_configs:
      - targets: ['checkout-api:8000']

# Fixed buckets: the _bucket series carries the le label
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route))

# Native histogram: pass the metric itself, without le
histogram_quantile(0.99, sum(rate(http_request_duration_seconds[5m])) by (route))

It's safer to convert starting with new metrics. Change an existing metric, and at that point both past data and queries break continuity.

Recording Rules — When, and Under What Name

A recording rule moves computation from query time to evaluation time. There are four criteria for creating one.

1. When the same expression repeats in three or more places
2. When query execution exceeds 2 seconds
3. When an alert rule runs a heavy expression on every evaluation
4. When you want to aggregate a high-cardinality source and retain it long-term

Names follow the level:metric:operation convention. Colons are used only in recording rules and never in a raw metric name. Follow this convention, and you can tell what dimension is left just by looking at the name.

# rules/http.yml
groups:
  - name: http_sli
    interval: 30s
    rules:
      # Tier 1 — aggregate from the source exactly once
      - record: route:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (route)

      - record: route:http_requests_errors:rate5m
        expr: sum(rate(http_requests_total{status_class="5xx"}[5m])) by (route)

      # Tier 2 — references tier 1. Doesn't rescan the source
      - record: route:http_error_ratio:rate5m
        expr: |
          route:http_requests_errors:rate5m
            /
          route:http_requests:rate5m

      - record: route:http_request_duration_seconds:p99_rate5m
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route)
          )

      # Service-wide — a summary with the route dimension removed
      - record: service:http_error_ratio:rate5m
        expr: |
          sum(rate(http_requests_total{status_class="5xx"}[5m]))
            /
          sum(rate(http_requests_total[5m]))

The layered structure is the key part. When multiple rules reuse tier 1, scanning the source time series ends in one pass. That said, rules must never form a circular reference, and promtool doesn't catch this, so it has to be checked in review.

Calculate how many time series a recording rule creates before deploying it. With 120 routes and a rule created for each of 5 windows, that's 600 new time series. With 50 rules, that's 30,000.

Run validation in CI.

# Syntax check
promtool check rules rules/http.yml

# Unit tests — feed input series and verify expected values
promtool test rules tests/http_test.yml

# Full config check
promtool check config prometheus.yml

# tests/http_test.yml
rule_files:
  - ../rules/http.yml

evaluation_interval: 30s

tests:
  - interval: 15s
    input_series:
      - series: 'http_requests_total{route="/v1/orders", status_class="2xx"}'
        values: '0+150x40'
      - series: 'http_requests_total{route="/v1/orders", status_class="5xx"}'
        values: '0+3x40'
    promql_expr_test:
      - expr: route:http_error_ratio:rate5m
        eval_time: 8m
        exp_samples:
          - labels: 'route:http_error_ratio:rate5m{route="/v1/orders"}'
            value: 0.0196078431372549

Compute the expected value yourself ahead of time, and CI will catch it if someone later changes the meaning of a rule while "optimizing" it.

Five Questions to Ask Yourself Before Putting Something on a Dashboard

Every time you build a panel, run it through the checks below. If it doesn't pass, don't build that panel.

1. Can you write the question this panel answers in one sentence? "CPU utilization" isn't a question. "Is this service hitting a CPU limit and causing latency" is a question.
2. Is it decided what to do when the value gets worse? A metric you can look at but that triggers no action stays an exploratory query, not a dashboard panel.
3. How does this value connect to user experience? GC count means nothing by itself. It becomes meaningful only placed alongside latency.
4. Do you know the normal range? If you don't know what's normal, you can't know what's anomalous either. Decide the axis range and threshold line together.
5. Is this query graphing a counter as-is, or averaging a quantile? Double-check the traps covered in the two sections above.

Panel order follows question order too. The top is user-facing metrics, below that is candidate causes, and the bottom is infrastructure resources. Reading top to bottom gives you the flow "is there impact, where's it from, is it a resource issue."

Closing — A Metric's Value Comes From Answering Questions, Not From Its Count

It's common for on-call to come up empty in a system with 300 metrics. On the flip side, some teams narrow down most incidents with a well-chosen 20. The difference isn't the volume collected — it's whether each metric has an explicit statement of which question it exists to answer.

There are two cheap checks you can run right now. First, pull the top 10 metrics by time-series count and write down, for each, "what question does this answer." The ones you can't answer usually make up half the cost. Second, search dashboards for every query containing histogram_quantile and check whether by (le is present. There's usually one missing it.

Further reading.

• Prometheus: Metric Types
• Prometheus: Metric and Label Naming
• Prometheus: Histograms and Summaries
• Prometheus: Native Histograms specification
• Prometheus: Recording Rules