Skip to content
Published on

Native Histograms Are Stable Now — So Why Haven't You Turned Them On Yet

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Introduction — What the Word "Stable" Actually Changed

Prometheus native histograms have been a long time coming. The first sentence of the official spec summarizes the history — introduced as an experimental feature in November 2022, first supported in server version v2.40.0, and requiring the --enable-feature=native-histograms flag to turn on. And in the same paragraph:

Starting with v3.8.0, native histograms are supported as a stable feature.

3.8.0 was released on December 2, 2025 (per the GitHub release; the CHANGELOG header says November 28). A feature that had worn the "experimental" tag for more than three years was finally stable.

But at the time of writing, there's a more urgent event. 3.13 came out on July 1, 2026, and it's the new LTS. And the 3.5 LTS that many organizations are likely running right now goes end-of-support on July 31, 2026. That's two weeks out. Here's the LTS table from the release cycle doc, reproduced as-is.

Release          Date          End of support   Status
Prometheus 2.53  2024-06-16    2025-07-31       End of life
Prometheus 3.5   2025-07-14    2026-07-31       Supported
Prometheus 3.13  2026-07-01    2027-07-31       Supported
TBD              2027-06       2028-07-31       Upcoming

Overlay the two facts and one conclusion falls out. The 3.5 LTS (July 2025) was cut four months before native histograms went stable. And the LTS doc explicitly names what's excluded from support scope — things classified unstable under the API stability guarantee, experimental features, and OpenBSD support. In other words, if you were on 3.5 LTS, native histograms were an experimental feature outside the LTS support scope from the start, and 3.13 is the first LTS to carry this feature in stable state. That's why, for organizations on the LTS track, this is an actual decision point right now — not "we'll get to it eventually."

The Timeseries Count a Classic Histogram Produces

Let's start with why this feature was wanted in the first place. The problem is cardinality.

A classic histogram looks like a histogram in the exposition format, but the moment it's scraped it splits into multiple float timeseries. The spec describes this decomposition as follows — summaries and (the classic version of) histograms are decomposed into float components upon ingestion, and both types have sum and count components, with histograms additionally carrying a number of bucket samples.

The spec's own example shows exactly what happens in practice. A single histogram called rpc_latency_seconds, scraped classically, becomes rpc_latency_seconds_sum, rpc_latency_seconds_count, and a number of rpc_latency_seconds_bucket timeseries with different le labels. If there are N buckets, that's N+2 timeseries. Scraped as a native histogram, it's one rpc_latency_seconds timeseries, exactly as the name suggests.

This difference multiplies when you partition by labels. In the spec's own words, if you attach labels to an already-heavy classic histogram and end up with 100 histograms, the cost goes up 100x. Native histograms are different — when partitioned, the individual histograms each tend to fill fewer buckets (a histogram that only holds requests with status code 404 usually has a narrow distribution), so the total number of filled buckets grows much more slowly than a multiple of the histogram count. That's the basis for the spec's claim that "even partitioning by labels becomes much more affordable now."

There's a temptation here to invent a number, and I won't. Both the Prometheus spec and the Grafana Mimir docs describe native histograms as offering "higher resolution at much lower resource cost than classic histograms" — but that's a qualitative statement from vendor/project docs, not a conditioned benchmark figure. The magnitude of the savings depends entirely on how many buckets you were already using and how wide your distribution is — a histogram with 5 buckets won't gain much.

The Schema — Why Bucket Boundaries Are a Formula, Not a Setting

The core idea of native histograms is that bucket boundaries are determined by a formula rather than set by a human. The value that determines this is the schema, and per the spec it's an int8 integer whose currently valid values are -53, and -4 through +8. -53 is for custom bucket boundaries (NHCB); the rest are the standard exponential schemas.

For standard schema n, the boundaries for the positive bucket at index i are computed as follows (the spec writes it in Python syntax).

# upper bound (inclusive):  (2**2**-n)**i
# lower bound (exclusive):  (2**2**-n)**(i-1)

An important property falls out of this. Schema n has half the resolution of schema n+1, so a histogram at n+1 can be converted to n by merging neighboring buckets. This is why standard schemas are mutually mergeable, and it's the basis for queries that sum histograms across multiple instances. The problem in classic histograms — where two services using different bucket boundaries couldn't be combined — disappears here.

On the instrumentation side, you don't pick the schema directly; you specify a growth factor. That's the Go client's NativeHistogramBucketFactor, and the comment in the client_golang source gives the exact figures — a generally good tradeoff between cost and accuracy is 1.1 (each bucket at most 10% wider than the previous one), which works out to 8 buckets per power-of-two interval (8 between 1 and 2, 8 between 2 and 4, 8 between 4 and 8, and so on). The actual factor is computed as 2^(2^-n), where n is chosen as the largest integer between -4 and 8 that yields a factor no greater than the one specified. So the smallest possible factor is about 1.00271 (= 2^(2^-8)).

The area near zero is handled separately. Exponential buckets approach zero infinitely, so any observation whose absolute value is at or below a threshold is collected into a "zero bucket." The Go client's default threshold is DefNativeHistogramZeroThreshold = 2.938735877055719e-39.

And storage is a sparse representation — empty buckets are (nearly) free. That's why claiming to "cover the entire float64 range" doesn't bankrupt you. However wide the covered range is, only the buckets that are actually filled cost anything.

What Stable Doesn't Guarantee

This is where the real substance of this post begins. The stable declaration means the way the server handles this data type won't break before v4 — it does not mean "you can just turn it on now." The spec itself attaches caveats right after the stable declaration.

You still have to turn it on explicitly. Scraping must be enabled with the scrape_native_histograms setting, and Remote-Write transmission has to be turned on separately with send_native_histograms. In 3.8, the legacy feature flag only had the effect of setting this config to true; starting with 3.9, the flag is a complete no-op, so you must set the config explicitly. A parenthetical in the spec tells you the roadmap — starting with v4, both settings default to true. So right now we're in a middle stretch: "stable but opt-in," with v4 as "the default."

Exposition is protobuf-only. This is the thing that trips people up most often in practice. The spec is explicit that the classic Prometheus text format has not been extended for native histograms and there's no plan to. OpenMetrics text format support has a proposal document, but its implementation status remains Not implemented. The PR for OpenMetrics protobuf support (OpenMetrics#256) was closed without merging on March 19, 2026 — and the reason is interesting: it's effectively a branding cleanup. The decision was that since the Prometheus protobuf exposition format already supports native histograms and everything else, that format becomes the official protobuf format, while OpenMetrics is treated as referring to the text-based format. To sum up, as of July 2026, there's exactly one path to exposing native histograms: the classic Prometheus protobuf format.

Client libraries haven't caught up yet. This one is worth checking for yourself. The latest client_golang release is v1.23.2 (September 5, 2025), and the comment on HistogramOpts still reads, word for word:

NOTE: Native Histograms are still an experimental feature. Their behavior might still change without a major version bump. Subsequently, all NativeHistogram... options here might still change their behavior or name (or might completely disappear) without a major version bump.

You could argue v1.23.2 predates 3.8.0 (December 2025) and the wording is just stale — and in fact the same sentence is still present on main. The point is that the subject of the stable declaration is the server. The server's data model and its storage/query behavior are stable, but the instrumentation library's option API is not documented as carrying the same guarantee.

Python is simpler — the issue tracking native histogram implementation in client_python is still open. This is one of the motivations the text-format proposal document cites explicitly: some libraries, like the Python client, don't want to add a protobuf dependency just to expose native histograms.

Kubernetes is alpha. Native histograms landed in Kubernetes component metrics starting with v1.36 (released April 22, 2026), and the NativeHistograms feature gate's status is alpha, default false. It applies to kube-apiserver, kube-controller-manager, kube-scheduler, kubelet, and kube-proxy, and can be turned on per component. When enabled, it's a dual exposition scheme exposing both classic and native simultaneously, and the request's Accept header decides which one you get.

The Protobuf Scrape Cost — the Problem That Originally Blocked Stable

There's an interesting bit of history here. Turning on scrape_native_histograms changes content negotiation so that scrape config prefers protobuf. As the spec states, when no default is specified, the effective priority order becomes:

[ PrometheusProto, OpenMetricsText1.0.0, OpenMetricsText0.0.1, PrometheusText0.0.4 ]

The problem was that this applied to every target under that scrape config. Scraping a handful of native histograms meant flipping your entire scrape over to protobuf. Issue #14668 tackled this head-on, and its diagnosis was — protobuf parsing generates a lot of memory churn, which mostly shows up as increased GC CPU usage, and depending on usage patterns CPU can rise significantly; in real-world cases a 2x increase has been observed. And the same issue body states, in bold:

(Or in other words: This issue needs to get resolved to declare native histograms a stable feature.)

In other words, this performance problem was a precondition for the stable declaration. The issue was closed as completed on February 13, 2025, and the stable declaration followed in December 2025. In discussion before it closed, a maintainer reported that in realistic benchmarks after optimization, it was 3x faster than the text parser and used 10% less allocated memory.

There's something to be honest about here. The spec doc still carries a note saying that scraping via a text-based format generally uses far fewer resources than protobuf, and links this same issue as its evidence — but that issue is closed as completed. The doc's note looks like it reflects the pre-optimization state and is stale. So it would be an overreach, given current sources, to say either "protobuf scraping is still 2x more expensive" or "this is now fully resolved." You have no choice but to measure CPU in your own environment. Compare the CPU of both the scraped process and Prometheus itself before and after turning it on.

There is one practical legacy this issue left behind, though: tools for narrowing exposition scope. Setting scrape_protocols explicitly lets you force specific targets onto non-protobuf formats even when scrape_native_histograms is on. And 3.13 added internal labels for per-target overrides via relabeling — __scrape_native_histograms__ and __always_scrape_classic_histograms__ (#18929), and __convert_classic_histograms_to_nhcb__ (#18840). That means you can turn native histograms on precisely for the targets that expose them, without splitting your scrape config.

NHCB — a Path That Doesn't Touch Instrumentation Code

Re-instrumenting every classic histogram to native from scratch is a big job. So there's a middle path: schema -53, a native histogram with custom bucket boundaries (NHCB).

Turning on the convert_classic_histograms_to_nhcb scrape option makes Prometheus convert classic histograms into NHCB at ingestion time. You leave the instrumentation code untouched and get the storage efficiency anyway. In the spec's words, NHCB's main advantage is that storage cost is generally much cheaper, and in particular the marginal cost of adding a bucket is low, so even classic histograms with many buckets become affordable to ingest.

The tradeoff, though, is clear. NHCB is fundamentally limited in its ability to merge with other histograms. The spec warns as follows — automatic reconciliation between different bucket layouts only keeps the boundaries that match exactly, so if most of the boundaries line up you get a usable result, but if bucket layouts diverge arbitrarily you can end up with a histogram where no boundary matches, leaving only a single overflow bucket. That's why the spec recommends using schema -53 only as an informed decision for specific use cases. If you push a codebase with inconsistent bucket boundaries across services entirely into NHCB and run aggregate queries, you'll quietly get useless results.

An Honest Look at the Tradeoffs — Why Migration Is Hard

The spec's migration section boils the problem down to three points. This is the real difficulty of adopting this feature.

First, queries work differently. In most cases the changes are minimal and intuitive, but the spec's own account is that there are tricky edge cases that make reliable automatic conversion hard. In other words, you have to move dashboards and alerting rules by hand.

Second, classic and native don't aggregate with each other. This is the most painful part. If you flip from classic to native at some point in time, it's hard to build a dashboard that straddles that cutover, and any range vector that spans the cutover is necessarily incomplete — pick classic and you only get the earlier data, pick native and you only get the later data. The spec's own example is concrete.

histogram_quantile(0.9, rate(rpc_duration_seconds[1d]))

This query, computing the 90th-percentile latency over a day, only covers a shorter window than that if you haven't been collecting native histograms for at least a full day. So the spec's recommendation is conservative — collect classic and native in parallel for a while. If you've run them in parallel for a month, any dashboard that doesn't look back further than a month can just switch over without worrying about the cutover point. Parallel collection is turned on with the always_scrape_classic_histograms option. Classic ends up as suffixed timeseries and native as a single timeseries under the plain name, so they don't collide in the TSDB. Of course, you pay for both the whole time.

Third, you can't pin arbitrary bucket boundaries. This can be decisive for teams doing SLOs. A classic histogram can be custom-designed with a boundary planted exactly where you care about it — for instance, if "percentage of responses within 300ms" is your SLO, pinning a boundary at 300ms makes that percentage an exact value rather than an estimate. A native histogram on a standard schema has high resolution but can't put a boundary at an arbitrary value. The Grafana Mimir docs flag the same point as a drawback — there's no way to configure arbitrary bucket boundaries such as thresholds that matter for SLO definitions, and the fraction of observations above or below a given threshold generally has to be estimated via interpolation. The spec is candid that in these cases "the user experience with native histograms may in fact get worse," and offers the option of simply leaving that particular histogram unmigrated as a solution.

One more thing on top of this. Because buckets are created dynamically as they're first filled, a wider-than-expected distribution creates more buckets than expected and uses more memory. The spec even warns that if observations can be manipulated externally, this can become a denial-of-service vector via memory exhaustion. Yet the Go client's NativeHistogramMaxBucketNumber defaults to 0, i.e. unlimited (the spec itself leaves this as a TODO). A commonly used value is 160, which matches the default for OTel exponential histograms. You can also defend on the collection side — the scrape config's native_histogram_bucket_limit (a cap on the number of buckets per histogram) and native_histogram_min_bucket_factor (a floor on the growth factor), both of which reduce resolution to fit when exceeded. Note, though, that if NHCB exceeds the bucket limit it can't reduce resolution, so the scrape simply fails.

So, Should You Turn It On?

Worth turning on now

  • You're using classic histograms with many (dozens of) buckets, and cardinality is already hurting because of label partitioning. Since the savings scale with the original bucket count, this is where it pays off the most.
  • Observations whose range is hard to know in advance, like latency distributions. Not having to guess bucket boundaries up front is a real, practical gain here.
  • You're instrumenting in Go and planning to upgrade your server to the 3.13 LTS. This is the smoothest path on your current stack right now.
  • You can't touch instrumentation code but want to cut storage cost — here you have the option of turning on NHCB conversion alone. Just understand the merge limitations first.

Not yet

  • Histograms whose bucket boundaries are pinned exactly to your SLO thresholds. The moment you switch to interpolated estimates, the nature of the SLO calculation changes. Just leave these classic.
  • Anywhere instrumented with the Python client. It's still unimplemented, so there's no choice to make yet.
  • Histograms with only a handful of buckets. There's little to gain, while you still pay the full migration cost.
  • Setups where most scrape targets don't expose native histograms but the scrape config would flip everything to protobuf as a unit. Narrow the scope with 3.13's per-target relabeling, or control it with scrape_protocols.
  • Anywhere the goal is Kubernetes component metrics. It's disabled by default in the 1.36 alpha, so turning on an alpha feature gate in a production cluster is a separate decision you'd need to make on its own.

Closing

To sum up: native histograms really are stable now (3.8.0, December 2025), 3.13 is the first LTS to carry this feature as stable (July 1, 2026), and the 3.5 LTS ends July 31. The data model is genuinely good — N+2 timeseries become 1, you no longer have to guess bucket boundaries, and standard schemas merge with each other.

But you have to read the subject of the word "stable" precisely. It means the way the server handles this data type won't break before v4 — it does not mean the ecosystem is ready. Exposition is still protobuf-only, the Go client still documents its own options as experimental, Python is unimplemented, and Kubernetes is alpha. The direction toward defaults flipping on in v4 is clear, but anyone crossing this in-between stretch right now has to check these gaps for themselves.

So the practical order of operations is this: upgrading to 3.13 is something you have to do anyway because of July 31, so do that first, and treat native histograms as a separate decision. Then pick just a handful of your most cardinality-painful histograms and start them with parallel collection, measure the CPU impact of the protobuf switch yourself, and set the parallel-collection period long enough to give yourself time to move dashboards over. This is a feature that took three years to go stable. There's no reason you need to migrate everything in two weeks.

References

Discuss on Twitter • View on GitHub