Skip to content
←

필사 모드: Parquet 2.13.0 Adds nan_count and IEEE 754 Total Order — How Float Statistics Got Fixed After 3 Years and 3 Months

English
0%
정확도 0%
💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — Statistics Exist to Let You Skip, But That Broke for Float

A big part of why Parquet files are fast comes down to statistics. Every row group, every column chunk, every page has a summary written down — min, max, null_count — and a query engine looks at just that summary, decides "there's no answer in this chunk," and skips it wholesale. Predicate pushdown and page pruning stand on top of this. The full picture of columnar storage is laid out in Columnar Storage Deep Dive 2025: Parquet, ORC, Apache Arrow, Dremel — Why Analytics DBs Are 10,000x Faster.

For integers or strings, this contract is simple. The writer records the true min and max, and the reader judges by that range. But on FLOAT and DOUBLE columns, this simple model breaks down. And it doesn't break down by "getting a bit slower" — it breaks down in a way where queries fail to find rows that are there.

parquet-format 2.13.0, released on June 13, 2026, closed this hole at the spec level. This post looks at exactly what that change is, why it took more than 3 years, and whether it's actually usable right now.

Why Float Statistics Are Uniquely Hard

There are two reasons. The Parquet community's official blog post (Jan Finis, Gang Wu, 2026-05-29) lays both of them out.

First, signed zero. -0.0 and +0.0 are treated as equal under normal arithmetic, but their bit patterns differ. Without a defined order, different libraries produce different statistics from the same data. Parquet's longstanding workaround was a dodge — force the minimum to be written as -0.0 whenever it is 0.0, regardless of the actual sign, and force the maximum to be written as +0.0 whenever it is 0.0. Readers are then warned that "a min of +0.0 may still mean -0.0 is present." That's a defense, not an ordering.

Second, NaN. This is the real problem. In IEEE 754, NaN has no order at all. x < NaN, x > NaN, and x == NaN are all false. If a writer simply feeds NaN into the min/max calculation, the resulting bounds are useless. So Parquet chose the opposite direction — exclude NaN from the statistics.

This is where the accident happens. To borrow the blog post's own example: say a page's max statistic is 0.0, and that page contains one NaN. A query engine that treats NaN as greater than every value issues a predicate like x > 1.0. The engine looks at the statistics, sees that max is 0.0, and can wrongly skip the page even though a row satisfying the condition is actually in it. In the original's own words, without knowing whether NaN is present, an engine cannot safely perform this kind of page pruning on floating-point columns at all.

In other words, the statistics were "the range with NaN removed," but readers had no way to know that. There was no nan_count.

This Isn't Theoretical — It Still Reproduces in DataFusion Today

If that sounds abstract, there's an actual reproduction. DataFusion issue #15812, "Pruning of floating point Parquet columns is incorrect when NaN is present", was filed on April 22, 2025, and is still open as of today (2026-07-16).

The example in the issue body is this. If the column is [1.0, 0.0, -1.0, NaN, -2.0], max is recorded as 1 and min as -2 (NaN excluded). But issuing select * from ... where x > 2 returns 0 rows, because no chunk has max > 2. Since DataFusion treats NaN as greater than 2, the correct answer should be one row — the NaN.

The reproduction attached to the issue uses an actual file from the parquet-testing repository.

> select * from 'parquet-testing/data/float16_nonzeros_and_nans.parquet' where x > arrow_cast(2.0, 'Float16');
+---+
| x |
+---+
+---+
0 row(s) fetched.

The expected result is one row containing NaN. And that file is still in the repository today.

What makes the problem harder is that engines don't even agree on NaN comparisons. Looking at the execution results alamb posted on arrow-rs tracking issue #8156: on the same data, DataFusion judges NaN > 34 as true and -NaN > 34 as false (because arrow-rs uses total order for floating-point comparisons), while PostgreSQL returns true for both. On the same thread, tustvold goes as far as saying — so many implementations have bugs (his phrasing includes past versions of arrow-rs itself) that DataFusion should not push down floating-point predicates at all, and if it is doing so, that is a bug that needs fixing.

This is why the spec had to be fixed. Leaving it to each engine to get right on its own was not going to work.

PARQUET-2249 — The Impossible Requirement ColumnIndex Was Stuck With

The problem has a name: PARQUET-2249. Its title is "Parquet spec (parquet.thrift) is inconsistent w.r.t. ColumnIndex + NaNs," and it was filed by Jan Finis on February 19, 2023.

The core of it is this. Say a data page is entirely NaN. This page is not a "null page" — the data is physically there, and NaN is different from missing (null). Yet the existing guidance says to exclude NaN from the min/max calculation. Meanwhile, ColumnIndex requires valid min_values and max_values for any page that isn't null. You must write a value when there is not a single usable non-NaN value to write — the contradiction inside the spec never closes.

On the Statistics side, you could combine num_values, null_count, and nan_count to derive "all non-null values are NaN." But ColumnIndex has no num_values field. Reading parquet.thrift at 2.13.0 directly, ColumnIndex's fields are only null_pages, min_values, max_values, boundary_order, null_counts, the level histograms, and the newly added nan_counts. You simply cannot do that arithmetic here.

3 Years, 3 Months — Two Failed Attempts, Then Consensus

  • 2023-02-19 — PARQUET-2249 filed (Jan Finis).
  • 2023-03-22 — PR #196, "Add nan_count to statistics". A proposal to add nan_count to statistics. It made the presence of NaN explicit and, being an optional field, was a safe migration path that wouldn't break old readers. After 99 comments (31 general + 68 review), it closed without being merged — because it never solved the ColumnIndex dilemma.
  • 2023-11-22 — PR #221, "Introduce IEEE 754 total order". A proposal to introduce an entirely new column order. In exchange for giving signed zero and NaN bit patterns a deterministic place, it had a fatal weakness — because NaN sits at both extremes of the total order, even a single NaN would contaminate min or max, completely disabling ordinary numeric predicate pushdown. After 52 comments, it too went unmerged.
  • 2025-04-22 — DataFusion #15812 filed. While the spec discussion was still circling, it showed up as an actual engine returning wrong answers.
  • 2025-08-09 — PR #514, "PARQUET-2249: Introduce IEEE 754 total order & NaN-counts". It combines the two.
  • 2026-05-26 — PR #514 merged. Open for about 9.5 months, with 67 comments (17 general + 50 review); the net change was 98 lines added / 6 lines removed across 3 files.
  • 2026-06-13 — parquet-format 2.13.0 released.

From the issue being filed to the spec being merged took 3 years and 3 months. Given that the final change was 98 lines, it's clear that what was hard about this project wasn't the code — it was reaching consensus.

The logic behind the consensus is this. The new total order strictly defines how bounds are compared, and nan_count explicitly marks the presence of NaN. And decisively — since none of the existing writers use the new order, the spec can safely mandate nan_count whenever IEEE_754_TOTAL_ORDER is used, while still treating old files leniently. It amounts to creating a clean, brand-new blank slate.

Exactly What 2.13.0 Changed in the Spec

It's faster to look at the IDL than to talk about it. In 2.12.0's parquet.thrift, neither nan_count nor IEEE754TotalOrder exists (zero hits on a string search). Three new things appear in 2.13.0.

First, field 9 was added to Statistics.

   /**
    * Count of NaN values in the column; only present if physical type is FLOAT
    * or DOUBLE, or logical type is FLOAT16.
    * If this field is not present, readers MUST assume NaNs may be present
    * (i.e. MUST assume nan_count > 0 and MAY NOT assume nan_count == 0).
    */
   9: optional i64 nan_count;

The important part here is the last sentence. A field being absent and being zero are different things. Absent means "unknown," and readers must assume NaN might be present. Without this rule, every old file would be misread as "no NaN."

Second, a per-page list was added to ColumnIndex.

   /**
    * A list containing the number of NaN values for each page. Only present
    * for columns of physical type FLOAT or DOUBLE, or logical type FLOAT16.
    * If this field is not present, readers MUST assume that there might be
    * NaN values in any page.
    */
   8: optional list<i64> nan_counts

Third, the ColumnOrder union gained a new variant.

/** Empty struct to signal IEEE 754 total order for floating point types */
struct IEEE754TotalOrder {}

union ColumnOrder {
  1: TypeDefinedOrder TYPE_ORDER;
  2: IEEE754TotalOrder IEEE_754_TOTAL_ORDER;
}

The spec describes the new order like this — it follows the totalOrder predicate from clause 5.10 of IEEE-754 (2008 revision), and only FLOAT, DOUBLE, or logical-type FLOAT16 columns may use it. Intuitively, it sorts floating-point values in mathematical order while placing -0 below +0, -NaN below everything, and +NaN above everything, and it defines an order even between different bit representations of the same value.

And when IEEE_754_TOTAL_ORDER is used, recording nan_count becomes mandatory. min/max hold the min/max among non-NaN values, but if every non-null value is NaN, they fall back to the min/max NaN under the total order — this is the exact point that resolves PARQUET-2249's impossible requirement. Since ColumnIndex has no num_values, the mere appearance of NaN in min_values or max_values itself becomes the signal that "this page is entirely NaN."

The existing TYPE_ORDER wasn't simply discarded either. For TYPE_ORDER, 2.13.0 now states that "float types have underspecified, ambiguous handling of NaN and -0/+0, so writers are recommended to use IEEE_754_TOTAL_ORDER," and newly spells out that if you use it anyway, you must write nan_count (even if it's 0) and compute min/max from non-NaN values only. A good number of what were "should" in 2.12.0 have been tightened to "must."

The Total Order Implementation Is Surprisingly One Line

The name sounds grand, as if it needs a heavy comparison engine, but it doesn't. Here's the Rust sketch the official blog gives.

pub fn totalOrder(x: f64, y: f64) -> bool {
    let mut x_int = x.to_bits() as i64;
    let mut y_int = y.to_bits() as i64;
    x_int ^= (((x_int >> 63) as u64) >> 1) as i64;
    y_int ^= (((y_int >> 63) as u64) >> 1) as i64;
    return x_int <= y_int;
}

It's a trick that preserves the IEEE bit pattern, flips only the negative values so the integer representation sorts in the correct order, and then just compares as integers. 32-bit floats work the same way.

And the actual implementation really is done this way. Open up parquet-java's PrimitiveComparator and you'll find the same trick.

public int compare(double d1, double d2) {
  long d1Long = Double.doubleToRawLongBits(d1);
  long d2Long = Double.doubleToRawLongBits(d2);
  d1Long ^= ((d1Long >> 63) >>> 1);
  d2Long ^= ((d2Long >> 63) >>> 1);
  return Long.compare(d1Long, d2Long);
}

The key point is that it's doubleToRawLongBits, not doubleToLongBits. The former normalizes NaN away, while the latter preserves NaN's sign and payload bits as-is. For the total order to "define an order even between different bit representations of the same value," it has to be raw.

Current Implementation Status — the Spec Shipped, the Engines Haven't Caught Up

This is the part of the post that needs to be the most honest. Just because the spec shipped doesn't mean your pipeline is fixed. What follows is the result of checking each implementation directly, as of July 16, 2026.

parquet-java — implemented, but not yet released. PR #3393, "PARQUET-2249: Add IEEE-754 total order and nan count for floating types", was opened by wgtmac (Gang Wu, co-author of the official blog post) on February 12, 2026, and merged on June 25, 2026. Its scope is 40 files, 4,857 lines added / 229 lines removed. It contains real classes like IEEE754FloatStatistics, IEEE754DoubleStatistics, and IEEE754Float16Statistics, plus E2E tests like TestIeee754TotalOrderE2E, and pom.xml's parquet.format.version is already 2.13.0. However, parquet-java's latest release is 1.17.1 (2026-05-12) — which predates the merge. In other words, it's not in the version you'd pull from Maven right now.

Arrow C++ — only the thrift was synced. A June 29, 2026 commit, "GH-50265: [C++][Parquet] Update parquet.thrift to sync with 2.13.0," brought cpp/src/parquet/parquet.thrift up to date with 2.13.0, and the fields also made it into generated files like parquet_types.h. But cpp/src/parquet/statistics.cc, which actually computes the statistics, has zero mentions of nan_count. The wire-format definition exists, but there's no logic that fills it in or reads it.

arrow-rs / DataFusion — still a draft. PR #9619, "Implement PARQUET-2249: Introduce IEEE 754 total order", was opened by etseidl on March 26, 2026, and is still a draft (17 files, 1,545 lines added / 252 lines removed, 106 comments, last updated 2026-07-14 — actively in progress). Earlier attempts — Xuanwo's #8158 was closed without merging on July 4, 2026, and the PoCs #7408 and #9669 also went unmerged. Searching the current main's hand-written thrift decoder (parquet/src/file/metadata/thrift/mod.rs) turns up zero hits for nan_count, and the ColumnOrder enum in basic.rs only knows TYPE_DEFINED_ORDER, UNDEFINED, and UNKNOWN. In other words, it's not in the Rust parquet crate's latest release (59.1.0, 2026-07-07).

To summarize — the spec shipped on June 13, and a month later, this feature is not in any major implementation's release. parquet-java is furthest along and likely to land in its next release, but that hasn't happened yet either.

The Honest Cost — This Fix Isn't Free

"An old bug got fixed" usually sounds like pure good news. This one isn't.

It's opt-in. The body of parquet-java PR #3393 states it explicitly — "The existing type-defined order remains the default." Looking at the code confirms it: PrimitiveType falls back to ColumnOrder.typeDefined() whenever columnOrder isn't specified. You have to turn it on per column, in the schema. Do nothing, and nothing changes.

Pruning becomes more conservative, not less. Quoting the same PR body directly, "Filtering around NaN values may be more conservative to avoid dropping data that can still match." This is the essence of the tradeoff — the "speed" you had until now stood on top of some wrong answers, and the moment you guarantee correctness, some queries get skipped less. If nan_count isn't 0, a predicate like col > 34 can no longer be pushed down from statistics alone. This is exactly what tustvold pointed out on the arrow-rs thread. No source publishes a measurement of how much slower this makes things, so I won't invent a number here.

The blast radius is much bigger than the spec's 98 lines. Looking at what's inside parquet-java's 40 changed files gives you a feel for it — beyond statistics and the column index, it touches DictionaryValuesWriter, ByteStreamSplitValuesWriter, BloomFilterImpl, DictionaryFilter, StatisticsFilter, and even LittleEndianDataOutputStream. According to the PR body, this is to preserve NaN's sign and payload bits exactly as the application supplied them. Put the other way around, this also means parquet-java had not been preserving NaN bit patterns until now.

Old-reader compatibility is different in theory than in practice. The spec states that "if a reader doesn't support a value of this union, it must ignore the min/max statistics for that column," and arrow-rs's current main does exactly that — it drops unrecognized union variants into ColumnOrder::UNKNOWN and ignores the statistics, per spec. But there's a concern emkornfield raised on arrow-rs issue #8156 on May 19, 2026. Old releases (in his words, "I think prior to arrow 57") use generated thrift code, and the parser might panic when reading a new file — shouldn't that be backported? etseidl's answer was that the problem lies not in the crate code but in the Rust thrift generator, and alamb replied, "let's backport it once we have actual evidence someone wants it." Either way — even though the spec was designed to be "safe" via optional fields and union extension, adding a field and adding a variant to an order union have different compatibility properties. The former, an old reader can simply ignore; the latter depends entirely on how the old reader handles it.

So What Should You Actually Do Right Now

Do this right now: check whether your float predicates are already wrong. This isn't a future problem, it's a present one. If you're issuing > / < predicates against a float column that can contain NaN, and the engine does statistics-based pruning, you have the same kind of wrong-answer risk as DataFusion #15812. The test is simple — can NaN actually get into that column? Sensor data, scientific pipelines that represent missing values as NaN, aggregations that a divide-by-zero can flow into — these are the candidates. If NaN can't get into a column in principle, this whole post is irrelevant to you.

When you shouldn't turn it on: right now, in most cases. The first reason is you can't — it isn't in any release yet. The second is interoperability. A file written with the new column order can only have its statistics used by a reader that understands that order, and there are barely any of those yet. What you'd gain by turning this on now is "metadata that will become valid in the future," and what you'd pay in the meantime is untested interaction with old readers. That's exactly why interop test files were added to the parquet-testing repository, and why parquet-java attached tests like TestInterOpReadFloatingPointNanCount — this is not a problem that gets solved by one implementation doing a good job.

When it becomes worth turning on. Once parquet-java's next release ships, arrow-rs #9619 sheds its draft status and merges, and Arrow C++ attaches real logic beyond the thrift definitions — that's when it makes sense to turn it on, column by column, starting with columns where NaN actually occurs. The design itself — per-column opt-in rather than a blanket switch — is telling you to use it exactly that way.

Until then, the safe default is to be conservative. tustvold's suggestion is the clearest one — given the state of the implementations, the right call is to not push down float predicates at all. Slow can be fixed; wrong is invisible.

Closing

To sum up: Parquet's min/max statistics contract broke down in the face of NaN — a value with no order — and -0.0/+0.0 — two bit patterns for the same value. Excluding NaN from statistics gave readers no way to know that had happened, and the result was wrong answers that failed to find rows that were actually there. ColumnIndex was outright stuck in an internal contradiction in the spec.

2.13.0's solution combines two pieces — making NaN's presence explicit with nan_count, pinning comparison down deterministically with IEEE_754_TOTAL_ORDER, and mandating nan_count whenever the new order is used, creating a clean signal: "new order = trustworthy statistics." The reason it took 3 years and 3 months and two abandoned PRs is that each individual proposal was only half the answer, and it took time to admit that.

If there's one thing to take from this post, it's this. The spec shipping and the problem being solved are two different things. Right now, the spec is complete, the reproducible wrong answer is still an open issue, and no major implementation's release contains the fix. Instead of reading the format spec and concluding "it's fixed," check whether it actually works in the version of the library you actually use. In this case, the answer is still "no."

References

현재 단락 (1/106)

A big part of why Parquet files are fast comes down to statistics. Every row group, every column chu...

작성 글자: 0자원문 글자: 19,638자작성 단락: 0/106