Skip to content
Published on

PySpark 4.2 Puts Python UDFs on Arrow by Default — the Story of 132 out of 225 Type-Coercion Cells Changing

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Introduction — A Default Flipped Quietly Two Days Ago

Apache Spark 4.2.0 was released on July 14, 2026. It's the third release in the 4.x line and, by the release notes' own count, packs in more than 1,700 Jira tickets and 250-plus contributors. The headline items are flashy — GEOMETRY/GEOGRAPHY geospatial types, CDC via the SQL CHANGES clause, NEAREST BY top-K joins, Data Source V2 transactions.

But for anyone running batch jobs on PySpark, the line that will actually cost you sleep tonight is a much less conspicuous entry on that list.

[SPARK-54555] Enable Arrow-optimized Python UDFs and Arrow-based PySpark IPC by default

Just one default flipped. No new API, no new syntax. And yet this single line can change the output data of a Python UDF you've been running for three years. No error, no warning.

This post traces exactly what that default changes, using only material actually checked into the Spark repository. The short version: the interesting part isn't speed — it's type coercion.

What Changed — Three Settings

Let's start by fact-checking. SPARK-54555 landed as PR #53264, and commit ea0a35e065d3 merged on December 2, 2025. The Jira ticket's fixVersion is 4.2.0, and this commit is present in the v4.2.0 tag but absent from v4.1.0.

What actually changed is a set of defaults in SQLConf.scala. Diffing the same file between branch-4.1 and branch-4.2 gives you this:

Setting                                          4.1     4.2
------------------------------------------------------------
spark.sql.execution.pythonUDF.arrow.enabled     false -> true
spark.sql.execution.pythonUDTF.arrow.enabled    false -> true
spark.sql.execution.arrow.enabled               false -> true

The third entry is a little confusing: spark.sql.execution.arrow.pyspark.enabled, the setting the docs and migration guide actually talk about, has no default of its own — it points to spark.sql.execution.arrow.enabled via fallbackConf. So the one that actually flipped in source is the latter. The effect is the same either way — DataFrame.toPandas and SparkSession.createDataFrame fed pandas/NumPy input now use Arrow columnar transfer by default.

In other words, this release flips Arrow on across three fronts in Python at once: plain Python UDFs, Python UDTFs, and the columnar data exchange (IPC) between the JVM and Python.

How Much Faster — Nobody Has Published a Number

I'll be honest about this part.

PR #53264's "Why are the changes needed?" section says Arrow's columnar IPC "significantly improves" throughput between the JVM and Python and cuts serialization/deserialization overhead. But the PR itself contains zero benchmark numbers. Databricks' Spark 4.2 introduction blog post says only that "existing UDFs can use the faster columnar path without any code changes," without offering a figure either.

The Spark repository does have a checked-in benchmark result at sql/core/benchmarks/UDFBenchmark-results.txt, but open it and it's a whole-stage-codegen on/off comparison for Scala UDFs — it doesn't measure Arrow against pickle Python UDFs at all. A benchmark harness like python/pyspark/sql/tests/pandas/bench_arrow_columnar_udf.py exists in the repo, but no results file is checked in.

So this post won't claim "N times faster." No such number has ever been published alongside this release. The direction is plausible enough — the case for why passing columnar batches beats row-by-row pickle serialization is the same logic covered in Catalyst and Tungsten, and it's the same family of story as why Polars beat pandas. But how many percentage points faster your own UDF gets is something you have to measure yourself. That's the entirety of what's confirmable from this release.

And either way, the part that actually matters in this default change isn't speed.

The Real Story Is Type Coercion

When you write a Python UDF, you declare a return type.

@udf("int")
def f(x):
    return some_value

The question is what happens when some_value turns out to be a different type than the declared int. Pickle and Arrow follow different rules at exactly this point. This is a known fact — it was even sitting in a comment in the PySpark source, above the _create_py_udf function in python/pyspark/sql/udf.py.

# Arrow and Pickle have different type coercion rules, so a UDF might have a
# different result with/without Arrow optimization. That's the main reason the
# Arrow optimization for Python UDFs is disabled by default.

The funny part is that this comment is still sitting there, unchanged, on branch-4.2. It says "that's why it's disabled by default," except on 4.2 it's actually enabled — the comment simply didn't keep up with the release. It also points to a coercion table at python/pyspark/sql/tests/udf_type_tests, a path that doesn't exist on branch-4.2.

The table moved. Its real location is python/pyspark/sql/tests/coercion/, which holds golden files that CI actually validates.

golden_python_udf_return_type_coercion_vanilla.csv           <- pickle (4.1 default)
golden_python_udf_return_type_coercion_with_arrow.csv        <- Arrow  (4.2 default)
golden_python_udf_return_type_coercion_with_arrow_and_pandas.csv

The third file is the pre-4.1 Arrow behavior (back when there was an extra pandas conversion step in the middle). Since spark.sql.legacy.execution.pythonUDF.pandas.conversion.enabled defaults to false on 4.2, the table that's actually live on 4.2 is the second file. For what it's worth, the official PySpark docs still point to the coercion differences via PR #41706, a "DO NOT MERGE" documentation-only PR opened back in 2023 — but the golden files in the repo are more current and are what CI actually enforces.

So I diffed those two golden files directly.

132 of 225 Cells

The table covers 15 declared SQL types × 15 Python return values = 225 cells. Comparing the two files cell by cell, 132 cells differ. That's more than half. X denotes an error.

Broken down by category, it looks like this:

Category                                         Cells
---------------------------------------------------------
1. None  -> error (a silent NULL now fails)          99
2. None  -> value (a silent NULL now has data)       18
3. value -> error (what worked now fails)             2
4. error -> value (what failed now returns a value)   7
5. value -> different value (output string changes)   6
---------------------------------------------------------
Total                                                132

Category 1 (99 cells) dominates overwhelmingly. Pickle Python UDFs were designed to "silently produce None" whenever the return value didn't match the declared type. That's gone. Now most mismatches raise an exception. Directionally this is a clear improvement — swallowing a type mistake into NULL is one of the nastiest classes of bugs a data pipeline can have. But if your previously working job suddenly dies on 4.2, odds are overwhelming it's one of these 99 cells. And dying is the correct behavior. The problem is only that you find out about it from a single release-notes line.

Let's pull out a few actual cells. The values are exactly as written in the golden files.

Declared type   Python value returned      pickle(4.1)                        Arrow(4.2)
----------------------------------------------------------------------------------------------
int             1.0        (float)       None                               1
int             1970-01-01 (date)        None                               0
bigint          1970-01-01 00:00:00      None                               0
boolean         1          (int)         None                               True
double          1          (Decimal)     None                               1.0
int             'a'        (str)         None                               X  (error)
tinyint         True       (bool)        None                               X  (error)
binary          'a'        (str)         b'a'                               X  (error)
struct<_1:int>  [1]        (list)        Row(_1=1)                          X  (error)
date            1          (int)         X  (error)                          datetime.date(1970, 1, 2)
string          1970-01-01 (date)        'java.util.GregorianCalendar[...'  '1970-01-01'
string          (1,)       (tuple)       '[Ljava.lang.Object;@<hash>'       '(1,)'
string          bytearray(b'ABC')        '[B@<hash>'                        "bytearray(b'ABC')"

The last three rows are pure bug fixes. On the pickle path, declaring date as string meant java.util.GregorianCalendar[time=?,... — the toString of a JVM-internal object flowed straight into your data. The bytearray case, [B@<hash>, is a Java byte array's identity hash. If something like that had made it into a production table, 4.2 fixes it into '1970-01-01'. Welcome, but if there's a downstream consumer reading that column, its value changes too.

Category 4 (7 cells) is also worth a look. Declaring date and returning the integer 1 was an error under pickle, but under Arrow it becomes 1970-01-02 — epoch plus one day. It's logically consistent, but it also means a mistake that used to be filtered out as an error now sails through as a plausible-looking date.

The Most Dangerous 18 Cells — Nothing Fails

The 99 cells fail loudly. Loud is manageable — you deploy, it blows up, you roll back or fix it.

The problem is Category 2, the 18 cells. Here, nothing happens at all. No exception, no warning, no log line. A cell that was NULL yesterday just has a value in it today.

Here's the full list.

Declared    Return value             pickle -> Arrow
--------------------------------------------------
boolean    1        (int)          None -> True
boolean    1.0      (float)        None -> True
tinyint    1.0      (float)        None -> 1
tinyint    1        (Decimal)      None -> 1
smallint   1.0      (float)        None -> 1
smallint   1        (Decimal)      None -> 1
int        1970-01-01 (date)       None -> 0
int        1.0      (float)        None -> 1
int        1        (Decimal)      None -> 1
bigint     1970-01-01 00:00:00     None -> 0
bigint     1.0      (float)        None -> 1
bigint     1        (Decimal)      None -> 1
float      True     (bool)         None -> 1.0
float      1        (int)          None -> 1.0
float      1        (Decimal)      None -> 1.0
double     True     (bool)         None -> 1.0
double     1        (int)          None -> 1.0
double     1        (Decimal)      None -> 1.0

A pattern jumps out of this list. Most of it is "declared one numeric type, returned a different numeric type." And that is the single easiest mistake to make in practice. Slap @udf("int") on a function that does division inside, and Python hands back a float. Declaring @udf("double") and returning a Decimal is just as common.

Through 4.1, every one of these UDFs was quietly emitting a NULL column. If nobody noticed, that NULL is already baked into everything downstream — COUNT(*), WHERE x IS NULL filters, AVG aggregates, join keys, dashboard numbers, all of it. The moment you move to 4.2, those cells fill in with real values, and all of that shifts quietly along with them.

This isn't a regression. 4.2 is the one that's correct — declare int, return 1.0, and the right output is 1, not NULL. But "getting fixed" and "output changing" are the same event operationally, and if you don't treat it as the latter, you get hurt — especially when a backfill produces different results for old batches versus new ones.

Your UDF Might Not Be on Arrow — the Import-Order Trap

Here's where it gets genuinely perverse: this default doesn't apply uniformly.

Look at _create_py_udf in python/pyspark/sql/udf.py and this is the logic:

is_arrow_enabled = False
if useArrow is None:
    from pyspark.sql import SparkSession

    session = SparkSession._instantiatedSession
    is_arrow_enabled = (
        False
        if session is None
        else session.conf.get("spark.sql.execution.pythonUDF.arrow.enabled") == "true"
    )
else:
    is_arrow_enabled = useArrow

If SparkSession._instantiatedSession is None, it's False without even reading the config. And this decision fires once — not when the UDF is called, but when the UDF object is created — and it hardens into the eval type from then on.

What that means: a UDF written at the top of a module like this,

# udfs.py — the decorator is evaluated at import time
from pyspark.sql.functions import udf

@udf("int")
def parse_score(s):
    return float(s) / 2

locks in as a pickle UDF if it's imported before SparkSession.builder.getOrCreate(), because no session exists yet — even on 4.2. Import that exact same code after the session is created, and it becomes an Arrow UDF.

Which means within a single job, different UDFs can end up under different coercion rules, purely based on import order. The example above declares int and returns a float, which lands exactly on one of those 18 cells (int + 1.0(float)) — depending on import order, you get NULL or you get a value.

This logic itself isn't new. The exact same code, letter for letter, is in udf.py on branch-4.1 too. It's just that on 4.1 the config already defaulted to false, so the session is None branch produced the same answer regardless. Now that the default has flipped on 4.2, this branch has become a path that silently overrides the new default.

There's a similarly quiet escape hatch. In the same function, if PyArrow or pandas isn't available:

except ImportError:
    is_arrow_enabled = False
    warnings.warn(
        "Arrow optimization failed to enable because PyArrow or Pandas is not installed. "
        "Falling back to a non-Arrow-optimized UDF.",
        RuntimeWarning,
    )

It throws one RuntimeWarning and falls back to pickle. If your driver image has PyArrow and some other environment doesn't, the exact same code runs under different coercion rules depending on the environment. For what it's worth, 4.2 also raised the minimum PyArrow version from 15.0.0 to 18.0.0; the pandas minimum stays at 2.2.0.

One more thing worth flagging: this fallback doesn't exist for UDFs. There's a setting, spark.sql.execution.arrow.pyspark.fallback.enabled (default true), that automatically falls back to a non-optimized implementation "if Arrow errors out" — but read the config docs carefully and this only applies to the optimization that spark.sql.execution.arrow.pyspark.enabled turns on, i.e., the toPandas and createDataFrame paths. A coercion error that happens during UDF execution does not fall back — it just kills the job.

What Comes Along With It

The 4.2 migration guide lists a few more Python-side changes. They're independent of the Arrow default, but they ride along in the same upgrade.

  • Nullable integer columns in pandas UDFs. A nullable integer column with nulls mixed into the batch is now passed as pandas' nullable integer extension dtype (Int8/Int16/Int32/Int64) rather than float64. UDF code that assumed float64 input needs to be revisited.
  • PyPy support dropped. As of 4.2, PyPy is no longer officially supported. The guidance is to use CPython.
  • createDataFrame from a NumPy ndarray. This now requires PyArrow instead of pandas, and converts straight to Arrow without going through pandas. If you relied on NumPy-dtype-based schema inference with Arrow turned off, the guide explicitly warns you to double-check the inferred schema.
  • Python Data Source schema mismatches. If the column type of the returned Arrow data differs from the declared schema, it now fails with DATA_SOURCE_RETURN_SCHEMA_MISMATCH. Previously, only a mismatch in column count or names triggered this error.

Arrow batch size is controlled by spark.sql.execution.arrow.maxRecordsPerBatch, which defaults to 10000. If your UDF is heavy on per-row memory, this value now affects the executor's memory profile — what used to stream row by row now arrives bundled into batches.

So How Should You Upgrade

To sum up, this is the sane path.

1. Upgrade with it off first. Moving to 4.2 and switching to Arrow UDFs are two separate changes. Don't do them at once.

spark.sql.execution.pythonUDF.arrow.enabled   false
spark.sql.execution.pythonUDTF.arrow.enabled  false
spark.sql.execution.arrow.pyspark.enabled     false

This lets you get everything else 4.2 brings — geospatial types, CDC, DSv2 transactions, Java 25 — with the same Python semantics as 4.1.

2. Then build a UDF inventory. The only thing actually at risk is "a UDF whose declared type differs from its return type." If the two match, none of the 132 cells concern you — the golden table's diagonal (declared type = return type) is identical between the two files. So there's exactly one question: does my UDF return what it declared?

You can't answer that by reading the code. One division inside a numeric UDF, one Decimal, and that's the end of it. Running the same job under both settings on a small sample and diffing the output is the only reliable way to check.

3. Watch the NULL count when you compare. The 18-cell category is silent, so you will never catch it by watching error logs. If a column's NULL count drops, that's exactly those 18 cells. The drop might be welcome — but that's your call to make, not the release's.

4. Pin down import order. Because of the trap above, "I turned the config on" isn't enough to guarantee Arrow is actually on. To be certain, declare it explicitly with udf(..., useArrow=True). When useArrow is set explicitly, it overrides both the session state and the config and just uses that value. Conversely, for a UDF you want to keep pinned to pickle, nail it down with useArrow=False. Leaving it ambiguous to the default is the worst option of all.

When to Keep This Default Off

Honestly, there are cases where staying off indefinitely is the right call.

  • You have few UDFs and all of them are light. Arrow's payoff shows up when a lot of data moves between the JVM and Python. If your job processes a few tens of thousands of rows a day, the cost of re-validating outweighs the gain.
  • You have a lot of UDFs where declared type and return type don't match, and you don't have the bandwidth to audit all of them right now. The 132 cells are your problem. Turn it off, schedule the work, do it later.
  • Your output is bound by a contract. If downstream expects not just the schema but the actual distribution of values (regulatory reports, financial close batches, for example), a quiet drop in NULLs is itself an incident. Get agreement first, then turn it on.

On the flip side, here's when it's reasonable to turn it on right away:

  • Your UDFs are heavy and data volume is high. The columnar path is more likely to be worth it — though as noted, you still have to measure the number yourself.
  • You can guarantee with tests that declared type matches return type. Then the 132 cells are irrelevant to you, and it's simply free.
  • And don't forget the case where the best answer is not to use a UDF at all. If you've been writing logic in a Python UDF that could be expressed with built-in functions or SQL, dropping the UDF is a far bigger win than Arrow speeding it up. 4.2 added more built-ins as well, including time_bucket.

Closing

The Python story in Spark 4.2.0 doesn't come through in a single release-notes line. "Arrow-optimized Python UDFs enabled by default" reads like a performance item, but open the golden tables actually checked into the repo and the semantics of 132 out of 225 cells change. Of those, 99 cells turn a once-silent NULL into an error, and 18 cells turn NULL into a value with no signal at all.

The Apache migration guide writes this out honestly — down to the exact config names and how to revert them. The vendor blog, by contrast, only describes it as "a faster columnar path without rewriting code" and never brings up coercion at all. Neither is lying, exactly, but the one you need to read when planning an upgrade is the former.

And I think the judgment underneath this change is correct on its own terms. "Silently return NULL when the type doesn't match" was never a defensible default to begin with. But there are pipelines that have been running on top of that wrong default for about three years, and for them this fix arrives not as a fix, but as an output change. Upgrade with it off, measure, then turn it on. The order is everything.

References

Discuss on Twitter • View on GitHub