Apache Spark 4.3.0 Adds Stateful Queries to Real-Time Mode and SCD Type 2 to Auto CDC, and Starts a Quarterly Feature-Release Cadence

Overview

The Apache Spark project announced Spark 4.3.0 on October 30, 2026, according to its news page. Per the release notes, it is the fourth release of the Spark 4.x line and addressed over 1,100 Jira tickets with contributions from more than 210 individuals. The notes say the release brings stateful processing to Structured Streaming Real-Time Mode, adds history-preserving SCD Type 2 Auto CDC to Spark Declarative Pipelines, and makes Spark faster to iterate with on small data (the notes’ wording).

What Is New

Stateful Real-Time Mode

According to the release notes, streaming aggregation, dropDuplicates and transformWithState now run in Real-Time Mode with end-to-end latency on the order of 100 milliseconds. The notes list support for Scala/Java transformWithState in Real-Time Mode and the removal of the @Experimental annotation from Real-Time Mode.

A Databricks engineering blog post dated November 5, 2026, whose authors include Jungtaek Lim, Wenchen Fan and Reynold Xin, gives vendor-reported figures from benchmarks it attributes to SPARK-58884. It reports P99 end-to-end latency of 136 ms for dropDuplicates, 139 ms for window aggregation and 141 ms for transformWithState. The post states that these are “measurements of the three tested workloads, not a latency guarantee for every pipeline.”

SCD Type 2 in Auto CDC

The release notes say Auto CDC now supports SCD Type 2 end to end, with each key’s history kept as versioned rows bounded by __START_AT and __END_AT. They add that the target table must be in a catalog supporting DSv2 row-level operations, with Apache Iceberg given as an example. The capability is available through Python, SQL and Spark Connect.

Small-Data Execution

The release notes describe opt-in shuffle-free single-task execution for eligible small, single-partition query fragments, controlled by spark.sql.optimizer.singleTaskExecution.enabled. The Databricks post says that with the default eligibility limits, qualifying file scans read one nonempty small file that fits within a single partition, while qualifying local in-memory relations contain up to 1,000 rows. It also says both this optimization and the Arrow-native Dataset cache are opt-in.

Other Changes

The release notes list SQL ASOF JOIN ... MATCH_CONDITION syntax, enabled with spark.sql.join.asofJoin.enabled and off by default. They also list a nanosecond-precision timestamp preview behind spark.sql.timestampNanosTypes.enabled, also off by default, and a BIN BY relation operator behind spark.sql.binByRelationOperator.enabled, off by default. Two settings are now enabled by default: spark.rdd.compress and spark.ui.contentSecurityPolicy.enabled. The notes also list the removal of Python 3.10 support.

Release Cadence

The release notes say 4.3.0 marks the start of a quarterly feature-release cadence. The Databricks post describes this as moving from one release every six months to one every three months.

What We Don’t Know

• The Databricks latency figures come from one post by Databricks authors; the benchmark hardware and configuration are not described in the sources cited here. This article cites no independent outlet.
• The Databricks post tells readers to consult the Databricks Runtime release notes for Databricks availability, and this article does not cover that.
• The release notes give the Real-Time Mode latency as “on the order of 100 milliseconds” and do not state percentile figures.