31 Pull requests opened by 23 people

• [SPARK-53209][YARN] Add ActiveProcessorCount JVM option to YARN executor and AM

  #51948 opened Aug 9, 2025

• [SPARK-53230][SQL] Assign a name to error class _LEGACY_ERROR_TEMP_1011

  #51955 opened Aug 10, 2025

• [SPARK-53207][SDP] Send Pipeline Event to Client Asynchronously

  #51956 opened Aug 10, 2025

• [WIP][TESTS] Upgrade pypy to 3.11

  #51966 opened Aug 11, 2025

• [SPARK-53254][PYTHON][TESTS] Skip hanging tests in PyPy daily workflow

  #51983 opened Aug 12, 2025

• [SPARK-53260]Reducing number of JDBC overhead connections creation

  #51991 opened Aug 12, 2025

• [SPARK-52844][PYTHON] Update pyyaml to 5.4

  #51993 opened Aug 12, 2025

• [SPARK-53262][SS] Support schema evolution for streaming dedupe operation

  #51996 opened Aug 12, 2025

• [SPARK-53012][PYHTON] Support Arrow Python UDTF in Spark Connect

  #51998 opened Aug 12, 2025

• [SPARK-52482][DOCS][FOLLOW-UP] Mention behavior changes in migration guide

  #51999 opened Aug 13, 2025

• temp:

  #52001 opened Aug 13, 2025

• [SPARK-53264][SQL][CATALYST]. Incorrect nullability when correlated scalar subquery ge…

  #52003 opened Aug 13, 2025

• [SPARK-52969][SQL] Support DSv2 OrcScan Dynamic Partition Pruning

  #52009 opened Aug 13, 2025

• [TEST-ONLY] Test PyPy 7.3.19 with Python 3.10

  #52014 opened Aug 13, 2025

• [SPARK-52677][SQL] Simplify DataTypeUtils.canWrite and TableOutputResolver

  #52016 opened Aug 13, 2025

• [SPARK-53265][PYTHON][DOCS] Add Arrow Python UDF Type Coercion Tables in Arrow Python UDF Docs

  #52025 opened Aug 14, 2025

• [SPARK-53273][CONNECT][SQL] Make RegisterUserDefinedFunction in SparkConnectPlanner side effect free

  #52026 opened Aug 14, 2025

• [SPARK-52336][CORE] Prepend Spark identifier to GCS user agent

  #52027 opened Aug 14, 2025

• [SPARK-53275][SQL] Handle stateful expressions when ordering in interpreted mode

  #52028 opened Aug 14, 2025

• [SPARK-53287][PS] Add ANSI Migration Guide

  #52034 opened Aug 15, 2025

• [SPARK-53288][SS] Fix assertion error with streaming global limit

  #52035 opened Aug 15, 2025

• [SPARK-53289] Invoke cancelJobGroup before start passing the ExecutionEnd event to prevent execution of canceled tasks.

  #52039 opened Aug 15, 2025

• [TEST-ONLY] A branch-3.5 PR to check the CI

  #52040 opened Aug 15, 2025

• [WIP][INFRA] Rebalance the test modules of pandas API on connect

  #52041 opened Aug 15, 2025

• [SPARK][CONNECT][CLIENT][PYTHON][SCALA] Remove RetriesExceeded error from Spark Connect Python and Scala clients

  #52042 opened Aug 15, 2025

• [SPARK-53291][SQL] Fix nullability for value column

  #52043 opened Aug 15, 2025

• [SPARK-53283][CONNECT] Make CheckpointCommand/RemoveCachedRemoteRelationCommand in SparkConnectPlanner side effect free

  #52044 opened Aug 15, 2025

• [SPARK-53295][PS] Turn on ANSI by default for Pandas API on Spark

  #52045 opened Aug 15, 2025

• [SPARK-53293] modify exprIdToOrdinal implementation for speedup on queries with wide tables

  #52046 opened Aug 15, 2025

• [SPARK-53294][SS] Enable StateDataSource with state checkpoint v2 (only batchId option)

  #52047 opened Aug 15, 2025

• [SPARK-52982][PYTHON] Disallow lateral join with Arrow Python UDTFs

  #52048 opened Aug 15, 2025

40 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [SPARK-53103][SS] Throw an error if state directory is not empty on batch 0

  #51817 commented on Aug 15, 2025 • 24 new comments

• [SPARK-52991][SQL] Implement MERGE INTO with SCHEMA EVOLUTION for V2 Data Source

  #51698 commented on Aug 15, 2025 • 22 new comments

• [SPARK-52582][SQL] Improve the memory usage of XML parser

  #51287 commented on Aug 15, 2025 • 20 new comments

• [SPARK-52777][SQL] Enable shuffle cleanup mode configuration in Spark SQL

  #51458 commented on Aug 15, 2025 • 7 new comments

• [SPARK-53108][SQL] Implement the time_diff function in Scala

  #51826 commented on Aug 12, 2025 • 4 new comments

• [SPARK-51920][SS][PYTHON] Fix composite/nested type in value state for python TWS

  #51621 commented on Aug 15, 2025 • 4 new comments

• [SPARK-53030][PYTHON] Support Arrow writer for streaming Python data sources

  #51757 commented on Aug 12, 2025 • 3 new comments

• [SPARK-53198][CORE] Support terminating driver JVM after SparkContext is stopped

  #51929 commented on Aug 15, 2025 • 2 new comments

• [SPARK-52621][SQL] Cast TIME to/from VARIANT

  #51553 commented on Aug 11, 2025 • 2 new comments

• [SPARK-53125][TEST] RemoteSparkSession prints whole `spark-submit` command

  #51846 commented on Aug 13, 2025 • 2 new comments

• [SPARK-53127][SQL] Enable LIMIT ALL to override recursion row limit

  #51847 commented on Aug 14, 2025 • 2 new comments

• [SPARK-52844][PYTHON][TESTS] Update black to 24.3.0

  #51687 commented on Aug 13, 2025 • 1 new comment

• [SPARK-53174][CORE] Add TMPDIR environment variable with the value of java.io.tmpdir

  #51902 commented on Aug 13, 2025 • 1 new comment

• [SPARK-53193][DOCS] Add advanced JVM optimization parameters to tuning guide

  #51920 commented on Aug 12, 2025 • 1 new comment

• [SPARK-53212][PYTHON] improve error handling for scalar Pandas UDFs

  #51937 commented on Aug 12, 2025 • 1 new comment

• [SPARK-53182][PYTHON][DOCS] Fix broken and missing links in PySpark DataFrames user guide

  #51851 commented on Aug 13, 2025 • 0 new comments

• [SPARK-53142][SQL] Support dynamic expression addition in SemanticComparator

  #51871 commented on Aug 12, 2025 • 0 new comments

• [SPARK-53143][SQL] Fix self join in DataFrame API - Join is not the only expected output from analyzer

  #51873 commented on Aug 12, 2025 • 0 new comments

• [SPARK-53105][Structured Streaming] Fix tests for checkpoint v2 in RocksDBSuite

  #51834 commented on Aug 11, 2025 • 0 new comments

• [SPARK-53144][CONNECT][SQL] Make CreateViewCommand in SparkConnectPlanner side effect free

  #51874 commented on Aug 15, 2025 • 0 new comments

• [SPARK-53156][CORE] Track Driver Memory Metrics when the Application ends

  #51882 commented on Aug 14, 2025 • 0 new comments

• [SPARK-53148][CONNECT][SQL] Make SqlCommand in SparkConnectPlanner side effect free

  #51903 commented on Aug 15, 2025 • 0 new comments

• [IN PROGRESS][K8S] Introduce pending pod limit per ResourceProfile

  #51913 commented on Aug 15, 2025 • 0 new comments

• [SPARK-35564][SQL] Support subexpression elimination for conditionally evaluated expressions

  #32987 commented on Aug 15, 2025 • 0 new comments

• [SPARK-37019][SQL] Add codegen support to array higher-order functions

  #34558 commented on Aug 13, 2025 • 0 new comments

• [SPARK-44639][SS][YARN] Use Java tmp dir for local RocksDB state storage on Yarn

  #42301 commented on Aug 15, 2025 • 0 new comments

• [SPARK-22876][YARN] Respect YARN AM failure validity interval

  #42570 commented on Aug 13, 2025 • 0 new comments

• [SPARK-49547][SQL][PYTHON] Add iterator of `RecordBatch` API to `applyInArrow`

  #49005 commented on Aug 15, 2025 • 0 new comments

• [SPARK-51710][PYTHON] Passing an empty array to PySpark Dataframe.dropDuplicates should behave the same as passing no arguments

  #50714 commented on Aug 16, 2025 • 0 new comments

• [SPARK-51972][SS] State Store file integrity verification using checksum

  #50773 commented on Aug 16, 2025 • 0 new comments

• [SPARK-52029][SQL] Support external field-id to column-name mapping for Parquet

  #50819 commented on Aug 16, 2025 • 0 new comments

• [SPARK-52031][SQL]Part 3. Subquery generator for testing queries with nested correlations

  #50821 commented on Aug 16, 2025 • 0 new comments

• [SPARK-50603][SQL] Respect user-provided basePath for streaming file source reads without glob

  #51267 commented on Aug 15, 2025 • 0 new comments

• [SPARK-52407][SQL] Add support for Theta Sketch

  #51298 commented on Aug 10, 2025 • 0 new comments

• [SPARK-33737][K8S] Support getting pod state using Informers + Listers

  #51396 commented on Aug 15, 2025 • 0 new comments

• [SPARK-52769][SQL] InjectRuntimeFilter should take into account join type and hints

  #51453 commented on Aug 11, 2025 • 0 new comments

• [WIP][SPARK-51169] Set up a daily job for Python 3.14

  #51532 commented on Aug 11, 2025 • 0 new comments

• [SPARK-52844][PYTHON] Update protobuf to 5.29.5

  #51747 commented on Aug 15, 2025 • 0 new comments

• [SPARK-42360][SQL] Rule to convert Left Outer Join with suitable filter to Left Anti Join

  #51762 commented on Aug 11, 2025 • 0 new comments

• [SPARK-52844][PYTHON] Update mlflow to 3.1.0

  #51774 commented on Aug 11, 2025 • 0 new comments