[GLUTEN-12555][CORE] Propagate child outputPartitioning and outputOrdering in ColumnarToColumnarExec - #12556
Merged
jackylee-ch merged 1 commit intoJul 20, 2026
Conversation
|
Run Gluten Clickhouse CI on x86 |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates ColumnarToColumnarExec (the core base class for columnar “transparent” wrapper operators) to correctly propagate physical properties from its child plan node, so Spark’s planner/AQE can continue to recognize existing distribution and ordering through these wrappers.
Changes:
- Propagate
outputPartitioningfromchild.outputPartitioninginColumnarToColumnarExec. - Propagate
outputOrderingfromchild.outputOrderinginColumnarToColumnarExec. - Add the required Spark SQL imports (
Partitioning,SortOrder) to support these overrides.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Run Gluten Clickhouse CI on x86 |
Comment on lines
+99
to
+101
| override def outputPartitioning: Partitioning = child.outputPartitioning | ||
|
|
||
| override def outputOrdering: Seq[SortOrder] = child.outputOrdering |
wForget
approved these changes
Jul 20, 2026
Contributor
|
@wangyum The CI problem has been fixed in main branch. Please update the branch and rerun CI |
…ering in ColumnarToColumnarExec ColumnarToColumnarExec is the abstract base for operators that only reshape, resize, or convert columnar batches without altering row distribution or order (e.g. VeloxResizeBatchesExec, ArrowColumnarToVeloxColumnarExec, OffloadArrowDataExec, LoadArrowDataExec, GpuResizeBufferColumnarBatchExec). It previously inherited the default outputPartitioning (UnknownPartitioning) and outputOrdering (Nil) from SparkPlan via UnaryExecNode, misrepresenting the physical layout. As a result: - storage-partitioned joins / v2 bucketing were not recognized when a batch-resize or format-convert node sat between a BatchScanExec and the join, causing a redundant shuffle; - SortMergeJoin lost the child's sort order across the wrapper and inserted an extra Sort; - exchange reuse / AQE coalescing / ordering-based optimizations made suboptimal decisions. Propagate the child's partitioning and ordering so the wrapper is transparent to the physical planner, mirroring how vanilla Spark's ColumnarToRowExec / RowToColumnarExec and Gluten's ColumnarInputAdapter handle transparency. Closes apache#12555. Generated-by: opencode ebay-glm-5-2-fp8-chat
wangyum
force-pushed
the
fix/columnar-to-columnar-output-properties
branch
from
July 20, 2026 07:59
f8c8f5a to
9f31bb9
Compare
|
Run Gluten Clickhouse CI on x86 |
jackylee-ch
approved these changes
Jul 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are included in this PR?
ColumnarToColumnarExecnow propagates the child'soutputPartitioningandoutputOrderinginstead of inheriting the defaults (UnknownPartitioning/Nil) fromSparkPlanviaUnaryExecNode.plus the required
Partitioning/SortOrderimports.Why are the changes needed?
ColumnarToColumnarExec(ingluten-core) is the abstract base for operators that only reshape / resize / convert columnar batches without altering row distribution or order. Concrete subclasses includeVeloxResizeBatchesExec,ArrowColumnarToVeloxColumnarExec,OffloadArrowDataExec,LoadArrowDataExec, andGpuResizeBufferColumnarBatchExec.Previously these wrappers reported
UnknownPartitioning(numColumns)andNilordering, which misrepresents the physical layout. Consequences:BatchScanExec(reportingKeyGroupedPartitioning) and the join, so the planner inserts a redundant shuffle, defeating the scan-reported partitioning.Sort.This mirrors how vanilla Spark's
ColumnarToRowExec/RowToColumnarExecand Gluten's ownColumnarInputAdaptertreat partitioning/ordering transparency. It applies to all supported Spark versions and all backends.Closes #12555.
Does this PR introduce any user-facing change?
No API change. Plans that previously inserted a redundant shuffle / sort around a columnar reshape can become more efficient.
How was this patch tested?
Existing unit tests (no behavioral change to data, only to plan shape); relied on the fact that the wrappers are data-transparent. Follow-up: a plan-shape assertion test for SPJ through a resize node can be added if reviewers want explicit coverage.
Was this patch authored or co-authored using generative AI tooling?
Yes.
Generated-by: opencode GLM 5.2