(benchmax) perf: bound multi-column join cardinality by the composite key - #24437
Closed
Dandandan wants to merge 1 commit into
Closed
(benchmax) perf: bound multi-column join cardinality by the composite key#24437Dandandan wants to merge 1 commit into
Dandandan wants to merge 1 commit into
Conversation
For a join on more than one column the join key is the tuple of all key columns, but the cardinality estimate used the distinct count of the single most selective column. A composite key is more selective than any of its parts, so the estimate came out too large. Estimate the key tuple's distinct count instead: it can be no larger than the product of the per-column counts, and no larger than the number of rows. Single-column joins are unaffected, and no TPC-H plan changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #24437 +/- ##
==========================================
- Coverage 81.23% 81.23% -0.01%
==========================================
Files 1112 1112
Lines 390635 390674 +39
Branches 390635 390674 +39
==========================================
+ Hits 317350 317378 +28
- Misses 54650 54657 +7
- Partials 18635 18639 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
Author
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/composite-key-join-cardinality (891804d) to bb038a6 (merge-base) diff Run configurationrun benchmark tpcdsResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/composite-key-join-cardinality (891804d) to bb038a6 (merge-base) diff Run configurationrun benchmark clickbench_partitionedResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/composite-key-join-cardinality (891804d) to bb038a6 (merge-base) diff Run configurationrun benchmark tpchResults will be posted here when complete File an issue against this benchmark runner |
Dandandan
changed the title
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/composite-key-join-cardinality (891804d) to bb038a6 (merge-base) diff Run configurationrun benchmark tpchCPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/composite-key-join-cardinality (891804d) to bb038a6 (merge-base) diff Run configurationrun benchmark tpcdsCPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/composite-key-join-cardinality (891804d) to bb038a6 (merge-base) diff Run configurationrun benchmark clickbench_partitionedCPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
For a join on more than one column the join key is the tuple of all key columns, but
estimate_inner_join_cardinalityused the distinct count of the single most selective column. A composite key is more selective than any of its parts, so the estimate comes out too large.TPC-H q9 joins
partsuppon(ps_partkey, ps_suppkey)— its primary key, so 800k distinct pairs — while the largest per-column count is only 200k. At SF1 that join is estimated at 4,800,972 rows; the true answer is 319,404. With this change it estimates 800,000.What changes are included in this PR?
Estimate the distinct count of the key tuple instead: it can be no larger than the product of the per-column counts, and no larger than the number of rows, so take the smaller of the two.
Single-column joins are unaffected — the product of one count is that count, and
max_distinct_countalready caps it at the row count.Are these changes tested?
Yes.
test_inner_join_cardinality_multiple_columnis updated for the new bound, andtest_inner_join_cardinality_multiple_column_below_row_countis added to cover the case where the product rather than the row count is the binding limit.I also compared the build and probe row counts of every hash join across TPC-H SF1 (62 joins, from
dfbench tpch --debugmetrics) before and after: the results are identical, so this changes no TPC-H plan and carries no measured runtime effect either way. It is an estimate accuracy improvement, not a performance fix.Are there any user-facing changes?
No API changes. Multi-column joins may pick a different build side, though none do in TPC-H.