fix: do not remove DISTINCT when a unique key was downgraded by a join#23548
Open
simonvandel wants to merge 2 commits into
Open
fix: do not remove DISTINCT when a unique key was downgraded by a join#23548simonvandel wants to merge 2 commits into
simonvandel wants to merge 2 commits into
Conversation
…a join `ReplaceDistinctWithAggregate` removes `Distinct::All` when a functional dependence covers all input fields, but it ignores the dependency mode: after a join a formerly-unique key is downgraded to `Dependency::Multi`, so the input may still contain duplicates. This SLT test records the current wrong behavior: `SELECT DISTINCT u.id` over a LEFT JOIN returns one row per matching order instead of one row per user.
`ReplaceDistinctWithAggregate` removes `Distinct::All` when a functional
dependence's determinant columns cover all input fields, treating the
input as already deduplicated. However it ignored the dependency mode:
after a join, `FunctionalDependencies::join` downgrades dependencies to
`Dependency::Multi` because a formerly-unique key may now occur in many
rows. Removing the DISTINCT in that case returns duplicate rows.
For example, with `users.id` as a primary key and several orders per
user:
SELECT DISTINCT users.id FROM users LEFT JOIN orders ON users.id = orders.user_id
returned `id` values repeated once per matching order.
Only remove the DISTINCT when the dependency mode is `Single`, i.e. the
determinant key is guaranteed to occur at most once. The sqllogictest
added in the previous commit now shows the correct deduplicated result
and an Aggregate in the plan.
github-actions
Bot
added
optimizer
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
The
ReplaceDistinctWithAggregateoptimization pass was incorrectly removing deduplication.What changes are included in this PR?
First commit reproduces the bug in an SLT. I opted for SLT instead of a test inside
datafusion/optimizer/src/replace_distinct_aggregate.rssince SLT seems easier to maintain. But let me know if you prefer the test elsewhere.Second commit fixes the bug. Also includes a drive-by documentation change for
Dependencyto surface the field docs.Are these changes tested?
Yes
Are there any user-facing changes?
Yes, query now deduplicates correctly