Skip to content

fix: align dictionary coercion across typed signatures#23549

Open
lyne7-sc wants to merge 3 commits into
apache:mainfrom
lyne7-sc:fix/dictionary-coercion-semantics
Open

fix: align dictionary coercion across typed signatures#23549
lyne7-sc wants to merge 3 commits into
apache:mainfrom
lyne7-sc:fix/dictionary-coercion-semantics

Conversation

@lyne7-sc

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

#22905 introduced explicit dictionary encoding preservation for coercible function signatures, but dictionary inputs were still handled differently across TypeSignatureClass variants:

Signature category Before After
Native(...) Materialized by default; preserved when explicitly requested Same default/opt-in contract
Typed non-Native (e.g. Integer, Numeric, Binary) Retained the physical dictionary type by default Materialized by default; preserved when explicitly requested
Any Passed through the original physical input type Unchanged

This PR makes the encoding preservation contract consistent across all typed signature classes: coercion operates on the dictionary value type, and the dictionary encoding is restored only when EncodingPreservation::dictionary() is enabled.

An audit of the affected built-ins found two functions, Spark hex and bitmap_count, that intentionally handle dictionary inputs; both now opt in explicitly. Other affected functions generally expect materialized value arrays, so the new default also avoids cases where signature matching accepted a dictionary but the function implementation rejected it at execution time. Functions that continue to materialize dictionary inputs do not gain dictionary-aware execution efficiency yet, but they can opt in later if they add support for encoded inputs.

What changes are included in this PR?

  • Align dictionary coercion across typed signature classes and preserve dictionary encoding when explicitly requested.
  • Explicitly enable dictionary preservation for Spark bitmap_count and the binary variant of Spark hex.
  • Document the behavior change and migration guidance in the DataFusion 55.0.0 upgrade guide.

Are these changes tested?

  • Unit tests cover materialization and preservation for Native, non-Native, and Any inputs.
  • SLTs cover to_hex materialization and verify that bitmap_count preserves its dictionary input without an additional cast to Binary.
  • Existing Spark hex dictionary tests cover its opt-in preservation behavior.

Are there any user-facing changes?

This is a behavioral API change for UDFs using typed non-Native classes such as Integer or Binary. UDFs relying on implicit dictionary preservation must now enable EncodingPreservation::dictionary() explicitly.

TypeSignatureClass::Any is unaffected. The upgrade guide has been updated, and this PR should carry the api change label.

@github-actions github-actions Bot added documentation Improvements or additions to documentation logical-expr Logical plan and expressions sqllogictest SQL Logic Tests (.slt) spark labels Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation logical-expr Logical plan and expressions spark sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant