Skip to content

feat(datafusion): add push down temporal literals#430

Open
liujiwen-up wants to merge 3 commits into
apache:mainfrom
liujiwen-up:fix/datafusion-time-literals
Open

feat(datafusion): add push down temporal literals#430
liujiwen-up wants to merge 3 commits into
apache:mainfrom
liujiwen-up:fix/datafusion-time-literals

Conversation

@liujiwen-up

@liujiwen-up liujiwen-up commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Purpose

DataFusion filter pushdown currently cannot translate temporal literal ScalarValues into Paimon Datums, so predicates on Paimon TIME, TIMESTAMP, and TIMESTAMP WITH LOCAL TIME ZONE columns fall open unnecessarily.

Changes

  • Translate ScalarValue::Time32Millisecond into Datum::Time for Paimon Time fields.
  • Translate timezone-free TimestampSecond / TimestampMillisecond / TimestampMicrosecond / TimestampNanosecond into Datum::Timestamp.
  • Translate timezone-bearing timestamp scalar values into Datum::LocalZonedTimestamp.
  • Use div_euclid / rem_euclid for sub-millisecond timestamp parts so negative timestamps before 1970 split correctly.
  • Keep timezone semantic mismatches unsupported so DataFusion residual filters continue to preserve correctness.

Tests

  • cargo test -p paimon-datafusion filter_pushdown
  • cargo test -p paimon-datafusion --test read_tables test_temporal_filter_pushdown_via_datafusion_scan -- --nocapture
  • cargo test -p paimon-datafusion was also run locally; all new filter_pushdown tests passed, but the full package run hit existing fixture-dependent failures for missing default.partitioned_log_table / default.multi_partitioned_log_table in table::tests::*scan* tests.

@liujiwen-up liujiwen-up changed the title fix(datafusion): push down temporal literals feat(datafusion): add push down temporal literals Jul 1, 2026
.await
.expect("INSERT should collect");

let timestamp_sql = "SELECT id, name FROM paimon.test_db.temporal_filter_pushdown \

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This residual predicate is true for the row matched by the temporal predicate, so the result would stay the same even if the residual filter were dropped. Could we make the residual condition actually filter out a temporal match, or assert the plan still contains a residual FilterExec?

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@QuakeWang QuakeWang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants