[python] Support Date/Time/Timestamp/Decimal filter literals in py_to_datum#443
Merged
JingsongLi merged 2 commits intoJul 3, 2026
Merged
Conversation
…_datum
py_to_datum previously raised NotImplementedError for any non-primitive
literal, so filters on temporal or decimal columns could never be pushed
down through the Python binding. Add DataType-driven conversions:
- Date: datetime.date -> epoch days; rejects datetime.datetime (a date
subclass) so the time-of-day part is never silently dropped.
- Time: naive datetime.time -> millis-of-day; rejects tz-aware times and
sub-millisecond microseconds (Datum::Time carries millis; truncation
would change comparison semantics).
- Timestamp: naive datetime.datetime -> {millis, nanos} with a Euclidean
split for pre-epoch instants; rejects tz-aware datetimes, matching the
DataFusion pushdown path which refuses zoned literals for TIMESTAMP.
- LocalZonedTimestamp: tz-aware datetime.datetime normalized through
astimezone(utc) (works with zoneinfo/pytz/fixed offsets); rejects naive
datetimes as ambiguous.
- Decimal: decimal.Decimal or int, rescaled losslessly to the column
scale via as_tuple(); rejects rounding, precision overflow, NaN/Inf,
bool, and binary floats.
All rejections are ValueError (type mismatch), keeping NotImplementedError
for genuinely unsupported types (Bytes/complex).
…literals The aware check was tzinfo-is-set, but Python defines aware as tzinfo set AND utcoffset() not None. A tzinfo whose utcoffset() returns None (a 'floating' timezone) passed the check, and astimezone(utc) then interpreted the value as process-local time — the same literal would normalize to different instants depending on the machine's timezone. Check utcoffset() explicitly and reject such pseudo-naive datetimes with the same ValueError as plain naive ones. Timestamp keeps rejecting ANY tzinfo (stricter than Python's definition, but explicit and deterministic); a new test documents that choice.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
py_to_datumin the Python binding raisesNotImplementedErrorfor any non-primitive literal, so a filter on a temporal or decimal column can never be pushed down through the DataFrame-style read path — every such query fails even though the core predicate machinery fully supports theseDatumvariants. This adds DataType-driven literal conversion for Date / Time / Timestamp / LocalZonedTimestamp / Decimal.Complements #430, which adds the equivalent temporal conversions on the DataFusion (SQL) path; this PR covers the Python binding path.
Linked issue: none.
Brief change log
datetime.date→ epoch days.datetime.datetime(adatesubclass) is rejected so the time-of-day part is never silently dropped.datetime.time→ millis-of-day. Timezone-aware times and sub-millisecond microseconds are rejected (Datum::Timecarries millis; silent truncation would change comparison semantics).datetime.datetime→{millis, nanos}with a Euclidean split so pre-epoch instants keep a non-negative nano remainder — same convention asBinaryRow::get_timestamp_rawand the DataFusion pushdown conversion. Timezone-aware datetimes are rejected (parity with the SQL path, which refuses zoned literals for TIMESTAMP).datetime.datetimenormalized throughastimezone(utc), so any tzinfo implementation (zoneinfo, pytz, fixed offsets) is handled by Python itself. Naive datetimes are rejected as ambiguous.decimal.Decimalorint, rescaled losslessly to the column's scale viaas_tuple()(avoids Decimal context rounding). Values needing rounding, exceeding the column precision, non-finite values (NaN/Infinity),bool, and binaryfloats are rejected.ValueError(type mismatch);NotImplementedErrorremains for genuinely unsupported types (Bytes/complex).chronofeature of pyo3 (chrono is already in the workspace dependency tree; no new crates).Tests
cargo test -p pypaimon_rust— 51 passed. New unit tests cover epoch-day conversion (incl. pre-epoch), datetime-as-date rejection, time tz/sub-milli rejection, naive/aware timestamp acceptance+rejection for both TIMESTAMP and TIMESTAMP_LTZ, pre-epoch Euclidean split, and decimal exact-scale / lossless-rescale / scientific-notation / negative / int-literal / rounding / overflow / NaN / float / bool cases.uv run --no-sync pytest tests/test_read.py— 39 passed. New end-to-end tests build a table with DATE/TIMESTAMP/DECIMAL columns across two files and assert the filter both prunes splits (stats-based) and returns the matching rows viaTableRead.read.API and Format
No API surface change (the same
with_filter(dict)accepts more literal types). No storage format change.Documentation
Behavior documented on
py_to_datumand the new conversion helpers.