Skip to content

fix: format identity partition paths for nanosecond timestamps#3659

Open
anxkhn wants to merge 1 commit into
apache:mainfrom
anxkhn:patch-18
Open

fix: format identity partition paths for nanosecond timestamps#3659
anxkhn wants to merge 1 commit into
apache:mainfrom
anxkhn:patch-18

Conversation

@anxkhn

@anxkhn anxkhn commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

An identity transform on a timestamp_ns / timestamptz_ns column renders the
raw integer value in its human string, so identity partition directory paths
come out as ts=1740342104375612001 instead of an ISO-8601 timestamp. The
microsecond timestamp / timestamptz types render correctly (e.g.
ts=2025-02-23T20:21:44.375612), so this is an inconsistency in the nanosecond
timestamp types added for the v3 spec.

The cause is in _int_to_human_string (pyiceberg/transforms.py): it dispatches
DateType, TimeType, TimestampType, and TimestamptzType, but has no
registration for TimestampNanoType / TimestamptzNanoType, so those values
fall through to the base handler str(value). Because
IdentityTransform.to_human_string and PartitionSpec.partition_to_path
(pyiceberg/partitioning.py) both route through _int_to_human_string, an
identity-partitioned nanosecond timestamp column produces a raw-integer
partition path.

This change adds to_human_timestamp_ns / to_human_timestamptz_ns helpers in
pyiceberg/utils/datetime.py and registers them for the two nanosecond
timestamp types. Python's datetime only supports microsecond precision, so the
helpers render the microsecond instant at a fixed 6 fractional digits and append
the remaining 3 sub-microsecond digits, yielding a full 9 fractional-digit
ISO-8601 string with no trailing-zero trimming. This matches the Java
DateTimeUtil.nanosToIsoTimestamp / nanosToIsoTimestamptz behaviour used by
Transforms.identity().toHumanString() (for example
1510871468000001001 -> 2017-11-16T22:31:08.000001001, with the zone offset kept
after the fraction for the tz variant).

Are these changes tested?

Yes.

  • tests/utils/test_datetime.py: new test_to_human_timestamp_ns and
    test_to_human_timestamptz_ns cover the epoch boundary and the
    sub-microsecond digit boundaries (0, 1, 999, 1000), a whole-second-plus-1ns
    case (where the microsecond fraction is zero), and the Java reference value.
  • tests/test_transforms.py::test_identity_human_string: adds timestamp_ns
    and timestamptz_ns rows alongside the existing type rows.
  • tests/table/test_partitioning.py::test_partition_spec_to_path_timestamp: an
    end-to-end PartitionSpec.partition_to_path check for the microsecond and
    nanosecond timestamp/timestamptz types (the user-facing symptom).

The targeted subset passes (14 passed). Running the full tests/test_transforms.py,
tests/utils/test_datetime.py, and tests/table/test_partitioning.py files
shows only the pre-existing failures that require the optional pyiceberg_core
extension; these new tests do not touch that path. Integration tests
(Docker + Spark) were not run in this environment; the behaviour is covered by
the unit tests end-to-end through partition_to_path.

Are there any user-facing changes?

Yes. Identity partition paths for timestamp_ns and timestamptz_ns columns are
now ISO-8601 strings (matching the microsecond timestamp types and the Java
implementation) instead of raw integers. IdentityTransform.to_human_string for
these types changes correspondingly.

Identity transform on timestamp_ns / timestamptz_ns rendered the raw
integer for to_human_string, producing non-ISO partition directory
paths (e.g. ts=1740342104375612001) unlike the microsecond timestamp
types which produce ISO-8601 strings.

_int_to_human_string dispatched Date/Time/Timestamp/Timestamptz but had
no registration for TimestampNanoType / TimestamptzNanoType, so those
values fell through to str(value). Add to_human_timestamp_ns and
to_human_timestamptz_ns helpers that render full 9 fractional-digit
ISO-8601 strings (matching the Java Transforms behaviour), and register
them for the nanosecond timestamp types.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant