Skip to content

Make partition expression evaluation stateless#4

Draft
dossett wants to merge 3 commits into
mainfrom
stateless-expression-evaluator
Draft

Make partition expression evaluation stateless#4
dossett wants to merge 3 commits into
mainfrom
stateless-expression-evaluator

Conversation

@dossett

@dossett dossett commented Jul 14, 2026

Copy link
Copy Markdown
Owner

Partition pruning currently rebuilds and binds the same expression for every data file because _ExpressionEvaluator stores the current record on itself and cannot safely be shared across manifest worker threads. Even for a single equality predicate, repeated binding costs more than the actual partition evaluation work.

This change separates immutable expression preparation from call-local evaluation state. That makes one prepared evaluator safe to share for every manifest using a partition spec, without locks or thread-local state. It is a fresh alternative to apache#3656, rooted directly at Apache main rather than stacked on that branch.

Summary

  • Keep the bound expression on _ExpressionEvaluator and move each record into a short-lived visitor.
  • Prepare one partition evaluator per spec and share it across manifest tasks.
  • Add deterministic concurrency, immutable-state, and planner-sharing coverage.
  • Add a benchmark using a single equality predicate across dense and fragmented manifests.

Performance

The focused benchmark evaluates 1,000 files against x == 5 using a two-field partition spec. Each result is the mean of five runs, with each run averaging 100 iterations.

Revision 1,000 files/manifest 1 file/manifest
main 5.001ms 5.199ms
Task-local reuse (apache#3656) 0.671ms 5.312ms
This change 0.756ms 0.928ms

Compared with main, stateless per-spec sharing is about 6.6x faster for a dense manifest and 5.6x faster for fragmented manifests. Compared with task-local reuse, constructing a call-local visitor is about 13% slower in the dense case, but avoiding one bind per manifest is about 5.7x faster in the fragmented case.

This is a focused evaluator benchmark, not an end-to-end query-runtime claim. It excludes catalog access, manifest Avro I/O and decompression, metrics evaluation, residual planning, and downstream data reads.

Testing

  • .venv/bin/python -m pytest tests/expressions/test_visitors.py -q
  • .venv/bin/python -m pytest tests/expressions/test_evaluator.py -q
  • .venv/bin/python -m pytest tests/table/test_init.py -q
  • .venv/bin/python -m pytest tests/table/test_partition_evaluator_planning.py -q
  • .venv/bin/python -m pytest tests/benchmark/test_partition_evaluator_benchmark.py -v -s -m benchmark
  • UV_NO_CONFIG=1 .venv/bin/prek run -a

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant