perf: cache residuals during scan planning

[dossett]

I used CODEX to analyze this problem and create this PR. I've reviewed the code and test and stand by them. This summary is written completely by a human (me) other than very light copy editing by an LLM.

Problem: During planning, a predicate is applied to determine the appropriate contents. Parts of the predicate can be removed because they're already satisfied by the partition scheme in a table. What can't be removed is the residual. Currently, a residual is constructed for every file. However, files written with the same partition spec and the same relevant partition values will have the same residual, so recalculating it is wasteful.

This change caches the file-level residual, but not the mutable ResidualEvaluator. On a cache miss, it constructs a fresh evaluator, calculates and caches the residual, and then discards the evaluator. The cache key contains the file's partition spec ID and the values of partition fields whose source columns are referenced by the predicate. If a predicate doesn't use one of the partition columns for a table, that column won't be in the key.

The cache is local to one plan_files() call and limited to 128 keys. This retains a small working set of repeated partition values while preventing unbounded growth when every file has a distinct relevant partition value.

CODEX-generated summary follows:

---

Rationale for this change

Scan planning currently calculates a residual for every data file and repeatedly rebuilds the partition schema while projecting predicates. For scans with many files and a large combined filter, this repeated work can dominate driver-side planning time.

Files can also differ in partition fields that the scan filter never references. Those fields cannot affect the residual, but including them in a cache key prevents reuse. This is particularly costly for layouts with a high-cardinality partition hash: files with the same predicate-relevant partition values still appear unique.

This change:

• Constructs each residual evaluator's partition schema once.
• Binds the scan filter once to identify its referenced source field IDs.
• Derives relevant partition-field positions separately for each spec, including every transform of a referenced source.
• Caches residual expressions by partition spec ID and only the relevant partition values.
• Constructs a fresh residual evaluator on every cache miss and constructs none on a cache hit.

The evaluator lifetime is intentional. ResidualEvaluator stores the partition being evaluated in self.struct, so sharing an instance across concurrent evaluations would be unsafe. An earlier version of #2695 attempted to cache evaluator instances; this PR instead caches only the resulting residual expressions. Keeping the spec ID in the key preserves isolation across evolved partition specs. Partition pruning and delete-file matching continue to use the complete partition record.

Performance

The benchmark plans 2,000 files with a 66-leaf filter. Every file has a unique high-cardinality partition-hash value that is not referenced by the filter. Results report the mean of three in-process runs using separate main and PR worktrees on the same machine and Python environment:

Predicate-relevant partitions	main	This PR	Improvement
7	2.886 s	0.010 s	~289x faster
2,000	3.273 s	1.563 s	~2.1x faster

A same-machine comparison with the previous PR revision, which reused one mutable evaluator per spec, measured 0.009 seconds for seven relevant partitions and 1.468 seconds for 2,000. Fresh evaluator construction therefore leaves the cache-hit-heavy case effectively unchanged and adds about 6% in the all-miss case, while eliminating shared evaluator state.

This is an isolated scan-planning benchmark: manifest entry loading is stubbed so it measures residual evaluation and FileScanTask construction. It does not include catalog access, manifest I/O, or downstream data execution, so these numbers are not an end-to-end query-speed claim.

Are these changes tested?

Yes.

• Added coverage proving partition schemas are constructed once per residual evaluator.
• Added coverage proving every cache miss receives a fresh evaluator and cache hits construct none.
• Added coverage for reuse by relevant partition values and isolation across evolved specs.
• Added coverage proving unreferenced partition fields are ignored and referenced fields remain in the key.
• Added coverage for multiple partition transforms of the same referenced source.
• Added coverage for bounded-cache eviction and correctness after eviction.
• Added an EtsyML-shaped benchmark with a unique irrelevant hash per file.
• Focused table-planning and residual suite: 126 passed.
• All repository hooks pass, including Ruff, mypy, pydocstyle, codespell, and lockfile validation.

Are there any user-facing changes?

There are no API or result-semantics changes. Local scan planning is faster for filtered scans, especially when many files differ only in partition fields unrelated to the scan predicate.

AI assistance

Codex assisted with implementation, test scaffolding, benchmark design, and PR drafting. I reviewed the logic and validated it with focused tests, repository hooks, and isolated before/after benchmarks.

[dossett]

In the Iceberg slack @jayceslesar pointed out that #2695 addresses a related problem. One key objection in that PR was that the ResidualEvaluator should not be cached and reused because it is not thread safe. My original PR was doing that, but I changed it to never reuse the ResidualEvaluator but it still does cache the resulting residual-expression, which can be safely reused across files. In certain circumstances this is a significant benefit.

(cc @jayceslesar @Fokko @kevinjqliu @ForeverAngry from that first PR)