Push down variant extractions in DataFusion

[JingsongLi]

Summary

This PR adds DataFusion-side Variant extraction pushdown for simple variant_get / try_variant_get expressions over Paimon VARIANT columns. Instead of always reading and rebuilding the full Variant value, eligible projections and filters can request connector-defined extraction read types and evaluate on extracted typed fields.

Changes

• Add Java-compatible Variant extraction metadata markers and allow ReadBuilder / PaimonTableScan to carry full read types, not only top-level projections.
• Teach shredding reads to assemble extraction structs from both plain Variant arrays and shredded Variant storage.
• Add a DataFusion optimizer rule and extension planner that rewrite eligible variant_get calls to scan-level Variant extraction plus get_field access.
• Keep correctness conservative when the same query also needs the full Variant column.
• Update SQL documentation to distinguish extraction pushdown from predicate/statistics pushdown.

Testing

• cargo test -p paimon variant_extraction -- --nocapture
• cargo test -p paimon read_data_fields -- --nocapture
• cargo test -p paimon-datafusion --test variant_pushdown -- --nocapture
• cargo check -p paimon-datafusion

Notes

Predicate translation through variant_get is still not pushed into Paimon/Parquet statistics. DataFusion evaluates those predicates after reading the extracted field.

[leaves12138]

I think this PR currently changes try_variant_get / Variant path semantics after extraction pushdown.

In the original UDF path, try_variant_get catches errors from variant.get_path(...) and returns NULL. With the pushed extraction path, assemble_variant_extraction_array calls variant.get_path(metadata[field_idx].path())? before cast_variant_to_shredded_value(..., fail_on_error), so an invalid path still fails the whole query even for try_variant_get.

There is also an encoding issue in the metadata description. build_variant_metadata concatenates path;fail_on_error;timezone using ; as a raw delimiter, and parse_variant_metadata splits by ;. However valid Variant paths may contain ;, e.g. a key path like $.a;b / $['a;b']. Such a query can work through the normal variant_get path but fails after pushdown with Malformed Variant metadata description.

I verified both with temporary repro tests:

• try_variant_get(payload, 'invalid_path', 'int') fails with Invalid Variant path instead of returning NULL.
• variant_get(payload, '$.a;b', 'int') fails with Malformed Variant metadata description.

Could we encode the metadata in a delimiter-safe format, such as JSON / length-prefix / escaping / base64, and apply fail_on_error to get_path errors as well? It would be good to add tests for both an invalid path under try_variant_get and a valid path containing ;.

[leaves12138]

Thanks for the update. The JSON metadata encoding and the invalid-path try_variant_get case look fixed now, and the PR's own variant_pushdown tests pass for me.

I found one more semantic regression around scalar casts after extraction pushdown. The pushed path eventually uses cast_variant_to_shredded_value / try_typed_shred, which only accepts exact VariantKind matches for many scalar targets. The existing DataFusion UDF path uses cast_variant_to_scalar, which supports broader variant_get casts such as Long/Decimal/String to numeric targets.

For example, with a shredded Variant row like {"age":27}, this query is pushed down but fails:

SELECT variant_get(payload, '$.age', 'double') AS value
FROM paimon.test_db.t
ORDER BY id

The pushed path returns Cannot cast Variant value to Double(DoubleType { nullable: true }). If I disable the pushdown by also projecting payload, the original UDF path returns 27.0 / 32.0 as expected.

I verified this with temporary tests:

• cargo test -p paimon-datafusion --test variant_pushdown on the clean PR head: 5 passed.
• Temporary variant_get(payload, '$.age', 'double') pushdown test: failed with the cast error above.
• Temporary non-pushed equivalent: passed and returned [27.0, 32.0].

Could we either reuse the same cast semantics as variant_functions.rs for extraction pushdown, or only push down scalar extractions whose pushed cast behavior is identical to the UDF? It would be good to add tests for at least Long -> Double, and probably a string-to-number try_variant_get case too.

[QuakeWang]

This cast path does not preserve the existing variant_get / try_variant_get UDF semantics. try_typed_shred only handles exact VariantKind-to-target cases plus a few narrow numeric conversions, while the current UDF also casts strings to numeric/bool/decimal, long/decimal/string to double, and object/array to string. For example, after pushdown try_variant_get(payload, '$.age_text', 'int') returns NULL for {"age_text":"27"} instead of 27. We should either reuse the same cast semantics here or only push down cases that are provably equivalent, with regression tests for these casts.

[QuakeWang]

This uses the table schema fields directly, so the variant extraction scan drops DataFusion-only fields such as _ROW_ID that the normal PaimonTableProvider::scan exposes via datafusion_read_fields when data evolution is enabled. A query selecting _ROW_ID together with a pushed variant_get can build an extension scan schema without _ROW_ID, causing later projection/field resolution to fail. This rewrite should use the same read-field/projection mapping as the normal scan and keep system fields untouched; please add a regression for data-evolution + _ROW_ID + variant_get.

[leaves12138]

One small consistency issue: this pushdown type parser accepts date, but the public variant_get / try_variant_get UDF type parser in variant_functions.rs does not support date (and the UDF cast path has no Arrow Date output branch). This makes the supported type list diverge from the non-pushdown path. Could we either remove date here, or add matching UDF support and tests for variant_get(..., 'date')?

[leaves12138]

Thanks, this is addressed now by removing date from the pushdown-supported type list and adding a regression test that keeps variant_get(..., 'date') unsupported consistently.

[leaves12138]

LGTM. The previous consistency issue has been addressed, and the variant_pushdown test passed locally.