Support DataFusion lateral vector search joins

[JingsongLi]

Summary

Adds DataFusion support for CROSS JOIN LATERAL vector_search(...) when the query vector comes from the left input column, planning it as batched vector search per input batch. The existing literal JSON vector_search path remains unchanged.

Changes

• Registers a custom optimizer rule and query planner in SQLContext to rewrite supported lateral joins into LateralVectorSearchExec.
• Extends the vector_search UDTF with a lateral marker provider for column query vector arguments.
• Executes batch vector search per left record batch, reads target rows by _ROW_ID, and emits joined left/right rows.
• Adds DataFusion coverage with per-row query vectors that produce different top-k matches.

Testing

• cargo test -p paimon-datafusion --test read_tables vector_search_tests::test_vector_search_lateral_join_uses_query_vectors -- --nocapture
• cargo test -p paimon-datafusion --test read_tables vector_search_tests::test_vector_search_java_vindex_table -- --nocapture
• cargo test -p paimon-datafusion --test read_tables vector_search_tests::test_vector_search_without_matching_index_returns_empty -- --nocapture
• cargo test -p paimon table::vector_search_builder -- --nocapture
• cargo fmt --all --check
• git diff --check
• cargo clippy -p paimon-datafusion --test read_tables -- -D warnings

Notes

• First version supports CROSS JOIN LATERAL vector_search(...) / inner joins with no join predicate. Unsupported shapes continue to fail through the marker provider scan path.
• The lateral query vector currently must be a column expression yielding List<Float32> or FixedSizeList<Float32>.

[leaves12138]

Reviewed the lateral vector_search implementation. The existing literal JSON vector_search path remains intact, and the new lateral path is cleanly routed through a logical rewrite plus a custom physical extension planner. The execution path batches query vectors from the left input, preserves the per-query result mapping, and reads target rows by _ROW_ID without falling back to a full table scan.

Validation passed locally:

• git diff --cached --check
• cargo fmt --all -- --check
• cargo test -p paimon-datafusion --test read_tables vector_search_tests::test_vector_search_lateral_join_uses_query_vectors -- --nocapture
• cargo test -p paimon-datafusion --test read_tables vector_search_tests::test_vector_search_java_vindex_table -- --nocapture
• cargo test -p paimon-datafusion --test read_tables vector_search_tests::test_vector_search_without_matching_index_returns_empty -- --nocapture
• cargo test -p paimon table::vector_search_builder -- --nocapture
• cargo clippy -p paimon-datafusion --test read_tables -- -D warnings

No blocking issues found.

[QuakeWang]

LGTM. The change is well-scoped, reuses the existing batch vector search/read path cleanly, and the added lateral join test covers the main behavior. I only left one non-blocking nit around the extension node Eq/Hash contract.

[QuakeWang]

nit: Should input also participate in dyn_hash / dyn_eq? The node's semantics depend on the child plan too, and DataFusion extension nodes generally expect Eq/Hash to represent the full logical node. Non-blocking, but including it would make the contract tighter.

[JingsongLi]

Fixed by including the child logical plan in both dyn_hash and dyn_eq. Thanks for the catch.