Skip to content

feat(index): add single-column btree global index build#454

Open
QuakeWang wants to merge 1 commit into
apache:mainfrom
QuakeWang:feat-btree-global-index
Open

feat(index): add single-column btree global index build#454
QuakeWang wants to merge 1 commit into
apache:mainfrom
QuakeWang:feat-btree-global-index

Conversation

@QuakeWang

Copy link
Copy Markdown
Member

Purpose

Rust already has BTree global index read support, scan integration, and row-id coverage semantics, but it did not have an entry point to build BTree global index files. This left the Rust read path unable to close the write/read lifecycle by itself.

Brief change log

  • Add a single-column BTree global index builder for data-evolution row-tracking append tables.
  • Plan index shards by contiguous global row-id ranges and reject missing first_row_id.
  • Write BTree index files under {table}/index/ with Java-compatible BTree/index manifest metadata.
  • Add Table::new_btree_global_index_build_builder().
  • Add CALL sys.create_global_index(..., index_type => 'btree') with unsupported handling for non-BTree types.
  • Add tests for shard planning, row-id gaps, missing row ids, overlap rejection, manifest metadata, SQL creation, and fast/full/detail scan behavior after appends.

Tests

  • cargo fmt --all -- --check
  • cargo clippy --all-targets --workspace --features fulltext,vortex,mosaic -- -D warnings
  • cargo test -p paimon btree
  • cargo test -p paimon table::global_index_scanner::tests --lib
  • cargo test -p paimon table::table_commit::tests::test_global_index --lib
  • cargo test -p paimon-datafusion --test procedures

API and Format

Adds a minimal Rust API and SQL procedure for BTree global index creation. The generated files use Java-compatible BTree/index manifest formats.

Documentation

Add a BTree global index builder for single-column data-evolution append tables. The builder plans row-id coverage conservatively, writes Java-compatible BTree/index manifest metadata, and exposes the minimal Rust API plus DataFusion create_global_index procedure.

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant