Skip to content

[client] Implement BatchScanner with limit-based scan#515

Merged
fresh-borzoni merged 2 commits into
apache:mainfrom
Arnav-panjla:feat/batch-scanner-316
May 28, 2026
Merged

[client] Implement BatchScanner with limit-based scan#515
fresh-borzoni merged 2 commits into
apache:mainfrom
Arnav-panjla:feat/batch-scanner-316

Conversation

@Arnav-panjla
Copy link
Copy Markdown
Contributor

@Arnav-panjla Arnav-panjla commented Apr 26, 2026

Purpose

Fixes #316

Implement one-shot BatchScanner using LimitScanRequest.

Brief change log

  • Added limit to TableScan
  • Added create_batch_scanner
  • Implemented BatchScanner (eager RPC, single poll_batch)
  • Added Arrow IPC (log) and KV → RecordBatch (PK) decoding
  • Added projection support

Tests

  • All tests pass (cargo test)
  • Verified with cargo check, clippy, fmt

API and Format

API Change: Yes

  • New BatchScanner, TableScan::limit, create_batch_scanner

Storage Format: No

Documentation

No

fresh-borzoni
fresh-borzoni reviewed Apr 30, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@fresh-borzoni fresh-borzoni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Arnav-panjla Ty for the PR, left some comments, PTAL

Comment thread Cargo.lock
Comment thread crates/fluss/src/client/table/batch_scanner.rs
Comment thread crates/fluss/src/client/table/batch_scanner.rs Outdated
@fresh-borzoni
Copy link
Copy Markdown
Member

fresh-borzoni commented May 9, 2026

@Arnav-panjla Do you need any help, will you be able to address comments in the review?

@fresh-borzoni
Copy link
Copy Markdown
Member

fresh-borzoni commented May 13, 2026

@Arnav-panjla Let us know if you wish to continue working on this PR, or we can take it over to finish

@Arnav-panjla
Copy link
Copy Markdown
Contributor Author

Arnav-panjla commented May 13, 2026

very sry ,
had my exams going on
I’ll update you with the next iteration within a day.

@fresh-borzoni
Copy link
Copy Markdown
Member

fresh-borzoni commented May 13, 2026

very sry , had my exams going on I’ll update you with the next iteration within a day.

@Arnav-panjla No rush, I just need to understand that you wish to continue working on this.
Good luck with the exams!

@Arnav-panjla Arnav-panjla force-pushed the feat/batch-scanner-316 branch from e43866c to 9ca191f Compare May 23, 2026 10:20
fresh-borzoni pushed a commit to Arnav-panjla/fluss-rust that referenced this pull request May 27, 2026
@Arnav-panjla @fresh-borzoni
[client] Implement BatchScanner with limit-based scan
4fc2135 
Implements a one-shot bounded BatchScanner backed by a single
LimitScanRequest RPC (fixes apache#316):
- adds TableScan::limit and create_batch_scanner
- eager RPC with leader resolution (mirrors Lookuper)
- Arrow IPC (log) and KV -> RecordBatch (PK) decoding
- projection support

Squashed from PR apache#515.
@fresh-borzoni fresh-borzoni force-pushed the feat/batch-scanner-316 branch 2 times, most recently from 1c3290c to a7b994f Compare May 27, 2026 13:02
@fresh-borzoni
Copy link
Copy Markdown
Member

fresh-borzoni commented May 27, 2026 •
edited
Loading

@Arnav-panjla Ty for the changes, LGTM overall
I added some code to address schema evolution and kv wiring, hope you don't mind my help with it, PTAL

@charlesdong1991 @leekeiabstraction Can you take a look as well?

@fresh-borzoni fresh-borzoni force-pushed the feat/batch-scanner-316 branch from a7b994f to 6521684 Compare May 27, 2026 15:27
charlesdong1991
charlesdong1991 reviewed May 27, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@charlesdong1991 charlesdong1991 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very nice pr! just a couple minor comments/questions

Comment thread crates/fluss/src/client/table/scanner.rs
Comment thread crates/fluss/tests/integration/batch_scanner.rs Outdated
Comment thread crates/fluss/src/client/table/batch_scanner.rs Outdated
Comment thread crates/fluss/src/client/table/batch_scanner.rs
@fresh-borzoni
Copy link
Copy Markdown
Member

fresh-borzoni commented May 27, 2026

@charlesdong1991 Ty for the review, addressed, PTAL 🙏

leekeiabstraction
leekeiabstraction approved these changes May 28, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@leekeiabstraction leekeiabstraction left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you @Arnav-panjla for the PR and @fresh-borzoni for taking it over the line!

charlesdong1991
charlesdong1991 approved these changes May 28, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@charlesdong1991 charlesdong1991 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for the great initial work @Arnav-panjla 👍
and @fresh-borzoni for pushing it through! 🥇

Arnav-panjla and others added 2 commits May 28, 2026 09:19
@Arnav-panjla @fresh-borzoni
[client] Implement BatchScanner with limit-based scan
7850eae 
Implements a one-shot bounded BatchScanner backed by a single
LimitScanRequest RPC (fixes apache#316):
- adds TableScan::limit and create_batch_scanner
- eager RPC with leader resolution (mirrors Lookuper)
- Arrow IPC (log) and KV -> RecordBatch (PK) decoding
- projection support

Squashed from PR apache#515.
@fresh-borzoni
[client] Fix KV limit-scan decode + enforce limit, support schema evo…
e101ee2 
…lution
@fresh-borzoni fresh-borzoni force-pushed the feat/batch-scanner-316 branch from 5793992 to e101ee2 Compare May 28, 2026 08:44
@fresh-borzoni fresh-borzoni merged commit 426bb1a into apache:main May 28, 2026
11 checks passed
fresh-borzoni pushed a commit that referenced this pull request May 28, 2026
@Arnav-panjla @fresh-borzoni
[client] Implement BatchScanner with limit-based scan
1aa24ac 
Implements a one-shot bounded BatchScanner backed by a single
LimitScanRequest RPC (fixes #316):
- adds TableScan::limit and create_batch_scanner
- eager RPC with leader resolution (mirrors Lookuper)
- Arrow IPC (log) and KV -> RecordBatch (PK) decoding
- projection support

Squashed from PR #515.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fresh-borzoni fresh-borzoni fresh-borzoni left review comments

@leekeiabstraction leekeiabstraction leekeiabstraction approved these changes

+1 more reviewer

@charlesdong1991 charlesdong1991 charlesdong1991 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Core BatchScanner implementation

4 participants

@Arnav-panjla @fresh-borzoni @charlesdong1991 @leekeiabstraction