Skip to content

Batch INSERT statements in executemany to reduce round trips#605

Open
srchilukoori wants to merge 1 commit into
trinodb:masterfrom
srchilukoori:batch-executemany-insert
Open

Batch INSERT statements in executemany to reduce round trips#605
srchilukoori wants to merge 1 commit into
trinodb:masterfrom
srchilukoori:batch-executemany-insert

Conversation

@srchilukoori
Copy link
Copy Markdown

Description

Fixes #602.

Cursor.executemany() executes INSERT...VALUES one row at a time.
For 1,000 rows that means 1,000 HTTP round trips to Trino.

This adds a regex check for INSERT...VALUES patterns and batches
rows into multi-row VALUES clauses, 100 per request. Non-INSERT
statements still use the existing row-by-row path. Parameters are
formatted with the existing _format_prepared_param() method.

Scenario Row-by-row Batched Speedup
500 rows x 2 cols 8.9s 0.23s ~39x
1,000 rows x 25 cols (mixed types) timeout 3.1s (322 rows/s)

Non-technical explanation

Bulk inserts via SQLAlchemy text() or cursor.executemany() now
send fewer queries to Trino instead of one per row.

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text:

* Batch INSERT...VALUES in `executemany()` into multi-row statements to reduce HTTP round trips. ({issue}`602`)

@cla-bot
Copy link
Copy Markdown

cla-bot Bot commented May 25, 2026

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Sadha Chilukoori.
This is most likely caused by a git client misconfiguration; please make sure to:

  1. check if your git client is configured with an email to sign commits git config --list | grep email
  2. If not, set it up using git config --global user.email email@example.com
  3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

@srchilukoori srchilukoori force-pushed the batch-executemany-insert branch from 35385b1 to 92069b8 Compare May 26, 2026 15:52
@cla-bot
Copy link
Copy Markdown

cla-bot Bot commented May 26, 2026

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

SQLAlchemy bulk insert is executed per row

1 participant