Support fetching query results in chunks#682
Open
staticlibs wants to merge 1 commit into
Open
Conversation
354997e to
4c32b11
Compare
This PR allows to read a query result as a lazily-fetched sequence of [data chunks](https://github.com/duckdb/duckdb-java/blob/32d68a448d27f00e0e86f59e454dcf7a674e9cc8/src/duckdb/src/include/duckdb/common/types/data_chunk.hpp#L26-L30). It effectively exposes the [duckdb_fetch_chunk](https://github.com/duckdb/duckdb-java/blob/32d68a448d27f00e0e86f59e454dcf7a674e9cc8/src/duckdb/src/include/duckdb.h#L5376-L5386) call to Java allowing to read the results in batches avoiding the per-row overhead mandated by the JDBC specification. For accessing the chunks contents the same `DuckDBDataChunkReader` API is used as with [Java user-defined functions](https://github.com/duckdb/duckdb-java/blob/32d68a448d27f00e0e86f59e454dcf7a674e9cc8/UDF.MD). Usage example: ```java try (DuckDBConnection conn = DriverManager.getConnection("jdbc:duckdb:").unwrap(DuckDBConnection.class); DuckDBPreparedStatement ps = conn.prepare("SELECT ? AS col1")) { ps.setInt(1, 42); // statement parameters are still 1-based try (DuckDBChunkedResult res = ps.query()) { // advance to the next chunk, returns true on success while (res.nextChunk()) { // get the current chunk from the result DuckDBDataChunkReader chunk = res.chunk(); // iterate over the chunk columns, all indices are 0-based for (long columnIndex = 0; columnIndex < chunk.columnCount(); columnIndex++) { // get a vector for the specified column DuckDBReadableVector vector = chunk.vector(columnIndex); // iterate over vector rows for (long rowIndex = 0; rowIndex < chunk.rowCount(); rowIndex++) { // get a value in the vector on the specified row int val = vector.getInt(rowIndex); System.out.println(val); } } } } } ``` Note1: Currently it only supports basic data types, support for composite types (`LIST`, `STRUCT`) is going to be added in future. Note2: the `query()` method can only be used on prepared statements, currently there is no `query(String)` overload.
4c32b11 to
e6b2d6c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR allows to read a query result as a lazily-fetched sequence of data chunks.
It effectively exposes the duckdb_fetch_chunk call to Java allowing to read the results in batches avoiding the per-row overhead mandated by the JDBC specification.
For accessing the chunks contents the same
DuckDBDataChunkReaderAPI is used as with Java user-defined functions.Usage example:
Note1: Currently it only supports basic data types, support for composite types (
LIST,STRUCT) is going to be added in future.Note2: the
query()method can only be used on prepared statements, currently there is noquery(String)overload.