Skip to content

Appender rework (1.3)#305

Merged
staticlibs merged 1 commit intoduckdb:v1.3-ossivalisfrom
staticlibs:appender_datachunk_13
Jul 8, 2025
Merged

Appender rework (1.3)#305
staticlibs merged 1 commit intoduckdb:v1.3-ossivalisfrom
staticlibs:appender_datachunk_13

Conversation

@staticlibs
Copy link
Collaborator

This is a backport of the PR #295 to v1.3-ossivalis stable branch.

This change implements the access to the Appender interface from Java with the following features:

  • C API is used to access the native Appender
  • necessary C API calls are exposed to Java using JNI wrappers as thin as possible - Java calls mirror corresponding C API calls 1 to 1
  • the data chunk interface of the Appender API is used: vector data is exposed as a direct ByteBuffer, all primitive appended values are written to this buffer from Java without going through JNI + C API ( that is still used for some types with calls like: duckdb_vector_assign_string_element_len)
  • Java-side Appender/DataChunk/Vector data structures follow the Go lang's Appender implementation closely (with nested arrays initialization etc)
  • Java Appender usage is made thread-safe for concurrent Appender or Connection closure; append() calls are remained not thread-safe (to minimize the overhead), but their usage cannot cause the JNI-side crash
  • Java API of the new Appender is modeled after the java.lang.StringBuilder class and intended to be used with method chaining
  • support primitive arrays (one and two dimensional)
  • support for NULL and DEFAULT values
  • type checks between Java types and DB column types are enforced

Note: previous version of the Appender (that internally creates Value instances and appends them one by one) is still available for compatibility. It can be created using
Connection#createSingleValueAppender method. It is marked as 'deprecated' and intended to be removed in future versions.

Testing: existing Appender test suite is extended and adapted to new Appender API.

Fixes: #84, #100, #110, #139, #157, #163, #219, #249

This is a backport of the PR duckdb#295 to `v1.3-ossivalis` stable branch.

This change implements the access to the Appender interface from Java with
the following features:

 - C API is used to access the native Appender
 - necessary C API calls are exposed to Java using JNI wrappers as thin
   as possible - Java calls mirror corresponding C API calls 1 to 1
 - the data chunk interface of the Appender API is used: vector data is
   exposed as a direct ByteBuffer, all primitive appended values are
   written to this buffer from Java without going through JNI + C API (
   that is still used for some types with calls like:
   `duckdb_vector_assign_string_element_len`)
 - Java-side Appender/DataChunk/Vector data structures follow the Go
   lang's Appender implementation closely (with nested arrays
   initialization etc)
 - Java Appender usage is made thread-safe for concurrent Appender or
   Connection closure; `append()` calls are remained not thread-safe (to
   minimize the overhead), but their usage cannot cause the JNI-side
   crash
 - Java API of the new Appender is modeled after the
   `java.lang.StringBuilder` class and intended to be used with method
   chaining
 - support primitive arrays (one and two dimensional)
 - support for `NULL` and `DEFAULT` values
 - type checks between Java types and DB column types are enforced

Note: previous version of the Appender (that internally creates
`Value` instances and appends them one by one) is still available for
compatibility. It can be created using
`Connection#createSingleValueAppender` method. It is marked as
'deprecated' and intended to be removed in future versions.

Testing: existing Appender test suite is extended and adapted to new
Appender API.

Fixes: duckdb#84, duckdb#100, duckdb#110, duckdb#139, duckdb#157, duckdb#163, duckdb#219, duckdb#249
@staticlibs staticlibs merged commit 7b84064 into duckdb:v1.3-ossivalis Jul 8, 2025
@staticlibs staticlibs deleted the appender_datachunk_13 branch July 8, 2025 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant