Support running an SQL file on init

[staticlibs]

This change adds support for session_init_sql_file connection option, that allows to speficy the path to an SQL file in local file system, that will be read by the driver and executed in a newly created connection before passing it to user.

By default the file is initalized only once per database, on the first connection established to this DB.

For :memory: connection-private DBs it effectively executed once per connection.

In addition to the DB init, it supports executing a part of the SQL file for every connection. It looks for the specific marker:

/* DUCKDB_CONNECTION_INIT_BELOW_MARKER */

in the SQL file. If this marker is present - everything before the marker is executed on DB init, and everything after this marker - on connection init.

DB init is not re-run when the DB is closed and re-opened after the last connection to it was closed and then new one created. If such re-init is necessary - jdbc_pin_db option is supposed to be used instead.

It is understood, that this feature can be security sensitive (it effectively implements an RCE entry) in contexts, where other applications/processes/users can control the appending to user-specified connection string or re-writing the specified file in local file system. The following security measures are taken to mitigate that:

• session_init_sql_file option can only be specified in the connection string itself, it is not accepted as part of connection Properties
• session_init_sql_file option must be specified as the first option in the connection string, for example: 'jdbc:duckdb:;session_init_sql_file=/path/to/init.sql'
• session_init_sql_file_sha256=<sha56sum_of_sql_file> option can be specified, the file contents SHA-256 sum is checked againts this value
• session_init_sql_file_sha256 option can only be specified in the connection string itself
• session_init_sql_file and session_init_sql_file_sha256 options cannot be specified multiple times
• content of the SQL file are available to the running code using DuckDBConnection#getSessionInitSQL() method

Testing: new tests added in a separate file.

[maiadegraaf]

Hi @staticlibs, great PR, think it will add a lot of versatility.

I wonder if it would be better to go back to your original idea of two files instead of one file with a marker. You could have the option to pass two separate files, eg: db_init_sql_file and connection_init_sql_file. The user could then choose to attach one or both of the files. It would mean we could pass the file on to DuckDB without parsing it in the driver.

[staticlibs]

@maiadegraaf

Thanks for your comments! I've experimented with 2 files, the problem with them is that you also need to pass 2 sha256 sums for them. All of this in connection string. In BI tools connectors the input field for these options (and, in general, a single input field for Extra options is expected) will have limited (and pretty narrow) UI width. Requiring user to pass there 4 parameters concatenated with ; and 2 of these parameters are 64-characters SHA256 strings - this makes the UX absolutely atrocious. The goal is to give the file name max visibility in connection UIs (outside of DuckDB specific BI connectors) to prevent its malicious replacement. And 2 files are losing this security feature. And to require power-users to prepare not one, but 2 init files (that can be lost in email or mixed up) for ordinary users is also not nice.

The DUCKDB_CONNECTION_INIT_BELOW_MARKER is super clunky, but effective and easy to implement. We don't parse SQL in drivers (before actual prepare+exec calls), just split the string on the marker.

So the proposal would be to leave a single file - rather keep the marker, or drop the marker logic completely along with per-connection init logic. Connection init is less useful than DB init, for example, in DuckLake only the USE is run per-connection. And this USE is expected to be optional in almost any tool - I cannot think of a tool that won't be able to use <ducklake_alias>. prefix in front of table names.

[maiadegraaf]

Thanks for the explanation. In that case, I agree that the marker is the best of both worlds. Perhaps for now, the connection init is less useful, but this way we also future-proof it.