Skip to content

Feature Request - Support create_native_delta_table for Delta Targets | Glue Crawler #37

@pkapoor-blackline

Description

@pkapoor-blackline

Describe the Feature

Current Behavior:

The delta_target block in AWS Glue crawler module does not support the create_native_delta_table argument, which limits compatibility with engines that can query Delta transaction logs directly.

dynamic "delta_target" {
  for_each = var.delta_target != null ? var.delta_target : []

  content {
    connection_name = delta_target.value.connection_name
    delta_tables    = delta_target.value.delta_tables
    write_manifest  = delta_target.value.write_manifest
  }
}

Expected Behavior

Add support for the create_native_delta_table argument per AWS Glue Crawler spec:

create_native_delta_table – (Optional) Specifies whether the crawler will create native tables, to allow integration with query engines that support querying of the Delta transaction log directly.

create_native_delta_table = delta_target.value.create_native_delta_table

Use Case

By allowing the delta target to specify argument create_native_delta_table will allow integration with query engines that support querying of the Delta transaction log directly.

Describe Ideal Solution

Update the delta target dynamic block to include the missing argument

dynamic "delta_target" {
  for_each = var.delta_target != null ? var.delta_target : []

  content {
    connection_name           = delta_target.value.connection_name
    delta_tables              = delta_target.value.delta_tables
    create_native_delta_table = delta_target.value.create_native_delta_table
    write_manifest            = delta_target.value.write_manifest
  }
}

Alternatives Considered

N/A

Additional Context

N/A

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions