Skip to content

Remove mutable Parquet buffer use from native_iceberg_compat #2152

@andygrove

Description

@andygrove

What is the problem the feature request solves?

As discovered in #2147, the native_iceberg_compat scan currently re-uses some legacy code that uses mutable Parquet buffers. As a result, we often need to perform a deep copy of all batches produced from this scan, which impacts performance. There is also the risk of memory corruption if our logic for determining when to make copies is not correct.

Describe the potential solution

I would like natice_iceberg_compat scans to stop using mutable Parquet buffers so that we can stop making copies of the output batches.

Additional context

No response

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions