Skip to content

Adding support for Row Id mapping for composite merge#20608

Open
darjisagar7 wants to merge 1 commit intoopensearch-project:feature/datafusionfrom
darjisagar7:RowIdMapping
Open

Adding support for Row Id mapping for composite merge#20608
darjisagar7 wants to merge 1 commit intoopensearch-project:feature/datafusionfrom
darjisagar7:RowIdMapping

Conversation

@darjisagar7
Copy link

Description

This PR will create the RowIdMapping which will be used for the composite merges

image image

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Sagar Darji <darjisagar7@gmail.com>
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Feb 12, 2026

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Tip

Issue Planner is now in beta. Read the docs and try it out! Share your feedback on Discord.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link
Contributor

❌ Gradle check result for f2560c5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

}

// Create mappings for this file
for old_row_id in 0..file_rows as i64 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we are creating RowIdMappingData struct for every row, for a merge of files totaling N rows, this will be 3N Java heap allocations triggered from Rust through JNI + 4N JNI calls from create_row_id_mapping_object
Instead, since the mapping is just new_row_id = file_start_offset + old_row_id, we can pass only per-file segment metadata (file ID, start offset, row count) across JNI and materialize the full mapping in Java.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants