Skip to content

FEAT-#7676: in-place casting between DataFrame engines #7666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
sfc-gh-jkew merged 32 commits into from
Sep 26, 2025
Merged

FEAT-#7676: in-place casting between DataFrame engines #7666

sfc-gh-jkew merged 32 commits into from
Sep 26, 2025

Conversation

@sfc-gh-jkew
Copy link
Contributor

@sfc-gh-jkew sfc-gh-jkew commented Sep 11, 2025 •
edited
Loading

Addresses a performance issue with hybrid execution where a sequence of merge operations using the same DataFrames could result in transfer thrashing. In this PR we have the arguments of the operation move in-place so that subsequent operations all stay on the same backend. This behavior can be turned off by setting the BackendMergeCastInPlace variable to false.

  • first commit message and PR title follow format outlined here

    NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.

  • passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
  • passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
  • signed commit with git commit -s
  • Resolves Convert the QueryCompilerCaster to In-Place Casting to Reduce Data Migration Churn #7676
  • tests added and passing
  • module layout described at docs/development/architecture.rst is up-to-date

@sfc-gh-jkew sfc-gh-jkew force-pushed the jkew/hybrid-merge-in-place branch from 1df6492 to 55141ee Compare September 19, 2025 23:25
@sfc-gh-jkew sfc-gh-jkew changed the title WIP in-place casting between dataframe engines FEAT-#7676: in-place casting between dataframe engines Sep 22, 2025
@sfc-gh-jkew sfc-gh-jkew changed the title FEAT-#7676: in-place casting between dataframe engines FEAT-#7676: in-place casting between DataFrame engines Sep 22, 2025
@sfc-gh-jkew sfc-gh-jkew marked this pull request as ready for review September 22, 2025 23:44
sfc-gh-mvashishtha
sfc-gh-mvashishtha reviewed Sep 23, 2025
View reviewed changes
sfc-gh-joshi
sfc-gh-joshi reviewed Sep 24, 2025
View reviewed changes
sfc-gh-joshi
sfc-gh-joshi reviewed Sep 25, 2025
View reviewed changes
Copy link
Contributor

@sfc-gh-joshi sfc-gh-joshi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few more minor nits, LGTM after they're addressed.

sfc-gh-mvashishtha
sfc-gh-mvashishtha reviewed Sep 25, 2025
View reviewed changes
sfc-gh-jkew and others added 12 commits September 25, 2025 14:40
@sfc-gh-jkew
get more info on exception
55dc657
@sfc-gh-jkew @sfc-gh-joshi
Update modin/tests/pandas/extensions/test_pd_extensions.py
a4a79d4 
Co-authored-by: Jonathan Shi <[email protected]>
@sfc-gh-jkew @sfc-gh-joshi
Update modin/tests/pandas/native_df_interoperability/test_compiler_ca…
14b201b 
…ster.py

Co-authored-by: Jonathan Shi <[email protected]>
@sfc-gh-jkew @sfc-gh-joshi
Update modin/tests/pandas/native_df_interoperability/test_compiler_ca…
a45c3bb 
…ster.py

Co-authored-by: Jonathan Shi <[email protected]>
@sfc-gh-jkew @sfc-gh-joshi
Update modin/tests/pandas/native_df_interoperability/test_compiler_ca…
b9d24ad 
…ster.py

Co-authored-by: Jonathan Shi <[email protected]>
@sfc-gh-jkew @sfc-gh-joshi
Update modin/tests/pandas/native_df_interoperability/test_compiler_ca…
851cfd2 
…ster.py

Co-authored-by: Jonathan Shi <[email protected]>
@sfc-gh-jkew @sfc-gh-joshi
Update modin/tests/pandas/native_df_interoperability/test_compiler_ca…
d23a9bb 
…ster.py

Co-authored-by: Jonathan Shi <[email protected]>
@sfc-gh-jkew @sfc-gh-joshi
Update modin/tests/pandas/native_df_interoperability/test_compiler_ca…
dfb472e 
…ster.py

Co-authored-by: Jonathan Shi <[email protected]>
@sfc-gh-jkew @sfc-gh-mvashishtha
Update modin/tests/pandas/native_df_interoperability/utils.py
3d3004e 
Co-authored-by: Mahesh Vashishtha <[email protected]>
@sfc-gh-jkew
fix import
bfacfa6
@sfc-gh-jkew @sfc-gh-joshi
Update modin/tests/pandas/native_df_interoperability/test_compiler_ca…
9fd776b 
…ster.py

Co-authored-by: Jonathan Shi <[email protected]>
@sfc-gh-jkew @sfc-gh-joshi
Update modin/tests/pandas/native_df_interoperability/test_compiler_ca…
f1cd589 
…ster.py

Co-authored-by: Jonathan Shi <[email protected]>
@sfc-gh-jkew sfc-gh-jkew merged commit 6e861d0 into main Sep 26, 2025
75 of 76 checks passed
@sfc-gh-jkew sfc-gh-jkew mentioned this pull request Sep 26, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sfc-gh-joshi sfc-gh-joshi sfc-gh-joshi left review comments

@sfc-gh-mvashishtha sfc-gh-mvashishtha sfc-gh-mvashishtha approved these changes

@devin-petersohn devin-petersohn Awaiting requested review from devin-petersohn devin-petersohn is a code owner

@mvashishtha mvashishtha Awaiting requested review from mvashishtha mvashishtha is a code owner

@RehanSD RehanSD Awaiting requested review from RehanSD RehanSD is a code owner

@YarShev YarShev Awaiting requested review from YarShev YarShev is a code owner

@vnlitvinov vnlitvinov Awaiting requested review from vnlitvinov vnlitvinov is a code owner

@anmyachev anmyachev Awaiting requested review from anmyachev anmyachev is a code owner

@dchigarev dchigarev Awaiting requested review from dchigarev dchigarev is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Convert the QueryCompilerCaster to In-Place Casting to Reduce Data Migration Churn

4 participants

@sfc-gh-jkew @sfc-gh-mvashishtha @sfc-gh-joshi