Exercise Python PVR test on Flink 2.0 #37313

Abacn · 2026-01-14T21:21:42Z

Please add a meaningful description for your change here

Fix #37583

Part of #36947

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

gemini-code-assist · 2026-01-14T21:22:07Z

Summary of Changes

Hello @Abacn, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on preparing the build environment and continuous integration (CI) for Apache Flink 2.0. It refines the Gradle build configuration to dynamically manage Flink versions, specifically enabling the system to recognize Flink 2.0 as the latest version. Additionally, various CI trigger files have been updated, including the introduction of new Flink-related test triggers, to ensure comprehensive testing with the anticipated Flink 2.0 integration.

Highlights

Flink Versioning Logic Update: The build system's logic for determining the latestFlinkVersion has been updated. It now dynamically selects the last version from the allFlinkVersions list, removing a previous hardcoded value of '1.20' and a TODO comment related to Flink 2.0 support, indicating readiness for Flink 2.0.
CI Trigger File Modifications: Numerous .github/trigger_files have been modified. These changes include incrementing modification/revision counters and removing outdated pull request references, primarily to re-trigger or update existing Flink-related continuous integration tests.
New Flink Test Suite Activations: New empty trigger files have been added for beam_PostCommit_Java_Tpcds_Flink.json, beam_PostCommit_PortableJar_Flink.json, and beam_PostCommit_Python_Examples_Flink.json. This suggests the activation or preparation of new Flink-specific test suites within the CI pipeline.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Ignored Files

Ignored by pattern: .github/workflows/** (7)
- .github/workflows/beam_PostCommit_Java_Nexmark_Flink.yml
- .github/workflows/beam_PostCommit_Java_PVR_Flink_Streaming.yml
- .github/workflows/beam_PostCommit_Java_Tpcds_Flink.yml
- .github/workflows/beam_PostCommit_Java_ValidatesRunner_Flink.yml
- .github/workflows/beam_PostCommit_XVR_Flink.yml
- .github/workflows/beam_PreCommit_Java_PVR_Flink_Batch.yml
- .github/workflows/beam_PreCommit_Java_PVR_Flink_Docker.yml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

Abacn · 2026-02-03T19:29:37Z

PostCommit Java Nexmark Flink: https://github.com/apache/beam/actions/runs/21644527625

PostCommit Java PVR Flink Streaming: https://github.com/apache/beam/actions/runs/21644546969

PostCommit Java ValidatesRunner Flink: https://github.com/apache/beam/actions/runs/21644604617

Abacn · 2026-02-09T21:39:19Z

Current test failures:

FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestOptimized::test_batch_pardo_fusion_break
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestOptimized::test_batch_to_element_pardo
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestOptimized::test_flattened_side_input
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestOptimized::test_gbk_side_input
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestOptimized::test_multimap_multiside_input
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestOptimized::test_multimap_side_input
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestOptimized::test_multimap_side_input_type_coercion
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestOptimized::test_pack_combiners
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestOptimized::test_pardo_side_input_dependencies
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestOptimized::test_pardo_side_inputs
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestOptimized::test_pardo_unfusable_side_inputs
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestOptimized::test_pardo_unfusable_side_inputs_with_separation
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestOptimized::test_pardo_windowed_side_inputs
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestOptimized::test_sdf_synthetic_source

java.lang.IllegalArgumentException
	at org.apache.beam.model.pipeline.v1.RunnerApi$Components.getTransformsOrThrow(RunnerApi.java:1038)
	at org.apache.beam.runners.flink.FlinkStreamingPortablePipelineTranslator.getSideInputIdToPCollectionViewMap(FlinkStreamingPortablePipelineTranslator.java:991)
	at org.apache.beam.runners.flink.FlinkStreamingPortablePipelineTranslator.transformSideInputs(FlinkStreamingPortablePipelineTranslator.java:1033)
	at org.apache.beam.runners.flink.FlinkStreamingPortablePipelineTranslator.translateExecutableStage(FlinkStreamingPortablePipelineTranslator.java:755)
	at org.apache.beam.runners.flink.FlinkStreamingPortablePipelineTranslator.translate(FlinkStreamingPortablePipelineTranslator.java:280)
	at org.apache.beam.runners.flink.FlinkStreamingPortablePipelineTranslator.translate(FlinkStreamingPortablePipelineTranslator.java:127)
	at org.apache.beam.runners.flink.FlinkPipelineRunner.runPipelineWithTranslator(FlinkPipelineRunner.java:96)
	at org.apache.beam.runners.flink.FlinkPipelineRunner.run(FlinkPipelineRunner.java:86)
	at org.apache.beam.runners.jobsubmission.JobInvocation.runPipeline(JobInvocation.java:86)
	at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75)
	at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

problems related to side input

https://github.com/apache/beam/actions/runs/21840420138/job/63022922889

Abacn · 2026-02-13T01:53:01Z

.github/workflows/beam_PostCommit_Python_Portable_Flink.yml

-        environment_type: ['DOCKER', 'LOOPBACK']
+        # all environment_type: ['DOCKER', 'LOOPBACK', 'PROCESS']
+        # Run modes not covered by PreCommit_Python_PVR_Flink (i.e. other than 'LOOPBACK')
+        environment_type: ['DOCKER']


LOOPBACK for both Flink 1.20 and Flink 2.0 already have coverage, dedup'd here

Abacn · 2026-02-13T01:53:49Z

sdks/java/extensions/sql/expansion-service/build.gradle

  manifest {
    attributes(["Multi-Release": true])
  }
-  outputs.upToDateWhen { false }


This was a workaround leftover for corrupted remote Gradle cache, could cause elevated compile time.

Abacn · 2026-02-13T01:54:38Z

sdks/python/apache_beam/runners/portability/flink_runner_test.py

-      return [
-          'java',
-          '-XX:-UseContainerSupport',
+      return ['java'] + platform_specific_opts + [


Fix flink_runner_test on MacOS (and persumably windows)

codecov · 2026-02-13T02:24:46Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 54.88%. Comparing base (9ea7380) to head (f29e3d3).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #37313       +/-   ##
=============================================
+ Coverage     35.88%   54.88%   +18.99%     
  Complexity     1691     1691               
=============================================
  Files          1063     1063               
  Lines        166668   166741       +73     
  Branches       1227     1227               
=============================================
+ Hits          59804    91509    +31705     
+ Misses       104665    73035    -31630     
+ Partials       2199     2197        -2

Flag	Coverage Δ
python	`80.38% <ø> (+40.69%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Abacn · 2026-02-13T03:29:59Z

Flink 2 Tests passed:

PreCommit Python PVR Flink (runs Flink 2 LOOPBACK):

https://github.com/apache/beam/actions/runs/21971664799

PostCommit Python Portable Flink (runs Flink 2 DOCKER):

https://github.com/apache/beam/actions/runs/21971674796

github-actions · 2026-02-13T04:19:55Z

Assigning reviewers:

R: @shunping for label python.
R: @kennknowles for label java.
R: @liferoad for label build.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

github-actions bot added the build label Jan 14, 2026

Abacn force-pushed the flink2.0-tag branch from 3202eee to 774627a Compare February 9, 2026 19:53

github-actions bot added the python label Feb 9, 2026

Abacn force-pushed the flink2.0-tag branch 2 times, most recently from 1565d23 to 2c39771 Compare February 9, 2026 20:58

github-actions bot added the runners label Feb 9, 2026

Abacn changed the title ~~Exercise Flink 2.0 test~~ Exercise Python PVR test on Flink 2.0 Feb 9, 2026

Abacn mentioned this pull request Feb 12, 2026

[Bug]: Python pipeline having side input fails on Flink runner when useDataStreamForBatch and pre_optimization #37583

Open

17 tasks

Exercise Flink 2.0 Python Validates Runner tests

f29e3d3

Abacn force-pushed the flink2.0-tag branch from 2c39771 to f29e3d3 Compare February 13, 2026 01:51

github-actions bot added java extensions sql flink labels Feb 13, 2026

Abacn commented Feb 13, 2026

View reviewed changes

Abacn marked this pull request as ready for review February 13, 2026 03:30

github-actions bot added the Next Action: Reviewers label Feb 13, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exercise Python PVR test on Flink 2.0 #37313

Exercise Python PVR test on Flink 2.0 #37313

Uh oh!

Abacn commented Jan 14, 2026 •

edited

Loading

Uh oh!

gemini-code-assist bot commented Jan 14, 2026

Uh oh!

Abacn commented Feb 3, 2026 •

edited

Loading

Uh oh!

Abacn commented Feb 9, 2026

Uh oh!

Abacn Feb 13, 2026

Uh oh!

Abacn Feb 13, 2026 •

edited

Loading

Uh oh!

Abacn Feb 13, 2026

Uh oh!

codecov bot commented Feb 13, 2026 •

edited

Loading

Uh oh!

Abacn commented Feb 13, 2026

Uh oh!

github-actions bot commented Feb 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Exercise Python PVR test on Flink 2.0 #37313

Are you sure you want to change the base?

Exercise Python PVR test on Flink 2.0 #37313

Uh oh!

Conversation

Abacn commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Actions Tests Status (on master branch)

Uh oh!

gemini-code-assist bot commented Jan 14, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

Abacn commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Abacn commented Feb 9, 2026

Uh oh!

Abacn Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

Abacn Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Abacn Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Abacn commented Feb 13, 2026

Uh oh!

github-actions bot commented Feb 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Abacn commented Jan 14, 2026 •

edited

Loading

Abacn commented Feb 3, 2026 •

edited

Loading

Abacn Feb 13, 2026 •

edited

Loading

codecov bot commented Feb 13, 2026 •

edited

Loading