Skip to content

gh-122941: Fix test_launcher sporadic failures via py.ini isolation#145090

Merged
zooba merged 2 commits intopython:mainfrom
itamaro:windows-test-launcher-isolation
Mar 4, 2026
Merged

gh-122941: Fix test_launcher sporadic failures via py.ini isolation#145090
zooba merged 2 commits intopython:mainfrom
itamaro:windows-test-launcher-isolation

Conversation

@itamaro
Copy link
Contributor

@itamaro itamaro commented Feb 21, 2026

Note: this PR was authored with assistance from Claude Code Opus 4.6

The launcher reads py.ini from %LOCALAPPDATA% via SHGetFolderPathW, which is shared across all processes for a given user. When multiple test processes run concurrently on the same machine, one process's py.ini writes leak into another process's launcher invocations, causing assertions like "PythonCore != PythonTestSuite" to fail.

Add a PYLAUNCHER_INIDIR environment variable to the launcher that, when set, redirects py.ini lookup to the specified directory and skips the default locations. Update test_launcher.py to create an isolated temp directory per test class and pass it via PYLAUNCHER_INIDIR.

I verified this fix on my Windows buildbot worker that's currently failing due to this issue.

PCbuild\amd64\python3.15t_d.exe -m test test_launcher -v

with this PR, all tests pass:

== CPython 3.15.0a6+ free-threading build (heads/windows-test-launcher-isolation:a8427d8c434, Feb 21 2026, 20:23:53) [MSC v.1950 64 bit (AMD64)]
== Windows-2022Server-10.0.20348-SP0 little-endian
== Python build: free_threading debug
== cwd: C:\Users\Administrator\cpython\build\test_python_worker_10400æ
== CPU count: 16
== encodings: locale=cp1252 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 3074267489
0:00:00 Run 1 test sequentially in a single process
0:00:00 [1/1] test_launcher
test_filter_to_company (test.test_launcher.TestLauncher.test_filter_to_company) ... ok
test_filter_to_company_and_tag (test.test_launcher.TestLauncher.test_filter_to_company_and_tag) ... ok
test_filter_to_company_with_default (test.test_launcher.TestLauncher.test_filter_to_company_with_default) ... ok
test_filter_to_tag (test.test_launcher.TestLauncher.test_filter_to_tag) ... ok
test_filter_with_single_install (test.test_launcher.TestLauncher.test_filter_with_single_install) ... ok
test_help_option (test.test_launcher.TestLauncher.test_help_option) ... ok
test_install (test.test_launcher.TestLauncher.test_install) ... ok
test_list (test.test_launcher.TestLauncher.test_list) ... ok
test_list_option (test.test_launcher.TestLauncher.test_list_option) ... ok
test_list_paths (test.test_launcher.TestLauncher.test_list_paths) ... ok
test_literal_shebang_absolute (test.test_launcher.TestLauncher.test_literal_shebang_absolute) ... ok
test_literal_shebang_command (test.test_launcher.TestLauncher.test_literal_shebang_command) ... ok
test_literal_shebang_invalid_template (test.test_launcher.TestLauncher.test_literal_shebang_invalid_template) ... ok
test_literal_shebang_quoted (test.test_launcher.TestLauncher.test_literal_shebang_quoted) ... ok
test_literal_shebang_quoted_escape (test.test_launcher.TestLauncher.test_literal_shebang_quoted_escape) ... ok
test_literal_shebang_relative (test.test_launcher.TestLauncher.test_literal_shebang_relative) ... ok
test_py2_default (test.test_launcher.TestLauncher.test_py2_default) ... ok
test_py2_default_env (test.test_launcher.TestLauncher.test_py2_default_env) ... ok
test_py2_shebang (test.test_launcher.TestLauncher.test_py2_shebang) ... ok
test_py2_shebang_nl (test.test_launcher.TestLauncher.test_py2_shebang_nl) ... ok
test_py3_default (test.test_launcher.TestLauncher.test_py3_default) ... ok
test_py3_default_env (test.test_launcher.TestLauncher.test_py3_default_env) ... ok
test_py3_shebang (test.test_launcher.TestLauncher.test_py3_shebang) ... ok
test_py3_shebang_nl (test.test_launcher.TestLauncher.test_py3_shebang_nl) ... ok
test_py_default (test.test_launcher.TestLauncher.test_py_default) ... ok
test_py_default_env (test.test_launcher.TestLauncher.test_py_default_env) ... ok
test_py_default_in_list (test.test_launcher.TestLauncher.test_py_default_in_list) ... ok
test_py_default_short_argv0 (test.test_launcher.TestLauncher.test_py_default_short_argv0) ... ok
test_py_handle_64_in_ini (test.test_launcher.TestLauncher.test_py_handle_64_in_ini) ... ok
test_py_shebang (test.test_launcher.TestLauncher.test_py_shebang) ... ok
test_py_shebang_invalid_bom (test.test_launcher.TestLauncher.test_py_shebang_invalid_bom) ... ok
test_py_shebang_nl (test.test_launcher.TestLauncher.test_py_shebang_nl) ... ok
test_py_shebang_short_argv0 (test.test_launcher.TestLauncher.test_py_shebang_short_argv0) ... ok
test_py_shebang_valid_bom (test.test_launcher.TestLauncher.test_py_shebang_valid_bom) ... ok
test_python_shebang (test.test_launcher.TestLauncher.test_python_shebang) ... ok
test_recursive_search_path (test.test_launcher.TestLauncher.test_recursive_search_path) ... ok
test_search_major_2 (test.test_launcher.TestLauncher.test_search_major_2) ... skipped 'requires at least one Python 2.x install'
test_search_major_3 (test.test_launcher.TestLauncher.test_search_major_3) ... ok
test_search_major_3_32 (test.test_launcher.TestLauncher.test_search_major_3_32) ... skipped 'requires at least one 32-bit Python 3.x install'
test_search_path (test.test_launcher.TestLauncher.test_search_path) ... ok
test_search_path_exe (test.test_launcher.TestLauncher.test_search_path_exe) ... ok
test_shebang_command_in_venv (test.test_launcher.TestLauncher.test_shebang_command_in_venv) ... ok
test_shebang_executable_extension (test.test_launcher.TestLauncher.test_shebang_executable_extension) ... ok
test_version (test.test_launcher.TestLauncher.test_version) ... ok
test_virtualenv_in_list (test.test_launcher.TestLauncher.test_virtualenv_in_list) ... ok
test_virtualenv_with_env (test.test_launcher.TestLauncher.test_virtualenv_with_env) ... ok

----------------------------------------------------------------------
Ran 46 tests in 1.353s

OK (skipped=2)
0:00:01 [1/1] test_launcher passed

== Tests result: SUCCESS ==

1 test OK.

Total duration: 1.6 sec
Total tests: run=46 skipped=2
Total test files: run=1/1
Result: SUCCESS

without it, 2 tests fail:

== CPython 3.15.0a6+ free-threading build (heads/main:c9380aebbe3, Feb 21 2026, 18:29:02) [MSC v.1950 64 bit (AMD64)]
== Windows-2022Server-10.0.20348-SP0 little-endian
== Python build: free_threading debug
== cwd: C:\Users\Administrator\cpython\build\test_python_worker_4368æ
== CPU count: 16
== encodings: locale=cp1252 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 3866893247
0:00:00 Run 1 test sequentially in a single process
0:00:00 [1/1] test_launcher
test_filter_to_company (test.test_launcher.TestLauncher.test_filter_to_company) ... ok
test_filter_to_company_and_tag (test.test_launcher.TestLauncher.test_filter_to_company_and_tag) ... ok
test_filter_to_company_with_default (test.test_launcher.TestLauncher.test_filter_to_company_with_default) ... ok
test_filter_to_tag (test.test_launcher.TestLauncher.test_filter_to_tag) ... ok
test_filter_with_single_install (test.test_launcher.TestLauncher.test_filter_with_single_install) ... ok
test_help_option (test.test_launcher.TestLauncher.test_help_option) ... ok
test_install (test.test_launcher.TestLauncher.test_install) ... ok
test_list (test.test_launcher.TestLauncher.test_list) ... ok
test_list_option (test.test_launcher.TestLauncher.test_list_option) ... ok
test_list_paths (test.test_launcher.TestLauncher.test_list_paths) ... ok
test_literal_shebang_absolute (test.test_launcher.TestLauncher.test_literal_shebang_absolute) ... ok
test_literal_shebang_command (test.test_launcher.TestLauncher.test_literal_shebang_command) ... ok
test_literal_shebang_invalid_template (test.test_launcher.TestLauncher.test_literal_shebang_invalid_template) ... ok
test_literal_shebang_quoted (test.test_launcher.TestLauncher.test_literal_shebang_quoted) ... ok
test_literal_shebang_quoted_escape (test.test_launcher.TestLauncher.test_literal_shebang_quoted_escape) ... ok
test_literal_shebang_relative (test.test_launcher.TestLauncher.test_literal_shebang_relative) ... ok
test_py2_default (test.test_launcher.TestLauncher.test_py2_default) ... ok
test_py2_default_env (test.test_launcher.TestLauncher.test_py2_default_env) ... ok
test_py2_shebang (test.test_launcher.TestLauncher.test_py2_shebang) ... ok
test_py2_shebang_nl (test.test_launcher.TestLauncher.test_py2_shebang_nl) ... ok
test_py3_default (test.test_launcher.TestLauncher.test_py3_default) ... ok
test_py3_default_env (test.test_launcher.TestLauncher.test_py3_default_env) ... ok
test_py3_shebang (test.test_launcher.TestLauncher.test_py3_shebang) ... ok
test_py3_shebang_nl (test.test_launcher.TestLauncher.test_py3_shebang_nl) ... ok
test_py_default (test.test_launcher.TestLauncher.test_py_default) ... ok
test_py_default_env (test.test_launcher.TestLauncher.test_py_default_env) ... ok
test_py_default_in_list (test.test_launcher.TestLauncher.test_py_default_in_list) ... ok
test_py_default_short_argv0 (test.test_launcher.TestLauncher.test_py_default_short_argv0) ... ok
test_py_handle_64_in_ini (test.test_launcher.TestLauncher.test_py_handle_64_in_ini) ... ok
test_py_shebang (test.test_launcher.TestLauncher.test_py_shebang) ... ok
test_py_shebang_invalid_bom (test.test_launcher.TestLauncher.test_py_shebang_invalid_bom) ... ok
test_py_shebang_nl (test.test_launcher.TestLauncher.test_py_shebang_nl) ... ok
test_py_shebang_short_argv0 (test.test_launcher.TestLauncher.test_py_shebang_short_argv0) ... ok
test_py_shebang_valid_bom (test.test_launcher.TestLauncher.test_py_shebang_valid_bom) ... ok
test_python_shebang (test.test_launcher.TestLauncher.test_python_shebang) ... ok
test_recursive_search_path (test.test_launcher.TestLauncher.test_recursive_search_path) ... ok
test_search_major_2 (test.test_launcher.TestLauncher.test_search_major_2) ... FAIL
test_search_major_3 (test.test_launcher.TestLauncher.test_search_major_3) ... FAIL
test_search_major_3_32 (test.test_launcher.TestLauncher.test_search_major_3_32) ... skipped 'requires at least one 32-bit Python 3.x install'
test_search_path (test.test_launcher.TestLauncher.test_search_path) ... ok
test_search_path_exe (test.test_launcher.TestLauncher.test_search_path_exe) ... ok
test_shebang_command_in_venv (test.test_launcher.TestLauncher.test_shebang_command_in_venv) ... ok
test_shebang_executable_extension (test.test_launcher.TestLauncher.test_shebang_executable_extension) ... ok
test_version (test.test_launcher.TestLauncher.test_version) ... ok
test_virtualenv_in_list (test.test_launcher.TestLauncher.test_virtualenv_in_list) ... ok
test_virtualenv_with_env (test.test_launcher.TestLauncher.test_virtualenv_with_env) ... ok

======================================================================
FAIL: test_search_major_2 (test.test_launcher.TestLauncher.test_search_major_2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Administrator\cpython\Lib\test\test_launcher.py", line 465, in test_search_major_2
    self.assertEqual("PythonCore", data["env.company"])
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'PythonCore' != 'PythonTestSuite'
- PythonCore
+ PythonTestSuite


======================================================================
FAIL: test_search_major_3 (test.test_launcher.TestLauncher.test_search_major_3)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Administrator\cpython\Lib\test\test_launcher.py", line 445, in test_search_major_3
    self.assertEqual("PythonCore", data["env.company"])
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'PythonCore' != 'PythonTestSuite'
- PythonCore
+ PythonTestSuite


----------------------------------------------------------------------
Ran 46 tests in 1.302s

FAILED (failures=2, skipped=1)
test test_launcher failed
0:00:01 [1/1/1] test_launcher failed (2 failures)

== Tests result: FAILURE ==

1 test failed:
    test_launcher

Total duration: 1.5 sec
Total tests: run=46 failures=2 skipped=1
Total test files: run=1/1 failed=1
Result: FAILURE

The launcher reads py.ini from %LOCALAPPDATA% via SHGetFolderPathW,
which is shared across all processes for a given user. When multiple
test processes run concurrently on the same machine, one process's
py.ini writes (e.g. python3=PythonTestSuite/3.100-arm64) leak into
another process's launcher invocations, causing assertions like
"PythonCore != PythonTestSuite" to fail.

Add a PYLAUNCHER_INIDIR environment variable to the launcher that,
when set, redirects py.ini lookup to the specified directory and skips
the default locations. Update test_launcher.py to create an isolated
temp directory per test class and pass it via PYLAUNCHER_INIDIR.
@itamaro
Copy link
Contributor Author

itamaro commented Feb 21, 2026

!buildbot AMD64 Windows Server 2022 NoGIL

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @itamaro for commit a8427d8 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F145090%2Fmerge

The command will test the builders whose names match following regular expression: AMD64 Windows Server 2022 NoGIL

The builders matched are:

  • AMD64 Windows Server 2022 NoGIL PR

@benediktjohannes
Copy link
Contributor

LGTM!

@itamaro
Copy link
Contributor Author

itamaro commented Feb 28, 2026

@zooba any concerns with this change?

@zooba zooba merged commit 6cdbd7b into python:main Mar 4, 2026
81 of 83 checks passed
@itamaro itamaro deleted the windows-test-launcher-isolation branch March 5, 2026 23:30
@itamaro itamaro added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Mar 5, 2026
@miss-islington-app
Copy link

Thanks @itamaro for the PR, and @zooba for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

@miss-islington-app
Copy link

Thanks @itamaro for the PR, and @zooba for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Mar 5, 2026
…tion (pythonGH-145090)

Adds _PYLAUNCHER_INIDIR as a private variable since the launcher is deprecated and not getting new features.
(cherry picked from commit 6cdbd7b)

Co-authored-by: Itamar Oren <itamarost@gmail.com>
@bedevere-app
Copy link

bedevere-app bot commented Mar 5, 2026

GH-145572 is a backport of this pull request to the 3.13 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Mar 5, 2026
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Mar 5, 2026
…tion (pythonGH-145090)

Adds _PYLAUNCHER_INIDIR as a private variable since the launcher is deprecated and not getting new features.
(cherry picked from commit 6cdbd7b)

Co-authored-by: Itamar Oren <itamarost@gmail.com>
@bedevere-app
Copy link

bedevere-app bot commented Mar 5, 2026

GH-145573 is a backport of this pull request to the 3.14 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Mar 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants