fix(core): fallback to child_process for WSL Windows binaries to prevent hang

[hsm207]

Summary

Fixes a critical deadlock where running Windows binaries (like pwsh.exe) via WSL in a PTY causes an indefinite hang due to unanswered Cursor Position Requests (\u001b[6n).

Details

Detects if a command resolves to a Windows path (e.g., /mnt/c/...) on Linux and forces a fallback to child_process (standard pipes) instead of using node-pty. This ensures that interactive-only escape sequences are not triggered or expected.

Related Issues

Fixes #15233

How to Validate

Run gemini -p "pwsh --version" in a WSL environment.

Pre-Merge Checklist

• Updated relevant documentation and README (if needed)
• Added/updated tests (if needed)
• Noted breaking changes (if any)
• Validated on required platforms/methods:
  • MacOS
    • npm run
    • npx
    • Docker
    • Podman
    • Seatbelt
  • Windows
    • npm run
    • npx
    • Docker
  • Linux
    • npm run
    • npx
    • Docker

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[gemini-code-assist]

Summary of Changes

Hello @hsm207, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical deadlock issue encountered when executing Windows binaries via WSL in a pseudo-terminal (PTY). The solution involves intelligently identifying such scenarios and gracefully falling back to standard child process execution, thereby bypassing the PTY-related hang caused by unhandled cursor position requests. This ensures smoother and more reliable command execution in WSL environments.

Highlights

• WSL Windows Binary Detection: Implemented logic to detect when a command executed on Linux (specifically within WSL) resolves to a Windows binary path (e.g., /mnt/c/...). This detection is crucial for preventing deadlocks.
• Forced child_process Fallback: When a WSL Windows binary is detected, the system now forces the use of node:child_process (standard pipes) instead of node-pty. This avoids issues with interactive-only escape sequences that cause hangs in PTYs.
• New Test Case: Added a dedicated test case to shellExecutionService.test.ts that specifically validates the fallback mechanism when a WSL Windows binary is identified, ensuring the fix works as intended.

Changelog

• packages/core/src/services/shellExecutionService.test.ts
  • Added mockRealpath and mockGetCommandRoot to hoisted mocks for testing file system path resolution.
  • Updated the shell-utils.js mock to include getCommandRoot.
  • Mocked node:fs to provide a controlled realpath implementation for testing.
  • Introduced a new test case to verify that child_process is used instead of PTY when a WSL Windows binary is detected.
• packages/core/src/services/shellExecutionService.ts
  • Imported the node:fs module for file system operations, specifically realpath.
  • Imported debugLogger for enhanced logging during the WSL detection process.
  • Imported getCommandRoot from shell-utils.js to extract the base command for path resolution.
  • Added a conditional block within the execute method to check for WSL Windows binaries on Linux platforms.
  • Implemented logic to resolve the command's real path and, if it matches a WSL mount pattern, disable node-pty usage.

Activity

• The author has completed the pre-merge checklist, including adding/updating tests and validating on Linux.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request aims to fix a critical deadlock when running Windows binaries on WSL through a PTY. The approach of detecting the binary and falling back to child_process is correct. However, the current implementation uses a path-based regex to detect Windows binaries, which is not robust. It will fail for users with custom WSL mount configurations, leaving them exposed to the original hang. I've suggested a more reliable method based on checking the file's magic bytes, which would make the fix comprehensive.

[gemini-code-assist]

The regex /^\/mnt\/[a-z]\// for detecting Windows binaries in WSL is brittle. It will not work for users with custom WSL mount configurations (e.g., mounting C: at /c via /etc/wsl.conf), and it could cause false positives for valid Linux paths like /mnt/a/some/dir. This means the "critical deadlock" this PR aims to fix will still occur for users with non-default configurations.

A more robust approach is to check the file's magic number to identify it as a Windows PE executable (MZ at the beginning of the file). This is independent of the file's path.

Here is an example of how you could implement this check:

async function isWindowsExecutable(filePath: string): Promise<boolean> {
  try {
    const buffer = Buffer.alloc(2);
    const fd = await fs.promises.open(filePath, 'r');
    await fd.read(buffer, 0, 2, 0);
    await fd.close();
    // Check for 'MZ' magic number
    return buffer.toString() === 'MZ';
  } catch {
    return false;
  }
}

You could then use if (await isWindowsExecutable(realPath)) instead of the regex test. When calling isWindowsExecutable, ensure filePath is a resolved path (e.g., using fs.promises.realpath) to prevent potential path traversal vulnerabilities. Also, remember to import the fs module using the node: prefix (e.g., import * as fs from 'node:fs';). It would be best to place this helper function in a utility file.

Note that the test should force child_process when detecting WSL Windows binary will also need to be updated to support this, likely by mocking fs.promises.open and fs.promises.read to return the MZ magic bytes for the given path.

async function isWindowsExecutable(filePath: string): Promise<boolean> {
  try {
    const buffer = Buffer.alloc(2);
    const fd = await fs.promises.open(filePath, 'r');
    await fd.read(buffer, 0, 2, 0);
    await fd.close();
    // Check for 'MZ' magic number
    return buffer.toString() === 'MZ';
  } catch {
    return false;
  }
}

References

1. When requesting file access permissions, resolve symbolic links first to display the actual path being accessed, preventing potential path traversal vulnerabilities.
2. Use the node: prefix when importing built-in Node.js modules for consistency across the codebase.

[hsm207]

Thanks for the feedback! I've refactored the detection logic to follow the suggested 'MZ' magic byte approach. I've also flattened the implementation using guard clauses and modern async patterns, and encapsulated it within a private helper method (isWslWindowsBinary) inside ShellExecutionService. Updated the tests to verify the magic byte detection works correctly.