feat: Add SGLang rollout backend and tests

[RolaoDenthu]

What does this PR do ?

Add comprehensive test coverage for SGLang generation backend, including functional tests, unit tests, and nightly tests.

• Functional Test (tests/functional/grpo_sglang.sh): Quick validation of SGLang-based GRPO training
• Unit Tests (tests/unit/models/generation/test_sglang_generation.py): unit tests covering:
  • Basic configuration validation
  • Policy generation and tensor parallelism
  • Worker seed behavior for RLHF diversity
  • HTTP server direct API access
  • Weight updates with DTensor policy (colocated mode)
  • Prefix cache reset after weight updates
• Nightly Test (tests/test_suites/llm/grpo-qwen3-0.6b-1n8g-sglang.sh): End-to-end convergence test for SGLang backend

Usage

• You can potentially add a usage example below

# Run functional test
uv add coverage
bash tests/functional/grpo_sglang.sh

# Run unit tests
uv sync --extra sglang --group test
uv run python -m pytest tests/unit/models/generation/test_sglang_generation.py -v --sglang-only

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Summary by CodeRabbit

Release Notes

• New Features

  • Distributed generation engine using SGLang backend with HTTP weight streaming and multi-GPU support.

• Configuration

  • New YAML configuration templates for SGLang-based experiments with customizable generation parameters.

• Tests

  • Comprehensive test coverage for SGLang generation, including tensor parallelism, batching, and dynamic weight updates.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[guyueh1]

quote lint check

File nemo_rl/models/generation/sglang/init.py has zero errors but is not in pyrefly.toml in the 'project-includes' list. Please add it to this whitelist.

[guyueh1]

quote build-container job

The lockfile at uv.lock needs to be updated, but --locked was provided. To update the lockfile, run uv lock.
You need to try locally build docker first to make sure it works (check docs/docker.md for instructions). During the process you will find you need to update the uv.lock file, add that change to the PR

[RolaoDenthu]

quote build-container job

The lockfile at uv.lock needs to be updated, but --locked was provided. To update the lockfile, run uv lock.
You need to try locally build docker first to make sure it works (check docs/docker.md for instructions). During the process you will find you need to update the uv.lock file, add that change to the PR

I have updated the uv.lock and pyrefly.toml.

[guyueh1]

@RolaoDenthu lint seems to be complaining a new file not added to the pyrefly.toml
Please check the test log, it has the shell command, please run it locally to make sure it passes, or use the pre-commit hook (as here )

[github-actions]

⚠️ File Consistency Check

Check based on commit: 0513cbf (PR #1674 from add-tests)

⚠️ DTensor Policy Worker Synchronization Warning

The file nemo_rl/models/policy/workers/dtensor_policy_worker_v2.py was modified in this PR, but nemo_rl/models/policy/workers/dtensor_policy_worker.py was not updated.

Why this matters:
These files contain related DTensor policy worker implementations that should be kept synchronized to ensure consistency across different versions.

Action required:

• Please review if the changes in nemo_rl/models/policy/workers/dtensor_policy_worker_v2.py should also be applied to nemo_rl/models/policy/workers/dtensor_policy_worker.py
• Update nemo_rl/models/policy/workers/dtensor_policy_worker.py if necessary to maintain consistency
• If the files are intentionally different, please add a comment in the PR explaining why

Files to check:

• Modified: nemo_rl/models/policy/workers/dtensor_policy_worker_v2.py
• Not modified: nemo_rl/models/policy/workers/dtensor_policy_worker.py

---

_{This check ensures that related file implementations remain synchronized across the codebase. If you believe this warning is incorrect or the files should intentionally differ, please add a comment explaining the reasoning.}

[RolaoDenthu]

@guyueh1 Hi I’ve fixed the environment and it should now be able to run with the current uv lock. Could you please restart the test?

[github-actions]

⚠️ File Consistency Check

Check based on commit: 570f996 (PR #1674 from add-tests)

⚠️ DTensor Policy Worker Synchronization Warning

The file nemo_rl/models/policy/workers/dtensor_policy_worker_v2.py was modified in this PR, but nemo_rl/models/policy/workers/dtensor_policy_worker.py was not updated.

Why this matters:
These files contain related DTensor policy worker implementations that should be kept synchronized to ensure consistency across different versions.

Action required:

• Please review if the changes in nemo_rl/models/policy/workers/dtensor_policy_worker_v2.py should also be applied to nemo_rl/models/policy/workers/dtensor_policy_worker.py
• Update nemo_rl/models/policy/workers/dtensor_policy_worker.py if necessary to maintain consistency
• If the files are intentionally different, please add a comment in the PR explaining why

Files to check:

• Modified: nemo_rl/models/policy/workers/dtensor_policy_worker_v2.py
• Not modified: nemo_rl/models/policy/workers/dtensor_policy_worker.py

---

_{This check ensures that related file implementations remain synchronized across the codebase. If you believe this warning is incorrect or the files should intentionally differ, please add a comment explaining the reasoning.}

[terrykong]

hey @RolaoDenthu . i'm trying to get the examples in this PR to run and running into some issues. I'll submit a PR against your branch tomorrow with some fixes