[trainer] feat: add per-round logprob mismatch metrics for multi-turn training

[aoshen524]

What does this PR do?

Add per-round logprob mismatch metrics for multi-turn RL training. In multi-turn trajectories, the response_mask contains contiguous segments of 1s for each round of model generation, separated by 0s for environment tokens (e.g., images). This PR detects those segments and computes per-round mean absolute logprob difference between rollout and actor, making it easy to identify which round diverges most.

This extends the existing debug metrics (#1712, #2808) without changing the existing API or adding any new dependencies.

Checklist Before Starting

• Search for similar PRs: https://github.com/verl-project/verl/pulls?q=is%3Apr+mismatch+logprob
• Format the PR title as [{modules}] {type}: {description}

Test

Existing unit test passes unchanged:

tests/utils/debug/test_metrics.py::TestMetrics::test_calculate_debug_metrics PASSED

Tested with multi-turn VLM RL training (Qwen2.5-VL) on 8-turn GUI agent trajectories.

API and Usage Example

No API changes. calculate_debug_metrics(data) signature is unchanged. The returned dict now includes additional per_round/ prefixed keys:

from verl.utils.debug.metrics import calculate_debug_metrics

metrics = calculate_debug_metrics(batch)
# Existing keys still present:
#   training/rollout_probs_diff_mean, training/rollout_probs_diff_max, etc.
#
# New per-round keys (only meaningful for multi-turn):
#   per_round/total_rounds: 8
#   per_round/round_0_abs_diff_mean: 0.000123
#   per_round/round_0_token_count: 304
#   ...
#   per_round/round_7_abs_diff_mean: 0.045678
#   per_round/round_7_token_count: 376
#   per_round/max_round_diff: 7
#   per_round/max_diff_value: 0.045678

Design & Code Changes

verl/utils/debug/metrics.py (+120 lines, single file change):

• _find_contiguous_segments(mask_1d): Finds contiguous segments of 1s in response_mask to identify round boundaries
• _calculate_per_round_metrics(train_log_probs, rollout_log_probs, response_mask): Computes mean absolute logprob diff per round, aggregated across the batch
• calculate_debug_metrics(): Now calls _calculate_per_round_metrics and includes the results

Checklist Before Submitting

• Read the Contribute Guide
• Apply pre-commit checks: all hooks passed
• Add / Update the documentation
• Add unit or end-to-end test(s): existing test passes with the updated function
• Once your PR is ready for CI, send a message in the ci-request channel

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive set of metrics for debugging training-inference mismatch in multi-turn RL scenarios. The changes are primarily in verl/utils/debug/metrics.py, with a minor change in verl/trainer/ppo/ray_trainer.py to enable the new metrics. The new metrics include perplexity, KL divergence, importance weights, and more, with per-round analysis. The code is well-structured and includes good documentation. I found one issue where a newly implemented metric function is not being used, which I've commented on.

[gemini-code-assist]

The function calculate_sequence_level_mismatch_metrics is defined but never called within calculate_debug_metrics. This seems like an oversight, as the PR description mentions adding sequence-level metrics. You should add a call to this function to include its metrics in the output.

Suggested change

	metrics.update(calculate_sequence_level_mismatch_metrics(train_log_probs, rollout_log_probs, mask_tensor))