How to get the entire rollout in multiturn setting?

[Blair1213]

I use multi turn setting for a stepwise verification and implement it with customize reward. However, I can’t get the whole rollout in reward function by solution_str. Which variable or config can make the whole rollout available?

[wuxibin89]

@Blair1213 I think solution_str already contains whole multi-turn rollout including assistant and tool response.
https://github.com/volcengine/verl/blob/main/verl/experimental/reward_loop/reward_manager/naive.py#L54-L56

cc @yyDing1

[yyDing1]

Yes, the solution_str contains the whole multi-turn info. In the multi-turn reward scenario, I suggest the following practice:

• (most suggested for research purpose): directly compute reward in the customized agent loop, AgentLoopOutput can accept reward score
• Decode solution_str into chat history. This can be done in reward_manager, where you can restore chat_history from response and its corresponding mask. Specifically, the response_mask of the tool part is 0 while that of the assistant is 1. This is not very recommended as the response_ids would contain some unexpected special tokens, which are hard to tackle.

[Blair1213]

Thank you @yyDing1 and @wuxibin89 for replying. Could you please share a tutorial of customized agent loop with me?

[yyDing1]

Tool agent loop directly returns turn-level rewards:

verl/verl/experimental/agent_loop/tool_agent_loop.py

Line 208 in 8b01d0b

output.extra_fields.update({"turn_scores": agent_data.turn_scores, "tool_rewards": agent_data.tool_rewards})

[wuxibin89]

Basically, you just need to inherit AgentLoopBase and implement the run method, then you can hook it in:

actor_rollout_ref.rollout.agent.agent_loop_config_path=/path/to/agent_loop.yaml

agent_loop.yaml:

- name: math_expression
  _target_: recipe.langgraph_agent.example.math_expression.MathExpressionReactAgentLoop

See the LangGraphAgentLoop example:
https://github.com/verl-project/verl-recipe/blob/main/langgraph_agent/example/run_qwen2.5_3b.sh#L129

[Blair1213]

Thank you for your reply. I checked these two files and make sure the tool works. However, I found that the response_str or the output ToolAgentLoop didn't contain the response of tools when I prompt the policy to put the response between <tool_response>...</tool_response>. Does this is normal that the response_str didn't contain the response from tools or I use it wrongly. Could I prompt it to put the response between two tags?