[BUG] Potential Memory Leak Issue

[dalek-who]

I am using agentlightning 0.3.0 to train an agent with RL. The total memory of the node is 250GB, and the memory usage keeps increasing during training. The program shuts down when the usage reaches around 220GB. Here are my logs for analysis, including stdout, stderr, config, tensorboard, and ray's session_latest, among others.

debug_log.tar.gz

[dalek-who]

Ray's session_latest is too large to upload, even after compression. If you need it, I can try to upload it later.

[dalek-who]

Here are the session_last corresponding logs for Ray. Extracting it to the same directory will restore the original directory structure.
session_latest_part_001.tar.gz
session_latest_part_002.tar.gz
session_latest_part_003.tar.gz
session_latest_part_004.tar.gz
session_latest_part_005.tar.gz
session_latest_part_006.tar.gz

[ultmaster]

I've seen similar reports before but didn't really figure out why that happens. It could be memory leakage of litellm or vllm or verl; too difficult to debug.

The easiest way would be terminate the training and resume from a checkpoint. However, if you are willing to investigate, I'd suggest you ask ChatGPT "how to debug memory leakage issue in ray worker?" and get the full memory footprint. I'll be happy to assist you if you have any further information.

[dalek-who]

Is it possible that the traces generated by the rollout have not been released? These traces are still visible in the dashboard.

[ultmaster]

In my experience, the traces won't take that much memory. Also, the older spans will be automatically evicted when the memory is near full.

If you are not sure, you can run agl store in a separated process and monitor its memory usage separately.

[dalek-who]

In my experience, the traces won't take that much memory.

Yes, I had dumped all prompts and responses, and they only occupied 2.1GB until the shutdown.

[dalek-who]

memory_analysis.tar.gz

Here is the collected data for further analysis. I monitored the program using the Ray dashboard and ran memray for 30 minutes (covering ~5 training steps) during the training process.

Attached files:

• Memory usage growth logs.

• Memray flame graph.

• Analysis report automatically generated by Cursor based on the flame graph.

memray configuration: leak=1, native=0, trace_python_allocators=0. (Note: Enabling all three options simultaneously caused the program to crash and generated excessively large files, but I can attempt to provide them if necessary.)

Please let me know if you need any further information!

[dalek-who]

2585744_memory_profiling_20260109153455.flamegraph.html

2585744_memory_profiling_20260109153455.table.html

2585744_memory_profiling_20260109153455.status.txt

This is another flame graph, a table, and a report with leak=1, native=0, trace_python_allocators=1.

[ultmaster]

@dalek-who

The whole spans created just grow up to 1 gigabytes. I think that's expected because the span needs to be stored. Will it grow to hundreds of GBs? If it's not, that's not actually the root cause of the memory leakage.

[dalek-who]

This is the result I monitored within half an hour (5 training steps). In reality, it keeps leaking as it runs.

Moreover, the leakage appears to be greater than what is shown here. Running for a day would occupy 300G of memory.

[ultmaster]

In half an hour it accumulates about 1GB. With this trend, the memory usage will be around 50GB after a day or so. It doesn't add up to the 300GB you mentioned.

[dalek-who]

I will further analyze where the memory was leaked to.