Termination due to SIGUSR1 is called and process is killed abruptly

[AdarshRai3]

Bug Description

These are the logs :

DEFAULT 2026-01-05T15:59:01.559673Z livekit::rtc_engine:470:livekit::rtc_engine - received session close: "server request to leave" Migration Resume
DEFAULT 2026-01-05T15:59:01.559687Z livekit::rtc_engine:470:livekit::rtc_engine - received session close: "signal client closed: \"stream closed\"" UnknownReason Resume
DEFAULT 2026-01-05T15:59:01.560258Z livekit::rtc_engine:766:livekit::rtc_engine - resuming connection... attempt: 0
DEFAULT 2026-01-05T15:59:14.110770Z livekit::rtc_engine::rtc_session:711:livekit::rtc_engine::rtc_session - signal_event taking too much time: Answer(SessionDescription { r#type: "answer", sdp: "v=0\r\no=- 5241511117965315297 1767628744 IN IP4 0.0.0.0\r\ns=-\r\nt=0 0\r\na=msid-semantic:WMS *\r\na=fingerprint:sha-256 3F:79:F5:2C:D2:1C:4C:72:1A:A8:9D:1D:D6:44:E7:EA:CD:26:7E:9F:EC:95:B1:26:48:EE:D7:2D:5B:D8:31:25\r\na=ice-lite\r\na=extmap-allow-mixed\r\na=group:BUNDLE 0 1\r\nm=audio 9 UDP/TLS/RTP/SAVPF 63 111 0 8\r\nc=IN IP4 0.0.0.0\r\na=setup:active\r\na=mid:0\r\na=ice-ufrag:yltvXnSPulzkxXVW\r\na=ice-pwd:swgBSsbYhtKzDSvDPekKSZzBCoIdRJMf\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:63 red/48000/2\r\na=fmtp:63 111/111\r\na=rtpmap:111 opus/48000/2\r\na=fmtp:111 minptime=10;useinbandfec=1;usedtx=1\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:8 PCMA/8000\r\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=recvonly\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 0.0.0.0\r\na=setup:active\r\na=mid:1\r\na=sendrecv\r\na=sctp-port:5000\r\na=max-message-size:65535\r\na=ice-ufrag:yltvXnSPulzkxXVW\r\na=ice-pwd:swgBSsbYhtKzDSvDPekKSZzBCoIdRJMf\r\n", id: 0 })
DEFAULT 2026-01-05T15:59:19.394358Z process exiting
DEFAULT 2026-01-05T16:14:19.386271Z process did not exit in time, killing process
DEFAULT 2026-01-05T16:14:19.386288Z killing process
DEFAULT 2026-01-05T16:14:19.386293Z sending SIGUSR1 signal to process
DEFAULT 2026-01-05T16:14:19.421846Z process exited with non-zero exit code -10

I am using version of livekit version 1.3.10, as you can see from the logs the termination signal is generated from the server which is sending sigusr1 and process is exiting with code zero.

logs are generated from this part of livekit, this file
livekit-agents/livekit/agents/ipc/supervised_proc.py
line 310-317

 if hasattr(signal, "SIGUSR1"):
                try:
                    logger.info("sending SIGUSR1 signal to process", extra=self.logging_extra())
                    os.kill(self._proc.pid, signal.SIGUSR1)  # type: ignore[arg-type]
                    await asyncio.sleep(0.5)
                except Exception:
                    pass
            self._proc.kill()

Kindly give me stable version in which I can switch to resolve this error , I have checked the code for blocker code I cannot find any.

Expected Behavior

Just want agent to not kill process in the middle of the session and call cleanup every time when it terminated by user or self terminate.

Reproduction Steps

1. It is not reproducing every session but in some session , so I cannot reproduce it.
...
- Sample code snippet, or a GitHub Gist link -

Operating System

windows

Models Used

open ai

Package Versions

# -------- LiveKit (VERSION-ALIGNED) --------
    "livekit>=1.0.23",
    "livekit-agents>=1.3.10",
    "livekit-api>=1.1.0",

    # -------- LiveKit Plugins --------
    "livekit-plugins-aws>=1.3.10",
    "livekit-plugins-azure>=1.3.10",
    "livekit-plugins-cartesia>=1.3.10",
    "livekit-plugins-deepgram>=1.3.10",
    "livekit-plugins-google>=1.3.10",
    "livekit-plugins-openai>=1.3.10",
    "livekit-plugins-silero>=1.3.10",
    "livekit-plugins-noise-cancellation>=0.2.5",

Session/Room/Call IDs

No response

Proposed Solution

Additional Context

No response

Screenshots and Recordings

[AdarshRai3]

I have checked thrice in my code , I have not find any blocker code.
Due this my cleanup is not running which have some essential process which need to completed for my session.
I have also set my : shutdown_process_timeout=300.0 but it is still giving me these issues

[longcw]

SIGUSR1 is not the reason, it's the agent process didn't exit gracefully so it was killed after a timeout. does this happens consistently? it will be helpful if you can share a reproducible example.

[AdarshRai3]

SIGUSR1 is not the reason, it's the agent process didn't exit gracefully so it was killed after a timeout. does this happens consistently? it will be helpful if you can share a reproducible example.

I agree, the process is getting in a infinite loop, since I set shutdown_process_timeout=900.0 and process is strucking in blocker code for 15 minutes and then it got terminated, therefore I check my code for blocker code but I have not find any.

I am getting this error in 1/5 sessions , I have removed all the method from the cleanup and just add logs but I still got the issue, I will again check my code .

this is how I am calling cleanup_session , can you guide me :

            ctx.add_shutdown_callback(lambda: asyncio.create_task(
                cleanup_session(ctx,session,llm_instance)
            ))

I have added enter logs in cleanup_session when that logs are not working.

[chenghao-mou]

asyncio.create_task() schedules the coroutine and returns a Task object immediately and it doesn't wait. The callback can finish before cleanup_session even starts executing.

During shutdown, this is dangerous because:

• The shutdown sequence continues without waiting for cleanup
• Resources the cleanup depends on may already be torn down, which means it will hang/keep waiting

[AdarshRai3]

asyncio.create_task() schedules the coroutine and returns a Task object immediately and it doesn't wait. The callback can finish before cleanup_session even starts executing.

During shutdown, this is dangerous because:

• The shutdown sequence continues without waiting for cleanup
• Resources the cleanup depends on may already be torn down, which means it will hang/keep waiting

Thanks but how can I solve this issues that completing the cleanup task essential for my application , what could be proper and safer method to call cleanup according to livekit

[chenghao-mou]

Use sync method is a good way to start. Can you share what you are doing inside the clean up function?

[AdarshRai3]

Use sync method is a good way to start. Can you share what you are doing inside the clean up function?

I have used the sync method now , it is still causing the issue , also I have add log in the first line of the method even that logs are not printing.

[chenghao-mou]

Then it is likely you have other blocking code even before shutdown callbacks are called. e.g. any other similar dangling async Tasks. One way to find out is to set up debug mode:

LK_DUMP_STACK_TRACES=1 LK_DUMP_DIR="path to a directory" python main.py

When SIGUSR1 is triggered, hopefully you will see relevant function calls that are causing the hang.

[davidzhao]

possibly related to #4500, which had just been fixed

[AdarshRai3]

possibly related to #4500, which had just been fixed

Thanks , I will just update to the latest version.

[AdarshRai3]

Then it is likely you have other blocking code even before shutdown callbacks are called. e.g. any other similar dangling async Tasks. One way to find out is to set up debug mode:

LK_DUMP_STACK_TRACES=1 LK_DUMP_DIR="path to a directory" python main.py

When SIGUSR1 is triggered, hopefully you will see relevant function calls that are causing the hang.

Actually, I just received the solution of this problem but still can you please elaborate on this how to use LK_DUMP to diagnose issues like this. I will be thankful if you write step by step diagnoses.

[chenghao-mou]

Then it is likely you have other blocking code even before shutdown callbacks are called. e.g. any other similar dangling async Tasks. One way to find out is to set up debug mode:
LK_DUMP_STACK_TRACES=1 LK_DUMP_DIR="path to a directory" python main.py

When SIGUSR1 is triggered, hopefully you will see relevant function calls that are causing the hang.

Actually, I just received the solution of this problem but still can you please elaborate on this how to use LK_DUMP to diagnose issues like this. I will be thankful if you write step by step diagnoses.

It's just two extra environment variables to help export the stack traces. It's like snapshot, so if you see any user code/function, that's usually a red flag. You can always share the dump file here so we can help you locate the issue if you want.

[AdarshRai3]

possibly related to #4500, which had just been fixed

I am currently on 1.3.10 , will I get the fix in the next release.

[AdarshRai3]

Then it is likely you have other blocking code even before shutdown callbacks are called. e.g. any other similar dangling async Tasks. One way to find out is to set up debug mode:
LK_DUMP_STACK_TRACES=1 LK_DUMP_DIR="path to a directory" python main.py
When SIGUSR1 is triggered, hopefully you will see relevant function calls that are causing the hang.

Actually, I just received the solution of this problem but still can you please elaborate on this how to use LK_DUMP to diagnose issues like this. I will be thankful if you write step by step diagnoses.

It's just two extra environment variables to help export the stack traces. It's like snapshot, so if you see any user code/function, that's usually a red flag. You can always share the dump file here so we can help you locate the issue if you want.

I am new to livekit , I have added those two variables in env but I dont get dump in the file, if you give me steps, I will be thankful : LK_DUMP_STACK_TRACES=1 LK_DUMP_DIR="/c/pocs/project/dump" python src/main.py

User@DESKTOP-HHKIOE4 MINGW64 /c/pocs/project (fix/technical-orchestrator)
$ LK_DUMP_STACK_TRACES=1 LK_DUMP_DIR="/c/pocs/project/dump" python src/main.py
Usage: main.py [OPTIONS] COMMAND [ARGS]...
Try 'main.py --help' for help.
╭─ Error ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Missing command.                                                                                                          │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯