Choppy Audio in HomeKit

[scottlow]

Hey there,

I'm trying to add a camera stream to HomeKit but am noticing that the audio cuts out every ~3 seconds. AFAICT, this issue is specific to HomeKit as the stream's audio works correctly in VLC (using the raw RTMP stream) and Frigate (using the go2rtc re-stream). Here's the config I'm using:

go2rtc:
  streams:
    nanit:
      - rtmp://<cameraIP>
  homekit:
    nanit:
      name: Nanit
      pin: <pin>

Here's the info on the RTMP stream from VLC:

I've tried re-encoding the stream using ffmpeg:rtmp://<cameraIP>#video=copy#audio=opus and ffmpeg:rtmp://<cameraIP>#video=copy#audio=aac, but neither seems to address the issue. I've also tried adding /16000 to the end of the audio parameter to force a specific sample rate.

My understanding is that this should have been fixed in #667, but is there something obvious I'm missing here? Or logs I can provide to help debug? Thanks in advance!

[AlexxIT]

Try to use #audio=opus/16000

go2rtc/pkg/hap/camera/ch115_supported_audio.go

Lines 23 to 25 in 753d661

	AudioCodecSampleRate8Khz = 0
	AudioCodecSampleRate16Khz = 1
	AudioCodecSampleRate24Khz = 2

[scottlow]

Try to use #audio=opus/16000

go2rtc/pkg/hap/camera/ch115_supported_audio.go

Lines 23 to 25 in 753d661

	AudioCodecSampleRate8Khz = 0
	AudioCodecSampleRate16Khz = 1
	AudioCodecSampleRate24Khz = 2

I tried that, but still no luck:

ffmpeg:rtmp://<cameraIP>#video=copy#audio=opus/16000

One thing I just noticed in the go2rtc WebUI is that adding #audio=opus/16000 doesn't seem to actually change the audio codec sample rate. Is this expected?

EDIT: Nvm. Just realized that per the Opus spec, audio will always be at 48 kHz and this parameter just impacts band limiting. Is there something else I should be trying?

{
  "producers": [
    {
      "id": 62,
      "format_name": "rtsp",
      "protocol": "rtsp+tcp",
      "remote_addr": "127.0.0.1:56930 forwarded 192.168.1.101:1935",
      "source": "exec:/usr/lib/ffmpeg/7.0/bin/ffmpeg -hide_banner -v error -fflags nobuffer -flags low_delay -i rtmp://192.168.1.101:1935 -c:v copy -c:a libopus -application:a lowdelay -min_comp 0 -ar:a 16000 -ac:a 1 -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:8554/5a8cb3ba73ff45c787a0430b40268272",
      "url": "rtmp://192.168.1.101:1935",
      "sdp": "v=0\r\no=- 0 0 IN IP4 127.0.0.1\r\ns=No Name\r\nc=IN IP4 127.0.0.1\r\nt=0 0\r\na=tool:libavformat 61.1.100\r\nm=video 0 RTP/AVP 96\r\nb=AS:3145\r\na=rtpmap:96 H264/90000\r\na=fmtp:96 packetization-mode=1; sprop-parameter-sets=Z01AKI2NQDwBE/LgLcBAQFAAAD6AAATiDoYA/8AAF9eC7y40MAf+AAC+vBd5cKA=,aO44gAA=; profile-level-id=4D4028\r\na=control:streamid=0\r\nm=audio 0 RTP/AVP 97\r\nb=AS:64\r\na=rtpmap:97 opus/48000/2\r\na=control:streamid=1\r\n",
      "user_agent": "ffmpeg/go2rtc",
      "medias": [
        "video, recvonly, H264",
        "audio, recvonly, OPUS/48000/2"
      ],
      "receivers": [
        {
          "id": 63,
          "codec": {
            "codec_name": "h264",
            "codec_type": "video",
            "level": 40,
            "profile": "Main"
          },
          "childs": [
            65,
            67,
            116
          ],
          "bytes": 26077979,
          "packets": 18975
        },
        {
          "id": 64,
          "codec": {
            "channels": 2,
            "codec_name": "opus",
            "codec_type": "audio",
            "sample_rate": 48000
          },
          "childs": [
            66,
            68,
            117
          ],
          "bytes": 1550723,
          "packets": 10419
        }
      ],
      "bytes_recv": 27983726
    }
  ],
  "consumers": [
    {
      "id": 13,
      "format_name": "rtsp",
      "protocol": "rtsp+tcp",
      "remote_addr": "127.0.0.1:56742",
      "sdp": "v=0\r\no=- 1 1 IN IP4 0.0.0.0\r\ns=go2rtc/1.9.10\r\nc=IN IP4 0.0.0.0\r\nt=0 0\r\nm=video 0 RTP/AVP 96\r\na=rtpmap:96 H264/90000\r\na=fmtp:96 packetization-mode=1; sprop-parameter-sets=Z01AKI2NQDwBE/LgLcBAQFAAAD6AAATiDoYA/8AAF9eC7y40MAf+AAC+vBd5cKA=,aO44gAA=; profile-level-id=4D4028\r\na=recvonly\r\na=control:trackID=0\r\nm=audio 0 RTP/AVP 97\r\na=rtpmap:97 OPUS/48000/2\r\na=recvonly\r\na=control:trackID=1\r\n",
      "user_agent": "FFmpeg Frigate/0.17.0-430cebe",
      "medias": [
        "video, sendonly, ANY",
        "audio, sendonly, ANY"
      ],
      "senders": [
        {
          "id": 65,
          "codec": {
            "codec_name": "h264",
            "codec_type": "video",
            "level": 40,
            "profile": "Main"
          },
          "parent": 63,
          "bytes": 26077979,
          "packets": 18975
        },
        {
          "id": 66,
          "codec": {
            "channels": 2,
            "codec_name": "opus",
            "codec_type": "audio",
            "sample_rate": 48000
          },
          "parent": 64,
          "bytes": 1550723,
          "packets": 10419
        }
      ],
      "bytes_send": 28099006
    },
    {
      "id": 27,
      "format_name": "rtsp",
      "protocol": "rtsp+tcp",
      "remote_addr": "127.0.0.1:56888",
      "sdp": "v=0\r\no=- 1 1 IN IP4 0.0.0.0\r\ns=go2rtc/1.9.10\r\nc=IN IP4 0.0.0.0\r\nt=0 0\r\nm=video 0 RTP/AVP 96\r\na=rtpmap:96 H264/90000\r\na=fmtp:96 packetization-mode=1; sprop-parameter-sets=Z01AKI2NQDwBE/LgLcBAQFAAAD6AAATiDoYA/8AAF9eC7y40MAf+AAC+vBd5cKA=,aO44gAA=; profile-level-id=4D4028\r\na=recvonly\r\na=control:trackID=0\r\nm=audio 0 RTP/AVP 97\r\na=rtpmap:97 OPUS/48000/2\r\na=recvonly\r\na=control:trackID=1\r\n",
      "user_agent": "FFmpeg Frigate/0.17.0-430cebe",
      "medias": [
        "video, sendonly, ANY",
        "audio, sendonly, ANY"
      ],
      "senders": [
        {
          "id": 67,
          "codec": {
            "codec_name": "h264",
            "codec_type": "video",
            "level": 40,
            "profile": "Main"
          },
          "parent": 63,
          "bytes": 26077979,
          "packets": 18975
        },
        {
          "id": 68,
          "codec": {
            "channels": 2,
            "codec_name": "opus",
            "codec_type": "audio",
            "sample_rate": 48000
          },
          "parent": 64,
          "bytes": 1550723,
          "packets": 10419
        }
      ],
      "bytes_send": 28099006
    },
    {
      "id": 115,
      "format_name": "probe",
      "protocol": "http",
      "remote_addr": "192.168.1.29:54172",
      "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 18_7 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.2 Mobile/15E148 Safari/604.1",
      "medias": [
        "video, sendonly, ALL",
        "audio, sendonly, ALL",
        "audio, recvonly, ANY"
      ],
      "senders": [
        {
          "id": 116,
          "codec": {
            "codec_name": "h264",
            "codec_type": "video",
            "level": 40,
            "profile": "Main"
          },
          "parent": 63
        },
        {
          "id": 117,
          "codec": {
            "channels": 2,
            "codec_name": "opus",
            "codec_type": "audio",
            "sample_rate": 48000
          },
          "parent": 64
        }
      ]
    }
  ]
}