I noticed this in the Bunlabs.Naudio.Flac fork, but it looks like the issue originated here.
BunLabs/NAudio.Flac#3 is a diff that at least prevents the pinned memory from being lost in multiple channel audio decoding. In case someone wants to fix the issue here too.