NOTE: I originally discovered/reported this last year on UE5.4, but did not use this forum. Original post: https://forums.unrealengine.com/t/deadlock-in-fgameplaymediaencoder/2106412/1
Our team has since fixed it by implementing this change: https://github.com/EpicGames/UnrealEngine/pull/12507
Hi all, there appears to be a possibility of deadlock when restarting highlight recording, i.e. via FHighlightRecorder::Start specifically when there is not a dedicated audio thread. The two deadlocking threads are as follows:
GameThread:
FHighlightRecorder::Start() calls
FGameplayMediaEncoder::Get()->RegisterListener(this), which takes a scoped lock on
FGameplayMediaEncoder::ListenersCS. Then it calls (conditionally)
FGameplayMediaEncoder::Start(), which eventually calls
FAudioEncoder::RegisterListener(*this), which takes another lock:
FAudioEncoder::ListenersMutex
AudioMixerRenderThread:
Run()
FOutputBuffer::MixNextBuffer()
FMixerDevice::OnProcessAudioStream()
FMixerSubmix::ProcessAudio()
FMixerSubmix::SendAudioToSubmixBufferListeners()
FGameplayMediaEncoder::OnNewSubmixBuffer()
FGameplayMediaEncoder::ProcessAudioFrame()
FWmfAudioEncoder::Encode()
FAudioEncoder::OnEncodedAudioFrame(), which locks
FAudioEncoder::ListenersMutex. In that scope it calls
FGameplayMediaEncoder::OnEncodedAudioFrame() and locks
FGameplayMediaEncoder::ListenersCS, causing a deadlock with the Game thread