We had some success (on UE5.2.1) to get a stable experience by using PS with minimal quantization parameter (min QP) set to 20 (on Linux, not sure if this works out on Windows too). Make sure to restart the stream if you set that minimal limit after the stream is already started…
The quality is of course suffering from this setting, but at least we get a stable stream.
We would expect that PS is detecting and handling the QP on its own.
Seems this is not working correctly anymore.