Welcome to the great “Video Resolution for Pixel Streaming” topic :).
The ongoing challenge is to have a video streaming resolution that is in line with (or equal to) the rendered game resolution.
There are quite a few forum posts around this topic. Here you will find them summarized so we can hopefully all use one resource.
UE Version Summary
4.26:
Everything works fine. Video resolution is updated automatically after r.setRes command
4.27:
Video resolution is captured on creation of webRtc Player and not updated afterwards.
Forum Posts on the topic: (link1) (link2) (link3)
Automatic or manual updating can be re-enabled or with commands:
5.0:
Video resolution is captured on creation of webRtc Player and not updated afterwards. The commands to enable it (in 4.27) have been removed.
See (link) (section ‘Pixel Streaming’) for details on the updates in UE5.0
Forum Posts on the topic (link1) (link2)
UE5-0 Workarounds
Currently there are no known workarounds for the issue, short of setting a resolution, refreshing the page and loging in again. We hope that a discussion in this forum topic with bring us a solution.
UE5-0 Solutions to explore
So far the most promising solution seems
Running the following functions in sequence (from app.js) does results in the player restarting with a fresh stream:
This doesn’t work if you change resolution before running the functions. No video appears on the screen.
Running the functions repeatedly results in severe performance degradation. Refreshing the browser resolved this.
adding ws.close() to the sequence forces the user to click on the screen before video resumes, but strangely also shows 2 parallel streams in the stats section of the browser
I’m hoping that we can come up with a working solution together!
I suspect that the Javascript sequence need some adjustments, as it may not be closing the old stream as expected. It is possible to have the streamer send a command to client to notify when the resolution change has been completed, only then will the client run the sequence.
Looks like there is one important thing that needs to be added to the plugin, that is to destroy the existing NvEnc encoding sessions as soon as possible and clean up everything related to it, so that they don’t take up processing power, and to avoid hitting the session limit if you are using Geforce instead of Quadro line.
Yes, I’m pretty sure it needs some adjustments… Unfortunately what I’m trying here is waaay over my head so it’s more like educated guessing than programming…
I’m only going base on my use case, which may not be applicable to yours. I’m also creating a custom web frontend which means it is very different from the sample provided by Epic. Basically what I do is after the client sent the resolution change command, it calls ws.close(), wait 100 msec, call connect(). I do not call neither webRtcPlayerObj.close() nor playVideoStream() directly. For the case of the sample app.js, you may want to set shouldShowPlayOverlay = false, make sure that inside ws.onclose handler you prevent the timeout call to the start function.
On the UE side, I know that for my app it is always 1 user <-> 1 instance of UE, which means I can eagerly destroy the old hardware encoders by inserting a line into UE::PixelStreaming::FVideoEncoderFactory::GetOrCreateHardwareEncoder in EncoderFactory.cpp like this:
So far this setup works alright for me, there is a delay when the reconnection is going on, but I can easily cover it up with a loading screen or something.
I just feel like the way Epic built Pixel Streaming is overkill, too complicated for what it should be. At some point I probably will have to build my own simplified plugin that only do what I need and no more.
I’m here to help shed a bit of light on the issue,
So the lack of stream resolution changing in Pixel Streaming in UE 5.0 - 5.0.3 is a known regression. With the changes needed to fix the issue, it couldn’t quite make it into the hot-fix release, much to our chagrin.
Good news however, is that it is already fixed in the ue5-main branch on Github!
Sorry for the inconvenience, but know that we’re aware and tackling these issues.
I tested the plugin on ue5-main, seems to work alright, except for one thing, which is there is often a delay before the resolution is updated, and sometimes it may get stuck in the old resolution until I force it to change by resizing the window again.
Are you seeing similar things? On my side I can reproduce this quite consistently by changing to a resoltution of 2048x2048 (no idea why this number… higher is fine, lower is fine…)
Edit:
When this happens, the UE application continues on running, but the PixelStreaming part seems to hang. Refreshing the browser doesn’t help, only restarting the UE application.
Edit2:
I’m also seeing crashes when closing the UE application that say
Assertion failed: InBuffer [File:D:\build++UE5\Sync\Engine\Plugins\Media\HardwareEncoders\Source\EncoderNVENC\Private\NVENC_EncoderH264.cpp] [Line: 827] Cannot destroy buffer in NVENC - buffer was nullptr.
You can refer to my earlier message about modifying C++ code inside the Pixel Streaming plugin. That is required, in order to shutdown the unused hardware video encoder instances, particularly important if you are not running the UE game instance on Quadro level graphics card. If you do not want to recompile the entire engine source for this purpose, you can copy the plugin folder from engine source and put into your project’s plugin folder, use that plugin instead of the pre-bundled plugin.
@trqan97 I’m trying to follow your instructions (UE5.0.3), but I keep on running into compiling issues:
LNK2019 Unresolved external symbol … webrtc.lib …
How do I resolve this?
Then, I modified below function in EncoderFactory.ccp like this:
TOptional<UE::PixelStreaming::FVideoEncoderFactory::FHardwareEncoderId> UE::PixelStreaming::FVideoEncoderFactory::GetOrCreateHardwareEncoder(int Width, int Height, int MaxBitrate, int TargetBitrate, int MaxFramerate)
{
FHardwareEncoderId EncoderId = HashResolution(Width, Height);
UE::PixelStreaming::FVideoEncoderH264Wrapper* Existing = GetHardwareEncoder(EncoderId);
TOptional<FHardwareEncoderId> Result = EncoderId;
if (Existing == nullptr)
{
// Make AVEncoder frame factory.
TUniquePtr<FEncoderFrameFactory> FrameFactory = MakeUnique<FEncoderFrameFactory>();
FrameFactory->SetResolution(Width, Height);
// Make the encoder config
AVEncoder::FVideoEncoder::FLayerConfig EncoderConfig;
EncoderConfig.Width = Width;
EncoderConfig.Height = Height;
EncoderConfig.MaxFramerate = MaxFramerate;
EncoderConfig.TargetBitrate = TargetBitrate;
EncoderConfig.MaxBitrate = MaxBitrate;
// Make the actual AVEncoder encoder.
const TArray<AVEncoder::FVideoEncoderInfo>& Available = AVEncoder::FVideoEncoderFactory::Get().GetAvailable();
TUniquePtr<AVEncoder::FVideoEncoder> Encoder = AVEncoder::FVideoEncoderFactory::Get().Create(Available[0].ID, FrameFactory->GetOrCreateVideoEncoderInput(EncoderConfig.Width, EncoderConfig.Height), EncoderConfig);
if (Encoder.IsValid())
{
Encoder->SetOnEncodedPacket([EncoderId, this](uint32 InLayerIndex, const AVEncoder::FVideoEncoderInputFrame* InFrame, const AVEncoder::FCodecPacket& InPacket) {
// Note: this is a static method call.
FVideoEncoderH264Wrapper::OnEncodedPacket(EncoderId, this, InLayerIndex, InFrame, InPacket);
});
// Make the hardware encoder wrapper
auto EncoderWrapper = MakeUnique<FVideoEncoderH264Wrapper>(EncoderId, MoveTemp(FrameFactory), MoveTemp(Encoder));
FVideoEncoderH264Wrapper* ReturnValue = EncoderWrapper.Get();
{
FScopeLock Lock(&HardwareEncodersGuard);
// MY ADDED CODE:: Destroy any old encoders
HardwareEncoders.Empty();
// END OF MY ADDED CODE
HardwareEncoders.Add(EncoderId, MoveTemp(EncoderWrapper));
}
return Result;
}
else
{
UE_LOG(LogPixelStreaming, Error, TEXT("Could not create encoder. Check encoder config or perhaps you used up all your HW encoders."));
// We could not make the encoder, so indicate the id was not set successfully.
Result.Reset();
return Result;
}
}
else
{
return Result;
}
}
Is this what you meant? So I’m destroying any active encoders before creating a new one.
Sorry to ask again for your help (but it’s the first time that I’m modifying c++ code ;-)).
Pleaes refer to the code snippet above, which was taken from my modified PixelStreaming.Build.cs. The 2 additional dependencies required are “OpenSSL” and “libOpus”.
emm…
if I use 'r.setres ’ change the screen size
then refresh web page(F5) in 1 second
many times (more than 10?)
the video memory will overflow (just a little project TopDownTemplate)
so 5.1 will fix?
I happen to use a custom resolution on my Unreal app with Pixel Streaming with UE 5.0.3, the pixel streaming module is able to set to correct aspect ratio automatically, although it is not my native custom resolution, but just half, however this is enough for using browser to see the whole UI. I achieved this without any command or any code, just changed the Nvidia Control panel resolution setting to the custom resolution I set in the Unreal app, on my monitor.
