Changing [video] resolution for PixelStreaming

Welcome to the great “Video Resolution for Pixel Streaming” topic :).

The ongoing challenge is to have a video streaming resolution that is in line with (or equal to) the rendered game resolution.

There are quite a few forum posts around this topic. Here you will find them summarized so we can hopefully all use one resource.

UE Version Summary

  • 4.26:
    Everything works fine. Video resolution is updated automatically after r.setRes command
  • 4.27:
    Video resolution is captured on creation of webRtc Player and not updated afterwards.
    Forum Posts on the topic: (link1) (link2) (link3)
    Automatic or manual updating can be re-enabled or with commands:
Automatic:
PixelStreaming.WebRTC.DisableResolutionChange false
PixelStreaming.Capturer.UseBackBufferSize true
Manual:
PixelStreaming.WebRTC.DisableResolutionChange false
PixelStreaming.Capturer.UseBackBufferSize false
PixelStreaming.Capturer.CaptureSize *target resolution*
  • 5.0:
    Video resolution is captured on creation of webRtc Player and not updated afterwards. The commands to enable it (in 4.27) have been removed.
    See (link) (section ‘Pixel Streaming’) for details on the updates in UE5.0
    Forum Posts on the topic (link1) (link2)

UE5-0 Workarounds
Currently there are no known workarounds for the issue, short of setting a resolution, refreshing the page and loging in again. We hope that a discussion in this forum topic with bring us a solution.

UE5-0 Solutions to explore
So far the most promising solution seems

Running the following functions in sequence (from app.js) does results in the player restarting with a fresh stream:

webRtcPlayerObj.close();
connect();
playVideoStream();

But…!

  • This doesn’t work if you change resolution before running the functions. No video appears on the screen.
  • Running the functions repeatedly results in severe performance degradation. Refreshing the browser resolved this.
  • adding ws.close() to the sequence forces the user to click on the screen before video resumes, but strangely also shows 2 parallel streams in the stats section of the browser

I’m hoping that we can come up with a working solution together!

I suspect that the Javascript sequence need some adjustments, as it may not be closing the old stream as expected. It is possible to have the streamer send a command to client to notify when the resolution change has been completed, only then will the client run the sequence.

Looks like there is one important thing that needs to be added to the plugin, that is to destroy the existing NvEnc encoding sessions as soon as possible and clean up everything related to it, so that they don’t take up processing power, and to avoid hitting the session limit if you are using Geforce instead of Quadro line.

Yes, I’m pretty sure it needs some adjustments… Unfortunately what I’m trying here is waaay over my head so it’s more like educated guessing than programming…

Any help would be greatly appreciated.

I’m only going base on my use case, which may not be applicable to yours. I’m also creating a custom web frontend which means it is very different from the sample provided by Epic. Basically what I do is after the client sent the resolution change command, it calls ws.close(), wait 100 msec, call connect(). I do not call neither webRtcPlayerObj.close() nor playVideoStream() directly. For the case of the sample app.js, you may want to set shouldShowPlayOverlay = false, make sure that inside ws.onclose handler you prevent the timeout call to the start function.

On the UE side, I know that for my app it is always 1 user <-> 1 instance of UE, which means I can eagerly destroy the old hardware encoders by inserting a line into UE::PixelStreaming::FVideoEncoderFactory::GetOrCreateHardwareEncoder in EncoderFactory.cpp like this:

{
	FScopeLock Lock(&HardwareEncodersGuard);
	HardwareEncoders.Empty();
}

So far this setup works alright for me, there is a delay when the reconnection is going on, but I can easily cover it up with a loading screen or something.

I just feel like the way Epic built Pixel Streaming is overkill, too complicated for what it should be. At some point I probably will have to build my own simplified plugin that only do what I need and no more.

Hey guys!

I’m here to help shed a bit of light on the issue,

So the lack of stream resolution changing in Pixel Streaming in UE 5.0 - 5.0.3 is a known regression. With the changes needed to fix the issue, it couldn’t quite make it into the hot-fix release, much to our chagrin.

Good news however, is that it is already fixed in the ue5-main branch on Github!

Sorry for the inconvenience, but know that we’re aware and tackling these issues.

That clarifies a lot. Thanks a lot for the update!

I tested the plugin on ue5-main, seems to work alright, except for one thing, which is there is often a delay before the resolution is updated, and sometimes it may get stuck in the old resolution until I force it to change by resizing the window again.

I got it to work following your notes (thanks for that!). Everything seems fine… most of the time.

However, in some cases I’m having issues setting up the video stream and it just remains stuck at WebRTC data channel connected... waiting for video

I found that when this happens it’s because:

a. In WebRtcPlayer.js, this event never gets called:

    video.addEventListener('loadedmetadata', function(e){
        if(self.onVideoInitialised){
            self.onVideoInitialised();
        }
    }, true);

b. In app.js this event never gets called:

webRtcPlayerObj.onVideoInitialised = function() {
	console.log("webRtcPlayerObj.onVideoInitialised");
    if (ws && ws.readyState === WS_OPEN_STATE) {
        if (shouldShowPlayOverlay) {
            showPlayOverlay();
            resizePlayerStyle();
        }
        else {
			resizePlayerStyle();
            playVideoStream();
        }
    }
};

Are you seeing similar things? On my side I can reproduce this quite consistently by changing to a resoltution of 2048x2048 (no idea why this number… higher is fine, lower is fine…)

Edit:
When this happens, the UE application continues on running, but the PixelStreaming part seems to hang. Refreshing the browser doesn’t help, only restarting the UE application.

Edit2:
I’m also seeing crashes when closing the UE application that say

Assertion failed: InBuffer [File:D:\build++UE5\Sync\Engine\Plugins\Media\HardwareEncoders\Source\EncoderNVENC\Private\NVENC_EncoderH264.cpp] [Line: 827] Cannot destroy buffer in NVENC - buffer was nullptr.

You can refer to my earlier message about modifying C++ code inside the Pixel Streaming plugin. That is required, in order to shutdown the unused hardware video encoder instances, particularly important if you are not running the UE game instance on Quadro level graphics card. If you do not want to recompile the entire engine source for this purpose, you can copy the plugin folder from engine source and put into your project’s plugin folder, use that plugin instead of the pre-bundled plugin.

@trqan97 I’m trying to follow your instructions (UE5.0.3), but I keep on running into compiling issues:

LNK2019 Unresolved external symbol … webrtc.lib …

How do I resolve this?

Then, I modified below function in EncoderFactory.ccp like this:

TOptional<UE::PixelStreaming::FVideoEncoderFactory::FHardwareEncoderId> UE::PixelStreaming::FVideoEncoderFactory::GetOrCreateHardwareEncoder(int Width, int Height, int MaxBitrate, int TargetBitrate, int MaxFramerate)
{
	FHardwareEncoderId EncoderId = HashResolution(Width, Height);
	UE::PixelStreaming::FVideoEncoderH264Wrapper* Existing = GetHardwareEncoder(EncoderId);

	TOptional<FHardwareEncoderId> Result = EncoderId;

	if (Existing == nullptr)
	{
		// Make AVEncoder frame factory.
		TUniquePtr<FEncoderFrameFactory> FrameFactory = MakeUnique<FEncoderFrameFactory>();
		FrameFactory->SetResolution(Width, Height);

		// Make the encoder config
		AVEncoder::FVideoEncoder::FLayerConfig EncoderConfig;
		EncoderConfig.Width = Width;
		EncoderConfig.Height = Height;
		EncoderConfig.MaxFramerate = MaxFramerate;
		EncoderConfig.TargetBitrate = TargetBitrate;
		EncoderConfig.MaxBitrate = MaxBitrate;

		// Make the actual AVEncoder encoder.
		const TArray<AVEncoder::FVideoEncoderInfo>& Available = AVEncoder::FVideoEncoderFactory::Get().GetAvailable();
		TUniquePtr<AVEncoder::FVideoEncoder> Encoder = AVEncoder::FVideoEncoderFactory::Get().Create(Available[0].ID, FrameFactory->GetOrCreateVideoEncoderInput(EncoderConfig.Width, EncoderConfig.Height), EncoderConfig);
		if (Encoder.IsValid())
		{
			Encoder->SetOnEncodedPacket([EncoderId, this](uint32 InLayerIndex, const AVEncoder::FVideoEncoderInputFrame* InFrame, const AVEncoder::FCodecPacket& InPacket) {
				// Note: this is a static method call.
				FVideoEncoderH264Wrapper::OnEncodedPacket(EncoderId, this, InLayerIndex, InFrame, InPacket);
			});

			// Make the hardware encoder wrapper
			auto EncoderWrapper = MakeUnique<FVideoEncoderH264Wrapper>(EncoderId, MoveTemp(FrameFactory), MoveTemp(Encoder));
			FVideoEncoderH264Wrapper* ReturnValue = EncoderWrapper.Get();
			{
				FScopeLock Lock(&HardwareEncodersGuard);
				// MY ADDED CODE:: Destroy any old encoders
				HardwareEncoders.Empty();
				// END OF MY ADDED CODE
				HardwareEncoders.Add(EncoderId, MoveTemp(EncoderWrapper));
			}

			return Result;
		}
		else
		{
			UE_LOG(LogPixelStreaming, Error, TEXT("Could not create encoder. Check encoder config or perhaps you used up all your HW encoders."));
			// We could not make the encoder, so indicate the id was not set successfully.
			Result.Reset();
			return Result;
		}
	}
	else
	{
		return Result;
	}
}

Is this what you meant? So I’m destroying any active encoders before creating a new one.

Sorry to ask again for your help (but it’s the first time that I’m modifying c++ code ;-)).

So my main issue appears to be

LNK2019	unresolved external symbol somethingreference MyProject D:\Unreal Projects\Virtuality_5\Intermediate\ProjectFiles\webrtc.lib(openssl_identity.obj)

I tried to put some webrtc.lib file in the suggested directory, but that doesn’t make any difference.

PrivateDependencyModuleNames.AddRange(new string[]
			{
				"ApplicationCore",
				"Core",
				"CoreUObject",
				"Engine",
				"InputCore",
				"Json",
				"RenderCore",
				"RHI",
				"RHICore",
				"Slate",
				"SlateCore",
				"AudioMixer",
				"WebRTC",
				"WebSockets",
				"Sockets",
				"MediaUtils",
				"DeveloperSettings",
				"AVEncoder",
				"OpenSSL",
			});

			AddEngineThirdPartyPrivateStaticDependencies(Target, "libOpus");

Pleaes refer to the code snippet above, which was taken from my modified PixelStreaming.Build.cs. The 2 additional dependencies required are “OpenSSL” and “libOpus”.

1 Like

Thank you for this.

Since I’m out on holiday I don’t have access to my workstation to try this out - but as soon as I’m back I’ll try adding this.