Use Media FVideoEncoder to send backbuffer stream over udp

Hello,

I have an issue trying to decode H264 frames using the UE_5.0\Engine\Source\Runtime\AVEncoder\Private\Decoders\Windows\VideoDecoderH264_Windows.cpp decoder through the FVideoEncoder interface.

For a little more context, i am trying to stream a unreal engine game instance’s backbuffer to another instance. To fetch the backbuffer and use gpu encoding i took code from the pixelstreaming engine plugin. In the current process i send the compressed const AVEncoder::FCodecPacket& Packet directly to a decoder function in the same program to avoid any network related issues.

While Unreal Engine implemented a Nvidia NVENC interface (and for amd), as far as I have checked none exist for NVDEC to decode the stream. Thus I decided to use the windows h264 video decoder instance that supports hardware decoding.

I implemented everything assuming a directx11 rendering context and here is the function receiving the decoded FVideoDecoderOutput (a buffer containing the pointer of a shared directx11-2d-texture). However no matter how hard i try i cannot manage to show the decoded image on the target texture (texture of a material in myh scene).

Do you guys have an idea what i am maybe doing wrong ?

following this link https://forums.unrealengine.com/t/media-framework-roadmap/25203, I am trying to copy the given temporary frame like told : “frame buffer copies, […] from the temporary buffer into the render thread.”

Thanks a lot and have a wonderful day !

	Decoder->SetOnDecodedFrame([this](const AVEncoder::FVideoDecoderOutput* InDecodedFrame) {
		// Fetch Resulting texture from buffer
		const FVideoDecoderAllocFrameBufferResult* ResultBuffer = InDecodedFrame->GetAllocatedBuffer();
		ID3D11Texture2D** SourceSharedTextureBuffer = static_cast<ID3D11Texture2D**>(ResultBuffer->AllocatedBuffer);
		ID3D11Texture2D* SourceSharedTexture = *SourceSharedTextureBuffer;
		TRefCountPtr<ID3D11Device> SourceSharedTextureDevice;

		// Get destination texture
		ID3D11Texture2D* DxTargetTexture = static_cast<ID3D11Texture2D*>(
			StreamingTexture->GetResource()->TextureRHI->GetTexture2D()->GetNativeResource());

		// Get dx11 device and context
		ID3D11DeviceContext* Dx11Context = nullptr;
		ID3D11Device* Dx11Device = static_cast<ID3D11Device*>(GDynamicRHI->RHIGetNativeDevice());
		Dx11Device->GetImmediateContext(&Dx11Context);

		// Get from source shared texture the tex handle
		TRefCountPtr<IDXGIResource> POtherResource;
		HRESULT Result = S_OK;
		if (FAILED(Result = SourceSharedTexture->QueryInterface(__uuidof(IDXGIResource), (void**)POtherResource.GetInitReference()))) {
			UE_LOG(LogTemp, Error, TEXT("SourceSharedtexture->QueryInterface failed with 0x%08x (%s)"), Result, *GetComErrorDescription(Result));
			return;
		}
		HANDLE SharedHandle;
		if (FAILED(Result = POtherResource->GetSharedHandle(&SharedHandle))) {
			UE_LOG(LogTemp, Error, TEXT("POtherResource->GetSharedHandle failed with 0x%08x (%s)"), Result, *GetComErrorDescription(Result));
			return;
		}
		TRefCountPtr<ID3D11Texture2D> SourceTexture = nullptr;
		if (FAILED(Result = Dx11Device->OpenSharedResource(SharedHandle, __uuidof(ID3D11Texture2D), (void**)SourceTexture.GetInitReference()))) {
			UE_LOG(LogTemp, Error, TEXT("Dx11Device->OpenSharedResource failed with 0x%08x (%s)"), Result, *GetComErrorDescription(Result));
			return;
		}

		TRefCountPtr<IDXGIKeyedMutex> KeyedMutex;
		if (FAILED(Result = SourceTexture->QueryInterface(_uuidof(IDXGIKeyedMutex), (void**)KeyedMutex.GetInitReference()))) {
			UE_LOG(LogTemp, Error, TEXT("SourceTexture->QueryInterface failed with 0x%08x (%s)"), Result, *GetComErrorDescription(Result));
			return;
		}
		if (KeyedMutex) {
			Result = KeyedMutex->AcquireSync(1, 5000);
			if (Result == WAIT_OBJECT_0) {
				Dx11Context->CopyResource(DxTargetTexture, SourceTexture);
			} else {
				// Unable to get handle
			}
			Result = KeyedMutex->ReleaseSync(0);
			if (Result == WAIT_OBJECT_0) {
				UE_LOG(LogTemp, Warning, TEXT("Success copying resource !"));
			} else {
				UE_LOG(LogTemp, Warning, TEXT("Failed to copy resource !"));
			}
		}

		// Make sure texture is updated before giving access to the sample in the rendering thread.
		Dx11Context->Flush();

		ResultBuffer->ReleaseCallback.Execute(ResultBuffer->CallbackValue, ResultBuffer->AllocatedBuffer);
	});