Copy backbuffer to texture

[Developer_A]

Hello! I'm implementing some features and i need to copy backbuffer to rendertarget (and next resize/blur it, but it doesn't matter, i just need a copy of backbuffer). Of course I'm looking for a solution to do that with GPU/GPU Ram only (without extra coping texture data to RAM and the usage of CPU or blocking game thread). I've found event that is fired from rendering pipeline and succesfully subscribed that:

FDelegateHandle OnBackBufferReadyToPresent = FSlateApplication::Get().GetRenderer()->OnBackBufferReadyToPresent().AddUObject(this, &UCachedFrameBufferComponent::OnBackBufferReady_RenderThread);

It seems that I'm able to get the access to backbuffer here and copy it to created render target with copy texture command, but that doesn't work (back buffer contains data that i need, but i cannot copy that to my rendertarget):

void UCachedFrameBufferComponent::OnBackBufferReady_RenderThread(SWindow& SlateWindow, const FTexture2DRHIRef& BackBuffer)
{
	ensure(IsInRenderingThread());

	FRHICommandListImmediate& RHICmdList = GRHICommandList.GetImmediateCommandList();
	FRHITexture2D* CachedTexture = CachedFrameBuffer->Resource->TextureRHI->GetTexture2D();

	FRHICopyTextureInfo CopyInfo;
	RHICmdList.CopyTexture(CachedTexture, BackBuffer, CopyInfo);
}

CachedFrameBuffer is blank. If it makes sense I create CachedFrameBuffer from editor and tried different resolutions (including back buffer resolution), formats and etc. Moreover, I tried to create it manually and this doesn't affect too.

UTextureRenderTarget2D* CachedFrameBuffer = UKismetRenderingLibrary::CreateRenderTarget2D(GetOwner()->GetWorld(), ViewportSize.X * ScreenPercentage, ViewportSize.Y * ScreenPercentage, ETextureRenderTargetFormat::RTF_RGBA8);

After investigation of engine's sources I find more progressive way to do that:

void UCachedFrameBufferComponent::CopyBackBuffer(const FTexture2DRHIRef& BackBuffer, const FTexture2DRHIRef& CachedFrame)
{
	IRendererModule* RendererModule = &FModuleManager::GetModuleChecked<IRendererModule>("Renderer");
	FRHICommandListImmediate& RHICmdList = FRHICommandListExecutor::GetImmediateCommandList();

	if (BackBuffer->GetFormat() == CachedFrame->GetFormat() &&
		BackBuffer->GetSizeXY() == CachedFrame->GetSizeXY())
	{
		RHICmdList.CopyToResolveTarget(BackBuffer, CachedFrame, FResolveParams{});
	}
	else // Texture format mismatch, use a shader to do the copy.
	{
		// #todo-renderpasses there's no explicit resolve here? Do we need one?
		FRHIRenderPassInfo RPInfo(CachedFrame, ERenderTargetActions::Load_Store);
		RHICmdList.BeginRenderPass(RPInfo, TEXT("CopyBackbuffer"));
		{
			RHICmdList.SetViewport(0, 0, 0.0f, CachedFrame->GetSizeX(), CachedFrame->GetSizeY(), 1.0f);

			FGraphicsPipelineStateInitializer GraphicsPSOInit;
			RHICmdList.ApplyCachedRenderTargets(GraphicsPSOInit);
			GraphicsPSOInit.BlendState = TStaticBlendState<>::GetRHI();
			GraphicsPSOInit.RasterizerState = TStaticRasterizerState<>::GetRHI();
			GraphicsPSOInit.DepthStencilState = TStaticDepthStencilState<false, CF_Always>::GetRHI();

			TShaderMap<FGlobalShaderType>* ShaderMap = GetGlobalShaderMap(GMaxRHIFeatureLevel);
			TShaderMapRef<FScreenVS> VertexShader(ShaderMap);
			TShaderMapRef<FScreenPS> PixelShader(ShaderMap);

			GraphicsPSOInit.BoundShaderState.VertexDeclarationRHI = GFilterVertexDeclaration.VertexDeclarationRHI;
			GraphicsPSOInit.BoundShaderState.VertexShaderRHI = GETSAFERHISHADER_VERTEX(*VertexShader);
			GraphicsPSOInit.BoundShaderState.PixelShaderRHI = GETSAFERHISHADER_PIXEL(*PixelShader);
			GraphicsPSOInit.PrimitiveType = PT_TriangleList;

			SetGraphicsPipelineState(RHICmdList, GraphicsPSOInit);

			if (CachedFrame->GetSizeX() != BackBuffer->GetSizeX() || CachedFrame->GetSizeY() != BackBuffer->GetSizeY())
			{
				PixelShader->SetParameters(RHICmdList, TStaticSamplerState<SF_Bilinear>::GetRHI(), BackBuffer);
			}
			else
			{
				PixelShader->SetParameters(RHICmdList, TStaticSamplerState<SF_Point>::GetRHI(), BackBuffer);
			}

			RendererModule->DrawRectangle(
				RHICmdList,
				0, 0,									// Dest X, Y
				CachedFrame->GetSizeX(),			// Dest Width
				CachedFrame->GetSizeY(),			// Dest Height
				0, 0,									// Source U, V
				1, 1,									// Source USize, VSize
				CachedFrame->GetSizeXY(),		// Target buffer size
				FIntPoint(1, 1),						// Source texture size
				*VertexShader,
				EDRF_Default);
		}
		RHICmdList.EndRenderPass();
	}
}

The approach is quite obvious: bind render target and render back buffer texture to it. But it doesn't work, my render target still is blank.

Maybe I'm missing something during operating with RHI texture objects instead of Unreal's UTextureRenderTarget2D*?

FRHITexture2D* CachedTexture = CachedFrameBuffer->Resource->TextureRHI->GetTexture2D();

Also I tried to wrap RHI commands with ENQUEUE_RENDER_COMMAND macros,(btw should i do this for code that is run in rendering thread in BackBufferReady callback? I think not, anyway that doesn't help, render target still is blank).

So, please help me or share the info where I should dig in engine to achieve that I need.

Thank you!

[Developer_A]

Also, If it makes sense I use the latest version of Engine: 4.24.1

[Developer_A]

Just carefully redo that in test project, and that works. Seems there was a silly misprint/mistake, probably in render texture initialization. The correct code is next:

void ACacheFrameBufferActor::BeginPlay()
{
    Super::BeginPlay();

    OnBackBufferReadyToPresent = FSlateApplication::Get().GetRenderer()->OnBackBufferReadyToPresent().AddUObject(this, &ACacheFrameBufferActor::OnBackBufferReady_RenderThread);
    bIsCalculatedCaptureRect = CalculateCaptureRect(CaptureRect);

    CaptureFrameTexture = UKismetRenderingLibrary::CreateRenderTarget2D(GetWorld(), 128, 128);
    MaterialInstanceDynamic->SetTextureParameterValue(FName("Texture"), CaptureFrameTexture);
}

bool ACacheFrameBufferActor::CalculateCaptureRect(FIntRect& CaptureRect) const
{

#if WITH_EDITOR

    TSharedPtr<FSceneViewport> SceneViewport;

    if (GIsEditor)
    {
        for (const FWorldContext& Context : GEngine->GetWorldContexts())
        {
            if (Context.WorldType == EWorldType::PIE)
            {
                FSlatePlayInEditorInfo* SlatePlayInEditorSession = GEditor->SlatePlayInEditorMap.Find(Context.ContextHandle);
                if (SlatePlayInEditorSession)
                {
                    if (SlatePlayInEditorSession->DestinationSlateViewport.IsValid())
                    {
                        TSharedPtr<IAssetViewport> DestinationLevelViewport = SlatePlayInEditorSession->DestinationSlateViewport.Pin();

                        SceneViewport = DestinationLevelViewport->GetSharedActiveViewport();
                    }
                    else if (SlatePlayInEditorSession->SlatePlayInEditorWindowViewport.IsValid())
                    {
                        SceneViewport = SlatePlayInEditorSession->SlatePlayInEditorWindowViewport;
                    }
                }
            }
        }
    }

    if (SceneViewport == nullptr || !SceneViewport.IsValid())
    {
        return false;
    }

    CaptureRect = FIntRect(0, 0, SceneViewport->GetSize().X, SceneViewport->GetSize().Y);
    FIntPoint WindowSize(0, 0);

    // Set up the capture rectangle
    TSharedPtr<SViewport> ViewportWidget = SceneViewport->GetViewportWidget().Pin();
    if (ViewportWidget.IsValid())
    {
        TSharedPtr<SWindow> Window = FSlateApplication::Get().FindWidgetWindow(ViewportWidget.ToSharedRef());
        if (Window.IsValid())
        {
            //TargetWindowPtr = Window.Get();
            FGeometry InnerWindowGeometry = Window->GetWindowGeometryInWindow();

            // Find the widget path relative to the window
            FArrangedChildren JustWindow(EVisibility::Visible);
            JustWindow.AddWidget(FArrangedWidget(Window.ToSharedRef(), InnerWindowGeometry));

            FWidgetPath WidgetPath(Window.ToSharedRef(), JustWindow);
            if (WidgetPath.ExtendPathTo(FWidgetMatcher(ViewportWidget.ToSharedRef()), EVisibility::Visible))
            {
                FArrangedWidget ArrangedWidget = WidgetPath.FindArrangedWidget(ViewportWidget.ToSharedRef()).Get(FArrangedWidget::GetNullWidget());

                FVector2D Position = ArrangedWidget.Geometry.GetAbsolutePosition();
                FVector2D Size = ArrangedWidget.Geometry.GetAbsoluteSize();

                CaptureRect = FIntRect(
                    Position.X,
                    Position.Y,
                    Position.X + Size.X,
                    Position.Y + Size.Y);

                FVector2D AbsoluteSize = InnerWindowGeometry.GetAbsoluteSize();
                WindowSize = FIntPoint(AbsoluteSize.X, AbsoluteSize.Y);

            }
        }
    }

    return true;

#endif

    return false;
}

void ACacheFrameBufferActor::OnBackBufferReady_RenderThread(SWindow& SlateWindow, const FTexture2DRHIRef& BackBuffer)
{
    ensure(IsInRenderingThread());

    if (CaptureFrameTexture != nullptr)
    {
        FRHICommandListImmediate& RHICmdList = GRHICommandList.GetImmediateCommandList();
        FRHITexture2D* CachedTexture = CaptureFrameTexture->Resource->TextureRHI->GetTexture2D();

        FBox2D Region = FBox2D(FVector2D::ZeroVector, FVector2D::UnitVector);

        if (bIsCalculatedCaptureRect)
        {
            float U = float(CaptureRect.Min.X) / float(BackBuffer->GetSizeX());
            float V = float(CaptureRect.Min.Y) / float(BackBuffer->GetSizeY());
            float SizeU = float(CaptureRect.Max.X) / float(BackBuffer->GetSizeX());
            float SizeV = float(CaptureRect.Max.Y) / float(BackBuffer->GetSizeY());

            Region = FBox2D(FVector2D(U,V), FVector2D(SizeU, SizeV));
        }

        CopyBackBuffer(BackBuffer, CachedTexture, Region);
    }
}

void ACacheFrameBufferActor::CopyBackBuffer(const FTexture2DRHIRef& BackBuffer, const FTexture2DRHIRef& CachedFrame, const FBox2D& BackBufferRegion)
{
    IRendererModule* RendererModule = &FModuleManager::GetModuleChecked<IRendererModule>("Renderer");
    FRHICommandListImmediate& RHICmdList = FRHICommandListExecutor::GetImmediateCommandList();

    if (BackBuffer->GetFormat() == CachedFrame->GetFormat() &&
        BackBuffer->GetSizeXY() == CachedFrame->GetSizeXY())
    {
        RHICmdList.CopyToResolveTarget(BackBuffer, CachedFrame, FResolveParams{});
    }
    else // Texture format mismatch, use a shader to do the copy.
    {
        // #todo-renderpasses there's no explicit resolve here? Do we need one?
        FRHIRenderPassInfo RPInfo(CachedFrame, ERenderTargetActions::Load_Store);
        RHICmdList.BeginRenderPass(RPInfo, TEXT("CopyBackbuffer"));
        {
            RHICmdList.SetViewport(0, 0, 0.0f, CachedFrame->GetSizeX(), CachedFrame->GetSizeY(), 1.0f);

            FGraphicsPipelineStateInitializer GraphicsPSOInit;
            RHICmdList.ApplyCachedRenderTargets(GraphicsPSOInit);
            GraphicsPSOInit.BlendState = TStaticBlendState<>::GetRHI();
            GraphicsPSOInit.RasterizerState = TStaticRasterizerState<>::GetRHI();
            GraphicsPSOInit.DepthStencilState = TStaticDepthStencilState<false, CF_Always>::GetRHI();

            TShaderMap<FGlobalShaderType>* ShaderMap = GetGlobalShaderMap(GMaxRHIFeatureLevel);
            TShaderMapRef<FScreenVS> VertexShader(ShaderMap);
            TShaderMapRef<FScreenPS> PixelShader(ShaderMap);

            GraphicsPSOInit.BoundShaderState.VertexDeclarationRHI = GFilterVertexDeclaration.VertexDeclarationRHI;
            GraphicsPSOInit.BoundShaderState.VertexShaderRHI = GETSAFERHISHADER_VERTEX(*VertexShader);
            GraphicsPSOInit.BoundShaderState.PixelShaderRHI = GETSAFERHISHADER_PIXEL(*PixelShader);
            GraphicsPSOInit.PrimitiveType = PT_TriangleList;

            SetGraphicsPipelineState(RHICmdList, GraphicsPSOInit);

            if (CachedFrame->GetSizeX() != BackBuffer->GetSizeX() || CachedFrame->GetSizeY() != BackBuffer->GetSizeY())
            {
                PixelShader->SetParameters(RHICmdList, TStaticSamplerState<SF_Bilinear>::GetRHI(), BackBuffer);
            }
            else
            {
                PixelShader->SetParameters(RHICmdList, TStaticSamplerState<SF_Point>::GetRHI(), BackBuffer);
            }

            FVector2D Size = BackBufferRegion.GetSize();

            RendererModule->DrawRectangle(
                RHICmdList,
                0, 0,                                                // Dest X, Y
                CachedFrame->GetSizeX(),                            // Dest Width
                CachedFrame->GetSizeY(),                            // Dest Height
                BackBufferRegion.Min.X, BackBufferRegion.Min.Y,
                Size.X, Size.Y,                                        // Source USize, VSize
                CachedFrame->GetSizeXY(),                            // Target buffer size
                FIntPoint(1, 1),                                    // Source texture size
                *VertexShader,
                EDRF_Default);
        }
        RHICmdList.EndRenderPass();
    }
}