How to prevent pausing caused by scene capture?

Development Rendering
, ,
1

Hey community,

I am working on a simulation that has multiple cameras that all need to take scene captures every time a HTTP request is being made, and then send the captured scene back in HTTP response. Those requests come multiple times in a minute, so quite frequently. The issue is the small pause that the scene capture causes which will make the simulation super laggy. I tried handling it in another thread but it doesn’t work either since rendering stuff should not be done in background threads. This is one version I have tried:

***

requestGET() to handle the HTTP request:

bool ASimulationHttpServer::requestGET(const FHttpServerRequest& Request, const FHttpResultCallback& OnComplete)
{
	// reset waitForFence boolean
	myWaitForFence = true;

	if (!IsValid(mySimulationGameState) || !myServerStarted || myServerPort <= 0)
	{
		return false;
	}
	TArray<ACameraPawn*> arrayOfCameras = mySimulationGameState->getCamerasInGroup(myCameraGroup->getMyCameraGroupID());
	for (auto& camera : arrayOfCameras)
	{
		for (auto& queryParam : Request.QueryParams)
		{
			if (queryParam.Key == "id" && FCString::Atoi(*queryParam.Value) == camera->getMyCameraID())
			{
				camera->createAndCaptureScene();

				while(myWaitForFence)
				{
					if (camera->getIsFenceReady())
					{
						myWaitForFence = false;
						const std::vector<unsigned char> resultVector = camera->encodeToJpeg();
						if (!resultVector.empty())
						{
							TUniquePtr<FHttpServerResponse> response = FHttpServerResponse::Create(TEXT(""), TEXT("image/jpeg"));
							if (response != nullptr)
							{
								// Content-Type header
								TArray<FString> contentTypeValues;
								contentTypeValues.Add(TEXT("image/jpeg"));
								response->Headers.Add(TEXT("Content-Type"), contentTypeValues);

								// Add image data to response body
								response->Body.Append(reinterpret_cast<const uint8*>(resultVector.data()), resultVector.size());

								// Set the Content-Length header
								FString contentLengthValue = FString::Printf(TEXT("%d"), response->Body.Num());
								TArray<FString> contentLengthValues;
								contentLengthValues.Add(contentLengthValue);
								response->Headers.Add(TEXT("Content-Length"), contentLengthValues);

								OnComplete(MoveTemp(response));
								return true;
							}
							else
							{
								return false;
							}
						}
					}
				}
				return true;
			}
		}
	}
	return false;
}

Inside it the createAndCaptureScene():

void ACameraPawn::createAndCaptureScene()
{
    // reset readPixelsStarted and imagePixels in the beginning of scene capture
    myReadPixelsStarted = false;
    myImagePixels.Empty();

    if (IsValid(myRenderTarget))
    {
        myRenderTarget->SRGB = true;
        myRenderTarget->TargetGamma = 2.0f;

        mySceneCaptureComponent = FindComponentByClass<USceneCaptureComponent2D>();
        if (IsValid(mySceneCaptureComponent))
        {
            mySceneCaptureComponent->TextureTarget = myRenderTarget;

            // Capture Scene:
            mySceneCaptureComponent->CaptureScene();
            FTextureRenderTargetResource* renderTargetResource = myRenderTarget->GameThread_GetRenderTargetResource();
            readPixels(myImagePixels, renderTargetResource); // reads pixels and saves them in ImagePixels array (diy function)
        }
    }
}

I made a modified version of the ReadPixels() because in the engine version FlushRenderingCommands() is called which stops the thread for a while. So my version looks like:

bool ACameraPawn::readPixels(TArray< FColor >& OutImageData, FTextureRenderTargetResource* renderTargetResourceInstance, FReadSurfaceDataFlags InFlags, FIntRect InRect)
{
    // pretty much a straight copy from UnrealClient::ReadPixels() function to execute the pixel reading without the thread stopping FlushRenderingCommands() function;
    myReadPixelsStarted = true;

    if (InRect == FIntRect(0, 0, 0, 0))
    {
        InRect = FIntRect(0, 0, renderTargetResourceInstance->GetSizeXY().X, renderTargetResourceInstance->GetSizeXY().Y);
    }

    // Read the render target surface data back.	
    struct FReadSurfaceContext
    {
        FRenderTarget* SrcRenderTarget;
        TArray<FColor>* OutData;
        FIntRect Rect;
        FReadSurfaceDataFlags Flags;
    };

    OutImageData.Reset();
    FReadSurfaceContext Context =
    {
        renderTargetResourceInstance,
        &OutImageData,
        InRect,
        FReadSurfaceDataFlags(RCM_UNorm, CubeFace_MAX)
    };

    ENQUEUE_RENDER_COMMAND(ReadSurfaceCommand)(
        [Context](FRHICommandListImmediate& RHICmdList)
        {
            RHICmdList.ReadSurfaceData(
                Context.SrcRenderTarget->GetRenderTargetTexture(),
                Context.Rect,
                *Context.OutData,
                Context.Flags
            );
        });
    myReadPixelFence.BeginFence();

    return OutImageData.Num() > 0;
}

I have tried also AsyncTask:

void ACameraPawn::createAndCaptureScene(const std::function<void(const std::vector<unsigned char>&)>& resultCallback)
{
    if (IsValid(myRenderTarget))
    {
        myRenderTarget->SRGB = true;
        myRenderTarget->TargetGamma = 2.0f;

        mySceneCaptureComponent = FindComponentByClass<USceneCaptureComponent2D>();
        if (IsValid(mySceneCaptureComponent))
        {
            mySceneCaptureComponent->TextureTarget = myRenderTarget;

            // Capture Scene:
            mySceneCaptureComponent->CaptureScene();
            std::lock_guard<std::mutex> Lock(RenderTargetMutex);
            FRenderTarget* renderTargetResource = myRenderTarget->GameThread_GetRenderTargetResource();
            RenderTargetMutex.unlock();

            // add FRenderTarget and callback function in struct that will be passed in async task
            captureTaskParams taskParams;
            taskParams.renderTargetResource = renderTargetResource;
            taskParams.captureResultCallback = resultCallback;

            // create a new async task
            const auto asyncTraceTask = new FAutoDeleteAsyncTask<UAsyncTaskForSceneCapture>(this, taskParams);
            asyncTraceTask->StartBackgroundTask();
        }
    }
}

I also stumbled upon experimental Capture Scene Async class, but I guess that doesn’t work unless I move lot of the functionality to blueprints..? At least trying to do it in C++ failed.

The issue with all these is that it STILL always pauses the game thread. Is there any way to do this without the pausing happening?
Any help is appreciated! If more code or other explanation is needed, please let me know :slight_smile:

2

I am facing this same issue with scene capture stopping the game for a brief moment. Some hints on how to solve this would be greatly appreciated!

Or is it even possible to do asynchronous renders in UE? I believe it should somehow be possible since for example render target can render another image of the same scene without any interruptions: https://www.youtube.com/watch?v=c4YCMK9L9qI Or am I missing something here, is this render target somehow fundamentally different compared to how scene capture works?

3

the pause is the readback, not the capture itself. scene capture renders fine async, but the moment you read pixels back on the game thread the cpu has to wait for the gpu, so every http request costs you a hitch.

the fix is queueing the copy on the render thread and only touching the pixels once a fence says its ready, game thread never blocks. polling one capture per request keeps it smooth even at several requests a minute.

if you dont want to wire fences yourself, Async Render Target Reader, a render target readback plugin for Unreal Engine 5, wraps exactly that as a Read Render Target Pixels Async node with On Success and On Failure: Async Render Target Reader — Non-Blocking GPU Pixel Readback for Blueprints | Fab

4

One likely cause is in requestGET itself: the Http Server router normally calls your handler on the game thread, so the while(myWaitForFence) loop blocks the game thread. The render thread can still progress, but anything that needs the game thread to tick (the capture being kicked off, your fence-ready flag being updated) has to wait for you, so you get a hitch every request.

Suggestions:

  1. Don’t spin. Store the OnComplete callback (with the camera ID), return true right away, and complete the response later, e.g. from a Tick/FTSTicker that checks whether the readback fence is ready. FHttpResultCallback can be called later from any point.
  2. Do the readback asynchronously with FRHIGPUTextureReadback (EnqueueCopy on the render thread, then poll IsReady() each frame and Lock/Unlock), instead of anything that flushes rendering commands.
  3. Do the JPEG encode off the game thread (IImageWrapper compress inside an Async(EAsyncExecution::ThreadPool, …) task) since it’s the CPU-heavy part, and send the result when it finishes.
  4. Make the capture itself cheaper: keep bCaptureEveryFrame and bCaptureOnMovement false, use the smallest resolution you need, and limit the Show Flags on the SceneCapture (disable Lumen/Nanite/post effects you don’t need), or use a lighter capture source such as SceneColor/LDR.
  5. If several cameras are asked in one request, stagger the captures over a few frames.

This should remove the visible stall.