Hey community,
I am working on a simulation that has multiple cameras that all need to take scene captures every time a HTTP request is being made, and then send the captured scene back in HTTP response. Those requests come multiple times in a minute, so quite frequently. The issue is the small pause that the scene capture causes which will make the simulation super laggy. I tried handling it in another thread but it doesn’t work either since rendering stuff should not be done in background threads. This is one version I have tried:
***
requestGET() to handle the HTTP request:
bool ASimulationHttpServer::requestGET(const FHttpServerRequest& Request, const FHttpResultCallback& OnComplete)
{
// reset waitForFence boolean
myWaitForFence = true;
if (!IsValid(mySimulationGameState) || !myServerStarted || myServerPort <= 0)
{
return false;
}
TArray<ACameraPawn*> arrayOfCameras = mySimulationGameState->getCamerasInGroup(myCameraGroup->getMyCameraGroupID());
for (auto& camera : arrayOfCameras)
{
for (auto& queryParam : Request.QueryParams)
{
if (queryParam.Key == "id" && FCString::Atoi(*queryParam.Value) == camera->getMyCameraID())
{
camera->createAndCaptureScene();
while(myWaitForFence)
{
if (camera->getIsFenceReady())
{
myWaitForFence = false;
const std::vector<unsigned char> resultVector = camera->encodeToJpeg();
if (!resultVector.empty())
{
TUniquePtr<FHttpServerResponse> response = FHttpServerResponse::Create(TEXT(""), TEXT("image/jpeg"));
if (response != nullptr)
{
// Content-Type header
TArray<FString> contentTypeValues;
contentTypeValues.Add(TEXT("image/jpeg"));
response->Headers.Add(TEXT("Content-Type"), contentTypeValues);
// Add image data to response body
response->Body.Append(reinterpret_cast<const uint8*>(resultVector.data()), resultVector.size());
// Set the Content-Length header
FString contentLengthValue = FString::Printf(TEXT("%d"), response->Body.Num());
TArray<FString> contentLengthValues;
contentLengthValues.Add(contentLengthValue);
response->Headers.Add(TEXT("Content-Length"), contentLengthValues);
OnComplete(MoveTemp(response));
return true;
}
else
{
return false;
}
}
}
}
return true;
}
}
}
return false;
}
Inside it the createAndCaptureScene():
void ACameraPawn::createAndCaptureScene()
{
// reset readPixelsStarted and imagePixels in the beginning of scene capture
myReadPixelsStarted = false;
myImagePixels.Empty();
if (IsValid(myRenderTarget))
{
myRenderTarget->SRGB = true;
myRenderTarget->TargetGamma = 2.0f;
mySceneCaptureComponent = FindComponentByClass<USceneCaptureComponent2D>();
if (IsValid(mySceneCaptureComponent))
{
mySceneCaptureComponent->TextureTarget = myRenderTarget;
// Capture Scene:
mySceneCaptureComponent->CaptureScene();
FTextureRenderTargetResource* renderTargetResource = myRenderTarget->GameThread_GetRenderTargetResource();
readPixels(myImagePixels, renderTargetResource); // reads pixels and saves them in ImagePixels array (diy function)
}
}
}
I made a modified version of the ReadPixels() because in the engine version FlushRenderingCommands() is called which stops the thread for a while. So my version looks like:
bool ACameraPawn::readPixels(TArray< FColor >& OutImageData, FTextureRenderTargetResource* renderTargetResourceInstance, FReadSurfaceDataFlags InFlags, FIntRect InRect)
{
// pretty much a straight copy from UnrealClient::ReadPixels() function to execute the pixel reading without the thread stopping FlushRenderingCommands() function;
myReadPixelsStarted = true;
if (InRect == FIntRect(0, 0, 0, 0))
{
InRect = FIntRect(0, 0, renderTargetResourceInstance->GetSizeXY().X, renderTargetResourceInstance->GetSizeXY().Y);
}
// Read the render target surface data back.
struct FReadSurfaceContext
{
FRenderTarget* SrcRenderTarget;
TArray<FColor>* OutData;
FIntRect Rect;
FReadSurfaceDataFlags Flags;
};
OutImageData.Reset();
FReadSurfaceContext Context =
{
renderTargetResourceInstance,
&OutImageData,
InRect,
FReadSurfaceDataFlags(RCM_UNorm, CubeFace_MAX)
};
ENQUEUE_RENDER_COMMAND(ReadSurfaceCommand)(
[Context](FRHICommandListImmediate& RHICmdList)
{
RHICmdList.ReadSurfaceData(
Context.SrcRenderTarget->GetRenderTargetTexture(),
Context.Rect,
*Context.OutData,
Context.Flags
);
});
myReadPixelFence.BeginFence();
return OutImageData.Num() > 0;
}
I have tried also AsyncTask:
void ACameraPawn::createAndCaptureScene(const std::function<void(const std::vector<unsigned char>&)>& resultCallback)
{
if (IsValid(myRenderTarget))
{
myRenderTarget->SRGB = true;
myRenderTarget->TargetGamma = 2.0f;
mySceneCaptureComponent = FindComponentByClass<USceneCaptureComponent2D>();
if (IsValid(mySceneCaptureComponent))
{
mySceneCaptureComponent->TextureTarget = myRenderTarget;
// Capture Scene:
mySceneCaptureComponent->CaptureScene();
std::lock_guard<std::mutex> Lock(RenderTargetMutex);
FRenderTarget* renderTargetResource = myRenderTarget->GameThread_GetRenderTargetResource();
RenderTargetMutex.unlock();
// add FRenderTarget and callback function in struct that will be passed in async task
captureTaskParams taskParams;
taskParams.renderTargetResource = renderTargetResource;
taskParams.captureResultCallback = resultCallback;
// create a new async task
const auto asyncTraceTask = new FAutoDeleteAsyncTask<UAsyncTaskForSceneCapture>(this, taskParams);
asyncTraceTask->StartBackgroundTask();
}
}
}
I also stumbled upon experimental Capture Scene Async class, but I guess that doesn’t work unless I move lot of the functionality to blueprints…? At least trying to do it in C++ failed.
The issue with all these is that it STILL always pauses the game thread. Is there any way to do this without the pausing happening?
Any help is appreciated! If more code or other explanation is needed, please let me know 