Download

Question about accessing render target images from a dedicated thread

I have been working on an open source UAV simulation (project by Microsoft called AirSim.), where one of the goals is to record images from the drone’s point of view. The drone has a camera that’s erpresented in the code as a 2D capture component, from which the render target is accessed and this needs to be stored to disk as the drone is flying around. The original implementation contained a ReadPixels() call directly from the game thread: which was blocking the game thread until the scene is rendered, and thus causing really low FPS. To get around this, I am trying to implement my own multithreaded version of it, but have been running into a few problems while doing that.

In my approach, I have a dedicated thread for this data logging which is initialized the first time the game thread is run. Once this thread is initialized, the main purpose of this thread is to fire up at a certain frequency and store a screenshot from the camera’s point of view. To access the image data, I could not directly use the RenderTarget->ReadPixels() function because that function as it is, contains a call to FlushRenderingCommands(), which should not be called from a non-game thread. Hence, I rewrote ReadPixels() in my own code without this call: but then I had to be sure the rendering was finished before the image was saved. I tried to use a RenderCommandFence for this, but apparently even that can only be used from a game thread, so this issue is still potentially unsolved. But to get around that, I coded in a sleep function so that the thread has enough time to wait for the rendering. My main problem now follows:

This threaded framework does save the images as desired, but I am still seeing a jerk in the gameplay FPS whenever a screenshot is being stored. To further debug this, I removed the actual save-to-disk part in the code and just left the ReadPixels() call, and a 1 second sleep between trying to access the render target. And sure enough, every second, there is a drop in the FPS in the game play, and I am not sure what’s causing it. My code can be seen at https://github.com/saihv/AirSim/blob/cameralogging/Unreal/Plugins/AirSim/Source/CameraLogger.cpp, but a very simplified version of what I am doing looks like this:



void ReadPixels_Modified()
{
    APIPCamera* cam = GameThread->CameraDirector->getCamera(0);
    USceneCaptureComponent2D* capture = cam->getCaptureComponent(EPIPCameraType::PIP_CAMERA_TYPE_SCENE, true);
    if (capture != nullptr) {
	if (capture->TextureTarget != nullptr) {
		FTextureRenderTargetResource* RenderResource = capture->TextureTarget->GetRenderTargetResource();
                // Create struct for render target data
                
				ENQUEUE_UNIQUE_RENDER_COMMAND_ONEPARAMETER(
					ReadSurfaceCommand,
					FReadSurfaceContext, Context, ReadSurfaceContext,
					{
						RHICmdList.ReadSurfaceData(
							Context.SrcRenderTarget->GetRenderTargetTexture(),
							Context.Rect,
							*Context.OutData,
							Context.Flags
						);
					});
                 }
           }
     }
}

uint32 Thread::Run()
{
	ReadPixels_Modified();
        Thread::Sleep(1);
        // Save image
}



It would be great if someone can shed some light on what I am doing wrong here!

well do you understand the concept of what your doing?

That depends on a vast majority of things.

I hope I am understanding your question right: Mainly I am just looking for an approach where I can access pixel data and save images to disk without blocking the game thread. Which is why I tried to recreate what ReadPixels() does, but without having to block the game thread (inspired by this tutorial https://wiki.unrealengine.com/Render_Target_Lookup). I apologize if my question wasn’t detailed enough, I am still a beginner when it comes to Unreal: I can provide more information about any specific way things are set up if required.

I believe ENQUEUE_UNIQUE_RENDER_COMMAND_ONEPARAMETER enqueues the function to run in the render thread. So the work of reading the pixels isn’t actually running on your new thread. Your hitch is probably coming from slowing down the render thread now.

Looks like ReadSurfaceData() as it is, contains a check to see if it’s being called from a render thread. (IsRenderingThread()): so I assumed it could only be queued into the rendering thread. Are there any alternative approaches that I could take?

Ideally what you want is to simply create a “CPU Read Only” buffer that you can simply queue up a CopyTextureToBuffer type of command each frame along with a render fence. When you go to read the texture, make sure the fence has passed (or wait on it for some small amount of time), and then simply read the data.

That’s the concept at least. How you do that specifically using Unreal I haven’t dug into yet as I haven’t had to do much rendering work with UE4 yet.