I have been working on an open source UAV simulation (project by Microsoft called AirSim.), where one of the goals is to record images from the drone’s point of view. The drone has a camera that’s erpresented in the code as a 2D capture component, from which the render target is accessed and this needs to be stored to disk as the drone is flying around. The original implementation contained a ReadPixels() call directly from the game thread: which was blocking the game thread until the scene is rendered, and thus causing really low FPS. To get around this, I am trying to implement my own multithreaded version of it, but have been running into a few problems while doing that.
In my approach, I have a dedicated thread for this data logging which is initialized the first time the game thread is run. Once this thread is initialized, the main purpose of this thread is to fire up at a certain frequency and store a screenshot from the camera’s point of view. To access the image data, I could not directly use the RenderTarget->ReadPixels() function because that function as it is, contains a call to FlushRenderingCommands(), which should not be called from a non-game thread. Hence, I rewrote ReadPixels() in my own code without this call: but then I had to be sure the rendering was finished before the image was saved. I tried to use a RenderCommandFence for this, but apparently even that can only be used from a game thread, so this issue is still potentially unsolved. But to get around that, I coded in a sleep function so that the thread has enough time to wait for the rendering. My main problem now follows:
This threaded framework does save the images as desired, but I am still seeing a jerk in the gameplay FPS whenever a screenshot is being stored. To further debug this, I removed the actual save-to-disk part in the code and just left the ReadPixels() call, and a 1 second sleep between trying to access the render target. And sure enough, every second, there is a drop in the FPS in the game play, and I am not sure what’s causing it. My code can be seen at https://github.com/saihv/AirSim/blob/cameralogging/Unreal/Plugins/AirSim/Source/CameraLogger.cpp, but a very simplified version of what I am doing looks like this:
void ReadPixels_Modified()
{
APIPCamera* cam = GameThread->CameraDirector->getCamera(0);
USceneCaptureComponent2D* capture = cam->getCaptureComponent(EPIPCameraType::PIP_CAMERA_TYPE_SCENE, true);
if (capture != nullptr) {
if (capture->TextureTarget != nullptr) {
FTextureRenderTargetResource* RenderResource = capture->TextureTarget->GetRenderTargetResource();
// Create struct for render target data
ENQUEUE_UNIQUE_RENDER_COMMAND_ONEPARAMETER(
ReadSurfaceCommand,
FReadSurfaceContext, Context, ReadSurfaceContext,
{
RHICmdList.ReadSurfaceData(
Context.SrcRenderTarget->GetRenderTargetTexture(),
Context.Rect,
*Context.OutData,
Context.Flags
);
});
}
}
}
}
uint32 Thread::Run()
{
ReadPixels_Modified();
Thread::Sleep(1);
// Save image
}
It would be great if someone can shed some light on what I am doing wrong here!