Get image from SceneCapture2D

Development Programming & Scripting C++
1

I am trying to read an image from a USceneCapture2D from C++ to pass to a cv::Mat.

in the header:


    
    USceneCaptureComponent2D* sceneCapture;
    UTextureRenderTarget2D* renderTarget;

in the cpp:



    BeginPlay()
    {
        renderTarget = NewObject<UTextureRenderTarget2D>();
        renderTarget->InitAutoFormat(internResolution, internResolution);
        renderTarget->UpdateResourceImmediate();

        sceneCapture = CreateDefaultSubobject<USceneCaptureComponent2D>(TEXT("SceneCapture"));
        sceneCapture->CaptureSource = SCS_SceneColorSceneDepth;

        sceneCapture->TextureTarget = renderTarget;
        sceneCapture->SetupAttachment(OurCamera);
    }

    Tick()
    {
        if (renderTarget == nullptr)
        {
            renderTarget = NewObject<UTextureRenderTarget2D>();
            renderTarget->InitAutoFormat(internResolution, internResolution);
            renderTarget->UpdateResourceImmediate();
        }

        sceneCapture->TextureTarget = renderTarget;
        sceneCapture->UpdateContent();

        imageRendered = sceneCapture->TextureTarget->ConstructTexture2D(this, "CameraImage", EObjectFlags::RF_NoFlags, CTF_DeferCompression);

        FTexture2DMipMap* topMipMap = &imageRendered->PlatformData->Mips[0];
        FByteBulkData* RawImage = &topMipMap->BulkData;

        char* dataPointer = (char*)RawImage->Lock(LOCK_READ_ONLY);
    }

The resulting image looks like this:

[Album] imgur.com

The code to view the image:

Why are the colors so odd? What is going on here?
Something from my scene clearly makes it to the images, I can see contours of object I placed in the scene, so it somewhat works, but mostly not.
I checked my Float16 implementation against a IEEE754 website, and my results were the same, so I will assume that it works as intended. Even the depth looks funky (the foreground works, the background isā€¦ just off).

2

Did you try with initializing your render target with InitCustomFormat and choosing B8G8R8A8 instead? Or do you really need the precision/HDR output? Also, maybe you have an endianness problem? I.e. try flipping the two bytes when you read the raw data back (line 74 of your viewer code). Just throwing around ideasā€¦
You also should not rely on ConstructTexture2D as itā€™s EditorOnly and requires a power of two texture size iircā€¦ instead, I think you should use ReadPixels (see below)
I did some tests with this code, and the cv::Mat seemed to contain reasonable contents, so you might have to check your serialization/deserialization code:



void ASceneCaptureRecorder::Tick(float DeltaTime) {
  Super::Tick(DeltaTime);

  if (OutputVideoFile.IsEmpty()) return;

  if (RenderTarget == nullptr) {
    RenderTarget = NewObject<UTextureRenderTarget2D>();
    RenderTarget->InitCustomFormat(512, 512, EPixelFormat::PF_B8G8R8A8, true);
    RenderTarget->UpdateResourceImmediate();
  }

  SceneCapture->TextureTarget = RenderTarget;
  SceneCapture->CaptureScene();

  auto RenderTargetResource = RenderTarget->GameThread_GetRenderTargetResource();

  if (RenderTargetResource) {
    TArray<FColor> buffer;
    RenderTargetResource->ReadPixels(buffer);

    cv::Mat wrappedImage(RenderTarget->GetSurfaceHeight(), RenderTarget->GetSurfaceWidth(), CV_8UC4,
                         buffer.GetData());

    std::string OutputFile(TCHAR_TO_UTF8(*OutputVideoFile));
    cv::imwrite(OutputFile, wrappedImage);
  }
}
1 Like
3

Okay, I have rewritten some of my code due to your answer, but now I get absolutely nothing in the TArray<FColor> buffer;

I pasted the code at CustomScreenCapture - Pastebin.com and when I run it, buffer is just full of zeros. What am I doing wrong here? Why isnā€™t it rendering something into that TextureTarget? (Note, I changed it to FinalColor only for debugging reasons).

In the UE4 Editor GUI, when I click on an instance of the CustomScreenCapture, it displays the view of the camera in the preview just fine. But the buffer itself is blank.

CameraCapture.jpg
CameraCapture.jpg1929Ɨ1049 415 KB

4

Check out Ramaā€™s Victory plugin.
Donated some code to that years ago that grabbed texture from scene capture components and saved it.
Though its probably way out of date now, it may be of some use to you still.

5

I ran through that code, and it was fairly similar to what I had.

What I noticed though is that I absolutely have to call

imageRendered = sceneCapture->TextureTarget->ConstructTexture2D(this, ā€œCameraImageā€, EObjectFlags::RF_NoFlags, CTF_DeferCompression);

otherwise the ReadPixels() are blank. If I call that ConstructTexture2D, RenderTargetResource->ReadPixels() returns valid values. Is there a way for it to render into the texture target without reconstructing the Texture2D every frame, which is apparently fairly computational expensive?

Edit: disregard this, the first frame had no values, when I checked on the fifth frame, it was fine.

6

Okay, finally I have everything working as intended. My color display was wrong due to an issue with endianess and the unpredictability of the bitfield (the compiler decides the order of mantissa, exponent, sign), so I instead transform the color to char, save those, transform the depth to float, save that, and then everything works just fine.

Thanks for the feedback, while it didnā€™t really help with the problems itself, the codes you guys showed my made my program like 20 times fasterā€¦

7

Iā€™ve been spending hours here. And what Iā€™m getting is, yup, you guessed it, all BGRA 0 \0 entries throughout the buffer. I think Iā€™ve cleaned up the code pretty well, but Iā€™m just not seeing why the output array is just blanks.

My code: GitHub - zipzit/UnrealCustomScreenCapture: Unreal Game Engine -- Custom Scene Capture 2D (export video feed out of game engine...)

  • Iā€™m running the code on Windows64 bit, using VisualStudio 2019. Iā€™ve carefully aimed the CustomScreenCapture camera, and the preview thing works well. Iā€™m using the debugging mode to step thru the code and inspect the values contained within.
  • In debugging mode, I can see that the name of my CustomScreenCapture object matches that listed in the debugger.
  • I can preview the camera stuff in the game editor and it look absolutely correct.
  • I can see that the location and rotation of the camera in the game editor matches the AActor.PrimaryActorTick.Target.RootComponent.RelativeLocation and .RelativeRotation in the debugger in Visual Studio.
  • I have debugger stops set at the last printFString() in the program, where I can easily examine the TArray output.
  • The size / Num() of the FColor buffer TArray is 1048576 which is exactly 1024 x 1024. Thatā€™s a really good indicator.

ā€¦ But, all I get are 0,0.0ā€¦ all black.

Iā€™ve gone over all the details listed above. SinisterMJ and TheHugeManatee stuff super helpful, but there is still some stuff I donā€™t understand.

  • I donā€™t believe endianess would explain the all 0ā€™s in TArray.
  • I donā€™t understand the SinisterMJ comment ā€œEdit: disregard this, the first frame had no values, when I checked on the fifth frame, it was fine.ā€ I have no idea what ā€œTHISā€ refers to. Iā€™m assuming its the ConstructTexture2D() call, but Iā€™m not sure. From looking at the notes in TheHugeManatee posting, I donā€™t believe that is necessary.
  • Could I be failing on clipping? (sceneCapture.bEnableClipPlane = false)
  • Problem with PostProcessSettings? sceneCapture.PostProcessSettings {bOverride_WhiteTemp=0 '\0\ bOverride_WhiteTint=0 ā€˜\0ā€™ā€¦}
  • Iā€™ve tried other image formats, most fail in debugger with an exception thrown.

Other ideas on how to get successful output here? Many thanksā€¦

Editor View.jpg
Editor View.jpg1456Ɨ874 212 KB

VisualStudioDebugger.png
VisualStudioDebugger.png1143Ɨ1000 66.6 KB

8

My final code that did work:

CustomScreenCapture.h:



// Fill out your copyright notice in the Description page of Project Settings.

#pragma once

#include "CoreMinimal.h"

#include <string>
#include <memory>
#include <vector>

#include "GameFramework/Actor.h"
#include "Classes/Camera/CameraComponent.h"
#include "Classes/Components/SceneCaptureComponent2D.h"
#include "Classes/Engine/TextureRenderTarget2D.h"
#include "CustomScreenCapture.generated.h"

UCLASS()
class RTW_SIMULATION_API ACustomScreenCapture : public AActor
{
GENERATED_BODY()

public:
// Sets default values for this actor's properties
ACustomScreenCapture();

// Called every frame
virtual void Tick(float DeltaTime) override;

UPROPERTY(EditAnywhere, Category = "Output Information", meta = (ClampMin = "32", ClampMax = "4096", UIMin = "32", UIMax = "4096"))
uint32 resolutionX;

UPROPERTY(EditAnywhere, Category = "Output Information", meta = (ClampMin = "32", ClampMax = "4096", UIMin = "32", UIMax = "4096"))
uint32 resolutionY;

UPROPERTY(EditAnywhere, Category = "Output Information", meta = (ClampMin = "20.0", ClampMax = "170.0", UIMin = "20.0", UIMax = "179.9"))
float field_of_view;

UPROPERTY(EditAnywhere, Category = "Output Information")
FString outputFolderPath;

UPROPERTY(EditAnywhere, Category = "Stereo Setup")
FVector colorCameraTranslation;

UPROPERTY(EditAnywhere, Category = "Stereo Setup")
FQuat colorCameraRotation;
protected:
// Member variables

uint32_t internResolutionX; // Textures need to be power of 2
uint32_t internResolutionY; // Textures need to be power of 2

// Current counter of image
uint32_t counterImage;

// Complete compacted base filename
std::string baseFilenameDepth;
std::string baseFilenameColor;
std::string basePathFolder;
std::string sCounter;

UTextureRenderTarget2D* renderTargetDepth;
class USceneCaptureComponent2D* sceneCaptureDepth;
class UCameraComponent* OurCameraDepth;

UTextureRenderTarget2D* renderTargetColor;
class USceneCaptureComponent2D* sceneCaptureColor;
class UCameraComponent* OurCameraColor;

std::vector<uint16_t> depthVector;

// Called when the game starts or when spawned
virtual void BeginPlay() override;

void SaveTextureDepthmap();
void SaveTextureColor();


};

CustomScreenCapture.cpp:



// Fill out your copyright notice in the Description page of Project Settings.

#include "CustomScreenCapture.h"
#include "RTW_WorldSettings.h"
#include "nlohmann/json.hpp"
#include "HAL/PlatformFilemanager.h"
#include "GenericPlatform/GenericPlatformFile.h"

#include <fstream>

using json = nlohmann::json;

// Sets default values
ACustomScreenCapture::ACustomScreenCapture()
: resolutionX(1024)
, resolutionY(1024)
, field_of_view(90.0f)
, outputFolderPath(TEXT("."))
, colorCameraTranslation(0.0f, 0.0f, 0.0f)
, colorCameraRotation(0., 0., 0., 1.)
, counterImage(0)
, baseFilenameDepth("")
, baseFilenameColor("")
, basePathFolder("")
{
PrimaryActorTick.bCanEverTick = true;

// Only tick once all updates regarding movement and physics have happened
PrimaryActorTick.TickGroup = TG_PostUpdateWork;

RootComponent = CreateDefaultSubobject<USceneComponent>(TEXT("RootComponent"));

OurCameraDepth = CreateDefaultSubobject<UCameraComponent>(TEXT("ViewportCameraDepth"));
OurCameraDepth->SetupAttachment(RootComponent);

OurCameraColor = CreateDefaultSubobject<UCameraComponent>(TEXT("ViewportCameraColor"));
OurCameraColor->SetupAttachment(RootComponent);

// Resolution has to be a power of 2. This code finds the lowest RxR resolution which has more pixel than set

sceneCaptureDepth = CreateDefaultSubobject<USceneCaptureComponent2D>(TEXT("SceneCaptureDepth"));
sceneCaptureDepth->SetupAttachment(OurCameraDepth);

sceneCaptureColor = CreateDefaultSubobject<USceneCaptureComponent2D>(TEXT("SceneCaptureColor"));
sceneCaptureColor->SetupAttachment(OurCameraColor);
}

// Called when the game starts or when spawned
void ACustomScreenCapture::BeginPlay()
{
Super::BeginPlay();
basePathFolder = std::string(TCHAR_TO_UTF8(*outputFolderPath));

IPlatformFile& PlatformFile = FPlatformFileManager::Get().GetPlatformFile();
PlatformFile.CreateDirectory(*outputFolderPath);

// Go on with the file name
if (basePathFolder.back() != '/')
basePathFolder.append("/");

#pragma region Get_Resolution_Power_of_2
// Resolution has to be a power of 2. This code finds the lowest RxR resolution which has equal or more pixel than set
uint32_t higherX = resolutionX;

higherX--;
higherX |= higherX >> 1;
higherX |= higherX >> 2;
higherX |= higherX >> 4;
higherX |= higherX >> 8;
higherX |= higherX >> 16;
higherX++;

internResolutionX = higherX;

uint32_t higherY = resolutionY;

higherY--;
higherY |= higherY >> 1;
higherY |= higherY >> 2;
higherY |= higherY >> 4;
higherY |= higherY >> 8;
higherY |= higherY >> 16;
higherY++;

internResolutionY = higherY;
#pragma endregion

OurCameraDepth->FieldOfView = field_of_view;
OurCameraColor->FieldOfView = field_of_view;
sceneCaptureColor->FOVAngle = field_of_view;
sceneCaptureDepth->FOVAngle = field_of_view;

OurCameraColor->SetRelativeLocation(colorCameraTranslation);
OurCameraColor->SetRelativeRotation(colorCameraRotation);

renderTargetDepth = NewObject<UTextureRenderTarget2D>();
renderTargetDepth->InitCustomFormat(internResolutionX, internResolutionY, EPixelFormat::PF_FloatRGBA, true);

renderTargetDepth->UpdateResourceImmediate();

renderTargetColor = NewObject<UTextureRenderTarget2D>();
renderTargetColor->InitCustomFormat(internResolutionX, internResolutionY, EPixelFormat::PF_B8G8R8A8, true);

renderTargetColor->UpdateResourceImmediate();

sceneCaptureDepth->CaptureSource = SCS_SceneDepth;
sceneCaptureDepth->TextureTarget = renderTargetDepth;
sceneCaptureDepth->bCaptureEveryFrame = true;

sceneCaptureColor->CaptureSource = SCS_FinalColorLDR;
sceneCaptureColor->TextureTarget = renderTargetColor;
sceneCaptureColor->bCaptureEveryFrame = true;

//imageRendered = sceneCaptureDepth->TextureTarget->ConstructTexture2D(this, "CameraImage", EObjectFlags::RF_NoFlags, CTF_DeferCompression);

#pragma region Get_File_Name
// Temporary buffer
char targetBuffer[10];

std::ofstream metaData;

// World location, as string
std::string strPosX;
std::string strPosY;
std::string strPosZ;

// World rotation, as string
std::string strRotPitch;
std::string strRotRoll;
std::string strRotYaw;

std::string strRotX;
std::string strRotY;
std::string strRotZ;
std::string strRotW;


// Field of view (same for both cameras)
sprintf(targetBuffer, "%.3f", field_of_view);
std::string fov = std::string(targetBuffer);

// Get world location of Actor
FVector location = OurCameraDepth->GetComponentLocation();
sprintf(targetBuffer, "%.3f", location.X);
strPosX = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", location.Y);
strPosY = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", location.Z);
strPosZ = std::string(targetBuffer);

// Get yaw pitch roll of actor
FRotator rotation = OurCameraDepth->GetComponentRotation();
FQuat quaternion = rotation.Quaternion();

sprintf(targetBuffer, "%.3f", quaternion.X);
strRotX = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", quaternion.Y);
strRotY = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", quaternion.Z);
strRotZ = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", quaternion.W);
strRotW = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", rotation.Pitch);
strRotPitch = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", rotation.Roll);
strRotRoll = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", rotation.Yaw);
strRotYaw = std::string(targetBuffer);

ARTW_WorldSettings* tempPtr = reinterpret_cast<ARTW_WorldSettings*>(GetWorldSettings());
sprintf(targetBuffer, "%.3f", tempPtr->frames_per_second);
std::string strFPS = std::string(targetBuffer);

json j;
j"fps"] = strFPS;
j"fov"] = fov;
j"width"] = std::to_string(internResolutionX);
j"height"] = std::to_string(internResolutionY);

j"depth_image"]"pos_x"] = strPosX;
j"depth_image"]"pos_y"] = strPosY;
j"depth_image"]"pos_z"] = strPosZ;
j"depth_image"]"rot_pitch"] = strRotPitch;
j"depth_image"]"rot_roll"] = strRotRoll;
j"depth_image"]"rot_yaw"] = strRotYaw;

j"depth_image"]"rot_X"] = strRotX;
j"depth_image"]"rot_Y"] = strRotY;
j"depth_image"]"rot_Z"] = strRotZ;
j"depth_image"]"rot_W"] = strRotW;

baseFilenameDepth = basePathFolder + std::string("image");
baseFilenameDepth += std::string("_number_");

location = OurCameraColor->GetComponentLocation();

sprintf(targetBuffer, "%.3f", location.X);
strPosX = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", location.Y);
strPosY = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", location.Z);
strPosZ = std::string(targetBuffer);

// Get yaw pitch roll of actor
rotation = OurCameraColor->GetComponentRotation();
quaternion = rotation.Quaternion();

sprintf(targetBuffer, "%.3f", quaternion.X);
strRotX = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", quaternion.Y);
strRotY = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", quaternion.Z);
strRotZ = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", quaternion.W);
strRotW = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", rotation.Pitch);
strRotPitch = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", rotation.Roll);
strRotRoll = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", rotation.Yaw);
strRotYaw = std::string(targetBuffer);

baseFilenameColor = basePathFolder + std::string("image");
baseFilenameColor += std::string("_number_");

j"color_image"]"pos_x"] = strPosX;
j"color_image"]"pos_y"] = strPosY;
j"color_image"]"pos_z"] = strPosZ;
j"color_image"]"rot_pitch"] = strRotPitch;
j"color_image"]"rot_roll"] = strRotRoll;
j"color_image"]"rot_yaw"] = strRotYaw;

j"color_image"]"rot_X"] = strRotX;
j"color_image"]"rot_Y"] = strRotY;
j"color_image"]"rot_Z"] = strRotZ;
j"color_image"]"rot_W"] = strRotW;

// Get world location of Actor
location = colorCameraTranslation;
sprintf(targetBuffer, "%.3f", location.X);
strPosX = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", location.Y);
strPosY = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", location.Z);
strPosZ = std::string(targetBuffer);

// Get yaw pitch roll of actor
quaternion = colorCameraRotation;

sprintf(targetBuffer, "%.3f", quaternion.X);
strRotX = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", quaternion.Y);
strRotY = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", quaternion.Z);
strRotZ = std::string(targetBuffer);

sprintf(targetBuffer, "%.3f", quaternion.W);
strRotW = std::string(targetBuffer);

j"color_image"]"rel_pos_x"] = strPosX;
j"color_image"]"rel_pos_y"] = strPosY;
j"color_image"]"rel_pos_z"] = strPosZ;

j"color_image"]"rel_rot_X"] = strRotX;
j"color_image"]"rel_rot_Y"] = strRotY;
j"color_image"]"rel_rot_Z"] = strRotZ;
j"color_image"]"rel_rot_W"] = strRotW;

metaData.open(basePathFolder + "Metadata.json");
metaData << j.dump(2);
metaData.close();
#pragma endregion
}

// Called every frame
void ACustomScreenCapture::Tick(float DeltaTime)
{
Super::Tick(DeltaTime);

sCounter = std::to_string(counterImage);
sCounter = std::string(6 - sCounter.length(), '0') + sCounter;

if (counterImage > 0)
{
SaveTextureDepthmap();
SaveTextureColor();
}

counterImage++;
}

void ACustomScreenCapture::SaveTextureDepthmap()
{
auto RenderTargetResource = renderTargetDepth->GameThread_GetRenderTargetResource();

if (RenderTargetResource)
{
TArray<FFloat16Color> buffer16;
RenderTargetResource->ReadFloat16Pixels(buffer16);

std::string fileName = baseFilenameDepth;
fileName += sCounter + std::string(".depth16");
std::ofstream targetFileDepth(fileName, std::ofstream::binary);

depthVector.resize(buffer16.Num());

for (int32_t index = 0; index < buffer16.Num(); index++)
{
depthVector[index] = static_cast<uint16_t>(buffer16[index].R.GetFloat() * 10 + 0.5);
}

targetFileDepth.write(reinterpret_cast<char*>(depthVector.data()), depthVector.size() * sizeof(decltype(depthVector)::value_type));
targetFileDepth.close();
}
}

void ACustomScreenCapture::SaveTextureColor()
{
auto RenderTargetResource = renderTargetColor->GameThread_GetRenderTargetResource();

if (RenderTargetResource)
{
TArray<FColor> buffer8;
RenderTargetResource->ReadPixels(buffer8);

std::string fileName = baseFilenameColor;
fileName += sCounter + std::string(".bgr8");
std::ofstream targetFileColor(fileName, std::ofstream::binary);

targetFileColor.write(reinterpret_cast<char*>(buffer8.GetData()), buffer8.Num() * sizeof(FColor));
targetFileColor.close();
}
1 Like
9

Many thanks to SinisterMJ for his help here. I donā€™t understand exactly why my program failed, all generally had all the right pieces, but apparently not in exactly the right places. I had some commands that I ended up moving from BeginPlay() to the constructor(). A lot of stuff was redundant and removed. Note, I only used the 8Bit Color side of the final SinisterMJ final project, and not the 16bit depth stuff. Work is documented at my github repoā€¦ A couple of surprises.

  • I can confirm first capture buffer was all 0ā€™s.
  • Good data received from Tick #2 and beyond.
  • I could easily see the data results in the Visual Studio debugger.

I ended up saving the buffer data to disk in a simple 8Bit BGRA format (no hassle with endian stuff). I then opened the files using OpenCV, where they can be viewed and saved again as .jpg images. See SinisterMJ notes above for help on this.

Still a couple of things to work on.

  • The .jpg comes out pretty dark. Not sure what is going on there.
  • Its not clear if I require a perfect square, power of 2 size for the texture/image capture for the function calls I actually used: ***GameThread_GetRenderTargetResource(). ***Iā€™m really curious if I can capture a 1280x720 image. Lots of the other functions are clearly marked in the game engine header files as requiring power of 2 square elements, just not that one.

Again, many thanks for the help from SinisterMJ (and the rest of the folks hereā€¦)

Hereā€™s my first useful capture imageā€¦

test.jpg
test.jpg1024Ɨ1024 81 KB