Using USceneCaptureComponent2D to create a snapshot at runtime

[FrancescoDeso]

I’d like to be able to take a snapshot at runtime in certain moments. So, when some event occurs, I create a USceneCaptureComponent2D, place it where the player camera is located, take the snapshot and set it in a UMG widget. This is what I would like to accomplish.

This is how I’m creating the USceneCaptureComponent2D:

	// Creating the render target
	UTextureRenderTarget2D *renderTarget2D = NewObject<UTextureRenderTarget2D>();

	renderTarget2D->bHDR = false;
	renderTarget2D->ClearColor = FLinearColor::Blue;
	renderTarget2D->InitAutoFormat(512, 512);

	// Getting the camera's transform and creating the scene capturere actor
	FTransform cameraTransform = UGameplayStatics::GetPlayerCameraManager(GetWorld(), 0)->GetRootComponent()->GetComponentTransform();
	ASceneCapture2D *sceneCaptureActor = (ASceneCapture2D*) GetWorld()->SpawnActor<ASceneCapture2D>(ASceneCapture2D::StaticClass());

	if(sceneCaptureActor != nullptr)
	{
	    // Getting the actual component, registering and putting it where the camera was
		USceneCaptureComponent2D *sceneCaptureComponent = sceneCaptureActor->GetCaptureComponent2D();
		sceneCaptureComponent->RegisterComponent();
		sceneCaptureComponent->SetWorldTransform(cameraTransform);

		// Setting the target
		sceneCaptureComponent->TextureTarget = renderTarget2D;

		// Defining some properties; the most important one is that it won't capture every frame
		sceneCaptureComponent->ProjectionType = ECameraProjectionMode::Type::Perspective;
		sceneCaptureComponent->FOVAngle = 90.0f;
		sceneCaptureComponent->CaptureSource = ESceneCaptureSource::SCS_SceneColorHDR;
		sceneCaptureComponent->CompositeMode = ESceneCaptureCompositeMode::SCCM_Overwrite;
		sceneCaptureComponent->bCaptureOnMovement = false;
		sceneCaptureComponent->bCaptureEveryFrame = false;
		sceneCaptureComponent->MaxViewDistanceOverride = -1.0f;
		sceneCaptureComponent->bAutoActivate = true;
		sceneCaptureComponent->DetailMode = EDetailMode::DM_High;

		// Taking the snapshot
		sceneCaptureComponent->CaptureScene();

        // Telling the target to "refresh"
		renderTarget2D->UpdateResourceImmediate();

		// Creating the actual texture
		UTexture2D *newTexture = renderTarget2D->ConstructTexture2D(this, FString("Snapshot"), EObjectFlags::RF_NoFlags);

		// Calling a function that will trigger a Blueprint event, passing the texture to a UMG widget
		AddTexture(newTexture);
	}

And this is the Blueprint function that receives the texture and set it to an UMG widget:

Where “TextureWidget” is an Image widget.

The result of the code is just a 512x512 blue image (I set the texture render to use blue as clear color), so it’s kind of like an empty image.

I don’t understand what I’m doing wrong…

[Kris]

Try this:

1. Create it.
2. Tell it to capture on movement.
3. Shift it up and down slightly.
4. The information should now be up to date.
5. Grab the result.

[FrancescoDeso]

@Kris;696408

Try this:

1. Create it.
2. Tell it to capture on movement.
3. Shift it up and down slightly.
4. The information should now be up to date.
5. Grab the result.

The fact is that I would like to capture the scene in the moment I create the object and then discard it. It’s just a one time thing, I wouldn’t want to drag it with me for the next Ticks.

Isn’t it possible to take the snapshot at the creation?

[kamrann]

It’s probably actually more efficient to defer it one frame (CaptureSceneDeferred), since otherwise I’d imagine you’re blocking the game thread against the render thread.

Still, I don’t see why it wouldn’t be working.

[FrancescoDeso]

@kamrann;696537

It’s probably actually more efficient to defer it one frame (CaptureSceneDeferred), since otherwise I’d imagine you’re blocking the game thread against the render thread.

Still, I don’t see why it wouldn’t be working.

I also tried deferring it, but the result is the same.

[kamrann]

Just to clarify, you know that with the deferred call, you have to leave the actor/component in the level and grab the texture on the following tick?

If you did that and it’s still empty, then obviously something else is wrong somewhere, not sure what though.

[FrancescoDeso]

@kamrann;696559

Just to clarify, you know that with the deferred call, you have to leave the actor/component in the level and grab the texture on the following tick?

If you did that and it’s still empty, then obviously something else is wrong somewhere, not sure what though.

Yes I tried that.

I also tried creating the USceneCaptureComponent2D in the player class at compile time so that it’s not created on the fly, and then accessing it, setting the render and doing the rest. The result is still the same though…

[Kris]

@Beriol;696535

The fact is that I would like to capture the scene in the moment I create the object and then discard it. It’s just a one time thing, I wouldn’t want to drag it with me for the next Ticks.

Isn’t it possible to take the snapshot at the creation?

Understood, but I think that it needs a chance to capture something (bCaptureEveryFrame = true combined with a short delay) or will only capture something when it is moved (bCaptureOnMovement = true combined with a quick movement).

If you spawn it, then immediately move it slightly, I am hoping it will capture the information you need.

Spawn it, move it, get result, destroy it.

No delay

[FrancescoDeso]

@Kris;696956

Understood, but I think that it needs a chance to capture something (bCaptureEveryFrame = true combined with a short delay) or will only capture something when it is moved (bCaptureOnMovement = true combined with a quick movement).

If you spawn it, then immediately move it slightly, I am hoping it will capture the information you need.

Spawn it, move it, get result, destroy it.

No delay

I tried also that, but with no result! But actually I managed to get it working, and I found out that the things the UE4 didn’t like very much regarded pretty much how I used the result in UMG. The way I created the Texture2D and set it in UMG didn’t work properly, and I still don’t know why.

I ended up creating a material with a texture parameter, and when I capture something with the USceneCaptureComponent2D I set it as the texture parameter to the material, which is then used in UMG.

Here the whole thing:

The material:

The material domain is important. The default texture is not, since it will be changed later. UE4 couldn’t let me leave it empty though

The material is loaded from C++ in the constructor of the Actor (or whatever, in my case is an UUserWidget):

UMyWidget::UMyWidget(const FObjectInitializer& ObjectInitializer)
    : Super(ObjectInitializer)
{
    // Path to the material
    ConstructorHelpers::FObjectFinder<UMaterial> snapshotMaterialFinder(TEXT("Material'/Game/Materials/M_SnapshotMaterial.M_SnapshotMaterial'"));

    // Need to cast the pointer. The mSnapshotMaterial has type UMaterial*
    mSnapshotMaterial = (UMaterial*) snapshotMaterialFinder.Object;
}

I ended up creating the scene capture component only once in BeginPlay(). I create the actor and then get the component (I couldn’t make it work without spawning the actor):

    ASceneCapture2D *sceneCaptureActor = (ASceneCapture2D*) GetWorld()->SpawnActor<ASceneCapture2D>(ASceneCapture2D::StaticClass());
    mSceneCaptureComponent = sceneCaptureActor->GetCaptureComponent2D();

    mSceneCaptureComponent->ProjectionType = ECameraProjectionMode::Type::Perspective;
    mSceneCaptureComponent->FOVAngle = 90.0f;
    mSceneCaptureComponent->CaptureSource = ESceneCaptureSource::SCS_SceneColorHDR;
    mSceneCaptureComponent->CompositeMode = ESceneCaptureCompositeMode::SCCM_Overwrite;
    mSceneCaptureComponent->bCaptureOnMovement = false;
    mSceneCaptureComponent->bCaptureEveryFrame = false;
    mSceneCaptureComponent->MaxViewDistanceOverride = -1.0f;
    mSceneCaptureComponent->bAutoActivate = true;
    mSceneCaptureComponent->DetailMode = EDetailMode::DM_High;

Then when I need to take a picture I do this:

	// Getting the camera's transform and positioning the scene capture component there
	FTransform cameraTransform = UGameplayStatics::GetPlayerCameraManager(GetWorld(), 0)->GetRootComponent()->GetComponentTransform();
	mSceneCaptureComponent->SetWorldTransform(cameraTransform);

	// Creating the render target
	UTextureRenderTarget2D *renderTarget2D = NewObject<UTextureRenderTarget2D>();

	// Setting parameters and size
	renderTarget2D->bHDR = false;
	renderTarget2D->ClearColor = FLinearColor::Blue;
	renderTarget2D->InitAutoFormat(512, 512);

	// Setting the new target
	mSceneCaptureComponent->TextureTarget = renderTarget2D;

	// Capturing the scene. At the next tick we'll have what was captured
	mSceneCaptureComponent->CaptureSceneDeferred();

	// This boolean will be used later
	mCreateSnapshot = true;

Then, since I’m capturing the scene in a deferred way (because as kamrann said it’s more efficient) I’ll use the Tick function to get what was captured:

// In my case it's the NativeTick and not the Tick, since I'm using  UUserWidget
void UMyWidget::NativeTick(const FGeometry& MyGeometry, float InDeltaTime)
{
    Super::NativeTick(MyGeometry, InDeltaTime);

    // I only get the snapshot if I need to
    if(mCreateSnapshot)
    {
        // Retrieving the render target
        UTextureRenderTarget2D *renderTarget2D = mSceneCaptureComponent->TextureTarget;

        // Creating a new material from the base one and setting the render target as texture
        UMaterialInstanceDynamic* dynamicMaterial = UMaterialInstanceDynamic::Create(mSnapshotMaterial, this);
        dynamicMaterial->SetTextureParameterValue(FName("Texture"), renderTarget2D);

        // Telling the UMG to set the new material
        SetTextureMaterial(CurrentlySelectedStepIndex, dynamicMaterial);

        // The snapshot was taken, so no need to execute this code again
        mCreateSnapshot = false;
    }
}

This is all the C++ code. The UMG blueprint looks like this:

If you have suggestions on how to improve this please tell me

[Kris]

Besides using a one off timer, nope

[FrancescoDeso]

@Kris;697045

Besides using a one off timer, nope

You’re right of course, no need for the Tick for this