How do you get the 2D coordinate a SceneCaptureComponent2D is projecting to?

Development Programming & Scripting UI
, , , ,
1

Okay, So I have a minimap which is being drawn from a Scene Capture Component 2D, and it captures the whole terrain. Problem is that capturing all the actors in terrain every tick causes the game to drop below 60 FPS, so my work around is to just render the terrain once on event begin play. Then draw an image widget that overlays on top of the static minimap. Afterwards, whenever the player moves, then update the player icon on the image widget to the new position. This would circumnavigate all the computation of the minimap for each and every tick.

So, here in lies my problem: I want to get the player location in the world, and then project that into the 2D coordinates that would be on my minimap.

I was trying to find a way to get the projection matrix that the scene capture component 2D uses, and then maybe try to use that to map 3D points in world space to the 2D points that I’m trying to compute. Does anyone have any suggestions? The Scene Capture Component 2D allows you to access a custom projection matrix, but I don’t want to define a custom projection matrix, I just want to grab the projection matrix that’s already being used by the Capture Component, and use that to get the coordinates I want.

2

Hi guy, have you solved the problem yet? I’m looking for this solution too.

3

Hi,

I was facing the same issue, as I could not find any information about that I decided to try to calibrate the SceneCaptureComponent2D’s camera the same way I would do it for a real camera (with a chessboard etc) and it amees to work well.

The resulting camera matrix allows me to compute the 2D coordinates of an object based on the location of the object, the location of the camera and the rotation of the camera, it works fine for me. If that can help you the camera matrix I found for a 1920x1080 image is:

9.5971160723108721e+02, 0, 9.6055043729890588e+02,

0, 9.6132797478587622e+02, 5.4029398159070877e+02,

0, 0, 1;

And also the following distortions parameters:

k1 = 7.3166520138576770e-03

k2 = -8.2875850280479196e-02

p1 = 9.4918855815189523e-05

p2 = 2.4066111259195101e-04

k3 = 2.7776018336345953e-01

I don’t know how familiar you are with projection matrices, if you are not, here is a quick explanation on how to get the coordinates of the object you wish to project in the camera’s coordinate system:

249598-worldtocameraframe.png
249598-worldtocameraframe.png1080×736 1.18 MB

.
Once you have that you just need to divide all the coordinates by the Z coordinate and then multiply the resulting vector by the camera matrix.

You can also apply the following formula before multiplying with the camera matrix if you want to remove the distortion of your images.

249600-undistort.png

Where x’ and y’ is what you get after dividing by the Z coordinates. After that you just need to multiply the camera matrix with the vector (x’’, y’’, 1) and you should obtain the 2D coordinates of you object.

Hope that can help.

( ͡° ͜ʖ ͡°)

4

Hi,

Guess you could reference these functions from UE:
UGameplayStatics::GetViewProjectionMatri
https://github.com/EpicGames/UnrealEngine/blob/4.25/Engine/Source/Runtime/Engine/Private/GameplayStatics.cpp#L2842
, which calls UGameplayStatics::CalculateViewProjectionMatricesFromMinimalView
https://github.com/EpicGames/UnrealEngine/blob/4.25/Engine/Source/Runtime/Engine/Private/GameplayStatics.cpp#L2817

Basically, here is what I tried:

bool ProjectWorldLocationToCapturedScreen(USceneCaptureComponent2D* InCaptureComponent, const FVector& InWorldLocation,
                                                                            const FIntPoint& InRenderTarget2DSize,
                                                                            FVector2D& OutPixel)
{
    // Render Target's Rectangle
    verify(InRenderTarget2DSize.GetMin() > 0);
    const FIntRect renderTargetRect(0, 0, InRenderTarget2DSize.X, InRenderTarget2DSize.Y);

    // Initialise Viewinfo for projection matrix from [InCaptureComponent]
    FMinimalViewInfo minimalViewInfo;
    InCaptureComponent->GetCameraView(0.f, minimalViewInfo);

    // Fetch [captureComponent]'s [CustomProjectionMatrix]
    TOptional<FMatrix> customProjectionMatrix;
    if(InCaptureComponent->bUseCustomProjectionMatrix) {
        customProjectionMatrix = InCaptureComponent->CustomProjectionMatrix;
    }

    // Calculate [cameraViewProjectionMatrix]
    FMatrix captureViewMatrix, captureProjectionMatrix, captureViewProjectionMatrix;
    UGameplayStatics::CalculateViewProjectionMatricesFromMinimalView(minimalViewInfo, customProjectionMatrix,
        captureViewMatrix, captureProjectionMatrix, captureViewProjectionMatrix);

    bool result = FSceneView::ProjectWorldToScreen(InWorldLocation, renderTargetRect,
                                                   captureViewProjectionMatrix, OutPixel);

    UE_LOG(LogTemp, Verbose, TEXT("ON [%s] CAPTURED SCREEN: WORLD LOCATION [%s] HAS LOCAL PIXEL COORDINATES: (X) %lf [over %d] OR (Y) %lf [over %d]"),
                             *InCaptureComponent->GetName(), *InWorldLocation.ToString(),
                             OutPixel.X, InRenderTarget2DSize.X,
                             OutPixel.Y, InRenderTarget2DSize.Y);
    return result;
}
5

If your player is at the center of the sceneCaptureComponent you should easily display markers using the method displayed in this video: Unreal Engine 4 Tutorial - Minimap (Advanced) - Part 3 - NPC/Object Icons - YouTube