Project 3D coordinates to SceneCaptureComponent2D

General Feedback & Requests
1

I posted this request on the answer hub, and I was told this is a better place to post feature requests, so I’m reposting this question here:

I would like to be able to project from a world location, to a pixel location, on a SceneCaptureComponent’s render target, so I can render icons over a mini-map.

Currently, I am rendering mini-map icons as billboard components in the captured scene, but those Icons are cropped by the bounds of the MiniMapWidget.

I would like to be able to draw the Icons on top of the mini-map, and clamp them to the bounds of the mini-map, so they slide along the edge. To place the icons, I would need to project a 3D world location to a 2D location on the texture, and add that to the 2D coordinates of the mini-map to get the screen location where I would render the icon.

The problem is, as far as I know, you can only project to a HUD’s Canvas, and I have not found any functions for projecting to a SceneCaptureComponent2D.

so I’m asking for a SceneCaptureComponent2D::Project() function, exposed to blueprint, and maybe a SceneCaptureComponent2D::Deproject() function as well.

I would also love to know why a SceneCaptureComponent2D doesn’t have a mesh in the editor, like the Camera, so you can see where its pointing… or better yet: why even have a SceneCaptureComponent2D, when you could just combine scene capturing for textures and cube maps into the camera?

2

You can do it, it’s just a lot of math, I did it for my minimap and it’s working fine :slight_smile:

3

could you give some hints on how you did it?

its such a low level engine feature that i didn’t want to waste a bunch of time digging through projection matrices, and its not a very important feature for the games that i am working on, i am just trying to help improve the features on the engine, even if its just by pointing out some useful features that are missing. to be honest, the only reason i made a minimap at all was to answer someone’s question on the answerhub, and i noticed this feature was missing.

I think this is a generally useful engine feature that should be built into the engine, rather than programmed in game code. without this SceneCapture2D::Project() being a standard function in blueprint, i cant make a blueprint only 3D minimap.

------- WIP -----------------

ill spend an hour or two on this, and if i cant solve it by then, i have more important gameplay features to code… so here goes nothing:

looking at SceneView.cpp, i think i need a ViewProjectionMatrix to transform, but in SceneCapture2D, i just have a DrawFrustum.

so i need to transform the data from a DrawFrustum into a ViewProjectionMatrix, then copy a bunch of code from SceneView::Project() …

I cant find where a SceneCapture2D gets rendered… it doesn’t seem like anything uses its DrawFrustum…

it seems like all of this code expects me to have a viewport… i wish it designed more generically, projection should work from any point of view, as a general math function.

its a scenecomponent, so i can use GetComponentToWorld() to get its world transform, and as a transform, i can use ToMatrixNoScale(), and that will give me a world space matrix representing the location and rotation of the point of view… so the inverse of that should give me ViewSpace, which can be multiplied by the projectionMatrix to get ViewProjectionMatrix… i think… then i use TransformFVector4 to get a plane with X and Y representing Pixel coordinates, which probably need to be normalized and adjusted for texture size.

so inside GameplayStatics, i would have to write a function that did something like this:
FPlane Result = (MySceneCapture2D.GetComponentToWorld().ToMatrixNoScale() * ProjMatrix).TransformFVector4(FVector4(WorldPosition, 1.f));
FVector2D PixelLocation(Result.X,Result.Y);
Return PixelLocation;

but its been 2 hours, so i’m not going to spend any more time trying to figure out low level rendering code. testing this code would have been a nightmare of recompile time, and i have no idea how many days it would take to get it right, so i will leave it to the engine programmers at epic, or anyone else who wants to tackle the problem.

4

I don’t know if you are “on the right track”, I have never looked into the source code of UE4 for my minimap, it is 100% blueprint :slight_smile:

Maybe my approach does not work for you since the image you attached in the first post looks like a 3D minimap. I have honestly no idea how to convert world location to RT location in 3D, my minimap only supports a 2D view of the 3D world, so “view from above”, not “bird’s-eye view”.

5

oh. a flat 2D projection would be alot easier, and if i cant solve this in 3D, that’s a good backup solution.

6

For anyone who might stumble upon this in the future as I did, I looked at ScottSpadea’s work and thought it looked really hard so I took a completely different approach. Here’s my code (for UE4.25):



// Getting the field of view of the external camera
float FOVx = SceneCaptureComponent2D->FOVAngle;
float FOVy = 2 * UKismetMathLibrary::Atan(UKismetMathLibrary::Tan(FOVx * UKismetMathLibrary::GetPI() / 180 / 2) * 9 / 16) * 180 / UKismetMathLibrary::GetPI(); //Assuming 16:9 aspect ratio (which it is by default). Formula from: https://www.reddit.com/r/Planetside/comments/1xl1z5/brief_table_for_calculating_fieldofview_vertical/

FVector RelativePosition = (/*Relative position here, or WorldPosition - SceneCaptureComponent2D->GetComponentLocation()*/);
FRotator RelativeAngle = RelativePosition.Rotation();

FVector2D ScreenLocation = (RelativeAngle.Yaw, RelativeAngle.Pitch);    //The initialization is useless but makes it a bit more clear

//Convert from angle to UV coords
ScreenLocation.X = 0.5 + (RelativeAngle.Yaw / (FOVx * 1.11));
ScreenLocation.Y = 0.5 - (1.11 * RelativeAngle.Pitch / (FOVy)); // Weird adjustment here

It’s weird that I need those adjustments in the last two lines, which lead me to think that there might be something wrong with my approach, but they seem to do the trick. If anyone tries this, let me know if you need the adjustment and, if so, if it’s the same.

2 Likes
7

Was searching for world to Scene Capture projection myself.Maybe someone will need it in future as i did.
Got inspired by Engine code :

bool UPJCProjectionBlueprintLibrary::ProjectWorldToSceneCapture(USceneCaptureComponent2D* SceneCaptureComponent2D,
	const FVector& WorldPosition, FVector2D& ScreenPosition)
{

	bool bResult = false;
	if (SceneCaptureComponent2D)
	{

		FMatrix OutViewMatrix;
		FMatrix ProjectionMatrix;
		FMatrix ViewProjectionMatrix;
		TOptional<FMatrix> CustomProjectionMatrix;
		if(SceneCaptureComponent2D->bUseCustomProjectionMatrix)
		{
		CustomProjectionMatrix = 
        SceneCaptureComponent2D->CustomProjectionMatrix;
		}

		FMinimalViewInfo ViewInfo;
		SceneCaptureComponent2D->GetCameraView(0.0f, ViewInfo);
		
		UGameplayStatics::CalculateViewProjectionMatricesFromMinimalView(ViewInfo,CustomProjectionMatrix, OutViewMatrix,ProjectionMatrix,ViewProjectionMatrix);
		
		FIntPoint TargetSize = FIntPoint(
SceneCaptureComponent2D->TextureTarget->SizeX,
 SceneCaptureComponent2D->TextureTarget->SizeY);
		FIntRect(FIntPoint(0, 0), TargetSize);
		
		bResult = FSceneView::ProjectWorldToScreen(WorldPosition, FIntRect(FIntPoint(0, 0), TargetSize), ViewProjectionMatrix, ScreenPosition);
		return bResult;
	}
	
	ScreenPosition = FVector2d::Zero();
	return bResult;
}
1 Like