Deproject SceneCaptureComponent2D

I currently have SceneCaptureComponent2D rendering to a texture that I’m using in a UMG widget. I want to deproject mouse clicks on that UMG widget to the 3D space in front of the SceneCaptureComponent2D. I’ve dug around the source to see if there’s anything that I can use, but I couldn’t quite find anything useful. And, unfortunately, I’m still trying to wrap my head around the math needed to make it work.

Any suggestions?


Answering my own question. This is what I did. (It involves editing UE4 source.)

First, I added two variables to USceneCaptureComponent.

	FIntRect ViewRect;
	FMatrix InvViewProjectionMatrix;

Then I populate those two variables at the end of FScene::CreateSceneRenderer (in SceneCaptureRendering.cpp)

// Copy the data needed to do deprojection.
	SceneCaptureComponent->ViewRect = View->ViewRect;
	SceneCaptureComponent->InvViewProjectionMatrix = View->InvViewProjectionMatrix;

	return FSceneRenderer::CreateSceneRenderer(&ViewFamily, NULL);

With that in place, I can use the above and take the screen position of the click event (or whatever) and call FSceneView::DeprojectScreenToWorld.

bool ADeprojectTest::Deproject(const FVector2D& ScreenPosition, FVector& Origin, FVector& Direction)
	if (SceneCaptureComponent)
		FSceneView::DeprojectScreenToWorld(ScreenPosition, SceneCaptureComponent->ViewRect, SceneCaptureComponent->InvViewProjectionMatrix, Origin, Direction);
		return true;
	return false;

That’s it. It should return an origin and direction normal for your tracing needs.

It seems that InvViewProjectionMatrix is not in FSceneView anymore. Have you found a solution for Unreal 4.15 ?

I got it from View->ViewMatrices->GetInvViewProjectionMatrix(), but it seems broken.