How to get the screen location of a viewport?

Development Programming & Scripting
1

Hello everyone!

I’m currently implementing a plugin and I need to transform some coordinates in operating system screen space to the active viewport.
I’ve gotten as far as finding a reference to the viewport I’m interested in:
GEngine->GameViewport->Viewport

I’m guessing this will always resolve to the viewport in question (disregarding any potential situations where it might be null). The question then becomes; How do I transform my screen coordinates to local coordinates in this viewport?

I first approached the question from the mouse angle. Since the unreal engine somehow is able to transform the mouse coordinates (expressed in “operating system screen space”) to viewport space coordinates, I should be able to piggyback on that code to transform my coordinates… right?

Well it turned out to not be that easy. I think I’ve found the part of the code that does this, in SceneViewport.cpp:


void FSceneViewport::GetMousePos( FIntPoint& MousePosition, const bool bLocalPosition )
{
	if (bLocalPosition)
	{
		MousePosition = CachedMousePos;
	}
	else
	{
		const FVector2D AbsoluteMousePos = CachedGeometry.LocalToAbsolute(FVector2D(CachedMousePos.X, CachedMousePos.Y));
		MousePosition.X = AbsoluteMousePos.X;
		MousePosition.Y = AbsoluteMousePos.Y;
	}
}

This class is the actual type of GEngine->GameViewport->Viewport it seems.
The problem here is that CachedGeometry is private, and I can’t find a good way to get access to it.
I’m currently trying to find a way to intercept CachedGeometry before it reaches the SceneViewport class, but it’s proving to be hard.
I’ve tried a few hacks to subvert the encapsulation system to see if it could work, and it seems like it, but it’s hardly a good solution.

Does anyone know of a better solution, or of a good way to make this work?

UPDATE:

Rama suggested using the:

	/** Transforms a point from world-space to the view's screen-space. */
	FVector4 WorldToScreen(const FVector& WorldPoint) const;

	/** Transforms a point from the view's screen-space to world-space. */
	FVector ScreenToWorld(const FVector4& ScreenPoint) const;

	/** Transforms a point from the view's screen-space into pixel coordinates relative to the view's X,Y. */
	bool ScreenToPixel(const FVector4& ScreenPoint,FVector2D& OutPixelLocation) const;

	/** Transforms a point from pixel coordinates relative to the view's X,Y (left, top) into the view's screen-space. */
	FVector4 PixelToScreen(float X,float Y,float Z) const;

	/** Transforms a point from the view's world-space into pixel coordinates relative to the view's X,Y (left, top). */
	bool WorldToPixel(const FVector& WorldPoint,FVector2D& OutPixelLocation) const;

	/** Transforms a point from pixel coordinates relative to the view's X,Y (left, top) into the view's world-space. */
	FVector4 PixelToWorld(float X,float Y,float Z) const;

Class of functions on FSceneView, but the screen referenced here is just another coordinate system inside the viewport itself, and are not related to the O/S screen pixels.

Best regards,
Temaran

2

#SceneView Screen to Pixel

If you can get your FSceneView, then you want these two functions:

#SceneManagement.h

bool FSceneView::ScreenToPixel(const FVector4& ScreenPoint,FVector2D& OutPixelLocation) const

FVector4 FSceneView::PixelToScreen(float InX,float InY,float Z) const

#:heart:

Rama

3

Very nice, I’ll definately look into this!

Can’t believe I missed these two functions, I was grepping around for “ToScreen” etc, but I must have overlooked it.
Thank you Rama!

4

And if someone is wondering, the easiest way to get the sceneview is probably:


ULocalPlayer* LocalPlayer = Cast(Player);

	if (LocalPlayer != NULL && LocalPlayer->ViewportClient != NULL && LocalPlayer->ViewportClient->Viewport != NULL)
	{
		// Create a view family for the game viewport
		FSceneViewFamilyContext ViewFamily( FSceneViewFamily::ConstructionValues(
			LocalPlayer->ViewportClient->Viewport,
			GetWorld()->Scene,
			LocalPlayer->ViewportClient->EngineShowFlags )
			.SetRealtimeUpdate(true) );

		// Calculate a view where the player is to update the streaming from the players start location
		FVector ViewLocation;
		FRotator ViewRotation;
		FSceneView* SceneView = LocalPlayer->CalcSceneView( &ViewFamily, /*out*/ ViewLocation, /*out*/ ViewRotation, LocalPlayer->ViewportClient->Viewport );

		if (SceneView)
		{
			SceneView->ScreenToPixel(PROPERARGS);
		}
	}
5

After running a few tests, it seems that this wasn’t what I needed after all.
The screen referenced in these function names are just another coordinate system in the viewport it seems, not the actual physical screen.

My quest for a way to properly do this continues!

Thank you for the idea though :slight_smile:

6

I’m also very interested in the answer to this question, have you found out a solution?

7

I haven’t tried it myself, but I’m pretty sure you need FViewport::ViewportToVirtualDesktopPixel (FViewport::ViewportToVirtualDesktopPixel | Unreal Engine Documentation)

8

Haha, I’d completely forgotten about this thread. I was the one who added that function to the engine a couple of months ago, but ty anyways for taking the time :slight_smile:

9

I’ve been using a different method in PIE vs Standalone builds for a long time. It’s nice that I can finally unify them under one method (Slate’s TakeScreenshot). The problem I previously faced was this method would capture the whole editor window, and I only wanted the viewport. I could get the viewport size, but was struggling to get its position.

Thanks for adding ViewportToVirtualDesktopPixel, and I’m glad I found it mentioned in this thread. From there I could subtract the window position and have the viewport position in the window. Odd that UE makes this so hard, but at least this function makes it possible! I hadn’t found it before because it’s undocumented and I wasn’t sure if it was meant for use, or something internal.

For anyone who needs help with the conversion, here’s some code. Viewport is the PIE viewport. Size is that viewport’s size.

			TSharedPtr<SWindow> ViewportWindow = Viewport->FindWindow();
			TSharedRef<SWindow> Window = ViewportWindow ? ViewportWindow.ToSharedRef() : GEngine->GameViewport->GetWindow().ToSharedRef();

			// Attempt to only screenshot the PIE viewport section
			FVector2D UpperLeftInViewport(0, 0);
			FIntPoint UpperLeftInDesktop = Viewport->ViewportToVirtualDesktopPixel(UpperLeftInViewport);
			// Need to subtract window position to go from pos in desktop to pos in window.
			auto WindowRect = Window->GetRectInScreen();
			FIntPoint UpperLeftInWindow = UpperLeftInDesktop - FIntPoint(WindowRect.GetTopLeft2f().X, WindowRect.GetTopLeft2f().Y);
			FIntRect InnerWidgetArea(UpperLeftInWindow.X, UpperLeftInWindow.Y, UpperLeftInWindow.X + Size.X, UpperLeftInWindow.Y + Size.Y);
			bSuccess = FSlateApplication::Get().TakeScreenshot(Window, InnerWidgetArea, Bitmap, ScreenshotSize);

Context: For those curious, yes there are other ways to take a screenshot or get a texture from a camera (such as having a camera render target), but I have specific needs that those other methods kept failing at. I need to get the screenshot immediately, not waiting for a render frame to pass. I need the buffer in memory, not saved to a file. And most importantly, I need to have the screenshot perfectly match the current viewport’s buffer, so I can put the texture over the screen while I do other stuff, then fade it away in whatever cinematic way we like. Any discretion in postprocessing or colors etc is jarring when we display the texture over what the player was seeing, so directly fetching the viewport buffer or calling Slate’s TakeScreenshot (which does the same thing) is ideal.

Cheers, all!