Transforming O/S mouse coordinates to viewport pixels

Development Programming & Scripting C++
1

Hello everyone!

I’ve been stuck on this problem for some time now for one of my plug-ins, and I’ve also posted on the answerhub on it:

but the guys over there were not able to help me unfortunately, so I thought I’d ask the people here as well :slight_smile:

In a nutshell, I need to transform coordinates that I get from a third party device (origin in the top left of the primary screen, in pixels) to pixel coordinates in the game viewport, much like the engine already does with the mouse device.

My first approach was to try and figure out how the mouse is translated and do the same thing, but I soon discovered that this is controlled through the FSceneViewport class when in editor mode (the summary says “A viewport for use with Slate SViewport widgets.” so I’m assuming this is used in the editor). However I’m afraid that even though I manage to find a solution using this class, another implementation class might be used when building for release, although I haven’t been able to find any other overrides for the mouse methods in the FSceneViewport in any other parts of the solution, so maybe my fear is unwarranted.

But I digress, the problem with this class though (found in SceneViewport.cpp) is that it’s using a private member (and method parameters in associated functions) ‘FGeometry CachedGeometry;’ to perform the transformation, and there doesn’t seem to be any way to intercept or access it without modifying the engine source, which I would like to avoid since it would make using the plugin harder than needs be.

I got a suggestion on the answerhub to use the:


bool ScreenToPixel(const FVector4& ScreenPoint,FVector2D& OutPixelLocation) const;

class of functions in the FSceneView object which is readily accessible, but unfortunately, the screen mentioned here is just a different coordinate system for the same area, and not the actual O/S pixels, which makes it useless for my purposes :frowning:

Has anyone already succeeded in doing what I’m trying to do? Or has a good idea on how to achieve it?
I’ve gotten it to work using both engine modifications (just exposing the FGeometry as public) as well as with mem-offset hacks to access the private member, but these methods are obviously not stable enough to use in a release :slight_smile:

Any insight / help is much appreciated.

Best regards,

2

Hey !

If you are looking the direct screen pixel position of your mouse, even when it’s captured in a viewport, I’d imagine the best way to still receive OS pixel coordinates is to bypass the unreal coord system altogether.

You should look into dll binding user32.dll for windows.
(There’s a great new tutorial from ZKarmaKun here)!

It handles OS level mouse positions, and will return the [correct screen space vector2D

Not entirely sure if this is what you’re after but I hope it helps.

Word of warning though: you’d be using a platform specific dll, so you may need an intermediate layer to manage the given OS.

-

3

Hello Loken!

Thank you for your suggestion, but I’m afraid this is not what I’m after. What I want to do is transform a position in O/S screen space to the local coordinate system of the game (this means the editor viewport if I’m in editor mode, the local window space if I’m in windowed mode in release, or no transformation at all if I’m in fullscreen release.

Some examples:

I get the O/S screen position V1 = (42, 24) from my 3rd party hardware, and the game is running in windowed release with the window position V2 = (10, 20); this means my transformed coordinate is V1 - V2 = Vr = (32, 4).

If the game was instead in fullscreen, my Vr would be the same as V1 since the origin of the window is the same as the primary screen origin (42, 24).

If the game was running in the unreal editor, I would need to do a transformation relative the viewport there (this is the example I’m talking about in the opening post), let’s say the CachedGeometry of the FSceneViewport reveals that the editor viewport is at (500, 600) in this case my Vr would become negative (which means my 3rd party hardware is pointing somewhere not on the viewport and should not be considered).

What I’m looking for is primarily some way to do this agnostic from what mode the running in (the same function no matter the mode), but I’m not altogether convinced it is possible to do it that way, my backup solution is to detect the current mode (either with preprocessor or with runtime variables) and choose the correct code path. But in that case, I need to find a way to get the position of the editor FViewport (and that is what I’m discussing in my op post).

Best regards,

4

Ok, since I couldn’t find any way to accomplish this, I made one, yay! Open source, yay!

To UnrealClient.h in the FViewport class I added:

(Comments present in code)


	virtual bool OperatingSystemPixelToViewportPixel(FIntPoint const * const OperatingSystemPoint, FIntPoint& ViewportPoint) const { return false; }
	virtual bool ViewportPixelToOperatingSystemPixel(FIntPoint const * const ViewportPoint, FIntPoint& OperatingSystemPoint) const { return false; }

I override them in FSceneViewport.h and then created these implementations in SceneViewport.cpp:



bool FSceneViewport::OperatingSystemPixelToViewportPixel(FIntPoint const * const OperatingSystemPoint, FIntPoint& ViewportPoint) const
{
	auto TransformedPoint = CachedGeometry.AbsoluteToLocal(FVector2D(OperatingSystemPoint->X, OperatingSystemPoint->Y));
	ViewportPoint.X = TransformedPoint .X;
	ViewportPoint.Y = TransformedPoint .Y;

	return true;
}

bool FSceneViewport::ViewportPixelToOperatingSystemPixel(FIntPoint const * const ViewportPoint, FIntPoint& OperatingSystemPoint) const
{
	auto TransformedPoint = CachedGeometry.LocalToAbsolute(FVector2D(ViewportPoint->X, ViewportPoint->Y));
	OperatingSystemPoint.X = TransformedPoint .X;
	OperatingSystemPoint.Y = TransformedPoint .Y;

	return true;
}

So far, it seems to work in the editor, but I have yet to test my fears regarding release mode.

After that I’m going to try and find some docs on Epics policy on pull requests on private forks since my plugin will be useless without these changes.

If anyone knows a better way to do this, I’m still open to suggestions!

Best regards,