Mouse Interaction during stereoscopic rendering

Development Programming & Scripting C++
, , , ,
1

This is a report about a bug and a work-around.

When using SteamVR (it might also apply to other VR platforms, I did not check) UMG can’t correctly translate mouse coordinates to screenspace coordinates, because the viewport information it uses is set by SteamVRHMD.cpp.
Blueprint nodes like GetWindowClientSize, GetViewportSize (+GetViewportScale), and AbsoluteToViewport don’t give accurate information at that point anymore either. GetMousePositionOnPlatform does still seem to work correctly, but that still requires accurate knowledge about the coordinates and dimensions of the window.

It is worth noting that I wrote this thread largely here because it took quite some time to debug, and I don’t want to have to go through that again, and I don’t see why anyone else should have to. With a bit of luck I’ll have time to report a proper bug report, but given that performing interactions with UMG widgets using the mouse might not be the most common thing to do in VR projects, I wouldn’t expect a bug report like that to receive a lot of priority.

It is possible to work around it with a little bit of C++:

FVector4 rect = FVector4(0, 0, 0, 0);

TSharedPtr<SWindow> TopLevelWindow = FSlateApplication::Get().GetActiveTopLevelWindow();
if (TopLevelWindow.IsValid())
{
	FSlateRect coordinates = TopLevelWindow->GetClientRectInScreen();
	rect = FVector4(coordinates.Left, coordinates.Top, coordinates.Right - coordinates.Left, coordinates.Bottom - coordinates.Top);
}
return rect;

(There might be a neater way to pass it to Blueprints than through an FVector4, but I leave that as an exercise for the reader.)
Be sure to

#include "Widgets\SWindow.h"
#include "Framework\Application\SlateApplication.h"

You may also need to add “Engine”, “SlateCore”, and “Slate” to AdditionalDependencies in the uproject file, or plugin file. You probably don’t need all of them, but it’s been a long day, and I don’t want to wait for a bunch more compilations to find out for sure. Just try it with “Slate” only, might well work. (I needed the first two for other approaches of getting access to SWindow), which didn’t work in PIE)

That still leaves the challenge of displaying UMG through the spectator view (to prevent it from getting displayed in the HMD), by passing the Render Target of a widget to SetSpectatorScreenTexture, but there’s already a little bit of information about that in the form of Riley Florence’s video. Oh and remember that you need to tell playercontroller 0 to show the mouse cursor.

4 Likes
2

You might want to use

TArray<TSharedRef<SWindow>> VisibleWindows;
FSlateApplication::Get().GetAllVisibleWindowsOrdered(VisibleWindows);
if (VisibleWindows.Num() != 0)
{
	TSharedPtr<SWindow> TopLevelWindow = VisibleWindows[0];
	//Rest of the coordinate getting code here
}

instead of the GetActiveTopLevelWindow call, to get the TopLevelWindow, because the TopLevelWindow turns out to actually store the focused widget once you focus on one… Instead of… you know… the top level window.

Note, the index 0 assumption only holds while the editor and all of its windows are minimized.

3

Oh, and if you’re not in editor, you can use

UGameEngine* GameEngine = Cast<UGameEngine>(GEngine);
if (GameEngine != nullptr)
{
	TSharedPtr<SWindow> GameViewportWindow = GameEngine->GameViewportWindow.Pin();
	if (GameViewportWindow.IsValid())
	{
		// coordinate getting code here.
	}
}

Just like KismetSystemLibrary.cpp does for setting the title. Just be aware that the editor uses UUnrealEdEngine, which doesn’t have a GameViewportWindow reference :(.

4

Since this is now a thread full of ways to access the game window, here’s one I encountered used by epic:
void FindSceneViewport (OutInputWindow, OutSceneViewport) in VirtualCameraActor.cpp. It does the most straight forward UGameEngine approach in builds, and a widget search in editor. That editor dependency won’t work in builds, so you can modify your build.cs to add it conditionally.

if (Target.bBuildEditor == true)
{
	PrivateDependencyModuleNames.Add("UnrealEd");
}

Here’s hoping this is the last post I’ll have to make in this thread.

1 Like
5

Hey, this is really exciting stuff, thanks for sharing! I have a noob question though, since I never worked with C++ before - how can I setup this piece of script so that it’s exposed to BP? I would be grateful for just a high level overview, I think I will manage from there

7

Probably the easiest way to do this is with a C++ Blueprint Function Library
You can create one from the Unreal Editor, from File > New C++ Class, then scrolling down to Blueprint Function Library.

That should give you a .h header file (a short description of the function name, inputs and outputs) and a .cpp file (which contains the actual implementation).

For example, the function I created looks like this in the header of this new Blueprint Function Library:

UFUNCTION(BlueprintCallable, Category = "Window")
static FVector4 getWindowClientRect();

After compiling, the blueprint node looks like this:
image
(well… I split the outputs on that node, but close enough)

1 Like
8

So… it turns out, it’s even easier, it just wasn’t really documented outside of the source code.
Console command:
Slate.DrawToVRRenderTarget 0

Calling this will stop slate viewport widgets from showing up in the HMD (world widgets still display just fine). So you can just call AddToViewport, and use widgets as normal.

image
image761×307 39.3 KB

You may also want/have to

  • Make the window resizable so that viewport dimensions, and coordinates stay up to date.
  • Set ShowMouseCursor to true on the playercontroller

But yeah… much easier than manually copying over Widget RenderTargets to the Spectator screen texture.