Mouse Interaction during stereoscopic rendering

Development Programming & Scripting C++
, , , ,
1

This is a report about a bug and a work-around.

When using SteamVR (it might also apply to other VR platforms, I did not check) UMG can’t correctly translate mouse coordinates to screenspace coordinates, because the viewport information it uses is set by SteamVRHMD.cpp.
Blueprint nodes like GetWindowClientSize, GetViewportSize (+GetViewportScale), and AbsoluteToViewport don’t give accurate information at that point anymore either. GetMousePositionOnPlatform does still seem to work correctly, but that still requires accurate knowledge about the coordinates and dimensions of the window.

It is worth noting that I wrote this thread largely here because it took quite some time to debug, and I don’t want to have to go through that again, and I don’t see why anyone else should have to. With a bit of luck I’ll have time to report a proper bug report, but given that performing interactions with UMG widgets using the mouse might not be the most common thing to do in VR projects, I wouldn’t expect a bug report like that to receive a lot of priority.

It is possible to work around it with a little bit of C++:

FVector4 rect = FVector4(0, 0, 0, 0);

TSharedPtr<SWindow> TopLevelWindow = FSlateApplication::Get().GetActiveTopLevelWindow();
if (TopLevelWindow.IsValid())
{
	FSlateRect coordinates = TopLevelWindow->GetClientRectInScreen();
	rect = FVector4(coordinates.Left, coordinates.Top, coordinates.Right - coordinates.Left, coordinates.Bottom - coordinates.Top);
}
return rect;

(There might be a neater way to pass it to Blueprints than through an FVector4, but I leave that as an exercise for the reader.)
Be sure to

#include "Widgets\SWindow.h"
#include "Framework\Application\SlateApplication.h"

You may also need to add “Engine”, “SlateCore”, and “Slate” to AdditionalDependencies in the uproject file, or plugin file. You probably don’t need all of them, but it’s been a long day, and I don’t want to wait for a bunch more compilations to find out for sure. Just try it with “Slate” only, might well work. (I needed the first two for other approaches of getting access to SWindow), which didn’t work in PIE)

That still leaves the challenge of displaying UMG through the spectator view (to prevent it from getting displayed in the HMD), by passing the Render Target of a widget to SetSpectatorScreenTexture, but there’s already a little bit of information about that in the form of Riley Florence’s video. Oh and remember that you need to tell playercontroller 0 to show the mouse cursor.

4 Likes
2

You might want to use

TArray<TSharedRef<SWindow>> VisibleWindows;
FSlateApplication::Get().GetAllVisibleWindowsOrdered(VisibleWindows);
if (VisibleWindows.Num() != 0)
{
	TSharedPtr<SWindow> TopLevelWindow = VisibleWindows[0];
	//Rest of the coordinate getting code here
}

instead of the GetActiveTopLevelWindow call, to get the TopLevelWindow, because the TopLevelWindow turns out to actually store the focused widget once you focus on one… Instead of… you know… the top level window.

Note, the index 0 assumption only holds while the editor and all of its windows are minimized.

3

Oh, and if you’re not in editor, you can use

UGameEngine* GameEngine = Cast<UGameEngine>(GEngine);
if (GameEngine != nullptr)
{
	TSharedPtr<SWindow> GameViewportWindow = GameEngine->GameViewportWindow.Pin();
	if (GameViewportWindow.IsValid())
	{
		// coordinate getting code here.
	}
}

Just like KismetSystemLibrary.cpp does for setting the title. Just be aware that the editor uses UUnrealEdEngine, which doesn’t have a GameViewportWindow reference :(.

4

Since this is now a thread full of ways to access the game window, here’s one I encountered used by epic:
void FindSceneViewport (OutInputWindow, OutSceneViewport) in VirtualCameraActor.cpp. It does the most straight forward UGameEngine approach in builds, and a widget search in editor. That editor dependency won’t work in builds, so you can modify your build.cs to add it conditionally.

if (Target.bBuildEditor == true)
{
	PrivateDependencyModuleNames.Add("UnrealEd");
}

Here’s hoping this is the last post I’ll have to make in this thread.

1 Like
5

Hey, this is really exciting stuff, thanks for sharing! I have a noob question though, since I never worked with C++ before - how can I setup this piece of script so that it’s exposed to BP? I would be grateful for just a high level overview, I think I will manage from there

7

Probably the easiest way to do this is with a C++ Blueprint Function Library
You can create one from the Unreal Editor, from File > New C++ Class, then scrolling down to Blueprint Function Library.

That should give you a .h header file (a short description of the function name, inputs and outputs) and a .cpp file (which contains the actual implementation).

For example, the function I created looks like this in the header of this new Blueprint Function Library:

UFUNCTION(BlueprintCallable, Category = "Window")
static FVector4 getWindowClientRect();

After compiling, the blueprint node looks like this:
image
(well… I split the outputs on that node, but close enough)

1 Like
8

So… it turns out, it’s even easier, it just wasn’t really documented outside of the source code.
Console command:
Slate.DrawToVRRenderTarget 0

Calling this will stop slate viewport widgets from showing up in the HMD (world widgets still display just fine). So you can just call AddToViewport, and use widgets as normal.

image
image761×307 39.3 KB

You may also want/have to

  • Make the window resizable so that viewport dimensions, and coordinates stay up to date.
  • Set ShowMouseCursor to true on the playercontroller

But yeah… much easier than manually copying over Widget RenderTargets to the Spectator screen texture.

3 Likes
9

In unreal 5.5 the slate.DrawToVRRenderTarget command is gone. Any idea what is the replacement for it?

10

Hello @mkderin
Did you find any replacement ?

11

any updates on the missing command in ue5.5?

12

I don’t currently use Unreal Engine 5. The benefits were mostly aimed at open world stuff, and did not yet outweigh the cost of fixing all the issues that came up while porting my 4.27 VR project.

What I can tell is that the cvar (CVarDrawToVRRenderTarget) was checked in FSlateRHIRenderer::DrawWindow_RenderThread
The comment for the cvar had the text “for better or worse” in it, so it’d make most sense if they just removed the weird behavior of drawing to the HMD, removing the need for the cvar, but given that you’re posting here, I guess they didn’t? You could check the source code for 5.5 to see what changed to slate rendering, and where they’re drawing to the HMD render target, and how to disable it.

Otherwise the work-arounds I started the thread with might still work.

13

Oh, that’s a shame… I can’t really change the engine code, so I’ll have to see if there is another way to apply mouse interactions.
Riley Florence’s video renders perfectly, but it doesn’t allow any mouse interactions, while this post using retainer boxes also works but for some reason if I create my own DisplayMaterial, it doesn’t render…

14

The work-around I describe above does not require changing the engine code. But perhaps you’re saying C++ in general is the daunting part, in which case, fair enough.

The only reason I mention engine code is because that’s how I originally found the cvar. It looks like they completely changed the way Slate draws in the case of stereo rendering, first drawing to a separate buffer if it’s available. I have no idea how to fix the bug where it draws to one eye of the HMD.

15

Oh does the workaround work for the Riley Florence’s method, too? I just assumed your workaround was for recalibrating the mouse position. For me, UMG renders but doesn’t interact with the mouse at all… Even I if I try moving the mouse all over the screen, the button doesn’t change to “hover” state.