Implementing a Native Interaction Support for 3D Widgets in VR


I’ve been racking my brain and searching the internet for a good week now, as I’m looking to implement an interactable 3D UI widget for VR. I’ve tried to do this many ways within the editor, and I’m now looking to either edit the engine or make an extension to UUserWidget that allows me to emulate all of the touch invocations on interactable subwidgets from blueprints. I intend to do this the way getnamo mentioned by implementing a recursive geometry collision solver.

Looking through the engine code, the whole widget library is quite a mess, and it looks like it hasn’t seen an overhaul in years; it was not made to be extensible. This surprises me, but I suppose this was meant to map directly into an on-screen overlay, which is the antithesis for UI design in VR. There’s UUserWidget which has a “WidgetTree,” which contains Widget objects which aren’t actually widgets until they map out into SWidget objects, which apparently inherit their geometry from their parents in real time. I can’t seem to get the geometry of any SWidget from itself, nor from any children. Am I going about this in the wrong direction?

I just want to be able to use UMG Editor to design my 3D VR widgets instead of the apparently condoned method of manually dragging components for hit detection over my UI components. Users of UE4 should not have to roll their own VR UI for every specific project, especially if we’re just trying to add a simple UI to interact with a quick tech demo.

Is anyone interested in working on this with me? The community should really make a push to understand, fork, fix, and merge the existing UMG code to fit VR into the ecosystem. At the end of this, if I end up with a separate class for a VR UI Widget, I’ll share it on the forums. If it requires modifications to the engine and I get it working robustly enough, I’ll gladly make a merge request to the source.

Thanks for your help,

Punt the native method until later, let Epic fix it the right way. In the meantime I forked Rama’s victory plugin here GitHub - MadisonAster/VictoryPlugin: Rama's Victory BP Plugin

Using this you can then hack around the issue to use umg widgets by setting the cursor to the center of the viewport, and tying an input action to a simulated click event:

You’ll also need to grab the hmd position, and update the menu and crosshair position on tick to keep it at the same position in space relative to your character. I’ve opted to simply place the menu inside an actor offset it and update the actor rotation:

After doing this, you can use all of your widgets as they are, without any individual hacks. Just make sure you uncheck “isFocusable” on any buttons, place an invisible button in the background of each base widget to catch any unhandled events and keep it in focus, and use the parabola distortion to make it curve around the character.

If you want to get really fancy, I also added a function on that fork to simulate the mousewheel so that scroll boxes can also be used:

Thank you for the answer, I’m going to try this within the week, and I’ll get back to you on how it works.

Rama merged my changes into the the master branch of the victory plugin now. So get the code from here instead:
link text

Help No matter what I download the 50 mb the 66 kb and no matter which directory either engine or project folders… following the info at the Forum from Rama himself…

I can get the plugin pack to be seen in Plugins and even restart without error loading map up, but for the life of me I cannot find the elusive Victory Simulate Key Press…

going a bit bat**** here trying to get this…

Any help would be greatly appreciated and i wont forget, and you’ll be thanked like you should be.

Just tested with this zip

Assuming UE4.11, and visual studio 2015 community are installed you can use option a or option b for any step along the way (try both just to learn how it works):

1a: From the unreal project browser, create a new UE4 c++ project (so that it creates a solution file)

1b: From an existing UE4 project create a new c++ class (so that it creates a solution file) doesn’t matter what kind.

2: When it opens visual studio you can just close it out if you’re using step 5a.

3a: Navigate to the project directory (Or just right click on content, select show in explorer, then go up a directory), create a new folder “Plugins”. DO NOT create this under the content directory, has to go in the project directory above it.

3b: Open Edit>Plugins from the UE4 editor, then click new plugin at the bottom right pane. (Victory is a Blueprint Library) This will create the Plugins folder for you, and your new plugin inside of it.

4: Copy the VictoryPlugin-master folder out of the zip file and into the Plugins directory.

5a: Close the UE4 editor and reopen the project. It will ask you if you would like to rebuild. select yes.

5b: In UE4 editor click File>Refresh Visual Studio Project, then open the solution in visual studio (double clicking the .sln file should do), right click on the project in visual studio, and select build. Restart the UE4 editor.

If you hit a snag, it’s almost definitely a visual studio problem. Make sure the project builds properly if you just ask it to create a new c++ class. if that won’t build, try reinstalling VS2015 (Remember community version is free so long as you login with your microsoft account inside the Visual Studio Application)

Got it installed and seen, Thanks a million Man.

Not yet got the plugin implemented though.

Ok it works so far… some tinkering with widget properties etc. I’m sure i’ll want it to do more now and find there’s limitations but Wow, thank you for the install help and the above blueprints. Doing full GearVR testing today, will get back with results…

I just want to post here to let some people viewing in that this may be a plaster resolution to the problem, as it does not particularly resolve anything.

I have recently launched my game on Steam that implemented a similar method to this, and while for the most part it works perfectly, I am seeing use cases with about 1-2% of users where this just does not work. They will have some software that blocks the window or blocks the mouse, or something. We still do not know what is actually causing this to happen.
They end up getting frustrated or upset, and that is BAD news for everyone involved.

I would still like to see Epic come up with a solution to this, that does not require rewriting the entire UI. Even if the solution is something as simple as what is done in the vr community templates where we have to replace the buttons with special objects, that would be fine.

From what i can tell we really need a solution to interact with 3d widgets from traces, completely separate from the mouse input. And the only way to achieve this is to create objects with hits and arrays of buttons. This is far too complex if you have an existing UI that is more than a few buttons.

This solution would also need to work with different input devices, and different HMD devices.

There are ways around this, and starting out fresh may just be the only answer, it can just significantly increase development time if you already have a complex ui done in umg for non vr use.

Just to be clear i’m not getting angry here, and will probably end up having to rewrite my entire UI.
Just wanted to point some things out and get some discussion going around it as it can hurt both developers and the customers of developers.

take care!

look into this… Mitch just got a Dev Grant for his work… he has trace UI interaction…

Already have, that doesn’t do what i was asking.

This however, which he literally JUST posted.
Is EXACTLY the solution we have needed.

VR UMG Compatibility Plugin! - XR Development - Unreal Engine Forums!

Like, seriously guys check this out.

I know it’s great isn’t it… was hours off by posting… look into Nick Darnell’s work as well.

Hi, I’ve been trying to follow these instructions, to get a stationary screen in VR to display an interactable Widget, with a cursor that follows where the player is looking… But I still can’t seem to get the hover-feature to respond to where the HMD/mouse is looking. :confused:
I don’t think I need to worry about moving things with the player, or rotation, and I’m not making any kind of crosshair or cursor… I just want the player to be able to look at a Button, have the button highlight, and then they can push a button to simulate a click. I’ve followed ur instructions, but nothing seems to be interacting yet.
And this part is really confusing me:
", place an invisible button in the background of each base widget to catch any unhandled events and keep it in focus, "
I’m not sure what this means, or why?

This issue is now resolved in 4.13, see:


on this page: Unreal Engine 4.13 Released! - Unreal Engine