Trouble creating a virtual cursor to control an experience through widget on spectator screen

Development Platform & Builds XR Development
1

Hullo everyone,

context

Like many others, I want to control a virtual reality experience using a spectator screen.
I have everything working for controlling a widget in the spectator view with the keyboard, but I’m having a small issue when trying to get the mouse to work as expected.

what i’ve got so far

  1. A pawn with a UWidgetComponent and a UWidgetInteractionComponent that is turned to face the widget component.
  2. A function that gets the “absolute location” for a cursor based on the mouse position by getting the mouse x and y and dividing it by the viewport size x and y divided by the pixel density in virtual reality.
  3. A function that offsets the absolute location for the cursor such that the widget interaction component’s location is correctly set face the widget component, which needs a pivot of (.5f,.5f).
  4. A widget with a UImage behaving as a virtual cursor that has is visible to the user only (no hit).
  5. A render target that renders out the real scene and displays it on the UI through a user interface material.
  6. A behaviour such that when the P key is pressed when running in virtual reality the spectator mode is switched to (and from) texture and the texture is set to the render target of the widget component.

problems

This isn’t a hugely efficient setup, as the widget needs to be rendered in world space, but for now it seems to be the best option.
The only big problem now is that the virtual cursor has correct values while not running in VR but incorrect values in VR such that sometimes the cursor is in the incorrect position

what seems to be the root of all evil

It appears that GetViewportSize doesn’t actually return the correct values when in virtual reality.
Does anybody know a workaround for this?

If anybody has a moment to take a look at the project to help me figure out why the virtual reality position is incorrect, I’d really appreciate it!
The base code for the solution is MIT licensed in this repository: GitHub - calben/UnrealVrSpectatorControl: Example project for controlling a VR experience through UI a

Thanks for any help!!

2

For some reason, I can’t access your file.

If it’s rendered in world space, why can’t you use a LookAt function to shoot a line from the camera to the mouse, use that rotation to get a forward vector and shoot a widget interaction from the mouse to the widget.

The position issue when in vr mode is probably caused by incorrectly recentering on the center of the percieved field of view for the hmd and not the center of the left eye.

3

Hey @NotSoAccurateNo1 ! Thanks for the response. I ended up figuring it out. The problem ended up being that the viewport size was not the correct value to get, as the viewport was the VR display. The correct value could be gotten from a GEngine variable. I’ll post a working solution to this thread soon!