Unreal vr running in the background while also rendering to the screen

Development Platform & Builds XR Development
, , ,
1

I am running an unreal application in the background while running another application in the foreground. The unreal application is doing some calculations and then sending those somewhere else. To further complicate things, I’m doing this in VR.

The way to get unreal to run steam VR in the background rather than in the foreground, is to edit this file:
Engine\Plugins\Runtime\Steam\SteamVR\Source\SteamVR\Private\SteamVRHMD.cpp

and change this:

		VRSystem = vr::VR_Init(&VRInitErr, vr::VRApplication_Scene);

to this:

		VRSystem = vr::VR_Init(&VRInitErr, vr::VRApplication_Overlay);

or this:

		VRSystem = vr::VR_Init(&VRInitErr, vr::VRApplication_Background);

More info on that can be found here: API Documentation · ValveSoftware/openvr Wiki · GitHub

If you make the change above and rebuild unreal, what you see is a black window and nothing renders to that window or to the HMD. But if you use print statements, you can see that unreal is still tracking everything. This seems like it’s working correctly and that this is expected behavior. But I’d like one more thing. What I’d like is for it to continue to not render to the HMD (this is how it is currently and this is how I want it to stay since another app is rendering to the HMD), but to render something to the window. Since it pops up a window, I’d like to be able to see something and not just have an empty window. Right now all I can see in that window are my print statements. Is it possible to render a camera into that view? Does anyone have any thoughts on things I can try? I realize this is a very strange scenario.

Thanks in advance.

2

The XR plugin system is designed for foreground applications. It’s heavily integrated into the rendering systems, and will switch to XR rendering if it finds an active plugin. You would probably have to either modify the plugin to not register itself as an XR plugin, or write a new plugin from scratch. Not using the XR system would likely break most of the XR library functions though.

3

It turns out there was a very simple solution to this, which was to uncheck “Lock to HMD” on the camera. I honestly have no idea why this fixes the problem, but it does.

image

If you need the camera to still follow the HMD, you can then just set the position manually on tick by using the Get Tracked Device Position and Orientation and then setting the device ID to 0.

image
image801×306 61.1 KB

1 Like
4

Glad that you found a solution.

Note that there may till be some issues attempting to run the plugin in the background. It will still likely render a stereo view and attempt to submit it to SteamVR every frame.

The plugin also calls the IVRCompositor interface in many places. This interface is only designed to be called from the foreground scene application, so it might have consequences like messing with the frame timing and rendering of the scene app, and the frame timing and device poses of the background application will likely not be synchronized properly. Without the synchronization it will not vsync properly, so the app may run with an uncapped framerate.