I’m trying to implement an AR passthrough mode using the Valve Index stereo cameras to selectively overlay the camera feed on to of the game world. Using the OpenVR IVRTrackedCamera functions I can access the image textures just fine, but I’m not able to get the image projection and frame prediction right. UE4 seems to have an AR module with the same functionality built-in, but it doesn’t seem to have any support for SteamVR, or VR headsets in general.
Had anyone been able to implement this in UE4, or have any other tips?
It took forever, but I finally managed to implement it. It uses an FSceneView extension to to the appropriate tasks in the render thread. I currently got it setup in two ways, one passes a 3x3 projection matrices as material parameters so the screen UV’s can be projected in any scene material. The other is a simple post process shader (using the RDG) that projects and overlays the passthrough based on the Custom Depth Stencil.
Having to manually set up several vector parameters for the matrices in each leads to huge node mess, and the custom depth stencil it not rendered with MSAA leading to jaggy transitions. If anyone has any ideas on how to get around these things or other improvements, I’d be happy to hear them.
I added support for running the plugin while using OpenXR. The plugin will use a background OpenVR session in parallel with the XR system to get the camera data.
Note that it still is limited to using SteamVR for the runtime, with a HMD that supports the SteamVR camera interface.
I’ve only tested it with the Index and original Vive, but it should work with any headsets that expose the camera images to their SteamVR driver. I think most Vive headsets should work, it’s possible the Varjo XR headsets work as well.
Oculus and WMR headsets will not work. Oculus at least does not want to give applications direct access to the camera images.
I tried to use your plugin example (4.27) with my Valve Index, but can’t get it to work.
It compiles, the game starts and the VR interaction works and everything. However, the spheres that I think are supposed to show the passthrough video input just show their defaults textures. I checked the log and I get the following Warnings:
LogSpawn: Warning: SpawnActor failed because no class was specified
LogSteamVRPassthrough: Warning: CameraFrameSize error [100] on device Id 0
LogSteamVRPassthrough: Warning: Error enabling passthrough video!
I also get the same warnings when I set up a UE5 project on my own, include your plugin and call the EnableVideo function at BeginPlay.
Do you have any hints on what’s going wrong there?
