Download

SteamVR passthrough camera view

I’m trying to implement an AR passthrough mode using the Valve Index stereo cameras to selectively overlay the camera feed on to of the game world. Using the OpenVR IVRTrackedCamera functions I can access the image textures just fine, but I’m not able to get the image projection and frame prediction right. UE4 seems to have an AR module with the same functionality built-in, but it doesn’t seem to have any support for SteamVR, or VR headsets in general.

Had anyone been able to implement this in UE4, or have any other tips?

1 Like

It took forever, but I finally managed to implement it. It uses an FSceneView extension to to the appropriate tasks in the render thread. I currently got it setup in two ways, one passes a 3x3 projection matrices as material parameters so the screen UV’s can be projected in any scene material. The other is a simple post process shader (using the RDG) that projects and overlays the passthrough based on the Custom Depth Stencil.

Having to manually set up several vector parameters for the matrices in each leads to huge node mess, and the custom depth stencil it not rendered with MSAA leading to jaggy transitions. If anyone has any ideas on how to get around these things or other improvements, I’d be happy to hear them.

Applied to a material:
https://twitter.com/rectus_sa/status/1445132023803301895

Stenciled post process:

3 Likes

I got it packaged into a plugin. Supports 4.26 and 4.27. Any feedback is welcome.

https://github.com/Rectus/UE4SteamVRPassthrough

https://github.com/Rectus/UE4SteamVRPassthrough_Example

1 Like

I added support for running the plugin while using OpenXR. The plugin will use a background OpenVR session in parallel with the XR system to get the camera data.

Note that it still is limited to using SteamVR for the runtime, with a HMD that supports the SteamVR camera interface.

Thanks for sharing your work @Rectus_SA! It’s cool stuff, looking forward to see what you end up making with it

1 Like

This thread deserve more love. :kissing_heart:

2 Likes

Will this only work with Index?

I’ve only tested it with the Index and original Vive, but it should work with any headsets that expose the camera images to their SteamVR driver. I think most Vive headsets should work, it’s possible the Varjo XR headsets work as well.

Oculus and WMR headsets will not work. Oculus at least does not want to give applications direct access to the camera images.