Hi everyone,
I am quite new to developing VR experiences and using Unreal Engine 5. I have a project to undertake, and I own an HTC Vive Pro Eye. I would like to utilize its passthrough camera input similar to AR and then manipulate it within Unreal Engine. I have come across some videos featuring different HMDs, but I believe they were using Unity. If anyone could guide me on how to accomplish this in Unreal Engine, I would greatly appreciate it. Thanks in advance.
There is no official support AFAIK, but I wrote a plugin for it: GitHub - Rectus/UE4SteamVRPassthrough at ue5
It only provides the same projection the 2D Room View mode in SteamVR does, which at least on the Valve Index has a lot of issues. It does allow you to access the camera images inside materials though, if you need more access, you would need to modify the plugin.
Iâve also written a newer implementation as an OpenXR API layer. It doesnât allow accessing the camera images, but it has much better projection. I havenât seen any confirmation whether it works on the Vive Pro though. GitHub - Rectus/openxr-steamvr-passthrough: OpenXR API Layer for SteamVR camera passthrough
thanks a lot will try
Iâm having trouble managing the build plugins in version 5.2. Which version are you using? Let me explain the steps I follow:
When I try to build and reopen the project, it displays the following message: 'Try rebuilding from the source manually.
If you can help, that would be great. Thank you.
Iâve only tested it on 5.0 and 4.27. Unfortunately I donât have any pre-built version, you will have to build the plugin manually.
Were you able to do occlusion with the Vive?
The OpenXR API layer I linked above can do depth occlusion (fairly poorly) if you have something with stereo cameras. Iâve only tested it with the Index, but it should hopefully work with some of the Vive models as well.
Hi @Rectus_SA and @Drdodolord sorry to bother you both and bring back this topic. But would you be able to explain how to build the pluging manually so it can be used in new UE5 versions?
Thank you so much in advance for your assitance.
@Rectus_SA The error comes from the reference to OpenVR at SteamVRPassthrough.Build.cs , how could I fix this reference?
Here I attach how I tried to rebuild the plugin:
The plugin requires the OpenVR library to work. It used to be provided by Unreal for the SteamVR plugin, but since that has been removed in recent versions, you would have to either backport it, or provide it yourself instead from GitHub - ValveSoftware/openvr: OpenVR SDK.
Note that the plugin will still not work out of the box with newer UE5 versions. The SceneView API has some changes that need to be addressed, and the material based passthrough would need a lot of work to function with the upgraded material systems (or disabled).
Unless you absolutely need support for accessing and modifying the camera images, Iâd suggest using the OpenXR API layer instead of the plugin.
Thank you so much for the swift reply!
Understood, I will dive a little bit more in the topic and see what are viable options to get some camera feed into VR as kind of a âpassthrough cutoutâ using SteamVR enabled camera from the headset (in this case either Valve Index or Htc Vive Pro"
Very much appreciated, have a nice day!
Sorry to bother again @Rectus_SA
Recently I bumped into your project for GitHub - Rectus/openxr-steamvr-passthrough: OpenXR API Layer for SteamVR camera passthrough
(I have a few questions regarding the use of it) Since it seems you are quite aware of this topic I was wondering if you could point me in the right direction. I am currently developing using UE5.3 and would like to explore (if possible) how to use HTC Vive Pro camera feed to have kind of like a âpassthrough windowâ similar to what other manufacturers SDKs offer. Do you have any suggestion how to accomplish this using Open XR API layer or other approach that you may think is more suitable? (The end result would be something similar to what you developed with te orb passthrough project)
Thank you again.
Best regards
That should work fine with the API layer. The trickiest part with it is how to write a zeroed alpha channel in the right places. Unreal doesnât provide any mechanism for directly writing alpha in scene materials, so you have to do it in a post process material.
One way of punching out a hole is to use the Custom Depth Stencil feature to set a stencil value on an object you want to render passthrough on, read the stencil value in a post process material, and use that to drive the Opacity output. Make sure Output Alpha is enabled in the material.
Note that you need to set âEnable alpha channel support in post processing (experimental)â to âAllow though tonemapperâ in the project settings for alpha to be output.
Hi @Rectus_SA , I was trying to explore several options how to access that camera feed but have not been successful so far⌠Also I saw in Viveâs documentation they donât support it natively through openXR https://developer.vive.com/resources/openxr/unreal/unreal-overview/ , but SteamVR allows room view and so, so it is technically possible⌠But was wondering (since you are more experienced in this topic) how to use openXR API layer for that purpose of achieving a similar result to âroom viewâ in the controller as steamVR does.
Thanks
Basically it works by setting the OpenXR environment blend mode to be alpha blended, and setting the output alpha on the output framebuffer to 0 where you want the passthrough to be visible.
I havenât made any tutorials on this myself yet, but Varjo (whoâs OpenXR runtime supports this method natively) have some tutorials on how to set it up in Unreal. Note that the IsMixedRealitySupported and IsMixedRealityEnabled nodes require Varjos own Unreal plugin.
Basic setup:
Masking out specific objects:
