Hi everyone,
I am quite new to developing VR experiences and using Unreal Engine 5. I have a project to undertake, and I own an HTC Vive Pro Eye. I would like to utilize its passthrough camera input similar to AR and then manipulate it within Unreal Engine. I have come across some videos featuring different HMDs, but I believe they were using Unity. If anyone could guide me on how to accomplish this in Unreal Engine, I would greatly appreciate it. Thanks in advance.
There is no official support AFAIK, but I wrote a plugin for it: GitHub - Rectus/UE4SteamVRPassthrough at ue5
It only provides the same projection the 2D Room View mode in SteamVR does, which at least on the Valve Index has a lot of issues. It does allow you to access the camera images inside materials though, if you need more access, you would need to modify the plugin.
Iâve also written a newer implementation as an OpenXR API layer. It doesnât allow accessing the camera images, but it has much better projection. I havenât seen any confirmation whether it works on the Vive Pro though. GitHub - Rectus/openxr-steamvr-passthrough: OpenXR API Layer for SteamVR camera passthrough
thanks a lot will try
Iâm having trouble managing the build plugins in version 5.2. Which version are you using? Let me explain the steps I follow:
When I try to build and reopen the project, it displays the following message: 'Try rebuilding from the source manually.
If you can help, that would be great. Thank you.
Iâve only tested it on 5.0 and 4.27. Unfortunately I donât have any pre-built version, you will have to build the plugin manually.
Were you able to do occlusion with the Vive?
The OpenXR API layer I linked above can do depth occlusion (fairly poorly) if you have something with stereo cameras. Iâve only tested it with the Index, but it should hopefully work with some of the Vive models as well.
Hi @Rectus_SA and @Drdodolord sorry to bother you both and bring back this topic. But would you be able to explain how to build the pluging manually so it can be used in new UE5 versions?
Thank you so much in advance for your assitance.
@Rectus_SA The error comes from the reference to OpenVR at SteamVRPassthrough.Build.cs , how could I fix this reference?
Here I attach how I tried to rebuild the plugin:
The plugin requires the OpenVR library to work. It used to be provided by Unreal for the SteamVR plugin, but since that has been removed in recent versions, you would have to either backport it, or provide it yourself instead from GitHub - ValveSoftware/openvr: OpenVR SDK.
Note that the plugin will still not work out of the box with newer UE5 versions. The SceneView API has some changes that need to be addressed, and the material based passthrough would need a lot of work to function with the upgraded material systems (or disabled).
Unless you absolutely need support for accessing and modifying the camera images, Iâd suggest using the OpenXR API layer instead of the plugin.
Thank you so much for the swift reply!
Understood, I will dive a little bit more in the topic and see what are viable options to get some camera feed into VR as kind of a âpassthrough cutoutâ using SteamVR enabled camera from the headset (in this case either Valve Index or Htc Vive Pro"
Very much appreciated, have a nice day!
Sorry to bother again @Rectus_SA
Recently I bumped into your project for GitHub - Rectus/openxr-steamvr-passthrough: OpenXR API Layer for SteamVR camera passthrough
(I have a few questions regarding the use of it) Since it seems you are quite aware of this topic I was wondering if you could point me in the right direction. I am currently developing using UE5.3 and would like to explore (if possible) how to use HTC Vive Pro camera feed to have kind of like a âpassthrough windowâ similar to what other manufacturers SDKs offer. Do you have any suggestion how to accomplish this using Open XR API layer or other approach that you may think is more suitable? (The end result would be something similar to what you developed with te orb passthrough project)
Thank you again.
Best regards
That should work fine with the API layer. The trickiest part with it is how to write a zeroed alpha channel in the right places. Unreal doesnât provide any mechanism for directly writing alpha in scene materials, so you have to do it in a post process material.
One way of punching out a hole is to use the Custom Depth Stencil feature to set a stencil value on an object you want to render passthrough on, read the stencil value in a post process material, and use that to drive the Opacity output. Make sure Output Alpha is enabled in the material.
Note that you need to set âEnable alpha channel support in post processing (experimental)â to âAllow though tonemapperâ in the project settings for alpha to be output.
Hi @Rectus_SA , I was trying to explore several options how to access that camera feed but have not been successful so far⌠Also I saw in Viveâs documentation they donât support it natively through openXR https://developer.vive.com/resources/openxr/unreal/unreal-overview/ , but SteamVR allows room view and so, so it is technically possible⌠But was wondering (since you are more experienced in this topic) how to use openXR API layer for that purpose of achieving a similar result to âroom viewâ in the controller as steamVR does.
Thanks
Basically it works by setting the OpenXR environment blend mode to be alpha blended, and setting the output alpha on the output framebuffer to 0 where you want the passthrough to be visible.
I havenât made any tutorials on this myself yet, but Varjo (whoâs OpenXR runtime supports this method natively) have some tutorials on how to set it up in Unreal. Note that the IsMixedRealitySupported and IsMixedRealityEnabled nodes require Varjos own Unreal plugin.
Basic setup:
Masking out specific objects:
Hi @Rectus_SA thank you for pointing me out in the right direction. I follow their approach to get a mixed reality background, but have run into the issue that when I toggle that function the app freezes.
Of course I took into account and not used those required Varjo plugin nodes, and just bypassed it with some variables I control. The problem is whenever I run the command to blen the environment, the application within the headset enters in the âwaitingâŚâ status, which is funny because in the Preview window it is still running (but no passthrough or camera feed being shown of course)
I runned the steamvr webconsole in an attempt to get some more information and debug what could be going on, but only noticed the next statements
But in essence the only thing I am doing is the similar approach of âxr.OpenXREnvironmentBlendMode 3â but there is no mention of that in the log messages
But then the output log from UE actually shed some light into the topic, I am trying these with both Valve Index and HTC Vive Pro 1/2. But no sucess with either.
Any additional resource I should check out in an attempt to get that âmixed reality windowâ running through these devices?
Thanks for your time and assistance!
Do you have the API layer installed?
https://github.com/Rectus/openxr-steamvr-passthrough/wiki/Installation
If itâs working correctly, it should add an additional menu to the SteamVR dashboard while the application is running as seen here:
https://github.com/Rectus/openxr-steamvr-passthrough/wiki/Usage
You are completely right! @Rectus_SA After all these trial and error I had uninstalleld the API layer.
Once installed I am able to see the additional SteamVR menu, and this allows the application not to get stucked in that âWaitingâŚâ status in runtime when toggeling xr.OpenXREnvironmentBlendMode
The issue is nothing actually gets turned on, I played around a little bit with the settings, but nothing happens after so (I followed the varjo basic environment set up, with the postprocess material and so.
An extra issue arised that when I play this from UE, it runs well the first time, but when I stop it and try to run it again, the whole editor crashes under this error:
And to workaround that I have to close steamvr and open it againâŚ
Check that the SteamVR Room View works. The layer uses the SteamVR camara APIs, so if that isnât set up right, it wonât work.
You can also look at the logs in the debug tab to see if there are any errors printed. The log also gets written to C:\Users\<User>\AppData\Local\XR_APILAYER_NOVENDOR_steamvr_passthrough.log
@Rectus_SA yeah, camera test from desktop client and room view from vr dashboard work both fine
I donât see anything particularly wrong with the log outoput, it seems like the API layer was enabled successfully for the session, but can not visualize in VR
