# HTC Vive passthrough camera AR/VR development

**URL:** <https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330>\
**Category:** XR Development\
**Tags:** unreal-engine, question\
**Created:** [June 2, 2023, 3:04am UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330 "2023-06-02T03:04:38Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Drdodolord](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Drdodolord](https://forums.unrealengine.com/u/Drdodolord)\
**Post date:** [June 2, 2023, 3:04am UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/1 "2023-06-02T03:04:38Z")

</div>

Hi everyone,

I am quite new to developing VR experiences and using Unreal Engine 5. I have a project to undertake, and I own an HTC Vive Pro Eye. I would like to utilize its passthrough camera input similar to AR and then manipulate it within Unreal Engine. I have come across some videos featuring different HMDs, but I believe they were using Unity. If anyone could guide me on how to accomplish this in Unreal Engine, I would greatly appreciate it. Thanks in advance.

---

<div class="post-metadata">

**Author:** ![Rectus\_SA](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Rectus\_SA](https://forums.unrealengine.com/u/Rectus_SA)\
**Post date:** [June 2, 2023, 7:05pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/2 "2023-06-02T19:05:17Z")

</div>

There is no official support AFAIK, but I wrote a plugin for it: [GitHub - Rectus/UE4SteamVRPassthrough at ue5](https://github.com/Rectus/UE4SteamVRPassthrough/tree/ue5)

It only provides the same projection the 2D Room View mode in SteamVR does, which at least on the Valve Index has a lot of issues. It does allow you to access the camera images inside materials though, if you need more access, you would need to modify the plugin.

I’ve also written a newer implementation as an OpenXR API layer. It doesn’t allow accessing the camera images, but it has much better projection. I haven’t seen any confirmation whether it works on the Vive Pro though. [GitHub - Rectus/openxr-steamvr-passthrough: OpenXR API Layer for SteamVR camera passthrough](https://github.com/Rectus/openxr-steamvr-passthrough)

---

<div class="post-metadata">

**Author:** ![Drdodolord](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Drdodolord](https://forums.unrealengine.com/u/Drdodolord)\
**Post date:** [June 3, 2023, 8:16pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/3 "2023-06-03T20:16:14Z")

</div>

thanks a lot will try

---

<div class="post-metadata">

**Author:** ![Drdodolord](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Drdodolord](https://forums.unrealengine.com/u/Drdodolord)\
**Post date:** [June 6, 2023, 12:11pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/4 "2023-06-06T12:11:58Z")

</div>

I’m having trouble managing the build plugins in version 5.2. Which version are you using? Let me explain the steps I follow:

1. Create a project.
2. Create a ‘plugins’ folder in the root directory of the project.
3. Unzip your plugin and place it there.

When I try to build and reopen the project, it displays the following message: 'Try rebuilding from the source manually.

If you can help, that would be great. Thank you.

---

<div class="post-metadata">

**Author:** ![Rectus\_SA](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Rectus\_SA](https://forums.unrealengine.com/u/Rectus_SA)\
**Post date:** [June 6, 2023, 2:01pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/5 "2023-06-06T14:01:59Z")

</div>

I’ve only tested it on 5.0 and 4.27. Unfortunately I don’t have any pre-built version, you will have to build the plugin manually.

---

<div class="post-metadata">

**Author:** ![Ship-Destroyer](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Ship-Destroyer](https://forums.unrealengine.com/u/Ship-Destroyer)\
**Post date:** [July 13, 2023, 2:13pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/6 "2023-07-13T14:13:31Z")

</div>

Were you able to do occlusion with the Vive?

---

<div class="post-metadata">

**Author:** ![Rectus\_SA](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Rectus\_SA](https://forums.unrealengine.com/u/Rectus_SA)\
**Post date:** [July 13, 2023, 7:53pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/7 "2023-07-13T19:53:40Z")

</div>

The OpenXR API layer I linked above can do depth occlusion (fairly poorly) if you have something with stereo cameras. I’ve only tested it with the Index, but it should hopefully work with some of the Vive models as well.

---

<div class="post-metadata">

**Author:** ![nacho1998zgz](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@nacho1998zgz](https://forums.unrealengine.com/u/nacho1998zgz)\
**Post date:** [April 23, 2024, 11:53am UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/8 "2024-04-23T11:53:03Z")

</div>

Hi @Rectus_SA and @Drdodolord sorry to bother you both and bring back this topic. But would you be able to explain how to build the pluging manually so it can be used in new UE5 versions?

Thank you so much in advance for your assitance.

---

<div class="post-metadata">

**Author:** ![nacho1998zgz](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@nacho1998zgz](https://forums.unrealengine.com/u/nacho1998zgz)\
**Post date:** [April 23, 2024, 1:33pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/9 "2024-04-23T13:33:03Z")

</div>

@Rectus_SA The error comes from the reference to OpenVR at SteamVRPassthrough.Build.cs , how could I fix this reference?

Here I attach how I tried to rebuild the plugin:

 ![image](https://d3kjluh73b9h9o.cloudfront.net/original/4X/a/9/1/a91648dd4e04e015f98437cefdd5fe89d4c1b754.png)

---

<div class="post-metadata">

**Author:** ![Rectus\_SA](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Rectus\_SA](https://forums.unrealengine.com/u/Rectus_SA)\
**Post date:** [April 23, 2024, 2:05pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/10 "2024-04-23T14:05:28Z")

</div>

The plugin requires the OpenVR library to work. It used to be provided by Unreal for the SteamVR plugin, but since that has been removed in recent versions, you would have to either backport it, or provide it yourself instead from [GitHub - ValveSoftware/openvr: OpenVR SDK](https://github.com/ValveSoftware/openvr).

Note that the plugin will still not work out of the box with newer UE5 versions. The SceneView API has some changes that need to be addressed, and the material based passthrough would need a lot of work to function with the upgraded material systems (or disabled).

Unless you absolutely need support for accessing and modifying the camera images, I’d suggest using the OpenXR API layer instead of the plugin.

---

<div class="post-metadata">

**Author:** ![nacho1998zgz](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@nacho1998zgz](https://forums.unrealengine.com/u/nacho1998zgz)\
**Post date:** [April 23, 2024, 2:30pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/11 "2024-04-23T14:30:30Z")

</div>

Thank you so much for the swift reply!  
Understood, I will dive a little bit more in the topic and see what are viable options to get some camera feed into VR as kind of a “passthrough cutout” using SteamVR enabled camera from the headset (in this case either Valve Index or Htc Vive Pro"

Very much appreciated, have a nice day!

---

<div class="post-metadata">

**Author:** ![nacho1998zgz](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@nacho1998zgz](https://forums.unrealengine.com/u/nacho1998zgz)\
**Post date:** [April 24, 2024, 7:52am UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/12 "2024-04-24T07:52:54Z")

</div>

Sorry to bother again @Rectus_SA  
Recently I bumped into your project for [GitHub - Rectus/openxr-steamvr-passthrough: OpenXR API Layer for SteamVR camera passthrough](https://github.com/Rectus/openxr-steamvr-passthrough?tab=readme-ov-file)  
(I have a few questions regarding the use of it) Since it seems you are quite aware of this topic I was wondering if you could point me in the right direction. I am currently developing using UE5.3 and would like to explore (if possible) how to use HTC Vive Pro camera feed to have kind of like a “passthrough window” similar to what other manufacturers SDKs offer. Do you have any suggestion how to accomplish this using Open XR API layer or other approach that you may think is more suitable? (The end result would be something similar to what you developed with te orb passthrough project)

Thank you again.

Best regards

---

<div class="post-metadata">

**Author:** ![Rectus\_SA](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Rectus\_SA](https://forums.unrealengine.com/u/Rectus_SA)\
**Post date:** [April 24, 2024, 9:21am UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/13 "2024-04-24T09:21:36Z")

</div>

That should work fine with the API layer. The trickiest part with it is how to write a zeroed alpha channel in the right places. Unreal doesn’t provide any mechanism for directly writing alpha in scene materials, so you have to do it in a post process material.

One way of punching out a hole is to use the Custom Depth Stencil feature to set a stencil value on an object you want to render passthrough on, read the stencil value in a post process material, and use that to drive the Opacity output. Make sure Output Alpha is enabled in the material.

Note that you need to set “Enable alpha channel support in post processing (experimental)” to “Allow though tonemapper” in the project settings for alpha to be output.

---

<div class="post-metadata">

**Author:** ![nacho1998zgz](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@nacho1998zgz](https://forums.unrealengine.com/u/nacho1998zgz)\
**Post date:** [May 8, 2024, 12:20pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/14 "2024-05-08T12:20:29Z")

</div>

Hi @Rectus_SA , I was trying to explore several options how to access that camera feed but have not been successful so far… Also I saw in Vive’s documentation they don’t support it natively through openXR [https://developer.vive.com/resources/openxr/unreal/unreal-overview/](https://developer.vive.com/resources/openxr/unreal/unreal-overview/) , but SteamVR allows room view and so, so it is technically possible… But was wondering (since you are more experienced in this topic) how to use openXR API layer for that purpose of achieving a similar result to “room view” in the controller as steamVR does.

Thanks

---

<div class="post-metadata">

**Author:** ![Rectus\_SA](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Rectus\_SA](https://forums.unrealengine.com/u/Rectus_SA)\
**Post date:** [May 8, 2024, 1:07pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/15 "2024-05-08T13:07:16Z")

</div>

Basically it works by setting the OpenXR environment blend mode to be alpha blended, and setting the output alpha on the output framebuffer to 0 where you want the passthrough to be visible.

I haven’t made any tutorials on this myself yet, but Varjo (who’s OpenXR runtime supports this method natively) have some tutorials on how to set it up in Unreal. Note that the **IsMixedRealitySupported** and **IsMixedRealityEnabled** nodes require Varjos own Unreal plugin.

Basic setup:

> **[Mixed Reality with Unreal Engine 5](https://developer.varjo.com/docs/unreal/ue5/mixed-reality-with-unreal5)**
>
> Developing mixed reality applications for Varjo XR headsets is quick and easy with Varjo OpenXR plugin. Follow these instructions to get started.

Masking out specific objects:

> **[Masking with Unreal](https://developer.varjo.com/docs/unreal/ue5/masking-with-unreal5)**
>
> When developing for Varjo XR headsets, you can mask specific objects to act as a “window” into the real world. This is especially useful if you want to see real-world elements such as instruments or controller devices in the virtual environment....

---

<div class="post-metadata">

**Author:** ![nacho1998zgz](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@nacho1998zgz](https://forums.unrealengine.com/u/nacho1998zgz)\
**Post date:** [May 14, 2024, 10:53am UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/16 "2024-05-14T10:53:04Z")

</div>

Hi @Rectus_SA thank you for pointing me out in the right direction. I follow their approach to get a mixed reality background, but have run into the issue that when I toggle that function the app freezes.

Of course I took into account and not used those required Varjo plugin nodes, and just bypassed it with some variables I control. The problem is whenever I run the command to blen the environment, the application within the headset enters in the “waiting…” status, which is funny because in the Preview window it is still running (but no passthrough or camera feed being shown of course)

I runned the steamvr webconsole in an attempt to get some more information and debug what could be going on, but only noticed the next statements

 ![Screenshot 2024-05-14 124619](https://d3kjluh73b9h9o.cloudfront.net/original/4X/8/8/4/88461f5bf68bd748e0d75fcbc6552ff21d172779.png)

But in essence the only thing I am doing is the similar approach of “xr.OpenXREnvironmentBlendMode 3” but there is no mention of that in the log messages

 ![Screenshot 2024-05-14 124844](https://d3kjluh73b9h9o.cloudfront.net/original/4X/5/d/9/5d9b6643371f13f9a1628e278037dc2128546bba.png)

But then the output log from UE actually shed some light into the topic, I am trying these with both Valve Index and HTC Vive Pro 1/2. But no sucess with either.

 ![Screenshot 2024-05-14 125054](https://d3kjluh73b9h9o.cloudfront.net/original/4X/9/7/0/9706fa7a39797e93b117a13f7a423b5b53d3ac5c.png)

Any additional resource I should check out in an attempt to get that “mixed reality window” running through these devices?  
Thanks for your time and assistance!

---

<div class="post-metadata">

**Author:** ![Rectus\_SA](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Rectus\_SA](https://forums.unrealengine.com/u/Rectus_SA)\
**Post date:** [May 14, 2024, 10:59am UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/17 "2024-05-14T10:59:00Z")

</div>

Do you have the API layer installed?

[https://github.com/Rectus/openxr-steamvr-passthrough/wiki/Installation](https://github.com/Rectus/openxr-steamvr-passthrough/wiki/Installation)

If it’s working correctly, it should add an additional menu to the SteamVR dashboard while the application is running as seen here:  
[https://github.com/Rectus/openxr-steamvr-passthrough/wiki/Usage](https://github.com/Rectus/openxr-steamvr-passthrough/wiki/Usage)

---

<div class="post-metadata">

**Author:** ![nacho1998zgz](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@nacho1998zgz](https://forums.unrealengine.com/u/nacho1998zgz)\
**Post date:** [May 14, 2024, 12:03pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/18 "2024-05-14T12:03:14Z")

</div>

You are completely right! @Rectus_SA After all these trial and error I had uninstalleld the API layer.  
Once installed I am able to see the additional SteamVR menu, and this allows the application not to get stucked in that “Waiting…” status in runtime when toggeling xr.OpenXREnvironmentBlendMode  
 ![Screenshot 2024-05-14 135110](https://d3kjluh73b9h9o.cloudfront.net/original/4X/f/e/0/fe067ffe9fbe5e185ad8562bc7e37c1fdd88d5a6.png)

 ![Screenshot 2024-05-14 135934](https://d3kjluh73b9h9o.cloudfront.net/original/4X/9/4/f/94f319f22b0d1d0d3518aa0e4adaa24bab6c0fbf.jpeg)

The issue is nothing actually gets turned on, I played around a little bit with the settings, but nothing happens after so (I followed the varjo basic environment set up, with the postprocess material and so.  
An extra issue arised that when I play this from UE, it runs well the first time, but when I stop it and try to run it again, the whole editor crashes under this error:

 ![Screenshot 2024-05-14 134551](https://d3kjluh73b9h9o.cloudfront.net/original/4X/4/b/9/4b9555a4fd83f114d7061600770decd0c8a66759.png)

And to workaround that I have to close steamvr and open it again…

---

<div class="post-metadata">

**Author:** ![Rectus\_SA](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Rectus\_SA](https://forums.unrealengine.com/u/Rectus_SA)\
**Post date:** [May 14, 2024, 12:34pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/19 "2024-05-14T12:34:36Z")

</div>

Check that the SteamVR Room View works. The layer uses the SteamVR camara APIs, so if that isn’t set up right, it won’t work.

You can also look at the logs in the debug tab to see if there are any errors printed. The log also gets written to `C:\Users\<User>\AppData\Local\XR_APILAYER_NOVENDOR_steamvr_passthrough.log`

---

<div class="post-metadata">

**Author:** ![nacho1998zgz](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@nacho1998zgz](https://forums.unrealengine.com/u/nacho1998zgz)\
**Post date:** [May 14, 2024, 12:48pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/20 "2024-05-14T12:48:33Z")

</div>

@Rectus_SA yeah, camera test from desktop client and room view from vr dashboard work both fine

 ![Screenshot 2024-05-14 143951](https://d3kjluh73b9h9o.cloudfront.net/original/4X/3/c/a/3ca8ee80a3cfddbb67b935cbe42579f1d44b0b02.jpeg)

I don’t see anything particularly wrong with the log outoput, it seems like the API layer was enabled successfully for the session, but can not visualize in VR

 ![Screenshot 2024-05-14 144420](https://d3kjluh73b9h9o.cloudfront.net/original/4X/b/9/b/b9b22f041f96cfef5370338b2ea7520d2690d3aa.png)

[Next page](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330.md?page=2)
