# HTC Vive passthrough camera AR/VR development

**URL:** <https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330>\
**Category:** XR Development\
**Tags:** unreal-engine, question\
**Created:** [June 2, 2023, 3:04am UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330 "2023-06-02T03:04:38Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![Rectus\_SA](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Rectus\_SA](https://forums.unrealengine.com/u/Rectus_SA)\
**Post date:** [May 14, 2024, 12:56pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/21 "2024-05-14T12:56:45Z")

</div>

That looks like it is supposed to at least. Does it work with other OpenXR applications?

---

<div class="post-metadata">

**Author:** ![nacho1998zgz](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@nacho1998zgz](https://forums.unrealengine.com/u/nacho1998zgz)\
**Post date:** [May 14, 2024, 1:39pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/22 "2024-05-14T13:39:42Z")

</div>

@Rectus_SA as I did not know which other apps may use OpenXR I ended up trying The Blue (VR game) and it allowed me to acces the menu dashboard and I guess it worked as expected (as settings changes were having an effect on the look of the passtrhough)

 ![Screenshot 2024-05-14 153527](https://d3kjluh73b9h9o.cloudfront.net/original/4X/1/c/f/1cff838eaa1715ba3181785e1b672965fb090a65.jpeg)

So this is a temporary relief, but now the question arise of what could be happening that the DefaultVR Template from UE with those changes from Varjo does not effectively enable and use xr.OpenXREnvironmentBlendMode

I check the log out of curiosity, and it is pretty much identical to the previous one…

 ![Screenshot 2024-05-14 153921](https://d3kjluh73b9h9o.cloudfront.net/original/4X/4/6/6/4660de460d556f2be1bf8aa581a0dc1de2f89995.png)

---

<div class="post-metadata">

**Author:** ![Rectus\_SA](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Rectus\_SA](https://forums.unrealengine.com/u/Rectus_SA)\
**Post date:** [May 14, 2024, 2:07pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/23 "2024-05-14T14:07:58Z")

</div>

Looking at it, there seems to be an issue with the passthrough rendering in UE 5.4. It seems to render to a different swapchain image than the one actually being presented. I’ll have to look into why that is happening. Sorry!

---

<div class="post-metadata">

**Author:** ![nacho1998zgz](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@nacho1998zgz](https://forums.unrealengine.com/u/nacho1998zgz)\
**Post date:** [May 15, 2024, 6:47am UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/24 "2024-05-15T06:47:48Z")

</div>

Okay, thank you so much for your time!  
PS: In this case I was actually trying with UE5.3, but maybe is something recent too

---

<div class="post-metadata">

**Author:** ![Rectus\_SA](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Rectus\_SA](https://forums.unrealengine.com/u/Rectus_SA)\
**Post date:** [May 15, 2024, 2:07pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/25 "2024-05-15T14:07:56Z")

</div>

The issue seems to be that I wasn’t handling multiple images being open simultaneously in the same OpenXR swapchain (which is allowed by the spec). I guess Epic must have improved on the pipelining starting from 5.3.

I pushed an update to the layer, see if it works:  
[https://github.com/Rectus/openxr-steamvr-passthrough/releases/tag/v0.2.6-beta](https://github.com/Rectus/openxr-steamvr-passthrough/releases/tag/v0.2.6-beta)

---

<div class="post-metadata">

**Author:** ![nacho1998zgz](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@nacho1998zgz](https://forums.unrealengine.com/u/nacho1998zgz)\
**Post date:** [May 24, 2024, 12:35pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/26 "2024-05-24T12:35:05Z")

</div>

Hi @Rectus_SA sorry for gettin back after a few days. I was able to try it and it really worked! That’s amazing, great job! I tested it with the valve index and it worked like a charm basically out of the box. Then I tried using it with the HTC Vive Pro and 2 but I got the issue that for some reason it is overlapping or weird mixing the images feed from both cameras, resulting in just white/light colors. Any idea or setting that should fix this?

---

<div class="post-metadata">

**Author:** ![Rectus\_SA](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Rectus\_SA](https://forums.unrealengine.com/u/Rectus_SA)\
**Post date:** [May 24, 2024, 1:11pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/27 "2024-05-24T13:11:42Z")

</div>

Unfortunately I haven’t been able to test with any of the Vive Pros, and there is no documentation from either Valve or HTC on how the cameras need to be set up. Does it happen with all the three projection modes (in the settings main page)?

I’d also appreciate any screenshots of the issue.

---

<div class="post-metadata">

**Author:** ![nacho1998zgz](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@nacho1998zgz](https://forums.unrealengine.com/u/nacho1998zgz)\
**Post date:** [May 27, 2024, 7:02am UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/28 "2024-05-27T07:02:35Z")

</div>

Yeah this situation happened for all 3 main projection modes and I tried playing around with the rest of the setting also. Here I attach to screenshots of how it looks the view when projection mode was 2D room view on top of the application TheBlue, in the latest one I discovered that it looks that its mainly taking the right camera on top of the left one (as that’s how it looked when I covered it)

The camera test from steamvr passed so no hardware issue on that regard

 ![Screenshot 2024-05-27 085215](https://d3kjluh73b9h9o.cloudfront.net/original/4X/e/d/4/ed4a8a5dee6aa8474d4c54f9ad52d3a86e7d1ffe.jpeg)

Here you can see the right one is covered

 ![Screenshot 2024-05-27 085020](https://d3kjluh73b9h9o.cloudfront.net/original/4X/1/6/5/165f19f15e584ccd9ebfcd52235f4ef080b032d0.jpeg)

---

<div class="post-metadata">

**Author:** ![Rectus\_SA](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Rectus\_SA](https://forums.unrealengine.com/u/Rectus_SA)\
**Post date:** [May 27, 2024, 9:26am UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/29 "2024-05-27T09:26:16Z")

</div>

Thanks. The SteamVR camera test only shows its own output though. I would like to see what the actual in-game passthrough output looks like, ideally from the SteamVR VR View set to Both Eyes.

I’d also appreciate if you could take screenshots of the device properties of the Vive Pro’s from the passthrough settings menu. In the Debug tab of the menu click the Refresh button, and select the headset from the dropdown. It should show about two screens worth of parameters.

---

<div class="post-metadata">

**Author:** ![nacho1998zgz](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@nacho1998zgz](https://forums.unrealengine.com/u/nacho1998zgz)\
**Post date:** [May 27, 2024, 11:12am UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/30 "2024-05-27T11:12:51Z")

</div>

Thanks for the swift reply!  
Here I attach the screenshots for both eyes from the device passthrough on the VRTemplate project I was using (both cameras uncovered)

 ![Screenshot 2024-05-27 130610](https://d3kjluh73b9h9o.cloudfront.net/original/4X/e/2/d/e2d2d01f4bbe0c9747b7c6b6c4b51e9fccb432a5.jpeg)

And here is the debug output log within the layer app in the headset (Vive Pro 2)

 ![Screenshot 2024-05-27 130737](https://d3kjluh73b9h9o.cloudfront.net/original/4X/f/6/5/f65e303a9cab8451067ae2785bb973818cf11604.png)  
 ![Screenshot 2024-05-27 130824](https://d3kjluh73b9h9o.cloudfront.net/original/4X/7/7/4/77440278270bdeeb0d630c7cb068d2720a08e569.png)

---

<div class="post-metadata">

**Author:** ![Rectus\_SA](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Rectus\_SA](https://forums.unrealengine.com/u/Rectus_SA)\
**Post date:** [May 27, 2024, 2:51pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/31 "2024-05-27T14:51:15Z")

</div>

Thanks. I’ll have to dive into it and see if I can find the issue.

---

<div class="post-metadata">

**Author:** ![Rectus\_SA](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Rectus\_SA](https://forums.unrealengine.com/u/Rectus_SA)\
**Post date:** [June 5, 2024, 2:36pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/32 "2024-06-05T14:36:20Z")

</div>

I released an update. There were several issues with the vertical frame setup, and my code was assuming that the frame size was identical with the different distortion modes. I managed to get all the projection modes to output something that at least looks correct.

[https://github.com/Rectus/openxr-steamvr-passthrough/releases/tag/v0.2.7-beta](https://github.com/Rectus/openxr-steamvr-passthrough/releases/tag/v0.2.7-beta)

---

<div class="post-metadata">

**Author:** ![Szlimsedi](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Szlimsedi](https://forums.unrealengine.com/u/Szlimsedi)\
**Post date:** [March 27, 2025, 3:44pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/33 "2025-03-27T15:44:34Z")

</div>

Hello, sorry to revive the thread. I read through multiple times and tried to wrap my head around everything. I am testing on an HTC vive pro 2 and with the post process material in UE5.4. And it is working as expected with the 2D room view but when I switch to custom or 3D stereo I cannot see anything from the passthrough. I might be missing some simple thing but I would appreciate any insight on this.

Thank you for the amazing work though!

---

<div class="post-metadata">

**Author:** ![Rectus\_SA](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Rectus\_SA](https://forums.unrealengine.com/u/Rectus_SA)\
**Post date:** [March 27, 2025, 6:38pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/34 "2025-03-27T18:38:54Z")

</div>

I still haven’t been able to test it on real Vive Pro hardware. It’s likely another part of my custom projection code that still is broken. I’ll have another go at looking for issues.

It’s great that the 2D Room View mode works though. Thanks for testing it out and for reporting the issue!

One thing that would help is if you could check that all the device properties in the screenshots above matches yours, in case there are different hardware revisions with varying camera setups. Also if there are any error messages in the log.

---

<div class="post-metadata">

**Author:** ![Szlimsedi](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Szlimsedi](https://forums.unrealengine.com/u/Szlimsedi)\
**Post date:** [March 28, 2025, 10:03am UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/35 "2025-03-28T10:03:46Z")

</div>

Thanks for the quick reply. I took some screenshots of the device properties and the log as well

 ![passtrhough1](https://d3kjluh73b9h9o.cloudfront.net/original/4X/0/5/2/052fcd2bdc9d122fa2339e501d1d82319f0c8bba.png)  
 ![passtrhough2](https://d3kjluh73b9h9o.cloudfront.net/original/4X/c/7/a/c7a9e725a18866a05e9b54db99e2d2dcd436012a.jpeg)  
 ![passtrhough3](https://d3kjluh73b9h9o.cloudfront.net/original/4X/1/2/f/12fb981bed37bc41183ab16b08f60ffbe4881704.jpeg)  
 ![passtrhough4](https://d3kjluh73b9h9o.cloudfront.net/original/4X/a/0/0/a0090b0c60d7ddf38c9b03492f84c6d7f83b4caf.jpeg)

Only difference I could find compared to the screenshots above is with the individual cameras, most notibly in the intrinsics. But take a look, you understand it more than me.

---

<div class="post-metadata">

**Author:** ![Rectus\_SA](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Rectus\_SA](https://forums.unrealengine.com/u/Rectus_SA)\
**Post date:** [March 28, 2025, 6:35pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/36 "2025-03-28T18:35:38Z")

</div>

Thanks! Everything looks normal, the differences must be from factory calibration.

---

<div class="post-metadata">

**Author:** ![Rectus\_SA](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Rectus\_SA](https://forums.unrealengine.com/u/Rectus_SA)\
**Post date:** [April 2, 2025, 7:33am UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/37 "2025-04-02T07:33:57Z")

</div>

I think I have figured out what the issue is. I’m using a fisheye lens model for undistorting SteamVR cameras, but the Vive cameras have non-fisheye lenses. I’m getting plausible results using the regular lens model, but since the Vive Pro 2 doesn’t provide any distortion coefficients, it’s still possible the output won’t be correct.

I’m going to release an update with the fix, as well as an override to force a different lens model. The update has a lot of other changes as well, so I need some time for testing it.

---

<div class="post-metadata">

**Author:** ![Szlimsedi](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Szlimsedi](https://forums.unrealengine.com/u/Szlimsedi)\
**Post date:** [April 3, 2025, 8:39am UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/38 "2025-04-03T08:39:46Z")

</div>

Awesome, thank you for the amazing work! If I still have access to the headset when you roll the update around I will test it!

---

<div class="post-metadata">

**Author:** ![Rectus\_SA](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Rectus\_SA](https://forums.unrealengine.com/u/Rectus_SA)\
**Post date:** [April 5, 2025, 2:25pm UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/39 "2025-04-05T14:25:00Z")

</div>

I got it done: [https://github.com/Rectus/openxr-steamvr-passthrough/releases/tag/v0.3.3-beta](https://github.com/Rectus/openxr-steamvr-passthrough/releases/tag/v0.3.3-beta)

If it doesn’t work, I’d like to see what the input camera frames look like. I added a button to save a frame to file in the debug tab, it will put it in `%LOCALAPPDATA%`

---

<div class="post-metadata">

**Author:** ![Szlimsedi](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Szlimsedi](https://forums.unrealengine.com/u/Szlimsedi)\
**Post date:** [April 7, 2025, 10:29am UTC](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330/40 "2025-04-07T10:29:35Z")

</div>

It does work. It does display passthrough in the other 2 modes now as well. Idk if it’s how it’s supposed to work but the custom 2d view and the 3d stereo view look the same and they both feel more zoomed in compared to the steamvr view and it’s a bit hard to perceive depth. As if I only had one eye open. Tried to grab some stuff on my table meanwhile and missed them for the first time. It feels much more natural in the default steam vr view mode. Made me a little nauseaous. But hey it’s no longer dark in the other two views, thank you for the quick update!

[Previous page](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330.md?page=1)

[Next page](https://forums.unrealengine.com/t/htc-vive-passthrough-camera-ar-vr-development/1186330.md?page=3)
