VR Video Orchestra APK with Oculus Quest 2 Tips And Tricks XP

anodin
Wed, 15 Nov 2023 17:24:09 +0000
Mobile

Hi community,
I created a VR application last year on Oculus Quest 2 with Unreal Engine 4.27.2
It’s a serious game where you take place as a conductor, it integrate 4 larges 360 videos captured with the musician of the Montreal Symphony Orchestra.
Here a video of the final resuly :
I'm happy with the final result, and now I'm also happy to share with the community the troubleshot by where I past through, i hope it can help some people how gonna start this aventure.
First i dev only for Oculus Quest 2, add a different headset will need to create different build for each one…
And at the time VR was not implemented on UE5, and i can said it’s better to stay on old stable product that go on fancy new engine and get stuck forever.

1- Visual studio 2019
Pption Game / tablet

TechSpot

Visual Studio 2019

Developer tools and services for any platform with any language. IDE, DevOps, code editor & more.

2- Oculus + Android Studio :
https://developer.oculus.com/documentation/unreal/unreal-quick-start-guide-quest/?device=QUEST

Android Developers

Download Android Studio & App Tools - Android Developers

Android Studio provides app builders with an integrated development environment (IDE) optimized for Android apps. Download Android Studio today.

I installed Android API 31
I got save by this video ! At the end he show how rename and rewrite a java file : d8.jar to dx.jar

Unreal Engine - Set up SDK, NDK and JDK for VR and other Android Devices

Here a screeshot of my Android preference :

Yes i installed android version 31 but i write 29
I allow large OBB file because i got large video to feed.
But i you can try to keep you app light and package everything in the .apk, it easier to export and install.
Important is the Build section :
I really try all combinaison, and i try first new vulkan and at the end i have to go back to the calssic OpenGL 3 because Vulkan didn t allow my to use video inside Widget, it get black ! It was a terrible situation for me because all the Ui on the Conductor pupitre was done with Widget and discover that at the end of the projet, i had to delete all the widget and redo it with Blueprint.

Here my Oculus VR setting
Lot to say about colors, i didn t have the same color on the oculus plug in on my computer and since my apk exported, so you can imagine how had it was the back and forth…
Video :
For optimization we cut the 360 video to render only the part where was the musician move as video, the ‘room’ was render as Still image but fade each other because at the beginning and at the end, the lighting of the room change.

So use Color rec 709 in Ae.
In the Oculus doc they say that oculus is a Rec2020, but i used rec 709 too.
We export a lot a time video to match Image and video colors
A lot of coodec was used too.
I try vp8, vp9, av1, mkv, ogg, 3gp, too much to return to only the mp4, here the command line for ffmpeg
ffmpeg -i “input.mov” -c:v libx264 -preset slow -crf 17 -x264-params mvrange=511 -pix_fmt yuv420p -an -movflags faststart “output.mp4”

Audio :
Ochestra mean good audio. We thought ambisonic sound format will help us in this VR spacialisation… and noo
Unreal Engine only supports first-order ambisonic assets = So only 4 channel = Mono + L-R (AKA mid-side stereo)
So after this desilution the Sound Studio had to recreate all the audio source inside of Unreal and animated it depend of the choose of the user instrument selection. You can see the audio source in the screeneshot and the Sphere Mesh Cut.

To blend image and video, we had a nice god ray. When the user select an instrument, the partition appear over them.
What is the most performance consuming in VR mobile app ? TRANSPARENCY.
i just can say avoid it.

Important Foreward rendring for performance.
AntiAliasing : MSAA

I got problem with Scene capture in mobile, we try to avoid video at the begining, for the partition just a camera animated in X axis, could have make the job. but this is an old bug report :

Mobile VR - Multiview: Render to texture is black. XR Development

Hi to everybody, this is the first time I ask for help here on the community. I have a big problem in Mobile VR. Hardware requirements for the project… GearVR (on Samsung S10e). Necessary software setup: Forward render MultiView Oculus for Quest/GearVR Target: I am trying to have back mirrors in a drivable car: when I activate Multiview (to reach needed performances), the “mirror” goes black. The mirror is created via a Scenecapture 2D + rendertexture + unlit material. Issue retreavi…

A chance that it was a Unlut scene.
For material quality don t forget to use Full Precision inside Mobile option.

I learned a lot during this projet, but i’m not ready to do it again.
I hope it helps a bit some of you.

2 posts - 1 participant