UE4 into CAVE

Hi all

I’ve been working for a few weeks on UE4 to adapt it into a CAVE environment.
I would like to share my work, in order to have any advices.
I am working for a French laboratory specialized in VR (http://www.institutimage.fr/, http://www.ensam.eu). We own many devices including an “old” CAVE system. For years now we have successfully been using our own software based on OpenSceneGraph for all our devices. We are now trying to adapt UE4 as the engine for our immersive environments.

  1.  Our CAVE presentation
    

Our CAVE is very specific. It’s 4 sided (front, right, left, bottom), each wall measuring 32.7 meters, the stereo is passive so we have to generate 8 images (2 per walls) with a resolution of 14001050 pixel for each projector.
We don’t have any cluster, we are using a single computer equipped with 2 QuadroPlex 7000 (that means 4 Quadro 6000 GPUs). Each Quadro has 2 DVI graphic outputs ; so we’ve got a total of 8 outputs (which fits all our projectors).

With Nvidia driver mosaic software, we have made 4 extended monitors with a resolution of 2800x1050 per desktop. Each desktop is managed by one Quadro and is corresponding to left and right eye. We have to add black strips on the left and right side of each viewport to adapt the resolution to our CAVE walls.

Reso.jpg Move-V3.png

  1.  UE4 adaption
    

Here I would like to share my work on UE4, I hope I will have some feedback.

  •      View matrix
    

I made a VRPN library able to get the position and orientation of the headtracker. Within a C++ plugin I used a custom actor component which is wrapping the VPRN library. Then, I have coded a custom character actor (like first person shooter sample) which possesses the vrpn actor component.
My custom actor character contains a camera, a capsule for collision and 2 custom properties which are the position and the orientation of the headtracker (given by its component).
On tick event, I am updating the properties from the component, then I just have to update the position and rotation and my custom character by calling setActorLocationAndRotation in its blueprint (or by using a setTransform method in C++).
That works well, my custom actor component transformations are updated when I am moving the headtracker.

  •      Off axis projection matrix
    

For a CAVE system we had to compute an off axis projection matrix. This guy helped me a bit (Unreal Engine on a multi projector wall display - Architecture - Epic Developer Community Forums)
Always in my C++ plugin, I made a custom GameEngine class (A new, community-hosted Unreal Engine Wiki - Announcements - Epic Developer Community Forums) and I have override the InitializedHMDDevice() method. Into UE4 C++ code, this method is used to create a FakeStereoRendering device, I decided to create and use my own StereoRendering device instead, inherited from IStereRendering class. From this class I had override GetStereoProjectionMatrix(…) method to build my own off axis projection matrix. (don’t forget that UE4 is using a right handed coordinate system, see D3DXMatrixPerspectiveOffCenterLH function (D3dx9math.h) - Win32 apps | Microsoft Learn)
By using a renderingDevice, the viewport is automatically divided in two, which is perfect for my passive stereo system.

  •      Multi process (one process per wall)
    

I have looked a bit UE4 multiplayer system features, but I don’t think it is fitting well for a multi process application (it was designed for multiplayer games). I decided to use MPI (Message Passing Interface — Wikipédia), it is a free library which permits to launch many processes and allows them to communicate.
So I am creating 4 processes, one for each wall of my CAVE. Each process has a resolution of 2800x1050 and is corresponding to one wall/floor (and one extended desktop). It contains a split screen for left and right eyes. Each eye has a resolution of 1400x1050.
The process 0 is getting the tracker transforms and sending to the other processes with a synchronized broadcast. Then every process computes its view and projection matrix. Depending the process I am adding a rotation, according to the face my process is taking care of (+90 for the right face, -90 for the left face etc….)

  •      Black strips
    

This is very specific to our CAVE, so it may not be very interesting for everyone.
Our resolution of 1400x1050 per eye (our video projection resolution) is not corresponding to our 3x2.7 meter walls. I need a resolution of 1160x1050 to fit. So I have created with UE4 a custom post process material which scales the postprocess render image from 1400x1050 to 1160x1050 output image. Then, I am blending this result with a texture which contains two black strips on the left and right sides. I have to hide (1400 -1160) deformed pixels on each sides of the output image.

  •      Positioning and resolution
    

At least I had to place and set a full screen resolution for each of my process. In my custom character begin play event, I am calling custom methods which are using GetSystemResolution->RequestResolutionChange(…) and -WinX -WinY parameters command to achieve that.

  1.  results
    

It is working well into our CAVE, but the immersion is not perfect. I don’t feel like I am into the 3D scene (it was a far better with OpenSceneGraph). Maybe my computations are false, but the main problem is that the solution is very laggy and running low fps !
With a resolution of 2800x1050 for each of my 4 processes, I have less than 10fps, and less than 20fps with 1400*525 per process.
It is very important that the application must be very reactive (more than 30fps at least) to feel the immersion.
I have tried to launch 4 samples UE4 applications with the same resolution and I’ve got the same results (in terms of fps). So I guess the slowness is not coming from my code.

I have identified that only one graphic card / one GPU is working, and regarding to this posts, it seems that UE4 is not supporting multi-GPU ….

The ideal config should be using a cluster. So I think I am definitely stuck with my hardware config…


Anyway I just wanted to share the workflow I used to adapt UE4 into a CAVE. With a different system (if you have a cluster), maybe it could be useful to someone.
If anyone wants more detail (code, screenshot…), feel free to ask.
If anyone has advices, feel free to tell.

Thx for reading, sorry for my english ^^

It’s good to hear others are also testing the engine in a CAVE. We also come from OpenSceneGraph and are experimenting with UE4 now :slight_smile:
About the performance problen: I haven’t tried this but it used to be possible to say which graphics adapter UE4 uses, maybe this will help: Multiple Graphics Adapters (related to 4.1 QFE / very low fps rendering) - General Discussion - Unreal Engine Forums
We are actually using the UE4 multiplayer features, it’s hard to get it going but it’s worth it for the actor replication features. If you have animating objects or objects with physics then UE4 can be made to sync them for you every frame (but you need to disable all the throttling mechanisms in UE4 so it replicates all the objects that need replicating each frame.)

Hi ,
thank you for reading !!
About the multiplayer physics, yes, you are right. The animated objects (especially physich) won’t be synchronized with mpi, that could be a problem; using multiplayer features can be a solution (but very painfull)
I will read you link, thx for sharing.

Hi there,

The GPU utilization all going to the first GPU has to do with the lack of GPU affinity. By default all contexts on NVidia hardware will go to "the first’ GPU, whichever that may be. You’ll need to use the NV_gpu_affinity OpenGL extension to bind a render context to a physical GPU.

https://www.opengl.org/registry/specs/NV/gpu_affinity.txt

Hi all
thank your for your advice, I managed to assign one process per GPU with GraphicsAdapter UE4 (Multiple Graphics Adapters (related to 4.1 QFE / very low fps rendering) - Community & Industry Discussion - Epic Developer Community Forums
My front process is managed by 1st GPU, my 2nd process is on the 2nd GPU etc… and the perf are now acceptable (I will post some pictures later)
The “ugly” part is that I need 4 differents consolevariables.ini files for each my processes, so I had to duplicate my exe.
I have looked on the window creation in UE4 source code but I really don’t want to modify the engine code to add some affinity.

Event with the The “immersion feeling” sensation is still false, I am pretty sure I have a scale problem to resolve.

Thank you again for your feedback, it’s great to see people working with UE4 on a CAVE.

Hi,
here my work in progress, if someone is interested

some video,

the results are pretty amazing thanks to the UE4 renderings.
I still have to work on

  • active stereo (the system is currenlty using passive stereo system)
  • collision detection while navigating
  • event management between UE4 visual thread

That’s great overseb, I wish had the low level know how to implement this where I work (we have a big CAVE2 facility).

Hi, yes I’m very interested. I’m really wanting to build a ‘cave’ for my work to showcase some 3d scenes we are working on. Is there any downloadable examples of a simple setup, without head tracking, just 4 wall outputs and possibly floor and ceiling?
Thanks, really awesome work!!!

Hi thank you !

sorry but no downloadable example yet … but my goal it to put everything on opensource !I just need the time to do that and I would to like to complete few features before (active stereo…)

Any news?

Hi,

Great work and thanks for you effort and for sharing!

I’ll be sure that everyone would appreciate some more details on your implementation :slight_smile:
Please keep us updated! :slight_smile:

Any news? overseb how can I contact you?

Hey Overseb,

i am very interested in this, thus i would like to know whether you could post a little example code on how to do the message passing interface.

I want to render 2 screens with 2 projectors each(quad buffer) using only one single machine (2 quadro cards). And until yet i could not find out how create a new game window and attach the gameviewport client of a camera to it or to do it the multiplayer way in cooked version (using auto activate for player in a camera it works at least when i simulate that stuff).

Could you please help me out?

Thx in advance

I really need this information soon.

is there any other way to get in contact with each other?

I’m interested too.
It’s great work, and it will be wonderful to see some tech details and explanations.

Hi, please visit the “iLocalPlayer” UE4 C++ Plugin: iLocalPlayer4UE in Code Plugins - UE Marketplace;

Functions of this plugin :1. Side-by-Side Stereo 3D implementing; 2. Asymmetric projection(or named asymmetric view frustum) implementing.

Hope this helps you.

Hi all
I am sorry I didn’t have the opportunity to work on UE4 since few months, my project was on standby.
However, I will try to find some time to tr iLocalPlayer plugin.
Thank you for your answer.

If someone did adapt UE4 for a CAVE successfully, don’t hesitate to share

thank you for your advice

i am building A Cave with 5 walls and use UE4, can you give me your config of nDispaly to me?

Hi. I’m currently working on something similar for my bachelours and I would relly love to get in touch with you and ask a few questions if possible. Please let me knnow if I can contact you. Thanks