VR CAVE optimization issue [nDisplay]

Development Programming & Scripting
,
1

Hello, I’ve been trying for a few weeks to optimize the perfomance of Unreal Engine in our VR CAVE, but with very limited success.
I’m running Unreal Engine in our CAVE using nDisplay and Switchboard. At first without any change, the official nDisplay template was running between 10 and 20 FPS. Then I used the Perfectly Optimized template (I only kept DirectX12) and now it runs at 35 to 45 FPS. I created a new nDisplay configuration, and tried different variations, like putting one viewport per node, or putting two viewports in one node, etc. But it never really helped achieve stable 60 FPS (which is the max output that our projectors can deliver for one eye).

I used the Stat UNIT command to gather some statistics :

When the whole CAVE is running (using the one viewport per node config) :
(Stats from the Control PC node, but the other nodes have more or less the same values)
FPS : 31
Frame : 33 ms
Game : 25 ms
Draw : 27 ms
RHIT : 29 ms
GPU Time : 9 ms

When only the nodes of one Render PC are running :
FPS : 36
Frame : 27 ms
Game : 9 ms
Draw : 27 ms
RHIT : 21 ms
GPU Time : 2 ms

When only one node is running :
FPS : 60
Frame : 16 ms
Game : 4 ms
Draw : 16 ms
RHIT : 13 ms
GPU Time : 3 ms

When only one node is running and Render Sync Policy is set to None:
FPS : 280
Frame : 3 ms
Game : 2 ms
Draw : 3 ms
RHIT : 1 ms
GPU Time : 2 ms

When using the 2 Render PCs (no Control PC) and two viewports per node in the nDisplay config :

FPS : 36
Frame : 25 ms
Game : 9 ms
Draw : 26 ms
RHIT : 20 ms
GPU Time : 3 ms

Our current CAVE setup :

Our CAVE is composed of 3 different faces : 2 walls and 1 floor. Each face is split into two parts, so we have a total of 6 screens. Each screen’s image is projected using a BARCO F80-09 with active stereo enabled.

Projector config :
Input 1 : DisplayPort 1 | EDID : 2560x1600@60Hz
Input 2 : DisplayPort 2 | EDID : 2560x1600@60Hz
Stereo : Sync Delay : 0 | Dark Time : 1600
Display Mode : Display Setup : ActiveStereo | Frequency 120Hz

Computers config :

We have 3 computers running on a local network for our CAVE. 2 are for render and the other one is for control. Render PC config (same for both) :

CPU Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz 3.19 GHz (2 processors)
GPU 2 x NVIDIA RTX A6000
RAM 128 GB
Synchro Card: NVIDIA QUADRO G-SYNC II
OS : Windows 10
DirectX : DirectX12
Internet access : No

Control PC :
CPU Intel(R) Xeon(R) W-2225 CPU @ 4.10GHz (4.10 GHz)
GPU NVIDIA RTX A4000
RAM 64 GB
Synchro Card : None
OS : Windows 11
DirectX : DirectX12
Internet access : Yes

On each Render PC, one of the GPUs is connected to 2 projectors (4 outputs) and the other to one projector (2 outputs), so each Render PC must render 6 windows. We are tracking the user’s head movement using Optitrack.

Config nDisplay :

GPU : Each node is set to use the graphic adapter the output is plugged to.
Render Sync Policy: Ethernet
I tried using the NVIDIA Mosaic with Render Sync Policy set to NVIDIA and all the viewports under one node set to fullscreen, but it didn’t improve the performance.
All the viewports have the same dimensions as the nodes they are in.
Render API : DX12
Render mode : Mono

I tried launching it as both a standalone and a packaged game, but saw no significant improvement when launching as a packaged game.
All the LiveLink/Optitrack components were disabled for the tests.

I suspect the bottleneck might be related to the CPU or a synchronization issue, but I’m not sure how to address it.

Feel free to ask if you need more details. Any suggestions to improve performance are very welcome!

2

Hey there! Im the creator of Perfectly Optimized Template, first of all, thanks for using the template and im glad it was able to help even just a little. I see that you need stable 60 FPS on this project. Though I am not very experienced on this technology, I’ll try and help the best I can.

Here is my thought process:
Assuming we look at the stats of the whole CAVE running for real world load, your GPU time is pretty low, which means your correct about the CPU bottleneck. GPU isn’t to blame for the performance.

I’ve also noticed that your Draw time stays the same, around 26 - 27ms regardless of node count.

Single-PC (36 FPS) and full-cluster (31 FPS) have nearly identical Draw/RHIT times (27 - 29ms). This could be the cost of each Render PC submitting 6 separate viewport renders (6 windows/PC) to the RHI thread, causing strain on the CPU. This caps you around 36 FPS before cluster sync is even added. So please check that

I see that Game thread time scales directly with node count. nDisplay’s cluster manager does network barrier waits each frame (all nodes must reach sync points before proceeding) which is separate from “Render Sync Policy” (which only governs GPU). This barrier is TCP based and its cost grows with node count and network latency. This is probably why NVIDIA hardware sync didn’t do anything.

And could you clarify a bit about the control PC? if it’s acting as cluster Primary and blocking barriers unnecessarily, test running Primary role on a Render PC instead, Control PC has a weaker CPU/no sync card could be dragging barrier timing

And also Confirm you’re not CPU-bound on game thread logic, 25ms Game thread with LiveLink disabled is high for an otherwise simple scene

Could you please confrim these things and let me know if they helped, though they weren’t much
Have a great day!

3

yk? After typing all of that i realised that you posted this almost a month ago… my bad :sweat_smile: